diff --git a/.agents/skills/build-from-issue/SKILL.md b/.agents/skills/build-from-issue/SKILL.md index 4eb34036b..89d9e0421 100644 --- a/.agents/skills/build-from-issue/SKILL.md +++ b/.agents/skills/build-from-issue/SKILL.md @@ -1,6 +1,8 @@ --- name: build-from-issue description: Given a GitHub issue number, plan and implement the work described in the issue. Supports direct user requests and unattended queue processing through the `agent:*` workflow labels. Includes tests, documentation updates, and PR creation. Trigger keywords - build from issue, implement issue, work on issue, build issue, start issue. +metadata: + internal: true --- # Build From Issue diff --git a/.agents/skills/build-openshell-mxc-windows/SKILL.md b/.agents/skills/build-openshell-mxc-windows/SKILL.md index d61fc555e..5f0d617b0 100644 --- a/.agents/skills/build-openshell-mxc-windows/SKILL.md +++ b/.agents/skills/build-openshell-mxc-windows/SKILL.md @@ -1,6 +1,8 @@ --- name: build-openshell-mxc-windows description: Maintain and validate OpenShell's build-only Windows MSVC lane for x64 and ARM64. Use when working on Windows compilation, `windows:*` mise tasks, unsupported Windows compute-driver contracts, or Windows build reports. This skill does not implement Docker, Kubernetes, Podman, VM, MXC driver, policy translation, MSI, service, or supervisor runtime support on Windows. +metadata: + internal: true --- # Build OpenShell-MXC for Windows @@ -231,7 +233,7 @@ crypto dependency builds. | `windows:build:arm64` | Release-builds `openshell-gateway.exe` and `openshell.exe` for ARM64. | | `windows:test:x64` | Runs native x64 workspace tests with `--no-fail-fast`, excluding unsupported Windows packages as top-level workspace targets. | | `windows:test:arm64` | Runs native ARM64 workspace tests with `--no-fail-fast` and the same package exclusions. Rejects non-ARM64 hosts. | -| `windows:test:unsupported:x64` | Re-runs focused `openshell-server` tests for unsupported Windows driver behavior. | +| `windows:test:unsupported:x64` | Re-runs focused `openshell-gateway` tests for unsupported Windows driver behavior. | | `windows:test:unsupported:arm64` | Re-runs the same focused contracts natively on ARM64. Rejects non-ARM64 hosts. | | `windows:artifacts` | Reports size and SHA256 for release artifacts that exist. | | `windows:ci` | Runs the full ordered x64-host Windows CI lane, plus ARM64 check/build when not skipped. | @@ -248,10 +250,10 @@ Windows must continue to reject unsupported compute drivers clearly. | Driver | Windows build behavior | Runtime behavior | |---|---|---| -| Docker | Driver crate excluded; server config contract retained. | Gateway construction returns unsupported. | -| Kubernetes | Driver crate excluded; server config contract retained. | Gateway construction returns unsupported. | -| Podman | Driver crate excluded; server config contract retained. | Gateway construction returns unsupported. | -| VM | Driver crate excluded from workspace validation. | Gateway construction returns unsupported. | +| Docker | Driver crate excluded; gateway registration stub retained. | Gateway construction returns unsupported. | +| Kubernetes | Driver crate excluded; gateway registration stub retained. | Gateway construction returns unsupported. | +| Podman | Driver crate excluded; gateway registration stub retained. | Gateway construction returns unsupported. | +| VM | Driver crate excluded; gateway registration stub retained. | Gateway construction returns unsupported. | The focused contract tasks for either native architecture run: diff --git a/.agents/skills/build-openshell-mxc-windows/reference.md b/.agents/skills/build-openshell-mxc-windows/reference.md index 4a5f2668a..16b9a4858 100644 --- a/.agents/skills/build-openshell-mxc-windows/reference.md +++ b/.agents/skills/build-openshell-mxc-windows/reference.md @@ -85,7 +85,7 @@ Windows is a build target only. These runtimes remain unsupported: Rules: -- Keep config/library stubs where the gateway needs them. +- Keep registration stubs in the gateway composition crate where the gateway needs them. - Return clear unsupported errors at runtime. - Do not build standalone Windows driver binaries. - Do not add Docker Desktop, WSL, Hyper-V, Podman machine, Podman Desktop, or @@ -180,7 +180,7 @@ blocked dependency. ### Focused tests report many filtered-out tests This is expected for `windows:test:unsupported:x64`. Cargo runs one named test -and filters the other `openshell-server` tests. Report these as filtered, not +and filters the other `openshell-gateway` tests. Report these as filtered, not ignored. ## Reporting Counts diff --git a/.agents/skills/create-github-issue/SKILL.md b/.agents/skills/create-github-issue/SKILL.md index 1ab92a6ed..2510b7927 100644 --- a/.agents/skills/create-github-issue/SKILL.md +++ b/.agents/skills/create-github-issue/SKILL.md @@ -1,6 +1,8 @@ --- name: create-github-issue description: Create GitHub issues using the gh CLI. Use when the user wants to create a new issue, report a bug, request a feature, or create a task in GitHub. Trigger keywords - create issue, new issue, file bug, report bug, feature request, github issue. +metadata: + internal: true --- # Create GitHub Issue diff --git a/.agents/skills/create-github-pr/SKILL.md b/.agents/skills/create-github-pr/SKILL.md index d98aba37f..8e14bc5f9 100644 --- a/.agents/skills/create-github-pr/SKILL.md +++ b/.agents/skills/create-github-pr/SKILL.md @@ -1,6 +1,8 @@ --- name: create-github-pr description: Create GitHub pull requests using the gh CLI. Use when the user wants to create a new PR, submit code for review, or open a pull request. Trigger keywords - create PR, pull request, new PR, submit for review, code review. +metadata: + internal: true --- # Create GitHub Pull Request diff --git a/.agents/skills/create-rfc/SKILL.md b/.agents/skills/create-rfc/SKILL.md index f767e4758..6df87ef31 100644 --- a/.agents/skills/create-rfc/SKILL.md +++ b/.agents/skills/create-rfc/SKILL.md @@ -1,6 +1,8 @@ --- name: create-rfc description: Create OpenShell RFC proposals in rfc/ from a design request. Use when the user asks to write, draft, start, create, or update an RFC, Request for Comments, architecture proposal, API proposal, process proposal, or cross-cutting design proposal that should follow the OpenShell RFC process and template. +metadata: + internal: true --- # Create RFC diff --git a/.agents/skills/create-spike/SKILL.md b/.agents/skills/create-spike/SKILL.md index 01afee91e..baafd173d 100644 --- a/.agents/skills/create-spike/SKILL.md +++ b/.agents/skills/create-spike/SKILL.md @@ -1,6 +1,8 @@ --- name: create-spike description: Investigate a plain-language problem description by deeply exploring the codebase, then create a structured GitHub issue with technical findings. Prequel to build-from-issue — maps vague ideas to concrete, buildable issues. Trigger keywords - spike, investigate, explore, research issue, technical investigation, create spike, new spike, feasibility, codebase exploration. +metadata: + internal: true --- # Create Spike diff --git a/.agents/skills/debug-inference/SKILL.md b/.agents/skills/debug-inference/SKILL.md deleted file mode 100644 index 08462a475..000000000 --- a/.agents/skills/debug-inference/SKILL.md +++ /dev/null @@ -1,408 +0,0 @@ ---- -name: debug-inference -description: Debug why inference.local, direct external inference, or supervisor-only system inference is failing. Use when the user cannot reach a local model server, has provider base URL issues, sees inference verification failures, hits protocol mismatches, or needs to diagnose inference on local vs remote gateways. Trigger keywords - debug inference, inference.local, system inference, sandbox-system, local inference, ollama, vllm, sglang, trtllm, NIM, inference failing, model server unreachable, failed to verify inference endpoint, host.openshell.internal. ---- - -# Debug Inference - -Diagnose why OpenShell inference is failing and recommend exact fix commands. - -Use `openshell` CLI commands to inspect the active gateway, provider records, managed inference config, and sandbox behavior. Use a short sandbox probe when needed to confirm end-to-end routing. - -## Overview - -OpenShell supports three inference paths. Diagnose the correct one first. - -1. **Managed inference** through `https://inference.local` - - Configured by `openshell inference set` - - Shared by every sandbox on the active gateway - - Credentials and model are injected by OpenShell -2. **Direct external inference** to hosts like `api.openai.com` - - Controlled by `network_policies` - - Requires the application to call the external host directly - - Requires provider attachment and network access to be configured separately -3. **System inference** used by platform functions - - Configured by `openshell inference set --system` - - Uses the `sandbox-system` route - - Consumed in-process by the sandbox supervisor and not exposed to sandbox user code through `inference.local` - -For local or self-hosted engines such as Ollama, vLLM, SGLang, TRT-LLM, and many NIM deployments, the most common managed inference pattern is an `openai` provider with `OPENAI_BASE_URL` pointing at a host the gateway can reach. - -## Prerequisites - -- `openshell` is on the PATH -- The active gateway is running -- You know the failing setup, or can infer it from commands and config - -## Tools Available - -Use these commands first: - -```bash -# Which gateway is active, and can the CLI reach it? -openshell status - -# Show both the user-facing and system inference routes -openshell inference get - -# Show only the supervisor-only system route -openshell inference get --system - -# Inspect the provider record referenced by the relevant route -openshell provider get - -# Inspect gateway topology details when remote/local confusion is suspected -openshell gateway info - -# Run a minimal end-to-end probe from a sandbox -openshell sandbox create -- curl https://inference.local/v1/chat/completions --json '{"messages":[{"role":"user","content":"hello"}],"max_tokens":10}' -``` - -## Workflow - -When the user asks to debug inference, run diagnostics automatically in this order. Stop and report findings as soon as a root cause is identified. - -### Determine Context - -Establish these facts first: - -1. Is sandbox code calling `https://inference.local`, is the application calling a direct external host, or is a platform function using system inference? -2. Which gateway is active, and is it local, remote, or cloud? -3. Which provider, model, and timeout are configured for the relevant route? -4. Is the upstream local to the gateway host, or somewhere else? - -### Step 0: Check the Active Gateway - -Run: - -```bash -openshell status -openshell gateway info -``` - -Look for: - -- Active gateway name and endpoint -- Whether the gateway is local or remote -- Whether `host.openshell.internal` would point to the local machine or a remote host - -Common mistake: - -- **Laptop-local model + remote gateway**: `host.openshell.internal` points to the remote gateway host, not your laptop. A laptop-local Ollama or vLLM server will not be reachable without a tunnel or shared reachable network path. - -### Step 1: Check Whether the Relevant Route Is Configured - -Run: - -```bash -openshell inference get -openshell inference get --system -``` - -Interpretation: - -- `openshell inference get` shows both the user-facing `inference.local` route and the system route. `--system` isolates the system route. -- **The `inference.local` route is `Not configured`**: managed inference has no backend. Configure it without `--system`: - - ```bash - openshell inference set --provider --model - ``` - -- **System inference is `Not configured`**: platform functions have no system backend. Configure it separately: - - ```bash - openshell inference set --system --provider --model - ``` - -- **Provider, model, and timeout shown**: Continue to provider inspection for the relevant route. - -### Step 2: Inspect the Provider Record - -Run: - -```bash -openshell provider get -``` - -Check: - -- Provider type matches the client API shape and is supported for managed inference - - `openai` for OpenAI-compatible engines such as Ollama, vLLM, SGLang, TRT-LLM, and many NIM deployments - - `anthropic` for Anthropic Messages API - - `nvidia` for NVIDIA-hosted OpenAI-compatible endpoints - - `deepinfra` for DeepInfra's OpenAI-compatible endpoint - - `google-vertex-ai` for Vertex AI; Claude models use Anthropic Messages and other models use OpenAI Chat Completions - - `aws-bedrock` only through a configured Bedrock-compatible bridge today -- Required credential key exists -- `*_BASE_URL` override is correct when using a self-hosted endpoint - -Fix examples: - -```bash -openshell provider create --name ollama --type openai --credential OPENAI_API_KEY=empty --config OPENAI_BASE_URL=http://host.openshell.internal:11434/v1 - -openshell provider update ollama --credential OPENAI_API_KEY=empty --config OPENAI_BASE_URL=http://host.openshell.internal:11434/v1 -``` - -`provider update` preserves the provider type and does not accept `--type`. Prefer bare credential keys, such as `--credential OPENAI_API_KEY`, when reading a real secret from the CLI environment. - -### Step 3: Check Local Host Reachability - -For host-backed local inference, confirm the upstream server: - -- Binds to `0.0.0.0`, not only `127.0.0.1` -- Runs on the same machine as the gateway -- Is reachable through `host.openshell.internal`, the host's LAN IP, or another reachable hostname - -Common mistakes: - -- **Base URL uses `127.0.0.1` or `localhost`**: usually wrong for managed inference. Replace with `host.openshell.internal` or the host's LAN IP. -- **Server binds only to loopback**: reconfigure it to bind to `0.0.0.0`. -- **Inference engine runs as a system service**: changing the bind address may require updating the service configuration and restarting the service before the new listener becomes reachable. - -### Step 4: Check Request Shape - -User-facing managed inference only works for `https://inference.local` and supported inference API paths. - -Supported patterns include: - -- `POST /v1/chat/completions` -- `POST /v1/completions` -- `POST /v1/responses` -- `POST /v1/embeddings` -- `POST /v1/messages` -- `GET /v1/models` -- `GET /v1/models/*` -- `POST /model/{modelId}/invoke` for bridge-fronted `aws-bedrock` - -Common mistakes: - -- **Wrong scheme**: `http://inference.local` instead of `https://inference.local` -- **Unsupported path**: request does not match a known inference API -- **Protocol mismatch**: Anthropic client against an `openai` provider, or vice versa -- **Provider-specific mismatch**: Vertex Claude requests must use `/v1/messages`; other Vertex models currently use `/v1/chat/completions`; Bedrock uses its model-in-path invoke shape - -Fix guidance: - -- Use a supported path and provider type -- Point OpenAI-compatible SDKs at `https://inference.local/v1` -- If the SDK requires an API key, pass any non-empty placeholder such as `test` - -### Step 5: Probe from a Sandbox - -This probe validates the user-facing `inference.local` route. It does not exercise supervisor-only system inference. - -Run a minimal request from inside a sandbox: - -```bash -openshell sandbox create -- curl https://inference.local/v1/chat/completions --json '{"messages":[{"role":"user","content":"hello"}],"max_tokens":10}' -``` - -Interpretation: - -- **`cluster inference is not configured`**: set the managed gateway route with `openshell inference set` -- **`connection not allowed by policy`** on `inference.local`: unsupported method or path -- **`no compatible route`**: provider type and client API shape do not match -- **Connection refused / upstream unavailable / verification failures**: base URL, bind address, topology, or credentials are wrong - -For system inference failures, inspect the platform function and sandbox supervisor/network logs after confirming `openshell inference get --system`. User code cannot call the `sandbox-system` route directly. - -### Step 6: Reapply or Repair the Managed Route - -After fixing the provider, use `update` for a partial change or `set` to replace the route: - -```bash -openshell inference set --provider --model -openshell inference update --provider -openshell inference update --model -openshell inference update --timeout 120 -``` - -Add `--system` to target the system route. Without it, these commands target `inference.local`. A timeout of `0` uses the 60-second default; increase it for models with long reasoning or idle streaming phases. - -If the endpoint is intentionally offline and you only want to save the config: - -```bash -openshell inference set --provider --model --no-verify -``` - -Use `--no-verify` only when the endpoint is intentionally offline or the provider protocol cannot be verified, such as the current bridge-fronted Bedrock flow. Inference updates are hot-reloaded to running sandboxes within about 5 seconds by default. - -### Step 7: Diagnose Direct External Inference - -If the application calls `api.openai.com`, `api.anthropic.com`, or another external host directly, this is not a managed inference issue. - -Check instead: - -1. The application is configured to call the external hostname directly -2. A provider with the needed credentials exists -3. The sandbox has that provider attached (`openshell sandbox provider list [name]`) -4. `network_policies` allow that host, port, and HTTP rules - -If the response reports `credential_endpoint_mismatch`, the provider is attached -but its credential profile does not authorize that request recipient. Run -`openshell provider get ` to identify the provider type, then -inspect its profile endpoints with -`openshell provider profile export -o yaml`. That export uses the current -workspace scope; add `--global` when the provider was created with -`--global-profile`. Compare the profile's endpoint host, port, and path with the -direct request. Correct the provider selection or profile endpoint when that -recipient is intentional. Do not widen the sandbox network policy to work around -the mismatch: policy admission and credential endpoint authorization are -separate checks, and the provider profile should authorize only intended -credential recipients. - -If the response reports `request_authority_mismatch`, compare the HTTP request -authority with the CONNECT tunnel endpoint. The host and effective port must -match. For a tunnel to `api.example.com:8443`, send -`Host: api.example.com:8443`; omitting the non-default port makes the request -authority use the transport default and OpenShell rejects it. An absolute-form -request target must use the same authority. - -Attach or detach a provider on an existing sandbox with `openshell sandbox provider attach ` and `openshell sandbox provider detach `. - -Use the `generate-sandbox-policy` skill when the user needs help authoring policy YAML. - -## Fix: Local Host Inference Timeouts (Firewall) - -Use this fix when a sandbox can reach `https://inference.local`, but OpenShell reports an upstream timeout against a host-local backend such as Ollama. - -Example symptom: - -```json -{"error":"request to http://host.docker.internal:11434/v1/models timed out"} -``` - -### When This Happens - -This failure commonly appears on Linux hosts that: - -- Run the OpenShell gateway in Docker -- Route `inference.local` to a host-local OpenAI-compatible endpoint such as Ollama -- Have a host firewall or networking configuration that denies container-to-host traffic by default - -In this case, OpenShell routing is usually working correctly. The failing hop is container-to-host traffic on the backend port. - -### Why CoreDNS Is Not the Cause - -This is not the same issue as the Colima CoreDNS fix. - -OpenShell injects `host.docker.internal` and `host.openshell.internal` into sandbox workloads when the selected compute platform supports it. That path bypasses runtime DNS lookup. If the request still times out, the usual cause is host firewall or network policy, not DNS. - -### Verify the Problem - -1. Confirm the model server works on the host: - - ```bash - curl -sS http://127.0.0.1:11434/v1/models - ``` - -2. Confirm the host gateway address also works on the host: - - ```bash - curl -sS http://172.17.0.1:11434/v1/models - ``` - -3. Test the same endpoint from a gateway or sandbox container on the Docker network: - - ```bash - docker ps --filter name=openshell --format '{{.Names}}' - docker exec wget -qO- -T 5 http://host.docker.internal:11434/v1/models - ``` - -If steps 1 and 2 succeed but step 3 times out, the host firewall or network configuration is blocking the container-to-host path. - -### Fix - -Allow the Docker bridge network used by the OpenShell gateway and sandbox containers to reach the host-local inference port. The exact command depends on your firewall tooling (iptables, nftables, firewalld, UFW, etc.), but the rule should allow: - -- **Source**: the Docker bridge subnet used by OpenShell containers (commonly `172.18.0.0/16`) -- **Destination**: the host gateway IP injected into sandbox workloads for `host.docker.internal` (commonly `172.17.0.1`) -- **Port**: the inference server port (e.g. `11434/tcp` for Ollama) - -To find the actual values on your system: - -```bash -# Docker bridge subnet for the OpenShell network -docker network inspect $(docker network ls --filter name=openshell -q) --format '{{range .IPAM.Config}}{{.Subnet}}{{end}}' - -# Host gateway IP visible from inside the container -docker exec cat /etc/hosts | grep host.docker.internal -``` - -Adjust the source subnet, destination IP, or port to match your local Docker network layout. - -### Verify the Fix - -1. Re-run the container network check: - - ```bash - docker exec wget -qO- -T 5 http://host.docker.internal:11434/v1/models - ``` - -2. Re-test from a sandbox: - - ```bash - curl -sS https://inference.local/v1/models - ``` - -Both commands should return the upstream model list. - -### If It Still Fails - -- Confirm the backend listens on a host-reachable address: `ss -ltnp | rg ':11434\b'` -- Confirm the provider points at the host alias path you expect: `openshell provider get ` -- Confirm the active inference route: `openshell inference get` -- Inspect sandbox logs for upstream timeout details: `openshell logs --since 10m` - -## Common Failure Patterns - -| Symptom | Likely cause | Fix | -|---------|--------------|-----| -| `openshell inference get` shows `Not configured` | No managed inference route configured | `openshell inference set --provider --model ` | -| System inference is `Not configured` | Platform-only route has no backend | `openshell inference set --system --provider --model ` | -| `failed to verify inference endpoint` | Bad base URL, wrong credentials, wrong provider type, or upstream not reachable | Fix provider config, then rerun `openshell inference set`; use `--no-verify` only when the endpoint is intentionally offline | -| Base URL uses `127.0.0.1` | Loopback points at the wrong runtime | Use `host.openshell.internal` or another gateway-reachable host | -| Local engine works only when gateway is local | Gateway moved to remote host | Run the engine on the gateway host, add a tunnel, or use direct external access | -| `connection not allowed by policy` on `inference.local` | Unsupported path or method | Use a supported inference API path | -| `no compatible route` | Provider type does not match request shape | Create or select a provider of the matching type, or change the client API | -| `inference.local` works but a platform function fails | User route is configured but `sandbox-system` is missing or wrong | `openshell inference get --system`; configure or update with `--system`; inspect supervisor logs | -| Direct call to external host is denied | Missing policy or provider attachment | Update `network_policies` and launch sandbox with the right provider | -| Direct call returns `credential_endpoint_mismatch` | Attached provider profile does not authorize the request host, port, or path | Inspect the provider profile endpoints; select or update the profile only if it intentionally authorizes that recipient | -| Direct call returns `request_authority_mismatch` | HTTP authority does not match the CONNECT host and effective port | Include the explicit non-default port in `Host` and use the same authority in absolute-form targets | -| SDK fails on empty auth token | Client requires a non-empty API key even though OpenShell injects the real one | Use any placeholder token such as `test` | -| Upstream timeout from container to host-local backend | Host firewall or network config blocks container-to-host traffic | Allow the Docker bridge subnet to reach the inference port on the host gateway IP (see firewall fix section above) | - -## Full Diagnostic Dump - -Run this when you want a compact report before deciding on a fix: - -```bash -echo "=== Gateway Status ===" -openshell status - -echo "=== Gateway Info ===" -openshell gateway info - -echo "=== Managed Inference ===" -openshell inference get - -echo "=== System Inference Only ===" -openshell inference get --system - -echo "=== Providers ===" -openshell provider list - -echo "=== Selected Provider ===" -openshell provider get - -echo "=== Sandbox Probe ===" -openshell sandbox create -- curl https://inference.local/v1/chat/completions --json '{"messages":[{"role":"user","content":"hello"}],"max_tokens":10}' -``` - -When you report back, state: - -1. Which inference path is failing (`inference.local`, direct external, or system inference) -2. Whether gateway topology is part of the problem -3. The most likely root cause -4. The exact fix commands the user should run diff --git a/.agents/skills/fix-security-issue/SKILL.md b/.agents/skills/fix-security-issue/SKILL.md index ea3a12bfb..1452cbe66 100644 --- a/.agents/skills/fix-security-issue/SKILL.md +++ b/.agents/skills/fix-security-issue/SKILL.md @@ -1,6 +1,8 @@ --- name: fix-security-issue description: Implement a fix for a reviewed security issue. Takes a directly requested issue number or scans for issues labeled `topic:security` and `agent:implementation-requested`. Reads the security review from the issue comments and implements the remediation plan. Trigger keywords - fix security issue, remediate security, implement security fix, patch vulnerability. +metadata: + internal: true --- # Fix Security Issue diff --git a/.agents/skills/helm-dev-environment/SKILL.md b/.agents/skills/helm-dev-environment/SKILL.md index c577cd577..780cf8b9c 100644 --- a/.agents/skills/helm-dev-environment/SKILL.md +++ b/.agents/skills/helm-dev-environment/SKILL.md @@ -1,6 +1,8 @@ --- name: helm-dev-environment description: Start up, tear down, and configure the local Kubernetes development environment for OpenShell. Uses k3d (Docker-backed k3s) + Skaffold + Helm. Covers cluster lifecycle, optional add-ons (Keycloak OIDC, Envoy Gateway), HA testing, and port mappings. Trigger keywords - local k8s, local cluster, k3d, skaffold, helm dev, start cluster, stop cluster, tear down cluster, delete cluster, create cluster, helm:k3s, helm:skaffold, local dev environment, dev cluster, k8s dev, envoy gateway local, keycloak local, high availability, HA. +metadata: + internal: true --- # Helm Dev Environment diff --git a/.agents/skills/launch-openshell-gator/SKILL.md b/.agents/skills/launch-openshell-gator/SKILL.md index c8e9fc948..763dc779a 100644 --- a/.agents/skills/launch-openshell-gator/SKILL.md +++ b/.agents/skills/launch-openshell-gator/SKILL.md @@ -1,6 +1,8 @@ --- name: launch-openshell-gator description: Launch and supervise OpenShell gator agents. Use when starting gator on issues or PRs, checking gator sandboxes, building the gator sandbox image, restarting stuck gators, inspecting gator logs, or experimenting with gator harness/model overrides. Trigger keywords - launch gator, start gator, run gator, gator sandbox, supervised gator, gator logs, restart gator. +metadata: + internal: true --- # Launch OpenShell Gator @@ -24,7 +26,7 @@ For gator's PR/issue validation policy, load `gator-gate` inside the launched sa |---|---| | `scripts/agents/run.sh` | Manifest-driven OpenShell agent launcher. | | `scripts/agents/gator/agent.yaml` | Gator manifest: immutable payload version, default gateway, harness, providers, runtime, skills, and subagents. | -| `scripts/agents/gator/Dockerfile` | Gator sandbox image source. Local launches build this image through OpenShell. | +| `scripts/agents/gator/Dockerfile` | Gator sandbox image source. Local launches build it in gateway's Docker or Podman image store. | | `scripts/agents/gator/policy.yaml` | Sandbox policy for the gator agent. | | `scripts/agents/gator/bin/gh` | Gator-specific `gh` wrapper and same-SHA duplicate-post guard. | | `scripts/agents/gator/bin/review-feedback-ledger` | Builds tree-aware review scope, durable findings, convergence telemetry, and review-budget state. | @@ -158,7 +160,7 @@ sandbox_name="gator-pr-${pr_number}-supervised" "Review and monitor PR #${pr_number} through the gator-gate workflow. Scope this invocation only to PR #${pr_number}." ``` -The launcher builds the gator sandbox image when needed, stages the immutable payload, imports provider profiles, configures provider credentials and refresh, creates and uploads the sandbox payload, then starts the agent supervisor with `sandbox exec`. It writes a background log under `scripts/agents/gator/logs/`. +The launcher queries gateway's selected compute driver, builds gator image in matching Docker or Podman image store, stages immutable payload, imports provider profiles, configures provider credentials and refresh, creates and uploads sandbox payload, then starts agent supervisor with `sandbox exec`. It writes a background log under `scripts/agents/gator/logs/`. `CONTAINER_ENGINE`, when set, must match gateway driver. ### Launch An Issue Or Issue/PR Pair diff --git a/.agents/skills/openshell-cli/cli-reference.md b/.agents/skills/openshell-cli/cli-reference.md deleted file mode 100644 index 4d39143d8..000000000 --- a/.agents/skills/openshell-cli/cli-reference.md +++ /dev/null @@ -1,623 +0,0 @@ -# OpenShell CLI Reference - -Quick-reference for the `openshell` command-line interface. For workflow guidance, see [SKILL.md](SKILL.md). - -> **Self-teaching**: If a command or flag is not listed here, use `openshell --help` to discover it. The CLI has comprehensive built-in help at every level. - -## Global Options - -| Flag | Description | -|------|-------------| -| `-v`, `--verbose` | Increase verbosity (`-v` = info, `-vv` = debug, `-vvv` = trace) | -| `-g`, `--gateway ` | Gateway to operate on. Also settable via `OPENSHELL_GATEWAY` env var. Falls back to active gateway in `~/.config/openshell/active_gateway`. | -| `--gateway-endpoint ` | Connect directly to a gateway endpoint without looking up stored metadata. Also settable via `OPENSHELL_GATEWAY_ENDPOINT`. | -| `--gateway-insecure` | Skip TLS certificate verification. Also settable via `OPENSHELL_GATEWAY_INSECURE`; use only for trusted development endpoints. | - -## Environment Variables - -| Variable | Description | -|----------|-------------| -| `OPENSHELL_GATEWAY` | Override active gateway name (same as `--gateway`) | -| `OPENSHELL_GATEWAY_ENDPOINT` | Connect directly to a gateway endpoint (same as `--gateway-endpoint`) | -| `OPENSHELL_GATEWAY_INSECURE` | Skip TLS verification when set (same as `--gateway-insecure`) | -| `OPENSHELL_SANDBOX_POLICY` | Path to default sandbox policy YAML (fallback when `--policy` is not provided) | -| `OPENSHELL_COMMUNITY_REGISTRY` | Override the community sandbox image registry prefix used by `sandbox create --from ` | -| `OPENSHELL_THEME` | TUI theme: `auto`, `dark`, or `light` | - ---- - -## Complete Command Tree - -``` -openshell -├── gateway -│ ├── add [opts] -│ ├── login [name] -│ ├── logout [name] -│ ├── remove [name] -│ ├── info [--name] -│ ├── list -│ └── select [name] -├── status -├── whoami [--output ] -├── inference -│ ├── set --provider --model -│ ├── update [--provider] [--model] -│ └── get -├── sandbox -│ ├── create [opts] [-- CMD...] -│ ├── get [name] -│ ├── list [opts] -│ ├── stop [name] -│ ├── start [name] -│ ├── delete [name]... [--all] -│ ├── exec [--name ] [opts] -- CMD... -│ ├── connect [name] [--editor ] -│ ├── upload [dest] -│ ├── download [dest] -│ ├── ssh-config [name] -│ └── provider -│ ├── list [name] -│ ├── attach -│ └── detach -├── forward -│ ├── start [name] [-d] -│ ├── stop [name] -│ ├── list -│ └── service [name] --target-port [opts] -├── service -│ ├── expose [service] -│ ├── list [sandbox] -│ ├── get [service] -│ └── delete [service] -├── logs [name] [opts] -├── policy -│ ├── set [name] --policy [--global] [--wait] -│ ├── update [name] [opts] -│ ├── get [name] [--full|--base] [--global] -│ ├── list [name] [--global] -│ ├── delete --global -│ └── prove --policy --credentials [opts] -├── settings -│ ├── get [name] [--global] -│ ├── set [name] --key --value [--global] -│ └── delete [name] --key [--global] -├── rule (advanced; hidden from top-level help) -│ ├── get [name] [--status ] -│ ├── approve [name] --chunk-id -│ ├── reject [name] --chunk-id [--reason ] -│ ├── approve-all [name] [--include-security-flagged] -│ ├── clear [name] -│ └── history [name] -├── provider -│ ├── create --name --type [opts] -│ ├── refresh -│ │ ├── status [opts] -│ │ ├── configure [opts] -│ │ ├── rotate --credential-key -│ │ └── delete --credential-key -│ ├── get -│ ├── list [opts] -│ ├── list-profiles [opts] -│ ├── profile -│ │ ├── export [opts] -│ │ ├── import (--file |--from ) -│ │ ├── update --file -│ │ ├── lint (--file |--from ) -│ │ └── delete -│ ├── update [opts] -│ └── delete ... -├── doctor -│ └── check -├── term -├── completions -└── ssh-proxy [opts] -``` - ---- - -## Gateway Commands - -### `openshell gateway add ` - -Register an existing gateway endpoint. - -| Flag | Description | -|------|-------------| -| `--name ` | Gateway name | -| `--local` | Register a local mTLS gateway; with HTTP, store a local plaintext registration | -| `--remote ` | Register a remote mTLS gateway over SSH; with HTTP, store a remote plaintext registration | -| `--oidc-issuer ` | Register an OIDC-authenticated gateway | -| `--oidc-client-id ` | OIDC client ID (default: `openshell-cli`; requires `--oidc-issuer`) | -| `--oidc-audience ` | OIDC API audience (requires `--oidc-issuer`) | -| `--oidc-scopes ` | Space-separated OAuth2 scopes (requires `--oidc-issuer`) | - -Examples: - -- `openshell gateway add http://127.0.0.1:8080 --local --name local` -- `openshell gateway add https://gateway.example.com --name production` -- `openshell gateway add ssh://user@gateway.example.com:8080 --name remote` - -An `http://` endpoint is direct plaintext. A plain `https://` endpoint uses edge authentication. `--local` and `--remote` select mTLS registration modes when used with HTTPS; required certificates must already exist. An `ssh://` endpoint is shorthand for a remote gateway. - -### `openshell gateway remove [name]` - -Remove a local gateway registration. This removes CLI metadata and stored auth tokens only; package managers, systemd, Helm, Docker, and other platform tools still own the gateway process. - -### `openshell gateway login [name]` - -Refresh browser-based authentication for an edge-authenticated or OIDC gateway. - -### `openshell gateway logout [name]` - -Clear locally stored OIDC or edge credentials for a gateway. - -### `openshell gateway info` - -Show gateway details: endpoint, auth mode, and remote host metadata when present. - -| Flag | Description | -|------|-------------| -| `--name ` | Gateway name (defaults to active) | - -### `openshell gateway select [name]` - -Set the active gateway. Writes to `~/.config/openshell/active_gateway`. Without a name, opens an interactive chooser on a TTY or lists gateways in non-interactive mode. - -### `openshell gateway list` - -List registered gateways and mark the active one. `--output table|yaml|json` selects the format. - ---- - -## Doctor Commands - -### `openshell doctor check` - -Validate local Docker prerequisites for standalone gateway development. For -package-managed or Helm gateways, use `systemctl`, `journalctl`, `kubectl`, and -`helm` directly. - ---- - -## Status Command - -### `openshell status` - -Show server connectivity, authentication status, and version for the active -gateway. Connectivity uses the public health RPC; authentication is checked -with the protected gateway-info capability query and can fail while the gateway -remains connected. - -### `openshell whoami` - -Show the authenticated user identity: subject, display name, roles, scopes, and -identity provider. Requires an authenticated gateway connection. - -| Flag | Description | -|------|-------------| -| `--output ` | Output format: `table` (default), `json`, or `yaml` | - ---- - -## Sandbox Commands - -### `openshell sandbox create [OPTIONS] [-- COMMAND...]` - -Create a sandbox through the selected gateway and launch its canonical main -process. By default, the CLI attaches to that retained process after the -sandbox becomes ready. A trailing command defines the canonical main process; -without one, the default is `/bin/bash -l` with a PTY. Explicit commands remain -foreground in non-interactive automation: stdout and stderr stream to the -caller and the CLI returns the command's exact status. Exit 0 leaves -`Completed`; nonzero leaves `Error/MainProcessFailed`. -Starting either retained terminal result invalidates SSH sessions from the -previous runtime generation. - -| Flag | Description | -|------|-------------| -| `--name ` | Sandbox name (auto-generated if omitted) | -| `--from ` | Community name, Dockerfile path, directory, or image reference (BYOC) | -| `--no-keep` | Delete the sandbox after main output and the result drain | -| `--detach` | Start the canonical main process without attaching | -| `--editor vscode|cursor` | Launch a remote editor and keep the sandbox alive | -| `--gpu [COUNT]` | Request the driver's default GPU selection or a specific count | -| `--cpu ` | CPU limit (for example: `500m`, `1`, `2.5`) | -| `--memory ` | Memory limit (for example: `512Mi`, `4Gi`, `8G`) | - -`--detach` adds no attachment grace period: the sandbox reports the canonical -process result immediately when it exits. Foreground creation declares one -expected main-process SSH attachment; cleanup finalizes after that connection -drains and closes naturally. -| `--driver-config-json ` | Experimental driver-keyed configuration object | -| `--provider ` | Provider to attach (repeatable) | -| `--policy ` | Custom policy YAML; overrides the built-in default and `OPENSHELL_SANDBOX_POLICY` | -| `--forward <[BIND:]PORT>` | Start a local port forward and keep the sandbox alive | -| `--tty`, `--no-tty` | Force or disable pseudo-terminal allocation | -| `--auto-providers` | Auto-create missing providers from local credentials | -| `--no-auto-providers` | Never auto-create providers; error if a required provider is missing | -| `--label ` | Attach a label (repeatable) | -| `--env ` | Inject an environment variable (repeatable) | -| `--approval-mode manual|auto` | Handle agent-authored policy proposals; default: `manual` | -| `--upload [:]` | Upload local files to the working directory or an explicit destination (repeatable) | -| `--no-git-ignore` | Disable `.gitignore` filtering for `--upload` | -| `[-- COMMAND...]` | Canonical main command (defaults to `/bin/bash -l`) | - -`--upload` cannot be combined with a trailing main command because uploads -currently complete after the canonical process starts. Create the default -scratch sandbox, upload files, then use `sandbox exec`, or build the files into -the image. - -### `openshell sandbox get [name]` - -Show sandbox details and the active policy. Metadata identifies sandbox or global policy source and the corresponding revision. The name defaults to the last-used sandbox. - -| Flag | Description | -|------|-------------| -| `--policy-only` | Print only the active policy YAML to stdout | - -### `openshell sandbox list` - -| Flag | Default | Description | -|------|---------|-------------| -| `--limit ` | 100 | Maximum sandboxes | -| `--offset ` | 0 | Pagination offset | -| `--ids` | false | Print only sandbox IDs | -| `--names` | false | Print only sandbox names | -| `--selector ` | none | Filter by `key1=value1,key2=value2` | -| `--output table|yaml|json` | `table` | Output format | - -### `openshell sandbox delete [NAME]...` - -Delete one or more named sandboxes, or use `--all`. Deletion stops background port forwards. - -### `openshell sandbox stop [name]` - -Stop sandbox compute while retaining the sandbox and persistent workspace. The -name defaults to the last-used sandbox. The command stops background forwards -and waits for the `Stopped` phase. - -### `openshell sandbox start [name]` - -Start a stopped, failed, or completed sandbox and wait for `Ready`. This -launches a fresh canonical-main instance. The name defaults to the last-used -sandbox. - -### `openshell sandbox exec [OPTIONS] -- COMMAND...` - -Execute a command through the gRPC exec endpoint, stream its output, and exit with the remote command's exit code. - -| Flag | Default | Description | -|------|---------|-------------| -| `-n`, `--name ` | last-used | Sandbox name | -| `--workdir ` | none | Working directory in the sandbox | -| `--timeout ` | 0 | Command timeout; `0` disables it | -| `--tty`, `--no-tty` | auto | Force or disable a pseudo-terminal | -| `--env ` | none | Command environment variable (repeatable) | - -### `openshell sandbox connect [name]` - -Attach to the sandbox's retained canonical main process. Disconnecting leaves -the process running. Reconnecting targets the same process instance and -replays recent output. Use `sandbox exec --tty -- /bin/bash -l` when you need a -new shell. The name defaults to the last-used sandbox. - -`--editor vscode|cursor` launches a supported remote editor instead of -attaching to the canonical main process. - -### `openshell sandbox upload [dest]` - -Upload files using tar-over-SSH. The CLI discovers the canonical remote working directory when the destination is omitted. A named directory merges into an existing directory of the same name, overwriting matching entries without deleting unrelated entries. `.gitignore` filtering is enabled unless `--no-git-ignore` is passed. - -### `openshell sandbox download [dest]` - -Download files using tar-over-SSH. The sandbox source may be relative to the canonical remote working directory or an absolute path within it. The local destination defaults to `.`. - -### `openshell sandbox ssh-config [name]` - -Print an SSH config `Host` block. The name defaults to the last-used sandbox. - -### `openshell sandbox provider` - -Manage providers on an existing sandbox: - -- `openshell sandbox provider list [name]` -- `openshell sandbox provider attach ` -- `openshell sandbox provider detach ` - ---- - -## Port Forwarding Commands - -### `openshell forward start [name]` - -Start forwarding a local port to a sandbox. - -| Flag | Description | -|------|-------------| -| `` | `[bind_address:]port`; the port is used locally and remotely | -| `[name]` | Sandbox name (defaults to last-used) | -| `-d`, `--background` | Run in background | - -### `openshell forward stop [name]` - -Stop a background port forward. When the sandbox name is omitted, it is inferred from active forwards. - -### `openshell forward list` - -List all active port forwards (sandbox, port, PID, status). - -### `openshell forward service [name] --target-port ` - -Forward a local TCP port to a loopback service inside a sandbox over the gRPC relay. - -| Flag | Default | Description | -|------|---------|-------------| -| `--target-port ` | required | Service port inside the sandbox | -| `--target-host ` | `127.0.0.1` | Loopback service host | -| `--local <[BIND:]PORT>` | target port | Local bind; port `0` requests dynamic assignment | - ---- - -## Service Commands - -Gateway-managed HTTP service endpoints: - -- `openshell service expose [service]` -- `openshell service list [sandbox] [--limit N] [--offset N]` -- `openshell service get [service]` -- `openshell service delete [service]` - ---- - -## Logs Command - -### `openshell logs [name]` - -View sandbox logs. Supports one-shot and streaming. - -| Flag | Default | Description | -|------|---------|-------------| -| `-n ` | 200 | Number of log lines | -| `--tail` | false | Stream live logs | -| `--since ` | none | Only show logs from this duration ago (e.g., `5m`, `1h`) | -| `--source ` | `all` | Filter: `gateway`, `sandbox`, or `all` (repeatable) | -| `--level ` | none | Minimum level: `error`, `warn`, `info`, `debug`, `trace` | - -The sandbox name defaults to the last-used sandbox. - ---- - -## Policy Commands - -### `openshell policy update [name]` - -Incrementally merge live network policy changes into the current sandbox policy when the selected compute driver supports live updates. Multiple flags in one invocation are applied as one atomic batch and create at most one new revision. MXC rejects live policy merges; delete and recreate an MXC sandbox instead. - -| Flag | Default | Description | -|------|---------|-------------| -| `--add-endpoint ` | repeatable | `host:port[:access[:protocol[:enforcement[:options]]]]`. Adds or merges an endpoint. | -| `--remove-endpoint ` | repeatable | `host:port`. Removes the endpoint or just the requested port from a multi-port endpoint. | -| `--add-allow ` | repeatable | `host:port:METHOD:path_glob`. Adds REST or WebSocket allow rules. | -| `--add-deny ` | repeatable | `host:port:METHOD:path_glob`. Adds REST or WebSocket deny rules. | -| `--remove-rule ` | repeatable | Deletes a named network rule. | -| `--binary ` | repeatable | Adds binaries to each `--add-endpoint` rule. Valid only with `--add-endpoint`. | -| `--rule-name ` | none | Overrides the generated rule name. Valid only when exactly one `--add-endpoint` is provided. | -| `--dry-run` | false | Preview the merged policy locally without sending an update to the gateway. | -| `--wait` | false | Wait for the sandbox to confirm the new policy revision is loaded. | -| `--timeout ` | 60 | Timeout for `--wait`. | - -Notes: - -- The sandbox name defaults to the last-used sandbox. -- `--add-endpoint` options are comma-separated: `allowed-ip=`, `websocket-credential-rewrite`, `request-body-credential-rewrite`, and `allow-uninspected-credentials`. The last option is a security-sensitive exception for provider-credentialed L4-only, `tls: skip`, or otherwise uninspectable traffic. -- `protocol` accepts `tcp` for explicit L4-only host/port policy. It has the - same payload-handling behavior as omitting the protocol, but it requires a - valid DNS hostname and rejects hostless `allowed_ips` or literal-IP - selectors. It cannot be combined with `access`, `rules`, or L7 enforcement - options. -- `--add-allow` and `--add-deny` operate on REST and WebSocket endpoints. Use full YAML for JSON-RPC, MCP, SQL, or other policy structure. -- `--wait` cannot be combined with `--dry-run`. -- Use `policy set` when replacing the full policy or changing static sections. - -### `openshell policy set [name] --policy ` - -Replace the full policy on a live sandbox when the selected compute driver supports live updates. Only the dynamic `network_policies` field can be changed at runtime. MXC rejects live policy replacement; delete and recreate an MXC sandbox instead. - -| Flag | Default | Description | -|------|---------|-------------| -| `--policy ` | -- | Path to policy YAML (required) | -| `--global` | false | Apply as the gateway-global policy | -| `--yes` | false | Skip confirmation for a global update | -| `--wait` | false | Wait for sandbox to confirm policy is loaded | -| `--timeout ` | 60 | Timeout for `--wait` | - -Exit codes with `--wait`: 0 = loaded, 1 = failed, 124 = timeout. - -### `openshell policy get [name]` - -Show the current effective sandbox policy or stored global policy. - -| Flag | Default | Description | -|------|---------|-------------| -| `--rev ` | 0 | Show a stored revision; `0` shows the current effective policy | -| `--full` | false | Include the effective policy payload and provider-composed entries | -| `--base` | false | Include the base policy payload without provider-composed entries | -| `--output table|json` | `table` | Output format | -| `--global` | false | Show the global policy revision | - -### `openshell policy list [name]` - -List policy revision history (version, hash, status, created, error). - -| Flag | Default | Description | -|------|---------|-------------| -| `--limit ` | 20 | Max revisions to return | -| `--global` | false | List global policy revisions | - -### `openshell policy delete --global` - -Delete the global policy lock and restore sandbox-level policy control. `--yes` skips confirmation. - -### `openshell policy prove` - -Prove policy properties or find counterexamples. - -| Flag | Description | -|------|-------------| -| `--policy ` | Policy YAML (required) | -| `--credentials ` | Credential descriptor YAML (required) | -| `--registry ` | Capability registry directory (defaults to bundled) | -| `--accepted-risks ` | Accepted-risks YAML | -| `--compact` | One-line-per-finding output | - -### `openshell rule` (advanced) - -Review agent-authored network rule proposals. This command group is intentionally hidden from top-level help but is part of the policy-advisor workflow. - -- `openshell rule get [name] [--status pending|approved|rejected]` -- `openshell rule approve [name] --chunk-id ` -- `openshell rule reject [name] --chunk-id [--reason ]` -- `openshell rule approve-all [name] [--include-security-flagged]` -- `openshell rule clear [name]` -- `openshell rule history [name]` - -Sandbox names default to the last-used sandbox. The CLI fetches and submits each proposal's current review token; a changed live candidate remains pending until it is reviewed again. Bulk approval of security-flagged proposals requires explicit `--include-security-flagged`. - ---- - -## Settings Commands - -Settings support sandbox and gateway-global scopes: - -- `openshell settings get [name] [--global] [--json]` -- `openshell settings set [name] --key --value [--global] [--yes]` -- `openshell settings delete [name] --key [--global] [--yes]` - -Sandbox names default to the last-used sandbox. Global mutations prompt unless `--yes` is passed. - ---- - -## Provider Commands - -Provider types are defined by built-in and custom provider profiles. Use `openshell provider list-profiles` to discover the selected gateway's current inventory. - -### `openshell provider create --name --type ` - -Create a provider configuration. - -| Flag | Description | -|------|-------------| -| `--name ` | Provider name (required) | -| `--type ` | Provider type (required) | -| `--from-existing` | Load credentials and config from local state | -| `--credential KEY[=VALUE]` | Credential pair. Bare `KEY` reads from env var. Repeatable. | -| `--from-gcloud-adc` | Load a compatible credential from gcloud Application Default Credentials | -| `--runtime-credentials` | Resolve required credentials at runtime in the gateway or sandbox | -| `--config KEY=VALUE` | Config key/value pair. Repeatable. | - -Exactly one credential source is required. Credential-source flags conflict with one another. - -### `openshell provider get ` - -Show provider details (id, name, type, credential keys, config keys). - -### `openshell provider list` - -List providers in a table. - -| Flag | Default | Description | -|------|---------|-------------| -| `--limit ` | 100 | Max providers | -| `--offset ` | 0 | Pagination offset | -| `--names` | false | Print only names | -| `--output table|yaml|json` | `table` | Output format | - -### `openshell provider update ` - -Update an existing provider without changing its type. - -| Flag | Description | -|------|-------------| -| `--from-existing` | Rediscover local credentials and config | -| `--credential KEY[=VALUE]` | Update a credential (repeatable) | -| `--config KEY=VALUE` | Update config (repeatable) | -| `--credential-expires-at KEY=TIMESTAMP` | Set or clear credential expiry; accepts epoch milliseconds or RFC3339, and `0` clears | - -### `openshell provider delete ...` - -Delete one or more providers by name. - -### Provider profiles - -- `openshell provider list-profiles [--output table|yaml|json]` -- `openshell provider profile export [--output table|yaml|json]` -- `openshell provider profile import (--file |--from )` -- `openshell provider profile update --file ` -- `openshell provider profile lint (--file |--from )` -- `openshell provider profile delete ` - -### Provider credential refresh - -- `openshell provider refresh status [--credential-key ]` -- `openshell provider refresh rotate --credential-key ` -- `openshell provider refresh delete --credential-key ` - -`provider refresh configure ` accepts: - -| Flag | Description | -|------|-------------| -| `--credential-key ` | Injectable credential key (required) | -| `--strategy ` | `oauth2-refresh-token`, `oauth2-client-credentials`, or `google-service-account-jwt` | -| `--material KEY=VALUE` | Non-secret refresh material (repeatable) | -| `--secret-material-env KEY[=ENVVAR]` | Secret refresh material read from the CLI environment (repeatable) | -| `--secret-material-key KEY` | Mark a supplied material key secret (repeatable) | -| `--credential-expires-at TIMESTAMP` | Current credential expiry in epoch milliseconds or RFC3339 | - ---- - -## Inference Commands - -### `openshell inference set` - -Configure the gateway's user-facing `inference.local` route or the platform-only system route. Provider and model are required. - -| Flag | Default | Description | -|------|---------|-------------| -| `--provider ` | -- | Provider record name (required) | -| `--model ` | -- | Model identifier to use for generation requests (required) | -| `--system` | false | Configure the system inference route | -| `--no-verify` | false | Skip endpoint verification before saving | -| `--timeout ` | 0 | Request timeout; `0` uses the 60-second default | - -### `openshell inference update` - -Partially update the selected inference route. - -| Flag | Default | Description | -|------|---------|-------------| -| `--provider ` | unchanged | Provider record name | -| `--model ` | unchanged | Model identifier | -| `--system` | false | Target the system inference route | -| `--no-verify` | false | Skip endpoint verification before saving | -| `--timeout ` | unchanged | Request timeout; `0` uses the 60-second default | - -### `openshell inference get` - -Show both inference routes. `--system` shows only the system route. - ---- - -## Other Commands - -### `openshell term` - -Launch the OpenShell interactive TUI. `--theme auto|dark|light` overrides `OPENSHELL_THEME`. - -### `openshell completions ` - -Generate shell completion scripts. Supported shells: `bash`, `fish`, `zsh`, `powershell`. - -### `openshell ssh-proxy` - -SSH proxy used as a `ProxyCommand`. Not typically invoked directly. diff --git a/.agents/skills/review-github-pr/SKILL.md b/.agents/skills/review-github-pr/SKILL.md index 21cd13188..de56cb477 100644 --- a/.agents/skills/review-github-pr/SKILL.md +++ b/.agents/skills/review-github-pr/SKILL.md @@ -1,6 +1,8 @@ --- name: review-github-pr description: Review a GitHub pull request by summarizing its diff and key design decisions. Use when the user wants to review a PR, understand changes in a branch, or get a code review summary. Trigger keywords - review PR, review pull request, summarize PR, summarize diff, code review, review branch, PR summary, diff summary. +metadata: + internal: true --- # Review GitHub Pull Request diff --git a/.agents/skills/review-security-issue/SKILL.md b/.agents/skills/review-security-issue/SKILL.md index 03dc2974b..efb054df8 100644 --- a/.agents/skills/review-security-issue/SKILL.md +++ b/.agents/skills/review-security-issue/SKILL.md @@ -1,6 +1,8 @@ --- name: review-security-issue description: Given a GitHub issue, review the issue for security implications. You'll make a determination if the claim in the issue is legitimate and should be addressed or will be a "won't fix." Trigger keywords - security issue, review security ticket, review security issue. +metadata: + internal: true --- # Review Security Issue diff --git a/.agents/skills/sbom/SKILL.md b/.agents/skills/sbom/SKILL.md index 6996ffcf6..e5c3e48c2 100644 --- a/.agents/skills/sbom/SKILL.md +++ b/.agents/skills/sbom/SKILL.md @@ -1,6 +1,8 @@ --- name: sbom description: Generate and manage Software Bill of Materials (SBOMs) for the OpenShell project. Covers SBOM generation with Syft, license resolution via public registries, and CSV export for compliance review. Trigger keywords - SBOM, sbom, bill of materials, license audit, license resolution, generate sbom, sbom csv, dependency license, supply chain, license scan. +metadata: + internal: true --- # SBOM Generation and License Resolution diff --git a/.agents/skills/sync-agent-infra/SKILL.md b/.agents/skills/sync-agent-infra/SKILL.md index 5dfbef12c..d3c0190e8 100644 --- a/.agents/skills/sync-agent-infra/SKILL.md +++ b/.agents/skills/sync-agent-infra/SKILL.md @@ -1,6 +1,8 @@ --- name: sync-agent-infra description: Detect and fix drift across agent-first infrastructure files. Ensures skill inventories, workflow chains, architecture tables, issue/PR templates, and cross-references stay consistent when skills, crates, or workflows change. Run after adding, removing, or renaming skills or components. Trigger keywords - sync agent infra, sync skills, update agent docs, check agent consistency, agent infra drift, sync contributing, sync agents. +metadata: + internal: true --- # Sync Agent Infrastructure @@ -12,13 +14,13 @@ Detect and fix drift across the agent-first infrastructure files. These files re | `AGENTS.md` | Project identity, workflow chains, architecture overview, issue/PR conventions, skill maintenance pointer | | `CONTRIBUTING.md` | Skills table, workflow chains, "When to Open an Issue" guidance, skill references | | `CONTRIBUTING.md` issue lifecycle section | Human-facing issue states, roadmap decisions, acceptance signals, and direct-versus-queued agent ownership | -| `README.md` | "Built With Agents" section, "Explore with your agent" skill references | +| `README.md` | "Use OpenShell with Your Agent" and "Built With Agents" sections | | `.github/ISSUE_TEMPLATE/bug_report.yml` | Skill name references in diagnostic guidance | | `.github/ISSUE_TEMPLATE/feature_request.yml` | Skill name references in investigation guidance | | `.github/ISSUE_TEMPLATE/config.yml` | Contact link text referencing skills | | `.github/workflows/issue-triage.yml` | Comment text referencing skills | | `.agents/skills/triage-issue/SKILL.md` | Skill name references in gate check and diagnosis steps | -| `.agents/skills/openshell-cli/SKILL.md` | Companion skills table | +| `skills/*/SKILL.md` | Standalone user instructions and links to documentation, included files, and related skills | | `.agents/skills/create-github-pr/SKILL.md` | Pre-PR agent infrastructure check | | `.agents/skills/review-github-pr/SKILL.md` | Review-time agent infrastructure check | | `.agents/skills/build-from-issue/SKILL.md` | Label awareness and pre-commit agent infrastructure check | @@ -26,7 +28,7 @@ Detect and fix drift across the agent-first infrastructure files. These files re ## When to Run -- After adding, removing, or renaming a skill in `.agents/skills/` +- After adding, removing, renaming, or moving a skill in `skills/` or `.agents/skills/` - After adding, removing, or renaming a crate in `crates/` - After changing workflow chain relationships between skills - After changing which product or development areas a skill covers @@ -35,7 +37,7 @@ Detect and fix drift across the agent-first infrastructure files. These files re ## Skill Maintenance Map -Use this map when product behavior, commands, or development workflows change. It is a routing aid, not an exhaustive dependency list. Search `.agents/skills/` for the changed command, field, component, or workflow before concluding that no other skill needs an update. +Use this map when product behavior, commands, or development workflows change. It is a routing aid, not an exhaustive dependency list. Search both `skills/` and `.agents/skills/` for the changed command, field, component, or workflow before concluding that no other skill needs an update. | Change area | Skills to review | |---|---| @@ -43,7 +45,7 @@ Use this map when product behavior, commands, or development workflows change. I | Sandbox policy schema, presets, or enforcement behavior | `generate-sandbox-policy`, `openshell-cli` | | Supervisor middleware policy, registrations, runtime, or failure behavior | `generate-sandbox-policy`, `openshell-cli`, `debug-openshell-cluster` | | Gateway deployment, Helm, runtime drivers, or health checks | `debug-openshell-cluster`, `helm-dev-environment` | -| Inference routing, providers, or `inference.local` behavior | `debug-inference`, `openshell-cli` | +| Inference providers, native model endpoints, or migration from `inference.local` | `debug-inference`, `openshell-cli`, `generate-sandbox-policy` | | TUI architecture, navigation, data fetching, or UX | `tui-development` | | Release artifacts or post-publish smoke coverage | `test-release-canary` | | GitHub Actions workflows, required checks, or CI diagnostics | `watch-github-actions`; also `test-release-canary` for release smoke coverage | @@ -53,7 +55,7 @@ Use this map when product behavior, commands, or development workflows change. I | PR template, review conventions, or vouch behavior | `create-github-pr`, `review-github-pr`, `build-from-issue` | | Security review or remediation workflow | `review-security-issue`, `fix-security-issue` | | RFC template, numbering, or lifecycle | `create-rfc` | -| Documentation structure, navigation, or doc-update workflow | `update-docs` | +| Documentation structure, navigation, or doc-update workflow | `update-docs-from-commits` | | Skills, crates, workflow chains, issue/PR templates, or agent cross-references | `sync-agent-infra` | ## Prerequisites @@ -66,13 +68,14 @@ Gather the source of truth for each category. ### Skills -List all skill directories: +List public and contributor skill directories separately: ```bash +ls -1 skills/ ls -1 .agents/skills/ ``` -This is the canonical skill list. Every other file must agree with it. +The directories are canonical by audience: `skills/` contains public, installable user/operator skills and `.agents/skills/` contains internal contributor workflows. Every other file must agree with both inventories. ### Crates @@ -96,16 +99,19 @@ For each file in the table above, check for the following inconsistencies: ### `CONTRIBUTING.md` -1. **Skills table** — Every skill in `.agents/skills/` must appear in the "Agent Skills for Contributors" table. No skill in the table should reference a directory that doesn't exist. -2. **Workflow chains** — Must match `AGENTS.md` workflow chains exactly. -3. **Skill references in prose** — Any skill mentioned by name in "Before You Open an Issue", "When to Open an Issue", or "When NOT to Open an Issue" must exist in `.agents/skills/`. +1. **Public skills table** — Every skill in `skills/` must appear in "Skills for Using OpenShell" and no contributor skill may appear there. +2. **Contributor skills table** — Every skill in `.agents/skills/` must appear in "Agent Skills for Contributors" and no public skill may appear there. +3. **Inventory paths** — No skill in either table should reference a directory that does not exist. +4. **Workflow chains** — Must match `AGENTS.md` workflow chains exactly. +5. **Skill references in prose** — Any named skill must exist in exactly one canonical skill directory. ### `AGENTS.md` 1. **Architecture overview** — Every crate in `crates/` must appear in the architecture table. The `python/`, `proto/`, `deploy/`, `.agents/` rows must also be present. -2. **Workflow chains** — Verify each skill named in a chain exists in `.agents/skills/`. -3. **Issue/PR conventions** — Verify referenced skills (`create-github-issue`, `create-github-pr`, `build-from-issue`) exist. -4. **Skill maintenance pointer** — Verify it still points to `sync-agent-infra` and does not duplicate the maintenance map from this skill. +2. **Skill layout** — The architecture table must contain separate `skills/` and `.agents/skills/` rows with accurate audience descriptions. +3. **Workflow chains** — Verify each skill named in a chain exists in exactly one of the two skill directories. +4. **Issue/PR conventions** — Verify referenced skills (`create-github-issue`, `create-github-pr`, `build-from-issue`) exist. +5. **Skill maintenance pointer** — Verify it still points to `sync-agent-infra` and does not duplicate the maintenance map from this skill. ### Issue Lifecycle Documentation @@ -115,8 +121,8 @@ For each file in the table above, check for the following inconsistencies: ### `README.md` -1. **"Explore with your agent"** — Skill names referenced must exist in `.agents/skills/`. -2. **"Built With Agents"** — Skill names referenced must exist. Workflow descriptions should be consistent with `AGENTS.md` chains. +1. **Public installation guidance** — The README must distinguish `skills/` from `.agents/skills/`, include `npx skills add NVIDIA/OpenShell`, and list only canonical public skills as installable. +2. **"Built With Agents"** — Contributor skill names must exist under `.agents/skills/`. Workflow descriptions should be consistent with `AGENTS.md` chains. ### Issue Templates @@ -131,12 +137,23 @@ For each file in the table above, check for the following inconsistencies: ### Skill Cross-References 1. **`triage-issue`** — Skills referenced in gate check and diagnosis steps must exist. -2. **`openshell-cli`** — Companion skills table entries must exist. +2. **`openshell-cli`** — Companion skills table entries must exist in one canonical location. 3. **`build-from-issue`** — Label names must match the project's label taxonomy. Lifecycle and request labels must gate unattended queue pickup, while direct requests warn on workflow discrepancies and continue. 4. **`create-spike`** — Reference to `build-from-issue` as next step must be accurate. 5. **`review-security-issue`** / **`fix-security-issue`** — Cross-references between the two must be accurate. 6. **PR creation and review checks** — The `create-github-pr`, `review-github-pr`, `build-from-issue`, and `principal-engineer-reviewer` references to `sync-agent-infra` must exist and use trigger conditions aligned with this skill. +### Skill Layout, Metadata, and Portability + +1. **Placement** — The four public skills (`openshell-cli`, `generate-sandbox-policy`, `debug-inference`, and `debug-openshell-cluster`) must live only in `skills/`. Every other repository skill must live only in `.agents/skills/`. +2. **Internal metadata** — Every `.agents/skills/*/SKILL.md` must set `metadata.internal: true`. Public skills must not set internal metadata. Treat this as a discovery filter, not an access-control boundary. +3. **Unique names** — Parse the `name` field from every `SKILL.md` under both roots. Every name must be globally unique and match the documented inventory. +4. **Local references** — Every relative Markdown link and referenced file in a skill must resolve within that installed skill directory unless the reference is an explicit published URL. +5. **Canonical paths** — Contributor skills that name the source location of a public skill must use `skills//...`, never `.agents/skills//...`. +6. **Public portability** — Public skills must not require repository-relative files under `docs/`, `architecture/`, `crates/`, `deploy/`, or `.agents/`; source builds; `mise`; or repository E2E workflows. Use installed `openshell --help` for command syntax and Markdown endpoints under `https://docs.nvidia.com/openshell/latest/` (URLs ending in `.md`) for product documentation. +7. **No canonical documentation copies** — Review public reference files and large command/schema blocks. Remove material that merely copies CLI help, policy schemas, architecture docs, or published operational documentation; retain only skill-specific reasoning and worked interactions. +8. **Discovery** — Run `npx -y skills add . --list` from a clean checkout or disposable copy. It must list exactly the four public skills. Remove any generated lock file or installed directory after the check. + ## Step 3: Report Drift If any inconsistencies are found, report them in a structured format: @@ -145,9 +162,12 @@ If any inconsistencies are found, report them in a structured format: ## Agent Infrastructure Drift Report ### Skills Inventory -- ADDED (exists in .agents/skills/ but missing from CONTRIBUTING.md): -- REMOVED (in CONTRIBUTING.md but missing from .agents/skills/): -- OK: skills consistent +- PUBLIC ADDED (exists in skills/ but missing from CONTRIBUTING.md): +- PUBLIC REMOVED (documented as public but missing from skills/): +- CONTRIBUTOR ADDED (exists in .agents/skills/ but missing from CONTRIBUTING.md): +- CONTRIBUTOR REMOVED (documented as contributor but missing from .agents/skills/): +- METADATA/PATH/NAME ERRORS: +- OK: public and contributor skills consistent ### Architecture Table - ADDED (exists in crates/ but missing from AGENTS.md): @@ -178,6 +198,7 @@ If drift is found, fix it by updating the affected files: 5. **Removed crate** — Remove the row from the AGENTS.md architecture table. 6. **Changed workflow chain** — Update chains in both `AGENTS.md` and `CONTRIBUTING.md`. Update the "Built With Agents" section in `README.md` if the change is user-visible. 7. **Changed skill coverage** — Update the skill maintenance map in this file and any affected cross-references or companion-skill tables. +8. **Audience or portability drift** — Move the skill to its canonical root, fix internal metadata, replace stale public-skill paths, repair local links, and replace copied product documentation with CLI self-discovery or published documentation links. After fixing, re-run Step 2 to verify consistency. diff --git a/.agents/skills/test-release-canary/SKILL.md b/.agents/skills/test-release-canary/SKILL.md index 5e8bbf394..2d0fdfe36 100644 --- a/.agents/skills/test-release-canary/SKILL.md +++ b/.agents/skills/test-release-canary/SKILL.md @@ -1,6 +1,8 @@ --- name: test-release-canary description: Manually dispatch and iterate on the Release Canary workflow that smoke-tests published OpenShell artifacts (install.sh on macOS/Ubuntu/Fedora, Helm chart on kind) after each Release Dev publish. Use when changing `.github/workflows/release-canary.yml`, validating a release before tagging, debugging a canary failure, or reproducing a canary job locally. Trigger keywords - release canary, release-canary, canary failed, canary dispatch, test release canary, post-release smoke, install.sh canary, helm chart canary, kind canary, dispatch canary. +metadata: + internal: true --- # Test Release Canary @@ -14,6 +16,7 @@ The Release Canary (`.github/workflows/release-canary.yml`) smoke-tests the arti | `macos` | `macos-latest-xlarge` | `install.sh` resolves the Homebrew formula, brew installs the cask, and `openshell status` reaches the brew-services–backed local gateway with the VM driver. | | `ubuntu` | `ubuntu-latest` | `install.sh` installs the Debian package, the post-install systemd user service starts, and `openshell status` reaches the local gateway with the Docker driver. | | `fedora` | `fedora:latest` container | `install.sh` installs the RPM packages, the local gateway starts under Podman, and `openshell status` succeeds. | +| `ubuntu-snap` | `ubuntu-latest` | Downloads the Snap artifact from Release Dev, installs it with `--dangerous`, connects the required interfaces, and waits up to 30 seconds for the recovered local gateway. | | `kubernetes` | `ubuntu-latest` + kind | `helm install oci://ghcr.io/nvidia/openshell/helm-chart --version 0.0.0-dev` succeeds in a kind cluster, the gateway pod becomes Ready, port-forward exposes 8080, and the released CLI registers the in-cluster gateway and runs `openshell status` against it. | All canary jobs disable anonymous OpenShell telemetry. Host package jobs inject @@ -41,7 +44,7 @@ on: ``` - **Automatic.** Every successful `Release Dev` run (on `main` or a manual dispatch of Release Dev) fires the canary. Each job gates on `github.event.workflow_run.conclusion == 'success'` so a failed Release Dev does not run the canary. -- **Manual.** `workflow_dispatch` lets you run the canary on demand against any branch's workflow definition. +- **Manual.** `workflow_dispatch` lets you run the canary on demand against any branch's workflow definition. To include `ubuntu-snap`, supply `release-dev-run-id` for a successful Release Dev run whose Snap artifact should be tested; without it, that job is skipped because no artifact is available. When dispatched manually, `github.event.workflow_run.head_sha` is empty and the workflow falls back to `github.sha` (the branch tip) for the `install.sh` URL. @@ -53,6 +56,13 @@ Run the canary as-is on the current branch: gh workflow run release-canary.yml --ref "$(git branch --show-current)" ``` +To exercise the Ubuntu Snap job, pass the successful Release Dev run ID: + +```shell +gh workflow run release-canary.yml --ref "$(git branch --show-current)" \ + -f release-dev-run-id= +``` + Watch the run that starts: ```shell @@ -122,6 +132,7 @@ Loopback registration auto-derives the gateway name to `openshell` if `--name` i |---|---|---| | `macos`/`ubuntu`/`fedora` job fails on `install.sh` | Latest tagged release missing an asset, checksum mismatch, or `install.sh` regression on this branch. | Job log around the `curl … install.sh \| sh` step. | | `macos`/`ubuntu`/`fedora` job fails on `openshell status` | Local gateway service did not start (systemd/brew/podman). Often a driver issue. | Service logs in the job log; `OPENSHELL_DRIVERS` env in the "Ensure …" step. | +| `ubuntu-snap` fails after interface connection | The gateway did not recover after Docker became available, or did not become reachable within the 30-second bound. | Failure diagnostics dump Snap service/connection/change state, gateway and snapd journals, Snap logs, and port 17670 listeners. | | `kubernetes` job fails on `helm install --wait` | Chart did not deploy in 5 min — usually image pull failure or readiness probe failing. | "Diagnostics on failure" step dumps `helm status`, manifest, pod describe, pod logs. | | `kubernetes` job fails on `kubectl wait` | Gateway pod stuck `CrashLoopBackOff` or `ImagePullBackOff`. | Diagnostics dump; check `:dev` image existence at `ghcr.io/nvidia/openshell/gateway`. | | `kubernetes` job fails on `openshell gateway add` or `status` | Port-forward not reachable, or CLI/gateway proto mismatch. | `port-forward.log` and `openshell gateway list` in the diagnostics dump. | diff --git a/.agents/skills/triage-issue/SKILL.md b/.agents/skills/triage-issue/SKILL.md index 11594e13a..76ab01bdb 100644 --- a/.agents/skills/triage-issue/SKILL.md +++ b/.agents/skills/triage-issue/SKILL.md @@ -1,6 +1,8 @@ --- name: triage-issue description: Assess, validate, and route community-filed issues for human disposition and roadmap placement. Takes a specific issue number or processes a confirmed batch of issues labeled state:triage-needed. Investigates reported behavior, separates objective findings from product decisions, and prepares validated issues for a human yes/no decision. Trigger keywords - triage issue, triage, assess issue, review incoming issue, triage issues. +metadata: + internal: true --- # Triage Issue @@ -153,9 +155,9 @@ Based on the sub-agent's analysis, also attempt to validate the report directly: - For bug reports: check the relevant code paths, look for the described failure mode - For feature requests: assess feasibility against the existing architecture -- For gateway deployment or infrastructure issues: reference the `debug-openshell-cluster` skill's known failure patterns -- For inference and provider-topology issues: reference the `debug-inference` skill's known failure patterns -- For CLI/usage issues: reference the `openshell-cli` skill's command reference +- For gateway deployment or infrastructure issues: reference the known failure patterns in `skills/debug-openshell-cluster/SKILL.md` +- For inference and provider-topology issues: reference `skills/debug-inference/SKILL.md` +- For CLI/usage issues: reference the workflows in `skills/openshell-cli/SKILL.md` and confirm installed syntax with `openshell --help` Record impact signals for the human decision: affected users and scope, regression status, workaround availability, severity evidence, and evidence quality. Do not convert those facts into a roadmap or sequencing recommendation. diff --git a/.agents/skills/tui-development/SKILL.md b/.agents/skills/tui-development/SKILL.md index 7f11db26f..edd1397d5 100644 --- a/.agents/skills/tui-development/SKILL.md +++ b/.agents/skills/tui-development/SKILL.md @@ -1,6 +1,8 @@ --- name: tui-development description: Guide for developing the OpenShell TUI — a ratatui-based terminal UI for the OpenShell platform. Covers architecture, navigation, data fetching, theming, UX conventions, and development workflow. Trigger keywords - term, TUI, terminal UI, ratatui, openshell-tui, tui development, tui feature, tui bug. +metadata: + internal: true --- # OpenShell TUI Development Guide @@ -43,8 +45,8 @@ Gateway (discovered via openshell_bootstrap::list_gateways()) - **Gateways** are discovered from on-disk config via `openshell_bootstrap::list_gateways()`. Each gateway has a name, endpoint, local/remote flag, and source label. - **Workspaces** are fetched via `ListWorkspaces`. The user cycles through workspaces with `[w]`, or views all workspaces at once. The current workspace scopes provider and sandbox lists. -- **Provider Profiles** are fetched per-workspace via `ListProviderProfiles` when `providers_v2_enabled` is true. Profiles are cached in a `ProviderProfileCache` keyed by `(workspace, profile_id)` and matched to providers by type. They provide category, credential metadata, endpoint/binary counts, and inference capability. -- **Providers** are fetched via `ListProviders` scoped to the current workspace. Each `ProviderListEntry` pairs a provider with its optional cached profile. When `providers_v2_enabled` is true, CRUD operations are read-only in the TUI; when false, the TUI supports create/update/delete. +- **Provider Profiles** are fetched per-workspace via `ListProviderProfiles`. Profiles are cached in a `ProviderProfileCache` keyed by `(workspace, profile_id)` and matched to providers by type. They provide category, credential metadata, endpoint/binary counts, and inference capability. +- **Providers** are fetched via `ListProviders` scoped to the current workspace. Each `ProviderListEntry` pairs a provider with its optional cached profile. The TUI supports profile-backed create, update, and delete operations. - **Global Settings** are fetched via `GetGatewayConfig` and displayed in a tabbed pane alongside providers on the dashboard. Each setting is a registered key with a typed value (bool/int/string). Platform-admin access is required; `PermissionDenied` disables the pane. - **Sandboxes** belong to the active gateway and workspace. Fetched via `ListSandboxes` with a periodic tick refresh. - **Sandbox Settings** are effective settings returned by `GetSandboxConfig`, each with a scope (sandbox, global, or unset). Globally-managed settings are blocked from sandbox-level edits. @@ -53,7 +55,7 @@ Gateway (discovered via openshell_bootstrap::list_gateways()) The **title bar** always reflects this hierarchy, reading left-to-right from general to specific: ``` - OpenShell │ Current Gateway: [source] () │ Workspace: │ + OpenShell v │ Current Gateway: [source] () │ Workspace: │ ``` ## 3. Navigation & Screen Architecture @@ -142,8 +144,8 @@ Every frame renders four vertical regions: ### Title bar examples -- Dashboard: ` >_ OpenShell ALPHA | Current Gateway: openshell [local] (Healthy) | Workspace: default | Dashboard` -- Sandbox detail: ` >_ OpenShell ALPHA | Current Gateway: openshell [local] (Healthy) | Workspace: team-a | Sandbox: my-sandbox` +- Dashboard: ` >_ OpenShell v | Current Gateway: openshell [local] (Healthy) | Workspace: default | Dashboard` +- Sandbox detail: ` >_ OpenShell v | Current Gateway: openshell [local] (Healthy) | Workspace: team-a | Sandbox: my-sandbox` ### Adding a new screen @@ -170,7 +172,7 @@ Phase 2: WatchSandbox(follow_logs: true) → live tail → send via Event::L **Sandboxes**: Fetched via `ListSandboxes` on a 2-second tick, scoped to the current workspace (or all workspaces). -**Providers**: Fetched via `ListProviders` on each tick. When `providers_v2_enabled` is true, provider profiles are also fetched per-workspace via `ListProviderProfiles` and cached in a `ProviderProfileCache` keyed by `(workspace, profile_id)`. +**Providers**: Fetched via `ListProviders` on each tick. Provider profiles are fetched per-workspace via `ListProviderProfiles` and cached in a `ProviderProfileCache` keyed by `(workspace, profile_id)`. **Settings**: Global settings are fetched via `GetGatewayConfig` on each tick. Sandbox settings are fetched alongside the sandbox policy via `GetSandboxConfig` and refreshed on each tick when viewing a sandbox. @@ -321,7 +323,7 @@ TUI actions should parallel `openshell` CLI commands so users have familiar ment | `openshell sandbox connect` | `[s]` on sandbox policy view to launch SSH shell | | `openshell logs ` | `[l]` on sandbox detail to open log viewer | | `openshell provider list` | Provider table on Dashboard (middle pane) | -| `openshell provider create` | `[c]` on provider panel (when not providers_v2) | +| `openshell provider create` | `[c]` on provider panel | | `openshell status` | Status in title bar + gateway list | When adding new TUI features, check what the CLI offers and maintain consistency. @@ -383,11 +385,8 @@ All actions are accessible via keyboard shortcuts displayed in the nav bar. The **Dashboard (Gateways focus):** `[Tab] Switch Panel [Enter] Select [j/k] Navigate │ [:] Command [q] Quit` -**Dashboard (Providers focus, providers_v2):** -`[Tab] Switch Panel [h/l] Switch Tab [j/k] Navigate [Enter] Detail read-only │ [:] Command [q] Quit` - -**Dashboard (Providers focus, legacy):** -`[Tab] Switch Panel [h/l] Switch Tab [j/k] Navigate [Enter] Detail [c] Create [u] Update [d] Delete │ [:] Command [q] Quit` +**Dashboard (Providers focus):** +`[Tab] Switch Panel [h/l] Switch Tab [j/k] Navigate [Enter] Detail [c] Create [u] Update [d] Delete [w] Workspace │ [:] Command [q] Quit` **Dashboard (Global Settings focus):** `[Tab] Switch Panel [h/l] Switch Tab [j/k] Navigate [Enter] Edit [d] Delete │ [:] Command [q] Quit` @@ -412,7 +411,7 @@ All actions are accessible via keyboard shortcuts displayed in the nav bar. The | File | Purpose | | --- | --- | | `crates/openshell-tui/Cargo.toml` | Crate manifest — dependencies on `openshell-core`, `openshell-bootstrap`, `ratatui`, `crossterm`, `tonic`, `tokio` | -| `crates/openshell-tui/src/lib.rs` | Entry point. Event loop, gRPC calls (`refresh_data`, `refresh_providers`, `refresh_global_settings`, `refresh_workspaces`, `refresh_sandboxes`, `spawn_log_stream`, `handle_sandbox_delete`, `fetch_providers_v2_setting`), gateway switching, mTLS channel building, provider CRUD spawners, settings CRUD spawners, draft approval spawners | +| `crates/openshell-tui/src/lib.rs` | Entry point. Event loop, gRPC calls (`refresh_data`, `refresh_providers`, `refresh_global_settings`, `refresh_workspaces`, `refresh_sandboxes`, `spawn_log_stream`, `handle_sandbox_delete`), gateway switching, mTLS channel building, provider CRUD spawners, settings CRUD spawners, draft approval spawners | | `crates/openshell-tui/src/app.rs` | `App` state struct, `Screen`/`Focus`/`InputMode`/`LogSourceFilter`/`MiddlePaneTab`/`SandboxPolicyTab` enums, `LogLine`/`GatewayEntry`/`GlobalSettingEntry`/`SandboxSettingEntry`/`ProviderListEntry`/`ProviderDetailView` structs, create sandbox/provider form state, all key handling logic | | `crates/openshell-tui/src/event.rs` | `Event` enum (`Key`, `Mouse`, `Tick`, `Redraw`, `Resize`, `LogLines`, `CreateResult`, `ProviderCreateResult`, `ProviderDetailFetched`, `ProviderUpdateResult`, `ProviderDeleteResult`, `DraftActionResult`, `GlobalSettingsFetched`, `GlobalSettingSetResult`, `GlobalSettingDeleteResult`, `SandboxSettingSetResult`, `SandboxSettingDeleteResult`, `ForwardWarnings`), `EventHandler` with mpsc channels and crossterm polling | | `crates/openshell-tui/src/theme.rs` | `colors` module (NVIDIA_GREEN, EVERGLADE, BG, FG) and `styles` module (all `Style` constants) | @@ -470,14 +469,20 @@ Proto types come from `openshell-core` which generates them from `OUT_DIR` via ` ```rust use openshell_core::proto::openshell_client::OpenShellClient; -use openshell_core::proto::{ListSandboxesRequest, GetSandboxLogsRequest, ...}; +use openshell_core::proto::{ + all_workspaces_selector, workspace_selector, GetSandboxLogsRequest, + ListSandboxesRequest, ... +}; ``` ### Proto field gotchas - `DeleteSandboxRequest` uses the `name` field (not `id`): ```rust - let req = openshell_core::proto::DeleteSandboxRequest { name: sandbox_name }; + let req = openshell_core::proto::DeleteSandboxRequest { + name: sandbox_name, + workspace_scope: Some(workspace_selector(workspace)), + }; ``` - `WatchSandboxRequest` has extra fields beyond what you might need — always use `..Default::default()`: ```rust @@ -491,12 +496,24 @@ use openshell_core::proto::{ListSandboxesRequest, GetSandboxLogsRequest, ...}; }; ``` - `SandboxLogLine` proto fields: `sandbox_id`, `timestamp_ms`, `level`, `target`, `message`, `source`, `fields` (HashMap). -- `GetSandboxLogsRequest` fields: `sandbox_id`, `lines` (u32), `since_ms` (i64), `sources` (Vec), `min_level` (String), `workspace` (String). -- `ListSandboxesRequest` fields: `limit` (i64), `offset` (i64), `label_selector` (String), `workspace` (String), `all_workspaces` (bool). -- `ListProvidersRequest` fields: `limit` (i64), `offset` (i64), `workspace` (String), `all_workspaces` (bool). -- `ListWorkspacesRequest` fields: `limit` (i64), `offset` (i64), `label_selector` (String). -- `UpdateConfigRequest` fields: `name` (String, sandbox name or empty for global), `setting_key`, `setting_value`, `delete_setting` (bool), `global` (bool), `workspace`. -- Most resource requests include a `workspace` field that scopes the operation to the current workspace. +- Workspace-scoped request fields use `workspace_scope: Option`. + Select one workspace with `Some(workspace_selector(name))`. List requests that + explicitly support cross-workspace access also accept + `Some(all_workspaces_selector())`; do not use that marker on other requests. +- `GetSandboxLogsRequest` fields: `sandbox_id`, `lines` (u32), `since_ms` (i64), + `sources` (Vec), `min_level` (String), `workspace_scope`. +- `ListSandboxesRequest` fields: `limit` (u32), `offset` (u32), + `label_selector` (String), `workspace_scope`. +- `ListProvidersRequest` fields: `limit` (u32), `offset` (u32), + `workspace_scope`. +- `ListWorkspacesRequest` fields: `limit` (u32), `offset` (u32), + `label_selector` (String). +- `UpdateConfigRequest` fields include `name` (String, sandbox name or empty for + global), `setting_key`, `setting_value`, `delete_setting` (bool), `global` + (bool), and `workspace_scope`. Sandbox-scoped updates require a named selector; + gateway-global updates must leave `workspace_scope` as `None`. +- Most resource requests require an explicit named `workspace_scope`, including + the `default` workspace. An omitted selector is not an implicit default. ### gRPC timeouts @@ -526,7 +543,6 @@ The connect timeout for gateway switching is 10 seconds with HTTP/2 keepalive at 4. On success: - `app.client` is replaced with a new intercepted client - `reset_sandbox_state()` clears all sandbox/log/draft/policy data - - `fetch_providers_v2_setting()` probes the new gateway's `GetGatewayConfig` to determine whether providers_v2 mode is enabled, so provider CRUD controls render correctly - `refresh_data()` runs the full capability refresh sequence: `refresh_health` → `refresh_global_settings` → `refresh_workspaces` → `refresh_providers` → `refresh_sandboxes` 5. On failure: `status_text` shows the error @@ -534,13 +550,12 @@ The connect timeout for gateway switching is 10 seconds with HTTP/2 keepalive at On launch, before the event loop starts: -1. `fetch_providers_v2_setting()` — probe gateway capability -2. `refresh_gateway_list()` — discover gateways from disk -3. `refresh_data()` — full refresh (health, global settings, workspaces, providers, sandboxes) +1. `refresh_gateway_list()` — discover gateways from disk +2. `refresh_data()` — full refresh (health, global settings, workspaces, providers, sandboxes) ### Workspace switching lifecycle -1. User presses `[w]` on the sandboxes panel → `cycle_workspace()` advances through discovered workspace names, then "all" +1. User presses `[w]` on the providers or sandboxes panel → `cycle_workspace()` advances through discovered workspace names, then "all" 2. `pending_workspace_refresh = true` is set, cursor indices are reset 3. Event loop calls `refresh_providers()` and `refresh_sandboxes()` with the new workspace scope diff --git a/.agents/skills/update-docs/SKILL.md b/.agents/skills/update-docs/SKILL.md index f38d7f576..aa2db8cd5 100644 --- a/.agents/skills/update-docs/SKILL.md +++ b/.agents/skills/update-docs/SKILL.md @@ -1,6 +1,8 @@ --- name: update-docs-from-commits description: Scan recent git commits for changes that affect user-facing behavior, then draft or update the corresponding documentation pages. Use when docs have fallen behind code changes, after a batch of features lands, or when preparing a release. Trigger keywords - update docs, draft docs, docs from commits, sync docs, catch up docs, doc debt, docs behind, docs drift. +metadata: + internal: true --- # Update Docs from Commits diff --git a/.agents/skills/watch-github-actions/SKILL.md b/.agents/skills/watch-github-actions/SKILL.md index a7e7ea46d..c9e284331 100644 --- a/.agents/skills/watch-github-actions/SKILL.md +++ b/.agents/skills/watch-github-actions/SKILL.md @@ -1,6 +1,8 @@ --- name: watch-github-actions description: Watch and monitor GitHub Actions workflow runs using the gh CLI. Use when the user wants to check workflow status, watch a running workflow, view CI/CD jobs, or monitor build progress. Trigger keywords - watch pipeline, pipeline status, CI status, check build, monitor CI, view pipeline, pipeline progress, workflow status, actions status. +metadata: + internal: true --- # Watch GitHub Actions diff --git a/.config/nextest.toml b/.config/nextest.toml index 43783c18e..1c1235c6e 100644 --- a/.config/nextest.toml +++ b/.config/nextest.toml @@ -3,6 +3,8 @@ [profile.ci] fail-fast = false -status-level = "fail" +status-level = "slow" final-status-level = "fail" failure-output = "immediate-final" +# Print slow test names after 30 seconds and terminate them after two minutes. +slow-timeout = { period = "30s", terminate-after = 4 } diff --git a/.github/ISSUE_TEMPLATE/config.yml b/.github/ISSUE_TEMPLATE/config.yml index 28e6b201b..0f6486fde 100644 --- a/.github/ISSUE_TEMPLATE/config.yml +++ b/.github/ISSUE_TEMPLATE/config.yml @@ -7,11 +7,10 @@ contact_links: Vouch Request discussion describing what you want to work on. A maintainer will approve you with /vouch. - name: Have a question? - url: https://github.com/NVIDIA/OpenShell/blob/main/CONTRIBUTING.md#agent-skills-for-contributors + url: https://github.com/NVIDIA/OpenShell/blob/main/CONTRIBUTING.md#skills-for-using-openshell about: > - Point your agent at the repo. It has skills for CLI usage, cluster and - inference debugging, policy generation, and more. See CONTRIBUTING.md - for the full skills table. + Install the public OpenShell skills for CLI usage, cluster and inference + debugging, and policy generation. See CONTRIBUTING.md for details. - name: Security vulnerability? url: https://github.com/NVIDIA/OpenShell/blob/main/SECURITY.md about: > diff --git a/.github/actions/build-rust-binary/action.yml b/.github/actions/build-rust-binary/action.yml index 408219b57..74041e9fc 100644 --- a/.github/actions/build-rust-binary/action.yml +++ b/.github/actions/build-rust-binary/action.yml @@ -14,9 +14,6 @@ inputs: triple: description: Rust target triple required: true - dev-shell: - description: Nix development shell used to build the binary - required: true cargo-version: description: Cargo package version embedded in the binary required: true @@ -32,10 +29,6 @@ inputs: description: Additional flags passed to cargo build required: false default: "" - interpreter: - description: ELF interpreter for a dynamically linked Linux binary - required: false - default: "" runs: using: composite @@ -43,9 +36,9 @@ runs: - name: Hash development shell id: dev-shell shell: bash - env: - DEV_SHELL: ${{ inputs.dev-shell }} - run: echo "hash=$(nix hash file --type sha256 --base16 "$(nix eval --raw "${DEV_SHELL}.drvPath")")" >> "$GITHUB_OUTPUT" + run: | + shell_drv=$(nix eval --raw --impure .#devShells --apply 'shells: shells.${builtins.currentSystem}.default.drvPath') + echo "hash=$(nix hash file --type sha256 --base16 "$shell_drv")" >> "$GITHUB_OUTPUT" - name: Cache Rust artifacts uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6 # v2.9.2 @@ -54,90 +47,41 @@ runs: cache-on-failure: "true" cache-workspace-crates: "true" cache-bin: "false" - cmd-format: nix develop ${{ inputs.dev-shell }} -c {0} + cmd-format: nix develop -c {0} - name: Set version - shell: nix develop ${{ inputs.dev-shell }} -c bash -euo pipefail {0} + shell: nix develop -c bash -euo pipefail {0} env: INPUTS_CARGO_VERSION: ${{ inputs.cargo-version }} run: sed -i "s/^version = \"0\\.0\\.0\"$/version = \"${INPUTS_CARGO_VERSION}\"/" Cargo.toml - name: Build ${{ inputs.binary }} - shell: nix develop ${{ inputs.dev-shell }} -c bash -euo pipefail {0} + shell: nix develop -c bash -euo pipefail {0} env: + GIT_DIR: /nonexistent OPENSHELL_IMAGE_TAG: ${{ inputs.image-tag }} INPUTS_PACKAGE: ${{ inputs.package }} INPUTS_BINARY: ${{ inputs.binary }} + INPUTS_TRIPLE: ${{ inputs.triple }} INPUTS_EXTRA_CARGO_FLAGS: ${{ inputs.extra-cargo-flags }} - run: GIT_DIR=/nonexistent cargo auditable build --release --package "${INPUTS_PACKAGE}" --bin "${INPUTS_BINARY}" ${INPUTS_EXTRA_CARGO_FLAGS} - - - name: Normalize Darwin dynamic binary - if: endsWith(inputs.triple, '-apple-darwin') - shell: bash -euo pipefail {0} - env: - INPUTS_BINARY: ${{ inputs.binary }} - run: | - binary=target/release/${INPUTS_BINARY} - nix_iconv_paths=$(otool -L "$binary" | awk '$1 ~ "^/nix/store/.*/lib/libiconv\\.2\\.dylib$" { print $1 }') - if [[ -n "$nix_iconv_paths" ]]; then - while IFS= read -r nix_iconv_path; do - install_name_tool -change "$nix_iconv_path" /usr/lib/libiconv.2.dylib "$binary" - done <<< "$nix_iconv_paths" - codesign --force --sign - "$binary" - fi - - - name: Verify Darwin portability - if: endsWith(inputs.triple, '-apple-darwin') - shell: bash -euo pipefail {0} - env: - INPUTS_BINARY: ${{ inputs.binary }} - run: | - binary=target/release/${INPUTS_BINARY} - nix_dependencies=$(otool -L "$binary" | awk '$1 ~ "^/nix/store/" { print $1 }') - if [[ -n "$nix_dependencies" ]]; then - echo "error: $binary has non-portable Nix store dependencies:" >&2 - printf ' %s\n' "$nix_dependencies" >&2 - exit 1 - fi - codesign --verify --verbose=2 "$binary" + run: cargo auditable build --release --target "${INPUTS_TRIPLE}" --package "${INPUTS_PACKAGE}" --bin "${INPUTS_BINARY}" ${INPUTS_EXTRA_CARGO_FLAGS} - name: Verify ${{ inputs.binary }} - shell: nix develop ${{ inputs.dev-shell }} -c bash -euo pipefail {0} + shell: nix develop -c bash -euo pipefail {0} env: INPUTS_BINARY: ${{ inputs.binary }} + INPUTS_TRIPLE: ${{ inputs.triple }} run: | # Confirm the binary runs and reports the expected name. - target/release/${INPUTS_BINARY} --version | grep -q "^${INPUTS_BINARY} " + target/${INPUTS_TRIPLE}/release/${INPUTS_BINARY} --version | grep -q "^${INPUTS_BINARY} " # Confirm Syft can decode the embedded cargo-auditable metadata. - SYFT_CHECK_FOR_APP_UPDATE=false syft file:target/release/${INPUTS_BINARY} -o cyclonedx-json | grep 'pkg:cargo/' > /dev/null - - - name: Verify static linkage - if: endsWith(inputs.triple, '-linux-musl') - shell: nix develop ${{ inputs.dev-shell }} -c bash -euo pipefail {0} - env: - INPUTS_BINARY: ${{ inputs.binary }} - run: tasks/scripts/verify-static-binary.sh target/release/${INPUTS_BINARY} - - - name: Normalize Linux dynamic binary - if: endsWith(inputs.triple, '-linux-gnu') - shell: nix develop ${{ inputs.dev-shell }} -c bash -euo pipefail {0} - env: - INPUTS_BINARY: ${{ inputs.binary }} - INPUTS_INTERPRETER: ${{ inputs.interpreter }} - run: | - binary=target/release/${INPUTS_BINARY} - # Remove Nix store paths so the binary can run on other distributions. - patchelf --set-interpreter "${INPUTS_INTERPRETER}" --remove-rpath "$binary" - # Z3 must be embedded instead of loaded from the target system. - test -z "$(patchelf --print-needed "$binary" | grep '^libz3')" - # Reject symbols introduced after glibc 2.28. - tasks/scripts/verify-glibc-symbols.sh 2.28 "$binary" + SYFT_CHECK_FOR_APP_UPDATE=false syft file:target/${INPUTS_TRIPLE}/release/${INPUTS_BINARY} -o cyclonedx-json | grep 'pkg:cargo/' > /dev/null - name: Upload ${{ inputs.binary }} uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 with: name: ${{ inputs.artifact-name || format('{0}-{1}', inputs.binary, inputs.triple) }} - path: target/release/${{ inputs.binary }} + path: target/${{ inputs.triple }}/release/${{ inputs.binary }} compression-level: 0 retention-days: 5 if-no-files-found: error diff --git a/.github/actions/release-helm-oci/action.yml b/.github/actions/release-helm-oci/action.yml index d20691ad0..46649e0cb 100644 --- a/.github/actions/release-helm-oci/action.yml +++ b/.github/actions/release-helm-oci/action.yml @@ -4,7 +4,7 @@ name: Release Helm OCI description: > Patch chart version/appVersion, refuse duplicate OCI versions on public - releases, package the chart, and push to GHCR OCI. + releases, package the gateway and workspace charts, and push them to GHCR OCI. inputs: chart-version: @@ -51,11 +51,16 @@ runs: shell: bash run: | set -euo pipefail - CHART_DIR="${RUNNER_TEMP}/chart-build" - cp -a deploy/helm/openshell/. "${CHART_DIR}" - sed -i "s/^version:.*/version: ${CHART_VERSION}/" "${CHART_DIR}/Chart.yaml" - sed -i "s/^appVersion:.*/appVersion: \"${APP_VERSION}\"/" "${CHART_DIR}/Chart.yaml" - echo "chart_dir=${CHART_DIR}" >> "$GITHUB_OUTPUT" + GATEWAY_CHART_DIR="${RUNNER_TEMP}/gateway-chart-build" + WORKSPACE_CHART_DIR="${RUNNER_TEMP}/workspace-chart-build" + cp -a deploy/helm/openshell/. "${GATEWAY_CHART_DIR}" + cp -a deploy/helm/openshell-workspace/. "${WORKSPACE_CHART_DIR}" + for chart_dir in "${GATEWAY_CHART_DIR}" "${WORKSPACE_CHART_DIR}"; do + sed -i "s/^version:.*/version: ${CHART_VERSION}/" "${chart_dir}/Chart.yaml" + sed -i "s/^appVersion:.*/appVersion: \"${APP_VERSION}\"/" "${chart_dir}/Chart.yaml" + done + echo "gateway_chart_dir=${GATEWAY_CHART_DIR}" >> "$GITHUB_OUTPUT" + echo "workspace_chart_dir=${WORKSPACE_CHART_DIR}" >> "$GITHUB_OUTPUT" echo "chart_version=${CHART_VERSION}" >> "$GITHUB_OUTPUT" - name: Refuse duplicate chart version @@ -65,38 +70,52 @@ runs: shell: bash run: | set -euo pipefail - OCI_CHART="oci://ghcr.io/nvidia/openshell/helm-chart" - if helm show chart "${OCI_CHART}" --version "${CHART_VERSION}" >/dev/null 2>&1; then - echo "::error::Chart ${CHART_VERSION} is already published. Use a new tag or delete the existing package first." - exit 1 - fi + for chart in helm-chart openshell-workspace; do + OCI_CHART="oci://ghcr.io/nvidia/openshell/${chart}" + if helm show chart "${OCI_CHART}" --version "${CHART_VERSION}" >/dev/null 2>&1; then + echo "::error::Chart ${chart}:${CHART_VERSION} is already published. Use a new tag or delete the existing package first." + exit 1 + fi + done - name: Package Helm chart env: - CHART_DIR: ${{ steps.prep.outputs.chart_dir }} + GATEWAY_CHART_DIR: ${{ steps.prep.outputs.gateway_chart_dir }} + WORKSPACE_CHART_DIR: ${{ steps.prep.outputs.workspace_chart_dir }} shell: bash run: | set -euo pipefail - helm package "${CHART_DIR}" --destination /tmp - ls /tmp/helm-chart-*.tgz + mkdir -p /tmp/helm-charts + helm package "${GATEWAY_CHART_DIR}" --destination /tmp/helm-charts + helm package "${WORKSPACE_CHART_DIR}" --destination /tmp/helm-charts + ls /tmp/helm-charts/*.tgz - name: Push Helm chart to GHCR OCI shell: bash run: | set -euo pipefail - helm push /tmp/helm-chart-*.tgz oci://ghcr.io/nvidia/openshell + for archive in /tmp/helm-charts/*.tgz; do + helm push "${archive}" oci://ghcr.io/nvidia/openshell + done - name: Push SHA-pinned chart if: inputs.pin-sha != '' env: PIN_SHA: ${{ inputs.pin-sha }} - CHART_DIR: ${{ steps.prep.outputs.chart_dir }} + GATEWAY_CHART_DIR: ${{ steps.prep.outputs.gateway_chart_dir }} + WORKSPACE_CHART_DIR: ${{ steps.prep.outputs.workspace_chart_dir }} shell: bash run: | set -euo pipefail - SHA_CHART_DIR="${RUNNER_TEMP}/chart-build-sha" - cp -a "${CHART_DIR}/." "${SHA_CHART_DIR}" - sed -i "s/^version:.*/version: 0.0.0-dev.${PIN_SHA}/" "${SHA_CHART_DIR}/Chart.yaml" - sed -i "s/^appVersion:.*/appVersion: \"${PIN_SHA}\"/" "${SHA_CHART_DIR}/Chart.yaml" - helm package "${SHA_CHART_DIR}" --destination /tmp/sha-pin - helm push /tmp/sha-pin/helm-chart-*.tgz oci://ghcr.io/nvidia/openshell + mkdir -p /tmp/sha-pin + for source_dir in "${GATEWAY_CHART_DIR}" "${WORKSPACE_CHART_DIR}"; do + chart_name="$(basename "${source_dir}")" + sha_chart_dir="${RUNNER_TEMP}/${chart_name}-sha" + cp -a "${source_dir}/." "${sha_chart_dir}" + sed -i "s/^version:.*/version: 0.0.0-dev.${PIN_SHA}/" "${sha_chart_dir}/Chart.yaml" + sed -i "s/^appVersion:.*/appVersion: \"${PIN_SHA}\"/" "${sha_chart_dir}/Chart.yaml" + helm package "${sha_chart_dir}" --destination /tmp/sha-pin + done + for archive in /tmp/sha-pin/*.tgz; do + helm push "${archive}" oci://ghcr.io/nvidia/openshell + done diff --git a/.github/actions/setup-e2e-cli/action.yml b/.github/actions/setup-e2e-cli/action.yml index 4d43e232a..d7d1c8a29 100644 --- a/.github/actions/setup-e2e-cli/action.yml +++ b/.github/actions/setup-e2e-cli/action.yml @@ -1,5 +1,11 @@ name: Setup E2E CLI -description: Download an architecture-matched prebuilt OpenShell CLI for E2E tests +description: Download architecture-matched prebuilt OpenShell host CLIs for E2E tests + +inputs: + conformance-artifact-prefix: + description: Optional conformance CLI artifact name prefix; linux- is appended automatically + required: false + default: "" runs: using: composite @@ -17,3 +23,27 @@ runs: chmod +x "$cli" "$cli" --version echo "OPENSHELL_BIN=$cli" >> "$GITHUB_ENV" + + - name: Download prebuilt conformance CLI + if: inputs.conformance-artifact-prefix != '' + uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 + with: + name: ${{ format('{0}-{1}', inputs.conformance-artifact-prefix, runner.arch == 'X64' && 'x86_64-unknown-linux-musl' || 'aarch64-unknown-linux-musl') }} + path: .e2e/prebuilt-conformance + + - name: Configure prebuilt conformance CLI + if: inputs.conformance-artifact-prefix != '' + shell: bash + run: | # zizmor: ignore[github-env] validated filename under the trusted workspace path + set -euo pipefail + conformance="$GITHUB_WORKSPACE/.e2e/prebuilt-conformance/openshell-conformance" + if [[ ! -f "$conformance" ]]; then + echo "downloaded artifact is missing $conformance" >&2 + exit 1 + fi + # TODO: Remove this temporary mode diagnostic after CI confirms artifact permission handling. + echo "conformance binary mode before chmod: $(ls -l "$conformance")" + chmod +x "$conformance" + echo "conformance binary mode after chmod: $(ls -l "$conformance")" + "$conformance" list --output json >/dev/null + echo "OPENSHELL_CONFORMANCE_BIN=$conformance" >> "$GITHUB_ENV" diff --git a/.github/workflows/branch-checks.yml b/.github/workflows/branch-checks.yml index 2d9dc9914..6739434bf 100644 --- a/.github/workflows/branch-checks.yml +++ b/.github/workflows/branch-checks.yml @@ -41,6 +41,7 @@ jobs: needs: pr_metadata if: needs.pr_metadata.outputs.should_run == 'true' runs-on: linux-amd64-cpu8 + timeout-minutes: 30 container: image: ghcr.io/nvidia/openshell/ci:latest credentials: @@ -65,6 +66,7 @@ jobs: needs: pr_metadata if: needs.pr_metadata.outputs.should_run == 'true' runs-on: linux-amd64-cpu8 + timeout-minutes: 30 container: image: ghcr.io/nvidia/openshell/ci:latest credentials: @@ -84,6 +86,7 @@ jobs: needs: pr_metadata if: needs.pr_metadata.outputs.should_run == 'true' runs-on: linux-amd64-cpu8 + timeout-minutes: 30 defaults: run: shell: nix develop .#devShells.x86_64-linux.default -c bash -euo pipefail {0} @@ -117,6 +120,7 @@ jobs: - runner: macos-15-xlarge system: aarch64-darwin runs-on: ${{ matrix.runner }} + timeout-minutes: 90 defaults: run: shell: nix develop .#devShells.${{ matrix.system }}.default -c bash -euo pipefail {0} @@ -154,12 +158,14 @@ jobs: cargo fmt --all -- --check cargo fmt --manifest-path e2e/rust/Cargo.toml --all -- --check cargo fmt --manifest-path examples/governance-interceptor/Cargo.toml --all -- --check + cargo fmt --manifest-path examples/supervisor-middleware-content-guard/Cargo.toml --all -- --check - name: Lint run: | cargo clippy --workspace --all-targets -- -D warnings cargo clippy --manifest-path e2e/rust/Cargo.toml --all-targets -- -D warnings cargo check --manifest-path examples/governance-interceptor/Cargo.toml --all-targets + cargo check --manifest-path examples/supervisor-middleware-content-guard/Cargo.toml --all-targets - name: Test env: @@ -167,6 +173,30 @@ jobs: run: | cargo nextest run --profile ci --workspace --features openshell-server/test-support + - name: Verify telemetry can be compiled out + run: | + cargo build -p openshell-gateway --bin openshell-gateway + tasks/scripts/verify-telemetry-compiled-out.sh present target/debug/openshell-gateway + cargo build -p openshell-gateway --bin openshell-gateway --no-default-features --features defaults-without-telemetry + tasks/scripts/verify-telemetry-compiled-out.sh absent target/debug/openshell-gateway + cargo build -p openshell-sandbox --bin openshell-sandbox --no-default-features --features defaults-without-telemetry + tasks/scripts/verify-telemetry-compiled-out.sh absent target/debug/openshell-sandbox + + - name: Verify the defaults-without-telemetry feature alias tracks the default feature set + run: tasks/scripts/verify-defaults-without-telemetry.sh + + - name: Verify system CA roots build mode compiles and excludes bundled Mozilla roots + run: | + cargo check -p openshell-sandbox --all-targets --no-default-features --features system-ca-roots + if cargo tree -p openshell-sandbox -i webpki-roots --no-default-features --features system-ca-roots 2>/dev/null | grep -q webpki-roots; then + echo "ERROR: webpki-roots found in system CA roots build" >&2 + exit 1 + fi + if cargo tree -p openshell-sandbox -i webpki-root-certs --no-default-features --features system-ca-roots 2>/dev/null | grep -q webpki-root-certs; then + echo "ERROR: webpki-root-certs found in system CA roots build" >&2 + exit 1 + fi + python: name: Python (${{ matrix.runner }}) needs: pr_metadata @@ -176,6 +206,7 @@ jobs: matrix: runner: [linux-amd64-cpu8, linux-arm64-cpu8] runs-on: ${{ matrix.runner }} + timeout-minutes: 30 container: image: ghcr.io/nvidia/openshell/ci:latest credentials: @@ -207,6 +238,7 @@ jobs: needs: pr_metadata if: needs.pr_metadata.outputs.should_run == 'true' runs-on: linux-amd64-cpu8 + timeout-minutes: 30 container: image: ghcr.io/nvidia/openshell/ci:latest credentials: @@ -226,6 +258,7 @@ jobs: needs: pr_metadata if: needs.pr_metadata.outputs.should_run == 'true' runs-on: linux-amd64-cpu8 + timeout-minutes: 30 container: image: ghcr.io/nvidia/openshell/ci:latest credentials: @@ -245,6 +278,7 @@ jobs: needs: pr_metadata if: needs.pr_metadata.outputs.should_run == 'true' runs-on: linux-amd64-cpu8 + timeout-minutes: 30 container: image: ghcr.io/nvidia/openshell/ci:latest credentials: diff --git a/.github/workflows/branch-e2e.yml b/.github/workflows/branch-e2e.yml index 0b050d0e5..20183eecb 100644 --- a/.github/workflows/branch-e2e.yml +++ b/.github/workflows/branch-e2e.yml @@ -105,6 +105,28 @@ jobs: cargo-version: ${{ needs.version.outputs.cargo }} secrets: inherit + build-conformance: + needs: [pr_metadata, version] + if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_any_e2e == 'true' + permissions: + contents: read + strategy: + matrix: + include: + - triple: x86_64-unknown-linux-musl + runner: linux-amd64-cpu8 + - triple: aarch64-unknown-linux-musl + runner: linux-arm64-cpu8 + uses: ./.github/workflows/build-binaries.yml + with: + package: openshell-conformance-cli + binary: openshell-conformance + artifact-name: openshell-conformance-${{ matrix.triple }} + triple: ${{ matrix.triple }} + runner: ${{ matrix.runner }} + cargo-version: ${{ needs.version.outputs.cargo }} + secrets: inherit + build-sandbox: needs: version permissions: @@ -133,23 +155,17 @@ jobs: include: - triple: x86_64-unknown-linux-gnu runner: linux-amd64-cpu8 - dev_shell: .#devShells.x86_64-linux.glibc-2-28 - interpreter: /lib64/ld-linux-x86-64.so.2 - triple: aarch64-unknown-linux-gnu runner: linux-arm64-cpu8 - dev_shell: .#devShells.aarch64-linux.glibc-2-28 - interpreter: /lib/ld-linux-aarch64.so.1 uses: ./.github/workflows/build-binaries.yml with: - package: openshell-server + package: openshell-gateway binary: openshell-gateway artifact-name: openshell-gateway-plain-${{ matrix.triple }} triple: ${{ matrix.triple }} runner: ${{ matrix.runner }} - dev-shell: ${{ matrix.dev_shell }} cargo-version: ${{ needs.version.outputs.cargo }} extra-cargo-flags: --no-default-features - interpreter: ${{ matrix.interpreter }} secrets: inherit build-driver-docker: @@ -163,9 +179,7 @@ jobs: binary: openshell-driver-docker triple: aarch64-unknown-linux-gnu runner: linux-arm64-cpu8 - dev-shell: .#devShells.aarch64-linux.glibc-2-28 cargo-version: ${{ needs.version.outputs.cargo }} - interpreter: /lib/ld-linux-aarch64.so.1 secrets: inherit build-driver-podman: @@ -179,9 +193,7 @@ jobs: binary: openshell-driver-podman triple: x86_64-unknown-linux-gnu runner: linux-amd64-cpu8 - dev-shell: .#devShells.x86_64-linux.glibc-2-28 cargo-version: ${{ needs.version.outputs.cargo }} - interpreter: /lib64/ld-linux-x86-64.so.2 secrets: inherit build-driver-kubernetes: @@ -195,9 +207,7 @@ jobs: binary: openshell-driver-kubernetes triple: x86_64-unknown-linux-gnu runner: linux-amd64-cpu8 - dev-shell: .#devShells.x86_64-linux.glibc-2-28 cargo-version: ${{ needs.version.outputs.cargo }} - interpreter: /lib64/ld-linux-x86-64.so.2 secrets: inherit build-vm-driver: @@ -241,7 +251,7 @@ jobs: secrets: inherit docker-e2e: - needs: [pr_metadata, build-cli, build-gateway, build-supervisor-image] + needs: [pr_metadata, build-cli, build-conformance, build-gateway, build-supervisor-image] if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_core_e2e == 'true' permissions: actions: read @@ -251,9 +261,10 @@ jobs: with: image-tag: ${{ github.sha }} runner: linux-arm64-cpu8 + conformance-artifact-prefix: openshell-conformance podman-e2e: - needs: [pr_metadata, build-cli, build-gateway, build-supervisor-image] + needs: [pr_metadata, build-cli, build-conformance, build-gateway, build-supervisor-image] if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_core_e2e == 'true' permissions: actions: read @@ -262,18 +273,21 @@ jobs: uses: ./.github/workflows/e2e-podman-test.yml with: image-tag: ${{ github.sha }} + conformance-artifact-prefix: openshell-conformance vm-e2e: - needs: [pr_metadata, build-cli, build-gateway, build-vm-driver] + needs: [pr_metadata, build-cli, build-conformance, build-gateway, build-vm-driver] if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_core_e2e == 'true' permissions: actions: read contents: read packages: read uses: ./.github/workflows/e2e-vm-test.yml + with: + conformance-artifact-prefix: openshell-conformance docker-external-driver-e2e: - needs: [pr_metadata, build-cli, build-gateway-plain, build-driver-docker, build-supervisor-image] + needs: [pr_metadata, build-cli, build-conformance, build-gateway-plain, build-driver-docker, build-supervisor-image] if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_core_e2e == 'true' permissions: actions: read @@ -285,11 +299,12 @@ jobs: runner: linux-arm64-cpu8 gateway-artifact: openshell-gateway-plain-aarch64-unknown-linux-gnu external-driver-binary: openshell-driver-docker + conformance-artifact-prefix: openshell-conformance suite-matrix: >- [{"suite":"external-driver","cmd":"mise run --no-deps --skip-deps e2e:docker:external-driver","apt_packages":"openssh-client","python_proto":false,"mcp":false}] podman-external-driver-e2e: - needs: [pr_metadata, build-cli, build-gateway-plain, build-driver-podman, build-supervisor-image] + needs: [pr_metadata, build-cli, build-conformance, build-gateway-plain, build-driver-podman, build-supervisor-image] if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_core_e2e == 'true' permissions: actions: read @@ -300,11 +315,12 @@ jobs: image-tag: ${{ github.sha }} gateway-artifact: openshell-gateway-plain-x86_64-unknown-linux-gnu external-driver-binary: openshell-driver-podman + conformance-artifact-prefix: openshell-conformance suite-matrix: >- [{"suite":"external-driver","runner":"ubuntu-26.04","podman_major":"5","podman_package_version":"5.7.0+ds2-3build1","conmon_package_version":"2.1.13+ds1-2","cmd":"mise run --no-deps --skip-deps e2e:podman:external-driver"}] vm-external-driver-e2e: - needs: [pr_metadata, build-cli, build-gateway-plain, build-vm-driver] + needs: [pr_metadata, build-cli, build-conformance, build-gateway-plain, build-vm-driver] if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_core_e2e == 'true' permissions: actions: read @@ -315,9 +331,10 @@ jobs: gateway-artifact: openshell-gateway-plain-x86_64-unknown-linux-gnu suite-name: external-driver e2e-task: e2e:vm:external-driver + conformance-artifact-prefix: openshell-conformance gpu-e2e: - needs: [pr_metadata, build-cli, build-gateway, build-supervisor-image] + needs: [pr_metadata, build-cli, build-conformance, build-gateway, build-supervisor-image] if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_gpu_e2e == 'true' permissions: actions: read @@ -326,9 +343,10 @@ jobs: uses: ./.github/workflows/e2e-gpu-test.yaml with: image-tag: ${{ github.sha }} + conformance-artifact-prefix: openshell-conformance kubernetes-e2e: - needs: [pr_metadata, build-cli, build-gateway-image, build-supervisor-image] + needs: [pr_metadata, build-cli, build-conformance, build-gateway-image, build-supervisor-image] if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_core_e2e == 'true' strategy: fail-fast: false @@ -356,9 +374,10 @@ jobs: job-name: Kubernetes E2E (Rust smoke, ${{ matrix.topology }}, Agent Sandbox ${{ matrix.agent_sandbox_api }}) agent-sandbox-version: ${{ matrix.agent_sandbox_version }} extra-helm-values: ${{ matrix.extra_helm_values }} + conformance-artifact-prefix: openshell-conformance kubernetes-workspace-managed-e2e: - needs: [pr_metadata, build-cli, build-gateway-image, build-supervisor-image] + needs: [pr_metadata, build-cli, build-conformance, build-gateway-image, build-supervisor-image] if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_core_e2e == 'true' permissions: actions: read @@ -369,9 +388,10 @@ jobs: image-tag: ${{ github.sha }} job-name: Kubernetes E2E (workspace managed mode) e2e-task: e2e:kubernetes:workspace-managed + conformance-artifact-prefix: openshell-conformance kubernetes-external-driver-e2e: - needs: [pr_metadata, build-cli, build-gateway-plain, build-driver-kubernetes, build-supervisor-image] + needs: [pr_metadata, build-cli, build-conformance, build-gateway-plain, build-driver-kubernetes, build-supervisor-image] if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_core_e2e == 'true' permissions: actions: read @@ -385,9 +405,10 @@ jobs: gateway-artifact: openshell-gateway-plain-x86_64-unknown-linux-gnu external-driver-binary: openshell-driver-kubernetes cluster-images: supervisor + conformance-artifact-prefix: openshell-conformance kubernetes-workspace-operator-e2e: - needs: [pr_metadata, build-cli, build-gateway-image, build-supervisor-image] + needs: [pr_metadata, build-cli, build-conformance, build-gateway-image, build-supervisor-image] if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_core_e2e == 'true' permissions: actions: read @@ -398,9 +419,10 @@ jobs: image-tag: ${{ github.sha }} job-name: Kubernetes E2E (workspace operator mode) e2e-task: e2e:kubernetes:workspace-operator + conformance-artifact-prefix: openshell-conformance kubernetes-ha-e2e: - needs: [pr_metadata, build-cli, build-gateway-image, build-supervisor-image] + needs: [pr_metadata, build-cli, build-conformance, build-gateway-image, build-supervisor-image] if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_kubernetes_ha_e2e == 'true' permissions: actions: read @@ -412,9 +434,10 @@ jobs: job-name: Kubernetes HA E2E (Rust smoke) extra-helm-values: deploy/helm/openshell/ci/values-high-availability.yaml external-postgres-secret: openshell-ha-pg + conformance-artifact-prefix: openshell-conformance kubernetes-credential-drivers-e2e: - needs: [pr_metadata, build-cli, build-gateway-image, build-supervisor-image] + needs: [pr_metadata, build-cli, build-conformance, build-gateway-image, build-supervisor-image] if: needs.pr_metadata.outputs.should_run == 'true' && needs.pr_metadata.outputs.run_kubernetes_credential_drivers_e2e == 'true' permissions: actions: read @@ -425,6 +448,7 @@ jobs: image-tag: ${{ github.sha }} job-name: Kubernetes Credential Drivers E2E e2e-task: e2e:kubernetes:credential-drivers + conformance-artifact-prefix: openshell-conformance core-e2e-result: name: Core E2E result diff --git a/.github/workflows/build-binaries.yml b/.github/workflows/build-binaries.yml index 1305108fd..4d95b525a 100644 --- a/.github/workflows/build-binaries.yml +++ b/.github/workflows/build-binaries.yml @@ -18,9 +18,6 @@ on: runner: required: true type: string - dev-shell: - required: true - type: string cargo-version: required: true type: string @@ -36,10 +33,6 @@ on: required: false type: string default: "" - interpreter: - required: false - type: string - default: "" checkout-ref: required: false type: string @@ -70,9 +63,7 @@ jobs: package: ${{ inputs.package }} binary: ${{ inputs.binary }} triple: ${{ inputs.triple }} - dev-shell: ${{ inputs['dev-shell'] }} cargo-version: ${{ inputs['cargo-version'] }} image-tag: ${{ inputs['image-tag'] }} artifact-name: ${{ inputs['artifact-name'] }} extra-cargo-flags: ${{ inputs['extra-cargo-flags'] }} - interpreter: ${{ inputs.interpreter }} diff --git a/.github/workflows/build-cli-binaries.yml b/.github/workflows/build-cli-binaries.yml index b8fa96dd7..95fd3e391 100644 --- a/.github/workflows/build-cli-binaries.yml +++ b/.github/workflows/build-cli-binaries.yml @@ -31,20 +31,16 @@ jobs: include: - triple: x86_64-unknown-linux-musl runner: linux-amd64-cpu8 - dev_shell: .#devShells.x86_64-linux.musl - triple: aarch64-unknown-linux-musl runner: linux-arm64-cpu8 - dev_shell: .#devShells.aarch64-linux.musl - triple: aarch64-apple-darwin runner: macos-15-xlarge - dev_shell: .#devShells.aarch64-darwin.default uses: ./.github/workflows/build-binaries.yml with: package: openshell-cli binary: openshell triple: ${{ matrix.triple }} runner: ${{ matrix.runner }} - dev-shell: ${{ matrix.dev_shell }} cargo-version: ${{ inputs.cargo-version }} image-tag: ${{ inputs.image-tag }} checkout-ref: ${{ inputs.checkout-ref }} diff --git a/.github/workflows/build-gateway-binaries.yml b/.github/workflows/build-gateway-binaries.yml index ed5219e7c..c53aeac0d 100644 --- a/.github/workflows/build-gateway-binaries.yml +++ b/.github/workflows/build-gateway-binaries.yml @@ -31,25 +31,17 @@ jobs: include: - triple: x86_64-unknown-linux-gnu runner: linux-amd64-cpu8 - dev_shell: .#devShells.x86_64-linux.glibc-2-28 - interpreter: /lib64/ld-linux-x86-64.so.2 - triple: aarch64-unknown-linux-gnu runner: linux-arm64-cpu8 - dev_shell: .#devShells.aarch64-linux.glibc-2-28 - interpreter: /lib/ld-linux-aarch64.so.1 - triple: aarch64-apple-darwin runner: macos-15-xlarge - dev_shell: .#devShells.aarch64-darwin.default - interpreter: "" uses: ./.github/workflows/build-binaries.yml with: - package: openshell-server + package: openshell-gateway binary: openshell-gateway triple: ${{ matrix.triple }} runner: ${{ matrix.runner }} - dev-shell: ${{ matrix.dev_shell }} cargo-version: ${{ inputs.cargo-version }} image-tag: ${{ inputs.image-tag }} - interpreter: ${{ matrix.interpreter }} checkout-ref: ${{ inputs.checkout-ref }} secrets: inherit diff --git a/.github/workflows/build-rpm.yml b/.github/workflows/build-rpm.yml new file mode 100644 index 000000000..71aa04ddd --- /dev/null +++ b/.github/workflows/build-rpm.yml @@ -0,0 +1,123 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +name: Build RPM + +on: + workflow_call: + inputs: + checkout-ref: + required: true + type: string + arch: + required: true + type: string + runner: + required: true + type: string + cli-target: + required: true + type: string + gateway-target: + required: true + type: string + rpm-version: + required: false + type: string + default: "" + rpm-release: + required: false + type: string + default: "" + cargo-version: + required: false + type: string + default: "" + +permissions: + contents: read + +defaults: + run: + shell: bash + +jobs: + build: + name: Build RPM Package (Linux ${{ inputs.arch }}) + runs-on: ${{ inputs.runner }} + timeout-minutes: 60 + container: + image: docker.io/library/fedora:44@sha256:43b29f65a41eb9c35e1cd5323e3bdf3b655c2357a9f4f1ff2f9c2798e5045d80 + steps: + - name: Install packaging dependencies + run: | + dnf install -y \ + packit rpm-build \ + cargo cargo-rpm-macros git-core \ + pandoc python3-devel systemd-rpm-macros + + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + ref: ${{ inputs.checkout-ref }} + fetch-depth: 0 + + - name: Cache Cargo dependencies + uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6 # v2.9.2 + with: + shared-key: rpm-vendor-${{ inputs.arch }} + cache-targets: "false" + cache-bin: "false" + cache-on-failure: "true" + + - name: Download CLI artifact + uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 + with: + name: openshell-${{ inputs.cli-target }} + path: package-binaries/ + + - name: Download gateway artifact + uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 + with: + name: openshell-gateway-${{ inputs.gateway-target }} + path: package-binaries/ + + - name: Configure package inputs + run: | + set -euo pipefail + chmod +x package-binaries/openshell{,-gateway} + ls -lah package-binaries + + - name: Mark workspace safe for git + run: git config --global --add safe.directory "$GITHUB_WORKSPACE" + + - name: Fetch tags + run: git fetch --tags --force + + - name: Build RPMs via Packit + env: + OPENSHELL_RPM_VERSION: ${{ inputs.rpm-version }} + OPENSHELL_RPM_RELEASE: ${{ inputs.rpm-release }} + OPENSHELL_CARGO_VERSION: ${{ inputs.cargo-version }} + OPENSHELL_PREBUILT_BINARIES_DIR: ${{ github.workspace }}/package-binaries + run: packit build locally + + - name: Collect RPM artifacts + run: | + set -euo pipefail + mkdir -p artifacts + mapfile -t rpms < <(find "$GITHUB_WORKSPACE" -maxdepth 3 -type f -name '*.rpm' ! -name '*.src.rpm' | sort) + if [ "${#rpms[@]}" -eq 0 ]; then + echo "::error::No RPM artifacts found under $GITHUB_WORKSPACE" + find "$GITHUB_WORKSPACE" -maxdepth 3 -type f | sort + exit 1 + fi + cp "${rpms[@]}" artifacts/ + echo "=== Built RPMs ===" + ls -lah artifacts/ + + - name: Upload RPM artifacts + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 + with: + name: rpm-linux-${{ inputs.arch }} + path: artifacts/*.rpm + retention-days: 5 diff --git a/.github/workflows/build-sandbox-binaries.yml b/.github/workflows/build-sandbox-binaries.yml index 53f81bec4..006d396cf 100644 --- a/.github/workflows/build-sandbox-binaries.yml +++ b/.github/workflows/build-sandbox-binaries.yml @@ -31,17 +31,14 @@ jobs: include: - triple: x86_64-unknown-linux-musl runner: linux-amd64-cpu8 - dev_shell: .#devShells.x86_64-linux.musl - triple: aarch64-unknown-linux-musl runner: linux-arm64-cpu8 - dev_shell: .#devShells.aarch64-linux.musl uses: ./.github/workflows/build-binaries.yml with: package: openshell-sandbox binary: openshell-sandbox triple: ${{ matrix.triple }} runner: ${{ matrix.runner }} - dev-shell: ${{ matrix.dev_shell }} cargo-version: ${{ inputs.cargo-version }} image-tag: ${{ inputs.image-tag }} checkout-ref: ${{ inputs.checkout-ref }} diff --git a/.github/workflows/build-vm-driver.yml b/.github/workflows/build-vm-driver.yml index 9df710cb8..1e303b842 100644 --- a/.github/workflows/build-vm-driver.yml +++ b/.github/workflows/build-vm-driver.yml @@ -33,22 +33,17 @@ jobs: - arch: x86_64 triple: x86_64-unknown-linux-gnu runner: linux-amd64-cpu8 - dev_shell: .#devShells.x86_64-linux.glibc-2-28 - interpreter: /lib64/ld-linux-x86-64.so.2 - arch: aarch64 triple: aarch64-unknown-linux-gnu runner: linux-arm64-cpu8 - dev_shell: .#devShells.aarch64-linux.glibc-2-28 - interpreter: /lib/ld-linux-aarch64.so.1 - arch: aarch64 triple: aarch64-apple-darwin runner: macos-15-xlarge - dev_shell: .#devShells.aarch64-darwin.default runs-on: ${{ matrix.runner }} timeout-minutes: 60 defaults: run: - shell: nix develop ${{ matrix.dev_shell }} -c bash -euo pipefail {0} + shell: nix develop -c bash -euo pipefail {0} steps: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: @@ -82,7 +77,5 @@ jobs: package: openshell-driver-vm binary: openshell-driver-vm triple: ${{ matrix.triple }} - dev-shell: ${{ matrix.dev_shell }} cargo-version: ${{ inputs['cargo-version'] }} image-tag: ${{ inputs['image-tag'] }} - interpreter: ${{ matrix.interpreter }} diff --git a/.github/workflows/codex-security.yml b/.github/workflows/codex-security.yml new file mode 100644 index 000000000..a008af019 --- /dev/null +++ b/.github/workflows/codex-security.yml @@ -0,0 +1,286 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +name: Codex Security Release Qualification + +on: + push: + tags: + - "v*.*.*-pre.*" + workflow_call: + inputs: + candidate_ref: + description: Pre-release tag to scan (vMAJOR.MINOR.PATCH-pre.N) + required: true + type: string + stable_ref: + description: Optional previous stable tag override + required: false + default: "" + type: string + allow_full_bootstrap: + description: Allow a full scan when no previous stable tag exists + required: false + default: false + type: boolean + outputs: + base_sha: + description: Previous stable commit, empty for a full bootstrap scan + value: ${{ jobs.analyze.outputs.base_sha }} + candidate_sha: + description: Qualified pre-release commit + value: ${{ jobs.analyze.outputs.candidate_sha }} + category: + description: Code Scanning category for the release train + value: ${{ jobs.analyze.outputs.category }} + train: + description: Stable version targeted by the pre-release train + value: ${{ jobs.analyze.outputs.train }} + secrets: + CODEX_SECURITY_API_KEY: + required: true + workflow_dispatch: + inputs: + candidate_ref: + description: Pre-release tag to scan (vMAJOR.MINOR.PATCH-pre.N) + required: true + type: string + stable_ref: + description: Optional previous stable tag override + required: false + type: string + upload_sarif: + description: Upload manual-run results to Code Scanning + required: false + default: false + type: boolean + allow_full_bootstrap: + description: Allow a full scan when no previous stable tag exists + required: false + default: false + type: boolean + +permissions: + actions: read + contents: read + security-events: write + +concurrency: + group: codex-security-release-qualification + cancel-in-progress: true + +env: + CODEX_SECURITY_MAX_CONCURRENT_THREADS: "8" + CODEX_SECURITY_REASONING_EFFORT: medium + NVIDIA_INFERENCE_BASE_URL: https://inference-api.nvidia.com/v1 + NVIDIA_INFERENCE_MODEL: openai/openai/gpt-5.6-sol + +jobs: + analyze: + name: Codex Security (${{ inputs.candidate_ref || github.ref_name }}) + # The agent executes no shell commands on the repository self-hosted runner, + # so its preflight never scopes the diff and it seals no draft. + runs-on: ubuntu-latest + timeout-minutes: 120 + outputs: + base_sha: ${{ steps.range.outputs.base_sha }} + candidate_sha: ${{ steps.range.outputs.candidate_sha }} + category: ${{ steps.range.outputs.category }} + train: ${{ steps.range.outputs.train }} + steps: + # Install the trusted scanner before repository-controlled files exist in + # the workspace, and invoke it later through its absolute path. + - name: Set up Node.js + uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0 + with: + node-version: "26" + package-manager-cache: false + + - name: Set up Python + uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 + with: + python-version: "3.14" + + # Codex confines model-run commands with bubblewrap, which needs + # unprivileged user namespaces. Ubuntu 24.04 restricts those through + # AppArmor, so bubblewrap cannot set up the sandbox network namespace and + # the scan agent executes nothing at all. + - name: Allow unprivileged user namespaces + run: | + set -euo pipefail + if [ -e /proc/sys/kernel/apparmor_restrict_unprivileged_userns ]; then + sudo sysctl -w kernel.apparmor_restrict_unprivileged_userns=0 + fi + + - name: Install Codex Security + run: | + set -euo pipefail + npm install \ + --prefix "$RUNNER_TEMP/codex-security" \ + --ignore-scripts \ + --no-audit \ + --no-fund \ + @openai/codex-security@0.1.24 + + - name: Verify Codex Security + env: + CODEX_SECURITY_BIN: ${{ runner.temp }}/codex-security/node_modules/.bin/codex-security + run: | + set -euo pipefail + test -x "$CODEX_SECURITY_BIN" + "$CODEX_SECURITY_BIN" --version + + # The range resolver has to come from the workflow's own revision. A + # scanned candidate predates it, and running the resolver from the + # revision under scan would let that revision pick its own scan range. + - name: Check out the workflow revision + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + path: workflow-revision + sparse-checkout: tasks/scripts + persist-credentials: false + + - name: Stage the range resolver + run: | + set -euo pipefail + install -d -m 700 "$RUNNER_TEMP/range-resolver" + cp workflow-revision/tasks/scripts/release.py \ + workflow-revision/tasks/scripts/codex_security_range.py \ + "$RUNNER_TEMP/range-resolver/" + rm -rf workflow-revision + + - name: Check out the pre-release + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + ref: ${{ inputs.candidate_ref || github.ref }} + fetch-depth: 0 + persist-credentials: false + + - name: Resolve release range + id: range + env: + ALLOW_FULL_BOOTSTRAP: ${{ inputs.allow_full_bootstrap || false }} + CANDIDATE_REF: ${{ inputs.candidate_ref || github.ref_name }} + STABLE_REF: ${{ inputs.stable_ref || '' }} + run: | + set -euo pipefail + git show-ref --verify --quiet refs/remotes/origin/main + + args=( + --candidate "$CANDIDATE_REF" + --main-ref origin/main + ) + if [ -n "$STABLE_REF" ]; then + args+=(--stable "$STABLE_REF") + fi + if [ "$ALLOW_FULL_BOOTSTRAP" = "true" ]; then + args+=(--allow-full-bootstrap) + fi + + python3 "$RUNNER_TEMP/range-resolver/codex_security_range.py" "${args[@]}" + + - name: Scan changes since the previous stable + env: + BASE_SHA: ${{ steps.range.outputs.base_sha }} + CODEX_SECURITY_BIN: ${{ runner.temp }}/codex-security/node_modules/.bin/codex-security + CODEX_SECURITY_STATE_DIR: ${{ runner.temp }}/codex-security-state-${{ github.run_id }}-${{ github.run_attempt }} + HEAD_SHA: ${{ steps.range.outputs.candidate_sha }} + NVIDIA_INFERENCE_API_KEY: ${{ secrets.CODEX_SECURITY_API_KEY }} + OPENAI_API_KEY: ${{ secrets.CODEX_SECURITY_API_KEY }} + SCAN_DIR: ${{ runner.temp }}/codex-security-results-${{ github.run_id }}-${{ github.run_attempt }} + SCAN_SCOPE: ${{ steps.range.outputs.scan_scope }} + run: | + set -euo pipefail + install -d -m 700 "$CODEX_SECURITY_STATE_DIR" "$SCAN_DIR" + + target_args=() + if [ "$SCAN_SCOPE" = "diff" ]; then + target_args=(--diff "$BASE_SHA" --head "$HEAD_SHA") + elif [ "$SCAN_SCOPE" != "full" ]; then + echo "::error::Unsupported Codex Security scan scope: $SCAN_SCOPE" + exit 2 + fi + + "$CODEX_SECURITY_BIN" scan . \ + "${target_args[@]}" \ + --auth api-key \ + --model "$NVIDIA_INFERENCE_MODEL" \ + --effort "$CODEX_SECURITY_REASONING_EFFORT" \ + --codex 'model_provider="nvidia"' \ + --codex 'model_providers.nvidia.name="NVIDIA Inference"' \ + --codex "model_providers.nvidia.base_url=\"$NVIDIA_INFERENCE_BASE_URL\"" \ + --codex 'model_providers.nvidia.env_key="NVIDIA_INFERENCE_API_KEY"' \ + --codex 'model_providers.nvidia.wire_api="responses"' \ + --codex 'model_providers.nvidia.supports_websockets=false' \ + --codex "features.multi_agent_v2.max_concurrent_threads_per_session=$CODEX_SECURITY_MAX_CONCURRENT_THREADS" \ + --codex 'approval_policy="never"' \ + --output-dir "$SCAN_DIR" \ + --headless > /dev/null + + - name: Export SARIF + env: + CODEX_SECURITY_BIN: ${{ runner.temp }}/codex-security/node_modules/.bin/codex-security + SARIF_FILE: ${{ runner.temp }}/codex-security.sarif + SCAN_DIR: ${{ runner.temp }}/codex-security-results-${{ github.run_id }}-${{ github.run_attempt }} + run: | + set -euo pipefail + "$CODEX_SECURITY_BIN" export "$SCAN_DIR" \ + --export-format sarif \ + --source-root "$GITHUB_WORKSPACE" \ + --output "$SARIF_FILE" + + - name: Summarize findings + env: + BASE_TAG: ${{ steps.range.outputs.base_tag }} + CANDIDATE_TAG: ${{ steps.range.outputs.candidate_tag }} + COMMIT_COUNT: ${{ steps.range.outputs.commit_count }} + SARIF_FILE: ${{ runner.temp }}/codex-security.sarif + SCAN_SCOPE: ${{ steps.range.outputs.scan_scope }} + TRAIN: ${{ steps.range.outputs.train }} + run: | + set -euo pipefail + finding_count=$(jq '[.runs[]?.results[]?] | length' "$SARIF_FILE") + { + echo "### Codex Security release qualification" + echo + echo "- Train: \`$TRAIN\`" + echo "- Candidate: \`$CANDIDATE_TAG\`" + echo "- Maximum concurrent agent threads: $CODEX_SECURITY_MAX_CONCURRENT_THREADS" + if [ "$SCAN_SCOPE" = "diff" ]; then + echo "- Previous stable: \`$BASE_TAG\`" + echo "- Commits in cumulative diff: $COMMIT_COUNT" + else + echo "- Scope: approved full bootstrap scan" + fi + echo "- Coverage: complete" + echo "- Findings: $finding_count" + echo + echo "Findings are informational during the observation phase." + } >> "$GITHUB_STEP_SUMMARY" + + - name: Upload SARIF to Code Scanning + if: ${{ github.event_name != 'workflow_dispatch' || inputs.upload_sarif }} + uses: github/codeql-action/upload-sarif@ff2f1c621b7f889edc0d3c761ac2e6a3f8cdb0dd # v4.37.7 + with: + sarif_file: ${{ runner.temp }}/codex-security.sarif + ref: refs/heads/main + sha: ${{ steps.range.outputs.candidate_sha }} + category: ${{ steps.range.outputs.category }} + + result: + name: OpenShell / Codex Security (informational) + if: ${{ always() }} + needs: analyze + runs-on: ubuntu-latest + permissions: {} + steps: + - name: Evaluate scanner execution + env: + ANALYZE_RESULT: ${{ needs.analyze.result }} + run: | + set -euo pipefail + if [ "$ANALYZE_RESULT" != "success" ]; then + echo "::error::Codex Security release qualification did not complete successfully." + exit 1 + fi + echo "Codex Security completed; findings remain informational." diff --git a/.github/workflows/conformance.yml b/.github/workflows/conformance.yml new file mode 100644 index 000000000..242f8c7d0 --- /dev/null +++ b/.github/workflows/conformance.yml @@ -0,0 +1,179 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +name: Conformance + +on: + workflow_dispatch: {} + +permissions: {} + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: true + +jobs: + pr_metadata: + name: Resolve PR metadata + runs-on: ubuntu-latest + permissions: + contents: read + pull-requests: read + outputs: + should_run: ${{ steps.gate.outputs.should_run }} + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + - id: gate + uses: ./.github/actions/pr-gate + + version: + needs: pr_metadata + if: needs.pr_metadata.outputs.should_run == 'true' + runs-on: ubuntu-latest + timeout-minutes: 5 + permissions: + contents: read + outputs: + cargo: ${{ steps.version.outputs.cargo }} + rpm_version: ${{ steps.version.outputs.rpm_version }} + rpm_release: ${{ steps.version.outputs.rpm_release }} + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + fetch-depth: 0 + + - name: Compute versions + id: version + run: | + cargo="$(python3 tasks/scripts/release.py get-version --cargo)" + rpm_version="$(python3 tasks/scripts/release.py get-version --rpm-version)" + rpm_release="$(python3 tasks/scripts/release.py get-version --rpm-release)" + { + echo "cargo=$cargo" + echo "rpm_version=$rpm_version" + echo "rpm_release=$rpm_release" + } >> "$GITHUB_OUTPUT" + + build-cli: + needs: version + permissions: + contents: read + uses: ./.github/workflows/build-binaries.yml + with: + package: openshell-cli + binary: openshell + triple: x86_64-unknown-linux-musl + runner: linux-amd64-cpu8 + cargo-version: ${{ needs.version.outputs.cargo }} + checkout-ref: ${{ github.sha }} + secrets: inherit + + build-conformance: + needs: version + permissions: + contents: read + uses: ./.github/workflows/build-binaries.yml + with: + package: openshell-conformance-cli + binary: openshell-conformance + triple: x86_64-unknown-linux-musl + runner: linux-amd64-cpu8 + cargo-version: ${{ needs.version.outputs.cargo }} + checkout-ref: ${{ github.sha }} + secrets: inherit + + build-gateway: + needs: version + permissions: + contents: read + uses: ./.github/workflows/build-binaries.yml + with: + package: openshell-gateway + binary: openshell-gateway + triple: x86_64-unknown-linux-gnu + runner: linux-amd64-cpu8 + cargo-version: ${{ needs.version.outputs.cargo }} + image-tag: dev + checkout-ref: ${{ github.sha }} + secrets: inherit + + build-rpm: + needs: [version, build-cli, build-gateway] + permissions: + contents: read + uses: ./.github/workflows/build-rpm.yml + with: + checkout-ref: ${{ github.sha }} + arch: x86_64 + runner: linux-amd64-cpu8 + cli-target: x86_64-unknown-linux-musl + gateway-target: x86_64-unknown-linux-gnu + cargo-version: ${{ needs.version.outputs.cargo }} + rpm-version: ${{ needs.version.outputs.rpm_version }} + rpm-release: ${{ needs.version.outputs.rpm_release }} + + fedora: + name: Fedora with Rootless Podman + needs: [build-conformance, build-rpm] + runs-on: ubuntu-24.04 + timeout-minutes: 45 + permissions: + actions: read + contents: read + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + + - name: Enable KVM access + run: | + set -euo pipefail + if [[ ! -c /dev/kvm ]]; then + echo "::error::The runner did not expose /dev/kvm" + exit 1 + fi + sudo chmod 0666 /dev/kvm + exec 3<>/dev/kvm + exec 3>&- + + - uses: ./.github/actions/setup-nix + + - name: Download RPM artifacts + uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 + with: + name: rpm-linux-x86_64 + path: rpm-input + + - name: Download conformance CLI + uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 + with: + name: openshell-conformance-x86_64-unknown-linux-musl + path: conformance-input + + - name: Run RPM gateway continuity conformance + shell: bash + run: | + set -euo pipefail + chmod +x conformance-input/openshell-conformance + shopt -s nullglob + candidate_cli_package=(rpm-input/openshell-[0-9]*.rpm) + candidate_gateway_package=(rpm-input/openshell-gateway-[0-9]*.rpm) + if [[ ${#candidate_cli_package[@]} -ne 1 || ${#candidate_gateway_package[@]} -ne 1 ]]; then + echo "expected one candidate CLI and gateway RPM" >&2 + printf 'RPM artifacts:\n' >&2 + printf ' %s\n' rpm-input/* >&2 + exit 1 + fi + + OPENSHELL_TEST_GUEST_CACHE_DISABLE=1 nix run .#test-guest -- \ + --distro fedora \ + --with podman-rootless \ + --with selinux \ + --copy "${candidate_cli_package[0]}:/var/lib/openshell-conformance/candidate/openshell.rpm" \ + --copy "${candidate_gateway_package[0]}:/var/lib/openshell-conformance/candidate/openshell-gateway.rpm" \ + --copy conformance-input/openshell-conformance:/tmp/openshell-conformance \ + --copy nix/test-guest/conformance-plans/gateway-upgrade-restart.toml:/tmp/conformance-plan.toml \ + --provision openshell-rpm-latest-release \ + --provision gateway-podman \ + --provision openshell-rpm-gateway-upgrade \ + -- /tmp/openshell-conformance run --plan /tmp/conformance-plan.toml diff --git a/.github/workflows/e2e-docker-test.yml b/.github/workflows/e2e-docker-test.yml index d70340230..0c6365617 100644 --- a/.github/workflows/e2e-docker-test.yml +++ b/.github/workflows/e2e-docker-test.yml @@ -25,6 +25,10 @@ on: required: false type: string default: "" + conformance-artifact-prefix: + required: false + type: string + default: "" suite-matrix: required: false type: string @@ -66,6 +70,8 @@ jobs: persist-credentials: false - uses: ./.github/actions/setup-e2e-cli + with: + conformance-artifact-prefix: ${{ inputs.conformance-artifact-prefix }} - uses: ./.github/actions/setup-e2e-gateway with: diff --git a/.github/workflows/e2e-gpu-test.yaml b/.github/workflows/e2e-gpu-test.yaml index 1a8275a27..b363a0509 100644 --- a/.github/workflows/e2e-gpu-test.yaml +++ b/.github/workflows/e2e-gpu-test.yaml @@ -7,6 +7,11 @@ on: description: "Image tag to test (typically the commit SHA)" required: true type: string + conformance-artifact-prefix: + description: "Optional prebuilt conformance CLI artifact prefix (artifact suffix is )" + required: false + type: string + default: "" permissions: actions: read @@ -61,6 +66,8 @@ jobs: - name: Use prebuilt OpenShell CLI uses: ./.github/actions/setup-e2e-cli + with: + conformance-artifact-prefix: ${{ inputs.conformance-artifact-prefix }} - name: Use prebuilt OpenShell gateway uses: ./.github/actions/setup-e2e-gateway diff --git a/.github/workflows/e2e-kubernetes-test.yml b/.github/workflows/e2e-kubernetes-test.yml index 4498f9d44..52ee27bc7 100644 --- a/.github/workflows/e2e-kubernetes-test.yml +++ b/.github/workflows/e2e-kubernetes-test.yml @@ -42,6 +42,16 @@ on: required: false type: string default: "e2e:kubernetes" + test-name: + description: "Rust e2e test target to run (sets OPENSHELL_E2E_KUBE_TEST)" + required: false + type: string + default: "" + kubernetes-features: + description: "Cargo feature list for the Kubernetes e2e crate (uses script defaults when empty)" + required: false + type: string + default: "" mise-version: description: "mise version to install on the bare Kubernetes e2e runner" required: false @@ -52,6 +62,11 @@ on: required: false type: string default: "" + conformance-artifact-prefix: + description: "Optional prebuilt conformance CLI artifact prefix (artifact suffix is )" + required: false + type: string + default: "" external-driver-binary: description: "Optional standalone compute driver binary used to compose a local test image" required: false @@ -92,6 +107,8 @@ jobs: - name: Use prebuilt OpenShell CLI uses: ./.github/actions/setup-e2e-cli + with: + conformance-artifact-prefix: ${{ inputs.conformance-artifact-prefix }} - name: Use prebuilt OpenShell gateway if: inputs.gateway-artifact != '' @@ -121,6 +138,8 @@ jobs: OPENSHELL_E2E_KUBE_CONTEXT: kind-${{ env.KIND_CLUSTER_NAME }} OPENSHELL_E2E_KUBE_EXTRA_VALUES: ${{ inputs.extra-helm-values }} OPENSHELL_E2E_KUBE_EXTERNAL_POSTGRES_SECRET: ${{ inputs.external-postgres-secret }} + OPENSHELL_E2E_KUBE_TEST: ${{ inputs.test-name }} + OPENSHELL_E2E_KUBERNETES_FEATURES: ${{ inputs.kubernetes-features }} IMAGE_TAG: ${{ inputs.image-tag }} OPENSHELL_REGISTRY: ghcr.io/nvidia/openshell E2E_TASK: ${{ inputs.e2e-task }} diff --git a/.github/workflows/e2e-podman-test.yml b/.github/workflows/e2e-podman-test.yml index 4a5a6d38d..3ec5294c4 100644 --- a/.github/workflows/e2e-podman-test.yml +++ b/.github/workflows/e2e-podman-test.yml @@ -21,6 +21,10 @@ on: required: false type: string default: "" + conformance-artifact-prefix: + required: false + type: string + default: "" suite-matrix: required: false type: string @@ -53,6 +57,8 @@ jobs: persist-credentials: false - uses: ./.github/actions/setup-e2e-cli + with: + conformance-artifact-prefix: ${{ inputs.conformance-artifact-prefix }} - uses: ./.github/actions/setup-e2e-gateway with: diff --git a/.github/workflows/e2e-vm-test.yml b/.github/workflows/e2e-vm-test.yml index 57e04c32b..954de9669 100644 --- a/.github/workflows/e2e-vm-test.yml +++ b/.github/workflows/e2e-vm-test.yml @@ -22,6 +22,10 @@ on: required: false type: string default: e2e:vm + conformance-artifact-prefix: + required: false + type: string + default: "" permissions: actions: read @@ -43,6 +47,8 @@ jobs: persist-credentials: false - uses: ./.github/actions/setup-e2e-cli + with: + conformance-artifact-prefix: ${{ inputs.conformance-artifact-prefix }} - uses: ./.github/actions/setup-e2e-gateway with: diff --git a/.github/workflows/publish-docs-website.yml b/.github/workflows/publish-docs-website.yml index dfb24b9ce..5c99492b6 100644 --- a/.github/workflows/publish-docs-website.yml +++ b/.github/workflows/publish-docs-website.yml @@ -25,6 +25,7 @@ permissions: concurrency: group: docs-website cancel-in-progress: false + queue: max defaults: run: diff --git a/.github/workflows/release-canary.yml b/.github/workflows/release-canary.yml index 937e774db..ee4074142 100644 --- a/.github/workflows/release-canary.yml +++ b/.github/workflows/release-canary.yml @@ -2,6 +2,11 @@ name: Release Canary on: workflow_dispatch: + inputs: + release-dev-run-id: + description: "Successful Release Dev run ID whose Snap artifact to test" + required: false + type: string workflow_run: workflows: ["Release Dev"] types: [completed] @@ -154,7 +159,7 @@ jobs: ubuntu-snap: name: Ubuntu Snap - if: ${{ github.event.workflow_run.conclusion == 'success' }} + if: ${{ github.event.workflow_run.conclusion == 'success' || inputs.release-dev-run-id != '' }} runs-on: ubuntu-latest timeout-minutes: 20 steps: @@ -176,7 +181,7 @@ jobs: uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 with: github-token: ${{ github.token }} - run-id: ${{ github.event.workflow_run.id }} + run-id: ${{ inputs.release-dev-run-id || github.event.workflow_run.id }} pattern: snap-linux-amd64 path: release/ merge-multiple: true @@ -202,7 +207,27 @@ jobs: sudo snap services openshell openshell gateway add http://127.0.0.1:17670 --local --name snap-docker openshell gateway select snap-docker - openshell status + for _ in $(seq 1 30); do + if openshell status; then + exit 0 + fi + sleep 1 + done + echo "Gateway did not become ready within 30 seconds" >&2 + exit 1 + + - name: Collect Snap diagnostics + if: failure() + run: | + set +e + sudo snap services openshell + sudo snap connections openshell + sudo snap changes + sudo systemctl status snap.openshell.gateway.service --no-pager + sudo journalctl -b -u snap.openshell.gateway.service --no-pager -n 300 + sudo journalctl -b -u snapd.service --no-pager -n 300 + sudo snap logs openshell.gateway -n=300 + sudo ss -ltnp '( sport = :17670 )' kubernetes: name: Kubernetes Helm (kind) diff --git a/.github/workflows/release-dev.yml b/.github/workflows/release-dev.yml index 7982a8164..fb364b4e1 100644 --- a/.github/workflows/release-dev.yml +++ b/.github/workflows/release-dev.yml @@ -28,6 +28,7 @@ jobs: password: ${{ secrets.GITHUB_TOKEN }} outputs: python_version: ${{ steps.v.outputs.python }} + docs_version: ${{ steps.v.outputs.docs_version }} cargo_version: ${{ steps.v.outputs.cargo }} deb_version: ${{ steps.v.outputs.deb }} rpm_version: ${{ steps.v.outputs.rpm_version }} @@ -47,11 +48,15 @@ jobs: id: v run: | set -euo pipefail - echo "python=$(uv run python tasks/scripts/release.py get-version --python)" >> "$GITHUB_OUTPUT" - echo "cargo=$(uv run python tasks/scripts/release.py get-version --cargo)" >> "$GITHUB_OUTPUT" - echo "deb=$(uv run python tasks/scripts/release.py get-version --deb)" >> "$GITHUB_OUTPUT" - echo "rpm_version=$(uv run python tasks/scripts/release.py get-version --rpm-version)" >> "$GITHUB_OUTPUT" - echo "rpm_release=$(uv run python tasks/scripts/release.py get-version --rpm-release)" >> "$GITHUB_OUTPUT" + python_version=$(uv run python tasks/scripts/release.py get-version --dev --python) + { + echo "python=${python_version}" + echo "docs_version=${python_version%%+*}" + echo "cargo=$(uv run python tasks/scripts/release.py get-version --dev --cargo)" + echo "deb=$(uv run python tasks/scripts/release.py get-version --dev --deb)" + echo "rpm_version=$(uv run python tasks/scripts/release.py get-version --dev --rpm-version)" + echo "rpm_release=$(uv run python tasks/scripts/release.py get-version --dev --rpm-release)" + } >> "$GITHUB_OUTPUT" build-cli: needs: compute-versions @@ -63,6 +68,27 @@ jobs: image-tag: dev secrets: inherit + build-conformance: + needs: compute-versions + permissions: + contents: read + strategy: + matrix: + include: + - triple: x86_64-unknown-linux-musl + runner: linux-amd64-cpu8 + - triple: aarch64-unknown-linux-musl + runner: linux-arm64-cpu8 + uses: ./.github/workflows/build-binaries.yml + with: + package: openshell-conformance-cli + binary: openshell-conformance + artifact-name: openshell-conformance-${{ matrix.triple }} + triple: ${{ matrix.triple }} + runner: ${{ matrix.runner }} + cargo-version: ${{ needs.compute-versions.outputs.cargo_version }} + secrets: inherit + build-gateway: needs: compute-versions permissions: @@ -125,7 +151,7 @@ jobs: secrets: inherit docker-e2e: - needs: [build-cli, build-gateway, build-supervisor-image] + needs: [build-cli, build-conformance, build-gateway, build-supervisor-image] permissions: actions: read contents: read @@ -134,9 +160,10 @@ jobs: with: image-tag: ${{ github.sha }} runner: linux-arm64-cpu8 + conformance-artifact-prefix: openshell-conformance podman-e2e: - needs: [build-cli, build-gateway, build-supervisor-image] + needs: [build-cli, build-conformance, build-gateway, build-supervisor-image] permissions: actions: read contents: read @@ -144,14 +171,17 @@ jobs: uses: ./.github/workflows/e2e-podman-test.yml with: image-tag: ${{ github.sha }} + conformance-artifact-prefix: openshell-conformance vm-e2e: - needs: [build-cli, build-gateway, build-vm-driver] + needs: [build-cli, build-conformance, build-gateway, build-vm-driver] permissions: actions: read contents: read packages: read uses: ./.github/workflows/e2e-vm-test.yml + with: + conformance-artifact-prefix: openshell-conformance tag-ghcr-dev: name: Tag GHCR Images as Dev @@ -600,6 +630,22 @@ jobs: release-kind: dev pin-sha: ${{ github.sha }} + publish-fern-docs: + name: Sync and Publish Fern Docs + needs: [compute-versions, release-dev, release-helm, trigger-wheel-publish] + permissions: + contents: write + uses: ./.github/workflows/sync-docs.yml + with: + operation: sync + channel: dev + source_ref: ${{ github.sha }} + release_version: ${{ needs.compute-versions.outputs.docs_version }} + display_name: Dev + availability: beta + publish: true + secrets: inherit + trigger-wheel-publish: name: Trigger Wheel Publish needs: [compute-versions, release-dev] diff --git a/.github/workflows/release-tag.yml b/.github/workflows/release-tag.yml index 92eb4adf1..962d84640 100644 --- a/.github/workflows/release-tag.yml +++ b/.github/workflows/release-tag.yml @@ -48,6 +48,7 @@ jobs: semver: ${{ steps.v.outputs.semver }} # Commit resolved from RELEASE_TAG, used for image tags and downstream metadata source_sha: ${{ steps.v.outputs.source_sha }} + is_prerelease: ${{ steps.v.outputs.is_prerelease }} steps: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: @@ -64,14 +65,25 @@ jobs: id: v run: | set -euo pipefail - echo "python=$(uv run python tasks/scripts/release.py get-version --python)" >> "$GITHUB_OUTPUT" - echo "cargo=$(uv run python tasks/scripts/release.py get-version --cargo)" >> "$GITHUB_OUTPUT" - echo "npm=$(uv run python tasks/scripts/release.py get-version --npm)" >> "$GITHUB_OUTPUT" - echo "deb=$(uv run python tasks/scripts/release.py get-version --deb)" >> "$GITHUB_OUTPUT" - echo "rpm_version=$(uv run python tasks/scripts/release.py get-version --rpm-version)" >> "$GITHUB_OUTPUT" - echo "rpm_release=$(uv run python tasks/scripts/release.py get-version --rpm-release)" >> "$GITHUB_OUTPUT" - echo "semver=${RELEASE_TAG#v}" >> "$GITHUB_OUTPUT" - echo "source_sha=$(git rev-parse HEAD)" >> "$GITHUB_OUTPUT" + if [[ "${RELEASE_TAG}" =~ ^v[0-9]+\.[0-9]+\.[0-9]+-pre\.[1-9][0-9]*$ ]]; then + is_prerelease=true + elif [[ "${RELEASE_TAG}" =~ ^v[0-9]+\.[0-9]+\.[0-9]+$ ]]; then + is_prerelease=false + else + echo "Unsupported release tag: ${RELEASE_TAG}" >&2 + exit 1 + fi + { + echo "is_prerelease=${is_prerelease}" + echo "python=$(uv run python tasks/scripts/release.py get-version --python)" + echo "cargo=$(uv run python tasks/scripts/release.py get-version --cargo)" + echo "npm=$(uv run python tasks/scripts/release.py get-version --npm)" + echo "deb=$(uv run python tasks/scripts/release.py get-version --deb)" + echo "rpm_version=$(uv run python tasks/scripts/release.py get-version --rpm-version)" + echo "rpm_release=$(uv run python tasks/scripts/release.py get-version --rpm-release)" + echo "semver=${RELEASE_TAG#v}" + echo "source_sha=$(git rev-parse HEAD)" + } >> "$GITHUB_OUTPUT" build-cli: needs: compute-versions @@ -84,6 +96,28 @@ jobs: checkout-ref: ${{ inputs.tag || github.ref }} secrets: inherit + build-conformance: + needs: compute-versions + permissions: + contents: read + strategy: + matrix: + include: + - triple: x86_64-unknown-linux-musl + runner: linux-amd64-cpu8 + - triple: aarch64-unknown-linux-musl + runner: linux-arm64-cpu8 + uses: ./.github/workflows/build-binaries.yml + with: + package: openshell-conformance-cli + binary: openshell-conformance + artifact-name: openshell-conformance-${{ matrix.triple }} + triple: ${{ matrix.triple }} + runner: ${{ matrix.runner }} + cargo-version: ${{ needs.compute-versions.outputs.cargo_version }} + checkout-ref: ${{ inputs.tag || github.ref }} + secrets: inherit + build-gateway: needs: compute-versions permissions: @@ -153,7 +187,7 @@ jobs: secrets: inherit docker-e2e: - needs: [compute-versions, build-cli, build-gateway, build-supervisor-image] + needs: [compute-versions, build-cli, build-conformance, build-gateway, build-supervisor-image] permissions: actions: read contents: read @@ -163,9 +197,10 @@ jobs: image-tag: ${{ needs.compute-versions.outputs.source_sha }} checkout-ref: ${{ inputs.tag || github.ref }} runner: linux-arm64-cpu8 + conformance-artifact-prefix: openshell-conformance podman-e2e: - needs: [compute-versions, build-cli, build-gateway, build-supervisor-image] + needs: [compute-versions, build-cli, build-conformance, build-gateway, build-supervisor-image] permissions: actions: read contents: read @@ -174,9 +209,10 @@ jobs: with: image-tag: ${{ needs.compute-versions.outputs.source_sha }} checkout-ref: ${{ inputs.tag || github.ref }} + conformance-artifact-prefix: openshell-conformance vm-e2e: - needs: [compute-versions, build-cli, build-gateway, build-vm-driver] + needs: [compute-versions, build-cli, build-conformance, build-gateway, build-vm-driver] permissions: actions: read contents: read @@ -184,9 +220,10 @@ jobs: uses: ./.github/workflows/e2e-vm-test.yml with: checkout-ref: ${{ inputs.tag || github.ref }} + conformance-artifact-prefix: openshell-conformance tag-ghcr-release: - name: Tag GHCR Images for Release + name: Tag GHCR Images needs: [compute-versions, build-gateway-image, build-supervisor-image, release] runs-on: linux-amd64-cpu8 timeout-minutes: 10 @@ -194,22 +231,27 @@ jobs: - name: Log in to GHCR run: echo "${{ secrets.GITHUB_TOKEN }}" | docker login ghcr.io -u "${GITHUB_ACTOR}" --password-stdin - - name: Tag images with version and latest + - name: Tag images with release version + env: + IS_PRERELEASE: ${{ needs.compute-versions.outputs.is_prerelease }} run: | set -euo pipefail REGISTRY="ghcr.io/nvidia/openshell" VERSION="${{ needs.compute-versions.outputs.semver }}" SOURCE_TAG="${{ needs.compute-versions.outputs.source_sha }}" for component in gateway supervisor; do - echo "Tagging ${REGISTRY}/${component}:${SOURCE_TAG} as ${VERSION} and latest..." + echo "Tagging ${REGISTRY}/${component}:${SOURCE_TAG} as ${VERSION}..." docker buildx imagetools create \ --prefer-index=false \ -t "${REGISTRY}/${component}:${VERSION}" \ "${REGISTRY}/${component}:${SOURCE_TAG}" - docker buildx imagetools create \ - --prefer-index=false \ - -t "${REGISTRY}/${component}:latest" \ - "${REGISTRY}/${component}:${SOURCE_TAG}" + if [[ "${IS_PRERELEASE}" != "true" ]]; then + echo "Tagging ${REGISTRY}/${component}:${SOURCE_TAG} as latest..." + docker buildx imagetools create \ + --prefer-index=false \ + -t "${REGISTRY}/${component}:latest" \ + "${REGISTRY}/${component}:${SOURCE_TAG}" + fi done build-python-wheel: @@ -265,8 +307,9 @@ jobs: uses: ./.github/workflows/snap-package.yml with: checkout-ref: ${{ inputs.tag || github.ref }} - upload-channel: latest/stable - github-environment: latest/stable + upload-channel: ${{ needs.compute-versions.outputs.is_prerelease == 'true' && 'latest/edge' || 'latest/stable' }} + github-environment: ${{ needs.compute-versions.outputs.is_prerelease == 'true' && 'latest/edge' || 'latest/stable' }} + publish: ${{ needs.compute-versions.outputs.is_prerelease != 'true' }} secrets: publish-credentials: ${{ secrets.SNAPCRAFT_STORE_CREDENTIALS }} @@ -529,6 +572,7 @@ jobs: cat openshell-sandbox-checksums-sha256.txt - name: Generate Homebrew formula + if: needs.compute-versions.outputs.is_prerelease != 'true' run: | set -euo pipefail python3 tasks/scripts/release.py generate-homebrew-formula \ @@ -547,6 +591,7 @@ jobs: release/*.whl - name: Prune removed VM checksum asset + if: needs.compute-versions.outputs.is_prerelease != 'true' uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9 with: script: | @@ -569,6 +614,7 @@ jobs: } - name: Create GitHub Release + if: needs.compute-versions.outputs.is_prerelease != 'true' uses: softprops/action-gh-release@3d0d9888cb7fd7b750713d6e236d1fcb99157228 # v3.0.2 with: name: OpenShell ${{ env.RELEASE_TAG }} @@ -605,35 +651,35 @@ jobs: release/openshell-gateway-checksums-sha256.txt release/openshell-sandbox-checksums-sha256.txt + - name: Upload pre-release artifacts + if: needs.compute-versions.outputs.is_prerelease == 'true' + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 + with: + name: openshell-${{ env.RELEASE_TAG }} + path: release/ + retention-days: 90 + if-no-files-found: error + publish-fern-docs: - name: Publish Fern Docs - needs: [release] - runs-on: ubuntu-latest - timeout-minutes: 15 - steps: - - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - with: - ref: ${{ inputs.tag || github.ref }} - - - name: Setup Node.js - uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0 - with: - node-version: "24" - - - name: Install Fern CLI - run: | - FERN_VERSION=$(node -p "require('./fern/fern.config.json').version") - npm install -g "fern-api@${FERN_VERSION}" - - - name: Publish Fern docs - env: - FERN_TOKEN: ${{ secrets.FERN_TOKEN }} - working-directory: ./fern - run: fern generate --docs + name: Sync and Publish Fern Docs + needs: [compute-versions, release, publish-sdk-typescript, release-helm, trigger-wheel-publish] + if: needs.compute-versions.outputs.is_prerelease != 'true' + permissions: + contents: write + uses: ./.github/workflows/sync-docs.yml + with: + operation: sync + channel: latest + source_ref: ${{ needs.compute-versions.outputs.source_sha }} + release_version: ${{ needs.compute-versions.outputs.semver }} + display_name: Latest (v${{ needs.compute-versions.outputs.semver }}) + publish: true + secrets: inherit publish-sdk-typescript: name: Publish TypeScript SDK needs: [compute-versions, release] + if: needs.compute-versions.outputs.is_prerelease != 'true' runs-on: linux-amd64-cpu8 timeout-minutes: 15 permissions: @@ -690,6 +736,7 @@ jobs: trigger-wheel-publish: name: Trigger Wheel Publish needs: [compute-versions, release] + if: needs.compute-versions.outputs.is_prerelease != 'true' runs-on: [self-hosted, nv] timeout-minutes: 10 steps: diff --git a/.github/workflows/rpm-package.yml b/.github/workflows/rpm-package.yml index 870164458..5cda9a88c 100644 --- a/.github/workflows/rpm-package.yml +++ b/.github/workflows/rpm-package.yml @@ -37,83 +37,20 @@ jobs: matrix: include: - arch: x86_64 - artifact_arch: amd64 runner: linux-amd64-cpu8 cli_target: x86_64-unknown-linux-musl - gnu_target: x86_64-unknown-linux-gnu + gateway_target: x86_64-unknown-linux-gnu - arch: aarch64 - artifact_arch: arm64 runner: linux-arm64-cpu8 cli_target: aarch64-unknown-linux-musl - gnu_target: aarch64-unknown-linux-gnu - runs-on: ${{ matrix.runner }} - timeout-minutes: 60 - container: - image: fedora:latest - steps: - - name: Install build dependencies - run: | - dnf install -y \ - packit rpm-build \ - rust cargo gcc gcc-c++ make cmake pkg-config \ - clang-devel z3-devel systemd-rpm-macros \ - pandoc python3-devel git-core \ - cargo-rpm-macros - - - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - with: - ref: ${{ inputs.checkout-ref }} - fetch-depth: 0 - - - name: Download CLI artifact - uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 - with: - name: openshell-${{ matrix.cli_target }} - path: package-binaries/ - - - name: Download gateway artifact - uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 - with: - name: openshell-gateway-${{ matrix.gnu_target }} - path: package-binaries/ - - - name: Configure package inputs - run: | - set -euo pipefail - chmod +x package-binaries/openshell{,-gateway} - ls -lah package-binaries - - - name: Mark workspace safe for git - run: git config --global --add safe.directory "$GITHUB_WORKSPACE" - - - name: Fetch tags - run: git fetch --tags --force - - - name: Build RPMs via Packit - env: - OPENSHELL_RPM_VERSION: ${{ inputs['rpm-version'] }} - OPENSHELL_RPM_RELEASE: ${{ inputs['rpm-release'] }} - OPENSHELL_CARGO_VERSION: ${{ inputs['cargo-version'] }} - OPENSHELL_PREBUILT_BINARIES_DIR: ${{ github.workspace }}/package-binaries - run: packit build locally - - - name: Collect RPM artifacts - run: | - set -euo pipefail - mkdir -p artifacts - mapfile -t rpms < <(find "$GITHUB_WORKSPACE" -maxdepth 3 -type f -name '*.rpm' ! -name '*.src.rpm' | sort) - if [ "${#rpms[@]}" -eq 0 ]; then - echo "::error::No RPM artifacts found under $GITHUB_WORKSPACE" - find "$GITHUB_WORKSPACE" -maxdepth 3 -type f | sort - exit 1 - fi - cp "${rpms[@]}" artifacts/ - echo "=== Built RPMs ===" - ls -lah artifacts/ - - - name: Upload RPM artifacts - uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 - with: - name: rpm-linux-${{ matrix.arch }} - path: artifacts/*.rpm - retention-days: 5 + gateway_target: aarch64-unknown-linux-gnu + uses: ./.github/workflows/build-rpm.yml + with: + checkout-ref: ${{ inputs.checkout-ref }} + arch: ${{ matrix.arch }} + runner: ${{ matrix.runner }} + cli-target: ${{ matrix.cli_target }} + gateway-target: ${{ matrix.gateway_target }} + rpm-version: ${{ inputs.rpm-version }} + rpm-release: ${{ inputs.rpm-release }} + cargo-version: ${{ inputs.cargo-version }} diff --git a/.github/workflows/rust-cache-seed.yml b/.github/workflows/rust-cache-seed.yml deleted file mode 100644 index ed6d1028a..000000000 --- a/.github/workflows/rust-cache-seed.yml +++ /dev/null @@ -1,72 +0,0 @@ -name: Rust Cache Seed - -on: - push: - branches: - - main - workflow_dispatch: - -env: - CARGO_TERM_COLOR: always - CARGO_INCREMENTAL: "0" - MISE_GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} - SCCACHE_GHA_ENABLED: "true" - -permissions: - contents: read - packages: read - -concurrency: - group: ${{ github.workflow }}-${{ github.ref }} - cancel-in-progress: true - -jobs: - rust: - name: Rust (${{ matrix.runner }}) - strategy: - fail-fast: false - matrix: - runner: [linux-amd64-cpu8, linux-arm64-cpu8] - runs-on: ${{ matrix.runner }} - env: - SCCACHE_GHA_VERSION: branch-checks-rust-${{ matrix.runner }} - container: - image: ghcr.io/nvidia/openshell/ci:latest - credentials: - username: ${{ github.actor }} - password: ${{ secrets.GITHUB_TOKEN }} - steps: - - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - - - name: Install tools - run: mise install --locked - - - name: Configure GHA sccache backend - uses: mozilla-actions/sccache-action@9e7fa8a12102821edf02ca5dbea1acd0f89a2696 # v0.0.10 - - - name: Cache Rust target and registry - uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6 # v2 - with: - shared-key: rust-checks-${{ matrix.runner }} - cache-on-failure: true - - - name: Format - run: mise run rust:format:check - - - name: Lint - run: mise run rust:lint - - - name: Test - run: mise run test:rust - - - name: sccache stats - if: always() - run: | - set +e - stats_bin="${SCCACHE_PATH:-sccache}" - "$stats_bin" --show-stats - status=$? - if [ "$status" -ne 0 ]; then - echo "::warning::sccache stats unavailable (exit $status)" - fi - exit 0 diff --git a/.github/workflows/snap-package.yml b/.github/workflows/snap-package.yml index 6de2a587f..7eb78d3a2 100644 --- a/.github/workflows/snap-package.yml +++ b/.github/workflows/snap-package.yml @@ -17,6 +17,11 @@ on: required: true type: string description: "GitHub deployment environment for approval gates (e.g., latest/edge, latest/stable)" + publish: + required: false + type: boolean + default: true + description: "Whether to upload the built snap to the Snap Store" secrets: publish-credentials: @@ -161,6 +166,7 @@ jobs: retention-days: 5 - name: Upload snap to Snap Store + if: inputs.publish env: SNAPCRAFT_STORE_CREDENTIALS: ${{ secrets.publish-credentials }} INPUTS_UPLOAD_CHANNEL: ${{ inputs.upload-channel }} diff --git a/.github/workflows/sync-docs.yml b/.github/workflows/sync-docs.yml index db2f29f73..63c4a6b8b 100644 --- a/.github/workflows/sync-docs.yml +++ b/.github/workflows/sync-docs.yml @@ -4,6 +4,49 @@ name: Sync Docs Website on: + workflow_call: + inputs: + operation: + description: "Whether to sync or remove a docs snapshot" + required: true + type: string + channel: + description: "Docs channel to update or remove" + required: true + type: string + source_ref: + description: "Source commit SHA, branch, or tag to snapshot when operation=sync" + required: false + type: string + release_version: + description: "Release version used to order mutable channels" + required: false + type: string + version_slug: + description: "Version slug when channel=version" + required: false + type: string + display_name: + description: "Optional version selector display name" + required: false + type: string + availability: + description: "Optional Fern availability status" + required: false + type: string + publish: + description: "Publish production docs after syncing" + required: false + default: false + type: boolean + allow_rollback: + description: "Allow an explicitly requested mutable-channel rollback" + required: false + default: false + type: boolean + secrets: + FERN_TOKEN: + required: false workflow_dispatch: inputs: operation: @@ -21,19 +64,38 @@ on: options: - dev - latest + - stable - version source_ref: description: "Source commit SHA, branch, or tag to snapshot when operation=sync" required: false type: string + release_version: + description: "Release version, e.g. 0.1.2 or 0.1.3.dev4" + required: false + type: string version_slug: description: "Version slug when channel=version, e.g. v0.0.36" required: false type: string display_name: - description: "Optional version selector display name" + description: "Optional selector name, e.g. Dev" required: false type: string + availability: + description: "Optional Fern status: beta, deprecated, ga, or stable" + required: false + type: string + publish: + description: "Publish production docs after syncing" + required: false + default: false + type: boolean + allow_rollback: + description: "Allow an explicitly requested mutable-channel rollback" + required: false + default: false + type: boolean permissions: contents: write @@ -41,6 +103,7 @@ permissions: concurrency: group: docs-website cancel-in-progress: false + queue: max defaults: run: @@ -65,6 +128,7 @@ jobs: OPERATION: ${{ inputs.operation }} CHANNEL: ${{ inputs.channel }} SOURCE_REF: ${{ inputs.source_ref }} + RELEASE_VERSION: ${{ inputs.release_version }} VERSION_SLUG: ${{ inputs.version_slug }} run: | set -euo pipefail @@ -72,8 +136,12 @@ jobs: echo "source_ref is required when operation=sync" >&2 exit 1 fi - if [[ "$CHANNEL" == "version" && -z "$VERSION_SLUG" ]]; then - echo "version_slug is required when channel=version" >&2 + if [[ "$CHANNEL" =~ ^(dev|latest|stable)$ && -z "$RELEASE_VERSION" ]]; then + echo "release_version is required for dev, latest, and stable channels" >&2 + exit 1 + fi + if [[ "$CHANNEL" =~ ^(stable|version)$ && -z "$VERSION_SLUG" ]]; then + echo "version_slug is required for stable and version channels" >&2 exit 1 fi @@ -96,6 +164,7 @@ jobs: uses: astral-sh/setup-uv@ae62891fec2bb8e7d6c99fc78c9fec3a63790f8d # v10.0.0 with: version: "0.10.12" + python-version: "3.13" - name: Update docs snapshot # Inputs flow in as quoted env vars to avoid shell injection; see the @@ -104,17 +173,32 @@ jobs: OPERATION: ${{ inputs.operation }} CHANNEL: ${{ inputs.channel }} SOURCE_REF: ${{ inputs.source_ref }} + RELEASE_VERSION: ${{ inputs.release_version }} VERSION_SLUG: ${{ inputs.version_slug }} DISPLAY_NAME: ${{ inputs.display_name }} + AVAILABILITY: ${{ inputs.availability }} + ALLOW_ROLLBACK: ${{ inputs.allow_rollback }} run: | + SOURCE_SHA="" + if [[ "$OPERATION" == "sync" ]]; then + SOURCE_SHA=$(git -C source rev-parse HEAD) + fi + rollback_args=() + if [[ "$ALLOW_ROLLBACK" == "true" ]]; then + rollback_args+=(--allow-rollback) + fi uv run automation/tasks/scripts/sync_docs_website.py \ --operation "$OPERATION" \ --source-root source \ --docs-website-root docs-website \ --channel "$CHANNEL" \ --source-ref "$SOURCE_REF" \ + --source-sha "$SOURCE_SHA" \ + --release-version "$RELEASE_VERSION" \ --version-slug "$VERSION_SLUG" \ - --display-name "$DISPLAY_NAME" + --display-name "$DISPLAY_NAME" \ + --availability "$AVAILABILITY" \ + "${rollback_args[@]}" - name: Setup Node.js uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0 @@ -160,3 +244,10 @@ jobs: git commit -m "docs(website): remove ${target} docs" fi git push origin HEAD:docs-website + + - name: Publish Fern docs + if: ${{ inputs.publish }} + env: + FERN_TOKEN: ${{ secrets.FERN_TOKEN }} + working-directory: docs-website/fern + run: fern generate --docs diff --git a/.github/workflows/trivy-changes.yml b/.github/workflows/trivy-changes.yml new file mode 100644 index 000000000..612ce891c --- /dev/null +++ b/.github/workflows/trivy-changes.yml @@ -0,0 +1,171 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +name: Trivy Changes + +on: + pull_request: + merge_group: + types: [checks_requested] + workflow_dispatch: + inputs: + base_sha: + description: Base commit SHA to compare + required: true + type: string + head_sha: + description: Candidate commit SHA to compare + required: true + type: string + +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: true + +jobs: + changes: + name: Detect deployment configuration changes + runs-on: ubuntu-latest + permissions: + contents: read + pull-requests: read + outputs: + should_run: ${{ steps.default.outputs.should_run || steps.changed.outputs.any_modified }} + steps: + - id: default + if: github.event_name != 'pull_request' + run: echo "should_run=true" >> "$GITHUB_OUTPUT" + + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + if: github.event_name == 'pull_request' + with: + persist-credentials: false + + - id: changed + if: github.event_name == 'pull_request' + uses: tj-actions/changed-files@aa08304bd477b800d468db44fe10f6c61f7f7b11 # v42.1.0 + with: + # `any_modified` covers deletions, which `any_changed` omits, and a + # failed diff has to fail the job: both otherwise report no relevant + # change, and removing the scanner or a value fixture would skip the + # scan behind a green status. + fail_on_initial_diff_error: true + files: | + deploy/docker/** + deploy/helm/** + deploy/kube/** + .trivyignore.yaml + flake.nix + flake.lock + tasks/scripts/trivy-scan.sh + tasks/scripts/trivy-scan-test.sh + tasks/scripts/trivy-config-report.jq + .github/workflows/trivy-scan.yml + .github/workflows/trivy-changes.yml + + scan: + name: Scan changed deployment configuration + needs: changes + if: needs.changes.outputs.should_run == 'true' + runs-on: ubuntu-latest + timeout-minutes: 30 + env: + BASE_REF: ${{ github.event.pull_request.base.sha || github.event.merge_group.base_sha || inputs.base_sha }} + HEAD_REF: ${{ inputs.head_sha || github.sha }} + defaults: + run: + shell: nix develop --command bash -euo pipefail {0} + steps: + - name: Check out candidate + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + ref: ${{ env.HEAD_REF }} + persist-credentials: false + + - name: Check out baseline + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + ref: ${{ env.BASE_REF }} + path: .trivy-base + persist-credentials: false + + - name: Set up Nix + uses: ./.github/actions/setup-nix + + - name: Test report comparison + run: tasks/scripts/trivy-scan-test.sh + + - name: Validate candidate ignore policy + run: tasks/scripts/trivy-scan.sh validate-ignore + + # Ignore-policy changes take effect only after merge. Applying the + # baseline policy to both scans prevents a candidate from exempting a new + # finding in the same change that introduces it. + - name: Prepare baseline ignore policy + run: | + if [ -f .trivy-base/.trivyignore.yaml ]; then + cp .trivy-base/.trivyignore.yaml "$RUNNER_TEMP/trivy-baseline-ignore.yaml" + else + printf 'misconfigurations: []\n' >"$RUNNER_TEMP/trivy-baseline-ignore.yaml" + fi + + - name: Scan baseline + env: + TRIVY_SOURCE_ROOT: ${{ github.workspace }}/.trivy-base + TRIVY_IGNORE_FILE: ${{ runner.temp }}/trivy-baseline-ignore.yaml + TRIVY_REPORT_DIR: ${{ runner.temp }}/trivy-base + run: | + mkdir -p "$TRIVY_REPORT_DIR" + "$GITHUB_WORKSPACE/tasks/scripts/trivy-scan.sh" config + + - name: Scan candidate + env: + TRIVY_IGNORE_FILE: ${{ runner.temp }}/trivy-baseline-ignore.yaml + TRIVY_REPORT_DIR: ${{ runner.temp }}/trivy-head + run: | + mkdir -p "$TRIVY_REPORT_DIR" + tasks/scripts/trivy-scan.sh config + + - name: Reject new high or critical findings + run: | + tasks/scripts/trivy-scan.sh gate-config-diff \ + "$RUNNER_TEMP/trivy-base" "$RUNNER_TEMP/trivy-head" + + - name: Upload reports + if: always() + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 + with: + name: trivy-changes-${{ github.run_id }} + path: | + ${{ runner.temp }}/trivy-base + ${{ runner.temp }}/trivy-head + if-no-files-found: ignore + retention-days: 14 + + result: + name: OpenShell / Trivy Changes + needs: [changes, scan] + if: always() + runs-on: ubuntu-latest + steps: + - name: Check scan result + env: + CHANGES_RESULT: ${{ needs.changes.result }} + SHOULD_RUN: ${{ needs.changes.outputs.should_run }} + SCAN_RESULT: ${{ needs.scan.result }} + run: | + set -euo pipefail + if [ "$CHANGES_RESULT" != "success" ]; then + echo "::error::Change detection concluded $CHANGES_RESULT." + exit 1 + fi + if [ "$SHOULD_RUN" = "true" ] && [ "$SCAN_RESULT" != "success" ]; then + echo "::error::Trivy scan concluded $SCAN_RESULT." + exit 1 + fi + if [ "$SHOULD_RUN" != "true" ]; then + echo "No Helm or Dockerfile changes to scan." + fi diff --git a/.github/workflows/trivy-scan.yml b/.github/workflows/trivy-scan.yml new file mode 100644 index 000000000..ccfe80bd6 --- /dev/null +++ b/.github/workflows/trivy-scan.yml @@ -0,0 +1,219 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +name: Trivy Scan + +# Manual or reusable scan of deployment configuration and supplied OCI +# artifacts. Findings are informational by default; scanner errors remain fatal. + +on: + workflow_call: + inputs: + images: + description: Newline-separated image references to scan + type: string + default: "" + charts: + description: Newline-separated packaged Helm chart OCI references + type: string + default: "" + severity: + description: Severities that fail the workflow + type: string + default: HIGH,CRITICAL + ignore-unfixed: + description: Ignore image vulnerabilities with no upstream fix + type: boolean + default: true + fail-on-findings: + description: Fail the run on findings instead of warning + type: boolean + default: false + upload-sarif: + description: Upload results to GitHub Code Scanning + type: boolean + default: true + secrets: + CACHIX_AUTH_TOKEN: + description: Token used to write Nix build outputs to Cachix + + workflow_dispatch: + inputs: + images: + description: Newline-separated image references to scan + type: string + default: "" + charts: + description: Newline-separated packaged Helm chart OCI references to scan + type: string + default: "" + severity: + description: Severities that fail the workflow + type: string + default: HIGH,CRITICAL + ignore-unfixed: + description: Ignore image vulnerabilities with no upstream fix + type: boolean + default: true + fail-on-findings: + description: Fail the run on findings instead of warning + type: boolean + default: false + upload-sarif: + description: Upload results to GitHub Code Scanning + type: boolean + default: true + +permissions: + contents: read + +defaults: + run: + shell: nix develop --command bash -euo pipefail {0} + +env: + TRIVY_SEVERITY: ${{ inputs.severity || 'HIGH,CRITICAL' }} + TRIVY_REPORT_DIR: reports/trivy + +jobs: + scan: + name: OpenShell / Trivy (informational) + runs-on: ubuntu-latest + timeout-minutes: 60 + outputs: + sarif-batches: ${{ steps.sarif.outputs.batches }} + artifact-id: ${{ steps.reports.outputs.artifact-id }} + permissions: + contents: read + packages: read + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + + - uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4.6.0 + with: + registry: ghcr.io + username: ${{ github.actor }} + password: ${{ github.token }} + + - name: Set up Nix + uses: ./.github/actions/setup-nix + with: + cachix-auth-token: ${{ github.event_name != 'pull_request' && secrets.CACHIX_AUTH_TOKEN || '' }} + + - name: Test scan reporting + run: tasks/scripts/trivy-scan-test.sh + + - name: Scan configuration + id: config + env: + CHARTS: ${{ inputs.charts }} + run: | + args=() + while IFS= read -r ref; do + [ -n "$ref" ] || continue + args+=(--chart-ref "$ref") + done <<<"$CHARTS" + tasks/scripts/trivy-scan.sh config "${args[@]}" + + - name: Scan images + id: images + if: ${{ !cancelled() }} + env: + IMAGES: ${{ inputs.images }} + TRIVY_IGNORE_UNFIXED: ${{ inputs.ignore-unfixed }} + run: | + refs=() + while IFS= read -r ref; do + [ -n "$ref" ] || continue + refs+=("$ref") + done <<<"$IMAGES" + if [ "${#refs[@]}" -gt 0 ]; then + tasks/scripts/trivy-scan.sh images "${refs[@]}" + fi + + - name: Prepare consolidated SARIF uploads + id: sarif + if: >- + ${{ + !cancelled() + && steps.config.conclusion == 'success' + && steps.images.conclusion == 'success' + && inputs.upload-sarif + }} + run: tasks/scripts/trivy-scan.sh prepare-sarif + + - name: Upload reports + id: reports + if: always() + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 + with: + name: trivy-${{ github.run_id }} + path: reports/trivy + if-no-files-found: ignore + retention-days: 14 + + - name: Report findings + if: >- + ${{ + !cancelled() + && steps.config.conclusion == 'success' + && steps.images.conclusion == 'success' + }} + env: + FAIL_ON_FINDINGS: ${{ inputs.fail-on-findings }} + run: | + set +e + tasks/scripts/trivy-scan.sh gate + status=$? + set -e + case "$status" in + 0) echo "No findings at ${TRIVY_SEVERITY}." ;; + 10) + if [ "$FAIL_ON_FINDINGS" = "true" ]; then + echo "::error::Trivy findings at ${TRIVY_SEVERITY}." + exit 1 + fi + echo "::warning::Trivy findings at ${TRIVY_SEVERITY}; this check is informational." + ;; + *) echo "::error::Trivy could not evaluate the reports (exit $status)."; exit "$status" ;; + esac + + upload-sarif: + name: Publish Trivy SARIF (${{ matrix.batch }}) + needs: scan + # Findings may fail the scan job after reports are prepared. Publish those + # complete results too, but never publish an incomplete/failed scan. + if: >- + ${{ + !cancelled() + && inputs.upload-sarif + && needs.scan.outputs.sarif-batches != '' + && needs.scan.outputs.artifact-id != '' + }} + runs-on: ubuntu-latest + timeout-minutes: 10 + permissions: + contents: read + security-events: write + strategy: + fail-fast: false + matrix: + batch: ${{ fromJSON(needs.scan.outputs.sarif-batches) }} + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + + - uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 + with: + artifact-ids: ${{ needs.scan.outputs.artifact-id }} + merge-multiple: true + path: reports/trivy + + # One configuration run, plus one run per image/platform or packaged + # chart. Each batch holds at most GitHub's limit of 20 SARIF runs. + - uses: github/codeql-action/upload-sarif@ff2f1c621b7f889edc0d3c761ac2e6a3f8cdb0dd # v4.37.7 + with: + sarif_file: reports/trivy/code-scanning/uploads/${{ matrix.batch }} diff --git a/.gitignore b/.gitignore index 3ef1a3769..a7867fe2f 100644 --- a/.gitignore +++ b/.gitignore @@ -67,6 +67,9 @@ pip-delete-this-directory.txt coverage.out coverage/ htmlcov/ + +# Trivy scan reports (tasks/scripts/trivy-scan-*.sh) +/reports/ .tox/ .nox/ .coverage diff --git a/.packit.yaml b/.packit.yaml index 6379d8db8..d3b92eafa 100644 --- a/.packit.yaml +++ b/.packit.yaml @@ -11,7 +11,6 @@ specfile_path: openshell.spec # Packages needed in the SRPM build environment to create vendor tarball srpm_build_deps: - - rust - cargo - git-core @@ -45,10 +44,6 @@ actions: # dist-info stays at the RPM Version; dev build identity is carried by # Release so Fedora's Python RPM post-processing can normalize metadata. - 'bash -c "if [ -n \"${OPENSHELL_CARGO_VERSION:-}\" ]; then sed -i -r \"s/^%global openshell_cargo_version .*/%global openshell_cargo_version ${OPENSHELL_CARGO_VERSION}/\" openshell.spec; fi"' - # Override image_tag to 'latest' for tagged stable releases. - # For PR and commit-to-main builds the spec default ('dev') is kept, - # matching the :dev images pushed by release-dev.yml. - - 'bash -c "if git describe --exact-match --tags HEAD 2>/dev/null | grep -qE ''^v[0-9]+\.[0-9]+\.[0-9]+$''; then sed -i ''s/^%global image_tag.*/%global image_tag latest/'' openshell.spec; fi"' jobs: # Build on every pull request targeting main for CI validation diff --git a/.trivyignore.yaml b/.trivyignore.yaml new file mode 100644 index 000000000..5e6afc010 --- /dev/null +++ b/.trivyignore.yaml @@ -0,0 +1,34 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +# False positives only; unimplemented hardening and accepted risks stay visible. +# The script passes this YAML file explicitly and requires each exception to use +# `**/`, which matches source and packaged-chart paths. +# Trivy cannot scope Helm exceptions to one occurrence, so keep IDs and paths +# narrow. + +misconfigurations: + # The namespace comes from `helm install -n`, not the rendered workload. + - id: KSV-0110 + paths: + - "**/statefulset.yaml" + - "**/deployment.yaml" + statement: >- + An artifact of rendering the chart outside a cluster. The namespace is + supplied at install time. + + # The ConfigMap stores an external Secret key name, not a credential. + - id: KSV-01010 + paths: + - "**/gateway-config.yaml" + statement: >- + The ConfigMap holds the name of a key in an external Secret, not a + credential. + + # Trivy cannot add this project's GHCR namespace to its trusted registries. + - id: KSV-0125 + paths: + - "**/statefulset.yaml" + - "**/deployment.yaml" + statement: >- + Images come from ghcr.io/nvidia/openshell, this project's own registry. diff --git a/AGENTS.md b/AGENTS.md index 2f2a166f5..0b35bd02f 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -10,7 +10,12 @@ OpenShell is built agent-first. We design systems and use agents to implement th ## Skills -Agent skills live in `.agents/skills/`. Your harness can discover and load them natively — do not rely on this file for a full inventory. The detailed skills table is in [CONTRIBUTING.md](CONTRIBUTING.md) (for humans). +OpenShell has two skill collections: + +- `skills/` contains public, installable skills for using and operating OpenShell. These skills must work outside a source checkout and use installed CLI help plus published documentation as their sources of truth. +- `.agents/skills/` contains internal contributor and maintainer workflows for developing OpenShell. Your repository-aware harness can discover and load them natively. + +Do not rely on this file for a full inventory. The detailed public and contributor skill tables are in [CONTRIBUTING.md](CONTRIBUTING.md) (for humans). ## Workflow Chains @@ -30,10 +35,11 @@ These pipelines connect skills into end-to-end workflows. Individual skill files | Path | Components | Purpose | |------|-----------|---------| | `crates/openshell-cli/` | CLI binary | User-facing command-line interface | +| `crates/openshell-conformance/` | CLI conformance library | Reusable driver-agnostic scenarios and command runner | +| `crates/openshell-conformance-cli/` | Conformance CLI | Distributable `list` and `run` entrypoint for gateway conformance | | `crates/openshell-server/` | Gateway server | Control-plane API, sandbox lifecycle, auth boundary | | `crates/openshell-sandbox/` | Sandbox runtime | Container supervision, policy-enforced egress routing | -| `crates/openshell-policy/` | Policy engine | Filesystem, network, process, and inference constraints | -| `crates/openshell-router/` | Privacy router | Privacy-aware LLM routing | +| `crates/openshell-policy/` | Policy engine | Filesystem, network, and process constraints | | `crates/openshell-bootstrap/` | Gateway metadata | Gateway registration metadata, auth token storage, mTLS bundle storage | | `crates/openshell-gateway-interceptors/` | Gateway interceptors | Intercepts and transforms configured gRPC requests at the gateway routing boundary | | `crates/openshell-ocsf/` | OCSF logging | OCSF v1.8.0 event types, builders, shorthand/JSONL formatters, tracing layers | @@ -41,6 +47,7 @@ These pipelines connect skills into end-to-end workflows. Individual skill files | `crates/openshell-otel-test-support/` | OpenTelemetry test support | Shared loopback OTLP collector fixture for tracing tests | | `crates/openshell-core/` | Shared core | Common types, configuration, error handling | | `crates/openshell-extension-core/` | Extension core | Shared extension identity, JWT claims, bearer-token rotation, and TLS transport primitives | +| `crates/openshell-gateway/` | Gateway binary composition | Links selected first-party compute drivers into the backend-agnostic server registry | | `crates/openshell-sdk/` | Shared client SDK | Async Rust gateway client (gRPC transport, TLS, OIDC refresh, edge tunnel); consumed by CLI, TUI, and `@openshell/sdk` | | `crates/openshell-providers/` | Provider management | Credential provider backends | | `crates/openshell-tui/` | Terminal UI | Ratatui-based dashboard for monitoring | @@ -51,11 +58,12 @@ These pipelines connect skills into end-to-end workflows. Individual skill files | `crates/openshell-driver-docker/` | Docker compute driver | In-process `ComputeDriver` backend for local Docker sandbox containers | | `crates/openshell-driver-podman/` | Podman compute driver | In-process `ComputeDriver` backend for local Podman sandbox containers | | `crates/openshell-driver-vm/` | VM compute driver | Standalone libkrun-backed `ComputeDriver` subprocess (embeds its own rootfs + runtime) | +| `crates/openshell-driver-mxc/` | Microsoft MXC compute driver | In-process Windows AppContainer and isolation-session compute backend | | `crates/openshell-prover/` | Policy prover | Policy verification and proof generation | | `crates/openshell-server-macros/` | Server macros | Compile-time helpers for gateway RPC authorization | | `crates/openshell-supervisor-middleware/` | Middleware runtime | Generic middleware registry, remote service integration, and chain execution | | `crates/openshell-supervisor-middleware-builtins/` | Built-in middleware | First-party in-process middleware implementations | -| `crates/openshell-supervisor-network/` | Network supervisor | Proxying, L7 enforcement, policy evaluation, and inference routing | +| `crates/openshell-supervisor-network/` | Network supervisor | Proxying, L7 enforcement, policy evaluation, and provider credential injection | | `crates/openshell-supervisor-process/` | Process supervisor | Process lifecycle, namespace, and bypass monitoring | | `crates/openshell-vfio/` | VFIO support | PCI and GPU passthrough preparation and lifecycle | | `python/openshell/` | Python SDK | Python bindings and CLI packaging | @@ -64,7 +72,8 @@ These pipelines connect skills into end-to-end workflows. Individual skill files | `deploy/` | Docker, Helm, K8s | Dockerfiles, Helm chart, manifests | | `docs/` | Published docs | MDX pages, navigation, and content assets | | `fern/` | Docs site config | Fern site config, components, and theme assets | -| `.agents/skills/` | Agent skills | Workflow automation for development | +| `skills/` | Public agent skills | Installable workflows for using and operating OpenShell | +| `.agents/skills/` | Contributor agent skills | Repository-aware workflows for developing OpenShell | | `.agents/agents/` | Agent personas | Sub-agent definitions (e.g., reviewer, doc writer) | | `architecture/` | Architecture docs | Design decisions and component documentation | @@ -105,7 +114,7 @@ Use an OCSF builder + `ocsf_emit!()` for events that represent **observable sand - SSH authentication (accepted, denied, nonce replay) - Process lifecycle (start, exit, timeout, signal failure) - Security findings (unsafe policy, unavailable controls, replay attacks) -- Configuration changes (policy load/reload, TLS setup, inference routes, settings) +- Configuration changes (policy load/reload, TLS setup, provider attachments, settings) - Application lifecycle (supervisor start, SSH server ready) ### When to use plain tracing @@ -127,7 +136,7 @@ Use `info!()`, `debug!()`, `warn!()` for **internal operational plumbing** that | SSH sessions | `SshActivityBuilder` | Authentication, channel operations | | Process start/stop | `ProcessActivityBuilder` | Entrypoint lifecycle, signal failures | | Security alerts | `DetectionFindingBuilder` | Nonce replay, bypass detection, unsafe policy. Dual-emit with the domain event. | -| Policy/config changes | `ConfigStateChangeBuilder` | Policy load, Landlock apply, TLS setup, inference routes, settings | +| Policy/config changes | `ConfigStateChangeBuilder` | Policy load, Landlock apply, TLS setup, provider attachments, settings | | Supervisor lifecycle | `AppLifecycleBuilder` | Sandbox start, SSH server ready/failed | ### Severity guidelines @@ -235,7 +244,7 @@ ocsf_emit!(event); ## Cluster Infrastructure Changes -- If you change gateway deployment infrastructure (e.g., Helm values/templates, gateway image packaging, or deploy logic in `openshell-cli`), update the `debug-openshell-cluster` skill in `.agents/skills/debug-openshell-cluster/SKILL.md` to reflect those changes. +- If you change gateway deployment infrastructure (e.g., Helm values/templates, gateway image packaging, or deploy logic in `openshell-cli`), update the `debug-openshell-cluster` skill in `skills/debug-openshell-cluster/SKILL.md` to reflect those changes. ## Skill Maintenance @@ -246,10 +255,10 @@ When behavior, commands, or development workflows change, review the related age - When making changes, update the relevant documentation in the `architecture/` directory. - When changes affect user-facing behavior, update the relevant published docs pages under `docs/` and navigation in `docs/index.yml`. - When changing gateway TOML fields, driver-specific config options, config defaults, or Helm rendering of `gateway.toml`, update `docs/reference/gateway-config.mdx` in the same branch. -- `fern/` contains the Fern site config, components, preview workflow inputs, and publish settings. +- `fern/` contains the Fern site config, components, preview workflow inputs, publish settings, and publishing documentation in `fern/README.md`. - Follow the docs style guide in [docs/CONTRIBUTING.mdx](docs/CONTRIBUTING.mdx): active voice, minimal formatting, no filler introductions, `shell` fences for copyable commands, and no duplicate body H1. -- Fern PR previews run through `.github/workflows/branch-docs.yml`, and production publish runs through the `publish-fern-docs` job in `.github/workflows/release-tag.yml`. -- Use the `update-docs` skill to scan recent commits and draft doc updates. +- Fern PR previews run through `.github/workflows/branch-docs.yml`. Release Dev publishes `dev`, and Release Tag publishes an immutable stable version plus `latest`. Both production paths call `.github/workflows/sync-docs.yml` once. +- Use the `update-docs-from-commits` skill to scan recent commits and draft doc updates. ### Architecture Docs diff --git a/CI.md b/CI.md index c0ec57f87..a84bd9063 100644 --- a/CI.md +++ b/CI.md @@ -24,17 +24,21 @@ Three opt-in labels enable the long-running E2E suites: When multiple labels are present, `Branch E2E Checks` builds each generic multi-architecture artifact set once and fans out enabled suites in parallel. Runtime-specific reusable workflows define the Docker, Podman, VM, and Kubernetes lanes. Composite actions own the replaceable Podman, KVM, kind, and mise setup. Each lane depends only on the artifact categories it consumes: VM does not wait for container-driver artifacts or supervisor images, and GPU does not wait for the gateway image. Docker, Podman, GPU, Rust, Python, MCP, and VM E2E reuse matching prebuilt gateway and CLI binaries instead of compiling debug binaries in test jobs. Standalone-driver lanes additionally reuse driver-free gateway and compute-driver artifacts. Kubernetes managed-driver lanes consume published gateway and supervisor images, while the standalone-driver lane composes its gateway image from prebuilt binaries. The `OpenShell / E2E` and `OpenShell / GPU E2E` required statuses are evaluated from separate suite result jobs inside that workflow. `test:e2e-kubernetes` is optional while Kubernetes HA and credential-driver behavior are under active iteration: failures are visible in the workflow run but do not publish a required CI gate status. -The GitHub ruleset should require the `OpenShell / ...` statuses published by `Required CI Gates`, not the push-triggered workflow jobs directly. +The GitHub ruleset should require the `OpenShell / ...` statuses published by +`Required CI Gates` plus the direct `OpenShell / Trivy Changes` result, not the +push-triggered workflow jobs themselves. ## Informational security reports -Security analysis that does not need NVIDIA infrastructure runs directly on -GitHub-hosted runners. These workflows receive no secrets. The PR-oriented -reports run on fork pull requests without waiting for copy-pr-bot. Scanner jobs -request `security-events: write` to publish SARIF to Code Scanning. GitHub -permits Code Scanning uploads from `pull_request` runs even when fork and -Dependabot contexts receive a read-only `GITHUB_TOKEN`, so those scanners upload -results directly and also retain report artifacts: +Security workflow compute runs directly on GitHub-hosted runners instead of +NVIDIA self-hosted runners. The PR-oriented workflows receive no secrets and run +on fork pull requests without waiting for copy-pr-bot. Codex Security release +qualification is the exception: it runs only for maintainer-created +pre-release tags and receives a scoped NVIDIA Inference API key for the scan +step. Scanner jobs request `security-events: write` to publish SARIF to Code +Scanning. GitHub permits Code Scanning uploads from `pull_request` runs even +when fork and Dependabot contexts receive a read-only `GITHUB_TOKEN`, so those +scanners upload results directly: - `Workflow Security Reports` runs Actionlint and Zizmor. Actionlint reports workflow syntax and expression findings. Zizmor reports only High severity, @@ -50,6 +54,20 @@ results directly and also retain report artifacts: E2E test code are excluded. It runs nightly on `main`, remains manually dispatchable for diagnostics, uploads results to Code Scanning, and retains workflow artifacts. +- `Codex Security Release Qualification` analyzes each `vX.Y.Z-pre.N` + candidate against the previous stable release. Every candidate in a release + train therefore rescans the cumulative stable-to-candidate diff. It calls + `https://inference-api.nvidia.com/v1` with + `openai/openai/gpt-5.6-sol` at medium reasoning effort. The + `CODEX_SECURITY_API_KEY` secret must contain an API key authorized for that + NVIDIA endpoint. Results are uploaded against the candidate commit on `main` + under a category shared by the train, for example + `codex-security/v0.1.1`, so later candidates replace earlier analyses. The + slash-qualified NVIDIA model identifier prevents Codex Security 0.1.24 from + enforcing `--max-cost`, so the workflow relies on its timeout, serialized + concurrency, and the inference account's spend controls. Raw reports are not + retained as workflow artifacts. Pre-release creation and stable-promotion + enforcement remain part of RFC 0014 and are not implemented by this workflow. Findings do not fail these workflows. Tool startup, configuration, build, and analysis failures still fail so a broken scanner cannot appear healthy. The @@ -63,6 +81,66 @@ nix develop --command actionlint -shellcheck= -pyflakes= nix develop --command zizmor --offline --persona=regular --min-severity=high --no-exit-codes . ``` +## Artifact scanning + +`Trivy Scan` is a self-contained `workflow_dispatch`/`workflow_call` step. It +always scans deployment configuration and optionally scans supplied OCI image +and chart references. It is not wired into a release workflow; a future analysis +orchestrator can call it directly. + +The scan job checks static deployment files (including Dockerfiles) once, both +local charts with default values, the OpenShell `HELM_PROFILES` selected in +`tasks/scripts/trivy-scan.sh`, requested packaged charts, and both Linux +architectures of each requested image. The profile list includes development +and E2E overlays for regression coverage, but excludes `values-spire-stack.yaml`, +which belongs to the external SPIRE chart. A new OpenShell values fixture needs +an explicit entry in this list to receive Trivy coverage. + +Findings are informational by default, but scanner failures still fail the job +and `fail-on-findings` enables enforcement. Reports retain every severity; the +gate uses `HIGH,CRITICAL` by default. + +All detailed JSON reports are retained in one artifact. The workflow summary and +Code Scanning configuration report deduplicate the same rule, target, resource, +and message across profiles, listing the affected profiles in each finding. +Images and packaged charts retain separate analyses, keyed by full reference +and platform so different registries or versions cannot overwrite each other. +SARIF is generated only for publication. The upload job publishes +only this consolidated configuration SARIF and the artifact analyses, in batches +of at most 20 runs to respect GitHub's per-file limit. It also publishes complete +reports when `fail-on-findings` makes the scan job fail; incomplete scans are +retained as artifacts but are not published to Code Scanning. + +`.trivyignore.yaml` is reserved for false positives. Every entry must use at +least one `**/` path; `yq` and `jq` validate this structure +before scanning. + +Run the scanner locally with: + +```shell +nix develop --command tasks/scripts/trivy-scan.sh config +nix develop --command tasks/scripts/trivy-scan.sh images ghcr.io/nvidia/openshell/gateway:dev +nix develop --command tasks/scripts/trivy-scan.sh gate +nix develop --command tasks/scripts/trivy-scan.sh prepare-sarif +``` + +Use a fresh `TRIVY_REPORT_DIR` for each scan session. `prepare-sarif` creates +`code-scanning/uploads//`; only these directories are intended for upload. + +### Pull-request change gate + +`Trivy Changes` scans the base and candidate when a pull request or merge group +changes deployment configuration or scanner inputs. It fails only for new +`HIGH` or `CRITICAL` misconfigurations and retains both report sets. + +The candidate ignore file is validated, but the baseline policy applies to both +scans so a change cannot exempt its own finding. Existing profiles are compared +independently using the detailed reports, before any presentation deduplication; +a new profile reuses the maximum known occurrence count. A selected fixture +missing from the candidate fails the scan; a new fixture absent from the base +has no baseline profile. Invalid Trivy reports fail closed. Image package and +operating-system CVEs remain the responsibility of the standalone scan. + ## Commit signing copy-pr-bot decides whether to mirror a PR automatically based on whether the author is trusted. For org members and collaborators, "trusted" means **all commits in the PR are cryptographically signed**. Unsigned commits, even from an org member, force the bot to wait for a maintainer's `/ok to test `. @@ -142,14 +220,18 @@ GitHub merge queue is required for `main`. Repository administrators must enable - `OpenShell / E2E` - `OpenShell / GPU E2E` - `OpenShell / Helm Lint` +- `OpenShell / Trivy Changes` -Do not require the underlying workflow job names directly. `Required CI Gates` publishes stable commit statuses for both PR-head mirror commits and merge-group commits. +`Required CI Gates` publishes the stable statuses for mirror-based workflows. +`Trivy Changes` runs directly on pull requests and merge groups and publishes +its own stable result status. Merge-group runs use the `merge_group` event. The event is distinct from `pull_request` and `push`, and GitHub will not report required checks for queued PRs unless the workflows include it. In this repository: - `Branch Checks` runs the standard non-E2E gates on the merge-group SHA. - `Branch E2E Checks` runs core E2E and GPU E2E for merge groups. Kubernetes HA E2E remains optional and label-driven on PRs. - `Helm Lint` runs for merge groups without the PR diff optimization, because the merge-group branch is the final integration state. +- `Trivy Changes` compares the merge-group configuration with its base and rejects new High or Critical findings. - `Required CI Gates` posts the same `OpenShell / ...` statuses to the merge-group SHA and does not require a `pull-request/` mirror for merge-group events. Maintainers should add ready PRs to the queue rather than pressing a direct merge button. GitHub removes a PR from the queue if the merge-group checks fail or time out. @@ -187,6 +269,9 @@ The bot's full administrator documentation is internal to NVIDIA. The only comma | `.github/workflows/workflow-security.yml` | Runs informational Actionlint and High-severity Zizmor reports on GitHub-hosted runners. | | `.github/workflows/dependency-review.yml` | Reports dependency changes when GitHub Dependency Graph is available; otherwise publishes a neutral warning. | | `.github/workflows/codeql.yml` | Runs nightly informational CodeQL analysis on `main` for Rust and the Go, Python, and TypeScript SDKs and retains SARIF artifacts. | +| `.github/workflows/codex-security.yml` | Scans the cumulative diff from the previous stable release to each pre-release candidate and publishes train-scoped SARIF on `main`. | +| `.github/workflows/trivy-changes.yml` | Blocks pull requests and merge groups that introduce new High or Critical Helm or Dockerfile misconfigurations. | +| `.github/workflows/trivy-scan.yml` | Manual or reusable scan of supplied OCI image/chart references and deployment configuration. Findings are informational by default and can be configured to fail the workflow. | ## Release workflows @@ -195,7 +280,7 @@ These workflows run after merge to publish dev/tagged artifacts and verify them. | File | Role | |---|---| | `.github/workflows/release-dev.yml` | Publishes the rolling `dev` build on every push to `main`. Builds gateway/supervisor images and binaries, packages, wheels, and pushes the Helm chart as `oci://ghcr.io/nvidia/openshell/helm-chart:0.0.0-dev` (plus an immutable `0.0.0-dev.` pin). Also dispatchable manually. | -| `.github/workflows/release-tag.yml` | Publishes a tagged public release. | +| `.github/workflows/release-tag.yml` | Publishes a tagged stable release. Its automatic tag trigger excludes `-pre.*`; manual dispatch remains maintainer-controlled. | | `.github/workflows/release-canary.yml` | Smoke-tests published artifacts on `macos`, `ubuntu`, `fedora`, and `kubernetes` (kind + Helm) runners. Triggers automatically when `Release Dev` succeeds, and via `workflow_dispatch` on any branch (`gh workflow run release-canary.yml --ref `). The `kubernetes` job pins to `0.0.0-dev` artifacts; the other jobs install the latest tagged release via `install.sh`. See the `test-release-canary` skill for the manual-dispatch playbook and local kind reproduction. | ## Required status contexts @@ -206,8 +291,13 @@ Require these statuses in the branch ruleset for PR and merge-queue CI: - `OpenShell / E2E` - `OpenShell / GPU E2E` - `OpenShell / Helm Lint` +- `OpenShell / Trivy Changes` -Do not require the underlying workflow jobs directly. PR workflow jobs only appear after copy-pr-bot mirrors trusted code, and merge-group workflow jobs run on temporary queue branches. The stable `OpenShell / ...` contexts prove the expected workflow completed for the commit that GitHub is about to merge. +For mirror-based workflows, require the statuses published by +`Required CI Gates`, not their underlying jobs. `OpenShell / Trivy Changes` is +the stable result job of the direct pull-request workflow. Together these +contexts prove the expected checks completed for the commit GitHub is about to +merge. Do not add the informational Actionlint, Zizmor, Dependency Review, or CodeQL jobs to the required status list while they remain in observation mode. diff --git a/CODE_OF_CONDUCT.md b/CODE_OF_CONDUCT.md new file mode 100644 index 000000000..4575d1600 --- /dev/null +++ b/CODE_OF_CONDUCT.md @@ -0,0 +1,5 @@ +# Code of Conduct + +We follow the [CNCF Code of Conduct](https://github.com/cncf/foundation/blob/main/code-of-conduct.md). + +Please contact [openshell-maintainers@ai-openshell.org](mailto:openshell-maintainers@ai-openshell.org) or the [CNCF Code of Conduct Committee](mailto:conduct@cncf.io) to report violations of the Code of Conduct. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 8b33c58fa..12fb5c0f4 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -48,7 +48,7 @@ Search open and closed issues for the same need. Bug reports and feature request Feature requests must also propose a user-facing workflow and describe alternatives considered. Define the externally observable behavior and leave internal implementation choices open. Bug reports instead include minimal reproduction steps, the OpenShell version and relevant environment, and a small, redacted log excerpt when it materially clarifies the behavior. -The project includes optional [agent skills](#agent-skills-for-contributors) for self-service troubleshooting and exploration. Use them when they help you, but summarize any useful result in your own words rather than pasting a diagnostic transcript. +The project includes optional [agent skills](#agent-skills) for using OpenShell and contributing to the repository. Use them when they help you, but summarize any useful result in your own words rather than pasting a diagnostic transcript. ### When to Open an Issue @@ -68,15 +68,29 @@ Do not start substantial issue-backed work until a maintainer has accepted the i Use agents and the repository skills as needed to understand the affected code, evaluate tradeoffs, implement the smallest coherent change, and verify it. The pull request should explain what changed and how it was tested; it should not substitute an agent transcript for the contributor's understanding. -## Agent Skills for Contributors +## Agent Skills -Skills live in `.agents/skills/`. Your agent's harness can discover and load them natively. Here is the full inventory: +OpenShell keeps skills for using the product separate from skills for developing the repository. + +### Skills for Using OpenShell + +Public skills live in `skills/` and work without an OpenShell source checkout. Install them with `npx skills add NVIDIA/OpenShell`. + +| Skill | Purpose | +| --- | --- | +| `openshell-cli` | CLI usage, sandbox lifecycle, provider management, and BYOC workflows | +| `debug-openshell-cluster` | Diagnose gateway deployment and health issues | +| `debug-inference` | Diagnose attached-provider inference, native endpoints, and migration from `inference.local` | +| `generate-sandbox-policy` | Generate YAML sandbox policies from requirements or API documentation | + +Public skills use `openshell --help` for installed command syntax and published OpenShell documentation for product concepts and configuration. They must not depend on repository-relative source or documentation files. + +### Agent Skills for Contributors + +Contributor and maintainer skills live in `.agents/skills/`. They are marked internal so the Agent Skills CLI excludes them from ordinary public discovery, but repository-aware agent harnesses can discover and load them natively. Internal metadata is a discovery filter, not an access-control boundary. | Category | Skill | Purpose | | --------------- | ------------------------- | --------------------------------------------------------------------------------------------------- | -| Getting Started | `openshell-cli` | CLI usage, sandbox lifecycle, provider management, BYOC workflows | -| Getting Started | `debug-openshell-cluster` | Diagnose gateway deployment and health issues | -| Getting Started | `debug-inference` | Diagnose `inference.local`, host-backed local inference, and direct external inference setup issues | | Contributing | `create-spike` | Investigate a problem, produce a structured GitHub issue | | Contributing | `create-rfc` | Create RFC proposals from the repository template | | Contributing | `build-from-issue` | Plan and implement work from a GitHub issue (maintainer workflow) | @@ -89,11 +103,10 @@ Skills live in `.agents/skills/`. Your agent's harness can discover and load the | Reviewing | `launch-openshell-gator` | Launch and supervise OpenShell gator agents for issue and PR monitoring | | Reviewing | `test-release-canary` | Dispatch and iterate on the Release Canary workflow that smoke-tests published artifacts | | Triage | `triage-issue` | Assess, classify, and route community-filed issues | -| Platform | `generate-sandbox-policy` | Generate YAML sandbox policies from requirements or API docs | | Platform | `helm-dev-environment` | Start and manage the local Kubernetes development environment | | Platform | `tui-development` | Development guide for the ratatui-based terminal UI | | Platform | `build-openshell-mxc-windows` | Maintain and validate the build-only x64 and ARM64 Windows MSVC lane | -| Documentation | `update-docs` | Scan recent commits and draft doc updates for user-facing changes | +| Documentation | `update-docs-from-commits` | Scan recent commits and draft doc updates for user-facing changes | | Maintenance | `sync-agent-infra` | Detect and fix drift across agent-first infrastructure files | | Reference | `sbom` | Generate SBOMs and resolve dependency licenses | @@ -293,14 +306,19 @@ echo 'eval "$(~/.local/bin/mise activate zsh)"' >> ~/.zshrc Project requirements: -- Rust 1.90+ +- Rust 1.94+ - Python 3.11+ - Docker (running) +- CMake 3.16+ (only required when building with the `bundled-z3` feature) ### Z3 installation -The `openshell-prover` crate links against Z3. On macOS and Linux, install the -system Z3 development package; `z3-sys` discovers it through `pkg-config`. +The `openshell-prover` crate links directly against Z3. The `openshell-server` +crate depends on the prover, and the `openshell-gateway` binary crate depends +on `openshell-server` in turn; both forward a `bundled-z3` feature down to +`openshell-prover/bundled-z3`. The `openshell-cli` crate does not depend on +Z3. On macOS and Linux, install the system Z3 development package; `z3-sys` +discovers it through `pkg-config`. ```bash # macOS @@ -314,7 +332,7 @@ sudo dnf install z3-devel ``` If you prefer not to install Z3 system-wide, use the bundled Z3 feature. This -compiles Z3 from source during the Rust build: +compiles Z3 from source during the Rust build and requires CMake 3.16+: ```bash cargo build -p openshell-prover --features bundled-z3 @@ -328,16 +346,29 @@ For x86-64 Windows MSVC builds, use one of these Z3 paths: is set. - Bundled Z3: pass `--features bundled-z3` so `z3-sys` builds Z3 from source. -Both Windows paths still require `libclang.dll` for `bindgen`. If LLVM is not on -the default search path, set `LIBCLANG_PATH` to the directory containing -`libclang.dll`. +`openshell-prover` itself has no `bindgen`/`libclang` dependency, so building +just this crate does not require `LIBCLANG_PATH`: + +```powershell +cargo build -p openshell-prover --target x86_64-pc-windows-msvc --features bundled-z3 +``` + +### Windows full build + +To build the full set of Windows binaries, including `openshell-gateway.exe` +and `openshell.exe`, use the `windows:build:x64` mise task instead of a +single-crate `cargo build`. It builds Z3 from source (bundled) by default. A +full build also compiles crates that use `bindgen` (e.g. the MXC driver on +Windows), so it requires `libclang.dll`; if LLVM is not on the default search +path, set `LIBCLANG_PATH` to the directory containing `libclang.dll`: ```powershell $env:LIBCLANG_PATH='C:\Program Files\Microsoft Visual Studio\2022\\VC\Tools\Llvm\x64\bin' -cargo build -p openshell-cli --target x86_64-pc-windows-msvc --features bundled-z3 +mise run --skip-tools windows:build:x64 ``` -To use a local x64 Z3 release with the Windows task wrapper: +To use a local x64 Z3 release instead of the bundled build, set +`Z3_LIBRARY_PATH_OVERRIDE` and `Z3_SYS_Z3_HEADER` before running the task: ```powershell $env:Z3_LIBRARY_PATH_OVERRIDE='C:\path\to\z3-4.16.0-x64-win\bin' @@ -429,7 +460,8 @@ These are the primary `mise` tasks for day-to-day development: | `fern/` | Fern site config, components, and theme assets | | `architecture/` | Architecture docs and plans | | `rfc/` | Request for Comments proposals | -| `.agents/` | Agent skills and persona definitions | +| `skills/` | Public skills for using and operating OpenShell | +| `.agents/` | Contributor skills and persona definitions | ## RFCs @@ -439,7 +471,7 @@ New features always start as GitHub issues using the feature request template. F If your change affects user-facing behavior (new flags, changed defaults, new features, bug fixes that contradict existing docs), update the relevant pages under `docs/` in the same PR and adjust `docs/index.yml` if navigation changes. For explicit navigation entries, keep `page:` aligned with `sidebar-title` when present and put relative `slug:` values in `docs/index.yml`. Reserve frontmatter `slug` for folder-discovered pages or absolute URL overrides. -To ensure your doc changes follow NVIDIA documentation style, use the `update-docs` skill. +To ensure your doc changes follow NVIDIA documentation style, use the `update-docs-from-commits` skill. It scans commits, identifies doc pages that need updates, and drafts content that follows the style guide in `docs/CONTRIBUTING.mdx`. To preview Fern docs locally: @@ -456,9 +488,9 @@ mise run docs PRs that touch `docs/**` or `fern/**` are validated by `.github/workflows/branch-docs.yml`, and they get a preview when `FERN_TOKEN` is available to the workflow. -Fern docs publishing is handled by the `publish-fern-docs` job in `.github/workflows/release-tag.yml` when a release tag is created. +Release Dev publishes the `dev` docs version from `main`. Release Tag publishes an immutable stable version and updates `latest`. See [fern/README.md](fern/README.md) for the source layout, version model, and publishing workflows. -`docs/` is the source-of-truth docs tree. `fern/` contains the site config, components, and theme assets that publish those pages. +`docs/` is the source-of-truth docs tree. `fern/` contains the site configuration, components, theme assets, and its README. See [docs/CONTRIBUTING.mdx](docs/CONTRIBUTING.mdx) for the current docs authoring guide. diff --git a/Cargo.lock b/Cargo.lock index 5b30879aa..e2fad6fc7 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -228,6 +228,40 @@ dependencies = [ "serde_json", ] +[[package]] +name = "async-broadcast" +version = "0.7.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "435a87a52755b8f27fcf321ac4f04b2802e337c8c4872923137471ec39c37532" +dependencies = [ + "event-listener", + "event-listener-strategy", + "futures-core", + "pin-project-lite", +] + +[[package]] +name = "async-stream" +version = "0.3.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0b5a71a6f37880a80d1d7f19efd781e4b5de42c88f0722cc13bcb6cc2cfe8476" +dependencies = [ + "async-stream-impl", + "futures-core", + "pin-project-lite", +] + +[[package]] +name = "async-stream-impl" +version = "0.3.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c7c24de15d275a1ecfd47a380fb4d5ec9bfe0933f309ed5e705b775596a3574d" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.117", +] + [[package]] name = "async-trait" version = "0.1.89" @@ -434,24 +468,18 @@ dependencies = [ "aws-smithy-async", "aws-smithy-runtime-api", "aws-smithy-types", - "h2 0.3.27", - "h2 0.4.13", - "http 0.2.12", + "h2", "http 1.4.0", - "http-body 0.4.6", - "hyper 0.14.32", - "hyper 1.9.0", - "hyper-rustls 0.24.2", - "hyper-rustls 0.27.9", + "hyper", + "hyper-rustls", "hyper-util", "pin-project-lite", - "rustls 0.21.12", - "rustls 0.23.38", + "rustls", "rustls-native-certs", "rustls-pki-types", "tokio", - "tokio-rustls 0.26.4", - "tower 0.5.3", + "tokio-rustls", + "tower", "tracing", ] @@ -608,14 +636,14 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "31b698c5f9a010f6573133b09e0de5408834d0c82f8d7475a89fc1867a71cd90" dependencies = [ "axum-core", - "base64 0.22.1", + "base64", "bytes", "form_urlencoded", "futures-util", "http 1.4.0", "http-body 1.0.1", "http-body-util", - "hyper 1.9.0", + "hyper", "hyper-util", "itoa", "matchit", @@ -631,7 +659,7 @@ dependencies = [ "sync_wrapper", "tokio", "tokio-tungstenite 0.29.0", - "tower 0.5.3", + "tower", "tower-layer", "tower-service", "tracing", @@ -657,14 +685,14 @@ dependencies = [ ] [[package]] -name = "backoff" -version = "0.4.0" +name = "backon" +version = "1.6.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b62ddb9cb1ec0a098ad4bbf9344d0713fa193ae1a80af55febcff2627b6a00c1" +checksum = "cffb0e931875b666fc4fcb20fee52e9bbd1ef836fd9e9e04ec21555f9f85f7ef" dependencies = [ - "getrandom 0.2.17", - "instant", - "rand 0.8.6", + "fastrand", + "gloo-timers", + "tokio", ] [[package]] @@ -703,12 +731,6 @@ version = "1.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "fd307490d624467aa6f74b0eabb77633d1f758a7b25f12bceb0b22e08d9726f6" -[[package]] -name = "base64" -version = "0.21.7" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9d297deb1925b89f2ccc13d7635fa0714f12c87adce1c75356b39ca9b7178567" - [[package]] name = "base64" version = "0.22.1" @@ -810,7 +832,7 @@ version = "0.20.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ee04c4c84f1f811b017f2fbb7dd8815c976e7ca98593de9c1e2afad0f636bff4" dependencies = [ - "base64 0.22.1", + "base64", "bollard-stubs", "bytes", "futures-core", @@ -818,7 +840,7 @@ dependencies = [ "hex", "http 1.4.0", "http-body-util", - "hyper 1.9.0", + "hyper", "hyper-named-pipe", "hyper-util", "hyperlocal", @@ -1529,17 +1551,6 @@ dependencies = [ "powerfmt", ] -[[package]] -name = "derivative" -version = "2.2.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "fcc3dd5e9e9c0b295d6e1e4d811fb6f157d5ffd784b8d202fc62eac8035a770b" -dependencies = [ - "proc-macro2", - "quote", - "syn 1.0.109", -] - [[package]] name = "derive_builder" version = "0.20.2" @@ -1725,6 +1736,18 @@ dependencies = [ "zeroize", ] +[[package]] +name = "educe" +version = "0.6.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1d7bc049e1bd8cdeb31b68bbd586a9464ecf9f3944af3958a7a9d0f8b9799417" +dependencies = [ + "enum-ordinalize", + "proc-macro2", + "quote", + "syn 2.0.117", +] + [[package]] name = "either" version = "1.15.0" @@ -1783,6 +1806,26 @@ version = "1.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "34aa73646ffb006b8f5147f3dc182bd4bcb190227ce861fc4a4844bf8e3cb2c0" +[[package]] +name = "enum-ordinalize" +version = "4.4.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "89dd01549b09589510cf0647475075d12071456586d70f5c75c98ae2a5537677" +dependencies = [ + "enum-ordinalize-derive", +] + +[[package]] +name = "enum-ordinalize-derive" +version = "4.4.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a65863d15a4ce2888bd2f0f543cc963d3879c3a022c8ee43f6141d479a3ac815" +dependencies = [ + "proc-macro2", + "quote", + "syn 3.0.5", +] + [[package]] name = "enum_dispatch" version = "0.3.13" @@ -1833,13 +1876,12 @@ dependencies = [ [[package]] name = "etcetera" -version = "0.8.0" +version = "0.11.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "136d1b5283a1ab77bd9257427ffd09d8667ced0570b6f938942bc7568ed5b943" +checksum = "de48cc4d1c1d97a20fd819def54b890cadde72ed3ad0c614822a0a433361be96" dependencies = [ "cfg-if", - "home", - "windows-sys 0.48.0", + "windows-sys 0.61.2", ] [[package]] @@ -1853,6 +1895,16 @@ dependencies = [ "pin-project-lite", ] +[[package]] +name = "event-listener-strategy" +version = "0.5.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8be9f3dfaaffdae2972880079a491a1a8bb7cbed0b8dd7a347f668b4150a3b93" +dependencies = [ + "event-listener", + "pin-project-lite", +] + [[package]] name = "fastrand" version = "2.4.1" @@ -1926,9 +1978,9 @@ dependencies = [ [[package]] name = "flume" -version = "0.11.1" +version = "0.12.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "da0e4dd2a88388a1f4ccc7c9ce104604dab68d9f408dc34cd45823d5a9069095" +checksum = "5e139bc46ca777eb5efaf62df0ab8cc5fd400866427e56c68b22e414e53bd3be" dependencies = [ "futures-core", "futures-sink", @@ -2184,6 +2236,18 @@ dependencies = [ "regex-syntax", ] +[[package]] +name = "gloo-timers" +version = "0.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bbb143cf96099802033e0d4f4963b19fd2e0b728bcf076cd9cf7f6634f092994" +dependencies = [ + "futures-channel", + "futures-core", + "js-sys", + "wasm-bindgen", +] + [[package]] name = "group" version = "0.13.0" @@ -2208,28 +2272,9 @@ dependencies = [ [[package]] name = "h2" -version = "0.3.27" +version = "0.4.19" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0beca50380b1fc32983fc1cb4587bfa4bb9e78fc259aad4a0032d2080309222d" -dependencies = [ - "bytes", - "fnv", - "futures-core", - "futures-sink", - "futures-util", - "http 0.2.12", - "indexmap", - "slab", - "tokio", - "tokio-util", - "tracing", -] - -[[package]] -name = "h2" -version = "0.4.13" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2f44da3a8150a6703ed5d34e164b875fd14c2cdab9af1252a9a1020bde2bdc54" +checksum = "ef8e5e5a340588f4452631496976cf8636d4a7ecf600239fdc27615d2530bc16" dependencies = [ "atomic-waker", "bytes", @@ -2249,10 +2294,6 @@ name = "hashbrown" version = "0.14.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e5274423e17b7c9fc20b6e7e208532f9b19825d82dfd615708b70edd83df41f1" -dependencies = [ - "ahash", - "allocator-api2", -] [[package]] name = "hashbrown" @@ -2271,6 +2312,8 @@ version = "0.16.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "841d1cc9bed7f9236f321df977030373f4a4163ae1a7dbfe1a51a2c1a51d9100" dependencies = [ + "allocator-api2", + "equivalent", "foldhash 0.2.0", ] @@ -2282,11 +2325,35 @@ checksum = "4f467dd6dccf739c208452f8014c75c18bb8301b050ad1cfb27153803edb0f51" [[package]] name = "hashlink" -version = "0.10.0" +version = "0.11.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7382cf6263419f2d8df38c55d7da83da5c18aef87fc7a7fc1fb1e344edfe14c1" +checksum = "824e001ac4f3012dd16a264bec811403a67ca9deb6c102fc5049b32c4574b35f" dependencies = [ - "hashbrown 0.15.5", + "hashbrown 0.16.1", +] + +[[package]] +name = "headers" +version = "0.4.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b3314d5adb5d94bcdf56771f2e50dbbc80bb4bdf88967526706205ac9eff24eb" +dependencies = [ + "base64", + "bytes", + "headers-core", + "http 1.4.0", + "httpdate", + "mime", + "sha1 0.10.6", +] + +[[package]] +name = "headers-core" +version = "0.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "54b4a22553d4242c49fddb9ba998a99962b5cc6f22cb5a3482bec22522403ce4" +dependencies = [ + "http 1.4.0", ] [[package]] @@ -2377,6 +2444,17 @@ dependencies = [ "windows-sys 0.61.2", ] +[[package]] +name = "hostname" +version = "0.4.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "617aaa3557aef3810a6369d0a99fac8a080891b68bd9f9812a1eeda0c0730cbd" +dependencies = [ + "cfg-if", + "libc", + "windows-link", +] + [[package]] name = "http" version = "0.2.12" @@ -2465,30 +2543,6 @@ dependencies = [ "zeroize", ] -[[package]] -name = "hyper" -version = "0.14.32" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "41dfc780fdec9373c01bae43289ea34c972e40ee3c9f6b3c8801a35f35586ce7" -dependencies = [ - "bytes", - "futures-channel", - "futures-core", - "futures-util", - "h2 0.3.27", - "http 0.2.12", - "http-body 0.4.6", - "httparse", - "httpdate", - "itoa", - "pin-project-lite", - "socket2 0.5.10", - "tokio", - "tower-service", - "tracing", - "want", -] - [[package]] name = "hyper" version = "1.9.0" @@ -2499,7 +2553,7 @@ dependencies = [ "bytes", "futures-channel", "futures-core", - "h2 0.4.13", + "h2", "http 1.4.0", "http-body 1.0.1", "httparse", @@ -2511,6 +2565,25 @@ dependencies = [ "want", ] +[[package]] +name = "hyper-http-proxy" +version = "1.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bd1d471ea2f65ba45eddb1d6ab7d58ac2671d2d4ac14da5c6516ae1d97e1327a" +dependencies = [ + "bytes", + "futures-util", + "headers", + "http 1.4.0", + "hyper", + "hyper-rustls", + "hyper-util", + "pin-project-lite", + "tokio", + "tokio-rustls", + "tower-service", +] + [[package]] name = "hyper-named-pipe" version = "0.1.0" @@ -2518,7 +2591,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "73b7d8abf35697b81a825e386fc151e0d503e8cb5fcb93cc8669c376dfd6f278" dependencies = [ "hex", - "hyper 1.9.0", + "hyper", "hyper-util", "pin-project-lite", "tokio", @@ -2526,21 +2599,6 @@ dependencies = [ "winapi", ] -[[package]] -name = "hyper-rustls" -version = "0.24.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ec3efd23720e2049821a693cbc7e65ea87c72f1c58ff2f9522ff332b1491e590" -dependencies = [ - "futures-util", - "http 0.2.12", - "hyper 0.14.32", - "log", - "rustls 0.21.12", - "tokio", - "tokio-rustls 0.24.1", -] - [[package]] name = "hyper-rustls" version = "0.27.9" @@ -2548,13 +2606,13 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "33ca68d021ef39cf6463ab54c1d0f5daf03377b70561305bb89a8f83aab66e0f" dependencies = [ "http 1.4.0", - "hyper 1.9.0", + "hyper", "hyper-util", "log", - "rustls 0.23.38", + "rustls", "rustls-native-certs", "tokio", - "tokio-rustls 0.26.4", + "tokio-rustls", "tower-service", ] @@ -2564,7 +2622,7 @@ version = "0.5.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "2b90d566bffbce6a75bd8b09a05aa8c2cb1fabb6cb348f8840c9e4c90a0d83b0" dependencies = [ - "hyper 1.9.0", + "hyper", "hyper-util", "pin-project-lite", "tokio", @@ -2577,18 +2635,18 @@ version = "0.1.20" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "96547c2556ec9d12fb1578c4eaf448b04993e7fb79cbaad930a656880a6bdfa0" dependencies = [ - "base64 0.22.1", + "base64", "bytes", "futures-channel", "futures-util", "http 1.4.0", "http-body 1.0.1", - "hyper 1.9.0", + "hyper", "ipnet", "libc", "percent-encoding", "pin-project-lite", - "socket2 0.6.3", + "socket2", "tokio", "tower-service", "tracing", @@ -2602,7 +2660,7 @@ checksum = "986c5ce3b994526b3cd75578e62554abd09f0899d6206de48b3e96ab34ccc8c7" dependencies = [ "hex", "http-body-util", - "hyper 1.9.0", + "hyper", "hyper-util", "pin-project-lite", "tokio", @@ -2822,15 +2880,6 @@ dependencies = [ "hybrid-array", ] -[[package]] -name = "instant" -version = "0.1.13" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e0242819d153cba4b4b05a5a8f2a7e9bbf97b6055b2a002b395c96b5ff3c0222" -dependencies = [ - "cfg-if", -] - [[package]] name = "internal-russh-num-bigint" version = "0.5.0" @@ -3021,18 +3070,38 @@ dependencies = [ ] [[package]] -name = "jsonpath-rust" -version = "0.5.1" +name = "json-patch" +version = "4.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "19d8fe85bd70ff715f31ce8c739194b423d79811a19602115d611a3ec85d6200" +checksum = "7421438de105a0827e44fadd05377727847d717c80ce29a229f85fd04c427b72" +dependencies = [ + "jsonptr", + "serde", + "serde_json", + "thiserror 2.0.18", +] + +[[package]] +name = "jsonpath-rust" +version = "0.7.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0c00ae348f9f8fd2d09f82a98ca381c60df9e0820d8d79fce43e649b4dc3128b" dependencies = [ - "lazy_static", - "once_cell", "pest", "pest_derive", "regex", "serde_json", - "thiserror 1.0.69", + "thiserror 2.0.18", +] + +[[package]] +name = "jsonptr" +version = "0.7.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a5a3cc660ba5d72bce0b3bb295bf20847ccbb40fd423f3f05b61273672e561fe" +dependencies = [ + "serde", + "serde_json", ] [[package]] @@ -3042,7 +3111,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "0529410abe238729a60b108898784df8984c87f6054c9c4fcacc47e4803c1ce1" dependencies = [ "aws-lc-rs", - "base64 0.22.1", + "base64", "ed25519-dalek 2.2.0", "getrandom 0.2.17", "hmac 0.12.1", @@ -3061,11 +3130,11 @@ dependencies = [ [[package]] name = "k8s-openapi" -version = "0.21.1" +version = "0.24.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "550f99d93aa4c2b25de527bce492d772caf5e21d7ac9bd4b508ba781c8d91e30" +checksum = "2c75b990324f09bef15e791606b7b7a296d02fc88a344f6eba9390970a870ad5" dependencies = [ - "base64 0.21.7", + "base64", "chrono", "serde", "serde-value", @@ -3129,9 +3198,9 @@ dependencies = [ [[package]] name = "kube" -version = "0.90.0" +version = "0.99.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "65bfada4e00dac93a7b94e454ae4cde04ff8786645ac1b98f31352272e2682b5" +checksum = "9a4eb20010536b48abe97fec37d23d43069bcbe9686adcf9932202327bc5ca6e" dependencies = [ "k8s-openapi", "kube-client", @@ -3142,11 +3211,11 @@ dependencies = [ [[package]] name = "kube-client" -version = "0.90.0" +version = "0.99.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e0708306b5c0085f249f5e3d2d56a9bbfe0cbbf4fd4eb9ed4bbba542ba7649a7" +checksum = "7fc2ed952042df20d15ac2fe9614d0ec14b6118eab89633985d4b36e688dccf1" dependencies = [ - "base64 0.22.1", + "base64", "bytes", "chrono", "either", @@ -3155,79 +3224,83 @@ dependencies = [ "http 1.4.0", "http-body 1.0.1", "http-body-util", - "hyper 1.9.0", - "hyper-rustls 0.27.9", + "hyper", + "hyper-http-proxy", + "hyper-rustls", "hyper-timeout", "hyper-util", "jsonpath-rust", "k8s-openapi", "kube-core", "pem", - "rustls 0.23.38", - "rustls-pemfile", + "rustls", "secrecy", "serde", "serde_json", "serde_yaml", - "thiserror 1.0.69", + "thiserror 2.0.18", "tokio", "tokio-util", - "tower 0.4.13", - "tower-http 0.5.2", + "tower", + "tower-http", "tracing", ] [[package]] name = "kube-core" -version = "0.90.0" +version = "0.99.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7845bcc3e0f422df4d9049570baedd9bc1942f0504594e393e72fe24092559cf" +checksum = "ff0d0793db58e70ca6d689489183816cb3aa481673e7433dc618cf7e8007c675" dependencies = [ "chrono", "form_urlencoded", "http 1.4.0", - "json-patch", + "json-patch 4.2.0", "k8s-openapi", "schemars", "serde", + "serde-value", "serde_json", - "thiserror 1.0.69", + "thiserror 2.0.18", ] [[package]] name = "kube-derive" -version = "0.90.0" +version = "0.99.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5d0d2527a6ff7adf00b34d558c4c5de9404abe28808cb0a4c64b57e2c1b0716a" +checksum = "c562f58dc9f7ca5feac8a6ee5850ca221edd6f04ce0dd2ee873202a88cd494c9" dependencies = [ "darling", "proc-macro2", "quote", + "serde", "serde_json", "syn 2.0.117", ] [[package]] name = "kube-runtime" -version = "0.90.0" +version = "0.99.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4560e2c5c71366f6dceb6500ce33cf72299aede92381bb875dc2d4ba4f102c21" +checksum = "88f34cfab9b4bd8633062e0e85edb81df23cb09f159f2e31c60b069ae826ffdc" dependencies = [ "ahash", + "async-broadcast", + "async-stream", "async-trait", - "backoff", - "derivative", + "backon", + "educe", "futures", - "hashbrown 0.14.5", - "json-patch", + "hashbrown 0.15.5", + "hostname", + "json-patch 4.2.0", "k8s-openapi", "kube-client", "parking_lot", "pin-project", "serde", "serde_json", - "smallvec", - "thiserror 1.0.69", + "thiserror 2.0.18", "tokio", "tokio-util", "tracing", @@ -3304,16 +3377,6 @@ dependencies = [ "vcpkg", ] -[[package]] -name = "libyml" -version = "0.0.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3302702afa434ffa30847a83305f0a69d6abd74293b6554c18ec85c7ef30c980" -dependencies = [ - "anyhow", - "version_check", -] - [[package]] name = "linux-raw-sys" version = "0.4.15" @@ -3379,12 +3442,12 @@ checksum = "47e1ffaa40ddd1f3ed91f717a33c8c0ee23fff369e3aa8772b9605cc1d22f4c3" [[package]] name = "md-5" -version = "0.10.6" +version = "0.11.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d89e7ee0cfbedfc4da3340218492196241d89eefb6dab27de5df917a6d2e78cf" +checksum = "69b6441f590336821bb897fb28fc622898ccceb1d6cea3fde5ea86b090c4de98" dependencies = [ "cfg-if", - "digest 0.10.7", + "digest 0.11.2", ] [[package]] @@ -3415,9 +3478,9 @@ version = "0.18.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "3589659543c04c7dc5526ec858591015b87cd8746583b51b48ef4353f99dbcda" dependencies = [ - "base64 0.22.1", + "base64", "http-body-util", - "hyper 1.9.0", + "hyper", "hyper-util", "indexmap", "ipnet", @@ -3622,6 +3685,22 @@ dependencies = [ "bitflags 2.11.1", ] +[[package]] +name = "noyalib" +version = "0.0.28" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9f075ef19fa3bcf8697c0ef96c37d5c435d339a40ab8081cae3aac3a4e7fee9a" +dependencies = [ + "hashbrown 0.17.0", + "indexmap", + "libm", + "memchr", + "rustc-hash 2.1.2", + "serde", + "serde_core", + "smallvec", +] + [[package]] name = "nu-ansi-term" version = "0.50.3" @@ -3715,7 +3794,7 @@ version = "5.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "51e219e79014df21a225b1860a479e2dcd7cbd9130f4defd4bd0e191ea31d67d" dependencies = [ - "base64 0.22.1", + "base64", "chrono", "getrandom 0.2.17", "http 1.4.0", @@ -3821,16 +3900,13 @@ checksum = "384b8ab6d37215f3c5301a95a4accb5d64aa607f1fcb26a11b5303878451b4fe" name = "openshell-bootstrap" version = "0.0.0" dependencies = [ - "bollard", "bytes", - "futures", "miette", "openshell-core", "rcgen", "serde", "serde_json", "sha2 0.10.9", - "tar", "tempfile", "tokio", "tracing", @@ -3841,21 +3917,23 @@ name = "openshell-cli" version = "0.0.0" dependencies = [ "anyhow", - "base64 0.22.1", + "base64", "bytes", "chrono", "clap", "clap_complete", + "console", "crossterm 0.28.1", "dialoguer", "futures", "http-body-util", - "hyper 1.9.0", - "hyper-rustls 0.27.9", + "hyper", + "hyper-rustls", "hyper-util", "indicatif", "miette", "nix 0.29.0", + "noyalib", "oauth2", "openshell-bootstrap", "openshell-core", @@ -3867,32 +3945,54 @@ dependencies = [ "prost-types", "rcgen", "reqwest 0.12.28", - "rustls 0.23.38", + "rustls", "rustls-pemfile", "serde", "serde_json", - "serde_yml", "tar", "temp-env", "tempfile", "thiserror 2.0.18", "tokio", - "tokio-rustls 0.26.4", + "tokio-rustls", "tokio-stream", "tokio-tungstenite 0.26.2", "tonic", - "tower 0.5.3", + "tower", "tracing", "tracing-subscriber", "url", ] +[[package]] +name = "openshell-conformance" +version = "0.0.0" +dependencies = [ + "rand 0.9.4", + "serde", + "serde_json", + "tempfile", + "tokio", + "toml", +] + +[[package]] +name = "openshell-conformance-cli" +version = "0.0.0" +dependencies = [ + "clap", + "openshell-conformance", + "serde", + "serde_json", + "tokio", +] + [[package]] name = "openshell-core" version = "0.0.0" dependencies = [ "async-trait", - "base64 0.22.1", + "base64", "chrono", "glob", "ipnet", @@ -3902,8 +4002,11 @@ dependencies = [ "prost", "prost-types", "protoc-bin-vendored", + "rcgen", "reqwest 0.12.28", "rustix 1.1.4", + "rustls", + "rustls-pemfile", "serde", "serde_json", "tar", @@ -3923,10 +4026,10 @@ name = "openshell-driver-db-credstore" version = "0.0.0" dependencies = [ "async-trait", - "base64 0.22.1", + "aws-lc-rs", + "base64", "futures", "openshell-core", - "ring", "serde", "serde_json", "sha2 0.10.9", @@ -3961,7 +4064,7 @@ dependencies = [ "tokio-stream", "toml", "tonic", - "tower-http 0.6.8", + "tower-http", "tracing", "tracing-opentelemetry", "tracing-subscriber", @@ -3998,7 +4101,7 @@ dependencies = [ "tokio-stream", "toml", "tonic", - "tower 0.5.3", + "tower", "tracing", "tracing-subscriber", ] @@ -4026,13 +4129,13 @@ dependencies = [ name = "openshell-driver-mxc" version = "0.0.0" dependencies = [ - "base64 0.22.1", + "base64", "futures", + "noyalib", "openshell-core", "openshell-policy", "serde", "serde_json", - "serde_yml", "tempfile", "thiserror 2.0.18", "tokio", @@ -4050,7 +4153,7 @@ dependencies = [ "futures", "http 1.4.0", "http-body-util", - "hyper 1.9.0", + "hyper", "hyper-util", "miette", "nix 0.29.0", @@ -4068,7 +4171,7 @@ dependencies = [ "tokio", "tokio-stream", "tonic", - "tower-http 0.6.8", + "tower-http", "tracing", "tracing-opentelemetry", "tracing-subscriber", @@ -4084,6 +4187,7 @@ dependencies = [ "miette", "openshell-core", "reqwest 0.12.28", + "rustls", "serde", "serde_json", "sha2 0.10.9", @@ -4111,6 +4215,7 @@ dependencies = [ "nix 0.29.0", "oci-client", "openshell-core", + "openshell-driver-podman", "openshell-otel", "openshell-otel-test-support", "openshell-policy", @@ -4130,7 +4235,7 @@ dependencies = [ "tokio", "tokio-stream", "tonic", - "tower-http 0.6.8", + "tower-http", "tracing", "tracing-opentelemetry", "tracing-subscriber", @@ -4145,21 +4250,45 @@ dependencies = [ "http 1.4.0", "hyper-util", "rcgen", - "rustls 0.23.38", + "rustls", "serde", "serde_json", "thiserror 2.0.18", "tokio", "tokio-stream", "tonic", - "tower 0.5.3", + "tower", +] + +[[package]] +name = "openshell-gateway" +version = "0.0.0" +dependencies = [ + "async-trait", + "hyper-util", + "miette", + "nix 0.29.0", + "openshell-core", + "openshell-driver-docker", + "openshell-driver-kubernetes", + "openshell-driver-mxc", + "openshell-driver-podman", + "openshell-otel", + "openshell-server", + "rustix 1.1.4", + "serde", + "tempfile", + "tokio", + "tonic", + "tower", + "tracing", ] [[package]] name = "openshell-gateway-interceptors" version = "0.0.0" dependencies = [ - "json-patch", + "json-patch 1.4.0", "metrics", "openshell-core", "openshell-extension-core", @@ -4185,12 +4314,14 @@ dependencies = [ "serde_repr", "tracing", "tracing-subscriber", + "uuid", ] [[package]] name = "openshell-otel" version = "0.0.0" dependencies = [ + "futures", "http 1.4.0", "opentelemetry", "opentelemetry-otlp", @@ -4198,7 +4329,7 @@ dependencies = [ "thiserror 2.0.18", "tokio", "tonic", - "tower-http 0.6.8", + "tower-http", "tracing", "tracing-opentelemetry", "tracing-subscriber", @@ -4208,10 +4339,13 @@ dependencies = [ name = "openshell-otel-test-support" version = "0.0.0" dependencies = [ + "openshell-otel", "opentelemetry-proto", "tokio", "tokio-stream", "tonic", + "tracing", + "tracing-subscriber", ] [[package]] @@ -4220,11 +4354,11 @@ version = "0.0.0" dependencies = [ "hickory-proto", "miette", + "noyalib", "openshell-core", "prost-types", "serde", "serde_json", - "serde_yml", ] [[package]] @@ -4234,9 +4368,9 @@ dependencies = [ "glob", "include_dir", "miette", + "noyalib", "owo-colors", "serde", - "serde_yml", "z3", ] @@ -4245,33 +4379,15 @@ name = "openshell-providers" version = "0.0.0" dependencies = [ "glob", + "noyalib", "openshell-core", "openshell-policy", "serde", "serde_json", - "serde_yml", "thiserror 2.0.18", "url", ] -[[package]] -name = "openshell-router" -version = "0.0.0" -dependencies = [ - "bytes", - "openshell-core", - "reqwest 0.12.28", - "serde", - "serde_json", - "serde_yml", - "tempfile", - "thiserror 2.0.18", - "tokio", - "tracing", - "uuid", - "wiremock", -] - [[package]] name = "openshell-sandbox" version = "0.0.0" @@ -4290,7 +4406,7 @@ dependencies = [ "openshell-supervisor-process", "prost", "prost-types", - "rustls 0.23.38", + "rustls", "serde", "serde_json", "temp-env", @@ -4310,23 +4426,23 @@ version = "0.0.0" dependencies = [ "async-trait", "futures", - "hyper 1.9.0", + "hyper", "hyper-util", "jsonwebtoken", "miette", "oauth2", "openshell-core", "reqwest 0.12.28", - "rustls 0.23.38", + "rustls", "serde", "serde_json", "thiserror 2.0.18", "tokio", - "tokio-rustls 0.26.4", + "tokio-rustls", "tokio-stream", "tokio-tungstenite 0.26.2", "tonic", - "tower 0.5.3", + "tower", "tracing", "wiremock", ] @@ -4341,7 +4457,7 @@ dependencies = [ "aws-config", "aws-sdk-sts", "axum", - "base64 0.22.1", + "base64", "bytes", "clap", "futures", @@ -4352,8 +4468,8 @@ dependencies = [ "http 1.4.0", "http-body 1.0.1", "http-body-util", - "hyper 1.9.0", - "hyper-rustls 0.27.9", + "hyper", + "hyper-rustls", "hyper-util", "ipnet", "jsonwebtoken", @@ -4367,11 +4483,7 @@ dependencies = [ "openshell-bootstrap", "openshell-core", "openshell-driver-db-credstore", - "openshell-driver-docker", - "openshell-driver-kubernetes", "openshell-driver-kubernetes-secrets", - "openshell-driver-mxc", - "openshell-driver-podman", "openshell-driver-vault", "openshell-extension-core", "openshell-gateway-interceptors", @@ -4380,7 +4492,6 @@ dependencies = [ "openshell-policy", "openshell-prover", "openshell-providers", - "openshell-router", "openshell-supervisor-middleware", "openshell-supervisor-middleware-builtins", "opentelemetry", @@ -4390,31 +4501,33 @@ dependencies = [ "prost", "prost-reflect", "prost-types", + "protoc-bin-vendored", "rand 0.9.4", "rcgen", "reqwest 0.12.28", - "ring", "rsa 0.9.10", "russh", "rustix 1.1.4", - "rustls 0.23.38", + "rustls", "rustls-pemfile", "serde", "serde_json", "sha2 0.10.9", - "socket2 0.6.3", + "socket2", "spiffe", "sqlx", + "sqlx-core", "tempfile", "thiserror 2.0.18", "tokio", - "tokio-rustls 0.26.4", + "tokio-rustls", "tokio-stream", "tokio-tungstenite 0.26.2", "toml", "tonic", - "tower 0.5.3", - "tower-http 0.6.8", + "tonic-prost-build", + "tower", + "tower-http", "tracing", "tracing-opentelemetry", "tracing-subscriber", @@ -4473,7 +4586,7 @@ dependencies = [ "aws-credential-types", "aws-sigv4", "aws-smithy-runtime-api", - "base64 0.22.1", + "base64", "bytes", "flate2", "futures", @@ -4484,22 +4597,21 @@ dependencies = [ "ipnet", "libc", "miette", + "noyalib", "openshell-core", "openshell-ocsf", "openshell-policy", - "openshell-router", "openshell-supervisor-middleware", "openshell-supervisor-middleware-builtins", "prost-types", "rcgen", "regorus", "reqwest 0.12.28", - "rustls 0.23.38", + "rustls", "rustls-native-certs", "rustls-pemfile", "serde", "serde_json", - "serde_yml", "sha1 0.10.6", "sha2 0.10.9", "spiffe", @@ -4507,7 +4619,7 @@ dependencies = [ "tempfile", "thiserror 2.0.18", "tokio", - "tokio-rustls 0.26.4", + "tokio-rustls", "tokio-stream", "tokio-tungstenite 0.26.2", "tonic", @@ -4523,7 +4635,7 @@ name = "openshell-supervisor-process" version = "0.0.0" dependencies = [ "anyhow", - "base64 0.22.1", + "base64", "bytes", "capctl", "hex", @@ -4541,7 +4653,7 @@ dependencies = [ "seccompiler", "serde_json", "sha2 0.10.9", - "socket2 0.6.3", + "socket2", "tempfile", "tokio", "tokio-stream", @@ -4555,7 +4667,7 @@ dependencies = [ name = "openshell-tui" version = "0.0.0" dependencies = [ - "base64 0.22.1", + "base64", "crossterm 0.28.1", "indexmap", "miette", @@ -4830,7 +4942,7 @@ version = "3.0.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1d30c53c26bc5b31a98cd02d20f25a7c8567146caf63ed593a9d87b2775291be" dependencies = [ - "base64 0.22.1", + "base64", "serde_core", ] @@ -5233,7 +5345,7 @@ version = "0.16.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "01b80ea363c31af2de2b92e3c07ed1156628f7838c4afb4df75ee78a37fedbd1" dependencies = [ - "base64 0.22.1", + "base64", "prost", "prost-types", "serde", @@ -5360,8 +5472,8 @@ dependencies = [ "quinn-proto", "quinn-udp", "rustc-hash 2.1.2", - "rustls 0.23.38", - "socket2 0.6.3", + "rustls", + "socket2", "thiserror 2.0.18", "tokio", "tracing", @@ -5382,7 +5494,7 @@ dependencies = [ "rand_pcg", "ring", "rustc-hash 2.1.2", - "rustls 0.23.38", + "rustls", "rustls-pki-types", "slab", "thiserror 2.0.18", @@ -5400,7 +5512,7 @@ dependencies = [ "cfg_aliases", "libc", "once_cell", - "socket2 0.6.3", + "socket2", "tracing", "windows-sys 0.60.2", ] @@ -5555,8 +5667,8 @@ version = "0.13.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "75e669e5202259b5314d1ea5397316ad400819437857b90861765f24c4cf80a2" dependencies = [ + "aws-lc-rs", "pem", - "ring", "rustls-pki-types", "time", "yasna", @@ -5640,7 +5752,7 @@ version = "0.12.28" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "eddd3ca559203180a307f12d114c268abf583f59b03cb906fd0b3ff8646c1147" dependencies = [ - "base64 0.22.1", + "base64", "bytes", "futures-channel", "futures-core", @@ -5648,15 +5760,14 @@ dependencies = [ "http 1.4.0", "http-body 1.0.1", "http-body-util", - "hyper 1.9.0", - "hyper-rustls 0.27.9", + "hyper", + "hyper-rustls", "hyper-util", "js-sys", "log", "percent-encoding", "pin-project-lite", - "quinn", - "rustls 0.23.38", + "rustls", "rustls-native-certs", "rustls-pki-types", "serde", @@ -5664,9 +5775,9 @@ dependencies = [ "serde_urlencoded", "sync_wrapper", "tokio", - "tokio-rustls 0.26.4", - "tower 0.5.3", - "tower-http 0.6.8", + "tokio-rustls", + "tower", + "tower-http", "tower-service", "url", "wasm-bindgen", @@ -5680,22 +5791,22 @@ version = "0.13.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ab3f43e3283ab1488b624b44b0e988d0acea0b3214e694730a055cb6b2efa801" dependencies = [ - "base64 0.22.1", + "base64", "bytes", "futures-core", "futures-util", "http 1.4.0", "http-body 1.0.1", "http-body-util", - "hyper 1.9.0", - "hyper-rustls 0.27.9", + "hyper", + "hyper-rustls", "hyper-util", "js-sys", "log", "percent-encoding", "pin-project-lite", "quinn", - "rustls 0.23.38", + "rustls", "rustls-pki-types", "rustls-platform-verifier", "serde", @@ -5703,10 +5814,10 @@ dependencies = [ "serde_urlencoded", "sync_wrapper", "tokio", - "tokio-rustls 0.26.4", + "tokio-rustls", "tokio-util", - "tower 0.5.3", - "tower-http 0.6.8", + "tower", + "tower-http", "tower-service", "url", "wasm-bindgen", @@ -5958,18 +6069,6 @@ dependencies = [ "windows-sys 0.61.2", ] -[[package]] -name = "rustls" -version = "0.21.12" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3f56a14d1f48b391359b22f731fd4bd7e43c97f3c50eee276f3aa09c94784d3e" -dependencies = [ - "log", - "ring", - "rustls-webpki 0.101.7", - "sct", -] - [[package]] name = "rustls" version = "0.23.38" @@ -5979,9 +6078,8 @@ dependencies = [ "aws-lc-rs", "log", "once_cell", - "ring", "rustls-pki-types", - "rustls-webpki 0.103.13", + "rustls-webpki", "subtle", "zeroize", ] @@ -6028,10 +6126,10 @@ dependencies = [ "jni 0.21.1", "log", "once_cell", - "rustls 0.23.38", + "rustls", "rustls-native-certs", "rustls-platform-verifier-android", - "rustls-webpki 0.103.13", + "rustls-webpki", "security-framework", "security-framework-sys", "webpki-root-certs", @@ -6044,16 +6142,6 @@ version = "0.1.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f87165f0995f63a9fbeea62b64d10b4d9d8e78ec6d7d51fb2125fda7bb36788f" -[[package]] -name = "rustls-webpki" -version = "0.101.7" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8b6275d1ee7a1cd780b64aca7726599a1dbc893b1e64144529e55c3c2f745765" -dependencies = [ - "ring", - "untrusted 0.9.0", -] - [[package]] name = "rustls-webpki" version = "0.103.13" @@ -6148,16 +6236,6 @@ dependencies = [ "sha2 0.11.0", ] -[[package]] -name = "sct" -version = "0.7.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "da046153aa2352493d6cb7da4b6e5c0c057d8a1d0a9aa8560baffdd945acd414" -dependencies = [ - "ring", - "untrusted 0.9.0", -] - [[package]] name = "sec1" version = "0.7.3" @@ -6197,11 +6275,10 @@ dependencies = [ [[package]] name = "secrecy" -version = "0.8.0" +version = "0.10.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9bd1c54ea06cfd2f6b63219704de0b9b4f72dcc2b8fdef820be6cd799780e91e" +checksum = "e891af845473308773346dc847b2c23ee78fe442e0472ac50e22a18a93d3ae5a" dependencies = [ - "serde", "zeroize", ] @@ -6354,21 +6431,6 @@ dependencies = [ "unsafe-libyaml", ] -[[package]] -name = "serde_yml" -version = "0.0.12" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "59e2dd588bf1597a252c3b920e0143eb99b0f76e4e082f4c92ce34fbc9e71ddd" -dependencies = [ - "indexmap", - "itoa", - "libyml", - "memchr", - "ryu", - "serde", - "version_check", -] - [[package]] name = "serdect" version = "0.4.2" @@ -6578,16 +6640,6 @@ dependencies = [ "serde", ] -[[package]] -name = "socket2" -version = "0.5.10" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e22376abed350d73dd1cd119b57ffccad95b4e585a7cda43e286245ce23c0678" -dependencies = [ - "libc", - "windows-sys 0.52.0", -] - [[package]] name = "socket2" version = "0.6.3" @@ -6621,7 +6673,7 @@ dependencies = [ "tokio-util", "tonic", "tonic-prost", - "tower 0.5.3", + "tower", "tracing", "url", "zeroize", @@ -6664,9 +6716,9 @@ checksum = "3a0219bd7d979d58245a4f41f695e1ac9f8befdffadd7f61f1bae9e39abc6620" [[package]] name = "sqlx" -version = "0.8.6" +version = "0.9.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1fefb893899429669dcdd979aff487bd78f4064e5e7907e4269081e0ef7d97dc" +checksum = "378620ccc25c62c89d8be1c819e76a88d59bdcc3304733330788948e619bfd71" dependencies = [ "sqlx-core", "sqlx-macros", @@ -6677,12 +6729,13 @@ dependencies = [ [[package]] name = "sqlx-core" -version = "0.8.6" +version = "0.9.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ee6798b1838b6a0f69c007c133b8df5866302197e404e8b6ee8ed3e3a5e68dc6" +checksum = "05b44e85bf579a8eeb4ceaa77a3a523baf2bf0e9bac7e40f405d537b5d2d5ccb" dependencies = [ - "base64 0.22.1", + "base64", "bytes", + "cfg-if", "crc", "crossbeam-queue", "either", @@ -6691,14 +6744,13 @@ dependencies = [ "futures-intrusive", "futures-io", "futures-util", - "hashbrown 0.15.5", + "hashbrown 0.16.1", "hashlink", "indexmap", "log", "memchr", - "once_cell", "percent-encoding", - "rustls 0.23.38", + "rustls", "rustls-native-certs", "serde", "serde_json", @@ -6709,13 +6761,14 @@ dependencies = [ "tokio-stream", "tracing", "url", + "webpki-roots", ] [[package]] name = "sqlx-macros" -version = "0.8.6" +version = "0.9.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a2d452988ccaacfbf5e0bdbc348fb91d7c8af5bee192173ac3636b5fb6e6715d" +checksum = "bd2b84f2bc39a5705ef27ec785a11c934a41bbd4a24941e257927cddc26b60bf" dependencies = [ "proc-macro2", "quote", @@ -6726,15 +6779,15 @@ dependencies = [ [[package]] name = "sqlx-macros-core" -version = "0.8.6" +version = "0.9.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "19a9c1841124ac5a61741f96e1d9e2ec77424bf323962dd894bdb93f37d5219b" +checksum = "fb8d96de5fdc85a5c4ec813432b523ec637e80ba98f046555f75f7908ddac7c3" dependencies = [ + "cfg-if", "dotenvy", "either", "heck", "hex", - "once_cell", "proc-macro2", "quote", "serde", @@ -6744,59 +6797,45 @@ dependencies = [ "sqlx-postgres", "sqlx-sqlite", "syn 2.0.117", + "thiserror 2.0.18", "tokio", "url", ] [[package]] name = "sqlx-mysql" -version = "0.8.6" +version = "0.9.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "aa003f0038df784eb8fecbbac13affe3da23b45194bd57dba231c8f48199c526" +checksum = "90b8020fe17c5f2c245bfa2505d7ef59c5604839527c740266ad2214acebea27" dependencies = [ - "atoi", - "base64 0.22.1", "bitflags 2.11.1", "byteorder", "bytes", "crc", - "digest 0.10.7", + "digest 0.11.2", "dotenvy", "either", - "futures-channel", "futures-core", - "futures-io", "futures-util", "generic-array 0.14.7", - "hex", - "hkdf 0.12.4", - "hmac 0.12.1", - "itoa", "log", - "md-5", - "memchr", - "once_cell", "percent-encoding", - "rand 0.8.6", - "rsa 0.9.10", - "sha1 0.10.6", - "sha2 0.10.9", - "smallvec", + "serde", + "sha1 0.11.0", + "sha2 0.11.0", "sqlx-core", - "stringprep", "thiserror 2.0.18", "tracing", - "whoami", ] [[package]] name = "sqlx-postgres" -version = "0.8.6" +version = "0.9.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "db58fcd5a53cf07c184b154801ff91347e4c30d17a3562a635ff028ad5deda46" +checksum = "87a2bdd6e83f6b3ea525ca9fee568030508b58355a43d0b2c1674d5f79dcd65e" dependencies = [ "atoi", - "base64 0.22.1", + "base64", "bitflags 2.11.1", "byteorder", "crc", @@ -6806,18 +6845,16 @@ dependencies = [ "futures-core", "futures-util", "hex", - "hkdf 0.12.4", - "hmac 0.12.1", - "home", + "hkdf 0.13.0", + "hmac 0.13.0", "itoa", "log", "md-5", "memchr", - "once_cell", - "rand 0.8.6", + "rand 0.10.2", "serde", "serde_json", - "sha2 0.10.9", + "sha2 0.11.0", "smallvec", "sqlx-core", "stringprep", @@ -6828,12 +6865,13 @@ dependencies = [ [[package]] name = "sqlx-sqlite" -version = "0.8.6" +version = "0.9.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c2d12fe70b2c1b4401038055f90f151b78208de1f9f89a7dbfd41587a10c3eea" +checksum = "488e99c397a62007e4229aec669a179816339afc6d2620ca6fa420dbee2e982c" dependencies = [ "atoi", "flume", + "form_urlencoded", "futures-channel", "futures-core", "futures-executor", @@ -6843,7 +6881,6 @@ dependencies = [ "log", "percent-encoding", "serde", - "serde_urlencoded", "sqlx-core", "thiserror 2.0.18", "tracing", @@ -7023,9 +7060,9 @@ checksum = "a7973cce6668464ea31f176d85b13c7ab3bba2cb3b77a2ed26abd7801688010a" [[package]] name = "syn" -version = "1.0.109" +version = "2.0.117" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "72b64191b275b66ffe2469e8af2c1cfe3bafa67b529ead792a6d0160888b4237" +checksum = "e665b8803e7b1d2a727f4023456bbbbe74da67099c585258af0ad9c5013b9b99" dependencies = [ "proc-macro2", "quote", @@ -7034,9 +7071,9 @@ dependencies = [ [[package]] name = "syn" -version = "2.0.117" +version = "3.0.5" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e665b8803e7b1d2a727f4023456bbbbe74da67099c585258af0ad9c5013b9b99" +checksum = "12df2e0110f65b775f769bb17ef989067a1d931b2eb822bd4346631eeada89f9" dependencies = [ "proc-macro2", "quote", @@ -7080,6 +7117,7 @@ version = "0.3.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "96374855068f47402c3121c6eed88d29cb1de8f3ab27090e273e420bdabcf050" dependencies = [ + "futures", "parking_lot", ] @@ -7265,7 +7303,7 @@ dependencies = [ "parking_lot", "pin-project-lite", "signal-hook-registry", - "socket2 0.6.3", + "socket2", "tokio-macros", "windows-sys 0.61.2", ] @@ -7281,23 +7319,13 @@ dependencies = [ "syn 2.0.117", ] -[[package]] -name = "tokio-rustls" -version = "0.24.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c28327cf380ac148141087fbfb9de9d7bd4e84ab5d2c28fbc911d753de8a7081" -dependencies = [ - "rustls 0.21.12", - "tokio", -] - [[package]] name = "tokio-rustls" version = "0.26.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1729aa945f29d91ba541258c8df89027d5792d85a8841fb65e8bf0f4ede4ef61" dependencies = [ - "rustls 0.23.38", + "rustls", "tokio", ] @@ -7320,11 +7348,11 @@ checksum = "7a9daff607c6d2bf6c16fd681ccb7eecc83e4e2cdc1ca067ffaadfca5de7f084" dependencies = [ "futures-util", "log", - "rustls 0.23.38", + "rustls", "rustls-native-certs", "rustls-pki-types", "tokio", - "tokio-rustls 0.26.4", + "tokio-rustls", "tungstenite 0.26.2", ] @@ -7403,24 +7431,24 @@ checksum = "ac2a5518c70fa84342385732db33fb3f44bc4cc748936eb5833d2df34d6445ef" dependencies = [ "async-trait", "axum", - "base64 0.22.1", + "base64", "bytes", - "h2 0.4.13", + "h2", "http 1.4.0", "http-body 1.0.1", "http-body-util", - "hyper 1.9.0", + "hyper", "hyper-timeout", "hyper-util", "percent-encoding", "pin-project", "rustls-native-certs", - "socket2 0.6.3", + "socket2", "sync_wrapper", "tokio", - "tokio-rustls 0.26.4", + "tokio-rustls", "tokio-stream", - "tower 0.5.3", + "tower", "tower-layer", "tower-service", "tracing", @@ -7476,23 +7504,6 @@ dependencies = [ "tonic", ] -[[package]] -name = "tower" -version = "0.4.13" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b8fa9be0de6cf49e536ce1851f987bd21a43b771b09473c3549a6c853db37c1c" -dependencies = [ - "futures-core", - "futures-util", - "pin-project", - "pin-project-lite", - "tokio", - "tokio-util", - "tower-layer", - "tower-service", - "tracing", -] - [[package]] name = "tower" version = "0.5.3" @@ -7512,39 +7523,22 @@ dependencies = [ "tracing", ] -[[package]] -name = "tower-http" -version = "0.5.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1e9cd434a998747dd2c4276bc96ee2e0c7a2eadf3cae88e52be55a05fa9053f5" -dependencies = [ - "base64 0.21.7", - "bitflags 2.11.1", - "bytes", - "http 1.4.0", - "http-body 1.0.1", - "http-body-util", - "mime", - "pin-project-lite", - "tower-layer", - "tower-service", - "tracing", -] - [[package]] name = "tower-http" version = "0.6.8" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d4e6559d53cc268e5031cd8429d05415bc4cb4aefc4aa5d6cc35fbf5b924a1f8" dependencies = [ + "base64", "bitflags 2.11.1", "bytes", "futures-util", "http 1.4.0", "http-body 1.0.1", "iri-string", + "mime", "pin-project-lite", - "tower 0.5.3", + "tower", "tower-layer", "tower-service", "tracing", @@ -7563,7 +7557,7 @@ version = "0.12.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "6511f1f32c7cb7fd4525edc0eb4dcf307db8f7eceb2833ab24a37b4cc10cda61" dependencies = [ - "base64 0.22.1", + "base64", "serde", "serde_json", "thiserror 2.0.18", @@ -7696,7 +7690,7 @@ dependencies = [ "httparse", "log", "rand 0.9.4", - "rustls 0.23.38", + "rustls", "rustls-pki-types", "sha1 0.10.6", "thiserror 2.0.18", @@ -7948,12 +7942,6 @@ dependencies = [ "wit-bindgen 0.51.0", ] -[[package]] -name = "wasite" -version = "0.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b8dad83b4f25e74f184f64c43b150b91efe7647395b42289f38e50566d82855b" - [[package]] name = "wasm-bindgen" version = "0.2.118" @@ -8096,13 +8084,9 @@ dependencies = [ [[package]] name = "whoami" -version = "1.6.1" +version = "2.1.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5d4a4db5077702ca3015d3d02d74974948aba2ad9e12ab7df718ee64ccd7e97d" -dependencies = [ - "libredox", - "wasite", -] +checksum = "626c4bac6755d76ffc12cb01b2eac751db1996b9e0041de9aa02c8c211ddc82c" [[package]] name = "winapi" @@ -8558,12 +8542,12 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "08db1edfb05d9b3c1542e521aea074442088292f00b5f28e435c714a98f85031" dependencies = [ "assert-json-diff", - "base64 0.22.1", + "base64", "deadpool", "futures", "http 1.4.0", "http-body-util", - "hyper 1.9.0", + "hyper", "hyper-util", "log", "once_cell", diff --git a/Cargo.toml b/Cargo.toml index 57b77d071..dcf698e2f 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -8,7 +8,7 @@ members = ["crates/*"] [workspace.package] version = "0.0.0" edition = "2024" -rust-version = "1.90" +rust-version = "1.94" license = "Apache-2.0" repository = "https://github.com/NVIDIA/OpenShell" @@ -35,10 +35,10 @@ http-body = "1.0" http-body-util = "0.1" # TLS -tokio-rustls = { version = "0.26", default-features = false, features = ["logging", "tls12", "ring"] } -rustls = { version = "0.23", default-features = false, features = ["std", "logging", "tls12", "ring"] } +tokio-rustls = { version = "0.26", default-features = false, features = ["logging", "tls12", "aws_lc_rs"] } +rustls = { version = "0.23", default-features = false, features = ["std", "logging", "tls12", "aws_lc_rs"] } rustls-pemfile = "2" -rcgen = { version = "0.13", features = ["crypto", "pem"] } +rcgen = { version = "0.13", default-features = false, features = ["crypto", "pem", "aws_lc_rs"] } webpki-roots = "1" rustls-native-certs = "0.8" @@ -80,18 +80,18 @@ socket2 = "0.6" # Serialization serde = { version = "1", features = ["derive"] } serde_json = "1" -serde_yml = "0.0.12" +serde_yml = { package = "noyalib", version = "0.0.28", default-features = false, features = ["std", "compat-serde-yaml"] } toml = "0.8" apollo-parser = "0.8.5" tower-mcp-types = "0.12.0" regex = "1" # HTTP client -reqwest = { version = "0.12", default-features = false, features = ["json", "rustls-tls-native-roots"] } +reqwest = { version = "0.12.28", default-features = false, features = ["json", "rustls-tls-native-roots-no-provider"] } # AWS SDK -aws-config = { version = "1", default-features = false, features = ["rustls", "rt-tokio", "behavior-version-latest"] } -aws-sdk-sts = { version = "1", default-features = false, features = ["rustls", "rt-tokio", "behavior-version-latest"] } +aws-config = { version = "1", default-features = false, features = ["default-https-client", "rt-tokio", "behavior-version-latest"] } +aws-sdk-sts = { version = "1", default-features = false, features = ["default-https-client", "rt-tokio", "behavior-version-latest"] } # WebSocket tokio-tungstenite = { version = "0.26", default-features = false, features = ["connect", "rustls-tls-native-roots"] } @@ -104,7 +104,7 @@ sha2 = "0.10" rand = "0.9" jsonwebtoken = { version = "10", features = ["aws_lc_rs"] } getrandom = "0.3" -ring = "0.17" +aws-lc-rs = "1.16" spiffe = { version = "0.15", default-features = false, features = ["workload-api-jwt", "jwt-verify-rust-crypto", "tracing"] } # Filesystem embedding @@ -124,12 +124,14 @@ url = "2" indexmap = "2" # Database -sqlx = { version = "0.8", default-features = false, features = ["runtime-tokio", "tls-rustls-ring-native-roots", "postgres", "sqlite", "migrate", "macros"] } +sqlx = { version = "0.9", default-features = false, features = ["runtime-tokio", "tls-rustls-aws-lc-rs", "postgres", "sqlite", "migrate", "macros"] } +# SQLx's facade couples native roots to ring; select native roots independently. +sqlx-core = { version = "0.9", default-features = false, features = ["rustls-native-certs"] } # Kubernetes -kube = { version = "0.90", default-features = false, features = ["client", "runtime", "derive", "rustls-tls"] } -kube-runtime = "0.90" -k8s-openapi = { version = "0.21.1", features = ["v1_26"] } +kube = { version = "0.99", default-features = false, features = ["client", "runtime", "derive", "rustls-tls", "aws-lc-rs"] } +kube-runtime = "0.99" +k8s-openapi = { version = "0.24", features = ["v1_29"] } # IDs uuid = { version = "1.10", features = ["v4"] } diff --git a/GOVERNANCE.md b/GOVERNANCE.md new file mode 100644 index 000000000..e2c7cfbbe --- /dev/null +++ b/GOVERNANCE.md @@ -0,0 +1,166 @@ +# OpenShell Project Governance + +OpenShell is dedicated to creating safe, private, policy-governed runtimes for autonomous AI agents. This document explains how the project is governed. + +## Values + +OpenShell and its leadership embrace the following values: + +- **Openness:** Communication and decision-making happen in the open and remain discoverable for future reference. As much as possible, discussions and work take place in public forums and repositories. +- **Fairness:** All stakeholders may provide feedback and submit contributions. The project considers contributions on their merits. +- **Community over product or company:** Sustaining and growing the community takes priority over shipping code or advancing one sponsor's organizational goals. Contributors participate as individuals. +- **Vendor neutrality:** No single organization controls project direction or decisions. Maintainer selection, roadmap priorities, and release decisions are based on the interests of the project rather than employer affiliation. +- **Inclusivity:** Different perspectives and skills make OpenShell better. The project provides a welcoming and respectful environment for participation. +- **Participation:** Responsibilities and privileges are earned through sustained participation, demonstrated judgment, and earned community trust. + +## Contributor Ladder + +OpenShell uses a contributor ladder to provide a path from participation to project leadership. Movement through the ladder reflects increasing trust, responsibility, and project-wide stewardship. Contributions may include code, documentation, design, testing, issue triage, reviews, release work, community support, and other work that advances the project. + +### Contributor + +A Contributor is anyone who participates constructively in the OpenShell community. + +Contributors are responsible for: + +- Following the project Code of Conduct and contribution guidelines. +- Taking responsibility for the correctness and provenance of their work, including work created with AI tools or agents. +- Engaging constructively with reviewers, users, and other contributors. + +Contributors may: + +- Participate in project discussions. +- Submit issues, designs, code, documentation, tests, and reviews. +- Help other contributors and users. + +### Reviewer + +A Reviewer is a trusted Contributor with demonstrated knowledge in one or more project areas. Reviewers help maintain quality and provide consistent, actionable feedback, but do not receive project-wide merge or voting rights by virtue of this role. + +Reviewers are responsible for: + +- Regularly reviewing contributions in their areas of knowledge. +- Reviewing for correctness, security, maintainability, tests, documentation, and consistency with project direction. +- Supporting new and occasional contributors and helping them improve useful contributions. +- Escalating decisions that require broader expertise or project-wide input. + +A Reviewer demonstrates: + +- A record of constructive, technically valuable reviews. +- A working knowledge of the relevant project areas and their operational context. +- Reliable collaboration and follow-through. +- Judgment exercised for the good of OpenShell, independent of employer, product, or team interests. + +An existing Maintainer may nominate a Contributor to become a Reviewer. A simple majority vote of the Maintainers approves the nomination. Maintainers may grant the GitHub permissions needed for the Reviewer's responsibilities. + +### Maintainer + +Maintainers have project-wide write, merge, and voting privileges. They are collectively responsible for steering OpenShell in a positive direction and for managing project resources and contributor access. The current Maintainers are listed in [MAINTAINERS.md](MAINTAINERS.md). + +Maintainer status is held by individual humans. AI agents, automated systems, service accounts, companies, and other organizations cannot be Maintainers or exercise Maintainer votes. + +Maintainers are responsible for: + +- Reviewing and merging contributions across the project. +- Protecting project quality, security, architectural coherence, and long-term maintainability. +- Participating in project-wide technical, governance, roadmap, and release decisions. +- Seeking review from the people most knowledgeable about affected areas. +- Mentoring Contributors, Reviewers, and prospective Maintainers. +- Supporting contributions that benefit the broader community, including work outside their employer's immediate priorities. +- Following through on regressions and other issues arising from decisions they approve. +- Representing OpenShell and its community responsibly. + +Maintainers may: + +- Approve and merge changes after applicable review and automated checks. +- Vote on project matters. +- Nominate Contributors for Reviewer or Maintainer roles. +- Represent OpenShell in public and communicate with the CNCF on behalf of the project. + +The collective team of Maintainers is the Maintainer Council, the governing body for OpenShell. + +## Becoming a Maintainer + +Maintainer status reflects established trust and readiness to steward the whole project. Activity metrics are an eligibility floor, not a scorecard, an automatic promotion mechanism, or sufficient evidence of readiness. + +A candidate must first meet all of the following minimum participation requirements: + +- At least three months of sustained participation in OpenShell discussions, contributions, reviews, or other project work. +- At least three substantial pull requests merged into the project. +- Constructive reviews of at least five substantial pull requests. +- Demonstrated contribution to the general core product through reviews and feedback on third-party contributions beyond a feature, product, or project primarily needed by the candidate or their organization. + +For these requirements, a substantial pull request is work that requires meaningful technical or project judgment. Mechanical changes, generated volume, lines changed, and a series of closely related pull requests do not by themselves demonstrate substance. Maintainers evaluate the scope, difficulty, quality, impact, and judgment shown by the work in context. + +Meeting the minimum participation requirements only establishes eligibility. A candidate must also demonstrate: + +- Deep understanding of OpenShell in one or more project areas, including how those areas interact with the broader platform. +- Sound technical and product judgment, including an ability to identify risk, evaluate tradeoffs, and preserve long-term maintainability. +- Consistently high-quality implementation or documentation work and technically useful reviews. +- Understanding of project architecture, contribution policies, testing, security expectations, review practices, and development workflows. +- Reliable collaboration, respectful communication, and follow-through. +- Support for new and occasional contributors. +- A community-first perspective and an ability to act independently of the candidate's employer, friends, team, product, or organizational priorities. +- Personal accountability for contributions and reviews, including the ability to explain, validate, and take responsibility for work produced with AI tools or agents. +- Direct participation in OpenShell community calls, sufficient for the community to know the person behind the contributions and for Maintainers to assess the candidate's understanding, judgment, communication, and accountability. + +The use of AI or agents neither qualifies nor disqualifies a candidate. The Maintainer Council evaluates the candidate's demonstrated understanding, judgment, accountability, community participation, and the value they add to the project's human review and decision-making capacity. Raw contribution velocity, lines of code, and activity counts beyond the eligibility floor are not proxies for those qualities. + +Community call attendance is qualitative evidence, not an attendance quota or substitute for the other requirements. Candidates who cannot reasonably attend the regular call because of time zone, accessibility, or similar constraints may arrange equivalent direct, synchronous participation with community members and Maintainers. Video participation is not required. + +An existing Maintainer must nominate a candidate by sending a message to the [project mailing list](mailto:openshell-maintainers@ai-openshell.org). A simple majority vote of all active Maintainers approves the nomination. Nominations are evaluated without prejudice to employer or demographics and should consider the organizational diversity of the Maintainer Council. + +Approved Maintainers receive the GitHub rights needed for the role, are added to [MAINTAINERS.md](MAINTAINERS.md), and are invited to Maintainer communication channels. + +## Removing a Maintainer + +A Maintainer may resign at any time when they can no longer fulfill the role. + +A Maintainer may also be removed for inactivity, persistent failure to fulfill Maintainer responsibilities, a Code of Conduct violation, behavior detrimental to the project, or other cause. Inactivity means six months of very low or no project activity without a definite plan to resume active participation. Non-code project work counts as activity. + +Removal requires a two-thirds vote of the remaining active Maintainers. A Maintainer whose removal is under consideration may participate in the discussion but does not vote on their own removal. + +### Emeritus Maintainers + +Depending on the circumstances of a resignation or removal, a Maintainer may move to Emeritus status. Emeritus Maintainers are recognized for past contributions and may be consulted on project matters, but they do not have voting rights or merge access. They are listed separately in [MAINTAINERS.md](MAINTAINERS.md). + +An Emeritus Maintainer may return to active status through the standard Maintainer nomination and voting process, provided they meet the current requirements and can commit to active participation. + +## Meetings and Communication + +Project governance ordinarily takes place through public GitHub issues, discussions, and pull requests so decisions are visible and discoverable. Anyone interested in OpenShell may join the [OpenShell Community Google Group](https://groups.google.com/a/ai-openshell.org/g/openshell-community/about) to receive the community meeting invitation and view the community meeting agenda. Consequential decisions made during a meeting must be recorded in a public project artifact. + +Maintainers may meet privately when handling security reports, Code of Conduct reports, personnel matters, or other sensitive information. All Maintainers must be invited except anyone whose participation would create a conflict of interest, including a Maintainer who is the subject of a report. + +## CNCF Resources + +Any Maintainer may propose a request for CNCF resources through the [project mailing list](mailto:openshell-maintainers@ai-openshell.org) or a project meeting. A simple majority vote of active Maintainers approves the request. The Maintainer Council may delegate CNCF coordination to a non-Maintainer community member and arrange the access needed for that work. + +## Code of Conduct + +OpenShell follows the [CNCF Code of Conduct](https://github.com/cncf/foundation/blob/main/code-of-conduct.md). The Maintainer Council handles reports privately and coordinates with the CNCF Code of Conduct Committee when appropriate. If a Maintainer is directly involved in a report, that Maintainer is excluded from its handling, and the remaining Maintainers designate at least two uninvolved Maintainers to coordinate the response. + +## Security Response + +Security reports are handled according to the project [security policy](SECURITY.md). The Maintainer Council appoints or recognizes the people responsible for coordinating security responses and reviews the response process and membership at least annually. + +## Voting + +OpenShell uses [lazy consensus](https://community.apache.org/committers/decisionMaking.html#lazy-consensus) for most decisions: a proposal proceeds when there is general agreement and no unresolved, reasoned objection. Maintainers should allow time appropriate to the impact and urgency of a decision so that affected contributors across time zones can participate. + +Any Maintainer may call for a formal vote on the [project mailing list](mailto:openshell-maintainers@ai-openshell.org) or at a project meeting. Unless this document states otherwise, a proposal requires a simple majority of all active Maintainers to pass. A two-thirds vote requires support from at least two-thirds of all active Maintainers. Maintainers must disclose relevant conflicts of interest and recuse themselves when they cannot act solely in the project's interest. + +## Evolving This Governance + +The Maintainer Council should revisit this model as OpenShell grows. Signals that the project may need working groups, area-specific approvers, subproject governance, organization-balanced voting, or an elected steering body include: + +- Decisions routinely stall because the Maintainer Council is too large or different project areas need delegated authority. +- Contributors cannot find a clear path to greater responsibility. +- One organization dominates project decisions or Maintainer membership. +- Subprojects develop distinct contributor communities or release cadences. + +Such changes are a sign of project growth rather than governance failure. + +## Modifying This Governance + +Changes to this governance document and its supporting governance documents require a two-thirds vote of all active Maintainers. diff --git a/MAINTAINERS.md b/MAINTAINERS.md new file mode 100644 index 000000000..3ecde4ded --- /dev/null +++ b/MAINTAINERS.md @@ -0,0 +1,19 @@ +# Maintainers + +The current maintainers of OpenShell are listed below. + +| Name | GitHub ID | Company/Organization | +| --- | --- | --- | +| Derek Carr | [@derekwaynecarr](https://github.com/derekwaynecarr) | Red Hat | +| Matthew Grossman | [@matthewgrossman](https://github.com/matthewgrossman) | NVIDIA | +| Kris Hicks | [@krishicks](https://github.com/krishicks) | NVIDIA | +| Seth Jennings | [@sjenning](https://github.com/sjenning) | Red Hat | +| Adrien Langou | [@alangou](https://github.com/alangou) | NVIDIA | +| Evan Lezar | [@elezar](https://github.com/elezar) | NVIDIA | +| Jim Meyer | [@purp](https://github.com/purp) | NVIDIA | +| Piotr Mlocek | [@pimlock](https://github.com/pimlock) | NVIDIA | +| Taylor Mutch | [@TaylorMutch](https://github.com/TaylorMutch) | NVIDIA | +| John T. Myers | [@johntmyers](https://github.com/johntmyers) | NVIDIA | +| Drew Newberry | [@drew](https://github.com/drew) | NVIDIA | +| Mrunal Patel | [@mrunalp](https://github.com/mrunalp) | Red Hat | +| Simon Scatton | [@SDAChess](https://github.com/SDAChess) | NVIDIA | diff --git a/README.md b/README.md index ba1ccf985..35faf5c1b 100644 --- a/README.md +++ b/README.md @@ -16,7 +16,7 @@ OpenShell is the safe, private runtime for autonomous AI agents. It provides sandboxed execution environments that protect your data, credentials, and infrastructure — governed by declarative YAML policies that prevent unauthorized file access, data exfiltration, and uncontrolled network activity. -OpenShell is built agent-first. The project ships with agent skills for everything from gateway troubleshooting to policy generation, and we expect contributors to use them. +OpenShell is built agent-first. It ships public agent skills for using and operating OpenShell, plus separate repository-aware workflows for contributors and maintainers. ## Quickstart @@ -108,7 +108,7 @@ bash examples/sandbox-policy-quickstart/demo.sh OpenShell isolates each sandbox in its own container with policy-enforced egress routing. A lightweight gateway coordinates sandbox lifecycle, and every outbound connection is intercepted by the policy engine, which does one of three things: - **Allows** — the destination and binary match a policy block. -- **Routes for inference** — strips caller credentials, injects backend credentials, and forwards to the managed model. +- **Binds credentials to endpoints** — injects provider credentials only after policy admits a request to a profile-authorized endpoint. - **Denies** — blocks the request and logs it. | Component | Role | @@ -116,7 +116,7 @@ OpenShell isolates each sandbox in its own container with policy-enforced egress | **Gateway** | Control-plane API that coordinates sandbox lifecycle and acts as the auth boundary. | | **Sandbox** | Isolated runtime with container supervision and policy-enforced egress routing. | | **Policy Engine** | Enforces filesystem, network, and process constraints from application layer down to kernel. | -| **Privacy Router** | Privacy-aware LLM routing that keeps sensitive context on sandbox compute. | +| **Provider Access** | Profile-defined endpoints, binary policy, and endpoint-bound credential injection for model APIs and other services. | OpenShell runs a gateway control plane that manages sandbox lifecycle through a configured compute driver. Supported compute platforms include Docker, Podman, MicroVM, and Kubernetes. @@ -129,14 +129,16 @@ OpenShell applies defense in depth across four policy domains: | Filesystem | Prevents reads/writes outside allowed paths. | Locked at sandbox creation. | | Network | Blocks unauthorized outbound connections. | Hot-reloadable at runtime. | | Process | Blocks privilege escalation and dangerous syscalls. | Locked at sandbox creation. | -| Inference | Reroutes model API calls to controlled backends. | Hot-reloadable at runtime. | +| Providers | Grants endpoint-bound credentials and network access. | Hot-reloadable at runtime. | -Policies are declarative YAML files. Static sections (filesystem, process) are locked at creation; dynamic sections (network, inference) can be hot-reloaded on a running sandbox with `openshell policy set`. +Policies are declarative YAML files. Static sections (filesystem, process) are locked at creation; network policy and provider attachments can be updated on a running sandbox. ## Providers Agents need credentials — API keys, tokens, service accounts. OpenShell manages these as **providers**: named credential bundles that are injected into sandboxes at creation. The CLI auto-discovers credentials for recognized agents (Claude, Codex, OpenCode, Copilot) from your shell environment, or you can create providers explicitly with `openshell provider create`. Credentials never leak into the sandbox filesystem; they are injected as environment variables at runtime. +Inference access uses the same provider workflow. Attach an inference-capable provider to a sandbox, call the provider's native endpoint, and select the model in the client. Provider profiles contribute the endpoint policy and bind credential placeholders to the authorized destination. + ## GPU Support (Experimental) > **Experimental** — GPU passthrough works on supported hosts but is under active development. Expect rough edges and breaking changes. @@ -159,8 +161,8 @@ Docker-backed GPU sandboxes auto-select CDI when available and otherwise fall ba | [OpenCode](https://opencode.ai/) | [`base`](https://github.com/NVIDIA/OpenShell-Community/tree/main/sandboxes/base) | Works out of the box. Provider uses `OPENAI_API_KEY` or `OPENROUTER_API_KEY`. | | [Codex](https://developers.openai.com/codex) | [`base`](https://github.com/NVIDIA/OpenShell-Community/tree/main/sandboxes/base) | Works out of the box. Provider uses `OPENAI_API_KEY`. | | [GitHub Copilot CLI](https://docs.github.com/en/copilot/github-copilot-in-the-cli) | [`base`](https://github.com/NVIDIA/OpenShell-Community/tree/main/sandboxes/base) | Works out of the box. Provider uses `GITHUB_TOKEN` or `COPILOT_GITHUB_TOKEN`. | -| [OpenClaw](https://openclaw.ai/) | [NemoClaw](https://github.com/NVIDIA/NemoClaw) | Run OpenClaw more securely inside NVIDIA OpenShell with managed inference using NemoClaw. | -| [Hermes Agent](https://github.com/NousResearch/hermes-agent) | [NemoClaw](https://github.com/NVIDIA/NemoClaw) | Run Hermes Agent more securely inside NVIDIA OpenShell with managed inference using NemoClaw. | +| [OpenClaw](https://openclaw.ai/) | [NemoClaw](https://github.com/NVIDIA/NemoClaw) | Run OpenClaw more securely inside NVIDIA OpenShell with the NemoClaw blueprint. | +| [Hermes Agent](https://github.com/NousResearch/hermes-agent) | [NemoClaw](https://github.com/NVIDIA/NemoClaw) | Run Hermes Agent more securely inside NVIDIA OpenShell with the NemoClaw blueprint. | | [Ollama](https://ollama.com/) | [Community](https://github.com/NVIDIA/OpenShell-Community) | Launch with `openshell sandbox create --from ollama`. | | [Pi](https://pi.dev/) | [Community](https://github.com/NVIDIA/OpenShell-Community) | Launch with `openshell sandbox create --from pi`. | @@ -172,9 +174,9 @@ Docker-backed GPU sandboxes auto-select CDI when available and otherwise fall ba | `openshell sandbox connect [name]` | SSH into a running sandbox. | | `openshell sandbox list` | List all sandboxes. | | `openshell provider create --type [type] --from-existing` | Create a credential provider from env vars. | +| `openshell sandbox provider attach ` | Attach a provider to a running sandbox. | | `openshell policy set --policy file.yaml` | Apply or update a policy on a running sandbox. | | `openshell policy get ` | Show the active policy. | -| `openshell inference set --provider

--model ` | Configure the `inference.local` endpoint. | | `openshell logs [name] --tail` | Stream sandbox logs. | | `openshell term` | Launch the real-time terminal UI for debugging. | @@ -196,45 +198,49 @@ The TUI gives you a live, keyboard-driven view of your gateway and sandboxes. Na ## Community Sandboxes and BYOC -Use `--from` to create sandboxes from the [OpenShell Community](https://github.com/NVIDIA/OpenShell-Community) catalog, a local directory, or a container image: +Use `--from` to create sandboxes from the [OpenShell Community](https://github.com/NVIDIA/OpenShell-Community) catalog or a container image: ```bash openshell sandbox create --from gemini # community catalog -openshell sandbox create --from ./my-sandbox-dir # local Dockerfile +docker build -t my-sandbox:latest ./my-sandbox-dir # Docker gateway +openshell sandbox create --from my-sandbox:latest # Docker built image +podman build -t localhost/my-sandbox:latest ./my-sandbox-dir # Podman gateway +openshell sandbox create --from localhost/my-sandbox:latest # Podman built image openshell sandbox create --from registry.io/img:v1 # container image ``` +Build with the container engine used by your local gateway. For a remote +gateway, push the image to a registry that the gateway can pull from. + See the [OpenShell Community](https://github.com/NVIDIA/OpenShell-Community) catalog and the [BYOC example](https://github.com/NVIDIA/OpenShell/tree/main/examples/bring-your-own-container) for details. -## Explore with Your Agent +## Use OpenShell with Your Agent -Clone the repo and point your coding agent at it. The project includes agent skills that can answer questions, walk you through workflows, and diagnose problems — no issue filing required. +OpenShell provides four portable skills for users and operators: CLI workflows (`openshell-cli`), gateway troubleshooting (`debug-openshell-cluster`), inference troubleshooting (`debug-inference`), and policy generation (`generate-sandbox-policy`). Install them with the Agent Skills CLI: ```bash -git clone https://github.com/NVIDIA/OpenShell.git # or git@github.com:NVIDIA/OpenShell.git -cd OpenShell -# Point your agent here — it will discover the skills in .agents/skills/ automatically +npx skills add NVIDIA/OpenShell ``` -Your agent can load skills for CLI usage (`openshell-cli`), gateway troubleshooting (`debug-openshell-cluster`), inference troubleshooting (`debug-inference`), policy generation (`generate-sandbox-policy`), and more. See [CONTRIBUTING.md](CONTRIBUTING.md) for the full skills table. +These public, installable skills live in [`skills/`](skills/) and use the installed CLI help and [published documentation](https://docs.nvidia.com/openshell/latest/index.html) as their sources of truth. They do not require an OpenShell source checkout. ## Built With Agents -OpenShell is developed using the same agent-driven workflows it enables. The `.agents/skills/` directory contains workflow automation that powers the project's development cycle: +OpenShell is developed using the same agent-driven workflows it enables. Contributor and maintainer skills live separately in [`.agents/skills/`](.agents/skills/); they automate work on the OpenShell repository and are not included when users install the public skills: - **Spike and build:** Investigate a problem with `create-spike`; a human accepts it with `state:accepted` or [roadmap](https://github.com/orgs/NVIDIA/projects/233) placement, or declines it. Accepted work can remain human-owned or enter the optional, human-gated `agent:*` planning and implementation workflow. - **Triage and route:** Community issues are assessed with `triage-issue`. Agents establish technical validity and impact; humans decide whether the project should act and where the work sits on the roadmap. - **Security review:** `review-security-issue` produces a severity assessment and remediation plan. `fix-security-issue` implements it. -- **Policy authoring:** `generate-sandbox-policy` creates YAML policies from plain-language requirements or API documentation. +- **Repository maintenance:** `sync-agent-infra`, `update-docs-from-commits`, and other internal workflows keep code, documentation, and agent infrastructure consistent. -All agent implementation work is human-gated: maintainers explicitly request a plan, agents propose it, maintainers approve it, and agents build. See [AGENTS.md](AGENTS.md) for the full workflow chain documentation. +Agent implementation is human-directed: a user may request a phase directly, or maintainers may use the optional `agent:*` workflow to queue and approve planning and implementation. See [AGENTS.md](AGENTS.md) for the full workflow chain documentation. ## Getting Help - **Questions and discussion:** [GitHub Discussions](https://github.com/NVIDIA/OpenShell/discussions) - **Bug reports:** [GitHub Issues](https://github.com/NVIDIA/OpenShell/issues) — use the bug report template - **Security vulnerabilities:** See [SECURITY.md](SECURITY.md) — do not use GitHub Issues -- **Agent-assisted help:** Clone the repo and use the agent skills in `.agents/skills/` for self-service diagnostics +- **Agent-assisted help:** Install the public OpenShell skills with `npx skills add NVIDIA/OpenShell` ## Learn More @@ -250,7 +256,7 @@ All agent implementation work is human-gated: maintainers explicitly request a p ## Contributing -OpenShell is built agent-first. Issues should include a user story, problem statement, impact, and acceptance criteria. The impact should explain the consequences of the current behavior and why existing workarounds are insufficient. Feature requests also require a workflow-level proposed design and alternatives; bug reports add reproduction steps, environment details, and relevant logs. Once maintainers accept work, contributors should use the skills in `.agents/skills/` to investigate the current code and behavior, implement the change, and verify it. If an issue contains earlier diagnostics, verify them rather than relying on them. See [CONTRIBUTING.md](CONTRIBUTING.md) for the full agent skills table, contribution workflow, and development setup. +OpenShell is built agent-first. Issues should include a user story, problem statement, impact, and acceptance criteria. The impact should explain the consequences of the current behavior and why existing workarounds are insufficient. Feature requests also require a workflow-level proposed design and alternatives; bug reports add reproduction steps, environment details, and relevant logs. Once work is authorized through the project workflow or a direct request, contributors should use the skills in `.agents/skills/` to investigate the current code and behavior, implement the change, and verify it. If an issue contains earlier diagnostics, verify them rather than relying on them. See [CONTRIBUTING.md](CONTRIBUTING.md) for the full agent skills table, contribution workflow, and development setup. ## Telemetry @@ -258,7 +264,15 @@ OpenShell collects anonymous telemetry to help improve the project for developer Disable telemetry at runtime by setting `OPENSHELL_TELEMETRY_ENABLED=false` on the gateway deployment. For Helm installs, set `server.telemetryEnabled=false`. OpenShell propagates this deployment setting into sandbox supervisor environments so sandbox-side telemetry collection is disabled as well. -You can also compile telemetry out entirely. Telemetry support is a default-on `telemetry` Cargo feature; building with `--no-default-features` produces binaries that contain no telemetry endpoint, no telemetry HTTP client, and no emission code. Build telemetry-free artifacts with, for example, `cargo build --release -p openshell-server --no-default-features` (gateway) and the equivalent for `openshell-sandbox` and `openshell-driver-vm`. With telemetry compiled out, the gateway emits nothing and reports telemetry disabled to the sandboxes it launches. +You can also compile telemetry out entirely. Telemetry support is a default-on `telemetry` Cargo feature, and each crate that carries it also defines a `defaults-without-telemetry` alias covering every other default feature. Build telemetry-free artifacts with `--no-default-features --features defaults-without-telemetry`: + +```shell +cargo build --release -p openshell-gateway --no-default-features --features defaults-without-telemetry +cargo build --release -p openshell-sandbox --no-default-features --features defaults-without-telemetry +cargo build --release -p openshell-driver-vm --no-default-features --features defaults-without-telemetry +``` + +The resulting binaries contain no telemetry endpoint, no telemetry HTTP client, and no emission code. With telemetry compiled out, the gateway emits nothing and reports telemetry disabled to the sandboxes it launches. Cargo has no way to subtract a single default feature, so `defaults-without-telemetry` must be paired with `--no-default-features`; passing it on its own leaves the defaults in place and fails the build rather than producing a binary that still emits. Telemetry events are limited to anonymous operational categories and counts, such as sandbox lifecycle outcomes, provider profile buckets, policy decision counts, and aggregate network activity denial categories. OpenShell telemetry does not collect sandbox names or IDs, hostnames, file paths, binary paths, prompts, credentials, provider names, model names, or user content. diff --git a/TESTING.md b/TESTING.md index 6c0829060..5d9674dd0 100644 --- a/TESTING.md +++ b/TESTING.md @@ -136,8 +136,6 @@ def test_multiply(sandbox): |---|---|---| | `sandbox_client` | session | gRPC client connected to the active gateway | | `sandbox` | function | Factory returning a `Sandbox` context manager | -| `inference_client` | session | Client for managing inference routes | -| `mock_inference_route` | session | Creates a mock OpenAI-protocol route for tests | ### Rust CLI E2E (`e2e/rust/`) @@ -148,7 +146,12 @@ lifecycle management, output parsing, and cleanup. Suites: - Common suite (`--features e2e`) - driver-neutral CLI behavior, sandbox lifecycle, sync, port forwarding, policy, and provider tests. -- Docker suite (`--features e2e-docker`) - common suite plus Docker-only coverage such as Dockerfile image builds, Docker preflight checks, and managed Docker gateway start. +- CLI conformance (`openshell-conformance`) - the portable deployment smoke + scenario plus focused tests for its reusable command runner. +- Driver suites (`--features e2e-docker`, `e2e-podman`, `e2e-kubernetes`, or + `e2e-vm`) - CLI conformance plus the common and driver-specific coverage for + the selected deployment. +- Docker suite (`--features e2e-docker`) - includes Docker-only coverage such as Dockerfile image builds, Docker preflight checks, and managed Docker gateway start. - Docker GPU suite (`--features e2e-docker-gpu`) - Docker suite plus GPU sandbox smoke coverage. - VM suite (`--features e2e-vm`) - runs e2e tests on a VM. - Kubernetes credential-driver suite (`--features e2e-kubernetes-credential-drivers`) - targeted Kubernetes Secrets and Vault provider credential storage coverage. @@ -166,10 +169,48 @@ key. Run the Docker-backed Rust CLI e2e suite: ```shell -mise run e2e:rust +mise run e2e:docker ``` -Run the Podman-backed Rust CLI e2e suite: +Run the minimal portable CLI conformance profile against the gateway selected +in your OpenShell CLI configuration: + +```shell +mise run e2e:cli-conformance +``` + +The gateway must already be installed, reachable, and selected before the task +starts. The task does not provision a gateway or select a compute driver. Set +`OPENSHELL_BIN` to test a prebuilt CLI; otherwise, the task builds the CLI from +the current checkout. + +The phase-1 scenario verifies the complete CLI-to-gateway-to-driver path without +depending on how the gateway was installed or which driver is configured. It +requires machine-readable gRPC status, creates a uniquely named detached +sandbox with `--from base`, verifies the sandbox is `Ready` by finding its +unique name in paginated JSON list output, executes `echo` with a run-specific +marker, deletes the sandbox, and verifies that its name no longer appears. +Driver suites enable the same profile +instead of maintaining a separate smoke implementation. Sandbox lifecycle, +label matrices, VM overlay, and TLS-key permission assertions remain regular +E2E coverage. + +Each invocation prints a ten-character run ID before creating resources. +Conformance sandboxes use names such as `ct--01`. The runner tracks the +exact name and uses it for cleanup; phase 1 does not add ownership labels. + +The runner deletes owned resources after both success and failure. If the test +process is interrupted before cleanup, locate leftovers without touching +unrelated gateway state: + +```shell +openshell sandbox list --output json +openshell sandbox delete +``` + +Gateway-backed Rust E2E tasks build the standalone conformance CLI, run its +registered scenarios against the configured gateway, then run any lane-specific +Rust tests that still apply. Run the Podman-backed Rust CLI e2e suite: ```shell mise run e2e:podman @@ -189,6 +230,128 @@ Secrets and Vault storage backends one at a time: mise run e2e:kubernetes:credential-drivers ``` +### Kubernetes E2E (`e2e/rust/e2e-kubernetes.sh`) + +Kubernetes e2e tests deploy an OpenShell gateway into a real Kubernetes cluster +via Helm and run the Rust e2e suite against it. On vanilla Kubernetes the harness +reaches the gateway through `kubectl port-forward`; on OpenShift it instead uses a +passthrough Route secured with mandatory mTLS (see the OpenShift note below). + +Run with an ephemeral k3d cluster (macOS; created and torn down automatically): + +```shell +mise run e2e:kubernetes +``` + +Target an existing cluster (kind, k3d, or OpenShift): + +```shell +OPENSHELL_E2E_KUBE_CONTEXT=my-context mise run e2e:kubernetes +``` + +Scope to a single test for local debugging: + +```shell +OPENSHELL_E2E_KUBE_TEST=smoke mise run e2e:kubernetes +``` + +**OpenShift**: when the target cluster exposes the `route.openshift.io` API +group, the harness automatically applies SCC-compatible Helm overrides, grants +the required SCCs (`privileged` to `openshell-sandbox`, and `anyuid` to the +PostgreSQL fixture for DB scenarios), and drives the gateway through a +passthrough Route with mandatory mTLS instead of port-forward. No extra flags are +needed, but `oc` must be installed and authenticated against the target cluster +with permission to modify SCC bindings (`oc adm policy add-scc-to-user`) — the +harness exits early if `oc` is missing. The SCC grants and extracted client +mTLS material are removed during cleanup, including on failure or interrupt. + +On a **remote** cluster, drop the `e2e-host-gateway` feature. Those tests rely +on the sandbox-side `host.openshell.internal` alias reaching the machine running +the tests, which is unreachable from pods on a remote cluster, so they fail. +Left enabled, the `host_gateway_alias` suite fails because +`host.openshell.internal` does not resolve inside the pod, so the gateway +SSRF-denies the request (`DNS resolution failed` / `ssrf_denied`) — a networking +property of remote pods, not a gateway or transport fault. Override +`OPENSHELL_E2E_KUBERNETES_FEATURES` to exclude it: + +```shell +OPENSHELL_E2E_KUBE_CONTEXT=$(oc config current-context) \ + OPENSHELL_E2E_KUBERNETES_FEATURES="e2e,e2e-kubernetes" \ + mise run e2e:kubernetes +``` + +On an existing cluster the harness builds the CLI from your branch but pulls the +**published** gateway/supervisor image (default tag `latest`). The CLI and the +image can therefore be different versions. If tests fail because of this version +difference — for example, sandbox tests fail with `Pod exists with phase: Failed` +or connect-based tests stall because the deployed image predates a feature your +branch CLI needs — set `IMAGE_TAG` to an image that matches your branch. + +The `latest` tag lags to the last semver release, so it is often older than +`main`. Two better choices: + +- `IMAGE_TAG=dev` — a floating tag that tracks the latest `main` build. Good for + an ad-hoc run when your branch is close to `main` HEAD. Because it floats, two + runs on different days can pull different images, so it is not reproducible. +- **Pin the exact commit your branch is based on** — deterministic and immune to + a floating tag moving. Published tags are the full 40-char git SHA (semver tags + without a `v` prefix also exist but only for released versions): + +```shell +OPENSHELL_E2E_KUBE_CONTEXT=$(oc config current-context) \ + OPENSHELL_E2E_KUBERNETES_FEATURES="e2e,e2e-kubernetes" \ + IMAGE_TAG=$(git rev-parse "$(git merge-base HEAD upstream/main)") \ + mise run e2e:kubernetes +``` + +To pin a specific released version, use its semver tag without a `v` prefix +(`0.0.115`, not `v0.0.115`): + +```shell +OPENSHELL_E2E_KUBE_CONTEXT=$(oc config current-context) \ + OPENSHELL_E2E_KUBERNETES_FEATURES="e2e,e2e-kubernetes" \ + IMAGE_TAG=0.0.115 \ + mise run e2e:kubernetes +``` + +A semver tag matches a released commit, which may be behind `main`; if your +branch CLI needs a newer feature, pin the SHA of your branch's base instead. + +Confirm a tag exists before relying on it (set `TAG` to the tag you plan to use): + +```shell +TAG=0.0.115 +skopeo inspect "docker://ghcr.io/nvidia/openshell/gateway:${TAG}" +``` + +`IMAGE_TAG` sets only the gateway/supervisor image; the CLI under test is always +built from your branch. To validate against images from your exact commit +instead, build and push them and point `OPENSHELL_REGISTRY`/`IMAGE_TAG` at them. + +Available task variants: + +| Task | Purpose | +|---|---| +| `e2e:kubernetes` | Default Rust e2e against Helm-deployed gateway | +| `e2e:kubernetes:db` | All database backend scenarios (SQLite + external PostgreSQL) | +| `e2e:kubernetes:sidecar` | Supervisor sidecar topology overlay | +| `e2e:kubernetes:credential-drivers` | Kubernetes Secrets and Vault credential storage | +| `e2e:kubernetes:workspace-managed` | Managed workspace mode (auto-created namespaces) | +| `e2e:kubernetes:workspace-operator` | Operator workspace mode (pre-provisioned namespaces) | +| `e2e:kubernetes:v1alpha1` | Agent Sandbox v1alpha1 compatibility | +| `e2e:kubernetes:external-driver` | External Kubernetes driver sidecar | + +Kubernetes e2e environment variables: + +| Variable | Purpose | +|---|---| +| `OPENSHELL_E2E_KUBE_CONTEXT` | kubectl context for an existing cluster (skips k3d creation) | +| `OPENSHELL_E2E_KUBE_TEST` | Scope to a single test (e.g. `smoke`) | +| `OPENSHELL_E2E_KUBE_EXTRA_VALUES` | Colon-separated additional Helm values files | +| `OPENSHELL_E2E_KUBERNETES_FEATURES` | Cargo feature flags (default: `e2e,e2e-host-gateway,e2e-kubernetes`) | +| `IMAGE_TAG` | Gateway/supervisor image tag (default: `latest` for existing clusters) | +| `OPENSHELL_REGISTRY` | Image registry prefix (default: `ghcr.io/nvidia/openshell`) | + Run a single test directly with cargo: ```shell @@ -220,3 +383,5 @@ The harness (`e2e/rust/src/harness/`) provides: | `OPENSHELL_GATEWAY_ENDPOINT` | Run E2E tests against an existing plaintext HTTP gateway endpoint | | `OPENSHELL_E2E_DRIVER` | Driver name exported by the e2e gateway wrapper (`docker`, `podman`, or `vm`) | | `OPENSHELL_E2E_CREDENTIAL_DRIVERS` | Enables the Kubernetes credential-driver fixture path in `e2e/with-kube-gateway.sh` | +| `OPENSHELL_E2E_KUBE_CONTEXT` | kubectl context for Kubernetes e2e (skips ephemeral k3d) | +| `OPENSHELL_E2E_KUBE_TEST` | Scope Kubernetes e2e to a single test by name | diff --git a/architecture/README.md b/architecture/README.md index 0d814a7a2..706db67b1 100644 --- a/architecture/README.md +++ b/architecture/README.md @@ -7,7 +7,7 @@ the **Supervisor**. The CLI, SDK, and TUI provide user-facing access. The gateway is the authenticated control plane: it owns API access, durable state, policy and -settings delivery, provider and inference configuration, and relay +settings delivery, provider configuration and attachments, and relay coordination. The supervisor runs inside every sandbox workload and is the local security boundary. It launches the agent as a restricted child process and enforces policy where process identity, filesystem access, network egress, and @@ -50,7 +50,6 @@ flowchart TB subgraph DP["Sandbox Data Plane"] SUP["Supervisor"] PROXY["Policy proxy"] - ROUTER["Inference router"] POLICY["OPA policy engine"] AGENT["Restricted agent process"] end @@ -81,8 +80,7 @@ flowchart TB AGENT -->|"all ordinary egress"| PROXY PROXY -->|"evaluate"| POLICY PROXY -->|"allowed traffic"| EXT["External services"] - PROXY -->|"inference.local"| ROUTER - ROUTER -->|"managed inference"| MODEL["Inference backends"] + PROXY -->|"profile-authorized traffic"| MODEL["Model providers"] ``` ## Core Boundaries @@ -90,14 +88,13 @@ flowchart TB | Component | Boundary | |---|---| | CLI, SDK, TUI | User-facing management surfaces. They talk to the gateway and do not need to know which infrastructure drivers are active. | -| Gateway | Authenticated control plane, API server, durable state, policy and settings delivery, provider and inference config, supervisor session ownership, and relay coordination. | +| Gateway | Authenticated control plane, API server, durable state, policy and settings delivery, provider config, supervisor session ownership, and relay coordination. | | Compute subsystem | Sandbox lifecycle semantics: creation, deletion, watching, reconciliation, and state transitions. Platform provisioning details belong to the compute driver. | | Credentials subsystem | Logical provider and credential resolution. Secret storage and platform-native credential access belong to credentials drivers. | | Control-plane identity | Authentication and authorization for users, operators, and API clients. External identity verification belongs to identity drivers. | | Sandbox identity | Workload identity for supervisors and sandbox-to-sandbox authorization. Identity issuance or verification belongs to sandbox identity drivers. | | Supervisor | Sandbox-local security boundary. It prepares isolation, fetches config, injects credentials, runs relay endpoints, starts the proxy, and launches restricted agent processes. | -| Policy proxy | Mandatory egress path for agent traffic. It enforces destination, binary identity, SSRF, TLS/L7, credential injection, and inference interception rules. | -| Inference router | Sandbox-local forwarding for `https://inference.local` to configured model backends. | +| Policy proxy | Mandatory egress path for agent traffic. It enforces destination, binary identity, SSRF, TLS/L7, and endpoint-bound provider credential injection. | ## Integrating with the Ecosystem @@ -109,14 +106,14 @@ platforms that already provide them. The gateway owns OpenShell control-plane semantics: sandbox state, lifecycle ordering, policy and settings resolution, credential mapping, authorization, -inference configuration, and relay coordination. Drivers translate those -semantics into platform-native operations. They should stay thin, preserve +and relay coordination. Drivers translate those semantics into platform-native +operations. They should stay thin, preserve native behavior by default, and report platform lifecycle events back through the shared contracts. The supervisor owns OpenShell sandbox semantics. Filesystem policy, process -privilege reduction, network proxying, inference interception, credential -injection, security logging, and gateway relay behavior should remain +privilege reduction, network proxying, endpoint-bound credential injection, +security logging, and gateway relay behavior should remain consistent across runtimes. This keeps OpenShell usable in local single-player setups, Kubernetes @@ -128,7 +125,7 @@ ecosystem. The gateway and sandbox split control-plane authority from runtime enforcement. The gateway owns durable platform state: sandboxes, policy revisions, runtime -settings, provider records, inference configuration, session records, and +settings, provider records and profiles, session records, and authorization decisions. A sandbox owns the local execution boundary: process identity, filesystem access, network egress, credential injection, local logs, and the agent child process. @@ -141,13 +138,17 @@ bridge networks, port mappings, NAT traversal, or bespoke tunnels. The common runtime requirement is narrower: the supervisor must be able to reach the gateway. -The Windows MXC driver is an explicit exception. It launches and monitors a -one-shot workload in the driver, self-reports readiness, and does not expose a -supervisor session, interactive connect, live policy delivery, or governed egress. +The compute-driver capability contract identifies whether a driver reports +runtime readiness. Most drivers use the supervisor session model above. A +driver that sets `driver_reports_runtime_readiness` may self-report readiness +without a supervisor session. Every driver receives the canonical create-time +policy in `DriverSandboxSpec`; drivers without a supervisor use the existing +sandbox configuration API for later revisions. The Windows MXC driver reports +its own readiness and does not expose interactive connect or governed egress. The gateway delivers desired state; the sandbox applies it locally. Policy, -settings, credentials, and inference routes flow from the gateway to the -supervisor. The supervisor validates and applies what can change at runtime, +settings, provider attachments, and credential bindings flow from the gateway +to the supervisor. The supervisor validates and applies what can change at runtime, keeps last-known-good config when refresh fails, and leaves static isolation controls in place until the sandbox is recreated. @@ -166,7 +167,7 @@ that crate's `README.md`. | Document | Purpose | |---|---| | [Gateway](gateway.md) | Gateway control plane, auth, APIs, persistence, settings, and relay coordination. | -| [Sandbox](sandbox.md) | Sandbox supervisor, child process isolation, proxy, credentials, inference, connect, and logs. | +| [Sandbox](sandbox.md) | Sandbox supervisor, child process isolation, proxy, provider credentials, connect, and logs. | | [Sandbox Limits](sandbox-limits.md) | Sandbox supervisor and egress safety ceilings, ownership rules, current enforcement, and known gaps. | | [Security Policy](security-policy.md) | Policy model, enforcement layers, policy updates, policy advisor, and security logging. | | [Compute Runtimes](compute-runtimes.md) | Docker, Podman, Kubernetes, VM, sandbox images, and runtime-specific responsibilities. | diff --git a/architecture/build.md b/architecture/build.md index 4ffec5b71..5f3b913e1 100644 --- a/architecture/build.md +++ b/architecture/build.md @@ -10,8 +10,9 @@ OpenShell builds these main artifacts: | Artifact | Source | |---|---| -| Gateway binary | `crates/openshell-server` | +| Gateway binary | `crates/openshell-gateway` | | CLI binaries and system packages | `crates/openshell-cli` plus release packaging | +| E2E conformance CLI | `crates/openshell-conformance-cli` | | Python SDK wheel | `python/openshell` | | TypeScript SDK package | `sdk/typescript` | | Gateway container image | `deploy/docker/Dockerfile.gateway` | @@ -24,22 +25,47 @@ Sandbox community images are built outside this repository. ## Build Features +Rust builds require Rust 1.94 or newer. TLS and certificate generation use +AWS-LC, including the CLI and standalone examples. Native and cross-build +environments must provide the C toolchain required by aws-lc-sys; the Nix +development shells provide static AWS-LC libraries. + +SQLx uses AWS-LC with native certificate roots. The server enables +`sqlx-core/rustls-native-certs` directly because SQLx's facade does not expose +that root selection independently of the crypto provider. Credential storage +continues to use the same AES-256-GCM envelope format across backend changes. + Anonymous telemetry emission is gated behind a default-on `telemetry` Cargo feature. It is defined in `openshell-core` (where the emission code, HTTP client, and endpoint live) and forwarded by the binary crates that emit or -collect telemetry: `openshell-server` (gateway), `openshell-sandbox` +collect telemetry: `openshell-gateway`, `openshell-sandbox` (supervisor), and `openshell-driver-vm`. Every crate depends on `openshell-core` with `default-features = false`, so the binary crate's feature is the single switch that enables `openshell-core/telemetry` for its build graph. In-process drivers (`docker`, `kubernetes`, `podman`) inherit the gateway's setting through feature unification and carry no passthrough. -Building a binary with `--no-default-features` compiles out telemetry entirely: -no endpoint, no telemetry HTTP client, and no emission code. With telemetry -compiled out, `telemetry::enabled()` is always `false` and the `emit_*` helpers -are no-ops, so the data-model types stay available and dependent crates compile -unchanged. The runtime `OPENSHELL_TELEMETRY_ENABLED` switch remains the way to -disable telemetry in a default (telemetry-enabled) build. +Building a binary without the `telemetry` feature compiles out telemetry +entirely: no endpoint, no telemetry HTTP client, and no emission code. With +telemetry compiled out, `telemetry::enabled()` is always `false` and the +`emit_*` helpers are no-ops, so the data-model types stay available and +dependent crates compile unchanged. The runtime `OPENSHELL_TELEMETRY_ENABLED` +switch remains the way to disable telemetry in a default (telemetry-enabled) +build. + +Cargo cannot subtract a single default feature, so each of the three binary +crates also defines a `defaults-without-telemetry` alias listing every default +except `telemetry`. Telemetry-free builds use +`--no-default-features --features defaults-without-telemetry` and stay correct +as the default set grows, instead of dropping unrelated defaults the way a bare +`--no-default-features` does on `openshell-sandbox`. The alias is a keep-list, +not a switch: enabling it on top of the defaults would otherwise yield a +telemetry-on binary that reads as telemetry-free, so each crate root carries a +`compile_error!` for the `telemetry` + `defaults-without-telemetry` combination. +`rust:verify:defaults-without-telemetry` guards both properties — that each +alias still equals its crate's defaults minus `telemetry`, and that the +mutual-exclusion error is wired up — and `rust:verify:telemetry-off` builds +through the alias and inspects the resulting binaries for telemetry markers. Supervisor upstream TLS root-store selection is controlled by the `bundled-ca-roots` Cargo feature (on by default). Default builds use Mozilla @@ -131,6 +157,28 @@ in platform-specific Nix development shells through reusable workflows and the shared `build-rust-binary` action. The image build downloads each binary artifact into the staging directory before running Buildx. +The Nix flake exposes one development shell with target-specific toolchains. +The shared `mkToolchain` function in `nix/toolchain/default.nix` assembles native +libraries and Cargo environment settings. Linux and Darwin modules select the +compiler and sysroot and generate the compiler wrapper for their platform. +The `nix/toolchain/glibc-2.28/` directory contains the pinned glibc build, +GCC environment, and sysroot assembly used by GNU Linux targets. +The glibc build reuses a pinned historical Nixpkgs recipe with current build +tools; its headers, shared libraries, and static archives are assembled into +the sysroot from separate outputs. +Each toolchain supplies its compiler driver, assembler, archiver, native +libraries, and Cargo environment through derivation passthru. The shell omits +an implicit host C compiler; Cargo builds select the appropriate tools with +`--target`. GNU targets use a glibc 2.28 sysroot and static GCC runtimes, while +musl targets produce static executables. + +On macOS, the shell also provides a native Darwin toolchain with static Z3 +and AWS-LC. Its Clang driver uses the pinned, unprocessed Apple SDK so system +library stubs, including libiconv and libc++, retain their Apple install names. +System libraries and frameworks remain dynamically linked. The deployment +target matches the Nix host platform's minimum macOS version. The Rust toolchain +does not propagate Nix's replacement system libraries into the link environment. + Gateway and supervisor binaries staged into branch E2E, Release Dev, and Release Tag images are compiled through `cargo auditable` (pinned in `mise.toml`), which embeds a `.dep-v0` section describing the Rust dependencies actually compiled @@ -141,14 +189,15 @@ is a different artifact from the source SBOM produced by `syft dir:.` in `tasks/sbom.toml`, which describes the checkout, and from the image SBOM attestation below, which describes a published image. -The shared binary build action compiles release artifacts with `cargo auditable`. +The shared binary build action uses the default Nix shell and compiles release +artifacts with `cargo auditable build --target `. Verification and upload +read binaries from `target//release/`. Branch E2E, Release Dev, and Release Tag image jobs stage those same artifacts instead of rebuilding binaries in Docker. Each binary build scans its output with Syft and requires at least one decoded Cargo package before uploading the -artifact. Darwin builds replace Nix's `libiconv` load command with the macOS -system install name, ad-hoc sign the modified binary, and fail if `otool -L` -reports any remaining `/nix/store` dependency. Runtime and Syft verification -run after that normalization. The CI image gains the pinned `cargo-auditable` +artifact. The action checks each binary's `--version` output and leaves its +linkage as produced by the Nix toolchain, without post-link rewriting or +platform-specific linkage checks. The CI image gains the pinned `cargo-auditable` tool through `mise install --locked` but ships no auditable OpenShell binary of its own. @@ -240,6 +289,13 @@ pulls and explicit publication. OCI pulls require a trusted manifest digest and retain that provenance with the local entry; mutable tags are used only for explicit publication. +CLI conformance runs after target provisioning. Action-free scenarios operate +only through the configured OpenShell CLI. A versioned conformance plan may add +an ordered sequence of target-supplied host-side actions, such as a gateway +restart, while the scenario remains responsible for black-box sandbox +continuity checks. The plan exposes opaque executable paths and timeouts rather +than driver or package-manager configuration; target setup owns those details. + ## Python Wheel Packaging The generated protobuf/gRPC stubs under `python/openshell/_proto/` are gitignored @@ -272,7 +328,7 @@ the release tag. ## CI and E2E -Required checks run on GitHub Actions. Workflows that use NVIDIA self-hosted runners trigger from copy-pr-bot mirror branches, so trusted PRs are mirrored into `pull-request/` branches before those workflows run. `main` also uses GitHub merge queue so the final queued integration commit is validated before it merges. +Required checks run on GitHub Actions. Pull-request workflows that use NVIDIA self-hosted runners trigger from copy-pr-bot mirror branches, so trusted PRs are mirrored into `pull-request/` branches before those workflows run. `main` also uses GitHub merge queue so the final queued integration commit is validated before it merges. The high-level CI model: @@ -289,22 +345,34 @@ disables emission for Rust tests, E2E runs, and release canaries. This prevents synthetic activity from contributing to product usage metrics. Static security checks are deliberately outside the mirror-branch path. They run -directly on GitHub-hosted runners with no secrets, so the pull-request-triggered -ones also cover fork pull requests, and none of them consume NVIDIA self-hosted -capacity. Scanner jobs request `security-events: write` and upload SARIF to Code -Scanning directly on every event they run on, including fork and Dependabot pull -requests, which Code Scanning permits for `pull_request` runs despite their -read-only `GITHUB_TOKEN`. Each scanner also retains its report as a workflow -artifact. No privileged intermediate workflow relays those uploads. +directly on GitHub-hosted runners and none of them consume NVIDIA self-hosted +capacity. The change-oriented ones receive no secrets, so they also cover fork +pull requests. Codex Security release qualification is the exception: it needs a +scoped API key, which routes its model calls to NVIDIA-hosted inference while +the job itself stays GitHub-hosted. That placement is load-bearing rather than +incidental: on the repository self-hosted runner the scan agent executes no +shell commands at all, so its preflight never scopes the diff and it seals no +draft. Scanner jobs request `security-events: write` and upload SARIF to Code +Scanning directly on every event they run on, including fork and Dependabot +pull requests, which Code Scanning permits for +`pull_request` runs despite their read-only `GITHUB_TOKEN`. No privileged +intermediate workflow relays those uploads. Manually dispatched Codex Security +runs are the one opt-in exception, described below. Report retention differs by +scanner: Actionlint, Zizmor, and CodeQL keep their reports as workflow artifacts, +and Codex Security keeps no raw report. Triggers differ by workflow: `.github/workflows/workflow-security.yml` runs on `pull_request`, `merge_group`, `main`, and a weekly schedule; `.github/workflows/dependency-review.yml` runs on `pull_request` and -`merge_group` only, because it needs a base and head commit to compare; and +`merge_group` only, because it needs a base and head commit to compare; `.github/workflows/codeql.yml` runs nightly on the default branch (`main`) via -`schedule`, with `workflow_dispatch` kept for manual diagnostics. CodeQL does +`schedule`, with `workflow_dispatch` kept for manual diagnostics; and +`.github/workflows/codex-security.yml` runs on pushed `v*.*.*-pre.*` tags, and is +also callable through `workflow_call` and `workflow_dispatch`. CodeQL does not run on `pull_request`, `merge_group`, or pushes to `main`, so it reports repository-level Code Scanning state on the default branch instead of per-PR results, and its four-language matrix stays off the per-change critical path. +Codex Security is release-scoped rather than change-scoped, so it never runs on +a pull request or merge group. - **Actionlint and Zizmor** analyze the workflow definitions themselves. Repository configuration lives in `.github/actionlint.yml` (self-hosted runner @@ -329,24 +397,120 @@ results, and its four-language matrix stays off the per-change critical path. Go requires a build; the other languages use build mode `none`. Analysis runs on the nightly schedule or by manual dispatch. Results are uploaded to Code Scanning and always retained as workflow artifacts. +- **Codex Security** qualifies release candidates rather than individual + changes. The job installs a pinned `@openai/codex-security` release into the + runner temp directory before the repository is checked out and invokes it by + absolute path, so repository-controlled files cannot shadow the scanner. Model + calls go to NVIDIA-hosted inference at `https://inference-api.nvidia.com/v1`, + declared as a custom Codex provider named `nvidia` that uses the Responses + wire API with WebSockets disabled. The scan runs `openai/openai/gpt-5.6-sol` + at `medium` reasoning effort, with the multi-agent runtime capped at eight + concurrent threads through + `features.multi_agent_v2.max_concurrent_threads_per_session`. The + `CODEX_SECURITY_API_KEY` secret holds the + NVIDIA key and is exposed to the scan step alone, as `OPENAI_API_KEY` so the + CLI selects API-key auth and as `NVIDIA_INFERENCE_API_KEY`, the provider + `env_key` read by the Codex child process. `CODEX_SECURITY_STATE_DIR` and + `SCAN_DIR` are suffixed with `github.run_id` and `github.run_attempt` and + created mode `700`, so no scanner state or result set from a previous run or + retry attempt is reused even on a runner with a reusable temp directory. + `tasks/scripts/codex_security_range.py` resolves the scan range, reusing the + tag parsers in `tasks/scripts/release.py` so both stay on one definition of a + release tag while requiring the `v` prefix that a release workflow needs. The + job stages both files out of the workspace from the workflow's own revision + and runs the resolver by absolute path, because a scanned candidate predates + them and a revision under scan must not choose its own scan range. The range + itself is resolved against the checked-out candidate: the + candidate must be a `vX.Y.Z-pre.N` tag that is an ancestor of `origin/main`, + and the base is the newest stable `vX.Y.Z` tag merged into the candidate that + is strictly older than the release train `vX.Y.Z` the candidate targets. A + full-repository scan is only possible when no such stable tag exists and the + caller passes `allow_full_bootstrap`. Each candidate scans the cumulative + stable-to-candidate diff, so later candidates re-cover earlier ones. SARIF is + uploaded against `refs/heads/main` at the candidate commit under the + train-scoped category `codex-security/vX.Y.Z`, which makes each candidate's + analysis replace the previous one for that train. Automatic pre-release tag + pushes and `workflow_call` runs always upload. `workflow_dispatch` runs still + perform the scan and the SARIF export, but skip the Code Scanning upload + unless the caller sets the `upload_sarif` input, so manual diagnostics do not + overwrite a train's published analysis by default. Codex Security 0.1.24 + cannot apply `--max-cost` to a slash-qualified model identifier, so the run + has no CLI-enforced cost ceiling. Spend is bounded instead by the 120-minute + job timeout, a single repository-wide concurrency group that serializes + qualification so starting a newer candidate cancels an in-flight one, and + NVIDIA account-side controls. No raw report is retained. +- The job clears `kernel.apparmor_restrict_unprivileged_userns` before + installing the scanner. Codex confines model-run commands with bubblewrap, + which needs unprivileged user namespaces; Ubuntu 24.04 restricts those through + AppArmor, so bubblewrap fails to configure the sandbox network namespace + (`bwrap: loopback: Failed RTM_NEWADDR`) and the agent executes no commands at + all. The failure is silent: the agent retries its shell tool, gives up, and + seals no draft, while the scanner only reports a missing or incomplete draft. + Lifting a kernel restriction on the runner is what allows the sandbox that + confines the agent to start, and the runner is ephemeral and GitHub-hosted. +- The scan sets `approval_policy="never"`. Codex Security keeps + `approvals_reviewer="auto_review"` unconditionally, and that reviewer runs on + its own model rather than the configured one. Because the workflow declares a + single provider that serves only `openai/openai/gpt-5.6-sol`, any approval + request reaches a model the endpoint does not serve, so the agent never gets a + shell command approved and seals no draft. The scan stays confined by its + `workspace-write` sandbox with network access disabled and by the scanner's + own permission profile, which grants read access to the filesystem root and + write access only to the workspace roots. +- A scan that cannot execute commands reports only a missing or incomplete + draft, so diagnosing one means reading the scanner's session rollouts under + `CODEX_SECURITY_STATE_DIR`, where every shell command the agent ran is + recorded. No command at all is the signal that the sandbox failed to start. Findings never fail these checks; scanner and build failures do. A scanner that -cannot run, a CodeQL analyzer that does not complete, and an unexpected -Dependency Graph API error are all errors, which keeps an informational check -from silently degrading into a no-op. None of these checks are required -statuses, so they do not gate merges. +cannot run, a CodeQL analyzer that does not complete, an unexpected Dependency +Graph API error, and a Codex Security range, scan, or export failure are all +errors, which keeps an informational check from silently degrading into a no-op. +Codex Security also rejects any scan scope other than the resolved +cumulative diff or an approved full bootstrap, so a qualification run either +covers the whole stable-to-candidate range or fails; a separate no-permission +job republishes the analysis job's outcome as the +`OpenShell / Codex Security (informational)` status. None of these checks are +required statuses, so they do not gate merges. -See `CI.md` for the contributor workflow, labels, and maintainer merge-queue workflow. +Codex Security findings are informational during the observation phase, and the +workflow only reports on candidates that already exist. Creating pre-release +tags and gating stable promotion on qualification results are part of +[RFC 0014](../rfc/0014-release-stability/release-qualification.md) and are not +implemented yet. + +## Artifact Scanning + +Two entry points share `tasks/scripts/trivy-scan.sh`: a standalone analysis +workflow and a pull-request change gate. Nix supplies Trivy, Helm and `yq`. + +The standalone workflow scans deployment configuration and supplied OCI +references independently of release publication. Detailed JSON reports feed the +differential gate; the summary and published SARIF consolidate configuration +findings across profiles while preserving resource identity and affected profiles. +Images and packaged charts retain separate identities based on their full +references. Publication batches respect GitHub's limit of 20 SARIF runs. + +The PR/merge-group gate scans base and candidate with the same scanner and rejects +new `HIGH` or `CRITICAL` configuration findings. Its stable +`OpenShell / Trivy Changes` status succeeds when nothing relevant changed. +Image CVEs need the standalone scan. The reporting and gate invariants are: + +- A structurally invalid Trivy report is an error, not an empty finding set. +- Scanner failures prevent publication of incomplete analyses; findings alone + do not prevent publishing complete reports. +- Findings compare per profile against the same baseline profile, by semantic + identity and count rather than line number; a profile absent from the baseline + falls back to that identity's maximum across all profiles. +- The candidate's ignore file is validated, but the baseline's policy applies to + both scans, so an exemption takes effect only after merge. + +See [CI.md](../CI.md#artifact-scanning) for profiles, report paths, severity +settings, exceptions, and the contributor and maintainer workflows. ## Docs Site -Published docs live in `docs/`. Navigation lives in `docs/index.yml`. Fern site -configuration, components, theme assets, and publish settings live in `fern/`. - -Use `mise run docs` for strict validation and `mise run docs:serve` for local -preview. PR previews are produced by `.github/workflows/branch-docs.yml` when -Fern credentials are available. Production docs publish from the release tag -workflow. +Published docs live in `docs/`, and Fern site configuration lives in `fern/`. See [fern/README.md](../fern/README.md) for the source layout, local development commands, version model, and publishing workflows. ## Validation Expectations diff --git a/architecture/compute-runtimes.md b/architecture/compute-runtimes.md index b21dd0dc8..e1e731a0c 100644 --- a/architecture/compute-runtimes.md +++ b/architecture/compute-runtimes.md @@ -1,34 +1,43 @@ # Compute Runtimes Compute runtimes create, stop, start, delete, and watch sandbox workloads for the -gateway. They do not replace sandbox policy enforcement. Every runtime starts a -workload that runs the `openshell-sandbox` supervisor, and the supervisor -enforces the sandbox contract locally. +gateway. Supervisor-controlled runtimes start a workload that runs the +`openshell-sandbox` supervisor, which enforces the sandbox contract locally. +Driver-controlled runtimes apply the canonical sandbox policy while +provisioning and report workload readiness directly. ## Driver Contract -Each runtime receives a sandbox spec from the gateway and is responsible for: +Each runtime receives a sandbox spec and canonical policy from the gateway and +is responsible for: - Selecting the sandbox image. -- Injecting sandbox identity and gateway callback configuration. -- Supplying TLS or secret material for supervisor callbacks. -- Providing the supervisor binary or image in the workload. +- For supervisor-controlled runtimes, injecting sandbox identity and gateway + callback configuration, supplying callback credentials, and providing the + supervisor binary or image. +- For runtimes without the standard supervisor, validating and applying the + canonical policy before launching the workload. - Forwarding the exact canonical main-process argv and TTY mode without shell reconstruction. The sandbox-level environment and policy workspace apply to the main process. - Reporting lifecycle and platform events back to the gateway. - Cleaning up runtime-owned resources. -Drivers report **backend state only**. A driver snapshot with `Ready=True` means -the underlying compute resource (container, pod, VM) is healthy and running — -nothing more. Drivers must not gate on supervisor session state or hold -references to gateway-internal types. The gateway owns the public -`SandboxPhase::Ready` decision. This applies equally to extension drivers -implementing `ComputeDriver` out of tree. +Drivers report **runtime-observed state only** and must not hold references to +gateway-internal types. For supervisor-controlled runtimes, `Ready=True` means +only that the compute resource is healthy; the gateway also requires a +supervisor session before publishing `SandboxPhase::Ready`. For +drivers that report runtime readiness, `Ready=True` is authoritative because the driver +launches and monitors the policy-constrained workload itself. `compute_driver.proto` is the supported gateway/driver extension boundary. At initialization the gateway snapshots the driver's identity, version, -default image, and gateway-lifecycle preference from `GetCapabilities`. +default image, gateway-lifecycle preference, and +`driver_reports_runtime_readiness` from `GetCapabilities`. The gateway includes +the canonical `SandboxPolicy` in `DriverSandboxSpec.policy` for validation and +creation. Drivers that enforce policy outside the standard supervisor fetch +later revisions through `GetSandboxConfig` and acknowledge them through +`ReportPolicyStatus`. Process-identity omissions are preserved across this boundary so every driver can apply its native image or runtime defaults. Driver-requested listeners are structurally validated and remain restricted to sandbox callback RPCs. @@ -45,28 +54,29 @@ reason strings. ## Sandbox Readiness Composition -The gateway composes driver backend state with supervisor session presence to -produce the public `SandboxPhase`. This composition is gateway-owned and applied -uniformly across all drivers: +The gateway composes driver state with the advertised readiness behavior to +produce the public `SandboxPhase`: ``` backend_phase = derive_phase(driver_status) public_phase = if backend_phase in {Error, Deleting}: → pass through (terminal precedence) + if driver_reports_runtime_readiness && backend_phase == Ready: → Ready if backend_phase == Ready && session connected: → Ready if backend_phase == Ready && no session: → Provisioning if backend_phase in {Provisioning, Unknown} && session: → Ready if backend_phase in {Provisioning, Unknown} && no session: → Provisioning ``` -When `public_phase == Ready` the sandbox is usable through the gateway — both the +For a supervisor-controlled runtime, `public_phase == Ready` means both the backend resource is healthy and a supervisor session is registered. A sandbox whose backend reports ready but has no supervisor session yet holds `Provisioning` with a `Ready=False`, `SupervisorNotConnected` condition and the message `Backend ready; waiting for supervisor session`. This distinguishes it from a sandbox whose compute resource is still provisioning without exposing contradictory public -readiness signals. +readiness signals. When the driver reports runtime readiness, its ready condition +is published without waiting for a supervisor session. **Session precedence over lagging driver snapshots:** A supervisor session can only be established by a running workload. When `set_supervisor_session_state` promotes the @@ -85,10 +95,12 @@ session-owning replica. That work is deferred to GitHub issue #1868. Until then, deployments that require reliable readiness composition must run a single gateway replica. -**Extension point:** The readiness decision is a safety invariant, not an -operator-configurable hook. The driver contract is the correct extension point for -custom backend readiness semantics. RFC-0010 lifecycle hooks may observe readiness -transitions via `post_commit`; they do not override the composition rule. +**Extension point:** Driver-reported readiness is a capability, not an +operator-configurable hook. A driver may enable it only when it owns workload +readiness. Policy delivery remains independent: create-time policy is embedded +in the sandbox specification, and later revisions use the existing sandbox +configuration API. RFC-0010 lifecycle hooks may observe readiness transitions via +`post_commit`; they do not override the composition rule. The capability RPC reports driver identity, version, and the default sandbox image used by the gateway. GPU availability stays driver-local and is validated @@ -103,9 +115,8 @@ of re-querying drivers on each request. The gateway binary explicitly installs the compute drivers compiled into that binary before entering server startup. The server selects a configured driver by normalized registry name. When no driver is configured, it evaluates only -the installed drivers' probes in registered priority order, records every -available registration, and selects the first. Drivers without a probe, -including VM, remain opt-in. +the installed drivers' probes and chooses the lowest registered priority. +Drivers without a probe, including VM, remain opt-in. Startup computes this selection once after merging configuration. The same selection drives authentication defaults and runtime construction, so a probe @@ -117,17 +128,18 @@ registry. Adding or removing a compiled driver therefore changes registration rather than the server's selection flow. Alternate gateway binaries can install their own `ComputeDriverFactory` registrations and hand the completed registry to `run_cli_with_compute_drivers`; factories receive merged driver config and -finish through the same in-process runtime adapter. A configured UDS endpoint -still takes precedence over a compiled registration with the same name. +return either an in-process driver or a gateway-managed remote endpoint. The +server constructs the common runtime adapter and snapshots `GetCapabilities` +for either result. A configured UDS endpoint still takes precedence over a +compiled registration with the same name. -The standard server crate groups first-party registrations behind the -`in-tree-compute-drivers` feature. Protocol-only gateway builds disable that -feature and link no compute-driver crates. E2E lanes compose that gateway with -Docker, Podman, Kubernetes, and VM driver executables over the public UDS gRPC -contract so an in-tree driver cannot silently depend on a server-only API. -External Kubernetes drivers support shared and managed workspace modes. -Operator mode requires an in-process dynamic namespace allowlist and is -rejected when Kubernetes is configured through an external endpoint. +The `openshell-gateway` composition crate groups first-party registrations +behind the `in-tree-compute-drivers` feature. `openshell-server` has no compute +driver dependencies or backend-name dispatch. Protocol-only gateway builds +disable the composition feature and link no compute-driver crates. E2E lanes +compose that gateway with Docker, Podman, Kubernetes, and VM driver executables +over the public UDS gRPC contract so an in-tree driver cannot silently depend +on a server-only API. ## Stop and Start Lifecycle @@ -208,6 +220,22 @@ backend is already absent (`deleted = false`), the request removes gateway state synchronously. Sandbox row removal remains bound to the stable ID and resource version. Settings retain their existing best-effort name-based cleanup; SSH sessions, indexes, and watch/log buses are cleaned after confirmed removal. +Owned-record cleanup discovers records before mutating them and uses bounded +set-based deletes so teardown cannot amplify one sandbox into an unbounded +sequence of individual persistence writes. + +When a sandbox is instead discovered gone out-of-band — a watcher deletion +event, or the periodic prune sweep finding no matching driver resource, with +no explicit `DeleteSandbox` request involved at all — the gateway also +releases driver-owned resources (for example Podman's per-sandbox secrets and +workspace volume) by calling the driver's idempotent `DeleteSandbox`, not just +gateway state. Both paths skip that call when a request-side lifecycle +operation already holds the sandbox's gate, since that operation already owns +driver-side cleanup. The watch path defers the call itself to a background +task after a non-blocking gate check, so a slow driver call cannot stall the +sequential watch loop; the prune sweep calls the driver inline, since it +already makes a blocking `GetSandbox` call per sandbox as part of its normal +operation. The request acquires both locks before starting owned work, so cancellation while queued does not leave a delete armed. After that commitment point, the @@ -230,6 +258,16 @@ template resource limits. Docker and Podman apply them as runtime limits. Kubernetes mirrors each limit into the matching request. VM accepts the fields but currently ignores them. +Reusable sandbox workload templates are resolved before the compute-driver +boundary. Drivers do not receive a separate template resource; the gateway +lowers the selected `SandboxWorkloadTemplate` into the existing sandbox spec +and validates that spec before calling `ValidateSandboxCreate` or +`CreateSandbox`. Template CPU and memory become the same typed resource limits +described above. Template GPU settings become `ResourceRequirements`, preserving +the driver's default GPU assignment when the count is omitted. Template +`driver_config` remains a driver-keyed envelope until the compute layer selects +the active driver block and forwards only that block to the driver. + Docker and Podman also accept per-sandbox driver-config mounts for existing runtime-managed named volumes and tmpfs mounts. Podman additionally accepts image mounts through its image-volume API. User-supplied bind and volume mounts @@ -255,6 +293,17 @@ through the driver configuration. The Helm chart defaults sandbox agents to `Unconfined` so runtime/default AppArmor profiles do not block supervisor network namespace setup on AppArmor-enabled nodes. +The Kubernetes deployment packaging has two ownership boundaries. The gateway +chart owns the gateway workload, configuration, Services, PKI, and +cluster-scoped gateway resources. It can retain the legacy combined behavior, +or omit workspace resources. The workspace chart is installed into a +pre-provisioned sandbox namespace and owns only the sandbox ServiceAccount, +namespaced RBAC, and sandbox ingress NetworkPolicy. Its RoleBinding names the +gateway ServiceAccount and namespace explicitly, so the two releases have +disjoint lifecycle ownership. A shared-mode gateway can target one external +namespace, while operator mode maps workspace names to multiple +platform-provisioned namespaces. + Resource requirements enter the driver layer through `SandboxSpec.resource_requirements`. This includes a set of GPU requirements, where a user can request a specific number of GPUs or the driver-specific default behaviour. For all in-tree drivers, this is equivalent to selecting a single GPU. @@ -455,13 +504,13 @@ image-pull Secrets in every operator-managed namespace. **Operator** uses pre-provisioned namespaces discovered through two optional sources: a K8s label selector (`operator_namespace_label`) and a drop-in -allowlist file (`operator_namespace_file`). At least one must be configured. -The `OperatorNamespaceAllowlist` (`Arc>>`) is populated -at runtime by background watchers and read by the namespace resolver. Sandbox -creation fails closed if the workspace is not in the current allowlist. Platform -teams manage namespace lifecycle externally. RBAC uses the same ClusterRole as -managed mode but without namespace `create`/`delete` or ServiceAccount -permissions. +allowlist file (`operator_namespace_file`). Exactly one must be configured. +The compute driver and the gateway's ServiceAccount authenticator independently +watch that public config source; no in-process driver state crosses into the +server. Sandbox creation and token bootstrap fail closed if the workspace is +not in the current allowlist. Platform teams manage namespace lifecycle +externally. RBAC uses the same ClusterRole as managed mode but without namespace +`create`/`delete` or ServiceAccount permissions. ### Watching and Querying diff --git a/architecture/gateway.md b/architecture/gateway.md index 0430d9515..59b2bd52e 100644 --- a/architecture/gateway.md +++ b/architecture/gateway.md @@ -2,17 +2,16 @@ The gateway is the OpenShell control plane. It exposes the API used by the CLI, SDK, and TUI; persists platform state; manages provider credentials and -inference configuration; and asks compute runtimes to create or delete sandbox -workloads. +attachments; and asks compute runtimes to create or delete sandbox workloads. ## Responsibilities - Authenticate clients and sandbox callbacks. - Serve gRPC APIs for sandbox lifecycle, provider management, policy updates, - settings, inference configuration, logs, watch streams, and relay forwarding. + settings, logs, watch streams, and relay forwarding. - Serve HTTP endpoints for health, WebSocket tunnels, and edge-auth flows. - Persist domain objects in SQLite or Postgres. -- Resolve provider credentials and inference bundles for sandbox supervisors. +- Resolve endpoint-bound provider environments for sandbox supervisors. - Coordinate supervisor relay sessions for connect, exec, file sync, and service forwarding. - Persist the canonical main-process instance ID and normalized exit code on @@ -55,6 +54,16 @@ health, metrics, or tunnel routes. The plaintext service router also rejects browser requests whose Fetch Metadata, Origin, or Referer headers indicate a cross-origin or sibling-subdomain request. +Public workspace-scoped RPCs carry a typed `WorkspaceSelector`. A request must +select one non-empty workspace explicitly; `default` is an ordinary explicit +name, not an omitted-value fallback. Sandbox, sandbox template, provider, and +service list RPCs also accept an all-workspaces marker after Platform Admin +authorization. Single-workspace handlers reject that marker. Platform-global +policy operations require the selector to be absent, while workspace policy +operations require it. The gateway authorizes the selected scope before +performing resource lookup so malformed, unsupported, and unauthorized scopes +have consistent behavior across resource types. + Docker and Podman report the local address through which their sandboxes can reach the gateway. When the primary listener covers that address, the gateway reuses it; sandbox JWT authentication and its RPC allowlist remain the callback @@ -62,7 +71,7 @@ authorization boundary. When the primary listener does not cover the address, the gateway adds a callback-only listener. Additional callback listeners accept only gRPC methods classified as sandbox-callable by the gateway's generated authorization metadata. They reject user and administrator APIs, health, -reflection, non-callback inference APIs, and HTTP routes before normal request +reflection, and HTTP routes before normal request authentication. The operator-configured primary listener retains the full multiplexed API surface. @@ -192,7 +201,7 @@ Supported auth modes: | Plaintext | Local development or a trusted reverse proxy boundary. | | Unauthenticated local users | Trusted Kubernetes dev or fully trusted proxy deployments only. | | Cloudflare JWT | Edge-authenticated deployments where Cloudflare Access supplies identity. | -| OIDC | Bearer-token auth for users, with browser or device-code PKCE and client credentials login. | +| OIDC | Bearer-token auth for users, with browser or device-code PKCE and client credentials login. JWKS validation accepts RS256, RS384, RS512, PS256, PS384, PS512, ES256, ES384, and EdDSA (Ed25519) signing keys. | The CLI persists the scopes requested during OIDC login in gateway metadata and reuses them when refreshing an access token. This preserves the intended API @@ -277,7 +286,6 @@ The gateway API is organized around platform objects and operational streams: | Sandbox lifecycle | Create, list, delete, watch, exec, SSH session bootstrap, ForwardTcp service forwarding. | | Providers | Store provider records, discover credentials, resolve runtime environment. | | Policy and settings | Get effective sandbox config, update sandbox policy, manage global settings. | -| Inference | Set gateway-level model/provider config and resolve sandbox route bundles. | | Observability | Push sandbox logs, stream sandbox status and logs to clients. | Domain objects use shared metadata: stable server-generated IDs, human-readable @@ -299,7 +307,7 @@ The storage schema is intentionally narrow: | Column | Purpose | |---|---| | `id` | Stable gateway-generated object ID and primary key. | -| `object_type` | Logical resource kind, such as `sandbox`, `provider`, `ssh_session`, `inference_route`, `sandbox_policy`, or `draft_policy_chunk`. | +| `object_type` | Logical resource kind, such as `sandbox`, `provider`, `provider_profile`, `ssh_session`, `sandbox_policy`, or `draft_policy_chunk`. | | `name` | Human-readable name, unique within an object type when present. | | `scope` | Optional owner or namespace for scoped/versioned records, such as a sandbox ID for policy revisions. | | `version` | Optional monotonically increasing version for scoped records. | @@ -310,6 +318,81 @@ The storage schema is intentionally narrow: | `created_at_ms` and `updated_at_ms` | Gateway timestamps used for ordering and list output. | | `labels` | JSON object carrying Kubernetes-style object labels for filtering and organization. | +### Protobuf API and storage boundaries + +Public RPC contracts and durable protobuf formats have separate ownership. The +`openshell.v1.OpenShell` service currently has 74 RPCs. Their request and +response roots, streaming flags, and transitive message closure come from the +public descriptor set generated by `openshell-core`; a fingerprint test in +`openshell-server` requires this inventory to be reviewed whenever it changes. +Compute-driver, credential-driver, gateway-interceptor, and +supervisor-middleware services are compiled contracts for internal extension +boundaries, not public gateway RPCs. The current public inventory has 74 +methods, 278 messages, and 12 enums +(`c95ae90962c10fb28747db2b645adf4044562a3d208d84dfe4699d677e4364ee`). + +Storage-only messages live in the private, versioned +`openshell.storage.v1` package under `crates/openshell-server/proto`. The server +generates these types separately, so the public descriptor set and the Rust, +Go, Python, and TypeScript client generation inputs do not advertise them. + +| Storage classification | Protobuf messages | Durable use | +|---|---|---| +| Encoded storage roots | `StoredProviderCredentialRefreshState`, `StoredProviderProfile`, `PolicyRevisionPayload`, `DraftChunkPayload` | Complete protobuf payload stored in an object row or a scoped policy row. | +| Nested storage-only type | `StoredRefreshMaterialDeletion` | Repeated child records inside provider refresh state. | +| SQL materializations | `StoredPolicyRevision`, `StoredDraftChunk` | Server-only typed results assembled from indexed columns and decoded payloads; not public RPC messages. | +| Public messages used directly as encoded storage roots | `Sandbox`, `SandboxWorkloadTemplate`, `Provider`, `Workspace`, `WorkspaceMember`, `SshSession`, `ServiceEndpoint` | The generated public type is also the persisted payload. `SshSession` is not in the current public RPC message closure. | +| Embedded encoded root | `SandboxPolicy` | Stored in policy rows and inside the JSON settings envelope. | + +The 12 encoded durable roots above have a closure of 81 messages and eight +enums (`920a5243dfb37ce709f0f562a47d17791a5ede90fd7f662ed01542abd60a0dfb`). +Its intersection with the public RPC closure contains 71 messages and eight +enums (`05add438ba041defc98d791038ae593d3f09352677cae43f2276d494205ce415`). +The descriptor-derived test owns these full inventories; the tables here record +the reviewed roots and classifications. + +| Dual-purpose encoded root | Current decision | +|---|---| +| `Sandbox` | Defer a storage twin; govern its complete dependency closure as durable. | +| `SandboxWorkloadTemplate` | Defer a storage twin; govern its complete dependency closure as durable. | +| `Provider` | Defer a storage twin; govern its complete dependency closure as durable. | +| `Workspace` | Defer a storage twin; govern its complete dependency closure as durable. | +| `WorkspaceMember` | Defer a storage twin; govern its complete dependency closure as durable. | +| `SshSession` | Defer a storage twin; govern its complete dependency closure as durable. | +| `ServiceEndpoint` | Defer a storage twin; govern its complete dependency closure as durable. | +| `SandboxPolicy` | Defer a storage twin; govern its complete dependency closure as durable. | + +The public/storage overlap is deliberate for the current format. Storage twins +for the public roots are deferred: introducing them would require a broad +conversion boundary, and Prost does not retain unknown fields through a +decode-and-reencode conversion. Each root therefore carries a reviewed decision +to remain dual-purpose, and its complete transitive dependency closure is also +a durable format. Important embedded dependencies include `ObjectMeta`, +`ProviderProfile`, `CredentialHandle`, `SandboxPolicy`, and +`NetworkPolicyRule`. Global and sandbox settings additionally store an encoded +`SandboxPolicy` inside their JSON envelope. + +Public API compatibility and storage compatibility are reviewed independently: + +- Public compatibility is evaluated from public service descriptors and SDK + generation inputs. Storage-only packages must never enter that closure. +- `openshell.storage.v1` is frozen. Its test fingerprint covers message names, + field numbers, cardinality, scalar wire types, referenced types, map-entry + shapes, and optional presence. Keep its decoder available and introduce a + new versioned package plus an explicit migration or fallback decoder for a + format change; never reuse removed tags or names. +- Checked-in synthetic byte fixtures were encoded with the former + `openshell.v1` declarations. Current storage types must continue to decode + them semantically, which proves the package move does not require a database + rewrite. Protobuf payload bytes do not encode a message's package name. +- A change to a dual-purpose public message or any transitive durable + dependency requires both public-wire review and storage-migration review. + Wire-incompatible changes require a migration or fallback decoder and a + fixture for the earlier format. +- Mixed-version writers are unsupported. An older Prost writer can discard + fields it does not know when it reads and rewrites a record, even when the + newer field is wire-compatible. + Common resources use generic helpers that derive `object_type`, `id`, `name`, and labels from protobuf metadata traits before encoding the full message into `payload`. Policy revisions and draft policy chunks use the same table but also @@ -336,31 +419,45 @@ This keeps the gateway data model portable across storage backends and leaves room for future stores that can provide the same object, label, version, and scope semantics. +For in-memory SQLite, the adapter retains a dedicated keepalive connection for +the store lifetime. Operational connection replacement therefore preserves the +shared in-memory schema and objects instead of creating an empty database. + The SQLite adapter tightens the on-disk database file to mode `0o600` on every connect so that provider API keys, SSH session tokens, and sandbox metadata are not readable by other local users on shared hosts. The same restriction is reapplied to the `-wal` and `-shm` sidecars (created by SQLite's default WAL journal mode), which mirror the same sensitive contents. -Persisted state includes sandboxes, providers, provider credential refresh -state, SSH sessions, policy revisions, settings, inference configuration, and -deployment records. Provider refresh state is stored as a separate object -scoped to the provider instance through `objects.scope`. Its non-secret -configuration remains inline, while refresh tokens, client secrets, private -keys, and other secret source material are stored through the active credential -driver and represented by opaque handles. The provider record keeps only the -current injectable credential handles and optional per-credential expiry -timestamps. A refresh normally mints one credential, but a strategy may -co-mint several (AWS STS mints the access key, secret key, and session token in -one call); the refresh state pins the resolved set of env keys it owns so -collision checks reserve all of them before the first mint. Provider records -keep inline credential values only for legacy records created before credential -driver storage. New provider and refresh-material writes keep driver-owned -credential handles. When no external credential driver is configured, gateways -use server-owned encrypted database credential storage for defense in depth. -Multi-replica deployments can use that default with a shared database and -shared key-encryption key, or opt into an external backend such as Vault or -Kubernetes Secrets. +Persisted state includes sandboxes, providers, provider profiles, provider +credential refresh state, SSH sessions, policy revisions, settings, deployment +records, and reusable sandbox workload templates. Provider refresh +state is stored as a separate object scoped to the provider instance through +`objects.scope`. Its non-secret configuration remains inline, while refresh +tokens, client secrets, private keys, and other secret source material are +stored through the active credential driver and represented by opaque handles. +The provider record keeps only the current injectable credential handles and +optional per-credential expiry timestamps. A refresh normally mints one +credential, but a strategy may co-mint several (AWS STS mints the access key, +secret key, and session token in one call); the refresh state pins the resolved +set of env keys it owns so collision checks reserve all of them before the +first mint. Provider records keep inline credential values only for legacy +records created before credential driver storage. New provider and +refresh-material writes keep driver-owned credential handles. When no external +credential driver is configured, gateways use server-owned encrypted database +credential storage for defense in depth. Multi-replica deployments can use that +default with a shared database and shared key-encryption key, or opt into an +external backend such as Vault or Kubernetes Secrets. + +Sandbox workload templates are workspace-scoped gateway resources. Workspace +admins create and delete them; workspace users can read and list them. A +template owns reusable workload intent: image, environment, CPU and memory +limits, GPU request, driver-specific config, and service-level hints. A sandbox +created from a template resolves that resource once and persists an ordinary +`SandboxSpec` snapshot. The create request still owns per-sandbox governance: +name, labels, annotations, provider attachments, and policy. The sandbox stores +template provenance as the template name and resource version used for the +snapshot, so later template edits or deletes do not mutate existing sandboxes. OAuth refresh failures retain a gateway-owned recovery classification alongside the refresh state. The gateway reads only a bounded error response and maps @@ -450,7 +547,6 @@ coverage: | Sandbox | `MustCreate` | `update_message_cas` | `list_messages` | | Provider | `MustCreate` | `update_message_cas` | `list_messages` | | ProviderProfile | `MustCreate` | `MatchResourceVersion` | `list_messages` | -| InferenceRoute | `MustCreate` | `update_message_cas` | `list_messages` | | SandboxPolicy | scoped versioning | scoped versioning | scoped query | | Settings | `Mutex`-guarded | `Mutex`-guarded | single-row | @@ -512,69 +608,25 @@ configuration, valid endpoint-bound static credentials from other attached providers, and the dynamic credential snapshot. Provider environment revisions include profile endpoint and binding changes. -## Inference Resolution +## Provider Environment Resolution -Cluster inference routes store only `provider_name`, `model_id`, and optional -timeout. The gateway resolves endpoint URLs, protocols, credentials, auth -style, and route-shaping metadata from the provider record when supervisors call -`GetInferenceBundle`. Supported provider types for cluster inference are -`openai`, `anthropic`, `nvidia`, `deepinfra`, and `google-vertex-ai`. +The gateway resolves only the providers attached to a sandbox. It combines each +provider instance with its profile, returns non-secret configuration, and marks +credentials with the profile's host, port, and path boundaries. The supervisor +uses those bindings when it replaces credential placeholders in policy-allowed +requests. -The bundle carries enough information for sandbox-local routers to construct -upstream URLs without re-deriving provider-specific routing logic. Each resolved -route may include: - -| Field | Meaning | -|---|---| -| `model_in_path` | When true, the model identifier is part of the upstream URL path, not only the request body. | -| `request_path_override` | Path override or suffix. With `model_in_path=false`, replaces the protocol-derived path; with `model_in_path=true`, appended after the model ID. | - -For standard providers these fields stay unset and the sandbox router uses default -protocol paths. Vertex AI is model-aware: the gateway constructs the base URL -from provider config (`VERTEX_AI_PROJECT_ID`, `VERTEX_AI_REGION`, optional -`VERTEX_AI_PUBLISHER`) and emits route-shaping metadata so the sandbox router -stays provider-agnostic. - -Host selection follows the configured region: - -| Region value | Vertex host | -|---|---| -| `global` | `aiplatform.googleapis.com` | -| `us` or `eu` | `aiplatform.{region}.rep.googleapis.com` | -| Any other (e.g. `us-central1`) | `{region}-aiplatform.googleapis.com` | - -Route shaping by publisher: - -- **Anthropic (Claude)** — `model_in_path=true`, base path under - `publishers/anthropic/models`, protocol `anthropic_messages` only. The gateway - resolves `request_path_override=:rawPredict`; the sandbox router keeps - `:rawPredict` for buffered requests and upgrades to `:streamRawPredict` only - for streaming proxy calls. -- **All other models** (Gemini, third-party, unknown) — OpenAI-compatible - `.../endpoints/openapi` base with `request_path_override=/chat/completions`; - protocol `openai_chat_completions`. - -Callers may supply `GOOGLE_VERTEX_AI_BASE_URL` or `VERTEX_AI_BASE_URL` only for -non-Anthropic routes. Anthropic base URL overrides are rejected because they -cannot safely preserve model-path shaping and `anthropic_version` body -adaptation. Overrides still pin `request_path_override=/chat/completions` and -must use `https` with an official Vertex AI hostname (`aiplatform.googleapis.com`, -`aiplatform.{us,eu}.rep.googleapis.com`, or `{region}-aiplatform.googleapis.com`). - -Header passthrough is protocol-dependent. Vertex Claude rawPredict routes strip -client `anthropic-beta` headers; `anthropic-version` is not forwarded because -the sandbox router injects `anthropic_version` into the request body for Vertex -rawPredict. Non-Anthropic Vertex routes do not inherit Anthropic passthrough -headers. +Model selection, API protocol, request and response shapes, streaming behavior, +and endpoint URL construction remain responsibilities of the workload's native +client. The gateway does not parse or transform model API requests. For `google-vertex-ai` providers created with CLI `--from-gcloud-adc`, the CLI calls gateway `ConfigureProviderRefresh` with OAuth2 refresh material from gcloud ADC, then `RotateProviderCredential` to mint the first access token before reporting success. ADC-backed providers mint into `GOOGLE_VERTEX_AI_TOKEN`. A successful create therefore yields an immediately usable provider; failures roll -back the provider record. Service-account JSON and private keys are gateway-side -refresh bootstrap material only; sandbox runtime inference receives minted -access tokens, not raw service-account material. +back the provider record. Service-account JSON and private keys remain gateway-side +refresh bootstrap material; sandboxes receive minted access tokens instead. ## Supervisor Relay @@ -746,9 +798,22 @@ identity and configured compute driver. The gateway forwards OTLP configuration, its configured gateway name, and W3C trace context to managed external drivers. Built-in drivers use dedicated -in-process providers that preserve the same RPC trace boundary. Each driver +in-process providers that preserve the same RPC trace boundary. A shared +compute-driver tracing descriptor derives provider layers and standalone +installation from each driver's service identity, and routes both the shared +RPC boundary target and that driver's crate target prefix. A shared RPC tracer +owns unary and streaming boundary outcomes. Each driver exports to the configured collector under its own service name and carries the -gateway name as a resource attribute. +gateway name and configured compute driver as resource attributes. +Compute-driver client and server spans share the fully qualified protobuf +operation name, such as `openshell.compute.v1.ComputeDriver/CreateSandbox`, +in both the span name and `rpc.method`; the current RPC semantic conventions +integrate the service into that fully qualified method and do not emit +`rpc.service`. `service.name` and span kind distinguish the two sides. +Backend-prefixed spans describe implementation work beneath that boundary. +Streaming `WatchSandboxes` spans remain open for the stream lifetime on both +sides. A terminal stream status records its outcome; +consumer teardown without a terminal status leaves the span status unset. Two invariants shape the failure behavior. Telemetry is diagnostic, so no OTLP failure stops the gateway from serving: a malformed endpoint is logged at diff --git a/architecture/google-vertex-ai-provider.md b/architecture/google-vertex-ai-provider.md index aac0c11c2..8779bd287 100644 --- a/architecture/google-vertex-ai-provider.md +++ b/architecture/google-vertex-ai-provider.md @@ -1,544 +1,98 @@ -# Google Vertex AI Provider — Implementation Reference +# Google Vertex AI Provider -This document covers the full implementation of the `google-vertex-ai` provider in -OpenShell. It is the canonical reference for maintainers working on anything in the -Vertex AI request path, from CLI through gateway to sandbox. +The `google-vertex-ai` provider gives selected sandboxes direct access to +Google Vertex AI endpoints without exposing long-lived Google credentials. +The provider profile owns endpoint policy and credential metadata; the +sandbox attachment owns which workload receives access. ---- +## Boundaries -## 1. Overview - -OpenShell's `google-vertex-ai` provider routes `inference.local` traffic through -Google Cloud's Vertex AI platform. It differs from a direct Anthropic or OpenAI -integration in two ways that touch nearly every layer of the stack: - -1. **Authentication is OAuth2 bearer, not a static API key.** Vertex AI accepts - short-lived GCP access tokens (`ya29.*`) as `Authorization: Bearer` headers. The - gateway mints and rotates these tokens from one of two refresh sources: a GCP - service account key (JWT-bearer grant) or gcloud Application Default Credentials - (OAuth2 refresh-token grant). - -2. **The URL and wire format depend on the model family.** Anthropic Claude models - use Vertex AI's native rawPredict surface (`/publishers/anthropic/models/{model}:rawPredict`) - with the Anthropic Messages API body shape. Gemini and all other models use Vertex - AI's OpenAI-compatible Chat Completions surface - (`/v1beta1/.../endpoints/openapi/chat/completions`). The gateway selects the right - route at `openshell inference set` time based on the model name (or the explicit - `VERTEX_AI_PUBLISHER` config key). - -### Canonical provider type - -The canonical provider type string is `google-vertex-ai`. The following aliases are -accepted everywhere and normalized to the canonical string: - -| Input | Resolves to | +| Component | Responsibility | |---|---| -| `google-vertex-ai` | `google-vertex-ai` | -| `vertex` | `google-vertex-ai` | -| `vertex-ai` | `google-vertex-ai` | -| `google-vertex` | `google-vertex-ai` | -| `gcp-vertex` | `google-vertex-ai` | +| CLI | Discovers ADC or accepts service-account bootstrap material and creates the provider record. | +| Gateway | Stores refresh material through the credential driver and rotates short-lived access tokens. | +| Provider profile | Declares Vertex hosts, credential aliases, refresh constraints, and permitted binaries. | +| Sandbox supervisor | Delivers opaque credential placeholders and resolves them only for profile-authorized Vertex requests. | +| Workload | Selects the native Vertex endpoint, model, request format, streaming mode, and timeout. | -Alias resolution lives in `openshell_core::inference::normalize_inference_provider_type` -and is the single source of truth shared by `openshell-server`, `openshell-providers`, -and the CLI. +The provider does not select a model or transform requests. Anthropic Claude +uses Vertex's publisher-model `rawPredict` or `streamRawPredict` paths. Gemini +and other models use their documented native or OpenAI-compatible Vertex API. ---- +## Credential Flows -## 2. Architecture — How the Pieces Fit Together +Vertex accepts short-lived Google OAuth2 access tokens. Both supported setup +flows converge on a rotating access token stored in the provider record. -``` -CLI (openshell provider create / openshell inference set) - │ - ├── read_gcloud_adc() reads ~/.config/gcloud/application_default_credentials.json - ├── CreateProviderRequest persists provider object (type, credentials, config) - └── ConfigureProviderRefreshRequest registers a refresh state record +### Service account -Gateway (openshell-server) - │ - ├── provider_refresh worker background loop that rotates access tokens - │ ├── mint_oauth2_refresh_token() for gcloud ADC flow - │ └── mint_google_service_account_jwt() for service account key flow - │ - ├── SetClusterInferenceRequest - │ └── resolve_vertex_ai_route() builds RouterResolvedRoute - │ ├── infer_vertex_publisher() model → publisher - │ └── vertex_location_and_host() region → Vertex API host - │ - └── GetInferenceBundleRequest (from sandbox on connect) - └── resolve_route_by_name() re-resolves live route+credentials +The operator creates the provider with service-account bootstrap material and +configures `google-service-account-jwt` refresh. The private key remains in the +gateway credential store. The gateway mints +`GOOGLE_VERTEX_AI_SERVICE_ACCOUNT_TOKEN` and refreshes it before expiry. -Router (openshell-router) - │ - ├── proxy_with_candidates_streaming() - │ ├── build_provider_url() appends model/:rawPredict or /chat/completions - │ ├── sanitize_request_headers() strips auth, strips anthropic-beta for rawPredict - │ └── prepare_backend_request() - │ ├── bearer_auth(access_token) Authorization: Bearer ya29.* - │ ├── remove "model" from body (rawPredict only — model is in the URL) - │ └── inject "anthropic_version" (rawPredict only — must be in body, not header) - │ - └── proxy_to_backend() / proxy_to_backend_streaming() +### gcloud ADC -Sandbox (inference.local) - └── agent connects to inference.local → gateway proxy → Vertex AI +For local development, `--from-gcloud-adc` reads an authorized-user ADC file, +stores the refresh grant at the gateway, and mints +`GOOGLE_VERTEX_AI_TOKEN`. The ADC file and refresh token do not enter the +sandbox. + +## Runtime Data Flow + +1. The operator attaches the provider to a sandbox. +2. The effective policy includes the profile's Vertex endpoint and binary + rules. +3. A newly launched workload receives the current token as an opaque + placeholder plus non-secret project and region configuration. +4. The workload calls the native Vertex endpoint with that placeholder in the + `Authorization: Bearer` header. +5. After policy and endpoint binding pass, the proxy substitutes the current + real access token and forwards the request. +6. Token refresh updates the resolver; the workload continues using the same + placeholder. + +The raw `GOOGLE_SERVICE_ACCOUNT_KEY` credential is bootstrap-only and is never +part of sandbox runtime material. + +## Endpoint Boundary + +The built-in profile permits official Vertex hosts: + +- `-aiplatform.googleapis.com` +- `aiplatform.googleapis.com` +- `aiplatform.us.rep.googleapis.com` +- `aiplatform.eu.rep.googleapis.com` + +The workload constructs the documented project, location, publisher, and model +path. The proxy does not infer the publisher or rewrite the body. + +For Claude on Vertex, a request uses this form: + +```text +https://-aiplatform.googleapis.com/v1/projects//locations//publishers/anthropic/models/:rawPredict ``` -Key crates and their roles: - -| Crate | Role | -|---|---| -| `openshell-core` (`inference.rs`) | Canonical type aliases, profile constants, URL alias resolution | -| `openshell-providers` | Environment-based credential discovery for `--from-existing` | -| `openshell-server` (`inference.rs`) | Route resolution: maps provider + model → `RouterResolvedRoute` | -| `openshell-server` (`provider_refresh.rs`) | Credential refresh worker; mints access tokens | -| `openshell-router` (`backend.rs`) | Proxy: URL construction, header sanitization, body rewriting | -| `openshell-cli` (`run.rs`) | `provider create` and `--from-gcloud-adc` CLI flow | -| `providers/google-vertex-ai.yaml` | Provider type profile: credential keys, refresh strategy, endpoints | - ---- - -## 3. Credential Model — Two Flows - -Vertex AI accepts only short-lived GCP access tokens. The gateway never sends the raw -service account JSON or gcloud ADC material to Vertex AI. Both flows converge on the -same runtime secret: a `ya29.*` token stored under one of these credential keys, -searched in priority order: - -``` -GOOGLE_VERTEX_AI_SERVICE_ACCOUNT_TOKEN -VERTEX_AI_SERVICE_ACCOUNT_TOKEN -GOOGLE_VERTEX_AI_TOKEN -VERTEX_AI_TOKEN -``` - -These names are defined in `openshell_core::inference::VERTEX_AI_CREDENTIAL_KEY_NAMES` -and are shared between the profile, the CLI, and the route resolver. - -### 3a. Service Account Key Flow (production) - -``` -User: openshell provider create --type google-vertex-ai \ - --credential GOOGLE_SERVICE_ACCOUNT_KEY="$(cat key.json)" \ - --config VERTEX_AI_PROJECT_ID=my-project \ - --config VERTEX_AI_REGION=us-central1 - -User: openshell provider refresh configure vertex-prod \ - --credential-key GOOGLE_VERTEX_AI_SERVICE_ACCOUNT_TOKEN \ - --strategy google-service-account-jwt \ - --material client_email="sa@..." \ - --material private_key="..." \ - --secret-material-key private_key - -Gateway: mint_google_service_account_jwt() - 1. build JWT claims: iss=client_email, scope=cloud-platform, aud=token_url - 2. sign with RS256 using private_key - 3. POST assertion to https://oauth2.googleapis.com/token - 4. store access_token as GOOGLE_VERTEX_AI_SERVICE_ACCOUNT_TOKEN - 5. schedule next refresh 300 s before expiry (max 3600 s lifetime) -``` - -The raw `GOOGLE_SERVICE_ACCOUNT_KEY` is stored as bootstrap material for the refresh -worker. It is never exposed to sandboxes; the sandbox only ever sees the short-lived -access token. - -### 3b. gcloud Application Default Credentials Flow (local dev) - -``` -User: gcloud auth application-default login -User: openshell provider create --type google-vertex-ai \ - --from-gcloud-adc \ - --config VERTEX_AI_PROJECT_ID=my-project - -CLI: read_gcloud_adc() - checks GOOGLE_APPLICATION_CREDENTIALS → $CLOUDSDK_CONFIG/... → ~/.config/gcloud/adc.json - validates type == "authorized_user" (rejects service_account — different flow) - extracts client_id, client_secret, refresh_token - - configure_provider_refresh(strategy=oauth2_refresh_token, ...) - rotate_provider_credential(...) ← mints first token immediately - -Gateway: mint_oauth2_refresh_token() - POST to https://oauth2.googleapis.com/token - grant_type=refresh_token, client_id, client_secret, refresh_token - stores access_token as GOOGLE_VERTEX_AI_TOKEN - if response includes a new refresh_token, rotates it in state -``` - -The `--from-gcloud-adc` flag is rejected for any provider type other than -`google-vertex-ai`. The CLI validates and reads the ADC file before creating the -provider, so a missing or malformed ADC file results in a clean error with no orphaned -gateway state. - -### 3c. Refresh Worker - -`provider_refresh.rs` runs a background tokio task (`spawn_refresh_worker`) that -sweeps all `StoredProviderCredentialRefreshState` records on a configurable interval. -For each record where `next_refresh_at_ms <= now` or `status == "rotation_requested"`, -it calls `refresh_provider_credential`, which calls `mint_credential` and then -`apply_minted_credential`. The minted access token is written back to the provider's -`credentials` map under the configured `credential_key` via a CAS update. - -Key timing constants: - -- Default `refresh_before_seconds`: 300 (refresh 5 minutes before expiry) -- Default `max_lifetime_seconds`: 3600 (token lifetime cap) -- Error retry interval: 60 seconds - ---- - -## 4. Route Resolution — From Provider to RouterResolvedRoute - -When `openshell inference set --provider --model ` is called, the server -runs `resolve_vertex_ai_route` in `openshell-server/src/inference.rs`. This function -produces a `RouterResolvedRoute` that the router uses verbatim for every proxied -request. - -### 4a. Publisher Inference - -`infer_vertex_publisher(model_id)` maps model name prefixes to Vertex AI publishers: - -| Prefix | Publisher | Routing | -|---|---|---| -| `claude-*` | `anthropic` | Anthropic Messages API (rawPredict) | -| `gemini-*`, `text-bison-*`, `chat-bison-*` | `google` | OpenAI-compat Chat Completions | -| `llama-*` | `meta` | OpenAI-compat Chat Completions | -| `mistral-*`, `codestral-*` | `mistralai` | OpenAI-compat Chat Completions | -| `jamba-*` | `ai21` | OpenAI-compat Chat Completions | -| `deepseek-*` | `deepseek` | OpenAI-compat Chat Completions | -| (unrecognized) | `None` | OpenAI-compat Chat Completions | - -Only the `anthropic` result changes routing; all non-Anthropic publishers use the same -OpenAI-compatible Vertex surface. The `VERTEX_AI_PUBLISHER` config key overrides -inference: set it to `anthropic` to force rawPredict for a non-standard model name. - -### 4b. Host Resolution - -`vertex_location_and_host(region)` maps the `VERTEX_AI_REGION` config value to a -Vertex API host: - -| Region value | Host | -|---|---| -| `global` | `aiplatform.googleapis.com` | -| `us` | `aiplatform.us.rep.googleapis.com` | -| `eu` | `aiplatform.eu.rep.googleapis.com` | -| `us-central1`, `europe-west4`, etc. | `-aiplatform.googleapis.com` | - -Default region when not set: `us-central1`. - -### 4c. Endpoint and Protocol Selection - -**Anthropic (Claude) models:** - -``` -endpoint = https://{host}/v1/projects/{project}/locations/{location}/publishers/anthropic/models -protocol = ["anthropic_messages"] -model_in_path = true -request_path_override = ":rawPredict" -``` - -The model ID is NOT embedded in the endpoint URL. It is stored in `route.model` and -appended by `build_provider_url` at proxy time: `{endpoint}/{model}:rawPredict` for -buffered requests, `{endpoint}/{model}:streamRawPredict` for streaming. - -**Non-Anthropic models (Gemini, Llama, Mistral, etc.):** - -``` -endpoint = https://{host}/v1beta1/projects/{project}/locations/{location}/endpoints/openapi -protocol = ["openai_chat_completions"] -model_in_path = false -request_path_override = "/chat/completions" -``` - -**Base URL override (escape hatch, non-Anthropic only):** -When `GOOGLE_VERTEX_AI_BASE_URL` or `VERTEX_AI_BASE_URL` is set: - -- `GOOGLE_VERTEX_AI_BASE_URL` takes priority over `VERTEX_AI_BASE_URL` -- Rejected with `InvalidArgument` for Anthropic models — Anthropic routes require - model-path shaping that a bare URL override cannot preserve safely -- Must be `https://`, no IP literals, no userinfo, port 443 only if explicit, must - target an official Vertex AI hostname (pattern validated by `is_allowed_vertex_override_host`) - -### 4d. Credential Lookup - -For Vertex AI specifically, `find_provider_api_key` uses `CredentialLookup::PreferredOnly`: -it searches only the four `VERTEX_AI_CREDENTIAL_KEY_NAMES` keys and returns `None` if -none match. This prevents the raw service account JSON stored under -`GOOGLE_SERVICE_ACCOUNT_KEY` from being mistakenly used as a bearer token, which would -produce a confusing auth failure from Vertex AI. - -### 4e. Model ID Validation - -Before any URL is constructed, `validate_vertex_model_id` rejects model IDs that -contain path separators (`/`, `\`), traversal segments (`..`), URL delimiters (`?`, -`#`, `%`), control characters, or surrounding whitespace. This is defense-in-depth -against injection into the URL path that appears in Anthropic rawPredict routes. - ---- - -## 5. Request Proxying — Backend Transformations - -The router (`openshell-router/src/backend.rs`) applies four Vertex-specific -transformations on every proxied request: - -### 5a. URL Construction - -`build_provider_url` is called with `model_in_path` and `request_path_override` from -the resolved route: - -- **Anthropic buffered:** `{endpoint}/{model_id}:rawPredict` -- **Anthropic streaming:** `{endpoint}/{model_id}:streamRawPredict` - (`:rawPredict` suffix is upgraded to `:streamRawPredict` when `stream_response=true`) -- **OpenAI-compat:** `{endpoint}/chat/completions` - -### 5b. Authentication - -All Vertex AI routes use `AuthHeader::Bearer`. The router injects -`Authorization: Bearer {access_token}` where `access_token` is the `ya29.*` token -read from the provider's credentials at route resolution time. - -### 5c. Header Sanitization — Stripping `anthropic-beta` - -For rawPredict routes (`is_vertex_anthropic_rawpredict_route`), `sanitize_request_headers` -strips the `anthropic-beta` header even though it is in the route's `passthrough_headers` -list. Vertex AI's rawPredict endpoint rejects requests that include `anthropic-beta` with -HTTP 400. Beta feature enablement for Vertex AI is controlled through Google Cloud -(Model Garden access), not HTTP headers. Claude Code always sends `anthropic-beta` flags; -stripping them here prevents spurious 400 errors. - -Direct Anthropic API routes (non-Vertex) still forward `anthropic-beta` unchanged. - -### 5d. Body Rewriting — `model` and `anthropic_version` - -For rawPredict routes, `prepare_backend_request` rewrites the JSON request body: - -- **Removes `"model"` field.** Vertex AI rawPredict encodes the model in the URL path. - Sending `"model"` in the body causes HTTP 400 "Extra inputs are not permitted". Claude - Code and other Anthropic SDK clients always include `"model"` in the body; the router - strips it unconditionally for rawPredict routes. - -- **Injects `"anthropic_version": "vertex-2023-10-16"`.** The standard Anthropic API - sends this as the `anthropic-version` request header. Vertex AI's rawPredict expects - it as a JSON body field instead. The router injects it only when the client has not - already sent it (`!obj.contains_key("anthropic_version")`). The constant - `VERTEX_ANTHROPIC_VERSION = "vertex-2023-10-16"` is the Google-published value. - -For OpenAI-compatible routes, the standard model rewrite applies: `"model"` in the body -is overwritten with `route.model` (the model ID configured at `openshell inference set` -time). - ---- - -## 6. Configuration Keys - -These keys are set at `openshell provider create` time via `--config KEY=VALUE` and -stored in the provider's `config` map. They are re-read on every bundle resolution -(i.e. on every sandbox connect), so changing them with `openshell provider update` takes -effect for new sandboxes without restarting the gateway. - -| Key | Required | Default | Description | -|---|---|---|---| -| `VERTEX_AI_PROJECT_ID` | Yes (unless base URL override set) | — | GCP project ID. Must be 6–30 chars, lowercase letters/digits/hyphens, no leading/trailing hyphen. | -| `VERTEX_AI_REGION` | No | `us-central1` | GCP region or `global`/`us`/`eu`. Determines the Vertex API host. | -| `GOOGLE_VERTEX_AI_BASE_URL` | No | — | Full base URL override for non-Anthropic routes. Takes priority over `VERTEX_AI_BASE_URL`. | -| `VERTEX_AI_BASE_URL` | No | — | Backward-compatible alias for `GOOGLE_VERTEX_AI_BASE_URL`. | -| `VERTEX_AI_PUBLISHER` | No | Inferred from model name | Set to `anthropic` to force rawPredict routing. | - -Config key constants are defined in `openshell_core::inference`: -`VERTEX_AI_PROJECT_ID_KEY`, `VERTEX_AI_REGION_KEY`, `VERTEX_AI_PUBLISHER_KEY`. - -The full list of config keys scanned during `--from-existing` discovery is -`VERTEX_AI_CONFIG_KEY_NAMES` in the same module. - ---- - -## 7. Credential Keys and the Provider Profile - -`providers/google-vertex-ai.yaml` is the authoritative provider type profile. It -defines: - -- **`service_account_key` (`GOOGLE_SERVICE_ACCOUNT_KEY`):** The raw service account JSON. - This is gateway-side bootstrap material, not a sandbox credential. Not injected into - sandboxes. `required: false` because some deployments use gcloud ADC instead. - -- **`service_account_token` (`GOOGLE_VERTEX_AI_SERVICE_ACCOUNT_TOKEN`, - `VERTEX_AI_SERVICE_ACCOUNT_TOKEN`):** Short-lived access token minted from the service - account key via `google_service_account_jwt` refresh strategy. `auth_style: bearer`. - Refreshed 300 s before expiry; max lifetime 3600 s; scope - `https://www.googleapis.com/auth/cloud-platform`. - -- **`gcloud_adc_token` (`GOOGLE_VERTEX_AI_TOKEN`, `VERTEX_AI_TOKEN`):** Short-lived - access token minted from gcloud ADC via `oauth2_refresh_token` refresh strategy. - Same `auth_style: bearer`, same timing. - -The `discovery` section lists `[service_account_token, gcloud_adc_token]` as the two -credential sources the gateway will scan during `--from-existing`. - -The `endpoints` section enumerates all Vertex AI API hosts that sandbox network -policies must permit when `providers_v2_enabled=true`: - -- `*-aiplatform.googleapis.com:443` (regional endpoints) -- `aiplatform.googleapis.com:443` (global endpoint) -- `aiplatform.us.rep.googleapis.com:443` (US multi-region) -- `aiplatform.eu.rep.googleapis.com:443` (EU multi-region) - ---- - -## 8. `openshell-providers` — Discovery Plugin - -`openshell-providers` handles credential discovery for `--from-existing`. There is no -dedicated `vertex.rs` plugin file because Vertex AI token discovery is profile-driven: -the gateway reads the `providers/google-vertex-ai.yaml` profile and scans the -credential env vars listed there. - -Vertex AI config keys (`VERTEX_AI_PROJECT_ID`, `VERTEX_AI_REGION`, etc.) are not listed -in the profile's `discovery.credentials` section, so they are scanned separately in -`discover_existing_provider_data` in the CLI: - -```rust -if provider_type == VERTEX_AI_PROVIDER_TYPE { - for key in openshell_core::inference::VERTEX_AI_CONFIG_KEY_NAMES { - if let Ok(val) = std::env::var(key) { ... } - } -} -``` - -Provider type normalization for the `ProviderRegistry` (non-inference providers like -`claude-code`, `github`, `gitlab`) is handled by `normalize_provider_type` in -`openshell-providers/src/lib.rs`, which delegates Vertex AI aliases to -`normalize_inference_provider_type` in `openshell-core`. - ---- - -## 9. Inference Routing in the Sandbox - -When a sandbox agent connects to `https://inference.local`, the sandbox fetches the -inference bundle from the gateway (`GetInferenceBundleRequest`). The bundle contains one -or more `ResolvedRoute` proto messages built by `resolve_route_by_name`. For a Vertex AI -route the bundle contains: - -``` -ResolvedRoute { - name: "inference.local", - base_url: "https://us-central1-aiplatform.googleapis.com/v1/projects/.../publishers/anthropic/models", - model_id: "claude-sonnet-4-20250514", - api_key: "ya29.", - protocols: ["anthropic_messages"], - provider_type: "google-vertex-ai", - model_in_path: true, - request_path_override: ":rawPredict", -} -``` - -The sandbox proxy uses this bundle to configure the local `inference.local` route. The -bundle is re-fetched on reconnect, which picks up rotated access tokens automatically -without sandbox restart. - -The gateway does NOT expose GCP credentials (project ID, region, service account key) -in the bundle. Sandboxes see only the short-lived access token. - ---- - -## 10. Sandbox Usage Pattern - -Agents inside sandboxes connect to Vertex AI through `inference.local`. The correct -setup differs by model family: - -**Claude (Anthropic Messages API):** - -```sh -ANTHROPIC_BASE_URL="https://inference.local" ANTHROPIC_API_KEY=unused claude --bare -``` - -The `ANTHROPIC_API_KEY` value is stripped by the gateway and replaced with the real -GCP token. `--bare` skips Claude Code's OAuth flow. Do NOT set -`CLAUDE_CODE_USE_VERTEX=1` inside a sandbox — that makes Claude Code connect directly -to Vertex AI and attempt GCP ADC discovery, which fails in the sandbox environment. - -**Gemini / other models (OpenAI-compat):** -Point the SDK's base URL at `https://inference.local/v1` and use any non-empty value -as the API key. - -**Common sandbox policy denials to expect:** - -- `metadata.google.internal:80` — resolves to `169.254.169.254` (GCE metadata service). - Always blocked by the proxy unconditionally. -- `downloads.claude.ai:443` — Claude Code update checking. Block or approve per policy. -- `storage.googleapis.com:443` — GCS access. Optional; approve if the agent needs it. - ---- - -## 11. Key Files Reference - -| File | Purpose | -|---|---| -| `providers/google-vertex-ai.yaml` | Provider type profile: credential keys, refresh strategy params, allowed endpoints | -| `docs/providers/google-vertex-ai.mdx` | User-facing documentation | -| `crates/openshell-core/src/inference.rs` | Canonical provider type aliases, `InferenceProviderProfile`, `VERTEX_AI_*` constants, auth header logic | -| `crates/openshell-server/src/inference.rs` | Route resolution: `resolve_vertex_ai_route`, `infer_vertex_publisher`, `vertex_location_and_host`, model ID validation | -| `crates/openshell-server/src/provider_refresh.rs` | Refresh worker: `mint_google_service_account_jwt`, `mint_oauth2_refresh_token`, `apply_minted_credential` | -| `crates/openshell-router/src/backend.rs` | Proxy engine: `build_provider_url`, `sanitize_request_headers`, `prepare_backend_request` (model strip + anthropic_version inject) | -| `crates/openshell-router/src/config.rs` | `ResolvedRoute` struct with `model_in_path`, `request_path_override` fields | -| `crates/openshell-cli/src/run.rs` | `provider_create`, `read_gcloud_adc`, `rollback_provider_create_after_vertex_adc_failure` | -| `crates/openshell-providers/src/lib.rs` | `ProviderRegistry`, `normalize_provider_type`, `discover_existing_provider_data` | - ---- - -## 12. Maintenance Notes - -### Adding a new Vertex AI region - -No code changes are needed for standard regional endpoints (`-aiplatform.googleapis.com`). -`vertex_location_and_host` constructs the host from the region string dynamically. For -new special-case multi-region endpoints (analogous to `us` and `eu`), update the match -arm in `vertex_location_and_host` in `openshell-server/src/inference.rs`. - -Add the new host pattern to the `endpoints` list in `providers/google-vertex-ai.yaml` -so it is included in sandbox network policy injection. - -### Adding a new model family publisher - -Add a prefix match arm to `infer_vertex_publisher` in `openshell-server/src/inference.rs`. -Unless the new publisher uses a different wire format than the OpenAI-compatible -endpoint, no other changes are needed — all non-Anthropic publishers currently route -to the same `endpoints/openapi/chat/completions` surface. - -If the new publisher requires a separate rawPredict-style surface (like Anthropic), -update `resolve_vertex_ai_route` to add a new branch for that publisher, and add the -corresponding body transformation logic in `prepare_backend_request` in -`openshell-router/src/backend.rs`. - -### Updating `VERTEX_ANTHROPIC_VERSION` - -The constant `VERTEX_ANTHROPIC_VERSION = "vertex-2023-10-16"` in -`openshell-router/src/backend.rs` is the version string Google requires in the body of -rawPredict requests. Update it when Google publishes a new required version for the -Vertex AI Anthropic API. Check the -[Google Cloud Claude documentation](https://cloud.google.com/vertex-ai/generative-ai/docs/partner-models/use-claude) -for the current required value. - -### Rotating the GCP OAuth2 token endpoint URL - -`google_token_url` in `provider_refresh.rs` defaults to `https://oauth2.googleapis.com/token` -when `state.token_url` is empty. The `providers/google-vertex-ai.yaml` profile sets this -in the `refresh.token_url` field. If Google changes the token endpoint, update both: - -- `refresh.token_url` in `providers/google-vertex-ai.yaml` -- The default in `google_token_url` in `provider_refresh.rs` - -### Changing the `cloud-platform` OAuth scope - -The scope `https://www.googleapis.com/auth/cloud-platform` is set in the -`refresh.scopes` list in `providers/google-vertex-ai.yaml`. The gateway uses it when -constructing JWT claims and OAuth2 refresh requests. If a narrower scope becomes -available (e.g. a Vertex AI-specific scope), update the profile's `scopes` list. - -### Adding a new Vertex AI body invariant - -Body transformations for rawPredict live in `prepare_backend_request` in -`openshell-router/src/backend.rs`, gated by `is_vertex_anthropic_rawpredict_route`. -When adding a new transformation: - -1. Add the transformation in the `needs_vertex_anthropic_version` branch. -2. Add a wiremock-based integration test directly below the existing tests in that file. -3. Ensure the new transformation does not apply to standard Anthropic or OpenAI routes. +The request body includes the Vertex Anthropic API version expected by Google. +Streaming uses the corresponding native streaming endpoint. + +## Invariants + +- Refresh bootstrap material remains gateway-only. +- Sandboxes receive placeholders, never real access tokens. +- A token resolves only at endpoints covered by the attached provider profile. +- Detach and expiry revoke placeholder resolution. +- Project IDs, regions, publishers, and model IDs are non-secret workload + configuration. +- Provider attachment does not grant access when a gateway global policy + override suppresses provider-derived policy. +- Provider environment keys and dynamic credential bindings remain + unambiguous across all providers attached to one sandbox. + +## Operational Notes + +Attach or detach the provider with the normal sandbox provider lifecycle. A +running sandbox observes policy and resolver changes, but a process must be +launched after attachment to receive new environment variables. Direct native +requests provide the end-to-end verification path; provider creation does not +probe a model endpoint or validate a model ID. diff --git a/architecture/sandbox-limits.md b/architecture/sandbox-limits.md index 49f1ab7ce..9045ddc35 100644 --- a/architecture/sandbox-limits.md +++ b/architecture/sandbox-limits.md @@ -40,6 +40,15 @@ New limits should follow these rules: query parameters, or external free-form diagnostics. - Test time bounds with simulated time and test shared budgets under saturation. +## Gateway Sandbox Resources + +Gateway-owned sandbox resources also carry admission limits before they can +produce supervisor work. Reusable workload templates are capped at 1000 per +workspace. Template payloads reuse sandbox spec validation for environment +entry count and size, image and resource field sizes, driver-config serialized +size, and GPU count. Template names use the same DNS-style resource-name rules +as other named gateway resources. + ## Middleware Middleware limits are process-wide per sandbox. Registry replacement preserves @@ -112,21 +121,15 @@ middleware. A passed binary logical message still advances the active middleware session sequence and emits coverage telemetry, so a later text RPC can contain a valid sequence gap. -## Inference and Upstream Proxying +## Network and Upstream Proxying | Path | Current bound | Terminal behavior | |---|---:|---| -| `inference.local` request parse buffer | 10 MiB | Return `413` for an oversized request. | -| Chunked inference request | 10 MiB and 4,096 chunks | Reject an invalid or over-limit request. | -| Streaming inference response | 32 MiB and 120 s chunk idle | Truncate the stream and attempt a safe SSE error. | | Corporate proxy CONNECT response headers | 8 KiB | Fail the tunnel. | | Corporate proxy CONNECT handshake | 30 s total | Fail the tunnel; validated-address attempts share the aggregate budget. | | Token-grant HTTP request | 30 s request and connect | Fail credential resolution. | | Response-derived token cache TTL | 5 min default; 1 h response cap; 30 s expiry margin | A positive profile `cache_ttl_seconds` override replaces the response-derived calculation. | -Streaming response byte limits are integrity-relevant. Protocols whose clients -require one complete buffered object do not use the truncating SSE path. - ## Sandbox-Local Surfaces | Surface | Current bound | Scope and behavior | diff --git a/architecture/sandbox.md b/architecture/sandbox.md index 6365ed063..7be66e97e 100644 --- a/architecture/sandbox.md +++ b/architecture/sandbox.md @@ -28,7 +28,7 @@ only when the set is already empty; any other outcome fails the spawn. 2. The supervisor loads policy and runtime settings from local files or the gateway, depending on mode. 3. It prepares filesystem access, process restrictions, network namespace - routing, trust stores, provider credential resolution, and inference routes. + routing, trust stores, and provider credential resolution. 4. It launches the persisted canonical main-process argv and retains its PTY or pipes in the main-session multiplexer. 5. It starts the policy proxy and local SSH server. @@ -45,12 +45,12 @@ OpenShell uses overlapping controls rather than a single sandbox primitive: | Process policy | The child process runs as a non-root user with reduced privileges. | | Seccomp | Blocks dangerous syscalls, including raw socket paths that bypass the proxy. | | Network namespace | Forces ordinary agent egress through the local CONNECT proxy. | -| Policy proxy | Evaluates destination, binary identity, TLS/L7 rules, SSRF checks, and inference interception. | +| Policy proxy | Evaluates destination, binary identity, TLS/L7 rules, SSRF checks, and endpoint-bound credential injection. | The supervisor may enrich baseline filesystem allowances for runtime-required paths, such as proxy support files or GPU device paths when a GPU is present. -## Network and Inference +## Network and Provider Access See [Sandbox Limits](sandbox-limits.md) for the current numeric safety ceilings, their ownership, terminal behavior, and known gaps. @@ -157,6 +157,8 @@ support `params` matchers; generic JSON-RPC rules match only the method. JSON-RPC responses and server-to-client MCP messages on response or SSE streams are relayed but are not currently parsed for policy enforcement. +Every `protocol: mcp` endpoint carries a canonical, nonempty `mcp.versions` allowlist drawn from OpenShell's exact revision registry: `2025-03-26`, `2025-06-18`, and `2025-11-25`. A policy author may omit the entire `mcp` object when using the other endpoint defaults, or omit `mcp.versions` while setting another MCP option. Both forms resolve immediately to the exact allowlist `["2025-11-25"]`; omission never means latest or all known revisions. Defaulting applies only when the corresponding YAML key is absent: `mcp: null`, `versions: null`, and an explicit `versions: []` are invalid. At protobuf ingress, an empty repeated field means omission and uses the same default because protobuf repeated fields do not preserve presence. Normalization stores and serializes the materialized allowlist in semantic order, so adding a supported revision to the registry never widens a previously normalized policy. An explicit nonempty allowlist remains available as an advanced compatibility or downgrade control. The registry is a closed set rather than a date range, so duplicate or padded values, unknown dates, and moving aliases such as `draft` or `latest` are rejected. The sessionless `2026-07-28` revision is not accepted until OpenShell supports its distinct per-request contract. A version names a core protocol revision only; there is no policy syntax for layering a separately named SEP onto it. For every MCP HTTP request except a valid standalone `initialize`, the supervisor reads exactly one `MCP-Protocol-Version` header and requires that revision to appear in the endpoint allowlist. If the header is absent, the MCP transport specification defines `2025-03-26` as the compatibility fallback; OpenShell permits that fallback only when the allowlist contains it. Duplicate, empty, or unsupported header values receive `400 Bad Request`, while a supported revision outside the allowlist receives `403 Forbidden`. The supervisor repeats this check after middleware changes the request and before any upstream write. This check does not store session state or infer a version from a previous connection request. The registry also owns immutable batch-shape metadata: `2025-03-26` permits nonempty same-side top-level JSON-RPC batches, which OpenShell's planned enforcement caps at 64 members, while `2025-06-18` and `2025-11-25` prohibit top-level arrays. The current request parser does not yet apply these version-specific batch rules. + For admitted HTTP requests, the proxy can run an ordered supervisor middleware chain after L7 policy evaluation and before credential injection. Destination host selectors choose the chain independently of the network rule that admitted @@ -169,6 +171,10 @@ the remote adapter materializes an owned HTTP evaluation only when a request crosses that transport boundary. Both paths support bounded bidirectional WebSocket sessions, so a manifest advertises capabilities independently of transport. +When a stage ends, the remote adapter sends its terminal event, half-closes the +request stream, and briefly drains the response stream before releasing the +transport. This keeps a queued terminal event from being canceled with the +bidirectional RPC. The runtime keeps three states distinct: host selection attaches policy configs, manifest operation and phase bindings select the active chain, and the parsed message type determines whether that chain can inspect an individual payload. @@ -200,8 +206,9 @@ polling runs far more frequently than credentials expire, so the loop rotates only when a credential is missing or has passed four fifths of its lifetime, and bounds its sleep by the soonest rotation deadline. -Middleware cannot observe injected credentials or mutate supervisor-owned -credential, routing, or framing headers. Body transformations are re-evaluated +Middleware cannot observe injected credentials, introduce credential +placeholders, or mutate supervisor-owned credential, routing, or framing +headers. Body transformations are re-evaluated against body-aware L7 policy before later stages or the upstream can observe them. Requests, results, chain length, execution time, and diagnostics are bounded; external free-form diagnostic text is not exposed in responses or @@ -209,17 +216,11 @@ security logs. See [Supervisor Middleware](../docs/extensibility/supervisor-middleware.mdx) for configuration and protocol details. -`https://inference.local` is special. It bypasses OPA network policy and is -handled by the inference interception path: - -1. The proxy terminates the local TLS connection with the sandbox CA. -2. It detects known OpenAI, Anthropic, and compatible inference request shapes. -3. It strips caller-supplied credentials and disallowed headers. -4. It forwards through `openshell-router` using the route bundle fetched from - the gateway. - -External inference endpoints that do not use `inference.local` are treated like -ordinary network traffic and must be allowed by policy. +Inference providers use the same egress path as other external services. An +attached provider profile contributes endpoint and binary policy. The proxy +then resolves the provider's credential placeholder only when both policy and +the profile's endpoint binding authorize the native request. Model selection, +request shape, headers, streaming, and timeouts remain client concerns. In proxy-required networks, the supervisor chains upstream TLS tunnels through a corporate forward proxy with HTTP CONNECT instead of connecting directly, @@ -259,6 +260,12 @@ own DNS view, e.g. DoH tunneled via CONNECT, is a possible future enhancement and out of scope.) The workload child's proxy variables are unaffected — they are always rewritten to point at the local policy proxy. +Template environment is treated like user-provided sandbox environment. It can +shape the workload child, but it cannot override driver-controlled identity, +gateway callback, TLS, relay socket, proxy, provider, or supervisor coordination +variables. Drivers and the supervisor rewrite those reserved values after image +and template environment are considered. + The configuration is fail-closed: a setting that is present but invalid — an empty value, an unsupported or malformed proxy URL, an unreadable auth file or CA bundle, a malformed credential, or an auth file, `NO_PROXY` list, or CA @@ -294,6 +301,35 @@ file and builds the `Proxy-Authorization: Basic` header; a credential that is empty, contains control characters, or is not in `user:pass` form is fatal on both sides. +The VM driver has no argv seam of its own: its guest init script runs as PID 1 +and execs a fixed supervisor command line, and the libkrun and QEMU launch +backends both reach the supervisor through that script. Driver-owned +supervisor arguments therefore travel in a per-sandbox file the driver writes +into the overlay upperdir at a fixed guest path, one argument per line, which +the guest reads verbatim (no word splitting or globbing) and appends to every +supervisor exec. The file is written on **every** launch, including an empty +file when there is nothing to pass: the upperdir copy always shadows the +read-only image layer, so a sandbox image can neither supply its own +supervisor arguments by baking a file at that path nor disable the operator's +by omitting one. This mirrors the driver-authored `init.d` manifest, which +solves the same trust problem for guest init drop-ins. + +A microVM has no bind mounts or container secrets, so the VM driver stages the +credential and the CA bundle into the per-sandbox overlay disk instead — the +credential root-only, the CA world-readable, both at fixed `/opt/openshell` +paths and both removed with the sandbox state directory. The consequence, +which differs from the Podman secret model, is that the credential is at rest +inside that overlay image on the gateway host; the per-sandbox gateway JWT +already travels the same path. Proxy reachability differs by VM backend. libkrun-backed +sandboxes egress through gvproxy, so a proxy on the gateway host's loopback is +reachable through the host alias `host.openshell.internal`, which gvproxy NATs +to the host's `127.0.0.1`. QEMU/TAP sandboxes (GPU) have no equivalent: that +alias resolves to the TAP host address, and the driver's nftables `input` +chain accepts only the gateway port from the guest, so no gateway-host proxy +is reachable. The driver rejects a gateway-host proxy URL on the QEMU path at +launch rather than producing CONNECT timeouts. The guest's gateway callback is +unaffected in both backends and never traverses the proxy. + For Kubernetes sandboxes, the operator configures a Secret name and key rather than a gateway-host file path. Kubernetes projects that Secret only into the container that runs network supervision. Proxy credential Secrets require the @@ -419,6 +455,8 @@ sandbox workload directly. The relay supports: Sandbox logs are emitted locally and can also be pushed back to the gateway. Security-relevant sandbox behavior uses OCSF structured events; internal diagnostics use ordinary tracing. +The OCSF device describes the sandbox environment, with type ID Other and type +label `Sandbox`; its operating system is a separate attribute. ## Policy Proposals diff --git a/architecture/security-policy.md b/architecture/security-policy.md index 3f582eb25..5b49118b9 100644 --- a/architecture/security-policy.md +++ b/architecture/security-policy.md @@ -15,7 +15,7 @@ For the field-by-field YAML reference, use | Filesystem | Landlock restricts read-only and read-write paths. | | Process | The supervisor launches the agent as an unprivileged user with reduced capabilities. | | Network | The proxy evaluates destination, port, calling binary, and optional L7 rules. | -| Inference | `inference.local` is configured through gateway inference settings, not OPA network policy. | +| Provider access | Attached provider profiles contribute endpoint and binary rules; credentials remain bound to profile-authorized endpoints. | | Runtime settings | Typed settings are delivered with policy and can be global or sandbox scoped. | Filesystem and process policy are startup-time controls. Network policy is @@ -97,6 +97,13 @@ raw relay by default. A `protocol: rest` endpoint can opt in to after an allowed `101` upgrade; server-to-client traffic and all other upgraded protocols remain raw passthrough. +A `protocol: tcp` hostname is a connection-routing constraint, not an +application-authority boundary. Transparent capture validates the approved DNS +name, pinned destination address, port, and calling binary before opening the +stream, but it does not inspect TLS SNI, HTTP `Host`, or another protocol-level +destination. Compatible shared infrastructure can therefore let a client +select another tenant, virtual host, or service behind the approved front door. + ## Credentialed Endpoints OpenShell keeps provider credentials on paths it can inspect or rewrite by @@ -216,6 +223,12 @@ through the proposal loop instead of treating the denial as terminal. 1. **Submit.** Both proposers POST through the same `SubmitPolicyAnalysis` path. Each chunk is persisted with its `analysis_mode` for audit provenance. + Agent-authored chunks cannot request `protocol: tcp` or `tls: skip`; the + sandbox-local API rejects those transport choices for immediate feedback, + and the gateway repeats the check before persistence. + Omitted-protocol endpoints remain available through the explicit proxy with + default TLS termination and HTTP authority checks. Administrators can still + author native TCP and raw TLS policy directly. 2. **Build and validate the candidate.** The gateway first canonicalizes a mechanistic proposal against the live effective policy. If an endpoint is already governed by an inspected or provider-owned contract, the candidate @@ -268,6 +281,22 @@ After any successful policy write, pending chunks already covered by the new live effective policy are rejected as redundant. This keeps the review inbox aligned with what the sandbox currently enforces. +Endpoint and binary advisor markers are provenance, not authorization or +connection metadata. Provider- or user-authored declarations carry explicit +provenance; `policy.local` declarations carry advisor provenance. A difference +in endpoint provenance alone is compatible during effective-policy ambiguity +validation. When identical endpoint or binary identities merge, an explicit +declaration dominates an advisor declaration. Proposal coverage likewise +ignores provenance so an approved overlay converges when an existing explicit +declaration already supplies the same identity. + +This compatibility does not weaken SSRF classification. Exact-host trust +requires one matching rule to contain both an exact explicit endpoint and an +explicit binary identity. An advisor-only endpoint or binary cannot assemble +that trust from unrelated rules. A provider rule may independently establish +trust for its own explicit endpoint and binary pair, but an advisor overlay +does not broaden that pair to a different binary. + ### Security-notes gate Separately from the prover, each chunk carries advisory `security_notes`. diff --git a/architecture/windows-msvc-build.md b/architecture/windows-msvc-build.md index e940b8327..6b0d675a9 100644 --- a/architecture/windows-msvc-build.md +++ b/architecture/windows-msvc-build.md @@ -10,6 +10,9 @@ driver. It does not make Windows a Docker, Kubernetes, Podman, or VM runtime hos - Keep the Linux and macOS build paths unchanged. - Preserve gateway configuration parsing for all existing compute driver names. - Build and test the in-process MXC driver on supported Windows hosts. +- Use the ordinary in-process compute-driver composition path; MXC receives the + canonical sandbox policy through `DriverSandboxSpec` and advertises that it + reports runtime readiness. - Return clear unsupported errors when a Windows gateway is configured to use Docker, Kubernetes, Podman, or VM. - Keep dedicated `windows:*` validation tasks while allowing the repository-wide `pre-commit` task to delegate compiler-bearing Rust checks to the native @@ -23,9 +26,10 @@ driver. It does not make Windows a Docker, Kubernetes, Podman, or VM runtime hos ## Unsupported Driver Strategy -The gateway uses platform-specific configuration contracts on Windows. These -contracts preserve config-file parsing and reject unsupported driver selection -with a clear error without depending on the runtime driver crates. +The gateway composition crate installs platform-specific registration stubs on +Windows. These registrations preserve config-file selection and reject +unsupported drivers with a clear error without depending on their runtime +crates. The Windows lane does not build, release, package, or smoke-test standalone driver binaries for Docker, Kubernetes, Podman, or VM. Those binaries are Linux @@ -39,10 +43,10 @@ on Windows. | Driver | Windows build behavior | Runtime behavior | |---|---|---| -| Docker | Driver crate excluded; server config contract retained. | Gateway construction returns unsupported. | -| Kubernetes | Driver crate excluded; server config contract retained. | Gateway construction returns unsupported. | -| Podman | Driver crate excluded; server config contract retained. | Gateway construction returns unsupported. | -| VM | Driver crate excluded from workspace validation. | Gateway construction returns unsupported. | +| Docker | Driver crate excluded; gateway registration stub retained. | Gateway construction returns unsupported. | +| Kubernetes | Driver crate excluded; gateway registration stub retained. | Gateway construction returns unsupported. | +| Podman | Driver crate excluded; gateway registration stub retained. | Gateway construction returns unsupported. | +| VM | Driver crate excluded; gateway registration stub retained. | Gateway construction returns unsupported. | | MXC | Driver links into the native gateway and runs in Windows validation. | `process_container` is default-deny; grant-only `isolation_session` requires explicit configuration. | This keeps Windows behavior explicit without carrying runtime dependencies or @@ -66,7 +70,7 @@ Windows validation is exposed through `tasks/windows.toml`: | `windows:build:arm64` | Build release ARM64 `openshell-gateway.exe` and `openshell.exe`. | | `windows:test:x64` | Run native x64 workspace tests, including MXC mapper and lifecycle tests, while excluding unsupported Windows packages as top-level test targets. | | `windows:test:arm64` | Run native ARM64 workspace tests with the same package exclusions. | -| `windows:test:unsupported:x64` | Run focused server/runtime tests for unsupported driver contracts. | +| `windows:test:unsupported:x64` | Run focused gateway-composition tests for unsupported driver contracts. | | `windows:test:unsupported:arm64` | Run the same focused contracts natively on ARM64. | | `windows:ci` | Run check, build, test, unsupported-contract tests, and artifact reporting. | diff --git a/buf.yaml b/buf.yaml index a9ada8c9e..77fde3656 100644 --- a/buf.yaml +++ b/buf.yaml @@ -9,6 +9,7 @@ version: v2 modules: - path: proto + - path: crates/openshell-server/proto lint: use: - STANDARD diff --git a/crates/openshell-bootstrap/Cargo.toml b/crates/openshell-bootstrap/Cargo.toml index 96a798508..435024e30 100644 --- a/crates/openshell-bootstrap/Cargo.toml +++ b/crates/openshell-bootstrap/Cargo.toml @@ -12,20 +12,15 @@ rust-version.workspace = true [dependencies] openshell-core = { path = "../openshell-core", default-features = false } bytes = { workspace = true } -futures = { workspace = true } miette = { workspace = true } rcgen = { workspace = true } sha2 = { workspace = true } serde = { workspace = true } serde_json = { workspace = true } -tar = "0.4" tempfile = "3" tokio = { workspace = true } tracing = { workspace = true } -[target.'cfg(not(target_os = "windows"))'.dependencies] -bollard = "0.20" - [dev-dependencies] [lints] diff --git a/crates/openshell-bootstrap/src/build.rs b/crates/openshell-bootstrap/src/build.rs deleted file mode 100644 index f0e03fcd8..000000000 --- a/crates/openshell-bootstrap/src/build.rs +++ /dev/null @@ -1,677 +0,0 @@ -// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -// SPDX-License-Identifier: Apache-2.0 - -//! Build container images for sandbox runtimes. -//! -//! This module wraps bollard's `build_image()` API to build a container image -//! from a Dockerfile and build context. Package-managed local gateways use the -//! host Docker daemon, so the resulting tag is passed to the gateway directly. - -use std::collections::HashMap; -use std::path::Path; -use std::time::Duration; - -use bollard::Docker; -use bollard::query_parameters::BuildImageOptionsBuilder; -use futures::StreamExt; -use miette::{IntoDiagnostic, Result, WrapErr}; -use tokio::time::timeout; - -/// Maximum gap between Docker build stream events before a build is treated -/// as stuck. -/// -/// Total silence longer than this on under-provisioned container runtimes -/// (e.g. default Colima 2 vCPU / 2 GiB on macOS) reliably indicates a -/// deadlocked builder that will never recover. The default leaves headroom -/// for legitimately quiet steps (a single long `RUN` that produces no output) -/// — override with `OPENSHELL_BUILD_NO_PROGRESS_TIMEOUT_SECS` if a specific -/// build needs more time, or shorter for CI tightening. -const DEFAULT_BUILD_NO_PROGRESS_TIMEOUT_SECS: u64 = 1800; - -async fn docker_daemon_platform(docker: &Docker) -> Result { - let info = docker - .info() - .await - .into_diagnostic() - .wrap_err("failed to query local Docker daemon info")?; - let os = info - .os_type - .as_deref() - .filter(|value| !value.is_empty()) - .ok_or_else(|| miette::miette!("Docker daemon did not report an operating system"))?; - let arch = info - .architecture - .as_deref() - .filter(|value| !value.is_empty()) - .ok_or_else(|| miette::miette!("Docker daemon did not report an architecture"))?; - - normalize_docker_platform(os, arch) -} - -fn normalize_docker_platform(os: &str, arch: &str) -> Result { - // Remote or non-standard daemons may return mixed-case arch strings. - let arch_lower = arch.to_lowercase(); - let normalized_arch = match arch_lower.as_str() { - "amd64" | "x86_64" => "amd64", - "arm64" | "aarch64" => "arm64", - "arm" | "armv7" | "armv7l" | "armhf" => "arm/v7", - "armv6" | "armv6l" => "arm/v6", - "386" | "i386" | "i686" => "386", - "ppc64le" => "ppc64le", - "s390x" => "s390x", - _ => { - return Err(miette::miette!( - "unsupported Docker daemon architecture for local image build: {arch}" - )); - } - }; - Ok(format!("{os}/{normalized_arch}")) -} - -/// Parse `os/arch[/variant]` and reject malformed empty segments. -fn parse_platform(platform: &str) -> Result<(&str, &str, &str)> { - let mut parts = platform.split('/'); - let os = parts - .next() - .filter(|v| !v.is_empty()) - .ok_or_else(|| miette::miette!("platform is missing an operating system"))?; - let arch = parts - .next() - .filter(|v| !v.is_empty()) - .ok_or_else(|| miette::miette!("platform is missing an architecture"))?; - // A trailing slash creates a malformed empty variant segment. - let variant = match parts.next() { - Some("") => { - return Err(miette::miette!( - "platform variant must not be empty (trailing slash?): '{platform}'" - )); - } - Some(v) => v, - None => "", - }; - if parts.next().is_some() { - return Err(miette::miette!( - "platform must be os/arch[/variant], got '{platform}'" - )); - } - Ok((os, arch, variant)) -} - -/// Seed Docker's implicit `BuildKit` platform args without replacing caller values. -/// -/// `target_platform` identifies the output image. `build_platform` identifies -/// the Docker daemon doing the build. Variant args are only injected when the -/// platform string has an explicit variant, which preserves Dockerfile ARG -/// defaults for no-variant platforms. -fn insert_implicit_platform_build_args( - build_args: &mut HashMap, - target_platform: &str, - build_platform: &str, -) -> Result<()> { - let (target_os, target_arch, target_variant) = parse_platform(target_platform) - .wrap_err_with(|| format!("invalid target platform '{target_platform}'"))?; - let (build_os, build_arch, build_variant) = parse_platform(build_platform) - .wrap_err_with(|| format!("invalid build platform '{build_platform}'"))?; - - for (key, val) in [ - ("TARGETPLATFORM", target_platform), - ("TARGETOS", target_os), - ("TARGETARCH", target_arch), - ("BUILDPLATFORM", build_platform), - ("BUILDOS", build_os), - ("BUILDARCH", build_arch), - ] { - build_args - .entry(key.to_string()) - .or_insert_with(|| val.to_string()); - } - if !target_variant.is_empty() { - build_args - .entry("TARGETVARIANT".to_string()) - .or_insert_with(|| target_variant.to_string()); - } - if !build_variant.is_empty() { - build_args - .entry("BUILDVARIANT".to_string()) - .or_insert_with(|| build_variant.to_string()); - } - - Ok(()) -} - -/// Build a container image from a Dockerfile using the local Docker daemon. -/// -/// This is used by `openshell sandbox create --from `. The image -/// remains available in the local Docker daemon so the gateway's active local -/// compute driver can resolve the tag. -#[allow(clippy::implicit_hasher)] -pub async fn build_local_image( - dockerfile_path: &Path, - tag: &str, - context_dir: &Path, - build_args: &HashMap, - on_log: &mut impl FnMut(String), -) -> Result<()> { - on_log(format!( - "Building image {tag} from {}", - dockerfile_path.display() - )); - build_image(dockerfile_path, tag, context_dir, build_args, on_log).await?; - on_log(format!("Built image {tag}")); - Ok(()) -} - -/// Build a container image using the local Docker daemon. -/// -/// Creates a tar archive of `context_dir`, sends it to Docker with the -/// specified Dockerfile path and tag, and streams build output to `on_log`. -async fn build_image( - dockerfile_path: &Path, - tag: &str, - context_dir: &Path, - build_args: &HashMap, - on_log: &mut impl FnMut(String), -) -> Result<()> { - let docker = Docker::connect_with_local_defaults() - .into_diagnostic() - .wrap_err("failed to connect to local Docker daemon")?; - // BUILD* always comes from daemon info. A TARGETPLATFORM override only - // changes the output image platform. - let build_platform = docker_daemon_platform(&docker).await?; - let target_platform = build_args - .get("TARGETPLATFORM") - .map_or_else(|| build_platform.clone(), Clone::clone); - let mut effective_build_args = build_args.clone(); - insert_implicit_platform_build_args( - &mut effective_build_args, - &target_platform, - &build_platform, - )?; - - // Compute the relative path of the Dockerfile within the context. - let dockerfile_relative = dockerfile_path - .strip_prefix(context_dir) - .unwrap_or(dockerfile_path); - let dockerfile_str = dockerfile_relative - .to_str() - .ok_or_else(|| miette::miette!("Dockerfile path is not valid UTF-8"))?; - - // Create a tar archive of the build context, respecting .dockerignore. - let context_tar = create_build_context_tar(context_dir)?; - - let mut builder = BuildImageOptionsBuilder::default() - .dockerfile(dockerfile_str) - .t(tag) - .rm(true) - .platform(&target_platform); - - // Pass build args to Docker. - if !effective_build_args.is_empty() { - builder = builder.buildargs(&effective_build_args); - } - - let options = builder.build(); - - let body = bollard::body_full(bytes::Bytes::from(context_tar)); - let mut stream = docker.build_image(options, None, Some(body)); - let no_progress_secs: u64 = std::env::var("OPENSHELL_BUILD_NO_PROGRESS_TIMEOUT_SECS") - .ok() - .and_then(|s| s.parse().ok()) - .filter(|&n| n > 0) - .unwrap_or(DEFAULT_BUILD_NO_PROGRESS_TIMEOUT_SECS); - let no_progress_timeout = Duration::from_secs(no_progress_secs); - - loop { - let next = match timeout(no_progress_timeout, stream.next()).await { - Ok(Some(result)) => result, - Ok(None) => break, - Err(_) => { - return Err(miette::miette!( - "Docker build produced no output for {}s. This usually means the container \ - runtime is under-provisioned (CPU/memory) and the builder has deadlocked; \ - check `docker info` (NCPU, MemTotal) and increase Colima/Docker Desktop \ - resources before retrying. If a legitimate build step is just quiet, raise \ - the threshold with OPENSHELL_BUILD_NO_PROGRESS_TIMEOUT_SECS=.", - no_progress_timeout.as_secs() - )); - } - }; - - let info = next - .into_diagnostic() - .wrap_err("Docker build stream error")?; - - // Forward build output lines. - if let Some(stream_line) = &info.stream { - let trimmed = stream_line.trim_end(); - if !trimmed.is_empty() { - on_log(trimmed.to_string()); - } - } - - // Check for build errors. - if let Some(error_detail) = &info.error_detail { - let msg = error_detail - .message - .as_deref() - .unwrap_or("unknown build error"); - return Err(miette::miette!("Docker build failed: {msg}")); - } - } - - Ok(()) -} - -/// Create a tar archive of a directory for use as a Docker build context. -/// -/// Walks `context_dir` recursively, respects a `.dockerignore` file if present, -/// and adds matching files with paths relative to the context root. -fn create_build_context_tar(context_dir: &Path) -> Result> { - let ignore_patterns = load_dockerignore(context_dir); - - let mut builder = tar::Builder::new(Vec::new()); - - // Walk the directory tree and add entries, skipping ignored paths. - walk_and_add(context_dir, context_dir, &ignore_patterns, &mut builder)?; - - builder - .into_inner() - .into_diagnostic() - .wrap_err("failed to finalize build context tar") -} - -/// Recursively walk a directory and add entries to a tar archive, -/// skipping paths that match `.dockerignore` patterns. -fn walk_and_add( - root: &Path, - current: &Path, - ignore_patterns: &[IgnorePattern], - builder: &mut tar::Builder>, -) -> Result<()> { - let entries = std::fs::read_dir(current) - .into_diagnostic() - .wrap_err_with(|| format!("failed to read directory: {}", current.display()))?; - - for entry in entries { - let entry = entry - .into_diagnostic() - .wrap_err("failed to read directory entry")?; - let path = entry.path(); - let relative = path - .strip_prefix(root) - .unwrap_or(&path) - .to_string_lossy() - .to_string(); - - // Normalize to forward slashes for pattern matching. - let relative_normalized = relative.replace('\\', "/"); - - if is_ignored(&relative_normalized, path.is_dir(), ignore_patterns) { - continue; - } - - if path.is_dir() { - walk_and_add(root, &path, ignore_patterns, builder)?; - } else { - // Use append_path_with_name which handles GNU LongName extensions - // for paths exceeding 100 bytes (the POSIX tar name field limit). - builder - .append_path_with_name(&path, &relative_normalized) - .into_diagnostic() - .wrap_err_with(|| format!("failed to add file to tar: {relative_normalized}"))?; - } - } - - Ok(()) -} - -/// A parsed `.dockerignore` pattern. -#[derive(Debug, Clone)] -struct IgnorePattern { - /// The glob pattern (may contain `*`, `**`, `?`). - pattern: String, - /// Whether this is a negation pattern (starts with `!`). - negated: bool, -} - -/// Load and parse a `.dockerignore` file from the context directory. -/// -/// Returns an empty list if no `.dockerignore` exists. -fn load_dockerignore(context_dir: &Path) -> Vec { - let dockerignore_path = context_dir.join(".dockerignore"); - let Ok(contents) = std::fs::read_to_string(&dockerignore_path) else { - return Vec::new(); - }; - - contents - .lines() - .map(str::trim) - .filter(|line| !line.is_empty() && !line.starts_with('#')) - .map(|line| { - line.strip_prefix('!').map_or_else( - || IgnorePattern { - pattern: line.to_string(), - negated: false, - }, - |rest| IgnorePattern { - pattern: rest.trim().to_string(), - negated: true, - }, - ) - }) - .collect() -} - -/// Check whether a relative path should be ignored based on `.dockerignore` patterns. -/// -/// Uses a simple glob-matching approach: patterns are matched against the -/// full relative path. A leading `/` anchors to the context root. The last -/// matching pattern wins (negation patterns re-include files). -fn is_ignored(relative_path: &str, is_dir: bool, patterns: &[IgnorePattern]) -> bool { - let mut ignored = false; - - for pat in patterns { - let pattern = pat.pattern.trim_start_matches('/'); - - // Check if the pattern matches. - let matches = glob_match(pattern, relative_path, is_dir); - - if matches { - ignored = !pat.negated; - } - } - - ignored -} - -/// Simple glob matching supporting `*`, `**`, and `?`. -/// -/// This is intentionally simple — it covers the common `.dockerignore` cases -/// without pulling in a full glob crate. For complex patterns, Docker's own -/// builder handles them during the build step anyway; this is just for -/// reducing the context tar size. -fn glob_match(pattern: &str, path: &str, is_dir: bool) -> bool { - // Handle ** prefix (match any number of directories) - if let Some(rest) = pattern.strip_prefix("**/") { - // Match against the path itself and any suffix after a / - if glob_match(rest, path, is_dir) { - return true; - } - for (idx, _) in path.match_indices('/') { - if let Some(suffix) = path.get(idx + 1..) - && glob_match(rest, suffix, is_dir) - { - return true; - } - } - return false; - } - - // Handle pattern that is just a name (no slashes) — match against any - // path component or as a prefix directory match. - if !pattern.contains('/') { - // Match the final component of the path. - let basename = path.rsplit('/').next().unwrap_or(path); - if simple_glob_match(pattern, basename) { - return true; - } - // Also match as a directory prefix: pattern "node_modules" should - // match "node_modules/foo/bar". - if let Some(first) = path.split('/').next() - && simple_glob_match(pattern, first) - { - return true; - } - return false; - } - - // Pattern contains slashes — match against the full path. - simple_glob_match(pattern, path) || (is_dir && path.starts_with(pattern.trim_end_matches('/'))) -} - -/// Match a simple glob pattern (with `*` and `?` but not `**`) against a string. -fn simple_glob_match(pattern: &str, text: &str) -> bool { - let mut p_star: Option = None; - let mut t_star: Option = None; - - let p_bytes: Vec = pattern.chars().collect(); - let t_bytes: Vec = text.chars().collect(); - let mut pi = 0; - let mut ti = 0; - - while ti < t_bytes.len() { - if pi < p_bytes.len() && (p_bytes[pi] == '?' || p_bytes[pi] == t_bytes[ti]) { - pi += 1; - ti += 1; - } else if pi < p_bytes.len() && p_bytes[pi] == '*' { - p_star = Some(pi); - t_star = Some(ti); - pi += 1; - } else if let Some(ps) = p_star { - pi = ps + 1; - t_star = Some(t_star.unwrap() + 1); - ti = t_star.unwrap(); - } else { - return false; - } - } - - while pi < p_bytes.len() && p_bytes[pi] == '*' { - pi += 1; - } - - pi == p_bytes.len() -} - -#[cfg(test)] -mod tests { - use super::*; - use std::fs; - - #[test] - fn test_create_build_context_tar() { - let dir = tempfile::tempdir().unwrap(); - let dir_path = dir.path(); - - // Create a simple Dockerfile and a file. - fs::write(dir_path.join("Dockerfile"), "FROM ubuntu:24.04\n").unwrap(); - fs::write(dir_path.join("hello.txt"), "hello world\n").unwrap(); - fs::create_dir(dir_path.join("subdir")).unwrap(); - fs::write(dir_path.join("subdir/nested.txt"), "nested\n").unwrap(); - - let tar_bytes = create_build_context_tar(dir_path).unwrap(); - assert!(!tar_bytes.is_empty()); - - // Verify the tar contains the expected entries. - let mut archive = tar::Archive::new(tar_bytes.as_slice()); - let entries: Vec = archive - .entries() - .unwrap() - .filter_map(std::result::Result::ok) - .map(|e| e.path().unwrap().to_string_lossy().to_string()) - .collect(); - - assert!(entries.iter().any(|e| e.contains("Dockerfile"))); - assert!(entries.iter().any(|e| e.contains("hello.txt"))); - assert!(entries.iter().any(|e| e.contains("subdir/nested.txt"))); - } - - #[test] - fn test_dockerignore_excludes_files() { - let dir = tempfile::tempdir().unwrap(); - let dir_path = dir.path(); - - fs::write(dir_path.join("Dockerfile"), "FROM ubuntu:24.04\n").unwrap(); - fs::write(dir_path.join("hello.txt"), "hello\n").unwrap(); - fs::write(dir_path.join("secret.env"), "SECRET=foo\n").unwrap(); - fs::create_dir(dir_path.join("node_modules")).unwrap(); - fs::write(dir_path.join("node_modules/pkg.js"), "module\n").unwrap(); - fs::write(dir_path.join(".dockerignore"), "*.env\nnode_modules\n").unwrap(); - - let tar_bytes = create_build_context_tar(dir_path).unwrap(); - let mut archive = tar::Archive::new(tar_bytes.as_slice()); - let entries: Vec = archive - .entries() - .unwrap() - .filter_map(std::result::Result::ok) - .map(|e| e.path().unwrap().to_string_lossy().to_string()) - .collect(); - - assert!(entries.iter().any(|e| e.contains("Dockerfile"))); - assert!(entries.iter().any(|e| e.contains("hello.txt"))); - assert!(!entries.iter().any(|e| e.contains("secret.env"))); - assert!(!entries.iter().any(|e| e.contains("node_modules"))); - } - - #[test] - fn test_dockerignore_negation() { - let dir = tempfile::tempdir().unwrap(); - let dir_path = dir.path(); - - fs::write(dir_path.join("Dockerfile"), "FROM ubuntu:24.04\n").unwrap(); - fs::write(dir_path.join("a.log"), "log\n").unwrap(); - fs::write(dir_path.join("important.log"), "keep\n").unwrap(); - fs::write(dir_path.join(".dockerignore"), "*.log\n!important.log\n").unwrap(); - - let tar_bytes = create_build_context_tar(dir_path).unwrap(); - let mut archive = tar::Archive::new(tar_bytes.as_slice()); - let entries: Vec = archive - .entries() - .unwrap() - .filter_map(std::result::Result::ok) - .map(|e| e.path().unwrap().to_string_lossy().to_string()) - .collect(); - - assert!(!entries.iter().any(|e| e.contains("a.log"))); - assert!(entries.iter().any(|e| e.contains("important.log"))); - } - - #[test] - fn test_long_path_exceeding_100_bytes() { - let dir = tempfile::tempdir().unwrap(); - let dir_path = dir.path(); - - // Build a nested path that exceeds 100 bytes when relative to root. - let deep_dir = dir_path.join( - "a/deeply/nested/directory/path/that/exceeds/one/hundred/bytes/total/from/the/build/context/root", - ); - fs::create_dir_all(&deep_dir).unwrap(); - fs::write(deep_dir.join("file.txt"), "deep content\n").unwrap(); - fs::write(dir_path.join("Dockerfile"), "FROM ubuntu:24.04\n").unwrap(); - - let tar_bytes = create_build_context_tar(dir_path).unwrap(); - let mut archive = tar::Archive::new(tar_bytes.as_slice()); - let entries: Vec = archive - .entries() - .unwrap() - .filter_map(std::result::Result::ok) - .map(|e| e.path().unwrap().to_string_lossy().to_string()) - .collect(); - - let long_entry = entries.iter().find(|e| e.contains("file.txt")); - assert!( - long_entry.is_some(), - "tar should contain deeply nested file; entries: {entries:?}" - ); - assert!( - long_entry.unwrap().len() > 100, - "path should exceed 100 bytes to exercise GNU LongName handling" - ); - } - - #[test] - fn test_simple_glob_match() { - assert!(simple_glob_match("*.txt", "hello.txt")); - assert!(!simple_glob_match("*.txt", "hello.rs")); - assert!(simple_glob_match("test?", "test1")); - assert!(!simple_glob_match("test?", "test12")); - assert!(simple_glob_match("*", "anything")); - assert!(simple_glob_match("foo*bar", "fooXYZbar")); - } - - #[test] - fn test_glob_match_double_star() { - assert!(glob_match("**/*.log", "a/b/c.log", false)); - assert!(glob_match("**/*.log", "c.log", false)); - assert!(!glob_match("**/*.log", "c.txt", false)); - assert!(glob_match("**/föö.log", "a/b/föö.log", false)); - } - - #[test] - fn test_is_ignored_directory_prefix() { - let patterns = vec![IgnorePattern { - pattern: "node_modules".to_string(), - negated: false, - }]; - assert!(is_ignored("node_modules", true, &patterns)); - assert!(is_ignored("node_modules/foo.js", false, &patterns)); - } - - #[test] - fn test_normalize_docker_platform_maps_docker_desktop_arch() { - assert_eq!( - normalize_docker_platform("linux", "aarch64").unwrap(), - "linux/arm64" - ); - assert_eq!( - normalize_docker_platform("linux", "x86_64").unwrap(), - "linux/amd64" - ); - } - - #[test] - fn test_normalize_docker_platform_case_insensitive_arch() { - // Some remote or non-standard Docker daemons return mixed-case arch strings. - assert_eq!( - normalize_docker_platform("linux", "ARM64").unwrap(), - "linux/arm64" - ); - assert_eq!( - normalize_docker_platform("linux", "X86_64").unwrap(), - "linux/amd64" - ); - } - - #[test] - fn test_insert_implicit_platform_build_args_native() { - // Native builds use one platform for both target and build args. - let mut build_args = HashMap::new(); - insert_implicit_platform_build_args(&mut build_args, "linux/arm/v7", "linux/arm/v7") - .unwrap(); - - assert_eq!(build_args.get("TARGETPLATFORM").unwrap(), "linux/arm/v7"); - assert_eq!(build_args.get("TARGETOS").unwrap(), "linux"); - assert_eq!(build_args.get("TARGETARCH").unwrap(), "arm"); - assert_eq!(build_args.get("TARGETVARIANT").unwrap(), "v7"); - assert_eq!(build_args.get("BUILDPLATFORM").unwrap(), "linux/arm/v7"); - assert_eq!(build_args.get("BUILDOS").unwrap(), "linux"); - assert_eq!(build_args.get("BUILDARCH").unwrap(), "arm"); - assert_eq!(build_args.get("BUILDVARIANT").unwrap(), "v7"); - } - - #[test] - fn test_insert_implicit_platform_build_args_cross_compile() { - // Cross-builds keep daemon BUILD* args separate from target args. - let mut build_args = HashMap::new(); - insert_implicit_platform_build_args(&mut build_args, "linux/arm64", "linux/amd64").unwrap(); - - assert_eq!(build_args.get("TARGETPLATFORM").unwrap(), "linux/arm64"); - assert_eq!(build_args.get("TARGETARCH").unwrap(), "arm64"); - assert_eq!(build_args.get("BUILDPLATFORM").unwrap(), "linux/amd64"); - assert_eq!(build_args.get("BUILDARCH").unwrap(), "amd64"); - // Leaving variant args absent preserves Dockerfile ARG defaults. - assert!(!build_args.contains_key("TARGETVARIANT")); - assert!(!build_args.contains_key("BUILDVARIANT")); - } - - #[test] - fn test_insert_implicit_platform_build_args_trailing_slash_rejected() { - let mut build_args = HashMap::new(); - let result = - insert_implicit_platform_build_args(&mut build_args, "linux/arm64/", "linux/amd64"); - assert!( - result.is_err(), - "trailing slash should be rejected as a malformed variant segment" - ); - } -} diff --git a/crates/openshell-bootstrap/src/build_windows.rs b/crates/openshell-bootstrap/src/build_windows.rs deleted file mode 100644 index 93af1345d..000000000 --- a/crates/openshell-bootstrap/src/build_windows.rs +++ /dev/null @@ -1,23 +0,0 @@ -// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -// SPDX-License-Identifier: Apache-2.0 - -//! Windows stub for local Dockerfile image builds. - -use std::collections::HashMap; -use std::path::Path; - -use miette::Result; - -// Keep this stub's signature aligned with the supported-platform implementation. -#[allow(clippy::implicit_hasher)] -pub async fn build_local_image( - _dockerfile_path: &Path, - _tag: &str, - _context_dir: &Path, - _build_args: &HashMap, - _on_log: &mut impl FnMut(String), -) -> Result<()> { - Err(miette::miette!( - "local Dockerfile sandbox sources are unsupported on Windows" - )) -} diff --git a/crates/openshell-bootstrap/src/lib.rs b/crates/openshell-bootstrap/src/lib.rs index 5598d524e..36ab7a8c7 100644 --- a/crates/openshell-bootstrap/src/lib.rs +++ b/crates/openshell-bootstrap/src/lib.rs @@ -1,11 +1,6 @@ // SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 -#[cfg(not(target_os = "windows"))] -pub mod build; -#[cfg(target_os = "windows")] -#[path = "build_windows.rs"] -pub mod build; pub mod edge_token; pub mod jwt; pub mod oidc_token; diff --git a/crates/openshell-bootstrap/src/oidc_token.rs b/crates/openshell-bootstrap/src/oidc_token.rs index f21a115c8..282d2ddec 100644 --- a/crates/openshell-bootstrap/src/oidc_token.rs +++ b/crates/openshell-bootstrap/src/oidc_token.rs @@ -124,6 +124,21 @@ pub fn is_token_expired(bundle: &OidcTokenBundle) -> bool { now + 30 >= expires_at } +/// Check whether the stored access token has actually expired. +/// +/// Unlike [`is_token_expired`], this does not include the 30-second refresh +/// window. Tokens without expiry metadata are treated as valid. +pub fn is_token_actually_expired(bundle: &OidcTokenBundle) -> bool { + let Some(expires_at) = bundle.expires_at else { + return false; + }; + let now = std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .unwrap_or_default() + .as_secs(); + now >= expires_at +} + #[cfg(test)] mod tests { use super::*; @@ -163,6 +178,30 @@ mod tests { }); } + #[test] + fn expiry_checks_distinguish_refresh_window_from_actual_expiry() { + let now = std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .unwrap() + .as_secs(); + let bundle = OidcTokenBundle { + access_token: "token".to_string(), + refresh_token: None, + expires_at: Some(now + 10), + issuer: "https://issuer.example.com".to_string(), + client_id: "openshell-cli".to_string(), + }; + + assert!(is_token_expired(&bundle)); + assert!(!is_token_actually_expired(&bundle)); + + let expired = OidcTokenBundle { + expires_at: Some(now.saturating_sub(1)), + ..bundle + }; + assert!(is_token_actually_expired(&expired)); + } + #[test] fn oidc_login_prompt_marker_is_per_gateway() { let tmp = tempfile::tempdir().unwrap(); diff --git a/crates/openshell-cli/Cargo.toml b/crates/openshell-cli/Cargo.toml index 4b9625331..9a803b77d 100644 --- a/crates/openshell-cli/Cargo.toml +++ b/crates/openshell-cli/Cargo.toml @@ -37,6 +37,9 @@ chrono = "0.4" clap = { workspace = true } clap_complete = { workspace = true } crossterm = { workspace = true } +# Styling backend shared by dialoguer and indicatif; `color` overrides its +# global switch so both follow the CLI's --color setting. +console = "0.15" dialoguer = "0.11" indicatif = { workspace = true } owo-colors = { workspace = true } @@ -46,7 +49,7 @@ bytes = { workspace = true } http-body-util = { workspace = true } hyper = { workspace = true } hyper-util = { workspace = true } -hyper-rustls = { version = "0.27", default-features = false, features = ["native-tokio", "http1", "http2", "tls12", "logging", "ring"] } +hyper-rustls = { version = "0.27", default-features = false, features = ["native-tokio", "http1", "http2", "tls12", "logging", "aws-lc-rs"] } rustls = { workspace = true } rustls-pemfile = { workspace = true } tokio-rustls = { workspace = true } @@ -88,7 +91,7 @@ nix = { workspace = true } [dev-dependencies] futures = { workspace = true } -rcgen = { version = "0.13", features = ["crypto", "pem"] } +rcgen = { workspace = true } reqwest = { workspace = true } serde_json = { workspace = true } temp-env = "0.3" diff --git a/crates/openshell-cli/src/auth.rs b/crates/openshell-cli/src/auth.rs index 56601e2a3..bc1baae26 100644 --- a/crates/openshell-cli/src/auth.rs +++ b/crates/openshell-cli/src/auth.rs @@ -35,7 +35,7 @@ use tokio::sync::oneshot; use tracing::debug; /// Timeout for the browser auth flow. -const AUTH_TIMEOUT: Duration = Duration::from_secs(120); +const AUTH_TIMEOUT: Duration = Duration::from_mins(2); /// Length of the confirmation code (alphanumeric characters). const CODE_LENGTH: usize = 7; diff --git a/crates/openshell-cli/src/color.rs b/crates/openshell-cli/src/color.rs new file mode 100644 index 000000000..9b78b4ca9 --- /dev/null +++ b/crates/openshell-cli/src/color.rs @@ -0,0 +1,613 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! Runtime color control for CLI output. +//! +//! The CLI styles its human-readable tables and status lines with ANSI escape +//! sequences. Those sequences must not appear when the output is being consumed +//! by another program, or callers end up writing brittle patterns against bytes +//! they cannot see. +//! +//! `owo_colors::OwoColorize` emits escapes unconditionally, so this module wraps +//! it with a process-wide switch resolved once at startup by [`init`]. Command +//! modules import [`Colorize`] instead of `OwoColorize`; the method names match, +//! so call sites are unchanged, but each one now checks the switch when it +//! renders. +//! +//! Styling is not confined to `owo-colors`, and the other paths each carry +//! their own default, so [`init`] brings them under the same setting: +//! +//! - `tracing_subscriber` formats with ANSI on, does no terminal detection, and +//! writes to stdout. Left alone, `openshell -v ... | ...` leaks escapes into a +//! pipe exactly like the styled tables did. `main` passes [`stdout_enabled`] +//! to `with_ansi`. +//! - `indicatif` and `dialoguer` both style through `console`, which has its own +//! detection and honors `NO_COLOR` but cannot know about `--color`. [`init`] +//! overrides it globally, which covers every progress bar and prompt rather +//! than the specific ones the CLI happens to construct today. +//! - `miette` renders errors to stderr with its own detection, likewise unaware +//! of `--color`. [`init`] installs a report handler built from +//! [`stderr_enabled`]. +//! +//! Resolution order, highest precedence first: +//! +//! 1. `--color always|never` on the command line. +//! 2. `NO_COLOR`, set and non-empty, disables color (). +//! 3. `FORCE_COLOR`, set and non-empty, forces color on (). +//! 4. Otherwise the stream is styled only when that stream is a terminal *and* +//! that terminal renders ANSI. +//! +//! Attachment and capability are separate questions. `TERM=dumb` is a terminal +//! that does not interpret escapes, so `auto` must not style it — and neither +//! `console` nor `miette` can apply their own `TERM` checks any more, because +//! [`init`] overrides both. Capability is consulted only under `auto`, so +//! `--color always` and `FORCE_COLOR` still force styling on a `dumb` terminal +//! for anyone who wants it. +//! +//! Step 4 is resolved per stream. Redirecting one must not decide for the other: +//! `openshell ... 2> build.log` from a terminal should keep a styled stdout and +//! write a plain-text log, and `openshell ... | grep` should keep styled +//! diagnostics on the terminal while feeding the pipe clean bytes. [`init`] +//! therefore stores one answer per stream and hands each library the one for the +//! stream it writes to. +//! +//! The `owo-colors` wrapper is the exception, because its call sites are split +//! across `println!` and `eprintln!` and a [`Painted`] value cannot tell which +//! macro will consume it. It styles only when both streams accept escapes, which +//! errs toward plain text rather than risk writing escapes to a redirected +//! stream. + +use std::ffi::OsStr; +use std::fmt::{self, Display}; +use std::io::IsTerminal; +use std::sync::atomic::{AtomicBool, Ordering}; + +use owo_colors::{OwoColorize, Style}; + +/// Whether stdout may carry escapes. Consulted for `tracing` and console's +/// stdout switch. +/// +/// Defaults to disabled so that any output produced before [`init`] runs — and +/// output from unit tests, which never call `init` — stays free of escapes. +static STDOUT_ENABLED: AtomicBool = AtomicBool::new(false); + +/// Whether stderr may carry escapes. Consulted for `miette` and console's stderr +/// switch. +static STDERR_ENABLED: AtomicBool = AtomicBool::new(false); + +/// When to colorize CLI output. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, clap::ValueEnum)] +pub enum ColorChoice { + /// Colorize a stream only when that stream is a terminal and no environment + /// override applies. + #[default] + Auto, + /// Always colorize, even when output is redirected. + Always, + /// Never colorize. + Never, +} + +/// Resolve the color setting and store it for the rest of the process. +/// +/// Call once, as early as possible after argument parsing and before any output +/// is written. +pub fn init(choice: ColorChoice) { + let no_color = std::env::var_os("NO_COLOR"); + let force_color = std::env::var_os("FORCE_COLOR"); + let term = std::env::var_os("TERM"); + + // Under `auto` each stream answers for itself. Redirecting one must not + // decide for the other: `openshell ... 2> build.log` from a terminal has a + // styled stdout and a plain-text stderr, and vice versa for `| grep`. + let stdout_enabled = resolve( + choice, + no_color.as_deref(), + force_color.as_deref(), + terminal_supports_ansi(std::io::stdout().is_terminal(), term.as_deref()), + ); + let stderr_enabled = resolve( + choice, + no_color.as_deref(), + force_color.as_deref(), + terminal_supports_ansi(std::io::stderr().is_terminal(), term.as_deref()), + ); + STDOUT_ENABLED.store(stdout_enabled, Ordering::Relaxed); + STDERR_ENABLED.store(stderr_enabled, Ordering::Relaxed); + + // `indicatif` and `dialoguer` both style through `console`, which keeps its + // own detection. Override it so progress bars and prompts follow the same + // setting as everything else — including `--color`, which console has no way + // to learn about on its own. + console::set_colors_enabled(stdout_enabled); + console::set_colors_enabled_stderr(stderr_enabled); + + // miette renders errors to stderr. The hook can only be installed once per + // process; a failure means something already installed one, and error + // rendering is not worth aborting the command over. + let _ = miette::set_hook(Box::new(move |_| { + Box::new( + miette::MietteHandlerOpts::new() + .color(stderr_enabled) + .build(), + ) + })); +} + +/// Whether stdout may carry ANSI escapes. +/// +/// [`init`] configures `console` and `miette` directly; `tracing` is wired up by +/// the caller, which writes to stdout and passes this to `with_ansi`. +#[must_use] +pub fn stdout_enabled() -> bool { + STDOUT_ENABLED.load(Ordering::Relaxed) +} + +/// Whether stderr may carry ANSI escapes. +#[must_use] +pub fn stderr_enabled() -> bool { + STDERR_ENABLED.load(Ordering::Relaxed) +} + +/// Whether a [`Painted`] value should render styled. +/// +/// The `owo-colors` call sites are split across `println!` and `eprintln!` and +/// the wrapper cannot tell which one will consume it, so it styles only when +/// *both* streams accept escapes. Erring toward plain text keeps a redirected +/// stream clean, which is the whole point of the switch; the cost is that +/// `openshell ... 2>/dev/null` from a terminal prints an uncolored table. +/// `--color always` overrides it. +#[must_use] +fn painted_enabled() -> bool { + stdout_enabled() && stderr_enabled() +} + +/// Decide whether one stream may carry escapes. Split out from [`init`] so the +/// precedence rules are testable without mutating process state. +fn resolve( + choice: ColorChoice, + no_color: Option<&OsStr>, + force_color: Option<&OsStr>, + stream_supports_ansi: bool, +) -> bool { + match choice { + ColorChoice::Always => return true, + ColorChoice::Never => return false, + ColorChoice::Auto => {} + } + + // Both conventions key on presence rather than value: set and non-empty + // means yes, regardless of what the value is. and + // . + if is_set(no_color) { + return false; + } + if is_set(force_color) { + return true; + } + + // Only `auto` consults the terminal. An explicit request above has already + // returned, so `--color always` and `FORCE_COLOR` still win on a terminal + // that reports no ANSI support. + stream_supports_ansi +} + +/// Whether this output stream's terminal renders ANSI escapes. +/// +/// Being attached to a terminal is not the same as that terminal rendering +/// ANSI. `TERM` is the unix signal for it; Windows consoles enable virtual +/// terminal processing instead and do not set `TERM`, so the check does not +/// apply there. +fn terminal_supports_ansi(stream_is_terminal: bool, term: Option<&OsStr>) -> bool { + stream_is_terminal + && if cfg!(unix) { + term_supports_ansi(term) + } else { + true + } +} + +/// Whether the terminal named by `TERM` renders ANSI escapes on Unix. +/// +/// Follows the rule `console` applies on unix, which this module overrides: +/// `dumb` means no, and an unset `TERM` means no because nothing identifies a +/// capable terminal. +/// +/// Empty is treated as unset, which is a deliberate divergence: `console` reads +/// `TERM=""` as `Ok("")`, and since that is not `"dumb"` it counts as capable. +/// An empty value names no terminal type, and every other variable here already +/// treats empty as unset, so it is handled the same way. +fn term_supports_ansi(term: Option<&OsStr>) -> bool { + is_set(term) && term != Some(OsStr::new("dumb")) +} + +/// Whether an environment variable counts as set: present and not empty. +fn is_set(value: Option<&OsStr>) -> bool { + value.is_some_and(|value| !value.is_empty()) +} + +/// A value tagged with a style, rendered only when color is enabled. +/// +/// Borrows its value and forwards the caller's format specification to the +/// inner `Display`, so width and alignment apply to the text rather than to the +/// text plus escapes. +pub struct Painted<'a, T: ?Sized> { + value: &'a T, + style: Style, +} + +impl Display for Painted<'_, T> { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + if !painted_enabled() { + return Display::fmt(&self.value, f); + } + // Hand the whole style to owo-colors in one go. `Styled` writes the + // prefix, forwards `f` to the inner `Display` so padding still measures + // the text, then writes the reset. + Display::fmt(&OwoColorize::style(&self.value, self.style), f) + } +} + +/// Style-combining methods, so `x.green().bold()` renders as one escape +/// sequence rather than nesting two. +/// +/// These are inherent so they take precedence over the [`Colorize`] blanket +/// impl, which would otherwise wrap a `Painted` in another `Painted`. +impl Painted<'_, T> { + /// Add bold to the current style. + #[must_use] + pub fn bold(self) -> Self { + Self { + style: self.style.bold(), + ..self + } + } + /// Add cyan to the current style. + #[must_use] + pub fn cyan(self) -> Self { + Self { + style: self.style.cyan(), + ..self + } + } + /// Add dimmed to the current style. + #[must_use] + pub fn dimmed(self) -> Self { + Self { + style: self.style.dimmed(), + ..self + } + } + /// Add green to the current style. + #[must_use] + pub fn green(self) -> Self { + Self { + style: self.style.green(), + ..self + } + } + /// Add red to the current style. + #[must_use] + pub fn red(self) -> Self { + Self { + style: self.style.red(), + ..self + } + } + /// Add yellow to the current style. + #[must_use] + pub fn yellow(self) -> Self { + Self { + style: self.style.yellow(), + ..self + } + } +} + +/// Styling methods mirroring the `owo_colors::OwoColorize` surface the CLI uses. +/// +/// Import this instead of `OwoColorize` so styled output honors [`init`]. The +/// two traits have colliding method names on purpose: importing both in one +/// module is an ambiguity error, which keeps unconditional coloring from +/// creeping back in. +pub trait Colorize { + /// Render in bold. + fn bold(&self) -> Painted<'_, Self>; + /// Render in cyan. + fn cyan(&self) -> Painted<'_, Self>; + /// Render dimmed. + fn dimmed(&self) -> Painted<'_, Self>; + /// Render in green. + fn green(&self) -> Painted<'_, Self>; + /// Render in red. + fn red(&self) -> Painted<'_, Self>; + /// Render in yellow. + fn yellow(&self) -> Painted<'_, Self>; +} + +impl Colorize for T { + fn bold(&self) -> Painted<'_, Self> { + Painted { + value: self, + style: Style::new().bold(), + } + } + fn cyan(&self) -> Painted<'_, Self> { + Painted { + value: self, + style: Style::new().cyan(), + } + } + fn dimmed(&self) -> Painted<'_, Self> { + Painted { + value: self, + style: Style::new().dimmed(), + } + } + fn green(&self) -> Painted<'_, Self> { + Painted { + value: self, + style: Style::new().green(), + } + } + fn red(&self) -> Painted<'_, Self> { + Painted { + value: self, + style: Style::new().red(), + } + } + fn yellow(&self) -> Painted<'_, Self> { + Painted { + value: self, + style: Style::new().yellow(), + } + } +} + +#[cfg(test)] +mod tests { + // Deliberately not `use super::*`: that would also pull in `OwoColorize` + // and make every `.green()` below ambiguous. + use super::{ + ColorChoice, Colorize, Ordering, STDERR_ENABLED, STDOUT_ENABLED, Style, painted_enabled, + resolve, term_supports_ansi, + }; + use std::ffi::OsStr; + + /// Serializes tests that flip the process-wide switches. + static SWITCH_LOCK: std::sync::Mutex<()> = std::sync::Mutex::new(()); + + /// Set both stream switches, since [`Painted`] requires both. + fn with_color(on: bool, body: impl FnOnce() -> R) -> R { + with_streams(on, on, body) + } + + fn with_streams(stdout: bool, stderr: bool, body: impl FnOnce() -> R) -> R { + let _guard = SWITCH_LOCK + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner); + let prev_out = STDOUT_ENABLED.swap(stdout, Ordering::Relaxed); + let prev_err = STDERR_ENABLED.swap(stderr, Ordering::Relaxed); + let result = body(); + STDOUT_ENABLED.store(prev_out, Ordering::Relaxed); + STDERR_ENABLED.store(prev_err, Ordering::Relaxed); + result + } + + /// Helper so the `resolve` cases read as plain strings. + fn env(value: &str) -> &OsStr { + OsStr::new(value) + } + + #[test] + fn disabled_output_has_no_escapes() { + with_color(false, || { + assert_eq!("running".green().to_string(), "running"); + assert_eq!("dead".red().to_string(), "dead"); + assert_eq!("STATUS".bold().to_string(), "STATUS"); + }); + } + + #[test] + fn enabled_output_is_wrapped_in_the_expected_escapes() { + with_color(true, || { + // The styling the bug report observed from `forward list`. + assert_eq!("running".green().to_string(), "\u{1b}[32mrunning\u{1b}[0m"); + // Delegation is to owo-colors, so the bytes match what it would + // produce for the same style. + assert_eq!( + "running".green().to_string(), + owo_colors::OwoColorize::style(&"running", Style::new().green()).to_string() + ); + assert_eq!( + "x".dimmed().to_string(), + owo_colors::OwoColorize::style(&"x", Style::new().dimmed()).to_string() + ); + }); + } + + #[test] + fn format_width_applies_to_text_not_escapes() { + // Padding must measure the value, so columns line up identically whether + // or not color is on. + let plain = with_color(false, || format!("[{:<10}]", "running".green())); + let colored = with_color(true, || format!("[{:<10}]", "running".green())); + + assert_eq!(plain, "[running ]"); + assert_eq!(colored, "[\u{1b}[32mrunning \u{1b}[0m]"); + } + + #[test] + fn styles_merge_into_one_escape_sequence() { + with_color(true, || { + // Chaining combines into a single style rather than nesting two + // wrappers, so there is one prefix and one reset. + assert_eq!("hi".green().bold().to_string(), "\u{1b}[32;1mhi\u{1b}[0m"); + assert_eq!( + "hi".green().bold().to_string(), + owo_colors::OwoColorize::style(&"hi", Style::new().green().bold()).to_string() + ); + // Order of the calls does not change the resulting style. + assert_eq!( + "hi".green().bold().to_string(), + "hi".bold().green().to_string() + ); + }); + with_color(false, || { + assert_eq!("hi".green().bold().to_string(), "hi"); + }); + } + + #[test] + fn non_string_values_render() { + with_color(false, || { + assert_eq!(8443.green().to_string(), "8443"); + assert_eq!(true.yellow().to_string(), "true"); + }); + } + + /// Render a `dialoguer` confirm prompt, which styles through `console`. + /// + /// Asserting on emitted bytes rather than on `console::colors_enabled()` + /// keeps this honest about what a user would actually see. + fn rendered_prompt() -> String { + use dialoguer::theme::Theme as _; + + let mut out = String::new(); + dialoguer::theme::ColorfulTheme::default() + .format_confirm_prompt(&mut out, "Continue?", Some(false)) + .expect("format confirm prompt"); + out + } + + #[test] + fn console_override_governs_indicatif_and_dialoguer_styling() { + let _guard = SWITCH_LOCK + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner); + // Prompts and progress bars draw to stderr, so their styles consult + // console's stderr switch. `init` sets both; so does this test. + let previous = console::colors_enabled_stderr(); + + console::set_colors_enabled_stderr(true); + let styled = rendered_prompt(); + console::set_colors_enabled_stderr(false); + let plain = rendered_prompt(); + + console::set_colors_enabled_stderr(previous); + + // Positive control first: without it, the plain assertion could pass + // simply because dialoguer stopped emitting anything at all. + assert!( + styled.contains('\u{1b}'), + "console override should permit styling, got: {styled:?}" + ); + assert!( + !plain.contains('\u{1b}'), + "console override should suppress styling, got: {plain:?}" + ); + assert!(plain.contains("Continue?"), "got: {plain:?}"); + } + + #[test] + fn explicit_choice_overrides_environment_and_terminal() { + assert!(resolve(ColorChoice::Always, Some(env("1")), None, false)); + assert!(!resolve(ColorChoice::Never, None, Some(env("1")), true)); + } + + #[test] + fn no_color_disables_when_set_and_non_empty() { + assert!(!resolve(ColorChoice::Auto, Some(env("1")), None, true)); + // Presence is what counts; the value is not interpreted, so a value that + // reads as falsy still disables color. + assert!(!resolve(ColorChoice::Auto, Some(env("0")), None, true)); + // NO_COLOR outranks FORCE_COLOR. + assert!(!resolve( + ColorChoice::Auto, + Some(env("1")), + Some(env("1")), + true + )); + // An empty value is not "set" for the purposes of the convention. + assert!(resolve(ColorChoice::Auto, Some(env("")), None, true)); + } + + #[test] + fn force_color_enables_when_set_and_non_empty() { + assert!(resolve(ColorChoice::Auto, None, Some(env("1")), false)); + // Same presence rule as NO_COLOR: `0` is a value, not an opt-out. + // keys on presence and non-emptiness only. + assert!(resolve(ColorChoice::Auto, None, Some(env("0")), false)); + assert!(!resolve(ColorChoice::Auto, None, Some(env("")), false)); + } + + #[test] + fn term_capability_follows_the_console_rule() { + assert!(term_supports_ansi(Some(OsStr::new("xterm-256color")))); + assert!(term_supports_ansi(Some(OsStr::new("screen")))); + assert!(!term_supports_ansi(Some(OsStr::new("dumb")))); + // Nothing to suggest a capable terminal, so assume none. + assert!(!term_supports_ansi(None)); + // Empty names no terminal type; treated as unset, unlike `console`. + assert!(!term_supports_ansi(Some(OsStr::new("")))); + // Only an exact match counts; `dumb-something` is a different terminal. + assert!(term_supports_ansi(Some(OsStr::new("dumb-but-color")))); + } + + #[test] + fn auto_does_not_style_an_incapable_terminal() { + // A `dumb` terminal is still a terminal, so `is_terminal()` alone would + // wrongly enable color. + assert!(!resolve(ColorChoice::Auto, None, None, false)); + assert!(resolve(ColorChoice::Auto, None, None, true)); + } + + #[test] + fn explicit_requests_outrank_terminal_capability() { + // `--color always` and FORCE_COLOR are for callers who know better than + // the detection, so an incapable terminal must not veto them. + assert!(resolve(ColorChoice::Always, None, None, false)); + assert!(resolve(ColorChoice::Auto, None, Some(env("1")), false)); + // The negative direction still wins over capability too. + assert!(!resolve(ColorChoice::Never, None, None, true)); + assert!(!resolve(ColorChoice::Auto, Some(env("1")), None, true)); + } + + #[test] + fn capability_does_not_rescue_a_redirected_stream() { + // Capability is an additional requirement, not an alternative one. + assert!(!resolve(ColorChoice::Auto, None, None, false)); + } + + #[test] + fn auto_resolves_each_stream_independently() { + // `openshell ... 2> build.log` from a terminal: stdout is styled, the + // log file is not. Resolving both from stdout's answer would put escapes + // in the log. + assert!(resolve(ColorChoice::Auto, None, None, true)); + assert!(!resolve(ColorChoice::Auto, None, None, false)); + } + + #[test] + fn painted_requires_both_streams() { + // A `Painted` value cannot tell whether it is bound for stdout or + // stderr, so it stays plain unless both streams accept escapes. + assert!(with_streams(true, true, painted_enabled)); + assert!(!with_streams(true, false, painted_enabled)); + assert!(!with_streams(false, true, painted_enabled)); + assert!(!with_streams(false, false, painted_enabled)); + + // The rendered consequence: stderr redirected to a file leaves the + // styled text plain rather than writing escapes into it. + assert_eq!( + with_streams(true, false, || "running".green().to_string()), + "running" + ); + } + + #[test] + fn auto_follows_the_terminal() { + assert!(resolve(ColorChoice::Auto, None, None, true)); + assert!(!resolve(ColorChoice::Auto, None, None, false)); + } +} diff --git a/crates/openshell-cli/src/commands/common.rs b/crates/openshell-cli/src/commands/common.rs index 03cb3260c..0f12c3df4 100644 --- a/crates/openshell-cli/src/commands/common.rs +++ b/crates/openshell-cli/src/commands/common.rs @@ -3,6 +3,7 @@ //! Shared helpers, types, and parsing utilities used across CLI command groups. +use crate::color::Colorize; use chrono::DateTime; use dialoguer::{Confirm, theme::ColorfulTheme}; use indicatif::{MultiProgress, ProgressBar, ProgressStyle}; @@ -17,7 +18,6 @@ use openshell_core::proto::{ }; use openshell_core::settings::{self, SettingValueKind}; use openshell_providers::builtin_profiles; -use owo_colors::OwoColorize; use std::collections::HashMap; use std::io::IsTerminal; use std::process::Command; diff --git a/crates/openshell-cli/src/commands/gateway.rs b/crates/openshell-cli/src/commands/gateway.rs index 7a3ebf01f..0e1fe9242 100644 --- a/crates/openshell-cli/src/commands/gateway.rs +++ b/crates/openshell-cli/src/commands/gateway.rs @@ -1,6 +1,7 @@ // SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 +use crate::color::Colorize; use crate::tls::{ TlsOptions, build_insecure_rustls_config, build_rustls_config, grpc_client, require_tls_materials, @@ -20,7 +21,6 @@ use openshell_bootstrap::{ }; use openshell_bootstrap::{GatewayMetadataSource, ListedGateway}; use openshell_core::proto::{GetGatewayInfoRequest, HealthRequest, ServiceStatus}; -use owo_colors::OwoColorize; use std::io::IsTerminal; use std::path::PathBuf; use tonic::{Code, Status}; @@ -1970,7 +1970,6 @@ mod tests { #[test] fn gateway_add_registers_plaintext_loopback_gateway_without_local_flag() { - let _ = rustls::crypto::ring::default_provider().install_default(); let tmpdir = tempfile::tempdir().expect("create tmpdir"); with_tmp_xdg(tmpdir.path(), || { let runtime = tokio::runtime::Runtime::new().expect("create runtime"); @@ -2002,7 +2001,6 @@ mod tests { #[test] fn gateway_add_respects_local_flag_for_plaintext_registrations() { - let _ = rustls::crypto::ring::default_provider().install_default(); let tmpdir = tempfile::tempdir().expect("create tmpdir"); with_tmp_xdg(tmpdir.path(), || { let runtime = tokio::runtime::Runtime::new().expect("create runtime"); @@ -2032,7 +2030,6 @@ mod tests { #[tokio::test] async fn http_health_check_supports_plain_http_endpoints() { - let _ = rustls::crypto::ring::default_provider().install_default(); let listener = TcpListener::bind("127.0.0.1:0").expect("bind listener"); let addr = listener.local_addr().expect("listener addr"); let server = thread::spawn(move || { @@ -2061,7 +2058,6 @@ mod tests { #[test] fn gateway_add_oidc_rolls_back_on_auth_failure() { - let _ = rustls::crypto::ring::default_provider().install_default(); let tmpdir = tempfile::tempdir().expect("create tmpdir"); with_tmp_xdg(tmpdir.path(), || { let runtime = tokio::runtime::Runtime::new().expect("create runtime"); @@ -2118,7 +2114,6 @@ mod tests { } #[test] fn gateway_add_oidc_rollback_keeps_system_active_fallback_userless() { - let _ = rustls::crypto::ring::default_provider().install_default(); let user = tempfile::tempdir().expect("create user tmpdir"); let system = tempfile::tempdir().expect("create system tmpdir"); with_tmp_xdg_and_system(user.path(), system.path(), || { @@ -2159,7 +2154,6 @@ mod tests { #[test] fn gateway_add_cloud_rolls_back_on_auth_failure() { - let _ = rustls::crypto::ring::default_provider().install_default(); let tmpdir = tempfile::tempdir().expect("create tmpdir"); with_tmp_xdg(tmpdir.path(), || { let _no_browser = EnvVarGuard::set("OPENSHELL_NO_BROWSER", "0"); @@ -2217,7 +2211,6 @@ mod tests { } #[test] fn gateway_add_cloud_rollback_keeps_system_active_fallback_userless() { - let _ = rustls::crypto::ring::default_provider().install_default(); let user = tempfile::tempdir().expect("create user tmpdir"); let system = tempfile::tempdir().expect("create system tmpdir"); with_tmp_xdg_and_system(user.path(), system.path(), || { diff --git a/crates/openshell-cli/src/commands/mod.rs b/crates/openshell-cli/src/commands/mod.rs index 09ea2941a..8d75cf5ae 100644 --- a/crates/openshell-cli/src/commands/mod.rs +++ b/crates/openshell-cli/src/commands/mod.rs @@ -3,3 +3,4 @@ pub mod common; pub mod gateway; +pub mod provider; diff --git a/crates/openshell-cli/src/commands/provider.rs b/crates/openshell-cli/src/commands/provider.rs new file mode 100644 index 000000000..e966eb690 --- /dev/null +++ b/crates/openshell-cli/src/commands/provider.rs @@ -0,0 +1,3019 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +use crate::color::Colorize; +use crate::commands::common::{ + format_epoch_ms, format_optional_epoch_ms, parse_credential_expiry_pairs, + parse_credential_pairs, parse_key_value_pairs, parse_secret_material_env_pairs, + truncate_display, truncate_status_field, +}; +use crate::tls::{TlsOptions, grpc_client}; +use dialoguer::Confirm; +use miette::{IntoDiagnostic, Result, WrapErr, miette}; +use openshell_core::proto::ProviderProfileCategory; +use openshell_core::proto::{ + AttachSandboxProviderRequest, ConfigureProviderRefreshRequest, CreateProviderRequest, + DeleteProviderProfileRequest, DeleteProviderRefreshRequest, DeleteProviderRequest, + DetachSandboxProviderRequest, GetProviderProfileRequest, GetProviderRefreshStatusRequest, + GetProviderRequest, GetSandboxRequest, ImportProviderProfilesRequest, + LintProviderProfilesRequest, ListProviderProfilesRequest, ListProvidersRequest, + ListSandboxProvidersRequest, Provider, ProviderCredentialRefreshRecoveryAction, + ProviderCredentialRefreshStatus, ProviderCredentialRefreshStrategy, + ProviderCredentialTokenGrantType, ProviderProfile, ProviderProfileDiagnostic, + ProviderProfileImportItem, RotateProviderCredentialRequest, UpdateProviderProfilesRequest, + UpdateProviderRequest, +}; +use openshell_core::{ObjectId, ObjectName, ObjectWorkspace}; +use openshell_providers::{ + ProviderTypeProfile, RealDiscoveryContext, detect_provider_from_command, discover_from_profile, + normalize_profile_id, normalize_provider_type, parse_profile_json, parse_profile_yaml, + profile_to_json, profile_to_yaml, profiles_to_json, profiles_to_yaml, +}; +use std::collections::{HashMap, HashSet}; +use std::io::IsTerminal; +use std::path::{Path, PathBuf}; +use tonic::{Code, Status}; + +fn aggregate_delete_failures(resource: &str, failures: &[String]) -> Result<()> { + if failures.is_empty() { + Ok(()) + } else { + Err(miette!( + "failed to delete {} {}{}: {}", + failures.len(), + resource, + if failures.len() == 1 { "" } else { "s" }, + failures.join(", ") + )) + } +} + +pub async fn sandbox_provider_list( + server: &str, + name: &str, + output: &str, + workspace: &str, + tls: &TlsOptions, +) -> Result<()> { + let mut client = grpc_client(server, tls).await?; + let response = client + .list_sandbox_providers(ListSandboxProvidersRequest { + sandbox_name: name.to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector(workspace)), + }) + .await + .into_diagnostic()?; + let providers = response.into_inner().providers; + + if crate::output::print_output_collection(output, &providers, attached_provider_to_json)? { + return Ok(()); + } + + if providers.is_empty() { + println!("No providers attached to sandbox {name}."); + return Ok(()); + } + + print_provider_attachment_table(&providers); + Ok(()) +} + +pub async fn sandbox_provider_attach( + server: &str, + name: &str, + provider: &str, + workspace: &str, + tls: &TlsOptions, +) -> Result<()> { + let mut client = grpc_client(server, tls).await?; + + // Fetch current sandbox to get resource_version for CAS + let sandbox = client + .get_sandbox(GetSandboxRequest { + name: name.to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector(workspace)), + }) + .await + .into_diagnostic()? + .into_inner() + .sandbox + .ok_or_else(|| miette::miette!("sandbox not found"))?; + + let resource_version = sandbox.metadata.as_ref().map_or(0, |m| m.resource_version); + + let response = match client + .attach_sandbox_provider(AttachSandboxProviderRequest { + sandbox_name: name.to_string(), + provider_name: provider.to_string(), + expected_resource_version: resource_version, + workspace_scope: Some(openshell_core::proto::workspace_selector(workspace)), + }) + .await + { + Ok(response) => response.into_inner(), + Err(status) if status.code() == Code::Aborted => { + return Err(miette::miette!( + "Failed to attach provider: sandbox was modified by another operation.\n\ + Please retry the command." + ) + .with_source_code(status.message().to_string())); + } + Err(e) => return Err(e).into_diagnostic(), + }; + + if response.attached { + println!( + "{} Attached provider {} to sandbox {}", + "✓".green().bold(), + provider, + name + ); + } else { + println!("Provider {provider} is already attached to sandbox {name}."); + } + Ok(()) +} + +pub async fn sandbox_provider_detach( + server: &str, + name: &str, + provider: &str, + workspace: &str, + tls: &TlsOptions, +) -> Result<()> { + let mut client = grpc_client(server, tls).await?; + + // Fetch current sandbox to get resource_version for CAS + let sandbox = client + .get_sandbox(GetSandboxRequest { + name: name.to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector(workspace)), + }) + .await + .into_diagnostic()? + .into_inner() + .sandbox + .ok_or_else(|| miette::miette!("sandbox not found"))?; + + let resource_version = sandbox.metadata.as_ref().map_or(0, |m| m.resource_version); + + let response = match client + .detach_sandbox_provider(DetachSandboxProviderRequest { + sandbox_name: name.to_string(), + provider_name: provider.to_string(), + expected_resource_version: resource_version, + workspace_scope: Some(openshell_core::proto::workspace_selector(workspace)), + }) + .await + { + Ok(response) => response.into_inner(), + Err(status) if status.code() == Code::Aborted => { + return Err(miette::miette!( + "Failed to detach provider: sandbox was modified by another operation.\n\ + Please retry the command." + ) + .with_source_code(status.message().to_string())); + } + Err(e) => return Err(e).into_diagnostic(), + }; + + if response.detached { + println!( + "{} Detached provider {} from sandbox {}", + "✓".green().bold(), + provider, + name + ); + } else { + println!("Provider {provider} was not attached to sandbox {name}."); + } + Ok(()) +} + +fn print_provider_attachment_table(providers: &[Provider]) { + print!("{}", format_provider_attachment_table(providers, true)); +} + +fn attached_provider_to_json(provider: &Provider) -> serde_json::Value { + let mut config_keys = provider.config.keys().cloned().collect::>(); + config_keys.sort(); + + serde_json::json!({ + "name": provider.object_name(), + "type": provider.r#type, + "credential_keys": provider_credential_keys(provider), + "config_keys": config_keys, + }) +} + +fn format_provider_attachment_table(providers: &[Provider], color: bool) -> String { + use std::fmt::Write as _; + + let name_width = providers + .iter() + .map(|provider| provider.object_name().len()) + .max() + .unwrap_or(4) + .max(4); + let type_width = providers + .iter() + .map(|provider| provider.r#type.len()) + .max() + .unwrap_or(4) + .max(4); + + let name_header = if color { + "NAME".bold().to_string() + } else { + "NAME".to_string() + }; + let type_header = if color { + "TYPE".bold().to_string() + } else { + "TYPE".to_string() + }; + let credential_keys_header = if color { + "CREDENTIAL_KEYS".bold().to_string() + } else { + "CREDENTIAL_KEYS".to_string() + }; + let config_keys_header = if color { + "CONFIG_KEYS".bold().to_string() + } else { + "CONFIG_KEYS".to_string() + }; + + let mut output = String::new(); + let _ = writeln!( + output, + "{name_header: Option { + detect_provider_from_command(command).map(str::to_string) +} + +/// Ensure all required providers exist. +/// +/// `explicit_names` are provider **names** supplied via `--provider`. They are +/// passed through directly; the server validates they exist at sandbox creation. +/// +/// `inferred_types` are provider **types** inferred from the trailing command +/// (e.g. `claude` -> type `"claude-code"`). These are resolved to provider names via +/// a type→name lookup, and missing types may be auto-created interactively. +/// +/// Returns a deduplicated list of provider **names** suitable for +/// `SandboxSpec.providers`. +pub async fn ensure_required_providers( + client: &mut crate::tls::GrpcClient, + explicit_names: &[String], + inferred_types: &[String], + auto_providers_override: Option, + workspace: &str, +) -> Result> { + if explicit_names.is_empty() && inferred_types.is_empty() { + return Ok(Vec::new()); + } + + let mut configured_names: Vec = Vec::new(); + let mut seen_names: HashSet = HashSet::new(); + + // ── Fetch all existing providers ───────────────────────────────────── + // Build both a name set (for explicit --provider lookups) and a + // type-to-name map (for inferred provider resolution). + let mut known_names: HashSet = HashSet::new(); + let mut type_to_name: HashMap = HashMap::new(); + { + let mut offset = 0_u32; + let limit = 100_u32; + loop { + let response = client + .list_providers(ListProvidersRequest { + limit, + offset, + workspace_scope: Some(openshell_core::proto::workspace_selector(workspace)), + }) + .await + .into_diagnostic()?; + let providers = response.into_inner().providers; + for provider in &providers { + known_names.insert(provider.object_name().to_string()); + if !provider.r#type.is_empty() { + let type_lower = provider.r#type.to_ascii_lowercase(); + type_to_name + .entry(type_lower) + .or_insert_with(|| provider.object_name().to_string()); + } + } + if providers.len() < limit as usize { + break; + } + offset = offset.saturating_add(limit); + } + } + + // ── Explicit provider names ────────────────────────────────────────── + // If the name exists on the server, use it directly. Otherwise, if the + // name matches a known provider type, auto-create a provider of that + // type with the requested name. + for name in explicit_names { + if known_names.contains(name) { + if seen_names.insert(name.clone()) { + configured_names.push(name.clone()); + } + } else { + let profile_id = name.trim(); + let profile = fetch_provider_profile(client, profile_id, workspace) + .await + .map_err(|_| { + miette::miette!( + "provider '{name}' not found and no provider profile named '{profile_id}' is available. \ + Create or import the profile first, then create the provider" + ) + })?; + let provider_type = profile.id; + auto_create_provider( + client, + &provider_type, + Some(name), + auto_providers_override, + &mut seen_names, + &mut configured_names, + workspace, + ) + .await?; + // Record the type mapping so the inferred-types pass below + // doesn't attempt to create a duplicate provider. + type_to_name + .entry(provider_type.to_ascii_lowercase()) + .or_insert_with(|| name.clone()); + } + } + + // ── Resolve inferred provider types ────────────────────────────────── + if !inferred_types.is_empty() { + // Collect resolved names for types that already have a provider. + for t in inferred_types { + if let Some(name) = type_to_name.get(&t.to_ascii_lowercase()) + && seen_names.insert(name.clone()) + { + configured_names.push(name.clone()); + } + } + + let missing = inferred_types + .iter() + .filter(|t| !type_to_name.contains_key(&t.to_ascii_lowercase())) + .cloned() + .collect::>(); + + for provider_type in missing { + auto_create_provider( + client, + &provider_type, + None, + auto_providers_override, + &mut seen_names, + &mut configured_names, + workspace, + ) + .await?; + } + } + + Ok(configured_names) +} + +/// Prompt for (or auto-confirm) creation of a provider from local credentials. +/// +/// When `preferred_name` is `Some`, the provider is created with that exact +/// name (used for explicit `--provider ` values). When `None`, the name +/// defaults to the type and retries with suffixes on conflict (used for +/// inferred provider types). +async fn auto_create_provider( + client: &mut crate::tls::GrpcClient, + provider_type: &str, + preferred_name: Option<&str>, + auto_providers_override: Option, + seen_names: &mut HashSet, + configured_names: &mut Vec, + workspace: &str, +) -> Result<()> { + eprintln!("Missing provider: {provider_type}"); + + // --no-auto-providers: skip silently. + if auto_providers_override == Some(false) { + eprintln!( + "{} Skipping provider '{provider_type}' (--no-auto-providers)", + "!".yellow(), + ); + eprintln!(); + return Ok(()); + } + + // No override and non-interactive: error. + if auto_providers_override.is_none() && !std::io::stdin().is_terminal() { + return Err(miette::miette!( + "missing required provider '{provider_type}'. Create it first with \ + `openshell provider create --type {provider_type} --name {provider_type} --from-existing`, \ + pass --auto-providers to auto-create, or set it up manually from inside the sandbox" + )); + } + + // --auto-providers: auto-confirm; otherwise prompt. + let should_create = if auto_providers_override == Some(true) { + true + } else { + Confirm::new() + .with_prompt("Create from local credentials?") + .default(true) + .interact() + .into_diagnostic()? + }; + + if !should_create { + eprintln!("{} Skipping provider '{provider_type}'", "!".yellow()); + eprintln!(); + return Ok(()); + } + + let profile = fetch_provider_profile(client, provider_type, workspace).await?; + let discovered = discover_existing_provider_data(client, provider_type, workspace) + .await + .map_err(|err| miette::miette!("failed to discover provider '{provider_type}': {err}"))?; + let discovered = match discovered { + Some(discovered) => discovered, + None if provider_profile_allows_empty_credentials(&profile) => { + openshell_providers::DiscoveredProvider::default() + } + None => { + return Err(miette::miette!( + "no existing local credentials found for provider profile '{provider_type}'. \ + Create it first with `openshell provider create --type {provider_type} --name {provider_type} --credential `" + )); + } + }; + + if let Some(exact_name) = preferred_name { + // Explicit name: create with exactly that name, no retries. + let request = CreateProviderRequest { + provider: Some(Provider { + metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { + id: String::new(), + name: exact_name.to_string(), + created_at_ms: 0, + labels: HashMap::new(), + resource_version: 0, + annotations: HashMap::new(), + workspace: workspace.to_string(), + deletion_timestamp_ms: 0, + }), + r#type: provider_type.to_string(), + credentials: discovered.credentials.clone(), + config: discovered.config.clone(), + credential_expires_at_ms: HashMap::new(), + profile_workspace: workspace.to_string(), + credential_handles: HashMap::new(), + }), + workspace_scope: Some(openshell_core::proto::workspace_selector(workspace)), + }; + + let response = client.create_provider(request).await.map_err(|status| { + miette::miette!("failed to create provider '{exact_name}': {status}") + })?; + let provider = response + .into_inner() + .provider + .ok_or_else(|| miette::miette!("provider missing from response"))?; + eprintln!( + "{} Created provider {} ({}) from existing local state", + "✓".green().bold(), + provider.object_name(), + provider.r#type + ); + if seen_names.insert(provider.object_name().to_string()) { + configured_names.push(provider.object_name().to_string()); + } + } else { + // Inferred type: try type as name, then suffixed variants. + let mut created = false; + for attempt in 0..5 { + let name = if attempt == 0 { + provider_type.to_string() + } else { + format!("{provider_type}-{attempt}") + }; + + let request = CreateProviderRequest { + provider: Some(Provider { + metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { + id: String::new(), + name: name.clone(), + created_at_ms: 0, + labels: HashMap::new(), + resource_version: 0, + annotations: HashMap::new(), + workspace: workspace.to_string(), + deletion_timestamp_ms: 0, + }), + r#type: provider_type.to_string(), + credentials: discovered.credentials.clone(), + config: discovered.config.clone(), + credential_expires_at_ms: HashMap::new(), + profile_workspace: workspace.to_string(), + credential_handles: HashMap::new(), + }), + workspace_scope: Some(openshell_core::proto::workspace_selector(workspace)), + }; + + match client.create_provider(request).await { + Ok(response) => { + let provider = response + .into_inner() + .provider + .ok_or_else(|| miette::miette!("provider missing from response"))?; + eprintln!( + "{} Created provider {} ({}) from existing local state", + "✓".green().bold(), + provider.object_name(), + provider.r#type + ); + if seen_names.insert(provider.object_name().to_string()) { + configured_names.push(provider.object_name().to_string()); + } + created = true; + break; + } + Err(status) if status.code() == Code::AlreadyExists => {} + Err(status) => { + return Err(miette::miette!( + "failed to create provider for type '{provider_type}': {status}" + )); + } + } + } + + if !created { + return Err(miette::miette!( + "failed to create provider for type '{provider_type}' after name retries" + )); + } + } + + eprintln!(); + Ok(()) +} + +fn read_gcloud_adc() -> Result<(String, String, String)> { + let path = if let Some(env_path) = std::env::var("GOOGLE_APPLICATION_CREDENTIALS") + .ok() + .filter(|v| !v.is_empty()) + { + PathBuf::from(env_path) + } else if let Some(config_dir) = std::env::var("CLOUDSDK_CONFIG") + .ok() + .filter(|v| !v.is_empty()) + { + PathBuf::from(config_dir).join("application_default_credentials.json") + } else { + let home = std::env::var("HOME") + .map_err(|_| miette::miette!("HOME is not set; cannot locate gcloud ADC file"))?; + PathBuf::from(home) + .join(".config") + .join("gcloud") + .join("application_default_credentials.json") + }; + + let content = std::fs::read_to_string(&path).map_err(|err| { + miette::miette!( + "failed to read gcloud ADC file at {}: {}. \ + Run: gcloud auth application-default login", + path.display(), + err + ) + })?; + + let json: serde_json::Value = serde_json::from_str(&content) + .map_err(|err| miette::miette!("failed to parse gcloud ADC file: {err}"))?; + + let cred_type = json.get("type").and_then(|v| v.as_str()); + match cred_type { + Some("service_account") => { + return Err(miette::miette!( + "Application Default Credentials are a service account key, not user credentials. \ + To use a service account, create the provider with the service account JSON key \ + and configure gateway-managed refresh for 'GOOGLE_VERTEX_AI_SERVICE_ACCOUNT_TOKEN'. \ + See: openshell provider create --help" + )); + } + Some("authorized_user") => {} + Some(other) => { + return Err(miette::miette!( + "Application Default Credentials have unsupported type '{other}' \ + (expected 'authorized_user'). \ + Run: gcloud auth application-default login" + )); + } + None => { + return Err(miette::miette!( + "gcloud ADC file is missing the 'type' field. \ + The file may be malformed. \ + Run: gcloud auth application-default login" + )); + } + } + + let client_id = json + .get("client_id") + .and_then(|v| v.as_str()) + .filter(|s| !s.is_empty()) + .ok_or_else(|| miette::miette!("gcloud ADC file is missing 'client_id'"))? + .to_string(); + + let client_secret = json + .get("client_secret") + .and_then(|v| v.as_str()) + .filter(|s| !s.is_empty()) + .ok_or_else(|| miette::miette!("gcloud ADC file is missing 'client_secret'"))? + .to_string(); + + let refresh_token = json + .get("refresh_token") + .and_then(|v| v.as_str()) + .filter(|s| !s.is_empty()) + .ok_or_else(|| miette::miette!("gcloud ADC file is missing 'refresh_token'"))? + .to_string(); + + Ok((client_id, client_secret, refresh_token)) +} + +async fn rollback_provider_create_after_gcloud_adc_failure( + client: &mut crate::tls::GrpcClient, + provider_name: &str, + stage: &str, + source: &Status, + workspace: &str, +) -> Result<()> { + match client + .delete_provider(DeleteProviderRequest { + name: provider_name.to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector(workspace)), + }) + .await + { + Ok(_) => Err(miette!( + "failed to {stage} credentials from gcloud ADC for provider '{provider_name}': {source}. \ + The provider was rolled back successfully." + )), + Err(cleanup_err) => { + eprintln!( + "{} Failed to clean up provider '{}' after {} failed: {}. \ + Run 'openshell provider delete {}' to remove it manually.", + "⚠".yellow(), + provider_name, + stage, + cleanup_err, + provider_name + ); + Err(miette!( + "failed to {stage} credentials from gcloud ADC for provider '{provider_name}': {source}. \ + Cleanup also failed, so the provider may still exist. \ + Run 'openshell provider delete {provider_name}' to remove it manually." + )) + } + } +} + +async fn fetch_provider_profile( + client: &mut crate::tls::GrpcClient, + provider_type: &str, + workspace: &str, +) -> Result { + let requested = provider_type.trim(); + let response = match fetch_provider_profile_exact(client, requested, workspace).await { + Ok(response) => response, + Err(status) if status.code() == Code::NotFound => { + let Some(alias) = normalize_provider_type(requested) + .filter(|alias| normalize_profile_id(requested).as_deref() != Some(*alias)) + else { + return Err(miette::miette!( + "provider profile '{requested}' not found; import a matching profile before using this provider type" + )); + }; + fetch_provider_profile_exact(client, alias, workspace) + .await + .map_err(|fallback_status| { + if fallback_status.code() == Code::NotFound { + miette::miette!( + "provider profile '{requested}' not found; import a matching profile before using this provider type" + ) + } else { + miette::miette!(fallback_status.to_string()) + } + })? + } + Err(status) => return Err(miette::miette!(status.to_string())), + }; + + Ok(response) +} + +async fn fetch_provider_profile_exact( + client: &mut crate::tls::GrpcClient, + provider_type: &str, + workspace: &str, +) -> std::result::Result { + client + .get_provider_profile(GetProviderProfileRequest { + id: provider_type.to_string(), + workspace: workspace.to_string(), + }) + .await + .and_then(|response| { + response + .into_inner() + .profile + .ok_or_else(|| Status::internal("provider profile missing from response")) + }) +} + +async fn discover_existing_provider_data( + client: &mut crate::tls::GrpcClient, + provider_type: &str, + workspace: &str, +) -> Result> { + let profile = fetch_provider_profile(client, provider_type, workspace).await?; + let profile = ProviderTypeProfile::from_proto(&profile); + let discovered = discover_from_profile(&profile, &RealDiscoveryContext).map_err(|err| { + miette::miette!("failed to discover existing provider data from profile: {err}") + })?; + + Ok(discovered) +} + +/// Canonical provider type string for Google Vertex AI. +const VERTEX_AI_PROVIDER_TYPE: &str = "google-vertex-ai"; + +/// Canonical provider type string for Google Cloud (GCP APIs). +const GOOGLE_CLOUD_PROVIDER_TYPE: &str = "google-cloud"; + +fn missing_credentials_error(provider_type: &str) -> miette::Report { + if provider_type == VERTEX_AI_PROVIDER_TYPE { + return miette::miette!( + "no credentials resolved for provider type '{provider_type}'. \ + Set GOOGLE_VERTEX_AI_TOKEN, VERTEX_AI_TOKEN, \ + GOOGLE_VERTEX_AI_SERVICE_ACCOUNT_TOKEN, or VERTEX_AI_SERVICE_ACCOUNT_TOKEN; \ + or use --from-gcloud-adc or --from-existing with those env vars set." + ); + } + + if provider_type == GOOGLE_CLOUD_PROVIDER_TYPE { + return miette::miette!( + "no credentials resolved for provider type '{provider_type}'. \ + Set GCP_ADC_ACCESS_TOKEN or GCP_SA_ACCESS_TOKEN; \ + or use --from-gcloud-adc / --from-existing with those env vars set." + ); + } + + miette::miette!( + "no credentials resolved for provider type '{provider_type}'. \ + Use --credential KEY[=VALUE], --runtime-credentials for runtime-resolved profile credentials, or --from-existing \ + with the appropriate env vars set." + ) +} + +async fn provider_credential_from_oidc_token( + credentials: &[String], + profile: Option<&ProviderProfile>, + tls: &TlsOptions, +) -> Result<(HashMap, HashMap)> { + let credential_key = oidc_subject_credential_key(credentials, profile)?; + + let gateway_name = tls.gateway_name().ok_or_else(|| { + miette::miette!("--from-oidc-token requires an active named OIDC gateway") + })?; + let bundle = + crate::oidc_auth::ensure_valid_oidc_token_bundle(gateway_name, tls.gateway_insecure) + .await + .map_err(|err| { + miette::miette!( + "failed to load or refresh OIDC token for gateway '{gateway_name}' while preparing provider credential: {err}" + ) + })?; + + let mut credential_map = HashMap::new(); + credential_map.insert(credential_key.clone(), bundle.access_token); + + let mut credential_expires_at_ms = HashMap::new(); + if let Some(expires_at) = bundle.expires_at { + let expires_at_ms = i64::try_from(expires_at) + .unwrap_or(i64::MAX / 1000) + .saturating_mul(1000); + credential_expires_at_ms.insert(credential_key, expires_at_ms); + } + + Ok((credential_map, credential_expires_at_ms)) +} + +fn oidc_subject_credential_key( + credentials: &[String], + profile: Option<&ProviderProfile>, +) -> Result { + if credentials.len() > 1 { + return Err(miette::miette!( + "--from-oidc-token accepts at most one --credential KEY destination" + )); + } + + if let Some(credential) = credentials.first() { + let credential = credential.trim(); + if credential.is_empty() || credential.contains('=') { + return Err(miette::miette!( + "--from-oidc-token requires --credential KEY without an inline value" + )); + } + if let Some(profile) = profile { + ensure_profile_declares_subject_credential(profile, credential)?; + } + return Ok(credential.to_string()); + } + + let Some(profile) = profile else { + return Err(miette::miette!( + "--from-oidc-token requires --credential KEY when the provider profile is unavailable" + )); + }; + + infer_oidc_subject_credential_from_profile(profile) +} + +fn ensure_profile_declares_subject_credential( + profile: &ProviderProfile, + credential: &str, +) -> Result<()> { + let matches = token_exchange_subject_credentials(profile); + if matches.iter().any(|candidate| candidate == credential) { + return Ok(()); + } + Err(miette::miette!( + "credential '{credential}' is not declared as a token-exchange subject credential in provider profile '{}'; expected one of: {}", + profile.id, + matches.join(", ") + )) +} + +fn infer_oidc_subject_credential_from_profile(profile: &ProviderProfile) -> Result { + let matches = token_exchange_subject_credentials(profile); + match matches.as_slice() { + [credential] => Ok(credential.clone()), + [] => Err(miette::miette!( + "provider profile '{}' does not declare a token-exchange subject credential; pass --credential KEY", + profile.id + )), + _ => Err(miette::miette!( + "provider profile '{}' declares multiple token-exchange subject credentials ({}); pass --credential KEY", + profile.id, + matches.join(", ") + )), + } +} + +fn token_exchange_subject_credentials(profile: &ProviderProfile) -> Vec { + let mut matches = Vec::new(); + for credential in &profile.credentials { + let Some(token_grant) = credential.token_grant.as_ref() else { + continue; + }; + if ProviderCredentialTokenGrantType::try_from(token_grant.grant_type).ok() + != Some(ProviderCredentialTokenGrantType::TokenExchange) + { + continue; + } + let Some(subject_token) = token_grant.subject_token.as_ref() else { + continue; + }; + if subject_token.source != "provider_credential" || subject_token.credential.is_empty() { + continue; + } + if !matches.contains(&subject_token.credential) { + matches.push(subject_token.credential.clone()); + } + } + matches +} + +#[allow(clippy::too_many_arguments)] +pub async fn provider_create( + server: &str, + name: &str, + provider_type: &str, + from_existing: bool, + credentials: &[String], + from_gcloud_adc: bool, + config: &[String], + workspace: &str, + tls: &TlsOptions, +) -> Result<()> { + let credential_source = match (from_existing, from_gcloud_adc) { + (true, true) => { + return Err(miette::miette!( + "--from-gcloud-adc cannot be combined with --from-existing, --from-oidc-token, or --credential; it also cannot be combined with --runtime-credentials" + )); + } + (true, false) => ProviderCreateCredentialSource::Existing, + (false, true) => ProviderCreateCredentialSource::GcloudAdc, + (false, false) => ProviderCreateCredentialSource::ExplicitCredentials, + }; + provider_create_with_options(ProviderCreateOptions { + server, + name, + provider_type, + credentials, + credential_source, + config, + workspace, + profile_workspace: workspace, + tls, + }) + .await +} + +pub struct ProviderCreateOptions<'a> { + pub server: &'a str, + pub name: &'a str, + pub provider_type: &'a str, + pub credentials: &'a [String], + pub credential_source: ProviderCreateCredentialSource, + pub config: &'a [String], + pub workspace: &'a str, + pub profile_workspace: &'a str, + pub tls: &'a TlsOptions, +} + +#[derive(Clone, Copy, Debug, Eq, PartialEq)] +pub enum ProviderCreateCredentialSource { + ExplicitCredentials, + Existing, + GcloudAdc, + OidcToken, + Runtime, +} + +pub async fn provider_create_with_options(options: ProviderCreateOptions<'_>) -> Result<()> { + let ProviderCreateOptions { + server, + name, + provider_type, + credentials, + credential_source, + config, + workspace, + profile_workspace, + tls, + } = options; + + let from_existing = credential_source == ProviderCreateCredentialSource::Existing; + let from_gcloud_adc = credential_source == ProviderCreateCredentialSource::GcloudAdc; + let from_oidc_token = credential_source == ProviderCreateCredentialSource::OidcToken; + let runtime_credentials = credential_source == ProviderCreateCredentialSource::Runtime; + + if from_gcloud_adc && !credentials.is_empty() { + return Err(miette::miette!( + "--from-gcloud-adc cannot be combined with --from-existing, --from-oidc-token, or --credential; it also cannot be combined with --runtime-credentials" + )); + } + if from_existing && !credentials.is_empty() { + return Err(miette::miette!( + "--from-existing cannot be combined with --credential" + )); + } + if runtime_credentials && !credentials.is_empty() { + return Err(miette::miette!( + "--runtime-credentials cannot be combined with --credential" + )); + } + + let mut client = grpc_client(server, tls).await?; + + let profile_id = provider_type.trim(); + if profile_id.is_empty() { + return Err(miette::miette!("provider type is required")); + } + let provider_profile = fetch_provider_profile(&mut client, profile_id, profile_workspace) + .await + .map_err(|err| { + miette::miette!("unsupported provider type or profile: {profile_id} ({err})") + })?; + let provider_type = provider_profile.id.clone(); + + let adc_credential_key = if from_gcloud_adc { + let profile = ProviderTypeProfile::from_proto(&provider_profile); + let adc_cred = profile.adc_credential().ok_or_else(|| { + miette::miette!( + "--from-gcloud-adc is not supported for '{provider_type}' providers \ + (no ADC-compatible credential in the provider profile)" + ) + })?; + Some( + adc_cred + .env_vars + .first() + .ok_or_else(|| { + miette::miette!( + "ADC credential in '{provider_type}' profile has no env_vars declared" + ) + })? + .clone(), + ) + } else { + None + }; + + let oidc_profile = if from_oidc_token { + Some(provider_profile.clone()) + } else { + None + }; + + let (mut credential_map, oidc_credential_expires_at_ms) = if from_oidc_token { + provider_credential_from_oidc_token(credentials, oidc_profile.as_ref(), tls).await? + } else { + (parse_credential_pairs(credentials)?, HashMap::new()) + }; + let mut config_map = parse_key_value_pairs(config, "--config")?; + + if from_existing { + let discovered = + discover_existing_provider_data(&mut client, &provider_type, profile_workspace).await?; + let Some(discovered) = discovered else { + return Err(miette::miette!( + "no existing local credentials/config found for provider type '{provider_type}'" + )); + }; + + for (key, value) in discovered.credentials { + credential_map.entry(key).or_insert(value); + } + for (key, value) in discovered.config { + config_map.entry(key).or_insert(value); + } + } + + if credential_map.is_empty() { + if from_existing { + return Err(missing_credentials_error(&provider_type)); + } + if runtime_credentials && !provider_profile_allows_runtime_credentials(&provider_profile) { + return Err(miette::miette!( + "--runtime-credentials is only valid for provider profiles whose required credentials are resolved at runtime" + )); + } + if !provider_profile_allows_empty_credentials(&provider_profile) { + return Err(missing_credentials_error(&provider_type)); + } + } + + // Validate and read the ADC file BEFORE creating the provider so that + // a bad/missing ADC does not leave an orphan provider behind. Bundle the + // credential key with the material so they stay coupled. + let gcloud_adc_bootstrap = if from_gcloud_adc { + let (client_id, client_secret, refresh_token) = read_gcloud_adc()?; + let key = adc_credential_key.expect("set when from_gcloud_adc is true"); + Some((key, client_id, client_secret, refresh_token)) + } else { + None + }; + + let response = client + .create_provider(CreateProviderRequest { + provider: Some(Provider { + metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { + id: String::new(), + name: name.to_string(), + created_at_ms: 0, + labels: HashMap::new(), + resource_version: 0, + annotations: HashMap::new(), + workspace: workspace.to_string(), + deletion_timestamp_ms: 0, + }), + r#type: provider_type.clone(), + credentials: credential_map, + config: config_map, + credential_expires_at_ms: oidc_credential_expires_at_ms, + profile_workspace: profile_workspace.to_string(), + credential_handles: HashMap::new(), + }), + workspace_scope: Some(openshell_core::proto::workspace_selector(workspace)), + }) + .await + .into_diagnostic()?; + + let provider = response + .into_inner() + .provider + .ok_or_else(|| miette::miette!("provider missing from response"))?; + let provider_name = provider.object_name().to_string(); + + if let Some((adc_credential_key, client_id, client_secret, refresh_token)) = + gcloud_adc_bootstrap + { + let mut material = HashMap::new(); + material.insert("client_id".to_string(), client_id); + material.insert("client_secret".to_string(), client_secret); + material.insert("refresh_token".to_string(), refresh_token); + + if let Err(configure_err) = client + .configure_provider_refresh(ConfigureProviderRefreshRequest { + provider: provider_name.clone(), + credential_key: adc_credential_key.clone(), + strategy: ProviderCredentialRefreshStrategy::Oauth2RefreshToken as i32, + material, + secret_material_keys: vec![ + "client_secret".to_string(), + "refresh_token".to_string(), + ], + expires_at_ms: None, + workspace_scope: Some(openshell_core::proto::workspace_selector(workspace)), + }) + .await + { + return rollback_provider_create_after_gcloud_adc_failure( + &mut client, + &provider_name, + "configure", + &configure_err, + workspace, + ) + .await; + } + + if let Err(rotate_err) = client + .rotate_provider_credential(RotateProviderCredentialRequest { + provider: provider_name.clone(), + credential_key: adc_credential_key, + workspace_scope: Some(openshell_core::proto::workspace_selector(workspace)), + }) + .await + { + return rollback_provider_create_after_gcloud_adc_failure( + &mut client, + &provider_name, + "mint the initial access token for", + &rotate_err, + workspace, + ) + .await; + } + + println!("{} Created provider {}", "✓".green().bold(), provider_name); + println!("Configured GCP credentials from gcloud ADC and minted the initial access token"); + return Ok(()); + } + + println!("{} Created provider {}", "✓".green().bold(), provider_name); + Ok(()) +} + +fn provider_profile_allows_empty_credentials(profile: &ProviderProfile) -> bool { + ProviderTypeProfile::from_proto(profile).allows_empty_provider_credentials() +} + +fn provider_profile_allows_runtime_credentials(profile: &ProviderProfile) -> bool { + ProviderTypeProfile::from_proto(profile).allows_runtime_provider_credentials() +} + +pub async fn provider_get( + server: &str, + name: &str, + workspace: &str, + tls: &TlsOptions, +) -> Result<()> { + let mut client = grpc_client(server, tls).await?; + let response = client + .get_provider(GetProviderRequest { + name: name.to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector(workspace)), + }) + .await + .into_diagnostic()?; + + let provider = response + .into_inner() + .provider + .ok_or_else(|| miette::miette!("provider missing from response"))?; + + let credential_keys = provider_credential_keys(&provider); + let config_keys = provider.config.keys().cloned().collect::>(); + + println!("{}", "Provider:".cyan().bold()); + println!(); + println!(" {} {}", "Id:".dimmed(), provider.object_id()); + println!(" {} {}", "Name:".dimmed(), provider.object_name()); + println!(" {} {}", "Type:".dimmed(), provider.r#type); + println!( + " {} {}", + "Resource version:".dimmed(), + provider.metadata.as_ref().map_or(0, |m| m.resource_version) + ); + println!( + " {} {}", + "Credential keys:".dimmed(), + if credential_keys.is_empty() { + "".to_string() + } else { + credential_keys.join(", ") + } + ); + println!( + " {} {}", + "Config keys:".dimmed(), + if config_keys.is_empty() { + "".to_string() + } else { + config_keys.join(", ") + } + ); + + Ok(()) +} + +fn provider_to_json(provider: &Provider) -> serde_json::Value { + let mut obj = serde_json::Map::new(); + + // Core fields + obj.insert("id".to_string(), serde_json::json!(provider.object_id())); + obj.insert( + "name".to_string(), + serde_json::json!(provider.object_name()), + ); + obj.insert( + "workspace".to_string(), + serde_json::json!(provider.object_workspace()), + ); + obj.insert("type".to_string(), serde_json::json!(provider.r#type)); + + // Credential keys (NEVER values - security) + let credential_keys = provider_credential_keys(provider); + obj.insert( + "credential_keys".to_string(), + serde_json::json!(credential_keys), + ); + + // Config keys (keys only, not values) + if !provider.config.is_empty() { + let config_keys: Vec = provider.config.keys().cloned().collect(); + obj.insert("config_keys".to_string(), serde_json::json!(config_keys)); + } + + // Metadata fields (only if metadata exists) + if let Some(meta) = &provider.metadata { + if !meta.labels.is_empty() { + obj.insert("labels".to_string(), serde_json::json!(meta.labels)); + } + if meta.resource_version != 0 { + obj.insert( + "resource_version".to_string(), + serde_json::json!(meta.resource_version), + ); + } + if meta.created_at_ms != 0 { + obj.insert( + "created_at".to_string(), + serde_json::json!(format_epoch_ms(meta.created_at_ms)), + ); + } + } + + // Credential expiration times (only if present) + if !provider.credential_expires_at_ms.is_empty() { + obj.insert( + "credential_expires_at_ms".to_string(), + serde_json::json!(provider.credential_expires_at_ms), + ); + } + + serde_json::Value::Object(obj) +} + +fn provider_credential_keys(provider: &Provider) -> Vec { + let mut keys: Vec = provider + .credentials + .keys() + .chain(provider.credential_handles.keys()) + .cloned() + .collect(); + keys.sort(); + keys.dedup(); + keys +} + +#[allow(clippy::too_many_arguments)] +pub async fn provider_list( + server: &str, + limit: u32, + offset: u32, + names_only: bool, + output: &str, + workspace: &str, + all_workspaces: bool, + tls: &TlsOptions, +) -> Result<()> { + let mut client = grpc_client(server, tls).await?; + let response = client + .list_providers(ListProvidersRequest { + limit, + offset, + workspace_scope: Some(if all_workspaces { + openshell_core::proto::all_workspaces_selector() + } else { + openshell_core::proto::workspace_selector(workspace) + }), + }) + .await + .into_diagnostic()?; + let providers = response.into_inner().providers; + + // Handle structured output formats (json, yaml) + if crate::output::print_output_collection(output, &providers, provider_to_json)? { + return Ok(()); + } + + if providers.is_empty() { + if !names_only { + println!("No providers found."); + } + return Ok(()); + } + + if names_only { + for provider in &providers { + if all_workspaces { + println!("{}/{}", provider.object_workspace(), provider.object_name()); + } else { + println!("{}", provider.object_name()); + } + } + return Ok(()); + } + + let ws_width = if all_workspaces { + providers + .iter() + .map(|p| p.object_workspace().len()) + .max() + .unwrap_or(9) + .max(9) + } else { + 0 + }; + let name_width = providers + .iter() + .map(|provider| provider.object_name().len()) + .max() + .unwrap_or(4) + .max(4); + let type_width = providers + .iter() + .map(|provider| provider.r#type.len()) + .max() + .unwrap_or(4) + .max(4); + + if all_workspaces { + println!( + "{: Result<()> { + let mut client = grpc_client(server, tls).await?; + let response = client + .list_provider_profiles(ListProviderProfilesRequest { + limit: 100, + offset: 0, + workspace: workspace.to_string(), + }) + .await + .into_diagnostic()?; + let mut profiles = response.into_inner().profiles; + profiles.sort_by(|left, right| { + left.category + .cmp(&right.category) + .then_with(|| left.id.cmp(&right.id)) + }); + let dto_profiles = profiles + .iter() + .map(ProviderTypeProfile::from_proto) + .collect::>(); + + if crate::output::print_output_direct( + output, + || profiles_to_json(&dto_profiles).into_diagnostic(), + || profiles_to_yaml(&dto_profiles).into_diagnostic(), + )? { + return Ok(()); + } + + if profiles.is_empty() { + println!("No provider profiles found."); + return Ok(()); + } + + println!("{}", "Available Provider Profiles:".cyan().bold()); + let id_width = provider_profile_id_width(&profiles); + let display_width = provider_profile_display_width(&profiles); + let source_width = provider_profile_source_width(&profiles); + let scope_width = provider_profile_scope_width(&profiles); + let mut current_category = i32::MIN; + for profile in &profiles { + if profile.category != current_category { + current_category = profile.category; + println!(); + println!(" {}", display_provider_category(current_category).bold()); + print_provider_type_header(id_width, scope_width, source_width, display_width); + } + print_provider_type_row(profile, id_width, scope_width, source_width, display_width); + } + + Ok(()) +} + +pub async fn provider_profile_export( + server: &str, + id: &str, + output: &str, + workspace: &str, + tls: &TlsOptions, +) -> Result<()> { + let rendered = provider_profile_export_text(server, id, output, workspace, tls).await?; + if output == "json" { + println!("{rendered}"); + } else { + print!("{rendered}"); + } + Ok(()) +} + +pub async fn provider_profile_export_text( + server: &str, + id: &str, + output: &str, + workspace: &str, + tls: &TlsOptions, +) -> Result { + let mut client = grpc_client(server, tls).await?; + let response = client + .get_provider_profile(GetProviderProfileRequest { + id: id.to_string(), + workspace: workspace.to_string(), + }) + .await + .into_diagnostic()?; + let profile = response + .into_inner() + .profile + .ok_or_else(|| miette!("provider profile '{id}' not found"))?; + let profile = ProviderTypeProfile::from_proto(&profile); + + match output { + "json" => profile_to_json(&profile).into_diagnostic(), + "yaml" => profile_to_yaml(&profile).into_diagnostic(), + "table" => Err(miette!( + "profile export supports '-o yaml' and '-o json'; table output is not supported" + )), + _ => Err(miette!("unsupported output format: {output}")), + } +} + +pub async fn provider_profile_import( + server: &str, + file: Option<&Path>, + from: Option<&Path>, + workspace: &str, + tls: &TlsOptions, +) -> Result<()> { + let (items, mut diagnostics) = load_profile_import_items(file, from)?; + if items.is_empty() && diagnostics.is_empty() { + return Err(miette!("no provider profile files found")); + } + if profile_diagnostics_have_errors(&diagnostics) { + print_profile_diagnostics(&diagnostics); + return Err(miette!("provider profile import failed")); + } + + let mut client = grpc_client(server, tls).await?; + if !items.is_empty() { + let response = client + .import_provider_profiles(ImportProviderProfilesRequest { + profiles: items, + workspace: workspace.to_string(), + }) + .await + .into_diagnostic()? + .into_inner(); + diagnostics.extend(response.diagnostics); + if response.imported { + println!( + "Imported {} provider profile{}.", + response.profiles.len(), + if response.profiles.len() == 1 { + "" + } else { + "s" + } + ); + return Ok(()); + } + } + + print_profile_diagnostics(&diagnostics); + Err(miette!("provider profile import failed")) +} + +pub async fn provider_profile_update( + server: &str, + id: &str, + file: &Path, + workspace: &str, + tls: &TlsOptions, +) -> Result<()> { + let (mut items, mut diagnostics) = load_profile_import_items(Some(file), None)?; + if items.is_empty() && diagnostics.is_empty() { + return Err(miette!("no provider profile files found")); + } + if profile_diagnostics_have_errors(&diagnostics) { + print_profile_diagnostics(&diagnostics); + return Err(miette!("provider profile update failed")); + } + + let mut client = grpc_client(server, tls).await?; + if let Some(item) = items.pop() { + let expected_resource_version = item + .profile + .as_ref() + .map_or(0, |profile| profile.resource_version); + let response = client + .update_provider_profiles(UpdateProviderProfilesRequest { + profile: Some(item), + expected_resource_version, + id: id.to_string(), + workspace: workspace.to_string(), + }) + .await + .into_diagnostic()? + .into_inner(); + diagnostics.extend(response.diagnostics); + if response.updated { + println!("Updated provider profile."); + return Ok(()); + } + } + + print_profile_diagnostics(&diagnostics); + Err(miette!("provider profile update failed")) +} + +pub async fn provider_profile_lint( + server: &str, + file: Option<&Path>, + from: Option<&Path>, + workspace: &str, + tls: &TlsOptions, +) -> Result<()> { + let (items, mut diagnostics) = load_profile_import_items(file, from)?; + if items.is_empty() && diagnostics.is_empty() { + return Err(miette!("no provider profile files found")); + } + + if !items.is_empty() { + let mut client = grpc_client(server, tls).await?; + let response = client + .lint_provider_profiles(LintProviderProfilesRequest { + profiles: items, + workspace: workspace.to_string(), + }) + .await + .into_diagnostic()? + .into_inner(); + diagnostics.extend(response.diagnostics); + } + + if profile_diagnostics_have_errors(&diagnostics) { + print_profile_diagnostics(&diagnostics); + return Err(miette!("provider profile lint failed")); + } + + println!("Provider profile lint passed."); + Ok(()) +} + +pub async fn provider_profile_delete( + server: &str, + ids: &[String], + workspace: &str, + tls: &TlsOptions, +) -> Result<()> { + let mut client = grpc_client(server, tls).await?; + let mut failures = Vec::new(); + for id in ids { + let response = match client + .delete_provider_profile(DeleteProviderProfileRequest { + id: id.clone(), + workspace: workspace.to_string(), + }) + .await + { + Ok(response) => response.into_inner(), + Err(status) => { + eprintln!( + "{} Failed to delete provider profile {id}: {status}", + "!".red().bold() + ); + failures.push(id.clone()); + continue; + } + }; + if response.deleted { + println!("{} Deleted provider profile {id}", "✓".green().bold()); + } else { + println!("{} Provider profile {id} not found", "!".yellow()); + } + } + aggregate_delete_failures("provider profile", &failures) +} + +pub async fn provider_refresh_status( + server: &str, + name: &str, + credential_key: Option<&str>, + workspace: &str, + tls: &TlsOptions, +) -> Result<()> { + let mut client = grpc_client(server, tls).await?; + let response = client + .get_provider_refresh_status(GetProviderRefreshStatusRequest { + provider: name.to_string(), + credential_key: credential_key.unwrap_or_default().to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector(workspace)), + }) + .await + .into_diagnostic()? + .into_inner(); + + if response.credentials.is_empty() { + if let Some(credential_key) = credential_key { + println!( + "No refresh configuration found for provider '{name}' credential '{credential_key}'." + ); + } else { + println!("No refresh configurations found for provider '{name}'."); + } + return Ok(()); + } + + println!("{}", refresh_status_header()); + for status in response.credentials { + print_refresh_status_row(&status); + } + Ok(()) +} + +fn refresh_status_header() -> String { + format!( + "{:<24} {:<28} {:<28} {:<24} {:<18} {:<20} {:<20} {:<20} {:<44} {}", + "PROVIDER".bold(), + "CREDENTIAL_KEY".bold(), + "STRATEGY".bold(), + "STATUS".bold(), + "RECOVERY".bold(), + "EXPIRES_AT".bold(), + "NEXT_REFRESH".bold(), + "LAST_REFRESH".bold(), + "FAILURE_CODE".bold(), + "LAST_ERROR".bold(), + ) +} + +pub struct ProviderRefreshConfigInput<'a> { + pub name: &'a str, + pub credential_key: &'a str, + pub strategy: &'a str, + pub material: &'a [String], + pub secret_material_env: &'a [String], + pub secret_material_keys: &'a [String], + pub credential_expires_at_ms: Option, +} + +pub async fn provider_refresh_config( + server: &str, + input: ProviderRefreshConfigInput<'_>, + workspace: &str, + tls: &TlsOptions, +) -> Result<()> { + let strategy = provider_refresh_strategy(input.strategy)?; + let mut material = parse_key_value_pairs(input.material, "--material")?; + let mut secret_material_keys = input.secret_material_keys.to_vec(); + // Env-resolved secrets are auto-marked secret; duplicate keys are an + // error rather than a precedence order. + for (key, value) in parse_secret_material_env_pairs(input.secret_material_env)? { + if material.contains_key(&key) { + return Err(miette!( + "duplicate material key '{key}': supplied via both --material and --secret-material-env" + )); + } + if !secret_material_keys.contains(&key) { + secret_material_keys.push(key.clone()); + } + material.insert(key, value); + } + let mut client = grpc_client(server, tls).await?; + let status = client + .configure_provider_refresh(ConfigureProviderRefreshRequest { + provider: input.name.to_string(), + credential_key: input.credential_key.to_string(), + strategy: strategy as i32, + material, + secret_material_keys, + expires_at_ms: input.credential_expires_at_ms, + workspace_scope: Some(openshell_core::proto::workspace_selector(workspace)), + }) + .await + .into_diagnostic()? + .into_inner() + .status + .ok_or_else(|| miette!("provider refresh status missing from response"))?; + + println!( + "{} Configured refresh for {} {}", + "✓".green().bold(), + status.provider_name, + status.credential_key + ); + Ok(()) +} + +pub async fn provider_rotate( + server: &str, + name: &str, + credential_key: &str, + workspace: &str, + tls: &TlsOptions, +) -> Result<()> { + let mut client = grpc_client(server, tls).await?; + let status = client + .rotate_provider_credential(RotateProviderCredentialRequest { + provider: name.to_string(), + credential_key: credential_key.to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector(workspace)), + }) + .await + .into_diagnostic()? + .into_inner() + .status + .ok_or_else(|| miette!("provider refresh status missing from response"))?; + + if status.last_error.is_empty() { + println!( + "{} Rotation requested for {} {} ({})", + "✓".green().bold(), + status.provider_name, + status.credential_key, + status.status + ); + } else { + println!( + "Rotation request recorded for {} {} ({}): {}", + status.provider_name, status.credential_key, status.status, status.last_error + ); + } + Ok(()) +} + +pub async fn provider_refresh_delete( + server: &str, + name: &str, + credential_key: &str, + workspace: &str, + tls: &TlsOptions, +) -> Result<()> { + let mut client = grpc_client(server, tls).await?; + let response = client + .delete_provider_refresh(DeleteProviderRefreshRequest { + provider: name.to_string(), + credential_key: credential_key.to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector(workspace)), + }) + .await + .into_diagnostic()? + .into_inner(); + + if response.deleted { + println!( + "{} Deleted refresh config for {} {}", + "✓".green().bold(), + name, + credential_key + ); + } else { + println!("No refresh config found for provider '{name}' credential '{credential_key}'."); + } + Ok(()) +} + +fn provider_refresh_strategy(strategy: &str) -> Result { + match strategy { + "oauth2_refresh_token" => Ok(ProviderCredentialRefreshStrategy::Oauth2RefreshToken), + "oauth2_client_credentials" => { + Ok(ProviderCredentialRefreshStrategy::Oauth2ClientCredentials) + } + "google_service_account_jwt" => { + Ok(ProviderCredentialRefreshStrategy::GoogleServiceAccountJwt) + } + "aws_sts_assume_role" => Ok(ProviderCredentialRefreshStrategy::AwsStsAssumeRole), + _ => Err(miette!("unsupported provider refresh strategy: {strategy}")), + } +} + +fn print_refresh_status_row(status: &ProviderCredentialRefreshStatus) { + println!("{}", refresh_status_row(status)); +} + +fn refresh_status_row(status: &ProviderCredentialRefreshStatus) -> String { + let strategy = ProviderCredentialRefreshStrategy::try_from(status.strategy) + .unwrap_or(ProviderCredentialRefreshStrategy::Unspecified); + let recovery_action = ProviderCredentialRefreshRecoveryAction::try_from(status.recovery_action) + .unwrap_or(ProviderCredentialRefreshRecoveryAction::Unspecified); + format!( + "{:<24} {:<28} {:<28} {:<24} {:<18} {:<20} {:<20} {:<20} {:<44} {}", + status.provider_name, + status.credential_key, + provider_refresh_strategy_name(strategy), + status.status, + provider_refresh_recovery_action_name(recovery_action), + format_optional_epoch_ms(status.expires_at_ms), + format_refresh_next_at_ms(status.next_refresh_at_ms), + format_optional_epoch_ms(status.last_refresh_at_ms), + status.failure_code, + truncate_status_field(&status.last_error, 72), + ) +} + +fn format_refresh_next_at_ms(next_refresh_at_ms: i64) -> String { + if next_refresh_at_ms == i64::MAX { + "-".to_string() + } else { + format_optional_epoch_ms(next_refresh_at_ms) + } +} + +fn provider_refresh_recovery_action_name( + action: ProviderCredentialRefreshRecoveryAction, +) -> &'static str { + match action { + ProviderCredentialRefreshRecoveryAction::Retry => "retry", + ProviderCredentialRefreshRecoveryAction::Reauthorize => "reauthorize", + ProviderCredentialRefreshRecoveryAction::FixConfiguration => "fix_configuration", + ProviderCredentialRefreshRecoveryAction::Investigate => "investigate", + ProviderCredentialRefreshRecoveryAction::Unspecified => "-", + } +} + +fn provider_refresh_strategy_name(strategy: ProviderCredentialRefreshStrategy) -> &'static str { + match strategy { + ProviderCredentialRefreshStrategy::Static => "static", + ProviderCredentialRefreshStrategy::External => "external", + ProviderCredentialRefreshStrategy::Oauth2RefreshToken => "oauth2_refresh_token", + ProviderCredentialRefreshStrategy::Oauth2ClientCredentials => "oauth2_client_credentials", + ProviderCredentialRefreshStrategy::GoogleServiceAccountJwt => "google_service_account_jwt", + ProviderCredentialRefreshStrategy::AwsStsAssumeRole => "aws_sts_assume_role", + ProviderCredentialRefreshStrategy::Unspecified => "unspecified", + } +} + +fn load_profile_import_items( + file: Option<&Path>, + from: Option<&Path>, +) -> Result<( + Vec, + Vec, +)> { + let paths = profile_source_paths(file, from)?; + let mut items = Vec::new(); + let mut diagnostics = Vec::new(); + for path in paths { + match load_profile_import_item(&path) { + Ok(item) => items.push(item), + Err(diagnostic) => diagnostics.push(diagnostic), + } + } + Ok((items, diagnostics)) +} + +fn profile_source_paths(file: Option<&Path>, from: Option<&Path>) -> Result> { + if let Some(file) = file { + return Ok(vec![file.to_path_buf()]); + } + let Some(from) = from else { + return Ok(Vec::new()); + }; + let mut paths = Vec::new(); + for entry in std::fs::read_dir(from) + .into_diagnostic() + .wrap_err_with(|| format!("failed to read profile directory {}", from.display()))? + { + let entry = entry.into_diagnostic()?; + let path = entry.path(); + if path.is_file() && profile_extension_supported(&path) { + paths.push(path); + } + } + paths.sort(); + Ok(paths) +} + +fn profile_extension_supported(path: &Path) -> bool { + matches!( + path.extension().and_then(|ext| ext.to_str()), + Some("yaml" | "yml" | "json") + ) +} + +fn load_profile_import_item( + path: &Path, +) -> Result { + let source = path.display().to_string(); + let input = std::fs::read_to_string(path).map_err(|err| { + profile_file_diagnostic( + &source, + format!("failed to read provider profile file: {err}"), + ) + })?; + let profile = match path.extension().and_then(|ext| ext.to_str()) { + Some("yaml" | "yml") => parse_profile_yaml(&input), + Some("json") => parse_profile_json(&input), + _ => { + return Err(profile_file_diagnostic( + &source, + "unsupported provider profile file format".to_string(), + )); + } + } + .map_err(|err| profile_file_diagnostic(&source, err.to_string()))?; + + let pre_lower = profile.validate_before_lowering(&source); + if let Some(diag) = pre_lower.into_iter().find(|d| d.severity == "error") { + return Err(ProviderProfileDiagnostic { + source: diag.source, + profile_id: diag.profile_id, + field: diag.field, + message: diag.message, + severity: diag.severity, + }); + } + + Ok(ProviderProfileImportItem { + profile: Some(profile.to_proto()), + source, + }) +} + +fn profile_file_diagnostic(source: &str, message: String) -> ProviderProfileDiagnostic { + ProviderProfileDiagnostic { + source: source.to_string(), + profile_id: String::new(), + field: "file".to_string(), + message, + severity: "error".to_string(), + } +} + +fn print_profile_diagnostics(diagnostics: &[ProviderProfileDiagnostic]) { + if diagnostics.is_empty() { + return; + } + eprintln!("{}", "Provider profile diagnostics:".red().bold()); + for diagnostic in diagnostics { + let source = if diagnostic.source.is_empty() { + "" + } else { + &diagnostic.source + }; + let profile = if diagnostic.profile_id.is_empty() { + "-".to_string() + } else { + diagnostic.profile_id.clone() + }; + eprintln!( + " {} {} profile={} field={} {}", + diagnostic.severity.as_str().red(), + source, + profile, + diagnostic.field, + diagnostic.message + ); + } +} + +fn profile_diagnostics_have_errors(diagnostics: &[ProviderProfileDiagnostic]) -> bool { + diagnostics + .iter() + .any(|diagnostic| diagnostic.severity == "error") +} + +fn display_provider_category(category: i32) -> &'static str { + match ProviderProfileCategory::try_from(category).unwrap_or(ProviderProfileCategory::Other) { + ProviderProfileCategory::Inference => "INFERENCE", + ProviderProfileCategory::Agent => "AGENT", + ProviderProfileCategory::SourceControl => "SOURCE CONTROL", + ProviderProfileCategory::Messaging => "MESSAGING", + ProviderProfileCategory::Data => "DATA", + ProviderProfileCategory::Knowledge => "KNOWLEDGE", + ProviderProfileCategory::Other | ProviderProfileCategory::Unspecified => "OTHER", + } +} + +const PROVIDER_PROFILE_ID_MAX_WIDTH: usize = 32; +const PROVIDER_PROFILE_DISPLAY_MAX_WIDTH: usize = 40; +const PROVIDER_PROFILE_SOURCE_MAX_WIDTH: usize = 24; + +fn provider_profile_id_width(profiles: &[ProviderProfile]) -> usize { + profiles + .iter() + .map(|profile| { + profile + .id + .chars() + .count() + .min(PROVIDER_PROFILE_ID_MAX_WIDTH) + }) + .max() + .unwrap_or(2) + .max(2) +} + +fn provider_profile_display_width(profiles: &[ProviderProfile]) -> usize { + profiles + .iter() + .map(|profile| { + profile + .display_name + .chars() + .count() + .min(PROVIDER_PROFILE_DISPLAY_MAX_WIDTH) + }) + .max() + .unwrap_or(4) + .max(4) +} + +fn provider_profile_scope_width(profiles: &[ProviderProfile]) -> usize { + profiles + .iter() + .map(|profile| profile.scope.chars().count()) + .max() + .unwrap_or(5) + .max(5) +} + +fn provider_profile_source_width(profiles: &[ProviderProfile]) -> usize { + profiles + .iter() + .map(|profile| { + profile + .source + .chars() + .count() + .min(PROVIDER_PROFILE_SOURCE_MAX_WIDTH) + }) + .max() + .unwrap_or(6) + .max(6) +} + +fn print_provider_type_header( + id_width: usize, + scope_width: usize, + source_width: usize, + display_width: usize, +) { + let endpoints = "ENDPOINTS"; + println!( + " {: { + pub server: &'a str, + pub name: &'a str, + pub from_existing: bool, + pub from_oidc_token: bool, + pub credentials: &'a [String], + pub config: &'a [String], + pub credential_expires_at: &'a [String], + pub workspace: &'a str, + pub tls: &'a TlsOptions, +} + +pub async fn provider_update(options: ProviderUpdateOptions<'_>) -> Result<()> { + let ProviderUpdateOptions { + server, + name, + from_existing, + from_oidc_token, + credentials, + config, + credential_expires_at, + workspace, + tls, + } = options; + + if from_existing && !credentials.is_empty() { + return Err(miette::miette!( + "--from-existing cannot be combined with --credential" + )); + } + if from_existing && from_oidc_token { + return Err(miette::miette!( + "--from-existing cannot be combined with --from-oidc-token" + )); + } + + let mut client = grpc_client(server, tls).await?; + + // Look up the stored provider so the update can carry its type and profile + // workspace. Policy interceptors evaluate the request before the gateway + // merges it with stored state, so an update that omits them cannot be + // authorized against the profile that owns the provider. + // + // The read is best-effort. A caller holding `provider:write` without + // `provider:read` must still be able to rotate credentials, so a denied + // read keeps the previous behavior of sending empty metadata rather than + // failing the update. `--from-existing` and `--from-oidc-token` need the + // stored type, so they surface the error instead. + let existing = match client + .get_provider(GetProviderRequest { + name: name.to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector(workspace)), + }) + .await + { + Ok(response) => response.into_inner().provider, + Err(status) + if status.code() == Code::PermissionDenied && !from_existing && !from_oidc_token => + { + None + } + Err(status) => return Err(status).into_diagnostic(), + }; + + if existing.is_none() && (from_existing || from_oidc_token) { + return Err(miette::miette!("provider '{name}' not found")); + } + + let oidc_profile = if from_oidc_token { + let existing = existing.as_ref().expect("checked above"); + Some( + fetch_provider_profile(&mut client, &existing.r#type, &existing.profile_workspace) + .await?, + ) + } else { + None + }; + + let (mut credential_map, oidc_credential_expires_at_ms) = if from_oidc_token { + provider_credential_from_oidc_token(credentials, oidc_profile.as_ref(), tls).await? + } else { + (parse_credential_pairs(credentials)?, HashMap::new()) + }; + let mut config_map = parse_key_value_pairs(config, "--config")?; + let mut credential_expires_at_ms = parse_credential_expiry_pairs(credential_expires_at)?; + credential_expires_at_ms.extend(oidc_credential_expires_at_ms); + + if from_existing { + let stored = existing.as_ref().expect("checked above"); + let provider_type = stored.r#type.clone(); + let discovered = + discover_existing_provider_data(&mut client, &provider_type, &stored.profile_workspace) + .await?; + let Some(discovered) = discovered else { + return Err(miette::miette!( + "no existing local credentials/config found for provider type '{provider_type}'" + )); + }; + + for (key, value) in discovered.credentials { + credential_map.entry(key).or_insert(value); + } + for (key, value) in discovered.config { + config_map.entry(key).or_insert(value); + } + } + + let response = client + .update_provider(UpdateProviderRequest { + provider: Some(Provider { + metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { + id: String::new(), + name: name.to_string(), + created_at_ms: 0, + labels: HashMap::new(), + resource_version: 0, + annotations: HashMap::new(), + workspace: workspace.to_string(), + deletion_timestamp_ms: 0, + }), + r#type: existing + .as_ref() + .map(|provider| provider.r#type.clone()) + .unwrap_or_default(), + credentials: credential_map, + config: config_map, + credential_expires_at_ms: HashMap::new(), + profile_workspace: existing + .as_ref() + .map(|provider| provider.profile_workspace.clone()) + .unwrap_or_default(), + credential_handles: HashMap::new(), + }), + credential_expires_at_ms, + workspace_scope: Some(openshell_core::proto::workspace_selector(workspace)), + }) + .await + .into_diagnostic()?; + + let provider = response + .into_inner() + .provider + .ok_or_else(|| miette::miette!("provider missing from response"))?; + + println!( + "{} Updated provider {}", + "✓".green().bold(), + provider.object_name() + ); + Ok(()) +} + +pub async fn provider_delete( + server: &str, + names: &[String], + workspace: &str, + tls: &TlsOptions, +) -> Result<()> { + let mut client = grpc_client(server, tls).await?; + let mut failures = Vec::new(); + for name in names { + let response = match client + .delete_provider(DeleteProviderRequest { + name: name.clone(), + workspace_scope: Some(openshell_core::proto::workspace_selector(workspace)), + }) + .await + { + Ok(response) => response, + Err(status) => { + eprintln!( + "{} Failed to delete provider {name}: {status}", + "!".red().bold() + ); + failures.push(name.clone()); + continue; + } + }; + if response.into_inner().deleted { + println!("{} Deleted provider {name}", "✓".green().bold()); + } else { + println!("{} Provider {name} not found", "!".yellow()); + } + } + aggregate_delete_failures("provider", &failures) +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::TEST_ENV_LOCK; + use crate::test_utils::EnvVarGuard; + use std::fs; + use std::io::Write; + + use openshell_core::proto::{ + CredentialHandle, Provider, ProviderCredentialRefresh, + ProviderCredentialRefreshRecoveryAction, ProviderCredentialRefreshStatus, + ProviderCredentialRefreshStrategy, ProviderCredentialTokenGrant, ProviderProfile, + ProviderProfileCredential, datamodel::v1::ObjectMeta, + }; + + #[test] + fn attached_provider_json_is_sorted_and_secret_safe() { + let provider = Provider { + metadata: Some(ObjectMeta { + name: "github".to_string(), + ..Default::default() + }), + r#type: "github".to_string(), + credentials: HashMap::from([ + ("Z_TOKEN".to_string(), "inline-secret".to_string()), + ("SHARED".to_string(), "shared-secret".to_string()), + ]), + config: HashMap::from([ + ("Z_URL".to_string(), "https://secret.example".to_string()), + ("A_MODE".to_string(), "sensitive-config".to_string()), + ]), + credential_expires_at_ms: HashMap::from([("Z_TOKEN".to_string(), 123)]), + profile_workspace: "internal".to_string(), + credential_handles: HashMap::from([ + ( + "A_HANDLE".to_string(), + CredentialHandle { + driver: "vault".to_string(), + handle: "opaque-secret-handle".to_string(), + metadata: HashMap::from([( + "internal".to_string(), + "metadata-value".to_string(), + )]), + }, + ), + ("SHARED".to_string(), CredentialHandle::default()), + ]), + }; + + let value = attached_provider_to_json(&provider); + assert_eq!( + value, + serde_json::json!({ + "name": "github", + "type": "github", + "credential_keys": ["A_HANDLE", "SHARED", "Z_TOKEN"], + "config_keys": ["A_MODE", "Z_URL"], + }) + ); + let serialized = value.to_string(); + for secret in [ + "inline-secret", + "shared-secret", + "https://secret.example", + "sensitive-config", + "opaque-secret-handle", + "metadata-value", + "internal", + "123", + ] { + assert!(!serialized.contains(secret), "leaked {secret}"); + } + } + + #[test] + fn provider_attachment_table_formats_provider_counts() { + let output = format_provider_attachment_table( + &[Provider { + metadata: Some(ObjectMeta { + name: "work-custom".to_string(), + ..Default::default() + }), + r#type: "custom-api".to_string(), + credentials: [ + ("CUSTOM_API_KEY".to_string(), "REDACTED".to_string()), + ("CUSTOM_API_SECRET".to_string(), "REDACTED".to_string()), + ] + .into_iter() + .collect(), + config: std::iter::once(( + "BASE_URL".to_string(), + "https://api.custom.example".to_string(), + )) + .collect(), + credential_expires_at_ms: HashMap::new(), + profile_workspace: String::new(), + credential_handles: HashMap::new(), + }], + false, + ); + + assert!(output.contains("NAME")); + assert!(output.contains("TYPE")); + assert!(output.contains("CREDENTIAL_KEYS")); + assert!(output.contains("CONFIG_KEYS")); + assert!(output.contains("work-custom")); + assert!(output.contains("custom-api")); + assert!(output.contains('2')); + assert!(output.contains('1')); + } + + #[test] + fn refresh_status_table_includes_operational_fields() { + let header = refresh_status_header(); + assert!(header.contains("NEXT_REFRESH")); + assert!(header.contains("LAST_REFRESH")); + assert!(header.contains("RECOVERY")); + assert!(header.contains("FAILURE_CODE")); + assert!(header.contains("LAST_ERROR")); + + let row = refresh_status_row(&ProviderCredentialRefreshStatus { + provider_name: "my-graph".to_string(), + provider_id: "provider-id".to_string(), + credential_key: "MS_GRAPH_ACCESS_TOKEN".to_string(), + strategy: ProviderCredentialRefreshStrategy::Oauth2ClientCredentials as i32, + status: "error".to_string(), + expires_at_ms: 1_767_225_600_000, + next_refresh_at_ms: i64::MAX, + last_refresh_at_ms: 1_767_225_000_000, + last_error: "token endpoint returned a very long error message that should be truncated for table readability" + .to_string(), + recovery_action: ProviderCredentialRefreshRecoveryAction::Reauthorize as i32, + failure_code: "oauth_rotated_refresh_token_handle_missing".to_string(), + provider_error_subtype: "invalid_rapt".to_string(), + last_error_at_ms: 1_767_225_000_000, + }); + + assert!(row.contains("my-graph")); + assert!(row.contains("MS_GRAPH_ACCESS_TOKEN")); + assert!(row.contains("oauth2_client_credentials")); + assert!(row.contains("error")); + assert!(row.contains("reauthorize")); + assert!(row.contains("oauth_rotated_refresh_token_handle_missing")); + assert!(row.contains("2026-01-01 00:00:00")); + assert!(!row.contains("292278994")); + assert!(row.contains("...")); + } + + #[test] + fn empty_provider_credentials_require_all_required_credentials_to_be_runtime_resolvable() { + let refresh_token_profile = ProviderProfile { + credentials: vec![ProviderProfileCredential { + name: "MS_GRAPH_ACCESS_TOKEN".to_string(), + required: true, + refresh: Some(ProviderCredentialRefresh { + strategy: ProviderCredentialRefreshStrategy::Oauth2RefreshToken as i32, + ..Default::default() + }), + ..Default::default() + }], + ..Default::default() + }; + assert!(provider_profile_allows_empty_credentials( + &refresh_token_profile + )); + + let token_grant_profile = ProviderProfile { + credentials: vec![ProviderProfileCredential { + name: "ACCESS_TOKEN".to_string(), + required: true, + token_grant: Some(ProviderCredentialTokenGrant { + token_endpoint: "https://auth.example.com/token".to_string(), + ..Default::default() + }), + ..Default::default() + }], + ..Default::default() + }; + assert!(provider_profile_allows_empty_credentials( + &token_grant_profile + )); + + let mixed_static_profile = ProviderProfile { + credentials: vec![ + ProviderProfileCredential { + name: "ACCESS_TOKEN".to_string(), + required: true, + refresh: Some(ProviderCredentialRefresh { + strategy: ProviderCredentialRefreshStrategy::Oauth2ClientCredentials as i32, + ..Default::default() + }), + ..Default::default() + }, + ProviderProfileCredential { + name: "STATIC_API_KEY".to_string(), + required: true, + refresh: None, + ..Default::default() + }, + ], + ..Default::default() + }; + assert!(!provider_profile_allows_empty_credentials( + &mixed_static_profile + )); + + let optional_refresh_profile = ProviderProfile { + credentials: vec![ProviderProfileCredential { + name: "OPTIONAL_TOKEN".to_string(), + required: false, + refresh: Some(ProviderCredentialRefresh { + strategy: ProviderCredentialRefreshStrategy::GoogleServiceAccountJwt as i32, + ..Default::default() + }), + ..Default::default() + }], + ..Default::default() + }; + assert!(provider_profile_allows_empty_credentials( + &optional_refresh_profile + )); + } + + #[test] + fn inferred_provider_type_returns_type_for_known_command() { + let result = inferred_provider_type(&["claude".to_string(), "--help".to_string()]); + assert_eq!(result, Some("claude-code".to_string())); + } + + #[test] + fn inferred_provider_type_returns_none_for_unknown_command() { + let result = inferred_provider_type(&["bash".to_string()]); + assert_eq!(result, None); + } + + #[test] + fn inferred_provider_type_returns_none_for_empty_command() { + let result = inferred_provider_type(&[]); + assert_eq!(result, None); + } + + #[test] + fn inferred_provider_type_normalizes_aliases() { + // Retired legacy types are not inferred, even when a custom profile + // with the same ID could be imported and attached explicitly. + let result = inferred_provider_type(&["glab".to_string()]); + assert_eq!(result, None); + + // `gh` should resolve to `github` + let result = inferred_provider_type(&["gh".to_string()]); + assert_eq!(result, Some("github".to_string())); + } + + #[test] + fn inferred_provider_type_handles_full_path() { + let result = inferred_provider_type(&["/usr/local/bin/claude".to_string()]); + assert_eq!(result, Some("claude-code".to_string())); + } + + #[test] + fn read_gcloud_adc_missing_file_errors() { + let _lock = TEST_ENV_LOCK + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner); + let _guard = EnvVarGuard::set( + "GOOGLE_APPLICATION_CREDENTIALS", + "/nonexistent/path/to/adc.json", + ); + let err = read_gcloud_adc().expect_err("missing file should error"); + assert!( + err.to_string().contains("failed to read gcloud ADC file"), + "unexpected error: {err}" + ); + } + + #[test] + fn read_gcloud_adc_wrong_type_errors() { + let _lock = TEST_ENV_LOCK + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner); + let tmp = tempfile::NamedTempFile::new().expect("tempfile"); + let json = serde_json::json!({ + "type": "service_account", + "project_id": "my-project", + "private_key_id": "key123" + }); + Write::write_all(&mut tmp.as_file(), json.to_string().as_bytes()).expect("write tempfile"); + let _guard = EnvVarGuard::set( + "GOOGLE_APPLICATION_CREDENTIALS", + tmp.path().to_str().expect("tempfile path"), + ); + let err = read_gcloud_adc().expect_err("wrong type should error"); + // The service_account type gets a targeted message directing the user + // to the real Vertex service-account credential flow instead of the + // generic authorized_user hint. + assert!( + err.to_string() + .contains("GOOGLE_VERTEX_AI_SERVICE_ACCOUNT_TOKEN"), + "error should mention the service-account token key, got: {err}" + ); + } + + #[test] + fn read_gcloud_adc_parses_user_creds() { + let _lock = TEST_ENV_LOCK + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner); + let tmp = tempfile::NamedTempFile::new().expect("tempfile"); + let json = serde_json::json!({ + "type": "authorized_user", + "client_id": "test-client-id.apps.googleusercontent.com", + "client_secret": "test-client-secret", + "refresh_token": "test-refresh-token" + }); + Write::write_all(&mut tmp.as_file(), json.to_string().as_bytes()).expect("write tempfile"); + let _guard = EnvVarGuard::set( + "GOOGLE_APPLICATION_CREDENTIALS", + tmp.path().to_str().expect("tempfile path"), + ); + let (client_id, client_secret, refresh_token) = + read_gcloud_adc().expect("valid ADC should parse"); + assert_eq!(client_id, "test-client-id.apps.googleusercontent.com"); + assert_eq!(client_secret, "test-client-secret"); + assert_eq!(refresh_token, "test-refresh-token"); + } + + #[test] + fn read_gcloud_adc_uses_cloudsdk_config_fallback() { + let _lock = TEST_ENV_LOCK + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner); + let dir = tempfile::tempdir().expect("tempdir"); + let adc_path = dir.path().join("application_default_credentials.json"); + let json = serde_json::json!({ + "type": "authorized_user", + "client_id": "cloudsdk-client-id.apps.googleusercontent.com", + "client_secret": "cloudsdk-client-secret", + "refresh_token": "cloudsdk-refresh-token" + }); + fs::write(&adc_path, json.to_string()).expect("write adc file"); + let _adc_guard = EnvVarGuard::unset("GOOGLE_APPLICATION_CREDENTIALS"); + let _cloudsdk_guard = + EnvVarGuard::set("CLOUDSDK_CONFIG", dir.path().to_str().expect("config path")); + + let (client_id, client_secret, refresh_token) = + read_gcloud_adc().expect("valid CLOUDSDK_CONFIG ADC should parse"); + assert_eq!(client_id, "cloudsdk-client-id.apps.googleusercontent.com"); + assert_eq!(client_secret, "cloudsdk-client-secret"); + assert_eq!(refresh_token, "cloudsdk-refresh-token"); + } + + #[test] + fn read_gcloud_adc_malformed_json_errors() { + let _lock = TEST_ENV_LOCK + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner); + let tmp = tempfile::NamedTempFile::new().expect("tempfile"); + Write::write_all(&mut tmp.as_file(), b"not valid json at all {{{{") + .expect("write tempfile"); + let _guard = EnvVarGuard::set( + "GOOGLE_APPLICATION_CREDENTIALS", + tmp.path().to_str().expect("tempfile path"), + ); + let result = read_gcloud_adc(); + assert!( + result.is_err(), + "malformed JSON should produce an error, got: {result:?}" + ); + let err = result.unwrap_err(); + let msg = format!("{err}"); + assert!( + msg.contains("parse") + || msg.contains("JSON") + || msg.contains("json") + || msg.contains("invalid") + || msg.contains("failed"), + "error message should mention parse/JSON failure, got: {msg}" + ); + } + + #[test] + fn empty_provider_credentials_allow_oauth2_refresh_token() { + use openshell_core::proto::{ + ProviderCredentialRefresh, ProviderCredentialRefreshStrategy, ProviderProfile, + ProviderProfileCredential, + }; + + let strategy = ProviderCredentialRefreshStrategy::Oauth2RefreshToken as i32; + let profile = ProviderProfile { + credentials: vec![ProviderProfileCredential { + required: true, + refresh: Some(ProviderCredentialRefresh { + strategy, + ..Default::default() + }), + ..Default::default() + }], + ..Default::default() + }; + assert!( + provider_profile_allows_empty_credentials(&profile), + "Oauth2RefreshToken should be allowed for refresh bootstrap" + ); + } + + #[test] + fn provider_to_json_includes_core_fields() { + let metadata = ObjectMeta { + id: "prov-123".to_string(), + name: "test-provider".to_string(), + ..Default::default() + }; + + let provider = Provider { + metadata: Some(metadata), + r#type: "anthropic".to_string(), + credentials: HashMap::new(), + config: HashMap::new(), + credential_expires_at_ms: HashMap::new(), + profile_workspace: String::new(), + credential_handles: HashMap::new(), + }; + + let json = provider_to_json(&provider); + + assert_eq!(json["id"], "prov-123"); + assert_eq!(json["name"], "test-provider"); + assert_eq!(json["workspace"], ""); + assert_eq!(json["type"], "anthropic"); + } + + #[test] + fn provider_to_json_exposes_credential_keys_not_values() { + let mut credentials = HashMap::new(); + credentials.insert("ANTHROPIC_API_KEY".to_string(), "secret-value".to_string()); + credentials.insert("OTHER_KEY".to_string(), "other-secret".to_string()); + + let provider = Provider { + metadata: Some(ObjectMeta::default()), + r#type: "anthropic".to_string(), + credentials, + config: HashMap::new(), + credential_expires_at_ms: HashMap::new(), + profile_workspace: String::new(), + credential_handles: HashMap::new(), + }; + + let json = provider_to_json(&provider); + let json_str = json.to_string(); + + // Assert credential keys are present + let keys = json["credential_keys"].as_array().unwrap(); + assert_eq!(keys.len(), 2); + assert!(keys.iter().any(|k| k.as_str() == Some("ANTHROPIC_API_KEY"))); + assert!(keys.iter().any(|k| k.as_str() == Some("OTHER_KEY"))); + + // Assert credential values are NOT in the output (SECURITY) + assert!( + !json_str.contains("secret-value"), + "credential values must not be exposed" + ); + assert!( + !json_str.contains("other-secret"), + "credential values must not be exposed" + ); + } + + #[test] + fn provider_to_json_exposes_config_keys_not_values() { + let mut config = HashMap::new(); + config.insert("region".to_string(), "us-west".to_string()); + config.insert( + "endpoint".to_string(), + "https://api.example.com".to_string(), + ); + + let provider = Provider { + metadata: Some(ObjectMeta::default()), + r#type: "custom".to_string(), + credentials: HashMap::new(), + config, + credential_expires_at_ms: HashMap::new(), + profile_workspace: String::new(), + credential_handles: HashMap::new(), + }; + + let json = provider_to_json(&provider); + let json_str = json.to_string(); + + // Assert config keys are present + let keys = json["config_keys"].as_array().unwrap(); + assert_eq!(keys.len(), 2); + assert!(keys.iter().any(|k| k.as_str() == Some("region"))); + assert!(keys.iter().any(|k| k.as_str() == Some("endpoint"))); + + // Assert config values are NOT in the output (SECURITY) + assert!( + !json_str.contains("us-west"), + "config values must not be exposed" + ); + assert!( + !json_str.contains("https://api.example.com"), + "config values must not be exposed" + ); + } + + #[test] + fn provider_to_json_omits_empty_config() { + let provider = Provider { + metadata: Some(ObjectMeta::default()), + r#type: "anthropic".to_string(), + credentials: HashMap::new(), + config: HashMap::new(), // Empty config + credential_expires_at_ms: HashMap::new(), + profile_workspace: String::new(), + credential_handles: HashMap::new(), + }; + + let json = provider_to_json(&provider); + + assert!( + json.get("config_keys").is_none(), + "empty config_keys should be omitted" + ); + } + + #[test] + fn provider_to_json_includes_metadata_fields_when_present() { + let mut labels = HashMap::new(); + labels.insert("env".to_string(), "prod".to_string()); + + let metadata = ObjectMeta { + id: "prov-123".to_string(), + name: "test-provider".to_string(), + resource_version: 42, + created_at_ms: 1_234_567_890_000, + labels, + annotations: HashMap::new(), + workspace: String::new(), + deletion_timestamp_ms: 0, + }; + + let provider = Provider { + metadata: Some(metadata), + r#type: "anthropic".to_string(), + credentials: HashMap::new(), + config: HashMap::new(), + credential_expires_at_ms: HashMap::new(), + profile_workspace: String::new(), + credential_handles: HashMap::new(), + }; + + let json = provider_to_json(&provider); + + assert_eq!(json["resource_version"], 42); + assert_eq!(json["created_at"], "2009-02-13 23:31:30"); + assert_eq!(json["labels"]["env"], "prod"); + } + + #[test] + fn provider_to_json_omits_zero_metadata_fields() { + let metadata = ObjectMeta { + id: "prov-123".to_string(), + name: "test-provider".to_string(), + // resource_version and created_at_ms are 0 + // labels is empty + ..Default::default() + }; + + let provider = Provider { + metadata: Some(metadata), + r#type: "anthropic".to_string(), + credentials: HashMap::new(), + config: HashMap::new(), + credential_expires_at_ms: HashMap::new(), + profile_workspace: String::new(), + credential_handles: HashMap::new(), + }; + + let json = provider_to_json(&provider); + + assert!( + json.get("resource_version").is_none(), + "zero resource_version should be omitted" + ); + assert!( + json.get("created_at").is_none(), + "zero created_at should be omitted" + ); + assert!( + json.get("labels").is_none(), + "empty labels should be omitted" + ); + } + + #[test] + fn provider_to_json_includes_credential_expiration() { + let mut credential_expires_at_ms = HashMap::new(); + credential_expires_at_ms.insert("ACCESS_TOKEN".to_string(), 1_234_567_890); + + let provider = Provider { + metadata: Some(ObjectMeta::default()), + r#type: "oauth".to_string(), + credentials: HashMap::new(), + config: HashMap::new(), + credential_expires_at_ms, + profile_workspace: String::new(), + credential_handles: HashMap::new(), + }; + + let json = provider_to_json(&provider); + + assert_eq!( + json["credential_expires_at_ms"]["ACCESS_TOKEN"], + 1_234_567_890 + ); + } + + #[test] + fn provider_to_json_formats_created_at_as_human_readable() { + let metadata = ObjectMeta { + id: "prov-123".to_string(), + name: "test-provider".to_string(), + created_at_ms: 1_609_459_200_000, // 2021-01-01 00:00:00 + ..Default::default() + }; + + let provider = Provider { + metadata: Some(metadata), + r#type: "anthropic".to_string(), + credentials: HashMap::new(), + config: HashMap::new(), + credential_expires_at_ms: HashMap::new(), + profile_workspace: String::new(), + credential_handles: HashMap::new(), + }; + + let json = provider_to_json(&provider); + + // Should format as human-readable datetime, not raw milliseconds + assert_eq!(json["created_at"], "2021-01-01 00:00:00"); + assert!( + json.get("created_at_ms").is_none(), + "raw milliseconds field should not exist" + ); + } +} diff --git a/crates/openshell-cli/src/completers.rs b/crates/openshell-cli/src/completers.rs index 9d3b88d59..0778babbf 100644 --- a/crates/openshell-cli/src/completers.rs +++ b/crates/openshell-cli/src/completers.rs @@ -39,8 +39,9 @@ pub fn complete_sandbox_names(_prefix: &OsStr) -> Vec { limit: 200, offset: 0, label_selector: String::new(), - workspace: workspace_from_args(), - all_workspaces: false, + workspace_scope: Some(openshell_core::proto::workspace_selector( + workspace_from_args(), + )), }) .await .ok()?; @@ -64,8 +65,9 @@ pub fn complete_provider_names(_prefix: &OsStr) -> Vec { .list_providers(ListProvidersRequest { limit: 200, offset: 0, - workspace: workspace_from_args(), - all_workspaces: false, + workspace_scope: Some(openshell_core::proto::workspace_selector( + workspace_from_args(), + )), }) .await .ok()?; diff --git a/crates/openshell-cli/src/lib.rs b/crates/openshell-cli/src/lib.rs index e28b84b05..910920f42 100644 --- a/crates/openshell-cli/src/lib.rs +++ b/crates/openshell-cli/src/lib.rs @@ -11,6 +11,7 @@ pub(crate) static TEST_ENV_LOCK: std::sync::Mutex<()> = std::sync::Mutex::new(() pub(crate) mod test_utils; pub mod auth; +pub mod color; pub(crate) mod commands; pub mod completers; pub mod edge_tunnel; diff --git a/crates/openshell-cli/src/main.rs b/crates/openshell-cli/src/main.rs index ad03fb4ec..db18795b6 100644 --- a/crates/openshell-cli/src/main.rs +++ b/crates/openshell-cli/src/main.rs @@ -7,7 +7,7 @@ use clap::{CommandFactory, Parser, Subcommand, ValueEnum, ValueHint}; use clap_complete::engine::ArgValueCompleter; use clap_complete::env::CompleteEnv; use miette::Result; -use owo_colors::OwoColorize; +use openshell_cli::color::{self, ColorChoice, Colorize}; use std::collections::HashMap; use std::io::Write; use std::path::PathBuf; @@ -156,8 +156,11 @@ fn resolve_gateway_name(gateway_flag: &Option) -> Option { /// Handles Cloudflare Access and OIDC auth modes by loading the stored token /// and setting it on `TlsOptions`. For OIDC, automatically refreshes the token /// if it's near expiry. -fn apply_auth(tls: &mut TlsOptions, gateway_name: &str) { - let _ = apply_auth_with_status(tls, gateway_name); +fn apply_auth(tls: &mut TlsOptions, gateway_name: &str) -> Result<()> { + if let Some(error) = apply_auth_with_status(tls, gateway_name) { + return Err(miette::miette!(error)); + } + Ok(()) } /// Apply stored authentication and return a user-facing preparation failure, @@ -206,12 +209,17 @@ fn apply_auth_with_status(tls: &mut TlsOptions, gateway_name: &str) -> Option { tracing::warn!("OIDC token refresh failed: {e}"); - // Use the expired token anyway — server will reject it - // with a clear error prompting re-login. - tls.oidc_token = Some(bundle.access_token); - Some(format!( - "OIDC token refresh failed; run `openshell gateway login {gateway_name}`" - )) + if openshell_bootstrap::oidc_token::is_token_actually_expired(&bundle) { + Some(format!( + "OIDC token refresh failed: {e}\ncached OIDC token has expired; run `openshell gateway login {gateway_name}`" + )) + } else { + tls.oidc_token = Some(bundle.access_token); + eprintln!( + "Warning: OIDC token refresh failed; continuing with the cached token: {e}" + ); + None + } } } } else { @@ -264,7 +272,6 @@ const HELP_TEMPLATE: &str = "\ gateway: Manage gateways status: Show gateway status and information whoami: Show the authenticated user identity - inference: Manage inference configuration doctor: Diagnose gateway issues \x1b[1mADDITIONAL COMMANDS\x1b[0m @@ -368,10 +375,9 @@ const POLICY_EXAMPLES: &str = "\x1b[1mALIAS\x1b[0m const SETTINGS_EXAMPLES: &str = "\x1b[1mEXAMPLES\x1b[0m $ openshell settings get my-sandbox $ openshell settings get --global - $ openshell settings set --global --key providers_v2_enabled --value true $ openshell settings set my-sandbox --key ocsf_json_enabled --value true $ openshell settings set --global --key ocsf_json_enabled --value true - $ openshell settings delete --global --key providers_v2_enabled + $ openshell settings delete --global --key ocsf_json_enabled "; const PROVIDER_EXAMPLES: &str = "\x1b[1mEXAMPLES\x1b[0m @@ -402,12 +408,6 @@ const GATEWAY_EXAMPLES: &str = "\x1b[1mALIAS\x1b[0m $ openshell gateway remove my-gateway "; -const INFERENCE_EXAMPLES: &str = "\x1b[1mEXAMPLES\x1b[0m - $ openshell inference set --provider openai --model gpt-4 - $ openshell inference get - $ openshell inference update --model gpt-4-turbo -"; - const DOCTOR_HELP: &str = "\x1b[1mALIAS\x1b[0m dr @@ -469,6 +469,18 @@ struct Cli { #[arg(short, long, action = clap::ArgAction::Count, global = true, help_heading = "GLOBAL FLAGS")] verbose: u8, + /// When to colorize output. `auto` colorizes only when stdout is a terminal. + #[arg( + long, + global = true, + value_enum, + default_value_t = ColorChoice::Auto, + value_name = "WHEN", + env = "OPENSHELL_COLOR", + help_heading = "GLOBAL FLAGS" + )] + color: ColorChoice, + /// Print help. #[arg(short = 'h', long, action = clap::ArgAction::Help, global = true, help_heading = "GLOBAL FLAGS")] help: (), @@ -597,13 +609,6 @@ enum Commands { output: OutputFormat, }, - /// Manage inference configuration. - #[command(after_help = INFERENCE_EXAMPLES, help_template = SUBCOMMAND_HELP_TEMPLATE)] - Inference { - #[command(subcommand)] - command: Option, - }, - // =================================================================== // DIAGNOSTIC COMMANDS // =================================================================== @@ -742,7 +747,7 @@ fn normalize_completion_script(output: Vec, executable: &std::path::Path) -> Ok(script.replace(executable.to_string_lossy().as_ref(), "openshell")) } -#[derive(Clone, Debug, ValueEnum)] +#[derive(Clone, Debug, PartialEq, Eq, ValueEnum)] enum OutputFormat { Table, Yaml, @@ -811,7 +816,7 @@ impl From for openshell_cli::ssh::Editor { #[derive(Subcommand, Debug)] enum ProviderCommands { /// Create a provider config. - #[command(group = clap::ArgGroup::new("cred_source").required(true).multiple(true).args(["from_existing", "credentials", "from_gcloud_adc", "runtime_credentials", "from_oidc_token"]), help_template = LEAF_HELP_TEMPLATE, next_help_heading = "FLAGS")] + #[command(group = clap::ArgGroup::new("cred_source").required(false).multiple(true).args(["from_existing", "credentials", "from_gcloud_adc", "runtime_credentials", "from_oidc_token"]), help_template = LEAF_HELP_TEMPLATE, next_help_heading = "FLAGS")] Create { /// Provider name. #[arg(long)] @@ -1093,8 +1098,9 @@ enum ProviderProfileCommands { /// Delete a custom provider profile. #[command(help_template = LEAF_HELP_TEMPLATE, next_help_heading = "FLAGS")] Delete { - /// Provider profile id. - id: String, + /// Provider profile id(s). + #[arg(required = true, num_args = 1.., value_name = "ID")] + ids: Vec, /// Target platform-scoped profile (ignores --workspace). #[arg(long)] @@ -1234,80 +1240,6 @@ enum GatewayCommands { }, } -// ----------------------------------------------------------------------- -// Inference commands -// ----------------------------------------------------------------------- - -#[derive(Subcommand, Debug)] -enum InferenceCommands { - /// Set workspace-level inference provider and model. - #[command(help_template = LEAF_HELP_TEMPLATE, next_help_heading = "FLAGS")] - Set { - /// Provider name. - #[arg(long, add = ArgValueCompleter::new(completers::complete_provider_names))] - provider: String, - - /// Model identifier to force for generation calls. - #[arg(long)] - model: String, - - /// Configure the system inference route instead of the user-facing - /// route. System inference is used by platform functions (e.g. the - /// agent harness) and is not accessible to user code. - #[arg(long)] - system: bool, - - /// Skip endpoint verification before saving the route. - #[arg(long)] - no_verify: bool, - - /// Request timeout in seconds for inference calls (0 = default 60s). - #[arg(long, default_value_t = 0)] - timeout: u64, - }, - - /// Update workspace-level inference configuration (partial update). - #[command(help_template = LEAF_HELP_TEMPLATE, next_help_heading = "FLAGS")] - Update { - /// Provider name (unchanged if omitted). - #[arg(long, add = ArgValueCompleter::new(completers::complete_provider_names))] - provider: Option, - - /// Model identifier (unchanged if omitted). - #[arg(long)] - model: Option, - - /// Target the system inference route. - #[arg(long)] - system: bool, - - /// Skip endpoint verification before saving the route. - #[arg(long)] - no_verify: bool, - - /// Request timeout in seconds for inference calls (0 = default 60s, unchanged if omitted). - #[arg(long)] - timeout: Option, - }, - - /// Get workspace-level inference provider and model. - #[command(help_template = LEAF_HELP_TEMPLATE, next_help_heading = "FLAGS")] - Get { - /// Show the system inference route instead of the user-facing route. - /// When omitted, both routes are displayed. - #[arg(long)] - system: bool, - }, - - /// Delete a workspace-level inference route. - #[command(help_template = LEAF_HELP_TEMPLATE, next_help_heading = "FLAGS")] - Delete { - /// Delete the system inference route instead of the user-facing route. - #[arg(long)] - system: bool, - }, -} - // ----------------------------------------------------------------------- // Doctor (diagnostic) commands // ----------------------------------------------------------------------- @@ -1339,16 +1271,21 @@ enum SandboxCommands { #[arg(long, add = ArgValueCompleter::new(completers::complete_sandbox_names))] name: Option, - /// Sandbox source: a community sandbox name (e.g., `ollama`), a path - /// to a Dockerfile or directory containing one, or a full container + /// Create the sandbox from a named sandbox template. + #[arg(long, conflicts_with_all = ["from", "gpu", "cpu", "memory", "driver_config_json", "envs"])] + template: Option, + + /// Sandbox source: a community sandbox name (e.g., `ollama`), a rootfs + /// tar archive (`.tar`, `.tar.gz`, or `.tgz`), or a full container /// image reference (e.g., `myregistry.com/img:tag`). /// /// Community names are resolved to /// `ghcr.io/nvidia/openshell-community/sandboxes/:latest` /// (override the prefix with `OPENSHELL_COMMUNITY_REGISTRY`). /// - /// When given a Dockerfile or directory, the image is built into the - /// local Docker daemon before creating the sandbox. + /// To use a local Dockerfile, build and tag it with the container + /// engine used by your local gateway, then pass the resulting image + /// reference here. A rootfs tar is staged for the VM compute driver. #[arg(long, value_hint = ValueHint::AnyPath)] from: Option, @@ -1681,6 +1618,10 @@ enum SandboxCommands { /// Manage providers attached to a sandbox. #[command(subcommand)] Provider(SandboxProviderCommands), + + /// Manage reusable sandbox workload templates. + #[command(subcommand)] + Template(SandboxTemplateCommands), } #[derive(Subcommand, Debug)] @@ -1691,6 +1632,10 @@ enum SandboxProviderCommands { /// Sandbox name (defaults to last-used sandbox). #[arg(add = ArgValueCompleter::new(completers::complete_sandbox_names))] name: Option, + + /// Output format. + #[arg(short = 'o', long = "output", value_enum, default_value_t = OutputFormat::Table)] + output: OutputFormat, }, /// Attach a provider to a sandbox. @@ -1718,6 +1663,121 @@ enum SandboxProviderCommands { }, } +#[derive(Subcommand, Debug)] +// `Create` carries several optional strings and repeated key-value flags. This +// enum is only used for clap parsing, so boxing fields would add friction +// without a meaningful runtime win. +#[allow(clippy::large_enum_variant)] +enum SandboxTemplateCommands { + /// Create a reusable sandbox workload template. + #[command(help_template = LEAF_HELP_TEMPLATE, next_help_heading = "FLAGS")] + Create { + /// Template name. + name: String, + + /// Container image for sandboxes created from this template. + /// When omitted, the gateway's default sandbox image is used at create time. + #[arg(long)] + image: Option, + + /// CPU limit for sandboxes created from this template (for example: 500m, 1, 2.5). + #[arg(long)] + cpu: Option, + + /// Memory limit for sandboxes created from this template (for example: 512Mi, 4Gi, 8G). + #[arg(long)] + memory: Option, + + /// Request GPU resources for sandboxes created from this template. + /// + /// Omit COUNT for the driver's default GPU selection, or pass COUNT + /// to request a specific number of GPUs. + #[arg(long, num_args = 0..=1, value_name = "COUNT", default_missing_value = "", value_parser = parse_gpu_request)] + gpu: Option, + + /// Experimental driver-keyed JSON object for driver-specific sandbox settings. + #[arg(long, value_name = "JSON")] + driver_config_json: Option, + + /// Target startup readiness duration for this template (for example: 30s, 5m, 1h). + #[arg(long, value_name = "DURATION")] + ready_within: Option, + + /// Maximum startup burst associated with this template. + #[arg(long, value_name = "COUNT", value_parser = clap::value_parser!(u32).range(1..))] + max_burst: Option, + + /// Attach labels to the template (key=value format, repeatable). + #[arg(long = "label", value_name = "KEY=VALUE")] + labels: Vec, + + /// Attach annotations to the template (key=value format, repeatable). + #[arg(long = "annotation", value_name = "KEY=VALUE")] + annotations: Vec, + + /// Set a non-secret environment variable in sandboxes created from this template. + /// Do not use this option for API keys, tokens, or other secrets; create + /// a provider and attach it when creating sandboxes instead. Repeatable. + #[arg(long = "env", value_name = "KEY=VALUE")] + envs: Vec, + + /// Suppress warnings when --env values look like credentials. + #[arg(long = "no-credential-warnings")] + no_credential_warnings: bool, + + /// Output format. + #[arg(short = 'o', long = "output", value_enum, default_value_t = OutputFormat::Table)] + output: OutputFormat, + }, + + /// Fetch a sandbox workload template by name. + #[command(help_template = LEAF_HELP_TEMPLATE, next_help_heading = "FLAGS")] + Get { + /// Template name. + name: String, + + /// Output format. + #[arg(short = 'o', long = "output", value_enum, default_value_t = OutputFormat::Table)] + output: OutputFormat, + }, + + /// List sandbox workload templates. + #[command(help_template = LEAF_HELP_TEMPLATE, next_help_heading = "FLAGS")] + List { + /// Maximum number of templates to return. + #[arg(long, default_value_t = 100)] + limit: u32, + + /// Offset into the template list. + #[arg(long, default_value_t = 0)] + offset: u32, + + /// Filter templates by labels, e.g. env=prod,team=runtime. + #[arg(long)] + label_selector: Option, + + /// Print only template names (one per line). + #[arg(long, conflicts_with = "output")] + names: bool, + + /// Output format. + #[arg(short = 'o', long = "output", value_enum, default_value_t = OutputFormat::Table, conflicts_with = "names")] + output: OutputFormat, + + /// List templates across all workspaces (overrides --workspace). + #[arg(long)] + all_workspaces: bool, + }, + + /// Delete sandbox workload templates. + #[command(help_template = LEAF_HELP_TEMPLATE, next_help_heading = "FLAGS")] + Delete { + /// Template names. + #[arg(required = true, num_args = 1.., value_name = "NAME")] + names: Vec, + }, +} + #[derive(Subcommand, Debug)] enum DraftCommands { /// Show network rules for a sandbox. @@ -1905,6 +1965,10 @@ enum PolicyCommands { /// List global policy revisions. #[arg(long)] global: bool, + + /// Output format. + #[arg(short = 'o', long = "output", value_enum, default_value_t = OutputFormat::Table)] + output: OutputFormat, }, /// Delete the gateway-global policy lock, restoring sandbox-level policy control. @@ -2011,9 +2075,13 @@ enum ForwardCommands { name: Option, }, - /// List active port forwards. + /// List tracked port forwards. #[command(help_template = LEAF_HELP_TEMPLATE, next_help_heading = "FLAGS")] - List, + List { + /// Output format. + #[arg(short = 'o', long = "output", value_enum, default_value_t = OutputFormat::Table)] + output: OutputFormat, + }, /// Forward a local TCP port to a loopback service inside a sandbox over gRPC. #[command(help_template = LEAF_HELP_TEMPLATE, next_help_heading = "FLAGS")] @@ -2071,6 +2139,10 @@ enum ServiceCommands { /// List services across all workspaces (overrides --workspace). #[arg(long)] all_workspaces: bool, + + /// Output format. + #[arg(short = 'o', long = "output", value_enum, default_value_t = OutputFormat::Table)] + output: OutputFormat, }, /// Show one exposed sandbox service endpoint. @@ -2195,6 +2267,10 @@ enum WorkspaceMemberCommands { /// Offset into the member list. #[arg(long, default_value_t = 0)] offset: u32, + + /// Output format. + #[arg(short = 'o', long = "output", value_enum, default_value_t = OutputFormat::Table)] + output: OutputFormat, }, } @@ -2226,15 +2302,14 @@ fn run_main() -> Result<()> { #[allow(clippy::large_stack_frames)] // CLI dispatch holds many futures; run on an expanded Windows stack. async fn run_async() -> Result<()> { - // Install the rustls crypto provider before completion runs — completers may - // establish TLS connections to the gateway. - rustls::crypto::ring::default_provider() - .install_default() - .map_err(|e| miette::miette!("failed to install rustls crypto provider: {e:?}"))?; - CompleteEnv::with_factory(Cli::command).complete(); let cli = Cli::parse(); + + // Resolve colorization before anything is printed, so no command can emit + // escape sequences into a pipe. + color::init(cli.color); + let mut tls = TlsOptions::default(); tls.gateway_insecure = cli.gateway_insecure; @@ -2246,7 +2321,11 @@ async fn run_async() -> Result<()> { _ => "trace", }; + // The fmt formatter defaults to ANSI on with no terminal detection, and it + // writes to stdout, so without this `openshell -v ... | ...` pipes escapes + // to the caller. tracing_subscriber::fmt() + .with_ansi(color::stdout_enabled()) .with_env_filter( tracing_subscriber::EnvFilter::try_from_default_env() .unwrap_or_else(|_| tracing_subscriber::EnvFilter::new(log_level)), @@ -2342,7 +2421,7 @@ async fn run_async() -> Result<()> { GatewayCommands::Info { output } => { if let Ok(ctx) = resolve_gateway(&cli.gateway, &cli.gateway_endpoint) { let mut tls = tls.with_gateway_name(&ctx.name); - apply_auth(&mut tls, &ctx.name); + apply_auth(&mut tls, &ctx.name)?; run::gateway_info(&ctx.name, &ctx.endpoint, &tls, output.as_str()).await?; } else { run::gateway_info_not_configured()?; @@ -2410,7 +2489,7 @@ async fn run_async() -> Result<()> { Some(Commands::Whoami { output }) => { let ctx = resolve_gateway(&cli.gateway, &cli.gateway_endpoint)?; let mut tls = tls.with_gateway_name(&ctx.name); - apply_auth(&mut tls, &ctx.name); + apply_auth(&mut tls, &ctx.name)?; run::whoami(&ctx.endpoint, &tls, output.as_str()).await?; } @@ -2445,8 +2524,23 @@ async fn run_async() -> Result<()> { ); } } - ForwardCommands::List => { + ForwardCommands::List { output } => { let forwards = run::list_forwards()?; + if openshell_cli::output::print_output_collection( + output.as_str(), + &forwards, + |forward| { + serde_json::json!({ + "sandbox": forward.sandbox_name, + "bind_address": forward.bind_addr, + "port": forward.port, + "pid": forward.pid, + "alive": forward.validated_alive, + }) + }, + )? { + return Ok(()); + } if forwards.is_empty() { eprintln!("No active forwards."); } else { @@ -2498,7 +2592,7 @@ async fn run_async() -> Result<()> { } => { let ctx = resolve_gateway(&cli.gateway, &cli.gateway_endpoint)?; let mut tls = tls.with_gateway_name(&ctx.name); - apply_auth(&mut tls, &ctx.name); + apply_auth(&mut tls, &ctx.name)?; let name = resolve_sandbox_name(name, &ctx.name, &cli.workspace)?; let local = local.unwrap_or_else(|| target_port.to_string()); run::service_forward_tcp( @@ -2520,7 +2614,7 @@ async fn run_async() -> Result<()> { let spec = openshell_core::forward::ForwardSpec::parse(&port)?; let ctx = resolve_gateway(&cli.gateway, &cli.gateway_endpoint)?; let mut tls = tls.with_gateway_name(&ctx.name); - apply_auth(&mut tls, &ctx.name); + apply_auth(&mut tls, &ctx.name)?; let name = resolve_sandbox_name(name, &ctx.name, &cli.workspace)?; run::sandbox_forward( &ctx.endpoint, @@ -2551,7 +2645,7 @@ async fn run_async() -> Result<()> { }) => { let ctx = resolve_gateway(&cli.gateway, &cli.gateway_endpoint)?; let mut tls = tls.with_gateway_name(&ctx.name); - apply_auth(&mut tls, &ctx.name); + apply_auth(&mut tls, &ctx.name)?; match command { ServiceCommands::Expose { sandbox, @@ -2574,6 +2668,7 @@ async fn run_async() -> Result<()> { limit, offset, all_workspaces, + output, } => { run::service_list( &ctx.endpoint, @@ -2582,6 +2677,7 @@ async fn run_async() -> Result<()> { offset, &cli.workspace, all_workspaces, + output.as_str(), &tls, ) .await?; @@ -2611,7 +2707,7 @@ async fn run_async() -> Result<()> { }) => { let ctx = resolve_gateway(&cli.gateway, &cli.gateway_endpoint)?; let mut tls = tls.with_gateway_name(&ctx.name); - apply_auth(&mut tls, &ctx.name); + apply_auth(&mut tls, &ctx.name)?; let name = resolve_sandbox_name(name, &ctx.name, &cli.workspace)?; run::sandbox_logs( &ctx.endpoint, @@ -2635,7 +2731,7 @@ async fn run_async() -> Result<()> { }) => { let ctx = resolve_gateway(&cli.gateway, &cli.gateway_endpoint)?; let mut tls = tls.with_gateway_name(&ctx.name); - apply_auth(&mut tls, &ctx.name); + apply_auth(&mut tls, &ctx.name)?; match policy_cmd { PolicyCommands::Set { name, @@ -2745,14 +2841,28 @@ async fn run_async() -> Result<()> { name, limit, global, + output, } => { if global { - run::sandbox_policy_list_global(&ctx.endpoint, limit, &cli.workspace, &tls) - .await?; + run::sandbox_policy_list_global( + &ctx.endpoint, + limit, + output.as_str(), + &cli.workspace, + &tls, + ) + .await?; } else { let name = resolve_sandbox_name(name, &ctx.name, &cli.workspace)?; - run::sandbox_policy_list(&ctx.endpoint, &name, limit, &cli.workspace, &tls) - .await?; + run::sandbox_policy_list( + &ctx.endpoint, + &name, + limit, + output.as_str(), + &cli.workspace, + &tls, + ) + .await?; } } PolicyCommands::Delete { global, yes } => { @@ -2775,7 +2885,7 @@ async fn run_async() -> Result<()> { }) => { let ctx = resolve_gateway(&cli.gateway, &cli.gateway_endpoint)?; let mut tls = tls.with_gateway_name(&ctx.name); - apply_auth(&mut tls, &ctx.name); + apply_auth(&mut tls, &ctx.name)?; match settings_cmd { SettingsCommands::Get { name, global, json } => { @@ -2854,7 +2964,7 @@ async fn run_async() -> Result<()> { }) => { let ctx = resolve_gateway(&cli.gateway, &cli.gateway_endpoint)?; let mut tls = tls.with_gateway_name(&ctx.name); - apply_auth(&mut tls, &ctx.name); + apply_auth(&mut tls, &ctx.name)?; match draft_cmd { DraftCommands::Get { name, status } => { let name = resolve_sandbox_name(name, &ctx.name, &cli.workspace)?; @@ -2920,69 +3030,6 @@ async fn run_async() -> Result<()> { } } - // ----------------------------------------------------------- - // Inference commands - // ----------------------------------------------------------- - Some(Commands::Inference { - command: Some(command), - }) => { - let ctx = resolve_gateway(&cli.gateway, &cli.gateway_endpoint)?; - let endpoint = &ctx.endpoint; - let mut tls = tls.with_gateway_name(&ctx.name); - apply_auth(&mut tls, &ctx.name); - match command { - InferenceCommands::Set { - provider, - model, - system, - no_verify, - timeout, - } => { - let route_name = if system { "sandbox-system" } else { "" }; - run::gateway_inference_set( - endpoint, - &provider, - &model, - route_name, - no_verify, - timeout, - &cli.workspace, - &tls, - ) - .await?; - } - InferenceCommands::Update { - provider, - model, - system, - no_verify, - timeout, - } => { - let route_name = if system { "sandbox-system" } else { "" }; - run::gateway_inference_update( - endpoint, - provider.as_deref(), - model.as_deref(), - route_name, - no_verify, - timeout, - &cli.workspace, - &tls, - ) - .await?; - } - InferenceCommands::Get { system } => { - let route_name = if system { Some("sandbox-system") } else { None }; - run::gateway_inference_get(endpoint, route_name, &cli.workspace, &tls).await?; - } - InferenceCommands::Delete { system } => { - let route_name = if system { "sandbox-system" } else { "" }; - run::gateway_inference_delete(endpoint, route_name, &cli.workspace, &tls) - .await?; - } - } - } - // ----------------------------------------------------------- // Sandbox commands // ----------------------------------------------------------- @@ -2992,6 +3039,7 @@ async fn run_async() -> Result<()> { match command { SandboxCommands::Create { name, + template, from, upload, no_git_ignore, @@ -3079,12 +3127,13 @@ async fn run_async() -> Result<()> { let ctx = resolve_gateway(&cli.gateway, &cli.gateway_endpoint)?; let endpoint = &ctx.endpoint; let mut tls = tls.with_gateway_name(&ctx.name); - apply_auth(&mut tls, &ctx.name); + apply_auth(&mut tls, &ctx.name)?; let exit_code = Box::pin(run::sandbox_create( endpoint, &ctx.name, run::SandboxCreateConfig { name: name.as_deref(), + template: template.as_deref(), from: from.as_deref(), uploads: &upload_specs, keep, @@ -3121,7 +3170,7 @@ async fn run_async() -> Result<()> { } => { let ctx = resolve_gateway(&cli.gateway, &cli.gateway_endpoint)?; let mut tls = tls.with_gateway_name(&ctx.name); - apply_auth(&mut tls, &ctx.name); + apply_auth(&mut tls, &ctx.name)?; let local = std::path::Path::new(&local_path); run::sandbox_upload( &ctx.endpoint, @@ -3141,7 +3190,7 @@ async fn run_async() -> Result<()> { } => { let ctx = resolve_gateway(&cli.gateway, &cli.gateway_endpoint)?; let mut tls = tls.with_gateway_name(&ctx.name); - apply_auth(&mut tls, &ctx.name); + apply_auth(&mut tls, &ctx.name)?; let local_dest = dest.as_deref().unwrap_or("."); eprintln!("Downloading sandbox:{sandbox_path} -> {local_dest}"); run::sandbox_sync_down( @@ -3159,7 +3208,7 @@ async fn run_async() -> Result<()> { let ctx = resolve_gateway(&cli.gateway, &cli.gateway_endpoint)?; let endpoint = &ctx.endpoint; let mut tls = tls.with_gateway_name(&ctx.name); - apply_auth(&mut tls, &ctx.name); + apply_auth(&mut tls, &ctx.name)?; match other { SandboxCommands::Create { .. } | SandboxCommands::Upload { .. } @@ -3289,10 +3338,16 @@ async fn run_async() -> Result<()> { run::print_ssh_config(&ctx.name, &name, &cli.workspace); } SandboxCommands::Provider(command) => match command { - SandboxProviderCommands::List { name } => { + SandboxProviderCommands::List { name, output } => { let name = resolve_sandbox_name(name, &ctx.name, &cli.workspace)?; - run::sandbox_provider_list(endpoint, &name, &cli.workspace, &tls) - .await?; + run::sandbox_provider_list( + endpoint, + &name, + output.as_str(), + &cli.workspace, + &tls, + ) + .await?; } SandboxProviderCommands::Attach { name, provider } => { run::sandbox_provider_attach( @@ -3315,6 +3370,89 @@ async fn run_async() -> Result<()> { .await?; } }, + SandboxCommands::Template(command) => match command { + SandboxTemplateCommands::Create { + name, + image, + cpu, + memory, + gpu, + driver_config_json, + ready_within, + max_burst, + labels, + annotations, + envs, + no_credential_warnings, + output, + } => { + let labels = run::parse_key_value_pairs(&labels, "--label")?; + let annotations = + run::parse_key_value_pairs(&annotations, "--annotation")?; + let environment = run::parse_env_pairs(&envs)?; + run::warn_credential_env_vars(&environment, no_credential_warnings); + let gpu_requirements: Option = + gpu.map(Into::into); + run::sandbox_template_create( + endpoint, + &name, + image.as_deref(), + cpu.as_deref(), + memory.as_deref(), + gpu_requirements, + driver_config_json.as_deref(), + ready_within.as_deref(), + max_burst, + labels, + annotations, + environment, + output.as_str(), + &cli.workspace, + &tls, + ) + .await?; + } + SandboxTemplateCommands::Get { name, output } => { + run::sandbox_template_get( + endpoint, + &name, + output.as_str(), + &cli.workspace, + &tls, + ) + .await?; + } + SandboxTemplateCommands::List { + limit, + offset, + label_selector, + names, + output, + all_workspaces, + } => { + run::sandbox_template_list( + endpoint, + limit, + offset, + label_selector.as_deref(), + names, + output.as_str(), + &cli.workspace, + all_workspaces, + &tls, + ) + .await?; + } + SandboxTemplateCommands::Delete { names } => { + run::sandbox_template_delete( + endpoint, + &names, + &cli.workspace, + &tls, + ) + .await?; + } + }, } } } @@ -3329,7 +3467,7 @@ async fn run_async() -> Result<()> { let ctx = resolve_gateway(&cli.gateway, &cli.gateway_endpoint)?; let endpoint = &ctx.endpoint; let mut tls = tls.with_gateway_name(&ctx.name); - apply_auth(&mut tls, &ctx.name); + apply_auth(&mut tls, &ctx.name)?; match command { WorkspaceCommands::Create { name, labels } => { @@ -3373,9 +3511,17 @@ async fn run_async() -> Result<()> { workspace, limit, offset, + output, } => { - run::workspace_member_list(endpoint, &workspace, limit, offset, &tls) - .await?; + run::workspace_member_list( + endpoint, + &workspace, + limit, + offset, + output.as_str(), + &tls, + ) + .await?; } }, } @@ -3387,7 +3533,7 @@ async fn run_async() -> Result<()> { let ctx = resolve_gateway(&cli.gateway, &cli.gateway_endpoint)?; let endpoint = &ctx.endpoint; let mut tls = tls.with_gateway_name(&ctx.name); - apply_auth(&mut tls, &ctx.name); + apply_auth(&mut tls, &ctx.name)?; match command { ProviderCommands::Create { @@ -3562,10 +3708,10 @@ async fn run_async() -> Result<()> { ) .await?; } - ProviderProfileCommands::Delete { id, global } => { + ProviderProfileCommands::Delete { ids, global } => { run::provider_profile_delete( endpoint, - &id, + &ids, profile_workspace(global), &tls, ) @@ -3602,7 +3748,7 @@ async fn run_async() -> Result<()> { Some(Commands::Term { theme }) => { let ctx = resolve_gateway(&cli.gateway, &cli.gateway_endpoint)?; let mut tls = tls.with_gateway_name(&ctx.name); - apply_auth(&mut tls, &ctx.name); + apply_auth(&mut tls, &ctx.name)?; let channel = openshell_cli::tls::build_channel(&ctx.endpoint, &tls).await?; let interceptor = openshell_core::auth::EdgeAuthInterceptor::new( tls.oidc_token.as_deref(), @@ -3646,7 +3792,7 @@ async fn run_async() -> Result<()> { None => tls, }; if let Some(ref g) = gateway_name_opt { - apply_auth(&mut effective_tls, g); + apply_auth(&mut effective_tls, g)?; } run::sandbox_ssh_proxy(&gw, &sid, &tok, &effective_tls).await?; } @@ -3665,7 +3811,7 @@ async fn run_async() -> Result<()> { meta.gateway_endpoint }; let mut tls = tls.with_gateway_name(&g); - apply_auth(&mut tls, &g); + apply_auth(&mut tls, &g)?; run::sandbox_ssh_proxy_by_name(&endpoint, &n, &tls, &cli.workspace).await?; } // Legacy name mode with --server only (no --gateway-name). @@ -3737,13 +3883,6 @@ async fn run_async() -> Result<()> { .print_help() .expect("Failed to print help"); } - Some(Commands::Inference { command: None }) => { - Cli::command() - .find_subcommand_mut("inference") - .expect("inference subcommand exists") - .print_help() - .expect("Failed to print help"); - } Some(Commands::Rule { command: None }) => { Cli::command() .find_subcommand_mut("rule") @@ -3935,11 +4074,6 @@ mod tests { 4, "Dockerfile", ), - ( - vec!["openshell", "sandbox", "create", "--from", "Do"], - 4, - "Dockerfile", - ), ( vec![ "openshell", @@ -4093,6 +4227,11 @@ mod tests { assert!(Cli::try_parse_from(["openshell", "info"]).is_err()); } + #[test] + fn managed_inference_is_not_a_command() { + assert!(Cli::try_parse_from(["openshell", "inference", "get"]).is_err()); + } + #[test] fn hidden_aliases_still_parse() { let cli = Cli::try_parse_from(["openshell", "lg", "sandbox-1"]) @@ -4104,54 +4243,6 @@ mod tests { )); } - #[test] - fn inference_set_accepts_no_verify_flag() { - let cli = Cli::try_parse_from([ - "openshell", - "inference", - "set", - "--provider", - "openai-dev", - "--model", - "gpt-4.1", - "--no-verify", - ]) - .expect("inference set should parse --no-verify"); - - assert!(matches!( - cli.command, - Some(Commands::Inference { - command: Some(InferenceCommands::Set { - no_verify: true, - .. - }) - }) - )); - } - - #[test] - fn inference_update_accepts_no_verify_flag() { - let cli = Cli::try_parse_from([ - "openshell", - "inference", - "update", - "--provider", - "openai-dev", - "--no-verify", - ]) - .expect("inference update should parse --no-verify"); - - assert!(matches!( - cli.command, - Some(Commands::Inference { - command: Some(InferenceCommands::Update { - no_verify: true, - .. - }) - }) - )); - } - #[test] fn completion_script_uses_openshell_command_name() { let script = normalize_completion_script( @@ -4165,20 +4256,12 @@ mod tests { } #[test] - fn sandbox_create_and_download_use_path_value_hints() { + fn sandbox_download_uses_path_value_hint() { let cmd = Cli::command(); let sandbox = cmd .get_subcommands() .find(|c| c.get_name() == "sandbox") .expect("missing sandbox subcommand"); - let create = sandbox - .get_subcommands() - .find(|c| c.get_name() == "create") - .expect("missing create subcommand"); - let from = create - .get_arguments() - .find(|arg| arg.get_id() == "from") - .expect("missing from argument"); let download = sandbox .get_subcommands() .find(|c| c.get_name() == "download") @@ -4188,7 +4271,6 @@ mod tests { .find(|arg| arg.get_id() == "dest") .expect("missing dest argument"); - assert_eq!(from.get_value_hint(), ValueHint::AnyPath); assert_eq!(dest.get_value_hint(), ValueHint::AnyPath); } @@ -4351,7 +4433,7 @@ mod tests { store_edge_token("edge-gateway", "token-123").unwrap(); let mut tls = TlsOptions::default(); - apply_auth(&mut tls, "edge-gateway"); + apply_auth(&mut tls, "edge-gateway").unwrap(); assert_eq!(tls.edge_token.as_deref(), Some("token-123")); }); @@ -4513,20 +4595,172 @@ mod tests { }) if id == "custom-api" )); - let delete = - Cli::try_parse_from(["openshell", "provider", "profile", "delete", "custom-api"]) - .expect("provider profile delete should parse"); + let delete = Cli::try_parse_from([ + "openshell", + "provider", + "profile", + "delete", + "custom-api", + "custom-alt", + ]) + .expect("provider profile delete should parse"); assert!(matches!( delete.command, Some(Commands::Provider { command: Some(ProviderCommands::Profile(ProviderProfileCommands::Delete { - id, + ids, .. })) - }) if id == "custom-api" + }) if ids == vec!["custom-api".to_string(), "custom-alt".to_string()] )); } + #[test] + fn forward_list_default_output_is_table() { + let cli = Cli::try_parse_from(["openshell", "forward", "list"]) + .expect("forward list should parse"); + + assert!(matches!( + cli.command, + Some(Commands::Forward { + command: Some(ForwardCommands::List { + output: OutputFormat::Table, + }) + }) + )); + } + + #[test] + fn forward_list_accepts_structured_output() { + for (format, expected) in [("json", OutputFormat::Json), ("yaml", OutputFormat::Yaml)] { + let cli = Cli::try_parse_from(["openshell", "forward", "list", "-o", format]) + .unwrap_or_else(|error| panic!("forward list -o {format} should parse: {error}")); + + let Some(Commands::Forward { + command: Some(ForwardCommands::List { output }), + }) = cli.command + else { + panic!("expected forward list command"); + }; + assert_eq!(output, expected, "unexpected output format for {format}"); + } + } + + #[test] + fn remaining_list_commands_default_output_to_table() { + let sandbox_provider = Cli::try_parse_from(["openshell", "sandbox", "provider", "list"]) + .expect("sandbox provider list should parse"); + assert!(matches!( + sandbox_provider.command, + Some(Commands::Sandbox { + command: Some(SandboxCommands::Provider(SandboxProviderCommands::List { + output: OutputFormat::Table, + .. + })), + }) + )); + + let policy = + Cli::try_parse_from(["openshell", "policy", "list"]).expect("policy list should parse"); + assert!(matches!( + policy.command, + Some(Commands::Policy { + command: Some(PolicyCommands::List { + output: OutputFormat::Table, + .. + }), + }) + )); + + let service = Cli::try_parse_from(["openshell", "service", "list"]) + .expect("service list should parse"); + assert!(matches!( + service.command, + Some(Commands::Service { + command: Some(ServiceCommands::List { + output: OutputFormat::Table, + .. + }), + }) + )); + + let workspace_member = Cli::try_parse_from([ + "openshell", + "workspace", + "member", + "list", + "--workspace", + "default", + ]) + .expect("workspace member list should parse"); + assert!(matches!( + workspace_member.command, + Some(Commands::Workspace { + command: Some(WorkspaceCommands::Member(WorkspaceMemberCommands::List { + output: OutputFormat::Table, + .. + })), + }) + )); + } + + #[test] + fn remaining_list_commands_accept_structured_output() { + for (format, expected) in [("json", OutputFormat::Json), ("yaml", OutputFormat::Yaml)] { + let sandbox_provider = + Cli::try_parse_from(["openshell", "sandbox", "provider", "list", "-o", format]) + .unwrap_or_else(|error| panic!("sandbox provider list -o {format}: {error}")); + let Some(Commands::Sandbox { + command: + Some(SandboxCommands::Provider(SandboxProviderCommands::List { output, .. })), + }) = sandbox_provider.command + else { + panic!("expected sandbox provider list command"); + }; + assert_eq!(output, expected); + + let policy = Cli::try_parse_from(["openshell", "policy", "list", "--output", format]) + .unwrap_or_else(|error| panic!("policy list --output {format}: {error}")); + let Some(Commands::Policy { + command: Some(PolicyCommands::List { output, .. }), + }) = policy.command + else { + panic!("expected policy list command"); + }; + assert_eq!(output, expected); + + let service = Cli::try_parse_from(["openshell", "service", "list", "-o", format]) + .unwrap_or_else(|error| panic!("service list -o {format}: {error}")); + let Some(Commands::Service { + command: Some(ServiceCommands::List { output, .. }), + }) = service.command + else { + panic!("expected service list command"); + }; + assert_eq!(output, expected); + + let workspace_member = Cli::try_parse_from([ + "openshell", + "workspace", + "member", + "list", + "--workspace", + "default", + "-o", + format, + ]) + .unwrap_or_else(|error| panic!("workspace member list -o {format}: {error}")); + let Some(Commands::Workspace { + command: + Some(WorkspaceCommands::Member(WorkspaceMemberCommands::List { output, .. })), + }) = workspace_member.command + else { + panic!("expected workspace member list command"); + }; + assert_eq!(output, expected); + } + } + #[test] fn sandbox_list_default_output_is_table() { let cli = Cli::try_parse_from(["openshell", "sandbox", "list"]) @@ -4733,8 +4967,8 @@ mod tests { } #[test] - fn provider_create_requires_credential_source() { - let err = Cli::try_parse_from([ + fn provider_create_accepts_no_credential_source() { + let cli = Cli::try_parse_from([ "openshell", "provider", "create", @@ -4743,9 +4977,21 @@ mod tests { "--type", "spiffe-token-demo", ]) - .expect_err("provider create should require a credential source"); + .expect("provider create should allow profiles without static credentials"); - assert!(err.to_string().contains("--runtime-credentials")); + assert!(matches!( + cli.command, + Some(Commands::Provider { + command: Some(ProviderCommands::Create { + from_existing: false, + credentials, + from_gcloud_adc: false, + from_oidc_token: false, + runtime_credentials: false, + .. + }) + }) if credentials.is_empty() + )); } #[test] @@ -5361,6 +5607,226 @@ mod tests { } } + #[test] + fn sandbox_create_template_flag_parses() { + let cli = Cli::try_parse_from([ + "openshell", + "sandbox", + "create", + "--template", + "gpu-kata", + "--provider", + "github", + ]) + .expect("sandbox create template flag should parse"); + + match cli.command { + Some(Commands::Sandbox { + command: + Some(SandboxCommands::Create { + template, + providers, + .. + }), + .. + }) => { + assert_eq!(template.as_deref(), Some("gpu-kata")); + assert_eq!(providers, vec!["github".to_string()]); + } + other => panic!("expected SandboxCommands::Create, got: {other:?}"), + } + } + + #[test] + fn sandbox_create_template_conflicts_with_inline_workload_flags() { + for (label, extra_args) in [ + ("--from", &["--from", "python:3.12"][..]), + ("--gpu", &["--gpu"][..]), + ("--cpu", &["--cpu", "1"][..]), + ("--memory", &["--memory", "2Gi"][..]), + ("--env", &["--env", "FOO=bar"][..]), + ( + "--driver-config-json", + &["--driver-config-json", r#"{"kubernetes":{}}"#][..], + ), + ] { + let args = ["openshell", "sandbox", "create", "--template", "base"] + .into_iter() + .chain(extra_args.iter().copied()); + let result = Cli::try_parse_from(args); + assert!(result.is_err(), "--template should conflict with {label}"); + } + } + + #[test] + fn sandbox_template_create_parses_workload_flags() { + let json = r#"{"kubernetes":{"pod":{"node_selector":{"pool":"gpu"}}}}"#; + let cli = Cli::try_parse_from([ + "openshell", + "sandbox", + "template", + "create", + "gpu-kata", + "--image", + "registry.example.com/agent:latest", + "--cpu", + "2", + "--memory", + "4Gi", + "--gpu", + "1", + "--driver-config-json", + json, + "--ready-within", + "5m", + "--max-burst", + "3", + "--label", + "team=runtime", + "--annotation", + "owner=platform", + "--env", + "FEATURE_FLAG=on", + "--no-credential-warnings", + "--output", + "json", + ]) + .expect("sandbox template create should parse"); + + match cli.command { + Some(Commands::Sandbox { + command: + Some(SandboxCommands::Template(SandboxTemplateCommands::Create { + name, + image, + cpu, + memory, + gpu, + driver_config_json, + ready_within, + max_burst, + labels, + annotations, + envs, + no_credential_warnings, + output, + })), + .. + }) => { + assert_eq!(name, "gpu-kata"); + assert_eq!(image.as_deref(), Some("registry.example.com/agent:latest")); + assert_eq!(cpu.as_deref(), Some("2")); + assert_eq!(memory.as_deref(), Some("4Gi")); + assert_eq!(gpu, Some(GpuCliRequest::Count(1))); + assert_eq!(driver_config_json.as_deref(), Some(json)); + assert_eq!(ready_within.as_deref(), Some("5m")); + assert_eq!(max_burst, Some(3)); + assert_eq!(labels, vec!["team=runtime".to_string()]); + assert_eq!(annotations, vec!["owner=platform".to_string()]); + assert_eq!(envs, vec!["FEATURE_FLAG=on".to_string()]); + assert!(no_credential_warnings); + assert!(matches!(output, OutputFormat::Json)); + } + other => panic!("expected SandboxTemplateCommands::Create, got: {other:?}"), + } + } + + #[test] + fn sandbox_template_list_parses_names_and_all_workspaces() { + let cli = Cli::try_parse_from([ + "openshell", + "sandbox", + "template", + "list", + "--names", + "--all-workspaces", + "--label-selector", + "team=runtime", + "--limit", + "25", + "--offset", + "5", + ]) + .expect("sandbox template list should parse"); + + match cli.command { + Some(Commands::Sandbox { + command: + Some(SandboxCommands::Template(SandboxTemplateCommands::List { + limit, + offset, + label_selector, + names, + all_workspaces, + .. + })), + .. + }) => { + assert_eq!(limit, 25); + assert_eq!(offset, 5); + assert_eq!(label_selector.as_deref(), Some("team=runtime")); + assert!(names); + assert!(all_workspaces); + } + other => panic!("expected SandboxTemplateCommands::List, got: {other:?}"), + } + } + + #[test] + fn sandbox_template_list_names_conflicts_with_output() { + let result = Cli::try_parse_from([ + "openshell", + "sandbox", + "template", + "list", + "--names", + "--output", + "json", + ]); + assert!(result.is_err()); + } + + #[test] + fn sandbox_template_create_image_is_optional() { + let cli = Cli::try_parse_from(["openshell", "sandbox", "template", "create", "base"]) + .expect("sandbox template create without --image should parse"); + + match cli.command { + Some(Commands::Sandbox { + command: + Some(SandboxCommands::Template(SandboxTemplateCommands::Create { image, .. })), + .. + }) => { + assert_eq!(image, None); + } + other => panic!("expected SandboxTemplateCommands::Create, got: {other:?}"), + } + } + + #[test] + fn sandbox_template_create_gpu_parses_driver_default() { + let cli = Cli::try_parse_from([ + "openshell", + "sandbox", + "template", + "create", + "gpu-kata", + "--gpu", + ]) + .expect("sandbox template create --gpu should parse"); + + match cli.command { + Some(Commands::Sandbox { + command: + Some(SandboxCommands::Template(SandboxTemplateCommands::Create { gpu, .. })), + .. + }) => { + assert_eq!(gpu, Some(GpuCliRequest::DriverDefault)); + } + other => panic!("expected SandboxTemplateCommands::Create, got: {other:?}"), + } + } + #[test] fn sandbox_create_gpu_parses_driver_default() { let cli = Cli::try_parse_from(["openshell", "sandbox", "create", "--gpu"]) diff --git a/crates/openshell-cli/src/oidc_auth.rs b/crates/openshell-cli/src/oidc_auth.rs index 7fbdb5100..7e362bf42 100644 --- a/crates/openshell-cli/src/oidc_auth.rs +++ b/crates/openshell-cli/src/oidc_auth.rs @@ -29,7 +29,7 @@ use tokio::net::TcpListener; use tokio::sync::oneshot; use tracing::debug; -const AUTH_TIMEOUT: Duration = Duration::from_secs(120); +const AUTH_TIMEOUT: Duration = Duration::from_mins(2); /// OIDC discovery document (subset of fields we need). #[derive(Debug, Deserialize)] @@ -93,6 +93,7 @@ async fn discover(issuer: &str, insecure: bool) -> Result { } fn http_client(insecure: bool) -> reqwest::Client { + let _ = rustls::crypto::aws_lc_rs::default_provider().install_default(); let mut builder = reqwest::ClientBuilder::new().redirect(reqwest::redirect::Policy::none()); if insecure { builder = builder.danger_accept_invalid_certs(true); diff --git a/crates/openshell-cli/src/run.rs b/crates/openshell-cli/src/run.rs index bead4b431..3421f0f87 100644 --- a/crates/openshell-cli/src/run.rs +++ b/crates/openshell-cli/src/run.rs @@ -9,70 +9,68 @@ pub use crate::commands::common::{ }; use crate::commands::common::{ ProvisioningDisplay, ProvisioningStep, confirm_global_setting_delete, - confirm_global_setting_takeover, format_epoch_ms, format_optional_epoch_ms, - format_setting_value, format_timestamp, format_timestamp_ms, handle_platform_progress_event, - is_provisioning_progress_event, non_empty_or, parse_cli_setting_value, - parse_credential_expiry_pairs, parse_credential_pairs, parse_duration_to_ms, phase_name, + confirm_global_setting_takeover, format_epoch_ms, format_setting_value, format_timestamp, + format_timestamp_ms, handle_platform_progress_event, is_provisioning_progress_event, + non_empty_or, parse_cli_setting_value, parse_duration_to_ms, phase_name, print_policy_merge_warnings, print_sandbox_header, print_sandbox_policy, provisioning_timeout_message, ready_false_condition_message, scrub_git_env, short_hash, - truncate_display, truncate_status_field, + truncate_status_field, }; pub use crate::commands::gateway::{ gateway_add, gateway_info, gateway_info_not_configured, gateway_list, gateway_login, gateway_logout, gateway_remove, gateway_select, gateway_status, gateway_use, }; +use crate::commands::provider::inferred_provider_type; +pub use crate::commands::provider::{ + ProviderCreateCredentialSource, ProviderCreateOptions, ProviderRefreshConfigInput, + ProviderUpdateOptions, ensure_required_providers, provider_create, + provider_create_with_options, provider_delete, provider_get, provider_list, + provider_list_profiles, provider_profile_delete, provider_profile_export, + provider_profile_export_text, provider_profile_import, provider_profile_lint, + provider_profile_update, provider_refresh_config, provider_refresh_delete, + provider_refresh_status, provider_rotate, provider_update, sandbox_provider_attach, + sandbox_provider_detach, sandbox_provider_list, +}; + +use crate::color::Colorize; use crate::policy_update::build_policy_update_plan; -use crate::tls::{TlsOptions, grpc_client, grpc_inference_client}; -use dialoguer::Confirm; +use crate::tls::{TlsOptions, grpc_client}; use futures::StreamExt; -use indicatif::{ProgressBar, ProgressStyle}; use miette::{IntoDiagnostic, Result, WrapErr, miette}; use openshell_bootstrap::{ GatewayMetadata, clear_last_sandbox_if_matches, get_gateway_metadata, save_last_sandbox, }; use openshell_core::net::set_tcp_nodelay_best_effort; -use openshell_core::proto::ProviderProfileCategory; use openshell_core::proto::{ - ApproveAllDraftChunksRequest, ApproveDraftChunkRequest, AttachSandboxProviderRequest, - ClearDraftChunksRequest, ConfigureProviderRefreshRequest, CreateProviderRequest, - CreateSandboxRequest, CreateSshSessionRequest, DeleteInferenceRouteRequest, - DeleteProviderProfileRequest, DeleteProviderRefreshRequest, DeleteProviderRequest, - DeleteSandboxRequest, DeleteServiceRequest, DetachSandboxProviderRequest, ExecSandboxRequest, - ExposeServiceRequest, GetCurrentUserRequest, GetDraftHistoryRequest, GetDraftPolicyRequest, - GetGatewayConfigRequest, GetInferenceRouteRequest, GetProviderProfileRequest, - GetProviderRefreshStatusRequest, GetProviderRequest, GetSandboxConfigRequest, - GetSandboxConfigResponse, GetSandboxLogsRequest, GetSandboxPolicyStatusRequest, - GetSandboxRequest, GetServiceRequest, GpuResourceRequirements, ImportProviderProfilesRequest, - LintProviderProfilesRequest, ListProviderProfilesRequest, ListProvidersRequest, - ListSandboxPoliciesRequest, ListSandboxProvidersRequest, ListSandboxesRequest, - ListServicesRequest, PolicySource, PolicyStatus, Provider, - ProviderCredentialRefreshRecoveryAction, ProviderCredentialRefreshStatus, - ProviderCredentialRefreshStrategy, ProviderCredentialTokenGrantType, ProviderProfile, - ProviderProfileDiagnostic, ProviderProfileImportItem, RejectDraftChunkRequest, - ResourceRequirements, RevokeSshSessionRequest, RotateProviderCredentialRequest, Sandbox, - SandboxPhase, SandboxPolicy, SandboxSpec, SandboxTemplate, ServiceEndpointResponse, - SetInferenceRouteRequest, SettingScope, StartSandboxRequest, StopSandboxRequest, - TcpForwardFrame, TcpForwardInit, TcpRelayTarget, UpdateConfigRequest, - UpdateProviderProfilesRequest, UpdateProviderRequest, WatchSandboxRequest, exec_sandbox_event, - setting_value, tcp_forward_init, + ApproveAllDraftChunksRequest, ApproveDraftChunkRequest, BeginRootfsTarStagingRequest, + ClearDraftChunksRequest, CreateSandboxRequest, CreateSandboxTemplateRequest, + CreateSshSessionRequest, DeleteSandboxRequest, DeleteSandboxTemplateRequest, + DeleteServiceRequest, ExecSandboxRequest, ExposeServiceRequest, GetCurrentUserRequest, + GetDraftHistoryRequest, GetDraftPolicyRequest, GetGatewayConfigRequest, + GetSandboxConfigRequest, GetSandboxConfigResponse, GetSandboxLogsRequest, + GetSandboxPolicyStatusRequest, GetSandboxRequest, GetSandboxTemplateRequest, GetServiceRequest, + GpuResourceRequirements, ListSandboxPoliciesRequest, ListSandboxTemplatesRequest, + ListSandboxesRequest, ListServicesRequest, PolicySource, PolicyStatus, RejectDraftChunkRequest, + ResourceRequirements, RevokeSshSessionRequest, Sandbox, SandboxPhase, SandboxPolicy, + SandboxResources, SandboxServiceLevel, SandboxSpec, SandboxStartup, SandboxTemplate, + SandboxWorkloadConfig, SandboxWorkloadTemplate, SandboxWorkloadTemplateSpec, + ServiceEndpointResponse, SettingScope, StartSandboxRequest, StopSandboxRequest, + TcpForwardFrame, TcpForwardInit, TcpRelayTarget, UpdateConfigRequest, WatchSandboxRequest, + exec_sandbox_event, tcp_forward_init, }; use openshell_core::settings; use openshell_core::{ObjectId, ObjectName, ObjectWorkspace}; -use openshell_providers::{ - ProviderRegistry, ProviderTypeProfile, RealDiscoveryContext, detect_provider_from_command, - discover_from_profile, normalize_provider_type, parse_profile_json, parse_profile_yaml, - profile_to_json, profile_to_yaml, profiles_to_json, profiles_to_yaml, -}; -use owo_colors::OwoColorize; use std::borrow::Cow; -use std::collections::{HashMap, HashSet}; +use std::collections::HashMap; use std::io::{ErrorKind, IsTerminal, Read, Write}; use std::path::{Path, PathBuf}; use std::process::Command; use std::time::{Duration, Instant}; use tonic::{Code, Status}; +const PROVISIONAL_CONTAINER_EXIT_RECONCILIATION_TIMEOUT: Duration = Duration::from_secs(5); + // Re-export SSH functions for backward compatibility pub use crate::ssh::{Editor, print_ssh_config}; pub use crate::ssh::{ @@ -119,6 +117,20 @@ impl ProgressOutput { } } +fn aggregate_delete_failures(resource: &str, failures: &[String]) -> Result<()> { + if failures.is_empty() { + Ok(()) + } else { + Err(miette!( + "failed to delete {} {}{}: {}", + failures.len(), + resource, + if failures.len() == 1 { "" } else { "s" }, + failures.join(", ") + )) + } +} + #[derive(Debug, Clone)] struct CurrentUserView { subject: String, @@ -241,6 +253,19 @@ fn has_main_process_result(sandbox: &Sandbox) -> bool { }) } +fn is_provisional_container_exit(sandbox: &Sandbox) -> bool { + let phase = SandboxPhase::try_from(sandbox.phase()).unwrap_or(SandboxPhase::Unknown); + phase == SandboxPhase::Error + && sandbox.status.as_ref().is_some_and(|status| { + status.exit_code.is_none() + && status.conditions.iter().any(|condition| { + condition.r#type == "Ready" + && condition.status.eq_ignore_ascii_case("false") + && condition.reason == "ContainerExited" + }) + }) +} + fn build_sandbox_resource_limits( cpu: Option<&str>, memory: Option<&str>, @@ -386,6 +411,7 @@ async fn finalize_sandbox_create_session( #[derive(Debug)] pub struct SandboxCreateConfig<'a> { pub name: Option<&'a str>, + pub template: Option<&'a str>, pub from: Option<&'a str>, pub uploads: &'a [(String, Option, bool)], pub keep: bool, @@ -411,6 +437,7 @@ impl Default for SandboxCreateConfig<'_> { fn default() -> Self { Self { name: None, + template: None, from: None, uploads: &[], keep: false, @@ -444,6 +471,7 @@ pub async fn sandbox_create( ) -> Result { let SandboxCreateConfig { name, + template, from, uploads, keep, @@ -497,36 +525,41 @@ pub async fn sandbox_create( let effective_server = server.to_string(); let effective_tls = tls.clone(); - // Resolve the --from flag into a container image reference, building from - // a Dockerfile first if necessary. - let image: Option = match from { - Some(val) => { - let resolved = resolve_from(val)?; - match resolved { - ResolvedSource::Image(img) => Some(img), - ResolvedSource::Dockerfile { - dockerfile, - context, - } => { - let tag = build_from_dockerfile(&dockerfile, &context, gateway_name).await?; - Some(tag) + if template.is_some() + && (from.is_some() + || gpu_requirements.is_some() + || cpu.is_some() + || memory.is_some() + || driver_config_json.is_some() + || !environment.is_empty()) + { + return Err(miette::miette!( + "--template cannot be combined with inline workload flags" + )); + } + + // Resolve the --from flag into a container image reference, or stage a + // rootfs tar on the gateway and retain its staging token. Template creates + // resolve workload shape on the gateway and skip --from. + let (image, rootfs_tar_token): (Option, Option) = if template.is_some() { + (None, None) + } else { + match from { + Some(val) => { + let resolved = resolve_from(val)?; + match resolved { + ResolvedSource::Image(img) => (Some(img), None), + ResolvedSource::RootfsTar { path } => { + let token = + stage_rootfs_tar(gateway_name, &mut client, workspace, &path).await?; + (None, Some(token)) + } } } + None => (None, None), } - None => None, - }; - let inferred_provider = inferred_provider_type(command); - let providers_v2_enabled = - if inferred_provider.is_some() && auto_providers_override != Some(false) { - gateway_providers_v2_enabled(&mut client).await? - } else { - false - }; - let inferred_types: Vec = if providers_v2_enabled { - Vec::new() - } else { - inferred_provider.into_iter().collect() }; + let inferred_types: Vec = inferred_provider_type(command).into_iter().collect(); let configured_providers = ensure_required_providers( &mut client, providers, @@ -537,12 +570,28 @@ pub async fn sandbox_create( .await?; let policy = load_sandbox_policy(policy)?; - let resource_limits = build_sandbox_resource_limits(cpu, memory)?; - let driver_config = driver_config_json - .map(parse_driver_config_json) - .transpose()?; + let resource_limits = if template.is_none() { + build_sandbox_resource_limits(cpu, memory)? + } else { + None + }; + let mut driver_config = if template.is_none() { + driver_config_json + .map(parse_driver_config_json) + .transpose()? + } else { + None + }; - let template = if image.is_some() || resource_limits.is_some() || driver_config.is_some() { + if let Some(token) = &rootfs_tar_token { + driver_config = Some(merge_rootfs_tar_driver_config(driver_config, token)?); + } + + let inline_template = if image.is_some() + || resource_limits.is_some() + || driver_config.is_some() + || rootfs_tar_token.is_some() + { Some(SandboxTemplate { image: image.unwrap_or_default(), resources: resource_limits, @@ -557,11 +606,11 @@ pub async fn sandbox_create( let main_terminal = tty_override .unwrap_or_else(|| std::io::stdin().is_terminal() && std::io::stdout().is_terminal()); - let main_command = if command.is_empty() { - vec!["/bin/bash".to_string(), "-l".to_string()] - } else { - command.to_vec() - }; + // Forward the command as-is. When empty, the gateway persists it empty and + // the supervisor resolves the default login shell against the sandbox image + // (bash when present, otherwise /bin/sh on minimal images like Alpine). + // Baking a shell here would force a shell the image may not ship. + let main_command = command.to_vec(); let persist = sandbox_should_persist(keep, forward.as_ref()); let create_detaches = detach || (persist @@ -579,10 +628,14 @@ pub async fn sandbox_create( let request = CreateSandboxRequest { spec: Some(SandboxSpec { resource_requirements, - environment, + environment: if template.is_none() { + environment + } else { + HashMap::new() + }, policy, providers: configured_providers, - template, + template: inline_template, command: main_command, tty: main_terminal, ..SandboxSpec::default() @@ -590,8 +643,9 @@ pub async fn sandbox_create( name: name.unwrap_or_default().to_string(), labels, annotations, - workspace: workspace.to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector(workspace)), await_main_process_attachment, + workload_template_name: template.unwrap_or_default().to_string(), }; let response = match client.create_sandbox(request).await { @@ -635,7 +689,7 @@ pub async fn sandbox_create( name: sandbox_name.clone(), setting_key: settings::PROPOSAL_APPROVAL_MODE_KEY.to_string(), setting_value: Some(setting), - workspace: workspace.to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector(workspace)), ..Default::default() }) .await @@ -726,6 +780,12 @@ pub async fn sandbox_create( .unwrap_or(300), ); let mut provisioning_idle_deadline = Instant::now() + provision_timeout; + // The compute driver can publish ContainerExited while the supervisor's + // authoritative canonical-process result is waiting for the same gateway + // state lock. Keep watching briefly so the provisional error cannot race + // ephemeral cleanup, but retain a deadline for containers that exit before + // the supervisor can report a result. + let mut provisional_container_exit_deadline: Option = None; // Track whether we saw the gateway become ready (from log messages). let mut saw_gateway_ready = false; @@ -734,8 +794,15 @@ pub async fn sandbox_create( // longer than the default timeout pulling and preparing large images, // but only recognized progress events extend the idle deadline. Logs // and generic status churn must not keep a stuck sandbox alive forever. - let remaining = provisioning_idle_deadline.saturating_duration_since(Instant::now()); + let now = Instant::now(); + let mut remaining = provisioning_idle_deadline.saturating_duration_since(now); + if let Some(deadline) = provisional_container_exit_deadline { + remaining = remaining.min(deadline.saturating_duration_since(now)); + } if remaining.is_zero() { + if provisional_container_exit_deadline.is_some() { + break; + } let timeout_message = provisioning_timeout_message( provision_timeout.as_secs(), resource_requirements.as_ref(), @@ -755,6 +822,7 @@ pub async fn sandbox_create( let item = match maybe_item { Ok(Some(item)) => item, Ok(None) => break, // stream ended + Err(_elapsed) if provisional_container_exit_deadline.is_some() => break, Err(_elapsed) => { // Timeout fired — the stream was idle for too long. let timeout_message = provisioning_timeout_message( @@ -811,6 +879,12 @@ pub async fn sandbox_create( format!("{}: {}", condition.reason, condition.message); } } + if is_provisional_container_exit(&s) { + provisional_container_exit_deadline.get_or_insert_with(|| { + Instant::now() + PROVISIONAL_CONTAINER_EXIT_RECONCILIATION_TIMEOUT + }); + continue; + } break; } @@ -1107,181 +1181,249 @@ pub async fn sandbox_create( enum ResolvedSource { /// A ready-to-use container image reference. Image(String), - /// A Dockerfile that must be built before creating the sandbox. - Dockerfile { - dockerfile: PathBuf, - context: PathBuf, - }, + /// A flat rootfs tar archive (`.tar`, `.tar.gz`, `.tgz`) to stage for the + /// VM compute driver. + RootfsTar { path: PathBuf }, } -/// Classify the `--from` value into an image reference or a Dockerfile that -/// needs building. +/// Classify the `--from` value into an image reference or a rootfs tar to stage +/// for the VM driver. /// /// Resolution order: -/// 1. Existing file whose name contains "Dockerfile" → build from file. -/// 2. Existing directory that contains a `Dockerfile` → build from directory. -/// 3. Missing explicit local paths → local error, not image pull. -/// 4. Value contains `/`, `:`, or `.` → treat as a full image reference. -/// 5. Otherwise → community sandbox name, expanded via the registry prefix. +/// 1. Existing file with `.tar`, `.tar.gz`, or `.tgz` extension → rootfs tar archive. +/// 2. Local Dockerfile and directory paths → an actionable build-and-tag error. +/// 3. Other explicit local paths → an actionable error. +/// 4. Full image reference or community sandbox name → resolve as an image. fn resolve_from(value: &str) -> Result { let path = Path::new(value); - // 1. Existing file that looks like a Dockerfile. - if path.is_file() { - if filename_looks_like_dockerfile(path) { - let dockerfile = path - .canonicalize() - .into_diagnostic() - .wrap_err_with(|| format!("failed to resolve path: {}", path.display()))?; - let context = dockerfile - .parent() - .ok_or_else(|| miette::miette!("Dockerfile has no parent directory"))? - .to_path_buf(); - return Ok(ResolvedSource::Dockerfile { - dockerfile, - context, - }); - } + if path.is_file() && filename_looks_like_rootfs_tar(path) { + let tar_path = path + .canonicalize() + .into_diagnostic() + .wrap_err_with(|| format!("failed to resolve path: {}", path.display()))?; + return Ok(ResolvedSource::RootfsTar { path: tar_path }); + } - if value_looks_like_local_source(value) { + if value_looks_like_local_path(value) { + if !path.exists() && filename_looks_like_rootfs_tar(path) { return Err(miette::miette!( - "local --from file is not a Dockerfile: {}", + "local --from path does not exist: {}", path.display() )); } - } - // 2. Existing directory containing a Dockerfile. - if path.is_dir() { - let candidate = path.join("Dockerfile"); - if candidate.is_file() { - let context = path - .canonicalize() - .into_diagnostic() - .wrap_err_with(|| format!("failed to resolve path: {}", path.display()))?; - let dockerfile = context.join("Dockerfile"); - return Ok(ResolvedSource::Dockerfile { - dockerfile, - context, - }); - } + let build_context = if path.is_dir() { + path.display().to_string() + } else { + path.parent() + .map(|p| p.display().to_string()) + .filter(|p| !p.is_empty()) + .unwrap_or_else(|| ".".to_string()) + }; return Err(miette::miette!( - "No Dockerfile found in directory: {}", - path.display() + "'--from' no longer builds local Dockerfiles or directories: {}\n\ + Build and tag the image with the container engine used by the gateway, then pass the resulting image reference:\n \ + docker build -t {} # Docker gateway\n \ + podman build -t {} # Podman gateway\n \ + openshell sandbox create --from \n\ + If the gateway cannot access the selected engine's local image store (for example, a remote or Kubernetes gateway), push the image to a registry that the gateway can pull from.", + path.display(), + build_context, + build_context, )); } - if path.exists() { - return Err(miette::miette!( - "local --from path is not a regular file or directory: {}", - path.display() - )); - } - - // 3. Missing explicit local paths should fail locally. Otherwise values - // like `./Dockerfile` reach the gateway as image references and fail as - // Docker pull errors. - if value_looks_like_local_source(value) { - return Err(miette::miette!( - "local --from path does not exist: {}\n\ - Use an existing Dockerfile, a directory containing Dockerfile, or a container image reference.", - path.display() - )); - } - - // 4. Full image reference or community sandbox name — delegate to shared - // resolution in openshell-core. + // Full image reference or community sandbox name — delegate to shared + // resolution in openshell-core. Ok(ResolvedSource::Image( openshell_core::image::resolve_community_image(value), )) } -fn filename_looks_like_dockerfile(path: &Path) -> bool { +#[allow(clippy::case_sensitive_file_extension_comparisons)] // already lowercased +fn filename_looks_like_rootfs_tar(path: &Path) -> bool { let name = path .file_name() .map(|n| n.to_string_lossy()) .unwrap_or_default(); let lower = name.to_lowercase(); - lower.contains("dockerfile") || lower.ends_with(".dockerfile") + lower.ends_with(".tar.gz") || lower.ends_with(".tar") || lower.ends_with(".tgz") } -fn value_looks_like_local_source(value: &str) -> bool { - value_is_explicit_local_path(value) || value_looks_like_bare_dockerfile_name(value) -} - -fn value_is_explicit_local_path(value: &str) -> bool { +fn value_looks_like_local_path(value: &str) -> bool { let path = Path::new(value); path.is_absolute() || matches!(value, "." | "..") || value.starts_with("./") || value.starts_with("../") || value.starts_with("~/") + || value_looks_like_bare_dockerfile_name(value) } fn value_looks_like_bare_dockerfile_name(value: &str) -> bool { - !value.contains('/') && !value.contains(':') && filename_looks_like_dockerfile(Path::new(value)) + !value.contains('/') + && !value.contains(':') + && Path::new(value) + .file_name() + .is_some_and(|name| name.to_string_lossy().to_lowercase().contains("dockerfile")) } -fn dockerfile_sources_supported_for_gateway(metadata: Option<&GatewayMetadata>) -> bool { +fn rootfs_tar_sources_supported_for_gateway(metadata: Option<&GatewayMetadata>) -> bool { !metadata.is_some_and(|metadata| metadata.is_remote) } -/// Build a Dockerfile and return the local Docker tag. +/// Ask the gateway for a staging slot, then copy the archive into it. /// -/// Package-managed local gateways use the same Docker daemon that the CLI -/// builds into, so the tag is passed through directly and the active compute -/// driver resolves it. -async fn build_from_dockerfile( - dockerfile: &Path, - context: &Path, +/// The gateway owns the destination: it allocates a request-scoped directory +/// and returns a single-use token. We never name a path of our own choosing, +/// so a request cannot reach for another caller's archive or an arbitrary host +/// file. Returns the token to pass on `CreateSandbox`. +async fn stage_rootfs_tar( gateway_name: &str, + client: &mut crate::tls::GrpcClient, + workspace: &str, + tar_path: &Path, ) -> Result { let metadata = get_gateway_metadata(gateway_name); - if !dockerfile_sources_supported_for_gateway(metadata.as_ref()) { + if !rootfs_tar_sources_supported_for_gateway(metadata.as_ref()) { return Err(miette!( - "local Dockerfile sources are only supported for local gateways; gateway '{}' is remote", + "local rootfs tar sources are only supported for local gateways; gateway '{}' is remote", gateway_name )); } - let timestamp = std::time::SystemTime::now() - .duration_since(std::time::UNIX_EPOCH) - .unwrap_or_default() - .as_secs(); - let tag = format!("openshell/sandbox-from:{timestamp}"); + let file_name = tar_path + .file_name() + .ok_or_else(|| miette!("rootfs tar path has no filename"))? + .to_string_lossy() + .into_owned(); + let source_meta = tokio::fs::metadata(tar_path) + .await + .into_diagnostic() + .wrap_err_with(|| format!("failed to read {}", tar_path.display()))?; + // The gateway rejects a driver that cannot take rootfs tar sources, and an + // archive over its configured limit, before allocating anything. + let slot = client + .begin_rootfs_tar_staging(BeginRootfsTarStagingRequest { + workspace_scope: Some(openshell_core::proto::workspace_selector(workspace)), + file_name, + size_bytes: source_meta.len(), + }) + .await + .into_diagnostic() + .wrap_err("failed to allocate a rootfs tar staging slot on the gateway")? + .into_inner(); + + let staged_path = PathBuf::from(&slot.upload_path); eprintln!( - "Building image {} from {}", - tag.cyan(), - dockerfile.display() - ); - eprintln!(" {} {}", "Context:".dimmed(), context.display()); - eprintln!(" {} {}", "Gateway:".dimmed(), gateway_name); - eprintln!(); - - let mut on_log = |msg: String| { - eprintln!(" {msg}"); - }; - - openshell_bootstrap::build::build_local_image( - dockerfile, - &tag, - context, - &HashMap::new(), - &mut on_log, - ) - .await?; - - eprintln!(); - eprintln!( - "{} Image {} is available in the local Docker daemon for gateway '{}'.", - "✓".green().bold(), - tag.cyan(), + "Staging rootfs tar {} for gateway '{}'", + tar_path.display().to_string().cyan(), gateway_name, ); + // Enforced while streaming, so an archive that grows after the size check + // above still cannot exceed the limit. + if let Err(err) = copy_with_byte_limit(tar_path, &staged_path, slot.max_bytes).await { + // The staging directory belongs to the gateway, which reclaims it when + // the slot expires. Removing it here would reach into its state. + return Err(miette!( + "failed to stage rootfs tar to {}: {err}", + staged_path.display() + )); + } eprintln!(); - Ok(tag) + Ok(slot.staging_token) +} + +/// Copy `src` to `dst`, aborting if total bytes written exceeds `limit`. +/// A limit of 0 disables enforcement. +async fn copy_with_byte_limit( + src: &Path, + dst: &Path, + limit: u64, +) -> std::result::Result<(), String> { + use tokio::io::{AsyncReadExt, AsyncWriteExt}; + + let mut reader = tokio::fs::File::open(src) + .await + .map_err(|e| format!("open source: {e}"))?; + let mut writer = tokio::fs::File::create(dst) + .await + .map_err(|e| format!("create destination: {e}"))?; + + let mut buf = vec![0u8; 64 * 1024]; + let mut total: u64 = 0; + loop { + let n = reader + .read(&mut buf) + .await + .map_err(|e| format!("read: {e}"))?; + if n == 0 { + break; + } + total += n as u64; + if limit > 0 && total > limit { + return Err(format!( + "{} exceeds the {} byte limit", + src.display(), + limit + )); + } + writer + .write_all(&buf[..n]) + .await + .map_err(|e| format!("write: {e}"))?; + } + Ok(()) +} + +/// `driver_config` key for the VM compute driver. The gateway forwards only +/// `template.driver_config.` to the selected driver, so VM +/// settings must be nested under this key or they are dropped. +const VM_DRIVER_CONFIG_KEY: &str = "vm"; +/// VM `driver_config` field naming the gateway-issued staging slot. The +/// gateway swaps it for the resolved archive path before the driver sees it. +const ROOTFS_TAR_TOKEN_FIELD: &str = "rootfs_tar_staging_token"; + +/// Merge the staging token into `driver_config.vm`, preserving any VM settings +/// the caller already supplied through `--driver-config-json`. +fn merge_rootfs_tar_driver_config( + base: Option, + staging_token: &str, +) -> Result { + use prost_types::{Struct, Value, value::Kind}; + + let mut config = base.unwrap_or_default(); + let vm = config + .fields + .entry(VM_DRIVER_CONFIG_KEY.to_string()) + .or_insert_with(|| Value { + kind: Some(Kind::StructValue(Struct::default())), + }); + + let Some(Kind::StructValue(vm_config)) = vm.kind.as_mut() else { + return Err(miette!( + "--driver-config-json '{VM_DRIVER_CONFIG_KEY}' must be an object" + )); + }; + + if vm_config.fields.contains_key(ROOTFS_TAR_TOKEN_FIELD) { + return Err(miette!( + "--driver-config-json already sets {VM_DRIVER_CONFIG_KEY}.{ROOTFS_TAR_TOKEN_FIELD}; \ + remove it or drop the rootfs tar from --from" + )); + } + + vm_config.fields.insert( + ROOTFS_TAR_TOKEN_FIELD.to_string(), + Value { + kind: Some(Kind::StringValue(staging_token.to_string())), + }, + ); + + Ok(config) } /// Load sandbox policy YAML. @@ -1353,7 +1495,7 @@ pub async fn sandbox_get( let response = client .get_sandbox(GetSandboxRequest { name: name.to_string(), - workspace: workspace.to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector(workspace)), }) .await .into_diagnostic()?; @@ -1438,6 +1580,15 @@ pub async fn sandbox_get( } } + if let Some(provenance) = &sandbox.created_from_workload_template { + println!( + " {} {}@{}", + "Workload template:".dimmed(), + provenance.name, + provenance.resource_version + ); + } + let policy_from_global = config.policy_source == PolicySource::Global as i32; println!( " {} {}", @@ -1477,9 +1628,16 @@ pub async fn sandbox_get( /// data into memory before the server rejects an oversized message. const MAX_STDIN_PAYLOAD: usize = 4 * 1024 * 1024; +fn local_terminal_size() -> Option<(u32, u32)> { + crossterm::terminal::size() + .ok() + .map(|(cols, rows)| (u32::from(cols), u32::from(rows))) +} + /// Execute a command in a running sandbox via gRPC, streaming output to the terminal. /// -/// Returns the remote command's exit code. +/// Returns the remote command's exit code, or an error if the event stream +/// closes before the command reports an exit status. #[allow(clippy::too_many_arguments, clippy::implicit_hasher)] pub async fn sandbox_exec_grpc( server: &str, @@ -1499,7 +1657,7 @@ pub async fn sandbox_exec_grpc( let sandbox = client .get_sandbox(GetSandboxRequest { name: name.to_string(), - workspace: workspace.to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector(workspace)), }) .await .into_diagnostic()? @@ -1545,7 +1703,7 @@ pub async fn sandbox_exec_grpc( let tty = tty_override .unwrap_or_else(|| std::io::stdin().is_terminal() && std::io::stdout().is_terminal()); - if tty_override == Some(true) && std::io::stdin().is_terminal() { + if tty && std::io::stdin().is_terminal() { return sandbox_exec_interactive_grpc( client, &sandbox, @@ -1558,6 +1716,12 @@ pub async fn sandbox_exec_grpc( .await; } + let (cols, rows) = if tty { + local_terminal_size().unwrap_or_default() + } else { + (0, 0) + }; + // Make the streaming gRPC call. let mut stream = client .exec_sandbox(ExecSandboxRequest { @@ -1568,8 +1732,9 @@ pub async fn sandbox_exec_grpc( timeout_seconds, stdin: stdin_payload, tty, + cols, + rows, no_login_shell, - ..Default::default() }) .await .into_diagnostic()? @@ -1577,6 +1742,7 @@ pub async fn sandbox_exec_grpc( // Stream output to terminal in real-time. let mut exit_code = 0i32; + let mut exit_seen = false; let stdout = std::io::stdout(); let stderr = std::io::stderr(); @@ -1595,11 +1761,21 @@ pub async fn sandbox_exec_grpc( } Some(exec_sandbox_event::Payload::Exit(exit)) => { exit_code = exit.exit_code; + exit_seen = true; } None => {} } } + // A stream that closes without an Exit event means we never observed the + // command's outcome. The server treats the same condition as a relay + // failure; mirror that here so exit 0 stays meaningful. + if !exit_seen { + return Err(miette::miette!( + "sandbox exec relay closed before the command reported an exit status" + )); + } + Ok(exit_code) } @@ -1715,7 +1891,7 @@ async fn fetch_ready_sandbox_for_forward( let response = match client .get_sandbox(GetSandboxRequest { name: name.to_string(), - workspace: workspace.to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector(workspace)), }) .await { @@ -1925,7 +2101,7 @@ async fn sandbox_exec_interactive_grpc( use openshell_core::proto::{ExecSandboxInput, exec_sandbox_input}; use tokio_stream::wrappers::ReceiverStream; - let (cols, rows) = crossterm::terminal::size().unwrap_or((80, 24)); + let (cols, rows) = local_terminal_size().unwrap_or((80, 24)); let (input_tx, input_rx) = tokio::sync::mpsc::channel::(4096); @@ -1941,8 +2117,8 @@ async fn sandbox_exec_interactive_grpc( timeout_seconds, stdin: Vec::new(), tty: true, - cols: u32::from(cols), - rows: u32::from(rows), + cols, + rows, })), }) .await @@ -1992,13 +2168,10 @@ async fn sandbox_exec_interactive_grpc( tokio::signal::unix::signal(tokio::signal::unix::SignalKind::window_change()) .expect("failed to register SIGWINCH handler"); while sig.recv().await.is_some() { - if let Ok((c, r)) = crossterm::terminal::size() { + if let Some((cols, rows)) = local_terminal_size() { let msg = ExecSandboxInput { payload: Some(exec_sandbox_input::Payload::Resize( - ExecSandboxWindowResize { - cols: u32::from(c), - rows: u32::from(r), - }, + ExecSandboxWindowResize { cols, rows }, )), }; if resize_tx.send(msg).await.is_err() { @@ -2012,6 +2185,7 @@ async fn sandbox_exec_interactive_grpc( let _resize_guard = TaskGuard(resize_task); let mut exit_code = 0i32; + let mut exit_seen = false; let stdout = std::io::stdout(); let stderr = std::io::stderr(); @@ -2030,6 +2204,7 @@ async fn sandbox_exec_interactive_grpc( } Some(exec_sandbox_event::Payload::Exit(exit)) => { exit_code = exit.exit_code; + exit_seen = true; break; } None => {} @@ -2041,6 +2216,15 @@ async fn sandbox_exec_interactive_grpc( // Drop the raw mode guard to restore the terminal before returning. drop(raw_guard); + // A stream that closes without an Exit event means we never observed the + // command's outcome. Treat it as a relay failure rather than reporting a + // successful (0) exit. + if !exit_seen { + return Err(miette::miette!( + "sandbox exec relay closed before the command reported an exit status" + )); + } + Ok(exit_code) } @@ -2065,12 +2249,11 @@ pub async fn sandbox_list( limit, offset, label_selector: label_selector.unwrap_or("").to_string(), - workspace: if all_workspaces { - String::new() + workspace_scope: Some(if all_workspaces { + openshell_core::proto::all_workspaces_selector() } else { - workspace.to_string() - }, - all_workspaces, + openshell_core::proto::workspace_selector(workspace) + }), }) .await .into_diagnostic()?; @@ -2190,6 +2373,16 @@ fn sandbox_to_json(sandbox: &Sandbox) -> serde_json::Value { || serde_json::json!({}), |m| serde_json::json!(m.annotations), ); + let created_from_workload_template = + sandbox + .created_from_workload_template + .as_ref() + .map(|provenance| { + serde_json::json!({ + "name": provenance.name, + "resource_version": provenance.resource_version, + }) + }); serde_json::json!({ "id": sandbox.object_id(), "name": sandbox.object_name(), @@ -2201,6 +2394,7 @@ fn sandbox_to_json(sandbox: &Sandbox) -> serde_json::Value { "phase": phase_name(sandbox.phase()), "current_policy_version": sandbox.current_policy_version(), "exit_code": sandbox.status.as_ref().and_then(|status| status.exit_code), + "created_from_workload_template": created_from_workload_template, }) } @@ -2246,201 +2440,580 @@ fn sandbox_detail_to_json( Ok(value) } -pub async fn sandbox_provider_list( +#[allow(clippy::too_many_arguments, clippy::implicit_hasher)] +pub async fn sandbox_template_create( server: &str, name: &str, + image: Option<&str>, + cpu: Option<&str>, + memory: Option<&str>, + gpu_requirements: Option, + driver_config_json: Option<&str>, + ready_within: Option<&str>, + max_burst: Option, + labels: HashMap, + annotations: HashMap, + environment: HashMap, + output: &str, + workspace: &str, + tls: &TlsOptions, +) -> Result<()> { + let resources = if cpu.is_some() || memory.is_some() || gpu_requirements.is_some() { + Some(SandboxResources { + cpu: cpu + .map(validate_cpu_quantity) + .transpose()? + .unwrap_or_default(), + memory: memory + .map(validate_memory_quantity) + .transpose()? + .unwrap_or_default(), + gpu: gpu_requirements, + }) + } else { + None + }; + let driver_config = driver_config_json + .map(parse_driver_config_json) + .transpose()?; + let desired_service_level = build_template_service_level(ready_within, max_burst)?; + + let mut client = grpc_client(server, tls).await?; + let response = client + .create_sandbox_template(CreateSandboxTemplateRequest { + template: Some(SandboxWorkloadTemplate { + metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { + id: String::new(), + name: name.to_string(), + created_at_ms: 0, + labels, + resource_version: 0, + annotations, + workspace: workspace.to_string(), + deletion_timestamp_ms: 0, + }), + spec: Some(SandboxWorkloadTemplateSpec { + workload: Some(SandboxWorkloadConfig { + image: image.unwrap_or_default().to_string(), + environment, + resources, + }), + driver_config, + desired_service_level, + }), + }), + workspace_scope: Some(openshell_core::proto::workspace_selector(workspace)), + }) + .await + .into_diagnostic()?; + + let template = response + .into_inner() + .template + .ok_or_else(|| miette!("sandbox template missing from response"))?; + if crate::output::print_output_single(output, &template, sandbox_template_to_json)? { + return Ok(()); + } + println!( + "{} Created sandbox template {}", + "✓".green().bold(), + template.object_name().bold() + ); + Ok(()) +} + +fn build_template_service_level( + ready_within: Option<&str>, + max_burst: Option, +) -> Result> { + if ready_within.is_none() && max_burst.is_none() { + return Ok(None); + } + let ready_within = ready_within + .map(parse_duration_to_ms) + .transpose()? + .map(|ms| { + if ms <= 0 { + Err(miette!("--ready-within must be greater than zero")) + } else { + Ok(duration_ms_to_proto(ms)) + } + }) + .transpose()?; + Ok(Some(SandboxServiceLevel { + startup: Some(SandboxStartup { + ready_within, + max_burst: max_burst.unwrap_or_default(), + }), + })) +} + +fn duration_ms_to_proto(ms: i64) -> prost_types::Duration { + prost_types::Duration { + seconds: ms / 1_000, + nanos: i32::try_from((ms % 1_000) * 1_000_000) + .expect("duration millisecond remainder fits in protobuf nanos"), + } +} + +pub async fn sandbox_template_get( + server: &str, + name: &str, + output: &str, workspace: &str, tls: &TlsOptions, ) -> Result<()> { let mut client = grpc_client(server, tls).await?; let response = client - .list_sandbox_providers(ListSandboxProvidersRequest { - sandbox_name: name.to_string(), - workspace: workspace.to_string(), + .get_sandbox_template(GetSandboxTemplateRequest { + name: name.to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector(workspace)), }) .await .into_diagnostic()?; - let providers = response.into_inner().providers; + let template = response + .into_inner() + .template + .ok_or_else(|| miette!("sandbox template missing from response"))?; - if providers.is_empty() { - println!("No providers attached to sandbox {name}."); + if crate::output::print_output_single(output, &template, sandbox_template_to_json)? { return Ok(()); } - print_provider_attachment_table(&providers); + print_sandbox_template_detail(&template); Ok(()) } -pub async fn sandbox_provider_attach( +#[allow(clippy::too_many_arguments)] +pub async fn sandbox_template_list( server: &str, - name: &str, - provider: &str, + limit: u32, + offset: u32, + label_selector: Option<&str>, + names_only: bool, + output: &str, + workspace: &str, + all_workspaces: bool, + tls: &TlsOptions, +) -> Result<()> { + let mut client = grpc_client(server, tls).await?; + let response = client + .list_sandbox_templates(ListSandboxTemplatesRequest { + limit, + offset, + workspace_scope: Some(if all_workspaces { + openshell_core::proto::all_workspaces_selector() + } else { + openshell_core::proto::workspace_selector(workspace) + }), + label_selector: label_selector.unwrap_or_default().to_string(), + }) + .await + .into_diagnostic()?; + let templates = response.into_inner().templates; + + if crate::output::print_output_collection(output, &templates, sandbox_template_to_json)? { + return Ok(()); + } + + if templates.is_empty() { + if !names_only { + println!("No sandbox templates found."); + } + return Ok(()); + } + + if names_only { + for template in &templates { + if all_workspaces { + println!("{}/{}", template.object_workspace(), template.object_name()); + } else { + println!("{}", template.object_name()); + } + } + return Ok(()); + } + + print_sandbox_template_table(&templates, all_workspaces); + Ok(()) +} + +pub async fn sandbox_template_delete( + server: &str, + names: &[String], workspace: &str, tls: &TlsOptions, ) -> Result<()> { let mut client = grpc_client(server, tls).await?; - - // Fetch current sandbox to get resource_version for CAS - let sandbox = client - .get_sandbox(GetSandboxRequest { - name: name.to_string(), - workspace: workspace.to_string(), - }) - .await - .into_diagnostic()? - .into_inner() - .sandbox - .ok_or_else(|| miette::miette!("sandbox not found"))?; - - let resource_version = sandbox.metadata.as_ref().map_or(0, |m| m.resource_version); - - let response = match client - .attach_sandbox_provider(AttachSandboxProviderRequest { - sandbox_name: name.to_string(), - provider_name: provider.to_string(), - expected_resource_version: resource_version, - workspace: workspace.to_string(), - }) - .await - { - Ok(response) => response.into_inner(), - Err(status) if status.code() == Code::Aborted => { - return Err(miette::miette!( - "Failed to attach provider: sandbox was modified by another operation.\n\ - Please retry the command." - ) - .with_source_code(status.message().to_string())); + for name in names { + let response = client + .delete_sandbox_template(DeleteSandboxTemplateRequest { + name: name.clone(), + workspace_scope: Some(openshell_core::proto::workspace_selector(workspace)), + }) + .await + .into_diagnostic()?; + if response.into_inner().deleted { + println!("{} Deleted sandbox template {name}", "✓".green().bold()); + } else { + println!("Sandbox template {name} not found."); } - Err(e) => return Err(e).into_diagnostic(), - }; - - if response.attached { - println!( - "{} Attached provider {} to sandbox {}", - "✓".green().bold(), - provider, - name - ); - } else { - println!("Provider {provider} is already attached to sandbox {name}."); } Ok(()) } -pub async fn sandbox_provider_detach( - server: &str, - name: &str, - provider: &str, - workspace: &str, - tls: &TlsOptions, -) -> Result<()> { - let mut client = grpc_client(server, tls).await?; - - // Fetch current sandbox to get resource_version for CAS - let sandbox = client - .get_sandbox(GetSandboxRequest { - name: name.to_string(), - workspace: workspace.to_string(), - }) - .await - .into_diagnostic()? - .into_inner() - .sandbox - .ok_or_else(|| miette::miette!("sandbox not found"))?; - - let resource_version = sandbox.metadata.as_ref().map_or(0, |m| m.resource_version); - - let response = match client - .detach_sandbox_provider(DetachSandboxProviderRequest { - sandbox_name: name.to_string(), - provider_name: provider.to_string(), - expected_resource_version: resource_version, - workspace: workspace.to_string(), - }) - .await - { - Ok(response) => response.into_inner(), - Err(status) if status.code() == Code::Aborted => { - return Err(miette::miette!( - "Failed to detach provider: sandbox was modified by another operation.\n\ - Please retry the command." - ) - .with_source_code(status.message().to_string())); - } - Err(e) => return Err(e).into_diagnostic(), - }; - - if response.detached { - println!( - "{} Detached provider {} from sandbox {}", - "✓".green().bold(), - provider, - name - ); - } else { - println!("Provider {provider} was not attached to sandbox {name}."); - } - Ok(()) -} - -fn print_provider_attachment_table(providers: &[Provider]) { - print!("{}", format_provider_attachment_table(providers, true)); -} - -fn format_provider_attachment_table(providers: &[Provider], color: bool) -> String { - use std::fmt::Write as _; - - let name_width = providers - .iter() - .map(|provider| provider.object_name().len()) - .max() - .unwrap_or(4) - .max(4); - let type_width = providers - .iter() - .map(|provider| provider.r#type.len()) - .max() - .unwrap_or(4) - .max(4); - - let name_header = if color { - "NAME".bold().to_string() - } else { - "NAME".to_string() - }; - let type_header = if color { - "TYPE".bold().to_string() - } else { - "TYPE".to_string() - }; - let credential_keys_header = if color { - "CREDENTIAL_KEYS".bold().to_string() - } else { - "CREDENTIAL_KEYS".to_string() - }; - let config_keys_header = if color { - "CONFIG_KEYS".bold().to_string() - } else { - "CONFIG_KEYS".to_string() - }; - - let mut output = String::new(); - let _ = writeln!( - output, - "{name_header: serde_json::Value { + let mut obj = serde_json::Map::new(); + obj.insert("id".to_string(), serde_json::json!(template.object_id())); + obj.insert( + "name".to_string(), + serde_json::json!(template.object_name()), + ); + obj.insert( + "workspace".to_string(), + serde_json::json!(template.object_workspace()), ); - for provider in providers { - let provider_name = provider.object_name(); - let provider_type = &provider.r#type; - let credential_keys = provider_credential_keys(provider).len(); - let config_keys = provider.config.len(); - let _ = writeln!( - output, - "{provider_name:") ); } - output + if let Some(spec) = &template.spec + && let Some(workload) = &spec.workload + { + println!( + " {} {}", + "Image:".dimmed(), + non_empty_or(&workload.image, "") + ); + println!( + " {} {}", + "Environment:".dimmed(), + workload.environment.len() + ); + if let Some(resources) = &workload.resources { + println!( + " {} {}", + "CPU:".dimmed(), + non_empty_or(&resources.cpu, "") + ); + println!( + " {} {}", + "Memory:".dimmed(), + non_empty_or(&resources.memory, "") + ); + println!( + " {} {}", + "GPU:".dimmed(), + template_resources_gpu_display(resources).unwrap_or_else(|| "".to_string()) + ); + } + } + if let Some(startup) = template_startup(template) { + println!( + " {} {}", + "Ready within:".dimmed(), + startup + .ready_within + .as_ref() + .map_or_else(|| "".to_string(), duration_display) + ); + println!( + " {} {}", + "Max burst:".dimmed(), + if startup.max_burst == 0 { + "".to_string() + } else { + startup.max_burst.to_string() + } + ); + } +} + +fn print_sandbox_template_table(templates: &[SandboxWorkloadTemplate], show_workspace: bool) { + let name_width = templates + .iter() + .map(|template| template.object_name().len()) + .max() + .unwrap_or(4) + .max(4); + let workspace_width = if show_workspace { + templates + .iter() + .map(|template| template.object_workspace().len()) + .max() + .unwrap_or(9) + .max(9) + } else { + 0 + }; + let image_width = templates + .iter() + .map(|template| template_image(template).len()) + .max() + .unwrap_or(5) + .clamp(5, 48); + + if show_workspace { + println!( + "{: String { + template + .spec + .as_ref() + .and_then(|spec| spec.workload.as_ref()) + .map_or_else( + || "".to_string(), + |workload| non_empty_or(&workload.image, "").to_string(), + ) +} + +fn template_resources(template: &SandboxWorkloadTemplate) -> Option<&SandboxResources> { + template + .spec + .as_ref() + .and_then(|spec| spec.workload.as_ref()) + .and_then(|workload| workload.resources.as_ref()) +} + +fn template_resources_gpu_display(resources: &SandboxResources) -> Option { + if let Some(gpu) = &resources.gpu { + return Some( + gpu.count + .map_or_else(|| "default".to_string(), |count| count.to_string()), + ); + } + None +} + +fn template_startup(template: &SandboxWorkloadTemplate) -> Option<&SandboxStartup> { + template + .spec + .as_ref() + .and_then(|spec| spec.desired_service_level.as_ref()) + .and_then(|service_level| service_level.startup.as_ref()) +} + +fn duration_to_ms(duration: &prost_types::Duration) -> i64 { + duration.seconds.saturating_mul(1_000) + i64::from(duration.nanos / 1_000_000) +} + +fn duration_display(duration: &prost_types::Duration) -> String { + let total_ms = duration_to_ms(duration); + if total_ms % 3_600_000 == 0 { + format!("{}h", total_ms / 3_600_000) + } else if total_ms % 60_000 == 0 { + format!("{}m", total_ms / 60_000) + } else if total_ms % 1_000 == 0 { + format!("{}s", total_ms / 1_000) + } else { + format!("{total_ms}ms") + } +} + +fn labels_display(labels: &HashMap) -> String { + let mut pairs = labels + .iter() + .map(|(key, value)| format!("{key}={value}")) + .collect::>(); + pairs.sort(); + pairs.join(", ") } /// Delete a sandbox by name, or all sandboxes when `all` is true. @@ -2461,8 +3034,7 @@ pub async fn sandbox_delete( limit: 1000, offset: 0, label_selector: String::new(), - workspace: workspace.to_string(), - all_workspaces: false, + workspace_scope: Some(openshell_core::proto::workspace_selector(workspace)), }) .await .into_diagnostic()?; @@ -2479,6 +3051,7 @@ pub async fn sandbox_delete( names.to_vec() }; + let mut failures = Vec::new(); for name in &names_to_delete { // Stop any background port forwards for this sandbox before deleting. if let Ok(stopped) = stop_forwards_for_sandbox(name) { @@ -2493,7 +3066,7 @@ pub async fn sandbox_delete( let response = match client .delete_sandbox(DeleteSandboxRequest { name: name.clone(), - workspace: workspace.to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector(workspace)), }) .await { @@ -2503,7 +3076,14 @@ pub async fn sandbox_delete( println!("{} Sandbox {name} already deleted", "✓".green().bold()); continue; } - Err(status) => return Err(status).into_diagnostic(), + Err(status) => { + eprintln!( + "{} Failed to delete sandbox {name}: {status}", + "!".red().bold() + ); + failures.push(name.clone()); + continue; + } }; let deleted = response.into_inner().deleted; @@ -2515,7 +3095,7 @@ pub async fn sandbox_delete( } } - Ok(()) + aggregate_delete_failures("sandbox", &failures) } /// Stop a sandbox while retaining its persistent workspace. @@ -2538,7 +3118,7 @@ pub async fn sandbox_stop( let sandbox = client .stop_sandbox(StopSandboxRequest { name: name.to_string(), - workspace: workspace.to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector(workspace)), }) .await .into_diagnostic()? @@ -2561,7 +3141,7 @@ pub async fn sandbox_start( let sandbox = client .start_sandbox(StartSandboxRequest { name: name.to_string(), - workspace: workspace.to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector(workspace)), }) .await .into_diagnostic()? @@ -2647,314 +3227,6 @@ async fn wait_for_lifecycle_phase( } } -/// Return the provider type inferred from the trailing command, if any. -fn inferred_provider_type(command: &[String]) -> Option { - detect_provider_from_command(command).map(str::to_string) -} - -/// Ensure all required providers exist. -/// -/// `explicit_names` are provider **names** supplied via `--provider`. They are -/// passed through directly; the server validates they exist at sandbox creation. -/// -/// `inferred_types` are provider **types** inferred from the trailing command -/// (e.g. `claude` -> type `"claude-code"`). These are resolved to provider names via -/// a type→name lookup, and missing types may be auto-created interactively. -/// -/// Returns a deduplicated list of provider **names** suitable for -/// `SandboxSpec.providers`. -pub async fn ensure_required_providers( - client: &mut crate::tls::GrpcClient, - explicit_names: &[String], - inferred_types: &[String], - auto_providers_override: Option, - workspace: &str, -) -> Result> { - if explicit_names.is_empty() && inferred_types.is_empty() { - return Ok(Vec::new()); - } - - let mut configured_names: Vec = Vec::new(); - let mut seen_names: HashSet = HashSet::new(); - - // ── Fetch all existing providers ───────────────────────────────────── - // Build both a name set (for explicit --provider lookups) and a - // type-to-name map (for inferred provider resolution). - let mut known_names: HashSet = HashSet::new(); - let mut type_to_name: HashMap = HashMap::new(); - { - let mut offset = 0_u32; - let limit = 100_u32; - loop { - let response = client - .list_providers(ListProvidersRequest { - limit, - offset, - workspace: workspace.to_string(), - all_workspaces: false, - }) - .await - .into_diagnostic()?; - let providers = response.into_inner().providers; - for provider in &providers { - known_names.insert(provider.object_name().to_string()); - if !provider.r#type.is_empty() { - let type_lower = provider.r#type.to_ascii_lowercase(); - type_to_name - .entry(type_lower) - .or_insert_with(|| provider.object_name().to_string()); - } - } - if providers.len() < limit as usize { - break; - } - offset = offset.saturating_add(limit); - } - } - - // ── Explicit provider names ────────────────────────────────────────── - // If the name exists on the server, use it directly. Otherwise, if the - // name matches a known provider type, auto-create a provider of that - // type with the requested name. - for name in explicit_names { - if known_names.contains(name) { - if seen_names.insert(name.clone()) { - configured_names.push(name.clone()); - } - } else if let Some(provider_type) = normalize_provider_type(name) { - auto_create_provider( - client, - provider_type, - Some(name), - auto_providers_override, - &mut seen_names, - &mut configured_names, - workspace, - ) - .await?; - // Record the type mapping so the inferred-types pass below - // doesn't attempt to create a duplicate provider. - type_to_name - .entry(provider_type.to_ascii_lowercase()) - .or_insert_with(|| name.clone()); - } else { - return Err(miette::miette!( - "provider '{name}' not found and '{name}' is not a recognized provider type. \ - Create it first with `openshell provider create --type --name {name}`" - )); - } - } - - // ── Resolve inferred provider types ────────────────────────────────── - if !inferred_types.is_empty() { - // Collect resolved names for types that already have a provider. - for t in inferred_types { - if let Some(name) = type_to_name.get(&t.to_ascii_lowercase()) - && seen_names.insert(name.clone()) - { - configured_names.push(name.clone()); - } - } - - let missing = inferred_types - .iter() - .filter(|t| !type_to_name.contains_key(&t.to_ascii_lowercase())) - .cloned() - .collect::>(); - - for provider_type in missing { - auto_create_provider( - client, - &provider_type, - None, - auto_providers_override, - &mut seen_names, - &mut configured_names, - workspace, - ) - .await?; - } - } - - Ok(configured_names) -} - -/// Prompt for (or auto-confirm) creation of a provider from local credentials. -/// -/// When `preferred_name` is `Some`, the provider is created with that exact -/// name (used for explicit `--provider ` values). When `None`, the name -/// defaults to the type and retries with suffixes on conflict (used for -/// inferred provider types). -async fn auto_create_provider( - client: &mut crate::tls::GrpcClient, - provider_type: &str, - preferred_name: Option<&str>, - auto_providers_override: Option, - seen_names: &mut HashSet, - configured_names: &mut Vec, - workspace: &str, -) -> Result<()> { - eprintln!("Missing provider: {provider_type}"); - - // --no-auto-providers: skip silently. - if auto_providers_override == Some(false) { - eprintln!( - "{} Skipping provider '{provider_type}' (--no-auto-providers)", - "!".yellow(), - ); - eprintln!(); - return Ok(()); - } - - // No override and non-interactive: error. - if auto_providers_override.is_none() && !std::io::stdin().is_terminal() { - return Err(miette::miette!( - "missing required provider '{provider_type}'. Create it first with \ - `openshell provider create --type {provider_type} --name {provider_type} --from-existing`, \ - pass --auto-providers to auto-create, or set it up manually from inside the sandbox" - )); - } - - // --auto-providers: auto-confirm; otherwise prompt. - let should_create = if auto_providers_override == Some(true) { - true - } else { - Confirm::new() - .with_prompt("Create from local credentials?") - .default(true) - .interact() - .into_diagnostic()? - }; - - if !should_create { - eprintln!("{} Skipping provider '{provider_type}'", "!".yellow()); - eprintln!(); - return Ok(()); - } - - let discovered = discover_existing_provider_data(client, provider_type, workspace) - .await - .map_err(|err| miette::miette!("failed to discover provider '{provider_type}': {err}"))?; - let Some(discovered) = discovered else { - eprintln!( - "{} No existing local credentials/config found for '{}'. You can configure it from inside the sandbox.", - "!".yellow(), - provider_type - ); - eprintln!(); - return Ok(()); - }; - - if let Some(exact_name) = preferred_name { - // Explicit name: create with exactly that name, no retries. - let request = CreateProviderRequest { - provider: Some(Provider { - metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { - id: String::new(), - name: exact_name.to_string(), - created_at_ms: 0, - labels: HashMap::new(), - resource_version: 0, - annotations: HashMap::new(), - workspace: workspace.to_string(), - deletion_timestamp_ms: 0, - }), - r#type: provider_type.to_string(), - credentials: discovered.credentials.clone(), - config: discovered.config.clone(), - credential_expires_at_ms: HashMap::new(), - profile_workspace: workspace.to_string(), - credential_handles: HashMap::new(), - }), - workspace: workspace.to_string(), - }; - - let response = client.create_provider(request).await.map_err(|status| { - miette::miette!("failed to create provider '{exact_name}': {status}") - })?; - let provider = response - .into_inner() - .provider - .ok_or_else(|| miette::miette!("provider missing from response"))?; - eprintln!( - "{} Created provider {} ({}) from existing local state", - "✓".green().bold(), - provider.object_name(), - provider.r#type - ); - if seen_names.insert(provider.object_name().to_string()) { - configured_names.push(provider.object_name().to_string()); - } - } else { - // Inferred type: try type as name, then suffixed variants. - let mut created = false; - for attempt in 0..5 { - let name = if attempt == 0 { - provider_type.to_string() - } else { - format!("{provider_type}-{attempt}") - }; - - let request = CreateProviderRequest { - provider: Some(Provider { - metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { - id: String::new(), - name: name.clone(), - created_at_ms: 0, - labels: HashMap::new(), - resource_version: 0, - annotations: HashMap::new(), - workspace: workspace.to_string(), - deletion_timestamp_ms: 0, - }), - r#type: provider_type.to_string(), - credentials: discovered.credentials.clone(), - config: discovered.config.clone(), - credential_expires_at_ms: HashMap::new(), - profile_workspace: workspace.to_string(), - credential_handles: HashMap::new(), - }), - workspace: workspace.to_string(), - }; - - match client.create_provider(request).await { - Ok(response) => { - let provider = response - .into_inner() - .provider - .ok_or_else(|| miette::miette!("provider missing from response"))?; - eprintln!( - "{} Created provider {} ({}) from existing local state", - "✓".green().bold(), - provider.object_name(), - provider.r#type - ); - if seen_names.insert(provider.object_name().to_string()) { - configured_names.push(provider.object_name().to_string()); - } - created = true; - break; - } - Err(status) if status.code() == Code::AlreadyExists => {} - Err(status) => { - return Err(miette::miette!( - "failed to create provider for type '{provider_type}': {status}" - )); - } - } - } - - if !created { - return Err(miette::miette!( - "failed to create provider for type '{provider_type}' after name retries" - )); - } - } - - eprintln!(); - Ok(()) -} - pub async fn service_expose( server: &str, sandbox: &str, @@ -2970,7 +3242,7 @@ pub async fn service_expose( service: service.to_string(), target_port: u32::from(target_port), domain: true, - workspace: workspace.to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector(workspace)), }) .await .map_err(service_expose_status_error)? @@ -3003,6 +3275,7 @@ fn service_expose_status_error(status: Status) -> miette::Report { service_status_error("expose service", "sandbox:write", status) } +#[allow(clippy::too_many_arguments)] // user-facing CLI command pub async fn service_list( server: &str, sandbox: Option<&str>, @@ -3010,6 +3283,7 @@ pub async fn service_list( offset: u32, workspace: &str, all_workspaces: bool, + output: &str, tls: &TlsOptions, ) -> Result<()> { let mut client = grpc_client(server, tls).await?; @@ -3018,17 +3292,25 @@ pub async fn service_list( sandbox: sandbox.unwrap_or_default().to_string(), limit, offset, - workspace: if all_workspaces { - String::new() + workspace_scope: Some(if all_workspaces { + openshell_core::proto::all_workspaces_selector() } else { - workspace.to_string() - }, - all_workspaces, + openshell_core::proto::workspace_selector(workspace) + }), }) .await .map_err(|status| service_status_error("list services", "sandbox:read", status))? .into_inner(); + let services = response + .services + .iter() + .filter_map(|response| service_endpoint_to_json(response, server)) + .collect::>(); + if crate::output::print_output_collection(output, &services, Clone::clone)? { + return Ok(()); + } + if response.services.is_empty() { if let Some(sandbox) = sandbox { println!("No services exposed for sandbox {sandbox}."); @@ -3054,7 +3336,7 @@ pub async fn service_get( .get_service(GetServiceRequest { sandbox: sandbox.to_string(), service: service.to_string(), - workspace: workspace.to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector(workspace)), }) .await .map_err(|status| service_status_error("get service", "sandbox:read", status))? @@ -3076,7 +3358,7 @@ pub async fn service_delete( .delete_service(DeleteServiceRequest { sandbox: sandbox.to_string(), service: service.to_string(), - workspace: workspace.to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector(workspace)), }) .await .map_err(|status| service_status_error("delete service", "sandbox:write", status))? @@ -3217,6 +3499,30 @@ fn print_service_endpoint_table( } } +fn service_endpoint_to_json( + response: &ServiceEndpointResponse, + gateway_endpoint: &str, +) -> Option { + let endpoint = response.endpoint.as_ref()?; + let workspace = endpoint + .metadata + .as_ref() + .map_or("", |metadata| metadata.workspace.as_str()); + let url = if response.url.is_empty() { + String::new() + } else { + service_url_for_gateway(&response.url, gateway_endpoint) + }; + + Some(serde_json::json!({ + "workspace": workspace, + "sandbox": endpoint.sandbox_name, + "service": endpoint.service_name, + "target_port": endpoint.target_port, + "url": url, + })) +} + fn service_display_name(service: &str) -> &str { if service.is_empty() { "-" } else { service } } @@ -3228,126 +3534,6 @@ fn service_display_name(service: &str) -> &str { /// Checks `GOOGLE_APPLICATION_CREDENTIALS` first; falls back to /// `$CLOUDSDK_CONFIG/application_default_credentials.json` when set, then to /// `~/.config/gcloud/application_default_credentials.json`. -fn read_gcloud_adc() -> Result<(String, String, String)> { - let path = if let Some(env_path) = std::env::var("GOOGLE_APPLICATION_CREDENTIALS") - .ok() - .filter(|v| !v.is_empty()) - { - PathBuf::from(env_path) - } else if let Some(config_dir) = std::env::var("CLOUDSDK_CONFIG") - .ok() - .filter(|v| !v.is_empty()) - { - PathBuf::from(config_dir).join("application_default_credentials.json") - } else { - let home = std::env::var("HOME") - .map_err(|_| miette::miette!("HOME is not set; cannot locate gcloud ADC file"))?; - PathBuf::from(home) - .join(".config") - .join("gcloud") - .join("application_default_credentials.json") - }; - - let content = std::fs::read_to_string(&path).map_err(|err| { - miette::miette!( - "failed to read gcloud ADC file at {}: {}. \ - Run: gcloud auth application-default login", - path.display(), - err - ) - })?; - - let json: serde_json::Value = serde_json::from_str(&content) - .map_err(|err| miette::miette!("failed to parse gcloud ADC file: {err}"))?; - - let cred_type = json.get("type").and_then(|v| v.as_str()); - match cred_type { - Some("service_account") => { - return Err(miette::miette!( - "Application Default Credentials are a service account key, not user credentials. \ - To use a service account, create the provider with the service account JSON key \ - and configure gateway-managed refresh for 'GOOGLE_VERTEX_AI_SERVICE_ACCOUNT_TOKEN'. \ - See: openshell provider create --help" - )); - } - Some("authorized_user") => {} - Some(other) => { - return Err(miette::miette!( - "Application Default Credentials have unsupported type '{other}' \ - (expected 'authorized_user'). \ - Run: gcloud auth application-default login" - )); - } - None => { - return Err(miette::miette!( - "gcloud ADC file is missing the 'type' field. \ - The file may be malformed. \ - Run: gcloud auth application-default login" - )); - } - } - - let client_id = json - .get("client_id") - .and_then(|v| v.as_str()) - .filter(|s| !s.is_empty()) - .ok_or_else(|| miette::miette!("gcloud ADC file is missing 'client_id'"))? - .to_string(); - - let client_secret = json - .get("client_secret") - .and_then(|v| v.as_str()) - .filter(|s| !s.is_empty()) - .ok_or_else(|| miette::miette!("gcloud ADC file is missing 'client_secret'"))? - .to_string(); - - let refresh_token = json - .get("refresh_token") - .and_then(|v| v.as_str()) - .filter(|s| !s.is_empty()) - .ok_or_else(|| miette::miette!("gcloud ADC file is missing 'refresh_token'"))? - .to_string(); - - Ok((client_id, client_secret, refresh_token)) -} - -async fn rollback_provider_create_after_gcloud_adc_failure( - client: &mut crate::tls::GrpcClient, - provider_name: &str, - stage: &str, - source: &Status, - workspace: &str, -) -> Result<()> { - match client - .delete_provider(DeleteProviderRequest { - name: provider_name.to_string(), - workspace: workspace.to_string(), - }) - .await - { - Ok(_) => Err(miette!( - "failed to {stage} credentials from gcloud ADC for provider '{provider_name}': {source}. \ - The provider was rolled back successfully." - )), - Err(cleanup_err) => { - eprintln!( - "{} Failed to clean up provider '{}' after {} failed: {}. \ - Run 'openshell provider delete {}' to remove it manually.", - "⚠".yellow(), - provider_name, - stage, - cleanup_err, - provider_name - ); - Err(miette!( - "failed to {stage} credentials from gcloud ADC for provider '{provider_name}': {source}. \ - Cleanup also failed, so the provider may still exist. \ - Run 'openshell provider delete {provider_name}' to remove it manually." - )) - } - } -} - fn service_url_for_gateway(service_url: &str, gateway_endpoint: &str) -> String { let (Ok(mut service_url), Ok(gateway_endpoint)) = ( url::Url::parse(service_url), @@ -3366,1690 +3552,6 @@ fn service_url_for_gateway(service_url: &str, gateway_endpoint: &str) -> String service_url.to_string() } -async fn gateway_providers_v2_enabled(client: &mut crate::tls::GrpcClient) -> Result { - let response = client - .get_gateway_config(GetGatewayConfigRequest {}) - .await - .into_diagnostic()? - .into_inner(); - let Some(setting) = response.settings.get(settings::PROVIDERS_V2_ENABLED_KEY) else { - return Ok(false); - }; - match setting.value.as_ref() { - Some(setting_value::Value::BoolValue(enabled)) => Ok(*enabled), - None => Ok(false), - Some(_) => Err(miette::miette!( - "gateway setting '{}' has invalid value type; expected bool", - settings::PROVIDERS_V2_ENABLED_KEY - )), - } -} - -async fn fetch_provider_profile( - client: &mut crate::tls::GrpcClient, - provider_type: &str, - workspace: &str, -) -> Result { - let response = client - .get_provider_profile(GetProviderProfileRequest { - id: provider_type.to_string(), - workspace: workspace.to_string(), - }) - .await - .map_err(|status| { - if status.code() == Code::NotFound { - miette::miette!( - "provider profile '{provider_type}' not found; providers v2 discovery requires a provider profile" - ) - } else { - miette::miette!(status.to_string()) - } - })?; - - response - .into_inner() - .profile - .ok_or_else(|| miette::miette!("provider profile '{provider_type}' missing from response")) -} - -async fn discover_existing_provider_data( - client: &mut crate::tls::GrpcClient, - provider_type: &str, - workspace: &str, -) -> Result> { - if gateway_providers_v2_enabled(client).await? { - let profile = fetch_provider_profile(client, provider_type, workspace).await?; - let profile = ProviderTypeProfile::from_proto(&profile); - let mut discovered = - discover_from_profile(&profile, &RealDiscoveryContext).map_err(|err| { - miette::miette!("failed to discover existing provider data from profile: {err}") - })?; - - // Vertex AI config keys (project ID, region, base URL, publisher) are not - // declared in the profile's discovery.credentials list, so discover_from_profile - // does not scan them. Scan them directly here so --from-existing captures them. - if provider_type == VERTEX_AI_PROVIDER_TYPE { - let discovered = discovered.get_or_insert_with(Default::default); - for key in openshell_core::inference::VERTEX_AI_CONFIG_KEY_NAMES { - if let Ok(val) = std::env::var(key) { - let val = val.trim().to_string(); - if !val.is_empty() { - discovered.config.entry(key.to_string()).or_insert(val); - } - } - } - } - - Ok(discovered) - } else { - let registry = ProviderRegistry::new(); - registry - .discover_existing(provider_type) - .map_err(|err| miette::miette!("failed to discover existing provider data: {err}")) - } -} - -/// Canonical provider type string for Google Vertex AI. -const VERTEX_AI_PROVIDER_TYPE: &str = "google-vertex-ai"; - -/// Canonical provider type string for Google Cloud (GCP APIs). -const GOOGLE_CLOUD_PROVIDER_TYPE: &str = "google-cloud"; - -fn missing_credentials_error(provider_type: &str) -> miette::Report { - if provider_type == VERTEX_AI_PROVIDER_TYPE { - return miette::miette!( - "no credentials resolved for provider type '{provider_type}'. \ - Set GOOGLE_VERTEX_AI_TOKEN, VERTEX_AI_TOKEN, \ - GOOGLE_VERTEX_AI_SERVICE_ACCOUNT_TOKEN, or VERTEX_AI_SERVICE_ACCOUNT_TOKEN; \ - or use --from-gcloud-adc or --from-existing with those env vars set." - ); - } - - if provider_type == GOOGLE_CLOUD_PROVIDER_TYPE { - return miette::miette!( - "no credentials resolved for provider type '{provider_type}'. \ - Set GCP_ADC_ACCESS_TOKEN or GCP_SA_ACCESS_TOKEN; \ - or use --from-gcloud-adc / --from-existing with those env vars set." - ); - } - - miette::miette!( - "no credentials resolved for provider type '{provider_type}'. \ - Use --credential KEY[=VALUE], --runtime-credentials for runtime-resolved profile credentials, or --from-existing \ - with the appropriate env vars set." - ) -} - -async fn provider_credential_from_oidc_token( - credentials: &[String], - profile: Option<&ProviderProfile>, - tls: &TlsOptions, -) -> Result<(HashMap, HashMap)> { - let credential_key = oidc_subject_credential_key(credentials, profile)?; - - let gateway_name = tls.gateway_name().ok_or_else(|| { - miette::miette!("--from-oidc-token requires an active named OIDC gateway") - })?; - let bundle = - crate::oidc_auth::ensure_valid_oidc_token_bundle(gateway_name, tls.gateway_insecure) - .await - .map_err(|err| { - miette::miette!( - "failed to load or refresh OIDC token for gateway '{gateway_name}' while preparing provider credential: {err}" - ) - })?; - - let mut credential_map = HashMap::new(); - credential_map.insert(credential_key.clone(), bundle.access_token); - - let mut credential_expires_at_ms = HashMap::new(); - if let Some(expires_at) = bundle.expires_at { - let expires_at_ms = i64::try_from(expires_at) - .unwrap_or(i64::MAX / 1000) - .saturating_mul(1000); - credential_expires_at_ms.insert(credential_key, expires_at_ms); - } - - Ok((credential_map, credential_expires_at_ms)) -} - -fn oidc_subject_credential_key( - credentials: &[String], - profile: Option<&ProviderProfile>, -) -> Result { - if credentials.len() > 1 { - return Err(miette::miette!( - "--from-oidc-token accepts at most one --credential KEY destination" - )); - } - - if let Some(credential) = credentials.first() { - let credential = credential.trim(); - if credential.is_empty() || credential.contains('=') { - return Err(miette::miette!( - "--from-oidc-token requires --credential KEY without an inline value" - )); - } - if let Some(profile) = profile { - ensure_profile_declares_subject_credential(profile, credential)?; - } - return Ok(credential.to_string()); - } - - let Some(profile) = profile else { - return Err(miette::miette!( - "--from-oidc-token requires --credential KEY when the provider profile is unavailable" - )); - }; - - infer_oidc_subject_credential_from_profile(profile) -} - -fn ensure_profile_declares_subject_credential( - profile: &ProviderProfile, - credential: &str, -) -> Result<()> { - let matches = token_exchange_subject_credentials(profile); - if matches.iter().any(|candidate| candidate == credential) { - return Ok(()); - } - Err(miette::miette!( - "credential '{credential}' is not declared as a token-exchange subject credential in provider profile '{}'; expected one of: {}", - profile.id, - matches.join(", ") - )) -} - -fn infer_oidc_subject_credential_from_profile(profile: &ProviderProfile) -> Result { - let matches = token_exchange_subject_credentials(profile); - match matches.as_slice() { - [credential] => Ok(credential.clone()), - [] => Err(miette::miette!( - "provider profile '{}' does not declare a token-exchange subject credential; pass --credential KEY", - profile.id - )), - _ => Err(miette::miette!( - "provider profile '{}' declares multiple token-exchange subject credentials ({}); pass --credential KEY", - profile.id, - matches.join(", ") - )), - } -} - -fn token_exchange_subject_credentials(profile: &ProviderProfile) -> Vec { - let mut matches = Vec::new(); - for credential in &profile.credentials { - let Some(token_grant) = credential.token_grant.as_ref() else { - continue; - }; - if ProviderCredentialTokenGrantType::try_from(token_grant.grant_type).ok() - != Some(ProviderCredentialTokenGrantType::TokenExchange) - { - continue; - } - let Some(subject_token) = token_grant.subject_token.as_ref() else { - continue; - }; - if subject_token.source != "provider_credential" || subject_token.credential.is_empty() { - continue; - } - if !matches.contains(&subject_token.credential) { - matches.push(subject_token.credential.clone()); - } - } - matches -} - -#[allow(clippy::too_many_arguments)] -pub async fn provider_create( - server: &str, - name: &str, - provider_type: &str, - from_existing: bool, - credentials: &[String], - from_gcloud_adc: bool, - config: &[String], - workspace: &str, - tls: &TlsOptions, -) -> Result<()> { - let credential_source = match (from_existing, from_gcloud_adc) { - (true, true) => { - return Err(miette::miette!( - "--from-gcloud-adc cannot be combined with --from-existing, --from-oidc-token, or --credential; it also cannot be combined with --runtime-credentials" - )); - } - (true, false) => ProviderCreateCredentialSource::Existing, - (false, true) => ProviderCreateCredentialSource::GcloudAdc, - (false, false) => ProviderCreateCredentialSource::ExplicitCredentials, - }; - provider_create_with_options(ProviderCreateOptions { - server, - name, - provider_type, - credentials, - credential_source, - config, - workspace, - profile_workspace: workspace, - tls, - }) - .await -} - -pub struct ProviderCreateOptions<'a> { - pub server: &'a str, - pub name: &'a str, - pub provider_type: &'a str, - pub credentials: &'a [String], - pub credential_source: ProviderCreateCredentialSource, - pub config: &'a [String], - pub workspace: &'a str, - pub profile_workspace: &'a str, - pub tls: &'a TlsOptions, -} - -#[derive(Clone, Copy, Debug, Eq, PartialEq)] -pub enum ProviderCreateCredentialSource { - ExplicitCredentials, - Existing, - GcloudAdc, - OidcToken, - Runtime, -} - -pub async fn provider_create_with_options(options: ProviderCreateOptions<'_>) -> Result<()> { - let ProviderCreateOptions { - server, - name, - provider_type, - credentials, - credential_source, - config, - workspace, - profile_workspace, - tls, - } = options; - - let from_existing = credential_source == ProviderCreateCredentialSource::Existing; - let from_gcloud_adc = credential_source == ProviderCreateCredentialSource::GcloudAdc; - let from_oidc_token = credential_source == ProviderCreateCredentialSource::OidcToken; - let runtime_credentials = credential_source == ProviderCreateCredentialSource::Runtime; - - if from_gcloud_adc && !credentials.is_empty() { - return Err(miette::miette!( - "--from-gcloud-adc cannot be combined with --from-existing, --from-oidc-token, or --credential; it also cannot be combined with --runtime-credentials" - )); - } - if from_existing && !credentials.is_empty() { - return Err(miette::miette!( - "--from-existing cannot be combined with --credential" - )); - } - if runtime_credentials && !credentials.is_empty() { - return Err(miette::miette!( - "--runtime-credentials cannot be combined with --credential" - )); - } - - let mut client = grpc_client(server, tls).await?; - - let provider_type = if let Some(provider_type) = normalize_provider_type(provider_type) { - provider_type.to_string() - } else { - let profile_id = provider_type.trim(); - if profile_id.is_empty() { - return Err(miette::miette!("provider type is required")); - } - let response = client - .get_provider_profile(GetProviderProfileRequest { - id: profile_id.to_string(), - workspace: profile_workspace.to_string(), - }) - .await; - match response { - Ok(response) => response - .into_inner() - .profile - .map(|profile| profile.id) - .filter(|id| !id.trim().is_empty()) - .unwrap_or_else(|| profile_id.to_string()), - Err(status) if status.code() == Code::NotFound => { - return Err(miette::miette!( - "unsupported provider type or profile: {provider_type}" - )); - } - Err(status) => return Err(status).into_diagnostic(), - } - }; - - let adc_credential_key = if from_gcloud_adc { - let profile = fetch_provider_profile(&mut client, &provider_type, profile_workspace) - .await - .map_err(|err| { - miette::miette!( - "--from-gcloud-adc is not supported for '{provider_type}' providers ({err})" - ) - })?; - let profile = ProviderTypeProfile::from_proto(&profile); - let adc_cred = profile.adc_credential().ok_or_else(|| { - miette::miette!( - "--from-gcloud-adc is not supported for '{provider_type}' providers \ - (no ADC-compatible credential in the provider profile)" - ) - })?; - Some( - adc_cred - .env_vars - .first() - .ok_or_else(|| { - miette::miette!( - "ADC credential in '{provider_type}' profile has no env_vars declared" - ) - })? - .clone(), - ) - } else { - None - }; - - let oidc_profile = if from_oidc_token { - Some(fetch_provider_profile(&mut client, &provider_type, profile_workspace).await?) - } else { - None - }; - - let (mut credential_map, oidc_credential_expires_at_ms) = if from_oidc_token { - provider_credential_from_oidc_token(credentials, oidc_profile.as_ref(), tls).await? - } else { - (parse_credential_pairs(credentials)?, HashMap::new()) - }; - let mut config_map = parse_key_value_pairs(config, "--config")?; - - if from_existing { - let discovered = - discover_existing_provider_data(&mut client, &provider_type, profile_workspace).await?; - let Some(discovered) = discovered else { - return Err(miette::miette!( - "no existing local credentials/config found for provider type '{provider_type}'" - )); - }; - - for (key, value) in discovered.credentials { - credential_map.entry(key).or_insert(value); - } - for (key, value) in discovered.config { - config_map.entry(key).or_insert(value); - } - } - - if credential_map.is_empty() { - if from_existing { - return Err(missing_credentials_error(&provider_type)); - } - if !from_gcloud_adc && !runtime_credentials { - return Err(missing_credentials_error(&provider_type)); - } - let allows_empty_credentials = if runtime_credentials { - provider_profile_allows_empty_credentials( - &fetch_provider_profile(&mut client, &provider_type, profile_workspace).await?, - ) - } else { - fetch_provider_profile(&mut client, &provider_type, profile_workspace) - .await - .ok() - .is_some_and(|profile| provider_profile_allows_empty_credentials(&profile)) - }; - if !allows_empty_credentials { - if runtime_credentials { - return Err(miette::miette!( - "--runtime-credentials is only valid for provider profiles whose required credentials are resolved at runtime" - )); - } - return Err(missing_credentials_error(&provider_type)); - } - } - - // Validate and read the ADC file BEFORE creating the provider so that - // a bad/missing ADC does not leave an orphan provider behind. Bundle the - // credential key with the material so they stay coupled. - let gcloud_adc_bootstrap = if from_gcloud_adc { - let (client_id, client_secret, refresh_token) = read_gcloud_adc()?; - let key = adc_credential_key.expect("set when from_gcloud_adc is true"); - Some((key, client_id, client_secret, refresh_token)) - } else { - None - }; - - let response = client - .create_provider(CreateProviderRequest { - provider: Some(Provider { - metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { - id: String::new(), - name: name.to_string(), - created_at_ms: 0, - labels: HashMap::new(), - resource_version: 0, - annotations: HashMap::new(), - workspace: workspace.to_string(), - deletion_timestamp_ms: 0, - }), - r#type: provider_type.clone(), - credentials: credential_map, - config: config_map, - credential_expires_at_ms: oidc_credential_expires_at_ms, - profile_workspace: profile_workspace.to_string(), - credential_handles: HashMap::new(), - }), - workspace: workspace.to_string(), - }) - .await - .into_diagnostic()?; - - let provider = response - .into_inner() - .provider - .ok_or_else(|| miette::miette!("provider missing from response"))?; - let provider_name = provider.object_name().to_string(); - - if let Some((adc_credential_key, client_id, client_secret, refresh_token)) = - gcloud_adc_bootstrap - { - let mut material = HashMap::new(); - material.insert("client_id".to_string(), client_id); - material.insert("client_secret".to_string(), client_secret); - material.insert("refresh_token".to_string(), refresh_token); - - if let Err(configure_err) = client - .configure_provider_refresh(ConfigureProviderRefreshRequest { - provider: provider_name.clone(), - credential_key: adc_credential_key.clone(), - strategy: ProviderCredentialRefreshStrategy::Oauth2RefreshToken as i32, - material, - secret_material_keys: vec![ - "client_secret".to_string(), - "refresh_token".to_string(), - ], - expires_at_ms: None, - workspace: workspace.to_string(), - }) - .await - { - return rollback_provider_create_after_gcloud_adc_failure( - &mut client, - &provider_name, - "configure", - &configure_err, - workspace, - ) - .await; - } - - if let Err(rotate_err) = client - .rotate_provider_credential(RotateProviderCredentialRequest { - provider: provider_name.clone(), - credential_key: adc_credential_key, - workspace: workspace.to_string(), - }) - .await - { - return rollback_provider_create_after_gcloud_adc_failure( - &mut client, - &provider_name, - "mint the initial access token for", - &rotate_err, - workspace, - ) - .await; - } - - println!("{} Created provider {}", "✓".green().bold(), provider_name); - println!("Configured GCP credentials from gcloud ADC and minted the initial access token"); - return Ok(()); - } - - println!("{} Created provider {}", "✓".green().bold(), provider_name); - Ok(()) -} - -fn provider_profile_allows_empty_credentials(profile: &ProviderProfile) -> bool { - ProviderTypeProfile::from_proto(profile).allows_empty_provider_credentials() -} - -pub async fn provider_get( - server: &str, - name: &str, - workspace: &str, - tls: &TlsOptions, -) -> Result<()> { - let mut client = grpc_client(server, tls).await?; - let response = client - .get_provider(GetProviderRequest { - name: name.to_string(), - workspace: workspace.to_string(), - }) - .await - .into_diagnostic()?; - - let provider = response - .into_inner() - .provider - .ok_or_else(|| miette::miette!("provider missing from response"))?; - - let credential_keys = provider_credential_keys(&provider); - let config_keys = provider.config.keys().cloned().collect::>(); - - println!("{}", "Provider:".cyan().bold()); - println!(); - println!(" {} {}", "Id:".dimmed(), provider.object_id()); - println!(" {} {}", "Name:".dimmed(), provider.object_name()); - println!(" {} {}", "Type:".dimmed(), provider.r#type); - println!( - " {} {}", - "Resource version:".dimmed(), - provider.metadata.as_ref().map_or(0, |m| m.resource_version) - ); - println!( - " {} {}", - "Credential keys:".dimmed(), - if credential_keys.is_empty() { - "".to_string() - } else { - credential_keys.join(", ") - } - ); - println!( - " {} {}", - "Config keys:".dimmed(), - if config_keys.is_empty() { - "".to_string() - } else { - config_keys.join(", ") - } - ); - - Ok(()) -} - -fn provider_to_json(provider: &Provider) -> serde_json::Value { - let mut obj = serde_json::Map::new(); - - // Core fields - obj.insert("id".to_string(), serde_json::json!(provider.object_id())); - obj.insert( - "name".to_string(), - serde_json::json!(provider.object_name()), - ); - obj.insert( - "workspace".to_string(), - serde_json::json!(provider.object_workspace()), - ); - obj.insert("type".to_string(), serde_json::json!(provider.r#type)); - - // Credential keys (NEVER values - security) - let credential_keys = provider_credential_keys(provider); - obj.insert( - "credential_keys".to_string(), - serde_json::json!(credential_keys), - ); - - // Config keys (keys only, not values) - if !provider.config.is_empty() { - let config_keys: Vec = provider.config.keys().cloned().collect(); - obj.insert("config_keys".to_string(), serde_json::json!(config_keys)); - } - - // Metadata fields (only if metadata exists) - if let Some(meta) = &provider.metadata { - if !meta.labels.is_empty() { - obj.insert("labels".to_string(), serde_json::json!(meta.labels)); - } - if meta.resource_version != 0 { - obj.insert( - "resource_version".to_string(), - serde_json::json!(meta.resource_version), - ); - } - if meta.created_at_ms != 0 { - obj.insert( - "created_at".to_string(), - serde_json::json!(format_epoch_ms(meta.created_at_ms)), - ); - } - } - - // Credential expiration times (only if present) - if !provider.credential_expires_at_ms.is_empty() { - obj.insert( - "credential_expires_at_ms".to_string(), - serde_json::json!(provider.credential_expires_at_ms), - ); - } - - serde_json::Value::Object(obj) -} - -fn provider_credential_keys(provider: &Provider) -> Vec { - let mut keys: Vec = provider - .credentials - .keys() - .chain(provider.credential_handles.keys()) - .cloned() - .collect(); - keys.sort(); - keys.dedup(); - keys -} - -#[allow(clippy::too_many_arguments)] -pub async fn provider_list( - server: &str, - limit: u32, - offset: u32, - names_only: bool, - output: &str, - workspace: &str, - all_workspaces: bool, - tls: &TlsOptions, -) -> Result<()> { - let mut client = grpc_client(server, tls).await?; - let response = client - .list_providers(ListProvidersRequest { - limit, - offset, - workspace: if all_workspaces { - String::new() - } else { - workspace.to_string() - }, - all_workspaces, - }) - .await - .into_diagnostic()?; - let providers = response.into_inner().providers; - - // Handle structured output formats (json, yaml) - if crate::output::print_output_collection(output, &providers, provider_to_json)? { - return Ok(()); - } - - if providers.is_empty() { - if !names_only { - println!("No providers found."); - } - return Ok(()); - } - - if names_only { - for provider in &providers { - if all_workspaces { - println!("{}/{}", provider.object_workspace(), provider.object_name()); - } else { - println!("{}", provider.object_name()); - } - } - return Ok(()); - } - - let ws_width = if all_workspaces { - providers - .iter() - .map(|p| p.object_workspace().len()) - .max() - .unwrap_or(9) - .max(9) - } else { - 0 - }; - let name_width = providers - .iter() - .map(|provider| provider.object_name().len()) - .max() - .unwrap_or(4) - .max(4); - let type_width = providers - .iter() - .map(|provider| provider.r#type.len()) - .max() - .unwrap_or(4) - .max(4); - - if all_workspaces { - println!( - "{: Result<()> { - let mut client = grpc_client(server, tls).await?; - let response = client - .list_provider_profiles(ListProviderProfilesRequest { - limit: 100, - offset: 0, - workspace: workspace.to_string(), - }) - .await - .into_diagnostic()?; - let mut profiles = response.into_inner().profiles; - profiles.sort_by(|left, right| { - left.category - .cmp(&right.category) - .then_with(|| left.id.cmp(&right.id)) - }); - let dto_profiles = profiles - .iter() - .map(ProviderTypeProfile::from_proto) - .collect::>(); - - if crate::output::print_output_direct( - output, - || profiles_to_json(&dto_profiles).into_diagnostic(), - || profiles_to_yaml(&dto_profiles).into_diagnostic(), - )? { - return Ok(()); - } - - if profiles.is_empty() { - println!("No provider profiles found."); - return Ok(()); - } - - println!("{}", "Available Provider Profiles:".cyan().bold()); - let id_width = provider_profile_id_width(&profiles); - let display_width = provider_profile_display_width(&profiles); - let source_width = provider_profile_source_width(&profiles); - let scope_width = provider_profile_scope_width(&profiles); - let mut current_category = i32::MIN; - for profile in &profiles { - if profile.category != current_category { - current_category = profile.category; - println!(); - println!(" {}", display_provider_category(current_category).bold()); - print_provider_type_header(id_width, scope_width, source_width, display_width); - } - print_provider_type_row(profile, id_width, scope_width, source_width, display_width); - } - - Ok(()) -} - -pub async fn provider_profile_export( - server: &str, - id: &str, - output: &str, - workspace: &str, - tls: &TlsOptions, -) -> Result<()> { - let rendered = provider_profile_export_text(server, id, output, workspace, tls).await?; - if output == "json" { - println!("{rendered}"); - } else { - print!("{rendered}"); - } - Ok(()) -} - -pub async fn provider_profile_export_text( - server: &str, - id: &str, - output: &str, - workspace: &str, - tls: &TlsOptions, -) -> Result { - let mut client = grpc_client(server, tls).await?; - let response = client - .get_provider_profile(GetProviderProfileRequest { - id: id.to_string(), - workspace: workspace.to_string(), - }) - .await - .into_diagnostic()?; - let profile = response - .into_inner() - .profile - .ok_or_else(|| miette!("provider profile '{id}' not found"))?; - let profile = ProviderTypeProfile::from_proto(&profile); - - match output { - "json" => profile_to_json(&profile).into_diagnostic(), - "yaml" => profile_to_yaml(&profile).into_diagnostic(), - "table" => Err(miette!( - "profile export supports '-o yaml' and '-o json'; table output is not supported" - )), - _ => Err(miette!("unsupported output format: {output}")), - } -} - -pub async fn provider_profile_import( - server: &str, - file: Option<&Path>, - from: Option<&Path>, - workspace: &str, - tls: &TlsOptions, -) -> Result<()> { - let (items, mut diagnostics) = load_profile_import_items(file, from)?; - if items.is_empty() && diagnostics.is_empty() { - return Err(miette!("no provider profile files found")); - } - if profile_diagnostics_have_errors(&diagnostics) { - print_profile_diagnostics(&diagnostics); - return Err(miette!("provider profile import failed")); - } - - let mut client = grpc_client(server, tls).await?; - if !items.is_empty() { - let response = client - .import_provider_profiles(ImportProviderProfilesRequest { - profiles: items, - workspace: workspace.to_string(), - }) - .await - .into_diagnostic()? - .into_inner(); - diagnostics.extend(response.diagnostics); - if response.imported { - println!( - "Imported {} provider profile{}.", - response.profiles.len(), - if response.profiles.len() == 1 { - "" - } else { - "s" - } - ); - return Ok(()); - } - } - - print_profile_diagnostics(&diagnostics); - Err(miette!("provider profile import failed")) -} - -pub async fn provider_profile_update( - server: &str, - id: &str, - file: &Path, - workspace: &str, - tls: &TlsOptions, -) -> Result<()> { - let (mut items, mut diagnostics) = load_profile_import_items(Some(file), None)?; - if items.is_empty() && diagnostics.is_empty() { - return Err(miette!("no provider profile files found")); - } - if profile_diagnostics_have_errors(&diagnostics) { - print_profile_diagnostics(&diagnostics); - return Err(miette!("provider profile update failed")); - } - - let mut client = grpc_client(server, tls).await?; - if let Some(item) = items.pop() { - let expected_resource_version = item - .profile - .as_ref() - .map_or(0, |profile| profile.resource_version); - let response = client - .update_provider_profiles(UpdateProviderProfilesRequest { - profile: Some(item), - expected_resource_version, - id: id.to_string(), - workspace: workspace.to_string(), - }) - .await - .into_diagnostic()? - .into_inner(); - diagnostics.extend(response.diagnostics); - if response.updated { - println!("Updated provider profile."); - return Ok(()); - } - } - - print_profile_diagnostics(&diagnostics); - Err(miette!("provider profile update failed")) -} - -pub async fn provider_profile_lint( - server: &str, - file: Option<&Path>, - from: Option<&Path>, - workspace: &str, - tls: &TlsOptions, -) -> Result<()> { - let (items, mut diagnostics) = load_profile_import_items(file, from)?; - if items.is_empty() && diagnostics.is_empty() { - return Err(miette!("no provider profile files found")); - } - - if !items.is_empty() { - let mut client = grpc_client(server, tls).await?; - let response = client - .lint_provider_profiles(LintProviderProfilesRequest { - profiles: items, - workspace: workspace.to_string(), - }) - .await - .into_diagnostic()? - .into_inner(); - diagnostics.extend(response.diagnostics); - } - - if profile_diagnostics_have_errors(&diagnostics) { - print_profile_diagnostics(&diagnostics); - return Err(miette!("provider profile lint failed")); - } - - println!("Provider profile lint passed."); - Ok(()) -} - -pub async fn provider_profile_delete( - server: &str, - id: &str, - workspace: &str, - tls: &TlsOptions, -) -> Result<()> { - let mut client = grpc_client(server, tls).await?; - let response = client - .delete_provider_profile(DeleteProviderProfileRequest { - id: id.to_string(), - workspace: workspace.to_string(), - }) - .await - .into_diagnostic()? - .into_inner(); - if response.deleted { - println!("Deleted provider profile '{id}'."); - } else { - println!("Provider profile '{id}' was not deleted."); - } - Ok(()) -} - -pub async fn provider_refresh_status( - server: &str, - name: &str, - credential_key: Option<&str>, - workspace: &str, - tls: &TlsOptions, -) -> Result<()> { - let mut client = grpc_client(server, tls).await?; - let response = client - .get_provider_refresh_status(GetProviderRefreshStatusRequest { - provider: name.to_string(), - credential_key: credential_key.unwrap_or_default().to_string(), - workspace: workspace.to_string(), - }) - .await - .into_diagnostic()? - .into_inner(); - - if response.credentials.is_empty() { - if let Some(credential_key) = credential_key { - println!( - "No refresh configuration found for provider '{name}' credential '{credential_key}'." - ); - } else { - println!("No refresh configurations found for provider '{name}'."); - } - return Ok(()); - } - - println!("{}", refresh_status_header()); - for status in response.credentials { - print_refresh_status_row(&status); - } - Ok(()) -} - -fn refresh_status_header() -> String { - format!( - "{:<24} {:<28} {:<28} {:<24} {:<18} {:<20} {:<20} {:<20} {:<44} {}", - "PROVIDER".bold(), - "CREDENTIAL_KEY".bold(), - "STRATEGY".bold(), - "STATUS".bold(), - "RECOVERY".bold(), - "EXPIRES_AT".bold(), - "NEXT_REFRESH".bold(), - "LAST_REFRESH".bold(), - "FAILURE_CODE".bold(), - "LAST_ERROR".bold(), - ) -} - -pub struct ProviderRefreshConfigInput<'a> { - pub name: &'a str, - pub credential_key: &'a str, - pub strategy: &'a str, - pub material: &'a [String], - pub secret_material_env: &'a [String], - pub secret_material_keys: &'a [String], - pub credential_expires_at_ms: Option, -} - -pub async fn provider_refresh_config( - server: &str, - input: ProviderRefreshConfigInput<'_>, - workspace: &str, - tls: &TlsOptions, -) -> Result<()> { - let strategy = provider_refresh_strategy(input.strategy)?; - let mut material = parse_key_value_pairs(input.material, "--material")?; - let mut secret_material_keys = input.secret_material_keys.to_vec(); - // Env-resolved secrets are auto-marked secret; duplicate keys are an - // error rather than a precedence order. - for (key, value) in parse_secret_material_env_pairs(input.secret_material_env)? { - if material.contains_key(&key) { - return Err(miette!( - "duplicate material key '{key}': supplied via both --material and --secret-material-env" - )); - } - if !secret_material_keys.contains(&key) { - secret_material_keys.push(key.clone()); - } - material.insert(key, value); - } - let mut client = grpc_client(server, tls).await?; - let status = client - .configure_provider_refresh(ConfigureProviderRefreshRequest { - provider: input.name.to_string(), - credential_key: input.credential_key.to_string(), - strategy: strategy as i32, - material, - secret_material_keys, - expires_at_ms: input.credential_expires_at_ms, - workspace: workspace.to_string(), - }) - .await - .into_diagnostic()? - .into_inner() - .status - .ok_or_else(|| miette!("provider refresh status missing from response"))?; - - println!( - "{} Configured refresh for {} {}", - "✓".green().bold(), - status.provider_name, - status.credential_key - ); - Ok(()) -} - -pub async fn provider_rotate( - server: &str, - name: &str, - credential_key: &str, - workspace: &str, - tls: &TlsOptions, -) -> Result<()> { - let mut client = grpc_client(server, tls).await?; - let status = client - .rotate_provider_credential(RotateProviderCredentialRequest { - provider: name.to_string(), - credential_key: credential_key.to_string(), - workspace: workspace.to_string(), - }) - .await - .into_diagnostic()? - .into_inner() - .status - .ok_or_else(|| miette!("provider refresh status missing from response"))?; - - if status.last_error.is_empty() { - println!( - "{} Rotation requested for {} {} ({})", - "✓".green().bold(), - status.provider_name, - status.credential_key, - status.status - ); - } else { - println!( - "Rotation request recorded for {} {} ({}): {}", - status.provider_name, status.credential_key, status.status, status.last_error - ); - } - Ok(()) -} - -pub async fn provider_refresh_delete( - server: &str, - name: &str, - credential_key: &str, - workspace: &str, - tls: &TlsOptions, -) -> Result<()> { - let mut client = grpc_client(server, tls).await?; - let response = client - .delete_provider_refresh(DeleteProviderRefreshRequest { - provider: name.to_string(), - credential_key: credential_key.to_string(), - workspace: workspace.to_string(), - }) - .await - .into_diagnostic()? - .into_inner(); - - if response.deleted { - println!( - "{} Deleted refresh config for {} {}", - "✓".green().bold(), - name, - credential_key - ); - } else { - println!("No refresh config found for provider '{name}' credential '{credential_key}'."); - } - Ok(()) -} - -fn provider_refresh_strategy(strategy: &str) -> Result { - match strategy { - "oauth2_refresh_token" => Ok(ProviderCredentialRefreshStrategy::Oauth2RefreshToken), - "oauth2_client_credentials" => { - Ok(ProviderCredentialRefreshStrategy::Oauth2ClientCredentials) - } - "google_service_account_jwt" => { - Ok(ProviderCredentialRefreshStrategy::GoogleServiceAccountJwt) - } - "aws_sts_assume_role" => Ok(ProviderCredentialRefreshStrategy::AwsStsAssumeRole), - _ => Err(miette!("unsupported provider refresh strategy: {strategy}")), - } -} - -fn print_refresh_status_row(status: &ProviderCredentialRefreshStatus) { - println!("{}", refresh_status_row(status)); -} - -fn refresh_status_row(status: &ProviderCredentialRefreshStatus) -> String { - let strategy = ProviderCredentialRefreshStrategy::try_from(status.strategy) - .unwrap_or(ProviderCredentialRefreshStrategy::Unspecified); - let recovery_action = ProviderCredentialRefreshRecoveryAction::try_from(status.recovery_action) - .unwrap_or(ProviderCredentialRefreshRecoveryAction::Unspecified); - format!( - "{:<24} {:<28} {:<28} {:<24} {:<18} {:<20} {:<20} {:<20} {:<44} {}", - status.provider_name, - status.credential_key, - provider_refresh_strategy_name(strategy), - status.status, - provider_refresh_recovery_action_name(recovery_action), - format_optional_epoch_ms(status.expires_at_ms), - format_refresh_next_at_ms(status.next_refresh_at_ms), - format_optional_epoch_ms(status.last_refresh_at_ms), - status.failure_code, - truncate_status_field(&status.last_error, 72), - ) -} - -fn format_refresh_next_at_ms(next_refresh_at_ms: i64) -> String { - if next_refresh_at_ms == i64::MAX { - "-".to_string() - } else { - format_optional_epoch_ms(next_refresh_at_ms) - } -} - -fn provider_refresh_recovery_action_name( - action: ProviderCredentialRefreshRecoveryAction, -) -> &'static str { - match action { - ProviderCredentialRefreshRecoveryAction::Retry => "retry", - ProviderCredentialRefreshRecoveryAction::Reauthorize => "reauthorize", - ProviderCredentialRefreshRecoveryAction::FixConfiguration => "fix_configuration", - ProviderCredentialRefreshRecoveryAction::Investigate => "investigate", - ProviderCredentialRefreshRecoveryAction::Unspecified => "-", - } -} - -fn provider_refresh_strategy_name(strategy: ProviderCredentialRefreshStrategy) -> &'static str { - match strategy { - ProviderCredentialRefreshStrategy::Static => "static", - ProviderCredentialRefreshStrategy::External => "external", - ProviderCredentialRefreshStrategy::Oauth2RefreshToken => "oauth2_refresh_token", - ProviderCredentialRefreshStrategy::Oauth2ClientCredentials => "oauth2_client_credentials", - ProviderCredentialRefreshStrategy::GoogleServiceAccountJwt => "google_service_account_jwt", - ProviderCredentialRefreshStrategy::AwsStsAssumeRole => "aws_sts_assume_role", - ProviderCredentialRefreshStrategy::Unspecified => "unspecified", - } -} - -fn load_profile_import_items( - file: Option<&Path>, - from: Option<&Path>, -) -> Result<( - Vec, - Vec, -)> { - let paths = profile_source_paths(file, from)?; - let mut items = Vec::new(); - let mut diagnostics = Vec::new(); - for path in paths { - match load_profile_import_item(&path) { - Ok(item) => items.push(item), - Err(diagnostic) => diagnostics.push(diagnostic), - } - } - Ok((items, diagnostics)) -} - -fn profile_source_paths(file: Option<&Path>, from: Option<&Path>) -> Result> { - if let Some(file) = file { - return Ok(vec![file.to_path_buf()]); - } - let Some(from) = from else { - return Ok(Vec::new()); - }; - let mut paths = Vec::new(); - for entry in std::fs::read_dir(from) - .into_diagnostic() - .wrap_err_with(|| format!("failed to read profile directory {}", from.display()))? - { - let entry = entry.into_diagnostic()?; - let path = entry.path(); - if path.is_file() && profile_extension_supported(&path) { - paths.push(path); - } - } - paths.sort(); - Ok(paths) -} - -fn profile_extension_supported(path: &Path) -> bool { - matches!( - path.extension().and_then(|ext| ext.to_str()), - Some("yaml" | "yml" | "json") - ) -} - -fn load_profile_import_item( - path: &Path, -) -> Result { - let source = path.display().to_string(); - let input = std::fs::read_to_string(path).map_err(|err| { - profile_file_diagnostic( - &source, - format!("failed to read provider profile file: {err}"), - ) - })?; - let profile = match path.extension().and_then(|ext| ext.to_str()) { - Some("yaml" | "yml") => parse_profile_yaml(&input), - Some("json") => parse_profile_json(&input), - _ => { - return Err(profile_file_diagnostic( - &source, - "unsupported provider profile file format".to_string(), - )); - } - } - .map_err(|err| profile_file_diagnostic(&source, err.to_string()))?; - - let pre_lower = profile.validate_before_lowering(&source); - if let Some(diag) = pre_lower.into_iter().find(|d| d.severity == "error") { - return Err(ProviderProfileDiagnostic { - source: diag.source, - profile_id: diag.profile_id, - field: diag.field, - message: diag.message, - severity: diag.severity, - }); - } - - Ok(ProviderProfileImportItem { - profile: Some(profile.to_proto()), - source, - }) -} - -fn profile_file_diagnostic(source: &str, message: String) -> ProviderProfileDiagnostic { - ProviderProfileDiagnostic { - source: source.to_string(), - profile_id: String::new(), - field: "file".to_string(), - message, - severity: "error".to_string(), - } -} - -fn print_profile_diagnostics(diagnostics: &[ProviderProfileDiagnostic]) { - if diagnostics.is_empty() { - return; - } - eprintln!("{}", "Provider profile diagnostics:".red().bold()); - for diagnostic in diagnostics { - let source = if diagnostic.source.is_empty() { - "" - } else { - &diagnostic.source - }; - let profile = if diagnostic.profile_id.is_empty() { - "-".to_string() - } else { - diagnostic.profile_id.clone() - }; - eprintln!( - " {} {} profile={} field={} {}", - diagnostic.severity.as_str().red(), - source, - profile, - diagnostic.field, - diagnostic.message - ); - } -} - -fn profile_diagnostics_have_errors(diagnostics: &[ProviderProfileDiagnostic]) -> bool { - diagnostics - .iter() - .any(|diagnostic| diagnostic.severity == "error") -} - -fn display_provider_category(category: i32) -> &'static str { - match ProviderProfileCategory::try_from(category).unwrap_or(ProviderProfileCategory::Other) { - ProviderProfileCategory::Inference => "INFERENCE", - ProviderProfileCategory::Agent => "AGENT", - ProviderProfileCategory::SourceControl => "SOURCE CONTROL", - ProviderProfileCategory::Messaging => "MESSAGING", - ProviderProfileCategory::Data => "DATA", - ProviderProfileCategory::Knowledge => "KNOWLEDGE", - ProviderProfileCategory::Other | ProviderProfileCategory::Unspecified => "OTHER", - } -} - -const PROVIDER_PROFILE_ID_MAX_WIDTH: usize = 32; -const PROVIDER_PROFILE_DISPLAY_MAX_WIDTH: usize = 40; -const PROVIDER_PROFILE_SOURCE_MAX_WIDTH: usize = 24; - -fn provider_profile_id_width(profiles: &[ProviderProfile]) -> usize { - profiles - .iter() - .map(|profile| { - profile - .id - .chars() - .count() - .min(PROVIDER_PROFILE_ID_MAX_WIDTH) - }) - .max() - .unwrap_or(2) - .max(2) -} - -fn provider_profile_display_width(profiles: &[ProviderProfile]) -> usize { - profiles - .iter() - .map(|profile| { - profile - .display_name - .chars() - .count() - .min(PROVIDER_PROFILE_DISPLAY_MAX_WIDTH) - }) - .max() - .unwrap_or(4) - .max(4) -} - -fn provider_profile_scope_width(profiles: &[ProviderProfile]) -> usize { - profiles - .iter() - .map(|profile| profile.scope.chars().count()) - .max() - .unwrap_or(5) - .max(5) -} - -fn provider_profile_source_width(profiles: &[ProviderProfile]) -> usize { - profiles - .iter() - .map(|profile| { - profile - .source - .chars() - .count() - .min(PROVIDER_PROFILE_SOURCE_MAX_WIDTH) - }) - .max() - .unwrap_or(6) - .max(6) -} - -fn print_provider_type_header( - id_width: usize, - scope_width: usize, - source_width: usize, - display_width: usize, -) { - let endpoints = "ENDPOINTS"; - println!( - " {: { - pub server: &'a str, - pub name: &'a str, - pub from_existing: bool, - pub from_oidc_token: bool, - pub credentials: &'a [String], - pub config: &'a [String], - pub credential_expires_at: &'a [String], - pub workspace: &'a str, - pub tls: &'a TlsOptions, -} - -pub async fn provider_update(options: ProviderUpdateOptions<'_>) -> Result<()> { - let ProviderUpdateOptions { - server, - name, - from_existing, - from_oidc_token, - credentials, - config, - credential_expires_at, - workspace, - tls, - } = options; - - if from_existing && !credentials.is_empty() { - return Err(miette::miette!( - "--from-existing cannot be combined with --credential" - )); - } - if from_existing && from_oidc_token { - return Err(miette::miette!( - "--from-existing cannot be combined with --from-oidc-token" - )); - } - - let mut client = grpc_client(server, tls).await?; - let oidc_profile = if from_oidc_token { - let existing = client - .get_provider(GetProviderRequest { - name: name.to_string(), - workspace: workspace.to_string(), - }) - .await - .into_diagnostic()? - .into_inner() - .provider - .ok_or_else(|| miette::miette!("provider '{name}' not found"))?; - let profile_workspace = if existing.profile_workspace.is_empty() { - workspace - } else { - &existing.profile_workspace - }; - Some(fetch_provider_profile(&mut client, &existing.r#type, profile_workspace).await?) - } else { - None - }; - - let (mut credential_map, oidc_credential_expires_at_ms) = if from_oidc_token { - provider_credential_from_oidc_token(credentials, oidc_profile.as_ref(), tls).await? - } else { - (parse_credential_pairs(credentials)?, HashMap::new()) - }; - let mut config_map = parse_key_value_pairs(config, "--config")?; - let mut credential_expires_at_ms = parse_credential_expiry_pairs(credential_expires_at)?; - credential_expires_at_ms.extend(oidc_credential_expires_at_ms); - - if from_existing { - // Fetch the existing provider to discover its type for credential lookup. - let existing = client - .get_provider(GetProviderRequest { - name: name.to_string(), - workspace: workspace.to_string(), - }) - .await - .into_diagnostic()? - .into_inner() - .provider - .ok_or_else(|| miette::miette!("provider '{name}' not found"))?; - - let provider_type = existing.r#type; - let discovered = - discover_existing_provider_data(&mut client, &provider_type, workspace).await?; - let Some(discovered) = discovered else { - return Err(miette::miette!( - "no existing local credentials/config found for provider type '{provider_type}'" - )); - }; - - for (key, value) in discovered.credentials { - credential_map.entry(key).or_insert(value); - } - for (key, value) in discovered.config { - config_map.entry(key).or_insert(value); - } - } - - let response = client - .update_provider(UpdateProviderRequest { - provider: Some(Provider { - metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { - id: String::new(), - name: name.to_string(), - created_at_ms: 0, - labels: HashMap::new(), - resource_version: 0, - annotations: HashMap::new(), - workspace: workspace.to_string(), - deletion_timestamp_ms: 0, - }), - r#type: String::new(), - credentials: credential_map, - config: config_map, - credential_expires_at_ms: HashMap::new(), - profile_workspace: String::new(), - credential_handles: HashMap::new(), - }), - credential_expires_at_ms, - workspace: workspace.to_string(), - }) - .await - .into_diagnostic()?; - - let provider = response - .into_inner() - .provider - .ok_or_else(|| miette::miette!("provider missing from response"))?; - - println!( - "{} Updated provider {}", - "✓".green().bold(), - provider.object_name() - ); - Ok(()) -} - -pub async fn provider_delete( - server: &str, - names: &[String], - workspace: &str, - tls: &TlsOptions, -) -> Result<()> { - let mut client = grpc_client(server, tls).await?; - for name in names { - let response = client - .delete_provider(DeleteProviderRequest { - name: name.clone(), - workspace: workspace.to_string(), - }) - .await - .into_diagnostic()?; - if response.into_inner().deleted { - println!("{} Deleted provider {name}", "✓".green().bold()); - } else { - println!("{} Provider {name} not found", "!".yellow()); - } - } - Ok(()) -} - // --------------------------------------------------------------------------- // Workspace commands // --------------------------------------------------------------------------- @@ -5317,9 +3819,10 @@ pub async fn workspace_member_list( workspace: &str, limit: u32, offset: u32, + output: &str, tls: &TlsOptions, ) -> Result<()> { - use openshell_core::proto::{ListWorkspaceMembersRequest, WorkspaceRole}; + use openshell_core::proto::ListWorkspaceMembersRequest; let mut client = grpc_client(server, tls).await?; let response = client @@ -5332,6 +3835,10 @@ pub async fn workspace_member_list( .into_diagnostic()?; let members = response.into_inner().members; + if crate::output::print_output_collection(output, &members, workspace_member_to_json)? { + return Ok(()); + } + if members.is_empty() { println!("No members found in workspace {workspace}."); return Ok(()); @@ -5347,17 +3854,30 @@ pub async fn workspace_member_list( println!("{: "admin", - Ok(WorkspaceRole::User) => "user", - _ => "unknown", - }; + let role_str = workspace_member_role_name(member.role); println!("{: &'static str { + use openshell_core::proto::WorkspaceRole; + + match WorkspaceRole::try_from(role) { + Ok(WorkspaceRole::Admin) => "admin", + Ok(WorkspaceRole::User) => "user", + _ => "unknown", + } +} + +fn workspace_member_to_json(member: &openshell_core::proto::WorkspaceMember) -> serde_json::Value { + serde_json::json!({ + "subject": member.principal_subject, + "role": workspace_member_role_name(member.role), + }) +} + fn workspace_phase_str(workspace: &openshell_core::proto::Workspace) -> &'static str { use openshell_core::proto::datamodel::v1::WorkspacePhase; let phase = workspace @@ -5406,283 +3926,6 @@ fn workspace_to_json(workspace: &openshell_core::proto::Workspace) -> serde_json serde_json::Value::Object(obj) } -#[allow(clippy::too_many_arguments)] -pub async fn gateway_inference_set( - server: &str, - provider_name: &str, - model_id: &str, - route_name: &str, - no_verify: bool, - timeout_secs: u64, - workspace: &str, - tls: &TlsOptions, -) -> Result<()> { - let progress = if std::io::stdout().is_terminal() { - let spinner = ProgressBar::new_spinner(); - spinner.set_style( - ProgressStyle::with_template("{spinner:.cyan} {msg} ({elapsed})") - .unwrap_or_else(|_| ProgressStyle::default_spinner()), - ); - spinner.set_message("Configuring inference..."); - spinner.enable_steady_tick(Duration::from_millis(120)); - Some(spinner) - } else { - None - }; - - let mut client = grpc_inference_client(server, tls).await?; - let response = client - .set_inference_route(SetInferenceRouteRequest { - provider_name: provider_name.to_string(), - model_id: model_id.to_string(), - route_name: route_name.to_string(), - verify: false, - no_verify, - timeout_secs, - workspace: workspace.to_string(), - }) - .await; - - if let Some(progress) = &progress { - progress.finish_and_clear(); - } - - let response = response.map_err(format_inference_status)?; - - let configured = response.into_inner(); - let label = if configured.route_name == "sandbox-system" { - "System inference configured:" - } else { - "Inference configured:" - }; - println!("{}", label.cyan().bold()); - println!(); - println!(" {} {}", "Workspace:".dimmed(), configured.workspace); - println!(" {} {}", "Route:".dimmed(), configured.route_name); - println!(" {} {}", "Provider:".dimmed(), configured.provider_name); - println!(" {} {}", "Model:".dimmed(), configured.model_id); - println!(" {} {}", "Version:".dimmed(), configured.version); - print_timeout(configured.timeout_secs); - if configured.validation_performed { - println!(" {}", "Validated Endpoints:".dimmed()); - for endpoint in configured.validated_endpoints { - println!(" - {} ({})", endpoint.url, endpoint.protocol); - } - } - Ok(()) -} - -#[allow(clippy::too_many_arguments)] -pub async fn gateway_inference_update( - server: &str, - provider_name: Option<&str>, - model_id: Option<&str>, - route_name: &str, - no_verify: bool, - timeout_secs: Option, - workspace: &str, - tls: &TlsOptions, -) -> Result<()> { - if provider_name.is_none() && model_id.is_none() && timeout_secs.is_none() { - return Err(miette::miette!( - "at least one of --provider, --model, or --timeout must be specified" - )); - } - - let mut client = grpc_inference_client(server, tls).await?; - - // Fetch current config to use as base for the partial update. - let current = client - .get_inference_route(GetInferenceRouteRequest { - route_name: route_name.to_string(), - workspace: workspace.to_string(), - }) - .await - .into_diagnostic()? - .into_inner(); - - let provider = provider_name.unwrap_or(¤t.provider_name); - let model = model_id.unwrap_or(¤t.model_id); - let timeout = timeout_secs.unwrap_or(current.timeout_secs); - - let progress = if std::io::stdout().is_terminal() { - let spinner = ProgressBar::new_spinner(); - spinner.set_style( - ProgressStyle::with_template("{spinner:.cyan} {msg} ({elapsed})") - .unwrap_or_else(|_| ProgressStyle::default_spinner()), - ); - spinner.set_message("Configuring inference..."); - spinner.enable_steady_tick(Duration::from_millis(120)); - Some(spinner) - } else { - None - }; - - let response = client - .set_inference_route(SetInferenceRouteRequest { - provider_name: provider.to_string(), - model_id: model.to_string(), - route_name: route_name.to_string(), - verify: false, - no_verify, - timeout_secs: timeout, - workspace: workspace.to_string(), - }) - .await; - - if let Some(progress) = &progress { - progress.finish_and_clear(); - } - - let response = response.map_err(format_inference_status)?; - - let configured = response.into_inner(); - let label = if configured.route_name == "sandbox-system" { - "System inference updated:" - } else { - "Inference updated:" - }; - println!("{}", label.cyan().bold()); - println!(); - println!(" {} {}", "Workspace:".dimmed(), configured.workspace); - println!(" {} {}", "Route:".dimmed(), configured.route_name); - println!(" {} {}", "Provider:".dimmed(), configured.provider_name); - println!(" {} {}", "Model:".dimmed(), configured.model_id); - println!(" {} {}", "Version:".dimmed(), configured.version); - print_timeout(configured.timeout_secs); - if configured.validation_performed { - println!(" {}", "Validated Endpoints:".dimmed()); - for endpoint in configured.validated_endpoints { - println!(" - {} ({})", endpoint.url, endpoint.protocol); - } - } - Ok(()) -} - -pub async fn gateway_inference_get( - server: &str, - route_name: Option<&str>, - workspace: &str, - tls: &TlsOptions, -) -> Result<()> { - let mut client = grpc_inference_client(server, tls).await?; - - if let Some(name) = route_name { - // Show a single route (--system was specified). - let response = client - .get_inference_route(GetInferenceRouteRequest { - route_name: name.to_string(), - workspace: workspace.to_string(), - }) - .await - .into_diagnostic()?; - - let configured = response.into_inner(); - let label = if name == "sandbox-system" { - "System inference:" - } else { - "Inference:" - }; - println!("{}", label.cyan().bold()); - println!(); - println!(" {} {}", "Workspace:".dimmed(), configured.workspace); - println!(" {} {}", "Provider:".dimmed(), configured.provider_name); - println!(" {} {}", "Model:".dimmed(), configured.model_id); - println!(" {} {}", "Version:".dimmed(), configured.version); - print_timeout(configured.timeout_secs); - } else { - // Show both routes by default. - print_inference_route(&mut client, "Inference", "", workspace).await; - println!(); - print_inference_route(&mut client, "System inference", "sandbox-system", workspace).await; - } - Ok(()) -} - -pub async fn gateway_inference_delete( - server: &str, - route_name: &str, - workspace: &str, - tls: &TlsOptions, -) -> Result<()> { - let mut client = grpc_inference_client(server, tls).await?; - - let response = client - .delete_inference_route(DeleteInferenceRouteRequest { - route_name: route_name.to_string(), - workspace: workspace.to_string(), - }) - .await - .into_diagnostic()?; - - let label = if route_name == "sandbox-system" { - "System inference route" - } else { - "Inference route" - }; - - if response.into_inner().deleted { - println!("{label} deleted."); - } else { - println!("{label} not found (already deleted)."); - } - Ok(()) -} - -async fn print_inference_route( - client: &mut crate::tls::GrpcInferenceClient, - label: &str, - route_name: &str, - workspace: &str, -) { - match client - .get_inference_route(GetInferenceRouteRequest { - route_name: route_name.to_string(), - workspace: workspace.to_string(), - }) - .await - { - Ok(response) => { - let configured = response.into_inner(); - println!("{}", format!("{label}:").cyan().bold()); - println!(); - println!(" {} {}", "Workspace:".dimmed(), configured.workspace); - println!(" {} {}", "Provider:".dimmed(), configured.provider_name); - println!(" {} {}", "Model:".dimmed(), configured.model_id); - println!(" {} {}", "Version:".dimmed(), configured.version); - print_timeout(configured.timeout_secs); - } - Err(e) if e.code() == Code::NotFound => { - println!("{}", format!("{label}:").cyan().bold()); - println!(); - println!(" {}", "Not configured".dimmed()); - } - Err(e) => { - println!("{}", format!("{label}:").cyan().bold()); - println!(); - println!(" {} {}", "Error:".red(), e.message()); - } - } -} - -fn print_timeout(timeout_secs: u64) { - if timeout_secs == 0 { - println!(" {} {}s (default)", "Timeout:".dimmed(), 60); - } else { - println!(" {} {}s", "Timeout:".dimmed(), timeout_secs); - } -} - -fn format_inference_status(status: Status) -> miette::Report { - let message = status.message().trim(); - - if message.is_empty() { - return miette::miette!("inference configuration failed ({})", status.code()); - } - - miette::miette!("{message}") -} - pub fn git_repo_root(local_path: &Path) -> Result { let git_dir = if local_path.is_dir() { local_path @@ -5892,7 +4135,7 @@ pub async fn sandbox_policy_set_global( yes: bool, wait: bool, _timeout_secs: u64, - workspace: &str, + _workspace: &str, tls: &TlsOptions, ) -> Result<()> { if wait { @@ -5912,7 +4155,6 @@ pub async fn sandbox_policy_set_global( name: String::new(), policy: Some(policy), global: true, - workspace: workspace.to_string(), ..Default::default() }) .await @@ -5943,7 +4185,7 @@ pub async fn sandbox_settings_get( let sandbox = client .get_sandbox(GetSandboxRequest { name: name.to_string(), - workspace: workspace.to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector(workspace)), }) .await .into_diagnostic()? @@ -6101,7 +4343,7 @@ pub async fn gateway_setting_set( key: &str, value: &str, yes: bool, - workspace: &str, + _workspace: &str, tls: &TlsOptions, ) -> Result<()> { let setting_value = parse_cli_setting_value(key, value)?; @@ -6114,7 +4356,6 @@ pub async fn gateway_setting_set( setting_key: key.to_string(), setting_value: Some(setting_value), global: true, - workspace: workspace.to_string(), ..Default::default() }) .await @@ -6147,7 +4388,7 @@ pub async fn sandbox_setting_set( name: name.to_string(), setting_key: key.to_string(), setting_value: Some(setting_value), - workspace: workspace.to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector(workspace)), ..Default::default() }) .await @@ -6169,7 +4410,7 @@ pub async fn gateway_setting_delete( server: &str, key: &str, yes: bool, - workspace: &str, + _workspace: &str, tls: &TlsOptions, ) -> Result<()> { confirm_global_setting_delete(key, yes)?; @@ -6181,7 +4422,6 @@ pub async fn gateway_setting_delete( setting_key: key.to_string(), delete_setting: true, global: true, - workspace: workspace.to_string(), ..Default::default() }) .await @@ -6214,7 +4454,7 @@ pub async fn sandbox_setting_delete( name: name.to_string(), setting_key: key.to_string(), delete_setting: true, - workspace: workspace.to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector(workspace)), ..Default::default() }) .await @@ -6260,7 +4500,7 @@ pub async fn sandbox_policy_set( name: name.to_string(), version: 0, global: false, - workspace: workspace.to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector(workspace)), }) .await .ok() @@ -6271,7 +4511,7 @@ pub async fn sandbox_policy_set( .update_config(UpdateConfigRequest { name: name.to_string(), policy: Some(policy), - workspace: workspace.to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector(workspace)), ..Default::default() }) .await @@ -6319,7 +4559,7 @@ pub async fn sandbox_policy_set( name: name.to_string(), version: resp.version, global: false, - workspace: workspace.to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector(workspace)), }) .await .into_diagnostic()?; @@ -6396,7 +4636,7 @@ pub async fn sandbox_policy_update( let sandbox = client .get_sandbox(GetSandboxRequest { name: name.to_string(), - workspace: workspace.to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector(workspace)), }) .await .into_diagnostic()? @@ -6445,7 +4685,7 @@ pub async fn sandbox_policy_update( .update_config(UpdateConfigRequest { name: name.to_string(), merge_operations: plan.merge_operations, - workspace: workspace.to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector(workspace)), ..Default::default() }) .await @@ -6493,7 +4733,7 @@ pub async fn sandbox_policy_update( name: name.to_string(), version: response.version, global: false, - workspace: workspace.to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector(workspace)), }) .await .into_diagnostic()?; @@ -6601,7 +4841,7 @@ where name: name.to_string(), version, global: false, - workspace: workspace.to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector(workspace)), }) .await .into_diagnostic()?; @@ -6683,7 +4923,7 @@ where let sandbox = client .get_sandbox(GetSandboxRequest { name: name.to_string(), - workspace: workspace.to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector(workspace)), }) .await .into_diagnostic()? @@ -6785,7 +5025,7 @@ pub async fn sandbox_policy_get_global( version: u32, view: PolicyGetView, output: &str, - workspace: &str, + _workspace: &str, tls: &TlsOptions, ) -> Result<()> { let mut client = grpc_client(server, tls).await?; @@ -6795,7 +5035,7 @@ pub async fn sandbox_policy_get_global( name: String::new(), version, global: true, - workspace: workspace.to_string(), + workspace_scope: None, }) .await .into_diagnostic()?; @@ -6924,6 +5164,7 @@ pub async fn sandbox_policy_list( server: &str, name: &str, limit: u32, + output: &str, workspace: &str, tls: &TlsOptions, ) -> Result<()> { @@ -6935,12 +5176,17 @@ pub async fn sandbox_policy_list( limit, offset: 0, global: false, - workspace: workspace.to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector(workspace)), }) .await .into_diagnostic()?; let revisions = resp.into_inner().revisions; + let structured = policy_revision_list_json("sandbox", Some(name), &revisions)?; + if crate::output::print_output_collection(output, &structured, Clone::clone)? { + return Ok(()); + } + if revisions.is_empty() { eprintln!("No policy history found for sandbox '{name}'"); return Ok(()); @@ -6953,7 +5199,8 @@ pub async fn sandbox_policy_list( pub async fn sandbox_policy_list_global( server: &str, limit: u32, - workspace: &str, + output: &str, + _workspace: &str, tls: &TlsOptions, ) -> Result<()> { let mut client = grpc_client(server, tls).await?; @@ -6964,12 +5211,17 @@ pub async fn sandbox_policy_list_global( limit, offset: 0, global: true, - workspace: workspace.to_string(), + workspace_scope: None, }) .await .into_diagnostic()?; let revisions = resp.into_inner().revisions; + let structured = policy_revision_list_json("global", None, &revisions)?; + if crate::output::print_output_collection(output, &structured, Clone::clone)? { + return Ok(()); + } + if revisions.is_empty() { eprintln!("No global policy history found"); return Ok(()); @@ -6979,6 +5231,28 @@ pub async fn sandbox_policy_list_global( Ok(()) } +fn policy_revision_list_json( + scope: &str, + sandbox: Option<&str>, + revisions: &[openshell_core::proto::SandboxPolicyRevision], +) -> Result> { + revisions + .iter() + .map(|revision| { + let status = + PolicyStatus::try_from(revision.status).unwrap_or(PolicyStatus::Unspecified); + policy_revision_to_json( + scope, + sandbox, + None, + revision, + status, + PolicyGetView::Metadata, + ) + }) + .collect() +} + fn print_policy_revision_table(revisions: &[openshell_core::proto::SandboxPolicyRevision]) { println!( "{:<8} {:<14} {:<12} {:<24} ERROR", @@ -7029,7 +5303,7 @@ pub async fn sandbox_logs( let sandbox = client .get_sandbox(GetSandboxRequest { name: name.to_string(), - workspace: workspace.to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector(workspace)), }) .await .into_diagnostic()? @@ -7094,7 +5368,7 @@ pub async fn sandbox_logs( since_ms, sources: source_filter, min_level: level.to_uppercase(), - workspace: workspace.to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector(workspace)), }) .await .into_diagnostic()?; @@ -7170,7 +5444,7 @@ pub async fn sandbox_draft_get( .get_draft_policy(GetDraftPolicyRequest { name: name.to_string(), status_filter: status_filter.unwrap_or("").to_string(), - workspace: workspace.to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector(workspace)), }) .await .into_diagnostic()?; @@ -7278,7 +5552,7 @@ pub async fn sandbox_draft_approve( .get_draft_policy(GetDraftPolicyRequest { name: name.to_string(), status_filter: String::new(), - workspace: workspace.to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector(workspace)), }) .await .into_diagnostic()? @@ -7293,7 +5567,7 @@ pub async fn sandbox_draft_approve( .approve_draft_chunk(ApproveDraftChunkRequest { name: name.to_string(), chunk_id: chunk_id.to_string(), - workspace: workspace.to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector(workspace)), review_token, }) .await @@ -7326,7 +5600,7 @@ pub async fn sandbox_draft_reject( name: name.to_string(), chunk_id: chunk_id.to_string(), reason: reason.to_string(), - workspace: workspace.to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector(workspace)), }) .await .into_diagnostic()?; @@ -7349,7 +5623,7 @@ pub async fn sandbox_draft_approve_all( .get_draft_policy(GetDraftPolicyRequest { name: name.to_string(), status_filter: "pending".to_string(), - workspace: workspace.to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector(workspace)), }) .await .into_diagnostic()? @@ -7366,7 +5640,7 @@ pub async fn sandbox_draft_approve_all( .approve_all_draft_chunks(ApproveAllDraftChunksRequest { name: name.to_string(), include_security_flagged, - workspace: workspace.to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector(workspace)), approvals, }) .await @@ -7396,7 +5670,7 @@ pub async fn sandbox_draft_clear( let response = client .clear_draft_chunks(ClearDraftChunksRequest { name: name.to_string(), - workspace: workspace.to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector(workspace)), }) .await .into_diagnostic()?; @@ -7423,7 +5697,7 @@ pub async fn sandbox_draft_history( let response = client .get_draft_history(GetDraftHistoryRequest { name: name.to_string(), - workspace: workspace.to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector(workspace)), }) .await .into_diagnostic()?; @@ -7512,38 +5786,37 @@ fn format_endpoint(endpoint: &openshell_core::proto::NetworkEndpoint) -> String #[cfg(test)] mod tests { use super::{ - PolicyGetView, ProvisioningStep, build_sandbox_resource_limits, - dockerfile_sources_supported_for_gateway, format_endpoint, format_log_line, - format_provider_attachment_table, git_sync_files, has_main_process_result, - inferred_provider_type, parse_cli_setting_value, parse_credential_expiry_cli_value, - parse_credential_expiry_pairs, parse_credential_pairs, parse_driver_config_json, - parse_secret_material_env_pairs, policy_revision_to_json, - provider_profile_allows_empty_credentials, provisioning_timeout_message, - ready_false_condition_message, refresh_status_header, refresh_status_row, resolve_from, - sandbox_should_persist, sandbox_upload_plan, service_expose_status_error, - service_url_for_gateway, + PolicyGetView, ProvisioningStep, build_sandbox_resource_limits, format_endpoint, + format_log_line, git_sync_files, has_main_process_result, parse_cli_setting_value, + parse_credential_expiry_cli_value, parse_driver_config_json, + parse_secret_material_env_pairs, policy_revision_list_json, policy_revision_to_json, + provisioning_timeout_message, ready_false_condition_message, resolve_from, + rootfs_tar_sources_supported_for_gateway, sandbox_should_persist, sandbox_upload_plan, + service_endpoint_to_json, service_expose_status_error, service_url_for_gateway, + workspace_member_to_json, }; use crate::TEST_ENV_LOCK; - use crate::commands::common::progress_step_from_metadata; + use crate::commands::common::{ + parse_credential_expiry_pairs, parse_credential_pairs, progress_step_from_metadata, + }; use crate::test_utils::EnvVarGuard; + use openshell_bootstrap::GatewayMetadata; use std::fs; - use std::io::Write; use std::path::Path; use std::process::Command; use tonic::Status; - use openshell_bootstrap::GatewayMetadata; use openshell_core::progress::{ PROGRESS_STEP_PULLING_IMAGE, PROGRESS_STEP_REQUESTING_SANDBOX, PROGRESS_STEP_STARTING_SANDBOX, }; use openshell_core::proto::{ - GetSandboxConfigResponse, GpuResourceRequirements, PolicySource, PolicyStatus, Provider, - ProviderCredentialRefresh, ProviderCredentialRefreshRecoveryAction, - ProviderCredentialRefreshStatus, ProviderCredentialRefreshStrategy, - ProviderCredentialTokenGrant, ProviderProfile, ProviderProfileCredential, - ResourceRequirements, Sandbox, SandboxCondition, SandboxPhase, SandboxPolicyRevision, - SandboxStatus, datamodel::v1::ObjectMeta, + GetSandboxConfigResponse, GpuResourceRequirements, PolicySource, PolicyStatus, + ResourceRequirements, Sandbox, SandboxCondition, SandboxPhase, SandboxPolicy, + SandboxPolicyRevision, SandboxResources, SandboxStatus, SandboxWorkloadConfig, + SandboxWorkloadTemplate, SandboxWorkloadTemplateProvenance, SandboxWorkloadTemplateSpec, + ServiceEndpoint, ServiceEndpointResponse, WorkspaceMember, WorkspaceRole, + datamodel::v1::ObjectMeta, }; #[test] @@ -7574,6 +5847,112 @@ mod tests { ); } + #[test] + fn policy_list_json_reuses_metadata_contract() { + let load_error = "policy failed after checking café.example/非常に長いパス"; + let revisions = vec![SandboxPolicyRevision { + version: 7, + policy_hash: "0123456789abcdef".to_string(), + status: PolicyStatus::Failed as i32, + load_error: load_error.to_string(), + created_at_ms: 100, + loaded_at_ms: 200, + policy: Some(SandboxPolicy::default()), + provenance: std::collections::HashMap::from([( + "source".to_string(), + "provider-composition".to_string(), + )]), + }]; + + let values = policy_revision_list_json("sandbox", Some("dev"), &revisions) + .expect("policy list JSON"); + + assert_eq!( + values[0], + serde_json::json!({ + "scope": "sandbox", + "sandbox": "dev", + "version": 7, + "hash": "0123456789abcdef", + "status": "failed", + "created_at_ms": 100, + "loaded_at_ms": 200, + "load_error": load_error, + "provenance": {"source": "provider-composition"}, + }) + ); + assert!(values[0].get("policy").is_none()); + assert!(values[0].get("active_version").is_none()); + + let unknown = policy_revision_list_json( + "global", + None, + &[SandboxPolicyRevision { + version: 8, + status: 999, + ..Default::default() + }], + ) + .expect("global policy list JSON"); + assert_eq!(unknown[0]["scope"], "global"); + assert_eq!(unknown[0]["status"], "unspecified"); + assert!(unknown[0].get("sandbox").is_none()); + } + + #[test] + fn service_endpoint_json_has_raw_fields_and_normalized_url() { + let response = ServiceEndpointResponse { + endpoint: Some(ServiceEndpoint { + metadata: Some(ObjectMeta { + workspace: "team-a".to_string(), + ..Default::default() + }), + sandbox_name: "api".to_string(), + service_name: String::new(), + target_port: 8080, + ..Default::default() + }), + url: "https://api.openshell.localhost:3000/".to_string(), + }; + + let value = service_endpoint_to_json(&response, "https://gateway.example:17670") + .expect("service endpoint JSON"); + assert_eq!( + value, + serde_json::json!({ + "workspace": "team-a", + "sandbox": "api", + "service": "", + "target_port": 8080, + "url": "https://api.openshell.localhost:17670/", + }) + ); + assert!(service_endpoint_to_json(&ServiceEndpointResponse::default(), "unused").is_none()); + } + + #[test] + fn workspace_member_json_uses_stable_role_names() { + for (role, expected) in [ + (WorkspaceRole::Admin as i32, "admin"), + (WorkspaceRole::User as i32, "user"), + (999, "unknown"), + ] { + let value = workspace_member_to_json(&WorkspaceMember { + metadata: Some(ObjectMeta { + id: "internal-id".to_string(), + ..Default::default() + }), + principal_subject: "oidc-subject".to_string(), + role, + }); + assert_eq!( + value, + serde_json::json!({"subject": "oidc-subject", "role": expected}) + ); + assert!(!value.to_string().contains("internal-id")); + } + } + #[test] fn parse_credential_pairs_accepts_key_value_form() { let parsed = parse_credential_pairs(&["API_KEY=abc123".to_string()]).expect("parse"); @@ -7723,43 +6102,6 @@ mod tests { assert_eq!(parsed, 1_767_225_600_000); } - #[test] - fn provider_attachment_table_formats_provider_counts() { - let output = format_provider_attachment_table( - &[Provider { - metadata: Some(ObjectMeta { - name: "work-custom".to_string(), - ..Default::default() - }), - r#type: "custom-api".to_string(), - credentials: [ - ("CUSTOM_API_KEY".to_string(), "REDACTED".to_string()), - ("CUSTOM_API_SECRET".to_string(), "REDACTED".to_string()), - ] - .into_iter() - .collect(), - config: std::iter::once(( - "BASE_URL".to_string(), - "https://api.custom.example".to_string(), - )) - .collect(), - credential_expires_at_ms: std::collections::HashMap::new(), - profile_workspace: String::new(), - credential_handles: std::collections::HashMap::new(), - }], - false, - ); - - assert!(output.contains("NAME")); - assert!(output.contains("TYPE")); - assert!(output.contains("CREDENTIAL_KEYS")); - assert!(output.contains("CONFIG_KEYS")); - assert!(output.contains("work-custom")); - assert!(output.contains("custom-api")); - assert!(output.contains('2')); - assert!(output.contains('1')); - } - #[test] fn progress_step_metadata_values_map_to_cli_steps() { assert_eq!( @@ -7777,118 +6119,6 @@ mod tests { assert_eq!(progress_step_from_metadata("driver-private-step"), None); } - #[test] - fn refresh_status_table_includes_operational_fields() { - let header = refresh_status_header(); - assert!(header.contains("NEXT_REFRESH")); - assert!(header.contains("LAST_REFRESH")); - assert!(header.contains("RECOVERY")); - assert!(header.contains("FAILURE_CODE")); - assert!(header.contains("LAST_ERROR")); - - let row = refresh_status_row(&ProviderCredentialRefreshStatus { - provider_name: "my-graph".to_string(), - provider_id: "provider-id".to_string(), - credential_key: "MS_GRAPH_ACCESS_TOKEN".to_string(), - strategy: ProviderCredentialRefreshStrategy::Oauth2ClientCredentials as i32, - status: "error".to_string(), - expires_at_ms: 1_767_225_600_000, - next_refresh_at_ms: i64::MAX, - last_refresh_at_ms: 1_767_225_000_000, - last_error: "token endpoint returned a very long error message that should be truncated for table readability" - .to_string(), - recovery_action: ProviderCredentialRefreshRecoveryAction::Reauthorize as i32, - failure_code: "oauth_rotated_refresh_token_handle_missing".to_string(), - provider_error_subtype: "invalid_rapt".to_string(), - last_error_at_ms: 1_767_225_000_000, - }); - - assert!(row.contains("my-graph")); - assert!(row.contains("MS_GRAPH_ACCESS_TOKEN")); - assert!(row.contains("oauth2_client_credentials")); - assert!(row.contains("error")); - assert!(row.contains("reauthorize")); - assert!(row.contains("oauth_rotated_refresh_token_handle_missing")); - assert!(row.contains("2026-01-01 00:00:00")); - assert!(!row.contains("292278994")); - assert!(row.contains("...")); - } - - #[test] - fn empty_provider_credentials_require_all_required_credentials_to_be_runtime_resolvable() { - let refresh_token_profile = ProviderProfile { - credentials: vec![ProviderProfileCredential { - name: "MS_GRAPH_ACCESS_TOKEN".to_string(), - required: true, - refresh: Some(ProviderCredentialRefresh { - strategy: ProviderCredentialRefreshStrategy::Oauth2RefreshToken as i32, - ..Default::default() - }), - ..Default::default() - }], - ..Default::default() - }; - assert!(provider_profile_allows_empty_credentials( - &refresh_token_profile - )); - - let token_grant_profile = ProviderProfile { - credentials: vec![ProviderProfileCredential { - name: "ACCESS_TOKEN".to_string(), - required: true, - token_grant: Some(ProviderCredentialTokenGrant { - token_endpoint: "https://auth.example.com/token".to_string(), - ..Default::default() - }), - ..Default::default() - }], - ..Default::default() - }; - assert!(provider_profile_allows_empty_credentials( - &token_grant_profile - )); - - let mixed_static_profile = ProviderProfile { - credentials: vec![ - ProviderProfileCredential { - name: "ACCESS_TOKEN".to_string(), - required: true, - refresh: Some(ProviderCredentialRefresh { - strategy: ProviderCredentialRefreshStrategy::Oauth2ClientCredentials as i32, - ..Default::default() - }), - ..Default::default() - }, - ProviderProfileCredential { - name: "STATIC_API_KEY".to_string(), - required: true, - refresh: None, - ..Default::default() - }, - ], - ..Default::default() - }; - assert!(!provider_profile_allows_empty_credentials( - &mixed_static_profile - )); - - let optional_refresh_profile = ProviderProfile { - credentials: vec![ProviderProfileCredential { - name: "OPTIONAL_TOKEN".to_string(), - required: false, - refresh: Some(ProviderCredentialRefresh { - strategy: ProviderCredentialRefreshStrategy::GoogleServiceAccountJwt as i32, - ..Default::default() - }), - ..Default::default() - }], - ..Default::default() - }; - assert!(provider_profile_allows_empty_credentials( - &optional_refresh_profile - )); - } - #[test] fn parse_cli_setting_value_parses_bool_aliases() { let yes_value = parse_cli_setting_value("ocsf_json_enabled", "yes").expect("parse yes"); @@ -8019,41 +6249,6 @@ mod tests { ); } - #[test] - fn inferred_provider_type_returns_type_for_known_command() { - let result = inferred_provider_type(&["claude".to_string(), "--help".to_string()]); - assert_eq!(result, Some("claude-code".to_string())); - } - - #[test] - fn inferred_provider_type_returns_none_for_unknown_command() { - let result = inferred_provider_type(&["bash".to_string()]); - assert_eq!(result, None); - } - - #[test] - fn inferred_provider_type_returns_none_for_empty_command() { - let result = inferred_provider_type(&[]); - assert_eq!(result, None); - } - - #[test] - fn inferred_provider_type_normalizes_aliases() { - // `glab` should resolve to `gitlab` - let result = inferred_provider_type(&["glab".to_string()]); - assert_eq!(result, Some("gitlab".to_string())); - - // `gh` should resolve to `github` - let result = inferred_provider_type(&["gh".to_string()]); - assert_eq!(result, Some("github".to_string())); - } - - #[test] - fn inferred_provider_type_handles_full_path() { - let result = inferred_provider_type(&["/usr/local/bin/claude".to_string()]); - assert_eq!(result, Some("claude-code".to_string())); - } - #[test] fn sandbox_should_persist_defaults_to_persistent() { assert!(sandbox_should_persist(true, None)); @@ -8113,65 +6308,274 @@ mod tests { } #[test] - fn resolve_from_classifies_existing_dockerfile_path() { + fn resolve_from_rejects_existing_dockerfile_path() { let temp = tempfile::tempdir().expect("failed to create tempdir"); let dockerfile = temp.path().join("Dockerfile"); fs::write(&dockerfile, "FROM scratch\n").expect("failed to write Dockerfile"); - match resolve_from(dockerfile.to_str().expect("temp path is not UTF-8")) - .expect("expected Dockerfile source") - { - super::ResolvedSource::Dockerfile { - dockerfile: resolved, - context, - } => { - assert_eq!( - resolved, - dockerfile - .canonicalize() - .expect("failed to canonicalize Dockerfile") - ); - assert_eq!( - context, - temp.path() - .canonicalize() - .expect("failed to canonicalize context") - ); - } - super::ResolvedSource::Image(image) => { - panic!("expected Dockerfile source, got image {image}"); - } - } + let err = resolve_from(dockerfile.to_str().expect("temp path is not UTF-8")) + .expect_err("expected local Dockerfile path to be rejected"); + + assert!( + err.to_string() + .contains("no longer builds local Dockerfiles or directories"), + "unexpected error: {err}" + ); + assert!( + err.to_string().contains("docker build -t "), + "expected actionable build guidance: {err}" + ); + assert!( + err.to_string().contains("podman build -t "), + "expected Podman build guidance: {err}" + ); } #[test] - fn resolve_from_rejects_missing_explicit_dockerfile_path() { + fn resolve_from_rejects_missing_explicit_local_path() { let temp = tempfile::tempdir().expect("failed to create tempdir"); let missing = temp.path().join("Dockerfile"); let err = resolve_from(missing.to_str().expect("temp path is not UTF-8")) - .expect_err("expected missing Dockerfile path to be rejected"); + .expect_err("expected missing explicit local path to be rejected"); assert!( - err.to_string().contains("local --from path does not exist"), + err.to_string() + .contains("no longer builds local Dockerfiles or directories"), "unexpected error: {err}" ); } + #[test] + fn resolve_from_rejects_bare_dockerfile_name() { + let err = resolve_from("Dockerfile").expect_err("expected bare Dockerfile to be rejected"); + + assert!( + err.to_string() + .contains("no longer builds local Dockerfiles or directories"), + "unexpected error: {err}" + ); + } + + #[test] + fn resolve_from_keeps_bare_community_name_when_local_directory_matches() { + let _lock = TEST_ENV_LOCK + .lock() + .unwrap_or_else(std::sync::PoisonError::into_inner); + let temp = tempfile::tempdir().expect("create tempdir"); + fs::create_dir(temp.path().join("python")).expect("create matching directory"); + let original_dir = std::env::current_dir().expect("read current directory"); + std::env::set_current_dir(temp.path()).expect("enter tempdir"); + + let result = resolve_from("python"); + + std::env::set_current_dir(original_dir).expect("restore current directory"); + match result.expect("bare community name should not be a local path") { + super::ResolvedSource::Image(image) => assert_eq!( + image, + "ghcr.io/nvidia/openshell-community/sandboxes/python:latest" + ), + other @ super::ResolvedSource::RootfsTar { .. } => { + panic!("expected image source, got {other:?}"); + } + } + } + #[test] fn resolve_from_keeps_dockerfile_named_image_refs_as_images() { let image_ref = "ghcr.io/acme/dockerfile-runner:latest"; match resolve_from(image_ref).expect("expected image source") { super::ResolvedSource::Image(image) => assert_eq!(image, image_ref), - super::ResolvedSource::Dockerfile { .. } => { - panic!("expected image ref, got Dockerfile source"); + other @ super::ResolvedSource::RootfsTar { .. } => { + panic!("expected image ref, got {other:?}"); } } } #[test] - fn dockerfile_sources_are_rejected_for_remote_gateways() { + fn resolve_from_classifies_tar_archive() { + let temp = tempfile::tempdir().expect("failed to create tempdir"); + let archive = temp.path().join("rootfs.tar"); + fs::write(&archive, b"fake tar content").expect("failed to write archive"); + + match resolve_from(archive.to_str().expect("temp path is not UTF-8")) + .expect("expected RootfsTar source") + { + super::ResolvedSource::RootfsTar { path } => { + assert_eq!( + path, + archive + .canonicalize() + .expect("failed to canonicalize archive") + ); + } + other @ super::ResolvedSource::Image(_) => { + panic!("expected RootfsTar source, got {other:?}"); + } + } + } + + #[test] + fn resolve_from_classifies_tar_gz_archive() { + let temp = tempfile::tempdir().expect("failed to create tempdir"); + let archive = temp.path().join("rootfs.tar.gz"); + fs::write(&archive, b"fake tar.gz content").expect("failed to write archive"); + + match resolve_from(archive.to_str().expect("temp path is not UTF-8")) + .expect("expected RootfsTar source") + { + super::ResolvedSource::RootfsTar { path } => { + assert_eq!( + path, + archive + .canonicalize() + .expect("failed to canonicalize archive") + ); + } + other @ super::ResolvedSource::Image(_) => { + panic!("expected RootfsTar source, got {other:?}"); + } + } + } + + #[test] + fn resolve_from_classifies_tgz_archive() { + let temp = tempfile::tempdir().expect("failed to create tempdir"); + let archive = temp.path().join("rootfs.tgz"); + fs::write(&archive, b"fake tgz content").expect("failed to write archive"); + + match resolve_from(archive.to_str().expect("temp path is not UTF-8")) + .expect("expected RootfsTar source") + { + super::ResolvedSource::RootfsTar { path } => { + assert_eq!( + path, + archive + .canonicalize() + .expect("failed to canonicalize archive") + ); + } + other @ super::ResolvedSource::Image(_) => { + panic!("expected RootfsTar source, got {other:?}"); + } + } + } + + #[test] + fn resolve_from_rejects_missing_tar_archive() { + let temp = tempfile::tempdir().expect("failed to create tempdir"); + let missing = temp.path().join("missing.tar"); + + let err = resolve_from(missing.to_str().expect("temp path is not UTF-8")) + .expect_err("expected missing archive to be rejected"); + + assert!( + err.to_string().contains("local --from path does not exist"), + "unexpected error: {err}" + ); + } + + #[test] + fn filename_looks_like_rootfs_tar_detects_extensions() { + use super::filename_looks_like_rootfs_tar; + assert!(filename_looks_like_rootfs_tar(Path::new("rootfs.tar"))); + assert!(filename_looks_like_rootfs_tar(Path::new("rootfs.tar.gz"))); + assert!(filename_looks_like_rootfs_tar(Path::new("rootfs.tgz"))); + assert!(filename_looks_like_rootfs_tar(Path::new("IMAGE.TAR"))); + assert!(filename_looks_like_rootfs_tar(Path::new("my-image.TAR.GZ"))); + assert!(!filename_looks_like_rootfs_tar(Path::new("Dockerfile"))); + assert!(!filename_looks_like_rootfs_tar(Path::new("image.zip"))); + } + + /// The gateway forwards only `template.driver_config.` to the + /// selected driver, so a top-level key is silently dropped and the archive + /// never reaches the VM driver. + #[test] + fn rootfs_tar_driver_config_nests_under_vm_key() { + use prost_types::value::Kind; + + let config = + super::merge_rootfs_tar_driver_config(None, "tok-abc").expect("merge should succeed"); + + assert_eq!( + config.fields.keys().collect::>(), + vec!["vm"], + "rootfs tar config must live under the vm driver key" + ); + let Some(Kind::StructValue(vm)) = config.fields["vm"].kind.as_ref() else { + panic!("vm entry must be an object"); + }; + let Some(Kind::StringValue(token)) = vm.fields["rootfs_tar_staging_token"].kind.as_ref() + else { + panic!("rootfs_tar_staging_token must be a string"); + }; + assert_eq!(token, "tok-abc"); + assert!( + !vm.fields.contains_key("rootfs_tar_path"), + "the CLI never names a host path; the gateway resolves one" + ); + } + + #[test] + fn rootfs_tar_driver_config_preserves_existing_vm_settings() { + use prost_types::value::Kind; + + let base = parse_driver_config_json( + r#"{"vm":{"gpu_device_ids":["0000:2d:00.0"]},"docker":{"userns":"host"}}"#, + ) + .expect("valid driver config json"); + + let config = super::merge_rootfs_tar_driver_config(Some(base), "tok-abc") + .expect("merge should succeed"); + + // The sibling driver block survives untouched. + assert!(config.fields.contains_key("docker")); + + let Some(Kind::StructValue(vm)) = config.fields["vm"].kind.as_ref() else { + panic!("vm entry must be an object"); + }; + assert!( + vm.fields.contains_key("gpu_device_ids"), + "pre-existing vm settings must not be clobbered" + ); + let Some(Kind::StringValue(token)) = vm.fields["rootfs_tar_staging_token"].kind.as_ref() + else { + panic!("rootfs_tar_staging_token must be a string"); + }; + assert_eq!(token, "tok-abc"); + } + + #[test] + fn rootfs_tar_driver_config_rejects_caller_supplied_token() { + let base = parse_driver_config_json(r#"{"vm":{"rootfs_tar_staging_token":"stolen"}}"#) + .expect("valid driver config json"); + + let err = super::merge_rootfs_tar_driver_config(Some(base), "tok-abc") + .expect_err("a caller-supplied staging token must not be silently overwritten"); + + assert!( + err.to_string() + .contains("already sets vm.rootfs_tar_staging_token"), + "unexpected error: {err}" + ); + } + + #[test] + fn rootfs_tar_driver_config_rejects_non_object_vm_block() { + let base = parse_driver_config_json(r#"{"vm":"nonsense"}"#).expect("valid json object"); + + let err = super::merge_rootfs_tar_driver_config(Some(base), "tok-abc") + .expect_err("a non-object vm block must be rejected"); + + assert!( + err.to_string().contains("must be an object"), + "unexpected error: {err}" + ); + } + + #[test] + fn rootfs_tar_sources_are_rejected_for_remote_gateways() { let metadata = GatewayMetadata { name: "remote".to_string(), gateway_endpoint: "https://gateway.example.com".to_string(), @@ -8186,11 +6590,11 @@ mod tests { ..Default::default() }; - assert!(!dockerfile_sources_supported_for_gateway(Some(&metadata))); + assert!(!rootfs_tar_sources_supported_for_gateway(Some(&metadata))); } #[test] - fn dockerfile_sources_are_allowed_for_local_gateways() { + fn rootfs_tar_sources_are_allowed_for_local_gateways() { let metadata = GatewayMetadata { name: "local".to_string(), gateway_endpoint: "http://127.0.0.1:8080".to_string(), @@ -8205,8 +6609,8 @@ mod tests { ..Default::default() }; - assert!(dockerfile_sources_supported_for_gateway(Some(&metadata))); - assert!(dockerfile_sources_supported_for_gateway(None)); + assert!(rootfs_tar_sources_supported_for_gateway(Some(&metadata))); + assert!(rootfs_tar_sources_supported_for_gateway(None)); } #[test] @@ -8538,391 +6942,71 @@ mod tests { } #[test] - fn read_gcloud_adc_missing_file_errors() { - let _lock = TEST_ENV_LOCK - .lock() - .unwrap_or_else(std::sync::PoisonError::into_inner); - let _guard = EnvVarGuard::set( - "GOOGLE_APPLICATION_CREDENTIALS", - "/nonexistent/path/to/adc.json", - ); - let err = super::read_gcloud_adc().expect_err("missing file should error"); - assert!( - err.to_string().contains("failed to read gcloud ADC file"), - "unexpected error: {err}" - ); - } - - #[test] - fn read_gcloud_adc_wrong_type_errors() { - let _lock = TEST_ENV_LOCK - .lock() - .unwrap_or_else(std::sync::PoisonError::into_inner); - let tmp = tempfile::NamedTempFile::new().expect("tempfile"); - let json = serde_json::json!({ - "type": "service_account", - "project_id": "my-project", - "private_key_id": "key123" - }); - Write::write_all(&mut tmp.as_file(), json.to_string().as_bytes()).expect("write tempfile"); - let _guard = EnvVarGuard::set( - "GOOGLE_APPLICATION_CREDENTIALS", - tmp.path().to_str().expect("tempfile path"), - ); - let err = super::read_gcloud_adc().expect_err("wrong type should error"); - // The service_account type gets a targeted message directing the user - // to the real Vertex service-account credential flow instead of the - // generic authorized_user hint. - assert!( - err.to_string() - .contains("GOOGLE_VERTEX_AI_SERVICE_ACCOUNT_TOKEN"), - "error should mention the service-account token key, got: {err}" - ); - } - - #[test] - fn read_gcloud_adc_parses_user_creds() { - let _lock = TEST_ENV_LOCK - .lock() - .unwrap_or_else(std::sync::PoisonError::into_inner); - let tmp = tempfile::NamedTempFile::new().expect("tempfile"); - let json = serde_json::json!({ - "type": "authorized_user", - "client_id": "test-client-id.apps.googleusercontent.com", - "client_secret": "test-client-secret", - "refresh_token": "test-refresh-token" - }); - Write::write_all(&mut tmp.as_file(), json.to_string().as_bytes()).expect("write tempfile"); - let _guard = EnvVarGuard::set( - "GOOGLE_APPLICATION_CREDENTIALS", - tmp.path().to_str().expect("tempfile path"), - ); - let (client_id, client_secret, refresh_token) = - super::read_gcloud_adc().expect("valid ADC should parse"); - assert_eq!(client_id, "test-client-id.apps.googleusercontent.com"); - assert_eq!(client_secret, "test-client-secret"); - assert_eq!(refresh_token, "test-refresh-token"); - } - - #[test] - fn read_gcloud_adc_uses_cloudsdk_config_fallback() { - let _lock = TEST_ENV_LOCK - .lock() - .unwrap_or_else(std::sync::PoisonError::into_inner); - let dir = tempfile::tempdir().expect("tempdir"); - let adc_path = dir.path().join("application_default_credentials.json"); - let json = serde_json::json!({ - "type": "authorized_user", - "client_id": "cloudsdk-client-id.apps.googleusercontent.com", - "client_secret": "cloudsdk-client-secret", - "refresh_token": "cloudsdk-refresh-token" - }); - fs::write(&adc_path, json.to_string()).expect("write adc file"); - let _adc_guard = EnvVarGuard::unset("GOOGLE_APPLICATION_CREDENTIALS"); - let _cloudsdk_guard = - EnvVarGuard::set("CLOUDSDK_CONFIG", dir.path().to_str().expect("config path")); - - let (client_id, client_secret, refresh_token) = - super::read_gcloud_adc().expect("valid CLOUDSDK_CONFIG ADC should parse"); - assert_eq!(client_id, "cloudsdk-client-id.apps.googleusercontent.com"); - assert_eq!(client_secret, "cloudsdk-client-secret"); - assert_eq!(refresh_token, "cloudsdk-refresh-token"); - } - - #[test] - fn read_gcloud_adc_malformed_json_errors() { - let _lock = TEST_ENV_LOCK - .lock() - .unwrap_or_else(std::sync::PoisonError::into_inner); - let tmp = tempfile::NamedTempFile::new().expect("tempfile"); - Write::write_all(&mut tmp.as_file(), b"not valid json at all {{{{") - .expect("write tempfile"); - let _guard = EnvVarGuard::set( - "GOOGLE_APPLICATION_CREDENTIALS", - tmp.path().to_str().expect("tempfile path"), - ); - let result = super::read_gcloud_adc(); - assert!( - result.is_err(), - "malformed JSON should produce an error, got: {result:?}" - ); - let err = result.unwrap_err(); - let msg = format!("{err}"); - assert!( - msg.contains("parse") - || msg.contains("JSON") - || msg.contains("json") - || msg.contains("invalid") - || msg.contains("failed"), - "error message should mention parse/JSON failure, got: {msg}" - ); - } - - #[test] - fn empty_provider_credentials_allow_oauth2_refresh_token() { - use openshell_core::proto::{ - ProviderCredentialRefresh, ProviderCredentialRefreshStrategy, ProviderProfile, - ProviderProfileCredential, + fn sandbox_template_to_json_includes_metadata_labels_and_annotations() { + let template = SandboxWorkloadTemplate { + metadata: Some(ObjectMeta { + id: "template-123".to_string(), + name: "gpu-kata".to_string(), + labels: std::collections::HashMap::from([( + "team".to_string(), + "runtime".to_string(), + )]), + annotations: std::collections::HashMap::from([( + "owner".to_string(), + "platform".to_string(), + )]), + workspace: "default".to_string(), + ..Default::default() + }), + ..Default::default() }; - let strategy = ProviderCredentialRefreshStrategy::Oauth2RefreshToken as i32; - let profile = ProviderProfile { - credentials: vec![ProviderProfileCredential { - required: true, - refresh: Some(ProviderCredentialRefresh { - strategy, + let json = super::sandbox_template_to_json(&template); + + assert_eq!(json["labels"]["team"], "runtime"); + assert_eq!(json["annotations"]["owner"], "platform"); + } + + #[test] + fn sandbox_template_to_json_formats_default_gpu_like_display_output() { + let template = SandboxWorkloadTemplate { + spec: Some(SandboxWorkloadTemplateSpec { + workload: Some(SandboxWorkloadConfig { + resources: Some(SandboxResources { + gpu: Some(GpuResourceRequirements { count: None }), + ..Default::default() + }), ..Default::default() }), ..Default::default() - }], - ..Default::default() - }; - assert!( - provider_profile_allows_empty_credentials(&profile), - "Oauth2RefreshToken should be allowed for refresh bootstrap" - ); - } - - #[test] - fn provider_to_json_includes_core_fields() { - let metadata = ObjectMeta { - id: "prov-123".to_string(), - name: "test-provider".to_string(), + }), ..Default::default() }; - let provider = Provider { - metadata: Some(metadata), - r#type: "anthropic".to_string(), - credentials: std::collections::HashMap::new(), - config: std::collections::HashMap::new(), - credential_expires_at_ms: std::collections::HashMap::new(), - profile_workspace: String::new(), - credential_handles: std::collections::HashMap::new(), - }; + let json = super::sandbox_template_to_json(&template); - let json = super::provider_to_json(&provider); - - assert_eq!(json["id"], "prov-123"); - assert_eq!(json["name"], "test-provider"); - assert_eq!(json["workspace"], ""); - assert_eq!(json["type"], "anthropic"); + assert_eq!(json["resources"]["gpu"], "default"); } #[test] - fn provider_to_json_exposes_credential_keys_not_values() { - let mut credentials = std::collections::HashMap::new(); - credentials.insert("ANTHROPIC_API_KEY".to_string(), "secret-value".to_string()); - credentials.insert("OTHER_KEY".to_string(), "other-secret".to_string()); - - let provider = Provider { - metadata: Some(ObjectMeta::default()), - r#type: "anthropic".to_string(), - credentials, - config: std::collections::HashMap::new(), - credential_expires_at_ms: std::collections::HashMap::new(), - profile_workspace: String::new(), - credential_handles: std::collections::HashMap::new(), - }; - - let json = super::provider_to_json(&provider); - let json_str = json.to_string(); - - // Assert credential keys are present - let keys = json["credential_keys"].as_array().unwrap(); - assert_eq!(keys.len(), 2); - assert!(keys.iter().any(|k| k.as_str() == Some("ANTHROPIC_API_KEY"))); - assert!(keys.iter().any(|k| k.as_str() == Some("OTHER_KEY"))); - - // Assert credential values are NOT in the output (SECURITY) - assert!( - !json_str.contains("secret-value"), - "credential values must not be exposed" - ); - assert!( - !json_str.contains("other-secret"), - "credential values must not be exposed" - ); - } - - #[test] - fn provider_to_json_exposes_config_keys_not_values() { - let mut config = std::collections::HashMap::new(); - config.insert("region".to_string(), "us-west".to_string()); - config.insert( - "endpoint".to_string(), - "https://api.example.com".to_string(), - ); - - let provider = Provider { - metadata: Some(ObjectMeta::default()), - r#type: "custom".to_string(), - credentials: std::collections::HashMap::new(), - config, - credential_expires_at_ms: std::collections::HashMap::new(), - profile_workspace: String::new(), - credential_handles: std::collections::HashMap::new(), - }; - - let json = super::provider_to_json(&provider); - let json_str = json.to_string(); - - // Assert config keys are present - let keys = json["config_keys"].as_array().unwrap(); - assert_eq!(keys.len(), 2); - assert!(keys.iter().any(|k| k.as_str() == Some("region"))); - assert!(keys.iter().any(|k| k.as_str() == Some("endpoint"))); - - // Assert config values are NOT in the output (SECURITY) - assert!( - !json_str.contains("us-west"), - "config values must not be exposed" - ); - assert!( - !json_str.contains("https://api.example.com"), - "config values must not be exposed" - ); - } - - #[test] - fn provider_to_json_omits_empty_config() { - let provider = Provider { - metadata: Some(ObjectMeta::default()), - r#type: "anthropic".to_string(), - credentials: std::collections::HashMap::new(), - config: std::collections::HashMap::new(), // Empty config - credential_expires_at_ms: std::collections::HashMap::new(), - profile_workspace: String::new(), - credential_handles: std::collections::HashMap::new(), - }; - - let json = super::provider_to_json(&provider); - - assert!( - json.get("config_keys").is_none(), - "empty config_keys should be omitted" - ); - } - - #[test] - fn provider_to_json_includes_metadata_fields_when_present() { - let mut labels = std::collections::HashMap::new(); - labels.insert("env".to_string(), "prod".to_string()); - - let metadata = ObjectMeta { - id: "prov-123".to_string(), - name: "test-provider".to_string(), - resource_version: 42, - created_at_ms: 1_234_567_890_000, - labels, - annotations: std::collections::HashMap::new(), - workspace: String::new(), - deletion_timestamp_ms: 0, - }; - - let provider = Provider { - metadata: Some(metadata), - r#type: "anthropic".to_string(), - credentials: std::collections::HashMap::new(), - config: std::collections::HashMap::new(), - credential_expires_at_ms: std::collections::HashMap::new(), - profile_workspace: String::new(), - credential_handles: std::collections::HashMap::new(), - }; - - let json = super::provider_to_json(&provider); - - assert_eq!(json["resource_version"], 42); - assert_eq!(json["created_at"], "2009-02-13 23:31:30"); - assert_eq!(json["labels"]["env"], "prod"); - } - - #[test] - fn provider_to_json_omits_zero_metadata_fields() { - let metadata = ObjectMeta { - id: "prov-123".to_string(), - name: "test-provider".to_string(), - // resource_version and created_at_ms are 0 - // labels is empty + fn sandbox_template_to_json_preserves_explicit_gpu_count_as_number() { + let template = SandboxWorkloadTemplate { + spec: Some(SandboxWorkloadTemplateSpec { + workload: Some(SandboxWorkloadConfig { + resources: Some(SandboxResources { + gpu: Some(GpuResourceRequirements { count: Some(2) }), + ..Default::default() + }), + ..Default::default() + }), + ..Default::default() + }), ..Default::default() }; - let provider = Provider { - metadata: Some(metadata), - r#type: "anthropic".to_string(), - credentials: std::collections::HashMap::new(), - config: std::collections::HashMap::new(), - credential_expires_at_ms: std::collections::HashMap::new(), - profile_workspace: String::new(), - credential_handles: std::collections::HashMap::new(), - }; + let json = super::sandbox_template_to_json(&template); - let json = super::provider_to_json(&provider); - - assert!( - json.get("resource_version").is_none(), - "zero resource_version should be omitted" - ); - assert!( - json.get("created_at").is_none(), - "zero created_at should be omitted" - ); - assert!( - json.get("labels").is_none(), - "empty labels should be omitted" - ); - } - - #[test] - fn provider_to_json_includes_credential_expiration() { - let mut credential_expires_at_ms = std::collections::HashMap::new(); - credential_expires_at_ms.insert("ACCESS_TOKEN".to_string(), 1_234_567_890); - - let provider = Provider { - metadata: Some(ObjectMeta::default()), - r#type: "oauth".to_string(), - credentials: std::collections::HashMap::new(), - config: std::collections::HashMap::new(), - credential_expires_at_ms, - profile_workspace: String::new(), - credential_handles: std::collections::HashMap::new(), - }; - - let json = super::provider_to_json(&provider); - - assert_eq!( - json["credential_expires_at_ms"]["ACCESS_TOKEN"], - 1_234_567_890 - ); - } - - #[test] - fn provider_to_json_formats_created_at_as_human_readable() { - let metadata = ObjectMeta { - id: "prov-123".to_string(), - name: "test-provider".to_string(), - created_at_ms: 1_609_459_200_000, // 2021-01-01 00:00:00 - ..Default::default() - }; - - let provider = Provider { - metadata: Some(metadata), - r#type: "anthropic".to_string(), - credentials: std::collections::HashMap::new(), - config: std::collections::HashMap::new(), - credential_expires_at_ms: std::collections::HashMap::new(), - profile_workspace: String::new(), - credential_handles: std::collections::HashMap::new(), - }; - - let json = super::provider_to_json(&provider); - - // Should format as human-readable datetime, not raw milliseconds - assert_eq!(json["created_at"], "2021-01-01 00:00:00"); - assert!( - json.get("created_at_ms").is_none(), - "raw milliseconds field should not exist" - ); + assert_eq!(json["resources"]["gpu"], 2); } #[test] @@ -8935,6 +7019,10 @@ mod tests { created_at_ms: 1_609_459_200_000, ..Default::default() }), + created_from_workload_template: Some(SandboxWorkloadTemplateProvenance { + name: "gpu-kata".to_string(), + resource_version: "7".to_string(), + }), ..Default::default() }; sandbox.set_phase(SandboxPhase::Ready as i32); @@ -8954,6 +7042,11 @@ mod tests { assert_eq!(json["policy_source"], "global"); assert_eq!(json["revision"], 3); assert!(json["policy"].is_null()); + assert_eq!(json["created_from_workload_template"]["name"], "gpu-kata"); + assert_eq!( + json["created_from_workload_template"]["resource_version"], + "7" + ); } #[test] diff --git a/crates/openshell-cli/src/ssh.rs b/crates/openshell-cli/src/ssh.rs index 1e1e48257..c0937db48 100644 --- a/crates/openshell-cli/src/ssh.rs +++ b/crates/openshell-cli/src/ssh.rs @@ -3,6 +3,7 @@ //! SSH connection and proxy utilities. +use crate::color::Colorize; use crate::tls::{TlsOptions, grpc_client}; use miette::{IntoDiagnostic, Result, WrapErr}; #[cfg(unix)] @@ -16,7 +17,6 @@ use openshell_core::proto::{ tcp_forward_init, }; use openshell_core::{ObjectId, driver_mounts}; -use owo_colors::OwoColorize; use std::fs; use std::future::Future; use std::io::{IsTerminal, Write}; @@ -92,7 +92,7 @@ async fn ssh_session_config( let sandbox = client .get_sandbox(GetSandboxRequest { name: name.to_string(), - workspace: workspace.to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector(workspace)), }) .await .into_diagnostic()? diff --git a/crates/openshell-cli/src/tls.rs b/crates/openshell-cli/src/tls.rs index c24b84c7d..2578ed961 100644 --- a/crates/openshell-cli/src/tls.rs +++ b/crates/openshell-cli/src/tls.rs @@ -4,7 +4,6 @@ use miette::{IntoDiagnostic, Result, WrapErr}; use openshell_core::auth::EdgeAuthInterceptor; use openshell_core::net::set_tcp_nodelay_best_effort; -use openshell_core::proto::inference_client::InferenceClient; use openshell_core::proto::open_shell_client::OpenShellClient; use rustls::{ RootCertStore, @@ -26,8 +25,6 @@ use url::{Host, Url}; /// Concrete gRPC client type used by all commands. pub type GrpcClient = OpenShellClient>; -/// Concrete inference client type. -pub type GrpcInferenceClient = InferenceClient>; #[derive(Clone, Debug, Default)] pub struct TlsOptions { @@ -287,7 +284,7 @@ impl ServerCertVerifier for InsecureServerCertVerifier { } fn supported_verify_schemes(&self) -> Vec { - rustls::crypto::ring::default_provider() + rustls::crypto::aws_lc_rs::default_provider() .signature_verification_algorithms .supported_schemes() } @@ -466,12 +463,6 @@ fn interceptor_from_tls(tls: &TlsOptions) -> Result { EdgeAuthInterceptor::new(tls.oidc_token.as_deref(), tls.edge_token.as_deref()) } -pub async fn grpc_inference_client(server: &str, tls: &TlsOptions) -> Result { - let channel = build_channel(server, tls).await?; - let interceptor = interceptor_from_tls(tls)?; - Ok(InferenceClient::with_interceptor(channel, interceptor)) -} - #[cfg(test)] mod tests { use super::tls_server_name; diff --git a/crates/openshell-cli/tests/cli_color_integration.rs b/crates/openshell-cli/tests/cli_color_integration.rs new file mode 100644 index 000000000..54f2dc2f0 --- /dev/null +++ b/crates/openshell-cli/tests/cli_color_integration.rs @@ -0,0 +1,444 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! End-to-end checks that `openshell` does not write ANSI escape sequences into +//! a pipe. +//! +//! `Command::output` gives the child a pipe rather than a terminal, which is +//! exactly the situation these tests care about: a supervisor script parsing +//! output. The unit tests in `color` cover the precedence rules; these cover the +//! wiring from flag to rendered byte, across both styling paths that reach +//! stdout — the styled tables and the `tracing` formatter. + +#![cfg(unix)] + +use std::path::Path; +use std::process::Command; + +const ESC: char = '\u{1b}'; +const SANDBOX: &str = "openclaw-saw"; +const PORT: u16 = 8443; + +/// Seed a forward PID file so `forward list` has a row to render. +/// +/// The PID is one that cannot be running, so the row reports `dead`. Status text +/// is irrelevant here — both states go through the same styling path. +fn config_dir_with_forward(root: &Path) { + let forwards = root.join("openshell").join("forwards"); + std::fs::create_dir_all(&forwards).expect("create forwards dir"); + std::fs::write( + forwards.join(format!("{SANDBOX}-{PORT}.pid")), + // Format: \t\t + "4000000\tsbx-test\t0.0.0.0", + ) + .expect("write forward pid file"); +} + +fn forward_list(config_dir: &Path, args: &[&str], no_color: Option<&str>) -> String { + let mut command = Command::new(env!("CARGO_BIN_EXE_openshell")); + command + .args(["forward", "list"]) + .args(args) + .env("XDG_CONFIG_HOME", config_dir) + .env_remove("NO_COLOR") + .env_remove("FORCE_COLOR") + .env_remove("OPENSHELL_COLOR"); + if let Some(value) = no_color { + command.env("NO_COLOR", value); + } + + let output = command.output().expect("run openshell forward list"); + let stdout = String::from_utf8(output.stdout).expect("stdout is utf-8"); + assert!( + stdout.contains(SANDBOX), + "expected the seeded forward in the table, got: {stdout:?}" + ); + stdout +} + +#[test] +fn piped_output_is_free_of_escape_sequences_by_default() { + let tmpdir = tempfile::tempdir().expect("create tmpdir"); + config_dir_with_forward(tmpdir.path()); + + let stdout = forward_list(tmpdir.path(), &[], None); + + assert!( + !stdout.contains(ESC), + "piped `forward list` must not emit ANSI escapes, got: {stdout:?}" + ); +} + +#[test] +fn no_color_environment_variable_is_honored() { + let tmpdir = tempfile::tempdir().expect("create tmpdir"); + config_dir_with_forward(tmpdir.path()); + + let stdout = forward_list(tmpdir.path(), &[], Some("1")); + + assert!(!stdout.contains(ESC), "NO_COLOR ignored, got: {stdout:?}"); +} + +#[test] +fn color_never_suppresses_escape_sequences() { + let tmpdir = tempfile::tempdir().expect("create tmpdir"); + config_dir_with_forward(tmpdir.path()); + + let stdout = forward_list(tmpdir.path(), &["--color", "never"], None); + + assert!( + !stdout.contains(ESC), + "--color never ignored, got: {stdout:?}" + ); +} + +#[test] +fn color_always_emits_escape_sequences_into_a_pipe() { + let tmpdir = tempfile::tempdir().expect("create tmpdir"); + config_dir_with_forward(tmpdir.path()); + + // The opt-in escape hatch for callers who pipe into a pager and still want + // styling. Also proves the other cases are suppressing real output rather + // than rendering an empty table. + let stdout = forward_list(tmpdir.path(), &["--color", "always"], None); + + assert!( + stdout.contains(ESC), + "--color always must still colorize, got: {stdout:?}" + ); +} + +#[test] +fn explicit_color_flag_overrides_no_color() { + let tmpdir = tempfile::tempdir().expect("create tmpdir"); + config_dir_with_forward(tmpdir.path()); + + let stdout = forward_list(tmpdir.path(), &["--color", "always"], Some("1")); + + assert!( + stdout.contains(ESC), + "--color always must outrank NO_COLOR, got: {stdout:?}" + ); +} + +/// Run a command that fails to connect, with logging turned up so the `tracing` +/// formatter produces output. +/// +/// `tracing_subscriber::fmt` writes to stdout with ANSI enabled and does no +/// terminal detection of its own, so this is a second, independent way for +/// escapes to reach a pipe. +fn failing_connect(args: &[&str]) -> (String, String) { + let tmpdir = tempfile::tempdir().expect("create tmpdir"); + let output = Command::new(env!("CARGO_BIN_EXE_openshell")) + .args([ + "sandbox", + "list", + "--gateway", + "test-gateway", + // Nothing listens on port 1; the connection is refused immediately. + "--gateway-endpoint", + "http://127.0.0.1:1", + ]) + .args(args) + .env("XDG_CONFIG_HOME", tmpdir.path()) + .env("RUST_LOG", "debug") + .env_remove("NO_COLOR") + .env_remove("FORCE_COLOR") + .env_remove("OPENSHELL_COLOR") + .output() + .expect("run openshell sandbox list"); + + ( + String::from_utf8_lossy(&output.stdout).into_owned(), + String::from_utf8_lossy(&output.stderr).into_owned(), + ) +} + +fn verbose_log_output(args: &[&str]) -> String { + failing_connect(args).0 +} + +#[test] +fn tracing_output_is_free_of_escape_sequences_when_piped() { + let plain = verbose_log_output(&[]); + let styled = verbose_log_output(&["--color", "always"]); + + // Positive control. If this fails, the command stopped emitting logs at + // debug level — fix the invocation rather than deleting the assertion + // below, which would otherwise pass vacuously. + assert!( + styled.contains(ESC), + "expected `--color always` to style log output; got: {styled:?}" + ); + assert!( + !plain.contains(ESC), + "piped log output must not emit ANSI escapes, got: {plain:?}" + ); +} + +/// Run a command with stdout attached to a pseudo-terminal and stderr on a +/// pipe, returning what each stream received. +#[cfg(target_os = "linux")] +fn run_with_stdout_tty(mut command: Command) -> (String, String) { + use std::io::Read; + use std::os::fd::{AsRawFd, OwnedFd}; + + let pty = nix::pty::openpty(None, None).expect("openpty"); + let controller: OwnedFd = pty.master; + let follower: OwnedFd = pty.slave; + + let mut child = command + .stdout(follower.try_clone().expect("dup pty follower")) + .stderr(std::process::Stdio::piped()) + .spawn() + .expect("spawn openshell"); + // `Command` retains its configured stdio handles after spawning. Drop it so + // the controller sees EIO once the child exits. + drop(command); + + // Drop every follower handle in this process, or reading the controller + // blocks forever instead of returning EIO once the child exits. + drop(follower); + + let mut stderr_buf = String::new(); + child + .stderr + .take() + .expect("stderr pipe") + .read_to_string(&mut stderr_buf) + .expect("read stderr"); + child.wait().expect("wait for openshell"); + + // The pty read ends with EIO rather than a clean EOF; treat that as done. + let mut stdout_buf = Vec::new(); + let mut chunk = [0u8; 4096]; + loop { + match nix::unistd::read(controller.as_raw_fd(), &mut chunk) { + Ok(0) | Err(_) => break, + Ok(n) => stdout_buf.extend_from_slice(&chunk[..n]), + } + } + + ( + String::from_utf8_lossy(&stdout_buf).into_owned(), + stderr_buf, + ) +} + +/// Run a failing command with stdout attached to a pseudo-terminal and stderr +/// on a pipe, returning what each stream received. +/// +/// `Command::output` gives both streams pipes, so it cannot distinguish a +/// per-stream decision from a single one resolved off stdout. This asymmetric +/// setup is the only way to catch a stream being handed the other stream's +/// answer. +#[cfg(target_os = "linux")] +fn split_streams_stdout_tty(args: &[&str]) -> (String, String) { + let tmpdir = tempfile::tempdir().expect("create tmpdir"); + let mut command = Command::new(env!("CARGO_BIN_EXE_openshell")); + command + .args([ + "sandbox", + "list", + "--gateway", + "test-gateway", + "--gateway-endpoint", + "http://127.0.0.1:1", + ]) + .args(args) + .env("XDG_CONFIG_HOME", tmpdir.path()) + .env("RUST_LOG", "debug") + // Pin TERM: `auto` now requires a capable terminal, and CI runners + // often leave TERM unset, which would make this test's outcome depend + // on the ambient environment. + .env("TERM", "xterm-256color") + .env_remove("NO_COLOR") + .env_remove("FORCE_COLOR") + .env_remove("OPENSHELL_COLOR"); + + run_with_stdout_tty(command) +} + +/// Run `forward list` with stdout on a pseudo-terminal, under the given `TERM`, +/// and return everything stdout received. +/// +/// When `stderr_on_tty` is true, both streams share the terminal so the +/// `owo-colors` table is styled too. Otherwise, stderr is redirected to +/// `/dev/null`, which verifies the conservative table behavior. +#[cfg(target_os = "linux")] +fn forward_list_on_pty(term: &str, args: &[&str], stderr_on_tty: bool) -> String { + use std::os::fd::{AsRawFd, OwnedFd}; + + let pty = nix::pty::openpty(None, None).expect("openpty"); + let controller: OwnedFd = pty.master; + let follower: OwnedFd = pty.slave; + + let tmpdir = tempfile::tempdir().expect("create tmpdir"); + config_dir_with_forward(tmpdir.path()); + + let mut command = Command::new(env!("CARGO_BIN_EXE_openshell")); + command + .args(["forward", "list"]) + .args(args) + .env("XDG_CONFIG_HOME", tmpdir.path()) + .env("TERM", term) + .env_remove("NO_COLOR") + .env_remove("FORCE_COLOR") + .env_remove("OPENSHELL_COLOR") + .stdout(follower.try_clone().expect("dup pty follower")); + if stderr_on_tty { + command.stderr(follower.try_clone().expect("dup pty follower")); + } else { + command.stderr(std::process::Stdio::null()); + } + let mut child = command.spawn().expect("spawn openshell"); + // `Command` retains its configured stdio handles after spawning. Drop it so + // the controller sees EIO once the child exits. + drop(command); + + // Drop every follower handle here, or the controller read never sees EIO. + drop(follower); + + let mut buf = Vec::new(); + let mut chunk = [0u8; 4096]; + loop { + match nix::unistd::read(controller.as_raw_fd(), &mut chunk) { + Ok(0) | Err(_) => break, + Ok(n) => buf.extend_from_slice(&chunk[..n]), + } + } + child.wait().expect("wait for openshell"); + + let out = String::from_utf8_lossy(&buf).into_owned(); + assert!( + out.contains(SANDBOX), + "expected the seeded forward in the table, got: {out:?}" + ); + out +} + +/// A terminal that does not render ANSI must not be styled under `auto`. +/// +/// `TERM=dumb` is still a terminal, so an `is_terminal()` check alone reports it +/// as styleable. `console` and `miette` apply their own `TERM` checks, but the +/// color switch overrides both, so the check has to live here. +#[cfg(target_os = "linux")] +#[test] +fn dumb_terminal_is_not_styled_under_auto() { + let dumb = forward_list_on_pty("dumb", &[], true); + // Positive control: the same session on a capable terminal is styled, so a + // plain result below means capability was consulted, not that the pty setup + // silently produced nothing. + let capable = forward_list_on_pty("xterm-256color", &[], true); + + assert!( + capable.contains(ESC), + "expected styling on a capable terminal; got: {capable:?}" + ); + assert!( + !dumb.contains(ESC), + "TERM=dumb must not be styled, got: {dumb:?}" + ); +} + +/// An explicit request outranks the capability check, for callers who know +/// their terminal better than `TERM` does. +#[cfg(target_os = "linux")] +#[test] +fn color_always_overrides_a_dumb_terminal() { + let forced = forward_list_on_pty("dumb", &["--color", "always"], true); + + assert!( + forced.contains(ESC), + "--color always must style even a dumb terminal, got: {forced:?}" + ); +} + +/// Regression test for a redirected stream inheriting the other stream's +/// terminal check. +/// +/// With stdout on a terminal and stderr redirected, `openshell ... 2> build.log` +/// must leave the log free of escapes while stdout stays styled. +#[cfg(target_os = "linux")] +#[test] +fn redirected_stderr_stays_plain_while_stdout_is_a_terminal() { + let (stdout, stderr) = split_streams_stdout_tty(&[]); + + // Positive control: stdout really is a terminal here, so something must be + // styled. Otherwise the stderr assertion could pass for the wrong reason. + assert!( + stdout.contains(ESC), + "expected styled stdout on a pty; got: {stdout:?}" + ); + assert!( + !stderr.contains(ESC), + "redirected stderr must not receive escapes, got: {stderr:?}" + ); +} + +/// `Painted` cannot identify its destination stream, so table styling is +/// deliberately disabled when either stream is redirected. +#[cfg(target_os = "linux")] +#[test] +fn status_table_is_plain_when_stderr_is_redirected() { + let stdout = forward_list_on_pty("xterm-256color", &[], false); + + assert!( + stdout.contains(SANDBOX), + "expected the seeded forward in the table, got: {stdout:?}" + ); + assert!( + !stdout.contains(ESC), + "STATUS table must stay plain when stderr is redirected, got: {stdout:?}" + ); +} + +#[test] +fn error_output_follows_the_color_setting() { + // miette renders errors to stderr through its own handler. It already + // suppresses color when stderr is redirected, but it has no way to learn + // about `--color`, so `init` installs a handler built from the setting. + let (_, plain) = failing_connect(&[]); + let (_, styled) = failing_connect(&["--color", "always"]); + + assert!( + styled.contains(ESC), + "expected `--color always` to style error output; got: {styled:?}" + ); + assert!( + !plain.contains(ESC), + "piped error output must not emit ANSI escapes, got: {plain:?}" + ); +} + +#[test] +fn status_column_is_matchable_by_a_whitespace_anchored_pattern() { + // The regression this whole change exists for: a supervisor deciding whether + // a forward is alive by matching the STATUS column. Colorized output put an + // escape byte immediately before the status word, so a pattern requiring + // whitespace ahead of it could never match. + let tmpdir = tempfile::tempdir().expect("create tmpdir"); + config_dir_with_forward(tmpdir.path()); + + let stdout = forward_list(tmpdir.path(), &[], None); + let row = stdout + .lines() + .find(|line| line.starts_with(SANDBOX)) + .unwrap_or_else(|| panic!("no row for {SANDBOX} in: {stdout:?}")); + + let status = row + .split_whitespace() + .next_back() + .expect("row has a status column"); + assert_eq!(status, "dead"); + + // Whitespace, not an escape byte, separates the PID from the status. + let boundary = row + .rfind(status) + .and_then(|index| row[..index].chars().next_back()) + .expect("status is preceded by the rest of the row"); + assert!( + boundary.is_whitespace(), + "expected whitespace before the status column, found {boundary:?} in {row:?}" + ); +} diff --git a/crates/openshell-cli/tests/ensure_providers_integration.rs b/crates/openshell-cli/tests/ensure_providers_integration.rs index 9999a7c08..7545786a1 100644 --- a/crates/openshell-cli/tests/ensure_providers_integration.rs +++ b/crates/openshell-cli/tests/ensure_providers_integration.rs @@ -7,9 +7,7 @@ mod helpers; -use helpers::{ - EnvVarGuard, build_ca, build_client_cert, build_server_cert, install_rustls_provider, -}; +use helpers::{EnvVarGuard, build_ca, build_client_cert, build_server_cert}; use openshell_cli::run; use openshell_cli::tls::TlsOptions; use openshell_core::proto::open_shell_server::{OpenShell, OpenShellServer}; @@ -81,6 +79,13 @@ impl TestOpenShell { #[tonic::async_trait] impl OpenShell for TestOpenShell { + async fn begin_rootfs_tar_staging( + &self, + _request: tonic::Request, + ) -> Result, Status> { + Err(Status::unimplemented("not used by this test server")) + } + async fn report_main_process_exit( &self, _request: tonic::Request, @@ -154,6 +159,8 @@ impl OpenShell for TestOpenShell { Ok(Response::new(ListSandboxesResponse::default())) } + unimplemented_sandbox_template_rpcs!(); + async fn list_sandbox_providers( &self, _request: tonic::Request, @@ -314,14 +321,30 @@ impl OpenShell for TestOpenShell { &self, _request: tonic::Request, ) -> Result, Status> { - Err(Status::unimplemented("not implemented in test")) + let profiles = openshell_providers::builtin_profiles() + .iter() + .map(openshell_providers::ProviderTypeProfile::to_proto) + .collect(); + Ok(Response::new( + openshell_core::proto::ListProviderProfilesResponse { profiles }, + )) } async fn get_provider_profile( &self, - _request: tonic::Request, + request: tonic::Request, ) -> Result, Status> { - Err(Status::unimplemented("not implemented in test")) + let id = request.into_inner().id; + let profile = openshell_providers::builtin_profiles() + .iter() + .find(|profile| profile.id == id) + .ok_or_else(|| Status::not_found("provider profile not found"))? + .to_proto(); + Ok(Response::new( + openshell_core::proto::ProviderProfileResponse { + profile: Some(profile), + }, + )) } async fn import_provider_profiles( @@ -708,8 +731,6 @@ struct TestServer { } async fn run_server() -> TestServer { - install_rustls_provider(); - let (ca, ca_key) = build_ca(); let (server_cert, server_key) = build_server_cert(&ca, &ca_key); let (client_cert, client_key) = build_client_cert(&ca, &ca_key); @@ -850,7 +871,7 @@ async fn explicit_provider_name_errors_for_unrecognised_name() { "error should mention the name: {msg}" ); assert!( - msg.contains("not a recognized provider type"), + msg.contains("no provider profile"), "error should explain why it failed: {msg}" ); } diff --git a/crates/openshell-cli/tests/forward_list_integration.rs b/crates/openshell-cli/tests/forward_list_integration.rs new file mode 100644 index 000000000..50abfec37 --- /dev/null +++ b/crates/openshell-cli/tests/forward_list_integration.rs @@ -0,0 +1,156 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +use std::fs; +use std::path::Path; +use std::process::{Command, Output}; + +const DEAD_PID: u32 = u32::MAX; + +fn run_forward_list(config_dir: &Path, args: &[&str]) -> Output { + let mut command = Command::new(env!("CARGO_BIN_EXE_openshell")); + for (key, _) in std::env::vars().filter(|(key, _)| key.starts_with("OPENSHELL_")) { + command.env_remove(key); + } + command + .args(["forward", "list"]) + .args(args) + .env("XDG_CONFIG_HOME", config_dir) + .env("HOME", config_dir) + .output() + .expect("run openshell forward list") +} + +fn write_dead_forward(config_dir: &Path) { + let forward_dir = config_dir.join("openshell/forwards"); + fs::create_dir_all(&forward_dir).expect("create forward directory"); + fs::write( + forward_dir.join("my-sandbox-8443.pid"), + format!("{DEAD_PID}\tsandbox-id\t0.0.0.0"), + ) + .expect("write forward PID record"); +} + +fn assert_success(output: &Output) { + assert!( + output.status.success(), + "forward list failed:\n{}", + String::from_utf8_lossy(&output.stderr) + ); +} + +#[test] +fn forward_list_json_emits_machine_readable_records() { + let config_dir = tempfile::tempdir().expect("create config directory"); + write_dead_forward(config_dir.path()); + + let output = run_forward_list(config_dir.path(), &["--output", "json"]); + assert_success(&output); + assert!( + output.stderr.is_empty(), + "structured output wrote to stderr" + ); + assert!( + !output.stdout.contains(&0x1b), + "JSON contained ANSI escapes" + ); + + let value: serde_json::Value = + serde_json::from_slice(&output.stdout).expect("stdout should contain only JSON"); + assert_eq!( + value, + serde_json::json!([{ + "sandbox": "my-sandbox", + "bind_address": "0.0.0.0", + "port": 8443, + "pid": DEAD_PID, + "alive": false, + }]) + ); +} + +#[test] +fn forward_list_yaml_emits_equivalent_records() { + let config_dir = tempfile::tempdir().expect("create config directory"); + write_dead_forward(config_dir.path()); + + let output = run_forward_list(config_dir.path(), &["-o", "yaml"]); + assert_success(&output); + assert!( + output.stderr.is_empty(), + "structured output wrote to stderr" + ); + assert!( + !output.stdout.contains(&0x1b), + "YAML contained ANSI escapes" + ); + + let value: serde_json::Value = + serde_yml::from_slice(&output.stdout).expect("stdout should contain only YAML"); + assert_eq!( + value, + serde_json::json!([{ + "sandbox": "my-sandbox", + "bind_address": "0.0.0.0", + "port": 8443, + "pid": DEAD_PID, + "alive": false, + }]) + ); +} + +#[test] +fn forward_list_structured_output_emits_empty_collections() { + let config_dir = tempfile::tempdir().expect("create config directory"); + + for format in ["json", "yaml"] { + let output = run_forward_list(config_dir.path(), &["--output", format]); + assert_success(&output); + assert!(output.stderr.is_empty(), "{format} output wrote to stderr"); + + let value: serde_json::Value = match format { + "json" => serde_json::from_slice(&output.stdout).expect("parse empty JSON collection"), + "yaml" => serde_yml::from_slice(&output.stdout).expect("parse empty YAML collection"), + _ => unreachable!(), + }; + assert_eq!(value, serde_json::json!([])); + } +} + +#[test] +fn forward_list_table_output_keeps_existing_columns() { + let config_dir = tempfile::tempdir().expect("create config directory"); + write_dead_forward(config_dir.path()); + + let output = run_forward_list(config_dir.path(), &[]); + assert_success(&output); + assert!(output.stderr.is_empty(), "table output wrote to stderr"); + + let stdout = String::from_utf8_lossy(&output.stdout); + let rows: Vec> = stdout + .lines() + .map(|line| line.split_whitespace().collect()) + .collect(); + assert_eq!(rows.len(), 2); + assert_eq!(rows[0], vec!["SANDBOX", "BIND", "PORT", "PID", "STATUS"]); + assert_eq!( + &rows[1][..4], + ["my-sandbox", "0.0.0.0", "8443", "4294967295"] + ); + assert!(rows[1][4].contains("dead")); +} + +#[test] +fn forward_list_help_documents_output_flag() { + let config_dir = tempfile::tempdir().expect("create config directory"); + let output = run_forward_list(config_dir.path(), &["--help"]); + assert_success(&output); + + let stdout = String::from_utf8(output.stdout).expect("help should be UTF-8"); + assert!(stdout.contains("-o, --output "), "{stdout}"); + assert!(stdout.contains("[default: table]"), "{stdout}"); + assert!( + stdout.contains("possible values: table, yaml, json"), + "{stdout}" + ); +} diff --git a/crates/openshell-cli/tests/helpers/mod.rs b/crates/openshell-cli/tests/helpers/mod.rs index a58e750b9..e37b84a7b 100644 --- a/crates/openshell-cli/tests/helpers/mod.rs +++ b/crates/openshell-cli/tests/helpers/mod.rs @@ -8,6 +8,95 @@ //! mod helpers; //! ``` +#[macro_export] +macro_rules! unimplemented_sandbox_template_rpcs { + () => { + fn create_sandbox_template<'life0, 'async_trait>( + &'life0 self, + _request: tonic::Request, + ) -> std::pin::Pin< + Box< + dyn std::future::Future< + Output = Result< + tonic::Response, + tonic::Status, + >, + > + Send + + 'async_trait, + >, + > + where + 'life0: 'async_trait, + Self: 'async_trait, + { + Box::pin(async { Err(tonic::Status::unimplemented("unused")) }) + } + + fn get_sandbox_template<'life0, 'async_trait>( + &'life0 self, + _request: tonic::Request, + ) -> std::pin::Pin< + Box< + dyn std::future::Future< + Output = Result< + tonic::Response, + tonic::Status, + >, + > + Send + + 'async_trait, + >, + > + where + 'life0: 'async_trait, + Self: 'async_trait, + { + Box::pin(async { Err(tonic::Status::unimplemented("unused")) }) + } + + fn list_sandbox_templates<'life0, 'async_trait>( + &'life0 self, + _request: tonic::Request, + ) -> std::pin::Pin< + Box< + dyn std::future::Future< + Output = Result< + tonic::Response, + tonic::Status, + >, + > + Send + + 'async_trait, + >, + > + where + 'life0: 'async_trait, + Self: 'async_trait, + { + Box::pin(async { Err(tonic::Status::unimplemented("unused")) }) + } + + fn delete_sandbox_template<'life0, 'async_trait>( + &'life0 self, + _request: tonic::Request, + ) -> std::pin::Pin< + Box< + dyn std::future::Future< + Output = Result< + tonic::Response, + tonic::Status, + >, + > + Send + + 'async_trait, + >, + > + where + 'life0: 'async_trait, + Self: 'async_trait, + { + Box::pin(async { Err(tonic::Status::unimplemented("unused")) }) + } + }; +} + use rcgen::{ BasicConstraints, Certificate, CertificateParams, ExtendedKeyUsagePurpose, IsCa, KeyPair, }; @@ -71,14 +160,6 @@ impl Drop for EnvVarGuard { // ── TLS helpers ────────────────────────────────────────────────────────────── -/// Install the `rustls` ring crypto provider as the process default. -/// -/// Safe to call multiple times — subsequent calls are no-ops. -#[allow(dead_code)] -pub fn install_rustls_provider() { - let _ = rustls::crypto::ring::default_provider().install_default(); -} - /// Generate a self-signed CA certificate and its key pair. #[allow(dead_code)] pub fn build_ca() -> (Certificate, KeyPair) { diff --git a/crates/openshell-cli/tests/mtls_integration.rs b/crates/openshell-cli/tests/mtls_integration.rs index e4735ac4c..29de1fe6f 100644 --- a/crates/openshell-cli/tests/mtls_integration.rs +++ b/crates/openshell-cli/tests/mtls_integration.rs @@ -3,9 +3,7 @@ mod helpers; -use helpers::{ - EnvVarGuard, build_ca, build_client_cert, build_server_cert, install_rustls_provider, -}; +use helpers::{EnvVarGuard, build_ca, build_client_cert, build_server_cert}; use openshell_bootstrap::{get_gateway_metadata, load_active_gateway}; use openshell_cli::{ run, @@ -34,6 +32,13 @@ struct TestOpenShell; #[tonic::async_trait] impl OpenShell for TestOpenShell { + async fn begin_rootfs_tar_staging( + &self, + _request: tonic::Request, + ) -> Result, Status> { + Err(Status::unimplemented("not used by this test server")) + } + async fn report_main_process_exit( &self, _request: tonic::Request, @@ -113,6 +118,8 @@ impl OpenShell for TestOpenShell { )) } + unimplemented_sandbox_template_rpcs!(); + async fn list_sandbox_providers( &self, _request: tonic::Request, @@ -642,8 +649,6 @@ fn isolated_gateway_add_env( #[tokio::test] async fn gateway_add_mtls_loopback_uses_explicit_gateway_name() { - install_rustls_provider(); - let (ca, ca_key) = build_ca(); let (server_cert, server_key) = build_server_cert(&ca, &ca_key); let (client_cert, client_key) = build_client_cert(&ca, &ca_key); @@ -686,8 +691,6 @@ async fn gateway_add_mtls_loopback_uses_explicit_gateway_name() { #[tokio::test] async fn gateway_add_mtls_loopback_without_name_uses_openshell_default() { - install_rustls_provider(); - let (ca, ca_key) = build_ca(); let (server_cert, server_key) = build_server_cert(&ca, &ca_key); let (client_cert, client_key) = build_client_cert(&ca, &ca_key); @@ -729,8 +732,6 @@ async fn gateway_add_mtls_loopback_without_name_uses_openshell_default() { #[tokio::test] async fn gateway_add_mtls_loopback_explicit_name_does_not_fallback_to_openshell_certs() { - install_rustls_provider(); - let (ca, ca_key) = build_ca(); let (client_cert, client_key) = build_client_cert(&ca, &ca_key); let ca_cert = ca.pem(); @@ -768,7 +769,6 @@ async fn gateway_add_mtls_loopback_explicit_name_does_not_fallback_to_openshell_ #[tokio::test] async fn cli_connects_with_client_cert() { let _env = EnvVarGuard::set(&[]); - install_rustls_provider(); let (ca, ca_key) = build_ca(); let (server_cert, server_key) = build_server_cert(&ca, &ca_key); @@ -794,8 +794,6 @@ async fn cli_connects_with_client_cert() { #[tokio::test] async fn cli_requires_client_cert_for_https() { - install_rustls_provider(); - let (ca, ca_key) = build_ca(); let (server_cert, server_key) = build_server_cert(&ca, &ca_key); let ca_cert = ca.pem(); @@ -842,7 +840,6 @@ async fn run_server_no_client_auth( #[tokio::test] async fn cli_connects_with_gateway_insecure() { let _env = EnvVarGuard::set(&[]); - install_rustls_provider(); let (ca, ca_key) = build_ca(); let (server_cert, server_key) = build_server_cert(&ca, &ca_key); diff --git a/crates/openshell-cli/tests/provider_commands_integration.rs b/crates/openshell-cli/tests/provider_commands_integration.rs index 84fb8f916..912835faa 100644 --- a/crates/openshell-cli/tests/provider_commands_integration.rs +++ b/crates/openshell-cli/tests/provider_commands_integration.rs @@ -3,9 +3,7 @@ mod helpers; -use helpers::{ - EnvVarGuard, build_ca, build_client_cert, build_server_cert, install_rustls_provider, -}; +use helpers::{EnvVarGuard, build_ca, build_client_cert, build_server_cert}; use openshell_cli::run; use openshell_cli::tls::TlsOptions; use openshell_core::proto::open_shell_server::{OpenShell, OpenShellServer}; @@ -23,11 +21,12 @@ use openshell_core::proto::{ HealthRequest, HealthResponse, ListProvidersRequest, ListProvidersResponse, ListSandboxProvidersRequest, ListSandboxProvidersResponse, ListSandboxesRequest, ListSandboxesResponse, Provider, ProviderCredentialRefresh, ProviderCredentialRefreshStatus, - ProviderCredentialRefreshStrategy, ProviderProfile, ProviderProfileCredential, - ProviderProfileDiscovery, ProviderResponse, RevokeSshSessionRequest, RevokeSshSessionResponse, - RotateProviderCredentialRequest, RotateProviderCredentialResponse, Sandbox, SandboxResponse, - SandboxStreamEvent, ServiceStatus, SettingValue, SupervisorMessage, UpdateProviderRequest, - WatchSandboxRequest, setting_value, + ProviderCredentialRefreshStrategy, ProviderCredentialTokenGrant, + ProviderCredentialTokenGrantSubjectToken, ProviderCredentialTokenGrantType, ProviderProfile, + ProviderProfileCredential, ProviderProfileDiscovery, ProviderResponse, RevokeSshSessionRequest, + RevokeSshSessionResponse, RotateProviderCredentialRequest, RotateProviderCredentialResponse, + Sandbox, SandboxResponse, SandboxStreamEvent, ServiceStatus, SettingValue, SupervisorMessage, + UpdateProviderRequest, WatchSandboxRequest, }; use openshell_core::{ObjectId, ObjectName}; use std::collections::HashMap; @@ -44,14 +43,17 @@ use tonic::{Response, Status}; struct ProviderState { providers: Arc>>, profiles: Arc>>, + scoped_profiles: Arc>>, refresh_statuses: Arc>>, refresh_requests: Arc>>, provider_update_requests: Arc>>, deny_provider_reads: Arc, delete_provider_requests: Arc>>, + delete_provider_profile_requests: Arc>>, fail_configure_refresh_message: Arc>>, fail_rotate_refresh_message: Arc>>, fail_delete_provider_message: Arc>>, + fail_delete_provider_profile_message: Arc>>, sandbox_providers: Arc>>>, sandbox_provider_requests: Arc>>, global_settings: Arc>>, @@ -102,6 +104,13 @@ struct TestOpenShell { #[tonic::async_trait] impl OpenShell for TestOpenShell { + async fn begin_rootfs_tar_staging( + &self, + _request: tonic::Request, + ) -> Result, Status> { + Err(Status::unimplemented("not used by this test server")) + } + async fn report_main_process_exit( &self, _request: tonic::Request, @@ -181,6 +190,7 @@ impl OpenShell for TestOpenShell { }), spec: None, status: None, + ..Sandbox::default() }), })) } @@ -192,6 +202,8 @@ impl OpenShell for TestOpenShell { Ok(Response::new(ListSandboxesResponse::default())) } + unimplemented_sandbox_template_rpcs!(); + async fn list_sandbox_providers( &self, request: tonic::Request, @@ -489,8 +501,18 @@ impl OpenShell for TestOpenShell { &self, request: tonic::Request, ) -> Result, Status> { - let id = request.into_inner().id; - let profile = if let Some(profile) = openshell_providers::builtin_profiles() + let request = request.into_inner(); + let id = request.id; + let scoped_profile = self + .state + .scoped_profiles + .lock() + .await + .get(&(request.workspace, id.clone())) + .cloned(); + let profile = if let Some(profile) = scoped_profile { + profile + } else if let Some(profile) = openshell_providers::builtin_profiles() .iter() .find(|profile| profile.id == id) { @@ -876,6 +898,20 @@ impl OpenShell for TestOpenShell { request: tonic::Request, ) -> Result, Status> { let id = request.into_inner().id; + self.state + .delete_provider_profile_requests + .lock() + .await + .push(id.clone()); + let delete_failure = self + .state + .fail_delete_provider_profile_message + .lock() + .await + .take(); + if let Some(message) = delete_failure { + return Err(Status::internal(message)); + } let deleted = self.state.profiles.lock().await.remove(&id).is_some(); Ok(Response::new( openshell_core::proto::DeleteProviderProfileResponse { deleted }, @@ -1124,8 +1160,6 @@ struct TestServer { } async fn run_server() -> TestServer { - install_rustls_provider(); - let (ca, ca_key) = build_ca(); let (server_cert, server_key) = build_server_cert(&ca, &ca_key); let (client_cert, client_key) = build_client_cert(&ca, &ca_key); @@ -1173,15 +1207,132 @@ async fn run_server() -> TestServer { } } -async fn enable_providers_v2(ts: &TestServer) { - ts.state.global_settings.lock().await.insert( - openshell_core::settings::PROVIDERS_V2_ENABLED_KEY.to_string(), - SettingValue { - value: Some(setting_value::Value::BoolValue(true)), +async fn install_test_profile(ts: &TestServer, id: &str, credential_key: &str) { + ts.state.profiles.lock().await.insert( + id.to_string(), + ProviderProfile { + id: id.to_string(), + display_name: id.to_string(), + credentials: vec![ProviderProfileCredential { + name: "access_token".to_string(), + env_vars: vec![credential_key.to_string()], + required: true, + ..Default::default() + }], + ..Default::default() }, ); } +/// A readable provider must carry its stored type and profile workspace into +/// the update request. Policy interceptors evaluate the request before the +/// gateway merges it with stored state, so an update that omits them cannot be +/// authorized against the profile that owns the provider. +/// +/// The stored `profile_workspace` is forwarded verbatim rather than recomputed +/// from the request workspace. The gateway treats it as immutable, so deriving +/// it here would look like a change and be rejected. +#[tokio::test] +async fn provider_update_preserves_stored_type_and_profile_workspace_when_readable() { + let ts = run_server().await; + + run::provider_create( + &ts.endpoint, + "my-claude", + "claude", + false, + &["API_KEY=abc".to_string()], + false, + &[], + "default", + &ts.tls, + ) + .await + .expect("provider create"); + + run::provider_update(run::ProviderUpdateOptions { + server: &ts.endpoint, + name: "my-claude", + from_existing: false, + from_oidc_token: false, + credentials: &["API_KEY=rotated".to_string()], + config: &[], + credential_expires_at: &[], + workspace: "default", + tls: &ts.tls, + }) + .await + .expect("provider update"); + + let requests = ts.state.provider_update_requests.lock().await; + let request = requests.last().expect("provider update request"); + // `claude` normalizes to the canonical `claude-code` at creation, so the + // update carries the stored type rather than the alias the caller typed. + assert_eq!(request.r#type, "claude-code"); + // Forwarded verbatim rather than recomputed. The gateway treats + // profile_workspace as immutable, so any substitution here would look like + // a change and be rejected. + let stored = ts.state.providers.lock().await; + let stored = stored.get("my-claude").expect("stored provider"); + assert_eq!(request.profile_workspace, stored.profile_workspace); +} + +#[tokio::test] +async fn provider_delete_continues_after_entry_failure() { + let ts = run_server().await; + *ts.state.fail_delete_provider_message.lock().await = + Some("simulated provider delete failure".to_string()); + + let err = run::provider_delete( + &ts.endpoint, + &["failing-provider".to_string(), "later-provider".to_string()], + "default", + &ts.tls, + ) + .await + .expect_err("provider delete should report aggregate failure"); + + let msg = err.to_string(); + assert!( + msg.contains("failed to delete 1 provider: failing-provider"), + "unexpected error: {msg}" + ); + assert_eq!( + ts.state.delete_provider_requests.lock().await.clone(), + vec!["failing-provider".to_string(), "later-provider".to_string()] + ); +} + +#[tokio::test] +async fn provider_profile_delete_continues_after_entry_failure() { + let ts = run_server().await; + *ts.state.fail_delete_provider_profile_message.lock().await = + Some("simulated provider profile delete failure".to_string()); + + let err = run::provider_profile_delete( + &ts.endpoint, + &["failing-profile".to_string(), "later-profile".to_string()], + "default", + &ts.tls, + ) + .await + .expect_err("provider profile delete should report aggregate failure"); + + let msg = err.to_string(); + assert!( + msg.contains("failed to delete 1 provider profile: failing-profile"), + "unexpected error: {msg}" + ); + assert_eq!( + ts.state + .delete_provider_profile_requests + .lock() + .await + .clone(), + vec!["failing-profile".to_string(), "later-profile".to_string()] + ); +} + #[tokio::test] async fn provider_cli_run_functions_support_full_crud_flow() { let ts = run_server().await; @@ -1362,11 +1513,12 @@ async fn provider_list_json_empty() { #[tokio::test] async fn provider_refresh_cli_run_functions_wire_requests() { let ts = run_server().await; + install_test_profile(&ts, "custom-graph", "MS_GRAPH_ACCESS_TOKEN").await; run::provider_create( &ts.endpoint, "my-graph", - "outlook", + "custom-graph", false, &["MS_GRAPH_ACCESS_TOKEN=token".to_string()], false, @@ -1451,11 +1603,12 @@ async fn provider_refresh_cli_run_functions_wire_requests() { #[tokio::test] async fn provider_refresh_configure_reads_secret_material_from_env_off_argv() { let ts = run_server().await; + install_test_profile(&ts, "custom-chat", "GOOGLE_CHAT_ACCESS_TOKEN").await; run::provider_create( &ts.endpoint, "gc-bridge", - "outlook", + "custom-chat", false, &["GOOGLE_CHAT_ACCESS_TOKEN=pending".to_string()], false, @@ -1606,7 +1759,7 @@ async fn provider_create_allows_empty_credentials_for_gateway_refresh_profiles() } #[tokio::test] -async fn provider_create_requires_runtime_credentials_for_empty_gateway_refresh_profiles() { +async fn provider_create_allows_no_source_for_runtime_resolved_profiles() { let ts = run_server().await; ts.state.profiles.lock().await.insert( "custom-refresh".to_string(), @@ -1626,7 +1779,7 @@ async fn provider_create_requires_runtime_credentials_for_empty_gateway_refresh_ }, ); - let err = run::provider_create( + run::provider_create( &ts.endpoint, "custom-refresh-provider", "custom-refresh", @@ -1638,11 +1791,10 @@ async fn provider_create_requires_runtime_credentials_for_empty_gateway_refresh_ &ts.tls, ) .await - .expect_err("empty runtime-resolved providers should require an explicit source"); + .expect("runtime-resolved provider should not require a credential source"); - assert!(err.to_string().contains("--runtime-credentials")); assert!( - !ts.state + ts.state .providers .lock() .await @@ -1650,6 +1802,36 @@ async fn provider_create_requires_runtime_credentials_for_empty_gateway_refresh_ ); } +#[tokio::test] +async fn provider_create_allows_credentialless_policy_profile() { + let ts = run_server().await; + + run::provider_create( + &ts.endpoint, + "pypi", + "pypi", + false, + &[], + false, + &[], + "default", + &ts.tls, + ) + .await + .expect("credential-less provider create"); + + let provider = ts + .state + .providers + .lock() + .await + .get("pypi") + .cloned() + .expect("pypi provider"); + assert!(provider.credentials.is_empty()); + assert_eq!(provider.r#type, "pypi"); +} + #[tokio::test] async fn sandbox_provider_cli_run_functions_wire_requests_and_idempotent_results() { let ts = run_server().await; @@ -1686,7 +1868,7 @@ async fn sandbox_provider_cli_run_functions_wire_requests_and_idempotent_results ) .await .expect("sandbox provider attach is idempotent"); - run::sandbox_provider_list(&ts.endpoint, "dev-sandbox", "default", &ts.tls) + run::sandbox_provider_list(&ts.endpoint, "dev-sandbox", "table", "default", &ts.tls) .await .expect("sandbox provider list"); run::sandbox_provider_detach( @@ -1858,6 +2040,21 @@ binaries: [/usr/bin/custom] .expect("custom provider should be stored"); assert_eq!(provider.r#type, "custom-api"); + let mut custom_alt_profile = ts + .state + .profiles + .lock() + .await + .get("custom-api") + .cloned() + .expect("custom-api profile should be stored"); + custom_alt_profile.id = "custom-alt".to_string(); + ts.state + .profiles + .lock() + .await + .insert("custom-alt".to_string(), custom_alt_profile); + run::provider_delete( &ts.endpoint, &["custom-provider".to_string()], @@ -1866,15 +2063,22 @@ binaries: [/usr/bin/custom] ) .await .expect("custom provider delete"); - run::provider_profile_delete(&ts.endpoint, "custom-api", "default", &ts.tls) - .await - .expect("profile delete"); + run::provider_profile_delete( + &ts.endpoint, + &["custom-api".to_string(), "custom-alt".to_string()], + "default", + &ts.tls, + ) + .await + .expect("profile delete"); + let profiles = ts.state.profiles.lock().await; + assert!(!profiles.contains_key("custom-api")); + assert!(!profiles.contains_key("custom-alt")); } #[tokio::test] -async fn provider_create_from_existing_uses_profile_discovery_when_v2_enabled() { +async fn provider_create_from_existing_uses_profile_discovery() { let ts = run_server().await; - enable_providers_v2(&ts).await; ts.state.profiles.lock().await.insert( "custom-discovery".to_string(), ProviderProfile { @@ -1924,7 +2128,7 @@ async fn provider_create_from_existing_uses_profile_discovery_when_v2_enabled() } #[tokio::test] -async fn provider_create_from_existing_uses_registry_discovery_when_v2_disabled() { +async fn provider_create_from_existing_uses_builtin_profile_discovery() { let ts = run_server().await; let _env = EnvVarGuard::set(&[("OPENAI_API_KEY", "legacy-openai-secret")]); @@ -1958,9 +2162,8 @@ async fn provider_create_from_existing_uses_registry_discovery_when_v2_disabled( } #[tokio::test] -async fn provider_create_from_existing_vertex_discovers_credentials_and_config_when_v2_enabled() { +async fn provider_create_from_existing_vertex_discovers_credentials_and_config() { let ts = run_server().await; - enable_providers_v2(&ts).await; let _env = EnvVarGuard::set(&[ ("VERTEX_AI_TOKEN", "ya29.vertex-v2-fallback"), ("VERTEX_AI_PROJECT_ID", "vertex-v2-project"), @@ -2021,9 +2224,8 @@ async fn provider_create_from_existing_vertex_discovers_credentials_and_config_w } #[tokio::test] -async fn provider_create_from_existing_requires_profile_when_v2_enabled() { +async fn provider_create_from_existing_requires_profile() { let ts = run_server().await; - enable_providers_v2(&ts).await; // Use "generic" which is a normalised type but has no built-in provider // profile, so v2 profile-based discovery fails with the expected message. let _env = EnvVarGuard::set(&[("GENERIC_API_KEY", "some-secret")]); @@ -2044,7 +2246,7 @@ async fn provider_create_from_existing_requires_profile_when_v2_enabled() { assert!( err.to_string() - .contains("providers v2 discovery requires a provider profile"), + .contains("import a matching profile before using this provider type"), "unexpected error: {err}" ); assert!(!ts.state.providers.lock().await.contains_key("v2-generic")); @@ -2053,7 +2255,6 @@ async fn provider_create_from_existing_requires_profile_when_v2_enabled() { #[tokio::test] async fn provider_create_from_existing_fails_when_profile_discovery_finds_nothing() { let ts = run_server().await; - enable_providers_v2(&ts).await; ts.state.profiles.lock().await.insert( "empty-discovery".to_string(), ProviderProfile { @@ -2101,9 +2302,8 @@ async fn provider_create_from_existing_fails_when_profile_discovery_finds_nothin } #[tokio::test] -async fn provider_update_from_existing_uses_profile_discovery_when_v2_enabled() { +async fn provider_update_from_existing_uses_profile_discovery() { let ts = run_server().await; - enable_providers_v2(&ts).await; ts.state.profiles.lock().await.insert( "custom-update-discovery".to_string(), ProviderProfile { @@ -2167,6 +2367,147 @@ async fn provider_update_from_existing_uses_profile_discovery_when_v2_enabled() ); } +#[tokio::test] +async fn provider_update_from_existing_preserves_global_profile_scope() { + let ts = run_server().await; + let profile_id = "shadowed-update-discovery"; + for (profile_workspace, env_var) in [ + ("", "GLOBAL_UPDATE_DISCOVERY_API_KEY"), + ("default", "WORKSPACE_UPDATE_DISCOVERY_API_KEY"), + ] { + ts.state.scoped_profiles.lock().await.insert( + (profile_workspace.to_string(), profile_id.to_string()), + ProviderProfile { + id: profile_id.to_string(), + credentials: vec![ProviderProfileCredential { + name: "api_key".to_string(), + env_vars: vec![env_var.to_string()], + required: true, + ..Default::default() + }], + discovery: Some(ProviderProfileDiscovery { + credentials: vec!["api_key".to_string()], + }), + ..Default::default() + }, + ); + } + ts.state.providers.lock().await.insert( + "global-update".to_string(), + Provider { + metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { + id: "id-global-update".to_string(), + name: "global-update".to_string(), + workspace: "default".to_string(), + ..Default::default() + }), + r#type: profile_id.to_string(), + profile_workspace: String::new(), + ..Default::default() + }, + ); + let _env = EnvVarGuard::set(&[ + ("GLOBAL_UPDATE_DISCOVERY_API_KEY", "global-secret"), + ("WORKSPACE_UPDATE_DISCOVERY_API_KEY", "workspace-secret"), + ]); + + run::provider_update(run::ProviderUpdateOptions { + server: &ts.endpoint, + name: "global-update", + from_existing: true, + from_oidc_token: false, + credentials: &[], + config: &[], + credential_expires_at: &[], + workspace: "default", + tls: &ts.tls, + }) + .await + .expect("global profile-backed provider update --from-existing"); + + let provider = ts + .state + .providers + .lock() + .await + .get("global-update") + .cloned() + .expect("global provider should still be stored"); + assert_eq!( + provider.credentials.get("GLOBAL_UPDATE_DISCOVERY_API_KEY"), + Some(&"global-secret".to_string()) + ); + assert!( + !provider + .credentials + .contains_key("WORKSPACE_UPDATE_DISCOVERY_API_KEY") + ); +} + +#[tokio::test] +async fn provider_update_from_oidc_token_preserves_global_profile_scope() { + let ts = run_server().await; + let profile_id = "shadowed-oidc-update"; + for (profile_workspace, subject_credential) in [ + ("", "GLOBAL_SUBJECT_TOKEN"), + ("default", "WORKSPACE_SUBJECT_TOKEN"), + ] { + ts.state.scoped_profiles.lock().await.insert( + (profile_workspace.to_string(), profile_id.to_string()), + ProviderProfile { + id: profile_id.to_string(), + credentials: vec![ProviderProfileCredential { + name: "dynamic_token".to_string(), + token_grant: Some(ProviderCredentialTokenGrant { + grant_type: ProviderCredentialTokenGrantType::TokenExchange as i32, + subject_token: Some(ProviderCredentialTokenGrantSubjectToken { + source: "provider_credential".to_string(), + credential: subject_credential.to_string(), + ..Default::default() + }), + ..Default::default() + }), + ..Default::default() + }], + ..Default::default() + }, + ); + } + ts.state.providers.lock().await.insert( + "global-oidc-update".to_string(), + Provider { + metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { + id: "id-global-oidc-update".to_string(), + name: "global-oidc-update".to_string(), + workspace: "default".to_string(), + ..Default::default() + }), + r#type: profile_id.to_string(), + profile_workspace: String::new(), + ..Default::default() + }, + ); + + let err = run::provider_update(run::ProviderUpdateOptions { + server: &ts.endpoint, + name: "global-oidc-update", + from_existing: false, + from_oidc_token: true, + credentials: &["GLOBAL_SUBJECT_TOKEN".to_string()], + config: &[], + credential_expires_at: &[], + workspace: "default", + tls: &ts.tls, + }) + .await + .expect_err("unnamed test gateway should stop after profile validation"); + + assert!( + err.to_string().contains("active named OIDC gateway"), + "global profile should accept GLOBAL_SUBJECT_TOKEN before OIDC loading: {err}" + ); +} + #[tokio::test] async fn provider_profile_import_from_directory_imports_supported_profile_files() { let ts = run_server().await; @@ -2363,11 +2704,11 @@ async fn provider_create_rejects_key_only_credentials_without_local_env_value() } #[tokio::test] -async fn provider_create_supports_generic_type_and_env_lookup_credentials() { +async fn provider_create_rejects_profileless_generic_type() { let ts = run_server().await; let _guard = EnvVarGuard::set(&[("NAV_GENERIC_TEST_KEY", "generic-value")]); - run::provider_create( + let err = run::provider_create( &ts.endpoint, "my-generic", "generic", @@ -2379,24 +2720,12 @@ async fn provider_create_supports_generic_type_and_env_lookup_credentials() { &ts.tls, ) .await - .expect("provider create"); + .expect_err("profileless generic provider creation should fail"); - let mut client = openshell_cli::tls::grpc_client(&ts.endpoint, &ts.tls) - .await - .expect("grpc client should connect"); - let response = client - .get_provider(GetProviderRequest { - name: "my-generic".to_string(), - workspace: String::new(), - }) - .await - .expect("get provider should succeed") - .into_inner(); - let provider = response.provider.expect("provider should exist"); - assert_eq!(provider.r#type, "generic"); - assert_eq!( - provider.credentials.get("NAV_GENERIC_TEST_KEY"), - Some(&"generic-value".to_string()) + assert!( + err.to_string() + .contains("provider profile 'generic' not found"), + "unexpected error: {err}" ); } @@ -2435,6 +2764,34 @@ async fn provider_create_sends_inline_credentials() { ); } +#[tokio::test] +async fn provider_create_prefers_exact_imported_alias_profile() { + let ts = run_server().await; + install_test_profile(&ts, "gh", "GITHUB_TOKEN").await; + + run::provider_create_with_options(run::ProviderCreateOptions { + server: &ts.endpoint, + name: "enterprise-github", + provider_type: "gh", + credentials: &["GITHUB_TOKEN=test-token".to_string()], + credential_source: run::ProviderCreateCredentialSource::ExplicitCredentials, + config: &[], + workspace: "default", + profile_workspace: "default", + tls: &ts.tls, + }) + .await + .expect("create provider from exact imported alias profile"); + + let stored = ts.state.providers.lock().await; + let provider = stored.get("enterprise-github").expect("provider"); + assert_eq!(provider.r#type, "gh"); + assert_eq!( + provider.credentials.get("GITHUB_TOKEN").map(String::as_str), + Some("test-token") + ); +} + #[tokio::test] async fn provider_create_rejects_combined_from_existing_and_credentials() { let ts = run_server().await; @@ -2515,14 +2872,14 @@ async fn provider_create_rejects_combined_from_gcloud_adc_and_credentials() { #[tokio::test] async fn provider_create_rejects_empty_env_var_for_key_only_credential() { let ts = run_server().await; - let _guard = EnvVarGuard::set(&[("NAV_EMPTY_ENV_KEY", "")]); + let _guard = EnvVarGuard::set(&[("NVIDIA_API_KEY", "")]); let err = run::provider_create( &ts.endpoint, "bad-provider", - "generic", + "nvidia", false, - &["NAV_EMPTY_ENV_KEY".to_string()], + &["NVIDIA_API_KEY".to_string()], false, &[], "default", @@ -2533,7 +2890,7 @@ async fn provider_create_rejects_empty_env_var_for_key_only_credential() { assert!( err.to_string() - .contains("requires local env var 'NAV_EMPTY_ENV_KEY' to be set to a non-empty value"), + .contains("requires local env var 'NVIDIA_API_KEY' to be set to a non-empty value"), "unexpected error: {err}" ); } @@ -2563,7 +2920,7 @@ async fn provider_create_supports_nvidia_type_with_nvidia_api_key() { let response = client .get_provider(GetProviderRequest { name: "my-nvidia".to_string(), - workspace: String::new(), + workspace_scope: Some(openshell_core::proto::workspace_selector("default")), }) .await .expect("get provider should succeed") @@ -2928,7 +3285,6 @@ async fn provider_create_from_gcloud_adc_rolls_back_provider_when_initial_rotate #[tokio::test] async fn provider_create_from_existing_vertex_config_only_reports_missing_vertex_credentials() { let ts = run_server().await; - enable_providers_v2(&ts).await; let _env = EnvVarGuard::set(&[ ("VERTEX_AI_PROJECT_ID", "vertex-config-only-project"), ("VERTEX_AI_REGION", "us-central1"), diff --git a/crates/openshell-cli/tests/sandbox_create_lifecycle_integration.rs b/crates/openshell-cli/tests/sandbox_create_lifecycle_integration.rs index 118daff90..6ed9a7433 100644 --- a/crates/openshell-cli/tests/sandbox_create_lifecycle_integration.rs +++ b/crates/openshell-cli/tests/sandbox_create_lifecycle_integration.rs @@ -5,30 +5,29 @@ mod helpers; -use helpers::{ - EnvVarGuard, build_ca, build_client_cert, build_server_cert, install_rustls_provider, -}; +use helpers::{EnvVarGuard, build_ca, build_client_cert, build_server_cert}; use openshell_bootstrap::load_last_sandbox; use openshell_cli::run; use openshell_cli::tls::TlsOptions; use openshell_core::proto::open_shell_server::{OpenShell, OpenShellServer}; use openshell_core::proto::{ AttachSandboxProviderRequest, AttachSandboxProviderResponse, CreateProviderRequest, - CreateSandboxRequest, CreateSshSessionRequest, CreateSshSessionResponse, DeleteProviderRequest, - DeleteProviderResponse, DeleteSandboxRequest, DeleteSandboxResponse, - DetachSandboxProviderRequest, DetachSandboxProviderResponse, - ExchangeProviderSubjectTokenRequest, ExchangeProviderSubjectTokenResponse, ExecSandboxEvent, - ExecSandboxInput, ExecSandboxRequest, GatewayMessage, GetGatewayConfigRequest, - GetGatewayConfigResponse, GetProviderRequest, GetSandboxConfigRequest, - GetSandboxConfigResponse, GetSandboxProviderEnvironmentRequest, - GetSandboxProviderEnvironmentResponse, GetSandboxRequest, GpuResourceRequirements, - HealthRequest, HealthResponse, ListProvidersRequest, ListProvidersResponse, - ListSandboxProvidersRequest, ListSandboxProvidersResponse, ListSandboxesRequest, - ListSandboxesResponse, PlatformEvent, ProviderResponse, RevokeSshSessionRequest, + CreateSandboxRequest, CreateSandboxTemplateRequest, CreateSshSessionRequest, + CreateSshSessionResponse, DeleteProviderRequest, DeleteProviderResponse, DeleteSandboxRequest, + DeleteSandboxResponse, DeleteSandboxTemplateRequest, DetachSandboxProviderRequest, + DetachSandboxProviderResponse, ExchangeProviderSubjectTokenRequest, + ExchangeProviderSubjectTokenResponse, ExecSandboxEvent, ExecSandboxInput, ExecSandboxRequest, + GatewayMessage, GetGatewayConfigRequest, GetGatewayConfigResponse, GetProviderRequest, + GetSandboxConfigRequest, GetSandboxConfigResponse, GetSandboxProviderEnvironmentRequest, + GetSandboxProviderEnvironmentResponse, GetSandboxRequest, GetSandboxTemplateRequest, + GpuResourceRequirements, HealthRequest, HealthResponse, ListProvidersRequest, + ListProvidersResponse, ListSandboxProvidersRequest, ListSandboxProvidersResponse, + ListSandboxTemplatesRequest, ListSandboxTemplatesResponse, ListSandboxesRequest, + ListSandboxesResponse, PlatformEvent, Provider, ProviderResponse, RevokeSshSessionRequest, RevokeSshSessionResponse, Sandbox, SandboxCondition, SandboxLogLine, SandboxPhase, - SandboxResponse, SandboxStatus, SandboxStreamEvent, ServiceStatus, SettingValue, - SupervisorMessage, UpdateProviderRequest, WatchSandboxRequest, sandbox_stream_event, - setting_value, + SandboxResponse, SandboxStatus, SandboxStreamEvent, SandboxTemplateResponse, + SandboxWorkloadTemplate, ServiceStatus, SettingValue, SupervisorMessage, UpdateProviderRequest, + WatchSandboxRequest, sandbox_stream_event, }; use std::collections::HashMap; use std::fs; @@ -37,25 +36,48 @@ use std::sync::Arc; use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering}; use std::time::{Duration, Instant}; use tempfile::TempDir; +use tokio::io::{AsyncReadExt, AsyncWriteExt}; use tokio::net::TcpListener; -use tokio::sync::{Mutex, mpsc}; +use tokio::sync::{Mutex, Notify, mpsc}; use tokio_stream::wrappers::TcpListenerStream; use tonic::transport::{Certificate as TlsCertificate, Identity, Server, ServerTlsConfig}; use tonic::{Response, Status}; +fn selected_workspace( + scope: &Option, +) -> Option<&str> { + match scope.as_ref()?.selection.as_ref()? { + openshell_core::proto::datamodel::v1::workspace_selector::Selection::Workspace( + workspace, + ) => Some(workspace), + openshell_core::proto::datamodel::v1::workspace_selector::Selection::AllWorkspaces(_) => { + None + } + } +} + #[derive(Clone, Default)] struct SandboxState { deleted_names: Arc>>>, create_requests: Arc>>, + fail_delete_sandbox_message: Arc>>, vm_error_after_started: Arc, vm_error_with_observed_exit: Arc, vm_slow_progress_before_ready: Arc, vm_log_churn_before_ready: Arc, terminal_before_relay: Arc, + terminal_after_provisional_container_exit: Arc, + provisional_container_exit_without_result: Arc, + provisional_container_exit_sent: Arc, ssh_session_failures_remaining: Arc, ssh_session_requests: Arc, global_settings: Arc>>, gateway_config_requests: Arc, + providers: Arc>>, + template_create_requests: Arc>>, + template_get_requests: Arc>>, + template_list_requests: Arc>>, + template_delete_requests: Arc>>, } #[derive(Clone, Default)] @@ -65,6 +87,13 @@ struct TestOpenShell { #[tonic::async_trait] impl OpenShell for TestOpenShell { + async fn begin_rootfs_tar_staging( + &self, + _request: tonic::Request, + ) -> Result, Status> { + Err(Status::unimplemented("not used by this test server")) + } + async fn report_main_process_exit( &self, _request: tonic::Request, @@ -180,6 +209,99 @@ impl OpenShell for TestOpenShell { Ok(Response::new(ListSandboxesResponse::default())) } + async fn create_sandbox_template( + &self, + request: tonic::Request, + ) -> Result, Status> { + let request = request.into_inner(); + let mut template = request.template.clone().unwrap_or_default(); + let name = template + .metadata + .as_ref() + .map_or_else(|| "template".to_string(), |metadata| metadata.name.clone()); + template.metadata = Some(openshell_core::proto::datamodel::v1::ObjectMeta { + id: format!("template-{name}"), + name, + created_at_ms: 0, + labels: template + .metadata + .as_ref() + .map(|metadata| metadata.labels.clone()) + .unwrap_or_default(), + resource_version: 1, + annotations: HashMap::new(), + workspace: selected_workspace(&request.workspace_scope) + .unwrap_or("default") + .to_string(), + deletion_timestamp_ms: 0, + }); + self.state + .template_create_requests + .lock() + .await + .push(request); + Ok(Response::new(SandboxTemplateResponse { + template: Some(template), + })) + } + + async fn get_sandbox_template( + &self, + request: tonic::Request, + ) -> Result, Status> { + let request = request.into_inner(); + self.state + .template_get_requests + .lock() + .await + .push(request.clone()); + Ok(Response::new(SandboxTemplateResponse { + template: Some(SandboxWorkloadTemplate { + metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { + id: format!("template-{}", request.name), + name: request.name, + created_at_ms: 0, + labels: HashMap::new(), + resource_version: 1, + annotations: HashMap::new(), + workspace: selected_workspace(&request.workspace_scope) + .unwrap_or("default") + .to_string(), + deletion_timestamp_ms: 0, + }), + spec: None, + }), + })) + } + + async fn list_sandbox_templates( + &self, + request: tonic::Request, + ) -> Result, Status> { + self.state + .template_list_requests + .lock() + .await + .push(request.into_inner()); + Ok(Response::new(ListSandboxTemplatesResponse { + templates: Vec::new(), + })) + } + + async fn delete_sandbox_template( + &self, + request: tonic::Request, + ) -> Result, Status> { + self.state + .template_delete_requests + .lock() + .await + .push(request.into_inner()); + Ok(Response::new( + openshell_core::proto::DeleteSandboxTemplateResponse { deleted: true }, + )) + } + async fn list_sandbox_providers( &self, _request: tonic::Request, @@ -210,7 +332,11 @@ impl OpenShell for TestOpenShell { .deleted_names .lock() .await - .push(vec![request.name]); + .push(vec![request.name.clone()]); + let delete_failure = self.state.fail_delete_sandbox_message.lock().await.take(); + if let Some(message) = delete_failure { + return Err(Status::internal(message)); + } Ok(Response::new(DeleteSandboxResponse { deleted: true })) } @@ -333,21 +459,39 @@ impl OpenShell for TestOpenShell { &self, _request: tonic::Request, ) -> Result, Status> { - Ok(Response::new(ListProvidersResponse::default())) + Ok(Response::new(ListProvidersResponse { + providers: self.state.providers.lock().await.clone(), + })) } async fn list_provider_profiles( &self, _request: tonic::Request, ) -> Result, Status> { - Err(Status::unimplemented("not implemented in test")) + let profiles = openshell_providers::builtin_profiles() + .iter() + .map(openshell_providers::ProviderTypeProfile::to_proto) + .collect(); + Ok(Response::new( + openshell_core::proto::ListProviderProfilesResponse { profiles }, + )) } async fn get_provider_profile( &self, - _request: tonic::Request, + request: tonic::Request, ) -> Result, Status> { - Err(Status::unimplemented("not implemented in test")) + let id = request.into_inner().id; + let profile = openshell_providers::builtin_profiles() + .iter() + .find(|profile| profile.id == id) + .ok_or_else(|| Status::not_found("provider profile not found"))? + .to_proto(); + Ok(Response::new( + openshell_core::proto::ProviderProfileResponse { + profile: Some(profile), + }, + )) } async fn import_provider_profiles( @@ -443,6 +587,16 @@ impl OpenShell for TestOpenShell { .load(Ordering::SeqCst); let vm_log_churn_before_ready = self.state.vm_log_churn_before_ready.load(Ordering::SeqCst); let terminal_before_relay = self.state.terminal_before_relay.load(Ordering::SeqCst); + let terminal_after_provisional_container_exit = self + .state + .terminal_after_provisional_container_exit + .load(Ordering::SeqCst); + let provisional_container_exit_without_result = self + .state + .provisional_container_exit_without_result + .load(Ordering::SeqCst); + let provisional_container_exit_sent = + Arc::clone(&self.state.provisional_container_exit_sent); tokio::spawn(async move { let mut provisioning = Sandbox { @@ -486,11 +640,44 @@ impl OpenShell for TestOpenShell { }); completed.set_phase(SandboxPhase::Completed as i32); + let mut provisional_container_exit = error.clone(); + if let Some(ready) = provisional_container_exit + .status + .as_mut() + .and_then(|status| status.conditions.first_mut()) + { + ready.reason = "ContainerExited".to_string(); + ready.message = "Sandbox container exited".to_string(); + } + let _ = tx .send(Ok(SandboxStreamEvent { payload: Some(sandbox_stream_event::Payload::Sandbox(provisioning)), })) .await; + if terminal_after_provisional_container_exit + || provisional_container_exit_without_result + { + let _ = tx + .send(Ok(SandboxStreamEvent { + payload: Some(sandbox_stream_event::Payload::Sandbox( + provisional_container_exit, + )), + })) + .await; + provisional_container_exit_sent.notify_waiters(); + if provisional_container_exit_without_result { + std::future::pending::<()>().await; + return; + } + tokio::task::yield_now().await; + let _ = tx + .send(Ok(SandboxStreamEvent { + payload: Some(sandbox_stream_event::Payload::Sandbox(completed)), + })) + .await; + return; + } if vm_error_after_started { let _ = tx .send(Ok(SandboxStreamEvent { @@ -818,8 +1005,6 @@ struct TestServer { } async fn run_server() -> TestServer { - install_rustls_provider(); - let (ca, ca_key) = build_ca(); let (server_cert, server_key) = build_server_cert(&ca, &ca_key); let (client_cert, client_key) = build_client_cert(&ca, &ca_key); @@ -1189,13 +1374,61 @@ async fn create_requests(server: &TestServer) -> Vec { server.openshell.state.create_requests.lock().await.clone() } -async fn enable_providers_v2(server: &TestServer) { - server.openshell.state.global_settings.lock().await.insert( - openshell_core::settings::PROVIDERS_V2_ENABLED_KEY.to_string(), - SettingValue { - value: Some(setting_value::Value::BoolValue(true)), - }, - ); +async fn template_create_requests(server: &TestServer) -> Vec { + server + .openshell + .state + .template_create_requests + .lock() + .await + .clone() +} + +async fn template_list_requests(server: &TestServer) -> Vec { + server + .openshell + .state + .template_list_requests + .lock() + .await + .clone() +} + +async fn template_delete_requests(server: &TestServer) -> Vec { + server + .openshell + .state + .template_delete_requests + .lock() + .await + .clone() +} + +async fn add_provider(server: &TestServer, name: &str, provider_type: &str) { + server + .openshell + .state + .providers + .lock() + .await + .push(Provider { + metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { + id: format!("provider-{name}"), + name: name.to_string(), + created_at_ms: 0, + labels: HashMap::new(), + resource_version: 0, + annotations: HashMap::new(), + workspace: "default".to_string(), + deletion_timestamp_ms: 0, + }), + r#type: provider_type.to_string(), + credentials: HashMap::new(), + config: HashMap::new(), + credential_expires_at_ms: HashMap::new(), + profile_workspace: "default".to_string(), + credential_handles: HashMap::new(), + }); } fn test_tls(server: &TestServer) -> TlsOptions { @@ -1219,6 +1452,42 @@ fn test_config() -> run::SandboxCreateConfig<'static> { } } +#[tokio::test] +async fn sandbox_delete_continues_after_entry_failure() { + let server = run_server().await; + let tls = test_tls(&server); + *server + .openshell + .state + .fail_delete_sandbox_message + .lock() + .await = Some("simulated sandbox delete failure".to_string()); + + let err = run::sandbox_delete( + &server.endpoint, + &["failing-sandbox".to_string(), "later-sandbox".to_string()], + false, + "default", + &tls, + "openshell", + ) + .await + .expect_err("sandbox delete should report aggregate failure"); + + let msg = err.to_string(); + assert!( + msg.contains("failed to delete 1 sandbox: failing-sandbox"), + "unexpected error: {msg}" + ); + assert_eq!( + deleted_names(&server).await, + vec![ + vec!["failing-sandbox".to_string()], + vec!["later-sandbox".to_string()] + ] + ); +} + #[tokio::test] async fn sandbox_create_keeps_command_sessions_by_default() { let server = run_server().await; @@ -1482,6 +1751,207 @@ async fn sandbox_create_sends_driver_config_json() { ); } +#[tokio::test] +async fn sandbox_create_with_template_sends_workload_template_name() { + let server = run_server().await; + add_provider(&server, "github", "github").await; + let fake_ssh_dir = tempfile::tempdir().unwrap(); + let xdg_dir = tempfile::tempdir().unwrap(); + let _env = test_env(&fake_ssh_dir, &xdg_dir); + let tls = test_tls(&server); + install_fake_ssh(&fake_ssh_dir); + + run::sandbox_create( + &server.endpoint, + "openshell", + run::SandboxCreateConfig { + name: Some("from-template"), + template: Some("gpu-kata"), + providers: &["github".to_string()], + command: &["echo".into(), "OK".into()], + ..test_config() + }, + "default", + &tls, + ) + .await + .expect("sandbox create should succeed"); + + let requests = create_requests(&server).await; + let request = requests.first().expect("create request should be recorded"); + assert_eq!(request.workload_template_name, "gpu-kata"); + let spec = request + .spec + .as_ref() + .expect("governance spec should be sent"); + assert_eq!(spec.providers, vec!["github".to_string()]); + assert!(spec.template.is_none()); + assert!(spec.environment.is_empty()); + assert!(spec.resource_requirements.is_none()); +} + +#[tokio::test] +async fn sandbox_template_create_sends_non_default_workspace_in_scope_and_metadata() { + let server = run_server().await; + let fake_ssh_dir = tempfile::tempdir().unwrap(); + let xdg_dir = tempfile::tempdir().unwrap(); + let _env = test_env(&fake_ssh_dir, &xdg_dir); + let tls = test_tls(&server); + + run::sandbox_template_create( + &server.endpoint, + "gpu-kata", + Some("registry.example.com/agent:latest"), + Some("2"), + Some("4Gi"), + Some(GpuResourceRequirements { count: Some(1) }), + Some(r#"{"kubernetes":{"pod":{"node_selector":{"pool":"gpu"}}}}"#), + Some("5m"), + Some(3), + HashMap::from([("team".to_string(), "runtime".to_string())]), + HashMap::from([("owner".to_string(), "platform".to_string())]), + HashMap::from([("FEATURE_FLAG".to_string(), "on".to_string())]), + "table", + "team-a", + &tls, + ) + .await + .expect("template create should succeed"); + + let requests = template_create_requests(&server).await; + let request = requests + .first() + .expect("template create request should be recorded"); + assert_eq!(selected_workspace(&request.workspace_scope), Some("team-a")); + let template = request.template.as_ref().expect("template should be sent"); + let metadata = template.metadata.as_ref().expect("metadata should be sent"); + assert_eq!(metadata.name, "gpu-kata"); + assert_eq!(metadata.workspace, "team-a"); + assert_eq!(metadata.labels.get("team"), Some(&"runtime".to_string())); + assert_eq!( + metadata.annotations.get("owner"), + Some(&"platform".to_string()) + ); + + let spec = template.spec.as_ref().expect("spec should be sent"); + let workload = spec.workload.as_ref().expect("workload should be sent"); + assert_eq!(workload.image, "registry.example.com/agent:latest"); + assert_eq!( + workload.environment.get("FEATURE_FLAG"), + Some(&"on".to_string()) + ); + let resources = workload + .resources + .as_ref() + .expect("resources should be sent"); + assert_eq!(resources.cpu, "2"); + assert_eq!(resources.memory, "4Gi"); + assert_eq!(resources.gpu.as_ref().and_then(|gpu| gpu.count), Some(1)); + assert!(spec.driver_config.is_some()); + let startup = spec + .desired_service_level + .as_ref() + .and_then(|service_level| service_level.startup.as_ref()) + .expect("startup service level should be sent"); + assert_eq!(startup.max_burst, 3); + assert_eq!( + startup + .ready_within + .as_ref() + .map(|duration| duration.seconds), + Some(300) + ); +} + +#[tokio::test] +async fn sandbox_template_list_and_delete_send_workspace_requests() { + let server = run_server().await; + let fake_ssh_dir = tempfile::tempdir().unwrap(); + let xdg_dir = tempfile::tempdir().unwrap(); + let _env = test_env(&fake_ssh_dir, &xdg_dir); + let tls = test_tls(&server); + + run::sandbox_template_list( + &server.endpoint, + 25, + 5, + Some("team=runtime"), + false, + "table", + "default", + false, + &tls, + ) + .await + .expect("template list should succeed"); + run::sandbox_template_delete(&server.endpoint, &["gpu-kata".to_string()], "default", &tls) + .await + .expect("template delete should succeed"); + + let list_requests = template_list_requests(&server).await; + let list_request = list_requests + .first() + .expect("template list request should be recorded"); + assert_eq!(list_request.limit, 25); + assert_eq!(list_request.offset, 5); + assert_eq!(list_request.label_selector, "team=runtime"); + assert_eq!( + selected_workspace(&list_request.workspace_scope), + Some("default") + ); + + let delete_requests = template_delete_requests(&server).await; + let delete_request = delete_requests + .first() + .expect("template delete request should be recorded"); + assert_eq!(delete_request.name, "gpu-kata"); + assert_eq!( + selected_workspace(&delete_request.workspace_scope), + Some("default") + ); +} + +#[tokio::test] +async fn sandbox_template_create_allows_omitted_image() { + let server = run_server().await; + let fake_ssh_dir = tempfile::tempdir().unwrap(); + let xdg_dir = tempfile::tempdir().unwrap(); + let _env = test_env(&fake_ssh_dir, &xdg_dir); + let tls = test_tls(&server); + + run::sandbox_template_create( + &server.endpoint, + "base", + None, + None, + None, + None, + None, + None, + None, + HashMap::new(), + HashMap::new(), + HashMap::new(), + "table", + "default", + &tls, + ) + .await + .expect("template create without image should succeed"); + + let requests = template_create_requests(&server).await; + let request = requests + .first() + .expect("template create request should be recorded"); + let workload = request + .template + .as_ref() + .and_then(|template| template.spec.as_ref()) + .and_then(|spec| spec.workload.as_ref()) + .expect("workload should be sent"); + assert_eq!(workload.image, ""); +} + #[tokio::test] async fn sandbox_create_sends_gpu_default_request() { let server = run_server().await; @@ -1553,9 +2023,8 @@ async fn sandbox_create_sends_gpu_count_request() { } #[tokio::test] -async fn sandbox_create_does_not_infer_command_providers_when_v2_enabled() { +async fn sandbox_create_skips_inferred_provider_without_local_credentials() { let server = run_server().await; - enable_providers_v2(&server).await; let fake_ssh_dir = tempfile::tempdir().unwrap(); let xdg_dir = tempfile::tempdir().unwrap(); let _env = test_env(&fake_ssh_dir, &xdg_dir); @@ -1566,7 +2035,7 @@ async fn sandbox_create_does_not_infer_command_providers_when_v2_enabled() { &server.endpoint, "openshell", run::SandboxCreateConfig { - name: Some("v2-no-inferred-provider"), + name: Some("no-inferred-provider"), command: &["claude".into(), "--version".into()], tty_override: Some(true), ..test_config() @@ -1586,7 +2055,7 @@ async fn sandbox_create_does_not_infer_command_providers_when_v2_enabled() { .clone(); assert!( providers.is_empty(), - "providers v2 should not infer command providers, got {providers:?}" + "missing local credentials should skip inferred providers, got {providers:?}" ); } @@ -1788,6 +2257,109 @@ async fn sandbox_create_retries_terminal_attachment_until_relay_registers() { ); } +#[tokio::test] +async fn sandbox_create_waits_for_main_result_after_provisional_container_exit() { + let server = run_server().await; + server + .openshell + .state + .terminal_after_provisional_container_exit + .store(true, Ordering::SeqCst); + let fake_ssh_dir = tempfile::tempdir().unwrap(); + let xdg_dir = tempfile::tempdir().unwrap(); + let _env = test_env(&fake_ssh_dir, &xdg_dir); + let tls = test_tls(&server); + install_fake_ssh(&fake_ssh_dir); + + let exit_code = run::sandbox_create( + &server.endpoint, + "openshell", + run::SandboxCreateConfig { + name: Some("fast-ephemeral-command"), + keep: false, + command: &["echo".into(), "OK".into()], + ..test_config() + }, + "default", + &tls, + ) + .await + .expect("a provisional container exit must yield to the canonical main-process result"); + + assert_eq!(exit_code, 0); + assert_eq!( + deleted_names(&server).await, + vec![vec!["fast-ephemeral-command".to_string()]] + ); +} + +#[tokio::test] +async fn sandbox_create_bounds_provisional_container_exit_reconciliation() { + let server = run_server().await; + server + .openshell + .state + .provisional_container_exit_without_result + .store(true, Ordering::SeqCst); + let fake_ssh_dir = tempfile::tempdir().unwrap(); + let xdg_dir = tempfile::tempdir().unwrap(); + let _env = test_env(&fake_ssh_dir, &xdg_dir); + let tls = test_tls(&server); + install_fake_ssh(&fake_ssh_dir); + + let provisional_container_exit_sent = server + .openshell + .state + .provisional_container_exit_sent + .notified(); + tokio::pin!(provisional_container_exit_sent); + let command = ["echo".into(), "OK".into()]; + let create = run::sandbox_create( + &server.endpoint, + "openshell", + run::SandboxCreateConfig { + name: Some("missing-main-result"), + command: &command, + ..test_config() + }, + "default", + &tls, + ); + tokio::pin!(create); + + tokio::select! { + () = &mut provisional_container_exit_sent => {} + result = &mut create => panic!("sandbox create returned before the provisional exit was observed: {result:?}"), + } + let reconciliation_started = Instant::now(); + let result = tokio::time::timeout(Duration::from_secs(10), &mut create) + .await + .expect("provisional container exit reconciliation must remain bounded"); + let reconciliation_elapsed = reconciliation_started.elapsed(); + + let err = result + .expect_err("a missing canonical main-process result must retain the container exit error"); + + let rendered = err.to_string(); + assert!( + rendered.contains("sandbox entered error phase while provisioning"), + "unexpected error: {rendered}" + ); + assert!( + rendered.contains("ContainerExited: Sandbox container exited"), + "unexpected error: {rendered}" + ); + assert!( + !rendered.contains("timed out"), + "unexpected error: {rendered}" + ); + assert!( + reconciliation_elapsed >= Duration::from_secs(5), + "provisional container exit returned before reconciliation: {reconciliation_elapsed:?}" + ); + assert!(deleted_names(&server).await.is_empty()); +} + #[tokio::test] async fn sandbox_create_deletes_command_sessions_with_no_keep() { let server = run_server().await; @@ -2175,18 +2747,20 @@ async fn sandbox_create_env_rejects_invalid_key_name() { ); } -async fn run_cli_sandbox_create( - server: &TestServer, - name: &str, - extra_args: &[&str], -) -> std::process::Output { - let xdg_dir = tempfile::tempdir().unwrap(); +fn prepare_cli_xdg(server: &TestServer, xdg_dir: &TempDir) { let tls_dir = xdg_dir.path().join("openshell/gateways/openshell/mtls"); fs::create_dir_all(&tls_dir).unwrap(); for filename in ["ca.crt", "tls.crt", "tls.key"] { fs::copy(server.dir.path().join(filename), tls_dir.join(filename)).unwrap(); } +} +async fn run_cli_sandbox_create_with_xdg( + server: &TestServer, + xdg_dir: &TempDir, + name: &str, + extra_args: &[&str], +) -> std::process::Output { let mut cmd = tokio::process::Command::new(env!("CARGO_BIN_EXE_openshell")); for (key, _) in std::env::vars().filter(|(k, _)| k.starts_with("OPENSHELL_")) { cmd.env_remove(&key); @@ -2212,6 +2786,213 @@ async fn run_cli_sandbox_create( .unwrap() } +async fn run_cli_sandbox_create( + server: &TestServer, + name: &str, + extra_args: &[&str], +) -> std::process::Output { + let xdg_dir = tempfile::tempdir().unwrap(); + prepare_cli_xdg(server, &xdg_dir); + run_cli_sandbox_create_with_xdg(server, &xdg_dir, name, extra_args).await +} + +async fn run_cli_sandbox_template_create( + server: &TestServer, + name: &str, + extra_args: &[&str], +) -> std::process::Output { + let xdg_dir = tempfile::tempdir().unwrap(); + let tls_dir = xdg_dir.path().join("openshell/gateways/openshell/mtls"); + fs::create_dir_all(&tls_dir).unwrap(); + for filename in ["ca.crt", "tls.crt", "tls.key"] { + fs::copy(server.dir.path().join(filename), tls_dir.join(filename)).unwrap(); + } + + let mut cmd = tokio::process::Command::new(env!("CARGO_BIN_EXE_openshell")); + for (key, _) in std::env::vars().filter(|(k, _)| k.starts_with("OPENSHELL_")) { + cmd.env_remove(&key); + } + cmd.args([ + "--gateway", + "openshell", + "--gateway-endpoint", + &server.endpoint, + "sandbox", + "template", + "create", + name, + "--image", + "registry.example.com/agent:latest", + "--output=json", + ]) + .args(extra_args) + .env("XDG_CONFIG_HOME", xdg_dir.path()) + .env("HOME", xdg_dir.path()) + .env("OPENSHELL_PROVISION_TIMEOUT", "5") + .output() + .await + .unwrap() +} + +async fn run_rejected_oidc_refresh_server() -> String { + let listener = TcpListener::bind("127.0.0.1:0").await.unwrap(); + let issuer = format!("http://{}", listener.local_addr().unwrap()); + let task_issuer = issuer.clone(); + + tokio::spawn(async move { + loop { + let Ok((mut stream, _)) = listener.accept().await else { + return; + }; + let issuer = task_issuer.clone(); + tokio::spawn(async move { + let mut request = vec![0; 8192]; + let Ok(read) = stream.read(&mut request).await else { + return; + }; + let request = String::from_utf8_lossy(&request[..read]); + let (status, body) = + if request.starts_with("GET /.well-known/openid-configuration ") { + ( + "200 OK", + serde_json::json!({ + "issuer": issuer, + "authorization_endpoint": format!("{issuer}/authorize"), + "token_endpoint": format!("{issuer}/token"), + }) + .to_string(), + ) + } else if request.starts_with("POST /token ") { + ( + "400 Bad Request", + serde_json::json!({ + "error": "invalid_grant", + "error_description": "Token is not active", + }) + .to_string(), + ) + } else { + ("404 Not Found", "{}".to_string()) + }; + let response = format!( + "HTTP/1.1 {status}\r\nContent-Type: application/json\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{body}", + body.len(), + ); + let _ = stream.write_all(response.as_bytes()).await; + }); + } + }); + + issuer +} + +fn write_oidc_test_credentials( + server: &TestServer, + xdg_dir: &TempDir, + issuer: &str, + expires_at: u64, +) { + let gateway_dir = xdg_dir.path().join("openshell/gateways/openshell"); + fs::write( + gateway_dir.join("metadata.json"), + serde_json::to_vec_pretty(&openshell_bootstrap::GatewayMetadata { + name: "openshell".to_string(), + gateway_endpoint: server.endpoint.clone(), + auth_mode: Some("oidc".to_string()), + oidc_issuer: Some(issuer.to_string()), + oidc_client_id: Some("openshell-cli".to_string()), + ..Default::default() + }) + .unwrap(), + ) + .unwrap(); + fs::write( + gateway_dir.join("oidc_token.json"), + serde_json::to_vec_pretty(&openshell_bootstrap::oidc_token::OidcTokenBundle { + access_token: "cached-access-token".to_string(), + refresh_token: Some("inactive-refresh-token".to_string()), + expires_at: Some(expires_at), + issuer: issuer.to_string(), + client_id: "openshell-cli".to_string(), + }) + .unwrap(), + ) + .unwrap(); +} + +/// Models the gateway's JWT-expiration leeway by using a test gateway that +/// still accepts the stale bearer. Before the CLI failed closed, the rejected +/// refresh was only logged and `CreateSandbox` reached this gateway anyway. +#[tokio::test] +async fn sandbox_create_fails_before_mutation_when_expired_oidc_refresh_is_rejected() { + let server = run_server().await; + let issuer = run_rejected_oidc_refresh_server().await; + let xdg_dir = tempfile::tempdir().unwrap(); + prepare_cli_xdg(&server, &xdg_dir); + + write_oidc_test_credentials(&server, &xdg_dir, &issuer, 0); + + let result = run_cli_sandbox_create_with_xdg( + &server, + &xdg_dir, + "must-not-be-created", + &["--output=json"], + ) + .await; + + assert!( + !result.status.success(), + "refresh failure must abort create" + ); + let stderr = String::from_utf8_lossy(&result.stderr); + assert!( + stderr.contains("invalid_grant"), + "unexpected error: {stderr}" + ); + assert!( + stderr.contains("openshell gateway login openshell"), + "error should explain how to re-authenticate: {stderr}" + ); + assert!( + create_requests(&server).await.is_empty(), + "CreateSandbox must not be called after OIDC refresh fails" + ); +} + +#[tokio::test] +async fn sandbox_create_continues_with_unexpired_cached_token_when_refresh_fails() { + let server = run_server().await; + let issuer = run_rejected_oidc_refresh_server().await; + let xdg_dir = tempfile::tempdir().unwrap(); + prepare_cli_xdg(&server, &xdg_dir); + let expires_at = std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .unwrap() + .as_secs() + + 20; + write_oidc_test_credentials(&server, &xdg_dir, &issuer, expires_at); + + let result = + run_cli_sandbox_create_with_xdg(&server, &xdg_dir, "uses-cached-token", &["--output=json"]) + .await; + + assert!( + result.status.success(), + "sandbox create should continue with an unexpired cached token:\n{}", + String::from_utf8_lossy(&result.stderr) + ); + let stderr = String::from_utf8_lossy(&result.stderr); + assert!( + stderr.contains("continuing with the cached token"), + "refresh fallback warning should be visible: {stderr}" + ); + assert_eq!( + create_requests(&server).await.len(), + 1, + "CreateSandbox should be reached with the cached token" + ); +} + #[tokio::test] async fn sandbox_create_json_stdout_is_parseable() { let server = run_server().await; @@ -2241,3 +3022,63 @@ async fn sandbox_create_yaml_stdout_is_parseable() { serde_yml::from_str::(&stdout) .unwrap_or_else(|err| panic!("stdout should contain only YAML: {err}\n{stdout}")); } + +#[tokio::test] +async fn sandbox_template_create_warns_for_credential_env_vars() { + let server = run_server().await; + + let result = run_cli_sandbox_template_create( + &server, + "credential-env", + &["--env", "OPENAI_API_KEY=plain-secret"], + ) + .await; + + assert!( + result.status.success(), + "sandbox template create failed:\n{}", + String::from_utf8_lossy(&result.stderr) + ); + let stderr = String::from_utf8_lossy(&result.stderr); + assert!( + stderr.contains("OPENAI_API_KEY looks like a credential"), + "template create should warn for credential-looking --env values: {stderr}" + ); + assert!( + stderr.contains("To hide it from the agent, use a provider instead"), + "warning should point users toward providers: {stderr}" + ); + + let requests = template_create_requests(&server).await; + assert_eq!(requests.len(), 1); +} + +#[tokio::test] +async fn sandbox_template_create_suppresses_credential_env_warnings() { + let server = run_server().await; + + let result = run_cli_sandbox_template_create( + &server, + "credential-env-suppressed", + &[ + "--env", + "OPENAI_API_KEY=plain-secret", + "--no-credential-warnings", + ], + ) + .await; + + assert!( + result.status.success(), + "sandbox template create failed:\n{}", + String::from_utf8_lossy(&result.stderr) + ); + let stderr = String::from_utf8_lossy(&result.stderr); + assert!( + !stderr.contains("OPENAI_API_KEY looks like a credential"), + "template create should suppress credential-looking --env warnings: {stderr}" + ); + + let requests = template_create_requests(&server).await; + assert_eq!(requests.len(), 1); +} diff --git a/crates/openshell-cli/tests/sandbox_name_fallback_integration.rs b/crates/openshell-cli/tests/sandbox_name_fallback_integration.rs index ee72728aa..fdb0ebbd0 100644 --- a/crates/openshell-cli/tests/sandbox_name_fallback_integration.rs +++ b/crates/openshell-cli/tests/sandbox_name_fallback_integration.rs @@ -3,9 +3,7 @@ mod helpers; -use helpers::{ - EnvVarGuard, build_ca, build_client_cert, build_server_cert, install_rustls_provider, -}; +use helpers::{EnvVarGuard, build_ca, build_client_cert, build_server_cert}; use openshell_bootstrap::{load_last_sandbox, save_last_sandbox}; use openshell_cli::run; use openshell_cli::tls::TlsOptions; @@ -49,6 +47,13 @@ struct TestOpenShell { #[tonic::async_trait] impl OpenShell for TestOpenShell { + async fn begin_rootfs_tar_staging( + &self, + _request: tonic::Request, + ) -> Result, Status> { + Err(Status::unimplemented("not used by this test server")) + } + async fn report_main_process_exit( &self, _request: tonic::Request, @@ -138,6 +143,8 @@ impl OpenShell for TestOpenShell { Ok(Response::new(ListSandboxesResponse::default())) } + unimplemented_sandbox_template_rpcs!(); + async fn list_sandbox_providers( &self, _request: tonic::Request, @@ -673,8 +680,6 @@ struct TestServer { } async fn run_server() -> TestServer { - install_rustls_provider(); - let (ca, ca_key) = build_ca(); let (server_cert, server_key) = build_server_cert(&ca, &ca_key); let (client_cert, client_key) = build_client_cert(&ca, &ca_key); diff --git a/crates/openshell-conformance-cli/Cargo.toml b/crates/openshell-conformance-cli/Cargo.toml new file mode 100644 index 000000000..ce9f89c17 --- /dev/null +++ b/crates/openshell-conformance-cli/Cargo.toml @@ -0,0 +1,25 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +[package] +name = "openshell-conformance-cli" +description = "Standalone OpenShell CLI conformance test runner" +version.workspace = true +edition.workspace = true +rust-version.workspace = true +license.workspace = true +repository.workspace = true + +[[bin]] +name = "openshell-conformance" +path = "src/main.rs" + +[dependencies] +clap.workspace = true +openshell-conformance = { path = "../openshell-conformance" } +serde.workspace = true +serde_json.workspace = true +tokio.workspace = true + +[lints] +workspace = true diff --git a/crates/openshell-conformance-cli/src/main.rs b/crates/openshell-conformance-cli/src/main.rs new file mode 100644 index 000000000..c7f35bc13 --- /dev/null +++ b/crates/openshell-conformance-cli/src/main.rs @@ -0,0 +1,412 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! Standalone runner for `OpenShell` CLI conformance scenarios. + +use std::fmt::Write; +use std::future::Future; +use std::io::Read; +use std::path::PathBuf; +use std::pin::Pin; +use std::process::ExitCode; +use std::sync::Arc; + +use clap::{Parser, Subcommand, ValueEnum}; +use openshell_conformance::{ + ConformancePlan, HostAction, HostActionExecutor, OpenShellRunner, PlanRun, Scenario, + default_scenarios, scenario, scenarios, +}; +use serde::Serialize; + +#[derive(Debug, Parser)] +#[command( + name = "openshell-conformance", + about = "Run OpenShell CLI conformance scenarios", + version +)] +struct Cli { + #[command(subcommand)] + command: Command, +} + +#[derive(Debug, Subcommand)] +enum Command { + /// List registered scenarios. + List { + #[arg(long, value_enum, default_value_t = OutputFormat::Text)] + output: OutputFormat, + }, + /// Run action-free scenarios, named scenarios, or an explicit plan. + Run { + /// Action-free scenario names. Omit to run every action-free scenario. + scenarios: Vec, + /// Explicit path to the `OpenShell` CLI. Defaults to `openshell` on PATH. + #[arg(long)] + openshell_bin: Option, + /// Versioned TOML conformance plan. Use '-' to read the plan from stdin. + #[arg(long, conflicts_with = "scenarios")] + plan: Option, + #[arg(long, value_enum, default_value_t = OutputFormat::Text)] + output: OutputFormat, + }, +} + +#[derive(Clone, Copy, Debug, Eq, PartialEq, ValueEnum)] +enum OutputFormat { + Text, + Json, +} + +#[derive(Serialize)] +struct ScenarioDescription<'a> { + name: &'a str, + description: &'a str, +} + +#[derive(Serialize)] +struct ScenarioResult<'a> { + name: &'a str, + passed: bool, + diagnostic: Option, +} + +#[derive(Serialize)] +struct RunReport<'a> { + scenarios: Vec>, + passed: bool, +} + +#[tokio::main] +async fn main() -> ExitCode { + match execute(Cli::parse()).await { + Ok(()) => ExitCode::SUCCESS, + Err(error) => { + eprintln!("openshell-conformance: {error}"); + ExitCode::FAILURE + } + } +} + +async fn execute(cli: Cli) -> Result<(), String> { + match cli.command { + Command::List { output } => list(output), + Command::Run { + scenarios: requested, + openshell_bin, + plan, + output, + } => run(&requested, openshell_bin, plan, output).await, + } +} + +fn list(output: OutputFormat) -> Result<(), String> { + match output { + OutputFormat::Text => { + for candidate in scenarios() { + println!("{:<16} {}", candidate.name, candidate.description); + } + } + OutputFormat::Json => { + let result = scenarios() + .iter() + .map(|candidate| ScenarioDescription { + name: candidate.name, + description: candidate.description, + }) + .collect::>(); + println!( + "{}", + serde_json::to_string_pretty(&result).map_err(|error| error.to_string())? + ); + } + } + Ok(()) +} + +async fn run( + requested: &[String], + binary: Option, + plan_path: Option, + output: OutputFormat, +) -> Result<(), String> { + if let Some(plan_path) = plan_path { + let plan = read_plan(&plan_path)?; + return run_plan(&plan, binary, output).await; + } + + let selected = select_scenarios(requested)?; + let mut results = Vec::with_capacity(selected.len()); + for candidate in selected { + let plan_run = default_plan_run(candidate.name); + results.push(run_scenario(candidate, &plan_run, binary.as_ref(), None).await); + } + + render_results(results, output) +} + +async fn run_plan( + plan: &ConformancePlan, + binary: Option, + output: OutputFormat, +) -> Result<(), String> { + let executor: Arc = Arc::new(ProcessHostAction); + let mut results = Vec::with_capacity(plan.runs.len()); + for plan_run in &plan.runs { + let candidate = scenario(&plan_run.scenario) + .expect("validated conformance plan references a registered scenario"); + let result = + run_scenario(candidate, plan_run, binary.as_ref(), Some(executor.clone())).await; + let result = match (result.passed, &plan.diagnostics) { + (false, Some(diagnostics)) => append_diagnostics(result, &executor, diagnostics).await, + _ => result, + }; + let passed = result.passed; + results.push(result); + if !passed { + break; + } + } + + render_results(results, output) +} + +fn default_plan_run(scenario: &str) -> PlanRun { + PlanRun { + scenario: scenario.to_string(), + workload_expectation: None, + actions: Vec::new(), + } +} + +async fn run_scenario( + candidate: &'static Scenario, + plan_run: &PlanRun, + binary: Option<&PathBuf>, + host_action_executor: Option>, +) -> ScenarioResult<'static> { + let runner = binary.map_or_else( + || OpenShellRunner::new(candidate.name), + |path| OpenShellRunner::with_binary(path.clone(), candidate.name), + ); + let mut runner = match runner { + Ok(runner) => runner, + Err(error) => { + return ScenarioResult { + name: candidate.name, + passed: false, + diagnostic: Some(error.to_string()), + }; + } + }; + if let Some(host_action_executor) = host_action_executor { + runner = runner.with_host_action_executor(host_action_executor); + } + eprintln!("CLI conformance run ID: {}", runner.id()); + let scenario_result = match runner.check_gateway_status().await { + Ok(()) => candidate.run(&mut runner, plan_run).await, + Err(error) => Err(error), + }; + let outcome = runner.finish(scenario_result).await; + ScenarioResult { + name: candidate.name, + passed: outcome.is_ok(), + diagnostic: outcome.err(), + } +} + +async fn append_diagnostics( + mut result: ScenarioResult<'static>, + executor: &Arc, + diagnostics: &openshell_conformance::PlanDiagnostics, +) -> ScenarioResult<'static> { + let action = diagnostics.as_action(); + if let Err(error) = executor.execute(&action).await { + let diagnostic = result.diagnostic.get_or_insert_default(); + let _ = write!(diagnostic, "\n\nsecondary diagnostics failure:\n{error}"); + } + result +} + +fn render_results( + results: Vec>, + output: OutputFormat, +) -> Result<(), String> { + let passed = results.iter().all(|result| result.passed); + match output { + OutputFormat::Text => { + for result in &results { + if result.passed { + println!("PASS {}", result.name); + } else { + println!( + "FAIL {}\n{}", + result.name, + result.diagnostic.as_deref().unwrap_or("unknown failure") + ); + } + } + } + OutputFormat::Json => println!( + "{}", + serde_json::to_string_pretty(&RunReport { + scenarios: results, + passed + }) + .map_err(|error| error.to_string())? + ), + } + if passed { + Ok(()) + } else { + Err("one or more scenarios failed".to_string()) + } +} + +fn read_plan(path: &PathBuf) -> Result { + let contents = if path.as_os_str() == "-" { + let mut input = String::new(); + std::io::stdin() + .read_to_string(&mut input) + .map_err(|error| format!("read conformance plan from stdin: {error}"))?; + input + } else { + std::fs::read_to_string(path) + .map_err(|error| format!("read conformance plan {}: {error}", path.display()))? + }; + ConformancePlan::parse(&contents).map_err(|error| format!("invalid conformance plan: {error}")) +} + +fn select_scenarios(requested: &[String]) -> Result, String> { + if requested.is_empty() { + return Ok(default_scenarios().collect()); + } + requested + .iter() + .map(|name| { + let candidate = scenario(name).ok_or_else(|| { + format!("unknown scenario '{name}'; run `openshell-conformance list`") + })?; + if candidate.requires_plan() { + return Err(format!( + "scenario '{name}' requires an explicit --plan; run `openshell-conformance list`" + )); + } + Ok(candidate) + }) + .collect() +} + +struct ProcessHostAction; + +impl HostActionExecutor for ProcessHostAction { + fn execute( + &self, + action: &HostAction, + ) -> Pin> + Send + '_>> { + let name = action.name.clone(); + let command = action.command.clone(); + let timeout = action.timeout(); + let timeout_secs = action.timeout_secs; + Box::pin(async move { + let mut process = tokio::process::Command::new(&command); + process.kill_on_drop(true); + let output = tokio::time::timeout(timeout, process.output()) + .await + .map_err(|_| { + format!( + "host action {:?} command '{}' timed out after {}s", + name, + command.display(), + timeout_secs, + ) + })? + .map_err(|error| { + format!( + "start host action {:?} command '{}': {error}", + name, + command.display(), + ) + })?; + if output.status.success() { + return Ok(()); + } + Err(format!( + "host action {:?} command '{}' exited {:?}:\nstdout:\n{}\nstderr:\n{}", + name, + command.display(), + output.status.code(), + String::from_utf8_lossy(&output.stdout), + String::from_utf8_lossy(&output.stderr), + )) + }) + } +} + +#[cfg(test)] +mod tests { + use clap::Parser; + + use super::*; + + #[test] + fn selects_all_scenarios_by_default() { + assert_eq!( + select_scenarios(&[]).expect("select all").len(), + default_scenarios().count() + ); + } + + #[test] + fn selects_named_scenario() { + let selected = select_scenarios(&["smoke".to_string()]).expect("select smoke"); + assert_eq!(selected[0].name, "smoke"); + } + + #[test] + fn unknown_scenario_has_actionable_diagnostic() { + let error = select_scenarios(&["missing".to_string()]).expect_err("unknown scenario"); + assert!(error.contains("openshell-conformance list")); + } + + #[test] + fn action_scenario_requires_an_explicit_plan() { + let error = select_scenarios(&["sandbox-continuity".to_string()]) + .expect_err("action scenario requires a plan"); + + assert!(error.contains("requires an explicit --plan")); + } + + #[test] + fn parses_binary_override_and_json_output() { + let cli = Cli::try_parse_from([ + "openshell-conformance", + "run", + "smoke", + "--openshell-bin", + "/opt/openshell", + "--output", + "json", + ]) + .expect("parse CLI"); + let Command::Run { + openshell_bin, + output, + .. + } = cli.command + else { + panic!("expected run") + }; + assert_eq!(openshell_bin, Some(PathBuf::from("/opt/openshell"))); + assert_eq!(output, OutputFormat::Json); + } + + #[test] + fn parses_plan_from_stdin() { + let cli = Cli::try_parse_from(["openshell-conformance", "run", "--plan", "-"]) + .expect("parse plan from stdin"); + let Command::Run { plan, .. } = cli.command else { + panic!("expected run") + }; + assert_eq!(plan, Some(PathBuf::from("-"))); + } +} diff --git a/crates/openshell-conformance/Cargo.toml b/crates/openshell-conformance/Cargo.toml new file mode 100644 index 000000000..2c355a005 --- /dev/null +++ b/crates/openshell-conformance/Cargo.toml @@ -0,0 +1,24 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +[package] +name = "openshell-conformance" +description = "Reusable OpenShell CLI conformance scenarios and runner" +version.workspace = true +edition.workspace = true +rust-version.workspace = true +license.workspace = true +repository.workspace = true + +[dependencies] +rand.workspace = true +serde.workspace = true +serde_json.workspace = true +tokio.workspace = true +toml.workspace = true + +[dev-dependencies] +tempfile = "3" + +[lints] +workspace = true diff --git a/crates/openshell-conformance/src/executor.rs b/crates/openshell-conformance/src/executor.rs new file mode 100644 index 000000000..b45c102ee --- /dev/null +++ b/crates/openshell-conformance/src/executor.rs @@ -0,0 +1,52 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! Process execution boundary for the CLI conformance runner. + +use std::future::Future; +use std::path::PathBuf; +use std::pin::Pin; +use std::process::{Output, Stdio}; +use std::time::Duration; + +use tokio::time::timeout; + +pub type CliExecution<'a> = + Pin> + Send + 'a>>; + +pub trait CliExecutor: Send + Sync { + fn execute(&self, args: Vec, command_timeout: Duration) -> CliExecution<'_>; +} + +pub enum CliExecutionError { + Spawn(std::io::Error), + Timeout, +} + +pub struct ProcessCli { + binary: PathBuf, +} + +impl ProcessCli { + pub fn new(binary: PathBuf) -> Self { + Self { binary } + } +} + +impl CliExecutor for ProcessCli { + fn execute(&self, args: Vec, command_timeout: Duration) -> CliExecution<'_> { + Box::pin(async move { + let mut process = tokio::process::Command::new(&self.binary); + process + .args(&args) + .stdout(Stdio::piped()) + .stderr(Stdio::piped()) + .kill_on_drop(true); + + timeout(command_timeout, process.output()) + .await + .map_err(|_| CliExecutionError::Timeout)? + .map_err(CliExecutionError::Spawn) + }) + } +} diff --git a/crates/openshell-conformance/src/lib.rs b/crates/openshell-conformance/src/lib.rs new file mode 100644 index 000000000..7b28fa688 --- /dev/null +++ b/crates/openshell-conformance/src/lib.rs @@ -0,0 +1,1070 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! Reusable support for portable `OpenShell` CLI conformance scenarios. + +pub mod executor; +pub mod plan; +mod scenarios; + +use std::collections::BTreeSet; +use std::error::Error; +use std::fmt; +use std::future::Future; +use std::path::PathBuf; +use std::pin::Pin; +use std::process::ExitStatus; +use std::sync::Arc; +use std::time::{Duration, Instant}; + +use rand::Rng; +use serde::Deserialize; +use serde::de::DeserializeOwned; +use tokio::time::sleep; + +use self::executor::{CliExecutionError, CliExecutor, ProcessCli}; + +pub use plan::{ConformancePlan, HostAction, PlanDiagnostics, PlanRun, WorkloadExpectation}; +pub use scenarios::{SANDBOX_CONTINUITY_SCENARIO, SMOKE_SCENARIO}; + +/// An installed conformance scenario. +#[derive(Debug)] +pub struct Scenario { + pub name: &'static str, + pub description: &'static str, + requires_plan: bool, + run: for<'a> fn(&'a mut OpenShellRunner, &'a PlanRun) -> ScenarioFuture<'a>, + validate_plan_run: Option, +} + +pub type ScenarioFuture<'a> = Pin> + Send + 'a>>; +type PlanRunValidator = fn(&PlanRun) -> Result<(), String>; + +impl Scenario { + pub async fn run( + &self, + runner: &mut OpenShellRunner, + plan_run: &PlanRun, + ) -> Result<(), String> { + self.validate_plan_run(plan_run)?; + (self.run)(runner, plan_run).await + } + + pub fn validate_plan_run(&self, plan_run: &PlanRun) -> Result<(), String> { + self.validate_plan_run.map_or_else( + || default_validate_plan_run(plan_run), + |validate| validate(plan_run), + ) + } + + /// Whether this scenario may run only through an explicit target plan. + pub fn requires_plan(&self) -> bool { + self.requires_plan + } +} + +fn default_validate_plan_run(plan_run: &PlanRun) -> Result<(), String> { + if plan_run.workload_expectation.is_some() || !plan_run.actions.is_empty() { + return Err(format!( + "scenario {:?} does not accept workload_expectation or actions", + plan_run.scenario + )); + } + Ok(()) +} + +const SCENARIOS: &[Scenario] = &[SMOKE_SCENARIO, SANDBOX_CONTINUITY_SCENARIO]; + +/// Returns every scenario compiled into this distribution. +pub fn scenarios() -> &'static [Scenario] { + SCENARIOS +} + +/// Finds a scenario by its stable command-line name. +pub fn scenario(name: &str) -> Option<&'static Scenario> { + scenarios().iter().find(|candidate| candidate.name == name) +} + +/// Returns scenarios that need no host-level disruption capability. +pub fn default_scenarios() -> impl Iterator { + scenarios() + .iter() + .filter(|scenario| !scenario.requires_plan) +} + +const CLEANUP_TIMEOUT: Duration = Duration::from_mins(2); +pub const STATUS_TIMEOUT: Duration = Duration::from_secs(30); +const GATEWAY_STATUS_ATTEMPT_TIMEOUT: Duration = Duration::from_secs(10); +const GATEWAY_STATUS_INTERVAL: Duration = Duration::from_secs(2); + +/// Generate the suite-owned identifier used in resource names and diagnostics. +fn generate_run_id() -> String { + const ALPHABET: &[u8] = b"0123456789abcdefghijklmnopqrstuvwxyz"; + let mut rng = rand::rng(); + (0..10) + .map(|_| ALPHABET[rng.random_range(0..ALPHABET.len())] as char) + .collect() +} + +/// The completed outcome of one `OpenShell` CLI process. +#[derive(Debug)] +pub struct CommandResult { + run_id: String, + scenario: String, + step: String, + expectation: String, + command: String, + status: ExitStatus, + elapsed: Duration, + stdout: String, + stderr: String, +} + +impl CommandResult { + pub fn success(&self) -> bool { + self.status.success() + } + + pub fn exit_code(&self) -> Option { + self.status.code() + } + + pub fn stdout(&self) -> &str { + &self.stdout + } + + pub fn stderr(&self) -> &str { + &self.stderr + } + + pub fn elapsed(&self) -> Duration { + self.elapsed + } + + pub fn json(&self) -> Result { + serde_json::from_str(&self.stdout).map_err(|source| RunnerError::InvalidJson { + context: self.context(), + source, + stdout: self.stdout.clone(), + stderr: self.stderr.clone(), + }) + } + + pub fn require_success(&self) -> Result<(), String> { + if self.success() { + return Ok(()); + } + Err(self.failure_diagnostic(&self.expectation)) + } + + pub fn failure_diagnostic(&self, expectation: &str) -> String { + format!( + "{}\nexpected: {expectation}\nactual: exit {} after {:.1?}\ncommand: {}\nstdout:\n{}\nstderr:\n{}", + self.context(), + exit_description(self.status), + self.elapsed, + self.command, + self.stdout, + self.stderr, + ) + } + + fn context(&self) -> String { + format!("[run {}][{}/{}]", self.run_id, self.scenario, self.step) + } +} + +/// Failures in reusable runner mechanics rather than scenario assertions. +#[derive(Debug)] +pub enum RunnerError { + BinaryUnavailable(String), + Spawn { + context: String, + command: String, + source: std::io::Error, + }, + Timeout { + context: String, + command: String, + timeout: Duration, + }, + InvalidJson { + context: String, + source: serde_json::Error, + stdout: String, + stderr: String, + }, + PollTimeout { + context: String, + timeout: Duration, + last_observation: String, + }, + ObservationFailed { + context: String, + diagnostic: String, + }, +} + +impl fmt::Display for RunnerError { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + match self { + Self::BinaryUnavailable(message) => write!(f, "{message}"), + Self::Spawn { + context, + command, + source, + } => write!( + f, + "{context} failed to spawn command: {source}\ncommand: {command}" + ), + Self::Timeout { + context, + command, + timeout, + } => write!( + f, + "{context} timed out after {timeout:.1?}\ncommand: {command}" + ), + Self::InvalidJson { + context, + source, + stdout, + stderr, + } => write!( + f, + "{context} returned invalid JSON: {source}\nstdout:\n{stdout}\nstderr:\n{stderr}" + ), + Self::PollTimeout { + context, + timeout, + last_observation, + } => write!( + f, + "{context} did not satisfy the observation within {timeout:.1?}\nlast observation:\n{last_observation}" + ), + Self::ObservationFailed { + context, + diagnostic, + } => write!(f, "{context} observation failed early:\n{diagnostic}"), + } + } +} + +impl Error for RunnerError { + fn source(&self) -> Option<&(dyn Error + 'static)> { + match self { + Self::Spawn { source, .. } => Some(source), + Self::InvalidJson { source, .. } => Some(source), + _ => None, + } + } +} + +/// Result of one read-only polling observation. +pub enum Poll { + Ready(T), + Pending(String), + Failed(String), +} + +#[derive(Debug, Deserialize)] +struct StatusOutput { + gateway: Option, + server: Option, + status: String, + version: Option, + authentication: Option, +} + +#[derive(Debug, Deserialize)] +struct AuthenticationOutput { + status: String, +} + +/// Runs `OpenShell` commands for one conformance scenario and owns its cleanup. +pub struct OpenShellRunner { + cli: Arc, + host_action_executor: Option>, + run_id: String, + scenario: String, + known_sandboxes: BTreeSet, + finished: bool, +} + +/// Executes one target-supplied host-side action from an explicit plan. +/// +/// This intentionally differs from [`CliExecutor`]: that executor models +/// `OpenShell` CLI invocations through the runner's configured binary and emits +/// structured command results, while an action is a plan-owned executable with +/// no caller-provided arguments. +pub trait HostActionExecutor: Send + Sync { + fn execute( + &self, + action: &HostAction, + ) -> Pin> + Send + '_>>; +} + +/// A runner command with diagnostic context but no timeout yet. +pub struct CommandStep<'a> { + runner: &'a OpenShellRunner, + step: String, + description: Option, +} + +/// A fully configured runner command ready to execute. +pub struct OpenShellCommand<'a> { + runner: &'a OpenShellRunner, + step: String, + description: String, + timeout: Duration, +} + +impl OpenShellRunner { + pub fn new(scenario: &str) -> Result { + Ok(Self::with_executor( + Arc::new(ProcessCli::new(PathBuf::from("openshell"))), + scenario, + )) + } + + /// Uses an explicit `openshell` binary rather than resolving it on `PATH`. + pub fn with_binary(binary: PathBuf, scenario: &str) -> Result { + if !binary.is_file() { + return Err(RunnerError::BinaryUnavailable(format!( + "OpenShell CLI binary not found at {}", + binary.display() + ))); + } + Ok(Self::with_executor( + Arc::new(ProcessCli::new(binary)), + scenario, + )) + } + + /// Creates a runner with an injected executor. This is useful for harness tests. + pub fn with_executor(cli: Arc, scenario: &str) -> Self { + Self { + cli, + host_action_executor: None, + run_id: generate_run_id(), + scenario: scenario.to_string(), + known_sandboxes: BTreeSet::new(), + finished: false, + } + } + + /// Attaches the target-side executor used only by explicit plan actions. + #[must_use] + pub fn with_host_action_executor( + mut self, + host_action_executor: Arc, + ) -> Self { + self.host_action_executor = Some(host_action_executor); + self + } + + /// Execute a target-supplied action declared by the active plan. + pub async fn execute_host_action(&self, action: &HostAction) -> Result<(), String> { + let Some(host_action_executor) = &self.host_action_executor else { + return Err(format!( + "{} requires a host action executor; run this scenario through an explicit plan", + self.context(&format!("action/{}", action.name)) + )); + }; + eprintln!( + "{} applying target host action", + self.context(&format!("action/{}", action.name)) + ); + host_action_executor.execute(action).await.map_err(|error| { + format!( + "{} failed: {error}", + self.context(&format!("action/{}", action.name)) + ) + }) + } + + pub fn id(&self) -> &str { + &self.run_id + } + + pub fn scenario(&self) -> &str { + &self.scenario + } + + pub fn step(&self, step: impl Into) -> CommandStep<'_> { + CommandStep { + runner: self, + step: step.into(), + description: None, + } + } + + /// Confirm that the configured gateway is reachable before a scenario runs. + pub async fn check_gateway_status(&mut self) -> Result<(), String> { + let status = self + .poll_until( + "preflight", + STATUS_TIMEOUT, + GATEWAY_STATUS_INTERVAL, + async |runner| match runner + .step("status/json") + .description("machine-readable gateway status succeeds") + .with_timeout(GATEWAY_STATUS_ATTEMPT_TIMEOUT) + .run(&["status", "--output", "json"]) + .await + { + Ok(result) if !result.success() => Poll::Pending( + result.failure_diagnostic("gateway status command succeeds"), + ), + Ok(result) => match result.json::() { + Ok(status) if status.status == "connected" => Poll::Ready(status), + Ok(status) if status.status == "not_configured" => Poll::Failed( + "no active gateway; register and select a gateway before running conformance" + .to_string(), + ), + Ok(status) => Poll::Pending(format!( + "gateway status is {:?}; expected \"connected\"\nstderr:\n{}", + status.status, + result.stderr() + )), + Err(error) => Poll::Failed(error.to_string()), + }, + Err(error) => Poll::Pending(error.to_string()), + }, + ) + .await + .map_err(|error| error.to_string())?; + + eprintln!( + "gateway preflight connected: gateway={}, server={}, version={}, authentication={}", + status.gateway.as_deref().unwrap_or("unknown"), + status.server.as_deref().unwrap_or("unknown"), + status.version.as_deref().unwrap_or("unknown"), + status + .authentication + .as_ref() + .map_or("unknown", |authentication| authentication.status.as_str()), + ); + Ok(()) + } + + /// Register a sandbox name for cleanup. + /// + /// Call this before running the command that may create the sandbox. The + /// scenario remains responsible for constructing and running that command. + pub fn track_sandbox(&mut self, name: &str) { + self.known_sandboxes.insert(name.to_string()); + } + + /// Stop tracking a sandbox after the scenario confirms it is absent. + pub fn forget_sandbox(&mut self, name: &str) { + self.known_sandboxes.remove(name); + } + + pub async fn poll_until( + &mut self, + step: &str, + poll_timeout: Duration, + interval: Duration, + mut observe: F, + ) -> Result + where + F: AsyncFnMut(&mut Self) -> Poll, + { + let started = Instant::now(); + let context = self.context(step); + + loop { + match observe(self).await { + Poll::Ready(value) => return Ok(value), + Poll::Pending(diagnostic) => { + if started.elapsed() >= poll_timeout { + return Err(RunnerError::PollTimeout { + context, + timeout: poll_timeout, + last_observation: diagnostic, + }); + } + } + Poll::Failed(diagnostic) => { + return Err(RunnerError::ObservationFailed { + context, + diagnostic, + }); + } + } + sleep(interval).await; + } + } + + pub async fn finish(mut self, scenario_result: Result<(), String>) -> Result<(), String> { + let cleanup_result = self.cleanup().await; + self.finished = true; + combine_results(scenario_result, cleanup_result) + } + + async fn run_strings( + &self, + step: &str, + expectation: &str, + args: Vec, + command_timeout: Duration, + ) -> Result { + let context = self.context(step); + let command = sanitized_command(&args); + eprintln!("{context} running: {command}"); + + let started = Instant::now(); + let output = + self.cli + .execute(args, command_timeout) + .await + .map_err(|error| match error { + CliExecutionError::Timeout => RunnerError::Timeout { + context: context.clone(), + command: command.clone(), + timeout: command_timeout, + }, + CliExecutionError::Spawn(source) => RunnerError::Spawn { + context: context.clone(), + command: command.clone(), + source, + }, + })?; + let elapsed = started.elapsed(); + eprintln!( + "{context} completed in {:.1?}: exit {}", + elapsed, + exit_description(output.status) + ); + + Ok(CommandResult { + run_id: self.run_id.clone(), + scenario: self.scenario.clone(), + step: step.to_string(), + expectation: expectation.to_string(), + command, + status: output.status, + elapsed, + stdout: String::from_utf8_lossy(&output.stdout).into_owned(), + stderr: String::from_utf8_lossy(&output.stderr).into_owned(), + }) + } + + async fn cleanup(&self) -> Result<(), String> { + if self.known_sandboxes.is_empty() { + return Ok(()); + } + + let cleanup_started = Instant::now(); + let mut failures = Vec::new(); + for name in self.known_sandboxes.clone() { + let remaining = remaining_cleanup_time(cleanup_started); + if remaining.is_zero() { + failures.push(format!( + "{} cleanup budget expired before deleting sandbox '{name}'", + self.context("cleanup/delete") + )); + break; + } + match self + .step("cleanup/delete") + .description(format!("sandbox '{name}' is deleted or already absent")) + .with_timeout(remaining) + .run(&["sandbox", "delete", &name]) + .await + { + Ok(result) if result.success() || output_reports_not_found(&result) => {} + Ok(result) => { + failures.push(result.failure_diagnostic(&format!( + "sandbox '{name}' is deleted or already absent" + ))); + } + Err(error) => failures.push(error.to_string()), + } + } + + if failures.is_empty() { + Ok(()) + } else { + Err(failures.join("\n\n")) + } + } + + fn context(&self, step: &str) -> String { + format!("[run {}][{}/{}]", self.run_id, self.scenario, step) + } +} + +impl<'a> CommandStep<'a> { + #[must_use] + pub fn description(mut self, description: impl Into) -> Self { + self.description = Some(description.into()); + self + } + + pub fn with_timeout(self, timeout: Duration) -> OpenShellCommand<'a> { + let description = self + .description + .unwrap_or_else(|| format!("step '{}' succeeds", self.step)); + OpenShellCommand { + runner: self.runner, + step: self.step, + description, + timeout, + } + } +} + +impl OpenShellCommand<'_> { + pub async fn run(&self, args: &[&str]) -> Result { + self.runner + .run_strings( + &self.step, + &self.description, + args.iter().map(|arg| (*arg).to_string()).collect(), + self.timeout, + ) + .await + } +} + +impl Drop for OpenShellRunner { + fn drop(&mut self) { + if self.finished { + return; + } + let resources = if self.known_sandboxes.is_empty() { + "none explicitly tracked".to_string() + } else { + self.known_sandboxes + .iter() + .cloned() + .collect::>() + .join(", ") + }; + eprintln!( + "[run {}][{}] WARNING: runner dropped without finish(); known sandboxes: {}", + self.run_id, self.scenario, resources + ); + } +} + +fn combine_results( + scenario_result: Result<(), String>, + cleanup_result: Result<(), String>, +) -> Result<(), String> { + match (scenario_result, cleanup_result) { + (Ok(()), Ok(())) => Ok(()), + (Err(functional), Ok(())) => Err(functional), + (Ok(()), Err(cleanup)) => Err(format!("cleanup failed:\n{cleanup}")), + (Err(functional), Err(cleanup)) => Err(format!( + "{functional}\n\nsecondary cleanup failure:\n{cleanup}" + )), + } +} + +fn remaining_cleanup_time(started: Instant) -> Duration { + CLEANUP_TIMEOUT.saturating_sub(started.elapsed()) +} + +fn output_reports_not_found(result: &CommandResult) -> bool { + result.stderr().to_ascii_lowercase().contains("not found") + || result.stdout().to_ascii_lowercase().contains("not found") +} + +fn exit_description(status: ExitStatus) -> String { + status.code().map_or_else( + || "terminated by signal".to_string(), + |code| code.to_string(), + ) +} + +fn sanitized_command(args: &[String]) -> String { + let mut redact_next = false; + let rendered = args.iter().map(|arg| { + let rendered = if redact_next { + redact_next = false; + "".to_string() + } else if sensitive_flag(arg) { + redact_next = !arg.contains('='); + arg.split_once('=') + .map_or_else(|| arg.clone(), |(flag, _)| format!("{flag}=")) + } else { + arg.clone() + }; + shell_escape(&rendered) + }); + std::iter::once("openshell".to_string()) + .chain(rendered) + .collect::>() + .join(" ") +} + +fn sensitive_flag(arg: &str) -> bool { + let flag = arg.split_once('=').map_or(arg, |(flag, _)| flag); + matches!( + flag, + "--credential" + | "--credentials" + | "--env" + | "--header" + | "--material" + | "--password" + | "--secret" + | "--token" + ) || flag.contains("client-secret") + || flag.contains("private-key") + || flag.ends_with("-token") +} + +fn shell_escape(arg: &str) -> String { + if arg + .chars() + .all(|character| character.is_ascii_alphanumeric() || "-_./:@=,".contains(character)) + { + return arg.to_string(); + } + format!("'{}'", arg.replace('\'', "'\\''")) +} + +#[cfg(all(test, unix))] +mod tests { + use std::collections::VecDeque; + use std::os::unix::process::ExitStatusExt; + use std::process::Output; + use std::sync::Mutex; + + use serde::Deserialize; + + use super::executor::CliExecution; + use super::*; + + struct MockCli { + state: Mutex, + } + + struct MockCliState { + responses: VecDeque, + invocations: Vec>, + } + + enum MockResponse { + Output { + exit_code: i32, + stdout: String, + stderr: String, + }, + Timeout, + } + + impl MockResponse { + fn output(exit_code: i32, stdout: &str, stderr: &str) -> Self { + Self::Output { + exit_code, + stdout: stdout.to_string(), + stderr: stderr.to_string(), + } + } + + fn success(stdout: &str) -> Self { + Self::output(0, stdout, "") + } + } + + impl MockCli { + fn new(responses: Vec) -> Self { + Self { + state: Mutex::new(MockCliState { + responses: responses.into(), + invocations: Vec::new(), + }), + } + } + + fn invocations(&self) -> Vec> { + self.state + .lock() + .expect("lock mock CLI state") + .invocations + .clone() + } + } + + impl CliExecutor for MockCli { + fn execute(&self, args: Vec, _command_timeout: Duration) -> CliExecution<'_> { + let response = { + let mut state = self.state.lock().expect("lock mock CLI state"); + state.invocations.push(args); + state + .responses + .pop_front() + .expect("mock CLI received an unexpected invocation") + }; + + Box::pin(async move { + match response { + MockResponse::Output { + exit_code, + stdout, + stderr, + } => Ok(Output { + status: ExitStatus::from_raw(exit_code << 8), + stdout: stdout.into_bytes(), + stderr: stderr.into_bytes(), + }), + MockResponse::Timeout => Err(CliExecutionError::Timeout), + } + }) + } + } + + fn test_runner(responses: Vec) -> (OpenShellRunner, Arc) { + let cli = Arc::new(MockCli::new(responses)); + let runner = OpenShellRunner::with_executor(cli.clone(), "smoke"); + (runner, cli) + } + + #[test] + fn with_binary_rejects_a_missing_file() { + let directory = tempfile::tempdir().expect("create missing CLI directory"); + let Err(error) = OpenShellRunner::with_binary(directory.path().join("missing"), "smoke") + else { + panic!("missing CLI should be rejected"); + }; + + assert!(matches!(error, RunnerError::BinaryUnavailable(_))); + } + + #[test] + fn exposes_generated_identity() { + let (mut runner, _cli) = test_runner(Vec::new()); + + assert_eq!(runner.id().len(), 10); + assert_eq!(runner.scenario(), "smoke"); + runner.finished = true; + } + + #[tokio::test] + async fn captures_streams_and_nonzero_exit_as_result() { + let (mut runner, _cli) = test_runner(vec![MockResponse::output(7, "stdout", "stderr")]); + + let result = runner + .step("capture") + .description("sandbox list succeeds") + .with_timeout(Duration::from_secs(1)) + .run(&["sandbox", "list"]) + .await + .expect("completed nonzero exit is a result"); + + assert_eq!(result.exit_code(), Some(7)); + assert_eq!(result.stdout(), "stdout"); + assert_eq!(result.stderr(), "stderr"); + assert!( + result + .require_success() + .expect_err("nonzero result should fail its expectation") + .contains("expected: sandbox list succeeds") + ); + runner.finished = true; + } + + #[tokio::test] + async fn reports_timeout_as_typed_error() { + let (mut runner, _cli) = test_runner(vec![MockResponse::Timeout]); + + let error = runner + .step("timeout") + .with_timeout(Duration::from_millis(10)) + .run(&["status"]) + .await + .expect_err("command should time out"); + + assert!(matches!(error, RunnerError::Timeout { .. })); + runner.finished = true; + } + + #[derive(Debug, Deserialize, PartialEq, Eq)] + struct JsonFixture { + status: String, + } + + #[tokio::test] + async fn deserializes_stdout_json_without_stderr() { + let (mut runner, _cli) = test_runner(vec![MockResponse::output( + 0, + "{\"status\":\"connected\"}", + "diagnostic", + )]); + + let result = runner + .step("json") + .with_timeout(Duration::from_secs(1)) + .run(&["status"]) + .await + .expect("run fake CLI"); + + assert_eq!( + result.json::().expect("parse JSON"), + JsonFixture { + status: "connected".to_string() + } + ); + runner.finished = true; + } + + #[tokio::test] + async fn accepts_connected_gateway_status() { + let (mut runner, _cli) = + test_runner(vec![MockResponse::success("{\"status\":\"connected\"}")]); + + runner + .check_gateway_status() + .await + .expect("connected gateway should satisfy preflight"); + + runner.finished = true; + } + + #[tokio::test] + async fn polling_returns_ready_value() { + let (mut runner, _cli) = test_runner(Vec::new()); + let mut attempts = 0; + + let value = runner + .poll_until( + "poll", + Duration::from_secs(1), + Duration::from_millis(1), + async |_runner| { + attempts += 1; + if attempts == 2 { + Poll::Ready("ready") + } else { + Poll::Pending("not ready".to_string()) + } + }, + ) + .await + .expect("poll should become ready"); + + assert_eq!(value, "ready"); + runner.finished = true; + } + + #[tokio::test] + async fn polling_timeout_preserves_last_observation() { + let (mut runner, _cli) = test_runner(Vec::new()); + + let error = runner + .poll_until( + "poll", + Duration::from_millis(1), + Duration::from_millis(1), + async |_runner| Poll::<()>::Pending("still pending".to_string()), + ) + .await + .expect_err("poll should time out"); + + assert!(matches!( + error, + RunnerError::PollTimeout { + last_observation, + .. + } if last_observation == "still pending" + )); + runner.finished = true; + } + + #[tokio::test] + async fn finish_skips_cleanup_commands_without_registered_resources() { + let (runner, cli) = test_runner(Vec::new()); + + let error = runner + .finish(Err("preflight failed".to_string())) + .await + .expect_err("functional failure should be preserved"); + + assert_eq!(error, "preflight failed"); + assert!(cli.invocations().is_empty()); + } + + #[test] + fn tracks_sandbox_for_cleanup() { + let (mut runner, _cli) = test_runner(Vec::new()); + + runner.track_sandbox("ct-0123456789-01"); + + assert!(runner.known_sandboxes.contains("ct-0123456789-01")); + runner.finished = true; + } + + #[test] + fn forgets_exact_name_cleanup() { + let (mut runner, _cli) = test_runner(Vec::new()); + + runner.track_sandbox("ct-0123456789-01"); + runner.forget_sandbox("ct-0123456789-01"); + + assert!(runner.known_sandboxes.is_empty()); + runner.finished = true; + } + + #[tokio::test] + async fn finish_deletes_explicitly_tracked_resources() { + let (mut runner, cli) = test_runner(vec![MockResponse::success("")]); + runner.track_sandbox("ct-0123456789-01"); + + runner + .finish(Ok(())) + .await + .expect("owned resource cleanup should succeed"); + + assert_eq!( + cli.invocations(), + vec![vec![ + "sandbox".to_string(), + "delete".to_string(), + "ct-0123456789-01".to_string(), + ]] + ); + } + + #[test] + fn functional_failure_remains_primary_when_cleanup_also_fails() { + let error = combine_results( + Err("functional failure".to_string()), + Err("cleanup failure".to_string()), + ) + .expect_err("combined result should fail"); + + assert!(error.starts_with("functional failure")); + assert!(error.contains("secondary cleanup failure:\ncleanup failure")); + } + + #[test] + fn generated_run_id_is_portable_and_compact() { + let run_id = generate_run_id(); + assert_eq!(run_id.len(), 10); + assert!( + run_id + .chars() + .all(|character| character.is_ascii_lowercase() || character.is_ascii_digit()) + ); + } + + #[test] + fn sanitizes_sensitive_cli_arguments() { + let command = sanitized_command(&[ + "provider".to_string(), + "create".to_string(), + "--credential".to_string(), + "TOKEN=secret".to_string(), + "--client-secret=hunter2".to_string(), + ]); + + assert!(!command.contains("TOKEN=secret")); + assert!(!command.contains("hunter2")); + assert!(command.contains("")); + } +} diff --git a/crates/openshell-conformance/src/plan.rs b/crates/openshell-conformance/src/plan.rs new file mode 100644 index 000000000..ad955ea5f --- /dev/null +++ b/crates/openshell-conformance/src/plan.rs @@ -0,0 +1,233 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! Versioned, target-supplied execution plans for conformance scenarios. + +use std::path::{Path, PathBuf}; +use std::time::Duration; + +use serde::Deserialize; + +use crate::scenario; + +pub const PLAN_VERSION: u32 = 1; + +#[derive(Debug, Deserialize)] +pub struct ConformancePlan { + pub version: u32, + #[serde(default)] + pub runs: Vec, + pub diagnostics: Option, +} + +#[derive(Debug, Deserialize)] +pub struct PlanRun { + pub scenario: String, + pub workload_expectation: Option, + #[serde(default)] + pub actions: Vec, +} + +#[derive(Clone, Copy, Debug, Deserialize, Eq, PartialEq)] +#[serde(rename_all = "kebab-case")] +pub enum WorkloadExpectation { + Reconciled, +} + +#[derive(Clone, Debug, Deserialize)] +pub struct HostAction { + pub name: String, + pub command: PathBuf, + pub timeout_secs: u64, +} + +impl HostAction { + pub fn timeout(&self) -> Duration { + Duration::from_secs(self.timeout_secs) + } +} + +#[derive(Debug, Deserialize)] +pub struct PlanDiagnostics { + pub command: PathBuf, + pub timeout_secs: u64, +} + +impl PlanDiagnostics { + pub fn as_action(&self) -> HostAction { + HostAction { + name: "diagnostics".to_string(), + command: self.command.clone(), + timeout_secs: self.timeout_secs, + } + } +} + +impl ConformancePlan { + pub fn parse(input: &str) -> Result { + let plan = toml::from_str::(input).map_err(|error| error.to_string())?; + plan.validate()?; + Ok(plan) + } + + fn validate(&self) -> Result<(), String> { + if self.version != PLAN_VERSION { + return Err(format!( + "unsupported conformance plan version {}; expected {PLAN_VERSION}", + self.version + )); + } + if self.runs.is_empty() { + return Err("conformance plan must contain at least one run".to_string()); + } + if let Some(diagnostics) = &self.diagnostics { + validate_command( + "diagnostics", + &diagnostics.command, + diagnostics.timeout_secs, + )?; + } + for run in &self.runs { + let Some(scenario) = scenario(&run.scenario) else { + return Err(format!( + "unknown scenario {:?}; run `openshell-conformance list`", + run.scenario + )); + }; + scenario.validate_plan_run(run)?; + for action in &run.actions { + validate_command( + &format!("action {:?}", action.name), + &action.command, + action.timeout_secs, + )?; + } + } + Ok(()) + } +} + +fn validate_command(label: &str, command: &Path, timeout_secs: u64) -> Result<(), String> { + if !command.is_absolute() { + return Err(format!( + "{label} command must be an absolute path: {}", + command.display() + )); + } + if timeout_secs == 0 { + return Err(format!("{label} timeout_secs must be greater than zero")); + } + Ok(()) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn parses_a_smoke_and_continuity_plan() { + let plan = ConformancePlan::parse( + r#" + version = 1 + + [[runs]] + scenario = "smoke" + + [[runs]] + scenario = "sandbox-continuity" + workload_expectation = "reconciled" + + [[runs.actions]] + name = "gateway-upgrade" + command = "/usr/local/libexec/restart-gateway" + timeout_secs = 120 + "#, + ) + .expect("valid plan"); + + assert_eq!(plan.runs.len(), 2); + assert_eq!(plan.runs[1].actions[0].name, "gateway-upgrade"); + } + + #[test] + fn parses_plan_diagnostics() { + let plan = ConformancePlan::parse( + r#" + version = 1 + + [diagnostics] + command = "/usr/local/libexec/diagnostics" + timeout_secs = 60 + + [[runs]] + scenario = "smoke" + "#, + ) + .expect("valid plan with diagnostics"); + + assert_eq!( + plan.diagnostics + .expect("diagnostics configured") + .as_action() + .name, + "diagnostics" + ); + } + + #[test] + fn rejects_a_relative_action_command() { + let error = ConformancePlan::parse( + r#" + version = 1 + + [[runs]] + scenario = "sandbox-continuity" + workload_expectation = "reconciled" + + [[runs.actions]] + name = "gateway-restart" + command = "restart-gateway" + timeout_secs = 120 + "#, + ) + .expect_err("relative command must fail"); + + assert!(error.contains("absolute path")); + } + + #[test] + fn rejects_an_actionless_continuity_run() { + let error = ConformancePlan::parse( + r#" + version = 1 + + [[runs]] + scenario = "sandbox-continuity" + workload_expectation = "reconciled" + "#, + ) + .expect_err("continuity requires an action"); + + assert!(error.contains("requires at least one action")); + } + + #[test] + fn default_validation_rejects_actions_for_smoke() { + let error = ConformancePlan::parse( + r#" + version = 1 + + [[runs]] + scenario = "smoke" + + [[runs.actions]] + name = "gateway-restart" + command = "/usr/local/libexec/restart-gateway" + timeout_secs = 120 + "#, + ) + .expect_err("smoke does not accept actions"); + + assert!(error.contains("does not accept workload_expectation or actions")); + } +} diff --git a/crates/openshell-conformance/src/scenarios/mod.rs b/crates/openshell-conformance/src/scenarios/mod.rs new file mode 100644 index 000000000..c5211b369 --- /dev/null +++ b/crates/openshell-conformance/src/scenarios/mod.rs @@ -0,0 +1,10 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! Registered, portable conformance scenarios. + +mod sandbox_continuity; +mod smoke; + +pub use sandbox_continuity::SANDBOX_CONTINUITY_SCENARIO; +pub use smoke::SMOKE_SCENARIO; diff --git a/crates/openshell-conformance/src/scenarios/sandbox_continuity.rs b/crates/openshell-conformance/src/scenarios/sandbox_continuity.rs new file mode 100644 index 000000000..81f8d4657 --- /dev/null +++ b/crates/openshell-conformance/src/scenarios/sandbox_continuity.rs @@ -0,0 +1,207 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! Product-visible sandbox continuity across host-side actions. + +use std::time::Duration; + +use serde::Deserialize; + +use crate::{ + HostAction, OpenShellRunner, PlanRun, Poll, Scenario, ScenarioFuture, WorkloadExpectation, +}; + +const CREATE_TIMEOUT: Duration = Duration::from_mins(10); +const COMMAND_TIMEOUT: Duration = Duration::from_mins(2); +const RECOVERY_TIMEOUT: Duration = Duration::from_mins(4); +const RECOVERY_INTERVAL: Duration = Duration::from_secs(2); + +#[derive(Debug, Deserialize)] +struct SandboxState { + name: String, + phase: String, +} + +/// Certify sandbox state and workspace continuity across host-side actions. +pub const SANDBOX_CONTINUITY_SCENARIO: Scenario = Scenario { + name: "sandbox-continuity", + description: "Verify sandbox state and workspace continuity across planned host actions.", + requires_plan: true, + run: run_sandbox_continuity, + validate_plan_run: Some(validate_plan_run), +}; + +fn run_sandbox_continuity<'a>( + runner: &'a mut OpenShellRunner, + plan_run: &'a PlanRun, +) -> ScenarioFuture<'a> { + Box::pin(async move { run_sandbox_continuity_inner(runner, plan_run).await }) +} + +fn validate_plan_run(plan_run: &PlanRun) -> Result<(), String> { + if plan_run.workload_expectation != Some(WorkloadExpectation::Reconciled) { + return Err( + "scenario 'sandbox-continuity' requires workload_expectation = \"reconciled\"" + .to_string(), + ); + } + if plan_run.actions.is_empty() { + return Err("scenario 'sandbox-continuity' requires at least one action".to_string()); + } + Ok(()) +} + +async fn run_sandbox_continuity_inner( + runner: &mut OpenShellRunner, + plan_run: &PlanRun, +) -> Result<(), String> { + // The VM driver permits at most 19 characters in a sandbox name. + let running_name = format!("ct-{}-r", runner.id()); + let stopped_name = format!("ct-{}-s", runner.id()); + let marker = format!("openshell-sandbox-continuity-{}", runner.id()); + let marker_path = "/sandbox/.openshell-sandbox-continuity"; + let running_script = + format!("printf '%s\\n' '{marker}' > {marker_path}; while true; do sleep 1; done"); + + create_retained_sandbox(runner, &running_name, &running_script, "running").await?; + assert_marker(runner, &running_name, marker_path, &marker, "pre-action").await?; + + create_retained_sandbox( + runner, + &stopped_name, + "while true; do sleep 1; done", + "stopped", + ) + .await?; + let stop = runner + .step("stop") + .description(format!("sandbox '{stopped_name}' stops")) + .with_timeout(COMMAND_TIMEOUT) + .run(&["sandbox", "stop", &stopped_name]) + .await + .map_err(|error| error.to_string())?; + stop.require_success()?; + wait_for_phase(runner, &stopped_name, "Stopped", "stopped-before-actions").await?; + + for action in &plan_run.actions { + apply_action_and_assert( + runner, + action, + &running_name, + &stopped_name, + marker_path, + &marker, + ) + .await?; + } + Ok(()) +} + +#[allow(clippy::too_many_arguments)] +async fn apply_action_and_assert( + runner: &mut OpenShellRunner, + action: &HostAction, + running_name: &str, + stopped_name: &str, + marker_path: &str, + marker: &str, +) -> Result<(), String> { + let step = format!("after-{}", action.name); + runner.execute_host_action(action).await?; + runner.check_gateway_status().await?; + wait_for_phase(runner, running_name, "Ready", &format!("running-{step}")).await?; + wait_for_phase(runner, stopped_name, "Stopped", &format!("stopped-{step}")).await?; + assert_marker(runner, running_name, marker_path, marker, &step).await +} + +async fn create_retained_sandbox( + runner: &mut OpenShellRunner, + name: &str, + script: &str, + step: &str, +) -> Result<(), String> { + runner.track_sandbox(name); + let create = runner + .step(format!("create-{step}")) + .description(format!("retained sandbox '{name}' is created")) + .with_timeout(CREATE_TIMEOUT) + .run(&[ + "sandbox", "create", "--name", name, "--from", "base", "--detach", "--no-tty", "--", + "sh", "-lc", script, + ]) + .await + .map_err(|error| error.to_string())?; + create.require_success() +} + +async fn assert_marker( + runner: &OpenShellRunner, + name: &str, + path: &str, + marker: &str, + step: &str, +) -> Result<(), String> { + let result = runner + .step(format!("marker-{step}")) + .description(format!("sandbox '{name}' retains its workspace marker")) + .with_timeout(COMMAND_TIMEOUT) + .run(&[ + "sandbox", "exec", "--name", name, "--no-tty", "--", "cat", path, + ]) + .await + .map_err(|error| error.to_string())?; + result.require_success()?; + if result.stdout().lines().any(|line| line.trim() == marker) { + Ok(()) + } else { + Err(result.failure_diagnostic(&format!( + "sandbox '{name}' stdout contains marker {marker:?}" + ))) + } +} + +async fn wait_for_phase( + runner: &mut OpenShellRunner, + name: &str, + expected_phase: &str, + step: &str, +) -> Result<(), String> { + let name = name.to_string(); + let expected_phase = expected_phase.to_string(); + let step = step.to_string(); + let poll_step = step.clone(); + runner + .poll_until( + &poll_step, + RECOVERY_TIMEOUT, + RECOVERY_INTERVAL, + async move |runner| { + let result = runner + .step(format!("{step}/get")) + .description(format!("sandbox '{name}' reaches phase {expected_phase}")) + .with_timeout(COMMAND_TIMEOUT) + .run(&["sandbox", "get", &name, "--output", "json"]) + .await; + match result { + Ok(result) if !result.success() => Poll::Pending( + result.failure_diagnostic(&format!("sandbox '{name}' can be retrieved")), + ), + Ok(result) => match result.json::() { + Ok(state) if state.name != name => Poll::Failed(format!( + "sandbox get returned {:?}; expected '{name}'", + state.name + )), + Ok(state) if state.phase == expected_phase => Poll::Ready(()), + Ok(state) => Poll::Pending(format!( + "sandbox '{name}' phase is {:?}; expected {expected_phase:?}", + state.phase + )), + Err(error) => Poll::Failed(error.to_string()), + }, + Err(error) => Poll::Pending(error.to_string()), + } + }, + ) + .await + .map_err(|error| error.to_string()) +} diff --git a/crates/openshell-conformance/src/scenarios/smoke.rs b/crates/openshell-conformance/src/scenarios/smoke.rs new file mode 100644 index 000000000..7c5fa0068 --- /dev/null +++ b/crates/openshell-conformance/src/scenarios/smoke.rs @@ -0,0 +1,208 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! Portable phase-1 CLI conformance scenario. + +use std::time::{Duration, Instant}; + +use crate::{OpenShellRunner, PlanRun, STATUS_TIMEOUT, Scenario, ScenarioFuture}; +use serde::Deserialize; +use tokio::time::sleep; + +const CREATE_TIMEOUT: Duration = Duration::from_mins(10); +const LIST_ATTEMPT_TIMEOUT: Duration = Duration::from_secs(10); +const LIST_PAGE_SIZE: u32 = 1_000; +const EXEC_TIMEOUT: Duration = Duration::from_mins(2); +const DELETE_TIMEOUT: Duration = Duration::from_mins(2); +const DELETE_POLL_INTERVAL: Duration = Duration::from_secs(1); + +#[derive(Debug, Deserialize)] +struct SandboxListEntry { + name: String, + phase: String, +} + +/// Certify status -> create -> list Ready -> exec -> delete -> list empty. +pub const SMOKE_SCENARIO: Scenario = Scenario { + name: "smoke", + description: "Create, inspect, execute in, and delete a base sandbox.", + requires_plan: false, + run: run_smoke, + validate_plan_run: None, +}; + +fn run_smoke<'a>(runner: &'a mut OpenShellRunner, _plan_run: &'a PlanRun) -> ScenarioFuture<'a> { + Box::pin(async move { run_smoke_inner(runner).await }) +} + +async fn run_smoke_inner(runner: &mut OpenShellRunner) -> Result<(), String> { + let status = runner + .step("status") + .description("openshell status succeeds") + .with_timeout(STATUS_TIMEOUT) + .run(&["status"]) + .await + .map_err(|error| error.to_string())?; + status.require_success()?; + + let sandbox_name = format!("ct-{}-01", runner.id()); + runner.track_sandbox(&sandbox_name); + let create = runner + .step("create") + .description("sandbox creation succeeds") + .with_timeout(CREATE_TIMEOUT) + .run(&[ + "sandbox", + "create", + "--name", + &sandbox_name, + "--from", + "base", + "--detach", + ]) + .await + .map_err(|error| error.to_string())?; + create.require_success()?; + + let get = runner + .step("get-ready") + .description(format!("sandbox '{sandbox_name}' can be retrieved")) + .with_timeout(LIST_ATTEMPT_TIMEOUT) + .run(&["sandbox", "get", &sandbox_name, "--output", "json"]) + .await + .map_err(|error| error.to_string())?; + get.require_success()?; + + let sandbox = get + .json::() + .map_err(|error| error.to_string())?; + if sandbox.name != sandbox_name { + return Err(format!( + "sandbox get returned {:?}; expected sandbox '{sandbox_name}'", + sandbox.name + )); + } + if sandbox.phase != "Ready" { + return Err(format!( + "sandbox '{sandbox_name}' is in phase {:?}; expected Ready", + sandbox.phase + )); + } + + check_sandbox_listed(runner, &sandbox_name).await?; + + let marker = format!("openshell-conformance-{}", runner.id()); + let exec = runner + .step("exec") + .description("sandbox exec exits successfully") + .with_timeout(EXEC_TIMEOUT) + .run(&[ + "sandbox", + "exec", + "--name", + &sandbox_name, + "--no-tty", + "--", + "echo", + &marker, + ]) + .await + .map_err(|error| error.to_string())?; + exec.require_success()?; + let expected_stdout = format!("{marker}\n"); + if exec.stdout() != expected_stdout { + return Err(exec.failure_diagnostic(&format!("stdout is exactly {expected_stdout:?}"))); + } + + let delete = runner + .step("delete") + .description("sandbox deletion succeeds") + .with_timeout(DELETE_TIMEOUT) + .run(&["sandbox", "delete", &sandbox_name]) + .await + .map_err(|error| error.to_string())?; + delete.require_success()?; + + check_empty_list(runner, &sandbox_name).await?; + runner.forget_sandbox(&sandbox_name); + Ok(()) +} + +async fn check_sandbox_listed(runner: &OpenShellRunner, sandbox_name: &str) -> Result<(), String> { + if find_sandbox(runner, sandbox_name, "list-visible") + .await? + .is_some() + { + return Ok(()); + } + Err(format!( + "sandbox '{sandbox_name}' does not appear in sandbox list" + )) +} + +async fn check_empty_list(runner: &OpenShellRunner, sandbox_name: &str) -> Result<(), String> { + let started = Instant::now(); + + loop { + match find_sandbox(runner, sandbox_name, "list-empty/query").await? { + None => return Ok(()), + Some(sandbox) if started.elapsed() >= DELETE_TIMEOUT => { + return Err(format!( + "sandbox '{sandbox_name}' remains listed in phase {:?} after {DELETE_TIMEOUT:.1?}", + sandbox.phase + )); + } + Some(_) => sleep(DELETE_POLL_INTERVAL).await, + } + } +} + +async fn find_sandbox( + runner: &OpenShellRunner, + sandbox_name: &str, + step: &str, +) -> Result, String> { + let mut offset = 0u32; + + loop { + let limit = LIST_PAGE_SIZE.to_string(); + let page_offset = offset.to_string(); + let result = runner + .step(format!("{step}/{offset}")) + .description(format!("sandbox list page at offset {offset} succeeds")) + .with_timeout(LIST_ATTEMPT_TIMEOUT) + .run(&[ + "sandbox", + "list", + "--limit", + &limit, + "--offset", + &page_offset, + "--output", + "json", + ]) + .await + .map_err(|error| error.to_string())?; + result.require_success()?; + + let sandboxes = result + .json::>() + .map_err(|error| error.to_string())?; + if let Some(sandbox) = sandboxes + .iter() + .find(|sandbox| sandbox.name == sandbox_name) + { + return Ok(Some(SandboxListEntry { + name: sandbox.name.clone(), + phase: sandbox.phase.clone(), + })); + } + if sandboxes.len() < LIST_PAGE_SIZE as usize { + return Ok(None); + } + + offset = offset + .checked_add(LIST_PAGE_SIZE) + .ok_or_else(|| "sandbox list pagination offset overflowed".to_string())?; + } +} diff --git a/crates/openshell-core/Cargo.toml b/crates/openshell-core/Cargo.toml index d8483bd2c..ff2cc638f 100644 --- a/crates/openshell-core/Cargo.toml +++ b/crates/openshell-core/Cargo.toml @@ -16,7 +16,7 @@ openshell-extension-core = { path = "../openshell-extension-core" } glob = { workspace = true } prost = { workspace = true } prost-types = { workspace = true } -tonic = { workspace = true, features = ["channel", "tls-ring"] } +tonic = { workspace = true, features = ["channel", "tls-aws-lc"] } tonic-prost = { workspace = true } tokio = { workspace = true } tokio-stream = { workspace = true } @@ -27,9 +27,11 @@ serde_json = { workspace = true } tracing = { workspace = true } url = { workspace = true } ipnet = "2" +rustls = { workspace = true } +rustls-pemfile = { workspace = true } base64 = { workspace = true } chrono = { version = "0.4", default-features = false, features = ["clock", "std"], optional = true } -reqwest = { workspace = true, features = ["blocking", "rustls-tls-native-roots"], optional = true } +reqwest = { workspace = true, features = ["blocking", "rustls-tls-native-roots-no-provider"], optional = true } tar = { version = "0.4", optional = true } tempfile = { version = "3", optional = true } @@ -53,6 +55,7 @@ protoc-bin-vendored = { workspace = true } [dev-dependencies] tempfile = "3" +rcgen = { workspace = true } [lints] workspace = true diff --git a/crates/openshell-core/build.rs b/crates/openshell-core/build.rs index 9caaf8eb1..38c961b1d 100644 --- a/crates/openshell-core/build.rs +++ b/crates/openshell-core/build.rs @@ -4,16 +4,20 @@ use std::env; use std::path::{Path, PathBuf}; +mod build_version; + const PROTO_REL: &str = "../../proto"; fn main() -> Result<(), Box> { // --- Git-derived version --- - // Compute a version from `git describe` for local builds. In Docker/CI - // builds where .git is absent, this silently does nothing and the binary - // falls back to CARGO_PKG_VERSION (which is already sed-patched by the - // build pipeline). + // Compute a version from tags and commit metadata for local builds. In + // Docker/CI builds where .git is absent, this silently does nothing and + // the binary falls back to CARGO_PKG_VERSION (which is already sed-patched + // by the build pipeline). println!("cargo:rerun-if-changed=../../.git/HEAD"); + println!("cargo:rerun-if-changed=../../.git/logs/HEAD"); println!("cargo:rerun-if-changed=../../.git/refs/tags"); + println!("cargo:rerun-if-changed=../../.git/packed-refs"); if let Some(version) = git_version() { println!("cargo:rustc-env=OPENSHELL_GIT_VERSION={version}"); @@ -72,53 +76,43 @@ fn collect_proto_files(dir: &Path, out: &mut Vec) -> std::io::Result<() Ok(()) } -/// Derive a version string from `git describe --tags`. +/// Derive the release or development version from git metadata. /// /// Implements the "guess-next-dev" convention used by the release pipeline -/// (`setuptools-scm`): when there are commits past the last tag, the patch -/// version is bumped and `-dev.+g` is appended. +/// (`tasks/scripts/release.py`): exact stable and prerelease tags retain their +/// version. Otherwise, the latest merged stable release gets a patch bump and +/// `-dev.+g` is appended. /// /// Examples: -/// on tag v0.0.3 → "0.0.3" -/// 3 commits past v0.0.3 → "0.0.4-dev.3+g2bf9969" +/// on tag v0.1.0-pre.1 → "0.1.0-pre.1" +/// 3 commits past v0.0.3 → "0.0.4-dev.3+g2bf9969ab" /// -/// Returns `None` when git is unavailable or the repo has no matching tags. +/// Returns `None` when git metadata cannot be read. fn git_version() -> Option { - // Match numeric release tags only (e.g. `v0.0.29`). The bare glob `v*` - // also matches non-release tags like `vm-dev` or `vm-prod`; when one of - // those lands on the same commit as a release tag, `git describe` picks - // it and the resulting version string collapses to `m-dev` after the - // leading `v` is stripped below. Requiring a digit after `v` excludes - // those development tags without losing any release tag. - let output = std::process::Command::new("git") - .args(["describe", "--tags", "--long", "--match", "v[0-9]*"]) - .output() - .ok()?; + let exact_tags = git_output(&["tag", "--points-at", "HEAD"])?; + if let Some(version) = build_version::exact_release_version(exact_tags.lines()) { + return Some(version); + } + let merged_tags = git_output(&["tag", "--merged", "HEAD", "--list", "v*.*.*"])?; + let latest_tag = build_version::latest_stable_tag(merged_tags.lines()); + let revision_range = latest_tag + .as_deref() + .map_or_else(|| "HEAD".to_string(), |tag| format!("{tag}..HEAD")); + let distance = git_output(&["rev-list", "--count", &revision_range])? + .parse() + .ok()?; + let sha = git_output(&["rev-parse", "--short=9", "HEAD"])?; + + build_version::next_dev_version(latest_tag.as_deref(), distance, &sha) +} + +fn git_output(args: &[&str]) -> Option { + let output = std::process::Command::new("git").args(args).output().ok()?; if !output.status.success() { return None; } - - let desc = String::from_utf8(output.stdout).ok()?; - let desc = desc.trim(); - let desc = desc.strip_prefix('v').unwrap_or(desc); - - // `git describe --long` format: --g - // Split from the right to handle tags that contain hyphens. - let (rest, sha) = desc.rsplit_once('-')?; - let (tag, commits_str) = rest.rsplit_once('-')?; - let commits: u32 = commits_str.parse().ok()?; - - if commits == 0 { - // Exactly on a tag — use the tag version as-is. - return Some(tag.to_string()); - } - - // Bump patch version (guess-next-dev scheme). - let mut parts = tag.splitn(3, '.'); - let major = parts.next()?; - let minor = parts.next()?; - let patch: u32 = parts.next()?.parse().ok()?; - - Some(format!("{major}.{minor}.{}-dev.{commits}+{sha}", patch + 1)) + String::from_utf8(output.stdout) + .ok() + .map(|output| output.trim().to_string()) } diff --git a/crates/openshell-core/build_version.rs b/crates/openshell-core/build_version.rs new file mode 100644 index 000000000..944808be9 --- /dev/null +++ b/crates/openshell-core/build_version.rs @@ -0,0 +1,103 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +type StableVersion = (u32, u32, u32); +type PrereleaseVersion = (u32, u32, u32, u32); + +fn parse_stable_tag(tag: &str) -> Option { + let tag = tag.strip_prefix('v').unwrap_or(tag); + let mut parts = tag.split('.'); + let version = ( + parts.next()?.parse().ok()?, + parts.next()?.parse().ok()?, + parts.next()?.parse().ok()?, + ); + parts.next().is_none().then_some(version) +} + +fn parse_prerelease_tag(tag: &str) -> Option { + let tag = tag.strip_prefix('v').unwrap_or(tag); + let (base, sequence) = tag.rsplit_once("-pre.")?; + let (major, minor, patch) = parse_stable_tag(base)?; + let sequence = sequence.parse().ok()?; + (sequence > 0).then_some((major, minor, patch, sequence)) +} + +pub fn exact_release_version<'a>(tags: impl Iterator) -> Option { + let tags = tags.collect::>(); + + if let Some(((major, minor, patch), _)) = tags + .iter() + .filter_map(|tag| parse_stable_tag(tag).map(|version| (version, tag))) + .max_by_key(|(version, _)| *version) + { + return Some(format!("{major}.{minor}.{patch}")); + } + + tags.iter() + .filter_map(|tag| parse_prerelease_tag(tag)) + .max() + .map(|(major, minor, patch, sequence)| format!("{major}.{minor}.{patch}-pre.{sequence}")) +} + +pub fn latest_stable_tag<'a>(tags: impl Iterator) -> Option { + tags.filter_map(|tag| parse_stable_tag(tag).map(|version| (version, tag))) + .max_by_key(|(version, _)| *version) + .map(|(_, tag)| tag.to_string()) +} + +pub fn next_dev_version(tag: Option<&str>, distance: u32, sha: &str) -> Option { + let (major, minor, patch) = tag.map_or(Some((0, 0, 0)), parse_stable_tag)?; + Some(format!( + "{major}.{minor}.{}-dev.{distance}+g{sha}", + patch + 1 + )) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn exact_stable_release_wins_over_prerelease() { + let tags = ["v0.1.0-pre.2", "v0.1.0", "vm-dev"]; + assert_eq!( + exact_release_version(tags.into_iter()).as_deref(), + Some("0.1.0") + ); + } + + #[test] + fn exact_prerelease_uses_highest_sequence() { + let tags = ["v0.1.0-pre.1", "v0.1.0-pre.2"]; + assert_eq!( + exact_release_version(tags.into_iter()).as_deref(), + Some("0.1.0-pre.2") + ); + } + + #[test] + fn latest_stable_ignores_prerelease_and_non_release_tags() { + let tags = ["v0.0.116", "v0.1.0-pre.1", "vm-dev", "v0.0.99"]; + assert_eq!( + latest_stable_tag(tags.into_iter()).as_deref(), + Some("v0.0.116") + ); + } + + #[test] + fn next_dev_version_bumps_latest_stable_patch() { + assert_eq!( + next_dev_version(Some("v0.0.116"), 32, "5b925dd8a").as_deref(), + Some("0.0.117-dev.32+g5b925dd8a") + ); + } + + #[test] + fn next_dev_version_without_a_release_starts_at_first_patch() { + assert_eq!( + next_dev_version(None, 7, "abcdef123").as_deref(), + Some("0.0.1-dev.7+gabcdef123") + ); + } +} diff --git a/crates/openshell-core/src/config.rs b/crates/openshell-core/src/config.rs index a1f1a6c84..76f534c88 100644 --- a/crates/openshell-core/src/config.rs +++ b/crates/openshell-core/src/config.rs @@ -6,16 +6,10 @@ use serde::{Deserialize, Serialize}; use std::borrow::Cow; use std::collections::BTreeMap; -use std::fmt; -#[cfg(unix)] -use std::io::{Read, Write}; use std::net::SocketAddr; -#[cfg(unix)] -use std::os::unix::fs::FileTypeExt; -use std::path::{Path, PathBuf}; -use std::process::{Command, Stdio}; +use std::path::PathBuf; use std::str::FromStr; -use std::time::{Duration, Instant}; +use std::time::Duration; // ── Public default constants ──────────────────────────────────────────── // @@ -35,9 +29,6 @@ pub const DEFAULT_GATEWAY_NAME: &str = "openshell"; /// Default container stop timeout in seconds (SIGTERM → SIGKILL). pub const DEFAULT_STOP_TIMEOUT_SECS: u32 = 10; -/// Default Docker bridge network name for local sandboxes. -pub const DEFAULT_DOCKER_NETWORK_NAME: &str = "openshell-docker"; - /// Default domain used for browser-facing sandbox service URLs. pub const DEFAULT_SERVICE_ROUTING_DOMAIN: &str = "openshell.localhost"; @@ -118,34 +109,9 @@ pub const CDI_GPU_DEVICE_ALL: &str = "nvidia.com/gpu=all"; /// Default maximum number of processes (PIDs) allowed inside a sandbox container. /// -/// Shared by the Docker and Podman drivers; override via driver config. +/// Compute drivers may override this through backend configuration. pub const DEFAULT_SANDBOX_PIDS_LIMIT: i64 = 2048; -/// Compute backends the gateway can orchestrate sandboxes through. -#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] -#[serde(rename_all = "snake_case")] -pub enum ComputeDriverKind { - Kubernetes, - Vm, - Docker, - Podman, - /// Microsoft MXC isolation session (Windows only). - Mxc, -} - -impl ComputeDriverKind { - #[must_use] - pub const fn as_str(self) -> &'static str { - match self { - Self::Kubernetes => "kubernetes", - Self::Vm => "vm", - Self::Docker => "docker", - Self::Podman => "podman", - Self::Mxc => "mxc", - } - } -} - /// Normalize a configured compute driver name. /// /// Built-in driver names and custom remote driver names share the same @@ -167,594 +133,6 @@ pub fn normalize_compute_driver_name(value: &str) -> Result { Ok(value.to_ascii_lowercase()) } -impl fmt::Display for ComputeDriverKind { - fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { - f.write_str(self.as_str()) - } -} - -impl FromStr for ComputeDriverKind { - type Err = String; - - fn from_str(value: &str) -> Result { - match value.trim().to_ascii_lowercase().as_str() { - "kubernetes" => Ok(Self::Kubernetes), - "vm" => Ok(Self::Vm), - "docker" => Ok(Self::Docker), - "podman" => Ok(Self::Podman), - "mxc" => Ok(Self::Mxc), - other => Err(format!( - "unsupported compute driver '{other}'. expected one of: kubernetes, vm, docker, podman, mxc" - )), - } - } -} - -/// Auto-detect the appropriate compute driver based on the runtime environment. -/// -/// Priority order: Kubernetes → Podman → Docker. -/// VM is never auto-detected (requires explicit `--drivers vm`). -/// -/// Returns the first driver where the environment check passes. -/// Returns `None` if no compatible driver is found. -pub fn detect_driver() -> Option { - // Kubernetes: check for KUBERNETES_SERVICE_HOST env var (set inside pods) - if std::env::var_os("KUBERNETES_SERVICE_HOST").is_some() { - return Some(ComputeDriverKind::Kubernetes); - } - - // Podman: check for a reachable local API socket. - if is_podman_available() { - return Some(ComputeDriverKind::Podman); - } - - // Docker: check for a reachable local API socket. - if is_docker_available() { - return Some(ComputeDriverKind::Docker); - } - - None -} - -/// Return whether a responsive local Podman API socket is available. -#[must_use] -pub fn is_podman_available() -> bool { - detect_podman_socket().is_some() -} - -/// Return the Podman API socket, or `None` if Podman is not available. -/// -/// Probes the well-known socket candidates first, then falls back to asking -/// the Podman CLI where its socket lives. The symlink at a well-known path is -/// not always present — it varies by Podman version, machine provider, and -/// platform — so the CLI fallback is what makes detection work on hosts where -/// Podman is functional but the socket is somewhere else. -pub fn detect_podman_socket() -> Option { - detect_podman_socket_from_candidates(&podman_socket_candidates()) - .or_else(discover_podman_socket) -} - -fn detect_podman_socket_from_candidates(candidates: &[PathBuf]) -> Option { - candidates - .iter() - .find(|path| podman_socket_responds(path)) - .cloned() -} - -/// Maximum time to wait for a Podman discovery subprocess. -/// -/// Driver auto-detection runs `podman info`, `podman machine inspect`, and -/// `podman system connection list` during gateway startup. A stalled Podman -/// machine, SSH connection, provider, or helper must not block startup forever, -/// so each probe is bounded and treated as "not found" on expiry — detection -/// then continues to the next candidate driver or fails with an actionable -/// error. -const PODMAN_DISCOVERY_TIMEOUT: Duration = Duration::from_secs(5); - -/// How often the bounded runner polls the child for completion. -const PODMAN_DISCOVERY_POLL_INTERVAL: Duration = Duration::from_millis(25); - -/// Run `podman ` with a bounded deadline, returning captured stdout on a -/// successful exit. Returns `None` on spawn failure, non-zero exit, or timeout. -fn run_podman_capture(args: &[&str]) -> Option> { - run_bounded_command("podman", args, PODMAN_DISCOVERY_TIMEOUT) -} - -/// Run `program ` with a deadline, capturing stdout. -/// -/// Unlike `Command::output()`, which blocks until the child exits, the deadline -/// is absolute: the call returns within `timeout` no matter what the probe or -/// its descendants do. A Podman probe can leave a daemonized descendant (an SSH -/// multiplexer, `gvproxy`, etc.) that inherited the stdout pipe, so -/// `read_to_end` would otherwise wait for EOF forever even after the direct -/// child exits — and such a descendant may even have escaped the probe's -/// process group. To stay bounded, the deadline covers both waiting for the -/// child and draining its stdout; on expiry the call best-effort kills the -/// process group (cleaning up in-group descendants) and gives up immediately, -/// abandoning the reader thread rather than waiting on it again. The abandoned -/// reader exits on its own once the pipe finally closes. Returns the captured -/// stdout only on a successful exit whose output was fully drained within the -/// deadline; otherwise `None`. -fn run_bounded_command(program: &str, args: &[&str], timeout: Duration) -> Option> { - use std::io::Read as _; - use std::sync::mpsc; - - let mut command = Command::new(program); - command - .args(args) - .stdin(Stdio::null()) - .stdout(Stdio::piped()) - .stderr(Stdio::null()); - // Run the probe as its own process-group leader so in-group descendants that - // inherited the stdout pipe can be terminated as a group on timeout. - set_new_process_group(&mut command); - let mut child = command.spawn().ok()?; - - // Drain stdout on a separate thread so a child that fills the pipe buffer - // cannot deadlock against the polling loop below. The reader reports through - // a channel so the drain can be bounded by the deadline and abandoned if it - // outlives it. - let mut stdout = child.stdout.take()?; - let (tx, rx) = mpsc::channel(); - std::thread::spawn(move || { - let mut buf = Vec::new(); - let _ = stdout.read_to_end(&mut buf); - let _ = tx.send(buf); - }); - - let deadline = Instant::now() + timeout; - - // Wait for the direct child to exit, bounded by the deadline. - let status = loop { - match child.try_wait() { - Ok(Some(status)) => break Some(status), - Ok(None) => { - if Instant::now() >= deadline { - break None; - } - std::thread::sleep(PODMAN_DISCOVERY_POLL_INTERVAL); - } - Err(_) => break None, - } - }; - - // The child never exited within the deadline: kill the group, reap the - // (now-killed) direct child, and give up. `wait()` is bounded because the - // direct child is dead. - let Some(status) = status else { - terminate_process_group(&mut child); - let _ = child.wait(); - return None; - }; - - // The direct child exited (already reaped by `try_wait`). Collect its stdout - // without ever blocking past the deadline. A surviving descendant — possibly - // one that escaped the process group — can hold the pipe open indefinitely, - // so on expiry best-effort kill the group for cleanup and return None rather - // than waiting on the reader again. - let remaining = deadline.saturating_duration_since(Instant::now()); - match rx.recv_timeout(remaining) { - Ok(stdout) if status.success() => Some(stdout), - Ok(_) => None, - Err(_) => { - terminate_process_group(&mut child); - None - } - } -} - -/// Configure `command` to start its child as a new process-group leader so the -/// group can be signaled as a unit. No-op on non-Unix platforms. -#[cfg(unix)] -fn set_new_process_group(command: &mut Command) { - use std::os::unix::process::CommandExt as _; - command.process_group(0); -} - -#[cfg(not(unix))] -fn set_new_process_group(_command: &mut Command) {} - -/// Kill the child's whole process group so daemonized descendants that inherited -/// the stdout pipe are terminated too. Falls back to killing just the child. -#[cfg(unix)] -fn terminate_process_group(child: &mut std::process::Child) { - // `set_new_process_group` made the child a group leader, so its PID doubles - // as the group ID; signaling the negated PID targets the entire group. The - // group stays valid while a descendant is alive, so this reaches survivors - // even after the leader has been reaped. - let raw_pid = i32::try_from(child.id()).unwrap_or(i32::MAX); - let pgid = nix::unistd::Pid::from_raw(-raw_pid); - let _ = nix::sys::signal::kill(pgid, nix::sys::signal::Signal::SIGKILL); - let _ = child.kill(); -} - -#[cfg(not(unix))] -fn terminate_process_group(child: &mut std::process::Child) { - let _ = child.kill(); -} - -/// Query the Podman CLI to discover the host-side API socket path. -/// -/// Strategy: -/// 1. Run `podman info --format json` to check connectivity and whether -/// the service is remote (macOS/Windows VM) or local (native Linux). -/// 2. If `CONTAINER_HOST` explicitly points at a Unix socket, `podman info` -/// just connected through it — use that path directly (a raw unix:// URL -/// has no machine to inspect and reports the VM-internal socket). -/// 3. If `serviceIsRemote` is true, run `podman machine inspect` to get -/// the host-side forwarded socket (the `remoteSocket` from `podman info` -/// is the VM-internal path, which is not reachable from the host). -/// 4. If `serviceIsRemote` is false, use `remoteSocket.path` directly -/// (on native Linux this IS the real local socket). -fn discover_podman_socket() -> Option { - let stdout = run_podman_capture(&["info", "--format", "json"])?; - - // podman info succeeded, so an explicit unix:// CONTAINER_HOST is the exact - // working host-side socket. This must be checked before the machine path, - // which cannot map a raw unix:// endpoint to a machine. - if let Some(path) = explicit_unix_container_host() { - return Some(path); - } - - let info: serde_json::Value = serde_json::from_slice(&stdout).ok()?; - let is_remote = info["host"]["serviceIsRemote"].as_bool().unwrap_or(false); - - if is_remote { - discover_podman_machine_socket() - } else { - parse_podman_info_socket(&info) - } -} - -/// Return the socket path when `CONTAINER_HOST` is an explicit `unix://` URL. -/// -/// Honors Podman's precedence: `CONTAINER_CONNECTION` outranks `CONTAINER_HOST`, -/// so a set `CONTAINER_CONNECTION` means `podman info` did not use -/// `CONTAINER_HOST` and this returns `None`. -fn explicit_unix_container_host() -> Option { - if env_var_nonempty("CONTAINER_CONNECTION").is_some() { - return None; - } - let host = env_var_nonempty("CONTAINER_HOST")?; - unix_url_socket_path(&host) -} - -/// Parse the socket path from a `unix://` URL, or `None` for other schemes. -fn unix_url_socket_path(url: &str) -> Option { - let path = url.trim().strip_prefix("unix://")?; - (!path.is_empty()).then(|| PathBuf::from(path)) -} - -/// Extract the socket path from `podman info` JSON output. -/// Used on native Linux where `remoteSocket.path` is the real local socket. -fn parse_podman_info_socket(info: &serde_json::Value) -> Option { - let path_str = info["host"]["remoteSocket"]["path"].as_str()?; - let path = path_str.strip_prefix("unix://").unwrap_or(path_str); - if path.is_empty() { - return None; - } - Some(PathBuf::from(path)) -} - -/// Which Podman machine `podman info` connected through. -/// -/// Podman resolves its endpoint (highest precedence first) from -/// `CONTAINER_CONNECTION` (a named connection), then `CONTAINER_HOST` (a URL), -/// then the default connection in `containers.conf`. -#[derive(Debug, PartialEq, Eq)] -enum ActiveMachine { - /// An explicit selector (`CONTAINER_CONNECTION`, or `CONTAINER_HOST` mapped - /// to a connection by URL) named this connection. It must match a machine - /// exactly; guessing another machine would connect to the wrong socket. - Explicit(String), - /// An explicit `CONTAINER_HOST` is set but maps to no known connection - /// (e.g. a plain remote server, not a local machine). The active machine - /// cannot be determined and must not be guessed. - UnresolvedExplicit, - /// No explicit selector; the `containers.conf` default connection name, if - /// any. When absent, Podman's built-in default machine is inspected by name. - Default(Option), -} - -/// Run `podman machine inspect ` to discover the host-side forwarded -/// socket. Used on macOS/Windows where the Podman service runs inside a VM. -/// -/// The active machine is resolved first and inspected *by name*: a no-argument -/// `podman machine inspect` inspects only `podman-machine-default`, so a host -/// whose default connection is a different machine would otherwise be pointed -/// at the wrong machine's socket. When the active machine cannot be mapped to a -/// name, this returns `None` rather than substituting an unrelated machine. -fn discover_podman_machine_socket() -> Option { - let targets = podman_machine_inspect_targets(&active_podman_machine())?; - targets.iter().find_map(|name| { - let stdout = run_podman_capture(&["machine", "inspect", name])?; - let machines: serde_json::Value = serde_json::from_slice(&stdout).ok()?; - parse_podman_machine_inspect_socket(&machines) - }) -} - -/// Machine names to try with `podman machine inspect`, most specific first. -/// -/// `None` means the active machine cannot be determined; inspection must not -/// guess, since picking an unrelated machine would return the wrong socket. A -/// rootful connection is named `-root` while the machine itself is -/// ``, so the `-root`-stripped name is offered as a fallback. -fn podman_machine_inspect_targets(active: &ActiveMachine) -> Option> { - fn names_for(connection: &str) -> Vec { - let mut names = vec![connection.to_string()]; - if let Some(stripped) = connection.strip_suffix("-root") - && !stripped.is_empty() - { - names.push(stripped.to_string()); - } - names - } - - match active { - ActiveMachine::Explicit(name) | ActiveMachine::Default(Some(name)) => Some(names_for(name)), - ActiveMachine::UnresolvedExplicit => None, - // No explicit selector and no default connection: `podman info` used - // Podman's built-in default machine, so inspect it by name rather than - // guessing an arbitrary entry. - ActiveMachine::Default(None) => Some(vec!["podman-machine-default".to_string()]), - } -} - -/// Determine which machine `podman info` connected through. -fn active_podman_machine() -> ActiveMachine { - if let Some(name) = env_var_nonempty("CONTAINER_CONNECTION") { - return ActiveMachine::Explicit(name); - } - let container_host = env_var_nonempty("CONTAINER_HOST"); - resolve_active_podman_machine(container_host.as_deref(), podman_connection_list().as_ref()) -} - -/// Resolve the active machine from `CONTAINER_HOST` and the connection list. -/// -/// `CONTAINER_CONNECTION` is handled by the caller (it needs no connection -/// list). This is split out as a pure function for testing. -fn resolve_active_podman_machine( - container_host: Option<&str>, - connections: Option<&serde_json::Value>, -) -> ActiveMachine { - if let Some(host) = container_host { - return connections - .and_then(|c| podman_connection_name_for_uri(c, host)) - .map_or(ActiveMachine::UnresolvedExplicit, ActiveMachine::Explicit); - } - ActiveMachine::Default(connections.and_then(parse_default_podman_connection)) -} - -fn env_var_nonempty(key: &str) -> Option { - std::env::var(key) - .ok() - .filter(|value| !value.trim().is_empty()) -} - -/// Run `podman system connection list --format json`. -fn podman_connection_list() -> Option { - let stdout = run_podman_capture(&["system", "connection", "list", "--format", "json"])?; - serde_json::from_slice(&stdout).ok() -} - -/// Extract the default machine connection name from -/// `podman system connection list --format json`. -fn parse_default_podman_connection(connections: &serde_json::Value) -> Option { - connections - .as_array()? - .iter() - .find(|c| { - c["Default"].as_bool().unwrap_or(false) && c["IsMachine"].as_bool().unwrap_or(false) - }) - .and_then(|c| c["Name"].as_str()) - .map(str::to_string) -} - -/// Find the machine connection whose URI matches `CONTAINER_HOST`. -/// -/// Only machine connections (`IsMachine: true`) map to a local socket, so a -/// `CONTAINER_HOST` pointing at a plain remote server yields `None`. -fn podman_connection_name_for_uri(connections: &serde_json::Value, uri: &str) -> Option { - connections - .as_array()? - .iter() - .find(|c| c["IsMachine"].as_bool().unwrap_or(false) && c["URI"].as_str() == Some(uri)) - .and_then(|c| c["Name"].as_str()) - .map(str::to_string) -} - -/// Extract the host-side socket path from a `podman machine inspect ` -/// JSON array (which contains only the inspected machine). -fn parse_podman_machine_inspect_socket(machines: &serde_json::Value) -> Option { - let machine = machines.as_array()?.first()?; - let path_str = machine["ConnectionInfo"]["PodmanSocket"]["Path"].as_str()?; - if path_str.is_empty() { - return None; - } - Some(PathBuf::from(path_str)) -} - -fn podman_socket_candidates() -> Vec { - let socket = std::env::var("OPENSHELL_PODMAN_SOCKET") - .ok() - .filter(|path| !path.trim().is_empty()) - .map(PathBuf::from); - podman_socket_candidates_from_env( - socket, - std::env::var_os("XDG_RUNTIME_DIR").map(PathBuf::from), - std::env::var_os("HOME").map(PathBuf::from), - ) -} - -fn podman_socket_candidates_from_env( - socket: Option, - runtime_dir: Option, - home: Option, -) -> Vec { - let mut candidates = Vec::new(); - - if let Some(path) = socket { - candidates.push(path); - } - - if let Some(runtime_dir) = runtime_dir { - candidates.push(runtime_dir.join("podman/podman.sock")); - } - - #[cfg(target_os = "linux")] - { - candidates.push(PathBuf::from(format!( - "/run/user/{}/podman/podman.sock", - current_uid() - ))); - } - - if let Some(home) = home { - candidates.push(home.join(".local/share/containers/podman/machine/podman.sock")); - } - - candidates -} - -/// Return whether a responsive local Docker API socket is available. -#[must_use] -pub fn is_docker_available() -> bool { - detect_docker_socket().is_some() -} - -pub fn detect_docker_socket() -> Option { - detect_docker_socket_from_candidates(&docker_socket_candidates()) -} - -fn detect_docker_socket_from_candidates(candidates: &[PathBuf]) -> Option { - candidates - .iter() - .find(|path| docker_socket_responds(path)) - .cloned() -} - -fn docker_socket_candidates() -> Vec { - let mut candidates = Vec::new(); - - if let Ok(host) = std::env::var("DOCKER_HOST") - && let Some(path) = docker_host_unix_socket_path(&host) - { - candidates.push(path); - } - - candidates.push(PathBuf::from("/var/run/docker.sock")); - - if let Some(home) = std::env::var_os("HOME") { - candidates.push(PathBuf::from(home).join(".docker/run/docker.sock")); - } - - if let Some(runtime_dir) = std::env::var_os("XDG_RUNTIME_DIR") { - candidates.push(PathBuf::from(runtime_dir).join("docker.sock")); - } - - candidates -} - -fn docker_host_unix_socket_path(host: &str) -> Option { - let path = host.trim().strip_prefix("unix://")?; - (!path.is_empty()).then(|| PathBuf::from(path)) -} - -#[cfg(unix)] -fn is_unix_socket(path: &Path) -> bool { - path.metadata() - .is_ok_and(|metadata| metadata.file_type().is_socket()) -} - -#[cfg(unix)] -fn podman_socket_responds(path: &Path) -> bool { - unix_socket_http_ping(path, |response| { - http_response_is_success(response) && contains_ascii(response, b"Libpod-Api-Version:") - }) -} - -#[cfg(unix)] -fn docker_socket_responds(path: &Path) -> bool { - unix_socket_http_ping(path, |response| { - http_response_is_success(response) - && contains_ascii(response, b"Api-Version:") - && !contains_ascii(response, b"Libpod-Api-Version:") - }) -} - -#[cfg(unix)] -fn unix_socket_http_ping(path: &Path, accepts_response: impl FnOnce(&[u8]) -> bool) -> bool { - const PROBE_TIMEOUT: Duration = Duration::from_secs(1); - const PING_REQUEST: &[u8] = - b"GET /_ping HTTP/1.1\r\nHost: localhost\r\nConnection: close\r\n\r\n"; - - if !is_unix_socket(path) { - return false; - } - - let Ok(mut stream) = std::os::unix::net::UnixStream::connect(path) else { - return false; - }; - if stream.set_read_timeout(Some(PROBE_TIMEOUT)).is_err() - || stream.set_write_timeout(Some(PROBE_TIMEOUT)).is_err() - || stream.write_all(PING_REQUEST).is_err() - { - return false; - } - - let mut response = [0_u8; 512]; - let mut total = 0; - while total < response.len() { - let Ok(n) = stream.read(&mut response[total..]) else { - return false; - }; - if n == 0 { - break; - } - total += n; - if contains_ascii(&response[..total], b"\r\n\r\n") { - break; - } - } - total > 0 && accepts_response(&response[..total]) -} - -#[cfg(unix)] -fn http_response_is_success(response: &[u8]) -> bool { - response.starts_with(b"HTTP/1.1 200") || response.starts_with(b"HTTP/1.0 200") -} - -#[cfg(unix)] -fn contains_ascii(haystack: &[u8], needle: &[u8]) -> bool { - haystack - .windows(needle.len()) - .any(|window| window.eq_ignore_ascii_case(needle)) -} - -#[cfg(all(unix, test))] -fn is_reachable_unix_socket(path: &Path) -> bool { - is_unix_socket(path) && std::os::unix::net::UnixStream::connect(path).is_ok() -} - -#[cfg(all(unix, target_os = "linux"))] -fn current_uid() -> u32 { - use std::os::unix::fs::MetadataExt; - - std::fs::metadata("/proc/self").map_or(0, |metadata| metadata.uid()) -} - -#[cfg(not(unix))] -fn podman_socket_responds(path: &Path) -> bool { - let _ = path; - false -} - -#[cfg(not(unix))] -fn docker_socket_responds(path: &Path) -> bool { - let _ = path; - false -} - /// Server configuration. /// /// Built programmatically in [`crate::Config::new`] and the gateway CLI from @@ -1447,53 +825,13 @@ const fn default_ssh_session_ttl_secs() -> u64 { #[cfg(test)] mod tests { use super::{ - ActiveMachine, ComputeDriverKind, Config, DEFAULT_SERVICE_ROUTING_DOMAIN, - GatewayInterceptorBindingPolicy, GatewayInterceptorConfig, GatewayInterceptorFailurePolicy, - GatewayJwtConfig, GatewayProviderProfileSourceConfig, PolicyValidationFailureMode, - detect_docker_socket_from_candidates, detect_driver, detect_podman_socket_from_candidates, - docker_host_unix_socket_path, docker_socket_responds, explicit_unix_container_host, - normalize_compute_driver_name, parse_default_podman_connection, parse_podman_info_socket, - parse_podman_machine_inspect_socket, podman_connection_name_for_uri, - podman_machine_inspect_targets, podman_socket_candidates_from_env, podman_socket_responds, - resolve_active_podman_machine, run_bounded_command, unix_url_socket_path, + Config, DEFAULT_SERVICE_ROUTING_DOMAIN, GatewayInterceptorBindingPolicy, + GatewayInterceptorConfig, GatewayInterceptorFailurePolicy, GatewayJwtConfig, + GatewayProviderProfileSourceConfig, PolicyValidationFailureMode, + normalize_compute_driver_name, }; - #[cfg(unix)] - use super::{is_reachable_unix_socket, is_unix_socket}; - #[cfg(unix)] - use std::io::{Read as _, Write as _}; use std::net::SocketAddr; - #[cfg(unix)] - use std::os::unix::net::UnixListener; - use std::path::PathBuf; use std::time::Duration; - #[cfg(unix)] - use std::time::Instant; - - #[test] - fn compute_driver_kind_parses_supported_values() { - assert_eq!( - "kubernetes".parse::().unwrap(), - ComputeDriverKind::Kubernetes - ); - assert_eq!( - "vm".parse::().unwrap(), - ComputeDriverKind::Vm - ); - assert_eq!( - "podman".parse::().unwrap(), - ComputeDriverKind::Podman - ); - assert_eq!( - "docker".parse::().unwrap(), - ComputeDriverKind::Docker - ); - } - - #[test] - fn compute_driver_kind_rejects_unknown_values() { - let err = "firecracker".parse::().unwrap_err(); - assert!(err.contains("unsupported compute driver 'firecracker'")); - } #[test] fn policy_validation_failure_mode_is_secure_by_default() { @@ -1657,7 +995,7 @@ mod tests { Config::new(None) .with_grpc_rate_limit(Some(10), Some(60)) .grpc_rate_limit(), - Some((10, Duration::from_secs(60))) + Some((10, Duration::from_mins(1))) ); } @@ -1714,244 +1052,6 @@ mod tests { assert_eq!(cfg.health_bind_address, Some(addr)); } - #[test] - fn detect_driver_returns_none_without_k8s_env_or_local_runtime() { - // When KUBERNETES_SERVICE_HOST is not set, no Docker binary/socket is - // available, and no Podman API socket is available, detect_driver - // should return None. - // This test may pass or fail depending on the test environment, - // but it documents the expected behavior. - let _ = detect_driver(); // Returns Some or None based on environment - } - - #[test] - fn docker_host_unix_socket_path_parses_unix_hosts() { - assert_eq!( - docker_host_unix_socket_path("unix:///var/run/docker.sock"), - Some(PathBuf::from("/var/run/docker.sock")) - ); - assert_eq!(docker_host_unix_socket_path("tcp://127.0.0.1:2375"), None); - assert_eq!(docker_host_unix_socket_path("unix://"), None); - } - - #[cfg(unix)] - #[test] - fn is_unix_socket_detects_socket_files() { - let temp_dir = tempfile::tempdir().expect("create temp dir"); - let socket_path = temp_dir.path().join("docker.sock"); - let _listener = UnixListener::bind(&socket_path).expect("bind unix socket"); - - assert!(is_unix_socket(&socket_path)); - assert!(is_reachable_unix_socket(&socket_path)); - - let regular_file = temp_dir.path().join("not-a-socket"); - std::fs::write(®ular_file, b"not a socket").expect("write regular file"); - assert!(!is_unix_socket(®ular_file)); - assert!(!is_reachable_unix_socket(®ular_file)); - } - - #[cfg(unix)] - #[test] - #[ignore = "flaky under concurrent test execution"] - fn podman_socket_probe_accepts_successful_ping_response() { - let temp_dir = tempfile::tempdir().expect("create temp dir"); - let socket_path = temp_dir.path().join("podman.sock"); - let listener = UnixListener::bind(&socket_path).expect("bind podman socket"); - - let handle = std::thread::spawn(move || { - let (mut stream, _) = listener.accept().expect("accept podman probe"); - let mut request = [0_u8; 128]; - let n = stream.read(&mut request).expect("read podman probe"); - assert!(request[..n].starts_with(b"GET /_ping HTTP/1.1\r\n")); - stream - .write_all( - b"HTTP/1.1 200 OK\r\nLibpod-Api-Version: 5.8.2\r\nContent-Length: 2\r\n\r\nOK", - ) - .expect("write podman ping response"); - }); - - assert!(podman_socket_responds(&socket_path)); - handle.join().expect("probe server exits"); - } - - #[cfg(unix)] - #[test] - #[ignore = "flaky under concurrent test execution"] - fn podman_socket_probe_rejects_docker_ping_response() { - let temp_dir = tempfile::tempdir().expect("create temp dir"); - let socket_path = temp_dir.path().join("podman.sock"); - let listener = UnixListener::bind(&socket_path).expect("bind podman socket"); - - let handle = std::thread::spawn(move || { - let (mut stream, _) = listener.accept().expect("accept podman probe"); - let mut request = [0_u8; 128]; - let n = stream.read(&mut request).expect("read podman probe"); - assert!(request[..n].starts_with(b"GET /_ping HTTP/1.1\r\n")); - stream - .write_all( - b"HTTP/1.1 200 OK\r\nServer: Docker/29.2.1\r\nContent-Length: 2\r\n\r\nOK", - ) - .expect("write docker ping response"); - }); - - assert!(!podman_socket_responds(&socket_path)); - handle.join().expect("probe server exits"); - } - - #[cfg(unix)] - #[test] - #[ignore = "flaky under concurrent test execution"] - fn docker_socket_probe_accepts_successful_ping_response() { - let temp_dir = tempfile::tempdir().expect("create temp dir"); - let socket_path = temp_dir.path().join("docker.sock"); - let listener = UnixListener::bind(&socket_path).expect("bind docker socket"); - - let handle = std::thread::spawn(move || { - let (mut stream, _) = listener.accept().expect("accept docker probe"); - let mut request = [0_u8; 128]; - let n = stream.read(&mut request).expect("read docker probe"); - assert!(request[..n].starts_with(b"GET /_ping HTTP/1.1\r\n")); - stream - .write_all( - b"HTTP/1.1 200 OK\r\nApi-Version: 1.51\r\nDocker-Experimental: false\r\nContent-Length: 2\r\n\r\nOK", - ) - .expect("write docker ping response"); - }); - - assert!(docker_socket_responds(&socket_path)); - handle.join().expect("probe server exits"); - } - - #[cfg(unix)] - #[test] - #[ignore = "flaky under concurrent test execution"] - fn docker_socket_probe_rejects_podman_ping_response() { - let temp_dir = tempfile::tempdir().expect("create temp dir"); - let socket_path = temp_dir.path().join("podman.sock"); - let listener = UnixListener::bind(&socket_path).expect("bind podman socket"); - - let handle = std::thread::spawn(move || { - let (mut stream, _) = listener.accept().expect("accept docker probe"); - let mut request = [0_u8; 128]; - let n = stream.read(&mut request).expect("read docker probe"); - assert!(request[..n].starts_with(b"GET /_ping HTTP/1.1\r\n")); - stream - .write_all( - b"HTTP/1.1 200 OK\r\nLibpod-Api-Version: 5.8.2\r\nContent-Length: 2\r\n\r\nOK", - ) - .expect("write podman ping response"); - }); - - assert!(!docker_socket_responds(&socket_path)); - handle.join().expect("probe server exits"); - } - - #[cfg(unix)] - #[test] - fn docker_socket_probe_rejects_inactive_socket() { - let temp_dir = tempfile::tempdir().expect("create temp dir"); - let socket_path = temp_dir.path().join("docker.sock"); - let listener = UnixListener::bind(&socket_path).expect("bind docker socket"); - drop(listener); - - assert!(is_unix_socket(&socket_path)); - assert!(!docker_socket_responds(&socket_path)); - } - - #[cfg(unix)] - #[test] - #[ignore = "flaky under concurrent test execution"] - fn docker_socket_detection_returns_the_responsive_candidate() { - let temp_dir = tempfile::tempdir().expect("create temp dir"); - let inactive_path = temp_dir.path().join("inactive.sock"); - let inactive_listener = UnixListener::bind(&inactive_path).expect("bind inactive socket"); - drop(inactive_listener); - - let responsive_path = temp_dir.path().join("responsive.sock"); - let listener = UnixListener::bind(&responsive_path).expect("bind responsive socket"); - let handle = std::thread::spawn(move || { - let (mut stream, _) = listener.accept().expect("accept docker probe"); - let mut request = [0_u8; 128]; - let _ = stream.read(&mut request).expect("read docker probe"); - stream - .write_all(b"HTTP/1.1 200 OK\r\nApi-Version: 1.51\r\nContent-Length: 2\r\n\r\nOK") - .expect("write docker ping response"); - }); - - assert_eq!( - detect_docker_socket_from_candidates(&[inactive_path, responsive_path.clone(),]), - Some(responsive_path) - ); - handle.join().expect("probe server exits"); - } - - #[test] - fn podman_socket_candidates_include_env_runtime_and_home_paths() { - let candidates = podman_socket_candidates_from_env( - Some(PathBuf::from("/tmp/custom-podman.sock")), - Some(PathBuf::from("/tmp/runtime")), - Some(PathBuf::from("/tmp/home")), - ); - - assert!(candidates.contains(&PathBuf::from("/tmp/custom-podman.sock"))); - assert!(candidates.contains(&PathBuf::from("/tmp/runtime/podman/podman.sock"))); - assert!(candidates.contains(&PathBuf::from( - "/tmp/home/.local/share/containers/podman/machine/podman.sock" - ))); - } - - #[cfg(unix)] - #[test] - #[ignore = "flaky under concurrent test execution"] - fn podman_socket_detection_returns_the_responsive_candidate() { - let temp_dir = tempfile::tempdir().expect("create temp dir"); - let inactive_path = temp_dir.path().join("inactive.sock"); - let inactive_listener = UnixListener::bind(&inactive_path).expect("bind inactive socket"); - drop(inactive_listener); - - let responsive_path = temp_dir.path().join("responsive.sock"); - let listener = UnixListener::bind(&responsive_path).expect("bind responsive socket"); - let handle = std::thread::spawn(move || { - let (mut stream, _) = listener.accept().expect("accept podman probe"); - let mut request = [0_u8; 128]; - let _ = stream.read(&mut request).expect("read podman probe"); - stream - .write_all( - b"HTTP/1.1 200 OK\r\nLibpod-Api-Version: 5.8.2\r\nContent-Length: 2\r\n\r\nOK", - ) - .expect("write podman ping response"); - }); - - assert_eq!( - detect_podman_socket_from_candidates(&[inactive_path, responsive_path.clone(),]), - Some(responsive_path) - ); - handle.join().expect("probe server exits"); - } - - #[test] - #[allow(unsafe_code)] // std::env::set_var/remove_var require unsafe in Rust 2024 - fn detect_driver_prefers_kubernetes_when_k8s_env_is_set() { - // Save the original env var - let original = std::env::var("KUBERNETES_SERVICE_HOST").ok(); - - // Set the env var - unsafe { - std::env::set_var("KUBERNETES_SERVICE_HOST", "127.0.0.1"); - } - - let result = detect_driver(); - assert_eq!(result, Some(ComputeDriverKind::Kubernetes)); - - // Restore the original env var - unsafe { - match original { - Some(val) => std::env::set_var("KUBERNETES_SERVICE_HOST", val), - None => std::env::remove_var("KUBERNETES_SERVICE_HOST"), - } - } - } - #[test] fn supervisor_image_tag_prefers_explicit_build_tags() { use super::resolve_supervisor_image_tag; @@ -1968,23 +1068,6 @@ mod tests { ); } - #[test] - fn parse_podman_info_socket_extracts_linux_local_socket() { - let info: serde_json::Value = serde_json::json!({ - "host": { - "serviceIsRemote": false, - "remoteSocket": { - "path": "unix:///run/user/1000/podman/podman.sock", - "exists": true - } - } - }); - assert_eq!( - parse_podman_info_socket(&info), - Some(PathBuf::from("/run/user/1000/podman/podman.sock")) - ); - } - #[test] fn supervisor_image_tag_sanitizes_build_metadata_for_oci() { use super::resolve_supervisor_image_tag; @@ -1998,22 +1081,6 @@ mod tests { ); } - #[test] - fn parse_podman_info_socket_handles_path_without_unix_prefix() { - let info: serde_json::Value = serde_json::json!({ - "host": { - "remoteSocket": { - "path": "/run/user/1000/podman/podman.sock", - "exists": true - } - } - }); - assert_eq!( - parse_podman_info_socket(&info), - Some(PathBuf::from("/run/user/1000/podman/podman.sock")) - ); - } - #[test] fn default_supervisor_image_is_version_pinned() { use super::default_supervisor_image; @@ -2022,348 +1089,4 @@ mod tests { let tag = image.rsplit_once(':').unwrap().1; assert!(!tag.is_empty()); } - - #[test] - fn parse_podman_info_socket_returns_none_for_missing_path() { - let info: serde_json::Value = serde_json::json!({ - "host": { - "remoteSocket": {} - } - }); - assert_eq!(parse_podman_info_socket(&info), None); - } - - #[test] - fn parse_podman_info_socket_returns_none_for_empty_path() { - let info: serde_json::Value = serde_json::json!({ - "host": { - "remoteSocket": { - "path": "", - "exists": false - } - } - }); - assert_eq!(parse_podman_info_socket(&info), None); - } - - #[test] - fn parse_podman_machine_inspect_socket_extracts_macos_socket() { - // `podman machine inspect ` returns only the inspected machine. - let machines: serde_json::Value = serde_json::json!([ - { - "ConnectionInfo": { - "PodmanSocket": { - "Path": "/var/folders/1q/jx7s14b928n8zvstgfk98lj00000gn/T/podman/podman-machine-default-api.sock" - }, - "PodmanPipe": null - }, - "Name": "podman-machine-default" - } - ]); - assert_eq!( - parse_podman_machine_inspect_socket(&machines), - Some(PathBuf::from( - "/var/folders/1q/jx7s14b928n8zvstgfk98lj00000gn/T/podman/podman-machine-default-api.sock" - )) - ); - } - - #[test] - fn parse_podman_machine_inspect_socket_returns_none_for_empty_array() { - let machines: serde_json::Value = serde_json::json!([]); - assert_eq!(parse_podman_machine_inspect_socket(&machines), None); - } - - #[test] - fn parse_podman_machine_inspect_socket_returns_none_for_missing_socket() { - let machines: serde_json::Value = serde_json::json!([ - { - "ConnectionInfo": {}, - "Name": "podman-machine-default" - } - ]); - assert_eq!(parse_podman_machine_inspect_socket(&machines), None); - } - - #[test] - fn podman_machine_inspect_targets_uses_explicit_machine_by_name() { - // The active connection points at `work`; discovery must inspect `work` - // explicitly rather than the no-argument default machine, which would - // return a different machine's socket. - assert_eq!( - podman_machine_inspect_targets(&ActiveMachine::Explicit("work".to_string())), - Some(vec!["work".to_string()]) - ); - } - - #[test] - fn podman_machine_inspect_targets_strips_rootful_suffix() { - // Rootful connections are named `-root`; the machine itself is - // ``, offered as a fallback after the connection name. - assert_eq!( - podman_machine_inspect_targets(&ActiveMachine::Explicit("work-root".to_string())), - Some(vec!["work-root".to_string(), "work".to_string()]) - ); - } - - #[test] - fn podman_machine_inspect_targets_uses_default_connection_name() { - assert_eq!( - podman_machine_inspect_targets(&ActiveMachine::Default(Some("work".to_string()))), - Some(vec!["work".to_string()]) - ); - } - - #[test] - fn podman_machine_inspect_targets_falls_back_to_builtin_default() { - // No explicit selector and no default connection: inspect Podman's own - // built-in default machine by name. - assert_eq!( - podman_machine_inspect_targets(&ActiveMachine::Default(None)), - Some(vec!["podman-machine-default".to_string()]) - ); - } - - #[test] - fn podman_machine_inspect_targets_does_not_guess_for_unresolved_explicit() { - // CONTAINER_HOST pointing at a non-machine endpoint cannot be mapped to - // a machine; discovery must not guess an unrelated one. - assert_eq!( - podman_machine_inspect_targets(&ActiveMachine::UnresolvedExplicit), - None - ); - } - - #[cfg(unix)] - #[test] - fn run_bounded_command_captures_stdout_on_success() { - assert_eq!( - run_bounded_command("printf", &["hello"], Duration::from_secs(5)), - Some(b"hello".to_vec()) - ); - } - - #[cfg(unix)] - #[test] - fn run_bounded_command_returns_none_on_nonzero_exit() { - assert_eq!( - run_bounded_command("false", &[], Duration::from_secs(5)), - None - ); - } - - #[cfg(unix)] - #[test] - fn run_bounded_command_kills_child_that_exceeds_deadline() { - // A process that would otherwise block startup indefinitely must be - // bounded: `run_bounded_command` returns within the deadline instead of - // hanging until the child exits. - let start = Instant::now(); - let result = run_bounded_command("sleep", &["30"], Duration::from_millis(200)); - let elapsed = start.elapsed(); - assert_eq!(result, None); - assert!( - elapsed < Duration::from_secs(5), - "bounded command did not return promptly: {elapsed:?}" - ); - } - - #[cfg(unix)] - #[test] - fn run_bounded_command_bounds_drain_when_in_group_descendant_holds_stdout() { - // The shell exits immediately after `echo`, but the backgrounded child - // (in the same process group) inherits and holds the stdout pipe open. - // Without a bounded drain, `read_to_end` would wait ~30s for EOF even - // though the direct child already exited. - let start = Instant::now(); - let result = run_bounded_command( - "sh", - &["-c", "sleep 30 & echo done"], - Duration::from_millis(300), - ); - let elapsed = start.elapsed(); - assert!( - elapsed < Duration::from_secs(2), - "drain blocked on a descendant holding stdout: {elapsed:?}" - ); - // Draining hit the deadline, so the probe is treated as "not found". - assert_eq!(result, None); - } - - #[cfg(unix)] - #[test] - fn run_bounded_command_bounds_drain_when_descendant_escapes_process_group() { - // `set -m` runs the background job in its OWN process group, so it - // survives the group kill while still holding the stdout pipe. The - // deadline must remain absolute: the call must not fall back to an - // untimed receive that waits for the escaped descendant's EOF. - let start = Instant::now(); - let result = run_bounded_command( - "bash", - &["-c", "set -m; sleep 5 & echo done"], - Duration::from_millis(300), - ); - let elapsed = start.elapsed(); - // A regression (blocking receive after the timeout) would wait ~5s for - // the escaped `sleep`; the bounded implementation returns promptly. - assert!( - elapsed < Duration::from_secs(2), - "drain blocked on a descendant that escaped the process group: {elapsed:?}" - ); - assert_eq!(result, None); - } - - #[cfg(unix)] - #[test] - fn run_bounded_command_returns_none_for_missing_program() { - assert_eq!( - run_bounded_command( - "openshell-nonexistent-binary-xyz", - &[], - Duration::from_secs(5) - ), - None - ); - } - - #[test] - fn resolve_active_podman_machine_maps_container_host_to_connection() { - let connections: serde_json::Value = serde_json::json!([ - { "Name": "work", "IsMachine": true, "Default": false, - "URI": "ssh://core@127.0.0.1:5555/run/user/1000/podman/podman.sock" }, - { "Name": "podman-machine-default", "IsMachine": true, "Default": true, - "URI": "ssh://core@127.0.0.1:4444/run/user/1000/podman/podman.sock" } - ]); - // CONTAINER_HOST pointing at the non-default machine's URI resolves to - // that machine, not the default. - assert_eq!( - resolve_active_podman_machine( - Some("ssh://core@127.0.0.1:5555/run/user/1000/podman/podman.sock"), - Some(&connections) - ), - ActiveMachine::Explicit("work".to_string()) - ); - } - - #[test] - fn resolve_active_podman_machine_unmatched_host_is_unresolved() { - let connections: serde_json::Value = serde_json::json!([ - { "Name": "podman-machine-default", "IsMachine": true, "Default": true, - "URI": "ssh://core@127.0.0.1:4444/run/user/1000/podman/podman.sock" } - ]); - // A CONTAINER_HOST that matches no machine connection is unresolved, - // never silently mapped to the default machine. - assert_eq!( - resolve_active_podman_machine(Some("tcp://192.0.2.10:2375"), Some(&connections)), - ActiveMachine::UnresolvedExplicit - ); - } - - #[test] - fn resolve_active_podman_machine_defaults_without_host() { - let connections: serde_json::Value = serde_json::json!([ - { "Name": "podman-machine-default", "IsMachine": true, "Default": true, - "URI": "ssh://core@127.0.0.1:4444/run/user/1000/podman/podman.sock" } - ]); - assert_eq!( - resolve_active_podman_machine(None, Some(&connections)), - ActiveMachine::Default(Some("podman-machine-default".to_string())) - ); - } - - #[test] - fn podman_connection_name_for_uri_ignores_non_machine_matches() { - let connections: serde_json::Value = serde_json::json!([ - { "Name": "remote", "IsMachine": false, "Default": false, - "URI": "tcp://192.0.2.10:2375" } - ]); - // A URI match against a non-machine connection does not map to a local - // machine socket. - assert_eq!( - podman_connection_name_for_uri(&connections, "tcp://192.0.2.10:2375"), - None - ); - } - - #[test] - fn unix_url_socket_path_parses_unix_urls() { - assert_eq!( - unix_url_socket_path("unix:///run/user/1000/podman/podman.sock"), - Some(PathBuf::from("/run/user/1000/podman/podman.sock")) - ); - // Non-unix schemes and empty paths are not sockets. - assert_eq!(unix_url_socket_path("ssh://core@127.0.0.1:22/x"), None); - assert_eq!(unix_url_socket_path("tcp://127.0.0.1:2375"), None); - assert_eq!(unix_url_socket_path("unix://"), None); - } - - #[test] - #[allow(unsafe_code)] // std::env::set_var/remove_var require unsafe in Rust 2024 - fn explicit_unix_container_host_honors_scheme_and_precedence() { - fn set(key: &str, value: Option<&str>) { - unsafe { - match value { - Some(v) => std::env::set_var(key, v), - None => std::env::remove_var(key), - } - } - } - - let original_host = std::env::var("CONTAINER_HOST").ok(); - let original_connection = std::env::var("CONTAINER_CONNECTION").ok(); - - // A unix:// CONTAINER_HOST with no CONTAINER_CONNECTION is used directly. - set("CONTAINER_CONNECTION", None); - set("CONTAINER_HOST", Some("unix:///tmp/podman/custom.sock")); - assert_eq!( - explicit_unix_container_host(), - Some(PathBuf::from("/tmp/podman/custom.sock")) - ); - - // CONTAINER_CONNECTION outranks CONTAINER_HOST. - set("CONTAINER_CONNECTION", Some("work")); - assert_eq!(explicit_unix_container_host(), None); - - // A non-unix CONTAINER_HOST is not a direct socket. - set("CONTAINER_CONNECTION", None); - set( - "CONTAINER_HOST", - Some("ssh://core@127.0.0.1:5555/run/podman.sock"), - ); - assert_eq!(explicit_unix_container_host(), None); - - // Nothing set. - set("CONTAINER_HOST", None); - assert_eq!(explicit_unix_container_host(), None); - - set("CONTAINER_HOST", original_host.as_deref()); - set("CONTAINER_CONNECTION", original_connection.as_deref()); - } - - #[test] - fn parse_default_podman_connection_picks_default_machine() { - let connections: serde_json::Value = serde_json::json!([ - { "Name": "podman-machine-default", "IsMachine": true, "Default": true }, - { "Name": "podman-machine-default-root", "IsMachine": true, "Default": false } - ]); - assert_eq!( - parse_default_podman_connection(&connections), - Some("podman-machine-default".to_string()) - ); - } - - #[test] - fn parse_default_podman_connection_ignores_non_machine_and_missing_default() { - // Default connection that is not a machine is ignored. - let non_machine: serde_json::Value = serde_json::json!([ - { "Name": "remote-host", "IsMachine": false, "Default": true } - ]); - assert_eq!(parse_default_podman_connection(&non_machine), None); - - // No default at all. - let no_default: serde_json::Value = serde_json::json!([ - { "Name": "work", "IsMachine": true, "Default": false } - ]); - assert_eq!(parse_default_podman_connection(&no_default), None); - } } diff --git a/crates/openshell-core/src/container_paths.rs b/crates/openshell-core/src/container_paths.rs index c63e4bcdd..e26ea53f7 100644 --- a/crates/openshell-core/src/container_paths.rs +++ b/crates/openshell-core/src/container_paths.rs @@ -65,6 +65,32 @@ pub const VM_GUEST_TLS_KEY_PATH: &str = "/opt/openshell/tls/tls.key"; pub const VM_GUEST_SANDBOX_TOKEN_PATH: &str = "/opt/openshell/auth/sandbox.jwt"; pub const VM_GUEST_INIT_DROPIN_DIR: &str = "/opt/openshell/init.d"; pub const VM_GUEST_INIT_DROPIN_MANIFEST: &str = "/opt/openshell/init.d.manifest"; + +/// Guest path for the corporate upstream-proxy credential in VM sandboxes. +/// +/// The VM driver stages the `user:pass` credential here (mode `0600`, +/// root-only) inside the per-sandbox overlay upperdir, and passes only this +/// path on the supervisor's argv. A microVM has no bind mounts or container +/// secrets, so this is the same delivery the per-sandbox JWT already uses. +pub const VM_GUEST_UPSTREAM_PROXY_AUTH_PATH: &str = "/opt/openshell/auth/upstream-proxy"; + +/// Guest path for the corporate proxy CA bundle in VM sandboxes. +/// +/// A CA certificate is not secret, so unlike the credential this is staged +/// world-readable. The supervisor trusts it for the handshake with an +/// `https://` proxy and for server certificates re-signed by a +/// TLS-intercepting proxy. +pub const VM_GUEST_PROXY_CA_PATH: &str = "/opt/openshell/tls/proxy-ca.pem"; + +/// Guest path for the driver-authored supervisor argument list in VM sandboxes. +/// +/// Podman and Kubernetes build the supervisor's command line directly; the VM +/// guest init script execs a fixed argv, so driver-owned arguments travel +/// through this file instead. The driver writes it into the overlay upperdir +/// on every launch — empty when it has no arguments to pass — so a sandbox +/// image can neither forge entries nor shadow the driver's copy, and the +/// guest appends exactly what it finds there and nothing else. +pub const VM_GUEST_SUPERVISOR_ARGS_PATH: &str = "/opt/openshell/supervisor-args"; pub const VM_UMOCI_PATH: &str = "/opt/openshell/bin/umoci"; pub const VM_SANDBOX_OWNER_NORMALIZED_MARKER: &str = "/opt/openshell/.sandbox-owner-normalized"; @@ -103,6 +129,9 @@ mod tests { VM_GUEST_SANDBOX_TOKEN_PATH, VM_GUEST_INIT_DROPIN_DIR, VM_GUEST_INIT_DROPIN_MANIFEST, + VM_GUEST_UPSTREAM_PROXY_AUTH_PATH, + VM_GUEST_PROXY_CA_PATH, + VM_GUEST_SUPERVISOR_ARGS_PATH, VM_UMOCI_PATH, VM_SANDBOX_OWNER_NORMALIZED_MARKER, ]; diff --git a/crates/openshell-core/src/driver_utils.rs b/crates/openshell-core/src/driver_utils.rs index f3279d810..ad1f42db6 100644 --- a/crates/openshell-core/src/driver_utils.rs +++ b/crates/openshell-core/src/driver_utils.rs @@ -52,6 +52,17 @@ pub fn openshell_sandbox_label_selector() -> String { /// being relaunched. pub const CONDITION_EXITED: &str = "ContainerExited"; +/// Ready-condition reason when the supervisor rejects an image-provided OCI +/// working directory because the sandbox identity lacks the required access. +pub const CONDITION_WORKSPACE_VALIDATION_FAILED: &str = "WorkspaceValidationFailed"; + +/// Supervisor exit status reserved for OCI workspace validation failures. +/// +/// Local container drivers translate this status into +/// [`CONDITION_WORKSPACE_VALIDATION_FAILED`] so users receive the specific +/// provisioning failure rather than a generic container exit. +pub const SUPERVISOR_EXIT_WORKSPACE_VALIDATION_FAILED: i32 = 78; + /// Ready-condition reason when a container was terminated by an external signal. /// /// SIGKILL/SIGTERM (exit 137/143) is what a Podman/Docker machine or daemon @@ -375,6 +386,123 @@ pub const MAX_UPSTREAM_PROXY_CREDENTIAL_BYTES: u64 = 4096; /// cannot be opened or stat'd, is not a regular file, or exceeds the size /// bound. pub fn read_upstream_proxy_credential_file(path: &str) -> Result { + read_regular_file_bounded(path, MAX_UPSTREAM_PROXY_CREDENTIAL_BYTES).map_err(|err| match err { + BoundedReadError::Open(e) => format!("failed to open proxy auth file '{path}': {e}"), + BoundedReadError::Stat(e) => format!("failed to stat proxy auth file '{path}': {e}"), + BoundedReadError::NotRegular => format!("proxy auth file '{path}' is not a regular file"), + BoundedReadError::TooLarge => format!( + "proxy auth file '{path}' exceeds the {MAX_UPSTREAM_PROXY_CREDENTIAL_BYTES}-byte limit" + ), + BoundedReadError::Read(e) => format!("failed to read proxy auth file '{path}': {e}"), + }) +} + +/// Hard upper bound on the size of a corporate proxy CA bundle file. +/// +/// A CA bundle holding every corporate trust anchor is a few tens of +/// kilobytes; this cap only exists so a hostile or misconfigured path (a huge +/// file, or a special file such as `/dev/zero`) cannot exhaust gateway, +/// driver, or supervisor memory during a bounded read. +pub const MAX_UPSTREAM_PROXY_CA_BUNDLE_BYTES: u64 = 1024 * 1024; + +/// Read and validate an operator corporate proxy CA bundle PEM file. +/// +/// Rejects non-regular files (e.g. `/dev/zero`, directories, FIFOs) and files +/// larger than [`MAX_UPSTREAM_PROXY_CA_BUNDLE_BYTES`], then requires the +/// bundle to contribute at least one trust anchor rustls actually accepts — +/// see [`validate_upstream_proxy_ca_bundle_pem`]. Returns the PEM contents. +/// +/// Shared by the compute driver (at sandbox-create time, so the operator gets +/// an error naming the setting) and the in-container supervisor (at startup), +/// so a bundle accepted on the host is never rejected inside the sandbox and +/// vice versa. This is a blocking read; async callers should wrap it (e.g. +/// `tokio::task::spawn_blocking`). +/// +/// `label` names the operator-facing setting (`proxy_ca_bundle`, or the +/// supervisor's argument name) and prefixes every error. +/// +/// # Errors +/// +/// Returns a descriptive error (never containing file contents) when the path +/// cannot be read, is not a regular file, exceeds the size bound, or holds no +/// usable certificate. +pub fn read_upstream_proxy_ca_bundle_file(path: &str, label: &str) -> Result { + let pem = read_regular_file_bounded(path, MAX_UPSTREAM_PROXY_CA_BUNDLE_BYTES).map_err( + |err| match err { + BoundedReadError::Open(e) | BoundedReadError::Stat(e) | BoundedReadError::Read(e) => { + format!("{label} '{path}' could not be read: {e}") + } + BoundedReadError::NotRegular => { + format!("{label} '{path}' is not a regular file") + } + BoundedReadError::TooLarge => format!( + "{label} '{path}' exceeds the {MAX_UPSTREAM_PROXY_CA_BUNDLE_BYTES}-byte limit" + ), + }, + )?; + validate_upstream_proxy_ca_bundle_pem(&pem, path, label)?; + Ok(pem) +} + +/// Require a CA bundle PEM to contribute at least one usable trust anchor. +/// +/// Fail-closed to match the rest of the operator-owned proxy configuration: +/// the operator explicitly pointed at this file, so a bundle with no usable +/// certificate is an error rather than a silent fall-back to the built-in +/// roots that would quietly weaken the trust boundary. +/// +/// Validating that rustls accepts an anchor — rather than only that PEM +/// framing base64-decodes — is what makes the host-side check equivalent to +/// the guest-side one: a PEM block holding invalid DER passes +/// `rustls_pemfile::certs` but is silently dropped by +/// `RootCertStore::add_parsable_certificates`, so counting PEM blocks alone +/// would accept on the host a bundle that contributes zero anchors at runtime. +/// +/// # Errors +/// +/// Returns a descriptive error, prefixed with `label` and naming `path`, when +/// the PEM holds no certificate block or no block contains valid X.509 DER. +pub fn validate_upstream_proxy_ca_bundle_pem( + pem: &str, + path: &str, + label: &str, +) -> Result<(), String> { + let certs: Vec<_> = rustls_pemfile::certs(&mut pem.as_bytes()) + .flatten() + .collect(); + if certs.is_empty() { + return Err(format!( + "{label} '{path}' contains no PEM certificate blocks" + )); + } + let mut store = rustls::RootCertStore::empty(); + let (added, _ignored) = store.add_parsable_certificates(certs); + if added == 0 { + return Err(format!( + "{label} '{path}' contains no usable trust anchors \ + (PEM blocks were found but none contain valid X.509 DER)" + )); + } + Ok(()) +} + +/// Failure modes of [`read_regular_file_bounded`], so each caller can phrase +/// them in terms of the operator setting it is reading. +enum BoundedReadError { + Open(std::io::Error), + Stat(std::io::Error), + NotRegular, + TooLarge, + Read(std::io::Error), +} + +/// Read a regular file into a `String`, rejecting anything larger than +/// `max_bytes` and anything that is not a regular file. +/// +/// Backs the operator-supplied proxy file readers, which must never let a +/// hostile or misconfigured path (`/dev/zero`, a FIFO, a directory, a huge +/// file) exhaust memory or block the caller. +fn read_regular_file_bounded(path: &str, max_bytes: u64) -> Result { use std::io::Read as _; // Windows rejects opening a directory before a file handle is available, @@ -384,10 +512,9 @@ pub fn read_upstream_proxy_credential_file(path: &str) -> Result // window if the path is replaced between these operations. #[cfg(target_os = "windows")] { - let path_metadata = std::fs::metadata(path) - .map_err(|e| format!("failed to open proxy auth file '{path}': {e}"))?; + let path_metadata = std::fs::metadata(path).map_err(BoundedReadError::Open)?; if !path_metadata.is_file() { - return Err(format!("proxy auth file '{path}' is not a regular file")); + return Err(BoundedReadError::NotRegular); } } @@ -405,31 +532,145 @@ pub fn read_upstream_proxy_credential_file(path: &str) -> Result #[cfg(not(unix))] let open_result = std::fs::File::open(path); - let file = open_result.map_err(|e| format!("failed to open proxy auth file '{path}': {e}"))?; - let metadata = file - .metadata() - .map_err(|e| format!("failed to stat proxy auth file '{path}': {e}"))?; + let file = open_result.map_err(BoundedReadError::Open)?; + let metadata = file.metadata().map_err(BoundedReadError::Stat)?; if !metadata.is_file() { - return Err(format!("proxy auth file '{path}' is not a regular file")); + return Err(BoundedReadError::NotRegular); } - if metadata.len() > MAX_UPSTREAM_PROXY_CREDENTIAL_BYTES { - return Err(format!( - "proxy auth file '{path}' exceeds the {MAX_UPSTREAM_PROXY_CREDENTIAL_BYTES}-byte limit" - )); + if metadata.len() > max_bytes { + return Err(BoundedReadError::TooLarge); } // Bound the read even if the file grows between stat and read. let mut buf = String::new(); - file.take(MAX_UPSTREAM_PROXY_CREDENTIAL_BYTES + 1) + file.take(max_bytes + 1) .read_to_string(&mut buf) - .map_err(|e| format!("failed to read proxy auth file '{path}': {e}"))?; - if buf.len() as u64 > MAX_UPSTREAM_PROXY_CREDENTIAL_BYTES { - return Err(format!( - "proxy auth file '{path}' exceeds the {MAX_UPSTREAM_PROXY_CREDENTIAL_BYTES}-byte limit" - )); + .map_err(BoundedReadError::Read)?; + if buf.len() as u64 > max_bytes { + return Err(BoundedReadError::TooLarge); } Ok(buf) } +/// Operator-supplied corporate upstream-proxy settings, as a borrowed view. +/// +/// Compute drivers store these keys under their own +/// `[openshell.drivers.]` table; this type exists so the pairing rules +/// between them live in one place instead of being restated per driver. +/// Field names map 1:1 onto the documented TOML keys `https_proxy`, +/// `no_proxy`, `proxy_auth_file`, `proxy_auth_allow_insecure`, +/// `proxy_connect_by_hostname`, and `proxy_ca_bundle`. +#[derive(Debug, Clone, Copy, Default)] +pub struct UpstreamProxySettings<'a> { + /// `https_proxy`: the corporate forward proxy URL. + pub url: Option<&'a str>, + /// `no_proxy`: comma-separated bypass list. + pub no_proxy: Option<&'a str>, + /// `proxy_auth_file`: host path to a `user:pass` credential file. + pub auth_file: Option<&'a str>, + /// `proxy_auth_allow_insecure`: acknowledgement that Basic auth to an + /// `http://` proxy travels in cleartext. + pub auth_allow_insecure: Option, + /// `proxy_connect_by_hostname`: send hostnames rather than validated IPs + /// in CONNECT requests. + pub connect_by_hostname: Option, + /// `proxy_ca_bundle`: host path to a PEM CA bundle trusted for the proxy. + pub ca_bundle: Option<&'a str>, +} + +/// Validate operator-supplied corporate upstream-proxy settings, fail-closed. +/// +/// Shares URL semantics with the in-container supervisor through +/// [`parse_upstream_proxy_url`], so a value accepted here can never be +/// rejected by the supervisor at sandbox startup (or vice versa). Every +/// auxiliary setting is only meaningful relative to a proxy boundary the +/// operator believed was in effect, so a stray one is rejected rather than +/// silently accepted while all egress dials directly. +/// +/// A present-but-empty string is rejected everywhere: the supervisor treats +/// an empty driver-supplied argument as a fatal misconfiguration, so a driver +/// must never accept (and later pass) one. +/// +/// # Errors +/// +/// Returns a message naming the offending key. +pub fn validate_upstream_proxy_settings( + settings: &UpstreamProxySettings<'_>, +) -> Result<(), String> { + let proxy_secure = if let Some(url) = settings.url { + let addr = parse_upstream_proxy_url(url).map_err(|err| match err { + UpstreamProxyUrlError::Empty => "https_proxy must not be empty when set".to_string(), + UpstreamProxyUrlError::InlineCredentials => { + "https_proxy must not embed credentials in the URL; supply them via \ + proxy_auth_file so they are not stored in config or sandbox metadata" + .to_string() + } + err => format!("https_proxy {err}"), + })?; + addr.secure + } else { + false + }; + + if let Some(list) = settings.no_proxy { + if list.trim().is_empty() { + return Err("no_proxy must not be empty when set; omit it instead".to_string()); + } + if settings.url.is_none() { + return Err("no_proxy is set but no https_proxy is configured".to_string()); + } + } + + if let Some(path) = settings.auth_file { + if path.trim().is_empty() { + return Err("proxy_auth_file must not be empty when set".to_string()); + } + if settings.url.is_none() { + return Err("proxy_auth_file is set but no https_proxy is configured".to_string()); + } + // Basic auth over the plain-TCP proxy connection is readable by + // anyone on the network path; sending it requires an explicit + // operator acknowledgement rather than being an implicit side effect + // of configuring credentials. For an https:// proxy the credential is + // inside the verified TLS session, so the acknowledgement is + // unnecessary (but tolerated). + if settings.auth_allow_insecure != Some(true) && !proxy_secure { + return Err( + "proxy_auth_file sends the credential as cleartext Basic auth over the \ + plain-TCP connection to the http:// proxy; set proxy_auth_allow_insecure \ + = true to accept that exposure, or remove proxy_auth_file" + .to_string(), + ); + } + } else if settings.auth_allow_insecure.is_some() { + // The acknowledgement without credentials means the operator believed + // an auth file was configured; surface the mismatch. + return Err( + "proxy_auth_allow_insecure is set but no proxy_auth_file is configured".to_string(), + ); + } + + if settings.connect_by_hostname.is_some() && settings.url.is_none() { + return Err( + "proxy_connect_by_hostname is set but no https_proxy is configured".to_string(), + ); + } + + // A CA bundle only makes sense relative to a proxy boundary (an https:// + // proxy handshake, or a TLS-intercepting proxy's re-sign CA). The file's + // readability and certificate content are checked at sandbox-create time + // by the driver and fail closed again in the supervisor. + if let Some(path) = settings.ca_bundle { + if path.trim().is_empty() { + return Err("proxy_ca_bundle must not be empty when set".to_string()); + } + if settings.url.is_none() { + return Err("proxy_ca_bundle is set but no https_proxy is configured".to_string()); + } + } + + Ok(()) +} + /// Container-side directory where the provider SPIFFE Workload API socket is mounted. pub const PROVIDER_SPIFFE_WORKLOAD_API_SOCKET_MOUNT_DIR: &str = "/spiffe-workload-api"; @@ -437,9 +678,9 @@ pub const PROVIDER_SPIFFE_WORKLOAD_API_SOCKET_MOUNT_DIR: &str = "/spiffe-workloa /// /// The resulting path is `$XDG_STATE_HOME/openshell/[/]//sandbox.jwt`. /// -/// `driver_subdir` is driver-specific, e.g. `"docker-sandbox-tokens"` or -/// `"podman-sandbox-tokens"`. When `namespace` is `Some`, it is appended as -/// an additional path component (with `/` and `\` replaced by `-`). +/// `driver_subdir` is driver-specific. When `namespace` is `Some`, it is +/// appended as an additional path component (with `/` and `\` replaced by +/// `-`). /// /// # Errors /// Returns an error if the XDG state directory cannot be resolved. @@ -472,7 +713,7 @@ pub fn sandbox_log_level(sandbox: &DriverSandbox, default_level: &str) -> String } // --------------------------------------------------------------------------- -// Supervisor image helpers (shared by Docker and Podman drivers) +// Supervisor image helpers shared by container-backed drivers // --------------------------------------------------------------------------- /// Return the tag portion of a supervisor image reference, or `None` if the @@ -507,7 +748,7 @@ pub fn supervisor_image_should_refresh(image: &str) -> bool { } // --------------------------------------------------------------------------- -// Supervisor binary extraction helpers (shared by Docker and Podman drivers) +// Supervisor binary extraction helpers shared by container-backed drivers // --------------------------------------------------------------------------- #[cfg(feature = "driver-extraction")] @@ -584,8 +825,7 @@ pub fn write_cache_binary_atomic(final_path: &Path, bytes: &[u8]) -> Result<(), /// Return the host-side cache path for an extracted supervisor binary. /// /// The path is `$XDG_DATA_HOME/openshell///openshell-sandbox`. -/// `driver_subdir` distinguishes caches across drivers (e.g. `"docker-supervisor"`, -/// `"podman-supervisor"`). +/// `driver_subdir` distinguishes caches across drivers. pub fn supervisor_cache_path(driver_subdir: &str, digest: &str) -> Result { let base = crate::paths::xdg_data_dir() .map_err(|err| format!("failed to resolve XDG data dir: {err}"))?; @@ -889,4 +1129,273 @@ mod tests { "reading a FIFO must not block" ); } + + /// Build settings with only the fields a case cares about. + #[test] + fn ca_bundle_file_accepts_a_real_certificate() { + // The positive case that pins host acceptance to guest acceptance: + // what the driver stages is exactly what rustls will trust. + let cert = rcgen::generate_simple_self_signed(vec!["proxy.corp.example".to_string()]) + .expect("test CA"); + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().join("proxy-ca.pem"); + std::fs::write(&path, cert.cert.pem()).unwrap(); + + let pem = + read_upstream_proxy_ca_bundle_file(path.to_str().unwrap(), "proxy_ca_bundle").unwrap(); + assert!(pem.contains("BEGIN CERTIFICATE")); + } + + #[test] + fn ca_bundle_file_rejects_non_regular_and_oversized_paths() { + // /dev/zero is the case that matters: an unbounded read of it would + // exhaust gateway or driver memory on any authorized sandbox create. + let dir = tempfile::tempdir().unwrap(); + let err = + read_upstream_proxy_ca_bundle_file(dir.path().to_str().unwrap(), "proxy_ca_bundle") + .unwrap_err(); + assert!(err.contains("regular file"), "{err}"); + assert!(err.contains("proxy_ca_bundle"), "{err}"); + + if Path::new("/dev/zero").exists() { + let err = + read_upstream_proxy_ca_bundle_file("/dev/zero", "proxy_ca_bundle").unwrap_err(); + assert!(err.contains("regular file"), "{err}"); + } + + let oversized = dir.path().join("oversized.pem"); + std::fs::write( + &oversized, + vec![b'x'; usize::try_from(MAX_UPSTREAM_PROXY_CA_BUNDLE_BYTES).unwrap() + 1], + ) + .unwrap(); + let err = + read_upstream_proxy_ca_bundle_file(oversized.to_str().unwrap(), "proxy_ca_bundle") + .unwrap_err(); + assert!(err.contains("exceeds"), "{err}"); + } + + #[test] + fn ca_bundle_file_missing_path_is_an_error() { + let err = + read_upstream_proxy_ca_bundle_file("/nonexistent/proxy-ca.pem", "proxy_ca_bundle") + .unwrap_err(); + assert!(err.contains("could not be read"), "{err}"); + } + + #[test] + fn ca_bundle_rejects_a_file_without_certificate_blocks() { + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().join("proxy-ca.pem"); + std::fs::write(&path, "this is not a certificate\n").unwrap(); + let err = read_upstream_proxy_ca_bundle_file(path.to_str().unwrap(), "proxy_ca_bundle") + .unwrap_err(); + assert!(err.contains("no PEM certificate blocks"), "{err}"); + + std::fs::write(&path, "").unwrap(); + let err = read_upstream_proxy_ca_bundle_file(path.to_str().unwrap(), "proxy_ca_bundle") + .unwrap_err(); + assert!(err.contains("no PEM certificate blocks"), "{err}"); + } + + #[test] + fn ca_bundle_rejects_pem_blocks_holding_invalid_der() { + // Passes `rustls_pemfile::certs` but contributes no trust anchor, so + // accepting it on the host would break every guest after boot. + let err = validate_upstream_proxy_ca_bundle_pem( + "-----BEGIN CERTIFICATE-----\nAQID\n-----END CERTIFICATE-----\n", + "/etc/openshell/tls/proxy-ca.pem", + "proxy_ca_bundle", + ) + .unwrap_err(); + assert!(err.contains("no usable trust anchors"), "{err}"); + } + + fn proxy_settings(url: Option<&str>) -> UpstreamProxySettings<'_> { + UpstreamProxySettings { + url, + ..UpstreamProxySettings::default() + } + } + + #[test] + fn upstream_proxy_settings_accept_a_bare_proxy_url() { + validate_upstream_proxy_settings(&proxy_settings(Some("http://proxy.corp.com:3128"))) + .expect("a lone proxy URL is a complete configuration"); + } + + #[test] + fn upstream_proxy_settings_accept_an_empty_configuration() { + validate_upstream_proxy_settings(&UpstreamProxySettings::default()) + .expect("no proxy configured at all is valid"); + } + + #[test] + fn upstream_proxy_settings_reject_an_unsupported_scheme() { + let err = validate_upstream_proxy_settings(&proxy_settings(Some("socks5://proxy:1080"))) + .expect_err("only http:// and https:// proxies are supported"); + assert!(err.starts_with("https_proxy "), "{err}"); + assert!(err.contains("unsupported proxy scheme"), "{err}"); + } + + #[test] + fn upstream_proxy_settings_reject_inline_credentials_by_naming_the_auth_file() { + let err = validate_upstream_proxy_settings(&proxy_settings(Some("http://u:p@proxy:3128"))) + .expect_err("inline credentials would be stored in gateway config"); + assert!(err.contains("proxy_auth_file"), "{err}"); + } + + #[test] + fn upstream_proxy_settings_reject_an_empty_proxy_url() { + let err = validate_upstream_proxy_settings(&proxy_settings(Some(" "))) + .expect_err("present-but-empty is a misconfiguration, not 'unset'"); + assert_eq!(err, "https_proxy must not be empty when set"); + } + + #[test] + fn upstream_proxy_settings_reject_auxiliary_keys_without_a_proxy_url() { + // Each auxiliary key implies a proxy boundary the operator believed + // was in effect; accepting one while every dial goes direct would + // hide a fail-open state. + for (settings, key) in [ + ( + UpstreamProxySettings { + no_proxy: Some("10.0.0.0/8"), + ..UpstreamProxySettings::default() + }, + "no_proxy", + ), + ( + UpstreamProxySettings { + auth_file: Some("/etc/openshell/secrets/proxy-auth"), + ..UpstreamProxySettings::default() + }, + "proxy_auth_file", + ), + ( + UpstreamProxySettings { + connect_by_hostname: Some(true), + ..UpstreamProxySettings::default() + }, + "proxy_connect_by_hostname", + ), + ( + UpstreamProxySettings { + ca_bundle: Some("/etc/openshell/tls/proxy-ca.pem"), + ..UpstreamProxySettings::default() + }, + "proxy_ca_bundle", + ), + ] { + let err = validate_upstream_proxy_settings(&settings) + .expect_err("an auxiliary key without a proxy URL must fail closed"); + assert_eq!( + err, + format!("{key} is set but no https_proxy is configured") + ); + } + } + + #[test] + fn upstream_proxy_settings_reject_empty_auxiliary_values() { + for (settings, expected) in [ + ( + UpstreamProxySettings { + url: Some("http://proxy:3128"), + no_proxy: Some(" "), + ..UpstreamProxySettings::default() + }, + "no_proxy must not be empty when set; omit it instead", + ), + ( + UpstreamProxySettings { + url: Some("http://proxy:3128"), + auth_file: Some(""), + ..UpstreamProxySettings::default() + }, + "proxy_auth_file must not be empty when set", + ), + ( + UpstreamProxySettings { + url: Some("http://proxy:3128"), + ca_bundle: Some(""), + ..UpstreamProxySettings::default() + }, + "proxy_ca_bundle must not be empty when set", + ), + ] { + let err = validate_upstream_proxy_settings(&settings) + .expect_err("present-but-empty must never be treated as unset"); + assert_eq!(err, expected); + } + } + + #[test] + fn upstream_proxy_credentials_require_the_cleartext_acknowledgement() { + let err = validate_upstream_proxy_settings(&UpstreamProxySettings { + url: Some("http://proxy:3128"), + auth_file: Some("/etc/openshell/secrets/proxy-auth"), + ..UpstreamProxySettings::default() + }) + .expect_err("Basic auth to an http:// proxy is cleartext on the wire"); + assert!(err.contains("proxy_auth_allow_insecure"), "{err}"); + + validate_upstream_proxy_settings(&UpstreamProxySettings { + url: Some("http://proxy:3128"), + auth_file: Some("/etc/openshell/secrets/proxy-auth"), + auth_allow_insecure: Some(true), + ..UpstreamProxySettings::default() + }) + .expect("the explicit acknowledgement makes the exposure an operator decision"); + } + + #[test] + fn upstream_proxy_credentials_need_no_acknowledgement_for_an_https_proxy() { + // The credential travels inside the verified TLS session to the proxy. + validate_upstream_proxy_settings(&UpstreamProxySettings { + url: Some("https://proxy:3130"), + auth_file: Some("/etc/openshell/secrets/proxy-auth"), + ..UpstreamProxySettings::default() + }) + .expect("an https:// proxy does not expose the credential on the wire"); + + // ... but setting it anyway is tolerated rather than an error. + validate_upstream_proxy_settings(&UpstreamProxySettings { + url: Some("https://proxy:3130"), + auth_file: Some("/etc/openshell/secrets/proxy-auth"), + auth_allow_insecure: Some(true), + ..UpstreamProxySettings::default() + }) + .expect("a redundant acknowledgement is tolerated"); + } + + #[test] + fn upstream_proxy_acknowledgement_without_credentials_is_rejected() { + // Including `= false`: the operator believed an auth file was + // configured, so the mismatch is surfaced rather than ignored. + for ack in [Some(true), Some(false)] { + let err = validate_upstream_proxy_settings(&UpstreamProxySettings { + url: Some("http://proxy:3128"), + auth_allow_insecure: ack, + ..UpstreamProxySettings::default() + }) + .expect_err("the acknowledgement is meaningless without a credential"); + assert_eq!( + err, + "proxy_auth_allow_insecure is set but no proxy_auth_file is configured" + ); + } + } + + #[test] + fn upstream_proxy_ca_bundle_is_valid_with_a_plain_http_proxy() { + // A TLS-intercepting proxy can be reached over plain HTTP while still + // re-signing tunneled server certificates with its own CA. + validate_upstream_proxy_settings(&UpstreamProxySettings { + url: Some("http://proxy:3128"), + ca_bundle: Some("/etc/openshell/tls/proxy-ca.pem"), + ..UpstreamProxySettings::default() + }) + .expect("an intercepting proxy's CA is meaningful without an https:// proxy URL"); + } } diff --git a/crates/openshell-core/src/operator_namespace_allowlist.rs b/crates/openshell-core/src/dynamic_string_allowlist.rs similarity index 84% rename from crates/openshell-core/src/operator_namespace_allowlist.rs rename to crates/openshell-core/src/dynamic_string_allowlist.rs index c8f0f7f3d..1a2968188 100644 --- a/crates/openshell-core/src/operator_namespace_allowlist.rs +++ b/crates/openshell-core/src/dynamic_string_allowlist.rs @@ -4,16 +4,13 @@ use std::collections::BTreeSet; use std::sync::{Arc, RwLock}; -/// Thread-safe dynamic allowlist of Kubernetes operator-mode namespaces. -/// -/// This type lives in the public core API because both the Kubernetes driver -/// and gateway authentication boundary consume it. +/// Thread-safe dynamic allowlist of strings shared across component boundaries. #[derive(Debug, Clone)] -pub struct OperatorNamespaceAllowlist { +pub struct DynamicStringAllowlist { inner: Arc>>, } -impl OperatorNamespaceAllowlist { +impl DynamicStringAllowlist { fn read_guard(&self) -> std::sync::RwLockReadGuard<'_, BTreeSet> { self.inner .read() @@ -71,7 +68,7 @@ impl OperatorNamespaceAllowlist { } } -impl Default for OperatorNamespaceAllowlist { +impl Default for DynamicStringAllowlist { fn default() -> Self { Self::new() } diff --git a/crates/openshell-core/src/error.rs b/crates/openshell-core/src/error.rs index 8c23e3019..145106012 100644 --- a/crates/openshell-core/src/error.rs +++ b/crates/openshell-core/src/error.rs @@ -106,8 +106,8 @@ impl Error { /// Error type shared by all compute driver implementations. /// -/// Both the Podman and Kubernetes drivers map their backend-specific -/// errors into these variants before crossing crate boundaries. +/// Drivers map backend-specific errors into these variants before crossing +/// crate boundaries. #[derive(Debug, Error)] pub enum ComputeDriverError { /// The requested sandbox already exists. diff --git a/crates/openshell-core/src/grpc_client.rs b/crates/openshell-core/src/grpc_client.rs index 54f0db690..38b91e850 100644 --- a/crates/openshell-core/src/grpc_client.rs +++ b/crates/openshell-core/src/grpc_client.rs @@ -1,8 +1,8 @@ // SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 -//! gRPC client for fetching sandbox policy, provider environment, and inference -//! route bundles from `OpenShell` server. +//! gRPC client for fetching sandbox policy and provider environment from the +//! `OpenShell` server. //! //! Every request carries a sandbox bearer credential in the `Authorization` //! header. The token is resolved at startup from one of three sources: @@ -24,11 +24,11 @@ use std::time::{Duration, SystemTime, UNIX_EPOCH}; use crate::proto::{ DenialSummary, ExchangeProviderSubjectTokenRequest, GetDraftPolicyRequest, - GetInferenceBundleRequest, GetInferenceBundleResponse, GetSandboxConfigRequest, - GetSandboxProviderEnvironmentRequest, IssueSandboxTokenRequest, NetworkActivitySummary, - PolicyChunk, PolicySource, PolicyStatus, RefreshSandboxTokenRequest, ReportPolicyStatusRequest, - SandboxPolicy as ProtoSandboxPolicy, SubmitPolicyAnalysisRequest, SubmitPolicyAnalysisResponse, - UpdateConfigRequest, inference_client::InferenceClient, open_shell_client::OpenShellClient, + GetSandboxConfigRequest, GetSandboxProviderEnvironmentRequest, IssueSandboxTokenRequest, + NetworkActivitySummary, PolicyChunk, PolicySource, PolicyStatus, RefreshSandboxTokenRequest, + ReportPolicyStatusRequest, SandboxPolicy as ProtoSandboxPolicy, SubmitPolicyAnalysisRequest, + SubmitPolicyAnalysisResponse, UpdateConfigRequest, open_shell_client::OpenShellClient, + workspace_selector, }; use crate::sandbox_env; use miette::{IntoDiagnostic, Result, WrapErr}; @@ -681,12 +681,6 @@ async fn connect(endpoint: &str) -> Result> { Ok(OpenShellClient::new(channel)) } -/// Connect to the inference service. -async fn connect_inference(endpoint: &str) -> Result> { - let channel = connect_channel(endpoint).await?; - Ok(InferenceClient::new(channel)) -} - /// Fetch sandbox policy from `OpenShell` server via gRPC. /// /// Returns `Ok(Some(policy))` when the server has a policy configured, @@ -759,7 +753,7 @@ async fn sync_policy_with_client( .update_config(UpdateConfigRequest { name: sandbox.to_string(), policy: Some(policy.clone()), - workspace: workspace.to_string(), + workspace_scope: Some(workspace_selector(workspace)), ..Default::default() }) .await @@ -1187,7 +1181,7 @@ impl CachedOpenShellClient { .get_draft_policy(GetDraftPolicyRequest { name: sandbox_name.to_string(), status_filter: status_filter.to_string(), - workspace: self.workspace(), + workspace_scope: Some(workspace_selector(self.workspace())), }) .await .into_diagnostic()?; @@ -1222,17 +1216,3 @@ impl CachedOpenShellClient { Ok(()) } } - -/// Fetch the resolved inference route bundle from the server. -pub async fn fetch_inference_bundle(endpoint: &str) -> Result { - debug!(endpoint = %endpoint, "Fetching inference route bundle"); - - let mut client = connect_inference(endpoint).await?; - - let response = client - .get_inference_bundle(GetInferenceBundleRequest {}) - .await - .into_diagnostic()?; - - Ok(response.into_inner()) -} diff --git a/crates/openshell-core/src/inference.rs b/crates/openshell-core/src/inference.rs deleted file mode 100644 index 2be79d45e..000000000 --- a/crates/openshell-core/src/inference.rs +++ /dev/null @@ -1,509 +0,0 @@ -// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -// SPDX-License-Identifier: Apache-2.0 - -use std::collections::HashSet; - -// --------------------------------------------------------------------------- -// Auth header abstraction -// --------------------------------------------------------------------------- - -/// How to inject an API key on outgoing inference requests. -/// -/// Defined in `openshell-core` so both `openshell-router` (which applies it) -/// and `openshell-server` / `openshell-sandbox` (which resolve it from -/// provider metadata) can share the same type. -#[derive(Debug, Clone, PartialEq, Eq)] -pub enum AuthHeader { - /// `Authorization: Bearer ` - Bearer, - /// Custom header name (e.g. `x-api-key` for Anthropic). - Custom(&'static str), - /// Do not inject any auth header on outgoing requests. The upstream - /// is expected to authenticate itself — used when the configured - /// `default_base_url` (or operator-supplied base-URL override) points - /// at a translating bridge / proxy that holds operator-side - /// credentials in its own pod and ignores caller-supplied auth. - /// Currently used by the `aws-bedrock` profile, where `SigV4` signing - /// is deferred to a follow-up PR; today the only supported shape is - /// a bridge-fronted upstream. - None, -} - -// --------------------------------------------------------------------------- -// Inference provider profiles -// --------------------------------------------------------------------------- - -/// Static metadata describing how to talk to a specific inference provider's API. -/// -/// This is the single source of truth for provider-specific inference knowledge: -/// default endpoint, supported protocols, credential key lookup order, auth -/// header style, default headers, and allowed client-supplied passthrough -/// headers. -/// -/// This is separate from [`openshell_providers::ProviderPlugin`] which handles -/// credential *discovery* (scanning env vars). `InferenceProviderProfile` handles -/// how to *use* discovered credentials to make inference API calls. -pub struct InferenceProviderProfile { - pub provider_type: &'static str, - pub default_base_url: &'static str, - pub protocols: &'static [&'static str], - /// Credential map key names to search for the API key, in priority order. - pub credential_key_names: &'static [&'static str], - /// Config map key names to search for a base URL override, in priority order. - pub base_url_config_keys: &'static [&'static str], - /// Auth header style for outgoing requests. - pub auth: AuthHeader, - /// Default headers injected on every outgoing request. - pub default_headers: &'static [(&'static str, &'static str)], - /// Client-supplied headers that may be forwarded to the upstream backend. - /// - /// Header names must be lowercase and must not include auth headers. - pub passthrough_headers: &'static [&'static str], -} - -const OPENAI_PROTOCOLS: &[&str] = &[ - "openai_chat_completions", - "openai_completions", - "openai_responses", - "openai_embeddings", - "model_discovery", -]; - -const ANTHROPIC_PROTOCOLS: &[&str] = &["anthropic_messages", "model_discovery"]; - -/// Default protocol set for the Vertex AI profile. These are overridden at route -/// resolution time in `resolve_vertex_ai_route`: Anthropic models use -/// `anthropic_messages`, while Gemini and other models use the OpenAI-compatible -/// endpoint with `openai_chat_completions`. This default applies only to the -/// base-URL-override escape hatch path. -const VERTEX_AI_PROTOCOLS: &[&str] = &["anthropic_messages", "model_discovery"]; - -// `aws_bedrock_invoke_stream` (`/model/{id}/invoke-with-response-stream`) is -// deferred to a follow-up alongside protocol-aware AWS event-stream error -// handling: the shared streaming relay's truncation/timeout path injects -// SSE-formatted error frames, which would corrupt downstream Bedrock -// event-stream parsers. Until that lands, this profile advertises only -// the buffered `InvokeModel` shape. -const AWS_BEDROCK_PROTOCOLS: &[&str] = &["aws_bedrock_invoke"]; - -static OPENAI_PROFILE: InferenceProviderProfile = InferenceProviderProfile { - provider_type: "openai", - default_base_url: "https://api.openai.com/v1", - protocols: OPENAI_PROTOCOLS, - credential_key_names: &["OPENAI_API_KEY"], - base_url_config_keys: &["OPENAI_BASE_URL"], - auth: AuthHeader::Bearer, - default_headers: &[], - passthrough_headers: &["openai-organization", "x-model-id"], -}; - -static ANTHROPIC_PROFILE: InferenceProviderProfile = InferenceProviderProfile { - provider_type: "anthropic", - default_base_url: "https://api.anthropic.com/v1", - protocols: ANTHROPIC_PROTOCOLS, - credential_key_names: &["ANTHROPIC_API_KEY"], - base_url_config_keys: &["ANTHROPIC_BASE_URL"], - auth: AuthHeader::Custom("x-api-key"), - default_headers: &[("anthropic-version", "2023-06-01")], - passthrough_headers: &["anthropic-version", "anthropic-beta"], -}; - -/// Credential environment variable names for the Vertex AI provider, in priority order. -/// -/// These are referenced by both the provider discovery logic in `openshell-providers` -/// and the inference profile here so both crates agree on which env vars hold credentials. -pub const VERTEX_AI_CREDENTIAL_KEY_NAMES: &[&str] = &[ - "GOOGLE_VERTEX_AI_SERVICE_ACCOUNT_TOKEN", - "VERTEX_AI_SERVICE_ACCOUNT_TOKEN", - "GOOGLE_VERTEX_AI_TOKEN", - "VERTEX_AI_TOKEN", -]; - -/// GCP project ID config key for Vertex AI providers. -pub const VERTEX_AI_PROJECT_ID_KEY: &str = "VERTEX_AI_PROJECT_ID"; - -/// GCP region/location config key for Vertex AI providers. -pub const VERTEX_AI_REGION_KEY: &str = "VERTEX_AI_REGION"; - -/// Publisher override config key for Vertex AI providers. -/// -/// Set to `"anthropic"` to force Anthropic Messages API routing regardless of model name, -/// or any other value to force OpenAI-compatible routing. -pub const VERTEX_AI_PUBLISHER_KEY: &str = "VERTEX_AI_PUBLISHER"; - -/// Config key names scanned during provider discovery, in addition to credential keys. -/// -/// These are referenced by the provider discovery plugin in `openshell-providers` to -/// collect Vertex AI config from the environment during `--from-existing` flows. -pub const VERTEX_AI_CONFIG_KEY_NAMES: &[&str] = &[ - VERTEX_AI_PROJECT_ID_KEY, - VERTEX_AI_REGION_KEY, - "GOOGLE_VERTEX_AI_BASE_URL", - "VERTEX_AI_BASE_URL", - VERTEX_AI_PUBLISHER_KEY, -]; - -static VERTEX_AI_PROFILE: InferenceProviderProfile = InferenceProviderProfile { - provider_type: "google-vertex-ai", - // Base URL is project/region specific and built at route resolution time. - default_base_url: "", - protocols: VERTEX_AI_PROTOCOLS, - credential_key_names: VERTEX_AI_CREDENTIAL_KEY_NAMES, - base_url_config_keys: &["GOOGLE_VERTEX_AI_BASE_URL", "VERTEX_AI_BASE_URL"], - auth: AuthHeader::Bearer, - default_headers: &[], - passthrough_headers: &[], -}; - -static NVIDIA_PROFILE: InferenceProviderProfile = InferenceProviderProfile { - provider_type: "nvidia", - default_base_url: "https://integrate.api.nvidia.com/v1", - protocols: OPENAI_PROTOCOLS, - credential_key_names: &["NVIDIA_API_KEY"], - base_url_config_keys: &["NVIDIA_BASE_URL"], - auth: AuthHeader::Bearer, - default_headers: &[], - passthrough_headers: &["x-model-id"], -}; - -static DEEPINFRA_PROFILE: InferenceProviderProfile = InferenceProviderProfile { - provider_type: "deepinfra", - default_base_url: "https://api.deepinfra.com/v1/openai", - protocols: OPENAI_PROTOCOLS, - credential_key_names: &["DEEPINFRA_API_KEY"], - base_url_config_keys: &["DEEPINFRA_BASE_URL"], - auth: AuthHeader::Bearer, - default_headers: &[], - passthrough_headers: &["x-model-id"], -}; - -// AWS Bedrock — registered as bridge-fronted (no router-side auth -// injection). Real AWS Bedrock requires `SigV4` signing of every request, -// which is deferred to a follow-up PR (see #1704 thread). Until then, -// operators point `BEDROCK_BASE_URL` at a translating bridge or -// Bedrock-compatible proxy that handles auth in its own pod. The router -// passes Bedrock `InvokeModel` requests through opaquely; the L7 pattern -// `/model/{modelId}/invoke` is wired up in -// `crates/openshell-sandbox/src/l7/inference.rs`. `InvokeModelWithResponseStream` -// is deferred to the same follow-up that adds protocol-aware error framing. -// -// Note: `default_base_url` is intentionally an empty string. Without -// `BEDROCK_BASE_URL` config, route resolution rejects the provider -// rather than silently forwarding prompts to real AWS Bedrock with -// `auth: None` (which would fail upstream and risks operator -// surprise). Once the `SigV4` follow-up lands, the default can revert -// to `https://bedrock-runtime.us-east-1.amazonaws.com`. -static AWS_BEDROCK_PROFILE: InferenceProviderProfile = InferenceProviderProfile { - provider_type: "aws-bedrock", - default_base_url: "", - protocols: AWS_BEDROCK_PROTOCOLS, - // No single API key for Bedrock — `SigV4` takes four credentials - // (access key id, secret, session token, region) and signs requests - // rather than injecting a header. Until the `SigV4` follow-up lands - // the router-side auth shape is `None` and no credential lookup is - // required at route time. - credential_key_names: &[], - base_url_config_keys: &["BEDROCK_BASE_URL"], - auth: AuthHeader::None, - default_headers: &[], - passthrough_headers: &[], -}; - -/// Canonicalize an inference provider type string to a well-known identifier. -/// -/// Returns `Some(canonical_name)` for recognized inference providers, -/// `None` for unrecognized inputs. This is the single source of truth for -/// Vertex AI (and other inference provider) alias resolution so that both -/// [`profile_for`] and `openshell-providers` normalization agree. -#[must_use] -pub fn normalize_inference_provider_type(input: &str) -> Option<&'static str> { - match input.trim().to_ascii_lowercase().as_str() { - "openai" => Some("openai"), - "anthropic" => Some("anthropic"), - "nvidia" => Some("nvidia"), - "deepinfra" => Some("deepinfra"), - "aws-bedrock" => Some("aws-bedrock"), - "google-vertex-ai" | "vertex" | "vertex-ai" | "google-vertex" | "gcp-vertex" => { - Some("google-vertex-ai") - } - _ => None, - } -} - -/// Look up the inference provider profile for a given provider type. -/// -/// Returns `None` for provider types that don't support inference routing -/// (e.g. `github`, `gitlab`, `outlook`). -pub fn profile_for(provider_type: &str) -> Option<&'static InferenceProviderProfile> { - match normalize_inference_provider_type(provider_type)? { - "openai" => Some(&OPENAI_PROFILE), - "anthropic" => Some(&ANTHROPIC_PROFILE), - "nvidia" => Some(&NVIDIA_PROFILE), - "deepinfra" => Some(&DEEPINFRA_PROFILE), - "google-vertex-ai" => Some(&VERTEX_AI_PROFILE), - "aws-bedrock" => Some(&AWS_BEDROCK_PROFILE), - _ => None, - } -} - -/// Derive the [`AuthHeader`] and default headers for a provider type string. -/// -/// This is a convenience wrapper around [`profile_for`] for callers that only -/// need the auth/header information (e.g. the sandbox bundle-to-route -/// conversion). -pub fn auth_for_provider_type(provider_type: &str) -> (AuthHeader, Vec<(String, String)>) { - let (auth, headers, _) = route_headers_for_provider_type(provider_type); - (auth, headers) -} - -/// Derive routing header policy for a provider type string. -/// -/// Returns the auth injection mode, route-level default headers, and the -/// allowed client-supplied passthrough headers for `inference.local`. -pub fn route_headers_for_provider_type( - provider_type: &str, -) -> (AuthHeader, Vec<(String, String)>, Vec) { - profile_for(provider_type).map_or_else( - || (AuthHeader::Bearer, Vec::new(), Vec::new()), - |profile| { - let headers = profile - .default_headers - .iter() - .map(|(k, v)| ((*k).to_string(), (*v).to_string())) - .collect(); - let passthrough_headers = profile - .passthrough_headers - .iter() - .map(|name| (*name).to_string()) - .collect(); - (profile.auth.clone(), headers, passthrough_headers) - }, - ) -} - -/// Derive routing header policy for a specific resolved route. -/// -/// Most providers only need their provider type. Vertex AI is special because -/// Claude routes should forward `anthropic-beta`, while Gemini/OpenAI-compatible -/// routes should not inherit Anthropic passthrough headers. -pub fn route_headers_for_route( - provider_type: &str, - protocols: &[String], -) -> (AuthHeader, Vec<(String, String)>, Vec) { - let (auth, headers, mut passthrough_headers) = route_headers_for_provider_type(provider_type); - if profile_for(provider_type).is_some_and(|profile| profile.provider_type == "google-vertex-ai") - { - let is_vertex_anthropic = protocols - .iter() - .any(|protocol| protocol == "anthropic_messages"); - if is_vertex_anthropic && !passthrough_headers.iter().any(|h| h == "anthropic-beta") { - passthrough_headers.push("anthropic-beta".to_string()); - } - } - (auth, headers, passthrough_headers) -} - -// --------------------------------------------------------------------------- -// Protocol normalization -// --------------------------------------------------------------------------- - -/// Normalize a list of protocol strings: trim, lowercase, deduplicate, skip empty. -pub fn normalize_protocols(protocols: &[String]) -> Vec { - let mut normalized = Vec::new(); - let mut seen = HashSet::new(); - - for protocol in protocols { - let candidate = protocol.trim().to_ascii_lowercase(); - if candidate.is_empty() { - continue; - } - if seen.insert(candidate.clone()) { - normalized.push(candidate); - } - } - - normalized -} - -#[cfg(test)] -mod tests { - use super::*; - - #[test] - fn normalizes_and_deduplicates() { - let input = vec![ - "OpenAI_Chat_Completions".to_string(), - " openai_chat_completions ".to_string(), - "anthropic_messages".to_string(), - ]; - let result = normalize_protocols(&input); - assert_eq!( - result, - vec!["openai_chat_completions", "anthropic_messages"] - ); - } - - #[test] - fn skips_empty_and_whitespace() { - let input = vec![String::new(), " ".to_string(), "valid".to_string()]; - let result = normalize_protocols(&input); - assert_eq!(result, vec!["valid"]); - } - - #[test] - fn empty_input() { - let result = normalize_protocols(&[]); - assert!(result.is_empty()); - } - - #[test] - fn profile_for_known_types() { - assert!(profile_for("openai").is_some()); - assert!(profile_for("anthropic").is_some()); - assert!(profile_for("nvidia").is_some()); - assert!(profile_for("deepinfra").is_some()); - assert!(profile_for("aws-bedrock").is_some()); - assert!(profile_for("OpenAI").is_some()); // case insensitive - assert!(profile_for("AWS-Bedrock").is_some()); // case insensitive - } - - #[test] - fn aws_bedrock_uses_no_auth_header() { - let (auth, headers) = auth_for_provider_type("aws-bedrock"); - assert_eq!(auth, AuthHeader::None); - assert!(headers.is_empty()); - } - - #[test] - fn aws_bedrock_profile_has_no_credential_keys() { - let profile = profile_for("aws-bedrock").expect("profile registered"); - // No router-side credential lookup until the `SigV4` follow-up. - assert!(profile.credential_key_names.is_empty()); - assert_eq!(profile.base_url_config_keys, &["BEDROCK_BASE_URL"]); - } - - #[test] - fn aws_bedrock_protocols_are_bedrock_specific() { - let profile = profile_for("aws-bedrock").expect("profile registered"); - assert!(profile.protocols.contains(&"aws_bedrock_invoke")); - // `aws_bedrock_invoke_stream` is deferred to the follow-up that adds - // protocol-aware AWS event-stream error framing; until then the - // profile advertises only the buffered `InvokeModel` shape. - assert!(!profile.protocols.contains(&"aws_bedrock_invoke_stream")); - } - - #[test] - fn profile_for_deepinfra() { - let profile = profile_for("deepinfra").expect("deepinfra profile should exist"); - assert_eq!(profile.provider_type, "deepinfra"); - assert_eq!( - profile.default_base_url, - "https://api.deepinfra.com/v1/openai" - ); - assert_eq!(profile.auth, AuthHeader::Bearer); - } - - #[test] - fn openai_compatible_profiles_include_embeddings() { - for provider_type in ["openai", "nvidia", "deepinfra"] { - let profile = profile_for(provider_type).expect("provider profile should exist"); - assert!( - profile.protocols.contains(&"openai_embeddings"), - "{provider_type} should route OpenAI-compatible embeddings" - ); - } - } - - #[test] - fn profile_for_unknown_types() { - assert!(profile_for("github").is_none()); - assert!(profile_for("gitlab").is_none()); - assert!(profile_for("unknown").is_none()); - } - - #[test] - fn auth_for_anthropic_uses_custom_header() { - let (auth, headers) = auth_for_provider_type("anthropic"); - assert_eq!(auth, AuthHeader::Custom("x-api-key")); - assert!(headers.iter().any(|(k, _)| k == "anthropic-version")); - } - - #[test] - fn route_headers_for_openai_include_passthrough_headers() { - let (_, _, passthrough_headers) = route_headers_for_provider_type("openai"); - assert!( - passthrough_headers - .iter() - .any(|name| name == "openai-organization") - ); - assert!(passthrough_headers.iter().any(|name| name == "x-model-id")); - } - - #[test] - fn route_headers_for_anthropic_include_passthrough_headers() { - let (_, _, passthrough_headers) = route_headers_for_provider_type("anthropic"); - assert!( - passthrough_headers - .iter() - .any(|name| name == "anthropic-version") - ); - assert!( - passthrough_headers - .iter() - .any(|name| name == "anthropic-beta") - ); - } - - #[test] - fn auth_for_openai_uses_bearer() { - let (auth, headers) = auth_for_provider_type("openai"); - assert_eq!(auth, AuthHeader::Bearer); - assert!(headers.is_empty()); - } - - #[test] - fn profile_for_vertex_types() { - for key in &["google-vertex-ai", "vertex", "vertex-ai"] { - let profile = profile_for(key).expect("vertex profile should be Some"); - assert_eq!(profile.provider_type, "google-vertex-ai"); - } - } - - #[test] - fn auth_for_vertex_uses_bearer() { - let (auth, headers) = auth_for_provider_type("google-vertex-ai"); - assert_eq!(auth, AuthHeader::Bearer); - assert!(headers.is_empty()); - } - - #[test] - fn route_headers_for_vertex_anthropic_route_forward_beta_only() { - let (_, headers, passthrough_headers) = - route_headers_for_route("google-vertex-ai", &["anthropic_messages".to_string()]); - assert!(headers.is_empty()); - assert_eq!(passthrough_headers, vec!["anthropic-beta".to_string()]); - } - - #[test] - fn route_headers_for_vertex_openai_route_do_not_forward_anthropic_headers() { - let (_, headers, passthrough_headers) = - route_headers_for_route("google-vertex-ai", &["openai_chat_completions".to_string()]); - assert!(headers.is_empty()); - assert!(passthrough_headers.is_empty()); - } - - #[test] - fn auth_for_unknown_defaults_to_bearer() { - let (auth, headers) = auth_for_provider_type("unknown"); - assert_eq!(auth, AuthHeader::Bearer); - assert!(headers.is_empty()); - } - - #[test] - fn route_headers_for_unknown_are_empty() { - let (auth, headers, passthrough_headers) = route_headers_for_provider_type("unknown"); - assert_eq!(auth, AuthHeader::Bearer); - assert!(headers.is_empty()); - assert!(passthrough_headers.is_empty()); - } -} diff --git a/crates/openshell-core/src/lib.rs b/crates/openshell-core/src/lib.rs index 315bb319b..1bded4469 100644 --- a/crates/openshell-core/src/lib.rs +++ b/crates/openshell-core/src/lib.rs @@ -16,6 +16,7 @@ pub mod container_paths; pub mod denial; pub mod driver_mounts; pub mod driver_utils; +pub mod dynamic_string_allowlist; pub mod endpoint_path; pub mod error; #[cfg(unix)] @@ -26,14 +27,14 @@ pub mod gpu; pub mod grpc_client; pub mod host_pattern; pub mod image; -pub mod inference; pub mod jwt; +pub mod local_api_socket; +pub mod mcp; pub mod metadata; pub mod middleware; pub mod net; #[cfg(feature = "oauth")] pub mod oauth; -pub mod operator_namespace_allowlist; pub mod paths; pub mod policy; pub mod progress; @@ -44,34 +45,39 @@ pub mod provider_credentials; pub mod sandbox_env; pub mod secrets; pub mod settings; +pub mod shell; pub mod spiffe; pub mod telemetry; pub mod time; pub mod transport_errors; pub use config::{ - ComputeDriverKind, Config, GatewayAuthConfig, GatewayInterceptorBindingOverride, - GatewayInterceptorBindingPolicy, GatewayInterceptorConfig, GatewayInterceptorFailurePolicy, - GatewayInterceptorPhaseConfig, GatewayJwtConfig, GatewayProviderProfileSourceConfig, - MtlsAuthConfig, OidcConfig, PolicyValidationFailureMode, TlsConfig, + Config, GatewayAuthConfig, GatewayInterceptorBindingOverride, GatewayInterceptorBindingPolicy, + GatewayInterceptorConfig, GatewayInterceptorFailurePolicy, GatewayInterceptorPhaseConfig, + GatewayJwtConfig, GatewayProviderProfileSourceConfig, MtlsAuthConfig, OidcConfig, + PolicyValidationFailureMode, TlsConfig, }; +pub use dynamic_string_allowlist::DynamicStringAllowlist; pub use error::{ComputeDriverError, Error, Result}; pub use metadata::{ GetResourceVersion, ObjectId, ObjectLabels, ObjectName, ObjectWorkspace, SetResourceVersion, }; -pub use operator_namespace_allowlist::OperatorNamespaceAllowlist; /// Build version string derived from git metadata. /// -/// For local builds this is computed by `build.rs` via `git describe` using -/// the guess-next-dev scheme (e.g. `0.0.4-dev.6+g2bf9969`). In Docker/CI -/// builds where `.git` is absent, falls back to `CARGO_PKG_VERSION` which -/// is already set correctly by the build pipeline's sed patch. +/// For local builds this is computed by `build.rs` from the exact release tag +/// or the latest merged stable tag using the guess-next-dev scheme (e.g. +/// `0.0.4-dev.6+g2bf9969ab`). In Docker/CI builds where `.git` is absent, it +/// falls back to `CARGO_PKG_VERSION`, which the build pipeline already stamps. pub const VERSION: &str = match option_env!("OPENSHELL_GIT_VERSION") { Some(v) => v, None => env!("CARGO_PKG_VERSION"), }; +#[cfg(test)] +#[path = "../build_version.rs"] +mod build_version; + /// Encoded protobuf `FileDescriptorSet` for every proto in `proto/`. /// /// Emitted by `build.rs` via `tonic_build::configure().file_descriptor_set_path(...)`. diff --git a/crates/openshell-core/src/local_api_socket.rs b/crates/openshell-core/src/local_api_socket.rs new file mode 100644 index 000000000..6804ae513 --- /dev/null +++ b/crates/openshell-core/src/local_api_socket.rs @@ -0,0 +1,81 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! Generic discovery and probing for local HTTP APIs over Unix sockets. + +use std::path::{Path, PathBuf}; + +/// Return the first candidate whose HTTP ping response is accepted. +#[must_use] +pub fn first_responsive_socket( + candidates: &[PathBuf], + accepts_response: impl Fn(&[u8]) -> bool, +) -> Option { + candidates + .iter() + .find(|path| socket_responds(path, &accepts_response)) + .cloned() +} + +/// Return whether a byte slice contains another, ignoring ASCII case. +#[must_use] +pub fn contains_ascii(haystack: &[u8], needle: &[u8]) -> bool { + haystack + .windows(needle.len()) + .any(|window| window.eq_ignore_ascii_case(needle)) +} + +/// Return whether an HTTP response starts with a successful status line. +#[must_use] +pub fn http_response_is_success(response: &[u8]) -> bool { + response.starts_with(b"HTTP/1.1 200") || response.starts_with(b"HTTP/1.0 200") +} + +#[cfg(unix)] +fn socket_responds(path: &Path, accepts_response: &impl Fn(&[u8]) -> bool) -> bool { + use std::io::{Read as _, Write as _}; + use std::os::unix::fs::FileTypeExt as _; + use std::time::Duration; + + const PROBE_TIMEOUT: Duration = Duration::from_secs(1); + const PING_REQUEST: &[u8] = + b"GET /_ping HTTP/1.1\r\nHost: localhost\r\nConnection: close\r\n\r\n"; + + if !path + .metadata() + .is_ok_and(|metadata| metadata.file_type().is_socket()) + { + return false; + } + let Ok(mut stream) = std::os::unix::net::UnixStream::connect(path) else { + return false; + }; + if stream.set_read_timeout(Some(PROBE_TIMEOUT)).is_err() + || stream.set_write_timeout(Some(PROBE_TIMEOUT)).is_err() + || stream.write_all(PING_REQUEST).is_err() + { + return false; + } + + let mut response = [0_u8; 512]; + let mut total = 0; + while total < response.len() { + let Ok(read) = stream.read(&mut response[total..]) else { + return false; + }; + if read == 0 { + break; + } + total += read; + if contains_ascii(&response[..total], b"\r\n\r\n") { + break; + } + } + total > 0 && accepts_response(&response[..total]) +} + +#[cfg(not(unix))] +fn socket_responds(path: &Path, accepts_response: &impl Fn(&[u8]) -> bool) -> bool { + let _ = (path, accepts_response); + false +} diff --git a/crates/openshell-core/src/mcp.rs b/crates/openshell-core/src/mcp.rs new file mode 100644 index 000000000..883ca119a --- /dev/null +++ b/crates/openshell-core/src/mcp.rs @@ -0,0 +1,467 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! OpenShell-owned MCP protocol revisions and immutable batch-shape metadata. + +use std::collections::BTreeSet; +use std::fmt; +use std::str::FromStr; + +use crate::proto::{McpOptions, ProviderProfile}; + +/// Fixed `OpenShell` batch-member bound for the MCP `2025-03-26` wire profile. +/// +/// The bound is distinct from the separately enforced request-body byte limit. +pub const MAX_MCP_LEGACY_BATCH_MESSAGES: usize = 64; + +/// Stable MCP protocol revisions accepted by `OpenShell` policy configuration. +/// +/// Variant order is the canonical semantic order used when normalizing policy +/// allowlists. +#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash)] +#[non_exhaustive] +pub enum McpProtocolVersion { + /// MCP protocol revision `2025-03-26`. + V2025_03_26, + /// MCP protocol revision `2025-06-18`. + V2025_06_18, + /// MCP protocol revision `2025-11-25`. + V2025_11_25, +} + +/// Pinned MCP protocol revision used when policy authoring omits a revision. +/// +/// This value is assigned directly rather than derived from +/// [`McpProtocolVersion::ALL`] or its order. Adding support for a newer +/// revision therefore cannot silently change versionless policies. +pub const DEFAULT_MCP_PROTOCOL_VERSION: McpProtocolVersion = McpProtocolVersion::V2025_11_25; + +/// Normalize only the MCP option fields in a provider profile. +/// +/// MCP protocol matching is case-insensitive. Missing or empty version state +/// materializes [`DEFAULT_MCP_PROTOCOL_VERSION`], while a valid explicit +/// allowlist is sorted in [`McpProtocolVersion::ALL`] semantic order. An +/// explicit list containing an unsupported, padded, or duplicate value is +/// left byte-for-byte unchanged so a later validation boundary can reject the +/// original evidence. MCP-shaped data on non-MCP endpoints is also untouched. +pub fn normalize_provider_profile_mcp_fields(profile: &mut ProviderProfile) { + for endpoint in &mut profile.endpoints { + if !endpoint.protocol.eq_ignore_ascii_case("mcp") { + continue; + } + + let Some(options) = endpoint.mcp.as_mut() else { + endpoint.mcp = Some(McpOptions { + versions: vec![DEFAULT_MCP_PROTOCOL_VERSION.as_str().to_string()], + ..McpOptions::default() + }); + continue; + }; + + if options.versions.is_empty() { + options.versions = vec![DEFAULT_MCP_PROTOCOL_VERSION.as_str().to_string()]; + continue; + } + + // Parse into the shared version type before mutation. Comparing the + // set size with the input length detects duplicates without erasing + // the duplicate values that a fail-closed validator must report. + let Ok(versions) = options + .versions + .iter() + .map(|version| version.parse::()) + .collect::, _>>() + else { + continue; + }; + if versions.len() != options.versions.len() { + continue; + } + + options.versions = versions + .into_iter() + .map(|version| version.as_str().to_string()) + .collect(); + } +} + +impl McpProtocolVersion { + /// Every supported MCP protocol revision in canonical semantic order. + pub const ALL: &'static [Self] = &[Self::V2025_03_26, Self::V2025_06_18, Self::V2025_11_25]; + + /// Return the exact protocol identifier used on the MCP wire. + #[must_use] + pub const fn as_str(self) -> &'static str { + match self { + Self::V2025_03_26 => "2025-03-26", + Self::V2025_06_18 => "2025-06-18", + Self::V2025_11_25 => "2025-11-25", + } + } + + /// Return the fixed batch-shape metadata for this protocol revision. + #[must_use] + pub const fn wire_profile(self) -> McpWireProfile { + match self { + Self::V2025_03_26 => McpWireProfile { + version: self, + allows_json_rpc_batches: true, + max_batch_messages: Some(MAX_MCP_LEGACY_BATCH_MESSAGES), + }, + Self::V2025_06_18 | Self::V2025_11_25 => McpWireProfile { + version: self, + allows_json_rpc_batches: false, + max_batch_messages: None, + }, + } + } +} + +impl fmt::Display for McpProtocolVersion { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + formatter.write_str(self.as_str()) + } +} + +impl FromStr for McpProtocolVersion { + type Err = ParseMcpProtocolVersionError; + + fn from_str(value: &str) -> Result { + match value { + "2025-03-26" => Ok(Self::V2025_03_26), + "2025-06-18" => Ok(Self::V2025_06_18), + "2025-11-25" => Ok(Self::V2025_11_25), + _ => Err(ParseMcpProtocolVersionError { + value: value.to_string(), + }), + } + } +} + +/// Error returned when a string is not an exact supported MCP revision. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct ParseMcpProtocolVersionError { + value: String, +} + +impl ParseMcpProtocolVersionError { + /// Return the exact input that failed MCP revision parsing. + #[must_use] + pub fn value(&self) -> &str { + &self.value + } +} + +impl fmt::Display for ParseMcpProtocolVersionError { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + write!( + formatter, + "unsupported MCP protocol version '{}'", + self.value + ) + } +} + +impl std::error::Error for ParseMcpProtocolVersionError {} + +/// Immutable batch-shape metadata for one exact MCP protocol revision. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)] +pub struct McpWireProfile { + version: McpProtocolVersion, + allows_json_rpc_batches: bool, + max_batch_messages: Option, +} + +impl McpWireProfile { + /// Return the protocol revision governed by this profile. + #[must_use] + pub const fn version(self) -> McpProtocolVersion { + self.version + } + + /// Return whether the revision permits a top-level JSON-RPC batch array. + #[must_use] + pub const fn allows_json_rpc_batches(self) -> bool { + self.allows_json_rpc_batches + } + + /// Return the maximum batch members when batching is permitted. + /// + /// `None` means the profile forbids batches; it never represents an + /// unbounded batch. + #[must_use] + pub const fn max_batch_messages(self) -> Option { + self.max_batch_messages + } +} + +#[cfg(test)] +mod tests { + use prost::Message; + use prost_types::{FileDescriptorSet, field_descriptor_proto}; + + use super::*; + + #[test] + fn mcp_protocol_version_all_values_are_in_canonical_order() { + assert_eq!( + McpProtocolVersion::ALL, + &[ + McpProtocolVersion::V2025_03_26, + McpProtocolVersion::V2025_06_18, + McpProtocolVersion::V2025_11_25, + ] + ); + assert!( + McpProtocolVersion::ALL + .windows(2) + .all(|versions| versions[0] < versions[1]) + ); + } + + #[test] + fn default_mcp_protocol_version_is_pinned_to_the_2025_11_25_profile() { + assert_eq!( + DEFAULT_MCP_PROTOCOL_VERSION, + McpProtocolVersion::V2025_11_25 + ); + assert_eq!(DEFAULT_MCP_PROTOCOL_VERSION.as_str(), "2025-11-25"); + + let profile = DEFAULT_MCP_PROTOCOL_VERSION.wire_profile(); + assert_eq!(profile.version(), McpProtocolVersion::V2025_11_25); + assert!(!profile.allows_json_rpc_batches()); + assert_eq!(profile.max_batch_messages(), None); + } + + fn provider_profile_with_mcp(protocol: &str, options: Option) -> ProviderProfile { + ProviderProfile { + id: "mcp-profile".to_string(), + display_name: "MCP profile".to_string(), + description: "source-owned description".to_string(), + endpoints: vec![crate::proto::NetworkEndpoint { + host: "mcp.example.com".to_string(), + port: 443, + protocol: protocol.to_string(), + mcp: options, + ..crate::proto::NetworkEndpoint::default() + }], + ..ProviderProfile::default() + } + } + + #[test] + fn provider_profile_mcp_normalization_materializes_omitted_and_empty_versions() { + let mut omitted = provider_profile_with_mcp("McP", None); + normalize_provider_profile_mcp_fields(&mut omitted); + assert_eq!( + omitted.endpoints[0] + .mcp + .as_ref() + .expect("omitted MCP options must materialize") + .versions, + ["2025-11-25"] + ); + + let mut empty = provider_profile_with_mcp( + "mcp", + Some(McpOptions { + strict_tool_names: Some(false), + allow_all_known_mcp_methods: Some(true), + versions: Vec::new(), + }), + ); + normalize_provider_profile_mcp_fields(&mut empty); + assert_eq!( + empty.endpoints[0] + .mcp + .as_ref() + .expect("empty MCP versions must materialize"), + &McpOptions { + strict_tool_names: Some(false), + allow_all_known_mcp_methods: Some(true), + versions: vec!["2025-11-25".to_string()], + } + ); + } + + #[test] + fn provider_profile_mcp_normalization_canonicalizes_valid_explicit_versions_only() { + let mut profile = provider_profile_with_mcp( + "mcp", + Some(McpOptions { + strict_tool_names: Some(true), + versions: vec![ + "2025-11-25".to_string(), + "2025-03-26".to_string(), + "2025-06-18".to_string(), + ], + ..McpOptions::default() + }), + ); + let original = profile.clone(); + + normalize_provider_profile_mcp_fields(&mut profile); + + assert_eq!(profile.id, original.id); + assert_eq!(profile.display_name, original.display_name); + assert_eq!(profile.description, original.description); + assert_eq!(profile.endpoints[0].host, original.endpoints[0].host); + assert_eq!(profile.endpoints[0].port, original.endpoints[0].port); + assert_eq!( + profile.endpoints[0].protocol, + original.endpoints[0].protocol + ); + assert_eq!( + profile.endpoints[0] + .mcp + .as_ref() + .expect("valid MCP options"), + &McpOptions { + strict_tool_names: Some(true), + versions: vec![ + "2025-03-26".to_string(), + "2025-06-18".to_string(), + "2025-11-25".to_string(), + ], + ..McpOptions::default() + } + ); + } + + #[test] + fn provider_profile_mcp_normalization_preserves_every_malformed_explicit_list() { + for versions in [ + vec!["2025-11-25", "2025-11-25"], + vec![" 2025-11-25"], + vec!["2025-11-25 "], + vec!["2026-07-28"], + vec!["latest"], + vec!["draft"], + ] { + let mut profile = provider_profile_with_mcp( + "mcp", + Some(McpOptions { + versions: versions.into_iter().map(ToString::to_string).collect(), + ..McpOptions::default() + }), + ); + let original = profile.clone(); + + normalize_provider_profile_mcp_fields(&mut profile); + + assert_eq!(profile, original); + } + } + + #[test] + fn provider_profile_mcp_normalization_ignores_non_mcp_endpoint_evidence() { + let mut profile = provider_profile_with_mcp( + "rest", + Some(McpOptions { + versions: vec!["latest".to_string()], + ..McpOptions::default() + }), + ); + let original = profile.clone(); + + normalize_provider_profile_mcp_fields(&mut profile); + + assert_eq!(profile, original); + } + + #[test] + fn mcp_protocol_version_parses_only_exact_supported_values() { + assert_eq!("2025-03-26".parse(), Ok(McpProtocolVersion::V2025_03_26)); + assert_eq!("2025-06-18".parse(), Ok(McpProtocolVersion::V2025_06_18)); + assert_eq!("2025-11-25".parse(), Ok(McpProtocolVersion::V2025_11_25)); + + for unsupported in [ + "", + "2025-03-26 ", + " 2025-06-18", + "2025-11-25\n", + "2025-11-24", + "2026-07-28", + "draft", + "latest", + ] { + let error = unsupported.parse::(); + assert_eq!( + error, + Err(ParseMcpProtocolVersionError { + value: unsupported.to_string(), + }) + ); + } + } + + #[test] + fn mcp_protocol_version_display_and_as_str_round_trip() { + for version in McpProtocolVersion::ALL.iter().copied() { + assert_eq!(version.to_string(), version.as_str()); + assert_eq!(version.as_str().parse(), Ok(version)); + } + } + + #[test] + fn mcp_protocol_version_error_reports_the_rejected_input() { + let rejected = " 2025-03-26"; + let error = rejected + .parse::() + .expect_err("padded MCP revision must be rejected"); + + assert_eq!( + error.to_string(), + format!("unsupported MCP protocol version '{rejected}'") + ); + assert_eq!(error.value(), rejected); + } + + #[test] + fn mcp_protocol_version_wire_profiles_define_batch_metadata() { + assert_eq!(MAX_MCP_LEGACY_BATCH_MESSAGES, 64); + + let legacy = McpProtocolVersion::V2025_03_26.wire_profile(); + assert_eq!(legacy.version(), McpProtocolVersion::V2025_03_26); + assert!(legacy.allows_json_rpc_batches()); + assert_eq!( + legacy.max_batch_messages(), + Some(MAX_MCP_LEGACY_BATCH_MESSAGES) + ); + + for version in [ + McpProtocolVersion::V2025_06_18, + McpProtocolVersion::V2025_11_25, + ] { + let profile = version.wire_profile(); + assert_eq!(profile.version(), version); + assert!(!profile.allows_json_rpc_batches()); + assert_eq!(profile.max_batch_messages(), None); + } + } + + #[test] + fn mcp_options_versions_field_number_is_stable() { + let descriptor_set = FileDescriptorSet::decode(crate::FILE_DESCRIPTOR_SET) + .expect("OpenShell descriptor set must decode"); + let mcp_options = descriptor_set + .file + .iter() + .find(|file| file.package.as_deref() == Some("openshell.sandbox.v1")) + .and_then(|file| { + file.message_type + .iter() + .find(|message| message.name.as_deref() == Some("McpOptions")) + }) + .expect("sandbox schema must define McpOptions"); + let versions = mcp_options + .field + .iter() + .find(|field| field.name.as_deref() == Some("versions")) + .expect("McpOptions must define versions"); + + assert_eq!(versions.number, Some(3)); + assert_eq!(versions.label(), field_descriptor_proto::Label::Repeated); + assert_eq!(versions.r#type(), field_descriptor_proto::Type::String); + } +} diff --git a/crates/openshell-core/src/metadata.rs b/crates/openshell-core/src/metadata.rs index 8794c11d5..ccff87fd8 100644 --- a/crates/openshell-core/src/metadata.rs +++ b/crates/openshell-core/src/metadata.rs @@ -6,8 +6,8 @@ //! These traits provide uniform access to `ObjectMeta` fields across all resource types. use crate::proto::{ - InferenceRoute, ObjectForTest, Provider, Sandbox, SandboxStatus, ServiceEndpoint, SshSession, - StoredProviderCredentialRefreshState, StoredProviderProfile, Workspace, WorkspaceMember, + ObjectForTest, Provider, Sandbox, SandboxStatus, SandboxWorkloadTemplate, ServiceEndpoint, + SshSession, Workspace, WorkspaceMember, }; use std::collections::HashMap; @@ -104,6 +104,49 @@ impl Sandbox { } } +// Implementations for SandboxWorkloadTemplate +impl ObjectId for SandboxWorkloadTemplate { + fn object_id(&self) -> &str { + self.metadata.as_ref().map_or("", |m| m.id.as_str()) + } +} + +impl ObjectName for SandboxWorkloadTemplate { + fn object_name(&self) -> &str { + self.metadata.as_ref().map_or("", |m| m.name.as_str()) + } +} + +impl ObjectLabels for SandboxWorkloadTemplate { + fn object_labels(&self) -> Option> { + self.metadata.as_ref().map(|m| m.labels.clone()) + } +} + +impl SetResourceVersion for SandboxWorkloadTemplate { + fn set_resource_version(&mut self, version: u64) { + if let Some(meta) = self.metadata.as_mut() { + meta.resource_version = version; + } + } +} + +impl GetResourceVersion for SandboxWorkloadTemplate { + fn get_resource_version(&self) -> u64 { + self.metadata.as_ref().map_or(0, |m| m.resource_version) + } +} + +impl ObjectWorkspace for SandboxWorkloadTemplate { + fn object_workspace(&self) -> &str { + self.metadata.as_ref().map_or("", |m| m.workspace.as_str()) + } + + fn requires_workspace() -> bool { + true + } +} + // Implementations for Workspace impl ObjectId for Workspace { fn object_id(&self) -> &str { @@ -189,90 +232,6 @@ impl ObjectWorkspace for Provider { } } -// Implementations for StoredProviderProfile -impl ObjectId for StoredProviderProfile { - fn object_id(&self) -> &str { - self.metadata.as_ref().map_or("", |m| m.id.as_str()) - } -} - -impl ObjectName for StoredProviderProfile { - fn object_name(&self) -> &str { - self.metadata.as_ref().map_or("", |m| m.name.as_str()) - } -} - -impl ObjectLabels for StoredProviderProfile { - fn object_labels(&self) -> Option> { - self.metadata.as_ref().map(|m| m.labels.clone()) - } -} - -impl SetResourceVersion for StoredProviderProfile { - fn set_resource_version(&mut self, version: u64) { - if let Some(meta) = self.metadata.as_mut() { - meta.resource_version = version; - } - } -} - -impl GetResourceVersion for StoredProviderProfile { - fn get_resource_version(&self) -> u64 { - self.metadata.as_ref().map_or(0, |m| m.resource_version) - } -} - -impl ObjectWorkspace for StoredProviderProfile { - fn object_workspace(&self) -> &str { - self.metadata.as_ref().map_or("", |m| m.workspace.as_str()) - } - fn requires_workspace() -> bool { - false - } -} - -// Implementations for StoredProviderCredentialRefreshState -impl ObjectId for StoredProviderCredentialRefreshState { - fn object_id(&self) -> &str { - self.metadata.as_ref().map_or("", |m| m.id.as_str()) - } -} - -impl ObjectName for StoredProviderCredentialRefreshState { - fn object_name(&self) -> &str { - self.metadata.as_ref().map_or("", |m| m.name.as_str()) - } -} - -impl ObjectLabels for StoredProviderCredentialRefreshState { - fn object_labels(&self) -> Option> { - self.metadata.as_ref().map(|m| m.labels.clone()) - } -} - -impl SetResourceVersion for StoredProviderCredentialRefreshState { - fn set_resource_version(&mut self, version: u64) { - if let Some(meta) = self.metadata.as_mut() { - meta.resource_version = version; - } - } -} - -impl GetResourceVersion for StoredProviderCredentialRefreshState { - fn get_resource_version(&self) -> u64 { - self.metadata.as_ref().map_or(0, |m| m.resource_version) - } -} - -impl ObjectWorkspace for StoredProviderCredentialRefreshState { - fn object_workspace(&self) -> &str { - self.metadata.as_ref().map_or("", |m| m.workspace.as_str()) - } - fn requires_workspace() -> bool { - true - } -} - // Implementations for SshSession impl ObjectId for SshSession { fn object_id(&self) -> &str { @@ -357,48 +316,6 @@ impl ObjectWorkspace for ServiceEndpoint { } } -// Implementations for InferenceRoute -impl ObjectId for InferenceRoute { - fn object_id(&self) -> &str { - self.metadata.as_ref().map_or("", |m| m.id.as_str()) - } -} - -impl ObjectName for InferenceRoute { - fn object_name(&self) -> &str { - self.metadata.as_ref().map_or("", |m| m.name.as_str()) - } -} - -impl ObjectLabels for InferenceRoute { - fn object_labels(&self) -> Option> { - self.metadata.as_ref().map(|m| m.labels.clone()) - } -} - -impl SetResourceVersion for InferenceRoute { - fn set_resource_version(&mut self, version: u64) { - if let Some(meta) = self.metadata.as_mut() { - meta.resource_version = version; - } - } -} - -impl GetResourceVersion for InferenceRoute { - fn get_resource_version(&self) -> u64 { - self.metadata.as_ref().map_or(0, |m| m.resource_version) - } -} - -impl ObjectWorkspace for InferenceRoute { - fn object_workspace(&self) -> &str { - self.metadata.as_ref().map_or("", |m| m.workspace.as_str()) - } - fn requires_workspace() -> bool { - true - } -} - // Implementations for WorkspaceMember impl ObjectId for WorkspaceMember { fn object_id(&self) -> &str { diff --git a/crates/openshell-core/src/middleware.rs b/crates/openshell-core/src/middleware.rs index 2b3fb1898..d1d59f2a8 100644 --- a/crates/openshell-core/src/middleware.rs +++ b/crates/openshell-core/src/middleware.rs @@ -11,11 +11,17 @@ use tokio::sync::mpsc; use tonic::{Request, Response, Status}; use crate::proto::{ - HttpHeader, HttpRequestEvaluation, HttpRequestResult, HttpRequestTarget, MiddlewareManifest, - RequestContext, SupervisorMiddlewarePhase, ValidateConfigRequest, ValidateConfigResponse, - WebSocketSessionEvent, WebSocketSessionEventResult, + HttpHeader, HttpRequestEvaluation, HttpRequestResult, HttpRequestTarget, HttpResponseEvent, + HttpResponseEventResult, MiddlewareManifest, RequestContext, SupervisorMiddlewarePhase, + ValidateConfigRequest, ValidateConfigResponse, WebSocketSessionEvent, + WebSocketSessionEventResult, }; +/// Transport-neutral result stream for one HTTP response middleware stage. +pub type HttpResponseResultStream = Pin< + Box> + Send + 'static>, +>; + /// Transport-neutral response stream for one WebSocket middleware stage. pub type WebSocketResponseStream = Pin< Box< @@ -47,6 +53,15 @@ pub trait SupervisorMiddlewareEndpoint: Send + Sync { &self, requests: mpsc::Receiver, ) -> Result; + + async fn open_http_response_pre_return( + &self, + _requests: mpsc::Receiver, + ) -> Result { + Err(Status::unimplemented( + "middleware does not implement HTTP response pre-return evaluation", + )) + } } /// Borrowed request state exposed to one in-process middleware invocation. @@ -242,6 +257,18 @@ pub trait InProcessMiddleware: Send + Sync { "middleware does not implement WebSocket sessions", )) } + + /// Open one HTTP response pre-return stream. + /// + /// Request-only implementations may keep the default unsupported response. + async fn open_http_response_pre_return( + &self, + _requests: mpsc::Receiver, + ) -> std::result::Result { + Err(Status::unimplemented( + "middleware does not implement HTTP response pre-return evaluation", + )) + } } /// Default timeout for one supervisor middleware RPC. diff --git a/crates/openshell-core/src/oauth.rs b/crates/openshell-core/src/oauth.rs index c68ecfa6b..d3a9e3472 100644 --- a/crates/openshell-core/src/oauth.rs +++ b/crates/openshell-core/src/oauth.rs @@ -312,6 +312,7 @@ mod tests { } fn test_client() -> reqwest::Client { + let _ = rustls::crypto::aws_lc_rs::default_provider().install_default(); reqwest::Client::builder() .timeout(std::time::Duration::from_secs(5)) .no_proxy() diff --git a/crates/openshell-core/src/policy.rs b/crates/openshell-core/src/policy.rs index 1645b9da4..af474325c 100644 --- a/crates/openshell-core/src/policy.rs +++ b/crates/openshell-core/src/policy.rs @@ -92,6 +92,19 @@ pub enum LandlockCompatibility { HardRequirement, } +/// Accepted `landlock.compatibility` values in their proto string form. +/// +/// Single source of truth shared by YAML parsing, proto→runtime conversion, +/// and gateway policy validation so the accepted set cannot drift. +pub const LANDLOCK_COMPATIBILITY_VALUES: [&str; 2] = ["best_effort", "hard_requirement"]; + +/// Returns `true` if `value` is an accepted `landlock.compatibility` string. +/// +/// The empty string is accepted and defaults to `best_effort`. +pub fn is_valid_landlock_compatibility(value: &str) -> bool { + value.is_empty() || LANDLOCK_COMPATIBILITY_VALUES.contains(&value) +} + // ============================================================================ // Proto to Rust type conversions // ============================================================================ @@ -101,7 +114,7 @@ impl TryFrom for SandboxPolicy { fn try_from(proto: ProtoSandboxPolicy) -> Result { // In cluster mode we always run with proxy networking so all egress - // can be evaluated by OPA and `inference.local` is always addressable. + // can be evaluated by OPA. let network = NetworkPolicy { mode: NetworkMode::Proxy, proxy: Some(ProxyPolicy { http_addr: None }), @@ -114,7 +127,11 @@ impl TryFrom for SandboxPolicy { .map(FilesystemPolicy::from) .unwrap_or_default(), network, - landlock: proto.landlock.map(LandlockPolicy::from).unwrap_or_default(), + landlock: proto + .landlock + .map(LandlockPolicy::try_from) + .transpose()? + .unwrap_or_default(), process: proto.process.map(ProcessPolicy::from).unwrap_or_default(), }) } @@ -138,14 +155,20 @@ impl From for FilesystemPolicy { } } -impl From for LandlockPolicy { - fn from(proto: ProtoLandlockPolicy) -> Self { - let compatibility = if proto.compatibility == "hard_requirement" { - LandlockCompatibility::HardRequirement - } else { - LandlockCompatibility::BestEffort +impl TryFrom for LandlockPolicy { + type Error = miette::Error; + + fn try_from(proto: ProtoLandlockPolicy) -> Result { + let compatibility = match proto.compatibility.as_str() { + "best_effort" | "" => LandlockCompatibility::BestEffort, + "hard_requirement" => LandlockCompatibility::HardRequirement, + otherwise => miette::bail!( + "invalid landlock.compatibility {:?}; accepted: {}", + otherwise, + LANDLOCK_COMPATIBILITY_VALUES.join(", ") + ), }; - Self { compatibility } + Ok(Self { compatibility }) } } @@ -165,3 +188,49 @@ impl From for ProcessPolicy { } } } + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn try_from_maps_known_compatibility_values() { + for (input, expected) in [ + ("", LandlockCompatibility::BestEffort), + ("best_effort", LandlockCompatibility::BestEffort), + ("hard_requirement", LandlockCompatibility::HardRequirement), + ] { + let proto = ProtoLandlockPolicy { + compatibility: input.into(), + }; + let policy = LandlockPolicy::try_from(proto).expect("should convert"); + assert_eq!( + std::mem::discriminant(&policy.compatibility), + std::mem::discriminant(&expected), + "input {input:?} mapped to unexpected variant", + ); + } + } + + #[test] + fn try_from_rejects_invalid_compatibility() { + let proto = ProtoLandlockPolicy { + compatibility: "hard-requirement".into(), + }; + let err = LandlockPolicy::try_from(proto).expect_err("should reject"); + let msg = format!("{err:?}"); + assert!( + msg.contains("best_effort") && msg.contains("hard_requirement"), + "error should list accepted values, got: {msg}", + ); + } + + #[test] + fn is_valid_landlock_compatibility_accepts_empty_and_known() { + assert!(is_valid_landlock_compatibility("")); + assert!(is_valid_landlock_compatibility("best_effort")); + assert!(is_valid_landlock_compatibility("hard_requirement")); + assert!(!is_valid_landlock_compatibility("nope")); + assert!(!is_valid_landlock_compatibility("BestEffort")); + } +} diff --git a/crates/openshell-core/src/proto/mod.rs b/crates/openshell-core/src/proto/mod.rs index d3b340581..d0b976ff2 100644 --- a/crates/openshell-core/src/proto/mod.rs +++ b/crates/openshell-core/src/proto/mod.rs @@ -59,10 +59,6 @@ pub mod test { pub use super::generated::openshell::test::v1::*; } -pub mod inference { - pub use super::generated::openshell::inference::v1; -} - pub mod middleware { pub use super::generated::openshell::middleware::v1; } @@ -73,8 +69,23 @@ pub mod gateway_interceptor { pub use datamodel::v1::*; pub use gateway_interceptor::v1::*; -pub use inference::v1::*; pub use middleware::v1::*; pub use openshell::*; pub use sandbox::v1::*; pub use test::ObjectForTest; + +/// Build a selector for one explicitly named workspace. +pub fn workspace_selector(workspace: impl Into) -> WorkspaceSelector { + WorkspaceSelector { + selection: Some(workspace_selector::Selection::Workspace(workspace.into())), + } +} + +/// Build a selector for every workspace supported by a cross-workspace request. +pub fn all_workspaces_selector() -> WorkspaceSelector { + WorkspaceSelector { + selection: Some(workspace_selector::Selection::AllWorkspaces( + AllWorkspaces {}, + )), + } +} diff --git a/crates/openshell-core/src/sandbox_env.rs b/crates/openshell-core/src/sandbox_env.rs index 8da9a7d2a..2ce8e4b05 100644 --- a/crates/openshell-core/src/sandbox_env.rs +++ b/crates/openshell-core/src/sandbox_env.rs @@ -38,6 +38,9 @@ const MAIN_PROCESS_SPEC_BASE64URL_PREFIX: &str = "base64url:"; #[derive(Clone, Debug, Deserialize, Eq, PartialEq, Serialize)] pub struct MainProcessConfig { pub version: u32, + /// Canonical command. Empty means "no command supplied": the supervisor + /// resolves the default login shell against the sandbox image. A non-empty + /// command is the exact program+args and is run verbatim. pub command: Vec, pub tty: bool, #[serde(default)] @@ -47,11 +50,15 @@ pub struct MainProcessConfig { impl MainProcessConfig { pub const VERSION: u32 = 1; + /// Default config for a sandbox created without a command. The command is + /// left empty on purpose: the supervisor picks a login shell that exists in + /// the sandbox image (bash when present, otherwise `/bin/sh`). A TTY is + /// requested because the default is an interactive login shell. #[must_use] pub fn scratch() -> Self { Self { version: Self::VERSION, - command: vec!["/bin/bash".to_string(), "-l".to_string()], + command: Vec::new(), tty: true, await_main_process_attachment: false, } @@ -91,8 +98,13 @@ impl MainProcessConfig { config.version )); } - if config.command.is_empty() || config.command[0].is_empty() { - return Err(format!("{MAIN_PROCESS_SPEC} command must not be empty")); + // An empty command is valid: it means "no command supplied", and the + // supervisor resolves the default login shell. Only a present-but-blank + // program is rejected. + if !config.command.is_empty() && config.command[0].is_empty() { + return Err(format!( + "{MAIN_PROCESS_SPEC} command program must not be empty" + )); } Ok(config) } @@ -105,7 +117,7 @@ impl MainProcessConfig { } /// Encode the versioned transport without whitespace for constrained - /// environment-variable transports such as libkrun. + /// environment-variable transports used by embedded runtimes. pub fn encode_driver_spec_base64url( spec: Option<&crate::proto::compute::v1::DriverSandboxSpec>, ) -> Result { @@ -283,4 +295,40 @@ mod tests { let encoded = serde_json::to_string(&config).unwrap(); assert_eq!(MainProcessConfig::decode(&encoded).unwrap(), config); } + + #[test] + fn omitted_command_stays_empty_for_supervisor_resolution() { + // No command supplied → empty command; the supervisor resolves the + // default login shell against the sandbox image. + let empty = crate::proto::compute::v1::DriverSandboxSpec::default(); + assert!( + MainProcessConfig::from_driver_spec(Some(&empty)) + .command + .is_empty() + ); + assert!(MainProcessConfig::from_driver_spec(None).command.is_empty()); + + // An explicit command is preserved verbatim and never rewritten. + let explicit = crate::proto::compute::v1::DriverSandboxSpec { + command: vec!["/bin/bash".into(), "-l".into()], + tty: true, + ..Default::default() + }; + assert_eq!( + MainProcessConfig::from_driver_spec(Some(&explicit)).command, + vec!["/bin/bash".to_string(), "-l".to_string()] + ); + + // An empty command survives the transport round-trip. + let encoded = serde_json::to_string(&MainProcessConfig::scratch()).unwrap(); + assert!( + MainProcessConfig::decode(&encoded) + .unwrap() + .command + .is_empty() + ); + + // A present-but-blank program is still rejected. + assert!(MainProcessConfig::decode(r#"{"version":1,"command":[""],"tty":false}"#).is_err()); + } } diff --git a/crates/openshell-core/src/secrets.rs b/crates/openshell-core/src/secrets.rs index bef39c8fe..0ec130bd8 100644 --- a/crates/openshell-core/src/secrets.rs +++ b/crates/openshell-core/src/secrets.rs @@ -54,6 +54,38 @@ pub fn contains_reserved_credential_marker(value: &str) -> bool { contains_raw_reserved_marker(&decoded) } +/// Return whether an HTTP header value contains syntax reserved for credential +/// placeholder rewriting, including the decoded value of Basic authentication. +pub fn header_value_contains_reserved_credential_marker(value: &str) -> bool { + let trimmed = value.trim(); + if contains_reserved_credential_marker(trimmed) { + return true; + } + + let Some(decoded) = decode_basic_auth_value(trimmed) else { + return false; + }; + contains_raw_reserved_marker(&decoded) +} + +fn basic_auth_token(value: &str) -> Option<&str> { + value + .strip_prefix("Basic ") + .or_else(|| value.strip_prefix("basic ")) + .map(str::trim) +} + +fn decode_basic_auth_value(value: &str) -> Option { + decode_basic_auth_token(basic_auth_token(value)?) +} + +fn decode_basic_auth_token(encoded: &str) -> Option { + let decoded = base64::engine::general_purpose::STANDARD + .decode(encoded) + .ok()?; + String::from_utf8(decoded).ok() +} + pub fn contains_reserved_credential_marker_bytes(value: &[u8]) -> bool { if value.is_empty() { return false; @@ -494,10 +526,7 @@ impl SecretResolver { // Basic auth decoding: `Basic ` where the decoded content // contains a placeholder (e.g. `user:openshell:resolve:env:PASS`). - if let Some(encoded) = trimmed - .strip_prefix("Basic ") - .or_else(|| trimmed.strip_prefix("basic ")) - .map(str::trim) + if let Some(encoded) = basic_auth_token(trimmed) && let Some(rewritten) = self.rewrite_basic_auth_token(encoded)? { return Ok(Some(format!("Basic {rewritten}"))); @@ -633,18 +662,15 @@ impl SecretResolver { encoded: &str, ) -> Result, UnresolvedPlaceholderError> { let b64 = base64::engine::general_purpose::STANDARD; - let Some(decoded_bytes) = b64.decode(encoded.trim()).ok() else { - return Ok(None); - }; - let Some(decoded) = std::str::from_utf8(&decoded_bytes).ok() else { + let Some(decoded) = decode_basic_auth_token(encoded.trim()) else { return Ok(None); }; - if !contains_raw_reserved_marker(decoded) { + if !contains_raw_reserved_marker(&decoded) { return Ok(None); } - let mut rewritten = decoded.to_string(); + let mut rewritten = decoded; let replacements = self.rewrite_text_placeholders(&mut rewritten, "header")?; if replacements == 0 { @@ -1413,6 +1439,28 @@ mod tests { ])); } + #[test] + fn header_value_marker_detection_covers_rewrite_forms() { + let basic = + base64::engine::general_purpose::STANDARD.encode("user:openshell:resolve:env:API_KEY"); + + for value in [ + "openshell:resolve:env:API_KEY".to_string(), + "Bearer openshell:resolve:env:API_KEY".to_string(), + "provider-OPENSHELL-RESOLVE-ENV-API_KEY".to_string(), + "openshell%3Aresolve%3Aenv%3AAPI_KEY".to_string(), + format!("Basic {basic}"), + ] { + assert!( + header_value_contains_reserved_credential_marker(&value), + "reserved credential marker in {value:?}" + ); + } + assert!(!header_value_contains_reserved_credential_marker( + "Bearer ordinary-token" + )); + } + fn fully_percent_encoded(marker: &str) -> String { const HEX: &[u8; 16] = b"0123456789ABCDEF"; let mut encoded = String::with_capacity(marker.len() * 3); diff --git a/crates/openshell-core/src/settings.rs b/crates/openshell-core/src/settings.rs index 640707158..84702c9aa 100644 --- a/crates/openshell-core/src/settings.rs +++ b/crates/openshell-core/src/settings.rs @@ -65,7 +65,7 @@ impl RegisteredSetting { /// /// 1. Add a [`RegisteredSetting`] entry to this array with the key name and /// [`SettingValueKind`]. -/// 2. Recompile `openshell-server` (gateway) and `openshell-sandbox` +/// 2. Recompile `openshell-gateway` and `openshell-sandbox` /// (supervisor). No database migration is needed -- new keys are stored in /// the existing settings JSON blob. /// 3. Add sandbox-side consumption in `openshell-sandbox` to read and act on @@ -74,8 +74,7 @@ impl RegisteredSetting { /// settable via `settings set`. The server validates that only registered /// keys are accepted. /// 5. Add a unit test in this module's `tests` section to cover the new key. -pub const PROVIDERS_V2_ENABLED_KEY: &str = "providers_v2_enabled"; - +/// /// Sandbox-level opt-in for the agent-driven policy proposal surface. /// /// When true, the supervisor installs the `policy_advisor` skill, serves @@ -107,14 +106,15 @@ pub const PROPOSAL_APPROVAL_MODE_KEY: &str = "proposal_approval_mode"; /// fail-closes on unknown persisted values for defense in depth. pub const PROPOSAL_APPROVAL_MODE_VALUES: &[&str] = &["manual", "auto"]; +/// Allowed values for `ocsf_schema_version`. +/// +/// Only versions with actual downgrade transforms in +/// `openshell_ocsf::format::downgrade` are accepted. Empty string disables +/// downgrade (equivalent to unsetting the key). Malformed or unsupported +/// versions (e.g. `"banana"`, `"1.6"`) are rejected at configure time. +pub const OCSF_SCHEMA_VERSION_VALUES: &[&str] = &["", "1.1", "1.3"]; + pub const REGISTERED_SETTINGS: &[RegisteredSetting] = &[ - // Gateway-level opt-in for provider profile policy composition. Defaults - // to false when unset. - RegisteredSetting { - key: PROVIDERS_V2_ENABLED_KEY, - kind: SettingValueKind::Bool, - allowed_string_values: None, - }, // When true the sandbox writes OCSF v1.8.0 JSONL records to // `/var/log/openshell-ocsf*.log` (daily rotation, 3 files) in addition // to the human-readable shorthand log. Defaults to false (no JSONL written). @@ -123,6 +123,14 @@ pub const REGISTERED_SETTINGS: &[RegisteredSetting] = &[ kind: SettingValueKind::Bool, allowed_string_values: None, }, + // Target OCSF schema version for JSONL downgrade. When set (e.g. "1.1" + // or "1.3"), the JSONL layer strips fields and profiles that don't exist + // in the target version. Empty or unset means no downgrade. + RegisteredSetting { + key: "ocsf_schema_version", + kind: SettingValueKind::String, + allowed_string_values: Some(OCSF_SCHEMA_VERSION_VALUES), + }, // Sandbox-level opt-in for the agent-driven policy proposal surface. // See AGENT_POLICY_PROPOSALS_ENABLED_KEY for details. Defaults to false. RegisteredSetting { @@ -168,7 +176,7 @@ pub fn parse_bool_like(raw: &str) -> Option { #[cfg(test)] mod tests { use super::{ - PROPOSAL_APPROVAL_MODE_KEY, PROPOSAL_APPROVAL_MODE_VALUES, PROVIDERS_V2_ENABLED_KEY, + OCSF_SCHEMA_VERSION_VALUES, PROPOSAL_APPROVAL_MODE_KEY, PROPOSAL_APPROVAL_MODE_VALUES, REGISTERED_SETTINGS, RegisteredSetting, SettingValueKind, parse_bool_like, registered_keys_csv, setting_for_key, }; @@ -185,18 +193,16 @@ mod tests { } #[test] - fn setting_for_key_returns_providers_v2_enabled() { - let setting = setting_for_key(PROVIDERS_V2_ENABLED_KEY) - .expect("providers_v2_enabled should be registered"); - assert_eq!(setting.kind, SettingValueKind::Bool); + fn setting_for_key_rejects_removed_providers_v2_enabled() { + assert!(setting_for_key("providers_v2_enabled").is_none()); } // ---- RegisteredSetting::validate_string_value ---- #[test] fn validate_string_value_accepts_anything_when_unconstrained() { - let setting = setting_for_key(PROVIDERS_V2_ENABLED_KEY) - .expect("providers_v2_enabled should be registered"); + let setting = + setting_for_key("ocsf_json_enabled").expect("ocsf_json_enabled should be registered"); // Bool-kind entries currently leave `allowed_string_values = None`; // the helper still returns Ok for arbitrary strings. assert!(setting.validate_string_value("anything").is_ok()); @@ -237,6 +243,36 @@ mod tests { } } + // ---- ocsf_schema_version validation ---- + + #[test] + fn ocsf_schema_version_accepts_supported_versions() { + let setting = setting_for_key("ocsf_schema_version") + .expect("ocsf_schema_version should be registered"); + assert_eq!(setting.kind, SettingValueKind::String); + assert_eq!( + setting.allowed_string_values, + Some(OCSF_SCHEMA_VERSION_VALUES) + ); + assert!(setting.validate_string_value("").is_ok()); + assert!(setting.validate_string_value("1.1").is_ok()); + assert!(setting.validate_string_value("1.3").is_ok()); + } + + #[test] + fn ocsf_schema_version_rejects_malformed_and_unsupported() { + let setting = setting_for_key("ocsf_schema_version") + .expect("ocsf_schema_version should be registered"); + for bad in [ + "banana", "1.6", "1.5", "1.7", "1.7.0", "1.1.0", "2.0", " 1.1", "1.1 ", "v1.1", + ] { + let err = setting + .validate_string_value(bad) + .expect_err(&format!("expected '{bad}' to be rejected")); + assert_eq!(err, OCSF_SCHEMA_VERSION_VALUES); + } + } + // ---- parse_bool_like ---- #[test] diff --git a/crates/openshell-core/src/shell.rs b/crates/openshell-core/src/shell.rs new file mode 100644 index 000000000..09610afe1 --- /dev/null +++ b/crates/openshell-core/src/shell.rs @@ -0,0 +1,107 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! Login-shell resolution for sandbox images. +//! +//! The default sandbox command and the interactive SSH session need a shell, +//! but not every base image ships the same one. Debian-based images provide +//! `bash`; minimal images such as Alpine only provide `/bin/sh` (`BusyBox` +//! `ash`). Hard-coding `/bin/bash` makes sandbox startup fail on those images +//! with an opaque `No such file or directory`. +//! +//! These helpers resolve a shell that actually exists in the current root +//! filesystem. They must run inside the sandbox (i.e. in the supervisor), not +//! on the gateway, because the answer depends on the sandbox image's contents. + +/// Preferred interactive shell when the image provides it. +pub const BASH: &str = "/bin/bash"; + +/// Preferred interactive shell on `usr`-merged images where `/bin` is not a +/// top-level directory. +pub const USR_BASH: &str = "/usr/bin/bash"; + +/// POSIX shell. Guaranteed on virtually every image, including Alpine/`BusyBox`. +/// Used as the ultimate fallback. +pub const POSIX_SH: &str = "/bin/sh"; + +/// Shell paths tried, in preference order, by [`detect_login_shell`]. +pub const SHELL_CANDIDATES: &[&str] = &[BASH, USR_BASH, POSIX_SH]; + +/// Return `true` if `path` is a regular, executable file in the current root +/// filesystem. +#[must_use] +pub fn is_executable(path: &str) -> bool { + let Ok(meta) = std::fs::metadata(path) else { + return false; + }; + if !meta.is_file() { + return false; + } + #[cfg(unix)] + { + use std::os::unix::fs::PermissionsExt; + meta.permissions().mode() & 0o111 != 0 + } + #[cfg(not(unix))] + { + true + } +} + +/// Resolve a login shell that exists in the current root filesystem. +/// +/// Tries [`SHELL_CANDIDATES`] in order and falls back to [`POSIX_SH`]. Because +/// this inspects the filesystem, call it from the supervisor (inside the +/// sandbox), never on the gateway. +/// +/// `$SHELL` is intentionally not consulted: it is image/user-controlled, the +/// result is later invoked with `-lc`, and an executable that is not a +/// compatible shell (e.g. `SHELL=/bin/false`) would pass the executable check +/// yet break command execution. Resolving only from known shell paths avoids +/// that footgun. +#[must_use] +pub fn detect_login_shell() -> String { + SHELL_CANDIDATES + .iter() + .find(|candidate| is_executable(candidate)) + .map_or_else( + || POSIX_SH.to_string(), + |candidate| (*candidate).to_string(), + ) +} + +#[cfg(test)] +mod tests { + use super::*; + + // These assume a Unix root filesystem (`/bin/sh`, POSIX permission bits) and + // are skipped on the Windows workspace lane where openshell-core also builds. + #[cfg(unix)] + #[test] + fn posix_sh_exists_on_test_host() { + // /bin/sh is present on all supported Unix CI hosts (Linux and macOS). + assert!(is_executable(POSIX_SH)); + } + + #[test] + fn missing_path_is_not_executable() { + assert!(!is_executable("/nonexistent/definitely/not/here")); + } + + #[cfg(unix)] + #[test] + fn non_file_is_not_executable() { + // A directory is not an executable shell. + assert!(!is_executable("/")); + } + + #[cfg(unix)] + #[test] + fn detect_returns_an_executable_shell() { + let shell = detect_login_shell(); + assert!( + is_executable(&shell), + "detected shell {shell} is not executable" + ); + } +} diff --git a/crates/openshell-core/src/telemetry.rs b/crates/openshell-core/src/telemetry.rs index ad1598994..9541598fe 100644 --- a/crates/openshell-core/src/telemetry.rs +++ b/crates/openshell-core/src/telemetry.rs @@ -145,6 +145,7 @@ impl PolicyDecisionOperation { pub enum SandboxTemplateSource { Default, Image, + WorkloadTemplate, Undefined, } @@ -154,56 +155,41 @@ impl SandboxTemplateSource { match self { Self::Default => "default", Self::Image => "image", + Self::WorkloadTemplate => "workload_template", Self::Undefined => "undefined", } } } #[derive(Debug, Clone, Copy, PartialEq, Eq)] -pub enum TelemetryComputeDriver { - Docker, - Kubernetes, - Podman, - Vm, - Mxc, - Unknown, -} +pub struct TelemetryComputeDriver(&'static str); impl TelemetryComputeDriver { #[must_use] pub const fn as_str(self) -> &'static str { - match self { - Self::Docker => "docker", - Self::Kubernetes => "kubernetes", - Self::Podman => "podman", - Self::Vm => "vm", - Self::Mxc => "mxc", - Self::Unknown => "unknown", - } + self.0 } + /// Classify an unregistered compute driver without exposing its configured + /// name. #[must_use] - pub fn from_raw(raw: &str) -> Self { - match raw.trim().to_ascii_lowercase().as_str() { - "docker" => Self::Docker, - "k8s" | "kubernetes" => Self::Kubernetes, - "podman" => Self::Podman, - "vm" => Self::Vm, - "mxc" => Self::Mxc, - _ => Self::Unknown, - } + pub const fn custom() -> Self { + Self("custom") } + /// Define a bounded, anonymous category at a binary composition boundary. + /// + /// The category must be a static operational label. Never construct it + /// from user input, configuration, resource names, or other runtime data. #[must_use] - pub const fn from_driver_kind(driver_kind: Option) -> Self { - match driver_kind { - Some(crate::ComputeDriverKind::Docker) => Self::Docker, - Some(crate::ComputeDriverKind::Kubernetes) => Self::Kubernetes, - Some(crate::ComputeDriverKind::Podman) => Self::Podman, - Some(crate::ComputeDriverKind::Vm) => Self::Vm, - Some(crate::ComputeDriverKind::Mxc) => Self::Mxc, - None => Self::Unknown, - } + pub const fn anonymous_category(category: &'static str) -> Self { + Self(category) + } +} + +impl Default for TelemetryComputeDriver { + fn default() -> Self { + Self::custom() } } @@ -440,6 +426,7 @@ fn telemetry_worker(rx: mpsc::Receiver) { #[cfg(feature = "telemetry")] fn publish_payload(endpoint: &str, payload: Value) -> Result<(), reqwest::Error> { + let _ = rustls::crypto::aws_lc_rs::default_provider().install_default(); Client::builder() .use_rustls_tls() .tls_built_in_root_certs(true) @@ -688,27 +675,11 @@ mod tests { } #[test] - fn compute_driver_values_are_sanitized() { + fn compute_driver_values_are_bounded_by_the_composition_boundary() { + assert_eq!(TelemetryComputeDriver::custom().as_str(), "custom"); assert_eq!( - TelemetryComputeDriver::from_raw("docker").as_str(), - "docker" - ); - assert_eq!( - TelemetryComputeDriver::from_raw("k8s").as_str(), - "kubernetes" - ); - assert_eq!( - TelemetryComputeDriver::from_raw("KUBERNETES").as_str(), - "kubernetes" - ); - assert_eq!(TelemetryComputeDriver::from_raw("vm").as_str(), "vm"); - assert_eq!( - TelemetryComputeDriver::from_raw("podman").as_str(), - "podman" - ); - assert_eq!( - TelemetryComputeDriver::from_raw("private-driver").as_str(), - "unknown" + TelemetryComputeDriver::anonymous_category("first_party").as_str(), + "first_party" ); } @@ -797,7 +768,7 @@ mod disabled_tests { 1, false, SandboxTemplateSource::Default, - TelemetryComputeDriver::Docker, + TelemetryComputeDriver::custom(), ); emit_policy_decision( PolicyDecisionOperation::Approve, diff --git a/crates/openshell-driver-db-credstore/Cargo.toml b/crates/openshell-driver-db-credstore/Cargo.toml index 805cb1c7d..9f3f5427b 100644 --- a/crates/openshell-driver-db-credstore/Cargo.toml +++ b/crates/openshell-driver-db-credstore/Cargo.toml @@ -16,7 +16,7 @@ openshell-core = { path = "../openshell-core", default-features = false } async-trait = "0.1" base64 = { workspace = true } futures = { workspace = true } -ring = { workspace = true } +aws-lc-rs = { workspace = true } serde = { workspace = true } serde_json = { workspace = true } sha2 = { workspace = true } diff --git a/crates/openshell-driver-db-credstore/src/lib.rs b/crates/openshell-driver-db-credstore/src/lib.rs index 24c21e993..c9813870b 100644 --- a/crates/openshell-driver-db-credstore/src/lib.rs +++ b/crates/openshell-driver-db-credstore/src/lib.rs @@ -17,6 +17,8 @@ use std::path::{Path, PathBuf}; use std::sync::Arc; use async_trait::async_trait; +use aws_lc_rs::aead::{AES_256_GCM, Aad, LessSafeKey, Nonce, UnboundKey}; +use aws_lc_rs::rand::{SecureRandom, SystemRandom}; use base64::{ Engine as _, engine::general_purpose::{STANDARD as BASE64, STANDARD_NO_PAD as BASE64_NO_PAD}, @@ -26,8 +28,6 @@ use openshell_core::proto::credentials::v1::{ DeleteCredentialRequest, ResolveCredentialRequest, ResolvedCredential, StoreCredentialRequest, }; use openshell_core::{Error, Result as CoreResult}; -use ring::aead::{AES_256_GCM, Aad, LessSafeKey, Nonce, UnboundKey}; -use ring::rand::{SecureRandom, SystemRandom}; use serde::{Deserialize, Serialize}; use sha2::{Digest, Sha256}; use tonic::Status; @@ -933,6 +933,46 @@ mod tests { use std::sync::{Arc, Mutex}; use tonic::Code; + #[test] + fn persisted_aes_gcm_ciphertexts_remain_compatible() { + // Fixed, non-secret ciphertexts produced by the previous crypto backend. + let cases = [ + ( + [0x11; KEY_LEN], + [0x22; NONCE_LEN], + dek_aad("fixture", "provider", "api_key"), + vec![0x33; KEY_LEN], + "JMQ0evP8rGzWDO0Pe5Xa5iyg/tFZsp94YXw52zrSVjCoSMvKkAYj5kygMADT9jIW", + ), + ( + [0x33; KEY_LEN], + [0x44; NONCE_LEN], + value_aad("fixture", "provider", "api_key"), + b"fixture-secret".to_vec(), + "Vvi85r906kbT58fgk+mUIc/KA4ar8d3Syu8Jz5h1", + ), + ]; + for (key, nonce, aad, plaintext, ciphertext) in cases { + let encrypted = EncryptedBytes { + nonce: BASE64.encode(nonce), + ciphertext: ciphertext.to_string(), + }; + assert_eq!(decrypt_bytes(&key, &aad, &encrypted).unwrap(), plaintext); + let mut sealed = plaintext; + aead_key(&key) + .unwrap() + .seal_in_place_append_tag( + Nonce::assume_unique_for_key(nonce), + Aad::from(aad.as_slice()), + &mut sealed, + ) + .unwrap(); + assert_eq!(BASE64.encode(sealed), ciphertext); + assert!(decrypt_bytes(&key, b"wrong-aad", &encrypted).is_err()); + assert!(decrypt_bytes(&[0xff; KEY_LEN], &aad, &encrypted).is_err()); + } + } + #[derive(Debug, Default)] struct MemoryObjectStore { objects: Mutex>, diff --git a/crates/openshell-driver-docker/Cargo.toml b/crates/openshell-driver-docker/Cargo.toml index 7a5fa3fe3..0b03fd388 100644 --- a/crates/openshell-driver-docker/Cargo.toml +++ b/crates/openshell-driver-docker/Cargo.toml @@ -45,7 +45,7 @@ opentelemetry = { workspace = true } opentelemetry_sdk = { workspace = true, features = ["testing"] } prost-types = { workspace = true } tar = "0.4" -temp-env = "0.3" +temp-env = { version = "0.3", features = ["async_closure"] } tempfile = "3" tracing-subscriber = { workspace = true } diff --git a/crates/openshell-driver-docker/src/lib.rs b/crates/openshell-driver-docker/src/lib.rs index 3941819c6..f3d47c0a2 100644 --- a/crates/openshell-driver-docker/src/lib.rs +++ b/crates/openshell-driver-docker/src/lib.rs @@ -21,13 +21,12 @@ use bollard::query_parameters::{ }; use bytes::Bytes; use futures::{Stream, StreamExt}; -use openshell_core::config::{ - DEFAULT_DOCKER_NETWORK_NAME, DEFAULT_SANDBOX_PIDS_LIMIT, DEFAULT_STOP_TIMEOUT_SECS, -}; +use openshell_core::config::{DEFAULT_SANDBOX_PIDS_LIMIT, DEFAULT_STOP_TIMEOUT_SECS}; use openshell_core::driver_mounts; use openshell_core::driver_utils::{ - CONDITION_EXITED, CONDITION_RUNTIME_RESTART, LABEL_MANAGED_BY, LABEL_MANAGED_BY_VALUE, - LABEL_SANDBOX_ID, LABEL_SANDBOX_NAME, LABEL_SANDBOX_NAMESPACE, LABEL_SANDBOX_WORKSPACE, + CONDITION_EXITED, CONDITION_RUNTIME_RESTART, CONDITION_WORKSPACE_VALIDATION_FAILED, + LABEL_MANAGED_BY, LABEL_MANAGED_BY_VALUE, LABEL_SANDBOX_ID, LABEL_SANDBOX_NAME, + LABEL_SANDBOX_NAMESPACE, LABEL_SANDBOX_WORKSPACE, SUPERVISOR_EXIT_WORKSPACE_VALIDATION_FAILED, SUPERVISOR_IMAGE_BINARY_PATH, extract_first_tar_entry, supervisor_image_should_refresh, temp_extract_container_name, validate_linux_elf_binary, write_cache_binary_atomic, }; @@ -40,14 +39,15 @@ use openshell_core::progress::{ format_bytes, mark_progress_active, mark_progress_complete, mark_progress_detail, }; use openshell_core::proto::compute::v1::{ - CreateSandboxRequest, CreateSandboxResponse, DeleteSandboxRequest, DeleteSandboxResponse, - DeleteWorkspaceRequest, DeleteWorkspaceResponse, DriverCondition, DriverPlatformEvent, - DriverSandbox, DriverSandboxStatus, DriverSandboxTemplate, EnsureWorkspaceRequest, - EnsureWorkspaceResponse, GatewayListenerRequirement, GetCapabilitiesRequest, - GetCapabilitiesResponse, GetGatewayListenerRequirementsRequest, + CpuResourceCapabilities, CreateSandboxRequest, CreateSandboxResponse, DeleteSandboxRequest, + DeleteSandboxResponse, DeleteWorkspaceRequest, DeleteWorkspaceResponse, DriverCondition, + DriverPlatformEvent, DriverSandbox, DriverSandboxStatus, DriverSandboxTemplate, + EnsureWorkspaceRequest, EnsureWorkspaceResponse, GatewayListenerRequirement, + GetCapabilitiesRequest, GetCapabilitiesResponse, GetGatewayListenerRequirementsRequest, GetGatewayListenerRequirementsResponse, GetSandboxRequest, GetSandboxResponse, - GpuResourceRequirements, ListSandboxesRequest, ListSandboxesResponse, StartSandboxRequest, - StartSandboxResponse, StopSandboxRequest, StopSandboxResponse, ValidateSandboxCreateRequest, + GpuResourceCapabilities, GpuResourceRequirements, ListSandboxesRequest, ListSandboxesResponse, + MemoryResourceCapabilities, ResourceCapabilities, StartSandboxRequest, StartSandboxResponse, + StopSandboxRequest, StopSandboxResponse, ValidateSandboxCreateRequest, ValidateSandboxCreateResponse, WatchSandboxesDeletedEvent, WatchSandboxesEvent, WatchSandboxesPlatformEvent, WatchSandboxesRequest, WatchSandboxesSandboxEvent, compute_driver_server::ComputeDriver, gateway_listener_requirement::Selector, @@ -56,15 +56,13 @@ use openshell_core::proto::compute::v1::{ use openshell_core::proto_struct::{ deserialize_optional_non_empty_string_list, struct_to_json_value, }; -use openshell_core::{Config, Error, Result as CoreResult}; +use openshell_core::{Error, Result as CoreResult}; use opentelemetry::trace::TraceContextExt as _; use std::collections::{HashMap, HashSet}; -use std::future::Future; use std::net::{IpAddr, Ipv4Addr, SocketAddr}; use std::path::{Path, PathBuf}; use std::pin::Pin; use std::sync::Arc; -use std::task::{Context, Poll}; use std::time::Duration; use tokio::sync::{Mutex, broadcast, mpsc}; use tokio::task::JoinHandle; @@ -212,12 +210,17 @@ struct DockerDriverRuntimeConfig { supervisor_bin: PathBuf, guest_tls: Option, daemon_version: String, - supports_gpu: bool, - allow_all_default_gpu: bool, + gpu: DockerGpuRuntimeCapabilities, sandbox_pids_limit: i64, enable_bind_mounts: bool, } +#[derive(Debug, Clone, Copy)] +struct DockerGpuRuntimeCapabilities { + cdi_supported: bool, + wsl_all_gpu_fallback_enabled: bool, +} + #[derive(Debug, Clone, PartialEq, Eq)] enum DockerGatewayRoute { Bridge { @@ -415,55 +418,15 @@ fn default_true() -> bool { type WatchStream = Pin> + Send + 'static>>; -struct TracedWatchStream { - inner: WatchStream, - span: tracing::Span, - finished: bool, -} - -impl Stream for TracedWatchStream { - type Item = Result; - - fn poll_next(mut self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll> { - let span = self.span.clone(); - let _entered = span.enter(); - let result = self.inner.as_mut().poll_next(cx); - if !self.finished { - match &result { - Poll::Ready(Some(Err(status))) => { - openshell_otel::mark_error(&self.span); - self.span - .record("rpc.grpc.status_code", status.code() as i32); - self.finished = true; - } - Poll::Ready(None) => { - self.span - .record("rpc.grpc.status_code", tonic::Code::Ok as i32); - self.finished = true; - } - Poll::Pending | Poll::Ready(Some(Ok(_))) => {} - } - } - result - } -} - -impl Drop for TracedWatchStream { - fn drop(&mut self) { - if !self.finished { - openshell_otel::mark_error(&self.span); - self.span - .record("rpc.grpc.status_code", tonic::Code::Cancelled as i32); - } - } -} +#[cfg(test)] +type TracedWatchStream = openshell_otel::TracedGrpcStream; /// Compute-driver service wrapper that preserves the standalone RPC trace /// boundary while Docker runs in the gateway process. #[derive(Clone)] pub struct ComputeDriverService { driver: DockerComputeDriver, - trace_in_process_rpc: bool, + rpc_tracer: openshell_otel::InProcessRpcTracer, } impl ComputeDriverService { @@ -471,7 +434,7 @@ impl ComputeDriverService { pub fn new(driver: DockerComputeDriver) -> Self { Self { driver, - trace_in_process_rpc: false, + rpc_tracer: openshell_otel::InProcessRpcTracer::disabled(), } } @@ -479,61 +442,50 @@ impl ComputeDriverService { pub fn new_in_process(driver: DockerComputeDriver) -> Self { Self { driver, - trace_in_process_rpc: true, + rpc_tracer: openshell_otel::InProcessRpcTracer::enabled(), } } +} - fn in_process_rpc_span( - &self, - operation: &'static str, - method: &'static str, - ) -> Option { - self.trace_in_process_rpc.then(|| { - tracing::info_span!( - target: "openshell_driver_docker::otel_tracing", - "driver_rpc", - otel.name = operation, - otel.kind = "server", - otel.status_code = tracing::field::Empty, - rpc.system = "grpc", - rpc.service = "openshell.compute.v1.ComputeDriver", - rpc.method = method, - rpc.grpc.status_code = tracing::field::Empty, - ) - }) +/// Return the first responsive local Docker API socket. +#[must_use] +pub fn detect_socket() -> Option { + let mut candidates = Vec::new(); + if let Ok(host) = std::env::var("DOCKER_HOST") + && let Some(path) = host.trim().strip_prefix("unix://") + && !path.is_empty() + { + candidates.push(PathBuf::from(path)); } + candidates.push(PathBuf::from("/var/run/docker.sock")); + if let Some(home) = std::env::var_os("HOME") { + candidates.push(PathBuf::from(home).join(".docker/run/docker.sock")); + } + if let Some(runtime_dir) = std::env::var_os("XDG_RUNTIME_DIR") { + candidates.push(PathBuf::from(runtime_dir).join("docker.sock")); + } + openshell_core::local_api_socket::first_responsive_socket(&candidates, |response| { + openshell_core::local_api_socket::http_response_is_success(response) + && openshell_core::local_api_socket::contains_ascii(response, b"Api-Version:") + && !openshell_core::local_api_socket::contains_ascii(response, b"Libpod-Api-Version:") + }) +} - async fn trace_rpc( - &self, - operation: &'static str, - method: &'static str, - future: impl Future>, - ) -> Result { - use tracing::Instrument as _; - - let Some(span) = self.in_process_rpc_span(operation, method) else { - return future.await; - }; - let result = future.instrument(span.clone()).await; - match &result { - Ok(_) => { - span.record("rpc.grpc.status_code", tonic::Code::Ok as i32); - } - Err(status) => { - openshell_otel::mark_error(&span); - span.record("rpc.grpc.status_code", status.code() as i32); - } - } - result - } +#[must_use] +pub fn is_available() -> bool { + detect_socket().is_some() } impl DockerComputeDriver { - pub async fn new(config: &Config, docker_config: &DockerComputeConfig) -> CoreResult { + pub async fn new( + gateway_bind_address: SocketAddr, + gateway_log_level: &str, + docker_config: &DockerComputeConfig, + ) -> CoreResult { let socket_path = docker_config .socket_path .clone() - .or_else(openshell_core::config::detect_docker_socket) + .or_else(detect_socket) .unwrap_or_else(|| PathBuf::from("/var/run/docker.sock")); let socket_path_str = socket_path.to_str().ok_or_else(|| { Error::config(format!( @@ -552,14 +504,18 @@ impl DockerComputeDriver { let info = docker.info().await.map_err(|err| { Error::execution(format!("failed to query Docker daemon info: {err}")) })?; - let supports_gpu = info + let cdi_supported = info .cdi_spec_dirs .as_ref() .is_some_and(|dirs| !dirs.is_empty()); let cdi_gpu_inventory = docker_cdi_gpu_inventory(&info); - let allow_all_default_gpu = docker_info_reports_wsl2(&info); + let wsl_all_gpu_fallback_enabled = docker_info_reports_wsl2(&info); + let gpu = DockerGpuRuntimeCapabilities { + cdi_supported, + wsl_all_gpu_fallback_enabled, + }; validate_sandbox_pids_limit(docker_config.sandbox_pids_limit)?; - let gateway_port = config.bind_address.port(); + let gateway_port = gateway_bind_address.port(); if gateway_port == 0 { return Err(Error::config( "docker compute driver requires a fixed non-zero gateway bind port", @@ -571,7 +527,7 @@ impl DockerComputeDriver { let gateway_route = docker_gateway_route(&info, bridge_gateway_ip, gateway_port, host_gateway_ip); let gateway_callback_bind_address = - docker_gateway_callback_bind_address(&gateway_route, config.bind_address); + docker_gateway_callback_bind_address(&gateway_route, gateway_bind_address); let mut docker_config = docker_config.clone(); if docker_config.grpc_endpoint.trim().is_empty() { let scheme = if docker_guest_tls_configured(&docker_config) { @@ -603,12 +559,11 @@ impl DockerComputeDriver { gateway_callback_bind_address, ssh_socket_path: docker_config.ssh_socket_path.clone(), stop_timeout_secs: DEFAULT_STOP_TIMEOUT_SECS, - log_level: config.log_level.clone(), + log_level: gateway_log_level.to_string(), supervisor_bin, guest_tls, daemon_version: version.version.unwrap_or_else(|| "unknown".to_string()), - supports_gpu, - allow_all_default_gpu, + gpu, sandbox_pids_limit: docker_config.sandbox_pids_limit, enable_bind_mounts: docker_config.enable_bind_mounts, }, @@ -616,7 +571,7 @@ impl DockerComputeDriver { pending: Arc::new(Mutex::new(HashMap::new())), gpu_selector: Arc::new(CdiGpuDefaultSelector::new( cdi_gpu_inventory, - allow_all_default_gpu, + gpu.wsl_all_gpu_fallback_enabled, )), lifecycle_event_fences: DockerLifecycleEventFences::default(), }; @@ -636,6 +591,21 @@ impl DockerComputeDriver { default_image: self.config.default_image.clone(), gateway_manages_lifecycle: true, supports_sandbox_authentication: false, + driver_reports_runtime_readiness: false, + resource_capabilities: Some(ResourceCapabilities { + cpu: Some(CpuResourceCapabilities { + limit_supported: true, + }), + memory: Some(MemoryResourceCapabilities { + limit_supported: true, + }), + gpu: Some(GpuResourceCapabilities { + default_selection_supported: self.config.gpu.cdi_supported, + count_selection_supported: self.config.gpu.cdi_supported, + }), + }), + rootfs_tar_staging_dir: String::new(), + rootfs_tar_max_bytes: 0, } } @@ -667,7 +637,7 @@ impl DockerComputeDriver { DockerSandboxDriverConfig::from_template(template).map_err(Status::invalid_argument)?; validate_docker_driver_mounts(&driver_config.mounts, config.enable_bind_mounts)?; let gpu_requirements = driver_gpu_requirements(spec.resource_requirements.as_ref()); - Self::validate_gpu_request(gpu_requirements, config.supports_gpu, &driver_config)?; + Self::validate_gpu_request(gpu_requirements, config.gpu.cdi_supported, &driver_config)?; Ok(ValidatedDockerSandbox { template, driver_config, @@ -775,7 +745,7 @@ impl DockerComputeDriver { .map_err(|err| internal_status("query Docker daemon info", err))?; self.gpu_selector.refresh( docker_cdi_gpu_inventory(&info), - self.config.allow_all_default_gpu, + self.config.gpu.wsl_all_gpu_fallback_enabled, ); Ok(()) } @@ -825,7 +795,7 @@ impl DockerComputeDriver { return Ok(Some(sandbox)); } - Ok(self.pending_snapshot(sandbox_id, sandbox_name).await) + self.pending_snapshot(sandbox_id, sandbox_name).await } async fn current_snapshots(&self) -> Result, Status> { @@ -939,16 +909,6 @@ impl DockerComputeDriver { } } - #[tracing::instrument( - name = "docker.provision_sandbox", - skip(self, sandbox), - fields( - otel.name = "docker.provision_sandbox", - otel.status_code = tracing::field::Empty, - sandbox.id = %sandbox.id, - sandbox.name = %sandbox.name, - ) - )] async fn provision_sandbox_inner( &self, sandbox: &DriverSandbox, @@ -1108,7 +1068,9 @@ impl DockerComputeDriver { sandbox_id: &str, sandbox_name: &str, ) -> Result { - let pending = self.remove_pending_sandbox(sandbox_id, sandbox_name).await; + let pending = self + .remove_pending_sandbox(sandbox_id, sandbox_name) + .await?; if let Some(record) = pending.as_ref() && let Some(task) = record.task.as_ref() { @@ -1142,6 +1104,11 @@ impl DockerComputeDriver { } } } + // Container gone and no in-memory record survived (gateway + // restarted after an out-of-band `docker rm`). DeleteSandbox is + // the only thing that ever reclaims the token file, so reclaim it + // here too. + cleanup_sandbox_token_file_for_delete(sandbox_id, None, &self.config); return Ok(false); }; let Some(target) = summary_container_target(&container) else { @@ -1173,7 +1140,10 @@ impl DockerComputeDriver { .find_managed_container_summary(sandbox_id, sandbox_name) .await? else { - if let Some(record) = self.remove_pending_sandbox(sandbox_id, sandbox_name).await { + if let Some(record) = self + .remove_pending_sandbox(sandbox_id, sandbox_name) + .await? + { if let Some(task) = record.task { task.abort(); } @@ -1290,10 +1260,11 @@ impl DockerComputeDriver { async fn reserve_pending_sandbox(&self, sandbox: &DriverSandbox) -> Result<(), Status> { let mut pending = self.pending.lock().await; - if pending - .values() - .any(|record| record.sandbox.id == sandbox.id || record.sandbox.name == sandbox.name) - { + if pending.values().any(|record| { + record.sandbox.id == sandbox.id + || (record.sandbox.name == sandbox.name + && record.sandbox.workspace == sandbox.workspace) + }) { return Err(Status::already_exists("sandbox already exists")); } @@ -1316,12 +1287,12 @@ impl DockerComputeDriver { &self, sandbox_id: &str, sandbox_name: &str, - ) -> Option { + ) -> Result, Status> { let pending = self.pending.lock().await; - pending - .values() - .find(|record| pending_sandbox_matches(&record.sandbox, sandbox_id, sandbox_name)) - .map(|record| record.sandbox.clone()) + let Some(id) = resolve_pending_id(&pending, sandbox_id, sandbox_name)? else { + return Ok(None); + }; + Ok(pending.get(&id).map(|record| record.sandbox.clone())) } async fn pending_snapshot_map(&self) -> HashMap { @@ -1341,12 +1312,12 @@ impl DockerComputeDriver { &self, sandbox_id: &str, sandbox_name: &str, - ) -> Option { + ) -> Result, Status> { let mut pending = self.pending.lock().await; - let id = pending.iter().find_map(|(id, record)| { - pending_sandbox_matches(&record.sandbox, sandbox_id, sandbox_name).then(|| id.clone()) - })?; - pending.remove(&id) + let Some(id) = resolve_pending_id(&pending, sandbox_id, sandbox_name)? else { + return Ok(None); + }; + Ok(pending.remove(&id)) } async fn fail_pending_sandbox( @@ -1601,21 +1572,14 @@ impl DockerComputeDriver { .map_err(|err| internal_status("find Docker sandbox container", err))?; Ok(containers.into_iter().find(|summary| { - let Some(labels) = summary.labels.as_ref() else { - return false; - }; - let namespace_matches = labels - .get(LABEL_SANDBOX_NAMESPACE) - .is_some_and(|value| value == &self.config.sandbox_namespace); - let id_matches = sandbox_id.is_empty() - || labels - .get(LABEL_SANDBOX_ID) - .is_some_and(|value| value == sandbox_id); - let name_matches = sandbox_name.is_empty() - || labels - .get(LABEL_SANDBOX_NAME) - .is_some_and(|value| value == sandbox_name); - namespace_matches && id_matches && name_matches + summary.labels.as_ref().is_some_and(|labels| { + managed_container_identity_matches( + labels, + &self.config.sandbox_namespace, + sandbox_id, + sandbox_name, + ) + }) })) } @@ -1737,6 +1701,9 @@ impl DockerComputeDriver { } } +// Standalone and in-process servers both use this wrapper. Delegating to the +// driver's canonical tonic implementation keeps request validation and Docker +// operation spans identical across both deployment modes. #[tonic::async_trait] impl ComputeDriver for ComputeDriverService { type WatchSandboxesStream = WatchStream; @@ -1746,169 +1713,159 @@ impl ComputeDriver for ComputeDriverService { request: Request, ) -> Result, Status> { - self.trace_rpc( - "driver.authenticate_sandbox", - "authenticate_sandbox", - ComputeDriver::authenticate_sandbox(&self.driver, request), - ) - .await + self.rpc_tracer + .trace( + openshell_otel::rpc::AUTHENTICATE_SANDBOX, + ComputeDriver::authenticate_sandbox(&self.driver, request), + ) + .await } async fn get_capabilities( &self, request: Request, ) -> Result, Status> { - self.trace_rpc( - "driver.get_capabilities", - "get_capabilities", - ComputeDriver::get_capabilities(&self.driver, request), - ) - .await + self.rpc_tracer + .trace( + openshell_otel::rpc::GET_CAPABILITIES, + ComputeDriver::get_capabilities(&self.driver, request), + ) + .await } async fn get_gateway_listener_requirements( &self, request: Request, ) -> Result, Status> { - self.trace_rpc( - "driver.get_gateway_listener_requirements", - "get_gateway_listener_requirements", - ComputeDriver::get_gateway_listener_requirements(&self.driver, request), - ) - .await + self.rpc_tracer + .trace( + openshell_otel::rpc::GET_GATEWAY_LISTENER_REQUIREMENTS, + ComputeDriver::get_gateway_listener_requirements(&self.driver, request), + ) + .await } async fn validate_sandbox_create( &self, request: Request, ) -> Result, Status> { - self.trace_rpc( - "driver.validate_sandbox_create", - "validate_sandbox_create", - ComputeDriver::validate_sandbox_create(&self.driver, request), - ) - .await + self.rpc_tracer + .trace( + openshell_otel::rpc::VALIDATE_SANDBOX_CREATE, + ComputeDriver::validate_sandbox_create(&self.driver, request), + ) + .await } async fn get_sandbox( &self, request: Request, ) -> Result, Status> { - self.trace_rpc( - "driver.get_sandbox", - "get_sandbox", - ComputeDriver::get_sandbox(&self.driver, request), - ) - .await + self.rpc_tracer + .trace( + openshell_otel::rpc::GET_SANDBOX, + ComputeDriver::get_sandbox(&self.driver, request), + ) + .await } async fn list_sandboxes( &self, request: Request, ) -> Result, Status> { - self.trace_rpc( - "driver.list_sandboxes", - "list_sandboxes", - ComputeDriver::list_sandboxes(&self.driver, request), - ) - .await + self.rpc_tracer + .trace( + openshell_otel::rpc::LIST_SANDBOXES, + ComputeDriver::list_sandboxes(&self.driver, request), + ) + .await } async fn create_sandbox( &self, request: Request, ) -> Result, Status> { - self.trace_rpc( - "driver.create_sandbox", - "create_sandbox", - ComputeDriver::create_sandbox(&self.driver, request), - ) - .await + self.rpc_tracer + .trace( + openshell_otel::rpc::CREATE_SANDBOX, + ComputeDriver::create_sandbox(&self.driver, request), + ) + .await } async fn stop_sandbox( &self, request: Request, ) -> Result, Status> { - self.trace_rpc( - "driver.stop_sandbox", - "stop_sandbox", - ComputeDriver::stop_sandbox(&self.driver, request), - ) - .await + self.rpc_tracer + .trace( + openshell_otel::rpc::STOP_SANDBOX, + ComputeDriver::stop_sandbox(&self.driver, request), + ) + .await } async fn start_sandbox( &self, request: Request, ) -> Result, Status> { - self.trace_rpc( - "driver.start_sandbox", - "start_sandbox", - ComputeDriver::start_sandbox(&self.driver, request), - ) - .await + self.rpc_tracer + .trace( + openshell_otel::rpc::START_SANDBOX, + ComputeDriver::start_sandbox(&self.driver, request), + ) + .await } async fn delete_sandbox( &self, request: Request, ) -> Result, Status> { - self.trace_rpc( - "driver.delete_sandbox", - "delete_sandbox", - ComputeDriver::delete_sandbox(&self.driver, request), - ) - .await + self.rpc_tracer + .trace( + openshell_otel::rpc::DELETE_SANDBOX, + ComputeDriver::delete_sandbox(&self.driver, request), + ) + .await } async fn watch_sandboxes( &self, request: Request, ) -> Result, Status> { - use tracing::Instrument as _; - - let create_stream = ComputeDriver::watch_sandboxes(&self.driver, request); - let Some(span) = self.in_process_rpc_span("driver.watch_sandboxes", "watch_sandboxes") - else { - return create_stream.await; + let create_stream = async { + ComputeDriver::watch_sandboxes(&self.driver, request) + .await + .map(Response::into_inner) }; - match create_stream.instrument(span.clone()).await { - Ok(response) => Ok(Response::new(Box::pin(TracedWatchStream { - inner: response.into_inner(), - span, - finished: false, - }))), - Err(status) => { - openshell_otel::mark_error(&span); - span.record("rpc.grpc.status_code", status.code() as i32); - Err(status) - } - } + self.rpc_tracer + .trace_stream(openshell_otel::rpc::WATCH_SANDBOXES, create_stream) + .await + .map(Response::new) } async fn ensure_workspace( &self, request: Request, ) -> Result, Status> { - self.trace_rpc( - "driver.ensure_workspace", - "ensure_workspace", - ComputeDriver::ensure_workspace(&self.driver, request), - ) - .await + self.rpc_tracer + .trace( + openshell_otel::rpc::ENSURE_WORKSPACE, + ComputeDriver::ensure_workspace(&self.driver, request), + ) + .await } async fn delete_workspace( &self, request: Request, ) -> Result, Status> { - self.trace_rpc( - "driver.delete_workspace", - "delete_workspace", - ComputeDriver::delete_workspace(&self.driver, request), - ) - .await + self.rpc_tracer + .trace( + openshell_otel::rpc::DELETE_WORKSPACE, + ComputeDriver::delete_workspace(&self.driver, request), + ) + .await } } @@ -2210,9 +2167,73 @@ fn pending_sandbox_snapshot( } } -fn pending_sandbox_matches(sandbox: &DriverSandbox, sandbox_id: &str, sandbox_name: &str) -> bool { - (!sandbox_id.is_empty() && sandbox.id == sandbox_id) - || (!sandbox_name.is_empty() && sandbox.name == sandbox_name) +/// Decides whether a managed container satisfies a lifecycle request. +/// +/// `sandbox_id` is authoritative, matching [`resolve_pending_id`]. Requiring +/// the name to agree as well would discard a correct id match whenever the +/// caller pairs it with a stale name, leaving the container and its token file +/// behind while the driver reports the sandbox as absent. +/// +/// A request with no identifier matches nothing. `require_sandbox_identifier` +/// rejects that upstream, but the label filters degenerate to "every managed +/// container in the namespace", so this does not rely on the caller to guard it. +fn managed_container_identity_matches( + labels: &HashMap, + namespace: &str, + sandbox_id: &str, + sandbox_name: &str, +) -> bool { + if labels + .get(LABEL_SANDBOX_NAMESPACE) + .is_none_or(|value| value != namespace) + { + return false; + } + if !sandbox_id.is_empty() { + return labels + .get(LABEL_SANDBOX_ID) + .is_some_and(|value| value == sandbox_id); + } + !sandbox_name.is_empty() + && labels + .get(LABEL_SANDBOX_NAME) + .is_some_and(|value| value == sandbox_name) +} + +/// Resolves a lifecycle request to at most one pending sandbox id. +/// +/// `sandbox_id` is authoritative: when the caller supplies one, the name is +/// never consulted as an alternative. The name fallback rejects ambiguity +/// instead of letting `HashMap` iteration order pick a match, because sandbox +/// names are unique per workspace and the driver request carries no workspace. +fn resolve_pending_id( + pending: &HashMap, + sandbox_id: &str, + sandbox_name: &str, +) -> Result, Status> { + if !sandbox_id.is_empty() { + return Ok(pending + .contains_key(sandbox_id) + .then(|| sandbox_id.to_string())); + } + if sandbox_name.is_empty() { + return Ok(None); + } + + let mut matches = pending + .iter() + .filter(|(_, record)| record.sandbox.name == sandbox_name) + .map(|(id, _)| id.clone()); + + let Some(id) = matches.next() else { + return Ok(None); + }; + if matches.next().is_some() { + return Err(Status::failed_precondition( + "sandbox_name matches multiple pending sandboxes; specify sandbox_id", + )); + } + Ok(Some(id)) } fn provisioning_condition() -> DriverCondition { @@ -3544,8 +3565,10 @@ fn driver_status_from_summary( /// Refine an exited Docker sandbox's `Ready` condition from inspected state. /// -/// A signal kill (exit 137/143 = SIGKILL/SIGTERM, not OOM) is the signature of -/// a machine/daemon restart terminating a running container. Reclassify it from +/// A workspace-validation exit is reported distinctly so users can repair the +/// OCI working directory rather than diagnose a generic crash. A signal kill +/// (exit 137/143 = SIGKILL/SIGTERM, not OOM) is the signature of a +/// machine/daemon restart terminating a running container. Reclassify it from /// the generic terminal `ContainerExited` to the recoverable /// `ContainerRuntimeRestart` so gateway startup can revive it. OOM kills and /// ordinary application exits stay `ContainerExited` and terminal. @@ -3553,7 +3576,7 @@ fn apply_docker_exit_classification(sandbox: &mut DriverSandbox, state: &Contain if state.oom_killed == Some(true) { return; } - let Some(code) = state.exit_code.filter(|&code| matches!(code, 137 | 143)) else { + let Some(code) = state.exit_code else { return; }; let Some(condition) = sandbox @@ -3566,8 +3589,13 @@ fn apply_docker_exit_classification(sandbox: &mut DriverSandbox, state: &Contain if condition.reason != CONDITION_EXITED { return; } - condition.reason = CONDITION_RUNTIME_RESTART.to_string(); - condition.message = format!("Container terminated by signal (exit code {code})"); + if code == i64::from(SUPERVISOR_EXIT_WORKSPACE_VALIDATION_FAILED) { + condition.reason = CONDITION_WORKSPACE_VALIDATION_FAILED.to_string(); + condition.message = "OCI WorkingDir is not usable by the sandbox identity".to_string(); + } else if matches!(code, 137 | 143) { + condition.reason = CONDITION_RUNTIME_RESTART.to_string(); + condition.message = format!("Container terminated by signal (exit code {code})"); + } } fn container_ready_condition( @@ -4140,3 +4168,4 @@ fn internal_status(operation: &str, err: BollardError) -> Status { #[cfg(test)] mod tests; +pub const DEFAULT_DOCKER_NETWORK_NAME: &str = "openshell-docker"; diff --git a/crates/openshell-driver-docker/src/main.rs b/crates/openshell-driver-docker/src/main.rs index 2c6dbfd4c..a5ca29887 100644 --- a/crates/openshell-driver-docker/src/main.rs +++ b/crates/openshell-driver-docker/src/main.rs @@ -6,13 +6,10 @@ use std::path::PathBuf; use clap::Parser; use miette::{IntoDiagnostic, Result}; +use openshell_core::VERSION; use openshell_core::proto::compute::v1::compute_driver_server::ComputeDriverServer; -use openshell_core::{Config, VERSION}; -use openshell_driver_docker::otel_tracing::compute_driver_rpc_layer; use openshell_driver_docker::{ComputeDriverService, DockerComputeConfig, DockerComputeDriver}; use tracing::info; -use tracing_subscriber::EnvFilter; -use tracing_subscriber::prelude::*; #[derive(Debug, Parser)] #[command(name = "openshell-driver-docker", version = VERSION)] @@ -46,29 +43,19 @@ struct Args { #[tokio::main] async fn main() -> Result<()> { let args = Args::parse(); - let (tracer_provider, setup_error) = openshell_driver_docker::otel_tracing::provider_for( - args.otlp_endpoint.as_deref(), - args.gateway_name.as_deref(), + let _tracing = openshell_otel::install_driver_tracing( + openshell_driver_docker::otel_tracing::TRACING, + openshell_otel::DriverTracingConfig { + endpoint: args.otlp_endpoint.as_deref(), + gateway_name: args.gateway_name.as_deref(), + service_version: VERSION, + log_level: &args.log_level, + }, ); - tracing_subscriber::registry() - .with(EnvFilter::try_from_default_env().unwrap_or_else(|_| EnvFilter::new(&args.log_level))) - .with(tracing_subscriber::fmt::layer()) - .with( - tracer_provider - .as_ref() - .map(openshell_driver_docker::otel_tracing::layer), - ) - .init(); - if let Some(error) = setup_error { - tracing::error!(%error, "OTLP exporting could not be started"); - } else if let Some(endpoint) = &args.otlp_endpoint { - info!(endpoint, "OTLP exporting enabled"); - } let config_source = std::fs::read_to_string(&args.config).into_diagnostic()?; let docker_config: DockerComputeConfig = toml::from_str(&config_source).into_diagnostic()?; - let gateway_config = Config::new(None).with_bind_address(args.gateway_bind); - let driver = DockerComputeDriver::new(&gateway_config, &docker_config) + let driver = DockerComputeDriver::new(args.gateway_bind, &args.log_level, &docker_config) .await .into_diagnostic()?; @@ -77,21 +64,15 @@ async fn main() -> Result<()> { let _cleanup = openshell_core::external_driver_socket::SocketCleanup::new(args.bind_socket.clone()); info!(socket = %args.bind_socket.display(), "Starting Docker compute driver"); - let result = tonic::transport::Server::builder() - .layer(compute_driver_rpc_layer()) + tonic::transport::Server::builder() + .layer(openshell_otel::compute_driver_rpc_layer()) .add_service(ComputeDriverServer::new(ComputeDriverService::new(driver))) .serve_with_incoming_shutdown( openshell_core::external_driver_socket::SameUidUnixIncoming::new(listener), shutdown_signal(), ) .await - .into_diagnostic(); - if let Some(provider) = &tracer_provider - && let Err(error) = provider.shutdown() - { - tracing::warn!(%error, "OTLP tracer provider shutdown failed"); - } - result + .into_diagnostic() } async fn shutdown_signal() { diff --git a/crates/openshell-driver-docker/src/otel_tracing.rs b/crates/openshell-driver-docker/src/otel_tracing.rs index 7623279c1..03090c2ed 100644 --- a/crates/openshell-driver-docker/src/otel_tracing.rs +++ b/crates/openshell-driver-docker/src/otel_tracing.rs @@ -1,201 +1,19 @@ // SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 -//! OpenTelemetry trace exporting for the Docker compute driver. +//! OpenTelemetry tracing identity for the Docker compute driver. -use http::Request; -use openshell_otel::{ - HeaderMapExtractor, OtlpTraceConfig, RecordGrpcFailure, RecordGrpcStatus, SdkTracerProvider, - ServiceName, SetupError, -}; -use opentelemetry::propagation::TextMapPropagator as _; -use opentelemetry::trace::TraceContextExt as _; -use opentelemetry_sdk::propagation::TraceContextPropagator; -use tower_http::trace::{GrpcMakeClassifier, MakeSpan, TraceLayer}; -use tracing::{Span, Subscriber}; -use tracing_opentelemetry::OpenTelemetrySpanExt as _; -use tracing_subscriber::registry::LookupSpan; - -const SERVICE_NAME: &str = "openshell-driver-docker"; -const INSTRUMENTATION_SCOPE: &str = "openshell-driver-docker"; -const COMPUTE_DRIVER_SERVICE: &str = "openshell.compute.v1.ComputeDriver"; -pub const IN_PROCESS_TARGET_PREFIX: &str = "openshell_driver_docker"; - -/// Trace inbound standalone compute-driver RPCs and continue gateway context. -pub fn compute_driver_rpc_layer() -> TraceLayer< - GrpcMakeClassifier, - ComputeDriverRpcSpan, - (), - RecordGrpcStatus, - (), - RecordGrpcStatus, - RecordGrpcFailure, -> { - TraceLayer::new_for_grpc() - .make_span_with(ComputeDriverRpcSpan) - .on_request(()) - .on_response(RecordGrpcStatus) - .on_body_chunk(()) - .on_eos(RecordGrpcStatus) - .on_failure(RecordGrpcFailure) -} - -#[derive(Debug, Clone, Copy)] -pub struct ComputeDriverRpcSpan; - -impl MakeSpan for ComputeDriverRpcSpan { - fn make_span(&mut self, request: &Request) -> Span { - let (operation, method) = compute_driver_rpc_operation(request.uri().path()); - let span = tracing::info_span!( - "driver_rpc", - otel.name = operation, - otel.kind = "server", - otel.status_code = tracing::field::Empty, - rpc.system = "grpc", - rpc.service = COMPUTE_DRIVER_SERVICE, - rpc.method = method, - rpc.grpc.status_code = tracing::field::Empty, - ); - let parent = TraceContextPropagator::new().extract_with_context( - &opentelemetry::Context::new(), - &HeaderMapExtractor::new(request.headers()), - ); - if parent.span().span_context().is_valid() { - let _ = span.set_parent(parent); - } - span - } -} - -pub(crate) fn compute_driver_rpc_operation(path: &str) -> (&'static str, &'static str) { - match path.rsplit('/').next() { - Some("GetCapabilities") => ("driver.get_capabilities", "get_capabilities"), - Some("GetGatewayListenerRequirements") => ( - "driver.get_gateway_listener_requirements", - "get_gateway_listener_requirements", - ), - Some("ValidateSandboxCreate") => { - ("driver.validate_sandbox_create", "validate_sandbox_create") - } - Some("CreateSandbox") => ("driver.create_sandbox", "create_sandbox"), - Some("GetSandbox") => ("driver.get_sandbox", "get_sandbox"), - Some("ListSandboxes") => ("driver.list_sandboxes", "list_sandboxes"), - Some("StopSandbox") => ("driver.stop_sandbox", "stop_sandbox"), - Some("StartSandbox") => ("driver.start_sandbox", "start_sandbox"), - Some("DeleteSandbox") => ("driver.delete_sandbox", "delete_sandbox"), - Some("WatchSandboxes") => ("driver.watch_sandboxes", "watch_sandboxes"), - Some("EnsureWorkspace") => ("driver.ensure_workspace", "ensure_workspace"), - Some("DeleteWorkspace") => ("driver.delete_workspace", "delete_workspace"), - _ => ("driver.unknown", "unknown"), - } -} - -/// Build a tracer provider for the configured OTLP/gRPC endpoint and gateway. -#[must_use] -pub fn provider_for( - endpoint: Option<&str>, - gateway_name: Option<&str>, -) -> (Option, Option) { - openshell_otel::provider_for(endpoint.map(|endpoint| { - OtlpTraceConfig { - endpoint, - service_name: ServiceName::Fixed(SERVICE_NAME), - service_version: Some(openshell_core::VERSION), - resource_attributes: gateway_name - .map(str::trim) - .filter(|name| !name.is_empty()) - .map(|name| { - vec![opentelemetry::KeyValue::new( - "openshell.gateway.name", - name.to_string(), - )] - }) - .unwrap_or_default(), - } - })) -} - -pub fn layer(provider: &SdkTracerProvider) -> openshell_otel::OtlpLayer -where - S: Subscriber + for<'span> LookupSpan<'span>, -{ - openshell_otel::layer(provider, INSTRUMENTATION_SCOPE) -} - -pub fn in_process_layer(provider: &SdkTracerProvider) -> openshell_otel::TargetOtlpLayer -where - S: Subscriber + for<'span> LookupSpan<'span>, -{ - openshell_otel::layer_for_target_prefix( - provider, - INSTRUMENTATION_SCOPE, - IN_PROCESS_TARGET_PREFIX, - ) -} - -#[cfg(test)] -pub(crate) async fn test_lock() -> tokio::sync::MutexGuard<'static, ()> { - static LOCK: tokio::sync::Mutex<()> = tokio::sync::Mutex::const_new(()); - static INITIALIZED: std::sync::LazyLock<()> = std::sync::LazyLock::new(|| { - tracing::subscriber::set_global_default(tracing_subscriber::registry()) - .expect("test tracing subscriber installs once"); - }); - - let guard = LOCK.lock().await; - std::sync::LazyLock::force(&INITIALIZED); - guard -} +pub const TRACING: openshell_otel::ComputeDriverTracing = openshell_otel::compute_driver_tracing!(); #[cfg(test)] mod tests { - use openshell_otel_test_support::OtlpTestServer; - use tracing_subscriber::layer::SubscriberExt as _; - - #[test] - fn compute_driver_rpc_names_are_explicitly_mapped_and_schema_bounded() { - assert_eq!( - super::compute_driver_rpc_operation( - "/openshell.compute.v1.ComputeDriver/CreateSandbox" - ), - ("driver.create_sandbox", "create_sandbox") - ); - assert_eq!( - super::compute_driver_rpc_operation("/openshell.compute.v1.ComputeDriver/FutureMethod"), - ("driver.unknown", "unknown") - ); - } - #[tokio::test(flavor = "multi_thread", worker_threads = 2)] - async fn tracing_docker_driver_spans_reach_otlp_collector_with_resource_identity() { - let _tracing_lock = super::test_lock().await; - let collector = OtlpTestServer::start().await; - - let (provider, error) = super::provider_for(Some(collector.endpoint()), Some("docker-dev")); - assert!(error.is_none()); - let provider = provider.expect("provider"); - let subscriber = tracing_subscriber::registry().with(super::layer(&provider)); - tracing::subscriber::with_default(subscriber, || { - let span = tracing::info_span!("docker.schedule_sandbox", sandbox.id = "sb-otlp"); - drop(span.enter()); - drop(span); - }); - provider.force_flush().unwrap(); - collector.wait_for_export().await; - provider.shutdown().unwrap(); - let received = collector.shutdown().await; - - assert!( - received - .spans - .iter() - .any(|span| span.name == "docker.schedule_sandbox") - ); - assert_eq!(received.gateway_names, ["docker-dev"]); - assert!( - received - .service_names - .iter() - .any(|name| name == "openshell-driver-docker") - ); + async fn driver_spans_reach_otlp_collector_with_resource_identity() { + openshell_otel_test_support::assert_compute_driver_tracing( + super::TRACING, + openshell_core::VERSION, + "docker.schedule_sandbox", + ) + .await; } } diff --git a/crates/openshell-driver-docker/src/tests.rs b/crates/openshell-driver-docker/src/tests.rs index fd81c9b63..c6321cc63 100644 --- a/crates/openshell-driver-docker/src/tests.rs +++ b/crates/openshell-driver-docker/src/tests.rs @@ -4,8 +4,9 @@ use super::*; use openshell_core::config::DEFAULT_SERVER_PORT; use openshell_core::driver_utils::{ - LABEL_MANAGED_BY, LABEL_MANAGED_BY_VALUE, LABEL_SANDBOX_ID, LABEL_SANDBOX_NAME, - LABEL_SANDBOX_NAMESPACE, supervisor_cache_path_with_base, + CONDITION_WORKSPACE_VALIDATION_FAILED, LABEL_MANAGED_BY, LABEL_MANAGED_BY_VALUE, + LABEL_SANDBOX_ID, LABEL_SANDBOX_NAME, LABEL_SANDBOX_NAMESPACE, + SUPERVISOR_EXIT_WORKSPACE_VALIDATION_FAILED, supervisor_cache_path_with_base, }; use openshell_core::progress::{ PROGRESS_ACTIVE_DETAIL_KEY, PROGRESS_ACTIVE_STEP_KEY, PROGRESS_COMPLETE_LABEL_KEY, @@ -37,12 +38,13 @@ fn test_sandbox() -> DriverSandbox { log_level: "debug".to_string(), environment: HashMap::from([("SPEC_ENV".to_string(), "spec".to_string())]), template: Some(DriverSandboxTemplate { - image: "ghcr.io/nvidia/openshell/sandbox:dev".to_string(), + image: "ghcr.io/nvidia/openshell-community/sandboxes/base:latest".to_string(), agent_socket_path: String::new(), labels: HashMap::new(), environment: HashMap::from([("TEMPLATE_ENV".to_string(), "template".to_string())]), ..Default::default() }), + policy: None, resource_requirements: None, sandbox_token: String::new(), command: Vec::new(), @@ -119,8 +121,10 @@ fn runtime_config() -> DockerDriverRuntimeConfig { key: PathBuf::from("/tmp/tls.key"), }), daemon_version: "28.0.0".to_string(), - supports_gpu: false, - allow_all_default_gpu: false, + gpu: DockerGpuRuntimeCapabilities { + cdi_supported: false, + wsl_all_gpu_fallback_enabled: false, + }, sandbox_pids_limit: DEFAULT_SANDBOX_PIDS_LIMIT, enable_bind_mounts: false, } @@ -150,7 +154,7 @@ fn inspected_volume(driver: &str, options: HashMap) -> bollard:: } fn test_driver_with_config(config: DockerDriverRuntimeConfig) -> DockerComputeDriver { - let allow_all_default_gpu = config.allow_all_default_gpu; + let wsl_all_gpu_fallback_enabled = config.gpu.wsl_all_gpu_fallback_enabled; DockerComputeDriver { docker: Arc::new( Docker::connect_with_http("http://127.0.0.1:2375", 1, bollard::API_DEFAULT_VERSION) @@ -161,19 +165,176 @@ fn test_driver_with_config(config: DockerDriverRuntimeConfig) -> DockerComputeDr pending: Arc::new(tokio::sync::Mutex::new(HashMap::new())), gpu_selector: Arc::new(CdiGpuDefaultSelector::new( CdiGpuInventory::default(), - allow_all_default_gpu, + wsl_all_gpu_fallback_enabled, )), lifecycle_event_fences: DockerLifecycleEventFences::default(), } } +#[test] +fn capabilities_report_static_resource_support() { + let mut config = runtime_config(); + let capabilities = test_driver_with_config(config.clone()).capabilities(); + let resources = capabilities.resource_capabilities.unwrap(); + assert!(resources.cpu.unwrap().limit_supported); + assert!(resources.memory.unwrap().limit_supported); + let gpu = resources.gpu.unwrap(); + assert!(!gpu.default_selection_supported); + assert!(!gpu.count_selection_supported); + + config.gpu.cdi_supported = true; + let gpu = test_driver_with_config(config) + .capabilities() + .resource_capabilities + .unwrap() + .gpu + .unwrap(); + assert!(gpu.default_selection_supported); + assert!(gpu.count_selection_supported); +} + +type TestDriverClient = + openshell_core::proto::compute::v1::compute_driver_client::ComputeDriverClient< + tonic::transport::Channel, + >; + +fn request_with_traceparent(message: T) -> Request { + let mut request = Request::new(message); + request.metadata_mut().insert( + "traceparent", + "00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01" + .parse() + .unwrap(), + ); + request +} + +async fn fake_docker_with_no_containers() -> (String, JoinHandle<()>) { + use tokio::io::{AsyncReadExt, AsyncWriteExt}; + + let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap(); + let address = listener.local_addr().unwrap(); + let server = tokio::spawn(async move { + while let Ok((mut stream, _)) = listener.accept().await { + openshell_core::net::set_tcp_nodelay_best_effort(&stream); + let mut scratch = [0_u8; 4096_usize]; + let _ = stream.read(&mut scratch).await; + let _ = stream + .write_all( + b"HTTP/1.1 200 OK\r\n\ + Content-Type: application/json\r\n\ + Content-Length: 2\r\n\r\n[]", + ) + .await; + let _ = stream.flush().await; + } + }); + (format!("http://{address}"), server) +} + +async fn standalone_traced_client() -> ( + TestDriverClient, + tokio::sync::oneshot::Sender<()>, + JoinHandle>, +) { + use openshell_core::proto::compute::v1::compute_driver_server::ComputeDriverServer; + + let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap(); + let address = listener.local_addr().unwrap(); + let (shutdown, shutdown_rx) = tokio::sync::oneshot::channel(); + let service = ComputeDriverService::new(test_driver_with_config(runtime_config())); + let server = tokio::spawn(async move { + tonic::transport::Server::builder() + .layer(openshell_otel::compute_driver_rpc_layer()) + .add_service(ComputeDriverServer::new(service)) + .serve_with_incoming_shutdown( + tokio_stream::wrappers::TcpListenerStream::new(listener), + async { + let _ = shutdown_rx.await; + }, + ) + .await + }); + let client = TestDriverClient::connect(format!("http://{address}")) + .await + .unwrap(); + (client, shutdown, server) +} + +#[tokio::test] +async fn tracing_standalone_rpc_layer_propagates_context_and_records_errors() { + use opentelemetry_sdk::trace::{InMemorySpanExporterBuilder, SdkTracerProvider}; + use tracing_subscriber::layer::SubscriberExt as _; + + let _tracing_lock = openshell_otel_test_support::tracing_test_lock().await; + let exporter = InMemorySpanExporterBuilder::new().build(); + let provider = SdkTracerProvider::builder() + .with_simple_exporter(exporter.clone()) + .build(); + let dispatch = tracing::Dispatch::new( + tracing_subscriber::registry().with(otel_tracing::TRACING.layer(&provider)), + ); + let _dispatch = tracing::dispatcher::set_default(&dispatch); + let (mut client, shutdown, server) = standalone_traced_client().await; + + client + .get_capabilities(request_with_traceparent(GetCapabilitiesRequest {})) + .await + .expect("capabilities should succeed"); + client + .validate_sandbox_create(request_with_traceparent(ValidateSandboxCreateRequest { + sandbox: None, + })) + .await + .expect_err("missing sandbox should fail"); + drop(client); + shutdown.send(()).unwrap(); + tokio::time::timeout(Duration::from_secs(5), server) + .await + .expect("standalone test server should stop") + .expect("standalone test server should not panic") + .expect("standalone test server should stop cleanly"); + provider.force_flush().unwrap(); + + let spans = exporter.get_finished_spans().unwrap(); + let capabilities = spans + .iter() + .find(|span| span.name == "openshell.compute.v1.ComputeDriver/GetCapabilities") + .expect("capabilities RPC span"); + assert_eq!( + capabilities.span_context.trace_id().to_string(), + "4bf92f3577b34da6a3ce929d0e0e4736" + ); + assert_eq!(capabilities.parent_span_id.to_string(), "00f067aa0ba902b7"); + assert!(capabilities.attributes.iter().any(|attribute| { + attribute.key.as_str() == "rpc.method" + && attribute.value.to_string() == "openshell.compute.v1.ComputeDriver/GetCapabilities" + })); + assert!( + capabilities + .attributes + .iter() + .all(|attribute| attribute.key.as_str() != "rpc.service"), + "the current RPC semantic conventions integrate the service into rpc.method" + ); + let failed = spans + .iter() + .find(|span| span.name == "openshell.compute.v1.ComputeDriver/ValidateSandboxCreate") + .expect("failed RPC span"); + assert!(matches!( + failed.status, + opentelemetry::trace::Status::Error { .. } + )); + provider.shutdown().unwrap(); +} + #[tokio::test] async fn tracing_in_process_service_preserves_the_driver_rpc_server_boundary() { use opentelemetry_sdk::trace::{InMemorySpanExporterBuilder, SdkTracerProvider}; use tracing::{Instrument as _, instrument::WithSubscriber as _}; use tracing_subscriber::layer::SubscriberExt as _; - let _tracing_lock = otel_tracing::test_lock().await; + let _tracing_lock = openshell_otel_test_support::tracing_test_lock().await; let gateway_exporter = InMemorySpanExporterBuilder::new().build(); let gateway_provider = SdkTracerProvider::builder() .with_simple_exporter(gateway_exporter.clone()) @@ -183,19 +344,19 @@ async fn tracing_in_process_service_preserves_the_driver_rpc_server_boundary() { .with_simple_exporter(driver_exporter.clone()) .build(); let subscriber = tracing_subscriber::registry() - .with(openshell_otel::layer_excluding_target_prefix( + .with(openshell_otel::layer_excluding_target_prefixes( &gateway_provider, "gateway-test", - Some(otel_tracing::IN_PROCESS_TARGET_PREFIX), + otel_tracing::TRACING.in_process_targets(), )) - .with(otel_tracing::in_process_layer(&driver_provider)); + .with(otel_tracing::TRACING.in_process_layer(&driver_provider)); let service = ComputeDriverService::new_in_process(test_driver_with_config(runtime_config())); async { let gateway_span = tracing::info_span!( target: "openshell_server::compute", "driver", - otel.name = "driver.get_capabilities", + otel.name = "openshell.compute.v1.ComputeDriver/GetCapabilities", otel.kind = "client" ); ComputeDriver::get_capabilities(&service, Request::new(GetCapabilitiesRequest {})) @@ -208,6 +369,12 @@ async fn tracing_in_process_service_preserves_the_driver_rpc_server_boundary() { ); drop(unrelated.enter()); drop(unrelated); + let selected_backend = tracing::info_span!( + target: "openshell_driver_docker::compute", + "docker.operation" + ); + drop(selected_backend.enter()); + drop(selected_backend); Ok::<_, Status>(()) } .with_subscriber(subscriber) @@ -217,7 +384,7 @@ async fn tracing_in_process_service_preserves_the_driver_rpc_server_boundary() { let gateway_span = tracing::info_span!( target: "openshell_server::compute", "driver", - otel.name = "driver.validate_sandbox_create", + otel.name = "openshell.compute.v1.ComputeDriver/ValidateSandboxCreate", otel.kind = "client" ); ComputeDriver::validate_sandbox_create( @@ -229,12 +396,12 @@ async fn tracing_in_process_service_preserves_the_driver_rpc_server_boundary() { } .with_subscriber( tracing_subscriber::registry() - .with(openshell_otel::layer_excluding_target_prefix( + .with(openshell_otel::layer_excluding_target_prefixes( &gateway_provider, "gateway-test", - Some(otel_tracing::IN_PROCESS_TARGET_PREFIX), + otel_tracing::TRACING.in_process_targets(), )) - .with(otel_tracing::in_process_layer(&driver_provider)), + .with(otel_tracing::TRACING.in_process_layer(&driver_provider)), ) .await .expect_err("missing sandbox should fail"); @@ -245,11 +412,11 @@ async fn tracing_in_process_service_preserves_the_driver_rpc_server_boundary() { let driver_spans = driver_exporter.get_finished_spans().unwrap(); let client = gateway_spans .iter() - .find(|span| span.name == "driver.get_capabilities") + .find(|span| span.name == "openshell.compute.v1.ComputeDriver/GetCapabilities") .unwrap(); let server = driver_spans .iter() - .find(|span| span.name == "driver.get_capabilities") + .find(|span| span.name == "openshell.compute.v1.ComputeDriver/GetCapabilities") .expect("in-process server span"); assert_eq!( server.span_context.trace_id(), @@ -258,8 +425,18 @@ async fn tracing_in_process_service_preserves_the_driver_rpc_server_boundary() { assert_eq!(server.parent_span_id, client.span_context.span_id()); assert_eq!(server.span_kind, opentelemetry::trace::SpanKind::Server); assert!(server.attributes.iter().any(|attribute| { - attribute.key.as_str() == "rpc.grpc.status_code" - && attribute.value.to_string() == (tonic::Code::Ok as i32).to_string() + attribute.key.as_str() == "rpc.method" + && attribute.value.to_string() == "openshell.compute.v1.ComputeDriver/GetCapabilities" + })); + assert!( + server + .attributes + .iter() + .all(|attribute| attribute.key.as_str() != "rpc.service"), + "the current RPC semantic conventions integrate the service into rpc.method" + ); + assert!(server.attributes.iter().any(|attribute| { + attribute.key.as_str() == "rpc.response.status_code" && attribute.value.to_string() == "OK" })); assert!( gateway_spans @@ -273,17 +450,29 @@ async fn tracing_in_process_service_preserves_the_driver_rpc_server_boundary() { .all(|span| span.name != "kubernetes.operation"), "the Docker provider must not claim unrelated driver spans" ); + assert!( + gateway_spans + .iter() + .all(|span| span.name != "docker.operation"), + "the gateway provider must not claim the selected driver's backend spans" + ); + assert!( + driver_spans + .iter() + .any(|span| span.name == "docker.operation"), + "the Docker provider must export backend spans from the selected driver" + ); let failed = driver_spans .iter() - .find(|span| span.name == "driver.validate_sandbox_create") + .find(|span| span.name == "openshell.compute.v1.ComputeDriver/ValidateSandboxCreate") .expect("failed in-process server span"); assert!(matches!( failed.status, opentelemetry::trace::Status::Error { .. } )); assert!(failed.attributes.iter().any(|attribute| { - attribute.key.as_str() == "rpc.grpc.status_code" - && attribute.value.to_string() == (tonic::Code::InvalidArgument as i32).to_string() + attribute.key.as_str() == "rpc.response.status_code" + && attribute.value.to_string() == "INVALID_ARGUMENT" })); gateway_provider.shutdown().unwrap(); driver_provider.shutdown().unwrap(); @@ -295,12 +484,12 @@ async fn tracing_lifecycle_rpc_failures_export_docker_operation_spans() { use tracing::instrument::WithSubscriber as _; use tracing_subscriber::layer::SubscriberExt as _; - let _tracing_lock = otel_tracing::test_lock().await; + let _tracing_lock = openshell_otel_test_support::tracing_test_lock().await; let exporter = InMemorySpanExporterBuilder::new().build(); let provider = SdkTracerProvider::builder() .with_simple_exporter(exporter.clone()) .build(); - let subscriber = tracing_subscriber::registry().with(otel_tracing::layer(&provider)); + let subscriber = tracing_subscriber::registry().with(otel_tracing::TRACING.layer(&provider)); let driver = test_driver_with_config(runtime_config()); async { @@ -349,12 +538,12 @@ async fn tracing_direct_start_exports_a_docker_start_span() { use tracing::instrument::WithSubscriber as _; use tracing_subscriber::layer::SubscriberExt as _; - let _tracing_lock = otel_tracing::test_lock().await; + let _tracing_lock = openshell_otel_test_support::tracing_test_lock().await; let exporter = InMemorySpanExporterBuilder::new().build(); let provider = SdkTracerProvider::builder() .with_simple_exporter(exporter.clone()) .build(); - let subscriber = tracing_subscriber::registry().with(otel_tracing::layer(&provider)); + let subscriber = tracing_subscriber::registry().with(otel_tracing::TRACING.layer(&provider)); let driver = test_driver_with_config(runtime_config()); DockerComputeDriver::start_sandbox(&driver, "", "") @@ -378,30 +567,37 @@ async fn tracing_direct_start_exports_a_docker_start_span() { #[tokio::test] async fn tracing_image_preparation_failure_exports_nested_failed_spans() { use opentelemetry_sdk::trace::{InMemorySpanExporterBuilder, SdkTracerProvider}; - use tracing::instrument::WithSubscriber as _; + use tracing::{Instrument as _, instrument::WithSubscriber as _}; use tracing_subscriber::layer::SubscriberExt as _; - let _tracing_lock = otel_tracing::test_lock().await; + let _tracing_lock = openshell_otel_test_support::tracing_test_lock().await; let exporter = InMemorySpanExporterBuilder::new().build(); let provider = SdkTracerProvider::builder() .with_simple_exporter(exporter.clone()) .build(); - let subscriber = tracing_subscriber::registry().with(otel_tracing::layer(&provider)); + let subscriber = tracing_subscriber::registry().with(otel_tracing::TRACING.layer(&provider)); let mut config = runtime_config(); config.image_pull_policy = "unsupported".to_string(); let driver = test_driver_with_config(config); - driver - .provision_sandbox_inner(&test_sandbox()) - .with_subscriber(subscriber) - .await - .expect_err("unsupported image pull policy should fail provisioning"); + async { + driver + .provision_sandbox_inner(&test_sandbox()) + .instrument(tracing::info_span!( + "docker.provision", + otel.status_code = tracing::field::Empty + )) + .await + } + .with_subscriber(subscriber) + .await + .expect_err("unsupported image pull policy should fail provisioning"); provider.force_flush().unwrap(); let spans = exporter.get_finished_spans().unwrap(); let provision = spans .iter() - .find(|span| span.name == "docker.provision_sandbox") + .find(|span| span.name == "docker.provision") .expect("provisioning span should be exported"); assert!(matches!( provision.status, @@ -429,12 +625,12 @@ async fn background_provisioning_does_not_extend_the_scheduling_span_lifetime() use tracing_opentelemetry::OpenTelemetrySpanExt as _; use tracing_subscriber::layer::SubscriberExt as _; - let _tracing_lock = otel_tracing::test_lock().await; + let _tracing_lock = openshell_otel_test_support::tracing_test_lock().await; let exporter = InMemorySpanExporterBuilder::new().build(); let provider = SdkTracerProvider::builder() .with_simple_exporter(exporter.clone()) .build(); - let subscriber = tracing_subscriber::registry().with(otel_tracing::layer(&provider)); + let subscriber = tracing_subscriber::registry().with(otel_tracing::TRACING.layer(&provider)); let dispatch = tracing::Dispatch::new(subscriber); let _dispatch = tracing::dispatcher::set_default(&dispatch); @@ -471,30 +667,27 @@ async fn tracing_in_process_stream_span_lives_until_stream_failure() { use tracing::instrument::WithSubscriber as _; use tracing_subscriber::layer::SubscriberExt as _; - let _tracing_lock = otel_tracing::test_lock().await; + let _tracing_lock = openshell_otel_test_support::tracing_test_lock().await; let exporter = InMemorySpanExporterBuilder::new().build(); let provider = SdkTracerProvider::builder() .with_simple_exporter(exporter.clone()) .build(); - let subscriber = tracing_subscriber::registry().with(otel_tracing::in_process_layer(&provider)); + let subscriber = + tracing_subscriber::registry().with(otel_tracing::TRACING.in_process_layer(&provider)); async { let span = tracing::info_span!( - target: "openshell_driver_docker::otel_tracing", + target: otel_tracing::TRACING.in_process_target(), "driver_rpc", - otel.name = "driver.watch_sandboxes", + otel.name = "openshell.compute.v1.ComputeDriver/WatchSandboxes", otel.kind = "server", otel.status_code = tracing::field::Empty, - rpc.grpc.status_code = tracing::field::Empty, + rpc.response.status_code = tracing::field::Empty, ); let inner: WatchStream = Box::pin(futures::stream::iter([Err(Status::internal( "watch failed", ))])); - let mut stream = TracedWatchStream { - inner, - span, - finished: false, - }; + let mut stream = TracedWatchStream::new(inner, span); provider.force_flush().unwrap(); assert!( @@ -515,7 +708,7 @@ async fn tracing_in_process_stream_span_lives_until_stream_failure() { let spans = exporter.get_finished_spans().unwrap(); let span = spans .iter() - .find(|span| span.name == "driver.watch_sandboxes") + .find(|span| span.name == "openshell.compute.v1.ComputeDriver/WatchSandboxes") .expect("watch server span should be exported when the stream ends"); assert!(matches!( span.status, @@ -530,28 +723,25 @@ async fn tracing_in_process_stream_records_ok_when_stream_completes() { use tracing::instrument::WithSubscriber as _; use tracing_subscriber::layer::SubscriberExt as _; - let _tracing_lock = otel_tracing::test_lock().await; + let _tracing_lock = openshell_otel_test_support::tracing_test_lock().await; let exporter = InMemorySpanExporterBuilder::new().build(); let provider = SdkTracerProvider::builder() .with_simple_exporter(exporter.clone()) .build(); - let subscriber = tracing_subscriber::registry().with(otel_tracing::in_process_layer(&provider)); + let subscriber = + tracing_subscriber::registry().with(otel_tracing::TRACING.in_process_layer(&provider)); async { let span = tracing::info_span!( - target: "openshell_driver_docker::otel_tracing", + target: otel_tracing::TRACING.in_process_target(), "driver_rpc", - otel.name = "driver.watch_sandboxes", + otel.name = "openshell.compute.v1.ComputeDriver/WatchSandboxes", otel.kind = "server", otel.status_code = tracing::field::Empty, - rpc.grpc.status_code = tracing::field::Empty, + rpc.response.status_code = tracing::field::Empty, ); let inner: WatchStream = Box::pin(futures::stream::empty()); - let mut stream = TracedWatchStream { - inner, - span, - finished: false, - }; + let mut stream = TracedWatchStream::new(inner, span); assert!(stream.next().await.is_none()); drop(stream); @@ -563,43 +753,39 @@ async fn tracing_in_process_stream_records_ok_when_stream_completes() { let spans = exporter.get_finished_spans().unwrap(); let span = spans .iter() - .find(|span| span.name == "driver.watch_sandboxes") + .find(|span| span.name == "openshell.compute.v1.ComputeDriver/WatchSandboxes") .expect("watch server span should be exported when the stream completes"); assert!(span.attributes.iter().any(|attribute| { - attribute.key.as_str() == "rpc.grpc.status_code" - && attribute.value.to_string() == (tonic::Code::Ok as i32).to_string() + attribute.key.as_str() == "rpc.response.status_code" && attribute.value.to_string() == "OK" })); provider.shutdown().unwrap(); } #[tokio::test] -async fn tracing_in_process_stream_records_cancelled_when_dropped() { +async fn tracing_in_process_stream_leaves_status_unset_when_dropped() { use opentelemetry_sdk::trace::{InMemorySpanExporterBuilder, SdkTracerProvider}; use tracing::instrument::WithSubscriber as _; use tracing_subscriber::layer::SubscriberExt as _; - let _tracing_lock = otel_tracing::test_lock().await; + let _tracing_lock = openshell_otel_test_support::tracing_test_lock().await; let exporter = InMemorySpanExporterBuilder::new().build(); let provider = SdkTracerProvider::builder() .with_simple_exporter(exporter.clone()) .build(); - let subscriber = tracing_subscriber::registry().with(otel_tracing::in_process_layer(&provider)); + let subscriber = + tracing_subscriber::registry().with(otel_tracing::TRACING.in_process_layer(&provider)); async { let span = tracing::info_span!( - target: "openshell_driver_docker::otel_tracing", + target: otel_tracing::TRACING.in_process_target(), "driver_rpc", - otel.name = "driver.watch_sandboxes", + otel.name = "openshell.compute.v1.ComputeDriver/WatchSandboxes", otel.kind = "server", otel.status_code = tracing::field::Empty, - rpc.grpc.status_code = tracing::field::Empty, + rpc.response.status_code = tracing::field::Empty, ); let inner: WatchStream = Box::pin(futures::stream::pending()); - let stream = TracedWatchStream { - inner, - span, - finished: false, - }; + let stream = TracedWatchStream::new(inner, span); drop(stream); } @@ -610,16 +796,14 @@ async fn tracing_in_process_stream_records_cancelled_when_dropped() { let spans = exporter.get_finished_spans().unwrap(); let span = spans .iter() - .find(|span| span.name == "driver.watch_sandboxes") - .expect("watch server span should be exported when the stream is cancelled"); - assert!(matches!( - span.status, - opentelemetry::trace::Status::Error { .. } - )); - assert!(span.attributes.iter().any(|attribute| { - attribute.key.as_str() == "rpc.grpc.status_code" - && attribute.value.to_string() == (tonic::Code::Cancelled as i32).to_string() - })); + .find(|span| span.name == "openshell.compute.v1.ComputeDriver/WatchSandboxes") + .expect("watch server span should be exported when the stream is dropped"); + assert!(matches!(span.status, opentelemetry::trace::Status::Unset)); + assert!( + span.attributes + .iter() + .all(|attribute| attribute.key.as_str() != "rpc.response.status_code") + ); provider.shutdown().unwrap(); } @@ -1091,7 +1275,9 @@ fn build_environment_sets_docker_tls_paths() { }) .expect("main-process transport"); let main = openshell_core::sandbox_env::MainProcessConfig::decode(&encoded).unwrap(); - assert_eq!(main.command, vec!["/bin/bash", "-l"]); + // An omitted command is forwarded empty; the supervisor resolves the default + // login shell against the sandbox image at startup. + assert!(main.command.is_empty()); assert!(main.tty); } @@ -2119,7 +2305,7 @@ fn validate_sandbox_rejects_unknown_driver_config_fields() { #[test] fn validate_sandbox_accepts_gpu_count_request_shape() { let mut config = runtime_config(); - config.supports_gpu = true; + config.gpu.cdi_supported = true; let mut sandbox = test_sandbox(); sandbox.spec.as_mut().unwrap().resource_requirements = Some(gpu_resources(Some(2))); @@ -2130,7 +2316,7 @@ fn validate_sandbox_accepts_gpu_count_request_shape() { #[test] fn validate_sandbox_accepts_gpu_count_matching_cdi_devices() { let mut config = runtime_config(); - config.supports_gpu = true; + config.gpu.cdi_supported = true; let mut sandbox = test_sandbox(); let spec = sandbox.spec.as_mut().unwrap(); spec.resource_requirements = Some(gpu_resources(Some(2))); @@ -2146,7 +2332,7 @@ fn validate_sandbox_accepts_gpu_count_matching_cdi_devices() { #[test] fn validate_sandbox_accepts_single_cdi_device_without_gpu_count() { let mut config = runtime_config(); - config.supports_gpu = true; + config.gpu.cdi_supported = true; let mut sandbox = test_sandbox(); let spec = sandbox.spec.as_mut().unwrap(); spec.resource_requirements = Some(gpu_resources(None)); @@ -2159,7 +2345,7 @@ fn validate_sandbox_accepts_single_cdi_device_without_gpu_count() { #[test] fn validate_sandbox_rejects_multiple_cdi_devices_without_gpu_count() { let mut config = runtime_config(); - config.supports_gpu = true; + config.gpu.cdi_supported = true; let mut sandbox = test_sandbox(); let spec = sandbox.spec.as_mut().unwrap(); spec.resource_requirements = Some(gpu_resources(None)); @@ -2180,7 +2366,7 @@ fn validate_sandbox_rejects_multiple_cdi_devices_without_gpu_count() { #[test] fn validate_sandbox_rejects_cdi_devices_without_gpu_request() { let mut config = runtime_config(); - config.supports_gpu = true; + config.gpu.cdi_supported = true; let mut sandbox = test_sandbox(); sandbox .spec @@ -2200,7 +2386,7 @@ fn validate_sandbox_rejects_cdi_devices_without_gpu_request() { #[test] fn validate_sandbox_rejects_gpu_count_mismatched_cdi_devices() { let mut config = runtime_config(); - config.supports_gpu = true; + config.gpu.cdi_supported = true; let mut sandbox = test_sandbox(); let spec = sandbox.spec.as_mut().unwrap(); spec.resource_requirements = Some(gpu_resources(Some(2))); @@ -2256,7 +2442,7 @@ fn validate_sandbox_auth_accepts_gateway_token() { #[test] fn build_container_create_body_maps_default_gpu_to_selected_cdi_device() { let mut config = runtime_config(); - config.supports_gpu = true; + config.gpu.cdi_supported = true; let mut sandbox = test_sandbox(); sandbox.spec.as_mut().unwrap().resource_requirements = Some(gpu_resources(None)); @@ -2286,7 +2472,7 @@ fn build_container_create_body_maps_default_gpu_to_selected_cdi_device() { #[test] fn build_container_create_body_omits_devices_without_resolved_default_cdi_devices() { let mut config = runtime_config(); - config.supports_gpu = true; + config.gpu.cdi_supported = true; let mut sandbox = test_sandbox(); sandbox.spec.as_mut().unwrap().resource_requirements = Some(gpu_resources(None)); @@ -2304,7 +2490,7 @@ fn build_container_create_body_omits_devices_without_resolved_default_cdi_device #[test] fn build_container_create_body_passes_explicit_cdi_device_id_through() { let mut config = runtime_config(); - config.supports_gpu = true; + config.gpu.cdi_supported = true; let mut sandbox = test_sandbox(); let spec = sandbox.spec.as_mut().unwrap(); spec.resource_requirements = Some(gpu_resources(None)); @@ -2328,7 +2514,7 @@ fn build_container_create_body_passes_explicit_cdi_device_id_through() { #[test] fn build_container_create_body_rejects_gpu_count_mismatched_cdi_devices() { let mut config = runtime_config(); - config.supports_gpu = true; + config.gpu.cdi_supported = true; let mut sandbox = test_sandbox(); let spec = sandbox.spec.as_mut().unwrap(); spec.resource_requirements = Some(gpu_resources(Some(2))); @@ -2375,7 +2561,7 @@ fn build_container_create_body_rejects_empty_cdi_devices() { #[test] fn driver_default_gpu_selection_consumes_distinct_devices_for_creates() { let mut config = runtime_config(); - config.supports_gpu = true; + config.gpu.cdi_supported = true; let driver = test_driver_with_config(config); driver.gpu_selector.refresh( CdiGpuInventory::new(["nvidia.com/gpu=0", "nvidia.com/gpu=1"]), @@ -2715,8 +2901,17 @@ fn pending_sandbox_snapshot_uses_docker_namespace_and_starting_condition() { assert_eq!(snapshot.name, "demo"); assert_eq!(snapshot.namespace, "docker-dev"); assert!(snapshot.spec.is_none()); - assert!(pending_sandbox_matches(&snapshot, "sbx-123", "")); - assert!(pending_sandbox_matches(&snapshot, "", "demo")); + let pending = pending_map(&[&snapshot]); + assert_eq!( + resolve_pending_id(&pending, "sbx-123", "") + .unwrap() + .as_deref(), + Some("sbx-123") + ); + assert_eq!( + resolve_pending_id(&pending, "", "demo").unwrap().as_deref(), + Some("sbx-123") + ); let status = snapshot.status.expect("status"); assert!(!status.deleting); @@ -3138,6 +3333,15 @@ fn ready_reason(sandbox: &DriverSandbox) -> &str { .expect("Ready condition present") } +fn ready_message(sandbox: &DriverSandbox) -> &str { + sandbox + .status + .as_ref() + .and_then(|status| status.conditions.iter().find(|c| c.r#type == "Ready")) + .map(|c| c.message.as_str()) + .expect("Ready condition present") +} + #[test] fn docker_signal_kill_reclassified_as_runtime_restart() { // 137 (128+SIGKILL) and 143 (128+SIGTERM) mark an external termination — @@ -3175,6 +3379,24 @@ fn docker_ordinary_exit_stays_terminal() { assert_eq!(ready_reason(&sandbox), CONDITION_EXITED); } +#[test] +fn docker_workspace_validation_exit_is_reported_explicitly() { + let mut sandbox = exited_sandbox_with_ready_reason(CONDITION_EXITED); + let state = ContainerState { + status: Some(ContainerStateStatusEnum::EXITED), + exit_code: Some(i64::from(SUPERVISOR_EXIT_WORKSPACE_VALIDATION_FAILED)), + ..Default::default() + }; + + apply_docker_exit_classification(&mut sandbox, &state); + + assert_eq!( + ready_reason(&sandbox), + CONDITION_WORKSPACE_VALIDATION_FAILED + ); + assert!(ready_message(&sandbox).contains("WorkingDir")); +} + #[test] fn docker_oom_kill_stays_terminal_despite_137() { // An OOM kill reports exit 137 but must NOT be treated as a recoverable @@ -3189,3 +3411,407 @@ fn docker_oom_kill_stays_terminal_despite_137() { apply_docker_exit_classification(&mut sandbox, &state); assert_eq!(ready_reason(&sandbox), CONDITION_EXITED); } + +/// Minimal pending-map entry. Only the identity fields matter for lookup +/// resolution, so the spec and status are left empty on purpose. +fn pending_sandbox(id: &str, name: &str, workspace: &str) -> DriverSandbox { + DriverSandbox { + id: id.to_string(), + name: name.to_string(), + namespace: String::new(), + spec: None, + status: None, + workspace: workspace.to_string(), + } +} + +fn pending_map(sandboxes: &[&DriverSandbox]) -> HashMap { + sandboxes + .iter() + .map(|sandbox| { + ( + sandbox.id.clone(), + PendingSandboxRecord { + sandbox: (*sandbox).clone(), + task: None, + }, + ) + }) + .collect() +} + +async fn driver_with_pending(sandboxes: &[&DriverSandbox]) -> DockerComputeDriver { + let driver = test_driver_with_config(runtime_config()); + for sandbox in sandboxes { + driver + .reserve_pending_sandbox(sandbox) + .await + .expect("reserving a distinct sandbox must succeed"); + } + driver +} + +fn pending_ids(pending: &HashMap) -> Vec { + let mut ids: Vec = pending.keys().cloned().collect(); + ids.sort(); + ids +} + +#[test] +fn resolve_pending_id_prefers_sandbox_id_over_sandbox_name() { + // The id is authoritative. A stale or mismatched name travelling in the + // same request must not change which record is resolved. + let alpha = pending_sandbox("sbx-alpha", "demo", "alpha"); + let pending = pending_map(&[&alpha]); + + assert_eq!( + resolve_pending_id(&pending, "sbx-alpha", "stale-name") + .unwrap() + .as_deref(), + Some("sbx-alpha") + ); +} + +#[test] +fn resolve_pending_id_ignores_the_name_when_the_id_is_not_pending() { + // Regression for the `id OR name` match. `demo` exists in two workspaces: + // the beta copy is still provisioning, the alpha copy is already running. + // Deleting the alpha copy sends alpha's id plus the shared name. Matching + // on the name alone resolved to the beta record and evicted it, aborting + // an unrelated sandbox's provisioning task. + let beta = pending_sandbox("sbx-beta", "demo", "beta"); + let pending = pending_map(&[&beta]); + + assert_eq!( + resolve_pending_id(&pending, "sbx-alpha", "demo").unwrap(), + None + ); +} + +#[test] +fn resolve_pending_id_falls_back_to_the_name_when_no_id_is_supplied() { + // Direct driver callers may omit the id; a unique name still resolves. + let alpha = pending_sandbox("sbx-alpha", "demo", "alpha"); + let pending = pending_map(&[&alpha]); + + assert_eq!( + resolve_pending_id(&pending, "", "demo").unwrap().as_deref(), + Some("sbx-alpha") + ); +} + +#[test] +fn resolve_pending_id_rejects_an_ambiguous_name_only_lookup() { + // Two pending sandboxes share a name across workspaces and the driver + // request carries no workspace. Picking either one would make the outcome + // depend on `HashMap` iteration order, so refuse instead. + let alpha = pending_sandbox("sbx-alpha", "demo", "alpha"); + let beta = pending_sandbox("sbx-beta", "demo", "beta"); + let pending = pending_map(&[&alpha, &beta]); + + let err = resolve_pending_id(&pending, "", "demo") + .expect_err("an ambiguous name-only lookup must be rejected"); + assert_eq!(err.code(), tonic::Code::FailedPrecondition); +} + +#[test] +fn resolve_pending_id_returns_none_without_any_identifier() { + // `require_sandbox_identifier` rejects this upstream, but the resolver + // stays total so an empty request can never match an arbitrary record. + let alpha = pending_sandbox("sbx-alpha", "demo", "alpha"); + let pending = pending_map(&[&alpha]); + + assert_eq!(resolve_pending_id(&pending, "", "").unwrap(), None); +} + +#[tokio::test] +async fn remove_pending_sandbox_by_id_keeps_a_same_named_sandbox_in_another_workspace() { + let alpha = pending_sandbox("sbx-alpha", "demo", "alpha"); + let beta = pending_sandbox("sbx-beta", "demo", "beta"); + let driver = driver_with_pending(&[&alpha, &beta]).await; + + let removed = driver + .remove_pending_sandbox("sbx-alpha", "demo") + .await + .expect("an id-scoped removal must succeed") + .expect("the alpha record must be removed"); + + assert_eq!(removed.sandbox.id, "sbx-alpha"); + assert_eq!( + pending_ids(&driver.pending_snapshot_map().await), + ["sbx-beta"] + ); +} + +#[tokio::test] +async fn remove_pending_sandbox_by_a_unique_name_still_removes_the_record() { + let alpha = pending_sandbox("sbx-alpha", "demo", "alpha"); + let driver = driver_with_pending(&[&alpha]).await; + + let removed = driver + .remove_pending_sandbox("", "demo") + .await + .expect("a unique name-only removal must succeed") + .expect("the alpha record must be removed"); + + assert_eq!(removed.sandbox.id, "sbx-alpha"); + assert!(driver.pending_snapshot_map().await.is_empty()); +} + +#[tokio::test] +async fn remove_pending_sandbox_rejects_an_ambiguous_name_and_keeps_both_records() { + let alpha = pending_sandbox("sbx-alpha", "demo", "alpha"); + let beta = pending_sandbox("sbx-beta", "demo", "beta"); + let driver = driver_with_pending(&[&alpha, &beta]).await; + + let err = driver + .remove_pending_sandbox("", "demo") + .await + .map(|record| record.map(|record| record.sandbox.id)) + .expect_err("an ambiguous name-only removal must be rejected"); + + assert_eq!(err.code(), tonic::Code::FailedPrecondition); + assert_eq!( + pending_ids(&driver.pending_snapshot_map().await), + ["sbx-alpha", "sbx-beta"] + ); +} + +#[tokio::test] +async fn pending_snapshot_by_id_ignores_a_same_named_sandbox_in_another_workspace() { + // `GetSandbox` falls through to the pending map when no container exists. + // Resolving by name there leaked another workspace's snapshot. + let beta = pending_sandbox("sbx-beta", "demo", "beta"); + let driver = driver_with_pending(&[&beta]).await; + + assert!( + driver + .pending_snapshot("sbx-alpha", "demo") + .await + .expect("an id-scoped snapshot lookup must succeed") + .is_none() + ); +} + +#[tokio::test] +async fn pending_snapshot_rejects_an_ambiguous_name_only_lookup() { + let alpha = pending_sandbox("sbx-alpha", "demo", "alpha"); + let beta = pending_sandbox("sbx-beta", "demo", "beta"); + let driver = driver_with_pending(&[&alpha, &beta]).await; + + let err = driver + .pending_snapshot("", "demo") + .await + .expect_err("an ambiguous name-only snapshot lookup must be rejected"); + + assert_eq!(err.code(), tonic::Code::FailedPrecondition); +} + +#[tokio::test] +async fn reserve_pending_sandbox_allows_the_same_name_in_a_different_workspace() { + // Sandbox names are unique per workspace, so this is a legitimate create. + let alpha = pending_sandbox("sbx-alpha", "demo", "alpha"); + let beta = pending_sandbox("sbx-beta", "demo", "beta"); + let driver = driver_with_pending(&[&alpha]).await; + + driver + .reserve_pending_sandbox(&beta) + .await + .expect("a same-named sandbox in another workspace must be allowed"); + + assert_eq!( + pending_ids(&driver.pending_snapshot_map().await), + ["sbx-alpha", "sbx-beta"] + ); +} + +#[tokio::test] +async fn reserve_pending_sandbox_rejects_a_duplicate_name_in_the_same_workspace() { + let alpha = pending_sandbox("sbx-alpha", "demo", "alpha"); + let duplicate = pending_sandbox("sbx-other", "demo", "alpha"); + let driver = driver_with_pending(&[&alpha]).await; + + let err = driver + .reserve_pending_sandbox(&duplicate) + .await + .expect_err("a duplicate name within one workspace must be rejected"); + + assert_eq!(err.code(), tonic::Code::AlreadyExists); + assert_eq!( + pending_ids(&driver.pending_snapshot_map().await), + ["sbx-alpha"] + ); +} + +#[tokio::test] +async fn reserve_pending_sandbox_rejects_a_duplicate_id() { + let alpha = pending_sandbox("sbx-alpha", "demo", "alpha"); + let duplicate = pending_sandbox("sbx-alpha", "other-name", "beta"); + let driver = driver_with_pending(&[&alpha]).await; + + let err = driver + .reserve_pending_sandbox(&duplicate) + .await + .expect_err("a duplicate sandbox id must be rejected regardless of workspace"); + + assert_eq!(err.code(), tonic::Code::AlreadyExists); + assert_eq!( + pending_ids(&driver.pending_snapshot_map().await), + ["sbx-alpha"] + ); +} + +fn managed_container_labels( + namespace: &str, + sandbox_id: &str, + sandbox_name: &str, +) -> HashMap { + HashMap::from([ + ( + LABEL_MANAGED_BY.to_string(), + LABEL_MANAGED_BY_VALUE.to_string(), + ), + (LABEL_SANDBOX_NAMESPACE.to_string(), namespace.to_string()), + (LABEL_SANDBOX_ID.to_string(), sandbox_id.to_string()), + (LABEL_SANDBOX_NAME.to_string(), sandbox_name.to_string()), + ]) +} + +#[test] +fn managed_container_identity_matches_on_id_despite_a_stale_name() { + // Requiring the name to agree with an authoritative id dropped the match + // and made the driver report a live sandbox as absent, stranding the + // container and leaking its token file. + let labels = managed_container_labels("default", "sbx-alpha", "demo"); + + assert!(managed_container_identity_matches( + &labels, + "default", + "sbx-alpha", + "stale-name" + )); +} + +#[test] +fn managed_container_identity_rejects_a_name_match_when_the_id_differs() { + // The mirror of the pending-map fix: a shared name must not stand in for + // an id that explicitly disagrees. + let labels = managed_container_labels("default", "sbx-beta", "demo"); + + assert!(!managed_container_identity_matches( + &labels, + "default", + "sbx-alpha", + "demo" + )); +} + +#[test] +fn managed_container_identity_falls_back_to_the_name_without_an_id() { + let labels = managed_container_labels("default", "sbx-alpha", "demo"); + + assert!(managed_container_identity_matches( + &labels, "default", "", "demo" + )); + assert!(!managed_container_identity_matches( + &labels, "default", "", "other" + )); +} + +#[test] +fn managed_container_identity_matches_nothing_without_an_identifier() { + // The label filters degenerate to "every managed container in the + // namespace" when neither identifier is supplied, so the predicate must + // not wave the container through. + let labels = managed_container_labels("default", "sbx-alpha", "demo"); + + assert!(!managed_container_identity_matches( + &labels, "default", "", "" + )); +} + +#[test] +fn managed_container_identity_requires_the_configured_namespace() { + let labels = managed_container_labels("other-namespace", "sbx-alpha", "demo"); + + assert!(!managed_container_identity_matches( + &labels, + "default", + "sbx-alpha", + "demo" + )); +} + +#[tokio::test] +async fn delete_sandbox_reclaims_token_file_when_container_and_pending_are_gone() { + let state_dir = tempfile::tempdir().unwrap(); + let (endpoint, server) = fake_docker_with_no_containers().await; + + temp_env::async_with_vars([("XDG_STATE_HOME", Some(state_dir.path()))], async { + let config = runtime_config(); + let mut driver = test_driver_with_config(config.clone()); + driver.docker = Arc::new( + Docker::connect_with_http(&endpoint, 5, bollard::API_DEFAULT_VERSION).unwrap(), + ); + + // Arrange the leak: token on disk, container gone, `pending` empty. + let token = openshell_core::driver_utils::sandbox_token_path( + "docker-sandbox-tokens", + Some(&config.sandbox_namespace), + "sandbox-1", + ) + .unwrap(); + + fs::create_dir_all(token.parent().unwrap()).unwrap(); + fs::write(&token, "jwt\n").unwrap(); + + let deleted = driver.delete_sandbox_inner("sandbox-1", "").await.unwrap(); + assert!(!deleted, "nothing was removed, must not claim a deletion"); + assert!(!token.exists(), "token file must be reclaimed"); + }) + .await; + + server.abort(); +} + +#[tokio::test] +async fn delete_sandbox_by_name_only_leaves_the_namespace_directory_alone() { + // `DeleteSandbox` accepts a name without an id. With no id there is no + // token path to derive, so the cleanup must be a no-op: deriving a path + // from an empty id yields `/sandbox.jwt`, whose parent is the + // shared namespace directory. + let state_dir = tempfile::tempdir().unwrap(); + let (endpoint, server) = fake_docker_with_no_containers().await; + + temp_env::async_with_vars([("XDG_STATE_HOME", Some(state_dir.path()))], async { + let config = runtime_config(); + let mut driver = test_driver_with_config(config.clone()); + driver.docker = Arc::new( + Docker::connect_with_http(&endpoint, 5, bollard::API_DEFAULT_VERSION).unwrap(), + ); + + let namespace_dir = openshell_core::driver_utils::sandbox_token_path( + "docker-sandbox-tokens", + Some(&config.sandbox_namespace), + "sandbox-1", + ) + .unwrap() + .parent() + .and_then(Path::parent) + .unwrap() + .to_path_buf(); + fs::create_dir_all(&namespace_dir).unwrap(); + + let deleted = driver.delete_sandbox_inner("", "sandbox-1").await.unwrap(); + + assert!(!deleted, "nothing was removed, must not claim a deletion"); + assert!( + namespace_dir.is_dir(), + "namespace directory must survive a name-only delete: {}", + namespace_dir.display() + ); + }) + .await; + + server.abort(); +} diff --git a/crates/openshell-driver-kubernetes/README.md b/crates/openshell-driver-kubernetes/README.md index 0d0186ae1..02dcfe5e8 100644 --- a/crates/openshell-driver-kubernetes/README.md +++ b/crates/openshell-driver-kubernetes/README.md @@ -79,9 +79,9 @@ its pod. The driver sets `spec.operatingMode: Suspended` for `v1beta1` or `spec.replicas: 0` for `v1alpha1`. Start sets `Running` or one replica for the same resource, so the replacement pod mounts the existing claim. Delete is the only lifecycle operation that removes the Sandbox resource and its owned -storage. The driver confirms the stop from the published `Suspended` -condition when available. Legacy `v1alpha1` controllers omit a zero replica -count from status, so the driver confirms that their backing pod is gone. +storage. The driver confirms the stop from both the published `Suspended` +condition and deletion of the backing pod. Legacy `v1alpha1` controllers omit +a usable stopped condition, so pod deletion alone confirms their stop. The workspace PVC size defaults to `workspace_default_storage_size`. Set `workspace_storage_class` to pin the PVC to a specific `StorageClass`; an empty diff --git a/crates/openshell-driver-kubernetes/src/config.rs b/crates/openshell-driver-kubernetes/src/config.rs index aadcb1342..805c0314b 100644 --- a/crates/openshell-driver-kubernetes/src/config.rs +++ b/crates/openshell-driver-kubernetes/src/config.rs @@ -1,7 +1,7 @@ // SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 -pub use openshell_core::OperatorNamespaceAllowlist; +pub use openshell_core::DynamicStringAllowlist as OperatorNamespaceAllowlist; use openshell_core::config; use serde::{Deserialize, Deserializer, Serialize}; use std::collections::BTreeMap; diff --git a/crates/openshell-driver-kubernetes/src/driver.rs b/crates/openshell-driver-kubernetes/src/driver.rs index f4addd384..1e790bf34 100644 --- a/crates/openshell-driver-kubernetes/src/driver.rs +++ b/crates/openshell-driver-kubernetes/src/driver.rs @@ -44,16 +44,17 @@ use openshell_core::progress::{ format_bytes, mark_progress_active, mark_progress_complete, mark_progress_detail, }; use openshell_core::proto::compute::v1::{ - DriverCondition as SandboxCondition, DriverPlatformEvent as PlatformEvent, - DriverSandbox as Sandbox, DriverSandboxSpec as SandboxSpec, - DriverSandboxStatus as SandboxStatus, DriverSandboxTemplate as SandboxTemplate, - GetCapabilitiesResponse, GpuResourceRequirements, WatchSandboxesDeletedEvent, - WatchSandboxesEvent, WatchSandboxesPlatformEvent, WatchSandboxesSandboxEvent, - watch_sandboxes_event, + CpuResourceCapabilities, DriverCondition as SandboxCondition, + DriverPlatformEvent as PlatformEvent, DriverSandbox as Sandbox, + DriverSandboxSpec as SandboxSpec, DriverSandboxStatus as SandboxStatus, + DriverSandboxTemplate as SandboxTemplate, GetCapabilitiesResponse, GpuResourceCapabilities, + GpuResourceRequirements, MemoryResourceCapabilities, ResourceCapabilities, + WatchSandboxesDeletedEvent, WatchSandboxesEvent, WatchSandboxesPlatformEvent, + WatchSandboxesSandboxEvent, watch_sandboxes_event, }; use openshell_core::proto_struct::{struct_to_json_object, value_to_json}; use serde::Deserialize; -use std::collections::{BTreeMap, HashSet}; +use std::collections::{BTreeMap, BTreeSet, HashSet}; use std::path::{Path, PathBuf}; use std::pin::Pin; use std::sync::Arc; @@ -575,6 +576,21 @@ impl KubernetesComputeDriver { default_image: self.config.default_image.clone(), gateway_manages_lifecycle: false, supports_sandbox_authentication: true, + driver_reports_runtime_readiness: false, + resource_capabilities: Some(ResourceCapabilities { + cpu: Some(CpuResourceCapabilities { + limit_supported: true, + }), + memory: Some(MemoryResourceCapabilities { + limit_supported: true, + }), + gpu: Some(GpuResourceCapabilities { + default_selection_supported: true, + count_selection_supported: true, + }), + }), + rootfs_tar_staging_dir: String::new(), + rootfs_tar_max_bytes: 0, }) } @@ -1427,10 +1443,10 @@ impl KubernetesComputeDriver { #[allow(clippy::similar_names)] #[tracing::instrument( - name = "kubernetes.create_sandbox", + name = "kubernetes.provision", skip(self, sandbox), fields( - otel.name = "kubernetes.create_sandbox", + otel.name = "kubernetes.provision", otel.status_code = tracing::field::Empty, sandbox.id = %sandbox.id, sandbox.name = %sandbox.name, @@ -1623,8 +1639,7 @@ impl KubernetesComputeDriver { let (agent_sandbox_api, kube_name, pod_name, namespace, stop_timeout) = self .patch_sandbox_operating_state(sandbox_id, false) .await?; - let legacy_pod_api = (agent_sandbox_api.resource.version == SANDBOX_VERSION_V1ALPHA1) - .then(|| Api::::namespaced(self.client.clone(), &namespace)); + let pod_api = Api::::namespaced(self.client.clone(), &namespace); let deadline = tokio::time::Instant::now() + stop_timeout; let mut poll_interval = STOP_INITIAL_POLL_INTERVAL; @@ -1649,17 +1664,18 @@ impl KubernetesComputeDriver { )) })? .map_err(KubernetesDriverError::from_kube)?; - if kubernetes_sandbox_has_stopped_condition(&object) { - return Ok(()); - } if let Some(error) = kubernetes_sandbox_stop_failure(&object) { return Err(KubernetesDriverError::Message(error)); } - if let Some(pod_api) = legacy_pod_api.as_ref() - && kubernetes_sandbox_pod_is_gone(pod_api, &pod_name, deadline) - .await - .map_err(KubernetesDriverError::Message)? - { + let pod_is_gone = kubernetes_sandbox_pod_is_gone(&pod_api, &pod_name, deadline) + .await + .map_err(KubernetesDriverError::Message)?; + let stop_is_complete = kubernetes_sandbox_stop_is_complete( + &agent_sandbox_api.resource.version, + &object, + pod_is_gone, + ); + if stop_is_complete { return Ok(()); } let now = tokio::time::Instant::now(); @@ -1949,7 +1965,7 @@ impl KubernetesComputeDriver { loop { tokio::select! { event = sandbox_stream.next() => match event { - Some(Event::Applied(obj)) => { + Some(Event::Apply(obj) | Event::InitApply(obj)) => { if let Ok((kube_name, sandbox)) = sandbox_from_object(&namespace, obj) { update_indexes(&mut sandbox_name_to_id, &mut agent_pod_to_id, &kube_name, &sandbox); let event = WatchSandboxesEvent { @@ -1962,7 +1978,7 @@ impl KubernetesComputeDriver { } } } - Some(Event::Deleted(obj)) => { + Some(Event::Delete(obj)) => { if is_openshell_managed(&obj) && let Ok(sandbox_id) = sandbox_id_from_object(&obj) { @@ -1977,21 +1993,7 @@ impl KubernetesComputeDriver { } } } - Some(Event::Restarted(objs)) => { - for obj in objs { - if let Ok((kube_name, sandbox)) = sandbox_from_object(&namespace, obj) { - update_indexes(&mut sandbox_name_to_id, &mut agent_pod_to_id, &kube_name, &sandbox); - let event = WatchSandboxesEvent { - payload: Some(watch_sandboxes_event::Payload::Sandbox( - WatchSandboxesSandboxEvent { sandbox: Some(sandbox) } - )), - }; - if tx.send(Ok(event)).await.is_err() { - return; - } - } - } - } + Some(Event::Init | Event::InitDone) => {} None => { let _ = tx.send(Err(KubernetesDriverError::Message( "sandbox watcher stream ended unexpectedly".to_string() @@ -2000,7 +2002,7 @@ impl KubernetesComputeDriver { } }, event = event_stream.next() => match event { - Some(Event::Applied(obj)) => { + Some(Event::Apply(obj)) => { if let Some((sandbox_id, event)) = map_kube_event_to_platform( &sandbox_name_to_id, &agent_pod_to_id, @@ -2016,8 +2018,8 @@ impl KubernetesComputeDriver { } } } - Some(Event::Deleted(_)) => {} - Some(Event::Restarted(_)) => { + Some(Event::Delete(_) | Event::InitApply(_) | Event::InitDone) => {} + Some(Event::Init) => { debug!(namespace = %namespace, "Kubernetes event watcher restarted"); } None => { @@ -2066,7 +2068,7 @@ where loop { tokio::select! { event = sandbox_stream.next() => match event { - Some(Event::Applied(obj)) => { + Some(Event::Apply(obj) | Event::InitApply(obj)) => { let ns = obj.metadata.namespace.clone() .unwrap_or_else(|| default_namespace.clone()); if let Ok((_kube_name, sandbox)) = sandbox_from_object(&ns, obj) { @@ -2080,7 +2082,7 @@ where } } } - Some(Event::Deleted(obj)) => { + Some(Event::Delete(obj)) => { if is_openshell_managed(&obj) && let Ok(sandbox_id) = sandbox_id_from_object(&obj) { @@ -2094,22 +2096,7 @@ where } } } - Some(Event::Restarted(objs)) => { - for obj in objs { - let ns = obj.metadata.namespace.clone() - .unwrap_or_else(|| default_namespace.clone()); - if let Ok((_kube_name, sandbox)) = sandbox_from_object(&ns, obj) { - let event = WatchSandboxesEvent { - payload: Some(watch_sandboxes_event::Payload::Sandbox( - WatchSandboxesSandboxEvent { sandbox: Some(sandbox) } - )), - }; - if tx.send(Ok(event)).await.is_err() { - return; - } - } - } - } + Some(Event::Init | Event::InitDone) => {} None => { let _ = tx.send(Err(KubernetesDriverError::Message( "sandbox watcher stream ended unexpectedly".to_string() @@ -2302,7 +2289,6 @@ fn managed_ssh_network_policy(namespace: &str, config: &KubernetesComputeConfig) }]), ..Default::default() }), - status: None, } } @@ -3864,9 +3850,14 @@ fn sandbox_template_to_k8s_with_validated_config( } apply_pod_driver_config(&mut spec, &driver_config.pod); - // Per-sandbox platform_config.host_users overrides the cluster-wide default. - let use_user_namespaces = platform_config_bool(template, "host_users") - .map_or(params.enable_user_namespaces, |host_users| !host_users); + // Per-sandbox portable intent overrides the cluster-wide default. This + // driver owns the Kubernetes-specific `hostUsers` translation. Accept the + // former platform_config encoding during rolling upgrades from gateways + // that predate the typed field. + let use_user_namespaces = template + .user_namespaces + .or_else(|| platform_config_bool(template, "host_users").map(|host_users| !host_users)) + .unwrap_or(params.enable_user_namespaces); if use_user_namespaces { spec.insert("hostUsers".to_string(), serde_json::json!(false)); @@ -4482,7 +4473,7 @@ fn platform_config_bool(template: &SandboxTemplate, key: &str) -> Option { let config = template.platform_config.as_ref()?; let value = config.fields.get(key)?; match value.kind.as_ref() { - Some(prost_types::value::Kind::BoolValue(b)) => Some(*b), + Some(prost_types::value::Kind::BoolValue(value)) => Some(*value), _ => None, } } @@ -4560,6 +4551,19 @@ fn kubernetes_sandbox_has_stopped_condition(obj: &DynamicObject) -> bool { }) } +fn kubernetes_sandbox_stop_is_complete( + api_version: &str, + obj: &DynamicObject, + pod_is_gone: bool, +) -> bool { + if api_version == SANDBOX_VERSION_V1ALPHA1 { + // v1alpha1 omits a usable stopped condition. + pod_is_gone + } else { + kubernetes_sandbox_has_stopped_condition(obj) && pod_is_gone + } +} + fn kubernetes_sandbox_stop_failure(obj: &DynamicObject) -> Option { obj.data .get("status")? @@ -4689,6 +4693,7 @@ fn spawn_namespace_label_watcher( let mut retry_attempt = 0; loop { let mut stream = watcher::watcher(ns_api.clone(), watcher_config.clone()).boxed(); + let mut relisted_names = BTreeSet::new(); loop { let event = tokio::select! { @@ -4701,37 +4706,9 @@ fn spawn_namespace_label_watcher( } }; match event { - Ok(Some(Event::Applied(ns))) => { + Ok(Some(event)) => { retry_attempt = 0; - if let Some(name) = ns.metadata.name.as_deref() - && allowlist.insert(name.to_string()) - { - info!(namespace = name, "operator namespace added to allowlist"); - } - } - Ok(Some(Event::Deleted(ns))) => { - retry_attempt = 0; - if let Some(name) = ns.metadata.name.as_deref() - && allowlist.remove(name) - { - info!( - namespace = name, - "operator namespace removed from allowlist" - ); - } - } - Ok(Some(Event::Restarted(namespaces))) => { - retry_attempt = 0; - let names: std::collections::BTreeSet = namespaces - .into_iter() - .filter_map(|ns| ns.metadata.name) - .collect(); - let count = names.len(); - allowlist.replace(names); - info!( - total = count, - "operator namespace allowlist replaced from full relist" - ); + apply_namespace_watch_event(&allowlist, &mut relisted_names, event); } Ok(None) => { warn!("operator namespace watcher stream ended unexpectedly"); @@ -4773,7 +4750,7 @@ fn namespace_watcher_retry_delay(attempt: u32, jitter_seed: u64) -> Duration { Duration::from_secs(base_secs + jitter_secs) } -fn load_namespace_file(path: &Path) -> Result, String> { +fn load_namespace_file(path: &Path) -> Result, String> { let contents = std::fs::read_to_string(path) .map_err(|e| format!("failed to read {}: {e}", path.display()))?; let names: Vec = serde_json::from_str(&contents) @@ -4781,6 +4758,47 @@ fn load_namespace_file(path: &Path) -> Result Ok(names.into_iter().collect()) } +fn apply_namespace_watch_event( + allowlist: &OperatorNamespaceAllowlist, + relisted_names: &mut BTreeSet, + event: Event, +) { + match event { + Event::Apply(ns) => { + if let Some(name) = ns.metadata.name + && allowlist.insert(name.clone()) + { + info!(namespace = name, "operator namespace added to allowlist"); + } + } + Event::Delete(ns) => { + if let Some(name) = ns.metadata.name + && allowlist.remove(&name) + { + info!( + namespace = name, + "operator namespace removed from allowlist" + ); + } + } + Event::Init => relisted_names.clear(), + Event::InitApply(ns) => { + if let Some(name) = ns.metadata.name { + relisted_names.insert(name); + } + } + Event::InitDone => { + // Readers must see a complete snapshot, including during interrupted relists. + let count = relisted_names.len(); + allowlist.replace(std::mem::take(relisted_names)); + info!( + total = count, + "operator namespace allowlist replaced from full relist" + ); + } + } +} + fn spawn_namespace_file_watcher( path: PathBuf, allowlist: OperatorNamespaceAllowlist, @@ -4926,12 +4944,13 @@ mod tests { use tracing::instrument::WithSubscriber as _; use tracing_subscriber::layer::SubscriberExt as _; - let _tracing_lock = crate::otel_tracing::test_lock().await; + let _tracing_lock = openshell_otel_test_support::tracing_test_lock().await; let exporter = InMemorySpanExporterBuilder::new().build(); let provider = SdkTracerProvider::builder() .with_simple_exporter(exporter.clone()) .build(); - let subscriber = tracing_subscriber::registry().with(crate::otel_tracing::layer(&provider)); + let subscriber = + tracing_subscriber::registry().with(crate::otel_tracing::TRACING.layer(&provider)); let driver = KubernetesComputeDriver::new_for_test(KubernetesComputeConfig::default()); driver @@ -4944,7 +4963,7 @@ mod tests { let spans = exporter.get_finished_spans().unwrap(); let span = spans .iter() - .find(|span| span.name == "kubernetes.create_sandbox") + .find(|span| span.name == "kubernetes.provision") .expect("create operation span"); assert!(matches!( span.status, @@ -4958,15 +4977,16 @@ mod tests { use opentelemetry_sdk::trace::{InMemorySpanExporterBuilder, SdkTracerProvider}; use tracing_subscriber::layer::SubscriberExt as _; - let _tracing_lock = crate::otel_tracing::test_lock().await; + let _tracing_lock = openshell_otel_test_support::tracing_test_lock().await; let exporter = InMemorySpanExporterBuilder::new().build(); let provider = SdkTracerProvider::builder() .with_simple_exporter(exporter) .build(); - let subscriber = tracing_subscriber::registry().with(crate::otel_tracing::layer(&provider)); + let subscriber = + tracing_subscriber::registry().with(crate::otel_tracing::TRACING.layer(&provider)); let annotations = tracing::subscriber::with_default(subscriber, || { - let span = tracing::info_span!("kubernetes.create_sandbox"); + let span = tracing::info_span!("kubernetes.provision"); let _entered = span.enter(); let mut annotations = BTreeMap::new(); add_trace_context_annotation(&mut annotations); @@ -5026,7 +5046,7 @@ mod tests { } #[tokio::test] - async fn sandbox_watcher_error_does_not_hide_restarted_recovery_event() { + async fn sandbox_watcher_error_does_not_hide_relist() { let recovered = DynamicObject { types: None, metadata: ObjectMeta { @@ -5037,22 +5057,22 @@ mod tests { }; let source = futures::stream::iter([ Err(expired_watch_error()), - Ok(Event::Restarted(vec![recovered])), + Ok(Event::Init), + Ok(Event::InitApply(recovered)), + Ok(Event::InitDone), ]); let mut stream = continue_on_watcher_errors(source, "sandbox-resource"); + assert!(matches!(stream.next().await, Some(Event::Init))); let event = stream .next() .await .expect("410 Expired must not terminate the watcher stream"); - let Event::Restarted(objects) = event else { - panic!("expected kube-runtime recovery to emit Restarted"); + let Event::InitApply(object) = event else { + panic!("expected kube-runtime recovery to emit InitApply"); }; - assert_eq!(objects.len(), 1); - assert_eq!( - objects[0].metadata.name.as_deref(), - Some("recovered-sandbox") - ); + assert_eq!(object.metadata.name.as_deref(), Some("recovered-sandbox")); + assert!(matches!(stream.next().await, Some(Event::InitDone))); assert!( stream.next().await.is_none(), "source closure must be preserved" @@ -5081,7 +5101,9 @@ mod tests { }; let source = futures::stream::iter([ Err(expired_watch_error()), - Ok(Event::Restarted(vec![recovered])), + Ok(Event::Init), + Ok(Event::InitApply(recovered)), + Ok(Event::InitDone), ]) .chain(futures::stream::pending()); let sandbox_stream = recovering_watcher_stream(source, "sandbox-resource").boxed(); @@ -5108,10 +5130,12 @@ mod tests { } #[tokio::test] - async fn kubernetes_event_watcher_error_does_not_hide_restarted_recovery_event() { + async fn kubernetes_event_watcher_error_does_not_hide_relist() { let source = futures::stream::iter([ Err(expired_watch_error()), - Ok(Event::Restarted(vec![KubeEventObj::default()])), + Ok(Event::Init), + Ok(Event::InitApply(KubeEventObj::default())), + Ok(Event::InitDone), ]); let mut stream = continue_on_watcher_errors(source, "kubernetes-event"); @@ -5119,16 +5143,65 @@ mod tests { .next() .await .expect("410 Expired must not terminate the watcher stream"); - let Event::Restarted(events) = event else { - panic!("expected kube-runtime recovery to emit Restarted"); - }; - assert_eq!(events.len(), 1); + assert!(matches!(event, Event::Init)); + assert!(matches!(stream.next().await, Some(Event::InitApply(_)))); + assert!(matches!(stream.next().await, Some(Event::InitDone))); assert!( stream.next().await.is_none(), "source closure must be preserved" ); } + #[test] + fn namespace_relist_replaces_only_completed_snapshots() { + let allowlist = OperatorNamespaceAllowlist::from_set(BTreeSet::from(["old".to_string()])); + let mut pending = BTreeSet::new(); + let namespace = |name: &str| Namespace { + metadata: ObjectMeta { + name: Some(name.to_string()), + ..Default::default() + }, + ..Default::default() + }; + let config = KubernetesComputeConfig { + workspace_mode: WorkspaceMode::Operator, + ..Default::default() + }; + + apply_namespace_watch_event(&allowlist, &mut pending, Event::Init); + apply_namespace_watch_event( + &allowlist, + &mut pending, + Event::InitApply(namespace("partial")), + ); + assert!(accepts_auth_namespace(&config, Some(&allowlist), "old")); + assert!(!accepts_auth_namespace( + &config, + Some(&allowlist), + "partial" + )); + + apply_namespace_watch_event(&allowlist, &mut pending, Event::Init); + apply_namespace_watch_event(&allowlist, &mut pending, Event::InitApply(namespace("new"))); + apply_namespace_watch_event(&allowlist, &mut pending, Event::InitDone); + assert!(accepts_auth_namespace(&config, Some(&allowlist), "new")); + assert!(!accepts_auth_namespace(&config, Some(&allowlist), "old")); + assert!(!accepts_auth_namespace( + &config, + Some(&allowlist), + "partial" + )); + + apply_namespace_watch_event(&allowlist, &mut pending, Event::Apply(namespace("live"))); + assert!(accepts_auth_namespace(&config, Some(&allowlist), "live")); + apply_namespace_watch_event(&allowlist, &mut pending, Event::Delete(namespace("live"))); + assert!(!accepts_auth_namespace(&config, Some(&allowlist), "live")); + + apply_namespace_watch_event(&allowlist, &mut pending, Event::Init); + apply_namespace_watch_event(&allowlist, &mut pending, Event::InitDone); + assert!(!accepts_auth_namespace(&config, Some(&allowlist), "new")); + } + fn authenticated_token_review(username: &str) -> TokenReviewStatus { TokenReviewStatus { authenticated: Some(true), @@ -5336,7 +5409,7 @@ mod tests { assert_eq!( kubernetes_sandbox_stop_timeout(&sandbox), - Duration::from_secs(60), + Duration::from_mins(1), "an omitted grace period uses the Kubernetes 30-second default" ); @@ -5392,6 +5465,43 @@ mod tests { assert!(kubernetes_sandbox_has_stopped_condition(&sandbox)); } + #[test] + fn beta_stop_requires_suspended_condition_and_deleted_pod() { + let resource = ApiResource::from_gvk(&GroupVersionKind::gvk( + SANDBOX_GROUP, + SANDBOX_VERSION_V1BETA1, + SANDBOX_KIND, + )); + let mut sandbox = DynamicObject::new("sandbox", &resource); + + assert!(!kubernetes_sandbox_stop_is_complete( + SANDBOX_VERSION_V1BETA1, + &sandbox, + true, + )); + + sandbox.data = serde_json::json!({ + "status": { + "conditions": [{"type": "Suspended", "status": "True"}] + } + }); + assert!(!kubernetes_sandbox_stop_is_complete( + SANDBOX_VERSION_V1BETA1, + &sandbox, + false, + )); + assert!(kubernetes_sandbox_stop_is_complete( + SANDBOX_VERSION_V1BETA1, + &sandbox, + true, + )); + assert!(kubernetes_sandbox_stop_is_complete( + SANDBOX_VERSION_V1ALPHA1, + &DynamicObject::new("sandbox", &resource), + true, + )); + } + #[test] fn stop_failure_only_rejects_terminal_suspension_condition() { let resource = ApiResource::from_gvk(&GroupVersionKind::gvk( @@ -7560,15 +7670,7 @@ mod tests { #[test] fn user_namespaces_per_sandbox_override_enables() { let template = SandboxTemplate { - platform_config: Some(Struct { - fields: std::iter::once(( - "host_users".to_string(), - Value { - kind: Some(Kind::BoolValue(false)), - }, - )) - .collect(), - }), + user_namespaces: Some(true), ..SandboxTemplate::default() }; @@ -7584,7 +7686,7 @@ mod tests { assert_eq!( pod_template["spec"]["hostUsers"], serde_json::json!(false), - "per-sandbox host_users: false must enable user namespaces" + "per-sandbox user namespace intent must set hostUsers: false" ); let caps = pod_template["spec"]["containers"][0]["securityContext"]["capabilities"]["add"] .as_array() @@ -7595,15 +7697,7 @@ mod tests { #[test] fn user_namespaces_per_sandbox_override_disables() { let template = SandboxTemplate { - platform_config: Some(Struct { - fields: std::iter::once(( - "host_users".to_string(), - Value { - kind: Some(Kind::BoolValue(true)), - }, - )) - .collect(), - }), + user_namespaces: Some(false), ..SandboxTemplate::default() }; @@ -7621,7 +7715,7 @@ mod tests { assert!( pod_template["spec"]["hostUsers"].is_null(), - "per-sandbox host_users: true must disable user namespaces even when cluster default is on" + "per-sandbox user namespace intent must override the cluster default" ); let caps = pod_template["spec"]["containers"][0]["securityContext"]["capabilities"]["add"] .as_array() @@ -7633,6 +7727,37 @@ mod tests { ); } + #[test] + fn user_namespaces_accepts_legacy_host_users_encoding() { + let template = SandboxTemplate { + platform_config: Some(Struct { + fields: std::iter::once(( + "host_users".to_string(), + Value { + kind: Some(Kind::BoolValue(false)), + }, + )) + .collect(), + }), + ..SandboxTemplate::default() + }; + + let params = SandboxPodParams::default(); + let pod_template = sandbox_template_to_k8s( + &template, + false, + &std::collections::HashMap::new(), + true, + ¶ms, + ); + + assert_eq!( + pod_template["spec"]["hostUsers"], + serde_json::json!(false), + "legacy host_users: false must still enable user namespaces" + ); + } + #[test] fn automount_service_account_token_is_disabled() { let pod_template = { @@ -7816,43 +7941,6 @@ mod tests { ); } - #[test] - fn platform_config_bool_extracts_value() { - let template = SandboxTemplate { - platform_config: Some(Struct { - fields: std::iter::once(( - "my_bool".to_string(), - Value { - kind: Some(Kind::BoolValue(true)), - }, - )) - .collect(), - }), - ..SandboxTemplate::default() - }; - - assert_eq!(platform_config_bool(&template, "my_bool"), Some(true)); - assert_eq!(platform_config_bool(&template, "missing"), None); - } - - #[test] - fn platform_config_bool_returns_none_for_non_bool() { - let template = SandboxTemplate { - platform_config: Some(Struct { - fields: std::iter::once(( - "a_string".to_string(), - Value { - kind: Some(Kind::StringValue("hello".to_string())), - }, - )) - .collect(), - }), - ..SandboxTemplate::default() - }; - - assert_eq!(platform_config_bool(&template, "a_string"), None); - } - #[test] fn log_level_propagates_as_env_var_to_sandbox_pod() { let spec = SandboxSpec { @@ -8682,4 +8770,19 @@ mod tests { namespace_watcher_retry_delay(3, 2) ); } + + #[tokio::test] + async fn capabilities_report_static_resource_support() { + let driver = KubernetesComputeDriver::new_for_test(KubernetesComputeConfig::default()); + let resources = driver + .capabilities() + .unwrap() + .resource_capabilities + .unwrap(); + assert!(resources.cpu.unwrap().limit_supported); + assert!(resources.memory.unwrap().limit_supported); + let gpu = resources.gpu.unwrap(); + assert!(gpu.default_selection_supported); + assert!(gpu.count_selection_supported); + } } diff --git a/crates/openshell-driver-kubernetes/src/grpc.rs b/crates/openshell-driver-kubernetes/src/grpc.rs index 027c350a2..095752d84 100644 --- a/crates/openshell-driver-kubernetes/src/grpc.rs +++ b/crates/openshell-driver-kubernetes/src/grpc.rs @@ -15,11 +15,8 @@ use openshell_core::proto::compute::v1::{ ValidateSandboxCreateResponse, WatchSandboxesEvent, WatchSandboxesRequest, compute_driver_server::ComputeDriver, }; -use std::future::Future; use std::pin::Pin; -use std::task::{Context, Poll}; use tonic::{Request, Response, Status}; -use tracing::Instrument as _; use crate::KubernetesComputeDriver; use crate::WorkspaceMode; @@ -27,38 +24,13 @@ use crate::WorkspaceMode; type ComputeDriverWatchStream = Pin> + Send + 'static>>; -struct TracedWatchStream { - inner: ComputeDriverWatchStream, - span: tracing::Span, -} - -impl Stream for TracedWatchStream { - type Item = Result; - - fn poll_next(mut self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll> { - let span = self.span.clone(); - let _entered = span.enter(); - let result = self.inner.as_mut().poll_next(cx); - match &result { - Poll::Ready(Some(Err(status))) => { - openshell_otel::mark_error(&self.span); - self.span - .record("rpc.grpc.status_code", status.code() as i32); - } - Poll::Ready(None) => { - self.span - .record("rpc.grpc.status_code", tonic::Code::Ok as i32); - } - Poll::Pending | Poll::Ready(Some(Ok(_))) => {} - } - result - } -} +#[cfg(test)] +type TracedWatchStream = openshell_otel::TracedGrpcStream; #[derive(Debug, Clone)] pub struct ComputeDriverService { driver: KubernetesComputeDriver, - trace_in_process_rpc: bool, + rpc_tracer: openshell_otel::InProcessRpcTracer, } impl ComputeDriverService { @@ -66,7 +38,7 @@ impl ComputeDriverService { pub fn new(driver: KubernetesComputeDriver) -> Self { Self { driver, - trace_in_process_rpc: false, + rpc_tracer: openshell_otel::InProcessRpcTracer::disabled(), } } @@ -74,51 +46,9 @@ impl ComputeDriverService { pub fn new_in_process(driver: KubernetesComputeDriver) -> Self { Self { driver, - trace_in_process_rpc: true, + rpc_tracer: openshell_otel::InProcessRpcTracer::enabled(), } } - - fn in_process_rpc_span( - &self, - operation: &'static str, - method: &'static str, - ) -> Option { - self.trace_in_process_rpc.then(|| { - tracing::info_span!( - target: "openshell_driver_kubernetes::otel_tracing", - "driver_rpc", - otel.name = operation, - otel.kind = "server", - otel.status_code = tracing::field::Empty, - rpc.system = "grpc", - rpc.service = "openshell.compute.v1.ComputeDriver", - rpc.method = method, - rpc.grpc.status_code = tracing::field::Empty, - ) - }) - } - - async fn trace_rpc( - &self, - operation: &'static str, - method: &'static str, - future: impl Future>, - ) -> Result { - let Some(span) = self.in_process_rpc_span(operation, method) else { - return future.await; - }; - let result = future.instrument(span.clone()).await; - match &result { - Ok(_) => { - span.record("rpc.grpc.status_code", tonic::Code::Ok as i32); - } - Err(status) => { - openshell_otel::mark_error(&span); - span.record("rpc.grpc.status_code", status.code() as i32); - } - } - result - } } #[tonic::async_trait] @@ -127,170 +57,182 @@ impl ComputeDriver for ComputeDriverService { &self, request: Request, ) -> Result, Status> { - let credential = request.into_inner().credential; - if credential.is_empty() { - return Err(Status::invalid_argument("credential is required")); - } - let sandbox_id = self.driver.authenticate_sandbox(&credential).await?; - Ok(Response::new(AuthenticateSandboxResponse { sandbox_id })) + self.rpc_tracer + .trace(openshell_otel::rpc::AUTHENTICATE_SANDBOX, async { + let credential = request.into_inner().credential; + if credential.is_empty() { + return Err(Status::invalid_argument("credential is required")); + } + let sandbox_id = self.driver.authenticate_sandbox(&credential).await?; + Ok(Response::new(AuthenticateSandboxResponse { sandbox_id })) + }) + .await } async fn get_capabilities( &self, _request: Request, ) -> Result, Status> { - self.trace_rpc("driver.get_capabilities", "get_capabilities", async { - self.driver - .capabilities() - .map(Response::new) - .map_err(Status::internal) - }) - .await + self.rpc_tracer + .trace(openshell_otel::rpc::GET_CAPABILITIES, async { + self.driver + .capabilities() + .map(Response::new) + .map_err(Status::internal) + }) + .await } async fn get_gateway_listener_requirements( &self, _request: Request, ) -> Result, Status> { - self.trace_rpc( - "driver.get_gateway_listener_requirements", - "get_gateway_listener_requirements", - async { - Ok(Response::new(GetGatewayListenerRequirementsResponse { - requirements: Vec::new(), - })) - }, - ) - .await + self.rpc_tracer + .trace( + openshell_otel::rpc::GET_GATEWAY_LISTENER_REQUIREMENTS, + async { + Ok(Response::new(GetGatewayListenerRequirementsResponse { + requirements: Vec::new(), + })) + }, + ) + .await } async fn validate_sandbox_create( &self, request: Request, ) -> Result, Status> { - self.trace_rpc( - "driver.validate_sandbox_create", - "validate_sandbox_create", - async { + self.rpc_tracer + .trace(openshell_otel::rpc::VALIDATE_SANDBOX_CREATE, async { let sandbox = request .into_inner() .sandbox .ok_or_else(|| Status::invalid_argument("sandbox is required"))?; self.driver.validate_sandbox_create(&sandbox).await?; Ok(Response::new(ValidateSandboxCreateResponse {})) - }, - ) - .await + }) + .await } async fn get_sandbox( &self, request: Request, ) -> Result, Status> { - self.trace_rpc("driver.get_sandbox", "get_sandbox", async { - let request = request.into_inner(); - if request.sandbox_id.is_empty() { - return Err(Status::invalid_argument("sandbox_id is required")); - } - let sandbox = self - .driver - .get_sandbox(&request.sandbox_id) - .await - .map_err(Status::internal)? - .ok_or_else(|| Status::not_found("sandbox not found"))?; - Ok(Response::new(GetSandboxResponse { - sandbox: Some(sandbox), - })) - }) - .await + self.rpc_tracer + .trace(openshell_otel::rpc::GET_SANDBOX, async { + let request = request.into_inner(); + if request.sandbox_id.is_empty() { + return Err(Status::invalid_argument("sandbox_id is required")); + } + let sandbox = self + .driver + .get_sandbox(&request.sandbox_id) + .await + .map_err(Status::internal)? + .ok_or_else(|| Status::not_found("sandbox not found"))?; + Ok(Response::new(GetSandboxResponse { + sandbox: Some(sandbox), + })) + }) + .await } async fn list_sandboxes( &self, _request: Request, ) -> Result, Status> { - self.trace_rpc("driver.list_sandboxes", "list_sandboxes", async { - let sandboxes = self - .driver - .list_sandboxes() - .await - .map_err(Status::internal)?; - Ok(Response::new(ListSandboxesResponse { sandboxes })) - }) - .await + self.rpc_tracer + .trace(openshell_otel::rpc::LIST_SANDBOXES, async { + let sandboxes = self + .driver + .list_sandboxes() + .await + .map_err(Status::internal)?; + Ok(Response::new(ListSandboxesResponse { sandboxes })) + }) + .await } async fn create_sandbox( &self, request: Request, ) -> Result, Status> { - self.trace_rpc("driver.create_sandbox", "create_sandbox", async { - let sandbox = request - .into_inner() - .sandbox - .ok_or_else(|| Status::invalid_argument("sandbox is required"))?; - self.driver - .create_sandbox(&sandbox) - .await - .map_err(|e| Status::from(openshell_core::ComputeDriverError::from(e)))?; - Ok(Response::new(CreateSandboxResponse {})) - }) - .await + self.rpc_tracer + .trace(openshell_otel::rpc::CREATE_SANDBOX, async { + let sandbox = request + .into_inner() + .sandbox + .ok_or_else(|| Status::invalid_argument("sandbox is required"))?; + self.driver + .create_sandbox(&sandbox) + .await + .map_err(|e| Status::from(openshell_core::ComputeDriverError::from(e)))?; + Ok(Response::new(CreateSandboxResponse {})) + }) + .await } async fn stop_sandbox( &self, request: Request, ) -> Result, Status> { - self.trace_rpc("driver.stop_sandbox", "stop_sandbox", async { - let request = request.into_inner(); - if request.sandbox_id.is_empty() { - return Err(Status::invalid_argument("sandbox_id is required")); - } - self.driver - .stop_sandbox(&request.sandbox_id) - .await - .map_err(|error| Status::from(openshell_core::ComputeDriverError::from(error)))?; - Ok(Response::new(StopSandboxResponse {})) - }) - .await + self.rpc_tracer + .trace(openshell_otel::rpc::STOP_SANDBOX, async { + let request = request.into_inner(); + if request.sandbox_id.is_empty() { + return Err(Status::invalid_argument("sandbox_id is required")); + } + self.driver + .stop_sandbox(&request.sandbox_id) + .await + .map_err(|error| { + Status::from(openshell_core::ComputeDriverError::from(error)) + })?; + Ok(Response::new(StopSandboxResponse {})) + }) + .await } async fn start_sandbox( &self, request: Request, ) -> Result, Status> { - self.trace_rpc("driver.start_sandbox", "start_sandbox", async { - let request = request.into_inner(); - if request.sandbox_id.is_empty() { - return Err(Status::invalid_argument("sandbox_id is required")); - } - self.driver - .start_sandbox(&request.sandbox_id) - .await - .map_err(|error| Status::from(openshell_core::ComputeDriverError::from(error)))?; - Ok(Response::new(StartSandboxResponse {})) - }) - .await + self.rpc_tracer + .trace(openshell_otel::rpc::START_SANDBOX, async { + let request = request.into_inner(); + if request.sandbox_id.is_empty() { + return Err(Status::invalid_argument("sandbox_id is required")); + } + self.driver + .start_sandbox(&request.sandbox_id) + .await + .map_err(|error| { + Status::from(openshell_core::ComputeDriverError::from(error)) + })?; + Ok(Response::new(StartSandboxResponse {})) + }) + .await } async fn delete_sandbox( &self, request: Request, ) -> Result, Status> { - self.trace_rpc("driver.delete_sandbox", "delete_sandbox", async { - let request = request.into_inner(); - if request.sandbox_id.is_empty() { - return Err(Status::invalid_argument("sandbox_id is required")); - } - let deleted = self - .driver - .delete_sandbox(&request.sandbox_id) - .await - .map_err(Status::internal)?; - Ok(Response::new(DeleteSandboxResponse { deleted })) - }) - .await + self.rpc_tracer + .trace(openshell_otel::rpc::DELETE_SANDBOX, async { + let request = request.into_inner(); + if request.sandbox_id.is_empty() { + return Err(Status::invalid_argument("sandbox_id is required")); + } + let deleted = self + .driver + .delete_sandbox(&request.sandbox_id) + .await + .map_err(Status::internal)?; + Ok(Response::new(DeleteSandboxResponse { deleted })) + }) + .await } type WatchSandboxesStream = ComputeDriverWatchStream; @@ -308,81 +250,74 @@ impl ComputeDriver for ComputeDriverService { let stream = stream.map(|item| item.map_err(|err| Status::internal(err.to_string()))); Ok::(Box::pin(stream)) }; - let Some(span) = self.in_process_rpc_span("driver.watch_sandboxes", "watch_sandboxes") - else { - return create_stream.await.map(Response::new); - }; - match create_stream.instrument(span.clone()).await { - Ok(stream) => Ok(Response::new(Box::pin(TracedWatchStream { - inner: stream, - span, - }))), - Err(status) => { - openshell_otel::mark_error(&span); - span.record("rpc.grpc.status_code", status.code() as i32); - Err(status) - } - } + self.rpc_tracer + .trace_stream(openshell_otel::rpc::WATCH_SANDBOXES, create_stream) + .await + .map(Response::new) } async fn ensure_workspace( &self, request: Request, ) -> Result, Status> { - self.trace_rpc("driver.ensure_workspace", "ensure_workspace", async { - let workspace = request.into_inner().workspace; - if workspace.is_empty() { - return Err(Status::invalid_argument("workspace is required")); - } - self.driver - .validate_workspace_namespace(&workspace) - .map_err(|error| Status::from(openshell_core::ComputeDriverError::from(error)))?; - match self.driver.workspace_mode() { - WorkspaceMode::Managed => { - self.driver - .ensure_namespace(&workspace) - .await - .map_err(|e| Status::internal(e.to_string()))?; + self.rpc_tracer + .trace(openshell_otel::rpc::ENSURE_WORKSPACE, async { + let workspace = request.into_inner().workspace; + if workspace.is_empty() { + return Err(Status::invalid_argument("workspace is required")); } - WorkspaceMode::Operator => { - if let Some(allowlist) = self.driver.operator_allowlist() - && !allowlist.contains(&workspace) - { - return Err(Status::permission_denied(format!( - "workspace '{workspace}' is not in the operator namespace allowlist" - ))); + self.driver + .validate_workspace_namespace(&workspace) + .map_err(|error| { + Status::from(openshell_core::ComputeDriverError::from(error)) + })?; + match self.driver.workspace_mode() { + WorkspaceMode::Managed => { + self.driver + .ensure_namespace(&workspace) + .await + .map_err(|e| Status::internal(e.to_string()))?; } + WorkspaceMode::Operator => { + if let Some(allowlist) = self.driver.operator_allowlist() + && !allowlist.contains(&workspace) + { + return Err(Status::permission_denied(format!( + "workspace '{workspace}' is not in the operator namespace allowlist" + ))); + } + } + WorkspaceMode::Shared => {} } - WorkspaceMode::Shared => {} - } - Ok(Response::new(EnsureWorkspaceResponse {})) - }) - .await + Ok(Response::new(EnsureWorkspaceResponse {})) + }) + .await } async fn delete_workspace( &self, request: Request, ) -> Result, Status> { - self.trace_rpc("driver.delete_workspace", "delete_workspace", async { - let workspace = request.into_inner().workspace; - if workspace.is_empty() { - return Err(Status::invalid_argument("workspace is required")); - } - if workspace_delete_requires_namespace_access(self.driver.workspace_mode()) { - self.driver - .validate_workspace_namespace(&workspace) - .map_err(|error| { - Status::from(openshell_core::ComputeDriverError::from(error)) - })?; - self.driver - .delete_namespace(&workspace) - .await - .map_err(|e| Status::internal(e.to_string()))?; - } - Ok(Response::new(DeleteWorkspaceResponse {})) - }) - .await + self.rpc_tracer + .trace(openshell_otel::rpc::DELETE_WORKSPACE, async { + let workspace = request.into_inner().workspace; + if workspace.is_empty() { + return Err(Status::invalid_argument("workspace is required")); + } + if workspace_delete_requires_namespace_access(self.driver.workspace_mode()) { + self.driver + .validate_workspace_namespace(&workspace) + .map_err(|error| { + Status::from(openshell_core::ComputeDriverError::from(error)) + })?; + self.driver + .delete_namespace(&workspace) + .await + .map_err(|e| Status::internal(e.to_string()))?; + } + Ok(Response::new(DeleteWorkspaceResponse {})) + }) + .await } } @@ -402,10 +337,10 @@ mod tests { use super::*; use crate::KubernetesComputeConfig; use opentelemetry_sdk::trace::{InMemorySpanExporterBuilder, SdkTracerProvider}; - use tracing::instrument::WithSubscriber as _; + use tracing::{Instrument as _, instrument::WithSubscriber as _}; use tracing_subscriber::layer::SubscriberExt as _; - let _tracing_lock = crate::otel_tracing::test_lock().await; + let _tracing_lock = openshell_otel_test_support::tracing_test_lock().await; let gateway_exporter = InMemorySpanExporterBuilder::new().build(); let gateway_provider = SdkTracerProvider::builder() .with_simple_exporter(gateway_exporter.clone()) @@ -415,12 +350,12 @@ mod tests { .with_simple_exporter(driver_exporter.clone()) .build(); let subscriber = tracing_subscriber::registry() - .with(openshell_otel::layer_excluding_target_prefix( + .with(openshell_otel::layer_excluding_target_prefixes( &gateway_provider, "gateway-test", - Some(crate::otel_tracing::IN_PROCESS_TARGET_PREFIX), + crate::otel_tracing::TRACING.in_process_targets(), )) - .with(crate::otel_tracing::in_process_layer(&driver_provider)); + .with(crate::otel_tracing::TRACING.in_process_layer(&driver_provider)); let service = ComputeDriverService::new_in_process(KubernetesComputeDriver::new_for_test( KubernetesComputeConfig::default(), )); @@ -429,7 +364,7 @@ mod tests { let gateway_span = tracing::info_span!( target: "openshell_server::compute", "driver", - otel.name = "driver.get_capabilities", + otel.name = "openshell.compute.v1.ComputeDriver/GetCapabilities", otel.kind = "client" ); ComputeDriver::get_capabilities(&service, Request::new(GetCapabilitiesRequest {})) @@ -452,11 +387,11 @@ mod tests { let driver_spans = driver_exporter.get_finished_spans().unwrap(); let client = gateway_spans .iter() - .find(|span| span.name == "driver.get_capabilities") + .find(|span| span.name == "openshell.compute.v1.ComputeDriver/GetCapabilities") .expect("gateway client span"); let server = driver_spans .iter() - .find(|span| span.name == "driver.get_capabilities") + .find(|span| span.name == "openshell.compute.v1.ComputeDriver/GetCapabilities") .expect("in-process server span"); assert_eq!( server.span_context.trace_id(), @@ -465,12 +400,24 @@ mod tests { assert_eq!(server.parent_span_id, client.span_context.span_id()); assert_eq!(server.span_kind, opentelemetry::trace::SpanKind::Server); assert!(server.attributes.iter().any(|attribute| { - attribute.key.as_str() == "rpc.grpc.status_code" - && attribute.value.to_string() == (tonic::Code::Ok as i32).to_string() + attribute.key.as_str() == "rpc.method" + && attribute.value.to_string() + == "openshell.compute.v1.ComputeDriver/GetCapabilities" + })); + assert!( + server + .attributes + .iter() + .all(|attribute| attribute.key.as_str() != "rpc.service"), + "the current RPC semantic conventions integrate the service into rpc.method" + ); + assert!(server.attributes.iter().any(|attribute| { + attribute.key.as_str() == "rpc.response.status_code" + && attribute.value.to_string() == "OK" })); let failed = driver_spans .iter() - .find(|span| span.name == "driver.validate_sandbox_create") + .find(|span| span.name == "openshell.compute.v1.ComputeDriver/ValidateSandboxCreate") .expect("failed in-process server span"); assert!(matches!( failed.status, @@ -487,27 +434,27 @@ mod tests { use tracing::instrument::WithSubscriber as _; use tracing_subscriber::layer::SubscriberExt as _; - let _tracing_lock = crate::otel_tracing::test_lock().await; + let _tracing_lock = openshell_otel_test_support::tracing_test_lock().await; let exporter = InMemorySpanExporterBuilder::new().build(); let provider = SdkTracerProvider::builder() .with_simple_exporter(exporter.clone()) .build(); - let subscriber = - tracing_subscriber::registry().with(crate::otel_tracing::in_process_layer(&provider)); + let subscriber = tracing_subscriber::registry() + .with(crate::otel_tracing::TRACING.in_process_layer(&provider)); async { let span = tracing::info_span!( - target: "openshell_driver_kubernetes::otel_tracing", + target: crate::otel_tracing::TRACING.in_process_target(), "driver_rpc", - otel.name = "driver.watch_sandboxes", + otel.name = "openshell.compute.v1.ComputeDriver/WatchSandboxes", otel.kind = "server", otel.status_code = tracing::field::Empty, - rpc.grpc.status_code = tracing::field::Empty, + rpc.response.status_code = tracing::field::Empty, ); let inner: ComputeDriverWatchStream = Box::pin(futures::stream::iter([Err( Status::internal("watch failed"), )])); - let mut stream = TracedWatchStream { inner, span }; + let mut stream = TracedWatchStream::new(inner, span); provider.force_flush().unwrap(); assert!( @@ -528,7 +475,7 @@ mod tests { let spans = exporter.get_finished_spans().unwrap(); let span = spans .iter() - .find(|span| span.name == "driver.watch_sandboxes") + .find(|span| span.name == "openshell.compute.v1.ComputeDriver/WatchSandboxes") .expect("watch server span should be exported when the stream ends"); assert!(matches!( span.status, @@ -544,25 +491,25 @@ mod tests { use tracing::instrument::WithSubscriber as _; use tracing_subscriber::layer::SubscriberExt as _; - let _tracing_lock = crate::otel_tracing::test_lock().await; + let _tracing_lock = openshell_otel_test_support::tracing_test_lock().await; let exporter = InMemorySpanExporterBuilder::new().build(); let provider = SdkTracerProvider::builder() .with_simple_exporter(exporter.clone()) .build(); - let subscriber = - tracing_subscriber::registry().with(crate::otel_tracing::in_process_layer(&provider)); + let subscriber = tracing_subscriber::registry() + .with(crate::otel_tracing::TRACING.in_process_layer(&provider)); async { let span = tracing::info_span!( - target: "openshell_driver_kubernetes::otel_tracing", + target: crate::otel_tracing::TRACING.in_process_target(), "driver_rpc", - otel.name = "driver.watch_sandboxes", + otel.name = "openshell.compute.v1.ComputeDriver/WatchSandboxes", otel.kind = "server", otel.status_code = tracing::field::Empty, - rpc.grpc.status_code = tracing::field::Empty, + rpc.response.status_code = tracing::field::Empty, ); let inner: ComputeDriverWatchStream = Box::pin(futures::stream::empty()); - let mut stream = TracedWatchStream { inner, span }; + let mut stream = TracedWatchStream::new(inner, span); assert!(stream.next().await.is_none()); drop(stream); @@ -574,15 +521,63 @@ mod tests { let spans = exporter.get_finished_spans().unwrap(); let span = spans .iter() - .find(|span| span.name == "driver.watch_sandboxes") + .find(|span| span.name == "openshell.compute.v1.ComputeDriver/WatchSandboxes") .expect("watch server span should be exported when the stream completes"); assert!(span.attributes.iter().any(|attribute| { - attribute.key.as_str() == "rpc.grpc.status_code" - && attribute.value.to_string() == (tonic::Code::Ok as i32).to_string() + attribute.key.as_str() == "rpc.response.status_code" + && attribute.value.to_string() == "OK" })); provider.shutdown().unwrap(); } + #[tokio::test] + async fn tracing_in_process_stream_leaves_status_unset_when_dropped() { + use super::*; + use opentelemetry_sdk::trace::{InMemorySpanExporterBuilder, SdkTracerProvider}; + use tracing::instrument::WithSubscriber as _; + use tracing_subscriber::layer::SubscriberExt as _; + + let _tracing_lock = openshell_otel_test_support::tracing_test_lock().await; + let exporter = InMemorySpanExporterBuilder::new().build(); + let provider = SdkTracerProvider::builder() + .with_simple_exporter(exporter.clone()) + .build(); + let subscriber = tracing_subscriber::registry() + .with(crate::otel_tracing::TRACING.in_process_layer(&provider)); + + async { + let span = tracing::info_span!( + target: crate::otel_tracing::TRACING.in_process_target(), + "driver_rpc", + otel.name = "openshell.compute.v1.ComputeDriver/WatchSandboxes", + otel.kind = "server", + otel.status_code = tracing::field::Empty, + rpc.response.status_code = tracing::field::Empty, + error.type = tracing::field::Empty, + ); + let inner: ComputeDriverWatchStream = Box::pin(futures::stream::pending()); + let stream = TracedWatchStream::new(inner, span); + + drop(stream); + } + .with_subscriber(subscriber) + .await; + provider.force_flush().unwrap(); + + let spans = exporter.get_finished_spans().unwrap(); + let span = spans + .iter() + .find(|span| span.name == "openshell.compute.v1.ComputeDriver/WatchSandboxes") + .expect("watch server span should be exported when the stream is dropped"); + assert!(matches!(span.status, opentelemetry::trace::Status::Unset)); + assert!( + span.attributes + .iter() + .all(|attribute| attribute.key.as_str() != "rpc.response.status_code") + ); + provider.shutdown().unwrap(); + } + #[test] fn precondition_driver_errors_map_to_failed_precondition_status() { let status: Status = ComputeDriverError::from(KubernetesDriverError::Precondition( diff --git a/crates/openshell-driver-kubernetes/src/lib.rs b/crates/openshell-driver-kubernetes/src/lib.rs index edc0a3e97..28d3c77a7 100644 --- a/crates/openshell-driver-kubernetes/src/lib.rs +++ b/crates/openshell-driver-kubernetes/src/lib.rs @@ -14,4 +14,4 @@ pub use config::{ }; pub use driver::{KubernetesComputeDriver, KubernetesDriverError}; pub use grpc::ComputeDriverService; -pub use openshell_core::OperatorNamespaceAllowlist; +pub use openshell_core::DynamicStringAllowlist as OperatorNamespaceAllowlist; diff --git a/crates/openshell-driver-kubernetes/src/main.rs b/crates/openshell-driver-kubernetes/src/main.rs index 7690ccee8..9b11f5da2 100644 --- a/crates/openshell-driver-kubernetes/src/main.rs +++ b/crates/openshell-driver-kubernetes/src/main.rs @@ -7,12 +7,9 @@ use std::collections::BTreeMap; use std::net::SocketAddr; use std::path::PathBuf; use tracing::info; -use tracing_subscriber::EnvFilter; -use tracing_subscriber::prelude::*; use openshell_core::VERSION; use openshell_core::proto::compute::v1::compute_driver_server::ComputeDriverServer; -use openshell_driver_kubernetes::otel_tracing::compute_driver_rpc_layer; use openshell_driver_kubernetes::{ AppArmorProfile, ComputeDriverService, DEFAULT_GATEWAY_ID, DEFAULT_PROXY_UID, DEFAULT_SANDBOX_SERVICE_ACCOUNT_NAME, KubernetesComputeConfig, KubernetesComputeDriver, @@ -218,24 +215,15 @@ async fn shutdown_signal() { #[tokio::main] async fn main() -> Result<()> { let args = Args::parse(); - let (tracer_provider, setup_error) = openshell_driver_kubernetes::otel_tracing::provider_for( - args.otlp_endpoint.as_deref(), - args.gateway_name.as_deref(), + let _tracing = openshell_otel::install_driver_tracing( + openshell_driver_kubernetes::otel_tracing::TRACING, + openshell_otel::DriverTracingConfig { + endpoint: args.otlp_endpoint.as_deref(), + gateway_name: args.gateway_name.as_deref(), + service_version: VERSION, + log_level: &args.log_level, + }, ); - tracing_subscriber::registry() - .with(EnvFilter::try_from_default_env().unwrap_or_else(|_| EnvFilter::new(&args.log_level))) - .with(tracing_subscriber::fmt::layer()) - .with( - tracer_provider - .as_ref() - .map(openshell_driver_kubernetes::otel_tracing::layer), - ) - .init(); - if let Some(error) = setup_error { - tracing::error!(%error, "OTLP exporting could not be started"); - } else if let Some(endpoint) = &args.otlp_endpoint { - info!(endpoint, "OTLP exporting enabled"); - } let managed_ssh_gateway_pod_selector = args .managed_ssh_gateway_pod_selector @@ -317,14 +305,14 @@ async fn main() -> Result<()> { shutdown_signal().await; let _ = shutdown_tx.send(true); }; - let result = if let Some(socket_path) = args.bind_socket { + if let Some(socket_path) = args.bind_socket { let listener = openshell_core::external_driver_socket::bind_private(&socket_path) .map_err(|err| miette::miette!("{err}"))?; let _cleanup = openshell_core::external_driver_socket::SocketCleanup::new(socket_path.clone()); info!(socket = %socket_path.display(), "Starting Kubernetes compute driver"); tonic::transport::Server::builder() - .layer(compute_driver_rpc_layer()) + .layer(openshell_otel::compute_driver_rpc_layer()) .add_service(service) .serve_with_incoming_shutdown( openshell_core::external_driver_socket::SameUidUnixIncoming::new(listener), @@ -335,18 +323,12 @@ async fn main() -> Result<()> { } else { info!(address = %args.bind_address, "Starting Kubernetes compute driver"); tonic::transport::Server::builder() - .layer(compute_driver_rpc_layer()) + .layer(openshell_otel::compute_driver_rpc_layer()) .add_service(service) .serve_with_shutdown(args.bind_address, shutdown) .await .into_diagnostic() - }; - if let Some(provider) = &tracer_provider - && let Err(error) = provider.shutdown() - { - tracing::warn!(%error, "OTLP tracer provider shutdown failed"); } - result } #[cfg(test)] diff --git a/crates/openshell-driver-kubernetes/src/otel_tracing.rs b/crates/openshell-driver-kubernetes/src/otel_tracing.rs index 02501651f..37e69c65b 100644 --- a/crates/openshell-driver-kubernetes/src/otel_tracing.rs +++ b/crates/openshell-driver-kubernetes/src/otel_tracing.rs @@ -1,125 +1,19 @@ // SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 -//! OpenTelemetry trace exporting for the Kubernetes compute driver. +//! OpenTelemetry tracing identity for the Kubernetes compute driver. -use openshell_otel::{OtlpTraceConfig, SdkTracerProvider, ServiceName, SetupError}; -pub use openshell_otel::{compute_driver_rpc_layer, compute_driver_rpc_operation}; -use tracing::Subscriber; -use tracing_subscriber::registry::LookupSpan; - -const SERVICE_NAME: &str = "openshell-driver-kubernetes"; -const INSTRUMENTATION_SCOPE: &str = "openshell-driver-kubernetes"; -pub const IN_PROCESS_TARGET_PREFIX: &str = "openshell_driver_kubernetes"; - -/// Build a tracer provider for the configured OTLP/gRPC endpoint and gateway. -#[must_use] -pub fn provider_for( - endpoint: Option<&str>, - gateway_name: Option<&str>, -) -> (Option, Option) { - openshell_otel::provider_for(endpoint.map(|endpoint| { - OtlpTraceConfig { - endpoint, - service_name: ServiceName::Fixed(SERVICE_NAME), - service_version: Some(openshell_core::VERSION), - resource_attributes: gateway_name - .map(str::trim) - .filter(|name| !name.is_empty()) - .map(|name| { - vec![opentelemetry::KeyValue::new( - "openshell.gateway.name", - name.to_string(), - )] - }) - .unwrap_or_default(), - } - })) -} - -pub fn layer(provider: &SdkTracerProvider) -> openshell_otel::OtlpLayer -where - S: Subscriber + for<'span> LookupSpan<'span>, -{ - openshell_otel::layer(provider, INSTRUMENTATION_SCOPE) -} - -pub fn in_process_layer(provider: &SdkTracerProvider) -> openshell_otel::TargetOtlpLayer -where - S: Subscriber + for<'span> LookupSpan<'span>, -{ - openshell_otel::layer_for_target_prefix( - provider, - INSTRUMENTATION_SCOPE, - IN_PROCESS_TARGET_PREFIX, - ) -} - -#[cfg(test)] -pub(crate) async fn test_lock() -> tokio::sync::MutexGuard<'static, ()> { - static LOCK: tokio::sync::Mutex<()> = tokio::sync::Mutex::const_new(()); - static INITIALIZED: std::sync::LazyLock<()> = std::sync::LazyLock::new(|| { - tracing::subscriber::set_global_default(tracing_subscriber::registry()) - .expect("test tracing subscriber installs once"); - }); - - let guard = LOCK.lock().await; - std::sync::LazyLock::force(&INITIALIZED); - guard -} +pub const TRACING: openshell_otel::ComputeDriverTracing = openshell_otel::compute_driver_tracing!(); #[cfg(test)] mod tests { - use openshell_otel_test_support::OtlpTestServer; - use tracing_subscriber::layer::SubscriberExt as _; - - #[test] - fn compute_driver_rpc_names_are_explicitly_mapped_and_schema_bounded() { - assert_eq!( - super::compute_driver_rpc_operation( - "/openshell.compute.v1.ComputeDriver/GetCapabilities" - ), - ("driver.get_capabilities", "get_capabilities") - ); - assert_eq!( - super::compute_driver_rpc_operation( - "/openshell.compute.v1.ComputeDriver/AttackerControlled12345" - ), - ("driver.unknown", "unknown") - ); - } - #[tokio::test(flavor = "multi_thread", worker_threads = 2)] - async fn tracing_kubernetes_driver_spans_reach_otlp_collector_with_resource_identity() { - let _tracing_lock = super::test_lock().await; - let collector = OtlpTestServer::start().await; - let (provider, error) = - super::provider_for(Some(collector.endpoint()), Some("kubernetes-dev")); - assert!(error.is_none()); - let provider = provider.expect("provider"); - let subscriber = tracing_subscriber::registry().with(super::layer(&provider)); - tracing::subscriber::with_default(subscriber, || { - let span = tracing::info_span!("kubernetes.create_sandbox", sandbox.id = "sb-otlp"); - drop(span.enter()); - drop(span); - }); - provider.force_flush().unwrap(); - collector.wait_for_export().await; - provider.shutdown().unwrap(); - let received = collector.shutdown().await; - - assert!( - received - .spans - .iter() - .any(|span| span.name == "kubernetes.create_sandbox") - ); - assert_eq!(received.gateway_names, ["kubernetes-dev"]); - assert!( - received - .service_names - .iter() - .any(|name| name == super::SERVICE_NAME) - ); + async fn driver_spans_reach_otlp_collector_with_resource_identity() { + openshell_otel_test_support::assert_compute_driver_tracing( + super::TRACING, + openshell_core::VERSION, + "kubernetes.provision", + ) + .await; } } diff --git a/crates/openshell-driver-mxc/README.md b/crates/openshell-driver-mxc/README.md index 6496dfe28..bd0b51f48 100644 --- a/crates/openshell-driver-mxc/README.md +++ b/crates/openshell-driver-mxc/README.md @@ -4,8 +4,11 @@ OpenShell compute driver backed by **Microsoft MXC** (`wxc-exec`) on Windows. ## Design -This driver implements the gateway's `ComputeDriver` contract as an in-process -library linked into `openshell-gateway`. `process_container` launches a one-shot +This driver implements the gateway's ordinary in-process `ComputeDriver` +contract and is linked into `openshell-gateway`. It sets +`driver_reports_runtime_readiness`, so the gateway accepts driver-reported +readiness without a supervisor session. The canonical create-time +`SandboxPolicy` is carried by `DriverSandboxSpec.policy`. `process_container` launches a one-shot AppContainer and is the default. The opt-in `isolation_session` backend uses the state-aware `provision` → `start` → `exec` → `stop` → `deprovision` lifecycle. The driver launches and monitors the configured workload itself and self-reports @@ -51,7 +54,9 @@ openshell sandbox create --name mxc-demo --policy demo.yaml ` The `command` array is required and preserves Windows argument boundaries. `cwd` is optional. Environment variables come from the standard sandbox and template environment maps; the driver never copies values from the gateway host environment. -Network policy and live policy replacement or merge updates are rejected while the gateway uses MXC. Delete and recreate the sandbox to apply a different filesystem policy. +Network policy support depends on the selected containment path. Policy +replacement and merge updates use the gateway's standard sandbox configuration +revision contract. ## Prerequisites (live runs) @@ -64,7 +69,11 @@ no isolation session needed), set `OPENSHELL_MXC_MOCK_WXC=1`. ## Policy mapping -The production driver maps the typed `SandboxPolicy` to MXC configuration before it inserts a registry entry or invokes `wxc-exec`. Mapping failure therefore returns from `CreateSandbox` without leaving a partial sandbox. +The production driver maps the typed `SandboxPolicy` carried by the standard +driver request to MXC configuration before it inserts a registry entry or +invokes `wxc-exec`. Mapping failure therefore returns from `CreateSandbox` +without leaving a partial sandbox. There is no in-process policy side channel +or MXC-specific gateway composition variant. `EmbeddedPolicyMapper` calls the embedded [`policy_map`](src/policy_map/) module directly and normalizes filesystem paths to Windows form. It does not add gateway-configured host paths. The policy supplied for the sandbox is the only source of filesystem grants. diff --git a/crates/openshell-driver-mxc/src/driver.rs b/crates/openshell-driver-mxc/src/driver.rs index d8e981a46..62f134205 100644 --- a/crates/openshell-driver-mxc/src/driver.rs +++ b/crates/openshell-driver-mxc/src/driver.rs @@ -177,13 +177,6 @@ pub struct MxcComputeBackend { registry: Arc>>, watch_tx: Arc>, policy_mapper: Arc, - /// Out-of-band side channel for the `SandboxPolicy` (A1). The proto driver - /// contract has no `policy` field and there is no driver-side - /// `GetSandboxConfig`, so `ComputeRuntime::create_sandbox` stages the policy - /// here keyed by sandbox id (mirroring the `sandbox_token` injection), - /// immediately before dispatching to this backend's `create_sandbox`, which - /// removes/consumes it. - pending_policies: Arc>>, } impl std::fmt::Debug for MxcComputeBackend { @@ -279,17 +272,9 @@ impl MxcComputeBackend { // Production policy translation is always handled by the embedded // mapper before any MXC lifecycle side effects begin. policy_mapper: Arc::new(EmbeddedPolicyMapper), - pending_policies: Arc::new(Mutex::new(HashMap::new())), } } - /// Returns a clone of the `pending_policies` side channel so the gateway's - /// `ComputeRuntime` can stage the typed `SandboxPolicy` by sandbox id right - /// before dispatching `create_sandbox` (A1 wiring). - pub fn policy_sink(&self) -> Arc>> { - self.pending_policies.clone() - } - /// Test-only constructor wiring the in-process mock `wxc-exec` shim. #[cfg(test)] pub(crate) fn new_mocked(config: MxcComputeConfig) -> Self { @@ -304,10 +289,14 @@ impl MxcComputeBackend { driver_version: DRIVER_VERSION.to_string(), default_image: DEFAULT_IMAGE_SENTINEL.to_string(), gateway_manages_lifecycle: false, + supports_sandbox_authentication: false, + driver_reports_runtime_readiness: true, + resource_capabilities: None, + rootfs_tar_staging_dir: String::new(), } } - pub fn validate_sandbox_create(&self, sandbox: &DriverSandbox) -> Result<(), tonic::Status> { + fn validate_sandbox_fields(&self, sandbox: &DriverSandbox) -> Result<(), tonic::Status> { if let Some(spec) = &sandbox.spec { if effective_driver_gpu_count(driver_gpu_requirements( spec.resource_requirements.as_ref(), @@ -330,6 +319,29 @@ impl MxcComputeBackend { sandbox_config(sandbox)?; Ok(()) } + + fn map_sandbox_policy( + &self, + sandbox_id: &str, + policy: Option<&SandboxPolicy>, + ) -> Result { + self.policy_mapper + .map( + policy, + &MapCtx { + sandbox_id: sandbox_id.to_string(), + egress: None, + }, + ) + .map_err(|error| tonic::Status::invalid_argument(error.to_string())) + } + + pub fn validate_sandbox_create(&self, sandbox: &DriverSandbox) -> Result<(), tonic::Status> { + self.validate_sandbox_fields(sandbox)?; + let policy = sandbox.spec.as_ref().and_then(|spec| spec.policy.as_ref()); + self.map_sandbox_policy(&sandbox.id, policy)?; + Ok(()) + } pub async fn get_sandbox(&self, sandbox_name: &str) -> Option { let registry = self.registry.lock().await; registry @@ -346,25 +358,14 @@ impl MxcComputeBackend { pub async fn create_sandbox(&self, sandbox: &DriverSandbox) -> Result<(), tonic::Status> { let sandbox_id = sandbox.id.clone(); - // Consume the out-of-band policy staged by `ComputeRuntime::create_sandbox` - // (A1). Always remove so rejected creates cannot leak policy state. - let policy = self.pending_policies.lock().await.remove(&sandbox_id); - self.validate_sandbox_create(sandbox)?; + self.validate_sandbox_fields(sandbox)?; let sandbox_config = sandbox_config(sandbox)?; // Policy translation is deterministic and side-effect free. Do it before // inserting the registry entry or launching MXC so invalid requests fail // synchronously at the CreateSandbox boundary. - let mapped = self - .policy_mapper - .map( - policy.as_ref(), - &MapCtx { - sandbox_id: sandbox_id.clone(), - egress: None, - }, - ) - .map_err(|error| tonic::Status::invalid_argument(error.to_string()))?; + let policy = sandbox.spec.as_ref().and_then(|spec| spec.policy.as_ref()); + let mapped = self.map_sandbox_policy(&sandbox_id, policy)?; if sandbox .spec @@ -941,6 +942,11 @@ mod lifecycle_tests { } } + fn with_policy(mut sandbox: DriverSandbox, policy: SandboxPolicy) -> DriverSandbox { + sandbox.spec.as_mut().unwrap().policy = Some(policy); + sandbox + } + fn ready_condition(sb: &DriverSandbox) -> Option { sb.status .as_ref()? @@ -1027,13 +1033,8 @@ mod lifecycle_tests { ]; let backend = MxcComputeBackend::new_mocked(MxcComputeConfig::default()); - // Stage the policy via the A1 side channel (as ComputeRuntime would). - let sink = backend.policy_sink(); - sink.lock() - .await - .insert("sb-pos".into(), fs_policy(&[&share])); - - let sb = driver_sandbox_with_command("sb-pos", &share, cmd); + let policy = fs_policy(&[&share]); + let sb = with_policy(driver_sandbox_with_command("sb-pos", &share, cmd), policy); backend.create_sandbox(&sb).await.expect("create accepted"); // Self-reported Ready=True (no supervisor) once the agent exec launches. @@ -1085,12 +1086,8 @@ mod lifecycle_tests { ]; let backend = MxcComputeBackend::new_mocked(MxcComputeConfig::default()); - let sink = backend.policy_sink(); - sink.lock() - .await - .insert("sb-pc".into(), fs_policy(&[&share])); - - let sb = driver_sandbox_with_command("sb-pc", &share, cmd); + let policy = fs_policy(&[&share]); + let sb = with_policy(driver_sandbox_with_command("sb-pc", &share, cmd), policy); backend.create_sandbox(&sb).await.expect("create accepted"); let ready = wait_for(&backend, "sb-pc", |s| { @@ -1142,12 +1139,10 @@ mod lifecycle_tests { // Subscribe to the watch stream BEFORE create so we catch the denial event. let mut stream = backend.watch_sandboxes().await; - let sink = backend.policy_sink(); - sink.lock() - .await - .insert("sb-neg".into(), fs_policy(&[&share])); + let policy = fs_policy(&[&share]); + let sandbox = with_policy(driver_sandbox_with_command("sb-neg", &share, cmd), policy); backend - .create_sandbox(&driver_sandbox_with_command("sb-neg", &share, cmd)) + .create_sandbox(&sandbox) .await .expect("create accepted"); @@ -1199,13 +1194,10 @@ mod lifecycle_tests { format!("$null = '{share}'; Start-Sleep -Seconds 60"), ]; let backend = MxcComputeBackend::new_mocked(MxcComputeConfig::default()); + let policy = fs_policy(&[&share]); + let sandbox = with_policy(driver_sandbox_with_command("sb-stop", "", command), policy); backend - .policy_sink() - .lock() - .await - .insert("sb-stop".into(), fs_policy(&[&share])); - backend - .create_sandbox(&driver_sandbox_with_command("sb-stop", "", command)) + .create_sandbox(&sandbox) .await .expect("create accepted"); wait_for(&backend, "sb-stop", |sandbox| { @@ -1245,13 +1237,9 @@ mod lifecycle_tests { binaries: Vec::new(), }, ); - backend - .policy_sink() - .lock() - .await - .insert("sb-net".into(), policy); + let sandbox = with_policy(driver_sandbox("sb-net"), policy); let error = backend - .create_sandbox(&driver_sandbox("sb-net")) + .create_sandbox(&sandbox) .await .expect_err("unmappable policy must fail CreateSandbox synchronously"); assert_eq!(error.code(), tonic::Code::InvalidArgument); diff --git a/crates/openshell-driver-mxc/src/grpc.rs b/crates/openshell-driver-mxc/src/grpc.rs index 9e7dada94..dfe8f66b2 100644 --- a/crates/openshell-driver-mxc/src/grpc.rs +++ b/crates/openshell-driver-mxc/src/grpc.rs @@ -9,14 +9,15 @@ use crate::driver::MxcComputeBackend; use futures::{Stream, StreamExt}; use openshell_core::proto::compute::v1::{ - CreateSandboxRequest, CreateSandboxResponse, DeleteSandboxRequest, DeleteSandboxResponse, - DeleteWorkspaceRequest, DeleteWorkspaceResponse, EnsureWorkspaceRequest, - EnsureWorkspaceResponse, GetCapabilitiesRequest, GetCapabilitiesResponse, - GetGatewayListenerRequirementsRequest, GetGatewayListenerRequirementsResponse, - GetSandboxRequest, GetSandboxResponse, ListSandboxesRequest, ListSandboxesResponse, - StartSandboxRequest, StartSandboxResponse, StopSandboxRequest, StopSandboxResponse, - ValidateSandboxCreateRequest, ValidateSandboxCreateResponse, WatchSandboxesEvent, - WatchSandboxesRequest, compute_driver_server::ComputeDriver, + AuthenticateSandboxRequest, AuthenticateSandboxResponse, CreateSandboxRequest, + CreateSandboxResponse, DeleteSandboxRequest, DeleteSandboxResponse, DeleteWorkspaceRequest, + DeleteWorkspaceResponse, EnsureWorkspaceRequest, EnsureWorkspaceResponse, + GetCapabilitiesRequest, GetCapabilitiesResponse, GetGatewayListenerRequirementsRequest, + GetGatewayListenerRequirementsResponse, GetSandboxRequest, GetSandboxResponse, + ListSandboxesRequest, ListSandboxesResponse, StartSandboxRequest, StartSandboxResponse, + StopSandboxRequest, StopSandboxResponse, ValidateSandboxCreateRequest, + ValidateSandboxCreateResponse, WatchSandboxesEvent, WatchSandboxesRequest, + compute_driver_server::ComputeDriver, }; use std::pin::Pin; use tonic::{Request, Response, Status}; @@ -41,6 +42,15 @@ impl ComputeDriver for ComputeDriverService { Ok(Response::new(self.backend.capabilities())) } + async fn authenticate_sandbox( + &self, + _request: Request, + ) -> Result, Status> { + Err(Status::unimplemented( + "mxc does not authenticate sandbox credentials", + )) + } + async fn get_gateway_listener_requirements( &self, _request: Request, @@ -56,8 +66,8 @@ impl ComputeDriver for ComputeDriverService { &self, request: Request, ) -> Result, Status> { + let request = request.into_inner(); let sandbox = request - .into_inner() .sandbox .ok_or_else(|| Status::invalid_argument("sandbox is required"))?; self.backend.validate_sandbox_create(&sandbox)?; @@ -99,8 +109,8 @@ impl ComputeDriver for ComputeDriverService { &self, request: Request, ) -> Result, Status> { + let request = request.into_inner(); let sandbox = request - .into_inner() .sandbox .ok_or_else(|| Status::invalid_argument("sandbox is required"))?; self.backend.create_sandbox(&sandbox).await?; diff --git a/crates/openshell-driver-mxc/src/policy.rs b/crates/openshell-driver-mxc/src/policy.rs index 1f26d186e..a90923885 100644 --- a/crates/openshell-driver-mxc/src/policy.rs +++ b/crates/openshell-driver-mxc/src/policy.rs @@ -78,8 +78,8 @@ fn format_loss(items: &[LossItem]) -> String { /// Translates an `OpenShell` `SandboxPolicy` into an MXC `ContainerConfig` /// fragment, returning a loss report of anything unrepresentable. pub trait PolicyMapper: Send + Sync { - /// `policy` is `None` only when the gateway failed to stage one (the MXC - /// path treats that as a hard error — the demo's whole point is enforcement). + /// `policy` is `None` only when the create request omitted it. The MXC path + /// treats that as a hard error because it cannot launch without enforcement. fn map(&self, policy: Option<&SandboxPolicy>, ctx: &MapCtx) -> Result; } diff --git a/crates/openshell-driver-podman/README.md b/crates/openshell-driver-podman/README.md index d55d20ec9..e53ddc9f3 100644 --- a/crates/openshell-driver-podman/README.md +++ b/crates/openshell-driver-podman/README.md @@ -81,7 +81,7 @@ The container spec in `container.rs` sets these security-critical fields: |---|---|---| | `user` | `0:0` | The supervisor needs root inside the container for namespace creation, proxy setup, Landlock, seccomp, and filesystem preparation. | | `cap_drop` | Selected unneeded defaults | Podman's default capability set is already restricted. The driver drops capabilities the supervisor does not need. | -| `cap_add` | `SYS_ADMIN`, `NET_ADMIN`, `SYS_PTRACE`, `SYSLOG`, `DAC_READ_SEARCH`, `SETPCAP` | Grants supervisor-only capabilities required for namespace setup, process identity, bypass diagnostics, and child bounding-set cleanup. Policy DNS binds an unprivileged supervisor port and does not require `NET_BIND_SERVICE`. | +| `cap_add` | `SYS_ADMIN`, `NET_ADMIN`, `SYS_PTRACE`, `SYSLOG`, `DAC_READ_SEARCH`, `SETPCAP`, `KILL` | Grants supervisor-only capabilities required for namespace setup, process identity, bypass diagnostics, child bounding-set cleanup, and forwarding shutdown signals to a workload that runs as the sandbox user. Policy DNS binds an unprivileged supervisor port and does not require `NET_BIND_SERVICE`. | | `no_new_privileges` | `true` | Prevents privilege escalation after exec. | | `seccomp_profile_path` | `unconfined` | The supervisor installs its own policy-aware BPF filter. A container-level profile can block Landlock/seccomp syscalls during setup. | | `mounts` | Private tmpfs at `/run/netns` | Lets the supervisor create named network namespaces in rootless Podman. | @@ -457,11 +457,11 @@ matter compared to cluster or rootful runtimes: ## Implementation References -- Gateway integration: `crates/openshell-server/src/compute/mod.rs` - (`new_podman` and `PodmanComputeDriver` wiring). -- Server configuration: `crates/openshell-server/src/lib.rs` - (`ComputeDriverKind::Podman` builds `PodmanComputeConfig` including - `sandbox_ssh_socket_path` from gateway `Config`). +- Gateway integration: `crates/openshell-gateway/src/lib.rs` registers the + driver factory and constructs `PodmanComputeConfig` from the generic server + build context. +- Server configuration: `crates/openshell-server/src/lib.rs` exposes the + backend-agnostic registry and factory context. - Gateway relay path: `openshell-core` `Config::sandbox_ssh_socket_path` in `crates/openshell-core/src/config.rs`. - SSRF mitigation: `crates/openshell-core/src/net.rs`, diff --git a/crates/openshell-driver-podman/src/client.rs b/crates/openshell-driver-podman/src/client.rs index 508b604ce..5f381daa9 100644 --- a/crates/openshell-driver-podman/src/client.rs +++ b/crates/openshell-driver-podman/src/client.rs @@ -719,7 +719,7 @@ impl PodmanClient { url_encode(policy), ); // Image pulls can be slow — use a generous timeout. - let pull_timeout = Duration::from_secs(600); + let pull_timeout = Duration::from_mins(10); let (status, bytes) = self .request(hyper::Method::POST, &path, None, pull_timeout) .await?; diff --git a/crates/openshell-driver-podman/src/container.rs b/crates/openshell-driver-podman/src/container.rs index abb9d69dd..a81ee13e1 100644 --- a/crates/openshell-driver-podman/src/container.rs +++ b/crates/openshell-driver-podman/src/container.rs @@ -1114,8 +1114,6 @@ pub fn build_container_spec_for_image( "DAC_OVERRIDE".into(), // Not needed: the supervisor does not create setuid/setgid executables. "FSETID".into(), - // Not needed: the supervisor does not send signals to arbitrary processes. - "KILL".into(), // Not needed: the supervisor does not bind privileged ports (<1024). "NET_BIND_SERVICE".into(), // Not in Podman's default set but explicitly denied in case the image @@ -1146,6 +1144,9 @@ pub fn build_container_spec_for_image( // Child setup clears the capability bounding set before exec, which // requires CAP_SETPCAP in the supervisor until drop_privileges(). "SETPCAP".into(), + // Forwarding shutdown signals to the canonical workload process + // group after it drops to the sandbox UID requires CAP_KILL. + "KILL".into(), ], // SETUID, SETGID, SETPCAP, CHOWN, and FOWNER are intentionally kept from // Podman's default set and not dropped: @@ -1885,6 +1886,7 @@ mod tests { "missing DAC_READ_SEARCH" ); assert!(added.contains(&"SETPCAP"), "missing SETPCAP"); + assert!(added.contains(&"KILL"), "missing KILL"); // SETUID and SETGID are NOT in cap_add — they remain available from the // default bounding set because we no longer use cap_drop:ALL. Verify they @@ -1916,6 +1918,10 @@ mod tests { !dropped.contains(&"SETPCAP"), "SETPCAP must not be dropped (needed for child bounding-set clear)" ); + assert!( + !dropped.contains(&"KILL"), + "KILL must not be dropped (needed to signal the sandbox workload on shutdown)" + ); assert!( !dropped.contains(&"ALL"), "must not use cap_drop:ALL in rootless Podman" diff --git a/crates/openshell-driver-podman/src/driver.rs b/crates/openshell-driver-podman/src/driver.rs index 8f7c0d32f..50eb01469 100644 --- a/crates/openshell-driver-podman/src/driver.rs +++ b/crates/openshell-driver-podman/src/driver.rs @@ -25,8 +25,9 @@ use openshell_core::proto::compute::v1::GatewayDefaultRouteInterfaceRequirement; #[cfg(target_os = "macos")] use openshell_core::proto::compute::v1::GatewayLoopbackInterfaceRequirement; use openshell_core::proto::compute::v1::{ - DriverSandbox, GatewayListenerRequirement, GetCapabilitiesResponse, GpuResourceRequirements, - gateway_listener_requirement::Selector, + CpuResourceCapabilities, DriverSandbox, GatewayListenerRequirement, GetCapabilitiesResponse, + GpuResourceCapabilities, GpuResourceRequirements, MemoryResourceCapabilities, + ResourceCapabilities, gateway_listener_requirement::Selector, }; #[cfg(target_os = "linux")] use std::net::{IpAddr, SocketAddr}; @@ -311,11 +312,21 @@ fn podman_gpu_selection_error(err: CdiGpuSelectionError) -> ComputeDriverError { ComputeDriverError::Precondition(err.to_string()) } +/// Return the first responsive local Podman API socket. +#[must_use] +pub fn detect_socket() -> Option { + crate::socket_discovery::detect_socket() +} + +#[must_use] +pub fn is_available() -> bool { + detect_socket().is_some() +} + /// Resolve the socket to connect to: explicit configuration wins, otherwise /// fall back to `detect`. Returns an error if neither resolves. /// -/// Takes `detect` as a parameter (rather than calling -/// [`openshell_core::config::detect_podman_socket`] directly) so tests can +/// Takes `detect` as a parameter so tests can /// exercise the precedence deterministically, without touching real /// environment variables or the filesystem. fn resolve_socket_path( @@ -337,10 +348,7 @@ impl PodmanComputeDriver { const MAX_PING_RETRIES: u32 = 5; const PING_RETRY_DELAY: Duration = Duration::from_secs(2); - let socket_path = resolve_socket_path( - config.socket_path.clone(), - openshell_core::config::detect_podman_socket, - )?; + let socket_path = resolve_socket_path(config.socket_path.clone(), detect_socket)?; config.socket_path = Some(socket_path.clone()); if !socket_path.exists() { @@ -508,6 +516,21 @@ impl PodmanComputeDriver { default_image: self.config.default_image.clone(), gateway_manages_lifecycle: true, supports_sandbox_authentication: false, + driver_reports_runtime_readiness: false, + resource_capabilities: Some(ResourceCapabilities { + cpu: Some(CpuResourceCapabilities { + limit_supported: true, + }), + memory: Some(MemoryResourceCapabilities { + limit_supported: true, + }), + gpu: Some(GpuResourceCapabilities { + default_selection_supported: true, + count_selection_supported: true, + }), + }), + rootfs_tar_staging_dir: String::new(), + rootfs_tar_max_bytes: 0, }) } @@ -720,10 +743,10 @@ impl PodmanComputeDriver { /// Create a sandbox container. #[tracing::instrument( - name = "podman.create_sandbox", + name = "podman.provision", skip(self, sandbox), fields( - otel.name = "podman.create_sandbox", + otel.name = "podman.provision", otel.status_code = tracing::field::Empty, sandbox.id = %sandbox.id, sandbox.name = %sandbox.name, @@ -1800,7 +1823,7 @@ mod tests { use tracing::instrument::WithSubscriber as _; use tracing_subscriber::layer::SubscriberExt as _; - let _tracing_lock = crate::otel_tracing::test_lock().await; + let _tracing_lock = openshell_otel_test_support::tracing_test_lock().await; let (socket_path, _requests, handle) = spawn_podman_stub( "trace-stop", vec![ @@ -1816,7 +1839,8 @@ mod tests { let provider = SdkTracerProvider::builder() .with_simple_exporter(exporter.clone()) .build(); - let subscriber = tracing_subscriber::registry().with(crate::otel_tracing::layer(&provider)); + let subscriber = + tracing_subscriber::registry().with(crate::otel_tracing::TRACING.layer(&provider)); test_driver(socket_path.clone()) .stop_sandbox("sandbox-1") @@ -1849,7 +1873,7 @@ mod tests { use tracing::instrument::WithSubscriber as _; use tracing_subscriber::layer::SubscriberExt as _; - let _tracing_lock = crate::otel_tracing::test_lock().await; + let _tracing_lock = openshell_otel_test_support::tracing_test_lock().await; let (socket_path, _requests, handle) = spawn_podman_stub( "trace-create", vec![ @@ -1868,7 +1892,8 @@ mod tests { let provider = SdkTracerProvider::builder() .with_simple_exporter(exporter.clone()) .build(); - let subscriber = tracing_subscriber::registry().with(crate::otel_tracing::layer(&provider)); + let subscriber = + tracing_subscriber::registry().with(crate::otel_tracing::TRACING.layer(&provider)); test_driver(socket_path.clone()) .create_sandbox(&plain_sandbox("sandbox-trace", "demo")) @@ -1881,7 +1906,7 @@ mod tests { let spans = exporter.get_finished_spans().unwrap(); let create = spans .iter() - .find(|span| span.name == "podman.create_sandbox") + .find(|span| span.name == "podman.provision") .expect("create operation should be exported"); for name in [ "podman.prepare_images", @@ -1909,7 +1934,7 @@ mod tests { use tracing::instrument::WithSubscriber as _; use tracing_subscriber::layer::SubscriberExt as _; - let _tracing_lock = crate::otel_tracing::test_lock().await; + let _tracing_lock = openshell_otel_test_support::tracing_test_lock().await; let (socket_path, _requests, handle) = spawn_podman_stub( "trace-image-failure", vec![ @@ -1921,7 +1946,8 @@ mod tests { let provider = SdkTracerProvider::builder() .with_simple_exporter(exporter.clone()) .build(); - let subscriber = tracing_subscriber::registry().with(crate::otel_tracing::layer(&provider)); + let subscriber = + tracing_subscriber::registry().with(crate::otel_tracing::TRACING.layer(&provider)); test_driver(socket_path.clone()) .create_sandbox(&plain_sandbox("sandbox-trace", "demo")) @@ -1950,12 +1976,13 @@ mod tests { use tracing::instrument::WithSubscriber as _; use tracing_subscriber::layer::SubscriberExt as _; - let _tracing_lock = crate::otel_tracing::test_lock().await; + let _tracing_lock = openshell_otel_test_support::tracing_test_lock().await; let exporter = InMemorySpanExporterBuilder::new().build(); let provider = SdkTracerProvider::builder() .with_simple_exporter(exporter.clone()) .build(); - let subscriber = tracing_subscriber::registry().with(crate::otel_tracing::layer(&provider)); + let subscriber = + tracing_subscriber::registry().with(crate::otel_tracing::TRACING.layer(&provider)); let (start_socket, _requests, start_handle) = spawn_podman_stub( "trace-start", @@ -1982,7 +2009,8 @@ mod tests { StubResponse::new(StatusCode::NO_CONTENT, ""), ], ); - let subscriber = tracing_subscriber::registry().with(crate::otel_tracing::layer(&provider)); + let subscriber = + tracing_subscriber::registry().with(crate::otel_tracing::TRACING.layer(&provider)); test_driver(delete_socket.clone()) .delete_sandbox("sandbox-1") .with_subscriber(subscriber) @@ -3097,4 +3125,19 @@ mod tests { assert!(userns_remaps_uids(Some("no-map"))); assert!(userns_remaps_uids(Some("private"))); } + + #[test] + fn capabilities_report_static_resource_support() { + let driver = PodmanComputeDriver::for_tests(PodmanComputeConfig::default()); + let resources = driver + .capabilities() + .unwrap() + .resource_capabilities + .unwrap(); + assert!(resources.cpu.unwrap().limit_supported); + assert!(resources.memory.unwrap().limit_supported); + let gpu = resources.gpu.unwrap(); + assert!(gpu.default_selection_supported); + assert!(gpu.count_selection_supported); + } } diff --git a/crates/openshell-driver-podman/src/grpc.rs b/crates/openshell-driver-podman/src/grpc.rs index 222c617c2..fedeea306 100644 --- a/crates/openshell-driver-podman/src/grpc.rs +++ b/crates/openshell-driver-podman/src/grpc.rs @@ -14,55 +14,21 @@ use openshell_core::proto::compute::v1::{ ValidateSandboxCreateRequest, ValidateSandboxCreateResponse, WatchSandboxesEvent, WatchSandboxesRequest, compute_driver_server::ComputeDriver, }; -use std::future::Future; use std::pin::Pin; -use std::task::{Context, Poll}; use tonic::{Request, Response, Status}; -use tracing::Instrument as _; use crate::PodmanComputeDriver; type ComputeDriverWatchStream = Pin> + Send + 'static>>; -struct TracedWatchStream { - inner: ComputeDriverWatchStream, - span: tracing::Span, -} - -impl TracedWatchStream { - fn new(inner: ComputeDriverWatchStream, span: tracing::Span) -> Self { - Self { inner, span } - } -} - -impl Stream for TracedWatchStream { - type Item = Result; - - fn poll_next(mut self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll> { - let span = self.span.clone(); - let _entered = span.enter(); - let result = self.inner.as_mut().poll_next(cx); - match &result { - Poll::Ready(Some(Err(status))) => { - openshell_otel::mark_error(&self.span); - self.span - .record("rpc.grpc.status_code", status.code() as i32); - } - Poll::Ready(None) => { - self.span - .record("rpc.grpc.status_code", tonic::Code::Ok as i32); - } - Poll::Pending | Poll::Ready(Some(Ok(_))) => {} - } - result - } -} +#[cfg(test)] +type TracedWatchStream = openshell_otel::TracedGrpcStream; #[derive(Debug, Clone)] pub struct ComputeDriverService { driver: PodmanComputeDriver, - trace_in_process_rpc: bool, + rpc_tracer: openshell_otel::InProcessRpcTracer, } impl ComputeDriverService { @@ -70,7 +36,7 @@ impl ComputeDriverService { pub fn new(driver: PodmanComputeDriver) -> Self { Self { driver, - trace_in_process_rpc: false, + rpc_tracer: openshell_otel::InProcessRpcTracer::disabled(), } } @@ -78,51 +44,9 @@ impl ComputeDriverService { pub fn new_in_process(driver: PodmanComputeDriver) -> Self { Self { driver, - trace_in_process_rpc: true, + rpc_tracer: openshell_otel::InProcessRpcTracer::enabled(), } } - - fn in_process_rpc_span( - &self, - operation: &'static str, - method: &'static str, - ) -> Option { - self.trace_in_process_rpc.then(|| { - tracing::info_span!( - target: "openshell_driver_podman::otel_tracing", - "driver_rpc", - otel.name = operation, - otel.kind = "server", - otel.status_code = tracing::field::Empty, - rpc.system = "grpc", - rpc.service = "openshell.compute.v1.ComputeDriver", - rpc.method = method, - rpc.grpc.status_code = tracing::field::Empty, - ) - }) - } - - async fn trace_rpc( - &self, - operation: &'static str, - method: &'static str, - future: impl Future>, - ) -> Result { - let Some(span) = self.in_process_rpc_span(operation, method) else { - return future.await; - }; - let result = future.instrument(span.clone()).await; - match &result { - Ok(_) => { - span.record("rpc.grpc.status_code", tonic::Code::Ok as i32); - } - Err(status) => { - openshell_otel::mark_error(&span); - span.record("rpc.grpc.status_code", status.code() as i32); - } - } - result - } } #[tonic::async_trait] @@ -132,51 +56,54 @@ impl ComputeDriver for ComputeDriverService { _request: Request, ) -> Result, Status> { - Err(Status::unimplemented( - "podman does not authenticate sandbox credentials", - )) + self.rpc_tracer + .trace(openshell_otel::rpc::AUTHENTICATE_SANDBOX, async { + Err(Status::unimplemented( + "podman does not authenticate sandbox credentials", + )) + }) + .await } async fn get_capabilities( &self, _request: Request, ) -> Result, Status> { - self.trace_rpc("driver.get_capabilities", "get_capabilities", async { - self.driver - .capabilities() - .map(Response::new) - .map_err(Status::from) - }) - .await + self.rpc_tracer + .trace(openshell_otel::rpc::GET_CAPABILITIES, async { + self.driver + .capabilities() + .map(Response::new) + .map_err(Status::from) + }) + .await } async fn get_gateway_listener_requirements( &self, _request: Request, ) -> Result, Status> { - self.trace_rpc( - "driver.get_gateway_listener_requirements", - "get_gateway_listener_requirements", - async { - Ok(Response::new(GetGatewayListenerRequirementsResponse { - requirements: self - .driver - .gateway_listener_requirements() - .map_err(Status::from)?, - })) - }, - ) - .await + self.rpc_tracer + .trace( + openshell_otel::rpc::GET_GATEWAY_LISTENER_REQUIREMENTS, + async { + Ok(Response::new(GetGatewayListenerRequirementsResponse { + requirements: self + .driver + .gateway_listener_requirements() + .map_err(Status::from)?, + })) + }, + ) + .await } async fn validate_sandbox_create( &self, request: Request, ) -> Result, Status> { - self.trace_rpc( - "driver.validate_sandbox_create", - "validate_sandbox_create", - async { + self.rpc_tracer + .trace(openshell_otel::rpc::VALIDATE_SANDBOX_CREATE, async { let sandbox = request .into_inner() .sandbox @@ -186,115 +113,120 @@ impl ComputeDriver for ComputeDriverService { .await .map_err(Status::from)?; Ok(Response::new(ValidateSandboxCreateResponse {})) - }, - ) - .await + }) + .await } async fn get_sandbox( &self, request: Request, ) -> Result, Status> { - self.trace_rpc("driver.get_sandbox", "get_sandbox", async { - let request = request.into_inner(); - if request.sandbox_id.is_empty() { - return Err(Status::invalid_argument("sandbox_id is required")); - } - let sandbox = self - .driver - .get_sandbox(&request.sandbox_id) - .await - .map_err(Status::from)? - .ok_or_else(|| Status::not_found("sandbox not found"))?; - Ok(Response::new(GetSandboxResponse { - sandbox: Some(sandbox), - })) - }) - .await + self.rpc_tracer + .trace(openshell_otel::rpc::GET_SANDBOX, async { + let request = request.into_inner(); + if request.sandbox_id.is_empty() { + return Err(Status::invalid_argument("sandbox_id is required")); + } + let sandbox = self + .driver + .get_sandbox(&request.sandbox_id) + .await + .map_err(Status::from)? + .ok_or_else(|| Status::not_found("sandbox not found"))?; + Ok(Response::new(GetSandboxResponse { + sandbox: Some(sandbox), + })) + }) + .await } async fn list_sandboxes( &self, _request: Request, ) -> Result, Status> { - self.trace_rpc("driver.list_sandboxes", "list_sandboxes", async { - let sandboxes = self.driver.list_sandboxes().await.map_err(Status::from)?; - Ok(Response::new(ListSandboxesResponse { sandboxes })) - }) - .await + self.rpc_tracer + .trace(openshell_otel::rpc::LIST_SANDBOXES, async { + let sandboxes = self.driver.list_sandboxes().await.map_err(Status::from)?; + Ok(Response::new(ListSandboxesResponse { sandboxes })) + }) + .await } async fn create_sandbox( &self, request: Request, ) -> Result, Status> { - self.trace_rpc("driver.create_sandbox", "create_sandbox", async { - let sandbox = request - .into_inner() - .sandbox - .ok_or_else(|| Status::invalid_argument("sandbox is required"))?; - self.driver - .create_sandbox(&sandbox) - .await - .map_err(Status::from)?; - Ok(Response::new(CreateSandboxResponse {})) - }) - .await + self.rpc_tracer + .trace(openshell_otel::rpc::CREATE_SANDBOX, async { + let sandbox = request + .into_inner() + .sandbox + .ok_or_else(|| Status::invalid_argument("sandbox is required"))?; + self.driver + .create_sandbox(&sandbox) + .await + .map_err(Status::from)?; + Ok(Response::new(CreateSandboxResponse {})) + }) + .await } async fn stop_sandbox( &self, request: Request, ) -> Result, Status> { - self.trace_rpc("driver.stop_sandbox", "stop_sandbox", async { - let request = request.into_inner(); - if request.sandbox_id.is_empty() { - return Err(Status::invalid_argument("sandbox_id is required")); - } - self.driver - .stop_sandbox(&request.sandbox_id) - .await - .map_err(Status::from)?; - Ok(Response::new(StopSandboxResponse {})) - }) - .await + self.rpc_tracer + .trace(openshell_otel::rpc::STOP_SANDBOX, async { + let request = request.into_inner(); + if request.sandbox_id.is_empty() { + return Err(Status::invalid_argument("sandbox_id is required")); + } + self.driver + .stop_sandbox(&request.sandbox_id) + .await + .map_err(Status::from)?; + Ok(Response::new(StopSandboxResponse {})) + }) + .await } async fn start_sandbox( &self, request: Request, ) -> Result, Status> { - self.trace_rpc("driver.start_sandbox", "start_sandbox", async { - let request = request.into_inner(); - if request.sandbox_id.is_empty() { - return Err(Status::invalid_argument("sandbox_id is required")); - } - self.driver - .start_sandbox(&request.sandbox_id) - .await - .map_err(Status::from)?; - Ok(Response::new(StartSandboxResponse {})) - }) - .await + self.rpc_tracer + .trace(openshell_otel::rpc::START_SANDBOX, async { + let request = request.into_inner(); + if request.sandbox_id.is_empty() { + return Err(Status::invalid_argument("sandbox_id is required")); + } + self.driver + .start_sandbox(&request.sandbox_id) + .await + .map_err(Status::from)?; + Ok(Response::new(StartSandboxResponse {})) + }) + .await } async fn delete_sandbox( &self, request: Request, ) -> Result, Status> { - self.trace_rpc("driver.delete_sandbox", "delete_sandbox", async { - let request = request.into_inner(); - if request.sandbox_id.is_empty() { - return Err(Status::invalid_argument("sandbox_id is required")); - } - let deleted = self - .driver - .delete_sandbox(&request.sandbox_id) - .await - .map_err(Status::from)?; - Ok(Response::new(DeleteSandboxResponse { deleted })) - }) - .await + self.rpc_tracer + .trace(openshell_otel::rpc::DELETE_SANDBOX, async { + let request = request.into_inner(); + if request.sandbox_id.is_empty() { + return Err(Status::invalid_argument("sandbox_id is required")); + } + let deleted = self + .driver + .delete_sandbox(&request.sandbox_id) + .await + .map_err(Status::from)?; + Ok(Response::new(DeleteSandboxResponse { deleted })) + }) + .await } type WatchSandboxesStream = ComputeDriverWatchStream; @@ -308,40 +240,32 @@ impl ComputeDriver for ComputeDriverService { let stream = stream.map(|item| item.map_err(|err| Status::internal(err.to_string()))); Ok::(Box::pin(stream)) }; - let Some(span) = self.in_process_rpc_span("driver.watch_sandboxes", "watch_sandboxes") - else { - return create_stream.await.map(Response::new); - }; - match create_stream.instrument(span.clone()).await { - Ok(stream) => Ok(Response::new(Box::pin(TracedWatchStream::new( - stream, span, - )))), - Err(status) => { - openshell_otel::mark_error(&span); - span.record("rpc.grpc.status_code", status.code() as i32); - Err(status) - } - } + self.rpc_tracer + .trace_stream(openshell_otel::rpc::WATCH_SANDBOXES, create_stream) + .await + .map(Response::new) } async fn ensure_workspace( &self, _request: Request, ) -> Result, Status> { - self.trace_rpc("driver.ensure_workspace", "ensure_workspace", async { - Ok(Response::new(EnsureWorkspaceResponse {})) - }) - .await + self.rpc_tracer + .trace(openshell_otel::rpc::ENSURE_WORKSPACE, async { + Ok(Response::new(EnsureWorkspaceResponse {})) + }) + .await } async fn delete_workspace( &self, _request: Request, ) -> Result, Status> { - self.trace_rpc("driver.delete_workspace", "delete_workspace", async { - Ok(Response::new(DeleteWorkspaceResponse {})) - }) - .await + self.rpc_tracer + .trace(openshell_otel::rpc::DELETE_WORKSPACE, async { + Ok(Response::new(DeleteWorkspaceResponse {})) + }) + .await } } @@ -386,7 +310,7 @@ mod tests { )); let server = tokio::spawn(async move { tonic::transport::Server::builder() - .layer(crate::otel_tracing::compute_driver_rpc_layer()) + .layer(openshell_otel::compute_driver_rpc_layer()) .add_service(ComputeDriverServer::new(service)) .serve_with_incoming_shutdown( tokio_stream::wrappers::TcpListenerStream::new(listener), @@ -429,7 +353,7 @@ mod tests { use tracing::{Instrument as _, instrument::WithSubscriber as _}; use tracing_subscriber::layer::SubscriberExt as _; - let _tracing_lock = crate::otel_tracing::test_lock().await; + let _tracing_lock = openshell_otel_test_support::tracing_test_lock().await; let gateway_exporter = InMemorySpanExporterBuilder::new().build(); let gateway_provider = SdkTracerProvider::builder() .with_simple_exporter(gateway_exporter.clone()) @@ -439,18 +363,18 @@ mod tests { .with_simple_exporter(driver_exporter.clone()) .build(); let subscriber = tracing_subscriber::registry() - .with(openshell_otel::layer_excluding_target_prefix( + .with(openshell_otel::layer_excluding_target_prefixes( &gateway_provider, "gateway-test", - Some(crate::otel_tracing::IN_PROCESS_TARGET_PREFIX), + crate::otel_tracing::TRACING.in_process_targets(), )) - .with(crate::otel_tracing::in_process_layer(&driver_provider)); + .with(crate::otel_tracing::TRACING.in_process_layer(&driver_provider)); let service = ComputeDriverService::new_in_process(PodmanComputeDriver::for_tests( PodmanComputeConfig::default(), )); async { - let gateway_span = tracing::info_span!(target: "openshell_server::compute", "driver", otel.name = "driver.get_capabilities", otel.kind = "client"); + let gateway_span = tracing::info_span!(target: "openshell_server::compute", "driver", otel.name = "openshell.compute.v1.ComputeDriver/GetCapabilities", otel.kind = "client"); ComputeDriver::get_capabilities(&service, Request::new(GetCapabilitiesRequest {})) .instrument(gateway_span) .await @@ -465,11 +389,11 @@ mod tests { let driver_spans = driver_exporter.get_finished_spans().unwrap(); let client = gateway_spans .iter() - .find(|span| span.name == "driver.get_capabilities") + .find(|span| span.name == "openshell.compute.v1.ComputeDriver/GetCapabilities") .unwrap(); let server = driver_spans .iter() - .find(|span| span.name == "driver.get_capabilities") + .find(|span| span.name == "openshell.compute.v1.ComputeDriver/GetCapabilities") .expect("in-process server span"); assert_eq!( server.span_context.trace_id(), @@ -478,8 +402,20 @@ mod tests { assert_eq!(server.parent_span_id, client.span_context.span_id()); assert_eq!(server.span_kind, opentelemetry::trace::SpanKind::Server); assert!(server.attributes.iter().any(|attribute| { - attribute.key.as_str() == "rpc.grpc.status_code" - && attribute.value.to_string() == (tonic::Code::Ok as i32).to_string() + attribute.key.as_str() == "rpc.method" + && attribute.value.to_string() + == "openshell.compute.v1.ComputeDriver/GetCapabilities" + })); + assert!( + server + .attributes + .iter() + .all(|attribute| attribute.key.as_str() != "rpc.service"), + "the current RPC semantic conventions integrate the service into rpc.method" + ); + assert!(server.attributes.iter().any(|attribute| { + attribute.key.as_str() == "rpc.response.status_code" + && attribute.value.to_string() == "OK" })); gateway_provider.shutdown().unwrap(); driver_provider.shutdown().unwrap(); @@ -490,13 +426,13 @@ mod tests { use opentelemetry_sdk::trace::{InMemorySpanExporterBuilder, SdkTracerProvider}; use tracing_subscriber::layer::SubscriberExt as _; - let _tracing_lock = crate::otel_tracing::test_lock().await; + let _tracing_lock = openshell_otel_test_support::tracing_test_lock().await; let exporter = InMemorySpanExporterBuilder::new().build(); let provider = SdkTracerProvider::builder() .with_simple_exporter(exporter.clone()) .build(); let dispatch = tracing::Dispatch::new( - tracing_subscriber::registry().with(crate::otel_tracing::layer(&provider)), + tracing_subscriber::registry().with(crate::otel_tracing::TRACING.layer(&provider)), ); let _dispatch = tracing::dispatcher::set_default(&dispatch); let (mut client, shutdown, server) = standalone_traced_client().await; @@ -523,7 +459,7 @@ mod tests { let spans = exporter.get_finished_spans().unwrap(); let capabilities = spans .iter() - .filter(|span| span.name == "driver.get_capabilities") + .filter(|span| span.name == "openshell.compute.v1.ComputeDriver/GetCapabilities") .collect::>(); assert_eq!(capabilities.len(), 1, "exactly one RPC span is expected"); assert_eq!( @@ -539,21 +475,21 @@ mod tests { opentelemetry::trace::SpanKind::Server ); assert!(capabilities[0].attributes.iter().any(|attribute| { - attribute.key.as_str() == "rpc.grpc.status_code" - && attribute.value.to_string() == (tonic::Code::Ok as i32).to_string() + attribute.key.as_str() == "rpc.response.status_code" + && attribute.value.to_string() == "OK" })); let failed = spans .iter() - .find(|span| span.name == "driver.validate_sandbox_create") + .find(|span| span.name == "openshell.compute.v1.ComputeDriver/ValidateSandboxCreate") .expect("failed RPC span should be exported"); assert!(matches!( failed.status, opentelemetry::trace::Status::Error { .. } )); assert!(failed.attributes.iter().any(|attribute| { - attribute.key.as_str() == "rpc.grpc.status_code" - && attribute.value.to_string() == (tonic::Code::InvalidArgument as i32).to_string() + attribute.key.as_str() == "rpc.response.status_code" + && attribute.value.to_string() == "INVALID_ARGUMENT" })); provider.shutdown().unwrap(); } @@ -564,22 +500,22 @@ mod tests { use tracing::instrument::WithSubscriber as _; use tracing_subscriber::layer::SubscriberExt as _; - let _tracing_lock = crate::otel_tracing::test_lock().await; + let _tracing_lock = openshell_otel_test_support::tracing_test_lock().await; let exporter = InMemorySpanExporterBuilder::new().build(); let provider = SdkTracerProvider::builder() .with_simple_exporter(exporter.clone()) .build(); - let subscriber = - tracing_subscriber::registry().with(crate::otel_tracing::in_process_layer(&provider)); + let subscriber = tracing_subscriber::registry() + .with(crate::otel_tracing::TRACING.in_process_layer(&provider)); async { let span = tracing::info_span!( - target: "openshell_driver_podman::otel_tracing", + target: crate::otel_tracing::TRACING.in_process_target(), "driver_rpc", - otel.name = "driver.watch_sandboxes", + otel.name = "openshell.compute.v1.ComputeDriver/WatchSandboxes", otel.kind = "server", otel.status_code = tracing::field::Empty, - rpc.grpc.status_code = tracing::field::Empty, + rpc.response.status_code = tracing::field::Empty, ); let inner: ComputeDriverWatchStream = Box::pin(futures::stream::iter([Err( Status::internal("watch failed"), @@ -605,7 +541,7 @@ mod tests { let spans = exporter.get_finished_spans().unwrap(); let span = spans .iter() - .find(|span| span.name == "driver.watch_sandboxes") + .find(|span| span.name == "openshell.compute.v1.ComputeDriver/WatchSandboxes") .expect("watch server span should be exported when the stream ends"); assert!(matches!( span.status, @@ -620,22 +556,22 @@ mod tests { use tracing::instrument::WithSubscriber as _; use tracing_subscriber::layer::SubscriberExt as _; - let _tracing_lock = crate::otel_tracing::test_lock().await; + let _tracing_lock = openshell_otel_test_support::tracing_test_lock().await; let exporter = InMemorySpanExporterBuilder::new().build(); let provider = SdkTracerProvider::builder() .with_simple_exporter(exporter.clone()) .build(); - let subscriber = - tracing_subscriber::registry().with(crate::otel_tracing::in_process_layer(&provider)); + let subscriber = tracing_subscriber::registry() + .with(crate::otel_tracing::TRACING.in_process_layer(&provider)); async { let span = tracing::info_span!( - target: "openshell_driver_podman::otel_tracing", + target: crate::otel_tracing::TRACING.in_process_target(), "driver_rpc", - otel.name = "driver.watch_sandboxes", + otel.name = "openshell.compute.v1.ComputeDriver/WatchSandboxes", otel.kind = "server", otel.status_code = tracing::field::Empty, - rpc.grpc.status_code = tracing::field::Empty, + rpc.response.status_code = tracing::field::Empty, ); let inner: ComputeDriverWatchStream = Box::pin(futures::stream::empty()); let mut stream = TracedWatchStream::new(inner, span); @@ -650,15 +586,62 @@ mod tests { let spans = exporter.get_finished_spans().unwrap(); let span = spans .iter() - .find(|span| span.name == "driver.watch_sandboxes") + .find(|span| span.name == "openshell.compute.v1.ComputeDriver/WatchSandboxes") .expect("watch server span should be exported when the stream completes"); assert!(span.attributes.iter().any(|attribute| { - attribute.key.as_str() == "rpc.grpc.status_code" - && attribute.value.to_string() == (tonic::Code::Ok as i32).to_string() + attribute.key.as_str() == "rpc.response.status_code" + && attribute.value.to_string() == "OK" })); provider.shutdown().unwrap(); } + #[tokio::test] + async fn in_process_stream_leaves_status_unset_when_dropped() { + use opentelemetry_sdk::trace::{InMemorySpanExporterBuilder, SdkTracerProvider}; + use tracing::instrument::WithSubscriber as _; + use tracing_subscriber::layer::SubscriberExt as _; + + let _tracing_lock = openshell_otel_test_support::tracing_test_lock().await; + let exporter = InMemorySpanExporterBuilder::new().build(); + let provider = SdkTracerProvider::builder() + .with_simple_exporter(exporter.clone()) + .build(); + let subscriber = tracing_subscriber::registry() + .with(crate::otel_tracing::TRACING.in_process_layer(&provider)); + + async { + let span = tracing::info_span!( + target: crate::otel_tracing::TRACING.in_process_target(), + "driver_rpc", + otel.name = "openshell.compute.v1.ComputeDriver/WatchSandboxes", + otel.kind = "server", + otel.status_code = tracing::field::Empty, + rpc.response.status_code = tracing::field::Empty, + error.type = tracing::field::Empty, + ); + let inner: ComputeDriverWatchStream = Box::pin(futures::stream::pending()); + let stream = TracedWatchStream::new(inner, span); + + drop(stream); + } + .with_subscriber(subscriber) + .await; + provider.force_flush().unwrap(); + + let spans = exporter.get_finished_spans().unwrap(); + let span = spans + .iter() + .find(|span| span.name == "openshell.compute.v1.ComputeDriver/WatchSandboxes") + .expect("watch server span should be exported when the stream is dropped"); + assert!(matches!(span.status, opentelemetry::trace::Status::Unset)); + assert!( + span.attributes + .iter() + .all(|attribute| attribute.key.as_str() != "rpc.response.status_code") + ); + provider.shutdown().unwrap(); + } + fn test_service(socket_path: PathBuf) -> ComputeDriverService { let config = PodmanComputeConfig { socket_path: Some(socket_path), diff --git a/crates/openshell-driver-podman/src/lib.rs b/crates/openshell-driver-podman/src/lib.rs index 7a78cf5d9..115e64eb2 100644 --- a/crates/openshell-driver-podman/src/lib.rs +++ b/crates/openshell-driver-podman/src/lib.rs @@ -7,6 +7,7 @@ pub(crate) mod container; pub mod driver; pub mod grpc; pub mod otel_tracing; +mod socket_discovery; #[cfg(test)] pub(crate) mod test_utils; pub(crate) mod watcher; diff --git a/crates/openshell-driver-podman/src/main.rs b/crates/openshell-driver-podman/src/main.rs index a0fa85d01..4c42ba969 100644 --- a/crates/openshell-driver-podman/src/main.rs +++ b/crates/openshell-driver-podman/src/main.rs @@ -7,8 +7,6 @@ use std::future::Future; use std::net::SocketAddr; use std::path::PathBuf; use tracing::info; -use tracing_subscriber::EnvFilter; -use tracing_subscriber::prelude::*; use openshell_core::VERSION; use openshell_core::proto::compute::v1::compute_driver_server::ComputeDriverServer; @@ -16,7 +14,6 @@ use openshell_driver_podman::config::{ DEFAULT_NETWORK_NAME, DEFAULT_PODMAN_STOP_TIMEOUT_SECS, DEFAULT_SANDBOX_PIDS_LIMIT, ImagePullPolicy, }; -use openshell_driver_podman::otel_tracing::compute_driver_rpc_layer; use openshell_driver_podman::{ComputeDriverService, PodmanComputeConfig, PodmanComputeDriver}; #[derive(Parser)] @@ -177,24 +174,15 @@ struct Args { #[tokio::main] async fn main() -> Result<()> { let args = Args::parse(); - let (tracer_provider, setup_error) = openshell_driver_podman::otel_tracing::provider_for( - args.otlp_endpoint.as_deref(), - args.gateway_name.as_deref(), + let _tracing = openshell_otel::install_driver_tracing( + openshell_driver_podman::otel_tracing::TRACING, + openshell_otel::DriverTracingConfig { + endpoint: args.otlp_endpoint.as_deref(), + gateway_name: args.gateway_name.as_deref(), + service_version: VERSION, + log_level: &args.log_level, + }, ); - tracing_subscriber::registry() - .with(EnvFilter::try_from_default_env().unwrap_or_else(|_| EnvFilter::new(&args.log_level))) - .with(tracing_subscriber::fmt::layer()) - .with( - tracer_provider - .as_ref() - .map(openshell_driver_podman::otel_tracing::layer), - ) - .init(); - if let Some(error) = setup_error { - tracing::error!(%error, "OTLP exporting could not be started"); - } else if let Some(endpoint) = &args.otlp_endpoint { - info!(endpoint, "OTLP exporting enabled"); - } let driver = PodmanComputeDriver::new(PodmanComputeConfig { socket_path: args.podman_socket, @@ -231,14 +219,14 @@ async fn main() -> Result<()> { .into_diagnostic()?; let service = ComputeDriverServer::new(ComputeDriverService::new(driver)); - let result = if let Some(socket_path) = args.bind_socket { + if let Some(socket_path) = args.bind_socket { let listener = openshell_core::external_driver_socket::bind_private(&socket_path) .map_err(|err| miette::miette!("{err}"))?; let _cleanup = openshell_core::external_driver_socket::SocketCleanup::new(socket_path.clone()); info!(socket = %socket_path.display(), "Starting Podman compute driver"); tonic::transport::Server::builder() - .layer(compute_driver_rpc_layer()) + .layer(openshell_otel::compute_driver_rpc_layer()) .add_service(service) .serve_with_incoming_shutdown( openshell_core::external_driver_socket::SameUidUnixIncoming::new(listener), @@ -249,18 +237,12 @@ async fn main() -> Result<()> { } else { info!(address = %args.bind_address, "Starting Podman compute driver"); tonic::transport::Server::builder() - .layer(compute_driver_rpc_layer()) + .layer(openshell_otel::compute_driver_rpc_layer()) .add_service(service) .serve_with_shutdown(args.bind_address, shutdown_signal()) .await .into_diagnostic() - }; - if let Some(provider) = &tracer_provider - && let Err(error) = provider.shutdown() - { - tracing::warn!(%error, "OTLP tracer provider shutdown failed"); } - result } async fn select_shutdown_signal( diff --git a/crates/openshell-driver-podman/src/otel_tracing.rs b/crates/openshell-driver-podman/src/otel_tracing.rs index 646472c8e..8d3314f82 100644 --- a/crates/openshell-driver-podman/src/otel_tracing.rs +++ b/crates/openshell-driver-podman/src/otel_tracing.rs @@ -1,165 +1,19 @@ // SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 -//! OpenTelemetry trace exporting for the Podman compute driver. +//! OpenTelemetry tracing identity for the Podman compute driver. -use openshell_otel::{OtlpTraceConfig, SdkTracerProvider, ServiceName, SetupError}; -pub use openshell_otel::{compute_driver_rpc_layer, compute_driver_rpc_operation}; -use tracing::Subscriber; -use tracing_subscriber::registry::LookupSpan; - -const SERVICE_NAME: &str = "openshell-driver-podman"; -const INSTRUMENTATION_SCOPE: &str = "openshell-driver-podman"; -pub const IN_PROCESS_TARGET_PREFIX: &str = "openshell_driver_podman"; - -/// Build a tracer provider for the configured OTLP/gRPC endpoint and gateway. -#[must_use] -pub fn provider_for( - endpoint: Option<&str>, - gateway_name: Option<&str>, -) -> (Option, Option) { - openshell_otel::provider_for(endpoint.map(|endpoint| { - OtlpTraceConfig { - endpoint, - service_name: ServiceName::Fixed(SERVICE_NAME), - service_version: Some(openshell_core::VERSION), - resource_attributes: gateway_name - .map(str::trim) - .filter(|name| !name.is_empty()) - .map(|name| { - vec![opentelemetry::KeyValue::new( - "openshell.gateway.name", - name.to_string(), - )] - }) - .unwrap_or_default(), - } - })) -} - -pub fn layer(provider: &SdkTracerProvider) -> openshell_otel::OtlpLayer -where - S: Subscriber + for<'span> LookupSpan<'span>, -{ - openshell_otel::layer(provider, INSTRUMENTATION_SCOPE) -} - -pub fn in_process_layer(provider: &SdkTracerProvider) -> openshell_otel::TargetOtlpLayer -where - S: Subscriber + for<'span> LookupSpan<'span>, -{ - openshell_otel::layer_for_target_prefix( - provider, - INSTRUMENTATION_SCOPE, - IN_PROCESS_TARGET_PREFIX, - ) -} - -#[cfg(test)] -pub(crate) async fn test_lock() -> tokio::sync::MutexGuard<'static, ()> { - static LOCK: tokio::sync::Mutex<()> = tokio::sync::Mutex::const_new(()); - static INITIALIZED: std::sync::LazyLock<()> = std::sync::LazyLock::new(|| { - tracing::subscriber::set_global_default(tracing_subscriber::registry()) - .expect("test tracing subscriber installs once"); - }); - - let guard = LOCK.lock().await; - std::sync::LazyLock::force(&INITIALIZED); - guard -} +pub const TRACING: openshell_otel::ComputeDriverTracing = openshell_otel::compute_driver_tracing!(); #[cfg(test)] mod tests { - use openshell_otel_test_support::OtlpTestServer; - use tracing_subscriber::layer::SubscriberExt as _; - - #[test] - fn compute_driver_rpc_names_are_explicitly_mapped_and_schema_bounded() { - for (rpc, operation, method) in [ - ( - "GetCapabilities", - "driver.get_capabilities", - "get_capabilities", - ), - ( - "GetGatewayListenerRequirements", - "driver.get_gateway_listener_requirements", - "get_gateway_listener_requirements", - ), - ( - "ValidateSandboxCreate", - "driver.validate_sandbox_create", - "validate_sandbox_create", - ), - ("CreateSandbox", "driver.create_sandbox", "create_sandbox"), - ("GetSandbox", "driver.get_sandbox", "get_sandbox"), - ("ListSandboxes", "driver.list_sandboxes", "list_sandboxes"), - ("StopSandbox", "driver.stop_sandbox", "stop_sandbox"), - ("StartSandbox", "driver.start_sandbox", "start_sandbox"), - ("DeleteSandbox", "driver.delete_sandbox", "delete_sandbox"), - ( - "WatchSandboxes", - "driver.watch_sandboxes", - "watch_sandboxes", - ), - ( - "EnsureWorkspace", - "driver.ensure_workspace", - "ensure_workspace", - ), - ( - "DeleteWorkspace", - "driver.delete_workspace", - "delete_workspace", - ), - ] { - assert_eq!( - super::compute_driver_rpc_operation(&format!( - "/openshell.compute.v1.ComputeDriver/{rpc}" - )), - (operation, method), - ); - } - assert_eq!( - super::compute_driver_rpc_operation( - "/openshell.compute.v1.ComputeDriver/AttackerControlled12345" - ), - ("driver.unknown", "unknown"), - ); - } - #[tokio::test(flavor = "multi_thread", worker_threads = 2)] - async fn podman_driver_spans_reach_otlp_collector_with_resource_identity() { - let _tracing_lock = super::test_lock().await; - let collector = OtlpTestServer::start().await; - - let (provider, error) = - super::provider_for(Some(collector.endpoint()), Some("production-us-west")); - assert!(error.is_none()); - let provider = provider.expect("provider"); - let subscriber = tracing_subscriber::registry().with(super::layer(&provider)); - tracing::subscriber::with_default(subscriber, || { - let span = tracing::info_span!("podman.create", sandbox.id = "sb-otlp"); - drop(span.enter()); - drop(span); - }); - provider.force_flush().unwrap(); - collector.wait_for_export().await; - provider.shutdown().unwrap(); - let received = collector.shutdown().await; - - assert!( - received - .spans - .iter() - .any(|span| span.name == "podman.create") - ); - assert_eq!(received.gateway_names, ["production-us-west"]); - assert!( - received - .service_names - .iter() - .any(|name| name == "openshell-driver-podman") - ); + async fn driver_spans_reach_otlp_collector_with_resource_identity() { + openshell_otel_test_support::assert_compute_driver_tracing( + super::TRACING, + openshell_core::VERSION, + "podman.create", + ) + .await; } } diff --git a/crates/openshell-driver-podman/src/socket_discovery.rs b/crates/openshell-driver-podman/src/socket_discovery.rs new file mode 100644 index 000000000..e7a6c79e7 --- /dev/null +++ b/crates/openshell-driver-podman/src/socket_discovery.rs @@ -0,0 +1,402 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! Podman socket discovery for local and machine-backed installations. + +use std::path::PathBuf; +use std::process::{Command, Stdio}; +use std::time::{Duration, Instant}; + +const DISCOVERY_TIMEOUT: Duration = Duration::from_secs(5); +const DISCOVERY_POLL_INTERVAL: Duration = Duration::from_millis(25); + +/// Return the first responsive well-known socket, then ask the Podman CLI for +/// the active native or machine-backed connection. +pub fn detect_socket() -> Option { + openshell_core::local_api_socket::first_responsive_socket(&socket_candidates(), |response| { + openshell_core::local_api_socket::http_response_is_success(response) + && openshell_core::local_api_socket::contains_ascii(response, b"Libpod-Api-Version:") + }) + .or_else(discover_socket) +} + +fn socket_candidates() -> Vec { + let mut candidates = Vec::new(); + if let Some(path) = env_var_nonempty("OPENSHELL_PODMAN_SOCKET") { + candidates.push(PathBuf::from(path)); + } + if let Some(runtime_dir) = std::env::var_os("XDG_RUNTIME_DIR") { + candidates.push(PathBuf::from(runtime_dir).join("podman/podman.sock")); + } + #[cfg(target_os = "linux")] + candidates.push(PathBuf::from(format!( + "/run/user/{}/podman/podman.sock", + rustix::process::geteuid().as_raw() + ))); + if let Some(home) = std::env::var_os("HOME") { + candidates + .push(PathBuf::from(home).join(".local/share/containers/podman/machine/podman.sock")); + } + candidates +} + +/// Query the same active connection selected by the Podman CLI. This covers +/// named/rootful machines and provider-specific forwarded socket locations that +/// do not have a stable well-known path. +fn discover_socket() -> Option { + let stdout = run_podman_capture(&["info", "--format", "json"])?; + + // A successful `podman info` proves this explicit socket is usable. A named + // connection has higher precedence, so only honor CONTAINER_HOST directly + // when no CONTAINER_CONNECTION is set. + if env_var_nonempty("CONTAINER_CONNECTION").is_none() + && let Some(path) = env_var_nonempty("CONTAINER_HOST") + .as_deref() + .and_then(unix_url_socket_path) + { + return Some(path); + } + + let info: serde_json::Value = serde_json::from_slice(&stdout).ok()?; + if !info["host"]["serviceIsRemote"].as_bool().unwrap_or(false) { + return parse_info_socket(&info); + } + + discover_machine_socket() +} + +fn discover_machine_socket() -> Option { + let connections = connection_list(); + let active = active_machine( + env_var_nonempty("CONTAINER_CONNECTION").as_deref(), + env_var_nonempty("CONTAINER_HOST").as_deref(), + connections.as_ref(), + )?; + machine_inspect_targets(&active) + .into_iter() + .find_map(|name| { + let stdout = run_podman_capture(&["machine", "inspect", &name])?; + let machines: serde_json::Value = serde_json::from_slice(&stdout).ok()?; + parse_machine_socket(&machines) + }) +} + +fn active_machine( + container_connection: Option<&str>, + container_host: Option<&str>, + connections: Option<&serde_json::Value>, +) -> Option { + if let Some(name) = container_connection.filter(|name| !name.trim().is_empty()) { + return Some(name.to_string()); + } + if let Some(host) = container_host.filter(|host| !host.trim().is_empty()) { + // An explicit non-machine endpoint must not fall back to an unrelated + // local machine. + return connection_name_for_uri(connections?, host); + } + default_machine_connection(connections).or_else(|| Some("podman-machine-default".to_string())) +} + +fn machine_inspect_targets(connection: &str) -> Vec { + let mut names = vec![connection.to_string()]; + if let Some(machine) = connection.strip_suffix("-root") + && !machine.is_empty() + { + names.push(machine.to_string()); + } + names +} + +fn connection_list() -> Option { + let stdout = run_podman_capture(&["system", "connection", "list", "--format", "json"])?; + serde_json::from_slice(&stdout).ok() +} + +fn default_machine_connection(connections: Option<&serde_json::Value>) -> Option { + connections? + .as_array()? + .iter() + .find(|connection| { + connection["Default"].as_bool().unwrap_or(false) + && connection["IsMachine"].as_bool().unwrap_or(false) + }) + .and_then(|connection| connection["Name"].as_str()) + .map(str::to_string) +} + +fn connection_name_for_uri(connections: &serde_json::Value, uri: &str) -> Option { + connections + .as_array()? + .iter() + .find(|connection| { + connection["IsMachine"].as_bool().unwrap_or(false) + && connection["URI"].as_str() == Some(uri) + }) + .and_then(|connection| connection["Name"].as_str()) + .map(str::to_string) +} + +fn parse_info_socket(info: &serde_json::Value) -> Option { + let path = info["host"]["remoteSocket"]["path"].as_str()?; + unix_url_socket_path(path).or_else(|| (!path.is_empty()).then(|| PathBuf::from(path))) +} + +fn parse_machine_socket(machines: &serde_json::Value) -> Option { + let path = machines.as_array()?.first()?["ConnectionInfo"]["PodmanSocket"]["Path"].as_str()?; + (!path.is_empty()).then(|| PathBuf::from(path)) +} + +fn unix_url_socket_path(url: &str) -> Option { + let path = url.trim().strip_prefix("unix://")?; + (!path.is_empty()).then(|| PathBuf::from(path)) +} + +fn env_var_nonempty(key: &str) -> Option { + std::env::var(key) + .ok() + .filter(|value| !value.trim().is_empty()) +} + +fn run_podman_capture(args: &[&str]) -> Option> { + run_bounded_command("podman", args, DISCOVERY_TIMEOUT) +} + +/// Capture stdout without allowing a stalled Podman machine, SSH transport, or +/// descendant holding the stdout pipe open to block gateway startup forever. +fn run_bounded_command(program: &str, args: &[&str], timeout: Duration) -> Option> { + use std::io::Read as _; + use std::sync::mpsc; + + let mut command = Command::new(program); + command + .args(args) + .stdin(Stdio::null()) + .stdout(Stdio::piped()) + .stderr(Stdio::null()); + set_new_process_group(&mut command); + let mut child = command.spawn().ok()?; + + let mut stdout = child.stdout.take()?; + let (sender, receiver) = mpsc::channel(); + std::thread::spawn(move || { + let mut output = Vec::new(); + let _ = stdout.read_to_end(&mut output); + let _ = sender.send(output); + }); + + let deadline = Instant::now() + timeout; + let status = loop { + match child.try_wait() { + Ok(Some(status)) => break Some(status), + Ok(None) if Instant::now() < deadline => { + std::thread::sleep(DISCOVERY_POLL_INTERVAL); + } + Ok(None) | Err(_) => break None, + } + }; + + let Some(status) = status else { + terminate_process_group(&mut child); + let _ = child.wait(); + return None; + }; + + match receiver.recv_timeout(deadline.saturating_duration_since(Instant::now())) { + Ok(output) if status.success() => Some(output), + Ok(_) => None, + Err(_) => { + terminate_process_group(&mut child); + None + } + } +} + +#[cfg(unix)] +fn set_new_process_group(command: &mut Command) { + use std::os::unix::process::CommandExt as _; + command.process_group(0); +} + +#[cfg(not(unix))] +fn set_new_process_group(_command: &mut Command) {} + +#[cfg(unix)] +fn terminate_process_group(child: &mut std::process::Child) { + let pid = i32::try_from(child.id()).unwrap_or(i32::MAX); + let group = nix::unistd::Pid::from_raw(-pid); + let _ = nix::sys::signal::kill(group, nix::sys::signal::Signal::SIGKILL); + let _ = child.kill(); +} + +#[cfg(not(unix))] +fn terminate_process_group(child: &mut std::process::Child) { + let _ = child.kill(); +} + +#[cfg(test)] +mod tests { + use super::*; + use serde_json::json; + + #[test] + fn parse_info_socket_rejects_missing_or_empty_paths() { + assert_eq!( + parse_info_socket(&json!({"host": {"remoteSocket": {}}})), + None + ); + assert_eq!( + parse_info_socket(&json!({"host": {"remoteSocket": {"path": ""}}})), + None + ); + } + + #[test] + fn parse_machine_socket_rejects_missing_socket_or_machine() { + assert_eq!(parse_machine_socket(&json!([])), None); + assert_eq!(parse_machine_socket(&json!([{"ConnectionInfo": {}}])), None); + } + + #[test] + fn parses_native_and_machine_socket_paths() { + let info = json!({"host": {"remoteSocket": {"path": "unix:///run/user/1000/podman.sock"}}}); + assert_eq!( + parse_info_socket(&info), + Some(PathBuf::from("/run/user/1000/podman.sock")) + ); + + let machine = json!([{"ConnectionInfo": {"PodmanSocket": {"Path": "/tmp/machine.sock"}}}]); + assert_eq!( + parse_machine_socket(&machine), + Some(PathBuf::from("/tmp/machine.sock")) + ); + } + + #[test] + fn resolves_named_rootful_and_default_machine_connections() { + let connections = json!([ + {"Name": "team-machine-root", "URI": "ssh://team", "Default": true, "IsMachine": true}, + {"Name": "remote", "URI": "ssh://remote", "Default": false, "IsMachine": false} + ]); + assert_eq!( + active_machine(Some("team-machine-root"), None, Some(&connections)), + Some("team-machine-root".to_string()) + ); + assert_eq!( + machine_inspect_targets("team-machine-root"), + ["team-machine-root", "team-machine"] + ); + assert_eq!( + active_machine(None, None, Some(&connections)), + Some("team-machine-root".to_string()) + ); + } + + #[test] + fn explicit_non_machine_endpoint_does_not_guess_a_machine() { + let connections = json!([ + {"Name": "local", "URI": "ssh://local", "Default": true, "IsMachine": true}, + {"Name": "remote", "URI": "ssh://remote", "Default": false, "IsMachine": false} + ]); + assert_eq!( + active_machine(None, Some("ssh://remote"), Some(&connections)), + None + ); + } + + #[test] + fn container_host_maps_to_the_matching_machine_connection() { + let connections = json!([ + {"Name": "work", "URI": "ssh://core@127.0.0.1:5555/run/podman.sock", "Default": false, "IsMachine": true}, + {"Name": "podman-machine-default", "URI": "ssh://core@127.0.0.1:4444/run/podman.sock", "Default": true, "IsMachine": true} + ]); + assert_eq!( + active_machine( + None, + Some("ssh://core@127.0.0.1:5555/run/podman.sock"), + Some(&connections), + ), + Some("work".to_string()) + ); + } + + #[test] + fn unix_url_socket_path_only_accepts_nonempty_unix_urls() { + assert_eq!( + unix_url_socket_path("unix:///run/user/1000/podman.sock"), + Some(PathBuf::from("/run/user/1000/podman.sock")) + ); + assert_eq!(unix_url_socket_path("ssh://core@127.0.0.1/x"), None); + assert_eq!(unix_url_socket_path("tcp://127.0.0.1:2375"), None); + assert_eq!(unix_url_socket_path("unix://"), None); + } + + #[cfg(unix)] + #[test] + fn bounded_command_captures_stdout_on_success() { + assert_eq!( + run_bounded_command("printf", &["hello"], Duration::from_secs(5)), + Some(b"hello".to_vec()) + ); + } + + #[cfg(unix)] + #[test] + fn bounded_command_returns_none_on_nonzero_exit_or_missing_program() { + assert_eq!( + run_bounded_command("false", &[], Duration::from_secs(5)), + None + ); + assert_eq!( + run_bounded_command( + "openshell-nonexistent-binary-xyz", + &[], + Duration::from_secs(5), + ), + None + ); + } + + #[cfg(unix)] + #[test] + fn bounded_command_kills_child_that_exceeds_deadline() { + let start = Instant::now(); + let result = run_bounded_command("sleep", &["30"], Duration::from_millis(200)); + assert_eq!(result, None); + assert!( + start.elapsed() < Duration::from_secs(5), + "bounded command did not return promptly" + ); + } + + #[cfg(unix)] + #[test] + fn bounded_command_bounds_drain_when_in_group_descendant_holds_stdout() { + let start = Instant::now(); + let result = run_bounded_command( + "sh", + &["-c", "sleep 30 & echo done"], + Duration::from_millis(300), + ); + assert_eq!(result, None); + assert!( + start.elapsed() < Duration::from_secs(2), + "drain blocked on a descendant holding stdout" + ); + } + + #[cfg(unix)] + #[test] + fn bounded_command_bounds_drain_when_descendant_escapes_process_group() { + let start = Instant::now(); + let result = run_bounded_command( + "bash", + &["-c", "set -m; sleep 5 & echo done"], + Duration::from_millis(300), + ); + assert_eq!(result, None); + assert!( + start.elapsed() < Duration::from_secs(2), + "drain blocked on a descendant that escaped the process group" + ); + } +} diff --git a/crates/openshell-driver-podman/src/watcher.rs b/crates/openshell-driver-podman/src/watcher.rs index 0c94bf72b..57ea45e6e 100644 --- a/crates/openshell-driver-podman/src/watcher.rs +++ b/crates/openshell-driver-podman/src/watcher.rs @@ -28,6 +28,7 @@ const CONDITION_RUNNING: &str = "ContainerRunning"; const CONDITION_STARTING: &str = "ContainerStarting"; use openshell_core::driver_utils::{ CONDITION_EXITED, CONDITION_RUNTIME_RESTART, CONDITION_STOPPED, + CONDITION_WORKSPACE_VALIDATION_FAILED, SUPERVISOR_EXIT_WORKSPACE_VALIDATION_FAILED, }; pub type WatchStream = @@ -122,15 +123,12 @@ fn deleted_event(sandbox_id: String) -> WatchSandboxesEvent { /// drops (daemon restart, socket error, or clean shutdown), the stream /// terminates with a final error item and stops producing events. /// -/// Callers are responsible for reconnecting by calling [`start_watch`] again. -/// The server's `ComputeRuntime::watch_loop` in `openshell-server` provides -/// this behaviour with a 2-second backoff: when the stream terminates with an -/// error, `watch_loop` sleeps and then calls `watch_sandboxes()` again, which -/// ultimately calls `start_watch()` again and re-syncs state. +/// Callers are responsible for reconnecting by calling [`start_watch`] again +/// and re-synchronizing state. /// /// **Do not add reconnection logic inside this function.** A local reconnect -/// would race with `watch_loop`'s retry and produce duplicate initial-sync -/// events that corrupt the server's sandbox index. +/// would race with the consumer's retry and produce duplicate initial-sync +/// events. pub async fn start_watch( client: PodmanClient, lifecycle_event_fences: LifecycleEventFences, @@ -456,6 +454,11 @@ fn condition_from_state(state: &ContainerState) -> DriverCondition { "OOMKilled", "Container was killed by the OOM killer".to_string(), ) + } else if state.exit_code == i64::from(SUPERVISOR_EXIT_WORKSPACE_VALIDATION_FAILED) { + ( + CONDITION_WORKSPACE_VALIDATION_FAILED, + "OCI WorkingDir is not usable by the sandbox identity".to_string(), + ) } else if matches!(state.exit_code, 137 | 143) { ( CONDITION_RUNTIME_RESTART, @@ -615,6 +618,24 @@ mod tests { assert!(cond.message.contains("code 1")); } + #[test] + fn condition_workspace_validation_exit_is_reported_explicitly() { + let state = ContainerState { + status: "exited".to_string(), + running: false, + exit_code: i64::from(SUPERVISOR_EXIT_WORKSPACE_VALIDATION_FAILED), + oom_killed: false, + health: None, + started_at: None, + finished_at: Some("2026-04-14T12:00:00Z".to_string()), + }; + + let cond = condition_from_state(&state); + + assert_eq!(cond.reason, CONDITION_WORKSPACE_VALIDATION_FAILED); + assert!(cond.message.contains("WorkingDir")); + } + #[test] fn condition_signal_kill_is_runtime_restart() { // 137 (128+SIGKILL) and 143 (128+SIGTERM) are external terminations — diff --git a/crates/openshell-driver-vault/Cargo.toml b/crates/openshell-driver-vault/Cargo.toml index 2d3878ed6..fe12c2e20 100644 --- a/crates/openshell-driver-vault/Cargo.toml +++ b/crates/openshell-driver-vault/Cargo.toml @@ -21,6 +21,7 @@ clap = { workspace = true } futures = { workspace = true } miette = { workspace = true } reqwest = { workspace = true } +rustls = { workspace = true } serde = { workspace = true } serde_json = { workspace = true } sha2 = { workspace = true } diff --git a/crates/openshell-driver-vault/src/lib.rs b/crates/openshell-driver-vault/src/lib.rs index d93200570..9fee593fb 100644 --- a/crates/openshell-driver-vault/src/lib.rs +++ b/crates/openshell-driver-vault/src/lib.rs @@ -126,6 +126,7 @@ impl VaultCredentialDriver { pub fn from_config(config: &toml::Table) -> CoreResult { let settings = VaultDriverSettings::from_table(config)?; let timeout_secs = timeout_secs(config)?; + let _ = rustls::crypto::aws_lc_rs::default_provider().install_default(); let client = reqwest::Client::builder() .timeout(Duration::from_secs(timeout_secs)) .build() diff --git a/crates/openshell-driver-vm/Cargo.toml b/crates/openshell-driver-vm/Cargo.toml index ebcb9d2bc..0d250b6e7 100644 --- a/crates/openshell-driver-vm/Cargo.toml +++ b/crates/openshell-driver-vm/Cargo.toml @@ -22,6 +22,7 @@ path = "src/main.rs" openshell-core = { path = "../openshell-core", default-features = false } openshell-otel = { path = "../openshell-otel" } openshell-policy = { path = "../openshell-policy" } +openshell-driver-podman = { path = "../openshell-driver-podman" } openshell-vfio = { path = "../openshell-vfio" } bollard = { version = "0.20", features = ["ssh"] } @@ -59,6 +60,15 @@ default = ["telemetry"] ## default; build with `--no-default-features` for a telemetry-free VM driver ## that reports telemetry disabled to the sandboxes it launches. telemetry = ["openshell-core/telemetry"] +## Convenience alias: every default feature except `telemetry`. Build a +## telemetry-free VM driver with +## `--no-default-features --features defaults-without-telemetry` and stay +## correct as new default features are added. Cargo cannot subtract a single +## default feature, so this alias must be paired with `--no-default-features`; +## enabling it alongside `telemetry` is a compile error rather than a silent +## telemetry-on build. Kept in sync with `default` by +## `rust:verify:defaults-without-telemetry`. +defaults-without-telemetry = [] [dev-dependencies] openshell-otel-test-support = { path = "../openshell-otel-test-support" } diff --git a/crates/openshell-driver-vm/README.md b/crates/openshell-driver-vm/README.md index f455beaeb..05e6df454 100644 --- a/crates/openshell-driver-vm/README.md +++ b/crates/openshell-driver-vm/README.md @@ -9,7 +9,7 @@ Standalone libkrun-backed [`ComputeDriver`](../../proto/compute_driver.proto) fo ```mermaid flowchart LR subgraph host["Host process"] - gateway["openshell-server
(compute::vm::spawn)"] + gateway["openshell-gateway
(vm::spawn)"] driver["openshell-driver-vm
├── libkrun (VM)
├── gvproxy (net)
└── openshell-sandbox.zst"] gateway <-->|"gRPC over UDS
compute-driver.sock"| driver end @@ -102,7 +102,7 @@ mise run vm:supervisor # if openshell-sandbox.zst is not already presen # 2. Build both binaries with the staged artifacts embedded OPENSHELL_VM_RUNTIME_COMPRESSED_DIR=$PWD/target/vm-runtime-compressed \ - cargo build -p openshell-server -p openshell-driver-vm + cargo build -p openshell-gateway -p openshell-driver-vm # 3. macOS only: codesign the driver for Hypervisor.framework codesign \ @@ -154,6 +154,14 @@ Select the VM driver with `--drivers vm`, `OPENSHELL_DRIVERS=vm`, or `compute_dr | `guest_tls_ca` | unset | CA cert for the guest's mTLS client bundle. Required when `grpc_endpoint` uses `https://`. | | `guest_tls_cert` | unset | Guest client certificate. | | `guest_tls_key` | unset | Guest client private key. | +| `https_proxy` | unset | Corporate forward proxy (`http://host:port` or `https://host:port`) the in-guest supervisor chains policy-approved TLS CONNECT egress through. On the libkrun backend a proxy on the gateway host's loopback must be addressed as `http://host.openshell.internal:` — guest egress leaves through gvproxy, which NATs `192.168.127.254` to the host's `127.0.0.1`. The QEMU/TAP backend (GPU sandboxes) has no such NAT and its nftables rules expose only the gateway port to the guest, so a gateway-host proxy URL is rejected at launch there; use an address routable from the guest's masqueraded egress. | +| `no_proxy` | unset | Comma-separated bypass list for the corporate proxy only. OpenShell policy evaluation still applies. | +| `proxy_auth_file` | unset | Gateway-host path to a `user:pass` credential file. Staged root-only into the per-sandbox overlay and removed with the sandbox. | +| `proxy_auth_allow_insecure` | unset | Required with `proxy_auth_file` against an `http://` proxy: acknowledges that Basic auth is cleartext on the connection to the proxy. | +| `proxy_connect_by_hostname` | unset | Send hostnames rather than validated IPs in CONNECT. Last resort for proxies whose ACLs reject IP targets. | +| `proxy_ca_bundle` | unset | Gateway-host path to a PEM CA bundle trusted for an `https://` proxy and for certificates a TLS-intercepting proxy re-signs. | + +The proxy settings are operator-owned and deployment-level: they are not accepted through `template.driver_config.vm`, and they reach the supervisor on its command line through a per-sandbox argument file the driver writes into the overlay upperdir on every launch, so a sandbox image cannot forge or shadow them. Every present-but-invalid value is fatal at gateway or sandbox startup rather than degrading to a direct dial. See [`openshell-gateway --help`](../openshell-server/src/cli.rs) for the gateway process flag surface. @@ -254,9 +262,9 @@ Each table is created atomically via `nft -f` on VM start and torn down atomical - Matching rustup target: `rustup target add aarch64-unknown-linux-gnu` (or `x86_64-unknown-linux-gnu` for an amd64 guest) - `cargo install --locked cargo-zigbuild` and `brew install zig` (or distro equivalent). `vm:supervisor` uses `cargo zigbuild` to cross-compile the in-VM `openshell-sandbox` supervisor binary. - [mise](https://mise.jdx.dev/) task runner -- Docker or Podman socket on the local CLI/gateway host when using - `openshell sandbox create --from ./Dockerfile` or `--from ./dir`; the CLI - builds the image and the VM driver exports it via the local container engine. +- Docker or Podman socket on the local CLI/gateway host when building an image + before `openshell sandbox create --from `; the VM driver exports the + image via the local container engine. Docker is tried first; if unavailable, the driver falls back to the Podman socket. On Linux, enable the Podman API socket with `systemctl --user start podman.socket` @@ -291,5 +299,5 @@ the user explicitly overrides it. ## TODOs -- The gateway still configures the driver via CLI args; this will move to a gRPC bootstrap call so the driver interface is uniform across backends. See the `TODO(driver-abstraction)` notes in `crates/openshell-server/src/lib.rs` and `crates/openshell-server/src/compute/vm.rs`. +- The gateway still configures the driver via CLI args; this will move to a gRPC bootstrap call so the driver interface is uniform across backends. See the `TODO(driver-abstraction)` note in `crates/openshell-gateway/src/vm.rs`. - macOS local builds are codesigned by `tasks/scripts/gateway-vm.sh`; the generated Homebrew formula signs the release tarball driver for local installs. diff --git a/crates/openshell-driver-vm/runtime/README.md b/crates/openshell-driver-vm/runtime/README.md index 11aab67f4..b686874ba 100644 --- a/crates/openshell-driver-vm/runtime/README.md +++ b/crates/openshell-driver-vm/runtime/README.md @@ -41,7 +41,7 @@ mise run vm:supervisor # Build the gateway and VM driver with embedded runtime artifacts OPENSHELL_VM_RUNTIME_COMPRESSED_DIR=$PWD/target/vm-runtime-compressed \ - cargo build -p openshell-server -p openshell-driver-vm + cargo build -p openshell-gateway -p openshell-driver-vm ``` Use `FROM_SOURCE=1 mise run vm:setup` to build the runtime from source instead diff --git a/crates/openshell-driver-vm/scripts/openshell-vm-sandbox-init.sh b/crates/openshell-driver-vm/scripts/openshell-vm-sandbox-init.sh index 14dbc0466..32d6ed1df 100644 --- a/crates/openshell-driver-vm/scripts/openshell-vm-sandbox-init.sh +++ b/crates/openshell-driver-vm/scripts/openshell-vm-sandbox-init.sh @@ -192,6 +192,67 @@ prepare_guest_image_rootfs() { rm -rf "$payload_dir" } +# Driver-owned arguments appended to the supervisor's command line. +# +# The VM driver cannot build the supervisor's argv the way the container +# drivers do, so it writes the arguments it chose into the overlay upperdir +# and this script appends them verbatim. Populated by +# read_supervisor_extra_args; empty until then. +SUPERVISOR_EXTRA_ARGS=() + +# Upper bound on driver-supplied supervisor arguments. +# +# The corporate proxy settings are the only producer today and top out at ten +# entries. The cap exists so a corrupt or oversized file cannot expand into an +# unbounded command line. +SUPERVISOR_EXTRA_ARGS_MAX=32 + +read_supervisor_extra_args() { + # Read the driver-authored supervisor argument list, one argument per + # line, verbatim -- no word splitting, globbing, or expansion, so values + # containing spaces (e.g. a NO_PROXY list) survive intact. + # + # Security: this is the operator-owned egress boundary. The driver writes + # this file into the overlay upperdir on every launch, including an empty + # file when it has no arguments to pass, so the upperdir copy always + # shadows the read-only image layer. A sandbox image can therefore neither + # supply its own supervisor arguments by baking a file at this path nor + # disable the operator's by omitting one. A missing file means the driver + # passed nothing; a file it cannot read means the overlay is broken, and + # we fail closed rather than start a supervisor with a silently truncated + # egress configuration. + local args_file + args_file="$(root_path /opt/openshell/supervisor-args)" + + SUPERVISOR_EXTRA_ARGS=() + if [ ! -f "$args_file" ]; then + return 0 + fi + if [ ! -r "$args_file" ]; then + ts "FATAL: supervisor argument list ${args_file} is not readable" + exit 1 + fi + + local arg + while IFS= read -r arg; do + # render_guest_supervisor_args never emits a blank line, so one means + # the file was truncated or tampered with after the driver wrote it. + if [ -z "$arg" ]; then + ts "FATAL: empty entry in supervisor argument list" + exit 1 + fi + if [ "${#SUPERVISOR_EXTRA_ARGS[@]}" -ge "$SUPERVISOR_EXTRA_ARGS_MAX" ]; then + ts "FATAL: supervisor argument list exceeds ${SUPERVISOR_EXTRA_ARGS_MAX} entries" + exit 1 + fi + SUPERVISOR_EXTRA_ARGS+=("$arg") + done < "$args_file" + + if [ "${#SUPERVISOR_EXTRA_ARGS[@]}" -gt 0 ]; then + ts "supervisor arguments from driver: ${#SUPERVISOR_EXTRA_ARGS[@]} entries" + fi +} + exec_supervisor_in_newroot() { local chroot_bin local bootstrap="/.openshell-bootstrap" @@ -214,14 +275,16 @@ exec_supervisor_in_newroot() { "${bootstrap}/lib64/ld-linux-aarch64.so.1"; do if [ -x "/newroot${loader}" ]; then lib_path="${bootstrap}/lib:${bootstrap}/lib64:${bootstrap}/usr/lib:${bootstrap}/usr/lib64:${bootstrap}/lib/aarch64-linux-gnu:${bootstrap}/lib/x86_64-linux-gnu:${bootstrap}/usr/lib/aarch64-linux-gnu:${bootstrap}/usr/lib/x86_64-linux-gnu" - exec "$chroot_bin" /newroot "$loader" --library-path "$lib_path" "$supervisor" --workdir /sandbox + exec "$chroot_bin" /newroot "$loader" --library-path "$lib_path" \ + "$supervisor" --workdir /sandbox "${SUPERVISOR_EXTRA_ARGS[@]+"${SUPERVISOR_EXTRA_ARGS[@]}"}" fi done - exec "$chroot_bin" /newroot "$supervisor" --workdir /sandbox + exec "$chroot_bin" /newroot "$supervisor" --workdir /sandbox "${SUPERVISOR_EXTRA_ARGS[@]+"${SUPERVISOR_EXTRA_ARGS[@]}"}" fi if [ -x /newroot/opt/openshell/bin/openshell-sandbox ]; then - exec "$chroot_bin" /newroot /opt/openshell/bin/openshell-sandbox --workdir /sandbox + exec "$chroot_bin" /newroot /opt/openshell/bin/openshell-sandbox \ + --workdir /sandbox "${SUPERVISOR_EXTRA_ARGS[@]+"${SUPERVISOR_EXTRA_ARGS[@]}"}" fi done @@ -833,11 +896,13 @@ if [ -n "${OPENSHELL_SANDBOX_ID:-}" ]; then ts "OPENSHELL_SANDBOX_ID=${OPENSHELL_SANDBOX_ID}" fi +read_supervisor_extra_args + ts "starting openshell-sandbox supervisor" if [ "${ROOT_PREFIX:-}" = "/newroot" ]; then exec_supervisor_in_newroot fi -exec /opt/openshell/bin/openshell-sandbox --workdir /sandbox +exec /opt/openshell/bin/openshell-sandbox --workdir /sandbox "${SUPERVISOR_EXTRA_ARGS[@]+"${SUPERVISOR_EXTRA_ARGS[@]}"}" } if [ "${1:-}" != "--post-overlay" ]; then diff --git a/crates/openshell-driver-vm/src/driver.rs b/crates/openshell-driver-vm/src/driver.rs index 2de65c3ad..896daa48a 100644 --- a/crates/openshell-driver-vm/src/driver.rs +++ b/crates/openshell-driver-vm/src/driver.rs @@ -18,12 +18,13 @@ use bollard::Docker; use bollard::errors::Error as BollardError; use bollard::models::ContainerCreateBody; use bollard::query_parameters::{CreateContainerOptionsBuilder, RemoveContainerOptionsBuilder}; -use flate2::read::GzDecoder; +use flate2::read::{GzDecoder, MultiGzDecoder}; use futures::{Stream, StreamExt, TryStreamExt}; use nix::errno::Errno; use nix::sys::signal::{Signal, kill}; use nix::unistd::Pid; use oci_client::client::{Client as OciClient, ClientConfig}; +use oci_client::errors::{OciDistributionError, OciErrorCode}; use oci_client::manifest::{ ImageIndexEntry, OCI_IMAGE_MEDIA_TYPE, OciDescriptor, OciImageManifest, }; @@ -37,14 +38,15 @@ use openshell_core::progress::{ format_bytes, mark_progress_active, mark_progress_complete, mark_progress_detail, }; use openshell_core::proto::compute::v1::{ - CreateSandboxRequest, CreateSandboxResponse, DeleteSandboxRequest, DeleteSandboxResponse, - DeleteWorkspaceRequest, DeleteWorkspaceResponse, DriverCondition as SandboxCondition, - DriverPlatformEvent as PlatformEvent, DriverSandbox as Sandbox, - DriverSandboxStatus as SandboxStatus, DriverSandboxTemplate as SandboxTemplate, - EnsureWorkspaceRequest, EnsureWorkspaceResponse, GetCapabilitiesRequest, - GetCapabilitiesResponse, GetGatewayListenerRequirementsRequest, + CpuResourceCapabilities, CreateSandboxRequest, CreateSandboxResponse, DeleteSandboxRequest, + DeleteSandboxResponse, DeleteWorkspaceRequest, DeleteWorkspaceResponse, + DriverCondition as SandboxCondition, DriverPlatformEvent as PlatformEvent, + DriverSandbox as Sandbox, DriverSandboxStatus as SandboxStatus, + DriverSandboxTemplate as SandboxTemplate, EnsureWorkspaceRequest, EnsureWorkspaceResponse, + GetCapabilitiesRequest, GetCapabilitiesResponse, GetGatewayListenerRequirementsRequest, GetGatewayListenerRequirementsResponse, GetSandboxRequest, GetSandboxResponse, - ListSandboxesRequest, ListSandboxesResponse, StartSandboxRequest, StartSandboxResponse, + GpuResourceCapabilities, ListSandboxesRequest, ListSandboxesResponse, + MemoryResourceCapabilities, ResourceCapabilities, StartSandboxRequest, StartSandboxResponse, StopSandboxRequest, StopSandboxResponse, ValidateSandboxCreateRequest, ValidateSandboxCreateResponse, WatchSandboxesDeletedEvent, WatchSandboxesEvent, WatchSandboxesPlatformEvent, WatchSandboxesRequest, WatchSandboxesSandboxEvent, @@ -59,8 +61,9 @@ use prost::Message; use sha2::{Digest, Sha256}; use std::collections::{HashMap, HashSet}; use std::fs; -use std::io::Read; -use std::net::Ipv4Addr; +use std::future::Future; +use std::io::{BufRead, BufReader, BufWriter, Read, Write}; +use std::net::{IpAddr, Ipv4Addr}; #[cfg(unix)] use std::os::unix::fs::PermissionsExt; use std::path::{Component, Path, PathBuf}; @@ -86,6 +89,12 @@ const DEFAULT_MEM_MIB: u32 = 2048; const DEFAULT_OVERLAY_DISK_MIB: u64 = 4096; const DEFAULT_REGISTRY_LAYER_DOWNLOAD_CONCURRENCY: usize = 4; const MAX_REGISTRY_LAYER_DOWNLOAD_CONCURRENCY: usize = 16; +const REGISTRY_REQUEST_MAX_ATTEMPTS: usize = 4; +const REGISTRY_RETRY_INITIAL_DELAY: Duration = Duration::from_millis(250); +const REGISTRY_RETRY_MAX_DELAY: Duration = Duration::from_secs(1); +/// 10 GiB — configurable via `rootfs_tar_max_bytes`. +const DEFAULT_ROOTFS_TAR_MAX_BYTES: u64 = 10 * 1024 * 1024 * 1024; +const ROOTFS_TAR_STAGING_DIR: &str = "rootfs-tar-staging"; #[derive(Debug, Clone, Default, serde::Deserialize)] #[serde(default, deny_unknown_fields)] @@ -95,6 +104,7 @@ struct VmSandboxDriverConfig { deserialize_with = "deserialize_optional_non_empty_string_list" )] gpu_device_ids: Option>, + rootfs_tar_path: Option, } impl VmSandboxDriverConfig { @@ -132,7 +142,7 @@ impl VmSandboxDriverConfig { /// Code paths route via `GVPROXY_HOST_LOOPBACK_ALIAS` (DNS / /etc/hosts) /// instead so logs stay readable; this constant is kept for documentation /// and parity with the guest init script. -#[allow(dead_code)] +#[allow(dead_code)] // Documentation/parity anchor; all routing goes via the alias. const GVPROXY_HOST_LOOPBACK_IP: &str = "192.168.127.254"; const OPENSHELL_HOST_GATEWAY_ALIAS: &str = "host.openshell.internal"; /// Hostname gvproxy resolves (via its embedded DNS) to the host-loopback IP. @@ -162,6 +172,19 @@ const GUEST_INIT_DROPIN_DIR: &str = openshell_core::container_paths::VM_GUEST_IN /// upperdir on every launch, so the image cannot forge or shadow it. const GUEST_INIT_DROPIN_MANIFEST: &str = openshell_core::container_paths::VM_GUEST_INIT_DROPIN_MANIFEST; +/// Guest path of the root-only corporate proxy credential staged by the driver. +const GUEST_UPSTREAM_PROXY_AUTH_PATH: &str = + openshell_core::container_paths::VM_GUEST_UPSTREAM_PROXY_AUTH_PATH; +/// Guest path of the corporate proxy CA bundle staged by the driver. +const GUEST_PROXY_CA_PATH: &str = openshell_core::container_paths::VM_GUEST_PROXY_CA_PATH; +/// Guest path of the driver-authored supervisor argument list. +/// +/// The counterpart of [`GUEST_INIT_DROPIN_MANIFEST`] for the supervisor's own +/// command line: written into the overlay upperdir on every launch (empty +/// when there is nothing to pass) so the guest appends exactly the arguments +/// the driver chose and a sandbox image cannot forge or shadow them. +const GUEST_SUPERVISOR_ARGS_PATH: &str = + openshell_core::container_paths::VM_GUEST_SUPERVISOR_ARGS_PATH; const IMAGE_CACHE_ROOT_DIR: &str = "images"; const IMAGE_CACHE_ROOTFS_IMAGE: &str = "rootfs.ext4"; const OVERLAY_TEMPLATE_CACHE_DIR: &str = "overlay-templates"; @@ -217,7 +240,7 @@ enum GuestImagePayloadSource { LocalDocker { rootfs_archive: PathBuf }, } -#[derive(Debug, Clone, serde::Serialize, serde::Deserialize)] +#[derive(Clone, serde::Serialize, serde::Deserialize)] pub struct VmDriverConfig { pub openshell_endpoint: String, pub state_dir: PathBuf, @@ -243,6 +266,113 @@ pub struct VmDriverConfig { /// When empty, defaults to the resolved UID. #[serde(default, skip_serializing_if = "Option::is_none")] pub sandbox_gid: Option, + + /// Corporate forward proxy URL (`http://host:port` or `https://host:port`) + /// passed to the in-guest supervisor. + /// + /// The supervisor chains policy-approved TLS tunnels through this proxy + /// with HTTP CONNECT instead of dialing destinations directly. This is an + /// operator-owned egress boundary: it travels on the supervisor's argv, + /// which sandbox spec/template environment and image `ENV` cannot + /// influence. A proxy on the gateway host's loopback is reachable from the + /// guest only through the gvproxy host alias + /// (`host.openshell.internal`). + #[serde(default, skip_serializing_if = "Option::is_none")] + pub https_proxy: Option, + + /// Comma-separated `NO_PROXY` list passed alongside the proxy URL. + /// + /// Matching destinations are dialed directly instead of through the + /// corporate proxy. This bypasses only the corporate proxy, never + /// `OpenShell` policy evaluation. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub no_proxy: Option, + + /// Path (on the gateway host) to a file containing the corporate proxy + /// credential in `user:pass` form. + /// + /// The driver validates it at sandbox-create time and stages it into the + /// per-sandbox overlay at [`GUEST_UPSTREAM_PROXY_AUTH_PATH`], root-only. + /// Credentials are never embedded in the proxy URL and never reach the + /// guest environment. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub proxy_auth_file: Option, + + /// Explicit acknowledgement that proxy credentials are sent in cleartext. + /// + /// `Proxy-Authorization: Basic` over the plain-TCP connection to an + /// `http://` proxy is recoverable by anyone on the network path, so + /// [`Self::proxy_auth_file`] requires this acknowledgement. An `https://` + /// proxy carries the credential inside the verified TLS session and does + /// not need it. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub proxy_auth_allow_insecure: Option, + + /// Send the destination hostname in CONNECT requests instead of a + /// validated IP. + /// + /// The default binds the tunnel to an address that passed the sandbox's + /// SSRF and `allowed_ips` validation. Set this only when the proxy's ACLs + /// filter on hostnames and reject IP CONNECT targets: the proxy then + /// resolves the name itself and its own ACLs become the effective egress + /// control for proxied TLS. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub proxy_connect_by_hostname: Option, + + /// Path (on the gateway host) to a PEM CA bundle trusted for the + /// corporate proxy. + /// + /// The driver stages it into the per-sandbox overlay at + /// [`GUEST_PROXY_CA_PATH`] and passes that path via + /// `--upstream-proxy-ca-bundle`. It is trusted both for the handshake + /// with an `https://` proxy and for server certificates re-signed by a + /// TLS-intercepting proxy. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub proxy_ca_bundle: Option, + /// Directory where rootfs tar files must be staged before they can be + /// referenced in a `CreateSandbox` request. Defaults to `/rootfs-tar-staging`. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub rootfs_tar_staging_dir: Option, + /// Maximum rootfs tar file size in bytes. Defaults to 10 GiB. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub rootfs_tar_max_bytes: Option, +} + +/// Redacting `Debug` so a proxy URL or credential path never reaches a log. +/// +/// A validated proxy URL cannot embed credentials, but `Debug` can be emitted +/// before validation runs, so presence is logged rather than the value. +impl std::fmt::Debug for VmDriverConfig { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.debug_struct("VmDriverConfig") + .field("openshell_endpoint", &self.openshell_endpoint) + .field("state_dir", &self.state_dir) + .field("launcher_bin", &self.launcher_bin) + .field("default_image", &self.default_image) + .field("bootstrap_image", &self.bootstrap_image) + .field("log_level", &self.log_level) + .field("krun_log_level", &self.krun_log_level) + .field("vcpus", &self.vcpus) + .field("mem_mib", &self.mem_mib) + .field("overlay_disk_mib", &self.overlay_disk_mib) + .field("guest_tls_ca", &self.guest_tls_ca) + .field("guest_tls_cert", &self.guest_tls_cert) + .field("guest_tls_key", &self.guest_tls_key) + .field("gpu_enabled", &self.gpu_enabled) + .field("gpu_mem_mib", &self.gpu_mem_mib) + .field("gpu_vcpus", &self.gpu_vcpus) + .field("sandbox_uid", &self.sandbox_uid) + .field("sandbox_gid", &self.sandbox_gid) + .field("https_proxy", &self.https_proxy.is_some()) + .field("no_proxy", &self.no_proxy) + .field("proxy_auth_file", &self.proxy_auth_file.is_some()) + .field("proxy_auth_allow_insecure", &self.proxy_auth_allow_insecure) + .field("proxy_connect_by_hostname", &self.proxy_connect_by_hostname) + .field("proxy_ca_bundle", &self.proxy_ca_bundle) + .field("rootfs_tar_staging_dir", &self.rootfs_tar_staging_dir) + .field("rootfs_tar_max_bytes", &self.rootfs_tar_max_bytes) + .finish() + } } /// Default sandbox UID used by the VM driver when no config value is set. @@ -269,6 +399,14 @@ impl Default for VmDriverConfig { gpu_vcpus: 4, sandbox_uid: None, sandbox_gid: None, + https_proxy: None, + no_proxy: None, + proxy_auth_file: None, + proxy_auth_allow_insecure: None, + proxy_connect_by_hostname: None, + proxy_ca_bundle: None, + rootfs_tar_staging_dir: None, + rootfs_tar_max_bytes: None, } } } @@ -307,6 +445,40 @@ impl VmDriverConfig { Ok(()) } + /// Validate the operator's corporate upstream-proxy settings, fail-closed. + /// + /// Delegates to the validator shared with the Podman and Kubernetes + /// drivers and with the in-guest supervisor, so a value accepted here is + /// never rejected inside the guest — and no misconfiguration can silently + /// degrade to a direct dial. + /// + /// # Errors + /// + /// Returns a message naming the offending key. + pub fn validate_proxy_config(&self) -> Result<(), String> { + openshell_core::driver_utils::validate_upstream_proxy_settings( + &openshell_core::driver_utils::UpstreamProxySettings { + url: self.https_proxy.as_deref(), + no_proxy: self.no_proxy.as_deref(), + auth_file: self.proxy_auth_file.as_deref(), + auth_allow_insecure: self.proxy_auth_allow_insecure, + connect_by_hostname: self.proxy_connect_by_hostname, + ca_bundle: self.proxy_ca_bundle.as_deref(), + }, + ) + } + + fn rootfs_tar_staging_dir(&self) -> PathBuf { + self.rootfs_tar_staging_dir + .clone() + .unwrap_or_else(|| self.state_dir.join(ROOTFS_TAR_STAGING_DIR)) + } + + fn rootfs_tar_max_bytes(&self) -> u64 { + self.rootfs_tar_max_bytes + .unwrap_or(DEFAULT_ROOTFS_TAR_MAX_BYTES) + } + fn requires_tls_materials(&self) -> bool { self.openshell_endpoint.starts_with("https://") } @@ -447,6 +619,7 @@ impl VmDriver { .validate() .map_err(|err| err.message().to_string())?; config.validate_sandbox_identity()?; + config.validate_proxy_config()?; if config.openshell_endpoint.trim().is_empty() { return Err("openshell endpoint is required".to_string()); } @@ -477,6 +650,13 @@ impl VmDriver { image_cache_root.display() ) })?; + let staging_dir = config.rootfs_tar_staging_dir(); + create_private_dir_all(&staging_dir).await.map_err(|err| { + format!( + "failed to create rootfs tar staging dir '{}': {err}", + staging_dir.display() + ) + })?; let launcher_bin = if let Some(path) = config.launcher_bin.clone() { path @@ -517,6 +697,68 @@ impl VmDriver { Ok(driver) } + async fn validate_rootfs_tar_path(&self, raw: &Path) -> Result { + let staging_dir = self.config.rootfs_tar_staging_dir(); + let canonical_staging = tokio::fs::canonicalize(&staging_dir).await.map_err(|err| { + Status::internal(format!( + "rootfs tar staging dir not accessible at {}: {err}", + staging_dir.display() + )) + })?; + + let canonical = tokio::fs::canonicalize(raw).await.map_err(|err| { + Status::failed_precondition(format!( + "rootfs tar path not accessible at {}: {err}", + raw.display() + )) + })?; + + if !canonical.starts_with(&canonical_staging) { + return Err(Status::permission_denied(format!( + "rootfs tar path {} is outside the staging directory {}", + canonical.display(), + canonical_staging.display() + ))); + } + + let relative = canonical.strip_prefix(&canonical_staging).unwrap(); + let depth = relative.components().count(); + if depth != 2 { + return Err(Status::permission_denied(format!( + "rootfs tar path {} must be inside a request subdirectory of the staging root", + canonical.display(), + ))); + } + + let metadata = tokio::fs::symlink_metadata(&canonical) + .await + .map_err(|err| { + Status::failed_precondition(format!( + "rootfs tar not accessible at {}: {err}", + canonical.display() + )) + })?; + if !metadata.file_type().is_file() { + return Err(Status::invalid_argument(format!( + "rootfs tar path {} is not a regular file", + canonical.display() + ))); + } + + let max_bytes = self.config.rootfs_tar_max_bytes(); + let file_size = metadata.len(); + if file_size > max_bytes { + return Err(Status::invalid_argument(format!( + "rootfs tar {} is {} bytes, exceeding the {} byte limit", + canonical.display(), + file_size, + max_bytes + ))); + } + + Ok(canonical) + } + #[must_use] pub fn capabilities(&self) -> GetCapabilitiesResponse { GetCapabilitiesResponse { @@ -525,6 +767,25 @@ impl VmDriver { default_image: self.config.default_image.clone(), gateway_manages_lifecycle: true, supports_sandbox_authentication: false, + driver_reports_runtime_readiness: false, + resource_capabilities: Some(ResourceCapabilities { + cpu: Some(CpuResourceCapabilities { + limit_supported: false, + }), + memory: Some(MemoryResourceCapabilities { + limit_supported: false, + }), + gpu: Some(GpuResourceCapabilities { + default_selection_supported: self.config.gpu_enabled, + count_selection_supported: self.config.gpu_enabled, + }), + }), + rootfs_tar_staging_dir: self + .config + .rootfs_tar_staging_dir() + .to_string_lossy() + .into_owned(), + rootfs_tar_max_bytes: self.config.rootfs_tar_max_bytes(), } } @@ -533,9 +794,11 @@ impl VmDriver { #[allow(clippy::result_large_err)] pub fn validate_sandbox(&self, sandbox: &Sandbox) -> Result<(), Status> { validate_vm_sandbox(sandbox, self.config.gpu_enabled)?; - if self.resolved_sandbox_image(sandbox).is_none() { + let has_rootfs_tar = + VmSandboxDriverConfig::from_sandbox(sandbox).is_ok_and(|c| c.rootfs_tar_path.is_some()); + if self.resolved_sandbox_image(sandbox).is_none() && !has_rootfs_tar { return Err(Status::failed_precondition( - "vm sandboxes require template.image or a configured default sandbox image", + "vm sandboxes require template.image, rootfs_tar_path in driver_config, or a configured default sandbox image", )); } Ok(()) @@ -553,11 +816,20 @@ impl VmDriver { validate_vm_sandbox(sandbox, self.config.gpu_enabled)?; let state_dir = sandbox_state_dir(&self.config.state_dir, &sandbox.id)?; - let image_ref = self.resolved_sandbox_image(sandbox).ok_or_else(|| { - Status::failed_precondition( - "vm sandboxes require template.image or a configured default sandbox image", - ) - })?; + let has_rootfs_tar = + VmSandboxDriverConfig::from_sandbox(sandbox).is_ok_and(|c| c.rootfs_tar_path.is_some()); + let image_ref = self + .resolved_sandbox_image(sandbox) + .or_else(|| { + has_rootfs_tar + .then(|| self.bootstrap_image_ref_default()) + .flatten() + }) + .ok_or_else(|| { + Status::failed_precondition( + "vm sandboxes require template.image, rootfs_tar_path in driver_config, or a configured default sandbox image", + ) + })?; info!( sandbox_id = %sandbox.id, image_ref = %image_ref, @@ -722,6 +994,16 @@ impl VmDriver { .and_then(|spec| spec.resource_requirements.as_ref()) .and_then(|requirements| driver_gpu_requirements(Some(requirements))) .is_some(); + let driver_config = + VmSandboxDriverConfig::from_sandbox(&sandbox).map_err(Status::invalid_argument)?; + let driver_config_had_rootfs_tar = driver_config.rootfs_tar_path.is_some(); + let rootfs_tar_path = match driver_config.rootfs_tar_path { + Some(raw) if overlay_preparation == OverlayPreparation::Fresh => { + Some(self.validate_rootfs_tar_path(Path::new(&raw)).await?) + } + Some(_) | None => None, + }; + self.publish_platform_event( sandbox.id.clone(), platform_event( @@ -732,7 +1014,32 @@ impl VmDriver { ), ); - let image_plan = self.prepare_runtime_images(&sandbox.id, &image_ref).await?; + let image_plan = if overlay_preparation == OverlayPreparation::PreserveExisting + && driver_config_had_rootfs_tar + { + let persisted_identity = + read_persisted_image_identity(&state_dir).await.map_err(|err| { + Status::internal(format!( + "cannot restore rootfs-tar sandbox: persisted image identity not found: {err}" + )) + })?; + let bootstrap_image_ref = self.bootstrap_image_ref(&image_ref); + let bootstrap_image_identity = self + .ensure_cached_bootstrap_rootfs_image(&sandbox.id, &bootstrap_image_ref) + .await?; + let root_disk = + image_cache_rootfs_image(&self.config.state_dir, &bootstrap_image_identity); + let image_disk = image_cache_rootfs_image(&self.config.state_dir, &persisted_identity); + RuntimeImagePlan { + root_disk, + image_disk: Some(image_disk), + image_identity: persisted_identity, + bootstrap_image_identity, + } + } else { + self.prepare_runtime_images(&sandbox.id, &image_ref, rootfs_tar_path.as_deref()) + .await? + }; let image_identity = image_plan.image_identity.clone(); self.ensure_provisioning_active(&sandbox.id).await?; info!( @@ -907,6 +1214,16 @@ impl VmDriver { return Err(err); } + // Staged on every launch, including a restart onto a preserved + // overlay, so the driver's copy always shadows the image layer. + if let Err(err) = inject_guest_upstream_proxy(&overlay_disk, &self.config).await { + self.lifecycle_extensions + .after_launch_failed(&sandbox, &state_dir, LaunchAbortReason::GuestPrepareFailed) + .await; + self.release_gpu_and_subnet(&sandbox.id); + return Err(err); + } + let endpoint_override = if plan.backend == VmBackend::Qemu { plan.host_ip.as_deref().map(|host_ip| { guest_visible_openshell_endpoint_for_tap(&self.config.openshell_endpoint, host_ip) @@ -1190,10 +1507,10 @@ impl VmDriver { } #[tracing::instrument( - name = "vm.delete", + name = "vm.teardown", skip(self), fields( - otel.name = "vm.delete", + otel.name = "vm.teardown", otel.status_code = tracing::field::Empty, sandbox.id = %sandbox_id, sandbox.name = %sandbox_name, @@ -1463,7 +1780,14 @@ impl VmDriver { clear_stop_marker: bool, reconciliation_span: &tracing::Span, ) -> bool { - let Some(image_ref) = self.resolved_sandbox_image(&sandbox) else { + let has_rootfs_tar = VmSandboxDriverConfig::from_sandbox(&sandbox) + .is_ok_and(|c| c.rootfs_tar_path.is_some()); + + let Some(image_ref) = self.resolved_sandbox_image(&sandbox).or_else(|| { + has_rootfs_tar + .then(|| self.bootstrap_image_ref_default()) + .flatten() + }) else { warn!( sandbox_id = %sandbox.id, sandbox_name = %sandbox.name, @@ -1693,26 +2017,44 @@ impl VmDriver { if plan.gpu_bdf.is_none() { plan.gpu_bdf = gpu_bdf; } - if has_complete_qemu_network(plan) { - return Ok(()); + if !has_complete_qemu_network(plan) { + let subnet = self + .subnet_allocator + .lock() + .map_err(|e| Status::internal(format!("subnet allocator lock poisoned: {e}")))? + .allocate(sandbox_id) + .map_err(Status::failed_precondition)?; + let mac = mac_from_sandbox_id(sandbox_id); + plan.tap_device = Some(tap_device_name(sandbox_id)); + plan.guest_ip = Some(subnet.guest_ip.to_string()); + plan.host_ip = Some(subnet.host_ip.to_string()); + plan.vsock_cid = Some(allocate_vsock_cid()); + plan.guest_mac = Some(format!( + "{:02x}:{:02x}:{:02x}:{:02x}:{:02x}:{:02x}", + mac[0], mac[1], mac[2], mac[3], mac[4], mac[5] + )); + plan.gateway_port = gateway_port_from_endpoint(&self.config.openshell_endpoint); } - let subnet = self - .subnet_allocator - .lock() - .map_err(|e| Status::internal(format!("subnet allocator lock poisoned: {e}")))? - .allocate(sandbox_id) - .map_err(Status::failed_precondition)?; - let mac = mac_from_sandbox_id(sandbox_id); - plan.tap_device = Some(tap_device_name(sandbox_id)); - plan.guest_ip = Some(subnet.guest_ip.to_string()); - plan.host_ip = Some(subnet.host_ip.to_string()); - plan.vsock_cid = Some(allocate_vsock_cid()); - plan.guest_mac = Some(format!( - "{:02x}:{:02x}:{:02x}:{:02x}:{:02x}:{:02x}", - mac[0], mac[1], mac[2], mac[3], mac[4], mac[5] - )); - plan.gateway_port = gateway_port_from_endpoint(&self.config.openshell_endpoint); + // The corporate-proxy host-loopback recipe is a libkrun/gvproxy + // property and has no QEMU/TAP equivalent (see + // `proxy_url_targets_gateway_host`). Run it here, after the subnet + // allocation above has settled `plan.host_ip`, because the address to + // compare against is this sandbox's own TAP host address. Fail the + // create with the reason rather than boot a sandbox whose + // policy-approved CONNECTs all time out against an unreachable proxy. + if let Some(url) = self.config.https_proxy.as_deref() + && proxy_url_targets_gateway_host(url, plan.host_ip.as_deref()) + { + let tap_host = plan.host_ip.as_deref().unwrap_or("the TAP host address"); + return Err(Status::failed_precondition(format!( + "https_proxy '{url}' addresses the gateway host, which a QEMU/TAP sandbox \ + (GPU sandboxes) cannot reach: host.openshell.internal resolves to this \ + sandbox's TAP host address {tap_host} and the driver's nftables rules allow \ + only the gateway port from the guest. Configure a proxy address routable \ + from the guest's masqueraded egress, or run this sandbox without a GPU" + ))); + } Ok(()) } @@ -1989,6 +2331,7 @@ impl VmDriver { &self, sandbox_id: &str, image_ref: &str, + rootfs_tar_path: Option<&Path>, ) -> Result { let span_status = openshell_otel::ErrorStatusGuard::current(); let bootstrap_image_ref = self.bootstrap_image_ref(image_ref); @@ -1997,6 +2340,18 @@ impl VmDriver { .await?; let root_disk = image_cache_rootfs_image(&self.config.state_dir, &bootstrap_image_identity); + if let Some(tar_path) = rootfs_tar_path { + let prepared = self + .ensure_prepared_rootfs_tar_disk(sandbox_id, tar_path, &root_disk) + .await?; + return Ok(RuntimeImagePlan { + root_disk, + image_disk: Some(prepared.disk_path), + image_identity: prepared.image_identity, + bootstrap_image_identity, + }); + } + if image_ref.trim() == bootstrap_image_ref.trim() { return span_status.finish(Ok(RuntimeImagePlan { root_disk, @@ -2018,15 +2373,20 @@ impl VmDriver { } fn bootstrap_image_ref(&self, sandbox_image_ref: &str) -> String { + self.bootstrap_image_ref_default() + .unwrap_or_else(|| sandbox_image_ref.to_string()) + } + + fn bootstrap_image_ref_default(&self) -> Option { let configured = self.config.bootstrap_image.trim(); if !configured.is_empty() { - return configured.to_string(); + return Some(configured.to_string()); } let default = self.config.default_image.trim(); if !default.is_empty() { - return default.to_string(); + return Some(default.to_string()); } - sandbox_image_ref.to_string() + None } #[tracing::instrument( @@ -2143,14 +2503,15 @@ impl VmDriver { ("image_source".to_string(), "registry".to_string()), ]), ); - client - .auth(&reference, &auth, RegistryOperation::Pull) - .await - .map_err(|err| { - Status::failed_precondition(format!( - "failed to authenticate registry access for vm sandbox image '{image_ref}': {err}" - )) - })?; + retry_registry_request("authenticate with registry", || { + client.auth(&reference, &auth, RegistryOperation::Pull) + }) + .await + .map_err(|err| { + Status::failed_precondition(format!( + "failed to authenticate registry access for vm sandbox image '{image_ref}': {err}" + )) + })?; info!(image_ref = %image_ref, "vm driver: fetching manifest digest"); self.publish_vm_progress( sandbox_id, @@ -2161,14 +2522,15 @@ impl VmDriver { ("image_source".to_string(), "registry".to_string()), ]), ); - let source_image_identity = client - .fetch_manifest_digest(&reference, &auth) - .await - .map_err(|err| { - Status::failed_precondition(format!( - "failed to resolve vm sandbox image '{image_ref}': {err}" - )) - })?; + let source_image_identity = retry_registry_request("fetch manifest digest", || { + client.fetch_manifest_digest(&reference, &auth) + }) + .await + .map_err(|err| { + Status::failed_precondition(format!( + "failed to resolve vm sandbox image '{image_ref}': {err}" + )) + })?; info!( image_ref = %image_ref, image_identity = %source_image_identity, @@ -2526,6 +2888,149 @@ impl VmDriver { }) } + async fn ensure_prepared_rootfs_tar_disk( + &self, + sandbox_id: &str, + tar_path: &Path, + bootstrap_root_disk: &Path, + ) -> Result { + let request_staging_dir = tar_path.parent().map(Path::to_path_buf); + let cleanup_request_staging = || async { + if let Some(d) = &request_staging_dir { + let _ = tokio::fs::remove_dir_all(d).await; + } + }; + + // Identity comes from the archive contents. See `rootfs_tar_cache_identity`. + let hash_source = tar_path.to_path_buf(); + let source_digest = match tokio::task::spawn_blocking(move || { + compute_file_sha256_hex(&hash_source) + }) + .await + { + Ok(Ok(digest)) => digest, + Ok(Err(err)) => { + cleanup_request_staging().await; + return Err(Status::failed_precondition(format!( + "rootfs tar not readable at {}: {err}", + tar_path.display() + ))); + } + Err(err) => { + cleanup_request_staging().await; + return Err(Status::internal(format!( + "failed to hash rootfs tar at {}: {err}", + tar_path.display() + ))); + } + }; + let cache_identity = rootfs_tar_cache_identity(&source_digest); + let image_path = image_cache_rootfs_image(&self.config.state_dir, &cache_identity); + let tar_display = tar_path.display().to_string(); + + if tokio::fs::metadata(&image_path).await.is_ok() { + self.publish_prepared_cache_hit( + sandbox_id, + &tar_display, + "rootfs_tar", + &cache_identity, + ); + cleanup_request_staging().await; + return Ok(PreparedImageDisk { + image_identity: cache_identity, + disk_path: image_path, + }); + } + + self.publish_prepared_cache_miss(sandbox_id, &tar_display, "rootfs_tar", &cache_identity); + let _cache_guard = self.image_cache_lock.lock().await; + if tokio::fs::metadata(&image_path).await.is_ok() { + self.publish_prepared_cache_hit( + sandbox_id, + &tar_display, + "rootfs_tar", + &cache_identity, + ); + cleanup_request_staging().await; + return Ok(PreparedImageDisk { + image_identity: cache_identity, + disk_path: image_path, + }); + } + + let staging_dir = image_cache_staging_dir(&self.config.state_dir, &cache_identity); + let rootfs_archive = staging_dir.join(IMAGE_EXPORT_ROOTFS_ARCHIVE); + self.reset_image_staging_dir(&staging_dir).await?; + + self.publish_vm_progress( + sandbox_id, + "CopyingRootfsTar", + format!("Copying rootfs tar \"{tar_display}\""), + HashMap::from([ + ("rootfs_tar_path".to_string(), tar_display.clone()), + ("image_source".to_string(), "rootfs_tar".to_string()), + ("image_identity".to_string(), cache_identity.clone()), + ]), + ); + let copy_src = tar_path.to_path_buf(); + let copy_dst = rootfs_archive.clone(); + let max_bytes = self.config.rootfs_tar_max_bytes(); + let copied_digest = match tokio::task::spawn_blocking(move || { + stage_rootfs_tar_archive(©_src, ©_dst, max_bytes) + }) + .await + { + Ok(Ok(digest)) => digest, + Ok(Err(err)) => { + let _ = tokio::fs::remove_dir_all(&staging_dir).await; + cleanup_request_staging().await; + return Err(Status::internal(format!( + "failed to copy rootfs tar to staging: {err}" + ))); + } + Err(err) => { + let _ = tokio::fs::remove_dir_all(&staging_dir).await; + cleanup_request_staging().await; + return Err(Status::internal(format!( + "failed to copy rootfs tar to staging: {err}" + ))); + } + }; + + // The archive changed between the hash pass and the copy: the prepared + // disk we are about to build would not match the identity it is cached + // under. Reject rather than poison the cache. + if copied_digest != source_digest { + let _ = tokio::fs::remove_dir_all(&staging_dir).await; + cleanup_request_staging().await; + return Err(Status::aborted(format!( + "rootfs tar {tar_display} changed while it was being staged; retry the request" + ))); + } + cleanup_request_staging().await; + + let payload = GuestImagePayload { + image_ref: tar_display.clone(), + image_identity: cache_identity.clone(), + source: GuestImagePayloadSource::LocalDocker { rootfs_archive }, + }; + self.build_prepared_image_disk( + sandbox_id, + &tar_display, + "rootfs_tar", + &cache_identity, + bootstrap_root_disk, + &staging_dir, + &payload, + ) + .await?; + + Ok(PreparedImageDisk { + image_identity: cache_identity, + disk_path: image_path, + }) + } + async fn ensure_prepared_registry_image_disk( &self, sandbox_id: &str, @@ -2545,14 +3050,15 @@ impl VmDriver { ("image_source".to_string(), "registry".to_string()), ]), ); - client - .auth(&reference, &auth, RegistryOperation::Pull) - .await - .map_err(|err| { - Status::failed_precondition(format!( - "failed to authenticate registry access for vm sandbox image '{image_ref}': {err}" - )) - })?; + retry_registry_request("authenticate with registry", || { + client.auth(&reference, &auth, RegistryOperation::Pull) + }) + .await + .map_err(|err| { + Status::failed_precondition(format!( + "failed to authenticate registry access for vm sandbox image '{image_ref}': {err}" + )) + })?; self.publish_vm_progress( sandbox_id, @@ -2563,14 +3069,15 @@ impl VmDriver { ("image_source".to_string(), "registry".to_string()), ]), ); - let source_image_identity = client - .fetch_manifest_digest(&reference, &auth) - .await - .map_err(|err| { - Status::failed_precondition(format!( - "failed to resolve vm sandbox image '{image_ref}': {err}" - )) - })?; + let source_image_identity = retry_registry_request("fetch manifest digest", || { + client.fetch_manifest_digest(&reference, &auth) + }) + .await + .map_err(|err| { + Status::failed_precondition(format!( + "failed to resolve vm sandbox image '{image_ref}': {err}" + )) + })?; let cache_identity = prepared_image_cache_identity(&source_image_identity); let image_path = image_cache_rootfs_image(&self.config.state_dir, &cache_identity); @@ -2596,14 +3103,15 @@ impl VmDriver { self.reset_image_staging_dir(&staging_dir).await?; let layout_dir = staging_dir.join(GUEST_IMAGE_OCI_LAYOUT_DIR); - let (manifest, _) = client - .pull_image_manifest(&reference, &auth) - .await - .map_err(|err| { - Status::failed_precondition(format!( - "failed to pull vm sandbox image manifest '{image_ref}': {err}" - )) - })?; + let (manifest, _) = retry_registry_request("pull image manifest", || { + client.pull_image_manifest(&reference, &auth) + }) + .await + .map_err(|err| { + Status::failed_precondition(format!( + "failed to pull vm sandbox image manifest '{image_ref}': {err}" + )) + })?; tokio::fs::create_dir_all(oci_layout_blobs_dir(&layout_dir)) .await .map_err(|err| Status::internal(format!("create guest OCI layout failed: {err}")))?; @@ -3663,7 +4171,7 @@ async fn connect_local_container_engine() -> Option { return Some(docker); } - let podman_socket = openshell_core::config::detect_podman_socket()?; + let podman_socket = detect_podman_socket()?; if let Ok(docker) = Docker::connect_with_unix(podman_socket.to_str()?, 120, bollard::API_DEFAULT_VERSION) && docker.ping().await.is_ok() @@ -3678,6 +4186,10 @@ async fn connect_local_container_engine() -> Option { None } +fn detect_podman_socket() -> Option { + openshell_driver_podman::driver::detect_socket() +} + fn is_openshell_local_build_image_ref(image_ref: &str) -> bool { image_ref.starts_with("openshell/sandbox-from:") } @@ -3792,6 +4304,134 @@ fn registry_client() -> OciClient { }) } +async fn retry_registry_request( + operation: &str, + request: F, +) -> Result +where + F: FnMut() -> Fut, + Fut: Future>, +{ + retry_registry_request_with_delay(operation, REGISTRY_RETRY_INITIAL_DELAY, request).await +} + +async fn retry_registry_request_with_delay( + operation: &str, + initial_delay: Duration, + mut request: F, +) -> Result +where + F: FnMut() -> Fut, + Fut: Future>, +{ + let mut delay = initial_delay; + for attempt in 1..=REGISTRY_REQUEST_MAX_ATTEMPTS { + match request().await { + Ok(value) => return Ok(value), + Err(err) + if attempt < REGISTRY_REQUEST_MAX_ATTEMPTS && registry_error_is_retryable(&err) => + { + warn!( + operation, + attempt, + max_attempts = REGISTRY_REQUEST_MAX_ATTEMPTS, + retry_delay_ms = delay.as_millis(), + error = %err, + "vm driver: transient registry request failed; retrying" + ); + tokio::time::sleep(delay).await; + delay = delay.saturating_mul(2).min(REGISTRY_RETRY_MAX_DELAY); + } + Err(err) => return Err(err), + } + } + + unreachable!("registry request retry loop always returns") +} + +fn registry_error_is_retryable(err: &OciDistributionError) -> bool { + match err { + OciDistributionError::AuthenticationFailure(message) => { + registry_message_is_retryable(message) + } + OciDistributionError::RegistryError { envelope, .. } => { + envelope.errors.iter().any(|error| { + error.code == OciErrorCode::Toomanyrequests + || registry_message_is_retryable(&error.message) + }) + } + OciDistributionError::RequestError(err) => { + err.is_connect() + || err.is_timeout() + || err + .status() + .is_some_and(|status| matches!(status.as_u16(), 408 | 425 | 429 | 500..=599)) + } + OciDistributionError::ServerError { code, message, .. } => { + matches!(code, 408 | 425 | 429 | 500..=599) || registry_message_is_retryable(message) + } + OciDistributionError::IoError(err) => matches!( + err.kind(), + std::io::ErrorKind::Interrupted + | std::io::ErrorKind::TimedOut + | std::io::ErrorKind::ConnectionAborted + | std::io::ErrorKind::ConnectionReset + | std::io::ErrorKind::ConnectionRefused + | std::io::ErrorKind::BrokenPipe + | std::io::ErrorKind::UnexpectedEof + | std::io::ErrorKind::WouldBlock + ), + _ => false, + } +} + +fn registry_message_is_retryable(message: &str) -> bool { + let message = message.to_ascii_lowercase(); + message.contains("retry-after") + || message.contains("too many requests") + || message.contains("rate limit") +} + +enum RegistryBlobPullError { + CreateFile(std::io::Error), + Registry(OciDistributionError), +} + +async fn pull_registry_blob_file( + operation: &str, + client: &OciClient, + reference: &Reference, + descriptor: &OciDescriptor, + blob_path: &Path, +) -> Result { + let mut delay = REGISTRY_RETRY_INITIAL_DELAY; + for attempt in 1..=REGISTRY_REQUEST_MAX_ATTEMPTS { + let mut file = tokio::fs::File::create(blob_path) + .await + .map_err(RegistryBlobPullError::CreateFile)?; + match client.pull_blob(reference, descriptor, &mut file).await { + Ok(()) => return Ok(file), + Err(err) + if attempt < REGISTRY_REQUEST_MAX_ATTEMPTS && registry_error_is_retryable(&err) => + { + warn!( + operation, + attempt, + max_attempts = REGISTRY_REQUEST_MAX_ATTEMPTS, + retry_delay_ms = delay.as_millis(), + error = %err, + "vm driver: transient registry request failed; retrying" + ); + tokio::time::sleep(delay).await; + delay = delay.saturating_mul(2).min(REGISTRY_RETRY_MAX_DELAY); + } + Err(err) => return Err(RegistryBlobPullError::Registry(err)), + } + } + + unreachable!("registry blob retry loop always returns") +} + fn linux_platform_resolver(manifests: &[ImageIndexEntry]) -> Option { let expected_arch = linux_oci_arch(); manifests @@ -3876,22 +4516,24 @@ impl VmDriver { staging_dir: &Path, rootfs: &Path, ) -> Result<(), Status> { - client - .auth(reference, auth, RegistryOperation::Pull) - .await - .map_err(|err| { - Status::failed_precondition(format!( - "failed to authenticate registry access for vm sandbox image '{image_ref}': {err}" - )) - })?; - let (manifest, _) = client - .pull_image_manifest(reference, auth) - .await - .map_err(|err| { - Status::failed_precondition(format!( - "failed to pull vm sandbox image manifest '{image_ref}': {err}" - )) - })?; + retry_registry_request("authenticate with registry", || { + client.auth(reference, auth, RegistryOperation::Pull) + }) + .await + .map_err(|err| { + Status::failed_precondition(format!( + "failed to authenticate registry access for vm sandbox image '{image_ref}': {err}" + )) + })?; + let (manifest, _) = retry_registry_request("pull image manifest", || { + client.pull_image_manifest(reference, auth) + }) + .await + .map_err(|err| { + Status::failed_precondition(format!( + "failed to pull vm sandbox image manifest '{image_ref}': {err}" + )) + })?; tokio::fs::create_dir_all(rootfs) .await @@ -4011,18 +4653,23 @@ async fn download_registry_layer_blob( .join("layers") .join(format!("{index:02}-{digest_component}.root")); - let mut file = tokio::fs::File::create(&blob_path) - .await - .map_err(|err| Status::internal(format!("create layer blob failed: {err}")))?; - client - .pull_blob(reference, &layer, &mut file) - .await - .map_err(|err| { - Status::failed_precondition(format!( - "failed to download layer '{}' for vm sandbox image '{image_ref}': {err}", - layer.digest - )) - })?; + let mut file = pull_registry_blob_file( + "download image layer", + client, + reference, + &layer, + &blob_path, + ) + .await + .map_err(|err| match err { + RegistryBlobPullError::CreateFile(err) => { + Status::internal(format!("create layer blob failed: {err}")) + } + RegistryBlobPullError::Registry(err) => Status::failed_precondition(format!( + "failed to download layer '{}' for vm sandbox image '{image_ref}': {err}", + layer.digest + )), + })?; file.flush() .await .map_err(|err| Status::internal(format!("flush layer blob failed: {err}")))?; @@ -4109,18 +4756,23 @@ async fn download_registry_descriptor_blob_file( .map_err(|err| Status::internal(format!("create OCI blob dir failed: {err}")))?; } - let mut file = tokio::fs::File::create(&blob_path) - .await - .map_err(|err| Status::internal(format!("create OCI {kind} blob failed: {err}")))?; - client - .pull_blob(reference, descriptor, &mut file) - .await - .map_err(|err| { - Status::failed_precondition(format!( - "failed to download {kind} '{}' for vm sandbox image '{image_ref}': {err}", - descriptor.digest - )) - })?; + let mut file = pull_registry_blob_file( + "download image blob", + client, + reference, + descriptor, + &blob_path, + ) + .await + .map_err(|err| match err { + RegistryBlobPullError::CreateFile(err) => { + Status::internal(format!("create OCI {kind} blob failed: {err}")) + } + RegistryBlobPullError::Registry(err) => Status::failed_precondition(format!( + "failed to download {kind} '{}' for vm sandbox image '{image_ref}': {err}", + descriptor.digest + )), + })?; file.flush() .await .map_err(|err| Status::internal(format!("flush OCI {kind} blob failed: {err}")))?; @@ -4177,6 +4829,113 @@ fn compute_bytes_sha256_hex(bytes: &[u8]) -> String { format!("{:x}", hasher.finalize()) } +/// Stage the caller-supplied rootfs archive at `src` into the image cache at +/// `dst`, and return the SHA-256 of the source bytes that were read. +/// +/// The staged file is always an uncompressed tar. `--from` accepts `.tar.gz` +/// and `.tgz`, but the guest image-prep VM extracts the staged file with a +/// plain `tar -xpf`, and the prepared disk is sized from that file's length, +/// so leaving gzip bytes on disk would both depend on the guest tar +/// auto-detecting compression and size the disk from the compressed length. +/// Compression is detected from the magic bytes: the driver only ever sees a +/// gateway-issued staging path, never the caller's file name. +/// +/// Expansion is bounded by `max_bytes` — the same limit the driver applies to +/// the archive it accepts — so a compression bomb cannot fill the host disk. +/// +/// The digest covers the source bytes rather than the bytes written, which is +/// what lets the caller detect an archive that changed underneath it during +/// staging: it stays comparable with the pre-copy hash pass whether or not the +/// source was compressed. +fn stage_rootfs_tar_archive(src: &Path, dst: &Path, max_bytes: u64) -> Result { + let file = fs::File::open(src).map_err(|err| format!("open {}: {err}", src.display()))?; + let mut reader = BufReader::new(file); + let compressed = reader + .fill_buf() + .map_err(|err| format!("read {}: {err}", src.display()))? + .starts_with(&crate::rootfs::GZIP_MAGIC); + + let mut source = HashingReader::new(reader); + if compressed { + write_stream_to_file(MultiGzDecoder::new(&mut source), dst, max_bytes)?; + } else { + write_stream_to_file(&mut source, dst, max_bytes)?; + } + + // A decoder stops at the end of the compressed stream, so drain whatever + // it left behind: the digest has to describe the whole source file for the + // caller's change-detection comparison to mean anything. + std::io::copy(&mut source, &mut std::io::sink()) + .map_err(|err| format!("read {}: {err}", src.display()))?; + Ok(source.finish()) +} + +/// Reader adapter that digests every byte it yields. +struct HashingReader { + inner: R, + hasher: Sha256, +} + +impl HashingReader { + fn new(inner: R) -> Self { + Self { + inner, + hasher: Sha256::new(), + } + } + + fn finish(self) -> String { + format!("{:x}", self.hasher.finalize()) + } +} + +impl Read for HashingReader { + fn read(&mut self, buf: &mut [u8]) -> std::io::Result { + let read = self.inner.read(buf)?; + self.hasher.update(&buf[..read]); + Ok(read) + } +} + +fn write_stream_to_file(mut reader: impl Read, dst: &Path, max_bytes: u64) -> Result<(), String> { + let mut writer = BufWriter::new( + fs::File::create(dst).map_err(|err| format!("create {}: {err}", dst.display()))?, + ); + let mut buffer = vec![0_u8; 64 * 1024].into_boxed_slice(); + let mut written = 0_u64; + loop { + let read = reader + .read(&mut buffer) + .map_err(|err| format!("read rootfs tar: {err}"))?; + if read == 0 { + break; + } + written = written.saturating_add(u64::try_from(read).unwrap_or(u64::MAX)); + if written > max_bytes { + return Err(format!( + "rootfs tar expands to more than the {max_bytes} byte limit" + )); + } + writer + .write_all(&buffer[..read]) + .map_err(|err| format!("write {}: {err}", dst.display()))?; + } + writer + .flush() + .map_err(|err| format!("flush {}: {err}", dst.display())) +} + +/// Cache identity for a rootfs tar archive, derived from its contents. +/// +/// Deliberately not path- or mtime-derived: staging directories are unique per +/// request, so a path-based key would never hit the cache, and a +/// seconds-truncated mtime cannot distinguish two writes within the same +/// second. A fixed-length digest also keeps the cache directory name inside +/// filesystem component limits regardless of how long the source path was. +fn rootfs_tar_cache_identity(digest: &str) -> String { + prepared_image_cache_identity(&format!("rootfs-tar:sha256:{digest}")) +} + fn extract_layer_blob_to_dir( blob_path: &Path, media_type: &str, @@ -4432,6 +5191,53 @@ fn guest_visible_openshell_endpoint(endpoint: &str) -> String { endpoint.to_string() } +/// Whether a corporate proxy URL points at the gateway host itself, as seen +/// from a QEMU/TAP guest whose TAP host address is `tap_host_ip`. +/// +/// On the libkrun backend gvproxy NATs the host-loopback alias +/// `host.openshell.internal` (and any loopback URL, which the driver rewrites +/// to that alias) to the gateway host's `127.0.0.1`, so a proxy bound to host +/// loopback is reachable from the guest. The QEMU/TAP backend used for GPU +/// sandboxes has no equivalent: `host.openshell.internal` resolves to the TAP +/// host address, and the driver's own nftables `input` chain accepts only the +/// gateway port from the guest and drops the rest, so no proxy on the gateway +/// host is reachable regardless of the address it binds. +/// +/// The gateway host is therefore reached from a QEMU guest under exactly three +/// spellings: the guest's own loopback (never the host's, but a configuration +/// that plainly means the host), the documented host aliases that +/// `write_host_gateway_aliases` seeds to the TAP host address, and that TAP +/// host address written literally. `tap_host_ip` is this sandbox's allocated +/// address, so the comparison must be made after the launch plan's subnet +/// allocation; `None` means the plan carries no TAP host and only the +/// address-independent spellings are classified. +/// +/// gvproxy's `GVPROXY_HOST_LOOPBACK_IP` is deliberately **not** matched here. +/// It is special only to libkrun; on QEMU/TAP it is an ordinary address that +/// may well be routable through the guest's masqueraded egress, and rejecting +/// it would refuse a working configuration. +/// +/// Used to reject an unreachable configuration up front on the QEMU path +/// instead of letting every policy-approved CONNECT time out. +fn proxy_url_targets_gateway_host(url: &str, tap_host_ip: Option<&str>) -> bool { + let Ok(parsed) = Url::parse(url) else { + // Unparseable URLs are rejected by shared validation before launch. + return false; + }; + let tap_host = tap_host_ip.and_then(|ip| ip.parse::().ok()); + match parsed.host() { + Some(Host::Ipv4(ip)) => ip.is_loopback() || tap_host == Some(IpAddr::V4(ip)), + Some(Host::Ipv6(ip)) => ip.is_loopback() || tap_host == Some(IpAddr::V6(ip)), + Some(Host::Domain(host)) => { + host.eq_ignore_ascii_case("localhost") + || host.eq_ignore_ascii_case(OPENSHELL_HOST_GATEWAY_ALIAS) + || host.eq_ignore_ascii_case("host.containers.internal") + || host.eq_ignore_ascii_case("host.docker.internal") + } + None => false, + } +} + fn gateway_port_from_endpoint(endpoint: &str) -> Option { Url::parse(endpoint).ok().and_then(|url| url.port()) } @@ -4828,6 +5634,11 @@ async fn write_sandbox_image_metadata( Ok(()) } +async fn read_persisted_image_identity(state_dir: &Path) -> Result { + let raw = tokio::fs::read_to_string(state_dir.join(IMAGE_IDENTITY_FILE)).await?; + Ok(raw.trim().to_string()) +} + async fn write_sandbox_request(state_dir: &Path, sandbox: &Sandbox) -> Result<(), std::io::Error> { restrict_owner_only_dir(state_dir).await?; write_private_file( @@ -5125,6 +5936,182 @@ fn inject_guest_init_dropins( span_status.finish(Ok(())) } +/// Build the corporate upstream-proxy arguments passed to the guest supervisor. +/// +/// This operator-owned egress boundary travels on the supervisor's argv, +/// which sandbox spec/template environment and image `ENV` cannot influence. +/// Credentials are never on argv — only the root-only guest path is passed; +/// the supervisor reads the credential from that file. +fn upstream_proxy_cli_args(config: &VmDriverConfig) -> Vec { + let mut args = Vec::new(); + if let Some(url) = &config.https_proxy { + args.push("--upstream-proxy".to_string()); + args.push(url.clone()); + } + if let Some(list) = &config.no_proxy { + args.push("--upstream-no-proxy".to_string()); + args.push(list.clone()); + } + if config.proxy_auth_file.is_some() { + args.push("--upstream-proxy-auth-file".to_string()); + // The guest path, never the gateway-host path the operator configured. + args.push(GUEST_UPSTREAM_PROXY_AUTH_PATH.to_string()); + } + // Config validation guarantees the acknowledgement is `true` whenever an + // auth file is configured against an http:// proxy; the supervisor + // independently refuses credentials without it. + if config.proxy_auth_allow_insecure == Some(true) { + args.push("--upstream-proxy-auth-allow-insecure".to_string()); + } + // Absent means the default validated-IP CONNECT binding; only the + // explicit hostname opt-in is passed through. + if config.proxy_connect_by_hostname == Some(true) { + args.push("--upstream-proxy-connect-by-hostname".to_string()); + } + if config.proxy_ca_bundle.is_some() { + args.push("--upstream-proxy-ca-bundle".to_string()); + args.push(GUEST_PROXY_CA_PATH.to_string()); + } + args +} + +/// Render the supervisor argument list as newline-separated arguments. +/// +/// One argument per line, verbatim: the guest reads the lines into an array +/// without word splitting or globbing, so values containing spaces survive +/// intact. An empty list renders an empty file, which the guest reads as "no +/// extra arguments". +fn render_guest_supervisor_args(args: &[String]) -> Vec { + let mut body = args.join("\n"); + if !body.is_empty() { + body.push('\n'); + } + body.into_bytes() +} + +/// Reject argument values the newline-delimited guest file cannot represent. +/// +/// Every value here is operator-supplied config, so this is a guard against +/// misconfiguration rather than an attack: a stray newline would otherwise +/// split one value into two arguments in the guest. +fn validate_guest_supervisor_args(args: &[String]) -> Result<(), String> { + for arg in args { + if arg.contains('\n') || arg.contains('\r') || arg.contains('\0') { + return Err( + "corporate proxy settings must not contain newline or NUL characters".to_string(), + ); + } + } + Ok(()) +} + +/// Read and validate the corporate proxy credential from the gateway host. +/// +/// Uses the validators shared with the supervisor, so a credential accepted +/// here is never rejected inside the guest. The error never carries the file +/// contents. +async fn read_sandbox_proxy_credential(path: &str) -> Result { + let path_owned = path.to_string(); + let raw = tokio::task::spawn_blocking(move || { + openshell_core::driver_utils::read_upstream_proxy_credential_file(&path_owned) + }) + .await + .map_err(|err| Status::internal(format!("proxy_auth_file read task failed: {err}")))? + .map_err(Status::invalid_argument)?; + let credential = openshell_core::driver_utils::parse_upstream_proxy_credential(&raw) + .map_err(|err| Status::invalid_argument(format!("proxy_auth_file '{path}': {err}")))?; + Ok(credential.to_string()) +} + +/// Read and validate the corporate proxy CA bundle from the gateway host. +/// +/// Uses the reader shared with the supervisor, so the bundle is bounded and +/// non-regular files are rejected (an operator path such as `/dev/zero` can +/// otherwise exhaust driver memory), and a bundle accepted here contributes at +/// least one trust anchor rustls accepts rather than merely looking like PEM. +/// Checked here rather than only in the guest so the operator gets an error +/// attributable to `proxy_ca_bundle` instead of an opaque supervisor startup +/// failure inside every sandbox. The error never carries the file contents. +async fn read_sandbox_proxy_ca_bundle(path: &str) -> Result, Status> { + let path_owned = path.to_string(); + let pem = tokio::task::spawn_blocking(move || { + openshell_core::driver_utils::read_upstream_proxy_ca_bundle_file( + &path_owned, + "proxy_ca_bundle", + ) + }) + .await + .map_err(|err| Status::internal(format!("proxy_ca_bundle read task failed: {err}")))? + .map_err(Status::invalid_argument)?; + Ok(pem.into_bytes()) +} + +/// Stage the corporate upstream-proxy configuration into the guest overlay. +/// +/// Writes three files into the overlay upperdir the driver owns: +/// +/// * the credential at [`GUEST_UPSTREAM_PROXY_AUTH_PATH`], mode `0600`; +/// * the CA bundle at [`GUEST_PROXY_CA_PATH`], mode `0644` (a CA certificate +/// is not secret); +/// * the supervisor argument list at [`GUEST_SUPERVISOR_ARGS_PATH`], mode +/// `0644`. +/// +/// All three are written on every launch, empty when the corresponding +/// setting is absent. Writing rather than skipping is what makes the channel +/// unforgeable: the upperdir copy always shadows the read-only image layer, so +/// a sandbox image cannot supply its own arguments or credential by baking a +/// file at these paths, and cannot disable the operator's by omitting one. It +/// also clears material a previous launch staged into a preserved overlay +/// after the operator removed the setting. +/// +/// A microVM has no bind mounts or container secrets, so the credential lives +/// at rest inside the per-sandbox overlay disk on the host — the same +/// delivery the per-sandbox gateway JWT already uses. It is removed with the +/// sandbox when the state directory is deleted. +#[allow(clippy::result_large_err)] +async fn inject_guest_upstream_proxy( + overlay_disk: &Path, + config: &VmDriverConfig, +) -> Result<(), Status> { + // Written whether or not they are configured. Writing empty files when + // the operator removed a setting clears material a previous launch staged + // into a preserved overlay, and shadows anything an image baked at these + // paths, so a staged file is only ever the one this launch produced. + let credential = match config.proxy_auth_file.as_deref() { + Some(path) => format!("{}\n", read_sandbox_proxy_credential(path).await?).into_bytes(), + None => Vec::new(), + }; + let credential_path = overlay_upper_path(GUEST_UPSTREAM_PROXY_AUTH_PATH); + write_rootfs_image_file(overlay_disk, &credential_path, &credential) + .map_err(|err| Status::internal(format!("write VM guest proxy credential: {err}")))?; + set_rootfs_image_file_mode(overlay_disk, &credential_path, 0o600) + .map_err(|err| Status::internal(format!("set VM guest proxy credential mode: {err}")))?; + + let ca_bundle = match config.proxy_ca_bundle.as_deref() { + Some(path) => read_sandbox_proxy_ca_bundle(path).await?, + None => Vec::new(), + }; + let ca_path = overlay_upper_path(GUEST_PROXY_CA_PATH); + write_rootfs_image_file(overlay_disk, &ca_path, &ca_bundle) + .map_err(|err| Status::internal(format!("write VM guest proxy CA bundle: {err}")))?; + set_rootfs_image_file_mode(overlay_disk, &ca_path, 0o644) + .map_err(|err| Status::internal(format!("set VM guest proxy CA bundle mode: {err}")))?; + + let args = upstream_proxy_cli_args(config); + validate_guest_supervisor_args(&args).map_err(Status::failed_precondition)?; + let guest_path = overlay_upper_path(GUEST_SUPERVISOR_ARGS_PATH); + write_rootfs_image_file( + overlay_disk, + &guest_path, + &render_guest_supervisor_args(&args), + ) + .map_err(|err| Status::internal(format!("write VM guest supervisor arguments: {err}")))?; + set_rootfs_image_file_mode(overlay_disk, &guest_path, 0o644).map_err(|err| { + Status::internal(format!("set VM guest supervisor arguments mode: {err}")) + })?; + Ok(()) +} + /// Render the drop-in allow-list as newline-separated, ASCII-sorted, /// de-duplicated names. Names are already validated to be path-safe by /// [`validate_guest_init_dropins`]. @@ -5596,12 +6583,59 @@ mod tests { use prost_types::{Struct, Value, value::Kind}; use std::fs; use std::path::Path; - use std::sync::atomic::{AtomicU64, Ordering}; + use std::sync::atomic::{AtomicU64, AtomicUsize, Ordering}; use std::time::{SystemTime, UNIX_EPOCH}; use tonic::Code; static ENV_LOCK: std::sync::LazyLock> = std::sync::LazyLock::new(|| std::sync::Mutex::new(())); + + #[test] + fn registry_throttling_errors_are_retryable() { + let error = OciDistributionError::RegistryError { + envelope: oci_client::errors::OciEnvelope { + errors: vec![oci_client::errors::OciError { + code: OciErrorCode::Toomanyrequests, + message: "retry-after: 829.756µs, allowed: 44000/minute".to_string(), + detail: serde_json::Value::Null, + }], + }, + url: "https://ghcr.io/v2/example/manifests/latest".to_string(), + }; + + assert!(registry_error_is_retryable(&error)); + } + + #[test] + fn permanent_registry_errors_are_not_retryable() { + let error = OciDistributionError::UnauthorizedError { + url: "https://example.invalid/v2/image/manifests/latest".to_string(), + }; + + assert!(!registry_error_is_retryable(&error)); + } + + #[tokio::test] + async fn registry_request_retries_transient_errors_until_success() { + let attempts = AtomicUsize::new(0); + let result = retry_registry_request_with_delay("test request", Duration::ZERO, || async { + let attempt = attempts.fetch_add(1, Ordering::Relaxed); + if attempt < 2 { + Err(OciDistributionError::ServerError { + code: 503, + url: "https://example.invalid/v2/".to_string(), + message: "temporarily unavailable".to_string(), + }) + } else { + Ok("success") + } + }) + .await; + + assert_eq!(result.unwrap(), "success"); + assert_eq!(attempts.load(Ordering::Relaxed), 3); + } + struct TestTracing { exporter: opentelemetry_sdk::trace::InMemorySpanExporter, _provider: opentelemetry_sdk::trace::SdkTracerProvider, @@ -5707,7 +6741,7 @@ mod tests { let (shutdown, shutdown_rx) = tokio::sync::oneshot::channel(); let server = tokio::spawn(async move { tonic::transport::Server::builder() - .layer(crate::otel_tracing::compute_driver_rpc_layer()) + .layer(openshell_otel::compute_driver_rpc_layer()) .add_service(ComputeDriverServer::new(driver)) .serve_with_incoming_shutdown( tokio_stream::wrappers::TcpListenerStream::new(listener), @@ -5744,7 +6778,7 @@ mod tests { let spans = traced.exporter.get_finished_spans().unwrap(); let rpc_spans = spans .iter() - .filter(|span| span.name == "driver.get_capabilities") + .filter(|span| span.name == "openshell.compute.v1.ComputeDriver/GetCapabilities") .collect::>(); assert_eq!( rpc_spans.len(), @@ -5825,14 +6859,14 @@ mod tests { let spans = traced.exporter.get_finished_spans().unwrap(); let expected = [ - "driver.get_capabilities", - "driver.validate_sandbox_create", - "driver.create_sandbox", - "driver.get_sandbox", - "driver.list_sandboxes", - "driver.stop_sandbox", - "driver.delete_sandbox", - "driver.watch_sandboxes", + "openshell.compute.v1.ComputeDriver/GetCapabilities", + "openshell.compute.v1.ComputeDriver/ValidateSandboxCreate", + "openshell.compute.v1.ComputeDriver/CreateSandbox", + "openshell.compute.v1.ComputeDriver/GetSandbox", + "openshell.compute.v1.ComputeDriver/ListSandboxes", + "openshell.compute.v1.ComputeDriver/StopSandbox", + "openshell.compute.v1.ComputeDriver/DeleteSandbox", + "openshell.compute.v1.ComputeDriver/WatchSandboxes", ]; for name in expected { let span = spans @@ -5843,10 +6877,10 @@ mod tests { assert_has_parent(span); } for name in [ - "driver.validate_sandbox_create", - "driver.create_sandbox", - "driver.get_sandbox", - "driver.stop_sandbox", + "openshell.compute.v1.ComputeDriver/ValidateSandboxCreate", + "openshell.compute.v1.ComputeDriver/CreateSandbox", + "openshell.compute.v1.ComputeDriver/GetSandbox", + "openshell.compute.v1.ComputeDriver/StopSandbox", ] { let span = spans.iter().find(|span| span.name == name).unwrap(); assert!( @@ -5857,12 +6891,12 @@ mod tests { } let delete_rpc = spans .iter() - .find(|span| span.name == "driver.delete_sandbox") + .find(|span| span.name == "openshell.compute.v1.ComputeDriver/DeleteSandbox") .expect("delete RPC span"); let cleanup = spans .iter() .find(|span| { - span.name == "vm.delete" + span.name == "vm.teardown" && span.span_context.trace_id() == delete_rpc.span_context.trace_id() }) .expect("delete cleanup span"); @@ -6031,7 +7065,7 @@ mod tests { async fn background_provisioning_does_not_extend_the_rpc_span_lifetime() { let traced = TestTracing::new(); let _dispatch = tracing::dispatcher::set_default(&traced.dispatch); - let rpc = tracing::info_span!("driver.create_sandbox"); + let rpc = tracing::info_span!("openshell.compute.v1.ComputeDriver/CreateSandbox"); let entered = rpc.enter(); let provisioning = provisioning_span(&rpc.context(), "sb-lifetime", "invalid image reference"); @@ -6044,7 +7078,7 @@ mod tests { .get_finished_spans() .unwrap() .iter() - .any(|span| span.name == "driver.create_sandbox"), + .any(|span| span.name == "openshell.compute.v1.ComputeDriver/CreateSandbox"), "the RPC span should finish while background provisioning is still active" ); drop(provisioning); @@ -6175,7 +7209,7 @@ mod tests { let spans = traced.exporter.get_finished_spans().unwrap(); let deletion = spans .iter() - .find(|span| span.name == "vm.delete") + .find(|span| span.name == "vm.teardown") .expect("delete span"); assert!( matches!(deletion.status, opentelemetry::trace::Status::Error { .. }), @@ -8045,6 +9079,312 @@ mod tests { }; use crate::runtime::VmBackend; + /// Driver whose rootfs tar staging root is an isolated temp directory. + fn rootfs_tar_test_driver(staging_root: &Path, max_bytes: Option) -> VmDriver { + let (events, _) = broadcast::channel(WATCH_BUFFER); + VmDriver { + config: VmDriverConfig { + rootfs_tar_staging_dir: Some(staging_root.to_path_buf()), + rootfs_tar_max_bytes: max_bytes, + ..Default::default() + }, + launcher_bin: PathBuf::from("openshell-driver-vm"), + registry: Arc::new(Mutex::new(HashMap::new())), + image_cache_lock: Arc::new(Mutex::new(())), + events, + gpu_inventory: None, + subnet_allocator: Arc::new(std::sync::Mutex::new(SubnetAllocator::new( + Ipv4Addr::new(10, 0, 128, 0), + 17, + ))), + lifecycle_extensions: Arc::new(LifecycleExtensionRegistry::new()), + } + } + + /// `/req-/` with `contents`, the shape the + /// gateway allocates for one create request. + fn staged_rootfs_tar(staging_root: &Path, request: &str, contents: &[u8]) -> PathBuf { + let request_dir = staging_root.join(format!("req-{request}")); + std::fs::create_dir_all(&request_dir).expect("create request dir"); + let archive = request_dir.join("rootfs.tar"); + std::fs::write(&archive, contents).expect("write archive"); + archive + } + + #[tokio::test] + async fn validate_rootfs_tar_path_accepts_staged_archive() { + let root = unique_temp_dir(); + std::fs::create_dir_all(&root).expect("create staging root"); + let archive = staged_rootfs_tar(&root, "a", b"payload"); + let driver = rootfs_tar_test_driver(&root, None); + + let resolved = driver + .validate_rootfs_tar_path(&archive) + .await + .expect("a correctly staged archive is accepted"); + + assert_eq!( + resolved, + archive.canonicalize().expect("canonicalize archive") + ); + let _ = std::fs::remove_dir_all(&root); + } + + /// The core of the fix: a caller-named host path must never reach + /// privileged driver I/O, even if the caller is authenticated. + #[tokio::test] + async fn validate_rootfs_tar_path_rejects_arbitrary_host_paths() { + let root = unique_temp_dir(); + std::fs::create_dir_all(&root).expect("create staging root"); + let driver = rootfs_tar_test_driver(&root, None); + + for candidate in ["/etc/passwd", "/dev/zero"] { + let path = Path::new(candidate); + if !path.exists() { + continue; + } + let Err(err) = driver.validate_rootfs_tar_path(path).await else { + panic!("{candidate} must be rejected"); + }; + assert_eq!( + err.code(), + Code::PermissionDenied, + "{candidate} should be denied, got: {err}" + ); + } + let _ = std::fs::remove_dir_all(&root); + } + + #[tokio::test] + async fn validate_rootfs_tar_path_rejects_symlink_escape() { + let root = unique_temp_dir(); + let request_dir = root.join("req-a"); + std::fs::create_dir_all(&request_dir).expect("create request dir"); + let target = unique_temp_dir(); + std::fs::create_dir_all(&target).expect("create escape target dir"); + let secret = target.join("secret.tar"); + std::fs::write(&secret, b"not yours").expect("write escape target"); + let link = request_dir.join("rootfs.tar"); + std::os::unix::fs::symlink(&secret, &link).expect("create symlink"); + let driver = rootfs_tar_test_driver(&root, None); + + let err = driver + .validate_rootfs_tar_path(&link) + .await + .expect_err("a symlink out of the staging root must be rejected"); + + assert_eq!(err.code(), Code::PermissionDenied, "{err}"); + let _ = std::fs::remove_dir_all(&root); + let _ = std::fs::remove_dir_all(&target); + } + + #[tokio::test] + async fn validate_rootfs_tar_path_rejects_wrong_depth() { + let root = unique_temp_dir(); + std::fs::create_dir_all(&root).expect("create staging root"); + let shallow = root.join("rootfs.tar"); + std::fs::write(&shallow, b"payload").expect("write shallow archive"); + let deep_dir = root.join("req-a").join("nested"); + std::fs::create_dir_all(&deep_dir).expect("create deep dir"); + let deep = deep_dir.join("rootfs.tar"); + std::fs::write(&deep, b"payload").expect("write deep archive"); + let driver = rootfs_tar_test_driver(&root, None); + + for path in [&shallow, &deep] { + let err = driver + .validate_rootfs_tar_path(path) + .await + .expect_err("only request-directory depth is accepted"); + assert_eq!(err.code(), Code::PermissionDenied, "{err}"); + } + let _ = std::fs::remove_dir_all(&root); + } + + #[tokio::test] + async fn validate_rootfs_tar_path_rejects_directory() { + let root = unique_temp_dir(); + let request_dir = root.join("req-a"); + let not_a_file = request_dir.join("rootfs.tar"); + std::fs::create_dir_all(¬_a_file).expect("create directory in archive position"); + let driver = rootfs_tar_test_driver(&root, None); + + let err = driver + .validate_rootfs_tar_path(¬_a_file) + .await + .expect_err("a directory is not a rootfs tar"); + + assert_eq!(err.code(), Code::InvalidArgument, "{err}"); + let _ = std::fs::remove_dir_all(&root); + } + + #[tokio::test] + async fn validate_rootfs_tar_path_enforces_max_bytes() { + let root = unique_temp_dir(); + std::fs::create_dir_all(&root).expect("create staging root"); + let archive = staged_rootfs_tar(&root, "a", &[0_u8; 64]); + let driver = rootfs_tar_test_driver(&root, Some(16)); + + let err = driver + .validate_rootfs_tar_path(&archive) + .await + .expect_err("an oversized archive must be rejected"); + + assert_eq!(err.code(), Code::InvalidArgument, "{err}"); + let _ = std::fs::remove_dir_all(&root); + } + + /// Identity must follow the bytes, not the path. The gateway hands every + /// request its own staging directory, so a path-derived key would miss the + /// cache on every single create. + #[test] + fn rootfs_tar_cache_identity_tracks_contents_not_path() { + let same_a = rootfs_tar_cache_identity(&compute_bytes_sha256_hex(b"rootfs-bytes")); + let same_b = rootfs_tar_cache_identity(&compute_bytes_sha256_hex(b"rootfs-bytes")); + let different = rootfs_tar_cache_identity(&compute_bytes_sha256_hex(b"other-bytes")); + + assert_eq!( + same_a, same_b, + "identical contents must share one prepared disk" + ); + assert_ne!( + same_a, different, + "different contents must not collide on one prepared disk" + ); + } + + /// The old key was `path + seconds-truncated mtime` run through a + /// punctuation sanitizer, so `/tmp/a/b.tar` and `/tmp/a-b.tar` collided and + /// a long path could blow past filesystem component limits. + #[test] + fn rootfs_tar_cache_identity_is_bounded_and_separator_safe() { + let long_path_digest = compute_bytes_sha256_hex(&vec![7_u8; 4096]); + let identity = rootfs_tar_cache_identity(&long_path_digest); + let sanitized = sanitize_image_identity(&identity); + + assert!( + sanitized.len() < 255, + "cache directory component must stay within filesystem limits, got {}", + sanitized.len() + ); + assert_ne!( + rootfs_tar_cache_identity(&compute_bytes_sha256_hex(b"/tmp/a/b.tar")), + rootfs_tar_cache_identity(&compute_bytes_sha256_hex(b"/tmp/a-b.tar")), + "separator-colliding inputs must not share an identity" + ); + } + + const TEST_STAGING_LIMIT: u64 = 10 * 1024 * 1024; + + /// Build an uncompressed tar holding a single file. + fn tar_bytes_with_file(name: &str, contents: &[u8]) -> Vec { + let mut builder = tar::Builder::new(Vec::new()); + let mut header = tar::Header::new_gnu(); + header.set_size(u64::try_from(contents.len()).expect("tar entry size fits u64")); + header.set_mode(0o644); + header.set_cksum(); + builder + .append_data(&mut header, name, contents) + .expect("append tar entry"); + builder.into_inner().expect("finish tar") + } + + fn gzip_bytes(bytes: &[u8]) -> Vec { + let mut encoder = flate2::write::GzEncoder::new(Vec::new(), flate2::Compression::default()); + encoder.write_all(bytes).expect("gzip payload"); + encoder.finish().expect("finish gzip") + } + + #[test] + fn stage_rootfs_tar_archive_matches_source_digest() { + let base = unique_temp_dir(); + std::fs::create_dir_all(&base).expect("create base dir"); + let src = base.join("src.tar"); + let dst = base.join("dst.tar"); + let payload = vec![3_u8; 200 * 1024]; + std::fs::write(&src, &payload).expect("write source"); + + let copied = + stage_rootfs_tar_archive(&src, &dst, TEST_STAGING_LIMIT).expect("copy should succeed"); + + assert_eq!(copied, compute_file_sha256_hex(&src).expect("hash source")); + assert_eq!(copied, compute_bytes_sha256_hex(&payload)); + assert_eq!(std::fs::read(&dst).expect("read copy"), payload); + let _ = std::fs::remove_dir_all(&base); + } + + /// An archive rewritten between the hash pass and the copy pass yields a + /// different digest, which is what lets the caller reject it instead of + /// caching a disk under an identity that does not describe it. + #[test] + fn stage_rootfs_tar_archive_detects_content_change_between_passes() { + let base = unique_temp_dir(); + std::fs::create_dir_all(&base).expect("create base dir"); + let src = base.join("src.tar"); + std::fs::write(&src, b"original").expect("write source"); + let first = compute_file_sha256_hex(&src).expect("hash source"); + + std::fs::write(&src, b"replaced").expect("rewrite source"); + let second = stage_rootfs_tar_archive(&src, &base.join("dst.tar"), TEST_STAGING_LIMIT) + .expect("copy"); + + assert_ne!( + first, second, + "a mid-staging rewrite must produce a different digest" + ); + let _ = std::fs::remove_dir_all(&base); + } + + /// `--from` accepts `.tar.gz`/`.tgz`, and the guest extracts the staged + /// file as a plain tar, so staging has to decompress on the way in. + #[test] + fn stage_rootfs_tar_archive_decompresses_gzip_sources() { + let base = unique_temp_dir(); + std::fs::create_dir_all(&base).expect("create base dir"); + let tar = tar_bytes_with_file("etc/marker.txt", b"rootfs-tar-gzip\n"); + let gzipped = gzip_bytes(&tar); + let src = base.join("src.tar.gz"); + let dst = base.join("source-rootfs.tar"); + std::fs::write(&src, &gzipped).expect("write source"); + + let digest = + stage_rootfs_tar_archive(&src, &dst, TEST_STAGING_LIMIT).expect("stage gzip archive"); + + assert_eq!( + digest, + compute_bytes_sha256_hex(&gzipped), + "the digest must cover the whole compressed source" + ); + assert_eq!( + std::fs::read(&dst).expect("read staged archive"), + tar, + "the staged archive must be an uncompressed tar" + ); + + let extracted = base.join("extracted"); + extract_rootfs_archive_to(&dst, &extracted).expect("extract staged archive"); + assert_eq!( + std::fs::read_to_string(extracted.join("etc/marker.txt")).expect("read marker"), + "rootfs-tar-gzip\n" + ); + let _ = std::fs::remove_dir_all(&base); + } + + /// The configured limit bounds what the driver writes, not just what it + /// accepts, so a highly compressible archive cannot fill the host disk. + #[test] + fn stage_rootfs_tar_archive_rejects_oversized_expansion() { + let base = unique_temp_dir(); + std::fs::create_dir_all(&base).expect("create base dir"); + let src = base.join("bomb.tar.gz"); + std::fs::write(&src, gzip_bytes(&vec![0_u8; 4 * 1024 * 1024])).expect("write source"); + + let err = stage_rootfs_tar_archive(&src, &base.join("dst.tar"), 64 * 1024) + .expect_err("expansion beyond the limit must be rejected"); + + assert!(err.contains("65536"), "unexpected error: {err}"); + let _ = std::fs::remove_dir_all(&base); + } + fn test_driver_with_extensions(extensions: LifecycleExtensionRegistry) -> VmDriver { let (events, _) = broadcast::channel(WATCH_BUFFER); VmDriver { @@ -8069,6 +9409,12 @@ mod tests { } } + fn test_driver_with_proxy(https_proxy: &str) -> VmDriver { + let mut driver = test_driver_with_extensions(LifecycleExtensionRegistry::new()); + driver.config.https_proxy = Some(https_proxy.to_string()); + driver + } + #[derive(Debug)] struct QemuRequiringExtension { name: String, @@ -8398,4 +9744,487 @@ mod tests { assert!(err.is_resource_exhausted()); assert_eq!(err.message(), "pool empty"); } + + /// A driver config carrying only corporate proxy settings. + fn proxy_config( + https_proxy: Option<&str>, + auth_file: Option<&str>, + ca_bundle: Option<&str>, + ) -> VmDriverConfig { + VmDriverConfig { + openshell_endpoint: "http://127.0.0.1:8080".to_string(), + https_proxy: https_proxy.map(ToString::to_string), + proxy_auth_file: auth_file.map(ToString::to_string), + proxy_auth_allow_insecure: auth_file.map(|_| true), + proxy_ca_bundle: ca_bundle.map(ToString::to_string), + ..Default::default() + } + } + + #[test] + fn driver_config_debug_redacts_the_proxy_url_and_credential_path() { + // `Debug` can be emitted before validation runs, and an unvalidated + // proxy URL may still carry inline `user:pass@` credentials. + let rendered = format!( + "{:?}", + proxy_config( + Some("http://user:secret@proxy.corp.test:3128"), + Some("/etc/openshell/secrets/proxy-auth"), + Some("/etc/openshell/tls/corp-ca.pem"), + ) + ); + assert!( + !rendered.contains("secret") && !rendered.contains("proxy.corp.test"), + "the proxy URL must be logged as presence only: {rendered}" + ); + assert!( + !rendered.contains("/etc/openshell/secrets/proxy-auth"), + "the credential path must be logged as presence only: {rendered}" + ); + assert!( + rendered.contains("https_proxy: true") && rendered.contains("proxy_auth_file: true"), + "presence of each must still be visible for debugging: {rendered}" + ); + // A CA path is not sensitive and stays readable. + assert!( + rendered.contains("corp-ca.pem"), + "the CA bundle path is not a secret and should stay legible: {rendered}" + ); + } + + #[test] + fn proxy_material_is_staged_inside_the_per_sandbox_overlay() { + // Everything the driver stages lands in the overlay upperdir, which + // lives in the sandbox's own state directory. That is what makes the + // credential removable with the sandbox (remove_sandbox_state_dir + // deletes the whole directory) and unforgeable by the guest image + // (the upperdir shadows the read-only image layer). + for guest_path in [ + GUEST_UPSTREAM_PROXY_AUTH_PATH, + GUEST_PROXY_CA_PATH, + GUEST_SUPERVISOR_ARGS_PATH, + ] { + assert!( + guest_path.starts_with("/opt/openshell/"), + "{guest_path} must be under the reserved guest control root" + ); + assert_eq!( + overlay_upper_path(guest_path), + format!("/upper{guest_path}"), + "{guest_path} must be staged into the overlay upperdir" + ); + } + } + + #[test] + fn upstream_proxy_args_are_empty_without_a_configured_proxy() { + assert!(upstream_proxy_cli_args(&VmDriverConfig::default()).is_empty()); + // The file is still written, empty, so the guest cannot fall back to + // an image-baked argument list. + assert!(render_guest_supervisor_args(&[]).is_empty()); + } + + #[test] + fn upstream_proxy_args_pass_guest_paths_not_host_paths() { + let config = proxy_config( + Some("http://proxy.corp.test:3128"), + Some("/etc/openshell/secrets/proxy-auth"), + Some("/etc/openshell/tls/corp-ca.pem"), + ); + let args = upstream_proxy_cli_args(&config); + + // The credential and CA live at fixed guest paths; the gateway-host + // paths the operator configured must never reach the guest argv. + let auth = args + .iter() + .position(|arg| arg == "--upstream-proxy-auth-file") + .map(|i| args[i + 1].as_str()); + assert_eq!(auth, Some(GUEST_UPSTREAM_PROXY_AUTH_PATH)); + let ca = args + .iter() + .position(|arg| arg == "--upstream-proxy-ca-bundle") + .map(|i| args[i + 1].as_str()); + assert_eq!(ca, Some(GUEST_PROXY_CA_PATH)); + assert!( + !args + .iter() + .any(|arg| arg.contains("/etc/openshell/secrets") || arg.contains("corp-ca.pem")), + "host paths leaked into the guest argv: {args:?}" + ); + } + + #[test] + fn upstream_proxy_args_pass_only_explicit_opt_ins() { + let mut config = proxy_config(Some("https://proxy.corp.test:3130"), None, None); + config.no_proxy = Some("10.0.0.0/8,.svc.cluster.local".to_string()); + let args = upstream_proxy_cli_args(&config); + assert_eq!( + args, + vec![ + "--upstream-proxy".to_string(), + "https://proxy.corp.test:3130".to_string(), + "--upstream-no-proxy".to_string(), + "10.0.0.0/8,.svc.cluster.local".to_string(), + ] + ); + + // `Some(false)` must not be passed as the presence flag it is on the + // supervisor side. + config.proxy_connect_by_hostname = Some(false); + assert!( + !upstream_proxy_cli_args(&config) + .iter() + .any(|arg| arg == "--upstream-proxy-connect-by-hostname") + ); + config.proxy_connect_by_hostname = Some(true); + assert!( + upstream_proxy_cli_args(&config) + .iter() + .any(|arg| arg == "--upstream-proxy-connect-by-hostname") + ); + } + + #[test] + fn guest_supervisor_args_render_one_argument_per_line() { + let args = vec![ + "--upstream-proxy".to_string(), + "http://proxy.corp.test:3128".to_string(), + "--upstream-no-proxy".to_string(), + "a.example, b.example".to_string(), + ]; + // A value containing a space stays one line, so the guest reads it + // back as a single argument rather than word-splitting it. + assert_eq!( + String::from_utf8(render_guest_supervisor_args(&args)).unwrap(), + "--upstream-proxy\nhttp://proxy.corp.test:3128\n--upstream-no-proxy\na.example, b.example\n" + ); + } + + #[test] + fn guest_supervisor_args_reject_line_breaking_values() { + // A newline would split one operator value into two guest arguments. + for bad in ["a\nb", "a\rb", "a\0b"] { + assert!( + validate_guest_supervisor_args(&[bad.to_string()]).is_err(), + "{bad:?} must be rejected" + ); + } + validate_guest_supervisor_args(&["--upstream-proxy".to_string()]) + .expect("ordinary arguments are accepted"); + } + + #[test] + fn proxy_config_validation_rejects_settings_without_a_proxy_url() { + let config = VmDriverConfig { + no_proxy: Some("10.0.0.0/8".to_string()), + ..Default::default() + }; + let err = config + .validate_proxy_config() + .expect_err("a bypass list without a proxy would hide a fail-open state"); + assert!(err.contains("no_proxy"), "{err}"); + + let config = proxy_config(Some("http://proxy.corp.test:3128"), None, None); + config + .validate_proxy_config() + .expect("a lone proxy URL is a complete configuration"); + } + + #[test] + fn proxy_config_validation_requires_the_cleartext_acknowledgement() { + let mut config = proxy_config( + Some("http://proxy.corp.test:3128"), + Some("/etc/openshell/secrets/proxy-auth"), + None, + ); + config.proxy_auth_allow_insecure = None; + let err = config + .validate_proxy_config() + .expect_err("Basic auth to an http:// proxy is cleartext on the wire"); + assert!(err.contains("proxy_auth_allow_insecure"), "{err}"); + } + + #[tokio::test] + async fn proxy_ca_bundle_without_a_certificate_fails_the_sandbox() { + let dir = std::env::temp_dir().join(format!("openshell-vm-ca-{}", std::process::id())); + std::fs::create_dir_all(&dir).unwrap(); + let path = dir.join("not-a-ca.pem"); + std::fs::write(&path, b"this is not a certificate\n").unwrap(); + + let err = read_sandbox_proxy_ca_bundle(path.to_str().unwrap()) + .await + .expect_err("a certificate-free bundle must fail closed"); + assert_eq!(err.code(), Code::InvalidArgument); + assert!(err.message().contains("no PEM certificate"), "{err}"); + + std::fs::write(&path, b"").unwrap(); + let err = read_sandbox_proxy_ca_bundle(path.to_str().unwrap()) + .await + .expect_err("an empty bundle must fail closed"); + assert!(err.message().contains("no PEM certificate"), "{err}"); + + // PEM framing that base64-decodes but is not X.509 DER: accepted by + // `rustls_pemfile` alone, contributes zero trust anchors at runtime, + // and so would make every guest supervisor fail after boot. + std::fs::write( + &path, + b"-----BEGIN CERTIFICATE-----\nAQID\n-----END CERTIFICATE-----\n", + ) + .unwrap(); + let err = read_sandbox_proxy_ca_bundle(path.to_str().unwrap()) + .await + .expect_err("a bundle with invalid DER must fail closed"); + assert_eq!(err.code(), Code::InvalidArgument); + assert!(err.message().contains("no usable trust anchors"), "{err}"); + + let err = read_sandbox_proxy_ca_bundle(dir.join("missing.pem").to_str().unwrap()) + .await + .expect_err("an unreadable bundle must fail closed"); + assert!(err.message().contains("could not be read"), "{err}"); + + // A special file must be rejected on its type, not read: an + // unbounded read of /dev/zero would exhaust driver memory. + #[cfg(unix)] + { + let err = read_sandbox_proxy_ca_bundle("/dev/zero") + .await + .expect_err("a non-regular bundle path must fail closed"); + assert_eq!(err.code(), Code::InvalidArgument); + assert!(err.message().contains("not a regular file"), "{err}"); + } + + // Oversized regular file: rejected on the stat'd length, again + // without reading it whole. + let oversized = dir.join("oversized.pem"); + let bound = openshell_core::driver_utils::MAX_UPSTREAM_PROXY_CA_BUNDLE_BYTES; + std::fs::write(&oversized, vec![b'x'; usize::try_from(bound).unwrap() + 1]).unwrap(); + let err = read_sandbox_proxy_ca_bundle(oversized.to_str().unwrap()) + .await + .expect_err("an oversized bundle must fail closed"); + assert_eq!(err.code(), Code::InvalidArgument); + assert!(err.message().contains("exceeds"), "{err}"); + + std::fs::remove_dir_all(&dir).unwrap(); + } + + #[test] + fn qemu_backend_rejects_a_gateway_host_proxy() { + // gvproxy's host-loopback NAT has no QEMU/TAP equivalent, so a proxy + // on the gateway host is unreachable from a GPU sandbox and must be + // rejected rather than time out on every CONNECT. The address that + // reaches the gateway host from a QEMU guest is this sandbox's own + // TAP host address, so the classifier is parameterized by it. + let tap_host = Some("10.0.128.1"); + for url in [ + "http://host.openshell.internal:8080", + "http://host.containers.internal:8080", + "http://host.docker.internal:8080", + "http://127.0.0.1:8080", + "http://localhost:8080", + "https://[::1]:8080", + // The address the aliases above resolve to inside the guest. + "http://10.0.128.1:8080", + ] { + assert!(proxy_url_targets_gateway_host(url, tap_host), "{url}"); + } + for url in [ + "http://proxy.corp.example:8080", + "https://10.1.2.3:3128", + // Special only to libkrun/gvproxy. On QEMU/TAP it is an ordinary + // address that may be routable through the guest's masqueraded + // egress, so rejecting it would refuse a working configuration. + "http://192.168.127.254:8080", + // Another sandbox's TAP host, not this one's. + "http://10.0.128.5:8080", + "not a url", + ] { + assert!(!proxy_url_targets_gateway_host(url, tap_host), "{url}"); + } + + // Without an allocated TAP host only the address-independent + // spellings classify; the loopback and alias guards still hold. + assert!(proxy_url_targets_gateway_host( + "http://127.0.0.1:8080", + None + )); + assert!(proxy_url_targets_gateway_host( + "http://host.openshell.internal:8080", + None + )); + assert!(!proxy_url_targets_gateway_host( + "http://10.0.128.1:8080", + None + )); + } + + #[test] + fn qemu_launch_plan_rejects_a_proxy_at_the_allocated_tap_host() { + // The preflight has to run against the address this sandbox actually + // got, which only exists once the launch plan's subnet is allocated. + // A proxy there is what `host.openshell.internal` resolves to in the + // guest, and the driver's own nftables input chain drops the port. + let probe = test_driver_with_extensions(LifecycleExtensionRegistry::new()); + let tap_host = probe + .build_vm_launch_plan("sandbox-proxy-tap", true, true, None) + .expect("gpu plan should build") + .host_ip + .expect("a QEMU plan carries a TAP host address"); + probe.release_subnet("sandbox-proxy-tap"); + + let driver = test_driver_with_proxy(&format!("http://{tap_host}:8080")); + let mut plan = driver + .build_vm_launch_plan("sandbox-proxy-tap", true, true, None) + .expect("gpu plan should build"); + assert_eq!(plan.host_ip.as_deref(), Some(tap_host.as_str())); + + let err = driver + .resolve_launch_plan_backend("sandbox-proxy-tap", true, None, &mut plan) + .expect_err("a proxy at the TAP host address is unreachable from the guest"); + assert_eq!(err.code(), Code::FailedPrecondition); + assert!(err.message().contains(&tap_host), "{err}"); + + driver.release_subnet("sandbox-proxy-tap"); + } + + #[test] + fn qemu_launch_plan_allows_a_proxy_at_the_gvproxy_host_loopback_address() { + // 192.168.127.254 carries no meaning on QEMU/TAP, so a launch must + // proceed rather than be refused for a libkrun-only reason. + let driver = test_driver_with_proxy(&format!("http://{GVPROXY_HOST_LOOPBACK_IP}:8080")); + let mut plan = driver + .build_vm_launch_plan("sandbox-proxy-gvproxy", true, true, None) + .expect("gpu plan should build"); + assert_ne!(plan.host_ip.as_deref(), Some(GVPROXY_HOST_LOOPBACK_IP)); + + driver + .resolve_launch_plan_backend("sandbox-proxy-gvproxy", true, None, &mut plan) + .expect("a routable proxy address must not block a GPU launch"); + assert_eq!(plan.backend, VmBackend::Qemu); + + driver.release_subnet("sandbox-proxy-gvproxy"); + } + + #[tokio::test] + async fn proxy_credential_is_validated_against_the_supervisor_rules() { + let dir = std::env::temp_dir().join(format!("openshell-vm-cred-{}", std::process::id())); + std::fs::create_dir_all(&dir).unwrap(); + let path = dir.join("proxy-auth"); + + std::fs::write(&path, "proxyuser:proxypass\n").unwrap(); + assert_eq!( + read_sandbox_proxy_credential(path.to_str().unwrap()) + .await + .expect("a well-formed credential is accepted"), + "proxyuser:proxypass" + ); + + // Rejected here rather than inside every sandbox's supervisor. + std::fs::write(&path, "no-separator\n").unwrap(); + let err = read_sandbox_proxy_credential(path.to_str().unwrap()) + .await + .expect_err("a malformed credential must fail closed"); + assert_eq!(err.code(), Code::InvalidArgument); + assert!( + !err.message().contains("no-separator"), + "the error must not echo credential file contents: {err}" + ); + + std::fs::remove_dir_all(&dir).unwrap(); + } + + #[test] + fn guest_environment_carries_no_corporate_proxy_settings() { + // The egress boundary is argv-only: `build_guest_environment` merges + // user-supplied environment, so anything it emitted here would be + // attacker-influenced. + let config = proxy_config( + Some("http://proxy.corp.test:3128"), + Some("/etc/openshell/secrets/proxy-auth"), + Some("/etc/openshell/tls/corp-ca.pem"), + ); + let sandbox = Sandbox { + id: "sb-proxy".to_string(), + name: "proxy".to_string(), + spec: Some(SandboxSpec { + environment: [ + ( + "HTTPS_PROXY".to_string(), + "http://attacker:3128".to_string(), + ), + ("NO_PROXY".to_string(), "*".to_string()), + ] + .into_iter() + .collect(), + ..Default::default() + }), + ..Default::default() + }; + + let env = build_guest_environment(&sandbox, &config, None); + assert!( + !env.iter().any(|entry| entry.starts_with("--upstream")), + "driver environment must never carry supervisor arguments: {env:?}" + ); + // A sandbox may still set the conventional variables for its own + // workload, but the supervisor ignores them on this path -- what + // matters is that the driver never derives the boundary from them. + assert!( + !env.iter() + .any(|entry| entry.contains("proxy.corp.test") || entry.contains("proxy-auth")), + "operator proxy settings must not reach the guest environment: {env:?}" + ); + } + + #[test] + fn sandbox_driver_config_cannot_carry_proxy_settings() { + // The upstream proxy is host network topology, not a per-sandbox + // setting: the caller-supplied envelope must reject it outright + // rather than silently ignoring it. + for key in [ + "https_proxy", + "no_proxy", + "proxy_auth_file", + "proxy_auth_allow_insecure", + "proxy_connect_by_hostname", + "proxy_ca_bundle", + ] { + let template = SandboxTemplate { + driver_config: Some(Struct { + fields: std::iter::once(( + key.to_string(), + Value { + kind: Some(Kind::StringValue("http://attacker:3128".to_string())), + }, + )) + .collect(), + }), + ..Default::default() + }; + assert!( + VmSandboxDriverConfig::from_template(&template).is_err(), + "template.driver_config.vm must reject '{key}'" + ); + } + } + + #[test] + fn capabilities_report_static_resource_support() { + let mut driver = test_driver_with_extensions(LifecycleExtensionRegistry::new()); + let resources = driver.capabilities().resource_capabilities.unwrap(); + assert!(!resources.cpu.unwrap().limit_supported); + assert!(!resources.memory.unwrap().limit_supported); + let gpu = resources.gpu.unwrap(); + assert!(!gpu.default_selection_supported); + assert!(!gpu.count_selection_supported); + + driver.config.gpu_enabled = true; + let gpu = driver + .capabilities() + .resource_capabilities + .unwrap() + .gpu + .unwrap(); + assert!(gpu.default_selection_supported); + assert!(gpu.count_selection_supported); + } } diff --git a/crates/openshell-driver-vm/src/lib.rs b/crates/openshell-driver-vm/src/lib.rs index 98ba6b0c9..f34c7dda8 100644 --- a/crates/openshell-driver-vm/src/lib.rs +++ b/crates/openshell-driver-vm/src/lib.rs @@ -1,6 +1,16 @@ // SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 +// `defaults-without-telemetry` is an alias for the default feature set minus +// `telemetry`, not a switch that turns telemetry off. Cargo cannot subtract a +// default feature, so adding it on top of the defaults would otherwise produce +// a telemetry-on build that reads as telemetry-free. Fail the build instead. +#[cfg(all(feature = "telemetry", feature = "defaults-without-telemetry"))] +compile_error!( + "features `telemetry` and `defaults-without-telemetry` are mutually exclusive; \ + build a telemetry-free VM driver with `--no-default-features --features defaults-without-telemetry`" +); + pub mod driver; mod embedded_runtime; mod ffi; diff --git a/crates/openshell-driver-vm/src/main.rs b/crates/openshell-driver-vm/src/main.rs index 95ebf0f8b..1570ba841 100644 --- a/crates/openshell-driver-vm/src/main.rs +++ b/crates/openshell-driver-vm/src/main.rs @@ -6,7 +6,6 @@ use futures::Stream; use miette::{IntoDiagnostic, Result}; use openshell_core::VERSION; use openshell_core::proto::compute::v1::compute_driver_server::ComputeDriverServer; -use openshell_driver_vm::otel_tracing::compute_driver_rpc_layer; #[cfg(target_os = "macos")] use openshell_driver_vm::{VM_RUNTIME_DIR_ENV, configured_runtime_dir}; use openshell_driver_vm::{VmBackend, VmDriver, VmDriverConfig, VmLaunchConfig, procguard, run_vm}; @@ -18,8 +17,6 @@ use std::pin::Pin; use std::task::{Context, Poll}; use tokio::net::{UnixListener, UnixStream}; use tracing::info; -use tracing_subscriber::EnvFilter; -use tracing_subscriber::prelude::*; #[derive(Parser, Debug)] #[command(name = "openshell-driver-vm")] @@ -146,6 +143,36 @@ struct Args { #[arg(long, env = "OPENSHELL_VM_SANDBOX_GID")] sandbox_gid: Option, + // Corporate forward proxy for sandbox egress. Operator-owned: these reach + // the guest supervisor on its argv, which the sandbox image and the + // user-supplied environment cannot influence. + #[arg(long, env = "OPENSHELL_VM_HTTPS_PROXY")] + https_proxy: Option, + + #[arg(long, env = "OPENSHELL_VM_NO_PROXY")] + no_proxy: Option, + + #[arg(long, env = "OPENSHELL_VM_PROXY_AUTH_FILE")] + proxy_auth_file: Option, + + // Value-taking rather than a presence flag so an explicit `false` in + // `[openshell.drivers.vm]` survives the gateway -> driver hop and still + // trips the "acknowledgement without a credential" check. + #[arg(long, env = "OPENSHELL_VM_PROXY_AUTH_ALLOW_INSECURE")] + proxy_auth_allow_insecure: Option, + + #[arg(long, env = "OPENSHELL_VM_PROXY_CONNECT_BY_HOSTNAME")] + proxy_connect_by_hostname: Option, + + #[arg(long, env = "OPENSHELL_VM_PROXY_CA_BUNDLE")] + proxy_ca_bundle: Option, + + #[arg(long, env = "OPENSHELL_VM_ROOTFS_TAR_STAGING_DIR")] + rootfs_tar_staging_dir: Option, + + #[arg(long, env = "OPENSHELL_VM_ROOTFS_TAR_MAX_BYTES")] + rootfs_tar_max_bytes: Option, + #[arg(long, hide = true)] vm_backend: Option, @@ -189,24 +216,15 @@ async fn main() -> Result<()> { return Ok(()); } - let (tracer_provider, setup_error) = openshell_driver_vm::otel_tracing::provider_for( - args.otlp_endpoint.as_deref(), - args.gateway_name.as_deref(), + let _tracing = openshell_otel::install_driver_tracing( + openshell_driver_vm::otel_tracing::TRACING, + openshell_otel::DriverTracingConfig { + endpoint: args.otlp_endpoint.as_deref(), + gateway_name: args.gateway_name.as_deref(), + service_version: VERSION, + log_level: &args.log_level, + }, ); - tracing_subscriber::registry() - .with(EnvFilter::try_from_default_env().unwrap_or_else(|_| EnvFilter::new(&args.log_level))) - .with(tracing_subscriber::fmt::layer()) - .with( - tracer_provider - .as_ref() - .map(openshell_driver_vm::otel_tracing::layer), - ) - .init(); - if let Some(error) = setup_error { - tracing::error!(%error, "OTLP exporting could not be started"); - } else if let Some(endpoint) = &args.otlp_endpoint { - info!(endpoint, "OTLP exporting enabled"); - } let listen_mode = compute_driver_listen_mode(&args).map_err(|err| miette::miette!("{err}"))?; @@ -243,11 +261,19 @@ async fn main() -> Result<()> { gpu_vcpus: args.gpu_vcpus, sandbox_uid: args.sandbox_uid, sandbox_gid: args.sandbox_gid, + https_proxy: args.https_proxy.clone(), + no_proxy: args.no_proxy.clone(), + proxy_auth_file: args.proxy_auth_file.clone(), + proxy_auth_allow_insecure: args.proxy_auth_allow_insecure, + proxy_connect_by_hostname: args.proxy_connect_by_hostname, + proxy_ca_bundle: args.proxy_ca_bundle.clone(), + rootfs_tar_staging_dir: args.rootfs_tar_staging_dir.clone(), + rootfs_tar_max_bytes: args.rootfs_tar_max_bytes, }) .await .map_err(|err| miette::miette!("{err}"))?; - let result = match listen_mode { + match listen_mode { ComputeDriverListenMode::Unix { socket_path, expected_peer_pid, @@ -258,7 +284,7 @@ async fn main() -> Result<()> { let listener = UnixListener::bind(&socket_path).into_diagnostic()?; restrict_socket_permissions(&socket_path).map_err(|err| miette::miette!("{err}"))?; let result = tonic::transport::Server::builder() - .layer(compute_driver_rpc_layer()) + .layer(openshell_otel::compute_driver_rpc_layer()) .add_service(ComputeDriverServer::new(driver)) .serve_with_incoming_shutdown( AuthenticatedUnixIncoming::new(listener, expected_peer_pid), @@ -272,19 +298,13 @@ async fn main() -> Result<()> { ComputeDriverListenMode::Tcp(bind_address) => { info!(address = %bind_address, "Starting unauthenticated dev vm compute driver"); tonic::transport::Server::builder() - .layer(compute_driver_rpc_layer()) + .layer(openshell_otel::compute_driver_rpc_layer()) .add_service(ComputeDriverServer::new(driver)) .serve_with_shutdown(bind_address, shutdown_signal()) .await .into_diagnostic() } - }; - if let Some(provider) = &tracer_provider - && let Err(error) = provider.shutdown() - { - tracing::warn!(%error, "OTLP tracer provider shutdown failed"); } - result } async fn shutdown_signal() { @@ -620,6 +640,63 @@ mod tests { use clap::Parser; use std::path::PathBuf; + #[test] + fn corporate_proxy_flags_parse_into_driver_settings() { + let args = Args::parse_from([ + "openshell-driver-vm", + "--openshell-endpoint", + "https://host.openshell.internal:17670", + "--https-proxy", + "http://proxy.corp.com:8080", + "--no-proxy", + "10.0.0.0/8,.svc.cluster.local", + "--proxy-auth-file", + "/etc/openshell/secrets/proxy-auth", + "--proxy-auth-allow-insecure", + "true", + "--proxy-connect-by-hostname", + "false", + "--proxy-ca-bundle", + "/etc/openshell/tls/proxy-ca.pem", + ]); + + assert_eq!( + args.https_proxy.as_deref(), + Some("http://proxy.corp.com:8080") + ); + assert_eq!( + args.no_proxy.as_deref(), + Some("10.0.0.0/8,.svc.cluster.local") + ); + assert_eq!( + args.proxy_auth_file.as_deref(), + Some("/etc/openshell/secrets/proxy-auth") + ); + assert_eq!(args.proxy_auth_allow_insecure, Some(true)); + // Value-taking rather than a presence flag, so the gateway can + // forward an explicit `false` from `[openshell.drivers.vm]`. + assert_eq!(args.proxy_connect_by_hostname, Some(false)); + assert_eq!( + args.proxy_ca_bundle.as_deref(), + Some("/etc/openshell/tls/proxy-ca.pem") + ); + } + + #[test] + fn corporate_proxy_settings_default_to_unset() { + let args = Args::parse_from([ + "openshell-driver-vm", + "--openshell-endpoint", + "https://host.openshell.internal:17670", + ]); + assert!(args.https_proxy.is_none()); + assert!(args.no_proxy.is_none()); + assert!(args.proxy_auth_file.is_none()); + assert!(args.proxy_auth_allow_insecure.is_none()); + assert!(args.proxy_connect_by_hostname.is_none()); + assert!(args.proxy_ca_bundle.is_none()); + } + #[test] fn peer_authorization_accepts_matching_uid_and_pid() { authorize_peer_credentials( diff --git a/crates/openshell-driver-vm/src/otel_tracing.rs b/crates/openshell-driver-vm/src/otel_tracing.rs index 47e2dd5f7..a931ff502 100644 --- a/crates/openshell-driver-vm/src/otel_tracing.rs +++ b/crates/openshell-driver-vm/src/otel_tracing.rs @@ -1,133 +1,19 @@ // SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 -//! OpenTelemetry trace exporting. +//! OpenTelemetry tracing identity for the VM compute driver. -use openshell_otel::{OtlpTraceConfig, SdkTracerProvider, ServiceName, SetupError}; -pub use openshell_otel::{compute_driver_rpc_layer, compute_driver_rpc_operation}; -use tracing::Subscriber; -use tracing_subscriber::registry::LookupSpan; - -const SERVICE_NAME: &str = "openshell-driver-vm"; -const INSTRUMENTATION_SCOPE: &str = "openshell-driver-vm"; - -/// Build a tracer provider for the configured OTLP/gRPC endpoint and gateway. -#[must_use] -pub fn provider_for( - endpoint: Option<&str>, - gateway_name: Option<&str>, -) -> (Option, Option) { - openshell_otel::provider_for(endpoint.map(|endpoint| { - OtlpTraceConfig { - endpoint, - service_name: ServiceName::Fixed(SERVICE_NAME), - service_version: Some(openshell_core::VERSION), - resource_attributes: gateway_name - .map(str::trim) - .filter(|name| !name.is_empty()) - .map(|name| { - vec![opentelemetry::KeyValue::new( - "openshell.gateway.name", - name.to_string(), - )] - }) - .unwrap_or_default(), - } - })) -} - -/// Build the tracing layer that exports VM-driver spans. -pub fn layer(provider: &SdkTracerProvider) -> openshell_otel::OtlpLayer -where - S: Subscriber + for<'span> LookupSpan<'span>, -{ - openshell_otel::layer(provider, INSTRUMENTATION_SCOPE) -} +pub const TRACING: openshell_otel::ComputeDriverTracing = openshell_otel::compute_driver_tracing!(); #[cfg(test)] mod tests { - use openshell_otel_test_support::OtlpTestServer; - use tracing_subscriber::layer::SubscriberExt as _; - - #[test] - fn compute_driver_rpc_names_are_explicitly_mapped_and_schema_bounded() { - for (rpc, operation, method) in [ - ( - "GetCapabilities", - "driver.get_capabilities", - "get_capabilities", - ), - ( - "GetGatewayListenerRequirements", - "driver.get_gateway_listener_requirements", - "get_gateway_listener_requirements", - ), - ( - "ValidateSandboxCreate", - "driver.validate_sandbox_create", - "validate_sandbox_create", - ), - ("CreateSandbox", "driver.create_sandbox", "create_sandbox"), - ("GetSandbox", "driver.get_sandbox", "get_sandbox"), - ("ListSandboxes", "driver.list_sandboxes", "list_sandboxes"), - ("StopSandbox", "driver.stop_sandbox", "stop_sandbox"), - ("StartSandbox", "driver.start_sandbox", "start_sandbox"), - ("DeleteSandbox", "driver.delete_sandbox", "delete_sandbox"), - ( - "WatchSandboxes", - "driver.watch_sandboxes", - "watch_sandboxes", - ), - ] { - assert_eq!( - super::compute_driver_rpc_operation(&format!( - "/openshell.compute.v1.ComputeDriver/{rpc}" - )), - (operation, method), - "{rpc} must keep an explicit low-cardinality span identity" - ); - } - assert_eq!( - super::compute_driver_rpc_operation( - "/openshell.compute.v1.ComputeDriver/AttackerControlled12345" - ), - ("driver.unknown", "unknown"), - "paths absent from the protobuf schema must not create span names" - ); - } - #[tokio::test(flavor = "multi_thread", worker_threads = 2)] - async fn vm_driver_spans_reach_otlp_collector_with_resource_identity() { - let collector = OtlpTestServer::start().await; - - let (provider, error) = - super::provider_for(Some(collector.endpoint()), Some("production-us-west")); - assert!(error.is_none(), "valid OTLP endpoint should configure"); - let provider = provider.expect("provider"); - let subscriber = tracing_subscriber::registry().with(super::layer(&provider)); - tracing::subscriber::with_default(subscriber, || { - let span = tracing::info_span!("vm.provision", sandbox.id = "sb-otlp"); - drop(span.enter()); - drop(span); - }); - provider.force_flush().unwrap(); - collector.wait_for_export().await; - provider.shutdown().unwrap(); - let received = collector.shutdown().await; - - received - .spans - .iter() - .find(|span| span.name == "vm.provision") - .expect("VM span should reach collector"); - assert!( - received - .service_names - .iter() - .any(|name| name == "openshell-driver-vm"), - "VM spans should use a distinct service name, got {:?}", - received.service_names - ); - assert_eq!(received.gateway_names, ["production-us-west"]); + async fn driver_spans_reach_otlp_collector_with_resource_identity() { + openshell_otel_test_support::assert_compute_driver_tracing( + super::TRACING, + openshell_core::VERSION, + "vm.provision", + ) + .await; } } diff --git a/crates/openshell-driver-vm/src/rootfs.rs b/crates/openshell-driver-vm/src/rootfs.rs index 9046913c9..5e8de5f69 100644 --- a/crates/openshell-driver-vm/src/rootfs.rs +++ b/crates/openshell-driver-vm/src/rootfs.rs @@ -1,11 +1,12 @@ // SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 +use flate2::read::MultiGzDecoder; use std::fs; use std::fs::File; #[cfg(test)] use std::io::BufWriter; -use std::io::{Cursor, Read, Seek, SeekFrom, Write}; +use std::io::{BufRead, BufReader, Cursor, Read, Seek, SeekFrom, Write}; use std::path::{Path, PathBuf}; use std::process::Command; use std::sync::atomic::{AtomicU64, Ordering}; @@ -13,6 +14,9 @@ use std::sync::atomic::{AtomicU64, Ordering}; const SUPERVISOR: &[u8] = include_bytes!(concat!(env!("OUT_DIR"), "/openshell-sandbox.zst")); const UMOCI: &[u8] = include_bytes!(concat!(env!("OUT_DIR"), "/umoci.zst")); const ROOTFS_VARIANT_MARKER: &str = ".openshell-rootfs-variant"; +/// Leading bytes of a gzip stream, used to recognize `.tar.gz`/`.tgz` input +/// without trusting the file name. +pub const GZIP_MAGIC: [u8; 2] = [0x1f, 0x8b]; const SANDBOX_GUEST_INIT_PATH: &str = "/srv/openshell-vm-sandbox-init.sh"; const SANDBOX_SUPERVISOR_PATH: &str = openshell_core::driver_utils::SUPERVISOR_CONTAINER_BINARY; const SANDBOX_UMOCI_PATH: &str = openshell_core::container_paths::VM_UMOCI_PATH; @@ -44,6 +48,12 @@ pub fn prepare_sandbox_rootfs_from_image_root( Ok(()) } +/// Extract a rootfs tarball, transparently decompressing gzip archives. +/// +/// Compression is detected from the magic bytes rather than the file name: +/// `--from` accepts `.tar.gz` and `.tgz`, but nothing guarantees a caller's +/// extension matches the bytes, and the archives this crate stages internally +/// carry no extension at all. pub fn extract_rootfs_archive_to(archive_path: &Path, dest: &Path) -> Result<(), String> { if dest.exists() { fs::remove_dir_all(dest) @@ -53,8 +63,20 @@ pub fn extract_rootfs_archive_to(archive_path: &Path, dest: &Path) -> Result<(), fs::create_dir_all(dest).map_err(|e| format!("create rootfs dir {}: {e}", dest.display()))?; let file = File::open(archive_path).map_err(|e| format!("open {}: {e}", archive_path.display()))?; - let mut archive = tar::Archive::new(file); - archive + let mut reader = BufReader::new(file); + let compressed = reader + .fill_buf() + .map_err(|e| format!("read {}: {e}", archive_path.display()))? + .starts_with(&GZIP_MAGIC); + if compressed { + unpack_tar_reader(MultiGzDecoder::new(reader), dest) + } else { + unpack_tar_reader(reader, dest) + } +} + +fn unpack_tar_reader(reader: impl Read, dest: &Path) -> Result<(), String> { + tar::Archive::new(reader) .unpack(dest) .map_err(|e| format!("extract rootfs tarball into {}: {e}", dest.display())) } @@ -1031,6 +1053,36 @@ mod tests { let _ = fs::remove_dir_all(&dir); } + /// `--from` accepts `.tar.gz` and `.tgz`, so extraction must recognize a + /// gzip stream instead of handing compressed bytes to the tar reader. + #[test] + fn extract_rootfs_archive_accepts_gzip_archives() { + let dir = unique_temp_dir(); + let rootfs = dir.join("rootfs"); + let extracted = dir.join("extracted"); + let archive = dir.join("rootfs.tar"); + let gz_archive = dir.join("rootfs.tar.gz"); + + fs::create_dir_all(rootfs.join("etc")).expect("create etc"); + fs::write(rootfs.join("etc/marker.txt"), "gzip-rootfs\n").expect("write marker"); + create_rootfs_archive_from_dir(&rootfs, &archive).expect("archive rootfs"); + + let mut encoder = flate2::write::GzEncoder::new(Vec::new(), flate2::Compression::default()); + encoder + .write_all(&fs::read(&archive).expect("read archive")) + .expect("gzip archive"); + fs::write(&gz_archive, encoder.finish().expect("finish gzip")).expect("write gzip archive"); + + extract_rootfs_archive_to(&gz_archive, &extracted).expect("extract gzip rootfs"); + + assert_eq!( + fs::read_to_string(extracted.join("etc/marker.txt")).expect("read extracted marker"), + "gzip-rootfs\n" + ); + + let _ = fs::remove_dir_all(&dir); + } + #[cfg(unix)] #[test] fn create_rootfs_archive_preserves_broken_symlinks() { diff --git a/crates/openshell-extension-core/src/jwt.rs b/crates/openshell-extension-core/src/jwt.rs index 4bf5d3918..9d85858de 100644 --- a/crates/openshell-extension-core/src/jwt.rs +++ b/crates/openshell-extension-core/src/jwt.rs @@ -9,7 +9,7 @@ use serde::{Deserialize, Serialize}; /// /// Extension credentials cross the gateway trust boundary and must remain /// short-lived even when legacy sandbox bootstrap credentials do not expire. -pub const MAX_EXTENSION_TOKEN_TTL: Duration = Duration::from_secs(3_600); +pub const MAX_EXTENSION_TOKEN_TTL: Duration = Duration::from_hours(1); /// Explicit `typ` header value carried by every extension bearer token. /// diff --git a/crates/openshell-extension-core/src/transport.rs b/crates/openshell-extension-core/src/transport.rs index 4552faf70..98bbeb89d 100644 --- a/crates/openshell-extension-core/src/transport.rs +++ b/crates/openshell-extension-core/src/transport.rs @@ -292,7 +292,6 @@ mod tests { #[tokio::test] async fn custom_ca_verifies_certificate_and_hostname() { - let _ = rustls::crypto::ring::default_provider().install_default(); let ca_key = KeyPair::generate().unwrap(); let mut ca_params = CertificateParams::new(Vec::::new()).unwrap(); ca_params.is_ca = IsCa::Ca(BasicConstraints::Unconstrained); diff --git a/crates/openshell-gateway-interceptors/src/profile_source.rs b/crates/openshell-gateway-interceptors/src/profile_source.rs index 48014258a..11ee46555 100644 --- a/crates/openshell-gateway-interceptors/src/profile_source.rs +++ b/crates/openshell-gateway-interceptors/src/profile_source.rs @@ -5,6 +5,7 @@ use std::time::Duration; +use openshell_core::mcp::normalize_provider_profile_mcp_fields; use openshell_core::proto::ProviderProfile; use openshell_core::proto::gateway_interceptor::v1::{ ProviderProfileSnapshotRequest, gateway_interceptor_client::GatewayInterceptorClient, @@ -70,11 +71,8 @@ impl GatewayInterceptorProfileSource { })? .into_inner(); - let revision = if response.revision.trim().is_empty() { - provider_profile_snapshot_revision(&response.profiles) - } else { - response.revision - }; + let revision = + resolve_provider_profile_snapshot_revision(response.revision, &response.profiles); Ok(ProviderProfileSourceSnapshot { revision, profiles: response.profiles, @@ -94,6 +92,9 @@ impl std::fmt::Debug for GatewayInterceptorProfileSource { fn provider_profile_snapshot_revision(profiles: &[ProviderProfile]) -> String { let mut profiles = profiles.to_vec(); + profiles + .iter_mut() + .for_each(normalize_provider_profile_mcp_fields); profiles.sort_by(|left, right| left.id.cmp(&right.id)); let mut hasher = sha2::Sha256::new(); hasher.update(b"openshell-provider-profile-snapshot-v1"); @@ -102,3 +103,107 @@ fn provider_profile_snapshot_revision(profiles: &[ProviderProfile]) -> String { } format!("sha256:{:x}", hasher.finalize()) } + +fn resolve_provider_profile_snapshot_revision( + revision: String, + profiles: &[ProviderProfile], +) -> String { + if revision.trim().is_empty() { + // Whitespace-only carries no source identity, so treat it as omitted + // and derive a stable revision from the canonical profile payloads. + provider_profile_snapshot_revision(profiles) + } else { + // A nonblank revision is opaque source-owned state. Preserve it + // exactly instead of substituting a locally derived fingerprint. + revision + } +} + +#[cfg(test)] +mod tests { + use openshell_core::proto::{McpOptions, NetworkEndpoint}; + + use super::*; + + fn profile_with_mcp_versions(versions: Option<&[&str]>) -> ProviderProfile { + ProviderProfile { + id: "governed-mcp".to_string(), + endpoints: vec![NetworkEndpoint { + host: "mcp.example.com".to_string(), + port: 443, + protocol: "mcp".to_string(), + mcp: versions.map(|versions| McpOptions { + versions: versions + .iter() + .map(|version| (*version).to_string()) + .collect(), + ..McpOptions::default() + }), + ..NetworkEndpoint::default() + }], + ..ProviderProfile::default() + } + } + + #[test] + fn fallback_revision_equates_omitted_empty_and_explicit_default_mcp_versions() { + let omitted = profile_with_mcp_versions(None); + let empty = profile_with_mcp_versions(Some(&[])); + let explicit = profile_with_mcp_versions(Some(&["2025-11-25"])); + + let expected = resolve_provider_profile_snapshot_revision( + " \t".to_string(), + std::slice::from_ref(&explicit), + ); + assert_eq!( + resolve_provider_profile_snapshot_revision( + String::new(), + std::slice::from_ref(&omitted), + ), + expected + ); + assert_eq!( + resolve_provider_profile_snapshot_revision(String::new(), std::slice::from_ref(&empty),), + expected + ); + } + + #[test] + fn fallback_revision_equates_valid_mcp_version_orderings() { + let canonical = profile_with_mcp_versions(Some(&["2025-03-26", "2025-11-25"])); + let reordered = profile_with_mcp_versions(Some(&["2025-11-25", "2025-03-26"])); + + assert_eq!( + provider_profile_snapshot_revision(std::slice::from_ref(&canonical)), + provider_profile_snapshot_revision(std::slice::from_ref(&reordered)) + ); + } + + #[test] + fn fallback_revision_preserves_malformed_mcp_evidence() { + let malformed = profile_with_mcp_versions(Some(&["latest", "2025-11-25", "2025-11-25"])); + let original = malformed.clone(); + let canonical = profile_with_mcp_versions(Some(&["2025-11-25"])); + + assert_ne!( + provider_profile_snapshot_revision(std::slice::from_ref(&malformed)), + provider_profile_snapshot_revision(std::slice::from_ref(&canonical)) + ); + assert_eq!(malformed, original); + } + + #[test] + fn source_supplied_revision_remains_opaque_and_exact() { + let profiles = [profile_with_mcp_versions(None)]; + let source_revision = " governance:v7 ".to_string(); + + assert_eq!( + resolve_provider_profile_snapshot_revision(source_revision.clone(), &profiles), + source_revision + ); + assert_eq!( + resolve_provider_profile_snapshot_revision(" \t".to_string(), &profiles), + provider_profile_snapshot_revision(&profiles) + ); + } +} diff --git a/crates/openshell-gateway-interceptors/src/proto_json.rs b/crates/openshell-gateway-interceptors/src/proto_json.rs index b0d6b0f53..dde37190e 100644 --- a/crates/openshell-gateway-interceptors/src/proto_json.rs +++ b/crates/openshell-gateway-interceptors/src/proto_json.rs @@ -291,7 +291,7 @@ mod tests { use openshell_core::proto::{ CreateProviderRequest, CreateSandboxRequest, GpuResourceRequirements, Provider, - SandboxSpec, UpdateConfigRequest, + SandboxSpec, UpdateConfigRequest, workspace_selector, }; use prost::Message as _; use prost_types::{ @@ -315,8 +315,9 @@ mod tests { name: "demo".to_string(), labels: HashMap::from([("team".to_string(), "agent".to_string())]), annotations: HashMap::new(), - workspace: String::new(), + workspace_scope: Some(workspace_selector("default")), await_main_process_attachment: false, + workload_template_name: String::new(), }; let bytes = request.encode_to_vec(); let json = codec @@ -344,7 +345,7 @@ mod tests { config: HashMap::from([("region".to_string(), "us-west".to_string())]), ..Provider::default() }), - workspace: String::new(), + workspace_scope: Some(workspace_selector("default")), }; let encoded = request.encode_to_vec(); @@ -368,7 +369,6 @@ mod tests { let codec = ProtoJsonCodec::openshell().unwrap(); for (message_name, field_name) in [ ("openshell.datamodel.v1.Provider", "credentials"), - ("openshell.inference.v1.ResolvedRoute", "api_key"), ("openshell.compute.v1.DriverSandboxSpec", "sandbox_token"), ("openshell.v1.IssueSandboxTokenResponse", "token"), ("openshell.v1.RefreshSandboxTokenResponse", "token"), @@ -377,10 +377,6 @@ mod tests { ("openshell.v1.RevokeSshSessionRequest", "token"), ("openshell.v1.TcpForwardInit", "authorization_token"), ("openshell.v1.SshSession", "token"), - ( - "openshell.v1.StoredProviderCredentialRefreshState", - "material", - ), ("openshell.v1.ConfigureProviderRefreshRequest", "material"), ( "openshell.v1.GetSandboxProviderEnvironmentResponse", @@ -404,6 +400,7 @@ mod tests { environment: HashMap::from([("FEATURE_FLAG".to_string(), "on".to_string())]), ..SandboxSpec::default() }), + workspace_scope: Some(workspace_selector("default")), ..CreateSandboxRequest::default() }; @@ -458,6 +455,7 @@ mod tests { "openshell.nvidia.com/policy-signature".to_string(), "signed".to_string(), )]), + workspace_scope: Some(workspace_selector("default")), ..Default::default() }; let bytes = request.encode_to_vec(); diff --git a/crates/openshell-gateway-interceptors/src/runtime.rs b/crates/openshell-gateway-interceptors/src/runtime.rs index 02afddab7..0bd1ba2b4 100644 --- a/crates/openshell-gateway-interceptors/src/runtime.rs +++ b/crates/openshell-gateway-interceptors/src/runtime.rs @@ -764,7 +764,7 @@ mod tests { config: HashMap::from([("region".to_string(), "old".to_string())]), ..Provider::default() }), - workspace: String::new(), + workspace_scope: Some(openshell_core::proto::workspace_selector("default")), }; let json = codec .decode_message_to_json("openshell.v1.CreateProviderRequest", &request) @@ -1017,6 +1017,7 @@ mod tests { ("policy-signature-kid".to_string(), "kid".to_string()), ("correlation-id".to_string(), "reload-1".to_string()), ]), + workspace_scope: Some(openshell_core::proto::workspace_selector("default")), ..UpdateConfigRequest::default() }; let body = GrpcFrame { @@ -1074,8 +1075,9 @@ mod tests { name: "demo".to_string(), labels: HashMap::new(), annotations: HashMap::new(), - workspace: String::new(), + workspace_scope: Some(openshell_core::proto::workspace_selector("default")), await_main_process_attachment: false, + workload_template_name: String::new(), }; let bytes = request.encode_to_vec(); diff --git a/crates/openshell-gateway/Cargo.toml b/crates/openshell-gateway/Cargo.toml new file mode 100644 index 000000000..f9d02027e --- /dev/null +++ b/crates/openshell-gateway/Cargo.toml @@ -0,0 +1,72 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +[package] +name = "openshell-gateway" +description = "OpenShell gateway binary composition" +version.workspace = true +edition.workspace = true +rust-version.workspace = true +license.workspace = true +repository.workspace = true + +[[bin]] +name = "openshell-gateway" +path = "src/main.rs" + +[dependencies] +openshell-core = { path = "../openshell-core", default-features = false } +openshell-server = { path = "../openshell-server", default-features = false } +openshell-otel = { path = "../openshell-otel", optional = true } +async-trait = "0.1" +miette = { workspace = true } +tokio = { workspace = true } + +[target.'cfg(not(target_os = "windows"))'.dependencies] +openshell-driver-docker = { path = "../openshell-driver-docker", optional = true } +openshell-driver-kubernetes = { path = "../openshell-driver-kubernetes", optional = true } +openshell-driver-podman = { path = "../openshell-driver-podman", optional = true } +hyper-util = { workspace = true, optional = true } +nix = { workspace = true, optional = true } +serde = { workspace = true, optional = true } +rustix = { workspace = true, optional = true } +tonic = { workspace = true, optional = true } +tower = { workspace = true, optional = true } +tracing = { workspace = true, optional = true } + +[target.'cfg(target_os = "windows")'.dependencies] +openshell-driver-mxc = { path = "../openshell-driver-mxc", optional = true } + +[features] +default = ["telemetry", "in-tree-compute-drivers"] +in-tree-compute-drivers = [ + "dep:openshell-driver-docker", + "dep:openshell-driver-kubernetes", + "dep:openshell-driver-podman", + "dep:openshell-otel", + "dep:hyper-util", + "dep:nix", + "dep:serde", + "dep:rustix", + "dep:tonic", + "dep:tower", + "dep:tracing", + "dep:openshell-driver-mxc", +] +telemetry = ["openshell-core/telemetry", "openshell-server/telemetry"] +## Convenience alias: every default feature except `telemetry`. Build a +## telemetry-free gateway with +## `--no-default-features --features defaults-without-telemetry` and stay +## correct as new default features are added. Cargo cannot subtract a single +## default feature, so this alias must be paired with `--no-default-features`; +## enabling it alongside `telemetry` is a compile error rather than a silent +## telemetry-on build. Kept in sync with `default` by +## `rust:verify:defaults-without-telemetry`. +defaults-without-telemetry = ["in-tree-compute-drivers"] +bundled-z3 = ["openshell-server/bundled-z3"] + +[lints] +workspace = true + +[dev-dependencies] +tempfile = "3" diff --git a/crates/openshell-gateway/src/lib.rs b/crates/openshell-gateway/src/lib.rs new file mode 100644 index 000000000..54ae5e5de --- /dev/null +++ b/crates/openshell-gateway/src/lib.rs @@ -0,0 +1,374 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! Standard gateway binary composition. +//! +//! The server remains backend-agnostic. This crate is the composition boundary +//! that links first-party compute drivers into the distributed gateway binary. + +// `defaults-without-telemetry` is an alias for the default feature set minus +// `telemetry`, not a switch that turns telemetry off. Cargo cannot subtract a +// default feature, so adding it on top of the defaults would otherwise produce +// a telemetry-on build that reads as telemetry-free. Fail the build instead. +#[cfg(all(feature = "telemetry", feature = "defaults-without-telemetry"))] +compile_error!( + "features `telemetry` and `defaults-without-telemetry` are mutually exclusive; \ + build a telemetry-free gateway with `--no-default-features --features defaults-without-telemetry`" +); + +#[cfg(all(not(target_os = "windows"), feature = "in-tree-compute-drivers"))] +mod vm; + +#[cfg(feature = "in-tree-compute-drivers")] +use openshell_core::telemetry::TelemetryComputeDriver; +#[cfg(feature = "in-tree-compute-drivers")] +use openshell_server::ComputeDriverRegistration; +use openshell_server::ComputeDriverRegistry; + +/// Install every first-party compute driver linked into the standard gateway. +#[must_use] +pub fn install_default_compute_drivers() -> ComputeDriverRegistry { + #[allow(unused_mut)] + let mut registry = ComputeDriverRegistry::new(); + #[cfg(all(not(target_os = "windows"), feature = "in-tree-compute-drivers"))] + install_in_tree_compute_drivers(&mut registry); + #[cfg(all(target_os = "windows", feature = "in-tree-compute-drivers"))] + install_mxc_compute_driver(&mut registry); + registry +} + +#[cfg(all(target_os = "windows", feature = "in-tree-compute-drivers"))] +fn install_mxc_compute_driver(registry: &mut ComputeDriverRegistry) { + let registration = ComputeDriverRegistration::new("mxc", u16::MAX, None, MxcFactory) + .expect("first-party driver name is valid") + .with_telemetry_category(TelemetryComputeDriver::anonymous_category("mxc")) + .with_local_singleplayer(); + registry + .install(registration) + .expect("first-party driver names are unique"); + + for name in ["docker", "kubernetes", "podman", "vm"] { + let registration = ComputeDriverRegistration::new( + name, + u16::MAX, + None, + UnsupportedWindowsFactory { name }, + ) + .expect("first-party driver name is valid"); + registry + .install(registration) + .expect("first-party driver names are unique"); + } +} + +#[cfg(all(target_os = "windows", feature = "in-tree-compute-drivers"))] +#[derive(Clone, Copy)] +struct UnsupportedWindowsFactory { + name: &'static str, +} + +#[cfg(all(target_os = "windows", feature = "in-tree-compute-drivers"))] +#[async_trait::async_trait] +impl openshell_server::ComputeDriverFactory for UnsupportedWindowsFactory { + async fn build( + &self, + _context: openshell_server::ComputeDriverBuildContext<'_>, + ) -> openshell_core::Result { + Err(unsupported_windows_compute_driver(self.name)) + } +} + +#[cfg(all(target_os = "windows", feature = "in-tree-compute-drivers"))] +fn unsupported_windows_compute_driver(name: &str) -> openshell_core::Error { + openshell_core::Error::config(format!("compute driver '{name}' is unsupported on Windows")) +} + +#[cfg(all(target_os = "windows", feature = "in-tree-compute-drivers"))] +#[derive(Clone, Copy)] +struct MxcFactory; + +#[cfg(all(target_os = "windows", feature = "in-tree-compute-drivers"))] +#[async_trait::async_trait] +impl openshell_server::ComputeDriverFactory for MxcFactory { + async fn build( + &self, + context: openshell_server::ComputeDriverBuildContext<'_>, + ) -> openshell_core::Result { + let config: openshell_driver_mxc::MxcComputeConfig = context.driver_config()?; + let backend = openshell_driver_mxc::MxcComputeBackend::new(config); + let driver = openshell_driver_mxc::ComputeDriverService::new(backend); + Ok(openshell_server::ComputeDriverInstance::InProcess( + std::sync::Arc::new(driver), + )) + } +} + +#[cfg(all(not(target_os = "windows"), feature = "in-tree-compute-drivers"))] +fn install_in_tree_compute_drivers(registry: &mut ComputeDriverRegistry) { + for registration in [ + ComputeDriverRegistration::new( + "kubernetes", + 100, + Some(|| std::env::var_os("KUBERNETES_SERVICE_HOST").is_some()), + KubernetesFactory, + ) + .map(|registration| { + registration + .with_telemetry_category(TelemetryComputeDriver::anonymous_category("kubernetes")) + .without_mtls_user_auth() + .with_in_process_tracing(openshell_driver_kubernetes::otel_tracing::TRACING) + .with_inherited_config_keys(&[ + "namespace", + "default_image", + "supervisor_image", + "client_tls_secret_name", + "service_account_name", + "host_gateway_ip", + "enable_user_namespaces", + "sa_token_ttl_secs", + ]) + }), + ComputeDriverRegistration::new( + "podman", + 200, + Some(openshell_driver_podman::driver::is_available), + PodmanFactory, + ) + .map(|registration| { + registration + .with_telemetry_category(TelemetryComputeDriver::anonymous_category("podman")) + .with_local_singleplayer() + .with_in_process_tracing(openshell_driver_podman::otel_tracing::TRACING) + .with_inherited_config_keys(&[ + "default_image", + "supervisor_image", + "host_gateway_ip", + "guest_tls_ca", + "guest_tls_cert", + "guest_tls_key", + ]) + }), + ComputeDriverRegistration::new( + "docker", + 300, + Some(openshell_driver_docker::is_available), + DockerFactory, + ) + .map(|registration| { + registration + .with_telemetry_category(TelemetryComputeDriver::anonymous_category("docker")) + .with_local_singleplayer() + .with_in_process_tracing(openshell_driver_docker::otel_tracing::TRACING) + .with_inherited_config_keys(&[ + "sandbox_namespace", + "default_image", + "supervisor_image", + "host_gateway_ip", + "guest_tls_ca", + "guest_tls_cert", + "guest_tls_key", + ]) + }), + ComputeDriverRegistration::new("vm", u16::MAX, None, VmFactory).map(|registration| { + registration + .with_telemetry_category(TelemetryComputeDriver::anonymous_category("vm")) + .with_local_singleplayer() + .with_inherited_config_keys(&[ + "default_image", + "guest_tls_ca", + "guest_tls_cert", + "guest_tls_key", + ]) + }), + ] { + registry + .install(registration.expect("first-party driver name is valid")) + .expect("first-party driver names are unique"); + } +} + +#[cfg(all(not(target_os = "windows"), feature = "in-tree-compute-drivers"))] +#[derive(Clone, Copy)] +struct KubernetesFactory; + +#[cfg(all(not(target_os = "windows"), feature = "in-tree-compute-drivers"))] +#[async_trait::async_trait] +impl openshell_server::ComputeDriverFactory for KubernetesFactory { + async fn build( + &self, + context: openshell_server::ComputeDriverBuildContext<'_>, + ) -> openshell_core::Result { + let mut config: openshell_driver_kubernetes::KubernetesComputeConfig = + context.driver_config()?; + if let Ok(size) = std::env::var("OPENSHELL_K8S_WORKSPACE_DEFAULT_STORAGE_SIZE") { + config.workspace_default_storage_size = size; + } + if let Ok(storage_class) = std::env::var("OPENSHELL_K8S_WORKSPACE_STORAGE_CLASS") { + config.workspace_storage_class = storage_class; + } + let driver = openshell_driver_kubernetes::KubernetesComputeDriver::new( + config, + context.shutdown_receiver(), + ) + .await + .map_err(|error| openshell_core::Error::execution(error.to_string()))?; + let driver = openshell_driver_kubernetes::ComputeDriverService::new_in_process(driver); + Ok(openshell_server::ComputeDriverInstance::InProcess( + std::sync::Arc::new(driver), + )) + } +} + +#[cfg(all(not(target_os = "windows"), feature = "in-tree-compute-drivers"))] +#[derive(Clone, Copy)] +struct DockerFactory; + +#[cfg(all(not(target_os = "windows"), feature = "in-tree-compute-drivers"))] +#[async_trait::async_trait] +impl openshell_server::ComputeDriverFactory for DockerFactory { + async fn build( + &self, + context: openshell_server::ComputeDriverBuildContext<'_>, + ) -> openshell_core::Result { + let mut config: openshell_driver_docker::DockerComputeConfig = context.driver_config()?; + apply_guest_tls( + &mut config.guest_tls_ca, + &mut config.guest_tls_cert, + &mut config.guest_tls_key, + context.guest_tls_paths(), + ); + let driver = openshell_driver_docker::DockerComputeDriver::new( + context.gateway_bind_address(), + context.gateway_log_level(), + &config, + ) + .await + .map_err(|error| openshell_core::Error::execution(error.to_string()))?; + let driver = openshell_driver_docker::ComputeDriverService::new_in_process(driver); + Ok(openshell_server::ComputeDriverInstance::InProcess( + std::sync::Arc::new(driver), + )) + } +} + +#[cfg(all(not(target_os = "windows"), feature = "in-tree-compute-drivers"))] +#[derive(Clone, Copy)] +struct PodmanFactory; + +#[cfg(all(not(target_os = "windows"), feature = "in-tree-compute-drivers"))] +#[async_trait::async_trait] +impl openshell_server::ComputeDriverFactory for PodmanFactory { + async fn build( + &self, + context: openshell_server::ComputeDriverBuildContext<'_>, + ) -> openshell_core::Result { + let mut config: openshell_driver_podman::PodmanComputeConfig = context.driver_config()?; + config.gateway_port = context.gateway_port(); + if let Ok(path) = std::env::var("OPENSHELL_PODMAN_SOCKET") { + config.socket_path = Some(path.into()); + } + if let Ok(ip) = std::env::var("OPENSHELL_PODMAN_HOST_GATEWAY_IP") { + config.host_gateway_ip = ip; + } + if let Ok(mode) = std::env::var("OPENSHELL_PODMAN_USERNS") { + config.userns = Some(mode); + } + apply_guest_tls( + &mut config.guest_tls_ca, + &mut config.guest_tls_cert, + &mut config.guest_tls_key, + context.guest_tls_paths(), + ); + let driver = openshell_driver_podman::PodmanComputeDriver::new(config) + .await + .map_err(|error| openshell_core::Error::execution(error.to_string()))?; + let driver = openshell_driver_podman::ComputeDriverService::new_in_process(driver); + Ok(openshell_server::ComputeDriverInstance::InProcess( + std::sync::Arc::new(driver), + )) + } +} + +#[cfg(all(not(target_os = "windows"), feature = "in-tree-compute-drivers"))] +#[derive(Clone, Copy)] +struct VmFactory; + +#[cfg(all(not(target_os = "windows"), feature = "in-tree-compute-drivers"))] +#[async_trait::async_trait] +impl openshell_server::ComputeDriverFactory for VmFactory { + async fn build( + &self, + context: openshell_server::ComputeDriverBuildContext<'_>, + ) -> openshell_core::Result { + let mut config: vm::VmComputeConfig = context.driver_config()?; + if config.state_dir.as_os_str().is_empty() { + config.state_dir = vm::VmComputeConfig::default_state_dir(); + } + if config.grpc_endpoint.trim().is_empty() + && (!context.gateway_tls_enabled() || context.guest_tls_paths().is_some()) + { + let scheme = if context.gateway_tls_enabled() { + "https" + } else { + "http" + }; + config.grpc_endpoint = format!("{scheme}://127.0.0.1:{}", context.gateway_port()); + } + apply_guest_tls( + &mut config.guest_tls_ca, + &mut config.guest_tls_cert, + &mut config.guest_tls_key, + context.guest_tls_paths(), + ); + let endpoint = vm::spawn( + context.gateway_log_level(), + context.gateway_name(), + &config, + context.otlp_config(), + ) + .await?; + Ok(openshell_server::ComputeDriverInstance::ManagedRemote( + endpoint, + )) + } +} + +#[cfg(all(not(target_os = "windows"), feature = "in-tree-compute-drivers"))] +fn apply_guest_tls( + ca: &mut Option, + cert: &mut Option, + key: &mut Option, + defaults: Option<(&std::path::Path, &std::path::Path, &std::path::Path)>, +) { + if ca.is_none() + && cert.is_none() + && key.is_none() + && let Some((default_ca, default_cert, default_key)) = defaults + { + *ca = Some(default_ca.to_owned()); + *cert = Some(default_cert.to_owned()); + *key = Some(default_key.to_owned()); + } +} + +#[cfg(all(test, target_os = "windows", feature = "in-tree-compute-drivers"))] +mod windows_tests { + use super::*; + + #[test] + fn windows_builtin_compute_drivers_report_unsupported() { + let registry = install_default_compute_drivers(); + assert_eq!( + registry.installed_driver_names().collect::>(), + ["docker", "kubernetes", "mxc", "podman", "vm"] + ); + + for name in ["docker", "kubernetes", "podman", "vm"] { + let message = unsupported_windows_compute_driver(name).to_string(); + assert!( + message.contains("unsupported on Windows"), + "{name} rejection should be explicit, got: {message}" + ); + } + } +} diff --git a/crates/openshell-server/src/main.rs b/crates/openshell-gateway/src/main.rs similarity index 59% rename from crates/openshell-server/src/main.rs rename to crates/openshell-gateway/src/main.rs index c76761016..85d061186 100644 --- a/crates/openshell-server/src/main.rs +++ b/crates/openshell-gateway/src/main.rs @@ -1,14 +1,10 @@ // SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 -//! `OpenShell` Gateway binary entrypoint. - -use miette::Result; - #[tokio::main] -async fn main() -> Result<()> { +async fn main() -> miette::Result<()> { openshell_server::cli::run_cli_with_compute_drivers( - openshell_server::install_default_compute_drivers(), + openshell_gateway::install_default_compute_drivers(), ) .await } diff --git a/crates/openshell-server/src/compute/vm.rs b/crates/openshell-gateway/src/vm.rs similarity index 79% rename from crates/openshell-server/src/compute/vm.rs rename to crates/openshell-gateway/src/vm.rs index 6a66fc8aa..f52bd9ddf 100644 --- a/crates/openshell-server/src/compute/vm.rs +++ b/crates/openshell-gateway/src/vm.rs @@ -20,28 +20,24 @@ //! [`openshell_core::Config`] so the shared core stays free of driver-specific //! plumbing. //! -//! TODO(driver-abstraction): this module still assumes the concrete VM driver -//! (argv shape, guest-TLS flags, libkrun-specific settings). Once we land the -//! generalized compute-driver interface, the CLI-arg plumbing below should -//! be replaced with a driver-agnostic launcher that speaks gRPC to -//! configure the driver — and this file should collapse to the types that -//! are genuinely VM-specific (libkrun log level, vCPU / memory shape) plus a -//! trait implementation registering the VM driver against the generic -//! interface. +//! Process launch remains deliberately VM-specific at this binary composition +//! boundary: it translates gateway configuration into the standalone driver's +//! argv and then connects through the same public compute-driver RPC interface +//! used by operator-managed external drivers. -use super::AcquiredRemoteDriverEndpoint; -#[cfg(unix)] -use super::ManagedDriverProcess; -use crate::config_file::OtlpConfig; -#[cfg(unix)] -use crate::otel_tracing::TraceContextInterceptor; #[cfg(unix)] use hyper_util::rt::TokioIo; #[cfg(unix)] use openshell_core::proto::compute::v1::{ GetCapabilitiesRequest, compute_driver_client::ComputeDriverClient, }; -use openshell_core::{ComputeDriverKind, Config, Error, Result}; +use openshell_core::{Error, Result}; +#[cfg(unix)] +use openshell_otel::TraceContextInterceptor; +use openshell_server::AcquiredRemoteDriverEndpoint; +#[cfg(unix)] +use openshell_server::ManagedDriverProcess; +use openshell_server::config_file::OtlpConfig; #[cfg(unix)] use std::os::unix::fs::{FileTypeExt, MetadataExt, PermissionsExt}; #[cfg(unix)] @@ -103,6 +99,34 @@ pub struct VmComputeConfig { /// Host-side private key for the guest's mTLS client bundle. pub guest_tls_key: Option, + + /// Corporate forward proxy URL (`http://host:port` or `https://host:port`) + /// for policy-approved TLS egress from VM sandboxes. + /// + /// Deployment-level configuration, not a per-sandbox setting: it is passed + /// to the driver, which puts it on the guest supervisor's argv. A proxy on + /// this host's loopback is reachable from a guest only through the gvproxy + /// host alias `host.openshell.internal`. + pub https_proxy: Option, + + /// Comma-separated `NO_PROXY` list. Bypasses only the corporate proxy, + /// never `OpenShell` policy evaluation. + pub no_proxy: Option, + + /// Path on this host to a `user:pass` corporate proxy credential file. + pub proxy_auth_file: Option, + + /// Acknowledgement that Basic auth to an `http://` proxy is cleartext. + /// Required alongside `proxy_auth_file` unless the proxy is `https://`. + pub proxy_auth_allow_insecure: Option, + + /// Send hostnames rather than validated IPs in CONNECT requests. Last + /// resort for proxies whose ACLs reject IP CONNECT targets. + pub proxy_connect_by_hostname: Option, + + /// Path on this host to a PEM CA bundle trusted for the corporate proxy + /// and for server certificates a TLS-intercepting proxy re-signs. + pub proxy_ca_bundle: Option, } impl VmComputeConfig { @@ -139,6 +163,29 @@ impl VmComputeConfig { 4096 } + /// Validate the corporate upstream-proxy settings, fail-closed. + /// + /// Runs in the gateway as well as in the driver so an invalid + /// `[openshell.drivers.vm]` table reports the offending key instead of + /// surfacing as an opaque driver-startup timeout. + /// + /// # Errors + /// + /// Returns a [`Error::config`] naming the offending key. + pub fn validate_proxy_config(&self) -> Result<()> { + openshell_core::driver_utils::validate_upstream_proxy_settings( + &openshell_core::driver_utils::UpstreamProxySettings { + url: self.https_proxy.as_deref(), + no_proxy: self.no_proxy.as_deref(), + auth_file: self.proxy_auth_file.as_deref(), + auth_allow_insecure: self.proxy_auth_allow_insecure, + connect_by_hostname: self.proxy_connect_by_hostname, + ca_bundle: self.proxy_ca_bundle.as_deref(), + }, + ) + .map_err(Error::config) + } + #[must_use] fn default_driver_search_dirs(home: Option) -> Vec { let mut dirs = Vec::new(); @@ -167,6 +214,12 @@ impl Default for VmComputeConfig { guest_tls_ca: None, guest_tls_cert: None, guest_tls_key: None, + https_proxy: None, + no_proxy: None, + proxy_auth_file: None, + proxy_auth_allow_insecure: None, + proxy_connect_by_hostname: None, + proxy_ca_bundle: None, } } } @@ -453,7 +506,8 @@ pub fn compute_driver_guest_tls_paths( /// kills the subprocess and removes the socket on drop. #[cfg(unix)] pub async fn spawn( - config: &Config, + gateway_log_level: &str, + gateway_name: &str, vm_config: &VmComputeConfig, otlp_config: Option<&OtlpConfig>, ) -> Result { @@ -463,6 +517,8 @@ pub async fn spawn( )); } + vm_config.validate_proxy_config()?; + let driver_bin = resolve_compute_driver_bin(vm_config)?; let socket_path = compute_driver_socket_path(vm_config); let guest_tls_paths = compute_driver_guest_tls_paths(vm_config)?; @@ -477,8 +533,8 @@ pub async fn spawn( command .arg("--expected-peer-pid") .arg(std::process::id().to_string()); - command.arg("--log-level").arg(&config.log_level); - append_otlp_args(&mut command, otlp_config, &config.name); + command.arg("--log-level").arg(gateway_log_level); + append_otlp_args(&mut command, otlp_config, gateway_name); command .arg("--openshell-endpoint") .arg(&vm_config.grpc_endpoint); @@ -504,6 +560,7 @@ pub async fn spawn( command.arg("--guest-tls-cert").arg(tls.cert); command.arg("--guest-tls-key").arg(tls.key); } + append_upstream_proxy_args(&mut command, vm_config); let mut child = command.spawn().map_err(|e| { Error::execution(format!( @@ -513,13 +570,43 @@ pub async fn spawn( })?; let channel = wait_for_compute_driver(&socket_path, &mut child).await?; let process = Arc::new(ManagedDriverProcess::new(child, socket_path)); - Ok(AcquiredRemoteDriverEndpoint::managed_builtin( - ComputeDriverKind::Vm, - channel, - process, + Ok(AcquiredRemoteDriverEndpoint::managed( + "vm", channel, process, )) } +/// Forward the operator's corporate proxy settings to the driver subprocess. +/// +/// Only keys the operator actually set are passed, so the driver keeps the +/// same "omitted means no proxy" contract the supervisor enforces. The +/// booleans travel as explicit values rather than presence flags so an +/// explicit `false` still trips the driver's pairing checks. +#[cfg(unix)] +fn append_upstream_proxy_args(command: &mut Command, vm_config: &VmComputeConfig) { + if let Some(url) = &vm_config.https_proxy { + command.arg("--https-proxy").arg(url); + } + if let Some(list) = &vm_config.no_proxy { + command.arg("--no-proxy").arg(list); + } + if let Some(path) = &vm_config.proxy_auth_file { + command.arg("--proxy-auth-file").arg(path); + } + if let Some(allow) = vm_config.proxy_auth_allow_insecure { + command + .arg("--proxy-auth-allow-insecure") + .arg(allow.to_string()); + } + if let Some(by_hostname) = vm_config.proxy_connect_by_hostname { + command + .arg("--proxy-connect-by-hostname") + .arg(by_hostname.to_string()); + } + if let Some(path) = &vm_config.proxy_ca_bundle { + command.arg("--proxy-ca-bundle").arg(path); + } +} + #[cfg(unix)] fn append_otlp_args(command: &mut Command, otlp_config: Option<&OtlpConfig>, gateway_name: &str) { if let Some(config) = otlp_config { @@ -530,7 +617,8 @@ fn append_otlp_args(command: &mut Command, otlp_config: Option<&OtlpConfig>, gat #[cfg(not(unix))] pub async fn spawn( - _config: &Config, + _gateway_log_level: &str, + _gateway_name: &str, _vm_config: &VmComputeConfig, _otlp_config: Option<&OtlpConfig>, ) -> Result { @@ -610,12 +698,12 @@ async fn connect_compute_driver(socket_path: &Path) -> Result { #[cfg(all(test, unix))] mod tests { use super::{ - VmComputeConfig, append_otlp_args, compute_driver_guest_tls_paths, - compute_driver_socket_path, current_euid, prepare_compute_driver_socket_path, - prepare_vm_state_dir, resolve_compute_driver_bin, resolve_driver_search_dirs, - wait_for_compute_driver, + VmComputeConfig, append_otlp_args, append_upstream_proxy_args, + compute_driver_guest_tls_paths, compute_driver_socket_path, current_euid, + prepare_compute_driver_socket_path, prepare_vm_state_dir, resolve_compute_driver_bin, + resolve_driver_search_dirs, }; - use crate::config_file::OtlpConfig; + use openshell_server::config_file::OtlpConfig; use std::os::unix::fs::PermissionsExt; use std::os::unix::net::UnixListener as StdUnixListener; use std::path::PathBuf; @@ -649,43 +737,83 @@ mod tests { ); } - #[tokio::test] - async fn readiness_probe_propagates_the_active_trace() { - use crate::otel_tracing::test_exporter; - use crate::test_support::FakeComputeDriver; + #[test] + fn vm_driver_command_forwards_corporate_proxy_settings() { + let mut command = tokio::process::Command::new("openshell-driver-vm"); + append_upstream_proxy_args( + &mut command, + &VmComputeConfig { + https_proxy: Some("http://proxy.corp.com:8080".to_string()), + no_proxy: Some("10.0.0.0/8".to_string()), + proxy_auth_file: Some("/etc/openshell/secrets/proxy-auth".to_string()), + proxy_auth_allow_insecure: Some(true), + proxy_connect_by_hostname: Some(false), + proxy_ca_bundle: Some("/etc/openshell/tls/proxy-ca.pem".to_string()), + ..VmComputeConfig::default() + }, + ); - let dir = tempdir().unwrap(); - let socket_path = dir.path().join("compute-driver.sock"); - let driver = FakeComputeDriver::new(); - let _server = driver.serve_uds(&socket_path).unwrap(); - let mut child = tokio::process::Command::new("sh") - .arg("-c") - .arg("read _") - .stdin(std::process::Stdio::piped()) - .kill_on_drop(true) - .spawn() - .unwrap(); - - let traced = test_exporter::install_traced(); - wait_for_compute_driver(&socket_path, &mut child) - .await - .unwrap(); - - let readiness = traced.spans_named("driver.wait_for_ready"); - assert_eq!(readiness.len(), 1, "one readiness operation should finish"); - test_exporter::assert_is_root(&readiness[0]); - let trace_id = readiness[0].span_context.trace_id().to_string(); + let args = command + .as_std() + .get_args() + .map(|arg| arg.to_string_lossy().into_owned()) + .collect::>(); assert_eq!( - driver.traceparents().len(), - 1, - "the readiness capability probe should carry trace context" - ); - assert!( - driver.traceparents()[0].contains(&trace_id), - "the readiness probe should be part of the active trace" + args, + [ + "--https-proxy", + "http://proxy.corp.com:8080", + "--no-proxy", + "10.0.0.0/8", + "--proxy-auth-file", + "/etc/openshell/secrets/proxy-auth", + "--proxy-auth-allow-insecure", + "true", + // Passed as an explicit value, not a presence flag, so the + // driver still sees the operator's `false`. + "--proxy-connect-by-hostname", + "false", + "--proxy-ca-bundle", + "/etc/openshell/tls/proxy-ca.pem", + ] ); } + #[test] + fn vm_driver_command_omits_unset_corporate_proxy_settings() { + let mut command = tokio::process::Command::new("openshell-driver-vm"); + append_upstream_proxy_args(&mut command, &VmComputeConfig::default()); + assert_eq!(command.as_std().get_args().count(), 0); + } + + #[test] + fn invalid_corporate_proxy_config_is_rejected_before_the_driver_starts() { + // Without this the operator would see an opaque driver-readiness + // timeout instead of an error naming the offending key. + let err = VmComputeConfig { + https_proxy: Some("socks5://proxy.corp.com:1080".to_string()), + ..VmComputeConfig::default() + } + .validate_proxy_config() + .expect_err("only http:// and https:// proxies are supported"); + assert!(err.to_string().contains("https_proxy"), "{err}"); + + let err = VmComputeConfig { + proxy_ca_bundle: Some("/etc/openshell/tls/proxy-ca.pem".to_string()), + ..VmComputeConfig::default() + } + .validate_proxy_config() + .expect_err("a CA bundle without a proxy URL would hide a fail-open state"); + assert!(err.to_string().contains("proxy_ca_bundle"), "{err}"); + + VmComputeConfig { + https_proxy: Some("http://proxy.corp.com:8080".to_string()), + ..VmComputeConfig::default() + } + .validate_proxy_config() + .expect("a lone proxy URL is a complete configuration"); + } + #[test] fn resolve_driver_bin_uses_driver_dir_when_binary_present() { let dir = tempdir().unwrap(); diff --git a/crates/openshell-ocsf/Cargo.toml b/crates/openshell-ocsf/Cargo.toml index be91b1547..69c7b7e0a 100644 --- a/crates/openshell-ocsf/Cargo.toml +++ b/crates/openshell-ocsf/Cargo.toml @@ -16,6 +16,7 @@ serde = { workspace = true } serde_json = { workspace = true } serde_repr = "0.1" tracing = { workspace = true } +uuid = { workspace = true } tracing-subscriber = { workspace = true } [dev-dependencies] diff --git a/crates/openshell-ocsf/src/builders/api_activity.rs b/crates/openshell-ocsf/src/builders/api_activity.rs index 03c857f08..f3cddbad8 100644 --- a/crates/openshell-ocsf/src/builders/api_activity.rs +++ b/crates/openshell-ocsf/src/builders/api_activity.rs @@ -159,9 +159,9 @@ mod tests { .ai_model(AiModel::new("claude-3-haiku", "anthropic")) .http_request(HttpRequest::new( "POST", - Url::new("https", "inference.local", "/v1/messages", 443), + Url::new("https", "api.anthropic.com", "/v1/messages", 443), )) - .dst_endpoint(Endpoint::from_domain("inference.local", 443)) + .dst_endpoint(Endpoint::from_domain("api.anthropic.com", 443)) .unmapped("latency_ms", 701_u64) .unmapped("input_tokens", 12_u64) .message("Model call: claude-3-haiku via anthropic") diff --git a/crates/openshell-ocsf/src/builders/mod.rs b/crates/openshell-ocsf/src/builders/mod.rs index e63b2be88..f51579c60 100644 --- a/crates/openshell-ocsf/src/builders/mod.rs +++ b/crates/openshell-ocsf/src/builders/mod.rs @@ -205,21 +205,24 @@ impl SandboxContext { version: OCSF_VERSION.to_string(), product: Product::openshell_sandbox(&self.product_version), profiles: profiles.iter().map(|s| (*s).to_string()).collect(), - uid: Some(self.sandbox_id.clone()), + uid: Some(uuid::Uuid::new_v4().to_string()), log_source: None, } } - /// Build the OCSF `Container` object. + /// Build the OCSF `Container` object when the event concerns a sandbox. #[must_use] - pub fn container(&self) -> Container { - Container { + pub fn container(&self) -> Option { + if self.sandbox_id.is_empty() { + return None; + } + Some(Container { name: self.sandbox_name.clone(), uid: Some(self.sandbox_id.clone()), - image: Some(Image { + image: (!self.container_image.is_empty()).then(|| Image { name: self.container_image.clone(), }), - } + }) } /// Build the OCSF `Device` object. @@ -249,7 +252,9 @@ impl SandboxContext { base.set_message(m); } base.set_device(self.device()); - base.set_container(self.container()); + if let Some(container) = self.container() { + base.set_container(container); + } } } @@ -277,13 +282,27 @@ mod tests { assert_eq!(meta.version, "1.8.0"); assert_eq!(meta.product.name, "OpenShell Sandbox Supervisor"); assert_eq!(meta.profiles.len(), 2); - assert_eq!(meta.uid.as_deref(), Some("sandbox-abc123")); + let uid = meta.uid.as_deref().expect("uid is set"); + assert!(!uid.is_empty()); + assert_ne!(uid, "sandbox-abc123"); + } + + #[test] + fn test_emitted_device_matches_vendored_schema() { + use crate::validation::schema::{ + load_object_schema, validate_enum_value, validate_required_fields, + }; + + let device = serde_json::to_value(test_sandbox_context().device()).unwrap(); + let schema = load_object_schema("device"); + validate_required_fields(&device, &schema); + validate_enum_value(&device, "type_id", &schema); } #[test] fn test_sandbox_context_container() { let ctx = test_sandbox_context(); - let container = ctx.container(); + let container = ctx.container().expect("sandbox context has a container"); assert_eq!(container.name, "my-sandbox"); assert_eq!(container.uid.as_deref(), Some("sandbox-abc123")); } diff --git a/crates/openshell-ocsf/src/enums/device_type.rs b/crates/openshell-ocsf/src/enums/device_type.rs new file mode 100644 index 000000000..b47596e9d --- /dev/null +++ b/crates/openshell-ocsf/src/enums/device_type.rs @@ -0,0 +1,62 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! OCSF `device.type_id` enum. + +use serde_repr::{Deserialize_repr, Serialize_repr}; + +/// OCSF Device Type ID. +/// +/// Only the values `OpenShell` can produce are modelled; the schema defines a +/// wider set (desktop, mobile, firewall, router, ...). +/// Values come from the OCSF device `type_id` enumeration. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Serialize_repr, Deserialize_repr)] +#[repr(u8)] +pub enum DeviceTypeId { + /// 0 — Unknown + Unknown = 0, + /// 99 — Other + Other = 99, +} + +impl DeviceTypeId { + #[must_use] + pub fn as_u8(self) -> u8 { + self as u8 + } +} + +impl std::fmt::Display for DeviceTypeId { + fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + formatter.write_str(match self { + Self::Unknown => "Unknown", + Self::Other => "Other", + }) + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn device_type_display_uses_schema_labels() { + for (device_type, expected) in [ + (DeviceTypeId::Unknown, "Unknown"), + (DeviceTypeId::Other, "Other"), + ] { + assert_eq!(device_type.to_string(), expected); + assert_eq!(format!("{device_type}"), expected); + } + } + + #[test] + fn device_type_json_roundtrip() { + for (device_type, expected) in [(DeviceTypeId::Unknown, 0), (DeviceTypeId::Other, 99)] { + let json = serde_json::to_value(device_type).unwrap(); + assert_eq!(json, serde_json::json!(expected)); + let decoded: DeviceTypeId = serde_json::from_value(json).unwrap(); + assert_eq!(decoded, device_type); + } + } +} diff --git a/crates/openshell-ocsf/src/enums/mod.rs b/crates/openshell-ocsf/src/enums/mod.rs index 96567f86d..b9ae43229 100644 --- a/crates/openshell-ocsf/src/enums/mod.rs +++ b/crates/openshell-ocsf/src/enums/mod.rs @@ -6,6 +6,7 @@ mod action; mod activity; mod auth; +mod device_type; mod disposition; mod http_method; mod launch; @@ -16,6 +17,7 @@ mod status; pub use action::ActionId; pub use activity::ActivityId; pub use auth::AuthTypeId; +pub use device_type::DeviceTypeId; pub use disposition::DispositionId; pub use http_method::HttpMethod; pub use launch::LaunchTypeId; diff --git a/crates/openshell-ocsf/src/events/api_activity.rs b/crates/openshell-ocsf/src/events/api_activity.rs index 55c2f70d6..456c0c768 100644 --- a/crates/openshell-ocsf/src/events/api_activity.rs +++ b/crates/openshell-ocsf/src/events/api_activity.rs @@ -110,7 +110,7 @@ mod tests { actor: Actor { process: Process::new("supervisor", 1), }, - src_endpoint: Endpoint::from_domain("inference.local", 443), + src_endpoint: Endpoint::from_domain("api.anthropic.com", 443), http_request: None, http_response: None, dst_endpoint: None, @@ -128,7 +128,7 @@ mod tests { assert_eq!(json["activity_id"], 99); assert_eq!(json["api"]["operation"], "POST /v1/messages"); assert_eq!(json["actor"]["process"]["name"], "supervisor"); - assert_eq!(json["src_endpoint"]["domain"], "inference.local"); + assert_eq!(json["src_endpoint"]["domain"], "api.anthropic.com"); assert_eq!(json["ai_model"]["name"], "claude-3-haiku"); } diff --git a/crates/openshell-ocsf/src/events/config_state_change.rs b/crates/openshell-ocsf/src/events/config_state_change.rs index 69333554a..d2c8b900c 100644 --- a/crates/openshell-ocsf/src/events/config_state_change.rs +++ b/crates/openshell-ocsf/src/events/config_state_change.rs @@ -10,7 +10,7 @@ use crate::events::base_event::BaseEventData; /// OCSF Device Config State Change Event [5019]. /// -/// Policy engine and inference routing configuration changes. +/// Policy engine and runtime configuration changes. #[derive(Debug, Clone, PartialEq, Eq, Deserialize)] pub struct DeviceConfigStateChangeEvent { /// Common base event fields. diff --git a/crates/openshell-ocsf/src/format/downgrade.rs b/crates/openshell-ocsf/src/format/downgrade.rs new file mode 100644 index 000000000..97f2fe88e --- /dev/null +++ b/crates/openshell-ocsf/src/format/downgrade.rs @@ -0,0 +1,248 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! OCSF schema version downgrade filter. +//! +//! Transforms serialized OCSF JSON events to conform to older schema versions +//! by stripping fields and profiles that don't exist in the target version. + +use serde_json::Value; + +/// Fields to strip when downgrading to v1.3.0 or earlier. +const STRIP_FOR_V1_3: &[&str] = &["ai_model", "container", "observation_point_id"]; + +/// Profile names to remove from `metadata.profiles` when downgrading to v1.3.0 or earlier. +const STRIP_PROFILES_V1_3: &[&str] = &["ai_operation", "container"]; + +/// Downgrade a serialized OCSF event to the target schema version. +/// +/// Modifies the JSON in place: strips fields that don't exist in the target +/// version, removes unknown profile names from `metadata.profiles`, and +/// rewrites `metadata.version` to match. +/// +/// Returns `true` if the event was modified, `false` if no changes were needed +/// (target is current version or newer). +pub fn downgrade_event(event: &mut Value, target_version: &str) -> bool { + let target = parse_version(target_version); + let v1_3 = (1, 3, 0); + + if target >= parse_version(crate::OCSF_VERSION) { + return false; + } + + let Some(obj) = event.as_object_mut() else { + return false; + }; + + let mut modified = false; + + if target <= v1_3 { + for field in STRIP_FOR_V1_3 { + if obj.remove(*field).is_some() { + modified = true; + } + } + + if let Some(profiles) = obj + .get_mut("metadata") + .and_then(Value::as_object_mut) + .and_then(|m| m.get_mut("profiles")) + .and_then(Value::as_array_mut) + { + let before = profiles.len(); + profiles.retain(|p| !p.as_str().is_some_and(|s| STRIP_PROFILES_V1_3.contains(&s))); + if profiles.len() != before { + modified = true; + } + } + } + + if let Some(metadata) = obj.get_mut("metadata").and_then(Value::as_object_mut) { + let original_version = metadata + .get("version") + .and_then(Value::as_str) + .unwrap_or(crate::OCSF_VERSION) + .to_string(); + metadata.insert( + "version".to_string(), + Value::String(target_version.to_string()), + ); + modified = true; + + let unmapped = obj + .entry("unmapped") + .or_insert_with(|| Value::Object(serde_json::Map::new())); + if let Some(u) = unmapped.as_object_mut() { + u.insert( + "downgraded_from".to_string(), + Value::String(original_version), + ); + } + } + + modified +} + +fn parse_version(v: &str) -> (u32, u32, u32) { + let parts: Vec = v.split('.').filter_map(|s| s.parse().ok()).collect(); + ( + parts.first().copied().unwrap_or(0), + parts.get(1).copied().unwrap_or(0), + parts.get(2).copied().unwrap_or(0), + ) +} + +#[cfg(test)] +mod tests { + use super::*; + + fn test_event() -> Value { + serde_json::json!({ + "class_uid": 4002, + "class_name": "HTTP Activity", + "time": 1_234_567_890, + "severity_id": 1, + "metadata": { + "version": crate::OCSF_VERSION, + "profiles": ["security_control", "network_proxy", "container", "host"] + }, + "device": {"hostname": "sandbox-1"}, + "container": {"name": "test-sandbox"}, + "observation_point_id": 2, + "unmapped": {"key": "value"} + }) + } + + #[test] + fn test_downgrade_to_v1_3_strips_fields() { + let mut event = test_event(); + let modified = downgrade_event(&mut event, "1.3.0"); + + assert!(modified); + assert!(event.get("container").is_none()); + assert!(event.get("observation_point_id").is_none()); + assert!(event.get("device").is_some()); + assert!(event.get("unmapped").is_some()); + } + + #[test] + fn test_downgrade_to_v1_1_strips_fields() { + let mut event = test_event(); + let modified = downgrade_event(&mut event, "1.1.0"); + + assert!(modified); + assert!(event.get("container").is_none()); + assert!(event.get("observation_point_id").is_none()); + } + + #[test] + fn test_downgrade_strips_profiles() { + let mut event = test_event(); + downgrade_event(&mut event, "1.3.0"); + + let profiles = event["metadata"]["profiles"].as_array().unwrap(); + assert!(!profiles.iter().any(|p| p == "container")); + assert!(profiles.iter().any(|p| p == "security_control")); + assert!(profiles.iter().any(|p| p == "host")); + } + + #[test] + fn test_downgrade_rewrites_version() { + let mut event = test_event(); + downgrade_event(&mut event, "1.1.0"); + + assert_eq!(event["metadata"]["version"], "1.1.0"); + assert_eq!(event["unmapped"]["downgraded_from"], crate::OCSF_VERSION); + } + + #[test] + fn test_no_downgrade_for_current_version() { + let mut event = test_event(); + let modified = downgrade_event(&mut event, crate::OCSF_VERSION); + + assert!(!modified); + assert_eq!(event["metadata"]["version"], crate::OCSF_VERSION); + } + + #[test] + fn test_no_downgrade_for_newer_version() { + let mut event = test_event(); + let modified = downgrade_event(&mut event, "1.9.0"); + + assert!(!modified); + } + + #[test] + fn test_downgrade_strips_ai_model_when_present() { + let mut event = serde_json::json!({ + "class_uid": 6003, + "metadata": { + "version": "1.8.0", + "profiles": ["container", "host", "ai_operation"] + }, + "ai_model": {"name": "claude-3-haiku", "ai_provider": "anthropic"}, + "unmapped": {"latency_ms": 701} + }); + let modified = downgrade_event(&mut event, "1.3.0"); + + assert!(modified); + assert!(event.get("ai_model").is_none()); + assert!( + !event["metadata"]["profiles"] + .as_array() + .unwrap() + .iter() + .any(|p| p == "ai_operation") + ); + assert_eq!(event["metadata"]["version"], "1.3.0"); + assert_eq!(event["unmapped"]["downgraded_from"], "1.8.0"); + assert_eq!(event["unmapped"]["latency_ms"], 701); + } + + #[test] + fn test_downgrade_creates_unmapped_when_absent() { + let mut event = serde_json::json!({ + "class_uid": 4001, + "metadata": { + "version": crate::OCSF_VERSION, + "profiles": ["container"] + }, + "container": {"name": "sandbox-1"} + }); + let modified = downgrade_event(&mut event, "1.1.0"); + + assert!(modified); + assert_eq!(event["unmapped"]["downgraded_from"], crate::OCSF_VERSION); + } + + #[test] + fn test_downgrade_rewrites_version_without_strippable_fields() { + let mut event = serde_json::json!({ + "class_uid": 0, + "metadata": { + "version": crate::OCSF_VERSION, + "profiles": ["host"] + } + }); + + let modified = downgrade_event(&mut event, "1.3.0"); + + assert!(modified); + assert_eq!(event["metadata"]["version"], "1.3.0"); + assert_eq!(event["unmapped"]["downgraded_from"], crate::OCSF_VERSION); + } + + #[test] + fn test_no_downgrade_omits_breadcrumb() { + let mut event = test_event(); + downgrade_event(&mut event, crate::OCSF_VERSION); + + assert!( + event + .get("unmapped") + .and_then(Value::as_object) + .and_then(|u| u.get("downgraded_from")) + .is_none() + ); + } +} diff --git a/crates/openshell-ocsf/src/format/mod.rs b/crates/openshell-ocsf/src/format/mod.rs index 084a013d9..17518a561 100644 --- a/crates/openshell-ocsf/src/format/mod.rs +++ b/crates/openshell-ocsf/src/format/mod.rs @@ -3,5 +3,6 @@ //! OCSF event formatters: shorthand (human-readable) and JSONL. +pub mod downgrade; pub mod jsonl; pub mod shorthand; diff --git a/crates/openshell-ocsf/src/lib.rs b/crates/openshell-ocsf/src/lib.rs index 345ea5717..c954fc562 100644 --- a/crates/openshell-ocsf/src/lib.rs +++ b/crates/openshell-ocsf/src/lib.rs @@ -44,8 +44,8 @@ pub use events::{ // --- Enum types --- pub use enums::{ - ActionId, ActivityId, AuthTypeId, ConfidenceId, DispositionId, HttpMethod, LaunchTypeId, - OcsfEnum, RiskLevelId, SecurityLevelId, SeverityId, StateId, StatusId, + ActionId, ActivityId, AuthTypeId, ConfidenceId, DeviceTypeId, DispositionId, HttpMethod, + LaunchTypeId, OcsfEnum, RiskLevelId, SecurityLevelId, SeverityId, StateId, StatusId, }; // --- Object types --- diff --git a/crates/openshell-ocsf/src/objects/device.rs b/crates/openshell-ocsf/src/objects/device.rs index 4c42fb4a1..b373a958c 100644 --- a/crates/openshell-ocsf/src/objects/device.rs +++ b/crates/openshell-ocsf/src/objects/device.rs @@ -5,12 +5,29 @@ use serde::{Deserialize, Serialize}; +use crate::enums::DeviceTypeId; + /// OCSF Device object. #[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] pub struct Device { /// Device hostname. pub hostname: String, + /// Administrator-assigned device name, when one exists. + #[serde(skip_serializing_if = "Option::is_none")] + pub name: Option, + + /// Stable unique identifier for the device. + #[serde(skip_serializing_if = "Option::is_none")] + pub uid: Option, + + /// Device type id. Required by the OCSF schema. + pub type_id: DeviceTypeId, + + /// Sibling label for `type_id`. + #[serde(rename = "type")] + pub type_label: String, + /// Operating system info. #[serde(skip_serializing_if = "Option::is_none")] pub os: Option, @@ -24,11 +41,15 @@ pub struct OsInfo { } impl Device { - /// Create a Linux device with the given hostname. + /// Create a Linux sandbox device with the given hostname. #[must_use] pub fn linux(hostname: &str) -> Self { Self { hostname: hostname.to_string(), + name: None, + uid: None, + type_id: DeviceTypeId::Other, + type_label: "Sandbox".to_string(), os: Some(OsInfo { name: "Linux".to_string(), }), @@ -47,4 +68,21 @@ mod tests { assert_eq!(json["hostname"], "sandbox-abc123"); assert_eq!(json["os"]["name"], "Linux"); } + + #[test] + fn sandbox_device_type_is_independent_of_its_os() { + let json = serde_json::to_value(Device::linux("sandbox-abc123")).unwrap(); + assert_eq!(json["type_id"], DeviceTypeId::Other.as_u8()); + assert_eq!(json["type"], "Sandbox"); + assert_eq!(json["os"]["name"], "Linux"); + } + + #[test] + fn device_round_trips() { + let device = Device::linux("sandbox-abc123"); + let json = serde_json::to_value(&device).unwrap(); + let decoded: Device = serde_json::from_value(json.clone()).unwrap(); + assert_eq!(decoded, device); + assert_eq!(serde_json::to_value(&decoded).unwrap(), json); + } } diff --git a/crates/openshell-ocsf/src/objects/metadata.rs b/crates/openshell-ocsf/src/objects/metadata.rs index 060f13ac6..14f5f8af6 100644 --- a/crates/openshell-ocsf/src/objects/metadata.rs +++ b/crates/openshell-ocsf/src/objects/metadata.rs @@ -18,7 +18,7 @@ pub struct Metadata { #[serde(skip_serializing_if = "Vec::is_empty")] pub profiles: Vec, - /// Unique event source identifier (sandbox ID). + /// Unique event identifier. #[serde(skip_serializing_if = "Option::is_none")] pub uid: Option, diff --git a/crates/openshell-ocsf/src/tracing_layers/jsonl_layer.rs b/crates/openshell-ocsf/src/tracing_layers/jsonl_layer.rs index 920483700..9414c7417 100644 --- a/crates/openshell-ocsf/src/tracing_layers/jsonl_layer.rs +++ b/crates/openshell-ocsf/src/tracing_layers/jsonl_layer.rs @@ -12,6 +12,7 @@ use tracing::Subscriber; use tracing_subscriber::Layer; use tracing_subscriber::layer::Context; +use crate::format::downgrade::downgrade_event; use crate::tracing_layers::event_bridge::{OCSF_TARGET, clone_current_event}; /// A tracing `Layer` that intercepts OCSF events and writes JSONL output. @@ -23,9 +24,15 @@ use crate::tracing_layers::event_bridge::{OCSF_TARGET, clone_current_event}; /// `false`, the layer short-circuits without writing. This allows the sandbox /// to hot-toggle OCSF JSONL output at runtime via the `ocsf_json_enabled` /// setting without rebuilding the subscriber. +/// +/// An optional target schema version can be set via +/// [`with_target_version`](Self::with_target_version). When set, events are +/// downgraded to the target version before writing (stripping fields and +/// profiles that don't exist in older schema versions). pub struct OcsfJsonlLayer { writer: Mutex, enabled: Option>, + target_version: Option>>, } impl OcsfJsonlLayer { @@ -35,6 +42,7 @@ impl OcsfJsonlLayer { Self { writer: Mutex::new(writer), enabled: None, + target_version: None, } } @@ -47,6 +55,16 @@ impl OcsfJsonlLayer { self.enabled = Some(flag); self } + + /// Attach a shared target schema version for downgrade filtering. + /// + /// When set, events are downgraded to the target version before writing. + /// The version can be changed at runtime via the shared mutex. + #[must_use] + pub fn with_target_version(mut self, version: Arc>) -> Self { + self.target_version = Some(version); + self + } } impl Layer for OcsfJsonlLayer @@ -67,9 +85,29 @@ where } if let Some(ocsf_event) = clone_current_event() - && let Ok(line) = ocsf_event.to_json_line() && let Ok(mut w) = self.writer.lock() { + let line = if let Some(ref target) = self.target_version + && let Ok(version) = target.lock() + && !version.is_empty() + { + let Ok(mut json) = serde_json::to_value(&ocsf_event) else { + return; + }; + downgrade_event(&mut json, &version); + match serde_json::to_string(&json) { + Ok(mut s) => { + s.push('\n'); + s + } + Err(_) => return, + } + } else { + match ocsf_event.to_json_line() { + Ok(l) => l, + Err(_) => return, + } + }; let _ = w.write_all(line.as_bytes()); } } diff --git a/crates/openshell-ocsf/tests/event_identity.rs b/crates/openshell-ocsf/tests/event_identity.rs new file mode 100644 index 000000000..3d8618bdd --- /dev/null +++ b/crates/openshell-ocsf/tests/event_identity.rs @@ -0,0 +1,79 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! `metadata.uid` identifies the event; `container.uid` identifies the sandbox. + +use std::net::{IpAddr, Ipv4Addr}; + +use openshell_ocsf::{ActivityId, NetworkActivityBuilder, OcsfEvent, SandboxContext, SeverityId}; + +fn sandbox_ctx(container_image: &str) -> SandboxContext { + SandboxContext { + sandbox_id: "sb-1".to_string(), + sandbox_name: "agent-01".to_string(), + container_image: container_image.to_string(), + hostname: "openshell-sb-1".to_string(), + product_version: "0.42.1".to_string(), + proxy_ip: IpAddr::V4(Ipv4Addr::LOCALHOST), + proxy_port: 8888, + } +} + +fn event(ctx: &SandboxContext) -> OcsfEvent { + NetworkActivityBuilder::new(ctx) + .activity(ActivityId::Open) + .severity(SeverityId::Medium) + .message("CONNECT api.example.com:443") + .build() +} + +#[test] +fn each_event_gets_its_own_metadata_uid() { + let ctx = sandbox_ctx("ghcr.io/nvidia/openshell/sandbox:0.42.1"); + let first = event(&ctx); + let second = event(&ctx); + + let first_uid = first.base().metadata.uid.clone().expect("uid is set"); + let second_uid = second.base().metadata.uid.clone().expect("uid is set"); + + assert!(!first_uid.is_empty()); + assert_ne!(first_uid, second_uid); +} + +#[test] +fn metadata_uid_is_no_longer_the_sandbox_id() { + let event = event(&sandbox_ctx("ghcr.io/nvidia/openshell/sandbox:0.42.1")); + assert_ne!(event.base().metadata.uid.as_deref(), Some("sb-1")); +} + +#[test] +fn the_sandbox_id_is_carried_by_the_container() { + let json = event(&sandbox_ctx("ghcr.io/nvidia/openshell/sandbox:0.42.1")) + .to_json() + .expect("serializes"); + + assert_eq!(json["container"]["uid"], "sb-1"); + assert_eq!(json["container"]["name"], "agent-01"); +} + +#[test] +fn a_container_without_an_image_omits_the_image() { + let json = event(&sandbox_ctx("")).to_json().expect("serializes"); + assert!(json["container"].get("image").is_none()); +} + +#[test] +fn device_identifies_the_sandbox_environment() { + let json = event(&sandbox_ctx("image:1")).to_json().unwrap(); + + assert_eq!(json["device"]["type_id"], 99); + assert_eq!(json["device"]["type"], "Sandbox"); + assert_eq!(json["device"]["os"]["name"], "Linux"); +} + +#[test] +fn event_identity_survives_serialization() { + let original = event(&sandbox_ctx("image:1")); + let decoded: OcsfEvent = serde_json::from_value(original.to_json().unwrap()).unwrap(); + assert_eq!(decoded.base().metadata.uid, original.base().metadata.uid); +} diff --git a/crates/openshell-ocsf/tests/roundtrip.rs b/crates/openshell-ocsf/tests/roundtrip.rs index 1c8a8bb30..d179510b1 100644 --- a/crates/openshell-ocsf/tests/roundtrip.rs +++ b/crates/openshell-ocsf/tests/roundtrip.rs @@ -22,7 +22,7 @@ fn ctx() -> SandboxContext { SandboxContext { sandbox_id: "sb-7f3a9c2e14b8".to_string(), sandbox_name: "agent-workspace-01".to_string(), - container_image: "ghcr.io/nvidia/openshell/sandbox:0.42.1".to_string(), + container_image: "ghcr.io/nvidia/openshell-community/sandboxes/base:latest".to_string(), hostname: "openshell-sb-7f3a9c2e14b8".to_string(), product_version: "0.42.1".to_string(), proxy_ip: IpAddr::V4(Ipv4Addr::LOCALHOST), @@ -197,12 +197,11 @@ fn api_activity_round_trips() { .status(StatusId::Success) .http_request(HttpRequest { http_method: HttpMethod::Post, - url: Some(Url::new("https", "inference.local", "/v1/chat", 443)), + url: Some(Url::new("https", "api.example.com", "/v1/chat", 443)), }) - .dst_endpoint(Endpoint::from_domain("inference.local", 443)) + .dst_endpoint(Endpoint::from_domain("api.example.com", 443)) .ai_model(AiModel::new("llama-3.1-8b", "nvidia")) - .message("inference request routed") - .unmapped("route", "system") + .message("inference request completed") .build(); assert_round_trips("api_activity", &event); diff --git a/crates/openshell-otel-test-support/Cargo.toml b/crates/openshell-otel-test-support/Cargo.toml index dac6ecbe7..48be77e1c 100644 --- a/crates/openshell-otel-test-support/Cargo.toml +++ b/crates/openshell-otel-test-support/Cargo.toml @@ -11,10 +11,13 @@ license.workspace = true repository.workspace = true [dependencies] +openshell-otel = { path = "../openshell-otel" } opentelemetry-proto = { version = "0.32", default-features = false, features = ["gen-tonic", "trace"] } tokio = { workspace = true } tokio-stream = { workspace = true } tonic = { workspace = true } +tracing = { workspace = true } +tracing-subscriber = { workspace = true } [lints] workspace = true diff --git a/crates/openshell-otel-test-support/src/lib.rs b/crates/openshell-otel-test-support/src/lib.rs index 4edbe8b7c..5d6db481a 100644 --- a/crates/openshell-otel-test-support/src/lib.rs +++ b/crates/openshell-otel-test-support/src/lib.rs @@ -12,11 +12,65 @@ use opentelemetry_proto::tonic::collector::trace::v1::{ }; use opentelemetry_proto::tonic::trace::v1::Span; +/// Serialize tracing tests and keep their callsites enabled process-wide. +pub async fn tracing_test_lock() -> tokio::sync::MutexGuard<'static, ()> { + static LOCK: tokio::sync::Mutex<()> = tokio::sync::Mutex::const_new(()); + static INITIALIZED: std::sync::LazyLock<()> = std::sync::LazyLock::new(|| { + tracing::subscriber::set_global_default(tracing_subscriber::registry()) + .expect("test tracing subscriber installs once"); + }); + + let guard = LOCK.lock().await; + std::sync::LazyLock::force(&INITIALIZED); + guard +} + +/// Verify one compute-driver descriptor exports spans and shared resources. +pub async fn assert_compute_driver_tracing( + descriptor: openshell_otel::ComputeDriverTracing, + service_version: &'static str, + span_name: &'static str, +) { + use tracing_subscriber::layer::SubscriberExt as _; + + let _tracing_lock = tracing_test_lock().await; + let collector = OtlpTestServer::start().await; + let (provider, error) = descriptor.provider_for( + Some(collector.endpoint()), + service_version, + Some("test-gateway"), + Some(descriptor.compute_driver()), + ); + assert!(error.is_none(), "valid OTLP endpoint should configure"); + let provider = provider.expect("provider"); + let subscriber = tracing_subscriber::registry().with(descriptor.layer(&provider)); + tracing::subscriber::with_default(subscriber, || { + let span = tracing::info_span!("driver.test", otel.name = span_name); + drop(span.enter()); + drop(span); + }); + provider.force_flush().unwrap(); + collector.wait_for_export().await; + provider.shutdown().unwrap(); + let received = collector.shutdown().await; + + assert!(received.spans.iter().any(|span| span.name == span_name)); + assert_eq!(received.gateway_names, ["test-gateway"]); + assert_eq!(received.compute_drivers, [descriptor.compute_driver()]); + assert!( + received + .service_names + .iter() + .any(|name| name == descriptor.service_name()) + ); +} + #[derive(Clone, Debug, Default)] pub struct ReceivedTraces { pub spans: Vec, pub service_names: Vec, pub gateway_names: Vec, + pub compute_drivers: Vec, } #[derive(Clone)] @@ -51,6 +105,9 @@ impl TraceService for Collector { match attribute.key.as_str() { "service.name" => received.service_names.push(value), "openshell.gateway.name" => received.gateway_names.push(value), + "openshell.gateway.compute_driver" => { + received.compute_drivers.push(value); + } _ => {} } } diff --git a/crates/openshell-otel/Cargo.toml b/crates/openshell-otel/Cargo.toml index 815598982..e4b5de770 100644 --- a/crates/openshell-otel/Cargo.toml +++ b/crates/openshell-otel/Cargo.toml @@ -11,6 +11,7 @@ license.workspace = true repository.workspace = true [dependencies] +futures = { workspace = true } http = { workspace = true } opentelemetry = { workspace = true } opentelemetry_sdk = { workspace = true } diff --git a/crates/openshell-otel/src/driver.rs b/crates/openshell-otel/src/driver.rs new file mode 100644 index 000000000..c28f5df55 --- /dev/null +++ b/crates/openshell-otel/src/driver.rs @@ -0,0 +1,258 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! Shared compute-driver tracing setup and in-process RPC instrumentation. + +use std::future::Future; +use std::pin::Pin; + +use futures::Stream; +use tracing::Instrument as _; +use tracing_subscriber::EnvFilter; +use tracing_subscriber::prelude::*; +use tracing_subscriber::registry::LookupSpan; + +use crate::{OtlpTraceConfig, SdkTracerProvider, ServiceName, SetupError}; + +/// Target used by every in-process compute-driver RPC boundary. +pub const IN_PROCESS_COMPUTE_DRIVER_TARGET: &str = "openshell_otel::in_process_compute_driver"; + +/// Define the calling driver crate's [`ComputeDriverTracing`] identity from +/// its Cargo package and crate names. +#[macro_export] +macro_rules! compute_driver_tracing { + () => { + $crate::ComputeDriverTracing::new(env!("CARGO_PKG_NAME"), env!("CARGO_CRATE_NAME")) + }; +} + +/// Tracing identity and layer factory for one compute-driver service. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct ComputeDriverTracing { + service_name: &'static str, + crate_target_prefix: &'static str, +} + +impl ComputeDriverTracing { + /// `crate_target_prefix` must be the driver's Rust crate name; it routes + /// that crate's spans to the driver's provider. Prefer + /// [`compute_driver_tracing!`](crate::compute_driver_tracing). + #[must_use] + pub const fn new(service_name: &'static str, crate_target_prefix: &'static str) -> Self { + Self { + service_name, + crate_target_prefix, + } + } + + #[must_use] + pub const fn service_name(self) -> &'static str { + self.service_name + } + + /// Canonical compute-driver name derived from `service.name`. + #[must_use] + pub fn compute_driver(self) -> &'static str { + self.service_name + .strip_prefix("openshell-driver-") + .unwrap_or(self.service_name) + } + + #[must_use] + pub const fn in_process_target(self) -> &'static str { + IN_PROCESS_COMPUTE_DRIVER_TARGET + } + + /// Targets routed to this driver's provider when it runs in-process. + #[must_use] + pub const fn in_process_targets(self) -> [&'static str; 2] { + [self.in_process_target(), self.crate_target_prefix] + } + + #[must_use] + pub fn provider_for( + self, + endpoint: Option<&str>, + service_version: &'static str, + gateway_name: Option<&str>, + compute_driver: Option<&str>, + ) -> (Option, Option) { + crate::provider_for(endpoint.map(|endpoint| OtlpTraceConfig { + endpoint, + service_name: ServiceName::Fixed(self.service_name), + service_version: Some(service_version), + resource_attributes: crate::gateway_resource_attributes(gateway_name, compute_driver), + })) + } + + pub fn layer(self, provider: &SdkTracerProvider) -> crate::OtlpLayer + where + S: tracing::Subscriber + for<'span> LookupSpan<'span>, + { + crate::layer(provider, self.service_name) + } + + pub fn in_process_layer(self, provider: &SdkTracerProvider) -> crate::TargetOtlpLayer + where + S: tracing::Subscriber + for<'span> LookupSpan<'span>, + { + crate::layer_for_target_prefixes(provider, self.service_name, self.in_process_targets()) + } +} + +/// Optional in-process server boundary tracing for a compute-driver service. +#[derive(Debug, Clone, Copy)] +pub struct InProcessRpcTracer { + enabled: bool, +} + +impl InProcessRpcTracer { + #[must_use] + pub const fn disabled() -> Self { + Self { enabled: false } + } + + #[must_use] + pub const fn enabled() -> Self { + Self { enabled: true } + } + + fn span(self, rpc: crate::ComputeDriverRpc) -> Option { + self.enabled.then(|| { + tracing::info_span!( + target: IN_PROCESS_COMPUTE_DRIVER_TARGET, + "driver_rpc", + otel.name = rpc.operation, + otel.kind = "server", + otel.status_code = tracing::field::Empty, + rpc.system.name = "grpc", + rpc.method = rpc.operation, + rpc.response.status_code = tracing::field::Empty, + error.type = tracing::field::Empty, + ) + }) + } + + pub async fn trace( + self, + rpc: crate::ComputeDriverRpc, + future: impl Future>, + ) -> Result { + let Some(span) = self.span(rpc) else { + return future.await; + }; + let result = future.instrument(span.clone()).await; + record_result(&span, &result); + result + } + + pub async fn trace_stream( + self, + rpc: crate::ComputeDriverRpc, + future: impl Future, tonic::Status>>, + ) -> Result, tonic::Status> + where + T: 'static, + { + let Some(span) = self.span(rpc) else { + return future.await; + }; + match future.instrument(span.clone()).await { + Ok(stream) => Ok(Box::pin(crate::TracedGrpcStream::new(stream, span))), + Err(status) => { + crate::record_grpc_status(&span, status.code()); + Err(status) + } + } + } +} + +pub type BoxGrpcStream = Pin> + Send + 'static>>; + +fn record_result(span: &tracing::Span, result: &Result) { + crate::record_grpc_status( + span, + result + .as_ref() + .map_or_else(tonic::Status::code, |_| tonic::Code::Ok), + ); +} + +/// Owns and shuts down a standalone compute driver's tracer provider. +pub struct DriverTracingHandle { + provider: Option, +} + +/// Named inputs for standalone compute-driver tracing installation. +#[derive(Debug, Clone, Copy)] +pub struct DriverTracingConfig<'a> { + pub endpoint: Option<&'a str>, + pub gateway_name: Option<&'a str>, + pub service_version: &'static str, + pub log_level: &'a str, +} + +impl Drop for DriverTracingHandle { + fn drop(&mut self) { + if let Some(provider) = &self.provider + && let Err(error) = provider.shutdown() + { + tracing::warn!(%error, "OTLP tracer provider shutdown failed"); + } + } +} + +/// Install standalone compute-driver logging and OTLP tracing. +#[must_use] +pub fn install_driver_tracing( + descriptor: ComputeDriverTracing, + config: DriverTracingConfig<'_>, +) -> DriverTracingHandle { + let (provider, setup_error) = descriptor.provider_for( + config.endpoint, + config.service_version, + config.gateway_name, + Some(descriptor.compute_driver()), + ); + tracing_subscriber::registry() + .with( + EnvFilter::try_from_default_env().unwrap_or_else(|_| EnvFilter::new(config.log_level)), + ) + .with(tracing_subscriber::fmt::layer()) + .with(provider.as_ref().map(|provider| descriptor.layer(provider))) + .init(); + + if let Some(error) = setup_error { + tracing::error!(%error, "OTLP exporting could not be started"); + } else if let Some(endpoint) = config.endpoint { + tracing::info!(endpoint, "OTLP exporting enabled"); + } + + DriverTracingHandle { provider } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn tracing_macro_derives_identity_from_the_calling_crate() { + const TRACING: ComputeDriverTracing = crate::compute_driver_tracing!(); + + assert_eq!(TRACING.service_name(), "openshell-otel"); + assert_eq!(TRACING.in_process_targets()[1], "openshell_otel"); + } + + #[tokio::test] + async fn disabled_stream_tracing_returns_the_original_box() { + let stream: BoxGrpcStream<()> = Box::pin(futures::stream::empty()); + let original = std::ptr::from_ref(stream.as_ref().get_ref()); + + let returned = InProcessRpcTracer::disabled() + .trace_stream(crate::rpc::WATCH_SANDBOXES, async move { Ok(stream) }) + .await + .unwrap(); + + assert!(std::ptr::eq(original, returned.as_ref().get_ref())); + } +} diff --git a/crates/openshell-otel/src/grpc.rs b/crates/openshell-otel/src/grpc.rs index 3c38ddd74..65d498ecc 100644 --- a/crates/openshell-otel/src/grpc.rs +++ b/crates/openshell-otel/src/grpc.rs @@ -3,6 +3,7 @@ //! Shared gRPC tracing adapters. +use futures::Stream; use http::Request; use opentelemetry::propagation::TextMapPropagator as _; use opentelemetry::trace::TraceContextExt as _; @@ -12,7 +13,132 @@ use tower_http::trace::{GrpcMakeClassifier, MakeSpan, OnEos, OnFailure, OnRespon use tracing::Span; use tracing_opentelemetry::OpenTelemetrySpanExt as _; -const COMPUTE_DRIVER_SERVICE: &str = "openshell.compute.v1.ComputeDriver"; +/// Keeps a gRPC server span alive for a response stream and records its outcome. +pub struct TracedGrpcStream { + inner: S, + span: Span, + finished: bool, +} + +impl TracedGrpcStream { + #[must_use] + pub fn new(inner: S, span: Span) -> Self { + Self { + inner, + span, + finished: false, + } + } +} + +impl Stream for TracedGrpcStream +where + S: Stream> + Unpin, +{ + type Item = Result; + + fn poll_next( + self: std::pin::Pin<&mut Self>, + cx: &mut std::task::Context<'_>, + ) -> std::task::Poll> { + let this = self.get_mut(); + let span = this.span.clone(); + let _entered = span.enter(); + let result = std::pin::Pin::new(&mut this.inner).poll_next(cx); + if !this.finished { + match &result { + std::task::Poll::Ready(Some(Err(status))) => { + record_grpc_status(&this.span, status.code()); + this.finished = true; + } + std::task::Poll::Ready(None) => { + record_grpc_status(&this.span, tonic::Code::Ok); + this.finished = true; + } + std::task::Poll::Pending | std::task::Poll::Ready(Some(Ok(_))) => {} + } + } + result + } +} + +pub const COMPUTE_DRIVER_RPC_SERVICE: &str = "openshell.compute.v1.ComputeDriver"; + +/// Low-cardinality semantic-convention identity for a compute-driver RPC. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct ComputeDriverRpc { + pub service: &'static str, + pub method: &'static str, + pub operation: &'static str, +} + +impl ComputeDriverRpc { + const fn new(method: &'static str, operation: &'static str) -> Self { + Self { + service: COMPUTE_DRIVER_RPC_SERVICE, + method, + operation, + } + } +} + +/// Typed identities for every RPC in the generated compute-driver service. +pub mod rpc { + use super::ComputeDriverRpc; + + pub const AUTHENTICATE_SANDBOX: ComputeDriverRpc = ComputeDriverRpc::new( + "AuthenticateSandbox", + "openshell.compute.v1.ComputeDriver/AuthenticateSandbox", + ); + pub const GET_CAPABILITIES: ComputeDriverRpc = ComputeDriverRpc::new( + "GetCapabilities", + "openshell.compute.v1.ComputeDriver/GetCapabilities", + ); + pub const GET_GATEWAY_LISTENER_REQUIREMENTS: ComputeDriverRpc = ComputeDriverRpc::new( + "GetGatewayListenerRequirements", + "openshell.compute.v1.ComputeDriver/GetGatewayListenerRequirements", + ); + pub const VALIDATE_SANDBOX_CREATE: ComputeDriverRpc = ComputeDriverRpc::new( + "ValidateSandboxCreate", + "openshell.compute.v1.ComputeDriver/ValidateSandboxCreate", + ); + pub const CREATE_SANDBOX: ComputeDriverRpc = ComputeDriverRpc::new( + "CreateSandbox", + "openshell.compute.v1.ComputeDriver/CreateSandbox", + ); + pub const GET_SANDBOX: ComputeDriverRpc = ComputeDriverRpc::new( + "GetSandbox", + "openshell.compute.v1.ComputeDriver/GetSandbox", + ); + pub const LIST_SANDBOXES: ComputeDriverRpc = ComputeDriverRpc::new( + "ListSandboxes", + "openshell.compute.v1.ComputeDriver/ListSandboxes", + ); + pub const STOP_SANDBOX: ComputeDriverRpc = ComputeDriverRpc::new( + "StopSandbox", + "openshell.compute.v1.ComputeDriver/StopSandbox", + ); + pub const START_SANDBOX: ComputeDriverRpc = ComputeDriverRpc::new( + "StartSandbox", + "openshell.compute.v1.ComputeDriver/StartSandbox", + ); + pub const DELETE_SANDBOX: ComputeDriverRpc = ComputeDriverRpc::new( + "DeleteSandbox", + "openshell.compute.v1.ComputeDriver/DeleteSandbox", + ); + pub const WATCH_SANDBOXES: ComputeDriverRpc = ComputeDriverRpc::new( + "WatchSandboxes", + "openshell.compute.v1.ComputeDriver/WatchSandboxes", + ); + pub const ENSURE_WORKSPACE: ComputeDriverRpc = ComputeDriverRpc::new( + "EnsureWorkspace", + "openshell.compute.v1.ComputeDriver/EnsureWorkspace", + ); + pub const DELETE_WORKSPACE: ComputeDriverRpc = ComputeDriverRpc::new( + "DeleteWorkspace", + "openshell.compute.v1.ComputeDriver/DeleteWorkspace", + ); +} /// Trace every inbound compute-driver RPC at the tonic service boundary. pub fn compute_driver_rpc_layer() -> TraceLayer< @@ -39,16 +165,16 @@ pub struct ComputeDriverRpcSpan; impl MakeSpan for ComputeDriverRpcSpan { fn make_span(&mut self, request: &Request) -> Span { - let (operation, method) = compute_driver_rpc_operation(request.uri().path()); + let rpc = compute_driver_rpc_operation(request.uri().path()); let span = tracing::info_span!( "driver_rpc", - otel.name = operation, + otel.name = rpc.map_or("grpc", |rpc| rpc.operation), otel.kind = "server", otel.status_code = tracing::field::Empty, - rpc.system = "grpc", - rpc.service = COMPUTE_DRIVER_SERVICE, - rpc.method = method, - rpc.grpc.status_code = tracing::field::Empty, + rpc.system.name = "grpc", + rpc.method = rpc.map_or("_OTHER", |rpc| rpc.operation), + rpc.response.status_code = tracing::field::Empty, + error.type = tracing::field::Empty, ); let parent = TraceContextPropagator::new().extract_with_context( &opentelemetry::Context::new(), @@ -62,26 +188,62 @@ impl MakeSpan for ComputeDriverRpcSpan { } /// Maps the generated compute-driver RPC schema to low-cardinality span names. -pub fn compute_driver_rpc_operation(path: &str) -> (&'static str, &'static str) { +pub fn compute_driver_rpc_operation(path: &str) -> Option { match path.rsplit('/').next() { - Some("GetCapabilities") => ("driver.get_capabilities", "get_capabilities"), - Some("GetGatewayListenerRequirements") => ( - "driver.get_gateway_listener_requirements", - "get_gateway_listener_requirements", - ), - Some("ValidateSandboxCreate") => { - ("driver.validate_sandbox_create", "validate_sandbox_create") - } - Some("CreateSandbox") => ("driver.create_sandbox", "create_sandbox"), - Some("GetSandbox") => ("driver.get_sandbox", "get_sandbox"), - Some("ListSandboxes") => ("driver.list_sandboxes", "list_sandboxes"), - Some("StopSandbox") => ("driver.stop_sandbox", "stop_sandbox"), - Some("StartSandbox") => ("driver.start_sandbox", "start_sandbox"), - Some("DeleteSandbox") => ("driver.delete_sandbox", "delete_sandbox"), - Some("WatchSandboxes") => ("driver.watch_sandboxes", "watch_sandboxes"), - Some("EnsureWorkspace") => ("driver.ensure_workspace", "ensure_workspace"), - Some("DeleteWorkspace") => ("driver.delete_workspace", "delete_workspace"), - _ => ("driver.unknown", "unknown"), + Some("AuthenticateSandbox") => Some(rpc::AUTHENTICATE_SANDBOX), + Some("GetCapabilities") => Some(rpc::GET_CAPABILITIES), + Some("GetGatewayListenerRequirements") => Some(rpc::GET_GATEWAY_LISTENER_REQUIREMENTS), + Some("ValidateSandboxCreate") => Some(rpc::VALIDATE_SANDBOX_CREATE), + Some("CreateSandbox") => Some(rpc::CREATE_SANDBOX), + Some("GetSandbox") => Some(rpc::GET_SANDBOX), + Some("ListSandboxes") => Some(rpc::LIST_SANDBOXES), + Some("StopSandbox") => Some(rpc::STOP_SANDBOX), + Some("StartSandbox") => Some(rpc::START_SANDBOX), + Some("DeleteSandbox") => Some(rpc::DELETE_SANDBOX), + Some("WatchSandboxes") => Some(rpc::WATCH_SANDBOXES), + Some("EnsureWorkspace") => Some(rpc::ENSURE_WORKSPACE), + Some("DeleteWorkspace") => Some(rpc::DELETE_WORKSPACE), + _ => None, + } +} + +/// Return the stable OpenTelemetry spelling for a gRPC response status. +#[must_use] +pub const fn grpc_status_code_name(code: tonic::Code) -> &'static str { + match code { + tonic::Code::Ok => "OK", + tonic::Code::Cancelled => "CANCELLED", + tonic::Code::Unknown => "UNKNOWN", + tonic::Code::InvalidArgument => "INVALID_ARGUMENT", + tonic::Code::DeadlineExceeded => "DEADLINE_EXCEEDED", + tonic::Code::NotFound => "NOT_FOUND", + tonic::Code::AlreadyExists => "ALREADY_EXISTS", + tonic::Code::PermissionDenied => "PERMISSION_DENIED", + tonic::Code::ResourceExhausted => "RESOURCE_EXHAUSTED", + tonic::Code::FailedPrecondition => "FAILED_PRECONDITION", + tonic::Code::Aborted => "ABORTED", + tonic::Code::OutOfRange => "OUT_OF_RANGE", + tonic::Code::Unimplemented => "UNIMPLEMENTED", + tonic::Code::Internal => "INTERNAL", + tonic::Code::Unavailable => "UNAVAILABLE", + tonic::Code::DataLoss => "DATA_LOSS", + tonic::Code::Unauthenticated => "UNAUTHENTICATED", + } +} + +/// Records a complete gRPC outcome on `span`. +/// +/// Non-OK codes also set `otel.status_code` to `ERROR` and record the code as +/// `error.type`. OK codes leave the OpenTelemetry span status unset. +/// +/// The span must declare `otel.status_code`, `rpc.response.status_code`, and +/// `error.type` when it is created because `tracing` ignores undeclared fields. +pub fn record_grpc_status(span: &Span, code: tonic::Code) { + let code_name = grpc_status_code_name(code); + span.record("rpc.response.status_code", code_name); + if code != tonic::Code::Ok { + crate::mark_error(span); + span.record("error.type", code_name); } } @@ -96,9 +258,11 @@ impl OnFailure for RecordGrpcFailure { _latency: std::time::Duration, span: &Span, ) { - crate::mark_error(span); if let GrpcFailureClass::Code(code) = failure { - span.record("rpc.grpc.status_code", code.get()); + let code = tonic::Code::from_i32(code.get()); + record_grpc_status(span, code); + } else { + crate::mark_error(span); } } } @@ -116,10 +280,8 @@ impl RecordGrpcStatus { else { return; }; - if code != tonic::Code::Ok as i32 { - crate::mark_error(span); - } - span.record("rpc.grpc.status_code", code); + let code = tonic::Code::from_i32(code); + record_grpc_status(span, code); } } @@ -150,57 +312,77 @@ mod tests { #[test] fn compute_driver_rpc_names_are_explicitly_mapped_and_schema_bounded() { - for (rpc, operation, method) in [ - ( - "GetCapabilities", - "driver.get_capabilities", - "get_capabilities", - ), - ( - "GetGatewayListenerRequirements", - "driver.get_gateway_listener_requirements", - "get_gateway_listener_requirements", - ), - ( - "ValidateSandboxCreate", - "driver.validate_sandbox_create", - "validate_sandbox_create", - ), - ("CreateSandbox", "driver.create_sandbox", "create_sandbox"), - ("GetSandbox", "driver.get_sandbox", "get_sandbox"), - ("ListSandboxes", "driver.list_sandboxes", "list_sandboxes"), - ("StopSandbox", "driver.stop_sandbox", "stop_sandbox"), - ("StartSandbox", "driver.start_sandbox", "start_sandbox"), - ("DeleteSandbox", "driver.delete_sandbox", "delete_sandbox"), - ( - "WatchSandboxes", - "driver.watch_sandboxes", - "watch_sandboxes", - ), - ( - "EnsureWorkspace", - "driver.ensure_workspace", - "ensure_workspace", - ), - ( - "DeleteWorkspace", - "driver.delete_workspace", - "delete_workspace", - ), + for rpc in [ + rpc::AUTHENTICATE_SANDBOX, + rpc::GET_CAPABILITIES, + rpc::GET_GATEWAY_LISTENER_REQUIREMENTS, + rpc::VALIDATE_SANDBOX_CREATE, + rpc::CREATE_SANDBOX, + rpc::GET_SANDBOX, + rpc::LIST_SANDBOXES, + rpc::STOP_SANDBOX, + rpc::START_SANDBOX, + rpc::DELETE_SANDBOX, + rpc::WATCH_SANDBOXES, + rpc::ENSURE_WORKSPACE, + rpc::DELETE_WORKSPACE, ] { + assert_eq!(rpc.operation, format!("{}/{}", rpc.service, rpc.method)); assert_eq!( - compute_driver_rpc_operation(&format!("/openshell.compute.v1.ComputeDriver/{rpc}")), - (operation, method) + compute_driver_rpc_operation(&format!("/{}/{}", rpc.service, rpc.method)), + Some(rpc) ); } assert_eq!( compute_driver_rpc_operation( "/openshell.compute.v1.ComputeDriver/AttackerControlled12345" ), - ("driver.unknown", "unknown") + None ); } + #[test] + fn traced_stream_records_an_observed_cancelled_status() { + let _tracing_lock = crate::test_lock(); + let exporter = InMemorySpanExporterBuilder::new().build(); + let provider = SdkTracerProvider::builder() + .with_simple_exporter(exporter.clone()) + .build(); + let subscriber = tracing_subscriber::registry().with(crate::layer(&provider, "test")); + + tracing::subscriber::with_default(subscriber, || { + let span = tracing::info_span!( + "watch", + otel.status_code = tracing::field::Empty, + rpc.response.status_code = tracing::field::Empty, + error.type = tracing::field::Empty, + ); + let inner = + futures::stream::iter([Err::<(), _>(tonic::Status::cancelled("peer cancelled"))]); + let mut stream = TracedGrpcStream::new(inner, span); + let mut cx = std::task::Context::from_waker(futures::task::noop_waker_ref()); + let result = std::pin::Pin::new(&mut stream).poll_next(&mut cx); + assert!(matches!( + result, + std::task::Poll::Ready(Some(Err(status))) + if status.code() == tonic::Code::Cancelled + )); + }); + provider.force_flush().unwrap(); + + let spans = exporter.get_finished_spans().unwrap(); + assert_eq!(spans.len(), 1); + assert!(matches!( + spans[0].status, + opentelemetry::trace::Status::Error { .. } + )); + assert!(spans[0].attributes.iter().any(|attribute| { + attribute.key.as_str() == "rpc.response.status_code" + && attribute.value.to_string() == "CANCELLED" + })); + provider.shutdown().unwrap(); + } + #[test] fn grpc_status_records_and_marks_non_ok_trailer_status() { let _tracing_lock = crate::test_lock(); @@ -216,7 +398,8 @@ mod tests { let span = tracing::info_span!( "rpc", otel.status_code = tracing::field::Empty, - rpc.grpc.status_code = tracing::field::Empty, + rpc.response.status_code = tracing::field::Empty, + error.type = tracing::field::Empty, ); RecordGrpcStatus.on_eos(Some(&trailers), std::time::Duration::ZERO, &span); }); @@ -229,8 +412,51 @@ mod tests { opentelemetry::trace::Status::Error { .. } )); assert!(spans[0].attributes.iter().any(|attribute| { - attribute.key.as_str() == "rpc.grpc.status_code" && attribute.value.to_string() == "13" + attribute.key.as_str() == "rpc.response.status_code" + && attribute.value.to_string() == "INTERNAL" })); + assert!(spans[0].attributes.iter().any(|attribute| { + attribute.key.as_str() == "error.type" && attribute.value.to_string() == "INTERNAL" + })); + provider.shutdown().unwrap(); + } + + #[test] + fn grpc_status_records_ok_without_marking_an_error() { + let _tracing_lock = crate::test_lock(); + let exporter = InMemorySpanExporterBuilder::new().build(); + let provider = SdkTracerProvider::builder() + .with_simple_exporter(exporter.clone()) + .build(); + let subscriber = tracing_subscriber::registry().with(crate::layer(&provider, "test")); + + tracing::subscriber::with_default(subscriber, || { + let span = tracing::info_span!( + "rpc", + otel.status_code = tracing::field::Empty, + rpc.response.status_code = tracing::field::Empty, + error.type = tracing::field::Empty, + ); + record_grpc_status(&span, tonic::Code::Ok); + }); + provider.force_flush().unwrap(); + + let spans = exporter.get_finished_spans().unwrap(); + assert_eq!(spans.len(), 1); + assert!(matches!( + spans[0].status, + opentelemetry::trace::Status::Unset + )); + assert!(spans[0].attributes.iter().any(|attribute| { + attribute.key.as_str() == "rpc.response.status_code" + && attribute.value.to_string() == "OK" + })); + assert!( + spans[0] + .attributes + .iter() + .all(|attribute| attribute.key.as_str() != "error.type") + ); provider.shutdown().unwrap(); } @@ -251,7 +477,7 @@ mod tests { let span = tracing::info_span!( "rpc", otel.status_code = tracing::field::Empty, - rpc.grpc.status_code = tracing::field::Empty, + rpc.response.status_code = tracing::field::Empty, ); RecordGrpcStatus.on_response(&response, std::time::Duration::ZERO, &span); RecordGrpcStatus.on_eos(None, std::time::Duration::ZERO, &span); @@ -265,7 +491,8 @@ mod tests { opentelemetry::trace::Status::Error { .. } )); assert!(spans[0].attributes.iter().any(|attribute| { - attribute.key.as_str() == "rpc.grpc.status_code" && attribute.value.to_string() == "13" + attribute.key.as_str() == "rpc.response.status_code" + && attribute.value.to_string() == "INTERNAL" })); provider.shutdown().unwrap(); } diff --git a/crates/openshell-otel/src/lib.rs b/crates/openshell-otel/src/lib.rs index 30a9d2973..7a9162ab9 100644 --- a/crates/openshell-otel/src/lib.rs +++ b/crates/openshell-otel/src/lib.rs @@ -3,12 +3,19 @@ //! Shared OpenTelemetry trace export support for `OpenShell` services. +mod driver; mod grpc; mod propagation; +pub use driver::{ + BoxGrpcStream, ComputeDriverTracing, DriverTracingConfig, DriverTracingHandle, + IN_PROCESS_COMPUTE_DRIVER_TARGET, InProcessRpcTracer, install_driver_tracing, +}; + pub use grpc::{ - ComputeDriverRpcSpan, RecordGrpcFailure, RecordGrpcStatus, compute_driver_rpc_layer, - compute_driver_rpc_operation, + COMPUTE_DRIVER_RPC_SERVICE, ComputeDriverRpc, ComputeDriverRpcSpan, RecordGrpcFailure, + RecordGrpcStatus, TracedGrpcStream, compute_driver_rpc_layer, compute_driver_rpc_operation, + grpc_status_code_name, record_grpc_status, rpc, }; pub use propagation::{ HeaderMapExtractor, MetadataMapInjector, TraceContextInterceptor, current_trace_context_carrier, @@ -28,6 +35,27 @@ use tracing_subscriber::registry::LookupSpan; const SDK_UNKNOWN_SERVICE_PREFIX: &str = "unknown_service"; +/// Build the resource attributes shared by gateway and compute-driver providers. +pub fn gateway_resource_attributes( + gateway_name: Option<&str>, + compute_driver: Option<&str>, +) -> Vec { + let mut attributes = Vec::new(); + if let Some(name) = gateway_name.map(str::trim).filter(|name| !name.is_empty()) { + attributes.push(KeyValue::new("openshell.gateway.name", name.to_string())); + } + if let Some(driver) = compute_driver + .map(str::trim) + .filter(|driver| !driver.is_empty()) + { + attributes.push(KeyValue::new( + "openshell.gateway.compute_driver", + driver.to_string(), + )); + } + attributes +} + /// Mark `span` as failed. /// /// The field must be declared on the span at creation because `tracing` drops @@ -202,17 +230,18 @@ pub type OtlpLayer = tracing_subscriber::filter::Filtered< pub type TargetOtlpLayer = tracing_subscriber::filter::Filtered, TargetPrefixFilter, S>; -#[derive(Debug, Clone, Copy)] +#[derive(Debug, Clone)] pub struct TargetPrefixFilter { - prefix: Option<&'static str>, + prefixes: Vec<&'static str>, include: bool, } impl Filter for TargetPrefixFilter { fn enabled(&self, metadata: &tracing::Metadata<'_>, _ctx: &Context<'_, S>) -> bool { let matches_prefix = self - .prefix - .is_some_and(|prefix| metadata.target().starts_with(prefix)); + .prefixes + .iter() + .any(|prefix| metadata.target().starts_with(prefix)); metadata.is_span() && !metadata.target().starts_with("opentelemetry") && (matches_prefix == self.include) @@ -237,13 +266,25 @@ pub fn layer_for_target_prefix( instrumentation_scope: &'static str, target_prefix: &'static str, ) -> TargetOtlpLayer +where + S: Subscriber + for<'span> LookupSpan<'span>, +{ + layer_for_target_prefixes(provider, instrumentation_scope, [target_prefix]) +} + +/// Build a tracing layer that exports only spans from `target_prefixes`. +pub fn layer_for_target_prefixes( + provider: &SdkTracerProvider, + instrumentation_scope: &'static str, + target_prefixes: impl IntoIterator, +) -> TargetOtlpLayer where S: Subscriber + for<'span> LookupSpan<'span>, { tracing_opentelemetry::layer() .with_tracer(provider.tracer(instrumentation_scope)) .with_filter(TargetPrefixFilter { - prefix: Some(target_prefix), + prefixes: target_prefixes.into_iter().collect(), include: true, }) } @@ -256,13 +297,27 @@ pub fn layer_excluding_target_prefix( instrumentation_scope: &'static str, target_prefix: Option<&'static str>, ) -> TargetOtlpLayer +where + S: Subscriber + for<'span> LookupSpan<'span>, +{ + layer_excluding_target_prefixes(provider, instrumentation_scope, target_prefix) +} + +/// Build a tracing layer that excludes spans from `target_prefixes`. +/// +/// An empty iterator excludes no application spans. +pub fn layer_excluding_target_prefixes( + provider: &SdkTracerProvider, + instrumentation_scope: &'static str, + target_prefixes: impl IntoIterator, +) -> TargetOtlpLayer where S: Subscriber + for<'span> LookupSpan<'span>, { tracing_opentelemetry::layer() .with_tracer(provider.tracer(instrumentation_scope)) .with_filter(TargetPrefixFilter { - prefix: target_prefix, + prefixes: target_prefixes.into_iter().collect(), include: false, }) } diff --git a/crates/openshell-policy/src/ambiguity.rs b/crates/openshell-policy/src/ambiguity.rs index 5c2b9d917..2c0b1b194 100644 --- a/crates/openshell-policy/src/ambiguity.rs +++ b/crates/openshell-policy/src/ambiguity.rs @@ -181,12 +181,6 @@ fn connection_conflicts(left: &NetworkEndpoint, right: &NetworkEndpoint) -> Vec< &normalized_strings(&left.allowed_ips), &normalized_strings(&right.allowed_ips), ); - push_conflict( - &mut conflicts, - "advisor_proposed", - &left.advisor_proposed, - &right.advisor_proposed, - ); conflicts } @@ -787,6 +781,15 @@ mod tests { assert!(find_endpoint_ambiguities(&policy_with(left, right)).is_empty()); } + #[test] + fn advisor_provenance_does_not_make_endpoints_ambiguous() { + let explicit = endpoint("api.example.com", 443); + let mut proposed = explicit.clone(); + proposed.advisor_proposed = true; + + assert!(find_endpoint_ambiguities(&policy_with(explicit, proposed)).is_empty()); + } + #[test] fn plain_l4_endpoint_does_not_compete_with_l7_endpoint_metadata() { let left = endpoint("api.example.com", 443); diff --git a/crates/openshell-policy/src/l7_validate.rs b/crates/openshell-policy/src/l7_validate.rs index d60491c4b..fe07df123 100644 --- a/crates/openshell-policy/src/l7_validate.rs +++ b/crates/openshell-policy/src/l7_validate.rs @@ -49,6 +49,26 @@ pub fn is_explicit_tcp_protocol(protocol: &str) -> bool { protocol.eq_ignore_ascii_case("tcp") } +/// Reject transport choices that an in-sandbox agent must not grant itself. +/// +/// An omitted protocol remains allowed: it uses the established explicit +/// proxy, which canonicalizes forward HTTP authorities and terminates TLS by +/// default. Native transparent TCP and `tls: skip` bypass those application +/// authority checks, so only an administrator may author them directly. +pub fn agent_authored_transport_rejection(protocol: &str, tls: &str) -> Option<&'static str> { + if is_explicit_tcp_protocol(protocol) { + return Some( + "agent-authored proposals cannot request protocol tcp; ask an administrator to add native TCP access explicitly", + ); + } + if tls.eq_ignore_ascii_case("skip") { + return Some( + "agent-authored proposals cannot request tls: skip; ask an administrator to add raw TLS access explicitly", + ); + } + None +} + /// Reject additional L7-only fields represented outside /// [`L7EndpointFields`] by the runtime and provider-profile schemas. /// @@ -69,6 +89,24 @@ pub fn validate_explicit_tcp_additional_fields( )] } +#[cfg(test)] +mod agent_transport_tests { + use super::agent_authored_transport_rejection; + + #[test] + fn omitted_protocol_with_default_tls_remains_available_to_agents() { + assert_eq!(agent_authored_transport_rejection("", ""), None); + } + + #[test] + fn agent_cannot_request_native_tcp_or_skip_tls_inspection() { + assert!(agent_authored_transport_rejection("tcp", "").is_some()); + assert!(agent_authored_transport_rejection("TCP", "terminate").is_some()); + assert!(agent_authored_transport_rejection("", "skip").is_some()); + assert!(agent_authored_transport_rejection("rest", "SKIP").is_some()); + } +} + /// Fields extracted from an endpoint definition needed for L7 semantic /// validation. Both profile lint and the runtime validator construct this /// from their own data representation. diff --git a/crates/openshell-policy/src/lib.rs b/crates/openshell-policy/src/lib.rs index 5be8173e3..fe644c3d6 100644 --- a/crates/openshell-policy/src/lib.rs +++ b/crates/openshell-policy/src/lib.rs @@ -14,7 +14,7 @@ mod l7_validate; mod merge; mod middleware; -use std::collections::{BTreeMap, HashMap}; +use std::collections::{BTreeMap, BTreeSet, HashMap}; use std::fmt; use std::net::IpAddr; use std::path::Path; @@ -25,20 +25,21 @@ pub use ambiguity::{EndpointAmbiguity, find_endpoint_ambiguities}; use hickory_proto::rr::Name; use miette::{IntoDiagnostic, Result, WrapErr}; +use openshell_core::mcp::{DEFAULT_MCP_PROTOCOL_VERSION, McpProtocolVersion}; use openshell_core::proto::{ FilesystemPolicy, GraphqlOperation, L7Allow, L7DenyRule, L7QueryMatcher, L7Rule, LandlockPolicy, McpOptions, NetworkBinary, NetworkEndpoint, NetworkPolicyRule, ProcessPolicy, SandboxPolicy, }; -use serde::{Deserialize, Serialize}; +use serde::{Deserialize, Deserializer, Serialize}; pub use compose::{ PROVIDER_RULE_NAME_PREFIX, ProviderPolicyLayer, compose_effective_policy, is_provider_rule_name, provider_rule_name, strip_provider_rule_names, }; pub use l7_validate::{ - L7EndpointFields, L7Protocol, validate_explicit_tcp_additional_fields, - validate_l7_endpoint_semantics, + L7EndpointFields, L7Protocol, agent_authored_transport_rejection, + validate_explicit_tcp_additional_fields, validate_l7_endpoint_semantics, }; pub use merge::{ PolicyMergeError, PolicyMergeOp, PolicyMergeResult, PolicyMergeWarning, @@ -79,11 +80,19 @@ struct FilesystemDef { read_write: Vec, } +#[derive(Debug, Default, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +enum LandlockCompatibilityDef { + #[default] + BestEffort, + HardRequirement, +} + #[derive(Debug, Serialize, Deserialize)] #[serde(deny_unknown_fields)] struct LandlockDef { - #[serde(default, skip_serializing_if = "String::is_empty")] - compatibility: String, + #[serde(default)] + compatibility: LandlockCompatibilityDef, } #[derive(Debug, Serialize, Deserialize)] @@ -173,7 +182,11 @@ struct NetworkEndpointDef { credential_binding: Option, #[serde(default, skip_serializing_if = "Option::is_none")] json_rpc: Option, - #[serde(default, skip_serializing_if = "Option::is_none")] + #[serde( + default, + deserialize_with = "deserialize_non_null_optional_field", + skip_serializing_if = "Option::is_none" + )] mcp: Option, } @@ -195,6 +208,19 @@ fn is_zero_u32(v: &u32) -> bool { *v == 0 } +fn deserialize_non_null_optional_field<'de, D, T>( + deserializer: D, +) -> std::result::Result, D::Error> +where + D: Deserializer<'de>, + T: Deserialize<'de>, +{ + // Serde skips this function when the field is absent because the field has + // `default`. When it is present, deserialize `T` directly so an explicit + // YAML or JSON null is rejected instead of collapsing into omission. + T::deserialize(deserializer).map(Some) +} + #[derive(Debug, Serialize, Deserialize)] #[serde(deny_unknown_fields)] struct JsonRpcConfigDef { @@ -212,6 +238,15 @@ fn json_rpc_config_from_proto(max_body_bytes: u32) -> Option { #[derive(Debug, Serialize, Deserialize)] #[serde(deny_unknown_fields)] struct McpConfigDef { + // Presence is retained until authored-policy validation so an omitted + // allowlist can select the pinned default while an explicit empty list is + // rejected as an authoring mistake. + #[serde( + default, + deserialize_with = "deserialize_non_null_optional_field", + skip_serializing_if = "Option::is_none" + )] + versions: Option>, #[serde(default, skip_serializing_if = "is_zero_u32")] max_body_bytes: u32, #[serde(default, skip_serializing_if = "Option::is_none")] @@ -220,15 +255,49 @@ struct McpConfigDef { allow_all_known_mcp_methods: Option, } +const MCP_VERSION_REMEDIATION: &str = "omit mcp.versions to use the pinned default revision, use an exact supported revision, or omit protocol and mcp for deliberate uninspected L4 passthrough only when that weaker boundary is acceptable"; + +fn validate_authored_mcp_versions(versions: Option<&[String]>, context: &str) -> Result<()> { + let Some(versions) = versions else { + return Ok(()); + }; + if versions.is_empty() { + return Err(miette::miette!( + "{context} has an empty mcp.versions list; omit mcp.versions to use the pinned default revision" + )); + } + + let mut seen = BTreeSet::new(); + for value in versions { + let version = value + .parse::() + .map_err(|error| miette::miette!("{context}: {error}; {MCP_VERSION_REMEDIATION}"))?; + if !seen.insert(version) { + return Err(miette::miette!( + "{context} has duplicate protocol version '{value}'" + )); + } + } + Ok(()) +} + fn mcp_config_from_proto(max_body_bytes: u32, mcp: Option<&McpOptions>) -> Option { + let mut versions = mcp + .map(|config| config.versions.clone()) + .unwrap_or_default(); + canonicalize_mcp_versions(&mut versions); let strict_tool_names = mcp.and_then(|config| config.strict_tool_names); let allow_all_known_mcp_methods = mcp.and_then(|config| config.allow_all_known_mcp_methods); - (max_body_bytes > 0 || strict_tool_names.is_some() || allow_all_known_mcp_methods.is_some()) - .then_some(McpConfigDef { - max_body_bytes, - strict_tool_names, - allow_all_known_mcp_methods, - }) + (!versions.is_empty() + || max_body_bytes > 0 + || strict_tool_names.is_some() + || allow_all_known_mcp_methods.is_some()) + .then_some(McpConfigDef { + versions: Some(versions), + max_body_bytes, + strict_tool_names, + allow_all_known_mcp_methods, + }) } /// Nested L7 config stanzas accepted by the YAML policy schema. @@ -255,7 +324,9 @@ impl L7ConfigStanza { /// /// The stanza schema stays tied to this crate's canonical serde definitions, so /// adding a new supported field requires updating this conversion next to the -/// type that parses it. +/// type that parses it. MCP revision fields are validated and materialized +/// before the alias is flattened so both runtime ingress paths receive the +/// same canonical allowlist. pub fn l7_config_alias_runtime_fields( stanza: L7ConfigStanza, value: serde_json::Value, @@ -271,13 +342,19 @@ pub fn l7_config_alias_runtime_fields( Ok(fields) } L7ConfigStanza::Mcp => { + let config: McpConfigDef = serde_json::from_value(value) + .map_err(|error| miette::miette!("invalid mcp config: {error}"))?; + validate_authored_mcp_versions(config.versions.as_deref(), "invalid mcp config")?; let McpConfigDef { + versions, max_body_bytes, strict_tool_names, allow_all_known_mcp_methods, - } = serde_json::from_value(value) - .map_err(|error| miette::miette!("invalid mcp config: {error}"))?; + } = config; + let mut versions = versions.unwrap_or_else(default_mcp_versions); + canonicalize_mcp_versions(&mut versions); let mut fields = Vec::new(); + fields.push(("mcp_versions", serde_json::json!(versions))); if max_body_bytes > 0 { fields.push(("json_rpc_max_body_bytes", serde_json::json!(max_body_bytes))); } @@ -580,21 +657,24 @@ fn json_rpc_max_body_bytes(json_rpc: &Option, mcp: &Option) -> Option { - mcp.as_ref().and_then(|config| config.strict_tool_names) -} +fn mcp_options(protocol: &str, mcp: &Option) -> Option { + if !is_mcp_protocol(protocol) { + return None; + } -fn mcp_allow_all_known_mcp_methods(mcp: &Option) -> Option { - mcp.as_ref() - .and_then(|config| config.allow_all_known_mcp_methods) -} - -fn mcp_options(mcp: &Option) -> Option { - let strict_tool_names = mcp_strict_tool_names(mcp); - let allow_all_known_mcp_methods = mcp_allow_all_known_mcp_methods(mcp); - (strict_tool_names.is_some() || allow_all_known_mcp_methods.is_some()).then_some(McpOptions { - strict_tool_names, - allow_all_known_mcp_methods, + let mut versions = mcp + .as_ref() + .and_then(|config| config.versions.clone()) + .unwrap_or_else(default_mcp_versions); + // Authored YAML is validated before this conversion. Sort only after that + // boundary so duplicate or unsupported input cannot be hidden. + canonicalize_mcp_versions(&mut versions); + Some(McpOptions { + strict_tool_names: mcp.as_ref().and_then(|config| config.strict_tool_names), + allow_all_known_mcp_methods: mcp + .as_ref() + .and_then(|config| config.allow_all_known_mcp_methods), + versions, }) } @@ -602,6 +682,37 @@ fn is_mcp_protocol(protocol: &str) -> bool { protocol.eq_ignore_ascii_case("mcp") } +fn default_mcp_versions() -> Vec { + // The single pinned constant is intentionally independent of the registry + // size and order, so adding support for a revision cannot widen an + // existing versionless policy. + vec![DEFAULT_MCP_PROTOCOL_VERSION.as_str().to_string()] +} + +fn canonicalize_mcp_versions(versions: &mut [String]) { + // Unknown and duplicate values remain present so canonicalization cannot + // erase evidence that the raw policy was invalid. + versions.sort_by(|left, right| { + match ( + left.parse::(), + right.parse::(), + ) { + (Ok(left), Ok(right)) => left.cmp(&right), + (Ok(_), Err(_)) => std::cmp::Ordering::Less, + (Err(_), Ok(_)) => std::cmp::Ordering::Greater, + (Err(_), Err(_)) => left.cmp(right), + } + }); +} + +/// Sort one protobuf MCP contract without hiding invalid input. +/// +/// Exact supported revisions use semantic catalog order. Duplicate and +/// unsupported identifiers remain present for subsequent validation. +pub(crate) fn canonicalize_mcp_options(options: &mut McpOptions) { + canonicalize_mcp_versions(&mut options.versions); +} + fn split_tool_param( protocol: &str, params: BTreeMap, @@ -790,7 +901,7 @@ fn to_proto(raw: PolicyFile) -> Result { } }), json_rpc_max_body_bytes: json_rpc_max_body_bytes(&e.json_rpc, &e.mcp), - mcp: mcp_options(&e.mcp), + mcp: mcp_options(&protocol, &e.mcp), } }) .collect(), @@ -815,7 +926,10 @@ fn to_proto(raw: PolicyFile) -> Result { read_write: fs.read_write, }), landlock: raw.landlock.map(|ll| LandlockPolicy { - compatibility: ll.compatibility, + compatibility: match ll.compatibility { + LandlockCompatibilityDef::BestEffort => "best_effort".to_string(), + LandlockCompatibilityDef::HardRequirement => "hard_requirement".to_string(), + }, }), process: raw.process.map(|p| ProcessPolicy { run_as_user: p.run_as_user, @@ -830,16 +944,28 @@ fn to_proto(raw: PolicyFile) -> Result { // Proto → YAML conversion // --------------------------------------------------------------------------- -fn from_proto(policy: &SandboxPolicy) -> PolicyFile { +fn from_proto(policy: &SandboxPolicy) -> Result { let filesystem_policy = policy.filesystem.as_ref().map(|fs| FilesystemDef { include_workdir: fs.include_workdir, read_only: fs.read_only.clone(), read_write: fs.read_write.clone(), }); - let landlock = policy.landlock.as_ref().map(|ll| LandlockDef { - compatibility: ll.compatibility.clone(), - }); + let landlock = match policy.landlock.as_ref() { + Some(ll) => { + let compatibility = match ll.compatibility.as_str() { + "hard_requirement" => LandlockCompatibilityDef::HardRequirement, + "best_effort" | "" => LandlockCompatibilityDef::BestEffort, + otherwise => miette::bail!( + "invalid landlock.compatibility {:?}; accepted: {}", + otherwise, + openshell_core::policy::LANDLOCK_COMPATIBILITY_VALUES.join(", ") + ), + }; + Some(LandlockDef { compatibility }) + } + _ => None, + }; let process = policy.process.as_ref().and_then(|p| { if p.run_as_user.is_empty() && p.run_as_group.is_empty() { @@ -963,14 +1089,14 @@ fn from_proto(policy: &SandboxPolicy) -> PolicyFile { let network_middlewares = middleware::from_proto(&policy.network_middlewares); - PolicyFile { + Ok(PolicyFile { version: policy.version, filesystem_policy, landlock, process, network_policies, network_middlewares, - } + }) } // --------------------------------------------------------------------------- @@ -1022,11 +1148,46 @@ pub fn is_valid_sandbox_identity(value: &str) -> bool { // Public API // --------------------------------------------------------------------------- +// Validate raw authored values and their relationship to the endpoint protocol +// before conversion. Keeping validation outside the Serde error wrapper makes +// actionable MCP diagnostics the top-level user-facing error. +fn validate_mcp_version_schema(policy: &PolicyFile) -> Result<()> { + for (policy_key, rule) in &policy.network_policies { + let policy_name = if rule.name.is_empty() { + policy_key + } else { + &rule.name + }; + for endpoint in &rule.endpoints { + if is_mcp_protocol(&endpoint.protocol) { + let context = format!( + "network policy '{policy_name}': MCP endpoint '{}'", + endpoint.host + ); + validate_authored_mcp_versions( + endpoint + .mcp + .as_ref() + .and_then(|config| config.versions.as_deref()), + &context, + )?; + } else if endpoint.mcp.is_some() { + return Err(miette::miette!( + "network policy '{policy_name}': non-MCP endpoint '{}' cannot configure mcp options", + endpoint.host + )); + } + } + } + Ok(()) +} + /// Parse a sandbox policy from a YAML string. pub fn parse_sandbox_policy(yaml: &str) -> Result { let raw: PolicyFile = serde_yml::from_str(yaml) .into_diagnostic() .wrap_err("failed to parse sandbox policy YAML")?; + validate_mcp_version_schema(&raw)?; to_proto(raw) } @@ -1036,7 +1197,9 @@ pub fn parse_sandbox_policy(yaml: &str) -> Result { /// canonical YAML field names (e.g. `filesystem_policy`, not `filesystem`) /// and is round-trippable through `parse_sandbox_policy`. pub fn serialize_sandbox_policy(policy: &SandboxPolicy) -> Result { - let yaml_repr = from_proto(policy); + let canonical = validate_and_canonicalize_mcp_policy_schema(policy.clone()) + .map_err(|error| miette::miette!("cannot serialize invalid sandbox policy: {error}"))?; + let yaml_repr = from_proto(&canonical)?; serde_yml::to_string(&yaml_repr) .into_diagnostic() .wrap_err("failed to serialize policy to YAML") @@ -1047,7 +1210,9 @@ pub fn serialize_sandbox_policy(policy: &SandboxPolicy) -> Result { /// The shape mirrors the YAML schema used by [`serialize_sandbox_policy`], so /// automation can use the same documented field names in either format. pub fn sandbox_policy_to_json_value(policy: &SandboxPolicy) -> Result { - let json_repr = from_proto(policy); + let canonical = validate_and_canonicalize_mcp_policy_schema(policy.clone()) + .map_err(|error| miette::miette!("cannot serialize invalid sandbox policy: {error}"))?; + let json_repr = from_proto(&canonical)?; serde_json::to_value(&json_repr) .into_diagnostic() .wrap_err("failed to serialize policy to JSON") @@ -1106,7 +1271,7 @@ pub const LEGACY_CONTAINER_POLICY_PATH: &str = "/etc/navigator/policy.yaml"; /// This policy grants filesystem access to standard system paths, leaves /// process identity selection to the compute runtime, enables Landlock in /// best-effort mode, and **blocks all network access** (no network policies, -/// no inference routing). +/// and no provider-derived endpoint access). pub fn restrictive_default_policy() -> SandboxPolicy { SandboxPolicy { version: 1, @@ -1230,6 +1395,28 @@ pub enum PolicyViolation { policy_name: String, host: String, }, + /// `landlock.compatibility` has an unrecognized value. + InvalidLandlockCompatibility { value: String }, + /// An effective MCP endpoint has not materialized a protocol revision. + MissingMcpVersions { policy_name: String, host: String }, + /// A non-MCP endpoint carries MCP-only configuration. + McpOptionsOnNonMcpEndpoint { + policy_name: String, + host: String, + protocol: String, + }, + /// An MCP revision allowlist contains an unsupported exact identifier. + UnsupportedMcpVersion { + policy_name: String, + host: String, + version: String, + }, + /// An MCP revision allowlist contains the same identifier more than once. + DuplicateMcpVersion { + policy_name: String, + host: String, + version: String, + }, } impl fmt::Display for PolicyViolation { @@ -1392,6 +1579,49 @@ impl fmt::Display for PolicyViolation { '{policy_name}' tls: skip endpoint '{host}'" ) } + Self::InvalidLandlockCompatibility { value } => { + write!( + f, + "invalid landlock.compatibility '{value}'; accepted: {}", + openshell_core::policy::LANDLOCK_COMPATIBILITY_VALUES.join(", ") + ) + } + Self::MissingMcpVersions { policy_name, host } => { + write!( + f, + "network policy '{policy_name}': MCP endpoint '{host}' has no materialized protocol version" + ) + } + Self::McpOptionsOnNonMcpEndpoint { + policy_name, + host, + protocol, + } => { + write!( + f, + "network policy '{policy_name}': endpoint '{host}' uses protocol '{protocol}' and cannot configure mcp options" + ) + } + Self::UnsupportedMcpVersion { + policy_name, + host, + version, + } => { + write!( + f, + "network policy '{policy_name}': MCP endpoint '{host}' has unsupported protocol version '{version}'; {MCP_VERSION_REMEDIATION}" + ) + } + Self::DuplicateMcpVersion { + policy_name, + host, + version, + } => { + write!( + f, + "network policy '{policy_name}': MCP endpoint '{host}' repeats protocol version '{version}'" + ) + } } } } @@ -1413,8 +1643,24 @@ impl fmt::Display for PolicyViolation { /// - Middleware names, implementations, failure modes, selectors, and built-in /// configurations must be valid /// - Middleware selectors must not match endpoints that skip TLS inspection +/// - MCP endpoints must carry a nonempty, unique allowlist of exact supported +/// revisions +/// - Non-MCP endpoints must not carry MCP options pub fn validate_sandbox_policy( policy: &SandboxPolicy, +) -> std::result::Result<(), Vec> { + validate_sandbox_policy_with_mcp_presence(policy, McpVersionPresence::RequireMaterialized) +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum McpVersionPresence { + RequireMaterialized, + AllowDefaultable, +} + +fn validate_sandbox_policy_with_mcp_presence( + policy: &SandboxPolicy, + mcp_version_presence: McpVersionPresence, ) -> std::result::Result<(), Vec> { let mut violations = Vec::new(); @@ -1436,6 +1682,17 @@ pub fn validate_sandbox_policy( } } + // Check landlock compatibility mode is a recognized value. Direct gRPC/SDK + // clients bypass YAML serde validation, so reject invalid values here at the + // gateway create path rather than deferring rejection to sandbox startup. + if let Some(ref landlock) = policy.landlock + && !openshell_core::policy::is_valid_landlock_compatibility(&landlock.compatibility) + { + violations.push(PolicyViolation::InvalidLandlockCompatibility { + value: landlock.compatibility.clone(), + }); + } + // Check filesystem paths if let Some(ref fs) = policy.filesystem { let total_paths = fs.read_only.len() + fs.read_write.len(); @@ -1482,8 +1739,13 @@ pub fn validate_sandbox_policy( } } + // Protobuf maps do not preserve iteration order. Sort rule keys so callers + // receive stable validation errors for equivalent policy inputs. + let mut network_policies: Vec<_> = policy.network_policies.iter().collect(); + network_policies.sort_by_key(|(name, _)| *name); + // Check network policy endpoint hosts for TLD wildcards. - for (key, rule) in &policy.network_policies { + for (key, rule) in network_policies { let name = if rule.name.is_empty() { key.clone() } else { @@ -1649,9 +1911,6 @@ pub fn validate_sandbox_policy( "SQL enforcement requires full SQL parsing; use enforcement: audit".to_string(), ); } - if ep.mcp.is_some() && ep.protocol != "mcp" { - l7_errors.push("mcp options are only valid for protocol mcp".to_string()); - } if ep.protocol == "graphql" { for (rule_index, rule) in ep.rules.iter().enumerate() { let operation_type = rule @@ -1693,6 +1952,7 @@ pub fn validate_sandbox_policy( reason, } })); + collect_mcp_endpoint_violations(&name, ep, mcp_version_presence, &mut violations); } } @@ -1757,6 +2017,186 @@ fn validate_tcp_dns_host_selector(host: &str) -> std::result::Result<(), String> Ok(()) } +fn collect_mcp_endpoint_violations( + policy_name: &str, + endpoint: &NetworkEndpoint, + mcp_version_presence: McpVersionPresence, + violations: &mut Vec, +) { + if is_mcp_protocol(&endpoint.protocol) { + let Some(mcp) = endpoint.mcp.as_ref() else { + if mcp_version_presence == McpVersionPresence::RequireMaterialized { + violations.push(PolicyViolation::MissingMcpVersions { + policy_name: policy_name.to_string(), + host: endpoint.host.clone(), + }); + } + return; + }; + if mcp.versions.is_empty() { + if mcp_version_presence == McpVersionPresence::RequireMaterialized { + violations.push(PolicyViolation::MissingMcpVersions { + policy_name: policy_name.to_string(), + host: endpoint.host.clone(), + }); + } + return; + } + + let mut seen = BTreeSet::new(); + for version in &mcp.versions { + if !seen.insert(version.as_str()) { + violations.push(PolicyViolation::DuplicateMcpVersion { + policy_name: policy_name.to_string(), + host: endpoint.host.clone(), + version: version.clone(), + }); + } + if version.parse::().is_err() { + violations.push(PolicyViolation::UnsupportedMcpVersion { + policy_name: policy_name.to_string(), + host: endpoint.host.clone(), + version: version.clone(), + }); + } + } + } else if endpoint.mcp.is_some() { + violations.push(PolicyViolation::McpOptionsOnNonMcpEndpoint { + policy_name: policy_name.to_string(), + host: endpoint.host.clone(), + protocol: endpoint.protocol.clone(), + }); + } +} + +fn validate_mcp_policy_schema( + policy: &SandboxPolicy, + mcp_version_presence: McpVersionPresence, +) -> std::result::Result<(), Vec> { + let mut violations = Vec::new(); + let mut network_policies: Vec<_> = policy.network_policies.iter().collect(); + network_policies.sort_by_key(|(name, _)| *name); + for (key, rule) in network_policies { + let policy_name = if rule.name.is_empty() { + key.as_str() + } else { + rule.name.as_str() + }; + for endpoint in &rule.endpoints { + collect_mcp_endpoint_violations( + policy_name, + endpoint, + mcp_version_presence, + &mut violations, + ); + } + } + if violations.is_empty() { + Ok(()) + } else { + Err(violations) + } +} + +/// Error returned when a checked policy boundary receives invalid protobuf state. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct PolicyValidationError { + violations: Vec, +} + +impl PolicyValidationError { + /// Borrow every violation found while validating the policy. + #[must_use] + pub fn violations(&self) -> &[PolicyViolation] { + &self.violations + } + + /// Consume the error and return every violation. + #[must_use] + pub fn into_violations(self) -> Vec { + self.violations + } +} + +impl fmt::Display for PolicyValidationError { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + write!(formatter, "sandbox policy validation failed")?; + for violation in &self.violations { + write!(formatter, "; {violation}")?; + } + Ok(()) + } +} + +impl std::error::Error for PolicyValidationError {} + +fn validate_and_canonicalize_mcp_policy_schema( + mut policy: SandboxPolicy, +) -> std::result::Result { + validate_mcp_policy_schema(&policy, McpVersionPresence::AllowDefaultable) + .map_err(|violations| PolicyValidationError { violations })?; + materialize_default_mcp_versions(&mut policy); + canonicalize_mcp_version_allowlists(&mut policy); + validate_mcp_policy_schema(&policy, McpVersionPresence::RequireMaterialized) + .map_err(|violations| PolicyValidationError { violations })?; + Ok(policy) +} + +/// Validate raw protobuf policy state, then return its canonical representation. +/// +/// Validation deliberately runs before default materialization and sorting so +/// malformed, duplicate, or misplaced MCP state cannot be repaired or hidden +/// by normalization. Missing or empty protobuf MCP options select the pinned +/// default because proto3 repeated fields do not preserve authoring presence. +/// Canonicalization only inserts the known pinned default and sorts allowlists +/// that the first pass already proved valid, so it cannot invalidate another +/// policy field. Debug builds recheck that local transformation invariant +/// without imposing a second whole-policy traversal on production read paths. +pub fn validate_and_canonicalize_sandbox_policy( + mut policy: SandboxPolicy, +) -> std::result::Result { + validate_sandbox_policy_with_mcp_presence(&policy, McpVersionPresence::AllowDefaultable) + .map_err(|violations| PolicyValidationError { violations })?; + materialize_default_mcp_versions(&mut policy); + canonicalize_mcp_version_allowlists(&mut policy); + debug_assert!( + validate_sandbox_policy(&policy).is_ok(), + "validated MCP canonicalization must preserve every policy invariant" + ); + Ok(policy) +} + +/// Replace absent protobuf MCP options and empty revision lists with the +/// single pinned policy default while preserving every explicit MCP option. +pub(crate) fn materialize_default_mcp_versions(policy: &mut SandboxPolicy) { + for rule in policy.network_policies.values_mut() { + for endpoint in &mut rule.endpoints { + if !is_mcp_protocol(&endpoint.protocol) { + continue; + } + let options = endpoint.mcp.get_or_insert_default(); + if options.versions.is_empty() { + options.versions = default_mcp_versions(); + } + } + } +} + +/// Canonicalize every MCP revision allowlist without deleting invalid input. +/// +/// This helper is crate-private because external callers should use +/// [`validate_and_canonicalize_sandbox_policy`] and cannot safely normalize an +/// unvalidated policy in place. +pub(crate) fn canonicalize_mcp_version_allowlists(policy: &mut SandboxPolicy) { + for rule in policy.network_policies.values_mut() { + for endpoint in &mut rule.endpoints { + if let Some(options) = endpoint.mcp.as_mut() { + canonicalize_mcp_options(options); + } + } + } +} + /// Truncate a string for safe inclusion in error messages. fn truncate_for_display(s: &str) -> String { if s.len() <= 80 { @@ -2155,6 +2595,7 @@ network_policies: let fields = l7_config_alias_runtime_fields( L7ConfigStanza::Mcp, serde_json::json!({ + "versions": ["2025-11-25", "2025-03-26", "2025-06-18"], "max_body_bytes": 131_072, "strict_tool_names": false, "allow_all_known_mcp_methods": true @@ -2165,12 +2606,32 @@ network_policies: assert_eq!( fields, vec![ + ( + "mcp_versions", + serde_json::json!(["2025-03-26", "2025-06-18", "2025-11-25"]) + ), ("json_rpc_max_body_bytes", serde_json::json!(131_072)), ("mcp_strict_tool_names", serde_json::json!(false)), ("mcp_allow_all_known_mcp_methods", serde_json::json!(true)), ] ); + let runtime_only_fields = l7_config_alias_runtime_fields( + L7ConfigStanza::Mcp, + serde_json::json!({"strict_tool_names": false}), + ) + .expect("runtime alias parsing supplies the pinned wire profile"); + assert_eq!( + runtime_only_fields, + vec![ + ( + "mcp_versions", + serde_json::json!([DEFAULT_MCP_PROTOCOL_VERSION.as_str()]) + ), + ("mcp_strict_tool_names", serde_json::json!(false)) + ] + ); + let err = l7_config_alias_runtime_fields( L7ConfigStanza::JsonRpc, serde_json::json!({"on_parse_error": "allow"}), @@ -2179,6 +2640,426 @@ network_policies: assert!(err.to_string().contains("on_parse_error")); } + #[test] + fn l7_mcp_alias_rejects_invalid_raw_version_authoring() { + let cases = [ + ("null config", serde_json::Value::Null), + ("null versions", serde_json::json!({"versions": null})), + ("empty versions", serde_json::json!({"versions": []})), + ( + "duplicate revision", + serde_json::json!({"versions": ["2025-11-25", "2025-11-25"]}), + ), + ( + "unsupported revision", + serde_json::json!({"versions": ["2025-11-26"]}), + ), + ( + "leading whitespace", + serde_json::json!({"versions": [" 2025-11-25"]}), + ), + ( + "trailing whitespace", + serde_json::json!({"versions": ["2025-11-25 "]}), + ), + ]; + + for (case, value) in cases { + let error = l7_config_alias_runtime_fields(L7ConfigStanza::Mcp, value) + .expect_err("invalid raw MCP version authoring must fail"); + assert!( + error.to_string().contains("invalid mcp config"), + "{case} returned an unexpected diagnostic: {error}" + ); + } + } + + const MCP_VERSIONS: [&str; 3] = ["2025-03-26", "2025-06-18", "2025-11-25"]; + + fn mcp_version_options(versions: &[&str]) -> McpOptions { + McpOptions { + versions: versions.iter().map(ToString::to_string).collect(), + ..Default::default() + } + } + + fn mcp_version_policy(protocol: &str, mcp: Option) -> SandboxPolicy { + let mut policy = restrictive_default_policy(); + policy.network_policies.insert( + "versioned".to_string(), + NetworkPolicyRule { + name: "versioned".to_string(), + endpoints: vec![NetworkEndpoint { + host: "mcp.example.com".to_string(), + port: 443, + protocol: protocol.to_string(), + rules: vec![L7Rule { + allow: Some(L7Allow { + method: "tools/list".to_string(), + ..Default::default() + }), + }], + mcp, + ..Default::default() + }], + ..Default::default() + }, + ); + policy + } + + fn mcp_version_endpoint_yaml(protocol: &str, mcp_body: Option<&str>) -> String { + let mcp = mcp_body.map_or_else(String::new, |body| format!(" mcp:\n{body}")); + format!( + "version: 1\nnetwork_policies:\n versioned:\n endpoints:\n - host: mcp.example.com\n port: 443\n protocol: {protocol}\n{mcp}" + ) + } + + #[test] + fn mcp_version_profile_fixture_round_trips_in_canonical_order() { + let fixture = include_str!("../testdata/mcp-version-profiles.yaml"); + let policy = parse_sandbox_policy(fixture).expect("fixed MCP profile fixture must parse"); + let endpoint = &policy.network_policies["versioned_mcp"].endpoints[0]; + assert_eq!( + endpoint.mcp.as_ref().expect("MCP options").versions, + MCP_VERSIONS + ); + + let canonical = + serialize_sandbox_policy(&policy).expect("MCP profile fixture must serialize"); + let positions = + MCP_VERSIONS.map(|version| canonical.find(version).expect("serialized revision")); + assert!(positions.windows(2).all(|pair| pair[0] < pair[1])); + assert_eq!( + parse_sandbox_policy(&canonical).expect("canonical fixture must parse"), + policy + ); + } + + #[test] + fn omitted_authored_mcp_versions_materialize_the_pinned_default() { + let omitted_stanza = parse_sandbox_policy(&mcp_version_endpoint_yaml("mcp", None)) + .expect("an omitted MCP stanza must select the default revision"); + let explicit_default = parse_sandbox_policy(&mcp_version_endpoint_yaml( + "mcp", + Some(" versions: [\"2025-11-25\"]\n"), + )) + .expect("the explicit default revision must parse"); + assert_eq!(omitted_stanza, explicit_default); + + let omitted_versions = parse_sandbox_policy(&mcp_version_endpoint_yaml( + "mcp", + Some(" strict_tool_names: false\n"), + )) + .expect("an omitted versions key must select the default revision"); + let options = omitted_versions.network_policies["versioned"].endpoints[0] + .mcp + .as_ref() + .expect("MCP options must be materialized"); + assert_eq!(options.versions, default_mcp_versions()); + assert_eq!(options.strict_tool_names, Some(false)); + + let yaml = serialize_sandbox_policy(&omitted_stanza) + .expect("the materialized default must serialize"); + assert!(yaml.contains("versions:")); + assert!(yaml.contains(DEFAULT_MCP_PROTOCOL_VERSION.as_str())); + } + + #[test] + fn pinned_mcp_default_does_not_expand_to_the_supported_registry() { + assert_eq!(default_mcp_versions().len(), 1); + assert_eq!( + default_mcp_versions(), + [DEFAULT_MCP_PROTOCOL_VERSION.as_str()] + ); + assert!(McpProtocolVersion::ALL.len() > default_mcp_versions().len()); + } + + #[test] + fn mcp_version_yaml_rejects_explicit_empty_duplicate_unknown_and_misplaced_values() { + let cases = [ + ("null allowlist", "mcp", Some(" versions: null\n")), + ("empty allowlist", "mcp", Some(" versions: []\n")), + ( + "empty identifier", + "mcp", + Some(" versions: [\"\"]\n"), + ), + ( + "duplicate revision", + "mcp", + Some(" versions: [\"2025-03-26\", \"2025-03-26\"]\n"), + ), + ( + "unsupported revision", + "mcp", + Some(" versions: [\"2025-03-27\"]\n"), + ), + ( + "unsupported alias", + "mcp", + Some(" versions: [\"latest\"]\n"), + ), + ( + "moving draft alias", + "mcp", + Some(" versions: [\"draft\"]\n"), + ), + ( + "leading whitespace", + "mcp", + Some(" versions: [\" 2025-03-26\"]\n"), + ), + ( + "trailing whitespace", + "mcp", + Some(" versions: [\"2025-03-26 \"]\n"), + ), + ( + "non-MCP placement", + "json-rpc", + Some(" versions: [\"2025-03-26\"]\n"), + ), + ]; + + for (case, protocol, body) in cases { + let yaml = mcp_version_endpoint_yaml(protocol, body); + assert!(parse_sandbox_policy(&yaml).is_err(), "{case} must fail"); + } + + let mut null_mcp = mcp_version_endpoint_yaml("mcp", None); + null_mcp.push_str(" mcp: null\n"); + assert!( + parse_sandbox_policy(&null_mcp).is_err(), + "an explicit null MCP stanza must fail" + ); + } + + #[test] + fn mcp_version_proto_validation_preserves_raw_invalid_values() { + let cases = [ + ("empty identifier", vec![""]), + ("unsupported revision", vec!["2025-03-27"]), + ("unsupported alias", vec!["latest"]), + ("moving draft alias", vec!["draft"]), + ("leading whitespace", vec![" 2025-03-26"]), + ("trailing whitespace", vec!["2025-03-26 "]), + ]; + for (case, versions) in cases { + let policy = mcp_version_policy("mcp", Some(mcp_version_options(&versions))); + let violations = validate_sandbox_policy(&policy).expect_err(case); + assert!(violations.iter().any(|violation| matches!( + violation, + PolicyViolation::UnsupportedMcpVersion { version, .. } + if version == versions[0] + ))); + } + + let duplicate = mcp_version_policy( + "mcp", + Some(mcp_version_options(&["2025-03-26", "2025-03-26"])), + ); + assert!( + validate_sandbox_policy(&duplicate) + .expect_err("duplicate must fail") + .iter() + .any(|violation| matches!( + violation, + PolicyViolation::DuplicateMcpVersion { version, .. } + if version == "2025-03-26" + )) + ); + + for misplaced in [McpOptions::default(), mcp_version_options(&["2025-03-26"])] { + let policy = mcp_version_policy("rest", Some(misplaced)); + assert!(matches!( + validate_sandbox_policy(&policy) + .expect_err("misplaced options must fail") + .as_slice(), + [PolicyViolation::McpOptionsOnNonMcpEndpoint { .. }] + )); + } + } + + #[test] + fn protobuf_missing_and_empty_mcp_options_materialize_the_same_default() { + let missing = mcp_version_policy("mcp", None); + let empty = mcp_version_policy("mcp", Some(McpOptions::default())); + let explicit = mcp_version_policy( + "mcp", + Some(mcp_version_options( + &[DEFAULT_MCP_PROTOCOL_VERSION.as_str()], + )), + ); + + for raw in [&missing, &empty] { + assert!(matches!( + validate_sandbox_policy(raw) + .expect_err("borrowed validation requires canonical effective state") + .as_slice(), + [PolicyViolation::MissingMcpVersions { .. }] + )); + let canonical = validate_and_canonicalize_sandbox_policy(raw.clone()) + .expect("checked protobuf boundaries must materialize the default"); + assert_eq!(canonical, explicit); + assert_eq!( + canonical.network_policies["versioned"].endpoints[0] + .mcp + .as_ref() + .expect("MCP options must be materialized") + .versions, + default_mcp_versions() + ); + } + + assert_eq!( + serialize_sandbox_policy(&missing).expect("missing options must serialize"), + serialize_sandbox_policy(&explicit).expect("explicit options must serialize") + ); + let defaulted_json = + sandbox_policy_to_json_value(&empty).expect("empty options must serialize"); + assert_eq!( + defaulted_json, + sandbox_policy_to_json_value(&explicit).expect("explicit options must serialize") + ); + assert_eq!( + defaulted_json["network_policies"]["versioned"]["endpoints"][0]["mcp"]["versions"], + serde_json::json!([DEFAULT_MCP_PROTOCOL_VERSION.as_str()]) + ); + } + + #[test] + fn unsupported_mcp_version_errors_explain_the_explicit_l4_escape_hatch() { + const DEFAULT_REMEDIATION: &str = "omit mcp.versions to use the pinned default revision"; + const L4_REMEDIATION: &str = + "omit protocol and mcp for deliberate uninspected L4 passthrough"; + + let yaml = mcp_version_endpoint_yaml("mcp", Some(" versions: [\"draft\"]\n")); + let authored_diagnostic = parse_sandbox_policy(&yaml) + .expect_err("moving draft aliases must fail closed") + .to_string(); + assert!( + authored_diagnostic.contains(DEFAULT_REMEDIATION) + && authored_diagnostic.contains(L4_REMEDIATION), + "rendered diagnostic omitted remediation choices: {authored_diagnostic}" + ); + + let policy = mcp_version_policy("mcp", Some(mcp_version_options(&["2026-07-28"]))); + let violations = validate_sandbox_policy(&policy) + .expect_err("unsupported protobuf revisions must fail closed"); + assert!(violations.iter().map(ToString::to_string).any(|message| { + message.contains(DEFAULT_REMEDIATION) && message.contains(L4_REMEDIATION) + })); + } + + #[test] + fn mcp_version_canonicalization_preserves_invalid_and_duplicate_entries() { + let mut policy = mcp_version_policy( + "mcp", + Some(mcp_version_options(&[ + "unknown-z", + "2025-11-25", + "2025-03-26", + "2025-03-26", + "unknown-a", + ])), + ); + + canonicalize_mcp_version_allowlists(&mut policy); + + assert_eq!( + policy.network_policies["versioned"].endpoints[0] + .mcp + .as_ref() + .expect("MCP options") + .versions, + [ + "2025-03-26", + "2025-03-26", + "2025-11-25", + "unknown-a", + "unknown-z", + ] + ); + let violations = validate_sandbox_policy(&policy) + .expect_err("normalization must not repair invalid input"); + assert!( + violations + .iter() + .any(|violation| matches!(violation, PolicyViolation::DuplicateMcpVersion { .. })) + ); + assert!( + violations.iter().any(|violation| matches!( + violation, + PolicyViolation::UnsupportedMcpVersion { .. } + )) + ); + } + + #[test] + fn canonical_serializers_reject_invalid_mcp_policy_before_conversion() { + let invalid = [ + mcp_version_policy( + "mcp", + Some(mcp_version_options(&["2025-03-26", "2025-03-26"])), + ), + mcp_version_policy("mcp", Some(mcp_version_options(&["latest"]))), + mcp_version_policy("mcp", Some(mcp_version_options(&["2025-03-26 "]))), + mcp_version_policy("rest", Some(McpOptions::default())), + mcp_version_policy("json-rpc", Some(mcp_version_options(&["2025-03-26"]))), + ]; + + for policy in invalid { + assert!(serialize_sandbox_policy(&policy).is_err()); + assert!(sandbox_policy_to_json_value(&policy).is_err()); + assert!(serialize_sandbox_policy_json(&policy).is_err()); + } + } + + #[test] + fn canonical_serializers_do_not_enforce_unrelated_mutation_safety_rules() { + let mut policy = mcp_version_policy( + "mcp", + Some(mcp_version_options(&["2025-11-25", "2025-03-26"])), + ); + policy.process = Some(ProcessPolicy { + run_as_user: "root".to_string(), + run_as_group: "sandbox".to_string(), + }); + assert!(validate_sandbox_policy(&policy).is_err()); + + let yaml = serialize_sandbox_policy(&policy) + .expect("serialization must remain available for policy inspection"); + let first = yaml.find("2025-03-26").expect("first version"); + let second = yaml.find("2025-11-25").expect("second version"); + assert!(first < second); + assert!(sandbox_policy_to_json_value(&policy).is_ok()); + } + + #[test] + fn mcp_version_checked_canonicalization_returns_valid_semantic_order() { + let policy = mcp_version_policy( + "mcp", + Some(mcp_version_options(&[ + "2025-11-25", + "2025-03-26", + "2025-06-18", + ])), + ); + let canonical = validate_and_canonicalize_sandbox_policy(policy) + .expect("valid policy must canonicalize"); + + assert_eq!( + canonical.network_policies["versioned"].endpoints[0] + .mcp + .as_ref() + .expect("MCP options") + .versions, + MCP_VERSIONS + ); + assert!(validate_sandbox_policy(&canonical).is_ok()); + } + #[test] fn container_policy_path_is_expected() { assert_eq!(CONTAINER_POLICY_PATH, "/etc/openshell/policy.yaml"); @@ -2314,6 +3195,80 @@ network_policies: assert_eq!(violations.len(), 2); } + #[test] + fn parse_rejects_invalid_landlock_compatibility() { + let err = parse_sandbox_policy("version: 1\nlandlock:\n compatibility: bogus\n") + .expect_err("should reject invalid YAML enum value"); + let msg = format!("{err:?}"); + assert!( + msg.contains("best_effort") && msg.contains("hard_requirement"), + "error should list accepted values, got: {msg}", + ); + } + + #[test] + fn parse_accepts_known_landlock_compatibility() { + for value in ["best_effort", "hard_requirement"] { + let yaml = format!("version: 1\nlandlock:\n compatibility: {value}\n"); + let policy = parse_sandbox_policy(&yaml).expect("should parse"); + assert_eq!( + policy.landlock.as_ref().expect("landlock").compatibility, + value, + ); + } + } + + #[test] + fn validate_rejects_invalid_landlock_compatibility_proto() { + let mut policy = restrictive_default_policy(); + policy.landlock = Some(LandlockPolicy { + compatibility: "nope".into(), + }); + let violations = validate_sandbox_policy(&policy).unwrap_err(); + assert!( + violations + .iter() + .any(|v| matches!(v, PolicyViolation::InvalidLandlockCompatibility { .. })), + "expected InvalidLandlockCompatibility, got: {violations:?}", + ); + } + + #[test] + fn validate_accepts_empty_landlock_compatibility() { + // Empty string is the proto default and maps to best_effort. + let mut policy = restrictive_default_policy(); + policy.landlock = Some(LandlockPolicy { + compatibility: String::new(), + }); + assert!(validate_sandbox_policy(&policy).is_ok()); + } + + #[test] + fn serialize_rejects_invalid_landlock_compatibility() { + // Old policies persisted before gateway validation can hold invalid + // values; serialization must error rather than normalize to best_effort. + let mut policy = restrictive_default_policy(); + policy.landlock = Some(LandlockPolicy { + compatibility: "hard-requirement".to_string(), + }); + let err = serialize_sandbox_policy(&policy).expect_err("should reject"); + let msg = format!("{err:?}"); + assert!( + msg.contains("best_effort") && msg.contains("hard_requirement"), + "error should list accepted values, got: {msg}", + ); + } + + #[test] + fn serialize_accepts_empty_landlock_compatibility() { + // Empty string is the proto default; serialize must not error on it. + let mut policy = restrictive_default_policy(); + policy.landlock = Some(LandlockPolicy { + compatibility: String::new(), + }); + assert!(serialize_sandbox_policy(&policy).is_ok()); + } + #[test] fn validate_rejects_invalid_middleware_control_fields() { let cases = [ @@ -3823,6 +4778,7 @@ network_policies: protocol: mcp enforcement: enforce mcp: + versions: [2025-03-26] max_body_bytes: 131072 strict_tool_names: false rules: @@ -3874,6 +4830,7 @@ network_policies: port: 443 protocol: mcp mcp: + versions: [2025-03-26] max_body_bytes: 131072 strict_tool_names: false rules: diff --git a/crates/openshell-policy/src/merge.rs b/crates/openshell-policy/src/merge.rs index 12026bbea..e6d72d489 100644 --- a/crates/openshell-policy/src/merge.rs +++ b/crates/openshell-policy/src/merge.rs @@ -7,7 +7,10 @@ use openshell_core::proto::{ L7Allow, L7DenyRule, L7Rule, NetworkBinary, NetworkEndpoint, NetworkPolicyRule, SandboxPolicy, }; -use crate::is_provider_rule_name; +use crate::{ + PolicyViolation, canonicalize_mcp_options, is_provider_rule_name, restrictive_default_policy, + validate_and_canonicalize_sandbox_policy, +}; const DEFAULT_JSON_RPC_MAX_BODY_BYTES: u32 = 64 * 1024; @@ -45,9 +48,11 @@ pub fn canonicalize_advisor_add_rule( }) .cloned() .map(|mut endpoint| { - // This marker is derived from provider/credential context by the - // gateway and must never be persisted from an advisor proposal. + // Provenance does not change the endpoint contract. The gateway + // derives the credential marker, and the advisor marker records + // where a persisted endpoint came from. endpoint.provider_credentialed = false; + endpoint.advisor_proposed = false; // A denial observes one binary-to-port authorization. Preserve the // existing inspection contract, but never copy sibling ports from // a multi-port endpoint into the proposal. @@ -84,6 +89,7 @@ pub fn canonicalize_advisor_add_rule( .any(|endpoint| { let mut normalized = endpoint.clone(); normalized.provider_credentialed = false; + normalized.advisor_proposed = false; normalize_endpoint(&mut normalized); normalized == contract }) @@ -92,6 +98,12 @@ pub fn canonicalize_advisor_add_rule( .collect::>(); sandbox_owners.sort(); + let mut contract = contract; + if sandbox_owners.is_empty() { + // A provider-owned contract is mirrored into a new sandbox-owned + // advisor overlay, so retain the incoming proposal provenance. + contract.advisor_proposed = incoming_endpoint.advisor_proposed; + } let target_name = sandbox_owners .first() .cloned() @@ -265,6 +277,15 @@ pub const ANY_BINARY_SCOPE: &str = "any binary"; #[derive(Debug, Clone, PartialEq, Eq)] pub enum PolicyMergeError { + /// The current policy was invalid before any operation was considered. + InvalidInputPolicy { + violations: Vec, + }, + /// An `AddRule` operation carried an invalid raw policy fragment. + InvalidOperationPolicy { + operation_index: usize, + violations: Vec, + }, MissingRuleNameForAddRule, /// An `AddRule` operation has no endpoint authorization to merge. EmptyAddRuleEndpoints { @@ -334,6 +355,10 @@ pub enum PolicyMergeError { rule_name: String, binary_path: String, }, + /// Valid inputs produced an invalid output, indicating a merge invariant failure. + InvalidMergedPolicy { + violations: Vec, + }, InvalidEndpointReference { host: String, port: u32, @@ -365,6 +390,17 @@ pub enum PolicyMergeError { impl std::fmt::Display for PolicyMergeError { fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { match self { + Self::InvalidInputPolicy { violations } => { + write_policy_violations(f, "current policy is invalid", violations) + } + Self::InvalidOperationPolicy { + operation_index, + violations, + } => write_policy_violations( + f, + &format!("merge operation {operation_index} carries an invalid policy rule"), + violations, + ), Self::MissingRuleNameForAddRule => write!(f, "add-rule operation requires a rule name"), Self::EmptyAddRuleEndpoints { operation_index, @@ -439,6 +475,9 @@ impl std::fmt::Display for PolicyMergeError { f, "merge operation {operation_index} cannot remove binary '{binary_path}' from rule '{rule_name}' because the rule authorizes any binary; replace the policy with an explicit binary list or remove the rule" ), + Self::InvalidMergedPolicy { violations } => { + write_policy_violations(f, "merge produced an invalid policy", violations) + } Self::InvalidEndpointReference { host, port } => { write!(f, "invalid endpoint reference '{host}:{port}'") } @@ -474,6 +513,18 @@ impl std::fmt::Display for PolicyMergeError { impl std::error::Error for PolicyMergeError {} +fn write_policy_violations( + formatter: &mut std::fmt::Formatter<'_>, + context: &str, + violations: &[PolicyViolation], +) -> std::fmt::Result { + formatter.write_str(context)?; + for violation in violations { + write!(formatter, "; {violation}")?; + } + Ok(()) +} + #[derive(Debug, Clone, PartialEq)] pub struct PolicyMergeResult { pub policy: SandboxPolicy, @@ -515,6 +566,16 @@ pub struct PolicyMergeResult { /// treats an unset proposal value as unspecified for fields the merge retains, /// and exact-matches only the fields the proposal actually set. pub fn policy_covers_rule(policy: &SandboxPolicy, proposed: &NetworkPolicyRule) -> bool { + // Coverage is used as an admission signal. Invalid loaded or proposed + // state must fail closed instead of being treated as semantic equality. + // Compare the checked canonical values so authored omissions such as the + // pinned MCP revision default resolve exactly as they do during merge. + let Ok(policy) = validate_and_canonicalize_sandbox_policy(policy.clone()) else { + return false; + }; + let Ok(proposed) = validate_rule_fragment("coverage-proposal", proposed.clone()) else { + return false; + }; if proposed.endpoints.is_empty() { return false; } @@ -632,7 +693,8 @@ fn endpoint_attributes_cover(loaded: &NetworkEndpoint, proposed: &NetworkEndpoin return false; } - // Widened fields (list appends and `|=` flags) use containment: merging + // Widened fields (list appends and authorization flags) use containment: + // merging // into an endpoint that already carries them leaves the loaded copy a // superset of the proposal, so equality would report "not covered" for a // proposal that did land. @@ -648,7 +710,6 @@ fn endpoint_attributes_cover(loaded: &NetworkEndpoint, proposed: &NetworkEndpoin loaded.request_body_credential_rewrite, proposed.request_body_credential_rewrite, ) - && flag_covers(loaded.advisor_proposed, proposed.advisor_proposed) // Fields the merge neither widens nor retains: it drops them entirely. // An unset proposal value asks for nothing and is satisfied by whatever // is loaded; a set value that differs was dropped, so the proposal is @@ -726,30 +787,29 @@ fn effective_enforcement(value: &str) -> &str { if value.is_empty() { "audit" } else { value } } -#[derive(Debug, Clone, Copy, PartialEq, Eq)] +#[derive(Debug, Clone, PartialEq, Eq)] struct EffectiveMcpContract { + versions: Vec, strict_tool_names: bool, allow_all_known_mcp_methods: bool, max_body_bytes: u32, } fn effective_mcp_contract(endpoint: &NetworkEndpoint) -> Option { - endpoint - .protocol - .eq_ignore_ascii_case("mcp") - .then(|| EffectiveMcpContract { - strict_tool_names: endpoint - .mcp - .as_ref() - .and_then(|options| options.strict_tool_names) - .unwrap_or(true), - allow_all_known_mcp_methods: endpoint - .mcp - .as_ref() - .and_then(|options| options.allow_all_known_mcp_methods) - .unwrap_or(false), - max_body_bytes: effective_json_rpc_max_body_bytes(endpoint.json_rpc_max_body_bytes), - }) + if !endpoint.protocol.eq_ignore_ascii_case("mcp") { + return None; + } + + // Public merge entry points validate MCP options before comparison. Keep + // this helper fail closed for internal coverage checks as well. + let mut options = endpoint.mcp.clone()?; + canonicalize_mcp_options(&mut options); + Some(EffectiveMcpContract { + versions: options.versions, + strict_tool_names: options.strict_tool_names.unwrap_or(true), + allow_all_known_mcp_methods: options.allow_all_known_mcp_methods.unwrap_or(false), + max_body_bytes: effective_json_rpc_max_body_bytes(endpoint.json_rpc_max_body_bytes), + }) } fn effective_json_rpc_max_body_bytes(value: u32) -> u32 { @@ -772,24 +832,41 @@ fn mcp_contracts_match(left: &NetworkEndpoint, right: &NetworkEndpoint) -> bool } } +/// Apply policy operations only after validating and canonicalizing every input. +/// +/// The current policy is validated first. Each `AddRule` is then validated +/// immediately before it is applied to a private clone, preserving request-order +/// errors without exposing partial mutations. The final policy is validated +/// again so an internal merge invariant failure cannot reach a caller. pub fn merge_policy( policy: SandboxPolicy, operations: &[PolicyMergeOp], ) -> Result { - let mut merged = policy.clone(); + let canonical_input = validate_and_canonicalize_sandbox_policy(policy).map_err(|error| { + PolicyMergeError::InvalidInputPolicy { + violations: error.into_violations(), + } + })?; + let mut merged = canonical_input.clone(); let mut warnings = Vec::new(); // Validate and apply in request order. `merged` is private until every // operation succeeds, so failures remain atomic without allowing a later // malformed operation to replace the error from an earlier operation. - let conflicts_before = conflicting_inspection_contracts(&policy); + let conflicts_before = conflicting_inspection_contracts(&canonical_input); for (operation_index, operation) in operations.iter().enumerate() { validate_operation(operation_index, operation)?; - apply_operation(&mut merged, operation_index, operation, &mut warnings)?; + let operation = validate_and_canonicalize_operation(operation_index, operation)?; + apply_operation(&mut merged, operation_index, &operation, &mut warnings)?; } ensure_no_new_inspection_conflicts(&merged, &conflicts_before)?; - let changed = merged != policy; + let merged = validate_and_canonicalize_sandbox_policy(merged).map_err(|error| { + PolicyMergeError::InvalidMergedPolicy { + violations: error.into_violations(), + } + })?; + let changed = merged != canonical_input; Ok(PolicyMergeResult { policy: merged, warnings, @@ -864,8 +941,8 @@ fn conflicting_inspection_contracts( /// a plain REST rule on an overlapping path can satisfy authorization for a tool /// call the MCP endpoint never allowed, and the relay forwards it. MCP therefore /// cannot share a host and port with a differently inspected endpoint. Two MCP -/// endpoints must also agree on one contract, because MCP options are not -/// path-selected. +/// endpoints must also agree on one contract, including the exact revision +/// allowlist, because MCP options are not path-selected. fn inspections_conflict(found: &[(EffectiveInspection, String)]) -> bool { let mut mcp_contracts = found .iter() @@ -881,16 +958,20 @@ fn inspections_conflict(found: &[(EffectiveInspection, String)]) -> bool { /// Rejects an operation that introduces an L7 inspection-contract conflict. /// -/// Only conflicts absent from `before` are rejected. A policy that already -/// carries one, for instance through a provider profile composed outside this -/// merge, would otherwise make every later update fail with an error the -/// operation did nothing to cause. +/// A policy that already carries a conflict, for instance through a provider +/// profile composed outside this merge, must still permit unrelated updates and +/// operations that reduce its existing contract set. A post-merge conflict is +/// rejected when it contains any contract that was absent before, because that +/// operation introduced a new inspection ambiguity at the host and port. fn ensure_no_new_inspection_conflicts( merged: &SandboxPolicy, before: &BTreeMap<(String, u32), Vec>, ) -> Result<(), PolicyMergeError> { for ((host, port), contracts) in conflicting_inspection_contracts(merged) { - if before.contains_key(&(host.clone(), port)) { + let contains_only_preexisting_contracts = before + .get(&(host.clone(), port)) + .is_some_and(|prior| contracts.iter().all(|contract| prior.contains(contract))); + if contains_only_preexisting_contracts { continue; } return Err(PolicyMergeError::ConflictingInspectionContracts { @@ -902,6 +983,45 @@ fn ensure_no_new_inspection_conflicts( Ok(()) } +fn validate_and_canonicalize_operation( + operation_index: usize, + operation: &PolicyMergeOp, +) -> Result { + let PolicyMergeOp::AddRule { rule_name, rule } = operation else { + return Ok(operation.clone()); + }; + + let rule = validate_rule_fragment(rule_name, rule.clone()).map_err(|violations| { + PolicyMergeError::InvalidOperationPolicy { + operation_index, + violations, + } + })?; + Ok(PolicyMergeOp::AddRule { + rule_name: rule_name.clone(), + rule, + }) +} + +fn validate_rule_fragment( + rule_name: &str, + rule: NetworkPolicyRule, +) -> Result> { + let mut fragment = restrictive_default_policy(); + fragment + .network_policies + .insert(rule_name.to_string(), rule); + let mut canonical = validate_and_canonicalize_sandbox_policy(fragment) + .map_err(crate::PolicyValidationError::into_violations)?; + + // Canonicalization only mutates endpoint fields, so a validated fragment + // must retain the key inserted immediately above. + Ok(canonical + .network_policies + .remove(rule_name) + .expect("validated fragment must retain its rule")) +} + fn validate_operation( operation_index: usize, operation: &PolicyMergeOp, @@ -1189,7 +1309,10 @@ fn is_authorization_inheritance_conflict(error: &PolicyMergeError) -> bool { // Malformed operations, and operations `merge_rules` never produces. // Neither is answered by choosing a different rule to write to. - PolicyMergeError::MissingRuleNameForAddRule + PolicyMergeError::InvalidInputPolicy { .. } + | PolicyMergeError::InvalidOperationPolicy { .. } + | PolicyMergeError::InvalidMergedPolicy { .. } + | PolicyMergeError::MissingRuleNameForAddRule | PolicyMergeError::EmptyAddRuleEndpoints { .. } | PolicyMergeError::CannotRemoveBinaryFromAnyBinaryScope { .. } => false, } @@ -1374,7 +1497,11 @@ fn merge_endpoint( existing.websocket_credential_rewrite |= incoming.websocket_credential_rewrite; existing.request_body_credential_rewrite |= incoming.request_body_credential_rewrite; existing.allow_uninspected_credentials |= incoming.allow_uninspected_credentials; - existing.advisor_proposed |= incoming.advisor_proposed; + // Provenance is not an authorization bit. If either declaration came + // directly from a user or provider, keep the endpoint explicit. This + // mirrors binary provenance and prevents an advisor overlay from tainting + // an already explicit endpoint for exact-host SSRF evaluation. + existing.advisor_proposed &= incoming.advisor_proposed; normalize_endpoint(existing); Ok(()) } @@ -1408,7 +1535,8 @@ fn describe_mcp_contract(endpoint: &NetworkEndpoint) -> String { || format!("non-mcp(protocol='{}')", endpoint.protocol), |contract| { format!( - "mcp(strict_tool_names={}, allow_all_known_mcp_methods={}, max_body_bytes={})", + "mcp(versions={:?}, strict_tool_names={}, allow_all_known_mcp_methods={}, max_body_bytes={})", + contract.versions, contract.strict_tool_names, contract.allow_all_known_mcp_methods, contract.max_body_bytes @@ -1979,6 +2107,9 @@ fn normalize_endpoint(endpoint: &mut NetworkEndpoint) { dedup_strings(&mut endpoint.allowed_ips); dedup_l7_rules(&mut endpoint.rules); dedup_deny_rules(&mut endpoint.deny_rules); + if let Some(options) = endpoint.mcp.as_mut() { + canonicalize_mcp_options(options); + } } fn dedup_strings(values: &mut Vec) { @@ -2088,12 +2219,18 @@ mod tests { PolicyMergeWarning, canonical_ports, canonicalize_advisor_add_rule, generated_rule_name, merge_policy, policy_covers_rule, }; - use crate::restrictive_default_policy; - use openshell_core::proto::{ - L7Allow, L7DenyRule, L7QueryMatcher, L7Rule, McpOptions, NetworkBinary, NetworkEndpoint, - NetworkPolicyRule, SandboxPolicy, + use crate::{restrictive_default_policy, validate_sandbox_policy}; + use openshell_core::{ + mcp::DEFAULT_MCP_PROTOCOL_VERSION, + proto::{ + L7Allow, L7DenyRule, L7QueryMatcher, L7Rule, McpOptions, NetworkBinary, + NetworkEndpoint, NetworkPolicyRule, SandboxPolicy, + }, }; + const MCP_VERSION: &str = "2025-03-26"; + const DEFAULT_MCP_VERSION: &str = DEFAULT_MCP_PROTOCOL_VERSION.as_str(); + fn endpoint(host: &str, port: u32) -> NetworkEndpoint { NetworkEndpoint { host: host.to_string(), @@ -2216,12 +2353,65 @@ mod tests { assert_eq!(canonical.endpoints[0].protocol, "rest"); assert_eq!(canonical.endpoints[0].access, "read-only"); assert!(!canonical.endpoints[0].provider_credentialed); + assert!(canonical.endpoints[0].advisor_proposed); assert_eq!( effective.network_policies["_provider_example"].endpoints[0], provider_endpoint ); } + #[test] + fn canonicalize_advisor_ignores_endpoint_provenance_when_inferring_contract() { + let mut provider_endpoint = endpoint("api.example.com", 443); + provider_endpoint.protocol = "rest".to_string(); + provider_endpoint.enforcement = "enforce".to_string(); + provider_endpoint.access = "read-only".to_string(); + provider_endpoint.provider_credentialed = true; + + let mut advisor_endpoint = provider_endpoint.clone(); + advisor_endpoint.provider_credentialed = false; + advisor_endpoint.advisor_proposed = true; + + let mut base = SandboxPolicy::default(); + base.network_policies.insert( + "existing_advisor".to_string(), + NetworkPolicyRule { + name: "existing-advisor".to_string(), + endpoints: vec![advisor_endpoint], + binaries: vec![advisor_binary("/usr/bin/curl")], + }, + ); + + let mut effective = base.clone(); + effective.network_policies.insert( + "_provider_example".to_string(), + NetworkPolicyRule { + name: "provider-example".to_string(), + endpoints: vec![provider_endpoint], + binaries: vec![binary("/usr/bin/gh")], + }, + ); + + let incoming = NetworkPolicyRule { + name: "advisor_example".to_string(), + endpoints: vec![NetworkEndpoint { + host: "api.example.com".to_string(), + port: 443, + advisor_proposed: true, + ..Default::default() + }], + binaries: vec![advisor_binary("/usr/bin/python")], + }; + + let (rule_name, canonical) = + canonicalize_advisor_add_rule(&base, &effective, "advisor_example", &incoming) + .expect("provenance alone must not create multiple endpoint contracts"); + + assert_eq!(rule_name, "existing_advisor"); + assert_eq!(canonical.endpoints[0].protocol, "rest"); + assert_eq!(canonical.endpoints[0].access, "read-only"); + } + #[test] fn canonicalize_advisor_narrows_multi_port_contract_and_keeps_overlay() { let mut existing_endpoint = endpoint("index.crates.io", 443); @@ -2327,6 +2517,7 @@ mod tests { mcp: Some(McpOptions { strict_tool_names, allow_all_known_mcp_methods, + versions: vec![MCP_VERSION.to_string()], }), ..Default::default() } @@ -2350,6 +2541,52 @@ mod tests { policy } + fn mcp_endpoint_with_versions(versions: &[&str]) -> NetworkEndpoint { + let mut endpoint = mcp_endpoint( + "mcp.example.com", + &[443], + None, + None, + 0, + vec![mcp_tool_rule("fixture-tool")], + ); + endpoint + .mcp + .as_mut() + .expect("MCP test endpoint must contain options") + .versions = versions.iter().map(ToString::to_string).collect(); + endpoint + } + + fn mcp_endpoint_without_options() -> NetworkEndpoint { + let mut endpoint = mcp_endpoint_with_versions(&[]); + endpoint.mcp = None; + endpoint + } + + fn default_mcp_endpoint_representations() -> [(&'static str, NetworkEndpoint); 3] { + [ + ("omitted", mcp_endpoint_without_options()), + ("empty", mcp_endpoint_with_versions(&[])), + ( + "explicit default", + mcp_endpoint_with_versions(&[DEFAULT_MCP_VERSION]), + ), + ] + } + + fn invalid_explicit_mcp_endpoints() -> Vec { + let mut misplaced_options = mcp_endpoint_with_versions(&[MCP_VERSION]); + misplaced_options.protocol = "rest".to_string(); + + vec![ + mcp_endpoint_with_versions(&[MCP_VERSION, MCP_VERSION]), + mcp_endpoint_with_versions(&["latest"]), + mcp_endpoint_with_versions(&["2025-03-26 "]), + misplaced_options, + ] + } + #[test] fn generated_rule_name_sanitizes_host() { assert_eq!( @@ -2401,6 +2638,311 @@ mod tests { ); } + #[test] + fn merge_rejects_invalid_mcp_input_before_operations() { + for endpoint in invalid_explicit_mcp_endpoints() { + let policy = policy_with_rule( + "existing", + rule_with_authorizations("existing", vec![endpoint], &["/usr/bin/client"]), + ); + let error = merge_policy( + policy, + &[PolicyMergeOp::RemoveRule { + rule_name: "existing".to_string(), + }], + ) + .expect_err("an operation cannot hide invalid input policy state"); + + assert!(matches!( + error, + PolicyMergeError::InvalidInputPolicy { violations } + if !violations.is_empty() + )); + } + } + + #[test] + fn merge_rejects_invalid_mcp_add_rule_before_overlap() { + let existing = rule_with_authorizations( + "existing", + vec![mcp_endpoint_with_versions(&[MCP_VERSION])], + &["/usr/bin/client"], + ); + + for endpoint in invalid_explicit_mcp_endpoints() { + let incoming = + rule_with_authorizations("existing", vec![endpoint], &["/usr/bin/client"]); + let error = merge_policy( + policy_with_rule("existing", existing.clone()), + &[PolicyMergeOp::AddRule { + rule_name: "existing".to_string(), + rule: incoming, + }], + ) + .expect_err("an invalid AddRule must fail before endpoint folding"); + + assert!(matches!( + error, + PolicyMergeError::InvalidOperationPolicy { + operation_index: 0, + violations, + } if !violations.is_empty() + )); + } + } + + #[test] + fn invalid_mcp_add_rule_reports_its_request_index() { + let operations = [ + PolicyMergeOp::RemoveRule { + rule_name: "absent".to_string(), + }, + PolicyMergeOp::AddRule { + rule_name: "mcp".to_string(), + rule: rule_with_authorizations( + "mcp", + vec![mcp_endpoint_with_versions(&[MCP_VERSION, MCP_VERSION])], + &["/usr/bin/client"], + ), + }, + ]; + + assert!(matches!( + merge_policy(restrictive_default_policy(), &operations), + Err(PolicyMergeError::InvalidOperationPolicy { + operation_index: 1, + .. + }) + )); + } + + #[test] + fn omitted_empty_and_explicit_default_mcp_versions_are_semantically_equal() { + let representations = default_mcp_endpoint_representations(); + let mut expected_merged_policy = None; + + for (base_label, base_endpoint) in &representations { + for (operation_label, operation_endpoint) in &representations { + let mut base_endpoint = base_endpoint.clone(); + base_endpoint.rules = vec![mcp_tool_rule("existing-tool")]; + let base_rule = + rule_with_authorizations("existing", vec![base_endpoint], &["/usr/bin/client"]); + let base_policy = policy_with_rule("existing", base_rule); + + // Coverage and merge are separate public admission boundaries. + // Both must resolve the same default before comparing contracts. + let mut equivalent_endpoint = operation_endpoint.clone(); + equivalent_endpoint.rules = vec![mcp_tool_rule("existing-tool")]; + let equivalent_rule = rule_with_authorizations( + "equivalent", + vec![equivalent_endpoint], + &["/usr/bin/client"], + ); + assert!( + policy_covers_rule(&base_policy, &equivalent_rule), + "{base_label} loaded state must cover an {operation_label} proposal" + ); + + let mut incoming_endpoint = operation_endpoint.clone(); + incoming_endpoint.rules = vec![mcp_tool_rule("new-tool")]; + let incoming_rule = rule_with_authorizations( + "existing", + vec![incoming_endpoint], + &["/usr/bin/client"], + ); + let result = merge_policy( + base_policy, + &[PolicyMergeOp::AddRule { + rule_name: "existing".to_string(), + rule: incoming_rule, + }], + ) + .unwrap_or_else(|error| { + panic!("{base_label} base must merge with {operation_label} operation: {error}") + }); + + assert!(result.changed); + assert_eq!( + result.policy.network_policies["existing"].endpoints[0] + .mcp + .as_ref() + .expect("canonical MCP output must contain explicit options") + .versions, + [DEFAULT_MCP_VERSION], + "{base_label} base and {operation_label} operation must materialize the pinned default" + ); + if let Some(expected) = expected_merged_policy.as_ref() { + assert_eq!( + &result.policy, expected, + "{base_label} base and {operation_label} operation must have one canonical result" + ); + } else { + expected_merged_policy = Some(result.policy); + } + } + } + } + + #[test] + fn explicit_non_default_mcp_allowlists_conflict_with_the_materialized_default() { + let expected_default_contract = format!( + "mcp(versions=[\"{DEFAULT_MCP_VERSION}\"], strict_tool_names=true, allow_all_known_mcp_methods=false, max_body_bytes={DEFAULT_JSON_RPC_MAX_BODY_BYTES})" + ); + + for versions in [ + &["2025-03-26"][..], + &["2025-03-26", DEFAULT_MCP_VERSION][..], + ] { + let existing = rule_with_authorizations( + "existing", + vec![mcp_endpoint_without_options()], + &["/usr/bin/client"], + ); + let incoming = rule_with_authorizations( + "existing", + vec![mcp_endpoint_with_versions(versions)], + &["/usr/bin/client"], + ); + + let error = merge_policy( + policy_with_rule("existing", existing), + &[PolicyMergeOp::AddRule { + rule_name: "existing".to_string(), + rule: incoming, + }], + ) + .expect_err("an explicit non-default allowlist must remain a distinct contract"); + + match error { + PolicyMergeError::McpContractConflict { + operation_index, + existing, + incoming, + .. + } => { + assert_eq!(operation_index, 0); + assert_eq!(existing, expected_default_contract); + assert!( + versions.iter().all(|version| incoming.contains(*version)), + "incoming contract must render every explicit revision: {incoming}" + ); + } + other => panic!("expected an MCP contract conflict, got {other:?}"), + } + } + } + + #[test] + fn merge_canonicalizes_equivalent_mcp_version_order() { + let existing = rule_with_authorizations( + "existing", + vec![mcp_endpoint_with_versions(&[ + "2025-11-25", + "2025-03-26", + "2025-06-18", + ])], + &["/usr/bin/trusted"], + ); + let incoming = rule_with_authorizations( + "existing", + vec![mcp_endpoint_with_versions(&[ + "2025-06-18", + "2025-11-25", + "2025-03-26", + ])], + &["/usr/bin/new"], + ); + + let result = merge_policy( + policy_with_rule("existing", existing), + &[PolicyMergeOp::AddRule { + rule_name: "existing".to_string(), + rule: incoming, + }], + ) + .expect("equivalent complete MCP contracts must merge"); + + assert!(result.changed); + assert!(validate_sandbox_policy(&result.policy).is_ok()); + assert_eq!( + result.policy.network_policies["existing"].endpoints[0] + .mcp + .as_ref() + .expect("MCP endpoint must retain options") + .versions, + ["2025-03-26", "2025-06-18", "2025-11-25"] + ); + } + + #[test] + fn canonicalizing_input_order_is_not_a_policy_change() { + let input = policy_with_rule( + "mcp", + rule_with_authorizations( + "mcp", + vec![mcp_endpoint_with_versions(&[ + "2025-11-25", + "2025-03-26", + "2025-06-18", + ])], + &["/usr/bin/client"], + ), + ); + + let result = merge_policy(input, &[]).expect("valid input must canonicalize"); + + assert!(!result.changed); + assert_eq!( + result.policy.network_policies["mcp"].endpoints[0] + .mcp + .as_ref() + .expect("MCP endpoint must retain options") + .versions, + ["2025-03-26", "2025-06-18", "2025-11-25"] + ); + } + + #[test] + fn policy_coverage_fails_closed_for_invalid_mcp_contracts() { + let loaded_rule = rule_with_authorizations( + "loaded", + vec![mcp_endpoint_with_versions(&[ + "2025-03-26", + "2025-06-18", + "2025-11-25", + ])], + &["/usr/bin/client"], + ); + let loaded = policy_with_rule("loaded", loaded_rule); + + for endpoint in invalid_explicit_mcp_endpoints() { + let proposed = + rule_with_authorizations("proposed", vec![endpoint], &["/usr/bin/client"]); + assert!(!policy_covers_rule(&loaded, &proposed)); + } + + let equivalent = rule_with_authorizations( + "proposed", + vec![mcp_endpoint_with_versions(&[ + "2025-11-25", + "2025-03-26", + "2025-06-18", + ])], + &["/usr/bin/client"], + ); + assert!(policy_covers_rule(&loaded, &equivalent)); + + let invalid_loaded = policy_with_rule( + "loaded", + rule_with_authorizations( + "loaded", + vec![mcp_endpoint_with_versions(&[MCP_VERSION, MCP_VERSION])], + &["/usr/bin/client"], + ), + ); + assert!(!policy_covers_rule(&invalid_loaded, &equivalent)); + } + #[test] fn new_binary_cannot_inherit_an_undeclared_existing_rest_endpoint() { let existing_endpoint = NetworkEndpoint { @@ -2678,6 +3220,7 @@ mod tests { Some(McpOptions { strict_tool_names: Some(false), allow_all_known_mcp_methods: Some(true), + versions: vec![MCP_VERSION.to_string()], }) ); assert_eq!(promoted.rules, vec![mcp_tool_rule("new-tool")]); @@ -3025,7 +3568,7 @@ mod tests { } #[test] - fn policy_coverage_requires_endpoint_advisor_provenance_to_be_loaded() { + fn policy_coverage_ignores_endpoint_advisor_provenance() { let loaded_endpoint = endpoint("api.example.com", 443); let mut proposed_endpoint = loaded_endpoint.clone(); proposed_endpoint.advisor_proposed = true; @@ -3036,7 +3579,37 @@ mod tests { let proposed = rule_with_authorizations("proposed", vec![proposed_endpoint], &["/usr/bin/client"]); - assert!(!policy_covers_rule(&loaded, &proposed)); + assert!(policy_covers_rule(&loaded, &proposed)); + } + + #[test] + fn explicit_endpoint_provenance_wins_in_either_merge_order() { + let explicit_endpoint = endpoint("api.example.com", 443); + let mut proposed_endpoint = explicit_endpoint.clone(); + proposed_endpoint.advisor_proposed = true; + + for (existing_endpoint, incoming_endpoint) in [ + (explicit_endpoint.clone(), proposed_endpoint.clone()), + (proposed_endpoint, explicit_endpoint), + ] { + let incoming = + rule_with_authorizations("api", vec![incoming_endpoint], &["/usr/bin/client"]); + let merged = merge_policy( + policy_with_rule( + "api", + rule_with_authorizations("api", vec![existing_endpoint], &["/usr/bin/client"]), + ), + &[PolicyMergeOp::AddRule { + rule_name: "api".to_string(), + rule: incoming.clone(), + }], + ) + .expect("compatible explicit and advisor rules should merge"); + + let endpoint = &merged.policy.network_policies["api"].endpoints[0]; + assert!(!endpoint.advisor_proposed); + assert!(policy_covers_rule(&merged.policy, &incoming)); + } } #[test] @@ -3277,6 +3850,7 @@ mod tests { port: 443, ports: vec![443], protocol: "websocket".to_string(), + access: "read-write".to_string(), websocket_credential_rewrite: true, ..Default::default() }], @@ -3322,6 +3896,7 @@ mod tests { port: 443, ports: vec![443], protocol: "rest".to_string(), + access: "read-write".to_string(), request_body_credential_rewrite: true, ..Default::default() }], @@ -4118,6 +4693,7 @@ mod tests { "protocol", NetworkEndpoint { protocol: "rest".to_string(), + access: "read-write".to_string(), ..endpoint("api.example.com", 443) }, ), @@ -4570,6 +5146,7 @@ mod tests { "protocol", NetworkEndpoint { protocol: "rest".to_string(), + access: "read-write".to_string(), ..endpoint("api.example.com", 443) }, ), @@ -4644,6 +5221,7 @@ mod tests { "existing", vec![NetworkEndpoint { protocol: "rest".to_string(), + access: "read-write".to_string(), ..endpoint("api.example.com", 443) }], &["/usr/bin/trusted"], @@ -4652,6 +5230,7 @@ mod tests { "existing", vec![NetworkEndpoint { protocol: "websocket".to_string(), + access: "read-write".to_string(), ..endpoint("api.example.com", 443) }], &["/usr/bin/trusted", "/usr/bin/second"], @@ -5154,6 +5733,82 @@ mod tests { .expect("an unrelated endpoint must not inherit a baseline conflict"); } + /// A baseline conflict does not license later operations to add another + /// incompatible contract at the same host and port. + #[test] + fn a_preexisting_mcp_conflict_does_not_license_a_new_revision_contract() { + let mut policy = policy_with_rule( + "first_rule", + rule_with_authorizations( + "first_rule", + vec![mcp_endpoint( + "mcp.example.com", + &[443], + None, + None, + 65_536, + vec![mcp_tool_rule("first")], + )], + &["/usr/bin/a"], + ), + ); + policy.network_policies.insert( + "second_rule".to_string(), + rule_with_authorizations( + "second_rule", + vec![NetworkEndpoint { + path: "/other".to_string(), + ..mcp_endpoint( + "mcp.example.com", + &[443], + None, + None, + 131_072, + vec![mcp_tool_rule("second")], + ) + }], + &["/usr/bin/b"], + ), + ); + + let error = merge_policy( + policy, + &[PolicyMergeOp::AddRule { + rule_name: "third_rule".to_string(), + rule: rule_with_authorizations( + "third_rule", + vec![NetworkEndpoint { + path: "/third".to_string(), + json_rpc_max_body_bytes: 65_536, + rules: vec![mcp_tool_rule("third")], + ..mcp_endpoint_with_versions(&["2025-06-18"]) + }], + &["/usr/bin/c"], + ), + }], + ) + .expect_err("a new revision contract must not hide behind a baseline conflict"); + + match error { + PolicyMergeError::ConflictingInspectionContracts { + host, + port, + contracts, + } => { + assert_eq!(host, "mcp.example.com"); + assert_eq!(port, 443); + assert_eq!(contracts.len(), 3); + assert!( + contracts + .iter() + .any(|contract| contract.contains("2025-06-18")), + "the diagnostic must include the newly introduced revision contract" + ); + } + other => panic!("expected an inspection-contract conflict, got {other:?}"), + } + } + /// An additive operation must never revoke access. Appending a named binary /// to a rule that authorizes any binary would restrict it to that one path. #[test] @@ -5365,6 +6020,7 @@ mod tests { vec![NetworkEndpoint { path: "/graphql".to_string(), protocol: "graphql".to_string(), + access: "read-only".to_string(), ..endpoint("api.github.com", 443) }], &["/usr/bin/only"], diff --git a/crates/openshell-policy/testdata/mcp-version-profiles.yaml b/crates/openshell-policy/testdata/mcp-version-profiles.yaml new file mode 100644 index 000000000..1d6106029 --- /dev/null +++ b/crates/openshell-policy/testdata/mcp-version-profiles.yaml @@ -0,0 +1,23 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +version: 1 + +network_policies: + versioned_mcp: + name: versioned_mcp + endpoints: + - host: mcp.example.com + port: 443 + protocol: mcp + enforcement: enforce + mcp: + versions: + - "2025-11-25" + - "2025-03-26" + - "2025-06-18" + rules: + - allow: + method: initialize + binaries: + - path: /usr/bin/mcp-client diff --git a/crates/openshell-prover/testdata/accepted-risks.yaml b/crates/openshell-prover/testdata/accepted-risks.yaml deleted file mode 100644 index 6af5c7146..000000000 --- a/crates/openshell-prover/testdata/accepted-risks.yaml +++ /dev/null @@ -1,7 +0,0 @@ -# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -# SPDX-License-Identifier: Apache-2.0 - -accepted_risks: - - query: inference_relay - reason: "Demo environment — inference.local not configured." - accepted_by: demo diff --git a/crates/openshell-providers/src/discovery.rs b/crates/openshell-providers/src/discovery.rs index f1af8fbfa..00e8da189 100644 --- a/crates/openshell-providers/src/discovery.rs +++ b/crates/openshell-providers/src/discovery.rs @@ -66,6 +66,16 @@ pub fn discover_from_profile( } } + if profile.id == "google-vertex-ai" { + for key in crate::VERTEX_AI_CONFIG_KEY_NAMES { + if let Some(value) = context.env_var(key) + && !value.trim().is_empty() + { + discovered.config.entry((*key).to_string()).or_insert(value); + } + } + } + if discovered.is_empty() { Ok(None) } else { @@ -165,4 +175,27 @@ mod tests { } if profile_id == "custom" && credential_name == "missing" )); } + + #[test] + fn vertex_profile_discovery_includes_supported_configuration() { + let mut profile = profile(); + profile.id = "google-vertex-ai".to_string(); + let ctx = MockDiscoveryContext::new() + .with_env("CUSTOM_API_KEY", "vertex-token") + .with_env("VERTEX_AI_PROJECT_ID", "project-a") + .with_env("VERTEX_AI_REGION", "us-central1"); + + let discovered = discover_from_profile(&profile, &ctx) + .expect("discovery should succeed") + .expect("provider should be discovered"); + + assert_eq!( + discovered.config.get("VERTEX_AI_PROJECT_ID"), + Some(&"project-a".to_string()) + ); + assert_eq!( + discovered.config.get("VERTEX_AI_REGION"), + Some(&"us-central1".to_string()) + ); + } } diff --git a/crates/openshell-providers/src/lib.rs b/crates/openshell-providers/src/lib.rs index fabe1a5be..8b9f0d313 100644 --- a/crates/openshell-providers/src/lib.rs +++ b/crates/openshell-providers/src/lib.rs @@ -24,6 +24,16 @@ pub use profiles::{ strategy_output_env_key, strategy_output_spec, strategy_primary_env_key, validate_profile_set, }; +pub const VERTEX_AI_PROJECT_ID_KEY: &str = "VERTEX_AI_PROJECT_ID"; +pub const VERTEX_AI_REGION_KEY: &str = "VERTEX_AI_REGION"; +pub const VERTEX_AI_CONFIG_KEY_NAMES: &[&str] = &[ + VERTEX_AI_PROJECT_ID_KEY, + VERTEX_AI_REGION_KEY, + "GOOGLE_VERTEX_AI_BASE_URL", + "VERTEX_AI_BASE_URL", + "VERTEX_AI_PUBLISHER", +]; + #[derive(Debug, thiserror::Error)] pub enum ProviderError { #[error("unsupported provider type: {0}")] @@ -56,21 +66,10 @@ pub struct ProviderDiscoverySpec { pub credential_env_vars: &'static [&'static str], } -pub trait ProviderPlugin: Send + Sync { - /// Canonical provider id (for example: "claude", "gitlab"). +trait ProviderPlugin: Send + Sync { + /// Canonical provider id. fn id(&self) -> &'static str; - /// Discover provider credentials and config from the local machine. - fn discover_existing(&self) -> Result, ProviderError>; - - /// Return the known credential environment variable names for this provider type. - /// - /// Used by the TUI to label BYO key entry fields and to choose which - /// env var name to store a manually-entered credential under. - fn credential_env_vars(&self) -> &'static [&'static str] { - &[] - } - /// Inject provider-specific environment variables into the sandbox env. /// /// Called during sandbox creation to project provider config (project IDs, @@ -79,25 +78,6 @@ pub trait ProviderPlugin: Send + Sync { fn inject_env(&self, _provider: &Provider, _env: &mut HashMap) {} } -/// Blanket implementation of [`ProviderPlugin`] for [`ProviderDiscoverySpec`]. -/// -/// Providers that only need standard env-var discovery can register their -/// `SPEC` constant directly, instead of defining a dedicated struct and -/// repeating the same three-method delegation. -impl ProviderPlugin for ProviderDiscoverySpec { - fn id(&self) -> &'static str { - self.id - } - - fn discover_existing(&self) -> Result, ProviderError> { - discover_with_spec(self, &RealDiscoveryContext) - } - - fn credential_env_vars(&self) -> &'static [&'static str] { - self.credential_env_vars - } -} - #[derive(Default)] pub struct ProviderRegistry { plugins: HashMap<&'static str, Box>, @@ -107,24 +87,15 @@ impl ProviderRegistry { #[must_use] pub fn new() -> Self { let mut registry = Self::default(); - registry.register(providers::claude::SPEC); - registry.register(providers::codex::SPEC); - registry.register(providers::copilot::SPEC); - registry.register(providers::opencode::OpencodeProvider); - registry.register(providers::generic::GenericProvider); - registry.register(providers::openai::SPEC); - registry.register(providers::anthropic::SPEC); - registry.register(providers::nvidia::SPEC); - registry.register(providers::deepinfra::SPEC); - registry.register(providers::github::SPEC); - registry.register(providers::gitlab::SPEC); + // Keep only the legacy config projectors required to run existing + // Google Cloud and Vertex records. Public provider discovery is + // profile-driven; this registry is an internal compatibility adapter. registry.register(providers::google_cloud::GoogleCloudProvider); - registry.register(providers::outlook::OutlookProvider); registry.register(providers::vertex::VertexProvider); registry } - pub fn register

(&mut self, plugin: P) + fn register

(&mut self, plugin: P) where P: ProviderPlugin + 'static, { @@ -132,36 +103,10 @@ impl ProviderRegistry { } #[must_use] - pub fn get(&self, id: &str) -> Option<&dyn ProviderPlugin> { + fn get(&self, id: &str) -> Option<&dyn ProviderPlugin> { self.plugins.get(id).map(Box::as_ref) } - pub fn discover_existing(&self, id: &str) -> Result, ProviderError> { - let Some(plugin) = self.get(id) else { - return Err(ProviderError::UnsupportedProvider(id.to_string())); - }; - plugin.discover_existing() - } - - /// Return the known credential env var names for a provider type. - #[must_use] - pub fn credential_env_vars(&self, id: &str) -> &'static [&'static str] { - self.get(id) - .map_or(&[], ProviderPlugin::credential_env_vars) - } - - #[must_use] - pub fn profile(&self, id: &str) -> Option<&'static ProviderTypeProfile> { - builtin_profiles() - .iter() - .find(|profile| profile.id.eq_ignore_ascii_case(id)) - } - - #[must_use] - pub fn profiles(&self) -> Vec<&'static ProviderTypeProfile> { - builtin_profiles().iter().collect() - } - /// Inject provider-specific env vars via the registered plugin. /// /// Normalizes the provider type and delegates to the plugin's `inject_env`. @@ -176,32 +121,38 @@ impl ProviderRegistry { } } - #[must_use] - pub fn known_types(&self) -> Vec<&'static str> { - let mut types = self.plugins.keys().copied().collect::>(); - types.sort_unstable(); - types + /// Inject config for an already-resolved profile ID without alias + /// normalization. This prevents an exact custom profile whose ID resembles + /// a legacy alias from selecting an unrelated built-in compatibility plugin. + pub fn inject_env_for_profile_id( + &self, + provider: &Provider, + profile_id: &str, + env: &mut HashMap, + ) { + if let Some(plugin) = self.get(profile_id) { + plugin.inject_env(provider, env); + } } } #[must_use] pub fn normalize_provider_type(input: &str) -> Option<&'static str> { - // Inference provider aliases are canonicalized in openshell-core so that - // openshell-server and openshell-providers agree on the same mapping. - if let Some(canonical) = openshell_core::inference::normalize_inference_provider_type(input) { - return Some(canonical); - } let normalized = input.trim().to_ascii_lowercase(); match normalized.as_str() { + "openai" => Some("openai"), + "anthropic" => Some("anthropic"), + "nvidia" => Some("nvidia"), + "deepinfra" => Some("deepinfra"), + "aws-bedrock" => Some("aws-bedrock"), + "google-vertex-ai" | "vertex" | "vertex-ai" | "google-vertex" | "gcp-vertex" => { + Some("google-vertex-ai") + } "claude" | "claude-code" | "claude_code" => Some("claude-code"), "codex" => Some("codex"), "copilot" => Some("copilot"), - "opencode" => Some("opencode"), "gcp" | "google-cloud" => Some("google-cloud"), - "generic" => Some("generic"), - "gitlab" | "glab" => Some("gitlab"), "github" | "gh" => Some("github"), - "outlook" => Some("outlook"), _ => None, } } @@ -222,25 +173,27 @@ mod tests { #[test] fn normalizes_known_provider_aliases() { - assert_eq!(normalize_provider_type("gitlab"), Some("gitlab")); - assert_eq!(normalize_provider_type("glab"), Some("gitlab")); assert_eq!(normalize_provider_type("gh"), Some("github")); assert_eq!(normalize_provider_type("CLAUDE"), Some("claude-code")); assert_eq!(normalize_provider_type("claude-code"), Some("claude-code")); - assert_eq!(normalize_provider_type("generic"), Some("generic")); + for retired in ["generic", "gitlab", "glab", "opencode", "outlook"] { + assert_eq!(normalize_provider_type(retired), None); + } assert_eq!(normalize_provider_type("openai"), Some("openai")); assert_eq!(normalize_provider_type("anthropic"), Some("anthropic")); assert_eq!(normalize_provider_type("nvidia"), Some("nvidia")); + assert_eq!(normalize_provider_type("deepinfra"), Some("deepinfra")); + assert_eq!(normalize_provider_type("aws-bedrock"), Some("aws-bedrock")); assert_eq!(normalize_provider_type("copilot"), Some("copilot")); - assert_eq!( - normalize_provider_type("google-vertex-ai"), - Some("google-vertex-ai") - ); - assert_eq!(normalize_provider_type("vertex"), Some("google-vertex-ai")); - assert_eq!( - normalize_provider_type("vertex-ai"), - Some("google-vertex-ai") - ); + for alias in [ + "google-vertex-ai", + "vertex", + "vertex-ai", + "google-vertex", + "gcp-vertex", + ] { + assert_eq!(normalize_provider_type(alias), Some("google-vertex-ai")); + } assert_eq!(normalize_provider_type("unknown"), None); } @@ -252,7 +205,7 @@ mod tests { ); assert_eq!( detect_provider_from_command(&["/usr/bin/glab".to_string()]), - Some("gitlab") + None ); assert_eq!( detect_provider_from_command(&["/usr/bin/bash".to_string()]), diff --git a/crates/openshell-providers/src/profiles.rs b/crates/openshell-providers/src/profiles.rs index 6e87826c4..9cdf2257a 100644 --- a/crates/openshell-providers/src/profiles.rs +++ b/crates/openshell-providers/src/profiles.rs @@ -5,6 +5,7 @@ #![allow(deprecated)] // NetworkBinary::harness remains in the public proto for compatibility. +use openshell_core::mcp::{DEFAULT_MCP_PROTOCOL_VERSION, McpProtocolVersion}; use openshell_core::proto::{ GraphqlOperation, L7Allow, L7DenyRule, L7QueryMatcher, L7Rule, McpOptions, NetworkBinary, NetworkEndpoint, NetworkPolicyRule, ProviderCredentialRefresh, @@ -15,20 +16,23 @@ use openshell_core::proto::{ }; use openshell_core::secrets::uses_reserved_revision_namespace; use openshell_policy::{ - L7EndpointFields, validate_explicit_tcp_additional_fields, validate_l7_endpoint_semantics, + L7EndpointFields, L7Protocol, validate_explicit_tcp_additional_fields, + validate_l7_endpoint_semantics, }; use serde::ser::SerializeStruct; use serde::{Deserialize, Deserializer, Serialize, Serializer, de}; -use std::collections::{HashMap, HashSet}; +use std::collections::{BTreeSet, HashMap, HashSet}; use std::net::IpAddr; use std::sync::OnceLock; const PATH_TEMPLATE_CREDENTIAL_PLACEHOLDER: &str = "{credential}"; +const MCP_VERSION_REMEDIATION: &str = "omit mcp.versions to use the pinned default revision; use an exact supported revision; or omit protocol and mcp for deliberate uninspected L4 passthrough only when that weaker boundary is acceptable"; const BUILT_IN_PROFILE_YAMLS: &[&str] = &[ include_str!("../../../providers/aws.yaml"), include_str!("../../../providers/aws-bedrock.yaml"), include_str!("../../../providers/aws-s3.yaml"), + include_str!("../../../providers/anthropic.yaml"), include_str!("../../../providers/claude-code.yaml"), include_str!("../../../providers/codex.yaml"), include_str!("../../../providers/copilot.yaml"), @@ -38,6 +42,7 @@ const BUILT_IN_PROFILE_YAMLS: &[&str] = &[ include_str!("../../../providers/google-cloud.yaml"), include_str!("../../../providers/google-vertex-ai.yaml"), include_str!("../../../providers/nvidia.yaml"), + include_str!("../../../providers/openai.yaml"), include_str!("../../../providers/pypi.yaml"), ]; @@ -53,6 +58,13 @@ pub enum ProfileError { DuplicateId(String), #[error("provider profile '{id}' has invalid endpoint '{host}:{port}'")] InvalidEndpoint { id: String, host: String, port: u32 }, + /// An MCP endpoint declared a malformed exact revision allowlist. + #[error("provider profile '{id}' has invalid MCP configuration in '{field}': {message}")] + InvalidMcpConfiguration { + id: String, + field: String, + message: String, + }, #[error("provider profile '{id}' has duplicate credential env var '{env_var}'")] DuplicateCredentialEnvVar { id: String, env_var: String }, #[error("provider profile '{id}' validation error: {field}: {message}")] @@ -216,67 +228,251 @@ pub struct DiscoveryProfile { // is added to NetworkEndpoint, L7Rule, L7Allow, L7DenyRule, L7QueryMatcher, // GraphqlOperation, or NetworkBinary, add it here and in both conversion // directions unless the import/lint path explicitly rejects it. -#[derive(Debug, Clone, Deserialize, Serialize, PartialEq, Eq)] +#[derive(Debug, Clone, PartialEq, Eq)] #[allow( clippy::struct_excessive_bools, reason = "Endpoint profile mirrors independent policy schema toggles." )] pub struct EndpointProfile { pub host: String, - #[serde(default, skip_serializing_if = "is_zero")] pub port: u32, - #[serde(default, skip_serializing_if = "String::is_empty")] pub protocol: String, - #[serde(default, skip_serializing_if = "String::is_empty")] pub tls: String, - #[serde(default, skip_serializing_if = "String::is_empty")] pub access: String, - #[serde(default, skip_serializing_if = "String::is_empty")] pub enforcement: String, - #[serde(default, skip_serializing_if = "Option::is_none")] pub rules: Option>, - #[serde(default, skip_serializing_if = "Vec::is_empty")] pub allowed_ips: Vec, - #[serde(default, skip_serializing_if = "Vec::is_empty")] pub ports: Vec, - #[serde(default, skip_serializing_if = "Option::is_none")] pub deny_rules: Option>, - #[serde(default, skip_serializing_if = "is_false")] pub allow_encoded_slash: bool, - #[serde(default, skip_serializing_if = "is_false")] pub websocket_credential_rewrite: bool, - #[serde(default, skip_serializing_if = "is_false")] pub request_body_credential_rewrite: bool, - #[serde(default, skip_serializing_if = "is_false")] pub allow_uninspected_credentials: bool, - #[serde(default, skip_serializing_if = "String::is_empty")] pub persisted_queries: String, - #[serde(default, skip_serializing_if = "HashMap::is_empty")] pub graphql_persisted_queries: HashMap, - #[serde(default, skip_serializing_if = "is_zero")] pub graphql_max_body_bytes: u32, - #[serde(default, skip_serializing_if = "is_zero")] pub json_rpc_max_body_bytes: u32, - #[serde(default, skip_serializing_if = "Option::is_none")] + /// MCP-specific policy options for endpoints whose protocol is `mcp`. + /// + /// Omission materializes [`DEFAULT_MCP_PROTOCOL_VERSION`], while an + /// explicitly null value is rejected as an ambiguous authored contract. + /// Declaring an allowlist does not yet select or enforce a runtime wire + /// profile; later runtime configuration owns that boundary. pub mcp: Option, - #[serde(default, skip_serializing_if = "String::is_empty")] pub path: String, - #[serde(default, skip_serializing_if = "String::is_empty")] pub credential_signing: String, - #[serde(default, skip_serializing_if = "String::is_empty")] pub signing_service: String, - #[serde(default, skip_serializing_if = "String::is_empty")] pub signing_region: String, } -#[derive(Debug, Clone, Default, Deserialize, Serialize, PartialEq, Eq)] +// EndpointProfile needs the complete endpoint before it can decide whether an +// omitted MCP object is meaningful. Keep this remote Serde shape field-for- +// field with EndpointProfile so every direct and nested parsing route applies +// the same protocol-aware materialization after the wire shape is decoded. +#[derive(Deserialize, Serialize)] +#[allow( + clippy::struct_excessive_bools, + reason = "Endpoint profile mirror preserves independent policy schema toggles." +)] +#[serde(remote = "EndpointProfile")] +struct EndpointProfileSerde { + host: String, + #[serde(default, skip_serializing_if = "is_zero")] + port: u32, + #[serde(default, skip_serializing_if = "String::is_empty")] + protocol: String, + #[serde(default, skip_serializing_if = "String::is_empty")] + tls: String, + #[serde(default, skip_serializing_if = "String::is_empty")] + access: String, + #[serde(default, skip_serializing_if = "String::is_empty")] + enforcement: String, + #[serde(default, skip_serializing_if = "Option::is_none")] + rules: Option>, + #[serde(default, skip_serializing_if = "Vec::is_empty")] + allowed_ips: Vec, + #[serde(default, skip_serializing_if = "Vec::is_empty")] + ports: Vec, + #[serde(default, skip_serializing_if = "Option::is_none")] + deny_rules: Option>, + #[serde(default, skip_serializing_if = "is_false")] + allow_encoded_slash: bool, + #[serde(default, skip_serializing_if = "is_false")] + websocket_credential_rewrite: bool, + #[serde(default, skip_serializing_if = "is_false")] + request_body_credential_rewrite: bool, + #[serde(default, skip_serializing_if = "is_false")] + allow_uninspected_credentials: bool, + #[serde(default, skip_serializing_if = "String::is_empty")] + persisted_queries: String, + #[serde(default, skip_serializing_if = "HashMap::is_empty")] + graphql_persisted_queries: HashMap, + #[serde(default, skip_serializing_if = "is_zero")] + graphql_max_body_bytes: u32, + #[serde(default, skip_serializing_if = "is_zero")] + json_rpc_max_body_bytes: u32, + #[serde( + default, + deserialize_with = "deserialize_non_null_mcp_options", + skip_serializing_if = "Option::is_none" + )] + mcp: Option, + #[serde(default, skip_serializing_if = "String::is_empty")] + path: String, + #[serde(default, skip_serializing_if = "String::is_empty")] + credential_signing: String, + #[serde(default, skip_serializing_if = "String::is_empty")] + signing_service: String, + #[serde(default, skip_serializing_if = "String::is_empty")] + signing_region: String, +} + +impl<'de> Deserialize<'de> for EndpointProfile { + fn deserialize(deserializer: D) -> Result + where + D: Deserializer<'de>, + { + let mut endpoint = EndpointProfileSerde::deserialize(deserializer)?; + materialize_mcp_endpoint_defaults(&mut endpoint); + Ok(endpoint) + } +} + +impl Serialize for EndpointProfile { + fn serialize(&self, serializer: S) -> Result + where + S: Serializer, + { + let mut endpoint = self.clone(); + materialize_mcp_endpoint_defaults(&mut endpoint); + EndpointProfileSerde::serialize(&endpoint, serializer) + } +} + +/// Version-aware policy metadata for one MCP provider endpoint. +/// +/// Omitted versions materialize the pinned `OpenShell` default. Explicit values +/// must contain exact, supported, unique revisions, and deserialization stores +/// them in canonical semantic order. Runtime profile selection and enforcement +/// are deliberately outside this schema-only contract. +#[derive(Debug, Clone, Deserialize, Serialize, PartialEq, Eq)] +#[serde(deny_unknown_fields)] pub struct McpOptionsProfile { + /// Exact MCP protocol revisions this provider endpoint permits. + /// + /// Omission selects [`DEFAULT_MCP_PROTOCOL_VERSION`]. An explicitly empty + /// authored list is rejected so a likely authoring mistake cannot silently + /// become the default. + #[serde( + default = "default_mcp_profile_versions", + deserialize_with = "deserialize_mcp_profile_versions", + serialize_with = "serialize_mcp_profile_versions" + )] + pub versions: Vec, + /// Whether runtime inspection should enforce recommended MCP tool names. #[serde(default, skip_serializing_if = "Option::is_none")] pub strict_tool_names: Option, + /// Whether known MCP methods are allowed without explicit method rules. #[serde(default, skip_serializing_if = "Option::is_none")] pub allow_all_known_mcp_methods: Option, } +impl Default for McpOptionsProfile { + fn default() -> Self { + Self { + versions: default_mcp_profile_versions(), + strict_tool_names: None, + allow_all_known_mcp_methods: None, + } + } +} + +fn deserialize_non_null_mcp_options<'de, D>( + deserializer: D, +) -> Result, D::Error> +where + D: Deserializer<'de>, +{ + Option::::deserialize(deserializer)? + .map(Some) + .ok_or_else(|| de::Error::custom("mcp must be an object when present")) +} + +fn default_mcp_profile_versions() -> Vec { + vec![DEFAULT_MCP_PROTOCOL_VERSION.as_str().to_string()] +} + +fn validate_mcp_profile_versions( + values: &[String], +) -> Result, String> { + if values.is_empty() { + return Err( + "mcp.versions must contain at least one supported protocol version".to_string(), + ); + } + + let mut versions = BTreeSet::new(); + for value in values { + let version = value + .parse::() + .map_err(|error| format!("{error}; {MCP_VERSION_REMEDIATION}"))?; + if !versions.insert(version) { + return Err(format!("duplicate MCP protocol version '{value}'")); + } + } + + Ok(versions) +} + +fn deserialize_mcp_profile_versions<'de, D>(deserializer: D) -> Result, D::Error> +where + D: Deserializer<'de>, +{ + let values = Vec::::deserialize(deserializer)?; + let versions = validate_mcp_profile_versions(&values).map_err(de::Error::custom)?; + Ok(versions + .into_iter() + .map(|version| version.as_str().to_string()) + .collect()) +} + +fn serialize_mcp_profile_versions(values: &[String], serializer: S) -> Result +where + S: Serializer, +{ + let default_versions; + let values = if values.is_empty() { + // Programmatic and protobuf callers cannot preserve authored field + // presence. Treat their empty representation as omission at the + // checked serialization boundary and emit the pinned explicit value. + default_versions = default_mcp_profile_versions(); + default_versions.as_slice() + } else { + values + }; + let versions = + validate_mcp_profile_versions(values).map_err(::custom)?; + versions + .into_iter() + .map(McpProtocolVersion::as_str) + .collect::>() + .serialize(serializer) +} + +fn materialize_mcp_endpoint_defaults(endpoint: &mut EndpointProfile) { + if !is_mcp_protocol(&endpoint.protocol) { + return; + } + + let options = endpoint.mcp.get_or_insert_with(McpOptionsProfile::default); + if options.versions.is_empty() { + // Empty protobuf and programmatic values cannot encode whether the + // author omitted the field. Normalize them to the same fixed contract + // as an omitted authored value before persistence or lowering. + options.versions = default_mcp_profile_versions(); + } +} + #[derive(Debug, Clone, Deserialize, Serialize, PartialEq, Eq)] pub struct L7RuleProfile { #[serde(default, skip_serializing_if = "Option::is_none")] @@ -492,13 +688,25 @@ impl ProviderTypeProfile { /// Whether this profile can be created without initial static credentials. /// - /// Empty provider creation is allowed when at least one credential can be - /// resolved at runtime, and every required credential can be resolved at - /// runtime. Runtime-resolvable credentials are either gateway-mintable - /// refresh credentials, sandbox-side dynamic token grants, or additional - /// outputs co-minted by another credential's gateway-mintable refresh. + /// Empty provider creation is allowed when every required credential can + /// be resolved at runtime. This includes profiles with no credentials and + /// profiles whose credentials are all optional. #[must_use] pub fn allows_empty_provider_credentials(&self) -> bool { + let co_minted = self.co_minted_credential_names(); + self.credentials.iter().all(|credential| { + let is_runtime_resolvable = + credential.is_runtime_resolvable() || co_minted.contains(credential.name.as_str()); + !credential.required || is_runtime_resolvable + }) + } + + /// Whether `--runtime-credentials` is meaningful for this profile. + /// + /// At least one credential must be runtime-resolvable and every required + /// credential must be resolvable without an initial static value. + #[must_use] + pub fn allows_runtime_provider_credentials(&self) -> bool { let co_minted = self.co_minted_credential_names(); let mut has_runtime_resolvable_credential = false; for credential in &self.credentials { @@ -512,6 +720,20 @@ impl ProviderTypeProfile { has_runtime_resolvable_credential } + /// Required credentials that must have an initial static value. + #[must_use] + pub fn required_static_credentials(&self) -> Vec<&CredentialProfile> { + let co_minted = self.co_minted_credential_names(); + self.credentials + .iter() + .filter(|credential| { + credential.required + && !credential.is_runtime_resolvable() + && !co_minted.contains(credential.name.as_str()) + }) + .collect() + } + /// Names of credentials produced as `additional_outputs` of a /// gateway-mintable refresh on some other credential. fn co_minted_credential_names(&self) -> HashSet<&str> { @@ -626,6 +848,7 @@ impl ProviderTypeProfile { pub fn validate_before_lowering(&self, source: &str) -> Vec { let mut diagnostics = Vec::new(); for (index, endpoint) in self.endpoints.iter().enumerate() { + collect_mcp_profile_diagnostics(source, &self.id, index, endpoint, &mut diagnostics); if endpoint.rules.as_ref().is_some_and(Vec::is_empty) { diagnostics.push(ProfileValidationDiagnostic::error( source, @@ -645,7 +868,7 @@ impl ProviderTypeProfile { )); } - if endpoint.protocol != "mcp" { + if !is_mcp_protocol(&endpoint.protocol) { continue; } if let Some(rules) = &endpoint.rules { @@ -702,6 +925,21 @@ fn is_u64_zero(value: &u64) -> bool { } impl CredentialProfile { + /// Keys accepted when storing this credential on a provider. + /// + /// Workload-injectable credentials use their declared environment aliases. + /// Broker-only credentials have no environment aliases and use their + /// logical profile name instead. + #[must_use] + pub fn accepted_stored_keys(&self) -> Vec<&str> { + if self.env_vars.is_empty() { + let name = self.name.trim(); + return (!name.is_empty()).then_some(name).into_iter().collect(); + } + + self.env_vars.iter().map(String::as_str).collect() + } + #[must_use] pub fn is_runtime_resolvable(&self) -> bool { self.token_grant.is_some() @@ -1225,7 +1463,7 @@ fn endpoint_to_proto(endpoint: &EndpointProfile) -> NetworkEndpoint { .collect(), graphql_max_body_bytes: endpoint.graphql_max_body_bytes, json_rpc_max_body_bytes: endpoint.json_rpc_max_body_bytes, - mcp: endpoint.mcp.as_ref().map(mcp_options_to_proto), + mcp: endpoint_mcp_options_to_proto(endpoint), path: endpoint.path.clone(), credential_signing: endpoint.credential_signing.clone(), signing_service: endpoint.signing_service.clone(), @@ -1237,7 +1475,7 @@ fn endpoint_to_proto(endpoint: &EndpointProfile) -> NetworkEndpoint { } fn endpoint_from_proto(endpoint: &NetworkEndpoint) -> EndpointProfile { - EndpointProfile { + let mut profile = EndpointProfile { host: endpoint.host.clone(), port: endpoint.port, protocol: endpoint.protocol.clone(), @@ -1274,28 +1512,70 @@ fn endpoint_from_proto(endpoint: &NetworkEndpoint) -> EndpointProfile { .collect(), graphql_max_body_bytes: endpoint.graphql_max_body_bytes, json_rpc_max_body_bytes: endpoint.json_rpc_max_body_bytes, - mcp: endpoint.mcp.map(mcp_options_from_proto), + mcp: endpoint.mcp.as_ref().map(mcp_options_from_proto), path: endpoint.path.clone(), credential_signing: endpoint.credential_signing.clone(), signing_service: endpoint.signing_service.clone(), signing_region: endpoint.signing_region.clone(), + }; + materialize_mcp_endpoint_defaults(&mut profile); + profile +} + +fn endpoint_mcp_options_to_proto(endpoint: &EndpointProfile) -> Option { + match endpoint.mcp.as_ref() { + Some(options) => Some(mcp_options_to_proto(options)), + None if is_mcp_protocol(&endpoint.protocol) => { + Some(mcp_options_to_proto(&McpOptionsProfile::default())) + } + None => None, } } fn mcp_options_to_proto(options: &McpOptionsProfile) -> McpOptions { + let mut versions = options.versions.clone(); + materialize_and_canonicalize_mcp_profile_versions(&mut versions); McpOptions { + versions, strict_tool_names: options.strict_tool_names, allow_all_known_mcp_methods: options.allow_all_known_mcp_methods, } } -fn mcp_options_from_proto(options: McpOptions) -> McpOptionsProfile { +fn mcp_options_from_proto(options: &McpOptions) -> McpOptionsProfile { + let mut versions = options.versions.clone(); + materialize_and_canonicalize_mcp_profile_versions(&mut versions); McpOptionsProfile { + versions, strict_tool_names: options.strict_tool_names, allow_all_known_mcp_methods: options.allow_all_known_mcp_methods, } } +fn materialize_and_canonicalize_mcp_profile_versions(versions: &mut Vec) { + if versions.is_empty() { + *versions = default_mcp_profile_versions(); + } else { + canonicalize_mcp_profile_versions(versions); + } +} + +fn canonicalize_mcp_profile_versions(versions: &mut [String]) { + // Preserve unsupported and duplicate values so subsequent validation can + // reject them; sorting must never repair malformed protobuf input. + versions.sort_by(|left, right| { + match ( + left.parse::(), + right.parse::(), + ) { + (Ok(left), Ok(right)) => left.cmp(&right), + (Ok(_), Err(_)) => std::cmp::Ordering::Less, + (Err(_), Ok(_)) => std::cmp::Ordering::Greater, + (Err(_), Err(_)) => left.cmp(right), + } + }); +} + fn binary_to_proto(binary: &BinaryProfile) -> NetworkBinary { NetworkBinary { path: binary.path.clone(), @@ -1534,6 +1814,10 @@ pub fn parse_profile_catalog_yamls( Ok(profiles) } +fn is_mcp_diagnostic_field(field: &str) -> bool { + field.split('.').any(|segment| segment == "mcp") +} + fn validate_profiles(profiles: &[ProviderTypeProfile]) -> Result<(), ProfileError> { let diagnostics = validate_profile_set( &profiles @@ -1541,50 +1825,56 @@ fn validate_profiles(profiles: &[ProviderTypeProfile]) -> Result<(), ProfileErro .map(|profile| (String::new(), profile.clone())) .collect::>(), ); - if let Some(diagnostic) = diagnostics.first() { - if diagnostic.field == "id" && diagnostic.message == "provider profile id is required" { - return Err(ProfileError::MissingId); - } - if diagnostic.field == "id" - && diagnostic + let Some(diagnostic) = diagnostics.first() else { + return Ok(()); + }; + if diagnostic.field == "id" && diagnostic.message == "provider profile id is required" { + return Err(ProfileError::MissingId); + } + if diagnostic.field == "id" + && diagnostic + .message + .starts_with("duplicate provider profile id") + { + return Err(ProfileError::DuplicateId(diagnostic.profile_id.clone())); + } + if diagnostic.field.starts_with("credentials.env_vars") { + return Err(ProfileError::DuplicateCredentialEnvVar { + id: diagnostic.profile_id.clone(), + env_var: diagnostic .message - .starts_with("duplicate provider profile id") - { - return Err(ProfileError::DuplicateId(diagnostic.profile_id.clone())); - } - if diagnostic.field.starts_with("credentials.env_vars") { - return Err(ProfileError::DuplicateCredentialEnvVar { - id: diagnostic.profile_id.clone(), - env_var: diagnostic - .message - .trim_start_matches("duplicate credential env var '") - .trim_end_matches('\'') - .to_string(), - }); - } - if diagnostic.field.starts_with("endpoints") - && let Some(profile) = profiles - .iter() - .find(|profile| profile.id == diagnostic.profile_id) - && let Some(endpoint) = profile - .endpoints - .iter() - .find(|endpoint| !endpoint_is_valid(endpoint)) - { - return Err(ProfileError::InvalidEndpoint { - id: profile.id.clone(), - host: endpoint.host.clone(), - port: endpoint.port, - }); - } - return Err(ProfileError::ValidationError { + .trim_start_matches("duplicate credential env var '") + .trim_end_matches('\'') + .to_string(), + }); + } + if is_mcp_diagnostic_field(&diagnostic.field) { + return Err(ProfileError::InvalidMcpConfiguration { id: diagnostic.profile_id.clone(), field: diagnostic.field.clone(), message: diagnostic.message.clone(), }); } - - Ok(()) + if diagnostic.field.starts_with("endpoints") + && let Some(profile) = profiles + .iter() + .find(|profile| profile.id == diagnostic.profile_id) + && let Some(endpoint) = profile + .endpoints + .iter() + .find(|endpoint| !endpoint_is_valid(endpoint)) + { + return Err(ProfileError::InvalidEndpoint { + id: profile.id.clone(), + host: endpoint.host.clone(), + port: endpoint.port, + }); + } + Err(ProfileError::ValidationError { + id: diagnostic.profile_id.clone(), + field: diagnostic.field.clone(), + message: diagnostic.message.clone(), + }) } #[must_use] @@ -2006,6 +2296,7 @@ pub fn validate_profile_set( format!("invalid endpoint '{}:{}'", endpoint.host, endpoint.port), )); } + collect_mcp_profile_diagnostics(source, profile_id, index, endpoint, &mut diagnostics); if endpoint.rules.as_ref().is_some_and(Vec::is_empty) { diagnostics.push(ProfileValidationDiagnostic::error( @@ -2064,7 +2355,7 @@ pub fn validate_profile_set( )); } - if endpoint.protocol == "mcp" { + if is_mcp_protocol(&endpoint.protocol) { let strict_tool_names = endpoint .mcp .as_ref() @@ -2302,14 +2593,6 @@ pub fn validate_profile_set( } } } else { - if endpoint.mcp.is_some() { - diagnostics.push(ProfileValidationDiagnostic::error( - source, - profile_id, - format!("endpoints[{index}]"), - "mcp options are only valid for protocol mcp", - )); - } if let Some(rules) = &endpoint.rules { for (rule_idx, rule) in rules.iter().enumerate() { if let Some(allow) = &rule.allow { @@ -2390,6 +2673,66 @@ pub fn validate_profile_set( diagnostics } +fn collect_mcp_profile_diagnostics( + source: &str, + profile_id: &str, + endpoint_index: usize, + endpoint: &EndpointProfile, + diagnostics: &mut Vec, +) { + let mcp_field = format!("endpoints[{endpoint_index}].mcp"); + if !is_mcp_protocol(&endpoint.protocol) { + if endpoint.mcp.is_some() { + diagnostics.push(ProfileValidationDiagnostic::error( + source, + profile_id, + mcp_field, + "mcp options are only valid for protocol mcp", + )); + } + return; + } + + let Some(options) = endpoint.mcp.as_ref() else { + // Programmatic callers may construct the pre-materialized shape. + // Lowering and serialization bind it to the pinned default. + return; + }; + if options.versions.is_empty() { + // Protobuf repeated fields collapse omission and explicit emptiness. + // The conversion boundary materializes both as the pinned default. + return; + } + + let versions_field = format!("{mcp_field}.versions"); + let mut seen = HashSet::new(); + for version in &options.versions { + if !seen.insert(version.as_str()) { + diagnostics.push(ProfileValidationDiagnostic::error( + source, + profile_id, + versions_field.clone(), + format!("duplicate MCP protocol version '{version}'"), + )); + } + if version.parse::().is_err() { + diagnostics.push(ProfileValidationDiagnostic::error( + source, + profile_id, + versions_field.clone(), + format!("unsupported MCP protocol version '{version}'; {MCP_VERSION_REMEDIATION}"), + )); + } + } +} + +// Protocol parsing is case-insensitive throughout the shared policy schema. +// Provider validation must use the same predicate so preflight and lowering +// cannot disagree about whether the MCP options are required or misplaced. +fn is_mcp_protocol(protocol: &str) -> bool { + matches!(L7Protocol::parse(protocol), Some(L7Protocol::Mcp)) +} + fn endpoint_is_valid(endpoint: &EndpointProfile) -> bool { if endpoint.host.trim().is_empty() { return false; @@ -2911,12 +3254,14 @@ pub fn builtin_profiles() -> &'static [ProviderTypeProfile] { mod tests { use std::collections::HashMap; + use openshell_core::mcp::{DEFAULT_MCP_PROTOCOL_VERSION, McpProtocolVersion}; use openshell_core::proto::{ProviderCredentialTokenGrantType, ProviderProfileCategory}; use super::{ - DiscoveryProfile, L7AllowProfile, L7QueryMatcherProfile, ProfileError, ProviderTypeProfile, - builtin_profiles, normalize_profile_id, parse_profile_catalog_yamls, parse_profile_json, - parse_profile_yaml, profile_to_json, profile_to_yaml, validate_profile_set, + DiscoveryProfile, EndpointProfile, L7AllowProfile, L7QueryMatcherProfile, ProfileError, + ProviderTypeProfile, builtin_profiles, is_mcp_diagnostic_field, normalize_profile_id, + parse_profile_catalog_yamls, parse_profile_json, parse_profile_yaml, profile_to_json, + profile_to_yaml, profiles_to_json, profiles_to_yaml, validate_profile_set, }; fn builtin_profile(id: &str) -> &'static ProviderTypeProfile { @@ -3042,6 +3387,31 @@ mod tests { ); } + #[test] + fn accepted_stored_keys_use_logical_name_for_broker_only_credentials() { + let profile = parse_profile_yaml( + r" +id: token-exchange +display_name: Token Exchange +credentials: + - name: subject_token + required: true + - name: access_token + env_vars: [ACCESS_TOKEN, ACCESS_TOKEN_FALLBACK] +", + ) + .expect("profile"); + + assert_eq!( + profile.credentials[0].accepted_stored_keys(), + vec!["subject_token"] + ); + assert_eq!( + profile.credentials[1].accepted_stored_keys(), + vec!["ACCESS_TOKEN", "ACCESS_TOKEN_FALLBACK"] + ); + } + #[test] fn vertex_profile_declares_discovery_and_fallback_token_env_vars() { let profile = builtin_profile("google-vertex-ai"); @@ -3081,8 +3451,7 @@ mod tests { } #[test] - fn empty_provider_credentials_require_a_runtime_resolvable_path_and_no_required_static_credentials() - { + fn empty_provider_credentials_require_no_required_static_credentials() { let optional_refresh_profile = parse_profile_yaml( r" id: optional-refresh @@ -3096,6 +3465,7 @@ credentials: ) .expect("profile"); assert!(optional_refresh_profile.allows_empty_provider_credentials()); + assert!(optional_refresh_profile.allows_runtime_provider_credentials()); let token_grant_profile = parse_profile_yaml( r" @@ -3110,6 +3480,7 @@ credentials: ) .expect("profile"); assert!(token_grant_profile.allows_empty_provider_credentials()); + assert!(token_grant_profile.allows_runtime_provider_credentials()); let mixed_required_profile = parse_profile_yaml( r" @@ -3126,6 +3497,7 @@ credentials: ) .expect("profile"); assert!(!mixed_required_profile.allows_empty_provider_credentials()); + assert!(!mixed_required_profile.allows_runtime_provider_credentials()); let static_only_profile = parse_profile_yaml( r" @@ -3137,7 +3509,21 @@ credentials: ", ) .expect("profile"); - assert!(!static_only_profile.allows_empty_provider_credentials()); + assert!(static_only_profile.allows_empty_provider_credentials()); + assert!(!static_only_profile.allows_runtime_provider_credentials()); + + let policy_only_profile = parse_profile_yaml( + r" +id: policy-only +display_name: Policy Only +endpoints: + - host: example.com + port: 443 +", + ) + .expect("profile"); + assert!(policy_only_profile.allows_empty_provider_credentials()); + assert!(!policy_only_profile.allows_runtime_provider_credentials()); } #[test] @@ -3249,9 +3635,9 @@ discovery: } #[test] - fn mcp_endpoint_strict_tool_names_round_trips_through_proto_and_yaml() { + fn mcp_endpoint_contract_round_trips_through_policy_proto_and_yaml() { let profile = parse_profile_yaml( - r" + r#" id: mcp-example display_name: MCP Example endpoints: @@ -3260,21 +3646,64 @@ endpoints: path: /mcp protocol: mcp mcp: + versions: ["2025-11-25", "2025-03-26", "2025-06-18"] strict_tool_names: false binaries: - /usr/bin/example-agent -", +"#, ) .expect("profile should parse"); + let expected_versions = ["2025-03-26", "2025-06-18", "2025-11-25"]; assert_eq!( profile.endpoints[0] .mcp .as_ref() - .and_then(|options| options.strict_tool_names), - Some(false) + .expect("MCP options") + .versions, + expected_versions + ); + let mut proto = profile.to_proto(); + assert_eq!( + proto.endpoints[0] + .mcp + .as_ref() + .expect("MCP options") + .versions, + expected_versions + ); + proto.endpoints[0] + .mcp + .as_mut() + .expect("MCP options") + .versions + .reverse(); + assert_eq!( + profile.network_policy_rule("provider").endpoints[0] + .mcp + .as_ref() + .expect("MCP options") + .versions, + expected_versions + ); + + let from_proto = ProviderTypeProfile::from_proto(&proto); + assert_eq!( + from_proto.endpoints[0] + .mcp + .as_ref() + .expect("MCP options") + .versions, + expected_versions + ); + assert_eq!( + from_proto.to_proto().endpoints[0] + .mcp + .as_ref() + .expect("MCP options") + .versions, + expected_versions ); - let from_proto = ProviderTypeProfile::from_proto(&profile.to_proto()); assert_eq!( from_proto.endpoints[0] .mcp @@ -3285,9 +3714,150 @@ binaries: let exported = profile_to_yaml(&from_proto).expect("yaml"); assert!(exported.contains("mcp:")); + let positions = expected_versions.map(|version| { + exported + .find(version) + .expect("serialized version must be present") + }); + assert!(positions.windows(2).all(|pair| pair[0] < pair[1])); assert!(exported.contains("strict_tool_names: false")); } + fn mcp_profile_for_serialization() -> ProviderTypeProfile { + parse_profile_yaml( + r#" +id: mcp-example +display_name: MCP Example +endpoints: + - host: mcp.example.com + port: 443 + protocol: mcp + mcp: + versions: ["2025-03-26", "2025-11-25"] +"#, + ) + .expect("valid MCP profile") + } + + #[test] + fn provider_serializers_canonicalize_programmatic_mcp_versions() { + let mut profile = mcp_profile_for_serialization(); + profile.endpoints[0] + .mcp + .as_mut() + .expect("MCP options") + .versions + .reverse(); + + let outputs = [ + profile_to_yaml(&profile).expect("single YAML"), + profile_to_json(&profile).expect("single JSON"), + profiles_to_yaml(std::slice::from_ref(&profile)).expect("catalog YAML"), + profiles_to_json(std::slice::from_ref(&profile)).expect("catalog JSON"), + ]; + + for output in outputs { + let legacy = output.find("2025-03-26").expect("legacy revision"); + let current = output.find("2025-11-25").expect("current revision"); + assert!( + legacy < current, + "serialized revisions were not canonical: {output}" + ); + } + } + + #[test] + fn provider_serializers_reject_invalid_programmatic_mcp_versions() { + let mut profile = mcp_profile_for_serialization(); + + for invalid in [ + vec!["2025-03-26".to_string(), "2025-03-26".to_string()], + vec!["latest".to_string()], + ] { + profile.endpoints[0] + .mcp + .as_mut() + .expect("MCP options") + .versions = invalid; + + assert!(profile_to_yaml(&profile).is_err()); + assert!(profile_to_json(&profile).is_err()); + assert!(profiles_to_yaml(std::slice::from_ref(&profile)).is_err()); + assert!(profiles_to_json(std::slice::from_ref(&profile)).is_err()); + } + } + + #[test] + fn provider_boundaries_materialize_programmatic_and_protobuf_empty_versions() { + let mut profile = mcp_profile_for_serialization(); + profile.endpoints[0] + .mcp + .as_mut() + .expect("MCP options") + .versions + .clear(); + + let expected = [DEFAULT_MCP_PROTOCOL_VERSION.as_str()]; + assert!( + profile + .validate_before_lowering("programmatic-empty.yaml") + .iter() + .all(|diagnostic| !is_mcp_diagnostic_field(&diagnostic.field)), + "programmatic emptiness must use the pinned default" + ); + let yaml = profile_to_yaml(&profile).expect("single YAML"); + let json = profile_to_json(&profile).expect("single JSON"); + for reparsed in [ + parse_profile_yaml(&yaml).expect("materialized YAML must parse"), + parse_profile_json(&json).expect("materialized JSON must parse"), + ] { + assert_eq!( + reparsed.endpoints[0] + .mcp + .as_ref() + .expect("MCP options") + .versions, + expected + ); + } + for output in [ + yaml, + json, + profiles_to_yaml(std::slice::from_ref(&profile)).expect("catalog YAML"), + profiles_to_json(std::slice::from_ref(&profile)).expect("catalog JSON"), + ] { + assert!(output.contains("versions")); + assert!(output.contains(DEFAULT_MCP_PROTOCOL_VERSION.as_str())); + } + + let proto = profile.to_proto(); + assert_eq!( + proto.endpoints[0] + .mcp + .as_ref() + .expect("MCP options") + .versions, + expected + ); + + let mut proto_with_empty_versions = proto; + proto_with_empty_versions.endpoints[0] + .mcp + .as_mut() + .expect("MCP options") + .versions + .clear(); + let from_proto = ProviderTypeProfile::from_proto(&proto_with_empty_versions); + assert_eq!( + from_proto.endpoints[0] + .mcp + .as_ref() + .expect("materialized MCP options") + .versions, + expected + ); + } + #[test] fn mcp_allow_params_round_trips_through_proto() { let profile = parse_profile_yaml( @@ -3299,6 +3869,7 @@ endpoints: port: 443 protocol: mcp mcp: + versions: ['2025-11-25'] allow_all_known_mcp_methods: true rules: - allow: @@ -3335,6 +3906,480 @@ binaries: ); } + #[test] + fn mcp_endpoint_profile_direct_serde_materializes_and_emits_the_pinned_default() { + let endpoint_yaml = r" +host: mcp.example.com +port: 443 +protocol: mcp +"; + let endpoint = serde_yml::from_str::(endpoint_yaml) + .expect("direct endpoint omission must default"); + let yaml_vector = serde_yml::from_str::>(&format!( + "- {}", + endpoint_yaml.trim_start().replace('\n', "\n ") + )) + .expect("YAML endpoint vector omission must default"); + let json_vector = serde_json::from_str::>( + r#"[{"host":"mcp.example.com","port":443,"protocol":"mcp"},{"host":"mcp.example.com","port":443,"protocol":"mcp","mcp":{}}]"#, + ) + .expect("JSON endpoint vector omissions must default"); + + let expected = [DEFAULT_MCP_PROTOCOL_VERSION.as_str()]; + for parsed in [&endpoint, &yaml_vector[0], &json_vector[0], &json_vector[1]] { + assert_eq!( + parsed + .mcp + .as_ref() + .expect("direct endpoint Serde must materialize MCP options") + .versions, + expected + ); + } + + let mut programmatic = endpoint; + programmatic.mcp = None; + let emitted_yaml = serde_yml::to_string(&programmatic) + .expect("direct endpoint YAML serialization must materialize"); + let emitted_json = serde_json::to_string(&vec![programmatic.clone()]) + .expect("direct endpoint vector JSON serialization must materialize"); + for emitted in [&emitted_yaml, &emitted_json] { + assert!(emitted.contains("mcp")); + assert!(emitted.contains("versions")); + assert!(emitted.contains(DEFAULT_MCP_PROTOCOL_VERSION.as_str())); + } + + programmatic.mcp = Some(super::McpOptionsProfile { + versions: vec!["2025-11-25".to_string(), "2025-03-26".to_string()], + strict_tool_names: None, + allow_all_known_mcp_methods: None, + }); + let canonical = serde_json::to_string(&programmatic) + .expect("direct endpoint serialization must canonicalize revisions"); + assert!( + canonical.find("2025-03-26").expect("legacy revision") + < canonical.find("2025-11-25").expect("current revision") + ); + + for invalid in [ + vec!["2025-03-26".to_string(), "2025-03-26".to_string()], + vec!["2025-03-26 ".to_string()], + vec!["draft".to_string()], + ] { + programmatic.mcp.as_mut().expect("MCP options").versions = invalid; + assert!(serde_yml::to_string(&programmatic).is_err()); + assert!(serde_json::to_string(&vec![programmatic.clone()]).is_err()); + } + } + + #[test] + fn mcp_endpoint_profile_direct_vector_serde_rejects_malformed_presence_and_versions() { + for invalid_mcp in [ + "null", + "{versions: null}", + "{versions: []}", + "{versions: ['2025-03-26', '2025-03-26']}", + "{versions: ['2025-03-26 ']}", + "{versions: [draft]}", + "{versions: [latest]}", + ] { + let yaml = format!( + "- host: mcp.example.com\n port: 443\n protocol: mcp\n mcp: {invalid_mcp}\n" + ); + assert!( + serde_yml::from_str::>(&yaml).is_err(), + "direct endpoint vector unexpectedly accepted {invalid_mcp}" + ); + } + + for json in [ + r#"[{"host":"mcp.example.com","protocol":"mcp","mcp":null}]"#, + r#"[{"host":"mcp.example.com","protocol":"mcp","mcp":{"versions":null}}]"#, + r#"[{"host":"mcp.example.com","protocol":"mcp","mcp":{"versions":[]}}]"#, + r#"[{"host":"mcp.example.com","protocol":"mcp","mcp":{"versions":["2025-03-26","2025-03-26"]}}]"#, + r#"[{"host":"mcp.example.com","protocol":"mcp","mcp":{"versions":["2025-03-26 "]}}]"#, + r#"[{"host":"mcp.example.com","protocol":"mcp","mcp":{"versions":["draft"]}}]"#, + r#"[{"host":"mcp.example.com","protocol":"mcp","mcp":{"versions":["latest"]}}]"#, + ] { + assert!(serde_json::from_str::>(json).is_err()); + } + } + + #[test] + fn mcp_endpoint_profile_materializes_omitted_versions_to_the_pinned_default() { + let omitted_options = r" +id: mcp-example +display_name: MCP Example +endpoints: + - host: mcp.example.com + port: 443 + protocol: mcp +"; + let omitted_versions = r" +id: mcp-example +display_name: MCP Example +endpoints: + - host: mcp.example.com + port: 443 + protocol: mcp + mcp: {} +"; + let explicit_default = format!( + r" +id: mcp-example +display_name: MCP Example +endpoints: + - host: mcp.example.com + port: 443 + protocol: mcp + mcp: + versions: ['{}'] +", + DEFAULT_MCP_PROTOCOL_VERSION.as_str() + ); + + let omitted_options = + parse_profile_yaml(omitted_options).expect("omitted MCP options must default"); + let omitted_versions = + parse_profile_yaml(omitted_versions).expect("omitted versions must default"); + let omitted_options_json = parse_profile_json( + r#"{"id":"mcp-example","display_name":"MCP Example","endpoints":[{"host":"mcp.example.com","port":443,"protocol":"mcp"}]}"#, + ) + .expect("omitted JSON MCP options must default"); + let omitted_versions_json = parse_profile_json( + r#"{"id":"mcp-example","display_name":"MCP Example","endpoints":[{"host":"mcp.example.com","port":443,"protocol":"mcp","mcp":{}}]}"#, + ) + .expect("omitted JSON versions must default"); + let explicit_default = + parse_profile_yaml(&explicit_default).expect("explicit default must parse"); + assert_eq!(omitted_options, omitted_versions); + assert_eq!(omitted_options, omitted_options_json); + assert_eq!(omitted_options, omitted_versions_json); + assert_eq!(omitted_options, explicit_default); + + let versions = &omitted_options.endpoints[0] + .mcp + .as_ref() + .expect("omitted MCP options must materialize") + .versions; + assert_eq!( + DEFAULT_MCP_PROTOCOL_VERSION, + McpProtocolVersion::V2025_11_25, + "provider omission must stay pinned to the declared default" + ); + assert_eq!(versions, &[DEFAULT_MCP_PROTOCOL_VERSION.as_str()]); + assert_ne!( + versions.len(), + McpProtocolVersion::ALL.len(), + "the default must not expand to every known revision" + ); + assert!( + omitted_options + .validate_before_lowering("omitted-options.yaml") + .iter() + .all(|diagnostic| !is_mcp_diagnostic_field(&diagnostic.field)), + "the materialized default must pass MCP validation" + ); + + let exported = profile_to_yaml(&omitted_options).expect("materialized YAML"); + assert!(exported.contains("mcp:")); + assert!(exported.contains("versions:")); + assert!(exported.contains(DEFAULT_MCP_PROTOCOL_VERSION.as_str())); + + let proto = omitted_options.to_proto(); + assert_eq!( + proto.endpoints[0] + .mcp + .as_ref() + .expect("materialized MCP options") + .versions, + [DEFAULT_MCP_PROTOCOL_VERSION.as_str()] + ); + } + + #[test] + fn mcp_endpoint_profile_rejects_explicit_empty_or_invalid_versions() { + for invalid_mcp in [ + "versions: []", + "versions: null", + "versions: [\"2025-03-26\", \"2025-03-26\"]", + "versions: [latest]", + "versions: [draft]", + "versions: ['2026-07-28']", + "versions: [\"2025-03-26 \"]", + ] { + let yaml = format!( + r" +id: mcp-example +display_name: MCP Example +endpoints: + - host: mcp.example.com + port: 443 + protocol: mcp + mcp: + {invalid_mcp} +" + ); + assert!( + parse_profile_yaml(&yaml).is_err(), + "invalid MCP profile unexpectedly parsed: {invalid_mcp}" + ); + } + + let misplaced_options = r" +id: rest-example +display_name: REST Example +endpoints: + - host: api.example.com + port: 443 + protocol: rest + mcp: + versions: ['2025-11-25'] +"; + assert!(matches!( + parse_profile_catalog_yamls(&[misplaced_options]), + Err(ProfileError::InvalidMcpConfiguration { field, .. }) + if field == "endpoints[0].mcp" + )); + } + + #[test] + fn mcp_endpoint_profile_rejects_explicit_null_options_across_parsing_routes() { + let yaml = r" +id: mcp-example +display_name: MCP Example +endpoints: + - host: mcp.example.com + port: 443 + protocol: mcp + mcp: null +"; + assert!(parse_profile_yaml(yaml).is_err()); + assert!(parse_profile_catalog_yamls(&[yaml]).is_err()); + + let value = + serde_yml::from_str::(yaml).expect("generic YAML value must parse"); + assert!(serde_yml::from_value::(&value).is_err()); + + for json in [ + r#"{"id":"mcp-example","display_name":"MCP Example","endpoints":[{"host":"mcp.example.com","port":443,"protocol":"mcp","mcp":null}]}"#, + r#"{"id":"mcp-example","display_name":"MCP Example","endpoints":[{"host":"mcp.example.com","port":443,"protocol":"mcp","mcp":{"versions":null}}]}"#, + ] { + assert!(parse_profile_json(json).is_err()); + } + } + + #[test] + fn endpoint_profile_preserves_unknown_field_tolerance() { + let endpoint_yaml = r" +host: api.example.com +port: 443 +protocol: rest +access: full +future_endpoint_option: true +"; + let endpoint_json = r#"{"host":"api.example.com","port":443,"protocol":"rest","access":"full","future_endpoint_option":true}"#; + + let yaml_endpoint = serde_yml::from_str::(endpoint_yaml) + .expect("unknown endpoint fields remain forward-compatible in YAML"); + let json_endpoint = serde_json::from_str::(endpoint_json) + .expect("unknown endpoint fields remain forward-compatible in JSON"); + assert_eq!(yaml_endpoint.host, "api.example.com"); + assert_eq!(json_endpoint, yaml_endpoint); + + let profile_yaml = format!( + "id: future-profile\ndisplay_name: Future profile\nendpoints:\n - {}", + endpoint_yaml.trim_start().replace('\n', "\n ") + ); + parse_profile_yaml(&profile_yaml) + .expect("nested endpoint parsing must retain the prior unknown-field tolerance"); + } + + #[test] + fn mcp_options_profile_rejects_unknown_fields_across_parsing_routes() { + let endpoint_yaml = r" +host: mcp.example.com +port: 443 +protocol: mcp +mcp: + version: ['2025-11-25'] +"; + let endpoint_json = r#"{"host":"mcp.example.com","port":443,"protocol":"mcp","mcp":{"versionss":["2025-11-25"]}}"#; + + assert!(serde_yml::from_str::(endpoint_yaml).is_err()); + assert!(serde_json::from_str::(endpoint_json).is_err()); + + let endpoint_vector_yaml = + format!("- {}", endpoint_yaml.trim_start().replace('\n', "\n ")); + assert!(serde_yml::from_str::>(&endpoint_vector_yaml).is_err()); + assert!( + serde_json::from_str::>(&format!("[{endpoint_json}]")).is_err() + ); + + let profile_yaml = format!( + "id: mcp-example\ndisplay_name: MCP Example\nendpoints:\n - {}", + endpoint_yaml.trim_start().replace('\n', "\n ") + ); + assert!(parse_profile_yaml(&profile_yaml).is_err()); + let value = serde_yml::from_str::(&profile_yaml) + .expect("generic YAML value must parse"); + assert!(serde_yml::from_value::(&value).is_err()); + + let profile_json = format!( + r#"{{"id":"mcp-example","display_name":"MCP Example","endpoints":[{endpoint_json}]}}"# + ); + assert!(parse_profile_json(&profile_json).is_err()); + } + + #[test] + fn unsupported_provider_mcp_version_explains_default_and_explicit_l4_escape_hatch() { + const DEFAULT_REMEDIATION: &str = "omit mcp.versions to use the pinned default revision"; + const L4_REMEDIATION: &str = + "omit protocol and mcp for deliberate uninspected L4 passthrough"; + + let yaml = r" +id: mcp-example +display_name: MCP Example +endpoints: + - host: mcp.example.com + port: 443 + protocol: mcp + mcp: + versions: [draft] +"; + let error = parse_profile_yaml(yaml) + .expect_err("moving draft aliases must fail closed") + .to_string(); + assert!(error.contains(DEFAULT_REMEDIATION)); + assert!(error.contains(L4_REMEDIATION)); + + let mut profile = parse_profile_yaml(&yaml.replace("draft", "2025-11-25")) + .expect("supported revision must parse"); + profile.endpoints[0] + .mcp + .as_mut() + .expect("MCP options") + .versions = vec!["draft".to_string()]; + let diagnostics = profile.validate_before_lowering("draft-version.yaml"); + assert!(diagnostics.iter().any(|diagnostic| { + diagnostic.field == "endpoints[0].mcp.versions" + && diagnostic.message.contains(DEFAULT_REMEDIATION) + && diagnostic.message.contains(L4_REMEDIATION) + })); + } + + #[test] + fn mcp_profile_validation_uses_shared_case_insensitive_protocol_parsing() { + let uppercase_mcp = r" +id: uppercase-mcp +display_name: Uppercase MCP +endpoints: + - host: mcp.example.com + port: 443 + protocol: MCP + mcp: + versions: ['2025-11-25', '2025-03-26'] + rules: + - allow: + method: initialize +"; + let profiles = parse_profile_catalog_yamls(&[uppercase_mcp]) + .expect("the shared protocol parser recognizes uppercase MCP"); + assert_eq!( + profiles[0].endpoints[0] + .mcp + .as_ref() + .expect("MCP options") + .versions, + ["2025-03-26", "2025-11-25"] + ); + + let uppercase_missing = r" +id: uppercase-mcp +display_name: Uppercase MCP +endpoints: + - host: mcp.example.com + port: 443 + protocol: MCP + rules: + - allow: + method: initialize +"; + let profiles = parse_profile_catalog_yamls(&[uppercase_missing]) + .expect("uppercase MCP must receive the pinned default"); + assert_eq!( + profiles[0].endpoints[0] + .mcp + .as_ref() + .expect("materialized MCP options") + .versions, + [DEFAULT_MCP_PROTOCOL_VERSION.as_str()] + ); + } + + #[test] + fn profile_catalog_rejects_mcp_contract_after_an_unrelated_diagnostic() { + let profiles = [ + r" +id: Invalid-Id +display_name: MCP Example +endpoints: + - host: mcp.example.com + port: 443 + protocol: rest + mcp: + versions: ['2025-11-25'] +", + r" +id: Invalid-Id +display_name: REST Example +endpoints: + - host: api.example.com + port: 443 + protocol: rest + mcp: + versions: ['2025-11-25'] +", + ]; + + for yaml in profiles { + let profile = parse_profile_yaml(yaml).expect("profile shape should parse"); + let diagnostics = validate_profile_set(&[(String::new(), profile)]); + assert_eq!(diagnostics[0].field, "id"); + assert!( + diagnostics + .iter() + .any(|diagnostic| is_mcp_diagnostic_field(&diagnostic.field)) + ); + assert!(matches!( + parse_profile_catalog_yamls(&[yaml]), + Err(ProfileError::ValidationError { ref field, .. }) if field == "id" + )); + } + } + + #[test] + fn profile_catalog_reports_the_first_mcp_diagnostic_before_a_later_invalid_endpoint() { + let yaml = r#" +id: mcp-priority +display_name: MCP Priority +endpoints: + - host: mcp.example.com + port: 443 + protocol: rest + mcp: + versions: ['2025-11-25'] + - host: "" + port: 443 +"#; + + assert!(matches!( + parse_profile_catalog_yamls(&[yaml]), + Err(ProfileError::InvalidMcpConfiguration { field, .. }) + if field == "endpoints[0].mcp" + )); + } + #[test] fn profile_refresh_metadata_round_trips_through_proto_and_yaml() { let profile = parse_profile_yaml( @@ -5154,6 +6199,7 @@ endpoints: port: 443 protocol: mcp mcp: + versions: ['2025-11-25'] allow_all_known_mcp_methods: true rules: - allow: @@ -5193,6 +6239,7 @@ endpoints: port: 443 protocol: mcp mcp: + versions: ['2025-11-25'] allow_all_known_mcp_methods: true rules: - allow: @@ -5234,6 +6281,7 @@ endpoints: port: 443 protocol: mcp mcp: + versions: ['2025-11-25'] allow_all_known_mcp_methods: true rules: - allow: @@ -5272,6 +6320,7 @@ endpoints: port: 443 protocol: mcp mcp: + versions: ['2025-11-25'] allow_all_known_mcp_methods: true deny_rules: - method: tools/call @@ -5315,6 +6364,7 @@ endpoints: port: 443 protocol: mcp mcp: + versions: ['2025-11-25'] allow_all_known_mcp_methods: true rules: - allow: @@ -5361,6 +6411,7 @@ endpoints: port: 443 protocol: mcp mcp: + versions: ['2025-11-25'] allow_all_known_mcp_methods: true deny_rules: - method: 'tools/call' @@ -5402,6 +6453,7 @@ endpoints: port: 443 protocol: mcp mcp: + versions: ['2025-11-25'] allow_all_known_mcp_methods: true rules: - allow: @@ -5457,6 +6509,8 @@ endpoints: - host: mcp.example.com port: 443 protocol: mcp + mcp: + versions: ['2025-11-25'] rules: - allow: method: 'tools/call' @@ -5491,6 +6545,7 @@ endpoints: port: 443 protocol: mcp mcp: + versions: ['2025-11-25'] allow_all_known_mcp_methods: true rules: - allow: @@ -5525,6 +6580,7 @@ endpoints: port: 443 protocol: mcp mcp: + versions: ['2025-11-25'] allow_all_known_mcp_methods: true rules: - allow: @@ -5559,6 +6615,7 @@ endpoints: port: 443 protocol: mcp mcp: + versions: ['2025-11-25'] allow_all_known_mcp_methods: true rules: - allow: @@ -5594,6 +6651,8 @@ endpoints: - host: mcp.example.com port: 443 protocol: mcp + mcp: + versions: ['2025-11-25'] rules: - allow: method: tools/call @@ -5647,6 +6706,7 @@ endpoints: port: 443 protocol: mcp mcp: + versions: ['2025-11-25'] allow_all_known_mcp_methods: true deny_rules: - method: tools/call @@ -5680,6 +6740,8 @@ endpoints: - host: mcp.example.com port: 443 protocol: mcp + mcp: + versions: ['2025-11-25'] rules: - allow: method: resources/read @@ -5711,6 +6773,7 @@ endpoints: port: 443 protocol: mcp mcp: + versions: ['2025-11-25'] allow_all_known_mcp_methods: true deny_rules: - method: resources/read @@ -5741,6 +6804,8 @@ endpoints: - host: mcp.example.com port: 443 protocol: mcp + mcp: + versions: ['2025-11-25'] rules: - allow: tool: @@ -5770,6 +6835,8 @@ endpoints: - host: mcp.example.com port: 443 protocol: mcp + mcp: + versions: ['2025-11-25'] rules: - allow: method: tools/call @@ -5803,6 +6870,8 @@ endpoints: - host: mcp.example.com port: 443 protocol: mcp + mcp: + versions: ['2025-11-25'] rules: - allow: method: tools/call @@ -5832,6 +6901,7 @@ endpoints: port: 443 protocol: mcp mcp: + versions: ['2025-11-25'] allow_all_known_mcp_methods: true deny_rules: - method: tools/call @@ -5898,6 +6968,8 @@ endpoints: - host: api.example.com port: 443 protocol: mcp + mcp: + versions: ['2025-11-25'] rules: [] binaries: - /usr/bin/example-agent @@ -5925,6 +6997,7 @@ endpoints: port: 443 protocol: mcp mcp: + versions: ['2025-11-25'] allow_all_known_mcp_methods: true rules: - allow: @@ -5956,6 +7029,7 @@ endpoints: port: 443 protocol: mcp mcp: + versions: ['2025-11-25'] allow_all_known_mcp_methods: true rules: - allow: @@ -5989,6 +7063,7 @@ endpoints: port: 443 protocol: mcp mcp: + versions: ['2025-11-25'] allow_all_known_mcp_methods: true rules: - allow: @@ -6022,6 +7097,7 @@ endpoints: port: 443 protocol: mcp mcp: + versions: ['2025-11-25'] allow_all_known_mcp_methods: true rules: - allow: @@ -6051,6 +7127,7 @@ endpoints: port: 443 protocol: mcp mcp: + versions: ['2025-11-25'] allow_all_known_mcp_methods: true strict_tool_names: false rules: @@ -6084,6 +7161,7 @@ endpoints: port: 443 protocol: mcp mcp: + versions: ['2025-11-25'] allow_all_known_mcp_methods: true strict_tool_names: true rules: @@ -6117,6 +7195,7 @@ endpoints: port: 443 protocol: mcp mcp: + versions: ['2025-11-25'] allow_all_known_mcp_methods: true rules: - allow: @@ -6147,6 +7226,7 @@ endpoints: port: 443 protocol: mcp mcp: + versions: ['2025-11-25'] allow_all_known_mcp_methods: true rules: - allow: @@ -6208,6 +7288,7 @@ endpoints: port: 443 protocol: rest mcp: + versions: ['2025-11-25'] strict_tool_names: true rules: - allow: diff --git a/crates/openshell-providers/src/providers/google_cloud.rs b/crates/openshell-providers/src/providers/google_cloud.rs index 470d1d206..3bea53d82 100644 --- a/crates/openshell-providers/src/providers/google_cloud.rs +++ b/crates/openshell-providers/src/providers/google_cloud.rs @@ -5,29 +5,13 @@ use std::collections::HashMap; use openshell_core::google_cloud; -use crate::{ - DiscoveredProvider, Provider, ProviderDiscoverySpec, ProviderError, ProviderPlugin, - RealDiscoveryContext, discover_with_spec, -}; +use crate::{Provider, ProviderPlugin}; pub struct GoogleCloudProvider; -const SPEC: ProviderDiscoverySpec = ProviderDiscoverySpec { - id: "google-cloud", - credential_env_vars: google_cloud::TOKEN_ENV_KEYS, -}; - impl ProviderPlugin for GoogleCloudProvider { fn id(&self) -> &'static str { - SPEC.id - } - - fn discover_existing(&self) -> Result, ProviderError> { - discover_with_spec(&SPEC, &RealDiscoveryContext) - } - - fn credential_env_vars(&self) -> &'static [&'static str] { - SPEC.credential_env_vars + "google-cloud" } fn inject_env(&self, provider: &Provider, env: &mut HashMap) { diff --git a/crates/openshell-providers/src/providers/mod.rs b/crates/openshell-providers/src/providers/mod.rs index 277021031..b6450e170 100644 --- a/crates/openshell-providers/src/providers/mod.rs +++ b/crates/openshell-providers/src/providers/mod.rs @@ -1,46 +1,5 @@ // SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 -/// Generate a standard discovery smoke-test for a provider whose only test is -/// checking that an env-var credential is picked up by `discover_with_spec`. -/// -/// # Usage -/// ```ignore -/// test_discovers_env_credential!(discovers_openai_env_credentials, "OPENAI_API_KEY", "sk-test"); -/// ``` -macro_rules! test_discovers_env_credential { - ($test_name:ident, $env_var:expr, $env_value:expr) => { - #[cfg(test)] - mod tests { - use super::SPEC; - use crate::discover_with_spec; - use crate::test_helpers::MockDiscoveryContext; - - #[test] - fn $test_name() { - let ctx = MockDiscoveryContext::new().with_env($env_var, $env_value); - let discovered = discover_with_spec(&SPEC, &ctx) - .expect("discovery") - .expect("provider"); - assert_eq!( - discovered.credentials.get($env_var), - Some(&$env_value.to_string()) - ); - } - } - }; -} -pub mod anthropic; -pub mod claude; -pub mod codex; -pub mod copilot; -pub mod deepinfra; -pub mod generic; -pub mod github; -pub mod gitlab; pub mod google_cloud; -pub mod nvidia; -pub mod openai; -pub mod opencode; -pub mod outlook; pub mod vertex; diff --git a/crates/openshell-providers/src/providers/vertex.rs b/crates/openshell-providers/src/providers/vertex.rs index ad52bc51c..e8e1857a9 100644 --- a/crates/openshell-providers/src/providers/vertex.rs +++ b/crates/openshell-providers/src/providers/vertex.rs @@ -4,51 +4,20 @@ use std::collections::HashMap; use openshell_core::google_cloud; -use openshell_core::inference; -use crate::{ - DiscoveredProvider, Provider, ProviderDiscoverySpec, ProviderError, ProviderPlugin, - RealDiscoveryContext, discover_with_spec, -}; +use crate::{Provider, ProviderPlugin, VERTEX_AI_PROJECT_ID_KEY, VERTEX_AI_REGION_KEY}; pub struct VertexProvider; -const SPEC: ProviderDiscoverySpec = ProviderDiscoverySpec { - id: "google-vertex-ai", - credential_env_vars: inference::VERTEX_AI_CREDENTIAL_KEY_NAMES, -}; - impl ProviderPlugin for VertexProvider { fn id(&self) -> &'static str { - SPEC.id - } - - fn discover_existing(&self) -> Result, ProviderError> { - let mut discovered = discover_with_spec(&SPEC, &RealDiscoveryContext)?.unwrap_or_default(); - - for key in inference::VERTEX_AI_CONFIG_KEY_NAMES { - if let Ok(val) = std::env::var(key) - && !val.trim().is_empty() - { - discovered.config.entry(key.to_string()).or_insert(val); - } - } - - if discovered.is_empty() { - Ok(None) - } else { - Ok(Some(discovered)) - } - } - - fn credential_env_vars(&self) -> &'static [&'static str] { - SPEC.credential_env_vars + "google-vertex-ai" } fn inject_env(&self, provider: &Provider, env: &mut HashMap) { if let Some(project) = provider .config - .get(inference::VERTEX_AI_PROJECT_ID_KEY) + .get(VERTEX_AI_PROJECT_ID_KEY) .filter(|v| !v.trim().is_empty()) { let trimmed = project.trim().to_string(); @@ -62,7 +31,7 @@ impl ProviderPlugin for VertexProvider { if let Some(region) = provider .config - .get(inference::VERTEX_AI_REGION_KEY) + .get(VERTEX_AI_REGION_KEY) .filter(|v| !v.trim().is_empty()) { let trimmed = region.trim().to_string(); diff --git a/crates/openshell-router/Cargo.toml b/crates/openshell-router/Cargo.toml deleted file mode 100644 index 97bbf4dc7..000000000 --- a/crates/openshell-router/Cargo.toml +++ /dev/null @@ -1,31 +0,0 @@ -# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -# SPDX-License-Identifier: Apache-2.0 - -[package] -name = "openshell-router" -description = "Inference routing library for OpenShell" -version.workspace = true -edition.workspace = true -rust-version.workspace = true -license.workspace = true -repository.workspace = true - -[dependencies] -openshell-core = { path = "../openshell-core", default-features = false } -bytes = { workspace = true } -reqwest = { workspace = true } -serde = { workspace = true } -serde_json = { workspace = true } -thiserror = { workspace = true } -tracing = { workspace = true } -tokio = { workspace = true } -serde_yml = { workspace = true } -uuid = { workspace = true } - -[dev-dependencies] -wiremock = "0.6" -tokio = { workspace = true, features = ["test-util"] } -tempfile = "3" - -[lints] -workspace = true diff --git a/crates/openshell-router/README.md b/crates/openshell-router/README.md deleted file mode 100644 index 8c9527da3..000000000 --- a/crates/openshell-router/README.md +++ /dev/null @@ -1,47 +0,0 @@ -# openshell-router - -`openshell-router` is the inference routing and upstream execution engine used -by the sandbox proxy and gateway inference validation paths. - -## Responsibilities - -- Select an upstream route from a candidate set (based on protocol compatibility). -- Forward raw HTTP requests to the selected upstream backend. -- Normalize upstream failures into router-level errors (`unauthorized`, `unavailable`, protocol/internal errors). -- Keep routing decision logic in one place so strategies can evolve (fallbacks, scoring, health-based routing). - -## Non-responsibilities - -- Authentication and sandbox identity. -- Authorization and policy enforcement. -- Persistence of routes/entities. -- Loading sandbox or policy objects. - -These are owned by `openshell-server` and `openshell-sandbox`. - -## Integration Contract - -Current split: - -- `openshell-server`: - - authenticates user-facing inference configuration changes - - resolves managed route candidates from provider records - - validates backend endpoints -- `openshell-sandbox`: - - intercepts `https://inference.local` - - detects the source inference protocol - - passes sanitized requests and resolved route candidates to the router -- `openshell-router`: - - picks a route from candidates (`proxy_with_candidates`) - - forwards the HTTP request upstream and returns the raw response - -## Public APIs - -- `Router::proxy_with_candidates(source_protocol, method, path, headers, body, &[ResolvedRoute])` - - Filters candidates by protocol compatibility, then forwards the request to the first match. - - Preferred path for entity-driven server routing. - -## Notes - -- Route selection matches candidates by `protocol` field (e.g. `openai_chat_completions`). -- Route selection is intentionally simple and will evolve. diff --git a/crates/openshell-router/src/backend.rs b/crates/openshell-router/src/backend.rs deleted file mode 100644 index 84a5c6acf..000000000 --- a/crates/openshell-router/src/backend.rs +++ /dev/null @@ -1,2729 +0,0 @@ -// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -// SPDX-License-Identifier: Apache-2.0 - -use crate::RouterError; -use crate::config::{AuthHeader, ResolvedRoute}; -use crate::mock; -use std::collections::HashSet; - -/// Maximum buffered inference response body, in bytes. The buffered path -/// reads the whole response into memory; the route timeout bounds time, not -/// memory, so without this cap an oversized upstream could force unbounded -/// allocation. Mirrors the sandbox streaming byte cap. Over-cap responses fail -/// as an upstream protocol error. -const MAX_BUFFERED_RESPONSE_BODY: usize = 32 * 1024 * 1024; - -#[derive(Debug, Clone, PartialEq, Eq)] -pub struct ValidatedEndpoint { - pub url: String, - pub protocol: String, -} - -#[derive(Debug, Clone, Copy, PartialEq, Eq)] -pub enum ValidationFailureKind { - RequestShape, - Credentials, - RateLimited, - Connectivity, - UpstreamHealth, - Unexpected, -} - -#[derive(Debug, Clone, PartialEq, Eq)] -pub struct ValidationFailure { - pub kind: ValidationFailureKind, - pub details: String, -} - -struct ValidationProbe { - path: &'static str, - protocol: &'static str, - body: bytes::Bytes, - /// Alternate body to try when the primary probe is rejected specifically - /// for `max_completion_tokens`. Used for `OpenAI` chat completions where - /// newer models require `max_completion_tokens` while legacy/self-hosted - /// backends only accept `max_tokens`. The retry is gated on the error - /// body naming that parameter, so an unrelated request-shape rejection - /// (wrong protocol for the model) falls through instead. - fallback_body: Option, -} - -/// Response from a proxied HTTP request to a backend (fully buffered). -#[derive(Debug)] -pub struct ProxyResponse { - pub status: u16, - pub headers: Vec<(String, String)>, - pub body: bytes::Bytes, - /// Model identity of the route that produced this response. - pub route_model: Option, - /// Endpoint of the route that produced this response. - pub route_endpoint: Option, -} - -/// Response from a proxied HTTP request where the body can be streamed -/// incrementally via [`StreamingProxyResponse::next_chunk`]. -pub struct StreamingProxyResponse { - pub status: u16, - pub headers: Vec<(String, String)>, - /// Model identity of the route that produced this response. - pub route_model: Option, - /// Endpoint of the route that produced this response. - pub route_endpoint: Option, - /// Either a live response to stream from, or a pre-buffered body (for mock routes). - body: StreamingBody, -} - -enum StreamingBody { - /// Live upstream response — call `chunk().await` to read incrementally. - Live(reqwest::Response), - /// Pre-buffered body (e.g. from mock routes). Drained on first `next_chunk()`. - Buffered(Option), -} - -/// The `anthropic_version` value required by Vertex AI's rawPredict endpoint for -/// Anthropic Claude models. Google publishes this version string; update here if -/// the Vertex AI Anthropic API version changes. -/// -/// See: -const VERTEX_ANTHROPIC_VERSION: &str = "vertex-2023-10-16"; - -const COMMON_INFERENCE_REQUEST_HEADERS: [&str; 4] = - ["content-type", "accept", "accept-encoding", "user-agent"]; - -/// Anthropic-SDK-only body fields that Vertex AI rawPredict does not accept. -/// Vertex AI's rawPredict endpoint uses strict pydantic validation and rejects -/// any extra inputs not in its schema. The Anthropic SDKs' native Vertex -/// transport (`AnthropicVertex`) strips these automatically; we must do the same -/// when proxying through `inference.local`. -/// -/// `model` is handled separately (rewritten or stripped depending on route type) -/// and is intentionally omitted here. -const VERTEX_UNSUPPORTED_BODY_FIELDS: &[&str] = &["context_management"]; - -impl StreamingProxyResponse { - /// Create from a fully-buffered [`ProxyResponse`] (for mock routes). - pub fn from_buffered(resp: ProxyResponse) -> Self { - Self { - status: resp.status, - headers: resp.headers, - route_model: resp.route_model, - route_endpoint: resp.route_endpoint, - body: StreamingBody::Buffered(Some(resp.body)), - } - } - - /// Read the next body chunk. Returns `None` when the body is exhausted. - pub async fn next_chunk(&mut self) -> Result, RouterError> { - match &mut self.body { - StreamingBody::Live(response) => response.chunk().await.map_err(|e| { - RouterError::UpstreamProtocol(format!("failed to read response chunk: {e}")) - }), - StreamingBody::Buffered(buf) => Ok(buf.take()), - } - } -} - -fn sanitize_request_headers( - route: &ResolvedRoute, - headers: &[(String, String)], -) -> Vec<(String, String)> { - let mut allowed = HashSet::new(); - allowed.extend( - COMMON_INFERENCE_REQUEST_HEADERS - .iter() - .map(|name| (*name).to_string()), - ); - allowed.extend( - route - .passthrough_headers - .iter() - .map(|name| name.to_ascii_lowercase()), - ); - allowed.extend( - route - .default_headers - .iter() - .map(|(name, _)| name.to_ascii_lowercase()), - ); - - // Vertex AI Anthropic rawPredict endpoints do not accept the - // `anthropic-beta` header. Beta feature enablement for Vertex AI is - // controlled through Google Cloud, not HTTP headers. Strip it here so - // clients (e.g. Claude Code) that always send beta flags don't cause - // HTTP 400 errors from the Vertex AI backend. - let strip_anthropic_beta = is_vertex_anthropic_rawpredict_route(route); - - headers - .iter() - .filter_map(|(name, value)| { - let name_lc = name.to_ascii_lowercase(); - if should_strip_request_header(&name_lc) || !allowed.contains(&name_lc) { - return None; - } - if strip_anthropic_beta && name_lc == "anthropic-beta" { - return None; - } - Some((name.clone(), value.clone())) - }) - .collect() -} - -fn should_strip_request_header(name: &str) -> bool { - matches!( - name, - "authorization" | "x-api-key" | "host" | "content-length" - ) || is_hop_by_hop_header(name) -} - -fn is_hop_by_hop_header(name: &str) -> bool { - matches!( - name, - "connection" - | "keep-alive" - | "proxy-authenticate" - | "proxy-authorization" - | "proxy-connection" - | "te" - | "trailer" - | "transfer-encoding" - | "upgrade" - ) -} - -/// Build and send an HTTP request to the backend configured in `route`. -/// -/// Returns the prepared [`reqwest::RequestBuilder`] with auth, headers, model -/// rewrite, and body applied. The caller decides whether to apply a total -/// request timeout before sending. -/// -/// `stream_response` controls whether Vertex AI Anthropic routes upgrade the -/// stored `:rawPredict` suffix to `:streamRawPredict` in the upstream URL. -/// It must match the transport the caller intends to use: -/// -/// | Caller | `stream_response` | Vertex suffix used | -/// |-----------------------------|-------------------|------------------------| -/// | `send_backend_request` | `false` | `:rawPredict` (unary) | -/// | `send_backend_request_streaming` | `true` | `:streamRawPredict` | -/// -/// `verify_backend_endpoint` explicitly passes `false` to probe the unary -/// `:rawPredict` endpoint during validation. The `inference.local` intercept -/// path always calls `send_backend_request_streaming` (and therefore always -/// passes `true`), but `:streamRawPredict` accepts both streaming and -/// non-streaming request bodies, so the behaviour is correct in all cases. -fn prepare_backend_request( - client: &reqwest::Client, - route: &ResolvedRoute, - method: &str, - path: &str, - headers: &[(String, String)], - body: bytes::Bytes, - stream_response: bool, -) -> Result<(reqwest::RequestBuilder, String), RouterError> { - // For AWS Bedrock routes the model id is encoded in the URL path - // (`/model/{modelId}/invoke[-with-response-stream]`), not in the - // JSON body. The caller's path can carry any model id; rewrite it - // to the operator-configured `route.model` so a sandbox cannot - // pick a different upstream model than what `inference set` - // configured. If the path is not a recognized Bedrock shape on a - // Bedrock route, reject the request rather than forwarding - // verbatim. - let rewritten_path: String; - let path = if route_is_bedrock(route) { - match rewrite_bedrock_path(route, path) { - Some(p) => { - rewritten_path = p; - rewritten_path.as_str() - } - None => { - return Err(RouterError::Internal(format!( - "AWS Bedrock route received unprocessable path '{path}' or invalid \ - route.model; expected /model//invoke and a model id with no \ - path separators, URL delimiters, percent escapes, traversal \ - segments, whitespace, or control characters" - ))); - } - } - } else { - path - }; - let url = build_provider_url(route, &route.model, path, stream_response); - let headers = sanitize_request_headers(route, headers); - - let reqwest_method: reqwest::Method = method - .parse() - .map_err(|_| RouterError::Internal(format!("invalid HTTP method: {method}")))?; - - let mut builder = client.request(reqwest_method, &url); - - // Inject API key using the route's configured auth mechanism. - match &route.auth { - AuthHeader::Bearer => { - builder = builder.bearer_auth(&route.api_key); - } - AuthHeader::Custom(header_name) => { - builder = builder.header(*header_name, &route.api_key); - } - AuthHeader::None => { - // Bridge-fronted upstream: no router-side auth injection. - // The configured `endpoint` is expected to be a translating - // bridge / proxy whose own pod holds operator-side - // credentials. Used today by the `aws-bedrock` profile - // (SigV4 signing is a separate follow-up). - } - } - for (name, value) in &headers { - builder = builder.header(name.as_str(), value.as_str()); - } - - // Apply route-level default headers (e.g. anthropic-version) unless - // the client already sent them. - for (name, value) in &route.default_headers { - let already_sent = headers.iter().any(|(h, _)| h.eq_ignore_ascii_case(name)); - if !already_sent { - builder = builder.header(name.as_str(), value.as_str()); - } - } - - // Rewrite the JSON body for backend compatibility: - // - Standard routes: set "model" to the route's configured model so the - // backend receives the correct model ID regardless of what the client sent. - // - Vertex AI rawPredict routes: remove "model" (it is encoded in the URL - // path) and inject "anthropic_version" (required in the body, not a header). - // Non-JSON bodies pass through unchanged; model rewrite and version injection - // are silently skipped. Such bodies would be rejected by the upstream anyway. - let body = match serde_json::from_slice::(&body) { - Ok(mut json) => { - if let Some(obj) = json.as_object_mut() { - // Vertex AI Anthropic endpoints require anthropic_version in the body. - // Standard Anthropic SDK sends it as a header; Vertex AI needs it as a body field. - // We inject it only for the Vertex rawPredict-style route contract used for - // Anthropic publisher endpoints, not for arbitrary model-in-path routes. - let needs_vertex_anthropic_version = is_vertex_anthropic_rawpredict_route(route); - if needs_vertex_anthropic_version { - // Vertex AI rawPredict encodes the model in the URL path, not - // the request body. Strip "model" and any Anthropic-SDK-only - // beta fields that Vertex's strict pydantic validation rejects. - obj.remove("model"); - for field in VERTEX_UNSUPPORTED_BODY_FIELDS { - obj.remove(*field); - } - } else if route_is_bedrock(route) { - // AWS Bedrock InvokeModel encodes the model in the URL - // path; the request body is the raw provider-specific - // payload (e.g. an Anthropic Messages body for Claude - // models, a Mistral payload for Mistral models). The - // body must not be mutated — injecting a "model" field - // here would either be silently ignored or rejected as - // an unexpected key by the upstream / bridge. - } else { - obj.insert( - "model".to_string(), - serde_json::Value::String(route.model.clone()), - ); - } - if needs_vertex_anthropic_version && !obj.contains_key("anthropic_version") { - obj.insert( - "anthropic_version".to_string(), - serde_json::Value::String(VERTEX_ANTHROPIC_VERSION.to_string()), - ); - } - } - - bytes::Bytes::from(serde_json::to_vec(&json).map_err(|err| { - RouterError::Internal(format!( - "failed to serialize rewritten inference request body: {err}" - )) - })?) - } - Err(_) => body, - }; - builder = builder.body(body); - - Ok((builder, url)) -} - -/// Send an error-mapped request, shared by both buffered and streaming paths. -fn map_send_error(e: reqwest::Error, url: &str) -> RouterError { - if e.is_timeout() { - RouterError::UpstreamUnavailable(format!("request to {url} timed out")) - } else if e.is_connect() { - RouterError::UpstreamUnavailable(format!("failed to connect to {url}: {e}")) - } else { - RouterError::Internal(format!("HTTP request failed: {e}")) - } -} - -/// Build and send an HTTP request to the backend with a total request timeout. -/// -/// The timeout covers the entire request lifecycle (connect + headers + body). -/// Suitable for non-streaming responses where the body is buffered completely. -async fn send_backend_request( - client: &reqwest::Client, - route: &ResolvedRoute, - method: &str, - path: &str, - headers: &[(String, String)], - body: bytes::Bytes, -) -> Result { - let (builder, url) = - prepare_backend_request(client, route, method, path, headers, body, false)?; - builder - .timeout(route.timeout) - .send() - .await - .map_err(|e| map_send_error(e, &url)) -} - -/// Build and send an HTTP request without a total request timeout. -/// -/// For streaming responses, the total duration is unbounded — liveness is -/// enforced by the caller's per-chunk idle timeout instead. Connection -/// establishment is still bounded by the client-level `connect_timeout`. -async fn send_backend_request_streaming( - client: &reqwest::Client, - route: &ResolvedRoute, - method: &str, - path: &str, - headers: &[(String, String)], - body: bytes::Bytes, -) -> Result { - let (builder, url) = prepare_backend_request(client, route, method, path, headers, body, true)?; - builder.send().await.map_err(|e| map_send_error(e, &url)) -} - -/// Validation probes for a route, in preference order. -/// -/// A managed route advertises every protocol in its provider profile, so an -/// embeddings model resolves to a route that also lists chat/completions. The -/// caller tries these in order and falls through to the next on a request-shape -/// rejection, so such a model validates against `/v1/embeddings` even though -/// the chat probe rejects it. Embeddings is ordered last so a genuinely -/// chat-capable route still validates against chat. Empty when the route -/// exposes no writable protocol. -fn validation_probes(route: &ResolvedRoute) -> Vec { - let has = |protocol: &str| route.protocols.iter().any(|p| p == protocol); - let mut probes = Vec::new(); - - if has("openai_chat_completions") { - // Use max_completion_tokens (modern OpenAI parameter, required by GPT-5+) - // with max_tokens as fallback for legacy/self-hosted backends. - probes.push(ValidationProbe { - path: "/v1/chat/completions", - protocol: "openai_chat_completions", - body: bytes::Bytes::from_static( - br#"{"messages":[{"role":"user","content":"ping"}],"max_completion_tokens":32}"#, - ), - fallback_body: Some(bytes::Bytes::from_static( - br#"{"messages":[{"role":"user","content":"ping"}],"max_tokens":32}"#, - )), - }); - } - - if has("anthropic_messages") { - probes.push(ValidationProbe { - path: "/v1/messages", - protocol: "anthropic_messages", - body: bytes::Bytes::from_static( - br#"{"messages":[{"role":"user","content":"ping"}],"max_tokens":32}"#, - ), - fallback_body: None, - }); - } - - if has("openai_responses") { - probes.push(ValidationProbe { - path: "/v1/responses", - protocol: "openai_responses", - body: bytes::Bytes::from_static(br#"{"input":"ping","max_output_tokens":32}"#), - fallback_body: None, - }); - } - - if has("openai_completions") { - probes.push(ValidationProbe { - path: "/v1/completions", - protocol: "openai_completions", - body: bytes::Bytes::from_static(br#"{"prompt":"ping","max_tokens":32}"#), - fallback_body: None, - }); - } - - // Last so a chat-capable route prefers a chat probe, but an embeddings-only - // model still validates against its single writable endpoint. - if has("openai_embeddings") { - probes.push(ValidationProbe { - path: "/v1/embeddings", - protocol: "openai_embeddings", - body: bytes::Bytes::from_static(br#"{"input":"ping"}"#), - fallback_body: None, - }); - } - - probes -} - -/// The request-shape failure for a route that advertises no writable protocol. -/// -/// Shared by the empty-probe guard and the all-probes-failed fallback so the -/// otherwise-unreachable terminal case is a value rather than a panic. -fn no_writable_protocol_failure(route: &ResolvedRoute) -> ValidationFailure { - ValidationFailure { - kind: ValidationFailureKind::RequestShape, - details: format!( - "route '{}' does not expose a writable inference protocol for validation", - route.name - ), - } -} - -pub async fn verify_backend_endpoint( - client: &reqwest::Client, - route: &ResolvedRoute, -) -> Result { - let probes = validation_probes(route); - let Some(first) = probes.first() else { - return Err(no_writable_protocol_failure(route)); - }; - - if mock::is_mock_route(route) { - return Ok(ValidatedEndpoint { - url: build_provider_url(route, &route.model, first.path, false), - protocol: first.protocol.to_string(), - }); - } - - let headers = vec![("content-type".to_string(), "application/json".to_string())]; - let mut last_shape_failure = None; - - for probe in &probes { - match try_validation_probe(client, route, probe, &headers).await { - Ok(endpoint) => return Ok(endpoint), - // A request-shape rejection means this protocol is wrong for the - // model (e.g. a chat probe against an embeddings model), so fall - // through to the next advertised protocol. Any other failure - // describes the backend itself (credentials, rate limit, - // connectivity, health) and is terminal across all protocols. - // - // Keep the first shape failure: it is the most-preferred protocol's - // rejection and the most actionable error to report. - Err(err) if err.kind == ValidationFailureKind::RequestShape => { - last_shape_failure.get_or_insert(err); - } - Err(err) => return Err(err), - } - } - - Err(last_shape_failure.unwrap_or_else(|| no_writable_protocol_failure(route))) -} - -/// Run one validation probe, retrying with its fallback body only when the -/// upstream specifically rejected `max_completion_tokens`. -/// -/// That retry exists for the GPT-5+ (`max_completion_tokens`) versus legacy -/// (`max_tokens`) chat split. Firing it for any request-shape rejection would -/// issue a second, pointless probe when the real signal is "wrong protocol for -/// this model", and a transient `429`/`5xx` on that retry could become a -/// terminal failure that stops the caller from reaching a protocol that would -/// have validated. -async fn try_validation_probe( - client: &reqwest::Client, - route: &ResolvedRoute, - probe: &ValidationProbe, - headers: &[(String, String)], -) -> Result { - let result = try_validation_request( - client, - route, - probe.path, - probe.protocol, - headers, - probe.body.clone(), - ) - .await; - - if let (Err(err), Some(fallback_body)) = (&result, &probe.fallback_body) - && err.kind == ValidationFailureKind::RequestShape - && err.details.contains("max_completion_tokens") - { - return try_validation_request( - client, - route, - probe.path, - probe.protocol, - headers, - fallback_body.clone(), - ) - .await; - } - - result -} - -/// Send a single validation request and classify the response. -async fn try_validation_request( - client: &reqwest::Client, - route: &ResolvedRoute, - path: &str, - protocol: &str, - headers: &[(String, String)], - body: bytes::Bytes, -) -> Result { - let response = send_backend_request(client, route, "POST", path, headers, body) - .await - .map_err(|err| match err { - RouterError::UpstreamUnavailable(details) => ValidationFailure { - kind: ValidationFailureKind::Connectivity, - details, - }, - RouterError::Internal(details) | RouterError::UpstreamProtocol(details) => { - ValidationFailure { - kind: ValidationFailureKind::Unexpected, - details, - } - } - RouterError::RouteNotFound(details) - | RouterError::NoCompatibleRoute(details) - | RouterError::Unauthorized(details) => ValidationFailure { - kind: ValidationFailureKind::Unexpected, - details, - }, - })?; - let url = build_provider_url(route, &route.model, path, false); - - if response.status().is_success() { - return Ok(ValidatedEndpoint { - url, - protocol: protocol.to_string(), - }); - } - - let status = response.status(); - let body = response.text().await.map_err(|e| ValidationFailure { - kind: ValidationFailureKind::Unexpected, - details: format!("failed to read validation response body: {e}"), - })?; - let body = body.trim(); - let body_suffix = if body.is_empty() { - String::new() - } else { - format!( - " Response body: {}", - body.chars().take(200).collect::() - ) - }; - - // Some OpenAI-compatible providers report an auth failure as 400/404/422 - // with an auth-shaped error body rather than 401/403. Classify those as a - // terminal credential failure so a bad key is not mistaken for a - // wrong-protocol probe and masked by a later probe that accepts it. - let kind = match status.as_u16() { - 401 | 403 => ValidationFailureKind::Credentials, - 400 | 404 | 422 if body_looks_like_auth_error(body) => ValidationFailureKind::Credentials, - 400 | 404 | 405 | 422 => ValidationFailureKind::RequestShape, - 429 => ValidationFailureKind::RateLimited, - 500..=599 => ValidationFailureKind::UpstreamHealth, - _ => ValidationFailureKind::Unexpected, - }; - - let summary = match kind { - ValidationFailureKind::Credentials => "upstream rejected credentials", - ValidationFailureKind::RateLimited => "upstream rate-limited the validation request", - ValidationFailureKind::UpstreamHealth => "upstream returned a server error", - ValidationFailureKind::RequestShape => "upstream rejected the validation request", - _ => "upstream returned an unexpected response", - }; - - Err(ValidationFailure { - kind, - details: format!("{summary} with HTTP {status}.{body_suffix}"), - }) -} - -/// Whether an upstream error body reads as an authentication or authorization -/// failure. Some OpenAI-compatible providers return these as HTTP 400/404/422 -/// rather than 401/403, so validation inspects the body to avoid classifying a -/// bad key as a wrong-protocol probe. Matching is conservative: only strong, -/// auth-specific phrases, lowercased, to avoid catching generic "invalid model" -/// request-shape errors. -fn body_looks_like_auth_error(body: &str) -> bool { - let body = body.to_ascii_lowercase(); - [ - "invalid_api_key", - "invalid api key", - "incorrect api key", - "invalid_authentication", - "authentication_error", - "authentication failed", - "unauthorized", - "permission_denied", - "permission denied", - "missing api key", - ] - .iter() - .any(|needle| body.contains(needle)) -} - -/// Extract status and headers from a [`reqwest::Response`]. -fn extract_response_metadata(response: &reqwest::Response) -> (u16, Vec<(String, String)>) { - let status = response.status().as_u16(); - let headers: Vec<(String, String)> = response - .headers() - .iter() - .map(|(k, v)| (k.to_string(), v.to_str().unwrap_or("").to_string())) - .collect(); - (status, headers) -} - -/// Forward a raw HTTP request to the backend configured in `route`. -/// -/// Buffers the entire response body before returning. Suitable for -/// non-streaming responses or mock routes. -pub async fn proxy_to_backend( - client: &reqwest::Client, - route: &ResolvedRoute, - _source_protocol: &str, - method: &str, - path: &str, - headers: Vec<(String, String)>, - body: bytes::Bytes, -) -> Result { - let response = send_backend_request(client, route, method, path, &headers, body).await?; - let (status, resp_headers) = extract_response_metadata(&response); - let body = read_capped_response_body(response, MAX_BUFFERED_RESPONSE_BODY).await?; - - Ok(ProxyResponse { - status, - headers: resp_headers, - body, - route_model: Some(route.model.clone()), - route_endpoint: Some(route.endpoint.clone()), - }) -} - -/// Read a response body fully into memory, rejecting anything over `max` bytes. -/// -/// Used by the buffered proxy path so a misbehaving upstream cannot force -/// unbounded allocation. The `Content-Length` check is a fast early-out; the -/// chunk loop is the real guard and bounds an absent, chunked, or -/// under-reported length. The cap counts the bytes reqwest yields: with no -/// decompression features enabled (see `Cargo.toml`) those are wire bytes, so -/// enabling a compression feature later would change what the cap measures. -/// Over-cap responses fail as `UpstreamProtocol` and are never partially -/// returned. -async fn read_capped_response_body( - mut response: reqwest::Response, - max: usize, -) -> Result { - if let Some(len) = response.content_length() - && len > max as u64 - { - return Err(RouterError::UpstreamProtocol(format!( - "inference response body of {len} bytes exceeds the {max} byte cap" - ))); - } - - // Preallocate to the advertised length when it is within the cap; the loop - // still enforces the bound for an absent or under-reported length. - let mut body: Vec = match response.content_length() { - Some(len) if len <= max as u64 => Vec::with_capacity(usize::try_from(len).unwrap_or(max)), - _ => Vec::new(), - }; - while let Some(chunk) = response - .chunk() - .await - .map_err(|e| RouterError::UpstreamProtocol(format!("failed to read response body: {e}")))? - { - if body.len() + chunk.len() > max { - return Err(RouterError::UpstreamProtocol(format!( - "inference response body exceeds the {max} byte cap" - ))); - } - body.extend_from_slice(&chunk); - } - Ok(bytes::Bytes::from(body)) -} - -/// Forward a raw HTTP request to the backend, returning response headers -/// immediately without buffering the body. -/// -/// The caller streams the body incrementally via -/// [`StreamingProxyResponse::response`] using `chunk().await`. -pub async fn proxy_to_backend_streaming( - client: &reqwest::Client, - route: &ResolvedRoute, - _source_protocol: &str, - method: &str, - path: &str, - headers: Vec<(String, String)>, - body: bytes::Bytes, -) -> Result { - let response = - send_backend_request_streaming(client, route, method, path, &headers, body).await?; - let (status, resp_headers) = extract_response_metadata(&response); - - Ok(StreamingProxyResponse { - status, - headers: resp_headers, - route_model: Some(route.model.clone()), - route_endpoint: Some(route.endpoint.clone()), - body: StreamingBody::Live(response), - }) -} - -/// Build the upstream URL for a provider route. -/// -/// `stream_response` selects between the unary and streaming Vertex AI -/// Anthropic endpoint suffixes. Pass the same value used for the enclosing -/// [`prepare_backend_request`] call. See that function's documentation for the -/// full caller table. -/// -/// Behavior matrix (`request_path_override`, `model_in_path`): -/// - `(Some(suffix), true)`: `{endpoint}/{model_id}{suffix}` -/// Used by Vertex AI Anthropic: `stream_response=false` keeps `:rawPredict` -/// (unary); `stream_response=true` upgrades to `:streamRawPredict`. -/// - `(Some(override_path), false)`: `{endpoint}{override_path}` -/// Used when a fixed path replaces the protocol-derived path. -/// - `(None, true)`: `{endpoint}/{model_id}/{protocol_path}` -/// Model embedded before protocol path. -/// - `(None, false)`: delegates to `build_backend_url` (default, with /v1 dedup). -fn build_provider_url( - route: &ResolvedRoute, - model_id: &str, - protocol_path: &str, - stream_response: bool, -) -> String { - let base = route.endpoint.trim_end_matches('/'); - match (&route.request_path_override, route.model_in_path) { - // Vertex AI publisher endpoint: model in URL path with suffix - // e.g. .../publishers/anthropic/models/claude-3-5-sonnet@20241022:rawPredict - (Some(suffix), true) => { - // suffix is appended directly after model_id (e.g. ":rawPredict"). - // It must not start with '/' — use the (Some, false) arm for path overrides. - debug_assert!( - !suffix.starts_with('/'), - "suffix in model_in_path branch must not start with '/'; got: {suffix:?}" - ); - let suffix = if stream_response - && suffix == ":rawPredict" - && is_vertex_anthropic_rawpredict_route(route) - { - ":streamRawPredict" - } else { - suffix.as_str() - }; - format!("{base}/{model_id}{suffix}") - } - // Explicit path override, model NOT in URL. - // Normalize: ensure override_path begins with '/' so the concatenation - // never produces a broken URL like `https://host.compath`. - (Some(override_path), false) => { - if override_path.starts_with('/') || override_path.is_empty() { - format!("{base}{override_path}") - } else { - format!("{base}/{override_path}") - } - } - // Model in path, no override — append model then protocol-derived path - (None, true) => { - let path = protocol_path.trim_start_matches('/'); - format!("{base}/{model_id}/{path}") - } - // Default: existing behavior (includes /v1 deduplication) - (None, false) => build_backend_url(&route.endpoint, protocol_path), - } -} - -fn build_backend_url(endpoint: &str, path: &str) -> String { - let base = endpoint.trim_end_matches('/'); - // Strip the /v1 prefix from the request path when the base URL's path - // component has /v1 as its first segment (e.g. openai/nvidia: "/v1", - // deepinfra: "/v1/openai") or its final segment (e.g. groq: - // "/openai/v1"). This covers all known provider shapes while preserving - // the full path for proxy endpoints where /v1 is buried in the middle - // (e.g. "https://proxy.example/api/v1/openai" → path "/api/v1/openai", - // neither first nor last segment). - let base_path_has_v1_edge_segment = base - .find("://") - .and_then(|i| base[i + 3..].find('/').map(|j| i + 3 + j)) - .is_some_and(|path_start| { - let base_path = &base[path_start..]; - base_path.starts_with("/v1/") || base_path.ends_with("/v1") - }); - if base_path_has_v1_edge_segment - && let Some(rest) = path - .strip_prefix("/v1") - .filter(|rest| rest.is_empty() || rest.starts_with('/')) - { - return format!("{base}{rest}"); - } - - format!("{base}{path}") -} - -/// Check whether a route targets an AWS Bedrock `InvokeModel` endpoint. -/// -/// Returns true when any of the route's protocols is one of the Bedrock -/// invocation protocols. Used to gate Bedrock-specific request shaping -/// (path-segment rewriting, skipped body-model injection) in -/// [`prepare_backend_request`]. -/// -/// `aws_bedrock_invoke_stream` is recognized for forward-compatibility -/// with the streaming follow-up but is not currently advertised by the -/// L7 pattern set. -fn route_is_bedrock(route: &ResolvedRoute) -> bool { - route - .protocols - .iter() - .any(|p| p == "aws_bedrock_invoke" || p == "aws_bedrock_invoke_stream") -} - -/// Parse a Bedrock invocation path into its `(model_id, action_suffix, query_tail)` -/// components. -/// -/// Recognized shape (caller's path on the way into the router): -/// - `/model//invoke[?]` → action `/invoke` -/// -/// `` must be non-empty and contain no `/`. The query tail -/// (including the leading `?`) is preserved so [`rewrite_bedrock_path`] -/// can restore it; the L7 matcher accepts queries, so silently dropping -/// them here would mutate the request shape between the matcher and -/// the upstream. Returns `None` when the path does not match — the -/// caller treats that as a malformed request and rejects rather than -/// forwarding verbatim. -/// -/// `InvokeModelWithResponseStream` (`/invoke-with-response-stream`) is -/// deferred until the streaming relay grows protocol-aware AWS -/// event-stream error termination; the L7 pattern set does not -/// advertise it today, so it cannot reach this parser. -fn parse_bedrock_invocation_path(path: &str) -> Option<(&str, &'static str, &str)> { - // Slice up to but not including `?`, then keep the `?`-prefixed - // tail so callers can re-attach it without reconstructing the - // delimiter. - let (path_only, query_tail) = path - .find('?') - .map_or((path, ""), |idx| (&path[..idx], &path[idx..])); - let rest = path_only.strip_prefix("/model/")?; - let slash_at = rest.find('/')?; - if slash_at == 0 { - return None; - } - let model_id = &rest[..slash_at]; - let suffix = &rest[slash_at..]; - let action: &'static str = match suffix { - "/invoke" => "/invoke", - _ => return None, - }; - Some((model_id, action, query_tail)) -} - -/// Rewrite a Bedrock invocation path so the model segment is the -/// operator-configured `route.model` rather than whatever the caller -/// supplied. Returns the rewritten path on success, or `None` when the -/// inbound path is not a recognized Bedrock invocation shape or when -/// `route.model` is not a valid Bedrock model id. -/// -/// Why rewrite rather than reject: the inbound L7 pattern detector -/// already accepts only `/model/{x}/invoke` shapes for Bedrock routes, -/// so a caller-supplied model segment that differs from the -/// operator-configured one is the only case this function changes — -/// and changing it (vs. rejecting) lets sandbox code that hardcodes a -/// different model continue to work, while still guaranteeing the -/// operator's chosen model is what reaches the upstream. -/// -/// Defense-in-depth model-ID validation: the server-side resolver -/// (`openshell-server::inference::resolve_provider_route`) already -/// rejects malformed Bedrock model ids at route-save time, but the -/// router enforces the same contract before interpolating -/// `route.model` into a URL path segment. Values containing `/`, `\`, -/// `?`, `#`, `%`, traversal segments, whitespace, or control chars -/// are rejected so a stale or hand-edited route store cannot produce -/// ambiguous or malformed upstream paths. -fn rewrite_bedrock_path(route: &ResolvedRoute, path: &str) -> Option { - if !is_valid_bedrock_model_id(&route.model) { - return None; - } - let (_caller_model, action, query_tail) = parse_bedrock_invocation_path(path)?; - Some(format!("/model/{}{}{}", route.model, action, query_tail)) -} - -/// Defense-in-depth predicate matching the server-side -/// `validate_aws_bedrock_model_id` contract — see that function for the -/// authoritative reasoning. Returns `true` when `value` is safe to -/// interpolate into a Bedrock URL path segment. The router uses this -/// before constructing an upstream path so a stale or out-of-band route -/// store cannot bypass the resolver's validation. -fn is_valid_bedrock_model_id(value: &str) -> bool { - if value.is_empty() || value != value.trim() { - return false; - } - if value.contains('/') || value.contains('\\') { - return false; - } - if value.chars().any(|c| matches!(c, '?' | '#' | '%')) { - return false; - } - if value.contains("..") { - return false; - } - if value.chars().any(|c| c.is_control() || c.is_whitespace()) { - return false; - } - true -} - -/// Check whether a route targets a Vertex AI Anthropic rawPredict endpoint. -/// -/// The predicate is purely structural — it tests `model_in_path`, -/// `anthropic_messages` protocol, and `:rawPredict` suffix — so any future -/// provider with the same route shape automatically inherits the same -/// transforms without code changes. -/// -/// The router stores the neutral `:rawPredict` suffix on resolved routes. -/// [`build_provider_url`] upgrades it to `:streamRawPredict` when -/// `stream_response=true` (see [`prepare_backend_request`] for the caller -/// table). [`verify_backend_endpoint`] deliberately passes `stream_response=false` -/// to probe the unary endpoint during validation. -fn is_vertex_anthropic_rawpredict_route(route: &ResolvedRoute) -> bool { - route.model_in_path - && route.protocols.iter().any(|p| p == "anthropic_messages") - && route - .request_path_override - .as_deref() - .is_some_and(|suffix| suffix == ":rawPredict") -} - -#[cfg(test)] -mod tests { - use super::{ - ValidationFailure, ValidationFailureKind, build_backend_url, build_provider_url, - parse_bedrock_invocation_path, prepare_backend_request, proxy_to_backend, - rewrite_bedrock_path, route_is_bedrock, verify_backend_endpoint, - }; - use crate::RouterError; - use crate::config::{DEFAULT_ROUTE_TIMEOUT, ResolvedRoute}; - use openshell_core::inference::AuthHeader; - use std::time::Duration; - use wiremock::matchers::{body_partial_json, header, method, path}; - use wiremock::{Mock, MockServer, ResponseTemplate}; - - #[test] - fn build_backend_url_dedupes_v1_prefix() { - assert_eq!( - build_backend_url("https://api.openai.com/v1", "/v1/chat/completions"), - "https://api.openai.com/v1/chat/completions" - ); - } - - #[test] - fn build_backend_url_preserves_non_versioned_base() { - assert_eq!( - build_backend_url("https://api.anthropic.com", "/v1/messages"), - "https://api.anthropic.com/v1/messages" - ); - } - - #[test] - fn build_backend_url_handles_exact_v1_path() { - assert_eq!( - build_backend_url("https://api.openai.com/v1", "/v1"), - "https://api.openai.com/v1" - ); - } - - #[test] - fn build_backend_url_dedupes_v1_for_base_with_v1_subpath() { - // DeepInfra base URL contains /v1/ internally — /v1 in the request - // path must still be stripped so chat/completions is not doubled. - assert_eq!( - build_backend_url( - "https://api.deepinfra.com/v1/openai", - "/v1/chat/completions" - ), - "https://api.deepinfra.com/v1/openai/chat/completions" - ); - } - - #[test] - fn build_backend_url_dedupes_v1_for_base_ending_with_v1() { - // Providers like Groq use a base URL where /v1 is the final segment - // below a non-root prefix (e.g. /openai/v1). The /v1 in the request - // path must still be stripped so it is not doubled. - assert_eq!( - build_backend_url("https://api.groq.com/openai/v1", "/v1/chat/completions"), - "https://api.groq.com/openai/v1/chat/completions" - ); - } - - #[test] - fn build_backend_url_preserves_v1_for_nested_proxy_path() { - // A proxy whose base path has /v1 buried in the middle (neither first - // nor last segment) must NOT have /v1 stripped — the full request - // path must be appended so the upstream receives the correct prefix. - assert_eq!( - build_backend_url( - "https://proxy.example/api/v1/openai", - "/v1/chat/completions" - ), - "https://proxy.example/api/v1/openai/v1/chat/completions" - ); - } - - fn test_route(endpoint: &str, protocols: &[&str], auth: AuthHeader) -> ResolvedRoute { - ResolvedRoute { - name: "inference.local".to_string(), - endpoint: endpoint.to_string(), - model: "test-model".to_string(), - api_key: "sk-test".to_string(), - protocols: protocols.iter().map(|p| (*p).to_string()).collect(), - auth, - default_headers: vec![("anthropic-version".to_string(), "2023-06-01".to_string())], - passthrough_headers: vec![ - "anthropic-version".to_string(), - "anthropic-beta".to_string(), - ], - timeout: DEFAULT_ROUTE_TIMEOUT, - model_in_path: false, - request_path_override: None, - } - } - - /// The buffered path must reject an over-cap upstream response rather than - /// buffer it. Guards the DoS/OOM exposure of reading the body unbounded. - #[tokio::test] - async fn proxy_to_backend_rejects_over_cap_response_body() { - use super::{MAX_BUFFERED_RESPONSE_BODY, proxy_to_backend}; - - let mock_server = MockServer::start().await; - // One byte over the cap. wiremock sets an accurate Content-Length, so - // the size check rejects before the body is buffered. - let oversized = vec![b'a'; MAX_BUFFERED_RESPONSE_BODY + 1]; - Mock::given(method("GET")) - .and(path("/v1/models")) - .respond_with(ResponseTemplate::new(200).set_body_bytes(oversized)) - .mount(&mock_server) - .await; - - let route = test_route(&mock_server.uri(), &["model_discovery"], AuthHeader::Bearer); - let client = reqwest::Client::new(); - let result = proxy_to_backend( - &client, - &route, - "model_discovery", - "GET", - "/v1/models", - vec![], - bytes::Bytes::new(), - ) - .await; - - assert!( - matches!(result, Err(RouterError::UpstreamProtocol(_))), - "over-cap response must fail as UpstreamProtocol, got: {result:?}" - ); - } - - /// Spawn a one-shot HTTP/1.1 upstream that replies with a chunked body and - /// no `Content-Length`, so the buffered read cannot pre-check a length and - /// must enforce the cap inside the chunk loop. - async fn spawn_chunked_upstream(chunks: &'static [&'static str]) -> std::net::SocketAddr { - use std::fmt::Write as _; - use tokio::io::{AsyncReadExt, AsyncWriteExt}; - let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap(); - let addr = listener.local_addr().unwrap(); - tokio::spawn(async move { - let (mut sock, _) = listener.accept().await.unwrap(); - let mut buf = [0u8; 1024]; - let _ = sock.read(&mut buf).await; - let mut resp = String::from( - "HTTP/1.1 200 OK\r\nContent-Type: application/json\r\nTransfer-Encoding: chunked\r\n\r\n", - ); - for c in chunks { - let _ = write!(resp, "{:x}\r\n{c}\r\n", c.len()); - } - resp.push_str("0\r\n\r\n"); - sock.write_all(resp.as_bytes()).await.unwrap(); - }); - addr - } - - /// The chunk-accumulation guard (not the `Content-Length` pre-check) must - /// reject an over-cap body when the response advertises no length. - #[tokio::test] - async fn read_capped_response_body_rejects_over_cap_chunked() { - let addr = spawn_chunked_upstream(&["aaaa", "bbbb", "cccc"]).await; - let response = reqwest::Client::new() - .get(format!("http://{addr}/")) - .send() - .await - .unwrap(); - assert!( - response.content_length().is_none(), - "chunked response should advertise no Content-Length" - ); - let result = super::read_capped_response_body(response, 8).await; - assert!( - matches!(result, Err(RouterError::UpstreamProtocol(_))), - "over-cap chunked body must be rejected by the loop, got: {result:?}" - ); - } - - /// A body exactly at the cap is accepted (inclusive bound) and returned - /// intact through the chunk loop. - #[tokio::test] - async fn read_capped_response_body_accepts_body_at_cap() { - let addr = spawn_chunked_upstream(&["aaaa", "bbbb"]).await; - let response = reqwest::Client::new() - .get(format!("http://{addr}/")) - .send() - .await - .unwrap(); - let body = super::read_capped_response_body(response, 8).await.unwrap(); - assert_eq!(&body[..], b"aaaabbbb"); - } - - #[test] - fn sanitize_request_headers_drops_unknown_sensitive_headers() { - let route = ResolvedRoute { - name: "inference.local".to_string(), - endpoint: "https://api.example.com/v1".to_string(), - model: "test-model".to_string(), - api_key: "sk-test".to_string(), - protocols: vec!["openai_chat_completions".to_string()], - auth: AuthHeader::Bearer, - default_headers: Vec::new(), - passthrough_headers: vec!["openai-organization".to_string()], - timeout: DEFAULT_ROUTE_TIMEOUT, - model_in_path: false, - request_path_override: None, - }; - - let kept = super::sanitize_request_headers( - &route, - &[ - ("content-type".to_string(), "application/json".to_string()), - ("authorization".to_string(), "Bearer client".to_string()), - ("cookie".to_string(), "session=1".to_string()), - ("x-amz-security-token".to_string(), "token".to_string()), - ("openai-organization".to_string(), "org_123".to_string()), - ], - ); - - assert!( - kept.iter() - .any(|(name, _)| name.eq_ignore_ascii_case("content-type")) - ); - assert!( - kept.iter() - .any(|(name, _)| name.eq_ignore_ascii_case("openai-organization")) - ); - assert!( - kept.iter() - .all(|(name, _)| !name.eq_ignore_ascii_case("authorization")) - ); - assert!( - kept.iter() - .all(|(name, _)| !name.eq_ignore_ascii_case("cookie")) - ); - assert!( - kept.iter() - .all(|(name, _)| !name.eq_ignore_ascii_case("x-amz-security-token")) - ); - } - - #[test] - fn sanitize_request_headers_preserves_allowed_provider_headers() { - let route = test_route( - "https://api.anthropic.com/v1", - &["anthropic_messages"], - AuthHeader::Custom("x-api-key"), - ); - - let kept = super::sanitize_request_headers( - &route, - &[ - ("anthropic-version".to_string(), "2024-10-22".to_string()), - ( - "anthropic-beta".to_string(), - "tool-use-2024-10-22".to_string(), - ), - ("x-api-key".to_string(), "client-key".to_string()), - ], - ); - - assert!(kept.iter().any( - |(name, value)| name.eq_ignore_ascii_case("anthropic-version") && value == "2024-10-22" - )); - assert!( - kept.iter() - .any(|(name, value)| name.eq_ignore_ascii_case("anthropic-beta") - && value == "tool-use-2024-10-22") - ); - assert!( - kept.iter() - .all(|(name, _)| !name.eq_ignore_ascii_case("x-api-key")) - ); - } - - #[test] - fn vertex_anthropic_rawpredict_strips_anthropic_beta() { - // Vertex AI rawPredict endpoints reject the anthropic-beta header. - // The router must strip it before forwarding to avoid HTTP 400 errors - // from the Vertex AI backend when clients (e.g. Claude Code) always - // send beta feature flags. - let route = ResolvedRoute { - name: "inference.local".to_string(), - endpoint: "https://us-central1-aiplatform.googleapis.com/v1/projects/proj/locations/us-central1/publishers/anthropic/models".to_string(), - model: "claude-sonnet-4-20250514".to_string(), - api_key: "ya29.token".to_string(), - protocols: vec!["anthropic_messages".to_string()], - auth: AuthHeader::Bearer, - default_headers: vec![], - passthrough_headers: vec!["anthropic-beta".to_string()], - timeout: DEFAULT_ROUTE_TIMEOUT, - model_in_path: true, - request_path_override: Some(":rawPredict".to_string()), - }; - - let headers = vec![ - ("content-type".to_string(), "application/json".to_string()), - ( - "anthropic-beta".to_string(), - "prompt-caching-scope-2026-01-05,redact-thinking-2026-02-12".to_string(), - ), - ]; - - let kept = super::sanitize_request_headers(&route, &headers); - - assert!( - kept.iter() - .any(|(name, _)| name.eq_ignore_ascii_case("content-type")), - "content-type should be preserved" - ); - assert!( - kept.iter() - .all(|(name, _)| !name.eq_ignore_ascii_case("anthropic-beta")), - "anthropic-beta must be stripped for Vertex AI rawPredict routes" - ); - } - - #[test] - fn direct_anthropic_preserves_anthropic_beta() { - // The anthropic-beta header must still pass through for direct - // Anthropic API routes -- only Vertex AI rawPredict strips it. - let route = test_route( - "https://api.anthropic.com/v1", - &["anthropic_messages"], - AuthHeader::Custom("x-api-key"), - ); - - let headers = vec![ - ("content-type".to_string(), "application/json".to_string()), - ( - "anthropic-beta".to_string(), - "prompt-caching-2024-07-31".to_string(), - ), - ]; - - let kept = super::sanitize_request_headers(&route, &headers); - - assert!( - kept.iter() - .any(|(name, value)| name.eq_ignore_ascii_case("anthropic-beta") - && value == "prompt-caching-2024-07-31"), - "anthropic-beta must be preserved for direct Anthropic API routes" - ); - } - - #[tokio::test] - async fn verify_backend_endpoint_uses_route_auth_and_shape() { - let mock_server = MockServer::start().await; - let route = test_route( - &mock_server.uri(), - &["anthropic_messages"], - AuthHeader::Custom("x-api-key"), - ); - - Mock::given(method("POST")) - .and(path("/v1/messages")) - .and(header("x-api-key", "sk-test")) - .and(header("content-type", "application/json")) - .and(header("anthropic-version", "2023-06-01")) - .and(body_partial_json(serde_json::json!({ - "model": "test-model", - "max_tokens": 32, - }))) - .respond_with( - ResponseTemplate::new(200).set_body_json(serde_json::json!({"id": "msg_1"})), - ) - .mount(&mock_server) - .await; - - let client = reqwest::Client::builder().build().unwrap(); - let validated = verify_backend_endpoint(&client, &route).await.unwrap(); - - assert_eq!(validated.protocol, "anthropic_messages"); - assert_eq!(validated.url, format!("{}/v1/messages", mock_server.uri())); - } - - #[tokio::test] - async fn verify_backend_endpoint_accepts_mock_routes() { - let route = test_route( - "mock://test-backend", - &["openai_chat_completions"], - AuthHeader::Bearer, - ); - - let client = reqwest::Client::builder().build().unwrap(); - let validated = verify_backend_endpoint(&client, &route).await.unwrap(); - - assert_eq!(validated.protocol, "openai_chat_completions"); - assert_eq!(validated.url, "mock://test-backend/v1/chat/completions"); - } - - /// GPT-5+ models reject `max_tokens` — the primary probe uses - /// `max_completion_tokens` so validation should succeed directly. - #[tokio::test] - async fn verify_openai_chat_uses_max_completion_tokens() { - let mock_server = MockServer::start().await; - let route = test_route( - &mock_server.uri(), - &["openai_chat_completions"], - AuthHeader::Bearer, - ); - - Mock::given(method("POST")) - .and(path("/v1/chat/completions")) - .and(body_partial_json(serde_json::json!({ - "max_completion_tokens": 32, - }))) - .respond_with( - ResponseTemplate::new(200).set_body_json(serde_json::json!({"id": "chatcmpl-1"})), - ) - .mount(&mock_server) - .await; - - let client = reqwest::Client::builder().build().unwrap(); - let validated = verify_backend_endpoint(&client, &route).await.unwrap(); - - assert_eq!(validated.protocol, "openai_chat_completions"); - } - - /// Legacy/self-hosted backends that reject `max_completion_tokens` - /// should succeed on the fallback probe using `max_tokens`. - #[tokio::test] - async fn verify_openai_chat_falls_back_to_max_tokens() { - let mock_server = MockServer::start().await; - let route = test_route( - &mock_server.uri(), - &["openai_chat_completions"], - AuthHeader::Bearer, - ); - - // Reject the primary probe (max_completion_tokens) with 400. - Mock::given(method("POST")) - .and(path("/v1/chat/completions")) - .and(body_partial_json(serde_json::json!({ - "max_completion_tokens": 32, - }))) - .respond_with(ResponseTemplate::new(400).set_body_string( - r#"{"error":{"message":"Unsupported parameter: 'max_completion_tokens'"}}"#, - )) - .expect(1) - .mount(&mock_server) - .await; - - // Accept the fallback probe (max_tokens). - Mock::given(method("POST")) - .and(path("/v1/chat/completions")) - .and(body_partial_json(serde_json::json!({ - "max_tokens": 32, - }))) - .respond_with( - ResponseTemplate::new(200).set_body_json(serde_json::json!({"id": "chatcmpl-2"})), - ) - .expect(1) - .mount(&mock_server) - .await; - - let client = reqwest::Client::builder().build().unwrap(); - let validated = verify_backend_endpoint(&client, &route).await.unwrap(); - - assert_eq!(validated.protocol, "openai_chat_completions"); - } - - /// A managed route for an embeddings model advertises the full provider - /// protocol set. The chat probe (tried first) rejects the embeddings model - /// as wrong-shape, so validation must fall through to the embeddings probe - /// rather than fail the route. - #[tokio::test] - async fn verify_embeddings_model_falls_through_chat_probe() { - let mock_server = MockServer::start().await; - let route = test_route( - &mock_server.uri(), - &[ - "openai_chat_completions", - "openai_completions", - "openai_responses", - "openai_embeddings", - "model_discovery", - ], - AuthHeader::Bearer, - ); - - // Chat, completions, and responses probes reject the embedding model. - for chat_path in ["/v1/chat/completions", "/v1/completions", "/v1/responses"] { - Mock::given(method("POST")) - .and(path(chat_path)) - .respond_with( - ResponseTemplate::new(400) - .set_body_string(r#"{"error":{"message":"not a chat model"}}"#), - ) - .mount(&mock_server) - .await; - } - // The embeddings probe accepts it. - Mock::given(method("POST")) - .and(path("/v1/embeddings")) - .respond_with( - ResponseTemplate::new(200) - .set_body_json(serde_json::json!({"object": "list", "data": []})), - ) - .mount(&mock_server) - .await; - - let client = reqwest::Client::new(); - let validated = verify_backend_endpoint(&client, &route) - .await - .expect("embeddings model should validate via the embeddings probe"); - assert_eq!(validated.protocol, "openai_embeddings"); - } - - /// A non-request-shape failure (credentials) is terminal: validation must - /// stop at the first probe and not fall through to a protocol that would - /// succeed, so a bad key is reported as such rather than masked. - #[tokio::test] - async fn verify_stops_on_credentials_failure() { - let mock_server = MockServer::start().await; - let route = test_route( - &mock_server.uri(), - &["openai_chat_completions", "openai_embeddings"], - AuthHeader::Bearer, - ); - - Mock::given(method("POST")) - .and(path("/v1/chat/completions")) - .respond_with(ResponseTemplate::new(401).set_body_string(r#"{"error":"bad key"}"#)) - .mount(&mock_server) - .await; - // Would succeed, but credentials failure on the first probe is terminal - // and this must never be reached. - Mock::given(method("POST")) - .and(path("/v1/embeddings")) - .respond_with( - ResponseTemplate::new(200) - .set_body_json(serde_json::json!({"object": "list", "data": []})), - ) - .mount(&mock_server) - .await; - - let client = reqwest::Client::new(); - let err = verify_backend_endpoint(&client, &route) - .await - .expect_err("a 401 must fail validation"); - assert_eq!(err.kind, ValidationFailureKind::Credentials); - } - - /// A 429 on the first probe is terminal (`RateLimited`) and must not fall - /// through to a later probe that would succeed. - #[tokio::test] - async fn verify_stops_on_rate_limit() { - let mock_server = MockServer::start().await; - let route = test_route( - &mock_server.uri(), - &["openai_chat_completions", "openai_embeddings"], - AuthHeader::Bearer, - ); - Mock::given(method("POST")) - .and(path("/v1/chat/completions")) - .respond_with(ResponseTemplate::new(429).set_body_string(r#"{"error":"slow down"}"#)) - .mount(&mock_server) - .await; - Mock::given(method("POST")) - .and(path("/v1/embeddings")) - .respond_with( - ResponseTemplate::new(200) - .set_body_json(serde_json::json!({"object": "list", "data": []})), - ) - .mount(&mock_server) - .await; - - let err = reqwest_verify(&route).await; - assert_eq!(err.kind, ValidationFailureKind::RateLimited); - } - - /// An auth failure reported as HTTP 400 with an auth-shaped body is terminal - /// (`Credentials`), not a request-shape fall-through, so a bad key cannot be - /// masked by a later probe that accepts it. - #[tokio::test] - async fn verify_auth_error_as_400_is_terminal() { - let mock_server = MockServer::start().await; - let route = test_route( - &mock_server.uri(), - &["openai_chat_completions", "openai_embeddings"], - AuthHeader::Bearer, - ); - Mock::given(method("POST")) - .and(path("/v1/chat/completions")) - .respond_with(ResponseTemplate::new(400).set_body_string( - r#"{"error":{"code":"invalid_api_key","message":"Incorrect API key provided"}}"#, - )) - .mount(&mock_server) - .await; - Mock::given(method("POST")) - .and(path("/v1/embeddings")) - .respond_with( - ResponseTemplate::new(200) - .set_body_json(serde_json::json!({"object": "list", "data": []})), - ) - .mount(&mock_server) - .await; - - let err = reqwest_verify(&route).await; - assert_eq!(err.kind, ValidationFailureKind::Credentials); - } - - /// When every probe is rejected as request-shape, validation returns the - /// first (most-preferred protocol's) failure, not the last. - #[tokio::test] - async fn verify_all_probes_request_shape_returns_first() { - let mock_server = MockServer::start().await; - let route = test_route( - &mock_server.uri(), - &["openai_chat_completions", "openai_embeddings"], - AuthHeader::Bearer, - ); - Mock::given(method("POST")) - .and(path("/v1/chat/completions")) - .respond_with( - ResponseTemplate::new(404).set_body_string(r#"{"error":"model not found: chat"}"#), - ) - .mount(&mock_server) - .await; - Mock::given(method("POST")) - .and(path("/v1/embeddings")) - .respond_with( - ResponseTemplate::new(400) - .set_body_string(r#"{"error":"not an embeddings model"}"#), - ) - .mount(&mock_server) - .await; - - let err = reqwest_verify(&route).await; - assert_eq!(err.kind, ValidationFailureKind::RequestShape); - assert!( - err.details.contains("model not found: chat"), - "should report the first (chat) failure, got: {}", - err.details - ); - } - - /// Helper: run `verify_backend_endpoint` and return the expected failure. - async fn reqwest_verify(route: &ResolvedRoute) -> ValidationFailure { - verify_backend_endpoint(&reqwest::Client::new(), route) - .await - .expect_err("validation should fail") - } - - /// Non-chat-completions probes (e.g. `anthropic_messages`) should not - /// have a fallback — a 400 remains a hard failure. - #[tokio::test] - async fn verify_non_chat_completions_no_fallback() { - let mock_server = MockServer::start().await; - let route = test_route( - &mock_server.uri(), - &["anthropic_messages"], - AuthHeader::Custom("x-api-key"), - ); - - Mock::given(method("POST")) - .and(path("/v1/messages")) - .respond_with(ResponseTemplate::new(400).set_body_string("bad request")) - .mount(&mock_server) - .await; - - let client = reqwest::Client::builder().build().unwrap(); - let result = verify_backend_endpoint(&client, &route).await; - - assert!(result.is_err()); - assert_eq!( - result.unwrap_err().kind, - ValidationFailureKind::RequestShape - ); - } - - #[tokio::test] - async fn verify_vertex_anthropic_route_uses_buffered_rawpredict_probe() { - let mock_server = MockServer::start().await; - let route = ResolvedRoute { - name: "vertex-anthropic".to_string(), - endpoint: format!( - "{}/v1/projects/my-project/locations/us-east5/publishers/anthropic/models", - mock_server.uri() - ), - model: "claude-3-5-sonnet@20241022".to_string(), - api_key: "ya29.token".to_string(), - protocols: vec!["anthropic_messages".to_string()], - auth: AuthHeader::Bearer, - default_headers: Vec::new(), - passthrough_headers: Vec::new(), - timeout: DEFAULT_ROUTE_TIMEOUT, - model_in_path: true, - request_path_override: Some(":rawPredict".to_string()), - }; - - Mock::given(method("POST")) - .and(path( - "/v1/projects/my-project/locations/us-east5/publishers/anthropic/models/claude-3-5-sonnet@20241022:rawPredict", - )) - .respond_with(ResponseTemplate::new(200).set_body_json(serde_json::json!({ - "id": "msg_vertex_verify" - }))) - .mount(&mock_server) - .await; - - let client = reqwest::Client::builder().build().unwrap(); - let validated = verify_backend_endpoint(&client, &route).await.unwrap(); - assert!( - validated.url.ends_with(":rawPredict"), - "buffered verification should probe the unary Vertex endpoint, got: {}", - validated.url - ); - } - - /// Vertex AI pattern: `model_in_path=true`, `request_path_override=Some(":rawPredict")` - /// means buffered requests POST to `base_url/model_id:rawPredict`. - #[test] - fn build_provider_url_model_in_path_with_suffix() { - let route = ResolvedRoute { - name: "inference.local".to_string(), - endpoint: - "https://us-east5-aiplatform.googleapis.com/v1/projects/my-project/locations/us-east5/publishers/anthropic/models" - .to_string(), - model: "claude-3-5-sonnet@20241022".to_string(), - api_key: "token".to_string(), - protocols: vec!["anthropic_messages".to_string()], - auth: AuthHeader::Bearer, - default_headers: Vec::new(), - passthrough_headers: Vec::new(), - timeout: DEFAULT_ROUTE_TIMEOUT, - model_in_path: true, - request_path_override: Some(":rawPredict".to_string()), - }; - - let url = build_provider_url(&route, "claude-3-5-sonnet@20241022", "/v1/messages", false); - assert!( - url.ends_with("/claude-3-5-sonnet@20241022:rawPredict"), - "expected URL to end with model id and suffix, got: {url}" - ); - assert!( - !url.contains("/v1/messages"), - "expected no protocol path appended, got: {url}" - ); - } - - #[test] - fn build_provider_url_vertex_anthropic_streaming_upgrades_to_stream_rawpredict() { - let route = ResolvedRoute { - name: "inference.local".to_string(), - endpoint: - "https://us-east5-aiplatform.googleapis.com/v1/projects/my-project/locations/us-east5/publishers/anthropic/models" - .to_string(), - model: "claude-3-5-sonnet@20241022".to_string(), - api_key: "token".to_string(), - protocols: vec!["anthropic_messages".to_string()], - auth: AuthHeader::Bearer, - default_headers: Vec::new(), - passthrough_headers: Vec::new(), - timeout: DEFAULT_ROUTE_TIMEOUT, - model_in_path: true, - request_path_override: Some(":rawPredict".to_string()), - }; - - let url = build_provider_url(&route, "claude-3-5-sonnet@20241022", "/v1/messages", true); - assert!( - url.ends_with("/claude-3-5-sonnet@20241022:streamRawPredict"), - "expected streaming URL to upgrade the suffix, got: {url}" - ); - } - - /// Vertex AI pattern: `model_in_path=true`, `request_path_override=Some("")` (empty suffix) - /// means POST directly to `base_url/model_id` with no additional path segment. - #[test] - fn build_provider_url_model_in_path_empty_suffix() { - let route = ResolvedRoute { - name: "inference.local".to_string(), - endpoint: "https://example.com/models".to_string(), - model: "my-model".to_string(), - api_key: "token".to_string(), - protocols: vec!["anthropic_messages".to_string()], - auth: AuthHeader::Bearer, - default_headers: Vec::new(), - passthrough_headers: Vec::new(), - timeout: DEFAULT_ROUTE_TIMEOUT, - model_in_path: true, - request_path_override: Some(String::new()), - }; - - let url = build_provider_url(&route, "my-model", "/v1/messages", false); - assert_eq!(url, "https://example.com/models/my-model"); - } - - /// Explicit path override: `request_path_override=Some("/v1/chat/completions")` - /// appends the override path to `base_url`, ignoring `model_in_path`. - #[test] - fn build_provider_url_with_path_override() { - let route = ResolvedRoute { - name: "inference.local".to_string(), - endpoint: "https://api.example.com".to_string(), - model: "some-model".to_string(), - api_key: "key".to_string(), - protocols: vec!["openai_chat_completions".to_string()], - auth: AuthHeader::Bearer, - default_headers: Vec::new(), - passthrough_headers: Vec::new(), - timeout: DEFAULT_ROUTE_TIMEOUT, - model_in_path: false, - request_path_override: Some("/v1/chat/completions".to_string()), - }; - - let url = build_provider_url(&route, "some-model", "/v1/chat/completions", false); - assert!( - url.ends_with("/v1/chat/completions"), - "expected URL to end with path override, got: {url}" - ); - } - - /// Default behavior: `model_in_path=false`, `request_path_override=None` uses - /// the existing `build_backend_url` logic (protocol-derived path only). - #[test] - fn build_provider_url_default_behavior() { - let route = ResolvedRoute { - name: "inference.local".to_string(), - endpoint: "https://api.openai.com/v1".to_string(), - model: "gpt-4o".to_string(), - api_key: "key".to_string(), - protocols: vec!["openai_chat_completions".to_string()], - auth: AuthHeader::Bearer, - default_headers: Vec::new(), - passthrough_headers: Vec::new(), - timeout: DEFAULT_ROUTE_TIMEOUT, - model_in_path: false, - request_path_override: None, - }; - - let url = build_provider_url(&route, "gpt-4o", "/v1/chat/completions", false); - assert_eq!( - url, "https://api.openai.com/v1/chat/completions", - "default behavior should dedupe v1 prefix and use protocol path" - ); - } - - #[test] - fn build_provider_url_override_path_normalizes_missing_leading_slash() { - // An override_path without a leading '/' must not produce a broken URL. - let route = ResolvedRoute { - name: "test".to_string(), - endpoint: "https://example.com/v1/projects/proj/locations/us/endpoints/openapi" - .to_string(), - model: "gemini-pro".to_string(), - api_key: "key".to_string(), - protocols: vec!["openai_chat_completions".to_string()], - auth: AuthHeader::Bearer, - default_headers: Vec::new(), - passthrough_headers: Vec::new(), - timeout: DEFAULT_ROUTE_TIMEOUT, - model_in_path: false, - request_path_override: Some("chat/completions".to_string()), // no leading slash - }; - let url = build_provider_url(&route, &route.model, "/v1/chat/completions", false); - // Must not produce https://...openaichat/completions - assert!( - url.contains("/chat/completions"), - "URL must contain /chat/completions, got: {url}" - ); - assert!( - !url.contains("openaichat"), - "URL must not smash endpoint and path, got: {url}" - ); - assert_eq!( - url, - "https://example.com/v1/projects/proj/locations/us/endpoints/openapi/chat/completions" - ); - } - - /// Vertex AI Anthropic routes require `anthropic_version` in the request body. - /// Verify it is injected on the buffered `:rawPredict` path when the client - /// did not already include it. - #[tokio::test] - async fn vertex_ai_body_injects_anthropic_version() { - let mock_server = MockServer::start().await; - - // Build a Vertex-AI-style route: model in path, suffix :rawPredict - let base_path = "/v1/projects/my-project/locations/us-east5/publishers/anthropic/models"; - let route = ResolvedRoute { - name: "vertex-anthropic".to_string(), - endpoint: format!("{}{base_path}", mock_server.uri()), - model: "claude-3-5-sonnet@20241022".to_string(), - api_key: "ya29.token".to_string(), - protocols: vec!["anthropic_messages".to_string()], - auth: AuthHeader::Bearer, - default_headers: Vec::new(), - passthrough_headers: Vec::new(), - timeout: DEFAULT_ROUTE_TIMEOUT, - model_in_path: true, - request_path_override: Some(":rawPredict".to_string()), - }; - - Mock::given(method("POST")) - .and(path(format!( - "{base_path}/claude-3-5-sonnet@20241022:rawPredict" - ))) - .and(body_partial_json(serde_json::json!({ - "anthropic_version": "vertex-2023-10-16", - }))) - .respond_with( - ResponseTemplate::new(200).set_body_json(serde_json::json!({"id": "msg_vertex_1"})), - ) - .mount(&mock_server) - .await; - - let client = reqwest::Client::builder().build().unwrap(); - let body = bytes::Bytes::from( - serde_json::to_vec(&serde_json::json!({ - "messages": [{"role": "user", "content": "ping"}], - "max_tokens": 32, - })) - .unwrap(), - ); - let headers = vec![("content-type".to_string(), "application/json".to_string())]; - - let (builder, _url) = prepare_backend_request( - &client, - &route, - "POST", - "/v1/messages", - &headers, - body, - false, - ) - .unwrap(); - - let response = builder.send().await.unwrap(); - assert_eq!( - response.status().as_u16(), - 200, - "mock should match body with anthropic_version injected" - ); - let received = mock_server.received_requests().await.unwrap(); - assert_eq!(received.len(), 1); - let received_body: serde_json::Value = serde_json::from_slice(&received[0].body).unwrap(); - assert!( - !received_body.as_object().unwrap().contains_key("model"), - "Vertex Anthropic route must not inject model into the body, got: {received_body}" - ); - } - - /// Claude Code and other Anthropic SDK clients always send "model" in the - /// request body. For Vertex AI rawPredict routes the model is in the URL - /// path; the body field must be stripped to avoid HTTP 400 - /// "Extra inputs are not permitted" from the Vertex AI backend. - #[tokio::test] - async fn vertex_ai_body_strips_client_model_field() { - let mock_server = MockServer::start().await; - - let base_path = "/v1/projects/my-project/locations/us-east5/publishers/anthropic/models"; - let route = ResolvedRoute { - name: "vertex-anthropic".to_string(), - endpoint: format!("{}{base_path}", mock_server.uri()), - model: "claude-3-5-sonnet@20241022".to_string(), - api_key: "ya29.token".to_string(), - protocols: vec!["anthropic_messages".to_string()], - auth: AuthHeader::Bearer, - default_headers: Vec::new(), - passthrough_headers: Vec::new(), - timeout: DEFAULT_ROUTE_TIMEOUT, - model_in_path: true, - request_path_override: Some(":rawPredict".to_string()), - }; - - Mock::given(method("POST")) - .and(path(format!( - "{base_path}/claude-3-5-sonnet@20241022:rawPredict" - ))) - .respond_with( - ResponseTemplate::new(200).set_body_json(serde_json::json!({"id": "msg_1"})), - ) - .mount(&mock_server) - .await; - - let client = reqwest::Client::builder().build().unwrap(); - // Simulate a client (e.g. Claude Code) that always sends "model" in the body. - let body = bytes::Bytes::from( - serde_json::to_vec(&serde_json::json!({ - "model": "claude-3-5-sonnet-20241022", - "messages": [{"role": "user", "content": "ping"}], - "max_tokens": 32, - })) - .unwrap(), - ); - let headers = vec![("content-type".to_string(), "application/json".to_string())]; - - let (builder, _url) = prepare_backend_request( - &client, - &route, - "POST", - "/v1/messages", - &headers, - body, - false, - ) - .unwrap(); - - let response = builder.send().await.unwrap(); - assert_eq!(response.status().as_u16(), 200); - let received = mock_server.received_requests().await.unwrap(); - let received_body: serde_json::Value = serde_json::from_slice(&received[0].body).unwrap(); - assert!( - !received_body.as_object().unwrap().contains_key("model"), - "model field must be stripped from Vertex AI rawPredict body, got: {received_body}" - ); - } - - #[tokio::test] - async fn vertex_ai_body_strips_unsupported_beta_fields() { - let mock_server = MockServer::start().await; - - let base_path = "/v1/projects/my-project/locations/us-east5/publishers/anthropic/models"; - let route = ResolvedRoute { - name: "vertex-anthropic".to_string(), - endpoint: format!("{}{base_path}", mock_server.uri()), - model: "claude-sonnet-4-6@20250514".to_string(), - api_key: "ya29.token".to_string(), - protocols: vec!["anthropic_messages".to_string()], - auth: AuthHeader::Bearer, - default_headers: Vec::new(), - passthrough_headers: Vec::new(), - timeout: DEFAULT_ROUTE_TIMEOUT, - model_in_path: true, - request_path_override: Some(":rawPredict".to_string()), - }; - - Mock::given(method("POST")) - .and(path(format!( - "{base_path}/claude-sonnet-4-6@20250514:rawPredict" - ))) - .respond_with( - ResponseTemplate::new(200).set_body_json(serde_json::json!({"id": "msg_1"})), - ) - .mount(&mock_server) - .await; - - let client = reqwest::Client::builder().build().unwrap(); - let body = bytes::Bytes::from( - serde_json::to_vec(&serde_json::json!({ - "model": "claude-sonnet-4-6-20250514", - "messages": [{"role": "user", "content": "ping"}], - "max_tokens": 32, - "context_management": {"enabled": true}, - })) - .unwrap(), - ); - let headers = vec![("content-type".to_string(), "application/json".to_string())]; - - let (builder, _url) = prepare_backend_request( - &client, - &route, - "POST", - "/v1/messages", - &headers, - body, - false, - ) - .unwrap(); - - let response = builder.send().await.unwrap(); - assert_eq!(response.status().as_u16(), 200); - let received = mock_server.received_requests().await.unwrap(); - let received_body: serde_json::Value = serde_json::from_slice(&received[0].body).unwrap(); - let obj = received_body.as_object().unwrap(); - assert!( - !obj.contains_key("context_management"), - "context_management must be stripped for Vertex AI rawPredict, got: {received_body}" - ); - assert!( - !obj.contains_key("model"), - "model must also be stripped for Vertex AI rawPredict, got: {received_body}" - ); - assert!( - obj.contains_key("messages"), - "standard fields must be preserved, got: {received_body}" - ); - } - - #[tokio::test] - async fn direct_anthropic_preserves_beta_fields() { - let route = ResolvedRoute { - name: "direct-anthropic".to_string(), - endpoint: "https://api.anthropic.com/v1".to_string(), - model: "claude-sonnet-4-6-20250514".to_string(), - api_key: "sk-test".to_string(), - protocols: vec!["anthropic_messages".to_string()], - auth: AuthHeader::Custom("x-api-key"), - default_headers: Vec::new(), - passthrough_headers: Vec::new(), - timeout: DEFAULT_ROUTE_TIMEOUT, - model_in_path: false, - request_path_override: None, - }; - - let client = reqwest::Client::builder().build().unwrap(); - let body = bytes::Bytes::from( - serde_json::to_vec(&serde_json::json!({ - "model": "claude-sonnet-4-6-20250514", - "messages": [{"role": "user", "content": "ping"}], - "max_tokens": 32, - "context_management": {"enabled": true}, - })) - .unwrap(), - ); - let headers = vec![("content-type".to_string(), "application/json".to_string())]; - - let (builder, _url) = prepare_backend_request( - &client, - &route, - "POST", - "/v1/messages", - &headers, - body, - false, - ) - .unwrap(); - - let request = builder.build().unwrap(); - let sent_body: serde_json::Value = - serde_json::from_slice(request.body().unwrap().as_bytes().unwrap()).unwrap(); - assert!( - sent_body - .as_object() - .unwrap() - .contains_key("context_management"), - "context_management must be preserved for direct Anthropic API routes" - ); - } - - #[tokio::test] - async fn vertex_ai_body_preserves_client_anthropic_version() { - // When the client already sends anthropic_version, the router must NOT overwrite it. - let mock_server = MockServer::start().await; - - // Expect the body to contain the client's version, NOT "vertex-2023-10-16" - Mock::given(method("POST")) - .and(body_partial_json(serde_json::json!({ - "anthropic_version": "custom-client-version", - }))) - .respond_with(ResponseTemplate::new(200).set_body_json(serde_json::json!({ - "id": "msg_1", - "type": "message", - "role": "assistant", - "model": "claude-3-5-sonnet@20241022", - "content": [{"type": "text", "text": "ok"}] - }))) - .mount(&mock_server) - .await; - - let router = crate::Router::new().unwrap(); - let candidates = vec![ResolvedRoute { - name: "vertex-test".to_string(), - endpoint: format!( - "{}/v1/projects/proj/locations/us-east5/publishers/anthropic/models", - mock_server.uri() - ), - model: "claude-3-5-sonnet@20241022".to_string(), - api_key: "ya29.test".to_string(), - protocols: vec!["anthropic_messages".to_string()], - auth: AuthHeader::Bearer, - default_headers: Vec::new(), - passthrough_headers: Vec::new(), - timeout: DEFAULT_ROUTE_TIMEOUT, - model_in_path: true, - request_path_override: Some(":rawPredict".to_string()), - }]; - - let body = serde_json::to_vec(&serde_json::json!({ - "messages": [{"role": "user", "content": "ping"}], - "max_tokens": 32, - "anthropic_version": "custom-client-version", - })) - .unwrap(); - - let response = router - .proxy_with_candidates( - "anthropic_messages", - "POST", - "/v1/messages", - vec![("content-type".to_string(), "application/json".to_string())], - bytes::Bytes::from(body), - &candidates, - ) - .await - .unwrap(); - - assert_eq!( - response.status, 200, - "proxy should succeed when client sends anthropic_version" - ); - } - - /// Standard Anthropic route (`model_in_path=false`) must NOT inject `anthropic_version`. - /// Vertex body injection must not affect non-Vertex Anthropic providers. - #[tokio::test] - async fn standard_anthropic_body_does_not_inject_vertex_anthropic_version() { - let mock_server = MockServer::start().await; - - let route = ResolvedRoute { - name: "anthropic-direct".to_string(), - endpoint: mock_server.uri(), - model: "claude-3-5-sonnet-20241022".to_string(), - api_key: "sk-ant-test".to_string(), - protocols: vec!["anthropic_messages".to_string()], - auth: AuthHeader::Custom("x-api-key"), - default_headers: Vec::new(), - passthrough_headers: Vec::new(), - timeout: DEFAULT_ROUTE_TIMEOUT, - model_in_path: false, - request_path_override: None, - }; - - Mock::given(method("POST")) - .and(path("/v1/messages")) - .respond_with( - ResponseTemplate::new(200).set_body_json(serde_json::json!({"id": "msg_1"})), - ) - .mount(&mock_server) - .await; - - let client = reqwest::Client::builder().build().unwrap(); - let body = bytes::Bytes::from( - serde_json::to_vec(&serde_json::json!({ - "messages": [{"role": "user", "content": "ping"}], - "max_tokens": 32, - })) - .unwrap(), - ); - let headers = vec![("content-type".to_string(), "application/json".to_string())]; - - let (builder, _url) = prepare_backend_request( - &client, - &route, - "POST", - "/v1/messages", - &headers, - body, - false, - ) - .unwrap(); - - builder.send().await.unwrap(); - - let received = mock_server.received_requests().await.unwrap(); - assert_eq!(received.len(), 1); - let received_body: serde_json::Value = serde_json::from_slice(&received[0].body).unwrap(); - assert!( - !received_body - .as_object() - .unwrap() - .contains_key("anthropic_version"), - "standard Anthropic route must not inject anthropic_version, got: {received_body}" - ); - } - - /// Model-in-path alone is not enough; only Vertex rawPredict-style routes should inject. - #[tokio::test] - async fn anthropic_model_in_path_without_rawpredict_suffix_does_not_inject_version() { - let mock_server = MockServer::start().await; - - let route = ResolvedRoute { - name: "non-vertex-model-path".to_string(), - endpoint: format!("{}/publisher/models", mock_server.uri()), - model: "claude-3-5-sonnet@20241022".to_string(), - api_key: "token".to_string(), - protocols: vec!["anthropic_messages".to_string()], - auth: AuthHeader::Bearer, - default_headers: Vec::new(), - passthrough_headers: Vec::new(), - timeout: DEFAULT_ROUTE_TIMEOUT, - model_in_path: true, - request_path_override: Some(String::new()), - }; - - Mock::given(method("POST")) - .respond_with( - ResponseTemplate::new(200) - .set_body_json(serde_json::json!({"id": "msg_model_path"})), - ) - .mount(&mock_server) - .await; - - let client = reqwest::Client::builder().build().unwrap(); - let body = bytes::Bytes::from( - serde_json::to_vec(&serde_json::json!({ - "messages": [{"role": "user", "content": "ping"}], - "max_tokens": 32, - })) - .unwrap(), - ); - let headers = vec![("content-type".to_string(), "application/json".to_string())]; - - let (builder, _url) = prepare_backend_request( - &client, - &route, - "POST", - "/v1/messages", - &headers, - body, - false, - ) - .unwrap(); - - builder.send().await.unwrap(); - - let received = mock_server.received_requests().await.unwrap(); - assert_eq!(received.len(), 1); - let received_body: serde_json::Value = serde_json::from_slice(&received[0].body).unwrap(); - assert!( - !received_body - .as_object() - .unwrap() - .contains_key("anthropic_version"), - "non-rawPredict model-in-path routes must not inject anthropic_version, got: {received_body}" - ); - } - - /// Vertex AI Gemini route (`model_in_path=false`, `openai_chat_completions`) must NOT inject. - #[tokio::test] - async fn vertex_gemini_body_does_not_inject_vertex_anthropic_version() { - let mock_server = MockServer::start().await; - - let route = ResolvedRoute { - name: "vertex-gemini".to_string(), - endpoint: format!( - "{}/v1beta1/projects/my-project/locations/us-central1/endpoints/openapi", - mock_server.uri() - ), - model: "gemini-pro".to_string(), - api_key: "ya29.token".to_string(), - protocols: vec!["openai_chat_completions".to_string()], - auth: AuthHeader::Bearer, - default_headers: Vec::new(), - passthrough_headers: Vec::new(), - timeout: DEFAULT_ROUTE_TIMEOUT, - model_in_path: false, - request_path_override: None, - }; - - Mock::given(method("POST")) - .respond_with( - ResponseTemplate::new(200).set_body_json(serde_json::json!({"id": "msg_gemini"})), - ) - .mount(&mock_server) - .await; - - let client = reqwest::Client::builder().build().unwrap(); - let body = bytes::Bytes::from( - serde_json::to_vec(&serde_json::json!({ - "messages": [{"role": "user", "content": "ping"}], - "max_tokens": 32, - })) - .unwrap(), - ); - let headers = vec![("content-type".to_string(), "application/json".to_string())]; - - let (builder, _url) = prepare_backend_request( - &client, - &route, - "POST", - "/v1/chat/completions", - &headers, - body, - false, - ) - .unwrap(); - - builder.send().await.unwrap(); - - let received = mock_server.received_requests().await.unwrap(); - assert_eq!(received.len(), 1); - let received_body: serde_json::Value = serde_json::from_slice(&received[0].body).unwrap(); - assert!( - !received_body - .as_object() - .unwrap() - .contains_key("anthropic_version"), - "Vertex Gemini route must not inject anthropic_version, got: {received_body}" - ); - assert_eq!( - received_body - .as_object() - .unwrap() - .get("model") - .and_then(serde_json::Value::as_str), - Some("gemini-pro"), - "Vertex Gemini route must still rewrite the model field, got: {received_body}" - ); - } - - // ============================================================ - // AWS Bedrock route shaping (path rewriting + body preservation) - // ============================================================ - - /// `parse_bedrock_invocation_path` rejects malformed paths. - #[test] - fn parse_bedrock_invocation_path_rejects_malformed() { - // Empty model id: `/model//invoke` - assert!(parse_bedrock_invocation_path("/model//invoke").is_none()); - // Multi-segment model id: `/model/a/b/invoke` - assert!(parse_bedrock_invocation_path("/model/a/b/invoke").is_none()); - // Unknown action: `/model/foo/converse` - assert!(parse_bedrock_invocation_path("/model/foo/converse").is_none()); - // Streaming variant is deferred until protocol-aware error - // framing exists; the parser must reject it the same way it - // rejects any other unknown action. - assert!(parse_bedrock_invocation_path("/model/foo/invoke-with-response-stream").is_none()); - // Wrong prefix: `/v1/messages` - assert!(parse_bedrock_invocation_path("/v1/messages").is_none()); - // Missing slash before action - assert!(parse_bedrock_invocation_path("/model/foo").is_none()); - } - - #[test] - fn parse_bedrock_invocation_path_accepts_invoke() { - let parsed = parse_bedrock_invocation_path( - "/model/anthropic.claude-3-5-sonnet-20241022-v2:0/invoke", - ); - assert_eq!( - parsed, - Some(("anthropic.claude-3-5-sonnet-20241022-v2:0", "/invoke", "")) - ); - } - - /// Query strings on Bedrock invoke paths are preserved through the - /// rewrite so the matcher (which accepts queries) and the upstream - /// see the same shape. - #[test] - fn parse_bedrock_invocation_path_preserves_query_string() { - let parsed = - parse_bedrock_invocation_path("/model/anthropic.claude-opus-4-7/invoke?trace=1"); - assert_eq!( - parsed, - Some(("anthropic.claude-opus-4-7", "/invoke", "?trace=1")) - ); - } - - /// `route_is_bedrock` matches the Bedrock invocation protocol(s). - /// `aws_bedrock_invoke_stream` is recognized for forward-compatibility - /// even though no L7 pattern advertises it today. - #[test] - fn route_is_bedrock_matches_invoke_protocols() { - let invoke_only = test_route( - "https://example.com", - &["aws_bedrock_invoke"], - AuthHeader::None, - ); - assert!(route_is_bedrock(&invoke_only)); - - let stream_forward_compat = test_route( - "https://example.com", - &["aws_bedrock_invoke_stream"], - AuthHeader::None, - ); - assert!(route_is_bedrock(&stream_forward_compat)); - - let openai = test_route( - "https://example.com", - &["openai_chat_completions"], - AuthHeader::Bearer, - ); - assert!(!route_is_bedrock(&openai)); - } - - /// `rewrite_bedrock_path` swaps caller's model segment for the - /// route-configured model and preserves any query string. - #[test] - fn rewrite_bedrock_path_substitutes_operator_model() { - let mut route = test_route( - "https://bedrock-bridge.example", - &["aws_bedrock_invoke"], - AuthHeader::None, - ); - route.model = "anthropic.claude-opus-4-7".to_string(); - - let rewritten = rewrite_bedrock_path(&route, "/model/some-other-model/invoke"); - assert_eq!( - rewritten, - Some("/model/anthropic.claude-opus-4-7/invoke".to_string()) - ); - - let rewritten_with_query = - rewrite_bedrock_path(&route, "/model/some-other-model/invoke?trace=1"); - assert_eq!( - rewritten_with_query, - Some("/model/anthropic.claude-opus-4-7/invoke?trace=1".to_string()) - ); - } - - #[test] - fn rewrite_bedrock_path_returns_none_for_non_bedrock_path() { - let route = test_route( - "https://bedrock-bridge.example", - &["aws_bedrock_invoke"], - AuthHeader::None, - ); - assert_eq!(rewrite_bedrock_path(&route, "/v1/messages"), None); - assert_eq!(rewrite_bedrock_path(&route, "/model//invoke"), None); - assert_eq!(rewrite_bedrock_path(&route, "/model/a/b/invoke"), None); - // Streaming variant is deferred at the L7 layer; the router - // must not produce an upstream path for it either. - assert_eq!( - rewrite_bedrock_path(&route, "/model/x/invoke-with-response-stream"), - None - ); - } - - /// Defense-in-depth: `rewrite_bedrock_path` rejects route models - /// that would produce ambiguous or malformed upstream URL paths, - /// even if a malformed value somehow reached the router store. - #[test] - fn rewrite_bedrock_path_rejects_unsafe_route_model() { - let mut route = test_route( - "https://bedrock-bridge.example", - &["aws_bedrock_invoke"], - AuthHeader::None, - ); - - for unsafe_model in [ - "anthropic.claude/../../etc/passwd", - "anthropic.claude\\backslash", - "model?injected=1", - "model#fragment", - "percent%2fencoded", - "..", - " leading-space", - "trailing-space ", - "tab\there", - "newline\nhere", - "", - ] { - route.model = unsafe_model.to_string(); - assert!( - rewrite_bedrock_path(&route, "/model/foo/invoke").is_none(), - "rewrite_bedrock_path must reject unsafe route.model: {unsafe_model:?}" - ); - } - } - - /// End-to-end: an inbound Bedrock request that names a different - /// model in the path arrives at the upstream/bridge with the - /// operator's model, and the body is unchanged (no `"model"` - /// injection). - #[tokio::test] - async fn bedrock_route_rewrites_model_in_path_and_preserves_body() { - let mock_server = MockServer::start().await; - let mut route = test_route( - &mock_server.uri(), - &["aws_bedrock_invoke"], - AuthHeader::None, - ); - route.model = "anthropic.claude-opus-4-7".to_string(); - - // The mock asserts the upstream sees the operator's model in - // the path, NOT the caller's model. - Mock::given(method("POST")) - .and(path("/model/anthropic.claude-opus-4-7/invoke")) - // Caller body has a "model" key; we expect it to pass - // through unchanged. The mock uses body_partial_json so - // additional fields are OK; the assertion below pins the - // body more tightly. - .respond_with(ResponseTemplate::new(200).set_body_json(serde_json::json!({"ok": true}))) - .mount(&mock_server) - .await; - - let client = reqwest::Client::builder() - .timeout(Duration::from_secs(5)) - .build() - .expect("client"); - - // Caller-supplied body — we deliberately include a "model" - // field naming a DIFFERENT model than the operator's, to - // verify the router does not inject route.model on top of - // it. The body should pass through verbatim because Bedrock - // encodes the model in the path. - let caller_body = serde_json::json!({ - "model": "caller-supplied-model-name", - "messages": [{"role": "user", "content": "hi"}], - }); - - let (builder, url) = prepare_backend_request( - &client, - &route, - "POST", - "/model/some-other-model/invoke", - &[], - bytes::Bytes::from(caller_body.to_string()), - false, - ) - .expect("prepare should succeed"); - - // URL should target the operator's model, not the caller's. - assert!( - url.ends_with("/model/anthropic.claude-opus-4-7/invoke"), - "URL must use operator model, got: {url}" - ); - - let resp = builder.send().await.expect("send"); - assert_eq!(resp.status(), 200); - - // Inspect what wiremock actually received. - let received = mock_server.received_requests().await.expect("requests"); - assert_eq!(received.len(), 1); - let req = &received[0]; - let received_body: serde_json::Value = - serde_json::from_slice(&req.body).expect("json body"); - // Caller's model name should pass through (NOT replaced by - // route.model). This proves the body is untouched. - assert_eq!( - received_body.get("model").and_then(|v| v.as_str()), - Some("caller-supplied-model-name"), - "Bedrock route must NOT rewrite body model, got: {received_body}" - ); - assert!( - received_body.get("messages").is_some(), - "messages field should pass through unchanged" - ); - } - - /// Vertex AI's OpenAI-compatible endpoint requires the body `model` field to - /// carry a publisher prefix (e.g. `google/gemini-2.5-flash`). This test - /// simulates the fix: `resolve_vertex_ai_route` sets `route.model` to the - /// prefixed form, and the body rewrite here forwards that value to Vertex. - /// - /// The mock server only accepts the prefixed form — matching Vertex's - /// behaviour — and returns 400 "Malformed publisher model" for the bare name. - #[tokio::test] - async fn vertex_openai_compat_rewrites_body_model_to_publisher_prefixed_form() { - let mock_server = MockServer::start().await; - - // Simulate Vertex accepting only the publisher-prefixed model name. - Mock::given(method("POST")) - .and(path("/chat/completions")) - .and(body_partial_json( - serde_json::json!({"model": "google/gemini-2.5-flash"}), - )) - .respond_with( - ResponseTemplate::new(200).set_body_json( - serde_json::json!({"choices": [{"message": {"content": "hi"}}]}), - ), - ) - .expect(1) - .mount(&mock_server) - .await; - - // Simulate Vertex rejecting the bare model name — the pre-fix failure. - Mock::given(method("POST")) - .and(path("/chat/completions")) - .and(body_partial_json( - serde_json::json!({"model": "gemini-2.5-flash"}), - )) - .respond_with(ResponseTemplate::new(400).set_body_json( - serde_json::json!({"error": {"message": "Malformed publisher model"}}), - )) - .expect(0) // must never be reached after the fix - .mount(&mock_server) - .await; - - // Route as produced by resolve_vertex_ai_route after the fix: - // route.model carries the publisher prefix. - let route = ResolvedRoute { - name: "vertex-gemini".to_string(), - endpoint: mock_server.uri(), - model: "google/gemini-2.5-flash".to_string(), - api_key: "ya29.token".to_string(), - protocols: vec!["openai_chat_completions".to_string()], - auth: AuthHeader::Bearer, - default_headers: vec![], - passthrough_headers: vec![], - timeout: DEFAULT_ROUTE_TIMEOUT, - model_in_path: false, - request_path_override: Some("/chat/completions".to_string()), - }; - - // The client sends the bare model name; the body rewrite must replace it - // with route.model (the publisher-prefixed form) before forwarding. - let client_body = serde_json::to_vec(&serde_json::json!({ - "model": "gemini-2.5-flash", - "messages": [{"role": "user", "content": "hello"}] - })) - .unwrap(); - - let client = reqwest::Client::new(); - let result = proxy_to_backend( - &client, - &route, - "openai_chat_completions", - "POST", - "/chat/completions", - vec![("content-type".to_string(), "application/json".to_string())], - bytes::Bytes::from(client_body), - ) - .await - .expect("proxy should succeed"); - - assert_eq!( - result.status, 200, - "Vertex mock must accept the publisher-prefixed model; \ - got {}: body rewrite did not apply the prefix", - result.status - ); - } - - /// Defense-in-depth: a Bedrock route receiving a non-Bedrock path - /// is rejected rather than forwarded. The L7 pattern detector - /// upstream of the router should never produce this combination, - /// but if it ever did, we must not silently forward. - #[test] - fn bedrock_route_rejects_non_bedrock_path() { - let client = reqwest::Client::new(); - let route = test_route( - "https://bedrock-bridge.example", - &["aws_bedrock_invoke"], - AuthHeader::None, - ); - let result = prepare_backend_request( - &client, - &route, - "POST", - "/v1/messages", - &[], - bytes::Bytes::from(r"{}"), - false, - ); - match result { - Err(RouterError::Internal(msg)) => { - assert!( - msg.contains("Bedrock") && msg.contains("/v1/messages"), - "error must name the offending path, got: {msg}" - ); - } - other => panic!("expected RouterError::Internal, got {other:?}"), - } - } -} diff --git a/crates/openshell-router/src/config.rs b/crates/openshell-router/src/config.rs deleted file mode 100644 index 81fac6048..000000000 --- a/crates/openshell-router/src/config.rs +++ /dev/null @@ -1,348 +0,0 @@ -// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -// SPDX-License-Identifier: Apache-2.0 - -use serde::Deserialize; -use std::path::Path; -use std::time::Duration; - -pub use openshell_core::inference::AuthHeader; - -use crate::RouterError; - -pub const DEFAULT_ROUTE_TIMEOUT: Duration = Duration::from_secs(60); - -#[derive(Debug, Clone, Deserialize)] -pub struct RouterConfig { - pub routes: Vec, -} - -#[derive(Debug, Clone, Deserialize)] -pub struct RouteConfig { - pub name: String, - pub endpoint: String, - pub model: String, - #[serde(default)] - pub provider_type: Option, - #[serde(default)] - pub protocols: Vec, - #[serde(default)] - pub api_key: Option, - #[serde(default)] - pub api_key_env: Option, -} - -/// A fully-resolved route ready for the router to forward requests. -/// -/// The router is provider-agnostic — all provider-specific decisions -/// (auth header style, default headers, passthrough headers, base URL) are made by the -/// caller during resolution. -#[derive(Clone)] -pub struct ResolvedRoute { - /// Route name used for identification (e.g. "inference.local", "sandbox-system"). - pub name: String, - pub endpoint: String, - pub model: String, - pub api_key: String, - pub protocols: Vec, - /// How to inject the API key on outgoing requests. - pub auth: AuthHeader, - /// Extra headers injected on every request (e.g. `anthropic-version`). - pub default_headers: Vec<(String, String)>, - /// Client-supplied headers that may be forwarded to the upstream backend. - pub passthrough_headers: Vec, - /// Per-request timeout for proxied inference calls. - pub timeout: Duration, - /// When true, the model identifier is embedded in the URL path (e.g. Vertex AI). - pub model_in_path: bool, - /// Optional override for the request path. When set, replaces the protocol-derived path. - /// An empty string means POST directly to `base_url/model_id` with no additional path. - pub request_path_override: Option, -} - -impl std::fmt::Debug for ResolvedRoute { - fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { - f.debug_struct("ResolvedRoute") - .field("name", &self.name) - .field("endpoint", &self.endpoint) - .field("model", &self.model) - .field("api_key", &"[REDACTED]") - .field("protocols", &self.protocols) - .field("auth", &self.auth) - .field("default_headers", &self.default_headers) - .field("passthrough_headers", &self.passthrough_headers) - .field("timeout", &self.timeout) - .field("model_in_path", &self.model_in_path) - .field("request_path_override", &self.request_path_override) - .finish() - } -} - -impl RouterConfig { - pub fn load_from_file(path: &Path) -> Result { - let content = std::fs::read_to_string(path).map_err(|e| { - RouterError::Internal(format!( - "failed to read router config {}: {e}", - path.display() - )) - })?; - let config: Self = serde_yml::from_str(&content).map_err(|e| { - RouterError::Internal(format!( - "failed to parse router config {}: {e}", - path.display() - )) - })?; - config.resolve() - } - - fn resolve(self) -> Result { - // Validate that all routes can resolve their API keys - for route in &self.routes { - route.resolve_api_key()?; - } - Ok(self) - } - - pub fn resolve_routes(&self) -> Result, RouterError> { - self.routes.iter().map(RouteConfig::resolve).collect() - } -} - -impl RouteConfig { - fn resolve_api_key(&self) -> Result { - if let Some(key) = &self.api_key { - return Ok(key.clone()); - } - if let Some(env_var) = &self.api_key_env { - return std::env::var(env_var).map_err(|_| { - RouterError::Internal(format!( - "environment variable {env_var} not set for route '{}'", - self.name - )) - }); - } - Err(RouterError::Internal(format!( - "route '{}' has neither api_key nor api_key_env", - self.name - ))) - } - - fn resolve(&self) -> Result { - let protocols = openshell_core::inference::normalize_protocols(&self.protocols); - if protocols.is_empty() { - return Err(RouterError::Internal(format!( - "route '{}' has no protocols", - self.name - ))); - } - - let (auth, default_headers, passthrough_headers) = - route_headers_from_provider_type(self.provider_type.as_deref(), &protocols); - - Ok(ResolvedRoute { - name: self.name.clone(), - endpoint: self.endpoint.clone(), - model: self.model.clone(), - api_key: self.resolve_api_key()?, - protocols, - auth, - default_headers, - passthrough_headers, - timeout: DEFAULT_ROUTE_TIMEOUT, - model_in_path: false, - request_path_override: None, - }) - } -} - -/// Derive auth header style, default headers, and passthrough headers from a -/// provider type string. -/// -/// Delegates to [`openshell_core::inference::route_headers_for_provider_type`] -/// which uses the centralized `InferenceProviderProfile` registry. -fn route_headers_from_provider_type( - provider_type: Option<&str>, - protocols: &[String], -) -> (AuthHeader, Vec<(String, String)>, Vec) { - openshell_core::inference::route_headers_for_route(provider_type.unwrap_or(""), protocols) -} - -#[cfg(test)] -mod tests { - use super::*; - use std::io::Write; - - #[test] - fn load_from_file_valid_yaml_round_trip() { - let yaml = r" -routes: - - name: inference.local - endpoint: http://localhost:8000/v1 - model: llama-3 - protocols: [openai_chat_completions] - api_key: sk-test-key - - name: inference.local - endpoint: https://api.openai.com/v1 - model: gpt-4o - protocols: [openai_chat_completions, anthropic_messages] - api_key: sk-prod-key -"; - let mut f = tempfile::NamedTempFile::new().unwrap(); - f.write_all(yaml.as_bytes()).unwrap(); - - let config = RouterConfig::load_from_file(f.path()).unwrap(); - assert_eq!(config.routes.len(), 2); - assert_eq!(config.routes[0].name, "inference.local"); - assert_eq!(config.routes[1].name, "inference.local"); - - let resolved = config.resolve_routes().unwrap(); - assert_eq!(resolved.len(), 2); - assert_eq!(resolved[0].api_key, "sk-test-key"); - assert_eq!(resolved[1].model, "gpt-4o"); - assert_eq!( - resolved[1].protocols, - vec!["openai_chat_completions", "anthropic_messages"] - ); - } - - #[test] - fn load_from_file_invalid_yaml_returns_error() { - let mut f = tempfile::NamedTempFile::new().unwrap(); - f.write_all(b"not: [valid: yaml: {{{{").unwrap(); - - let err = RouterConfig::load_from_file(f.path()).unwrap_err(); - assert!( - matches!(err, RouterError::Internal(_)), - "expected Internal error, got: {err:?}" - ); - } - - #[test] - fn load_from_file_missing_api_key_returns_error() { - let yaml = r" -routes: - - name: inference.local - endpoint: http://localhost:8000/v1 - model: llama-3 - protocols: [openai_chat_completions] -"; - let mut f = tempfile::NamedTempFile::new().unwrap(); - f.write_all(yaml.as_bytes()).unwrap(); - - let err = RouterConfig::load_from_file(f.path()).unwrap_err(); - assert!( - matches!(err, RouterError::Internal(_)), - "expected Internal error for missing api_key, got: {err:?}" - ); - let msg = err.to_string(); - assert!( - msg.contains("neither api_key nor api_key_env"), - "error should mention missing key: {msg}" - ); - } - - #[test] - #[allow(unsafe_code)] // std::env::set_var/remove_var require unsafe in Rust 2024 - fn load_from_file_api_key_env_resolves_from_environment() { - let yaml = r" -routes: - - name: inference.local - endpoint: http://localhost:8000/v1 - model: llama-3 - protocols: [openai_chat_completions] - api_key_env: NAV_TEST_API_KEY_FOR_YAML_TEST -"; - // SAFETY: this test runs single-threaded; no other thread reads this var. - unsafe { std::env::set_var("NAV_TEST_API_KEY_FOR_YAML_TEST", "from-env") }; - let mut f = tempfile::NamedTempFile::new().unwrap(); - f.write_all(yaml.as_bytes()).unwrap(); - - let config = RouterConfig::load_from_file(f.path()).unwrap(); - let resolved = config.resolve_routes().unwrap(); - assert_eq!(resolved[0].api_key, "from-env"); - - unsafe { std::env::remove_var("NAV_TEST_API_KEY_FOR_YAML_TEST") }; - } - - #[test] - fn load_from_file_nonexistent_path_returns_error() { - let err = RouterConfig::load_from_file(Path::new("/nonexistent/routes.yaml")).unwrap_err(); - assert!( - matches!(err, RouterError::Internal(_)), - "expected Internal error, got: {err:?}" - ); - } - - #[test] - fn resolved_route_debug_redacts_api_key() { - let route = ResolvedRoute { - name: "test".to_string(), - endpoint: "https://api.example.com/v1".to_string(), - model: "test-model".to_string(), - api_key: "sk-super-secret-key-12345".to_string(), - protocols: vec!["openai_chat_completions".to_string()], - auth: AuthHeader::Bearer, - default_headers: Vec::new(), - passthrough_headers: Vec::new(), - timeout: DEFAULT_ROUTE_TIMEOUT, - model_in_path: false, - request_path_override: None, - }; - let debug_output = format!("{route:?}"); - assert!( - !debug_output.contains("sk-super-secret-key-12345"), - "Debug output must not contain raw API key: {debug_output}" - ); - assert!( - debug_output.contains("[REDACTED]"), - "Debug output should show [REDACTED] for api_key: {debug_output}" - ); - } - - #[test] - fn auth_from_anthropic_provider_uses_custom_header() { - let (auth, headers, passthrough_headers) = route_headers_from_provider_type( - Some("anthropic"), - &["anthropic_messages".to_string()], - ); - assert_eq!(auth, AuthHeader::Custom("x-api-key")); - assert!(headers.iter().any(|(k, _)| k == "anthropic-version")); - assert!( - passthrough_headers - .iter() - .any(|name| name == "anthropic-beta") - ); - } - - #[test] - fn auth_from_openai_provider_uses_bearer() { - let (auth, headers, passthrough_headers) = route_headers_from_provider_type( - Some("openai"), - &["openai_chat_completions".to_string()], - ); - assert_eq!(auth, AuthHeader::Bearer); - assert!(headers.is_empty()); - assert!( - passthrough_headers - .iter() - .any(|name| name == "openai-organization") - ); - } - - #[test] - fn auth_from_none_defaults_to_bearer() { - let (auth, headers, passthrough_headers) = route_headers_from_provider_type(None, &[]); - assert_eq!(auth, AuthHeader::Bearer); - assert!(headers.is_empty()); - assert!(passthrough_headers.is_empty()); - } - - #[test] - fn vertex_anthropic_route_forwards_beta_only() { - let (_, headers, passthrough_headers) = route_headers_from_provider_type( - Some("google-vertex-ai"), - &["anthropic_messages".to_string()], - ); - assert!(headers.is_empty()); - assert_eq!(passthrough_headers, vec!["anthropic-beta".to_string()]); - } -} diff --git a/crates/openshell-router/src/lib.rs b/crates/openshell-router/src/lib.rs deleted file mode 100644 index 79bbfe6ca..000000000 --- a/crates/openshell-router/src/lib.rs +++ /dev/null @@ -1,189 +0,0 @@ -// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -// SPDX-License-Identifier: Apache-2.0 - -mod backend; -pub mod config; -mod mock; - -pub use backend::{ - ProxyResponse, StreamingProxyResponse, ValidatedEndpoint, ValidationFailure, - ValidationFailureKind, verify_backend_endpoint, -}; -use config::{ResolvedRoute, RouterConfig}; -use std::time::Duration; -use tracing::info; - -#[derive(Debug, thiserror::Error)] -pub enum RouterError { - #[error("route not found for route '{0}'")] - RouteNotFound(String), - #[error("no compatible route for protocol '{0}'")] - NoCompatibleRoute(String), - #[error("unauthorized: {0}")] - Unauthorized(String), - #[error("upstream unavailable: {0}")] - UpstreamUnavailable(String), - #[error("upstream protocol error: {0}")] - UpstreamProtocol(String), - #[error("internal error: {0}")] - Internal(String), -} - -#[derive(Debug)] -pub struct Router { - routes: Vec, - client: reqwest::Client, -} - -impl Router { - pub fn new() -> Result { - let client = reqwest::Client::builder() - .connect_timeout(Duration::from_secs(30)) - .build() - .map_err(|e| RouterError::Internal(format!("failed to build HTTP client: {e}")))?; - Ok(Self { - routes: Vec::new(), - client, - }) - } - - pub fn from_config(config: &RouterConfig) -> Result { - let resolved = config.resolve_routes()?; - let mut router = Self::new()?; - router.routes = resolved; - Ok(router) - } - - /// Proxy a raw HTTP request to the first compatible route from `candidates`. - /// - /// Filters candidates by `source_protocol` compatibility (exact match against - /// one of the route's `protocols`), then forwards to the first match. - pub async fn proxy_with_candidates( - &self, - source_protocol: &str, - method: &str, - path: &str, - headers: Vec<(String, String)>, - body: bytes::Bytes, - candidates: &[ResolvedRoute], - ) -> Result { - let normalized_source = source_protocol.trim().to_ascii_lowercase(); - let route = candidates - .iter() - .find(|r| r.protocols.iter().any(|p| p == &normalized_source)) - .ok_or_else(|| RouterError::NoCompatibleRoute(source_protocol.to_string()))?; - - info!( - protocols = %route.protocols.join(","), - endpoint = %route.endpoint, - method = %method, - path = %path, - "routing proxy inference request" - ); - - if mock::is_mock_route(route) { - info!(endpoint = %route.endpoint, "returning mock response"); - return Ok(mock::mock_response(route, &normalized_source)); - } - - backend::proxy_to_backend( - &self.client, - route, - &normalized_source, - method, - path, - headers, - body, - ) - .await - } - - /// Streaming variant of [`proxy_with_candidates`](Self::proxy_with_candidates). - /// - /// Returns response headers immediately without buffering the body. - /// The caller streams body chunks via [`StreamingProxyResponse::response`]. - pub async fn proxy_with_candidates_streaming( - &self, - source_protocol: &str, - method: &str, - path: &str, - headers: Vec<(String, String)>, - body: bytes::Bytes, - candidates: &[ResolvedRoute], - ) -> Result { - let normalized_source = source_protocol.trim().to_ascii_lowercase(); - let route = candidates - .iter() - .find(|r| r.protocols.iter().any(|p| p == &normalized_source)) - .ok_or_else(|| RouterError::NoCompatibleRoute(source_protocol.to_string()))?; - - info!( - protocols = %route.protocols.join(","), - endpoint = %route.endpoint, - method = %method, - path = %path, - "routing proxy inference request (streaming)" - ); - - if mock::is_mock_route(route) { - info!(endpoint = %route.endpoint, "returning mock response (buffered)"); - let buffered = mock::mock_response(route, &normalized_source); - return Ok(StreamingProxyResponse::from_buffered(buffered)); - } - - backend::proxy_to_backend_streaming( - &self.client, - route, - &normalized_source, - method, - path, - headers, - body, - ) - .await - } -} - -#[cfg(test)] -mod tests { - use super::*; - use config::{RouteConfig, RouterConfig}; - - fn test_config() -> RouterConfig { - RouterConfig { - routes: vec![RouteConfig { - name: "inference.local".to_string(), - endpoint: "http://localhost:8000/v1".to_string(), - model: "meta/llama-3.1-8b-instruct".to_string(), - provider_type: None, - protocols: vec!["openai_chat_completions".to_string()], - api_key: Some("test-key".to_string()), - api_key_env: None, - }], - } - } - - #[test] - fn router_resolves_routes_from_config() { - let router = Router::from_config(&test_config()).unwrap(); - assert_eq!(router.routes.len(), 1); - assert_eq!(router.routes[0].protocols, vec!["openai_chat_completions"]); - } - - #[test] - fn config_missing_api_key_returns_error() { - let config = RouterConfig { - routes: vec![RouteConfig { - name: "inference.local".to_string(), - endpoint: "http://localhost".to_string(), - model: "test-model".to_string(), - provider_type: None, - protocols: vec!["openai_chat_completions".to_string()], - api_key: None, - api_key_env: None, - }], - }; - let err = Router::from_config(&config).unwrap_err(); - assert!(matches!(err, RouterError::Internal(_))); - } -} diff --git a/crates/openshell-router/src/mock.rs b/crates/openshell-router/src/mock.rs deleted file mode 100644 index 946839bae..000000000 --- a/crates/openshell-router/src/mock.rs +++ /dev/null @@ -1,263 +0,0 @@ -// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -// SPDX-License-Identifier: Apache-2.0 - -use crate::backend::ProxyResponse; -use crate::config::ResolvedRoute; - -const MOCK_SCHEME: &str = "mock://"; - -/// Returns true if the route's endpoint uses the `mock://` scheme, -/// indicating the router should return a canned response. -pub fn is_mock_route(route: &ResolvedRoute) -> bool { - route.endpoint.starts_with(MOCK_SCHEME) -} - -/// Generate a canned HTTP response appropriate for the route's protocol. -/// -/// The response is protocol-aware: for `openai_chat_completions` it returns -/// a valid `OpenAI` chat completion JSON, for `anthropic_messages` a valid -/// Anthropic response, etc. The route's `model` field is echoed in the response. -pub fn mock_response(route: &ResolvedRoute, source_protocol: &str) -> ProxyResponse { - tracing::warn!( - endpoint = %route.endpoint, - "Serving mock response — mock:// routes should only be used in development/testing" - ); - - let protocol = if route.protocols.iter().any(|p| p == source_protocol) { - source_protocol - } else { - route.protocols.first().map_or("", String::as_str) - }; - - let body = match protocol { - "openai_chat_completions" => openai_chat_completion_body(&route.model), - "openai_completions" => openai_completion_body(&route.model), - "openai_embeddings" => openai_embeddings_body(&route.model), - "anthropic_messages" => anthropic_messages_body(&route.model), - _ => generic_body(&route.model), - }; - - let body_bytes = bytes::Bytes::from(body); - ProxyResponse { - status: 200, - headers: vec![ - ("content-type".to_string(), "application/json".to_string()), - ("x-openshell-mock".to_string(), "true".to_string()), - ], - body: body_bytes, - route_model: Some(route.model.clone()), - route_endpoint: Some(route.endpoint.clone()), - } -} - -fn openai_chat_completion_body(model: &str) -> Vec { - serde_json::to_vec(&serde_json::json!({ - "id": "mock-chatcmpl-001", - "object": "chat.completion", - "created": 1_700_000_000_i64, - "model": model, - "choices": [{ - "index": 0, - "message": { - "role": "assistant", - "content": "Hello from openshell mock backend" - }, - "finish_reason": "stop" - }], - "usage": { - "prompt_tokens": 1, - "completion_tokens": 5, - "total_tokens": 6 - } - })) - .expect("static JSON must serialize") -} - -fn openai_completion_body(model: &str) -> Vec { - serde_json::to_vec(&serde_json::json!({ - "id": "mock-cmpl-001", - "object": "text_completion", - "created": 1_700_000_000_i64, - "model": model, - "choices": [{ - "index": 0, - "text": "Hello from openshell mock backend", - "finish_reason": "stop" - }], - "usage": { - "prompt_tokens": 1, - "completion_tokens": 5, - "total_tokens": 6 - } - })) - .expect("static JSON must serialize") -} - -fn openai_embeddings_body(model: &str) -> Vec { - // Shape must match the OpenAI embeddings response (`object: "list"` with a - // `data` array of `{object, index, embedding}`) so callers that deserialize - // into an embeddings type get a structurally valid — if canned — vector. - serde_json::to_vec(&serde_json::json!({ - "object": "list", - "data": [{ - "object": "embedding", - "index": 0, - "embedding": [0.0_f32, 0.0_f32, 0.0_f32] - }], - "model": model, - "usage": { - "prompt_tokens": 1, - "total_tokens": 1 - } - })) - .expect("static JSON must serialize") -} - -fn anthropic_messages_body(model: &str) -> Vec { - serde_json::to_vec(&serde_json::json!({ - "id": "mock-msg-001", - "type": "message", - "role": "assistant", - "model": model, - "content": [{ - "type": "text", - "text": "Hello from openshell mock backend" - }], - "stop_reason": "end_turn", - "usage": { - "input_tokens": 1, - "output_tokens": 5 - } - })) - .expect("static JSON must serialize") -} - -fn generic_body(model: &str) -> Vec { - serde_json::to_vec(&serde_json::json!({ - "mock": true, - "model": model, - "message": "Hello from openshell mock backend" - })) - .expect("static JSON must serialize") -} - -#[cfg(test)] -mod tests { - use super::*; - - fn make_route(endpoint: &str, protocols: &[&str], model: &str) -> ResolvedRoute { - ResolvedRoute { - name: "test".to_string(), - endpoint: endpoint.to_string(), - model: model.to_string(), - api_key: "key".to_string(), - protocols: protocols.iter().map(ToString::to_string).collect(), - auth: crate::config::AuthHeader::Bearer, - default_headers: Vec::new(), - passthrough_headers: Vec::new(), - timeout: crate::config::DEFAULT_ROUTE_TIMEOUT, - model_in_path: false, - request_path_override: None, - } - } - - #[test] - fn is_mock_route_detects_mock_scheme() { - assert!(is_mock_route(&make_route( - "mock://test", - &["openai_chat_completions"], - "m" - ))); - assert!(is_mock_route(&make_route( - "mock://", - &["openai_chat_completions"], - "m" - ))); - } - - #[test] - fn is_mock_route_rejects_real_urls() { - assert!(!is_mock_route(&make_route( - "https://api.openai.com", - &["openai_chat_completions"], - "m" - ))); - assert!(!is_mock_route(&make_route( - "http://localhost:8000", - &["openai_chat_completions"], - "m" - ))); - } - - #[test] - fn mock_openai_chat_completion() { - let route = make_route("mock://test", &["openai_chat_completions"], "gpt-4"); - let resp = mock_response(&route, "openai_chat_completions"); - assert_eq!(resp.status, 200); - - let body: serde_json::Value = serde_json::from_slice(&resp.body).unwrap(); - assert_eq!(body["model"], "gpt-4"); - assert_eq!(body["object"], "chat.completion"); - assert_eq!( - body["choices"][0]["message"]["content"], - "Hello from openshell mock backend" - ); - } - - #[test] - fn mock_anthropic_messages() { - let route = make_route("mock://test", &["anthropic_messages"], "claude-3"); - let resp = mock_response(&route, "anthropic_messages"); - assert_eq!(resp.status, 200); - - let body: serde_json::Value = serde_json::from_slice(&resp.body).unwrap(); - assert_eq!(body["model"], "claude-3"); - assert_eq!(body["type"], "message"); - assert_eq!( - body["content"][0]["text"], - "Hello from openshell mock backend" - ); - } - - #[test] - fn mock_openai_embeddings() { - let route = make_route( - "mock://test", - &["openai_embeddings"], - "text-embedding-3-small", - ); - let resp = mock_response(&route, "openai_embeddings"); - assert_eq!(resp.status, 200); - - let body: serde_json::Value = serde_json::from_slice(&resp.body).unwrap(); - assert_eq!(body["object"], "list"); - assert_eq!(body["model"], "text-embedding-3-small"); - assert_eq!(body["data"][0]["object"], "embedding"); - assert!( - body["data"][0]["embedding"].is_array(), - "embedding must be a numeric array, got: {body}" - ); - } - - #[test] - fn mock_generic_protocol() { - let route = make_route("mock://test", &["unknown_protocol"], "some-model"); - let resp = mock_response(&route, "unknown_protocol"); - assert_eq!(resp.status, 200); - - let body: serde_json::Value = serde_json::from_slice(&resp.body).unwrap(); - assert_eq!(body["mock"], true); - assert_eq!(body["model"], "some-model"); - } - - #[test] - fn mock_response_includes_marker_header() { - let route = make_route("mock://test", &["openai_chat_completions"], "m"); - let resp = mock_response(&route, "openai_chat_completions"); - assert!( - resp.headers - .iter() - .any(|(k, v)| k == "x-openshell-mock" && v == "true") - ); - } -} diff --git a/crates/openshell-router/tests/backend_integration.rs b/crates/openshell-router/tests/backend_integration.rs deleted file mode 100644 index fe5c2dd0b..000000000 --- a/crates/openshell-router/tests/backend_integration.rs +++ /dev/null @@ -1,994 +0,0 @@ -// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -// SPDX-License-Identifier: Apache-2.0 - -use openshell_router::Router; -use openshell_router::config::{AuthHeader, ResolvedRoute, RouteConfig, RouterConfig}; -use wiremock::matchers::{bearer_token, body_partial_json, header, method, path}; -use wiremock::{Match, Mock, MockServer, Request, ResponseTemplate}; - -fn mock_candidates(base_url: &str) -> Vec { - vec![ResolvedRoute { - name: "inference.local".to_string(), - endpoint: base_url.to_string(), - model: "meta/llama-3.1-8b-instruct".to_string(), - api_key: "test-api-key".to_string(), - protocols: vec!["openai_chat_completions".to_string()], - auth: AuthHeader::Bearer, - default_headers: Vec::new(), - passthrough_headers: vec!["openai-organization".to_string(), "x-model-id".to_string()], - timeout: openshell_router::config::DEFAULT_ROUTE_TIMEOUT, - model_in_path: false, - request_path_override: None, - }] -} - -#[tokio::test] -async fn proxy_forwards_request_to_backend() { - let mock_server = MockServer::start().await; - - let response_body = serde_json::json!({ - "id": "chatcmpl-123", - "object": "chat.completion", - "created": 1_700_000_000_i64, - "model": "meta/llama-3.1-8b-instruct", - "choices": [{ - "index": 0, - "message": { - "role": "assistant", - "content": "Hello! How can I help you?" - }, - "finish_reason": "stop" - }], - "usage": { - "prompt_tokens": 10, - "completion_tokens": 8, - "total_tokens": 18 - } - }); - - Mock::given(method("POST")) - .and(path("/v1/chat/completions")) - .and(bearer_token("test-api-key")) - .respond_with(ResponseTemplate::new(200).set_body_json(&response_body)) - .mount(&mock_server) - .await; - - let router = Router::new().unwrap(); - let candidates = mock_candidates(&mock_server.uri()); - - let body = serde_json::to_vec(&serde_json::json!({ - "model": "test", - "messages": [{"role": "user", "content": "Hello"}] - })) - .unwrap(); - - let response = router - .proxy_with_candidates( - "openai_chat_completions", - "POST", - "/v1/chat/completions", - vec![("content-type".to_string(), "application/json".to_string())], - bytes::Bytes::from(body), - &candidates, - ) - .await - .unwrap(); - - assert_eq!(response.status, 200); - let resp_body: serde_json::Value = serde_json::from_slice(&response.body).unwrap(); - assert_eq!(resp_body["id"], "chatcmpl-123"); -} - -#[tokio::test] -async fn proxy_upstream_401_returns_error() { - let mock_server = MockServer::start().await; - - Mock::given(method("POST")) - .and(path("/v1/chat/completions")) - .respond_with(ResponseTemplate::new(401).set_body_json(serde_json::json!({ - "error": { "message": "Invalid API key" } - }))) - .mount(&mock_server) - .await; - - let router = Router::new().unwrap(); - let candidates = mock_candidates(&mock_server.uri()); - - let response = router - .proxy_with_candidates( - "openai_chat_completions", - "POST", - "/v1/chat/completions", - vec![], - bytes::Bytes::new(), - &candidates, - ) - .await - .unwrap(); - - // Raw proxy returns the actual HTTP status, not a RouterError - assert_eq!(response.status, 401); -} - -#[tokio::test] -async fn proxy_no_compatible_route_returns_error() { - let router = Router::new().unwrap(); - let candidates = vec![ResolvedRoute { - name: "inference.local".to_string(), - endpoint: "http://localhost:1234".to_string(), - model: "test".to_string(), - api_key: "key".to_string(), - protocols: vec!["anthropic_messages".to_string()], - auth: AuthHeader::Custom("x-api-key"), - default_headers: Vec::new(), - passthrough_headers: Vec::new(), - timeout: openshell_router::config::DEFAULT_ROUTE_TIMEOUT, - model_in_path: false, - request_path_override: None, - }]; - - let err = router - .proxy_with_candidates( - "openai_chat_completions", - "POST", - "/v1/chat/completions", - vec![], - bytes::Bytes::new(), - &candidates, - ) - .await - .unwrap_err(); - - assert!( - matches!(err, openshell_router::RouterError::NoCompatibleRoute(_)), - "expected NoCompatibleRoute, got: {err:?}" - ); -} - -#[tokio::test] -async fn proxy_strips_auth_header() { - let mock_server = MockServer::start().await; - - Mock::given(method("POST")) - .and(path("/v1/chat/completions")) - .and(bearer_token("test-api-key")) - .respond_with(ResponseTemplate::new(200).set_body_string("{}")) - .mount(&mock_server) - .await; - - let router = Router::new().unwrap(); - let candidates = mock_candidates(&mock_server.uri()); - - // Client sends its own Authorization header — should be stripped and replaced - let response = router - .proxy_with_candidates( - "openai_chat_completions", - "POST", - "/v1/chat/completions", - vec![("authorization".to_string(), "Bearer client-key".to_string())], - bytes::Bytes::new(), - &candidates, - ) - .await - .unwrap(); - - assert_eq!(response.status, 200); -} - -#[tokio::test] -async fn proxy_forwards_openai_organization_header() { - let mock_server = MockServer::start().await; - - Mock::given(method("POST")) - .and(path("/v1/chat/completions")) - .and(bearer_token("test-api-key")) - .and(header("openai-organization", "org_123")) - .respond_with(ResponseTemplate::new(200).set_body_string("{}")) - .mount(&mock_server) - .await; - - let router = Router::new().unwrap(); - let candidates = mock_candidates(&mock_server.uri()); - - let response = router - .proxy_with_candidates( - "openai_chat_completions", - "POST", - "/v1/chat/completions", - vec![ - ("openai-organization".to_string(), "org_123".to_string()), - ("cookie".to_string(), "session=abc".to_string()), - ], - bytes::Bytes::new(), - &candidates, - ) - .await - .unwrap(); - - assert_eq!(response.status, 200); -} - -#[tokio::test] -async fn proxy_mock_route_returns_canned_response() { - let router = Router::new().unwrap(); - let candidates = vec![ResolvedRoute { - name: "inference.local".to_string(), - endpoint: "mock://test".to_string(), - model: "mock/test-model".to_string(), - api_key: "unused".to_string(), - protocols: vec!["openai_chat_completions".to_string()], - auth: AuthHeader::Bearer, - default_headers: Vec::new(), - passthrough_headers: Vec::new(), - timeout: openshell_router::config::DEFAULT_ROUTE_TIMEOUT, - model_in_path: false, - request_path_override: None, - }]; - - let body = serde_json::to_vec(&serde_json::json!({ - "model": "mock/test-model", - "messages": [{"role": "user", "content": "hello"}] - })) - .unwrap(); - - let response = router - .proxy_with_candidates( - "openai_chat_completions", - "POST", - "/v1/chat/completions", - vec![("content-type".to_string(), "application/json".to_string())], - bytes::Bytes::from(body), - &candidates, - ) - .await - .unwrap(); - - assert_eq!(response.status, 200); - let resp_body: serde_json::Value = serde_json::from_slice(&response.body).unwrap(); - assert_eq!(resp_body["model"], "mock/test-model"); - assert_eq!( - resp_body["choices"][0]["message"]["content"], - "Hello from openshell mock backend" - ); -} - -#[tokio::test] -async fn proxy_overrides_model_in_request_body() { - let mock_server = MockServer::start().await; - - // The mock expects the route's model, NOT the client's original model - Mock::given(method("POST")) - .and(path("/v1/chat/completions")) - .and(body_partial_json(serde_json::json!({ - "model": "meta/llama-3.1-8b-instruct" - }))) - .respond_with(ResponseTemplate::new(200).set_body_string("{}")) - .mount(&mock_server) - .await; - - let router = Router::new().unwrap(); - let candidates = mock_candidates(&mock_server.uri()); - - // Client sends "gpt-4o-mini" but route is configured with "meta/llama-3.1-8b-instruct" - let body = serde_json::to_vec(&serde_json::json!({ - "model": "gpt-4o-mini", - "messages": [{"role": "user", "content": "Hello"}] - })) - .unwrap(); - - let response = router - .proxy_with_candidates( - "openai_chat_completions", - "POST", - "/v1/chat/completions", - vec![("content-type".to_string(), "application/json".to_string())], - bytes::Bytes::from(body), - &candidates, - ) - .await - .unwrap(); - - assert_eq!(response.status, 200); -} - -#[tokio::test] -async fn proxy_inserts_model_when_absent_from_body() { - let mock_server = MockServer::start().await; - - // The mock expects the route's model to be inserted even though the client didn't send one - Mock::given(method("POST")) - .and(path("/v1/chat/completions")) - .and(body_partial_json(serde_json::json!({ - "model": "meta/llama-3.1-8b-instruct" - }))) - .respond_with(ResponseTemplate::new(200).set_body_string("{}")) - .mount(&mock_server) - .await; - - let router = Router::new().unwrap(); - let candidates = mock_candidates(&mock_server.uri()); - - // Client omits "model" entirely - let body = serde_json::to_vec(&serde_json::json!({ - "messages": [{"role": "user", "content": "Hello"}] - })) - .unwrap(); - - let response = router - .proxy_with_candidates( - "openai_chat_completions", - "POST", - "/v1/chat/completions", - vec![("content-type".to_string(), "application/json".to_string())], - bytes::Bytes::from(body), - &candidates, - ) - .await - .unwrap(); - - assert_eq!(response.status, 200); -} - -#[tokio::test] -async fn proxy_uses_x_api_key_for_anthropic_route() { - let mock_server = MockServer::start().await; - - Mock::given(method("POST")) - .and(path("/v1/messages")) - .and(header("x-api-key", "test-anthropic-key")) - .respond_with(ResponseTemplate::new(200).set_body_json(serde_json::json!({ - "id": "msg_123", - "type": "message", - "role": "assistant", - "content": [{"type": "text", "text": "Hello"}], - "model": "claude-sonnet-4-20250514", - "stop_reason": "end_turn", - "usage": {"input_tokens": 10, "output_tokens": 5} - }))) - .mount(&mock_server) - .await; - - let router = Router::new().unwrap(); - let candidates = vec![ResolvedRoute { - name: "inference.local".to_string(), - endpoint: mock_server.uri(), - model: "claude-sonnet-4-20250514".to_string(), - api_key: "test-anthropic-key".to_string(), - protocols: vec!["anthropic_messages".to_string()], - auth: AuthHeader::Custom("x-api-key"), - default_headers: vec![("anthropic-version".to_string(), "2023-06-01".to_string())], - passthrough_headers: vec![ - "anthropic-version".to_string(), - "anthropic-beta".to_string(), - ], - timeout: openshell_router::config::DEFAULT_ROUTE_TIMEOUT, - model_in_path: false, - request_path_override: None, - }]; - - let body = serde_json::to_vec(&serde_json::json!({ - "model": "claude-sonnet-4-20250514", - "max_tokens": 1, - "messages": [{"role": "user", "content": "hi"}] - })) - .unwrap(); - - let response = router - .proxy_with_candidates( - "anthropic_messages", - "POST", - "/v1/messages", - vec![ - ("content-type".to_string(), "application/json".to_string()), - ("anthropic-version".to_string(), "2023-06-01".to_string()), - ], - bytes::Bytes::from(body), - &candidates, - ) - .await - .unwrap(); - - assert_eq!(response.status, 200); - let resp_body: serde_json::Value = serde_json::from_slice(&response.body).unwrap(); - assert_eq!(resp_body["type"], "message"); -} - -#[tokio::test] -async fn proxy_anthropic_does_not_send_bearer_auth() { - let mock_server = MockServer::start().await; - - // This mock rejects requests that have a Bearer token — ensuring we DON'T send one - Mock::given(method("POST")) - .and(path("/v1/messages")) - .and(header("x-api-key", "anthropic-key")) - .respond_with(ResponseTemplate::new(200).set_body_string("{}")) - .mount(&mock_server) - .await; - - // Also mount a catch-all that returns 401 if Bearer is used - Mock::given(method("POST")) - .and(path("/v1/messages")) - .and(bearer_token("anthropic-key")) - .respond_with(ResponseTemplate::new(401).set_body_string("should not use bearer")) - .mount(&mock_server) - .await; - - let router = Router::new().unwrap(); - let candidates = vec![ResolvedRoute { - name: "inference.local".to_string(), - endpoint: mock_server.uri(), - model: "claude-sonnet-4-20250514".to_string(), - api_key: "anthropic-key".to_string(), - protocols: vec!["anthropic_messages".to_string()], - auth: AuthHeader::Custom("x-api-key"), - default_headers: vec![("anthropic-version".to_string(), "2023-06-01".to_string())], - passthrough_headers: vec![ - "anthropic-version".to_string(), - "anthropic-beta".to_string(), - ], - timeout: openshell_router::config::DEFAULT_ROUTE_TIMEOUT, - model_in_path: false, - request_path_override: None, - }]; - - let response = router - .proxy_with_candidates( - "anthropic_messages", - "POST", - "/v1/messages", - vec![("content-type".to_string(), "application/json".to_string())], - bytes::Bytes::from(b"{}".to_vec()), - &candidates, - ) - .await - .unwrap(); - - assert_eq!(response.status, 200); -} - -/// Regression test: when the client sends `anthropic-version`, the header must -/// reach the upstream. Previously, the header was added to the strip list -/// (because it appeared in `default_headers`) AND the default injection was -/// skipped (because `already_sent` checked the *original* input), so neither -/// the client's value nor the default reached the backend. -#[tokio::test] -async fn proxy_forwards_client_anthropic_version_header() { - let mock_server = MockServer::start().await; - - // The upstream requires anthropic-version — wiremock will reject if missing. - Mock::given(method("POST")) - .and(path("/v1/messages")) - .and(header("x-api-key", "test-anthropic-key")) - .and(header("anthropic-version", "2024-10-22")) - .respond_with(ResponseTemplate::new(200).set_body_string("{}")) - .mount(&mock_server) - .await; - - let router = Router::new().unwrap(); - let candidates = vec![ResolvedRoute { - name: "inference.local".to_string(), - endpoint: mock_server.uri(), - model: "claude-sonnet-4-20250514".to_string(), - api_key: "test-anthropic-key".to_string(), - protocols: vec!["anthropic_messages".to_string()], - auth: AuthHeader::Custom("x-api-key"), - default_headers: vec![("anthropic-version".to_string(), "2023-06-01".to_string())], - passthrough_headers: vec![ - "anthropic-version".to_string(), - "anthropic-beta".to_string(), - ], - timeout: openshell_router::config::DEFAULT_ROUTE_TIMEOUT, - model_in_path: false, - request_path_override: None, - }]; - - let body = serde_json::to_vec(&serde_json::json!({ - "model": "claude-sonnet-4-20250514", - "max_tokens": 1, - "messages": [{"role": "user", "content": "hi"}] - })) - .unwrap(); - - // Client explicitly sends anthropic-version: 2024-10-22 — this value should - // reach the upstream, NOT be silently dropped. - let response = router - .proxy_with_candidates( - "anthropic_messages", - "POST", - "/v1/messages", - vec![ - ("content-type".to_string(), "application/json".to_string()), - ("anthropic-version".to_string(), "2024-10-22".to_string()), - ], - bytes::Bytes::from(body), - &candidates, - ) - .await - .unwrap(); - - assert_eq!( - response.status, 200, - "upstream should have received anthropic-version header" - ); -} - -#[tokio::test] -async fn proxy_vertex_gemini_route_uses_chat_completions_override() { - let mock_server = MockServer::start().await; - - Mock::given(method("POST")) - .and(path( - "/v1beta1/projects/my-project/locations/us-central1/endpoints/openapi/chat/completions", - )) - .and(bearer_token("ya29.test-token")) - .and(body_partial_json(serde_json::json!({ - "model": "gemini-2.0-flash-001", - }))) - .respond_with(ResponseTemplate::new(200).set_body_json(serde_json::json!({ - "id": "chatcmpl-vertex", - "object": "chat.completion", - "created": 1_700_000_000_i64, - "model": "gemini-2.0-flash-001", - "choices": [{ - "index": 0, - "message": { "role": "assistant", "content": "pong" }, - "finish_reason": "stop" - }] - }))) - .mount(&mock_server) - .await; - - let router = Router::new().unwrap(); - let candidates = vec![ResolvedRoute { - name: "inference.local".to_string(), - endpoint: format!( - "{}/v1beta1/projects/my-project/locations/us-central1/endpoints/openapi", - mock_server.uri() - ), - model: "gemini-2.0-flash-001".to_string(), - api_key: "ya29.test-token".to_string(), - protocols: vec!["openai_chat_completions".to_string()], - auth: AuthHeader::Bearer, - default_headers: Vec::new(), - passthrough_headers: Vec::new(), - timeout: openshell_router::config::DEFAULT_ROUTE_TIMEOUT, - model_in_path: false, - request_path_override: Some("/chat/completions".to_string()), - }]; - - let body = serde_json::to_vec(&serde_json::json!({ - "model": "client-model", - "messages": [{"role": "user", "content": "ping"}] - })) - .unwrap(); - - let response = router - .proxy_with_candidates( - "openai_chat_completions", - "POST", - "/v1/chat/completions", - vec![("content-type".to_string(), "application/json".to_string())], - bytes::Bytes::from(body), - &candidates, - ) - .await - .unwrap(); - - assert_eq!(response.status, 200); -} - -#[tokio::test] -async fn proxy_vertex_anthropic_route_uses_model_path_suffix() { - let mock_server = MockServer::start().await; - - Mock::given(method("POST")) - .and(path( - "/v1/projects/my-project/locations/us-east5/publishers/anthropic/models/claude-3-5-sonnet@20241022:rawPredict", - )) - .and(bearer_token("ya29.vertex-token")) - .and(body_partial_json(serde_json::json!({ - "anthropic_version": "vertex-2023-10-16", - }))) - .respond_with(ResponseTemplate::new(200).set_body_json(serde_json::json!({ - "id": "msg_vertex_1", - "type": "message", - "role": "assistant", - "model": "claude-3-5-sonnet@20241022", - "content": [{"type": "text", "text": "pong"}] - }))) - .mount(&mock_server) - .await; - - let router = Router::new().unwrap(); - let candidates = vec![ResolvedRoute { - name: "inference.local".to_string(), - endpoint: format!( - "{}/v1/projects/my-project/locations/us-east5/publishers/anthropic/models", - mock_server.uri() - ), - model: "claude-3-5-sonnet@20241022".to_string(), - api_key: "ya29.vertex-token".to_string(), - protocols: vec!["anthropic_messages".to_string()], - auth: AuthHeader::Bearer, - default_headers: Vec::new(), - passthrough_headers: vec!["anthropic-beta".to_string()], - timeout: openshell_router::config::DEFAULT_ROUTE_TIMEOUT, - model_in_path: true, - request_path_override: Some(":rawPredict".to_string()), - }]; - - // Include "model" in the body, as Claude Code and other Anthropic SDK - // clients always do. The router must strip it for Vertex AI rawPredict. - let body = serde_json::to_vec(&serde_json::json!({ - "model": "claude-3-5-sonnet-20241022", - "messages": [{"role": "user", "content": "ping"}], - "max_tokens": 32 - })) - .unwrap(); - - let response = router - .proxy_with_candidates( - "anthropic_messages", - "POST", - "/v1/messages", - vec![ - ("content-type".to_string(), "application/json".to_string()), - ("anthropic-beta".to_string(), "tools-2024-05-16".to_string()), - ("anthropic-version".to_string(), "2023-06-01".to_string()), - ], - bytes::Bytes::from(body), - &candidates, - ) - .await - .unwrap(); - - assert_eq!(response.status, 200); - let received = mock_server.received_requests().await.unwrap(); - assert_eq!(received.len(), 1); - let received_body: serde_json::Value = serde_json::from_slice(&received[0].body).unwrap(); - assert_eq!( - received_body["anthropic_version"], - serde_json::json!("vertex-2023-10-16") - ); - // "model" must be stripped: Vertex AI encodes the model in the URL path - // and rejects "model" in the body with "Extra inputs are not permitted". - assert!( - received_body.get("model").is_none(), - "Vertex Anthropic requests must not have model in the body, got: {received_body}" - ); - // anthropic-beta must be stripped: Vertex AI rejects unknown beta values - // with HTTP 400 (e.g. prompt-caching-scope-2026-01-05). - assert!( - !received[0].headers.contains_key("anthropic-beta"), - "anthropic-beta must not reach the Vertex AI backend" - ); - assert!( - !received[0].headers.contains_key("anthropic-version"), - "anthropic-version must be converted to body anthropic_version, not forwarded as a header" - ); -} - -/// Fields that Vertex AI rawPredict actually accepts for Anthropic models. -const VERTEX_ACCEPTED_FIELDS: &[&str] = &[ - "anthropic_version", - "messages", - "max_tokens", - "stop_sequences", - "stream", - "system", - "temperature", - "thinking", - "tool_choice", - "tools", - "top_k", - "top_p", - "metadata", -]; - -/// Simulates Vertex AI's strict pydantic validation: rejects any body field not -/// in the known Anthropic Messages API schema. This is what causes the real 400 -/// error described in #2444. -struct VertexStrictBodyValidator; - -impl Match for VertexStrictBodyValidator { - fn matches(&self, request: &Request) -> bool { - let Ok(body) = serde_json::from_slice::(&request.body) else { - return false; - }; - let Some(obj) = body.as_object() else { - return false; - }; - obj.keys() - .all(|k| VERTEX_ACCEPTED_FIELDS.contains(&k.as_str())) - } -} - -/// End-to-end reproduction of #2444: Claude Code sends `context_management` in -/// the body, the router must strip it before it reaches Vertex. The mock uses -/// strict body validation (like Vertex's pydantic) so the request only succeeds -/// if the field was actually removed. -#[tokio::test] -async fn proxy_vertex_strips_beta_fields_e2e() { - let mock_server = MockServer::start().await; - - let model_path = "/v1/projects/my-project/locations/us-east5/publishers/anthropic/models"; - let model = "claude-sonnet-4-6@20250514"; - - // This mock only matches if the body passes strict validation — - // context_management would cause a mismatch (simulating a real 400). - Mock::given(method("POST")) - .and(path(format!("{model_path}/{model}:rawPredict"))) - .and(VertexStrictBodyValidator) - .respond_with(ResponseTemplate::new(200).set_body_json(serde_json::json!({ - "id": "msg_1", - "type": "message", - "role": "assistant", - "model": model, - "content": [{"type": "text", "text": "Pong"}] - }))) - .mount(&mock_server) - .await; - - let router = Router::new().unwrap(); - let candidates = vec![ResolvedRoute { - name: "vertex-test".to_string(), - endpoint: format!("{}{model_path}", mock_server.uri()), - model: model.to_string(), - api_key: "ya29.token".to_string(), - protocols: vec!["anthropic_messages".to_string()], - auth: AuthHeader::Bearer, - default_headers: Vec::new(), - passthrough_headers: vec!["anthropic-beta".to_string()], - timeout: openshell_router::config::DEFAULT_ROUTE_TIMEOUT, - model_in_path: true, - request_path_override: Some(":rawPredict".to_string()), - }]; - - // Exact payload Claude Code v2.1.156+ sends: includes model, - // context_management, and anthropic-beta header. - let body = serde_json::to_vec(&serde_json::json!({ - "model": "claude-sonnet-4-6-20250514", - "messages": [{"role": "user", "content": "say pong"}], - "max_tokens": 128, - "context_management": {"enabled": true}, - })) - .unwrap(); - - let response = router - .proxy_with_candidates( - "anthropic_messages", - "POST", - "/v1/messages", - vec![ - ("content-type".to_string(), "application/json".to_string()), - ( - "anthropic-beta".to_string(), - "context-management-2025-06-27".to_string(), - ), - ], - bytes::Bytes::from(body), - &candidates, - ) - .await - .unwrap(); - - // If context_management leaked through, the strict mock wouldn't match - // and wiremock would return 404. - assert_eq!( - response.status, 200, - "request must succeed — context_management should have been stripped. \ - A 404 here means the strict body validator rejected an unsupported field." - ); -} - -#[tokio::test] -async fn proxy_vertex_anthropic_streaming_route_uses_stream_rawpredict() { - let mock_server = MockServer::start().await; - - Mock::given(method("POST")) - .and(path( - "/v1/projects/my-project/locations/us-east5/publishers/anthropic/models/claude-3-5-sonnet@20241022:streamRawPredict", - )) - .and(bearer_token("ya29.vertex-token")) - .respond_with(ResponseTemplate::new(200).set_body_string("{\"id\":\"msg_vertex_stream\"}")) - .mount(&mock_server) - .await; - - let router = Router::new().unwrap(); - let candidates = vec![ResolvedRoute { - name: "inference.local".to_string(), - endpoint: format!( - "{}/v1/projects/my-project/locations/us-east5/publishers/anthropic/models", - mock_server.uri() - ), - model: "claude-3-5-sonnet@20241022".to_string(), - api_key: "ya29.vertex-token".to_string(), - protocols: vec!["anthropic_messages".to_string()], - auth: AuthHeader::Bearer, - default_headers: Vec::new(), - passthrough_headers: vec!["anthropic-beta".to_string()], - timeout: openshell_router::config::DEFAULT_ROUTE_TIMEOUT, - model_in_path: true, - request_path_override: Some(":rawPredict".to_string()), - }]; - - let body = serde_json::to_vec(&serde_json::json!({ - "messages": [{"role": "user", "content": "ping"}], - "max_tokens": 32, - "stream": true - })) - .unwrap(); - - let mut response = router - .proxy_with_candidates_streaming( - "anthropic_messages", - "POST", - "/v1/messages", - vec![("content-type".to_string(), "application/json".to_string())], - bytes::Bytes::from(body), - &candidates, - ) - .await - .unwrap(); - - assert_eq!(response.status, 200); - let first_chunk = response.next_chunk().await.unwrap(); - assert!( - first_chunk.is_some(), - "streaming response should yield a body chunk" - ); -} - -#[test] -fn config_resolves_routes_with_protocol() { - let config = RouterConfig { - routes: vec![RouteConfig { - name: "inference.local".to_string(), - endpoint: "http://localhost:8000".to_string(), - model: "test-model".to_string(), - provider_type: None, - protocols: vec!["openai_chat_completions".to_string()], - api_key: Some("key".to_string()), - api_key_env: None, - }], - }; - let routes = config.resolve_routes().unwrap(); - assert_eq!(routes[0].protocols, vec!["openai_chat_completions"]); -} - -/// Streaming proxy must not apply a total request timeout to the body stream. -/// -/// The backend delays its response longer than the route timeout. With the old -/// code this would fail (reqwest's total `.timeout()` fires), but the streaming -/// path now omits that timeout — only the client-level `connect_timeout` and -/// the sandbox idle timeout govern liveness. -#[tokio::test] -async fn streaming_proxy_completes_despite_exceeding_route_timeout() { - use std::time::Duration; - - let mock_server = MockServer::start().await; - - let sse_body = concat!( - "data: {\"choices\":[{\"delta\":{\"content\":\"hello\"}}]}\n\n", - "data: {\"choices\":[{\"delta\":{\"content\":\" world\"}}]}\n\n", - "data: [DONE]\n\n", - ); - - // Delay the response 3s — longer than the 1s route timeout. - Mock::given(method("POST")) - .and(path("/v1/chat/completions")) - .and(bearer_token("test-api-key")) - .respond_with( - ResponseTemplate::new(200) - .append_header("content-type", "text/event-stream") - .set_body_string(sse_body) - .set_delay(Duration::from_secs(3)), - ) - .mount(&mock_server) - .await; - - let router = Router::new().unwrap(); - let candidates = vec![ResolvedRoute { - name: "inference.local".to_string(), - endpoint: mock_server.uri(), - model: "test-model".to_string(), - api_key: "test-api-key".to_string(), - protocols: vec!["openai_chat_completions".to_string()], - auth: AuthHeader::Bearer, - default_headers: Vec::new(), - passthrough_headers: Vec::new(), - // Route timeout shorter than the backend delay — streaming must - // NOT be constrained by this. - timeout: Duration::from_secs(1), - model_in_path: false, - request_path_override: None, - }]; - - let body = serde_json::to_vec(&serde_json::json!({ - "model": "test-model", - "messages": [{"role": "user", "content": "hi"}], - "stream": true - })) - .unwrap(); - - // The streaming path should succeed despite the 3s delay exceeding - // the 1s route timeout. - let mut resp = router - .proxy_with_candidates_streaming( - "openai_chat_completions", - "POST", - "/v1/chat/completions", - vec![("content-type".to_string(), "application/json".to_string())], - bytes::Bytes::from(body), - &candidates, - ) - .await - .expect("streaming proxy should not be killed by route timeout"); - - assert_eq!(resp.status, 200); - - // Drain all chunks to verify the full body is received. - let mut total_bytes = 0; - while let Ok(Some(chunk)) = resp.next_chunk().await { - total_bytes += chunk.len(); - } - assert!(total_bytes > 0, "should have received body chunks"); -} - -/// Non-streaming (buffered) proxy must still enforce the route timeout. -#[tokio::test] -async fn buffered_proxy_enforces_route_timeout() { - use std::time::Duration; - - let mock_server = MockServer::start().await; - - Mock::given(method("POST")) - .and(path("/v1/chat/completions")) - .respond_with( - ResponseTemplate::new(200) - .set_body_string("{}") - // Delay longer than the route timeout. - .set_delay(Duration::from_secs(5)), - ) - .mount(&mock_server) - .await; - - let router = Router::new().unwrap(); - let candidates = vec![ResolvedRoute { - name: "inference.local".to_string(), - endpoint: mock_server.uri(), - model: "test-model".to_string(), - api_key: "test-api-key".to_string(), - protocols: vec!["openai_chat_completions".to_string()], - auth: AuthHeader::Bearer, - default_headers: Vec::new(), - passthrough_headers: Vec::new(), - timeout: Duration::from_secs(1), - model_in_path: false, - request_path_override: None, - }]; - - let body = serde_json::to_vec(&serde_json::json!({ - "model": "test-model", - "messages": [{"role": "user", "content": "hi"}] - })) - .unwrap(); - - let result = router - .proxy_with_candidates( - "openai_chat_completions", - "POST", - "/v1/chat/completions", - vec![("content-type".to_string(), "application/json".to_string())], - bytes::Bytes::from(body), - &candidates, - ) - .await; - - assert!(result.is_err(), "buffered proxy should timeout"); - let err = result.unwrap_err().to_string(); - assert!( - err.contains("timed out"), - "error should mention timeout, got: {err}" - ); -} diff --git a/crates/openshell-sandbox/Cargo.toml b/crates/openshell-sandbox/Cargo.toml index c653db84d..3463f0376 100644 --- a/crates/openshell-sandbox/Cargo.toml +++ b/crates/openshell-sandbox/Cargo.toml @@ -60,6 +60,17 @@ default = ["telemetry", "bundled-ca-roots"] ## `--no-default-features --features system-ca-roots` to build a supervisor ## that uses the platform trust store with telemetry intact. system-ca-roots = ["telemetry"] +## Convenience alias: every default feature except `telemetry`. Build a +## telemetry-free supervisor with +## `--no-default-features --features defaults-without-telemetry` and stay +## correct as new default features are added. Cargo cannot subtract a single +## default feature, so this alias must be paired with `--no-default-features`; +## enabling it alongside `telemetry` is a compile error rather than a silent +## telemetry-on build. Kept in sync with `default` by +## `rust:verify:defaults-without-telemetry`. Do not pair it with +## `system-ca-roots`, which re-enables `telemetry`; a build with neither +## telemetry nor bundled CA roots is plain `--no-default-features`. +defaults-without-telemetry = ["bundled-ca-roots"] telemetry = ["openshell-core/telemetry"] bundled-ca-roots = ["openshell-supervisor-network/bundled-ca-roots"] diff --git a/crates/openshell-sandbox/src/lib.rs b/crates/openshell-sandbox/src/lib.rs index e70aac7a9..d86c94b97 100644 --- a/crates/openshell-sandbox/src/lib.rs +++ b/crates/openshell-sandbox/src/lib.rs @@ -5,6 +5,16 @@ //! //! This crate provides process sandboxing and monitoring capabilities. +// `defaults-without-telemetry` is an alias for the default feature set minus +// `telemetry`, not a switch that turns telemetry off. Cargo cannot subtract a +// default feature, so adding it on top of the defaults would otherwise produce +// a telemetry-on build that reads as telemetry-free. Fail the build instead. +#[cfg(all(feature = "telemetry", feature = "defaults-without-telemetry"))] +compile_error!( + "features `telemetry` and `defaults-without-telemetry` are mutually exclusive; \ + build a telemetry-free supervisor with `--no-default-features --features defaults-without-telemetry`" +); + mod activity_aggregator; mod denial_aggregator; #[cfg_attr(not(target_os = "linux"), allow(dead_code))] @@ -112,8 +122,8 @@ pub async fn run_sandbox( ssh_socket_path: Option, _health_check: bool, _health_port: u16, - inference_routes: Option, ocsf_enabled: Arc, + ocsf_schema_version: Arc>, network_enabled: bool, process_enabled: bool, upstream_proxy_args: openshell_supervisor_network::upstream_proxy::UpstreamProxyArgs, @@ -537,7 +547,6 @@ pub async fn run_sandbox( sandbox_id.as_deref(), sandbox_name_for_agg.as_deref(), openshell_endpoint_for_proxy.as_deref(), - inference_routes.as_deref(), denial_tx, activity_tx, agent_proposals.clone(), @@ -734,6 +743,7 @@ pub async fn run_sandbox( let poll_endpoint = endpoint.to_string(); let poll_engine = engine.clone(); let poll_ocsf_enabled = ocsf_enabled.clone(); + let poll_ocsf_schema_version = ocsf_schema_version.clone(); let poll_pid = entrypoint_pid.clone(); let poll_provider_credentials = provider_credentials.clone(); let poll_policy_local = networking.as_ref().map(|n| n.policy_local_ctx.clone()); @@ -749,6 +759,7 @@ pub async fn run_sandbox( entrypoint_pid: poll_pid, interval_secs: poll_interval_secs, ocsf_enabled: poll_ocsf_enabled, + ocsf_schema_version: poll_ocsf_schema_version, provider_credentials: poll_provider_credentials, policy_local_ctx: poll_policy_local, agent_proposals: agent_proposals.clone(), @@ -3329,6 +3340,7 @@ struct PolicyPollLoopContext { entrypoint_pid: Arc, interval_secs: u64, ocsf_enabled: Arc, + ocsf_schema_version: Arc>, provider_credentials: ProviderCredentialState, policy_local_ctx: Option>, agent_proposals: AgentProposals, @@ -3771,6 +3783,7 @@ async fn run_policy_poll_loop_with_client( match initial_poll_disposition(&ctx.loaded_policy_origin, &result) { InitialPollDisposition::Acknowledge(candidate) => { apply_ocsf_json_setting(&ctx.ocsf_enabled, &result.settings); + apply_ocsf_schema_version_setting(&ctx.ocsf_schema_version, &result.settings); apply_agent_proposals_enabled( &ctx.agent_proposals, agent_proposals_enabled_from_settings(&result.settings), @@ -3798,6 +3811,7 @@ async fn run_policy_poll_loop_with_client( InitialPollDisposition::Reconcile => pending_result = Some(result), InitialPollDisposition::TrackOnly => { apply_ocsf_json_setting(&ctx.ocsf_enabled, &result.settings); + apply_ocsf_schema_version_setting(&ctx.ocsf_schema_version, &result.settings); apply_agent_proposals_enabled( &ctx.agent_proposals, agent_proposals_enabled_from_settings(&result.settings), @@ -4286,6 +4300,7 @@ async fn run_policy_poll_loop_with_client( // Apply OCSF JSON toggle from the `ocsf_json_enabled` setting. apply_ocsf_json_setting(&ctx.ocsf_enabled, &result.settings); + apply_ocsf_schema_version_setting(&ctx.ocsf_schema_version, &result.settings); // Apply the agent-proposals feature toggle. On a false→true transition // we lazily install the skill so a sandbox that started with the flag @@ -4340,6 +4355,37 @@ fn extract_bool_setting( }) } +fn apply_ocsf_schema_version_setting( + version: &std::sync::Mutex, + settings: &std::collections::HashMap, +) { + let new_version = extract_string_setting(settings, "ocsf_schema_version").unwrap_or_default(); + if let Ok(mut current) = version.lock() + && *current != new_version + { + info!( + ocsf_schema_version = %new_version, + "OCSF schema version target changed" + ); + *current = new_version; + } +} + +fn extract_string_setting( + settings: &std::collections::HashMap, + key: &str, +) -> Option { + use openshell_core::proto::setting_value; + settings + .get(key) + .and_then(|es| es.value.as_ref()) + .and_then(|sv| sv.value.as_ref()) + .and_then(|v| match v { + setting_value::Value::StringValue(s) => Some(s.clone()), + _ => None, + }) +} + fn agent_proposals_enabled_from_settings( settings: &std::collections::HashMap, ) -> bool { @@ -4866,8 +4912,8 @@ filesystem_policy: #[test] fn discover_policy_restrictive_default_blocks_network() { - // In cluster mode we keep proxy mode enabled so `inference.local` - // can always be routed through proxy/OPA controls. + // Cluster sandboxes keep proxy mode enabled so egress is evaluated by + // the network policy engine. let proto = openshell_policy::restrictive_default_policy(); let local_policy = SandboxPolicy::try_from(proto).expect("conversion should succeed"); assert!(matches!(local_policy.network.mode, NetworkMode::Proxy)); @@ -5035,6 +5081,7 @@ network_policies: entrypoint_pid: Arc::new(AtomicU32::new(0)), interval_secs: 0, ocsf_enabled: Arc::new(AtomicBool::new(false)), + ocsf_schema_version: Arc::new(std::sync::Mutex::new(String::new())), provider_credentials: ProviderCredentialState::from_child_env_snapshot( 0, std::collections::HashMap::new(), diff --git a/crates/openshell-sandbox/src/main.rs b/crates/openshell-sandbox/src/main.rs index 710837865..440676ad1 100644 --- a/crates/openshell-sandbox/src/main.rs +++ b/crates/openshell-sandbox/src/main.rs @@ -10,7 +10,7 @@ use std::sync::atomic::AtomicBool; use clap::Parser; use miette::{IntoDiagnostic, Result}; use openshell_ocsf::{OcsfJsonlLayer, OcsfShorthandLayer}; -use tracing::{info, warn}; +use tracing::{error, info, warn}; use tracing_subscriber::EnvFilter; use tracing_subscriber::filter::LevelFilter; use tracing_subscriber::{Layer, layer::SubscriberExt, util::SubscriberInitExt}; @@ -111,7 +111,9 @@ impl std::str::FromStr for Mode { #[command(about = "Process sandbox and monitor", long_about = None)] struct Args { /// Command to execute in the sandbox. - /// Defaults to `/bin/bash -l` if neither this nor the driver specification is provided. + /// Defaults to a login shell if neither this nor the driver specification is + /// provided: `/bin/bash -l` when available, otherwise a shell detected in the + /// sandbox image (e.g. `/bin/sh` on Alpine). #[arg(trailing_var_arg = true)] command: Vec, @@ -163,12 +165,6 @@ struct Args { #[arg(long, env = openshell_core::sandbox_env::SSH_SOCKET_PATH)] ssh_socket_path: Option, - /// Path to YAML inference routes for standalone routing. - /// When set, inference routes are loaded from this file instead of - /// fetching a bundle from the gateway. - #[arg(long, env = "OPENSHELL_INFERENCE_ROUTES")] - inference_routes: Option, - /// Enable health check endpoint. #[arg(long)] health_check: bool, @@ -531,7 +527,6 @@ fn main() -> Result<()> { .build() .into_diagnostic()?; return runtime.block_on(async move { - let _ = rustls::crypto::ring::default_provider().install_default(); let exit = openshell_supervisor_process::debug_rpc::run(&raw_args[2..]).await?; std::process::exit(exit); }); @@ -575,10 +570,7 @@ fn main() -> Result<()> { .build() .into_diagnostic()?; - let exit_code = runtime.block_on(async move { - // Install rustls crypto provider before any TLS connections (including log push). - let _ = rustls::crypto::ring::default_provider().install_default(); - + let result = runtime.block_on(async move { // Set up optional log push layer (gRPC mode only). let log_push_state = if let (Some(sandbox_id), Some(endpoint)) = (&args.sandbox_id, &args.openshell_endpoint) @@ -600,6 +592,7 @@ fn main() -> Result<()> { // `ocsf_json_enabled` setting changes. The JSONL layer checks it // on each event and short-circuits when false. let ocsf_enabled = Arc::new(AtomicBool::new(false)); + let ocsf_schema_version = Arc::new(std::sync::Mutex::new(String::new())); // Keep guards alive for the entire process. When a guard is dropped the // non-blocking writer flushes remaining logs. @@ -618,7 +611,9 @@ fn main() -> Result<()> { .ok() .map(|roller| { let (writer, guard) = tracing_appender::non_blocking(roller); - let layer = OcsfJsonlLayer::new(writer).with_enabled_flag(ocsf_enabled.clone()); + let layer = OcsfJsonlLayer::new(writer) + .with_enabled_flag(ocsf_enabled.clone()) + .with_target_version(ocsf_schema_version.clone()); (layer, guard) }); let (jsonl_layer, jsonl_guard) = match jsonl_logging { @@ -678,6 +673,12 @@ fn main() -> Result<()> { ) }; + // An omitted command (the gateway leaves the default empty rather than + // baking a shell it cannot verify) is resolved to a login shell here, in + // the supervisor, so it matches the sandbox image: bash when present, + // otherwise /bin/sh (e.g. Alpine). An explicit command is used verbatim. + let command = resolve_default_command(command); + info!(command = ?command, "Starting sandbox"); // Note: "Starting sandbox" stays as plain info!() since the OCSF context // is not yet initialized at this point (run_sandbox hasn't been called). @@ -706,18 +707,46 @@ fn main() -> Result<()> { args.ssh_socket_path, args.health_check, args.health_port, - args.inference_routes, ocsf_enabled, + ocsf_schema_version, args.mode.network, args.mode.process, upstream_proxy_args, ) .await - })?; + }); + + let exit_code = match result { + Ok(exit_code) => exit_code, + Err(error) + if error + .to_string() + .contains("image workspace validation failed") => + { + error!(%error, "Image workspace validation failed"); + eprintln!("{error:?}"); + openshell_core::driver_utils::SUPERVISOR_EXIT_WORKSPACE_VALIDATION_FAILED + } + Err(error) => return Err(error), + }; std::process::exit(exit_code); } +/// Resolve an omitted canonical command to a login shell that exists in this +/// sandbox image. Empty means "use the default": the gateway leaves an omitted +/// command empty rather than persisting a shell it cannot verify, so the +/// supervisor picks one here against the real sandbox filesystem (bash when +/// present, otherwise `/bin/sh`). An explicit command is returned unchanged. +fn resolve_default_command(command: Vec) -> Vec { + if !command.is_empty() { + return command; + } + let shell = openshell_core::shell::detect_login_shell(); + info!(shell = %shell, "no command specified; resolved default login shell"); + vec![shell, "-l".to_string()] +} + #[cfg(test)] mod tests { use super::*; diff --git a/crates/openshell-sandbox/src/sidecar_control.rs b/crates/openshell-sandbox/src/sidecar_control.rs index fdf88f050..11f3e68e2 100644 --- a/crates/openshell-sandbox/src/sidecar_control.rs +++ b/crates/openshell-sandbox/src/sidecar_control.rs @@ -254,6 +254,10 @@ impl TryFrom for BootstrapData { let policy_proto = openshell_core::proto::SandboxPolicy::decode(policy_proto.as_slice()) .into_diagnostic() .wrap_err("failed to decode sidecar bootstrap policy")?; + let policy_proto = canonicalize_sidecar_policy( + policy_proto, + "sidecar bootstrap policy failed validation", + )?; Ok(Self { policy_proto, @@ -290,6 +294,10 @@ impl TryFrom for ControlUpdate { openshell_core::proto::SandboxPolicy::decode(policy_proto.as_slice()) .into_diagnostic() .wrap_err("failed to decode sidecar policy update")?; + let policy_proto = canonicalize_sidecar_policy( + policy_proto, + "sidecar policy update failed validation", + )?; Ok(Self::Policy { policy_proto: Box::new(policy_proto), policy_hash, @@ -692,10 +700,75 @@ fn decode_server_message(line: &str) -> Result { .wrap_err("failed to decode sidecar server message") } +fn canonicalize_sidecar_policy( + policy: openshell_core::proto::SandboxPolicy, + error_message: &'static str, +) -> Result { + // Bootstrap and update messages must expose the same canonical typed + // policy to every process-supervisor consumer. Keep validation details + // out of this channel error because they can contain authored values. + openshell_policy::validate_and_canonicalize_sandbox_policy(policy) + .map_err(|_| miette::miette!(error_message)) +} + #[cfg(test)] mod tests { use super::*; - use openshell_core::proto::SandboxPolicy; + use openshell_core::proto::{McpOptions, NetworkEndpoint, NetworkPolicyRule, SandboxPolicy}; + + fn defaultable_mcp_policy(mcp: Option) -> SandboxPolicy { + let mut policy = openshell_policy::restrictive_default_policy(); + policy.network_policies.insert( + "mcp".to_string(), + NetworkPolicyRule { + name: "mcp".to_string(), + endpoints: vec![NetworkEndpoint { + host: "mcp.example.com".to_string(), + port: 443, + protocol: "mcp".to_string(), + mcp, + rules: vec![openshell_core::proto::L7Rule { + allow: Some(openshell_core::proto::L7Allow { + method: "tools/list".to_string(), + ..Default::default() + }), + }], + ..Default::default() + }], + ..Default::default() + }, + ); + policy + } + + fn mcp_versions(policy: &SandboxPolicy) -> &[String] { + policy.network_policies["mcp"].endpoints[0] + .mcp + .as_ref() + .expect("canonical MCP options") + .versions + .as_slice() + } + + fn bootstrap_message(policy: &SandboxPolicy) -> WireServerMessage { + WireServerMessage::BootstrapResponse { + policy_proto: policy.encode_to_vec(), + provider_env_revision: 0, + provider_env_generation: 0, + provider_child_env: HashMap::new(), + agent_proposals_enabled: false, + proxy_ca_cert_path: None, + proxy_ca_bundle_path: None, + } + } + + fn policy_update_message(policy: &SandboxPolicy) -> WireServerMessage { + WireServerMessage::PolicyUpdated { + policy_proto: policy.encode_to_vec(), + policy_hash: "hash".to_string(), + config_revision: 1, + } + } fn current_peer() -> ExpectedPeer { ExpectedPeer { @@ -704,6 +777,48 @@ mod tests { } } + #[test] + fn policy_messages_canonicalize_defaultable_mcp_versions() { + for raw in [ + defaultable_mcp_policy(None), + defaultable_mcp_policy(Some(McpOptions::default())), + ] { + let bootstrap = BootstrapData::try_from(bootstrap_message(&raw)) + .expect("defaultable MCP policy must pass bootstrap ingress"); + assert_eq!(mcp_versions(&bootstrap.policy_proto), ["2025-11-25"]); + + let update = ControlUpdate::try_from(policy_update_message(&raw)) + .expect("defaultable MCP policy must pass update ingress"); + let ControlUpdate::Policy { policy_proto, .. } = update else { + panic!("expected policy update"); + }; + assert_eq!(mcp_versions(&policy_proto), ["2025-11-25"]); + } + } + + #[test] + fn policy_messages_reject_invalid_mcp_versions_without_echoing_values() { + let invalid = defaultable_mcp_policy(Some(McpOptions { + versions: vec!["latest".to_string()], + ..Default::default() + })); + + let bootstrap_error = BootstrapData::try_from(bootstrap_message(&invalid)) + .expect_err("invalid MCP policy must not pass bootstrap ingress") + .to_string(); + assert_eq!( + bootstrap_error, + "sidecar bootstrap policy failed validation" + ); + assert!(!bootstrap_error.contains("latest")); + + let update_error = ControlUpdate::try_from(policy_update_message(&invalid)) + .expect_err("invalid MCP policy must not pass update ingress") + .to_string(); + assert_eq!(update_error, "sidecar policy update failed validation"); + assert!(!update_error.contains("latest")); + } + #[tokio::test] async fn bootstrap_round_trips_policy_and_provider_env() { let dir = tempfile::tempdir().unwrap(); diff --git a/crates/openshell-sdk/README.md b/crates/openshell-sdk/README.md index b5f6f1b50..76a527f4b 100644 --- a/crates/openshell-sdk/README.md +++ b/crates/openshell-sdk/README.md @@ -9,9 +9,10 @@ gateway-name resolution. ## Two layers - `OpenShellClient` — the curated, sandbox-focused surface: health, sandbox - CRUD, readiness/deletion waits, and non-streaming exec. + CRUD, reusable sandbox template CRUD, readiness/deletion waits, and + non-streaming exec. - `raw` — direct access to the generated tonic clients for RPCs the curated - surface doesn't yet cover (inference, providers, policy, logs, settings, SSH, + surface doesn't yet cover (providers, policy, logs, settings, SSH, forwarding). ## Auth and refresh @@ -20,9 +21,8 @@ The curated surface drives OIDC refresh automatically: proactively before a request and reactively on `Unauthenticated`. Refreshes are single-flight, so only one is in flight at a time. -The plain `raw_grpc`/`raw_inference` accessors do not refresh; they return a -client bound to the current token. When a refresher is wired, use -`raw_grpc_fresh`/`raw_inference_fresh` to refresh before the call, and +The plain `raw_grpc` accessor does not refresh; it returns a client bound to +the current token. When a refresher is wired, use `raw_grpc_fresh` to refresh before the call, and `force_refresh` to recover after a raw RPC returns `Unauthenticated`. The SDK consumes a `Refresh` trait that the caller implements; it does not run @@ -44,10 +44,58 @@ mTLS (client certificates) is not supported. `OpenShellClient::connect(ClientConfig)` returns a connected client exposing `health`, `create_sandbox`, `get_sandbox`, `list_sandboxes`, `delete_sandbox`, +`create_sandbox_from_template`, `create_sandbox_template`, +`get_sandbox_template`, `list_sandbox_templates`, `delete_sandbox_template`, +`list_sandboxes_all_workspaces`, `list_sandbox_templates_all_workspaces`, `wait_ready`, `wait_deleted`, and `exec`. Curated types (`SandboxSpec`, -`SandboxRef`, `Health`, `ListOptions`, `ExecOptions`, `SandboxPhase`) use -SDK-shaped enums rather than raw proto integers. Failures map to a typed -`SdkError` with a discriminable kind. +`SandboxRef`, `Health`, `ListOptions`, `SandboxTemplateListOptions`, +`ExecOptions`, `SandboxPhase`) use SDK-shaped enums rather than raw proto +integers where practical. Reusable template resources are exposed as +`SandboxWorkloadTemplate` proto aliases so callers can populate the full +portable workload shape and driver config. Failures map to a typed `SdkError` +with a discriminable kind. + +Curated calls without a workspace argument explicitly select the `default` +workspace. Cross-workspace listing uses the separate `*_all_workspaces` +methods and requires Platform Admin access. + +```rust +use openshell_sdk::{ + ClientConfig, OpenShellClient, SandboxTemplateCreateSpec, + SandboxWorkloadConfig, SandboxWorkloadTemplate, SandboxWorkloadTemplateSpec, +}; + +# async fn run() -> Result<(), openshell_sdk::SdkError> { +let client = OpenShellClient::connect(ClientConfig::new("http://127.0.0.1:8080")).await?; +client + .create_sandbox_template(SandboxWorkloadTemplate { + metadata: Some(openshell_sdk::raw::proto::datamodel::v1::ObjectMeta { + name: "python".to_string(), + ..Default::default() + }), + spec: Some(SandboxWorkloadTemplateSpec { + workload: Some(SandboxWorkloadConfig { + image: "ghcr.io/nvidia/openshell-community/sandboxes/python:latest".to_string(), + ..Default::default() + }), + ..Default::default() + }), + }) + .await?; + +let _sandbox = client + .create_sandbox_from_template(SandboxTemplateCreateSpec { + template_name: "python".to_string(), + policy: Some(openshell_sdk::raw::proto::SandboxPolicy { + version: 1, + ..Default::default() + }), + ..Default::default() + }) + .await?; +# Ok(()) +# } +``` Enable the `extension` Cargo feature when building an OpenShell extension service: diff --git a/crates/openshell-sdk/src/client.rs b/crates/openshell-sdk/src/client.rs index a02b5735a..4713f0ed1 100644 --- a/crates/openshell-sdk/src/client.rs +++ b/crates/openshell-sdk/src/client.rs @@ -4,19 +4,19 @@ //! High-level async client over the gateway gRPC surface. //! //! Covers the sandbox-focused MVP slice: health, sandbox CRUD, readiness / -//! deletion waits, and non-streaming exec. Other RPCs (inference, providers, -//! policy, logs, settings, SSH, forwarding) are reachable via -//! [`OpenShellClient::raw_grpc`] / [`OpenShellClient::raw_inference`]. +//! deletion waits, and non-streaming exec. Other RPCs (providers, policy, +//! logs, settings, SSH, and forwarding) are reachable via +//! [`OpenShellClient::raw_grpc`]. use crate::auth::{BearerSlot, EdgeAuthInterceptor, bearer_metadata}; use crate::config::{AuthConfig, ClientConfig}; use crate::error::{Result, SdkError}; -use crate::raw::{AuthedGrpcClient, AuthedInferenceClient}; +use crate::raw::AuthedGrpcClient; use crate::refresh::{RefreshedToken, TokenSource}; use crate::transport; use crate::types::{ ExecOptions, ExecResult, Health, ListOptions, SandboxPhase, SandboxRef, SandboxSpec, - WorkspaceRef, + SandboxTemplateCreateSpec, SandboxTemplateListOptions, SandboxWorkloadTemplate, WorkspaceRef, }; use futures::StreamExt; use openshell_core::proto; @@ -114,24 +114,6 @@ impl OpenShellClient { Ok(self.raw_grpc()) } - /// Authenticated gRPC client for the inference service. - /// - /// Like [`OpenShellClient::raw_grpc`], this does not drive OIDC refresh; - /// use [`OpenShellClient::raw_inference_fresh`] when a refresher is wired. - pub fn raw_inference(&self) -> AuthedInferenceClient { - proto::inference_client::InferenceClient::with_interceptor( - self.channel.clone(), - self.interceptor.clone(), - ) - } - - /// Like [`OpenShellClient::raw_inference`], but proactively refreshes the - /// bearer token first (see [`OpenShellClient::raw_grpc_fresh`]). - pub async fn raw_inference_fresh(&self) -> Result { - self.ensure_fresh().await?; - Ok(self.raw_inference()) - } - /// Force an OIDC refresh and write the new token into the live bearer /// slot, regardless of expiry. Returns `true` when a refresher is wired /// and a fresh token was minted, `false` for static auth. Use after a @@ -163,13 +145,111 @@ impl OpenShellClient { sandbox_from_response(response.sandbox) } + /// Create a new sandbox from a workspace-scoped workload template name. + pub async fn create_sandbox_from_template( + &self, + spec: SandboxTemplateCreateSpec, + ) -> Result { + let request = create_sandbox_from_template_request(spec); + let response = self + .unary(|mut grpc| { + let request = request.clone(); + async move { grpc.create_sandbox(request).await } + }) + .await?; + sandbox_from_response(response.sandbox) + } + + /// Create a reusable sandbox template in the default workspace. + pub async fn create_sandbox_template( + &self, + template: SandboxWorkloadTemplate, + ) -> Result { + let response = self + .unary(|mut grpc| { + let request = proto::CreateSandboxTemplateRequest { + template: Some(template.clone()), + workspace_scope: Some(proto::workspace_selector("default")), + }; + async move { grpc.create_sandbox_template(request).await } + }) + .await?; + sandbox_template_from_response(response.template) + } + + /// Fetch a reusable sandbox template by name from the default workspace. + pub async fn get_sandbox_template(&self, name: &str) -> Result { + let response = self + .unary(|mut grpc| { + let request = proto::GetSandboxTemplateRequest { + name: name.to_string(), + workspace_scope: Some(proto::workspace_selector("default")), + }; + async move { grpc.get_sandbox_template(request).await } + }) + .await?; + sandbox_template_from_response(response.template) + } + + /// List reusable sandbox templates in the default workspace. + pub async fn list_sandbox_templates( + &self, + opts: SandboxTemplateListOptions, + ) -> Result> { + let response = self + .unary(|mut grpc| { + let request = proto::ListSandboxTemplatesRequest { + limit: opts.limit, + offset: opts.offset, + label_selector: opts.label_selector.clone(), + workspace_scope: Some(proto::workspace_selector("default")), + }; + async move { grpc.list_sandbox_templates(request).await } + }) + .await?; + Ok(response.templates) + } + + /// List reusable sandbox templates across all workspaces. + pub async fn list_sandbox_templates_all_workspaces( + &self, + opts: SandboxTemplateListOptions, + ) -> Result> { + let response = self + .unary(|mut grpc| { + let request = proto::ListSandboxTemplatesRequest { + limit: opts.limit, + offset: opts.offset, + label_selector: opts.label_selector.clone(), + workspace_scope: Some(proto::all_workspaces_selector()), + }; + async move { grpc.list_sandbox_templates(request).await } + }) + .await?; + Ok(response.templates) + } + + /// Delete a reusable sandbox template by name from the default workspace. + pub async fn delete_sandbox_template(&self, name: &str) -> Result { + let response = self + .unary(|mut grpc| { + let request = proto::DeleteSandboxTemplateRequest { + name: name.to_string(), + workspace_scope: Some(proto::workspace_selector("default")), + }; + async move { grpc.delete_sandbox_template(request).await } + }) + .await?; + Ok(response.deleted) + } + /// Fetch a sandbox by name. pub async fn get_sandbox(&self, name: &str) -> Result { let response = self .unary(|mut grpc| { let request = proto::GetSandboxRequest { name: name.to_string(), - workspace: String::new(), + workspace_scope: Some(proto::workspace_selector("default")), }; async move { grpc.get_sandbox(request).await } }) @@ -185,8 +265,7 @@ impl OpenShellClient { limit: opts.limit, offset: opts.offset, label_selector: opts.label_selector.clone().unwrap_or_default(), - workspace: String::new(), - all_workspaces: false, + workspace_scope: Some(proto::workspace_selector("default")), }; async move { grpc.list_sandboxes(request).await } }) @@ -209,7 +288,7 @@ impl OpenShellClient { .unary(|mut grpc| { let request = proto::DeleteSandboxRequest { name: name.to_string(), - workspace: String::new(), + workspace_scope: Some(proto::workspace_selector("default")), }; async move { grpc.delete_sandbox(request).await } }) @@ -223,7 +302,7 @@ impl OpenShellClient { .unary(|mut grpc| { let request = proto::StopSandboxRequest { name: name.to_string(), - workspace: String::new(), + workspace_scope: Some(proto::workspace_selector("default")), }; async move { grpc.stop_sandbox(request).await } }) @@ -237,7 +316,7 @@ impl OpenShellClient { .unary(|mut grpc| { let request = proto::StartSandboxRequest { name: name.to_string(), - workspace: String::new(), + workspace_scope: Some(proto::workspace_selector("default")), }; async move { grpc.start_sandbox(request).await } }) @@ -311,8 +390,7 @@ impl OpenShellClient { limit: opts.limit, offset: opts.offset, label_selector: opts.label_selector.clone().unwrap_or_default(), - workspace: String::new(), - all_workspaces: true, + workspace_scope: Some(proto::all_workspaces_selector()), }; async move { grpc.list_sandboxes(request).await } }) @@ -556,7 +634,7 @@ impl WorkspaceScopedClient { /// Create a new sandbox in this workspace. pub async fn create_sandbox(&self, spec: SandboxSpec) -> Result { let mut request = create_sandbox_request(spec); - request.workspace = self.workspace.clone(); + request.workspace_scope = Some(proto::workspace_selector(&self.workspace)); let response = self .client .unary(|mut grpc| { @@ -567,6 +645,91 @@ impl WorkspaceScopedClient { sandbox_from_response(response.sandbox) } + /// Create a new sandbox from a template in this workspace. + pub async fn create_sandbox_from_template( + &self, + spec: SandboxTemplateCreateSpec, + ) -> Result { + let mut request = create_sandbox_from_template_request(spec); + request.workspace_scope = Some(proto::workspace_selector(&self.workspace)); + let response = self + .client + .unary(|mut grpc| { + let request = request.clone(); + async move { grpc.create_sandbox(request).await } + }) + .await?; + sandbox_from_response(response.sandbox) + } + + /// Create a reusable sandbox template in this workspace. + pub async fn create_sandbox_template( + &self, + template: SandboxWorkloadTemplate, + ) -> Result { + let response = self + .client + .unary(|mut grpc| { + let request = proto::CreateSandboxTemplateRequest { + template: Some(template.clone()), + workspace_scope: Some(proto::workspace_selector(&self.workspace)), + }; + async move { grpc.create_sandbox_template(request).await } + }) + .await?; + sandbox_template_from_response(response.template) + } + + /// Fetch a reusable sandbox template by name in this workspace. + pub async fn get_sandbox_template(&self, name: &str) -> Result { + let response = self + .client + .unary(|mut grpc| { + let request = proto::GetSandboxTemplateRequest { + name: name.to_string(), + workspace_scope: Some(proto::workspace_selector(&self.workspace)), + }; + async move { grpc.get_sandbox_template(request).await } + }) + .await?; + sandbox_template_from_response(response.template) + } + + /// List reusable sandbox templates in this workspace. + pub async fn list_sandbox_templates( + &self, + opts: SandboxTemplateListOptions, + ) -> Result> { + let response = self + .client + .unary(|mut grpc| { + let request = proto::ListSandboxTemplatesRequest { + limit: opts.limit, + offset: opts.offset, + label_selector: opts.label_selector.clone(), + workspace_scope: Some(proto::workspace_selector(&self.workspace)), + }; + async move { grpc.list_sandbox_templates(request).await } + }) + .await?; + Ok(response.templates) + } + + /// Delete a reusable sandbox template by name in this workspace. + pub async fn delete_sandbox_template(&self, name: &str) -> Result { + let response = self + .client + .unary(|mut grpc| { + let request = proto::DeleteSandboxTemplateRequest { + name: name.to_string(), + workspace_scope: Some(proto::workspace_selector(&self.workspace)), + }; + async move { grpc.delete_sandbox_template(request).await } + }) + .await?; + Ok(response.deleted) + } + /// Fetch a sandbox by name in this workspace. pub async fn get_sandbox(&self, name: &str) -> Result { let response = self @@ -574,7 +737,7 @@ impl WorkspaceScopedClient { .unary(|mut grpc| { let request = proto::GetSandboxRequest { name: name.to_string(), - workspace: self.workspace.clone(), + workspace_scope: Some(proto::workspace_selector(&self.workspace)), }; async move { grpc.get_sandbox(request).await } }) @@ -591,8 +754,7 @@ impl WorkspaceScopedClient { limit: opts.limit, offset: opts.offset, label_selector: opts.label_selector.clone().unwrap_or_default(), - workspace: self.workspace.clone(), - all_workspaces: false, + workspace_scope: Some(proto::workspace_selector(&self.workspace)), }; async move { grpc.list_sandboxes(request).await } }) @@ -611,7 +773,7 @@ impl WorkspaceScopedClient { .unary(|mut grpc| { let request = proto::DeleteSandboxRequest { name: name.to_string(), - workspace: self.workspace.clone(), + workspace_scope: Some(proto::workspace_selector(&self.workspace)), }; async move { grpc.delete_sandbox(request).await } }) @@ -626,7 +788,7 @@ impl WorkspaceScopedClient { .unary(|mut grpc| { let request = proto::StopSandboxRequest { name: name.to_string(), - workspace: self.workspace.clone(), + workspace_scope: Some(proto::workspace_selector(&self.workspace)), }; async move { grpc.stop_sandbox(request).await } }) @@ -641,7 +803,7 @@ impl WorkspaceScopedClient { .unary(|mut grpc| { let request = proto::StartSandboxRequest { name: name.to_string(), - workspace: self.workspace.clone(), + workspace_scope: Some(proto::workspace_selector(&self.workspace)), }; async move { grpc.start_sandbox(request).await } }) @@ -835,7 +997,37 @@ fn create_sandbox_request(spec: SandboxSpec) -> proto::CreateSandboxRequest { name: name.unwrap_or_default(), labels, annotations: HashMap::new(), - workspace: String::new(), + workspace_scope: Some(proto::workspace_selector("default")), + await_main_process_attachment: false, + workload_template_name: String::new(), + } +} + +fn create_sandbox_from_template_request( + spec: SandboxTemplateCreateSpec, +) -> proto::CreateSandboxRequest { + let SandboxTemplateCreateSpec { + name, + template_name, + labels, + providers, + command, + tty, + policy, + } = spec; + proto::CreateSandboxRequest { + spec: Some(proto::SandboxSpec { + providers, + command, + tty, + policy, + ..proto::SandboxSpec::default() + }), + name: name.unwrap_or_default(), + labels, + annotations: HashMap::new(), + workspace_scope: Some(proto::workspace_selector("default")), + workload_template_name: template_name, await_main_process_attachment: false, } } @@ -846,6 +1038,13 @@ fn sandbox_from_response(sandbox: Option) -> Result .ok_or_else(|| SdkError::invalid_config("sandbox missing from gateway response")) } +fn sandbox_template_from_response( + template: Option, +) -> Result { + template + .ok_or_else(|| SdkError::invalid_config("sandbox template missing from gateway response")) +} + fn map_status(status: tonic::Status) -> SdkError { let message = status.message().to_string(); match status.code() { diff --git a/crates/openshell-sdk/src/config.rs b/crates/openshell-sdk/src/config.rs index 27a741173..bc4945c8e 100644 --- a/crates/openshell-sdk/src/config.rs +++ b/crates/openshell-sdk/src/config.rs @@ -56,8 +56,7 @@ impl AuthConfig { /// Configuration for opening a gRPC channel to an `OpenShell` gateway. /// -/// Consumed by `openshell_sdk::transport::grpc_client` and the -/// inference-client equivalent. One `ClientConfig` per logical connection; +/// Consumed by `openshell_sdk::transport::grpc_client`. One `ClientConfig` per logical connection; /// callers that want connection pooling cache the resulting `tonic::Channel`. // // NOTE: diff --git a/crates/openshell-sdk/src/lib.rs b/crates/openshell-sdk/src/lib.rs index c57944ca1..294471f0d 100644 --- a/crates/openshell-sdk/src/lib.rs +++ b/crates/openshell-sdk/src/lib.rs @@ -6,10 +6,11 @@ //! Two layers: //! //! - [`OpenShellClient`] — the high-level sandbox-focused MVP surface: -//! health, sandbox CRUD, readiness/deletion waits, non-streaming exec. +//! health, sandbox CRUD, reusable sandbox templates, readiness/deletion +//! waits, and non-streaming exec. //! - [`raw`] — direct access to the generated tonic clients for RPCs the -//! curated surface doesn't yet cover (inference, providers, policy, logs, -//! settings, SSH, forwarding). +//! curated surface doesn't yet cover (providers, policy, logs, settings, +//! SSH, forwarding). //! //! Owns the gRPC transport stack — channel construction, TLS material //! handling, request interceptors, OIDC token refresh, and the Cloudflare @@ -48,6 +49,8 @@ pub use config::{AuthConfig, ClientConfig}; pub use error::SdkError; pub use refresh::{Refresh, RefreshError, RefreshedToken, TokenSource}; pub use types::{ - ExecOptions, ExecResult, Health, ListOptions, SandboxPhase, SandboxRef, SandboxSpec, - ServiceStatus, WorkspaceRef, + ExecOptions, ExecResult, Health, ListOptions, SandboxPhase, SandboxRef, SandboxResources, + SandboxServiceLevel, SandboxSpec, SandboxStartup, SandboxTemplateCreateSpec, + SandboxTemplateListOptions, SandboxWorkloadConfig, SandboxWorkloadTemplate, + SandboxWorkloadTemplateProvenance, SandboxWorkloadTemplateSpec, ServiceStatus, WorkspaceRef, }; diff --git a/crates/openshell-sdk/src/oidc.rs b/crates/openshell-sdk/src/oidc.rs index 3f59c25cd..c39e8300a 100644 --- a/crates/openshell-sdk/src/oidc.rs +++ b/crates/openshell-sdk/src/oidc.rs @@ -139,6 +139,7 @@ pub async fn discover(issuer: &str, insecure: bool) -> Result { /// followed; OIDC providers should not redirect on the token endpoint. /// When `insecure` is true, TLS certificate verification is disabled. pub fn http_client(insecure: bool) -> reqwest::Client { + let _ = rustls::crypto::aws_lc_rs::default_provider().install_default(); let mut builder = reqwest::ClientBuilder::new().redirect(reqwest::redirect::Policy::none()); if insecure { builder = builder.danger_accept_invalid_certs(true); diff --git a/crates/openshell-sdk/src/raw.rs b/crates/openshell-sdk/src/raw.rs index e974b1925..3b1bb6605 100644 --- a/crates/openshell-sdk/src/raw.rs +++ b/crates/openshell-sdk/src/raw.rs @@ -6,8 +6,8 @@ //! //! Use this module when the curated high-level surface in //! [`crate::client::OpenShellClient`] doesn't expose the RPC or field you -//! need. The high-level surface is sandbox-focused for MVP; inference, -//! providers, policy, logs, settings, SSH, and forwarding all live here. +//! need. The high-level surface is sandbox-focused for MVP; providers, +//! policy, logs, settings, SSH, and forwarding all live here. //! //! ```ignore //! use openshell_sdk::{ClientConfig, OpenShellClient}; @@ -19,14 +19,17 @@ //! ``` pub use openshell_core::proto; -pub use openshell_core::proto::inference_client::InferenceClient; pub use openshell_core::proto::open_shell_client::OpenShellClient as GrpcClient; pub use openshell_core::proto::{ - CreateSandboxRequest, CreateWorkspaceRequest, DeleteSandboxRequest, DeleteWorkspaceRequest, - ExecSandboxRequest, GetSandboxRequest, GetWorkspaceRequest, HealthRequest, - ListProvidersRequest, ListSandboxesRequest, ListWorkspacesRequest, Sandbox, - SandboxPhase as ProtoSandboxPhase, SandboxSpec as ProtoSandboxSpec, SandboxTemplate, - ServiceStatus as ProtoServiceStatus, StartSandboxRequest, StopSandboxRequest, Workspace, + CreateSandboxRequest, CreateSandboxTemplateRequest, CreateWorkspaceRequest, + DeleteSandboxRequest, DeleteSandboxTemplateRequest, DeleteWorkspaceRequest, ExecSandboxRequest, + GetSandboxRequest, GetSandboxTemplateRequest, GetWorkspaceRequest, HealthRequest, + ListProvidersRequest, ListSandboxTemplatesRequest, ListSandboxesRequest, ListWorkspacesRequest, + Sandbox, SandboxPhase as ProtoSandboxPhase, SandboxResources, SandboxServiceLevel, + SandboxSpec as ProtoSandboxSpec, SandboxStartup, SandboxTemplate, SandboxTemplateResponse, + SandboxWorkloadConfig, SandboxWorkloadTemplate, SandboxWorkloadTemplateProvenance, + SandboxWorkloadTemplateSpec, ServiceStatus as ProtoServiceStatus, StartSandboxRequest, + StopSandboxRequest, Workspace, }; /// Type alias for the gRPC client wrapped in the SDK's auth interceptor. @@ -36,11 +39,3 @@ pub type AuthedGrpcClient = GrpcClient< crate::EdgeAuthInterceptor, >, >; - -/// Type alias for the inference client wrapped in the SDK's auth interceptor. -pub type AuthedInferenceClient = InferenceClient< - tonic::service::interceptor::InterceptedService< - tonic::transport::Channel, - crate::EdgeAuthInterceptor, - >, ->; diff --git a/crates/openshell-sdk/src/refresh.rs b/crates/openshell-sdk/src/refresh.rs index a70d88276..fd38e28ed 100644 --- a/crates/openshell-sdk/src/refresh.rs +++ b/crates/openshell-sdk/src/refresh.rs @@ -175,7 +175,7 @@ impl TokenSource { })), refresher, flight: Arc::new(Mutex::new(Flight::default())), - skew: Duration::from_secs(60), + skew: Duration::from_mins(1), } } diff --git a/crates/openshell-sdk/src/transport.rs b/crates/openshell-sdk/src/transport.rs index 9b25ced28..84f7096c0 100644 --- a/crates/openshell-sdk/src/transport.rs +++ b/crates/openshell-sdk/src/transport.rs @@ -183,7 +183,7 @@ impl ServerCertVerifier for InsecureServerCertVerifier { } fn supported_verify_schemes(&self) -> Vec { - rustls::crypto::ring::default_provider() + rustls::crypto::aws_lc_rs::default_provider() .signature_verification_algorithms .supported_schemes() } diff --git a/crates/openshell-sdk/src/types.rs b/crates/openshell-sdk/src/types.rs index 9cb7171fd..12b5a72cb 100644 --- a/crates/openshell-sdk/src/types.rs +++ b/crates/openshell-sdk/src/types.rs @@ -117,6 +117,58 @@ pub struct SandboxSpec { pub tty: bool, } +/// Caller intent for creating a sandbox from a named workload template. +#[derive(Clone, Debug, Default)] +pub struct SandboxTemplateCreateSpec { + /// Optional user-supplied sandbox name. When empty the server generates one. + pub name: Option, + /// Workspace-scoped template name to resolve at creation time. + pub template_name: String, + /// Labels attached to the sandbox. + pub labels: HashMap, + /// Provider names to attach. + pub providers: Vec, + /// Exact canonical command. Empty selects the gateway's scratch login shell. + pub command: Vec, + /// Allocate a retained pseudo-terminal for the canonical command. + pub tty: bool, + /// Create-time sandbox policy. The named workload template supplies runtime + /// workload fields; policy remains part of the sandbox's governance spec. + pub policy: Option, +} + +/// Reusable sandbox workload template resource. +/// +/// This is a raw proto alias because template specs intentionally expose the +/// full portable workload shape plus driver-owned config. +pub type SandboxWorkloadTemplate = proto::SandboxWorkloadTemplate; + +/// Desired reusable workload shape for a [`SandboxWorkloadTemplate`]. +pub type SandboxWorkloadTemplateSpec = proto::SandboxWorkloadTemplateSpec; + +/// Portable sandbox workload configuration for template-backed sandboxes. +pub type SandboxWorkloadConfig = proto::SandboxWorkloadConfig; + +/// Portable resource requirements for template-backed sandboxes. +pub type SandboxResources = proto::SandboxResources; + +/// Desired service level for sandboxes created from a template. +pub type SandboxServiceLevel = proto::SandboxServiceLevel; + +/// Startup service-level settings for template-backed sandboxes. +pub type SandboxStartup = proto::SandboxStartup; + +/// Options for listing reusable sandbox templates. +#[derive(Clone, Debug, Default)] +pub struct SandboxTemplateListOptions { + /// Maximum templates to return. `0` defers to the server default. + pub limit: u32, + /// Offset into the result list. + pub offset: u32, + /// Optional label selector in `key=value,key2=value2` form. + pub label_selector: String, +} + /// Reference to a sandbox owned by the gateway. #[derive(Clone, Debug)] #[non_exhaustive] @@ -128,12 +180,28 @@ pub struct SandboxRef { pub labels: HashMap, pub resource_version: u64, pub exit_code: Option, + pub created_from_workload_template: Option, +} + +/// Reusable workload template revision used to create a sandbox. +#[derive(Clone, Debug, Eq, PartialEq)] +#[non_exhaustive] +pub struct SandboxWorkloadTemplateProvenance { + pub name: String, + pub resource_version: String, } impl SandboxRef { pub(crate) fn from_proto(sandbox: proto::Sandbox) -> Self { let phase = sandbox.phase().into(); let exit_code = sandbox.status.as_ref().and_then(|status| status.exit_code); + let created_from_workload_template = + sandbox + .created_from_workload_template + .map(|p| SandboxWorkloadTemplateProvenance { + name: p.name, + resource_version: p.resource_version, + }); let meta = sandbox.metadata.unwrap_or_default(); Self { id: meta.id, @@ -143,6 +211,7 @@ impl SandboxRef { labels: meta.labels, resource_version: meta.resource_version, exit_code, + created_from_workload_template, } } } diff --git a/crates/openshell-sdk/tests/client_mock.rs b/crates/openshell-sdk/tests/client_mock.rs index 1cdac7da4..5fd2133b5 100644 --- a/crates/openshell-sdk/tests/client_mock.rs +++ b/crates/openshell-sdk/tests/client_mock.rs @@ -12,7 +12,8 @@ use openshell_core::proto; use openshell_core::proto::open_shell_server::{OpenShell, OpenShellServer}; use openshell_sdk::{ AuthConfig, ClientConfig, ExecOptions, ListOptions, OpenShellClient, Refresh, RefreshError, - RefreshedToken, SandboxPhase, SandboxSpec, ServiceStatus as SdkServiceStatus, + RefreshedToken, SandboxPhase, SandboxSpec, SandboxTemplateCreateSpec, + SandboxTemplateListOptions, ServiceStatus as SdkServiceStatus, }; use std::collections::HashMap; use std::sync::Arc; @@ -22,6 +23,22 @@ use tokio::sync::Mutex; use tokio_stream::wrappers::TcpListenerStream; use tonic::{Response, Status}; +fn selected_workspace(scope: &Option) -> Option<&str> { + match scope.as_ref()?.selection.as_ref()? { + proto::datamodel::v1::workspace_selector::Selection::Workspace(workspace) => { + Some(workspace) + } + proto::datamodel::v1::workspace_selector::Selection::AllWorkspaces(_) => None, + } +} + +fn selects_all_workspaces(scope: &Option) -> bool { + matches!( + scope.as_ref().and_then(|scope| scope.selection.as_ref()), + Some(proto::datamodel::v1::workspace_selector::Selection::AllWorkspaces(_)) + ) +} + /// Captured fixture state — what the mock observed and the canned replies it /// returned. One per test so assertions are scoped. #[derive(Default)] @@ -29,6 +46,10 @@ struct MockState { last_get_name: Mutex>, last_get_workspace: Mutex>, last_create: Mutex>, + last_template_create: Mutex>, + last_template_get: Mutex>, + last_template_list: Mutex>, + last_template_delete: Mutex>, last_delete_name: Mutex>, last_delete_workspace: Mutex>, last_stop: Mutex>, @@ -61,6 +82,11 @@ fn sandbox_with_phase_ws( phase: proto::SandboxPhase, workspace: &str, ) -> proto::Sandbox { + let created_from_workload_template = + (name == "from-template").then(|| proto::SandboxWorkloadTemplateProvenance { + name: "python".to_string(), + resource_version: "7".to_string(), + }); proto::Sandbox { metadata: Some(proto::datamodel::v1::ObjectMeta { id: format!("id-{name}"), @@ -77,6 +103,7 @@ fn sandbox_with_phase_ws( phase: phase.into(), ..Default::default() }), + created_from_workload_template, } } @@ -98,8 +125,43 @@ fn workspace_proto(name: &str, phase: proto::datamodel::v1::WorkspacePhase) -> p } } +fn workload_template_proto(name: &str, workspace: &str) -> proto::SandboxWorkloadTemplate { + proto::SandboxWorkloadTemplate { + metadata: Some(proto::datamodel::v1::ObjectMeta { + id: format!("template-{workspace}-{name}"), + name: name.to_string(), + created_at_ms: 1_000_000, + labels: HashMap::new(), + annotations: HashMap::new(), + resource_version: 1, + deletion_timestamp_ms: 0, + workspace: workspace.to_string(), + }), + spec: Some(proto::SandboxWorkloadTemplateSpec { + workload: Some(proto::SandboxWorkloadConfig { + image: format!("ghcr.io/test/{name}:latest"), + environment: HashMap::new(), + resources: Some(proto::SandboxResources { + cpu: "1".to_string(), + memory: "512Mi".to_string(), + ..proto::SandboxResources::default() + }), + }), + driver_config: None, + desired_service_level: None, + }), + } +} + #[tonic::async_trait] impl OpenShell for TestOpenShell { + async fn begin_rootfs_tar_staging( + &self, + _request: tonic::Request, + ) -> Result, Status> { + Err(Status::unimplemented("not used by this test server")) + } + async fn report_main_process_exit( &self, _request: tonic::Request, @@ -177,6 +239,58 @@ impl OpenShell for TestOpenShell { })) } + async fn create_sandbox_template( + &self, + request: tonic::Request, + ) -> Result, Status> { + let request = request.into_inner(); + let template = request + .template + .clone() + .ok_or_else(|| Status::invalid_argument("missing template"))?; + *self.state.last_template_create.lock().await = Some(request); + Ok(Response::new(proto::SandboxTemplateResponse { + template: Some(template), + })) + } + + async fn get_sandbox_template( + &self, + request: tonic::Request, + ) -> Result, Status> { + let request = request.into_inner(); + let workspace = selected_workspace(&request.workspace_scope).unwrap_or("default"); + let template = workload_template_proto(&request.name, workspace); + *self.state.last_template_get.lock().await = Some(request); + Ok(Response::new(proto::SandboxTemplateResponse { + template: Some(template), + })) + } + + async fn list_sandbox_templates( + &self, + request: tonic::Request, + ) -> Result, Status> { + let request = request.into_inner(); + *self.state.last_template_list.lock().await = Some(request); + Ok(Response::new(proto::ListSandboxTemplatesResponse { + templates: vec![ + workload_template_proto("python", "default"), + workload_template_proto("cuda", "gpu"), + ], + })) + } + + async fn delete_sandbox_template( + &self, + request: tonic::Request, + ) -> Result, Status> { + *self.state.last_template_delete.lock().await = Some(request.into_inner()); + Ok(Response::new(proto::DeleteSandboxTemplateResponse { + deleted: true, + })) + } + async fn stop_sandbox( &self, request: tonic::Request, @@ -185,7 +299,7 @@ impl OpenShell for TestOpenShell { let sandbox = sandbox_with_phase_ws( &request.name, proto::SandboxPhase::Stopped, - &request.workspace, + selected_workspace(&request.workspace_scope).unwrap_or("default"), ); *self.state.last_stop.lock().await = Some(request); Ok(Response::new(proto::SandboxResponse { @@ -201,7 +315,7 @@ impl OpenShell for TestOpenShell { let sandbox = sandbox_with_phase_ws( &request.name, proto::SandboxPhase::Starting, - &request.workspace, + selected_workspace(&request.workspace_scope).unwrap_or("default"), ); *self.state.last_start.lock().await = Some(request); Ok(Response::new(proto::SandboxResponse { @@ -216,7 +330,8 @@ impl OpenShell for TestOpenShell { let req = request.into_inner(); let name = req.name; *self.state.last_get_name.lock().await = Some(name.clone()); - *self.state.last_get_workspace.lock().await = Some(req.workspace.clone()); + *self.state.last_get_workspace.lock().await = + selected_workspace(&req.workspace_scope).map(str::to_string); let count = self.state.get_calls.fetch_add(1, Ordering::SeqCst); if self.state.get_returns_not_found { @@ -285,7 +400,8 @@ impl OpenShell for TestOpenShell { ) -> Result, Status> { let req = request.into_inner(); *self.state.last_delete_name.lock().await = Some(req.name); - *self.state.last_delete_workspace.lock().await = Some(req.workspace); + *self.state.last_delete_workspace.lock().await = + selected_workspace(&req.workspace_scope).map(str::to_string); Ok(Response::new(proto::DeleteSandboxResponse { deleted: true, })) @@ -817,6 +933,99 @@ async fn create_sandbox_passes_spec_through() { ); } +#[tokio::test] +async fn create_sandbox_from_template_passes_template_name() { + let state = Arc::new(MockState::default()); + let endpoint = start_mock(state.clone()).await; + let client = connect(&endpoint).await; + + let sandbox = client + .create_sandbox_from_template(SandboxTemplateCreateSpec { + name: Some("from-template".to_string()), + template_name: "python".to_string(), + providers: vec!["openai".to_string()], + command: vec!["python".to_string(), "-m".to_string(), "agent".to_string()], + tty: false, + policy: Some(proto::SandboxPolicy { + version: 1, + ..Default::default() + }), + ..Default::default() + }) + .await + .unwrap(); + assert_eq!(sandbox.name, "from-template"); + + let observed = state.last_create.lock().await.clone().unwrap(); + assert_eq!(observed.name, "from-template"); + assert_eq!(observed.workload_template_name, "python"); + let observed_spec = observed.spec.unwrap(); + assert_eq!(observed_spec.providers, vec!["openai".to_string()]); + assert_eq!(observed_spec.command, vec!["python", "-m", "agent"]); + assert!(!observed_spec.tty); + assert_eq!(observed_spec.policy.unwrap().version, 1); +} + +#[tokio::test] +async fn sandbox_template_crud_uses_default_workspace() { + let state = Arc::new(MockState::default()); + let endpoint = start_mock(state.clone()).await; + let client = connect(&endpoint).await; + + let created = client + .create_sandbox_template(workload_template_proto("python", "")) + .await + .unwrap(); + assert_eq!(created.metadata.as_ref().unwrap().name, "python"); + + let observed_create = state.last_template_create.lock().await.clone().unwrap(); + assert_eq!( + selected_workspace(&observed_create.workspace_scope), + Some("default") + ); + assert_eq!( + observed_create + .template + .as_ref() + .and_then(|template| template.metadata.as_ref()) + .unwrap() + .name, + "python" + ); + + let fetched = client.get_sandbox_template("python").await.unwrap(); + assert_eq!(fetched.metadata.as_ref().unwrap().name, "python"); + let observed_get = state.last_template_get.lock().await.clone().unwrap(); + assert_eq!(observed_get.name, "python"); + assert_eq!( + selected_workspace(&observed_get.workspace_scope), + Some("default") + ); + + let listed = client + .list_sandbox_templates_all_workspaces(SandboxTemplateListOptions { + limit: 10, + offset: 2, + label_selector: String::new(), + }) + .await + .unwrap(); + assert_eq!(listed.len(), 2); + let observed_list = state.last_template_list.lock().await.clone().unwrap(); + assert_eq!(observed_list.limit, 10); + assert_eq!(observed_list.offset, 2); + assert!(selects_all_workspaces(&observed_list.workspace_scope)); + + let deleted = client.delete_sandbox_template("python").await.unwrap(); + assert!(deleted); + let observed_delete = state.last_template_delete.lock().await.clone().unwrap(); + assert_eq!(observed_delete.name, "python"); + assert_eq!( + selected_workspace(&observed_delete.workspace_scope), + Some("default") + ); +} + #[tokio::test] async fn get_sandbox_sends_name_and_maps_phase() { let state = Arc::new(MockState { @@ -835,6 +1044,21 @@ async fn get_sandbox_sends_name_and_maps_phase() { assert_eq!(observed.as_deref(), Some("my-box")); } +#[tokio::test] +async fn get_sandbox_preserves_workload_template_provenance() { + let state = Arc::new(MockState::default()); + let endpoint = start_mock(state.clone()).await; + let client = connect(&endpoint).await; + + let sandbox = client.get_sandbox("from-template").await.unwrap(); + + let provenance = sandbox + .created_from_workload_template + .expect("template provenance"); + assert_eq!(provenance.name, "python"); + assert_eq!(provenance.resource_version, "7"); +} + #[tokio::test] async fn list_sandboxes_propagates_filters() { let state = Arc::new(MockState::default()); @@ -881,7 +1105,7 @@ async fn stop_and_start_map_requests_and_phases() { assert_eq!(stopped.phase, SandboxPhase::Stopped); let stop = state.last_stop.lock().await.clone().unwrap(); assert_eq!(stop.name, "sleepy"); - assert!(stop.workspace.is_empty()); + assert_eq!(selected_workspace(&stop.workspace_scope), Some("default")); let started = client .workspace("team-a") @@ -891,7 +1115,7 @@ async fn stop_and_start_map_requests_and_phases() { assert_eq!(started.phase, SandboxPhase::Starting); let start = state.last_start.lock().await.clone().unwrap(); assert_eq!(start.name, "sleepy"); - assert_eq!(start.workspace, "team-a"); + assert_eq!(selected_workspace(&start.workspace_scope), Some("team-a")); } #[tokio::test] @@ -1134,7 +1358,40 @@ async fn workspace_scoped_create_passes_workspace() { assert_eq!(result.name, "my-box"); let observed = state.last_create.lock().await.clone().unwrap(); - assert_eq!(observed.workspace, "staging"); + assert_eq!( + selected_workspace(&observed.workspace_scope), + Some("staging") + ); +} + +#[tokio::test] +async fn workspace_scoped_create_from_template_passes_workspace() { + let state = Arc::new(MockState::default()); + let endpoint = start_mock(state.clone()).await; + let client = connect(&endpoint).await; + + let sandbox = client + .workspace("staging") + .create_sandbox_from_template(SandboxTemplateCreateSpec { + name: Some("from-template".to_string()), + template_name: "python".to_string(), + policy: Some(proto::SandboxPolicy { + version: 2, + ..Default::default() + }), + ..Default::default() + }) + .await + .unwrap(); + assert_eq!(sandbox.name, "from-template"); + + let observed = state.last_create.lock().await.clone().unwrap(); + assert_eq!( + selected_workspace(&observed.workspace_scope), + Some("staging") + ); + assert_eq!(observed.workload_template_name, "python"); + assert_eq!(observed.spec.unwrap().policy.unwrap().version, 2); } #[tokio::test] @@ -1165,8 +1422,59 @@ async fn workspace_scoped_list_passes_workspace() { assert_eq!(items.len(), 2); let observed = state.last_list_request.lock().await.clone().unwrap(); - assert_eq!(observed.workspace, "dev"); - assert!(!observed.all_workspaces); + assert_eq!(selected_workspace(&observed.workspace_scope), Some("dev")); +} + +#[tokio::test] +async fn workspace_scoped_sandbox_template_crud_passes_workspace() { + let state = Arc::new(MockState::default()); + let endpoint = start_mock(state.clone()).await; + let client = connect(&endpoint).await; + let ws = client.workspace("staging"); + + ws.create_sandbox_template(workload_template_proto("python", "staging")) + .await + .unwrap(); + let observed_create = state.last_template_create.lock().await.clone().unwrap(); + assert_eq!( + selected_workspace(&observed_create.workspace_scope), + Some("staging") + ); + + ws.get_sandbox_template("python").await.unwrap(); + let observed_get = state.last_template_get.lock().await.clone().unwrap(); + assert_eq!(observed_get.name, "python"); + assert_eq!( + selected_workspace(&observed_get.workspace_scope), + Some("staging") + ); + + let listed = ws + .list_sandbox_templates(SandboxTemplateListOptions::default()) + .await + .unwrap(); + assert_eq!(listed.len(), 2); + let observed_list = state.last_template_list.lock().await.clone().unwrap(); + assert_eq!( + selected_workspace(&observed_list.workspace_scope), + Some("staging") + ); + + client + .list_sandbox_templates_all_workspaces(SandboxTemplateListOptions::default()) + .await + .unwrap(); + let observed_all = state.last_template_list.lock().await.clone().unwrap(); + assert!(selects_all_workspaces(&observed_all.workspace_scope)); + + let deleted = ws.delete_sandbox_template("python").await.unwrap(); + assert!(deleted); + let observed_delete = state.last_template_delete.lock().await.clone().unwrap(); + assert_eq!(observed_delete.name, "python"); + assert_eq!( + selected_workspace(&observed_delete.workspace_scope), + Some("staging") + ); } #[tokio::test] @@ -1196,8 +1504,7 @@ async fn list_sandboxes_all_workspaces_sets_flag() { assert_eq!(items.len(), 2); let observed = state.last_list_request.lock().await.clone().unwrap(); - assert!(observed.all_workspaces); - assert!(observed.workspace.is_empty()); + assert!(selects_all_workspaces(&observed.workspace_scope)); } // ---- Workspace CRUD tests ---- diff --git a/crates/openshell-server-macros/src/lib.rs b/crates/openshell-server-macros/src/lib.rs index a698ae662..e85d31629 100644 --- a/crates/openshell-server-macros/src/lib.rs +++ b/crates/openshell-server-macros/src/lib.rs @@ -199,7 +199,7 @@ fn snake_to_pascal(ident: &str) -> String { /// Name of the per-service const emitted alongside the impl block. The /// service module is what disambiguates between services — every impl /// lives in its own module (`crate::grpc::AUTH_METADATA`, -/// `crate::inference::AUTH_METADATA`), so a fixed name reads more +/// another service's `AUTH_METADATA`), so a fixed name reads more /// naturally than `OPENSHELL_AUTH_METADATA` / `INFERENCE_AUTH_METADATA`. const AUTH_METADATA_CONST: &str = "AUTH_METADATA"; diff --git a/crates/openshell-server/Cargo.toml b/crates/openshell-server/Cargo.toml index ae35fc0fb..efff4f36b 100644 --- a/crates/openshell-server/Cargo.toml +++ b/crates/openshell-server/Cargo.toml @@ -10,10 +10,6 @@ rust-version.workspace = true license.workspace = true repository.workspace = true -[[bin]] -name = "openshell-gateway" -path = "src/main.rs" - [dependencies] openshell-bootstrap = { path = "../openshell-bootstrap" } openshell-core = { path = "../openshell-core", default-features = false, features = ["oauth"] } @@ -27,11 +23,11 @@ openshell-otel = { path = "../openshell-otel" } openshell-policy = { path = "../openshell-policy" } openshell-prover = { path = "../openshell-prover" } openshell-providers = { path = "../openshell-providers" } -openshell-router = { path = "../openshell-router" } openshell-supervisor-middleware = { path = "../openshell-supervisor-middleware" } openshell-supervisor-middleware-builtins = { path = "../openshell-supervisor-middleware-builtins" } -# Kubernetes client (used by the `generate-certs` subcommand) +# Kubernetes client used by ServiceAccount bootstrap authentication and the +# `generate-certs` subcommand. kube = { workspace = true } k8s-openapi = { workspace = true } @@ -87,12 +83,12 @@ base64 = { workspace = true } futures = { workspace = true } bytes = { workspace = true } pin-project-lite = { workspace = true } -ring = { workspace = true } serde = { workspace = true } serde_json = { workspace = true } toml = { workspace = true } tokio-stream = { workspace = true } sqlx = { workspace = true } +sqlx-core = { workspace = true } reqwest = { workspace = true } aws-config = { workspace = true } aws-sdk-sts = { workspace = true } @@ -115,25 +111,8 @@ x509-parser = "0.16" arc-swap = "1" notify = "8" -[target.'cfg(not(target_os = "windows"))'.dependencies] -openshell-driver-docker = { path = "../openshell-driver-docker", optional = true } -openshell-driver-kubernetes = { path = "../openshell-driver-kubernetes", optional = true } -openshell-driver-podman = { path = "../openshell-driver-podman", optional = true } - -# MXC is the Windows-only in-process compute backend (openshell-driver-mxc is a -# no-op stub on other targets). It is only linked into the gateway on Windows. -[target.'cfg(target_os = "windows")'.dependencies] -openshell-driver-mxc = { path = "../openshell-driver-mxc" } - [features] -default = ["telemetry", "in-tree-compute-drivers"] -## Link the first-party compute drivers into the standard gateway binary. -## Disable this feature for a protocol-only gateway that uses external drivers. -in-tree-compute-drivers = [ - "dep:openshell-driver-docker", - "dep:openshell-driver-kubernetes", - "dep:openshell-driver-podman", -] +default = ["telemetry"] ## Compile in anonymous telemetry emission (forwards to openshell-core/telemetry). ## On by default; build with `--no-default-features` for a telemetry-free gateway ## that contains no telemetry endpoint, HTTP client, or emission code. @@ -141,11 +120,15 @@ telemetry = ["openshell-core/telemetry"] bundled-z3 = ["openshell-prover/bundled-z3"] test-support = [] +[build-dependencies] +tonic-prost-build = { workspace = true } +protoc-bin-vendored = { workspace = true } + [dev-dependencies] -hyper-rustls = { version = "0.27", default-features = false, features = ["native-tokio", "http1", "tls12", "logging", "ring"] } -rcgen = { version = "0.13", features = ["crypto", "pem"] } -rsa = { version = "0.9", features = ["pem"] } base64 = { workspace = true } +hyper-rustls = { version = "0.27", default-features = false, features = ["native-tokio", "http1", "tls12", "logging", "aws-lc-rs"] } +rcgen = { workspace = true } +rsa = { version = "0.9", features = ["pem"] } tokio-tungstenite = { workspace = true } futures-util = "0.3" wiremock = "0.6" diff --git a/crates/openshell-server/build.rs b/crates/openshell-server/build.rs new file mode 100644 index 000000000..b7999d28b --- /dev/null +++ b/crates/openshell-server/build.rs @@ -0,0 +1,50 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +use std::env; +use std::path::PathBuf; + +fn main() -> Result<(), Box> { + let manifest_dir = PathBuf::from(env::var("CARGO_MANIFEST_DIR")?); + let storage_proto_dir = manifest_dir.join("proto"); + let public_proto_dir = manifest_dir.join("../../proto"); + let storage_proto = storage_proto_dir.join("storage.proto"); + + println!("cargo:rerun-if-changed={}", storage_proto.display()); + for imported_proto in [ + "datamodel.proto", + "openshell.proto", + "options.proto", + "sandbox.proto", + ] { + println!( + "cargo:rerun-if-changed={}", + public_proto_dir.join(imported_proto).display() + ); + } + + // SAFETY: Build scripts run in their own single-threaded process. + #[allow(unsafe_code)] + unsafe { + env::set_var("PROTOC", protoc_bin_vendored::protoc_bin_path()?); + env::set_var("PROTOC_INCLUDE", protoc_bin_vendored::include_path()?); + } + + let descriptor_path = PathBuf::from(env::var("OUT_DIR")?).join("storage_descriptor.bin"); + tonic_prost_build::configure() + .build_server(false) + .build_client(false) + .extern_path(".openshell.v1", "::openshell_core::proto") + .extern_path( + ".openshell.datamodel.v1", + "::openshell_core::proto::datamodel::v1", + ) + .extern_path( + ".openshell.sandbox.v1", + "::openshell_core::proto::sandbox::v1", + ) + .file_descriptor_set_path(&descriptor_path) + .compile_protos(&[storage_proto], &[storage_proto_dir, public_proto_dir])?; + + Ok(()) +} diff --git a/crates/openshell-server/migrations/postgres/007_remove_inference_routes.sql b/crates/openshell-server/migrations/postgres/007_remove_inference_routes.sql new file mode 100644 index 000000000..291eeda39 --- /dev/null +++ b/crates/openshell-server/migrations/postgres/007_remove_inference_routes.sql @@ -0,0 +1,2 @@ +-- Managed inference routes were removed in favor of sandbox-scoped provider attachments. +DELETE FROM objects WHERE object_type = 'inference_route'; diff --git a/crates/openshell-server/migrations/sqlite/007_remove_inference_routes.sql b/crates/openshell-server/migrations/sqlite/007_remove_inference_routes.sql new file mode 100644 index 000000000..291eeda39 --- /dev/null +++ b/crates/openshell-server/migrations/sqlite/007_remove_inference_routes.sql @@ -0,0 +1,2 @@ +-- Managed inference routes were removed in favor of sandbox-scoped provider attachments. +DELETE FROM objects WHERE object_type = 'inference_route'; diff --git a/crates/openshell-server/proto/storage.proto b/crates/openshell-server/proto/storage.proto new file mode 100644 index 000000000..2ca7946f8 --- /dev/null +++ b/crates/openshell-server/proto/storage.proto @@ -0,0 +1,170 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +syntax = "proto3"; + +// Internal durable formats owned by the gateway. This package is not part of +// the public OpenShell API or any generated client SDK. +package openshell.storage.v1; + +import "datamodel.proto"; +import "openshell.proto"; +import "options.proto"; +import "sandbox.proto"; + +message StoredProviderCredentialRefreshState { + openshell.datamodel.v1.ObjectMeta metadata = 1; + string provider_id = 2; + string provider_name = 3; + string credential_key = 4; + openshell.v1.ProviderCredentialRefreshStrategy strategy = 5; + map material = 6 [(openshell.options.v1.secret) = true]; + // Material names classified as secret. Newly configured values live in the + // active credential driver and are absent from material. Legacy inline values + // are not automatically migrated before OpenShell 0.1.0. + repeated string secret_material_keys = 7; + int64 expires_at_ms = 8; + // int64 max parks the refresh until an explicit rotation or reconfiguration. + int64 next_refresh_at_ms = 9; + int64 last_refresh_at_ms = 10; + string status = 11; + string last_error = 12; + string token_url = 13; + repeated string scopes = 14; + int64 refresh_before_seconds = 15; + int64 max_lifetime_seconds = 16; + // Resolved mapping of strategy-defined output id -> concrete env key, pinned + // at configure time from the profile's additional_outputs. Read by minting, + // collision reservation, and env-key surfacing so later profile edits cannot + // silently redirect writes. + map additional_output_keys = 17; + // Opaque gateway-owned authorization epoch for the configured refresh + // grant. Explicit refresh configuration creates a new epoch; automatic and + // manual token rotation preserve it. It is never derived from or exposed + // with refresh material. + string authorization_epoch = 18; + // Secret refresh material is stored through the gateway's active credential + // driver. The persisted refresh state keeps only opaque handles; resolved + // values exist in gateway memory for the duration of one mint operation. + map secret_material_handles = 19; + // Handles replaced by reconfiguration or issuer-driven refresh-token + // rotation. This is a repeated entry rather than a material-keyed map so + // multiple superseded generations of the same material remain recoverable. + // Cleanup is retried by the refresh worker so a gateway crash or temporary + // credential-backend outage does not lose the deletion reference. + repeated StoredRefreshMaterialDeletion pending_secret_deletions = 20; + // Structured recovery details for the most recent refresh failure. These + // fields contain only gateway-owned codes and recognized bounded values. + openshell.v1.ProviderCredentialRefreshRecoveryAction recovery_action = 21; + string failure_code = 22; + string provider_error_subtype = 23; + int64 last_error_at_ms = 24; +} + +message StoredRefreshMaterialDeletion { + // Original material name used to derive the credential driver's storage key. + string material_key = 1; + // Opaque handle for the superseded secret object. + openshell.datamodel.v1.CredentialHandle handle = 2; +} + +// Stored custom provider profile object. +message StoredProviderProfile { + openshell.datamodel.v1.ObjectMeta metadata = 1; + openshell.v1.ProviderProfile profile = 2; +} + +// Stored payload for a policy revision row in the generic objects table. +message PolicyRevisionPayload { + // Serialized policy contents. + openshell.sandbox.v1.SandboxPolicy policy = 1; + // Deterministic hash of the policy payload. + string hash = 2; + // Load error reported by the sandbox, if any. + string load_error = 3; + // When the policy version was reported as loaded (ms since epoch). 0 if unset. + int64 loaded_at_ms = 4; + // Immutable provenance supplied when this revision was created. + map provenance = 5; +} + +// Stored payload for a draft policy chunk row in the generic objects table. +message DraftChunkPayload { + // Proposed network_policies map key. + string rule_name = 1; + // Proposed network policy rule. + openshell.sandbox.v1.NetworkPolicyRule proposed_rule = 2; + // Human-readable explanation of why this rule is proposed. + string rationale = 3; + // Security concerns flagged by analysis (empty if none). + string security_notes = 4; + // Analysis confidence (0.0-1.0). 0 for mechanistic mode. + float confidence = 5; + // When the user approved/rejected (ms since epoch). 0 if undecided. + int64 decided_at_ms = 6; + // Denormalized endpoint host for dedup and display. + string host = 7; + // Denormalized endpoint port for dedup and display. + int32 port = 8; + // Binary path that triggered the denial. + string binary = 9; + // Current draft version for the owning sandbox. + int64 draft_version = 10; + // Gateway prover verdict for this chunk; empty until prover runs. + // Mirrors PolicyChunk.validation_result. + string validation_result = 11; + // Operator-supplied free-form rejection text; empty for non-rejected + // chunks. Mirrors PolicyChunk.rejection_reason. + string rejection_reason = 12; + string application_error = 13; + string review_token = 14; + string current_effective_policy_hash = 15; + string candidate_effective_policy_hash = 16; + openshell.sandbox.v1.SandboxPolicy current_effective_policy = 17; + openshell.sandbox.v1.SandboxPolicy candidate_effective_policy = 18; +} + +// Internal stored policy revision row materialized from the generic objects table. +message StoredPolicyRevision { + string id = 1; + string sandbox_id = 2; + int64 version = 3; + bytes policy_payload = 4; + string policy_hash = 5; + string status = 6; + optional string load_error = 7; + int64 created_at_ms = 8; + optional int64 loaded_at_ms = 9; + map provenance = 10; +} + +// Internal stored draft chunk row materialized from the generic objects table. +message StoredDraftChunk { + string id = 1; + string sandbox_id = 2; + int64 draft_version = 3; + string status = 4; + string rule_name = 5; + bytes proposed_rule = 6; + string rationale = 7; + string security_notes = 8; + double confidence = 9; + int64 created_at_ms = 10; + optional int64 decided_at_ms = 11; + string host = 12; + int32 port = 13; + string binary = 14; + int32 hit_count = 15; + int64 first_seen_ms = 16; + int64 last_seen_ms = 17; + // Gateway prover verdict; empty until the prover runs. See PolicyChunk. + string validation_result = 18; + // Operator-supplied free-form rejection text. See PolicyChunk. + string rejection_reason = 19; + string application_error = 20; + string review_token = 21; + string current_effective_policy_hash = 22; + string candidate_effective_policy_hash = 23; + openshell.sandbox.v1.SandboxPolicy current_effective_policy = 24; + openshell.sandbox.v1.SandboxPolicy candidate_effective_policy = 25; +} diff --git a/crates/openshell-server/src/auth/compute_driver.rs b/crates/openshell-server/src/auth/compute_driver.rs index cedc2115b..04caee61b 100644 --- a/crates/openshell-server/src/auth/compute_driver.rs +++ b/crates/openshell-server/src/auth/compute_driver.rs @@ -52,7 +52,7 @@ impl Authenticator for ComputeDriverAuthenticator { Ok(Some(Principal::Sandbox(SandboxPrincipal { sandbox_id, source: SandboxIdentitySource::ComputeDriver { - driver_name: self.compute.selected_driver_name().to_string(), + driver_name: self.compute.configured_driver_name().to_string(), }, trust_domain: Some("openshell".to_string()), }))) diff --git a/crates/openshell-server/src/auth/descriptor_authz.rs b/crates/openshell-server/src/auth/descriptor_authz.rs index dbb9fa7ca..baea879bb 100644 --- a/crates/openshell-server/src/auth/descriptor_authz.rs +++ b/crates/openshell-server/src/auth/descriptor_authz.rs @@ -17,7 +17,7 @@ use super::method_authz::{AuthMode, Role}; const AUTHORIZATION_EXTENSION: &str = "openshell.options.v1.authorization"; /// Gateway-served protobuf packages. -const GATEWAY_PACKAGES: &[&str] = &["openshell.v1", "openshell.inference.v1"]; +const GATEWAY_PACKAGES: &[&str] = &["openshell.v1"]; /// Bearer-authenticated methods that deliberately require no role or scope. /// diff --git a/crates/openshell-server/src/auth/extension_mint_limit.rs b/crates/openshell-server/src/auth/extension_mint_limit.rs index 0495cf034..fc825404b 100644 --- a/crates/openshell-server/src/auth/extension_mint_limit.rs +++ b/crates/openshell-server/src/auth/extension_mint_limit.rs @@ -18,7 +18,7 @@ use std::collections::HashMap; use std::sync::Mutex; use std::time::{Duration, Instant}; -const DEFAULT_WINDOW: Duration = Duration::from_secs(60); +const DEFAULT_WINDOW: Duration = Duration::from_mins(1); const DEFAULT_MAX_PER_WINDOW: u32 = 10; /// Number of tracked sandboxes above which expired windows are pruned. @@ -102,7 +102,7 @@ mod tests { #[test] fn allows_a_burst_then_refuses_within_the_window() { - let limiter = ExtensionMintLimiter::new(Duration::from_secs(60), 3); + let limiter = ExtensionMintLimiter::new(Duration::from_mins(1), 3); let start = Instant::now(); for _ in 0..3 { @@ -113,17 +113,17 @@ mod tests { #[test] fn window_rollover_restores_capacity() { - let limiter = ExtensionMintLimiter::new(Duration::from_secs(60), 1); + let limiter = ExtensionMintLimiter::new(Duration::from_mins(1), 1); let start = Instant::now(); assert!(limiter.try_acquire_at("sandbox-a", start)); assert!(!limiter.try_acquire_at("sandbox-a", start + Duration::from_secs(59))); - assert!(limiter.try_acquire_at("sandbox-a", start + Duration::from_secs(60))); + assert!(limiter.try_acquire_at("sandbox-a", start + Duration::from_mins(1))); } #[test] fn sandboxes_are_limited_independently() { - let limiter = ExtensionMintLimiter::new(Duration::from_secs(60), 1); + let limiter = ExtensionMintLimiter::new(Duration::from_mins(1), 1); let start = Instant::now(); assert!(limiter.try_acquire_at("sandbox-a", start)); @@ -147,7 +147,7 @@ mod tests { #[test] fn a_zero_bound_disables_limiting() { - let limiter = ExtensionMintLimiter::new(Duration::from_secs(60), 0); + let limiter = ExtensionMintLimiter::new(Duration::from_mins(1), 0); let start = Instant::now(); for _ in 0..1_000 { assert!(limiter.try_acquire_at("sandbox-a", start)); diff --git a/crates/openshell-server/src/auth/method_authz.rs b/crates/openshell-server/src/auth/method_authz.rs index 71eb7acac..35ff82e92 100644 --- a/crates/openshell-server/src/auth/method_authz.rs +++ b/crates/openshell-server/src/auth/method_authz.rs @@ -139,9 +139,6 @@ mod tests { "/openshell.v1.OpenShell/ConnectSupervisor" )); assert!(!is_user_callable("/openshell.v1.OpenShell/RelayStream")); - assert!(!is_user_callable( - "/openshell.inference.v1.Inference/GetInferenceBundle" - )); // Unauthenticated methods are not "user callable" — they're // intercepted before principal evaluation. assert!(!is_user_callable("/openshell.v1.OpenShell/Health")); diff --git a/crates/openshell-server/src/auth/oidc.rs b/crates/openshell-server/src/auth/oidc.rs index 475c52ebe..303dcb3c5 100644 --- a/crates/openshell-server/src/auth/oidc.rs +++ b/crates/openshell-server/src/auth/oidc.rs @@ -18,12 +18,13 @@ use jsonwebtoken::{Algorithm, DecodingKey, Validation, decode, decode_header}; use openshell_core::OidcConfig; use reqwest::Client; use serde::Deserialize; -use std::collections::HashMap; +use std::collections::{HashMap, HashSet}; +use std::str::FromStr; use std::sync::Arc; use std::time::{Duration, Instant}; use tokio::sync::RwLock; use tonic::Status; -use tracing::{debug, info, warn}; +use tracing::{debug, error, info, warn}; /// Path prefixes that bypass OIDC validation (gRPC reflection, health probes). /// @@ -40,16 +41,34 @@ pub fn is_unauthenticated_method(path: &str) -> bool { .any(|prefix| path.starts_with(prefix)) } +/// How long previously-cached keys stay trusted after a refresh starts +/// returning zero usable keys, expressed as a multiple of the configured +/// TTL. Bounds exposure to keys the issuer may have deliberately revoked, +/// while still tolerating a transient empty or bad JWKS response. +const STALE_KEY_GRACE_MULTIPLIER: u32 = 3; + +/// Minimum interval between kid-miss-triggered refreshes. Bounds how much +/// a burst or steady stream of unknown-`kid` lookups can amplify requests +/// against the issuer's JWKS endpoint. +const KID_MISS_REFRESH_COOLDOWN: Duration = Duration::from_secs(1); + /// Cached JWKS key set fetched from the OIDC issuer. /// /// A `refresh_mutex` ensures that only one refresh runs at a time, /// preventing a "thundering herd" when the TTL expires or a new `kid` /// is encountered under concurrent load. pub struct JwksCache { - keys: Arc>>, + keys: Arc>>, jwks_uri: String, ttl: Duration, last_refresh: Arc>, + /// Timestamp of the last refresh that yielded at least one usable key. + /// Bounds how long a fully-empty or fully-unusable JWKS response can + /// keep serving stale cached keys before they're evicted. + last_nonempty_refresh: Arc>, + /// Timestamp of the last refresh triggered by an unknown `kid`, as + /// opposed to the regular TTL cadence. See `KID_MISS_REFRESH_COOLDOWN`. + last_kid_miss_refresh: Arc>, /// Serializes JWKS refresh operations so concurrent requests coalesce /// into a single HTTP fetch rather than stampeding the OIDC provider. refresh_mutex: tokio::sync::Mutex<()>, @@ -88,6 +107,218 @@ struct JwkKey { n: String, #[serde(default)] e: String, + #[serde(default)] + x: String, + #[serde(default)] + y: String, + #[serde(default)] + crv: String, + #[serde(default)] + alg: Option, + #[serde(rename = "use", default)] + use_: Option, + #[serde(default)] + key_ops: Vec, +} + +enum SkipReason { + MissingKid, + MissingComponents, + UnsupportedKeyType { + kty: String, + crv: Option, + }, + ParseError(String), + AlgMismatch { + declared: String, + derived: Algorithm, + }, + UseEnc, + UseNotSig, + KeyOpsNotForVerify, +} + +/// The RSA signature algorithms `jsonwebtoken` supports. Used to decide +/// whether a JWK's declared `alg` is a legitimate RSA algorithm selection +/// (see the RSA arm of `parse_jwk`) as opposed to an absent, unparseable, or +/// non-RSA value. +fn is_rsa_algorithm(algorithm: Algorithm) -> bool { + matches!( + algorithm, + Algorithm::RS256 + | Algorithm::RS384 + | Algorithm::RS512 + | Algorithm::PS256 + | Algorithm::PS384 + | Algorithm::PS512 + ) +} + +/// Log why a JWK was excluded from the cache, with the fields relevant to +/// diagnosing that specific `SkipReason`. +fn warn_skip(key: &JwkKey, reason: SkipReason) { + match reason { + SkipReason::MissingKid => { + warn!(kty = %key.kty, crv = %key.crv, "Skipping JWK without kid"); + } + SkipReason::MissingComponents => { + warn!( + kid = ?key.kid, + kty = %key.kty, + crv = %key.crv, + "Skipping JWK with missing key components" + ); + } + SkipReason::UnsupportedKeyType { kty, crv } => { + warn!(kid = ?key.kid, kty = %kty, crv = ?crv, "Skipping unsupported JWK key type"); + } + SkipReason::ParseError(error) => { + warn!(kid = ?key.kid, kty = %key.kty, error = %error, "Failed to parse JWK"); + } + SkipReason::AlgMismatch { declared, derived } => { + warn!( + kid = ?key.kid, + kty = %key.kty, + declared_alg = %declared, + derived_alg = ?derived, + "Skipping JWK with alg mismatch" + ); + } + SkipReason::UseEnc => { + warn!(kid = ?key.kid, kty = %key.kty, "Skipping encryption JWK (use: enc)"); + } + SkipReason::UseNotSig => { + warn!( + kid = ?key.kid, + kty = %key.kty, + use_ = ?key.use_, + "Skipping JWK with use not suitable for verify" + ); + } + SkipReason::KeyOpsNotForVerify => { + warn!(kid = ?key.kid, kty = %key.kty, "Skipping JWK with key_ops not suitable for verify"); + } + } +} + +/// Validate and decode a single JWK, returning its `kid`, decoding key, and +/// pinned algorithm, or the reason it must be excluded from the cache. +fn parse_jwk(key: &JwkKey) -> Result<(String, DecodingKey, Algorithm), SkipReason> { + // Must run before any early return below, not just before the + // `DecodingKey::from_*` calls further down: a JWKS response whose keys + // are all skipped by these checks would otherwise leave the process-wide + // crypto provider never installed on this path, and a caller reaching + // its own first `jsonwebtoken` operation without having installed one + // itself would panic. + crate::install_jsonwebtoken_crypto_provider(); + + // Subsumed by the `use_ != "sig"` check below (`"enc" != "sig"`); kept + // separate only to produce a more specific log line for the common + // `use: enc` case. + if key.use_.as_deref() == Some("enc") { + return Err(SkipReason::UseEnc); + } + if let Some(use_) = key.use_.as_deref().filter(|u| !u.is_empty()) + && use_ != "sig" + { + return Err(SkipReason::UseNotSig); + } + + // The gateway only ever verifies signatures, never creates them, so a + // key advertising `key_ops` must include "verify" — mirroring the + // strictness of the `use: "sig"` check above. `["sign"]` alone is + // semantically incoherent for a public verification key (RFC 7517 §4.3 + // treats "sign" and "verify" as distinct operations). + if !key.key_ops.is_empty() && !key.key_ops.iter().any(|op| op == "verify") { + return Err(SkipReason::KeyOpsNotForVerify); + } + + let kid = key.kid.as_deref().filter(|k| !k.is_empty()); + let Some(kid) = kid else { + return Err(SkipReason::MissingKid); + }; + + let (decoding_key, algorithm) = match key.kty.as_str() { + "RSA" => { + if key.n.is_empty() || key.e.is_empty() { + return Err(SkipReason::MissingComponents); + } + let dk = DecodingKey::from_rsa_components(&key.n, &key.e) + .map_err(|e| SkipReason::ParseError(e.to_string()))?; + // Select the algorithm from the JWK's declared `alg` when it + // names one of the RSA family; default to RS256 otherwise (most + // issuers, e.g. Microsoft Entra ID, omit `alg` entirely). The + // mismatch check below still rejects genuinely contradictory + // declarations (e.g. an RSA key declaring "ES256"). + let algorithm = key + .alg + .as_deref() + .and_then(|declared| Algorithm::from_str(declared).ok()) + .filter(|declared| is_rsa_algorithm(*declared)) + .unwrap_or(Algorithm::RS256); + (dk, algorithm) + } + "EC" => { + if key.x.is_empty() || key.y.is_empty() || key.crv.is_empty() { + return Err(SkipReason::MissingComponents); + } + let algorithm = match key.crv.as_str() { + "P-256" => Algorithm::ES256, + "P-384" => Algorithm::ES384, + _ => { + return Err(SkipReason::UnsupportedKeyType { + kty: key.kty.clone(), + crv: Some(key.crv.clone()), + }); + } + }; + let dk = DecodingKey::from_ec_components(&key.x, &key.y) + .map_err(|e| SkipReason::ParseError(e.to_string()))?; + (dk, algorithm) + } + "OKP" => { + if key.x.is_empty() || key.crv.is_empty() { + return Err(SkipReason::MissingComponents); + } + if key.crv != "Ed25519" { + return Err(SkipReason::UnsupportedKeyType { + kty: key.kty.clone(), + crv: Some(key.crv.clone()), + }); + } + let dk = DecodingKey::from_ed_components(&key.x) + .map_err(|e| SkipReason::ParseError(e.to_string()))?; + (dk, Algorithm::EdDSA) + } + other => { + return Err(SkipReason::UnsupportedKeyType { + kty: other.to_string(), + crv: if key.crv.is_empty() { + None + } else { + Some(key.crv.clone()) + }, + }); + } + }; + + if let Some(ref declared) = key.alg { + match Algorithm::from_str(declared) { + Ok(parsed) if parsed == algorithm => {} + // Either the declared algorithm genuinely contradicts the + // derived one, or it's an unrecognized string — in both cases + // we can't be sure the key is safe to use, so skip it rather + // than silently accepting our own default. + _ => { + return Err(SkipReason::AlgMismatch { + declared: declared.clone(), + derived: algorithm, + }); + } + } + } + + Ok((kid.to_string(), decoding_key, algorithm)) } /// Claims extracted from a validated JWT. @@ -168,6 +399,15 @@ impl JwksCache { /// Create a new JWKS cache, discovering the JWKS URI and fetching the /// initial key set. pub async fn new(config: &OidcConfig) -> Result { + if config.jwks_ttl_secs == 0 { + return Err( + "jwks_ttl_secs must be greater than zero (0 would refresh on every request and \ + collapse the stale-key grace period to zero)" + .to_string(), + ); + } + + let _ = rustls::crypto::aws_lc_rs::default_provider().install_default(); let http = Client::builder() .timeout(Duration::from_secs(10)) .build() @@ -209,6 +449,18 @@ impl JwksCache { .checked_sub(Duration::from_secs(config.jwks_ttl_secs + 1)) .unwrap_or_else(Instant::now), )), + // Placeholder value: the grace-period check is only reachable + // once `self.keys` has been non-empty, at which point this + // field always holds a freshly-stamped real value instead (see + // `refresh_keys`), so the value here is never actually read. + last_nonempty_refresh: Arc::new(RwLock::new(Instant::now())), + // Backdated beyond the cooldown so the very first kid-miss + // refresh is never throttled. + last_kid_miss_refresh: Arc::new(RwLock::new( + Instant::now() + .checked_sub(KID_MISS_REFRESH_COOLDOWN + Duration::from_secs(1)) + .unwrap_or_else(Instant::now), + )), refresh_mutex: tokio::sync::Mutex::new(()), http, config: config.clone(), @@ -233,26 +485,76 @@ impl JwksCache { .map_err(|e| format!("JWKS parse failed: {e}"))?; let mut new_keys = HashMap::new(); + let mut poisoned_kids = HashSet::new(); + let total = jwk_set.keys.len(); + for key in &jwk_set.keys { - if key.kty != "RSA" { + if let Some(kid) = key.kid.as_ref().filter(|k| !k.is_empty()) + && poisoned_kids.contains(kid) + { continue; } - let Some(ref kid) = key.kid else { - continue; - }; - crate::install_jsonwebtoken_crypto_provider(); - match DecodingKey::from_rsa_components(&key.n, &key.e) { - Ok(dk) => { - new_keys.insert(kid.clone(), dk); - } - Err(e) => { - warn!(kid = %kid, error = %e, "Failed to parse JWK"); + + match parse_jwk(key) { + Ok((kid, dk, alg)) => { + if let Some((_, existing_alg)) = new_keys.get(&kid) { + if *existing_alg == alg { + warn!( + kid = %kid, + algorithm = ?alg, + "Duplicate JWK kid, keeping first entry" + ); + continue; + } + let existing_alg = *existing_alg; + new_keys.remove(&kid); + poisoned_kids.insert(kid.clone()); + warn!( + kid = %kid, + existing_alg = ?existing_alg, + conflicting_alg = ?alg, + "Duplicate JWK kid with conflicting algorithms, poison-pilling kid" + ); + continue; + } + new_keys.insert(kid, (dk, alg)); } + Err(reason) => warn_skip(key, reason), } } - info!(count = new_keys.len(), "JWKS keys loaded"); - *self.keys.write().await = new_keys; + if new_keys.is_empty() { + // A refresh with zero usable keys must not immediately wipe a + // working cache, since that would turn a degraded issuer into a + // total outage. Keys stay trusted until a later refresh + // succeeds or the grace period expires. With no existing keys + // to fall back to (the first load, or after eviction), the + // empty set is committed immediately. + let grace = self.ttl.saturating_mul(STALE_KEY_GRACE_MULTIPLIER); + if self.keys.read().await.is_empty() { + error!( + total, + "JWKS has zero usable signing keys and no previous keys to fall back to" + ); + *self.keys.write().await = new_keys; + } else if self.last_nonempty_refresh.read().await.elapsed() > grace { + error!( + total, + grace_secs = grace.as_secs(), + "JWKS has had zero usable signing keys past the grace period; evicting cached keys" + ); + *self.keys.write().await = new_keys; + } else { + error!( + total, + "JWKS refresh loaded zero usable signing keys; keeping previous key set within grace period" + ); + } + } else { + info!(count = new_keys.len(), "JWKS keys loaded"); + *self.keys.write().await = new_keys; + *self.last_nonempty_refresh.write().await = Instant::now(); + } *self.last_refresh.write().await = Instant::now(); Ok(()) } @@ -276,9 +578,20 @@ impl JwksCache { self.refresh_keys().await } - /// Refresh keys unconditionally, coalescing concurrent callers. + /// Refresh keys on a kid-miss lookup, coalescing concurrent callers and + /// throttling how often such refreshes may hit the issuer. + /// + /// Without a cooldown, a caller presenting an unknown or fast-rotating + /// `kid` could force a fresh HTTP fetch on every single request. Only + /// the first kid-miss refresh within `KID_MISS_REFRESH_COOLDOWN` does + /// real work; the rest reuse its result. async fn refresh_keys_coalesced(&self) -> Result<(), String> { let _guard = self.refresh_mutex.lock().await; + let last = *self.last_kid_miss_refresh.read().await; + if last.elapsed() < KID_MISS_REFRESH_COOLDOWN { + return Ok(()); + } + *self.last_kid_miss_refresh.write().await = Instant::now(); self.refresh_keys().await } @@ -305,28 +618,39 @@ impl JwksCache { Status::unauthenticated("invalid token: missing kid") })?; - // Look up the key in cache. - let keys = self.keys.read().await; - let decoding_key = if let Some(k) = keys.get(&kid) { - k.clone() - } else { - // Key not found -- try refreshing once (key rotation). - drop(keys); + // Refresh once on kid miss. + let mut keys_guard = self.keys.read().await; + if !keys_guard.contains_key(&kid) { + drop(keys_guard); self.refresh_keys_coalesced().await.map_err(|e| { warn!(error = %e, "JWKS refresh on kid miss failed"); Status::internal("OIDC key refresh failed") })?; - let keys = self.keys.read().await; - keys.get(&kid).cloned().ok_or_else(|| { + keys_guard = self.keys.read().await; + } + + let (decoding_key, cached_algorithm) = keys_guard + .get(&kid) + .map(|(dk, alg)| (dk.clone(), *alg)) + .ok_or_else(|| { debug!(kid = %kid, "JWT kid not found in JWKS"); Status::unauthenticated("invalid token: unknown signing key") - })? - }; + })?; + drop(keys_guard); - // Validate the JWT. - let mut validation = Validation::new(Algorithm::RS256); + if header.alg != cached_algorithm { + debug!( + header_alg = ?header.alg, + expected = ?cached_algorithm, + "JWT algorithm mismatch for cached signing key" + ); + return Err(Status::unauthenticated("invalid token")); + } + + let mut validation = Validation::new(cached_algorithm); validation.set_issuer(&[&self.config.issuer]); validation.set_audience(&[&self.config.audience]); + validation.set_required_spec_claims(&["iss", "aud", "exp", "sub"]); let token_data = decode::(token, &decoding_key, &validation).map_err(|e| { debug!(error = %e, "JWT validation failed"); @@ -768,4 +1092,716 @@ mod tests { .await .expect_err("a token naming a kid absent from the JWKS must be rejected"); } + + mod jwks_validation { + use super::*; + use base64::Engine; + use base64::engine::general_purpose::URL_SAFE_NO_PAD; + use jsonwebtoken::{EncodingKey, Header, encode}; + use openshell_core::OidcConfig; + use rcgen::{KeyPair, PKCS_ECDSA_P256_SHA256, PKCS_ECDSA_P384_SHA384, PKCS_ED25519}; + use rsa::traits::PublicKeyParts; + use std::time::{SystemTime, UNIX_EPOCH}; + use wiremock::matchers::{method, path}; + use wiremock::{Mock, MockServer, ResponseTemplate}; + + struct TestSigningKey { + kid: String, + jwk: serde_json::Value, + encoding_key: EncodingKey, + algorithm: Algorithm, + } + + fn now_secs() -> i64 { + i64::try_from( + SystemTime::now() + .duration_since(UNIX_EPOCH) + .unwrap() + .as_secs(), + ) + .unwrap() + } + + fn ec_coords_from_spki(spki: &[u8], coord_len: usize) -> (String, String) { + // The uncompressed EC point (a 0x04 tag byte followed by X and Y) + // is always the tail of an SPKI DER encoding, so slice it + // directly at the known offset instead of scanning for the tag + // byte (which could false-match a coordinate byte at the wrong + // offset). + let point_len = 1 + 2 * coord_len; + let point = &spki[spki.len() - point_len..]; + assert_eq!(point[0], 0x04, "expected uncompressed EC point marker"); + let x = URL_SAFE_NO_PAD.encode(&point[1..=coord_len]); + let y = URL_SAFE_NO_PAD.encode(&point[1 + coord_len..point_len]); + (x, y) + } + + // Precomputed 2048-bit RSA keys (PKCS8 PEM). Generating a fresh RSA + // key per test call site is the dominant cost of this test module + // (~7s across ~15 call sites); two static fixtures are deterministic + // and effectively free to parse. Two distinct keys are kept because + // a couple of tests (key rotation, duplicate-kid poisoning) need + // genuinely different key material, not just different `kid`s. + const RSA_TEST_KEY_PEM_PRIMARY: &str = "-----BEGIN PRIVATE KEY-----\nMIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQDFDsXUQjvihWzC\nA8UfdQ+QiQrAt8RBEMXXxiYHTn+EwDl4umH+OFURRGIlGgZ8cFrewKDotaJylEal\nnRq+urYCS3JjoerIS1lc9XZFp5ri7XB+5fB0bZQPNX4KIJ7MlbP2XG0zqvawww1A\nZQ4w9Ni8X+pptB7QlGWldALlNiOPvlVNaRFRzvY4wc2JqwlWj0ZD/5fqVaO/7oa6\nyyKuZzXQ3ps0PMbt7sz9FcyMEgkQCm0eNIOancjeQKhFNFh5VN71PhyfnhN9Xkwn\n0vaV24SRrOurLZAH4vhum0aDQLFC0yDVCKEfwW1Wl7Cs0ns6kS/IbG+RPBGH4Snr\nRoI9OaiLAgMBAAECggEAWOS9IW9vjFQcJ7mDpxkrmEv56c38Xk2ushPU+97Rb5U3\nV9rccc3/sfZjP9Fps6ELnQjQjanCSmXRKMyiT//yMz7Nr1xPiWNUQLcKT4m4OT5b\nTSN1QVBdRi8fWHo2qJuvvycarAAnoL2csLvllvgc/X1XRa/XZshKwkR/Od8eU60C\nMHuCmks+NCpUL/lMIZecAbWFyQDYJqwB/lnJdEO5oDaIcGclAOQJzj3+xCfFnLLh\nYt/7tCxOboIeyP/fLh5OyqC6NppRUSwtfWNQNwgQWw7WWVd5TW9d2wjycgQKrm6P\n1tnbs8yRJ2h8jemJR65OukW1QOfdN7QFWBAoD8BEwQKBgQDiIXhmre2qw/d4aOZV\nqDzjlHFCAG9XCAWF8pio2Uy0SUXAI4CcK6d+Njm/U7CaKNUfbNyip0iv3YoADwAv\nDlI6kqo1DOGXYgo2z0zfmxWTixQsCvZHtLrSk4Z1sxiSlFXfbTsR3vBd7qLXgKQ/\nWR9b2chPYlGew8DBRTNa0aQ/KQKBgQDfFjVKMibXSi69x704LQ9QoO0SR+pgdyw6\nJ5auf0N3ACqIOTqdD001Dn1zNO35RdTbV3vYDCq24yED5xro+fpSSbZ2+ZpMjhEk\nZjLGSlTSwsjzXx4l41f8wX7Br5OgTjA9B23NNDTf/RaD+ui5mfVA3BJyoXMkHCC2\nIV5ZZt7EkwKBgHu8Xtqor5UymDaOCAO1BGRvdK3t+P7Bh+wsvDYgeaVpNr6VbqmG\nBae9WkoELG2ejEge1Hg4W0DIU9wGWU5mYr5kRLi0rLieUAJ/2ou8m8jZYJddBDhm\nf5f8W6YJ8xc6DectKRZ1TEfJ7ddIMBft14f2GnK91PWwHchj6l72ug5JAoGAd1Hc\njOPILIyL9YvY5CwNrfV097spXBFBwZUdHhYJkqOvHA9oD0t44zDt3mnoAtTb5bmk\nDslrK0jOhtTcatIRlmPAyV/1rI6sEojrDW4CcnwmmS095cv0asdfsd7kGfDYEjxf\n+Uq8ITWwDkVspqD3MYrD/zXlbOHyiRfN7Al+iysCgYEA4Q8+8d5UwF1jzUQFmSEt\nMjlGyycrLpzadgWTG9EjQN0FuSJFK8nyEyqcfzFm+rSsReZ37JwPTCQ/Ck4g4IXY\n3VvHY+6wj0NYwRwjKCZeqN8IAANsj5Xsh5pz4ETSwnwd0bUlGP6Bcqoc5edLnB85\nEMcbZoGCXLCoi/qf8T334k4=\n-----END PRIVATE KEY-----\n"; + const RSA_TEST_KEY_PEM_SECONDARY: &str = "-----BEGIN PRIVATE KEY-----\nMIIEvAIBADANBgkqhkiG9w0BAQEFAASCBKYwggSiAgEAAoIBAQCnRp9lR8xbtsXp\nCTTBkyAkhYji4Hp5FnAXxZF5DGfyw3I9ol0J8Hoe9RSBkQVIKXQe5Gfi+L7o9G6v\nVcucrzILxSJ4dCPKgNaj53G4zwkySpTsVO0XjxKanozQqdRbkAGaz9AsuzfGWB9m\n1AiC0JxKmOo8Ifxi4oj0hIBV4Mp3TCfHX8vFCIn5AZ/31Y29vT1mPzoCxwGw4Yxd\nh4kYQgIIWoAekebWeJ8iowgOvbV8rmT1geMWu7JRdpcStN9PWVZKUiZplILz9ci2\n+owm8UyxdQvBt6csEJFNdbAGUR1/Ywv/unoTQfrPvwQ4of5viywRra+iCMM0SrSA\nUO6Fu5a/AgMBAAECggEATDSTzD+73XJ0Ujhz9NYSeCDvnjBXC1AKDAJhRiy9NG8O\n3f5YdX09HVpYl7haGChuct5qZ5Ab5SPqQu2Kn5x+57bM/+QlJA2y+yOm/uMvFN6+\nXrZH9woilxcxHqSoDniaCo2vEJnQDIe78owZPoNMGH32hCOVh/UdIIw2rSkGA/eM\nYHHQXxx3sbi0AxCnlDzkgST15igP1baAZKUM2vIHVdg2Ccx2CnUiLM5BgvlWd98f\nS6GmTMk4WUWvmYYFZx+5GEhMmgz64xnyjIe+tRT1iP4p5NaCZc2CWLQdycEaEeKp\nmWFjxe7t/eXLUE3h1BBf/qZvcekpYbDGjw8JSaa00QKBgQDcctFl9P4npQUkoq0M\nDWeKSITsSwOVNKBLrQv5C4inz7QGPglGgIF8G6O7mvQ9ZulzDsvWfzxS0JE6ADkU\nHAkMY4/6Rfm7XKk8Yg1QoxCqsoI85nVQOhsmvVmIwXBDdINMY+g9jez9psiiiQiC\nh0nt3n9LPKu/YCzve50YTvR/bwKBgQDCQJSmZTFVsTC29e0CCgS1Zp9z4IAcZgth\nEJsaenE/ksQriH8gl3I4bwfI1p/xquBeWO40A016T67laCUBPTCwuZTGyqUiOtAw\nV5CoUiCvu38lNW6dk/RgouR0WoIQVvKq+5NKcnQtwbDHCt4U+hnB8c6EbT8dG2XU\noAamLcq1sQKBgADYI7srPAn01Nc2FEmWh439Bx1MkD/zCqYfjIswox5ZakwX0rtF\nZLmP9YmTZ1oQ2dYJ+Xfh1t5OVDAPrihIjzRP8U45FGLGUROdIIXtifPNaThIfayH\n/HCiiwQ+EWsAuDwDqfEKaRzzlZMhyTmOwRa7ImusWNAL00A7jfd43fDbAoGAa5u8\n/USXhOIIm4I2zmdgXmFAOcAHGDRLX3UEhzGHJPGX7InL6vEanDqdtFt49TZ03q8j\nHfsqY3Ra7ci4nywXmf7kdQ9zVTgBdpY7k5MTemZCtAkagv6gZRw3tGEjJgwUmDWP\nTbGDvIlM9aaGilZWCIN8pQ2j5er0iUoxBMPfRLECgYBu8/kcgM/sEeYh5vN5QqAh\n3/ezl+xboYDGGY1SLRwAzEsyIm9lyvvrFc+zGLpHqIP7jtrnu7CTqjxX0FQ2ECnd\nLIqNx0WMFZPq+GAD0hyUpYcqGFpHLGZGIOm8YhQUYvUL6ZWU4UQaND6gEq4ZYk9g\nk4dfOCu/AeloLlC3LUC9fw==\n-----END PRIVATE KEY-----\n"; + + enum RsaFixture { + Primary, + Secondary, + } + + /// Build an RSA `TestSigningKey` from a precomputed fixture, with an + /// optional declared `alg` and an explicit signing algorithm — used + /// by tests that exercise algorithm selection from the JWK's `alg` + /// field (RS256 default vs. a declared RS384/RS512/etc). + fn rsa_test_key_from( + kid: &str, + fixture: RsaFixture, + declared_alg: Option<&str>, + sign_algorithm: Algorithm, + ) -> TestSigningKey { + use rsa::pkcs8::DecodePrivateKey; + + let pem = match fixture { + RsaFixture::Primary => RSA_TEST_KEY_PEM_PRIMARY, + RsaFixture::Secondary => RSA_TEST_KEY_PEM_SECONDARY, + }; + let private_key = + rsa::RsaPrivateKey::from_pkcs8_pem(pem).expect("valid static test RSA key"); + let encoding_key = EncodingKey::from_rsa_pem(pem.as_bytes()).unwrap(); + let public_key = private_key.to_public_key(); + let n = URL_SAFE_NO_PAD.encode(public_key.n().to_bytes_be()); + let e = URL_SAFE_NO_PAD.encode(public_key.e().to_bytes_be()); + let mut jwk = serde_json::json!({ + "kty": "RSA", + "kid": kid, + "n": n, + "e": e, + "use": "sig", + }); + if let Some(alg) = declared_alg { + jwk["alg"] = serde_json::json!(alg); + } + TestSigningKey { + kid: kid.to_string(), + jwk, + encoding_key, + algorithm: sign_algorithm, + } + } + + fn rsa_test_key(kid: &str) -> TestSigningKey { + rsa_test_key_from(kid, RsaFixture::Primary, None, Algorithm::RS256) + } + + /// Distinct key material from `rsa_test_key`, for tests that need + /// two genuinely different RSA keys (e.g. key rotation, duplicate + /// `kid` poisoning) rather than just different `kid` labels. + fn rsa_test_key_secondary(kid: &str) -> TestSigningKey { + rsa_test_key_from(kid, RsaFixture::Secondary, None, Algorithm::RS256) + } + + fn ec_test_key(kid: &str, curve: &str) -> TestSigningKey { + let (key_pair, algorithm, coord_len) = match curve { + "P-256" => ( + KeyPair::generate_for(&PKCS_ECDSA_P256_SHA256).unwrap(), + Algorithm::ES256, + 32, + ), + "P-384" => ( + KeyPair::generate_for(&PKCS_ECDSA_P384_SHA384).unwrap(), + Algorithm::ES384, + 48, + ), + other => panic!("unsupported curve {other}"), + }; + let encoding_key = + EncodingKey::from_ec_pem(key_pair.serialize_pem().as_bytes()).unwrap(); + let (x, y) = ec_coords_from_spki(&key_pair.public_key_der(), coord_len); + let jwk = serde_json::json!({ + "kty": "EC", + "kid": kid, + "crv": curve, + "x": x, + "y": y, + "use": "sig", + }); + TestSigningKey { + kid: kid.to_string(), + jwk, + encoding_key, + algorithm, + } + } + + fn ed25519_test_key(kid: &str) -> TestSigningKey { + let key_pair = KeyPair::generate_for(&PKCS_ED25519).unwrap(); + let encoding_key = + EncodingKey::from_ed_pem(key_pair.serialize_pem().as_bytes()).unwrap(); + let spki = key_pair.public_key_der(); + let x = URL_SAFE_NO_PAD.encode(&spki[spki.len() - 32..]); + let jwk = serde_json::json!({ + "kty": "OKP", + "kid": kid, + "crv": "Ed25519", + "x": x, + "use": "sig", + }); + TestSigningKey { + kid: kid.to_string(), + jwk, + encoding_key, + algorithm: Algorithm::EdDSA, + } + } + + fn token_with_header_alg(token: &str, algorithm: Algorithm) -> String { + let mut parts: Vec = token.split('.').map(str::to_string).collect(); + assert_eq!(parts.len(), 3); + let header_bytes = URL_SAFE_NO_PAD.decode(&parts[0]).unwrap(); + let mut header: serde_json::Value = serde_json::from_slice(&header_bytes).unwrap(); + header["alg"] = serde_json::json!(match algorithm { + Algorithm::HS256 => "HS256", + other => panic!("unsupported test algorithm {other:?}"), + }); + parts[0] = URL_SAFE_NO_PAD.encode(serde_json::to_vec(&header).unwrap()); + parts.join(".") + } + + fn sign_token(key: &TestSigningKey, issuer: &str, audience: &str, sub: &str) -> String { + let mut header = Header::new(key.algorithm); + header.kid = Some(key.kid.clone()); + let claims = serde_json::json!({ + "sub": sub, + "iss": issuer, + "aud": audience, + "exp": now_secs() + 3600, + }); + encode(&header, &claims, &key.encoding_key).unwrap() + } + + fn test_oidc_config(issuer: &str) -> OidcConfig { + OidcConfig { + issuer: issuer.to_string(), + audience: "test-audience".to_string(), + jwks_ttl_secs: 3600, + roles_claim: "roles".to_string(), + admin_role: "admin".to_string(), + user_role: "user".to_string(), + scopes_claim: String::new(), + } + } + + async fn mount_oidc(server: &MockServer, jwks: serde_json::Value) -> (String, OidcConfig) { + let issuer = format!("{}/issuer", server.uri()); + let jwks_uri = format!("{issuer}/jwks"); + Mock::given(method("GET")) + .and(path("/issuer/.well-known/openid-configuration")) + .respond_with(ResponseTemplate::new(200).set_body_json(serde_json::json!({ + "issuer": issuer, + "jwks_uri": jwks_uri, + }))) + .mount(server) + .await; + Mock::given(method("GET")) + .and(path("/issuer/jwks")) + .respond_with(ResponseTemplate::new(200).set_body_json(jwks)) + .mount(server) + .await; + (issuer.clone(), test_oidc_config(&issuer)) + } + + async fn cache_from_jwks( + jwks: serde_json::Value, + ) -> (JwksCache, String, OidcConfig, MockServer) { + let server = MockServer::start().await; + let (issuer, config) = mount_oidc(&server, jwks).await; + let cache = JwksCache::new(&config).await.unwrap(); + // Keep the pooled wiremock server checked out until the caller has + // finished using the cache. A kid miss can trigger another JWKS + // request after this helper returns. + (cache, issuer, config, server) + } + + #[tokio::test] + async fn validate_rsa_es256_es384_eddsa_tokens() { + let rsa = rsa_test_key("rsa-key"); + let es256 = ec_test_key("es256-key", "P-256"); + let es384 = ec_test_key("es384-key", "P-384"); + let eddsa = ed25519_test_key("eddsa-key"); + let jwks = serde_json::json!({ + "keys": [rsa.jwk, es256.jwk, es384.jwk, eddsa.jwk] + }); + let (cache, issuer, config, _server) = cache_from_jwks(jwks).await; + + for (name, key) in [ + ("rsa", &rsa), + ("es256", &es256), + ("es384", &es384), + ("eddsa", &eddsa), + ] { + let token = sign_token(key, &issuer, &config.audience, "user-1"); + let result = cache.validate_token(&token).await; + assert!(result.is_ok(), "validation failed for {name}: {result:?}"); + assert_eq!(result.unwrap().subject, "user-1"); + } + } + + #[tokio::test] + async fn unsupported_curve_is_skipped() { + let signing = rsa_test_key("good"); + let jwks = serde_json::json!({ + "keys": [ + { "kty": "EC", "kid": "p521", "crv": "P-521", "x": "abc", "y": "def" }, + signing.jwk, + ] + }); + let (cache, issuer, config, _server) = cache_from_jwks(jwks).await; + let token = sign_token(&signing, &issuer, &config.audience, "user-1"); + assert!(cache.validate_token(&token).await.is_ok()); + } + + #[tokio::test] + async fn kid_miss_triggers_refresh() { + let initial = rsa_test_key("initial"); + let rotated = rsa_test_key_secondary("rotated"); + let server = MockServer::start().await; + let issuer = format!("{}/issuer", server.uri()); + let jwks_uri = format!("{issuer}/jwks"); + Mock::given(method("GET")) + .and(path("/issuer/.well-known/openid-configuration")) + .respond_with(ResponseTemplate::new(200).set_body_json(serde_json::json!({ + "issuer": issuer, + "jwks_uri": jwks_uri, + }))) + .mount(&server) + .await; + + Mock::given(method("GET")) + .and(path("/issuer/jwks")) + .respond_with(ResponseTemplate::new(200).set_body_json(serde_json::json!({ + "keys": [initial.jwk] + }))) + .up_to_n_times(1) + .mount(&server) + .await; + Mock::given(method("GET")) + .and(path("/issuer/jwks")) + .respond_with(ResponseTemplate::new(200).set_body_json(serde_json::json!({ + "keys": [rotated.jwk] + }))) + .mount(&server) + .await; + + let config = test_oidc_config(&issuer); + let cache = JwksCache::new(&config).await.unwrap(); + let initial_token = sign_token(&initial, &issuer, &config.audience, "user-initial"); + assert!(cache.validate_token(&initial_token).await.is_ok()); + + let rotated_token = sign_token(&rotated, &issuer, &config.audience, "user-rotated"); + let identity = cache.validate_token(&rotated_token).await.unwrap(); + assert_eq!(identity.subject, "user-rotated"); + } + + /// Shared setup for the "a refresh yields no usable keys" family of + /// regression tests: an initially-working cache, then a refresh + /// (triggered by an unknown `kid`) whose response is `bad_jwks`. + /// Asserts the previously cached key remains usable afterward. + async fn assert_refresh_preserves_cache(bad_jwks: serde_json::Value) { + let good = rsa_test_key("good"); + let server = MockServer::start().await; + let issuer = format!("{}/issuer", server.uri()); + let jwks_uri = format!("{issuer}/jwks"); + Mock::given(method("GET")) + .and(path("/issuer/.well-known/openid-configuration")) + .respond_with(ResponseTemplate::new(200).set_body_json(serde_json::json!({ + "issuer": issuer, + "jwks_uri": jwks_uri, + }))) + .mount(&server) + .await; + + Mock::given(method("GET")) + .and(path("/issuer/jwks")) + .respond_with(ResponseTemplate::new(200).set_body_json(serde_json::json!({ + "keys": [good.jwk] + }))) + .up_to_n_times(1) + .mount(&server) + .await; + Mock::given(method("GET")) + .and(path("/issuer/jwks")) + .respond_with(ResponseTemplate::new(200).set_body_json(bad_jwks)) + .mount(&server) + .await; + + let config = test_oidc_config(&issuer); + let cache = JwksCache::new(&config).await.unwrap(); + let token = sign_token(&good, &issuer, &config.audience, "user-1"); + assert!(cache.validate_token(&token).await.is_ok()); + + // An unknown kid triggers an unconditional refresh against the + // bad response, so this specific lookup still fails, but the + // previously cached key must remain usable afterward: the bad + // refresh must not have wiped it. + let mut header = Header::new(Algorithm::RS256); + header.kid = Some("unknown".to_string()); + let claims = serde_json::json!({ + "sub": "user-2", + "iss": issuer, + "aud": config.audience, + "exp": now_secs() + 3600, + }); + let unknown_kid_token = encode(&header, &claims, &good.encoding_key).unwrap(); + let err = cache.validate_token(&unknown_kid_token).await.unwrap_err(); + assert_eq!(err.code(), tonic::Code::Unauthenticated); + + assert!(cache.validate_token(&token).await.is_ok()); + } + + #[tokio::test] + async fn refresh_keeps_previous_keys_when_new_jwks_fully_unusable() { + // An issuer-side metadata change (e.g. every key switches to + // `use: "enc"`) must not wipe a previously-working key cache — + // otherwise a single bad JWKS response turns "working" into + // "every OIDC request rejected". + assert_refresh_preserves_cache(serde_json::json!({ + "keys": [{ "kty": "RSA", "kid": "good", "n": "abc", "e": "AQAB", "use": "enc" }] + })) + .await; + } + + #[tokio::test] + async fn refresh_keeps_previous_keys_when_new_jwks_response_empty() { + // A literally empty JWKS response (`total == 0`, as opposed to + // keys present but all filtered out) must be just as harmless + // to an existing cache as the fully-unusable case above. + assert_refresh_preserves_cache(serde_json::json!({ "keys": [] })).await; + } + + #[tokio::test] + async fn stale_keys_evicted_after_grace_period() { + // A JWKS that goes empty and stays empty must eventually stop + // being trusted; otherwise a key the issuer revoked could + // remain valid indefinitely behind a permanently broken JWKS + // endpoint. + let good = rsa_test_key("good"); + let server = MockServer::start().await; + let issuer = format!("{}/issuer", server.uri()); + let jwks_uri = format!("{issuer}/jwks"); + Mock::given(method("GET")) + .and(path("/issuer/.well-known/openid-configuration")) + .respond_with(ResponseTemplate::new(200).set_body_json(serde_json::json!({ + "issuer": issuer, + "jwks_uri": jwks_uri, + }))) + .mount(&server) + .await; + Mock::given(method("GET")) + .and(path("/issuer/jwks")) + .respond_with(ResponseTemplate::new(200).set_body_json(serde_json::json!({ + "keys": [good.jwk] + }))) + .up_to_n_times(1) + .mount(&server) + .await; + Mock::given(method("GET")) + .and(path("/issuer/jwks")) + .respond_with( + ResponseTemplate::new(200).set_body_json(serde_json::json!({ "keys": [] })), + ) + .mount(&server) + .await; + + let mut config = test_oidc_config(&issuer); + // Use a large real-time window, then place the cache timestamps + // explicitly on either side of it. This keeps scheduler delays + // from changing which branch the test exercises. + config.jwks_ttl_secs = 60; + let cache = JwksCache::new(&config).await.unwrap(); + let token = sign_token(&good, &issuer, &config.audience, "user-1"); + assert!(cache.validate_token(&token).await.is_ok()); + + // Past the TTL but within the grace period: a scheduled refresh + // sees the now-empty JWKS, but the cached key stays trusted. + *cache.last_refresh.write().await = Instant::now() + .checked_sub(cache.ttl + Duration::from_secs(1)) + .unwrap(); + assert!(cache.validate_token(&token).await.is_ok()); + + // Past the grace period with the JWKS still empty: the stale + // key must finally be evicted. + let grace = cache.ttl.saturating_mul(STALE_KEY_GRACE_MULTIPLIER); + *cache.last_refresh.write().await = Instant::now() + .checked_sub(cache.ttl + Duration::from_secs(1)) + .unwrap(); + *cache.last_nonempty_refresh.write().await = Instant::now() + .checked_sub(grace + Duration::from_secs(1)) + .unwrap(); + let err = cache.validate_token(&token).await.unwrap_err(); + assert_eq!(err.code(), tonic::Code::Unauthenticated); + } + + #[tokio::test] + async fn zero_ttl_rejected_at_construction() { + // A `jwks_ttl_secs` of 0 would refresh on every request and + // collapse the stale-key grace period to zero, so it's rejected + // outright rather than silently tolerated. This is consistent + // with how `AuthzPolicy::validate` rejects OIDC RBAC + // misconfiguration before the server starts, instead of + // working around it later. The check runs before any network + // I/O, so no mock server is needed here. + let mut config = test_oidc_config("http://unused.invalid"); + config.jwks_ttl_secs = 0; + let err = JwksCache::new(&config).await.unwrap_err(); + assert!( + err.contains("jwks_ttl_secs"), + "error should name the offending field: {err}" + ); + } + + #[tokio::test] + async fn kid_miss_refresh_throttled_within_cooldown() { + // Repeated lookups for an unknown kid within the cooldown + // window must not each trigger a fresh fetch. Otherwise, a + // burst (or steady stream) of unknown-kid requests could + // amplify load on the issuer's JWKS endpoint without bound. + let good = rsa_test_key("good"); + let server = MockServer::start().await; + let issuer = format!("{}/issuer", server.uri()); + let jwks_uri = format!("{issuer}/jwks"); + Mock::given(method("GET")) + .and(path("/issuer/.well-known/openid-configuration")) + .respond_with(ResponseTemplate::new(200).set_body_json(serde_json::json!({ + "issuer": issuer, + "jwks_uri": jwks_uri, + }))) + .mount(&server) + .await; + Mock::given(method("GET")) + .and(path("/issuer/jwks")) + .respond_with(ResponseTemplate::new(200).set_body_json(serde_json::json!({ + "keys": [good.jwk] + }))) + .mount(&server) + .await; + + let config = test_oidc_config(&issuer); + let cache = JwksCache::new(&config).await.unwrap(); + + let mut header = Header::new(Algorithm::RS256); + header.kid = Some("unknown".to_string()); + let claims = serde_json::json!({ + "sub": "user-2", + "iss": issuer, + "aud": config.audience, + "exp": now_secs() + 3600, + }); + let unknown_kid_token = encode(&header, &claims, &good.encoding_key).unwrap(); + + let previous_refresh = *cache.last_kid_miss_refresh.read().await; + let err = cache.validate_token(&unknown_kid_token).await.unwrap_err(); + assert_eq!(err.code(), tonic::Code::Unauthenticated); + assert!(*cache.last_kid_miss_refresh.read().await > previous_refresh); + + // Keep the remaining lookups deterministically inside the same + // cooldown window, even if the test task is descheduled while the + // rest of the server suite runs in parallel. + *cache.last_kid_miss_refresh.write().await = + Instant::now().checked_add(Duration::from_mins(1)).unwrap(); + for _ in 0..4 { + let err = cache.validate_token(&unknown_kid_token).await.unwrap_err(); + assert_eq!(err.code(), tonic::Code::Unauthenticated); + } + + let requests = server.received_requests().await.unwrap(); + let jwks_hits = requests + .iter() + .filter(|r| r.url.path() == "/issuer/jwks") + .count(); + // One fetch for the initial load, one more for the first + // kid-miss lookup; the remaining four must be throttled. + assert_eq!(jwks_hits, 2); + } + + #[tokio::test] + async fn header_alg_mismatch_rejected() { + let ec = ec_test_key("ec", "P-256"); + let jwks = serde_json::json!({ "keys": [ec.jwk] }); + let (cache, issuer, config, _server) = cache_from_jwks(jwks).await; + let token = sign_token(&ec, &issuer, &config.audience, "user-1"); + let forged = token_with_header_alg(&token, Algorithm::HS256); + let err = cache.validate_token(&forged).await.unwrap_err(); + assert_eq!(err.code(), tonic::Code::Unauthenticated); + } + + #[tokio::test] + async fn missing_sub_claim_rejected() { + let key = rsa_test_key("rsa"); + let jwks = serde_json::json!({ "keys": [key.jwk] }); + let (cache, issuer, config, _server) = cache_from_jwks(jwks).await; + let mut header = Header::new(key.algorithm); + header.kid = Some(key.kid.clone()); + let claims = serde_json::json!({ + "iss": issuer, + "aud": config.audience, + "exp": now_secs() + 3600, + }); + let token = encode(&header, &claims, &key.encoding_key).unwrap(); + let err = cache.validate_token(&token).await.unwrap_err(); + assert_eq!(err.code(), tonic::Code::Unauthenticated); + } + + #[tokio::test] + async fn jwk_alg_mismatch_skipped() { + let signing = rsa_test_key("rsa"); + let mut bad = signing.jwk.clone(); + bad["alg"] = serde_json::json!("ES256"); + let jwks = serde_json::json!({ "keys": [bad] }); + let (cache, issuer, config, _server) = cache_from_jwks(jwks).await; + let token = sign_token(&signing, &issuer, &config.audience, "user-1"); + let err = cache.validate_token(&token).await.unwrap_err(); + assert_eq!(err.code(), tonic::Code::Unauthenticated); + } + + #[tokio::test] + async fn rsa_declared_alg_rs512_selected_and_pinned() { + // An RSA JWK declaring "alg": "RS512" is validated as RS512, not + // vetoed against a hardcoded RS256 default that would silently + // discard the declared algorithm. + let key = rsa_test_key_from( + "rsa-rs512", + RsaFixture::Primary, + Some("RS512"), + Algorithm::RS512, + ); + let jwks = serde_json::json!({ "keys": [key.jwk.clone()] }); + let (cache, issuer, config, _server) = cache_from_jwks(jwks).await; + + let token = sign_token(&key, &issuer, &config.audience, "user-1"); + let result = cache.validate_token(&token).await; + assert!(result.is_ok(), "RS512 token should validate: {result:?}"); + + // The cached algorithm is pinned to RS512, not re-derived from + // the header — a token for the same kid claiming RS256 must + // still be rejected. + let mut rs256_variant = key; + rs256_variant.algorithm = Algorithm::RS256; + let mismatched = sign_token(&rs256_variant, &issuer, &config.audience, "user-1"); + let err = cache.validate_token(&mismatched).await.unwrap_err(); + assert_eq!(err.code(), tonic::Code::Unauthenticated); + } + + #[tokio::test] + async fn rsa_missing_alg_defaults_to_rs256() { + // No "alg" field at all (e.g. Microsoft Entra ID's JWKS) must + // still default to RS256, not be treated as unusable. + let key = rsa_test_key("rsa-no-alg"); + assert!(key.jwk.get("alg").is_none()); + let jwks = serde_json::json!({ "keys": [key.jwk.clone()] }); + let (cache, issuer, config, _server) = cache_from_jwks(jwks).await; + let token = sign_token(&key, &issuer, &config.audience, "user-1"); + assert!(cache.validate_token(&token).await.is_ok()); + } + + #[tokio::test] + async fn key_ops_verify_accepted_sign_only_rejected() { + // The gateway only ever verifies signatures, so `key_ops` must + // include "verify" — `["sign"]` alone (without "verify") on a + // public key is semantically incoherent per RFC 7517 §4.3 and + // must be rejected, not treated as interchangeable with "verify". + let mut verify_key = rsa_test_key("verify-key"); + verify_key.jwk["key_ops"] = serde_json::json!(["verify"]); + let mut sign_key = rsa_test_key_secondary("sign-key"); + sign_key.jwk["key_ops"] = serde_json::json!(["sign"]); + let jwks = serde_json::json!({ "keys": [verify_key.jwk, sign_key.jwk] }); + let (cache, issuer, config, _server) = cache_from_jwks(jwks).await; + + let verify_token = sign_token(&verify_key, &issuer, &config.audience, "verify-user"); + assert!(cache.validate_token(&verify_token).await.is_ok()); + + let sign_token_str = sign_token(&sign_key, &issuer, &config.audience, "sign-user"); + let err = cache.validate_token(&sign_token_str).await.unwrap_err(); + assert_eq!(err.code(), tonic::Code::Unauthenticated); + } + + #[tokio::test] + async fn key_ops_sign_and_encrypt_rejected() { + // RFC 7517 §4.3 permits "sign" with "verify" but not "sign" with + // "encrypt" ("other combinations SHOULD NOT be used"). Since + // "verify" is absent here, the key must be skipped. + let mut key = rsa_test_key("mixed-ops"); + key.jwk["key_ops"] = serde_json::json!(["sign", "encrypt"]); + let jwks = serde_json::json!({ "keys": [key.jwk] }); + let (cache, issuer, config, _server) = cache_from_jwks(jwks).await; + let token = sign_token(&key, &issuer, &config.audience, "user-1"); + let err = cache.validate_token(&token).await.unwrap_err(); + assert_eq!(err.code(), tonic::Code::Unauthenticated); + } + + #[tokio::test] + async fn key_ops_encrypt_only_skipped() { + let signing = rsa_test_key("good"); + let mut enc_ops = signing.jwk.clone(); + enc_ops["kid"] = serde_json::json!("enc-ops"); + enc_ops["key_ops"] = serde_json::json!(["encrypt"]); + let jwks = serde_json::json!({ "keys": [enc_ops, signing.jwk] }); + let (cache, issuer, config, _server) = cache_from_jwks(jwks).await; + let token = sign_token(&signing, &issuer, &config.audience, "user-1"); + assert!(cache.validate_token(&token).await.is_ok()); + } + + #[tokio::test] + async fn encryption_jwk_skipped_but_sig_key_works() { + let signing = rsa_test_key("sig"); + let mut enc = signing.jwk.clone(); + enc["use"] = serde_json::json!("enc"); + enc["kid"] = serde_json::json!("enc-key"); + let jwks = serde_json::json!({ "keys": [enc, signing.jwk] }); + let (cache, issuer, config, _server) = cache_from_jwks(jwks).await; + let token = sign_token(&signing, &issuer, &config.audience, "user-1"); + assert!(cache.validate_token(&token).await.is_ok()); + } + + #[tokio::test] + async fn duplicate_kid_same_algorithm_keeps_first() { + let first = rsa_test_key("dup"); + let second = rsa_test_key_secondary("other"); + let mut second_jwk = second.jwk.clone(); + second_jwk["kid"] = serde_json::json!("dup"); + let jwks = serde_json::json!({ "keys": [first.jwk, second_jwk] }); + let (cache, issuer, config, _server) = cache_from_jwks(jwks).await; + + let first_token = sign_token(&first, &issuer, &config.audience, "first-user"); + assert!(cache.validate_token(&first_token).await.is_ok()); + + let mut header = Header::new(second.algorithm); + header.kid = Some("dup".to_string()); + let claims = serde_json::json!({ + "sub": "second-user", + "iss": issuer, + "aud": config.audience, + "exp": now_secs() + 3600, + }); + let second_token = encode(&header, &claims, &second.encoding_key).unwrap(); + let err = cache.validate_token(&second_token).await.unwrap_err(); + assert_eq!(err.code(), tonic::Code::Unauthenticated); + } + + #[tokio::test] + async fn duplicate_kid_conflicting_algorithms_poison_pill() { + let rsa = rsa_test_key("conflict"); + let eddsa = ed25519_test_key("conflict"); + let jwks = serde_json::json!({ "keys": [rsa.jwk, eddsa.jwk] }); + let (cache, issuer, config, _server) = cache_from_jwks(jwks).await; + + let rsa_token = sign_token(&rsa, &issuer, &config.audience, "rsa-user"); + let err = cache.validate_token(&rsa_token).await.unwrap_err(); + assert_eq!(err.code(), tonic::Code::Unauthenticated); + + let eddsa_token = sign_token(&eddsa, &issuer, &config.audience, "eddsa-user"); + let err = cache.validate_token(&eddsa_token).await.unwrap_err(); + assert_eq!(err.code(), tonic::Code::Unauthenticated); + } + } } diff --git a/crates/openshell-server/src/auth/sandbox_jwt.rs b/crates/openshell-server/src/auth/sandbox_jwt.rs index 9dc10b840..4b086f49c 100644 --- a/crates/openshell-server/src/auth/sandbox_jwt.rs +++ b/crates/openshell-server/src/auth/sandbox_jwt.rs @@ -417,7 +417,7 @@ mod tests { } fn pair() -> (SandboxJwtIssuer, SandboxJwtAuthenticator) { - pair_with_ttl(Duration::from_secs(3600)) + pair_with_ttl(Duration::from_hours(1)) } fn pair_with_ttl(ttl: Duration) -> (SandboxJwtIssuer, SandboxJwtAuthenticator) { @@ -472,7 +472,7 @@ mod tests { mat.signing_key_pem.as_bytes(), mat.kid.clone(), "test-gateway", - Duration::from_secs(3600), + Duration::from_hours(1), ) .expect("issuer"); let auth = SandboxJwtAuthenticator::from_pem( @@ -582,7 +582,7 @@ mod tests { mat.signing_key_pem.as_bytes(), mat.kid.clone(), "g", - Duration::from_secs(3600), + Duration::from_hours(1), ) .unwrap(); let auth = @@ -623,7 +623,7 @@ mod tests { &extension_audience("urn:openshell:extension:middleware:scanner"), ExtensionCallerKind::Gateway, None, - Duration::from_secs(300), + Duration::from_mins(5), ) .expect("gateway token"); let mut validation = Validation::new(Algorithm::EdDSA); @@ -644,7 +644,7 @@ mod tests { &extension_audience("urn:openshell:extension:middleware:scanner"), ExtensionCallerKind::Supervisor, Some("sandbox-a"), - Duration::from_secs(300), + Duration::from_mins(5), ) .expect("supervisor token"); let claims = decode::(&supervisor.token, &decoding_key, &validation) @@ -662,7 +662,7 @@ mod tests { mat.signing_key_pem.as_bytes(), mat.kid.clone(), "gateway-a", - Duration::from_secs(3600), + Duration::from_hours(1), ) .expect("issuer"); @@ -671,7 +671,7 @@ mod tests { &extension_audience("urn:openshell:extension:middleware:scanner"), ExtensionCallerKind::Gateway, None, - Duration::from_secs(300), + Duration::from_mins(5), ) .expect("extension token"); assert_eq!( @@ -704,7 +704,7 @@ mod tests { &extension_audience("service-a"), ExtensionCallerKind::Gateway, None, - Duration::from_secs(60), + Duration::from_mins(1), ) .expect("token"); let decoding_key = DecodingKey::from_ed_pem(mat.public_key_pem.as_bytes()).unwrap(); @@ -722,7 +722,7 @@ mod tests { &extension_audience("openshell-gateway:test-gateway"), ExtensionCallerKind::Supervisor, Some("sandbox-a"), - Duration::from_secs(60), + Duration::from_mins(1), ) .expect_err("gateway sandbox audience must be reserved"); assert_eq!(error.code(), tonic::Code::InvalidArgument); @@ -755,7 +755,7 @@ mod tests { &extension_audience("service"), ExtensionCallerKind::Supervisor, None, - Duration::from_secs(60), + Duration::from_mins(1), ) .is_err() ); @@ -765,7 +765,7 @@ mod tests { &extension_audience("service"), ExtensionCallerKind::Gateway, Some("sandbox-a"), - Duration::from_secs(60), + Duration::from_mins(1), ) .is_err() ); diff --git a/crates/openshell-server/src/auth/sandbox_methods.rs b/crates/openshell-server/src/auth/sandbox_methods.rs index 89f34d125..f368298dd 100644 --- a/crates/openshell-server/src/auth/sandbox_methods.rs +++ b/crates/openshell-server/src/auth/sandbox_methods.rs @@ -29,9 +29,6 @@ mod tests { assert!(is_sandbox_callable( "/openshell.v1.OpenShell/GetSandboxConfig" )); - assert!(is_sandbox_callable( - "/openshell.inference.v1.Inference/GetInferenceBundle" - )); assert!(is_sandbox_callable( "/openshell.v1.OpenShell/ExchangeProviderSubjectToken" )); @@ -53,11 +50,5 @@ mod tests { assert!(!is_sandbox_callable( "/openshell.v1.OpenShell/ApproveDraftChunk" )); - assert!(!is_sandbox_callable( - "/openshell.inference.v1.Inference/GetInferenceRoute" - )); - assert!(!is_sandbox_callable( - "/openshell.inference.v1.Inference/SetInferenceRoute" - )); } } diff --git a/crates/openshell-server/src/auth/workspace_authz.rs b/crates/openshell-server/src/auth/workspace_authz.rs index e23d2287a..2958d656c 100644 --- a/crates/openshell-server/src/auth/workspace_authz.rs +++ b/crates/openshell-server/src/auth/workspace_authz.rs @@ -9,7 +9,10 @@ //! and workspace-level role. use super::principal::Principal; -use openshell_core::proto::WorkspaceRole as ProtoWorkspaceRole; +use openshell_core::proto::{ + WorkspaceRole as ProtoWorkspaceRole, WorkspaceSelector, + workspace_selector::Selection as WorkspaceSelection, +}; use tonic::Status; use crate::persistence::Store; @@ -39,7 +42,7 @@ impl MinWorkspaceRole { /// Result of a successful workspace authorization check. #[derive(Debug)] pub struct AuthorizedWorkspace { - /// Resolved workspace name (empty string normalized to `"default"`). + /// Explicit workspace name selected by the caller. pub workspace: String, /// How the caller was authorized. pub grant: AuthGrant, @@ -56,6 +59,75 @@ pub enum AuthGrant { Sandbox, } +/// Authorized scope for a request that supports one workspace or all workspaces. +#[derive(Debug)] +pub enum AuthorizedWorkspaceScope { + /// One explicitly named workspace. + Workspace(AuthorizedWorkspace), + /// All workspaces, authorized for a platform administrator. + AllWorkspaces, +} + +/// Authorize the required named selector on a single-workspace request. +#[allow(clippy::result_large_err)] +pub async fn authorize_workspace_selector( + store: &Store, + admin_role: &str, + principal: &Principal, + selector: Option<&WorkspaceSelector>, + min_role: MinWorkspaceRole, +) -> Result { + let workspace = selected_workspace_name(selector)?; + authorize_workspace(store, admin_role, principal, workspace, min_role).await +} + +/// Authorize a selector on a request that explicitly supports all workspaces. +#[allow(clippy::result_large_err)] +pub async fn authorize_list_workspace_selector( + store: &Store, + admin_role: &str, + principal: &Principal, + selector: Option<&WorkspaceSelector>, + min_role: MinWorkspaceRole, +) -> Result { + match selected_workspace(selector)? { + WorkspaceSelection::Workspace(workspace) => { + authorize_workspace(store, admin_role, principal, workspace, min_role) + .await + .map(AuthorizedWorkspaceScope::Workspace) + } + WorkspaceSelection::AllWorkspaces(_) => { + require_platform_admin(admin_role, principal)?; + Ok(AuthorizedWorkspaceScope::AllWorkspaces) + } + } +} + +/// Return the explicitly selected workspace name, rejecting missing, empty, +/// or all-workspaces selections. +#[allow(clippy::result_large_err)] +pub fn selected_workspace_name(selector: Option<&WorkspaceSelector>) -> Result<&str, Status> { + match selected_workspace(selector)? { + WorkspaceSelection::Workspace(workspace) => Ok(workspace), + WorkspaceSelection::AllWorkspaces(_) => Err(Status::invalid_argument( + "all_workspaces is not supported by this request", + )), + } +} + +#[allow(clippy::result_large_err)] +fn selected_workspace(selector: Option<&WorkspaceSelector>) -> Result<&WorkspaceSelection, Status> { + let selection = selector + .and_then(|selector| selector.selection.as_ref()) + .ok_or_else(|| Status::invalid_argument("workspace_scope is required"))?; + + if let WorkspaceSelection::Workspace(workspace) = selection { + crate::grpc::workspace::validate_workspace_name(workspace)?; + } + + Ok(selection) +} + /// Authorize a workspace-scoped operation for a user principal. /// /// Checks workspace membership and role. Platform admins (callers whose @@ -72,7 +144,7 @@ pub async fn authorize_workspace( workspace: &str, min_role: MinWorkspaceRole, ) -> Result { - let workspace = normalize_workspace(workspace); + let workspace = workspace.to_string(); match principal { Principal::User(user) => { @@ -186,14 +258,6 @@ fn role_satisfies(member_role: ProtoWorkspaceRole, min_role: MinWorkspaceRole) - } } -fn normalize_workspace(workspace: &str) -> String { - if workspace.is_empty() { - "default".to_string() - } else { - workspace.to_string() - } -} - #[cfg(test)] mod tests { use super::*; @@ -401,20 +465,75 @@ mod tests { } #[tokio::test] - async fn empty_workspace_normalizes_to_default() { + async fn empty_named_selector_is_rejected() { let store = test_store().await; add_member(&store, "default", "user-d", ProtoWorkspaceRole::User).await; let principal = user_principal("user-d", &["openshell-user"]); - let result = authorize_workspace( + let result = authorize_workspace_selector( &store, "openshell-admin", &principal, - "", + Some(&openshell_core::proto::workspace_selector("")), MinWorkspaceRole::User, ) .await; - assert!(result.is_ok()); - assert_eq!(result.unwrap().workspace, "default"); + let err = result.unwrap_err(); + assert_eq!(err.code(), tonic::Code::InvalidArgument); + assert_eq!(err.message(), "workspace name is required"); + } + + #[test] + fn missing_and_unset_selectors_are_rejected() { + let missing = selected_workspace_name(None).unwrap_err(); + assert_eq!(missing.code(), tonic::Code::InvalidArgument); + assert_eq!(missing.message(), "workspace_scope is required"); + + let unset = selected_workspace_name(Some(&WorkspaceSelector::default())).unwrap_err(); + assert_eq!(unset.code(), tonic::Code::InvalidArgument); + assert_eq!(unset.message(), "workspace_scope is required"); + } + + #[test] + fn all_workspaces_is_rejected_for_single_workspace_requests() { + let err = selected_workspace_name(Some(&openshell_core::proto::all_workspaces_selector())) + .unwrap_err(); + assert_eq!(err.code(), tonic::Code::InvalidArgument); + assert_eq!( + err.message(), + "all_workspaces is not supported by this request" + ); + } + + #[tokio::test] + async fn all_workspaces_requires_platform_admin() { + let store = test_store().await; + let selector = openshell_core::proto::all_workspaces_selector(); + let principal = user_principal("workspace-user", &["openshell-user"]); + let err = authorize_list_workspace_selector( + &store, + "openshell-admin", + &principal, + Some(&selector), + MinWorkspaceRole::User, + ) + .await + .unwrap_err(); + assert_eq!(err.code(), tonic::Code::PermissionDenied); + + let admin = user_principal("platform-admin", &["openshell-admin"]); + let authorized = authorize_list_workspace_selector( + &store, + "openshell-admin", + &admin, + Some(&selector), + MinWorkspaceRole::User, + ) + .await + .unwrap(); + assert!(matches!( + authorized, + AuthorizedWorkspaceScope::AllWorkspaces + )); } #[tokio::test] diff --git a/crates/openshell-server/src/cli.rs b/crates/openshell-server/src/cli.rs index f22e1355e..b8b7dfd07 100644 --- a/crates/openshell-server/src/cli.rs +++ b/crates/openshell-server/src/cli.rs @@ -6,7 +6,6 @@ use clap::parser::ValueSource; use clap::{ArgAction, ArgMatches, Command, CommandFactory, FromArgMatches, Parser}; use miette::{IntoDiagnostic, Result}; -use openshell_core::ComputeDriverKind; use openshell_core::config::{DEFAULT_GATEWAY_NAME, DEFAULT_SERVER_PORT}; use std::net::{IpAddr, SocketAddr}; use std::path::PathBuf; @@ -17,10 +16,7 @@ use crate::certgen; use crate::compute::driver_config::GuestTlsPaths; use crate::config_file::{self, ConfigFile, GatewayFileSection}; use crate::defaults::{self, LocalTlsPaths}; -use crate::{ - ComputeDriverRegistry, ServerStartupConfig, configured_compute_driver_for_startup, - install_default_compute_drivers, run_server, tracing_bus::TracingLogBus, -}; +use crate::{ComputeDriverRegistry, ServerStartupConfig, run_server, tracing_bus::TracingLogBus}; /// `OpenShell` gateway process - gRPC and HTTP server with protocol multiplexing. /// @@ -107,12 +103,10 @@ struct RunArgs { /// Compute drivers configured for this gateway. /// - /// Accepts a comma-delimited list such as `kubernetes` or - /// `kubernetes,podman`. The configuration format is future-proofed for - /// multiple drivers, but the gateway currently requires exactly one. - /// When unset, the gateway auto-detects the driver based on the runtime - /// environment (Kubernetes → Podman → Docker). VM is never - /// auto-detected and requires explicit configuration. + /// Accepts a comma-delimited list of registered driver names. The + /// configuration format is future-proofed for multiple drivers, but the + /// gateway currently requires exactly one. When unset, the gateway runs + /// detection probes supplied by the drivers compiled into the binary. #[arg( long, alias = "driver", @@ -127,9 +121,9 @@ struct RunArgs { /// /// When set, the socket is associated with the single driver name supplied /// by `--drivers` or `OPENSHELL_DRIVERS` and replaces normal construction - /// for that selected name, including canonical built-in names. The gateway - /// connects to this operator-provided endpoint; it does not provision the - /// remote driver. + /// for that selected name, including a compiled registration with the same + /// name. The gateway connects to this operator-provided endpoint; it does + /// not provision the remote driver. #[arg(long, env = "OPENSHELL_COMPUTE_DRIVER_SOCKET")] compute_driver_socket: Option, @@ -147,9 +141,9 @@ struct RunArgs { /// Enable mTLS client certificate authentication for local single-user gateways. /// - /// When unset, this defaults on for Docker, Podman, and VM gateways that - /// have client certificate verification configured and no OIDC issuer. - /// Kubernetes deployments must use OIDC or fronting-proxy auth instead. + /// When unset, this defaults on for drivers registered as local + /// single-player backends when client certificate verification is + /// configured and no OIDC issuer is present. #[arg( long = "enable-mtls-auth", env = "OPENSHELL_ENABLE_MTLS_AUTH", @@ -231,15 +225,11 @@ pub fn command() -> Command { } pub async fn run_cli() -> Result<()> { - run_cli_with_compute_drivers(install_default_compute_drivers()).await + run_cli_with_compute_drivers(ComputeDriverRegistry::new()).await } /// Run the gateway CLI with the compute drivers linked by the binary. pub async fn run_cli_with_compute_drivers(compute_drivers: ComputeDriverRegistry) -> Result<()> { - rustls::crypto::ring::default_provider() - .install_default() - .map_err(|e| miette::miette!("failed to install rustls crypto provider: {e:?}"))?; - let matches = command().get_matches(); let cli = Cli::from_arg_matches(&matches).expect("clap validated args"); @@ -249,7 +239,12 @@ pub async fn run_cli_with_compute_drivers(compute_drivers: ComputeDriverRegistry } } -fn prepare_server_config( +#[cfg(test)] +fn prepare_server_config(args: &mut RunArgs, matches: &ArgMatches) -> Result { + prepare_server_config_with_drivers(args, matches, &ComputeDriverRegistry::new()) +} + +fn prepare_server_config_with_drivers( args: &mut RunArgs, matches: &ArgMatches, compute_drivers: &ComputeDriverRegistry, @@ -270,7 +265,7 @@ fn prepare_server_config( let compute_driver = compute_drivers .select(&args.drivers) .map_err(|error| miette::miette!("{error}"))?; - let compute_driver_kind = compute_driver.name().parse::().ok(); + let selected_registration = compute_drivers.get(compute_driver.name()); let local_tls = apply_runtime_defaults(args)?; let guest_tls = local_tls.as_ref().map(GuestTlsPaths::from); @@ -281,7 +276,7 @@ fn prepare_server_config( let has_client_ca = args.tls_client_ca.is_some(); let has_oidc = args.oidc_issuer.is_some(); let mtls_auth_enabled = - resolve_mtls_auth_enabled(args, matches, file.as_ref(), compute_driver_kind); + resolve_mtls_auth_enabled(args, matches, file.as_ref(), selected_registration); if args.disable_tls && has_client_ca { return Err(miette::miette!( @@ -298,9 +293,11 @@ fn prepare_server_config( "mTLS user authentication requires --tls-client-ca so client certificates can be verified." )); } - if mtls_auth_enabled && matches!(compute_driver_kind, Some(ComputeDriverKind::Kubernetes)) { + if mtls_auth_enabled + && selected_registration.is_some_and(|registration| !registration.supports_mtls_user_auth()) + { return Err(miette::miette!( - "mTLS user authentication is not supported with the Kubernetes compute driver. Configure OIDC or a trusted fronting proxy for user authentication." + "mTLS user authentication is not supported with the selected compute driver. Configure OIDC or a trusted fronting proxy for user authentication." )); } @@ -501,8 +498,7 @@ async fn run_from_args( matches: ArgMatches, compute_drivers: ComputeDriverRegistry, ) -> Result<()> { - let prepared = prepare_server_config(&mut args, &matches, &compute_drivers)?; - let compute_driver = configured_compute_driver_for_startup(&compute_drivers, &prepared)?; + let prepared = prepare_server_config_with_drivers(&mut args, &matches, &compute_drivers)?; let tracing_log_bus = TracingLogBus::new(); let otlp_config = prepared @@ -511,14 +507,18 @@ async fn run_from_args( .and_then(|f| f.openshell.gateway.otlp.as_ref()); let gateway_resource = crate::otel_tracing::GatewayResourceAttributes::new( Some(prepared.config.name.as_str()), - Some(compute_driver.name()), + Some(prepared.compute_driver.name()), + ); + let compute_driver_tracing = compute_drivers.in_process_tracing( + &prepared.compute_driver, + &prepared.config.compute_driver_endpoints, ); let (tracing_handle, setup_error) = crate::tracing_setup::install( EnvFilter::try_from_default_env() .unwrap_or_else(|_| EnvFilter::new(&prepared.config.log_level)), &tracing_log_bus, otlp_config, - &compute_driver, + compute_driver_tracing, gateway_resource, ); @@ -576,7 +576,7 @@ async fn run_from_args( info!(bind = %prepared.config.bind_address, "Starting OpenShell server"); - let result = Box::pin(run_server(prepared, compute_driver, tracing_log_bus)).await; + let result = Box::pin(run_server(prepared, tracing_log_bus, compute_drivers)).await; tracing_handle.shutdown(); @@ -817,23 +817,15 @@ fn normalize_compute_driver_socket_args(args: &mut RunArgs, matches: &ArgMatches } } -fn is_singleplayer_driver(driver: Option) -> bool { - matches!( - driver, - Some( - ComputeDriverKind::Docker - | ComputeDriverKind::Podman - | ComputeDriverKind::Vm - | ComputeDriverKind::Mxc - ) - ) +fn is_singleplayer_driver(registration: Option<&crate::ComputeDriverRegistration>) -> bool { + registration.is_some_and(crate::ComputeDriverRegistration::is_local_singleplayer) } fn resolve_mtls_auth_enabled( args: &RunArgs, matches: &ArgMatches, file: Option<&ConfigFile>, - compute_driver: Option, + selected_registration: Option<&crate::ComputeDriverRegistration>, ) -> bool { let file_configured = file .and_then(|f| f.openshell.gateway.mtls_auth.as_ref()) @@ -846,7 +838,7 @@ fn resolve_mtls_auth_enabled( return false; } - is_singleplayer_driver(compute_driver) + is_singleplayer_driver(selected_registration) } #[cfg(test)] @@ -859,37 +851,49 @@ mod tests { static REGISTRY_DETECTION_CALLS: AtomicUsize = AtomicUsize::new(0); - fn detect_registered_docker() -> bool { + fn detect_registered_local() -> bool { REGISTRY_DETECTION_CALLS.fetch_add(1, Ordering::SeqCst); true } #[derive(Clone, Copy)] - struct TestComputeDriverFactory; + struct TestFactory; #[async_trait::async_trait] - impl crate::ComputeDriverFactory for TestComputeDriverFactory { + impl crate::ComputeDriverFactory for TestFactory { async fn build( &self, _context: crate::ComputeDriverBuildContext<'_>, - ) -> openshell_core::Result { - unreachable!("configuration tests do not construct the driver") + ) -> openshell_core::Result { + unreachable!("CLI metadata tests do not build drivers") } } - fn detected_docker_registry() -> crate::ComputeDriverRegistry { + fn test_registry(name: &str, singleplayer: bool, mtls: bool) -> crate::ComputeDriverRegistry { + let mut registration = + crate::ComputeDriverRegistration::new(name, 100, None, TestFactory).unwrap(); + if singleplayer { + registration = registration.with_local_singleplayer(); + } + if !mtls { + registration = registration.without_mtls_user_auth(); + } let mut registry = crate::ComputeDriverRegistry::new(); + registry.install(registration).unwrap(); registry - .install( - crate::ComputeDriverRegistration::new( - "docker", - 100, - Some(detect_registered_docker), - TestComputeDriverFactory, - ) - .unwrap(), - ) - .unwrap(); + } + + fn detected_local_registry() -> crate::ComputeDriverRegistry { + let registration = crate::ComputeDriverRegistration::new( + "local", + 100, + Some(detect_registered_local), + TestFactory, + ) + .unwrap() + .with_local_singleplayer(); + let mut registry = crate::ComputeDriverRegistry::new(); + registry.install(registration).unwrap(); registry } @@ -1350,7 +1354,7 @@ mod tests { "--db-url", "sqlite::memory:", "--drivers", - "docker", + "local", "--tls-cert", "/tmp/server.crt", "--tls-key", @@ -1363,7 +1367,7 @@ mod tests { &args, &matches, None, - Some(openshell_core::ComputeDriverKind::Docker) + test_registry("local", true, true).get("local") )); } @@ -1376,7 +1380,6 @@ mod tests { let config = tempfile::tempdir().unwrap(); let _state = EnvVarGuard::set("XDG_STATE_HOME", state.path().to_str().unwrap()); let _config = EnvVarGuard::set("XDG_CONFIG_HOME", config.path().to_str().unwrap()); - let _kubernetes = EnvVarGuard::set("KUBERNETES_SERVICE_HOST", "10.0.0.1"); let _mtls = EnvVarGuard::remove("OPENSHELL_ENABLE_MTLS_AUTH"); let _drivers = EnvVarGuard::remove("OPENSHELL_DRIVERS"); REGISTRY_DETECTION_CALLS.store(0, Ordering::SeqCst); @@ -1392,18 +1395,19 @@ mod tests { "--tls-client-ca", "/tmp/ca.crt", ]); - let registry = detected_docker_registry(); + let registry = detected_local_registry(); - let prepared = super::prepare_server_config(&mut args, &matches, ®istry).unwrap(); + let prepared = + super::prepare_server_config_with_drivers(&mut args, &matches, ®istry).unwrap(); - assert_eq!(prepared.compute_driver.name(), "docker"); + assert_eq!(prepared.compute_driver.name(), "local"); assert!(prepared.config.compute_drivers.is_empty()); assert!(prepared.config.mtls_auth.enabled); assert_eq!(REGISTRY_DETECTION_CALLS.load(Ordering::SeqCst), 1); } #[test] - fn mtls_auth_does_not_auto_default_for_kubernetes_driver() { + fn mtls_auth_does_not_auto_default_for_shared_driver() { let _lock = ENV_LOCK .lock() .unwrap_or_else(std::sync::PoisonError::into_inner); @@ -1414,7 +1418,7 @@ mod tests { "--db-url", "sqlite::memory:", "--drivers", - "kubernetes", + "shared", "--tls-cert", "/tmp/server.crt", "--tls-key", @@ -1427,7 +1431,7 @@ mod tests { &args, &matches, None, - Some(openshell_core::ComputeDriverKind::Kubernetes) + test_registry("shared", false, false).get("shared") )); } @@ -1443,7 +1447,7 @@ mod tests { "--db-url", "sqlite::memory:", "--drivers", - "docker", + "local", "--tls-cert", "/tmp/server.crt", "--tls-key", @@ -1464,7 +1468,7 @@ enabled = false &args, &matches, Some(&file), - Some(openshell_core::ComputeDriverKind::Docker) + test_registry("local", true, true).get("local") )); } @@ -1702,23 +1706,12 @@ ssh_session_ttl_secs = 1234 } #[test] - fn singleplayer_driver_matches_only_one_local_driver() { - for driver in [ - openshell_core::ComputeDriverKind::Docker, - openshell_core::ComputeDriverKind::Podman, - openshell_core::ComputeDriverKind::Vm, - openshell_core::ComputeDriverKind::Mxc, - ] { - assert!( - super::is_singleplayer_driver(Some(driver)), - "{driver} should be singleplayer" - ); - } + fn singleplayer_behavior_comes_from_registration() { + let local = test_registry("local", true, true); + assert!(super::is_singleplayer_driver(local.get("local"))); - assert!(!super::is_singleplayer_driver(Some( - openshell_core::ComputeDriverKind::Kubernetes - ))); - assert!(!super::is_singleplayer_driver(None)); + let shared = test_registry("shared", false, true); + assert!(!super::is_singleplayer_driver(shared.get("shared"))); } #[test] @@ -1744,11 +1737,6 @@ ssh_session_ttl_secs = 1234 Some(std::path::Path::new("/run/openshell/kyma.sock")) ); assert_eq!(args.drivers, ["kyma"]); - assert!( - args.drivers[0] - .parse::() - .is_err() - ); } #[test] @@ -1928,12 +1916,8 @@ mem_mib = "not-a-number" "--disable-tls", ]); - let prepared = super::prepare_server_config( - &mut args, - &matches, - &crate::install_default_compute_drivers(), - ) - .expect("server config is prepared"); + let prepared = + super::prepare_server_config(&mut args, &matches).expect("server config is prepared"); assert_eq!(prepared.config.compute_drivers, vec!["podman".to_string()]); assert_eq!( @@ -1944,53 +1928,4 @@ mem_mib = "not-a-number" assert!(file.openshell.drivers.contains_key("docker")); assert!(file.openshell.drivers.contains_key("vm")); } - - #[test] - #[cfg(not(target_os = "windows"))] - fn driver_inherits_shared_image_from_gateway_section() { - // [openshell.gateway].default_image inherits into the K8s driver - // table when the driver-specific table does not set it. - let file = config_file_from_toml( - r#" -[openshell.gateway] -default_image = "ghcr.io/nvidia/openshell/sandbox:1.0" - -[openshell.drivers.kubernetes] -namespace = "agents" -"#, - ); - let merged = crate::config_file::driver_table( - super::ComputeDriverKind::Kubernetes.as_str(), - &file.openshell.gateway, - file.openshell.drivers.get("kubernetes"), - ); - let parsed = merged - .try_into::() - .expect("merged table deserializes"); - assert_eq!(parsed.default_image, "ghcr.io/nvidia/openshell/sandbox:1.0"); - assert_eq!(parsed.namespace, "agents"); - } - - #[test] - #[cfg(not(target_os = "windows"))] - fn driver_specific_value_overrides_gateway_inheritance() { - let file = config_file_from_toml( - r#" -[openshell.gateway] -default_image = "gateway-default:1.0" - -[openshell.drivers.kubernetes] -default_image = "k8s-specific:1.0" -"#, - ); - let merged = crate::config_file::driver_table( - super::ComputeDriverKind::Kubernetes.as_str(), - &file.openshell.gateway, - file.openshell.drivers.get("kubernetes"), - ); - let parsed = merged - .try_into::() - .expect("deserializes"); - assert_eq!(parsed.default_image, "k8s-specific:1.0"); - } } diff --git a/crates/openshell-server/src/compute/driver_config.rs b/crates/openshell-server/src/compute/driver_config.rs index c25c63ded..d06e6fbc8 100644 --- a/crates/openshell-server/src/compute/driver_config.rs +++ b/crates/openshell-server/src/compute/driver_config.rs @@ -3,20 +3,13 @@ //! Selected compute-driver config construction. //! -//! This module owns loading the selected driver config from TOML, applying -//! driver-specific environment overrides, and applying gateway startup defaults. -//! It does not acquire, connect to, or start compute drivers. - -#[cfg(all(not(target_os = "windows"), feature = "in-tree-compute-drivers"))] -pub mod builtin; +//! This module owns loading the selected driver config from TOML and applying +//! gateway startup defaults and endpoint overrides. It does not acquire, +//! connect to, or start compute drivers. use crate::config_file; use crate::defaults::LocalTlsPaths; -#[cfg(target_os = "windows")] -use openshell_core::ComputeDriverKind; use openshell_core::{Error, Result}; -#[cfg(target_os = "windows")] -use openshell_driver_mxc::MxcComputeConfig; use serde::Deserialize; use std::collections::BTreeMap; use std::path::PathBuf; @@ -53,15 +46,6 @@ pub struct DriverStartupContext<'a> { pub endpoint_overrides: &'a BTreeMap, } -/// Build the selected MXC config from TOML. MXC is Windows-only and has no -/// runtime-default overlay; the driver reads its own settings from the config. -/// The Linux built-in driver configs now live in the `builtin` submodule -/// (compiled only off Windows). -#[cfg(target_os = "windows")] -pub fn mxc_config_from_context(context: DriverStartupContext<'_>) -> Result { - driver_config_from_context(context, ComputeDriverKind::Mxc.as_str()) -} - pub fn remote_driver_config_from_context( context: DriverStartupContext<'_>, name: &str, @@ -91,16 +75,18 @@ pub struct RemoteDriverConfig { pub fn driver_config_from_context( context: DriverStartupContext<'_>, driver_name: &str, + inherited_config_keys: &[&str], ) -> Result where T: Default + serde::de::DeserializeOwned, { - driver_config_from_file(context.file, driver_name) + driver_config_from_file(context.file, driver_name, inherited_config_keys) } fn driver_config_from_file( file: Option<&config_file::ConfigFile>, driver_name: &str, + inherited_config_keys: &[&str], ) -> Result where T: Default + serde::de::DeserializeOwned, @@ -108,10 +94,11 @@ where let Some(file) = file else { return Ok(T::default()); }; - let merged = config_file::driver_table( + let merged = config_file::driver_table_with_inherited_keys( driver_name, &file.openshell.gateway, file.openshell.drivers.get(driver_name), + inherited_config_keys, ); merged.try_into().map_err(|e| { Error::config(format!( diff --git a/crates/openshell-server/src/compute/driver_config/builtin.rs b/crates/openshell-server/src/compute/driver_config/builtin.rs deleted file mode 100644 index dea867237..000000000 --- a/crates/openshell-server/src/compute/driver_config/builtin.rs +++ /dev/null @@ -1,231 +0,0 @@ -// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -// SPDX-License-Identifier: Apache-2.0 - -//! Configuration construction for built-in compute drivers. - -use super::{DriverStartupContext, GuestTlsPaths, driver_config_from_context}; -use crate::compute::VmComputeConfig; -#[cfg(test)] -use crate::config_file; -use openshell_core::{ComputeDriverKind, Result}; -use openshell_driver_docker::DockerComputeConfig; -use openshell_driver_kubernetes::KubernetesComputeConfig; -use openshell_driver_podman::PodmanComputeConfig; -use std::path::PathBuf; - -/// Build the selected Kubernetes config from TOML plus runtime defaults. -pub fn kubernetes_config_from_context( - context: DriverStartupContext<'_>, -) -> Result { - let mut cfg = driver_config_from_context(context, ComputeDriverKind::Kubernetes.as_str())?; - apply_kubernetes_runtime_defaults(&mut cfg); - Ok(cfg) -} - -/// Build the selected Podman config from TOML plus runtime defaults. -pub fn podman_config_from_context( - context: DriverStartupContext<'_>, -) -> Result { - let mut podman = driver_config_from_context(context, ComputeDriverKind::Podman.as_str())?; - apply_podman_runtime_defaults(&mut podman, context); - Ok(podman) -} - -/// Build the selected Docker config from TOML plus runtime defaults. -pub fn docker_config_from_context( - context: DriverStartupContext<'_>, -) -> Result { - let mut cfg = driver_config_from_context(context, ComputeDriverKind::Docker.as_str())?; - apply_docker_runtime_defaults(&mut cfg, context); - Ok(cfg) -} - -/// Build the selected VM config from TOML plus runtime defaults. -pub fn vm_config_from_context(context: DriverStartupContext<'_>) -> Result { - let mut cfg = driver_config_from_context(context, ComputeDriverKind::Vm.as_str())?; - apply_vm_runtime_defaults(&mut cfg, context); - Ok(cfg) -} - -fn apply_kubernetes_runtime_defaults(k8s: &mut KubernetesComputeConfig) { - if let Ok(size) = std::env::var("OPENSHELL_K8S_WORKSPACE_DEFAULT_STORAGE_SIZE") { - k8s.workspace_default_storage_size = size; - } - if let Ok(storage_class) = std::env::var("OPENSHELL_K8S_WORKSPACE_STORAGE_CLASS") { - k8s.workspace_storage_class = storage_class; - } -} - -fn apply_podman_runtime_defaults( - podman: &mut PodmanComputeConfig, - context: DriverStartupContext<'_>, -) { - podman.gateway_port = context.gateway_port; - apply_podman_env_overrides(podman); - apply_guest_tls_defaults_to_split_fields( - &mut podman.guest_tls_ca, - &mut podman.guest_tls_cert, - &mut podman.guest_tls_key, - context.guest_tls, - ); -} - -fn apply_docker_runtime_defaults(cfg: &mut DockerComputeConfig, context: DriverStartupContext<'_>) { - apply_guest_tls_defaults_to_split_fields( - &mut cfg.guest_tls_ca, - &mut cfg.guest_tls_cert, - &mut cfg.guest_tls_key, - context.guest_tls, - ); -} - -fn apply_vm_runtime_defaults(cfg: &mut VmComputeConfig, context: DriverStartupContext<'_>) { - if cfg.state_dir.as_os_str().is_empty() { - cfg.state_dir = VmComputeConfig::default_state_dir(); - } - if cfg.grpc_endpoint.trim().is_empty() - && (!context.gateway_tls_enabled || context.guest_tls.is_some()) - { - let scheme = if context.gateway_tls_enabled { - "https" - } else { - "http" - }; - cfg.grpc_endpoint = format!("{scheme}://127.0.0.1:{}", context.gateway_port); - } - - apply_guest_tls_defaults_to_split_fields( - &mut cfg.guest_tls_ca, - &mut cfg.guest_tls_cert, - &mut cfg.guest_tls_key, - context.guest_tls, - ); -} - -fn apply_guest_tls_defaults_to_split_fields( - ca: &mut Option, - cert: &mut Option, - key: &mut Option, - defaults: Option<&GuestTlsPaths>, -) { - if ca.is_none() - && cert.is_none() - && key.is_none() - && let Some(paths) = defaults - { - *ca = Some(paths.ca.clone()); - *cert = Some(paths.cert.clone()); - *key = Some(paths.key.clone()); - } -} - -fn apply_podman_env_overrides(podman: &mut PodmanComputeConfig) { - if let Ok(p) = std::env::var("OPENSHELL_PODMAN_SOCKET") { - podman.socket_path = Some(PathBuf::from(p)); - } - if let Ok(ip) = std::env::var("OPENSHELL_PODMAN_HOST_GATEWAY_IP") { - podman.host_gateway_ip = ip; - } - if let Ok(mode) = std::env::var("OPENSHELL_PODMAN_USERNS") { - podman.userns = Some(mode); - } -} - -#[cfg(test)] -mod tests { - use super::*; - use std::collections::BTreeMap; - - fn test_context(file: Option<&config_file::ConfigFile>) -> DriverStartupContext<'_> { - static EMPTY_ENDPOINT_OVERRIDES: std::sync::LazyLock> = - std::sync::LazyLock::new(BTreeMap::new); - DriverStartupContext { - file, - guest_tls: None, - gateway_port: openshell_core::config::DEFAULT_SERVER_PORT, - gateway_tls_enabled: false, - endpoint_overrides: &EMPTY_ENDPOINT_OVERRIDES, - } - } - - #[test] - fn podman_config_reads_bind_mount_opt_in_from_driver_table() { - let file: config_file::ConfigFile = toml::from_str( - r" -[openshell.drivers.podman] -enable_bind_mounts = true -", - ) - .expect("valid config"); - - let cfg = podman_config_from_context(test_context(Some(&file))).expect("podman config"); - - assert!(cfg.enable_bind_mounts); - } - - #[test] - fn docker_config_reads_bind_mount_opt_in_from_driver_table() { - let file: config_file::ConfigFile = toml::from_str( - r" -[openshell.drivers.docker] -enable_bind_mounts = true -", - ) - .expect("valid config"); - - let cfg = docker_config_from_context(test_context(Some(&file))).expect("docker config"); - - assert!(cfg.enable_bind_mounts); - } - - #[test] - fn docker_config_reads_socket_path_from_driver_table() { - let file: config_file::ConfigFile = toml::from_str( - r#" -[openshell.drivers.docker] -socket_path = "/tmp/docker.sock" -"#, - ) - .expect("valid config"); - - let cfg = docker_config_from_context(test_context(Some(&file))).expect("docker config"); - - assert_eq!(cfg.socket_path, Some(PathBuf::from("/tmp/docker.sock"))); - } - - #[test] - fn docker_config_reports_selected_invalid_driver_table() { - let file: config_file::ConfigFile = toml::from_str( - r" -[openshell.drivers.docker] -unknown_docker_key = true -", - ) - .expect("valid config"); - - let err = docker_config_from_context(test_context(Some(&file))).unwrap_err(); - - assert!( - err.to_string() - .contains("invalid [openshell.drivers.docker] table") - ); - } - - #[test] - fn vm_config_reports_selected_invalid_driver_table() { - let file: config_file::ConfigFile = toml::from_str( - r#" -[openshell.drivers.vm] -mem_mib = "not-a-number" -"#, - ) - .expect("valid config"); - - let err = vm_config_from_context(test_context(Some(&file))).unwrap_err(); - - assert!( - err.to_string() - .contains("invalid [openshell.drivers.vm] table") - ); - } -} diff --git a/crates/openshell-server/src/compute/lease.rs b/crates/openshell-server/src/compute/lease.rs index bf58fae48..fbf039ec8 100644 --- a/crates/openshell-server/src/compute/lease.rs +++ b/crates/openshell-server/src/compute/lease.rs @@ -242,10 +242,10 @@ impl ReconcilerLease { /// Derive a stable replica identity for lease ownership. /// -/// Kubernetes sets `HOSTNAME` to the pod name, Docker sets it to the -/// container ID, and systemd units inherit the machine hostname. -/// `OPENSHELL_REPLICA_ID` allows explicit override. The UUID fallback -/// handles edge cases where neither env var is set. +/// Managed workloads commonly receive a stable runtime identity through +/// `HOSTNAME`, while systemd units inherit the machine hostname. +/// `OPENSHELL_REPLICA_ID` allows an explicit override. The UUID fallback +/// handles environments where neither variable is set. pub fn replica_id() -> String { std::env::var("OPENSHELL_REPLICA_ID") .or_else(|_| std::env::var("HOSTNAME")) @@ -505,7 +505,7 @@ mod tests { l2.renew(&mut guard2).await.unwrap(); // Replica-1 cannot re-acquire (lease exists) - let l1_retry = lease(store.clone(), "replica-1", Duration::from_secs(60)); + let l1_retry = lease(store.clone(), "replica-1", Duration::from_mins(1)); let err = l1_retry.try_acquire().await.unwrap_err(); assert!(matches!(err, LeaseError::AlreadyHeld)); diff --git a/crates/openshell-server/src/compute/mod.rs b/crates/openshell-server/src/compute/mod.rs index 3181f339a..70356e1ad 100644 --- a/crates/openshell-server/src/compute/mod.rs +++ b/crates/openshell-server/src/compute/mod.rs @@ -5,23 +5,13 @@ pub mod driver_config; pub mod lease; -#[cfg(all(not(target_os = "windows"), feature = "in-tree-compute-drivers"))] -pub mod vm; - -#[cfg(all(not(target_os = "windows"), feature = "in-tree-compute-drivers"))] -pub use openshell_driver_docker::DockerComputeConfig; -#[cfg(all(not(target_os = "windows"), feature = "in-tree-compute-drivers"))] -pub use openshell_driver_kubernetes::KubernetesComputeConfig; -#[cfg(all(not(target_os = "windows"), feature = "in-tree-compute-drivers"))] -pub use openshell_driver_podman::PodmanComputeConfig; -#[cfg(all(not(target_os = "windows"), feature = "in-tree-compute-drivers"))] -pub use vm::VmComputeConfig; +pub mod rootfs_tar; use crate::grpc::policy::SANDBOX_SETTINGS_OBJECT_TYPE; use crate::otel_tracing::TraceContextInterceptor; use crate::persistence::{ - DRAFT_CHUNK_OBJECT_TYPE, ObjectId, ObjectName, ObjectRecord, ObjectType, POLICY_OBJECT_TYPE, - Store, WriteCondition, + DRAFT_CHUNK_OBJECT_TYPE, ObjectCursor, ObjectId, ObjectName, ObjectRecord, ObjectType, + POLICY_OBJECT_TYPE, Store, WriteCondition, }; use crate::sandbox_index::SandboxIndex; use crate::sandbox_watch::SandboxWatchBus; @@ -30,9 +20,6 @@ use crate::tracing_bus::TracingLogBus; use futures::{Stream, StreamExt}; #[cfg(unix)] use hyper_util::rt::TokioIo; -use openshell_core::ComputeDriverKind; -#[cfg(target_os = "windows")] -use openshell_core::proto::SandboxPolicy; use openshell_core::proto::compute::v1::{ AuthenticateSandboxRequest, CreateSandboxRequest, DeleteSandboxRequest, DeleteWorkspaceRequest, DeleteWorkspaceResponse, DriverCondition, DriverPlatformEvent, DriverResourceRequirements, @@ -41,28 +28,18 @@ use openshell_core::proto::compute::v1::{ GatewayListenerRequirement as ProtoGatewayListenerRequirement, GetCapabilitiesRequest, GetGatewayListenerRequirementsRequest, GetGatewayListenerRequirementsResponse, GetSandboxRequest, GpuResourceRequirements as DriverGpuResourceRequirements, - ListSandboxesRequest, ResourceRequirements as DriverSandboxResourceRequirements, - StartSandboxRequest, StopSandboxRequest, ValidateSandboxCreateRequest, WatchSandboxesEvent, - WatchSandboxesRequest, compute_driver_client::ComputeDriverClient, - compute_driver_server::ComputeDriver, gateway_listener_requirement::Selector, - watch_sandboxes_event, + ListSandboxesRequest, ResourceCapabilities as DriverResourceCapabilities, + ResourceRequirements as DriverSandboxResourceRequirements, StartSandboxRequest, + StopSandboxRequest, ValidateSandboxCreateRequest, WatchSandboxesEvent, WatchSandboxesRequest, + compute_driver_client::ComputeDriverClient, compute_driver_server::ComputeDriver, + gateway_listener_requirement::Selector, watch_sandboxes_event, }; use openshell_core::proto::{ PlatformEvent, Sandbox, SandboxCondition, SandboxPhase, SandboxSpec, SandboxStatus, - SandboxTemplate, ServiceEndpoint, SshSession, + SandboxTemplate, SandboxWorkloadTemplate, ServiceEndpoint, SshSession, }; +use openshell_core::telemetry::TelemetryComputeDriver; use openshell_core::{ObjectLabels, ObjectWorkspace}; -#[cfg(all(not(target_os = "windows"), feature = "in-tree-compute-drivers"))] -use openshell_driver_docker::{ComputeDriverService as DockerDriverService, DockerComputeDriver}; -#[cfg(all(not(target_os = "windows"), feature = "in-tree-compute-drivers"))] -use openshell_driver_kubernetes::{ - ComputeDriverService as KubernetesDriverService, KubernetesComputeDriver, - OperatorNamespaceAllowlist, -}; -#[cfg(target_os = "windows")] -use openshell_driver_mxc::{ComputeDriverService as MxcDriverService, MxcComputeConfig}; -#[cfg(all(not(target_os = "windows"), feature = "in-tree-compute-drivers"))] -use openshell_driver_podman::{ComputeDriverService as PodmanDriverService, PodmanComputeDriver}; use prost::Message; use std::collections::HashMap; use std::fmt; @@ -70,7 +47,7 @@ use std::net::SocketAddr; use std::path::{Path, PathBuf}; use std::pin::Pin; use std::sync::{Arc, Mutex as StdMutex, Weak}; -use std::time::Duration; +use std::time::{Duration, Instant}; #[cfg(unix)] use tokio::net::UnixStream; use tokio::sync::{Mutex, watch}; @@ -99,7 +76,9 @@ mod traced_driver { use tonic::Status; use tracing::Instrument as _; - use super::SharedComputeDriver; + use super::{DriverWatchStream, SharedComputeDriver}; + + type TracedWatchStream = openshell_otel::TracedGrpcStream; #[derive(Clone)] pub(super) struct TracedDriver { @@ -112,45 +91,83 @@ mod traced_driver { Self { inner, name } } + fn span( + &self, + rpc: openshell_otel::ComputeDriverRpc, + sandbox_id: Option<&str>, + ) -> tracing::Span { + let span = tracing::info_span!( + "driver", + otel.name = rpc.operation, + otel.kind = "client", + otel.status_code = tracing::field::Empty, + driver.name = %self.name, + sandbox.id = tracing::field::Empty, + rpc.system.name = "grpc", + rpc.method = rpc.operation, + rpc.response.status_code = tracing::field::Empty, + error.type = tracing::field::Empty, + ); + if let Some(sandbox_id) = sandbox_id { + span.record("sandbox.id", sandbox_id); + } + span + } + /// Run one call across the driver boundary inside its span. /// /// Takes a closure rather than a future so the call cannot be built /// without going through here. pub(super) async fn call( &self, - operation: &'static str, + rpc: openshell_otel::ComputeDriverRpc, sandbox_id: Option<&str>, call: impl FnOnce(SharedComputeDriver) -> Fut, ) -> Result where Fut: Future>, { - let span = tracing::info_span!( - "driver", - otel.name = operation, - otel.kind = "client", - otel.status_code = tracing::field::Empty, - driver.name = %self.name, - sandbox.id = tracing::field::Empty, - grpc.code = tracing::field::Empty, - ); - if let Some(sandbox_id) = sandbox_id { - span.record("sandbox.id", sandbox_id); - } + let span = self.span(rpc, sandbox_id); let future = call(self.inner.clone()); async { let result = future.await; - if let Err(status) = &result { - let current = tracing::Span::current(); - crate::otel_tracing::mark_error(¤t); - current.record("grpc.code", status.code() as i32); + let current = tracing::Span::current(); + match &result { + Ok(_) => { + openshell_otel::record_grpc_status(¤t, tonic::Code::Ok); + } + Err(status) => { + openshell_otel::record_grpc_status(¤t, status.code()); + } } result } .instrument(span) .await } + + /// Open a driver watch while keeping the client span alive with the stream. + pub(super) async fn watch(&self) -> Result, Status> { + let span = self.span(openshell_otel::rpc::WATCH_SANDBOXES, None); + let result = self + .inner + .clone() + .watch_sandboxes(tonic::Request::new(super::WatchSandboxesRequest {})) + .instrument(span.clone()) + .await; + match result { + Ok(response) => { + let (metadata, inner, extensions) = response.into_parts(); + let stream: DriverWatchStream = Box::pin(TracedWatchStream::new(inner, span)); + Ok(tonic::Response::from_parts(metadata, stream, extensions)) + } + Err(status) => { + openshell_otel::record_grpc_status(&span, status.code()); + Err(status) + } + } + } } } @@ -287,14 +304,22 @@ pub struct ComputeDriverInfoSnapshot { pub gateway_manages_lifecycle: bool, /// Whether the driver authenticates driver-native sandbox credentials. pub supports_sandbox_authentication: bool, + /// Whether the driver reports runtime readiness without a supervisor session. + pub driver_reports_runtime_readiness: bool, + /// Static portable resource request forms from the startup capability snapshot. + pub resource_capabilities: Option, + /// Directory where rootfs tar files must be staged. + pub rootfs_tar_staging_dir: String, + /// Maximum rootfs tar file size in bytes. + pub rootfs_tar_max_bytes: u64, } /// Interval between store-vs-backend reconciliation sweeps. -const RECONCILE_INTERVAL: Duration = Duration::from_secs(60); +const RECONCILE_INTERVAL: Duration = Duration::from_mins(1); /// How long a sandbox can remain provisioning in the store without a /// corresponding backend resource before it is considered orphaned. -const ORPHAN_GRACE_PERIOD: Duration = Duration::from_secs(300); +const ORPHAN_GRACE_PERIOD: Duration = Duration::from_mins(5); // Re-export the shared error type under the name used by this module. pub use openshell_core::ComputeDriverError as ComputeError; @@ -306,8 +331,8 @@ pub struct ManagedDriverProcess { } impl ManagedDriverProcess { - #[cfg(all(unix, any(test, feature = "in-tree-compute-drivers")))] - pub(crate) fn new(child: tokio::process::Child, socket_path: PathBuf) -> Self { + #[cfg(unix)] + pub fn new(child: tokio::process::Child, socket_path: PathBuf) -> Self { Self { child: std::sync::Mutex::new(Some(child)), socket_path, @@ -404,14 +429,13 @@ pub struct AcquiredRemoteDriverEndpoint { } impl AcquiredRemoteDriverEndpoint { - #[cfg(any(test, feature = "in-tree-compute-drivers"))] - pub(crate) fn managed_builtin( - driver_kind: ComputeDriverKind, + pub fn managed( + name: impl Into, channel: Channel, driver_process: Arc, ) -> Self { Self { - name: driver_kind.as_str().to_string(), + name: name.into(), channel, driver_process: Some(driver_process), } @@ -575,6 +599,7 @@ impl ComputeDriver for RemoteComputeDriver { pub struct ComputeRuntime { driver: TracedDriver, driver_info: ComputeDriverInfoSnapshot, + telemetry_compute_driver: TelemetryComputeDriver, driver_process: Option>, default_image: String, store: Arc, @@ -586,14 +611,10 @@ pub struct ComputeRuntime { lifecycle_gates: Arc, gateway_listener_requirements: Vec, replica_id: String, - /// A1 policy side channel for the in-process MXC driver. `create_sandbox` - /// stages the typed `SandboxPolicy` here by sandbox id immediately before - /// dispatching to the driver, which consumes it. `None` for all other - /// drivers. The proto driver contract has no `policy` field and there is no - /// driver-side `GetSandboxConfig`, so this in-process map is how the policy - /// reaches the MXC backend without changing the cross-process contract. - #[cfg(target_os = "windows")] - mxc_policy_sink: Option>>>, + /// Gateway-issued staging slots for rootfs tar archives. Shared across + /// clones: `ServerState` holds `ComputeRuntime` by value, so a per-clone + /// table would make a token minted on one clone invisible to another. + rootfs_tar_staging: Arc, } impl fmt::Debug for ComputeRuntime { @@ -631,11 +652,9 @@ impl ComputeRuntime { compute_error_from_status(status) })? .into_inner(); - let driver_kind = driver_name.parse::().ok(); info!( configured_driver = %driver_name, advertised_driver = %capabilities.driver_name, - in_tree = driver_kind.is_some(), "Compute driver connected" ); let driver_info = ComputeDriverInfoSnapshot { @@ -644,6 +663,10 @@ impl ComputeRuntime { driver_version: capabilities.driver_version, gateway_manages_lifecycle: capabilities.gateway_manages_lifecycle, supports_sandbox_authentication: capabilities.supports_sandbox_authentication, + driver_reports_runtime_readiness: capabilities.driver_reports_runtime_readiness, + resource_capabilities: capabilities.resource_capabilities, + rootfs_tar_staging_dir: capabilities.rootfs_tar_staging_dir, + rootfs_tar_max_bytes: capabilities.rootfs_tar_max_bytes, }; let default_image = capabilities.default_image; let gateway_listener_requirements = match driver @@ -699,9 +722,16 @@ impl ComputeRuntime { } Err(status) => return Err(compute_error_from_status(status)), }; + let rootfs_tar_staging = Arc::new(rootfs_tar::RootfsTarStagingRegistry::new( + (!driver_info.rootfs_tar_staging_dir.is_empty()) + .then(|| PathBuf::from(&driver_info.rootfs_tar_staging_dir)), + driver_info.rootfs_tar_max_bytes, + )); + rootfs_tar_staging.sweep_orphans(); Ok(Self { driver: TracedDriver::new(driver, driver_name), driver_info, + telemetry_compute_driver: TelemetryComputeDriver::custom(), driver_process, default_image, store, @@ -713,8 +743,7 @@ impl ComputeRuntime { lifecycle_gates: Arc::new(LifecycleGateRegistry::default()), gateway_listener_requirements, replica_id: lease::replica_id(), - #[cfg(target_os = "windows")] - mxc_policy_sink: None, + rootfs_tar_staging, }) } @@ -743,62 +772,6 @@ impl ComputeRuntime { self.lifecycle_gates.entry_count() } - #[cfg(all(not(target_os = "windows"), feature = "in-tree-compute-drivers"))] - pub async fn new_docker( - config: openshell_core::Config, - docker_config: DockerComputeConfig, - store: Arc, - sandbox_index: SandboxIndex, - sandbox_watch_bus: SandboxWatchBus, - tracing_log_bus: TracingLogBus, - supervisor_sessions: Arc, - ) -> Result { - let driver = DockerComputeDriver::new(&config, &docker_config) - .await - .map_err(|err| ComputeError::Message(err.to_string()))?; - let driver: SharedComputeDriver = Arc::new(DockerDriverService::new_in_process(driver)); - Self::from_driver( - ComputeDriverKind::Docker.as_str().to_string(), - driver, - None, - store, - sandbox_index, - sandbox_watch_bus, - tracing_log_bus, - supervisor_sessions, - ) - .await - } - - #[cfg(all(not(target_os = "windows"), feature = "in-tree-compute-drivers"))] - pub async fn new_kubernetes( - config: KubernetesComputeConfig, - store: Arc, - sandbox_index: SandboxIndex, - sandbox_watch_bus: SandboxWatchBus, - tracing_log_bus: TracingLogBus, - supervisor_sessions: Arc, - shutdown_rx: watch::Receiver, - ) -> Result<(Self, Option), ComputeError> { - let driver = KubernetesComputeDriver::new(config, shutdown_rx) - .await - .map_err(|err| ComputeError::Message(err.to_string()))?; - let operator_allowlist_arc = driver.operator_allowlist().cloned(); - let driver: SharedComputeDriver = Arc::new(KubernetesDriverService::new_in_process(driver)); - let runtime = Self::from_driver( - ComputeDriverKind::Kubernetes.as_str().to_string(), - driver, - None, - store, - sandbox_index, - sandbox_watch_bus, - tracing_log_bus, - supervisor_sessions, - ) - .await?; - Ok((runtime, operator_allowlist_arc)) - } - pub(crate) async fn new_remote_driver( endpoint: AcquiredRemoteDriverEndpoint, store: Arc, @@ -821,64 +794,6 @@ impl ComputeRuntime { .await } - #[cfg(all(not(target_os = "windows"), feature = "in-tree-compute-drivers"))] - pub async fn new_podman( - config: PodmanComputeConfig, - store: Arc, - sandbox_index: SandboxIndex, - sandbox_watch_bus: SandboxWatchBus, - tracing_log_bus: TracingLogBus, - supervisor_sessions: Arc, - ) -> Result { - let driver = PodmanComputeDriver::new(config) - .await - .map_err(|err| ComputeError::Message(err.to_string()))?; - let driver: SharedComputeDriver = Arc::new(PodmanDriverService::new_in_process(driver)); - Self::from_driver( - ComputeDriverKind::Podman.as_str().to_string(), - driver, - None, - store, - sandbox_index, - sandbox_watch_bus, - tracing_log_bus, - supervisor_sessions, - ) - .await - } - - /// Construct a `ComputeRuntime` backed by the MXC compute driver. - /// - /// MXC is Windows-only, in-process, and self-reports `Ready` — there is - /// no supervisor session argument because no surrogate or relay is used. - #[cfg(target_os = "windows")] - pub async fn new_mxc( - mxc_config: MxcComputeConfig, - store: Arc, - sandbox_index: SandboxIndex, - sandbox_watch_bus: SandboxWatchBus, - tracing_log_bus: TracingLogBus, - supervisor_sessions: Arc, - ) -> Result { - let backend = openshell_driver_mxc::MxcComputeBackend::new(mxc_config); - // Grab the A1 policy side channel before moving `backend` into the service. - let sink = backend.policy_sink(); - let service: SharedComputeDriver = Arc::new(MxcDriverService::new(backend)); - let mut runtime = Self::from_driver( - ComputeDriverKind::Mxc.as_str().to_string(), - service, - None, - store, - sandbox_index, - sandbox_watch_bus, - tracing_log_bus, - supervisor_sessions, - ) - .await?; - runtime.mxc_policy_sink = Some(sink); - Ok(runtime) - } - #[must_use] pub fn default_image(&self) -> &str { &self.default_image @@ -890,12 +805,16 @@ impl ComputeRuntime { } #[must_use] - pub fn driver_kind(&self) -> Option { - self.driver_info.name.parse().ok() + pub(crate) fn rootfs_tar_staging(&self) -> &rootfs_tar::RootfsTarStagingRegistry { + &self.rootfs_tar_staging } + /// The `template.driver_config` key whose block this gateway forwards. + /// + /// This is the *configured* driver name, which is not necessarily the name + /// the driver reports for itself in `driver_info.driver_name`. #[must_use] - pub fn selected_driver_name(&self) -> &str { + pub fn configured_driver_name(&self) -> &str { &self.driver_info.name } @@ -914,13 +833,29 @@ impl ComputeRuntime { credential: credential.to_string(), }; self.driver - .call("driver.authenticate_sandbox", None, |driver| async move { - driver.authenticate_sandbox(Request::new(request)).await - }) + .call( + openshell_otel::rpc::AUTHENTICATE_SANDBOX, + None, + |driver| async move { driver.authenticate_sandbox(Request::new(request)).await }, + ) .await .map(|response| response.into_inner().sandbox_id) } + #[must_use] + pub(crate) fn telemetry_compute_driver(&self) -> TelemetryComputeDriver { + self.telemetry_compute_driver + } + + #[must_use] + pub(crate) fn with_telemetry_compute_driver( + mut self, + telemetry_compute_driver: TelemetryComputeDriver, + ) -> Self { + self.telemetry_compute_driver = telemetry_compute_driver; + self + } + #[must_use] pub(crate) fn gateway_listener_requirements(&self) -> &[GatewayListenerRequirement] { &self.gateway_listener_requirements @@ -930,11 +865,15 @@ impl ComputeRuntime { let workspace = workspace.to_string(); match self .driver - .call("driver.ensure_workspace", None, |driver| async move { - driver - .ensure_workspace(Request::new(EnsureWorkspaceRequest { workspace })) - .await - }) + .call( + openshell_otel::rpc::ENSURE_WORKSPACE, + None, + |driver| async move { + driver + .ensure_workspace(Request::new(EnsureWorkspaceRequest { workspace })) + .await + }, + ) .await { Ok(_) => Ok(()), @@ -947,11 +886,15 @@ impl ComputeRuntime { let workspace = workspace.to_string(); match self .driver - .call("driver.delete_workspace", None, |driver| async move { - driver - .delete_workspace(Request::new(DeleteWorkspaceRequest { workspace })) - .await - }) + .call( + openshell_otel::rpc::DELETE_WORKSPACE, + None, + |driver| async move { + driver + .delete_workspace(Request::new(DeleteWorkspaceRequest { workspace })) + .await + }, + ) .await { Ok(_) => Ok(()), @@ -961,11 +904,17 @@ impl ComputeRuntime { } pub async fn validate_sandbox_create(&self, sandbox: &Sandbox) -> Result<(), Status> { - let driver_sandbox = driver_sandbox_from_public(sandbox, &self.driver_info.name) + let mut driver_sandbox = driver_sandbox_from_public(sandbox, &self.driver_info.name) .map_err(|status| *status)?; + // Peek, never consume: create runs the same path immediately after and + // must still find the token. + if let Some(token) = take_staging_token(&mut driver_sandbox) { + let staged = self.rootfs_tar_staging.peek(&token)?; + set_rootfs_tar_path(&mut driver_sandbox, &staged); + } self.driver .call( - "driver.validate_sandbox_create", + openshell_otel::rpc::VALIDATE_SANDBOX_CREATE, Some(sandbox.object_id()), |driver| async move { driver @@ -986,12 +935,25 @@ impl ComputeRuntime { await_main_process_attachment: bool, ) -> Result { let sandbox_id = sandbox.object_id().to_string(); + let mut sandbox = sandbox; + + // Strip the staging token from the public sandbox before anything + // persists it: the object store copy is readable by every member of the + // workspace, and the token is a bearer credential for the staged + // archive. The driver gets the resolved path instead, on its own copy. + let staging_token = take_public_staging_token(&mut sandbox, &self.driver_info.name); + let mut staged = staging_token + .map(|token| self.rootfs_tar_staging.consume(&token)) + .transpose()?; + let mut driver_sandbox = driver_sandbox_from_public(&sandbox, &self.driver_info.name) .map_err(|status| *status)?; + if let Some(staged) = staged.as_ref() { + set_rootfs_tar_path(&mut driver_sandbox, staged.path()); + } // Create with MustCreate condition to prevent duplicate creation race self.sandbox_index.update_from_sandbox(&sandbox); - let mut sandbox = sandbox; let labels_map = sandbox.object_labels(); let labels_json = if labels_map.as_ref().is_none_or(HashMap::is_empty) { None @@ -1035,20 +997,10 @@ impl ComputeRuntime { if let Some(spec) = driver_sandbox.spec.as_mut() { spec.await_main_process_attachment = await_main_process_attachment; } - // A1: stage the typed SandboxPolicy out-of-band into the MXC backend's - // side channel, keyed by sandbox id (== DriverSandbox.id), immediately - // before dispatch. The driver removes/consumes it in create_sandbox. The - // proto driver contract has no policy field, so this is the only path. - #[cfg(target_os = "windows")] - if let Some(sink) = &self.mxc_policy_sink - && let Some(p) = sandbox.spec.as_ref().and_then(|s| s.policy.clone()) - { - sink.lock().await.insert(sandbox_id.clone(), p); - } match self .driver .call( - "driver.create_sandbox", + openshell_otel::rpc::CREATE_SANDBOX, Some(sandbox.object_id()), |driver| async move { driver @@ -1061,6 +1013,12 @@ impl ComputeRuntime { .await { Ok(_) => { + // The driver now owns the staged archive and removes the + // request directory once it has built the disk. Every other + // arm lets the guard drop and clean up. + if let Some(staged) = staged.as_mut() { + staged.disarm(); + } self.sandbox_watch_bus.notify(sandbox.object_id()); if let Some(metadata) = sandbox.metadata.as_mut() { metadata.resource_version = result.resource_version; @@ -1197,18 +1155,22 @@ impl ComputeRuntime { ) -> Result { let result = self .driver - .call("driver.stop_sandbox", Some(&sandbox_id), |driver| { - let sandbox_id = sandbox_id.clone(); - let sandbox_name = sandbox_name.clone(); - async move { - driver - .stop_sandbox(Request::new(StopSandboxRequest { - sandbox_id, - sandbox_name, - })) - .await - } - }) + .call( + openshell_otel::rpc::STOP_SANDBOX, + Some(&sandbox_id), + |driver| { + let sandbox_id = sandbox_id.clone(); + let sandbox_name = sandbox_name.clone(); + async move { + driver + .stop_sandbox(Request::new(StopSandboxRequest { + sandbox_id, + sandbox_name, + })) + .await + } + }, + ) .await; match result { @@ -1357,18 +1319,22 @@ impl ComputeRuntime { ) -> Result { let result = self .driver - .call("driver.start_sandbox", Some(&sandbox_id), |driver| { - let sandbox_id = sandbox_id.clone(); - let sandbox_name = sandbox_name.clone(); - async move { - driver - .start_sandbox(Request::new(StartSandboxRequest { - sandbox_id, - sandbox_name, - })) - .await - } - }) + .call( + openshell_otel::rpc::START_SANDBOX, + Some(&sandbox_id), + |driver| { + let sandbox_id = sandbox_id.clone(); + let sandbox_name = sandbox_name.clone(); + async move { + driver + .start_sandbox(Request::new(StartSandboxRequest { + sandbox_id, + sandbox_name, + })) + .await + } + }, + ) .await; match result { @@ -1418,11 +1384,14 @@ impl ComputeRuntime { || driver_snapshot_confirms_stopped(&snapshot); let suspension_progressing = expected_stopped && driver_snapshot_confirms_stopping(&snapshot); - if suspension_progressing { - // The Kubernetes controller has accepted the stop and - // is waiting for its pod to terminate. Preserve the - // durable transition so a later watch event can complete - // it instead of claiming the sandbox is running again. + let runtime_restart_during_stop = + expected_stopped && driver_snapshot_reports_runtime_restart(&snapshot); + if suspension_progressing || runtime_restart_during_stop { + // The backend has accepted the stop but has not finished + // terminating the sandbox. A runtime restart can likewise + // be the expected exit from an in-flight stop. Preserve the + // durable transition so completion or recovery, rather than + // a watch snapshot, determines its terminal state. debug!(sandbox_id, "Sandbox stop is still progressing"); } else if backend_phase == SandboxPhase::Error || observed_stopped == expected_stopped @@ -1467,7 +1436,12 @@ impl ComputeRuntime { match self .store .update_message_cas::(&sandbox_id, expected_resource_version, |sandbox| { - apply_driver_snapshot(sandbox, snapshot, session_connected); + apply_driver_snapshot( + sandbox, + snapshot, + session_connected, + self.driver_info.driver_reports_runtime_readiness, + ); }) .await { @@ -1644,7 +1618,7 @@ impl ComputeRuntime { let result = self .driver .call( - "driver.delete_sandbox", + openshell_otel::rpc::DELETE_SANDBOX, Some(transition.deleting.object_id()), |driver| { let sandbox_id = transition.deleting.object_id().to_string(); @@ -1915,7 +1889,14 @@ impl ComputeRuntime { sandbox_id, deleting_resource_version, "reconcile observed backend snapshot", - |sandbox| apply_driver_snapshot(sandbox, &snapshot, session_connected), + |sandbox| { + apply_driver_snapshot( + sandbox, + &snapshot, + session_connected, + self.driver_info.driver_reports_runtime_readiness, + ); + }, ) .await; } @@ -2127,7 +2108,7 @@ impl ComputeRuntime { let watch_runtime = runtime.clone(); let watch_shutdown = shutdown_rx.clone(); tokio::spawn(async move { - watch_runtime.watch_loop(watch_shutdown).await; + Box::pin(watch_runtime.watch_loop(watch_shutdown)).await; }); tokio::spawn(async move { runtime.reconcile_loop(shutdown_rx).await; @@ -2199,18 +2180,22 @@ impl ComputeRuntime { let sandbox_name = sandbox.object_name().to_string(); match self .driver - .call("driver.stop_sandbox", Some(&sandbox_id), |driver| { - let sandbox_id = sandbox_id.clone(); - let sandbox_name = sandbox_name.clone(); - async move { - driver - .stop_sandbox(Request::new(StopSandboxRequest { - sandbox_id, - sandbox_name, - })) - .await - } - }) + .call( + openshell_otel::rpc::STOP_SANDBOX, + Some(&sandbox_id), + |driver| { + let sandbox_id = sandbox_id.clone(); + let sandbox_name = sandbox_name.clone(); + async move { + driver + .stop_sandbox(Request::new(StopSandboxRequest { + sandbox_id, + sandbox_name, + })) + .await + } + }, + ) .await { Ok(_) => { @@ -2307,18 +2292,22 @@ impl ComputeRuntime { let sandbox_name = sandbox.object_name().to_string(); match self .driver - .call("driver.start_sandbox", Some(&sandbox_id), |driver| { - let sandbox_id = sandbox_id.clone(); - let sandbox_name = sandbox_name.clone(); - async move { - driver - .start_sandbox(Request::new(StartSandboxRequest { - sandbox_id, - sandbox_name, - })) - .await - } - }) + .call( + openshell_otel::rpc::START_SANDBOX, + Some(&sandbox_id), + |driver| { + let sandbox_id = sandbox_id.clone(); + let sandbox_name = sandbox_name.clone(); + async move { + driver + .start_sandbox(Request::new(StartSandboxRequest { + sandbox_id, + sandbox_name, + })) + .await + } + }, + ) .await { Ok(_) => { @@ -2432,7 +2421,7 @@ impl ComputeRuntime { match self .driver .call( - "driver.stop_sandbox", + openshell_otel::rpc::STOP_SANDBOX, Some(&sandbox_id), |driver| async move { driver @@ -2479,7 +2468,7 @@ impl ComputeRuntime { if let Err(err) = self .driver .call( - "driver.start_sandbox", + openshell_otel::rpc::START_SANDBOX, Some(&sandbox_id), |driver| async move { driver @@ -2653,7 +2642,7 @@ impl ComputeRuntime { let runtime = self.clone(); let watch_cancel = cancel_rx.clone(); let watch_handle = tokio::spawn(async move { - runtime.watch_loop(watch_cancel).await; + Box::pin(runtime.watch_loop(watch_cancel)).await; }); let runtime = self.clone(); @@ -2701,17 +2690,7 @@ impl ComputeRuntime { async fn watch_loop(self: Arc, mut cancel: watch::Receiver) { loop { - // Spans the stream open, not its lifetime: the future resolves - // once the driver accepts the watch. - let mut stream = match self - .driver - .call("driver.watch_sandboxes", None, |driver| async move { - driver - .watch_sandboxes(Request::new(WatchSandboxesRequest {})) - .await - }) - .await - { + let mut stream = match self.driver.watch().await { Ok(response) => response.into_inner(), Err(err) => { warn!(error = %err, "Compute driver watch stream failed to start"); @@ -2779,13 +2758,20 @@ impl ComputeRuntime { )] async fn reconcile_store_with_backend(&self, grace_period: Duration) -> Result<(), String> { let sweep_started_at_ms = openshell_core::time::now_ms(); + // Reclaims staging directories whose driver failed before its own + // cleanup ran, which the token table cannot see once consumed. + self.rootfs_tar_staging.sweep_orphans(); let backend_sandboxes = self .driver - .call("driver.list_sandboxes", None, |driver| async move { - driver - .list_sandboxes(Request::new(ListSandboxesRequest {})) - .await - }) + .call( + openshell_otel::rpc::LIST_SANDBOXES, + None, + |driver| async move { + driver + .list_sandboxes(Request::new(ListSandboxesRequest {})) + .await + }, + ) .await .map_err(|e| e.to_string()) .inspect_err(|_| crate::otel_tracing::mark_error(&tracing::Span::current()))? @@ -2874,7 +2860,7 @@ impl ComputeRuntime { match event.payload { Some(watch_sandboxes_event::Payload::Sandbox(sandbox)) => { if let Some(sandbox) = sandbox.sandbox { - self.apply_sandbox_update(sandbox).await?; + Box::pin(self.apply_sandbox_update(sandbox)).await?; } } Some(watch_sandboxes_event::Payload::Deleted(deleted)) => { @@ -2899,13 +2885,67 @@ impl ComputeRuntime { Ok(()) } - async fn apply_sandbox_update(&self, incoming: DriverSandbox) -> Result<(), String> { - let _guard = self.sync_lock.lock().await; - let existing = self + async fn apply_sandbox_update(&self, mut incoming: DriverSandbox) -> Result<(), String> { + let guard = self.sync_lock.lock().await; + let mut existing = self .store .get(Sandbox::object_type(), &incoming.id) .await .map_err(|e| e.to_string())?; + let existing_sandbox = existing.as_ref().map(decode_sandbox_record).transpose()?; + let existing_phase = existing_sandbox + .as_ref() + .map_or(SandboxPhase::Unknown, |sandbox| { + SandboxPhase::try_from(sandbox.phase()).unwrap_or(SandboxPhase::Unknown) + }); + + if !driver_snapshot_reports_terminal_container_exit(&incoming) + || existing_phase != SandboxPhase::Starting + { + return self.apply_sandbox_update_locked(incoming, existing).await; + } + + // A terminal snapshot can already be queued when StartSandbox moves + // the durable phase to Starting. Release the global watch lock, wait + // for that lifecycle operation, and then reread both the driver and + // store before applying the terminal observation. Taking the + // per-sandbox gate only for this ambiguous phase avoids delaying + // unrelated watch events behind slow lifecycle operations. + let existing_name = existing_sandbox.as_ref().map_or_else( + || incoming.name.clone(), + |sandbox| sandbox.object_name().to_string(), + ); + drop(guard); + let _lifecycle_guard = self.lifecycle_gates.lock_for(&incoming.id).await; + let observed = self.get_driver_sandbox(&incoming.id, &existing_name).await; + let _guard = self.sync_lock.lock().await; + existing = self + .store + .get(Sandbox::object_type(), &incoming.id) + .await + .map_err(|e| e.to_string())?; + let current_phase = existing + .as_ref() + .map(decode_sandbox_record) + .transpose()? + .map_or(SandboxPhase::Unknown, |sandbox| { + SandboxPhase::try_from(sandbox.phase()).unwrap_or(SandboxPhase::Unknown) + }); + + match observed { + Ok(Some(live)) if live.id == incoming.id && live.status.is_some() => incoming = live, + Ok(Some(_) | None) | Err(_) + if matches!(current_phase, SandboxPhase::Starting | SandboxPhase::Ready) => + { + warn!( + sandbox_id = %incoming.id, + "Could not validate terminal driver snapshot; retaining current sandbox state" + ); + return Ok(()); + } + Ok(Some(_) | None) | Err(_) => {} + } + self.apply_sandbox_update_locked(incoming, existing).await } @@ -2955,7 +2995,14 @@ impl ComputeRuntime { .update_message_cas::( &incoming.id, expected_resource_version, - |sandbox| apply_driver_snapshot(sandbox, &incoming, session_connected), + |sandbox| { + apply_driver_snapshot( + sandbox, + &incoming, + session_connected, + self.driver_info.driver_reports_runtime_readiness, + ); + }, ) .await .map_err(|e| match e { @@ -3183,6 +3230,17 @@ impl ComputeRuntime { else { return Ok(()); }; + let phase = SandboxPhase::try_from(sandbox.phase()).unwrap_or(SandboxPhase::Unknown); + if matches!( + phase, + SandboxPhase::Deleting | SandboxPhase::Stopping | SandboxPhase::Stopped + ) { + // Lifecycle shutdown intentionally discards the canonical process + // result. Finalization must still acknowledge that discarded + // result so the supervisor can exit before the compute backend's + // termination grace period expires. + return Ok(()); + } let Some(status) = sandbox.status.as_ref() else { return Err("main-process exit has not been reported".to_string()); }; @@ -3234,6 +3292,13 @@ impl ComputeRuntime { .await .map_err(|e| e.to_string())?; if let Some(sandbox) = sandbox.as_ref() { + // The watcher told us this sandbox's compute resource is gone, so + // no request-side DeleteSandbox call is coming — release + // driver-owned resources in the background. Watch events are + // processed sequentially, so this must not block on the driver + // call itself, only on the (instant, non-blocking) decision to + // make it. + self.spawn_driver_sandbox_cleanup(sandbox.object_id(), sandbox.object_name()); self.cleanup_sandbox_owned_records(sandbox).await?; } @@ -3285,26 +3350,142 @@ impl ComputeRuntime { Ok(()) } + /// Best-effort driver-side cleanup for a sandbox discovered gone + /// out-of-band — a watch deletion event, or the periodic prune sweep + /// finding no matching driver resource — rather than through an + /// explicit `DeleteSandbox` request. + /// + /// Skips the driver call entirely if a request-side lifecycle operation + /// (e.g. an in-flight explicit delete) already holds this sandbox's + /// lifecycle gate: that operation already owns driver-side cleanup for + /// it, and calling `DeleteSandbox` again here would race its own + /// in-flight call. + /// + /// The gate check is synchronous, but the actual `DeleteSandbox` RPC is + /// always deferred to a background task, never awaited inline: both + /// call sites run while holding a broader lock (the watch loop's + /// sequential event processing; the prune sweep's gateway-wide + /// `sync_lock`), and a slow or stuck driver call must never block that + /// wider scope. The gate itself is held for the background call's + /// duration, so this still can't race a concurrent request-side + /// operation — only the potentially-slow RPC is backgrounded. + fn spawn_driver_sandbox_cleanup(&self, sandbox_id: &str, sandbox_name: &str) { + let gate = self.lifecycle_gates.gate_for(sandbox_id); + let Ok(guard) = gate.try_lock_owned() else { + debug!( + sandbox_id, + sandbox_name, + "Skipping driver cleanup while a lifecycle operation is already in flight for this sandbox" + ); + return; + }; + + let runtime = self.clone(); + let sandbox_id = sandbox_id.to_string(); + let sandbox_name = sandbox_name.to_string(); + tokio::spawn(async move { + let _guard = guard; + runtime + .call_driver_delete_sandbox(&sandbox_id, &sandbox_name) + .await; + }); + } + + /// `DeleteSandbox` is idempotent: drivers must reclaim owned + /// secrets/volumes/etc. even when the underlying compute resource is + /// already gone. Failures here are logged, not propagated — callers + /// already consider this sandbox gone, so a driver hiccup must not + /// block store cleanup. + async fn call_driver_delete_sandbox(&self, sandbox_id: &str, sandbox_name: &str) { + let result = self + .driver + .call( + openshell_otel::rpc::DELETE_SANDBOX, + Some(sandbox_id), + |driver| { + let sandbox_id = sandbox_id.to_string(); + let sandbox_name = sandbox_name.to_string(); + async move { + driver + .delete_sandbox(Request::new(DeleteSandboxRequest { + sandbox_id, + sandbox_name, + })) + .await + } + }, + ) + .await; + + if let Err(status) = result { + warn!( + sandbox_id, + sandbox_name, + error = %status, + "Failed to release driver-owned resources while cleaning up a sandbox discovered gone out-of-band" + ); + } + } + async fn cleanup_sandbox_ssh_sessions( &self, sandbox_id: &str, workspace: &str, ) -> Result<(), String> { - let records = self - .store - .list(SshSession::object_type(), workspace, 1000, 0) - .await - .map_err(|e| format!("list SSH sessions: {e}"))?; + let started = Instant::now(); + let mut cursor = None; + let mut scanned = 0_usize; + let mut decode_failures = 0_usize; + let mut session_ids = Vec::new(); - for record in records { - if let Ok(session) = SshSession::decode(record.payload.as_slice()) - && session.sandbox_id == sandbox_id - { - self.store - .delete(SshSession::object_type(), session.object_id()) - .await - .map_err(|e| format!("delete SSH session {}: {e}", session.object_id()))?; + loop { + let records = self + .store + .list_after( + SshSession::object_type(), + workspace, + cursor.as_ref(), + LIFECYCLE_SWEEP_PAGE_SIZE, + ) + .await + .map_err(|e| format!("list SSH sessions: {e}"))?; + let page_len = records.len(); + scanned += page_len; + + cursor = records.last().map(ObjectCursor::from); + for record in records { + match SshSession::decode(record.payload.as_slice()) { + Ok(session) if session.sandbox_id == sandbox_id => { + session_ids.push(session.object_id().to_string()); + } + Ok(_) => {} + Err(_) => decode_failures += 1, + } } + + if page_len < LIFECYCLE_SWEEP_PAGE_SIZE as usize { + break; + } + } + + let matched = session_ids.len(); + let deleted = self + .store + .delete_many(SshSession::object_type(), &session_ids) + .await + .map_err(|e| format!("delete sandbox SSH sessions: {e}"))?; + + if matched > 0 || decode_failures > 0 { + debug!( + sandbox_id, + workspace, + scanned, + matched, + deleted, + decode_failures, + elapsed_ms = started.elapsed().as_millis(), + "Sandbox SSH session cleanup complete" + ); } Ok(()) @@ -3538,6 +3719,14 @@ impl ComputeRuntime { age_secs = age_ms / 1000, "Removing sandbox from store after it disappeared from the compute driver snapshot" ); + // The driver's own snapshot never reported this sandbox, so no + // request-side DeleteSandbox call is coming for it either — release + // driver-owned resources in the background. This function holds + // `sync_lock` (the gateway-wide state guard) through the rest of its + // body, so the driver call must not be awaited here: doing so would + // block every other sandbox operation gateway-wide on a single, + // potentially slow or stuck driver RPC. + self.spawn_driver_sandbox_cleanup(&sandbox_id, &sandbox_name); self.apply_deleted_if_version_locked(&sandbox, expected_resource_version) .await } @@ -3549,18 +3738,22 @@ impl ComputeRuntime { ) -> Result, String> { match self .driver - .call("driver.get_sandbox", Some(sandbox_id), |driver| { - let sandbox_id = sandbox_id.to_string(); - let sandbox_name = sandbox_name.to_string(); - async move { - driver - .get_sandbox(Request::new(GetSandboxRequest { - sandbox_id, - sandbox_name, - })) - .await - } - }) + .call( + openshell_otel::rpc::GET_SANDBOX, + Some(sandbox_id), + |driver| { + let sandbox_id = sandbox_id.to_string(); + let sandbox_name = sandbox_name.to_string(); + async move { + driver + .get_sandbox(Request::new(GetSandboxRequest { + sandbox_id, + sandbox_name, + })) + .await + } + }, + ) .await { Ok(response) => { @@ -3583,7 +3776,17 @@ impl ComputeRuntime { fn apply_main_process_exit(sandbox: &mut Sandbox, instance_id: &str, exit_code: i32) { let sandbox_name = sandbox.object_name().to_string(); - let preserve_infrastructure_error = sandbox.phase() == SandboxPhase::Error as i32; + // A driver can observe the container exit before the supervisor's + // authoritative main-process report arrives. In that ordering, + // ContainerExited is only a provisional classification: replace it with + // the canonical process result once its exit code is known. Preserve all + // other infrastructure errors. + let preserve_infrastructure_error = sandbox.phase() == SandboxPhase::Error as i32 + && !sandbox.status.as_ref().is_some_and(|status| { + status.conditions.iter().any(|condition| { + condition.r#type == "Ready" && condition.reason == "ContainerExited" + }) + }); let status = sandbox.status.get_or_insert_with(|| SandboxStatus { sandbox_name: sandbox_name.clone(), ..Default::default() @@ -3715,6 +3918,7 @@ fn driver_sandbox_spec_from_public( .as_ref() .map(|template| driver_sandbox_template_from_public(template, driver_name)) .transpose()?, + policy: spec.policy.clone(), resource_requirements: spec.resource_requirements.as_ref().map(|requirements| { DriverSandboxResourceRequirements { gpu: requirements @@ -3742,9 +3946,80 @@ fn driver_sandbox_template_from_public( resources: extract_typed_resources(&template.resources), platform_config: build_platform_config(template), driver_config: select_driver_config(&template.driver_config, driver_name)?, + user_namespaces: template.user_namespaces, }) } +/// Remove the staging token from a driver-native sandbox, if present. +/// +/// The driver config here has already been narrowed to the selected driver's +/// block, so the token sits at the top level. +fn take_staging_token(driver_sandbox: &mut DriverSandbox) -> Option { + let config = driver_sandbox + .spec + .as_mut()? + .template + .as_mut()? + .driver_config + .as_mut()?; + match config.fields.remove(rootfs_tar::STAGING_TOKEN_FIELD)?.kind { + Some(prost_types::value::Kind::StringValue(token)) => Some(token), + _ => None, + } +} + +/// Remove the staging token from the public sandbox, under the driver's key. +/// +/// Called before the sandbox is persisted so the token never reaches the object +/// store, where every workspace member could read it back. +fn take_public_staging_token(sandbox: &mut Sandbox, driver_name: &str) -> Option { + let config = sandbox + .spec + .as_mut()? + .template + .as_mut()? + .driver_config + .as_mut()?; + let Some(prost_types::value::Kind::StructValue(driver_config)) = config + .fields + .get_mut(driver_name) + .and_then(|v| v.kind.as_mut()) + else { + return None; + }; + match driver_config + .fields + .remove(rootfs_tar::STAGING_TOKEN_FIELD)? + .kind + { + Some(prost_types::value::Kind::StringValue(token)) => Some(token), + _ => None, + } +} + +/// Substitute the gateway-resolved archive path into the driver-native copy. +/// +/// This is the only writer of `rootfs_tar_path`; a caller-supplied value is +/// rejected in request validation before it ever reaches here. +fn set_rootfs_tar_path(driver_sandbox: &mut DriverSandbox, path: &Path) { + let Some(template) = driver_sandbox + .spec + .as_mut() + .and_then(|spec| spec.template.as_mut()) + else { + return; + }; + let config = template.driver_config.get_or_insert_with(Default::default); + config.fields.insert( + rootfs_tar::ROOTFS_TAR_PATH_FIELD.to_string(), + prost_types::Value { + kind: Some(prost_types::value::Kind::StringValue( + path.to_string_lossy().into_owned(), + )), + }, + ); +} + fn select_driver_config( config: &Option, driver_name: &str, @@ -3847,19 +4122,6 @@ fn build_platform_config(template: &SandboxTemplate) -> Option &'static str { + "sandbox_workload_template" + } +} + fn compute_error_from_status(status: Status) -> ComputeError { match status.code() { Code::AlreadyExists => ComputeError::AlreadyExists, @@ -4003,14 +4271,22 @@ fn public_status_from_driver( } } -fn apply_driver_snapshot(sandbox: &mut Sandbox, incoming: &DriverSandbox, session_connected: bool) { +fn apply_driver_snapshot( + sandbox: &mut Sandbox, + incoming: &DriverSandbox, + session_connected: bool, + driver_reports_runtime_readiness: bool, +) { let old_phase = SandboxPhase::try_from(sandbox.phase()).unwrap_or(SandboxPhase::Unknown); let sandbox_name = &incoming.name; // Infrastructure errors and successful main-process completions are // sticky until an explicit lifecycle operation changes desired state. A - // late backend snapshot must not revive either result. - if matches!(old_phase, SandboxPhase::Error | SandboxPhase::Completed) { + // late signal-exit snapshot must also not overwrite the terminal reason + // recorded by a completed explicit stop. + if matches!(old_phase, SandboxPhase::Error | SandboxPhase::Completed) + || (old_phase == SandboxPhase::Stopped && driver_snapshot_reports_runtime_restart(incoming)) + { if let Some(metadata) = sandbox.metadata.as_mut() { metadata.name.clone_from(sandbox_name); } @@ -4035,7 +4311,11 @@ fn apply_driver_snapshot(sandbox: &mut Sandbox, incoming: &DriverSandbox, sessio (phase, status) }, |incoming_status| { - let composed = ComposedPhase::new(incoming_status, session_connected); + let composed = ComposedPhase::new( + incoming_status, + session_connected, + driver_reports_runtime_readiness, + ); let mut status = Some(public_status_from_driver( incoming_status, composed.phase, @@ -4047,6 +4327,13 @@ fn apply_driver_snapshot(sandbox: &mut Sandbox, incoming: &DriverSandbox, sessio ); phase = match old_phase { + // SIGTERM-driven runtime exits are reported as a runtime restart by + // Docker and Podman. While an explicit stop owns this durable + // transition, preserve Stopping so the stop result chooses whether + // the sandbox actually reached Stopped or needs recovery. + SandboxPhase::Stopping if driver_snapshot_reports_runtime_restart(incoming) => { + SandboxPhase::Stopping + } SandboxPhase::Stopping if phase == SandboxPhase::Stopped || driver_snapshot_confirms_stopped(incoming) => { @@ -4128,6 +4415,29 @@ fn driver_snapshot_confirms_stopped(incoming: &DriverSandbox) -> bool { }) } +fn driver_snapshot_reports_terminal_container_exit(incoming: &DriverSandbox) -> bool { + incoming.status.as_ref().is_some_and(|status| { + status.conditions.iter().any(|condition| { + condition.status.eq_ignore_ascii_case("false") + && matches!( + condition.reason.to_ascii_lowercase().as_str(), + "containerexited" | "containerstopped" | "containerruntimerestart" + ) + }) + }) +} + +fn driver_snapshot_reports_runtime_restart(incoming: &DriverSandbox) -> bool { + incoming.status.as_ref().is_some_and(|status| { + status.conditions.iter().any(|condition| { + condition.status.eq_ignore_ascii_case("false") + && condition + .reason + .eq_ignore_ascii_case("ContainerRuntimeRestart") + }) + }) +} + fn driver_snapshot_confirms_stopping(incoming: &DriverSandbox) -> bool { incoming.status.as_ref().is_some_and(|status| { status.conditions.iter().any(|condition| { @@ -4168,7 +4478,11 @@ struct ComposedPhase { } impl ComposedPhase { - fn new(incoming_status: &DriverSandboxStatus, session_connected: bool) -> Self { + fn new( + incoming_status: &DriverSandboxStatus, + session_connected: bool, + driver_reports_runtime_readiness: bool, + ) -> Self { let backend_phase = derive_phase(Some(incoming_status)); // A live supervisor session is a stronger readiness signal than the backend phase. // set_supervisor_session_state may have already promoted the store record to Ready @@ -4176,13 +4490,15 @@ impl ComposedPhase { // backend phase overwrite it. let phase = match backend_phase { SandboxPhase::Error | SandboxPhase::Deleting | SandboxPhase::Stopped => backend_phase, + SandboxPhase::Ready if driver_reports_runtime_readiness => SandboxPhase::Ready, _ if session_connected => SandboxPhase::Ready, _ => SandboxPhase::Provisioning, }; Self { phase, session_connected, - backend_ready_without_session: backend_phase == SandboxPhase::Ready + backend_ready_without_session: !driver_reports_runtime_readiness + && backend_phase == SandboxPhase::Ready && !session_connected, } } @@ -4466,6 +4782,10 @@ impl ComputeDriver for NoopTestDriver { default_image: "openshell/sandbox:test".to_string(), gateway_manages_lifecycle: false, supports_sandbox_authentication: self.sandbox_authentication.is_some(), + driver_reports_runtime_readiness: false, + resource_capabilities: None, + rootfs_tar_staging_dir: String::new(), + rootfs_tar_max_bytes: 0, }, )) } @@ -4609,7 +4929,12 @@ pub async fn new_test_runtime_with_driver( driver_version: "test".to_string(), gateway_manages_lifecycle: false, supports_sandbox_authentication, + driver_reports_runtime_readiness: false, + resource_capabilities: None, + rootfs_tar_staging_dir: String::new(), + rootfs_tar_max_bytes: 0, }, + telemetry_compute_driver: TelemetryComputeDriver::custom(), driver_process: None, default_image: "openshell/sandbox:test".to_string(), store, @@ -4621,8 +4946,7 @@ pub async fn new_test_runtime_with_driver( lifecycle_gates: Arc::new(LifecycleGateRegistry::default()), gateway_listener_requirements: Vec::new(), replica_id: "test-replica".to_string(), - #[cfg(target_os = "windows")] - mxc_policy_sink: None, + rootfs_tar_staging: Arc::new(rootfs_tar::RootfsTarStagingRegistry::disabled()), } } @@ -4741,6 +5065,139 @@ mod tests { assert!(selected.fields.contains_key("pool")); } + /// The CLI builds `--from ` config as `{"vm": {...}}`. Guard the + /// CLI-to-driver transport: the rootfs tar field and any pre-existing VM + /// setting must both survive driver selection. A top-level field would be + /// dropped silently here and never reach the VM driver. + #[test] + fn select_driver_config_forwards_cli_rootfs_tar_template_to_vm_driver() { + let config = prost_types::Struct { + fields: std::iter::once(( + "vm".to_string(), + struct_value([ + ("rootfs_tar_path", string_value("/staging/req-a/rootfs.tar")), + ("gpu_device_ids", string_value("0000:2d:00.0")), + ]), + )) + .collect(), + }; + + let selected = select_driver_config(&Some(config), "vm").unwrap(); + let selected = selected.expect("vm config should be selected"); + + assert!(selected.fields.contains_key("rootfs_tar_path")); + assert!(selected.fields.contains_key("gpu_device_ids")); + } + + #[test] + fn select_driver_config_drops_top_level_rootfs_tar_path() { + let config = prost_types::Struct { + fields: std::iter::once(( + "rootfs_tar_path".to_string(), + string_value("/staging/req-a/rootfs.tar"), + )) + .collect(), + }; + + assert!( + select_driver_config(&Some(config), "vm").unwrap().is_none(), + "a top-level rootfs_tar_path never reaches the vm driver" + ); + } + + /// The staging token is a bearer credential for the staged archive, and the + /// persisted public sandbox is readable by every member of the workspace. + /// It must be stripped before anything writes that copy. + #[test] + fn take_public_staging_token_strips_it_from_the_public_sandbox() { + let mut sandbox = Sandbox { + spec: Some(SandboxSpec { + template: Some(SandboxTemplate { + driver_config: Some(prost_types::Struct { + fields: std::iter::once(( + "vm".to_string(), + struct_value([ + ("rootfs_tar_staging_token", string_value("tok-abc")), + ("gpu_device_ids", string_value("0000:2d:00.0")), + ]), + )) + .collect(), + }), + ..Default::default() + }), + ..Default::default() + }), + ..Default::default() + }; + + let token = take_public_staging_token(&mut sandbox, "vm"); + + assert_eq!(token.as_deref(), Some("tok-abc")); + let config = sandbox + .spec + .as_ref() + .and_then(|s| s.template.as_ref()) + .and_then(|t| t.driver_config.as_ref()) + .expect("driver config"); + let Some(prost_types::value::Kind::StructValue(vm)) = config.fields["vm"].kind.as_ref() + else { + panic!("vm block must survive"); + }; + assert!(!vm.fields.contains_key("rootfs_tar_staging_token")); + assert!( + vm.fields.contains_key("gpu_device_ids"), + "other vm settings must be left intact" + ); + } + + #[test] + fn take_public_staging_token_ignores_other_drivers() { + let mut sandbox = Sandbox { + spec: Some(SandboxSpec { + template: Some(SandboxTemplate { + driver_config: Some(prost_types::Struct { + fields: std::iter::once(( + "docker".to_string(), + struct_value([("rootfs_tar_staging_token", string_value("tok-abc"))]), + )) + .collect(), + }), + ..Default::default() + }), + ..Default::default() + }), + ..Default::default() + }; + + assert!(take_public_staging_token(&mut sandbox, "vm").is_none()); + } + + #[test] + fn set_rootfs_tar_path_writes_into_the_driver_copy() { + let mut driver_sandbox = DriverSandbox { + spec: Some(DriverSandboxSpec { + template: Some(DriverSandboxTemplate::default()), + ..Default::default() + }), + ..Default::default() + }; + + set_rootfs_tar_path(&mut driver_sandbox, Path::new("/staging/req-a/r.tar")); + + let config = driver_sandbox + .spec + .as_ref() + .and_then(|s| s.template.as_ref()) + .and_then(|t| t.driver_config.as_ref()) + .expect("driver config"); + let Some(prost_types::value::Kind::StringValue(path)) = + config.fields["rootfs_tar_path"].kind.as_ref() + else { + panic!("rootfs_tar_path must be a string"); + }; + assert_eq!(path, "/staging/req-a/r.tar"); + } + #[test] fn select_driver_config_rejects_non_object_matching_driver_block() { let config = prost_types::Struct { @@ -4787,6 +5244,10 @@ mod tests { default_image: "openshell/sandbox:test".to_string(), gateway_manages_lifecycle: false, supports_sandbox_authentication: false, + driver_reports_runtime_readiness: false, + resource_capabilities: None, + rootfs_tar_staging_dir: String::new(), + rootfs_tar_max_bytes: 0, })) } @@ -4948,6 +5409,7 @@ mod tests { delete_release: Semaphore, delete_blocked: AtomicBool, delete_calls: AtomicUsize, + delete_requests: TestMutex>, delete_outcome: TestMutex, stop_started: Notify, stop_finished: Notify, @@ -4980,6 +5442,7 @@ mod tests { delete_release: Semaphore::new(0), delete_blocked: AtomicBool::new(false), delete_calls: AtomicUsize::new(0), + delete_requests: TestMutex::new(Vec::new()), delete_outcome: TestMutex::new(ControlledDeleteOutcome::Ok(true)), stop_started: Notify::new(), stop_finished: Notify::new(), @@ -5063,6 +5526,13 @@ mod tests { self.delete_calls.load(Ordering::SeqCst) } + fn delete_requests(&self) -> Vec<(String, String)> { + self.delete_requests + .lock() + .expect("delete requests lock poisoned") + .clone() + } + fn stop_calls(&self) -> usize { self.stop_calls.load(Ordering::SeqCst) } @@ -5118,6 +5588,10 @@ mod tests { default_image: "openshell/sandbox:test".to_string(), gateway_manages_lifecycle: false, supports_sandbox_authentication: false, + driver_reports_runtime_readiness: false, + resource_capabilities: None, + rootfs_tar_staging_dir: String::new(), + rootfs_tar_max_bytes: 0, })) } @@ -5250,8 +5724,13 @@ mod tests { async fn delete_sandbox( &self, - _request: Request, + request: Request, ) -> Result, Status> { + let request = request.into_inner(); + self.delete_requests + .lock() + .expect("delete requests lock poisoned") + .push((request.sandbox_id, request.sandbox_name)); self.delete_calls.fetch_add(1, Ordering::SeqCst); self.delete_started.notify_one(); if self.delete_blocked.load(Ordering::SeqCst) { @@ -5322,7 +5801,12 @@ mod tests { driver_version: "test".to_string(), gateway_manages_lifecycle: false, supports_sandbox_authentication: false, + driver_reports_runtime_readiness: false, + resource_capabilities: None, + rootfs_tar_staging_dir: String::new(), + rootfs_tar_max_bytes: 0, }, + telemetry_compute_driver: TelemetryComputeDriver::custom(), driver_process: None, default_image: "openshell/sandbox:test".to_string(), store, @@ -5334,8 +5818,7 @@ mod tests { lifecycle_gates: Arc::new(LifecycleGateRegistry::default()), gateway_listener_requirements: Vec::new(), replica_id: "test-replica".to_string(), - #[cfg(target_os = "windows")] - mxc_policy_sink: None, + rootfs_tar_staging: Arc::new(rootfs_tar::RootfsTarStagingRegistry::disabled()), } } @@ -5415,6 +5898,42 @@ mod tests { })); } + #[test] + fn main_process_exit_replaces_provisional_container_exit() { + let mut sandbox = error_sandbox_record("sb-1", "sandbox-a", "ContainerExited"); + apply_main_process_exit(&mut sandbox, "instance-1", 0); + + assert_eq!( + SandboxPhase::try_from(sandbox.phase()), + Ok(SandboxPhase::Completed) + ); + let status = sandbox.status.as_ref().unwrap(); + assert_eq!(status.exit_code, Some(0)); + assert_eq!(status.main_process_instance_id, "instance-1"); + assert!(status.conditions.iter().any(|condition| { + condition.r#type == "Ready" + && condition.status == "False" + && condition.reason == "MainProcessCompleted" + })); + } + + #[test] + fn main_process_exit_preserves_specific_infrastructure_error() { + let mut sandbox = error_sandbox_record("sb-1", "sandbox-a", "BackendResourceMissing"); + apply_main_process_exit(&mut sandbox, "instance-1", 0); + + assert_eq!( + SandboxPhase::try_from(sandbox.phase()), + Ok(SandboxPhase::Error) + ); + let status = sandbox.status.as_ref().unwrap(); + assert_eq!(status.exit_code, Some(0)); + assert_eq!(status.main_process_instance_id, "instance-1"); + assert!(status.conditions.iter().any(|condition| { + condition.r#type == "Ready" && condition.reason == "BackendResourceMissing" + })); + } + #[tokio::test] async fn stale_main_process_exit_is_acknowledged_without_replacing_active_instance() { let runtime = test_runtime(Arc::new(TestDriver::default())).await; @@ -5600,6 +6119,10 @@ mod tests { .main_process_exited(id, "instance-1", 143) .await .unwrap(); + runtime + .finalize_main_process_exit(id, "instance-1") + .await + .expect("intentional shutdown finalization should be acknowledged"); let stored = runtime .store @@ -5914,7 +6437,7 @@ mod tests { ) -> (watch::Sender, tokio::task::JoinHandle<()>) { let (shutdown_tx, shutdown_rx) = watch::channel(false); let runtime = Arc::new(runtime.clone()); - let handle = tokio::spawn(async move { runtime.watch_loop(shutdown_rx).await }); + let handle = tokio::spawn(async move { Box::pin(runtime.watch_loop(shutdown_rx)).await }); tokio::time::timeout(Duration::from_secs(1), driver.watch_started.notified()) .await .expect("watch loop did not start"); @@ -5938,6 +6461,22 @@ mod tests { assert!(store.is_single_replica()); } + #[test] + fn driver_reported_runtime_uses_driver_readiness() { + let status = make_driver_status(DriverCondition { + r#type: "Ready".to_string(), + status: "True".to_string(), + reason: "AgentRunning".to_string(), + message: "MXC workload is running".to_string(), + last_transition_time: String::new(), + }); + + let composed = ComposedPhase::new(&status, false, true); + + assert_eq!(composed.phase, SandboxPhase::Ready); + assert!(!composed.backend_ready_without_session); + } + #[test] fn delete_gate_registry_removes_stale_entries() { let registry = LifecycleGateRegistry::default(); @@ -6253,8 +6792,7 @@ mod tests { )); } - /// Driver calls are a remote boundary even in-process: they reach the - /// Docker daemon, the Kubernetes API, or a Podman socket. + /// Driver calls are a remote boundary even when the driver is in-process. #[tokio::test] async fn driver_calls_export_spans_with_parents() { use tracing::Instrument as _; @@ -6274,7 +6812,11 @@ mod tests { .instrument(tracing::info_span!("request")) .await; - let driver_span = traced.span_with("driver.create_sandbox", "sandbox.id", "sb-trace"); + let driver_span = traced.span_with( + "openshell.compute.v1.ComputeDriver/CreateSandbox", + "sandbox.id", + "sb-trace", + ); test_exporter::assert_has_parent(&driver_span); assert_eq!( test_exporter::attribute(&driver_span, "driver.name").as_deref(), @@ -6285,6 +6827,18 @@ mod tests { test_exporter::attribute(&driver_span, "sandbox.id").as_deref(), Some("sb-trace"), ); + assert_eq!( + test_exporter::attribute(&driver_span, "rpc.method").as_deref(), + Some("openshell.compute.v1.ComputeDriver/CreateSandbox"), + ); + assert!( + test_exporter::attribute(&driver_span, "rpc.service").is_none(), + "the current RPC semantic conventions integrate the service into rpc.method" + ); + assert_eq!( + test_exporter::attribute(&driver_span, "rpc.response.status_code").as_deref(), + Some("OK"), + ); assert_eq!( driver_span.span_kind, opentelemetry::trace::SpanKind::Client, @@ -6300,6 +6854,39 @@ mod tests { ); } + #[tokio::test] + async fn driver_watch_client_span_lives_until_the_stream_completes() { + use futures::StreamExt as _; + + use crate::otel_tracing::test_exporter; + + let traced = test_exporter::install_traced(); + let driver = TracedDriver::new(Arc::new(TestDriver::default()), "test-driver".to_string()); + let response = driver.watch().await.expect("watch opens"); + assert!( + traced + .finished_spans() + .iter() + .all(|span| span.name != "openshell.compute.v1.ComputeDriver/WatchSandboxes"), + "the client span must remain open while the response stream is alive" + ); + + let mut stream = response.into_inner(); + assert!(stream.next().await.is_none()); + drop(stream); + + let spans = traced.finished_spans(); + let span = spans + .iter() + .find(|span| span.name == "openshell.compute.v1.ComputeDriver/WatchSandboxes") + .expect("watch client span should finish with the stream"); + assert_eq!(span.span_kind, opentelemetry::trace::SpanKind::Client); + assert_eq!( + test_exporter::attribute(span, "rpc.response.status_code").as_deref(), + Some("OK"), + ); + } + /// A failing driver call must be visible as a failure in the trace, not /// just as a span that happens to be followed by nothing. #[tokio::test] @@ -6431,7 +7018,11 @@ mod tests { .instrument(tracing::info_span!("request")) .await; - let driver_span = traced.span_with("driver.create_sandbox", "sandbox.id", "sb-fail"); + let driver_span = traced.span_with( + "openshell.compute.v1.ComputeDriver/CreateSandbox", + "sandbox.id", + "sb-fail", + ); assert!( matches!( @@ -6442,8 +7033,8 @@ mod tests { driver_span.status ); assert_eq!( - test_exporter::attribute(&driver_span, "grpc.code").as_deref(), - Some("14"), + test_exporter::attribute(&driver_span, "rpc.response.status_code").as_deref(), + Some("UNAVAILABLE"), "the gRPC code names the cause without reading the message" ); } @@ -6755,6 +7346,103 @@ mod tests { assert_eq!(driver.stop_calls(), 1); } + #[tokio::test] + async fn explicit_stop_completes_after_term_runtime_restart() { + let driver = ControlledDriver::new(); + driver.block_stop(); + let runtime = test_runtime(driver.clone()).await; + let sandbox = sandbox_record("sb-term-stop", "sandbox-term-stop", SandboxPhase::Ready); + runtime.store.put_message(&sandbox).await.unwrap(); + + let stop_runtime = runtime.clone(); + let stop = tokio::spawn(async move { + stop_runtime + .stop_sandbox("default", "sandbox-term-stop") + .await + }); + tokio::time::timeout(Duration::from_secs(1), driver.stop_started.notified()) + .await + .expect("stop did not reach the driver"); + + let mut runtime_restart = ready_driver_sandbox(sandbox.object_id(), sandbox.object_name()); + runtime_restart.status = Some(make_driver_status(make_driver_condition( + "ContainerRuntimeRestart", + "container exited with status 143", + ))); + runtime.apply_sandbox_update(runtime_restart).await.unwrap(); + + let stopping = runtime + .store + .get_message::(sandbox.object_id()) + .await + .unwrap() + .unwrap(); + assert_eq!(stopping.phase(), SandboxPhase::Stopping as i32); + assert_eq!( + stopping.status.unwrap().conditions[0].reason, + "ContainerRuntimeRestart" + ); + + driver.release_stop(); + let stopped = tokio::time::timeout(Duration::from_secs(1), stop) + .await + .expect("stop did not finish") + .unwrap() + .unwrap(); + assert_eq!(stopped.phase(), SandboxPhase::Stopped as i32); + assert_eq!(stopped.status.unwrap().conditions[0].reason, "Stopped"); + } + + #[tokio::test] + async fn failed_stop_does_not_report_term_runtime_restart_as_stopped() { + let driver = ControlledDriver::new(); + driver.block_stop(); + driver.set_stop_outcome(ControlledLifecycleOutcome::Error("stop timed out")); + let runtime = test_runtime(driver.clone()).await; + let sandbox = sandbox_record("sb-term-fail", "sandbox-term-fail", SandboxPhase::Ready); + runtime.store.put_message(&sandbox).await.unwrap(); + + let stop_runtime = runtime.clone(); + let stop = tokio::spawn(async move { + stop_runtime + .stop_sandbox("default", "sandbox-term-fail") + .await + }); + tokio::time::timeout(Duration::from_secs(1), driver.stop_started.notified()) + .await + .expect("stop did not reach the driver"); + + let mut runtime_restart = ready_driver_sandbox(sandbox.object_id(), sandbox.object_name()); + runtime_restart.status = Some(make_driver_status(make_driver_condition( + "ContainerRuntimeRestart", + "container exited with status 143", + ))); + driver.set_get_outcome(ControlledGetOutcome::Sandbox(Box::new( + runtime_restart.clone(), + ))); + runtime.apply_sandbox_update(runtime_restart).await.unwrap(); + + driver.release_stop(); + let err = tokio::time::timeout(Duration::from_secs(1), stop) + .await + .expect("stop did not finish") + .unwrap() + .unwrap_err(); + assert!(err.message().contains("stop timed out")); + + let stored = runtime + .store + .get_message::(sandbox.object_id()) + .await + .unwrap() + .unwrap(); + assert_eq!(stored.phase(), SandboxPhase::Stopping as i32); + assert_eq!( + stored.status.unwrap().conditions[0].reason, + "ContainerRuntimeRestart" + ); + } + #[tokio::test] async fn request_cancellation_does_not_cancel_start_worker() { let driver = ControlledDriver::new(); @@ -6930,6 +7618,105 @@ mod tests { assert_eq!(stored.phase(), SandboxPhase::Starting as i32); } + #[tokio::test] + async fn stale_container_exit_queued_before_start_cannot_regress_restart() { + for reason in [ + "ContainerExited", + "ContainerStopped", + "ContainerRuntimeRestart", + ] { + let driver = ControlledDriver::new(); + driver.block_start(); + let sandbox = + sandbox_record("sb-start-race", "sandbox-start-race", SandboxPhase::Stopped); + driver.set_get_outcome(ControlledGetOutcome::Sandbox(Box::new( + ready_driver_sandbox(sandbox.object_id(), sandbox.object_name()), + ))); + let runtime = test_runtime(driver.clone()).await; + runtime.store.put_message(&sandbox).await.unwrap(); + + let start_runtime = runtime.clone(); + let sandbox_name = sandbox.object_name().to_string(); + let start = + tokio::spawn( + async move { start_runtime.start_sandbox("default", &sandbox_name).await }, + ); + tokio::time::timeout(Duration::from_secs(1), driver.start_started.notified()) + .await + .expect("start did not reach the driver"); + + let mut stale_exit = ready_driver_sandbox(sandbox.object_id(), sandbox.object_name()); + stale_exit.status = Some(make_driver_status(make_driver_condition( + reason, + "container stopped before restart", + ))); + let update_runtime = runtime.clone(); + let mut update = + tokio::spawn(async move { update_runtime.apply_sandbox_update(stale_exit).await }); + + assert!( + tokio::time::timeout(Duration::from_millis(50), &mut update) + .await + .is_err(), + "{reason} watch update must wait for the active start operation" + ); + + driver.release_start(); + start.await.unwrap().unwrap(); + update.await.unwrap().unwrap(); + + let stored = runtime + .store + .get_message::(sandbox.object_id()) + .await + .unwrap() + .unwrap(); + assert_eq!( + stored.phase(), + SandboxPhase::Starting as i32, + "{reason} must not regress the restarted sandbox" + ); + } + } + + #[tokio::test] + async fn live_container_exit_during_start_still_transitions_to_error() { + for reason in [ + "ContainerExited", + "ContainerStopped", + "ContainerRuntimeRestart", + ] { + let driver = ControlledDriver::new(); + let sandbox = sandbox_record( + "sb-start-exited", + "sandbox-start-exited", + SandboxPhase::Starting, + ); + let mut exited = ready_driver_sandbox(sandbox.object_id(), sandbox.object_name()); + exited.status = Some(make_driver_status(make_driver_condition( + reason, + "restarted container exited", + ))); + driver.set_get_outcome(ControlledGetOutcome::Sandbox(Box::new(exited.clone()))); + let runtime = test_runtime(driver).await; + runtime.store.put_message(&sandbox).await.unwrap(); + + runtime.apply_sandbox_update(exited).await.unwrap(); + + let stored = runtime + .store + .get_message::(sandbox.object_id()) + .await + .unwrap() + .unwrap(); + assert_eq!( + stored.phase(), + SandboxPhase::Error as i32, + "current {reason} snapshot must remain terminal" + ); + } + } + #[tokio::test] async fn lifecycle_operations_reject_invalid_source_phases() { let runtime = test_runtime(Arc::new(TestDriver::default())).await; @@ -7666,6 +8453,43 @@ mod tests { assert_eq!(driver.delete_calls(), 1); } + #[tokio::test] + async fn sandbox_ssh_session_cleanup_batches_across_list_pages() { + let runtime = test_runtime(ControlledDriver::new()).await; + for idx in 0..(LIFECYCLE_SWEEP_PAGE_SIZE + 5) { + let session = ssh_session_record(&format!("owned-{idx:04}"), "sb-owned"); + runtime.store.put_message(&session).await.unwrap(); + } + for idx in 0..7 { + let session = ssh_session_record(&format!("unrelated-{idx:04}"), "sb-unrelated"); + runtime.store.put_message(&session).await.unwrap(); + } + + runtime + .cleanup_sandbox_ssh_sessions("sb-owned", "default") + .await + .unwrap(); + + assert_eq!( + runtime + .store + .count_in_workspace(SshSession::object_type(), "default") + .await + .unwrap(), + 7 + ); + for idx in 0..7 { + assert!( + runtime + .store + .get_message::(&format!("unrelated-{idx:04}")) + .await + .unwrap() + .is_some() + ); + } + } + #[tokio::test] async fn already_absent_driver_resource_is_removed_synchronously() { let driver = ControlledDriver::new(); @@ -7837,6 +8661,148 @@ mod tests { )); } + #[tokio::test] + async fn apply_deleted_releases_driver_resources_for_out_of_band_removal() { + // Regression test for #2352: a container removed without going + // through OpenShell's own DeleteSandbox (e.g. `podman rm -f`) must + // still release driver-owned secrets/volumes, not just the store + // record. This never calls `runtime.delete_sandbox`, matching the + // out-of-band removal the issue reports. + let driver = ControlledDriver::new(); + let runtime = test_runtime(driver.clone()).await; + let sandbox = sandbox_record("sb-1", "sandbox-a", SandboxPhase::Ready); + runtime.store.put_message(&sandbox).await.unwrap(); + let session = seed_sandbox_owned_records(&runtime, &sandbox).await; + + assert_eq!(driver.delete_calls(), 0); + + runtime.apply_deleted("sb-1").await.unwrap(); + + // The driver call is backgrounded (see `spawn_driver_sandbox_cleanup`) + // so it can't block the watch loop; wait for it to actually land + // before asserting on it. + tokio::time::timeout(Duration::from_secs(1), driver.delete_started.notified()) + .await + .expect("background driver cleanup did not run"); + assert_eq!( + driver.delete_requests(), + vec![("sb-1".to_string(), "sandbox-a".to_string())] + ); + assert_sandbox_owned_records(&runtime, &sandbox, &session, false).await; + assert!( + runtime + .store + .get_message::("sb-1") + .await + .unwrap() + .is_none() + ); + } + + #[tokio::test] + async fn apply_deleted_removes_store_record_even_when_driver_cleanup_fails() { + // The driver has already told us (via the watch/prune path that led + // here) that this sandbox is gone. A failure releasing its + // secrets/volumes must be logged, not block store cleanup — retrying + // forever on a driver hiccup would leave the store permanently out + // of sync with reality. + let driver = ControlledDriver::new(); + driver.set_delete_outcome(ControlledDeleteOutcome::Error("podman unreachable")); + let runtime = test_runtime(driver.clone()).await; + let sandbox = sandbox_record("sb-1", "sandbox-a", SandboxPhase::Ready); + runtime.store.put_message(&sandbox).await.unwrap(); + + runtime.apply_deleted("sb-1").await.unwrap(); + + // Store cleanup happens synchronously in `apply_deleted_locked`, so + // this is already true even though the driver call is backgrounded. + assert!( + runtime + .store + .get_message::("sb-1") + .await + .unwrap() + .is_none() + ); + + tokio::time::timeout(Duration::from_secs(1), driver.delete_started.notified()) + .await + .expect("background driver cleanup did not run"); + assert_eq!(driver.delete_calls(), 1); + } + + #[tokio::test] + async fn prune_missing_sandbox_releases_driver_resources() { + // Regression test for #2352's second reproduction: a sandbox whose + // container never survived to exist at all (e.g. the gateway was + // killed mid-create) is discovered missing by the periodic sweep, + // not the watcher. That path must also release driver resources. + let driver = ControlledDriver::new(); + driver.set_get_outcome(ControlledGetOutcome::Missing); + let runtime = test_runtime(driver.clone()).await; + let sandbox = sandbox_record("sb-1", "sandbox-a", SandboxPhase::Provisioning); + runtime.store.put_message(&sandbox).await.unwrap(); + + runtime + .reconcile_store_with_backend(Duration::ZERO) + .await + .unwrap(); + + assert!( + runtime + .store + .get_message::("sb-1") + .await + .unwrap() + .is_none() + ); + + // The driver call is backgrounded (see `spawn_driver_sandbox_cleanup`) + // so the prune sweep never awaits it while holding the gateway-wide + // sync_lock; wait for it to actually land before asserting on it. + tokio::time::timeout(Duration::from_secs(1), driver.delete_started.notified()) + .await + .expect("background driver cleanup did not run"); + assert_eq!( + driver.delete_requests(), + vec![("sb-1".to_string(), "sandbox-a".to_string())] + ); + } + + #[tokio::test] + async fn prune_sweep_does_not_block_on_a_stuck_driver_delete_call() { + // Regression test: the prune sweep's driver cleanup must not be + // awaited while holding `sync_lock` (the gateway-wide state guard). + // Block the driver's delete call indefinitely and confirm the sweep + // itself still completes promptly and removes the store record. + let driver = ControlledDriver::new(); + driver.block_delete(); + driver.set_get_outcome(ControlledGetOutcome::Missing); + let runtime = test_runtime(driver.clone()).await; + let sandbox = sandbox_record("sb-1", "sandbox-a", SandboxPhase::Provisioning); + runtime.store.put_message(&sandbox).await.unwrap(); + + tokio::time::timeout( + Duration::from_secs(1), + runtime.reconcile_store_with_backend(Duration::ZERO), + ) + .await + .expect("prune sweep blocked on the stuck driver delete call") + .unwrap(); + + assert!( + runtime + .store + .get_message::("sb-1") + .await + .unwrap() + .is_none() + ); + tokio::time::timeout(Duration::from_secs(1), driver.delete_started.notified()) + .await + .expect("background driver cleanup did not run"); + } + #[tokio::test] async fn accepted_delete_cleans_local_state_after_another_replica_removes_row() { let driver = ControlledDriver::new(); @@ -8345,6 +9311,73 @@ mod tests { assert_eq!(status.exit_code, None); } + #[tokio::test] + async fn unexpected_term_runtime_restart_transitions_to_error() { + let runtime = test_runtime(Arc::new(TestDriver::default())).await; + let sandbox = sandbox_record("sb-term-exit", "sandbox-term-exit", SandboxPhase::Ready); + runtime.store.put_message(&sandbox).await.unwrap(); + let mut runtime_restart = ready_driver_sandbox(sandbox.object_id(), sandbox.object_name()); + runtime_restart.status = Some(make_driver_status(make_driver_condition( + "ContainerRuntimeRestart", + "container exited with status 143", + ))); + + runtime.apply_sandbox_update(runtime_restart).await.unwrap(); + + let stored = runtime + .store + .get_message::(sandbox.object_id()) + .await + .unwrap() + .unwrap(); + assert_eq!(stored.phase(), SandboxPhase::Error as i32); + assert_eq!( + stored.status.unwrap().conditions[0].reason, + "ContainerRuntimeRestart" + ); + } + + #[tokio::test] + async fn late_term_runtime_restart_preserves_intentional_stop_status() { + let runtime = test_runtime(Arc::new(TestDriver::default())).await; + let mut sandbox = sandbox_record( + "sb-term-stopped", + "sandbox-term-stopped", + SandboxPhase::Stopped, + ); + sandbox.status = Some(SandboxStatus { + sandbox_name: sandbox.object_name().to_string(), + phase: SandboxPhase::Stopped as i32, + conditions: vec![SandboxCondition { + r#type: "Ready".to_string(), + status: "False".to_string(), + reason: "Stopped".to_string(), + message: "Sandbox compute is stopped".to_string(), + last_transition_time: String::new(), + }], + ..Default::default() + }); + runtime.store.put_message(&sandbox).await.unwrap(); + let mut runtime_restart = ready_driver_sandbox(sandbox.object_id(), sandbox.object_name()); + runtime_restart.status = Some(make_driver_status(make_driver_condition( + "ContainerRuntimeRestart", + "container exited with status 143", + ))); + + runtime.apply_sandbox_update(runtime_restart).await.unwrap(); + + let stored = runtime + .store + .get_message::(sandbox.object_id()) + .await + .unwrap() + .unwrap(); + assert_eq!(stored.phase(), SandboxPhase::Stopped as i32); + let ready = &stored.status.unwrap().conditions[0]; + assert_eq!(ready.reason, "Stopped"); + assert_eq!(ready.message, "Sandbox compute is stopped"); + } + #[tokio::test] async fn late_driver_exit_preserves_completed_main_result() { let runtime = test_runtime(Arc::new(TestDriver::default())).await; @@ -8669,7 +9702,7 @@ mod tests { #[tokio::test] async fn backend_not_ready_with_supervisor_becomes_ready() { - // VM path: supervisor connects before backend reports Ready. + // The supervisor may connect before the backend reports Ready. let runtime = test_runtime(Arc::new(TestDriver::default())).await; let sandbox = sandbox_record("sb-1", "sandbox-a", SandboxPhase::Provisioning); runtime.store.put_message(&sandbox).await.unwrap(); @@ -9001,7 +10034,7 @@ mod tests { .iter() .find(|root| { spans.iter().any(|span| { - span.name == "driver.list_sandboxes" + span.name == "openshell.compute.v1.ComputeDriver/ListSandboxes" && span.span_context.trace_id() == root.span_context.trace_id() }) && spans.iter().any(|span| { span.name.starts_with("store.") @@ -9014,7 +10047,7 @@ mod tests { let driver_span = spans .iter() .find(|span| { - span.name == "driver.list_sandboxes" + span.name == "openshell.compute.v1.ComputeDriver/ListSandboxes" && span.span_context.trace_id() == root.span_context.trace_id() }) .expect("the sweep records its driver call"); @@ -9831,45 +10864,16 @@ mod tests { } #[test] - fn build_platform_config_inverts_user_namespaces_to_host_users() { - use prost_types::value::Kind; - - // user_namespaces: true → host_users: false - let mut template = SandboxTemplate { + fn driver_template_preserves_user_namespace_intent() { + let template = SandboxTemplate { user_namespaces: Some(true), ..SandboxTemplate::default() }; - let config = build_platform_config(&template).expect("config should be Some"); - let host_users = config - .fields - .get("host_users") - .expect("host_users must exist"); - assert_eq!( - host_users.kind, - Some(Kind::BoolValue(false)), - "user_namespaces: true must produce host_users: false" - ); + let driver_template = driver_sandbox_template_from_public(&template, "test") + .expect("template conversion should succeed"); - // user_namespaces: false → host_users: true - template.user_namespaces = Some(false); - let config = build_platform_config(&template).expect("config should be Some"); - let host_users = config - .fields - .get("host_users") - .expect("host_users must exist"); - assert_eq!( - host_users.kind, - Some(Kind::BoolValue(true)), - "user_namespaces: false must produce host_users: true" - ); - - // user_namespaces: None → host_users absent - template.user_namespaces = None; - let config = build_platform_config(&template); - assert!( - config.is_none() || !config.as_ref().unwrap().fields.contains_key("host_users"), - "unset user_namespaces must not produce host_users" - ); + assert_eq!(driver_template.user_namespaces, Some(true)); + assert!(driver_template.platform_config.is_none()); } #[tokio::test] @@ -10075,8 +11079,12 @@ mod tests { let mut sandbox = sandbox_record("sb-uds", "uds-sandbox", SandboxPhase::Provisioning); sandbox.spec = Some(SandboxSpec { log_level: "debug".to_string(), + policy: Some(openshell_core::proto::SandboxPolicy { + version: 42, + ..Default::default() + }), template: Some(SandboxTemplate { - image: "ghcr.io/nvidia/openshell/sandbox:test".to_string(), + image: "ghcr.io/nvidia/openshell-community/sandboxes/base:latest".to_string(), driver_config: Some(prost_types::Struct { fields: [ ( @@ -10114,6 +11122,20 @@ mod tests { .expect("selected driver_config should be forwarded"); assert!(driver_config.fields.contains_key("pool")); assert!(!driver_config.fields.contains_key("network_mode")); + assert_eq!( + validated + .spec + .as_ref() + .and_then(|spec| spec.policy.as_ref()) + .map(|policy| policy.version), + Some(42) + ); + assert!(matches!( + &calls[3], + FakeComputeDriverCall::CreateSandbox { sandbox: Some(sandbox) } + if sandbox.spec.as_ref().and_then(|spec| spec.policy.as_ref()) + .is_some_and(|policy| policy.version == 42) + )); driver.clear_calls(); runtime diff --git a/crates/openshell-server/src/compute/rootfs_tar.rs b/crates/openshell-server/src/compute/rootfs_tar.rs new file mode 100644 index 000000000..3df1f831f --- /dev/null +++ b/crates/openshell-server/src/compute/rootfs_tar.rs @@ -0,0 +1,693 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! Gateway-owned staging slots for rootfs tar archives. +//! +//! A sandbox created from a flat rootfs tar needs the archive on the gateway +//! host before the compute driver can turn it into a disk. Callers never name +//! that location. The gateway allocates a request-scoped directory inside the +//! driver-advertised staging root and hands back an opaque single-use token; +//! `CreateSandbox` carries the token, and the gateway substitutes the resolved +//! path into the driver-native request. A caller-supplied `rootfs_tar_path` is +//! rejected outright during request validation, so a raw host path can never +//! reach the privileged driver. + +use std::collections::HashMap; +use std::path::{Component, Path, PathBuf}; +use std::sync::Mutex; +use std::time::{Duration, Instant}; + +use openshell_core::time::now_ms; +use rand::RngCore; +use tonic::Status; +use tracing::{info, warn}; + +/// How long an allocated slot survives without being consumed. +const STAGING_TOKEN_TTL: Duration = Duration::from_mins(30); +/// Outstanding slots one caller may hold. Bounds the directories a single +/// authenticated caller can create by calling `begin` in a loop. +const MAX_SLOTS_PER_CALLER: usize = 4; +/// Outstanding slots across all callers. Per-caller alone would let enough +/// distinct callers exhaust the staging filesystem; a global cap alone would +/// let one caller starve everyone else. +const MAX_TOTAL_SLOTS: usize = 64; +const STAGING_DIR_PREFIX: &str = "req-"; +const MAX_STAGED_FILE_NAME_LEN: usize = 128; + +/// `driver_config.` key the CLI sets to redeem a staging slot. +pub const STAGING_TOKEN_FIELD: &str = "rootfs_tar_staging_token"; +/// `driver_config.` key the gateway substitutes for the driver. Callers +/// may never set this themselves. +pub const ROOTFS_TAR_PATH_FIELD: &str = "rootfs_tar_path"; + +/// Deliberately identical for unknown and expired tokens: distinguishing them +/// would let a caller probe which tokens exist. +fn unknown_token() -> Status { + Status::failed_precondition( + "rootfs tar staging token is unknown or expired; re-stage the archive", + ) +} + +/// A slot handed back to the client by `BeginRootfsTarStaging`. +#[derive(Debug, Clone)] +pub struct StagingSlot { + pub token: String, + pub upload_path: PathBuf, + pub max_bytes: u64, + pub expires_at_ms: i64, +} + +#[derive(Debug)] +struct StagingEntry { + dir: PathBuf, + file: PathBuf, + workspace: String, + subject: String, + expires_at: Instant, +} + +/// Ownership of a consumed staging directory. +/// +/// Removes the directory on drop so every failure path after the token is +/// consumed cleans up, unless [`StagedRootfsTar::disarm`] has transferred +/// ownership to the driver (which deletes it once the archive is extracted). +#[derive(Debug)] +pub struct StagedRootfsTar { + path: PathBuf, + dir: Option, +} + +impl StagedRootfsTar { + pub fn path(&self) -> &Path { + &self.path + } + + /// Hand the directory to the driver, which removes it after staging. + pub fn disarm(&mut self) { + self.dir = None; + } +} + +impl Drop for StagedRootfsTar { + fn drop(&mut self) { + let Some(dir) = self.dir.take() else { + return; + }; + // Unlinking is a syscall, but a multi-GiB file makes it worth keeping + // off a reactor thread. + match tokio::runtime::Handle::try_current() { + Ok(handle) => { + handle.spawn_blocking(move || remove_staging_dir(&dir)); + } + Err(_) => remove_staging_dir(&dir), + } + } +} + +fn remove_staging_dir(dir: &Path) { + if let Err(err) = std::fs::remove_dir_all(dir) + && err.kind() != std::io::ErrorKind::NotFound + { + warn!( + dir = %dir.display(), + error = %err, + "Failed to remove rootfs tar staging directory" + ); + } +} + +/// Request-scoped staging slots, keyed by opaque single-use token. +#[derive(Debug)] +pub struct RootfsTarStagingRegistry { + /// `None` when the active driver does not support rootfs tar sources. + staging_root: Option, + max_bytes: u64, + entries: Mutex>, + ttl: Duration, +} + +impl RootfsTarStagingRegistry { + pub fn new(staging_root: Option, max_bytes: u64) -> Self { + Self::with_ttl(staging_root, max_bytes, STAGING_TOKEN_TTL) + } + + fn with_ttl(staging_root: Option, max_bytes: u64, ttl: Duration) -> Self { + Self { + staging_root, + max_bytes, + entries: Mutex::new(HashMap::new()), + ttl, + } + } + + /// Registry for a driver that does not accept rootfs tar sources. + #[cfg(test)] + pub fn disabled() -> Self { + Self::new(None, 0) + } + + /// Allocate a request-scoped directory and return its single-use token. + pub fn begin( + &self, + workspace: &str, + subject: &str, + file_name: &str, + size_bytes: u64, + ) -> Result { + let Some(staging_root) = self.staging_root.as_ref() else { + return Err(Status::failed_precondition( + "the active compute driver does not support rootfs tar sources", + )); + }; + + if self.max_bytes > 0 && size_bytes > self.max_bytes { + return Err(Status::invalid_argument(format!( + "rootfs tar is {size_bytes} bytes, exceeding the driver limit of {} bytes", + self.max_bytes + ))); + } + + let file_name = sanitize_staged_file_name(file_name)?; + + let mut entries = self.entries.lock().expect("staging registry poisoned"); + Self::purge_expired(&mut entries); + if entries.len() >= MAX_TOTAL_SLOTS { + return Err(Status::resource_exhausted( + "the gateway has too many outstanding rootfs tar staging slots; retry shortly", + )); + } + let held_by_caller = entries + .values() + .filter(|entry| entry.workspace == workspace && entry.subject == subject) + .count(); + if held_by_caller >= MAX_SLOTS_PER_CALLER { + return Err(Status::resource_exhausted( + "too many outstanding rootfs tar staging slots; retry once an earlier create completes", + )); + } + + // The directory name uses independent randomness so the token never + // appears in a filesystem path, a directory listing, or a log field. + let dir = staging_root.join(format!("{STAGING_DIR_PREFIX}{}", uuid::Uuid::new_v4())); + create_private_dir(&dir).map_err(|err| { + Status::internal(format!( + "failed to create rootfs tar staging directory: {err}" + )) + })?; + + let file = dir.join(&file_name); + let token = new_staging_token(); + let expires_at = Instant::now() + self.ttl; + let expires_at_ms = now_ms() + i64::try_from(self.ttl.as_millis()).unwrap_or(i64::MAX); + + entries.insert( + token.clone(), + StagingEntry { + dir, + file: file.clone(), + workspace: workspace.to_string(), + subject: subject.to_string(), + expires_at, + }, + ); + + Ok(StagingSlot { + token, + upload_path: file, + max_bytes: self.max_bytes, + expires_at_ms, + }) + } + + /// Confirm the token belongs to this caller. Does not consume it. + /// + /// This is what stops one caller redeeming a slot minted for another. + pub fn authorize(&self, token: &str, workspace: &str, subject: &str) -> Result<(), Status> { + let mut entries = self.entries.lock().expect("staging registry poisoned"); + Self::purge_expired(&mut entries); + let entry = entries.get(token).ok_or_else(unknown_token)?; + if entry.workspace != workspace || entry.subject != subject { + return Err(Status::permission_denied( + "rootfs tar staging token was issued to a different caller", + )); + } + Ok(()) + } + + /// Resolve the staged path without consuming the token, for validation. + pub fn peek(&self, token: &str) -> Result { + let mut entries = self.entries.lock().expect("staging registry poisoned"); + Self::purge_expired(&mut entries); + let entry = entries.get(token).ok_or_else(unknown_token)?; + Ok(entry.file.clone()) + } + + /// Consume the token. A second redemption of the same token fails. + pub fn consume(&self, token: &str) -> Result { + let mut entries = self.entries.lock().expect("staging registry poisoned"); + Self::purge_expired(&mut entries); + let entry = entries.remove(token).ok_or_else(unknown_token)?; + drop(entries); + + if !entry.file.is_file() { + // Nothing was uploaded, or it was replaced by a directory or link. + remove_staging_dir(&entry.dir); + return Err(Status::failed_precondition(format!( + "no rootfs tar archive was uploaded to the staging slot at {}", + entry.file.display() + ))); + } + + Ok(StagedRootfsTar { + path: entry.file, + dir: Some(entry.dir), + }) + } + + fn purge_expired(entries: &mut HashMap) { + let now = Instant::now(); + let expired: Vec = entries + .iter() + .filter(|(_, entry)| entry.expires_at <= now) + .map(|(token, _)| token.clone()) + .collect(); + for token in expired { + if let Some(entry) = entries.remove(&token) { + remove_staging_dir(&entry.dir); + } + } + } + + /// Remove request directories nothing owns any more. + /// + /// Runs at startup and on each reconcile sweep. It catches two cases the + /// token table cannot: directories left by a previous gateway process, and + /// directories whose token was consumed but whose driver failed before it + /// reached its own cleanup. + /// + /// Age-gated rather than an unconditional wipe, because a driver can still + /// be copying a multi-gigabyte archive out of a directory whose gateway + /// already restarted. + pub fn sweep_orphans(&self) { + let Some(staging_root) = self.staging_root.as_ref() else { + return; + }; + let Ok(read_dir) = std::fs::read_dir(staging_root) else { + return; + }; + + let mut removed = 0usize; + for entry in read_dir.flatten() { + let name = entry.file_name(); + let Some(name) = name.to_str() else { continue }; + if !name.starts_with(STAGING_DIR_PREFIX) { + continue; + } + let stale = entry + .metadata() + .and_then(|meta| meta.modified()) + .is_ok_and(|modified| modified.elapsed().is_ok_and(|age| age > self.ttl)); + if stale { + remove_staging_dir(&entry.path()); + removed += 1; + } + } + + if removed > 0 { + info!( + removed, + staging_root = %staging_root.display(), + "Removed orphaned rootfs tar staging directories" + ); + } + } +} + +fn new_staging_token() -> String { + let mut raw = [0u8; 32]; + rand::rng().fill_bytes(&mut raw); + hex::encode(raw) +} + +/// Reject anything that would let `dir.join(file_name)` escape the request +/// directory. This is the check that makes the joined path safe to trust. +fn sanitize_staged_file_name(file_name: &str) -> Result { + let invalid = |reason: &str| Status::invalid_argument(format!("rootfs tar file_name {reason}")); + + if file_name.is_empty() { + return Err(invalid("must not be empty")); + } + if file_name.len() > MAX_STAGED_FILE_NAME_LEN { + return Err(invalid(&format!( + "must be at most {MAX_STAGED_FILE_NAME_LEN} bytes" + ))); + } + if file_name.contains('/') || file_name.contains('\\') || file_name.contains('\0') { + return Err(invalid("must not contain path separators")); + } + if file_name.starts_with('.') { + return Err(invalid("must not start with '.'")); + } + + let path = Path::new(file_name); + let mut components = path.components(); + let Some(Component::Normal(only)) = components.next() else { + return Err(invalid("must be a plain file name")); + }; + if components.next().is_some() { + return Err(invalid("must be a plain file name")); + } + if only != file_name { + return Err(invalid("must be a plain file name")); + } + + Ok(file_name.to_string()) +} + +fn create_private_dir(dir: &Path) -> std::io::Result<()> { + std::fs::create_dir(dir)?; + #[cfg(unix)] + { + use std::os::unix::fs::PermissionsExt; + std::fs::set_permissions(dir, std::fs::Permissions::from_mode(0o700))?; + } + Ok(()) +} + +#[cfg(test)] +mod tests { + use super::*; + use tonic::Code; + + fn registry(root: &Path) -> RootfsTarStagingRegistry { + RootfsTarStagingRegistry::new(Some(root.to_path_buf()), 1024) + } + + fn temp_root() -> tempfile::TempDir { + tempfile::tempdir().expect("create staging root") + } + + fn upload(slot: &StagingSlot, contents: &[u8]) { + std::fs::write(&slot.upload_path, contents).expect("write staged archive"); + } + + #[test] + fn begin_allocates_private_request_dir_and_hides_the_token() { + let root = temp_root(); + let registry = registry(root.path()); + + let slot = registry + .begin("default", "alice", "rootfs.tar", 128) + .expect("slot allocated"); + + let dir = slot.upload_path.parent().expect("upload dir"); + assert!(dir.starts_with(root.path())); + assert!( + dir.file_name() + .and_then(|n| n.to_str()) + .expect("dir name") + .starts_with(STAGING_DIR_PREFIX) + ); + assert_eq!(slot.upload_path.file_name().unwrap(), "rootfs.tar"); + assert!( + !slot.upload_path.to_string_lossy().contains(&slot.token), + "the token must not be recoverable from a directory listing" + ); + #[cfg(unix)] + { + use std::os::unix::fs::PermissionsExt; + let mode = std::fs::metadata(dir) + .expect("dir metadata") + .permissions() + .mode(); + assert_eq!(mode & 0o777, 0o700); + } + } + + #[test] + fn begin_rejects_oversized_archive_before_allocating() { + let root = temp_root(); + let registry = registry(root.path()); + + let err = registry + .begin("default", "alice", "rootfs.tar", 4096) + .expect_err("oversized archive rejected"); + + assert_eq!(err.code(), Code::InvalidArgument); + assert_eq!( + std::fs::read_dir(root.path()).unwrap().count(), + 0, + "nothing may be allocated for a rejected request" + ); + } + + /// `dir.join(file_name)` must not be able to escape the request directory. + #[test] + fn begin_rejects_traversal_file_names() { + let root = temp_root(); + let registry = registry(root.path()); + + let too_long = "x".repeat(MAX_STAGED_FILE_NAME_LEN + 1); + for name in [ + "../../etc/passwd", + "a/b.tar", + "..", + "", + "\\evil.tar", + ".hidden.tar", + too_long.as_str(), + ] { + let err = match registry.begin("default", "alice", name, 16) { + Ok(slot) => panic!("expected rejection for {name:?}, allocated {slot:?}"), + Err(err) => err, + }; + assert_eq!(err.code(), Code::InvalidArgument, "{name:?}: {err}"); + } + + assert_eq!( + std::fs::read_dir(root.path()).unwrap().count(), + 0, + "a rejected file name must not leave a directory behind" + ); + } + + #[test] + fn begin_rejects_when_driver_has_no_staging_dir() { + let registry = RootfsTarStagingRegistry::disabled(); + + let err = registry + .begin("default", "alice", "rootfs.tar", 16) + .expect_err("a driver without tar support rejects staging"); + + assert_eq!(err.code(), Code::FailedPrecondition); + } + + #[test] + fn begin_bounds_outstanding_slots_per_caller() { + let root = temp_root(); + let registry = registry(root.path()); + + for _ in 0..MAX_SLOTS_PER_CALLER { + registry + .begin("default", "alice", "rootfs.tar", 16) + .expect("slot allocated"); + } + + let err = registry + .begin("default", "alice", "rootfs.tar", 16) + .expect_err("one caller's outstanding slots are bounded"); + assert_eq!(err.code(), Code::ResourceExhausted); + } + + /// The per-caller cap must not become a way for one caller to lock others + /// out of the feature. + #[test] + fn one_caller_at_its_cap_does_not_block_another() { + let root = temp_root(); + let registry = registry(root.path()); + + for _ in 0..MAX_SLOTS_PER_CALLER { + registry + .begin("default", "alice", "rootfs.tar", 16) + .expect("slot allocated"); + } + + registry + .begin("default", "bob", "rootfs.tar", 16) + .expect("a different caller is unaffected"); + registry + .begin("other-workspace", "alice", "rootfs.tar", 16) + .expect("the same caller in another workspace is unaffected"); + } + + /// The replay regression test: a token redeemed twice must fail. + #[test] + fn consume_is_single_use() { + let root = temp_root(); + let registry = registry(root.path()); + let slot = registry + .begin("default", "alice", "rootfs.tar", 16) + .expect("slot allocated"); + upload(&slot, b"payload"); + + let mut staged = registry.consume(&slot.token).expect("first consume"); + staged.disarm(); + + let err = registry + .consume(&slot.token) + .expect_err("a staging token may only be redeemed once"); + assert_eq!(err.code(), Code::FailedPrecondition); + } + + /// Validation peeks and creation consumes; peeking must not burn the token. + #[test] + fn peek_does_not_consume() { + let root = temp_root(); + let registry = registry(root.path()); + let slot = registry + .begin("default", "alice", "rootfs.tar", 16) + .expect("slot allocated"); + upload(&slot, b"payload"); + + assert_eq!(registry.peek(&slot.token).unwrap(), slot.upload_path); + assert_eq!(registry.peek(&slot.token).unwrap(), slot.upload_path); + + let mut staged = registry.consume(&slot.token).expect("consume after peeks"); + staged.disarm(); + } + + #[test] + fn unknown_token_is_rejected_the_same_way_as_an_expired_one() { + let root = temp_root(); + let registry = registry(root.path()); + + let unknown = registry + .consume(&"0".repeat(64)) + .expect_err("unknown token"); + + let expiring = RootfsTarStagingRegistry::with_ttl( + Some(root.path().to_path_buf()), + 1024, + Duration::ZERO, + ); + let slot = expiring + .begin("default", "alice", "rootfs.tar", 16) + .expect("slot allocated"); + upload(&slot, b"payload"); + let expired = expiring.consume(&slot.token).expect_err("expired token"); + + assert_eq!(unknown.code(), expired.code()); + assert_eq!(unknown.message(), expired.message()); + } + + #[test] + fn expired_slot_directory_is_removed() { + let root = temp_root(); + let registry = RootfsTarStagingRegistry::with_ttl( + Some(root.path().to_path_buf()), + 1024, + Duration::ZERO, + ); + let slot = registry + .begin("default", "alice", "rootfs.tar", 16) + .expect("slot allocated"); + upload(&slot, b"payload"); + let dir = slot.upload_path.parent().unwrap().to_path_buf(); + + let _ = registry.consume(&slot.token); + + assert!(!dir.exists(), "an expired slot must not leave data behind"); + } + + /// The cross-request regression test the review asked for by name. + #[test] + fn authorize_rejects_another_caller() { + let root = temp_root(); + let registry = registry(root.path()); + let slot = registry + .begin("team-a", "alice", "rootfs.tar", 16) + .expect("slot allocated"); + + for (workspace, subject) in [("team-b", "alice"), ("team-a", "bob")] { + let err = registry + .authorize(&slot.token, workspace, subject) + .expect_err("a token minted for another caller must be refused"); + assert_eq!(err.code(), Code::PermissionDenied); + } + + registry + .authorize(&slot.token, "team-a", "alice") + .expect("the rightful owner still holds the slot"); + } + + #[test] + fn consume_rejects_a_slot_that_was_never_uploaded_to() { + let root = temp_root(); + let registry = registry(root.path()); + let slot = registry + .begin("default", "alice", "rootfs.tar", 16) + .expect("slot allocated"); + + let err = registry + .consume(&slot.token) + .expect_err("an empty slot cannot be created from"); + + assert_eq!(err.code(), Code::FailedPrecondition); + } + + #[test] + fn staged_guard_removes_directory_unless_disarmed() { + let root = temp_root(); + let registry = registry(root.path()); + + let slot = registry + .begin("default", "alice", "rootfs.tar", 16) + .expect("slot allocated"); + upload(&slot, b"payload"); + let dir = slot.upload_path.parent().unwrap().to_path_buf(); + drop(registry.consume(&slot.token).expect("consume")); + assert!(!dir.exists(), "dropping the guard must clean up"); + + let slot = registry + .begin("default", "alice", "rootfs.tar", 16) + .expect("slot allocated"); + upload(&slot, b"payload"); + let dir = slot.upload_path.parent().unwrap().to_path_buf(); + let mut staged = registry.consume(&slot.token).expect("consume"); + staged.disarm(); + drop(staged); + assert!( + dir.exists(), + "a disarmed guard leaves the dir to the driver" + ); + } + + #[test] + fn orphan_sweep_removes_only_stale_request_dirs() { + let root = temp_root(); + let stale = root.path().join("req-stale"); + let fresh = root.path().join("req-fresh"); + let unrelated = root.path().join("keep-me"); + for dir in [&stale, &fresh, &unrelated] { + std::fs::create_dir(dir).expect("create dir"); + } + + // A long TTL means nothing on disk has aged out yet. This is what keeps + // a restart from wiping a directory the driver is still copying from. + let young = RootfsTarStagingRegistry::new(Some(root.path().to_path_buf()), 1024); + young.sweep_orphans(); + assert!(stale.exists() && fresh.exists() && unrelated.exists()); + + // A zero TTL ages everything out, but only request directories are ours. + let aged = RootfsTarStagingRegistry::with_ttl( + Some(root.path().to_path_buf()), + 1024, + Duration::ZERO, + ); + aged.sweep_orphans(); + + assert!(!stale.exists()); + assert!(!fresh.exists()); + assert!(unrelated.exists(), "unrelated entries must be left alone"); + } +} diff --git a/crates/openshell-server/src/config_file.rs b/crates/openshell-server/src/config_file.rs index 74b6aad01..64ac95362 100644 --- a/crates/openshell-server/src/config_file.rs +++ b/crates/openshell-server/src/config_file.rs @@ -26,7 +26,6 @@ use std::net::SocketAddr; use std::path::{Path, PathBuf}; use base64::Engine as _; -use openshell_core::config::ComputeDriverKind; use openshell_core::proto::SupervisorMiddlewareService; use openshell_core::{ GatewayAuthConfig, GatewayInterceptorConfig, GatewayJwtConfig, @@ -432,13 +431,22 @@ pub fn driver_table( driver_name: &str, gateway: &GatewayFileSection, raw: Option<&toml::Value>, +) -> toml::Value { + driver_table_with_inherited_keys(driver_name, gateway, raw, &[]) +} + +pub(crate) fn driver_table_with_inherited_keys( + _driver_name: &str, + gateway: &GatewayFileSection, + raw: Option<&toml::Value>, + inheritable_keys: &[&str], ) -> toml::Value { let mut merged = match raw { Some(toml::Value::Table(table)) => table.clone(), _ => toml::Table::new(), }; - for key in inheritable_keys(driver_name) { + for key in inheritable_keys { if merged.contains_key(*key) { continue; } @@ -450,50 +458,6 @@ pub fn driver_table( toml::Value::Table(merged) } -/// Inheritance allowlist (the Q4 "high-overlap set"). Each driver opts in -/// to a specific subset so a gateway-wide default does not accidentally land -/// in a driver table that does not understand the field. -fn inheritable_keys(driver_name: &str) -> &'static [&'static str] { - match driver_name.parse::().ok() { - Some(ComputeDriverKind::Kubernetes) => &[ - "namespace", - "default_image", - "supervisor_image", - "client_tls_secret_name", - "service_account_name", - "host_gateway_ip", - "enable_user_namespaces", - "sa_token_ttl_secs", - ], - Some(ComputeDriverKind::Docker) => &[ - "sandbox_namespace", - "default_image", - "supervisor_image", - "host_gateway_ip", - "guest_tls_ca", - "guest_tls_cert", - "guest_tls_key", - ], - Some(ComputeDriverKind::Podman) => &[ - "default_image", - "supervisor_image", - "host_gateway_ip", - "guest_tls_ca", - "guest_tls_cert", - "guest_tls_key", - ], - Some(ComputeDriverKind::Vm) => &[ - "default_image", - "guest_tls_ca", - "guest_tls_cert", - "guest_tls_key", - ], - // MXC reads its own settings from the driver config table and has no - // gateway-inherited required fields. - Some(ComputeDriverKind::Mxc) | None => &[], - } -} - fn gateway_inherited_value(g: &GatewayFileSection, key: &str) -> Option { match key { "namespace" | "sandbox_namespace" => g.sandbox_namespace.as_deref().map(string_value), @@ -558,7 +522,7 @@ sandbox_namespace = "agents" grpc_rate_limit_requests = 120 grpc_rate_limit_window_seconds = 60 policy_validation_failure_mode = "retain_last_valid" -default_image = "ghcr.io/nvidia/openshell/sandbox:latest" +default_image = "ghcr.io/nvidia/openshell-community/sandboxes/base:latest" supervisor_image = "ghcr.io/nvidia/openshell/supervisor:latest" client_tls_secret_name = "openshell-sandbox-tls" service_account_name = "openshell-sandbox" @@ -585,7 +549,7 @@ namespace = "agents" assert_eq!(gw.log_level.as_deref(), Some("info")); assert_eq!( gw.default_image.as_deref(), - Some("ghcr.io/nvidia/openshell/sandbox:latest") + Some("ghcr.io/nvidia/openshell-community/sandboxes/base:latest") ); assert_eq!(gw.grpc_rate_limit_requests, Some(120)); assert_eq!(gw.grpc_rate_limit_window_seconds, Some(60)); @@ -979,17 +943,20 @@ version = 2 #[test] fn driver_table_inherits_gateway_defaults() { let gateway = GatewayFileSection { - default_image: Some("ghcr.io/nvidia/openshell/sandbox:0.9".to_string()), + default_image: Some( + "ghcr.io/nvidia/openshell-community/sandboxes/base:latest".to_string(), + ), supervisor_image: Some("ghcr.io/nvidia/openshell/supervisor:0.9".to_string()), ..Default::default() }; let raw = toml::toml! { namespace = "agents" }; - let merged = driver_table( - ComputeDriverKind::Kubernetes.as_str(), + let merged = driver_table_with_inherited_keys( + "alpha", &gateway, Some(&toml::Value::Table(raw)), + &["default_image", "supervisor_image"], ); let table = merged.as_table().expect("table"); assert_eq!( @@ -998,7 +965,7 @@ version = 2 ); assert_eq!( table.get("default_image").and_then(|v| v.as_str()), - Some("ghcr.io/nvidia/openshell/sandbox:0.9") + Some("ghcr.io/nvidia/openshell-community/sandboxes/base:latest") ); assert_eq!( table.get("supervisor_image").and_then(|v| v.as_str()), @@ -1007,14 +974,21 @@ version = 2 } #[test] - fn docker_driver_table_inherits_gateway_defaults() { + fn registered_driver_table_inherits_selected_gateway_defaults() { let gateway = GatewayFileSection { sandbox_namespace: Some("agents".to_string()), - default_image: Some("ghcr.io/nvidia/openshell/sandbox:0.9".to_string()), + default_image: Some( + "ghcr.io/nvidia/openshell-community/sandboxes/base:latest".to_string(), + ), host_gateway_ip: Some("10.0.0.1".to_string()), ..Default::default() }; - let merged = driver_table(ComputeDriverKind::Docker.as_str(), &gateway, None); + let merged = driver_table_with_inherited_keys( + "alpha", + &gateway, + None, + &["sandbox_namespace", "default_image", "host_gateway_ip"], + ); let table = merged.as_table().expect("table"); assert_eq!( table.get("sandbox_namespace").and_then(|v| v.as_str()), @@ -1022,7 +996,7 @@ version = 2 ); assert_eq!( table.get("default_image").and_then(|v| v.as_str()), - Some("ghcr.io/nvidia/openshell/sandbox:0.9") + Some("ghcr.io/nvidia/openshell-community/sandboxes/base:latest") ); assert_eq!( table.get("host_gateway_ip").and_then(|v| v.as_str()), @@ -1031,17 +1005,24 @@ version = 2 } #[test] - fn podman_driver_table_inherits_gateway_host_gateway_ip() { + fn registered_driver_table_can_select_network_defaults() { let gateway = GatewayFileSection { - default_image: Some("ghcr.io/nvidia/openshell/sandbox:0.9".to_string()), + default_image: Some( + "ghcr.io/nvidia/openshell-community/sandboxes/base:latest".to_string(), + ), host_gateway_ip: Some("192.168.127.254".to_string()), ..Default::default() }; - let merged = driver_table(ComputeDriverKind::Podman.as_str(), &gateway, None); + let merged = driver_table_with_inherited_keys( + "beta", + &gateway, + None, + &["default_image", "host_gateway_ip"], + ); let table = merged.as_table().expect("table"); assert_eq!( table.get("default_image").and_then(|v| v.as_str()), - Some("ghcr.io/nvidia/openshell/sandbox:0.9") + Some("ghcr.io/nvidia/openshell-community/sandboxes/base:latest") ); assert_eq!( table.get("host_gateway_ip").and_then(|v| v.as_str()), @@ -1058,10 +1039,11 @@ version = 2 let raw = toml::toml! { default_image = "driver-specific" }; - let merged = driver_table( - ComputeDriverKind::Podman.as_str(), + let merged = driver_table_with_inherited_keys( + "alpha", &gateway, Some(&toml::Value::Table(raw)), + &["default_image"], ); assert_eq!( merged @@ -1075,13 +1057,12 @@ version = 2 #[test] fn driver_table_does_not_leak_keys_outside_allowlist() { - // `client_tls_secret_name` is K8s-only; Docker must not receive it - // even when set at gateway scope. + // Fields not selected by the registration must remain gateway-only. let gateway = GatewayFileSection { client_tls_secret_name: Some("openshell-sandbox-tls".to_string()), ..Default::default() }; - let merged = driver_table(ComputeDriverKind::Docker.as_str(), &gateway, None); + let merged = driver_table_with_inherited_keys("alpha", &gateway, None, &["default_image"]); assert!( !merged .as_table() diff --git a/crates/openshell-server/src/credentials.rs b/crates/openshell-server/src/credentials.rs index 12d2835be..e1ea5994c 100644 --- a/crates/openshell-server/src/credentials.rs +++ b/crates/openshell-server/src/credentials.rs @@ -30,7 +30,7 @@ use openshell_core::proto::credentials::v1::{ GetCredentialDriverCapabilitiesResponse, ResolveCredentialRequest, ResolveCredentialsRequest, ResolvedCredential, StoreCredentialRequest, credential_driver_client::CredentialDriverClient, }; -use openshell_core::proto::{CredentialHandle, Provider, StoredRefreshMaterialDeletion}; +use openshell_core::proto::{CredentialHandle, Provider}; use openshell_core::{Config, Error, Result as CoreResult}; use openshell_driver_db_credstore::{ CredentialObjectWrite, DbCredstoreCredentialDriver, DbCredstoreObjectStore, @@ -51,6 +51,7 @@ use tower::service_fn; use tracing::warn; use crate::persistence::{PersistenceError, Store, WriteCondition}; +use crate::storage_proto::StoredRefreshMaterialDeletion; const DEFAULT_CREDENTIAL_DRIVER_STARTUP_TIMEOUT_SECS: u64 = 10; const DEFAULT_CREDENTIAL_DRIVER_RPC_TIMEOUT_SECS: u64 = 30; diff --git a/crates/openshell-server/src/gateway_listener.rs b/crates/openshell-server/src/gateway_listener.rs index 640757bfb..b638fc33e 100644 --- a/crates/openshell-server/src/gateway_listener.rs +++ b/crates/openshell-server/src/gateway_listener.rs @@ -386,10 +386,10 @@ mod tests { #[test] fn gateway_listener_specs_reuse_primary_when_wildcard_covers_driver_address() { let primary: SocketAddr = "0.0.0.0:8080".parse().unwrap(); - let docker: SocketAddr = "172.18.0.1:8080".parse().unwrap(); + let callback: SocketAddr = "172.18.0.1:8080".parse().unwrap(); let requirements = [ - docker_listener_requirement(docker), - docker_listener_requirement(docker), + exact_listener_requirement(callback), + exact_listener_requirement(callback), ]; assert_eq!( @@ -401,15 +401,15 @@ mod tests { #[test] fn gateway_listener_scope_for_reused_primary_remains_primary() { let primary: SocketAddr = "0.0.0.0:8080".parse().unwrap(); - let docker: SocketAddr = "172.18.0.1:8080".parse().unwrap(); + let callback: SocketAddr = "172.18.0.1:8080".parse().unwrap(); let loopback: SocketAddr = "127.0.0.1:8080".parse().unwrap(); - let [spec] = gateway_listener_specs(primary, &[docker_listener_requirement(docker)]) + let [spec] = gateway_listener_specs(primary, &[exact_listener_requirement(callback)]) .unwrap() .try_into() .unwrap(); assert_eq!( - spec.scope_for_local_addr(docker), + spec.scope_for_local_addr(callback), GatewayListenerScope::Primary, ); assert_eq!( @@ -421,10 +421,10 @@ mod tests { #[test] fn gateway_listener_specs_preserve_driver_callback_scope() { let primary: SocketAddr = "127.0.0.1:8080".parse().unwrap(); - let docker: SocketAddr = "172.18.0.1:8080".parse().unwrap(); + let callback: SocketAddr = "172.18.0.1:8080".parse().unwrap(); let requirements = [ - docker_listener_requirement(docker), - docker_listener_requirement(docker), + exact_listener_requirement(callback), + exact_listener_requirement(callback), ]; assert_eq!( @@ -437,11 +437,11 @@ mod tests { provenance: None, }, GatewayListenerSpec { - address: docker, + address: callback, scope: GatewayListenerScope::ComputeDriverCallback, covered_addresses: Vec::new(), provenance: Some(GatewayListenerProvenance { - driver_name: "docker".to_string(), + driver_name: "alpha".to_string(), reason: "managed bridge".to_string(), }), }, @@ -473,7 +473,7 @@ mod tests { "172.18.0.1:0", "172.18.0.1:9090", ] { - let requirement = docker_listener_requirement(address.parse().unwrap()); + let requirement = exact_listener_requirement(address.parse().unwrap()); assert!( gateway_listener_specs(primary, &[requirement]).is_err(), "{address} should be rejected" @@ -482,41 +482,41 @@ mod tests { } #[test] - fn gateway_listener_specs_use_exact_podman_network_gateway() { + fn gateway_listener_specs_use_exact_network_gateway() { let primary: SocketAddr = "127.0.0.1:8080".parse().unwrap(); - let podman_gateway: SocketAddr = "10.89.1.1:8080".parse().unwrap(); + let network_gateway: SocketAddr = "10.89.1.1:8080".parse().unwrap(); assert_eq!( - gateway_listener_specs(primary, &[podman_listener_requirement(podman_gateway)]) + gateway_listener_specs(primary, &[network_listener_requirement(network_gateway)]) .unwrap(), vec![ primary_listener_spec(primary), - callback_listener_spec(podman_gateway, "podman", "Podman managed bridge",), + callback_listener_spec(network_gateway, "beta", "managed bridge",), ] ); } #[test] - fn gateway_listener_specs_reuse_primary_when_it_covers_podman_exact() { + fn gateway_listener_specs_reuse_primary_when_it_covers_exact_requirement() { let primary: SocketAddr = "0.0.0.0:8080".parse().unwrap(); - let podman_gateway: SocketAddr = "10.89.1.1:8080".parse().unwrap(); + let network_gateway: SocketAddr = "10.89.1.1:8080".parse().unwrap(); assert_eq!( - gateway_listener_specs(primary, &[podman_listener_requirement(podman_gateway)],) + gateway_listener_specs(primary, &[network_listener_requirement(network_gateway)],) .unwrap(), vec![primary_listener_spec(primary)] ); } #[test] - fn gateway_listener_specs_resolve_podman_default_route_source() { + fn gateway_listener_specs_resolve_default_route_source() { let primary: SocketAddr = "127.0.0.1:8080".parse().unwrap(); let default_route_ip = "192.168.20.20".parse().unwrap(); assert_eq!( gateway_listener_specs_with_default_route_ip( primary, - &[podman_default_route_listener_requirement()], + &[default_route_listener_requirement()], Some(default_route_ip), ) .unwrap(), @@ -524,8 +524,8 @@ mod tests { primary_listener_spec(primary), callback_listener_spec( "192.168.20.20:8080".parse().unwrap(), - "podman", - "rootless pasta upstream interface", + "beta", + "default route interface", ), ] ); @@ -537,7 +537,7 @@ mod tests { let err = gateway_listener_specs_with_default_route_ip( primary, - &[podman_default_route_listener_requirement()], + &[default_route_listener_requirement()], Some("203.0.113.20".parse().unwrap()), ) .unwrap_err(); @@ -552,7 +552,7 @@ mod tests { assert_eq!( gateway_listener_specs_with_default_route_ip( primary, - &[podman_default_route_listener_requirement()], + &[default_route_listener_requirement()], Some(default_route_ip), ) .unwrap(), @@ -561,28 +561,24 @@ mod tests { } #[test] - fn gateway_listener_specs_resolve_podman_loopback_separately() { + fn gateway_listener_specs_resolve_loopback_separately() { let primary: SocketAddr = "192.168.20.20:8080".parse().unwrap(); assert_eq!( - gateway_listener_specs(primary, &[podman_loopback_listener_requirement()]).unwrap(), + gateway_listener_specs(primary, &[loopback_listener_requirement()]).unwrap(), vec![ primary_listener_spec(primary), - callback_listener_spec( - "127.0.0.1:8080".parse().unwrap(), - "podman", - "Podman machine host forwarder", - ), + callback_listener_spec("127.0.0.1:8080".parse().unwrap(), "beta", "host forwarder",), ] ); } #[test] - fn gateway_listener_specs_reuse_wildcard_primary_for_podman_loopback() { + fn gateway_listener_specs_reuse_wildcard_primary_for_loopback() { let primary = "0.0.0.0:8080".parse().unwrap(); assert_eq!( - gateway_listener_specs(primary, &[podman_loopback_listener_requirement()]).unwrap(), + gateway_listener_specs(primary, &[loopback_listener_requirement()]).unwrap(), vec![primary_listener_spec(primary)] ); } @@ -592,7 +588,7 @@ mod tests { let primary = "127.0.0.1:8080".parse().unwrap(); assert_eq!( - gateway_listener_specs(primary, &[podman_loopback_listener_requirement()]).unwrap(), + gateway_listener_specs(primary, &[loopback_listener_requirement()]).unwrap(), vec![primary_listener_spec(primary)] ); } @@ -602,7 +598,7 @@ mod tests { for primary in ["[::1]:8080", "[::]:8080"] { let primary = primary.parse().unwrap(); let specs = - gateway_listener_specs(primary, &[podman_loopback_listener_requirement()]).unwrap(); + gateway_listener_specs(primary, &[loopback_listener_requirement()]).unwrap(); assert_eq!(specs.len(), 2); assert_eq!(specs[1].address, SocketAddr::from(([127, 0, 0, 1], 8080))); @@ -613,7 +609,7 @@ mod tests { fn gateway_listener_specs_validate_selector_independently_of_driver_name() { let primary: SocketAddr = "192.168.20.20:8080".parse().unwrap(); let requirement = GatewayListenerRequirement::LoopbackInterface { - driver_name: "docker".to_string(), + driver_name: "alpha".to_string(), reason: "wrong selector".to_string(), }; @@ -634,7 +630,7 @@ mod tests { let result: openshell_core::Result<()> = async { let _listeners = bind_gateway_listeners( primary_address, - &[docker_listener_requirement(occupied_address)], + &[exact_listener_requirement(occupied_address)], ) .await?; continuation_reached.store(true, Ordering::SeqCst); @@ -663,7 +659,7 @@ mod tests { drop(probe); let primary = format!("[::]:{port}").parse().unwrap(); - let listeners = bind_gateway_listeners(primary, &[podman_loopback_listener_requirement()]) + let listeners = bind_gateway_listeners(primary, &[loopback_listener_requirement()]) .await .expect("IPv6 wildcard and IPv4 callback listeners should both bind"); @@ -675,33 +671,33 @@ mod tests { ); } - fn docker_listener_requirement(address: SocketAddr) -> GatewayListenerRequirement { + fn exact_listener_requirement(address: SocketAddr) -> GatewayListenerRequirement { GatewayListenerRequirement::Exact { address, - driver_name: "docker".to_string(), + driver_name: "alpha".to_string(), reason: "managed bridge".to_string(), } } - fn podman_listener_requirement(address: SocketAddr) -> GatewayListenerRequirement { + fn network_listener_requirement(address: SocketAddr) -> GatewayListenerRequirement { GatewayListenerRequirement::Exact { address, - driver_name: "podman".to_string(), - reason: "Podman managed bridge".to_string(), + driver_name: "beta".to_string(), + reason: "managed bridge".to_string(), } } - fn podman_default_route_listener_requirement() -> GatewayListenerRequirement { + fn default_route_listener_requirement() -> GatewayListenerRequirement { GatewayListenerRequirement::DefaultRouteInterface { - driver_name: "podman".to_string(), - reason: "rootless pasta upstream interface".to_string(), + driver_name: "beta".to_string(), + reason: "default route interface".to_string(), } } - fn podman_loopback_listener_requirement() -> GatewayListenerRequirement { + fn loopback_listener_requirement() -> GatewayListenerRequirement { GatewayListenerRequirement::LoopbackInterface { - driver_name: "podman".to_string(), - reason: "Podman machine host forwarder".to_string(), + driver_name: "beta".to_string(), + reason: "host forwarder".to_string(), } } diff --git a/crates/openshell-server/src/grpc/auth_rpc.rs b/crates/openshell-server/src/grpc/auth_rpc.rs index 104d63958..62547cc29 100644 --- a/crates/openshell-server/src/grpc/auth_rpc.rs +++ b/crates/openshell-server/src/grpc/auth_rpc.rs @@ -193,7 +193,7 @@ pub async fn handle_refresh_sandbox_token( } const MAX_EXTENSION_CREDENTIALS_PER_REFRESH: usize = 64; -const DEFAULT_EXTENSION_TOKEN_TTL: Duration = Duration::from_secs(15 * 60); +const DEFAULT_EXTENSION_TOKEN_TTL: Duration = Duration::from_mins(15); #[allow(clippy::result_large_err)] fn mint_extension_credentials( @@ -323,7 +323,7 @@ mod tests { mat.signing_key_pem.as_bytes(), mat.kid, "test-gateway", - Duration::from_secs(3600), + Duration::from_hours(1), ) .unwrap(); state.sandbox_jwt_issuer = Some(Arc::new(issuer)); diff --git a/crates/openshell-server/src/grpc/mod.rs b/crates/openshell-server/src/grpc/mod.rs index 957a77cba..c5a12a15b 100644 --- a/crates/openshell-server/src/grpc/mod.rs +++ b/crates/openshell-server/src/grpc/mod.rs @@ -14,41 +14,45 @@ pub mod workspace; use openshell_core::proto::{ AddWorkspaceMemberRequest, AddWorkspaceMemberResponse, ApproveAllDraftChunksRequest, ApproveAllDraftChunksResponse, ApproveDraftChunkRequest, ApproveDraftChunkResponse, - AttachSandboxProviderRequest, AttachSandboxProviderResponse, ClearDraftChunksRequest, - ClearDraftChunksResponse, ComputeDriverCapabilities, ComputeDriverInfo, - ConfigureProviderRefreshRequest, ConfigureProviderRefreshResponse, CreateProviderRequest, - CreateSandboxRequest, CreateSshSessionRequest, CreateSshSessionResponse, - CreateWorkspaceRequest, CreateWorkspaceResponse, DeleteProviderProfileRequest, - DeleteProviderProfileResponse, DeleteProviderRefreshRequest, DeleteProviderRefreshResponse, - DeleteProviderRequest, DeleteProviderResponse, DeleteSandboxRequest, DeleteSandboxResponse, - DeleteServiceRequest, DeleteServiceResponse, DeleteWorkspaceRequest, DeleteWorkspaceResponse, - DetachSandboxProviderRequest, DetachSandboxProviderResponse, EditDraftChunkRequest, - EditDraftChunkResponse, ExchangeProviderSubjectTokenRequest, - ExchangeProviderSubjectTokenResponse, ExecSandboxEvent, ExecSandboxInput, ExecSandboxRequest, - ExposeServiceRequest, FinalizeMainProcessExitRequest, FinalizeMainProcessExitResponse, - GatewayMessage, GetCurrentUserRequest, GetCurrentUserResponse, GetDraftHistoryRequest, - GetDraftHistoryResponse, GetDraftPolicyRequest, GetDraftPolicyResponse, + AttachSandboxProviderRequest, AttachSandboxProviderResponse, BeginRootfsTarStagingRequest, + BeginRootfsTarStagingResponse, ClearDraftChunksRequest, ClearDraftChunksResponse, + ComputeDriverCapabilities, ComputeDriverInfo, ConfigureProviderRefreshRequest, + ConfigureProviderRefreshResponse, CpuResourceCapabilities, CreateProviderRequest, + CreateSandboxRequest, CreateSandboxTemplateRequest, CreateSshSessionRequest, + CreateSshSessionResponse, CreateWorkspaceRequest, CreateWorkspaceResponse, + DeleteProviderProfileRequest, DeleteProviderProfileResponse, DeleteProviderRefreshRequest, + DeleteProviderRefreshResponse, DeleteProviderRequest, DeleteProviderResponse, + DeleteSandboxRequest, DeleteSandboxResponse, DeleteSandboxTemplateRequest, + DeleteSandboxTemplateResponse, DeleteServiceRequest, DeleteServiceResponse, + DeleteWorkspaceRequest, DeleteWorkspaceResponse, DetachSandboxProviderRequest, + DetachSandboxProviderResponse, EditDraftChunkRequest, EditDraftChunkResponse, + ExchangeProviderSubjectTokenRequest, ExchangeProviderSubjectTokenResponse, ExecSandboxEvent, + ExecSandboxInput, ExecSandboxRequest, ExposeServiceRequest, FinalizeMainProcessExitRequest, + FinalizeMainProcessExitResponse, GatewayMessage, GetCurrentUserRequest, GetCurrentUserResponse, + GetDraftHistoryRequest, GetDraftHistoryResponse, GetDraftPolicyRequest, GetDraftPolicyResponse, GetGatewayConfigRequest, GetGatewayConfigResponse, GetGatewayInfoRequest, GetGatewayInfoResponse, GetProviderProfileRequest, GetProviderRefreshStatusRequest, GetProviderRefreshStatusResponse, GetProviderRequest, GetSandboxConfigRequest, GetSandboxConfigResponse, GetSandboxLogsRequest, GetSandboxLogsResponse, GetSandboxPolicyStatusRequest, GetSandboxPolicyStatusResponse, GetSandboxProviderEnvironmentRequest, GetSandboxProviderEnvironmentResponse, GetSandboxRequest, - GetServiceRequest, GetWorkspaceRequest, GetWorkspaceResponse, HealthRequest, HealthResponse, - ImportProviderProfilesRequest, ImportProviderProfilesResponse, IssueSandboxTokenRequest, - IssueSandboxTokenResponse, LintProviderProfilesRequest, LintProviderProfilesResponse, - ListProviderProfilesRequest, ListProviderProfilesResponse, ListProvidersRequest, - ListProvidersResponse, ListSandboxPoliciesRequest, ListSandboxPoliciesResponse, - ListSandboxProvidersRequest, ListSandboxProvidersResponse, ListSandboxesRequest, - ListSandboxesResponse, ListServicesRequest, ListServicesResponse, ListWorkspaceMembersRequest, - ListWorkspaceMembersResponse, ListWorkspacesRequest, ListWorkspacesResponse, - ProviderProfileResponse, ProviderResponse, PushSandboxLogsRequest, PushSandboxLogsResponse, - RefreshSandboxTokenRequest, RefreshSandboxTokenResponse, RejectDraftChunkRequest, - RejectDraftChunkResponse, RelayFrame, RemoveWorkspaceMemberRequest, - RemoveWorkspaceMemberResponse, ReportMainProcessExitRequest, ReportMainProcessExitResponse, - ReportPolicyStatusRequest, ReportPolicyStatusResponse, RevokeSshSessionRequest, - RevokeSshSessionResponse, RotateProviderCredentialRequest, RotateProviderCredentialResponse, - SandboxResponse, ServiceEndpointResponse, ServiceStatus, StartSandboxRequest, + GetSandboxTemplateRequest, GetServiceRequest, GetWorkspaceRequest, GetWorkspaceResponse, + GpuResourceCapabilities, HealthRequest, HealthResponse, ImportProviderProfilesRequest, + ImportProviderProfilesResponse, IssueSandboxTokenRequest, IssueSandboxTokenResponse, + LintProviderProfilesRequest, LintProviderProfilesResponse, ListProviderProfilesRequest, + ListProviderProfilesResponse, ListProvidersRequest, ListProvidersResponse, + ListSandboxPoliciesRequest, ListSandboxPoliciesResponse, ListSandboxProvidersRequest, + ListSandboxProvidersResponse, ListSandboxTemplatesRequest, ListSandboxTemplatesResponse, + ListSandboxesRequest, ListSandboxesResponse, ListServicesRequest, ListServicesResponse, + ListWorkspaceMembersRequest, ListWorkspaceMembersResponse, ListWorkspacesRequest, + ListWorkspacesResponse, MemoryResourceCapabilities, ProviderProfileResponse, ProviderResponse, + PushSandboxLogsRequest, PushSandboxLogsResponse, RefreshSandboxTokenRequest, + RefreshSandboxTokenResponse, RejectDraftChunkRequest, RejectDraftChunkResponse, RelayFrame, + RemoveWorkspaceMemberRequest, RemoveWorkspaceMemberResponse, ReportMainProcessExitRequest, + ReportMainProcessExitResponse, ReportPolicyStatusRequest, ReportPolicyStatusResponse, + ResourceCapabilities, RevokeSshSessionRequest, RevokeSshSessionResponse, + RotateProviderCredentialRequest, RotateProviderCredentialResponse, SandboxResponse, + SandboxTemplateResponse, ServiceEndpointResponse, ServiceStatus, StartSandboxRequest, StopSandboxRequest, SubmitPolicyAnalysisRequest, SubmitPolicyAnalysisResponse, SupervisorMessage, TcpForwardFrame, UndoDraftChunkRequest, UndoDraftChunkResponse, UpdateConfigRequest, UpdateConfigResponse, UpdateProviderProfilesRequest, @@ -187,19 +191,6 @@ enum StoredSettingValue { // Utility // --------------------------------------------------------------------------- -/// Validate that object metadata is present and contains required fields. -/// -/// This is a crate-level helper that wraps the validation module's implementation. -/// Use this from modules outside of `grpc` that need to validate metadata. -// `tonic::Status` is large but is the API surface of gRPC handlers. -#[allow(clippy::result_large_err)] -pub fn validate_object_metadata( - metadata: Option<&openshell_core::proto::datamodel::v1::ObjectMeta>, - resource_type: &str, -) -> Result<(), Status> { - validation::validate_object_metadata(metadata, resource_type) -} - // --------------------------------------------------------------------------- // Service struct // --------------------------------------------------------------------------- @@ -256,6 +247,10 @@ impl OpenShell for OpenShellService { capabilities: Some(ComputeDriverCapabilities { driver_name: driver.driver_name.clone(), driver_version: driver.driver_version.clone(), + resource_capabilities: driver + .resource_capabilities + .as_ref() + .map(|resources| public_resource_capabilities(*resources)), }), }) .collect(); @@ -276,6 +271,13 @@ impl OpenShell for OpenShellService { sandbox::handle_create_sandbox(&self.state, request).await } + async fn begin_rootfs_tar_staging( + &self, + request: Request, + ) -> Result, Status> { + sandbox::handle_begin_rootfs_tar_staging(&self.state, request).await + } + type WatchSandboxStream = sandbox::WatchSandboxStream; async fn watch_sandbox( @@ -299,6 +301,34 @@ impl OpenShell for OpenShellService { sandbox::handle_list_sandboxes(&self.state, request).await } + async fn create_sandbox_template( + &self, + request: Request, + ) -> Result, Status> { + sandbox::handle_create_sandbox_template(&self.state, request).await + } + + async fn get_sandbox_template( + &self, + request: Request, + ) -> Result, Status> { + sandbox::handle_get_sandbox_template(&self.state, request).await + } + + async fn list_sandbox_templates( + &self, + request: Request, + ) -> Result, Status> { + sandbox::handle_list_sandbox_templates(&self.state, request).await + } + + async fn delete_sandbox_template( + &self, + request: Request, + ) -> Result, Status> { + sandbox::handle_delete_sandbox_template(&self.state, request).await + } + async fn list_sandbox_providers( &self, request: Request, @@ -765,6 +795,26 @@ impl OpenShell for OpenShellService { } } +fn public_resource_capabilities( + resources: openshell_core::proto::compute::v1::ResourceCapabilities, +) -> ResourceCapabilities { + ResourceCapabilities { + cpu: resources.cpu.as_ref().map(|cpu| CpuResourceCapabilities { + limit_supported: cpu.limit_supported, + }), + memory: resources + .memory + .as_ref() + .map(|memory| MemoryResourceCapabilities { + limit_supported: memory.limit_supported, + }), + gpu: resources.gpu.as_ref().map(|gpu| GpuResourceCapabilities { + default_selection_supported: gpu.default_selection_supported, + count_selection_supported: gpu.count_selection_supported, + }), + } +} + // --------------------------------------------------------------------------- // Shared test support // --------------------------------------------------------------------------- @@ -875,6 +925,12 @@ pub mod test_support { #[cfg(test)] mod tests { use super::*; + use openshell_core::proto::compute::v1::{ + CpuResourceCapabilities as DriverCpuResourceCapabilities, + GpuResourceCapabilities as DriverGpuResourceCapabilities, + MemoryResourceCapabilities as DriverMemoryResourceCapabilities, + ResourceCapabilities as DriverResourceCapabilities, + }; #[test] fn clamp_limit_zero_returns_default() { @@ -900,4 +956,39 @@ mod tests { ); assert_eq!(clamp_limit(u32::MAX, 100, MAX_PAGE_SIZE), MAX_PAGE_SIZE); } + + #[test] + fn public_resource_capabilities_preserves_reported_fields() { + let driver_capabilities = DriverResourceCapabilities { + cpu: Some(DriverCpuResourceCapabilities { + limit_supported: true, + }), + memory: Some(DriverMemoryResourceCapabilities { + limit_supported: false, + }), + gpu: Some(DriverGpuResourceCapabilities { + default_selection_supported: true, + count_selection_supported: true, + }), + }; + + let capabilities = public_resource_capabilities(driver_capabilities); + + assert!(capabilities.cpu.expect("CPU capabilities").limit_supported); + assert!( + !capabilities + .memory + .expect("memory capabilities") + .limit_supported + ); + let gpu = capabilities.gpu.expect("GPU capabilities"); + assert!(gpu.default_selection_supported); + assert!(gpu.count_selection_supported); + } + + #[test] + fn public_resource_capabilities_preserves_absence() { + let absent: Option = None; + assert!(absent.map(public_resource_capabilities).is_none()); + } } diff --git a/crates/openshell-server/src/grpc/policy.rs b/crates/openshell-server/src/grpc/policy.rs index 0bda93a15..4f2855dd0 100644 --- a/crates/openshell-server/src/grpc/policy.rs +++ b/crates/openshell-server/src/grpc/policy.rs @@ -13,7 +13,8 @@ use crate::ServerState; use crate::auth::principal::Principal; use crate::auth::workspace_authz::{ - MinWorkspaceRole, authorize_sandbox_workspace, authorize_workspace, require_platform_admin, + MinWorkspaceRole, authorize_sandbox_workspace, authorize_workspace_selector, + require_platform_admin, selected_workspace_name, }; use crate::persistence::{ DraftChunkRecord, ObjectId, ObjectName, ObjectType, ObjectWorkspace, PolicyRecord, Store, @@ -22,6 +23,9 @@ use crate::policy_store::{AtomicPolicyRevisionWrite, PolicyStoreExt}; use crate::provider_profile_sources::EffectiveProviderProfileCatalog; #[cfg(test)] use crate::provider_profile_sources::ProviderProfileSources; +use crate::storage_proto::StoredProviderCredentialRefreshState; +#[cfg(test)] +use crate::storage_proto::StoredProviderProfile; use openshell_core::net::{is_always_blocked_ip, is_internal_ip}; use openshell_core::proto::policy_merge_operation; use openshell_core::proto::setting_value; @@ -71,7 +75,6 @@ use openshell_prover::{ registry::load_embedded_binary_registry, report::finding_shorthand, }; -use openshell_providers::normalize_provider_type; use prost::Message; use sha2::{Digest, Sha256}; use std::collections::{BTreeMap, HashMap, HashSet}; @@ -81,8 +84,9 @@ use tonic::{Request, Response, Status}; use tracing::{debug, info, warn}; use super::validation::{ - level_matches, source_matches, validate_annotations, validate_no_reserved_provider_policy_keys, - validate_policy_safety, validate_static_fields_unchanged, + level_matches, source_matches, validate_and_canonicalize_policy, validate_annotations, + validate_no_reserved_provider_policy_keys, validate_policy_safety, + validate_static_fields_unchanged, }; use super::{MAX_PAGE_SIZE, StoredSettingValue, StoredSettings, clamp_limit}; use crate::persistence::current_time_ms; @@ -100,6 +104,10 @@ pub const SANDBOX_SETTINGS_OBJECT_TYPE: &str = "sandbox_settings"; const POLICY_SETTING_KEY: &str = "policy"; /// Sentinel `sandbox_id` used to store global policy revisions. const GLOBAL_POLICY_SANDBOX_ID: &str = "__global__"; +/// Stable labels used when stored policy state fails validation. +const STORED_POLICY_SOURCE_HISTORY: &str = "sandbox policy history"; +const STORED_POLICY_SOURCE_SPEC: &str = "sandbox spec policy"; +const STORED_POLICY_SOURCE_GLOBAL: &str = "global policy setting"; /// Maximum number of optimistic retry attempts for policy version conflicts. const MERGE_RETRY_LIMIT: usize = 5; @@ -809,10 +817,9 @@ async fn build_credential_set_for_sandbox_with_catalog( }; let provider_type = provider.r#type.trim(); - let profile_id = normalize_provider_type(provider_type).unwrap_or(provider_type); let Some(profile) = super::provider::get_provider_type_profile_for_scope( catalog, - profile_id, + provider_type, &provider.profile_workspace, ) else { warn!( @@ -1582,8 +1589,8 @@ async fn auto_approve_chunk( } // TODO: share effective-policy lookup with `load_sandbox_policy` / -// `GetSandboxConfig`. They re-implement very similar global-settings + -// providers_v2 + compose logic; consolidating them is out of scope for the +// `GetSandboxConfig`. They re-implement very similar global-settings and +// profile-composition logic; consolidating them is out of scope for the // agent-authored proposal validation slice. async fn current_effective_policy_for_sandbox( state: &ServerState, @@ -1597,23 +1604,47 @@ async fn current_effective_policy_for_sandbox( .as_ref() .map(|spec| spec.providers.clone()) .unwrap_or_default(); + let global_settings = load_global_settings(state.store.as_ref()).await?; + if let Some(global_policy) = decode_policy_from_global_settings(&global_settings)? { + // A global policy is the complete effective policy. Dormant sandbox + // history and specs may predate the current schema, but they must not + // prevent the valid global policy from being served. + return apply_effective_policy_context( + state, + catalog, + workspace, + &provider_names, + global_policy, + PolicySource::Global, + ) + .await; + } + let policy = if let Some(record) = state .store .get_latest_policy(sandbox_id) .await .map_err(|e| Status::internal(format!("fetch latest policy failed: {e}")))? { - ProtoSandboxPolicy::decode(record.policy_payload.as_slice()) - .map_err(|e| Status::internal(format!("decode current policy failed: {e}")))? + canonical_policy_record_identity(&record)?.0 } else { - sandbox - .spec - .as_ref() - .and_then(|spec| spec.policy.clone()) - .unwrap_or_default() + match sandbox.spec.as_ref().and_then(|spec| spec.policy.clone()) { + Some(policy) => { + validate_and_canonicalize_stored_policy(policy, STORED_POLICY_SOURCE_SPEC)? + } + None => ProtoSandboxPolicy::default(), + } }; - effective_policy_for_source(state, catalog, workspace, &provider_names, policy).await + apply_effective_policy_context( + state, + catalog, + workspace, + &provider_names, + policy, + PolicySource::Sandbox, + ) + .await } async fn effective_policy_for_source( @@ -1632,8 +1663,25 @@ async fn effective_policy_for_source( }, ); - let providers_v2_enabled = - bool_setting_enabled(&global_settings, settings::PROVIDERS_V2_ENABLED_KEY)?; + apply_effective_policy_context( + state, + catalog, + workspace, + provider_names, + policy, + policy_source, + ) + .await +} + +async fn apply_effective_policy_context( + state: &ServerState, + catalog: &EffectiveProviderProfileCatalog, + workspace: &str, + provider_names: &[String], + mut policy: ProtoSandboxPolicy, + policy_source: PolicySource, +) -> Result { clear_provider_credentialed_markers(&mut policy); let mut provider_context = provider_policy_context_with_catalog( state.store.as_ref(), @@ -1642,10 +1690,7 @@ async fn effective_policy_for_source( provider_names, ) .await?; - if providers_v2_enabled - && !matches!(policy_source, PolicySource::Global) - && !provider_context.layers.is_empty() - { + if !matches!(policy_source, PolicySource::Global) && !provider_context.layers.is_empty() { policy = compose_effective_policy(&policy, &provider_context.layers); } let policy_credential_bindings = policy_static_credential_endpoint_bindings(Some(&policy))?; @@ -1782,11 +1827,9 @@ fn validate_policy_credential_binding_context( "credential_binding references provider '{provider_name}', but that provider is not attached to the sandbox" )) })?; - let profile_id = normalize_provider_type(&record.provider.r#type) - .unwrap_or(record.provider.r#type.as_str()); let profile = super::provider::get_provider_type_profile_for_scope( catalog, - profile_id, + &record.provider.r#type, &record.provider.profile_workspace, ) .ok_or_else(|| { @@ -1859,11 +1902,9 @@ fn signing_profile_for_record( catalog: &EffectiveProviderProfileCatalog, record: &super::provider::ProviderEnvironmentRecord, ) -> Option { - let profile_id = - normalize_provider_type(&record.provider.r#type).unwrap_or(record.provider.r#type.as_str()); super::provider::get_provider_type_profile_for_scope( catalog, - profile_id, + &record.provider.r#type, &record.provider.profile_workspace, ) } @@ -1966,9 +2007,7 @@ async fn provider_policy_layers_for_sandbox( provider_names: &[String], ) -> Result, Status> { let global_settings = load_global_settings(state.store.as_ref()).await?; - if decode_policy_from_global_settings(&global_settings)?.is_some() - || !bool_setting_enabled(&global_settings, settings::PROVIDERS_V2_ENABLED_KEY)? - { + if decode_policy_from_global_settings(&global_settings)?.is_some() { return Ok(Vec::new()); } let catalog = state @@ -2000,14 +2039,17 @@ pub(super) async fn current_base_policy_for_sandbox( .await .map_err(|e| Status::internal(format!("fetch latest policy failed: {e}")))? { - return ProtoSandboxPolicy::decode(record.policy_payload.as_slice()) - .map_err(|e| Status::internal(format!("decode current policy failed: {e}"))); + let (policy, _) = canonical_policy_record_identity(&record)?; + return Ok(policy); } - Ok(sandbox + sandbox .spec .as_ref() .and_then(|spec| spec.policy.clone()) - .unwrap_or_default()) + .map_or_else( + || Ok(ProtoSandboxPolicy::default()), + |policy| validate_and_canonicalize_stored_policy(policy, STORED_POLICY_SOURCE_SPEC), + ) } pub(super) async fn validate_candidate_provider_attachments( @@ -2046,8 +2088,7 @@ pub(super) async fn provider_policy_composition_enabled(store: &Store) -> Result } fn provider_policy_composition_enabled_in(settings: &StoredSettings) -> Result { - Ok(decode_policy_from_global_settings(settings)?.is_none() - && bool_setting_enabled(settings, settings::PROVIDERS_V2_ENABLED_KEY)?) + Ok(decode_policy_from_global_settings(settings)?.is_none()) } async fn validate_provider_composition_for_existing_sandboxes( @@ -2114,6 +2155,15 @@ async fn validate_provider_composition_for_existing_sandboxes( Ok(()) } +pub async fn validate_provider_composition_startup_preflight( + state: &ServerState, +) -> Result<(), Status> { + if provider_policy_composition_enabled(state.store.as_ref()).await? { + validate_provider_composition_for_existing_sandboxes(state).await?; + } + Ok(()) +} + pub(super) async fn validate_candidate_sandbox_credential_policy( state: &ServerState, workspace: &str, @@ -2332,26 +2382,41 @@ pub(super) async fn handle_get_sandbox_config( .snapshot_catalog(state.store.as_ref(), &workspace) .await?; - // Try to get the latest policy from the policy history table. + let global_settings = load_global_settings(state.store.as_ref()).await?; + let global_policy = decode_policy_from_global_settings(&global_settings)?; + let mut global_policy_version: u32 = 0; + + // Try to get the latest policy from the policy history table. Under a + // global override, only the sandbox version metadata is observed; the + // dormant payload is neither decoded nor validated. let latest = state .store .get_latest_policy(&sandbox_id) .await .map_err(|e| Status::internal(format!("fetch policy history failed: {e}")))?; - let mut policy_source = PolicySource::Sandbox; - let (mut policy, mut version, mut policy_hash) = if let Some(record) = latest { - let decoded = ProtoSandboxPolicy::decode(record.policy_payload.as_slice()) - .map_err(|e| Status::internal(format!("decode policy failed: {e}")))?; + let (mut policy, version, mut policy_hash, policy_source) = if let Some(global_policy) = + global_policy + { + let version = latest + .as_ref() + .map(|record| u32::try_from(record.version).unwrap_or(0)) + .filter(|version| *version > 0) + .unwrap_or(1); + let hash = deterministic_policy_hash(&global_policy); + (Some(global_policy), version, hash, PolicySource::Global) + } else if let Some(record) = latest { + let (policy, hash) = canonical_policy_record_identity(&record)?; debug!( sandbox_id = %sandbox_id, version = record.version, "GetSandboxConfig served from policy history" ); ( - Some(decoded), + Some(policy), u32::try_from(record.version).unwrap_or(0), - record.policy_hash, + hash, + PolicySource::Sandbox, ) } else { // Lazy backfill: no policy history exists yet. @@ -2366,9 +2431,16 @@ pub(super) async fn handle_get_sandbox_config( sandbox_id = %sandbox_id, "GetSandboxConfig: no policy configured, returning empty response" ); - (None, 0, String::new()) + (None, 0, String::new(), PolicySource::Sandbox) } Some(spec_policy) => { + // Stored specs may predate the current schema. Validate before + // creating policy history so malformed state is never copied or + // marked loaded, and hash the canonical representation. + let spec_policy = validate_and_canonicalize_stored_policy( + spec_policy, + STORED_POLICY_SOURCE_SPEC, + )?; let hash = deterministic_policy_hash(&spec_policy); let payload = spec_policy.encode_to_vec(); let policy_id = uuid::Uuid::new_v4().to_string(); @@ -2400,7 +2472,7 @@ pub(super) async fn handle_get_sandbox_config( "GetSandboxConfig served from spec (backfilled version 1)" ); - (Some(spec_policy), 1, hash) + (Some(spec_policy), 1, hash, PolicySource::Sandbox) } } }; @@ -2408,8 +2480,6 @@ pub(super) async fn handle_get_sandbox_config( let global_settings = load_global_settings(state.store.as_ref()).await?; let sandbox_settings = load_sandbox_settings(state.store.as_ref(), &workspace, sandbox.object_name()).await?; - let providers_v2_enabled = - bool_setting_enabled(&global_settings, settings::PROVIDERS_V2_ENABLED_KEY)?; let mut provider_policy_context = provider_policy_context_with_catalog( state.store.as_ref(), &provider_profile_catalog, @@ -2418,22 +2488,13 @@ pub(super) async fn handle_get_sandbox_config( ) .await?; - let mut global_policy_version: u32 = 0; - - if let Some(global_policy) = decode_policy_from_global_settings(&global_settings)? { - policy = Some(global_policy.clone()); - policy_hash = deterministic_policy_hash(&global_policy); - policy_source = PolicySource::Global; - if version == 0 { - version = 1; - } - if let Ok(Some(global_rev)) = state + if matches!(policy_source, PolicySource::Global) + && let Ok(Some(global_rev)) = state .store .get_latest_policy(GLOBAL_POLICY_SANDBOX_ID) .await - { - global_policy_version = u32::try_from(global_rev.version).unwrap_or(0); - } + { + global_policy_version = u32::try_from(global_rev.version).unwrap_or(0); } if let Some(source_policy) = policy.as_mut() { @@ -2442,13 +2503,19 @@ pub(super) async fn handle_get_sandbox_config( clear_provider_credentialed_markers(source_policy); } - if providers_v2_enabled - && !matches!(policy_source, PolicySource::Global) + if !matches!(policy_source, PolicySource::Global) && let Some(source_policy) = policy.as_ref() && !provider_policy_context.layers.is_empty() { let effective_policy = compose_effective_policy(source_policy, &provider_policy_context.layers); + let effective_policy = + validate_and_canonicalize_policy(effective_policy).map_err(|error| { + Status::failed_precondition(format!( + "provider composition produced an invalid effective policy: {}", + error.message() + )) + })?; validate_policy_safety(&effective_policy).map_err(|error| { Status::failed_precondition(format!( "provider composition produced an invalid effective policy: {}", @@ -2684,7 +2751,7 @@ fn compute_provider_env_revision_from_records_and_policy_bindings( } fn hash_provider_refresh_states( - states: &[openshell_core::proto::StoredProviderCredentialRefreshState], + states: &[StoredProviderCredentialRefreshState], hasher: &mut Sha256, ) -> Result<(), Status> { let mut states = states.iter().collect::>(); @@ -2743,8 +2810,7 @@ fn hash_provider_profile_revision( profile_workspace: &str, hasher: &mut Sha256, ) { - let profile_id = normalize_provider_type(provider_type).unwrap_or(provider_type); - catalog.hash_type_profile_revision_for_scope(profile_id, profile_workspace, hasher); + catalog.hash_type_profile_revision_for_scope(provider_type, profile_workspace, hasher); } #[cfg(test)] @@ -2804,10 +2870,9 @@ async fn provider_policy_context_with_catalog( .ok_or_else(|| Status::failed_precondition(format!("provider '{name}' not found")))?; let provider_type = provider.r#type.trim(); - let profile_id = normalize_provider_type(provider_type).unwrap_or(provider_type); let Some(profile) = super::provider::get_provider_type_profile_for_scope( catalog, - profile_id, + provider_type, &provider.profile_workspace, ) else { warn!( @@ -2818,6 +2883,11 @@ async fn provider_policy_context_with_catalog( continue; }; + if !super::provider::provider_profile_endpoints_are_active(&profile, &provider) { + endpointless_provider_names.insert(name.clone()); + continue; + } + let rule_name = openshell_policy::provider_rule_name(provider.object_name()); let mut rule = profile.network_policy_rule(&rule_name); if rule.endpoints.is_empty() { @@ -3005,16 +3075,6 @@ fn report_uninspected_credentialed_endpoints(policy: &ProtoSandboxPolicy, sandbo } } -pub(super) fn bool_setting_enabled(settings: &StoredSettings, key: &str) -> Result { - match settings.settings.get(key) { - None => Ok(false), - Some(StoredSettingValue::Bool(value)) => Ok(*value), - Some(_) => Err(Status::internal(format!( - "setting '{key}' has invalid value type; expected bool" - ))), - } -} - pub(super) async fn handle_get_gateway_config( state: &Arc, _request: Request, @@ -3150,20 +3210,6 @@ pub(super) async fn handle_get_sandbox_provider_environment( // Update config handler (policy + settings mutations) // --------------------------------------------------------------------------- -fn validate_live_policy_update_support( - driver_kind: Option, - has_policy: bool, - has_merge_ops: bool, -) -> Result<(), Status> { - if (has_policy || has_merge_ops) && driver_kind == Some(openshell_core::ComputeDriverKind::Mxc) - { - return Err(Status::failed_precondition( - "live policy updates are not supported for MXC sandboxes; recreate the sandbox so the new policy is mapped before launch", - )); - } - Ok(()) -} - pub(super) async fn handle_update_config( state: &Arc, request: Request, @@ -3173,7 +3219,13 @@ pub(super) async fn handle_update_config( let update = request.get_ref(); let should_emit_policy_failure = should_emit_config_update_policy_telemetry(sandbox_caller) && (update.policy.is_some() || !update.merge_operations.is_empty()); - let result = handle_update_config_inner(state, request, &principal, sandbox_caller).await; + let result = Box::pin(handle_update_config_inner( + state, + request, + &principal, + sandbox_caller, + )) + .await; if result.is_err() && should_emit_policy_failure { emit_sandbox_policy_update_failure(); } @@ -3189,6 +3241,11 @@ async fn handle_update_config_inner( let req = request.into_inner(); validate_annotations(&req.annotations, "annotations")?; let workspace = if req.global { + if req.workspace_scope.is_some() { + return Err(Status::invalid_argument( + "workspace_scope must be omitted when global is true", + )); + } require_platform_admin(&state.admin_role, principal)?; String::new() } else { @@ -3197,15 +3254,16 @@ async fn handle_update_config_inner( } else { MinWorkspaceRole::Admin }; + let workspace = selected_workspace_name(req.workspace_scope.as_ref())?; authorize_sandbox_workspace( &state.store, &state.admin_role, principal, - &req.workspace, + workspace, min_role, ) .await?; - super::workspace::resolve_workspace(state.store.as_ref(), &req.workspace) + super::workspace::resolve_workspace(state.store.as_ref(), workspace) .await? .name }; @@ -3238,7 +3296,6 @@ async fn handle_update_config_inner( "one of policy, setting_key, or merge_operations must be provided", )); } - validate_live_policy_update_support(state.compute.driver_kind(), has_policy, has_merge_ops)?; if req.global { if !req.annotations.is_empty() { return Err(Status::invalid_argument( @@ -3264,6 +3321,7 @@ async fn handle_update_config_inner( })?; clear_provider_credentialed_markers(&mut new_policy); validate_no_reserved_provider_policy_keys(&new_policy)?; + new_policy = validate_and_canonicalize_policy(new_policy)?; validate_policy_safety(&new_policy)?; crate::middleware::validate_policy(state.middleware_registry.as_ref(), &new_policy) .await?; @@ -3284,7 +3342,7 @@ async fn handle_update_config_inner( .map_err(|e| Status::internal(format!("fetch latest global policy failed: {e}")))?; if let Some(ref current) = latest - && current.policy_hash == hash + && canonical_policy_record_matches_for_deduplication(current, &hash) && current.status == "loaded" { let mut global_settings = load_global_settings(state.store.as_ref()).await?; @@ -3656,14 +3714,16 @@ async fn handle_update_config_inner( validate_no_reserved_provider_policy_keys(&new_policy)?; } - let backfill_policy = if let Some(baseline_policy) = spec.policy.as_ref() { + let should_backfill_policy = if let Some(baseline_policy) = spec.policy.as_ref() { let comparable_baseline = baseline_policy.clone(); validate_static_fields_unchanged(&comparable_baseline, &new_policy)?; - None + false } else { - Some(new_policy.clone()) + true }; + new_policy = validate_and_canonicalize_policy(new_policy)?; + let backfill_policy = should_backfill_policy.then(|| new_policy.clone()); validate_policy_safety(&new_policy)?; crate::middleware::validate_policy(state.middleware_registry.as_ref(), &new_policy).await?; let provider_layers = @@ -3717,7 +3777,7 @@ async fn handle_update_config_inner( .map_err(|e| Status::internal(format!("fetch latest policy failed: {e}")))?; if let Some(ref current) = latest - && current.policy_hash == hash + && canonical_policy_record_matches_for_deduplication(current, &hash) && current.provenance == req.annotations { response_annotations = persist_existing_policy_projection( @@ -3807,7 +3867,7 @@ async fn handle_update_config_inner( let hash = deterministic_policy_hash(&new_policy); if let Some(ref current) = latest - && current.policy_hash == hash + && canonical_policy_record_matches_for_deduplication(current, &hash) { return Ok(Response::new(UpdateConfigResponse { version: u32::try_from(current.version).unwrap_or(0), @@ -3869,14 +3929,19 @@ pub(super) async fn handle_get_sandbox_policy_status( let principal = super::extract_principal(&request)?; let req = request.into_inner(); let workspace = if req.global { + if req.workspace_scope.is_some() { + return Err(Status::invalid_argument( + "workspace_scope must be omitted when global is true", + )); + } require_platform_admin(&state.admin_role, &principal)?; String::new() } else { - let authz = authorize_workspace( + let authz = authorize_workspace_selector( &state.store, &state.admin_role, &principal, - &req.workspace, + req.workspace_scope.as_ref(), MinWorkspaceRole::User, ) .await?; @@ -3925,7 +3990,7 @@ pub(super) async fn handle_get_sandbox_policy_status( let record = record.ok_or_else(|| Status::not_found(not_found_msg))?; Ok(Response::new(GetSandboxPolicyStatusResponse { - revision: Some(policy_record_to_revision(&record, true)), + revision: Some(policy_record_to_revision(&record, true)?), active_version, })) } @@ -3937,14 +4002,19 @@ pub(super) async fn handle_list_sandbox_policies( let principal = super::extract_principal(&request)?; let req = request.into_inner(); let workspace = if req.global { + if req.workspace_scope.is_some() { + return Err(Status::invalid_argument( + "workspace_scope must be omitted when global is true", + )); + } require_platform_admin(&state.admin_role, &principal)?; String::new() } else { - let authz = authorize_workspace( + let authz = authorize_workspace_selector( &state.store, &state.admin_role, &principal, - &req.workspace, + req.workspace_scope.as_ref(), MinWorkspaceRole::User, ) .await?; @@ -3978,7 +4048,7 @@ pub(super) async fn handle_list_sandbox_policies( let revisions = records .iter() .map(|r| policy_record_to_revision(r, false)) - .collect(); + .collect::, Status>>()?; Ok(Response::new(ListSandboxPoliciesResponse { revisions })) } @@ -4076,8 +4146,22 @@ pub(super) async fn handle_get_sandbox_logs( if req.sandbox_id.is_empty() { return Err(Status::invalid_argument("sandbox_id is required")); } - let _sandbox = + let authz = authorize_workspace_selector( + &state.store, + &state.admin_role, + &principal, + req.workspace_scope.as_ref(), + MinWorkspaceRole::User, + ) + .await?; + let workspace = super::workspace::resolve_workspace(state.store.as_ref(), &authz.workspace) + .await? + .name; + let sandbox = super::sandbox::fetch_and_authorize_sandbox(state, &principal, &req.sandbox_id).await?; + if sandbox.object_workspace() != workspace { + return Err(Status::not_found("sandbox not found")); + } let lines = if req.lines == 0 { 2000 } else { req.lines }; let tail = state.tracing_log_bus.tail(&req.sandbox_id, lines as usize); @@ -4336,6 +4420,18 @@ pub(super) async fn handle_submit_policy_analysis( } let rule_ref = chunk.proposed_rule.as_ref().expect("checked above"); + if req.analysis_mode == "agent_authored" + && let Some(reason) = rule_ref.endpoints.iter().find_map(|endpoint| { + openshell_policy::agent_authored_transport_rejection( + &endpoint.protocol, + &endpoint.tls, + ) + }) + { + rejected += 1; + rejection_reasons.push(format!("chunk '{}': {reason}", chunk.rule_name)); + continue; + } let incoming_observation_key = rule_ref.endpoints.first().and_then(|endpoint| { rule_ref.binaries.first().map(|binary| { ( @@ -4543,7 +4639,7 @@ pub(super) async fn handle_submit_policy_analysis( // string means findings or infrastructure error, both of which // require human attention. if auto_approve_enabled - && let Err(err) = auto_approve_chunk( + && let Err(err) = Box::pin(auto_approve_chunk( state, &effective_id, AutoApproveChunkContext { @@ -4552,7 +4648,7 @@ pub(super) async fn handle_submit_policy_analysis( source: &req.analysis_mode, resolved_from, }, - ) + )) .await { persist_pending_application_error(state, &effective_id, &err).await; @@ -4596,15 +4692,16 @@ pub(super) async fn handle_get_draft_policy( .cloned() .ok_or_else(|| Status::unauthenticated("missing principal"))?; let req = request.into_inner(); + let workspace_name = selected_workspace_name(req.workspace_scope.as_ref())?; authorize_sandbox_workspace( &state.store, &state.admin_role, &principal, - &req.workspace, + workspace_name, MinWorkspaceRole::User, ) .await?; - let workspace = super::workspace::resolve_workspace(state.store.as_ref(), &req.workspace) + let workspace = super::workspace::resolve_workspace(state.store.as_ref(), workspace_name) .await? .name; if req.name.is_empty() { @@ -4677,11 +4774,11 @@ async fn handle_approve_draft_chunk_inner( ) -> Result, Status> { let principal = super::extract_principal(&request)?; let req = request.into_inner(); - let authz = authorize_workspace( + let authz = authorize_workspace_selector( &state.store, &state.admin_role, &principal, - &req.workspace, + req.workspace_scope.as_ref(), MinWorkspaceRole::Admin, ) .await?; @@ -4832,11 +4929,11 @@ async fn handle_reject_draft_chunk_inner( ) -> Result, Status> { let principal = super::extract_principal(&request)?; let req = request.into_inner(); - let authz = authorize_workspace( + let authz = authorize_workspace_selector( &state.store, &state.admin_role, &principal, - &req.workspace, + req.workspace_scope.as_ref(), MinWorkspaceRole::Admin, ) .await?; @@ -4942,11 +5039,11 @@ async fn handle_approve_all_draft_chunks_inner( ) -> Result, Status> { let principal = super::extract_principal(&request)?; let req = request.into_inner(); - let authz = authorize_workspace( + let authz = authorize_workspace_selector( &state.store, &state.admin_role, &principal, - &req.workspace, + req.workspace_scope.as_ref(), MinWorkspaceRole::Admin, ) .await?; @@ -5250,11 +5347,11 @@ pub(super) async fn handle_edit_draft_chunk( ) -> Result, Status> { let principal = super::extract_principal(&request)?; let req = request.into_inner(); - let authz = authorize_workspace( + let authz = authorize_workspace_selector( &state.store, &state.admin_role, &principal, - &req.workspace, + req.workspace_scope.as_ref(), MinWorkspaceRole::Admin, ) .await?; @@ -5331,11 +5428,11 @@ async fn handle_undo_draft_chunk_inner( ) -> Result, Status> { let principal = super::extract_principal(&request)?; let req = request.into_inner(); - let authz = authorize_workspace( + let authz = authorize_workspace_selector( &state.store, &state.admin_role, &principal, - &req.workspace, + req.workspace_scope.as_ref(), MinWorkspaceRole::Admin, ) .await?; @@ -5428,11 +5525,11 @@ pub(super) async fn handle_clear_draft_chunks( ) -> Result, Status> { let principal = super::extract_principal(&request)?; let req = request.into_inner(); - let authz = authorize_workspace( + let authz = authorize_workspace_selector( &state.store, &state.admin_role, &principal, - &req.workspace, + req.workspace_scope.as_ref(), MinWorkspaceRole::Admin, ) .await?; @@ -5476,11 +5573,11 @@ pub(super) async fn handle_get_draft_history( ) -> Result, Status> { let principal = super::extract_principal(&request)?; let req = request.into_inner(); - let authz = authorize_workspace( + let authz = authorize_workspace_selector( &state.store, &state.admin_role, &principal, - &req.workspace, + req.workspace_scope.as_ref(), MinWorkspaceRole::User, ) .await?; @@ -5741,6 +5838,46 @@ fn deterministic_policy_hash(policy: &ProtoSandboxPolicy) -> String { hex::encode(Sha256::digest(canonical_policy_bytes(policy))) } +/// Rebuilds a policy revision's identity from its checked canonical payload. +/// +/// `PolicyRecord::policy_hash` is persisted metadata and cannot prove what the +/// payload contains. Decode and validate every record before using its identity +/// so legacy encodings deduplicate semantically and damaged rows fail closed. +fn canonical_policy_record_identity( + record: &PolicyRecord, +) -> Result<(ProtoSandboxPolicy, String), Status> { + let decoded = ProtoSandboxPolicy::decode(record.policy_payload.as_slice()) + .map_err(|error| Status::internal(format!("decode policy revision failed: {error}")))?; + let policy = validate_and_canonicalize_stored_policy(decoded, STORED_POLICY_SOURCE_HISTORY)?; + let hash = deterministic_policy_hash(&policy); + Ok((policy, hash)) +} + +/// Compare a stored revision during no-op detection without blocking repair. +/// +/// Invalid durable state remains unusable everywhere that loads or serves a +/// policy. A full, already-validated replacement is different: treating an +/// unreadable current row as non-matching lets the write path append a good +/// revision instead of making the corrupt or legacy row permanently terminal. +fn canonical_policy_record_matches_for_deduplication( + record: &PolicyRecord, + expected_hash: &str, +) -> bool { + match canonical_policy_record_identity(record) { + Ok((_, hash)) => hash == expected_hash, + Err(error) => { + warn!( + policy_id = %record.id, + sandbox_id = %record.sandbox_id, + version = record.version, + error = %error, + "Invalid stored policy revision cannot satisfy deduplication; a valid replacement may proceed" + ); + false + } + } +} + /// Compute a fingerprint for the effective sandbox configuration. fn compute_config_revision_with_validation_mode( policy: Option<&ProtoSandboxPolicy>, @@ -5860,8 +5997,11 @@ fn draft_chunk_record_to_proto(record: &DraftChunkRecord) -> Result SandboxPolicyRevision { - let status = match record.status.as_str() { +fn policy_record_to_revision( + record: &PolicyRecord, + include_policy: bool, +) -> Result { + let stored_status = match record.status.as_str() { "pending" => PolicyStatus::Pending, "loaded" => PolicyStatus::Loaded, "failed" => PolicyStatus::Failed, @@ -5869,21 +6009,49 @@ fn policy_record_to_revision(record: &PolicyRecord, include_policy: bool) -> San _ => PolicyStatus::Unspecified, }; - let policy = if include_policy { - ProtoSandboxPolicy::decode(record.policy_payload.as_slice()).ok() - } else { - None - }; - - SandboxPolicyRevision { - version: u32::try_from(record.version).unwrap_or(0), - policy_hash: record.policy_hash.clone(), - status: status.into(), - load_error: record.load_error.clone().unwrap_or_default(), - created_at_ms: record.created_at_ms, - loaded_at_ms: record.loaded_at_ms.unwrap_or(0), - policy, - provenance: record.provenance.clone(), + match canonical_policy_record_identity(record) { + Ok((policy, policy_hash)) => Ok(SandboxPolicyRevision { + version: u32::try_from(record.version).unwrap_or(0), + policy_hash, + status: stored_status.into(), + load_error: record.load_error.clone().unwrap_or_default(), + created_at_ms: record.created_at_ms, + loaded_at_ms: record.loaded_at_ms.unwrap_or(0), + policy: include_policy.then_some(policy), + provenance: record.provenance.clone(), + }), + Err(error) if !include_policy => { + // History listing is a recovery surface, not an enforcement path. + // Preserve row metadata so one legacy or damaged payload cannot + // hide every usable revision in the page, but blank the untrusted + // hash and mark the projection failed. Detail and runtime callers + // still receive the hard error through the branch below. + let identity_error = format!( + "policy revision is invalid under the current schema: {}", + error.message() + ); + let load_error = record.load_error.as_deref().map_or_else( + || identity_error.clone(), + |stored_error| { + if stored_error.is_empty() { + identity_error.clone() + } else { + format!("{stored_error}; {identity_error}") + } + }, + ); + Ok(SandboxPolicyRevision { + version: u32::try_from(record.version).unwrap_or(0), + policy_hash: String::new(), + status: PolicyStatus::Failed.into(), + load_error, + created_at_ms: record.created_at_ms, + loaded_at_ms: record.loaded_at_ms.unwrap_or(0), + policy: None, + provenance: record.provenance.clone(), + }) + } + Err(error) => Err(error), } } @@ -6213,13 +6381,15 @@ fn validate_merge_operations_for_server(operations: &[PolicyMergeOp]) -> Result< fn map_policy_merge_error(error: openshell_policy::PolicyMergeError) -> Status { match error { - openshell_policy::PolicyMergeError::MissingRuleNameForAddRule + openshell_policy::PolicyMergeError::InvalidOperationPolicy { .. } + | openshell_policy::PolicyMergeError::MissingRuleNameForAddRule | openshell_policy::PolicyMergeError::EmptyAddRuleEndpoints { .. } | openshell_policy::PolicyMergeError::InvalidEndpointReference { .. } | openshell_policy::PolicyMergeError::UnsupportedAccessPreset { .. } => { Status::invalid_argument(error.to_string()) } - openshell_policy::PolicyMergeError::McpContractConflict { .. } + openshell_policy::PolicyMergeError::InvalidInputPolicy { .. } + | openshell_policy::PolicyMergeError::McpContractConflict { .. } | openshell_policy::PolicyMergeError::NewBinaryWouldInheritAuthorization { .. } | openshell_policy::PolicyMergeError::ExistingBinariesWouldInheritAuthorization { .. @@ -6234,6 +6404,9 @@ fn map_policy_merge_error(error: openshell_policy::PolicyMergeError) -> Status { | openshell_policy::PolicyMergeError::EndpointHasNoAllowBase { .. } => { Status::failed_precondition(error.to_string()) } + openshell_policy::PolicyMergeError::InvalidMergedPolicy { .. } => { + Status::internal(error.to_string()) + } } } @@ -6403,11 +6576,11 @@ async fn apply_merge_operations_with_retry( .await .map_err(|e| Status::internal(format!("fetch latest policy failed: {e}")))?; - let current_policy = if let Some(ref record) = latest { - ProtoSandboxPolicy::decode(record.policy_payload.as_slice()) - .map_err(|e| Status::internal(format!("decode current policy failed: {e}")))? + let (current_policy, current_hash) = if let Some(ref record) = latest { + let (policy, hash) = canonical_policy_record_identity(record)?; + (policy, Some(hash)) } else { - baseline_policy.cloned().unwrap_or_default() + (baseline_policy.cloned().unwrap_or_default(), None) }; if let Some(expected_hash) = expected_current_effective_hash { @@ -6451,7 +6624,7 @@ async fn apply_merge_operations_with_retry( } if let Some(ref current) = latest - && current.policy_hash == hash + && current_hash.as_deref() == Some(hash.as_str()) && atomic_context.is_none_or(|context| current.provenance == *context.provenance) { return Ok((current.version, hash, None)); @@ -6721,30 +6894,6 @@ pub(super) async fn load_global_settings(store: &Store) -> Result Result { - let global_settings = load_global_settings(store).await?; - bool_setting_enabled(&global_settings, key) -} - -/// Test helper: set a boolean global setting, loading current settings first so -/// the CAS write succeeds whether the record already exists or not. Available to -/// sibling test modules without exposing the private `StoredSettings` type. -#[cfg(test)] -pub async fn set_global_bool_setting_for_test( - store: &Store, - key: &str, - value: bool, -) -> Result<(), Status> { - let mut settings = load_global_settings(store).await?; - settings - .settings - .insert(key.to_string(), StoredSettingValue::Bool(value)); - save_global_settings(store, &settings).await -} - pub(super) async fn save_global_settings( store: &Store, settings: &StoredSettings, @@ -6863,7 +7012,23 @@ fn decode_policy_from_global_settings( .map_err(|e| Status::internal(format!("global policy decode failed: {e}")))?; let policy = ProtoSandboxPolicy::decode(raw.as_slice()) .map_err(|e| Status::internal(format!("global policy protobuf decode failed: {e}")))?; - Ok(Some(policy)) + validate_and_canonicalize_stored_policy(policy, STORED_POLICY_SOURCE_GLOBAL).map(Some) +} + +/// Validate a decoded stored policy before it is trusted, hashed, or copied. +/// +/// Stored rows may have been written by an older schema or damaged outside the +/// normal request path. Treat invalid durable state as a failed precondition and +/// return the canonical value so callers cannot accidentally reuse raw bytes. +fn validate_and_canonicalize_stored_policy( + policy: ProtoSandboxPolicy, + source: &'static str, +) -> Result { + openshell_policy::validate_and_canonicalize_sandbox_policy(policy).map_err(|error| { + Status::failed_precondition(format!( + "stored policy source '{source}' is invalid: {error}" + )) + }) } fn merge_effective_settings( @@ -6950,27 +7115,6 @@ mod tests { use std::sync::atomic::{AtomicUsize, Ordering}; use tonic::Code; - #[test] - fn mxc_rejects_sandbox_policy_replacement_and_merge_updates() { - for (has_policy, has_merge_ops) in [(true, false), (false, true)] { - let error = validate_live_policy_update_support( - Some(openshell_core::ComputeDriverKind::Mxc), - has_policy, - has_merge_ops, - ) - .expect_err("MXC must reject policy mutations after launch"); - assert_eq!(error.code(), Code::FailedPrecondition); - } - - let error = validate_live_policy_update_support( - Some(openshell_core::ComputeDriverKind::Mxc), - true, - false, - ) - .expect_err("global policy replacement also changes desired state for live MXC sandboxes"); - assert_eq!(error.code(), Code::FailedPrecondition); - } - /// Wrap a request with a user `Principal` so handler scope guards treat /// the test caller as a CLI user. Most handler tests exercise /// user-facing behavior and should not trip sandbox equality checks. @@ -7016,6 +7160,985 @@ mod tests { }) } + fn mcp_policy_with_versions(versions: &[&str]) -> ProtoSandboxPolicy { + let mut policy = openshell_policy::restrictive_default_policy(); + policy.network_policies.insert( + "mcp".to_string(), + NetworkPolicyRule { + name: "mcp".to_string(), + endpoints: vec![NetworkEndpoint { + host: "mcp.example.com".to_string(), + port: 443, + protocol: "mcp".to_string(), + rules: vec![L7Rule { + allow: Some(openshell_core::proto::L7Allow { + method: "tools/list".to_string(), + ..Default::default() + }), + }], + mcp: Some(openshell_core::proto::McpOptions { + versions: versions.iter().map(ToString::to_string).collect(), + ..Default::default() + }), + ..Default::default() + }], + ..Default::default() + }, + ); + policy + } + + fn mcp_policy_without_options() -> ProtoSandboxPolicy { + let mut policy = mcp_policy_with_versions(&[]); + policy + .network_policies + .get_mut("mcp") + .expect("MCP test rule") + .endpoints + .first_mut() + .expect("MCP test endpoint") + .mcp = None; + policy + } + + fn legacy_non_mcp_policy_with_mcp_options() -> ProtoSandboxPolicy { + let mut policy = mcp_policy_with_versions(&[]); + let endpoint = policy + .network_policies + .get_mut("mcp") + .expect("MCP test rule") + .endpoints + .first_mut() + .expect("MCP test endpoint"); + endpoint.protocol = "rest".to_string(); + endpoint.mcp = Some(openshell_core::proto::McpOptions { + strict_tool_names: Some(true), + ..Default::default() + }); + policy + } + + fn mcp_versions(policy: &ProtoSandboxPolicy) -> &[String] { + policy + .network_policies + .get("mcp") + .and_then(|rule| rule.endpoints.first()) + .and_then(|endpoint| endpoint.mcp.as_ref()) + .map(|mcp| mcp.versions.as_slice()) + .expect("canonical MCP test endpoint options") + } + + fn defaulted_mcp_policy_cases() -> [(&'static str, ProtoSandboxPolicy); 3] { + [ + ("omitted-options", mcp_policy_without_options()), + ("empty-versions", mcp_policy_with_versions(&[])), + ( + "explicit-default", + mcp_policy_with_versions(&["2025-11-25"]), + ), + ] + } + + /// Install a global policy through the normal write path and return the + /// canonical value that subsequent read paths must serve. + async fn install_test_global_policy(state: &Arc) -> ProtoSandboxPolicy { + handle_update_config( + state, + with_user(Request::new(UpdateConfigRequest { + global: true, + policy: Some(mcp_policy_with_versions(&[])), + ..Default::default() + })), + ) + .await + .expect("test global policy update must succeed"); + + let settings = load_global_settings(state.store.as_ref()) + .await + .expect("test global settings lookup"); + decode_policy_from_global_settings(&settings) + .expect("test global policy must decode") + .expect("test global policy must be present") + } + + #[tokio::test] + async fn get_sandbox_config_rejects_invalid_spec_policy_before_history_backfill() { + let state = test_server_state().await; + let cases = [ + ( + "duplicate", + mcp_policy_with_versions(&["2025-11-25", "2025-11-25"]), + ), + ("unsupported", mcp_policy_with_versions(&["latest"])), + ]; + + for (case, policy) in cases { + let sandbox_id = format!("stored-invalid-{case}"); + state + .store + .put_message(&test_sandbox( + &sandbox_id, + &format!("stored-invalid-{case}"), + policy, + Vec::new(), + )) + .await + .expect("store legacy sandbox spec"); + + let error = handle_get_sandbox_config( + &state, + with_sandbox( + Request::new(GetSandboxConfigRequest { + sandbox_id: sandbox_id.clone(), + }), + &sandbox_id, + ), + ) + .await + .expect_err("invalid stored spec must fail before history backfill"); + + assert_eq!(error.code(), Code::FailedPrecondition, "{case}"); + assert!( + error.message().contains(STORED_POLICY_SOURCE_SPEC), + "{case}" + ); + assert!( + state + .store + .get_latest_policy(&sandbox_id) + .await + .expect("policy history lookup") + .is_none(), + "{case} must not create policy history" + ); + } + } + + #[tokio::test] + async fn get_sandbox_config_backfills_canonical_spec_policy_bytes_and_hash() { + let state = test_server_state().await; + let sandbox_id = "stored-canonical-backfill"; + let raw = mcp_policy_with_versions(&["2025-11-25", "2025-03-26", "2025-06-18"]); + let canonical = validate_and_canonicalize_policy(raw.clone()) + .expect("supported stored MCP policy must canonicalize"); + state + .store + .put_message(&test_sandbox( + sandbox_id, + "stored-canonical-backfill", + raw, + Vec::new(), + )) + .await + .expect("store legacy sandbox spec"); + + let response = handle_get_sandbox_config( + &state, + with_sandbox( + Request::new(GetSandboxConfigRequest { + sandbox_id: sandbox_id.to_string(), + }), + sandbox_id, + ), + ) + .await + .expect("valid stored spec must backfill") + .into_inner(); + + let canonical_hash = deterministic_policy_hash(&canonical); + assert_eq!(response.policy.as_ref(), Some(&canonical)); + assert_eq!(response.policy_hash, canonical_hash); + assert_eq!(response.version, 1); + + let persisted = state + .store + .get_latest_policy(sandbox_id) + .await + .expect("policy history lookup") + .expect("canonical history revision"); + assert_eq!(persisted.policy_payload, canonical.encode_to_vec()); + assert_eq!(persisted.policy_hash, canonical_hash); + assert_eq!(persisted.status, "loaded"); + } + + #[tokio::test] + async fn get_sandbox_config_backfills_defaulted_mcp_policy_as_canonical_bytes_and_hash() { + let state = test_server_state().await; + let canonical = validate_and_canonicalize_policy(mcp_policy_with_versions(&["2025-11-25"])) + .expect("explicit default MCP policy must canonicalize"); + assert_eq!(mcp_versions(&canonical), &["2025-11-25".to_string()]); + let canonical_payload = canonical.encode_to_vec(); + let canonical_hash = deterministic_policy_hash(&canonical); + let cases = [ + ("omitted-options", mcp_policy_without_options()), + ("empty-versions", mcp_policy_with_versions(&[])), + ( + "explicit-default", + mcp_policy_with_versions(&["2025-11-25"]), + ), + ]; + + for (case, raw) in cases { + let sandbox_id = format!("stored-default-{case}"); + state + .store + .put_message(&test_sandbox(&sandbox_id, &sandbox_id, raw, Vec::new())) + .await + .expect("store legacy sandbox spec"); + + let response = handle_get_sandbox_config( + &state, + with_sandbox( + Request::new(GetSandboxConfigRequest { + sandbox_id: sandbox_id.clone(), + }), + &sandbox_id, + ), + ) + .await + .expect("defaulted stored spec must backfill") + .into_inner(); + + assert_eq!(response.policy.as_ref(), Some(&canonical), "{case}"); + assert_eq!(response.policy_hash, canonical_hash, "{case}"); + assert_eq!(response.version, 1, "{case}"); + assert_eq!( + mcp_versions(response.policy.as_ref().expect("backfilled policy")), + &["2025-11-25".to_string()], + "{case}" + ); + + let persisted = state + .store + .get_latest_policy(&sandbox_id) + .await + .expect("policy history lookup") + .expect("canonical history revision"); + assert_eq!(persisted.policy_payload, canonical_payload, "{case}"); + assert_eq!(persisted.policy_hash, canonical_hash, "{case}"); + assert_eq!(persisted.status, "loaded", "{case}"); + } + } + + #[tokio::test] + async fn legacy_defaulted_mcp_policy_canonicalizes_for_current_base_and_history_export() { + let store = test_store().await; + let canonical = validate_and_canonicalize_policy(mcp_policy_with_versions(&["2025-11-25"])) + .expect("explicit default MCP policy must canonicalize"); + let canonical_hash = deterministic_policy_hash(&canonical); + let cases = [ + ("omitted-options", mcp_policy_without_options()), + ("empty-versions", mcp_policy_with_versions(&[])), + ( + "explicit-default", + mcp_policy_with_versions(&["2025-11-25"]), + ), + ]; + + for (case, raw) in cases { + let sandbox_id = format!("legacy-default-history-{case}"); + let sandbox = test_sandbox(&sandbox_id, &sandbox_id, raw.clone(), Vec::new()); + + let spec_base = current_base_policy_for_sandbox(&store, &sandbox) + .await + .expect("legacy spec policy must canonicalize"); + assert_eq!(spec_base, canonical, "{case}"); + assert_eq!( + mcp_versions(&spec_base), + &["2025-11-25".to_string()], + "{case}" + ); + + store + .put_policy_revision( + &format!("legacy-default-history-revision-{case}"), + &sandbox_id, + "default", + 1, + &raw.encode_to_vec(), + "legacy-uncanonicalized-hash", + ) + .await + .expect("store legacy policy history"); + + let history_base = current_base_policy_for_sandbox(&store, &sandbox) + .await + .expect("legacy history policy must canonicalize"); + assert_eq!(history_base, canonical, "{case}"); + + let record = store + .get_latest_policy(&sandbox_id) + .await + .expect("policy history lookup") + .expect("legacy policy history"); + let revision = policy_record_to_revision(&record, true) + .expect("legacy history export must canonicalize"); + assert_eq!(revision.policy_hash, canonical_hash, "{case}"); + let exported = revision.policy.expect("exported history policy"); + assert_eq!(exported, canonical, "{case}"); + assert_eq!( + mcp_versions(&exported), + &["2025-11-25".to_string()], + "{case}" + ); + + let listed = policy_record_to_revision(&record, false) + .expect("legacy history list projection must canonicalize"); + assert_eq!(listed.policy_hash, canonical_hash, "{case}"); + assert!(listed.policy.is_none(), "{case}"); + } + } + + #[tokio::test] + async fn invalid_legacy_latest_history_fails_closed_but_remains_listable() { + let state = test_server_state().await; + let sandbox_id = "stored-invalid-history"; + let valid = validate_and_canonicalize_policy(mcp_policy_with_versions(&[])) + .expect("valid history policy must canonicalize"); + let valid_hash = deterministic_policy_hash(&valid); + state + .store + .put_message(&test_sandbox( + sandbox_id, + "stored-invalid-history", + valid.clone(), + Vec::new(), + )) + .await + .expect("store sandbox"); + state + .store + .put_policy_revision( + "stored-valid-history-revision", + sandbox_id, + "default", + 1, + &valid.encode_to_vec(), + &valid_hash, + ) + .await + .expect("store valid history"); + let invalid = legacy_non_mcp_policy_with_mcp_options(); + state + .store + .put_policy_revision( + "stored-invalid-history-revision", + sandbox_id, + "default", + 2, + &invalid.encode_to_vec(), + "uncanonicalized-hash", + ) + .await + .expect("store legacy invalid history"); + + let error = handle_get_sandbox_config( + &state, + with_sandbox( + Request::new(GetSandboxConfigRequest { + sandbox_id: sandbox_id.to_string(), + }), + sandbox_id, + ), + ) + .await + .expect_err("invalid latest history must fail closed"); + + assert_eq!(error.code(), Code::FailedPrecondition); + assert!(error.message().contains(STORED_POLICY_SOURCE_HISTORY)); + + let record = state + .store + .get_latest_policy(sandbox_id) + .await + .expect("policy history lookup") + .expect("invalid policy history"); + let listed_invalid = policy_record_to_revision(&record, false) + .expect("list projection must preserve invalid legacy history metadata"); + assert_eq!(listed_invalid.version, 2); + assert_eq!(listed_invalid.status, PolicyStatus::Failed as i32); + assert!(listed_invalid.policy_hash.is_empty()); + assert!(listed_invalid.policy.is_none()); + assert!( + listed_invalid + .load_error + .contains(STORED_POLICY_SOURCE_HISTORY) + ); + + let detail_error = handle_get_sandbox_policy_status( + &state, + with_user(Request::new(GetSandboxPolicyStatusRequest { + name: "stored-invalid-history".to_string(), + version: 2, + workspace_scope: Some(openshell_core::proto::workspace_selector("default")), + ..Default::default() + })), + ) + .await + .expect_err("invalid history detail must remain fail-closed"); + assert_eq!(detail_error.code(), Code::FailedPrecondition); + assert!( + detail_error + .message() + .contains(STORED_POLICY_SOURCE_HISTORY) + ); + + let listed = handle_list_sandbox_policies( + &state, + with_user(Request::new(ListSandboxPoliciesRequest { + name: "stored-invalid-history".to_string(), + limit: 10, + workspace_scope: Some(openshell_core::proto::workspace_selector("default")), + ..Default::default() + })), + ) + .await + .expect("one invalid revision must not hide the rest of the history") + .into_inner(); + + assert_eq!(listed.revisions.len(), 2); + assert_eq!(listed.revisions[0], listed_invalid); + assert_eq!(listed.revisions[1].version, 1); + assert_eq!(listed.revisions[1].policy_hash, valid_hash); + assert_eq!(listed.revisions[1].status, PolicyStatus::Pending as i32); + assert!(listed.revisions[1].load_error.is_empty()); + assert!(listed.revisions[1].policy.is_none()); + } + + #[tokio::test] + async fn valid_global_policy_overrides_invalid_legacy_history() { + let state = test_server_state().await; + let global_policy = install_test_global_policy(&state).await; + let sandbox_id = "global-overrides-invalid-history"; + let sandbox = test_sandbox( + sandbox_id, + sandbox_id, + ProtoSandboxPolicy::default(), + Vec::new(), + ); + state + .store + .put_message(&sandbox) + .await + .expect("store sandbox"); + + let invalid = legacy_non_mcp_policy_with_mcp_options(); + state + .store + .put_policy_revision( + "global-overrides-invalid-history-revision", + sandbox_id, + "default", + 2, + &invalid.encode_to_vec(), + "untrusted-legacy-hash", + ) + .await + .expect("store invalid legacy policy history"); + + let response = handle_get_sandbox_config( + &state, + with_sandbox( + Request::new(GetSandboxConfigRequest { + sandbox_id: sandbox_id.to_string(), + }), + sandbox_id, + ), + ) + .await + .expect("valid global policy must override invalid local history") + .into_inner(); + + assert_eq!(response.policy.as_ref(), Some(&global_policy)); + assert_eq!( + response.policy_hash, + deterministic_policy_hash(&global_policy) + ); + assert_eq!(response.policy_source, PolicySource::Global as i32); + assert_eq!(response.version, 2); + assert_eq!(response.global_policy_version, 1); + + let catalog = state + .provider_profile_sources + .snapshot_catalog(state.store.as_ref(), "default") + .await + .expect("provider profile catalog"); + let effective = current_effective_policy_for_sandbox( + state.as_ref(), + &catalog, + "default", + &sandbox, + sandbox_id, + ) + .await + .expect("effective-policy lookup must bypass invalid local history"); + assert_eq!(effective, global_policy); + } + + #[tokio::test] + async fn valid_global_policy_overrides_invalid_legacy_spec_without_backfill() { + let state = test_server_state().await; + let global_policy = install_test_global_policy(&state).await; + let sandbox_id = "global-overrides-invalid-spec"; + let sandbox = test_sandbox( + sandbox_id, + sandbox_id, + legacy_non_mcp_policy_with_mcp_options(), + Vec::new(), + ); + state + .store + .put_message(&sandbox) + .await + .expect("store sandbox with invalid legacy spec"); + + let response = handle_get_sandbox_config( + &state, + with_sandbox( + Request::new(GetSandboxConfigRequest { + sandbox_id: sandbox_id.to_string(), + }), + sandbox_id, + ), + ) + .await + .expect("valid global policy must override invalid local spec") + .into_inner(); + + assert_eq!(response.policy.as_ref(), Some(&global_policy)); + assert_eq!( + response.policy_hash, + deterministic_policy_hash(&global_policy) + ); + assert_eq!(response.policy_source, PolicySource::Global as i32); + assert_eq!(response.version, 1); + assert_eq!(response.global_policy_version, 1); + assert!( + state + .store + .get_latest_policy(sandbox_id) + .await + .expect("policy history lookup") + .is_none(), + "global override must not backfill invalid dormant spec state" + ); + + let catalog = state + .provider_profile_sources + .snapshot_catalog(state.store.as_ref(), "default") + .await + .expect("provider profile catalog"); + let effective = current_effective_policy_for_sandbox( + state.as_ref(), + &catalog, + "default", + &sandbox, + sandbox_id, + ) + .await + .expect("effective-policy lookup must bypass invalid local spec"); + assert_eq!(effective, global_policy); + assert!( + state + .store + .get_latest_policy(sandbox_id) + .await + .expect("policy history lookup") + .is_none(), + "effective-policy lookup must not create policy history" + ); + } + + #[tokio::test] + async fn canonical_mcp_version_order_produces_identical_policy_bytes_and_hashes() { + let state = test_server_state().await; + let forward = mcp_policy_with_versions(&["2025-03-26", "2025-06-18", "2025-11-25"]); + let forward = + validate_and_canonicalize_policy(forward).expect("forward policy must validate"); + let reverse = mcp_policy_with_versions(&["2025-11-25", "2025-06-18", "2025-03-26"]); + let reverse = + validate_and_canonicalize_policy(reverse).expect("reverse policy must validate"); + + assert_eq!(forward.encode_to_vec(), reverse.encode_to_vec()); + assert_eq!( + deterministic_policy_hash(&forward), + deterministic_policy_hash(&reverse) + ); + + handle_update_config( + &state, + with_user(Request::new(UpdateConfigRequest { + global: true, + policy: Some(mcp_policy_with_versions(&[ + "2025-11-25", + "2025-06-18", + "2025-03-26", + ])), + ..Default::default() + })), + ) + .await + .expect("global policy ingress must accept supported versions"); + + let persisted = state + .store + .get_latest_policy(GLOBAL_POLICY_SANDBOX_ID) + .await + .expect("global policy lookup") + .expect("global policy revision"); + assert_eq!(persisted.policy_payload, forward.encode_to_vec()); + assert_eq!(persisted.policy_hash, deterministic_policy_hash(&forward)); + } + + #[tokio::test] + async fn global_policy_ingress_persists_defaulted_mcp_versions_identically() { + let state = test_server_state().await; + let canonical = mcp_policy_with_versions(&["2025-11-25"]); + let canonical = validate_and_canonicalize_policy(canonical) + .expect("explicit default MCP policy must canonicalize"); + let canonical_payload = canonical.encode_to_vec(); + let canonical_hash = deterministic_policy_hash(&canonical); + let cases = [ + ("omitted-options", mcp_policy_without_options()), + ("empty-versions", mcp_policy_with_versions(&[])), + ( + "explicit-default", + mcp_policy_with_versions(&["2025-11-25"]), + ), + ]; + + for (case, policy) in cases { + let response = handle_update_config( + &state, + with_user(Request::new(UpdateConfigRequest { + global: true, + policy: Some(policy), + ..Default::default() + })), + ) + .await + .expect("defaulted global policy ingress must succeed") + .into_inner(); + + assert_eq!(response.version, 1, "{case}"); + assert_eq!(response.policy_hash, canonical_hash, "{case}"); + let persisted = state + .store + .get_latest_policy(GLOBAL_POLICY_SANDBOX_ID) + .await + .expect("global policy lookup") + .expect("global policy revision"); + assert_eq!(persisted.policy_payload, canonical_payload, "{case}"); + assert_eq!(persisted.policy_hash, canonical_hash, "{case}"); + + let global_settings = load_global_settings(state.store.as_ref()) + .await + .expect("global settings lookup"); + let stored_policy = decode_policy_from_global_settings(&global_settings) + .expect("stored global policy must decode") + .expect("stored global policy"); + assert_eq!(stored_policy, canonical, "{case}"); + assert_eq!( + mcp_versions(&stored_policy), + &["2025-11-25".to_string()], + "{case}" + ); + } + } + + #[tokio::test] + async fn policy_record_identity_global_deduplicates_defaulted_mcp_history() { + for (case, legacy_policy) in defaulted_mcp_policy_cases() { + let state = test_server_state().await; + let canonical = mcp_policy_with_versions(&["2025-11-25"]); + let canonical = validate_and_canonicalize_policy(canonical) + .expect("explicit default MCP policy must canonicalize"); + let canonical_hash = deterministic_policy_hash(&canonical); + + state + .store + .put_policy_revision( + &format!("global-legacy-default-{case}"), + GLOBAL_POLICY_SANDBOX_ID, + "", + 1, + &legacy_policy.encode_to_vec(), + "untrusted-legacy-hash", + ) + .await + .expect("store legacy global policy revision"); + state + .store + .update_policy_status( + GLOBAL_POLICY_SANDBOX_ID, + 1, + "loaded", + None, + Some(current_time_ms()), + ) + .await + .expect("mark legacy global policy loaded"); + + let response = handle_update_config( + &state, + with_user(Request::new(UpdateConfigRequest { + global: true, + policy: Some(mcp_policy_with_versions(&["2025-11-25"])), + ..Default::default() + })), + ) + .await + .expect("semantic default must reuse the legacy global revision") + .into_inner(); + + assert_eq!(response.version, 1, "{case}"); + assert_eq!(response.policy_hash, canonical_hash, "{case}"); + let revisions = state + .store + .list_policies(GLOBAL_POLICY_SANDBOX_ID, 10, 0) + .await + .expect("list global policy revisions"); + assert_eq!(revisions.len(), 1, "{case}"); + let settings = load_global_settings(state.store.as_ref()) + .await + .expect("load canonical global setting"); + assert_eq!( + decode_policy_from_global_settings(&settings) + .expect("decode canonical global setting") + .expect("global policy setting"), + canonical, + "{case}" + ); + } + } + + #[tokio::test] + async fn policy_record_identity_sandbox_deduplicates_defaulted_mcp_history() { + let state = test_server_state().await; + let canonical = mcp_policy_with_versions(&["2025-11-25"]); + let canonical = validate_and_canonicalize_policy(canonical) + .expect("explicit default MCP policy must canonicalize"); + let canonical_hash = deterministic_policy_hash(&canonical); + + for (case, legacy_policy) in defaulted_mcp_policy_cases() { + let sandbox_id = format!("identity-default-{case}"); + let sandbox_name = format!("identity-default-{case}"); + state + .store + .put_message(&test_sandbox( + &sandbox_id, + &sandbox_name, + canonical.clone(), + Vec::new(), + )) + .await + .expect("store sandbox"); + state + .store + .put_policy_revision( + &format!("policy-identity-default-{case}"), + &sandbox_id, + "default", + 1, + &legacy_policy.encode_to_vec(), + "untrusted-legacy-hash", + ) + .await + .expect("store legacy sandbox policy revision"); + + let response = handle_update_config( + &state, + with_user(Request::new(UpdateConfigRequest { + name: sandbox_name, + policy: Some(mcp_policy_with_versions(&["2025-11-25"])), + workspace_scope: Some(openshell_core::proto::workspace_selector("default")), + ..Default::default() + })), + ) + .await + .expect("semantic default must reuse the legacy sandbox revision") + .into_inner(); + + assert_eq!(response.version, 1, "{case}"); + assert_eq!(response.policy_hash, canonical_hash, "{case}"); + let revisions = state + .store + .list_policies(&sandbox_id, 10, 0) + .await + .expect("list sandbox policy revisions"); + assert_eq!(revisions.len(), 1, "{case}"); + } + } + + #[tokio::test] + async fn policy_record_identity_merge_deduplicates_defaulted_mcp_history() { + let store = test_store().await; + let canonical = validate_and_canonicalize_policy(mcp_policy_with_versions(&["2025-11-25"])) + .expect("explicit default MCP policy must canonicalize"); + let canonical_hash = deterministic_policy_hash(&canonical); + + for (case, legacy_policy) in defaulted_mcp_policy_cases() { + let sandbox_id = format!("identity-merge-default-{case}"); + store + .put_policy_revision( + &format!("policy-identity-merge-default-{case}"), + &sandbox_id, + "default", + 1, + &legacy_policy.encode_to_vec(), + "untrusted-legacy-hash", + ) + .await + .expect("store legacy merge base"); + + let (version, hash, _) = apply_merge_operations_with_retry( + &store, + &sandbox_id, + "default", + None, + &[], + PolicyMergeValidationContext { + provider_layers: &[], + credential_binding: None, + }, + None, + None, + ) + .await + .expect("empty merge must reuse the semantic policy revision"); + + assert_eq!(version, 1, "{case}"); + assert_eq!(hash, canonical_hash, "{case}"); + let revisions = store + .list_policies(&sandbox_id, 10, 0) + .await + .expect("list merge policy revisions"); + assert_eq!(revisions.len(), 1, "{case}"); + } + } + + #[tokio::test] + async fn policy_record_identity_ignores_stale_matching_metadata() { + let state = test_server_state().await; + let candidate = mcp_policy_with_versions(&["2025-11-25"]); + let candidate = validate_and_canonicalize_policy(candidate) + .expect("candidate MCP policy must canonicalize"); + let candidate_hash = deterministic_policy_hash(&candidate); + let different = mcp_policy_with_versions(&["2025-06-18"]); + let different = validate_and_canonicalize_policy(different) + .expect("different MCP policy must canonicalize"); + let sandbox_id = "identity-stale-metadata"; + let sandbox_name = "identity-stale-metadata"; + + state + .store + .put_message(&test_sandbox( + sandbox_id, + sandbox_name, + candidate.clone(), + Vec::new(), + )) + .await + .expect("store sandbox"); + state + .store + .put_policy_revision( + "policy-identity-stale-metadata", + sandbox_id, + "default", + 1, + &different.encode_to_vec(), + &candidate_hash, + ) + .await + .expect("store policy with stale matching metadata"); + + let response = handle_update_config( + &state, + with_user(Request::new(UpdateConfigRequest { + name: sandbox_name.to_string(), + policy: Some(candidate.clone()), + workspace_scope: Some(openshell_core::proto::workspace_selector("default")), + ..Default::default() + })), + ) + .await + .expect("different payload must create a new revision") + .into_inner(); + + assert_eq!(response.version, 2); + assert_eq!(response.policy_hash, candidate_hash); + let latest = state + .store + .get_latest_policy(sandbox_id) + .await + .expect("load latest policy") + .expect("new policy revision"); + assert_eq!(latest.version, 2); + assert_eq!(latest.policy_payload, candidate.encode_to_vec()); + } + + #[tokio::test] + async fn valid_full_replacement_recovers_from_malformed_payload_with_matching_metadata() { + let state = test_server_state().await; + let candidate = mcp_policy_with_versions(&["2025-11-25"]); + let candidate = validate_and_canonicalize_policy(candidate) + .expect("candidate MCP policy must canonicalize"); + let candidate_hash = deterministic_policy_hash(&candidate); + let malformed = mcp_policy_with_versions(&["2025-11-25", "2025-11-25"]); + let sandbox_id = "identity-malformed-payload"; + let sandbox_name = "identity-malformed-payload"; + + state + .store + .put_message(&test_sandbox( + sandbox_id, + sandbox_name, + candidate.clone(), + Vec::new(), + )) + .await + .expect("store sandbox"); + state + .store + .put_policy_revision( + "policy-identity-malformed-payload", + sandbox_id, + "default", + 1, + &malformed.encode_to_vec(), + &candidate_hash, + ) + .await + .expect("store malformed policy payload"); + + let response = handle_update_config( + &state, + with_user(Request::new(UpdateConfigRequest { + name: sandbox_name.to_string(), + policy: Some(candidate.clone()), + workspace_scope: Some(openshell_core::proto::workspace_selector("default")), + ..Default::default() + })), + ) + .await + .expect("valid full replacement must repair malformed latest history") + .into_inner(); + + assert_eq!(response.version, 2); + assert_eq!(response.policy_hash, candidate_hash); + let revisions = state + .store + .list_policies(sandbox_id, 10, 0) + .await + .expect("list sandbox policy revisions"); + assert_eq!(revisions.len(), 2); + assert_eq!(revisions[0].version, 2); + assert_eq!(revisions[0].policy_payload, candidate.encode_to_vec()); + assert_eq!(revisions[0].policy_hash, candidate_hash); + assert_eq!(revisions[1].version, 1); + } + #[test] fn provider_credentialed_stamping_matches_host_patterns_and_ports() { let mut policy = ProtoSandboxPolicy { @@ -7323,6 +8446,7 @@ mod tests { let req = UpdateConfigRequest { name: "sandbox-1".to_string(), policy: Some(ProtoSandboxPolicy::default()), + workspace_scope: Some(openshell_core::proto::workspace_selector("default")), ..Default::default() }; assert!(validate_sandbox_caller_update(&req).is_ok()); @@ -7345,6 +8469,7 @@ mod tests { name: "sandbox-1".to_string(), setting_key: "inference.model".to_string(), setting_value: Some(SettingValue { value: None }), + workspace_scope: Some(openshell_core::proto::workspace_selector("default")), ..Default::default() }; let err = validate_sandbox_caller_update(&req).unwrap_err(); @@ -7425,7 +8550,9 @@ mod tests { &state, with_user(Request::new(GetSandboxLogsRequest { sandbox_id: "sandbox-b-id".to_string(), - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), ..GetSandboxLogsRequest::default() })), ) @@ -7511,6 +8638,47 @@ mod tests { ); } + #[tokio::test] + async fn global_policy_requests_reject_workspace_selectors() { + let state = test_server_state().await; + + let update_error = handle_update_config( + &state, + authed_request(UpdateConfigRequest { + global: true, + workspace_scope: Some(openshell_core::proto::workspace_selector("default")), + ..Default::default() + }), + ) + .await + .unwrap_err(); + assert_eq!(update_error.code(), Code::InvalidArgument); + + let get_error = handle_get_sandbox_policy_status( + &state, + authed_request(GetSandboxPolicyStatusRequest { + global: true, + workspace_scope: Some(openshell_core::proto::workspace_selector("default")), + ..Default::default() + }), + ) + .await + .unwrap_err(); + assert_eq!(get_error.code(), Code::InvalidArgument); + + let list_error = handle_list_sandbox_policies( + &state, + authed_request(ListSandboxPoliciesRequest { + global: true, + workspace_scope: Some(openshell_core::proto::workspace_selector("default")), + ..Default::default() + }), + ) + .await + .unwrap_err(); + assert_eq!(list_error.code(), Code::InvalidArgument); + } + #[tokio::test] async fn update_config_rejects_missing_principal() { let state = test_server_state().await; @@ -7546,6 +8714,13 @@ mod tests { #[test] fn policy_merge_error_mapping_distinguishes_request_shape_from_state_conflicts() { + let invalid_operation = + map_policy_merge_error(openshell_policy::PolicyMergeError::InvalidOperationPolicy { + operation_index: 0, + violations: Vec::new(), + }); + assert_eq!(invalid_operation.code(), Code::InvalidArgument); + let empty = map_policy_merge_error(openshell_policy::PolicyMergeError::EmptyAddRuleEndpoints { operation_index: 0, @@ -7639,6 +8814,18 @@ mod tests { ); assert_eq!(any_binary.code(), Code::FailedPrecondition); assert!(any_binary.message().contains("/usr/bin/untrusted")); + + let invalid_input = + map_policy_merge_error(openshell_policy::PolicyMergeError::InvalidInputPolicy { + violations: Vec::new(), + }); + assert_eq!(invalid_input.code(), Code::FailedPrecondition); + + let invalid_merged = + map_policy_merge_error(openshell_policy::PolicyMergeError::InvalidMergedPolicy { + violations: Vec::new(), + }); + assert_eq!(invalid_merged.code(), Code::Internal); } // ---- Sandbox IDOR guard (issue #1354) ---- @@ -7783,7 +8970,9 @@ mod tests { Request::new(GetDraftPolicyRequest { name: "sandbox-b".to_string(), status_filter: String::new(), - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), }), "sb-a", ); @@ -7800,6 +8989,7 @@ mod tests { Request::new(UpdateConfigRequest { name: "missing-sandbox".to_string(), policy: Some(ProtoSandboxPolicy::default()), + workspace_scope: Some(openshell_core::proto::workspace_selector("default")), ..Default::default() }), "sb-a", @@ -7835,7 +9025,9 @@ mod tests { Request::new(GetDraftPolicyRequest { name: "missing-sandbox".to_string(), status_filter: String::new(), - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), }), "sb-a", ); @@ -8087,21 +9279,6 @@ mod tests { sandbox } - async fn enable_providers_v2(state: &Arc) { - let global_settings = StoredSettings { - revision: 1, - settings: std::iter::once(( - settings::PROVIDERS_V2_ENABLED_KEY.to_string(), - StoredSettingValue::Bool(true), - )) - .collect(), - ..Default::default() - }; - save_global_settings(state.store.as_ref(), &global_settings) - .await - .unwrap(); - } - async fn get_sandbox_policy(state: &Arc, sandbox_id: &str) -> ProtoSandboxPolicy { handle_get_sandbox_config( state, @@ -8199,7 +9376,6 @@ mod tests { Arc::clone(&fetch_count), ); let state = Arc::new(state); - enable_providers_v2(&state).await; let mut provider_a = test_provider("provider-a", "moving-a"); provider_a.credentials = HashMap::from([("TOKEN_A".to_string(), "a".to_string())]); @@ -8339,7 +9515,7 @@ mod tests { .await .unwrap(); store - .put_message(&openshell_core::proto::StoredProviderProfile { + .put_message(&StoredProviderProfile { metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { id: "profile-generic".to_string(), name: "generic".to_string(), @@ -8382,6 +9558,48 @@ mod tests { assert_eq!(layers[0].rule.endpoints[0].host, "backdoor.example"); } + #[tokio::test] + async fn provider_policy_layers_prefer_exact_imported_alias_profile() { + let store = test_store().await; + store + .put_message(&test_provider("enterprise-github", "gh")) + .await + .unwrap(); + store + .put_message(&StoredProviderProfile { + metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { + id: "profile-gh".to_string(), + name: "gh".to_string(), + workspace: "default".to_string(), + ..Default::default() + }), + profile: Some(openshell_core::proto::ProviderProfile { + id: "gh".to_string(), + display_name: "Enterprise GitHub".to_string(), + endpoints: vec![NetworkEndpoint { + host: "github.enterprise.example".to_string(), + port: 443, + ..Default::default() + }], + ..Default::default() + }), + }) + .await + .unwrap(); + + let layers = + profile_provider_policy_layers(&store, "default", &["enterprise-github".to_string()]) + .await + .unwrap(); + + assert_eq!(layers.len(), 1); + assert_eq!(layers[0].rule.endpoints.len(), 1); + assert_eq!( + layers[0].rule.endpoints[0].host, + "github.enterprise.example" + ); + } + #[tokio::test] #[allow(deprecated)] async fn provider_policy_layers_include_custom_provider_profiles() { @@ -8391,7 +9609,7 @@ mod tests { .await .unwrap(); store - .put_message(&openshell_core::proto::StoredProviderProfile { + .put_message(&StoredProviderProfile { metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { id: "profile-custom-api".to_string(), name: "custom-api".to_string(), @@ -8463,7 +9681,7 @@ mod tests { .await .unwrap(); store - .put_message(&openshell_core::proto::StoredProviderProfile { + .put_message(&StoredProviderProfile { metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { id: "profile-custom-api".to_string(), name: "custom-api".to_string(), @@ -8598,33 +9816,62 @@ mod tests { ); } + #[tokio::test] + async fn provider_policy_layers_skip_public_vendor_endpoints_for_alternate_upstreams() { + let store = test_store().await; + let mut openai = test_provider("alternate-openai", "openai"); + openai.config.insert( + "OPENAI_BASE_URL".to_string(), + "https://api.example.com/v1".to_string(), + ); + let mut anthropic = test_provider("alternate-anthropic", "anthropic"); + anthropic.config.insert( + "ANTHROPIC_BASE_URL".to_string(), + "https://api.example.com/v1".to_string(), + ); + store.put_message(&openai).await.unwrap(); + store.put_message(&anthropic).await.unwrap(); + + let layers = profile_provider_policy_layers( + &store, + "default", + &[ + "alternate-openai".to_string(), + "alternate-anthropic".to_string(), + ], + ) + .await + .unwrap(); + + assert!(layers.is_empty()); + } + #[tokio::test] async fn provider_policy_layers_respect_profile_workspace_scope() { let store = test_store().await; - let make_stored_profile = - |id: &str, workspace: &str, host: &str| openshell_core::proto::StoredProviderProfile { - metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { - id: format!("profile-{id}-{workspace}"), - name: id.to_string(), - created_at_ms: 1_000_000, - labels: HashMap::new(), - resource_version: 0, - annotations: HashMap::new(), - workspace: workspace.to_string(), - deletion_timestamp_ms: 0, - }), - profile: Some(openshell_core::proto::ProviderProfile { - id: id.to_string(), - display_name: format!("{host} profile"), - endpoints: vec![NetworkEndpoint { - host: host.to_string(), - port: 443, - ..Default::default() - }], + let make_stored_profile = |id: &str, workspace: &str, host: &str| StoredProviderProfile { + metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { + id: format!("profile-{id}-{workspace}"), + name: id.to_string(), + created_at_ms: 1_000_000, + labels: HashMap::new(), + resource_version: 0, + annotations: HashMap::new(), + workspace: workspace.to_string(), + deletion_timestamp_ms: 0, + }), + profile: Some(openshell_core::proto::ProviderProfile { + id: id.to_string(), + display_name: format!("{host} profile"), + endpoints: vec![NetworkEndpoint { + host: host.to_string(), + port: 443, ..Default::default() - }), - }; + }], + ..Default::default() + }), + }; store .put_message(&make_stored_profile( @@ -8671,65 +9918,9 @@ mod tests { ); } - #[test] - fn providers_v2_enabled_defaults_false_when_unset() { - assert!( - !bool_setting_enabled( - &StoredSettings::default(), - settings::PROVIDERS_V2_ENABLED_KEY - ) - .unwrap() - ); - } - - #[test] - fn providers_v2_enabled_reads_global_bool_setting() { - let mut settings = StoredSettings::default(); - settings.settings.insert( - settings::PROVIDERS_V2_ENABLED_KEY.to_string(), - StoredSettingValue::Bool(true), - ); - - assert!(bool_setting_enabled(&settings, settings::PROVIDERS_V2_ENABLED_KEY).unwrap()); - } - #[tokio::test] - async fn sandbox_config_omits_provider_layers_when_v2_disabled() { + async fn sandbox_config_always_composes_provider_layers() { let state = test_server_state().await; - state - .store - .put_message(&test_provider("work-github", "github")) - .await - .unwrap(); - state - .store - .put_message(&test_sandbox( - "sb-v2-disabled", - "v2-disabled", - test_policy_with_rule("sandbox_only", "sandbox.example.com"), - vec!["work-github".to_string()], - )) - .await - .unwrap(); - - let effective_policy = get_sandbox_policy(&state, "sb-v2-disabled").await; - - assert!( - effective_policy - .network_policies - .contains_key("sandbox_only") - ); - assert!( - !effective_policy - .network_policies - .contains_key("_provider_work_github") - ); - } - - #[tokio::test] - async fn sandbox_config_composes_provider_layers_when_v2_enabled() { - let state = test_server_state().await; - enable_providers_v2(&state).await; state .store .put_message(&test_provider("work-github", "github")) @@ -8769,6 +9960,93 @@ mod tests { ); } + #[tokio::test] + async fn sandbox_config_materializes_default_mcp_version_after_provider_composition() { + use openshell_core::proto::{ProviderProfile, ProviderProfileCategory}; + + let state = test_server_state().await; + state + .store + .put_message(&StoredProviderProfile { + metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { + id: "profile-mcp-default".to_string(), + name: "mcp-default".to_string(), + created_at_ms: 1_000_000, + labels: HashMap::new(), + resource_version: 0, + annotations: HashMap::new(), + workspace: "default".to_string(), + deletion_timestamp_ms: 0, + }), + profile: Some(ProviderProfile { + id: "mcp-default".to_string(), + display_name: "MCP default".to_string(), + category: ProviderProfileCategory::Other as i32, + endpoints: vec![NetworkEndpoint { + host: "mcp.example.com".to_string(), + port: 443, + protocol: "mcp".to_string(), + mcp: None, + rules: vec![L7Rule { + allow: Some(openshell_core::proto::L7Allow { + method: "tools/list".to_string(), + ..Default::default() + }), + }], + ..Default::default() + }], + ..Default::default() + }), + }) + .await + .expect("store versionless MCP provider profile"); + state + .store + .put_message(&test_provider("work-mcp-default", "mcp-default")) + .await + .expect("store MCP provider"); + state + .store + .put_message(&test_sandbox( + "sb-mcp-default-composed", + "mcp-default-composed", + test_policy_with_rule("sandbox_only", "sandbox.example.com"), + vec!["work-mcp-default".to_string()], + )) + .await + .expect("store MCP sandbox"); + + let response = handle_get_sandbox_config( + &state, + with_user(Request::new(GetSandboxConfigRequest { + sandbox_id: "sb-mcp-default-composed".to_string(), + })), + ) + .await + .expect("provider-composed MCP policy must materialize") + .into_inner(); + let effective_policy = response.policy.expect("effective composed policy"); + let endpoint = effective_policy + .network_policies + .values() + .flat_map(|rule| &rule.endpoints) + .find(|endpoint| endpoint.host == "mcp.example.com") + .expect("composed MCP endpoint"); + + assert_eq!( + endpoint + .mcp + .as_ref() + .expect("canonical MCP options") + .versions, + ["2025-11-25".to_string()] + ); + assert_eq!( + response.policy_hash, + deterministic_policy_hash(&effective_policy) + ); + } + #[test] fn candidate_effective_policy_rejects_provider_endpoint_ambiguity() { let base = test_policy_with_rule("base", "api.example.com"); @@ -8805,7 +10083,9 @@ mod tests { &state, with_user(Request::new(UpdateConfigRequest { name: "ambiguous-update".to_string(), - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), policy: Some(test_ambiguous_policy()), ..Default::default() })), @@ -8842,7 +10122,9 @@ mod tests { &state, with_user(Request::new(UpdateConfigRequest { name: "unattached-binding".to_string(), - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), policy: Some(test_policy_with_credential_binding( "cloud", "api.cloud.example", @@ -8887,7 +10169,9 @@ mod tests { &state, with_user(Request::new(UpdateConfigRequest { name: "double-binding".to_string(), - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), policy: Some(test_policy_with_credential_binding( "cloud", "api.cloud.example", @@ -8905,9 +10189,7 @@ mod tests { #[tokio::test] async fn update_config_gates_uninspected_endpointless_credential_binding() { - use openshell_core::proto::{ - ProviderProfile, ProviderProfileCategory, StoredProviderProfile, - }; + use openshell_core::proto::{ProviderProfile, ProviderProfileCategory}; let state = test_server_state().await; state @@ -8957,7 +10239,9 @@ mod tests { &state, with_user(Request::new(UpdateConfigRequest { name: "endpointless-gating".to_string(), - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), policy: Some(l4.clone()), ..Default::default() })), @@ -8980,7 +10264,9 @@ mod tests { &state, with_user(Request::new(UpdateConfigRequest { name: "endpointless-gating".to_string(), - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), policy: Some(tls_skip.clone()), ..Default::default() })), @@ -8994,7 +10280,9 @@ mod tests { &state, with_user(Request::new(UpdateConfigRequest { name: "endpointless-gating".to_string(), - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), merge_operations: vec![add_bound_rule(&l4)], ..Default::default() })), @@ -9008,7 +10296,9 @@ mod tests { &state, with_user(Request::new(UpdateConfigRequest { name: "endpointless-gating".to_string(), - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), merge_operations: vec![add_bound_rule(&tls_skip)], ..Default::default() })), @@ -9039,7 +10329,9 @@ mod tests { &state, with_user(Request::new(UpdateConfigRequest { name: "endpointless-gating".to_string(), - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), merge_operations: vec![add_bound_rule(&opted_in)], ..Default::default() })), @@ -9072,7 +10364,9 @@ mod tests { &state, with_user(Request::new(UpdateConfigRequest { name: "signing-no-source".to_string(), - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), policy: Some(test_sigv4_policy("s3.amazonaws.com", None)), ..Default::default() })), @@ -9118,7 +10412,9 @@ mod tests { &state, with_user(Request::new(UpdateConfigRequest { name: "signing-unbound-aws".to_string(), - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), policy: Some(test_sigv4_policy("s3.amazonaws.com", None)), ..Default::default() })), @@ -9155,7 +10451,9 @@ mod tests { &state, with_user(Request::new(UpdateConfigRequest { name: "signing-bound-aws".to_string(), - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), policy: Some(test_sigv4_policy("s3.amazonaws.com", Some("aws-prod"))), ..Default::default() })), @@ -9190,12 +10488,19 @@ mod tests { sandbox.spec.as_mut().unwrap().policy = None; state.store.put_message(&sandbox).await.unwrap(); + let mut policy = test_sigv4_policy("bucket.s3.amazonaws.com", None); + let endpoint = &mut policy.network_policies.get_mut("aws").unwrap().endpoints[0]; + endpoint.access = "read-write".to_string(); + endpoint.enforcement = "enforce".to_string(); + handle_update_config( &state, with_user(Request::new(UpdateConfigRequest { name: "signing-profile-endpoint".to_string(), - workspace: "default".to_string(), - policy: Some(test_sigv4_policy("bucket.s3.amazonaws.com", None)), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), + policy: Some(policy), ..Default::default() })), ) @@ -9224,7 +10529,9 @@ mod tests { &state, with_user(Request::new(UpdateConfigRequest { name: "signing-profile-mismatch".to_string(), - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), policy: Some(test_sigv4_policy("api.example.com", None)), ..Default::default() })), @@ -9282,12 +10589,9 @@ mod tests { #[tokio::test] async fn provider_attachment_preflight_rejects_composed_ambiguity() { - use openshell_core::proto::{ - ProviderProfile, ProviderProfileCategory, StoredProviderProfile, - }; + use openshell_core::proto::{ProviderProfile, ProviderProfileCategory}; let state = test_server_state().await; - enable_providers_v2(&state).await; state .store .put_message(&StoredProviderProfile { @@ -9335,7 +10639,9 @@ mod tests { sandbox_name: "provider-ambiguity".to_string(), provider_name: "candidate-provider".to_string(), expected_resource_version: 0, - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), }), ) .await @@ -9356,11 +10662,10 @@ mod tests { async fn sandbox_config_rejects_invalid_provider_composed_policy() { use openshell_core::proto::{ MiddlewareEndpointSelector, NetworkMiddlewareConfig, ProviderProfile, - ProviderProfileCategory, StoredProviderProfile, + ProviderProfileCategory, }; let state = test_server_state().await; - enable_providers_v2(&state).await; let profile = StoredProviderProfile { metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { @@ -9434,9 +10739,8 @@ mod tests { } #[tokio::test] - async fn sandbox_config_skips_profileless_provider_types_when_v2_enabled() { + async fn sandbox_config_skips_profileless_provider_types() { let state = test_server_state().await; - enable_providers_v2(&state).await; state .store .put_message(&test_provider("legacy-generic", "generic")) @@ -9471,7 +10775,6 @@ mod tests { #[tokio::test] async fn sandbox_config_composition_is_jit_and_does_not_persist_provider_layers() { let state = test_server_state().await; - enable_providers_v2(&state).await; state .store .put_message(&test_provider("work-github", "github")) @@ -9520,7 +10823,7 @@ mod tests { use crate::grpc::provider::handle_update_provider_profiles; use openshell_core::proto::{ ProviderProfile, ProviderProfileCategory, ProviderProfileImportItem, - StoredProviderProfile, UpdateProviderProfilesRequest, + UpdateProviderProfilesRequest, }; fn stored_profile(host: &str) -> StoredProviderProfile { @@ -9558,7 +10861,6 @@ mod tests { } let state = test_server_state().await; - enable_providers_v2(&state).await; state .store .put_message(&stored_profile("api.before.example")) @@ -9661,7 +10963,6 @@ mod tests { #[tokio::test] async fn sandbox_config_composes_user_and_provider_rules() { let state = test_server_state().await; - enable_providers_v2(&state).await; state .store .put_message(&test_provider("work-github", "github")) @@ -9710,7 +11011,7 @@ mod tests { } #[tokio::test] - async fn provider_environment_resolution_is_unchanged_by_providers_v2_setting() { + async fn provider_environment_resolution_is_stable_across_policy_composition() { use openshell_core::proto::GetSandboxProviderEnvironmentRequest; let state = test_server_state().await; @@ -9742,7 +11043,6 @@ mod tests { .into_inner() .environment; - enable_providers_v2(&state).await; let v2_env = handle_get_sandbox_provider_environment( &state, with_user(Request::new(GetSandboxProviderEnvironmentRequest { @@ -9805,7 +11105,7 @@ mod tests { .put_message(&test_provider("work-github", "github")) .await .unwrap(); - let mut profileless_openai = test_provider("gateway-openai", "openai"); + let mut profileless_openai = test_provider("gateway-openai", "legacy-openai"); profileless_openai.credentials = HashMap::from([("OPENAI_API_KEY".to_string(), "openai-secret".to_string())]); state.store.put_message(&profileless_openai).await.unwrap(); @@ -9854,7 +11154,6 @@ mod tests { use openshell_core::proto::{ GetSandboxConfigRequest, GetSandboxProviderEnvironmentRequest, NetworkCredentialBinding, ProviderProfile, ProviderProfileCategory, - StoredProviderProfile, }; let state = test_server_state().await; @@ -9871,6 +11170,11 @@ mod tests { id: "endpointless".to_string(), display_name: "Endpointless".to_string(), category: ProviderProfileCategory::Other as i32, + credentials: vec![openshell_core::proto::ProviderProfileCredential { + name: "cloud_token".to_string(), + env_vars: vec!["CLOUD_TOKEN".to_string()], + ..Default::default() + }], endpoints: Vec::new(), ..Default::default() }), @@ -9964,7 +11268,9 @@ mod tests { &state, with_user(Request::new(UpdateConfigRequest { name: "policy-binding".to_string(), - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), policy: Some(next_policy.clone()), ..Default::default() })), @@ -10019,7 +11325,9 @@ mod tests { &state, with_user(Request::new(UpdateConfigRequest { name: "policy-binding".to_string(), - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), policy: Some(unbound_policy), ..Default::default() })), @@ -10071,7 +11379,7 @@ mod tests { async fn invalid_static_binding_does_not_suppress_valid_dynamic_credentials() { use openshell_core::proto::{ GetSandboxProviderEnvironmentRequest, ProviderCredentialTokenGrant, ProviderProfile, - ProviderProfileCategory, ProviderProfileCredential, StoredProviderProfile, + ProviderProfileCategory, ProviderProfileCredential, }; let state = test_server_state().await; @@ -10164,7 +11472,6 @@ mod tests { GetSandboxProviderEnvironmentRequest, ProviderCredentialTokenGrant, ProviderCredentialTokenGrantSubjectToken, ProviderCredentialTokenGrantType, ProviderProfile, ProviderProfileCategory, ProviderProfileCredential, - StoredProviderProfile, }; let state = test_server_state().await; @@ -10369,7 +11676,7 @@ mod tests { async fn provider_environment_revision_and_payload_share_immutable_record_snapshot() { use openshell_core::proto::{ ProviderCredentialTokenGrant, ProviderProfile, ProviderProfileCategory, - ProviderProfileCredential, StoredProviderProfile, + ProviderProfileCredential, }; fn dynamic_profile( @@ -10394,6 +11701,7 @@ mod tests { category: ProviderProfileCategory::Other as i32, credentials: vec![ProviderProfileCredential { name: "access_token".to_string(), + env_vars: vec!["GITHUB_TOKEN".to_string()], auth_style: "bearer".to_string(), header_name: "authorization".to_string(), token_grant: Some(ProviderCredentialTokenGrant { @@ -10556,8 +11864,7 @@ mod tests { use crate::grpc::provider::handle_update_provider_profiles; use openshell_core::proto::{ ProviderCredentialTokenGrant, ProviderProfile, ProviderProfileCategory, - ProviderProfileCredential, ProviderProfileImportItem, StoredProviderProfile, - UpdateProviderProfilesRequest, + ProviderProfileCredential, ProviderProfileImportItem, UpdateProviderProfilesRequest, }; use std::time::Duration; @@ -10673,9 +11980,7 @@ mod tests { #[tokio::test] async fn platform_profile_narrowing_changes_platform_provider_revision_when_shadowed() { use crate::persistence::WriteCondition; - use openshell_core::proto::{ - ProviderProfile, ProviderProfileCategory, StoredProviderProfile, - }; + use openshell_core::proto::{ProviderProfile, ProviderProfileCategory}; fn stored_profile(workspace: &str, path: &str) -> StoredProviderProfile { StoredProviderProfile { @@ -10769,7 +12074,6 @@ mod tests { }; let state = test_server_state().await; - enable_providers_v2(&state).await; state .store .put_message(&test_provider("work-github", "github")) @@ -10809,7 +12113,9 @@ mod tests { sandbox_name: "attach-lifecycle".to_string(), provider_name: "work-github".to_string(), expected_resource_version: 0, - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), })), ) .await @@ -10847,7 +12153,9 @@ mod tests { sandbox_name: "attach-lifecycle".to_string(), provider_name: "work-github".to_string(), expected_resource_version: 0, - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), }), ) .await @@ -10892,7 +12200,6 @@ mod tests { }; let state = test_server_state().await; - enable_providers_v2(&state).await; handle_import_provider_profiles( &state, authed_request(ImportProviderProfilesRequest { @@ -10980,7 +12287,9 @@ mod tests { sandbox_name: "attach-lifecycle".to_string(), provider_name: "work-custom".to_string(), expected_resource_version: 0, - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), })), ) .await @@ -11021,7 +12330,9 @@ mod tests { sandbox_name: "attach-lifecycle".to_string(), provider_name: "work-custom".to_string(), expected_resource_version: 0, - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), }), ) .await @@ -11051,7 +12362,7 @@ mod tests { } #[tokio::test] - async fn global_policy_suppresses_provider_profile_layers_when_v2_enabled() { + async fn global_policy_suppresses_provider_profile_layers() { use openshell_core::proto::{ GetSandboxConfigRequest, NetworkEndpoint, NetworkPolicyRule, SandboxPhase, SandboxPolicy, SandboxSpec, @@ -11119,17 +12430,10 @@ mod tests { }; let global_settings = StoredSettings { revision: 1, - settings: [ - ( - settings::PROVIDERS_V2_ENABLED_KEY.to_string(), - StoredSettingValue::Bool(true), - ), - ( - POLICY_SETTING_KEY.to_string(), - StoredSettingValue::Bytes(hex::encode(global_policy.encode_to_vec())), - ), - ] - .into_iter() + settings: std::iter::once(( + POLICY_SETTING_KEY.to_string(), + StoredSettingValue::Bytes(hex::encode(global_policy.encode_to_vec())), + )) .collect(), ..Default::default() }; @@ -11335,7 +12639,9 @@ mod tests { &state, with_user(Request::new(ApproveAllDraftChunksRequest { name: sandbox_name.to_string(), - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), approvals: chunks .iter() .map(|chunk| openshell_core::proto::DraftChunkApproval { @@ -11378,7 +12684,7 @@ mod tests { } #[tokio::test] - async fn approve_all_skips_later_batch_conflict_and_applies_compatible_prefix() { + async fn approve_all_skips_later_endpoint_conflict_and_applies_compatible_prefix() { let state = test_server_state().await; let sandbox_id = "sb-approve-all-conflict"; let sandbox_name = "approve-all-conflict"; @@ -11419,12 +12725,15 @@ mod tests { ..Default::default() }, PolicyChunk { - rule_name: "passthrough".to_string(), + rule_name: "conflicting".to_string(), proposed_rule: Some(NetworkPolicyRule { - name: "passthrough".to_string(), + name: "conflicting".to_string(), endpoints: vec![NetworkEndpoint { host: "shared.example.com".to_string(), port: 443, + protocol: "graphql".to_string(), + enforcement: "enforce".to_string(), + access: "read-only".to_string(), advisor_proposed: true, ..Default::default() }], @@ -11459,7 +12768,9 @@ mod tests { &state, with_user(Request::new(ApproveAllDraftChunksRequest { name: sandbox_name.to_string(), - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), approvals: chunks .iter() .map(|chunk| openshell_core::proto::DraftChunkApproval { @@ -11502,7 +12813,7 @@ mod tests { .unwrap(); let policy = ProtoSandboxPolicy::decode(revision.policy_payload.as_slice()).unwrap(); assert!(policy.network_policies.contains_key("inspected")); - assert!(!policy.network_policies.contains_key("passthrough")); + assert!(!policy.network_policies.contains_key("conflicting")); } #[tokio::test] @@ -11661,7 +12972,9 @@ mod tests { with_user(Request::new(ApproveAllDraftChunksRequest { name: sandbox_name.to_string(), include_security_flagged: false, - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), approvals: vec![openshell_core::proto::DraftChunkApproval { chunk_id: chunk_id.clone(), review_token: chunk.review_token.clone(), @@ -11689,7 +13002,9 @@ mod tests { with_user(Request::new(ApproveAllDraftChunksRequest { name: sandbox_name.to_string(), include_security_flagged: true, - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), approvals: vec![openshell_core::proto::DraftChunkApproval { chunk_id: chunk_id.clone(), review_token: chunk.review_token.clone(), @@ -11767,7 +13082,9 @@ mod tests { name: sandbox_name.to_string(), chunk_id: chunk_id.clone(), proposed_rule: Some(private_rule), - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), })), ) .await @@ -11786,7 +13103,9 @@ mod tests { with_user(Request::new(GetDraftPolicyRequest { name: sandbox_name.to_string(), status_filter: String::new(), - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), })), ) .await @@ -11803,7 +13122,9 @@ mod tests { with_user(Request::new(ApproveAllDraftChunksRequest { name: sandbox_name.to_string(), include_security_flagged: false, - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), ..Default::default() })), ) @@ -11869,7 +13190,9 @@ mod tests { with_user(Request::new(ApproveAllDraftChunksRequest { name: sandbox_name.to_string(), include_security_flagged: false, - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), ..Default::default() })), ) @@ -11939,7 +13262,9 @@ mod tests { with_user(Request::new(GetDraftPolicyRequest { name: sandbox_name.to_string(), status_filter: String::new(), - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), })), ) .await @@ -12053,7 +13378,9 @@ mod tests { name: sandbox_name.to_string(), chunk_id: chunk_id.clone(), proposed_rule: Some(finding_rule), - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), })), ) .await @@ -12186,7 +13513,9 @@ mod tests { with_user(Request::new(GetDraftPolicyRequest { name: sandbox_name.clone(), status_filter: String::new(), - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), })), ) .await @@ -12206,7 +13535,9 @@ mod tests { authed_request(ApproveDraftChunkRequest { name: sandbox_name.clone(), chunk_id: chunk_id.clone(), - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), review_token, }), ) @@ -12220,7 +13551,9 @@ mod tests { &state, authed_request(GetDraftHistoryRequest { name: sandbox_name.clone(), - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), }), ) .await @@ -12238,7 +13571,9 @@ mod tests { limit: 10, offset: 0, global: false, - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), }), ) .await @@ -12252,7 +13587,9 @@ mod tests { authed_request(UndoDraftChunkRequest { name: sandbox_name.clone(), chunk_id: chunk_id.clone(), - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), }), ) .await @@ -12266,7 +13603,9 @@ mod tests { with_user(Request::new(GetDraftPolicyRequest { name: sandbox_name.clone(), status_filter: String::new(), - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), })), ) .await @@ -12279,7 +13618,9 @@ mod tests { &state, authed_request(GetDraftHistoryRequest { name: sandbox_name.clone(), - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), }), ) .await @@ -12295,7 +13636,9 @@ mod tests { limit: 10, offset: 0, global: false, - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), }), ) .await @@ -12309,7 +13652,9 @@ mod tests { &state, authed_request(ClearDraftChunksRequest { name: sandbox_name.clone(), - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), }), ) .await @@ -12322,7 +13667,9 @@ mod tests { with_user(Request::new(GetDraftPolicyRequest { name: sandbox_name.clone(), status_filter: String::new(), - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), })), ) .await @@ -12334,7 +13681,9 @@ mod tests { &state, authed_request(GetDraftHistoryRequest { name: sandbox_name, - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), }), ) .await @@ -12411,7 +13760,9 @@ mod tests { name: sandbox_name.clone(), chunk_id: chunk_id.clone(), reason: guidance.to_string(), - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), }), ) .await @@ -12422,7 +13773,9 @@ mod tests { with_user(Request::new(GetDraftPolicyRequest { name: sandbox_name, status_filter: String::new(), - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), })), ) .await @@ -12529,7 +13882,9 @@ mod tests { with_user(Request::new(GetDraftPolicyRequest { name: sandbox_name, status_filter: String::new(), - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), })), ) .await @@ -12642,7 +13997,9 @@ mod tests { with_user(Request::new(GetDraftPolicyRequest { name: sandbox_name.clone(), status_filter: String::new(), - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), })), ) .await @@ -12654,13 +14011,13 @@ mod tests { .find(|c| c.id == mechanistic_chunk_id) .expect("mechanistic chunk present"); assert_eq!(mech.status, "pending"); - // Mechanistic L4 with credential in scope flags as new credentialed - // reach for the binary on the host. + // The attached GitHub profile already grants credentialed reach for + // this host, so the mechanistic proposal does not expand reach. assert!( - mech.validation_result + !mech + .validation_result .contains("credential_reach_expansion"), - "mechanistic L4 with credential in scope should emit \ - credential_reach_expansion; got: {}", + "profile-composed reach should prevent a duplicate expansion finding; got: {}", mech.validation_result ); @@ -12718,7 +14075,9 @@ mod tests { with_user(Request::new(GetDraftPolicyRequest { name: sandbox_name, status_filter: String::new(), - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), })), ) .await @@ -12848,7 +14207,9 @@ mod tests { with_user(Request::new(GetDraftPolicyRequest { name: sandbox_name, status_filter: String::new(), - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), })), ) .await @@ -12952,7 +14313,9 @@ mod tests { &state, with_user(Request::new(GetDraftPolicyRequest { name: sandbox_name, - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), ..Default::default() })), ) @@ -12972,7 +14335,10 @@ mod tests { let canonical = chunk.proposed_rule.as_ref().unwrap(); assert_eq!(canonical.endpoints[0].protocol, "rest"); assert_eq!(canonical.endpoints[0].access, "read-only"); - assert!(!canonical.endpoints[0].advisor_proposed); + assert!( + canonical.endpoints[0].advisor_proposed, + "a new advisor overlay must retain proposal provenance" + ); let revision = state .store @@ -12993,6 +14359,10 @@ mod tests { assert_eq!(curl_rule.endpoints[0].ports, vec![443]); assert_eq!(curl_rule.endpoints[0].protocol, "rest"); assert_eq!(curl_rule.endpoints[0].access, "read-only"); + assert!( + curl_rule.endpoints[0].advisor_proposed, + "the persisted advisor overlay must retain proposal provenance" + ); assert_eq!(curl_rule.binaries.len(), 1); assert_eq!(curl_rule.binaries[0].path, "/usr/bin/curl"); } @@ -13068,7 +14438,9 @@ mod tests { &state, with_user(Request::new(GetDraftPolicyRequest { name: sandbox_name, - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), ..Default::default() })), ) @@ -13199,7 +14571,9 @@ mod tests { with_user(Request::new(ApproveDraftChunkRequest { name: sandbox_name, chunk_id: chunk_id.clone(), - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), review_token: before.review_token.clone(), })), ) @@ -13383,7 +14757,9 @@ mod tests { with_user(Request::new(GetDraftPolicyRequest { name: sandbox_name, status_filter: String::new(), - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), })), ) .await @@ -13487,7 +14863,9 @@ mod tests { with_user(Request::new(GetDraftPolicyRequest { name: sandbox_name, status_filter: String::new(), - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), })), ) .await @@ -13584,7 +14962,9 @@ mod tests { with_user(Request::new(GetDraftPolicyRequest { name: sandbox_name, status_filter: String::new(), - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), })), ) .await @@ -13672,7 +15052,9 @@ mod tests { with_user(Request::new(GetDraftPolicyRequest { name: sandbox_name, status_filter: String::new(), - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), })), ) .await @@ -13764,7 +15146,9 @@ mod tests { with_user(Request::new(GetDraftPolicyRequest { name: sandbox_name, status_filter: String::new(), - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), })), ) .await @@ -13859,7 +15243,9 @@ mod tests { with_user(Request::new(GetDraftPolicyRequest { name: sandbox_name, status_filter: String::new(), - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), })), ) .await @@ -13958,6 +15344,93 @@ mod tests { ); } + #[tokio::test] + async fn agent_authored_submit_rejects_native_tcp_and_tls_skip_but_allows_explicit_proxy() { + use openshell_core::proto::{NetworkBinary, NetworkEndpoint, NetworkPolicyRule}; + + let state = test_server_state().await; + let sandbox_name = "reject-agent-raw-transports"; + state + .store + .put_message(&test_sandbox( + "sb-reject-agent-raw-transports", + sandbox_name, + ProtoSandboxPolicy::default(), + vec![], + )) + .await + .unwrap(); + + let endpoint = |protocol: &str, tls: &str| NetworkEndpoint { + host: "api.example.com".to_string(), + port: 443, + protocol: protocol.to_string(), + tls: tls.to_string(), + ..Default::default() + }; + let chunk = |name: &str, endpoint: NetworkEndpoint| PolicyChunk { + rule_name: name.to_string(), + proposed_rule: Some(NetworkPolicyRule { + name: name.to_string(), + endpoints: vec![endpoint], + binaries: vec![NetworkBinary { + path: "/usr/bin/curl".to_string(), + ..Default::default() + }], + }), + ..Default::default() + }; + + let response = handle_submit_policy_analysis( + &state, + with_user(Request::new(SubmitPolicyAnalysisRequest { + name: sandbox_name.to_string(), + analysis_mode: "agent_authored".to_string(), + proposed_chunks: vec![ + chunk("native_tcp", endpoint("tcp", "")), + chunk("raw_tls", endpoint("", "skip")), + chunk("explicit_proxy", endpoint("", "")), + ], + ..Default::default() + })), + ) + .await + .unwrap() + .into_inner(); + + assert_eq!(response.accepted_chunks, 1); + assert_eq!(response.rejected_chunks, 2); + assert_eq!(response.rejection_reasons.len(), 2); + assert!( + response + .rejection_reasons + .iter() + .any(|reason| reason.contains("protocol tcp")) + ); + assert!( + response + .rejection_reasons + .iter() + .any(|reason| reason.contains("tls: skip")) + ); + + let draft = handle_get_draft_policy( + &state, + with_user(Request::new(GetDraftPolicyRequest { + name: sandbox_name.to_string(), + status_filter: String::new(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), + })), + ) + .await + .unwrap() + .into_inner(); + assert_eq!(draft.chunks.len(), 1); + assert_eq!(draft.chunks[0].rule_name, "explicit_proxy"); + } + #[tokio::test] async fn approve_draft_chunk_rejects_stored_reserved_provider_rule_name() { use openshell_core::proto::{NetworkBinary, NetworkEndpoint, NetworkPolicyRule}; @@ -14021,7 +15494,9 @@ mod tests { with_user(Request::new(ApproveDraftChunkRequest { name: sandbox_name.to_string(), chunk_id: chunk.id.clone(), - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), ..Default::default() })), ) @@ -14132,7 +15607,9 @@ mod tests { with_user(Request::new(GetDraftPolicyRequest { name: sandbox_name, status_filter: String::new(), - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), })), ) .await @@ -14232,7 +15709,9 @@ mod tests { with_user(Request::new(GetDraftPolicyRequest { name: sandbox_name, status_filter: String::new(), - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), })), ) .await @@ -14321,7 +15800,9 @@ mod tests { with_user(Request::new(GetDraftPolicyRequest { name: sandbox_name, status_filter: String::new(), - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), })), ) .await @@ -14340,15 +15821,13 @@ mod tests { } #[tokio::test] - async fn agent_authored_validation_uses_providers_v2_effective_policy() { + async fn agent_authored_validation_uses_profile_composed_effective_policy() { use openshell_core::proto::{ FilesystemPolicy, L7Allow, L7DenyRule, L7Rule, NetworkBinary, NetworkEndpoint, ProviderProfile, ProviderProfileCategory, SandboxPhase, SandboxPolicy, SandboxSpec, - StoredProviderProfile, }; let state = test_server_state().await; - enable_providers_v2(&state).await; state .store .put_message(&test_provider("work-custom", "custom-api")) @@ -14400,10 +15879,11 @@ mod tests { .await .unwrap(); + let sandbox_id = "sb-agent-provider-effective-policy"; let sandbox_name = "agent-provider-effective-policy".to_string(); let mut sandbox = Sandbox { metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { - id: "sb-agent-provider-effective-policy".to_string(), + id: sandbox_id.to_string(), name: sandbox_name.clone(), created_at_ms: 1_000_000, labels: HashMap::new(), @@ -14429,6 +15909,7 @@ mod tests { sandbox.set_phase(SandboxPhase::Ready as i32); state.store.put_message(&sandbox).await.unwrap(); + #[allow(deprecated)] let proposed_rule = NetworkPolicyRule { name: "github_contents_write".to_string(), endpoints: vec![NetworkEndpoint { @@ -14446,15 +15927,16 @@ mod tests { ..Default::default() }), }], + advisor_proposed: true, ..Default::default() }], binaries: vec![NetworkBinary { path: "/usr/bin/curl".to_string(), - ..Default::default() + harness: true, }], }; - handle_submit_policy_analysis( + let submit = handle_submit_policy_analysis( &state, with_user(Request::new(SubmitPolicyAnalysisRequest { name: sandbox_name.clone(), @@ -14469,20 +15951,32 @@ mod tests { })), ) .await - .unwrap(); + .unwrap() + .into_inner(); + assert_eq!(submit.accepted_chunks, 1); + assert_eq!(submit.rejected_chunks, 0); + let chunk_id = submit.accepted_chunk_ids[0].clone(); let draft = handle_get_draft_policy( &state, with_user(Request::new(GetDraftPolicyRequest { - name: sandbox_name, + name: sandbox_name.clone(), status_filter: String::new(), - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), })), ) .await .unwrap() .into_inner(); - let verdict = &draft.chunks[0].validation_result; + let chunk = draft + .chunks + .iter() + .find(|chunk| chunk.id == chunk_id) + .expect("provider-overlap proposal should reach the draft inbox"); + assert_eq!(chunk.status, "pending"); + let verdict = &chunk.validation_result; let first_line = verdict.lines().next().unwrap_or(""); assert!( first_line.starts_with("prover: "), @@ -14492,7 +15986,51 @@ mod tests { assert!( !verdict.contains("validation unavailable"), "providers-v2 composition must not break the prover pipeline; \ - got: {verdict}" + got: {verdict}" + ); + + handle_approve_draft_chunk( + &state, + authed_request(ApproveDraftChunkRequest { + name: sandbox_name, + chunk_id, + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), + review_token: chunk.review_token.clone(), + }), + ) + .await + .expect("provider-overlap proposal should approve"); + + let stored = state + .store + .get_latest_policy(sandbox_id) + .await + .unwrap() + .expect("approval should persist a base-policy revision"); + let base_policy = ProtoSandboxPolicy::decode(stored.policy_payload.as_slice()).unwrap(); + assert!( + base_policy + .network_policies + .contains_key("github_contents_write") + ); + assert!( + !base_policy + .network_policies + .contains_key("_provider_work_custom"), + "provider-composed rules must not be copied into the mutable base policy" + ); + + let effective_policy = get_sandbox_policy(&state, sandbox_id).await; + let provider_rule = &effective_policy.network_policies["_provider_work_custom"]; + assert_eq!(provider_rule.endpoints[0].access, "full"); + assert_eq!(provider_rule.endpoints[0].deny_rules.len(), 1); + assert!(!provider_rule.endpoints[0].advisor_proposed); + assert!( + effective_policy + .network_policies + .contains_key("github_contents_write") ); } @@ -14517,7 +16055,6 @@ mod tests { }; let state = test_server_state().await; - enable_providers_v2(&state).await; // Github provider attached: a credential ends up in scope for // api.github.com (PUT proposal flags MEDIUM). raw.githubusercontent.com @@ -14647,7 +16184,9 @@ mod tests { with_user(Request::new(GetDraftPolicyRequest { name: sandbox_name, status_filter: String::new(), - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), })), ) .await @@ -14798,7 +16337,9 @@ mod tests { with_user(Request::new(GetDraftPolicyRequest { name: sandbox_name.clone(), status_filter: String::new(), - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), })), ) .await @@ -14820,7 +16361,9 @@ mod tests { name: sandbox_name, chunk_id: second.accepted_chunk_ids[0].clone(), reason: "redraft test".to_string(), - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), }), ) .await @@ -14904,7 +16447,9 @@ mod tests { with_user(Request::new(GetDraftPolicyRequest { name: sandbox_name, status_filter: String::new(), - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), })), ) .await @@ -15020,7 +16565,9 @@ mod tests { with_user(Request::new(GetDraftPolicyRequest { name: sandbox_name.clone(), status_filter: String::new(), - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), })), ) .await @@ -15041,7 +16588,9 @@ mod tests { with_user(Request::new(GetDraftPolicyRequest { name: sandbox_name.clone(), status_filter: String::new(), - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), })), ) .await @@ -15284,7 +16833,9 @@ mod tests { name: sandbox_name.clone(), chunk_id: chunk_id.clone(), reason: "scope too broad".to_string(), - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), }), ) .await @@ -15295,7 +16846,9 @@ mod tests { authed_request(ApproveDraftChunkRequest { name: sandbox_name.clone(), chunk_id: chunk_id.clone(), - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), review_token, }), ) @@ -15307,7 +16860,9 @@ mod tests { authed_request(UndoDraftChunkRequest { name: sandbox_name.clone(), chunk_id: chunk_id.clone(), - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), }), ) .await @@ -15318,7 +16873,9 @@ mod tests { with_user(Request::new(GetDraftPolicyRequest { name: sandbox_name, status_filter: String::new(), - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), })), ) .await @@ -15430,7 +16987,9 @@ mod tests { with_user(Request::new(GetDraftPolicyRequest { name: sandbox_a.object_name().to_string(), status_filter: String::new(), - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), })), ) .await @@ -15445,7 +17004,9 @@ mod tests { authed_request(ApproveDraftChunkRequest { name: other_name.clone(), chunk_id: chunk_id.clone(), - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), review_token: String::new(), }), ) @@ -15459,7 +17020,9 @@ mod tests { name: other_name.clone(), chunk_id: chunk_id.clone(), reason: "wrong sandbox".to_string(), - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), }), ) .await @@ -15472,7 +17035,9 @@ mod tests { name: other_name.clone(), chunk_id: chunk_id.clone(), proposed_rule: Some(proposed_rule.clone()), - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), }), ) .await @@ -15484,7 +17049,9 @@ mod tests { authed_request(ApproveDraftChunkRequest { name: sandbox_a.object_name().to_string(), chunk_id: chunk_id.clone(), - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), review_token, }), ) @@ -15496,7 +17063,9 @@ mod tests { authed_request(UndoDraftChunkRequest { name: other_name, chunk_id, - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), }), ) .await @@ -16258,6 +17827,44 @@ mod tests { assert_eq!(decoded.version, 7); } + #[test] + fn decode_policy_from_global_settings_validates_and_canonicalizes_stored_policy() { + let invalid = mcp_policy_with_versions(&["latest"]); + let invalid_global = StoredSettings { + revision: 1, + settings: std::iter::once(( + POLICY_SETTING_KEY.to_string(), + StoredSettingValue::Bytes(hex::encode(invalid.encode_to_vec())), + )) + .collect(), + ..Default::default() + }; + let error = decode_policy_from_global_settings(&invalid_global) + .expect_err("invalid global policy must fail closed"); + assert_eq!(error.code(), Code::FailedPrecondition); + assert!(error.message().contains(STORED_POLICY_SOURCE_GLOBAL)); + + let reversed = mcp_policy_with_versions(&["2025-11-25", "2025-06-18", "2025-03-26"]); + let canonical = validate_and_canonicalize_policy(reversed.clone()) + .expect("supported global policy must canonicalize"); + let valid_global = StoredSettings { + revision: 1, + settings: std::iter::once(( + POLICY_SETTING_KEY.to_string(), + StoredSettingValue::Bytes(hex::encode(reversed.encode_to_vec())), + )) + .collect(), + ..Default::default() + }; + + assert_eq!( + decode_policy_from_global_settings(&valid_global) + .expect("valid global policy") + .expect("global policy present"), + canonical + ); + } + #[test] fn config_revision_changes_when_effective_setting_changes() { let policy = ProtoSandboxPolicy::default(); @@ -16440,9 +18047,7 @@ mod tests { } async fn install_ambiguous_provider_binding(state: &Arc, suffix: &str) { - use openshell_core::proto::{ - ProviderProfile, ProviderProfileCategory, StoredProviderProfile, - }; + use openshell_core::proto::{ProviderProfile, ProviderProfileCategory}; let profile_name = format!("ambiguous-{suffix}"); let provider_name = format!("provider-{suffix}"); @@ -16491,32 +18096,6 @@ mod tests { .unwrap(); } - #[tokio::test] - async fn enabling_provider_composition_rejects_existing_ambiguous_binding() { - let state = test_server_state().await; - install_ambiguous_provider_binding(&state, "enable").await; - - let error = handle_update_config( - &state, - with_user(Request::new(UpdateConfigRequest { - global: true, - setting_key: settings::PROVIDERS_V2_ENABLED_KEY.to_string(), - setting_value: Some(SettingValue { - value: Some(setting_value::Value::BoolValue(true)), - }), - ..Default::default() - })), - ) - .await - .expect_err("provider composition must be validated before activation"); - - assert_eq!(error.code(), Code::FailedPrecondition); - assert!(error.message().contains("sandbox-enable")); - assert!(error.message().contains("tls")); - let settings = load_global_settings(state.store.as_ref()).await.unwrap(); - assert!(!bool_setting_enabled(&settings, settings::PROVIDERS_V2_ENABLED_KEY).unwrap()); - } - #[tokio::test] async fn deleting_global_policy_rejects_reactivated_ambiguous_provider_binding() { let state = test_server_state().await; @@ -16532,20 +18111,6 @@ mod tests { ) .await .expect("global policy should suppress provider composition"); - handle_update_config( - &state, - with_user(Request::new(UpdateConfigRequest { - global: true, - setting_key: settings::PROVIDERS_V2_ENABLED_KEY.to_string(), - setting_value: Some(SettingValue { - value: Some(setting_value::Value::BoolValue(true)), - }), - ..Default::default() - })), - ) - .await - .expect("providers may be enabled while a global policy is active"); - let error = handle_update_config( &state, with_user(Request::new(UpdateConfigRequest { @@ -16564,21 +18129,28 @@ mod tests { assert!(settings.settings.contains_key(POLICY_SETTING_KEY)); } + #[tokio::test] + async fn startup_preflight_rejects_persisted_ambiguous_provider_binding() { + let state = test_server_state().await; + install_ambiguous_provider_binding(&state, "upgrade").await; + + let error = validate_provider_composition_startup_preflight(&state) + .await + .expect_err("startup must reject policy that unconditional composition would activate"); + + assert_eq!(error.code(), Code::FailedPrecondition); + assert!(error.message().contains("sandbox-upgrade")); + assert!(error.message().contains("invalid effective policy")); + } + #[test] fn merge_effective_settings_global_overrides_sandbox_key() { let global = StoredSettings { revision: 2, - settings: [ - ( - settings::PROVIDERS_V2_ENABLED_KEY.to_string(), - StoredSettingValue::Bool(false), - ), - ( - settings::AGENT_POLICY_PROPOSALS_ENABLED_KEY.to_string(), - StoredSettingValue::Bool(false), - ), - ] - .into_iter() + settings: std::iter::once(( + settings::AGENT_POLICY_PROPOSALS_ENABLED_KEY.to_string(), + StoredSettingValue::Bool(false), + )) .collect(), ..Default::default() }; @@ -16586,7 +18158,7 @@ mod tests { revision: 1, settings: [ ( - settings::PROVIDERS_V2_ENABLED_KEY.to_string(), + settings::AGENT_POLICY_PROPOSALS_ENABLED_KEY.to_string(), StoredSettingValue::Bool(true), ), ( @@ -16600,15 +18172,6 @@ mod tests { }; let merged = merge_effective_settings(&global, &sandbox).unwrap(); - let providers_v2 = merged - .get(settings::PROVIDERS_V2_ENABLED_KEY) - .expect("providers_v2_enabled present"); - assert_eq!(providers_v2.scope, SettingScope::Global as i32); - assert_eq!( - providers_v2.value.as_ref().and_then(|v| v.value.as_ref()), - Some(&setting_value::Value::BoolValue(false)) - ); - let ocsf_json = merged .get("ocsf_json_enabled") .expect("ocsf_json_enabled present"); @@ -17608,7 +19171,9 @@ mod tests { merge_operations: vec![], expected_resource_version: current_version, annotations: HashMap::new(), - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), }), ) .await @@ -17704,7 +19269,9 @@ mod tests { merge_operations: vec![], expected_resource_version: current_version, annotations: annotations.clone(), - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), }), ) .await @@ -17777,6 +19344,7 @@ mod tests { "openshell.nvidia.com/policy-signature".to_string(), "same-hash-signature".to_string(), )]), + workspace_scope: Some(openshell_core::proto::workspace_selector("default")), ..Default::default() }), ) @@ -17860,6 +19428,7 @@ mod tests { name: "idempotent-provenance".to_string(), policy: Some(policy.clone()), annotations: annotations.clone(), + workspace_scope: Some(openshell_core::proto::workspace_selector("default")), ..Default::default() })), ) @@ -17872,6 +19441,7 @@ mod tests { name: "idempotent-provenance".to_string(), policy: Some(policy), annotations: annotations.clone(), + workspace_scope: Some(openshell_core::proto::workspace_selector("default")), ..Default::default() })), ) @@ -17925,6 +19495,7 @@ mod tests { with_user(Request::new(UpdateConfigRequest { name: "preserve-full".to_string(), policy: Some(updated), + workspace_scope: Some(openshell_core::proto::workspace_selector("default")), ..Default::default() })), ) @@ -17988,6 +19559,7 @@ mod tests { }, )), }], + workspace_scope: Some(openshell_core::proto::workspace_selector("default")), ..Default::default() })), ) @@ -18060,6 +19632,7 @@ mod tests { )), }], annotations: provenance.clone(), + workspace_scope: Some(openshell_core::proto::workspace_selector("default")), ..Default::default() })), ) @@ -18121,6 +19694,7 @@ mod tests { name: "preserve-backfill".to_string(), policy: Some(ProtoSandboxPolicy::default()), expected_resource_version: current_version, + workspace_scope: Some(openshell_core::proto::workspace_selector("default")), ..Default::default() }), ) @@ -18157,6 +19731,329 @@ mod tests { ); } + #[tokio::test] + async fn update_config_reports_immutable_removal_before_policy_safety_errors() { + let state = test_server_state().await; + let sandbox_id = "sb-static-error-priority"; + let sandbox_name = "static-error-priority"; + let baseline = openshell_policy::restrictive_default_policy(); + state + .store + .put_message(&test_sandbox( + sandbox_id, + sandbox_name, + baseline.clone(), + Vec::new(), + )) + .await + .unwrap(); + + let current = state + .store + .get_message_by_name::("default", sandbox_name) + .await + .unwrap() + .unwrap(); + let current_version = current.metadata.as_ref().unwrap().resource_version; + + // The replacement removes baseline paths and adds an unsafe traversal. + // Live-policy immutability is the earlier contract, so it must remain + // the stable failure even when later whole-policy validation would fail. + let mut unsafe_replacement = baseline; + unsafe_replacement.filesystem.as_mut().unwrap().read_only = + vec!["/usr/../etc/shadow".to_string()]; + + let error = handle_update_config( + &state, + with_user(Request::new(UpdateConfigRequest { + name: sandbox_name.to_string(), + policy: Some(unsafe_replacement), + expected_resource_version: current_version, + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), + ..Default::default() + })), + ) + .await + .expect_err("immutable removal must fail before whole-policy validation"); + + assert_eq!(error.code(), Code::InvalidArgument); + assert!(error.message().contains("cannot be removed")); + let unchanged = state + .store + .get_message_by_name::("default", sandbox_name) + .await + .unwrap() + .unwrap(); + assert_eq!( + unchanged + .spec + .as_ref() + .and_then(|spec| spec.policy.as_ref()), + Some(&openshell_policy::restrictive_default_policy()) + ); + assert!( + state + .store + .get_latest_policy(sandbox_id) + .await + .unwrap() + .is_none() + ); + } + + #[tokio::test] + async fn update_config_policy_backfill_validates_before_persistence() { + use openshell_core::proto::{SandboxPhase, SandboxSpec}; + + let state = test_server_state().await; + let sandbox_id = "sb-invalid-first-sync"; + let sandbox_name = "invalid-first-sync"; + let mut sandbox = Sandbox { + metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { + id: sandbox_id.to_string(), + name: sandbox_name.to_string(), + created_at_ms: 1_000_000, + labels: HashMap::new(), + resource_version: 0, + annotations: HashMap::new(), + workspace: "default".to_string(), + deletion_timestamp_ms: 0, + }), + spec: Some(SandboxSpec { + policy: None, + providers: Vec::new(), + ..Default::default() + }), + ..Default::default() + }; + sandbox.set_phase(SandboxPhase::Provisioning as i32); + state.store.put_message(&sandbox).await.unwrap(); + + let current = state + .store + .get_message_by_name::("default", sandbox_name) + .await + .unwrap() + .unwrap(); + let current_version = current.metadata.as_ref().unwrap().resource_version; + + let invalid_version_sets: &[&[&str]] = &[&["latest"], &["2025-11-25", "2025-11-25"]]; + for versions in invalid_version_sets { + let error = handle_update_config( + &state, + with_user(Request::new(UpdateConfigRequest { + name: sandbox_name.to_string(), + policy: Some(mcp_policy_with_versions(versions)), + expected_resource_version: current_version, + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), + ..Default::default() + })), + ) + .await + .expect_err("invalid first-sync policy must fail before backfill"); + + assert_eq!(error.code(), Code::InvalidArgument); + let unchanged = state + .store + .get_message_by_name::("default", sandbox_name) + .await + .unwrap() + .unwrap(); + assert!(unchanged.spec.as_ref().unwrap().policy.is_none()); + assert!( + state + .store + .get_latest_policy(sandbox_id) + .await + .unwrap() + .is_none() + ); + } + } + + #[tokio::test] + async fn update_config_policy_backfill_persists_defaulted_mcp_versions_identically() { + use openshell_core::proto::{SandboxPhase, SandboxSpec}; + + let state = test_server_state().await; + let canonical_policy = mcp_policy_with_versions(&["2025-11-25"]); + let canonical_policy = validate_and_canonicalize_policy(canonical_policy) + .expect("explicit default MCP policy must canonicalize"); + let canonical_payload = canonical_policy.encode_to_vec(); + let canonical_hash = deterministic_policy_hash(&canonical_policy); + let cases = [ + ("omitted-options", mcp_policy_without_options()), + ("empty-versions", mcp_policy_with_versions(&[])), + ( + "explicit-default", + mcp_policy_with_versions(&["2025-11-25"]), + ), + ]; + + for (case, policy) in cases { + let sandbox_id = format!("sb-default-first-sync-{case}"); + let sandbox_name = format!("default-first-sync-{case}"); + let mut sandbox = Sandbox { + metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { + id: sandbox_id.clone(), + name: sandbox_name.clone(), + created_at_ms: 1_000_000, + labels: HashMap::new(), + resource_version: 0, + annotations: HashMap::new(), + workspace: "default".to_string(), + deletion_timestamp_ms: 0, + }), + spec: Some(SandboxSpec { + policy: None, + providers: Vec::new(), + ..Default::default() + }), + ..Default::default() + }; + sandbox.set_phase(SandboxPhase::Provisioning as i32); + state.store.put_message(&sandbox).await.unwrap(); + + let current = state + .store + .get_message_by_name::("default", &sandbox_name) + .await + .unwrap() + .unwrap(); + let current_version = current.metadata.as_ref().unwrap().resource_version; + let response = handle_update_config( + &state, + with_user(Request::new(UpdateConfigRequest { + name: sandbox_name.clone(), + policy: Some(policy), + expected_resource_version: current_version, + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), + ..Default::default() + })), + ) + .await + .expect("defaulted first-sync policy must persist") + .into_inner(); + + assert_eq!(response.version, 1, "{case}"); + assert_eq!(response.policy_hash, canonical_hash, "{case}"); + let stored = state + .store + .get_message_by_name::("default", &sandbox_name) + .await + .unwrap() + .unwrap(); + let stored_policy = stored + .spec + .as_ref() + .and_then(|spec| spec.policy.as_ref()) + .expect("backfilled sandbox policy"); + assert_eq!(stored_policy, &canonical_policy, "{case}"); + assert_eq!( + mcp_versions(stored_policy), + &["2025-11-25".to_string()], + "{case}" + ); + + let revision = state + .store + .get_latest_policy(&sandbox_id) + .await + .unwrap() + .expect("first-sync policy revision must exist"); + assert_eq!(revision.policy_payload, canonical_payload, "{case}"); + assert_eq!(revision.policy_hash, canonical_hash, "{case}"); + } + } + + #[tokio::test] + async fn update_config_policy_backfill_canonicalizes_mcp_versions_before_persistence() { + use openshell_core::proto::{SandboxPhase, SandboxSpec}; + + let state = test_server_state().await; + let sandbox_id = "sb-canonical-first-sync"; + let sandbox_name = "canonical-first-sync"; + let mut sandbox = Sandbox { + metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { + id: sandbox_id.to_string(), + name: sandbox_name.to_string(), + created_at_ms: 1_000_000, + labels: HashMap::new(), + resource_version: 0, + annotations: HashMap::new(), + workspace: "default".to_string(), + deletion_timestamp_ms: 0, + }), + spec: Some(SandboxSpec { + policy: None, + providers: Vec::new(), + ..Default::default() + }), + ..Default::default() + }; + sandbox.set_phase(SandboxPhase::Provisioning as i32); + state.store.put_message(&sandbox).await.unwrap(); + + let current = state + .store + .get_message_by_name::("default", sandbox_name) + .await + .unwrap() + .unwrap(); + let current_version = current.metadata.as_ref().unwrap().resource_version; + let canonical_policy = + mcp_policy_with_versions(&["2025-03-26", "2025-06-18", "2025-11-25"]); + let canonical_policy = validate_and_canonicalize_policy(canonical_policy) + .expect("canonical MCP policy must validate"); + + handle_update_config( + &state, + with_user(Request::new(UpdateConfigRequest { + name: sandbox_name.to_string(), + policy: Some(mcp_policy_with_versions(&[ + "2025-11-25", + "2025-06-18", + "2025-03-26", + ])), + expected_resource_version: current_version, + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), + ..Default::default() + })), + ) + .await + .expect("valid first-sync policy must persist"); + + let stored = state + .store + .get_message_by_name::("default", sandbox_name) + .await + .unwrap() + .unwrap(); + assert_eq!( + stored.spec.as_ref().and_then(|spec| spec.policy.as_ref()), + Some(&canonical_policy) + ); + let revision = state + .store + .get_latest_policy(sandbox_id) + .await + .unwrap() + .expect("first-sync policy revision must exist"); + assert_eq!(revision.policy_payload, canonical_policy.encode_to_vec()); + assert_eq!( + revision.policy_hash, + deterministic_policy_hash(&canonical_policy) + ); + } + #[tokio::test] async fn update_config_global_rejects_annotations() { let state = test_server_state().await; @@ -18202,6 +20099,7 @@ mod tests { name: "invalid-annotation".to_string(), policy: Some(ProtoSandboxPolicy::default()), annotations: HashMap::from([("bad key".to_string(), "value".to_string())]), + workspace_scope: Some(openshell_core::proto::workspace_selector("default")), ..Default::default() })), ) @@ -18234,6 +20132,7 @@ mod tests { "_provider_work_github", "api.github.com", )), + workspace_scope: Some(openshell_core::proto::workspace_selector("default")), ..Default::default() })), ) @@ -18300,6 +20199,7 @@ mod tests { name: "sync-strip".to_string(), policy: Some(synced_policy), expected_resource_version: current_version, + workspace_scope: Some(openshell_core::proto::workspace_selector("default")), ..Default::default() }), "sb-sync-strip", @@ -18401,7 +20301,9 @@ mod tests { merge_operations: vec![], expected_resource_version: 99, // stale version annotations: HashMap::new(), - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), }), ) .await @@ -18500,7 +20402,9 @@ mod tests { merge_operations: vec![], expected_resource_version: initial_version, annotations: HashMap::new(), - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), }), ) .await @@ -18582,7 +20486,7 @@ mod tests { let err = handle_get_sandbox_policy_status( &state, non_member_request(GetSandboxPolicyStatusRequest { - workspace: "no-such-ws".into(), + workspace_scope: Some(openshell_core::proto::workspace_selector("no-such-ws")), ..Default::default() }), ) @@ -18598,7 +20502,7 @@ mod tests { let err = handle_list_sandbox_policies( &state, non_member_request(ListSandboxPoliciesRequest { - workspace: "no-such-ws".into(), + workspace_scope: Some(openshell_core::proto::workspace_selector("no-such-ws")), ..Default::default() }), ) @@ -18614,7 +20518,7 @@ mod tests { let err = handle_update_config( &state, non_member_request(UpdateConfigRequest { - workspace: "no-such-ws".into(), + workspace_scope: Some(openshell_core::proto::workspace_selector("no-such-ws")), ..Default::default() }), ) @@ -18630,7 +20534,7 @@ mod tests { let err = handle_get_draft_policy( &state, non_member_request(GetDraftPolicyRequest { - workspace: "no-such-ws".into(), + workspace_scope: Some(openshell_core::proto::workspace_selector("no-such-ws")), ..Default::default() }), ) @@ -18646,7 +20550,7 @@ mod tests { let err = handle_approve_draft_chunk( &state, non_member_request(ApproveDraftChunkRequest { - workspace: "no-such-ws".into(), + workspace_scope: Some(openshell_core::proto::workspace_selector("no-such-ws")), ..Default::default() }), ) @@ -18662,7 +20566,7 @@ mod tests { let err = handle_reject_draft_chunk( &state, non_member_request(RejectDraftChunkRequest { - workspace: "no-such-ws".into(), + workspace_scope: Some(openshell_core::proto::workspace_selector("no-such-ws")), ..Default::default() }), ) @@ -18678,7 +20582,7 @@ mod tests { let err = handle_approve_all_draft_chunks( &state, non_member_request(ApproveAllDraftChunksRequest { - workspace: "no-such-ws".into(), + workspace_scope: Some(openshell_core::proto::workspace_selector("no-such-ws")), ..Default::default() }), ) @@ -18694,7 +20598,7 @@ mod tests { let err = handle_edit_draft_chunk( &state, non_member_request(EditDraftChunkRequest { - workspace: "no-such-ws".into(), + workspace_scope: Some(openshell_core::proto::workspace_selector("no-such-ws")), ..Default::default() }), ) @@ -18710,7 +20614,7 @@ mod tests { let err = handle_undo_draft_chunk( &state, non_member_request(UndoDraftChunkRequest { - workspace: "no-such-ws".into(), + workspace_scope: Some(openshell_core::proto::workspace_selector("no-such-ws")), ..Default::default() }), ) @@ -18726,7 +20630,7 @@ mod tests { let err = handle_clear_draft_chunks( &state, non_member_request(ClearDraftChunksRequest { - workspace: "no-such-ws".into(), + workspace_scope: Some(openshell_core::proto::workspace_selector("no-such-ws")), ..Default::default() }), ) @@ -18742,7 +20646,7 @@ mod tests { let err = handle_get_draft_history( &state, non_member_request(GetDraftHistoryRequest { - workspace: "no-such-ws".into(), + workspace_scope: Some(openshell_core::proto::workspace_selector("no-such-ws")), ..Default::default() }), ) @@ -18756,9 +20660,8 @@ mod tests { ); } - /// ID-based policy handlers must return `NOT_FOUND` — never - /// `PERMISSION_DENIED` — when the caller lacks workspace access, so that - /// cross-workspace sandbox existence cannot be inferred (CWE-203). + /// ID-only policy handlers hide cross-workspace resources, while requests + /// with an explicit workspace selector authorize that selector first. #[tokio::test] async fn id_based_policy_handlers_hide_cross_workspace_sandboxes() { let mut state = test_server_state().await; @@ -18807,6 +20710,7 @@ mod tests { &state, non_member_request(GetSandboxLogsRequest { sandbox_id: "sandbox-other".into(), + workspace_scope: Some(openshell_core::proto::workspace_selector("default")), ..Default::default() }), ) @@ -18814,8 +20718,8 @@ mod tests { .unwrap_err(); assert_eq!( err.code(), - Code::NotFound, - "handle_get_sandbox_logs must return NotFound, not PermissionDenied" + Code::PermissionDenied, + "handle_get_sandbox_logs must authorize the selected workspace before lookup" ); } diff --git a/crates/openshell-server/src/grpc/provider.rs b/crates/openshell-server/src/grpc/provider.rs index c8c8149a8..46eb9ce9b 100644 --- a/crates/openshell-server/src/grpc/provider.rs +++ b/crates/openshell-server/src/grpc/provider.rs @@ -14,12 +14,13 @@ use crate::provider_profile_sources::{ EffectiveProviderProfileCatalog, ProviderProfileSources, profile_response_payload, profile_storage_payload, stored_profile_resource_version, }; +use crate::storage_proto::{StoredProviderCredentialRefreshState, StoredProviderProfile}; use openshell_core::metadata::ObjectWorkspace; use openshell_core::proto::{ CredentialHandle, Provider, ProviderCredentialRefreshStrategy, ProviderCredentialTokenGrantAudienceOverride, ProviderCredentialTokenGrantType, ProviderProfile, ProviderProfileCredential, Sandbox, StaticCredentialBinding, - StaticCredentialEndpointBinding, StoredProviderCredentialRefreshState, + StaticCredentialEndpointBinding, }; use openshell_core::telemetry::{ LifecycleOperation, ProviderProfile as TelemetryProviderProfile, TelemetryOutcome, @@ -44,7 +45,7 @@ const GATEWAY_SPIFFE_WORKLOAD_API_SOCKET: &str = "OPENSHELL_GATEWAY_SPIFFE_WORKL /// Redact credential values from a provider before returning it in a gRPC /// response. Key names are preserved so callers can display credential counts -/// and key listings. Internal server paths (inference routing, sandbox env +/// and key listings. Internal server paths (sandbox env /// injection) read credentials from the store directly and are unaffected. fn redact_provider_credentials(mut provider: Provider) -> Provider { for value in provider.credentials.values_mut() { @@ -167,13 +168,18 @@ async fn create_provider_record_validating( "provider.credential_handles is internal gateway state and cannot be supplied", )); } + if !provider.profile_workspace.is_empty() && provider.profile_workspace != workspace { return Err(Status::invalid_argument( "profile_workspace must be empty (global) or match the provider workspace", )); } if provider.credentials.is_empty() - && !provider_type_allows_empty_credentials(catalog, &provider.r#type) + && !provider_type_allows_empty_credentials( + catalog, + &provider.r#type, + &provider.profile_workspace, + ) { return Err(Status::invalid_argument( "provider.credentials must not be empty", @@ -462,6 +468,13 @@ async fn update_provider_record_validating( let cas_result = async { validate_provider_mutable_fields(&candidate)?; + if let Some(profile) = get_provider_type_profile_for_scope( + catalog, + &candidate.r#type, + &candidate.profile_workspace, + ) { + validate_provider_credentials(&profile, &candidate, &updated_credential_values)?; + } validate_provider_update_against_attached_sandboxes_with_catalog( store, catalog, workspace, &candidate, ) @@ -645,6 +658,55 @@ where Ok(out) } +async fn providers_using_profile( + store: &Store, + workspace: &str, + profile_id: &str, +) -> Result, Status> { + let is_platform_scope = workspace.is_empty(); + let mut offset = 0u32; + let mut blocking = Vec::new(); + loop { + let records = if is_platform_scope { + store + .list_by_type(Provider::object_type(), 1000, offset) + .await + } else { + store + .list(Provider::object_type(), workspace, 1000, offset) + .await + } + .map_err(|e| Status::internal(format!("list providers failed: {e}")))?; + if records.is_empty() { + break; + } + offset = offset + .checked_add( + u32::try_from(records.len()) + .map_err(|_| Status::internal("provider page size exceeded u32"))?, + ) + .ok_or_else(|| Status::internal("provider pagination offset overflow"))?; + for record in records { + let provider = Provider::decode(record.payload.as_slice()) + .map_err(|e| Status::internal(format!("decode provider failed: {e}")))?; + if provider.profile_workspace != workspace + || normalize_profile_id(&provider.r#type).as_deref() != Some(profile_id) + { + continue; + } + let label = if is_platform_scope { + format!("{}/{}", provider.object_workspace(), provider.object_name()) + } else { + provider.object_name().to_string() + }; + blocking.push(label); + } + } + blocking.sort(); + blocking.dedup(); + Ok(blocking) +} + async fn sandboxes_using_provider( store: &Store, workspace: &str, @@ -1094,16 +1156,31 @@ pub(super) async fn resolve_provider_environment_from_records_with_policy_bindin let name = &record.name; let provider = &record.provider; let mut provider_env = HashMap::new(); - let profile_id = - normalize_provider_type(&provider.r#type).unwrap_or(provider.r#type.as_str()); - let profile = - get_provider_type_profile_for_scope(catalog, profile_id, &provider.profile_workspace); + let profile = get_provider_type_profile_for_scope( + catalog, + &provider.r#type, + &provider.profile_workspace, + ); + let accepted_stored_credential_keys = profile.as_ref().map(|profile| { + profile + .credentials + .iter() + .flat_map(|credential| credential.accepted_stored_keys()) + .map(str::to_string) + .collect::>() + }); + let profile_endpoints_are_active = profile + .as_ref() + .is_none_or(|profile| provider_profile_endpoints_are_active(profile, provider)); let profile_proto = profile.as_ref().map(ProviderTypeProfile::to_proto); let broker_only_credential_keys = profile_proto .as_ref() .map(broker_only_provider_credential_keys) .unwrap_or_default(); let profile_endpoints = profile_proto.as_ref().map(|profile| { + if !profile_endpoints_are_active { + return Vec::new(); + } profile .endpoints .iter() @@ -1143,6 +1220,17 @@ pub(super) async fn resolve_provider_environment_from_records_with_policy_bindin let refresh_epochs = refresh_authorization_epochs_by_key(record)?; for (key, value) in &provider.credentials { + if accepted_stored_credential_keys + .as_ref() + .is_some_and(|accepted| !accepted.contains(key)) + { + warn!( + provider_name = %name, + key = %key, + "withholding provider credential not declared by resolved profile" + ); + continue; + } if is_non_injectable_provider_credential(provider, key) || broker_only_credential_keys.contains(key) { @@ -1215,6 +1303,17 @@ pub(super) async fn resolve_provider_environment_from_records_with_policy_bindin .resolve_provider_handles(provider, now_ms) .await?; for (key, value) in resolved_refs.values { + if accepted_stored_credential_keys + .as_ref() + .is_some_and(|accepted| !accepted.contains(&key)) + { + warn!( + provider_name = %name, + key = %key, + "withholding provider credential handle not declared by resolved profile" + ); + continue; + } if is_non_injectable_provider_credential(provider, &key) || broker_only_credential_keys.contains(&key) { @@ -1274,7 +1373,7 @@ pub(super) async fn resolve_provider_environment_from_records_with_policy_bindin // provider's earlier output cannot change how this provider classifies // or populates its own keys. Cross-provider credential/config // collisions have already been rejected by the validation above. - registry.inject_env(provider, &mut provider_env); + inject_provider_plugin_environment(catalog, provider, ®istry, &mut provider_env); for (key, value) in provider_env { env.entry(key).or_insert(value); } @@ -1400,11 +1499,11 @@ fn resolve_dynamic_credentials_from_records( let mut dynamic_creds = HashMap::new(); for record in records { let provider = &record.provider; - let profile_id = - normalize_provider_type(&provider.r#type).unwrap_or(provider.r#type.as_str()); - let Some(profile) = - get_provider_type_profile_for_scope(catalog, profile_id, &provider.profile_workspace) - else { + let Some(profile) = get_provider_type_profile_for_scope( + catalog, + &provider.r#type, + &provider.profile_workspace, + ) else { continue; }; insert_dynamic_credentials_for_profile( @@ -1807,7 +1906,7 @@ async fn validate_provider_environment_keys_unique_at( &mut seen_plugin_config, &provider_name, active_provider_environment_keys(store, catalog, &provider, now_ms).await?, - provider_plugin_environment_keys(&provider), + provider_plugin_environment_keys(catalog, &provider), )?; dynamic_bindings.extend(dynamic_token_grant_bindings_for_provider_with_catalog( catalog, &provider, @@ -1840,7 +1939,7 @@ async fn validate_provider_environment_records_unique_at( now_ms, ) .await?, - provider_plugin_environment_keys(provider), + provider_plugin_environment_keys(catalog, provider), )?; dynamic_bindings.extend(dynamic_token_grant_bindings_for_provider_with_catalog( catalog, provider, @@ -1850,12 +1949,33 @@ async fn validate_provider_environment_records_unique_at( Ok(()) } -fn provider_plugin_environment_keys(provider: &Provider) -> Vec { +fn provider_plugin_environment_keys( + catalog: &EffectiveProviderProfileCatalog, + provider: &Provider, +) -> Vec { let mut plugin_environment = HashMap::new(); - openshell_providers::ProviderRegistry::new().inject_env(provider, &mut plugin_environment); + let registry = openshell_providers::ProviderRegistry::new(); + inject_provider_plugin_environment(catalog, provider, ®istry, &mut plugin_environment); plugin_environment.into_keys().collect() } +fn inject_provider_plugin_environment( + catalog: &EffectiveProviderProfileCatalog, + provider: &Provider, + registry: &openshell_providers::ProviderRegistry, + environment: &mut HashMap, +) { + if let Some(profile) = + get_provider_type_profile_for_scope(catalog, &provider.r#type, &provider.profile_workspace) + { + registry.inject_env_for_profile_id(provider, &profile.id, environment); + } else { + // Preserve config projection for legacy records when their profile + // source is temporarily unavailable. + registry.inject_env(provider, environment); + } +} + fn validate_provider_environment_key_ownership( seen_credentials: &mut HashMap, seen_plugin_config: &mut HashMap, @@ -1926,9 +2046,8 @@ fn dynamic_token_grant_bindings_for_provider_with_catalog( provider: &Provider, ) -> Vec { let provider_name = provider.object_name().to_string(); - let profile_id = normalize_provider_type(&provider.r#type).unwrap_or(provider.r#type.as_str()); let Some(profile) = - get_provider_type_profile_for_scope(catalog, profile_id, &provider.profile_workspace) + get_provider_type_profile_for_scope(catalog, &provider.r#type, &provider.profile_workspace) else { return Vec::new(); }; @@ -2141,8 +2260,7 @@ fn broker_only_provider_credential_keys_for_provider( catalog: &EffectiveProviderProfileCatalog, provider: &Provider, ) -> HashSet { - let profile_id = normalize_provider_type(&provider.r#type).unwrap_or(provider.r#type.as_str()); - get_provider_type_profile_for_scope(catalog, profile_id, &provider.profile_workspace) + get_provider_type_profile_for_scope(catalog, &provider.r#type, &provider.profile_workspace) .as_ref() .map(ProviderTypeProfile::to_proto) .map(|profile| broker_only_provider_credential_keys(&profile)) @@ -2176,8 +2294,7 @@ fn provider_credential_not_expired(provider: &Provider, key: &str, now_ms: i64) } fn is_non_injectable_provider_credential(provider: &Provider, key: &str) -> bool { - openshell_core::inference::normalize_inference_provider_type(&provider.r#type) - == Some("google-vertex-ai") + normalize_provider_type(&provider.r#type) == Some("google-vertex-ai") && key == "GOOGLE_SERVICE_ACCOUNT_KEY" } @@ -2218,8 +2335,7 @@ use openshell_core::proto::{ ListProviderProfilesResponse, ListProvidersRequest, ListProvidersResponse, ProviderProfileDiagnostic, ProviderProfileImportItem, ProviderProfileResponse, ProviderResponse, RotateProviderCredentialRequest, RotateProviderCredentialResponse, - StoredProviderProfile, UpdateProviderProfilesRequest, UpdateProviderProfilesResponse, - UpdateProviderRequest, + UpdateProviderProfilesRequest, UpdateProviderProfilesResponse, UpdateProviderRequest, }; use openshell_core::spiffe::{ JwtSvidParseError, SpiffeJwtClaims, parse_unverified_jwt_svid_claims, @@ -2234,7 +2350,10 @@ use std::sync::{Arc, LazyLock, RwLock}; use tonic::{Request, Response}; use crate::auth::principal::Principal; -use crate::auth::workspace_authz::{MinWorkspaceRole, authorize_workspace, require_platform_admin}; +use crate::auth::workspace_authz::{ + AuthorizedWorkspaceScope, MinWorkspaceRole, authorize_list_workspace_selector, + authorize_workspace, authorize_workspace_selector, require_platform_admin, +}; use openshell_core::oauth::{ self, TokenExchangeParams, effective_client_assertion_type, effective_token_type, }; @@ -2245,6 +2364,7 @@ const MAX_INTERMEDIATE_TOKEN_CACHE_ENTRIES: usize = 1024; static TOKEN_EXCHANGE_HTTP_CLIENT: LazyLock> = LazyLock::new(|| { + let _ = rustls::crypto::aws_lc_rs::default_provider().install_default(); reqwest::Client::builder() .timeout(std::time::Duration::from_secs(30)) .connect_timeout(std::time::Duration::from_secs(30)) @@ -2351,11 +2471,11 @@ pub(super) async fn handle_create_provider( ) -> Result, Status> { let principal = super::extract_principal(&request)?; let req = request.into_inner(); - let authz = authorize_workspace( + let authz = authorize_workspace_selector( &state.store, &state.admin_role, &principal, - &req.workspace, + req.workspace_scope.as_ref(), MinWorkspaceRole::Admin, ) .await?; @@ -2373,14 +2493,22 @@ pub(super) async fn handle_create_provider( if let Some(metadata) = provider.metadata.as_mut() { metadata.workspace.clone_from(&workspace); } - let provider_type = provider.r#type.clone(); - if state.credentials.stores_provider_credentials() && !provider.credentials.is_empty() { - state.compute.ensure_workspace(&workspace).await?; + if !provider.credential_handles.is_empty() { + return Err(Status::invalid_argument( + "provider.credential_handles is internal gateway state and cannot be supplied", + )); } + let provider_type = provider.r#type.clone(); + let _sandbox_sync_guard = state.compute.sandbox_sync_guard().await; let catalog = state .provider_profile_sources .snapshot_catalog(state.store.as_ref(), &workspace) .await?; + let profile = resolve_provider_create_profile(&catalog, &mut provider)?; + validate_provider_create_credentials(&profile, &provider)?; + if state.credentials.stores_provider_credentials() && !provider.credentials.is_empty() { + state.compute.ensure_workspace(&workspace).await?; + } let result = create_provider_record_validating( state.store.as_ref(), &workspace, @@ -2417,11 +2545,11 @@ pub(super) async fn handle_get_provider( ) -> Result, Status> { let principal = super::extract_principal(&request)?; let req = request.into_inner(); - let authz = authorize_workspace( + let authz = authorize_workspace_selector( &state.store, &state.admin_role, &principal, - &req.workspace, + req.workspace_scope.as_ref(), MinWorkspaceRole::User, ) .await?; @@ -2441,15 +2569,17 @@ pub(super) async fn handle_list_providers( ) -> Result, Status> { let principal = super::extract_principal(&request)?; let request = request.into_inner(); - if request.all_workspaces && !request.workspace.is_empty() { - return Err(Status::invalid_argument( - "all_workspaces and workspace are mutually exclusive", - )); - } let limit = clamp_limit(request.limit, 100, MAX_PAGE_SIZE); - let providers = if request.all_workspaces { - require_platform_admin(&state.admin_role, &principal)?; + let scope = authorize_list_workspace_selector( + &state.store, + &state.admin_role, + &principal, + request.workspace_scope.as_ref(), + MinWorkspaceRole::User, + ) + .await?; + let providers = if matches!(scope, AuthorizedWorkspaceScope::AllWorkspaces) { let all: Vec = state .store .list_all_messages(limit, request.offset) @@ -2457,14 +2587,9 @@ pub(super) async fn handle_list_providers( .map_err(|e| Status::internal(format!("list providers failed: {e}")))?; all.into_iter().map(redact_provider_credentials).collect() } else { - let authz = authorize_workspace( - &state.store, - &state.admin_role, - &principal, - &request.workspace, - MinWorkspaceRole::User, - ) - .await?; + let AuthorizedWorkspaceScope::Workspace(authz) = scope else { + unreachable!("all-workspaces scope handled above") + }; let workspace = super::workspace::resolve_workspace(state.store.as_ref(), &authz.workspace) .await? .name; @@ -2824,11 +2949,11 @@ pub(super) async fn handle_delete_provider_profile( return Err(Status::not_found("provider profile not found")); } - let blocking_sandboxes = sandboxes_using_profile(state.store.as_ref(), &workspace, &id).await?; - if !blocking_sandboxes.is_empty() { + let blocking_providers = providers_using_profile(state.store.as_ref(), &workspace, &id).await?; + if !blocking_providers.is_empty() { return Err(Status::failed_precondition(format!( - "provider profile '{id}' is in use by sandboxes: {}", - blocking_sandboxes.join(", ") + "provider profile '{id}' is in use by providers: {}", + blocking_providers.join(", ") ))); } @@ -2841,13 +2966,6 @@ pub(super) async fn handle_delete_provider_profile( Ok(Response::new(DeleteProviderProfileResponse { deleted })) } -pub(super) fn get_provider_type_profile_with_catalog( - catalog: &EffectiveProviderProfileCatalog, - id: &str, -) -> Option { - catalog.get_type_profile(id) -} - pub(super) fn get_provider_type_profile_for_scope( catalog: &EffectiveProviderProfileCatalog, id: &str, @@ -2856,6 +2974,36 @@ pub(super) fn get_provider_type_profile_for_scope( catalog.get_type_profile_for_scope(id, profile_workspace) } +/// Prevent a legacy alternate-upstream provider from binding its credential to +/// the built-in public vendor endpoint. Alternate endpoints must be expressed +/// by an explicitly imported endpoint-bearing profile. +pub(super) fn provider_profile_endpoints_are_active( + profile: &ProviderTypeProfile, + provider: &Provider, +) -> bool { + if profile.source != "builtin" { + return true; + } + + let (base_url_key, default_base_url) = match profile.id.as_str() { + "openai" => ("OPENAI_BASE_URL", "https://api.openai.com/v1"), + "anthropic" => ("ANTHROPIC_BASE_URL", "https://api.anthropic.com/v1"), + _ => return true, + }; + + provider + .config + .get(base_url_key) + .map(String::as_str) + .map(str::trim) + .filter(|value| !value.is_empty()) + .is_none_or(|configured| { + configured + .trim_end_matches('/') + .eq_ignore_ascii_case(default_base_url.trim_end_matches('/')) + }) +} + #[cfg(test)] pub(super) async fn get_provider_type_profile( store: &Store, @@ -3004,13 +3152,112 @@ fn validate_refresh_material( fn provider_type_allows_empty_credentials( catalog: &EffectiveProviderProfileCatalog, provider_type: &str, + profile_workspace: &str, ) -> bool { - let Some(profile) = get_provider_type_profile_with_catalog(catalog, provider_type) else { + let Some(profile) = + get_provider_type_profile_for_scope(catalog, provider_type, profile_workspace) + else { return false; }; profile.allows_empty_provider_credentials() } +fn resolve_provider_create_profile( + catalog: &EffectiveProviderProfileCatalog, + provider: &mut Provider, +) -> Result { + let requested_type = provider.r#type.trim(); + if requested_type.is_empty() { + return Err(Status::invalid_argument("provider.type is required")); + } + let profile = get_provider_type_profile_for_scope( + catalog, + requested_type, + &provider.profile_workspace, + ) + .ok_or_else(|| { + Status::invalid_argument(format!( + "provider profile '{requested_type}' was not found in the requested scope; import a matching profile before creating this provider" + )) + })?; + provider.r#type.clone_from(&profile.id); + Ok(profile) +} + +fn validate_provider_create_credentials( + profile: &ProviderTypeProfile, + provider: &Provider, +) -> Result<(), Status> { + validate_provider_credentials(profile, provider, &HashMap::new()) +} + +fn validate_provider_credentials( + profile: &ProviderTypeProfile, + provider: &Provider, + pending_credentials: &HashMap, +) -> Result<(), Status> { + let declared_keys = profile + .credentials + .iter() + .flat_map(|credential| credential.accepted_stored_keys()) + .collect::>(); + let mut unknown_keys = provider + .credentials + .keys() + .chain(provider.credential_handles.keys()) + .chain(pending_credentials.keys()) + .filter(|key| !declared_keys.contains(key.as_str())) + .cloned() + .collect::>(); + unknown_keys.sort(); + unknown_keys.dedup(); + if !unknown_keys.is_empty() { + return Err(Status::invalid_argument(format!( + "provider credentials are not declared by profile '{}': {}", + profile.id, + unknown_keys.join(", ") + ))); + } + + validate_required_static_credentials(profile, provider, pending_credentials) +} + +fn validate_required_static_credentials( + profile: &ProviderTypeProfile, + provider: &Provider, + pending_credentials: &HashMap, +) -> Result<(), Status> { + let mut missing = Vec::new(); + for credential in profile.required_static_credentials() { + let accepted_keys = credential.accepted_stored_keys(); + let supplied = accepted_keys.iter().any(|key| { + provider + .credentials + .get(*key) + .is_some_and(|value| !value.trim().is_empty()) + || provider.credential_handles.contains_key(*key) + || pending_credentials + .get(*key) + .is_some_and(|value| !value.trim().is_empty()) + }); + if !supplied { + missing.push( + accepted_keys + .first() + .map_or_else(|| credential.name.clone(), |key| (*key).to_string()), + ); + } + } + if !missing.is_empty() { + return Err(Status::invalid_argument(format!( + "provider profile '{}' requires static credentials: {}", + profile.id, + missing.join(", ") + ))); + } + Ok(()) +} + fn normalize_profile_id_request(id: &str) -> Result { if id.trim().is_empty() { return Err(Status::invalid_argument("id is required")); @@ -3261,8 +3508,15 @@ async fn profile_attached_sandbox_diagnostics( else { continue; }; - let profile_id = - normalize_provider_type(&provider.r#type).unwrap_or(provider.r#type.as_str()); + let requested_profile_id = normalize_profile_id(&provider.r#type) + .unwrap_or_else(|| provider.r#type.trim().to_string()); + let profile_id = if candidate_profiles.contains_key(&requested_profile_id) { + requested_profile_id + } else { + normalize_provider_type(&provider.r#type) + .filter(|alias| candidate_profiles.contains_key(*alias)) + .map_or(requested_profile_id, str::to_string) + }; let scope_mismatch = (is_platform_scope && !provider.profile_workspace.is_empty()) || (!is_platform_scope && provider.profile_workspace.is_empty()); if scope_mismatch { @@ -3272,7 +3526,7 @@ async fn profile_attached_sandbox_diagnostics( if validate_policy_composition && let Some(profile) = get_provider_type_profile_for_scope( catalog, - profile_id, + &provider.r#type, &provider.profile_workspace, ) { @@ -3284,7 +3538,7 @@ async fn profile_attached_sandbox_diagnostics( } continue; } - if let Some((source, profile)) = candidate_profiles.get(profile_id) { + if let Some((source, profile)) = candidate_profiles.get(&profile_id) { let has_static_credentials = provider .credentials .keys() @@ -3300,7 +3554,7 @@ async fn profile_attached_sandbox_diagnostics( if has_static_credentials && !has_usable_endpoint && !has_policy_binding { diagnostics.push(ProfileValidationDiagnostic { source: source.clone(), - profile_id: profile_id.to_string(), + profile_id: profile_id.clone(), field: "endpoints".to_string(), message: format!( "{operation} would leave static provider credentials without an authorized endpoint on sandbox '{sandbox_name}'" @@ -3311,7 +3565,7 @@ async fn profile_attached_sandbox_diagnostics( if has_usable_endpoint && has_policy_binding { diagnostics.push(ProfileValidationDiagnostic { source: source.clone(), - profile_id: profile_id.to_string(), + profile_id: profile_id.clone(), field: "endpoints".to_string(), message: format!( "{operation} would give provider '{provider_name}' both profile endpoint bindings and sandbox policy credential bindings on sandbox '{sandbox_name}'" @@ -3330,7 +3584,7 @@ async fn profile_attached_sandbox_diagnostics( rule_name, }); } - let used = (source.clone(), profile_id.to_string()); + let used = (source.clone(), profile_id.clone()); if !imported_profiles_used.contains(&used) { imported_profiles_used.push(used); } @@ -3341,7 +3595,7 @@ async fn profile_attached_sandbox_diagnostics( if validate_policy_composition && let Some(profile) = get_provider_type_profile_for_scope( catalog, - profile_id, + &provider.r#type, &provider.profile_workspace, ) { @@ -3437,83 +3691,17 @@ fn has_errors(diagnostics: &[ProfileValidationDiagnostic]) -> bool { .any(|diagnostic| diagnostic.severity == "error") } -async fn sandboxes_using_profile( - store: &Store, - workspace: &str, - profile_id: &str, -) -> Result, Status> { - let is_platform_scope = workspace.is_empty(); - - let candidates = if is_platform_scope { - scan_sandboxes_all(store, |sandbox| { - let has_providers = sandbox - .spec - .as_ref() - .is_some_and(|s| !s.providers.is_empty()); - has_providers.then_some(sandbox) - }) - .await? - } else { - scan_sandboxes(store, workspace, |sandbox| { - let has_providers = sandbox - .spec - .as_ref() - .is_some_and(|s| !s.providers.is_empty()); - has_providers.then_some(sandbox) - }) - .await? - }; - - let mut blocking = Vec::new(); - for sandbox in candidates { - let sandbox_workspace = sandbox.object_workspace().to_string(); - let spec = sandbox.spec.as_ref().expect("filtered by scan_sandboxes"); - for provider_name in &spec.providers { - let provider_ws = if is_platform_scope { - &sandbox_workspace - } else { - workspace - }; - let Some(provider) = store - .get_message_by_name::(provider_ws, provider_name) - .await - .map_err(|e| Status::internal(format!("fetch provider failed: {e}")))? - else { - continue; - }; - if is_platform_scope && !provider.profile_workspace.is_empty() { - continue; - } - if !is_platform_scope && provider.profile_workspace.is_empty() { - continue; - } - if normalize_profile_id(&provider.r#type).as_deref() == Some(profile_id) { - let label = if is_platform_scope { - format!("{}/{}", sandbox_workspace, sandbox.object_name()) - } else { - sandbox.object_name().to_string() - }; - blocking.push(label); - break; - } - } - } - blocking.sort(); - blocking.dedup(); - Ok(blocking) -} - pub(super) async fn handle_update_provider( state: &Arc, request: Request, ) -> Result, Status> { let principal = super::extract_principal(&request)?; let req = request.into_inner(); - let authz = authorize_workspace( + let authz = authorize_workspace_selector( &state.store, &state.admin_role, &principal, - &req.workspace, + req.workspace_scope.as_ref(), MinWorkspaceRole::Admin, ) .await?; @@ -3615,14 +3803,16 @@ pub(super) async fn handle_exchange_provider_subject_token( .await .map_err(|e| Status::internal(format!("fetch provider failed: {e}")))? .ok_or_else(|| Status::not_found("provider not found"))?; - let profile_id = normalize_provider_type(&provider.r#type).unwrap_or(provider.r#type.as_str()); let catalog = state .provider_profile_sources .snapshot_catalog(state.store.as_ref(), &workspace) .await?; - let profile = - get_provider_type_profile_for_scope(&catalog, profile_id, &provider.profile_workspace) - .ok_or_else(|| Status::failed_precondition("provider profile not found"))?; + let profile = get_provider_type_profile_for_scope( + &catalog, + &provider.r#type, + &provider.profile_workspace, + ) + .ok_or_else(|| Status::failed_precondition("provider profile not found"))?; let profile_proto = profile.to_proto(); let credential = profile_proto .credentials @@ -4038,11 +4228,11 @@ pub(super) async fn handle_get_provider_refresh_status( ) -> Result, Status> { let principal = super::extract_principal(&request)?; let request = request.into_inner(); - let authz = authorize_workspace( + let authz = authorize_workspace_selector( &state.store, &state.admin_role, &principal, - &request.workspace, + request.workspace_scope.as_ref(), MinWorkspaceRole::User, ) .await?; @@ -4091,11 +4281,11 @@ pub(super) async fn handle_configure_provider_refresh( ) -> Result, Status> { let principal = super::extract_principal(&request)?; let request = request.into_inner(); - let authz = authorize_workspace( + let authz = authorize_workspace_selector( &state.store, &state.admin_role, &principal, - &request.workspace, + request.workspace_scope.as_ref(), MinWorkspaceRole::Admin, ) .await?; @@ -4126,18 +4316,6 @@ pub(super) async fn handle_configure_provider_refresh( crate::provider_refresh::refresh_strategy_name(strategy as i32) ))); } - if strategy == ProviderCredentialRefreshStrategy::AwsStsAssumeRole { - let global_settings = - crate::grpc::policy::load_global_settings(state.store.as_ref()).await?; - if !crate::grpc::policy::bool_setting_enabled( - &global_settings, - openshell_core::settings::PROVIDERS_V2_ENABLED_KEY, - )? { - return Err(Status::failed_precondition( - "aws_sts_assume_role requires providers_v2_enabled=true", - )); - } - } if request.material.len() > MAX_PROVIDER_CONFIG_ENTRIES { return Err(Status::invalid_argument(format!( "material exceeds maximum entries ({} > {MAX_PROVIDER_CONFIG_ENTRIES})", @@ -4500,11 +4678,11 @@ pub(super) async fn handle_rotate_provider_credential( ) -> Result, Status> { let principal = super::extract_principal(&request)?; let request = request.into_inner(); - let authz = authorize_workspace( + let authz = authorize_workspace_selector( &state.store, &state.admin_role, &principal, - &request.workspace, + request.workspace_scope.as_ref(), MinWorkspaceRole::Admin, ) .await?; @@ -4570,11 +4748,11 @@ pub(super) async fn handle_delete_provider_refresh( ) -> Result, Status> { let principal = super::extract_principal(&request)?; let request = request.into_inner(); - let authz = authorize_workspace( + let authz = authorize_workspace_selector( &state.store, &state.admin_role, &principal, - &request.workspace, + request.workspace_scope.as_ref(), MinWorkspaceRole::Admin, ) .await?; @@ -4648,11 +4826,11 @@ pub(super) async fn handle_delete_provider( ) -> Result, Status> { let principal = super::extract_principal(&request)?; let req = request.into_inner(); - let authz = authorize_workspace( + let authz = authorize_workspace_selector( &state.store, &state.admin_role, &principal, - &req.workspace, + req.workspace_scope.as_ref(), MinWorkspaceRole::Admin, ) .await?; @@ -4749,17 +4927,17 @@ mod tests { use crate::grpc::{MAX_MAP_KEY_LEN, MAX_PROVIDER_TYPE_LEN}; use crate::persistence::test_store; use openshell_core::proto::{ - ConfigureProviderRefreshRequest, CreateProviderRequest, CreateWorkspaceRequest, - DeleteProviderProfileRequest, DeleteProviderRefreshRequest, DeleteProviderRequest, - GetProviderProfileRequest, GetProviderRefreshStatusRequest, GetProviderRequest, - ImportProviderProfilesRequest, L7Allow, L7Rule, LintProviderProfilesRequest, - ListProviderProfilesRequest, ListProvidersRequest, NetworkBinary, NetworkEndpoint, - NetworkPolicyRule, ProviderCredentialRefresh, ProviderCredentialRefreshMaterial, - ProviderCredentialRefreshRecoveryAction, ProviderCredentialTokenGrant, - ProviderCredentialTokenGrantAudienceOverride, ProviderProfile, ProviderProfileCategory, + AttachSandboxProviderRequest, ConfigureProviderRefreshRequest, CreateProviderRequest, + CreateWorkspaceRequest, DeleteProviderProfileRequest, DeleteProviderRefreshRequest, + DeleteProviderRequest, GetProviderProfileRequest, GetProviderRefreshStatusRequest, + GetProviderRequest, ImportProviderProfilesRequest, L7Allow, L7Rule, + LintProviderProfilesRequest, ListProviderProfilesRequest, ListProvidersRequest, + NetworkBinary, NetworkEndpoint, NetworkPolicyRule, ProviderCredentialRefresh, + ProviderCredentialRefreshMaterial, ProviderCredentialTokenGrant, + ProviderCredentialTokenGrantAudienceOverride, ProviderCredentialTokenGrantSubjectToken, + ProviderCredentialTokenGrantType, ProviderProfile, ProviderProfileCategory, ProviderProfileCredential, ProviderProfileImportItem, RotateProviderCredentialRequest, - Sandbox, SandboxPolicy, SandboxSpec, StoredProviderProfile, UpdateProviderProfilesRequest, - UpdateProviderRequest, + Sandbox, SandboxPolicy, SandboxSpec, UpdateProviderProfilesRequest, UpdateProviderRequest, }; use openshell_core::{ObjectId, ObjectName}; use tonic::{Code, Request}; @@ -4781,6 +4959,40 @@ mod tests { assert!(!is_valid_env_key("X;rm -rf /")); } + #[test] + fn create_validation_accepts_broker_only_credential_by_logical_name() { + let profile = ProviderTypeProfile::from_proto(&ProviderProfile { + id: "token-exchange".to_string(), + credentials: vec![ + ProviderProfileCredential { + name: "subject_token".to_string(), + required: true, + ..Default::default() + }, + ProviderProfileCredential { + name: "access_token".to_string(), + token_grant: Some(ProviderCredentialTokenGrant { + grant_type: ProviderCredentialTokenGrantType::TokenExchange as i32, + subject_token: Some(ProviderCredentialTokenGrantSubjectToken { + source: "provider_credential".to_string(), + credential: "subject_token".to_string(), + ..Default::default() + }), + ..Default::default() + }), + ..Default::default() + }, + ], + ..Default::default() + }); + let provider = Provider { + credentials: HashMap::from([("subject_token".to_string(), "test-token".to_string())]), + ..Default::default() + }; + + validate_provider_create_credentials(&profile, &provider).unwrap(); + } + #[test] fn telemetry_provider_profile_maps_unknown_to_custom() { assert_eq!( @@ -4797,11 +5009,11 @@ mod tests { ); assert_eq!( telemetry_provider_profile("glab"), - TelemetryProviderProfile::Gitlab + TelemetryProviderProfile::Custom ); assert_eq!( telemetry_provider_profile("outlook"), - TelemetryProviderProfile::Outlook + TelemetryProviderProfile::Custom ); assert_eq!( telemetry_provider_profile("generic"), @@ -5657,6 +5869,7 @@ mod tests { assert_eq!( ids, vec![ + "anthropic", "aws", "aws-bedrock", "aws-s3", @@ -5669,6 +5882,7 @@ mod tests { "google-cloud", "google-vertex-ai", "nvidia", + "openai", "pypi" ] ); @@ -5773,20 +5987,6 @@ mod tests { #[tokio::test] async fn profile_update_rejects_fanout_endpoint_ambiguity_without_persisting() { let state = test_server_state().await; - crate::grpc::policy::save_global_settings( - state.store.as_ref(), - &crate::grpc::StoredSettings { - revision: 1, - settings: std::iter::once(( - openshell_core::settings::PROVIDERS_V2_ENABLED_KEY.to_string(), - crate::grpc::StoredSettingValue::Bool(true), - )) - .collect(), - ..Default::default() - }, - ) - .await - .unwrap(); let mut initial_profile = custom_profile("fanout-ambiguity"); initial_profile.endpoints.push(NetworkEndpoint { @@ -6295,7 +6495,7 @@ mod tests { } #[tokio::test] - async fn delete_provider_profile_rejects_builtin_and_in_use_custom_profiles() { + async fn delete_provider_profile_rejects_builtin_and_provider_referenced_profiles() { let state = test_server_state().await; handle_import_provider_profiles( &state, @@ -6333,7 +6533,7 @@ mod tests { .put_message(&Sandbox { metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { id: "sandbox-id".to_string(), - name: "sandbox-using-custom".to_string(), + name: "sandbox-custom".to_string(), created_at_ms: 0, labels: HashMap::new(), resource_version: 0, @@ -6342,7 +6542,7 @@ mod tests { deletion_timestamp_ms: 0, }), spec: Some(SandboxSpec { - providers: vec!["custom-provider".to_string()], + providers: Vec::new(), ..Default::default() }), ..Default::default() @@ -6360,7 +6560,59 @@ mod tests { .await .unwrap_err(); assert_eq!(in_use_err.code(), Code::FailedPrecondition); - assert!(in_use_err.message().contains("sandbox-using-custom")); + assert!(in_use_err.message().contains("custom-provider")); + + let attached = super::super::sandbox::handle_attach_sandbox_provider( + &state, + authed_request(AttachSandboxProviderRequest { + sandbox_name: "sandbox-custom".to_string(), + provider_name: "custom-provider".to_string(), + expected_resource_version: 0, + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), + }), + ) + .await + .unwrap() + .into_inner(); + assert!(attached.attached); + } + + #[tokio::test] + async fn delete_global_provider_profile_checks_providers_in_every_workspace() { + let state = test_server_state().await; + state + .store + .put_message(&stored_provider_profile_for_workspace( + custom_profile("global-custom"), + "", + )) + .await + .unwrap(); + let catalog = state + .provider_profile_sources + .snapshot_catalog(state.store.as_ref(), "default") + .await + .unwrap(); + let mut provider = provider_with_values("global-provider", "global-custom"); + provider.profile_workspace = String::new(); + create_provider_record_with_catalog(state.store.as_ref(), &catalog, "default", provider) + .await + .unwrap(); + + let err = handle_delete_provider_profile( + &state, + authed_request(DeleteProviderProfileRequest { + id: "global-custom".to_string(), + workspace: String::new(), + }), + ) + .await + .unwrap_err(); + + assert_eq!(err.code(), Code::FailedPrecondition); + assert!(err.message().contains("default/global-provider")); } #[tokio::test] @@ -6413,7 +6665,9 @@ mod tests { // credential storage by omitting this advisory list. secret_material_keys: Vec::new(), expires_at_ms: Some(expires_at_ms), - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), }), ) .await @@ -6428,7 +6682,9 @@ mod tests { authed_request(GetProviderRefreshStatusRequest { provider: "msgraph".to_string(), credential_key: "MS_GRAPH_ACCESS_TOKEN".to_string(), - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), }), ) .await @@ -6495,7 +6751,9 @@ mod tests { ]), secret_material_keys: vec!["client_secret".to_string()], expires_at_ms: Some(expires_at_ms), - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), }), ) .await @@ -6520,7 +6778,9 @@ mod tests { authed_request(DeleteProviderRefreshRequest { provider: "msgraph".to_string(), credential_key: "MS_GRAPH_ACCESS_TOKEN".to_string(), - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), }), ) .await @@ -6534,7 +6794,9 @@ mod tests { authed_request(GetProviderRefreshStatusRequest { provider: "msgraph".to_string(), credential_key: "MS_GRAPH_ACCESS_TOKEN".to_string(), - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), }), ) .await @@ -6586,7 +6848,9 @@ mod tests { ]), secret_material_keys: vec!["client_secret".to_string()], expires_at_ms: None, - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), }; handle_configure_provider_refresh(&state, authed_request(request("original-secret"))) .await @@ -6706,7 +6970,9 @@ mod tests { ]), secret_material_keys: vec!["client_secret".to_string()], expires_at_ms: None, - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), }; let (first_store_hit, release_first_store) = first_state.credentials.gate_next_store(); @@ -6831,7 +7097,9 @@ mod tests { material: HashMap::new(), secret_material_keys: Vec::new(), expires_at_ms: Some(expires_at_ms), - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), }), ) .await @@ -6906,7 +7174,9 @@ mod tests { authed_request(DeleteProviderRefreshRequest { provider: "provider-a".to_string(), credential_key: "REFRESH_TOKEN".to_string(), - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), }), ) .await @@ -6975,7 +7245,9 @@ mod tests { ]), secret_material_keys: vec!["private_key".to_string()], expires_at_ms: None, - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), }), ) .await @@ -7041,7 +7313,9 @@ mod tests { ]), secret_material_keys: vec!["client_secret".to_string()], expires_at_ms: Some(refresh_expires_at_ms), - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), }), ) .await @@ -7081,7 +7355,9 @@ mod tests { authed_request(DeleteProviderRefreshRequest { provider: "msgraph".to_string(), credential_key: "MS_GRAPH_ACCESS_TOKEN".to_string(), - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), }), ) .await @@ -7105,17 +7381,7 @@ mod tests { #[tokio::test] async fn delete_aws_sts_refresh_clears_all_pinned_output_expiries() { - use crate::grpc::policy::set_global_bool_setting_for_test; - let state = test_server_state().await; - set_global_bool_setting_for_test( - state.store.as_ref(), - openshell_core::settings::PROVIDERS_V2_ENABLED_KEY, - true, - ) - .await - .unwrap(); - create_provider_record( state.store.as_ref(), "default", @@ -7154,7 +7420,9 @@ mod tests { )]), secret_material_keys: Vec::new(), expires_at_ms: Some(refresh_expires_at_ms), - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), }), ) .await @@ -7197,7 +7465,9 @@ mod tests { authed_request(DeleteProviderRefreshRequest { provider: "aws-delete".to_string(), credential_key: "AWS_ACCESS_KEY_ID".to_string(), - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), }), ) .await @@ -7375,7 +7645,9 @@ mod tests { ]), secret_material_keys: vec!["client_secret".to_string()], expires_at_ms: None, - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), }), ) .await @@ -7455,7 +7727,9 @@ mod tests { ]), secret_material_keys: vec!["client_secret".to_string()], expires_at_ms: None, - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), }), ) .await @@ -7474,7 +7748,9 @@ mod tests { ]), secret_material_keys: vec!["client_secret".to_string()], expires_at_ms: None, - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), }), ) .await @@ -7537,7 +7813,9 @@ mod tests { ]), secret_material_keys: vec!["client_secret".to_string()], expires_at_ms: None, - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), }), ) .await @@ -7554,7 +7832,9 @@ mod tests { material: HashMap::from([("tenant_id".to_string(), "tenant".to_string())]), secret_material_keys: vec!["client_secret".to_string()], expires_at_ms: None, - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), }), ) .await @@ -7608,7 +7888,9 @@ mod tests { material: HashMap::new(), secret_material_keys: Vec::new(), expires_at_ms: None, - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), }), ) .await @@ -7705,6 +7987,59 @@ mod tests { assert!(response.deleted); } + #[tokio::test] + async fn create_provider_waits_for_provider_profile_delete_guard() { + let state = test_server_state().await; + handle_import_provider_profiles( + &state, + authed_request(ImportProviderProfilesRequest { + profiles: vec![ProviderProfileImportItem { + profile: Some(custom_profile("guarded-create")), + source: "guarded-create.yaml".to_string(), + }], + workspace: "default".to_string(), + }), + ) + .await + .unwrap(); + + let guard = state.compute.sandbox_sync_guard().await; + let task_state = state.clone(); + let task = tokio::spawn(async move { + let mut provider = provider_with_values("guarded-provider", "guarded-create"); + provider.credentials.clear(); + provider.config.clear(); + handle_create_provider( + &task_state, + authed_request(CreateProviderRequest { + provider: Some(provider), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), + }), + ) + .await + }); + + tokio::time::sleep(std::time::Duration::from_millis(20)).await; + assert!( + !task.is_finished(), + "provider create should wait for provider profile mutation guard" + ); + drop(guard); + + let response = tokio::time::timeout(std::time::Duration::from_secs(5), task) + .await + .expect("create should finish after guard release") + .expect("join create task") + .expect("create should succeed") + .into_inner(); + assert_eq!( + response.provider.expect("provider").object_name(), + "guarded-provider" + ); + } + #[tokio::test] async fn provider_crud_round_trip_and_semantics() { let store = test_store().await; @@ -7969,7 +8304,7 @@ mod tests { create_provider_record( &store, "default", - provider_with_values("legacy-provider", "openai"), + provider_with_values("legacy-provider", "legacy-custom"), ) .await .unwrap(); @@ -8034,7 +8369,7 @@ mod tests { create_provider_record( &store, "default", - provider_with_values("legacy-provider", "openai"), + provider_with_values("legacy-provider", "legacy-custom"), ) .await .unwrap(); @@ -8114,7 +8449,9 @@ mod tests { "openai", "OPENAI_API_KEY", )), - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), }), ) .await @@ -8124,6 +8461,173 @@ mod tests { assert!(err.message().contains("internal gateway state")); } + #[tokio::test] + async fn handle_create_provider_rejects_profileless_type() { + let state = test_server_state().await; + let err = handle_create_provider( + &state, + authed_request(CreateProviderRequest { + provider: Some(provider_with_values("legacy-gitlab", "gitlab")), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), + }), + ) + .await + .unwrap_err(); + + assert_eq!(err.code(), Code::InvalidArgument); + assert!( + err.message() + .contains("provider profile 'gitlab' was not found") + ); + } + + #[tokio::test] + async fn handle_create_provider_allows_credentialless_policy_profile() { + let state = test_server_state().await; + let response = handle_create_provider( + &state, + authed_request(CreateProviderRequest { + provider: Some(Provider { + metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { + name: "pypi".to_string(), + workspace: "default".to_string(), + ..Default::default() + }), + r#type: "pypi".to_string(), + profile_workspace: "default".to_string(), + ..Default::default() + }), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), + }), + ) + .await + .unwrap() + .into_inner() + .provider + .expect("provider"); + + assert_eq!(response.r#type, "pypi"); + assert!(response.credentials.is_empty()); + } + + #[tokio::test] + async fn handle_create_provider_allows_retired_type_with_imported_profile() { + let state = test_server_state().await; + handle_import_provider_profiles( + &state, + authed_request(ImportProviderProfilesRequest { + profiles: vec![ProviderProfileImportItem { + profile: Some(custom_profile("gitlab")), + source: "custom-gitlab.yaml".to_string(), + }], + workspace: "default".to_string(), + }), + ) + .await + .unwrap(); + + let response = handle_create_provider( + &state, + authed_request(CreateProviderRequest { + provider: Some(Provider { + metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { + name: "private-gitlab".to_string(), + workspace: "default".to_string(), + ..Default::default() + }), + r#type: "gitlab".to_string(), + profile_workspace: "default".to_string(), + ..Default::default() + }), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), + }), + ) + .await + .unwrap() + .into_inner() + .provider + .expect("provider"); + + assert_eq!(response.r#type, "gitlab"); + } + + #[tokio::test] + async fn handle_create_provider_prefers_exact_imported_alias_profile() { + let mut state = test_server_state().await; + let config = state + .config + .clone() + .with_credential_drivers(["test-static"]); + let credentials = crate::credentials::CredentialRuntime::from_config(&config).unwrap(); + let state_mut = Arc::get_mut(&mut state).unwrap(); + state_mut.config = config; + state_mut.credentials = credentials; + + let mut profile = custom_profile("gh"); + profile.credentials = vec![static_credential("token", "GITHUB_TOKEN", true)]; + profile.endpoints = vec![NetworkEndpoint { + host: "github.enterprise.example".to_string(), + port: 443, + allow_uninspected_credentials: true, + ..Default::default() + }]; + let imported = handle_import_provider_profiles( + &state, + authed_request(ImportProviderProfilesRequest { + profiles: vec![ProviderProfileImportItem { + profile: Some(profile), + source: "enterprise-github.yaml".to_string(), + }], + workspace: "default".to_string(), + }), + ) + .await + .unwrap() + .into_inner(); + assert!(imported.imported, "diagnostics: {:?}", imported.diagnostics); + + let provider = handle_create_provider( + &state, + authed_request(CreateProviderRequest { + provider: Some(provider_with_credential_value( + "enterprise-github", + "gh", + "GITHUB_TOKEN", + "test-token", + )), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), + }), + ) + .await + .unwrap() + .into_inner() + .provider + .expect("provider"); + + assert_eq!(provider.r#type, "gh"); + let catalog = state + .provider_profile_sources + .snapshot_catalog(state.store.as_ref(), "default") + .await + .unwrap(); + let resolved = get_provider_type_profile_for_scope( + &catalog, + &provider.r#type, + &provider.profile_workspace, + ) + .expect("exact imported profile"); + assert_eq!(resolved.id, "gh"); + assert_eq!(resolved.endpoints[0].host, "github.enterprise.example"); + } + #[tokio::test] async fn handle_create_provider_stores_inline_credentials_with_enabled_driver() { let mut state = test_server_state().await; @@ -8145,7 +8649,9 @@ mod tests { "OPENAI_API_KEY", "sk-test", )), - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), }), ) .await @@ -8188,6 +8694,96 @@ mod tests { assert_eq!(result.get("OPENAI_API_KEY"), Some(&"sk-test".to_string())); } + #[tokio::test] + async fn handle_create_provider_accepts_broker_only_subject_token() { + let mut state = test_server_state().await; + let config = state + .config + .clone() + .with_credential_drivers(["test-static"]); + let credentials = crate::credentials::CredentialRuntime::from_config(&config).unwrap(); + let state_mut = Arc::get_mut(&mut state).unwrap(); + state_mut.config = config; + state_mut.credentials = credentials; + + let mut profile = custom_profile("spiffe-token-exchange-demo"); + profile.credentials = vec![ + ProviderProfileCredential { + name: "subject_token".to_string(), + description: "Broker-only subject token".to_string(), + required: true, + ..Default::default() + }, + ProviderProfileCredential { + name: "access_token".to_string(), + required: false, + auth_style: "bearer".to_string(), + header_name: "Authorization".to_string(), + token_grant: Some(ProviderCredentialTokenGrant { + grant_type: ProviderCredentialTokenGrantType::TokenExchange as i32, + token_endpoint: "https://issuer.example.com/token".to_string(), + jwt_svid_audience: "https://issuer.example.com".to_string(), + client_assertion_type: "urn:ietf:params:oauth:client-assertion-type:jwt-bearer" + .to_string(), + subject_token: Some(ProviderCredentialTokenGrantSubjectToken { + source: "provider_credential".to_string(), + credential: "subject_token".to_string(), + subject_token_type: "urn:ietf:params:oauth:token-type:access_token" + .to_string(), + }), + ..Default::default() + }), + ..Default::default() + }, + ]; + profile.endpoints = vec![NetworkEndpoint { + host: "api.example.com".to_string(), + port: 443, + allow_uninspected_credentials: true, + ..Default::default() + }]; + let imported = handle_import_provider_profiles( + &state, + authed_request(ImportProviderProfilesRequest { + profiles: vec![ProviderProfileImportItem { + profile: Some(profile), + source: "provider-profile.yaml".to_string(), + }], + workspace: "default".to_string(), + }), + ) + .await + .unwrap() + .into_inner(); + assert!(imported.imported, "diagnostics: {:?}", imported.diagnostics); + + handle_create_provider( + &state, + authed_request(CreateProviderRequest { + provider: Some(provider_with_credential_value( + "exchange", + "spiffe-token-exchange-demo", + "subject_token", + "test-token", + )), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), + }), + ) + .await + .unwrap(); + + let stored: Provider = state + .store + .get_message_by_name("default", "exchange") + .await + .unwrap() + .unwrap(); + assert!(stored.credentials.is_empty()); + assert!(stored.credential_handles.contains_key("subject_token")); + } + #[tokio::test] async fn handle_update_provider_rejects_user_supplied_credential_handles() { let state = test_server_state().await; @@ -8208,7 +8804,9 @@ mod tests { "OPENAI_API_KEY", )), credential_expires_at_ms: HashMap::new(), - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), }), ) .await @@ -8316,7 +8914,7 @@ mod tests { let store = test_store().await; // Create provider and verify resource_version: 1 in response - let created = provider_with_values("test-provider", "openai"); + let created = provider_with_values("test-provider", "legacy-custom"); let persisted = create_provider_record(&store, "default", created) .await .unwrap(); @@ -8341,7 +8939,7 @@ mod tests { workspace: "default".to_string(), deletion_timestamp_ms: 0, }), - r#type: "openai".to_string(), + r#type: "legacy-custom".to_string(), credentials: std::iter::once(( "OPENAI_API_KEY".to_string(), "updated-key".to_string(), @@ -8376,7 +8974,7 @@ mod tests { workspace: "default".to_string(), deletion_timestamp_ms: 0, }), - r#type: "openai".to_string(), + r#type: "legacy-custom".to_string(), credentials: std::iter::once(( "OPENAI_API_KEY".to_string(), "third-key".to_string(), @@ -8690,7 +9288,7 @@ mod tests { async fn update_provider_empty_maps_is_noop() { let store = test_store().await; - let created = provider_with_values("noop-test", "nvidia"); + let created = provider_with_values("noop-test", "legacy-custom"); let persisted = create_provider_record(&store, "default", created) .await .unwrap(); @@ -8721,7 +9319,7 @@ mod tests { .unwrap(); assert_eq!(updated.object_id(), persisted.object_id()); - assert_eq!(updated.r#type, "nvidia"); + assert_eq!(updated.r#type, "legacy-custom"); assert_eq!(updated.credentials.len(), 2); assert_eq!( updated.credentials.get("API_TOKEN"), @@ -8745,7 +9343,7 @@ mod tests { async fn update_provider_empty_value_deletes_key() { let store = test_store().await; - let created = provider_with_values("delete-key-test", "openai"); + let created = provider_with_values("delete-key-test", "legacy-custom"); create_provider_record(&store, "default", created) .await .unwrap(); @@ -8804,7 +9402,7 @@ mod tests { async fn update_provider_empty_type_preserves_existing() { let store = test_store().await; - let created = provider_with_values("type-preserve-test", "anthropic"); + let created = provider_with_values("type-preserve-test", "legacy-custom"); create_provider_record(&store, "default", created) .await .unwrap(); @@ -8834,7 +9432,7 @@ mod tests { .await .unwrap(); - assert_eq!(updated.r#type, "anthropic"); + assert_eq!(updated.r#type, "legacy-custom"); } #[tokio::test] @@ -9170,7 +9768,9 @@ mod tests { ]), secret_material_keys: vec!["client_secret".to_string()], expires_at_ms: None, - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), }; handle_configure_provider_refresh(&state, authed_request(configure())) .await @@ -9235,7 +9835,9 @@ mod tests { ..Default::default() }), credential_expires_at_ms: HashMap::new(), - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), }), ) .await @@ -9486,6 +10088,151 @@ mod tests { ); } + #[tokio::test] + async fn resolve_provider_env_withholds_undeclared_legacy_credentials() { + let store = test_store().await; + let mut legacy = provider_with_values("legacy-openai", "openai"); + legacy.credentials = HashMap::from([ + ("OPENAI_API_KEY".to_string(), "openai-key".to_string()), + ( + "AWS_SECRET_ACCESS_KEY".to_string(), + "unrelated-secret".to_string(), + ), + ]); + create_provider_record(&store, "default", legacy) + .await + .unwrap(); + + let result = + resolve_provider_environment(&store, "default", &["legacy-openai".to_string()]) + .await + .unwrap(); + + assert_eq!( + result.get("OPENAI_API_KEY"), + Some(&"openai-key".to_string()) + ); + assert!( + result + .static_credential_bindings + .contains_key("OPENAI_API_KEY") + ); + assert!(!result.contains_key("AWS_SECRET_ACCESS_KEY")); + assert!( + !result + .static_credential_bindings + .contains_key("AWS_SECRET_ACCESS_KEY") + ); + } + + #[tokio::test] + async fn resolve_provider_env_withholds_undeclared_legacy_credential_handles() { + let store = test_store().await; + let config = openshell_core::Config::new(None).with_credential_drivers(["test-static"]); + let credentials = crate::credentials::CredentialRuntime::from_config(&config).unwrap(); + let catalog = ProviderProfileSources::with_default_sources() + .snapshot_catalog(&store, "default") + .await + .unwrap(); + create_provider_record_validating( + &store, + "default", + &catalog, + provider_with_credential_value( + "legacy-openai-handle", + "openai", + "AWS_SECRET_ACCESS_KEY", + "unrelated-secret", + ), + Some(&credentials), + ) + .await + .unwrap(); + + let records = load_provider_environment_records( + &store, + "default", + &["legacy-openai-handle".to_string()], + ) + .await + .unwrap(); + assert!(records[0].provider.credentials.is_empty()); + assert!( + records[0] + .provider + .credential_handles + .contains_key("AWS_SECRET_ACCESS_KEY") + ); + + let result = + resolve_provider_environment_from_records_with_policy_bindings_and_credentials( + &store, + &catalog, + &records, + &HashMap::new(), + &credentials, + None, + ) + .await + .unwrap(); + assert!(!result.contains_key("AWS_SECRET_ACCESS_KEY")); + assert!( + !result + .static_credential_bindings + .contains_key("AWS_SECRET_ACCESS_KEY") + ); + } + + #[tokio::test] + async fn resolve_provider_env_does_not_bind_alternate_upstream_keys_to_public_vendors() { + let store = test_store().await; + let cases = [ + ( + "alternate-openai", + "openai", + "OPENAI_API_KEY", + "OPENAI_BASE_URL", + ), + ( + "alternate-anthropic", + "anthropic", + "ANTHROPIC_API_KEY", + "ANTHROPIC_BASE_URL", + ), + ]; + for (name, provider_type, credential_key, base_url_key) in cases { + let mut provider = + provider_with_credential_value(name, provider_type, credential_key, "private-key"); + provider.config.insert( + base_url_key.to_string(), + "https://api.example.com/v1".to_string(), + ); + create_provider_record(&store, "default", provider) + .await + .unwrap(); + } + + let result = resolve_provider_environment( + &store, + "default", + &[ + "alternate-openai".to_string(), + "alternate-anthropic".to_string(), + ], + ) + .await + .unwrap(); + + for credential_key in ["OPENAI_API_KEY", "ANTHROPIC_API_KEY"] { + assert!(!result.contains_key(credential_key)); + assert!( + !result + .static_credential_bindings + .contains_key(credential_key) + ); + } + } + #[tokio::test] async fn resolve_provider_env_binds_endpointless_profile_from_policy() { let store = test_store().await; @@ -10350,10 +11097,11 @@ mod tests { .await .unwrap(); - // Credential value wins over the injected static value. + // Profile-authoritative runtime projection withholds the undeclared + // credential, so the provider plugin's declared config wins. assert_eq!( result.get("GOOSE_PROVIDER"), - Some(&"custom-value".to_string()) + Some(&"gcp_vertex_ai".to_string()) ); } @@ -10683,6 +11431,7 @@ mod tests { ..SandboxSpec::default() }), status: None, + ..Sandbox::default() }; sandbox.set_phase(SandboxPhase::Ready as i32); store.put_message(&sandbox).await.unwrap(); @@ -10719,6 +11468,7 @@ mod tests { }), spec: Some(SandboxSpec::default()), status: None, + ..Sandbox::default() }; sandbox.set_phase(SandboxPhase::Ready as i32); store.put_message(&sandbox).await.unwrap(); @@ -10889,18 +11639,120 @@ mod tests { // ---- CAS (Client-driven optimistic concurrency) tests for UpdateProvider ---- + #[tokio::test] + async fn update_provider_rejects_undeclared_profile_credential() { + let state = test_server_state().await; + handle_create_provider( + &state, + authed_request(CreateProviderRequest { + provider: Some(provider_with_credential_value( + "profile-backed-openai", + "openai", + "OPENAI_API_KEY", + "sk-test", + )), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), + }), + ) + .await + .unwrap(); + + let current = state + .store + .get_message_by_name::("default", "profile-backed-openai") + .await + .unwrap() + .unwrap(); + let mut update = current; + update.credential_handles.clear(); + update.credentials.insert( + "AWS_SECRET_ACCESS_KEY".to_string(), + "unrelated-secret".to_string(), + ); + + let error = handle_update_provider( + &state, + authed_request(UpdateProviderRequest { + provider: Some(update), + credential_expires_at_ms: HashMap::new(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), + }), + ) + .await + .unwrap_err(); + + assert_eq!(error.code(), Code::InvalidArgument); + assert!(error.message().contains("AWS_SECRET_ACCESS_KEY")); + } + + #[tokio::test] + async fn update_provider_rejects_deleting_required_profile_credential() { + let state = test_server_state().await; + handle_create_provider( + &state, + authed_request(CreateProviderRequest { + provider: Some(provider_with_credential_value( + "required-openai", + "openai", + "OPENAI_API_KEY", + "sk-test", + )), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), + }), + ) + .await + .unwrap(); + + let current = state + .store + .get_message_by_name::("default", "required-openai") + .await + .unwrap() + .unwrap(); + let mut update = current; + update.credential_handles.clear(); + update + .credentials + .insert("OPENAI_API_KEY".to_string(), String::new()); + + let error = handle_update_provider( + &state, + authed_request(UpdateProviderRequest { + provider: Some(update), + credential_expires_at_ms: HashMap::new(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), + }), + ) + .await + .unwrap_err(); + + assert_eq!(error.code(), Code::InvalidArgument); + assert!(error.message().contains("requires static credentials")); + } + #[tokio::test] async fn update_provider_client_driven_cas_succeeds_with_correct_version() { let state = test_server_state().await; // Create a provider - let mut provider = provider_with_values("test-provider", "generic"); + let mut provider = + provider_with_credential_value("test-provider", "openai", "OPENAI_API_KEY", "sk-test"); provider.metadata.as_mut().unwrap().id = String::new(); handle_create_provider( &state, authed_request(CreateProviderRequest { provider: Some(provider.clone()), - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), }), ) .await @@ -10919,8 +11771,8 @@ mod tests { let mut updated_provider = current.clone(); updated_provider.credential_handles.clear(); updated_provider - .credentials - .insert("NEW_KEY".to_string(), "new-value".to_string()); + .config + .insert("NEW_CONFIG".to_string(), "new-value".to_string()); updated_provider.metadata.as_mut().unwrap().resource_version = current_version; // Update should succeed @@ -10929,7 +11781,9 @@ mod tests { authed_request(UpdateProviderRequest { provider: Some(updated_provider.clone()), credential_expires_at_ms: HashMap::new(), - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), }), ) .await @@ -10951,13 +11805,7 @@ mod tests { .resource_version, current_version + 1 ); - assert!( - response - .provider - .unwrap() - .credentials - .contains_key("NEW_KEY") - ); + assert!(response.provider.unwrap().config.contains_key("NEW_CONFIG")); } #[tokio::test] @@ -10965,13 +11813,16 @@ mod tests { let state = test_server_state().await; // Create a provider - let mut provider = provider_with_values("test-provider", "generic"); + let mut provider = + provider_with_credential_value("test-provider", "openai", "OPENAI_API_KEY", "sk-test"); provider.metadata.as_mut().unwrap().id = String::new(); handle_create_provider( &state, authed_request(CreateProviderRequest { provider: Some(provider.clone()), - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), }), ) .await @@ -10990,8 +11841,8 @@ mod tests { let mut stale_provider = current.clone(); stale_provider.credential_handles.clear(); stale_provider - .credentials - .insert("NEW_KEY".to_string(), "new-value".to_string()); + .config + .insert("NEW_CONFIG".to_string(), "new-value".to_string()); stale_provider.metadata.as_mut().unwrap().resource_version = 99; // stale version // Update should fail with ABORTED @@ -11000,7 +11851,9 @@ mod tests { authed_request(UpdateProviderRequest { provider: Some(stale_provider), credential_expires_at_ms: HashMap::new(), - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), }), ) .await @@ -11025,8 +11878,7 @@ mod tests { unchanged.metadata.as_ref().unwrap().resource_version, current_version ); - assert!(!unchanged.credentials.contains_key("NEW_KEY")); - assert!(!unchanged.credential_handles.contains_key("NEW_KEY")); + assert!(!unchanged.config.contains_key("NEW_CONFIG")); } #[tokio::test] @@ -11050,7 +11902,9 @@ mod tests { "OPENAI_API_KEY", "sk-first", )), - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), }), ) .await @@ -11074,7 +11928,9 @@ mod tests { authed_request(UpdateProviderRequest { provider: Some(stale_provider), credential_expires_at_ms: HashMap::new(), - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), }), ) .await @@ -11107,13 +11963,16 @@ mod tests { let state = Arc::new(test_server_state().await); // Create a provider - let mut provider = provider_with_values("test-provider", "generic"); + let mut provider = + provider_with_credential_value("test-provider", "openai", "OPENAI_API_KEY", "sk-test"); provider.metadata.as_mut().unwrap().id = String::new(); handle_create_provider( &state, authed_request(CreateProviderRequest { provider: Some(provider.clone()), - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), }), ) .await @@ -11135,7 +11994,7 @@ mod tests { let mut updated = initial.clone(); updated.credential_handles.clear(); updated - .credentials + .config .insert(format!("KEY_{i}"), format!("value-{i}")); updated.metadata.as_mut().unwrap().resource_version = initial_version; @@ -11145,7 +12004,9 @@ mod tests { authed_request(UpdateProviderRequest { provider: Some(updated), credential_expires_at_ms: HashMap::new(), - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), }), ) .await @@ -11187,91 +12048,16 @@ mod tests { initial_version + 1 ); - // Exactly one of KEY_0, KEY_1, or KEY_2 should be present + // Exactly one of KEY_0, KEY_1, or KEY_2 should be present. let new_keys_count = (0..3) - .filter(|i| { - final_provider - .credential_handles - .contains_key(&format!("KEY_{i}")) - }) + .filter(|i| final_provider.config.contains_key(&format!("KEY_{i}"))) .count(); assert_eq!(new_keys_count, 1); } #[tokio::test] - async fn configure_aws_sts_requires_v2_enabled() { + async fn configure_aws_sts_succeeds_without_feature_gate() { let state = test_server_state().await; - create_provider_record( - state.store.as_ref(), - "default", - Provider { - metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { - id: String::new(), - name: "my-aws".to_string(), - created_at_ms: 0, - labels: HashMap::new(), - resource_version: 0, - annotations: HashMap::new(), - workspace: "default".to_string(), - deletion_timestamp_ms: 0, - }), - r#type: "aws".to_string(), - credentials: std::iter::once(( - "AWS_ACCESS_KEY_ID".to_string(), - "placeholder".to_string(), - )) - .collect(), - config: HashMap::new(), - credential_expires_at_ms: HashMap::new(), - profile_workspace: "default".to_string(), - credential_handles: HashMap::new(), - }, - ) - .await - .unwrap(); - - let err = handle_configure_provider_refresh( - &state, - authed_request(ConfigureProviderRefreshRequest { - provider: "my-aws".to_string(), - credential_key: "AWS_ACCESS_KEY_ID".to_string(), - strategy: ProviderCredentialRefreshStrategy::AwsStsAssumeRole as i32, - material: HashMap::from([( - "role_arn".to_string(), - "arn:aws:iam::123456789012:role/Test".to_string(), - )]), - secret_material_keys: Vec::new(), - expires_at_ms: None, - workspace: "default".to_string(), - }), - ) - .await - .unwrap_err(); - - assert_eq!(err.code(), Code::FailedPrecondition); - assert!(err.message().contains("providers_v2_enabled")); - } - - #[tokio::test] - async fn configure_aws_sts_succeeds_with_v2_enabled() { - use crate::grpc::StoredSettingValue; - use crate::grpc::StoredSettings; - use crate::grpc::policy::save_global_settings; - - let state = test_server_state().await; - - let global_settings = StoredSettings { - revision: 1, - settings: std::iter::once(( - openshell_core::settings::PROVIDERS_V2_ENABLED_KEY.to_string(), - StoredSettingValue::Bool(true), - )) - .collect(), - ..Default::default() - }; - save_global_settings(state.store.as_ref(), &global_settings) - .await - .unwrap(); create_provider_record( state.store.as_ref(), @@ -11314,7 +12100,9 @@ mod tests { )]), secret_material_keys: Vec::new(), expires_at_ms: None, - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), }), ) .await @@ -11332,24 +12120,7 @@ mod tests { #[tokio::test] async fn configure_aws_sts_rejects_endpoint_override_material() { - use crate::grpc::StoredSettingValue; - use crate::grpc::StoredSettings; - use crate::grpc::policy::save_global_settings; - let state = test_server_state().await; - let global_settings = StoredSettings { - revision: 1, - settings: std::iter::once(( - openshell_core::settings::PROVIDERS_V2_ENABLED_KEY.to_string(), - StoredSettingValue::Bool(true), - )) - .collect(), - ..Default::default() - }; - save_global_settings(state.store.as_ref(), &global_settings) - .await - .unwrap(); - create_provider_record( state.store.as_ref(), "default", @@ -11393,7 +12164,9 @@ mod tests { ]), secret_material_keys: Vec::new(), expires_at_ms: None, - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), }), ) .await @@ -11424,24 +12197,7 @@ mod tests { #[tokio::test] async fn configure_aws_sts_rejects_partial_source_credentials() { - use crate::grpc::StoredSettingValue; - use crate::grpc::StoredSettings; - use crate::grpc::policy::save_global_settings; - let state = test_server_state().await; - let global_settings = StoredSettings { - revision: 1, - settings: std::iter::once(( - openshell_core::settings::PROVIDERS_V2_ENABLED_KEY.to_string(), - StoredSettingValue::Bool(true), - )) - .collect(), - ..Default::default() - }; - save_global_settings(state.store.as_ref(), &global_settings) - .await - .unwrap(); - create_provider_record( state.store.as_ref(), "default", @@ -11484,7 +12240,9 @@ mod tests { ]), secret_material_keys: Vec::new(), expires_at_ms: None, - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), }), ) .await @@ -11496,17 +12254,7 @@ mod tests { #[tokio::test] async fn configure_aws_sts_rejects_session_token_without_pair() { - use crate::grpc::policy::set_global_bool_setting_for_test; - let state = test_server_state().await; - set_global_bool_setting_for_test( - state.store.as_ref(), - openshell_core::settings::PROVIDERS_V2_ENABLED_KEY, - true, - ) - .await - .unwrap(); - create_provider_record( state.store.as_ref(), "default", @@ -11550,7 +12298,9 @@ mod tests { ]), secret_material_keys: vec!["aws_session_token".to_string()], expires_at_ms: None, - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), }), ) .await @@ -11562,24 +12312,7 @@ mod tests { #[tokio::test] async fn configure_aws_sts_persists_resolved_additional_output_keys() { - use crate::grpc::StoredSettingValue; - use crate::grpc::StoredSettings; - use crate::grpc::policy::save_global_settings; - let state = test_server_state().await; - let global_settings = StoredSettings { - revision: 1, - settings: std::iter::once(( - openshell_core::settings::PROVIDERS_V2_ENABLED_KEY.to_string(), - StoredSettingValue::Bool(true), - )) - .collect(), - ..Default::default() - }; - save_global_settings(state.store.as_ref(), &global_settings) - .await - .unwrap(); - create_provider_record( state.store.as_ref(), "default", @@ -11617,7 +12350,9 @@ mod tests { )]), secret_material_keys: Vec::new(), expires_at_ms: None, - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), }), ) .await @@ -11651,17 +12386,7 @@ mod tests { #[tokio::test] async fn update_provider_rejects_gateway_refresh_primary_and_additional_output_keys() { - use crate::grpc::policy::set_global_bool_setting_for_test; - let state = test_server_state().await; - set_global_bool_setting_for_test( - state.store.as_ref(), - openshell_core::settings::PROVIDERS_V2_ENABLED_KEY, - true, - ) - .await - .unwrap(); - let original_credentials = HashMap::from([ ( "AWS_ACCESS_KEY_ID".to_string(), @@ -11705,7 +12430,9 @@ mod tests { )]), secret_material_keys: Vec::new(), expires_at_ms: None, - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), }), ) .await @@ -11729,7 +12456,9 @@ mod tests { ..Default::default() }), credential_expires_at_ms: HashMap::new(), - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), }), ) .await @@ -11751,24 +12480,7 @@ mod tests { #[tokio::test] async fn configure_aws_sts_requires_profile_declaring_the_refresh() { - use crate::grpc::StoredSettingValue; - use crate::grpc::StoredSettings; - use crate::grpc::policy::save_global_settings; - let state = test_server_state().await; - let global_settings = StoredSettings { - revision: 1, - settings: std::iter::once(( - openshell_core::settings::PROVIDERS_V2_ENABLED_KEY.to_string(), - StoredSettingValue::Bool(true), - )) - .collect(), - ..Default::default() - }; - save_global_settings(state.store.as_ref(), &global_settings) - .await - .unwrap(); - // A generic provider has no profile, so it declares no STS refresh // binding. STS must not be configurable against it. create_provider_record( @@ -11812,7 +12524,9 @@ mod tests { )]), secret_material_keys: Vec::new(), expires_at_ms: None, - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), }), ) .await @@ -11824,24 +12538,7 @@ mod tests { #[tokio::test] async fn configure_aws_sts_rejects_non_canonical_credential_key() { - use crate::grpc::StoredSettingValue; - use crate::grpc::StoredSettings; - use crate::grpc::policy::save_global_settings; - let state = test_server_state().await; - let global_settings = StoredSettings { - revision: 1, - settings: std::iter::once(( - openshell_core::settings::PROVIDERS_V2_ENABLED_KEY.to_string(), - StoredSettingValue::Bool(true), - )) - .collect(), - ..Default::default() - }; - save_global_settings(state.store.as_ref(), &global_settings) - .await - .unwrap(); - create_provider_record( state.store.as_ref(), "default", @@ -11881,7 +12578,9 @@ mod tests { )]), secret_material_keys: Vec::new(), expires_at_ms: None, - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), }), ) .await @@ -11891,116 +12590,6 @@ mod tests { assert!(err.message().contains("AWS_ACCESS_KEY_ID")); } - #[tokio::test] - async fn rotate_aws_sts_blocked_after_providers_v2_disabled() { - use crate::grpc::policy::set_global_bool_setting_for_test; - - let state = test_server_state().await; - - set_global_bool_setting_for_test( - state.store.as_ref(), - openshell_core::settings::PROVIDERS_V2_ENABLED_KEY, - true, - ) - .await - .unwrap(); - - create_provider_record( - state.store.as_ref(), - "default", - Provider { - metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { - id: String::new(), - name: "aws-gate".to_string(), - created_at_ms: 0, - labels: HashMap::new(), - resource_version: 0, - annotations: HashMap::new(), - workspace: "default".to_string(), - deletion_timestamp_ms: 0, - }), - r#type: "aws".to_string(), - credentials: HashMap::new(), - config: HashMap::new(), - credential_expires_at_ms: HashMap::new(), - profile_workspace: "default".to_string(), - credential_handles: HashMap::new(), - }, - ) - .await - .unwrap(); - - handle_configure_provider_refresh( - &state, - authed_request(ConfigureProviderRefreshRequest { - provider: "aws-gate".to_string(), - credential_key: "AWS_ACCESS_KEY_ID".to_string(), - strategy: ProviderCredentialRefreshStrategy::AwsStsAssumeRole as i32, - material: HashMap::from([( - "role_arn".to_string(), - "arn:aws:iam::123456789012:role/Test".to_string(), - )]), - secret_material_keys: Vec::new(), - expires_at_ms: None, - workspace: "default".to_string(), - }), - ) - .await - .unwrap(); - - // Disable the gate after the refresh is already configured. - set_global_bool_setting_for_test( - state.store.as_ref(), - openshell_core::settings::PROVIDERS_V2_ENABLED_KEY, - false, - ) - .await - .unwrap(); - - let err = handle_rotate_provider_credential( - &state, - authed_request(RotateProviderCredentialRequest { - provider: "aws-gate".to_string(), - credential_key: "AWS_ACCESS_KEY_ID".to_string(), - workspace: "default".to_string(), - }), - ) - .await - .unwrap_err(); - - assert_eq!(err.code(), Code::FailedPrecondition); - assert!(err.message().contains("providers_v2_enabled")); - - // The gate rejection is recorded on the refresh state, and no credential - // was minted into the provider. - let provider = state - .store - .get_message_by_name::("default", "aws-gate") - .await - .unwrap() - .unwrap(); - assert!(!provider.credentials.contains_key("AWS_ACCESS_KEY_ID")); - let refresh_state = crate::provider_refresh::get_refresh_state( - state.store.as_ref(), - "default", - provider.object_id(), - "AWS_ACCESS_KEY_ID", - ) - .await - .unwrap() - .expect("refresh state should exist"); - assert_eq!(refresh_state.status, "configuration_required"); - assert_eq!( - refresh_state.recovery_action, - ProviderCredentialRefreshRecoveryAction::FixConfiguration as i32 - ); - assert_eq!(refresh_state.failure_code, "refresh_configuration_invalid"); - assert_eq!( - refresh_state.next_refresh_at_ms - refresh_state.last_error_at_ms, - 60 * 60 * 1000 - ); - } - #[tokio::test] async fn active_provider_environment_keys_include_additional_output_keys() { let state = test_server_state().await; @@ -12078,25 +12667,8 @@ mod tests { #[tokio::test] async fn configure_aws_sts_validates_additional_credential_key_collision() { - use crate::grpc::StoredSettingValue; - use crate::grpc::StoredSettings; - use crate::grpc::policy::save_global_settings; - let state = test_server_state().await; - let global_settings = StoredSettings { - revision: 1, - settings: std::iter::once(( - openshell_core::settings::PROVIDERS_V2_ENABLED_KEY.to_string(), - StoredSettingValue::Bool(true), - )) - .collect(), - ..Default::default() - }; - save_global_settings(state.store.as_ref(), &global_settings) - .await - .unwrap(); - let mut existing_provider = Provider { metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { id: String::new(), @@ -12186,7 +12758,9 @@ mod tests { )]), secret_material_keys: Vec::new(), expires_at_ms: None, - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), }), ) .await @@ -12198,17 +12772,7 @@ mod tests { #[tokio::test(flavor = "multi_thread", worker_threads = 4)] async fn concurrent_configure_aws_sts_reserves_key_exactly_once() { - use crate::grpc::policy::set_global_bool_setting_for_test; - let state = test_server_state().await; - set_global_bool_setting_for_test( - state.store.as_ref(), - openshell_core::settings::PROVIDERS_V2_ENABLED_KEY, - true, - ) - .await - .unwrap(); - for name in ["aws-a", "aws-b"] { create_provider_record( state.store.as_ref(), @@ -12271,7 +12835,9 @@ mod tests { )]), secret_material_keys: Vec::new(), expires_at_ms: None, - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), }) }; @@ -12454,8 +13020,8 @@ mod tests { // Create same-named provider in each workspace via handlers. let make_provider = || Provider { metadata: None, - r#type: "custom".to_string(), - credentials: HashMap::from([("TOKEN".to_string(), "secret".to_string())]), + r#type: "pypi".to_string(), + credentials: HashMap::new(), config: HashMap::new(), credential_expires_at_ms: HashMap::new(), profile_workspace: String::new(), @@ -12474,12 +13040,14 @@ mod tests { labels: HashMap::new(), resource_version: 0, annotations: HashMap::new(), - workspace: String::new(), + workspace: "default".to_string(), deletion_timestamp_ms: 0, }); p }), - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), }), ) .await @@ -12504,12 +13072,14 @@ mod tests { labels: HashMap::new(), resource_version: 0, annotations: HashMap::new(), - workspace: String::new(), + workspace: "default".to_string(), deletion_timestamp_ms: 0, }); p }), - workspace: "beta".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "beta".to_string(), + )), }), ) .await @@ -12529,7 +13099,9 @@ mod tests { &state, authed_request(GetProviderRequest { name: "shared-name".to_string(), - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), }), ) .await @@ -12541,7 +13113,9 @@ mod tests { &state, authed_request(GetProviderRequest { name: "shared-name".to_string(), - workspace: "beta".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "beta".to_string(), + )), }), ) .await @@ -12555,8 +13129,9 @@ mod tests { authed_request(ListProvidersRequest { limit: 100, offset: 0, - workspace: "default".to_string(), - all_workspaces: false, + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), }), ) .await @@ -12570,8 +13145,9 @@ mod tests { authed_request(ListProvidersRequest { limit: 100, offset: 0, - workspace: "beta".to_string(), - all_workspaces: false, + workspace_scope: Some(openshell_core::proto::workspace_selector( + "beta".to_string(), + )), }), ) .await @@ -12585,7 +13161,9 @@ mod tests { &state, authed_request(DeleteProviderRequest { name: "shared-name".to_string(), - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), }), ) .await @@ -12598,8 +13176,9 @@ mod tests { authed_request(ListProvidersRequest { limit: 100, offset: 0, - workspace: "default".to_string(), - all_workspaces: false, + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), }), ) .await @@ -12611,7 +13190,9 @@ mod tests { &state, authed_request(GetProviderRequest { name: "shared-name".to_string(), - workspace: "beta".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "beta".to_string(), + )), }), ) .await @@ -12633,12 +13214,14 @@ mod tests { labels: HashMap::new(), resource_version: 0, annotations: HashMap::new(), - workspace: String::new(), + workspace: "default".to_string(), deletion_timestamp_ms: 0, }); p }), - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), }), ) .await @@ -12649,28 +13232,13 @@ mod tests { authed_request(ListProvidersRequest { limit: 100, offset: 0, - workspace: String::new(), - all_workspaces: true, + workspace_scope: Some(openshell_core::proto::all_workspaces_selector()), }), ) .await .unwrap() .into_inner(); assert_eq!(listed.providers.len(), 2); - - // all_workspaces with non-empty workspace is rejected. - let err = handle_list_providers( - &state, - authed_request(ListProvidersRequest { - limit: 100, - offset: 0, - workspace: "default".to_string(), - all_workspaces: true, - }), - ) - .await - .unwrap_err(); - assert_eq!(err.code(), Code::InvalidArgument); } #[tokio::test] @@ -13482,7 +14050,7 @@ mod tests { let err = handle_create_provider( &state, non_member_request(CreateProviderRequest { - workspace: "no-such-ws".into(), + workspace_scope: Some(openshell_core::proto::workspace_selector("no-such-ws")), ..Default::default() }), ) @@ -13497,7 +14065,7 @@ mod tests { let err = handle_get_provider( &state, non_member_request(GetProviderRequest { - workspace: "no-such-ws".into(), + workspace_scope: Some(openshell_core::proto::workspace_selector("no-such-ws")), ..Default::default() }), ) @@ -13512,7 +14080,7 @@ mod tests { let err = handle_list_providers( &state, non_member_request(ListProvidersRequest { - workspace: "no-such-ws".into(), + workspace_scope: Some(openshell_core::proto::workspace_selector("no-such-ws")), ..Default::default() }), ) @@ -13527,7 +14095,7 @@ mod tests { let err = handle_update_provider( &state, non_member_request(UpdateProviderRequest { - workspace: "no-such-ws".into(), + workspace_scope: Some(openshell_core::proto::workspace_selector("no-such-ws")), ..Default::default() }), ) @@ -13542,7 +14110,7 @@ mod tests { let err = handle_get_provider_refresh_status( &state, non_member_request(GetProviderRefreshStatusRequest { - workspace: "no-such-ws".into(), + workspace_scope: Some(openshell_core::proto::workspace_selector("no-such-ws")), ..Default::default() }), ) @@ -13557,7 +14125,7 @@ mod tests { let err = handle_configure_provider_refresh( &state, non_member_request(ConfigureProviderRefreshRequest { - workspace: "no-such-ws".into(), + workspace_scope: Some(openshell_core::proto::workspace_selector("no-such-ws")), ..Default::default() }), ) @@ -13572,7 +14140,7 @@ mod tests { let err = handle_rotate_provider_credential( &state, non_member_request(RotateProviderCredentialRequest { - workspace: "no-such-ws".into(), + workspace_scope: Some(openshell_core::proto::workspace_selector("no-such-ws")), ..Default::default() }), ) @@ -13587,7 +14155,7 @@ mod tests { let err = handle_delete_provider_refresh( &state, non_member_request(DeleteProviderRefreshRequest { - workspace: "no-such-ws".into(), + workspace_scope: Some(openshell_core::proto::workspace_selector("no-such-ws")), ..Default::default() }), ) @@ -13602,7 +14170,7 @@ mod tests { let err = handle_delete_provider( &state, non_member_request(DeleteProviderRequest { - workspace: "no-such-ws".into(), + workspace_scope: Some(openshell_core::proto::workspace_selector("no-such-ws")), ..Default::default() }), ) diff --git a/crates/openshell-server/src/grpc/sandbox.rs b/crates/openshell-server/src/grpc/sandbox.rs index a8d198bc8..17ae70d2f 100644 --- a/crates/openshell-server/src/grpc/sandbox.rs +++ b/crates/openshell-server/src/grpc/sandbox.rs @@ -11,29 +11,38 @@ use crate::ServerState; use crate::auth::workspace_authz::{ - MinWorkspaceRole, authorize_sandbox_workspace, authorize_workspace, require_platform_admin, + AuthorizedWorkspaceScope, MinWorkspaceRole, authorize_list_workspace_selector, + authorize_sandbox_workspace, authorize_workspace_selector, }; use crate::persistence::{ObjectLabels, ObjectType, WriteCondition, generate_name}; use futures::future; use openshell_core::net::set_tcp_nodelay_best_effort; +use openshell_core::proto::datamodel::v1::ObjectMeta; use openshell_core::proto::{ AttachSandboxProviderRequest, AttachSandboxProviderResponse, CreateSandboxRequest, - CreateSshSessionRequest, CreateSshSessionResponse, DeleteSandboxRequest, DeleteSandboxResponse, - DetachSandboxProviderRequest, DetachSandboxProviderResponse, ExecSandboxEvent, ExecSandboxExit, - ExecSandboxInput, ExecSandboxRequest, ExecSandboxStderr, ExecSandboxStdout, GetSandboxRequest, - ListSandboxProvidersRequest, ListSandboxProvidersResponse, ListSandboxesRequest, - ListSandboxesResponse, Provider, RevokeSshSessionRequest, RevokeSshSessionResponse, - SandboxResponse, SandboxStreamEvent, SshRelayTarget, StartSandboxRequest, StopSandboxRequest, + CreateSandboxTemplateRequest, CreateSshSessionRequest, CreateSshSessionResponse, + DeleteSandboxRequest, DeleteSandboxResponse, DeleteSandboxTemplateRequest, + DeleteSandboxTemplateResponse, DetachSandboxProviderRequest, DetachSandboxProviderResponse, + ExecSandboxEvent, ExecSandboxExit, ExecSandboxInput, ExecSandboxRequest, ExecSandboxStderr, + ExecSandboxStdout, GetSandboxRequest, GetSandboxTemplateRequest, ListSandboxProvidersRequest, + ListSandboxProvidersResponse, ListSandboxTemplatesRequest, ListSandboxTemplatesResponse, + ListSandboxesRequest, ListSandboxesResponse, Provider, ResourceRequirements, + RevokeSshSessionRequest, RevokeSshSessionResponse, SandboxResources, SandboxResponse, + SandboxSpec, SandboxStreamEvent, SandboxTemplateResponse, SandboxWorkloadTemplate, + SandboxWorkloadTemplateProvenance, SshRelayTarget, StartSandboxRequest, StopSandboxRequest, TcpForwardFrame, TcpForwardInit, TcpRelayTarget, WatchSandboxRequest, relay_open, tcp_forward_init, }; -use openshell_core::proto::{Sandbox, SandboxPhase, SandboxTemplate, SshSession}; -use openshell_core::telemetry::{ - LifecycleOperation, LifecycleResource, SandboxTemplateSource, TelemetryComputeDriver, - TelemetryOutcome, +use openshell_core::proto::{ + BeginRootfsTarStagingRequest, BeginRootfsTarStagingResponse, Sandbox, SandboxPhase, + SandboxTemplate, SshSession, }; -use openshell_core::{ObjectId, ObjectName, ObjectWorkspace}; +use openshell_core::telemetry::{ + LifecycleOperation, LifecycleResource, SandboxTemplateSource, TelemetryOutcome, +}; +use openshell_core::{GetResourceVersion, ObjectId, ObjectName, ObjectWorkspace}; use prost::Message; +use prost_types::{Struct, Value, value::Kind}; use std::collections::HashMap; use std::net::IpAddr; use std::pin::Pin; @@ -53,14 +62,16 @@ use super::provider::{ get_provider_record, is_valid_env_key, validate_provider_environment_keys_unique_with_catalog, }; use super::validation::{ - level_matches, source_matches, validate_exec_request_fields, - validate_no_reserved_provider_policy_keys, validate_policy_safety, validate_sandbox_spec, + level_matches, source_matches, validate_and_canonicalize_policy, validate_dns1123_label, + validate_exec_request_fields, validate_no_reserved_provider_policy_keys, + validate_policy_safety, validate_sandbox_governance_spec, validate_sandbox_spec, }; use super::{MAX_PAGE_SIZE, MAX_PROVIDERS, MAX_ROUTABLE_NAME_LEN, clamp_limit}; use crate::persistence::current_time_ms; const TCP_FORWARD_CHUNK_SIZE: usize = 64 * 1024; const NO_LOGIN_SHELL_ENV: (&str, &str) = ("OPENSHELL_NO_LOGIN_SHELL", "1"); +const MAX_TEMPLATES_PER_WORKSPACE: u32 = 1000; #[derive(Debug)] pub struct WatchSandboxStream { @@ -159,30 +170,136 @@ pub(super) async fn handle_create_sandbox( ) -> Result, Status> { let create_request = request.get_ref().clone(); let result = handle_create_sandbox_inner(state, request).await; + let created_sandbox = result + .as_ref() + .ok() + .and_then(|response| response.get_ref().sandbox.as_ref()); emit_sandbox_create_telemetry( state, &create_request, + created_sandbox, TelemetryOutcome::from_success(result.is_ok()), ); result } +/// Allocate a gateway-owned staging slot for a local rootfs tar archive. +/// +/// The caller writes the archive to the returned path and then names the token +/// on `CreateSandbox`. It never names a filesystem path of its own choosing. +pub(super) async fn handle_begin_rootfs_tar_staging( + state: &Arc, + request: Request, +) -> Result, Status> { + let principal = super::extract_principal(&request)?; + let request = request.into_inner(); + + let authz = authorize_workspace_selector( + &state.store, + &state.admin_role, + &principal, + request.workspace_scope.as_ref(), + MinWorkspaceRole::User, + ) + .await?; + let workspace = super::workspace::resolve_workspace(state.store.as_ref(), &authz.workspace) + .await? + .ensure_active()?; + + let subject = principal_subject(&principal)?; + let slot = state.compute.rootfs_tar_staging().begin( + &workspace, + &subject, + &request.file_name, + request.size_bytes, + )?; + + Ok(Response::new(BeginRootfsTarStagingResponse { + staging_token: slot.token, + upload_path: slot.upload_path.to_string_lossy().into_owned(), + max_bytes: slot.max_bytes, + expires_at_ms: slot.expires_at_ms, + })) +} + +/// Stable caller identity used to bind a staging slot to its requester. +fn principal_subject(principal: &crate::auth::principal::Principal) -> Result { + match principal { + crate::auth::principal::Principal::User(user) => Ok(user.identity.subject.clone()), + _ => Err(Status::permission_denied( + "rootfs tar staging requires a user principal", + )), + } +} + +/// Read the staging token a caller named for the active driver, without +/// removing it. The gateway consumes it later, inside `create_sandbox`. +fn staging_token_in_spec(spec: &SandboxSpec, driver_name: &str) -> Option { + let config = spec.template.as_ref()?.driver_config.as_ref()?; + let Kind::StructValue(driver_config) = config.fields.get(driver_name)?.kind.as_ref()? else { + return None; + }; + match driver_config + .fields + .get(crate::compute::rootfs_tar::STAGING_TOKEN_FIELD)? + .kind + .as_ref()? + { + Kind::StringValue(token) => Some(token.clone()), + _ => None, + } +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +struct SandboxCreateTelemetryAttrs { + requested_gpu: bool, + provider_count: u64, + has_custom_policy: bool, + template_source: SandboxTemplateSource, +} + fn emit_sandbox_create_telemetry( state: &Arc, request: &CreateSandboxRequest, + created_sandbox: Option<&Sandbox>, outcome: TelemetryOutcome, ) { - let compute_driver = telemetry_compute_driver(state.compute.driver_kind()); + let compute_driver = state.compute.telemetry_compute_driver(); + let attrs = sandbox_create_telemetry_attrs(request, created_sandbox); + openshell_core::telemetry::emit_sandbox_create( + outcome, + attrs.requested_gpu, + attrs.provider_count, + attrs.has_custom_policy, + attrs.template_source, + compute_driver, + ); +} + +fn sandbox_create_telemetry_attrs( + request: &CreateSandboxRequest, + created_sandbox: Option<&Sandbox>, +) -> SandboxCreateTelemetryAttrs { + if !request.workload_template_name.trim().is_empty() { + let spec = created_sandbox + .and_then(|sandbox| sandbox.spec.as_ref()) + .or(request.spec.as_ref()); + return SandboxCreateTelemetryAttrs { + requested_gpu: spec.is_some_and(|spec| { + openshell_core::gpu::sandbox_gpu_requested(spec.resource_requirements.as_ref()) + }), + provider_count: spec.map_or(0, |spec| spec.providers.len() as u64), + has_custom_policy: spec.is_some_and(|spec| spec.policy.is_some()), + template_source: SandboxTemplateSource::WorkloadTemplate, + }; + } let Some(spec) = request.spec.as_ref() else { - openshell_core::telemetry::emit_sandbox_create( - outcome, - false, - 0, - false, - SandboxTemplateSource::Undefined, - compute_driver, - ); - return; + return SandboxCreateTelemetryAttrs { + requested_gpu: false, + provider_count: 0, + has_custom_policy: false, + template_source: SandboxTemplateSource::Undefined, + }; }; let template_source = if spec .template @@ -195,20 +312,12 @@ fn emit_sandbox_create_telemetry( }; let gpu_requested = openshell_core::gpu::sandbox_gpu_requested(spec.resource_requirements.as_ref()); - openshell_core::telemetry::emit_sandbox_create( - outcome, - gpu_requested, - spec.providers.len() as u64, - spec.policy.is_some(), + SandboxCreateTelemetryAttrs { + requested_gpu: gpu_requested, + provider_count: spec.providers.len() as u64, + has_custom_policy: spec.policy.is_some(), template_source, - compute_driver, - ); -} - -fn telemetry_compute_driver( - driver_kind: Option, -) -> TelemetryComputeDriver { - TelemetryComputeDriver::from_driver_kind(driver_kind) + } } async fn handle_create_sandbox_inner( @@ -218,33 +327,15 @@ async fn handle_create_sandbox_inner( let principal = super::extract_principal(&request)?; let request = request.into_inner(); let await_main_process_attachment = request.await_main_process_attachment; - let mut spec = request - .spec - .ok_or_else(|| Status::invalid_argument("spec is required"))?; + let workload_template_name = request.workload_template_name.trim().to_string(); - // Every newly persisted sandbox has one explicit canonical process. This - // portable default also preserves compatibility with callers compiled - // before the main-process field was introduced. - if spec.command.is_empty() { - spec.command = vec!["/bin/bash".to_string(), "-l".to_string()]; - spec.tty = true; - } + validate_create_sandbox_request_pre_io(&request, &workload_template_name)?; - // Validate field sizes before any I/O (fail fast on oversized payloads). - validate_sandbox_spec(&request.name, &spec)?; - - // Validate labels (keys and values must meet Kubernetes requirements). - for (key, value) in &request.labels { - crate::grpc::validation::validate_label_key(key)?; - crate::grpc::validation::validate_label_value(value)?; - } - crate::grpc::validation::validate_annotations(&request.annotations, "annotations")?; - - let authz = authorize_workspace( + let authz = authorize_workspace_selector( &state.store, &state.admin_role, &principal, - &request.workspace, + request.workspace_scope.as_ref(), MinWorkspaceRole::User, ) .await?; @@ -252,6 +343,53 @@ async fn handle_create_sandbox_inner( .await? .ensure_active()?; + let (mut spec, created_from_workload_template) = if workload_template_name.is_empty() { + let spec = request + .spec + .ok_or_else(|| Status::invalid_argument("spec is required"))?; + (spec, None) + } else { + let governance_spec = request.spec.unwrap_or_default(); + let template = state + .store + .get_message_by_name::(&workspace, &workload_template_name) + .await + .map_err(|e| Status::internal(format!("fetch sandbox template failed: {e}")))? + .ok_or_else(|| Status::not_found("sandbox template not found"))?; + let provenance = SandboxWorkloadTemplateProvenance { + name: template.object_name().to_string(), + resource_version: template.get_resource_version().to_string(), + }; + let mut resolved = sandbox_spec_from_stored_workload_template(&template)?; + resolved.policy = governance_spec.policy; + resolved.providers = governance_spec.providers; + resolved.command = governance_spec.command; + resolved.tty = governance_spec.tty; + (resolved, Some(provenance)) + }; + + // Leave an omitted command empty rather than persisting a concrete shell: + // the supervisor resolves the default login shell against the sandbox image + // (bash when present, otherwise /bin/sh on minimal images like Alpine), + // which the gateway cannot do since it does not see the sandbox filesystem. + // The default is an interactive login shell, so request a TTY. + if spec.command.is_empty() { + spec.tty = true; + } + + // Validate field sizes before any create-side effects. + validate_sandbox_spec(&request.name, &spec)?; + + // A staging slot may only be redeemed by the caller it was issued to. This + // is the only point in the create path where the principal is in scope. + if let Some(token) = staging_token_in_spec(&spec, state.compute.configured_driver_name()) { + let subject = principal_subject(&principal)?; + state + .compute + .rootfs_tar_staging() + .authorize(&token, &workspace, &subject)?; + } + let _sandbox_sync_guard = if spec.providers.is_empty() { None } else { @@ -281,13 +419,23 @@ async fn handle_create_sandbox_inner( // Ensure the template always carries the resolved image. let template = spec.template.get_or_insert_with(SandboxTemplate::default); - if template.image.is_empty() { + if template.image.trim().is_empty() { template.image = state.compute.default_image().to_string(); } if let Some(ref mut policy) = spec.policy { super::policy::clear_provider_credentialed_markers(policy); validate_no_reserved_provider_policy_keys(policy)?; + *policy = validate_and_canonicalize_policy(policy.clone())?; + } + + // Process identity and MCP default materialization can increase the + // protobuf size. Recheck the exact canonical spec before any middleware or + // compute boundary can observe or persist it. The initial check remains + // above so requests that are already oversized still fail before I/O. + validate_sandbox_spec(&request.name, &spec)?; + + if let Some(ref policy) = spec.policy { validate_policy_safety(policy)?; crate::middleware::validate_policy(state.middleware_registry.as_ref(), policy).await?; } @@ -309,7 +457,7 @@ async fn handle_create_sandbox_inner( let now_ms = current_time_ms(); let mut sandbox = Sandbox { - metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { + metadata: Some(ObjectMeta { id: id.clone(), name: name.clone(), created_at_ms: now_ms, @@ -321,6 +469,7 @@ async fn handle_create_sandbox_inner( }), spec: Some(spec), status: None, + created_from_workload_template, }; sandbox.set_phase(SandboxPhase::Provisioning as i32); @@ -347,11 +496,8 @@ async fn handle_create_sandbox_inner( status })?; - // Mint the gateway JWT for singleplayer drivers. K8s sandboxes skip - // this mint and bootstrap via `IssueSandboxToken` at supervisor - // startup; identifying "is this K8s?" lives in the compute layer, so - // we mint unconditionally here when the issuer is configured and let - // the K8s driver simply ignore the field. + // Mint a gateway JWT whenever the issuer is configured. Compute runtimes + // that bootstrap through another authentication mechanism may ignore it. let sandbox_token = state.sandbox_jwt_issuer.as_ref().map(|issuer| { issuer.mint(&id).map(|minted| { tracing::info!( @@ -382,6 +528,135 @@ async fn handle_create_sandbox_inner( })) } +fn validate_create_sandbox_request_pre_io( + request: &CreateSandboxRequest, + workload_template_name: &str, +) -> Result<(), Status> { + // Validate labels (keys and values must meet Kubernetes requirements). + for (key, value) in &request.labels { + crate::grpc::validation::validate_label_key(key)?; + crate::grpc::validation::validate_label_value(value)?; + } + crate::grpc::validation::validate_annotations(&request.annotations, "annotations")?; + + if workload_template_name.is_empty() { + let spec = request + .spec + .as_ref() + .ok_or_else(|| Status::invalid_argument("spec is required"))?; + return validate_sandbox_spec(&request.name, spec); + } + + validate_dns1123_label(workload_template_name, "workload_template_name")?; + if let Some(spec) = request.spec.as_ref() { + validate_template_create_governance_spec(spec)?; + validate_sandbox_governance_spec(&request.name, spec)?; + } else { + validate_sandbox_governance_spec(&request.name, &SandboxSpec::default())?; + } + Ok(()) +} + +fn validate_template_create_governance_spec(spec: &SandboxSpec) -> Result<(), Status> { + if !spec.log_level.is_empty() { + return Err(Status::invalid_argument( + "spec.log_level cannot be set when workload_template_name is set", + )); + } + if !spec.environment.is_empty() { + return Err(Status::invalid_argument( + "spec.environment cannot be set when workload_template_name is set", + )); + } + if spec.template.is_some() { + return Err(Status::invalid_argument( + "spec.template cannot be set when workload_template_name is set", + )); + } + if spec.resource_requirements.is_some() { + return Err(Status::invalid_argument( + "spec.resource_requirements cannot be set when workload_template_name is set", + )); + } + Ok(()) +} + +fn sandbox_spec_from_stored_workload_template( + template: &SandboxWorkloadTemplate, +) -> Result { + sandbox_spec_from_workload_template(template, tonic::Code::Internal) +} + +fn sandbox_spec_from_user_workload_template( + template: &SandboxWorkloadTemplate, +) -> Result { + sandbox_spec_from_workload_template(template, tonic::Code::InvalidArgument) +} + +fn sandbox_spec_from_workload_template( + template: &SandboxWorkloadTemplate, + missing_field_code: tonic::Code, +) -> Result { + let spec = template + .spec + .as_ref() + .ok_or_else(|| Status::new(missing_field_code, "sandbox template spec is required"))?; + let workload = spec + .workload + .as_ref() + .ok_or_else(|| Status::new(missing_field_code, "sandbox template workload is required"))?; + let resources = workload.resources.as_ref(); + Ok(SandboxSpec { + environment: workload.environment.clone(), + template: Some(SandboxTemplate { + image: workload.image.clone(), + resources: resources.and_then(template_resource_struct), + driver_config: spec.driver_config.clone(), + ..SandboxTemplate::default() + }), + resource_requirements: resources.and_then(template_gpu_requirements), + ..SandboxSpec::default() + }) +} + +fn template_gpu_requirements(resources: &SandboxResources) -> Option { + Some(ResourceRequirements { + gpu: Some(resources.gpu?), + }) +} + +fn template_resource_struct(resources: &SandboxResources) -> Option { + let mut limits = std::collections::BTreeMap::new(); + if !resources.cpu.is_empty() { + limits.insert( + "cpu".to_string(), + Value { + kind: Some(Kind::StringValue(resources.cpu.clone())), + }, + ); + } + if !resources.memory.is_empty() { + limits.insert( + "memory".to_string(), + Value { + kind: Some(Kind::StringValue(resources.memory.clone())), + }, + ); + } + if limits.is_empty() { + None + } else { + let mut fields = std::collections::BTreeMap::new(); + fields.insert( + "limits".to_string(), + Value { + kind: Some(Kind::StructValue(Struct { fields: limits })), + }, + ); + Some(Struct { fields }) + } +} + pub(super) async fn handle_get_sandbox( state: &Arc, request: Request, @@ -391,11 +666,11 @@ pub(super) async fn handle_get_sandbox( if req.name.is_empty() { return Err(Status::invalid_argument("name is required")); } - let authz = authorize_workspace( + let authz = authorize_workspace_selector( &state.store, &state.admin_role, &principal, - &req.workspace, + req.workspace_scope.as_ref(), MinWorkspaceRole::User, ) .await?; @@ -421,15 +696,17 @@ pub(super) async fn handle_list_sandboxes( ) -> Result, Status> { let principal = super::extract_principal(&request)?; let request = request.into_inner(); - if request.all_workspaces && !request.workspace.is_empty() { - return Err(Status::invalid_argument( - "all_workspaces and workspace are mutually exclusive", - )); - } let limit = clamp_limit(request.limit, 100, MAX_PAGE_SIZE); - let sandboxes: Vec = if request.all_workspaces { - require_platform_admin(&state.admin_role, &principal)?; + let scope = authorize_list_workspace_selector( + &state.store, + &state.admin_role, + &principal, + request.workspace_scope.as_ref(), + MinWorkspaceRole::User, + ) + .await?; + let sandboxes: Vec = if matches!(scope, AuthorizedWorkspaceScope::AllWorkspaces) { if request.label_selector.is_empty() { state .store @@ -445,14 +722,9 @@ pub(super) async fn handle_list_sandboxes( .map_err(|e| Status::internal(format!("list sandboxes failed: {e}")))? } } else { - let authz = authorize_workspace( - &state.store, - &state.admin_role, - &principal, - &request.workspace, - MinWorkspaceRole::User, - ) - .await?; + let AuthorizedWorkspaceScope::Workspace(authz) = scope else { + unreachable!("all-workspaces scope handled above") + }; let workspace = super::workspace::resolve_workspace(state.store.as_ref(), &authz.workspace) .await? .name; @@ -482,17 +754,294 @@ pub(super) async fn handle_list_sandboxes( Ok(Response::new(ListSandboxesResponse { sandboxes })) } +pub(super) async fn handle_create_sandbox_template( + state: &Arc, + request: Request, +) -> Result, Status> { + let principal = super::extract_principal(&request)?; + let req = request.into_inner(); + let template = req + .template + .ok_or_else(|| Status::invalid_argument("template is required"))?; + let metadata = template.metadata.clone().unwrap_or_default(); + let authz = authorize_workspace_selector( + &state.store, + &state.admin_role, + &principal, + req.workspace_scope.as_ref(), + MinWorkspaceRole::Admin, + ) + .await?; + let workspace = super::workspace::resolve_workspace(state.store.as_ref(), &authz.workspace) + .await? + .ensure_active()?; + if !metadata.workspace.is_empty() && metadata.workspace != workspace { + return Err(Status::invalid_argument( + "template.metadata.workspace must match request workspace", + )); + } + if metadata.name.is_empty() { + return Err(Status::invalid_argument( + "template.metadata.name is required", + )); + } + + let mut resolved = template; + resolved.metadata = Some(ObjectMeta { + id: uuid::Uuid::new_v4().to_string(), + name: metadata.name, + created_at_ms: current_time_ms(), + labels: metadata.labels, + resource_version: 0, + annotations: metadata.annotations, + workspace: workspace.clone(), + deletion_timestamp_ms: 0, + }); + validate_sandbox_workload_template(&resolved)?; + + let labels_map = resolved.object_labels(); + let labels_json = if labels_map.as_ref().is_none_or(HashMap::is_empty) { + None + } else { + Some( + serde_json::to_string(&labels_map) + .map_err(|e| Status::internal(format!("failed to serialize labels: {e}")))?, + ) + }; + let write = state + .store + .create_if_workspace_count_below( + SandboxWorkloadTemplate::object_type(), + resolved.object_id(), + resolved.object_name(), + &workspace, + &resolved.encode_to_vec(), + labels_json.as_deref(), + u64::from(MAX_TEMPLATES_PER_WORKSPACE), + ) + .await; + let write = match write { + Ok(Some(write)) => write, + Ok(None) => { + return Err(Status::resource_exhausted(format!( + "workspace has reached the maximum of {MAX_TEMPLATES_PER_WORKSPACE} sandbox templates" + ))); + } + Err(crate::persistence::PersistenceError::UniqueViolation { .. }) => { + return Err(Status::already_exists("sandbox template already exists")); + } + Err(err) => { + return Err(Status::internal(format!( + "persist sandbox template failed: {err}" + ))); + } + }; + if let Some(metadata) = resolved.metadata.as_mut() { + metadata.resource_version = write.resource_version; + } + + Ok(Response::new(SandboxTemplateResponse { + template: Some(resolved), + })) +} + +pub(super) async fn handle_get_sandbox_template( + state: &Arc, + request: Request, +) -> Result, Status> { + let principal = super::extract_principal(&request)?; + let req = request.into_inner(); + if req.name.is_empty() { + return Err(Status::invalid_argument("name is required")); + } + let authz = authorize_workspace_selector( + &state.store, + &state.admin_role, + &principal, + req.workspace_scope.as_ref(), + MinWorkspaceRole::User, + ) + .await?; + let workspace = super::workspace::resolve_workspace(state.store.as_ref(), &authz.workspace) + .await? + .name; + let template = state + .store + .get_message_by_name::(&workspace, &req.name) + .await + .map_err(|e| Status::internal(format!("fetch sandbox template failed: {e}")))? + .ok_or_else(|| Status::not_found("sandbox template not found"))?; + Ok(Response::new(SandboxTemplateResponse { + template: Some(template), + })) +} + +pub(super) async fn handle_list_sandbox_templates( + state: &Arc, + request: Request, +) -> Result, Status> { + let principal = super::extract_principal(&request)?; + let request = request.into_inner(); + let limit = clamp_limit(request.limit, 100, MAX_PAGE_SIZE); + let scope = authorize_list_workspace_selector( + &state.store, + &state.admin_role, + &principal, + request.workspace_scope.as_ref(), + MinWorkspaceRole::User, + ) + .await?; + let templates = if matches!(scope, AuthorizedWorkspaceScope::AllWorkspaces) { + if request.label_selector.is_empty() { + state + .store + .list_all_messages::(limit, request.offset) + .await + .map_err(|e| Status::internal(format!("list sandbox templates failed: {e}")))? + } else { + crate::grpc::validation::validate_label_selector(&request.label_selector)?; + state + .store + .list_all_messages_with_selector::( + &request.label_selector, + limit, + request.offset, + ) + .await + .map_err(|e| Status::internal(format!("list sandbox templates failed: {e}")))? + } + } else { + let AuthorizedWorkspaceScope::Workspace(authz) = scope else { + unreachable!("all-workspaces scope handled above") + }; + let workspace = super::workspace::resolve_workspace(state.store.as_ref(), &authz.workspace) + .await? + .name; + if request.label_selector.is_empty() { + state + .store + .list_messages::(&workspace, limit, request.offset) + .await + .map_err(|e| Status::internal(format!("list sandbox templates failed: {e}")))? + } else { + crate::grpc::validation::validate_label_selector(&request.label_selector)?; + state + .store + .list_messages_with_selector::( + &workspace, + &request.label_selector, + limit, + request.offset, + ) + .await + .map_err(|e| { + Status::internal(format!("list sandbox templates with selector failed: {e}")) + })? + } + }; + Ok(Response::new(ListSandboxTemplatesResponse { templates })) +} + +pub(super) async fn handle_delete_sandbox_template( + state: &Arc, + request: Request, +) -> Result, Status> { + let principal = super::extract_principal(&request)?; + let req = request.into_inner(); + if req.name.is_empty() { + return Err(Status::invalid_argument("name is required")); + } + let authz = authorize_workspace_selector( + &state.store, + &state.admin_role, + &principal, + req.workspace_scope.as_ref(), + MinWorkspaceRole::Admin, + ) + .await?; + let workspace = super::workspace::resolve_workspace(state.store.as_ref(), &authz.workspace) + .await? + .name; + let deleted = state + .store + .delete_by_name( + SandboxWorkloadTemplate::object_type(), + &workspace, + &req.name, + ) + .await + .map_err(|e| Status::internal(format!("delete sandbox template failed: {e}")))?; + Ok(Response::new(DeleteSandboxTemplateResponse { deleted })) +} + +fn validate_sandbox_workload_template(template: &SandboxWorkloadTemplate) -> Result<(), Status> { + super::validation::validate_object_metadata(template.metadata.as_ref(), "sandbox_template")?; + let name = template.object_name().to_string(); + validate_dns1123_label(&name, "template.metadata.name")?; + validate_sandbox_workload_template_service_level(template)?; + let spec = sandbox_spec_from_user_workload_template(template)?; + validate_sandbox_spec(&name, &spec)?; + Ok(()) +} + +fn validate_sandbox_workload_template_service_level( + template: &SandboxWorkloadTemplate, +) -> Result<(), Status> { + let Some(startup) = template + .spec + .as_ref() + .and_then(|spec| spec.desired_service_level.as_ref()) + .and_then(|service_level| service_level.startup.as_ref()) + else { + return Ok(()); + }; + if let Some(ready_within) = &startup.ready_within { + validate_positive_normalized_duration( + ready_within, + "template.spec.desired_service_level.startup.ready_within", + )?; + } + Ok(()) +} + +fn validate_positive_normalized_duration( + duration: &prost_types::Duration, + field: &str, +) -> Result<(), Status> { + const MAX_DURATION_SECONDS: u64 = 315_576_000_000; + if duration.seconds.unsigned_abs() > MAX_DURATION_SECONDS + || duration.nanos.unsigned_abs() >= 1_000_000_000 + { + return Err(Status::invalid_argument(format!( + "{field} must be a valid protobuf Duration" + ))); + } + if (duration.seconds > 0 && duration.nanos < 0) || (duration.seconds < 0 && duration.nanos > 0) + { + return Err(Status::invalid_argument(format!( + "{field} must be a normalized protobuf Duration" + ))); + } + if duration.seconds < 0 || duration.nanos < 0 || (duration.seconds == 0 && duration.nanos == 0) + { + return Err(Status::invalid_argument(format!( + "{field} must be greater than zero" + ))); + } + Ok(()) +} + pub(super) async fn handle_list_sandbox_providers( state: &Arc, request: Request, ) -> Result, Status> { let principal = super::extract_principal(&request)?; let req = request.into_inner(); - let authz = authorize_workspace( + let authz = authorize_workspace_selector( &state.store, &state.admin_role, &principal, - &req.workspace, + req.workspace_scope.as_ref(), MinWorkspaceRole::User, ) .await?; @@ -510,11 +1059,11 @@ pub(super) async fn handle_attach_sandbox_provider( ) -> Result, Status> { let principal = super::extract_principal(&request)?; let request = request.into_inner(); - let authz = authorize_workspace( + let authz = authorize_workspace_selector( &state.store, &state.admin_role, &principal, - &request.workspace, + request.workspace_scope.as_ref(), MinWorkspaceRole::User, ) .await?; @@ -659,11 +1208,11 @@ pub(super) async fn handle_detach_sandbox_provider( ) -> Result, Status> { let principal = super::extract_principal(&request)?; let request = request.into_inner(); - let authz = authorize_workspace( + let authz = authorize_workspace_selector( &state.store, &state.admin_role, &principal, - &request.workspace, + request.workspace_scope.as_ref(), MinWorkspaceRole::User, ) .await?; @@ -779,11 +1328,11 @@ async fn handle_delete_sandbox_inner( if name.is_empty() { return Err(Status::invalid_argument("name is required")); } - let authz = authorize_workspace( + let authz = authorize_workspace_selector( &state.store, &state.admin_role, &principal, - &req.workspace, + req.workspace_scope.as_ref(), MinWorkspaceRole::User, ) .await?; @@ -827,11 +1376,11 @@ async fn handle_stop_sandbox_inner( if req.name.is_empty() { return Err(Status::invalid_argument("name is required")); } - let authz = authorize_workspace( + let authz = authorize_workspace_selector( &state.store, &state.admin_role, &principal, - &req.workspace, + req.workspace_scope.as_ref(), MinWorkspaceRole::User, ) .await?; @@ -871,11 +1420,11 @@ async fn handle_start_sandbox_inner( if req.name.is_empty() { return Err(Status::invalid_argument("name is required")); } - let authz = authorize_workspace( + let authz = authorize_workspace_selector( &state.store, &state.admin_role, &principal, - &req.workspace, + req.workspace_scope.as_ref(), MinWorkspaceRole::User, ) .await?; @@ -1195,6 +1744,16 @@ fn is_watch_terminal(phase: SandboxPhase) -> bool { // Exec handler // --------------------------------------------------------------------------- +const DEFAULT_PTY_COLS: u32 = 80; +const DEFAULT_PTY_ROWS: u32 = 24; + +fn pty_dimensions(cols: u32, rows: u32) -> (u32, u32) { + ( + if cols == 0 { DEFAULT_PTY_COLS } else { cols }, + if rows == 0 { DEFAULT_PTY_ROWS } else { rows }, + ) +} + pub(super) async fn handle_exec_sandbox( state: &Arc, request: Request, @@ -1236,6 +1795,7 @@ pub(super) async fn handle_exec_sandbox( let stdin_payload = req.stdin; let timeout_seconds = req.timeout_seconds; let request_tty = req.tty; + let (cols, rows) = pty_dimensions(req.cols, req.rows); let sandbox_id = sandbox.object_id().to_string(); @@ -1260,6 +1820,8 @@ pub(super) async fn handle_exec_sandbox( timeout_seconds, request_tty, no_login_shell, + cols, + rows, ) .await { @@ -1671,8 +2233,7 @@ pub(super) async fn handle_exec_sandbox_interactive( let request_tty = req.tty; let no_login_shell = req.no_login_shell; let timeout_seconds = req.timeout_seconds; - let cols = if req.cols == 0 { 80 } else { req.cols }; - let rows = if req.rows == 0 { 24 } else { req.rows }; + let (cols, rows) = pty_dimensions(req.cols, req.rows); let sandbox_id = sandbox.object_id().to_string(); @@ -1751,7 +2312,7 @@ pub(super) async fn handle_create_ssh_session( 0 }; let session = SshSession { - metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { + metadata: Some(ObjectMeta { id: token.clone(), name: generate_name(), created_at_ms: now_ms, @@ -1997,6 +2558,8 @@ async fn stream_exec_over_relay( timeout_seconds: u32, request_tty: bool, no_login_shell: bool, + cols: u32, + rows: u32, ) -> Result<(), Status> { let command_preview: String = command .chars() @@ -2022,6 +2585,7 @@ async fn stream_exec_over_relay( stdin_payload, request_tty, no_login_shell, + (cols, rows), tx.clone(), ); @@ -2373,8 +2937,11 @@ async fn run_exec_with_russh( stdin_payload: Vec, request_tty: bool, no_shell_login: bool, + pty_size: (u32, u32), tx: mpsc::Sender>, ) -> Result { + let (cols, rows) = pty_size; + // Defense-in-depth: validate command at the transport boundary. if command.as_bytes().contains(&0) { return Err(Status::invalid_argument( @@ -2423,7 +2990,7 @@ async fn run_exec_with_russh( if request_tty { channel - .request_pty(false, "xterm-256color", 0, 0, 0, 0, &[]) + .request_pty(false, "xterm-256color", cols, rows, 0, 0, &[]) .await .map_err(|e| Status::internal(format!("failed to allocate PTY: {e}")))?; } @@ -2524,6 +3091,7 @@ mod tests { }; use crate::provider_profile_sources::ProviderProfileSources; use openshell_core::GatewayProviderProfileSourceConfig; + use openshell_core::proto::GpuResourceRequirements; use openshell_core::proto::datamodel::v1::ObjectMeta; async fn test_server_state_with_user_only_github_profile() -> Arc { @@ -2551,6 +3119,13 @@ mod tests { // ---- shell_escape ---- + #[test] + fn pty_dimensions_default_zero_values_without_overwriting_explicit_values() { + assert_eq!(pty_dimensions(0, 0), (80, 24)); + assert_eq!(pty_dimensions(120, 40), (120, 40)); + assert_eq!(pty_dimensions(0, 40), (80, 40)); + } + #[test] fn watch_terminal_phases_include_command_results_and_errors() { for phase in [ @@ -2576,26 +3151,64 @@ mod tests { } #[test] - fn telemetry_compute_driver_uses_resolved_driver_kind() { + fn sandbox_create_telemetry_uses_resolved_template_gpu_request() { + let request = CreateSandboxRequest { + spec: Some(SandboxSpec { + providers: vec!["github".to_string()], + policy: Some(openshell_core::proto::SandboxPolicy::default()), + ..SandboxSpec::default() + }), + workload_template_name: "gpu-kata".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector("default")), + ..CreateSandboxRequest::default() + }; + let created = Sandbox { + spec: Some(SandboxSpec { + providers: vec!["github".to_string()], + policy: Some(openshell_core::proto::SandboxPolicy::default()), + resource_requirements: Some(ResourceRequirements { + gpu: Some(GpuResourceRequirements { count: Some(1) }), + }), + ..SandboxSpec::default() + }), + created_from_workload_template: Some(SandboxWorkloadTemplateProvenance { + name: "gpu-kata".to_string(), + resource_version: "7".to_string(), + }), + ..Sandbox::default() + }; + assert_eq!( - telemetry_compute_driver(Some(openshell_core::ComputeDriverKind::Docker)), - TelemetryComputeDriver::Docker + sandbox_create_telemetry_attrs(&request, Some(&created)), + SandboxCreateTelemetryAttrs { + requested_gpu: true, + provider_count: 1, + has_custom_policy: true, + template_source: SandboxTemplateSource::WorkloadTemplate, + } ); + } + + #[test] + fn sandbox_create_telemetry_falls_back_to_request_for_unresolved_template() { + let request = CreateSandboxRequest { + spec: Some(SandboxSpec { + providers: vec!["github".to_string()], + ..SandboxSpec::default() + }), + workload_template_name: "missing-template".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector("default")), + ..CreateSandboxRequest::default() + }; + assert_eq!( - telemetry_compute_driver(Some(openshell_core::ComputeDriverKind::Kubernetes)), - TelemetryComputeDriver::Kubernetes - ); - assert_eq!( - telemetry_compute_driver(Some(openshell_core::ComputeDriverKind::Podman)), - TelemetryComputeDriver::Podman - ); - assert_eq!( - telemetry_compute_driver(Some(openshell_core::ComputeDriverKind::Vm)), - TelemetryComputeDriver::Vm - ); - assert_eq!( - telemetry_compute_driver(None), - TelemetryComputeDriver::Unknown + sandbox_create_telemetry_attrs(&request, None), + SandboxCreateTelemetryAttrs { + requested_gpu: false, + provider_count: 1, + has_custom_policy: false, + template_source: SandboxTemplateSource::WorkloadTemplate, + } ); } @@ -2908,7 +3521,7 @@ mod tests { workspace: "default".to_string(), deletion_timestamp_ms: 0, }), - spec: Some(openshell_core::proto::SandboxSpec { + spec: Some(SandboxSpec { log_level: "debug".to_string(), policy: Some(openshell_core::proto::SandboxPolicy::default()), providers, @@ -2921,6 +3534,41 @@ mod tests { sandbox } + fn test_workload_template(name: &str) -> SandboxWorkloadTemplate { + SandboxWorkloadTemplate { + metadata: Some(ObjectMeta { + id: String::new(), + name: name.to_string(), + created_at_ms: 0, + labels: HashMap::from([("team".to_string(), "runtime".to_string())]), + resource_version: 0, + annotations: HashMap::new(), + workspace: String::new(), + deletion_timestamp_ms: 0, + }), + spec: Some(openshell_core::proto::SandboxWorkloadTemplateSpec { + workload: Some(openshell_core::proto::SandboxWorkloadConfig { + image: "registry.example.com/agent:latest".to_string(), + environment: HashMap::from([("FEATURE_FLAG".to_string(), "on".to_string())]), + resources: Some(SandboxResources { + cpu: "2".to_string(), + memory: "4Gi".to_string(), + gpu: Some(GpuResourceRequirements { count: Some(1) }), + }), + }), + driver_config: None, + desired_service_level: None, + }), + } + } + + fn proto_string_value(value: &Value) -> Option<&str> { + match value.kind.as_ref() { + Some(Kind::StringValue(value)) => Some(value.as_str()), + _ => None, + } + } + #[tokio::test] #[ignore = "flaky under concurrent test execution"] async fn watch_producer_releases_request_span_when_client_disconnects() { @@ -2982,7 +3630,9 @@ mod tests { &delete_state, authed_request(DeleteSandboxRequest { name: "reused-name".to_string(), - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), }), ) .await @@ -3041,7 +3691,7 @@ mod tests { sandbox_name: "work".to_string(), provider_name: "work-github".to_string(), expected_resource_version: 0, - workspace: String::new(), + workspace_scope: Some(openshell_core::proto::workspace_selector("default")), }), ) .await @@ -3082,7 +3732,7 @@ mod tests { sandbox_name: "work".to_string(), provider_name: "work-github".to_string(), expected_resource_version: 0, - workspace: String::new(), + workspace_scope: Some(openshell_core::proto::workspace_selector("default")), }), ) .await @@ -3120,7 +3770,7 @@ mod tests { sandbox_name: "work".to_string(), provider_name: "work-github".to_string(), expected_resource_version: 0, - workspace: String::new(), + workspace_scope: Some(openshell_core::proto::workspace_selector("default")), }), ) .await @@ -3143,6 +3793,15 @@ mod tests { #[tokio::test] async fn detach_sandbox_provider_is_idempotent_and_removes_all_matches() { let state = test_server_state().await; + state + .store + .put_message(&test_provider_with_credential_key( + "other", + "github", + "GITHUB_TOKEN", + )) + .await + .unwrap(); state .store .put_message(&test_sandbox( @@ -3162,7 +3821,7 @@ mod tests { sandbox_name: "work".to_string(), provider_name: "work-github".to_string(), expected_resource_version: 0, - workspace: String::new(), + workspace_scope: Some(openshell_core::proto::workspace_selector("default")), }), ) .await @@ -3187,7 +3846,7 @@ mod tests { sandbox_name: "work".to_string(), provider_name: "work-github".to_string(), expected_resource_version: 0, - workspace: String::new(), + workspace_scope: Some(openshell_core::proto::workspace_selector("default")), }), ) .await @@ -3234,7 +3893,7 @@ mod tests { sandbox_name: "work".to_string(), provider_name: "work-gcp".to_string(), expected_resource_version: 0, - workspace: String::new(), + workspace_scope: Some(openshell_core::proto::workspace_selector("default")), }), ) .await @@ -3272,7 +3931,7 @@ mod tests { &state, authed_request(ListSandboxProvidersRequest { sandbox_name: "work".to_string(), - workspace: String::new(), + workspace_scope: Some(openshell_core::proto::workspace_selector("default")), }), ) .await @@ -3302,7 +3961,7 @@ mod tests { sandbox_name: "work".to_string(), provider_name: "missing".to_string(), expected_resource_version: 0, - workspace: String::new(), + workspace_scope: Some(openshell_core::proto::workspace_selector("default")), }), ) .await @@ -3471,14 +4130,15 @@ mod tests { &state, authed_request(CreateSandboxRequest { name: "collision".to_string(), - spec: Some(openshell_core::proto::SandboxSpec { + spec: Some(SandboxSpec { providers: vec!["provider-a".to_string(), "provider-b".to_string()], ..Default::default() }), labels: HashMap::new(), annotations: HashMap::new(), - workspace: String::new(), + workspace_scope: Some(openshell_core::proto::workspace_selector("default")), await_main_process_attachment: false, + workload_template_name: String::new(), }), ) .await @@ -3498,11 +4158,12 @@ mod tests { &state, authed_request(CreateSandboxRequest { name: "user-catalog".to_string(), - spec: Some(openshell_core::proto::SandboxSpec::default()), + spec: Some(SandboxSpec::default()), labels: HashMap::new(), annotations: HashMap::new(), - workspace: String::new(), + workspace_scope: Some(openshell_core::proto::workspace_selector("default")), await_main_process_attachment: false, + workload_template_name: String::new(), }), ) .await @@ -3531,14 +4192,15 @@ mod tests { &state, authed_request(CreateSandboxRequest { name: "reserved-policy-key".to_string(), - spec: Some(openshell_core::proto::SandboxSpec { + spec: Some(SandboxSpec { policy: Some(policy), ..Default::default() }), labels: HashMap::new(), annotations: HashMap::new(), - workspace: String::new(), + workspace_scope: Some(openshell_core::proto::workspace_selector("default")), await_main_process_attachment: false, + workload_template_name: String::new(), }), ) .await @@ -3549,6 +4211,344 @@ mod tests { assert!(err.message().contains("reserved '_provider_' prefix")); } + fn mcp_policy_with_options( + mcp: Option, + ) -> openshell_core::proto::SandboxPolicy { + let mut policy = openshell_policy::restrictive_default_policy(); + policy.network_policies.insert( + "mcp".to_string(), + openshell_core::proto::NetworkPolicyRule { + name: "mcp".to_string(), + endpoints: vec![openshell_core::proto::NetworkEndpoint { + host: "mcp.example.com".to_string(), + port: 443, + protocol: "mcp".to_string(), + mcp, + // Keep this fixture valid independently of MCP version + // defaulting: without allow-all, MCP endpoints require an + // explicit method rule at the L7 validation boundary. + rules: vec![openshell_core::proto::L7Rule { + allow: Some(openshell_core::proto::L7Allow { + method: "tools/list".to_string(), + ..Default::default() + }), + }], + ..Default::default() + }], + ..Default::default() + }, + ); + policy + } + + fn mcp_policy_with_versions(versions: &[&str]) -> openshell_core::proto::SandboxPolicy { + mcp_policy_with_options(Some(openshell_core::proto::McpOptions { + versions: versions.iter().map(ToString::to_string).collect(), + ..Default::default() + })) + } + + fn mcp_padding_rule_name_for_policy_size(target_size: usize) -> String { + let mut policy = mcp_policy_with_versions(&["2025-11-25"]); + openshell_policy::ensure_sandbox_process_identity(&mut policy); + let mut policy = validate_and_canonicalize_policy(policy) + .expect("explicit default MCP policy must be canonicalizable"); + policy.network_policies.insert( + "padding".to_string(), + openshell_core::proto::NetworkPolicyRule::default(), + ); + + // The padding rule has no endpoints and therefore does not change the + // policy's behavior. Iterative adjustment accounts for protobuf varint + // length-prefix growth while targeting the exact encoded boundary. + for _ in 0..4 { + let current_size = policy.encoded_len(); + if current_size == target_size { + break; + } + + let padding = &mut policy + .network_policies + .get_mut("padding") + .expect("padding rule must exist") + .name; + if current_size < target_size { + padding.push_str(&"x".repeat(target_size - current_size)); + } else { + let new_len = padding + .len() + .checked_sub(current_size - target_size) + .expect("padding adjustment must remain nonnegative"); + padding.truncate(new_len); + } + } + + assert_eq!(policy.encoded_len(), target_size); + policy + .network_policies + .remove("padding") + .expect("padding rule must exist") + .name + } + + #[tokio::test] + async fn create_sandbox_canonicalizes_mcp_versions_before_persistence() { + let state = test_server_state().await; + + handle_create_sandbox( + &state, + authed_request(CreateSandboxRequest { + name: "mcp-canonical".to_string(), + spec: Some(SandboxSpec { + policy: Some(mcp_policy_with_versions(&["2025-11-25", "2025-03-26"])), + ..Default::default() + }), + labels: HashMap::new(), + annotations: HashMap::new(), + workspace_scope: Some(openshell_core::proto::workspace_selector("default")), + ..Default::default() + }), + ) + .await + .expect("supported MCP versions must be accepted"); + + let stored = state + .store + .get_message_by_name::("default", "mcp-canonical") + .await + .expect("stored sandbox lookup must succeed") + .expect("created sandbox must be persisted"); + let policy = stored + .spec + .expect("sandbox spec") + .policy + .expect("sandbox policy"); + let versions = &policy.network_policies["mcp"].endpoints[0] + .mcp + .as_ref() + .expect("MCP options") + .versions; + assert_eq!(versions, &["2025-03-26", "2025-11-25"]); + } + + #[tokio::test] + async fn create_sandbox_materializes_default_mcp_version_before_persistence() { + let state = test_server_state().await; + let cases = [ + ("mcp-def-no-opts", None), + ( + "mcp-def-empty", + Some(openshell_core::proto::McpOptions::default()), + ), + ( + "mcp-def-explicit", + Some(openshell_core::proto::McpOptions { + versions: vec!["2025-11-25".to_string()], + ..Default::default() + }), + ), + ]; + let mut expected_policy = None; + let mut expected_bytes = None; + + for (sandbox_name, mcp) in cases { + handle_create_sandbox( + &state, + authed_request(CreateSandboxRequest { + name: sandbox_name.to_string(), + spec: Some(SandboxSpec { + policy: Some(mcp_policy_with_options(mcp)), + ..Default::default() + }), + labels: HashMap::new(), + annotations: HashMap::new(), + workspace_scope: Some(openshell_core::proto::workspace_selector("default")), + ..Default::default() + }), + ) + .await + .expect("defaultable MCP versions must be accepted"); + + let stored = state + .store + .get_message_by_name::("default", sandbox_name) + .await + .expect("stored sandbox lookup must succeed") + .expect("created sandbox must be persisted"); + let policy = stored + .spec + .expect("sandbox spec") + .policy + .expect("sandbox policy"); + let options = policy.network_policies["mcp"].endpoints[0] + .mcp + .as_ref() + .expect("MCP options must be materialized before persistence"); + assert_eq!(options.versions, ["2025-11-25"], "{sandbox_name}"); + + let encoded = policy.encode_to_vec(); + if let Some(expected) = expected_policy.as_ref() { + assert_eq!( + &policy, expected, + "default spellings must persist the same canonical policy: {sandbox_name}" + ); + assert_eq!( + encoded, + *expected_bytes + .as_ref() + .expect("canonical policy bytes accompany the expected policy"), + "default spellings must persist identical policy bytes: {sandbox_name}" + ); + } else { + expected_policy = Some(policy); + expected_bytes = Some(encoded); + } + } + } + + #[tokio::test] + async fn create_sandbox_bounds_canonical_mcp_policy_size_for_all_default_spellings() { + let state = test_server_state().await; + let max_policy_size = super::super::MAX_POLICY_SIZE; + + for (boundary, target_size, should_accept) in [ + ("fit", max_policy_size, true), + ("over", max_policy_size + 1, false), + ] { + let padding_name = mcp_padding_rule_name_for_policy_size(target_size); + let cases = [ + ("omitted", None), + ("empty", Some(openshell_core::proto::McpOptions::default())), + ( + "explicit", + Some(openshell_core::proto::McpOptions { + versions: vec!["2025-11-25".to_string()], + ..Default::default() + }), + ), + ]; + + for (spelling, mcp) in cases { + let sandbox_name = format!("mcp-{boundary}-{spelling}"); + let mut policy = mcp_policy_with_options(mcp); + openshell_policy::ensure_sandbox_process_identity(&mut policy); + policy.network_policies.insert( + "padding".to_string(), + openshell_core::proto::NetworkPolicyRule { + name: padding_name.clone(), + ..Default::default() + }, + ); + + let raw_size = policy.encoded_len(); + if spelling == "explicit" { + assert_eq!(raw_size, target_size, "{sandbox_name}"); + } else { + assert!(raw_size < target_size, "{sandbox_name}: {raw_size}"); + } + + let canonical = validate_and_canonicalize_policy(policy.clone()) + .expect("defaultable MCP policy must be canonicalizable"); + assert_eq!(canonical.encoded_len(), target_size, "{sandbox_name}"); + + let result = handle_create_sandbox( + &state, + authed_request(CreateSandboxRequest { + name: sandbox_name.clone(), + spec: Some(SandboxSpec { + policy: Some(policy), + ..Default::default() + }), + labels: HashMap::new(), + annotations: HashMap::new(), + workspace_scope: Some(openshell_core::proto::workspace_selector("default")), + ..Default::default() + }), + ) + .await; + + if should_accept { + result.expect("canonical policy at the size limit must be accepted"); + let stored = state + .store + .get_message_by_name::("default", &sandbox_name) + .await + .expect("stored sandbox lookup must succeed") + .expect("accepted sandbox must be persisted"); + let stored_policy = stored + .spec + .expect("sandbox spec") + .policy + .expect("sandbox policy"); + assert_eq!(stored_policy, canonical, "{sandbox_name}"); + assert_eq!( + stored_policy.encoded_len(), + max_policy_size, + "{sandbox_name}" + ); + } else { + let error = result.expect_err("oversized canonical policy must be rejected"); + assert_eq!(error.code(), tonic::Code::InvalidArgument, "{sandbox_name}"); + assert_eq!( + error.message(), + format!( + "policy serialized size exceeds maximum ({target_size} > {max_policy_size})" + ), + "{sandbox_name}" + ); + let stored = state + .store + .get_message_by_name::("default", &sandbox_name) + .await + .expect("stored sandbox lookup must succeed"); + assert!( + stored.is_none(), + "oversized policy must not persist sandbox {sandbox_name}" + ); + } + } + } + } + + #[tokio::test] + async fn create_sandbox_rejects_invalid_mcp_versions_without_persisting() { + let state = test_server_state().await; + let cases: &[(&str, &[&str])] = &[ + ("mcp-duplicate-versions", &["2025-11-25", "2025-11-25"]), + ("mcp-unsupported-version", &["2026-07-28"]), + ]; + + for &(sandbox_name, versions) in cases { + let error = handle_create_sandbox( + &state, + authed_request(CreateSandboxRequest { + name: sandbox_name.to_string(), + spec: Some(SandboxSpec { + policy: Some(mcp_policy_with_versions(versions)), + ..Default::default() + }), + labels: HashMap::new(), + annotations: HashMap::new(), + workspace_scope: Some(openshell_core::proto::workspace_selector("default")), + ..Default::default() + }), + ) + .await + .expect_err("invalid MCP versions must reject sandbox creation"); + + assert_eq!(error.code(), tonic::Code::InvalidArgument, "{sandbox_name}"); + let stored = state + .store + .get_message_by_name::("default", sandbox_name) + .await + .expect("stored sandbox lookup must succeed"); + assert!( + stored.is_none(), + "invalid MCP versions must not persist sandbox {sandbox_name}" + ); + } + } + #[tokio::test] async fn create_sandbox_persists_long_metadata_annotations() { let state = test_server_state().await; @@ -3559,11 +4559,12 @@ mod tests { &state, authed_request(CreateSandboxRequest { name: "annotated".to_string(), - spec: Some(openshell_core::proto::SandboxSpec::default()), + spec: Some(SandboxSpec::default()), labels: HashMap::new(), annotations: HashMap::from([(annotation_key.clone(), annotation_value.clone())]), - workspace: String::new(), + workspace_scope: Some(openshell_core::proto::workspace_selector("default")), await_main_process_attachment: false, + workload_template_name: String::new(), }), ) .await @@ -3583,7 +4584,7 @@ mod tests { &state, authed_request(GetSandboxRequest { name: "annotated".to_string(), - workspace: String::new(), + workspace_scope: Some(openshell_core::proto::workspace_selector("default")), }), ) .await @@ -3602,8 +4603,7 @@ mod tests { #[tokio::test] async fn create_and_get_preserve_partial_process_identity() { - let state = - test_server_state_with_driver(openshell_core::ComputeDriverKind::Docker.as_str()).await; + let state = test_server_state_with_driver("docker").await; let policy = openshell_core::proto::SandboxPolicy { version: 1, process: Some(openshell_core::proto::ProcessPolicy { @@ -3617,14 +4617,15 @@ mod tests { &state, authed_request(CreateSandboxRequest { name: "partial-id".to_string(), - spec: Some(openshell_core::proto::SandboxSpec { + spec: Some(SandboxSpec { policy: Some(policy), ..Default::default() }), labels: HashMap::new(), annotations: HashMap::new(), - workspace: String::new(), + workspace_scope: Some(openshell_core::proto::workspace_selector("default")), await_main_process_attachment: false, + workload_template_name: String::new(), }), ) .await @@ -3647,7 +4648,7 @@ mod tests { &state, authed_request(GetSandboxRequest { name: "partial-id".to_string(), - workspace: String::new(), + workspace_scope: Some(openshell_core::proto::workspace_selector("default")), }), ) .await @@ -3667,9 +4668,7 @@ mod tests { #[tokio::test] async fn create_and_get_preserve_partial_process_identity_for_kubernetes() { - let state = - test_server_state_with_driver(openshell_core::ComputeDriverKind::Kubernetes.as_str()) - .await; + let state = test_server_state_with_driver("kubernetes").await; let policy = openshell_core::proto::SandboxPolicy { version: 1, process: Some(openshell_core::proto::ProcessPolicy { @@ -3683,14 +4682,15 @@ mod tests { &state, authed_request(CreateSandboxRequest { name: "kube-partial-id".to_string(), - spec: Some(openshell_core::proto::SandboxSpec { + spec: Some(SandboxSpec { policy: Some(policy), ..Default::default() }), labels: HashMap::new(), annotations: HashMap::new(), - workspace: String::new(), + workspace_scope: Some(openshell_core::proto::workspace_selector("default")), await_main_process_attachment: false, + workload_template_name: String::new(), }), ) .await @@ -3717,11 +4717,12 @@ mod tests { &state, authed_request(CreateSandboxRequest { name: "bad-label".to_string(), - spec: Some(openshell_core::proto::SandboxSpec::default()), + spec: Some(SandboxSpec::default()), labels: HashMap::from([("team".to_string(), "x".repeat(512))]), annotations: HashMap::new(), - workspace: String::new(), + workspace_scope: Some(openshell_core::proto::workspace_selector("default")), await_main_process_attachment: false, + workload_template_name: String::new(), }), ) .await @@ -3747,14 +4748,15 @@ mod tests { &task_state, authed_request(CreateSandboxRequest { name: "guarded-create".to_string(), - spec: Some(openshell_core::proto::SandboxSpec { + spec: Some(SandboxSpec { providers: vec!["work-github".to_string()], ..Default::default() }), labels: HashMap::new(), annotations: HashMap::new(), - workspace: String::new(), + workspace_scope: Some(openshell_core::proto::workspace_selector("default")), await_main_process_attachment: false, + workload_template_name: String::new(), }), ) .await @@ -3779,6 +4781,863 @@ mod tests { ); } + #[tokio::test] + async fn sandbox_template_handlers_create_get_list_and_delete_workspace_resource() { + let state = test_server_state().await; + + let created = handle_create_sandbox_template( + &state, + authed_request(CreateSandboxTemplateRequest { + template: Some(test_workload_template("gpu-kata")), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), + }), + ) + .await + .expect("template create should succeed") + .into_inner() + .template + .expect("template response"); + + let metadata = created.metadata.as_ref().expect("metadata"); + assert_eq!(metadata.name, "gpu-kata"); + assert_eq!(metadata.workspace, "default"); + assert!(!metadata.id.is_empty()); + assert_ne!(metadata.resource_version, 0); + + let fetched = handle_get_sandbox_template( + &state, + authed_request(GetSandboxTemplateRequest { + name: "gpu-kata".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), + }), + ) + .await + .expect("template get should succeed") + .into_inner() + .template + .expect("fetched template"); + assert_eq!(fetched.object_name(), "gpu-kata"); + assert_eq!(fetched.object_workspace(), "default"); + + let listed = handle_list_sandbox_templates( + &state, + authed_request(ListSandboxTemplatesRequest { + limit: 100, + offset: 0, + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), + label_selector: String::new(), + }), + ) + .await + .expect("template list should succeed") + .into_inner() + .templates; + assert_eq!(listed.len(), 1); + assert_eq!(listed[0].object_name(), "gpu-kata"); + + let deleted = handle_delete_sandbox_template( + &state, + authed_request(DeleteSandboxTemplateRequest { + name: "gpu-kata".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), + }), + ) + .await + .expect("template delete should succeed") + .into_inner(); + assert!(deleted.deleted); + + let missing = handle_get_sandbox_template( + &state, + authed_request(GetSandboxTemplateRequest { + name: "gpu-kata".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), + }), + ) + .await + .expect_err("deleted template should not be fetchable"); + assert_eq!(missing.code(), tonic::Code::NotFound); + } + + #[tokio::test] + async fn sandbox_template_list_filters_by_label_selector() { + let state = test_server_state().await; + + let mut gpu = test_workload_template("gpu-kata"); + gpu.metadata + .as_mut() + .expect("metadata") + .labels + .insert("team".to_string(), "runtime".to_string()); + handle_create_sandbox_template( + &state, + authed_request(CreateSandboxTemplateRequest { + template: Some(gpu), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), + }), + ) + .await + .expect("gpu template create should succeed"); + + let mut cpu = test_workload_template("cpu-base"); + cpu.metadata + .as_mut() + .expect("metadata") + .labels + .insert("team".to_string(), "batch".to_string()); + handle_create_sandbox_template( + &state, + authed_request(CreateSandboxTemplateRequest { + template: Some(cpu), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), + }), + ) + .await + .expect("cpu template create should succeed"); + + let listed = handle_list_sandbox_templates( + &state, + authed_request(ListSandboxTemplatesRequest { + limit: 100, + offset: 0, + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), + label_selector: "team=runtime".to_string(), + }), + ) + .await + .expect("template list with label selector should succeed") + .into_inner() + .templates; + + assert_eq!(listed.len(), 1); + assert_eq!(listed[0].object_name(), "gpu-kata"); + } + + #[tokio::test] + async fn sandbox_template_create_rejects_whitespace_name() { + let state = test_server_state().await; + + let err = handle_create_sandbox_template( + &state, + authed_request(CreateSandboxTemplateRequest { + template: Some(test_workload_template(" gpu-kata ")), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), + }), + ) + .await + .expect_err("template names must be canonical DNS-1123 labels"); + + assert_eq!(err.code(), tonic::Code::InvalidArgument); + assert!(err.message().contains("template.metadata.name")); + + let listed = handle_list_sandbox_templates( + &state, + authed_request(ListSandboxTemplatesRequest { + limit: 100, + offset: 0, + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), + label_selector: String::new(), + }), + ) + .await + .expect("template list should succeed") + .into_inner() + .templates; + assert!(listed.is_empty()); + } + + #[tokio::test] + async fn sandbox_template_create_empty_workspace_ignores_metadata_workspace() { + use openshell_core::proto::CreateWorkspaceRequest; + + let state = test_server_state().await; + crate::grpc::workspace::handle_create_workspace( + &state, + Request::new(CreateWorkspaceRequest { + name: "beta".to_string(), + labels: HashMap::new(), + }), + ) + .await + .expect("beta workspace should be created"); + + let mut template = test_workload_template("copied-template"); + template.metadata.as_mut().unwrap().workspace = "beta".to_string(); + + let err = handle_create_sandbox_template( + &state, + authed_request(CreateSandboxTemplateRequest { + template: Some(template), + workspace_scope: Some(openshell_core::proto::workspace_selector("default")), + }), + ) + .await + .expect_err("empty request workspace must default to default, not metadata workspace"); + + assert_eq!(err.code(), tonic::Code::InvalidArgument); + assert!(err.message().contains("template.metadata.workspace")); + + let listed = handle_list_sandbox_templates( + &state, + authed_request(ListSandboxTemplatesRequest { + limit: 100, + offset: 0, + workspace_scope: Some(openshell_core::proto::workspace_selector( + "beta".to_string(), + )), + label_selector: String::new(), + }), + ) + .await + .expect("template list should succeed") + .into_inner() + .templates; + assert!(listed.is_empty()); + } + + #[tokio::test] + async fn sandbox_template_create_rejects_missing_spec_as_invalid_argument() { + let state = test_server_state().await; + let mut template = test_workload_template("missing-spec"); + template.spec = None; + + let err = handle_create_sandbox_template( + &state, + authed_request(CreateSandboxTemplateRequest { + template: Some(template), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), + }), + ) + .await + .expect_err("template create should reject missing spec"); + + assert_eq!(err.code(), tonic::Code::InvalidArgument); + assert!(err.message().contains("sandbox template spec")); + } + + #[test] + fn sandbox_template_validation_allows_positive_ready_within() { + let mut template = test_workload_template("gpu-kata"); + template.metadata.as_mut().unwrap().id = "template-gpu-kata".to_string(); + template.spec.as_mut().unwrap().desired_service_level = + Some(openshell_core::proto::SandboxServiceLevel { + startup: Some(openshell_core::proto::SandboxStartup { + ready_within: Some(prost_types::Duration { + seconds: 1, + nanos: 0, + }), + max_burst: 1, + }), + }); + + validate_sandbox_workload_template(&template).expect("positive ready_within should pass"); + } + + #[test] + fn sandbox_template_validation_rejects_non_positive_ready_within() { + for (duration, expected) in [ + ( + prost_types::Duration { + seconds: 0, + nanos: 0, + }, + "greater than zero", + ), + ( + prost_types::Duration { + seconds: -1, + nanos: 0, + }, + "greater than zero", + ), + ( + prost_types::Duration { + seconds: 0, + nanos: -1, + }, + "greater than zero", + ), + ] { + let mut template = test_workload_template("gpu-kata"); + template.metadata.as_mut().unwrap().id = "template-gpu-kata".to_string(); + template.spec.as_mut().unwrap().desired_service_level = + Some(openshell_core::proto::SandboxServiceLevel { + startup: Some(openshell_core::proto::SandboxStartup { + ready_within: Some(duration), + max_burst: 1, + }), + }); + + let err = validate_sandbox_workload_template(&template) + .expect_err("non-positive ready_within should be rejected"); + assert_eq!(err.code(), tonic::Code::InvalidArgument); + assert!(err.message().contains(expected), "{err:?}"); + assert!( + err.message() + .contains("template.spec.desired_service_level.startup.ready_within"), + "{err:?}" + ); + } + } + + #[test] + fn sandbox_template_validation_rejects_malformed_ready_within() { + for (duration, expected) in [ + ( + prost_types::Duration { + seconds: 1, + nanos: -1, + }, + "normalized", + ), + ( + prost_types::Duration { + seconds: 0, + nanos: 1_000_000_000, + }, + "valid protobuf Duration", + ), + ( + prost_types::Duration { + seconds: 315_576_000_001, + nanos: 0, + }, + "valid protobuf Duration", + ), + ] { + let mut template = test_workload_template("gpu-kata"); + template.metadata.as_mut().unwrap().id = "template-gpu-kata".to_string(); + template.spec.as_mut().unwrap().desired_service_level = + Some(openshell_core::proto::SandboxServiceLevel { + startup: Some(openshell_core::proto::SandboxStartup { + ready_within: Some(duration), + max_burst: 1, + }), + }); + + let err = validate_sandbox_workload_template(&template) + .expect_err("malformed ready_within should be rejected"); + assert_eq!(err.code(), tonic::Code::InvalidArgument); + assert!(err.message().contains(expected), "{err:?}"); + assert!( + err.message() + .contains("template.spec.desired_service_level.startup.ready_within"), + "{err:?}" + ); + } + } + + #[tokio::test] + async fn sandbox_template_create_rejects_workspace_quota() { + let state = test_server_state().await; + for index in 0..MAX_TEMPLATES_PER_WORKSPACE { + let mut template = test_workload_template(&format!("tmpl-{index}")); + let metadata = template.metadata.as_mut().expect("metadata"); + metadata.id = format!("template-{index}"); + metadata.workspace = "default".to_string(); + state.store.put_message(&template).await.unwrap(); + } + + let err = handle_create_sandbox_template( + &state, + authed_request(CreateSandboxTemplateRequest { + template: Some(test_workload_template("overflow")), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), + }), + ) + .await + .expect_err("template create must reject a full workspace"); + + assert_eq!(err.code(), tonic::Code::ResourceExhausted); + assert!(err.message().contains("1000 sandbox templates")); + } + + #[tokio::test] + async fn sandbox_template_create_enforces_workspace_quota_concurrently() { + let state = test_server_state().await; + for index in 0..(MAX_TEMPLATES_PER_WORKSPACE - 1) { + let mut template = test_workload_template(&format!("tmpl-{index}")); + let metadata = template.metadata.as_mut().expect("metadata"); + metadata.id = format!("template-{index}"); + metadata.workspace = "default".to_string(); + state.store.put_message(&template).await.unwrap(); + } + + let mut handles = vec![]; + for index in 0..8 { + let state = Arc::clone(&state); + let handle = tokio::spawn(async move { + handle_create_sandbox_template( + &state, + authed_request(CreateSandboxTemplateRequest { + template: Some(test_workload_template(&format!("overflow-{index}"))), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), + }), + ) + .await + }); + handles.push(handle); + } + + let results: Vec<_> = future::join_all(handles) + .await + .into_iter() + .map(|r| r.unwrap()) + .collect(); + + let successes = results.iter().filter(|r| r.is_ok()).count(); + let exhausted = results + .iter() + .filter(|r| { + r.as_ref() + .err() + .is_some_and(|e| e.code() == tonic::Code::ResourceExhausted) + }) + .count(); + + assert_eq!(successes, 1); + assert_eq!(exhausted, 7); + let count = state + .store + .count_in_workspace(SandboxWorkloadTemplate::object_type(), "default") + .await + .unwrap(); + assert_eq!(count, u64::from(MAX_TEMPLATES_PER_WORKSPACE)); + } + + #[test] + fn template_create_sandbox_spec_field_policy_is_exhaustive() { + assert_proto_fields_classified( + "openshell.v1.SandboxSpec", + &["policy", "providers", "command", "tty"], + &[ + "log_level", + "environment", + "template", + "resource_requirements", + ], + ); + } + + fn assert_proto_fields_classified( + message_name: &str, + copied_from_create_request: &[&str], + rejected_template_workload_overrides: &[&str], + ) { + let pool = prost_reflect::DescriptorPool::decode(openshell_core::FILE_DESCRIPTOR_SET) + .expect("decode descriptor set"); + let message = pool + .get_message_by_name(message_name) + .expect("message descriptor"); + let classified: std::collections::HashSet<&str> = copied_from_create_request + .iter() + .chain(rejected_template_workload_overrides.iter()) + .copied() + .collect(); + let actual: std::collections::HashSet = message + .fields() + .map(|field| field.name().to_string()) + .collect(); + + for field in &actual { + assert!( + classified.contains(field.as_str()), + "{message_name}.{field} is not classified for template-backed sandbox creates. \ + Add it to copied_from_create_request when callers own the create-time value, \ + or to rejected_template_workload_overrides when the workload template owns it." + ); + } + + for field in classified { + assert!( + actual.contains(field), + "{message_name}.{field} is classified for template-backed sandbox creates, \ + but the proto field no longer exists" + ); + } + } + + #[tokio::test] + async fn create_sandbox_from_workload_template_resolves_workload_and_preserves_governance() { + let state = test_server_state().await; + state + .store + .put_message(&test_provider("work-github", "github")) + .await + .unwrap(); + handle_create_sandbox_template( + &state, + authed_request(CreateSandboxTemplateRequest { + template: Some(test_workload_template("gpu-kata")), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), + }), + ) + .await + .expect("template create should succeed"); + + let mut policy = openshell_core::proto::SandboxPolicy { + version: 1, + ..Default::default() + }; + policy.network_policies.insert( + "example".to_string(), + openshell_core::proto::NetworkPolicyRule { + name: "example".to_string(), + ..Default::default() + }, + ); + + let created = handle_create_sandbox( + &state, + authed_request(CreateSandboxRequest { + name: "from-template".to_string(), + spec: Some(SandboxSpec { + providers: vec!["work-github".to_string()], + policy: Some(policy), + command: vec!["echo".to_string(), "template-create".to_string()], + tty: false, + ..Default::default() + }), + labels: HashMap::new(), + annotations: HashMap::new(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), + workload_template_name: "gpu-kata".to_string(), + await_main_process_attachment: false, + }), + ) + .await + .expect("sandbox create from template should succeed") + .into_inner() + .sandbox + .expect("created sandbox"); + + let provenance = created + .created_from_workload_template + .expect("template provenance"); + assert_eq!(provenance.name, "gpu-kata"); + assert!(!provenance.resource_version.is_empty()); + + let spec = created.spec.expect("resolved sandbox spec"); + assert_eq!(spec.providers, vec!["work-github".to_string()]); + assert!(spec.policy.is_some()); + assert_eq!( + spec.command, + vec!["echo".to_string(), "template-create".to_string()] + ); + assert!(!spec.tty); + assert_eq!( + spec.environment.get("FEATURE_FLAG"), + Some(&"on".to_string()) + ); + + let template = spec.template.expect("resolved inline template"); + assert_eq!(template.image, "registry.example.com/agent:latest"); + let limits = template + .resources + .as_ref() + .and_then(|resources| resources.fields.get("limits")) + .and_then(|limits| limits.kind.as_ref()) + .and_then(|kind| match kind { + Kind::StructValue(value) => Some(&value.fields), + _ => None, + }) + .expect("resource limits"); + assert_eq!(limits.get("cpu").and_then(proto_string_value), Some("2")); + assert_eq!( + limits.get("memory").and_then(proto_string_value), + Some("4Gi") + ); + assert_eq!( + spec.resource_requirements + .and_then(|requirements| requirements.gpu) + .and_then(|gpu| gpu.count), + Some(1) + ); + } + + #[tokio::test] + async fn create_sandbox_from_workload_template_defaults_whitespace_image() { + let state = test_server_state().await; + let mut template = test_workload_template("default-image"); + template + .spec + .as_mut() + .and_then(|spec| spec.workload.as_mut()) + .expect("test template workload") + .image = " ".to_string(); + handle_create_sandbox_template( + &state, + authed_request(CreateSandboxTemplateRequest { + template: Some(template), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), + }), + ) + .await + .expect("template create should succeed"); + + let created = handle_create_sandbox( + &state, + authed_request(CreateSandboxRequest { + name: "from-template".to_string(), + spec: Some(SandboxSpec::default()), + labels: HashMap::new(), + annotations: HashMap::new(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), + workload_template_name: "default-image".to_string(), + await_main_process_attachment: false, + }), + ) + .await + .expect("sandbox create from template should succeed") + .into_inner() + .sandbox + .expect("created sandbox"); + + let image = created + .spec + .and_then(|spec| spec.template) + .map(|template| template.image) + .expect("resolved template image"); + assert_eq!(image, state.compute.default_image()); + } + + #[tokio::test] + async fn create_sandbox_from_workload_template_preserves_default_gpu_request() { + let state = test_server_state().await; + let mut template = test_workload_template("default-gpu"); + template + .spec + .as_mut() + .and_then(|spec| spec.workload.as_mut()) + .and_then(|workload| workload.resources.as_mut()) + .expect("test template resources") + .gpu = Some(GpuResourceRequirements { count: None }); + handle_create_sandbox_template( + &state, + authed_request(CreateSandboxTemplateRequest { + template: Some(template), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), + }), + ) + .await + .expect("template create should succeed"); + + let created = handle_create_sandbox( + &state, + authed_request(CreateSandboxRequest { + name: "from-template".to_string(), + spec: Some(SandboxSpec::default()), + labels: HashMap::new(), + annotations: HashMap::new(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), + workload_template_name: "default-gpu".to_string(), + await_main_process_attachment: false, + }), + ) + .await + .expect("sandbox create from template should succeed") + .into_inner() + .sandbox + .expect("created sandbox"); + + let gpu = created + .spec + .as_ref() + .and_then(|spec| spec.resource_requirements.as_ref()) + .and_then(|requirements| requirements.gpu.as_ref()) + .expect("default GPU request should be preserved"); + assert_eq!(gpu.count, None); + } + + #[tokio::test] + async fn create_sandbox_from_corrupted_workload_template_returns_internal() { + let state = test_server_state().await; + let mut template = test_workload_template("corrupt-template"); + let metadata = template.metadata.as_mut().expect("metadata"); + metadata.id = "template-corrupt-template".to_string(); + metadata.workspace = "default".to_string(); + template.spec = None; + state.store.put_message(&template).await.unwrap(); + + let err = handle_create_sandbox( + &state, + authed_request(CreateSandboxRequest { + name: "from-corrupt".to_string(), + spec: Some(SandboxSpec::default()), + labels: HashMap::new(), + annotations: HashMap::new(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), + workload_template_name: "corrupt-template".to_string(), + await_main_process_attachment: false, + }), + ) + .await + .expect_err("corrupted stored template should fail as server data corruption"); + + assert_eq!(err.code(), tonic::Code::Internal, "{}", err.message()); + assert!(err.message().contains("sandbox template spec")); + } + + #[tokio::test] + async fn create_sandbox_from_workload_template_rejects_inline_workload_overrides() { + let state = test_server_state().await; + handle_create_sandbox_template( + &state, + authed_request(CreateSandboxTemplateRequest { + template: Some(test_workload_template("gpu-kata")), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), + }), + ) + .await + .expect("template create should succeed"); + + let err = handle_create_sandbox( + &state, + authed_request(CreateSandboxRequest { + name: "bad-template-create".to_string(), + spec: Some(SandboxSpec { + environment: HashMap::from([("INLINE".to_string(), "blocked".to_string())]), + ..Default::default() + }), + labels: HashMap::new(), + annotations: HashMap::new(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), + workload_template_name: "gpu-kata".to_string(), + await_main_process_attachment: false, + }), + ) + .await + .expect_err("inline workload overrides should be rejected"); + + assert_eq!(err.code(), tonic::Code::InvalidArgument); + assert!(err.message().contains("spec.environment")); + } + + #[tokio::test] + async fn create_sandbox_from_workload_template_rejects_malformed_template_name() { + let state = test_server_state().await; + + let err = handle_create_sandbox( + &state, + authed_request(CreateSandboxRequest { + name: "bad-template-create".to_string(), + spec: Some(SandboxSpec::default()), + labels: HashMap::new(), + annotations: HashMap::new(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), + workload_template_name: "Invalid_Template_Name".to_string(), + await_main_process_attachment: false, + }), + ) + .await + .expect_err("malformed template name should be rejected before lookup"); + + assert_eq!(err.code(), tonic::Code::InvalidArgument); + assert!(err.message().contains("workload_template_name")); + } + + #[tokio::test] + async fn create_sandbox_from_workload_template_rejects_oversized_governance_before_lookup() { + let state = test_server_state().await; + + let err = handle_create_sandbox( + &state, + authed_request(CreateSandboxRequest { + name: "bad-template-create".to_string(), + spec: Some(SandboxSpec { + providers: (0..=MAX_PROVIDERS).map(|i| format!("p-{i}")).collect(), + ..Default::default() + }), + labels: HashMap::new(), + annotations: HashMap::new(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), + workload_template_name: "missing-template".to_string(), + await_main_process_attachment: false, + }), + ) + .await + .expect_err("oversized governance spec should be rejected before template lookup"); + + assert_eq!(err.code(), tonic::Code::InvalidArgument); + assert!(err.message().contains("providers")); + } + + #[tokio::test] + async fn create_sandbox_rejects_oversized_direct_spec_before_workspace_lookup() { + let state = test_server_state().await; + + let err = handle_create_sandbox( + &state, + authed_request(CreateSandboxRequest { + name: "bad-direct-create".to_string(), + spec: Some(SandboxSpec { + providers: (0..=MAX_PROVIDERS).map(|i| format!("p-{i}")).collect(), + ..Default::default() + }), + labels: HashMap::new(), + annotations: HashMap::new(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "missing-workspace".to_string(), + )), + workload_template_name: String::new(), + await_main_process_attachment: false, + }), + ) + .await + .expect_err("oversized direct spec should be rejected before workspace lookup"); + + assert_eq!(err.code(), tonic::Code::InvalidArgument); + assert!(err.message().contains("providers")); + } + #[tokio::test] async fn attach_sandbox_provider_rejects_credential_key_collisions() { let state = test_server_state().await; @@ -3804,7 +5663,7 @@ mod tests { sandbox_name: "work".to_string(), provider_name: "provider-b".to_string(), expected_resource_version: 0, - workspace: String::new(), + workspace_scope: Some(openshell_core::proto::workspace_selector("default")), }), ) .await @@ -3851,7 +5710,7 @@ mod tests { sandbox_name: "work".to_string(), provider_name: "provider-31".to_string(), expected_resource_version: 0, - workspace: String::new(), + workspace_scope: Some(openshell_core::proto::workspace_selector("default")), }), ) .await @@ -3906,7 +5765,7 @@ mod tests { sandbox_name: "work".to_string(), provider_name: "provider-32".to_string(), expected_resource_version: 0, - workspace: String::new(), + workspace_scope: Some(openshell_core::proto::workspace_selector("default")), }), ) .await @@ -3953,7 +5812,7 @@ mod tests { sandbox_name: "work".to_string(), provider_name: long_name, expected_resource_version: 0, - workspace: String::new(), + workspace_scope: Some(openshell_core::proto::workspace_selector("default")), }), ) .await @@ -3980,7 +5839,7 @@ mod tests { sandbox_name: "work".to_string(), provider_name: long_name, expected_resource_version: 0, - workspace: String::new(), + workspace_scope: Some(openshell_core::proto::workspace_selector("default")), }), ) .await @@ -4179,7 +6038,7 @@ mod tests { sandbox_name: "work".to_string(), provider_name: "github".to_string(), expected_resource_version: current_version, - workspace: String::new(), + workspace_scope: Some(openshell_core::proto::workspace_selector("default")), }), ) .await @@ -4231,7 +6090,7 @@ mod tests { sandbox_name: "work".to_string(), provider_name: "github".to_string(), expected_resource_version: 99, - workspace: String::new(), + workspace_scope: Some(openshell_core::proto::workspace_selector("default")), }), ) .await @@ -4294,7 +6153,7 @@ mod tests { sandbox_name: "work".to_string(), provider_name: "github".to_string(), expected_resource_version: current_version, - workspace: String::new(), + workspace_scope: Some(openshell_core::proto::workspace_selector("default")), }), ) .await @@ -4346,7 +6205,7 @@ mod tests { sandbox_name: "work".to_string(), provider_name: "github".to_string(), expected_resource_version: 99, - workspace: String::new(), + workspace_scope: Some(openshell_core::proto::workspace_selector("default")), }), ) .await @@ -4427,7 +6286,7 @@ mod tests { sandbox_name: "work".to_string(), provider_name: format!("provider-{i}"), expected_resource_version: initial_version, - workspace: String::new(), + workspace_scope: Some(openshell_core::proto::workspace_selector("default")), }), ) .await @@ -4511,7 +6370,9 @@ mod tests { &state, authed_request(GetSandboxRequest { name: "shared-name".to_string(), - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), }), ) .await @@ -4524,7 +6385,9 @@ mod tests { &state, authed_request(GetSandboxRequest { name: "shared-name".to_string(), - workspace: "beta".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "beta".to_string(), + )), }), ) .await @@ -4539,8 +6402,9 @@ mod tests { limit: 100, offset: 0, label_selector: String::new(), - workspace: "default".to_string(), - all_workspaces: false, + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), }), ) .await @@ -4556,8 +6420,9 @@ mod tests { limit: 100, offset: 0, label_selector: String::new(), - workspace: "beta".to_string(), - all_workspaces: false, + workspace_scope: Some(openshell_core::proto::workspace_selector( + "beta".to_string(), + )), }), ) .await @@ -4580,8 +6445,9 @@ mod tests { limit: 100, offset: 0, label_selector: String::new(), - workspace: "default".to_string(), - all_workspaces: false, + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), }), ) .await @@ -4594,7 +6460,9 @@ mod tests { &state, authed_request(GetSandboxRequest { name: "shared-name".to_string(), - workspace: "beta".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "beta".to_string(), + )), }), ) .await @@ -4622,29 +6490,13 @@ mod tests { limit: 100, offset: 0, label_selector: String::new(), - workspace: String::new(), - all_workspaces: true, + workspace_scope: Some(openshell_core::proto::all_workspaces_selector()), }), ) .await .unwrap() .into_inner(); assert_eq!(listed.sandboxes.len(), 2); - - // all_workspaces with non-empty workspace is rejected. - let err = handle_list_sandboxes( - &state, - authed_request(ListSandboxesRequest { - limit: 100, - offset: 0, - label_selector: String::new(), - workspace: "default".to_string(), - all_workspaces: true, - }), - ) - .await - .unwrap_err(); - assert_eq!(err.code(), tonic::Code::InvalidArgument); } /// Non-members must receive `PERMISSION_DENIED` — never `NOT_FOUND` — when @@ -4680,8 +6532,8 @@ mod tests { let err = handle_create_sandbox( &state, non_member_request(CreateSandboxRequest { - workspace: "no-such-ws".into(), - spec: Some(openshell_core::proto::SandboxSpec::default()), + workspace_scope: Some(openshell_core::proto::workspace_selector("no-such-ws")), + spec: Some(SandboxSpec::default()), ..Default::default() }), ) @@ -4698,7 +6550,7 @@ mod tests { let err = handle_get_sandbox( &state, non_member_request(GetSandboxRequest { - workspace: "no-such-ws".into(), + workspace_scope: Some(openshell_core::proto::workspace_selector("no-such-ws")), name: "any".into(), }), ) @@ -4714,7 +6566,7 @@ mod tests { let err = handle_list_sandboxes( &state, non_member_request(ListSandboxesRequest { - workspace: "no-such-ws".into(), + workspace_scope: Some(openshell_core::proto::workspace_selector("no-such-ws")), ..Default::default() }), ) @@ -4730,7 +6582,7 @@ mod tests { let err = handle_list_sandbox_providers( &state, non_member_request(ListSandboxProvidersRequest { - workspace: "no-such-ws".into(), + workspace_scope: Some(openshell_core::proto::workspace_selector("no-such-ws")), ..Default::default() }), ) @@ -4746,7 +6598,7 @@ mod tests { let err = handle_attach_sandbox_provider( &state, non_member_request(AttachSandboxProviderRequest { - workspace: "no-such-ws".into(), + workspace_scope: Some(openshell_core::proto::workspace_selector("no-such-ws")), ..Default::default() }), ) @@ -4762,7 +6614,7 @@ mod tests { let err = handle_detach_sandbox_provider( &state, non_member_request(DetachSandboxProviderRequest { - workspace: "no-such-ws".into(), + workspace_scope: Some(openshell_core::proto::workspace_selector("no-such-ws")), ..Default::default() }), ) @@ -4779,7 +6631,7 @@ mod tests { let err = handle_delete_sandbox( &state, non_member_request(DeleteSandboxRequest { - workspace: "no-such-ws".into(), + workspace_scope: Some(openshell_core::proto::workspace_selector("no-such-ws")), name: "any".into(), }), ) @@ -4795,7 +6647,7 @@ mod tests { handle_stop_sandbox( &state, non_member_request(StopSandboxRequest { - workspace: "no-such-ws".into(), + workspace_scope: Some(openshell_core::proto::workspace_selector("no-such-ws")), name: "any".into(), }), ) @@ -4803,7 +6655,7 @@ mod tests { handle_start_sandbox( &state, non_member_request(StartSandboxRequest { - workspace: "no-such-ws".into(), + workspace_scope: Some(openshell_core::proto::workspace_selector("no-such-ws")), name: "any".into(), }), ) diff --git a/crates/openshell-server/src/grpc/service.rs b/crates/openshell-server/src/grpc/service.rs index 790e26d61..34c7d62d3 100644 --- a/crates/openshell-server/src/grpc/service.rs +++ b/crates/openshell-server/src/grpc/service.rs @@ -15,7 +15,10 @@ use tonic::{Request, Response, Status}; use uuid::Uuid; use crate::ServerState; -use crate::auth::workspace_authz::{MinWorkspaceRole, authorize_workspace, require_platform_admin}; +use crate::auth::workspace_authz::{ + AuthorizedWorkspaceScope, MinWorkspaceRole, authorize_list_workspace_selector, + authorize_workspace_selector, +}; use crate::persistence::{ObjectType, WriteCondition}; use crate::service_routing; @@ -28,11 +31,11 @@ pub(super) async fn handle_expose_service( ) -> Result, Status> { let principal = super::extract_principal(&request)?; let req = request.into_inner(); - let authz = authorize_workspace( + let authz = authorize_workspace_selector( &state.store, &state.admin_role, &principal, - &req.workspace, + req.workspace_scope.as_ref(), MinWorkspaceRole::User, ) .await?; @@ -147,11 +150,11 @@ pub(super) async fn handle_get_service( ) -> Result, Status> { let principal = super::extract_principal(&request)?; let req = request.into_inner(); - let authz = authorize_workspace( + let authz = authorize_workspace_selector( &state.store, &state.admin_role, &principal, - &req.workspace, + req.workspace_scope.as_ref(), MinWorkspaceRole::User, ) .await?; @@ -174,54 +177,53 @@ pub(super) async fn handle_list_services( ) -> Result, Status> { let principal = super::extract_principal(&request)?; let req = request.into_inner(); - if req.all_workspaces && !req.workspace.is_empty() { - return Err(Status::invalid_argument( - "all_workspaces and workspace are mutually exclusive", - )); - } if !req.sandbox.is_empty() { validate_endpoint_name("sandbox", &req.sandbox, MAX_SANDBOX_NAME_LEN)?; } let limit = super::clamp_limit(req.limit, 100, super::MAX_PAGE_SIZE); - let endpoints: Vec = if req.all_workspaces { - require_platform_admin(&state.admin_role, &principal)?; - if !req.sandbox.is_empty() { - return Err(Status::invalid_argument( - "sandbox filter is not supported with all_workspaces", - )); - } - state.store.list_all_messages(limit, req.offset).await - } else { - let authz = authorize_workspace( - &state.store, - &state.admin_role, - &principal, - &req.workspace, - MinWorkspaceRole::User, - ) - .await?; - let workspace = super::workspace::resolve_workspace(state.store.as_ref(), &authz.workspace) - .await? - .name; - if req.sandbox.is_empty() { - state - .store - .list_messages(&workspace, limit, req.offset) - .await + let scope = authorize_list_workspace_selector( + &state.store, + &state.admin_role, + &principal, + req.workspace_scope.as_ref(), + MinWorkspaceRole::User, + ) + .await?; + let endpoints: Vec = + if matches!(scope, AuthorizedWorkspaceScope::AllWorkspaces) { + if !req.sandbox.is_empty() { + return Err(Status::invalid_argument( + "sandbox filter is not supported with all_workspaces", + )); + } + state.store.list_all_messages(limit, req.offset).await } else { - state - .store - .list_messages_with_selector( - &workspace, - &format!("sandbox={}", req.sandbox), - limit, - req.offset, - ) - .await + let AuthorizedWorkspaceScope::Workspace(authz) = scope else { + unreachable!("all-workspaces scope handled above") + }; + let workspace = + super::workspace::resolve_workspace(state.store.as_ref(), &authz.workspace) + .await? + .name; + if req.sandbox.is_empty() { + state + .store + .list_messages(&workspace, limit, req.offset) + .await + } else { + state + .store + .list_messages_with_selector( + &workspace, + &format!("sandbox={}", req.sandbox), + limit, + req.offset, + ) + .await + } } - } - .map_err(|e| Status::internal(format!("list endpoints failed: {e}")))?; + .map_err(|e| Status::internal(format!("list endpoints failed: {e}")))?; let services = endpoints .into_iter() @@ -237,11 +239,11 @@ pub(super) async fn handle_delete_service( ) -> Result, Status> { let principal = super::extract_principal(&request)?; let req = request.into_inner(); - let authz = authorize_workspace( + let authz = authorize_workspace_selector( &state.store, &state.admin_role, &principal, - &req.workspace, + req.workspace_scope.as_ref(), MinWorkspaceRole::User, ) .await?; @@ -413,7 +415,9 @@ mod tests { service: "web".to_string(), target_port: 8080, domain: true, - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), }), ) .await @@ -427,8 +431,9 @@ mod tests { sandbox: "my-sandbox".to_string(), limit: 0, offset: 0, - workspace: "default".to_string(), - all_workspaces: false, + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), }), ) .await @@ -445,7 +450,9 @@ mod tests { authed_request(GetServiceRequest { sandbox: "my-sandbox".to_string(), service: "web".to_string(), - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), }), ) .await @@ -458,7 +465,9 @@ mod tests { authed_request(DeleteServiceRequest { sandbox: "my-sandbox".to_string(), service: "web".to_string(), - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), }), ) .await @@ -471,7 +480,9 @@ mod tests { authed_request(GetServiceRequest { sandbox: "my-sandbox".to_string(), service: "web".to_string(), - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), }), ) .await @@ -484,8 +495,9 @@ mod tests { sandbox: "my-sandbox".to_string(), limit: 0, offset: 0, - workspace: "default".to_string(), - all_workspaces: false, + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), }), ) .await @@ -509,7 +521,9 @@ mod tests { service: "web".to_string(), target_port: 8080, domain: true, - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), }), ) .await @@ -524,7 +538,9 @@ mod tests { service: "web".to_string(), target_port: 9090, domain: true, - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), }), ) .await @@ -549,8 +565,9 @@ mod tests { sandbox: "my-sandbox".to_string(), limit: 0, offset: 0, - workspace: "default".to_string(), - all_workspaces: false, + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), }), ) .await @@ -572,7 +589,9 @@ mod tests { service: "web".to_string(), target_port: 7070, domain: true, - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), }), ) .await @@ -588,7 +607,9 @@ mod tests { service: "web".to_string(), target_port: 8080, domain: true, - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), }), ) .await @@ -603,7 +624,9 @@ mod tests { service: "web".to_string(), target_port: 9090, domain: true, - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), }), ) .await @@ -626,7 +649,9 @@ mod tests { authed_request(GetServiceRequest { sandbox: "my-sandbox".to_string(), service: "web".to_string(), - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), }), ) .await @@ -687,7 +712,9 @@ mod tests { service: "web".to_string(), target_port: 8080, domain: true, - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), }), ) .await @@ -700,7 +727,9 @@ mod tests { service: "web".to_string(), target_port: 9090, domain: true, - workspace: "beta".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "beta".to_string(), + )), }), ) .await @@ -712,7 +741,9 @@ mod tests { authed_request(GetServiceRequest { sandbox: "my-sandbox".to_string(), service: "web".to_string(), - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), }), ) .await @@ -726,7 +757,9 @@ mod tests { authed_request(GetServiceRequest { sandbox: "my-sandbox".to_string(), service: "web".to_string(), - workspace: "beta".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "beta".to_string(), + )), }), ) .await @@ -741,8 +774,9 @@ mod tests { sandbox: "my-sandbox".to_string(), limit: 100, offset: 0, - workspace: "default".to_string(), - all_workspaces: false, + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), }), ) .await @@ -760,8 +794,9 @@ mod tests { sandbox: "my-sandbox".to_string(), limit: 100, offset: 0, - workspace: "beta".to_string(), - all_workspaces: false, + workspace_scope: Some(openshell_core::proto::workspace_selector( + "beta".to_string(), + )), }), ) .await @@ -779,7 +814,9 @@ mod tests { authed_request(DeleteServiceRequest { sandbox: "my-sandbox".to_string(), service: "web".to_string(), - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), }), ) .await @@ -793,8 +830,9 @@ mod tests { sandbox: "my-sandbox".to_string(), limit: 100, offset: 0, - workspace: "default".to_string(), - all_workspaces: false, + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), }), ) .await @@ -807,7 +845,9 @@ mod tests { authed_request(GetServiceRequest { sandbox: "my-sandbox".to_string(), service: "web".to_string(), - workspace: "beta".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "beta".to_string(), + )), }), ) .await @@ -824,7 +864,9 @@ mod tests { service: "api".to_string(), target_port: 3000, domain: true, - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), }), ) .await @@ -836,29 +878,13 @@ mod tests { sandbox: String::new(), limit: 100, offset: 0, - workspace: String::new(), - all_workspaces: true, + workspace_scope: Some(openshell_core::proto::all_workspaces_selector()), }), ) .await .unwrap() .into_inner(); assert_eq!(listed.services.len(), 2); - - // all_workspaces with non-empty workspace is rejected. - let err = handle_list_services( - &state, - authed_request(ListServicesRequest { - sandbox: String::new(), - limit: 100, - offset: 0, - workspace: "default".to_string(), - all_workspaces: true, - }), - ) - .await - .unwrap_err(); - assert_eq!(err.code(), tonic::Code::InvalidArgument); } /// Non-member callers must receive `PERMISSION_DENIED` — not `NOT_FOUND` — @@ -889,7 +915,7 @@ mod tests { let err = handle_expose_service( &state, non_member_request(ExposeServiceRequest { - workspace: "no-such-ws".into(), + workspace_scope: Some(openshell_core::proto::workspace_selector("no-such-ws")), ..Default::default() }), ) @@ -905,7 +931,7 @@ mod tests { let err = handle_get_service( &state, non_member_request(GetServiceRequest { - workspace: "no-such-ws".into(), + workspace_scope: Some(openshell_core::proto::workspace_selector("no-such-ws")), ..Default::default() }), ) @@ -921,7 +947,7 @@ mod tests { let err = handle_list_services( &state, non_member_request(ListServicesRequest { - workspace: "no-such-ws".into(), + workspace_scope: Some(openshell_core::proto::workspace_selector("no-such-ws")), ..Default::default() }), ) @@ -937,7 +963,7 @@ mod tests { let err = handle_delete_service( &state, non_member_request(DeleteServiceRequest { - workspace: "no-such-ws".into(), + workspace_scope: Some(openshell_core::proto::workspace_selector("no-such-ws")), ..Default::default() }), ) diff --git a/crates/openshell-server/src/grpc/validation.rs b/crates/openshell-server/src/grpc/validation.rs index eb53c80f2..dac34524c 100644 --- a/crates/openshell-server/src/grpc/validation.rs +++ b/crates/openshell-server/src/grpc/validation.rs @@ -4,13 +4,13 @@ //! Request validation helpers for the gRPC service. //! //! All functions in this module are pure — they take proto types or primitives -//! and return `Result<(), Status>`. No server state is required. +//! and return validated values or `Status` errors. No server state is required. #![allow(clippy::result_large_err)] // Validation returns Result<_, Status> use openshell_core::proto::{ CredentialHandle, ExecSandboxRequest, Provider, SandboxPolicy as ProtoSandboxPolicy, - SandboxTemplate, + SandboxSpec, SandboxTemplate, }; use prost::Message; use tonic::Status; @@ -150,26 +150,12 @@ pub(super) fn validate_dns1123_label(name: &str, field: &str) -> Result<(), Stat /// Validate field sizes on a `CreateSandboxRequest` before persisting. /// /// Returns `INVALID_ARGUMENT` on the first field that exceeds its limit. -pub(super) fn validate_sandbox_spec( - name: &str, - spec: &openshell_core::proto::SandboxSpec, -) -> Result<(), Status> { +pub(super) fn validate_sandbox_spec(name: &str, spec: &SandboxSpec) -> Result<(), Status> { // --- request.name --- - if !name.is_empty() && name.len() > MAX_ROUTABLE_NAME_LEN { - return Err(Status::invalid_argument(format!( - "name exceeds maximum length ({} > {MAX_ROUTABLE_NAME_LEN})", - name.len() - ))); - } - validate_dns1123_label(name, "name")?; + validate_sandbox_name(name)?; // --- spec.providers --- - if spec.providers.len() > MAX_PROVIDERS { - return Err(Status::invalid_argument(format!( - "providers list exceeds maximum ({} > {MAX_PROVIDERS})", - spec.providers.len() - ))); - } + validate_sandbox_provider_count(spec)?; // --- spec.log_level --- if spec.log_level.len() > MAX_LOG_LEVEL_LEN { @@ -203,6 +189,45 @@ pub(super) fn validate_sandbox_spec( } // --- spec.policy serialized size --- + validate_sandbox_policy_size(spec)?; + + Ok(()) +} + +pub(super) fn validate_sandbox_governance_spec( + name: &str, + spec: &SandboxSpec, +) -> Result<(), Status> { + validate_sandbox_name(name)?; + validate_sandbox_provider_count(spec)?; + if !spec.command.is_empty() { + validate_main_process_command(&spec.command)?; + } + validate_sandbox_policy_size(spec)?; + Ok(()) +} + +fn validate_sandbox_name(name: &str) -> Result<(), Status> { + if !name.is_empty() && name.len() > MAX_ROUTABLE_NAME_LEN { + return Err(Status::invalid_argument(format!( + "name exceeds maximum length ({} > {MAX_ROUTABLE_NAME_LEN})", + name.len() + ))); + } + validate_dns1123_label(name, "name") +} + +fn validate_sandbox_provider_count(spec: &SandboxSpec) -> Result<(), Status> { + if spec.providers.len() > MAX_PROVIDERS { + return Err(Status::invalid_argument(format!( + "providers list exceeds maximum ({} > {MAX_PROVIDERS})", + spec.providers.len() + ))); + } + Ok(()) +} + +fn validate_sandbox_policy_size(spec: &SandboxSpec) -> Result<(), Status> { if let Some(ref policy) = spec.policy { let size = policy.encoded_len(); if size > MAX_POLICY_SIZE { @@ -244,7 +269,7 @@ fn validate_main_process_command(command: &[String]) -> Result<(), Status> { Ok(()) } -fn validate_gpu_request_fields(spec: &openshell_core::proto::SandboxSpec) -> Result<(), Status> { +fn validate_gpu_request_fields(spec: &SandboxSpec) -> Result<(), Status> { if openshell_core::gpu::sandbox_gpu_count(spec.resource_requirements.as_ref()) == Some(0) { return Err(Status::invalid_argument("gpu count must be greater than 0")); } @@ -307,11 +332,36 @@ fn validate_sandbox_template(tmpl: &SandboxTemplate) -> Result<(), Status> { "template.driver_config serialized size exceeds maximum ({size} > {MAX_TEMPLATE_STRUCT_SIZE})" ))); } + reject_gateway_owned_driver_config_keys(s)?; } Ok(()) } +/// `driver_config` fields the gateway resolves and writes itself. +/// +/// A caller who could set these would hand a raw host path straight to a +/// privileged compute driver. Clients name a staging token instead, and the +/// gateway substitutes the path it allocated. +const GATEWAY_OWNED_DRIVER_CONFIG_KEYS: &[&str] = &["rootfs_tar_path"]; + +fn reject_gateway_owned_driver_config_keys(config: &prost_types::Struct) -> Result<(), Status> { + for (driver_name, value) in &config.fields { + let Some(prost_types::value::Kind::StructValue(driver_config)) = value.kind.as_ref() else { + continue; + }; + for key in GATEWAY_OWNED_DRIVER_CONFIG_KEYS { + if driver_config.fields.contains_key(*key) { + return Err(Status::invalid_argument(format!( + "template.driver_config.{driver_name}.{key} is set by the gateway \ + and cannot be supplied by the caller" + ))); + } + } + } + Ok(()) +} + /// Validate a `map` field: entry count, key length, value length. pub(super) fn validate_string_map( map: &std::collections::HashMap, @@ -853,6 +903,19 @@ pub(super) fn validate_policy_safety(policy: &ProtoSandboxPolicy) -> Result<(), Ok(()) } +/// Validate a policy and return the canonical value safe to hash and persist. +/// +/// Validation runs before canonicalization so sorting cannot hide duplicate or +/// unsupported MCP revisions. Callers must use the returned value because the +/// input may carry an equivalent but noncanonical revision order. +pub(super) fn validate_and_canonicalize_policy( + policy: ProtoSandboxPolicy, +) -> Result { + openshell_policy::validate_and_canonicalize_sandbox_policy(policy).map_err(|error| { + Status::invalid_argument(format!("policy contains unsafe content: {error}")) + }) +} + /// Validate that user-authored policy does not use provider-derived rule keys. pub(super) fn validate_no_reserved_provider_policy_keys( policy: &ProtoSandboxPolicy, @@ -2229,4 +2292,43 @@ mod tests { let err = validate_exec_request_fields(&req).unwrap_err(); assert!(err.message().contains("newline")); } + + fn driver_config(json: &str) -> prost_types::Struct { + let serde_json::Value::Object(fields) = + serde_json::from_str::(json).expect("valid json") + else { + panic!("driver_config test input must be a JSON object"); + }; + openshell_core::proto_struct::json_object_to_struct(fields).expect("encodable") + } + + /// The security boundary: only the gateway may name a host path for the + /// compute driver. A direct API request that supplies one is refused. + #[test] + fn rejects_caller_supplied_rootfs_tar_path() { + for json in [ + r#"{"vm":{"rootfs_tar_path":"/etc/passwd"}}"#, + r#"{"vm":{"rootfs_tar_path":"/dev/zero"}}"#, + // Driver-agnostic: no driver block may carry a gateway-owned key. + r#"{"docker":{"rootfs_tar_path":"/etc/shadow"}}"#, + ] { + let err = reject_gateway_owned_driver_config_keys(&driver_config(json)) + .expect_err("a caller-supplied rootfs_tar_path must be rejected"); + assert_eq!(err.code(), Code::InvalidArgument, "{json}: {err}"); + assert!(err.message().contains("rootfs_tar_path"), "{json}: {err}"); + } + } + + #[test] + fn accepts_driver_config_without_gateway_owned_keys() { + for json in [ + r#"{"vm":{"rootfs_tar_staging_token":"tok-abc"}}"#, + r#"{"vm":{"gpu_device_ids":["0000:2d:00.0"]}}"#, + r#"{"kubernetes":{"pod":{"nodeName":"gpu-1"}}}"#, + r"{}", + ] { + reject_gateway_owned_driver_config_keys(&driver_config(json)) + .unwrap_or_else(|err| panic!("{json} should be accepted: {err}")); + } + } } diff --git a/crates/openshell-server/src/grpc/workspace.rs b/crates/openshell-server/src/grpc/workspace.rs index d83ffab0e..546934a6e 100644 --- a/crates/openshell-server/src/grpc/workspace.rs +++ b/crates/openshell-server/src/grpc/workspace.rs @@ -12,11 +12,10 @@ use openshell_core::proto::datamodel::v1::{ObjectMeta, WorkspacePhase, Workspace use openshell_core::proto::{ AddWorkspaceMemberRequest, AddWorkspaceMemberResponse, CreateWorkspaceRequest, CreateWorkspaceResponse, DeleteWorkspaceRequest, DeleteWorkspaceResponse, GetWorkspaceRequest, - GetWorkspaceResponse, InferenceRoute, ListWorkspaceMembersRequest, - ListWorkspaceMembersResponse, ListWorkspacesRequest, ListWorkspacesResponse, Provider, - RemoveWorkspaceMemberRequest, RemoveWorkspaceMemberResponse, Sandbox, ServiceEndpoint, - SshSession, StoredProviderCredentialRefreshState, StoredProviderProfile, Workspace, - WorkspaceMember, WorkspaceRole, + GetWorkspaceResponse, ListWorkspaceMembersRequest, ListWorkspaceMembersResponse, + ListWorkspacesRequest, ListWorkspacesResponse, Provider, RemoveWorkspaceMemberRequest, + RemoveWorkspaceMemberResponse, Sandbox, SandboxWorkloadTemplate, ServiceEndpoint, SshSession, + Workspace, WorkspaceMember, WorkspaceRole, }; use prost::Message; use tonic::{Request, Response, Status}; @@ -28,6 +27,7 @@ use crate::persistence::{ DRAFT_CHUNK_OBJECT_TYPE, ObjectLabels, ObjectType, POLICY_OBJECT_TYPE, WriteCondition, current_time_ms, }; +use crate::storage_proto::{StoredProviderCredentialRefreshState, StoredProviderProfile}; use std::collections::HashMap; use super::{MAX_PAGE_SIZE, clamp_limit}; @@ -70,7 +70,7 @@ fn membership_filter_subject<'a>( } } -fn validate_workspace_name(name: &str) -> Result<(), Status> { +pub fn validate_workspace_name(name: &str) -> Result<(), Status> { if name.is_empty() { return Err(Status::invalid_argument("workspace name is required")); } @@ -375,6 +375,7 @@ pub(super) async fn handle_delete_workspace( let mut blocking = Vec::new(); for (object_type, label) in [ (Sandbox::object_type(), "sandbox"), + (SandboxWorkloadTemplate::object_type(), "sandbox template"), (Provider::object_type(), "provider"), (StoredProviderProfile::object_type(), "provider profile"), (ServiceEndpoint::object_type(), "service"), @@ -410,13 +411,7 @@ pub(super) async fn handle_delete_workspace( // Cascade-delete non-blocking resources before the final CAS delete. // This is safe without a transaction: the workspace is Terminating, so // ensure_active rejects new resource creation. If delete_if conflicts - // below, the retry will find no routes/members to delete and succeed. - state - .store - .delete_all_in_workspace(InferenceRoute::object_type(), &name) - .await - .map_err(|e| Status::internal(format!("delete inference routes failed: {e}")))?; - + // below, the retry will find no members to delete and succeed. state .store .delete_all_in_workspace(WorkspaceMember::object_type(), &name) @@ -819,6 +814,72 @@ mod tests { assert!(resp.deleted); } + #[tokio::test] + async fn delete_workspace_blocked_by_sandbox_template() { + let state = test_server_state().await; + + handle_create_workspace( + &state, + Request::new(CreateWorkspaceRequest { + name: "templated".to_string(), + labels: HashMap::new(), + }), + ) + .await + .unwrap(); + + let template = SandboxWorkloadTemplate { + metadata: Some(ObjectMeta { + id: "template-1".to_string(), + name: "gpu-kata".to_string(), + created_at_ms: 1_000_000, + labels: HashMap::new(), + annotations: HashMap::new(), + resource_version: 0, + workspace: "templated".to_string(), + deletion_timestamp_ms: 0, + }), + spec: None, + }; + state.store.put_message(&template).await.unwrap(); + + let err = handle_delete_workspace( + &state, + Request::new(DeleteWorkspaceRequest { + name: "templated".to_string(), + }), + ) + .await + .unwrap_err(); + assert_eq!(err.code(), Code::FailedPrecondition); + assert!( + err.message().contains("sandbox template"), + "error should name sandbox templates as blocking resources: {}", + err.message() + ); + + state + .store + .delete_by_name( + SandboxWorkloadTemplate::object_type(), + "templated", + "gpu-kata", + ) + .await + .unwrap(); + + let resp = handle_delete_workspace( + &state, + Request::new(DeleteWorkspaceRequest { + name: "templated".to_string(), + }), + ) + .await + .unwrap() + .into_inner(); + assert!(resp.deleted); + } + #[tokio::test] async fn delete_workspace_blocked_by_ssh_session() { let state = test_server_state().await; @@ -1458,64 +1519,6 @@ mod tests { assert!(err.message().contains("being deleted")); } - #[tokio::test] - async fn delete_workspace_cascade_deletes_inference_routes() { - let state = test_server_state().await; - - handle_create_workspace( - &state, - Request::new(CreateWorkspaceRequest { - name: "route-test".to_string(), - labels: HashMap::new(), - }), - ) - .await - .unwrap(); - - let route = InferenceRoute { - metadata: Some(ObjectMeta { - id: "route-1".to_string(), - name: "inference.local".to_string(), - created_at_ms: 1_000_000, - labels: HashMap::new(), - annotations: HashMap::new(), - resource_version: 0, - workspace: "route-test".to_string(), - deletion_timestamp_ms: 0, - }), - config: Some(openshell_core::proto::InferenceRouteConfig { - provider_name: "test-provider".to_string(), - model_id: "gpt-4o".to_string(), - timeout_secs: 0, - }), - version: 1, - }; - state.store.put_message(&route).await.unwrap(); - - // Inference route should NOT block workspace deletion. - let resp = handle_delete_workspace( - &state, - Request::new(DeleteWorkspaceRequest { - name: "route-test".to_string(), - }), - ) - .await - .unwrap() - .into_inner(); - assert!(resp.deleted); - - // Inference route should have been cascade-deleted. - let remaining: Vec = state - .store - .list_messages("route-test", 100, 0) - .await - .unwrap(); - assert!( - remaining.is_empty(), - "inference routes should be cascade-deleted with workspace" - ); - } - /// Non-member callers must receive `PERMISSION_DENIED` — not `NOT_FOUND` — /// when targeting a workspace that does not exist. Returning `NOT_FOUND` /// would create a CWE-203 workspace-name oracle. diff --git a/crates/openshell-server/src/inference.rs b/crates/openshell-server/src/inference.rs deleted file mode 100644 index b83fd6be4..000000000 --- a/crates/openshell-server/src/inference.rs +++ /dev/null @@ -1,3839 +0,0 @@ -// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -// SPDX-License-Identifier: Apache-2.0 - -#![allow(clippy::result_large_err)] // gRPC handlers return Result, Status> - -use openshell_core::inference::{ - VERTEX_AI_PROJECT_ID_KEY, VERTEX_AI_PUBLISHER_KEY, VERTEX_AI_REGION_KEY, -}; -use openshell_core::proto::{ - DeleteInferenceRouteRequest, DeleteInferenceRouteResponse, GetInferenceBundleRequest, - GetInferenceBundleResponse, GetInferenceRouteRequest, GetInferenceRouteResponse, - InferenceRoute, InferenceRouteConfig, Provider, ResolvedRoute, Sandbox, - SetInferenceRouteRequest, SetInferenceRouteResponse, ValidatedEndpoint, - inference_server::Inference, -}; -use openshell_core::{ObjectId, ObjectLabels, ObjectWorkspace}; -use openshell_providers::normalize_provider_type; -use openshell_router::config::ResolvedRoute as RouterResolvedRoute; -use openshell_router::{ValidationFailureKind, verify_backend_endpoint}; -use prost::Message as _; -use std::collections::HashMap; -use std::sync::Arc; -use std::time::Duration; -use tonic::{Request, Response, Status}; - -use crate::{ - ServerState, - auth::workspace_authz::{MinWorkspaceRole, authorize_workspace}, - persistence::{ObjectName, ObjectType, Store, WriteCondition, current_time_ms}, -}; - -#[derive(Debug)] -pub struct InferenceService { - state: Arc, -} - -impl InferenceService { - pub fn new(state: Arc) -> Self { - Self { state } - } -} - -const CLUSTER_INFERENCE_ROUTE_NAME: &str = "inference.local"; -const SANDBOX_SYSTEM_ROUTE_NAME: &str = "sandbox-system"; - -/// Map a request `route_name` to the canonical store key. -/// -/// Empty string defaults to `CLUSTER_INFERENCE_ROUTE_NAME` for backward compat. -fn effective_route_name(name: &str) -> Result<&str, Status> { - match name.trim() { - "" | "inference.local" => Ok(CLUSTER_INFERENCE_ROUTE_NAME), - "sandbox-system" => Ok(SANDBOX_SYSTEM_ROUTE_NAME), - other => Err(Status::invalid_argument(format!( - "unknown route_name '{other}'; expected 'inference.local' or 'sandbox-system'" - ))), - } -} - -impl ObjectType for InferenceRoute { - fn object_type() -> &'static str { - "inference_route" - } -} - -#[tonic::async_trait] -impl Inference for InferenceService { - async fn get_inference_bundle( - &self, - request: Request, - ) -> Result, Status> { - let sandbox_id = authorize_inference_bundle( - request - .extensions() - .get::(), - )?; - let sandbox: Sandbox = self - .state - .store - .get_message::(&sandbox_id) - .await - .map_err(|e| Status::internal(format!("fetch sandbox failed: {e}")))? - .ok_or_else(|| Status::not_found(format!("sandbox '{sandbox_id}' not found")))?; - let workspace = sandbox.object_workspace(); - resolve_inference_bundle_with_credentials( - self.state.store.as_ref(), - workspace, - Some(&self.state.credentials), - ) - .await - .map(Response::new) - } - - async fn set_inference_route( - &self, - request: Request, - ) -> Result, Status> { - let principal = crate::grpc::extract_principal(&request)?; - let req = request.into_inner(); - let authz = authorize_workspace( - &self.state.store, - &self.state.admin_role, - &principal, - &req.workspace, - MinWorkspaceRole::Admin, - ) - .await?; - let workspace = - crate::grpc::workspace::resolve_workspace(self.state.store.as_ref(), &authz.workspace) - .await? - .ensure_active()?; - let route_name = effective_route_name(&req.route_name)?; - let verify = !req.no_verify; - let route = upsert_cluster_inference_route_with_credentials( - self.state.store.as_ref(), - &workspace, - Some(&self.state.credentials), - route_name, - &req.provider_name, - &req.model_id, - req.timeout_secs, - verify, - ) - .await?; - - let config = route - .route - .config - .as_ref() - .ok_or_else(|| Status::internal("managed route missing config"))?; - - Ok(Response::new(SetInferenceRouteResponse { - provider_name: config.provider_name.clone(), - model_id: config.model_id.clone(), - version: route.route.version, - route_name: route_name.to_string(), - validation_performed: !route.validation.is_empty(), - validated_endpoints: route.validation, - timeout_secs: config.timeout_secs, - workspace, - })) - } - - async fn get_inference_route( - &self, - request: Request, - ) -> Result, Status> { - let principal = crate::grpc::extract_principal(&request)?; - let req = request.into_inner(); - let authz = authorize_workspace( - &self.state.store, - &self.state.admin_role, - &principal, - &req.workspace, - MinWorkspaceRole::User, - ) - .await?; - let workspace = - crate::grpc::workspace::resolve_workspace(self.state.store.as_ref(), &authz.workspace) - .await? - .name; - let route_name = effective_route_name(&req.route_name)?; - let route = self - .state - .store - .get_message_by_name::(&workspace, route_name) - .await - .map_err(|e| Status::internal(format!("fetch route failed: {e}")))? - .ok_or_else(|| { - Status::not_found(format!( - "inference route '{route_name}' is not configured in workspace '{workspace}'; run 'openshell inference set --provider --model '" - )) - })?; - - let config = route - .config - .as_ref() - .ok_or_else(|| Status::internal("managed route missing config"))?; - - if config.provider_name.trim().is_empty() || config.model_id.trim().is_empty() { - return Err(Status::failed_precondition( - "managed route is missing provider/model metadata", - )); - } - - Ok(Response::new(GetInferenceRouteResponse { - provider_name: config.provider_name.clone(), - model_id: config.model_id.clone(), - version: route.version, - route_name: route_name.to_string(), - timeout_secs: config.timeout_secs, - workspace, - })) - } - - async fn delete_inference_route( - &self, - request: Request, - ) -> Result, Status> { - let principal = crate::grpc::extract_principal(&request)?; - let req = request.into_inner(); - let authz = authorize_workspace( - &self.state.store, - &self.state.admin_role, - &principal, - &req.workspace, - MinWorkspaceRole::Admin, - ) - .await?; - let workspace = - crate::grpc::workspace::resolve_workspace(self.state.store.as_ref(), &authz.workspace) - .await? - .name; - let route_name = effective_route_name(&req.route_name)?; - let deleted = self - .state - .store - .delete_by_name(InferenceRoute::object_type(), &workspace, route_name) - .await - .map_err(|e| Status::internal(format!("delete route failed: {e}")))?; - Ok(Response::new(DeleteInferenceRouteResponse { deleted })) - } -} - -#[cfg(test)] -async fn upsert_cluster_inference_route( - store: &Store, - workspace: &str, - route_name: &str, - provider_name: &str, - model_id: &str, - timeout_secs: u64, - verify: bool, -) -> Result { - upsert_cluster_inference_route_with_credentials( - store, - workspace, - None, - route_name, - provider_name, - model_id, - timeout_secs, - verify, - ) - .await -} - -#[cfg(test)] -async fn upsert_inference_route( - store: &Store, - workspace: &str, - route_name: &str, - provider_name: &str, - model_id: &str, - timeout_secs: u64, - verify: bool, -) -> Result { - upsert_cluster_inference_route( - store, - workspace, - route_name, - provider_name, - model_id, - timeout_secs, - verify, - ) - .await -} - -#[allow(clippy::too_many_arguments)] -async fn upsert_cluster_inference_route_with_credentials( - store: &Store, - workspace: &str, - credentials: Option<&crate::credentials::CredentialRuntime>, - route_name: &str, - provider_name: &str, - model_id: &str, - timeout_secs: u64, - verify: bool, -) -> Result { - if provider_name.trim().is_empty() { - return Err(Status::invalid_argument("provider_name is required")); - } - if model_id.trim().is_empty() { - return Err(Status::invalid_argument("model_id is required")); - } - - let provider = store - .get_message_by_name::(workspace, provider_name) - .await - .map_err(|e| Status::internal(format!("fetch provider failed: {e}")))? - .ok_or_else(|| { - Status::failed_precondition(format!( - "provider '{provider_name}' not found in workspace '{workspace}'" - )) - })?; - let provider = resolve_provider_credentials(provider, credentials).await?; - - let resolved = resolve_provider_route(&provider, model_id)?; - let validation = if verify { - vec![verify_provider_endpoint(provider.object_name(), model_id, &resolved).await?] - } else { - Vec::new() - }; - - let config = build_inference_route_config(&provider, model_id, timeout_secs); - - let existing = store - .get_message_by_name::(workspace, route_name) - .await - .map_err(|e| Status::internal(format!("fetch route failed: {e}")))?; - - let now_ms = current_time_ms(); - - let (id, metadata, new_version, condition) = if let Some(existing) = existing { - let resource_version = existing.metadata.as_ref().map_or(0, |m| m.resource_version); - ( - existing.object_id().to_string(), - existing.metadata.clone(), - existing.version.saturating_add(1), - WriteCondition::MatchResourceVersion(resource_version), - ) - } else { - let new_id = uuid::Uuid::new_v4().to_string(); - let new_metadata = Some(openshell_core::proto::datamodel::v1::ObjectMeta { - id: new_id.clone(), - name: route_name.to_string(), - created_at_ms: now_ms, - labels: HashMap::new(), - resource_version: 0, - annotations: HashMap::new(), - workspace: workspace.to_string(), - deletion_timestamp_ms: 0, - }); - (new_id, new_metadata, 1, WriteCondition::MustCreate) - }; - - let route = InferenceRoute { - metadata, - config: Some(config), - version: new_version, - }; - - crate::grpc::validate_object_metadata(route.metadata.as_ref(), "inference_route")?; - - let labels_map = route.object_labels(); - let labels_json = if labels_map.as_ref().is_none_or(HashMap::is_empty) { - None - } else { - Some( - serde_json::to_string(&labels_map) - .map_err(|e| Status::internal(format!("failed to serialize labels: {e}")))?, - ) - }; - store - .put_if( - InferenceRoute::object_type(), - &id, - route_name, - workspace, - &route.encode_to_vec(), - labels_json.as_deref(), - condition, - ) - .await - .map_err(|e| crate::grpc::persistence_error_to_status(e, "upsert inference route"))?; - - Ok(UpsertedInferenceRoute { route, validation }) -} - -fn build_inference_route_config( - provider: &Provider, - model_id: &str, - timeout_secs: u64, -) -> InferenceRouteConfig { - InferenceRouteConfig { - provider_name: provider.object_name().to_string(), - model_id: model_id.to_string(), - timeout_secs, - } -} - -#[derive(Debug)] -struct ResolvedProviderRoute { - provider_type: String, - route: RouterResolvedRoute, -} - -#[derive(Debug)] -struct UpsertedInferenceRoute { - route: InferenceRoute, - validation: Vec, -} - -/// Infer the Vertex AI publisher segment from a model identifier. -/// -/// Currently only the `"anthropic"` result is consumed by -/// `resolve_vertex_ai_route` to select between the native Anthropic -/// Messages API (`rawPredict`) and the OpenAI-compatible endpoint. -/// Non-Anthropic publisher mappings (`meta`, `mistralai`, `ai21`, -/// `deepseek`, `google`) are maintained for forward compatibility -/// and documentation value — all non-Anthropic models route to the -/// same OpenAI-compatible endpoint regardless of publisher. -/// -/// Returns `None` for unrecognized models, which causes resolution to -/// fall back to the OpenAI-compatible endpoint -/// (`v1beta1/.../endpoints/openapi`). -fn infer_vertex_publisher(model_id: &str) -> Option<&'static str> { - if model_id.starts_with("claude-") { - Some("anthropic") - } else if model_id.starts_with("gemini-") - || model_id.starts_with("text-bison-") - || model_id.starts_with("chat-bison-") - { - Some("google") - } else if model_id.starts_with("llama-") { - Some("meta") - } else if model_id.starts_with("mistral-") || model_id.starts_with("codestral-") { - Some("mistralai") - } else if model_id.starts_with("jamba-") { - Some("ai21") - } else if model_id.starts_with("deepseek-") { - Some("deepseek") - } else { - None - } -} - -/// Return a required Vertex AI config value, or a `FailedPrecondition` status. -fn required_vertex_config<'a>( - config: &'a HashMap, - key: &str, -) -> Result<&'a str, Status> { - config - .get(key) - .map(String::as_str) - .filter(|v| !v.trim().is_empty()) - .ok_or_else(|| { - Status::failed_precondition(format!("Vertex AI provider requires {key} config")) - }) -} - -/// Validate a GCP project ID against the documented format. -/// -/// GCP project IDs must be 6–30 characters, start with a lowercase letter, -/// contain only lowercase letters, digits, and hyphens, and not end with a hyphen. -fn validate_gcp_project_id(value: &str) -> Result<(), Status> { - let valid = value.len() >= 6 - && value.len() <= 30 - && value.starts_with(|c: char| c.is_ascii_lowercase()) - && !value.ends_with('-') - && value - .chars() - .all(|c| c.is_ascii_lowercase() || c.is_ascii_digit() || c == '-'); - if valid { - Ok(()) - } else { - Err(Status::invalid_argument(format!( - "VERTEX_AI_PROJECT_ID has invalid format: {value:?}. \ - GCP project IDs must be 6-30 characters, start with a lowercase letter, \ - contain only lowercase letters, digits, and hyphens, and not end with a hyphen." - ))) - } -} - -/// Validate a GCP region/location value. -/// -/// Accepts the special keywords `global`, `us`, and `eu`, plus standard -/// regional patterns like `us-central1`, `europe-west4`, `us-east4-a`. -fn validate_gcp_region(value: &str) -> Result<(), Status> { - let lower = value.trim().to_ascii_lowercase(); - let valid = matches!(lower.as_str(), "global" | "us" | "eu") - || (lower - .chars() - .all(|c| c.is_ascii_lowercase() || c.is_ascii_digit() || c == '-') - && lower.contains('-') - && !lower.starts_with('-') - && !lower.ends_with('-')); - if valid { - Ok(()) - } else { - Err(Status::invalid_argument(format!( - "VERTEX_AI_REGION has invalid format: {value:?}. \ - Expected a GCP region (e.g. us-central1, europe-west4) \ - or one of: global, us, eu." - ))) - } -} - -/// Resolve the Vertex AI API host and normalized location from a configured region. -fn vertex_location_and_host(region: &str) -> (String, String) { - let location = region.trim().to_ascii_lowercase(); - let host = match location.as_str() { - "global" => "aiplatform.googleapis.com".to_string(), - "us" | "eu" => format!("aiplatform.{location}.rep.googleapis.com"), - _ => format!("{location}-aiplatform.googleapis.com"), - }; - (location, host) -} - -/// Reject Bedrock model ids that would produce ambiguous or malformed -/// upstream URL paths. -/// -/// AWS Bedrock encodes the model in `/model//invoke`, so the value -/// is interpolated directly into a URL path segment. Without -/// validation, a value containing `/`, `\`, percent escapes, query or -/// fragment delimiters, traversal segments, whitespace, or control -/// characters could break out of the path segment, smuggle a different -/// upstream route, or produce ambiguous/malformed paths upstream. -/// -/// Mirrors [`validate_vertex_model_id`] — Bedrock has the same exposure -/// for the same reason, and the contract is enforced again at the -/// router layer (`is_valid_bedrock_model_id`) as defense-in-depth. -fn validate_aws_bedrock_model_id(value: &str) -> Result<(), Status> { - let trimmed = value.trim(); - if trimmed.is_empty() { - return Err(Status::invalid_argument("model_id is required")); - } - if value != trimmed { - return Err(Status::invalid_argument(format!( - "AWS Bedrock model_id must not include leading or trailing whitespace: {value:?}" - ))); - } - if value.contains('/') || value.contains('\\') { - return Err(Status::invalid_argument(format!( - "AWS Bedrock model_id must not contain path separators: {value:?}" - ))); - } - if value.chars().any(|c| matches!(c, '?' | '#' | '%')) { - return Err(Status::invalid_argument(format!( - "AWS Bedrock model_id must not contain URL delimiters or percent escapes: {value:?}" - ))); - } - if value.contains("..") { - return Err(Status::invalid_argument(format!( - "AWS Bedrock model_id must not contain traversal segments: {value:?}" - ))); - } - if value.chars().any(|c| c.is_control() || c.is_whitespace()) { - return Err(Status::invalid_argument(format!( - "AWS Bedrock model_id must not contain whitespace or control characters: {value:?}" - ))); - } - Ok(()) -} - -fn validate_vertex_model_id(value: &str) -> Result<(), Status> { - let trimmed = value.trim(); - if trimmed.is_empty() { - return Err(Status::invalid_argument("model_id is required")); - } - if value != trimmed { - return Err(Status::invalid_argument(format!( - "Vertex AI model_id must not include leading or trailing whitespace: {value:?}" - ))); - } - if value.contains('/') || value.contains('\\') { - return Err(Status::invalid_argument(format!( - "Vertex AI model_id must not contain path separators: {value:?}" - ))); - } - if value.chars().any(|c| matches!(c, '?' | '#' | '%')) { - return Err(Status::invalid_argument(format!( - "Vertex AI model_id must not contain URL delimiters or percent escapes: {value:?}" - ))); - } - if value.contains("..") { - return Err(Status::invalid_argument(format!( - "Vertex AI model_id must not contain traversal segments: {value:?}" - ))); - } - if value.chars().any(|c| c.is_control() || c.is_whitespace()) { - return Err(Status::invalid_argument(format!( - "Vertex AI model_id must not contain whitespace or control characters: {value:?}" - ))); - } - Ok(()) -} - -fn is_allowed_vertex_override_host(host: &str) -> bool { - matches!( - host, - "aiplatform.googleapis.com" - | "aiplatform.us.rep.googleapis.com" - | "aiplatform.eu.rep.googleapis.com" - ) || host.ends_with("-aiplatform.googleapis.com") -} - -fn validate_vertex_base_url(value: &str) -> Result { - let trimmed = value.trim(); - let url = url::Url::parse(trimmed).map_err(|err| { - Status::invalid_argument(format!("Vertex AI base URL override is invalid: {err}")) - })?; - - if url.scheme() != "https" { - return Err(Status::invalid_argument( - "Vertex AI base URL override must use https".to_string(), - )); - } - if !url.username().is_empty() || url.password().is_some() { - return Err(Status::invalid_argument( - "Vertex AI base URL override must not include userinfo".to_string(), - )); - } - if url.query().is_some() || url.fragment().is_some() { - return Err(Status::invalid_argument( - "Vertex AI base URL override must not include query or fragment components".to_string(), - )); - } - if let Some(port) = url.port() - && port != 443 - { - return Err(Status::invalid_argument(format!( - "Vertex AI base URL override must use port 443 when an explicit port is set, got {port}" - ))); - } - - match url.host() { - Some(url::Host::Domain(host)) if is_allowed_vertex_override_host(host) => {} - Some(url::Host::Domain(host)) => { - return Err(Status::invalid_argument(format!( - "Vertex AI base URL override must target an official Vertex AI hostname, got {host:?}" - ))); - } - Some(url::Host::Ipv4(_) | url::Host::Ipv6(_)) => { - return Err(Status::invalid_argument(format!( - "Vertex AI base URL override must not use IP literal hosts: {}", - url.host_str().unwrap_or("") - ))); - } - None => { - return Err(Status::invalid_argument( - "Vertex AI base URL override must include a host".to_string(), - )); - } - } - - Ok(trimmed.to_string()) -} - -/// Build a [`RouterResolvedRoute`] for Vertex AI without duplicating the 15-field struct. -#[allow(clippy::too_many_arguments)] -fn build_vertex_route( - route_name: &str, - endpoint: String, - model_id: &str, - api_key: &str, - protocols: Vec, - profile: &openshell_core::inference::InferenceProviderProfile, - model_in_path: bool, - request_path_override: Option, -) -> RouterResolvedRoute { - RouterResolvedRoute { - name: route_name.to_string(), - endpoint, - model: model_id.to_string(), - api_key: api_key.to_string(), - protocols, - auth: profile.auth.clone(), - default_headers: profile - .default_headers - .iter() - .map(|(k, v)| ((*k).to_string(), (*v).to_string())) - .collect(), - passthrough_headers: profile - .passthrough_headers - .iter() - .map(|p| (*p).to_string()) - .collect(), - timeout: openshell_router::config::DEFAULT_ROUTE_TIMEOUT, - model_in_path, - request_path_override, - } -} - -/// Resolve a Vertex AI route given provider config, model, and bearer token. -fn resolve_vertex_ai_route( - config: &HashMap, - model_id: &str, - route_name: &str, - api_key: &str, - profile: &openshell_core::inference::InferenceProviderProfile, -) -> Result { - // Validate model_id early — it appears in URL paths for Anthropic routes - // and in JSON request bodies for all routes. Rejecting path separators, - // traversal segments, and control characters up front is defense-in-depth. - validate_vertex_model_id(model_id)?; - - // Determine if this is an Anthropic model. - // Explicit VERTEX_AI_PUBLISHER=anthropic overrides inference. - // All non-Anthropic models route to the OpenAI-compatible endpoint. - let explicit_publisher = config - .get(VERTEX_AI_PUBLISHER_KEY) - .map(String::as_str) - .filter(|v| !v.trim().is_empty()); - - let is_anthropic = explicit_publisher.map_or_else( - || infer_vertex_publisher(model_id) == Some("anthropic"), - |p| p.eq_ignore_ascii_case("anthropic"), - ); - - // Vertex's OpenAI-compatible endpoint requires the request body's `model` - // field to carry a publisher prefix: `/` (e.g. - // `google/gemini-2.5-flash`). The publisher is taken from the explicit - // VERTEX_AI_PUBLISHER config value (when set to a non-Anthropic value) or - // inferred from the model name. For unrecognised models with no explicit - // publisher, the bare model ID is forwarded unchanged; Vertex will return - // a 400 in that case, which is the correct observable signal to the caller. - // Anthropic rawPredict routes encode the model in the URL path, not the - // body, so they are unaffected. - let body_model_id: String = if is_anthropic { - model_id.to_string() - } else { - let publisher = explicit_publisher.or_else(|| infer_vertex_publisher(model_id)); - publisher.map_or_else(|| model_id.to_string(), |p| format!("{p}/{model_id}")) - }; - - // Escape hatch: caller-supplied full base URL still uses the model-derived - // protocol and path contract, but only for the OpenAI-compatible Vertex surface. - // Anthropic-on-Vertex needs model-path shaping and body adaptation that a fully - // caller-controlled URL cannot safely preserve. - if let Some(base_url) = profile - .base_url_config_keys - .iter() - .find_map(|key| config.get(*key).filter(|v| !v.trim().is_empty())) - .map(String::as_str) - { - if is_anthropic { - return Err(Status::invalid_argument( - "Vertex AI base URL overrides are not supported for Anthropic models. \ - Remove GOOGLE_VERTEX_AI_BASE_URL / VERTEX_AI_BASE_URL and configure \ - VERTEX_AI_PROJECT_ID + VERTEX_AI_REGION instead." - .to_string(), - )); - } - let base_url = validate_vertex_base_url(base_url)?; - - return Ok(build_vertex_route( - route_name, - base_url, - &body_model_id, - api_key, - vec!["openai_chat_completions".to_string()], - profile, - false, - Some("/chat/completions".to_string()), - )); - } - - let project = required_vertex_config(config, VERTEX_AI_PROJECT_ID_KEY)?; - validate_gcp_project_id(project)?; - let region = config - .get(VERTEX_AI_REGION_KEY) - .map(String::as_str) - .filter(|v| !v.trim().is_empty()) - .unwrap_or("us-central1"); - validate_gcp_region(region)?; - let (location, host) = vertex_location_and_host(region); - - if is_anthropic { - // Native Anthropic Messages API via rawPredict. - // model_id is NOT embedded in the endpoint — it is carried in route.model - // and appended with the suffix by build_provider_url(). The router upgrades - // `:rawPredict` to `:streamRawPredict` only for streaming proxy calls. - let endpoint = format!( - "https://{host}/v1/projects/{project}/locations/{location}/publishers/anthropic/models" - ); - let protocols = vec!["anthropic_messages".to_string()]; - Ok(build_vertex_route( - route_name, - endpoint, - model_id, - api_key, - protocols, - profile, - true, - Some(":rawPredict".to_string()), - )) - } else { - // OpenAI-compatible endpoint for all non-Anthropic models - // (Gemini, Llama, Mistral, unknown, etc.). Vertex's OpenAI-compatible - // surface uses `/chat/completions` under the `.../endpoints/openapi` - // base, so we pin the route to that path instead of appending the - // router's default `/v1/...` protocol path. - let endpoint = format!( - "https://{host}/v1beta1/projects/{project}/locations/{location}/endpoints/openapi" - ); - let protocols = vec!["openai_chat_completions".to_string()]; - Ok(build_vertex_route( - route_name, - endpoint, - &body_model_id, - api_key, - protocols, - profile, - false, - Some("/chat/completions".to_string()), - )) - } -} - -fn resolve_provider_route( - provider: &Provider, - model_id: &str, -) -> Result { - let raw_provider_type = provider.r#type.trim(); - let provider_type = normalize_provider_type(raw_provider_type) - .map_or_else(|| raw_provider_type.to_ascii_lowercase(), str::to_string); - - let profile = openshell_core::inference::profile_for(&provider_type).ok_or_else(|| { - Status::invalid_argument(format!( - "provider '{name}' has unsupported type '{raw_provider_type}' for cluster inference \ - (supported: openai, anthropic, nvidia, deepinfra, google-vertex-ai, aws-bedrock)", - name = provider.object_name() - )) - })?; - - // Profiles with `auth: None` are bridge-fronted — the upstream - // authenticates itself, so the router doesn't need a credential at - // route-resolution time. Today this is `aws-bedrock`. - let api_key = if matches!(profile.auth, openshell_core::inference::AuthHeader::None) { - String::new() - } else { - find_provider_api_key( - provider, - profile.credential_key_names, - if provider_type == "google-vertex-ai" { - CredentialLookup::PreferredOnly - } else { - CredentialLookup::PreferredThenAny - }, - ) - .ok_or_else(|| { - Status::invalid_argument(format!( - "provider '{name}' has no usable API key credential", - name = provider.object_name() - )) - })? - }; - - // Vertex AI requires a model-aware URL; delegate to specialised resolver. - if provider_type == "google-vertex-ai" { - let route = resolve_vertex_ai_route( - &provider.config, - model_id, - provider.object_name(), - &api_key, - profile, - )?; - return Ok(ResolvedProviderRoute { - provider_type, - route, - }); - } - - // AWS Bedrock encodes the model in the URL path - // (`/model//invoke`), so the model id is interpolated directly - // into a path segment by the router. Validate up front so the route - // store cannot hold a model id that would produce ambiguous or - // malformed upstream paths. Defense-in-depth: the router enforces - // the same contract again before constructing an upstream URL. - if provider_type == "aws-bedrock" { - validate_aws_bedrock_model_id(model_id)?; - } - - let base_url = find_provider_config_value(provider, profile.base_url_config_keys) - .unwrap_or_else(|| profile.default_base_url.to_string()) - .trim() - .to_string(); - - if base_url.is_empty() { - return Err(Status::invalid_argument(format!( - "provider '{name}' resolved to empty base_url", - name = provider.object_name() - ))); - } - - Ok(ResolvedProviderRoute { - provider_type, - route: RouterResolvedRoute { - name: provider.object_name().to_string(), - endpoint: base_url, - model: model_id.to_string(), - api_key, - protocols: profile.protocols.iter().map(|p| (*p).to_string()).collect(), - auth: profile.auth.clone(), - default_headers: profile - .default_headers - .iter() - .map(|(name, value)| ((*name).to_string(), (*value).to_string())) - .collect(), - passthrough_headers: profile - .passthrough_headers - .iter() - .map(|name| (*name).to_string()) - .collect(), - timeout: openshell_router::config::DEFAULT_ROUTE_TIMEOUT, - model_in_path: false, - request_path_override: None, - }, - }) -} - -fn validation_failure( - provider_name: &str, - model_id: &str, - base_url: &str, - details: &str, - next_steps: &str, -) -> Status { - Status::failed_precondition(format!( - "failed to verify inference endpoint for provider '{provider_name}' and model '{model_id}' at '{base_url}': {details}. Next steps: {next_steps}, or retry with '--no-verify' if you want to skip verification" - )) -} - -fn validation_next_steps(kind: ValidationFailureKind) -> &'static str { - match kind { - ValidationFailureKind::Credentials => { - "verify the provider API key and any required auth headers" - } - ValidationFailureKind::RateLimited => { - "retry later or verify quota/limits on the upstream provider" - } - ValidationFailureKind::RequestShape => { - "confirm the provider type, base URL, and model identifier" - } - ValidationFailureKind::Connectivity => { - "check that the service is running, confirm the base URL and protocol, and verify credentials" - } - ValidationFailureKind::UpstreamHealth => { - "check whether the endpoint is healthy and serving requests" - } - ValidationFailureKind::Unexpected => { - "confirm the endpoint URL, protocol, credentials, and model identifier" - } - } -} - -async fn verify_provider_endpoint( - provider_name: &str, - model_id: &str, - route: &ResolvedProviderRoute, -) -> Result { - let client = reqwest::Client::builder() - .timeout(Duration::from_secs(30)) - .build() - .map_err(|err| Status::internal(format!("build validation client failed: {err}")))?; - verify_backend_endpoint(&client, &route.route) - .await - .map(|validated| ValidatedEndpoint { - url: validated.url, - protocol: validated.protocol, - }) - .map_err(|err| { - validation_failure( - provider_name, - model_id, - &route.route.endpoint, - &err.details, - validation_next_steps(err.kind), - ) - }) -} - -/// Controls whether `find_provider_api_key` is allowed to fall back to any -/// non-empty credential when the preferred key names produce no match. -/// -/// `PreferredOnly` is used for providers like Vertex AI where the fallback -/// would pick up JSON bootstrap material (e.g. service account keys) that -/// are not valid bearer tokens. -#[derive(Debug, Clone, Copy, PartialEq, Eq)] -enum CredentialLookup { - /// Only search `preferred_key_names`. Return `None` if none match. - PreferredOnly, - /// Search `preferred_key_names` first, then fall back to any non-empty credential. - PreferredThenAny, -} - -fn find_provider_api_key( - provider: &Provider, - preferred_key_names: &[&str], - lookup: CredentialLookup, -) -> Option { - for key in preferred_key_names { - if let Some(value) = provider.credentials.get(*key) - && !value.trim().is_empty() - { - return Some(value.clone()); - } - } - - if lookup == CredentialLookup::PreferredOnly { - return None; - } - - let mut keys = provider.credentials.keys().collect::>(); - keys.sort(); - for key in keys { - if let Some(value) = provider.credentials.get(key) - && !value.trim().is_empty() - { - return Some(value.clone()); - } - } - - None -} - -fn find_provider_config_value(provider: &Provider, preferred_keys: &[&str]) -> Option { - for key in preferred_keys { - if let Some(value) = provider.config.get(*key) - && !value.trim().is_empty() - { - return Some(value.clone()); - } - } - None -} - -fn authorize_inference_bundle( - principal: Option<&crate::auth::principal::Principal>, -) -> Result { - match principal { - Some(crate::auth::principal::Principal::Sandbox(s)) => Ok(s.sandbox_id.clone()), - Some(crate::auth::principal::Principal::User(_)) => Err(Status::permission_denied( - "GetInferenceBundle requires a sandbox principal", - )), - Some(crate::auth::principal::Principal::Anonymous) | None => Err(Status::unauthenticated( - "GetInferenceBundle requires an authenticated sandbox principal", - )), - } -} - -/// Resolve the inference bundle (all managed routes + revision hash). -#[cfg(test)] -async fn resolve_inference_bundle( - store: &Store, - workspace: &str, -) -> Result { - resolve_inference_bundle_with_credentials(store, workspace, None).await -} - -async fn resolve_inference_bundle_with_credentials( - store: &Store, - workspace: &str, - credentials: Option<&crate::credentials::CredentialRuntime>, -) -> Result { - let mut routes = Vec::new(); - if let Some(r) = resolve_route_by_name_with_credentials( - store, - workspace, - credentials, - CLUSTER_INFERENCE_ROUTE_NAME, - ) - .await? - { - routes.push(r); - } - if let Some(r) = resolve_route_by_name_with_credentials( - store, - workspace, - credentials, - SANDBOX_SYSTEM_ROUTE_NAME, - ) - .await? - { - routes.push(r); - } - - let now_ms = i64::try_from( - std::time::SystemTime::now() - .duration_since(std::time::UNIX_EPOCH) - .unwrap_or_default() - .as_millis(), - ) - .unwrap_or(i64::MAX); - - // Compute a simple revision from route contents for cache freshness checks. - let revision = { - use std::hash::{Hash, Hasher}; - let mut hasher = std::collections::hash_map::DefaultHasher::new(); - for r in &routes { - r.name.hash(&mut hasher); - r.base_url.hash(&mut hasher); - r.model_id.hash(&mut hasher); - r.api_key.hash(&mut hasher); - r.protocols.hash(&mut hasher); - r.provider_type.hash(&mut hasher); - r.timeout_secs.hash(&mut hasher); - r.model_in_path.hash(&mut hasher); - r.request_path_override.hash(&mut hasher); - } - format!("{:016x}", hasher.finish()) - }; - - Ok(GetInferenceBundleResponse { - routes, - revision, - generated_at_ms: now_ms, - }) -} - -#[cfg(test)] -async fn resolve_route_by_name( - store: &Store, - workspace: &str, - route_name: &str, -) -> Result, Status> { - resolve_route_by_name_with_credentials(store, workspace, None, route_name).await -} - -async fn resolve_route_by_name_with_credentials( - store: &Store, - workspace: &str, - credentials: Option<&crate::credentials::CredentialRuntime>, - route_name: &str, -) -> Result, Status> { - let route = store - .get_message_by_name::(workspace, route_name) - .await - .map_err(|e| Status::internal(format!("fetch route failed: {e}")))?; - - let Some(route) = route else { - return Ok(None); - }; - - let Some(config) = route.config.as_ref() else { - return Ok(None); - }; - - if config.provider_name.trim().is_empty() { - return Err(Status::failed_precondition(format!( - "route '{route_name}' is missing provider_name" - ))); - } - - if config.model_id.trim().is_empty() { - return Err(Status::failed_precondition(format!( - "route '{route_name}' is missing model_id" - ))); - } - - let provider = store - .get_message_by_name::(workspace, &config.provider_name) - .await - .map_err(|e| Status::internal(format!("fetch provider failed: {e}")))? - .ok_or_else(|| { - Status::failed_precondition(format!( - "configured provider '{}' was not found in workspace '{workspace}'", - config.provider_name - )) - })?; - let provider = resolve_provider_credentials(provider, credentials).await?; - - let resolved = resolve_provider_route(&provider, &config.model_id)?; - - Ok(Some(ResolvedRoute { - name: route_name.to_string(), - base_url: resolved.route.endpoint, - model_id: resolved.route.model.clone(), - api_key: resolved.route.api_key, - protocols: resolved.route.protocols, - provider_type: resolved.provider_type, - timeout_secs: config.timeout_secs, - model_in_path: resolved.route.model_in_path, - request_path_override: resolved.route.request_path_override, - })) -} - -async fn resolve_provider_credentials( - mut provider: Provider, - credentials: Option<&crate::credentials::CredentialRuntime>, -) -> Result { - if provider.credential_handles.is_empty() { - return Ok(provider); - } - - let credentials = credentials.ok_or_else(|| { - Status::failed_precondition(format!( - "provider '{}' stores credentials as handles, but credential storage is unavailable", - provider.object_name() - )) - })?; - let resolved = credentials - .resolve_provider_handles(&provider, current_time_ms()) - .await?; - provider.credentials.extend(resolved.values); - - // Merge expiration times, keeping the earliest non-zero value - for (key, driver_expires_at_ms) in resolved.expires_at_ms { - let provider_expires_at_ms = provider - .credential_expires_at_ms - .get(&key) - .copied() - .unwrap_or(0); - - let effective_expires_at_ms = match (provider_expires_at_ms, driver_expires_at_ms) { - (0, driver) => driver, - (provider, 0) => provider, - (provider, driver) => provider.min(driver), - }; - - if effective_expires_at_ms > 0 { - provider - .credential_expires_at_ms - .insert(key, effective_expires_at_ms); - } - } - Ok(provider) -} - -#[cfg(test)] -mod tests { - use super::*; - use crate::auth::identity::{Identity, IdentityProvider}; - use crate::auth::principal::{ - Principal, SandboxIdentitySource, SandboxPrincipal, UserPrincipal, - }; - use openshell_core::ObjectId; - use wiremock::matchers::{body_partial_json, header, method, path}; - use wiremock::{Mock, MockServer, ResponseTemplate}; - - async fn test_store() -> Store { - crate::persistence::test_store().await - } - - fn test_user_principal() -> Principal { - Principal::User(UserPrincipal { - identity: Identity { - subject: "user-a".to_string(), - display_name: None, - roles: vec!["openshell-user".to_string()], - scopes: vec![], - provider: IdentityProvider::Oidc, - }, - }) - } - - fn test_sandbox_principal() -> Principal { - Principal::Sandbox(SandboxPrincipal { - sandbox_id: "sandbox-a".to_string(), - source: SandboxIdentitySource::BootstrapJwt { - issuer: "openshell-gateway:test".to_string(), - }, - trust_domain: Some("openshell".to_string()), - }) - } - - fn make_route(name: &str, provider_name: &str, model_id: &str) -> InferenceRoute { - InferenceRoute { - metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { - id: format!("id-{name}"), - name: name.to_string(), - created_at_ms: 1_000_000, - labels: HashMap::new(), - resource_version: 0, - annotations: HashMap::new(), - workspace: "default".to_string(), - deletion_timestamp_ms: 0, - }), - config: Some(InferenceRouteConfig { - provider_name: provider_name.to_string(), - model_id: model_id.to_string(), - timeout_secs: 0, - }), - version: 0, - } - } - - fn make_provider(name: &str, provider_type: &str, key_name: &str, key_value: &str) -> Provider { - Provider { - metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { - id: format!("provider-{name}"), - name: name.to_string(), - created_at_ms: 1_000_000, - labels: HashMap::new(), - resource_version: 0, - annotations: HashMap::new(), - workspace: "default".to_string(), - deletion_timestamp_ms: 0, - }), - r#type: provider_type.to_string(), - credentials: std::iter::once((key_name.to_string(), key_value.to_string())).collect(), - config: HashMap::new(), - credential_expires_at_ms: HashMap::new(), - profile_workspace: String::new(), - credential_handles: HashMap::new(), - } - } - - fn make_provider_with_base_url( - name: &str, - provider_type: &str, - key_name: &str, - key_value: &str, - base_url_key: &str, - base_url: &str, - ) -> Provider { - Provider { - config: std::iter::once((base_url_key.to_string(), base_url.to_string())).collect(), - ..make_provider(name, provider_type, key_name, key_value) - } - } - - #[test] - fn resolve_vertex_ai_route_handles_empty_base_url_keys() { - let config = HashMap::new(); - let profile = openshell_core::inference::InferenceProviderProfile { - provider_type: "google_vertex_ai", - default_base_url: "https://example.com", - protocols: &[], - credential_key_names: &[], - base_url_config_keys: &[], - auth: openshell_core::inference::AuthHeader::Bearer, - default_headers: &[], - passthrough_headers: &[], - }; - let result = resolve_vertex_ai_route(&config, "model-id", "route", "api-key", &profile); - // Empty base_url_config_keys must not panic. The route still errors because - // the minimal Vertex config is missing, so we only assert reachability. - assert!( - result.is_err(), - "expected missing config to error, got {result:?}" - ); - } - - #[test] - fn resolve_vertex_ai_route_skips_blank_preferred_for_fallback() { - let mut config = HashMap::new(); - config.insert("GOOGLE_VERTEX_AI_BASE_URL".to_string(), " ".to_string()); - config.insert( - "VERTEX_AI_BASE_URL".to_string(), - "https://us-central1-aiplatform.googleapis.com".to_string(), - ); - config.insert( - VERTEX_AI_PROJECT_ID_KEY.to_string(), - "my-project".to_string(), - ); - config.insert(VERTEX_AI_REGION_KEY.to_string(), "us-central1".to_string()); - - let profile = openshell_core::inference::InferenceProviderProfile { - provider_type: "google-vertex-ai", - default_base_url: "", - protocols: &[], - credential_key_names: &[], - base_url_config_keys: &["GOOGLE_VERTEX_AI_BASE_URL", "VERTEX_AI_BASE_URL"], - auth: openshell_core::inference::AuthHeader::Bearer, - default_headers: &[], - passthrough_headers: &[], - }; - - let result = resolve_vertex_ai_route(&config, "model-id", "route", "api-key", &profile); - assert!( - result.is_ok(), - "blank preferred key should fall back to valid alias: {result:?}" - ); - let route = result.unwrap(); - assert_eq!( - route.endpoint, - "https://us-central1-aiplatform.googleapis.com" - ); - } - - #[test] - fn inference_bundle_requires_sandbox_principal() { - let sandbox = test_sandbox_principal(); - assert!(authorize_inference_bundle(Some(&sandbox)).is_ok()); - - let user = test_user_principal(); - let err = authorize_inference_bundle(Some(&user)).expect_err("users cannot fetch bundle"); - assert_eq!(err.code(), tonic::Code::PermissionDenied); - - let err = authorize_inference_bundle(None).expect_err("missing principal rejected"); - assert_eq!(err.code(), tonic::Code::Unauthenticated); - } - - #[tokio::test] - async fn upsert_cluster_route_creates_and_increments_version() { - let store = test_store().await; - - let provider = make_provider("openai-dev", "openai", "OPENAI_API_KEY", "sk-test"); - store - .put_message(&provider) - .await - .expect("provider should persist"); - - let first = upsert_inference_route( - &store, - "default", - CLUSTER_INFERENCE_ROUTE_NAME, - "openai-dev", - "gpt-4o", - 0, - false, - ) - .await - .expect("first set should succeed"); - assert_eq!(first.route.object_name(), CLUSTER_INFERENCE_ROUTE_NAME); - - let second = upsert_inference_route( - &store, - "default", - CLUSTER_INFERENCE_ROUTE_NAME, - "openai-dev", - "gpt-4.1", - 0, - false, - ) - .await - .expect("second set should succeed"); - assert_eq!(second.route.object_id(), first.route.object_id()); - - let config = second.route.config.as_ref().expect("config"); - assert_eq!(config.provider_name, "openai-dev"); - assert_eq!(config.model_id, "gpt-4.1"); - } - - #[tokio::test] - async fn upsert_cluster_route_succeeds_for_aws_bedrock_with_bridge_url() { - // aws-bedrock is registered with `auth: AuthHeader::None` (the - // bridge-fronted shape) so route resolution does NOT require a - // real API key — but `provider create` still requires a - // non-empty credentials map at the gRPC layer, so operators - // pass a placeholder credential per the docs. The router - // ignores it on the outbound path. - // - // The other half of the contract is `BEDROCK_BASE_URL`: with - // `default_base_url: ""` in the core profile, providers - // without it fail route resolution rather than silently - // forwarding prompts to AWS Bedrock with no usable auth. This - // test pins down the success path. - let store = test_store().await; - - let provider = Provider { - metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { - id: "provider-bedrock-bridge".to_string(), - name: "bedrock-bridge".to_string(), - created_at_ms: 1_000_000, - labels: HashMap::new(), - resource_version: 0, - annotations: HashMap::new(), - workspace: "default".to_string(), - deletion_timestamp_ms: 0, - }), - r#type: "aws-bedrock".to_string(), - // Placeholder credential — the router ignores it because - // auth: None skips header injection. Mirrors the - // doc-recommended `--credential AWS_ACCESS_KEY_ID=unused-bridge-fronted-shape`. - credentials: std::iter::once(( - "AWS_ACCESS_KEY_ID".to_string(), - "unused-bridge-fronted-shape".to_string(), - )) - .collect(), - config: std::iter::once(( - "BEDROCK_BASE_URL".to_string(), - "http://bedrock-bridge.demo.svc.cluster.local:8080".to_string(), - )) - .collect(), - credential_expires_at_ms: HashMap::new(), - profile_workspace: String::new(), - credential_handles: HashMap::new(), - }; - store - .put_message(&provider) - .await - .expect("provider should persist"); - - let upserted = upsert_inference_route( - &store, - "default", - CLUSTER_INFERENCE_ROUTE_NAME, - "bedrock-bridge", - "anthropic.claude-3-5-sonnet-20241022-v2:0", - 0, - false, - ) - .await - .expect("upsert should succeed for aws-bedrock provider"); - - assert_eq!(upserted.route.object_name(), CLUSTER_INFERENCE_ROUTE_NAME); - let config = upserted.route.config.as_ref().expect("config"); - assert_eq!(config.provider_name, "bedrock-bridge"); - assert_eq!(config.model_id, "anthropic.claude-3-5-sonnet-20241022-v2:0"); - - // Verify the resolved route metadata reflects bridge-fronted - // auth (empty api_key + provider_type = "aws-bedrock"). Note - // the api_key is empty even though the provider has a - // credential — auth: None skips api-key lookup entirely. - let managed = resolve_route_by_name(&store, "default", CLUSTER_INFERENCE_ROUTE_NAME) - .await - .expect("route should resolve") - .expect("managed route should exist"); - assert_eq!(managed.provider_type, "aws-bedrock"); - assert_eq!( - managed.base_url, - "http://bedrock-bridge.demo.svc.cluster.local:8080" - ); - assert_eq!(managed.api_key, ""); - } - - #[tokio::test] - async fn upsert_cluster_route_rejects_aws_bedrock_without_bedrock_base_url() { - // The companion to upsert_cluster_route_succeeds_for_aws_bedrock_with_bridge_url: - // an aws-bedrock provider without BEDROCK_BASE_URL must be - // rejected at route resolution. This pins down the safety - // contract johntmyers asked for — until the SigV4 follow-up - // lands, the router must NOT silently forward prompts to AWS - // with auth: None. - // - // Mechanism: AWS_BEDROCK_PROFILE.default_base_url is "". When - // the provider has no BEDROCK_BASE_URL config, base_url - // resolves to empty, triggering the existing - // empty-base_url check in resolve_provider_route. - let store = test_store().await; - - let provider = Provider { - metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { - id: "provider-bedrock-misconfigured".to_string(), - name: "bedrock-misconfigured".to_string(), - created_at_ms: 1_000_000, - labels: HashMap::new(), - resource_version: 0, - annotations: HashMap::new(), - workspace: "default".to_string(), - deletion_timestamp_ms: 0, - }), - r#type: "aws-bedrock".to_string(), - credentials: std::iter::once(( - "AWS_ACCESS_KEY_ID".to_string(), - "unused-bridge-fronted-shape".to_string(), - )) - .collect(), - // Intentionally no BEDROCK_BASE_URL. - config: HashMap::new(), - credential_expires_at_ms: HashMap::new(), - profile_workspace: String::new(), - credential_handles: HashMap::new(), - }; - store - .put_message(&provider) - .await - .expect("provider should persist"); - - let err = upsert_inference_route( - &store, - "default", - CLUSTER_INFERENCE_ROUTE_NAME, - "bedrock-misconfigured", - "anthropic.claude-3-5-sonnet-20241022-v2:0", - 0, - false, - ) - .await - .expect_err("upsert should reject aws-bedrock provider without BEDROCK_BASE_URL"); - assert_eq!(err.code(), tonic::Code::InvalidArgument); - assert!( - err.message().contains("empty base_url"), - "error should name the missing base_url, got: {}", - err.message() - ); - } - - /// Bedrock route resolution must reject model ids that would - /// produce ambiguous or malformed upstream URL paths. The Vertex - /// suite has equivalent coverage; this is the Bedrock companion. - #[tokio::test] - async fn upsert_cluster_route_rejects_aws_bedrock_unsafe_model_id() { - let store = test_store().await; - - let provider = Provider { - metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { - id: "provider-bedrock-bridge".to_string(), - name: "bedrock-bridge".to_string(), - created_at_ms: 1_000_000, - labels: HashMap::new(), - resource_version: 0, - annotations: HashMap::new(), - workspace: "default".to_string(), - deletion_timestamp_ms: 0, - }), - r#type: "aws-bedrock".to_string(), - credentials: HashMap::new(), - config: std::iter::once(( - "BEDROCK_BASE_URL".to_string(), - "http://bedrock-bridge.demo.svc.cluster.local:8080".to_string(), - )) - .collect(), - credential_expires_at_ms: HashMap::new(), - profile_workspace: String::new(), - credential_handles: HashMap::new(), - }; - store - .put_message(&provider) - .await - .expect("provider should persist"); - - for unsafe_model in [ - "anthropic.claude/../../etc/passwd", - "back\\slash-id", - "model?injected=1", - "model#fragment", - "percent%2fencoded", - "model..v2", - " leading-space", - "trailing-space ", - "tab\there", - "newline\nhere", - ] { - let err = upsert_inference_route( - &store, - "default", - CLUSTER_INFERENCE_ROUTE_NAME, - "bedrock-bridge", - unsafe_model, - 0, - false, - ) - .await - .expect_err(unsafe_model); - assert_eq!( - err.code(), - tonic::Code::InvalidArgument, - "{unsafe_model:?} should fail with InvalidArgument" - ); - assert!( - err.message().contains("AWS Bedrock model_id"), - "error must name AWS Bedrock model_id for {unsafe_model:?}, got: {}", - err.message() - ); - } - } - - #[tokio::test] - async fn resolve_managed_route_returns_none_when_missing() { - let store = test_store().await; - - let route = resolve_route_by_name(&store, "default", CLUSTER_INFERENCE_ROUTE_NAME) - .await - .expect("resolution should not fail"); - assert!(route.is_none()); - } - - #[tokio::test] - async fn bundle_happy_path_returns_managed_route() { - let store = test_store().await; - - let provider = make_provider("openai-dev", "openai", "OPENAI_API_KEY", "sk-test"); - store - .put_message(&provider) - .await - .expect("persist provider"); - - let route = make_route(CLUSTER_INFERENCE_ROUTE_NAME, "openai-dev", "mock/model-a"); - store.put_message(&route).await.expect("persist route"); - - let resp = resolve_inference_bundle(&store, "default") - .await - .expect("bundle should resolve"); - - assert_eq!(resp.routes.len(), 1); - assert_eq!(resp.routes[0].name, CLUSTER_INFERENCE_ROUTE_NAME); - assert_eq!(resp.routes[0].model_id, "mock/model-a"); - assert_eq!(resp.routes[0].provider_type, "openai"); - assert_eq!(resp.routes[0].api_key, "sk-test"); - assert_eq!(resp.routes[0].base_url, "https://api.openai.com/v1"); - assert!(!resp.revision.is_empty()); - assert!(resp.generated_at_ms > 0); - } - - #[tokio::test] - async fn bundle_vertex_ai_anthropic_route_preserves_model_path_and_rawpredict() { - let store = test_store().await; - let config = [ - ( - "VERTEX_AI_PROJECT_ID".to_string(), - "my-gcp-project".to_string(), - ), - ("VERTEX_AI_REGION".to_string(), "us-central1".to_string()), - ] - .into_iter() - .collect(); - let provider = make_vertex_provider_with_config("vertex-dev", config); - store - .put_message(&provider) - .await - .expect("persist provider"); - let route = make_route( - CLUSTER_INFERENCE_ROUTE_NAME, - "vertex-dev", - "claude-3-5-sonnet@20241022", - ); - store.put_message(&route).await.expect("persist route"); - - let resp = resolve_inference_bundle(&store, "default") - .await - .expect("bundle should resolve"); - - assert_eq!(resp.routes.len(), 1); - let route = &resp.routes[0]; - assert_eq!(route.provider_type, "google-vertex-ai"); - assert_eq!(route.api_key, "ya29.test-token"); - assert_eq!(route.protocols, vec!["anthropic_messages"]); - assert!(route.model_in_path); - assert_eq!(route.request_path_override, Some(":rawPredict".to_string())); - assert_eq!(route.model_id, "claude-3-5-sonnet@20241022"); - assert_eq!( - route.base_url, - "https://us-central1-aiplatform.googleapis.com/v1/projects/my-gcp-project/locations/us-central1/publishers/anthropic/models" - ); - } - - #[tokio::test] - async fn bundle_vertex_ai_gemini_route_preserves_chat_completions_override() { - let store = test_store().await; - let config = [ - ( - "VERTEX_AI_PROJECT_ID".to_string(), - "my-gcp-project".to_string(), - ), - ("VERTEX_AI_REGION".to_string(), "us-central1".to_string()), - ] - .into_iter() - .collect(); - let provider = make_vertex_provider_with_config("vertex-dev", config); - store - .put_message(&provider) - .await - .expect("persist provider"); - let route = make_route( - CLUSTER_INFERENCE_ROUTE_NAME, - "vertex-dev", - "gemini-2.0-flash-001", - ); - store.put_message(&route).await.expect("persist route"); - - let resp = resolve_inference_bundle(&store, "default") - .await - .expect("bundle should resolve"); - - assert_eq!(resp.routes.len(), 1); - let route = &resp.routes[0]; - assert_eq!(route.provider_type, "google-vertex-ai"); - assert_eq!(route.api_key, "ya29.test-token"); - assert_eq!(route.protocols, vec!["openai_chat_completions"]); - assert!(!route.model_in_path); - assert_eq!( - route.request_path_override, - Some("/chat/completions".to_string()) - ); - assert_eq!(route.model_id, "google/gemini-2.0-flash-001"); - assert_eq!( - route.base_url, - "https://us-central1-aiplatform.googleapis.com/v1beta1/projects/my-gcp-project/locations/us-central1/endpoints/openapi" - ); - } - - #[tokio::test] - async fn bundle_vertex_ai_non_anthropic_model_id_carries_publisher_prefix() { - // Regression test: the bundle's model_id must carry the publisher prefix - // so the router sends e.g. "google/gemini-2.5-flash" in the request body, - // not the bare "gemini-2.5-flash" that Vertex AI rejects with HTTP 400. - let store = test_store().await; - let config = [ - ( - "VERTEX_AI_PROJECT_ID".to_string(), - "my-gcp-project".to_string(), - ), - ("VERTEX_AI_REGION".to_string(), "us-central1".to_string()), - ] - .into_iter() - .collect(); - let provider = make_vertex_provider_with_config("vertex-dev", config); - store - .put_message(&provider) - .await - .expect("persist provider"); - let route = make_route( - CLUSTER_INFERENCE_ROUTE_NAME, - "vertex-dev", - "gemini-2.5-flash", - ); - store.put_message(&route).await.expect("persist route"); - - let resp = resolve_inference_bundle(&store, "default") - .await - .expect("bundle should resolve"); - - assert_eq!(resp.routes.len(), 1); - assert_eq!( - resp.routes[0].model_id, "google/gemini-2.5-flash", - "bundle model_id must carry publisher prefix for non-Anthropic Vertex routes" - ); - } - - #[tokio::test] - async fn bundle_without_cluster_route_returns_empty_routes() { - let store = test_store().await; - - let resp = resolve_inference_bundle(&store, "default") - .await - .expect("bundle should resolve"); - assert!(resp.routes.is_empty()); - } - - #[tokio::test] - async fn bundle_revision_is_stable_for_same_route() { - let store = test_store().await; - - let provider = make_provider("openai-dev", "openai", "OPENAI_API_KEY", "sk-test"); - store - .put_message(&provider) - .await - .expect("persist provider"); - - let route = make_route( - CLUSTER_INFERENCE_ROUTE_NAME, - "openai-dev", - "mock/model-stable", - ); - store.put_message(&route).await.expect("persist route"); - - let resp1 = resolve_inference_bundle(&store, "default") - .await - .expect("first resolve"); - let resp2 = resolve_inference_bundle(&store, "default") - .await - .expect("second resolve"); - - assert_eq!( - resp1.revision, resp2.revision, - "same route should produce same revision" - ); - } - - #[tokio::test] - async fn resolve_managed_route_derives_from_provider() { - let store = test_store().await; - - let provider = Provider { - metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { - id: "provider-1".to_string(), - name: "openai-dev".to_string(), - created_at_ms: 1_000_000, - labels: HashMap::new(), - resource_version: 0, - annotations: HashMap::new(), - workspace: "default".to_string(), - deletion_timestamp_ms: 0, - }), - r#type: "openai".to_string(), - credentials: std::iter::once(("OPENAI_API_KEY".to_string(), "sk-test".to_string())) - .collect(), - config: std::iter::once(( - "OPENAI_BASE_URL".to_string(), - "https://station.example.com/v1".to_string(), - )) - .collect(), - credential_expires_at_ms: HashMap::new(), - profile_workspace: String::new(), - credential_handles: HashMap::new(), - }; - store - .put_message(&provider) - .await - .expect("provider should persist"); - - let route = InferenceRoute { - metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { - id: "r-1".to_string(), - name: CLUSTER_INFERENCE_ROUTE_NAME.to_string(), - created_at_ms: 1_000_000, - labels: HashMap::new(), - resource_version: 0, - annotations: HashMap::new(), - workspace: "default".to_string(), - deletion_timestamp_ms: 0, - }), - config: Some(InferenceRouteConfig { - provider_name: "openai-dev".to_string(), - model_id: "test/model".to_string(), - timeout_secs: 0, - }), - version: 1, - }; - store - .put_message(&route) - .await - .expect("route should persist"); - - let managed = resolve_route_by_name(&store, "default", CLUSTER_INFERENCE_ROUTE_NAME) - .await - .expect("route should resolve") - .expect("managed route should exist"); - - assert_eq!(managed.base_url, "https://station.example.com/v1"); - assert_eq!(managed.api_key, "sk-test"); - assert_eq!(managed.provider_type, "openai"); - assert_eq!( - managed.protocols, - vec![ - "openai_chat_completions".to_string(), - "openai_completions".to_string(), - "openai_responses".to_string(), - "openai_embeddings".to_string(), - "model_discovery".to_string(), - ] - ); - } - - #[tokio::test] - async fn managed_route_resolves_default_credential_handles() { - let store = test_store().await; - let credentials = crate::credentials::CredentialRuntime::from_config_with_store( - &openshell_core::Config::new(None), - Arc::new(store.clone()), - ) - .expect("credential runtime should connect to default encrypted store"); - let handles = credentials - .store_provider_credentials( - "openai-dev", - "default", - "provider-1", - &HashMap::from([("OPENAI_API_KEY".to_string(), "sk-encrypted".to_string())]), - &HashMap::new(), - ) - .await - .expect("credential should be stored"); - - let provider = Provider { - metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { - id: "provider-1".to_string(), - name: "openai-dev".to_string(), - created_at_ms: 1_000_000, - labels: HashMap::new(), - resource_version: 0, - workspace: "default".to_string(), - ..Default::default() - }), - r#type: "openai".to_string(), - credentials: HashMap::new(), - config: std::iter::once(( - "OPENAI_BASE_URL".to_string(), - "https://station.example.com/v1".to_string(), - )) - .collect(), - credential_expires_at_ms: HashMap::new(), - credential_handles: handles, - profile_workspace: String::new(), - }; - store - .put_message(&provider) - .await - .expect("provider should persist"); - - upsert_cluster_inference_route_with_credentials( - &store, - "default", - Some(&credentials), - CLUSTER_INFERENCE_ROUTE_NAME, - "openai-dev", - "test/model", - 0, - false, - ) - .await - .expect("route should be created from handle-backed provider"); - - let managed = resolve_route_by_name_with_credentials( - &store, - "default", - Some(&credentials), - CLUSTER_INFERENCE_ROUTE_NAME, - ) - .await - .expect("route should resolve") - .expect("managed route should exist"); - - assert_eq!(managed.base_url, "https://station.example.com/v1"); - assert_eq!(managed.api_key, "sk-encrypted"); - } - - #[tokio::test] - async fn resolve_managed_route_reflects_provider_key_rotation() { - let store = test_store().await; - - let provider = make_provider("openai-dev", "openai", "OPENAI_API_KEY", "sk-initial"); - store - .put_message(&provider) - .await - .expect("provider should persist"); - - let route = make_route(CLUSTER_INFERENCE_ROUTE_NAME, "openai-dev", "test/model"); - store - .put_message(&route) - .await - .expect("route should persist"); - - let first = resolve_route_by_name(&store, "default", CLUSTER_INFERENCE_ROUTE_NAME) - .await - .expect("route should resolve") - .expect("managed route should exist"); - assert_eq!(first.api_key, "sk-initial"); - - let rotated_provider = Provider { - metadata: provider.metadata.clone(), - r#type: provider.r#type.clone(), - credentials: std::iter::once(("OPENAI_API_KEY".to_string(), "sk-rotated".to_string())) - .collect(), - config: provider.config.clone(), - credential_expires_at_ms: provider.credential_expires_at_ms.clone(), - profile_workspace: provider.profile_workspace.clone(), - credential_handles: HashMap::new(), - }; - store - .put_message(&rotated_provider) - .await - .expect("provider rotation should persist"); - - let second = resolve_route_by_name(&store, "default", CLUSTER_INFERENCE_ROUTE_NAME) - .await - .expect("route should resolve") - .expect("managed route should exist"); - assert_eq!(second.api_key, "sk-rotated"); - } - - #[tokio::test] - async fn upsert_system_route_creates_with_correct_name() { - let store = test_store().await; - - let provider = make_provider("anthropic-dev", "anthropic", "ANTHROPIC_API_KEY", "sk-ant"); - store.put_message(&provider).await.expect("persist"); - - let route = upsert_inference_route( - &store, - "default", - SANDBOX_SYSTEM_ROUTE_NAME, - "anthropic-dev", - "claude-sonnet-4-20250514", - 0, - false, - ) - .await - .expect("should succeed"); - - assert_eq!(route.route.object_name(), SANDBOX_SYSTEM_ROUTE_NAME); - let config = route.route.config.as_ref().expect("config"); - assert_eq!(config.provider_name, "anthropic-dev"); - assert_eq!(config.model_id, "claude-sonnet-4-20250514"); - } - - #[tokio::test] - async fn upsert_inference_route_vertex_ai_anthropic_sets_model_in_path() { - let store = test_store().await; - - // Build a Vertex AI provider with the required config and a minted access token. - let provider = Provider { - metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { - id: "provider-vertex-test".to_string(), - name: "vertex-test".to_string(), - created_at_ms: 1_000_000, - labels: HashMap::new(), - resource_version: 0, - annotations: HashMap::new(), - workspace: "default".to_string(), - deletion_timestamp_ms: 0, - }), - r#type: "google-vertex-ai".to_string(), - credentials: std::iter::once(( - "GOOGLE_VERTEX_AI_TOKEN".to_string(), - "ya29.test-access-token".to_string(), - )) - .collect(), - config: [ - ( - "VERTEX_AI_PROJECT_ID".to_string(), - "my-gcp-project".to_string(), - ), - ("VERTEX_AI_REGION".to_string(), "us-central1".to_string()), - ] - .into_iter() - .collect(), - credential_expires_at_ms: HashMap::new(), - profile_workspace: String::new(), - credential_handles: HashMap::new(), - }; - store - .put_message(&provider) - .await - .expect("persist provider"); - - let result = upsert_inference_route( - &store, - "default", - CLUSTER_INFERENCE_ROUTE_NAME, - "vertex-test", - "claude-3-5-sonnet@20241022", - 0, - false, // skip verification — no live endpoint - ) - .await - .expect("upsert should succeed for Vertex AI Anthropic model"); - - // Confirm the route was persisted with correct metadata - assert_eq!(result.route.object_name(), CLUSTER_INFERENCE_ROUTE_NAME); - let config = result.route.config.as_ref().expect("config"); - assert_eq!(config.provider_name, "vertex-test"); - assert_eq!(config.model_id, "claude-3-5-sonnet@20241022"); - - // Resolve the persisted route and assert Vertex AI Anthropic path contract - let resolved = resolve_route_by_name(&store, "default", CLUSTER_INFERENCE_ROUTE_NAME) - .await - .expect("resolve should not fail") - .expect("route should exist after upsert"); - - assert!( - resolved.model_in_path, - "Anthropic-on-Vertex routes must set model_in_path=true" - ); - assert_eq!( - resolved.request_path_override, - Some(":rawPredict".to_string()), - "Anthropic-on-Vertex routes must persist the rawPredict suffix" - ); - assert_eq!(resolved.provider_type, "google-vertex-ai"); - assert!( - resolved.base_url.contains("publishers/anthropic/models"), - "endpoint must end with /publishers/anthropic/models, got: {}", - resolved.base_url - ); - assert!( - !resolved.base_url.contains("claude-3-5-sonnet"), - "model_id must not be embedded in the endpoint, got: {}", - resolved.base_url - ); - } - - #[tokio::test] - async fn bundle_includes_both_user_and_system_routes() { - let store = test_store().await; - - let openai = make_provider("openai-dev", "openai", "OPENAI_API_KEY", "sk-oai"); - store.put_message(&openai).await.expect("persist openai"); - let anthropic = make_provider("anthropic-dev", "anthropic", "ANTHROPIC_API_KEY", "sk-ant"); - store - .put_message(&anthropic) - .await - .expect("persist anthropic"); - - let user_route = make_route(CLUSTER_INFERENCE_ROUTE_NAME, "openai-dev", "gpt-4o"); - store - .put_message(&user_route) - .await - .expect("persist user route"); - let system_route = make_route( - SANDBOX_SYSTEM_ROUTE_NAME, - "anthropic-dev", - "claude-sonnet-4-20250514", - ); - store - .put_message(&system_route) - .await - .expect("persist system route"); - - let resp = resolve_inference_bundle(&store, "default") - .await - .expect("bundle should resolve"); - - assert_eq!(resp.routes.len(), 2); - assert_eq!(resp.routes[0].name, CLUSTER_INFERENCE_ROUTE_NAME); - assert_eq!(resp.routes[0].model_id, "gpt-4o"); - assert_eq!(resp.routes[1].name, SANDBOX_SYSTEM_ROUTE_NAME); - assert_eq!(resp.routes[1].model_id, "claude-sonnet-4-20250514"); - } - - #[tokio::test] - async fn bundle_with_only_system_route() { - let store = test_store().await; - - let provider = make_provider("openai-dev", "openai", "OPENAI_API_KEY", "sk-test"); - store.put_message(&provider).await.expect("persist"); - let system_route = make_route(SANDBOX_SYSTEM_ROUTE_NAME, "openai-dev", "gpt-4o-mini"); - store.put_message(&system_route).await.expect("persist"); - - let resp = resolve_inference_bundle(&store, "default") - .await - .expect("bundle should resolve"); - - assert_eq!(resp.routes.len(), 1); - assert_eq!(resp.routes[0].name, SANDBOX_SYSTEM_ROUTE_NAME); - assert_eq!(resp.routes[0].model_id, "gpt-4o-mini"); - } - - #[tokio::test] - async fn get_returns_system_route_when_requested() { - let store = test_store().await; - - let provider = make_provider("openai-dev", "openai", "OPENAI_API_KEY", "sk-test"); - store.put_message(&provider).await.expect("persist"); - - upsert_inference_route( - &store, - "default", - SANDBOX_SYSTEM_ROUTE_NAME, - "openai-dev", - "gpt-4o-mini", - 0, - false, - ) - .await - .expect("upsert should succeed"); - - let route = store - .get_message_by_name::("default", SANDBOX_SYSTEM_ROUTE_NAME) - .await - .expect("fetch should succeed") - .expect("route should exist"); - - assert_eq!(route.object_name(), SANDBOX_SYSTEM_ROUTE_NAME); - let config = route.config.as_ref().expect("config"); - assert_eq!(config.model_id, "gpt-4o-mini"); - } - - #[tokio::test] - async fn upsert_cluster_route_verifies_endpoint_when_requested() { - let store = test_store().await; - let mock_server = MockServer::start().await; - - Mock::given(method("POST")) - .and(path("/v1/chat/completions")) - .and(header("authorization", "Bearer sk-test")) - .and(header("content-type", "application/json")) - .and(body_partial_json(serde_json::json!({ - "model": "gpt-4o-mini", - "max_completion_tokens": 32, - }))) - .respond_with(ResponseTemplate::new(200).set_body_json(serde_json::json!({ - "id": "chatcmpl-123", - "object": "chat.completion", - "choices": [{"index": 0, "message": {"role": "assistant", "content": "ok"}, "finish_reason": "stop"}], - "model": "gpt-4o-mini" - }))) - .mount(&mock_server) - .await; - - let provider = make_provider_with_base_url( - "openai-dev", - "openai", - "OPENAI_API_KEY", - "sk-test", - "OPENAI_BASE_URL", - &mock_server.uri(), - ); - store - .put_message(&provider) - .await - .expect("persist provider"); - - let route = upsert_inference_route( - &store, - "default", - CLUSTER_INFERENCE_ROUTE_NAME, - "openai-dev", - "gpt-4o-mini", - 0, - true, - ) - .await - .expect("validation should succeed"); - - assert_eq!(route.route.version, 1); - assert_eq!(route.validation.len(), 1); - assert_eq!(route.validation[0].protocol, "openai_chat_completions"); - } - - #[tokio::test] - async fn upsert_cluster_route_rejects_failed_validation() { - let store = test_store().await; - let mock_server = MockServer::start().await; - - Mock::given(method("POST")) - .and(path("/v1/chat/completions")) - .respond_with(ResponseTemplate::new(401).set_body_string("bad key")) - .mount(&mock_server) - .await; - - let provider = make_provider_with_base_url( - "openai-dev", - "openai", - "OPENAI_API_KEY", - "sk-test", - "OPENAI_BASE_URL", - &mock_server.uri(), - ); - store - .put_message(&provider) - .await - .expect("persist provider"); - - let err = upsert_inference_route( - &store, - "default", - CLUSTER_INFERENCE_ROUTE_NAME, - "openai-dev", - "gpt-4o-mini", - 0, - true, - ) - .await - .expect_err("validation should fail"); - - assert_eq!(err.code(), tonic::Code::FailedPrecondition); - assert!( - err.message() - .contains("failed to verify inference endpoint") - ); - assert!(err.message().contains("verify the provider API key")); - assert!(err.message().contains("--no-verify")); - - let persisted = store - .get_message_by_name::("default", CLUSTER_INFERENCE_ROUTE_NAME) - .await - .expect("fetch route") - .is_none(); - assert!(persisted, "route should not persist on failed validation"); - } - - #[tokio::test] - async fn upsert_cluster_route_skips_validation_by_default() { - let store = test_store().await; - let provider = make_provider_with_base_url( - "openai-dev", - "openai", - "OPENAI_API_KEY", - "sk-test", - "OPENAI_BASE_URL", - "http://127.0.0.1:9", - ); - store - .put_message(&provider) - .await - .expect("persist provider"); - - let route = upsert_inference_route( - &store, - "default", - CLUSTER_INFERENCE_ROUTE_NAME, - "openai-dev", - "gpt-4o-mini", - 0, - false, - ) - .await - .expect("non-verified route should persist"); - - assert_eq!(route.route.version, 1); - assert!(route.validation.is_empty()); - } - - // ------------------------------------------------------------------------- - // infer_vertex_publisher tests - // ------------------------------------------------------------------------- - - #[test] - fn infer_vertex_publisher_anthropic() { - assert_eq!( - infer_vertex_publisher("claude-3-5-sonnet@20241022"), - Some("anthropic") - ); - assert_eq!(infer_vertex_publisher("claude-opus-4"), Some("anthropic")); - } - - #[test] - fn infer_vertex_publisher_gemini() { - assert_eq!(infer_vertex_publisher("gemini-pro"), Some("google")); - assert_eq!(infer_vertex_publisher("gemini-1.5-flash"), Some("google")); - assert_eq!(infer_vertex_publisher("text-bison-001"), Some("google")); - assert_eq!(infer_vertex_publisher("chat-bison-001"), Some("google")); - } - - #[test] - fn infer_vertex_publisher_unknown() { - assert_eq!(infer_vertex_publisher("some-unknown-model"), None); - assert_eq!(infer_vertex_publisher("gpt-4o"), None); - } - - #[test] - fn infer_vertex_publisher_other_publishers() { - assert_eq!(infer_vertex_publisher("llama-3-70b"), Some("meta")); - assert_eq!(infer_vertex_publisher("mistral-large"), Some("mistralai")); - assert_eq!(infer_vertex_publisher("codestral-22b"), Some("mistralai")); - assert_eq!(infer_vertex_publisher("jamba-1.5-large"), Some("ai21")); - assert_eq!(infer_vertex_publisher("deepseek-r1"), Some("deepseek")); - } - - // ------------------------------------------------------------------------- - // resolve_vertex_ai_route tests - // ------------------------------------------------------------------------- - - fn make_vertex_provider_with_config(name: &str, config: HashMap) -> Provider { - Provider { - metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { - id: format!("provider-{name}"), - name: name.to_string(), - created_at_ms: 1_000_000, - labels: HashMap::new(), - resource_version: 1, - annotations: HashMap::new(), - workspace: "default".to_string(), - deletion_timestamp_ms: 0, - }), - r#type: "google-vertex-ai".to_string(), - credentials: std::iter::once(( - "GOOGLE_VERTEX_AI_TOKEN".to_string(), - "ya29.test-token".to_string(), - )) - .collect(), - config, - credential_expires_at_ms: HashMap::new(), - profile_workspace: String::new(), - credential_handles: HashMap::new(), - } - } - - #[test] - fn resolve_vertex_ai_route_anthropic_model() { - let config = [ - ("VERTEX_AI_PROJECT_ID".to_string(), "my-project".to_string()), - ("VERTEX_AI_REGION".to_string(), "us-east1".to_string()), - ] - .into_iter() - .collect(); - let provider = make_vertex_provider_with_config("vertex-dev", config); - - let resolved = resolve_provider_route(&provider, "claude-3-5-sonnet@20241022") - .expect("should resolve"); - - assert_eq!(resolved.provider_type, "google-vertex-ai"); - assert!(resolved.route.model_in_path); - assert_eq!( - resolved.route.request_path_override, - Some(":rawPredict".to_string()) - ); - // model_id must NOT be embedded in the endpoint — it travels via route.model - assert!( - !resolved.route.endpoint.contains("claude-3-5-sonnet"), - "model_id must not be in endpoint, got: {}", - resolved.route.endpoint - ); - assert!( - resolved - .route - .endpoint - .ends_with("/publishers/anthropic/models"), - "endpoint should end with /publishers/anthropic/models, got: {}", - resolved.route.endpoint - ); - assert!( - resolved - .route - .endpoint - .starts_with("https://us-east1-aiplatform.googleapis.com/"), - "expected regional Vertex host, got: {}", - resolved.route.endpoint - ); - assert!( - resolved.route.endpoint.contains("my-project"), - "expected project in URL" - ); - assert!( - resolved - .route - .protocols - .contains(&"anthropic_messages".to_string()), - "expected anthropic_messages protocol" - ); - assert_eq!(resolved.route.model, "claude-3-5-sonnet@20241022"); - } - - #[test] - fn resolve_vertex_ai_route_base_url_override() { - let config = std::iter::once(( - "VERTEX_AI_BASE_URL".to_string(), - "https://us-central1-aiplatform.googleapis.com/v1beta1/projects/my-project/locations/us-central1/endpoints/openapi".to_string(), - )) - .collect(); - let provider = make_vertex_provider_with_config("vertex-custom", config); - - let resolved = resolve_provider_route(&provider, "any-model").expect("should resolve"); - - assert_eq!( - resolved.route.endpoint, - "https://us-central1-aiplatform.googleapis.com/v1beta1/projects/my-project/locations/us-central1/endpoints/openapi" - ); - assert!(!resolved.route.model_in_path); - assert_eq!( - resolved.route.request_path_override, - Some("/chat/completions".to_string()) - ); - assert_eq!( - resolved.route.protocols, - vec!["openai_chat_completions".to_string()] - ); - assert_eq!(resolved.route.model, "any-model"); - } - - #[test] - fn resolve_vertex_ai_route_google_prefixed_base_url_override() { - // GOOGLE_VERTEX_AI_BASE_URL (the preferred key) must work on its own. - let config = std::iter::once(( - "GOOGLE_VERTEX_AI_BASE_URL".to_string(), - "https://aiplatform.googleapis.com/v1beta1/projects/my-project/locations/global/endpoints/openapi".to_string(), - )) - .collect(); - let provider = make_vertex_provider_with_config("vertex-custom-google", config); - - let resolved = resolve_provider_route(&provider, "any-model").expect("should resolve"); - - assert_eq!( - resolved.route.endpoint, - "https://aiplatform.googleapis.com/v1beta1/projects/my-project/locations/global/endpoints/openapi" - ); - assert!(!resolved.route.model_in_path); - assert_eq!( - resolved.route.request_path_override, - Some("/chat/completions".to_string()) - ); - } - - #[test] - fn resolve_vertex_ai_route_base_url_priority_google_wins() { - // When both override keys are set, GOOGLE_VERTEX_AI_BASE_URL takes priority. - let config = [ - ( - "GOOGLE_VERTEX_AI_BASE_URL".to_string(), - "https://aiplatform.googleapis.com/v1beta1/projects/my-project/locations/global/endpoints/openapi".to_string(), - ), - ( - "VERTEX_AI_BASE_URL".to_string(), - "https://us-central1-aiplatform.googleapis.com/v1beta1/projects/my-project/locations/us-central1/endpoints/openapi".to_string(), - ), - ] - .into_iter() - .collect(); - let provider = make_vertex_provider_with_config("vertex-priority", config); - - let resolved = resolve_provider_route(&provider, "any-model").expect("should resolve"); - - assert_eq!( - resolved.route.endpoint, - "https://aiplatform.googleapis.com/v1beta1/projects/my-project/locations/global/endpoints/openapi", - "GOOGLE_VERTEX_AI_BASE_URL must win over VERTEX_AI_BASE_URL" - ); - } - - #[test] - fn resolve_vertex_ai_route_base_url_override_rejects_anthropic_models() { - let config = std::iter::once(( - "GOOGLE_VERTEX_AI_BASE_URL".to_string(), - "https://aiplatform.googleapis.com/v1beta1/projects/my-project/locations/global/endpoints/openapi".to_string(), - )) - .collect(); - let provider = make_vertex_provider_with_config("vertex-custom-anthropic", config); - - let err = resolve_provider_route(&provider, "claude-3-5-sonnet@20241022") - .expect_err("anthropic overrides should fail closed"); - - assert_eq!(err.code(), tonic::Code::InvalidArgument); - assert!( - err.message() - .contains("base URL overrides are not supported") - ); - } - - #[test] - fn resolve_vertex_ai_route_base_url_override_rejects_non_vertex_host() { - let config = std::iter::once(( - "VERTEX_AI_BASE_URL".to_string(), - "https://custom.example.com/v1".to_string(), - )) - .collect(); - let provider = make_vertex_provider_with_config("vertex-custom-invalid-host", config); - - let err = resolve_provider_route(&provider, "gemini-pro") - .expect_err("non-Vertex hosts must be rejected"); - - assert_eq!(err.code(), tonic::Code::InvalidArgument); - assert!( - err.message() - .contains("must target an official Vertex AI hostname") - ); - } - - #[test] - fn resolve_vertex_ai_route_base_url_override_rejects_non_https() { - let config = std::iter::once(( - "VERTEX_AI_BASE_URL".to_string(), - "http://us-central1-aiplatform.googleapis.com/v1beta1/projects/my-project/locations/us-central1/endpoints/openapi".to_string(), - )) - .collect(); - let provider = make_vertex_provider_with_config("vertex-custom-http", config); - - let err = resolve_provider_route(&provider, "gemini-pro") - .expect_err("non-https overrides must be rejected"); - - assert_eq!(err.code(), tonic::Code::InvalidArgument); - assert!(err.message().contains("must use https")); - } - - #[test] - fn resolve_vertex_ai_route_base_url_override_rejects_ip_literal() { - let config = std::iter::once(( - "VERTEX_AI_BASE_URL".to_string(), - "https://127.0.0.1/v1beta1/projects/my-project/locations/us-central1/endpoints/openapi" - .to_string(), - )) - .collect(); - let provider = make_vertex_provider_with_config("vertex-custom-ip", config); - - let err = resolve_provider_route(&provider, "gemini-pro") - .expect_err("IP literal overrides must be rejected"); - - assert_eq!(err.code(), tonic::Code::InvalidArgument); - assert!(err.message().contains("must not use IP literal hosts")); - } - - #[test] - fn resolve_vertex_ai_route_gemini_model() { - let config = - std::iter::once(("VERTEX_AI_PROJECT_ID".to_string(), "proj-123".to_string())).collect(); - let provider = make_vertex_provider_with_config("vertex-gemini", config); - - let resolved = resolve_provider_route(&provider, "gemini-pro").expect("should resolve"); - - // Gemini routes to OpenAI-compatible endpoint, not publisher endpoint - assert!(!resolved.route.model_in_path); - assert_eq!( - resolved.route.request_path_override, - Some("/chat/completions".to_string()) - ); - assert!( - resolved.route.endpoint.contains("v1beta1"), - "gemini should use v1beta1 endpoint, got: {}", - resolved.route.endpoint - ); - assert!( - resolved.route.endpoint.contains("endpoints/openapi"), - "gemini should use openapi endpoint, got: {}", - resolved.route.endpoint - ); - assert!( - !resolved.route.endpoint.contains("publishers/google"), - "gemini must not embed publisher in endpoint, got: {}", - resolved.route.endpoint - ); - // Default region - assert!(resolved.route.endpoint.contains("us-central1")); - assert!( - resolved - .route - .protocols - .contains(&"openai_chat_completions".to_string()), - "expected openai_chat_completions protocol" - ); - assert!( - !resolved - .route - .protocols - .contains(&"anthropic_messages".to_string()), - "must not have anthropic_messages protocol for gemini" - ); - // Vertex OpenAI-compatible endpoint requires publisher prefix in body model field - assert_eq!( - resolved.route.model, "google/gemini-pro", - "Vertex non-Anthropic body model must carry publisher prefix" - ); - } - - #[test] - fn resolve_vertex_ai_route_unknown_model_uses_openai_compat() { - let config = - std::iter::once(("VERTEX_AI_PROJECT_ID".to_string(), "proj-abc".to_string())).collect(); - let provider = make_vertex_provider_with_config("vertex-compat", config); - - let resolved = - resolve_provider_route(&provider, "some-unknown-model").expect("should resolve"); - - assert!(!resolved.route.model_in_path); - assert_eq!( - resolved.route.request_path_override, - Some("/chat/completions".to_string()) - ); - assert!( - resolved.route.endpoint.contains("v1beta1"), - "unknown model should use v1beta1 endpoint" - ); - assert!( - resolved.route.endpoint.contains("endpoints/openapi"), - "unknown model should use openapi endpoint" - ); - assert!( - resolved - .route - .protocols - .contains(&"openai_chat_completions".to_string()), - "expected openai_chat_completions protocol for unknown model" - ); - assert!( - !resolved - .route - .protocols - .contains(&"anthropic_messages".to_string()), - "must not have anthropic_messages for unknown model" - ); - // Unknown models have no inferred publisher; body model ID is unchanged - assert_eq!(resolved.route.model, "some-unknown-model"); - } - - #[test] - fn resolve_vertex_ai_route_non_anthropic_publisher_prefix_gemini() { - // Gemini models must get `google/` in route.model so the - // OpenAI-compatible Vertex endpoint accepts the request body. - let config = - std::iter::once(("VERTEX_AI_PROJECT_ID".to_string(), "proj-abc".to_string())).collect(); - let provider = make_vertex_provider_with_config("vertex-gemini-flash", config); - - let resolved = - resolve_provider_route(&provider, "gemini-2.5-flash").expect("should resolve"); - - assert_eq!(resolved.route.model, "google/gemini-2.5-flash"); - } - - #[test] - fn resolve_vertex_ai_route_non_anthropic_publisher_prefix_llama() { - let config = - std::iter::once(("VERTEX_AI_PROJECT_ID".to_string(), "proj-abc".to_string())).collect(); - let provider = make_vertex_provider_with_config("vertex-llama", config); - - let resolved = resolve_provider_route(&provider, "llama-3-70b").expect("should resolve"); - - assert_eq!(resolved.route.model, "meta/llama-3-70b"); - } - - #[test] - fn resolve_vertex_ai_route_explicit_publisher_overrides_inference() { - // VERTEX_AI_PUBLISHER takes precedence over infer_vertex_publisher for - // unknown model names. - let config = [ - ("VERTEX_AI_PROJECT_ID".to_string(), "proj-abc".to_string()), - ("VERTEX_AI_PUBLISHER".to_string(), "acme".to_string()), - ] - .into_iter() - .collect(); - let provider = make_vertex_provider_with_config("vertex-explicit", config); - - let resolved = - resolve_provider_route(&provider, "some-acme-model").expect("should resolve"); - - assert_eq!(resolved.route.model, "acme/some-acme-model"); - } - - #[test] - fn resolve_vertex_ai_route_base_url_override_gemini_gets_publisher_prefix() { - // Publisher prefix must also be applied when a base URL override is used. - let config = std::iter::once(( - "VERTEX_AI_BASE_URL".to_string(), - "https://us-central1-aiplatform.googleapis.com/v1beta1/projects/my-project/locations/us-central1/endpoints/openapi".to_string(), - )) - .collect(); - let provider = make_vertex_provider_with_config("vertex-base-url-gemini", config); - - let resolved = - resolve_provider_route(&provider, "gemini-2.0-flash").expect("should resolve"); - - assert_eq!(resolved.route.model, "google/gemini-2.0-flash"); - } - - #[test] - fn resolve_vertex_ai_route_global_region_uses_global_host() { - let config = [ - ( - "VERTEX_AI_PROJECT_ID".to_string(), - "proj-global".to_string(), - ), - ("VERTEX_AI_REGION".to_string(), "GLOBAL".to_string()), - ] - .into_iter() - .collect(); - let provider = make_vertex_provider_with_config("vertex-global", config); - - let resolved = - resolve_provider_route(&provider, "claude-opus-4-7").expect("should resolve"); - - assert_eq!( - resolved.route.endpoint, - "https://aiplatform.googleapis.com/v1/projects/proj-global/locations/global/publishers/anthropic/models" - ); - assert!(resolved.route.model_in_path); - assert_eq!( - resolved.route.request_path_override, - Some(":rawPredict".to_string()) - ); - } - - #[test] - fn resolve_vertex_ai_route_us_multiregion_uses_rep_host() { - let config = [ - ("VERTEX_AI_PROJECT_ID".to_string(), "proj-us".to_string()), - ("VERTEX_AI_REGION".to_string(), "us".to_string()), - ] - .into_iter() - .collect(); - let provider = make_vertex_provider_with_config("vertex-us", config); - - let resolved = resolve_provider_route(&provider, "gemini-pro").expect("should resolve"); - - assert_eq!( - resolved.route.endpoint, - "https://aiplatform.us.rep.googleapis.com/v1beta1/projects/proj-us/locations/us/endpoints/openapi" - ); - assert_eq!( - resolved.route.request_path_override, - Some("/chat/completions".to_string()) - ); - } - - #[test] - fn resolve_vertex_ai_route_eu_multiregion_uses_rep_host() { - let config = [ - ("VERTEX_AI_PROJECT_ID".to_string(), "proj-eu".to_string()), - ("VERTEX_AI_REGION".to_string(), "eu".to_string()), - ] - .into_iter() - .collect(); - let provider = make_vertex_provider_with_config("vertex-eu", config); - - let resolved = resolve_provider_route(&provider, "gemini-pro").expect("should resolve"); - - assert_eq!( - resolved.route.endpoint, - "https://aiplatform.eu.rep.googleapis.com/v1beta1/projects/proj-eu/locations/eu/endpoints/openapi" - ); - assert_eq!( - resolved.route.request_path_override, - Some("/chat/completions".to_string()) - ); - } - - #[test] - fn resolve_vertex_ai_route_explicit_publisher_anthropic_override() { - // Explicit VERTEX_AI_PUBLISHER=anthropic → Anthropic Messages API path - let config = [ - ("VERTEX_AI_PROJECT_ID".to_string(), "my-proj".to_string()), - ("VERTEX_AI_PUBLISHER".to_string(), "anthropic".to_string()), - ] - .into_iter() - .collect(); - let provider = make_vertex_provider_with_config("vertex-pub-anthropic", config); - - let resolved = resolve_provider_route(&provider, "some-model").expect("should resolve"); - - assert!(resolved.route.model_in_path); - assert_eq!( - resolved.route.request_path_override, - Some(":rawPredict".to_string()) - ); - assert!( - resolved - .route - .endpoint - .ends_with("/publishers/anthropic/models"), - "expected anthropic publisher endpoint, got: {}", - resolved.route.endpoint - ); - assert!( - !resolved.route.endpoint.contains("some-model"), - "model must not be in endpoint" - ); - assert!( - resolved - .route - .protocols - .contains(&"anthropic_messages".to_string()), - "expected anthropic_messages protocol" - ); - } - - #[test] - fn resolve_vertex_ai_route_explicit_publisher_non_anthropic_uses_openai_compat() { - // Explicit VERTEX_AI_PUBLISHER=google (any non-anthropic) → OpenAI-compat endpoint - let config = [ - ("VERTEX_AI_PROJECT_ID".to_string(), "my-proj".to_string()), - ("VERTEX_AI_PUBLISHER".to_string(), "google".to_string()), - ] - .into_iter() - .collect(); - let provider = make_vertex_provider_with_config("vertex-pub-google", config); - - let resolved = resolve_provider_route(&provider, "some-model").expect("should resolve"); - - assert!(!resolved.route.model_in_path); - assert_eq!( - resolved.route.request_path_override, - Some("/chat/completions".to_string()) - ); - assert!( - resolved.route.endpoint.contains("v1beta1"), - "non-anthropic publisher should use v1beta1 endpoint, got: {}", - resolved.route.endpoint - ); - assert!( - resolved.route.endpoint.contains("endpoints/openapi"), - "non-anthropic publisher should use openapi endpoint, got: {}", - resolved.route.endpoint - ); - assert!( - !resolved.route.endpoint.contains("publishers/google"), - "must not embed publisher in endpoint, got: {}", - resolved.route.endpoint - ); - assert!( - resolved - .route - .protocols - .contains(&"openai_chat_completions".to_string()), - "expected openai_chat_completions for non-anthropic publisher" - ); - } - - #[test] - fn resolve_vertex_ai_route_missing_project_fails() { - let config = HashMap::new(); - let provider = make_vertex_provider_with_config("vertex-no-proj", config); - - let err = resolve_provider_route(&provider, "claude-3-5-sonnet@20241022") - .expect_err("should fail without project"); - assert_eq!(err.code(), tonic::Code::FailedPrecondition); - assert!(err.message().contains("VERTEX_AI_PROJECT_ID")); - } - - #[test] - fn resolve_vertex_ai_route_whitespace_only_project_fails() { - // required_vertex_config rejects whitespace-only values via .filter(|v| !v.trim().is_empty()) - let config = [ - ("VERTEX_AI_PROJECT_ID".to_string(), " ".to_string()), - ("VERTEX_AI_REGION".to_string(), "us-central1".to_string()), - ] - .into_iter() - .collect(); - let result = resolve_vertex_ai_route( - &config, - "claude-3-5-sonnet@20241022", - "test-route", - "dummy-token", - openshell_core::inference::profile_for("google-vertex-ai").unwrap(), - ); - assert!( - result.is_err(), - "whitespace-only project should fail, got: {result:?}" - ); - let status = result.unwrap_err(); - assert_eq!(status.code(), tonic::Code::FailedPrecondition); - } - - #[test] - fn resolve_vertex_ai_route_requires_minted_access_token() { - let config = - std::iter::once(("VERTEX_AI_PROJECT_ID".to_string(), "proj-id".to_string())).collect(); - let provider = Provider { - credentials: std::iter::once(( - "GOOGLE_SERVICE_ACCOUNT_KEY".to_string(), - "{\"type\":\"service_account\"}".to_string(), - )) - .collect(), - config, - ..make_vertex_provider_with_config("vertex-bootstrap-only", HashMap::new()) - }; - - let err = resolve_provider_route(&provider, "claude-3-5-sonnet@20241022") - .expect_err("bootstrap JSON must not be treated as a bearer token"); - assert_eq!(err.code(), tonic::Code::InvalidArgument); - assert!(err.message().contains("no usable API key credential")); - } - - #[test] - fn resolve_vertex_ai_route_alias_canonicalizes_provider_type() { - let config = - std::iter::once(("VERTEX_AI_PROJECT_ID".to_string(), "proj-id".to_string())).collect(); - let mut provider = make_vertex_provider_with_config("vertex-alias", config); - provider.r#type = "vertex-ai".to_string(); - - let resolved = resolve_provider_route(&provider, "claude-3-5-sonnet@20241022") - .expect("alias should resolve through Vertex routing"); - - assert_eq!(resolved.provider_type, "google-vertex-ai"); - assert!(resolved.route.model_in_path); - assert_eq!( - resolved.route.request_path_override, - Some(":rawPredict".to_string()) - ); - } - - #[test] - fn resolve_vertex_ai_route_anthropic_protocols() { - let config = - std::iter::once(("VERTEX_AI_PROJECT_ID".to_string(), "proj-id".to_string())).collect(); - let provider = make_vertex_provider_with_config("v", config); - let resolved = resolve_provider_route(&provider, "claude-3-5-sonnet@20241022").unwrap(); - assert!( - resolved - .route - .protocols - .contains(&"anthropic_messages".to_string()) - ); - assert!( - !resolved - .route - .protocols - .contains(&"openai_chat_completions".to_string()) - ); - assert_eq!( - resolved.route.protocols, - vec!["anthropic_messages".to_string()] - ); - } - - #[test] - fn resolve_vertex_ai_route_openai_compat_protocols() { - let config = - std::iter::once(("VERTEX_AI_PROJECT_ID".to_string(), "proj-id".to_string())).collect(); - let provider = make_vertex_provider_with_config("v", config); - let resolved = resolve_provider_route(&provider, "gemini-pro").unwrap(); - assert!( - resolved - .route - .protocols - .contains(&"openai_chat_completions".to_string()) - ); - assert!( - resolved - .route - .protocols - .iter() - .all(|protocol| protocol == "openai_chat_completions") - ); - } - - #[test] - fn resolve_vertex_ai_route_model_not_in_endpoint() { - // model_id must NOT appear in the endpoint URL — it travels via route.model - let config = [ - ("VERTEX_AI_PROJECT_ID".to_string(), "proj-id".to_string()), - ("VERTEX_AI_REGION".to_string(), "us-east1".to_string()), - ] - .into_iter() - .collect(); - let provider = make_vertex_provider_with_config("v", config); - let resolved = resolve_provider_route(&provider, "claude-3-5-sonnet@20241022").unwrap(); - assert!( - !resolved.route.endpoint.contains("claude-3-5-sonnet"), - "model_id must not be in endpoint, got: {}", - resolved.route.endpoint - ); - assert!( - resolved - .route - .endpoint - .ends_with("/publishers/anthropic/models") - ); - } - - #[test] - fn resolve_vertex_ai_route_rejects_model_ids_with_path_separators() { - let config = [ - ("VERTEX_AI_PROJECT_ID".to_string(), "proj-id".to_string()), - ("VERTEX_AI_PUBLISHER".to_string(), "anthropic".to_string()), - ] - .into_iter() - .collect(); - let provider = make_vertex_provider_with_config("vertex-bad-model", config); - - let err = resolve_provider_route(&provider, "claude/3-sonnet") - .expect_err("path-like model IDs must be rejected"); - - assert_eq!(err.code(), tonic::Code::InvalidArgument); - assert!(err.message().contains("must not contain path separators")); - } - - #[test] - fn resolve_vertex_ai_route_rejects_model_ids_with_url_delimiters() { - let config = [ - ("VERTEX_AI_PROJECT_ID".to_string(), "proj-id".to_string()), - ("VERTEX_AI_PUBLISHER".to_string(), "anthropic".to_string()), - ] - .into_iter() - .collect(); - let provider = make_vertex_provider_with_config("vertex-bad-model-url", config); - - for model_id in ["claude?alt=1", "claude#fragment", "claude%2Fbad"] { - let err = resolve_provider_route(&provider, model_id) - .expect_err("URL delimiter-bearing model IDs must be rejected"); - assert_eq!(err.code(), tonic::Code::InvalidArgument); - assert!( - err.message() - .contains("must not contain URL delimiters or percent escapes"), - "unexpected error for {model_id:?}: {}", - err.message() - ); - } - } - - #[test] - fn resolve_vertex_ai_route_accepts_versioned_claude_model_id() { - let config = [ - ("VERTEX_AI_PROJECT_ID".to_string(), "proj-id".to_string()), - ("VERTEX_AI_PUBLISHER".to_string(), "anthropic".to_string()), - ] - .into_iter() - .collect(); - let provider = make_vertex_provider_with_config("vertex-good-model", config); - - let resolved = resolve_provider_route(&provider, "claude-3-5-sonnet@20241022") - .expect("versioned Claude model IDs must remain valid"); - - assert!(resolved.route.model_in_path); - assert_eq!(resolved.route.model, "claude-3-5-sonnet@20241022"); - } - - #[test] - fn resolve_vertex_ai_route_rejects_model_ids_with_whitespace() { - let config = [ - ("VERTEX_AI_PROJECT_ID".to_string(), "proj-id".to_string()), - ("VERTEX_AI_PUBLISHER".to_string(), "anthropic".to_string()), - ] - .into_iter() - .collect(); - let provider = make_vertex_provider_with_config("vertex-bad-model-whitespace", config); - - let err = resolve_provider_route(&provider, "some model") - .expect_err("whitespace in Anthropic Vertex model IDs must be rejected"); - - assert_eq!(err.code(), tonic::Code::InvalidArgument); - assert!( - err.message() - .contains("must not contain whitespace or control characters") - ); - } - - #[test] - fn validate_gcp_project_id_accepts_valid() { - assert!(validate_gcp_project_id("my-project").is_ok()); - assert!(validate_gcp_project_id("my-project-123").is_ok()); - assert!(validate_gcp_project_id("abcdef").is_ok()); // min length 6 - } - - #[test] - fn validate_gcp_project_id_rejects_invalid() { - assert!(validate_gcp_project_id("").is_err()); // empty - assert!(validate_gcp_project_id("ab").is_err()); // too short - assert!(validate_gcp_project_id("../admin").is_err()); // path traversal - assert!(validate_gcp_project_id("MY-PROJECT").is_err()); // uppercase - assert!(validate_gcp_project_id("my-project-").is_err()); // trailing hyphen - assert!(validate_gcp_project_id("1my-project").is_err()); // starts with digit - } - - #[test] - fn validate_gcp_region_accepts_valid() { - assert!(validate_gcp_region("us-central1").is_ok()); - assert!(validate_gcp_region("europe-west4").is_ok()); - assert!(validate_gcp_region("global").is_ok()); - assert!(validate_gcp_region("us").is_ok()); - assert!(validate_gcp_region("eu").is_ok()); - assert!(validate_gcp_region("us-east4-a").is_ok()); // zone-like - } - - #[test] - fn validate_gcp_region_rejects_invalid() { - assert!(validate_gcp_region("").is_err()); - assert!(validate_gcp_region("../../etc").is_err()); // path traversal - assert!(validate_gcp_region("us central1").is_err()); // space - assert!(validate_gcp_region("-us-central1").is_err()); // leading hyphen - assert!(validate_gcp_region("us-central1-").is_err()); // trailing hyphen - } - - // ------------------------------------------------------------------------- - // validate_vertex_base_url edge-case tests - // ------------------------------------------------------------------------- - - #[test] - fn validate_vertex_base_url_rejects_ipv6_literal() { - let err = validate_vertex_base_url( - "https://[::1]/v1beta1/projects/p/locations/l/endpoints/openapi", - ) - .expect_err("IPv6 literals must be rejected"); - assert_eq!(err.code(), tonic::Code::InvalidArgument); - assert!( - err.message().contains("IP literal"), - "expected IP literal error, got: {}", - err.message() - ); - } - - #[test] - fn validate_vertex_base_url_rejects_userinfo() { - let err = - validate_vertex_base_url("https://user:pass@us-central1-aiplatform.googleapis.com/v1") - .expect_err("userinfo must be rejected"); - assert_eq!(err.code(), tonic::Code::InvalidArgument); - assert!( - err.message().contains("userinfo"), - "expected userinfo error, got: {}", - err.message() - ); - } - - #[test] - fn validate_vertex_base_url_rejects_query_string() { - let err = - validate_vertex_base_url("https://us-central1-aiplatform.googleapis.com/v1?key=val") - .expect_err("query string must be rejected"); - assert_eq!(err.code(), tonic::Code::InvalidArgument); - assert!( - err.message().contains("query or fragment"), - "expected query/fragment error, got: {}", - err.message() - ); - } - - #[test] - fn validate_vertex_base_url_rejects_fragment() { - let err = - validate_vertex_base_url("https://us-central1-aiplatform.googleapis.com/v1#section") - .expect_err("fragment must be rejected"); - assert_eq!(err.code(), tonic::Code::InvalidArgument); - assert!( - err.message().contains("query or fragment"), - "expected query/fragment error, got: {}", - err.message() - ); - } - - #[test] - fn validate_vertex_base_url_rejects_non_443_port() { - let err = validate_vertex_base_url("https://us-central1-aiplatform.googleapis.com:8443/v1") - .expect_err("non-443 port must be rejected"); - assert_eq!(err.code(), tonic::Code::InvalidArgument); - assert!( - err.message().contains("443"), - "expected port 443 error, got: {}", - err.message() - ); - } - - #[test] - fn validate_vertex_model_id_rejects_double_dot_traversal() { - // ".." without a slash should still be rejected as a path traversal segment. - let err = validate_vertex_model_id("model..v2") - .expect_err("double-dot traversal must be rejected"); - assert_eq!(err.code(), tonic::Code::InvalidArgument); - assert!( - err.message().contains("traversal"), - "expected path traversal error, got: {}", - err.message() - ); - } - - /// Bedrock model ids appear as a URL path segment in - /// `/model//invoke`. Mirrors the Vertex validation suite. - #[test] - fn validate_aws_bedrock_model_id_accepts_well_formed_ids() { - // Real Bedrock model ids: provider-prefixed, dotted, hyphenated, - // possibly versioned with `:0` suffix. - validate_aws_bedrock_model_id("anthropic.claude-opus-4-7").expect("dotted id"); - validate_aws_bedrock_model_id("anthropic.claude-3-5-sonnet-20241022-v2:0") - .expect("versioned id"); - validate_aws_bedrock_model_id("meta.llama3-70b-instruct-v1:0").expect("meta id"); - validate_aws_bedrock_model_id("mistral.mixtral-8x7b-instruct-v0:1").expect("mistral id"); - } - - #[test] - fn validate_aws_bedrock_model_id_rejects_empty() { - let err = validate_aws_bedrock_model_id("").expect_err("empty must be rejected"); - assert_eq!(err.code(), tonic::Code::InvalidArgument); - assert!(err.message().contains("required")); - } - - #[test] - fn validate_aws_bedrock_model_id_rejects_path_separators() { - for value in ["foo/bar", "anthropic.claude/../passwd", "back\\slash"] { - let err = validate_aws_bedrock_model_id(value).expect_err(value); - assert!( - err.message().contains("path separators"), - "expected path-separator error for {value:?}, got: {}", - err.message() - ); - } - } - - #[test] - fn validate_aws_bedrock_model_id_rejects_url_delimiters() { - for value in ["model?injected=1", "model#fragment", "percent%2fencoded"] { - let err = validate_aws_bedrock_model_id(value).expect_err(value); - assert!( - err.message().contains("URL delimiters"), - "expected URL-delimiter error for {value:?}, got: {}", - err.message() - ); - } - } - - #[test] - fn validate_aws_bedrock_model_id_rejects_traversal() { - let err = validate_aws_bedrock_model_id("model..v2") - .expect_err("double-dot traversal must be rejected"); - assert!( - err.message().contains("traversal"), - "expected path traversal error, got: {}", - err.message() - ); - } - - #[test] - fn validate_aws_bedrock_model_id_rejects_whitespace_and_control() { - for value in [ - " leading", - "trailing ", - "in middle", - "tab\tin", - "newline\nin", - ] { - let err = validate_aws_bedrock_model_id(value).expect_err(value); - assert!( - err.message().contains("whitespace") || err.message().contains("control"), - "expected whitespace/control error for {value:?}, got: {}", - err.message() - ); - } - } - - #[test] - fn effective_route_name_defaults_empty_to_inference_local() { - assert_eq!( - effective_route_name("").unwrap(), - CLUSTER_INFERENCE_ROUTE_NAME - ); - assert_eq!( - effective_route_name(" ").unwrap(), - CLUSTER_INFERENCE_ROUTE_NAME - ); - assert_eq!( - effective_route_name("inference.local").unwrap(), - CLUSTER_INFERENCE_ROUTE_NAME - ); - } - - #[test] - fn effective_route_name_accepts_sandbox_system() { - assert_eq!( - effective_route_name("sandbox-system").unwrap(), - SANDBOX_SYSTEM_ROUTE_NAME - ); - } - - #[test] - fn effective_route_name_rejects_unknown_name() { - let err = effective_route_name("unknown-route").unwrap_err(); - assert_eq!(err.code(), tonic::Code::InvalidArgument); - } - - #[tokio::test] - async fn concurrent_upsert_route_create_uses_must_create() { - let store = test_store().await; - - let provider = make_provider("openai-dev", "openai", "OPENAI_API_KEY", "sk-test"); - store.put_message(&provider).await.expect("persist"); - - // Spawn two concurrent upsert calls for the same route (create path) - let store1 = store.clone(); - let handle1 = tokio::spawn(async move { - upsert_inference_route( - &store1, - "default", - CLUSTER_INFERENCE_ROUTE_NAME, - "openai-dev", - "gpt-4o", - 0, - false, - ) - .await - }); - - let store2 = store.clone(); - let handle2 = tokio::spawn(async move { - upsert_inference_route( - &store2, - "default", - CLUSTER_INFERENCE_ROUTE_NAME, - "openai-dev", - "gpt-4.1", - 0, - false, - ) - .await - }); - - let result1 = handle1.await.unwrap(); - let result2 = handle2.await.unwrap(); - - // If both tasks observe a missing route before either insert commits, MustCreate - // should let exactly one win. If the scheduler serializes them, the second call - // may legitimately observe the new route and take the update path. - let successes = [&result1, &result2].iter().filter(|r| r.is_ok()).count(); - let failures = [&result1, &result2] - .iter() - .filter(|r| { - r.as_ref().is_err_and(|e| { - // Accept either ABORTED (from CAS) or Internal (from DB unique constraint) - e.code() == tonic::Code::Aborted - || (e.code() == tonic::Code::Internal - && e.message().contains("unique violation")) - }) - }) - .count(); - - assert!( - successes == 1 || successes == 2, - "one racing create should succeed, or both serialized upserts should succeed, got: {result1:?}, {result2:?}" - ); - if successes == 1 { - assert_eq!( - failures, 1, - "the losing racing create should fail, got: {result1:?}, {result2:?}" - ); - } else { - assert_eq!( - failures, 0, - "serialized upserts should not fail, got: {result1:?}, {result2:?}" - ); - let mut versions = [&result1, &result2] - .into_iter() - .map(|result| result.as_ref().expect("success").route.version) - .collect::>(); - versions.sort_unstable(); - assert_eq!( - versions, - vec![1, 2], - "serialized create-then-update should return versions 1 and 2" - ); - } - - // Only one route should exist. - let route = store - .get_message_by_name::("default", CLUSTER_INFERENCE_ROUTE_NAME) - .await - .expect("fetch") - .expect("route should exist"); - let expected_version = if successes == 1 { 1 } else { 2 }; - assert_eq!(route.version, expected_version); - } - - #[tokio::test] - async fn concurrent_upsert_route_update_uses_cas() { - let store = test_store().await; - - let provider = make_provider("openai-dev", "openai", "OPENAI_API_KEY", "sk-test"); - store.put_message(&provider).await.expect("persist"); - - // Create initial route - upsert_inference_route( - &store, - "default", - CLUSTER_INFERENCE_ROUTE_NAME, - "openai-dev", - "gpt-3.5", - 0, - false, - ) - .await - .expect("initial create should succeed"); - - // Spawn two concurrent updates - let store1 = store.clone(); - let handle1 = tokio::spawn(async move { - upsert_inference_route( - &store1, - "default", - CLUSTER_INFERENCE_ROUTE_NAME, - "openai-dev", - "gpt-4o", - 0, - false, - ) - .await - }); - - let store2 = store.clone(); - let handle2 = tokio::spawn(async move { - upsert_inference_route( - &store2, - "default", - CLUSTER_INFERENCE_ROUTE_NAME, - "openai-dev", - "gpt-4.1", - 0, - false, - ) - .await - }); - - let result1 = handle1.await.unwrap(); - let result2 = handle2.await.unwrap(); - - // One should succeed, one may fail with ABORTED due to CAS conflict - let successes = [&result1, &result2].iter().filter(|r| r.is_ok()).count(); - - assert!( - successes >= 1, - "at least one update should succeed, got: {result1:?}, {result2:?}" - ); - - // The route should have one of the new model values and version 2 - let route = store - .get_message_by_name::("default", CLUSTER_INFERENCE_ROUTE_NAME) - .await - .expect("fetch") - .expect("route should exist"); - let config = route.config.expect("config"); - assert!( - config.model_id == "gpt-4o" || config.model_id == "gpt-4.1", - "model should be one of the updated values, got {}", - config.model_id - ); - assert_ne!( - config.model_id, "gpt-3.5", - "model should not be the original value" - ); - assert!( - route.version >= 2 && route.version <= 3, - "version should be 2 (one update won, one conflicted) or 3 (both succeeded sequentially), got {}", - route.version - ); - } - - // ------------------------------------------------------------------------- - // Workspace isolation tests - // ------------------------------------------------------------------------- - - #[tokio::test] - async fn inference_bundle_resolves_workspace_scoped_route() { - let store = test_store().await; - - let alpha_provider = Provider { - metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { - id: "provider-alpha".to_string(), - name: "openai-alpha".to_string(), - created_at_ms: 1_000_000, - labels: HashMap::new(), - annotations: HashMap::new(), - resource_version: 0, - workspace: "alpha".to_string(), - deletion_timestamp_ms: 0, - }), - r#type: "openai".to_string(), - credentials: std::iter::once(( - "OPENAI_API_KEY".to_string(), - "sk-alpha-key".to_string(), - )) - .collect(), - config: HashMap::new(), - credential_expires_at_ms: HashMap::new(), - profile_workspace: String::new(), - credential_handles: HashMap::new(), - }; - store - .put_message(&alpha_provider) - .await - .expect("persist alpha provider"); - - let beta_provider = Provider { - metadata: Some(openshell_core::proto::datamodel::v1::ObjectMeta { - id: "provider-beta".to_string(), - name: "anthropic-beta".to_string(), - created_at_ms: 1_000_000, - labels: HashMap::new(), - annotations: HashMap::new(), - resource_version: 0, - workspace: "beta".to_string(), - deletion_timestamp_ms: 0, - }), - r#type: "anthropic".to_string(), - credentials: std::iter::once(( - "ANTHROPIC_API_KEY".to_string(), - "sk-beta-key".to_string(), - )) - .collect(), - config: HashMap::new(), - credential_expires_at_ms: HashMap::new(), - profile_workspace: String::new(), - credential_handles: HashMap::new(), - }; - store - .put_message(&beta_provider) - .await - .expect("persist beta provider"); - - upsert_inference_route( - &store, - "alpha", - CLUSTER_INFERENCE_ROUTE_NAME, - "openai-alpha", - "gpt-4", - 0, - false, - ) - .await - .expect("set alpha route"); - - upsert_inference_route( - &store, - "beta", - CLUSTER_INFERENCE_ROUTE_NAME, - "anthropic-beta", - "claude-sonnet-4-20250514", - 0, - false, - ) - .await - .expect("set beta route"); - - let alpha_bundle = resolve_inference_bundle(&store, "alpha") - .await - .expect("alpha bundle should resolve"); - assert_eq!(alpha_bundle.routes.len(), 1); - assert_eq!(alpha_bundle.routes[0].api_key, "sk-alpha-key"); - assert_eq!(alpha_bundle.routes[0].model_id, "gpt-4"); - assert_eq!(alpha_bundle.routes[0].provider_type, "openai"); - - let beta_bundle = resolve_inference_bundle(&store, "beta") - .await - .expect("beta bundle should resolve"); - assert_eq!(beta_bundle.routes.len(), 1); - assert_eq!(beta_bundle.routes[0].api_key, "sk-beta-key"); - assert_eq!(beta_bundle.routes[0].model_id, "claude-sonnet-4-20250514"); - assert_eq!(beta_bundle.routes[0].provider_type, "anthropic"); - } - - #[tokio::test] - async fn inference_bundle_empty_for_workspace_without_route() { - let store = test_store().await; - - let provider = make_provider("openai-dev", "openai", "OPENAI_API_KEY", "sk-test"); - store - .put_message(&provider) - .await - .expect("persist provider"); - - upsert_inference_route( - &store, - "default", - CLUSTER_INFERENCE_ROUTE_NAME, - "openai-dev", - "gpt-4", - 0, - false, - ) - .await - .expect("set default route"); - - let other_bundle = resolve_inference_bundle(&store, "other-workspace") - .await - .expect("bundle should resolve"); - assert!( - other_bundle.routes.is_empty(), - "workspace with no route should get empty bundle, not inherit from another workspace" - ); - } - - #[tokio::test] - async fn delete_route_removes_existing_route() { - let store = test_store().await; - - let provider = make_provider("openai-dev", "openai", "OPENAI_API_KEY", "sk-test"); - store.put_message(&provider).await.expect("persist"); - - upsert_inference_route( - &store, - "default", - CLUSTER_INFERENCE_ROUTE_NAME, - "openai-dev", - "gpt-4o", - 0, - false, - ) - .await - .expect("upsert should succeed"); - - let deleted = store - .delete_by_name( - InferenceRoute::object_type(), - "default", - CLUSTER_INFERENCE_ROUTE_NAME, - ) - .await - .expect("delete should succeed"); - assert!(deleted, "route should have been deleted"); - - let route: Option = store - .get_message_by_name("default", CLUSTER_INFERENCE_ROUTE_NAME) - .await - .expect("fetch should succeed"); - assert!(route.is_none(), "route should no longer exist"); - } - - #[tokio::test] - async fn delete_route_returns_false_when_not_found() { - let store = test_store().await; - - let deleted = store - .delete_by_name( - InferenceRoute::object_type(), - "default", - CLUSTER_INFERENCE_ROUTE_NAME, - ) - .await - .expect("delete should succeed"); - assert!(!deleted, "nothing to delete"); - } - - #[tokio::test] - async fn delete_route_bundle_becomes_empty() { - let store = test_store().await; - - let provider = make_provider("openai-dev", "openai", "OPENAI_API_KEY", "sk-test"); - store.put_message(&provider).await.expect("persist"); - - upsert_inference_route( - &store, - "default", - CLUSTER_INFERENCE_ROUTE_NAME, - "openai-dev", - "gpt-4o", - 0, - false, - ) - .await - .expect("upsert should succeed"); - - let bundle = resolve_inference_bundle(&store, "default") - .await - .expect("bundle should resolve"); - assert_eq!(bundle.routes.len(), 1); - - store - .delete_by_name( - InferenceRoute::object_type(), - "default", - CLUSTER_INFERENCE_ROUTE_NAME, - ) - .await - .expect("delete should succeed"); - - let bundle = resolve_inference_bundle(&store, "default") - .await - .expect("bundle should resolve"); - assert!( - bundle.routes.is_empty(), - "bundle should be empty after route deletion" - ); - } - - /// Non-member callers must receive `PERMISSION_DENIED` — not `NOT_FOUND` — - /// when targeting a workspace that does not exist. Returning `NOT_FOUND` - /// would create a CWE-203 workspace-name oracle. - #[tokio::test] - async fn non_member_gets_permission_denied_not_workspace_oracle() { - use crate::grpc::test_support::test_server_state; - use crate::inference::InferenceService; - use openshell_core::proto::inference_server::Inference; - - fn non_member_request(inner: T) -> Request { - let mut req = Request::new(inner); - req.extensions_mut().insert(Principal::User(UserPrincipal { - identity: Identity { - subject: "non-member".to_string(), - display_name: None, - roles: vec![], - scopes: vec![], - provider: IdentityProvider::Oidc, - }, - })); - req - } - - let mut state = test_server_state().await; - Arc::get_mut(&mut state).unwrap().admin_role = "openshell-admin".to_string(); - - let svc = InferenceService::new(state.clone()); - - let err = svc - .set_inference_route(non_member_request(SetInferenceRouteRequest { - workspace: "no-such-ws".into(), - ..Default::default() - })) - .await - .unwrap_err(); - assert_eq!( - err.code(), - tonic::Code::PermissionDenied, - "set_inference_route should return PermissionDenied, got {:?}", - err.code() - ); - - let err = svc - .get_inference_route(non_member_request(GetInferenceRouteRequest { - workspace: "no-such-ws".into(), - ..Default::default() - })) - .await - .unwrap_err(); - assert_eq!( - err.code(), - tonic::Code::PermissionDenied, - "get_inference_route should return PermissionDenied, got {:?}", - err.code() - ); - - let err = svc - .delete_inference_route(non_member_request(DeleteInferenceRouteRequest { - workspace: "no-such-ws".into(), - ..Default::default() - })) - .await - .unwrap_err(); - assert_eq!( - err.code(), - tonic::Code::PermissionDenied, - "delete_inference_route should return PermissionDenied, got {:?}", - err.code() - ); - } -} diff --git a/crates/openshell-server/src/lib.rs b/crates/openshell-server/src/lib.rs index aafc0cd36..50c4f9869 100644 --- a/crates/openshell-server/src/lib.rs +++ b/crates/openshell-server/src/lib.rs @@ -23,7 +23,6 @@ mod defaults; mod gateway_listener; mod grpc; mod http; -mod inference; mod middleware; mod multiplex; mod otel_tracing; @@ -36,6 +35,7 @@ mod sandbox_index; mod sandbox_watch; mod service_routing; mod ssh_sessions; +mod storage_proto; pub mod supervisor_session; mod telemetry; #[cfg(any(test, feature = "test-support"))] @@ -48,9 +48,8 @@ mod tracing_setup; mod ws_tunnel; use metrics_exporter_prometheus::PrometheusBuilder; -#[cfg(target_os = "windows")] -use openshell_core::ComputeDriverKind; use openshell_core::net::set_tcp_nodelay_best_effort; +use openshell_core::telemetry::TelemetryComputeDriver; use openshell_core::{Config, Error, ObjectLabels, Result}; use openshell_extension_core::{ BearerTokenSlot, ExtensionAudience, ExtensionCallerKind, ExtensionKind, MAX_EXTENSION_TOKEN_TTL, @@ -59,7 +58,7 @@ use openshell_supervisor_middleware::MiddlewareRegistry; use std::collections::{BTreeMap, HashMap}; use std::io::ErrorKind; use std::net::SocketAddr; -use std::path::Path; +use std::path::{Path, PathBuf}; #[cfg(test)] use std::sync::LazyLock; use std::sync::{ @@ -99,7 +98,7 @@ struct GatewayExtensionCredential { fn extension_token_ttl(issuer: &auth::sandbox_jwt::SandboxJwtIssuer) -> Duration { if issuer.ttl().is_zero() { - Duration::from_secs(15 * 60) + Duration::from_mins(15) } else { issuer.ttl().min(MAX_EXTENSION_TOKEN_TTL) } @@ -293,8 +292,8 @@ pub struct ServerState { /// Registry of active supervisor sessions and pending relay channels. /// - /// Stored as `Arc` so compute drivers (e.g. the Docker driver) - /// can be constructed before `ServerState` and still + /// Stored as `Arc` so compiled compute drivers can be constructed before + /// `ServerState` and still /// query session state to surface supervisor readiness. pub supervisor_sessions: Arc, @@ -442,14 +441,14 @@ impl ServerState { /// Returns an error if the server fails to start or encounters a fatal error. pub(crate) async fn run_server( startup: ServerStartupConfig, - compute_driver: ConfiguredComputeDriver, tracing_log_bus: TracingLogBus, + compute_drivers: ComputeDriverRegistry, ) -> Result<()> { let ServerStartupConfig { config, config_file, guest_tls, - compute_driver: _, + compute_driver, } = startup; let (shutdown_tx, shutdown_rx) = watch::channel(false); @@ -590,12 +589,18 @@ pub(crate) async fn run_server( let sandbox_index = SandboxIndex::new(); let sandbox_watch_bus = SandboxWatchBus::new(); let supervisor_sessions = Arc::new(supervisor_session::SupervisorSessionRegistry::new()); - let driver_startup = - compute_driver_startup_context(&config, config_file.as_ref(), guest_tls.as_ref()); - let (compute, _operator_allowlist) = build_compute_runtime( + let driver_startup = compute::driver_config::DriverStartupContext { + file: config_file.as_ref(), + guest_tls: guest_tls.as_ref(), + gateway_port: config.bind_address.port(), + gateway_tls_enabled: config.tls.is_some(), + endpoint_overrides: &config.compute_driver_endpoints, + }; + let compute = build_compute_runtime( + &compute_drivers, + &compute_driver, &config, driver_startup, - compute_driver, store.clone(), sandbox_index.clone(), sandbox_watch_bus.clone(), @@ -667,7 +672,7 @@ pub(crate) async fn run_server( auth::compute_driver::ComputeDriverAuthenticator::new(state.compute.clone()), )); info!( - driver = state.compute.selected_driver_name(), + driver = state.compute.configured_driver_name(), "compute-driver sandbox bootstrap authenticator enabled" ); } @@ -678,6 +683,14 @@ pub(crate) async fn run_server( // first snapshots observe the post-start backend state. Explicitly stopped // sandboxes remain stopped. ensure_default_workspace(&store).await?; + grpc::policy::validate_provider_composition_startup_preflight(&state) + .await + .map_err(|error| { + Error::config(format!( + "provider policy composition startup preflight failed: {}", + error.message() + )) + })?; let gateway_listeners = bind_gateway_listeners( config.bind_address, @@ -690,9 +703,9 @@ pub(crate) async fn run_server( } state.compute.spawn_watchers(shutdown_rx.clone()); - ssh_sessions::spawn_session_reaper(store.clone(), Duration::from_secs(3600)); + ssh_sessions::spawn_session_reaper(store.clone(), Duration::from_hours(1)); supervisor_session::spawn_relay_reaper(state.clone(), Duration::from_secs(30)); - provider_refresh::spawn_refresh_worker(state.clone(), Duration::from_secs(60)); + provider_refresh::spawn_refresh_worker(state.clone(), Duration::from_mins(1)); // Create the multiplexed service let service = MultiplexService::new(state.clone()); @@ -1030,30 +1043,22 @@ async fn terminate_signal() { let _ = signal.recv().await; } -#[cfg(all(target_os = "windows", feature = "in-tree-compute-drivers"))] -fn unsupported_builtin_compute_driver(driver: ComputeDriverKind) -> compute::ComputeError { - compute::ComputeError::Message(format!( - "{} compute driver is unsupported on Windows", - driver.as_str() - )) -} +pub use compute::{ + AcquiredRemoteDriverEndpoint, DriverWatchStream, ManagedDriverProcess, SharedComputeDriver, +}; -type OperatorAllowlistArc = Option; -pub use compute::{DriverWatchStream, SharedComputeDriver}; - -/// Opaque result returned by a compiled compute-driver factory. -pub struct ComputeDriverBuildOutput { - runtime: ComputeRuntime, - operator_allowlist: OperatorAllowlistArc, +/// Driver instance returned by a compiled compute-driver factory. +pub enum ComputeDriverInstance { + /// A driver hosted in the gateway process. + InProcess(SharedComputeDriver), + /// A driver process launched and owned by the gateway. + ManagedRemote(AcquiredRemoteDriverEndpoint), } /// Factory for a compute driver linked into a gateway binary. #[async_trait::async_trait] pub trait ComputeDriverFactory: Send + Sync { - async fn build( - &self, - context: ComputeDriverBuildContext<'_>, - ) -> Result; + async fn build(&self, context: ComputeDriverBuildContext<'_>) -> Result; } /// One named compiled-driver registration. @@ -1063,6 +1068,11 @@ pub struct ComputeDriverRegistration { detection_priority: u16, detect: Option bool>, factory: Arc, + telemetry_category: TelemetryComputeDriver, + inherited_config_keys: &'static [&'static str], + local_singleplayer: bool, + supports_mtls_user_auth: bool, + in_process_tracing: Option, } impl std::fmt::Debug for ComputeDriverRegistration { @@ -1091,8 +1101,67 @@ impl ComputeDriverRegistration { detection_priority, detect, factory: Arc::new(factory), + telemetry_category: TelemetryComputeDriver::custom(), + inherited_config_keys: &[], + local_singleplayer: false, + supports_mtls_user_auth: true, + in_process_tracing: None, }) } + + /// Select gateway-wide defaults understood by this driver's config type. + #[must_use] + pub fn with_inherited_config_keys(mut self, keys: &'static [&'static str]) -> Self { + self.inherited_config_keys = keys; + self + } + + /// Assign a bounded telemetry category chosen by the binary composition + /// boundary. Runtime driver names are never used as telemetry values. + #[must_use] + pub fn with_telemetry_category(mut self, category: TelemetryComputeDriver) -> Self { + self.telemetry_category = category; + self + } + + /// Mark a backend whose local deployment should use single-player defaults. + #[must_use] + pub fn with_local_singleplayer(mut self) -> Self { + self.local_singleplayer = true; + self + } + + /// Mark a backend that requires user authentication other than mTLS. + #[must_use] + pub fn without_mtls_user_auth(mut self) -> Self { + self.supports_mtls_user_auth = false; + self + } + + /// Attach process-wide tracing for this in-process compiled driver. + #[must_use] + pub fn with_in_process_tracing( + mut self, + tracing: openshell_otel::ComputeDriverTracing, + ) -> Self { + self.in_process_tracing = Some(tracing); + self + } + + #[must_use] + pub(crate) fn is_local_singleplayer(&self) -> bool { + self.local_singleplayer + } + + #[must_use] + pub(crate) fn supports_mtls_user_auth(&self) -> bool { + self.supports_mtls_user_auth + } + + #[must_use] + pub fn in_process_tracing(&self) -> Option { + self.in_process_tracing + } } /// Registry of compute drivers compiled into this gateway binary. @@ -1157,10 +1226,23 @@ impl ComputeDriverRegistry { self.drivers.keys().map(String::as_str) } - fn get(&self, name: &str) -> Option<&ComputeDriverRegistration> { + pub(crate) fn get(&self, name: &str) -> Option<&ComputeDriverRegistration> { self.drivers.get(name) } + fn in_process_tracing( + &self, + selection: &ComputeDriverSelection, + endpoint_overrides: &BTreeMap, + ) -> Option { + let name = selection.name(); + if endpoint_overrides.contains_key(name) { + return None; + } + self.get(name) + .and_then(ComputeDriverRegistration::in_process_tracing) + } + fn detect(&self) -> ComputeDriverDetection { let mut candidates = self .drivers @@ -1187,7 +1269,7 @@ impl ComputeDriverRegistry { if detection.selected().is_none() { return Err(Error::config( "no compute driver configured and auto-detection found no suitable installed \ - driver; set --drivers or OPENSHELL_DRIVERS=", + driver; set --drivers or OPENSHELL_DRIVERS=", )); } Ok(ComputeDriverSelection::AutoDetected(detection)) @@ -1205,88 +1287,14 @@ impl ComputeDriverRegistry { } } -/// Install every first-party compute driver linked into the standard gateway. -#[must_use] -pub fn install_default_compute_drivers() -> ComputeDriverRegistry { - #[allow(unused_mut)] - let mut registry = ComputeDriverRegistry::new(); - #[cfg(all(not(target_os = "windows"), feature = "in-tree-compute-drivers"))] - { - registry - .install( - ComputeDriverRegistration::new( - "kubernetes", - 100, - Some(|| std::env::var_os("KUBERNETES_SERVICE_HOST").is_some()), - KubernetesComputeDriverFactory, - ) - .expect("valid kubernetes registration"), - ) - .expect("unique kubernetes registration"); - registry - .install( - ComputeDriverRegistration::new( - "podman", - 200, - Some(openshell_core::config::is_podman_available), - PodmanComputeDriverFactory, - ) - .expect("valid podman registration"), - ) - .expect("unique podman registration"); - registry - .install( - ComputeDriverRegistration::new( - "docker", - 300, - Some(openshell_core::config::is_docker_available), - DockerComputeDriverFactory, - ) - .expect("valid docker registration"), - ) - .expect("unique docker registration"); - registry - .install( - ComputeDriverRegistration::new("vm", u16::MAX, None, VmComputeDriverFactory) - .expect("valid vm registration"), - ) - .expect("unique vm registration"); - } - #[cfg(all(target_os = "windows", feature = "in-tree-compute-drivers"))] - { - registry - .install( - ComputeDriverRegistration::new("mxc", u16::MAX, None, MxcComputeDriverFactory) - .expect("valid mxc registration"), - ) - .expect("unique mxc registration"); - for name in ["kubernetes", "podman", "docker", "vm"] { - registry - .install( - ComputeDriverRegistration::new( - name, - u16::MAX, - None, - UnsupportedComputeDriverFactory, - ) - .expect("valid unsupported registration"), - ) - .expect("unique unsupported registration"); - } - } - registry -} - pub struct ComputeDriverBuildContext<'a> { driver_name: String, - config: &'a Config, + gateway_name: &'a str, + gateway_bind_address: SocketAddr, + gateway_log_level: &'a str, driver_startup: compute::driver_config::DriverStartupContext<'a>, - store: Arc, - sandbox_index: SandboxIndex, - sandbox_watch_bus: SandboxWatchBus, - tracing_log_bus: TracingLogBus, - supervisor_sessions: Arc, shutdown_rx: watch::Receiver, + inherited_config_keys: &'static [&'static str], } impl ComputeDriverBuildContext<'_> { @@ -1296,8 +1304,18 @@ impl ComputeDriverBuildContext<'_> { } #[must_use] - pub fn gateway_config(&self) -> &Config { - self.config + pub fn gateway_name(&self) -> &str { + self.gateway_name + } + + #[must_use] + pub fn gateway_bind_address(&self) -> SocketAddr { + self.gateway_bind_address + } + + #[must_use] + pub fn gateway_log_level(&self) -> &str { + self.gateway_log_level } #[must_use] @@ -1323,7 +1341,11 @@ impl ComputeDriverBuildContext<'_> { where T: Default + serde::de::DeserializeOwned, { - compute::driver_config::driver_config_from_context(self.driver_startup, &self.driver_name) + compute::driver_config::driver_config_from_context( + self.driver_startup, + &self.driver_name, + self.inherited_config_keys, + ) } #[must_use] @@ -1331,242 +1353,84 @@ impl ComputeDriverBuildContext<'_> { self.shutdown_rx.clone() } - /// Finish construction of an in-process driver through the common runtime path. - pub async fn finish_in_process( - self, - driver: SharedComputeDriver, - ) -> Result { - let runtime = ComputeRuntime::from_driver( - self.driver_name, - driver, - None, - self.store, - self.sandbox_index, - self.sandbox_watch_bus, - self.tracing_log_bus, - self.supervisor_sessions, - ) - .await - .map_err(|error| Error::execution(format!("failed to create compute runtime: {error}")))?; - Ok(ComputeDriverBuildOutput { - runtime, - operator_allowlist: None, - }) - } -} - -#[cfg(all(target_os = "windows", feature = "in-tree-compute-drivers"))] -#[derive(Clone, Copy)] -struct MxcComputeDriverFactory; - -#[cfg(all(target_os = "windows", feature = "in-tree-compute-drivers"))] -#[async_trait::async_trait] -impl ComputeDriverFactory for MxcComputeDriverFactory { - async fn build( - &self, - context: ComputeDriverBuildContext<'_>, - ) -> Result { - let mxc_config = compute::driver_config::mxc_config_from_context(context.driver_startup)?; - let runtime = ComputeRuntime::new_mxc( - mxc_config, - context.store, - context.sandbox_index, - context.sandbox_watch_bus, - context.tracing_log_bus, - context.supervisor_sessions, - ) - .await - .map_err(|error| Error::execution(format!("failed to create compute runtime: {error}")))?; - Ok(ComputeDriverBuildOutput { - runtime, - operator_allowlist: None, - }) - } -} - -#[cfg(all(target_os = "windows", feature = "in-tree-compute-drivers"))] -#[derive(Clone, Copy)] -struct UnsupportedComputeDriverFactory; - -#[cfg(all(target_os = "windows", feature = "in-tree-compute-drivers"))] -#[async_trait::async_trait] -impl ComputeDriverFactory for UnsupportedComputeDriverFactory { - async fn build( - &self, - context: ComputeDriverBuildContext<'_>, - ) -> Result { - Err(Error::execution( - unsupported_builtin_compute_driver( - context - .driver_name - .parse() - .expect("default driver names are valid"), - ) - .to_string(), - )) - } -} - -#[cfg(all(not(target_os = "windows"), feature = "in-tree-compute-drivers"))] -#[derive(Clone, Copy)] -struct KubernetesComputeDriverFactory; - -#[cfg(all(not(target_os = "windows"), feature = "in-tree-compute-drivers"))] -#[async_trait::async_trait] -impl ComputeDriverFactory for KubernetesComputeDriverFactory { - async fn build( - &self, - context: ComputeDriverBuildContext<'_>, - ) -> Result { - warn_if_kubernetes_sandbox_jwt_expiry_disabled(context.config); - let config = compute::driver_config::builtin::kubernetes_config_from_context( - context.driver_startup, - )?; - let (runtime, operator_allowlist) = ComputeRuntime::new_kubernetes( - config, - context.store, - context.sandbox_index, - context.sandbox_watch_bus, - context.tracing_log_bus, - context.supervisor_sessions, - context.shutdown_rx, - ) - .await - .map_err(|error| Error::execution(format!("failed to create compute runtime: {error}")))?; - Ok(ComputeDriverBuildOutput { - runtime, - operator_allowlist, - }) - } -} - -#[cfg(all(not(target_os = "windows"), feature = "in-tree-compute-drivers"))] -#[derive(Clone, Copy)] -struct DockerComputeDriverFactory; - -#[cfg(all(not(target_os = "windows"), feature = "in-tree-compute-drivers"))] -#[async_trait::async_trait] -impl ComputeDriverFactory for DockerComputeDriverFactory { - async fn build( - &self, - context: ComputeDriverBuildContext<'_>, - ) -> Result { - let driver_config = - compute::driver_config::builtin::docker_config_from_context(context.driver_startup)?; - let runtime = ComputeRuntime::new_docker( - context.config.clone(), - driver_config, - context.store, - context.sandbox_index, - context.sandbox_watch_bus, - context.tracing_log_bus, - context.supervisor_sessions, - ) - .await - .map_err(|error| Error::execution(format!("failed to create compute runtime: {error}")))?; - Ok(ComputeDriverBuildOutput { - runtime, - operator_allowlist: None, - }) - } -} - -#[cfg(all(not(target_os = "windows"), feature = "in-tree-compute-drivers"))] -#[derive(Clone, Copy)] -struct PodmanComputeDriverFactory; - -#[cfg(all(not(target_os = "windows"), feature = "in-tree-compute-drivers"))] -#[async_trait::async_trait] -impl ComputeDriverFactory for PodmanComputeDriverFactory { - async fn build( - &self, - context: ComputeDriverBuildContext<'_>, - ) -> Result { - let driver_config = - compute::driver_config::builtin::podman_config_from_context(context.driver_startup)?; - let runtime = ComputeRuntime::new_podman( - driver_config, - context.store, - context.sandbox_index, - context.sandbox_watch_bus, - context.tracing_log_bus, - context.supervisor_sessions, - ) - .await - .map_err(|error| Error::execution(format!("failed to create compute runtime: {error}")))?; - Ok(ComputeDriverBuildOutput { - runtime, - operator_allowlist: None, - }) - } -} - -#[cfg(all(not(target_os = "windows"), feature = "in-tree-compute-drivers"))] -#[derive(Clone, Copy)] -struct VmComputeDriverFactory; - -#[cfg(all(not(target_os = "windows"), feature = "in-tree-compute-drivers"))] -#[async_trait::async_trait] -impl ComputeDriverFactory for VmComputeDriverFactory { - async fn build( - &self, - context: ComputeDriverBuildContext<'_>, - ) -> Result { - let driver_config = - compute::driver_config::builtin::vm_config_from_context(context.driver_startup)?; - let otlp_config = context - .driver_startup + #[must_use] + pub fn otlp_config(&self) -> Option<&config_file::OtlpConfig> { + self.driver_startup .file - .and_then(|file| file.openshell.gateway.otlp.as_ref()); - let endpoint = compute::vm::spawn(context.config, &driver_config, otlp_config).await?; - let runtime = ComputeRuntime::new_remote_driver( - endpoint, - context.store, - context.sandbox_index, - context.sandbox_watch_bus, - context.tracing_log_bus, - context.supervisor_sessions, - ) - .await - .map_err(|error| Error::execution(format!("failed to create compute runtime: {error}")))?; - Ok(ComputeDriverBuildOutput { - runtime, - operator_allowlist: None, - }) + .and_then(|file| file.openshell.gateway.otlp.as_ref()) } } #[allow(clippy::too_many_arguments)] async fn build_compute_runtime( + registry: &ComputeDriverRegistry, + selection: &ComputeDriverSelection, config: &Config, driver_startup: compute::driver_config::DriverStartupContext<'_>, - driver: ConfiguredComputeDriver, store: Arc, sandbox_index: SandboxIndex, sandbox_watch_bus: SandboxWatchBus, tracing_log_bus: TracingLogBus, supervisor_sessions: Arc, shutdown_rx: watch::Receiver, -) -> Result<(ComputeRuntime, OperatorAllowlistArc)> { +) -> Result { + let driver = resolve_configured_compute_driver(registry, selection.name(), driver_startup)?; + let telemetry_compute_driver = driver.telemetry_compute_driver(registry); info!(driver = %driver.name(), "Using compute driver"); + if config + .gateway_jwt + .as_ref() + .is_some_and(|jwt| jwt.ttl_secs == 0) + && !driver.is_local_singleplayer(registry) + { + warn!( + "Gateway configured with non-expiring sandbox JWTs; set gateway_jwt.ttl_secs > 0 for shared deployments" + ); + } - let (runtime, operator_allowlist) = match driver { + let runtime = match driver { ConfiguredComputeDriver::Registered(registration) => { - let output = registration - .factory - .build(ComputeDriverBuildContext { - driver_name: registration.name, - config, - driver_startup, + let build_context = ComputeDriverBuildContext { + driver_name: registration.name.clone(), + gateway_name: &config.name, + gateway_bind_address: config.bind_address, + gateway_log_level: &config.log_level, + driver_startup, + shutdown_rx, + inherited_config_keys: registration.inherited_config_keys, + }; + let instance = registration.factory.build(build_context).await?; + match instance { + ComputeDriverInstance::InProcess(driver) => ComputeRuntime::from_driver( + registration.name, + driver, + None, store, sandbox_index, sandbox_watch_bus, tracing_log_bus, supervisor_sessions, - shutdown_rx, - }) - .await?; - (output.runtime, output.operator_allowlist) + ) + .await + .map_err(|error| { + Error::execution(format!("failed to create compute runtime: {error}")) + })?, + ComputeDriverInstance::ManagedRemote(mut endpoint) => { + endpoint.name = registration.name; + ComputeRuntime::new_remote_driver( + endpoint, + store, + sandbox_index, + sandbox_watch_bus, + tracing_log_bus, + supervisor_sessions, + ) + .await + .map_err(|error| { + Error::execution(format!("failed to create compute runtime: {error}")) + })? + } + } } ConfiguredComputeDriver::Remote { name } => { let remote_config = @@ -1579,7 +1443,7 @@ async fn build_compute_runtime( let endpoint = compute::connect_remote_compute_driver(name, &remote_config.socket_path) .await .map_err(|e| Error::execution(format!("failed to create compute runtime: {e}")))?; - let rt = ComputeRuntime::new_remote_driver( + ComputeRuntime::new_remote_driver( endpoint, store, sandbox_index, @@ -1588,58 +1452,56 @@ async fn build_compute_runtime( supervisor_sessions, ) .await - .map_err(|e| Error::execution(format!("failed to create compute runtime: {e}")))?; - (rt, None) + .map_err(|e| Error::execution(format!("failed to create compute runtime: {e}")))? } }; - Ok((runtime, operator_allowlist)) -} - -fn compute_driver_startup_context<'a>( - config: &'a Config, - config_file: Option<&'a config_file::ConfigFile>, - guest_tls: Option<&'a compute::driver_config::GuestTlsPaths>, -) -> compute::driver_config::DriverStartupContext<'a> { - compute::driver_config::DriverStartupContext { - file: config_file, - guest_tls, - gateway_port: config.bind_address.port(), - gateway_tls_enabled: config.tls.is_some(), - endpoint_overrides: &config.compute_driver_endpoints, - } + Ok(runtime.with_telemetry_compute_driver(telemetry_compute_driver)) } #[derive(Debug, Clone)] -pub(crate) enum ConfiguredComputeDriver { +enum ConfiguredComputeDriver { Registered(ComputeDriverRegistration), Remote { name: String }, } impl ConfiguredComputeDriver { - pub(crate) fn name(&self) -> &str { + fn name(&self) -> &str { match self { Self::Registered(registration) => ®istration.name, Self::Remote { name } => name, } } + + fn is_local_singleplayer(&self, registry: &ComputeDriverRegistry) -> bool { + match self { + Self::Registered(registration) => registration.is_local_singleplayer(), + Self::Remote { name } => registry + .get(name) + .is_some_and(ComputeDriverRegistration::is_local_singleplayer), + } + } + + fn telemetry_compute_driver(&self, registry: &ComputeDriverRegistry) -> TelemetryComputeDriver { + match self { + Self::Registered(registration) => registration.telemetry_category, + Self::Remote { name } => registry + .get(name) + .map_or_else(TelemetryComputeDriver::custom, |registration| { + registration.telemetry_category + }), + } + } } -pub(crate) fn configured_compute_driver_for_startup( +#[cfg(test)] +fn configured_compute_driver( registry: &ComputeDriverRegistry, - startup: &ServerStartupConfig, + config: &Config, + driver_startup: compute::driver_config::DriverStartupContext<'_>, ) -> Result { - resolve_configured_compute_driver( - registry, - startup.compute_driver.name(), - compute::driver_config::DriverStartupContext { - file: startup.config_file.as_ref(), - guest_tls: startup.guest_tls.as_ref(), - gateway_port: startup.config.bind_address.port(), - gateway_tls_enabled: startup.config.tls.is_some(), - endpoint_overrides: &startup.config.compute_driver_endpoints, - }, - ) + let selection = registry.select(&config.compute_drivers)?; + resolve_configured_compute_driver(registry, selection.name(), driver_startup) } fn resolve_configured_compute_driver( @@ -1663,23 +1525,6 @@ fn resolve_configured_compute_driver( Ok(ConfiguredComputeDriver::Remote { name }) } -#[cfg(any(test, feature = "in-tree-compute-drivers"))] -fn kubernetes_sandbox_jwt_expiry_disabled(config: &Config) -> bool { - config - .gateway_jwt - .as_ref() - .is_some_and(|jwt| jwt.ttl_secs == 0) -} - -#[cfg(feature = "in-tree-compute-drivers")] -fn warn_if_kubernetes_sandbox_jwt_expiry_disabled(config: &Config) { - if kubernetes_sandbox_jwt_expiry_disabled(config) { - warn!( - "Kubernetes gateway configured with non-expiring sandbox JWTs (gateway_jwt.ttl_secs = 0); set ttl_secs > 0 for shared Kubernetes deployments" - ); - } -} - pub(crate) async fn ensure_default_workspace(store: &Store) -> Result<()> { use grpc::workspace::{DEFAULT_WORKSPACE_NAME, WORKSPACE_OBJECT_TYPE}; use openshell_core::proto::Workspace; @@ -1745,11 +1590,11 @@ mod tests { BoundGatewayListener, ConfiguredComputeDriver, ConnectionProtocol, ExtensionKind, GatewayListenerScope, MultiplexService, ServerState, TlsAcceptor, allow_plaintext_service_http, bind_gateway_listeners, classify_initial_bytes, - is_benign_tls_handshake_failure, kubernetes_sandbox_jwt_expiry_disabled, + configured_compute_driver, is_benign_tls_handshake_failure, mint_gateway_extension_credential, serve_gateway_listener, }; use openshell_core::{ - ComputeDriverKind, Config, + Config, proto::{HealthRequest, open_shell_client::OpenShellClient}, }; use std::io::{Error, ErrorKind}; @@ -1765,11 +1610,30 @@ mod tests { use tokio::sync::watch; use crate::{ - compute::GatewayListenerRequirement, - gateway_listener::GatewayListenerSpec, - tls_test_utils::{generate_test_certs_with_ca, install_rustls_provider}, + compute::GatewayListenerRequirement, gateway_listener::GatewayListenerSpec, + tls_test_utils::generate_test_certs_with_ca, }; + static DETECTION_PROBE_ORDER: LazyLock>> = + LazyLock::new(|| Mutex::new(Vec::new())); + + fn record_detection_probe(name: &'static str, available: bool) -> bool { + DETECTION_PROBE_ORDER.lock().unwrap().push(name); + available + } + + fn unavailable_first_probe() -> bool { + record_detection_probe("first", false) + } + + fn available_second_probe() -> bool { + record_detection_probe("second", true) + } + + fn available_third_probe() -> bool { + record_detection_probe("third", true) + } + fn extension_test_issuer() -> Arc { let material = openshell_bootstrap::jwt::generate_jwt_key().expect("jwt key"); Arc::new( @@ -1777,7 +1641,7 @@ mod tests { material.signing_key_pem.as_bytes(), material.kid, "gateway-a", - Duration::from_secs(900), + Duration::from_mins(15), ) .expect("issuer"), ) @@ -1872,54 +1736,42 @@ mod tests { } fn test_compute_drivers() -> super::ComputeDriverRegistry { - super::install_default_compute_drivers() - } - - fn select_compute_driver( - registry: &super::ComputeDriverRegistry, - config: &Config, - driver_startup: crate::compute::driver_config::DriverStartupContext<'_>, - ) -> openshell_core::Result { - let selection = registry.select(&config.compute_drivers)?; - super::resolve_configured_compute_driver(registry, selection.name(), driver_startup) + let mut registry = super::ComputeDriverRegistry::new(); + for (name, priority) in [("alpha", 100), ("beta", 200), ("gamma", 300)] { + registry + .install( + super::ComputeDriverRegistration::new( + name, + priority, + None, + TestComputeDriverFactory, + ) + .unwrap() + .with_telemetry_category( + openshell_core::telemetry::TelemetryComputeDriver::anonymous_category( + "registered", + ), + ), + ) + .unwrap(); + } + registry } #[derive(Clone, Copy)] struct TestComputeDriverFactory; - static DETECTION_PROBE_ORDER: LazyLock>> = - LazyLock::new(|| Mutex::new(Vec::new())); - - fn record_detection_probe(name: &'static str, available: bool) -> bool { - DETECTION_PROBE_ORDER.lock().unwrap().push(name); - available - } - - fn unavailable_first_probe() -> bool { - record_detection_probe("first", false) - } - - fn available_second_probe() -> bool { - record_detection_probe("second", true) - } - - fn available_third_probe() -> bool { - record_detection_probe("third", true) - } - #[async_trait::async_trait] impl super::ComputeDriverFactory for TestComputeDriverFactory { async fn build( &self, _context: super::ComputeDriverBuildContext<'_>, - ) -> openshell_core::Result { + ) -> openshell_core::Result { unreachable!("selection tests do not construct the driver") } } fn test_tls_acceptor() -> (TempDir, TlsAcceptor) { - install_rustls_provider(); - let dir = tempdir().expect("failed to create tempdir"); generate_test_certs_with_ca(dir.path()); @@ -2226,38 +2078,31 @@ mod tests { #[test] fn configured_compute_driver_triggers_auto_detection_when_empty() { - let config = Config::new(None).with_compute_drivers(std::iter::empty::()); - // Empty drivers triggers auto-detection, which may return Some or None - // depending on the environment. This test verifies the auto-detection path - // is taken rather than immediately returning an error. - let result = select_compute_driver( - &test_compute_drivers(), - &config, - test_driver_startup(&config, None), - ); - // Either we get a detected driver or an error about none being detected. - match result { - Ok(ConfiguredComputeDriver::Registered(registration)) => { - assert!( - matches!( - registration.name.as_str(), - "kubernetes" | "docker" | "podman" - ), - "auto-detected unexpected driver: {}", - registration.name - ); - } - Ok(ConfiguredComputeDriver::Remote { name }) => { - panic!("auto-detection returned remote driver: {name}"); - } - Err(e) => { - assert!( - e.to_string() - .contains("auto-detection found no suitable installed driver"), - "unexpected error: {e}" - ); - } + fn available() -> bool { + true } + + let mut registry = super::ComputeDriverRegistry::new(); + registry + .install( + super::ComputeDriverRegistration::new( + "detected", + 100, + Some(available), + TestComputeDriverFactory, + ) + .unwrap(), + ) + .unwrap(); + let config = Config::new(None).with_compute_drivers(std::iter::empty::()); + let result = + configured_compute_driver(®istry, &config, test_driver_startup(&config, None)) + .unwrap(); + + let ConfiguredComputeDriver::Registered(registration) = result else { + panic!("auto-detection must select a registered driver"); + }; + assert_eq!(registration.name, "detected"); } #[test] @@ -2320,9 +2165,8 @@ mod tests { #[test] fn configured_compute_driver_rejects_multiple_entries() { - let config = Config::new(None) - .with_compute_drivers([ComputeDriverKind::Kubernetes, ComputeDriverKind::Podman]); - let err = select_compute_driver( + let config = Config::new(None).with_compute_drivers(["alpha", "beta"]); + let err = configured_compute_driver( &test_compute_drivers(), &config, test_driver_startup(&config, None), @@ -2332,64 +2176,38 @@ mod tests { err.to_string() .contains("multiple compute drivers are not supported yet") ); - assert!(err.to_string().contains("kubernetes,podman")); + assert!(err.to_string().contains("alpha,beta")); } #[test] - fn configured_compute_driver_accepts_podman() { - let config = Config::new(None).with_compute_drivers([ComputeDriverKind::Podman]); - let driver = select_compute_driver( - &test_compute_drivers(), - &config, - test_driver_startup(&config, None), - ) - .unwrap(); + fn configured_compute_driver_accepts_registered_name() { + let config = Config::new(None).with_compute_drivers(["beta"]); + let registry = test_compute_drivers(); + let driver = + configured_compute_driver(®istry, &config, test_driver_startup(&config, None)) + .unwrap(); + assert_eq!( + driver.telemetry_compute_driver(®istry).as_str(), + "registered" + ); assert!(matches!( driver, - ConfiguredComputeDriver::Registered(registration) if registration.name == "podman" - )); - } - - #[test] - fn configured_compute_driver_accepts_vm() { - let config = Config::new(None).with_compute_drivers([ComputeDriverKind::Vm]); - let driver = select_compute_driver( - &test_compute_drivers(), - &config, - test_driver_startup(&config, None), - ) - .unwrap(); - assert!(matches!( - driver, - ConfiguredComputeDriver::Registered(registration) if registration.name == "vm" - )); - } - - #[test] - fn configured_compute_driver_accepts_docker() { - let config = Config::new(None).with_compute_drivers([ComputeDriverKind::Docker]); - let driver = select_compute_driver( - &test_compute_drivers(), - &config, - test_driver_startup(&config, None), - ) - .unwrap(); - assert!(matches!( - driver, - ConfiguredComputeDriver::Registered(registration) if registration.name == "docker" + ConfiguredComputeDriver::Registered(registration) if registration.name == "beta" )); } #[test] fn configured_compute_driver_resolves_named_remote() { let config = Config::new(None).with_compute_drivers(["kyma"]); + let registry = test_compute_drivers(); - let driver = select_compute_driver( - &test_compute_drivers(), - &config, - test_driver_startup(&config, None), - ) - .unwrap(); + let driver = + configured_compute_driver(®istry, &config, test_driver_startup(&config, None)) + .unwrap(); + assert_eq!( + driver.telemetry_compute_driver(®istry).as_str(), + "custom" + ); match driver { ConfiguredComputeDriver::Remote { name } => { @@ -2405,30 +2223,32 @@ mod tests { } #[test] - fn configured_compute_driver_uses_vm_endpoint_override() { + fn configured_compute_driver_uses_endpoint_override() { let config = Config::new(None) - .with_compute_drivers([ComputeDriverKind::Vm]) - .with_compute_driver_endpoint("vm", "/run/openshell/vm.sock"); + .with_compute_drivers(["alpha"]) + .with_compute_driver_endpoint("alpha", "/run/openshell/alpha.sock"); + let registry = test_compute_drivers(); - let driver = select_compute_driver( - &test_compute_drivers(), - &config, - test_driver_startup(&config, None), - ) - .unwrap(); + let driver = + configured_compute_driver(®istry, &config, test_driver_startup(&config, None)) + .unwrap(); + assert_eq!( + driver.telemetry_compute_driver(®istry).as_str(), + "registered" + ); assert!(matches!( driver, - ConfiguredComputeDriver::Remote { name } if name == "vm" + ConfiguredComputeDriver::Remote { name } if name == "alpha" )); } #[test] fn configured_compute_driver_uses_builtin_endpoint_override() { let config = Config::new(None) - .with_compute_drivers([ComputeDriverKind::Docker]) - .with_compute_driver_endpoint("docker", "/run/openshell/docker.sock"); + .with_compute_drivers(["beta"]) + .with_compute_driver_endpoint("beta", "/run/openshell/beta.sock"); - let driver = select_compute_driver( + let driver = configured_compute_driver( &test_compute_drivers(), &config, test_driver_startup(&config, None), @@ -2436,50 +2256,10 @@ mod tests { .unwrap(); assert!(matches!( driver, - ConfiguredComputeDriver::Remote { name } if name == "docker" + ConfiguredComputeDriver::Remote { name } if name == "beta" )); } - #[test] - fn kubernetes_sandbox_jwt_expiry_disabled_warns_for_zero_ttl() { - fn config_with_jwt_ttl(ttl_secs: u64) -> Config { - let mut config = Config::new(None); - config.gateway_jwt = Some(openshell_core::GatewayJwtConfig { - signing_key_path: "/tmp/signing.pem".into(), - public_key_path: "/tmp/public.pem".into(), - kid_path: "/tmp/kid".into(), - gateway_id: "openshell".to_string(), - ttl_secs, - }); - config - } - - assert!(kubernetes_sandbox_jwt_expiry_disabled( - &config_with_jwt_ttl(0) - )); - assert!(!kubernetes_sandbox_jwt_expiry_disabled( - &config_with_jwt_ttl(3600) - )); - assert!(!kubernetes_sandbox_jwt_expiry_disabled(&Config::new(None))); - } - - #[cfg(target_os = "windows")] - #[test] - fn windows_builtin_compute_drivers_report_unsupported() { - for driver in [ - ComputeDriverKind::Docker, - ComputeDriverKind::Kubernetes, - ComputeDriverKind::Podman, - ComputeDriverKind::Vm, - ] { - let message = super::unsupported_builtin_compute_driver(driver).to_string(); - assert!( - message.contains("unsupported on Windows"), - "{driver} rejection should be explicit, got: {message}" - ); - } - } - #[tokio::test] async fn failed_gateway_listener_bind_does_not_attempt_persisted_sandbox_start() { let occupied_listener = TcpListener::bind("127.0.0.1:0").await.unwrap(); diff --git a/crates/openshell-server/src/multiplex.rs b/crates/openshell-server/src/multiplex.rs index 3ef774e4c..f03aa4d07 100644 --- a/crates/openshell-server/src/multiplex.rs +++ b/crates/openshell-server/src/multiplex.rs @@ -17,9 +17,7 @@ use hyper_util::{ service::TowerToHyperService, }; use metrics::{counter, histogram}; -use openshell_core::proto::{ - inference_server::InferenceServer, open_shell_server::OpenShellServer, -}; +use openshell_core::proto::open_shell_server::OpenShellServer; use openshell_core::{ Config, proto::{Provider, UpdateProviderRequest}, @@ -50,11 +48,9 @@ use crate::{ auth::identity::Identity, auth::oidc::{self, OidcAuthenticator}, auth::principal::{Principal, UserPrincipal}, - auth::workspace_authz::{MinWorkspaceRole, authorize_workspace}, + auth::workspace_authz::{MinWorkspaceRole, authorize_workspace_selector}, gateway_listener::GatewayListenerScope, - http_router, - inference::InferenceService, - service_http_router, + http_router, service_http_router, }; /// Request-ID generator that produces a UUID v4 for each inbound request. @@ -277,8 +273,6 @@ impl MultiplexService { self.state.gateway_interceptors.clone(), Some(self.state.clone()), ); - let inference = InferenceServer::new(InferenceService::new(self.state.clone())) - .max_decoding_message_size(MAX_GRPC_DECODE_SIZE); let authz_policy = self.state.config.oidc.as_ref().map(|oidc| AuthzPolicy { admin_role: oidc.admin_role.clone(), user_role: oidc.user_role.clone(), @@ -286,7 +280,7 @@ impl MultiplexService { }); let authenticator_chain = build_authenticator_chain(&self.state); let grpc_service = AuthGrpcRouter::with_peer_identity( - GrpcRouter::new(openshell, inference), + openshell, authenticator_chain, authz_policy, self.state @@ -558,11 +552,11 @@ async fn hydrate_update_provider_identity( let principal = principal.ok_or_else(|| tonic::Status::unauthenticated("authentication required"))?; - let authorized = authorize_workspace( + let authorized = authorize_workspace_selector( state.store.as_ref(), &state.admin_role, principal, - &request.workspace, + request.workspace_scope.as_ref(), MinWorkspaceRole::Admin, ) .await?; @@ -920,59 +914,6 @@ where } } -/// Combined gRPC service that routes between `OpenShell` and Inference services -/// based on the request path prefix. -#[derive(Clone)] -pub struct GrpcRouter { - openshell: N, - inference: I, -} - -impl GrpcRouter { - fn new(openshell: N, inference: I) -> Self { - Self { - openshell, - inference, - } - } -} - -const INFERENCE_PATH_PREFIX: &str = "/openshell.inference.v1.Inference/"; - -impl tower::Service> for GrpcRouter -where - N: tower::Service> + Clone + Send + 'static, - N::Response: Send, - N::Future: Send, - N::Error: Send, - I: tower::Service, Response = N::Response, Error = N::Error> - + Clone - + Send - + 'static, - I::Future: Send, - B: Send + 'static, -{ - type Response = N::Response; - type Error = N::Error; - type Future = Pin> + Send>>; - - fn poll_ready(&mut self, _cx: &mut Context<'_>) -> Poll> { - Poll::Ready(Ok(())) - } - - fn call(&mut self, req: Request) -> Self::Future { - let is_inference = req.uri().path().starts_with(INFERENCE_PATH_PREFIX); - - if is_inference { - let mut svc = self.inference.clone(); - Box::pin(async move { svc.ready().await?.call(req).await }) - } else { - let mut svc = self.openshell.clone(); - Box::pin(async move { svc.ready().await?.call(req).await }) - } - } -} - /// Assemble the authenticator chain for the gateway. /// /// Chain order (first-match-wins): @@ -1488,7 +1429,6 @@ mod tests { "/openshell.v1.OpenShell/GetSandboxProviderEnvironment", "/openshell.v1.OpenShell/SubmitPolicyAnalysis", "/openshell.v1.OpenShell/RefreshSandboxToken", - "/openshell.inference.v1.Inference/GetInferenceBundle", ]; for path in callback_paths { @@ -1521,8 +1461,6 @@ mod tests { "/openshell.v1.OpenShell/ListSandboxes", "/openshell.v1.OpenShell/DeleteSandbox", "/openshell.v1.OpenShell/CreateProvider", - "/openshell.inference.v1.Inference/GetInferenceRoute", - "/openshell.inference.v1.Inference/SetInferenceRoute", ]; for path in rejected_grpc_paths { @@ -1541,7 +1479,6 @@ mod tests { let paths = [ "/grpc.health.v1.Health/Check", "/openshell.v1.OpenShell/ListSandboxes", - "/openshell.inference.v1.Inference/GetInferenceRoute", "/health", "/service", ]; @@ -1803,7 +1740,9 @@ mod tests { )]), ..Default::default() }), - workspace: "default".to_string(), + workspace_scope: Some(openshell_core::proto::workspace_selector( + "default".to_string(), + )), ..Default::default() }; let authed = crate::grpc::test_support::authed_request(()); @@ -2492,7 +2431,6 @@ mod tests { "/openshell.v1.OpenShell/CreateSandbox", "/openshell.v1.OpenShell/ListSandboxes", "/openshell.v1.OpenShell/DeleteSandbox", - "/openshell.inference.v1.Inference/GetInferenceBundle", "/metrics", ]; @@ -2515,7 +2453,6 @@ mod tests { let expected = [ "GET", - "openshell.inference.v1.Inference/GetInferenceBundle", "openshell.v1.OpenShell/CreateSandbox", "openshell.v1.OpenShell/DeleteSandbox", "openshell.v1.OpenShell/ListSandboxes", @@ -2542,13 +2479,6 @@ mod tests { otel_span_name(&http::Method::POST, "/openshell.v1.OpenShell/CreateSandbox"), "openshell.v1.OpenShell/CreateSandbox" ); - assert_eq!( - otel_span_name( - &http::Method::POST, - "/openshell.inference.v1.Inference/GetInferenceBundle" - ), - "openshell.inference.v1.Inference/GetInferenceBundle" - ); } /// Non-RPC paths use a low-cardinality method-only name because sandbox @@ -2575,14 +2505,6 @@ mod tests { ); } - #[test] - fn grpc_method_extracts_inference_service() { - assert_eq!( - grpc_method_from_path("/openshell.inference.v1.Inference/GetInferenceBundle"), - "GetInferenceBundle" - ); - } - #[test] fn grpc_method_handles_bare_path() { assert_eq!(grpc_method_from_path("Health"), "Health"); @@ -2910,27 +2832,6 @@ mod tests { )); } - #[tokio::test] - async fn sandbox_principal_can_fetch_inference_bundle() { - let mock = Arc::new(MockAuthenticator::returning(Ok(Some(sandbox_principal())))); - let chain = AuthenticatorChain::new(vec![mock]); - let (recorder, seen) = PrincipalRecorder::new(); - let mut router = AuthGrpcRouter::new(recorder, Some(chain), None); - - let res = router - .call(empty_request( - "/openshell.inference.v1.Inference/GetInferenceBundle", - )) - .await - .unwrap(); - - assert_eq!(res.status(), 200); - assert!(matches!( - seen.lock().unwrap().as_ref(), - Some(Principal::Sandbox(_)) - )); - } - /// A user principal — even one carrying `openshell:all` and the /// admin role — must not reach a `sandbox`-annotated method. The /// router enforces this from the per-handler auth-mode declarations @@ -2964,7 +2865,6 @@ mod tests { "/openshell.v1.OpenShell/RelayStream", "/openshell.v1.OpenShell/IssueSandboxToken", "/openshell.v1.OpenShell/RefreshSandboxToken", - "/openshell.inference.v1.Inference/GetInferenceBundle", ] { let mock = Arc::new(MockAuthenticator::returning(Ok(Some(admin_user())))); let chain = AuthenticatorChain::new(vec![mock]); @@ -3021,8 +2921,6 @@ mod tests { "/openshell.v1.OpenShell/DeleteSandbox", "/openshell.v1.OpenShell/CreateProvider", "/openshell.v1.OpenShell/ApproveDraftChunk", - "/openshell.inference.v1.Inference/GetInferenceRoute", - "/openshell.inference.v1.Inference/SetInferenceRoute", ] { let mock = Arc::new(MockAuthenticator::returning(Ok(Some(sandbox_principal())))); let chain = AuthenticatorChain::new(vec![mock]); diff --git a/crates/openshell-server/src/otel_tracing.rs b/crates/openshell-server/src/otel_tracing.rs index b4e8d9e50..da1250005 100644 --- a/crates/openshell-server/src/otel_tracing.rs +++ b/crates/openshell-server/src/otel_tracing.rs @@ -51,11 +51,15 @@ impl<'a> GatewayResourceAttributes<'a> { compute_driver, } } - /// The configured gateway installation name, if any. pub fn name(&self) -> Option<&'a str> { self.name } + + /// The configured compute-driver name, if any. + pub fn compute_driver(&self) -> Option<&'a str> { + self.compute_driver + } } fn trace_config<'cfg>( @@ -72,29 +76,14 @@ fn trace_config<'cfg>( ServiceName::Fixed, ); - let mut resource_attributes = Vec::new(); - if let Some(name) = gateway.name.map(str::trim).filter(|s| !s.is_empty()) { - resource_attributes.push(opentelemetry::KeyValue::new( - "openshell.gateway.name", - name.to_string(), - )); - } - if let Some(compute_driver) = gateway - .compute_driver - .map(str::trim) - .filter(|s| !s.is_empty()) - { - resource_attributes.push(opentelemetry::KeyValue::new( - "openshell.gateway.compute_driver", - compute_driver.to_string(), - )); - } - OtlpTraceConfig { endpoint: &cfg.endpoint, service_name, service_version: Some(openshell_core::VERSION), - resource_attributes, + resource_attributes: openshell_otel::gateway_resource_attributes( + gateway.name(), + gateway.compute_driver(), + ), } } @@ -134,19 +123,23 @@ pub fn provider_for( openshell_otel::provider_for(cfg.map(|cfg| trace_config(cfg, gateway))) } -/// Build the gateway layer while routing one selected in-process driver to -/// its own tracer provider. -pub fn layer_excluding_driver( +/// Build the `tracing` layer that forwards spans to `provider`. +/// +/// Events stay on the gateway's logging layers. Spans emitted by the +/// OpenTelemetry crates are excluded to prevent recursive export traffic. +pub fn layer( provider: &SdkTracerProvider, - driver_target_prefix: Option<&'static str>, + driver: Option, ) -> openshell_otel::TargetOtlpLayer where S: Subscriber + for<'span> LookupSpan<'span>, { - openshell_otel::layer_excluding_target_prefix( + openshell_otel::layer_excluding_target_prefixes( provider, INSTRUMENTATION_SCOPE, - driver_target_prefix, + driver + .into_iter() + .flat_map(openshell_otel::ComputeDriverTracing::in_process_targets), ) } @@ -180,8 +173,7 @@ pub mod test_exporter { let provider = opentelemetry_sdk::trace::SdkTracerProvider::builder() .with_simple_exporter(exporter.clone()) .build(); - let subscriber = - tracing_subscriber::registry().with(super::layer_excluding_driver(&provider, None)); + let subscriber = tracing_subscriber::registry().with(super::layer(&provider, None)); let dispatch = tracing::Dispatch::new(subscriber); TracingTestGuard { _default: tracing::dispatcher::set_default(&dispatch), diff --git a/crates/openshell-server/src/persistence/mod.rs b/crates/openshell-server/src/persistence/mod.rs index 516faf4fe..066ce6cc3 100644 --- a/crates/openshell-server/src/persistence/mod.rs +++ b/crates/openshell-server/src/persistence/mod.rs @@ -6,7 +6,7 @@ mod postgres; mod sqlite; -pub use openshell_core::proto::{ +pub use crate::storage_proto::{ StoredDraftChunk as DraftChunkRecord, StoredPolicyRevision as PolicyRecord, }; @@ -26,6 +26,12 @@ pub const DRAFT_CHUNK_OBJECT_TYPE: &str = "draft_policy_chunk"; pub type PersistenceResult = Result; +/// Maximum number of object ids sent in one set-based delete statement. +/// +/// Keep this well below `SQLite`'s bind-variable limit. Backends split larger +/// requests into independently retryable, bounded write statements. +pub const DELETE_MANY_BATCH_SIZE: usize = 128; + /// Persistence-layer error type. #[derive(Debug, Error, Clone)] pub enum PersistenceError { @@ -101,6 +107,30 @@ pub struct ObjectRecord { pub resource_version: u64, } +/// Stable position in the global object-listing order. +/// +/// Keyset consumers must use the matching store method for the order encoded +/// here: workspace-scoped lists use `created_at_ms`, `name`, and `id`; global +/// lists additionally include `workspace`. +#[derive(Debug, Clone)] +pub struct ObjectCursor { + pub created_at_ms: i64, + pub name: String, + pub workspace: String, + pub id: String, +} + +impl From<&ObjectRecord> for ObjectCursor { + fn from(record: &ObjectRecord) -> Self { + Self { + created_at_ms: record.created_at_ms, + name: record.name.clone(), + workspace: record.workspace.clone(), + id: record.id.clone(), + } + } +} + /// Write condition for compare-and-swap operations. #[derive(Debug, Clone, Copy)] pub enum WriteCondition { @@ -132,8 +162,9 @@ pub trait ObjectType { fn object_type() -> &'static str; } -// Import object metadata accessor traits from openshell-core -// (implementations for all proto types are in openshell-core::metadata) +// Import object metadata accessor traits from openshell-core. Implementations +// for public resource types live there; private storage types implement them +// in crate::storage_proto. pub use openshell_core::{ GetResourceVersion, ObjectId, ObjectLabels, ObjectName, ObjectWorkspace, SetResourceVersion, }; @@ -371,6 +402,39 @@ impl Store { )) } + /// Atomically insert a named object only if its workspace has fewer than + /// `max_count` objects of the same type. + /// + /// Returns `Ok(None)` when the quota is already full. A duplicate id or + /// `(object_type, workspace, name)` still returns + /// [`PersistenceError::UniqueViolation`]. + #[allow(clippy::too_many_arguments)] + #[tracing::instrument( + name = "store", + skip_all, + fields(otel.name = "store.create_if_workspace_count_below", otel.status_code = tracing::field::Empty, object_type = %object_type, object.id = %id, object.name = %name, workspace = %workspace, max_count = max_count) + )] + pub async fn create_if_workspace_count_below( + &self, + object_type: &str, + id: &str, + name: &str, + workspace: &str, + payload: &[u8], + labels: Option<&str>, + max_count: u64, + ) -> PersistenceResult> { + store_dispatch_traced!(self.create_if_workspace_count_below( + object_type, + id, + name, + workspace, + payload, + labels, + max_count + )) + } + /// Fetch an object by id. #[tracing::instrument( name = "store", @@ -415,6 +479,22 @@ impl Store { store_dispatch_traced!(self.delete(object_type, id)) } + /// Delete objects of one type by id in bounded, set-based statements. + #[tracing::instrument( + name = "store", + skip_all, + fields( + otel.name = "store.delete_many", + otel.status_code = tracing::field::Empty, + object_type = %object_type, + object_count = ids.len(), + batch_count = ids.len().div_ceil(DELETE_MANY_BATCH_SIZE), + ) + )] + pub async fn delete_many(&self, object_type: &str, ids: &[String]) -> PersistenceResult { + store_dispatch_traced!(self.delete_many(object_type, ids)) + } + /// Count objects of a given type within a workspace. #[tracing::instrument( name = "store", @@ -499,6 +579,46 @@ impl Store { store_dispatch_traced!(self.list_by_type(object_type, limit, offset)) } + /// List workspace objects after a stable cursor, without offset drift. + #[tracing::instrument( + name = "store", + skip_all, + fields( + otel.name = "store.list_after", + otel.status_code = tracing::field::Empty, + object_type = %object_type, + workspace = %workspace, + ) + )] + pub async fn list_after( + &self, + object_type: &str, + workspace: &str, + after: Option<&ObjectCursor>, + limit: u32, + ) -> PersistenceResult> { + store_dispatch_traced!(self.list_after(object_type, workspace, after, limit)) + } + + /// List objects across workspaces after a stable cursor, without offset drift. + #[tracing::instrument( + name = "store", + skip_all, + fields( + otel.name = "store.list_by_type_after", + otel.status_code = tracing::field::Empty, + object_type = %object_type, + ) + )] + pub async fn list_by_type_after( + &self, + object_type: &str, + after: Option<&ObjectCursor>, + limit: u32, + ) -> PersistenceResult> { + store_dispatch_traced!(self.list_by_type_after(object_type, after, limit)) + } + /// List objects by type and application-owned scope. /// /// Workspace filtering is intentionally omitted: scope values are sandbox diff --git a/crates/openshell-server/src/persistence/postgres.rs b/crates/openshell-server/src/persistence/postgres.rs index 8bab0ada9..449f8f6df 100644 --- a/crates/openshell-server/src/persistence/postgres.rs +++ b/crates/openshell-server/src/persistence/postgres.rs @@ -2,8 +2,8 @@ // SPDX-License-Identifier: Apache-2.0 use super::{ - DraftChunkRecord, ObjectRecord, PersistenceError, PersistenceResult, PolicyRecord, - WriteCondition, WriteResult, current_time_ms, map_db_error, map_migrate_error, + DraftChunkRecord, ObjectCursor, ObjectRecord, PersistenceError, PersistenceResult, + PolicyRecord, WriteCondition, WriteResult, current_time_ms, map_db_error, map_migrate_error, }; use crate::policy_store::{ AtomicPolicyRevisionWrite, draft_chunk_payload_from_record, draft_chunk_record_from_parts, @@ -14,11 +14,19 @@ use openshell_core::SetResourceVersion; use openshell_core::proto::Sandbox; use prost::Message; use sqlx::postgres::PgPoolOptions; -use sqlx::{Connection, PgPool, Row}; +use sqlx::{Connection, PgPool, Postgres, QueryBuilder, Row}; static POSTGRES_MIGRATOR: sqlx::migrate::Migrator = sqlx::migrate!("./migrations/postgres"); -use super::{DRAFT_CHUNK_OBJECT_TYPE, POLICY_OBJECT_TYPE}; +#[cfg(test)] +pub(super) fn embedded_migration_sql(version: i64) -> Option<&'static str> { + POSTGRES_MIGRATOR + .iter() + .find(|migration| migration.version == version) + .map(|migration| migration.sql.as_ref()) +} + +use super::{DELETE_MANY_BATCH_SIZE, DRAFT_CHUNK_OBJECT_TYPE, POLICY_OBJECT_TYPE}; #[derive(Debug, Clone)] pub struct PostgresStore { @@ -337,6 +345,73 @@ RETURNING resource_version, created_at_ms, updated_at_ms }) } + #[allow(clippy::too_many_arguments)] + pub async fn create_if_workspace_count_below( + &self, + object_type: &str, + id: &str, + name: &str, + workspace: &str, + payload: &[u8], + labels: Option<&str>, + max_count: u64, + ) -> PersistenceResult> { + let now_ms = current_time_ms(); + let labels_jsonb: Option = labels + .map(serde_json::from_str) + .transpose() + .map_err(|e| PersistenceError::Encode(format!("invalid labels JSON: {e}")))?; + let mut tx = self.pool.begin().await.map_err(|e| map_db_error(&e))?; + + sqlx::query("SELECT pg_advisory_xact_lock(hashtext($1), hashtext($2))") + .bind(object_type) + .bind(workspace) + .execute(&mut *tx) + .await + .map_err(|e| map_db_error(&e))?; + + let row: (i64,) = sqlx::query_as( + "SELECT COUNT(*) FROM objects WHERE object_type = $1 AND workspace = $2", + ) + .bind(object_type) + .bind(workspace) + .fetch_one(&mut *tx) + .await + .map_err(|e| map_db_error(&e))?; + let count = u64::try_from(row.0).unwrap_or(0); + if count >= max_count { + tx.commit().await.map_err(|e| map_db_error(&e))?; + return Ok(None); + } + + let row = sqlx::query( + r" +INSERT INTO objects (object_type, id, name, workspace, payload, created_at_ms, updated_at_ms, labels, resource_version) +VALUES ($1, $2, $3, $4, $5, $6, $6, COALESCE($7, '{}'::jsonb), 1) +RETURNING resource_version, created_at_ms, updated_at_ms +", + ) + .bind(object_type) + .bind(id) + .bind(name) + .bind(workspace) + .bind(payload) + .bind(now_ms) + .bind(labels_jsonb) + .fetch_one(&mut *tx) + .await + .map_err(|e| map_db_error(&e))?; + + tx.commit().await.map_err(|e| map_db_error(&e))?; + + let resource_version_i64: i64 = row.try_get("resource_version").unwrap_or(1); + Ok(Some(WriteResult { + resource_version: resource_version_i64.max(1).cast_unsigned(), + created_at_ms: row.get("created_at_ms"), + updated_at_ms: row.get("updated_at_ms"), + })) + } + pub async fn get( &self, object_type: &str, @@ -391,6 +466,28 @@ WHERE object_type = $1 AND workspace = $2 AND name = $3 Ok(result.rows_affected() > 0) } + pub async fn delete_many(&self, object_type: &str, ids: &[String]) -> PersistenceResult { + let mut deleted = 0_u64; + for ids in ids.chunks(DELETE_MANY_BATCH_SIZE) { + let mut query = + QueryBuilder::::new("DELETE FROM objects WHERE object_type = "); + query.push_bind(object_type).push(" AND id IN ("); + let mut separated = query.separated(", "); + for id in ids { + separated.push_bind(id); + } + separated.push_unseparated(")"); + + deleted += query + .build() + .execute(&self.pool) + .await + .map_err(|e| map_db_error(&e))? + .rows_affected(); + } + Ok(deleted) + } + pub async fn count_in_workspace( &self, object_type: &str, @@ -500,6 +597,33 @@ LIMIT $2 OFFSET $3 Ok(rows.into_iter().map(row_to_object_record).collect()) } + pub async fn list_after( + &self, + object_type: &str, + workspace: &str, + after: Option<&ObjectCursor>, + limit: u32, + ) -> PersistenceResult> { + let rows = if let Some(cursor) = after { + sqlx::query("SELECT object_type, id, name, workspace, payload, created_at_ms, updated_at_ms, labels, resource_version FROM objects WHERE object_type = $1 AND workspace = $2 AND (created_at_ms, name, id) > ($3, $4, $5) ORDER BY created_at_ms, name, id LIMIT $6").bind(object_type).bind(workspace).bind(cursor.created_at_ms).bind(&cursor.name).bind(&cursor.id).bind(i64::from(limit)).fetch_all(&self.pool).await + } else { + sqlx::query("SELECT object_type, id, name, workspace, payload, created_at_ms, updated_at_ms, labels, resource_version FROM objects WHERE object_type = $1 AND workspace = $2 ORDER BY created_at_ms, name, id LIMIT $3").bind(object_type).bind(workspace).bind(i64::from(limit)).fetch_all(&self.pool).await + }.map_err(|e| map_db_error(&e))?; + Ok(rows.into_iter().map(row_to_object_record).collect()) + } + pub async fn list_by_type_after( + &self, + object_type: &str, + after: Option<&ObjectCursor>, + limit: u32, + ) -> PersistenceResult> { + let rows = if let Some(cursor) = after { + sqlx::query("SELECT object_type, id, name, workspace, payload, created_at_ms, updated_at_ms, labels, resource_version FROM objects WHERE object_type = $1 AND (created_at_ms, name, workspace, id) > ($2, $3, $4, $5) ORDER BY created_at_ms, name, workspace, id LIMIT $6").bind(object_type).bind(cursor.created_at_ms).bind(&cursor.name).bind(&cursor.workspace).bind(&cursor.id).bind(i64::from(limit)).fetch_all(&self.pool).await + } else { + sqlx::query("SELECT object_type, id, name, workspace, payload, created_at_ms, updated_at_ms, labels, resource_version FROM objects WHERE object_type = $1 ORDER BY created_at_ms, name, workspace, id LIMIT $2").bind(object_type).bind(i64::from(limit)).fetch_all(&self.pool).await + }.map_err(|e| map_db_error(&e))?; + Ok(rows.into_iter().map(row_to_object_record).collect()) + } pub async fn list_with_membership( &self, diff --git a/crates/openshell-server/src/persistence/sqlite.rs b/crates/openshell-server/src/persistence/sqlite.rs index 658530f75..5497b96c2 100644 --- a/crates/openshell-server/src/persistence/sqlite.rs +++ b/crates/openshell-server/src/persistence/sqlite.rs @@ -2,8 +2,8 @@ // SPDX-License-Identifier: Apache-2.0 use super::{ - DraftChunkRecord, ObjectRecord, PersistenceError, PersistenceResult, PolicyRecord, - WriteCondition, WriteResult, current_time_ms, map_db_error, map_migrate_error, + DraftChunkRecord, ObjectCursor, ObjectRecord, PersistenceError, PersistenceResult, + PolicyRecord, WriteCondition, WriteResult, current_time_ms, map_db_error, map_migrate_error, }; use crate::policy_store::{ AtomicPolicyRevisionWrite, draft_chunk_payload_from_record, draft_chunk_record_from_parts, @@ -14,35 +14,65 @@ use openshell_core::SetResourceVersion; use openshell_core::paths::set_file_owner_only; use openshell_core::proto::Sandbox; use prost::Message; -use sqlx::sqlite::{SqliteConnectOptions, SqlitePoolOptions}; -use sqlx::{Connection, Row, SqlitePool}; +use sqlx::sqlite::{SqliteConnectOptions, SqliteConnection, SqlitePoolOptions}; +use sqlx::{Connection, QueryBuilder, Row, Sqlite, SqlitePool}; use std::path::{Path, PathBuf}; use std::str::FromStr; +use std::sync::Arc; +use std::sync::atomic::{AtomicU64, Ordering}; +use tokio::sync::Mutex; static SQLITE_MIGRATOR: sqlx::migrate::Migrator = sqlx::migrate!("./migrations/sqlite"); -use super::{DRAFT_CHUNK_OBJECT_TYPE, POLICY_OBJECT_TYPE}; +#[cfg(test)] +pub(super) fn embedded_migration_sql(version: i64) -> Option<&'static str> { + SQLITE_MIGRATOR + .iter() + .find(|migration| migration.version == version) + .map(|migration| migration.sql.as_ref()) +} +static IN_MEMORY_DB_SEQUENCE: AtomicU64 = AtomicU64::new(0); + +use super::{DELETE_MANY_BATCH_SIZE, DRAFT_CHUNK_OBJECT_TYPE, POLICY_OBJECT_TYPE}; #[derive(Debug, Clone)] pub struct SqliteStore { pool: SqlitePool, + #[cfg_attr(not(any(test, feature = "test-support")), allow(dead_code))] + in_memory_keepalive: Option>>>, +} + +#[cfg(test)] +pub(super) async fn replace_pool_connection(store: &SqliteStore) -> PersistenceResult<()> { + let connection = store.pool.acquire().await.map_err(|e| map_db_error(&e))?; + connection.close().await.map_err(|e| map_db_error(&e)) } impl SqliteStore { /// Closes the connection pool. #[cfg(test)] pub(crate) async fn close_for_test(&self) { - self.pool.close().await; + self.close().await; } pub async fn connect(url: &str) -> PersistenceResult { let is_in_memory = url.contains(":memory:") || url.contains("mode=memory"); let max_connections = if is_in_memory { 1 } else { 5 }; - let options = SqliteConnectOptions::from_str(url) + let mut options = SqliteConnectOptions::from_str(url) .map_err(|e| map_db_error(&e))? .create_if_missing(true); + if is_in_memory { + if options.get_filename().as_os_str().is_empty() + || options.get_filename() == Path::new(":memory:") + { + let sequence = IN_MEMORY_DB_SEQUENCE.fetch_add(1, Ordering::Relaxed); + options = options.filename(format!("file:openshell-in-memory-{sequence}")); + } + options = options.shared_cache(true); + } + let mut pool_options = SqlitePoolOptions::new() .max_connections(max_connections) .min_connections(max_connections); @@ -55,6 +85,15 @@ impl SqliteStore { // so we can restrict the permissions after the database is connected. let db_path = (!is_in_memory).then(|| options.get_filename().to_path_buf()); + let in_memory_keepalive = if is_in_memory { + let connection = SqliteConnection::connect_with(&options) + .await + .map_err(|e| map_db_error(&e))?; + Some(Arc::new(Mutex::new(Some(connection)))) + } else { + None + }; + let pool = pool_options .connect_with(options) .await @@ -65,7 +104,10 @@ impl SqliteStore { restrict_db_file_permissions(&path)?; } - Ok(Self { pool }) + Ok(Self { + pool, + in_memory_keepalive, + }) } pub async fn migrate(&self) -> PersistenceResult<()> { @@ -88,6 +130,12 @@ impl SqliteStore { #[cfg(any(test, feature = "test-support"))] pub async fn close(&self) { self.pool.close().await; + if let Some(keepalive) = &self.in_memory_keepalive { + let connection = keepalive.lock().await.take(); + if let Some(connection) = connection { + let _ = connection.close().await; + } + } } pub async fn put( @@ -357,6 +405,67 @@ VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?7, ?8, 1) }) } + #[allow(clippy::too_many_arguments)] + pub async fn create_if_workspace_count_below( + &self, + object_type: &str, + id: &str, + name: &str, + workspace: &str, + payload: &[u8], + labels: Option<&str>, + max_count: u64, + ) -> PersistenceResult> { + let now_ms = current_time_ms(); + let mut tx = self + .pool + .begin_with("BEGIN IMMEDIATE") + .await + .map_err(|e| map_db_error(&e))?; + + let row: (i64,) = sqlx::query_as( + r#" +SELECT COUNT(*) FROM "objects" +WHERE "object_type" = ?1 AND "workspace" = ?2 +"#, + ) + .bind(object_type) + .bind(workspace) + .fetch_one(&mut *tx) + .await + .map_err(|e| map_db_error(&e))?; + let count = u64::try_from(row.0).unwrap_or(0); + if count >= max_count { + tx.commit().await.map_err(|e| map_db_error(&e))?; + return Ok(None); + } + + sqlx::query( + r#" +INSERT INTO "objects" ("object_type", "id", "name", "workspace", "payload", "created_at_ms", "updated_at_ms", "labels", "resource_version") +VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?6, ?7, 1) +"#, + ) + .bind(object_type) + .bind(id) + .bind(name) + .bind(workspace) + .bind(payload) + .bind(now_ms) + .bind(labels.unwrap_or("{}")) + .execute(&mut *tx) + .await + .map_err(|e| map_db_error(&e))?; + + tx.commit().await.map_err(|e| map_db_error(&e))?; + + Ok(Some(WriteResult { + resource_version: 1, + created_at_ms: now_ms, + updated_at_ms: now_ms, + })) + } + pub async fn get( &self, object_type: &str, @@ -416,6 +525,27 @@ WHERE "object_type" = ?1 AND "id" = ?2 Ok(result.rows_affected() > 0) } + pub async fn delete_many(&self, object_type: &str, ids: &[String]) -> PersistenceResult { + let mut deleted = 0_u64; + for ids in ids.chunks(DELETE_MANY_BATCH_SIZE) { + let mut query = QueryBuilder::::new("DELETE FROM objects WHERE object_type = "); + query.push_bind(object_type).push(" AND id IN ("); + let mut separated = query.separated(", "); + for id in ids { + separated.push_bind(id); + } + separated.push_unseparated(")"); + + deleted += query + .build() + .execute(&self.pool) + .await + .map_err(|e| map_db_error(&e))? + .rows_affected(); + } + Ok(deleted) + } + pub async fn count_in_workspace( &self, object_type: &str, @@ -541,6 +671,77 @@ LIMIT ?2 OFFSET ?3 Ok(rows.into_iter().map(row_to_object_record).collect()) } + pub async fn list_after( + &self, + object_type: &str, + workspace: &str, + after: Option<&ObjectCursor>, + limit: u32, + ) -> PersistenceResult> { + let rows = if let Some(cursor) = after { + sqlx::query( + r#" +SELECT "object_type", "id", "name", "workspace", "payload", "created_at_ms", "updated_at_ms", "labels", "resource_version" +FROM "objects" +WHERE "object_type" = ?1 AND "workspace" = ?2 + AND ("created_at_ms", "name", "id") > (?3, ?4, ?5) +ORDER BY "created_at_ms" ASC, "name" ASC, "id" ASC +LIMIT ?6 +"#, + ) + .bind(object_type) + .bind(workspace) + .bind(cursor.created_at_ms) + .bind(&cursor.name) + .bind(&cursor.id) + .bind(i64::from(limit)) + .fetch_all(&self.pool) + .await + } else { + sqlx::query( + r#" +SELECT "object_type", "id", "name", "workspace", "payload", "created_at_ms", "updated_at_ms", "labels", "resource_version" +FROM "objects" +WHERE "object_type" = ?1 AND "workspace" = ?2 +ORDER BY "created_at_ms" ASC, "name" ASC, "id" ASC +LIMIT ?3 +"#, + ) + .bind(object_type) + .bind(workspace) + .bind(i64::from(limit)) + .fetch_all(&self.pool) + .await + } + .map_err(|e| map_db_error(&e))?; + Ok(rows.into_iter().map(row_to_object_record).collect()) + } + pub async fn list_by_type_after( + &self, + object_type: &str, + after: Option<&ObjectCursor>, + limit: u32, + ) -> PersistenceResult> { + let rows = if let Some(cursor) = after { + sqlx::query(r#" +SELECT "object_type", "id", "name", "workspace", "payload", "created_at_ms", "updated_at_ms", "labels", "resource_version" +FROM "objects" +WHERE "object_type" = ?1 + AND ("created_at_ms", "name", "workspace", "id") > (?2, ?3, ?4, ?5) +ORDER BY "created_at_ms" ASC, "name" ASC, "workspace" ASC, "id" ASC +LIMIT ?6 +"#).bind(object_type).bind(cursor.created_at_ms).bind(&cursor.name).bind(&cursor.workspace).bind(&cursor.id).bind(i64::from(limit)).fetch_all(&self.pool).await + } else { + sqlx::query(r#" +SELECT "object_type", "id", "name", "workspace", "payload", "created_at_ms", "updated_at_ms", "labels", "resource_version" +FROM "objects" +WHERE "object_type" = ?1 +ORDER BY "created_at_ms" ASC, "name" ASC, "workspace" ASC, "id" ASC +LIMIT ?2 +"#).bind(object_type).bind(i64::from(limit)).fetch_all(&self.pool).await + }.map_err(|e| map_db_error(&e))?; + Ok(rows.into_iter().map(row_to_object_record).collect()) + } pub async fn list_with_membership( &self, @@ -629,7 +830,8 @@ AND EXISTS ( ) .unwrap(); - let mut query = sqlx::query(&sql) + // Label paths above escape SQL quotes; all values remain bound parameters. + let mut query = sqlx::query(sqlx::AssertSqlSafe(sql.as_str())) .bind(object_type) .bind(member_type) .bind(member_name); diff --git a/crates/openshell-server/src/persistence/tests.rs b/crates/openshell-server/src/persistence/tests.rs index 8802ac8d1..169fa8153 100644 --- a/crates/openshell-server/src/persistence/tests.rs +++ b/crates/openshell-server/src/persistence/tests.rs @@ -132,6 +132,108 @@ async fn sqlite_connect_runs_embedded_migrations() { assert!(records.is_empty()); } +#[tokio::test] +async fn sqlite_inference_route_removal_migration_deletes_only_managed_routes() { + use sqlx::{Connection, SqliteConnection}; + + let migration = super::sqlite::embedded_migration_sql(7) + .expect("SQLite migrator must embed removal migration 007"); + let mut connection = SqliteConnection::connect("sqlite::memory:") + .await + .expect("connect to migration test database"); + sqlx::raw_sql( + "CREATE TABLE objects (object_type TEXT NOT NULL, id TEXT NOT NULL);\ + INSERT INTO objects VALUES ('inference_route', 'managed-route');\ + INSERT INTO objects VALUES ('sandbox', 'preserved-sandbox');", + ) + .execute(&mut connection) + .await + .expect("seed pre-migration objects"); + + sqlx::raw_sql(migration) + .execute(&mut connection) + .await + .expect("run SQLite removal migration"); + + let remaining: Vec<(String, String)> = + sqlx::query_as("SELECT object_type, id FROM objects ORDER BY object_type, id") + .fetch_all(&mut connection) + .await + .expect("read migrated objects"); + assert_eq!( + remaining, + vec![("sandbox".to_string(), "preserved-sandbox".to_string())], + "removal migration must purge managed routes without touching other objects" + ); +} + +#[test] +fn embedded_migrators_include_inference_route_removal() { + for (backend, migration) in [ + ("sqlite", super::sqlite::embedded_migration_sql(7)), + ("postgres", super::postgres::embedded_migration_sql(7)), + ] { + let sql = + migration.unwrap_or_else(|| panic!("{backend} migrator is missing migration 007")); + assert!( + sql.contains("DELETE FROM objects WHERE object_type = 'inference_route'"), + "{backend} migration 007 must purge managed inference route objects" + ); + } +} + +#[tokio::test] +async fn sqlite_in_memory_store_survives_pool_connection_replacement() { + for url in ["sqlite::memory:", "sqlite://?mode=memory"] { + let store = super::sqlite::SqliteStore::connect(url) + .await + .expect("connect to in-memory SQLite"); + store.migrate().await.expect("migrate in-memory SQLite"); + store + .put( + "sandbox", + "before-replacement", + "before-replacement", + "default", + b"before", + None, + ) + .await + .expect("write before connection replacement"); + + super::sqlite::replace_pool_connection(&store) + .await + .expect("replace operational pool connection"); + + let preserved = store + .get("sandbox", "before-replacement") + .await + .expect("schema survives connection replacement") + .expect("existing object survives connection replacement"); + assert_eq!(preserved.payload, b"before", "database URL: {url}"); + + store + .put( + "sandbox", + "after-replacement", + "after-replacement", + "default", + b"after", + None, + ) + .await + .expect("write after connection replacement"); + assert!( + store + .get("sandbox", "after-replacement") + .await + .expect("read after connection replacement") + .is_some(), + "database URL: {url}" + ); + } +} + #[cfg(unix)] #[tokio::test] async fn sqlite_connect_restricts_db_file_permissions() { @@ -366,6 +468,116 @@ async fn sqlite_delete_behavior() { assert!(!deleted_again); } +#[tokio::test] +async fn delete_many_is_bounded_idempotent_and_type_scoped() { + let store = test_store().await; + let mut ids = Vec::new(); + for idx in 0..(super::DELETE_MANY_BATCH_SIZE + 12) { + let id = format!("sandbox-{idx}"); + store + .put( + "sandbox", + &id, + &format!("name-{idx}"), + "default", + b"payload", + None, + ) + .await + .unwrap(); + ids.push(id); + } + store + .put( + "provider", + "other-type", + "other-type", + "default", + b"payload", + None, + ) + .await + .unwrap(); + + ids.extend([ + "missing".to_string(), + "other-type".to_string(), + "sandbox-0".to_string(), + ]); + let expected = u64::try_from(super::DELETE_MANY_BATCH_SIZE + 12).unwrap(); + assert_eq!(store.delete_many("sandbox", &ids).await.unwrap(), expected); + assert_eq!(store.delete_many("sandbox", &ids).await.unwrap(), 0); + assert_eq!(store.delete_many("sandbox", &[]).await.unwrap(), 0); + assert!(store.get("provider", "other-type").await.unwrap().is_some()); +} + +#[tokio::test] +async fn file_backed_sqlite_bulk_delete_allows_concurrent_control_reads() { + use std::sync::Arc; + use std::sync::atomic::{AtomicBool, Ordering}; + + let tmp = tempfile::tempdir().expect("tempdir"); + let url = format!("sqlite:{}?mode=rwc", tmp.path().join("bulk.db").display()); + let store = Store::connect(&url) + .await + .expect("connect file-backed store"); + store + .put( + "provider", + "control-row", + "control-row", + "default", + b"control", + None, + ) + .await + .unwrap(); + + let mut ids = Vec::new(); + for idx in 0..500 { + let id = format!("session-{idx}"); + store + .put("ssh_session", &id, &id, "default", b"payload", None) + .await + .unwrap(); + ids.push(id); + } + + let stop = Arc::new(AtomicBool::new(false)); + let read_store = store.clone(); + let read_stop = stop.clone(); + let (started_tx, started_rx) = tokio::sync::oneshot::channel(); + let reader = tokio::spawn(async move { + let mut reads = 0_usize; + let mut started_tx = Some(started_tx); + while !read_stop.load(Ordering::Relaxed) { + read_store + .get("provider", "control-row") + .await + .map_err(|error| error.to_string())? + .ok_or_else(|| "control row disappeared".to_string())?; + reads += 1; + if let Some(started_tx) = started_tx.take() { + let _ = started_tx.send(()); + } + tokio::task::yield_now().await; + } + Ok::(reads) + }); + started_rx.await.expect("reader started"); + + assert_eq!(store.delete_many("ssh_session", &ids).await.unwrap(), 500); + stop.store(true, Ordering::Relaxed); + assert!(reader.await.unwrap().unwrap() > 0); + assert!( + store + .get("provider", "control-row") + .await + .unwrap() + .is_some() + ); +} + #[tokio::test] async fn sqlite_protobuf_round_trip() { let store = test_store().await; @@ -1709,6 +1921,7 @@ async fn cas_update_message_cas_succeeds() { }), spec: None, status: None, + ..Sandbox::default() }; store.put_message(&sandbox).await.unwrap(); @@ -1751,6 +1964,7 @@ async fn cas_update_message_cas_conflicts_on_concurrent_updates() { }), spec: None, status: None, + ..Sandbox::default() }; store.put_message(&sandbox).await.unwrap(); @@ -1821,6 +2035,7 @@ async fn cas_update_message_cas_rejects_workspace_change() { }), spec: None, status: None, + ..Sandbox::default() }; store.put_message(&sandbox).await.unwrap(); @@ -1863,6 +2078,7 @@ async fn cas_update_message_cas_rejects_name_change() { }), spec: None, status: None, + ..Sandbox::default() }; store.put_message(&sandbox).await.unwrap(); diff --git a/crates/openshell-server/src/policy_store.rs b/crates/openshell-server/src/policy_store.rs index bd044c871..8b1b6df33 100644 --- a/crates/openshell-server/src/policy_store.rs +++ b/crates/openshell-server/src/policy_store.rs @@ -4,10 +4,8 @@ use crate::persistence::{ DraftChunkRecord, PersistenceError, PersistenceResult, PolicyRecord, SetResourceVersion, Store, }; -use openshell_core::proto::{ - DraftChunkPayload, NetworkPolicyRule, PolicyRevisionPayload, Sandbox, - SandboxPolicy as ProtoSandboxPolicy, -}; +use crate::storage_proto::{DraftChunkPayload, PolicyRevisionPayload}; +use openshell_core::proto::{NetworkPolicyRule, Sandbox, SandboxPolicy as ProtoSandboxPolicy}; use prost::Message; use std::collections::HashMap; diff --git a/crates/openshell-server/src/provider_profile_sources.rs b/crates/openshell-server/src/provider_profile_sources.rs index cbe5ff892..cea7faf31 100644 --- a/crates/openshell-server/src/provider_profile_sources.rs +++ b/crates/openshell-server/src/provider_profile_sources.rs @@ -8,14 +8,15 @@ use std::sync::Arc; use async_trait::async_trait; use openshell_core::GatewayProviderProfileSourceConfig; -use openshell_core::proto::{ProviderProfile, StoredProviderProfile}; +use openshell_core::mcp::normalize_provider_profile_mcp_fields; +use openshell_core::proto::ProviderProfile; use openshell_gateway_interceptors::{ GatewayInterceptorProfileSource, GatewayInterceptorRuntime, ProviderProfileSourceSnapshot as InterceptorProfileSnapshot, }; use openshell_providers::{ ProfileValidationDiagnostic, ProviderTypeProfile, builtin_profiles, normalize_profile_id, - validate_profile_set, + normalize_provider_type, validate_profile_set, }; use prost::Message as _; use sha2::{Digest, Sha256}; @@ -23,6 +24,7 @@ use tonic::Status; use tracing::debug; use crate::persistence::{ObjectType, Store}; +use crate::storage_proto::StoredProviderProfile; const BUILTIN_SOURCE_ID: &str = "builtin"; const USER_SOURCE_ID: &str = "user"; @@ -136,7 +138,8 @@ impl ProviderProfileSource for UserProviderProfileSource { let resource_version = stored_profile_resource_version(&stored); hasher.update(resource_version.to_le_bytes()); if let Some(profile) = stored.profile { - let profile = profile_response_payload(profile, resource_version); + let mut profile = profile_response_payload(profile, resource_version); + normalize_provider_profile_mcp_fields(&mut profile); hasher.update(profile.encode_to_vec()); profiles.push(ScopedSnapshotProfile { scope: ProfileScope::Platform, @@ -156,7 +159,8 @@ impl ProviderProfileSource for UserProviderProfileSource { let resource_version = stored_profile_resource_version(&stored); hasher.update(resource_version.to_le_bytes()); if let Some(profile) = stored.profile { - let profile = profile_response_payload(profile, resource_version); + let mut profile = profile_response_payload(profile, resource_version); + normalize_provider_profile_mcp_fields(&mut profile); hasher.update(profile.encode_to_vec()); profiles.push(ScopedSnapshotProfile { scope: ProfileScope::Workspace, @@ -238,6 +242,7 @@ struct ScopedProfileEntry { struct EffectiveProfileEntry { effective: ScopedProfileEntry, platform_fallback: Option, + static_fallback: Option, } #[derive(Debug, Clone)] @@ -415,6 +420,9 @@ impl EffectiveProviderProfileCatalog { if let Some(fallback) = &entry.platform_fallback { result.push((fallback.scope, fallback.response.clone())); } + if let Some(fallback) = &entry.static_fallback { + result.push((fallback.scope, fallback.response.clone())); + } } result } @@ -427,6 +435,7 @@ impl EffectiveProviderProfileCatalog { .map(|entry| entry.effective.response.clone()) } + #[cfg(test)] pub(crate) fn get_type_profile(&self, id: &str) -> Option { let id = normalize_profile_id(id)?; self.profiles @@ -447,6 +456,22 @@ impl EffectiveProviderProfileCatalog { &self, id: &str, profile_workspace: &str, + ) -> Option<&ScopedProfileEntry> { + if let Some(entry) = self.exact_scoped_type_profile_for_scope(id, profile_workspace) { + return Some(entry); + } + + let alias = normalize_provider_type(id)?; + if normalize_profile_id(id).as_deref() == Some(alias) { + return None; + } + self.exact_scoped_type_profile_for_scope(alias, profile_workspace) + } + + fn exact_scoped_type_profile_for_scope( + &self, + id: &str, + profile_workspace: &str, ) -> Option<&ScopedProfileEntry> { let id = normalize_profile_id(id)?; let entry = self.profiles.get(&id)?; @@ -459,7 +484,7 @@ impl EffectiveProviderProfileCatalog { match &entry.platform_fallback { Some(fallback) => Some(fallback), None if entry.effective.scope == ProfileScope::Platform => Some(&entry.effective), - None => None, + None => entry.static_fallback.as_ref(), } } else { Some(&entry.effective) @@ -470,8 +495,14 @@ impl EffectiveProviderProfileCatalog { let id = normalize_profile_id(id)?; self.profiles .get(&id) - .filter(|entry| !entry.effective.user_managed) - .map(|entry| entry.effective.source_id.clone()) + .and_then(|entry| { + if entry.effective.user_managed { + entry.static_fallback.as_ref() + } else { + Some(&entry.effective) + } + }) + .map(|entry| entry.source_id.clone()) } pub(crate) fn hash_type_profile_revision_for_scope( @@ -537,12 +568,14 @@ fn build_effective_profiles( "duplicate provider profile source id '{source_id}'" ))); } - let source_revision = snapshot.revision.trim(); - if source_revision.is_empty() { + let source_revision = snapshot.revision; + if source_revision.trim().is_empty() { return Err(Status::failed_precondition(format!( "provider profile source '{source_id}' returned an empty revision" ))); } + // Revisions are opaque source-owned identities. Whitespace only is + // invalid, but a nonblank revision must otherwise remain byte-exact. if snapshot.profiles.is_empty() && !snapshot.allow_empty { return Err(Status::failed_precondition(format!( "provider profile source '{source_id}' returned no profiles" @@ -608,13 +641,20 @@ fn build_effective_profiles( let mut response = scoped_profile.profile; response.source = source_id.to_string(); response.scope = scope_to_string(scoped_profile.scope).to_string(); + let profile = ProviderTypeProfile::from_proto(&response); + + // Conversion and source normalization preserve unsupported and + // duplicate MCP revisions, so validation above cannot have + // malformed evidence repaired into an accepted profile. Normalize + // the valid response shape only after that check succeeds. + normalize_provider_profile_mcp_fields(&mut response); let new_entry = ScopedProfileEntry { source_id: source_id.to_string(), - source_revision: source_revision.to_string(), + source_revision: source_revision.clone(), user_managed: snapshot.user_managed, scope: scoped_profile.scope, - profile: ProviderTypeProfile::from_proto(&response), + profile, response, }; @@ -623,6 +663,19 @@ fn build_effective_profiles( let new_scope = new_entry.scope; match (existing_scope, new_scope) { + (ProfileScope::Static, _) + if existing.effective.source_id == BUILTIN_SOURCE_ID + && new_entry.user_managed => + { + let fallback = std::mem::replace(&mut existing.effective, new_entry); + existing.static_fallback = Some(fallback); + } + (_, ProfileScope::Static) + if new_entry.source_id == BUILTIN_SOURCE_ID + && existing.effective.user_managed => + { + existing.static_fallback = Some(new_entry); + } (ProfileScope::Static, _) | (_, ProfileScope::Static) => { let location = if existing.effective.source_id == source_id { format!("within source '{source_id}'") @@ -656,6 +709,7 @@ fn build_effective_profiles( EffectiveProfileEntry { effective: new_entry, platform_fallback: None, + static_fallback: None, }, ); } @@ -699,6 +753,9 @@ fn format_diagnostic(diagnostic: ProfileValidationDiagnostic) -> String { fn profile_snapshot_revision(profiles: &[ProviderProfile]) -> String { let mut profiles = profiles.to_vec(); + profiles + .iter_mut() + .for_each(normalize_provider_profile_mcp_fields); profiles.sort_by(|left, right| left.id.cmp(&right.id)); let mut hasher = Sha256::new(); hasher.update(b"openshell-provider-profile-snapshot-v1"); @@ -907,6 +964,191 @@ mod tests { profile.to_proto() } + fn profile_with_mcp_versions(id: &str, versions: &[&str]) -> ProviderProfile { + let mut profile = profile(id); + profile + .endpoints + .push(openshell_core::proto::NetworkEndpoint { + host: "mcp.example.com".to_string(), + port: 443, + protocol: "mcp".to_string(), + mcp: Some(openshell_core::proto::McpOptions { + versions: versions + .iter() + .map(|version| (*version).to_string()) + .collect(), + ..Default::default() + }), + rules: vec![openshell_core::proto::L7Rule { + allow: Some(openshell_core::proto::L7Allow { + method: "tools/list".to_string(), + ..Default::default() + }), + }], + ..Default::default() + }); + profile + } + + fn profile_without_mcp_options(id: &str) -> ProviderProfile { + let mut profile = profile(id); + profile + .endpoints + .push(openshell_core::proto::NetworkEndpoint { + host: "mcp.example.com".to_string(), + port: 443, + protocol: "mcp".to_string(), + mcp: None, + rules: vec![openshell_core::proto::L7Rule { + allow: Some(openshell_core::proto::L7Allow { + method: "tools/list".to_string(), + ..Default::default() + }), + }], + ..Default::default() + }); + profile + } + + fn mcp_versions(profile: &ProviderProfile) -> &[String] { + profile + .endpoints + .iter() + .find(|endpoint| endpoint.protocol == "mcp") + .and_then(|endpoint| endpoint.mcp.as_ref()) + .map(|mcp| mcp.versions.as_slice()) + .expect("canonical provider profile MCP options") + } + + #[test] + fn equivalent_mcp_version_order_produces_identical_source_profile_fingerprints() { + let catalog = |versions: &[&str]| { + build_effective_profiles(vec![CollectedProviderProfileSnapshot { + source_id: "external/test".to_string(), + revision: "same-revision".to_string(), + profiles: vec![ScopedSnapshotProfile { + scope: ProfileScope::Static, + profile: profile_with_mcp_versions("versioned-profile", versions), + }], + user_managed: false, + allow_empty: false, + }]) + .expect("valid source profile") + }; + let canonical = catalog(&["2025-03-26", "2025-11-25"]); + let reordered = catalog(&["2025-11-25", "2025-03-26"]); + + let mut canonical_hash = Sha256::new(); + canonical.hash_type_profile_revision_for_scope( + "versioned-profile", + "", + &mut canonical_hash, + ); + let mut reordered_hash = Sha256::new(); + reordered.hash_type_profile_revision_for_scope( + "versioned-profile", + "", + &mut reordered_hash, + ); + + assert_eq!(canonical_hash.finalize(), reordered_hash.finalize()); + assert_eq!( + canonical.get_profile("versioned-profile"), + reordered.get_profile("versioned-profile") + ); + } + + #[test] + fn defaulted_mcp_versions_produce_identical_source_profile_fingerprints() { + let catalog = |profile| { + build_effective_profiles(vec![CollectedProviderProfileSnapshot { + source_id: "external/test".to_string(), + revision: "same-revision".to_string(), + profiles: vec![ScopedSnapshotProfile { + scope: ProfileScope::Static, + profile, + }], + user_managed: false, + allow_empty: false, + }]) + .expect("valid source profile") + }; + let omitted_profile = profile_without_mcp_options("versioned-profile"); + let empty_profile = profile_with_mcp_versions("versioned-profile", &[]); + let explicit_profile = profile_with_mcp_versions("versioned-profile", &["2025-11-25"]); + assert_eq!( + profile_snapshot_revision(std::slice::from_ref(&omitted_profile)), + profile_snapshot_revision(std::slice::from_ref(&explicit_profile)) + ); + assert_eq!( + profile_snapshot_revision(std::slice::from_ref(&empty_profile)), + profile_snapshot_revision(std::slice::from_ref(&explicit_profile)) + ); + + let omitted = catalog(omitted_profile); + let empty = catalog(empty_profile); + let explicit = catalog(explicit_profile); + + let fingerprint = |catalog: &EffectiveProviderProfileCatalog| { + let mut hash = Sha256::new(); + catalog.hash_type_profile_revision_for_scope("versioned-profile", "", &mut hash); + hash.finalize() + }; + assert_eq!(fingerprint(&omitted), fingerprint(&explicit)); + assert_eq!(fingerprint(&empty), fingerprint(&explicit)); + + let explicit_profile = explicit + .get_profile("versioned-profile") + .expect("explicit provider profile"); + assert_eq!(mcp_versions(&explicit_profile), &["2025-11-25".to_string()]); + assert_eq!( + omitted.get_profile("versioned-profile"), + Some(explicit_profile.clone()) + ); + assert_eq!( + empty.get_profile("versioned-profile"), + Some(explicit_profile) + ); + } + + #[test] + fn mcp_profile_normalization_preserves_malformed_explicit_evidence() { + let mut malformed = profile_with_mcp_versions( + "malformed-version-profile", + &["latest", "2025-11-25", "2025-11-25"], + ); + malformed.description = "unrelated source-owned field".to_string(); + let original = malformed.clone(); + + normalize_provider_profile_mcp_fields(&mut malformed); + + assert_eq!(malformed, original); + assert_ne!( + profile_snapshot_revision(std::slice::from_ref(&malformed)), + profile_snapshot_revision(&[profile_with_mcp_versions( + "malformed-version-profile", + &["2025-11-25"], + )]) + ); + let error = build_effective_profiles(vec![CollectedProviderProfileSnapshot { + source_id: "external/test".to_string(), + revision: "malformed".to_string(), + profiles: vec![ScopedSnapshotProfile { + scope: ProfileScope::Static, + profile: malformed, + }], + user_managed: false, + allow_empty: false, + }]) + .expect_err("malformed explicit revisions must remain invalid"); + assert!( + error + .message() + .contains("duplicate MCP protocol version '2025-11-25'"), + "validation must reject the preserved duplicate before the later unsupported alias: {error}" + ); + } + #[tokio::test] async fn captured_catalog_is_immutable_and_each_source_is_fetched_once() { let mut revision_a = profile("moving-profile"); @@ -977,6 +1219,49 @@ mod tests { assert!(err.message().contains("returned an empty revision")); } + #[test] + fn nonblank_source_revisions_remain_opaque_and_whitespace_sensitive() { + let catalog = |revision: &str| { + build_effective_profiles(vec![CollectedProviderProfileSnapshot { + source_id: "source-a".to_string(), + revision: revision.to_string(), + profiles: vec![scoped(ProfileScope::Static, "github")], + user_managed: false, + allow_empty: false, + }]) + .expect("nonblank source revision") + }; + let plain = catalog("opaque"); + let padded = catalog(" opaque "); + + assert_ne!(plain.revision(), padded.revision()); + assert_eq!( + plain + .profiles + .get("github") + .expect("plain profile") + .effective + .source_revision, + "opaque" + ); + assert_eq!( + padded + .profiles + .get("github") + .expect("padded profile") + .effective + .source_revision, + " opaque " + ); + + let profile_hash = |catalog: &EffectiveProviderProfileCatalog| { + let mut hasher = Sha256::new(); + catalog.hash_type_profile_revision_for_scope("github", "", &mut hasher); + hasher.finalize() + }; + assert_ne!(profile_hash(&plain), profile_hash(&padded)); + } + #[test] fn duplicate_profile_ids_across_sources_are_invalid() { let err = build_effective_profiles(vec![ @@ -1505,28 +1790,46 @@ mod tests { } #[test] - fn same_id_static_vs_user_still_errors() { - let err = build_effective_profiles(vec![ + fn user_managed_profiles_shadow_new_builtins_in_scope() { + let catalog = build_effective_profiles(vec![ CollectedProviderProfileSnapshot { source_id: "builtin".to_string(), revision: "v1".to_string(), - profiles: vec![scoped(ProfileScope::Static, "openai")], + profiles: vec![ + scoped(ProfileScope::Static, "openai"), + scoped(ProfileScope::Static, "anthropic"), + ], user_managed: false, allow_empty: false, }, CollectedProviderProfileSnapshot { source_id: "user".to_string(), revision: "v1".to_string(), - profiles: vec![scoped(ProfileScope::Platform, "openai")], + profiles: vec![ + scoped(ProfileScope::Platform, "openai"), + scoped(ProfileScope::Workspace, "anthropic"), + ], user_managed: true, allow_empty: true, }, ]) - .unwrap_err(); + .unwrap(); - assert!( - err.message() - .contains("duplicate provider profile id 'openai'") + let openai = catalog + .get_type_profile_for_scope("openai", "default") + .expect("platform override"); + assert_eq!(openai.source, "user"); + let anthropic = catalog + .get_type_profile_for_scope("anthropic", "default") + .expect("workspace override"); + assert_eq!(anthropic.source, "user"); + let platform_anthropic = catalog + .get_type_profile_for_scope("anthropic", "") + .expect("built-in fallback outside workspace scope"); + assert_eq!(platform_anthropic.source, "builtin"); + assert_eq!( + catalog.static_source_for_profile("openai").as_deref(), + Some("builtin") ); } @@ -1792,4 +2095,70 @@ mod tests { assert!(result.is_some()); assert_eq!(result.unwrap().display_name, "Workspace Anthropic"); } + + #[test] + fn exact_alias_shaped_profile_wins_before_builtin_alias_fallback() { + let mut builtin = profile("github"); + builtin.display_name = "Built-in GitHub".to_string(); + let mut custom = profile("gh"); + custom.display_name = "Enterprise GitHub".to_string(); + + let catalog = build_effective_profiles(vec![ + CollectedProviderProfileSnapshot { + source_id: "builtin".to_string(), + revision: "builtin-v1".to_string(), + profiles: vec![ScopedSnapshotProfile { + scope: ProfileScope::Static, + profile: builtin, + }], + user_managed: false, + allow_empty: false, + }, + CollectedProviderProfileSnapshot { + source_id: "user".to_string(), + revision: "user-v1".to_string(), + profiles: vec![ScopedSnapshotProfile { + scope: ProfileScope::Workspace, + profile: custom, + }], + user_managed: true, + allow_empty: true, + }, + ]) + .unwrap(); + + let exact = catalog + .get_type_profile_for_scope("gh", "default") + .expect("exact custom profile"); + assert_eq!(exact.id, "gh"); + assert_eq!(exact.display_name, "Enterprise GitHub"); + + let builtin = catalog + .get_type_profile_for_scope("github", "default") + .expect("built-in profile"); + assert_eq!(builtin.id, "github"); + } + + #[test] + fn custom_profile_can_reuse_default_id_when_builtin_source_is_not_loaded() { + let mut custom = profile("github"); + custom.display_name = "Private GitHub".to_string(); + let catalog = build_effective_profiles(vec![CollectedProviderProfileSnapshot { + source_id: "user".to_string(), + revision: "user-v1".to_string(), + profiles: vec![ScopedSnapshotProfile { + scope: ProfileScope::Workspace, + profile: custom, + }], + user_managed: true, + allow_empty: true, + }]) + .unwrap(); + + let resolved = catalog + .get_type_profile_for_scope("github", "default") + .expect("custom profile reusing unloaded default ID"); + assert_eq!(resolved.id, "github"); + assert_eq!(resolved.display_name, "Private GitHub"); + } } diff --git a/crates/openshell-server/src/provider_refresh.rs b/crates/openshell-server/src/provider_refresh.rs index 176f86ae2..e8386abcb 100644 --- a/crates/openshell-server/src/provider_refresh.rs +++ b/crates/openshell-server/src/provider_refresh.rs @@ -11,7 +11,6 @@ use openshell_core::ObjectWorkspace; use openshell_core::proto::{ CredentialHandle, Provider, ProviderCredentialRefreshRecoveryAction, ProviderCredentialRefreshStatus, ProviderCredentialRefreshStrategy, - StoredProviderCredentialRefreshState, StoredRefreshMaterialDeletion, }; use openshell_core::{ObjectId, ObjectName, SetResourceVersion}; use prost::Message; @@ -21,6 +20,8 @@ use std::time::Duration; use tonic::{Code, Status}; use tracing::{info, warn}; +use crate::storage_proto::{StoredProviderCredentialRefreshState, StoredRefreshMaterialDeletion}; + const DEFAULT_REFRESH_BEFORE_SECONDS: i64 = 300; const DEFAULT_MAX_LIFETIME_SECONDS: i64 = 3600; const REFRESH_ERROR_RETRY_SECONDS: i64 = 60; @@ -835,23 +836,6 @@ pub async fn refresh_provider_credential( "provider credential refresh started" ); - // Enforce the providers_v2 gate on every mint, not just at configure time. - // Otherwise disabling providers_v2_enabled leaves already-configured refresh - // states that the worker and manual rotation keep minting from. - if let Err(err) = ensure_refresh_providers_v2_gate(store, &state).await { - let failure = RefreshFailure::from_status(&err); - persist_refresh_failure_state(store, &mut state, expected_version, &failure).await?; - warn!( - provider = %state.provider_name, - credential_key = %state.credential_key, - strategy = %refresh_strategy_name(state.strategy), - status = %state.status, - error = %err, - "provider credential refresh gate rejected" - ); - return Err(err); - } - let mint_result = match resolve_refresh_material(Some(credentials), &state).await { Ok(transient_state) => mint_credential(&transient_state).await, Err(err) => Err(err.into()), @@ -1276,31 +1260,6 @@ async fn cleanup_staged_refresh_handles( } } -/// Reject minting for strategies that require `providers_v2_enabled` when the -/// setting is off. Runs on every refresh (worker sweep and manual rotation), so -/// disabling the setting halts further mints from already-configured states. -async fn ensure_refresh_providers_v2_gate( - store: &Store, - state: &StoredProviderCredentialRefreshState, -) -> Result<(), Status> { - let strategy = ProviderCredentialRefreshStrategy::try_from(state.strategy) - .unwrap_or(ProviderCredentialRefreshStrategy::Unspecified); - if strategy != ProviderCredentialRefreshStrategy::AwsStsAssumeRole { - return Ok(()); - } - if !crate::grpc::policy::global_bool_setting_enabled( - store, - openshell_core::settings::PROVIDERS_V2_ENABLED_KEY, - ) - .await? - { - return Err(Status::failed_precondition( - "aws_sts_assume_role requires providers_v2_enabled=true", - )); - } - Ok(()) -} - async fn mint_credential( state: &StoredProviderCredentialRefreshState, ) -> Result { @@ -1579,6 +1538,7 @@ async fn request_token( } } + let _ = rustls::crypto::aws_lc_rs::default_provider().install_default(); let client = reqwest::Client::builder() .timeout(Duration::from_secs(30)) .build() @@ -2058,12 +2018,12 @@ mod tests { }; use crate::credentials::CredentialRuntime; use crate::persistence::{current_time_ms, test_store}; + use crate::storage_proto::StoredProviderCredentialRefreshState; use openshell_core::Config; use openshell_core::proto::datamodel::v1::ObjectMeta; use openshell_core::proto::{ CredentialHandle, Provider, ProviderCredentialRefreshRecoveryAction, ProviderCredentialRefreshStrategy, Sandbox, SandboxSpec, - StoredProviderCredentialRefreshState, }; use openshell_core::{ObjectId, ObjectName, ObjectWorkspace}; use std::collections::HashMap; @@ -2443,6 +2403,7 @@ mod tests { .mount(&mock_server) .await; + let _ = rustls::crypto::aws_lc_rs::default_provider().install_default(); let response = reqwest::get(format!("{}/oversized", mock_server.uri())) .await .unwrap(); @@ -2518,62 +2479,6 @@ mod tests { ); } - #[tokio::test] - async fn disabled_provider_gate_requires_configuration_and_retries_hourly() { - let store = test_store().await; - let provider = provider("disabled-provider-gate", "aws-s3"); - store.put_message(&provider).await.unwrap(); - let state = new_refresh_state( - &provider, - "default", - "AWS_ACCESS_KEY_ID", - NewRefreshStateConfig { - strategy: ProviderCredentialRefreshStrategy::AwsStsAssumeRole, - material: HashMap::from([( - "role_arn".to_string(), - "arn:aws:iam::123456789012:role/test".to_string(), - )]), - secret_material_keys: Vec::new(), - expires_at_ms: 0, - token_url: String::new(), - scopes: Vec::new(), - refresh_before_seconds: 30, - max_lifetime_seconds: 60, - additional_output_keys: HashMap::new(), - }, - ) - .unwrap(); - put_refresh_state(&store, &state).await.unwrap(); - - let err = refresh_provider_credential( - &store, - "default", - &test_credentials(), - None, - "disabled-provider-gate", - "AWS_ACCESS_KEY_ID", - ) - .await - .unwrap_err(); - - assert_eq!(err.code(), tonic::Code::FailedPrecondition); - let stored = - get_refresh_state(&store, "default", provider.object_id(), "AWS_ACCESS_KEY_ID") - .await - .unwrap() - .unwrap(); - assert_eq!(stored.status, "configuration_required"); - assert_eq!( - stored.recovery_action, - ProviderCredentialRefreshRecoveryAction::FixConfiguration as i32 - ); - assert_eq!(stored.failure_code, "refresh_configuration_invalid"); - assert_eq!( - stored.next_refresh_at_ms - stored.last_error_at_ms, - 60 * 60 * 1000 - ); - } - #[tokio::test] async fn oauth_invalid_grant_persists_terminal_reauthorization_status() { let mock_server = MockServer::start().await; @@ -3590,13 +3495,6 @@ mod tests { .await; let store = test_store().await; - crate::grpc::policy::set_global_bool_setting_for_test( - &store, - openshell_core::settings::PROVIDERS_V2_ENABLED_KEY, - true, - ) - .await - .unwrap(); let prov = provider("aws-sts-test", "aws"); store.put_message(&prov).await.unwrap(); @@ -3695,13 +3593,6 @@ mod tests { .await; let store = test_store().await; - crate::grpc::policy::set_global_bool_setting_for_test( - &store, - openshell_core::settings::PROVIDERS_V2_ENABLED_KEY, - true, - ) - .await - .unwrap(); let prov = provider("aws-sts-custom", "aws"); store.put_message(&prov).await.unwrap(); @@ -3779,13 +3670,6 @@ mod tests { #[tokio::test] async fn aws_sts_mint_rejects_partial_source_credentials() { let store = test_store().await; - crate::grpc::policy::set_global_bool_setting_for_test( - &store, - openshell_core::settings::PROVIDERS_V2_ENABLED_KEY, - true, - ) - .await - .unwrap(); let prov = provider("aws-sts-partial", "aws"); store.put_message(&prov).await.unwrap(); @@ -4110,13 +3994,6 @@ mod tests { .await; let store = test_store().await; - crate::grpc::policy::set_global_bool_setting_for_test( - &store, - openshell_core::settings::PROVIDERS_V2_ENABLED_KEY, - true, - ) - .await - .unwrap(); let prov = provider("aws-sts-session", "aws"); store.put_message(&prov).await.unwrap(); @@ -4194,13 +4071,6 @@ mod tests { #[tokio::test] async fn aws_sts_mint_rejects_session_token_without_source_pair() { let store = test_store().await; - crate::grpc::policy::set_global_bool_setting_for_test( - &store, - openshell_core::settings::PROVIDERS_V2_ENABLED_KEY, - true, - ) - .await - .unwrap(); let prov = provider("aws-sts-lonesession", "aws"); store.put_message(&prov).await.unwrap(); @@ -4279,13 +4149,6 @@ mod tests { .await; let store = test_store().await; - crate::grpc::policy::set_global_bool_setting_for_test( - &store, - openshell_core::settings::PROVIDERS_V2_ENABLED_KEY, - true, - ) - .await - .unwrap(); let prov = provider("aws-race", "aws"); store.put_message(&prov).await.unwrap(); let provider_id = prov.object_id().to_string(); @@ -4407,13 +4270,6 @@ mod tests { .await; let store = test_store().await; - crate::grpc::policy::set_global_bool_setting_for_test( - &store, - openshell_core::settings::PROVIDERS_V2_ENABLED_KEY, - true, - ) - .await - .unwrap(); let prov = provider("aws-superseded", "aws"); store.put_message(&prov).await.unwrap(); let provider_id = prov.object_id().to_string(); diff --git a/crates/openshell-server/src/sandbox_index.rs b/crates/openshell-server/src/sandbox_index.rs index 589f88fd8..c119ca688 100644 --- a/crates/openshell-server/src/sandbox_index.rs +++ b/crates/openshell-server/src/sandbox_index.rs @@ -1,7 +1,7 @@ // SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 -//! In-memory indexes for correlating Kubernetes objects back to sandbox ids. +//! In-memory indexes for correlating compute resources back to sandbox ids. use std::collections::HashMap; use std::sync::{Arc, RwLock}; diff --git a/crates/openshell-server/src/ssh_sessions.rs b/crates/openshell-server/src/ssh_sessions.rs index 490f0dbb2..12081c217 100644 --- a/crates/openshell-server/src/ssh_sessions.rs +++ b/crates/openshell-server/src/ssh_sessions.rs @@ -7,11 +7,14 @@ use openshell_core::ObjectId; use openshell_core::proto::SshSession; use openshell_core::time::now_ms; use prost::Message; +use std::future::Future; use std::sync::Arc; use std::time::Duration; use tracing::{info, warn}; -use crate::persistence::{ObjectType, Store}; +use crate::persistence::{ObjectCursor, ObjectType, Store}; + +const SESSION_REAPER_PAGE_SIZE: u32 = 1000; impl ObjectType for SshSession { fn object_type() -> &'static str { @@ -34,37 +37,69 @@ pub fn spawn_session_reaper(store: Arc, interval: Duration) { } async fn reap_expired_sessions(store: &Store) -> Result<(), String> { + reap_expired_sessions_after_page(store, |_| std::future::ready(())).await +} + +async fn reap_expired_sessions_after_page( + store: &Store, + mut after_page: F, +) -> Result<(), String> +where + F: FnMut(usize) -> Fut, + Fut: Future, +{ let now_ms = now_ms(); + let started = std::time::Instant::now(); + let mut cursor = None; + let mut page_number = 0_usize; + let mut scanned = 0_usize; + let mut decode_failures = 0_usize; + let mut session_ids = Vec::new(); - let records = store - .list_by_type(SshSession::object_type(), 1000, 0) - .await - .map_err(|e| e.to_string())?; + loop { + let records = store + .list_by_type_after( + SshSession::object_type(), + cursor.as_ref(), + SESSION_REAPER_PAGE_SIZE, + ) + .await + .map_err(|e| e.to_string())?; + let page_len = records.len(); + scanned += page_len; - let mut reaped = 0u32; - for record in records { - let session: SshSession = match Message::decode(record.payload.as_slice()) { - Ok(s) => s, - Err(_) => continue, - }; - - let should_delete = - (session.expires_at_ms > 0 && now_ms > session.expires_at_ms) || session.revoked; - - if should_delete { - if let Err(e) = store - .delete(SshSession::object_type(), session.object_id()) - .await - { - warn!(session_id = %session.object_id(), error = %e, "Failed to reap SSH session"); - } else { - reaped += 1; + cursor = records.last().map(ObjectCursor::from); + for record in records { + let Ok(session) = SshSession::decode(record.payload.as_slice()) else { + decode_failures += 1; + continue; + }; + if (session.expires_at_ms > 0 && now_ms > session.expires_at_ms) || session.revoked { + session_ids.push(session.object_id().to_string()); } } + + if page_len < SESSION_REAPER_PAGE_SIZE as usize { + break; + } + page_number += 1; + after_page(page_number).await; } - if reaped > 0 { - info!(count = reaped, "SSH session reaper: cleaned up sessions"); + let matched = session_ids.len(); + let deleted = store + .delete_many(SshSession::object_type(), &session_ids) + .await + .map_err(|e| e.to_string())?; + if matched > 0 || decode_failures > 0 { + info!( + scanned, + matched, + deleted, + decode_failures, + elapsed_ms = started.elapsed().as_millis(), + "SSH session reaper sweep complete" + ); } Ok(()) } @@ -175,4 +210,71 @@ mod tests { "session with no expiry should be preserved" ); } + + #[tokio::test] + async fn reaper_batches_expired_and_revoked_sessions() { + let store = test_store().await; + let session_count = crate::persistence::DELETE_MANY_BATCH_SIZE + 9; + for idx in 0..session_count { + let session = make_session( + &format!("reap-{idx}"), + "sbx1", + if idx % 2 == 0 { now_ms() - 1 } else { 0 }, + idx % 2 != 0, + ); + store.put_message(&session).await.unwrap(); + } + let active = make_session("keep", "sbx1", now_ms() + 60_000, false); + store.put_message(&active).await.unwrap(); + + reap_expired_sessions(&store).await.unwrap(); + + assert_eq!( + store + .count_in_workspace(SshSession::object_type(), "default") + .await + .unwrap(), + 1 + ); + assert!( + store + .get_message::("keep") + .await + .unwrap() + .is_some() + ); + } + + #[tokio::test] + async fn reaper_removes_every_expired_session_when_an_earlier_page_row_is_deleted() { + let store = test_store().await; + for idx in 0..=SESSION_REAPER_PAGE_SIZE { + let session = make_session(&format!("reap-{idx:04}"), "sbx1", now_ms() - 1, false); + store.put_message(&session).await.unwrap(); + } + + let delete_store = store.clone(); + reap_expired_sessions_after_page(&store, move |page_number| { + let delete_store = delete_store.clone(); + async move { + if page_number == 1 { + delete_store + .delete(SshSession::object_type(), "reap-0000") + .await + .unwrap(); + } + } + }) + .await + .unwrap(); + + assert_eq!( + store + .count_in_workspace(SshSession::object_type(), "default") + .await + .unwrap(), + 0, + "the reaper must not leave an expired session behind" + ); + } } diff --git a/crates/openshell-server/src/storage_proto.rs b/crates/openshell-server/src/storage_proto.rs new file mode 100644 index 000000000..cd5e065de --- /dev/null +++ b/crates/openshell-server/src/storage_proto.rs @@ -0,0 +1,721 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! Gateway-private, versioned protobuf formats for durable storage. + +#![allow( + clippy::all, + clippy::pedantic, + clippy::nursery, + dead_code, + unused_imports, + unused_qualifications, + rust_2018_idioms +)] + +include!(concat!(env!("OUT_DIR"), "/openshell.storage.v1.rs")); + +#[cfg(test)] +const STORAGE_FILE_DESCRIPTOR_SET: &[u8] = + include_bytes!(concat!(env!("OUT_DIR"), "/storage_descriptor.bin")); + +use openshell_core::{ + GetResourceVersion, ObjectId, ObjectLabels, ObjectName, ObjectWorkspace, SetResourceVersion, +}; +use std::collections::HashMap; + +impl ObjectId for StoredProviderProfile { + fn object_id(&self) -> &str { + self.metadata.as_ref().map_or("", |m| m.id.as_str()) + } +} + +impl ObjectName for StoredProviderProfile { + fn object_name(&self) -> &str { + self.metadata.as_ref().map_or("", |m| m.name.as_str()) + } +} + +impl ObjectLabels for StoredProviderProfile { + fn object_labels(&self) -> Option> { + self.metadata.as_ref().map(|m| m.labels.clone()) + } +} + +impl SetResourceVersion for StoredProviderProfile { + fn set_resource_version(&mut self, version: u64) { + if let Some(meta) = self.metadata.as_mut() { + meta.resource_version = version; + } + } +} + +impl GetResourceVersion for StoredProviderProfile { + fn get_resource_version(&self) -> u64 { + self.metadata.as_ref().map_or(0, |m| m.resource_version) + } +} + +impl ObjectWorkspace for StoredProviderProfile { + fn object_workspace(&self) -> &str { + self.metadata.as_ref().map_or("", |m| m.workspace.as_str()) + } + + fn requires_workspace() -> bool { + false + } +} + +impl ObjectId for StoredProviderCredentialRefreshState { + fn object_id(&self) -> &str { + self.metadata.as_ref().map_or("", |m| m.id.as_str()) + } +} + +impl ObjectName for StoredProviderCredentialRefreshState { + fn object_name(&self) -> &str { + self.metadata.as_ref().map_or("", |m| m.name.as_str()) + } +} + +impl ObjectLabels for StoredProviderCredentialRefreshState { + fn object_labels(&self) -> Option> { + self.metadata.as_ref().map(|m| m.labels.clone()) + } +} + +impl SetResourceVersion for StoredProviderCredentialRefreshState { + fn set_resource_version(&mut self, version: u64) { + if let Some(meta) = self.metadata.as_mut() { + meta.resource_version = version; + } + } +} + +impl GetResourceVersion for StoredProviderCredentialRefreshState { + fn get_resource_version(&self) -> u64 { + self.metadata.as_ref().map_or(0, |m| m.resource_version) + } +} + +impl ObjectWorkspace for StoredProviderCredentialRefreshState { + fn object_workspace(&self) -> &str { + self.metadata.as_ref().map_or("", |m| m.workspace.as_str()) + } + + fn requires_workspace() -> bool { + true + } +} + +#[cfg(test)] +mod tests { + use super::*; + use prost::Message; + use prost_types::{DescriptorProto, EnumDescriptorProto, FileDescriptorSet}; + use sha2::{Digest, Sha256}; + use std::collections::{BTreeMap, BTreeSet, VecDeque}; + + const STORAGE_V1_SCHEMA_SHA256: &str = + "79c72615d957fc0653c672f61998bf7d8d21b757bc05d07b3fff92bd70fc8f52"; + const PUBLIC_RPC_SCHEMA_SHA256: &str = + "c95ae90962c10fb28747db2b645adf4044562a3d208d84dfe4699d677e4364ee"; + const DURABLE_SCHEMA_SHA256: &str = + "920a5243dfb37ce709f0f562a47d17791a5ede90fd7f662ed01542abd60a0dfb"; + const PUBLIC_DURABLE_OVERLAP_SHA256: &str = + "05add438ba041defc98d791038ae593d3f09352677cae43f2276d494205ce415"; + // Synthetic payloads generated with the public declarations at v0.0.116, + // before their relocation into openshell.storage.v1. Values are deliberately + // non-secret and the ordinary protobuf bytes contain no package names. + const V0_0_116_REFRESH_STATE: &str = "0a230a096c65676163792d6964120b6c65676163792d6e616d6528073a0764656661756c74120b70726f76696465722d69641a0870726f76696465722205544f4b454e32160a09636c69656e745f6964120973796e7468657469633a0d726566726573685f746f6b656e406448c8015a06616374697665720773636f70652d619a011f0a0d726566726573685f746f6b656e120e0a047465737412066f7061717565a201150a036f6c64120e0a047465737412066f7061717565"; + const V0_0_116_DELETION: &str = "0a036f6c64120e0a047465737412066f7061717565"; + const V0_0_116_PROFILE: &str = "0a230a096c65676163792d6964120b6c65676163792d6e616d6528073a0764656661756c7412110a0770726f66696c6512064c6567616379"; + const V0_0_116_POLICY_PAYLOAD: &str = + "0a0012067368613235361a046e6f6e6520ac022a110a06736f75726365120766697874757265"; + const V0_0_116_DRAFT_PAYLOAD: &str = + "0a0472756c651a07666978747572652d0000403f3a0b6578616d706c652e636f6d40bb035002"; + const V0_0_116_POLICY_RECORD: &str = "0a09706f6c6963792d6964120a73616e64626f782d6964180222030102032a0673686132353632066c6f616465643a046e6f6e6540fa0148ac0252110a06736f75726365120766697874757265"; + const V0_0_116_DRAFT_RECORD: &str = "0a086368756e6b2d6964120a73616e64626f782d69641802220770656e64696e672a0472756c65320204053a076669787475726549000000000000e83f50de02589003620b6578616d706c652e636f6d68bb037801"; + const STORAGE_MESSAGE_NAMES: [&str; 7] = [ + "DraftChunkPayload", + "PolicyRevisionPayload", + "StoredDraftChunk", + "StoredPolicyRevision", + "StoredProviderCredentialRefreshState", + "StoredProviderProfile", + "StoredRefreshMaterialDeletion", + ]; + const DURABLE_ROOTS: [&str; 12] = [ + ".openshell.datamodel.v1.Provider", + ".openshell.datamodel.v1.Workspace", + ".openshell.sandbox.v1.SandboxPolicy", + ".openshell.storage.v1.DraftChunkPayload", + ".openshell.storage.v1.PolicyRevisionPayload", + ".openshell.storage.v1.StoredProviderCredentialRefreshState", + ".openshell.storage.v1.StoredProviderProfile", + ".openshell.v1.Sandbox", + ".openshell.v1.SandboxWorkloadTemplate", + ".openshell.v1.ServiceEndpoint", + ".openshell.v1.SshSession", + ".openshell.v1.WorkspaceMember", + ]; + + #[derive(Default)] + struct SchemaIndex<'a> { + messages: BTreeMap, + enums: BTreeMap, + } + + #[derive(Debug)] + struct SchemaClosure { + messages: BTreeSet, + enums: BTreeSet, + } + + fn qualified_name(prefix: &str, name: &str) -> String { + if prefix.is_empty() { + format!(".{name}") + } else { + format!("{prefix}.{name}") + } + } + + fn index_message<'a>(index: &mut SchemaIndex<'a>, prefix: &str, message: &'a DescriptorProto) { + let name = qualified_name(prefix, message.name.as_deref().expect("message name")); + index.messages.insert(name.clone(), message); + for nested in &message.nested_type { + index_message(index, &name, nested); + } + for nested_enum in &message.enum_type { + index.enums.insert( + qualified_name(&name, nested_enum.name.as_deref().expect("enum name")), + nested_enum, + ); + } + } + + fn index_descriptor<'a>(index: &mut SchemaIndex<'a>, descriptor: &'a FileDescriptorSet) { + for file in &descriptor.file { + let package = file.package.as_deref().unwrap_or_default(); + let prefix = if package.is_empty() { + String::new() + } else { + format!(".{package}") + }; + for message in &file.message_type { + index_message(index, &prefix, message); + } + for r#enum in &file.enum_type { + index.enums.insert( + qualified_name(&prefix, r#enum.name.as_deref().expect("enum name")), + r#enum, + ); + } + } + } + + fn schema_closure( + index: &SchemaIndex<'_>, + roots: impl IntoIterator, + ) -> SchemaClosure { + let mut messages = BTreeSet::new(); + let mut enums = BTreeSet::new(); + let mut pending = roots.into_iter().collect::>(); + while let Some(name) = pending.pop_front() { + if let Some(message) = index.messages.get(&name) { + if !messages.insert(name) { + continue; + } + for field in &message.field { + let Some(type_name) = field.type_name.as_ref() else { + continue; + }; + if index.messages.contains_key(type_name) { + pending.push_back(type_name.clone()); + } else if index.enums.contains_key(type_name) { + enums.insert(type_name.clone()); + } + } + } else if index.enums.contains_key(&name) { + enums.insert(name); + } else { + panic!("schema root or dependency {name} is missing from descriptors"); + } + } + SchemaClosure { messages, enums } + } + + fn append_one_message_schema(output: &mut String, full_name: &str, message: &DescriptorProto) { + output.push_str(&format!( + "message|{}|{}|{:?}|{:?}|{:?}\n", + full_name, + message + .options + .as_ref() + .map(|options| hex::encode(options.encode_to_vec())) + .unwrap_or_default(), + message.reserved_range, + message.reserved_name, + message.oneof_decl, + )); + for field in &message.field { + output.push_str(&format!( + "field|{}|{}|{}|{}|{}|{}|{}|{}|{}\n", + full_name, + field.number.unwrap_or_default(), + field.name.as_deref().unwrap_or_default(), + field.label.unwrap_or_default(), + field.r#type.unwrap_or_default(), + field.type_name.as_deref().unwrap_or_default(), + field.oneof_index.unwrap_or(-1), + field.proto3_optional.unwrap_or(false), + field + .options + .as_ref() + .map(|options| hex::encode(options.encode_to_vec())) + .unwrap_or_default(), + )); + } + } + + fn schema_fingerprint(index: &SchemaIndex<'_>, closure: &SchemaClosure) -> String { + let mut schema = String::new(); + for name in &closure.messages { + append_one_message_schema(&mut schema, name, index.messages[name]); + } + for name in &closure.enums { + let r#enum = index.enums[name]; + schema.push_str(&format!( + "enum|{}|{}|{:?}|{:?}\n", + name, + r#enum + .options + .as_ref() + .map(|options| hex::encode(options.encode_to_vec())) + .unwrap_or_default(), + r#enum.reserved_range, + r#enum.reserved_name, + )); + for value in &r#enum.value { + schema.push_str(&format!( + "enum-value|{}|{}|{}|{}\n", + name, + value.number.unwrap_or_default(), + value.name.as_deref().unwrap_or_default(), + value + .options + .as_ref() + .map(|options| hex::encode(options.encode_to_vec())) + .unwrap_or_default(), + )); + } + } + format!("{:x}", Sha256::digest(schema.as_bytes())) + } + + fn append_message_schema(output: &mut String, prefix: &str, message: &DescriptorProto) { + let name = message.name.as_deref().expect("message name"); + let full_name = if prefix.is_empty() { + name.to_string() + } else { + format!("{prefix}.{name}") + }; + output.push_str(&format!( + "message|{}|{}|{:?}|{:?}\n", + full_name, + message + .options + .as_ref() + .map(|options| hex::encode(options.encode_to_vec())) + .unwrap_or_default(), + message.reserved_range, + message.reserved_name, + )); + + for field in &message.field { + output.push_str(&format!( + "field|{}|{}|{}|{}|{}|{}|{}|{}|{}\n", + full_name, + field.number.unwrap_or_default(), + field.name.as_deref().unwrap_or_default(), + field.label.unwrap_or_default(), + field.r#type.unwrap_or_default(), + field.type_name.as_deref().unwrap_or_default(), + field.oneof_index.unwrap_or(-1), + field.proto3_optional.unwrap_or(false), + field + .options + .as_ref() + .map(|options| hex::encode(options.encode_to_vec())) + .unwrap_or_default(), + )); + } + for nested in &message.nested_type { + append_message_schema(output, &full_name, nested); + } + } + + fn storage_schema() -> (Vec, String) { + let descriptor = FileDescriptorSet::decode(STORAGE_FILE_DESCRIPTOR_SET) + .expect("storage descriptor set must decode"); + let file = descriptor + .file + .iter() + .find(|file| file.package.as_deref() == Some("openshell.storage.v1")) + .expect("storage descriptor must contain openshell.storage.v1"); + let mut names = file + .message_type + .iter() + .map(|message| message.name.clone().expect("message name")) + .collect::>(); + names.sort(); + + let mut messages = file.message_type.iter().collect::>(); + messages.sort_by_key(|message| message.name.as_deref().unwrap_or_default()); + let mut schema = String::new(); + for message in messages { + append_message_schema(&mut schema, "", message); + } + (names, schema) + } + + #[test] + fn storage_v1_schema_is_frozen() { + let (names, schema) = storage_schema(); + assert_eq!(names, STORAGE_MESSAGE_NAMES); + let actual = format!("{:x}", Sha256::digest(schema.as_bytes())); + assert_eq!( + actual, STORAGE_V1_SCHEMA_SHA256, + "openshell.storage.v1 changed; keep v1 decoders intact and introduce a versioned migration path before updating this reviewed fingerprint" + ); + } + + #[test] + fn storage_types_are_absent_from_public_descriptor() { + let public = FileDescriptorSet::decode(openshell_core::FILE_DESCRIPTOR_SET) + .expect("public descriptor set must decode"); + for file in public.file { + assert_ne!(file.package.as_deref(), Some("openshell.storage.v1")); + for message in file.message_type { + let name = message.name.expect("message name"); + assert!( + !STORAGE_MESSAGE_NAMES.contains(&name.as_str()), + "storage-only message {name} leaked into the public descriptor" + ); + } + } + } + + #[test] + fn public_and_durable_schema_inventories_are_complete() { + let public = FileDescriptorSet::decode(openshell_core::FILE_DESCRIPTOR_SET) + .expect("public descriptor set must decode"); + let storage = FileDescriptorSet::decode(STORAGE_FILE_DESCRIPTOR_SET) + .expect("storage descriptor set must decode"); + let mut index = SchemaIndex::default(); + index_descriptor(&mut index, &storage); + index_descriptor(&mut index, &public); + + let mut methods = Vec::new(); + let mut public_roots = BTreeSet::new(); + let mut compiled_method_count = 0; + for file in &public.file { + let package = file.package.as_deref().unwrap_or_default(); + for service in &file.service { + let service_name = service.name.as_deref().expect("service name"); + compiled_method_count += service.method.len(); + if !matches!( + (package, service_name), + ("openshell.v1", "OpenShell") | ("openshell.inference.v1", "Inference") + ) { + continue; + } + for method in &service.method { + let input = method.input_type.as_deref().expect("method input type"); + let output = method.output_type.as_deref().expect("method output type"); + public_roots.insert(input.to_string()); + public_roots.insert(output.to_string()); + methods.push(format!( + "{package}.{service_name}/{}|{}|{}|{}|{}", + method.name.as_deref().expect("method name"), + input, + output, + method.client_streaming.unwrap_or(false), + method.server_streaming.unwrap_or(false), + )); + } + } + } + methods.sort(); + assert_eq!(compiled_method_count, 100, "classify every compiled RPC"); + assert_eq!(methods.len(), 74, "inventory every public gateway RPC"); + assert_eq!( + methods + .iter() + .filter(|method| method.starts_with("openshell.v1.OpenShell/")) + .count(), + 74 + ); + assert!(methods.iter().all(|method| !method.contains(".storage."))); + + let public_closure = schema_closure(&index, public_roots); + let durable_closure = schema_closure(&index, DURABLE_ROOTS.into_iter().map(str::to_string)); + let overlap_messages = public_closure + .messages + .intersection(&durable_closure.messages) + .cloned() + .collect::>(); + let overlap_enums = public_closure + .enums + .intersection(&durable_closure.enums) + .cloned() + .collect::>(); + let overlap_inventory = overlap_messages + .iter() + .map(|name| format!("message|{name}")) + .chain(overlap_enums.iter().map(|name| format!("enum|{name}"))) + .collect::>() + .join("\n"); + + let public_schema_hash = schema_fingerprint(&index, &public_closure); + let public_inventory_hash = format!( + "{:x}", + Sha256::digest(format!("{}\n{public_schema_hash}", methods.join("\n")).as_bytes()) + ); + let durable_inventory_hash = schema_fingerprint(&index, &durable_closure); + let overlap_hash = format!("{:x}", Sha256::digest(overlap_inventory.as_bytes())); + + assert_eq!( + (public_closure.messages.len(), public_closure.enums.len()), + (278, 12) + ); + assert_eq!( + (durable_closure.messages.len(), durable_closure.enums.len()), + (81, 8) + ); + assert_eq!((overlap_messages.len(), overlap_enums.len()), (71, 8)); + + assert_eq!( + public_inventory_hash, PUBLIC_RPC_SCHEMA_SHA256, + "the public RPC schema closure changed; review API compatibility and update the inventory and architecture/gateway.md" + ); + assert_eq!( + durable_inventory_hash, DURABLE_SCHEMA_SHA256, + "a durable protobuf root or transitive dependency changed; record migration handling and a prior-version fixture before updating this fingerprint" + ); + assert_eq!( + overlap_hash, PUBLIC_DURABLE_OVERLAP_SHA256, + "the public/durable protobuf overlap changed; review both API and storage compatibility before updating this inventory" + ); + } + + fn legacy_bytes(encoded: &str) -> Vec { + hex::decode(encoded).expect("checked-in legacy fixture must be valid hex") + } + + #[test] + fn pre_move_storage_payloads_decode_after_package_relocation() { + let refresh = StoredProviderCredentialRefreshState::decode( + legacy_bytes(V0_0_116_REFRESH_STATE).as_slice(), + ) + .expect("legacy refresh state must decode"); + assert_eq!(refresh.provider_id, "provider-id"); + assert_eq!(refresh.provider_name, "provider"); + assert_eq!(refresh.credential_key, "TOKEN"); + assert_eq!(refresh.material["client_id"], "synthetic"); + assert_eq!(refresh.secret_material_keys, ["refresh_token"]); + assert_eq!(refresh.expires_at_ms, 100); + assert_eq!(refresh.next_refresh_at_ms, 200); + assert_eq!(refresh.status, "active"); + assert_eq!(refresh.scopes, ["scope-a"]); + assert_eq!( + refresh.secret_material_handles["refresh_token"].driver, + "test" + ); + assert_eq!(refresh.pending_secret_deletions[0].material_key, "old"); + + let deletion = + StoredRefreshMaterialDeletion::decode(legacy_bytes(V0_0_116_DELETION).as_slice()) + .expect("legacy deletion must decode"); + assert_eq!(deletion.material_key, "old"); + assert_eq!(deletion.handle.expect("handle").driver, "test"); + + let profile = StoredProviderProfile::decode(legacy_bytes(V0_0_116_PROFILE).as_slice()) + .expect("legacy provider profile must decode"); + let profile = profile.profile.expect("profile"); + assert_eq!(profile.id, "profile"); + assert_eq!(profile.display_name, "Legacy"); + + let policy_payload = + PolicyRevisionPayload::decode(legacy_bytes(V0_0_116_POLICY_PAYLOAD).as_slice()) + .expect("legacy policy payload must decode"); + assert!(policy_payload.policy.is_some()); + assert_eq!(policy_payload.hash, "sha256"); + assert_eq!(policy_payload.load_error, "none"); + assert_eq!(policy_payload.loaded_at_ms, 300); + assert_eq!(policy_payload.provenance["source"], "fixture"); + + let draft_payload = + DraftChunkPayload::decode(legacy_bytes(V0_0_116_DRAFT_PAYLOAD).as_slice()) + .expect("legacy draft payload must decode"); + assert_eq!(draft_payload.rule_name, "rule"); + assert_eq!(draft_payload.rationale, "fixture"); + assert_eq!(draft_payload.confidence, 0.75); + assert_eq!(draft_payload.host, "example.com"); + assert_eq!(draft_payload.port, 443); + assert_eq!(draft_payload.draft_version, 2); + + let policy = StoredPolicyRevision::decode(legacy_bytes(V0_0_116_POLICY_RECORD).as_slice()) + .expect("legacy policy record must decode"); + assert_eq!(policy.id, "policy-id"); + assert_eq!(policy.sandbox_id, "sandbox-id"); + assert_eq!(policy.version, 2); + assert_eq!(policy.policy_payload, [1, 2, 3]); + assert_eq!(policy.policy_hash, "sha256"); + assert_eq!(policy.status, "loaded"); + assert_eq!(policy.load_error.as_deref(), Some("none")); + assert_eq!(policy.created_at_ms, 250); + assert_eq!(policy.loaded_at_ms, Some(300)); + assert_eq!(policy.provenance["source"], "fixture"); + + let draft = StoredDraftChunk::decode(legacy_bytes(V0_0_116_DRAFT_RECORD).as_slice()) + .expect("legacy draft record must decode"); + assert_eq!(draft.id, "chunk-id"); + assert_eq!(draft.sandbox_id, "sandbox-id"); + assert_eq!(draft.draft_version, 2); + assert_eq!(draft.status, "pending"); + assert_eq!(draft.rule_name, "rule"); + assert_eq!(draft.proposed_rule, [4, 5]); + assert_eq!(draft.rationale, "fixture"); + assert_eq!(draft.confidence, 0.75); + assert_eq!(draft.created_at_ms, 350); + assert_eq!(draft.decided_at_ms, Some(400)); + assert_eq!(draft.host, "example.com"); + assert_eq!(draft.port, 443); + assert_eq!(draft.hit_count, 1); + } + + #[tokio::test] + async fn v0_0_116_database_payloads_survive_current_migrations() { + use crate::persistence::{DRAFT_CHUNK_OBJECT_TYPE, POLICY_OBJECT_TYPE, Store}; + use crate::policy_store::{draft_chunk_record_from_parts, policy_record_from_parts}; + + let tempdir = tempfile::tempdir().expect("temporary database directory"); + let database_path = tempdir.path().join("v0.0.116.db"); + let database_url = format!("sqlite://{}", database_path.display()); + + // v0.0.116 and this change share migrations 001-006. Populate that + // historical on-disk shape with bytes emitted by the v0.0.116 schema, + // then reopen it through the current migration and loader path. + let old_store = Store::connect(&database_url) + .await + .expect("create legacy database"); + let fixtures = [ + ( + "provider_credential_refresh_state", + "legacy-id", + "legacy-name", + "default", + V0_0_116_REFRESH_STATE, + ), + ( + "provider_profile", + "legacy-profile-id", + "legacy-profile", + "", + V0_0_116_PROFILE, + ), + ( + POLICY_OBJECT_TYPE, + "policy-id", + "", + "default", + V0_0_116_POLICY_PAYLOAD, + ), + ( + DRAFT_CHUNK_OBJECT_TYPE, + "chunk-id", + "", + "default", + V0_0_116_DRAFT_PAYLOAD, + ), + ]; + for (object_type, id, name, workspace, payload) in fixtures { + old_store + .put( + object_type, + id, + name, + workspace, + &legacy_bytes(payload), + None, + ) + .await + .expect("insert v0.0.116 fixture"); + } + old_store.close().await; + + let current_store = Store::connect(&database_url) + .await + .expect("current migrations must accept legacy database"); + for (object_type, id, _, _, payload) in fixtures { + let record = current_store + .get(object_type, id) + .await + .expect("load fixture row") + .expect("fixture row must remain present"); + assert_eq!(record.payload, legacy_bytes(payload)); + } + + let refresh = current_store + .get_message::("legacy-id") + .await + .expect("decode refresh fixture") + .expect("refresh fixture must remain present"); + assert_eq!(refresh.provider_id, "provider-id"); + assert_eq!(refresh.get_resource_version(), 1); + + let profile = current_store + .get_message::("legacy-profile-id") + .await + .expect("decode profile fixture") + .expect("profile fixture must remain present"); + assert_eq!(profile.profile.expect("profile").display_name, "Legacy"); + + let policy_record = current_store + .get(POLICY_OBJECT_TYPE, "policy-id") + .await + .expect("load policy fixture") + .expect("policy fixture must remain present"); + let policy = policy_record_from_parts( + policy_record.id, + "sandbox-id".to_string(), + 2, + "loaded".to_string(), + &policy_record.payload, + policy_record.created_at_ms, + ) + .expect("current policy loader must decode v0.0.116 payload"); + assert_eq!(policy.policy_hash, "sha256"); + assert_eq!(policy.provenance["source"], "fixture"); + + let draft_record = current_store + .get(DRAFT_CHUNK_OBJECT_TYPE, "chunk-id") + .await + .expect("load draft fixture") + .expect("draft fixture must remain present"); + let draft = draft_chunk_record_from_parts( + draft_record.id, + "sandbox-id".to_string(), + "pending".to_string(), + 1, + &draft_record.payload, + draft_record.created_at_ms, + draft_record.updated_at_ms, + ) + .expect("current draft loader must decode v0.0.116 payload"); + assert_eq!(draft.rule_name, "rule"); + assert_eq!(draft.host, "example.com"); + assert_eq!(draft.port, 443); + } +} diff --git a/crates/openshell-server/src/supervisor_session.rs b/crates/openshell-server/src/supervisor_session.rs index c8491dc1e..24413f0f1 100644 --- a/crates/openshell-server/src/supervisor_session.rs +++ b/crates/openshell-server/src/supervisor_session.rs @@ -1663,7 +1663,7 @@ mod tests { "sbx-test", relay_tx, Instant::now() - .checked_sub(Duration::from_secs(60)) + .checked_sub(Duration::from_mins(1)) .expect("test duration should be before now"), ), ); @@ -1741,7 +1741,7 @@ mod tests { "sbx-test", relay_tx, Instant::now() - .checked_sub(Duration::from_secs(60)) + .checked_sub(Duration::from_mins(1)) .expect("test duration should be before now"), ), ); diff --git a/crates/openshell-server/src/test_support.rs b/crates/openshell-server/src/test_support.rs index 5f4063ae4..3bd5430ef 100644 --- a/crates/openshell-server/src/test_support.rs +++ b/crates/openshell-server/src/test_support.rs @@ -96,6 +96,10 @@ impl FakeComputeDriver { default_image: "openshell/sandbox:test".to_string(), gateway_manages_lifecycle: false, supports_sandbox_authentication: false, + driver_reports_runtime_readiness: false, + resource_capabilities: None, + rootfs_tar_staging_dir: String::new(), + rootfs_tar_max_bytes: 0, }, gateway_listener_requirements: Vec::new(), gateway_listener_requirements_supported: true, @@ -284,11 +288,13 @@ impl ComputeDriver for FakeComputeDriver { request: Request, ) -> Result, Status> { self.record_traceparent(request.metadata()); - let sandbox = request.into_inner().sandbox; + let request = request.into_inner(); self.with_state(|state| { state .calls - .push(FakeComputeDriverCall::ValidateSandboxCreate { sandbox }); + .push(FakeComputeDriverCall::ValidateSandboxCreate { + sandbox: request.sandbox, + }); }); Ok(Response::new(ValidateSandboxCreateResponse {})) } @@ -337,7 +343,8 @@ impl ComputeDriver for FakeComputeDriver { request: Request, ) -> Result, Status> { self.record_traceparent(request.metadata()); - let sandbox = request.into_inner().sandbox; + let request = request.into_inner(); + let sandbox = request.sandbox; self.with_state(|state| { if let Some(sandbox) = sandbox.as_ref() { state.sandboxes.insert(sandbox.id.clone(), sandbox.clone()); diff --git a/crates/openshell-server/src/tls.rs b/crates/openshell-server/src/tls.rs index aa627746c..531daf787 100644 --- a/crates/openshell-server/src/tls.rs +++ b/crates/openshell-server/src/tls.rs @@ -18,7 +18,7 @@ use openshell_ocsf::{ ConfigStateChangeBuilder, OCSF_TARGET, SandboxContext, SeverityId, StateId, StatusId, }; use rustls::ServerConfig; -use rustls::crypto::ring::sign; +use rustls::crypto::aws_lc_rs::sign; use rustls::pki_types::{CertificateDer, PrivateKeyDer}; use rustls::server::{ClientHello, ResolvesServerCert, WebPkiClientVerifier}; use rustls::sign::CertifiedKey; @@ -488,9 +488,7 @@ fn tls_ocsf_ctx() -> SandboxContext { #[cfg(test)] mod tests { use super::*; - use crate::tls_test_utils::{ - generate_test_certs_with_ca, install_rustls_provider, write_test_file, - }; + use crate::tls_test_utils::{generate_test_certs_with_ca, write_test_file}; use rcgen::{CertificateParams, IsCa, KeyPair, KeyUsagePurpose}; use tokio::net::{TcpListener, TcpStream}; @@ -532,7 +530,6 @@ mod tests { #[test] fn test_build_server_config() { - install_rustls_provider(); let dir = tempfile::tempdir().expect("failed to create tempdir"); let _ = generate_test_certs_with_ca(dir.path()); @@ -553,7 +550,6 @@ mod tests { #[test] fn test_reload_success() { - install_rustls_provider(); let dir = tempfile::tempdir().expect("failed to create tempdir"); let (ca_cert, ca_key) = generate_test_certs_with_ca(dir.path()); @@ -574,7 +570,6 @@ mod tests { #[test] fn test_reload_invalid_preserves_old() { - install_rustls_provider(); let dir = tempfile::tempdir().expect("failed to create tempdir"); generate_test_certs_with_ca(dir.path()); @@ -607,8 +602,6 @@ mod tests { #[tokio::test(flavor = "multi_thread")] async fn test_concurrent_handshake_and_reload() { - install_rustls_provider(); - let dir = tempfile::tempdir().expect("failed to create tempdir"); let (ca_cert, ca_key) = generate_test_certs_with_ca(dir.path()); let acceptor = TlsAcceptor::from_files( @@ -702,8 +695,6 @@ mod tests { #[tokio::test] async fn test_reload_serves_new_cert() { - install_rustls_provider(); - let dir = tempfile::tempdir().expect("failed to create tempdir"); let (ca_cert, ca_key) = generate_test_certs_with_ca(dir.path()); let acceptor = TlsAcceptor::from_files( @@ -778,8 +769,6 @@ mod tests { #[tokio::test] async fn test_reload_worker_shutdown() { - install_rustls_provider(); - let dir = tempfile::tempdir().expect("failed to create tempdir"); generate_test_certs_with_ca(dir.path()); let acceptor = TlsAcceptor::from_files( @@ -812,8 +801,6 @@ mod tests { #[tokio::test] async fn test_reload_worker_detects_file_change() { - install_rustls_provider(); - let dir = tempfile::tempdir().expect("failed to create tempdir"); let (ca_cert, ca_key) = generate_test_certs_with_ca(dir.path()); let acceptor = TlsAcceptor::from_files( @@ -902,8 +889,6 @@ mod tests { #[tokio::test] async fn test_reload_mtls_ca_rotation() { - install_rustls_provider(); - let dir = tempfile::tempdir().expect("failed to create tempdir"); let (initial_ca_cert, initial_ca_key) = generate_test_certs_with_ca(dir.path()); @@ -1110,7 +1095,6 @@ mod tests { #[test] fn test_build_cert_resolver_returns_none_when_no_external() { - install_rustls_provider(); let dir = tempfile::tempdir().expect("failed to create tempdir"); generate_test_certs_with_ca(dir.path()); @@ -1127,7 +1111,6 @@ mod tests { #[test] fn test_build_cert_resolver_errors_on_cert_without_key() { - install_rustls_provider(); let dir = tempfile::tempdir().expect("failed to create tempdir"); generate_test_certs_with_ca(dir.path()); @@ -1147,7 +1130,6 @@ mod tests { #[test] fn test_build_cert_resolver_errors_on_key_without_cert() { - install_rustls_provider(); let dir = tempfile::tempdir().expect("failed to create tempdir"); generate_test_certs_with_ca(dir.path()); @@ -1167,7 +1149,6 @@ mod tests { #[test] fn test_build_cert_resolver_errors_on_empty_server_names() { - install_rustls_provider(); let dir = tempfile::tempdir().expect("failed to create tempdir"); let (ca_cert, ca_key) = generate_test_certs_with_ca(dir.path()); generate_named_cert( @@ -1195,7 +1176,6 @@ mod tests { #[test] fn test_dual_cert_resolver_returns_external_on_sni_match() { - install_rustls_provider(); let dir = tempfile::tempdir().expect("failed to create tempdir"); let (ca_cert, ca_key) = generate_test_certs_with_ca(dir.path()); generate_named_cert( @@ -1233,8 +1213,6 @@ mod tests { #[tokio::test(flavor = "multi_thread")] async fn test_dual_cert_resolver_sni_selects_correct_cert() { - install_rustls_provider(); - let dir = tempfile::tempdir().expect("failed to create tempdir"); let (ca_cert, ca_key) = generate_test_certs_with_ca(dir.path()); generate_named_cert( diff --git a/crates/openshell-server/src/tls_test_utils.rs b/crates/openshell-server/src/tls_test_utils.rs index aee83c49e..bcb71b0b5 100644 --- a/crates/openshell-server/src/tls_test_utils.rs +++ b/crates/openshell-server/src/tls_test_utils.rs @@ -9,14 +9,6 @@ use std::path::Path; use rcgen::{CertificateParams, IsCa, KeyPair}; -/// Install the default rustls crypto provider. -/// -/// Must be called once at the start of any test that exercises TLS handshakes. -/// Multiple calls are harmless (subsequent calls return an error, ignored). -pub fn install_rustls_provider() { - let _ = rustls::crypto::ring::default_provider().install_default(); -} - /// Write bytes to a file inside `dir`, panicking on failure. pub fn write_test_file(dir: &Path, name: &str, data: &[u8]) { let path = dir.join(name); diff --git a/crates/openshell-server/src/tracing_setup.rs b/crates/openshell-server/src/tracing_setup.rs index edcf30307..ed3f56c6e 100644 --- a/crates/openshell-server/src/tracing_setup.rs +++ b/crates/openshell-server/src/tracing_setup.rs @@ -11,7 +11,6 @@ use opentelemetry_sdk::trace::SdkTracerProvider; use tracing_subscriber::EnvFilter; use tracing_subscriber::prelude::*; -use crate::ConfiguredComputeDriver; use crate::config_file::OtlpConfig; use crate::otel_tracing::{GatewayResourceAttributes, SetupError}; use crate::tracing_bus::TracingLogBus; @@ -36,153 +35,45 @@ impl TracingHandle { } } -#[derive(Debug, Clone, Copy, PartialEq, Eq)] -enum InProcessDriverTracing { - #[cfg(all(not(target_os = "windows"), feature = "in-tree-compute-drivers"))] - Docker, - #[cfg(all(not(target_os = "windows"), feature = "in-tree-compute-drivers"))] - Kubernetes, - #[cfg(all(not(target_os = "windows"), feature = "in-tree-compute-drivers"))] - Podman, -} - -impl InProcessDriverTracing { - #[cfg(all(not(target_os = "windows"), feature = "in-tree-compute-drivers"))] - fn target_prefix(self) -> &'static str { - match self { - Self::Docker => openshell_driver_docker::otel_tracing::IN_PROCESS_TARGET_PREFIX, - Self::Kubernetes => openshell_driver_kubernetes::otel_tracing::IN_PROCESS_TARGET_PREFIX, - Self::Podman => openshell_driver_podman::otel_tracing::IN_PROCESS_TARGET_PREFIX, - } - } -} - -fn in_process_driver_tracing(driver: &ConfiguredComputeDriver) -> Option { - #[cfg(all(not(target_os = "windows"), feature = "in-tree-compute-drivers"))] - match driver { - ConfiguredComputeDriver::Registered(registration) if registration.name == "docker" => { - Some(InProcessDriverTracing::Docker) - } - ConfiguredComputeDriver::Registered(registration) if registration.name == "podman" => { - Some(InProcessDriverTracing::Podman) - } - ConfiguredComputeDriver::Registered(registration) if registration.name == "kubernetes" => { - Some(InProcessDriverTracing::Kubernetes) - } - _ => None, - } - #[cfg(not(all(not(target_os = "windows"), feature = "in-tree-compute-drivers")))] - { - let _ = driver; - None - } -} - -fn in_process_driver_target_prefix(driver: Option) -> Option<&'static str> { - #[cfg(all(not(target_os = "windows"), feature = "in-tree-compute-drivers"))] - { - driver.map(InProcessDriverTracing::target_prefix) - } - #[cfg(not(all(not(target_os = "windows"), feature = "in-tree-compute-drivers")))] - { - let _ = driver; - None - } -} - -#[cfg(all(not(target_os = "windows"), feature = "in-tree-compute-drivers"))] -fn in_process_driver_provider( - driver: Option, - endpoint: Option<&str>, - gateway_name: Option<&str>, -) -> (Option, Option) { - match driver { - Some(InProcessDriverTracing::Docker) => { - openshell_driver_docker::otel_tracing::provider_for(endpoint, gateway_name) - } - Some(InProcessDriverTracing::Kubernetes) => { - openshell_driver_kubernetes::otel_tracing::provider_for(endpoint, gateway_name) - } - Some(InProcessDriverTracing::Podman) => { - openshell_driver_podman::otel_tracing::provider_for(endpoint, gateway_name) - } - None => (None, None), - } -} - -#[cfg(not(all(not(target_os = "windows"), feature = "in-tree-compute-drivers")))] -fn in_process_driver_provider( - _driver: Option, - _endpoint: Option<&str>, - _gateway_name: Option<&str>, -) -> (Option, Option) { - (None, None) -} - -#[cfg(all(not(target_os = "windows"), feature = "in-tree-compute-drivers"))] -fn in_process_driver_layer( - provider: &Option, - driver: Option, -) -> Option> -where - S: tracing::Subscriber + for<'span> tracing_subscriber::registry::LookupSpan<'span>, -{ - provider.as_ref().map(|provider| match driver { - Some(InProcessDriverTracing::Docker) => { - openshell_driver_docker::otel_tracing::in_process_layer(provider) - } - Some(InProcessDriverTracing::Kubernetes) => { - openshell_driver_kubernetes::otel_tracing::in_process_layer(provider) - } - Some(InProcessDriverTracing::Podman) => { - openshell_driver_podman::otel_tracing::in_process_layer(provider) - } - None => unreachable!("a driver provider requires a selected driver"), - }) -} - -#[cfg(not(all(not(target_os = "windows"), feature = "in-tree-compute-drivers")))] -fn in_process_driver_layer( - _provider: &Option, - _driver: Option, -) -> Option> -where - S: tracing::Subscriber + for<'span> tracing_subscriber::registry::LookupSpan<'span>, -{ - None -} - pub fn install( env_filter: EnvFilter, tracing_log_bus: &TracingLogBus, otlp_config: Option<&OtlpConfig>, - driver: &ConfiguredComputeDriver, + driver: Option, gateway: GatewayResourceAttributes<'_>, ) -> (TracingHandle, Option) { let (tracer_provider, setup_error) = crate::otel_tracing::provider_for(otlp_config, gateway); - let selected_driver = in_process_driver_tracing(driver); - let driver_endpoint = selected_driver + let driver_endpoint = driver .is_some() .then_some(otlp_config) .flatten() .map(|config| config.endpoint.as_str()); - let (driver_tracer_provider, driver_setup_error) = - in_process_driver_provider(selected_driver, driver_endpoint, gateway.name()); + let (driver_tracer_provider, driver_setup_error) = driver.map_or_else( + || (None, None), + |descriptor| { + descriptor.provider_for( + driver_endpoint, + openshell_core::VERSION, + gateway.name(), + gateway.compute_driver(), + ) + }, + ); tracing_subscriber::registry() .with(env_filter) .with(tracing_subscriber::fmt::layer()) .with(tracing_log_bus.layer()) - .with(tracer_provider.as_ref().map(|provider| { - crate::otel_tracing::layer_excluding_driver( - provider, - in_process_driver_target_prefix(selected_driver), - ) + .with( + tracer_provider + .as_ref() + .map(|provider| crate::otel_tracing::layer(provider, driver)), + ) + .with(driver_tracer_provider.as_ref().map(|provider| { + driver + .expect("a driver provider requires a selected driver") + .in_process_layer(provider) })) - .with(in_process_driver_layer( - &driver_tracer_provider, - selected_driver, - )) .init(); ( @@ -193,40 +84,3 @@ pub fn install( setup_error.or(driver_setup_error), ) } - -#[cfg(test)] -mod tests { - use super::*; - - #[cfg(all(not(target_os = "windows"), feature = "in-tree-compute-drivers"))] - #[test] - fn in_process_driver_tracing_selects_registered_compute_drivers() { - let registry = crate::install_default_compute_drivers(); - let registered = |name| { - ConfiguredComputeDriver::Registered( - registry - .get(name) - .unwrap_or_else(|| panic!("{name} driver is registered")) - .clone(), - ) - }; - assert_eq!( - in_process_driver_tracing(®istered("podman")), - Some(InProcessDriverTracing::Podman) - ); - assert_eq!( - in_process_driver_tracing(®istered("docker")), - Some(InProcessDriverTracing::Docker) - ); - assert_eq!( - in_process_driver_tracing(®istered("kubernetes")), - Some(InProcessDriverTracing::Kubernetes) - ); - assert_eq!( - in_process_driver_tracing(&ConfiguredComputeDriver::Remote { - name: "custom".to_string(), - }), - None - ); - } -} diff --git a/crates/openshell-server/tests/common/mod.rs b/crates/openshell-server/tests/common/mod.rs index 9e3957abb..695d8d8f2 100644 --- a/crates/openshell-server/tests/common/mod.rs +++ b/crates/openshell-server/tests/common/mod.rs @@ -53,6 +53,13 @@ pub struct TestOpenShell; #[tonic::async_trait] impl OpenShell for TestOpenShell { + async fn begin_rootfs_tar_staging( + &self, + _request: tonic::Request, + ) -> Result, Status> { + Err(Status::unimplemented("not used by this test server")) + } + async fn report_main_process_exit( &self, _request: tonic::Request, @@ -98,6 +105,34 @@ impl OpenShell for TestOpenShell { Ok(Response::new(SandboxResponse::default())) } + async fn create_sandbox_template( + &self, + _request: tonic::Request, + ) -> Result, Status> { + Err(Status::unimplemented("unused")) + } + + async fn get_sandbox_template( + &self, + _request: tonic::Request, + ) -> Result, Status> { + Err(Status::unimplemented("unused")) + } + + async fn list_sandbox_templates( + &self, + _request: tonic::Request, + ) -> Result, Status> { + Err(Status::unimplemented("unused")) + } + + async fn delete_sandbox_template( + &self, + _request: tonic::Request, + ) -> Result, Status> { + Err(Status::unimplemented("unused")) + } + async fn stop_sandbox( &self, _request: tonic::Request, @@ -578,11 +613,6 @@ impl OpenShell for TestOpenShell { // TLS / PKI helpers (used by TLS integration tests) // --------------------------------------------------------------------------- -/// Initialise the rustls crypto provider (idempotent). -pub fn install_rustls_provider() { - let _ = rustls::crypto::ring::default_provider().install_default(); -} - /// PKI bundle: CA cert, server cert+key, client cert+key (all PEM). #[allow(clippy::struct_field_names)] pub struct PkiBundle { diff --git a/crates/openshell-server/tests/edge_tunnel_auth.rs b/crates/openshell-server/tests/edge_tunnel_auth.rs index be70e4567..c236d9995 100644 --- a/crates/openshell-server/tests/edge_tunnel_auth.rs +++ b/crates/openshell-server/tests/edge_tunnel_auth.rs @@ -29,7 +29,7 @@ mod common; use bytes::Bytes; use common::{ PkiBundle, build_tls_root, generate_pki, generate_rogue_pki, grpc_client_mtls, - install_rustls_provider, start_test_server, + start_test_server, }; use http_body_util::Empty; use hyper::{Request, StatusCode}; @@ -160,7 +160,6 @@ fn https_client_no_cert( /// Valid client cert is accepted when a CA is configured. #[tokio::test] async fn mtls_valid_client_cert_accepted() { - install_rustls_provider(); let (temp, pki) = generate_pki(); let tls_acceptor = TlsAcceptor::from_files( @@ -204,7 +203,6 @@ async fn mtls_valid_client_cert_accepted() { /// always optional. Auth is deferred to the application layer. #[tokio::test] async fn no_client_cert_accepted_with_ca_configured() { - install_rustls_provider(); let (temp, pki) = generate_pki(); let tls_acceptor = TlsAcceptor::from_files( @@ -250,7 +248,6 @@ async fn no_client_cert_accepted_with_ca_configured() { /// cert is presented. #[tokio::test] async fn bearer_header_reaches_server_without_client_cert() { - install_rustls_provider(); let (temp, pki) = generate_pki(); let tls_acceptor = TlsAcceptor::from_files( @@ -284,7 +281,6 @@ async fn bearer_header_reaches_server_without_client_cert() { /// client certs are optional — presented certs are still validated. #[tokio::test] async fn rogue_cert_rejected() { - install_rustls_provider(); let (temp, pki) = generate_pki(); let tls_acceptor = TlsAcceptor::from_files( @@ -333,7 +329,6 @@ async fn rogue_cert_rejected() { /// client certificates. Clients connect with server-only TLS. #[tokio::test] async fn https_only_no_client_cert_required() { - install_rustls_provider(); let (temp, pki) = generate_pki(); let tls_acceptor = TlsAcceptor::from_files( diff --git a/crates/openshell-server/tests/multiplex_tls_integration.rs b/crates/openshell-server/tests/multiplex_tls_integration.rs index 3447aad51..adc6837cc 100644 --- a/crates/openshell-server/tests/multiplex_tls_integration.rs +++ b/crates/openshell-server/tests/multiplex_tls_integration.rs @@ -6,7 +6,7 @@ mod common; use bytes::Bytes; use common::{ PkiBundle, build_tls_root, generate_pki, generate_rogue_pki, grpc_client_mtls, - install_rustls_provider, start_test_server, + start_test_server, }; use http_body_util::Empty; use hyper::Request; @@ -54,7 +54,6 @@ fn https_client_mtls( #[tokio::test] async fn serves_grpc_and_http_over_tls_on_same_port() { - install_rustls_provider(); let (temp, pki) = generate_pki(); let tls_acceptor = openshell_server::TlsAcceptor::from_files( @@ -96,7 +95,6 @@ async fn serves_grpc_and_http_over_tls_on_same_port() { #[tokio::test] async fn mtls_valid_client_cert_accepted() { - install_rustls_provider(); let (temp, pki) = generate_pki(); let tls_acceptor = openshell_server::TlsAcceptor::from_files( @@ -127,7 +125,6 @@ async fn mtls_valid_client_cert_accepted() { #[tokio::test] async fn no_client_cert_accepted_with_ca() { - install_rustls_provider(); let (temp, pki) = generate_pki(); let tls_acceptor = openshell_server::TlsAcceptor::from_files( @@ -166,7 +163,6 @@ async fn no_client_cert_accepted_with_ca() { #[tokio::test] async fn no_client_cert_rejected_when_required() { - install_rustls_provider(); let (temp, pki) = generate_pki(); let tls_acceptor = openshell_server::TlsAcceptor::from_files( @@ -206,7 +202,6 @@ async fn no_client_cert_rejected_when_required() { #[tokio::test] async fn mtls_wrong_ca_client_cert_rejected() { - install_rustls_provider(); let (temp, pki) = generate_pki(); let tls_acceptor = openshell_server::TlsAcceptor::from_files( diff --git a/crates/openshell-server/tests/supervisor_relay_integration.rs b/crates/openshell-server/tests/supervisor_relay_integration.rs index 6cfd3b009..448ae2cc7 100644 --- a/crates/openshell-server/tests/supervisor_relay_integration.rs +++ b/crates/openshell-server/tests/supervisor_relay_integration.rs @@ -48,6 +48,13 @@ struct RelayGateway { #[tonic::async_trait] impl OpenShell for RelayGateway { + async fn begin_rootfs_tar_staging( + &self, + _request: tonic::Request, + ) -> Result, Status> { + Err(Status::unimplemented("not used by this test server")) + } + async fn report_main_process_exit( &self, _request: tonic::Request, @@ -82,6 +89,34 @@ impl OpenShell for RelayGateway { // ------ unused stubs ------ + async fn create_sandbox_template( + &self, + _request: tonic::Request, + ) -> Result, Status> { + Err(Status::unimplemented("unused")) + } + + async fn get_sandbox_template( + &self, + _request: tonic::Request, + ) -> Result, Status> { + Err(Status::unimplemented("unused")) + } + + async fn list_sandbox_templates( + &self, + _request: tonic::Request, + ) -> Result, Status> { + Err(Status::unimplemented("unused")) + } + + async fn delete_sandbox_template( + &self, + _request: tonic::Request, + ) -> Result, Status> { + Err(Status::unimplemented("unused")) + } + type ConnectSupervisorStream = ReceiverStream>; async fn connect_supervisor( &self, diff --git a/crates/openshell-supervisor-middleware/src/headers.rs b/crates/openshell-supervisor-middleware/src/headers.rs index d4b1168f6..1dc37bff2 100644 --- a/crates/openshell-supervisor-middleware/src/headers.rs +++ b/crates/openshell-supervisor-middleware/src/headers.rs @@ -1,25 +1,37 @@ // SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 -//! Validation and logical application of middleware request-header mutations. +//! Validation and logical application of HTTP middleware header mutations. -use openshell_core::proto::{ExistingHeaderAction, HeaderMutation, HttpHeader, header_mutation}; +use openshell_core::{ + proto::{ExistingHeaderAction, HeaderMutation, HttpHeader, header_mutation}, + secrets::header_value_contains_reserved_credential_marker, +}; pub const MAX_HEADER_MUTATIONS: usize = 64; pub const MAX_HEADER_MUTATION_BYTES: usize = 32 * 1024; +/// Selects the protected-header rules for the HTTP message being mutated. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum HeaderAuthority { + Request, + Response, + ResponseTrailers, +} + #[derive(Debug, Clone, PartialEq, Eq)] pub enum HeaderMutationError { TooMany { count: usize }, InvalidName { name: String }, Protected { name: String }, HopByHop { name: String }, - WriteNamespace { name: String }, UnsafeValue { name: String }, + CredentialPlaceholder { name: String }, TooLarge, InvalidExistingAction, MissingExistingAction { name: String }, UnsupportedExistingAction, + AbsentTrailerName { name: String }, Empty, } @@ -31,12 +43,13 @@ impl HeaderMutationError { Self::InvalidName { .. } => "header_mutation_invalid_name", Self::Protected { .. } => "header_mutation_protected_header", Self::HopByHop { .. } => "header_mutation_hop_by_hop_header", - Self::WriteNamespace { .. } => "header_mutation_write_namespace", Self::UnsafeValue { .. } => "header_mutation_unsafe_value", + Self::CredentialPlaceholder { .. } => "header_mutation_credential_placeholder", Self::TooLarge => "header_mutation_bytes_over_capacity", Self::InvalidExistingAction => "header_mutation_invalid_existing_action", Self::MissingExistingAction { .. } => "header_mutation_missing_existing_action", Self::UnsupportedExistingAction => "header_mutation_unsupported_existing_action", + Self::AbsentTrailerName { .. } => "trailer_mutation_absent_name", Self::Empty => "header_mutation_empty", } } @@ -67,16 +80,16 @@ impl std::fmt::Display for HeaderMutationError { "middleware cannot mutate hop-by-hop header '{name}'" ) } - Self::WriteNamespace { name } => write!( - formatter, - "middleware can only write request headers prefixed with x-openshell-middleware- and cannot write '{name}'" - ), Self::UnsafeValue { name } => { write!( formatter, "middleware cannot write header '{name}' with an unsafe value" ) } + Self::CredentialPlaceholder { name } => write!( + formatter, + "middleware cannot write credential placeholder in header '{name}'" + ), Self::TooLarge => write!( formatter, "middleware header mutations exceed {MAX_HEADER_MUTATION_BYTES} bytes" @@ -94,6 +107,10 @@ impl std::fmt::Display for HeaderMutationError { "middleware returned unsupported on_existing action" ) } + Self::AbsentTrailerName { name } => write!( + formatter, + "middleware cannot create absent response trailer '{name}'" + ), Self::Empty => write!(formatter, "middleware returned an empty header mutation"), } } @@ -105,6 +122,7 @@ impl std::error::Error for HeaderMutationError {} /// state observed by the next middleware. Repeated values and wire order are /// preserved; comparisons are case-insensitive. pub fn apply( + authority: HeaderAuthority, existing_headers: &[HttpHeader], connection_nominated_headers: &[String], mutations: &[HeaderMutation], @@ -121,18 +139,28 @@ pub fn apply( match mutation.operation.as_ref() { Some(header_mutation::Operation::Write(write)) => { let name = validate_name(&write.name)?; + validate_authority(authority, MutationKind::Write, &write.name, &name)?; + if authority == HeaderAuthority::ResponseTrailers + && !existing_headers + .iter() + .any(|existing| existing.name.eq_ignore_ascii_case(&name)) + { + return Err(HeaderMutationError::AbsentTrailerName { + name: write.name.clone(), + }); + } if is_connection_nominated(connection_nominated_headers, &name) { return Err(HeaderMutationError::HopByHop { name: write.name.clone(), }); } - if !name.starts_with("x-openshell-middleware-") { - return Err(HeaderMutationError::WriteNamespace { + if !is_safe_value(&write.value) { + return Err(HeaderMutationError::UnsafeValue { name: write.name.clone(), }); } - if !is_safe_value(&write.value) { - return Err(HeaderMutationError::UnsafeValue { + if header_value_contains_reserved_credential_marker(&write.value) { + return Err(HeaderMutationError::CredentialPlaceholder { name: write.name.clone(), }); } @@ -166,6 +194,7 @@ pub fn apply( } Some(header_mutation::Operation::Remove(remove)) => { let name = validate_name(&remove.name)?; + validate_authority(authority, MutationKind::Remove, &remove.name, &name)?; if is_connection_nominated(connection_nominated_headers, &name) { return Err(HeaderMutationError::HopByHop { name: remove.name.clone(), @@ -195,12 +224,35 @@ fn validate_name(name: &str) -> Result { name: name.to_string(), }); } - if is_protected(&lower) { + Ok(lower) +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum MutationKind { + Write, + Remove, +} + +fn validate_authority( + authority: HeaderAuthority, + kind: MutationKind, + original_name: &str, + normalized_name: &str, +) -> Result<(), HeaderMutationError> { + let protected = match authority { + HeaderAuthority::Request => is_request_protected(normalized_name), + HeaderAuthority::Response => { + is_response_protected(normalized_name) + || (kind == MutationKind::Write && is_response_remove_only(normalized_name)) + } + HeaderAuthority::ResponseTrailers => is_response_protected(normalized_name), + }; + if protected { return Err(HeaderMutationError::Protected { - name: name.to_string(), + name: original_name.to_string(), }); } - Ok(lower) + Ok(()) } fn is_name_token_byte(byte: u8) -> bool { @@ -233,7 +285,7 @@ fn is_safe_value(value: &str) -> bool { .all(|byte| byte == b'\t' || (0x20..=0x7e).contains(&byte) || byte >= 0x80) } -fn is_protected(name: &str) -> bool { +fn is_request_protected(name: &str) -> bool { matches!( name, "authorization" @@ -253,6 +305,42 @@ fn is_protected(name: &str) -> bool { || name.starts_with("x-openshell-credential") } +fn is_response_protected(name: &str) -> bool { + matches!( + name, + "authentication-info" + | "connection" + | "content-encoding" + | "content-length" + | "content-range" + | "keep-alive" + | "proxy-authenticate" + | "proxy-authentication-info" + | "proxy-authorization" + | "proxy-connection" + | "set-cookie" + | "te" + | "trailer" + | "transfer-encoding" + | "upgrade" + | "www-authenticate" + ) || name.starts_with("x-openshell-credential") +} + +fn is_response_remove_only(name: &str) -> bool { + matches!( + name, + "accept-ranges" + | "etag" + | "content-md5" + | "digest" + | "content-digest" + | "repr-digest" + | "signature" + | "signature-input" + ) +} + fn is_connection_nominated(connection_nominated_headers: &[String], name: &str) -> bool { connection_nominated_headers .iter() @@ -292,6 +380,7 @@ mod tests { #[test] fn protected_header_write_is_rejected() { let error = apply( + HeaderAuthority::Request, &[], &[], &[write( @@ -311,6 +400,7 @@ mod tests { #[test] fn unsafe_header_value_is_rejected() { let error = apply( + HeaderAuthority::Request, &[], &[], &[write( @@ -323,6 +413,57 @@ mod tests { assert!(error.to_string().contains("unsafe value")); } + #[test] + fn credential_placeholder_header_values_are_rejected() { + for value in [ + "openshell:resolve:env:API_KEY", + "Bearer openshell:resolve:env:API_KEY", + "provider-OPENSHELL-RESOLVE-ENV-API_KEY", + "openshell%3Aresolve%3Aenv%3AAPI_KEY", + "Basic dXNlcjpvcGVuc2hlbGw6cmVzb2x2ZTplbnY6QVBJX0tFWQ==", + ] { + for authority in [HeaderAuthority::Request, HeaderAuthority::Response] { + let error = apply( + authority, + &[], + &[], + &[write("x-api-key", value, ExistingHeaderAction::Overwrite)], + ) + .expect_err("credential placeholder write"); + assert_eq!( + error, + HeaderMutationError::CredentialPlaceholder { + name: "x-api-key".to_string() + } + ); + } + } + } + + #[test] + fn existing_credential_placeholder_header_value_is_preserved() { + let existing = [header("x-api-key", "openshell:resolve:env:API_KEY")]; + let updated = apply( + HeaderAuthority::Request, + &existing, + &[], + &[write( + "cache-control", + "no-store", + ExistingHeaderAction::Overwrite, + )], + ) + .expect("ordinary mutation beside an existing placeholder"); + + assert_eq!( + updated, + vec![ + header("x-api-key", "openshell:resolve:env:API_KEY"), + header("cache-control", "no-store"), + ] + ); + } + #[test] fn existing_header_write_obeys_collision_action() { let existing = [ @@ -330,6 +471,7 @@ mod tests { header("accept", "application/json"), ]; let appended = apply( + HeaderAuthority::Request, &existing, &[], &[write( @@ -349,6 +491,7 @@ mod tests { ); let overwritten = apply( + HeaderAuthority::Request, &existing, &[], &[write( @@ -367,6 +510,7 @@ mod tests { ); let skipped = apply( + HeaderAuthority::Request, &existing, &[], &[write( @@ -386,13 +530,25 @@ mod tests { header("accept", "application/json"), header("x-trace", "two"), ]; - let updated = apply(&existing, &[], &[remove("X-Trace")]).expect("remove visible header"); + let updated = apply( + HeaderAuthority::Request, + &existing, + &[], + &[remove("X-Trace")], + ) + .expect("remove visible header"); assert_eq!(updated, vec![header("accept", "application/json")]); } #[test] fn protected_header_remove_is_rejected_even_when_not_visible() { - let error = apply(&[], &[], &[remove("Authorization")]).expect_err("protected removal"); + let error = apply( + HeaderAuthority::Request, + &[], + &[], + &[remove("Authorization")], + ) + .expect_err("protected removal"); assert!( error .to_string() @@ -404,6 +560,7 @@ mod tests { fn connection_nominated_header_is_protected() { let nominated = vec!["x-openshell-middleware-tag".to_string()]; let write_error = apply( + HeaderAuthority::Request, &[], &nominated, &[write( @@ -419,12 +576,206 @@ mod tests { .contains("hop-by-hop header 'X-OpenShell-Middleware-Tag'") ); - let remove_error = apply(&[], &nominated, &[remove("X-OpenShell-Middleware-Tag")]) - .expect_err("hop-by-hop removal"); + let remove_error = apply( + HeaderAuthority::Request, + &[], + &nominated, + &[remove("X-OpenShell-Middleware-Tag")], + ) + .expect_err("hop-by-hop removal"); assert!( remove_error .to_string() .contains("hop-by-hop header 'X-OpenShell-Middleware-Tag'") ); } + + #[test] + fn request_write_accepts_end_to_end_header_without_namespace() { + let updated = apply( + HeaderAuthority::Request, + &[], + &[], + &[write( + "Cache-Control", + "no-store", + ExistingHeaderAction::Overwrite, + )], + ) + .expect("ordinary end-to-end request header"); + + assert_eq!(updated, vec![header("cache-control", "no-store")]); + } + + #[test] + fn response_authority_allows_end_to_end_writes_and_integrity_removal() { + let existing = [header("etag", "old"), header("content-type", "text/plain")]; + let updated = apply( + HeaderAuthority::Response, + &existing, + &[], + &[ + write("Cache-Control", "private", ExistingHeaderAction::Overwrite), + remove("ETag"), + ], + ) + .expect("permitted response mutations"); + + assert_eq!( + updated, + vec![ + header("content-type", "text/plain"), + header("cache-control", "private"), + ] + ); + } + + #[test] + fn response_authority_rejects_framing_and_integrity_writes() { + for mutation in [ + remove("Content-Length"), + write("ETag", "new", ExistingHeaderAction::Overwrite), + ] { + let error = apply(HeaderAuthority::Response, &[], &[], &[mutation]) + .expect_err("protected response mutation"); + assert!(matches!(error, HeaderMutationError::Protected { .. })); + } + } + + #[test] + fn response_authority_protects_credential_headers_from_writes_and_removals() { + let existing = [header("set-cookie", "session=upstream")]; + for name in [ + "Set-Cookie", + "WWW-Authenticate", + "Authentication-Info", + "Proxy-Authentication-Info", + "X-OpenShell-Credential-Token", + ] { + for mutation in [ + write(name, "planted", ExistingHeaderAction::Overwrite), + remove(name), + ] { + let error = apply(HeaderAuthority::Response, &existing, &[], &[mutation]) + .expect_err("credential response header mutation"); + assert_eq!( + error, + HeaderMutationError::Protected { + name: name.to_string() + } + ); + } + } + } + + #[test] + fn empty_response_trailers_accept_pass_through() { + assert_eq!( + apply(HeaderAuthority::ResponseTrailers, &[], &[], &[]), + Ok(Vec::new()) + ); + } + + #[test] + fn response_trailer_pass_through_preserves_fields_and_order() { + let existing = [ + header("x-checksum", "one"), + header("x-trace", "middle"), + header("x-checksum", "two"), + ]; + + let updated = apply(HeaderAuthority::ResponseTrailers, &existing, &[], &[]) + .expect("empty mutation list"); + + assert_eq!(updated, existing); + } + + #[test] + fn response_trailer_mutations_modify_remove_and_preserve_order() { + let existing = [ + header("x-checksum", "one"), + header("x-remove", "gone"), + header("x-trace", "middle"), + header("x-checksum", "two"), + ]; + + let updated = apply( + HeaderAuthority::ResponseTrailers, + &existing, + &[], + &[ + write("X-Checksum", "replacement", ExistingHeaderAction::Overwrite), + remove("X-Remove"), + write("X-Trace", "last", ExistingHeaderAction::Append), + ], + ) + .expect("permitted response trailer mutations"); + + assert_eq!( + updated, + vec![ + header("x-trace", "middle"), + header("x-checksum", "replacement"), + header("x-trace", "last"), + ] + ); + } + + #[test] + fn response_trailer_write_cannot_introduce_an_absent_name() { + let existing = [header("x-checksum", "one")]; + let error = apply( + HeaderAuthority::ResponseTrailers, + &existing, + &[], + &[write( + "X-New-Trailer", + "value", + ExistingHeaderAction::Overwrite, + )], + ) + .expect_err("absent response trailer name"); + + assert_eq!( + error, + HeaderMutationError::AbsentTrailerName { + name: "X-New-Trailer".into() + } + ); + } + + #[test] + fn response_trailer_removal_of_an_absent_name_is_a_noop() { + let existing = [header("x-checksum", "one")]; + let updated = apply( + HeaderAuthority::ResponseTrailers, + &existing, + &[], + &[remove("X-Missing")], + ) + .expect("absent response trailer removal"); + + assert_eq!(updated, existing); + } + + #[test] + fn response_trailer_protected_fields_cannot_be_mutated() { + for mutation in [ + write("Content-Length", "10", ExistingHeaderAction::Overwrite), + remove("Set-Cookie"), + ] { + let error = apply( + HeaderAuthority::ResponseTrailers, + &[ + header("content-length", "5"), + header("set-cookie", "session=upstream"), + ], + &[], + &[mutation], + ) + .expect_err("protected response trailer mutation"); + + assert!(matches!(error, HeaderMutationError::Protected { .. })); + } + } } diff --git a/crates/openshell-supervisor-middleware/src/lib.rs b/crates/openshell-supervisor-middleware/src/lib.rs index 902b5165c..1065c18fa 100644 --- a/crates/openshell-supervisor-middleware/src/lib.rs +++ b/crates/openshell-supervisor-middleware/src/lib.rs @@ -3,7 +3,7 @@ //! Supervisor middleware registration and chain execution. -mod headers; +pub mod headers; mod remote; mod websocket; @@ -33,7 +33,8 @@ use tokio::sync::{OnceCell, OwnedSemaphorePermit, Semaphore}; use tonic::{Request, Response as TonicResponse, Status as TonicStatus}; pub use openshell_core::middleware::{ - HttpRequestView, InProcessMiddleware, SupervisorMiddlewareEndpoint, WebSocketResponseStream, + HttpRequestView, HttpResponseResultStream, InProcessMiddleware, SupervisorMiddlewareEndpoint, + WebSocketResponseStream, }; pub type MiddlewareService = dyn SupervisorMiddleware; @@ -180,6 +181,13 @@ impl InProcessMiddleware for EndpointInProcessAdapter { ) -> std::result::Result { self.endpoint.open_websocket_session(requests).await } + + async fn open_http_response_pre_return( + &self, + requests: tokio::sync::mpsc::Receiver, + ) -> std::result::Result { + self.endpoint.open_http_response_pre_return(requests).await + } } /// Adapt a transport-neutral endpoint to the in-process registry contract. @@ -835,6 +843,12 @@ fn supported_binding(source: &str, binding: &MiddlewareBinding) -> Result Ok(SupportedBinding::HttpPreCredentials), + ( + Some(SupervisorMiddlewareOperation::HttpResponse), + Some(SupervisorMiddlewarePhase::PreReturn), + ) => Err(miette!( + "{source} advertises HTTP_RESPONSE/PRE_RETURN, which is not yet supported" + )), ( Some(SupervisorMiddlewareOperation::WebsocketMessage), Some(SupervisorMiddlewarePhase::PreCredentials), @@ -843,7 +857,7 @@ fn supported_binding(source: &str, binding: &MiddlewareBinding) -> Result Err(miette!( - "{source} advertises WEBSOCKET_MESSAGE/PRE_RETURN, which is reserved for PR 2" + "{source} advertises WEBSOCKET_MESSAGE/PRE_RETURN, which is not yet supported" )), _ => Err(miette!( "{source} advertises an unsupported middleware operation/phase pair" @@ -1435,6 +1449,20 @@ impl ChainRunner { .entries) } + pub async fn describe_http_response_chain( + &self, + entries: &[ChainEntry], + ) -> Result> { + Ok(self + .describe_chain_for( + entries, + SupervisorMiddlewareOperation::HttpResponse, + SupervisorMiddlewarePhase::PreReturn, + ) + .await? + .entries) + } + async fn describe_chain_for( &self, entries: &[ChainEntry], @@ -1848,6 +1876,7 @@ impl ChainRunner { None } else { match headers::apply( + headers::HeaderAuthority::Request, &headers, &connection_nominated_headers, &result.header_mutations, @@ -3025,6 +3054,56 @@ mod tests { received: std::sync::Mutex>, } + struct InProcessHeaderChainService { + received: std::sync::Mutex>>, + } + + #[tonic::async_trait] + impl InProcessMiddleware for InProcessHeaderChainService { + async fn describe(&self) -> MiddlewareManifest { + MiddlewareManifest { + name: "test/in-process-header-chain".into(), + service_version: "test".into(), + bindings: vec![MiddlewareBinding { + operation: SupervisorMiddlewareOperation::HttpRequest as i32, + phase: SupervisorMiddlewarePhase::PreCredentials as i32, + max_payload_bytes: 4096, + timeout: String::new(), + }], + expected_audience: String::new(), + } + } + + async fn validate_config( + &self, + _middleware_name: &str, + _config: &prost_types::Struct, + ) -> Result<()> { + Ok(()) + } + + async fn evaluate_http_request( + &self, + request: HttpRequestView<'_>, + ) -> Result { + let invocation = { + let mut received = self.received.lock().expect("in-process header chain lock"); + let invocation = received.len(); + received.push(request.headers().to_vec()); + invocation + }; + let mut result = allow_result(); + if invocation == 0 { + result.header_mutations.push(write_header( + "cache-control", + "no-store", + ExistingHeaderAction::Overwrite, + )); + } + Ok(result) + } + } + #[tonic::async_trait] impl SupervisorMiddleware for HeaderChainService { type EvaluateWebSocketSessionStream = WebSocketResponseStream; @@ -3081,13 +3160,13 @@ mod tests { let mut result = allow_result(); if invocation == 0 { result.header_mutations.push(write_header( - "x-openshell-middleware-chain", + "cache-control", "first", ExistingHeaderAction::Overwrite, )); } else if invocation == 1 { result.header_mutations.push(write_header( - "x-openshell-middleware-chain", + "cache-control", "second", self.second_action, )); @@ -3141,13 +3220,56 @@ mod tests { let observed: Vec<&str> = received[2] .headers .iter() - .filter(|header| header.name == "x-openshell-middleware-chain") + .filter(|header| header.name == "cache-control") .map(|header| header.value.as_str()) .collect(); assert_eq!(observed, expected, "action {action:?}"); } } + #[tokio::test] + async fn in_process_request_middleware_writes_end_to_end_header_without_namespace() { + let service = Arc::new(InProcessHeaderChainService { + received: std::sync::Mutex::new(Vec::new()), + }); + let runner = ChainRunner::new(service.clone()); + let entries = [ + ChainEntry { + name: "writer".into(), + implementation: "test/in-process-header-chain".into(), + order: 0, + config: prost_types::Struct::default(), + on_error: OnError::FailClosed, + }, + ChainEntry { + name: "observer".into(), + implementation: "test/in-process-header-chain".into(), + order: 10, + config: prost_types::Struct::default(), + on_error: OnError::FailClosed, + }, + ]; + + let outcome = runner + .evaluate(&entries, input("payload")) + .await + .expect("evaluate in-process header chain"); + let received = service + .received + .lock() + .expect("recorded in-process headers"); + + assert!(outcome.allowed); + assert_eq!( + received[1] + .iter() + .filter(|header| header.name == "cache-control") + .map(|header| header.value.as_str()) + .collect::>(), + vec!["no-store"] + ); + } + #[tokio::test] async fn repeated_request_headers_reach_middleware_in_wire_order() { // A map contract would collapse repeated header names to one value @@ -3563,6 +3685,30 @@ mod tests { ); } + #[test] + fn manifest_rejects_http_response_pre_return_binding_until_dispatch_is_available() { + let registration = external_registration(4096); + let manifest = MiddlewareManifest { + name: "example/response".into(), + service_version: "test".into(), + bindings: vec![MiddlewareBinding { + operation: SupervisorMiddlewareOperation::HttpResponse as i32, + phase: SupervisorMiddlewarePhase::PreReturn as i32, + max_payload_bytes: 4096, + timeout: "500ms".into(), + }], + expected_audience: String::new(), + }; + + let error = validate_external_manifest(®istration, &manifest, 4096, false) + .expect_err("HTTP response pre-return binding must remain unavailable"); + assert!( + error + .to_string() + .contains("HTTP_RESPONSE/PRE_RETURN, which is not yet supported") + ); + } + #[test] fn manifest_accepts_forward_websocket_binding_and_reserves_return_phase() { let binding = |phase| MiddlewareBinding { @@ -3582,8 +3728,8 @@ mod tests { manifest.bindings = vec![binding(SupervisorMiddlewarePhase::PreReturn)]; let error = validate_manifest_bindings("test WebSocket service", &manifest, None) - .expect_err("return-path binding stays reserved for PR 2"); - assert!(error.to_string().contains("reserved for PR 2")); + .expect_err("return-path WebSocket binding is not yet supported"); + assert!(error.to_string().contains("not yet supported")); } #[test] @@ -4142,6 +4288,53 @@ mod tests { assert!(!format!("{outcome:?}").contains(secret)); } + #[tokio::test] + async fn credential_placeholder_header_mutation_follows_on_error() { + let placeholder = "openshell:resolve:env:API_KEY"; + let service = Arc::new(ScriptedService { + manifest_name: "test/middleware".into(), + max_body_bytes: 4096, + result: openshell_core::proto::HttpRequestResult { + header_mutations: vec![write_header( + "x-api-key", + placeholder, + ExistingHeaderAction::Overwrite, + )], + ..allow_result() + }, + }); + let registry = registry_with_external(service, external_registration(4096)).await; + let runner = ChainRunner::from_registry(registry); + + for (on_error, allowed) in [(OnError::FailClosed, false), (OnError::FailOpen, true)] { + let outcome = runner + .evaluate( + &[ChainEntry { + name: "guard".into(), + implementation: "local-guard-service".into(), + order: 0, + config: prost_types::Struct::default(), + on_error, + }], + input("hello"), + ) + .await + .expect("evaluate credential placeholder mutation"); + + assert_eq!(outcome.allowed, allowed); + assert!(outcome.header_mutations.is_empty()); + assert_eq!(outcome.applied.len(), 1); + assert!(outcome.applied[0].failed); + assert!(!format!("{outcome:?}").contains(placeholder)); + if !allowed { + assert_eq!( + outcome.reason, + "middleware_failed: header_mutation_credential_placeholder" + ); + } + } + } + #[tokio::test] async fn connection_nominated_write_and_remove_are_rejected_after_filtering() { let mutations = [ @@ -4572,7 +4765,7 @@ mod tests { close_on_first_message: bool, messages: Arc, session_ends: Option< - tokio::sync::mpsc::UnboundedSender, + tokio::sync::mpsc::UnboundedSender, >, } @@ -4663,7 +4856,7 @@ mod tests { Some(web_socket_session_event::Event::SessionEnd(end)) => { if let Some(session_ends) = &session_ends && let Ok(reason) = - openshell_core::proto::WebSocketSessionEndReason::try_from( + openshell_core::proto::MiddlewareSessionEndReason::try_from( end.reason, ) { @@ -4948,14 +5141,14 @@ mod tests { assert!(!text.invocations[0].failed); session - .end(openshell_core::proto::WebSocketSessionEndReason::NormalClose) + .end(openshell_core::proto::MiddlewareSessionEndReason::Normal) .await; } } #[tokio::test] async fn explicit_websocket_preflight_denial_is_authoritative_for_both_error_modes() { - use openshell_core::proto::WebSocketSessionEndReason; + use openshell_core::proto::MiddlewareSessionEndReason; for on_error in [OnError::FailOpen, OnError::FailClosed] { let (session_ends_tx, mut session_ends_rx) = tokio::sync::mpsc::unbounded_channel(); @@ -4991,7 +5184,7 @@ mod tests { assert!(!outcome.allowed); assert_eq!( outcome.terminal_reason, - Some(WebSocketSessionEndReason::MiddlewareDenial) + Some(MiddlewareSessionEndReason::MiddlewareDenial) ); assert_eq!( outcome.reason, @@ -5029,7 +5222,7 @@ mod tests { assert!(!outcome.invocations[0].failed); assert_eq!( session_ends_rx.recv().await, - Some(WebSocketSessionEndReason::MiddlewareDenial) + Some(MiddlewareSessionEndReason::MiddlewareDenial) ); assert!( session_ends_rx.try_recv().is_err(), @@ -5040,7 +5233,7 @@ mod tests { #[tokio::test] async fn mixed_websocket_preflight_denial_ends_every_opened_stage() { - use openshell_core::proto::WebSocketSessionEndReason; + use openshell_core::proto::MiddlewareSessionEndReason; let (first_end_tx, mut first_end_rx) = tokio::sync::mpsc::unbounded_channel(); let (denier_end_tx, mut denier_end_rx) = tokio::sync::mpsc::unbounded_channel(); @@ -5082,7 +5275,7 @@ mod tests { assert!(!outcome.allowed); assert_eq!( outcome.terminal_reason, - Some(WebSocketSessionEndReason::MiddlewareDenial) + Some(MiddlewareSessionEndReason::MiddlewareDenial) ); assert_eq!( outcome @@ -5099,7 +5292,7 @@ mod tests { for receiver in [&mut first_end_rx, &mut denier_end_rx, &mut last_end_rx] { assert_eq!( receiver.recv().await, - Some(WebSocketSessionEndReason::MiddlewareDenial) + Some(MiddlewareSessionEndReason::MiddlewareDenial) ); assert!( receiver.try_recv().is_err(), @@ -5176,7 +5369,7 @@ mod tests { "middleware_failed: request_message_over_capacity" ); session - .end(openshell_core::proto::WebSocketSessionEndReason::NormalClose) + .end(openshell_core::proto::MiddlewareSessionEndReason::Normal) .await; } @@ -5233,7 +5426,7 @@ mod tests { assert!(!redacted.invocations[0].stage_disabled); session - .end(openshell_core::proto::WebSocketSessionEndReason::NormalClose) + .end(openshell_core::proto::MiddlewareSessionEndReason::Normal) .await; } @@ -5279,7 +5472,7 @@ mod tests { ); assert!(outcome.invocations[0].transformed); session - .end(openshell_core::proto::WebSocketSessionEndReason::NormalClose) + .end(openshell_core::proto::MiddlewareSessionEndReason::Normal) .await; } @@ -5365,7 +5558,7 @@ mod tests { assert!(target.query.is_empty()); assert_eq!(observed.requested_subprotocols, ["realtime"]); session - .end(openshell_core::proto::WebSocketSessionEndReason::NormalClose) + .end(openshell_core::proto::MiddlewareSessionEndReason::Normal) .await; let _ = shutdown_tx.send(()); server_task @@ -5476,7 +5669,7 @@ mod tests { drop(work); session - .end(openshell_core::proto::WebSocketSessionEndReason::NormalClose) + .end(openshell_core::proto::MiddlewareSessionEndReason::Normal) .await; let _ = shutdown_tx.send(()); server_task @@ -5735,8 +5928,10 @@ mod tests { "all-skip preflight must not retain session capacity" ); assert_eq!( - session_ends_rx.recv().await, - Some(openshell_core::proto::WebSocketSessionEndReason::StageSkipped) + tokio::time::timeout(Duration::from_secs(1), session_ends_rx.recv()) + .await + .expect("skipped stage must receive session_end"), + Some(openshell_core::proto::MiddlewareSessionEndReason::StageSkipped) ); assert!( session_ends_rx.try_recv().is_err(), @@ -5782,7 +5977,7 @@ mod tests { sessions .pop() .expect("retained session") - .end(openshell_core::proto::WebSocketSessionEndReason::NormalClose) + .end(openshell_core::proto::MiddlewareSessionEndReason::Normal) .await; assert_eq!(runner.registry.session_admission.available_permits(), 1); @@ -5824,7 +6019,7 @@ mod tests { sessions .pop() .expect("retained old-generation session") - .end(openshell_core::proto::WebSocketSessionEndReason::PolicyReload) + .end(openshell_core::proto::MiddlewareSessionEndReason::PolicyReload) .await; let admitted = replacement .preflight_websocket(&chain, websocket_preflight_input("new-generation-admitted")) diff --git a/crates/openshell-supervisor-middleware/src/remote.rs b/crates/openshell-supervisor-middleware/src/remote.rs index edc1e8066..944303810 100644 --- a/crates/openshell-supervisor-middleware/src/remote.rs +++ b/crates/openshell-supervisor-middleware/src/remote.rs @@ -3,12 +3,14 @@ use miette::{IntoDiagnostic, Result, WrapErr}; use openshell_core::middleware::{ - HttpRequestView, SupervisorMiddlewareEndpoint, WebSocketResponseStream, + HttpRequestView, HttpResponseResultStream, SupervisorMiddlewareEndpoint, + WebSocketResponseStream, }; +use openshell_core::proto::middleware::v1::http_response_pre_return_client::HttpResponsePreReturnClient; use openshell_core::proto::middleware::v1::supervisor_middleware_client::SupervisorMiddlewareClient; use openshell_core::proto::{ - HttpRequestEvaluation, HttpRequestResult, MiddlewareManifest, ValidateConfigRequest, - ValidateConfigResponse, WebSocketSessionEvent, + HttpRequestEvaluation, HttpRequestResult, HttpResponseEvent, MiddlewareManifest, + ValidateConfigRequest, ValidateConfigResponse, WebSocketSessionEvent, }; use openshell_extension_core::{ BearerTokenInterceptor, BearerTokenSlot, ExtensionChannelConfig, ExtensionServerTrust, @@ -106,6 +108,7 @@ impl GrpcMiddlewareService { #[derive(Clone)] pub struct RemoteMiddlewareService { client: SupervisorMiddlewareClient, + response_client: HttpResponsePreReturnClient, } impl RemoteMiddlewareService { @@ -133,7 +136,10 @@ impl RemoteMiddlewareService { let channel = InterceptedService::new(channel, interceptor); Ok(Self { - client: SupervisorMiddlewareClient::new(channel) + client: SupervisorMiddlewareClient::new(channel.clone()) + .max_decoding_message_size(MIDDLEWARE_GRPC_MESSAGE_BYTES) + .max_encoding_message_size(MIDDLEWARE_GRPC_MESSAGE_BYTES), + response_client: HttpResponsePreReturnClient::new(channel) .max_decoding_message_size(MIDDLEWARE_GRPC_MESSAGE_BYTES) .max_encoding_message_size(MIDDLEWARE_GRPC_MESSAGE_BYTES), }) @@ -179,4 +185,18 @@ impl SupervisorMiddlewareEndpoint for RemoteMiddlewareService { .into_inner(); Ok(Box::pin(responses)) } + + async fn open_http_response_pre_return( + &self, + receiver: tokio::sync::mpsc::Receiver, + ) -> std::result::Result { + let mut client = self.response_client.clone(); + let responses = client + .evaluate(Request::new(tokio_stream::wrappers::ReceiverStream::new( + receiver, + ))) + .await? + .into_inner(); + Ok(Box::pin(responses)) + } } diff --git a/crates/openshell-supervisor-middleware/src/websocket.rs b/crates/openshell-supervisor-middleware/src/websocket.rs index 1fd95021a..900a821e5 100644 --- a/crates/openshell-supervisor-middleware/src/websocket.rs +++ b/crates/openshell-supervisor-middleware/src/websocket.rs @@ -12,11 +12,12 @@ use tokio::sync::mpsc; use tokio::time::Instant; use openshell_core::proto::{ - Decision, HttpRequestTarget, RequestContext, SupervisorMiddlewarePhase, WebSocketMessage, + Decision, HttpRequestTarget, MiddlewareSessionEnd, MiddlewareSessionEndReason, + MiddlewareSessionProtocolError, RequestContext, SupervisorMiddlewarePhase, WebSocketMessage, WebSocketMessageResult, WebSocketPreflight, WebSocketPreflightAction, - WebSocketPreflightDecision, WebSocketSessionEnd, WebSocketSessionEndReason, - WebSocketSessionEvent, WebSocketSessionStart, web_socket_message, web_socket_message_result, - web_socket_session_event, web_socket_session_event_result, + WebSocketPreflightDecision, WebSocketProtocolError, WebSocketSessionEvent, + WebSocketSessionStart, middleware_session_protocol_error, web_socket_message, + web_socket_message_result, web_socket_session_event, web_socket_session_event_result, }; use super::{ @@ -30,6 +31,7 @@ use super::{ }; const STREAM_CHANNEL_CAPACITY: usize = 4; +const SESSION_END_TIMEOUT: Duration = Duration::from_millis(10); const MAX_REQUESTED_SUBPROTOCOLS: usize = 32; const MAX_SUBPROTOCOL_BYTES: usize = 4 * 1024; const MAX_SELECTED_SUBPROTOCOL_BYTES: usize = 256; @@ -106,7 +108,7 @@ pub struct WebSocketPreflightResult { pub allowed: bool, /// Typed terminal reason when preflight denied the upgrade. `None` means /// the request may continue, including voluntary skip and fail-open. - pub terminal_reason: Option, + pub terminal_reason: Option, pub reason: String, pub denial: Option, pub session: Option, @@ -122,7 +124,7 @@ pub struct WebSocketPreflightResult { pub struct WebSocketSessionStartOutcome { pub allowed: bool, /// Typed terminal reason when session start cannot continue. - pub terminal_reason: Option, + pub terminal_reason: Option, pub reason: String, pub invocations: Vec, } @@ -145,6 +147,42 @@ struct WebSocketStageTransport { responses: super::WebSocketResponseStream, } +impl WebSocketStageTransport { + async fn end(self, reason: MiddlewareSessionEndReason) { + let _ = tokio::time::timeout(SESSION_END_TIMEOUT, self.end_inner(reason)).await; + } + + async fn end_inner(self, reason: MiddlewareSessionEndReason) { + if self.sender.send(session_end_request(reason)).await.is_err() { + return; + } + self.drain().await; + } + + async fn drain(self) { + let Self { + sender, + mut responses, + } = self; + // Keep the response handle alive while half-closing the request stream. + // Dropping both handles together schedules an HTTP/2 CANCEL, which may + // discard the buffered session_end before the middleware receives it. + drop(sender); + while responses.next().await.is_some() {} + } + + fn end_now(self, reason: MiddlewareSessionEndReason) { + if self.sender.try_send(session_end_request(reason)).is_err() { + return; + } + if let Ok(runtime) = tokio::runtime::Handle::try_current() { + drop(runtime.spawn(async move { + let _ = tokio::time::timeout(SESSION_END_TIMEOUT, self.drain()).await; + })); + } + } +} + struct WebSocketStage { entry: DescribedChainEntry, transport: Option, @@ -156,16 +194,13 @@ impl WebSocketStage { } async fn disable(&mut self) { - self.end(WebSocketSessionEndReason::MiddlewareFailure).await; + self.end(MiddlewareSessionEndReason::MiddlewareFailure) + .await; } - async fn end(&mut self, reason: WebSocketSessionEndReason) { + async fn end(&mut self, reason: MiddlewareSessionEndReason) { if let Some(transport) = self.transport.take() { - let _ = tokio::time::timeout( - Duration::from_millis(10), - transport.sender.send(session_end_request(reason)), - ) - .await; + transport.end(reason).await; } } } @@ -295,10 +330,10 @@ impl ChainRunner { } if let Some(denial) = denial { - end_stages(&mut stages, WebSocketSessionEndReason::MiddlewareDenial).await; + end_stages(&mut stages, MiddlewareSessionEndReason::MiddlewareDenial).await; return Ok(WebSocketPreflightResult { allowed: false, - terminal_reason: Some(WebSocketSessionEndReason::MiddlewareDenial), + terminal_reason: Some(MiddlewareSessionEndReason::MiddlewareDenial), reason: middleware_denial_reason( &denial.config_name, denial.reason_code.as_deref(), @@ -315,10 +350,10 @@ impl ChainRunner { } if let Some(reason) = fail_closed_reason { - end_stages(&mut stages, WebSocketSessionEndReason::MiddlewareFailure).await; + end_stages(&mut stages, MiddlewareSessionEndReason::MiddlewareFailure).await; return Ok(WebSocketPreflightResult { allowed: false, - terminal_reason: Some(WebSocketSessionEndReason::MiddlewareFailure), + terminal_reason: Some(MiddlewareSessionEndReason::MiddlewareFailure), reason, denial: None, session: None, @@ -401,7 +436,7 @@ fn session_capacity_exhausted( .collect(); WebSocketPreflightResult { allowed: !fail_closed, - terminal_reason: fail_closed.then_some(WebSocketSessionEndReason::MiddlewareFailure), + terminal_reason: fail_closed.then_some(MiddlewareSessionEndReason::MiddlewareFailure), reason: if fail_closed { format!("middleware_failed: {reason}") } else { @@ -444,7 +479,7 @@ impl WebSocketSession { if selected_subprotocol.len() > MAX_SELECTED_SUBPROTOCOL_BYTES { return WebSocketSessionStartOutcome { allowed: false, - terminal_reason: Some(WebSocketSessionEndReason::MiddlewareFailure), + terminal_reason: Some(MiddlewareSessionEndReason::MiddlewareFailure), reason: "middleware_failed: selected_subprotocol_over_capacity".to_string(), invocations: Vec::new(), }; @@ -480,7 +515,7 @@ impl WebSocketSession { allowed: fail_closed.is_none(), terminal_reason: fail_closed .as_ref() - .map(|_| WebSocketSessionEndReason::MiddlewareFailure), + .map(|_| MiddlewareSessionEndReason::MiddlewareFailure), reason: fail_closed.unwrap_or_default(), invocations, } @@ -804,7 +839,7 @@ impl WebSocketSession { } } - pub async fn end(mut self, reason: WebSocketSessionEndReason) { + pub async fn end(mut self, reason: MiddlewareSessionEndReason) { end_stages(&mut self.stages, reason).await; self.reconcile_lifecycle(); } @@ -812,7 +847,7 @@ impl WebSocketSession { impl Drop for WebSocketSession { fn drop(&mut self) { - end_stages_now(&mut self.stages, WebSocketSessionEndReason::Cancellation); + end_stages_now(&mut self.stages, MiddlewareSessionEndReason::Cancellation); self.reconcile_lifecycle(); } } @@ -965,25 +1000,25 @@ async fn open_stage(entry: DescribedChainEntry, input: WebSocketPreflightInput) Err(_) => return OpenStage::Failed(entry, "middleware_timeout".into()), }; let Some(response) = response else { - let _ = sender.try_send(session_end_request( - WebSocketSessionEndReason::MiddlewareFailure, - )); + WebSocketStageTransport { sender, responses } + .end(MiddlewareSessionEndReason::MiddlewareFailure) + .await; return OpenStage::Failed(entry, "missing_preflight_decision".into()); }; let Some(web_socket_session_event_result::Result::PreflightDecision(decision)) = response.result else { - let _ = sender.try_send(session_end_request( - WebSocketSessionEndReason::MiddlewareFailure, - )); + WebSocketStageTransport { sender, responses } + .end(MiddlewareSessionEndReason::MiddlewareFailure) + .await; return OpenStage::Failed(entry, "invalid_preflight_decision".into()); }; let decision = match validate_preflight_decision(decision) { Ok(decision) => decision, Err(reason) => { - let _ = sender.try_send(session_end_request( - WebSocketSessionEndReason::MiddlewareFailure, - )); + WebSocketStageTransport { sender, responses } + .end(MiddlewareSessionEndReason::MiddlewareFailure) + .await; return OpenStage::Failed(entry, reason.into()); } }; @@ -1015,7 +1050,9 @@ async fn open_stage(entry: DescribedChainEntry, input: WebSocketPreflightInput) WebSocketPreflightAction::Skip => { let outcome = preflight_stage_outcome(&entry, WebSocketInvocationOutcome::Skip, decision); - let _ = sender.try_send(session_end_request(WebSocketSessionEndReason::StageSkipped)); + WebSocketStageTransport { sender, responses } + .end(MiddlewareSessionEndReason::StageSkipped) + .await; OpenStage::Skip(outcome) } WebSocketPreflightAction::Unspecified => { @@ -1299,25 +1336,33 @@ fn failure_invocation( } } -async fn end_stages(stages: &mut [WebSocketStage], reason: WebSocketSessionEndReason) { +async fn end_stages(stages: &mut [WebSocketStage], reason: MiddlewareSessionEndReason) { for stage in stages { stage.end(reason).await; } } -fn end_stages_now(stages: &mut [WebSocketStage], reason: WebSocketSessionEndReason) { +fn end_stages_now(stages: &mut [WebSocketStage], reason: MiddlewareSessionEndReason) { for stage in stages { if let Some(transport) = stage.transport.take() { - let _ = transport.sender.try_send(session_end_request(reason)); + transport.end_now(reason); } } } -fn session_end_request(reason: WebSocketSessionEndReason) -> WebSocketSessionEvent { +fn session_end_request(reason: MiddlewareSessionEndReason) -> WebSocketSessionEvent { + let protocol_error = (reason == MiddlewareSessionEndReason::ProtocolError).then_some({ + MiddlewareSessionProtocolError { + domain: Some(middleware_session_protocol_error::Domain::WebSocket( + WebSocketProtocolError {}, + )), + } + }); WebSocketSessionEvent { event: Some(web_socket_session_event::Event::SessionEnd( - WebSocketSessionEnd { + MiddlewareSessionEnd { reason: reason as i32, + protocol_error, }, )), } @@ -1327,6 +1372,30 @@ fn session_end_request(reason: WebSocketSessionEndReason) -> WebSocketSessionEve mod tests { use super::*; + #[test] + fn session_end_refines_only_protocol_errors() { + let protocol_event = session_end_request(MiddlewareSessionEndReason::ProtocolError); + let Some(web_socket_session_event::Event::SessionEnd(protocol_end)) = protocol_event.event + else { + panic!("expected session end"); + }; + assert_eq!( + MiddlewareSessionEndReason::try_from(protocol_end.reason), + Ok(MiddlewareSessionEndReason::ProtocolError) + ); + assert!(matches!( + protocol_end.protocol_error.and_then(|detail| detail.domain), + Some(middleware_session_protocol_error::Domain::WebSocket(_)) + )); + + let normal_event = session_end_request(MiddlewareSessionEndReason::Normal); + let Some(web_socket_session_event::Event::SessionEnd(normal_end)) = normal_event.event + else { + panic!("expected session end"); + }; + assert!(normal_end.protocol_error.is_none()); + } + #[test] fn protobuf_rejects_invalid_utf8_text_payload() { let encoded_text_with_invalid_utf8 = [0x12, 0x01, 0xff]; @@ -1401,8 +1470,8 @@ mod tests { panic!("disabled stage must receive session end"); }; assert_eq!( - WebSocketSessionEndReason::try_from(end.reason), - Ok(WebSocketSessionEndReason::MiddlewareFailure) + MiddlewareSessionEndReason::try_from(end.reason), + Ok(MiddlewareSessionEndReason::MiddlewareFailure) ); assert!( requests.try_recv().is_err(), diff --git a/crates/openshell-supervisor-network/Cargo.toml b/crates/openshell-supervisor-network/Cargo.toml index 34d9c32a4..edc5658c5 100644 --- a/crates/openshell-supervisor-network/Cargo.toml +++ b/crates/openshell-supervisor-network/Cargo.toml @@ -3,7 +3,7 @@ [package] name = "openshell-supervisor-network" -description = "Network component of the OpenShell supervisor: proxy, L7 enforcement, OPA, inference routing, denial aggregator" +description = "Network component of the OpenShell supervisor: proxy, L7 enforcement, OPA, and denial aggregation" version.workspace = true edition.workspace = true license.workspace = true @@ -14,7 +14,6 @@ rust-version.workspace = true openshell-core = { path = "../openshell-core", features = ["oauth"] } openshell-ocsf = { path = "../openshell-ocsf" } openshell-policy = { path = "../openshell-policy" } -openshell-router = { path = "../openshell-router" } openshell-supervisor-middleware = { path = "../openshell-supervisor-middleware" } apollo-parser = { workspace = true } diff --git a/crates/openshell-supervisor-network/src/inference_routes.rs b/crates/openshell-supervisor-network/src/inference_routes.rs deleted file mode 100644 index 22b406b8d..000000000 --- a/crates/openshell-supervisor-network/src/inference_routes.rs +++ /dev/null @@ -1,751 +0,0 @@ -// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -// SPDX-License-Identifier: Apache-2.0 - -//! Inference route bundle resolution and refresh. -//! -//! Resolves inference routes from one of two sources at sandbox startup: -//! a local YAML file (`--inference-routes`) or a cluster bundle fetched via -//! gRPC. Builds the [`InferenceContext`] consumed by the proxy's L7 layer -//! and spawns a background refresh loop in cluster mode so route changes -//! propagate without restarting the sandbox. -//! -//! Distinct from [`crate::l7::inference`], which parses HTTP requests and -//! matches them against API patterns at request time. -//! -//! [`InferenceContext`]: crate::proxy::InferenceContext - -use std::sync::Arc; -use std::time::Duration; - -use miette::Result; -use tracing::{info, trace, warn}; - -use openshell_ocsf::{ - ConfigStateChangeBuilder, SeverityId, StateId, StatusId, ctx::ctx as ocsf_ctx, ocsf_emit, -}; - -/// Default interval (seconds) for re-fetching the inference route bundle from -/// the gateway in cluster mode. -/// -/// Override at runtime with the `OPENSHELL_ROUTE_REFRESH_INTERVAL_SECS` -/// environment variable. File-based routes (`--inference-routes`) are loaded -/// once at startup and never refreshed. -pub const DEFAULT_ROUTE_REFRESH_INTERVAL_SECS: u64 = 5; - -/// Route name for the sandbox system inference route. -const SANDBOX_SYSTEM_ROUTE_NAME: &str = "sandbox-system"; - -#[derive(Debug, Clone, Copy, PartialEq, Eq)] -pub enum InferenceRouteSource { - File, - Cluster, - None, -} - -pub fn infer_route_source( - sandbox_id: Option<&str>, - openshell_endpoint: Option<&str>, - inference_routes: Option<&str>, -) -> InferenceRouteSource { - if inference_routes.is_some() { - InferenceRouteSource::File - } else if sandbox_id.is_some() && openshell_endpoint.is_some() { - InferenceRouteSource::Cluster - } else { - InferenceRouteSource::None - } -} - -pub fn disable_inference_on_empty_routes(source: InferenceRouteSource) -> bool { - !matches!(source, InferenceRouteSource::Cluster) -} - -pub fn route_refresh_interval_secs() -> u64 { - let Ok(value) = std::env::var("OPENSHELL_ROUTE_REFRESH_INTERVAL_SECS") else { - return DEFAULT_ROUTE_REFRESH_INTERVAL_SECS; - }; - match value.parse::() { - Ok(interval) if interval > 0 => interval, - Ok(_) => { - warn!( - default_interval_secs = DEFAULT_ROUTE_REFRESH_INTERVAL_SECS, - "Ignoring zero route refresh interval" - ); - DEFAULT_ROUTE_REFRESH_INTERVAL_SECS - } - Err(error) => { - warn!( - interval = %value, - error = %error, - default_interval_secs = DEFAULT_ROUTE_REFRESH_INTERVAL_SECS, - "Ignoring invalid route refresh interval" - ); - DEFAULT_ROUTE_REFRESH_INTERVAL_SECS - } - } -} - -/// Build an [`InferenceContext`](crate::proxy::InferenceContext) by resolving -/// inference routes from either a local YAML file or the gateway bundle. -/// -/// If both a routes file and cluster credentials are provided, the routes file -/// wins and the cluster bundle is not fetched. -/// -/// Returns `None` if neither source is configured (inference routing disabled). -/// -/// # Errors -/// -/// Returns an error if loading the routes file fails or the file's routes -/// cannot be resolved. gRPC errors are swallowed (logged) and produce -/// `Ok(None)` so a missing cluster bundle disables inference routing rather -/// than aborting sandbox startup. -// `routes`/`router` are intentionally distinct nouns (the route list vs the -// router that consumes them); both names are clearer than alternatives. -#[allow(clippy::similar_names)] -pub async fn build_inference_context( - sandbox_id: Option<&str>, - openshell_endpoint: Option<&str>, - inference_routes: Option<&str>, -) -> Result>> { - use openshell_router::Router; - use openshell_router::config::RouterConfig; - - let source = infer_route_source(sandbox_id, openshell_endpoint, inference_routes); - - // Captured during the initial cluster bundle fetch so the background refresh - // loop can skip no-op updates from the very first tick. - let mut initial_revision: Option = None; - - let routes = match source { - InferenceRouteSource::File => { - let Some(path) = inference_routes else { - return Ok(None); - }; - - // Standalone mode: load routes from file (fail-fast on errors) - if sandbox_id.is_some() { - ocsf_emit!(ConfigStateChangeBuilder::new(ocsf_ctx()) - .severity(SeverityId::Informational) - .status(StatusId::Success) - .state(StateId::Enabled, "loaded") - .unmapped("inference_routes", serde_json::json!(path)) - .message(format!( - "Inference routes file takes precedence over cluster bundle [path:{path}]" - )) - .build()); - } - ocsf_emit!( - ConfigStateChangeBuilder::new(ocsf_ctx()) - .severity(SeverityId::Informational) - .status(StatusId::Success) - .state(StateId::Other, "loading") - .unmapped("inference_routes", serde_json::json!(path)) - .message(format!("Loading inference routes from file [path:{path}]")) - .build() - ); - let config = RouterConfig::load_from_file(std::path::Path::new(path)) - .map_err(|e| miette::miette!("failed to load inference routes {path}: {e}"))?; - config - .resolve_routes() - .map_err(|e| miette::miette!("failed to resolve routes from {path}: {e}"))? - } - InferenceRouteSource::Cluster => { - let (Some(_id), Some(endpoint)) = (sandbox_id, openshell_endpoint) else { - return Ok(None); - }; - - // Cluster mode: fetch bundle from gateway - info!(endpoint = %endpoint, "Fetching inference route bundle from gateway"); - match openshell_core::grpc_client::fetch_inference_bundle(endpoint).await { - Ok(bundle) => { - initial_revision = Some(bundle.revision.clone()); - ocsf_emit!( - ConfigStateChangeBuilder::new(ocsf_ctx()) - .severity(SeverityId::Informational) - .status(StatusId::Success) - .state(StateId::Enabled, "loaded") - .unmapped("route_count", serde_json::json!(bundle.routes.len())) - .unmapped("revision", serde_json::json!(&bundle.revision)) - .message(format!( - "Loaded inference route bundle [route_count:{} revision:{}]", - bundle.routes.len(), - bundle.revision - )) - .build() - ); - bundle_to_resolved_routes(&bundle) - } - Err(e) => { - // Distinguish expected "not configured" states from server errors. - // gRPC PermissionDenied/NotFound means inference bundle is unavailable - // for this sandbox — skip gracefully. Other errors are unexpected. - let msg = e.to_string(); - if msg.contains("permission denied") || msg.contains("not found") { - ocsf_emit!( - ConfigStateChangeBuilder::new(ocsf_ctx()) - .severity(SeverityId::Informational) - .status(StatusId::Success) - .state(StateId::Disabled, "disabled") - .unmapped("error", serde_json::json!(e.to_string())) - .message(format!( - "Inference bundle unavailable, routing disabled [error:{e}]" - )) - .build() - ); - return Ok(None); - } - ocsf_emit!(ConfigStateChangeBuilder::new(ocsf_ctx()) - .severity(SeverityId::Medium) - .status(StatusId::Failure) - .state(StateId::Disabled, "disabled") - .unmapped("error", serde_json::json!(e.to_string())) - .message(format!( - "Failed to fetch inference bundle, inference routing disabled [error:{e}]" - )) - .build()); - return Ok(None); - } - } - } - InferenceRouteSource::None => { - // No route source — inference routing is not configured - return Ok(None); - } - }; - - if routes.is_empty() && disable_inference_on_empty_routes(source) { - ocsf_emit!( - ConfigStateChangeBuilder::new(ocsf_ctx()) - .severity(SeverityId::Informational) - .status(StatusId::Success) - .state(StateId::Disabled, "disabled") - .message("No usable inference routes, inference routing disabled") - .build() - ); - return Ok(None); - } - - if routes.is_empty() { - ocsf_emit!(ConfigStateChangeBuilder::new(ocsf_ctx()) - .severity(SeverityId::Informational) - .status(StatusId::Success) - .state(StateId::Other, "waiting") - .message("Inference route bundle is empty; keeping routing enabled and waiting for refresh") - .build()); - } - - ocsf_emit!( - ConfigStateChangeBuilder::new(ocsf_ctx()) - .severity(SeverityId::Informational) - .status(StatusId::Success) - .state(StateId::Enabled, "enabled") - .unmapped("route_count", serde_json::json!(routes.len())) - .message(format!( - "Inference routing enabled with local execution [route_count:{}]", - routes.len() - )) - .build() - ); - - // Partition routes by name into user-facing and system caches. - let (user_routes, system_routes) = partition_routes(routes); - - let router = - Router::new().map_err(|e| miette::miette!("failed to initialize inference router: {e}"))?; - let patterns = crate::l7::inference::default_patterns(); - - let ctx = Arc::new(crate::proxy::InferenceContext::new( - patterns, - router, - user_routes, - system_routes, - )); - - // Spawn background route cache refresh for cluster mode at startup so - // request handling never depends on control-plane latency. - if matches!(source, InferenceRouteSource::Cluster) - && let (Some(_id), Some(endpoint)) = (sandbox_id, openshell_endpoint) - { - spawn_route_refresh( - ctx.route_cache(), - ctx.system_route_cache(), - endpoint.to_string(), - route_refresh_interval_secs(), - initial_revision, - ); - } - - Ok(Some(ctx)) -} - -/// Split resolved routes into user-facing and system caches by route name. -/// -/// Routes named `"sandbox-system"` go to the system cache; everything else -/// (including `"inference.local"` and empty names) goes to the user cache. -pub fn partition_routes( - routes: Vec, -) -> ( - Vec, - Vec, -) { - let mut user = Vec::new(); - let mut system = Vec::new(); - for r in routes { - if r.name == SANDBOX_SYSTEM_ROUTE_NAME { - system.push(r); - } else { - user.push(r); - } - } - (user, system) -} - -/// Convert a proto bundle response into resolved routes for the router. -pub fn bundle_to_resolved_routes( - bundle: &openshell_core::proto::GetInferenceBundleResponse, -) -> Vec { - bundle - .routes - .iter() - .map(|r| { - let (auth, default_headers, passthrough_headers) = - openshell_core::inference::route_headers_for_provider_type(&r.provider_type); - let timeout = if r.timeout_secs == 0 { - openshell_router::config::DEFAULT_ROUTE_TIMEOUT - } else { - Duration::from_secs(r.timeout_secs) - }; - openshell_router::config::ResolvedRoute { - name: r.name.clone(), - endpoint: r.base_url.clone(), - model: r.model_id.clone(), - api_key: r.api_key.clone(), - protocols: r.protocols.clone(), - auth, - default_headers, - passthrough_headers, - timeout, - model_in_path: r.model_in_path, - request_path_override: r.request_path_override.clone(), - } - }) - .collect() -} - -/// Spawn a background task that periodically refreshes both route caches from the gateway. -/// -/// The loop uses the bundle `revision` hash to avoid unnecessary cache writes -/// when routes haven't changed. `initial_revision` is the revision captured -/// during the startup fetch in [`build_inference_context`] so the first refresh -/// cycle can already skip a no-op update. -pub fn spawn_route_refresh( - user_cache: Arc>>, - system_cache: Arc>>, - endpoint: String, - interval_secs: u64, - initial_revision: Option, -) { - tokio::spawn(async move { - use tokio::time::{MissedTickBehavior, interval}; - - let mut current_revision = initial_revision; - - let mut tick = interval(Duration::from_secs(interval_secs)); - tick.set_missed_tick_behavior(MissedTickBehavior::Skip); - - loop { - tick.tick().await; - - match openshell_core::grpc_client::fetch_inference_bundle(&endpoint).await { - Ok(bundle) => { - if current_revision.as_deref() == Some(&bundle.revision) { - trace!(revision = %bundle.revision, "Inference bundle unchanged"); - continue; - } - - let routes = bundle_to_resolved_routes(&bundle); - let (user_routes, system_routes) = partition_routes(routes); - ocsf_emit!(ConfigStateChangeBuilder::new(ocsf_ctx()) - .severity(SeverityId::Informational) - .status(StatusId::Success) - .state(StateId::Enabled, "updated") - .unmapped("user_route_count", serde_json::json!(user_routes.len())) - .unmapped("system_route_count", serde_json::json!(system_routes.len())) - .unmapped("revision", serde_json::json!(&bundle.revision)) - .message(format!( - "Inference routes updated [user_route_count:{} system_route_count:{} revision:{}]", - user_routes.len(), - system_routes.len(), - bundle.revision - )) - .build()); - current_revision = Some(bundle.revision); - *user_cache.write().await = user_routes; - *system_cache.write().await = system_routes; - } - Err(e) => { - ocsf_emit!(ConfigStateChangeBuilder::new(ocsf_ctx()) - .severity(SeverityId::Medium) - .status(StatusId::Failure) - .state(StateId::Other, "stale") - .unmapped("error", serde_json::json!(e.to_string())) - .message(format!( - "Failed to refresh inference route cache, keeping stale routes [error:{e}]" - )) - .build()); - } - } - } - }); -} - -#[cfg(test)] -#[allow( - clippy::needless_raw_string_hashes, - clippy::similar_names, - reason = "Test code: test fixtures often use idiomatic forms not flagged in production." -)] -mod tests { - use super::*; - use std::sync::{LazyLock, Mutex}; - use temp_env::with_vars; - - static ENV_LOCK: LazyLock> = LazyLock::new(|| Mutex::new(())); - - #[test] - fn bundle_to_resolved_routes_converts_all_fields() { - let bundle = openshell_core::proto::GetInferenceBundleResponse { - routes: vec![ - openshell_core::proto::ResolvedRoute { - name: "frontier".to_string(), - base_url: "https://api.example.com/v1".to_string(), - api_key: "sk-test-key".to_string(), - model_id: "gpt-4".to_string(), - protocols: vec![ - "openai_chat_completions".to_string(), - "openai_responses".to_string(), - ], - provider_type: "openai".to_string(), - timeout_secs: 0, - model_in_path: false, - request_path_override: None, - }, - openshell_core::proto::ResolvedRoute { - name: "local".to_string(), - base_url: "http://vllm:8000/v1".to_string(), - api_key: "local-key".to_string(), - model_id: "llama-3".to_string(), - protocols: vec!["openai_chat_completions".to_string()], - provider_type: String::new(), - timeout_secs: 120, - model_in_path: false, - request_path_override: None, - }, - ], - revision: "abc123".to_string(), - generated_at_ms: 1000, - }; - - let routes = bundle_to_resolved_routes(&bundle); - - assert_eq!(routes.len(), 2); - assert_eq!(routes[0].endpoint, "https://api.example.com/v1"); - assert_eq!(routes[0].model, "gpt-4"); - assert_eq!(routes[0].api_key, "sk-test-key"); - assert_eq!( - routes[0].auth, - openshell_core::inference::AuthHeader::Bearer - ); - assert_eq!( - routes[0].protocols, - vec!["openai_chat_completions", "openai_responses"] - ); - assert_eq!( - routes[0].timeout, - openshell_router::config::DEFAULT_ROUTE_TIMEOUT, - "timeout_secs=0 should map to default" - ); - assert_eq!(routes[1].endpoint, "http://vllm:8000/v1"); - assert_eq!( - routes[1].auth, - openshell_core::inference::AuthHeader::Bearer - ); - assert_eq!( - routes[1].timeout, - Duration::from_secs(120), - "timeout_secs=120 should map to 120s" - ); - } - - #[test] - fn bundle_to_resolved_routes_handles_empty_bundle() { - let bundle = openshell_core::proto::GetInferenceBundleResponse { - routes: vec![], - revision: "empty".to_string(), - generated_at_ms: 0, - }; - - let routes = bundle_to_resolved_routes(&bundle); - assert!(routes.is_empty()); - } - - #[test] - fn bundle_to_resolved_routes_preserves_name_field() { - let bundle = openshell_core::proto::GetInferenceBundleResponse { - routes: vec![openshell_core::proto::ResolvedRoute { - name: "sandbox-system".to_string(), - base_url: "https://api.example.com/v1".to_string(), - api_key: "key".to_string(), - model_id: "model".to_string(), - protocols: vec!["openai_chat_completions".to_string()], - provider_type: "openai".to_string(), - timeout_secs: 0, - model_in_path: false, - request_path_override: None, - }], - revision: "rev".to_string(), - generated_at_ms: 0, - }; - - let routes = bundle_to_resolved_routes(&bundle); - assert_eq!(routes[0].name, "sandbox-system"); - } - - #[test] - fn routes_segregated_by_name() { - let routes = vec![ - openshell_router::config::ResolvedRoute { - name: "inference.local".to_string(), - endpoint: "https://api.openai.com/v1".to_string(), - model: "gpt-4o".to_string(), - api_key: "key1".to_string(), - protocols: vec!["openai_chat_completions".to_string()], - auth: openshell_core::inference::AuthHeader::Bearer, - default_headers: vec![], - passthrough_headers: vec![], - timeout: openshell_router::config::DEFAULT_ROUTE_TIMEOUT, - model_in_path: false, - request_path_override: None, - }, - openshell_router::config::ResolvedRoute { - name: "sandbox-system".to_string(), - endpoint: "https://api.anthropic.com/v1".to_string(), - model: "claude-sonnet-4-20250514".to_string(), - api_key: "key2".to_string(), - protocols: vec!["anthropic_messages".to_string()], - auth: openshell_core::inference::AuthHeader::Custom("x-api-key"), - default_headers: vec![], - passthrough_headers: vec![], - timeout: openshell_router::config::DEFAULT_ROUTE_TIMEOUT, - model_in_path: false, - request_path_override: None, - }, - ]; - - let (user, system) = partition_routes(routes); - assert_eq!(user.len(), 1); - assert_eq!(user[0].name, "inference.local"); - assert_eq!(system.len(), 1); - assert_eq!(system[0].name, "sandbox-system"); - } - - // -- build_inference_context tests -- - - #[tokio::test] - async fn build_inference_context_route_file_loads_routes() { - use std::io::Write; - - let yaml = r#" -routes: - - name: inference.local - endpoint: http://localhost:8000/v1 - model: llama-3 - protocols: [openai_chat_completions] - api_key: test-key -"#; - let mut f = tempfile::NamedTempFile::new().unwrap(); - f.write_all(yaml.as_bytes()).unwrap(); - let path = f.path().to_str().unwrap(); - - let ctx = build_inference_context(None, None, Some(path)) - .await - .expect("should load routes from file"); - - let ctx = ctx.expect("context should be Some"); - let cache = ctx.route_cache(); - let routes = cache.read().await; - assert_eq!(routes.len(), 1); - assert_eq!(routes[0].endpoint, "http://localhost:8000/v1"); - } - - #[tokio::test] - async fn build_inference_context_empty_route_file_returns_none() { - use std::io::Write; - - // Route file with empty routes list → inference routing disabled (not an error) - let yaml = "routes: []\n"; - let mut f = tempfile::NamedTempFile::new().unwrap(); - f.write_all(yaml.as_bytes()).unwrap(); - let path = f.path().to_str().unwrap(); - - let ctx = build_inference_context(None, None, Some(path)) - .await - .expect("empty routes file should not error"); - assert!( - ctx.is_none(), - "empty routes should disable inference routing" - ); - } - - #[tokio::test] - async fn build_inference_context_no_sources_returns_none() { - let ctx = build_inference_context(None, None, None) - .await - .expect("should succeed with None"); - - assert!(ctx.is_none(), "no sources should return None"); - } - - #[tokio::test] - async fn build_inference_context_route_file_overrides_cluster() { - use std::io::Write; - - let yaml = r#" -routes: - - name: inference.local - endpoint: http://localhost:9999/v1 - model: file-model - protocols: [openai_chat_completions] - api_key: file-key -"#; - let mut f = tempfile::NamedTempFile::new().unwrap(); - f.write_all(yaml.as_bytes()).unwrap(); - let path = f.path().to_str().unwrap(); - - // Even with sandbox_id and endpoint, route_file takes precedence - let ctx = build_inference_context(Some("sb-1"), Some("http://localhost:50051"), Some(path)) - .await - .expect("should load from file"); - - let ctx = ctx.expect("context should be Some"); - let cache = ctx.route_cache(); - let routes = cache.read().await; - assert_eq!(routes[0].endpoint, "http://localhost:9999/v1"); - } - - #[test] - fn infer_route_source_prefers_file_mode() { - assert_eq!( - infer_route_source( - Some("sb-1"), - Some("http://localhost:50051"), - Some("routes.yaml") - ), - InferenceRouteSource::File - ); - } - - #[test] - fn infer_route_source_cluster_requires_id_and_endpoint() { - assert_eq!( - infer_route_source(Some("sb-1"), Some("http://localhost:50051"), None), - InferenceRouteSource::Cluster - ); - assert_eq!( - infer_route_source(Some("sb-1"), None, None), - InferenceRouteSource::None - ); - assert_eq!( - infer_route_source(None, Some("http://localhost:50051"), None), - InferenceRouteSource::None - ); - } - - #[test] - fn disable_inference_on_empty_routes_depends_on_source() { - assert!(disable_inference_on_empty_routes( - InferenceRouteSource::File - )); - assert!(!disable_inference_on_empty_routes( - InferenceRouteSource::Cluster - )); - assert!(disable_inference_on_empty_routes( - InferenceRouteSource::None - )); - } - - // ---- Route refresh interval + revision tests ---- - - #[test] - fn default_route_refresh_interval_is_five_seconds() { - assert_eq!(DEFAULT_ROUTE_REFRESH_INTERVAL_SECS, 5); - } - - #[test] - fn route_refresh_interval_uses_env_override() { - let _guard = ENV_LOCK.lock().unwrap(); - with_vars( - [("OPENSHELL_ROUTE_REFRESH_INTERVAL_SECS", Some("9"))], - || { - assert_eq!(route_refresh_interval_secs(), 9); - }, - ); - } - - #[test] - fn route_refresh_interval_rejects_zero() { - let _guard = ENV_LOCK.lock().unwrap(); - with_vars( - [("OPENSHELL_ROUTE_REFRESH_INTERVAL_SECS", Some("0"))], - || { - assert_eq!( - route_refresh_interval_secs(), - DEFAULT_ROUTE_REFRESH_INTERVAL_SECS - ); - }, - ); - } - - #[test] - fn route_refresh_interval_rejects_invalid_values() { - let _guard = ENV_LOCK.lock().unwrap(); - with_vars( - [("OPENSHELL_ROUTE_REFRESH_INTERVAL_SECS", Some("abc"))], - || { - assert_eq!( - route_refresh_interval_secs(), - DEFAULT_ROUTE_REFRESH_INTERVAL_SECS - ); - }, - ); - } - - #[tokio::test] - async fn route_cache_preserves_content_when_not_written() { - use std::sync::Arc; - use tokio::sync::RwLock; - - let routes = vec![openshell_router::config::ResolvedRoute { - name: "inference.local".to_string(), - endpoint: "http://original:8000/v1".to_string(), - model: "original-model".to_string(), - api_key: "key".to_string(), - auth: openshell_core::inference::AuthHeader::Bearer, - protocols: vec!["openai_chat_completions".to_string()], - default_headers: vec![], - passthrough_headers: vec![], - timeout: openshell_router::config::DEFAULT_ROUTE_TIMEOUT, - model_in_path: false, - request_path_override: None, - }]; - - let cache = Arc::new(RwLock::new(routes)); - - // Verify the cache preserves its content — the revision-based skip - // logic in spawn_route_refresh ensures the cache is only written - // when the revision actually changes. - let read = cache.read().await; - assert_eq!(read.len(), 1); - assert_eq!(read[0].model, "original-model"); - } -} diff --git a/crates/openshell-supervisor-network/src/l7/inference.rs b/crates/openshell-supervisor-network/src/l7/inference.rs deleted file mode 100644 index 5aefab9e9..000000000 --- a/crates/openshell-supervisor-network/src/l7/inference.rs +++ /dev/null @@ -1,1003 +0,0 @@ -// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -// SPDX-License-Identifier: Apache-2.0 - -//! Inference API pattern detection and gateway rerouting. -//! -//! For requests targeting `inference.local`, this module detects whether the -//! HTTP request is a known inference API call and routes it through the local -//! sandbox router. - -/// How an inference protocol delivers its response to the sandboxed client. -/// -/// `Streaming` protocols (chat completions, completions, responses, Anthropic -/// messages) emit a Server-Sent Events token stream and are served through the -/// chunked transfer-encoding path so tokens reach the client incrementally. -/// -/// `Buffered` protocols (embeddings, model discovery) return a single JSON -/// object the client parses whole. They must be served in one piece with an -/// accurate `Content-Length`. Sending them through the streaming path is -/// unsafe: a mid-body truncation (the streaming size cap or idle timeout) -/// appends an SSE error event to bytes the client decodes as one JSON object, -/// silently corrupting it. -/// -/// Framing is a property of the protocol, declared once per pattern in -/// [`default_patterns`], so the streaming-vs-buffered decision cannot drift -/// across the dispatch sites that consume it. -#[derive(Debug, Clone, Copy, PartialEq, Eq)] -pub enum ResponseFraming { - /// SSE token stream, served via chunked transfer-encoding. - /// - /// The `OpenAI` completion-style protocols are classified streaming - /// unconditionally. They are dual-mode (a `stream: false` request returns - /// one buffered JSON object), but the dispatch keys framing off the - /// protocol alone and does not inspect the request body, so they are - /// streamed defensively. Their buffered responses tolerate chunked framing; - /// only the embeddings and model-discovery shapes (no streaming mode at - /// all) must be served buffered. - Streaming, - /// Single JSON object, served buffered with an accurate `Content-Length`. - Buffered, -} - -/// An inference API pattern for detecting inference calls in intercepted traffic. -#[derive(Debug, Clone)] -pub struct InferenceApiPattern { - pub method: String, - pub path_glob: String, - pub protocol: String, - pub kind: String, - /// Response delivery mode for this protocol. Selects the buffered or - /// streaming proxy path; see [`ResponseFraming`]. - pub framing: ResponseFraming, -} - -impl InferenceApiPattern { - /// Whether this protocol's response must be served buffered (one JSON - /// object framed with an accurate `Content-Length`) rather than streamed. - #[must_use] - pub fn is_buffered(&self) -> bool { - matches!(self.framing, ResponseFraming::Buffered) - } -} - -/// Default patterns for known inference APIs (`OpenAI`, Anthropic, AWS Bedrock). -pub fn default_patterns() -> Vec { - vec![ - InferenceApiPattern { - method: "POST".to_string(), - path_glob: "/v1/chat/completions".to_string(), - protocol: "openai_chat_completions".to_string(), - kind: "chat_completion".to_string(), - framing: ResponseFraming::Streaming, - }, - InferenceApiPattern { - method: "POST".to_string(), - path_glob: "/v1/completions".to_string(), - protocol: "openai_completions".to_string(), - kind: "completion".to_string(), - framing: ResponseFraming::Streaming, - }, - InferenceApiPattern { - method: "POST".to_string(), - path_glob: "/v1/responses".to_string(), - protocol: "openai_responses".to_string(), - kind: "responses".to_string(), - framing: ResponseFraming::Streaming, - }, - InferenceApiPattern { - method: "POST".to_string(), - path_glob: "/v1/embeddings".to_string(), - protocol: "openai_embeddings".to_string(), - kind: "embeddings".to_string(), - framing: ResponseFraming::Buffered, - }, - InferenceApiPattern { - method: "POST".to_string(), - path_glob: "/v1/messages".to_string(), - protocol: "anthropic_messages".to_string(), - kind: "messages".to_string(), - framing: ResponseFraming::Streaming, - }, - // Model discovery returns one JSON object (a model list), never an SSE - // stream, so it is served buffered for the same reason as embeddings. - InferenceApiPattern { - method: "GET".to_string(), - path_glob: "/v1/models".to_string(), - protocol: "model_discovery".to_string(), - kind: "models_list".to_string(), - framing: ResponseFraming::Buffered, - }, - InferenceApiPattern { - method: "GET".to_string(), - path_glob: "/v1/models/*".to_string(), - protocol: "model_discovery".to_string(), - kind: "models_get".to_string(), - framing: ResponseFraming::Buffered, - }, - // AWS Bedrock InvokeModel. The `*` segment is the Bedrock model id - // (e.g. `anthropic.claude-opus-4-7`). - // - // InvokeModel returns ONE JSON object the client decodes whole — it - // must be served buffered with an accurate `Content-Length`, otherwise - // the streaming proxy's size-cap or idle-timeout failure mode would - // append an SSE error event to bytes the caller decodes as one JSON - // object, silently corrupting it. - // - // `InvokeModelWithResponseStream` - // (`/model/{id}/invoke-with-response-stream`) is deferred to a - // follow-up: Bedrock streams use AWS event-stream framing, but the - // shared streaming relay's truncation/timeout/upstream-error path - // emits SSE-formatted error frames, which would corrupt downstream - // event-stream parsers. The follow-up adds protocol-aware error - // termination before re-introducing the streaming pattern. - InferenceApiPattern { - method: "POST".to_string(), - path_glob: "/model/*/invoke".to_string(), - protocol: "aws_bedrock_invoke".to_string(), - kind: "messages".to_string(), - framing: ResponseFraming::Buffered, - }, - ] -} - -/// Check if an HTTP request matches a known inference API pattern. -/// -/// Path globs support two wildcard shapes (one per pattern, not both): -/// - **Trailing `/*`**: `/v1/models/*` matches `/v1/models` and any -/// `/v1/models/` (one or many path segments). -/// - **Middle `/*/`**: `/model/*/invoke` matches `/model//invoke` -/// for a single non-empty segment that contains no `/`. Used for -/// AWS Bedrock's `/model/{modelId}/invoke[-with-response-stream]`. -pub fn detect_inference_pattern<'a>( - method: &str, - path: &str, - patterns: &'a [InferenceApiPattern], -) -> Option<&'a InferenceApiPattern> { - // Strip query string for matching - let path_only = path.split('?').next().unwrap_or(path); - patterns.iter().find(|p| { - if !method.eq_ignore_ascii_case(&p.method) { - return false; - } - - if let Some(prefix) = p.path_glob.strip_suffix("/*") { - return path_only == prefix - || path_only - .strip_prefix(prefix) - .is_some_and(|suffix| suffix.starts_with('/')); - } - - if let Some((before, after)) = p.path_glob.split_once("/*/") { - let Some(rest) = path_only.strip_prefix(before) else { - return false; - }; - let Some(rest) = rest.strip_prefix('/') else { - return false; - }; - // rest must look like `/` where is non-empty - // and contains no `/` (single path segment). - let Some(slash_at) = rest.find('/') else { - return false; - }; - let Some(after_segment) = rest.get(slash_at + 1..) else { - return false; - }; - return slash_at > 0 && after_segment == after; - } - - path_only == p.path_glob - }) -} - -/// A parsed HTTP request from the intercepted tunnel. -pub struct ParsedHttpRequest { - pub method: String, - pub path: String, - pub headers: Vec<(String, String)>, - pub body: Vec, -} - -/// Result of attempting to parse an HTTP request from a buffer. -pub enum ParseResult { - /// A complete request was parsed, along with the byte count consumed. - Complete(ParsedHttpRequest, usize), - /// Headers are incomplete — caller should read more data. - Incomplete, - /// The request is malformed and must be rejected (e.g., duplicate Content-Length). - Invalid(String), -} - -/// Try to parse an HTTP/1.1 request from raw bytes. -/// -/// Returns [`ParseResult::Complete`] with the parsed request and bytes consumed, -/// or [`ParseResult::Incomplete`] if more data is needed. -pub fn try_parse_http_request(buf: &[u8]) -> ParseResult { - let Some(header_end) = buf.windows(4).position(|w| w == b"\r\n\r\n") else { - return ParseResult::Incomplete; - }; - let headers_bytes = &buf[..header_end]; - let Ok(header_str) = std::str::from_utf8(headers_bytes) else { - return ParseResult::Incomplete; - }; - let body_start = header_end + 4; - - let mut lines = header_str.split("\r\n"); - let Some(request_line) = lines.next() else { - return ParseResult::Incomplete; - }; - let mut parts = request_line.split_whitespace(); - let (Some(method), Some(path)) = (parts.next(), parts.next()) else { - return ParseResult::Incomplete; - }; - let method = method.to_string(); - let path = path.to_string(); - - let mut headers = Vec::new(); - let mut content_length: usize = 0; - let mut has_content_length = false; - let mut is_chunked = false; - for line in lines { - if line.is_empty() { - break; - } - if let Some((name, value)) = line.split_once(':') { - let name = name.trim().to_string(); - let value = value.trim().to_string(); - if name.eq_ignore_ascii_case("content-length") { - let new_len: usize = match value.parse() { - Ok(v) => v, - Err(_) => { - return ParseResult::Invalid(format!( - "invalid Content-Length value: {value}" - )); - } - }; - if has_content_length && new_len != content_length { - return ParseResult::Invalid(format!( - "duplicate Content-Length headers with differing values ({content_length} vs {new_len})" - )); - } - content_length = new_len; - has_content_length = true; - } - if name.eq_ignore_ascii_case("transfer-encoding") - && value - .split(',') - .any(|enc| enc.trim().eq_ignore_ascii_case("chunked")) - { - is_chunked = true; - } - headers.push((name, value)); - } - } - - if is_chunked && has_content_length { - return ParseResult::Invalid( - "Request contains both Transfer-Encoding and Content-Length headers".to_string(), - ); - } - - let (body, consumed) = if is_chunked { - let Some((decoded_body, consumed)) = parse_chunked_body(buf, body_start) else { - return ParseResult::Incomplete; - }; - (decoded_body, consumed) - } else { - let total_len = body_start + content_length; - if buf.len() < total_len { - return ParseResult::Incomplete; - } - (buf[body_start..total_len].to_vec(), total_len) - }; - - ParseResult::Complete( - ParsedHttpRequest { - method, - path, - headers, - body, - }, - consumed, - ) -} - -/// Maximum decoded body size from chunked transfer encoding (10 MiB). -/// Matches the caller's `MAX_INFERENCE_BUF` limit. -const MAX_CHUNKED_BODY: usize = 10 * 1024 * 1024; - -/// Maximum number of chunks to process. Normal HTTP clients send the body -/// in a handful of large chunks; thousands of tiny chunks indicate abuse. -const MAX_CHUNK_COUNT: usize = 4096; - -/// Parse an HTTP chunked body from `buf[start..]`. -/// -/// Returns `(decoded_body, total_consumed_bytes_from_buf_start)` when complete, -/// or `None` if more bytes are needed or resource limits are exceeded. -fn parse_chunked_body(buf: &[u8], start: usize) -> Option<(Vec, usize)> { - let mut pos = start; - let mut body = Vec::new(); - let mut chunk_count: usize = 0; - - loop { - chunk_count += 1; - if chunk_count > MAX_CHUNK_COUNT { - return None; - } - - let size_line_end = find_crlf(buf, pos)?; - let size_line = std::str::from_utf8(&buf[pos..size_line_end]).ok()?; - let size_token = size_line.split(';').next()?.trim(); - let chunk_size = usize::from_str_radix(size_token, 16).ok()?; - pos = size_line_end.checked_add(2)?; - - if chunk_size == 0 { - // Parse trailers (if any). Terminates on empty trailer line. - loop { - let trailer_end = find_crlf(buf, pos)?; - let trailer_line = &buf[pos..trailer_end]; - pos = trailer_end.checked_add(2)?; - if trailer_line.is_empty() { - return Some((body, pos)); - } - } - } - - // Early reject: chunk cannot possibly fit in remaining buffer. - let remaining = buf.len().saturating_sub(pos); - if chunk_size > remaining { - return None; - } - - // Reject if decoded body would exceed size limit. - if body.len().saturating_add(chunk_size) > MAX_CHUNKED_BODY { - return None; - } - - let chunk_end = pos.checked_add(chunk_size)?; - let chunk_crlf_end = chunk_end.checked_add(2)?; - if buf.len() < chunk_crlf_end { - return None; - } - if &buf[chunk_end..chunk_crlf_end] != b"\r\n" { - return None; - } - - body.extend_from_slice(&buf[pos..chunk_end]); - pos = chunk_crlf_end; - } -} - -fn find_crlf(buf: &[u8], start: usize) -> Option { - buf.get(start..)? - .windows(2) - .position(|w| w == b"\r\n") - .map(|offset| start + offset) -} - -/// Reason phrase for an HTTP status code used on the inference proxy path. -/// -/// Covers the statuses produced by the router error mapping (400/401/403/500/ -/// 502/503) and the upstream codes an inference backend can pass through -/// verbatim (404/405 on unknown model or method, 422 on malformed embeddings -/// input, 429 on rate limit). Unknown codes fall back to `"Unknown"` so the -/// status line is still well-formed. -fn http_status_text(status: u16) -> &'static str { - match status { - 200 => "OK", - 400 => "Bad Request", - 401 => "Unauthorized", - 403 => "Forbidden", - 404 => "Not Found", - 405 => "Method Not Allowed", - 411 => "Length Required", - 413 => "Payload Too Large", - 422 => "Unprocessable Entity", - 429 => "Too Many Requests", - 500 => "Internal Server Error", - 502 => "Bad Gateway", - 503 => "Service Unavailable", - _ => "Unknown", - } -} - -/// Format an HTTP/1.1 response from status, headers, and body. -pub fn format_http_response(status: u16, headers: &[(String, String)], body: &[u8]) -> Vec { - use std::fmt::Write; - - let status_text = http_status_text(status); - - let mut response = format!("HTTP/1.1 {status} {status_text}\r\n"); - let mut has_content_length = false; - for (name, value) in headers { - let _ = write!(response, "{name}: {value}\r\n"); - if name.eq_ignore_ascii_case("content-length") { - has_content_length = true; - } - } - if !has_content_length { - let _ = write!(response, "content-length: {}\r\n", body.len()); - } - response.push_str("\r\n"); - - let mut bytes = response.into_bytes(); - bytes.extend_from_slice(body); - bytes -} - -/// Format HTTP/1.1 response headers for a chunked (streaming) response. -/// -/// Emits the status line, supplied headers (stripping any `content-length` or -/// `transfer-encoding` the upstream may have sent), and a -/// `transfer-encoding: chunked` header. The body is **not** included — the -/// caller writes chunks separately via [`format_chunk`] and -/// [`format_chunk_terminator`]. -pub fn format_http_response_header(status: u16, headers: &[(String, String)]) -> Vec { - use std::fmt::Write; - - let status_text = http_status_text(status); - - let mut response = format!("HTTP/1.1 {status} {status_text}\r\n"); - for (name, value) in headers { - // Skip framing headers — we always emit chunked TE. - if name.eq_ignore_ascii_case("content-length") - || name.eq_ignore_ascii_case("transfer-encoding") - { - continue; - } - let _ = write!(response, "{name}: {value}\r\n"); - } - let _ = write!(response, "transfer-encoding: chunked\r\n"); - response.push_str("\r\n"); - response.into_bytes() -} - -/// Format a single HTTP chunked transfer-encoding segment. -/// -/// Returns `\r\n\r\n`. -pub fn format_chunk(data: &[u8]) -> Vec { - let mut buf = format!("{:x}\r\n", data.len()).into_bytes(); - buf.extend_from_slice(data); - buf.extend_from_slice(b"\r\n"); - buf -} - -/// The HTTP chunked transfer-encoding terminator: `0\r\n\r\n`. -pub fn format_chunk_terminator() -> &'static [u8] { - b"0\r\n\r\n" -} - -/// Format an SSE error event for injection into a streaming response. -/// -/// Sent just before the chunked terminator when the proxy truncates a stream -/// due to timeout, byte limit, or upstream error. Clients parsing SSE events -/// can detect this and surface the error instead of silently losing data. -/// -/// The `reason` must NOT contain internal URLs, hostnames, or credentials — -/// the OCSF log captures full detail server-side. -pub fn format_sse_error(reason: &str) -> Vec { - // Use serde_json to escape control characters, quotes, and backslashes - // correctly. A handwritten escape can't safely cover \u0000-\u001F, and - // an unescaped \n\n in `reason` would split the SSE event into two - // frames, allowing a malicious upstream to inject a forged event. - let payload = serde_json::json!({ - "error": { - "message": reason, - "type": "proxy_stream_error", - } - }); - let mut out = Vec::with_capacity(reason.len() + 64); - out.extend_from_slice(b"data: "); - // serde_json::to_writer is infallible for in-memory Vec. - serde_json::to_writer(&mut out, &payload).expect("serializing static schema cannot fail"); - out.extend_from_slice(b"\n\n"); - out -} - -#[cfg(test)] -mod tests { - use super::*; - - #[test] - fn detect_openai_chat_completions() { - let patterns = default_patterns(); - let result = detect_inference_pattern("POST", "/v1/chat/completions", &patterns); - assert!(result.is_some()); - assert_eq!(result.unwrap().protocol, "openai_chat_completions"); - } - - #[test] - fn detect_openai_responses() { - let patterns = default_patterns(); - let result = detect_inference_pattern("POST", "/v1/responses", &patterns); - assert!(result.is_some()); - assert_eq!(result.unwrap().protocol, "openai_responses"); - } - - #[test] - fn detect_anthropic_messages() { - let patterns = default_patterns(); - let result = detect_inference_pattern("POST", "/v1/messages", &patterns); - assert!(result.is_some()); - assert_eq!(result.unwrap().protocol, "anthropic_messages"); - } - - #[test] - fn detect_with_query_string() { - let patterns = default_patterns(); - let result = - detect_inference_pattern("POST", "/v1/chat/completions?stream=true", &patterns); - assert!(result.is_some()); - } - - #[test] - fn no_match_for_get() { - let patterns = default_patterns(); - let result = detect_inference_pattern("GET", "/v1/chat/completions", &patterns); - assert!(result.is_none()); - } - - #[test] - fn detect_get_models() { - let patterns = default_patterns(); - let result = detect_inference_pattern("GET", "/v1/models", &patterns); - assert!(result.is_some()); - let pattern = result.unwrap(); - assert_eq!(pattern.protocol, "model_discovery"); - // A model list is one JSON object; it must be served buffered, never - // through the SSE streaming path that could append an error frame. - assert!(pattern.is_buffered()); - } - - #[test] - fn detect_get_model_details() { - let patterns = default_patterns(); - let result = detect_inference_pattern("GET", "/v1/models/gpt-4.1", &patterns); - assert!(result.is_some()); - let pattern = result.unwrap(); - assert_eq!(pattern.protocol, "model_discovery"); - assert!(pattern.is_buffered()); - } - - #[test] - fn detect_openai_embeddings() { - let patterns = default_patterns(); - let result = detect_inference_pattern("POST", "/v1/embeddings", &patterns); - assert!(result.is_some()); - let pattern = result.unwrap(); - assert_eq!(pattern.protocol, "openai_embeddings"); - assert_eq!(pattern.kind, "embeddings"); - assert!(pattern.is_buffered()); - } - - /// Every default pattern must declare framing consistent with how its - /// protocol actually responds: single-JSON-object protocols buffered, - /// SSE token streams streaming. A wrong classification routes a response - /// through the path that can corrupt or stall it. - #[test] - fn protocol_framing_classification() { - let patterns = default_patterns(); - for pattern in &patterns { - let expected_buffered = matches!( - pattern.protocol.as_str(), - "model_discovery" | "openai_embeddings" | "aws_bedrock_invoke" - ); - assert_eq!( - pattern.is_buffered(), - expected_buffered, - "{} ({}) has wrong framing", - pattern.protocol, - pattern.path_glob - ); - } - } - - #[test] - fn detect_aws_bedrock_invoke() { - let patterns = default_patterns(); - let result = - detect_inference_pattern("POST", "/model/anthropic.claude-opus-4-7/invoke", &patterns); - assert!(result.is_some()); - assert_eq!(result.unwrap().protocol, "aws_bedrock_invoke"); - assert_eq!(result.unwrap().kind, "messages"); - } - - /// `InvokeModelWithResponseStream` is intentionally NOT advertised by - /// the default pattern set today. The shared streaming relay's - /// truncation/timeout/upstream-error path emits SSE-formatted error - /// frames, which would corrupt the AWS event-stream framing Bedrock - /// streams use. The pattern is restored alongside protocol-aware - /// error termination in a follow-up; until then, intercepted - /// `/invoke-with-response-stream` requests fall through to the - /// non-inference path rather than being mis-routed through the - /// SSE-injecting relay. - #[test] - fn aws_bedrock_invoke_stream_pattern_is_deferred() { - let patterns = default_patterns(); - assert!( - detect_inference_pattern( - "POST", - "/model/anthropic.claude-opus-4-7/invoke-with-response-stream", - &patterns, - ) - .is_none(), - "InvokeModelWithResponseStream must not be advertised until \ - protocol-aware AWS event-stream error framing exists" - ); - assert!( - !patterns - .iter() - .any(|p| p.protocol == "aws_bedrock_invoke_stream"), - "no pattern should declare protocol=aws_bedrock_invoke_stream" - ); - } - - #[test] - fn aws_bedrock_invoke_with_query_string() { - let patterns = default_patterns(); - let result = detect_inference_pattern("POST", "/model/foo.bar/invoke?trace=1", &patterns); - assert!(result.is_some()); - assert_eq!(result.unwrap().protocol, "aws_bedrock_invoke"); - } - - #[test] - fn aws_bedrock_rejects_empty_model_id() { - let patterns = default_patterns(); - // `/model//invoke` — empty wildcard segment is not a valid Bedrock id. - assert!(detect_inference_pattern("POST", "/model//invoke", &patterns).is_none()); - } - - #[test] - fn aws_bedrock_rejects_multi_segment_model_id() { - let patterns = default_patterns(); - // The `*` matches a single path segment only; multi-segment ids must - // not match (would be a path-traversal liability otherwise). - assert!(detect_inference_pattern("POST", "/model/foo/bar/invoke", &patterns).is_none()); - } - - #[test] - fn aws_bedrock_rejects_get() { - let patterns = default_patterns(); - assert!( - detect_inference_pattern("GET", "/model/anthropic.claude-opus-4-7/invoke", &patterns) - .is_none() - ); - } - - #[test] - fn aws_bedrock_rejects_unknown_action() { - let patterns = default_patterns(); - assert!(detect_inference_pattern("POST", "/model/foo/converse", &patterns).is_none()); - } - - /// `InvokeModel` returns one JSON object — must be served buffered. - /// Sending it through the streaming proxy would risk truncation or an - /// appended SSE error event corrupting the JSON body the caller decodes. - #[test] - fn aws_bedrock_invoke_is_buffered() { - let patterns = default_patterns(); - let invoke = - detect_inference_pattern("POST", "/model/anthropic.claude-opus-4-7/invoke", &patterns) - .expect("InvokeModel pattern must match"); - assert_eq!(invoke.protocol, "aws_bedrock_invoke"); - assert!( - invoke.is_buffered(), - "InvokeModel must be Buffered (one JSON object, accurate Content-Length); \ - streaming would risk corrupting the response" - ); - } - - #[test] - fn parse_simple_post_request() { - let body = b"{\"hello\":true}"; - let header = format!( - "POST /v1/chat/completions HTTP/1.1\r\nHost: api.openai.com\r\nContent-Length: {}\r\n\r\n", - body.len() - ); - let mut request = header.into_bytes(); - request.extend_from_slice(body); - let ParseResult::Complete(parsed, consumed) = try_parse_http_request(&request) else { - panic!("expected Complete"); - }; - assert_eq!(parsed.method, "POST"); - assert_eq!(parsed.path, "/v1/chat/completions"); - assert_eq!(parsed.body, body); - assert_eq!(consumed, request.len()); - } - - #[test] - fn parse_incomplete_headers() { - let request = b"POST /v1/chat/completions HTTP/1.1\r\nHost: api.openai.com\r\n"; - assert!(matches!( - try_parse_http_request(request), - ParseResult::Incomplete - )); - } - - #[test] - fn parse_chunked_decodes_body() { - let request = b"POST /v1/chat/completions HTTP/1.1\r\nHost: api.openai.com\r\nTransfer-Encoding: chunked\r\n\r\nA\r\n{\"a\":true}\r\n0\r\n\r\n"; - let ParseResult::Complete(parsed, consumed) = try_parse_http_request(request) else { - panic!("expected Complete"); - }; - assert_eq!(parsed.body, br#"{"a":true}"#); - assert_eq!(consumed, request.len()); - } - - #[test] - fn parse_chunked_incomplete() { - let request = b"POST /v1/chat/completions HTTP/1.1\r\nTransfer-Encoding: chunked\r\n\r\n4\r\n{\"a\r\n"; - assert!(matches!( - try_parse_http_request(request), - ParseResult::Incomplete - )); - } - - #[test] - fn format_response_basic() { - let body = b"{\"ok\":true}"; - let response = format_http_response(200, &[], body); - let response_str = String::from_utf8_lossy(&response); - assert!(response_str.starts_with("HTTP/1.1 200 OK\r\n")); - assert!(response_str.contains("content-length: 11\r\n")); - assert!(response_str.ends_with("{\"ok\":true}")); - } - - #[test] - fn format_response_header_chunked() { - let headers = vec![ - ("content-type".to_string(), "text/event-stream".to_string()), - ("x-request-id".to_string(), "abc123".to_string()), - ]; - let header = format_http_response_header(200, &headers); - let header_str = String::from_utf8_lossy(&header); - assert!(header_str.starts_with("HTTP/1.1 200 OK\r\n")); - assert!(header_str.contains("content-type: text/event-stream\r\n")); - assert!(header_str.contains("x-request-id: abc123\r\n")); - assert!(header_str.contains("transfer-encoding: chunked\r\n")); - assert!(header_str.ends_with("\r\n")); - // Must NOT contain content-length - assert!(!header_str.to_lowercase().contains("content-length")); - } - - #[test] - fn format_response_header_strips_upstream_framing() { - let headers = vec![ - ("content-length".to_string(), "9999".to_string()), - ("transfer-encoding".to_string(), "chunked".to_string()), - ("content-type".to_string(), "application/json".to_string()), - ]; - let header = format_http_response_header(200, &headers); - let header_str = String::from_utf8_lossy(&header); - // Should not contain the upstream content-length or transfer-encoding values - assert!(!header_str.contains("content-length: 9999")); - // Should contain exactly one transfer-encoding: chunked (ours) - assert_eq!(header_str.matches("transfer-encoding: chunked").count(), 1); - } - - #[test] - fn format_chunk_basic() { - let data = b"hello"; - let chunk = format_chunk(data); - assert_eq!(chunk, b"5\r\nhello\r\n"); - } - - #[test] - fn format_chunk_empty() { - // Empty chunk is NOT the terminator — it's a zero-length data segment - let chunk = format_chunk(b""); - assert_eq!(chunk, b"0\r\n\r\n"); - } - - #[test] - fn format_chunk_terminator_value() { - assert_eq!(format_chunk_terminator(), b"0\r\n\r\n"); - } - - #[test] - fn format_chunk_large_hex() { - let data = vec![0x41u8; 256]; // 0x100 bytes - let chunk = format_chunk(&data); - assert!(chunk.starts_with(b"100\r\n")); - assert!(chunk.ends_with(b"\r\n")); - assert_eq!(chunk.len(), 3 + 2 + 256 + 2); // "100" + \r\n + data + \r\n - } - - // ---- SEC-010: parse_chunked_body resource limits ---- - - #[test] - fn parse_chunked_multi_chunk_body() { - // Two chunks: 5 bytes + 6 bytes - let request = b"POST /v1/chat HTTP/1.1\r\nTransfer-Encoding: chunked\r\n\r\n5\r\nhello\r\n6\r\n world\r\n0\r\n\r\n"; - let ParseResult::Complete(parsed, _) = try_parse_http_request(request) else { - panic!("expected Complete"); - }; - assert_eq!(parsed.body, b"hello world"); - } - - #[test] - fn parse_chunked_rejects_too_many_chunks() { - // Build a request with MAX_CHUNK_COUNT + 1 tiny chunks - let mut buf = Vec::new(); - buf.extend_from_slice(b"POST /v1/chat HTTP/1.1\r\nTransfer-Encoding: chunked\r\n\r\n"); - for _ in 0..=MAX_CHUNK_COUNT { - buf.extend_from_slice(b"1\r\nX\r\n"); - } - buf.extend_from_slice(b"0\r\n\r\n"); - assert!(matches!( - try_parse_http_request(&buf), - ParseResult::Incomplete - )); - } - - #[test] - fn parse_chunked_within_chunk_count_limit() { - // MAX_CHUNK_COUNT chunks should succeed - let mut buf = Vec::new(); - buf.extend_from_slice(b"POST /v1/chat HTTP/1.1\r\nTransfer-Encoding: chunked\r\n\r\n"); - for _ in 0..100 { - buf.extend_from_slice(b"1\r\nX\r\n"); - } - buf.extend_from_slice(b"0\r\n\r\n"); - let ParseResult::Complete(parsed, _) = try_parse_http_request(&buf) else { - panic!("expected Complete for 100 chunks"); - }; - assert_eq!(parsed.body.len(), 100); - } - - /// SEC: Transfer-Encoding substring match must not match partial tokens. - #[test] - fn te_substring_not_chunked() { - let body = r#"{"model":"m","messages":[]}"#; - let request = format!( - "POST /v1/chat/completions HTTP/1.1\r\n\ - Host: x\r\n\ - Transfer-Encoding: chunkedx\r\n\ - Content-Length: {}\r\n\ - \r\n{body}", - body.len(), - ); - let ParseResult::Complete(parsed, _) = try_parse_http_request(request.as_bytes()) else { - panic!("expected Complete for non-matching TE with valid CL"); - }; - assert_eq!(parsed.body.len(), body.len()); - } - - // ---- SEC: Content-Length validation ---- - - #[test] - fn reject_differing_duplicate_content_length() { - let request = b"POST /v1/chat/completions HTTP/1.1\r\nHost: x\r\nContent-Length: 0\r\nContent-Length: 50\r\n\r\n"; - assert!(matches!( - try_parse_http_request(request), - ParseResult::Invalid(reason) if reason.contains("differing values") - )); - } - - #[test] - fn accept_identical_duplicate_content_length() { - let request = b"POST /v1/chat/completions HTTP/1.1\r\nHost: x\r\nContent-Length: 5\r\nContent-Length: 5\r\n\r\nhello"; - let ParseResult::Complete(parsed, _) = try_parse_http_request(request) else { - panic!("expected Complete for identical duplicate CL"); - }; - assert_eq!(parsed.body, b"hello"); - } - - #[test] - fn reject_non_numeric_content_length() { - let request = - b"POST /v1/chat/completions HTTP/1.1\r\nHost: x\r\nContent-Length: abc\r\n\r\n"; - assert!(matches!( - try_parse_http_request(request), - ParseResult::Invalid(reason) if reason.contains("invalid Content-Length") - )); - } - - #[test] - fn reject_two_non_numeric_content_lengths() { - let request = b"POST /v1/chat/completions HTTP/1.1\r\nHost: x\r\nContent-Length: abc\r\nContent-Length: def\r\n\r\n"; - assert!(matches!( - try_parse_http_request(request), - ParseResult::Invalid(_) - )); - } - - // ---- SEC-009: CL/TE desynchronisation ---- - - /// Reject requests with both Content-Length and Transfer-Encoding to - /// prevent CL/TE request smuggling (RFC 7230 Section 3.3.3). - #[test] - fn reject_dual_content_length_and_transfer_encoding() { - let request = b"POST /v1/chat/completions HTTP/1.1\r\nHost: x\r\nContent-Length: 5\r\nTransfer-Encoding: chunked\r\n\r\n"; - assert!( - matches!( - try_parse_http_request(request), - ParseResult::Invalid(reason) - if reason.contains("Transfer-Encoding") - && reason.contains("Content-Length") - ), - "Must reject request with both CL and TE" - ); - } - - /// Same rejection regardless of header order. - #[test] - fn reject_dual_transfer_encoding_and_content_length() { - let request = b"POST /v1/chat/completions HTTP/1.1\r\nHost: x\r\nTransfer-Encoding: chunked\r\nContent-Length: 5\r\n\r\n"; - assert!( - matches!( - try_parse_http_request(request), - ParseResult::Invalid(reason) - if reason.contains("Transfer-Encoding") - && reason.contains("Content-Length") - ), - "Must reject request with both TE and CL" - ); - } - - #[test] - fn format_sse_error_produces_valid_sse_json() { - let output = format_sse_error("chunk idle timeout exceeded"); - let text = std::str::from_utf8(&output).expect("should be valid utf8"); - - // Must start with "data: " (SSE format) - assert!(text.starts_with("data: "), "must be an SSE data line"); - - // Must end with double newline (SSE event boundary) - assert!(text.ends_with("\n\n"), "must end with SSE event boundary"); - - // The JSON payload between "data: " and "\n\n" must parse - let json_str = text.trim_start_matches("data: ").trim_end(); - let parsed: serde_json::Value = serde_json::from_str(json_str).expect("must be valid JSON"); - - assert_eq!(parsed["error"]["type"], "proxy_stream_error"); - assert_eq!(parsed["error"]["message"], "chunk idle timeout exceeded"); - } - - #[test] - fn format_sse_error_escapes_quotes_in_reason() { - let output = format_sse_error("error: \"bad\" response"); - let text = std::str::from_utf8(&output).unwrap(); - let json_str = text.trim_start_matches("data: ").trim_end(); - let parsed: serde_json::Value = - serde_json::from_str(json_str).expect("must produce valid JSON with escaped quotes"); - assert_eq!(parsed["error"]["message"], "error: \"bad\" response"); - } - - #[test] - fn format_sse_error_escapes_control_characters_in_reason() { - // A future caller passing a dynamic upstream error message (containing - // \n, \r, or \t — common in connection-reset errors and tracebacks) - // must still produce parseable SSE JSON. - let output = format_sse_error("upstream error: connection\nreset\tafter 0 bytes"); - let text = std::str::from_utf8(&output).unwrap(); - let json_str = text.trim_start_matches("data: ").trim_end(); - let parsed: serde_json::Value = serde_json::from_str(json_str) - .expect("must produce valid JSON when reason contains control characters"); - assert_eq!( - parsed["error"]["message"], - "upstream error: connection\nreset\tafter 0 bytes" - ); - } - - #[test] - fn format_sse_error_does_not_inject_extra_sse_events() { - // SSE events are separated by `\n\n`. If the reason string contains - // `\n\n`, an unescaped formatter would split the single error event - // into two SSE frames, allowing a malicious upstream to inject a - // forged event into the client's perceived stream - // (e.g. a fake tool_call delta). - let output = format_sse_error( - "safe prefix\n\ndata: {\"choices\":[{\"delta\":{\"tool_calls\":[{\"id\":\"FORGED\"}]}}]}", - ); - let text = std::str::from_utf8(&output).unwrap(); - - // Exactly one SSE event boundary (the trailing one) — the reason - // string must not introduce additional `\n\n` sequences. - let boundary_count = text.matches("\n\n").count(); - assert_eq!( - boundary_count, 1, - "format_sse_error must emit exactly one SSE event boundary; \ - reason string must not be able to inject extra events" - ); - } -} diff --git a/crates/openshell-supervisor-network/src/l7/jsonrpc.rs b/crates/openshell-supervisor-network/src/l7/jsonrpc.rs index 8469906a0..aea53edc0 100644 --- a/crates/openshell-supervisor-network/src/l7/jsonrpc.rs +++ b/crates/openshell-supervisor-network/src/l7/jsonrpc.rs @@ -86,6 +86,54 @@ pub(crate) async fn parse_jsonrpc_http_request Result { + if jsonrpc_receive_stream_request(request) { + return Ok(JsonRpcRequestInfo::receive_stream()); + } + + let header_end = request + .raw_header + .windows(4) + .position(|window| window == b"\r\n\r\n") + .ok_or_else(|| miette::miette!("HTTP request headers are missing the CRLF terminator"))? + + 4; + let body = &request.raw_header[header_end..]; + match request.body_length { + crate::l7::provider::BodyLength::None if body.is_empty() => {} + crate::l7::provider::BodyLength::ContentLength(length) => { + let length = usize::try_from(length) + .map_err(|_| miette::miette!("HTTP request body length exceeds platform limit"))?; + if body.len() != length { + return Err(miette::miette!( + "buffered HTTP request body length does not match Content-Length" + )); + } + } + crate::l7::provider::BodyLength::None => { + return Err(miette::miette!( + "buffered HTTP request has bytes without body framing" + )); + } + crate::l7::provider::BodyLength::Chunked => { + return Err(miette::miette!( + "buffered JSON-RPC request retained chunked framing" + )); + } + } + + Ok(parse_jsonrpc_body_with_options(body, inspection_options)) +} + #[derive(Debug, Clone, PartialEq, Eq)] pub struct JsonRpcRequestInfo { /// Calls found in the request body. Responses and receive-stream GETs have @@ -169,6 +217,12 @@ pub struct JsonRpcCallInfo { /// MCP `tools/call` tool name when known. Generic JSON-RPC leaves this /// unset because params are not inspected. pub tool: Option, + /// Whether this call is a JSON-RPC notification without an `id`. + /// + /// MCP initialization is a request, so transport code uses this bit to + /// avoid treating an extension notification named `initialize` as the + /// header-exempt initialization exchange. + pub is_notification: bool, } impl JsonRpcRequestInfo { @@ -252,7 +306,23 @@ pub fn parse_jsonrpc_body_with_options( let mut has_response = false; for item in &items { match parse_jsonrpc_message(item, inspection_options) { - Ok(JsonRpcMessageInfo::Call(call)) => calls.push(call), + Ok(JsonRpcMessageInfo::Call(call)) => { + // Initialization establishes the MCP exchange before any + // profile-specific request rules apply. A batch has no + // single bootstrap exchange, so fail closed before policy + // evaluation can forward any member. + if inspection_options.mode == JsonRpcInspectionMode::Mcp + && call.method == "initialize" + { + return JsonRpcRequestInfo::rejected( + true, + JsonRpcInspectionError::invalid_message( + "MCP initialize must not appear in a batch", + ), + ); + } + calls.push(call); + } Ok(JsonRpcMessageInfo::Response) => has_response = true, Err(error) => { return JsonRpcRequestInfo::rejected( @@ -349,6 +419,7 @@ fn parse_jsonrpc_call( method: method.to_string(), params: HashMap::new(), tool: None, + is_notification: value.get("id").is_none(), }) } @@ -405,6 +476,7 @@ fn parse_mcp_call( method: mcp_request.method_name().to_string(), params, tool, + is_notification: false, }); } @@ -425,6 +497,7 @@ fn parse_mcp_call( method: notification.method, params: HashMap::new(), tool: None, + is_notification: true, }) } @@ -806,6 +879,39 @@ mod tests { assert_eq!(info.calls[1].tool.as_deref(), Some("search_web")); } + #[test] + fn mcp_initialize_is_not_valid_in_a_batch() { + let body = br#"[ + {"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-11-25","capabilities":{},"clientInfo":{"name":"test","version":"1"}}}, + {"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"read_status","arguments":{}}} + ]"#; + let info = parse_jsonrpc_body(body, JsonRpcInspectionMode::Mcp); + + assert!(info.is_batch); + assert!(info.calls.is_empty()); + assert!(!info.has_response); + assert_eq!( + info.error.as_ref().map(JsonRpcInspectionError::detail), + Some("MCP initialize must not appear in a batch") + ); + } + + #[test] + fn generic_jsonrpc_keeps_batch_initialize_behavior() { + let body = br#"[ + {"jsonrpc":"2.0","id":1,"method":"initialize","params":{}}, + {"jsonrpc":"2.0","id":2,"method":"reports.list","params":{}} + ]"#; + let info = parse_jsonrpc_body(body, JsonRpcInspectionMode::JsonRpc); + + assert!( + info.error.is_none(), + "generic batch should remain valid: {info:?}" + ); + assert!(info.is_batch); + assert_eq!(info.calls.len(), 2); + } + #[test] fn parses_batch_with_calls_and_responses() { let body = br#"[ diff --git a/crates/openshell-supervisor-network/src/l7/mcp.rs b/crates/openshell-supervisor-network/src/l7/mcp.rs new file mode 100644 index 000000000..d5fbaed8b --- /dev/null +++ b/crates/openshell-supervisor-network/src/l7/mcp.rs @@ -0,0 +1,318 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! MCP Streamable HTTP request-version selection. + +use openshell_core::mcp::McpProtocolVersion; + +use crate::l7::jsonrpc::JsonRpcRequestInfo; +use crate::l7::provider::L7Request; + +const MCP_PROTOCOL_VERSION_HEADER: &str = "mcp-protocol-version"; + +/// Protocol revision selected for one MCP HTTP request. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(super) enum McpRequestProtocolVersion { + /// A valid standalone initialize request selects its revision in the JSON-RPC body. + Initialization, + /// A subsequent request selected an exact revision from its header or the legacy fallback. + Selected(McpProtocolVersion), +} + +/// Failure to select a policy-allowed protocol revision for an MCP HTTP request. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(super) enum McpProtocolVersionError { + /// The HTTP header block could not yield one unambiguous end-to-end value. + InvalidHeader, + /// The header value is not an MCP revision supported by this `OpenShell` build. + UnsupportedHeaderValue, + /// The selected supported revision is absent from the endpoint allowlist. + NotAllowed(McpProtocolVersion), +} + +impl McpProtocolVersionError { + /// Return the HTTP status for this transport or policy rejection. + #[must_use] + pub(super) const fn http_status(self) -> &'static str { + match self { + Self::InvalidHeader | Self::UnsupportedHeaderValue => "400 Bad Request", + Self::NotAllowed(_) => "403 Forbidden", + } + } + + /// Return a stable machine-readable response code. + #[must_use] + pub(super) const fn response_code(self) -> &'static str { + match self { + Self::InvalidHeader => "invalid_mcp_protocol_version_header", + Self::UnsupportedHeaderValue => "unsupported_mcp_protocol_version", + Self::NotAllowed(_) => "mcp_protocol_version_not_allowed", + } + } +} + +impl std::fmt::Display for McpProtocolVersionError { + fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + match self { + Self::InvalidHeader => formatter.write_str( + "MCP-Protocol-Version must contain one non-empty end-to-end header value", + ), + Self::UnsupportedHeaderValue => { + formatter.write_str("MCP-Protocol-Version names an unsupported protocol version") + } + Self::NotAllowed(version) => write!( + formatter, + "MCP protocol version {version} is not allowed by endpoint policy" + ), + } + } +} + +impl std::error::Error for McpProtocolVersionError {} + +/// Select and authorize the protocol revision for one MCP HTTP request. +/// +/// MCP initialization negotiates its revision in the JSON-RPC body and is the +/// only request exempt from the header. Every other request is self-contained: +/// an absent header selects the specification-defined `2025-03-26` fallback, +/// and the resulting supported revision must appear in the endpoint allowlist. +pub(super) fn select_request_protocol_version( + request: &L7Request, + info: &JsonRpcRequestInfo, + allowed_versions: &[McpProtocolVersion], +) -> Result { + if is_standalone_initialize(info) { + return Ok(McpRequestProtocolVersion::Initialization); + } + + let version = match request_protocol_version_header(&request.raw_header)? { + Some(value) => value + .parse::() + .map_err(|_| McpProtocolVersionError::UnsupportedHeaderValue)?, + None => McpProtocolVersion::V2025_03_26, + }; + if !allowed_versions.contains(&version) { + return Err(McpProtocolVersionError::NotAllowed(version)); + } + + Ok(McpRequestProtocolVersion::Selected(version)) +} + +fn is_standalone_initialize(info: &JsonRpcRequestInfo) -> bool { + !info.is_batch + && !info.has_response + && info.error.is_none() + && matches!( + info.calls.as_slice(), + [call] if call.method == "initialize" && !call.is_notification + ) +} + +fn request_protocol_version_header( + raw_header: &[u8], +) -> Result, McpProtocolVersionError> { + let header_end = raw_header + .windows(4) + .position(|window| window == b"\r\n\r\n") + .ok_or(McpProtocolVersionError::InvalidHeader)? + + 4; + let headers = std::str::from_utf8(&raw_header[..header_end]) + .map_err(|_| McpProtocolVersionError::InvalidHeader)?; + // Forwarding removes Connection-nominated fields. A revision field must + // survive that cleanup. Use the forwarding parser's canonical field names + // so authorization and removal agree, including after middleware rebuilds. + let nominated = crate::l7::rest::connection_nominated_header_names(&raw_header[..header_end]) + .map_err(|_| McpProtocolVersionError::InvalidHeader)?; + if nominated.contains(MCP_PROTOCOL_VERSION_HEADER) { + return Err(McpProtocolVersionError::InvalidHeader); + } + let mut values = headers.split("\r\n").skip(1).filter_map(|line| { + let (name, value) = line.split_once(':')?; + // HTTP field-value optional whitespace is only SP or HTAB. Using + // Unicode whitespace trimming here would accept bytes that are part + // of the protocol-version value rather than HTTP framing. + name.eq_ignore_ascii_case(MCP_PROTOCOL_VERSION_HEADER) + .then_some(value.trim_matches([' ', '\t'])) + }); + let Some(value) = values.next() else { + return Ok(None); + }; + if value.is_empty() || values.next().is_some() { + return Err(McpProtocolVersionError::InvalidHeader); + } + Ok(Some(value)) +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::l7::jsonrpc::{JsonRpcInspectionMode, parse_jsonrpc_body}; + use crate::l7::provider::BodyLength; + + fn request(method: &str, headers: &str) -> L7Request { + L7Request { + action: method.to_string(), + target: "/mcp".to_string(), + query_params: std::collections::HashMap::new(), + raw_header: format!("{method} /mcp HTTP/1.1\r\nHost: example.test\r\n{headers}\r\n") + .into_bytes(), + body_length: BodyLength::None, + } + } + + fn request_info(body: &[u8]) -> JsonRpcRequestInfo { + parse_jsonrpc_body(body, JsonRpcInspectionMode::Mcp) + } + + #[test] + fn standalone_initialize_uses_body_negotiation_only() { + let info = request_info( + br#"{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-11-25","capabilities":{},"clientInfo":{"name":"test","version":"1"}}}"#, + ); + + assert_eq!( + select_request_protocol_version(&request("POST", ""), &info, &[]), + Ok(McpRequestProtocolVersion::Initialization) + ); + } + + #[test] + fn initialize_notification_is_not_treated_as_initialization() { + let info = request_info(br#"{"jsonrpc":"2.0","method":"initialize","params":{}}"#); + + assert_eq!( + select_request_protocol_version( + &request("POST", ""), + &info, + &[McpProtocolVersion::V2025_03_26] + ), + Ok(McpRequestProtocolVersion::Selected( + McpProtocolVersion::V2025_03_26 + )) + ); + } + + #[test] + fn subsequent_requests_select_exact_header_for_every_http_method() { + let info = request_info(br#"{"jsonrpc":"2.0","id":1,"method":"tools/list"}"#); + + for method in ["POST", "GET", "DELETE"] { + assert_eq!( + select_request_protocol_version( + &request(method, "MCP-Protocol-Version: 2025-11-25\r\n"), + &info, + &[McpProtocolVersion::V2025_11_25] + ), + Ok(McpRequestProtocolVersion::Selected( + McpProtocolVersion::V2025_11_25 + )), + "method {method} must use the same per-request selection" + ); + } + } + + #[test] + fn missing_header_uses_only_the_legacy_specification_fallback() { + let info = request_info(br#"{"jsonrpc":"2.0","id":1,"method":"tools/list"}"#); + + assert_eq!( + select_request_protocol_version( + &request("POST", ""), + &info, + &[McpProtocolVersion::V2025_03_26] + ), + Ok(McpRequestProtocolVersion::Selected( + McpProtocolVersion::V2025_03_26 + )) + ); + assert_eq!( + select_request_protocol_version( + &request("POST", ""), + &info, + &[McpProtocolVersion::V2025_11_25] + ), + Err(McpProtocolVersionError::NotAllowed( + McpProtocolVersion::V2025_03_26 + )) + ); + } + + #[test] + fn repeated_or_empty_headers_are_bad_requests() { + let info = request_info(br#"{"jsonrpc":"2.0","id":1,"method":"tools/list"}"#); + + for headers in [ + "MCP-Protocol-Version:\r\n", + "MCP-Protocol-Version: 2025-11-25\r\nMCP-Protocol-Version: 2025-11-25\r\n", + "MCP-Protocol-Version: 2025-03-26\r\nmcp-protocol-version: 2025-11-25\r\n", + ] { + assert_eq!( + select_request_protocol_version( + &request("POST", headers), + &info, + &[McpProtocolVersion::V2025_11_25] + ), + Err(McpProtocolVersionError::InvalidHeader) + ); + } + } + + #[test] + fn protocol_version_must_remain_an_end_to_end_field() { + for connection_options in [ + "mcp-protocol-version", + "keep-alive, MCP-Protocol-Version", + "\tMcp-Protocol-Version\t, close", + ] { + let headers = request("POST", &format!("Connection: {connection_options}\r\n")); + assert_eq!( + request_protocol_version_header(&headers.raw_header), + Err(McpProtocolVersionError::InvalidHeader), + "revision metadata cannot be declared hop-by-hop" + ); + } + let headers = request("POST", "Connection: keep-alive, x-request-id\r\n"); + assert_eq!( + request_protocol_version_header(&headers.raw_header), + Ok(None) + ); + } + + #[test] + fn unsupported_or_non_exact_header_values_are_bad_requests() { + let info = request_info(br#"{"jsonrpc":"2.0","id":1,"method":"tools/list"}"#); + + for value in [ + "2026-07-28", + "2025-11-25, 2025-03-26", + "2025-11-25x", + "\u{00a0}2025-11-25", + ] { + assert_eq!( + select_request_protocol_version( + &request("POST", &format!("MCP-Protocol-Version: {value}\r\n")), + &info, + &[McpProtocolVersion::V2025_11_25] + ), + Err(McpProtocolVersionError::UnsupportedHeaderValue) + ); + } + } + + #[test] + fn supported_but_disallowed_header_is_forbidden() { + let info = request_info(br#"{"jsonrpc":"2.0","id":1,"method":"tools/list"}"#); + + let error = select_request_protocol_version( + &request("POST", "MCP-Protocol-Version: 2025-06-18\r\n"), + &info, + &[McpProtocolVersion::V2025_11_25], + ) + .expect_err("supported revision is outside endpoint policy"); + assert_eq!( + error, + McpProtocolVersionError::NotAllowed(McpProtocolVersion::V2025_06_18) + ); + assert_eq!(error.http_status(), "403 Forbidden"); + } +} diff --git a/crates/openshell-supervisor-network/src/l7/mod.rs b/crates/openshell-supervisor-network/src/l7/mod.rs index 70a980ba2..77e4497d4 100644 --- a/crates/openshell-supervisor-network/src/l7/mod.rs +++ b/crates/openshell-supervisor-network/src/l7/mod.rs @@ -10,8 +10,8 @@ pub mod graphql; pub(crate) mod http; -pub mod inference; pub mod jsonrpc; +pub(crate) mod mcp; pub(crate) mod middleware; pub mod path; pub mod provider; @@ -21,6 +21,7 @@ pub mod tls; pub(crate) mod token_grant_injection; pub(crate) mod websocket; +use openshell_core::mcp::McpProtocolVersion; pub use openshell_policy::L7Protocol; use openshell_policy::{ L7EndpointFields, validate_explicit_tcp_additional_fields, validate_l7_endpoint_semantics, @@ -119,6 +120,10 @@ pub struct L7EndpointConfig { /// MCP-only strict validation for tools/call params.name. Defaults to true /// for MCP endpoints and is ignored by other JSON-RPC-family protocols. pub mcp_strict_tool_names: bool, + /// Canonical MCP protocol revisions allowed by this endpoint. + /// + /// Non-MCP endpoints always carry an empty list. + pub mcp_versions: Vec, /// When true, percent-encoded `/` (`%2F`) is preserved in path segments /// rather than rejected at the parser. Needed by upstreams like GitLab /// that embed `%2F` in namespaced project paths. Defaults to false. @@ -172,7 +177,8 @@ pub struct L7RequestInfo { /// Parse an L7 endpoint config from a regorus Value (returned by Rego query). /// /// The value is expected to be the raw endpoint object from the Rego data, -/// containing fields: `protocol`, optionally `tls`, `enforcement`. +/// containing fields: `protocol`, optionally `tls`, `enforcement`, and the +/// canonical `mcp_versions` array for MCP endpoints. pub fn parse_l7_config(val: ®orus::Value) -> Option { let protocol_val = get_object_str(val, "protocol")?; let protocol = L7Protocol::parse(&protocol_val)?; @@ -231,6 +237,11 @@ pub fn parse_l7_config(val: ®orus::Value) -> Option { .unwrap_or(jsonrpc::DEFAULT_MAX_BODY_BYTES); let mcp_strict_tool_names = protocol == L7Protocol::Mcp && get_object_bool(val, "mcp_strict_tool_names").unwrap_or(true); + let mcp_versions = if protocol == L7Protocol::Mcp { + parse_canonical_mcp_versions(val)? + } else { + Vec::new() + }; let credential_signing = match get_object_str(val, "credential_signing").as_deref() { Some("sigv4") => CredentialSigning::SigV4, @@ -271,6 +282,7 @@ pub fn parse_l7_config(val: ®orus::Value) -> Option { graphql_max_body_bytes, json_rpc_max_body_bytes, mcp_strict_tool_names, + mcp_versions, allow_encoded_slash, websocket_credential_rewrite, request_body_credential_rewrite, @@ -402,6 +414,27 @@ fn get_object_str(val: ®orus::Value, key: &str) -> Option { } } +fn parse_canonical_mcp_versions(val: ®orus::Value) -> Option> { + let regorus::Value::Array(values) = get_object_value(val, "mcp_versions")? else { + return None; + }; + let versions = values + .iter() + .map(|value| match value { + regorus::Value::String(value) => value.parse::().ok(), + _ => None, + }) + .collect::>>()?; + + // Both policy ingress paths promise a non-empty, strictly ordered list. + // Rechecking that runtime contract prevents a dropped or corrupted + // projection from silently selecting a different wire profile. + if versions.is_empty() || !versions.windows(2).all(|pair| pair[0] < pair[1]) { + return None; + } + Some(versions) +} + fn endpoint_has_graphql_policy(val: ®orus::Value) -> bool { has_non_empty_object_field(val, "graphql_persisted_queries") || has_graphql_persisted_query_mode(val) @@ -1252,6 +1285,7 @@ pub fn validate_l7_policies(data_json: &serde_json::Value) -> (Vec, Vec< "{loc}: JSON-RPC-specific endpoint fields are ignored unless protocol is json-rpc or mcp" )); } + validate_mcp_versions_field(&mut errors, &loc, ep, l7_protocol); let has_mcp_strict_tool_names = ep.get("mcp_strict_tool_names").is_some(); let has_mcp_allow_all_known_mcp_methods = ep.get("mcp_allow_all_known_mcp_methods").is_some(); @@ -1571,6 +1605,55 @@ pub fn validate_l7_policies(data_json: &serde_json::Value) -> (Vec, Vec< (errors, warnings) } +fn validate_mcp_versions_field( + errors: &mut Vec, + loc: &str, + endpoint: &serde_json::Value, + protocol: Option, +) { + let Some(value) = endpoint.get("mcp_versions") else { + return; + }; + if protocol != Some(L7Protocol::Mcp) { + errors.push(format!( + "{loc}: mcp.versions is only valid for protocol mcp" + )); + return; + } + let Some(values) = value.as_array() else { + errors.push(format!("{loc}: mcp.versions must be an array")); + return; + }; + if values.is_empty() { + errors.push(format!("{loc}: mcp.versions must not be empty")); + return; + } + + let mut versions = Vec::with_capacity(values.len()); + for value in values { + let Some(value) = value.as_str() else { + errors.push(format!("{loc}: mcp.versions entries must be strings")); + return; + }; + let Ok(version) = value.parse::() else { + errors.push(format!( + "{loc}: mcp.versions contains an unsupported protocol version" + )); + return; + }; + versions.push(version); + } + + // Runtime ingress has already normalized the allowlist. Requiring strict + // order here catches duplicates and bypasses that could otherwise fail + // later by removing L7 inspection from the selected route. + if !versions.windows(2).all(|pair| pair[0] < pair[1]) { + errors.push(format!( + "{loc}: mcp.versions must be unique and in canonical order" + )); + } +} + /// Map a supported L7 `access` preset to explicit rules for `protocol`. /// /// Returns `None` when `access` is not `read-only`, `read-write`, or `full`. @@ -1884,7 +1967,7 @@ mod tests { #[test] fn parse_l7_config_mcp_strict_tool_names_defaults_true() { let val = regorus::Value::from_json_str( - r#"{"protocol": "mcp", "host": "mcp.example.com", "port": 443}"#, + r#"{"protocol": "mcp", "host": "mcp.example.com", "port": 443, "mcp_versions": ["2025-11-25"]}"#, ) .unwrap(); let config = parse_l7_config(&val).unwrap(); @@ -1894,13 +1977,53 @@ mod tests { #[test] fn parse_l7_config_mcp_strict_tool_names_can_disable() { let val = regorus::Value::from_json_str( - r#"{"protocol": "mcp", "host": "mcp.example.com", "port": 443, "mcp_strict_tool_names": false}"#, + r#"{"protocol": "mcp", "host": "mcp.example.com", "port": 443, "mcp_strict_tool_names": false, "mcp_versions": ["2025-11-25"]}"#, ) .unwrap(); let config = parse_l7_config(&val).unwrap(); assert!(!config.mcp_strict_tool_names); } + #[test] + fn parse_l7_config_requires_canonical_mcp_versions() { + let explicit = regorus::Value::from_json_str( + r#"{"protocol": "mcp", "mcp_versions": ["2025-03-26", "2025-11-25"]}"#, + ) + .unwrap(); + assert_eq!( + parse_l7_config(&explicit).unwrap().mcp_versions, + vec![ + McpProtocolVersion::V2025_03_26, + McpProtocolVersion::V2025_11_25 + ] + ); + + for invalid in [ + r#"{"protocol": "mcp"}"#, + r#"{"protocol": "mcp", "mcp_versions": []}"#, + r#"{"protocol": "mcp", "mcp_versions": ["2026-01-01"]}"#, + r#"{"protocol": "mcp", "mcp_versions": [1]}"#, + r#"{"protocol": "mcp", "mcp_versions": ["2025-11-25", "2025-11-25"]}"#, + r#"{"protocol": "mcp", "mcp_versions": ["2025-11-25", "2025-03-26"]}"#, + ] { + let value = regorus::Value::from_json_str(invalid).unwrap(); + assert!( + parse_l7_config(&value).is_none(), + "non-canonical MCP runtime config must be rejected: {invalid}" + ); + } + } + + #[test] + fn parse_l7_config_keeps_mcp_versions_off_other_protocols() { + let value = regorus::Value::from_json_str( + r#"{"protocol": "json-rpc", "mcp_versions": ["2025-11-25"]}"#, + ) + .unwrap(); + + assert!(parse_l7_config(&value).unwrap().mcp_versions.is_empty()); + } + #[test] fn parse_l7_config_websocket_credential_rewrite_defaults_false() { let val = regorus::Value::from_json_str( diff --git a/crates/openshell-supervisor-network/src/l7/relay.rs b/crates/openshell-supervisor-network/src/l7/relay.rs index 2697fedb3..d022622a3 100644 --- a/crates/openshell-supervisor-network/src/l7/relay.rs +++ b/crates/openshell-supervisor-network/src/l7/relay.rs @@ -164,6 +164,93 @@ where Ok(()) } +/// Enforce MCP request-version policy and emit a transport or policy rejection. +/// Non-MCP adapters share this entry point without changing their behavior. +pub(crate) async fn enforce_mcp_protocol_version( + config: &L7EndpointConfig, + request: &crate::l7::provider::L7Request, + info: &crate::l7::jsonrpc::JsonRpcRequestInfo, + client: &mut W, + ctx: &L7EvalContext, + redacted_target: &str, +) -> Result +where + W: AsyncWrite + Unpin, +{ + if config.protocol != L7Protocol::Mcp { + return Ok(true); + } + + match crate::l7::mcp::select_request_protocol_version(request, info, &config.mcp_versions) { + Ok(crate::l7::mcp::McpRequestProtocolVersion::Initialization) => Ok(true), + Ok(crate::l7::mcp::McpRequestProtocolVersion::Selected(version)) => { + debug!(mcp_protocol_version = %version, "Selected MCP request protocol version"); + Ok(true) + } + Err(error) => { + let reason = error.to_string(); + let summary = l7_protocol_log_summary(None, Some(info)); + ocsf_emit!(build_l7_request_event( + ctx, + &request.action, + redacted_target, + "deny", + "l7-mcp", + &reason, + summary.as_deref(), + )); + let deny_group = match error { + crate::l7::mcp::McpProtocolVersionError::NotAllowed(_) => "l7_policy", + crate::l7::mcp::McpProtocolVersionError::InvalidHeader + | crate::l7::mcp::McpProtocolVersionError::UnsupportedHeaderValue => { + "l7_parse_rejection" + } + }; + emit_activity(ctx, true, deny_group); + + let body = serde_json::json!({ + "error": error.response_code(), + "detail": reason, + "policy": ctx.policy_name, + "layer": "l7", + "protocol": "mcp", + "method": request.action, + "path": redacted_target, + }); + crate::l7::rest::send_json_response( + &ctx.policy_name, + body, + client, + error.http_status(), + ) + .await?; + Ok(false) + } + } +} + +/// Reinspect the buffered outgoing MCP request after request transformations. +/// The forwarding adapter must call this before any upstream request write. +pub(crate) async fn enforce_final_mcp_protocol_version( + config: &L7EndpointConfig, + request: &crate::l7::provider::L7Request, + client: &mut W, + ctx: &L7EvalContext, + redacted_target: &str, +) -> Result +where + W: AsyncWrite + Unpin, +{ + if config.protocol != L7Protocol::Mcp { + return Ok(true); + } + let info = crate::l7::jsonrpc::inspect_buffered_jsonrpc_http_request( + request, + crate::l7::jsonrpc::JsonRpcInspectionOptions::for_config(config), + )?; + enforce_mcp_protocol_version(config, request, &info, client, ctx, redacted_target).await +} + fn build_request_authority_mismatch_event(ctx: &L7EvalContext) -> openshell_ocsf::OcsfEvent { HttpActivityBuilder::new(openshell_ocsf::ctx::ctx()) .activity(ActivityId::Fail) @@ -688,6 +775,12 @@ where graphql: graphql_info.clone(), jsonrpc: jsonrpc_info.clone(), }; + if let Some(info) = jsonrpc_info.as_ref() + && !enforce_mcp_protocol_version(config, &req, info, client, ctx, &redacted_target) + .await? + { + return Ok(()); + } let websocket_request = crate::l7::rest::request_is_websocket_upgrade(&req.raw_header); if config.protocol == L7Protocol::Websocket && !websocket_request { crate::l7::rest::RestProvider::default() @@ -790,6 +883,11 @@ where return Ok(()); } }; + if !enforce_final_mcp_protocol_version(config, &req, client, ctx, &redacted_target) + .await? + { + return Ok(()); + } let scoped_ctx = scoped_context_for_request(ctx, &req); let ctx = scoped_ctx.as_ref().unwrap_or(ctx); let mut middleware_session = if let Some(chain) = websocket_chain.as_deref() { @@ -855,7 +953,7 @@ where Ok(None) => { if let Some(session) = middleware_session.take() { session - .end(openshell_core::proto::WebSocketSessionEndReason::Cancellation) + .end(openshell_core::proto::MiddlewareSessionEndReason::Cancellation) .await; } return Ok(()); @@ -875,14 +973,14 @@ where RelayOutcome::Reusable => { if let Some(session) = middleware_session.take() { session - .end(openshell_core::proto::WebSocketSessionEndReason::UpstreamRejected) + .end(openshell_core::proto::MiddlewareSessionEndReason::UpstreamFailure) .await; } } RelayOutcome::Consumed => { if let Some(session) = middleware_session.take() { session - .end(openshell_core::proto::WebSocketSessionEndReason::UpstreamRejected) + .end(openshell_core::proto::MiddlewareSessionEndReason::UpstreamFailure) .await; } return Ok(()); @@ -1217,7 +1315,7 @@ where emit_policy_reload(guard, host, port, &options.policy_name); if let Some(session) = options.middleware_session.take() { session - .end(openshell_core::proto::WebSocketSessionEndReason::PolicyReload) + .end(openshell_core::proto::MiddlewareSessionEndReason::PolicyReload) .await; } send_websocket_close(client, upstream, 1012).await; @@ -1594,7 +1692,7 @@ where Ok(None) => { if let Some(session) = middleware_session.take() { session - .end(openshell_core::proto::WebSocketSessionEndReason::Cancellation) + .end(openshell_core::proto::MiddlewareSessionEndReason::Cancellation) .await; } return Ok(()); @@ -1614,14 +1712,14 @@ where RelayOutcome::Reusable => { if let Some(session) = middleware_session.take() { session - .end(openshell_core::proto::WebSocketSessionEndReason::UpstreamRejected) + .end(openshell_core::proto::MiddlewareSessionEndReason::UpstreamFailure) .await; } } RelayOutcome::Consumed => { if let Some(session) = middleware_session.take() { session - .end(openshell_core::proto::WebSocketSessionEndReason::UpstreamRejected) + .end(openshell_core::proto::MiddlewareSessionEndReason::UpstreamFailure) .await; } debug!( @@ -1720,7 +1818,7 @@ pub(crate) async fn finalize_websocket_pre_upgrade( emit_policy_reload(guard, host, port, policy_name); if let Some(session) = session.take() { session - .end(openshell_core::proto::WebSocketSessionEndReason::PolicyReload) + .end(openshell_core::proto::MiddlewareSessionEndReason::PolicyReload) .await; } Err(error) @@ -1731,9 +1829,9 @@ pub(crate) async fn finalize_websocket_pre_upgrade( Err(error) => { let reason = if guard.is_stale() { emit_policy_reload(guard, host, port, policy_name); - openshell_core::proto::WebSocketSessionEndReason::PolicyReload + openshell_core::proto::MiddlewareSessionEndReason::PolicyReload } else { - openshell_core::proto::WebSocketSessionEndReason::UpstreamRejected + openshell_core::proto::MiddlewareSessionEndReason::UpstreamFailure }; if let Some(session) = session.take() { session.end(reason).await; @@ -1759,9 +1857,6 @@ where return Ok(()); } - // Future MCP version-profile request checks should hook here before OPA - // evaluation. See McpOptions in proto/sandbox.proto for the policy - // roadmap and source documentation. let parsed = match crate::l7::jsonrpc::parse_jsonrpc_http_request( client, config.json_rpc_max_body_bytes, @@ -1817,6 +1912,11 @@ where graphql: None, jsonrpc: Some(jsonrpc_info.clone()), }; + if !enforce_mcp_protocol_version(config, &req, &jsonrpc_info, client, ctx, &redacted_target) + .await? + { + return Ok(()); + } let hard_deny_reason = l7_request_hard_deny_reason(config.protocol, &request_info); let force_deny = hard_deny_reason.is_some(); @@ -1928,6 +2028,11 @@ where return Ok(()); } }; + if !enforce_final_mcp_protocol_version(config, &req, client, ctx, &redacted_target) + .await? + { + return Ok(()); + } let scoped_ctx = scoped_context_for_request(ctx, &req); let ctx = scoped_ctx.as_ref().unwrap_or(ctx); // Future MCP response/SSE introspection or rewrite would hook here @@ -7357,6 +7462,7 @@ network_policies: graphql_max_body_bytes: 0, json_rpc_max_body_bytes: crate::l7::jsonrpc::DEFAULT_MAX_BODY_BYTES, mcp_strict_tool_names: true, + mcp_versions: Vec::new(), allow_encoded_slash, websocket_credential_rewrite: false, request_body_credential_rewrite: false, @@ -7731,7 +7837,7 @@ network_policies: .await }); - let scenario = tokio::time::timeout(std::time::Duration::from_secs(60), async { + let scenario = tokio::time::timeout(std::time::Duration::from_mins(1), async { app.write_all( b"GET /ws HTTP/1.1\r\nHost: api.example.test\r\nUpgrade: websocket\r\nConnection: Upgrade\r\nSec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==\r\nSec-WebSocket-Version: 13\r\nSec-WebSocket-Extensions: permessage-deflate; client_no_context_takeover\r\n\r\n", ) @@ -7801,6 +7907,7 @@ network_policies: graphql_max_body_bytes: 0, json_rpc_max_body_bytes: crate::l7::jsonrpc::DEFAULT_MAX_BODY_BYTES, mcp_strict_tool_names: true, + mcp_versions: Vec::new(), allow_encoded_slash: false, websocket_credential_rewrite: true, request_body_credential_rewrite: false, @@ -8003,6 +8110,7 @@ network_policies: graphql_max_body_bytes: 0, json_rpc_max_body_bytes: crate::l7::jsonrpc::DEFAULT_MAX_BODY_BYTES, mcp_strict_tool_names: true, + mcp_versions: Vec::new(), allow_encoded_slash: false, websocket_credential_rewrite: true, request_body_credential_rewrite: false, @@ -8129,6 +8237,7 @@ network_policies: graphql_max_body_bytes: 0, json_rpc_max_body_bytes: crate::l7::jsonrpc::DEFAULT_MAX_BODY_BYTES, mcp_strict_tool_names: true, + mcp_versions: Vec::new(), allow_encoded_slash: false, websocket_credential_rewrite: true, request_body_credential_rewrite: false, @@ -8570,6 +8679,213 @@ network_policies: .unwrap(); } + #[tokio::test] + async fn mcp_relay_forwards_standalone_initialize_without_version_header() { + let (config, tunnel_engine, ctx) = mcp_test_relay_context(); + let (mut app, mut relay_client) = tokio::io::duplex(8192); + let (mut relay_upstream, mut upstream) = tokio::io::duplex(8192); + let relay = tokio::spawn(async move { + relay_with_inspection( + &config, + tunnel_engine, + &mut relay_client, + &mut relay_upstream, + &ctx, + ) + .await + }); + + let body = br#"{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-11-25","capabilities":{},"clientInfo":{"name":"test","version":"1"}}}"#; + let request = format!( + "POST /mcp HTTP/1.1\r\nHost: mcp.example.test:8000\r\nContent-Type: application/json\r\nContent-Length: {}\r\nConnection: close\r\n\r\n", + body.len() + ); + app.write_all(request.as_bytes()).await.unwrap(); + app.write_all(body).await.unwrap(); + + let mut upstream_bytes = vec![0; 2048]; + let count = tokio::time::timeout( + std::time::Duration::from_secs(1), + upstream.read(&mut upstream_bytes), + ) + .await + .expect("standalone initialize should reach upstream") + .unwrap(); + let upstream_request = String::from_utf8_lossy(&upstream_bytes[..count]); + assert!(upstream_request.contains(r#""method":"initialize""#)); + + upstream + .write_all( + b"HTTP/1.1 200 OK\r\nContent-Length: 36\r\nConnection: close\r\n\r\n{\"jsonrpc\":\"2.0\",\"id\":1,\"result\":{}}", + ) + .await + .unwrap(); + let mut response = [0; 512]; + let count = + tokio::time::timeout(std::time::Duration::from_secs(1), app.read(&mut response)) + .await + .expect("initialize response should reach client") + .unwrap(); + assert!(String::from_utf8_lossy(&response[..count]).contains("200 OK")); + + drop(app); + tokio::time::timeout(std::time::Duration::from_secs(1), relay) + .await + .expect("relay should complete") + .unwrap() + .unwrap(); + } + + async fn run_rejected_mcp_version_request( + route_selected: bool, + version_headers: &str, + ) -> (String, Vec) { + let (config, tunnel_engine, ctx) = mcp_test_relay_context(); + let (mut app, mut relay_client) = tokio::io::duplex(8192); + let (mut relay_upstream, mut upstream) = tokio::io::duplex(8192); + let relay = tokio::spawn(async move { + if route_selected { + relay_with_route_selection( + &[config], + tunnel_engine, + &mut relay_client, + &mut relay_upstream, + &ctx, + ) + .await + } else { + relay_with_inspection( + &config, + tunnel_engine, + &mut relay_client, + &mut relay_upstream, + &ctx, + ) + .await + } + }); + + let body = br#"{"jsonrpc":"2.0","id":7,"result":{"ok":true}}"#; + let request = format!( + "POST /mcp HTTP/1.1\r\nHost: mcp.example.test:8000\r\nContent-Type: application/json\r\n{version_headers}Content-Length: {}\r\nConnection: close\r\n\r\n", + body.len() + ); + app.write_all(request.as_bytes()).await.unwrap(); + app.write_all(body).await.unwrap(); + + let mut response = Vec::new(); + tokio::time::timeout( + std::time::Duration::from_secs(1), + app.read_to_end(&mut response), + ) + .await + .expect("MCP version rejection should close the client response") + .unwrap(); + tokio::time::timeout(std::time::Duration::from_secs(1), relay) + .await + .expect("relay should complete after version rejection") + .unwrap() + .unwrap(); + + let mut forwarded = Vec::new(); + tokio::time::timeout( + std::time::Duration::from_secs(1), + upstream.read_to_end(&mut forwarded), + ) + .await + .expect("version rejection should close upstream without forwarding") + .unwrap(); + ( + String::from_utf8(response).expect("UTF-8 response"), + forwarded, + ) + } + + #[tokio::test] + async fn mcp_relay_rejects_invalid_disallowed_and_missing_versions_without_forwarding() { + for (headers, status, code) in [ + ( + "MCP-Protocol-Version: 2026-07-28\r\n", + "400 Bad Request", + "unsupported_mcp_protocol_version", + ), + ( + "MCP-Protocol-Version: 2025-11-25\r\nMCP-Protocol-Version: 2025-11-25\r\n", + "400 Bad Request", + "invalid_mcp_protocol_version_header", + ), + ( + "MCP-Protocol-Version: 2025-06-18\r\n", + "403 Forbidden", + "mcp_protocol_version_not_allowed", + ), + ("", "403 Forbidden", "mcp_protocol_version_not_allowed"), + ] { + let (response, forwarded) = run_rejected_mcp_version_request(false, headers).await; + assert!( + response.starts_with(&format!("HTTP/1.1 {status}")), + "{response}" + ); + assert!(response.contains(code), "{response}"); + assert!(forwarded.is_empty(), "rejected request reached upstream"); + } + } + + #[tokio::test] + async fn route_selected_mcp_relay_enforces_request_version_before_forwarding() { + let (response, forwarded) = + run_rejected_mcp_version_request(true, "MCP-Protocol-Version: 2026-07-28\r\n").await; + + assert!( + response.starts_with("HTTP/1.1 400 Bad Request"), + "{response}" + ); + assert!( + response.contains("unsupported_mcp_protocol_version"), + "{response}" + ); + assert!(forwarded.is_empty(), "rejected request reached upstream"); + } + + #[tokio::test] + async fn final_mcp_version_check_reclassifies_a_rewritten_initialize_body() { + let (config, _, ctx) = mcp_test_relay_context(); + let final_body = br#"{"jsonrpc":"2.0","id":2,"method":"tools/list"}"#; + let mut raw_header = format!( + "POST /mcp HTTP/1.1\r\nHost: mcp.example.test:8000\r\nContent-Type: application/json\r\nContent-Length: {}\r\nConnection: close\r\n\r\n", + final_body.len() + ) + .into_bytes(); + raw_header.extend_from_slice(final_body); + let request = crate::l7::provider::L7Request { + action: "POST".to_string(), + target: "/mcp".to_string(), + query_params: std::collections::HashMap::new(), + raw_header, + body_length: crate::l7::provider::BodyLength::ContentLength(final_body.len() as u64), + }; + let (mut client, mut relay_client) = tokio::io::duplex(2048); + + let allowed = + enforce_final_mcp_protocol_version(&config, &request, &mut relay_client, &ctx, "/mcp") + .await + .expect("final request inspection"); + assert!( + !allowed, + "rewritten non-initialize request must require a version" + ); + drop(relay_client); + + let mut response = Vec::new(); + client.read_to_end(&mut response).await.unwrap(); + let response = String::from_utf8(response).expect("UTF-8 response"); + assert!(response.starts_with("HTTP/1.1 403 Forbidden"), "{response}"); + assert!( + response.contains("mcp_protocol_version_not_allowed"), + "{response}" + ); + } + #[tokio::test] async fn mcp_relay_forwards_jsonrpc_response_frame() { let (config, tunnel_engine, ctx) = mcp_test_relay_context(); @@ -8588,7 +8904,7 @@ network_policies: let body = br#"{"jsonrpc":"2.0","id":7,"result":{"action":"accept","content":{}}}"#; let request = format!( - "POST /mcp HTTP/1.1\r\nHost: mcp.example.test:8000\r\nContent-Type: application/json\r\nContent-Length: {}\r\nConnection: close\r\n\r\n", + "POST /mcp HTTP/1.1\r\nHost: mcp.example.test:8000\r\nContent-Type: application/json\r\nMCP-Protocol-Version: 2025-11-25\r\nContent-Length: {}\r\nConnection: close\r\n\r\n", body.len() ); app.write_all(request.as_bytes()).await.unwrap(); diff --git a/crates/openshell-supervisor-network/src/l7/rest.rs b/crates/openshell-supervisor-network/src/l7/rest.rs index 93315a671..42bf3a33d 100644 --- a/crates/openshell-supervisor-network/src/l7/rest.rs +++ b/crates/openshell-supervisor-network/src/l7/rest.rs @@ -2639,7 +2639,8 @@ async fn send_forbidden_json( send_json_response(policy_name, body, client, "403 Forbidden").await } -async fn send_json_response( +/// Send a platform-owned JSON response with an explicit HTTP status. +pub(crate) async fn send_json_response( policy_name: &str, body: serde_json::Value, client: &mut C, diff --git a/crates/openshell-supervisor-network/src/l7/tls.rs b/crates/openshell-supervisor-network/src/l7/tls.rs index 2275a60d3..63e04089c 100644 --- a/crates/openshell-supervisor-network/src/l7/tls.rs +++ b/crates/openshell-supervisor-network/src/l7/tls.rs @@ -465,7 +465,6 @@ mod tests { #[test] fn upstream_config_alpn() { - let _ = rustls::crypto::ring::default_provider().install_default(); let config = build_upstream_client_config("").unwrap(); assert_eq!(config.alpn_protocols, vec![b"http/1.1".to_vec()]); } diff --git a/crates/openshell-supervisor-network/src/l7/websocket.rs b/crates/openshell-supervisor-network/src/l7/websocket.rs index 6cd8aa481..dab11a111 100644 --- a/crates/openshell-supervisor-network/src/l7/websocket.rs +++ b/crates/openshell-supervisor-network/src/l7/websocket.rs @@ -30,8 +30,8 @@ pub const MAX_QUEUED_WEBSOCKET_ASSEMBLIES: usize = MAX_CONCURRENT_WEBSOCKET_ASSE const MAX_RAW_FRAME_PAYLOAD_BYTES: u64 = 16 * 1024 * 1024; const MAX_MESSAGE_FRAGMENTS: usize = 4096; const TEXT_MESSAGE_ASSEMBLY_IDLE_TIMEOUT: StdDuration = StdDuration::from_secs(30); -const TEXT_MESSAGE_ASSEMBLY_TOTAL_TIMEOUT: StdDuration = StdDuration::from_secs(120); -const TEXT_MESSAGE_FORWARD_TOTAL_TIMEOUT: StdDuration = StdDuration::from_secs(120); +const TEXT_MESSAGE_ASSEMBLY_TOTAL_TIMEOUT: StdDuration = StdDuration::from_mins(2); +const TEXT_MESSAGE_FORWARD_TOTAL_TIMEOUT: StdDuration = StdDuration::from_mins(2); const COPY_BUF_SIZE: usize = 8192; const OPCODE_CONTINUATION: u8 = 0x0; const OPCODE_TEXT: u8 = 0x1; @@ -97,7 +97,8 @@ pub enum WebSocketAssemblyAdmissionOutcome { #[derive(Debug, Clone, Copy, PartialEq, Eq)] enum WebSocketTerminationCause { - PeerDisconnect, + DownstreamDisconnect, + UpstreamDisconnect, PolicyReload, CapacityExhausted, MiddlewareDenial, @@ -111,7 +112,7 @@ enum WebSocketTerminationCause { impl WebSocketTerminationCause { fn close_code(self) -> Option { match self { - Self::PeerDisconnect => None, + Self::DownstreamDisconnect | Self::UpstreamDisconnect => None, Self::PolicyReload => Some(1012), Self::CapacityExhausted => Some(1013), Self::MiddlewareDenial | Self::MiddlewareFailure | Self::PolicyDenial => Some(1008), @@ -121,21 +122,24 @@ impl WebSocketTerminationCause { } } - fn session_end_reason(self) -> openshell_core::proto::WebSocketSessionEndReason { + fn session_end_reason(self) -> openshell_core::proto::MiddlewareSessionEndReason { match self { - Self::PeerDisconnect => { - openshell_core::proto::WebSocketSessionEndReason::PeerDisconnect + Self::DownstreamDisconnect => { + openshell_core::proto::MiddlewareSessionEndReason::DownstreamDisconnect } - Self::PolicyReload => openshell_core::proto::WebSocketSessionEndReason::PolicyReload, + Self::UpstreamDisconnect => { + openshell_core::proto::MiddlewareSessionEndReason::UpstreamDisconnect + } + Self::PolicyReload => openshell_core::proto::MiddlewareSessionEndReason::PolicyReload, Self::MiddlewareDenial => { - openshell_core::proto::WebSocketSessionEndReason::MiddlewareDenial + openshell_core::proto::MiddlewareSessionEndReason::MiddlewareDenial } - Self::PolicyDenial => openshell_core::proto::WebSocketSessionEndReason::PolicyDenial, + Self::PolicyDenial => openshell_core::proto::MiddlewareSessionEndReason::PolicyDenial, Self::CapacityExhausted | Self::MiddlewareFailure => { - openshell_core::proto::WebSocketSessionEndReason::MiddlewareFailure + openshell_core::proto::MiddlewareSessionEndReason::MiddlewareFailure } Self::InvalidUtf8 | Self::ProtocolError | Self::MessageTooBig => { - openshell_core::proto::WebSocketSessionEndReason::ProtocolError + openshell_core::proto::MiddlewareSessionEndReason::ProtocolError } } } @@ -189,7 +193,8 @@ enum AssemblyTimeoutKind { #[derive(Debug, Clone, Copy, PartialEq, Eq)] enum FrameErrorKind { - PeerDisconnect, + DownstreamDisconnect, + UpstreamDisconnect, Protocol(FrameFailureClass), InvalidUtf8, MessageTooBig, @@ -209,20 +214,27 @@ impl std::fmt::Display for FrameError { } impl FrameError { - fn peer_io(context: &str, error: std::io::Error) -> Self { + fn client_io(error: std::io::Error) -> Self { Self { - kind: FrameErrorKind::PeerDisconnect, - error: miette!("{context}: {error}"), + kind: FrameErrorKind::DownstreamDisconnect, + error: miette!("websocket client read failed: {error}"), } } - fn peer_disconnect(error: miette::Report) -> Self { + fn client_disconnect(error: miette::Report) -> Self { Self { - kind: FrameErrorKind::PeerDisconnect, + kind: FrameErrorKind::DownstreamDisconnect, error, } } + fn upstream_io(context: &str, error: std::io::Error) -> Self { + Self { + kind: FrameErrorKind::UpstreamDisconnect, + error: miette!("{context}: {error}"), + } + } + fn protocol(failure_class: FrameFailureClass, error: miette::Report) -> Self { Self { kind: FrameErrorKind::Protocol(failure_class), @@ -263,7 +275,12 @@ impl FrameError { impl From for WebSocketTermination { fn from(frame_error: FrameError) -> Self { let (cause, failure_class) = match frame_error.kind { - FrameErrorKind::PeerDisconnect => (WebSocketTerminationCause::PeerDisconnect, None), + FrameErrorKind::DownstreamDisconnect => { + (WebSocketTerminationCause::DownstreamDisconnect, None) + } + FrameErrorKind::UpstreamDisconnect => { + (WebSocketTerminationCause::UpstreamDisconnect, None) + } FrameErrorKind::Protocol(failure_class) => ( WebSocketTerminationCause::ProtocolError, Some(failure_class), @@ -381,7 +398,7 @@ impl TextMessageAssembly { Err(FrameError::assembly_timeout(AssemblyTimeoutKind::Idle)) } result = reader.read(buffer) => { - result.map_err(|error| FrameError::peer_io("websocket client read failed", error)) + result.map_err(FrameError::client_io) }, } } @@ -395,7 +412,7 @@ impl TextMessageAssembly { while filled < buffer.len() { let read = self.read_some(reader, &mut buffer[filled..]).await?; if read == 0 { - return Err(FrameError::peer_disconnect(miette!( + return Err(FrameError::client_disconnect(miette!( "websocket payload ended before declared length" ))); } @@ -524,22 +541,35 @@ where &mut options, ); let server_to_client = async { - tokio::io::copy(&mut upstream_read, &mut client_write) - .await - .map_err(|error| { + let mut buf = vec![0u8; COPY_BUF_SIZE]; + loop { + let read = upstream_read.read(&mut buf).await.map_err(|error| { terminate( - WebSocketTerminationCause::PeerDisconnect, - miette!("websocket upstream relay ended: {error}"), + WebSocketTerminationCause::UpstreamDisconnect, + miette!("websocket upstream read failed: {error}"), ) })?; - client_write.flush().await.map_err(|error| { - terminate( - WebSocketTerminationCause::PeerDisconnect, - miette!("websocket client relay ended: {error}"), - ) - })?; + if read == 0 { + break; + } + client_write + .write_all(&buf[..read]) + .await + .map_err(|error| { + terminate( + WebSocketTerminationCause::DownstreamDisconnect, + miette!("websocket client write failed: {error}"), + ) + })?; + client_write.flush().await.map_err(|error| { + terminate( + WebSocketTerminationCause::DownstreamDisconnect, + miette!("websocket client flush failed: {error}"), + ) + })?; + } Ok::<_, WebSocketTermination>( - openshell_core::proto::WebSocketSessionEndReason::PeerDisconnect, + openshell_core::proto::MiddlewareSessionEndReason::UpstreamDisconnect, ) }; @@ -619,7 +649,7 @@ async fn relay_client_to_server( host: &str, port: u16, options: &mut RelayOptions<'_>, -) -> WebSocketRelayResult +) -> WebSocketRelayResult where R: AsyncRead + Unpin, W: AsyncWrite + Unpin, @@ -637,9 +667,9 @@ where else { let _ = writer.shutdown().await; return Ok(if close_seen { - openshell_core::proto::WebSocketSessionEndReason::NormalClose + openshell_core::proto::MiddlewareSessionEndReason::Normal } else { - openshell_core::proto::WebSocketSessionEndReason::PeerDisconnect + openshell_core::proto::MiddlewareSessionEndReason::DownstreamDisconnect }); }; @@ -886,7 +916,7 @@ async fn read_exact_for_assembly( .read_exact(buffer) .await .map(|_| ()) - .map_err(|error| FrameError::peer_io("websocket client read failed", error)), + .map_err(FrameError::client_io), } } @@ -897,10 +927,7 @@ async fn read_frame_header( let mut first = [0u8; 1]; let first_read = match assembly { Some(assembly) => assembly.read_some(reader, &mut first).await, - None => reader - .read(&mut first) - .await - .map_err(|error| FrameError::peer_io("websocket client read failed", error)), + None => reader.read(&mut first).await.map_err(FrameError::client_io), }; let first = match first_read { Ok(0) => return Ok(None), @@ -1600,15 +1627,15 @@ where writer .write_all(&frame.raw_header) .await - .map_err(|error| FrameError::peer_io("websocket upstream write failed", error))?; + .map_err(|error| FrameError::upstream_io("websocket upstream write failed", error))?; writer .write_all(&raw_payload) .await - .map_err(|error| FrameError::peer_io("websocket upstream write failed", error))?; + .map_err(|error| FrameError::upstream_io("websocket upstream write failed", error))?; writer .flush() .await - .map_err(|error| FrameError::peer_io("websocket upstream flush failed", error))?; + .map_err(|error| FrameError::upstream_io("websocket upstream flush failed", error))?; Ok(()) } @@ -1654,7 +1681,7 @@ where writer .write_all(&frame.raw_header) .await - .map_err(|error| FrameError::peer_io("websocket upstream write failed", error))?; + .map_err(|error| FrameError::upstream_io("websocket upstream write failed", error))?; let mut remaining = frame.payload_len; let mut buf = [0u8; COPY_BUF_SIZE]; while remaining > 0 { @@ -1664,22 +1691,22 @@ where let n = reader .read(&mut buf[..to_read]) .await - .map_err(|error| FrameError::peer_io("websocket client read failed", error))?; + .map_err(FrameError::client_io)?; if n == 0 { - return Err(FrameError::peer_disconnect(miette!( + return Err(FrameError::client_disconnect(miette!( "websocket payload ended before declared length" ))); } writer .write_all(&buf[..n]) .await - .map_err(|error| FrameError::peer_io("websocket upstream write failed", error))?; + .map_err(|error| FrameError::upstream_io("websocket upstream write failed", error))?; remaining -= n as u64; } writer .flush() .await - .map_err(|error| FrameError::peer_io("websocket upstream flush failed", error))?; + .map_err(|error| FrameError::upstream_io("websocket upstream flush failed", error))?; Ok(()) } @@ -1762,19 +1789,19 @@ async fn write_text_frame_guarded( tokio::time::timeout(TEXT_MESSAGE_FORWARD_TOTAL_TIMEOUT, async { writer.write_all(header).await.map_err(|error| { terminate( - WebSocketTerminationCause::PeerDisconnect, + WebSocketTerminationCause::UpstreamDisconnect, miette!("websocket upstream write failed: {error}"), ) })?; writer.write_all(payload).await.map_err(|error| { terminate( - WebSocketTerminationCause::PeerDisconnect, + WebSocketTerminationCause::UpstreamDisconnect, miette!("websocket upstream write failed: {error}"), ) })?; writer.flush().await.map_err(|error| { terminate( - WebSocketTerminationCause::PeerDisconnect, + WebSocketTerminationCause::UpstreamDisconnect, miette!("websocket upstream flush failed: {error}"), ) }) @@ -1782,7 +1809,7 @@ async fn write_text_frame_guarded( .await .map_err(|_| { terminate( - WebSocketTerminationCause::PeerDisconnect, + WebSocketTerminationCause::UpstreamDisconnect, miette!("websocket upstream forwarding total timeout"), ) })??; @@ -2216,7 +2243,7 @@ network_policies: #[test] fn termination_causes_map_to_protocol_close_codes_and_session_reasons() { - use openshell_core::proto::WebSocketSessionEndReason as EndReason; + use openshell_core::proto::MiddlewareSessionEndReason as EndReason; assert_eq!( WebSocketTerminationCause::InvalidUtf8.close_code(), @@ -2238,7 +2265,14 @@ network_policies: WebSocketTerminationCause::PolicyReload.close_code(), Some(1012) ); - assert_eq!(WebSocketTerminationCause::PeerDisconnect.close_code(), None); + assert_eq!( + WebSocketTerminationCause::DownstreamDisconnect.close_code(), + None + ); + assert_eq!( + WebSocketTerminationCause::UpstreamDisconnect.close_code(), + None + ); assert_eq!( WebSocketTerminationCause::InvalidUtf8.session_end_reason(), @@ -2260,6 +2294,14 @@ network_policies: WebSocketTerminationCause::PolicyReload.session_end_reason(), EndReason::PolicyReload ); + assert_eq!( + WebSocketTerminationCause::DownstreamDisconnect.session_end_reason(), + EndReason::DownstreamDisconnect + ); + assert_eq!( + WebSocketTerminationCause::UpstreamDisconnect.session_end_reason(), + EndReason::UpstreamDisconnect + ); } fn resolver() -> (HashMap, SecretResolver) { @@ -2603,7 +2645,7 @@ network_policies: .await .expect("join forwarding") .expect_err("non-reading upstream must time out"); - assert_eq!(error.cause, WebSocketTerminationCause::PeerDisconnect); + assert_eq!(error.cause, WebSocketTerminationCause::UpstreamDisconnect); assert!( error .error @@ -3486,7 +3528,7 @@ network_policies: enum ObservedWebSocketRequest { SessionStart, Message { sequence: u64, payload: String }, - SessionEnd(openshell_core::proto::WebSocketSessionEndReason), + SessionEnd(openshell_core::proto::MiddlewareSessionEndReason), } #[derive(Clone, Default)] @@ -3648,7 +3690,7 @@ network_policies: Some(web_socket_session_event::Event::SessionEnd(end)) => { if let Some(observed) = &observed && let Ok(reason) = - openshell_core::proto::WebSocketSessionEndReason::try_from( + openshell_core::proto::MiddlewareSessionEndReason::try_from( end.reason, ) { @@ -3898,7 +3940,7 @@ network_policies: assert!(matches!( observed.recv().await, Some(ObservedWebSocketRequest::SessionEnd( - openshell_core::proto::WebSocketSessionEndReason::ProtocolError, + openshell_core::proto::MiddlewareSessionEndReason::ProtocolError, )) )); assert!( @@ -4018,7 +4060,7 @@ network_policies: assert!(matches!( observed.recv().await, Some(ObservedWebSocketRequest::SessionEnd( - openshell_core::proto::WebSocketSessionEndReason::PeerDisconnect, + openshell_core::proto::MiddlewareSessionEndReason::DownstreamDisconnect, )) )); @@ -4029,6 +4071,72 @@ network_policies: .expect("middleware server"); } + #[tokio::test] + async fn upstream_eof_reports_one_upstream_disconnect_while_downstream_is_open() { + let (mut session, mut observed, shutdown_tx, server_task) = + recording_middleware_session("wss").await; + assert!(session.start("").await.allowed); + assert!(matches!( + tokio::time::timeout(std::time::Duration::from_secs(2), observed.recv()) + .await + .expect("middleware observes session start"), + Some(ObservedWebSocketRequest::SessionStart) + )); + + let (client_app, mut relay_client) = tokio::io::duplex(4096); + let (mut relay_upstream, upstream_app) = tokio::io::duplex(4096); + let relay = tokio::spawn(async move { + relay_with_options( + &mut relay_client, + &mut relay_upstream, + Vec::new(), + "api.openai.com", + 443, + RelayOptions { + policy_name: "rest-api", + assembly_budget: WebSocketAssemblyBudget::default(), + resolver: None, + generation_guard: None, + provider_credentials: None, + target: "/", + inspector: None, + compression: WebSocketCompression::None, + middleware_session: Some(session), + middleware_context: None, + deny_uninspected_credentials: false, + }, + ) + .await + }); + + drop(upstream_app); + tokio::time::timeout(std::time::Duration::from_secs(2), relay) + .await + .expect("relay finishes after upstream disconnect") + .expect("join relay") + .expect("upstream EOF ends relay normally"); + assert!(matches!( + tokio::time::timeout(std::time::Duration::from_secs(2), observed.recv()) + .await + .expect("middleware observes session end"), + Some(ObservedWebSocketRequest::SessionEnd( + openshell_core::proto::MiddlewareSessionEndReason::UpstreamDisconnect, + )) + )); + assert!( + observed.try_recv().is_err(), + "upstream EOF must produce exactly one session end" + ); + + drop(client_app); + let _ = shutdown_tx.send(()); + tokio::time::timeout(std::time::Duration::from_secs(2), server_task) + .await + .expect("middleware server shuts down") + .expect("join middleware server") + .expect("middleware server"); + } + #[tokio::test] async fn denied_websocket_session_start_reports_middleware_failure_before_close() { let (session, mut observed, shutdown_tx, server_task) = @@ -4056,7 +4164,7 @@ network_policies: assert!(matches!( observed.recv().await, Some(ObservedWebSocketRequest::SessionEnd( - openshell_core::proto::WebSocketSessionEndReason::MiddlewareFailure, + openshell_core::proto::MiddlewareSessionEndReason::MiddlewareFailure, )) )); assert!( @@ -4258,7 +4366,7 @@ network_policies: assert!(error.to_string().contains("policy generation is stale")); match observed.recv().await { Some(ObservedWebSocketRequest::SessionEnd( - openshell_core::proto::WebSocketSessionEndReason::PolicyReload, + openshell_core::proto::MiddlewareSessionEndReason::PolicyReload, )) => {} Some(ObservedWebSocketRequest::Message { payload, .. }) => { panic!("stale message leaked {} bytes to middleware", payload.len()); @@ -4383,7 +4491,7 @@ network_policies: } assert_eq!( end_reason, - Some(openshell_core::proto::WebSocketSessionEndReason::PolicyReload) + Some(openshell_core::proto::MiddlewareSessionEndReason::PolicyReload) ); let _ = shutdown_tx.send(()); @@ -4423,7 +4531,7 @@ network_policies: assert!(matches!( observed.recv().await, Some(ObservedWebSocketRequest::SessionEnd( - openshell_core::proto::WebSocketSessionEndReason::PolicyReload + openshell_core::proto::MiddlewareSessionEndReason::PolicyReload )) )); @@ -4527,7 +4635,7 @@ network_policies: assert!(matches!( observed.recv().await, Some(ObservedWebSocketRequest::SessionEnd( - openshell_core::proto::WebSocketSessionEndReason::PolicyReload + openshell_core::proto::MiddlewareSessionEndReason::PolicyReload )) )); @@ -4637,7 +4745,7 @@ network_policies: ); match observed.recv().await { Some(ObservedWebSocketRequest::SessionEnd( - openshell_core::proto::WebSocketSessionEndReason::PolicyDenial, + openshell_core::proto::MiddlewareSessionEndReason::PolicyDenial, )) => {} Some(ObservedWebSocketRequest::Message { payload, .. }) => { panic!( diff --git a/crates/openshell-supervisor-network/src/lib.rs b/crates/openshell-supervisor-network/src/lib.rs index 4fec48b30..e00a5bf48 100644 --- a/crates/openshell-supervisor-network/src/lib.rs +++ b/crates/openshell-supervisor-network/src/lib.rs @@ -4,12 +4,11 @@ //! Networking component of the `OpenShell` supervisor. //! //! Owns the egress proxy, L7 enforcement, OPA policy engine, identity cache, -//! inference routing, and TLS interception. The denial-event channel is +//! TLS interception, and credential injection. The denial-event channel is //! owned by the orchestrator; this crate produces denials but does not //! aggregate them. pub mod identity; -pub mod inference_routes; pub mod l7; pub mod opa; pub(crate) mod policy_dns; diff --git a/crates/openshell-supervisor-network/src/opa.rs b/crates/openshell-supervisor-network/src/opa.rs index 63aa2c2c7..d2ed59378 100644 --- a/crates/openshell-supervisor-network/src/opa.rs +++ b/crates/openshell-supervisor-network/src/opa.rs @@ -13,7 +13,7 @@ use openshell_core::policy::{ FilesystemPolicy, LandlockCompatibility, LandlockPolicy, ProcessPolicy, }; use openshell_core::proto::SandboxPolicy as ProtoSandboxPolicy; -use openshell_policy::L7ConfigStanza; +use openshell_policy::{L7ConfigStanza, L7Protocol as PolicyL7Protocol}; use openshell_supervisor_middleware::{ChainEntry, ChainRunner, MiddlewareRegistry}; use std::path::{Path, PathBuf}; use std::sync::{ @@ -413,7 +413,21 @@ impl OpaEngine { entrypoint_pid: u32, require_binary_identity: bool, ) -> Result { - let ambiguities = openshell_policy::find_endpoint_ambiguities(proto); + // Protobuf cannot distinguish an omitted repeated MCP version field + // from an empty one. Canonicalize before any runtime consumer reads + // the policy so both representations select the pinned default. + let proto = openshell_policy::validate_and_canonicalize_sandbox_policy(proto.clone()) + .map_err(|error| { + let errors = error + .violations() + .iter() + .map(ToString::to_string) + .collect::>() + .join("\n"); + miette::miette!("policy validation failed:\n{errors}") + })?; + + let ambiguities = openshell_policy::find_endpoint_ambiguities(&proto); if !ambiguities.is_empty() { return Err(miette::miette!( "network endpoint ambiguity validation failed:\n{}", @@ -426,21 +440,13 @@ impl OpaEngine { } emit_binary_identity_mode(require_binary_identity, "proto"); - if let Err(violations) = openshell_policy::validate_sandbox_policy(proto) { - let errors = violations - .iter() - .map(ToString::to_string) - .collect::>() - .join("\n"); - return Err(miette::miette!("policy validation failed:\n{errors}")); - } - - let data_json_str = proto_to_opa_data_json(proto, entrypoint_pid); + let data_json_str = proto_to_opa_data_json(&proto, entrypoint_pid); // Parse back to Value for preprocessing, then re-serialize let mut data: serde_json::Value = serde_json::from_str(&data_json_str) .map_err(|e| miette::miette!("internal: failed to parse proto JSON: {e}"))?; inject_runtime_policy_data(&mut data, require_binary_identity); + normalize_endpoint_protocols(&mut data); // Validate BEFORE expanding presets let (errors, warnings) = crate::l7::validate_l7_policies(&data); @@ -1382,6 +1388,7 @@ fn preprocess_yaml_data( let mut data: serde_json::Value = serde_yml::from_str(yaml_str) .map_err(|e| miette::miette!("failed to parse YAML data: {e}"))?; inject_runtime_policy_data(&mut data, require_binary_identity); + normalize_endpoint_protocols(&mut data); // Normalize port → ports for all endpoints so Rego always sees "ports" array. normalize_endpoint_ports(&mut data); @@ -1431,6 +1438,57 @@ fn preprocess_yaml_data( serde_json::to_string(&data).map_err(|e| miette::miette!("failed to serialize data: {e}")) } +/// Canonicalize recognized protocol spellings before L7 validation or Rego use. +/// +/// Rust parses protocol names case-insensitively, while Rego compares stable +/// wire keys. Publishing one canonical spelling keeps rule validation, typed +/// routing, and authorization on the same protocol branch. +fn normalize_endpoint_protocols(data: &mut serde_json::Value) { + let Some(policies) = data + .get_mut("network_policies") + .and_then(serde_json::Value::as_object_mut) + else { + return; + }; + + for policy in policies.values_mut() { + let Some(endpoints) = policy + .get_mut("endpoints") + .and_then(serde_json::Value::as_array_mut) + else { + continue; + }; + + for endpoint in endpoints { + let Some(endpoint) = endpoint.as_object_mut() else { + continue; + }; + let Some(protocol) = endpoint.get("protocol").and_then(serde_json::Value::as_str) + else { + continue; + }; + let canonical = if protocol.eq_ignore_ascii_case("tcp") { + Some("tcp") + } else { + PolicyL7Protocol::parse(protocol).map(|protocol| match protocol { + PolicyL7Protocol::Rest => "rest", + PolicyL7Protocol::Websocket => "websocket", + PolicyL7Protocol::Graphql => "graphql", + PolicyL7Protocol::Sql => "sql", + PolicyL7Protocol::JsonRpc => "json-rpc", + PolicyL7Protocol::Mcp => "mcp", + }) + }; + if let Some(canonical) = canonical { + endpoint.insert( + "protocol".to_string(), + serde_json::Value::String(canonical.to_string()), + ); + } + } + } +} + /// Normalize endpoint port/ports in JSON data. /// /// YAML policies may use `port: N` (single) or `ports: [N, M]` (multi). @@ -1502,6 +1560,28 @@ fn normalize_l7_config_aliases(data: &mut serde_json::Value) -> Vec { for stanza in L7ConfigStanza::ALL { normalize_l7_config_alias(&mut errors, ep_obj, &loc, stanza); } + + // The nested MCP stanza is optional, but the runtime projection is + // not. Materialize the pinned default at this YAML boundary so a + // missing alias cannot later look like corrupted runtime state. + if ep_obj + .get("protocol") + .and_then(serde_json::Value::as_str) + .is_some_and(|protocol| protocol.eq_ignore_ascii_case("mcp")) + && !ep_obj.contains_key("mcp_versions") + { + match openshell_policy::l7_config_alias_runtime_fields( + L7ConfigStanza::Mcp, + serde_json::json!({}), + ) { + Ok(fields) => { + for (field, value) in fields { + ep_obj.insert(field.to_string(), value); + } + } + Err(error) => errors.push(format!("{loc}.mcp: {error}")), + } + } } } @@ -1520,12 +1600,24 @@ fn normalize_l7_config_alias( }; if config.is_null() { ep.remove(key); + if stanza == L7ConfigStanza::Mcp { + errors.push(format!("{loc}.{key}: mcp config must be an object")); + } return; } match openshell_policy::l7_config_alias_runtime_fields(stanza, config) { Ok(fields) => { ep.remove(key); for (field, value) in fields { + if stanza == L7ConfigStanza::Mcp + && field == "mcp_versions" + && ep.contains_key(field) + { + errors.push(format!( + "{loc}: mcp.versions and mcp_versions cannot both be set" + )); + continue; + } ep.entry(field.to_string()).or_insert(value); } } @@ -1931,29 +2023,48 @@ fn proto_to_opa_data_json(proto: &ProtoSandboxPolicy, entrypoint_pid: u32) -> St .iter() .map(|r| { let a = r.allow.as_ref(); - let mut allow = serde_json::json!({ - "method": a.map_or("", |a| &a.method), - "path": a.map_or("", |a| &a.path), - "command": a.map_or("", |a| &a.command), - "operation_type": a.map_or("", |a| &a.operation_type), - "operation_name": a.map_or("", |a| &a.operation_name), - }); - if let Some(a) = a - && !a.fields.is_empty() - { - allow["fields"] = a.fields.clone().into(); + let mut allow = serde_json::Map::new(); + if let Some(a) = a { + // Proto3 represents absent scalar selectors as empty + // strings. Omit them so protobuf and YAML rules expose + // the same selector families to runtime validation. + if !a.method.is_empty() { + allow.insert("method".to_string(), a.method.clone().into()); + } + if !a.path.is_empty() { + allow.insert("path".to_string(), a.path.clone().into()); + } + if !a.command.is_empty() { + allow + .insert("command".to_string(), a.command.clone().into()); + } + if !a.operation_type.is_empty() { + allow.insert( + "operation_type".to_string(), + a.operation_type.clone().into(), + ); + } + if !a.operation_name.is_empty() { + allow.insert( + "operation_name".to_string(), + a.operation_name.clone().into(), + ); + } + if !a.fields.is_empty() { + allow.insert("fields".to_string(), a.fields.clone().into()); + } } let query = a.map_or_else(serde_json::Map::new, |allow| { l7_matchers_to_json(&allow.query) }); if !query.is_empty() { - allow["query"] = query.into(); + allow.insert("query".to_string(), query.into()); } let params = a.map_or_else(serde_json::Map::new, |allow| { l7_matchers_to_json(&allow.params) }); if !params.is_empty() { - allow["params"] = params.into(); + allow.insert("params".to_string(), params.into()); } serde_json::json!({ "allow": allow }) }) @@ -2059,6 +2170,9 @@ fn proto_to_opa_data_json(proto: &ProtoSandboxPolicy, entrypoint_pid: u32) -> St ep["json_rpc_max_body_bytes"] = e.json_rpc_max_body_bytes.into(); } if let Some(mcp) = &e.mcp { + if e.protocol.eq_ignore_ascii_case("mcp") { + ep["mcp_versions"] = mcp.versions.clone().into(); + } if let Some(strict_tool_names) = mcp.strict_tool_names { ep["mcp_strict_tool_names"] = strict_tool_names.into(); } @@ -2192,8 +2306,9 @@ fn proto_to_opa_data_json(proto: &ProtoSandboxPolicy, entrypoint_pid: u32) -> St mod tests { use super::*; + use openshell_core::mcp::DEFAULT_MCP_PROTOCOL_VERSION; use openshell_core::proto::{ - FilesystemPolicy as ProtoFs, L7Allow, L7QueryMatcher, L7Rule, NetworkBinary, + FilesystemPolicy as ProtoFs, L7Allow, L7QueryMatcher, L7Rule, McpOptions, NetworkBinary, NetworkEndpoint, NetworkMiddlewareConfig, NetworkPolicyRule, ProcessPolicy as ProtoProc, SandboxPolicy as ProtoSandboxPolicy, }; @@ -2263,6 +2378,112 @@ mod tests { } } + fn defaultable_mcp_proto(mcp: Option) -> ProtoSandboxPolicy { + let mut policy = openshell_policy::restrictive_default_policy(); + policy.network_policies.insert( + "mcp".to_string(), + NetworkPolicyRule { + name: "mcp".to_string(), + endpoints: vec![NetworkEndpoint { + host: "mcp.example.com".to_string(), + port: 443, + protocol: "mcp".to_string(), + mcp, + rules: vec![L7Rule { + allow: Some(L7Allow { + method: "tools/list".to_string(), + ..Default::default() + }), + }], + ..Default::default() + }], + binaries: vec![NetworkBinary { + path: "/usr/bin/curl".to_string(), + ..Default::default() + }], + }, + ); + policy + } + + fn projected_allow( + protocol: &str, + allow: L7Allow, + ) -> serde_json::Map { + let proto = ProtoSandboxPolicy { + version: 1, + network_policies: std::collections::HashMap::from([( + "selector_projection".to_string(), + NetworkPolicyRule { + name: "selector_projection".to_string(), + endpoints: vec![NetworkEndpoint { + host: "api.example.com".to_string(), + port: 443, + protocol: protocol.to_string(), + rules: vec![L7Rule { allow: Some(allow) }], + ..Default::default() + }], + binaries: vec![], + }, + )]), + ..Default::default() + }; + let projected: serde_json::Value = serde_json::from_str(&proto_to_opa_data_json(&proto, 0)) + .expect("protobuf policy projection must produce JSON"); + projected["network_policies"]["selector_projection"]["endpoints"][0]["rules"][0]["allow"] + .as_object() + .expect("projected allow rule must be an object") + .clone() + } + + #[test] + fn proto_projection_omits_absent_allow_selectors() { + let allow = projected_allow( + "websocket", + L7Allow { + operation_type: "subscription".to_string(), + operation_name: "NewMessages".to_string(), + fields: vec!["messageAdded".to_string()], + ..Default::default() + }, + ); + + assert_eq!( + allow, + serde_json::json!({ + "operation_type": "subscription", + "operation_name": "NewMessages", + "fields": ["messageAdded"], + }) + .as_object() + .expect("expected object") + .clone() + ); + } + + #[test] + fn proto_projection_preserves_nonempty_allow_selectors() { + let allow = projected_allow( + "rest", + L7Allow { + method: "POST".to_string(), + path: "/repos/**".to_string(), + ..Default::default() + }, + ); + + assert_eq!( + allow, + serde_json::json!({ + "method": "POST", + "path": "/repos/**", + }) + .as_object() + .expect("expected object") + .clone() + ); + } + const POLICY_DNS_SNAPSHOT_DATA: &str = r#" network_policies: dns_transport: @@ -3898,6 +4119,10 @@ network_policies: path: "/mcp".to_string(), protocol: "mcp".to_string(), enforcement: "enforce".to_string(), + mcp: Some(McpOptions { + versions: vec![DEFAULT_MCP_PROTOCOL_VERSION.as_str().to_string()], + ..Default::default() + }), rules: vec![L7Rule { allow: Some(L7Allow { method: "tools/call".to_string(), @@ -4824,6 +5049,7 @@ network_policies: protocol: mcp enforcement: enforce mcp: + versions: ["2025-11-25", "2025-03-26"] strict_tool_names: false rules: - allow: @@ -4847,6 +5073,426 @@ network_policies: let l7 = crate::l7::parse_l7_config(&config).expect("parse l7 config"); assert_eq!(l7.protocol, crate::l7::L7Protocol::Mcp); assert!(!l7.mcp_strict_tool_names); + assert_eq!( + l7.mcp_versions, + vec![ + openshell_core::mcp::McpProtocolVersion::V2025_03_26, + openshell_core::mcp::McpProtocolVersion::V2025_11_25, + ] + ); + } + + const PROTOCOL_NAME_CASES: [(&str, &str); 7] = [ + ("tcp", "TcP"), + ("rest", "ReSt"), + ("websocket", "WeBsOcKeT"), + ("graphql", "GrApHqL"), + ("sql", "SqL"), + ("json-rpc", "JsOn-RpC"), + ("mcp", "McP"), + ]; + + fn assert_loaded_protocol_name(engine: &OpaEngine, authored: &str, canonical: &str) { + // Inspect stored data before typed parsing, which accepts mixed case and + // would otherwise hide a missing normalization step in either loader. + let data = engine.engine.lock().expect("OPA engine lock").get_data(); + assert_eq!( + data["network_policies"]["protocol_names"]["endpoints"][0]["protocol"], + regorus::Value::from(canonical), + "stored protocol for {authored}" + ); + + let input = NetworkInput { + host: "protocol.example.com".into(), + port: 443, + binary_path: PathBuf::from("/usr/bin/curl"), + binary_sha256: "unused".into(), + ancestors: vec![], + cmdline_paths: vec![], + }; + let config = engine + .query_endpoint_config(&input) + .expect("endpoint query"); + if canonical == "tcp" { + // TCP remains an L4 endpoint; canonical spelling must not opt it + // into request inspection or change its network-level allowance. + assert!(config.is_none(), "TCP must not return L7 configuration"); + assert!(matches!( + engine + .evaluate_network_action(&input) + .expect("network query"), + NetworkAction::Allow { .. } + )); + } else { + let config = config.expect("L7 endpoint configuration"); + assert_eq!( + config["protocol"], + regorus::Value::from(canonical), + "published endpoint protocol for {authored}" + ); + crate::l7::parse_l7_config(&config).expect("valid typed L7 configuration"); + } + } + + #[test] + fn protocol_names_yaml_load_canonicalizes_all_supported_values() { + for (canonical, mixed_case) in PROTOCOL_NAME_CASES { + let fields = match canonical { + "tcp" => "", + "rest" | "websocket" => { + "enforcement: enforce\n rules: [{allow: {method: GET, path: /status}}]" + } + "graphql" => { + "enforcement: enforce\n rules: [{allow: {operation_type: query, fields: [viewer]}}]" + } + // SQL inspection supports audit mode, not enforce mode. + "sql" => "enforcement: audit\n rules: [{allow: {command: SELECT}}]", + "json-rpc" => "enforcement: enforce\n rules: [{allow: {method: status}}]", + "mcp" => "enforcement: enforce\n rules: [{allow: {method: tools/list}}]", + _ => unreachable!("fixture must name a supported protocol"), + }; + for authored in [ + canonical.to_string(), + canonical.to_ascii_uppercase(), + mixed_case.into(), + ] { + let yaml = format!( + r#" +network_policies: + protocol_names: + name: protocol_names + endpoints: + - host: protocol.example.com + port: 443 + protocol: {authored} + {fields} + binaries: + - {{ path: /usr/bin/curl }} +"# + ); + let engine = OpaEngine::from_strings(TEST_POLICY, &yaml) + .unwrap_or_else(|error| panic!("valid YAML protocol {authored}: {error}")); + assert_loaded_protocol_name(&engine, &authored, canonical); + } + } + } + + #[test] + fn protocol_names_proto_load_canonicalizes_all_supported_values() { + for (canonical, mixed_case) in PROTOCOL_NAME_CASES { + let allow = match canonical { + "tcp" => None, + "rest" | "websocket" => Some(L7Allow { + method: "GET".into(), + path: "/status".into(), + ..Default::default() + }), + "graphql" => Some(L7Allow { + operation_type: "query".into(), + fields: vec!["viewer".into()], + ..Default::default() + }), + "sql" => Some(L7Allow { + command: "SELECT".into(), + ..Default::default() + }), + "json-rpc" | "mcp" => Some(L7Allow { + method: if canonical == "mcp" { + "tools/list" + } else { + "status" + } + .into(), + ..Default::default() + }), + _ => unreachable!("fixture must name a supported protocol"), + }; + for authored in [ + canonical.to_string(), + canonical.to_ascii_uppercase(), + mixed_case.into(), + ] { + let mut policy = openshell_policy::restrictive_default_policy(); + policy.network_policies.insert( + "protocol_names".into(), + NetworkPolicyRule { + name: "protocol_names".into(), + endpoints: vec![NetworkEndpoint { + host: "protocol.example.com".into(), + port: 443, + protocol: authored.clone(), + // TCP has no L7 settings; SQL only supports audit. + enforcement: match canonical { + "tcp" => "", + "sql" => "audit", + _ => "enforce", + } + .into(), + rules: allow + .clone() + .map(|allow| L7Rule { allow: Some(allow) }) + .into_iter() + .collect(), + ..Default::default() + }], + binaries: vec![NetworkBinary { + path: "/usr/bin/curl".into(), + ..Default::default() + }], + }, + ); + let engine = OpaEngine::from_proto(&policy) + .unwrap_or_else(|error| panic!("valid protobuf protocol {authored}: {error}")); + assert_loaded_protocol_name(&engine, &authored, canonical); + } + } + } + + #[test] + fn protocol_names_normalization_preserves_other_data_and_is_idempotent() { + let mut data = serde_json::json!({ + "network_policies": { + "first": { + "endpoints": [ + {"protocol": "ReSt", "path": "/KeepCase", "host": "KeepCase.example"}, + {"protocol": "FutureProtocol"}, + {"protocol": ""}, + {"protocol": null}, + {"protocol": 42}, + {"host": "no-protocol.example"} + ] + }, + "second": {"endpoints": [{"protocol": "JsOn-RpC"}]} + } + }); + let mut expected = data.clone(); + expected["network_policies"]["first"]["endpoints"][0]["protocol"] = "rest".into(); + expected["network_policies"]["second"]["endpoints"][0]["protocol"] = "json-rpc".into(); + + normalize_endpoint_protocols(&mut data); + assert_eq!(data, expected, "only recognized protocol names may change"); + normalize_endpoint_protocols(&mut data); + assert_eq!(data, expected, "normalization must be idempotent"); + } + + #[test] + fn yaml_load_accepts_mixed_case_mcp_protocol_with_default_versions() { + let data = r#" +network_policies: + mcp: + name: mcp + endpoints: + - host: mcp.example.com + port: 443 + protocol: MCP + rules: + - allow: + method: tools/list + binaries: + - { path: /usr/bin/curl } +"#; + let engine = OpaEngine::from_strings(TEST_POLICY, data).expect("engine from yaml"); + let input = NetworkInput { + host: "mcp.example.com".into(), + port: 443, + binary_path: PathBuf::from("/usr/bin/curl"), + binary_sha256: "unused".into(), + ancestors: vec![], + cmdline_paths: vec![], + }; + let config = engine + .query_endpoint_config(&input) + .expect("query endpoint config") + .expect("expected MCP endpoint config"); + let l7 = crate::l7::parse_l7_config(&config).expect("parse L7 endpoint config"); + + let config_json: serde_json::Value = serde_json::from_str( + &config + .to_json_str() + .expect("endpoint config must serialize as JSON"), + ) + .expect("endpoint config must be valid JSON"); + assert_eq!( + config_json["protocol"], + serde_json::json!("mcp"), + "OPA data must use the canonical protocol key" + ); + assert_eq!(l7.mcp_versions, vec![DEFAULT_MCP_PROTOCOL_VERSION]); + assert!(eval_l7( + &engine, + &l7_jsonrpc_input("mcp.example.com", 443, "/", "tools/list") + )); + assert!(!eval_l7( + &engine, + &l7_jsonrpc_input("mcp.example.com", 443, "/", "tools/call") + )); + } + + #[test] + fn yaml_load_rejects_rest_shaped_rules_on_mixed_case_mcp_protocol() { + let data = r#" +network_policies: + mcp: + name: mcp + endpoints: + - host: mcp.example.com + port: 443 + protocol: MCP + rules: + - allow: + method: POST + path: "**" + binaries: + - { path: /usr/bin/curl } +"#; + + let Err(error) = OpaEngine::from_strings(TEST_POLICY, data) else { + panic!("mixed-case MCP must not bypass MCP rule validation"); + }; + assert!( + error + .to_string() + .contains("mcp L7 rules must use method/tool, not path/query"), + "{error}" + ); + } + + #[test] + fn yaml_load_rejects_invalid_flat_mcp_versions_before_activation() { + for (case, versions) in [ + ("empty", "[]"), + ("non-string", "[1]"), + ("unsupported", "[\"2026-01-01\"]"), + ("duplicate", "[\"2025-11-25\", \"2025-11-25\"]"), + ("non-canonical", "[\"2025-11-25\", \"2025-03-26\"]"), + ] { + let data = format!( + r#" +network_policies: + mcp: + name: mcp + endpoints: + - host: mcp.example.com + port: 443 + protocol: mcp + mcp_versions: {versions} + rules: + - allow: + method: tools/list + binaries: + - {{ path: /usr/bin/curl }} +"# + ); + + let Err(error) = OpaEngine::from_strings(TEST_POLICY, &data) else { + panic!("invalid MCP runtime metadata must reject activation: {case}"); + }; + assert!( + error.to_string().contains("mcp.versions"), + "{case}: {error}" + ); + } + } + + #[test] + fn yaml_load_rejects_nested_and_flat_mcp_version_collision() { + let data = r#" +network_policies: + mcp: + name: mcp + endpoints: + - host: mcp.example.com + port: 443 + protocol: mcp + mcp_versions: ["2025-03-26"] + mcp: + versions: ["2025-11-25"] + rules: + - allow: + method: tools/list + binaries: + - { path: /usr/bin/curl } +"#; + + let Err(error) = OpaEngine::from_strings(TEST_POLICY, data) else { + panic!("ambiguous MCP revision sources must reject activation"); + }; + assert!( + error + .to_string() + .contains("mcp.versions and mcp_versions cannot both be set"), + "{error}" + ); + } + + #[test] + fn yaml_load_rejects_null_mcp_config_before_defaulting() { + for mcp_fields in [ + "mcp: null", + "mcp: null\n mcp_versions: [\"2025-11-25\"]", + ] { + let data = format!( + r#" +network_policies: + mcp: + name: mcp + endpoints: + - host: mcp.example.com + port: 443 + protocol: mcp + {mcp_fields} + rules: + - allow: + method: tools/list + binaries: + - {{ path: /usr/bin/curl }} +"# + ); + + let Err(error) = OpaEngine::from_strings(TEST_POLICY, &data) else { + panic!("null MCP config must reject activation"); + }; + assert!( + error.to_string().contains("mcp config must be an object"), + "{error}" + ); + } + } + + #[test] + fn yaml_load_rejects_mcp_versions_on_non_mcp_protocols() { + for mcp_fields in [ + "mcp:\n versions: [\"2025-11-25\"]", + "mcp_versions: [\"2025-11-25\"]", + ] { + let data = format!( + r#" +network_policies: + json_rpc: + name: json_rpc + endpoints: + - host: rpc.example.com + port: 443 + protocol: json-rpc + {mcp_fields} + rules: + - allow: + method: ping + binaries: + - {{ path: /usr/bin/curl }} +"# + ); + + let Err(error) = OpaEngine::from_strings(TEST_POLICY, &data) else { + panic!("MCP revision policy must not apply to generic JSON-RPC"); + }; + assert!( + error + .to_string() + .contains("mcp.versions is only valid for protocol mcp"), + "{error}" + ); + } } #[test] @@ -5266,6 +5912,97 @@ network_policies: assert!(message.contains("tls")); } + #[test] + fn proto_load_accepts_defaultable_mcp_versions_with_mixed_case_protocol() { + let mut implicit_defaults = defaultable_mcp_proto(None); + implicit_defaults + .network_policies + .get_mut("mcp") + .expect("defaultable MCP fixture contains the MCP policy") + .endpoints[0] + .protocol = "Mcp".to_string(); + let explicit_defaults = defaultable_mcp_proto(Some(McpOptions::default())); + + for policy in [implicit_defaults, explicit_defaults] { + let engine = OpaEngine::from_proto(&policy) + .expect("supervisor ingress must materialize the pinned MCP revision"); + let input = NetworkInput { + host: "mcp.example.com".into(), + port: 443, + binary_path: PathBuf::from("/usr/bin/curl"), + binary_sha256: "unused".into(), + ancestors: vec![], + cmdline_paths: vec![], + }; + let config = engine + .query_endpoint_config(&input) + .expect("query endpoint config") + .expect("expected MCP endpoint config"); + let config_json: serde_json::Value = serde_json::from_str( + &config + .to_json_str() + .expect("endpoint config must serialize as JSON"), + ) + .expect("endpoint config must be valid JSON"); + assert_eq!(config_json["protocol"], serde_json::json!("mcp")); + let l7 = crate::l7::parse_l7_config(&config).expect("parse L7 endpoint config"); + assert_eq!( + l7.mcp_versions, + vec![DEFAULT_MCP_PROTOCOL_VERSION], + "protobuf ingress must preserve the materialized default through OPA" + ); + } + } + + #[test] + fn proto_load_projects_canonical_mcp_versions_to_l7_config() { + let policy = defaultable_mcp_proto(Some(McpOptions { + versions: vec!["2025-11-25".to_string(), "2025-03-26".to_string()], + ..Default::default() + })); + let engine = OpaEngine::from_proto(&policy).expect("valid MCP policy"); + let input = NetworkInput { + host: "mcp.example.com".into(), + port: 443, + binary_path: PathBuf::from("/usr/bin/curl"), + binary_sha256: "unused".into(), + ancestors: vec![], + cmdline_paths: vec![], + }; + let config = engine + .query_endpoint_config(&input) + .expect("query endpoint config") + .expect("expected MCP endpoint config"); + let l7 = crate::l7::parse_l7_config(&config).expect("parse L7 endpoint config"); + + assert_eq!( + l7.mcp_versions, + vec![ + openshell_core::mcp::McpProtocolVersion::V2025_03_26, + openshell_core::mcp::McpProtocolVersion::V2025_11_25, + ], + "protobuf ingress must preserve the canonical allowlist through OPA" + ); + } + + #[test] + fn proto_load_rejects_unsupported_mcp_versions() { + let policy = defaultable_mcp_proto(Some(McpOptions { + versions: vec!["latest".to_string()], + ..Default::default() + })); + + let Err(error) = OpaEngine::from_proto(&policy) else { + panic!("canonicalization must not repair an unsupported MCP revision"); + }; + let error = error.to_string(); + assert!(error.contains("policy validation failed"), "{error}"); + assert!( + error.contains("unsupported protocol version 'latest'"), + "{error}" + ); + } + #[tokio::test] async fn fail_closed_quarantine_denies_and_wakes_generation_guards() { let engine = test_engine(); @@ -5730,7 +6467,7 @@ process: // network_action tests // ======================================================================== - const INFERENCE_TEST_DATA: &str = r#" + const PROVIDER_ENDPOINT_TEST_DATA: &str = r#" network_policies: claude_code: name: claude_code @@ -5755,7 +6492,7 @@ process: run_as_group: sandbox "#; - const NO_INFERENCE_TEST_DATA: &str = r#" + const OTHER_ENDPOINT_TEST_DATA: &str = r#" network_policies: gitlab: name: gitlab @@ -5774,19 +6511,19 @@ process: run_as_group: sandbox "#; - fn inference_engine() -> OpaEngine { - OpaEngine::from_strings(TEST_POLICY, INFERENCE_TEST_DATA) - .expect("Failed to load inference test data") + fn provider_endpoint_engine() -> OpaEngine { + OpaEngine::from_strings(TEST_POLICY, PROVIDER_ENDPOINT_TEST_DATA) + .expect("Failed to load provider endpoint test data") } - fn no_inference_engine() -> OpaEngine { - OpaEngine::from_strings(TEST_POLICY, NO_INFERENCE_TEST_DATA) - .expect("Failed to load no-inference test data") + fn other_endpoint_engine() -> OpaEngine { + OpaEngine::from_strings(TEST_POLICY, OTHER_ENDPOINT_TEST_DATA) + .expect("Failed to load alternate endpoint test data") } #[test] fn explicitly_allowed_endpoint_binary_returns_allow() { - let engine = inference_engine(); + let engine = provider_endpoint_engine(); let input = NetworkInput { host: "api.anthropic.com".into(), port: 443, @@ -5808,7 +6545,7 @@ process: fn relaxed_binary_identity_allows_declared_endpoint_without_binary_match() { let engine = OpaEngine::from_strings_with_binary_identity_required( TEST_POLICY, - INFERENCE_TEST_DATA, + PROVIDER_ENDPOINT_TEST_DATA, false, ) .expect("Failed to load relaxed binary identity test data"); @@ -5846,7 +6583,7 @@ process: #[test] fn unknown_endpoint_returns_deny() { - let engine = inference_engine(); + let engine = provider_endpoint_engine(); let input = NetworkInput { host: "api.openai.com".into(), port: 443, @@ -5863,8 +6600,8 @@ process: } #[test] - fn unknown_endpoint_without_inference_returns_deny() { - let engine = no_inference_engine(); + fn unknown_endpoint_with_other_policy_returns_deny() { + let engine = other_endpoint_engine(); let input = NetworkInput { host: "api.openai.com".into(), port: 443, @@ -5884,7 +6621,7 @@ process: fn endpoint_in_policy_binary_not_allowed_returns_deny() { // api.anthropic.com is declared but python3 is not in the binary list. // With binary allow/deny, this is denied. - let engine = inference_engine(); + let engine = provider_endpoint_engine(); let input = NetworkInput { host: "api.anthropic.com".into(), port: 443, @@ -5901,8 +6638,8 @@ process: } #[test] - fn endpoint_in_policy_binary_not_allowed_without_inference_returns_deny() { - let engine = no_inference_engine(); + fn endpoint_in_policy_binary_not_allowed_with_other_policy_returns_deny() { + let engine = other_endpoint_engine(); let input = NetworkInput { host: "gitlab.com".into(), port: 443, @@ -7999,7 +8736,7 @@ network_policies: old_sessions .pop() .expect("old-generation session") - .end(openshell_core::proto::WebSocketSessionEndReason::PolicyReload) + .end(openshell_core::proto::MiddlewareSessionEndReason::PolicyReload) .await; let admitted = current_runner .preflight_websocket( diff --git a/crates/openshell-supervisor-network/src/policy_dns/mod.rs b/crates/openshell-supervisor-network/src/policy_dns/mod.rs index b7dd13ca9..6c692e495 100644 --- a/crates/openshell-supervisor-network/src/policy_dns/mod.rs +++ b/crates/openshell-supervisor-network/src/policy_dns/mod.rs @@ -580,7 +580,7 @@ mod tests { calls: AtomicUsize::new(0), answer: TrustedAnswer { addresses, - ttl: Duration::from_secs(300), + ttl: Duration::from_mins(5), }, }, Arc::new(ResolvedEndpointStore::new( @@ -984,6 +984,6 @@ process: { run_as_user: sandbox, run_as_group: sandbox } clamp_mapping_ttl(Duration::from_secs(10)), Duration::from_secs(10) ); - assert_eq!(clamp_mapping_ttl(Duration::from_secs(300)), MAX_MAPPING_TTL); + assert_eq!(clamp_mapping_ttl(Duration::from_mins(5)), MAX_MAPPING_TTL); } } diff --git a/crates/openshell-supervisor-network/src/policy_local.rs b/crates/openshell-supervisor-network/src/policy_local.rs index 908587ed3..520a22adb 100644 --- a/crates/openshell-supervisor-network/src/policy_local.rs +++ b/crates/openshell-supervisor-network/src/policy_local.rs @@ -85,7 +85,7 @@ const MAX_DENIAL_LINE_BYTES: usize = 4096; #[derive(Debug)] pub struct PolicyLocalContext { - current_policy: Arc>>, + current_policy: Arc>>>, agent_proposals: AgentProposals, gateway_endpoint: Option, sandbox_name: Option, @@ -120,7 +120,7 @@ impl PolicyLocalContext { workspace_rx: tokio::sync::watch::Receiver, ) -> Self { Self { - current_policy: Arc::new(RwLock::new(current_policy)), + current_policy: Arc::new(RwLock::new(current_policy.map(Arc::new))), agent_proposals, gateway_endpoint, sandbox_name, @@ -130,7 +130,10 @@ impl PolicyLocalContext { } pub async fn set_current_policy(&self, policy: ProtoSandboxPolicy) { - *self.current_policy.write().await = Some(policy); + // Every successful reload receives a distinct Arc, including an + // identical policy installed again. Waiters use pointer identity to + // decide whether the installed snapshot needs another coverage scan. + *self.current_policy.write().await = Some(Arc::new(policy)); } pub fn workspace(&self) -> String { @@ -887,18 +890,41 @@ async fn wait_for_local_policy_to_cover( proposed_rule: &NetworkPolicyRule, deadline: tokio::time::Instant, ) -> bool { + wait_for_local_policy_to_cover_with( + ctx, + proposed_rule, + deadline, + openshell_policy::policy_covers_rule, + ) + .await +} + +async fn wait_for_local_policy_to_cover_with( + ctx: &PolicyLocalContext, + proposed_rule: &NetworkPolicyRule, + deadline: tokio::time::Instant, + covers_rule: F, +) -> bool +where + F: Fn(&ProtoSandboxPolicy, &NetworkPolicyRule) -> bool, +{ const TICK: std::time::Duration = std::time::Duration::from_millis(200); + let mut checked_snapshot: Option> = None; loop { - // Clone the snapshot out of the RwLock before running coverage — - // otherwise the read guard is held across `policy_covers_rule`'s - // iteration of `network_policies`, serializing a writer (supervisor - // reload) on the very thing we're waiting for. Clone-per-tick on - // a few-KB struct is cheap for the bounded wait window here. + // Clone only the Arc while holding the lock. Coverage runs once for + // each installed snapshot and never holds the read guard, so reloads + // cannot block behind a full network-policy scan. let snapshot = ctx.current_policy.read().await.clone(); if let Some(policy) = snapshot.as_ref() - && openshell_policy::policy_covers_rule(policy, proposed_rule) + && checked_snapshot + .as_ref() + .is_none_or(|checked| !Arc::ptr_eq(checked, policy)) { - return true; + let covered = covers_rule(policy, proposed_rule); + checked_snapshot = Some(Arc::clone(policy)); + if covered { + return true; + } } let remaining = deadline.saturating_duration_since(tokio::time::Instant::now()); if remaining.is_zero() { @@ -1112,6 +1138,11 @@ fn network_endpoint_from_json( if endpoint.host.trim().is_empty() { return Err("endpoint.host is required".to_string()); } + if let Some(reason) = + openshell_policy::agent_authored_transport_rejection(&endpoint.protocol, &endpoint.tls) + { + return Err(reason.to_string()); + } let mut ports = endpoint.ports; if ports.is_empty() && endpoint.port > 0 { @@ -1440,6 +1471,7 @@ mod tests { assert_eq!(rule.endpoints[0].port, 443); assert_eq!(rule.endpoints[0].ports, vec![443]); assert_eq!(rule.endpoints[0].protocol, "rest"); + assert!(rule.endpoints[0].advisor_proposed); #[allow(deprecated)] { assert!(rule.binaries[0].harness); @@ -1483,6 +1515,47 @@ mod tests { assert!(!error.contains("secret")); } + #[test] + fn proposal_chunks_from_body_rejects_native_tcp_and_tls_skip() { + for endpoint in [ + r#"{"host":"db.example.com","port":5432,"protocol":"tcp"}"#, + r#"{"host":"api.example.com","port":443,"tls":"skip"}"#, + ] { + let body = format!( + r#"{{ + "operations": [{{ + "addRule": {{ + "ruleName": "raw_transport", + "rule": {{"endpoints": [{endpoint}]}} + }} + }}] + }}"# + ); + + let error = proposal_chunks_from_body(body.as_bytes()).unwrap_err(); + assert!(error.contains("administrator"), "unexpected error: {error}"); + } + } + + #[test] + fn proposal_chunks_from_body_accepts_omitted_protocol_with_default_tls() { + let body = br#"{ + "operations": [{ + "addRule": { + "ruleName": "explicit_proxy", + "rule": { + "endpoints": [{"host":"api.example.com","port":443}] + } + } + }] + }"#; + + let chunks = proposal_chunks_from_body(body).unwrap(); + let endpoint = &chunks[0].proposed_rule.as_ref().unwrap().endpoints[0]; + assert!(endpoint.protocol.is_empty()); + assert!(endpoint.tls.is_empty()); + } + #[test] fn parse_last_query_clamps_to_max() { assert_eq!(parse_last_query("last=5"), Some(5)); @@ -2036,7 +2109,7 @@ mod tests { let policy = ctx.current_policy.clone(); tokio::spawn(async move { tokio::time::sleep(std::time::Duration::from_millis(50)).await; - *policy.write().await = Some(policy_with_rule(NetworkPolicyRule { + *policy.write().await = Some(Arc::new(policy_with_rule(NetworkPolicyRule { name: "unrelated".to_string(), endpoints: vec![NetworkEndpoint { host: "api.example.com".to_string(), @@ -2048,7 +2121,7 @@ mod tests { path: "/usr/bin/curl".to_string(), ..Default::default() }], - })); + }))); }) }; @@ -2089,7 +2162,7 @@ mod tests { let target = proposed.clone(); tokio::spawn(async move { tokio::time::sleep(std::time::Duration::from_millis(100)).await; - *policy.write().await = Some(policy_with_rule(target)); + *policy.write().await = Some(Arc::new(policy_with_rule(target))); }) }; @@ -2138,6 +2211,81 @@ mod tests { ); } + #[tokio::test(start_paused = true)] + async fn wait_checks_unchanged_policy_snapshot_once() { + let proposed = proposed_curl_rule_for_github(); + let ctx = PolicyLocalContext::new( + Some(ProtoSandboxPolicy { + version: 1, + ..Default::default() + }), + None, + None, + AgentProposals::new(false), + test_workspace_rx(), + ); + let checks = std::cell::Cell::new(0_u32); + let deadline = tokio::time::Instant::now() + std::time::Duration::from_millis(650); + + let reloaded = wait_for_local_policy_to_cover_with(&ctx, &proposed, deadline, |_, _| { + checks.set(checks.get() + 1); + false + }) + .await; + + assert!(!reloaded); + assert_eq!( + checks.get(), + 1, + "an unchanged snapshot must be checked once" + ); + } + + #[tokio::test(start_paused = true)] + async fn wait_checks_each_installed_snapshot_once() { + let proposed = proposed_curl_rule_for_github(); + let empty_policy = ProtoSandboxPolicy { + version: 1, + ..Default::default() + }; + let ctx = Arc::new(PolicyLocalContext::new( + Some(empty_policy.clone()), + None, + None, + AgentProposals::new(false), + test_workspace_rx(), + )); + let reloads = Arc::clone(&ctx); + let covering_rule = proposed.clone(); + let installations = tokio::spawn(async move { + tokio::time::sleep(std::time::Duration::from_millis(250)).await; + reloads.set_current_policy(empty_policy).await; + tokio::time::sleep(std::time::Duration::from_millis(200)).await; + reloads + .set_current_policy(policy_with_rule(covering_rule)) + .await; + }); + let checks = std::cell::Cell::new(0_u32); + let deadline = tokio::time::Instant::now() + std::time::Duration::from_secs(2); + + let reloaded = + wait_for_local_policy_to_cover_with(&ctx, &proposed, deadline, |policy, rule| { + checks.set(checks.get() + 1); + openshell_policy::policy_covers_rule(policy, rule) + }) + .await; + installations + .await + .expect("policy installation task must complete"); + + assert!(reloaded, "the covering snapshot must complete the wait"); + assert_eq!( + checks.get(), + 3, + "initial, identical replacement, and covering snapshots need one check each" + ); + } + #[test] fn sanitize_reason_for_audit_strips_control_chars_and_caps_length() { // Tabs and newlines are stripped; ordinary printable chars survive; diff --git a/crates/openshell-supervisor-network/src/proxy.rs b/crates/openshell-supervisor-network/src/proxy.rs index dc2736a4e..add7be9b3 100644 --- a/crates/openshell-supervisor-network/src/proxy.rs +++ b/crates/openshell-supervisor-network/src/proxy.rs @@ -25,9 +25,8 @@ use openshell_core::policy::ProxyPolicy; use openshell_core::provider_credentials::{ProviderCredentialSnapshot, ProviderCredentialState}; use openshell_core::secrets::{self, SecretResolver, rewrite_header_line_checked}; use openshell_ocsf::{ - ActionId, ActivityId, AiModel, ApiActivityBuilder, DispositionId, Endpoint, - HttpActivityBuilder, HttpRequest, NetworkActivityBuilder, Process, SeverityId, StatusId, - Url as OcsfUrl, ocsf_emit, + ActionId, ActivityId, DispositionId, Endpoint, HttpActivityBuilder, HttpRequest, + NetworkActivityBuilder, Process, SeverityId, StatusId, Url as OcsfUrl, ocsf_emit, }; #[cfg(target_os = "linux")] use std::mem::size_of; @@ -62,8 +61,6 @@ const TUNNEL_PROTOCOL_PEEK_TIMEOUT: std::time::Duration = std::time::Duration::f const TUNNEL_PROTOCOL_PEEK_POLL: std::time::Duration = std::time::Duration::from_millis(5); #[cfg(test)] const TUNNEL_PROTOCOL_PEEK_POLL: std::time::Duration = std::time::Duration::from_millis(1); -const INFERENCE_LOCAL_HOST: &str = "inference.local"; -const INFERENCE_LOCAL_PORT: u16 = 443; const FORWARD_ENCODED_SLASH_REJECTION_DETAIL: &str = "request-target contains an encoded '/' (%2F) which is not allowed on this endpoint"; #[cfg(target_os = "linux")] @@ -132,102 +129,6 @@ const CLOUD_METADATA_IPS: &[IpAddr] = &[ IpAddr::V4(std::net::Ipv4Addr::new(169, 254, 169, 254)), ]; -/// Maximum total bytes for a streaming inference response body (32 MiB). -#[cfg(not(test))] -const MAX_STREAMING_BODY: usize = 32 * 1024 * 1024; -// Keep unit tests deterministic without pushing tens of MiB through loopback. -#[cfg(test)] -const MAX_STREAMING_BODY: usize = 1024; - -/// Idle timeout per chunk when relaying streaming inference responses. -/// -/// Reasoning models (e.g. nemotron-3-super, o1, o3) can pause for 60+ seconds -/// between "thinking" and output phases. 120s provides headroom while still -/// catching genuinely stuck streams. -#[cfg(not(test))] -const CHUNK_IDLE_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(120); -// Exercise idle-timeout truncation without slowing the full package test suite. -#[cfg(test)] -const CHUNK_IDLE_TIMEOUT: std::time::Duration = std::time::Duration::from_millis(100); - -/// Outcome of an inference interception attempt. -/// -/// Returned by [`handle_inference_interception`] so the call site can emit -/// a structured CONNECT deny log when the connection is not successfully routed. -#[derive(Debug)] -enum InferenceOutcome { - /// At least one request was successfully routed to a local inference backend. - Routed, - /// The connection was denied (TLS failure, non-inference request, etc.). - Denied { reason: String }, -} - -/// Inference routing context for sandbox-local execution. -/// -/// Holds a `Router` (HTTP client) and cached sets of resolved routes. -/// User routes serve `inference.local` traffic; system routes are consumed -/// in-process by the supervisor for platform functions (e.g. agent harness). -pub struct InferenceContext { - pub patterns: Vec, - router: openshell_router::Router, - /// Routes for the user-facing `inference.local` endpoint. - routes: Arc>>, - /// Routes for supervisor-only system inference (`sandbox-system`). - system_routes: Arc>>, -} - -impl InferenceContext { - // `router`/`routes` are intentionally distinct nouns (the router and the - // route list it consumes); both names are clearer than alternatives. - #[allow(clippy::similar_names)] - pub fn new( - patterns: Vec, - router: openshell_router::Router, - routes: Vec, - system_routes: Vec, - ) -> Self { - Self { - patterns, - router, - routes: Arc::new(tokio::sync::RwLock::new(routes)), - system_routes: Arc::new(tokio::sync::RwLock::new(system_routes)), - } - } - - /// Get a handle to the user route cache for background refresh. - pub fn route_cache( - &self, - ) -> Arc>> { - self.routes.clone() - } - - /// Get a handle to the system route cache for background refresh. - pub fn system_route_cache( - &self, - ) -> Arc>> { - self.system_routes.clone() - } - - /// Make an inference call using system routes (supervisor-only). - /// - /// This is the in-process API for platform functions. It bypasses the - /// CONNECT proxy entirely — the supervisor calls the router directly - /// from the host network namespace. - pub async fn system_inference( - &self, - protocol: &str, - method: &str, - path: &str, - headers: Vec<(String, String)>, - body: bytes::Bytes, - ) -> Result { - let routes = self.system_routes.read().await; - self.router - .proxy_with_candidates(protocol, method, path, headers, body, &routes) - .await - } -} - pub struct ProxyHandle { #[allow(dead_code)] http_addr: Option, @@ -248,7 +149,6 @@ impl ProxyHandle { identity_cache: Arc, entrypoint_pid: Arc, tls_state: Option>, - inference_ctx: Option>, provider_credentials: Option, policy_local_ctx: Option>, denial_tx: Option>, @@ -380,7 +280,6 @@ impl ProxyHandle { let cache = identity_cache.clone(); let spid = entrypoint_pid.clone(); let tls = tls_state.clone(); - let inf = inference_ctx.clone(); let policy_local = policy_local_ctx.clone(); let proposals = agent_proposals.clone(); let gw = trusted_host_gateway.clone(); @@ -404,7 +303,6 @@ impl ProxyHandle { cache, spid, tls, - inf, policy_local, proposals, gw, @@ -1688,7 +1586,6 @@ async fn handle_tcp_connection( identity_cache: Arc, entrypoint_pid: Arc, tls_state: Option>, - inference_ctx: Option>, policy_local_ctx: Option>, agent_proposals: openshell_core::proposals::AgentProposals, trusted_host_gateway: Arc>, @@ -1776,33 +1673,6 @@ async fn handle_tcp_connection( let host = normalize_host(&raw_host); let (host_lc, raw_host_lc) = (host.to_ascii_lowercase(), raw_host.to_ascii_lowercase()); - if host_lc == INFERENCE_LOCAL_HOST && port == INFERENCE_LOCAL_PORT { - respond(&mut client, b"HTTP/1.1 200 Connection Established\r\n\r\n").await?; - let outcome = handle_inference_interception( - client, - INFERENCE_LOCAL_HOST, - port, - tls_state.as_ref(), - inference_ctx.as_ref(), - ) - .await?; - if let InferenceOutcome::Denied { reason } = outcome { - emit_activity(&activity_tx, true, "forward_policy"); - let event = NetworkActivityBuilder::new(openshell_ocsf::ctx::ctx()) - .activity(ActivityId::Open) - .action(ActionId::Denied) - .disposition(DispositionId::Blocked) - .severity(SeverityId::Medium) - .status(StatusId::Failure) - .dst_endpoint(Endpoint::from_domain(INFERENCE_LOCAL_HOST, port)) - .message(format!("Inference interception denied: {reason}")) - .status_detail(&reason) - .build(); - ocsf_emit!(event); - } - return Ok(()); - } - let workload_addr = client.peer_addr().into_diagnostic()?; let proxy_addr = client.local_addr().into_diagnostic()?; let connection = crate::procfs::WorkloadProxyTcpConnection::new(workload_addr, proxy_addr); @@ -2810,600 +2680,6 @@ fn authorize_egress_intent( } } -/// Maximum buffer size for inference request parsing (10 MiB). -const MAX_INFERENCE_BUF: usize = 10 * 1024 * 1024; - -/// Initial buffer size for inference request parsing (64 KiB). -const INITIAL_INFERENCE_BUF: usize = 65536; - -/// Handle an intercepted connection for inference routing. -/// -/// TLS-terminates the client connection, parses HTTP requests, and executes -/// inference API calls locally via `openshell-router`. -/// Non-inference requests are denied with 403. -/// -/// Returns [`InferenceOutcome::Routed`] if at least one request was successfully -/// routed, or [`InferenceOutcome::Denied`] with a reason for all denial cases. -async fn handle_inference_interception( - client: TcpStream, - host: &str, - port: u16, - tls_state: Option<&Arc>, - inference_ctx: Option<&Arc>, -) -> Result { - let Some(ctx) = inference_ctx else { - return Ok(InferenceOutcome::Denied { - reason: "cluster inference context not configured".to_string(), - }); - }; - - let Some(tls) = tls_state else { - return Ok(InferenceOutcome::Denied { - reason: "missing TLS state".to_string(), - }); - }; - - // TLS-terminate the client side (present a cert for the target host) - let mut tls_client = match crate::l7::tls::tls_terminate_client(client, tls, host).await { - Ok(c) => c, - Err(e) => { - return Ok(InferenceOutcome::Denied { - reason: format!("TLS handshake failed: {e}"), - }); - } - }; - - process_inference_keepalive(&mut tls_client, ctx, port).await -} - -/// Read and process HTTP requests from a TLS-terminated inference connection. -/// -/// Each request is matched against inference patterns and routed locally. -/// Any non-inference request is immediately denied and the connection is closed, -/// even if previous requests on the same keep-alive connection were routed -/// successfully. -async fn process_inference_keepalive( - stream: &mut S, - ctx: &InferenceContext, - port: u16, -) -> Result { - use crate::l7::inference::{ParseResult, format_http_response, try_parse_http_request}; - - let mut buf = vec![0u8; INITIAL_INFERENCE_BUF]; - let mut used = 0usize; - let mut routed_any = false; - - loop { - let n = match stream.read(&mut buf[used..]).await { - Ok(n) => n, - Err(e) => { - if routed_any { - break; - } - return Ok(InferenceOutcome::Denied { - reason: format!("I/O error: {e}"), - }); - } - }; - if n == 0 { - if routed_any { - break; - } - return Ok(InferenceOutcome::Denied { - reason: "client closed connection".to_string(), - }); - } - used += n; - - // Try to parse a complete HTTP request - match try_parse_http_request(&buf[..used]) { - ParseResult::Complete(request, consumed) => { - let was_routed = route_inference_request(&request, ctx, stream).await?; - if was_routed { - routed_any = true; - } else { - // Deny and close: a non-inference request must not be silently - // ignored on a keep-alive connection that previously routed - // inference traffic. - return Ok(InferenceOutcome::Denied { - reason: "connection not allowed by policy".to_string(), - }); - } - - // Shift buffer for next request - buf.copy_within(consumed..used, 0); - used -= consumed; - } - ParseResult::Incomplete => { - // Need more data — grow buffer if full - if used == buf.len() { - if buf.len() >= MAX_INFERENCE_BUF { - let response = format_http_response(413, &[], b"Payload Too Large"); - write_all(stream, &response).await?; - if routed_any { - break; - } - return Ok(InferenceOutcome::Denied { - reason: "payload too large".to_string(), - }); - } - buf.resize((buf.len() * 2).min(MAX_INFERENCE_BUF), 0); - } - } - ParseResult::Invalid(reason) => { - { - let event = NetworkActivityBuilder::new(openshell_ocsf::ctx::ctx()) - .activity(ActivityId::Refuse) - .action(ActionId::Denied) - .disposition(DispositionId::Rejected) - .severity(SeverityId::Medium) - .status(StatusId::Failure) - .dst_endpoint(Endpoint::from_domain(INFERENCE_LOCAL_HOST, port)) - .message(format!("Rejecting malformed inference request: {reason}")) - .status_detail(&reason) - .build(); - ocsf_emit!(event); - } - let response = format_http_response(400, &[], b"Bad Request"); - write_all(stream, &response).await?; - return Ok(InferenceOutcome::Denied { reason }); - } - } - } - - Ok(InferenceOutcome::Routed) -} - -/// Extract the model name from an inference request body. -fn extract_model_from_request(body: &[u8]) -> Option { - #[derive(serde::Deserialize)] - struct Req { - model: Option, - } - serde_json::from_slice::(body) - .ok() - .and_then(|r| r.model) - .filter(|m| !m.is_empty()) -} - -/// Extract token usage from an inference response body. -fn extract_usage_from_response(body: &[u8]) -> (Option, Option) { - #[derive(serde::Deserialize)] - struct Resp { - usage: Option, - } - #[derive(serde::Deserialize)] - struct Usage { - prompt_tokens: Option, - completion_tokens: Option, - } - match serde_json::from_slice::(body) { - Ok(r) => { - let u = r.usage.unwrap_or(Usage { - prompt_tokens: None, - completion_tokens: None, - }); - (u.prompt_tokens, u.completion_tokens) - } - Err(_) => (None, None), - } -} - -/// Emit an OCSF API Activity [6003] event with the `ai_operation` profile after an inference call. -#[allow(clippy::too_many_arguments)] -fn emit_ai_inference( - method: &str, - path: &str, - route_model: Option<&str>, - route_provider: Option<&str>, - status: StatusId, - input_tokens: Option, - output_tokens: Option, - latency: std::time::Duration, -) { - let model_name = route_model.unwrap_or("unknown"); - let provider_name = route_provider.unwrap_or("unknown"); - let latency_ms = u64::try_from(latency.as_millis()).unwrap_or(u64::MAX); - - let operation = format!("{method} {path}"); - - let mut builder = ApiActivityBuilder::new(openshell_ocsf::ctx::ctx(), &operation) - .severity(SeverityId::Informational) - .status(status) - .ai_model(AiModel::new(model_name, provider_name)) - .http_request(HttpRequest::new( - method, - OcsfUrl::new("https", INFERENCE_LOCAL_HOST, path, 443), - )) - .dst_endpoint(Endpoint::from_domain( - INFERENCE_LOCAL_HOST, - INFERENCE_LOCAL_PORT, - )) - .unmapped("latency_ms", latency_ms); - - if let Some(t) = input_tokens { - builder = builder.unmapped("input_tokens", t); - } - if let Some(t) = output_tokens { - builder = builder.unmapped("output_tokens", t); - } - - let msg = format!( - "Model call: {model_name} via {provider_name} ({}in, {}out)", - input_tokens.map_or_else(|| "?".to_string(), |t| t.to_string()), - output_tokens.map_or_else(|| "?".to_string(), |t| t.to_string()), - ); - builder = builder.message(msg); - - ocsf_emit!(builder.build()); -} - -/// Route a parsed inference request locally via the sandbox router, or deny it. -/// -/// Returns `Ok(true)` if the request was routed to an inference backend, -/// `Ok(false)` if it was denied as a non-inference request. -async fn route_inference_request( - request: &crate::l7::inference::ParsedHttpRequest, - ctx: &InferenceContext, - tls_client: &mut (impl tokio::io::AsyncWrite + Unpin), -) -> Result { - use crate::l7::inference::{detect_inference_pattern, format_http_response}; - - let normalized_path = normalize_inference_path(&request.path); - - if let Some(pattern) = - detect_inference_pattern(&request.method, &normalized_path, &ctx.patterns) - { - { - let event = NetworkActivityBuilder::new(openshell_ocsf::ctx::ctx()) - .activity(ActivityId::Open) - .action(ActionId::Allowed) - .disposition(DispositionId::Detected) - .severity(SeverityId::Informational) - .status(StatusId::Success) - .dst_endpoint(Endpoint::from_domain(INFERENCE_LOCAL_HOST, 443)) - .message(format!( - "Intercepted inference request, routing locally: {} {} (protocol={}, kind={})", - request.method, normalized_path, pattern.protocol, pattern.kind - )) - .build(); - ocsf_emit!(event); - } - - let routes = ctx.routes.read().await; - - if routes.is_empty() { - let body = serde_json::json!({ - "error": "cluster inference is not configured", - "hint": "run: openshell cluster inference set --help" - }); - let body_bytes = body.to_string(); - let response = format_http_response( - 503, - &[("content-type".to_string(), "application/json".to_string())], - body_bytes.as_bytes(), - ); - write_all(tls_client, &response).await?; - return Ok(true); - } - - // Buffered protocols (embeddings, model discovery) return a single JSON - // object, not an SSE token stream. Serve them buffered with an accurate - // Content-Length: the streaming path would append an SSE error frame to - // the body on a size-cap or idle-timeout truncation, corrupting a - // payload the client parses as one JSON object. Framing is declared per - // protocol on the matched pattern. - let _req_model = extract_model_from_request(&request.body); - let normalized_protocol = pattern.protocol.to_ascii_lowercase(); - let selected_route = routes - .iter() - .find(|r| r.protocols.iter().any(|p| p == &normalized_protocol)) - .or_else(|| routes.first()); - let route_model = selected_route.map(|r| r.model.clone()); - let route_endpoint = selected_route.map(|r| r.endpoint.clone()); - let infer_start = std::time::Instant::now(); - - if pattern.is_buffered() { - match ctx - .router - .proxy_with_candidates( - &pattern.protocol, - &request.method, - &normalized_path, - request.headers.clone(), - bytes::Bytes::from(request.body.clone()), - &routes, - ) - .await - { - Ok(resp) => { - let (input_tokens, output_tokens) = extract_usage_from_response(&resp.body); - let resp_status = if (200..300).contains(&resp.status) { - StatusId::Success - } else { - StatusId::Failure - }; - emit_ai_inference( - &request.method, - &normalized_path, - resp.route_model.as_deref(), - resp.route_endpoint.as_deref(), - resp_status, - input_tokens, - output_tokens, - infer_start.elapsed(), - ); - - let resp_headers = sanitize_inference_response_headers(resp.headers); - let response = format_http_response(resp.status, &resp_headers, &resp.body); - write_all(tls_client, &response).await?; - } - Err(e) => { - emit_ai_inference( - &request.method, - &normalized_path, - route_model.as_deref(), - route_endpoint.as_deref(), - StatusId::Failure, - None, - None, - infer_start.elapsed(), - ); - write_inference_router_error(tls_client, &e).await?; - } - } - return Ok(true); - } - - match ctx - .router - .proxy_with_candidates_streaming( - &pattern.protocol, - &request.method, - &normalized_path, - request.headers.clone(), - bytes::Bytes::from(request.body.clone()), - &routes, - ) - .await - { - Ok(mut resp) => { - use crate::l7::inference::{ - format_chunk, format_chunk_terminator, format_http_response_header, - format_sse_error, - }; - - let stream_route_model = resp.route_model.take(); - let stream_route_endpoint = resp.route_endpoint.take(); - - let resp_headers = sanitize_inference_response_headers( - std::mem::take(&mut resp.headers).into_iter().collect(), - ); - - // Write response headers immediately (chunked TE). - let header_bytes = format_http_response_header(resp.status, &resp_headers); - write_all(tls_client, &header_bytes).await?; - - // Stream body chunks with byte cap and idle timeout. - // - // Each upstream chunk is wrapped in HTTP chunked framing and - // flushed immediately so SSE events reach the client without - // delay. Unlike the previous per-byte write_all+flush, we - // coalesce the framing header + data + trailer into a single - // write_all call, reducing the number of TLS records per chunk - // from 3 to 1 while preserving incremental delivery. - let resp_status_ok = (200..300).contains(&resp.status); - let mut total_bytes: usize = 0; - let mut stream_failed = false; - loop { - match tokio::time::timeout(CHUNK_IDLE_TIMEOUT, resp.next_chunk()).await { - Ok(Ok(Some(chunk))) => { - total_bytes += chunk.len(); - if total_bytes > MAX_STREAMING_BODY { - warn!( - total_bytes = total_bytes, - limit = MAX_STREAMING_BODY, - "streaming response exceeded byte limit, truncating" - ); - let err = format_sse_error( - "response truncated: exceeded maximum streaming body size", - ); - let _ = write_all(tls_client, &format_chunk(&err)).await; - stream_failed = true; - break; - } - let encoded = format_chunk(&chunk); - write_all(tls_client, &encoded).await?; - } - Ok(Ok(None)) => break, - Ok(Err(e)) => { - let event = NetworkActivityBuilder::new(openshell_ocsf::ctx::ctx()) - .activity(ActivityId::Fail) - .severity(SeverityId::Medium) - .status(StatusId::Failure) - .dst_endpoint(Endpoint::from_domain(INFERENCE_LOCAL_HOST, 443)) - .message(format!( - "error reading upstream response chunk after \ - {total_bytes} bytes: {e}" - )) - .build(); - ocsf_emit!(event); - let err = format_sse_error("response truncated: upstream read error"); - let _ = write_all(tls_client, &format_chunk(&err)).await; - stream_failed = true; - break; - } - Err(_) => { - let event = NetworkActivityBuilder::new(openshell_ocsf::ctx::ctx()) - .activity(ActivityId::Fail) - .severity(SeverityId::Medium) - .status(StatusId::Failure) - .dst_endpoint(Endpoint::from_domain(INFERENCE_LOCAL_HOST, 443)) - .message(format!( - "streaming response chunk idle timeout after \ - {total_bytes} bytes, closing" - )) - .build(); - ocsf_emit!(event); - let err = - format_sse_error("response truncated: chunk idle timeout exceeded"); - let _ = write_all(tls_client, &format_chunk(&err)).await; - stream_failed = true; - break; - } - } - } - - // Terminate the chunked stream. - write_all(tls_client, format_chunk_terminator()).await?; - - // Emit API Activity for the completed streaming call. - let stream_status = if stream_failed || !resp_status_ok { - StatusId::Failure - } else { - StatusId::Success - }; - emit_ai_inference( - &request.method, - &normalized_path, - stream_route_model.as_deref(), - stream_route_endpoint.as_deref(), - stream_status, - None, - None, - infer_start.elapsed(), - ); - } - Err(e) => { - emit_ai_inference( - &request.method, - &normalized_path, - route_model.as_deref(), - route_endpoint.as_deref(), - StatusId::Failure, - None, - None, - infer_start.elapsed(), - ); - write_inference_router_error(tls_client, &e).await?; - } - } - Ok(true) - } else { - // Not an inference request — deny - { - let event = NetworkActivityBuilder::new(openshell_ocsf::ctx::ctx()) - .activity(ActivityId::Open) - .action(ActionId::Denied) - .disposition(DispositionId::Blocked) - .severity(SeverityId::Medium) - .status(StatusId::Failure) - .dst_endpoint(Endpoint::from_domain(INFERENCE_LOCAL_HOST, 443)) - .message(format!( - "connection not allowed by policy: {} {}", - request.method, normalized_path - )) - .build(); - ocsf_emit!(event); - } - let body = serde_json::json!({"error": "connection not allowed by policy"}); - let body_bytes = body.to_string(); - let response = format_http_response( - 403, - &[("content-type".to_string(), "application/json".to_string())], - body_bytes.as_bytes(), - ); - write_all(tls_client, &response).await?; - Ok(false) - } -} - -/// Emit an OCSF failure event and write a buffered JSON error response for a -/// router error hit while proxying an inference request. -/// -/// Shared by the streaming and buffered routing paths so both surface upstream -/// failures with the same status mapping and the same audit record. -async fn write_inference_router_error( - tls_client: &mut (impl tokio::io::AsyncWrite + Unpin), - err: &openshell_router::RouterError, -) -> Result<()> { - use crate::l7::inference::format_http_response; - - let event = NetworkActivityBuilder::new(openshell_ocsf::ctx::ctx()) - .activity(ActivityId::Fail) - .severity(SeverityId::Low) - .status(StatusId::Failure) - .dst_endpoint(Endpoint::from_domain(INFERENCE_LOCAL_HOST, 443)) - .message(format!( - "inference endpoint detected but upstream service failed: {err}" - )) - .build(); - ocsf_emit!(event); - - let (status, msg) = router_error_to_http(err); - let body = serde_json::json!({ "error": msg }).to_string(); - let response = format_http_response( - status, - &[("content-type".to_string(), "application/json".to_string())], - body.as_bytes(), - ); - write_all(tls_client, &response).await -} - -/// Map router errors to HTTP status codes and sanitized messages. -/// -/// Returns generic, client-safe messages instead of verbatim internal details; -/// the full error is recorded in the OCSF failure event by the caller. -fn router_error_to_http(err: &openshell_router::RouterError) -> (u16, String) { - use openshell_router::RouterError; - match err { - RouterError::RouteNotFound(_) => (400, "no inference route configured".to_string()), - RouterError::NoCompatibleRoute(_) => { - (400, "no compatible inference route available".to_string()) - } - RouterError::Unauthorized(_) => (401, "unauthorized".to_string()), - RouterError::UpstreamUnavailable(_) => (503, "inference service unavailable".to_string()), - RouterError::UpstreamProtocol(_) | RouterError::Internal(_) => { - (502, "inference service error".to_string()) - } - } -} - -fn sanitize_inference_response_headers(headers: Vec<(String, String)>) -> Vec<(String, String)> { - headers - .into_iter() - .filter(|(name, _)| !should_strip_response_header(name)) - .collect() -} - -fn should_strip_response_header(name: &str) -> bool { - let name_lc = name.to_ascii_lowercase(); - matches!(name_lc.as_str(), "content-length") || is_hop_by_hop_header(&name_lc) -} - -fn is_hop_by_hop_header(name: &str) -> bool { - matches!( - name, - "connection" - | "keep-alive" - | "proxy-authenticate" - | "proxy-authorization" - | "proxy-connection" - | "te" - | "trailer" - | "transfer-encoding" - | "upgrade" - ) -} - -/// Write all bytes to an async writer. -async fn write_all(writer: &mut (impl tokio::io::AsyncWrite + Unpin), data: &[u8]) -> Result<()> { - use tokio::io::AsyncWriteExt; - writer.write_all(data).await.into_diagnostic()?; - writer.flush().await.into_diagnostic()?; - Ok(()) -} - fn emit_l7_tunnel_close_after_policy_change(host: &str, port: u16, error: miette::Report) { let event = NetworkActivityBuilder::new(openshell_ocsf::ctx::ctx()) .activity(ActivityId::Open) @@ -4238,23 +3514,6 @@ fn endpoint_config_string_array(config: ®orus::Value, key: &str) -> Vec String { - match crate::l7::path::canonicalize_request_target( - path, - &crate::l7::path::CanonicalizeOptions::default(), - ) { - Ok((canon, _)) => canon.path, - Err(_) => path.to_string(), - } -} - /// Extract the hostname from an absolute-form URI used in plain HTTP proxy requests. /// /// For example, `"http://example.com/path"` yields `"example.com"` and @@ -5392,9 +4651,8 @@ async fn handle_forward_proxy( raw_header: forward_request_bytes, body_length, }; - if crate::l7::jsonrpc::jsonrpc_receive_stream_request(&jsonrpc_request) { - forward_request_bytes = jsonrpc_request.raw_header; - Some(crate::l7::jsonrpc::JsonRpcRequestInfo::receive_stream()) + let info = if crate::l7::jsonrpc::jsonrpc_receive_stream_request(&jsonrpc_request) { + crate::l7::jsonrpc::JsonRpcRequestInfo::receive_stream() } else { let body = match crate::l7::http::read_body_for_inspection( client, @@ -5427,12 +4685,28 @@ async fn handle_forward_proxy( return Ok(()); } }; - forward_request_bytes = jsonrpc_request.raw_header; - Some(crate::l7::jsonrpc::parse_jsonrpc_body_with_options( + crate::l7::jsonrpc::parse_jsonrpc_body_with_options( &body, crate::l7::jsonrpc::JsonRpcInspectionOptions::for_config(&l7_config.config), - )) + ) + }; + // Forward HTTP shares the MCP transport gate with CONNECT before + // method authorization. Borrow the buffered request so checking + // the version does not copy the inspected body. + if !crate::l7::relay::enforce_mcp_protocol_version( + &l7_config.config, + &jsonrpc_request, + &info, + client, + &l7_ctx, + &telemetry_path, + ) + .await? + { + return Ok(()); } + forward_request_bytes = jsonrpc_request.raw_header; + Some(info) } else { None }; @@ -5809,7 +5083,7 @@ async fn handle_forward_proxy( ); if let Some(session) = middleware_session.take() { session - .end(openshell_core::proto::WebSocketSessionEndReason::Cancellation) + .end(openshell_core::proto::MiddlewareSessionEndReason::Cancellation) .await; } respond( @@ -5869,7 +5143,7 @@ async fn handle_forward_proxy( ); if let Some(session) = middleware_session.take() { session - .end(openshell_core::proto::WebSocketSessionEndReason::Cancellation) + .end(openshell_core::proto::MiddlewareSessionEndReason::Cancellation) .await; } if e.is_endpoint_mismatch() { @@ -5900,6 +5174,38 @@ async fn handle_forward_proxy( } }; + // Middleware and credential rewriting can change request headers. Check + // the final origin-form bytes after hop-by-hop sanitization, before an + // upstream connection exists, so forwarding preserves the MCP decision. + let rewritten = match forward_l7_reeval.as_ref() { + Some((config, _)) if config.protocol == crate::l7::L7Protocol::Mcp => { + let request = crate::l7::rest::request_from_buffered_http( + method, + middleware_path, + &upstream_target, + rewritten, + )?; + if !crate::l7::relay::enforce_final_mcp_protocol_version( + config, + &request, + client, + &l7_ctx, + &telemetry_path, + ) + .await? + { + if let Some(session) = middleware_session.take() { + session + .end(openshell_core::proto::MiddlewareSessionEndReason::Cancellation) + .await; + } + return Ok(()); + } + request.raw_header + } + _ => rewritten, + }; + if let Err(e) = forward_generation_guard.ensure_current() { warn!( host = %host_lc, @@ -5912,7 +5218,7 @@ async fn handle_forward_proxy( emit_l7_tunnel_close_after_policy_change(&host_lc, port, e); if let Some(session) = middleware_session.take() { session - .end(openshell_core::proto::WebSocketSessionEndReason::PolicyReload) + .end(openshell_core::proto::MiddlewareSessionEndReason::PolicyReload) .await; } respond( @@ -5957,7 +5263,7 @@ async fn handle_forward_proxy( ocsf_emit!(event); if let Some(session) = middleware_session.take() { session - .end(openshell_core::proto::WebSocketSessionEndReason::UpstreamRejected) + .end(openshell_core::proto::MiddlewareSessionEndReason::UpstreamFailure) .await; } respond( @@ -5986,7 +5292,7 @@ async fn handle_forward_proxy( emit_l7_tunnel_close_after_policy_change(&host_lc, port, e); if let Some(session) = middleware_session.take() { session - .end(openshell_core::proto::WebSocketSessionEndReason::PolicyReload) + .end(openshell_core::proto::MiddlewareSessionEndReason::PolicyReload) .await; } respond( @@ -6039,7 +5345,7 @@ async fn handle_forward_proxy( if let Some(error) = report.downcast_ref::() { if let Some(session) = middleware_session.take() { session - .end(openshell_core::proto::WebSocketSessionEndReason::Cancellation) + .end(openshell_core::proto::MiddlewareSessionEndReason::Cancellation) .await; } crate::l7::relay::reject_credential_resolution(client, &l7_ctx, error).await?; @@ -6081,7 +5387,7 @@ async fn handle_forward_proxy( | crate::l7::provider::RelayOutcome::Consumed => { if let Some(session) = middleware_session.take() { session - .end(openshell_core::proto::WebSocketSessionEndReason::UpstreamRejected) + .end(openshell_core::proto::MiddlewareSessionEndReason::UpstreamFailure) .await; } } @@ -6349,10 +5655,9 @@ mod tests { use super::*; use openshell_core::proposals::AgentProposals; use std::collections::HashMap as TestHashMap; - use std::future::Future; use std::net::{IpAddr, Ipv4Addr, Ipv6Addr, SocketAddr}; use std::sync::Arc; - use tokio::io::{AsyncRead, AsyncReadExt, AsyncWriteExt}; + use tokio::io::{AsyncReadExt, AsyncWriteExt}; use tokio::net::{TcpListener, TcpStream}; struct DenyWebSocketPreflight; @@ -6526,7 +5831,6 @@ network_policies: {} Arc::new(AtomicU32::new(std::process::id())), None, None, - None, AgentProposals::default(), Arc::new(None), Arc::new(None), @@ -6556,6 +5860,158 @@ network_policies: {} } } + #[tokio::test] + async fn plaintext_mcp_forwarding_preserves_initialization_and_selected_revision() { + if !cfg!(target_os = "linux") { + eprintln!("skipping: handler identity binding requires /proc (Linux)"); + return; + } + let Some(upstream_ip) = non_loopback_test_ipv4() else { + eprintln!("skipping: no routable non-loopback IPv4 test address"); + return; + }; + + for (body, version_header) in [ + ( + r#"{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-11-25","capabilities":{},"clientInfo":{"name":"test","version":"1"}}}"#, + "", + ), + ( + r#"{"jsonrpc":"2.0","id":2,"method":"tools/list"}"#, + "MCP-Protocol-Version: 2025-11-25\r\n", + ), + ] { + let upstream_listener = TcpListener::bind((upstream_ip, 0)) + .await + .expect("bind MCP upstream listener"); + let upstream_port = upstream_listener.local_addr().unwrap().port(); + let executable = std::env::current_exe().expect("current executable"); + let data = format!( + r#" +network_middlewares: + inspect: + middleware: openshell/regex + on_error: fail_closed + endpoints: + include: ["{upstream_ip}"] +network_policies: + mcp-upstream: + name: mcp-upstream + endpoints: + - host: "{upstream_ip}" + port: {upstream_port} + path: /mcp + protocol: mcp + enforcement: enforce + rules: + - allow: + method: initialize + - allow: + method: tools/list + binaries: + - {{ path: "{executable}" }} +"#, + executable = executable.display(), + ); + let engine = Arc::new( + OpaEngine::from_strings(include_str!("../data/sandbox-policy.rego"), &data) + .expect("load MCP policy"), + ); + let registry = openshell_supervisor_middleware::MiddlewareRegistry::connect_services( + openshell_supervisor_middleware_builtins::services(), + Vec::new(), + ) + .await + .expect("connect built-in middleware"); + engine + .replace_middleware_registry(registry) + .expect("install built-in middleware"); + + let upstream = tokio::spawn(async move { + let (mut socket, _) = upstream_listener.accept().await.unwrap(); + let mut request = Vec::new(); + let mut chunk = [0; 2048]; + loop { + let count = socket.read(&mut chunk).await.unwrap(); + assert_ne!(count, 0, "MCP request closed before its body completed"); + request.extend_from_slice(&chunk[..count]); + if let Some(header_end) = request + .windows(4) + .position(|window| window == b"\r\n\r\n") + .map(|end| end + 4) + && request.len() >= header_end + body.len() + { + assert_eq!(&request[header_end..], body.as_bytes()); + break; + } + } + socket + .write_all( + b"HTTP/1.1 200 OK\r\nContent-Length: 2\r\nConnection: close\r\n\r\nok", + ) + .await + .unwrap(); + String::from_utf8(request).expect("UTF-8 MCP request") + }); + let proxy_listener = TcpListener::bind("127.0.0.1:0") + .await + .expect("bind proxy listener"); + let proxy_address = proxy_listener.local_addr().unwrap(); + let target = format!("http://{upstream_ip}:{upstream_port}/mcp"); + let request = format!( + "POST {target} HTTP/1.1\r\nHost: {upstream_ip}:{upstream_port}\r\nContent-Type: application/json\r\n{version_header}Content-Length: {}\r\nConnection: close\r\n\r\n{body}", + body.len(), + ); + let client = tokio::spawn(async move { + let mut socket = TcpStream::connect(proxy_address).await.unwrap(); + let mut response = Vec::new(); + socket.read_to_end(&mut response).await.unwrap(); + response + }); + let (mut proxy_connection, _) = proxy_listener.accept().await.unwrap(); + + tokio::time::timeout( + std::time::Duration::from_secs(30), + handle_forward_proxy( + "POST", + &target, + request.as_bytes(), + request.len(), + &mut proxy_connection, + engine, + Arc::new(BinaryIdentityCache::new()), + Arc::new(AtomicU32::new(std::process::id())), + None, + AgentProposals::default(), + Arc::new(None), + None, + None, + None, + None, + None, + ), + ) + .await + .expect("MCP forwarding should complete") + .expect("handle valid MCP request"); + drop(proxy_connection); + + let response = client.await.expect("join MCP client"); + assert!(response.starts_with(b"HTTP/1.1 200 OK")); + let forwarded = upstream.await.expect("join MCP upstream"); + assert!(forwarded.starts_with("POST /mcp HTTP/1.1\r\n")); + if version_header.is_empty() { + assert!( + !forwarded + .to_ascii_lowercase() + .contains("mcp-protocol-version:") + ); + } else { + assert!(forwarded.contains(version_header)); + } + } + } + #[tokio::test] async fn plaintext_websocket_preflight_denial_does_not_connect_upstream() { if !cfg!(target_os = "linux") { @@ -6783,7 +6239,7 @@ network_policies: ) .await }); - let scenario = tokio::time::timeout(std::time::Duration::from_secs(60), async { + let scenario = tokio::time::timeout(std::time::Duration::from_mins(1), async { let (client, upstream) = tokio::join!(client, upstream); client.expect("join plaintext WebSocket client"); assert_eq!( @@ -7306,6 +6762,7 @@ network_policies: graphql_max_body_bytes: crate::l7::graphql::DEFAULT_MAX_BODY_BYTES, json_rpc_max_body_bytes: crate::l7::jsonrpc::DEFAULT_MAX_BODY_BYTES, mcp_strict_tool_names: true, + mcp_versions: Vec::new(), allow_encoded_slash: false, websocket_credential_rewrite, request_body_credential_rewrite: false, @@ -8545,6 +8002,7 @@ network_policies: graphql_max_body_bytes: crate::l7::graphql::DEFAULT_MAX_BODY_BYTES, json_rpc_max_body_bytes: crate::l7::jsonrpc::DEFAULT_MAX_BODY_BYTES, mcp_strict_tool_names: true, + mcp_versions: Vec::new(), allow_encoded_slash: false, websocket_credential_rewrite: false, request_body_credential_rewrite: false, @@ -8565,6 +8023,7 @@ network_policies: graphql_max_body_bytes: crate::l7::graphql::DEFAULT_MAX_BODY_BYTES, json_rpc_max_body_bytes: crate::l7::jsonrpc::DEFAULT_MAX_BODY_BYTES, mcp_strict_tool_names: true, + mcp_versions: Vec::new(), allow_encoded_slash: false, websocket_credential_rewrite: false, request_body_credential_rewrite: false, @@ -9362,723 +8821,6 @@ network_policies: ); } - #[tokio::test] - async fn inference_interception_applies_router_header_allowlist() { - use tokio::io::{AsyncReadExt, AsyncWriteExt}; - use tokio::net::TcpListener; - - let listener = TcpListener::bind("127.0.0.1:0").await.unwrap(); - let upstream_addr = listener.local_addr().unwrap(); - let upstream_task = tokio::spawn(async move { - use crate::l7::inference::{ParseResult, try_parse_http_request}; - - let (mut upstream, _) = listener.accept().await.unwrap(); - let mut buf = Vec::new(); - let mut chunk = [0u8; 4096]; - - loop { - let n = upstream.read(&mut chunk).await.unwrap(); - assert!(n > 0, "upstream request closed before request completed"); - buf.extend_from_slice(&chunk[..n]); - - match try_parse_http_request(&buf) { - ParseResult::Complete(_, consumed) => { - upstream - .write_all(b"HTTP/1.1 200 OK\r\nContent-Length: 2\r\n\r\nok") - .await - .unwrap(); - return String::from_utf8_lossy(&buf[..consumed]).to_string(); - } - ParseResult::Incomplete => continue, - ParseResult::Invalid(reason) => { - panic!("forwarded request should parse cleanly: {reason}"); - } - } - } - }); - - let router = openshell_router::Router::new().unwrap(); - let patterns = crate::l7::inference::default_patterns(); - let ctx = InferenceContext::new( - patterns, - router, - vec![openshell_router::config::ResolvedRoute { - name: "inference.local".to_string(), - endpoint: format!("http://{upstream_addr}"), - model: "meta/llama-3.1-8b-instruct".to_string(), - api_key: "test-api-key".to_string(), - protocols: vec!["openai_chat_completions".to_string()], - auth: openshell_router::config::AuthHeader::Bearer, - default_headers: vec![], - passthrough_headers: vec![ - "openai-organization".to_string(), - "x-model-id".to_string(), - ], - timeout: openshell_router::config::DEFAULT_ROUTE_TIMEOUT, - model_in_path: false, - request_path_override: None, - }], - vec![], - ); - - let body = r#"{"model":"ignored","messages":[{"role":"user","content":"hi"}]}"#; - let request = format!( - "POST /v1/chat/completions HTTP/1.1\r\n\ - Host: inference.local\r\n\ - Content-Type: application/json\r\n\ - OpenAI-Organization: org_123\r\n\ - Authorization: Bearer client-key\r\n\ - Cookie: session=abc\r\n\ - Content-Length: {}\r\n\r\n{}", - body.len(), - body, - ); - - let (client, mut server) = tokio::io::duplex(65536); - let (mut client_read, mut client_write) = tokio::io::split(client); - - let server_task = - tokio::spawn(async move { process_inference_keepalive(&mut server, &ctx, 443).await }); - - client_write.write_all(request.as_bytes()).await.unwrap(); - client_write.shutdown().await.unwrap(); - - let mut response = Vec::new(); - client_read.read_to_end(&mut response).await.unwrap(); - let response_text = String::from_utf8_lossy(&response); - assert!(response_text.starts_with("HTTP/1.1 200")); - - let outcome = server_task.await.unwrap().unwrap(); - assert!( - matches!(outcome, InferenceOutcome::Routed), - "expected Routed outcome, got: {outcome:?}" - ); - - let forwarded = upstream_task.await.unwrap(); - let forwarded_lc = forwarded.to_ascii_lowercase(); - assert!(forwarded_lc.contains("openai-organization: org_123")); - assert!(forwarded_lc.contains("authorization: bearer test-api-key")); - assert!(!forwarded_lc.contains("authorization: bearer client-key")); - assert!(!forwarded_lc.contains("cookie:")); - } - - fn streaming_inference_route(endpoint: String) -> openshell_router::config::ResolvedRoute { - openshell_router::config::ResolvedRoute { - name: "inference.local".to_string(), - endpoint, - model: "meta/llama-3.1-8b-instruct".to_string(), - api_key: "test-api-key".to_string(), - protocols: vec!["openai_chat_completions".to_string()], - auth: openshell_router::config::AuthHeader::Bearer, - default_headers: vec![], - passthrough_headers: vec![], - timeout: openshell_router::config::DEFAULT_ROUTE_TIMEOUT, - model_in_path: false, - request_path_override: None, - } - } - - fn embeddings_inference_route(endpoint: String) -> openshell_router::config::ResolvedRoute { - openshell_router::config::ResolvedRoute { - name: "inference.local".to_string(), - endpoint, - model: "text-embedding-3-small".to_string(), - api_key: "test-api-key".to_string(), - protocols: vec!["openai_embeddings".to_string()], - auth: openshell_router::config::AuthHeader::Bearer, - default_headers: vec![], - passthrough_headers: vec![], - timeout: openshell_router::config::DEFAULT_ROUTE_TIMEOUT, - model_in_path: false, - request_path_override: None, - } - } - - /// Embeddings responses are a single buffered JSON object, not an SSE - /// stream. They must be framed with `Content-Length` and must never be sent - /// through the chunked streaming path, whose truncation handlers would - /// append an SSE `proxy_stream_error` frame into the JSON body. - #[tokio::test] - async fn inference_embeddings_served_buffered_with_content_length() { - let listener = TcpListener::bind("127.0.0.1:0").await.unwrap(); - let upstream_addr = listener.local_addr().unwrap(); - let upstream_body = r#"{"object":"list","data":[{"object":"embedding","index":0,"embedding":[0.1,0.2]}],"model":"text-embedding-3-small"}"#; - let upstream_task = tokio::spawn(async move { - let (mut upstream, _) = listener.accept().await.unwrap(); - read_forwarded_inference_request(&mut upstream).await; - // Buffered upstream response with Content-Length (no chunked TE). - let resp = format!( - "HTTP/1.1 200 OK\r\nContent-Type: application/json\r\nContent-Length: {}\r\n\r\n{}", - upstream_body.len(), - upstream_body, - ); - upstream.write_all(resp.as_bytes()).await.unwrap(); - }); - - let router = openshell_router::Router::new().unwrap(); - let patterns = crate::l7::inference::default_patterns(); - let ctx = InferenceContext::new( - patterns, - router, - vec![embeddings_inference_route(format!( - "http://{upstream_addr}" - ))], - vec![], - ); - - let body = r#"{"model":"text-embedding-3-small","input":"hello"}"#; - let request = format!( - "POST /v1/embeddings HTTP/1.1\r\n\ - Host: inference.local\r\n\ - Content-Type: application/json\r\n\ - Content-Length: {}\r\n\r\n{}", - body.len(), - body, - ); - - let (client, mut server) = tokio::io::duplex(65536); - let (mut client_read, mut client_write) = tokio::io::split(client); - let server_task = - tokio::spawn(async move { process_inference_keepalive(&mut server, &ctx, 443).await }); - - client_write.write_all(request.as_bytes()).await.unwrap(); - client_write.shutdown().await.unwrap(); - - let mut response = Vec::new(); - client_read.read_to_end(&mut response).await.unwrap(); - let response = String::from_utf8(response).unwrap(); - - server_task.await.unwrap().unwrap(); - upstream_task.await.unwrap(); - - assert!( - response.starts_with("HTTP/1.1 200 OK\r\n"), - "expected buffered 200 response, got: {response}" - ); - let lower = response.to_ascii_lowercase(); - assert!( - lower.contains("content-length:"), - "embeddings response must be Content-Length framed, got: {response}" - ); - assert!( - !lower.contains("transfer-encoding: chunked"), - "embeddings response must NOT be chunked, got: {response}" - ); - assert!( - !response.contains("proxy_stream_error"), - "embeddings response must not carry an SSE error frame, got: {response}" - ); - assert!( - response.contains(r#""object":"list""#), - "embeddings JSON body must be forwarded intact, got: {response}" - ); - } - - fn model_discovery_inference_route( - endpoint: String, - ) -> openshell_router::config::ResolvedRoute { - openshell_router::config::ResolvedRoute { - name: "inference.local".to_string(), - endpoint, - model: "text-embedding-3-small".to_string(), - api_key: "test-api-key".to_string(), - protocols: vec!["model_discovery".to_string()], - auth: openshell_router::config::AuthHeader::Bearer, - default_headers: vec![], - passthrough_headers: vec![], - timeout: openshell_router::config::DEFAULT_ROUTE_TIMEOUT, - model_in_path: false, - request_path_override: None, - } - } - - /// `GET /v1/models` (model discovery) returns one JSON object — a model - /// list — exactly like embeddings. It must be served buffered with - /// `Content-Length`, never through the chunked streaming path whose - /// truncation handlers would append an SSE `proxy_stream_error` frame into - /// the JSON body. This guards the framing classification for the protocol. - #[tokio::test] - async fn inference_model_discovery_served_buffered_with_content_length() { - let listener = TcpListener::bind("127.0.0.1:0").await.unwrap(); - let upstream_addr = listener.local_addr().unwrap(); - let upstream_body = - r#"{"object":"list","data":[{"id":"text-embedding-3-small","object":"model"}]}"#; - let upstream_task = tokio::spawn(async move { - let (mut upstream, _) = listener.accept().await.unwrap(); - read_forwarded_inference_request(&mut upstream).await; - let resp = format!( - "HTTP/1.1 200 OK\r\nContent-Type: application/json\r\nContent-Length: {}\r\n\r\n{}", - upstream_body.len(), - upstream_body, - ); - upstream.write_all(resp.as_bytes()).await.unwrap(); - }); - - let router = openshell_router::Router::new().unwrap(); - let patterns = crate::l7::inference::default_patterns(); - let ctx = InferenceContext::new( - patterns, - router, - vec![model_discovery_inference_route(format!( - "http://{upstream_addr}" - ))], - vec![], - ); - - // GET model discovery carries no request body. - let request = "GET /v1/models HTTP/1.1\r\n\ - Host: inference.local\r\n\ - Content-Length: 0\r\n\r\n" - .to_string(); - - let (client, mut server) = tokio::io::duplex(65536); - let (mut client_read, mut client_write) = tokio::io::split(client); - let server_task = - tokio::spawn(async move { process_inference_keepalive(&mut server, &ctx, 443).await }); - - client_write.write_all(request.as_bytes()).await.unwrap(); - client_write.shutdown().await.unwrap(); - - let mut response = Vec::new(); - client_read.read_to_end(&mut response).await.unwrap(); - let response = String::from_utf8(response).unwrap(); - - server_task.await.unwrap().unwrap(); - upstream_task.await.unwrap(); - - assert!( - response.starts_with("HTTP/1.1 200 OK\r\n"), - "expected buffered 200 response, got: {response}" - ); - let lower = response.to_ascii_lowercase(); - assert!( - lower.contains("content-length:"), - "model discovery response must be Content-Length framed, got: {response}" - ); - assert!( - !lower.contains("transfer-encoding: chunked"), - "model discovery response must NOT be chunked, got: {response}" - ); - assert!( - !response.contains("proxy_stream_error"), - "model discovery response must not carry an SSE error frame, got: {response}" - ); - assert!( - response.contains(r#""object":"list""#), - "model discovery JSON body must be forwarded intact, got: {response}" - ); - } - - /// `GET /v1/models/{id}` (model discovery glob) must forward the model id in - /// the path through the buffered path with the id intact, never streamed. - #[tokio::test] - async fn inference_model_discovery_glob_path_served_buffered() { - let listener = TcpListener::bind("127.0.0.1:0").await.unwrap(); - let upstream_addr = listener.local_addr().unwrap(); - let upstream_body = r#"{"id":"gpt-4.1","object":"model"}"#; - let upstream_task = tokio::spawn(async move { - let (mut upstream, _) = listener.accept().await.unwrap(); - let forwarded = read_forwarded_request_line(&mut upstream).await; - let resp = format!( - "HTTP/1.1 200 OK\r\nContent-Type: application/json\r\nContent-Length: {}\r\n\r\n{}", - upstream_body.len(), - upstream_body, - ); - upstream.write_all(resp.as_bytes()).await.unwrap(); - forwarded - }); - - let router = openshell_router::Router::new().unwrap(); - let patterns = crate::l7::inference::default_patterns(); - let ctx = InferenceContext::new( - patterns, - router, - vec![model_discovery_inference_route(format!( - "http://{upstream_addr}" - ))], - vec![], - ); - - let request = "GET /v1/models/gpt-4.1 HTTP/1.1\r\n\ - Host: inference.local\r\n\ - Content-Length: 0\r\n\r\n" - .to_string(); - let (client, mut server) = tokio::io::duplex(65536); - let (mut client_read, mut client_write) = tokio::io::split(client); - let server_task = - tokio::spawn(async move { process_inference_keepalive(&mut server, &ctx, 443).await }); - client_write.write_all(request.as_bytes()).await.unwrap(); - client_write.shutdown().await.unwrap(); - let mut response = Vec::new(); - client_read.read_to_end(&mut response).await.unwrap(); - let response = String::from_utf8(response).unwrap(); - server_task.await.unwrap().unwrap(); - let (method, forwarded_path) = upstream_task.await.unwrap(); - - assert_eq!(method, "GET"); - assert_eq!( - forwarded_path, "/v1/models/gpt-4.1", - "the model id in the glob path must be forwarded intact" - ); - let lower = response.to_ascii_lowercase(); - assert!( - response.starts_with("HTTP/1.1 200 OK\r\n") - && lower.contains("content-length:") - && !lower.contains("transfer-encoding: chunked") - && !response.contains("proxy_stream_error"), - "glob model discovery must be buffered and Content-Length framed, got: {response}" - ); - } - - /// A failed model-discovery upstream must produce a buffered, Content-Length - /// framed JSON error, never a chunked SSE `proxy_stream_error` frame. - #[tokio::test] - async fn inference_model_discovery_error_served_buffered() { - // A port with no listener so the upstream connection is refused. - let dead_addr = { - let listener = TcpListener::bind("127.0.0.1:0").await.unwrap(); - let addr = listener.local_addr().unwrap(); - drop(listener); - addr - }; - - let router = openshell_router::Router::new().unwrap(); - let patterns = crate::l7::inference::default_patterns(); - let ctx = InferenceContext::new( - patterns, - router, - vec![model_discovery_inference_route(format!( - "http://{dead_addr}" - ))], - vec![], - ); - - let request = "GET /v1/models HTTP/1.1\r\n\ - Host: inference.local\r\n\ - Content-Length: 0\r\n\r\n" - .to_string(); - let (client, mut server) = tokio::io::duplex(65536); - let (mut client_read, mut client_write) = tokio::io::split(client); - let server_task = - tokio::spawn(async move { process_inference_keepalive(&mut server, &ctx, 443).await }); - client_write.write_all(request.as_bytes()).await.unwrap(); - client_write.shutdown().await.unwrap(); - let mut response = Vec::new(); - client_read.read_to_end(&mut response).await.unwrap(); - let response = String::from_utf8(response).unwrap(); - server_task.await.unwrap().unwrap(); - - let lower = response.to_ascii_lowercase(); - assert!( - response.starts_with("HTTP/1.1 5"), - "a refused upstream should yield a 5xx, got: {response}" - ); - assert!( - lower.contains("content-length:") - && !lower.contains("transfer-encoding: chunked") - && !response.contains("proxy_stream_error"), - "buffered model-discovery error must be Content-Length framed JSON, got: {response}" - ); - assert!( - response.contains("error"), - "error response should carry a JSON error body, got: {response}" - ); - } - - async fn read_forwarded_inference_request(stream: &mut S) { - use crate::l7::inference::{ParseResult, try_parse_http_request}; - - let mut buf = Vec::new(); - let mut chunk = [0u8; 4096]; - loop { - let n = stream.read(&mut chunk).await.unwrap(); - assert!(n > 0, "upstream request closed before completion"); - buf.extend_from_slice(&chunk[..n]); - - match try_parse_http_request(&buf) { - ParseResult::Complete(_, _) => return, - ParseResult::Incomplete => continue, - ParseResult::Invalid(reason) => { - panic!("forwarded request should parse cleanly: {reason}"); - } - } - } - } - - /// Like [`read_forwarded_inference_request`] but returns the forwarded - /// request line (method, path) so a test can assert the upstream URL path. - async fn read_forwarded_request_line(stream: &mut S) -> (String, String) { - use crate::l7::inference::{ParseResult, try_parse_http_request}; - - let mut buf = Vec::new(); - let mut chunk = [0u8; 4096]; - loop { - let n = stream.read(&mut chunk).await.unwrap(); - assert!(n > 0, "upstream request closed before completion"); - buf.extend_from_slice(&chunk[..n]); - - match try_parse_http_request(&buf) { - ParseResult::Complete(req, _) => return (req.method, req.path), - ParseResult::Incomplete => continue, - ParseResult::Invalid(reason) => { - panic!("forwarded request should parse cleanly: {reason}"); - } - } - } - } - - async fn run_live_streaming_inference(serve_upstream: F) -> String - where - F: FnOnce(TcpStream) -> Fut + Send + 'static, - Fut: Future + Send + 'static, - { - let listener = TcpListener::bind("127.0.0.1:0").await.unwrap(); - let upstream_addr = listener.local_addr().unwrap(); - let upstream_task = tokio::spawn(async move { - let (mut upstream, _) = listener.accept().await.unwrap(); - read_forwarded_inference_request(&mut upstream).await; - serve_upstream(upstream).await; - }); - - let router = openshell_router::Router::new().unwrap(); - let patterns = crate::l7::inference::default_patterns(); - let ctx = InferenceContext::new( - patterns, - router, - vec![streaming_inference_route(format!("http://{upstream_addr}"))], - vec![], - ); - - let body = r#"{"model":"ignored","messages":[{"role":"user","content":"hi"}]}"#; - let request = format!( - "POST /v1/chat/completions HTTP/1.1\r\n\ - Host: inference.local\r\n\ - Content-Type: application/json\r\n\ - Accept: text/event-stream\r\n\ - Content-Length: {}\r\n\r\n{}", - body.len(), - body, - ); - - let (client, mut server) = tokio::io::duplex(65536); - let (mut client_read, mut client_write) = tokio::io::split(client); - let server_task = - tokio::spawn(async move { process_inference_keepalive(&mut server, &ctx, 443).await }); - - client_write.write_all(request.as_bytes()).await.unwrap(); - client_write.shutdown().await.unwrap(); - - let mut response = Vec::new(); - client_read.read_to_end(&mut response).await.unwrap(); - - let outcome = server_task.await.unwrap().unwrap(); - assert!( - matches!(outcome, InferenceOutcome::Routed), - "expected Routed outcome, got: {outcome:?}" - ); - upstream_task.await.unwrap(); - - String::from_utf8(response).unwrap() - } - - fn assert_streaming_sse_error(response: &str, message: &str) { - assert!( - response.starts_with("HTTP/1.1 200 OK\r\n"), - "expected successful streaming response, got: {response}" - ); - assert!( - response - .to_ascii_lowercase() - .contains("transfer-encoding: chunked"), - "expected chunked streaming response, got: {response}" - ); - assert!( - response.contains("\"type\":\"proxy_stream_error\""), - "expected proxy_stream_error SSE event, got: {response}" - ); - assert!( - response.contains(&format!("\"message\":\"{message}\"")), - "expected SSE message {message:?}, got: {response}" - ); - assert!( - response.ends_with("0\r\n\r\n"), - "streaming response must end with chunked terminator, got: {response}" - ); - } - - #[tokio::test] - async fn inference_stream_byte_limit_injects_sse_error() { - let response = run_live_streaming_inference(|mut upstream| async move { - use crate::l7::inference::{format_chunk, format_chunk_terminator}; - - upstream - .write_all( - b"HTTP/1.1 200 OK\r\n\ - Content-Type: text/event-stream\r\n\ - Transfer-Encoding: chunked\r\n\r\n", - ) - .await - .unwrap(); - let body = vec![b'a'; MAX_STREAMING_BODY + 1]; - let _ = upstream.write_all(&format_chunk(&body)).await; - let _ = upstream.write_all(format_chunk_terminator()).await; - }) - .await; - - assert_streaming_sse_error( - &response, - "response truncated: exceeded maximum streaming body size", - ); - } - - #[tokio::test] - async fn inference_stream_upstream_read_error_injects_sse_error() { - let response = run_live_streaming_inference(|mut upstream| async move { - upstream - .write_all( - b"HTTP/1.1 200 OK\r\n\ - Content-Type: text/event-stream\r\n\ - Content-Length: 64\r\n\r\n\ - partial", - ) - .await - .unwrap(); - }) - .await; - - assert!( - response.contains("partial"), - "expected initial upstream bytes before truncation, got: {response}" - ); - assert_streaming_sse_error(&response, "response truncated: upstream read error"); - } - - #[tokio::test] - async fn inference_stream_idle_timeout_injects_sse_error() { - let response = run_live_streaming_inference(|mut upstream| async move { - upstream - .write_all( - b"HTTP/1.1 200 OK\r\n\ - Content-Type: text/event-stream\r\n\ - Transfer-Encoding: chunked\r\n\r\n", - ) - .await - .unwrap(); - tokio::time::sleep(CHUNK_IDLE_TIMEOUT + std::time::Duration::from_millis(50)).await; - }) - .await; - - assert_streaming_sse_error(&response, "response truncated: chunk idle timeout exceeded"); - } - - // -- router_error_to_http -- - - #[test] - fn router_error_route_not_found_maps_to_400() { - let err = openshell_router::RouterError::RouteNotFound("local".into()); - let (status, msg) = router_error_to_http(&err); - assert_eq!(status, 400); - assert_eq!(msg, "no inference route configured"); - // SEC-008: must NOT leak the route hint to sandboxed code - assert!(!msg.contains("local")); - } - - #[test] - fn router_error_no_compatible_route_maps_to_400() { - let err = openshell_router::RouterError::NoCompatibleRoute("anthropic_messages".into()); - let (status, msg) = router_error_to_http(&err); - assert_eq!(status, 400); - assert_eq!(msg, "no compatible inference route available"); - // SEC-008: must NOT leak the protocol name to sandboxed code - assert!(!msg.contains("anthropic_messages")); - } - - #[test] - fn router_error_unauthorized_maps_to_401() { - let err = - openshell_router::RouterError::Unauthorized("bad token from 10.0.0.5:8080".into()); - let (status, msg) = router_error_to_http(&err); - assert_eq!(status, 401); - assert_eq!(msg, "unauthorized"); - // SEC-008: must NOT leak upstream details to sandboxed code - assert!(!msg.contains("10.0.0.5")); - } - - #[test] - fn router_error_upstream_unavailable_maps_to_503() { - let err = openshell_router::RouterError::UpstreamUnavailable( - "connection refused to 10.0.0.5:8080".into(), - ); - let (status, msg) = router_error_to_http(&err); - assert_eq!(status, 503); - assert_eq!(msg, "inference service unavailable"); - // SEC-008: must NOT leak upstream address to sandboxed code - assert!(!msg.contains("10.0.0.5")); - } - - #[test] - fn router_error_upstream_protocol_maps_to_502() { - let err = openshell_router::RouterError::UpstreamProtocol( - "TLS handshake failed for nim.internal.svc:443".into(), - ); - let (status, msg) = router_error_to_http(&err); - assert_eq!(status, 502); - assert_eq!(msg, "inference service error"); - // SEC-008: must NOT leak internal hostnames to sandboxed code - assert!(!msg.contains("nim.internal")); - } - - #[test] - fn router_error_internal_maps_to_502() { - let err = openshell_router::RouterError::Internal( - "failed to read /etc/openshell/routes.json".into(), - ); - let (status, msg) = router_error_to_http(&err); - assert_eq!(status, 502); - assert_eq!(msg, "inference service error"); - // SEC-008: must NOT leak file paths to sandboxed code - assert!(!msg.contains("/etc/openshell")); - } - - #[test] - fn sanitize_response_headers_strips_hop_by_hop() { - let headers = vec![ - ("transfer-encoding".to_string(), "chunked".to_string()), - ("content-length".to_string(), "128".to_string()), - ("connection".to_string(), "keep-alive".to_string()), - ("content-type".to_string(), "text/event-stream".to_string()), - ("cache-control".to_string(), "no-cache".to_string()), - ]; - - let kept = sanitize_inference_response_headers(headers); - - assert!( - kept.iter() - .all(|(k, _)| !k.eq_ignore_ascii_case("transfer-encoding")), - "transfer-encoding should be stripped" - ); - assert!( - kept.iter() - .all(|(k, _)| !k.eq_ignore_ascii_case("content-length")), - "content-length should be stripped" - ); - assert!( - kept.iter() - .all(|(k, _)| !k.eq_ignore_ascii_case("connection")), - "connection should be stripped" - ); - assert!( - kept.iter() - .any(|(k, _)| k.eq_ignore_ascii_case("content-type")), - "content-type should be preserved" - ); - assert!( - kept.iter() - .any(|(k, _)| k.eq_ignore_ascii_case("cache-control")), - "cache-control should be preserved" - ); - } - // -- is_always_blocked_ip -- #[test] @@ -11782,87 +10524,6 @@ network_policies: assert!(result.is_empty()); } - /// Regression test: exercises the actual keep-alive interception loop to - /// verify that a non-inference request is denied even after a previous - /// inference request was successfully routed on the same connection. - /// - /// Before the fix, `handle_inference_interception` used - /// `else if !routed_any` which silently dropped denials once `routed_any` - /// was true, allowing non-inference HTTP requests to piggyback on a - /// keep-alive connection that had previously handled inference traffic. - /// Regression test: exercises the actual keep-alive interception loop to - /// verify that a non-inference request is denied even after a previous - /// inference request was successfully routed on the same connection. - /// - /// The server runs in a spawned task with empty routes (the inference - /// request gets a 503 "not configured" but is still recognized as - /// inference and returns Ok(true)). The client sends the inference - /// request, reads the 503 response, then sends a non-inference request - /// on the same connection. The server must return Denied. - #[tokio::test(flavor = "multi_thread", worker_threads = 2)] - async fn test_keepalive_denies_non_inference_after_routed() { - use openshell_router::Router; - use tokio::io::{AsyncReadExt, AsyncWriteExt}; - - let router = Router::new().unwrap(); - let patterns = crate::l7::inference::default_patterns(); - // Empty routes: inference request gets 503 but returns Ok(true). - let ctx = InferenceContext::new(patterns, router, vec![], vec![]); - - let body = r#"{"model":"test","messages":[{"role":"user","content":"hi"}]}"#; - let inference_req = format!( - "POST /v1/chat/completions HTTP/1.1\r\n\ - Host: inference.local\r\n\ - Content-Type: application/json\r\n\ - Content-Length: {}\r\n\r\n{}", - body.len(), - body, - ); - let non_inference_req = "GET /admin/config HTTP/1.1\r\nHost: inference.local\r\n\r\n"; - - let (client, mut server) = tokio::io::duplex(65536); - let (mut client_read, mut client_write) = tokio::io::split(client); - - // Spawn the server task so it runs concurrently. - let server_task = - tokio::spawn(async move { process_inference_keepalive(&mut server, &ctx, 443).await }); - - // Client: send inference request, read response, send non-inference. - client_write - .write_all(inference_req.as_bytes()) - .await - .unwrap(); - - // Read the 503 response so the server loops back to read. - let mut buf = vec![0u8; 4096]; - let _ = client_read.read(&mut buf).await.unwrap(); - - // Send non-inference request on the same keep-alive connection. - client_write - .write_all(non_inference_req.as_bytes()) - .await - .unwrap(); - drop(client_write); - - // Drain remaining response bytes. - tokio::spawn(async move { - let mut buf = vec![0u8; 4096]; - loop { - match client_read.read(&mut buf).await { - Ok(0) | Err(_) => break, - Ok(_) => continue, - } - } - }); - - let outcome = server_task.await.unwrap().unwrap(); - - assert!( - matches!(outcome, InferenceOutcome::Denied { .. }), - "expected Denied after non-inference request on keep-alive, got: {outcome:?}" - ); - } - // -- build_json_error_response -- #[test] @@ -12095,7 +10756,6 @@ network_policies: cache, entrypoint_pid, None, // tls_state — ephemeral CA unavailable - None, // inference_ctx None, // policy_local_ctx AgentProposals::default(), // agent_proposals Arc::new(None), // trusted_host_gateway @@ -12163,7 +10823,6 @@ network_policies: Arc::new(AtomicU32::new(std::process::id())), None, None, - None, AgentProposals::default(), Arc::new(None), Arc::new(None), diff --git a/crates/openshell-supervisor-network/src/proxy/relay.rs b/crates/openshell-supervisor-network/src/proxy/relay.rs index 9e00cfa25..d5d182862 100644 --- a/crates/openshell-supervisor-network/src/proxy/relay.rs +++ b/crates/openshell-supervisor-network/src/proxy/relay.rs @@ -409,6 +409,7 @@ mod tests { graphql_max_body_bytes: crate::l7::graphql::DEFAULT_MAX_BODY_BYTES, json_rpc_max_body_bytes: crate::l7::jsonrpc::DEFAULT_MAX_BODY_BYTES, mcp_strict_tool_names: true, + mcp_versions: Vec::new(), allow_encoded_slash: false, websocket_credential_rewrite: false, request_body_credential_rewrite: false, diff --git a/crates/openshell-supervisor-network/src/proxy/tests/compatibility.rs b/crates/openshell-supervisor-network/src/proxy/tests/compatibility.rs index 186d15608..30ee3d701 100644 --- a/crates/openshell-supervisor-network/src/proxy/tests/compatibility.rs +++ b/crates/openshell-supervisor-network/src/proxy/tests/compatibility.rs @@ -545,7 +545,6 @@ network_policies: Arc::new(AtomicU32::new(0)), None, None, - None, AgentProposals::default(), Arc::new(None), Arc::new(None), diff --git a/crates/openshell-supervisor-network/src/run.rs b/crates/openshell-supervisor-network/src/run.rs index 2a71702b4..af9ffb840 100644 --- a/crates/openshell-supervisor-network/src/run.rs +++ b/crates/openshell-supervisor-network/src/run.rs @@ -4,7 +4,7 @@ //! Networking stack startup for the sandbox. //! //! Builds the network namespace (Linux), the CONNECT proxy with TLS L7 -//! interception, the inference context, and wires the proxy to the +//! interception and wires the proxy to the //! caller-supplied denial-event channel. Returns a [`Networking`] handle //! whose RAII fields keep the proxy task alive for the lifetime of the //! sandbox supervisor. @@ -176,8 +176,8 @@ pub struct Networking { /// # Errors /// /// Returns an error if proxy mode is requested but the proxy configuration, -/// OPA engine, or identity cache is missing, if inference route resolution -/// fails, or if the proxy server fails to start. +/// OPA engine, or identity cache is missing, or if the proxy server fails to +/// start. #[allow(clippy::too_many_arguments)] pub async fn run_networking( policy: &SandboxPolicy, @@ -190,7 +190,6 @@ pub async fn run_networking( sandbox_id: Option<&str>, sandbox_name: Option<&str>, openshell_endpoint: Option<&str>, - #[allow(unused_variables)] inference_routes: Option<&str>, denial_tx: Option>, activity_tx: Option, agent_proposals: AgentProposals, @@ -425,14 +424,6 @@ pub async fn run_networking( SocketAddr::new(ip, port) }); - // Build inference context for local routing of intercepted inference calls. - let inference_ctx = crate::inference_routes::build_inference_context( - sandbox_id, - openshell_endpoint, - inference_routes, - ) - .await?; - let proxy_handle = ProxyHandle::start_with_bind_addr( proxy_policy, bind_addr, @@ -440,7 +431,6 @@ pub async fn run_networking( cache, entrypoint_pid.clone(), tls_state, - inference_ctx, Some(provider_credentials.clone()), Some(policy_local_ctx.clone()), denial_tx.clone(), diff --git a/crates/openshell-supervisor-network/src/sigv4.rs b/crates/openshell-supervisor-network/src/sigv4.rs index 40cc189dc..79e4c32e1 100644 --- a/crates/openshell-supervisor-network/src/sigv4.rs +++ b/crates/openshell-supervisor-network/src/sigv4.rs @@ -28,6 +28,13 @@ fn looks_like_region(s: &str) -> bool { /// hostnames. The region is the label immediately before `amazonaws.com` /// (or `amazonaws.com.cn`). pub fn extract_aws_region(host: &str) -> Option { + // The global S3 endpoint and its virtual-hosted bucket form always sign + // in us-east-1. This keeps policy metadata consistent across overlapping + // S3 hostname patterns while preserving correct global endpoint signing. + if host == "s3.amazonaws.com" || host.ends_with(".s3.amazonaws.com") { + return Some("us-east-1".to_string()); + } + let parts: Vec<&str> = host.split('.').collect(); // China partition: *.amazonaws.com.cn if parts.len() >= 5 @@ -386,13 +393,19 @@ mod tests { } #[test] - fn global_endpoint_returns_none() { - assert!(extract_aws_region("s3.amazonaws.com").is_none()); + fn global_s3_endpoint_uses_us_east_1() { + assert_eq!( + extract_aws_region("s3.amazonaws.com").as_deref(), + Some("us-east-1") + ); } #[test] - fn virtual_hosted_global_endpoint_returns_none() { - assert!(extract_aws_region("my-bucket.s3.amazonaws.com").is_none()); + fn virtual_hosted_global_s3_endpoint_uses_us_east_1() { + assert_eq!( + extract_aws_region("my-bucket.s3.amazonaws.com").as_deref(), + Some("us-east-1") + ); } #[test] diff --git a/crates/openshell-supervisor-network/src/token_grant.rs b/crates/openshell-supervisor-network/src/token_grant.rs index 15aea6fcc..4e4237f68 100644 --- a/crates/openshell-supervisor-network/src/token_grant.rs +++ b/crates/openshell-supervisor-network/src/token_grant.rs @@ -49,6 +49,7 @@ use spiffe::WorkloadApiClient; /// Token cache shared across all provider token grants. static TOKEN_CACHE: LazyLock = LazyLock::new(TokenCache::new); static TOKEN_GRANT_HTTP_CLIENT: LazyLock = LazyLock::new(|| { + let _ = rustls::crypto::aws_lc_rs::default_provider().install_default(); reqwest::Client::builder() .timeout(Duration::from_secs(30)) .connect_timeout(Duration::from_secs(30)) diff --git a/crates/openshell-supervisor-network/src/upstream_proxy.rs b/crates/openshell-supervisor-network/src/upstream_proxy.rs index 253fa5226..9cb5481df 100644 --- a/crates/openshell-supervisor-network/src/upstream_proxy.rs +++ b/crates/openshell-supervisor-network/src/upstream_proxy.rs @@ -612,31 +612,12 @@ fn parse_proxy_url(raw: &str, var_name: &str) -> Result<(ProxyEndpoint, bool), S /// fall-back to the built-in roots that would quietly weaken the trust /// boundary. The error names `var_name` so the operator can locate the setting. pub(crate) fn read_proxy_ca_bundle(path: &str, var_name: &str) -> Result { - let pem = std::fs::read_to_string(path) - .map_err(|err| format!("{var_name} '{path}' could not be read: {err}"))?; - // Validate that the bundle contributes at least one trust anchor that - // rustls actually accepts, not just that PEM framing base64-decodes. - // A PEM block with invalid DER passes `rustls_pemfile::certs` but is - // silently rejected by `RootCertStore::add_parsable_certificates`; - // counting only PEM blocks would let such a bundle satisfy the check - // while contributing zero usable anchors at runtime. - let certs: Vec<_> = rustls_pemfile::certs(&mut pem.as_bytes()) - .flatten() - .collect(); - if certs.is_empty() { - return Err(format!( - "{var_name} '{path}' contains no PEM certificate blocks" - )); - } - let mut store = rustls::RootCertStore::empty(); - let (added, _ignored) = store.add_parsable_certificates(certs); - if added == 0 { - return Err(format!( - "{var_name} '{path}' contains no usable trust anchors \ - (PEM blocks were found but none contain valid X.509 DER)" - )); - } - Ok(pem) + // Shared with the compute driver, which validates the same file on the + // gateway host before staging it, so host acceptance and guest acceptance + // cannot diverge. It also bounds the read: the file arrives from the + // driver, but a bundle the size of the sandbox disk should fail rather + // than be loaded whole. + openshell_core::driver_utils::read_upstream_proxy_ca_bundle_file(path, var_name) } /// Build the TLS client config used to connect to an `https://` corporate @@ -1148,13 +1129,6 @@ mod tests { config_from(pairs).unwrap().unwrap() } - /// Install the process-wide rustls crypto provider once. Building a - /// `ClientConfig` (for an `https://` proxy) requires it; the install is - /// idempotent, so tests that build TLS configs call this first. - fn install_crypto_provider() { - let _ = rustls::crypto::ring::default_provider().install_default(); - } - #[test] fn no_env_yields_none() { assert!(config_from(&[]).unwrap().is_none()); @@ -1289,7 +1263,6 @@ mod tests { #[test] fn https_proxy_scheme_enables_tls_and_requires_explicit_port() { - install_crypto_provider(); // An explicit port is required (no scheme-default fallback), matching // the http:// grammar. let err = config_from(&[(HTTPS_PROXY, "https://proxy.corp.com")]).unwrap_err(); @@ -1333,7 +1306,6 @@ mod tests { #[test] fn unreadable_ca_bundle_is_fatal_for_https_proxy() { - install_crypto_provider(); let err = config_from(&[ (HTTPS_PROXY, "https://proxy.corp.com:3130"), (PROXY_CA_BUNDLE, "/nonexistent/proxy-ca.pem"), @@ -1357,7 +1329,6 @@ mod tests { #[test] fn ca_bundle_with_no_certificates_is_fatal() { - install_crypto_provider(); let bundle = tempfile::NamedTempFile::new().unwrap(); std::fs::write(bundle.path(), "not a certificate\n").unwrap(); let path = bundle.path().to_string_lossy().into_owned(); @@ -1371,7 +1342,6 @@ mod tests { #[test] fn ca_bundle_with_invalid_der_certificates_is_fatal() { - install_crypto_provider(); let bundle = tempfile::NamedTempFile::new().unwrap(); std::fs::write( bundle.path(), @@ -1458,7 +1428,6 @@ mod tests { #[test] fn auth_file_without_insecure_acknowledgement_is_allowed_for_https_proxy() { - install_crypto_provider(); let file = tempfile::NamedTempFile::new().unwrap(); std::fs::write(file.path(), "user:secret\n").unwrap(); let path = file.path().to_str().unwrap().to_string(); @@ -2032,8 +2001,6 @@ mod tests { const SERVER_HOSTNAME: &str = "upstream.example.test"; - let _ = rustls::crypto::ring::default_provider().install_default(); - // Trusted CA; the client config trusts it, and the fake upstream // server presents a leaf for SERVER_HOSTNAME signed by it. let ca = tls::SandboxCa::generate().unwrap(); @@ -2265,7 +2232,6 @@ mod tests { /// the server task (yielding the received CONNECT request), and the /// server certificate PEM to use as the corporate CA bundle. async fn fake_tls_proxy() -> (SocketAddr, tokio::task::JoinHandle, String) { - install_crypto_provider(); let key = rcgen::KeyPair::generate().unwrap(); let cert = rcgen::CertificateParams::new(vec!["127.0.0.1".to_string()]) .unwrap() diff --git a/crates/openshell-supervisor-network/tests/system_inference.rs b/crates/openshell-supervisor-network/tests/system_inference.rs deleted file mode 100644 index 3c8e6ee8f..000000000 --- a/crates/openshell-supervisor-network/tests/system_inference.rs +++ /dev/null @@ -1,167 +0,0 @@ -// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -// SPDX-License-Identifier: Apache-2.0 - -//! Integration test for the in-process system inference API. -//! -//! Uses the router's built-in `mock://` route support to verify the full -//! in-process path: route selection → `proxy_with_candidates()` → -//! mock backend → response. - -use openshell_router::Router; -use openshell_router::config::{AuthHeader, ResolvedRoute}; -use openshell_supervisor_network::proxy::InferenceContext; - -fn make_system_route() -> ResolvedRoute { - ResolvedRoute { - name: "sandbox-system".to_string(), - endpoint: "mock://system-test".to_string(), - model: "system/policy-analyzer".to_string(), - api_key: "system-key".to_string(), - protocols: vec!["openai_chat_completions".to_string()], - auth: AuthHeader::Bearer, - default_headers: Vec::new(), - passthrough_headers: Vec::new(), - timeout: openshell_router::config::DEFAULT_ROUTE_TIMEOUT, - model_in_path: false, - request_path_override: None, - } -} - -fn make_user_route() -> ResolvedRoute { - ResolvedRoute { - name: "inference.local".to_string(), - endpoint: "mock://user-test".to_string(), - model: "user/gpt-4o".to_string(), - api_key: "user-key".to_string(), - protocols: vec!["openai_chat_completions".to_string()], - auth: AuthHeader::Bearer, - default_headers: Vec::new(), - passthrough_headers: Vec::new(), - timeout: openshell_router::config::DEFAULT_ROUTE_TIMEOUT, - model_in_path: false, - request_path_override: None, - } -} - -#[tokio::test] -async fn system_inference_routes_to_mock_backend() { - let router = Router::new().unwrap(); - let patterns = openshell_supervisor_network::l7::inference::default_patterns(); - - let ctx = InferenceContext::new( - patterns, - router, - vec![make_user_route()], - vec![make_system_route()], - ); - - let body = serde_json::to_vec(&serde_json::json!({ - "model": "anything", - "messages": [{"role": "user", "content": "analyze this policy"}] - })) - .unwrap(); - - let response = ctx - .system_inference( - "openai_chat_completions", - "POST", - "/v1/chat/completions", - vec![("content-type".to_string(), "application/json".to_string())], - bytes::Bytes::from(body), - ) - .await - .expect("system_inference should succeed"); - - assert_eq!(response.status, 200); - - // Verify the mock response came back with the system model - let resp_body: serde_json::Value = serde_json::from_slice(&response.body).unwrap(); - assert_eq!(resp_body["model"], "system/policy-analyzer"); - - // Verify the mock header is present - assert!( - response - .headers - .iter() - .any(|(k, v)| k == "x-openshell-mock" && v == "true") - ); -} - -#[tokio::test] -async fn system_inference_uses_system_routes_not_user_routes() { - let router = Router::new().unwrap(); - let patterns = openshell_supervisor_network::l7::inference::default_patterns(); - - // Only user routes configured — no system routes - let ctx = InferenceContext::new(patterns, router, vec![make_user_route()], vec![]); - - let body = serde_json::to_vec(&serde_json::json!({ - "model": "gpt-4o", - "messages": [{"role": "user", "content": "hello"}] - })) - .unwrap(); - - let result = ctx - .system_inference( - "openai_chat_completions", - "POST", - "/v1/chat/completions", - vec![], - bytes::Bytes::from(body), - ) - .await; - - // Should fail because the system route cache is empty — user routes - // are not accessible through system_inference(). - assert!( - result.is_err(), - "system_inference should fail when no system routes are configured" - ); -} - -#[tokio::test] -async fn system_inference_with_anthropic_protocol() { - let router = Router::new().unwrap(); - let patterns = openshell_supervisor_network::l7::inference::default_patterns(); - - let system_route = ResolvedRoute { - name: "sandbox-system".to_string(), - endpoint: "mock://anthropic-system".to_string(), - model: "claude-sonnet-4-20250514".to_string(), - api_key: "ant-key".to_string(), - protocols: vec!["anthropic_messages".to_string()], - auth: AuthHeader::Custom("x-api-key"), - default_headers: vec![("anthropic-version".to_string(), "2023-06-01".to_string())], - passthrough_headers: vec![ - "anthropic-version".to_string(), - "anthropic-beta".to_string(), - ], - timeout: openshell_router::config::DEFAULT_ROUTE_TIMEOUT, - model_in_path: false, - request_path_override: None, - }; - - let ctx = InferenceContext::new(patterns, router, vec![], vec![system_route]); - - let body = serde_json::to_vec(&serde_json::json!({ - "model": "claude-sonnet-4-20250514", - "max_tokens": 1, - "messages": [{"role": "user", "content": "analyze policy"}] - })) - .unwrap(); - - let response = ctx - .system_inference( - "anthropic_messages", - "POST", - "/v1/messages", - vec![("content-type".to_string(), "application/json".to_string())], - bytes::Bytes::from(body), - ) - .await - .expect("anthropic system_inference should succeed"); - - assert_eq!(response.status, 200); - let resp_body: serde_json::Value = serde_json::from_slice(&response.body).unwrap(); - assert_eq!(resp_body["type"], "message"); -} diff --git a/crates/openshell-supervisor-process/src/process.rs b/crates/openshell-supervisor-process/src/process.rs index 52557493c..0ab1dd318 100644 --- a/crates/openshell-supervisor-process/src/process.rs +++ b/crates/openshell-supervisor-process/src/process.rs @@ -225,11 +225,14 @@ fn apply_canonical_process_environment( user_environment: &HashMap, ) { let (session_user, session_home) = session_user_and_home(policy, workspace.home()); + // Resolve a shell present in the sandbox image (minimal images such as + // Alpine ship only `/bin/sh`, not bash). Runs in the supervisor. + let shell = openshell_core::shell::detect_login_shell(); for (key, value) in [ ("HOME", session_home.as_str()), ("USER", session_user.as_str()), - ("SHELL", "/bin/bash"), + ("SHELL", shell.as_str()), ( "TERM", if interactive { @@ -900,15 +903,19 @@ impl ProcessHandle { } } + // Name the program in the error: a bare "No such file or directory" + // here is otherwise indistinguishable from a missing working directory + // or interpreter, and is a common failure on images that lack the + // requested shell/binary (e.g. bash on Alpine). #[cfg(target_os = "linux")] let mut child = spawn_command_with_supervisor_identity_namespace(cmd) .into_diagnostic() - .wrap_err("failed to spawn sandbox entrypoint process")?; + .wrap_err_with(|| format!("failed to spawn sandbox entrypoint process '{program}'"))?; #[cfg(not(target_os = "linux"))] let mut child = cmd .spawn() .into_diagnostic() - .wrap_err("failed to spawn sandbox entrypoint process")?; + .wrap_err_with(|| format!("failed to spawn sandbox entrypoint process '{program}'"))?; let pid = child.id().unwrap_or(0); managed_children::register(pid); @@ -2475,7 +2482,10 @@ mod tests { Some(¤t_user.dir.to_string_lossy().as_ref()) ); assert_eq!(variables.get("USER"), Some(¤t_user.name.as_str())); - assert_eq!(variables.get("SHELL"), Some(&"/bin/bash")); + // SHELL is the shell detected in the current root filesystem, not a + // hardcoded path (bash-less images resolve to /bin/sh). + let expected_shell = openshell_core::shell::detect_login_shell(); + assert_eq!(variables.get("SHELL"), Some(&expected_shell.as_str())); assert_eq!(variables.get("TERM"), Some(&"xterm-256color")); } diff --git a/crates/openshell-supervisor-process/src/run.rs b/crates/openshell-supervisor-process/src/run.rs index b2820d958..8c47e789b 100644 --- a/crates/openshell-supervisor-process/src/run.rs +++ b/crates/openshell-supervisor-process/src/run.rs @@ -453,24 +453,34 @@ pub async fn run_process( .build() ); - if let Some(tx) = sidecar_exit_tx.as_ref() { - report_sidecar_main_process_exit(tx, &main_instance_id, rendered_code).await?; - } else if let (Some(endpoint), Some(id)) = (openshell_endpoint, sandbox_id) { - report_main_process_exit_until_ack(endpoint, id, &main_instance_id, rendered_code).await; - info!(instance_id = %main_instance_id, "main-process exit acknowledged"); + if outcome.should_report_main_process_exit() { + if let Some(tx) = sidecar_exit_tx.as_ref() { + report_sidecar_main_process_exit(tx, &main_instance_id, rendered_code).await?; + } else if let (Some(endpoint), Some(id)) = (openshell_endpoint, sandbox_id) { + report_main_process_exit_until_ack(endpoint, id, &main_instance_id, rendered_code) + .await; + info!(instance_id = %main_instance_id, "main-process exit acknowledged"); + } + } else { + info!( + instance_id = %main_instance_id, + "skipping main-process exit report during supervisor shutdown" + ); } main_session.mark_terminal_reported(); - if drain_terminal && terminal_delivery_pending { + if outcome.should_report_main_process_exit() && drain_terminal && terminal_delivery_pending { // The peer's SSH channel-close confirms that the terminal frames sent // above traversed russh and the relay. Detached commands have no active // attachment and never enter this wait. main_session.wait_for_terminal_attachments().await; } - if let Some(tx) = sidecar_exit_tx.as_ref() { - finalize_sidecar_main_process_exit(tx, &main_instance_id).await?; - } else if let (Some(endpoint), Some(id)) = (openshell_endpoint, sandbox_id) { - finalize_main_process_exit_until_ack(endpoint, id, &main_instance_id).await; - info!(instance_id = %main_instance_id, "main-process terminal delivery finalized"); + if outcome.should_report_main_process_exit() { + if let Some(tx) = sidecar_exit_tx.as_ref() { + finalize_sidecar_main_process_exit(tx, &main_instance_id).await?; + } else if let (Some(endpoint), Some(id)) = (openshell_endpoint, sandbox_id) { + finalize_main_process_exit_until_ack(endpoint, id, &main_instance_id).await; + info!(instance_id = %main_instance_id, "main-process terminal delivery finalized"); + } } supervisor_terminating.store(true, Ordering::Release); @@ -572,6 +582,16 @@ enum ProcessWaitOutcome { }, } +impl ProcessWaitOutcome { + /// A gateway acknowledgement is required for ordinary canonical-process + /// completion, but cannot be awaited after the supervisor itself has been + /// asked to terminate. At that point the gateway may already be shutting + /// down and no longer able to acknowledge the report. + fn should_report_main_process_exit(&self) -> bool { + !matches!(self, Self::ShutdownSignal { .. }) + } +} + async fn wait_for_process_exit_or_shutdown( handle: &mut ProcessHandle, timeout_secs: u64, @@ -789,4 +809,22 @@ mod tests { assert_eq!(ssh_proxy_url_for_policy(&policy, None), None); } + + #[cfg(unix)] + #[test] + fn supervisor_shutdown_exit_skips_gateway_acknowledgement() { + use std::os::unix::process::ExitStatusExt; + + let status = ProcessStatus::from(std::process::ExitStatus::from_raw(libc::SIGTERM)); + + assert!(ProcessWaitOutcome::Exited(status).should_report_main_process_exit()); + assert!(ProcessWaitOutcome::TimedOut.should_report_main_process_exit()); + assert!( + !ProcessWaitOutcome::ShutdownSignal { + signal: "SIGTERM", + status, + } + .should_report_main_process_exit() + ); + } } diff --git a/crates/openshell-supervisor-process/src/sandbox/linux/landlock.rs b/crates/openshell-supervisor-process/src/sandbox/linux/landlock.rs index bf42faede..0a5bbfdc7 100644 --- a/crates/openshell-supervisor-process/src/sandbox/linux/landlock.rs +++ b/crates/openshell-supervisor-process/src/sandbox/linux/landlock.rs @@ -144,6 +144,14 @@ fn prepare_with_path_open_mode( } if read_only.is_empty() && read_write.is_empty() { + if matches!( + policy.landlock.compatibility, + LandlockCompatibility::HardRequirement + ) { + miette::bail!( + "landlock.compatibility is hard_requirement but no filesystem paths are configured" + ); + } return Ok(None); } @@ -522,6 +530,68 @@ mod tests { panic!("hard_requirement should accept mixed directory and device paths: {err}"); } } + #[test] + fn prepare_hard_requirement_no_paths_aborts() { + // Zero configured paths under hard_requirement must fail startup rather + // than silently running without filesystem restrictions. + let policy = hard_requirement_policy(vec![], vec![]); + let Err(err) = prepare(&policy, None) else { + panic!("should abort with no paths"); + }; + let msg = err.to_string(); + assert!( + msg.contains("hard_requirement") && msg.contains("no filesystem paths"), + "error should explain the empty hard_requirement policy: {msg}" + ); + } + + #[test] + fn prepare_best_effort_no_paths_is_noop() { + let policy = SandboxPolicy { + version: 1, + filesystem: FilesystemPolicy { + read_only: vec![], + read_write: vec![], + include_workdir: false, + }, + network: NetworkPolicy::default(), + landlock: LandlockPolicy { + compatibility: LandlockCompatibility::BestEffort, + }, + process: ProcessPolicy::default(), + }; + let prepared = prepare(&policy, None).expect("best_effort no-op should succeed"); + assert!(prepared.is_none(), "no paths should produce no ruleset"); + } + + #[test] + fn prepare_include_workdir_counts_as_configured_path() { + // With no explicit paths but include_workdir set, the workdir must be + // treated as a configured path — so the zero-path abort must NOT fire. + let policy = SandboxPolicy { + version: 1, + filesystem: FilesystemPolicy { + read_only: vec![], + read_write: vec![], + include_workdir: true, + }, + network: NetworkPolicy::default(), + landlock: LandlockPolicy { + compatibility: LandlockCompatibility::HardRequirement, + }, + process: ProcessPolicy::default(), + }; + // Any error (e.g. Landlock unavailable on this host) is acceptable, but + // it must not be the "no filesystem paths" abort. + if let Err(err) = prepare(&policy, Some("/tmp")) { + let msg = err.to_string(); + assert!( + !msg.contains("no filesystem paths"), + "workdir should count as a configured path: {msg}" + ); + } + } + fn tailored_access(path: &Path, requested_access: BitFlags) -> BitFlags { let path_fd = PathFd::new(path).unwrap(); access_for_path_fd(&path_fd, requested_access, ABI::V2).unwrap() diff --git a/crates/openshell-supervisor-process/src/sandbox/linux/mod.rs b/crates/openshell-supervisor-process/src/sandbox/linux/mod.rs index 107a50e37..8afb3caef 100644 --- a/crates/openshell-supervisor-process/src/sandbox/linux/mod.rs +++ b/crates/openshell-supervisor-process/src/sandbox/linux/mod.rs @@ -99,14 +99,60 @@ pub fn log_sandbox_readiness(policy: &SandboxPolicy, workdir: Option<&str>) { let total_paths = read_only.len() + read_write.len(); if total_paths == 0 { - openshell_ocsf::ocsf_emit!( - openshell_ocsf::ConfigStateChangeBuilder::new(openshell_ocsf::ctx::ctx()) - .severity(openshell_ocsf::SeverityId::Informational) - .status(openshell_ocsf::StatusId::Success) - .state(openshell_ocsf::StateId::Other, "skipped") - .message("Landlock filesystem sandbox skipped: no paths configured".to_string()) - .build() - ); + if matches!( + policy.landlock.compatibility, + openshell_core::policy::LandlockCompatibility::HardRequirement + ) { + // hard_requirement with no paths is fatal (see `landlock::prepare`). + // Emit a failure state so operators don't see a misleading "skipped" + // success event immediately before startup aborts. + openshell_ocsf::ocsf_emit!( + openshell_ocsf::ConfigStateChangeBuilder::new(openshell_ocsf::ctx::ctx()) + .severity(openshell_ocsf::SeverityId::High) + .status(openshell_ocsf::StatusId::Failure) + .state(openshell_ocsf::StateId::Other, "invalid") + .message( + "Landlock hard_requirement but no filesystem paths configured; \ + sandbox startup will abort" + .to_string(), + ) + .build() + ); + // Dual-emit a security finding for the unsafe policy (per OCSF + // guidance: pair the domain event with a DetectionFinding). + openshell_ocsf::ocsf_emit!( + openshell_ocsf::DetectionFindingBuilder::new(openshell_ocsf::ctx::ctx()) + .activity(openshell_ocsf::ActivityId::Open) + .severity(openshell_ocsf::SeverityId::High) + .confidence(openshell_ocsf::ConfidenceId::High) + .is_alert(true) + .finding_info( + openshell_ocsf::FindingInfo::new( + "landlock-hard-requirement-no-paths", + "Landlock Hard Requirement Without Paths", + ) + .with_desc( + "landlock.compatibility is hard_requirement but no filesystem \ + paths are configured; sandbox startup will abort.", + ), + ) + .message( + "Landlock hard_requirement with no filesystem paths configured".to_string(), + ) + .build() + ); + } else { + openshell_ocsf::ocsf_emit!( + openshell_ocsf::ConfigStateChangeBuilder::new(openshell_ocsf::ctx::ctx()) + .severity(openshell_ocsf::SeverityId::Informational) + .status(openshell_ocsf::StatusId::Success) + .state(openshell_ocsf::StateId::Other, "skipped") + .message( + "Landlock filesystem sandbox skipped: no paths configured".to_string(), + ) + .build() + ); + } return; } diff --git a/crates/openshell-supervisor-process/src/skills/policy_advisor.md b/crates/openshell-supervisor-process/src/skills/policy_advisor.md index 4b7074f6d..8d792aaed 100644 --- a/crates/openshell-supervisor-process/src/skills/policy_advisor.md +++ b/crates/openshell-supervisor-process/src/skills/policy_advisor.md @@ -50,10 +50,12 @@ operations. Each `addRule` carries a complete narrow `NetworkPolicyRule`. credential is in scope auto-approve** (see Auto-approval below). Any credentialed reach or capability change goes to human review — that is the design. L7 is still the agent-speed path because the prover can - precisely describe the change (which method was added on which path); - L4 to a credentialed host loses that precision. Use L4 only when the - binary's wire protocol is opaque to L7 inspection (`ssh`, `nc`, - `git-remote-http`) or the host has no documented REST surface. + precisely describe the change (which method was added on which path). + When a client can use the explicit proxy but its wire protocol is opaque + to L7 inspection (`ssh`, `nc`, `git-remote-http`), omit `protocol` and + retain default TLS handling. Never propose `protocol: tcp` or `tls: skip`: + those choices bypass application-authority inspection and require an + administrator to add the rule explicitly. 5. Draft the narrowest rule: exact host, exact port, exact binary when known, exact method, and the smallest safe path. 6. Submit the proposal, save `accepted_chunk_ids` from the response, and @@ -207,6 +209,8 @@ The new submission wins by structural overlap. `fe80::/10`) or known metadata hostnames such as `metadata.google.internal`. Cloud-metadata endpoints there can hand out the host's credentials. +- Do not propose `protocol: tcp` or `tls: skip`. If the task requires native + TCP or a raw TLS tunnel, ask an administrator to add that access explicitly. - Do not include query strings, tokens, credentials, or secret values in paths. - Explain uncertainty in `intent_summary` instead of widening the rule. diff --git a/crates/openshell-supervisor-process/src/ssh.rs b/crates/openshell-supervisor-process/src/ssh.rs index 357969fb5..fbd6d9275 100644 --- a/crates/openshell-supervisor-process/src/ssh.rs +++ b/crates/openshell-supervisor-process/src/ssh.rs @@ -1208,7 +1208,7 @@ fn apply_child_env( .env(openshell_core::sandbox_env::SANDBOX, "1") .env("HOME", session_home) .env("USER", session_user) - .env("SHELL", "/bin/bash") + .env("SHELL", openshell_core::shell::detect_login_shell()) .env("PATH", &path) .env("TERM", term); @@ -1242,6 +1242,33 @@ const fn login_shell_flag(no_login_shell: bool) -> &'static str { if no_login_shell { "-c" } else { "-lc" } } +/// Build the shell command for an SSH session using a shell that exists in the +/// sandbox image (minimal images such as Alpine ship only `/bin/sh`, not bash). +/// +/// `no_command_arg` is appended only when no explicit command is given: `-i` +/// for an interactive PTY session, or `None` for the non-PTY stdin path (a +/// bare shell already reads piped stdin line-by-line). With an explicit +/// command the login-shell flag is used per `no_login_shell`. +fn build_ssh_shell_command( + shell: &str, + command: Option, + no_login_shell: bool, + no_command_arg: Option<&str>, +) -> Command { + let mut cmd = Command::new(shell); + match command { + None => { + if let Some(arg) = no_command_arg { + cmd.arg(arg); + } + } + Some(command) => { + cmd.arg(login_shell_flag(no_login_shell)).arg(command); + } + } + cmd +} + #[allow(clippy::too_many_arguments)] fn spawn_pty_shell( policy: &SandboxPolicy, @@ -1276,18 +1303,11 @@ fn spawn_pty_shell( let mut reader = master.try_clone()?; let mut writer = master.try_clone()?; - let mut cmd = command.map_or_else( - || { - let mut c = Command::new("/bin/bash"); - c.arg("-i"); - c - }, - |command| { - let mut c = Command::new("/bin/bash"); - c.arg(login_shell_flag(no_login_shell)).arg(command); - c - }, - ); + // Resolve a shell present in the sandbox image; interactive PTY sessions + // pass `-i` when no command is given. Runs in the supervisor, so it + // inspects the sandbox filesystem. + let shell = openshell_core::shell::detect_login_shell(); + let mut cmd = build_ssh_shell_command(&shell, command, no_login_shell, Some("-i")); let term = if pty.term.is_empty() { "xterm-256color" @@ -1435,25 +1455,15 @@ fn spawn_pipe_exec( resolved_identity: ResolvedProcessIdentity, enforcement_mode: ProcessEnforcementMode, ) -> anyhow::Result>> { - let mut cmd = command.map_or_else( - || { - // No command — read from stdin. Do *not* pass `-i`; interactive - // mode reads .bashrc, writes prompts to stderr, and can introduce - // just enough latency for VS Code Remote-SSH's platform detection - // to time out and fall back to "windows". Plain `bash` with piped - // stdin already reads commands line-by-line (script mode), which is - // exactly what VS Code's local server expects. - Command::new("/bin/bash") - }, - |command| { - let mut c = Command::new("/bin/bash"); - // Login shell (-l) sources .profile/.bashrc so tool env vars - // (VIRTUAL_ENV, etc.) are available. Callers that need a predictable - // environment opt out via OPENSHELL_NO_LOGIN_SHELL → plain -c. - c.arg(login_shell_flag(no_login_shell)).arg(command); - c - }, - ); + // Resolve a shell present in the sandbox image; minimal images (e.g. Alpine) + // don't ship bash, only `/bin/sh`. Runs in the supervisor, so it inspects + // the sandbox filesystem. No command → read from stdin with no `-i`: + // interactive mode reads .bashrc, writes prompts to stderr, and can add + // just enough latency for VS Code Remote-SSH's platform detection to time + // out and fall back to "windows". A plain shell with piped stdin already + // reads commands line-by-line (script mode), which is what VS Code expects. + let shell = openshell_core::shell::detect_login_shell(); + let mut cmd = build_ssh_shell_command(&shell, command, no_login_shell, None); let (session_user, session_home) = session_user_and_home(policy, workspace.home()); apply_child_env( @@ -1797,8 +1807,40 @@ fn is_loopback_host(host: &str) -> bool { )] mod tests { use super::*; + use std::ffi::OsStr; use std::process::Stdio; + /// Regression test: SSH sessions run the shell they are given, never a + /// hardcoded bash, so sh-only images (e.g. Alpine) work. Covers both the + /// interactive PTY path (`-i` when no command) and the non-PTY path. + #[test] + fn build_ssh_shell_command_uses_given_shell() { + // PTY, no command → given shell + interactive flag. + let cmd = build_ssh_shell_command("/bin/sh", None, false, Some("-i")); + assert_eq!(cmd.get_program(), OsStr::new("/bin/sh")); + assert_eq!(cmd.get_args().collect::>(), vec![OsStr::new("-i")]); + + // Non-PTY, no command → bare shell, no args (reads piped stdin). + let cmd = build_ssh_shell_command("/bin/sh", None, false, None); + assert_eq!(cmd.get_program(), OsStr::new("/bin/sh")); + assert_eq!(cmd.get_args().count(), 0); + + // Explicit command → login-shell flag + command, still on the given shell. + let cmd = build_ssh_shell_command("/bin/sh", Some("echo hi".into()), false, Some("-i")); + assert_eq!(cmd.get_program(), OsStr::new("/bin/sh")); + assert_eq!( + cmd.get_args().collect::>(), + vec![OsStr::new("-lc"), OsStr::new("echo hi")] + ); + + // OPENSHELL_NO_LOGIN_SHELL → plain -c. + let cmd = build_ssh_shell_command("/bin/sh", Some("echo hi".into()), true, None); + assert_eq!( + cmd.get_args().collect::>(), + vec![OsStr::new("-c"), OsStr::new("echo hi")] + ); + } + /// Regression test: the direct-tcpip connect path sets `TCP_NODELAY`. #[tokio::test] async fn connect_in_netns_sets_tcp_nodelay() { diff --git a/crates/openshell-tui/src/app.rs b/crates/openshell-tui/src/app.rs index a5e14bcba..cf7464fcd 100644 --- a/crates/openshell-tui/src/app.rs +++ b/crates/openshell-tui/src/app.rs @@ -11,6 +11,9 @@ use openshell_core::auth::EdgeAuthInterceptor; use openshell_core::proto::open_shell_client::OpenShellClient; use openshell_core::proto::setting_value; use openshell_core::settings::{self, SettingValueKind}; +use openshell_providers::{ + DiscoveredProvider, ProviderTypeProfile, RealDiscoveryContext, discover_from_profile, +}; use tonic::service::interceptor::InterceptedService; use tonic::transport::Channel; @@ -350,9 +353,9 @@ pub enum ProviderKeyField { Name, /// Focused credential row for known types (index via `cred_cursor`). Credential, - /// Custom env var name (generic / no-known-env-vars types only). + /// Custom env var name (legacy no-known-env-vars types only). EnvVarName, - /// Custom env var value (generic / no-known-env-vars types only). + /// Custom env var value (legacy no-known-env-vars types only). GenericValue, /// Browsing/deleting existing config entries (Up/Down/Ctrl+D). ConfigList, @@ -385,14 +388,16 @@ pub struct CreateProviderForm { pub config_key_input: String, /// Config value being entered. pub config_value_input: String, - /// For generic / types with no known env vars: custom env var name. + /// For legacy types with no known env vars: custom env var name. pub generic_env_name: String, - /// For generic / types with no known env vars: custom value. + /// For legacy types with no known env vars: custom value. pub generic_value: String, /// Which field is focused in the key entry form. pub key_field: ProviderKeyField, - /// True when the provider type has no known env vars (generic, outlook). + /// True when the selected profile has no accepted stored credential keys. pub is_generic: bool, + /// True when the selected profile permits creation without stored credentials. + pub allows_empty_credentials: bool, /// Status message (errors, validation). pub status: Option, /// Warning shown at top of `EnterKey` modal (e.g. autodetect failure). @@ -405,6 +410,12 @@ pub struct CreateProviderForm { pub discovered_credentials: Option>, } +fn apply_discovered_provider(form: &mut CreateProviderForm, discovered: DiscoveredProvider) { + form.discovered_credentials = Some(discovered.credentials); + form.config = discovered.config.into_iter().collect(); + form.config_cursor = 0; +} + // --------------------------------------------------------------------------- // Provider detail view (Get) // --------------------------------------------------------------------------- @@ -590,7 +601,7 @@ pub struct App { pub pending_workspace_refresh: bool, // Provider list - pub providers_v2_enabled: bool, + pub provider_profiles: Vec, pub provider_entries: Vec, pub provider_names: Vec, pub provider_types: Vec, @@ -970,7 +981,7 @@ impl App { all_workspaces: false, workspace_names: Vec::new(), pending_workspace_refresh: false, - providers_v2_enabled: false, + provider_profiles: Vec::new(), provider_entries: Vec::new(), provider_names: Vec::new(), provider_types: Vec::new(), @@ -1396,6 +1407,9 @@ impl App { self.input_mode = InputMode::Command; self.command_input.clear(); } + KeyCode::Char('w') => { + self.cycle_workspace(); + } KeyCode::Char('j') | KeyCode::Down => { if self.provider_count > 0 && self.provider_selected < self.provider_count - 1 { self.provider_selected += 1; @@ -1410,7 +1424,7 @@ impl App { self.overflow_focus_up(); } } - KeyCode::Char('c') if !self.providers_v2_enabled => { + KeyCode::Char('c') => { if self.all_workspaces { self.status_text = "Switch to a specific workspace to create providers.".to_string(); @@ -1423,10 +1437,10 @@ impl App { self.pending_provider_get = true; } // Open update form for the selected provider. - KeyCode::Char('u') if self.provider_count > 0 && !self.providers_v2_enabled => { + KeyCode::Char('u') if self.provider_count > 0 => { self.open_update_provider_form(); } - KeyCode::Char('d') if self.provider_count > 0 && !self.providers_v2_enabled => { + KeyCode::Char('d') if self.provider_count > 0 => { self.confirm_provider_delete = true; } KeyCode::Char('h' | 'l') | KeyCode::Left | KeyCode::Right => { @@ -2383,15 +2397,45 @@ impl App { // ------------------------------------------------------------------ fn open_create_provider_form(&mut self) { - let known = openshell_providers::ProviderRegistry::new().known_types(); - let types: Vec = known.into_iter().map(String::from).collect(); + let types = self.available_provider_profile_types(); self.create_provider_form = Some(CreateProviderForm { types, + status: self.provider_profiles.is_empty().then(|| { + "Provider profiles unavailable. Wait for refresh, then retry.".to_string() + }), ..CreateProviderForm::default() }); } + fn available_provider_profile_types(&self) -> Vec { + let mut types = self + .provider_profiles + .iter() + .map(|profile| profile.id.clone()) + .collect::>(); + types.sort(); + types.dedup(); + types + } + + pub(crate) fn sync_create_provider_types(&mut self) { + let types = self.available_provider_profile_types(); + let Some(form) = self.create_provider_form.as_mut() else { + return; + }; + if form.phase != CreateProviderPhase::SelectType { + return; + } + + form.types = types; + form.type_cursor = form.type_cursor.min(form.types.len().saturating_sub(1)); + form.status = form + .types + .is_empty() + .then(|| "Provider profiles unavailable. Wait for refresh, then retry.".to_string()); + } + fn handle_create_provider_key(&mut self, key: KeyEvent) { let Some(form) = self.create_provider_form.as_mut() else { return; @@ -2409,27 +2453,53 @@ impl App { form.type_cursor = form.type_cursor.saturating_sub(1); } KeyCode::Enter => { - let selected = form.types[form.type_cursor].clone(); - let registry = openshell_providers::ProviderRegistry::new(); - let env_vars = registry.credential_env_vars(&selected); - form.is_generic = env_vars.is_empty(); - - // Populate credential rows from all known env vars. - form.credentials = env_vars + let Some(selected) = form.types.get(form.type_cursor).cloned() else { + form.status = Some( + "Provider profiles unavailable. Wait for refresh, then retry." + .to_string(), + ); + return; + }; + let Some(profile) = self + .provider_profiles .iter() - .map(|s| (s.to_string(), String::new())) + .find(|profile| profile.id == selected) + .cloned() + else { + form.status = Some(format!("Provider profile '{selected}' is unavailable")); + return; + }; + let profile = ProviderTypeProfile::from_proto(&profile); + let mut credential_keys = Vec::new(); + for key in profile + .credentials + .iter() + .flat_map(|credential| credential.accepted_stored_keys()) + { + if !credential_keys.iter().any(|existing| existing == key) { + credential_keys.push(key.to_string()); + } + } + form.is_generic = credential_keys.is_empty(); + form.allows_empty_credentials = profile.allows_empty_provider_credentials(); + + // Populate credential rows from all accepted storage keys, + // including logical names for broker-only credentials. + form.credentials = credential_keys + .into_iter() + .map(|key| (key, String::new())) .collect(); form.cred_cursor = 0; // Auto-generate a unique name. form.name = unique_provider_name(&selected, &self.provider_names); - if form.is_generic { - // No known env vars — skip straight to manual entry. - form.phase = CreateProviderPhase::EnterKey; - form.key_field = ProviderKeyField::Name; - form.status = None; - form.warning = None; + if form.credentials.is_empty() { + // Credential-less profiles can be created directly. + form.discovered_credentials = Some(HashMap::new()); + form.phase = CreateProviderPhase::Creating; + form.anim_start = Some(Instant::now()); + self.pending_provider_create = true; } else { form.phase = CreateProviderPhase::ChooseMethod; form.method_cursor = 0; @@ -2450,10 +2520,18 @@ impl App { KeyCode::Enter => { let ptype = form.types[form.type_cursor].clone(); if form.method_cursor == 0 { - // Autodetect — synchronous since we only check env vars now. - let registry = openshell_providers::ProviderRegistry::new(); - if let Ok(Some(discovered)) = registry.discover_existing(&ptype) { - form.discovered_credentials = Some(discovered.credentials); + let discovered = self + .provider_profiles + .iter() + .find(|profile| profile.id == ptype) + .map(ProviderTypeProfile::from_proto) + .and_then(|profile| { + discover_from_profile(&profile, &RealDiscoveryContext) + .ok() + .flatten() + }); + if let Some(discovered) = discovered { + apply_discovered_provider(form, discovered); if form.name.is_empty() { form.name = unique_provider_name(&ptype, &self.provider_names); } @@ -2465,7 +2543,12 @@ impl App { form.phase = CreateProviderPhase::EnterKey; form.key_field = ProviderKeyField::Name; form.warning = Some( - "No credentials found in environment. Enter manually.".to_string(), + if form.allows_empty_credentials { + "No credentials found in environment. Enter credentials or submit without them." + } else { + "No credentials found in environment. Enter manually." + } + .to_string(), ); form.status = None; } @@ -2789,7 +2872,7 @@ impl App { creds.insert(name.clone(), value.clone()); } } - if creds.is_empty() { + if creds.is_empty() && !form.allows_empty_credentials { form.status = Some("At least one credential is required.".to_string()); return; @@ -2893,13 +2976,19 @@ impl App { }) .unwrap_or_default(); - // If we don't know the credential key, derive from registry. + // If we don't know the credential key, derive it from the profile. let key = if cred_key.is_empty() { - let registry = openshell_providers::ProviderRegistry::new(); - registry - .credential_env_vars(&ptype) - .first() - .map_or(String::new(), ToString::to_string) + self.provider_entries + .get(self.provider_selected) + .and_then(|entry| entry.profile.as_ref()) + .map(ProviderTypeProfile::from_proto) + .and_then(|profile| { + profile + .credential_env_vars() + .first() + .map(ToString::to_string) + }) + .unwrap_or_default() } else { cred_key }; @@ -3336,7 +3425,7 @@ impl App { ); let raw_profile_yaml = profile.and_then(|profile| { - let dto = openshell_providers::ProviderTypeProfile::from_proto(profile); + let dto = ProviderTypeProfile::from_proto(profile); openshell_providers::profile_to_yaml(&dto).ok() }); @@ -3435,7 +3524,7 @@ impl App { self.global_policy_active = false; self.global_policy_version = 0; // Reset provider state too. - self.providers_v2_enabled = false; + self.provider_profiles.clear(); self.provider_entries.clear(); self.provider_names.clear(); self.provider_types.clear(); @@ -3501,22 +3590,115 @@ mod tests { ) } + fn provider_profile( + id: &str, + credentials: Vec, + ) -> openshell_core::proto::ProviderProfile { + openshell_core::proto::ProviderProfile { + id: id.to_string(), + credentials, + ..Default::default() + } + } + + fn credential( + name: &str, + env_vars: &[&str], + required: bool, + runtime_resolvable: bool, + ) -> openshell_core::proto::ProviderProfileCredential { + openshell_core::proto::ProviderProfileCredential { + name: name.to_string(), + env_vars: env_vars.iter().map(ToString::to_string).collect(), + required, + token_grant: runtime_resolvable.then(Default::default), + ..Default::default() + } + } + + fn key(code: KeyCode) -> KeyEvent { + KeyEvent::new(code, KeyModifiers::NONE) + } + #[tokio::test] - async fn global_settings_do_not_override_provider_api_capability() { + async fn create_provider_enter_with_empty_profile_catalog_is_recoverable() { let mut app = test_app(); - app.providers_v2_enabled = true; - let mut values = HashMap::new(); - values.insert( - settings::PROVIDERS_V2_ENABLED_KEY.to_string(), - openshell_core::proto::SettingValue { - value: Some(setting_value::Value::BoolValue(false)), - }, + app.open_create_provider_form(); + + app.handle_create_provider_key(key(KeyCode::Enter)); + + let form = app + .create_provider_form + .as_ref() + .expect("form remains open"); + assert_eq!(form.phase, CreateProviderPhase::SelectType); + assert!( + form.status + .as_deref() + .is_some_and(|status| status.contains("profiles unavailable")) ); + assert!(!app.pending_provider_create); - app.apply_global_settings(values, 7); + app.provider_profiles = vec![provider_profile("recovered", Vec::new())]; + app.sync_create_provider_types(); + let form = app + .create_provider_form + .as_ref() + .expect("form remains open"); + assert_eq!(form.types, vec!["recovered"]); + assert!(form.status.is_none()); + } - assert!(app.providers_v2_enabled); - assert_eq!(app.global_settings_revision, 7); + #[tokio::test] + async fn create_provider_accepts_empty_credentials_when_profile_allows_them() { + for profile in [ + provider_profile("policy-only", Vec::new()), + provider_profile( + "optional-static", + vec![credential("api_key", &["API_KEY"], false, false)], + ), + provider_profile( + "runtime-token", + vec![credential("access_token", &["ACCESS_TOKEN"], true, true)], + ), + ] { + let mut app = test_app(); + app.provider_profiles = vec![profile]; + app.open_create_provider_form(); + app.handle_create_provider_key(key(KeyCode::Enter)); + + let form = app.create_provider_form.as_mut().expect("form"); + if form.phase != CreateProviderPhase::Creating { + form.phase = CreateProviderPhase::EnterKey; + form.key_field = ProviderKeyField::Submit; + app.handle_create_provider_key(key(KeyCode::Enter)); + } + + let form = app.create_provider_form.as_ref().expect("form"); + assert_eq!(form.phase, CreateProviderPhase::Creating); + assert_eq!(form.discovered_credentials, Some(HashMap::new())); + assert!(app.pending_provider_create); + } + } + + #[tokio::test] + async fn create_provider_uses_logical_key_for_broker_only_credential() { + let mut app = test_app(); + app.provider_profiles = vec![provider_profile( + "token-exchange", + vec![credential("subject_token", &[], true, false)], + )]; + app.open_create_provider_form(); + + app.handle_create_provider_key(key(KeyCode::Enter)); + + let form = app.create_provider_form.as_ref().expect("form"); + assert_eq!(form.phase, CreateProviderPhase::ChooseMethod); + assert_eq!( + form.credentials, + vec![("subject_token".to_string(), String::new())] + ); + assert!(!form.allows_empty_credentials); } #[tokio::test] @@ -3646,6 +3828,32 @@ mod tests { assert_eq!(gateway.source_label(), "unknown"); } + #[tokio::test] + async fn providers_workspace_shortcut_cycles_scope_and_resets_selections() { + let mut app = test_app(); + app.screen = Screen::Dashboard; + app.focus = Focus::Providers; + app.workspace_names = vec!["default".to_string(), "team-b".to_string()]; + app.provider_selected = 3; + app.sandbox_selected = 4; + + app.handle_key(key(KeyCode::Char('w'))); + + assert_eq!(app.current_workspace, "team-b"); + assert!(!app.all_workspaces); + assert_eq!(app.provider_selected, 0); + assert_eq!(app.sandbox_selected, 0); + assert!(app.pending_workspace_refresh); + + app.handle_key(key(KeyCode::Char('w'))); + assert!(app.all_workspaces); + assert_eq!(app.workspace_display(), "all"); + + app.handle_key(key(KeyCode::Char('w'))); + assert!(!app.all_workspaces); + assert_eq!(app.current_workspace, "default"); + } + // -- selected_sandbox_workspace ---------------------------------------- #[test] @@ -3931,4 +4139,34 @@ mod tests { ); assert_eq!(config.get("FOO"), Some(&"bar".to_string())); } + + #[test] + fn autodetected_provider_configuration_is_preserved_in_create_form() { + let mut form = CreateProviderForm::default(); + apply_discovered_provider( + &mut form, + DiscoveredProvider { + credentials: HashMap::from([("VERTEX_AI_TOKEN".to_string(), "token".to_string())]), + config: HashMap::from([ + ("VERTEX_AI_PROJECT_ID".to_string(), "project-a".to_string()), + ("VERTEX_AI_REGION".to_string(), "us-central1".to_string()), + ]), + }, + ); + + assert_eq!( + form.discovered_credentials + .as_ref() + .and_then(|credentials| credentials.get("VERTEX_AI_TOKEN")), + Some(&"token".to_string()) + ); + assert_eq!( + form.config.get("VERTEX_AI_PROJECT_ID"), + Some(&"project-a".to_string()) + ); + assert_eq!( + form.config.get("VERTEX_AI_REGION"), + Some(&"us-central1".to_string()) + ); + } } diff --git a/crates/openshell-tui/src/lib.rs b/crates/openshell-tui/src/lib.rs index 625bc8a41..2ab19d89d 100644 --- a/crates/openshell-tui/src/lib.rs +++ b/crates/openshell-tui/src/lib.rs @@ -8,6 +8,7 @@ pub mod theme; mod ui; use std::collections::HashMap; +use std::future::Future; use std::io; use std::path::PathBuf; use std::time::Duration; @@ -35,9 +36,25 @@ use event::{Event, EventHandler}; /// Duration to show the splash screen before auto-dismissing. const SPLASH_DURATION: Duration = Duration::from_secs(3); const PROVIDER_PROFILE_SCOPE_WORKSPACE: &str = "workspace"; +const PROVIDER_PROFILE_PAGE_SIZE: u32 = 100; type ProviderProfileCache = HashMap<(String, String), openshell_core::proto::ProviderProfile>; +fn named_workspace_scope(workspace: impl Into) -> openshell_core::proto::WorkspaceSelector { + openshell_core::proto::workspace_selector(workspace) +} + +fn list_workspace_scope( + workspace: impl Into, + all_workspaces: bool, +) -> openshell_core::proto::WorkspaceSelector { + if all_workspaces { + openshell_core::proto::all_workspaces_selector() + } else { + openshell_core::proto::workspace_selector(workspace) + } +} + // Re-export for use by the CLI crate. pub use theme::ThemeMode; @@ -78,7 +95,6 @@ pub async fn run( let mut events = EventHandler::new(Duration::from_secs(2)); - fetch_providers_v2_setting(&mut app).await; refresh_gateway_list(&mut app); refresh_data(&mut app).await; @@ -500,10 +516,6 @@ async fn handle_gateway_switch(app: &mut App) { app.gateway_name = name; app.endpoint = endpoint; app.reset_sandbox_state(); - // Re-fetch the providers_v2 capability for the new gateway - // before refreshing data, so provider CRUD controls reflect - // the correct mode. - fetch_providers_v2_setting(app).await; refresh_data(app).await; } Err(e) => { @@ -651,7 +663,7 @@ fn spawn_log_stream(app: &mut App, tx: mpsc::UnboundedSender) { since_ms: 0, sources: vec![], min_level: String::new(), - workspace, + workspace_scope: Some(named_workspace_scope(workspace)), }; match tokio::time::timeout(Duration::from_secs(5), client.get_sandbox_logs(req)).await { @@ -754,7 +766,7 @@ async fn handle_sandbox_delete(app: &mut App) { let req = openshell_core::proto::DeleteSandboxRequest { name: sandbox_name, - workspace: app.selected_sandbox_workspace(), + workspace_scope: Some(named_workspace_scope(app.selected_sandbox_workspace())), }; match app.client.delete_sandbox(req).await { Ok(_) => { @@ -787,7 +799,7 @@ async fn fetch_sandbox_detail(app: &mut App) { let req = openshell_core::proto::GetSandboxRequest { name: sandbox_name.clone(), - workspace: app.selected_sandbox_workspace(), + workspace_scope: Some(named_workspace_scope(app.selected_sandbox_workspace())), }; // Step 1: Fetch sandbox metadata (providers, sandbox ID). @@ -873,7 +885,7 @@ async fn handle_shell_connect( let sandbox_id = { let req = openshell_core::proto::GetSandboxRequest { name: sandbox_name.clone(), - workspace: app.selected_sandbox_workspace(), + workspace_scope: Some(named_workspace_scope(app.selected_sandbox_workspace())), }; match tokio::time::timeout(Duration::from_secs(5), app.client.get_sandbox(req)).await { Ok(Ok(resp)) => { @@ -1030,7 +1042,7 @@ async fn handle_exec_command( let sandbox_id = { let req = openshell_core::proto::GetSandboxRequest { name: sandbox_name.to_string(), - workspace: workspace.to_string(), + workspace_scope: Some(named_workspace_scope(workspace)), }; match tokio::time::timeout(Duration::from_secs(5), app.client.get_sandbox(req)).await { Ok(Ok(resp)) => { @@ -1406,8 +1418,9 @@ fn spawn_create_sandbox(app: &mut App, tx: mpsc::UnboundedSender) { }), labels: HashMap::new(), annotations: HashMap::new(), - workspace: workspace.clone(), + workspace_scope: Some(named_workspace_scope(&workspace)), await_main_process_attachment: false, + workload_template_name: String::new(), }; let sandbox_name = @@ -1448,7 +1461,7 @@ fn spawn_create_sandbox(app: &mut App, tx: mpsc::UnboundedSender) { let req = openshell_core::proto::GetSandboxRequest { name: sandbox_name.clone(), - workspace: workspace.clone(), + workspace_scope: Some(named_workspace_scope(&workspace)), }; // Retry on transient errors. if let Ok(resp) = client.get_sandbox(req).await @@ -1690,7 +1703,7 @@ fn spawn_create_provider(app: &App, tx: mpsc::UnboundedSender) { profile_workspace: workspace.clone(), credential_handles: HashMap::default(), }), - workspace: workspace.clone(), + workspace_scope: Some(named_workspace_scope(&workspace)), }; match client.create_provider(req).await { @@ -1731,7 +1744,10 @@ fn spawn_get_provider(app: &App, tx: mpsc::UnboundedSender) { let workspace = app.selected_provider_workspace(); tokio::spawn(async move { - let req = openshell_core::proto::GetProviderRequest { name, workspace }; + let req = openshell_core::proto::GetProviderRequest { + name, + workspace_scope: Some(named_workspace_scope(workspace)), + }; match tokio::time::timeout(Duration::from_secs(5), client.get_provider(req)).await { Ok(Ok(resp)) => { if let Some(provider) = resp.into_inner().provider { @@ -1805,7 +1821,7 @@ fn spawn_update_provider(app: &App, tx: mpsc::UnboundedSender) { credential_handles: HashMap::default(), }), credential_expires_at_ms: HashMap::default(), - workspace, + workspace_scope: Some(named_workspace_scope(workspace)), }; match tokio::time::timeout(Duration::from_secs(5), client.update_provider(req)).await { @@ -1834,7 +1850,10 @@ fn spawn_delete_provider(app: &App, tx: mpsc::UnboundedSender) { let workspace = app.selected_provider_workspace(); tokio::spawn(async move { - let req = openshell_core::proto::DeleteProviderRequest { name, workspace }; + let req = openshell_core::proto::DeleteProviderRequest { + name, + workspace_scope: Some(named_workspace_scope(workspace)), + }; match tokio::time::timeout(Duration::from_secs(5), client.delete_provider(req)).await { Ok(Ok(resp)) => { let _ = tx.send(Event::ProviderDeleteResult(Ok(resp.into_inner().deleted))); @@ -1877,7 +1896,7 @@ fn spawn_draft_approve(app: &App, tx: mpsc::UnboundedSender) { let req = openshell_core::proto::ApproveDraftChunkRequest { name, chunk_id, - workspace, + workspace_scope: Some(named_workspace_scope(workspace)), review_token, }; match tokio::time::timeout(Duration::from_secs(5), client.approve_draft_chunk(req)).await { @@ -1923,7 +1942,7 @@ fn spawn_draft_reject(app: &App, tx: mpsc::UnboundedSender) { name, chunk_id, reason: String::new(), - workspace, + workspace_scope: Some(named_workspace_scope(workspace)), }; match tokio::time::timeout(Duration::from_secs(5), client.reject_draft_chunk(req)).await { Ok(Ok(_)) => { @@ -1972,7 +1991,7 @@ fn spawn_draft_approve_all( let req = openshell_core::proto::ApproveAllDraftChunksRequest { name, include_security_flagged: false, - workspace, + workspace_scope: Some(named_workspace_scope(workspace)), approvals, }; match tokio::time::timeout( @@ -2018,30 +2037,6 @@ fn format_draft_approve_all_result( // Data refresh // --------------------------------------------------------------------------- -async fn fetch_providers_v2_setting(app: &mut App) { - let req = openshell_core::proto::GetGatewayConfigRequest {}; - match tokio::time::timeout(Duration::from_secs(5), app.client.get_gateway_config(req)).await { - Ok(Ok(resp)) => { - let response = resp.into_inner(); - let enabled = response - .settings - .get(openshell_core::settings::PROVIDERS_V2_ENABLED_KEY) - .and_then(|s| match &s.value { - Some(openshell_core::proto::setting_value::Value::BoolValue(v)) => Some(*v), - _ => None, - }) - .unwrap_or(false); - app.providers_v2_enabled = enabled; - } - Ok(Err(e)) => { - app.status_text = format!("failed to fetch gateway config: {}", e.message()); - } - Err(_) => { - app.status_text = "gateway config fetch timed out".to_string(); - } - } -} - async fn refresh_data(app: &mut App) { refresh_health(app).await; refresh_global_settings(app).await; @@ -2120,12 +2115,10 @@ async fn refresh_providers(app: &mut App) { let req = openshell_core::proto::ListProvidersRequest { limit: 100, offset: 0, - workspace: if app.all_workspaces { - String::new() - } else { - app.current_workspace.clone() - }, - all_workspaces: app.all_workspaces, + workspace_scope: Some(list_workspace_scope( + &app.current_workspace, + app.all_workspaces, + )), }; let response = match tokio::time::timeout(Duration::from_secs(5), app.client.list_providers(req)).await { @@ -2141,36 +2134,52 @@ async fn refresh_providers(app: &mut App) { }; let providers = response.providers; - let profiles: ProviderProfileCache = if app.providers_v2_enabled { - let workspaces: std::collections::HashSet = providers - .iter() - .map(|provider| provider_profile_query_workspace(provider).to_string()) - // Legacy provider records can decode without an object workspace. Do not - // turn that missing context into a platform-scoped profile request. - .filter(|workspace| !workspace.is_empty()) - .collect(); - let mut all_profiles = HashMap::new(); - for ws in &workspaces { - let req = openshell_core::proto::ListProviderProfilesRequest { - limit: 100, - offset: 0, - workspace: ws.clone(), - }; - if let Ok(Ok(resp)) = tokio::time::timeout( - Duration::from_secs(5), - app.client.list_provider_profiles(req), - ) - .await - { - for profile in resp.into_inner().profiles { - cache_provider_profile(&mut all_profiles, ws, profile); + let mut workspaces: std::collections::HashSet = providers + .iter() + .map(|provider| provider_profile_query_workspace(provider).to_string()) + // Legacy provider records can decode without an object workspace. Do not + // turn that missing context into a platform-scoped profile request. + .filter(|workspace| !workspace.is_empty()) + .collect(); + if !app.all_workspaces { + workspaces.insert(app.current_workspace.clone()); + } + let mut profiles = HashMap::new(); + app.provider_profiles.clear(); + for ws in &workspaces { + let client = app.client.clone(); + let workspace = ws.clone(); + if let Some(listed) = collect_provider_profile_pages(move |offset| { + let mut client = client.clone(); + let workspace = workspace.clone(); + async move { + let req = openshell_core::proto::ListProviderProfilesRequest { + limit: PROVIDER_PROFILE_PAGE_SIZE, + offset, + workspace, + }; + match tokio::time::timeout( + Duration::from_secs(5), + client.list_provider_profiles(req), + ) + .await + { + Ok(Ok(response)) => Some(response.into_inner().profiles), + _ => None, } } + }) + .await + { + if !app.all_workspaces && ws == &app.current_workspace { + app.provider_profiles.clone_from(&listed); + } + for profile in listed { + cache_provider_profile(&mut profiles, ws, profile); + } } - all_profiles - } else { - HashMap::new() - }; + } + app.sync_create_provider_types(); app.provider_count = providers.len(); app.provider_entries = providers @@ -2205,6 +2214,26 @@ async fn refresh_providers(app: &mut App) { } } +async fn collect_provider_profile_pages( + mut fetch_page: F, +) -> Option> +where + F: FnMut(u32) -> Fut, + Fut: Future>>, +{ + let mut profiles = Vec::new(); + let mut offset = 0; + loop { + let page = fetch_page(offset).await?; + let page_len = page.len(); + profiles.extend(page); + if page_len < PROVIDER_PROFILE_PAGE_SIZE as usize { + return Some(profiles); + } + offset = offset.saturating_add(PROVIDER_PROFILE_PAGE_SIZE); + } +} + async fn refresh_global_settings(app: &mut App) { if !app.global_settings_access_denied { let req = openshell_core::proto::GetGatewayConfigRequest {}; @@ -2237,7 +2266,7 @@ async fn refresh_global_settings(app: &mut App) { limit: 1, offset: 0, global: true, - workspace: String::new(), + workspace_scope: None, }; match tokio::time::timeout( Duration::from_secs(5), @@ -2317,7 +2346,6 @@ fn spawn_set_global_setting(app: &App, tx: mpsc::UnboundedSender) { setting_key: key, setting_value: Some(SettingValue { value: Some(value) }), global: true, - workspace: String::new(), ..Default::default() }; @@ -2351,7 +2379,6 @@ fn spawn_delete_global_setting(app: &App, tx: mpsc::UnboundedSender) { setting_key: key, delete_setting: true, global: true, - workspace: String::new(), ..Default::default() }; @@ -2419,7 +2446,7 @@ fn spawn_set_sandbox_setting(app: &App, tx: mpsc::UnboundedSender) { name, setting_key: key, setting_value: Some(SettingValue { value: Some(value) }), - workspace, + workspace_scope: Some(named_workspace_scope(workspace)), ..Default::default() }; @@ -2457,7 +2484,7 @@ fn spawn_delete_sandbox_setting(app: &App, tx: mpsc::UnboundedSender) { name, setting_key: key, delete_setting: true, - workspace, + workspace_scope: Some(named_workspace_scope(workspace)), ..Default::default() }; @@ -2501,12 +2528,10 @@ async fn refresh_sandboxes(app: &mut App) { limit: 100, offset: 0, label_selector: String::new(), - workspace: if app.all_workspaces { - String::new() - } else { - app.current_workspace.clone() - }, - all_workspaces: app.all_workspaces, + workspace_scope: Some(list_workspace_scope( + &app.current_workspace, + app.all_workspaces, + )), }; let result = tokio::time::timeout(Duration::from_secs(5), app.client.list_sandboxes(req)).await; match result { @@ -2655,7 +2680,7 @@ async fn refresh_draft_chunks(app: &mut App) { let req = openshell_core::proto::GetDraftPolicyRequest { name: sandbox_name, status_filter: String::new(), - workspace: app.selected_sandbox_workspace(), + workspace_scope: Some(named_workspace_scope(app.selected_sandbox_workspace())), }; if let Ok(Ok(resp)) = @@ -2686,7 +2711,7 @@ async fn refresh_sandbox_draft_counts(app: &mut App) { let req = openshell_core::proto::GetDraftPolicyRequest { name: name.clone(), status_filter: "pending".to_string(), - workspace: ws, + workspace_scope: Some(named_workspace_scope(ws)), }; if let Ok(Ok(resp)) = tokio::time::timeout(Duration::from_secs(2), app.client.get_draft_policy(req)).await @@ -2863,3 +2888,48 @@ mod provider_profile_workspace_tests { } } } + +#[cfg(test)] +mod provider_profile_pagination_tests { + use super::*; + use std::sync::{Arc, Mutex}; + + #[tokio::test] + async fn profile_fetch_continues_until_page_two_is_collected() { + let requested_offsets = Arc::new(Mutex::new(Vec::new())); + let offsets = Arc::clone(&requested_offsets); + + let profiles = collect_provider_profile_pages(move |offset| { + let offsets = Arc::clone(&offsets); + async move { + offsets.lock().unwrap().push(offset); + match offset { + 0 => Some( + (0..PROVIDER_PROFILE_PAGE_SIZE) + .map(|index| openshell_core::proto::ProviderProfile { + id: format!("profile-{index}"), + ..Default::default() + }) + .collect(), + ), + PROVIDER_PROFILE_PAGE_SIZE => { + Some(vec![openshell_core::proto::ProviderProfile { + id: "page-two-profile".to_string(), + ..Default::default() + }]) + } + _ => panic!("unexpected profile page offset {offset}"), + } + } + }) + .await + .expect("all pages should load"); + + assert_eq!( + *requested_offsets.lock().unwrap(), + vec![0, PROVIDER_PROFILE_PAGE_SIZE] + ); + assert_eq!(profiles.len(), PROVIDER_PROFILE_PAGE_SIZE as usize + 1); + assert_eq!(profiles.last().unwrap().id, "page-two-profile"); + } +} diff --git a/crates/openshell-tui/src/ui/create_provider.rs b/crates/openshell-tui/src/ui/create_provider.rs index cb39bff34..b68ddf5ea 100644 --- a/crates/openshell-tui/src/ui/create_provider.rs +++ b/crates/openshell-tui/src/ui/create_provider.rs @@ -9,6 +9,7 @@ use ratatui::widgets::{Block, Borders, Clear, Padding, Paragraph}; use crate::app::{App, CreateProviderPhase, ProviderKeyField, UpdateProviderField}; use indexmap::IndexMap; +use std::ops::Range; use super::centered_rect; @@ -76,25 +77,48 @@ fn draw_select_type( chunks[0], ); - let lines: Vec> = form - .types - .iter() - .enumerate() - .map(|(i, ty)| { - let is_cursor = i == form.type_cursor; - let marker = if is_cursor { ">" } else { " " }; - let style = if is_cursor { t.accent } else { t.text }; - Line::from(vec![ - Span::styled(format!(" {marker} "), style), - Span::styled(ty.as_str(), style), - ]) - }) - .collect(); + let visible_range = visible_type_range( + form.types.len(), + form.type_cursor, + usize::from(chunks[2].height), + ); + let lines: Vec> = if form.types.is_empty() { + vec![Line::from(Span::styled( + form.status + .as_deref() + .unwrap_or("No provider profiles available."), + t.status_warn, + ))] + } else { + form.types + .iter() + .enumerate() + .skip(visible_range.start) + .take(visible_range.len()) + .map(|(i, ty)| { + let is_cursor = i == form.type_cursor; + let marker = if is_cursor { ">" } else { " " }; + let style = if is_cursor { t.accent } else { t.text }; + Line::from(vec![ + Span::styled(format!(" {marker} "), style), + Span::styled(ty.as_str(), style), + ]) + }) + .collect() + }; frame.render_widget(Paragraph::new(lines), chunks[2]); let hint = Line::from(vec![ Span::styled("[j/k]", t.key_hint), Span::styled(" Navigate ", t.muted), + Span::styled( + format!( + "[{}/{}] ", + form.type_cursor.saturating_add(1).min(form.types.len()), + form.types.len() + ), + t.muted, + ), Span::styled("[Enter]", t.key_hint), Span::styled(" Select ", t.muted), Span::styled("[Esc]", t.key_hint), @@ -103,6 +127,19 @@ fn draw_select_type( frame.render_widget(Paragraph::new(hint), chunks[4]); } +fn visible_type_range(total: usize, cursor: usize, visible_rows: usize) -> Range { + if total == 0 || visible_rows == 0 { + return 0..0; + } + + let visible_rows = visible_rows.min(total); + let cursor = cursor.min(total - 1); + let start = cursor + .saturating_sub(visible_rows - 1) + .min(total - visible_rows); + start..start + visible_rows +} + // --------------------------------------------------------------------------- // Phase 2: Choose method (autodetect vs manual) // --------------------------------------------------------------------------- @@ -1094,3 +1131,23 @@ fn render_status( ); } } + +#[cfg(test)] +mod tests { + use super::visible_type_range; + + #[test] + fn type_list_window_follows_cursor_past_visible_rows() { + assert_eq!(visible_type_range(25, 0, 10), 0..10); + assert_eq!(visible_type_range(25, 9, 10), 0..10); + assert_eq!(visible_type_range(25, 10, 10), 1..11); + assert_eq!(visible_type_range(25, 24, 10), 15..25); + } + + #[test] + fn type_list_window_handles_short_and_empty_lists() { + assert_eq!(visible_type_range(3, 2, 10), 0..3); + assert_eq!(visible_type_range(0, 0, 10), 0..0); + assert_eq!(visible_type_range(3, 2, 0), 0..0); + } +} diff --git a/crates/openshell-tui/src/ui/mod.rs b/crates/openshell-tui/src/ui/mod.rs index 3078ff793..cf5c18736 100644 --- a/crates/openshell-tui/src/ui/mod.rs +++ b/crates/openshell-tui/src/ui/mod.rs @@ -74,8 +74,8 @@ fn draw_sandbox_screen(frame: &mut Frame<'_>, app: &mut App, area: Rect) { let chunks = Layout::default() .direction(Direction::Vertical) .constraints([ - Constraint::Percentage(20), // metadata - Constraint::Percentage(80), // policy or logs + Constraint::Length(sandbox_detail::required_height(app)), // metadata + Constraint::Min(0), // policy or logs ]) .split(area); @@ -143,10 +143,8 @@ fn draw_title_bar(frame: &mut Frame<'_>, app: &App, area: Rect) { .find(|gateway| gateway.name == app.gateway_name) .map_or("unknown", app::GatewayEntry::source_label); - let mut parts: Vec> = vec![ - Span::styled(" >_ OpenShell ", t.accent_bold), - Span::styled(" ALPHA ", t.badge), - Span::styled(" | ", t.muted), + let mut parts: Vec> = title_bar_brand_spans(t); + parts.extend([ Span::styled("Current Gateway: ", t.text), Span::styled(&app.gateway_name, t.heading), Span::styled(" [", t.muted), @@ -155,7 +153,7 @@ fn draw_title_bar(frame: &mut Frame<'_>, app: &App, area: Rect) { status_span, Span::styled(")", t.muted), Span::styled(" | ", t.muted), - ]; + ]); parts.push(Span::styled("Workspace: ", t.text)); parts.push(Span::styled(app.workspace_display(), t.heading)); @@ -180,6 +178,14 @@ fn draw_title_bar(frame: &mut Frame<'_>, app: &App, area: Rect) { frame.render_widget(Paragraph::new(title).style(t.title_bar), area); } +fn title_bar_brand_spans(theme: &Theme) -> Vec> { + vec![ + Span::styled(" >_ OpenShell ", theme.accent_bold), + Span::styled(format!("v{}", openshell_core::VERSION), theme.muted), + Span::styled(" | ", theme.muted), + ] +} + fn draw_nav_bar(frame: &mut Frame<'_>, app: &App, area: Rect) { let t = &app.theme; let spans = match app.screen { @@ -207,27 +213,6 @@ fn draw_nav_bar(frame: &mut Frame<'_>, app: &App, area: Rect) { Span::styled("[q]", t.muted), Span::styled(" Quit", t.muted), ], - Focus::Providers if app.providers_v2_enabled => vec![ - Span::styled(" ", t.text), - Span::styled("[Tab]", t.key_hint), - Span::styled(" Switch Panel", t.text), - Span::styled(" ", t.text), - Span::styled("[h/l]", t.key_hint), - Span::styled(" Switch Tab", t.text), - Span::styled(" ", t.text), - Span::styled("[j/k]", t.key_hint), - Span::styled(" Navigate", t.text), - Span::styled(" ", t.text), - Span::styled("[Enter]", t.key_hint), - Span::styled(" Detail", t.text), - Span::styled(" ", t.text), - Span::styled("read-only", t.muted), - Span::styled(" | ", t.border), - Span::styled("[:]", t.muted), - Span::styled(" Command ", t.muted), - Span::styled("[q]", t.muted), - Span::styled(" Quit", t.muted), - ], Focus::Providers => vec![ Span::styled(" ", t.text), Span::styled("[Tab]", t.key_hint), @@ -242,14 +227,11 @@ fn draw_nav_bar(frame: &mut Frame<'_>, app: &App, area: Rect) { Span::styled("[Enter]", t.key_hint), Span::styled(" Detail", t.text), Span::styled(" ", t.text), - Span::styled("[c]", t.key_hint), - Span::styled(" Create", t.text), + Span::styled("[c/u/d]", t.key_hint), + Span::styled(" Create/Update/Delete", t.text), Span::styled(" ", t.text), - Span::styled("[u]", t.key_hint), - Span::styled(" Update", t.text), - Span::styled(" ", t.text), - Span::styled("[d]", t.key_hint), - Span::styled(" Delete", t.text), + Span::styled("[w]", t.key_hint), + Span::styled(" Workspace", t.text), Span::styled(" | ", t.border), Span::styled("[:]", t.muted), Span::styled(" Command ", t.muted), @@ -689,3 +671,95 @@ pub fn centered_popup(percent_x: u16, height: u16, area: Rect) -> Rect { ]) .split(vert[1])[1] } + +#[cfg(test)] +mod tests { + use super::*; + use openshell_core::auth::EdgeAuthInterceptor; + use openshell_core::proto::open_shell_client::OpenShellClient; + use ratatui::Terminal; + use ratatui::backend::TestBackend; + + fn test_app() -> App { + let channel = tonic::transport::Endpoint::from_static("http://127.0.0.1:1").connect_lazy(); + let client = OpenShellClient::with_interceptor(channel, EdgeAuthInterceptor::noop()); + let mut app = App::new( + client, + "test".to_string(), + "http://127.0.0.1:1".to_string(), + "default".to_string(), + Theme::dark(), + ); + app.screen = Screen::Dashboard; + app.focus = Focus::Providers; + app + } + + #[tokio::test] + async fn providers_navigation_advertises_workspace_shortcut() { + let app = test_app(); + let mut terminal = Terminal::new(TestBackend::new(180, 1)).unwrap(); + + terminal + .draw(|frame| draw_nav_bar(frame, &app, frame.size())) + .unwrap(); + + let text: String = terminal + .backend() + .buffer() + .content() + .iter() + .map(ratatui::buffer::Cell::symbol) + .collect(); + assert!(text.contains("[w] Workspace"), "nav bar was: {text:?}"); + } + + #[test] + fn title_bar_brand_renders_resolved_version_without_alpha_badge() { + let expected = format!(" >_ OpenShell v{} | ", openshell_core::VERSION); + let width = u16::try_from(expected.len()).unwrap(); + let backend = TestBackend::new(width, 1); + let mut terminal = Terminal::new(backend).unwrap(); + + terminal + .draw(|frame| { + frame.render_widget( + Paragraph::new(Line::from(title_bar_brand_spans(&Theme::dark()))), + frame.size(), + ); + }) + .unwrap(); + + let buffer = terminal.backend().buffer(); + let rendered = (0..width) + .map(|x| buffer.get(x, 0).symbol()) + .collect::(); + assert_eq!(rendered, expected); + assert!(!rendered.contains("ALPHA")); + } + + #[tokio::test] + async fn sandbox_delete_confirmation_is_visible_in_standard_terminal() { + let mut app = test_app(); + app.screen = Screen::Sandbox; + app.focus = Focus::SandboxPolicy; + app.sandbox_names = vec!["test-sandbox".to_string()]; + app.sandbox_count = 1; + app.confirm_delete = true; + let mut terminal = Terminal::new(TestBackend::new(100, 24)).unwrap(); + + terminal.draw(|frame| draw(frame, &mut app)).unwrap(); + + let text: String = terminal + .backend() + .buffer() + .content() + .iter() + .map(ratatui::buffer::Cell::symbol) + .collect(); + assert!( + text.contains("Delete sandbox 'test-sandbox'?"), + "sandbox screen was: {text:?}" + ); + } +} diff --git a/crates/openshell-tui/src/ui/providers.rs b/crates/openshell-tui/src/ui/providers.rs index e4f4d2d0b..3bf5c7f94 100644 --- a/crates/openshell-tui/src/ui/providers.rs +++ b/crates/openshell-tui/src/ui/providers.rs @@ -9,12 +9,12 @@ use ratatui::widgets::{Block, Borders, Cell, Padding, Row, Table}; use crate::app::App; pub fn draw(frame: &mut Frame<'_>, app: &App, area: Rect, focused: bool) { - let t = &app.theme; - if app.providers_v2_enabled { - draw_v2(frame, app, area, focused); - return; - } + draw_v2(frame, app, area, focused); +} +#[allow(dead_code)] +fn draw_legacy(frame: &mut Frame<'_>, app: &App, area: Rect, focused: bool) { + let t = &app.theme; let show_ws = app.all_workspaces; let mut header_cells = Vec::new(); diff --git a/crates/openshell-tui/src/ui/sandbox_detail.rs b/crates/openshell-tui/src/ui/sandbox_detail.rs index 4ff78617f..a398797e4 100644 --- a/crates/openshell-tui/src/ui/sandbox_detail.rs +++ b/crates/openshell-tui/src/ui/sandbox_detail.rs @@ -8,6 +8,37 @@ use ratatui::widgets::{Block, Borders, Padding, Paragraph}; use crate::app::App; +const BASE_CONTENT_ROWS: u16 = 6; +const BORDER_ROWS: u16 = 2; + +fn pending_draft_count(app: &App) -> usize { + let cached = app + .sandbox_draft_counts + .get(app.sandbox_selected) + .copied() + .unwrap_or(0); + if cached > 0 { + cached + } else { + app.draft_chunks + .iter() + .filter(|chunk| chunk.status == "pending") + .count() + } +} + +/// Return the rows needed to render every metadata line without clipping. +pub(super) fn required_height(app: &App) -> u16 { + let policy_rows = u16::from(app.sandbox_policy_is_global); + let action_rows = if app.confirm_delete { + 2 // spacer plus confirmation + } else { + u16::from(pending_draft_count(app) > 0) + }; + + BASE_CONTENT_ROWS + policy_rows + action_rows + BORDER_ROWS +} + /// Draw a compact metadata pane for the currently selected sandbox. /// /// This is non-interactive (no focus state) — always rendered with the @@ -37,16 +68,8 @@ pub fn draw(frame: &mut Frame<'_>, app: &App, area: Rect) { }; // Count pending draft recommendations for this sandbox. - let pending_count = app.sandbox_draft_counts.get(idx).copied().unwrap_or(0); - // Also check the live draft_chunks when on the sandbox screen (more up-to-date). - let pending_count = if pending_count > 0 { - pending_count - } else { - app.draft_chunks - .iter() - .filter(|c| c.status == "pending") - .count() - }; + // Fall back to the live chunks when the dashboard cache has no pending count. + let pending_count = pending_draft_count(app); // Row 1: Name + Status + optional draft badge let mut row1_spans = vec![ diff --git a/crates/openshell-tui/src/ui/splash.rs b/crates/openshell-tui/src/ui/splash.rs index 7f889f995..a46667fb3 100644 --- a/crates/openshell-tui/src/ui/splash.rs +++ b/crates/openshell-tui/src/ui/splash.rs @@ -104,16 +104,11 @@ pub fn draw(frame: &mut Frame<'_>, area: Rect, theme: &crate::theme::Theme) { frame.render_widget(Paragraph::new(content_lines), chunks[0]); - // -- Footer: version + ALPHA badge on line 1, prompt on line 2 -- + // -- Footer: version on line 1, prompt on line 2 -- let version = format!("v{}", openshell_core::VERSION); - let alpha_badge = "ALPHA"; let footer = Paragraph::new(vec![ - Line::from(vec![ - Span::styled(version, t.accent), - Span::styled(" ", t.muted), - Span::styled(alpha_badge, t.title_bar), - ]), + Line::from(Span::styled(version, t.accent)), Line::from(Span::styled("press any key ░", t.muted)), ]); diff --git a/deny.toml b/deny.toml index c4dab1233..b91a0699f 100644 --- a/deny.toml +++ b/deny.toml @@ -21,9 +21,6 @@ ignore = [ { id = "RUSTSEC-2026-0204", reason = "crossbeam-epoch pointer deref — transitive via metrics/quanta" }, { id = "RUSTSEC-2023-0071", reason = "rsa Marvin attack — transitive via spiffe, no direct exposure" }, { id = "RUSTSEC-2025-0134", reason = "rustls-pemfile unmaintained — transitive via older kube/hyper" }, - { id = "RUSTSEC-2026-0098", reason = "rustls-webpki URI name constraints — transitive via older rustls" }, - { id = "RUSTSEC-2026-0099", reason = "rustls-webpki wildcard name constraints — transitive via older rustls" }, - { id = "RUSTSEC-2026-0104", reason = "rustls-webpki CRL parsing panic — transitive via older rustls" }, { id = "RUSTSEC-2025-0068", reason = "serde_yml unsound+unmaintained — direct dep, no maintained alternative yet" }, ] @@ -56,6 +53,7 @@ registries = [] # -- Bans ---------------------------------------------------------------------- [bans] +deny = [{ name = "ring", reason = "Use the AWS-LC crypto backend instead." }] multiple-versions = "warn" wildcards = "allow" highlight = "all" diff --git a/deploy/docker/Dockerfile.cli-macos b/deploy/docker/Dockerfile.cli-macos index af509ee97..f86cd594b 100644 --- a/deploy/docker/Dockerfile.cli-macos +++ b/deploy/docker/Dockerfile.cli-macos @@ -74,9 +74,7 @@ COPY crates/openshell-prover/Cargo.toml crates/openshell-prover/Cargo.toml COPY crates/openshell-core/build.rs crates/openshell-core/build.rs COPY proto/ proto/ -# Scope workspace to CLI crates only — avoids compiling aws-lc-sys (pulled -# by russh in openshell-sandbox/openshell-server) which is difficult to -# cross-compile and unnecessary for the CLI binary. +# Scope workspace to CLI crates only to avoid compiling unrelated runtimes. RUN sed -i 's|members = \["crates/\*"\]|members = ["crates/openshell-cli", "crates/openshell-core", "crates/openshell-bootstrap", "crates/openshell-policy", "crates/openshell-prover", "crates/openshell-providers", "crates/openshell-tui"]|' Cargo.toml RUN mkdir -p crates/openshell-cli/src \ diff --git a/deploy/docker/Dockerfile.gateway-macos b/deploy/docker/Dockerfile.gateway-macos index 122f16eba..4ecbac039 100644 --- a/deploy/docker/Dockerfile.gateway-macos +++ b/deploy/docker/Dockerfile.gateway-macos @@ -53,47 +53,47 @@ ENV BINDGEN_EXTRA_CLANG_ARGS_aarch64_apple_darwin=--target=arm64-apple-macosx\ - COPY Cargo.toml Cargo.lock ./ COPY crates/openshell-core/Cargo.toml crates/openshell-core/Cargo.toml +COPY crates/openshell-gateway/Cargo.toml crates/openshell-gateway/Cargo.toml COPY crates/openshell-driver-kubernetes/Cargo.toml crates/openshell-driver-kubernetes/Cargo.toml COPY crates/openshell-policy/Cargo.toml crates/openshell-policy/Cargo.toml COPY crates/openshell-prover/Cargo.toml crates/openshell-prover/Cargo.toml -COPY crates/openshell-router/Cargo.toml crates/openshell-router/Cargo.toml COPY crates/openshell-server/Cargo.toml crates/openshell-server/Cargo.toml COPY crates/openshell-core/build.rs crates/openshell-core/build.rs COPY proto/ proto/ -RUN sed -i 's|members = \["crates/\*"\]|members = ["crates/openshell-server", "crates/openshell-core", "crates/openshell-driver-kubernetes", "crates/openshell-policy", "crates/openshell-prover", "crates/openshell-router"]|' Cargo.toml +RUN sed -i 's|members = \["crates/\*"\]|members = ["crates/openshell-gateway", "crates/openshell-server", "crates/openshell-core", "crates/openshell-driver-kubernetes", "crates/openshell-policy", "crates/openshell-prover"]|' Cargo.toml RUN mkdir -p crates/openshell-core/src \ + crates/openshell-gateway/src \ crates/openshell-driver-kubernetes/src \ crates/openshell-policy/src \ crates/openshell-prover/src \ - crates/openshell-router/src \ crates/openshell-server/src && \ touch crates/openshell-core/src/lib.rs && \ + touch crates/openshell-gateway/src/lib.rs && \ + printf 'fn main() {}\n' > crates/openshell-gateway/src/main.rs && \ touch crates/openshell-driver-kubernetes/src/lib.rs && \ printf 'fn main() {}\n' > crates/openshell-driver-kubernetes/src/main.rs && \ touch crates/openshell-policy/src/lib.rs && \ touch crates/openshell-prover/src/lib.rs && \ - touch crates/openshell-router/src/lib.rs && \ - touch crates/openshell-server/src/lib.rs && \ - printf 'fn main() {}\n' > crates/openshell-server/src/main.rs + touch crates/openshell-server/src/lib.rs RUN --mount=type=cache,id=cargo-registry-gateway-macos,sharing=locked,target=/root/.cargo/registry \ --mount=type=cache,id=cargo-git-gateway-macos,sharing=locked,target=/root/.cargo/git \ --mount=type=cache,id=cargo-target-gateway-macos-${CARGO_TARGET_CACHE_SCOPE},sharing=locked,target=/build/target \ - cargo build --release --target aarch64-apple-darwin -p openshell-server --features bundled-z3 2>/dev/null || true + cargo build --release --target aarch64-apple-darwin -p openshell-gateway --features bundled-z3 2>/dev/null || true COPY crates/ crates/ COPY providers/ providers/ RUN touch crates/openshell-core/src/lib.rs \ + crates/openshell-gateway/src/lib.rs \ + crates/openshell-gateway/src/main.rs \ crates/openshell-driver-kubernetes/src/lib.rs \ crates/openshell-driver-kubernetes/src/main.rs \ crates/openshell-policy/src/lib.rs \ crates/openshell-prover/src/lib.rs \ - crates/openshell-router/src/lib.rs \ crates/openshell-server/src/lib.rs \ - crates/openshell-server/src/main.rs \ crates/openshell-core/build.rs \ proto/*.proto @@ -105,7 +105,7 @@ RUN --mount=type=cache,id=cargo-registry-gateway-macos,sharing=locked,target=/ro if [ -n "${OPENSHELL_CARGO_VERSION:-}" ]; then \ sed -i -E '/^\[workspace\.package\]/,/^\[/{s/^version[[:space:]]*=[[:space:]]*".*"/version = "'"${OPENSHELL_CARGO_VERSION}"'"/}' Cargo.toml; \ fi && \ - cargo build --release --target aarch64-apple-darwin -p openshell-server --features bundled-z3 && \ + cargo build --release --target aarch64-apple-darwin -p openshell-gateway --features bundled-z3 && \ cp target/aarch64-apple-darwin/release/openshell-gateway /openshell-gateway FROM scratch AS binary diff --git a/deploy/helm/openshell-workspace/Chart.yaml b/deploy/helm/openshell-workspace/Chart.yaml new file mode 100644 index 000000000..03a3919d4 --- /dev/null +++ b/deploy/helm/openshell-workspace/Chart.yaml @@ -0,0 +1,9 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +apiVersion: v2 +name: openshell-workspace +description: Namespace-scoped prerequisites for OpenShell Kubernetes sandboxes +type: application +version: 0.0.0 +appVersion: "0.0.0" diff --git a/deploy/helm/openshell-workspace/README.md b/deploy/helm/openshell-workspace/README.md new file mode 100644 index 000000000..96ae4ca6e --- /dev/null +++ b/deploy/helm/openshell-workspace/README.md @@ -0,0 +1,47 @@ +# OpenShell Workspace Helm Chart + + + +> **Experimental** - the shared-gateway, multi-namespace deployment path is +> under active design. + +This chart installs the namespace-scoped ServiceAccount, RBAC, and NetworkPolicy +needed for OpenShell Kubernetes sandboxes. Install it once in every +platform-managed workspace namespace. It does not create a namespace or deploy +an OpenShell gateway. + +Install the gateway chart with `workspaceResources.enabled=false`, then install +this chart with the gateway ServiceAccount identity. Configure the gateway's +Kubernetes driver in `operator` workspace mode when it serves more than one +pre-provisioned workspace namespace: + +```shell +helm install openshell-workspace ./deploy/helm/openshell-workspace \ + --namespace app-a \ + --set gateway.serviceAccount.name=openshell \ + --set gateway.serviceAccount.namespace=openshell +``` + +Keep `sandboxServiceAccount.name` aligned with the gateway chart's +`sandboxServiceAccount.name`. The defaults for both charts are +`openshell-sandbox`. + +## Values + +| Key | Type | Default | Description | +|-----|------|---------|-------------| +| fullnameOverride | string | `""` | Override the full generated resource name. | +| gateway.networkPolicy.podSelector | object | `{"app.kubernetes.io/instance":"openshell","app.kubernetes.io/name":"openshell"}` | Labels selecting gateway pods allowed to reach sandbox SSH. | +| gateway.serviceAccount.name | string | `"openshell"` | Name of the shared gateway ServiceAccount. | +| gateway.serviceAccount.namespace | string | `"openshell"` | Namespace containing the shared gateway ServiceAccount. | +| nameOverride | string | `""` | Override the chart name used in generated resource names. | +| networkPolicy.enabled | bool | `true` | Restrict sandbox SSH ingress to the shared gateway pods. | +| sandboxServiceAccount.annotations | object | `{}` | Annotations added to the generated sandbox ServiceAccount. | +| sandboxServiceAccount.create | bool | `true` | Create the ServiceAccount assigned to sandbox pods. | +| sandboxServiceAccount.name | string | `"openshell-sandbox"` | Sandbox ServiceAccount name. | + +---------------------------------------------- +Autogenerated from chart metadata using [helm-docs v1.14.2](https://github.com/norwoodj/helm-docs/releases/v1.14.2) diff --git a/deploy/helm/openshell-workspace/README.md.gotmpl b/deploy/helm/openshell-workspace/README.md.gotmpl new file mode 100644 index 000000000..a233eef16 --- /dev/null +++ b/deploy/helm/openshell-workspace/README.md.gotmpl @@ -0,0 +1,33 @@ +# OpenShell Workspace Helm Chart + + + +> **Experimental** - the shared-gateway, multi-namespace deployment path is +> under active design. + +This chart installs the namespace-scoped ServiceAccount, RBAC, and NetworkPolicy +needed for OpenShell Kubernetes sandboxes. Install it once in every +platform-managed workspace namespace. It does not create a namespace or deploy +an OpenShell gateway. + +Install the gateway chart with `workspaceResources.enabled=false`, then install +this chart with the gateway ServiceAccount identity. Configure the gateway's +Kubernetes driver in `operator` workspace mode when it serves more than one +pre-provisioned workspace namespace: + +```shell +helm install openshell-workspace ./deploy/helm/openshell-workspace \ + --namespace app-a \ + --set gateway.serviceAccount.name=openshell \ + --set gateway.serviceAccount.namespace=openshell +``` + +Keep `sandboxServiceAccount.name` aligned with the gateway chart's +`sandboxServiceAccount.name`. The defaults for both charts are +`openshell-sandbox`. + +{{ template "chart.valuesSection" . }} +{{ template "helm-docs.versionFooter" . }} diff --git a/deploy/helm/openshell-workspace/templates/_helpers.tpl b/deploy/helm/openshell-workspace/templates/_helpers.tpl new file mode 100644 index 000000000..eb948b990 --- /dev/null +++ b/deploy/helm/openshell-workspace/templates/_helpers.tpl @@ -0,0 +1,44 @@ +{{/* +Expand the name of the chart. +*/}} +{{- define "openshell-workspace.name" -}} +{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" }} +{{- end }} + +{{/* +Create a default fully qualified app name. +*/}} +{{- define "openshell-workspace.fullname" -}} +{{- if .Values.fullnameOverride }} +{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" }} +{{- else }} +{{- $name := default .Chart.Name .Values.nameOverride }} +{{- if contains $name .Release.Name }} +{{- .Release.Name | trunc 63 | trimSuffix "-" }} +{{- else }} +{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" }} +{{- end }} +{{- end }} +{{- end }} + +{{/* +Common labels. +*/}} +{{- define "openshell-workspace.labels" -}} +helm.sh/chart: {{ printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" }} +app.kubernetes.io/name: {{ include "openshell-workspace.name" . }} +app.kubernetes.io/instance: {{ .Release.Name }} +{{- if .Chart.AppVersion }} +app.kubernetes.io/version: {{ .Chart.AppVersion | quote }} +{{- end }} +app.kubernetes.io/managed-by: {{ .Release.Service }} +{{- end }} + +{{/* +Validate required cross-namespace gateway identity values. +*/}} +{{- define "openshell-workspace.validateValues" -}} +{{- $gatewayServiceAccountName := required "gateway.serviceAccount.name is required" .Values.gateway.serviceAccount.name -}} +{{- $gatewayServiceAccountNamespace := required "gateway.serviceAccount.namespace is required" .Values.gateway.serviceAccount.namespace -}} +{{- $sandboxServiceAccountName := required "sandboxServiceAccount.name is required" .Values.sandboxServiceAccount.name -}} +{{- end }} diff --git a/deploy/helm/openshell-workspace/templates/networkpolicy.yaml b/deploy/helm/openshell-workspace/templates/networkpolicy.yaml new file mode 100644 index 000000000..f88cf84d0 --- /dev/null +++ b/deploy/helm/openshell-workspace/templates/networkpolicy.yaml @@ -0,0 +1,30 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +{{- include "openshell-workspace.validateValues" . }} +{{- if .Values.networkPolicy.enabled }} +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: {{ include "openshell-workspace.fullname" . }}-sandbox-ssh + namespace: {{ .Release.Namespace }} + labels: + {{- include "openshell-workspace.labels" . | nindent 4 }} +spec: + podSelector: + matchLabels: + openshell.ai/managed-by: openshell + policyTypes: + - Ingress + ingress: + - from: + - namespaceSelector: + matchLabels: + kubernetes.io/metadata.name: {{ .Values.gateway.serviceAccount.namespace }} + podSelector: + matchLabels: + {{- toYaml .Values.gateway.networkPolicy.podSelector | nindent 14 }} + ports: + - protocol: TCP + port: 2222 +{{- end }} diff --git a/deploy/helm/openshell-workspace/templates/role.yaml b/deploy/helm/openshell-workspace/templates/role.yaml new file mode 100644 index 000000000..45b3beb83 --- /dev/null +++ b/deploy/helm/openshell-workspace/templates/role.yaml @@ -0,0 +1,39 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +{{- include "openshell-workspace.validateValues" . }} +apiVersion: rbac.authorization.k8s.io/v1 +kind: Role +metadata: + name: {{ include "openshell-workspace.fullname" . }}-sandbox + namespace: {{ .Release.Namespace }} + labels: + {{- include "openshell-workspace.labels" . | nindent 4 }} +rules: + - apiGroups: + - agents.x-k8s.io + resources: + - sandboxes + - sandboxes/status + verbs: + - create + - delete + - get + - list + - patch + - update + - watch + - apiGroups: + - "" + resources: + - events + verbs: + - get + - list + - watch + - apiGroups: + - "" + resources: + - pods + verbs: + - get diff --git a/deploy/helm/openshell-workspace/templates/rolebinding.yaml b/deploy/helm/openshell-workspace/templates/rolebinding.yaml new file mode 100644 index 000000000..669ec4762 --- /dev/null +++ b/deploy/helm/openshell-workspace/templates/rolebinding.yaml @@ -0,0 +1,19 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +{{- include "openshell-workspace.validateValues" . }} +apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: + name: {{ include "openshell-workspace.fullname" . }}-sandbox + namespace: {{ .Release.Namespace }} + labels: + {{- include "openshell-workspace.labels" . | nindent 4 }} +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: Role + name: {{ include "openshell-workspace.fullname" . }}-sandbox +subjects: + - kind: ServiceAccount + name: {{ .Values.gateway.serviceAccount.name }} + namespace: {{ .Values.gateway.serviceAccount.namespace }} diff --git a/deploy/helm/openshell-workspace/templates/serviceaccount.yaml b/deploy/helm/openshell-workspace/templates/serviceaccount.yaml new file mode 100644 index 000000000..20bb263b5 --- /dev/null +++ b/deploy/helm/openshell-workspace/templates/serviceaccount.yaml @@ -0,0 +1,17 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +{{- include "openshell-workspace.validateValues" . }} +{{- if .Values.sandboxServiceAccount.create }} +apiVersion: v1 +kind: ServiceAccount +metadata: + name: {{ .Values.sandboxServiceAccount.name }} + namespace: {{ .Release.Namespace }} + labels: + {{- include "openshell-workspace.labels" . | nindent 4 }} + {{- with .Values.sandboxServiceAccount.annotations }} + annotations: + {{- toYaml . | nindent 4 }} + {{- end }} +{{- end }} diff --git a/deploy/helm/openshell-workspace/tests/workspace_test.yaml b/deploy/helm/openshell-workspace/tests/workspace_test.yaml new file mode 100644 index 000000000..2f71920ea --- /dev/null +++ b/deploy/helm/openshell-workspace/tests/workspace_test.yaml @@ -0,0 +1,61 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +suite: workspace namespace resources +templates: + - templates/serviceaccount.yaml + - templates/role.yaml + - templates/rolebinding.yaml + - templates/networkpolicy.yaml +release: + name: tenant-a + namespace: app-a + +tests: + - it: creates the workspace role in the release namespace + template: templates/role.yaml + asserts: + - hasDocuments: + count: 1 + - equal: + path: metadata.namespace + value: app-a + + - it: binds the shared gateway service account + template: templates/rolebinding.yaml + set: + gateway.serviceAccount.name: shared-gateway + gateway.serviceAccount.namespace: openshell-system + asserts: + - equal: + path: metadata.namespace + value: app-a + - equal: + path: subjects[0].name + value: shared-gateway + - equal: + path: subjects[0].namespace + value: openshell-system + + - it: selects gateway pods in the gateway namespace + template: templates/networkpolicy.yaml + set: + gateway.serviceAccount.namespace: openshell-system + gateway.networkPolicy.podSelector: + app.kubernetes.io/name: openshell + app.kubernetes.io/instance: central + asserts: + - equal: + path: spec.ingress[0].from[0].namespaceSelector.matchLabels["kubernetes.io/metadata.name"] + value: openshell-system + - equal: + path: spec.ingress[0].from[0].podSelector.matchLabels["app.kubernetes.io/instance"] + value: central + + - it: supports a pre-existing sandbox service account + template: templates/serviceaccount.yaml + set: + sandboxServiceAccount.create: false + asserts: + - hasDocuments: + count: 0 diff --git a/deploy/helm/openshell-workspace/values.yaml b/deploy/helm/openshell-workspace/values.yaml new file mode 100644 index 000000000..2c52b0460 --- /dev/null +++ b/deploy/helm/openshell-workspace/values.yaml @@ -0,0 +1,31 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +# -- Override the chart name used in generated resource names. +nameOverride: "" +# -- Override the full generated resource name. +fullnameOverride: "" + +gateway: + serviceAccount: + # -- Name of the shared gateway ServiceAccount. + name: openshell + # -- Namespace containing the shared gateway ServiceAccount. + namespace: openshell + networkPolicy: + # -- Labels selecting gateway pods allowed to reach sandbox SSH. + podSelector: + app.kubernetes.io/name: openshell + app.kubernetes.io/instance: openshell + +sandboxServiceAccount: + # -- Create the ServiceAccount assigned to sandbox pods. + create: true + # -- Sandbox ServiceAccount name. + name: openshell-sandbox + # -- Annotations added to the generated sandbox ServiceAccount. + annotations: {} + +networkPolicy: + # -- Restrict sandbox SSH ingress to the shared gateway pods. + enabled: true diff --git a/deploy/helm/openshell/README.md b/deploy/helm/openshell/README.md index 1786a46f6..fb3161a60 100644 --- a/deploy/helm/openshell/README.md +++ b/deploy/helm/openshell/README.md @@ -9,6 +9,14 @@ Edit README.md.gotmpl and values.yaml, then run `mise run helm:docs`. This chart deploys the OpenShell gateway into a Kubernetes cluster. It is published as an OCI artifact to GHCR at `oci://ghcr.io/nvidia/openshell/helm-chart`. +By default, this chart also creates the namespace-scoped resources needed by +sandboxes. For a shared-gateway deployment, install it with +`workspaceResources.enabled=false`, then install the +`deploy/helm/openshell-workspace` chart in every pre-provisioned workspace +namespace. The gateway and workspace releases can then be upgraded and removed +independently. Use Kubernetes `operator` workspace mode when one gateway serves +multiple pre-provisioned workspace namespaces. + ## Prerequisites The Kubernetes Agent Sandbox CRDs and controller must be installed on the cluster before deploying OpenShell. Install them with: @@ -258,7 +266,7 @@ discovery endpoint or its TLS CA. | server.oidc.audience | string | `"openshell-cli"` | Expected audience claim for the API resource server. This should match the server's --oidc-audience, NOT the CLI client ID. | | server.oidc.caConfigMapName | string | `""` | Name of a ConfigMap containing a CA certificate bundle (key: ca.crt) for verifying the OIDC issuer's TLS certificate. Required when the issuer uses a non-public CA (e.g. OpenShift ingress, private PKI). | | server.oidc.issuer | string | `""` | OIDC issuer URL (e.g. https://keycloak.example.com/realms/openshell). | -| server.oidc.jwksTtl | int | `3600` | JWKS key cache TTL in seconds. | +| server.oidc.jwksTtl | int | `3600` | JWKS key cache TTL in seconds. Must be greater than zero. | | server.oidc.rolesClaim | string | `""` | Dot-separated path to the roles array in the JWT claims. Keycloak: "realm_access.roles", Entra ID: "roles", Okta: "groups". | | server.oidc.scopesClaim | string | `""` | Dot-separated path to the scopes array in the JWT claims. | | server.oidc.userRole | string | `""` | Role name for standard user access. | @@ -278,7 +286,7 @@ discovery endpoint or its TLS CA. | server.sandboxNamespace | string | `""` | Namespace where sandbox pods are created. Defaults to the Helm release namespace (.Release.Namespace) when left empty. | | server.telemetryEnabled | bool | `true` | Enable anonymous OpenShell telemetry from the gateway and the sandbox supervisors it launches. | | server.tls.certSecretName | string | `"openshell-server-tls"` | K8s secret (type kubernetes.io/tls) with tls.crt and tls.key for the server. | -| server.tls.clientCaSecretName | string | `"openshell-server-client-ca"` | K8s secret with ca.crt for client certificate verification (mTLS). Set to "" to disable mTLS and run HTTPS-only (use OIDC for auth instead). | +| server.tls.clientCaSecretName | string | `"openshell-server-client-ca"` | K8s secret with ca.crt for client certificate verification (mTLS). Set to "" to disable mTLS and run HTTPS-only (use OIDC for auth instead). Do not set to null; omit the key to use the default secret name above. | | server.tls.clientTlsSecretName | string | `"openshell-client-tls"` | K8s secret mounted into sandbox pods for mTLS to the server. | | server.workspaceDefaultStorageSize | string | `""` | Default storage size for the workspace PVC in sandbox pods. Uses Kubernetes quantity syntax (e.g. "2Gi", "10Gi", "500Mi"). Empty = built-in default (2Gi). | | server.workspaceStorageClass | string | `""` | Kubernetes StorageClass for the workspace PVC in sandbox pods. Empty (default) = omit storageClassName, using the cluster's default StorageClass. Set this on clusters with no default StorageClass, otherwise the workspace PVC stays Pending and the sandbox never starts. | @@ -306,6 +314,7 @@ discovery endpoint or its TLS CA. | upstreamProxy.url | string | `""` | HTTP proxy URL in http://host:port form. HTTPS-to-proxy is not supported. | | workload.allowMultiReplicaStatefulSet | bool | `false` | Allow replicaCount > 1 while rendering a StatefulSet. Prefer workload.kind=deployment for external database-backed multi-replica gateways; this override exists for operators who explicitly require StatefulSet identity or storage semantics. | | workload.kind | string | `"statefulset"` | Gateway workload controller kind. Use `statefulset` for the default SQLite database, or `deployment` when server.externalDbSecret points at an external database. | +| workspaceResources.enabled | bool | `true` | Create the sandbox ServiceAccount, Role, RoleBinding, and NetworkPolicy from this chart. Disable for a gateway-only release. | ---------------------------------------------- Autogenerated from chart metadata using [helm-docs v1.14.2](https://github.com/norwoodj/helm-docs/releases/v1.14.2) diff --git a/deploy/helm/openshell/README.md.gotmpl b/deploy/helm/openshell/README.md.gotmpl index 73ebb39c8..cf8677741 100644 --- a/deploy/helm/openshell/README.md.gotmpl +++ b/deploy/helm/openshell/README.md.gotmpl @@ -9,6 +9,14 @@ Edit README.md.gotmpl and values.yaml, then run `mise run helm:docs`. This chart deploys the OpenShell gateway into a Kubernetes cluster. It is published as an OCI artifact to GHCR at `oci://ghcr.io/nvidia/openshell/helm-chart`. +By default, this chart also creates the namespace-scoped resources needed by +sandboxes. For a shared-gateway deployment, install it with +`workspaceResources.enabled=false`, then install the +`deploy/helm/openshell-workspace` chart in every pre-provisioned workspace +namespace. The gateway and workspace releases can then be upgraded and removed +independently. Use Kubernetes `operator` workspace mode when one gateway serves +multiple pre-provisioned workspace namespaces. + ## Prerequisites The Kubernetes Agent Sandbox CRDs and controller must be installed on the cluster before deploying OpenShell. Install them with: diff --git a/deploy/helm/openshell/ci/values-openshift-e2e.yaml b/deploy/helm/openshell/ci/values-openshift-e2e.yaml new file mode 100644 index 000000000..d2aaf0fac --- /dev/null +++ b/deploy/helm/openshell/ci/values-openshift-e2e.yaml @@ -0,0 +1,53 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +# OpenShift overlay for the Kubernetes e2e harness. +# +# Bundles the OpenShift-specific settings the harness needs: the Route/mTLS +# transport, and an image pull policy that avoids stale cached images. +# +# Route/mTLS transport: on OpenShift, `kubectl port-forward` stalls the SSH-relay +# `sandbox connect` path (round-trip-heavy SSH over SPDY), so the harness drives +# the gateway through a passthrough OpenShift Route with mTLS instead. This +# overlay turns TLS back on (values-skaffold.yaml disables it), enables the Route, +# and promotes the cert-verified caller to a dev principal. +# +# Image pull policy: force `Always` so runs against the `latest` upstream image +# actually use it, instead of a stale copy cached on the cluster nodes. +# +# Layered by e2e/with-kube-gateway.sh AFTER ci/values-skaffold.yaml and +# ci/values-openshift-scc.yaml when an OpenShift cluster is detected. The harness +# supplies `openshiftRoute.host` and `pkiInitJob.serverDnsNames[0]` via --set at +# install time (both are the cluster-derived Route hostname). +# +# Security: this is NOT an open gateway. `server.tls.clientCaSecretName` defaults +# to `openshell-server-client-ca` and there is no OIDC, so `require_client_auth` +# is true and mTLS is MANDATORY at the TLS handshake — a caller with only the +# Route URL and no client certificate is rejected before any RPC. The passthrough +# Route terminates TLS at the gateway pod, so this holds end-to-end. +# `allowUnauthenticatedUsers` only promotes the already cert-verified caller to a +# dev principal at the app layer (mtls_auth is unsupported with the Kubernetes +# driver). Both are required together; the client certificate is the access gate. +image: + pullPolicy: Always + +supervisor: + image: + pullPolicy: Always + +server: + disableTls: false + auth: + allowUnauthenticatedUsers: true + +openshiftRoute: + enabled: true + # host is supplied via --set at install time (cluster-derived Route hostname). + # The default HAProxy Route timeout is 30s, which severs long-lived transfers + # (large sandbox upload/download, SSH-relay `sandbox connect`) mid-stream. Raise + # both the connection timeout and the passthrough tunnel timeout so these paths + # survive. Passthrough Routes proxy in TCP mode, so timeout-tunnel governs the + # established tunnel while timeout covers the pre-tunnel phase. + annotations: + haproxy.router.openshift.io/timeout: 300s + haproxy.router.openshift.io/timeout-tunnel: 300s diff --git a/deploy/helm/openshell/ci/values-openshift-scc.yaml b/deploy/helm/openshell/ci/values-openshift-scc.yaml new file mode 100644 index 000000000..b7f37be6e --- /dev/null +++ b/deploy/helm/openshell/ci/values-openshift-scc.yaml @@ -0,0 +1,20 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +# OpenShift SCC compatibility overlay. Removes the hardcoded runAsUser and +# fsGroup so that OpenShift's restricted-v2 SCC can inject the namespace- +# assigned UID/GID range. Layer after values.yaml: +# helm install openshell deploy/helm/openshell -f ci/values-openshift-scc.yaml +# +# The e2e Kubernetes harness applies this automatically when it detects an +# OpenShift cluster (route.openshift.io API present). + +podSecurityContext: null + +securityContext: + runAsNonRoot: true + runAsUser: null + allowPrivilegeEscalation: false + capabilities: + drop: + - ALL diff --git a/deploy/helm/openshell/templates/_gateway-workload.tpl b/deploy/helm/openshell/templates/_gateway-workload.tpl index a73acc981..25662518e 100644 --- a/deploy/helm/openshell/templates/_gateway-workload.tpl +++ b/deploy/helm/openshell/templates/_gateway-workload.tpl @@ -34,8 +34,10 @@ spec: - host.docker.internal - host.openshell.internal {{- end }} + {{- with .Values.podSecurityContext }} securityContext: - {{- toYaml .Values.podSecurityContext | nindent 4 }} + {{- toYaml . | nindent 4 }} + {{- end }} containers: - name: openshell-gateway securityContext: @@ -102,7 +104,7 @@ spec: mountPath: /etc/openshell-tls/server-external readOnly: true {{- end }} - {{- if or .Values.server.tls.clientCaSecretName (and .Values.pkiInitJob.enabled (not .Values.certManager.enabled)) (and .Values.certManager.enabled .Values.certManager.clientCaFromServerTlsSecret) }} + {{- if eq (include "openshell.gatewayClientCaEnabled" .) "true" }} - name: tls-client-ca mountPath: /etc/openshell-tls/client-ca readOnly: true @@ -172,7 +174,7 @@ spec: secret: secretName: {{ include "openshell.fullname" . }}-server-external-tls {{- end }} - {{- if or .Values.server.tls.clientCaSecretName (and .Values.pkiInitJob.enabled (not .Values.certManager.enabled)) (and .Values.certManager.enabled .Values.certManager.clientCaFromServerTlsSecret) }} + {{- if eq (include "openshell.gatewayClientCaEnabled" .) "true" }} - name: tls-client-ca secret: {{- if or (and .Values.pkiInitJob.enabled (not .Values.certManager.enabled)) (and .Values.certManager.enabled .Values.certManager.clientCaFromServerTlsSecret) }} diff --git a/deploy/helm/openshell/templates/_helpers.tpl b/deploy/helm/openshell/templates/_helpers.tpl index 548418abc..3d9f2f3e0 100644 --- a/deploy/helm/openshell/templates/_helpers.tpl +++ b/deploy/helm/openshell/templates/_helpers.tpl @@ -70,6 +70,19 @@ Create the name of the service account assigned to sandbox pods {{- end }} {{- end }} +{{/* +Whether this chart owns workspace-scoped resources. Missing legacy values +default to enabled so upgrades with --reuse-values preserve the old topology. +*/}} +{{- define "openshell.workspaceResourcesEnabled" -}} +{{- $workspaceResources := .Values.workspaceResources | default dict -}} +{{- $enabled := true -}} +{{- if hasKey $workspaceResources "enabled" -}} +{{- $enabled = get $workspaceResources "enabled" -}} +{{- end -}} +{{- if $enabled -}}true{{- end -}} +{{- end }} + {{/* Gateway image reference. Uses image.tag when set; falls back to .Chart.AppVersion so a released chart automatically pulls the matching image without extra overrides. @@ -83,6 +96,20 @@ so a released chart automatically pulls the matching image without extra overrid ghcr.io/nvidia/openshell/supervisor {{- end }} +{{/* +Whether the gateway listener should verify client certificates (mTLS). +An explicit empty server.tls.clientCaSecretName disables client-CA wiring in +both gateway.toml and the workload, overriding built-in PKI and cert-manager +defaults. +*/}} +{{- define "openshell.gatewayClientCaEnabled" -}} +{{- if .Values.server.disableTls -}} +{{- else if eq .Values.server.tls.clientCaSecretName "" -}} +{{- else if or .Values.server.tls.clientCaSecretName (and .Values.pkiInitJob.enabled (not .Values.certManager.enabled)) (and .Values.certManager.enabled .Values.certManager.clientCaFromServerTlsSecret) -}} +true +{{- end -}} +{{- end -}} + {{/* Whether Helm must propagate a supervisor image override into gateway.toml. The chart's documented repository and empty tag are the gateway-owned default. @@ -261,4 +288,7 @@ Validate chart values that Helm would otherwise accept silently. {{- if gt (len $credentialDrivers) 1 -}} {{- fail "only one external server.credentialDrivers backend can be enabled at a time." -}} {{- end -}} +{{- if kindIs "invalid" .Values.server.tls.clientCaSecretName -}} +{{- fail "server.tls.clientCaSecretName cannot be null; omit the key to use the chart default (openshell-server-client-ca), or set to \"\" to disable client certificate verification for HTTPS-only mode" -}} +{{- end -}} {{- end }} diff --git a/deploy/helm/openshell/templates/gateway-config.yaml b/deploy/helm/openshell/templates/gateway-config.yaml index d7f3cb9a8..083748aee 100644 --- a/deploy/helm/openshell/templates/gateway-config.yaml +++ b/deploy/helm/openshell/templates/gateway-config.yaml @@ -104,7 +104,9 @@ data: [openshell.gateway.tls] cert_path = "/etc/openshell-tls/server/tls.crt" key_path = "/etc/openshell-tls/server/tls.key" + {{- if eq (include "openshell.gatewayClientCaEnabled" .) "true" }} client_ca_path = "/etc/openshell-tls/client-ca/ca.crt" + {{- end }} {{- if .Values.certManager.serverIssuerRef.name }} external_cert_path = "/etc/openshell-tls/server-external/tls.crt" external_key_path = "/etc/openshell-tls/server-external/tls.key" diff --git a/deploy/helm/openshell/templates/networkpolicy.yaml b/deploy/helm/openshell/templates/networkpolicy.yaml index e85571e5f..c9e4a760e 100644 --- a/deploy/helm/openshell/templates/networkpolicy.yaml +++ b/deploy/helm/openshell/templates/networkpolicy.yaml @@ -1,7 +1,7 @@ # SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. # SPDX-License-Identifier: Apache-2.0 -{{- if .Values.networkPolicy.enabled }} +{{- if and (include "openshell.workspaceResourcesEnabled" .) .Values.networkPolicy.enabled }} # NetworkPolicy restricting SSH ingress on sandbox pods to the gateway pod. # Sandbox pods are dynamically created by the server and labelled with # openshell.ai/managed-by=openshell. This policy ensures only the gateway diff --git a/deploy/helm/openshell/templates/role.yaml b/deploy/helm/openshell/templates/role.yaml index 8def13f31..dfd642361 100644 --- a/deploy/helm/openshell/templates/role.yaml +++ b/deploy/helm/openshell/templates/role.yaml @@ -1,5 +1,5 @@ {{- $workspaceMode := .Values.server.drivers.kubernetes.workspaceMode | default "shared" -}} -{{- if eq $workspaceMode "shared" }} +{{- if and (eq $workspaceMode "shared") (include "openshell.workspaceResourcesEnabled" .) }} # SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. # SPDX-License-Identifier: Apache-2.0 apiVersion: rbac.authorization.k8s.io/v1 diff --git a/deploy/helm/openshell/templates/rolebinding.yaml b/deploy/helm/openshell/templates/rolebinding.yaml index 381473a58..32f11644b 100644 --- a/deploy/helm/openshell/templates/rolebinding.yaml +++ b/deploy/helm/openshell/templates/rolebinding.yaml @@ -1,5 +1,5 @@ {{- $workspaceMode := .Values.server.drivers.kubernetes.workspaceMode | default "shared" -}} -{{- if eq $workspaceMode "shared" }} +{{- if and (eq $workspaceMode "shared") (include "openshell.workspaceResourcesEnabled" .) }} # SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. # SPDX-License-Identifier: Apache-2.0 apiVersion: rbac.authorization.k8s.io/v1 diff --git a/deploy/helm/openshell/templates/serviceaccount.yaml b/deploy/helm/openshell/templates/serviceaccount.yaml index a98ad5363..1a9245d4d 100644 --- a/deploy/helm/openshell/templates/serviceaccount.yaml +++ b/deploy/helm/openshell/templates/serviceaccount.yaml @@ -13,10 +13,10 @@ metadata: {{- toYaml . | nindent 4 }} {{- end }} {{- end }} -{{- if and .Values.serviceAccount.create .Values.sandboxServiceAccount.create }} +{{- if and .Values.serviceAccount.create (include "openshell.workspaceResourcesEnabled" .) .Values.sandboxServiceAccount.create }} --- {{- end }} -{{- if .Values.sandboxServiceAccount.create }} +{{- if and (include "openshell.workspaceResourcesEnabled" .) .Values.sandboxServiceAccount.create }} apiVersion: v1 kind: ServiceAccount metadata: diff --git a/deploy/helm/openshell/tests/gateway_config_test.yaml b/deploy/helm/openshell/tests/gateway_config_test.yaml index b54b036c0..1380cb18c 100644 --- a/deploy/helm/openshell/tests/gateway_config_test.yaml +++ b/deploy/helm/openshell/tests/gateway_config_test.yaml @@ -422,6 +422,94 @@ tests: path: data["gateway.toml"] pattern: '\[openshell\.gateway\.tls\]' + - it: renders client_ca_path for built-in PKI by default + template: templates/gateway-config.yaml + asserts: + - matchRegex: + path: data["gateway.toml"] + pattern: 'client_ca_path\s*=\s*"/etc/openshell-tls/client-ca/ca\.crt"' + + - it: omits client_ca_path when clientCaSecretName is empty for HTTPS-only mode + template: templates/gateway-config.yaml + set: + server.tls.clientCaSecretName: "" + asserts: + - matchRegex: + path: data["gateway.toml"] + pattern: '\[openshell\.gateway\.tls\]' + - matchRegex: + path: data["gateway.toml"] + pattern: 'cert_path\s*=\s*"/etc/openshell-tls/server/tls\.crt"' + - matchRegex: + path: data["gateway.toml"] + pattern: 'key_path\s*=\s*"/etc/openshell-tls/server/tls\.key"' + - notMatchRegex: + path: data["gateway.toml"] + pattern: 'client_ca_path\s*=' + + - it: keeps external server certificate fields when clientCaSecretName is empty + template: templates/gateway-config.yaml + set: + certManager.enabled: true + certManager.serverIssuerRef.name: letsencrypt-prod + certManager.serverDnsNames: + - gateway.example.com + server.tls.clientCaSecretName: "" + asserts: + - notMatchRegex: + path: data["gateway.toml"] + pattern: 'client_ca_path\s*=' + - matchRegex: + path: data["gateway.toml"] + pattern: 'external_cert_path\s*=\s*"/etc/openshell-tls/server-external/tls\.crt"' + - matchRegex: + path: data["gateway.toml"] + pattern: 'external_key_path\s*=\s*"/etc/openshell-tls/server-external/tls\.key"' + - matchRegex: + path: data["gateway.toml"] + pattern: 'external_server_names\s*=\s*\["gateway\.example\.com"\]' + + - it: omits client_ca_path for cert-manager shared CA when clientCaSecretName is empty + template: templates/gateway-config.yaml + set: + certManager.enabled: true + certManager.clientCaFromServerTlsSecret: true + server.tls.clientCaSecretName: "" + asserts: + - matchRegex: + path: data["gateway.toml"] + pattern: '\[openshell\.gateway\.tls\]' + - matchRegex: + path: data["gateway.toml"] + pattern: 'cert_path\s*=\s*"/etc/openshell-tls/server/tls\.crt"' + - matchRegex: + path: data["gateway.toml"] + pattern: 'key_path\s*=\s*"/etc/openshell-tls/server/tls\.key"' + - notMatchRegex: + path: data["gateway.toml"] + pattern: 'client_ca_path\s*=' + + - it: omits client_ca_path when cert-manager owns TLS and no client CA secret is set + template: templates/gateway-config.yaml + set: + certManager.enabled: true + certManager.clientCaFromServerTlsSecret: false + pkiInitJob.enabled: true + server.tls.clientCaSecretName: "" + asserts: + - matchRegex: + path: data["gateway.toml"] + pattern: '\[openshell\.gateway\.tls\]' + - matchRegex: + path: data["gateway.toml"] + pattern: 'cert_path\s*=\s*"/etc/openshell-tls/server/tls\.crt"' + - matchRegex: + path: data["gateway.toml"] + pattern: 'key_path\s*=\s*"/etc/openshell-tls/server/tls\.key"' + - notMatchRegex: + path: data["gateway.toml"] + pattern: 'client_ca_path\s*=' + - it: renders server_sans from certManager.serverDnsNames set: certManager.enabled: true @@ -442,6 +530,14 @@ tests: path: spec.template.spec.containers[0].args content: "sqlite:/var/openshell/openshell.db" + - it: fails when clientCaSecretName is null + template: templates/statefulset.yaml + set: + server.tls.clientCaSecretName: null + asserts: + - failedTemplate: + errorMessage: "server.tls.clientCaSecretName cannot be null; omit the key to use the chart default (openshell-server-client-ca), or set to \"\" to disable client certificate verification for HTTPS-only mode" + - it: fails when legacy postgres.enabled is set template: templates/statefulset.yaml set: diff --git a/deploy/helm/openshell/tests/gateway_pod_security_context_test.yaml b/deploy/helm/openshell/tests/gateway_pod_security_context_test.yaml new file mode 100644 index 000000000..cdb70a05e --- /dev/null +++ b/deploy/helm/openshell/tests/gateway_pod_security_context_test.yaml @@ -0,0 +1,36 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +suite: gateway pod securityContext +templates: + - templates/gateway-config.yaml + - templates/statefulset.yaml +release: + name: openshell + namespace: my-namespace + +tests: + - it: renders the pod securityContext from podSecurityContext by default + template: templates/statefulset.yaml + asserts: + - equal: + path: spec.template.spec.securityContext.fsGroup + value: 1000 + + - it: renders an explicitly set podSecurityContext + template: templates/statefulset.yaml + set: + podSecurityContext: + fsGroup: 2000 + asserts: + - equal: + path: spec.template.spec.securityContext.fsGroup + value: 2000 + + - it: omits the pod securityContext block when podSecurityContext is null + template: templates/statefulset.yaml + set: + podSecurityContext: null + asserts: + - notExists: + path: spec.template.spec.securityContext diff --git a/deploy/helm/openshell/tests/sandbox_namespace_test.yaml b/deploy/helm/openshell/tests/sandbox_namespace_test.yaml index ee89fce53..864e3a851 100644 --- a/deploy/helm/openshell/tests/sandbox_namespace_test.yaml +++ b/deploy/helm/openshell/tests/sandbox_namespace_test.yaml @@ -75,3 +75,21 @@ tests: path: metadata.namespace value: other-ns documentIndex: 1 + + - it: omits workspace resources in gateway-only mode + set: + workspaceResources.enabled: false + networkPolicy.enabled: true + asserts: + - hasDocuments: + count: 1 + template: templates/gateway-config.yaml + - hasDocuments: + count: 0 + template: templates/networkpolicy.yaml + - hasDocuments: + count: 0 + template: templates/role.yaml + - hasDocuments: + count: 0 + template: templates/rolebinding.yaml diff --git a/deploy/helm/openshell/tests/sandbox_service_account_test.yaml b/deploy/helm/openshell/tests/sandbox_service_account_test.yaml index c42641582..c9f10868f 100644 --- a/deploy/helm/openshell/tests/sandbox_service_account_test.yaml +++ b/deploy/helm/openshell/tests/sandbox_service_account_test.yaml @@ -29,3 +29,13 @@ tests: asserts: - hasDocuments: count: 1 + + - it: renders only the gateway service account in gateway-only mode + set: + workspaceResources.enabled: false + asserts: + - hasDocuments: + count: 1 + - equal: + path: metadata.name + value: openshell diff --git a/deploy/helm/openshell/tests/statefulset_client_ca_test.yaml b/deploy/helm/openshell/tests/statefulset_client_ca_test.yaml index 1d744b35a..813e47fbe 100644 --- a/deploy/helm/openshell/tests/statefulset_client_ca_test.yaml +++ b/deploy/helm/openshell/tests/statefulset_client_ca_test.yaml @@ -84,3 +84,39 @@ tests: name: tls-client-ca mountPath: /etc/openshell-tls/client-ca readOnly: true + + # Explicit HTTPS-only opt-out must suppress built-in PKI client-CA wiring even + # when pkiInitJob remains enabled (the default install path from #2095). + - it: omits the client CA volume for built-in PKI when clientCaSecretName is empty + template: templates/statefulset.yaml + set: + pkiInitJob.enabled: true + certManager.enabled: false + server.tls.clientCaSecretName: "" + asserts: + - lengthEqual: + path: spec.template.spec.volumes + count: 3 + - notContains: + path: spec.template.spec.containers[0].volumeMounts + content: + name: tls-client-ca + mountPath: /etc/openshell-tls/client-ca + readOnly: true + + - it: omits the client CA volume when cert-manager shares server CA and clientCaSecretName is empty + template: templates/statefulset.yaml + set: + certManager.enabled: true + certManager.clientCaFromServerTlsSecret: true + server.tls.clientCaSecretName: "" + asserts: + - lengthEqual: + path: spec.template.spec.volumes + count: 3 + - notContains: + path: spec.template.spec.containers[0].volumeMounts + content: + name: tls-client-ca + mountPath: /etc/openshell-tls/client-ca + readOnly: true diff --git a/deploy/helm/openshell/values.yaml b/deploy/helm/openshell/values.yaml index 6b0b6242b..453654f79 100644 --- a/deploy/helm/openshell/values.yaml +++ b/deploy/helm/openshell/values.yaml @@ -102,6 +102,14 @@ sandboxServiceAccount: # -- Existing service account name for sandbox pods when sandboxServiceAccount.create is false. name: "" +# Namespace-scoped resources needed to run sandboxes. Disable this when the +# gateway and workspace prerequisites are managed as separate Helm releases +# using the openshell-workspace chart. +workspaceResources: + # -- Create the sandbox ServiceAccount, Role, RoleBinding, and NetworkPolicy + # from this chart. Disable for a gateway-only release. + enabled: true + # -- Extra annotations to add to the gateway pod. podAnnotations: {} # -- Extra labels to add to the gateway pod. @@ -359,6 +367,7 @@ server: certSecretName: openshell-server-tls # -- K8s secret with ca.crt for client certificate verification (mTLS). # Set to "" to disable mTLS and run HTTPS-only (use OIDC for auth instead). + # Do not set to null; omit the key to use the default secret name above. clientCaSecretName: openshell-server-client-ca # -- K8s secret mounted into sandbox pods for mTLS to the server. clientTlsSecretName: openshell-client-tls @@ -404,7 +413,7 @@ server: # -- Expected audience claim for the API resource server. # This should match the server's --oidc-audience, NOT the CLI client ID. audience: "openshell-cli" - # -- JWKS key cache TTL in seconds. + # -- JWKS key cache TTL in seconds. Must be greater than zero. jwksTtl: 3600 # -- Dot-separated path to the roles array in the JWT claims. # Keycloak: "realm_access.roles", Entra ID: "roles", Okta: "groups". diff --git a/deploy/helm/test-split-ownership.sh b/deploy/helm/test-split-ownership.sh new file mode 100755 index 000000000..30fd7360d --- /dev/null +++ b/deploy/helm/test-split-ownership.sh @@ -0,0 +1,66 @@ +#!/usr/bin/env bash +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +set -euo pipefail + +repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd -P)" +work_dir="$(mktemp -d)" +trap 'rm -rf "${work_dir}"' EXIT + +helm template openshell "${repo_root}/deploy/helm/openshell" \ + --namespace openshell \ + --set agentSandbox.preflight.enabled=false \ + --set workspaceResources.enabled=false \ + >"${work_dir}/gateway.yaml" + +if yq ea -e \ + 'select(.kind == "Role" and .metadata.name == "openshell-sandbox")' \ + "${work_dir}/gateway.yaml" >/dev/null 2>&1; then + echo "gateway chart rendered workspace resources despite workspaceResources.enabled=false" >&2 + exit 1 +fi + +helm template openshell-workspace "${repo_root}/deploy/helm/openshell-workspace" \ + --namespace app-a \ + --set gateway.serviceAccount.name=openshell \ + --set gateway.serviceAccount.namespace=openshell \ + >"${work_dir}/workspace.yaml" + +invalid_workspace_docs="$( + yq ea -N -r \ + 'select(. != null and (.apiVersion == null or .kind == null)) | document_index' \ + "${work_dir}/workspace.yaml" +)" +if [[ -n "${invalid_workspace_docs}" ]]; then + echo "workspace chart rendered documents without apiVersion or kind: ${invalid_workspace_docs}" >&2 + exit 1 +fi + +helm template openshell "${repo_root}/deploy/helm/openshell" \ + --namespace openshell \ + --set agentSandbox.preflight.enabled=false \ + --set-json workspaceResources=null \ + >"${work_dir}/legacy-reuse-values.yaml" + +yq ea -e \ + 'select(.kind == "Role" and .metadata.name == "openshell-sandbox") | + .apiVersion == "rbac.authorization.k8s.io/v1"' \ + "${work_dir}/legacy-reuse-values.yaml" >/dev/null + +yq ea -N -r \ + 'select(.kind != null) | [.apiVersion, .kind, (.metadata.namespace // "openshell"), .metadata.name] | @tsv' \ + "${work_dir}/gateway.yaml" | sort -u >"${work_dir}/gateway.objects" +yq ea -N -r \ + 'select(.kind != null) | [.apiVersion, .kind, (.metadata.namespace // "app-a"), .metadata.name] | @tsv' \ + "${work_dir}/workspace.yaml" | sort -u >"${work_dir}/workspace.objects" + +comm -12 "${work_dir}/gateway.objects" "${work_dir}/workspace.objects" \ + >"${work_dir}/overlap.objects" +if [[ -s "${work_dir}/overlap.objects" ]]; then + echo "gateway and workspace charts claim the same Kubernetes objects:" >&2 + cat "${work_dir}/overlap.objects" >&2 + exit 1 +fi + +echo "gateway and workspace chart object ownership is disjoint" diff --git a/deploy/man/openshell-gateway.8.md b/deploy/man/openshell-gateway.8.md index 2d584c4ba..ee26a4477 100644 --- a/deploy/man/openshell-gateway.8.md +++ b/deploy/man/openshell-gateway.8.md @@ -18,7 +18,7 @@ openshell-gateway - OpenShell gateway server daemon **openshell-gateway** is the control-plane server for OpenShell. It manages sandbox lifecycle, stores provider credentials, delivers -network and filesystem policies to sandboxes, routes inference +network and filesystem policies to sandboxes, manages provider access requests, and provides the SSH tunnel endpoint for CLI-to-sandbox connections. diff --git a/deploy/man/openshell.1.md b/deploy/man/openshell.1.md index 849d299cb..7dabf558a 100644 --- a/deploy/man/openshell.1.md +++ b/deploy/man/openshell.1.md @@ -19,7 +19,7 @@ openshell - CLI for managing OpenShell sandboxes, gateway registrations, and pro **openshell** is the command-line interface for OpenShell, a platform providing safe, sandboxed runtimes for autonomous AI agents. It manages gateway registrations, sandbox lifecycle, credential providers, -network policies, and inference routing. +network policies, and provider attachments. The CLI communicates with a gateway server over gRPC. The gateway can run as a package-managed systemd user service, a Helm deployment, a @@ -114,16 +114,12 @@ development task, or behind a cloud reverse proxy. **provider delete** *NAME* : Delete a provider. -## Inference Routing +**sandbox provider attach** *SANDBOX* *PROVIDER* +: Attach a provider to a running sandbox. New processes receive its + endpoint-bound credential placeholders and provider-derived policy. -**inference set** **--provider** *NAME* **--model** *MODEL* -: Configure inference routing. - -**inference get** -: Show current inference configuration. - -**inference update** \[**--model** *MODEL*\] -: Update inference configuration. +**sandbox provider detach** *SANDBOX* *PROVIDER* +: Revoke a provider attachment from a running sandbox. ## Other diff --git a/deploy/rpm/QUICKSTART.md b/deploy/rpm/QUICKSTART.md index 442458d09..f37f61135 100644 --- a/deploy/rpm/QUICKSTART.md +++ b/deploy/rpm/QUICKSTART.md @@ -144,13 +144,15 @@ openshell provider create --name openai --type openai \ openshell provider list ``` -## Configure inference routing (optional) +## Configure provider-backed inference (optional) -To route inference requests through a specific provider and model: +To grant a sandbox access to a model provider: ```shell -openshell inference set --provider openai --model gpt-4 -openshell inference get +openshell sandbox provider attach openai + +# Configure the workload to call the provider's native endpoint and select +# the model in the client. ``` ## Next steps diff --git a/deploy/rpm/TROUBLESHOOTING.md b/deploy/rpm/TROUBLESHOOTING.md index 103ce3bf9..950052084 100644 --- a/deploy/rpm/TROUBLESHOOTING.md +++ b/deploy/rpm/TROUBLESHOOTING.md @@ -13,7 +13,7 @@ and what to use instead. ### Commands that work normally -All sandbox, provider, policy, inference, and settings commands +All sandbox, provider, policy, and settings commands communicate with the gateway over gRPC and work identically regardless of deployment mode: @@ -23,7 +23,8 @@ openshell sandbox create|list|get|delete|connect|exec openshell logs openshell provider create|list|get|update|delete openshell policy get|set|update|list|prove -openshell inference set|get|update +openshell provider list +openshell sandbox provider list openshell settings get|set openshell forward start|stop|list openshell term @@ -47,13 +48,11 @@ systemd commands directly: ### Building from local Dockerfiles -`openshell sandbox create --from ./Dockerfile` builds via the local -Docker daemon. With the RPM Podman driver, build the image with Podman -and reference it directly: +Build the image with Podman, then reference it directly: ```shell -podman build -t my-sandbox ./my-dir -openshell sandbox create --from localhost/my-sandbox +podman build -t localhost/my-sandbox:latest ./my-dir +openshell sandbox create --from localhost/my-sandbox:latest ``` ## Remote CLI access diff --git a/docs/CONTRIBUTING.mdx b/docs/CONTRIBUTING.mdx index 9a69c4b44..b7a47ce98 100644 --- a/docs/CONTRIBUTING.mdx +++ b/docs/CONTRIBUTING.mdx @@ -14,10 +14,10 @@ If you use an AI coding agent (Cursor, Claude Code, Codex, etc.), the repo inclu | Skill | What it does | When to use | |---|---|---| -| `update-docs` | Scans recent commits for user-facing changes and drafts doc updates. | After landing features, before a release, or to find doc gaps. | +| `update-docs-from-commits` | Scans recent commits for user-facing changes and drafts doc updates. | After landing features, before a release, or to find doc gaps. | | `build-from-issue` | Plans and implements work from a GitHub issue, including doc updates. | When working from an issue that has doc impact. | -The skills live in `.agents/skills/` and follow the style guide below automatically. To use one, ask your agent to run it (e.g., "catch up the docs for everything merged since v0.2.0"). +These contributor workflows live in `.agents/skills/` and follow the style guide below automatically. They are separate from the public skills in `skills/`, which help users operate OpenShell and can be installed without cloning the repository. To use a contributor skill, ask your repository-aware agent to run it (e.g., "catch up the docs for everything merged since v0.2.0"). ## When to Update Docs diff --git a/docs/about/how-it-works.mdx b/docs/about/how-it-works.mdx index 5223072a2..9878fceb6 100644 --- a/docs/about/how-it-works.mdx +++ b/docs/about/how-it-works.mdx @@ -4,14 +4,14 @@ title: "How OpenShell Works" sidebar-title: "How It Works" description: "Understand the OpenShell architecture, runtime boundaries, gateways, sandboxes, and ecosystem integration points." -keywords: "Generative AI, Cybersecurity, AI Agents, Architecture, Gateway, Sandbox, Inference Routing" +keywords: "Generative AI, Cybersecurity, AI Agents, Architecture, Gateway, Sandbox, Providers" position: 2 --- OpenShell is built around three stable runtime components: the **CLI**, the **Gateway**, and the **Supervisor**. The CLI, SDK, and TUI provide user-facing access. The gateway is the -control plane: it owns API access, state, policy and settings delivery, provider and inference configuration, and relay coordination. The supervisor runs inside every sandbox workload and is the local security boundary. It launches the agent as a restricted child process and enforces policy where process identity, filesystem access, network egress, and +control plane: it owns API access, state, policy and settings delivery, provider configuration, and relay coordination. The supervisor runs inside every sandbox workload and is the local security boundary. It launches the agent as a restricted child process and enforces policy where process identity, filesystem access, network egress, and runtime credentials are visible. Infrastructure-specific work sits behind integration boundaries. Compute, @@ -46,7 +46,6 @@ flowchart TB AGENT["Restricted agent process"] PROXY["Policy proxy"] POLICY["OPA policy engine"] - ROUTER["Inference router"] end CLI -->|"gRPC / HTTP"| GW @@ -64,8 +63,7 @@ flowchart TB AGENT -->|"ordinary egress"| PROXY PROXY -->|"evaluate"| POLICY PROXY -->|"allowed traffic"| EXT["External services"] - PROXY -->|"inference.local"| ROUTER - ROUTER -->|"managed inference"| MODEL["Inference backends"] + PROXY -->|"profile-authorized traffic"| MODEL["Model providers"] ``` ## Deployment Models @@ -92,11 +90,11 @@ device plugins without changing the gateway and sandbox contract. | [Gateways](/sandboxes/manage-gateways) | Authenticated control plane that owns API access, durable state, sandbox lifecycle, settings delivery, authorization, and relay coordination. | | [Providers](/sandboxes/manage-providers) | Credential and provider records that map logical agent needs to platform or user-managed secrets without exposing raw credentials to the agent process. | | [Policies](/sandboxes/policies) | Declarative controls for filesystem access, process identity, network egress, L7 rules, credential injection, and runtime policy updates. | -| [Inference Routing](/sandboxes/inference-routing) | Managed `https://inference.local` path that routes model traffic to configured backends while keeping provider credentials outside the sandbox. | +| [Provider-backed Inference](/sandboxes/inference-routing) | Per-sandbox provider attachment, native model endpoints, and endpoint-bound credential injection. | ## Gateways and Sandboxes -The gateway and sandbox split control-plane authority from runtime enforcement. The gateway owns durable platform state: sandboxes, policy revisions, runtime settings, provider records, inference configuration, session records, and authorization decisions. A sandbox owns the local execution boundary: process identity, filesystem access, network egress, credential injection, local logs, and the agent child process. +The gateway and sandbox split control-plane authority from runtime enforcement. The gateway owns durable platform state: sandboxes, policy revisions, runtime settings, provider records, session records, and authorization decisions. A sandbox owns the local execution boundary: process identity, filesystem access, network egress, credential injection, local logs, and the agent child process. The relationship is supervisor initiated. Each sandbox supervisor connects outbound to a known gateway endpoint, authenticates as a sandbox workload, and keeps a live session open for control traffic and relays. This avoids requiring every compute driver to solve gateway-to-sandbox reachability through pod IPs, bridge networks, port mappings, NAT traversal, or custom tunnels. @@ -114,18 +112,18 @@ then launches the agent under the active policy. | Filesystem | Applies filesystem policy before the agent starts so undeclared paths are inaccessible and declared paths are read-only or read-write as configured. | | Network | Routes ordinary egress through the policy proxy so destination, port, binary identity, and L7 request rules can be evaluated before traffic leaves the sandbox. | | Credentials | Receives credential material from the gateway and injects it only through configured policy paths or request-time proxy rules. | -| Inference | Intercepts `https://inference.local` and forwards model traffic through the configured inference route instead of exposing provider credentials to the agent. | +| Provider access | Enforces profile-derived policy and substitutes provider credential placeholders only at profile-authorized endpoints. | | Observability | Emits local security and lifecycle logs, pushes sandbox logs to the gateway, and keeps relay endpoints available for connect, exec, and file transfer operations. | Static controls such as filesystem and process isolation are established at sandbox start and require sandbox recreation to change. Dynamic controls such as -network policy, credential delivery, and inference routing can refresh over the +network policy and credential delivery can refresh over the live gateway-supervisor session. ## Ecosystem Integration OpenShell integrates with infrastructure ecosystems instead of replacing them. Runtimes, schedulers, secret stores, identity providers, workload identity systems, image pipelines, storage, and GPU or device exposure remain owned by the platforms that provide them. -The gateway owns OpenShell control-plane semantics: sandbox state, lifecycle ordering, policy and settings resolution, credential mapping, authorization, inference configuration, and relay coordination. Drivers translate those semantics into platform-native operations. +The gateway owns OpenShell control-plane semantics: sandbox state, lifecycle ordering, policy and settings resolution, credential mapping, authorization, and relay coordination. Drivers translate those semantics into platform-native operations. -The supervisor owns OpenShell sandbox semantics. Filesystem policy, process privilege reduction, network proxying, inference interception, credential injection, security logging, and gateway relay behavior stay consistent across Docker, Podman, Kubernetes, VM-backed sandboxes, and future integrations. +The supervisor owns OpenShell sandbox semantics. Filesystem policy, process privilege reduction, network proxying, provider credential injection, security logging, and gateway relay behavior stay consistent across Docker, Podman, Kubernetes, VM-backed sandboxes, and future integrations. diff --git a/docs/about/overview.mdx b/docs/about/overview.mdx index 6ef41b34a..90b565ae3 100644 --- a/docs/about/overview.mdx +++ b/docs/about/overview.mdx @@ -4,7 +4,7 @@ title: "Overview of NVIDIA OpenShell" sidebar-title: "Overview" description: "OpenShell is the safe, private runtime for autonomous AI agents. Run agents in sandboxed environments that protect your data, credentials, and infrastructure." -keywords: "Generative AI, Cybersecurity, AI Agents, Sandboxing, Security, Privacy, Inference Routing" +keywords: "Generative AI, Cybersecurity, AI Agents, Sandboxing, Security, Privacy, Provider Credentials" position: 1 --- @@ -22,7 +22,7 @@ The table below summarizes common failure modes and how OpenShell mitigates them |---|---|---| | Data exfiltration | Agent uploads source code or internal files to unauthorized endpoints. | Network policies allow only approved destinations; other outbound traffic is denied. | | Credential theft | Agent reads local secrets such as SSH keys or cloud credentials. | Filesystem restrictions (Landlock) confine access to declared paths only. | -| Unauthorized API usage | Agent sends prompts or data to unapproved model providers. | Privacy routing and network policies control where inference traffic can go. | +| Unauthorized API usage | Agent sends prompts or data to unapproved model providers. | Provider profiles and network policies restrict model traffic to approved endpoints and binaries. | | Privilege escalation | Agent attempts `sudo`, setuid paths, or dangerous syscall behavior. | Unprivileged process identity and seccomp restrictions block escalation paths. | ## Protection Layers at a Glance @@ -34,7 +34,7 @@ OpenShell applies defense in depth across the following policy domains. | Filesystem | Prevents reads/writes outside allowed paths. | Locked at sandbox creation. | | Network | Blocks unauthorized outbound connections. | Hot-reloadable at runtime. | | Process | Blocks privilege escalation and dangerous syscalls. | Locked at sandbox creation. | -| Inference | Reroutes model API calls to controlled backends. | Hot-reloadable at runtime. | +| Provider credentials | Resolves opaque credential placeholders only at profile-authorized endpoints. | Attachments, rotation, and revocation update at runtime; new environment variables require a new process. | For details, refer to [Customize Sandbox Policies](/sandboxes/policies) and [Default Policy](/reference/default-policy). @@ -45,7 +45,7 @@ OpenShell supports a range of agent deployment patterns. | Use Case | Description | |-----------------------------|----------------------------------------------------------------------------------------------------------| | Secure coding agents | Run Claude Code, OpenCode, Codex, or GitHub Copilot CLI with constrained file and network access. | -| Private enterprise development | Route inference to self-hosted or private backends while keeping sensitive context under your control. | +| Private enterprise development | Grant selected sandboxes access to self-hosted or private model endpoints while keeping sensitive context under your control. | | Compliance and audit | Treat policy YAML as version-controlled security controls that can be reviewed and audited. | | Reusable environments | Use community sandbox images or bring your own containerized runtime. | diff --git a/docs/about/supported-agents.mdx b/docs/about/supported-agents.mdx index 9d38541dd..13704aa94 100644 --- a/docs/about/supported-agents.mdx +++ b/docs/about/supported-agents.mdx @@ -10,12 +10,12 @@ The following table summarizes the agents that run in OpenShell sandboxes. Most | Agent | Source | Default Policy | Notes | |---|---|---|---| -| [Claude Code](https://docs.anthropic.com/en/docs/claude-code) | [`base`](https://github.com/NVIDIA/OpenShell-Community/tree/main/sandboxes/base) | Full coverage | Works out of the box. Requires `ANTHROPIC_API_KEY` for direct Anthropic access, or use `inference.local` with a configured provider (e.g. Vertex AI). | -| [OpenCode](https://opencode.ai/) | [`base`](https://github.com/NVIDIA/OpenShell-Community/tree/main/sandboxes/base) | Partial coverage | Pre-installed. Use `ANTHROPIC_BASE_URL="https://inference.local/v1"` with a configured provider. Add `opencode.ai` endpoint and OpenCode binary paths to the policy for full functionality. | +| [Claude Code](https://docs.anthropic.com/en/docs/claude-code) | [`base`](https://github.com/NVIDIA/OpenShell-Community/tree/main/sandboxes/base) | Full coverage | Works out of the box with an attached `claude-code` provider. For another model platform, attach an endpoint-bearing profile and configure Claude Code for that platform's native API. | +| [OpenCode](https://opencode.ai/) | [`base`](https://github.com/NVIDIA/OpenShell-Community/tree/main/sandboxes/base) | Partial coverage | Pre-installed. Attach the selected model provider, configure its native base URL and model, and ensure the imported profile names OpenCode's service endpoint and binary paths. | | [Codex](https://developers.openai.com/codex) | [`base`](https://github.com/NVIDIA/OpenShell-Community/tree/main/sandboxes/base) | No coverage | Pre-installed. Requires a custom policy with OpenAI endpoints and Codex binary paths. Requires `OPENAI_API_KEY`. | | [GitHub Copilot CLI](https://docs.github.com/en/copilot/github-copilot-in-the-cli) | [`base`](https://github.com/NVIDIA/OpenShell-Community/tree/main/sandboxes/base) | Full coverage | Pre-installed. Works out of the box. Requires `GITHUB_TOKEN` or `COPILOT_GITHUB_TOKEN`. | -| [OpenClaw](https://openclaw.ai/) | [NemoClaw](https://github.com/NVIDIA/NemoClaw) | Blueprint-managed | Run OpenClaw more securely inside NVIDIA OpenShell with managed inference using NemoClaw. | -| [Hermes Agent](https://github.com/NousResearch/hermes-agent) | [NemoClaw](https://github.com/NVIDIA/NemoClaw) | Blueprint-managed | Run Hermes Agent more securely inside NVIDIA OpenShell with managed inference using NemoClaw. | +| [OpenClaw](https://openclaw.ai/) | [NemoClaw](https://github.com/NVIDIA/NemoClaw) | Blueprint-managed | Run OpenClaw more securely inside NVIDIA OpenShell with the NemoClaw blueprint. | +| [Hermes Agent](https://github.com/NousResearch/hermes-agent) | [NemoClaw](https://github.com/NVIDIA/NemoClaw) | Blueprint-managed | Run Hermes Agent more securely inside NVIDIA OpenShell with the NemoClaw blueprint. | | [Ollama](https://ollama.com/) | [`ollama`](https://github.com/NVIDIA/OpenShell-Community/tree/main/sandboxes/ollama) | Bundled | Run cloud and local models. Includes Claude Code, Codex, and OpenCode. Launch with `openshell sandbox create --from ollama`. | | [Pi](https://pi.dev/) | [`pi`](https://github.com/NVIDIA/OpenShell-Community/tree/main/sandboxes/pi) | Bundled | Comes with Pi pre-installed. Launch with `openshell sandbox create --from pi`. | diff --git a/docs/extensibility/supervisor-middleware.mdx b/docs/extensibility/supervisor-middleware.mdx index f568a52d9..48b13dfa0 100644 --- a/docs/extensibility/supervisor-middleware.mdx +++ b/docs/extensibility/supervisor-middleware.mdx @@ -27,7 +27,7 @@ For an RFC 6455 upgrade over `ws://` or `wss://`, the supervisor first finds eve 1. A preflight before the upgrade is sent upstream. The stage chooses `INSPECT`, voluntary `SKIP`, or authoritative `DENY` and may return a bounded diagnostic reason, stable reason code, findings, and metadata. OpenShell runs selected preflights concurrently; any `DENY` rejects the upgrade regardless of `on_error`. 2. A session-start event after the upstream accepts the upgrade, including the negotiated subprotocol. 3. Complete client-to-upstream text messages in sequence order. OpenShell reassembles fragmented messages and decompresses negotiated `permessage-deflate` messages before evaluation. -4. A best-effort session-end event when the stage stream remains writable. OpenShell attempts at most one terminal event for each opened stream, including streams opened during a preflight that rejects the upgrade before session start. +4. A best-effort session-end event when the stage stream remains writable. OpenShell attempts at most one terminal event for each opened stream, including streams opened during a preflight that rejects the upgrade before session start. It then half-closes the request stream and briefly drains the response stream so the terminal event can leave the local transport before the RPC closes. Middleware services should finish their response stream after the request stream reaches EOF. The protobuf represents each logical message with a `text` or `binary` payload variant. Text uses the protobuf `string` type, so invalid UTF-8 cannot enter the middleware contract. Results use an optional matching replacement variant: absence preserves the input, while presence represents a replacement even when its content is empty. OpenShell rejects attempts to change the message type. Allowed replacements are re-framed, re-compressed when required, and forwarded. Binary messages, control frames, and upstream-to-client traffic remain uninspected. Binary messages pass through under both `on_error` modes. For each active selected stage, OpenShell emits an informational `unsupported_message_type` coverage event and advances the session-global sequence; the next text message can therefore reach the stage with a valid sequence gap. @@ -191,7 +191,7 @@ A middleware result can return ordered header mutations before OpenShell injects A `remove` mutation removes every value for a case-insensitive header name. OpenShell applies each successful stage's mutations before invoking the next middleware, so later stages observe the accumulated header state. -Header writes must use the `x-openshell-middleware-` prefix. Removes may target other middleware-visible request headers. Protected credential, routing, framing, and hop-by-hop headers are always rejected. Header values must not contain control characters. +Writes and removals may target middleware-visible end-to-end request headers. Protected credential, routing, framing, and hop-by-hop headers are always rejected. Header values must not contain control characters or OpenShell credential placeholder syntax. Middleware runs before credential injection, but it cannot introduce a value that the later injection step would resolve. OpenShell validates and applies each stage's mutations atomically. An invalid operation discards every mutation from that stage and follows its `on_error` behavior. Built-in failures can name the offending header. Operator-run failures use a platform-owned error code so request-derived header text cannot reach logs or denied responses. diff --git a/docs/get-started/quickstart.mdx b/docs/get-started/quickstart.mdx index 8a8b82046..4d7a3e2de 100644 --- a/docs/get-started/quickstart.mdx +++ b/docs/get-started/quickstart.mdx @@ -33,7 +33,7 @@ The install script uses Homebrew, RPM, or a Debian package based on your machine After installing the CLI, run `openshell --help` in your terminal to view the full CLI reference. -You can also clone the [NVIDIA OpenShell GitHub repository](https://github.com/NVIDIA/OpenShell) and use the `/openshell-cli` skill to load the CLI reference into your agent. +Install the public OpenShell agent skills with `npx skills add NVIDIA/OpenShell`. The `openshell-cli` skill guides your agent through common workflows and uses the installed CLI help as the command reference; no OpenShell source checkout is required. ## Create Your First OpenShell Sandbox diff --git a/docs/get-started/tutorials/docker-compose.mdx b/docs/get-started/tutorials/docker-compose.mdx index 677f34595..fd8c13f05 100644 --- a/docs/get-started/tutorials/docker-compose.mdx +++ b/docs/get-started/tutorials/docker-compose.mdx @@ -137,9 +137,9 @@ docker pull ghcr.io/nvidia/openshell-community/sandboxes/base:latest -OpenClaw runs inside OpenShell through [NemoClaw](https://github.com/NVIDIA/NemoClaw), which manages the sandbox image, inference routing, and security policies. +OpenClaw runs inside OpenShell through [NemoClaw](https://github.com/NVIDIA/NemoClaw), which manages the sandbox image, model-provider setup, and security policies. -Follow the [NemoClaw Quickstart](https://docs.nvidia.com/nemoclaw/latest/get-started/quickstart/) to set up an OpenClaw sandbox with managed inference. +Follow the [NemoClaw Quickstart](https://docs.nvidia.com/nemoclaw/latest/get-started/quickstart/) to set up an OpenClaw sandbox. diff --git a/docs/get-started/tutorials/first-network-policy.mdx b/docs/get-started/tutorials/first-network-policy.mdx index 05888278b..cbf9eaaf9 100644 --- a/docs/get-started/tutorials/first-network-policy.mdx +++ b/docs/get-started/tutorials/first-network-policy.mdx @@ -65,15 +65,16 @@ The request fails. By default, all outbound network traffic is denied. The sandb curl: (56) Received HTTP code 403 from proxy after CONNECT ``` -Exit the sandbox. Sandboxes are kept running by default, so you can reconnect later. Use `--no-keep` at creation time if you want the sandbox deleted after exit: +Leave this sandbox shell open and switch to a second terminal on your host for the next steps. The remaining `openshell` commands (checking logs and applying the policy) run on your host, not inside the sandbox. -```shell -exit -``` + +Keep the interactive sandbox shell open for the rest of the tutorial. Exiting it stops the sandbox's main process, and with the default restart policy the sandbox is not reconnectable afterward. Add `--no-keep` at creation time if you want the sandbox deleted automatically when you exit. + + ## Check the Deny Log -Every denied connection produces a structured log entry. Query the sandbox logs from your host to confirm the denial and inspect the reason. +Every denied connection produces a structured log entry. In your second (host) terminal, query the sandbox logs to confirm the denial and inspect the reason. ```shell openshell logs demo --since 5m @@ -135,13 +136,7 @@ This tutorial uses `curl` and `read-only` access to keep things simple. When bui ## Verify If GET Requests Are Allowed -The policy is now active. Reconnect to the sandbox and retry the same request to confirm that read access works. - -```shell -openshell sandbox connect demo -``` - -Retry the same request: +The policy is now active. Return to your sandbox shell in the first terminal (it is still running) and retry the same request to confirm that read access works: ```shell curl -s https://api.github.com/zen @@ -169,15 +164,9 @@ curl -s -X POST https://api.github.com/repos/octocat/hello-world/issues \ The CONNECT request succeeded because `api.github.com` is allowed, but the L7 proxy inspected the HTTP method and returned `403`. `POST` is not in the `read-only` preset. An agent with this policy can read code from GitHub but cannot create issues, push commits, or modify anything. -Exit the sandbox: - -```shell -exit -``` - ## Check the L7 Deny Log -L7 denials are logged separately from connection-level denials. The log entry includes the exact HTTP method and path that the proxy rejected. +L7 denials are logged separately from connection-level denials. The log entry includes the exact HTTP method and path that the proxy rejected. Run this from your second (host) terminal, leaving the sandbox shell open: ```shell openshell logs demo --level warn --since 5m diff --git a/docs/get-started/tutorials/index.mdx b/docs/get-started/tutorials/index.mdx index bc30bcfc4..2050480b9 100644 --- a/docs/get-started/tutorials/index.mdx +++ b/docs/get-started/tutorials/index.mdx @@ -24,17 +24,17 @@ Launch Claude Code in a sandbox, diagnose a policy denial, and iterate on a cust -Configure a Providers v2 Microsoft Graph provider with gateway-managed OAuth2 refresh-token rotation. +Configure a Microsoft Graph provider profile with gateway-managed OAuth2 refresh-token rotation. -Route inference through Ollama using cloud-hosted or local models, and verify it from a sandbox. +Run Ollama models in a community sandbox or attach a profile for a host-level service. -Route inference to a local LM Studio server using the OpenAI-compatible or Anthropic-compatible APIs. +Attach an endpoint-bearing profile and call a local LM Studio server directly. diff --git a/docs/get-started/tutorials/inference-ollama.mdx b/docs/get-started/tutorials/inference-ollama.mdx index 4f46b847e..c23e6eaf6 100644 --- a/docs/get-started/tutorials/inference-ollama.mdx +++ b/docs/get-started/tutorials/inference-ollama.mdx @@ -4,54 +4,35 @@ title: "Run Local Inference with Ollama" sidebar-title: "Inference with Ollama" slug: "get-started/tutorials/inference-ollama" -description: "Run local and cloud models inside an OpenShell sandbox using the Ollama community sandbox, or route sandbox requests to a host-level Ollama server." -keywords: "Generative AI, Cybersecurity, Tutorial, Inference Routing, Ollama, Local Inference, Sandbox" +description: "Run Ollama inside a community sandbox or attach a profile for a host-level Ollama server." +keywords: "Generative AI, Cybersecurity, Tutorial, Ollama, Local Inference, Sandbox, Provider Profile" --- -This tutorial covers two ways of running Ollama with OpenShell: +This tutorial covers two Ollama deployments: -1. Ollama sandbox. This is the recommended way to run Ollama. A self-contained sandbox with Ollama, Claude Code, and Codex pre-installed. One command starts it. -2. Host-level Ollama. This is an alternative way to run Ollama. Run Ollama on the gateway host and route sandbox inference to it. Use this option when you want a single Ollama instance shared across multiple sandboxes. - -After completing this tutorial, you know how to: - -- Launch the Ollama community sandbox for a batteries-included experience. -- Use `ollama launch` to start coding agents inside a sandbox. -- Expose a host-level Ollama server to sandboxes through `inference.local`. +1. The self-contained Ollama community sandbox. This is the recommended path. +2. A host-level Ollama service shared by explicitly authorized sandboxes. ## Prerequisites -- A working OpenShell installation. Complete the [Quickstart](/get-started/quickstart) before proceeding. +Complete the [Quickstart](/get-started/quickstart) before proceeding. -## Option A: Ollama Community Sandbox (Recommended) +## Option A: Ollama Community Sandbox -The Ollama community sandbox bundles Ollama, Claude Code, OpenCode, and Codex into a single image. Ollama starts automatically when the sandbox launches. - - - -### Create the Sandbox +The community image bundles Ollama and supported coding agents: ```shell openshell sandbox create --from ollama ``` -This pulls the community sandbox image, applies the bundled policy, and drops you into a shell with Ollama running. - -### Chat with a Model - -Chat with a local model +Inside the sandbox, run a local or cloud model: ```shell ollama run qwen3.5 -``` - -Or a cloud model - -```shell ollama run kimi-k2.5:cloud ``` -Or use `ollama launch` to start a coding agent with Ollama as the model backend: +Or launch a coding agent against Ollama: ```shell ollama launch claude @@ -59,153 +40,107 @@ ollama launch codex ollama launch opencode ``` -For CI/CD and automated workflows, `ollama launch` supports a headless mode: +For an automated workflow: ```shell ollama launch claude --yes --model qwen3.5 ``` - +Useful starter models include `qwen3.5:0.8b` for smoke tests and `qwen3.5` +for coding and tool use. Check the +[Ollama model library](https://ollama.com/library) for current model details. -### Model Recommendations - -| Use case | Model | Notes | -|---|---|---| -| Smoke test | `qwen3.5:0.8b` | Fast, lightweight, good for verifying setup | -| Coding and reasoning | `qwen3.5` | Strong tool calling support for agentic workflows | -| Complex tasks | `nemotron-3-super` | 122B parameter model, needs 48GB+ VRAM | -| No local GPU | `qwen3.5:cloud` | Runs on Ollama's cloud infrastructure, no `ollama pull` required | - - -Cloud models use the `:cloud` tag suffix and do not require local hardware. - -```shell -openshell sandbox create --from ollama -``` - - - -### Tool Calling - -Agentic workflows (Claude Code, Codex, OpenCode) rely on tool calling. The following models have reliable tool calling support: Qwen 3.5, Nemotron-3-Super, GLM-5, and Kimi-K2.5. Check the [Ollama model library](https://ollama.com/library) for the latest models. - -### Updating Ollama - -To update Ollama inside a running sandbox: - -```shell -update-ollama -``` - -Or auto-update on every sandbox start: +Update the bundled Ollama installation with `update-ollama`, or request an +update at sandbox start: ```shell openshell sandbox create --from ollama -e OLLAMA_UPDATE=1 ``` -## Option B: Host-Level Ollama +## Option B: Host-level Ollama -Use this approach when you want a single Ollama instance on the gateway host, shared across multiple sandboxes through `inference.local`. +Use this path when Ollama runs on the same machine as the OpenShell gateway and +selected sandboxes should share it. - -This approach uses Ollama because it is easy to install and run locally, but you can substitute other inference engines such as vLLM, SGLang, TRT-LLM, and NVIDIA NIM by changing the startup command, base URL, and model name. - - - - - -### Install and Start Ollama - -Install [Ollama](https://ollama.com/) on the gateway host: +### Start Ollama on a Reachable Address ```shell curl -fsSL https://ollama.com/install.sh | sh -``` - -Start Ollama on all interfaces so it is reachable from sandboxes: - -```shell OLLAMA_HOST=0.0.0.0:11434 ollama serve ``` - -If you see `Error: listen tcp 0.0.0.0:11434: bind: address already in use`, Ollama is already running as a system service. Stop it first: +In another terminal, pull a model: ```shell -systemctl stop ollama -OLLAMA_HOST=0.0.0.0:11434 ollama serve +ollama pull qwen3.5:0.8b ``` - +### Import an Endpoint-bearing Profile -### Pull a Model +Save this as `ollama-openai.yaml`: -In a second terminal, pull a model: +```yaml +id: ollama-openai +display_name: Ollama +description: Host-level Ollama OpenAI-compatible API +category: inference +inference_capable: true +credentials: [] +endpoints: + - host: host.openshell.internal + port: 11434 + protocol: rest + access: read-write + enforcement: enforce +binaries: + - /usr/bin/curl + - /usr/local/bin/curl + - /usr/bin/python3 + - /usr/local/bin/python + - /sandbox/.uv/python/** + - /sandbox/.venv/** +``` ```shell -ollama run qwen3.5:0.8b +openshell provider profile lint -f ollama-openai.yaml +openshell provider profile import -f ollama-openai.yaml +openshell provider create --name ollama --type ollama-openai ``` -Type `/bye` to exit the interactive session. The model stays loaded. +The provider has no secret because this Ollama server does not authenticate. +Its attachment still carries the endpoint and binary policy. -### Create a Provider - -Create an OpenAI-compatible provider pointing at the host Ollama: +### Attach and Verify ```shell -openshell provider create \ - --name ollama \ - --type openai \ - --credential OPENAI_API_KEY=empty \ - --config OPENAI_BASE_URL=http://host.openshell.internal:11434/v1 +openshell sandbox create \ + --name ollama-client \ + --provider ollama \ + -- \ + curl http://host.openshell.internal:11434/v1/chat/completions \ + --json '{ + "model":"qwen3.5:0.8b", + "messages":[{"role":"user","content":"hello"}], + "max_tokens":10 + }' ``` -OpenShell injects `host.openshell.internal` so sandboxes and the gateway can reach the host machine. You can also use the host's LAN IP. - -### Set Inference Routing - -```shell -openshell inference set --provider ollama --model qwen3.5:0.8b -``` - -Confirm: - -```shell -openshell inference get -``` - -### Verify from a Sandbox - -```shell -openshell sandbox create -- \ - curl https://inference.local/v1/chat/completions \ - --json '{"messages":[{"role":"user","content":"hello"}],"max_tokens":10}' -``` - -The response should be JSON from the model. - - +For an OpenAI SDK client, set its base URL to +`http://host.openshell.internal:11434/v1`, use any non-empty API key value the +SDK accepts, and select the real Ollama model in the request. ## Troubleshooting -Common issues and fixes: - -- **Ollama not reachable from sandbox:** Ollama must be bound to `0.0.0.0`, not `127.0.0.1`. This applies to host-level Ollama only; the community sandbox handles this automatically. -- **`OPENAI_BASE_URL` wrong:** Use `http://host.openshell.internal:11434/v1`, not `localhost` or `127.0.0.1`. -- **Model not found:** Run `ollama ps` to confirm the model is loaded. Run `ollama pull ` if needed. -- **HTTPS instead of HTTP:** Code inside sandboxes must call `https://inference.local`, not `http://`. -- **AMD GPU driver issues:** Ollama v0.18+ requires ROCm 7 drivers for AMD GPUs. Update your drivers if you see GPU detection failures. - -Useful commands: - -```shell -openshell status -openshell inference get -openshell provider get ollama -``` +- Bind host-level Ollama to `0.0.0.0`, not `127.0.0.1`. +- Use `host.openshell.internal`, not `localhost`, from a sandbox. +- Confirm attachment with `openshell sandbox provider list ollama-client`. +- Inspect the effective policy with `openshell policy get ollama-client --full`. +- Run `ollama ps` and `ollama pull ` when the model is unavailable. +- If the gateway is remote, the hostname refers to the remote gateway host, + not your laptop. Use a shared service address or tunnel instead. ## Next Steps -- To learn more about managed inference, refer to [Inference Routing](/sandboxes/inference-routing). -- To configure a different self-hosted backend, refer to [Inference Routing](/sandboxes/inference-routing#configure-inference-routing). -- To learn how sandbox containers are selected, refer to [Sandboxes](/sandboxes/manage-sandboxes#custom-containers). +- [Provider-backed Inference](/sandboxes/inference-routing) +- [Profiles](/providers/profiles) +- [Customize Sandbox Policies](/sandboxes/policies) diff --git a/docs/get-started/tutorials/local-inference-lmstudio.mdx b/docs/get-started/tutorials/local-inference-lmstudio.mdx index 7ce604009..4f7ffc7ae 100644 --- a/docs/get-started/tutorials/local-inference-lmstudio.mdx +++ b/docs/get-started/tutorials/local-inference-lmstudio.mdx @@ -1,214 +1,139 @@ --- # SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. # SPDX-License-Identifier: Apache-2.0 -title: "Route Local Inference Requests to LM Studio" +title: "Use Local Inference with LM Studio" sidebar-title: "Local Inference with LM Studio" slug: "get-started/tutorials/local-inference-lmstudio" -description: "Configure inference.local to route sandbox requests to a local LM Studio server running on the gateway host." -keywords: "Generative AI, Cybersecurity, Tutorial, Inference Routing, LM Studio, Local Inference, Sandbox" +description: "Attach a provider profile for a local LM Studio server and call its native compatible endpoints." +keywords: "Generative AI, Cybersecurity, Tutorial, LM Studio, Local Inference, Sandbox, Provider Profile" --- -This tutorial describes how to configure OpenShell to route inference requests to a local LM Studio server. - - -The LM Studio server provides easy setup with both OpenAI and Anthropic compatible endpoints. - - - -This tutorial covers: - -- Expose a local inference server to OpenShell sandboxes. -- Verify end-to-end inference from inside a sandbox. +This tutorial exposes an LM Studio server on the gateway host to selected +sandboxes. The provider attachment grants access; the workload calls LM +Studio's OpenAI- or Anthropic-compatible endpoint directly. ## Prerequisites -First, complete OpenShell installation and follow the [Quickstart](/get-started/quickstart). +- Complete the [Quickstart](/get-started/quickstart). +- Install [LM Studio](https://lmstudio.ai/download) on the gateway host. -[Install the LM Studio app](https://lmstudio.ai/download). Make sure that your LM Studio is running in the same environment as your gateway. - -If you prefer to work without having to keep the LM Studio app open, download llmster (headless LM Studio) with the following command: +For a headless installation: - + + ```shell curl -fsSL https://lmstudio.ai/install.sh | bash ``` - + + ```shell irm https://lmstudio.ai/install.ps1 | iex ``` + -And start llmster: +Start the daemon and server: ```shell lms daemon up +lms server start --bind 0.0.0.0 ``` - - -## Start LM Studio Local Server - -Start the LM Studio local server from the Developer tab, and verify the OpenAI-compatible endpoint is enabled. - -LM Studio listens to `127.0.0.1:1234` by default. For use with OpenShell, configure LM Studio to listen on all interfaces (`0.0.0.0`). - -If you use the GUI, go to the Developer Tab, select Server Settings, then enable Serve on Local Network. - -If you use llmster in headless mode, run `lms server start --bind 0.0.0.0`. - -## Test with a small model - -In the LM Studio app, head to the Model Search tab to download a small model like Qwen3.5 2B. - -In the terminal, use the following command to download and load the model: +Load a model: ```shell lms get qwen/qwen3.5-2b lms load qwen/qwen3.5-2b ``` -## Add LM Studio as a provider +## Create the Provider Profile -Choose the provider type that matches the client protocol you want to route through `inference.local`. +Save this as `lmstudio.yaml`: - - - -Add LM Studio as an OpenAI-compatible provider through `host.openshell.internal`: - -```shell -openshell provider create \ - --name lmstudio \ - --type openai \ - --credential OPENAI_API_KEY=lmstudio \ - --config OPENAI_BASE_URL=http://host.openshell.internal:1234/v1 +```yaml +id: lmstudio +display_name: LM Studio +description: Host-level LM Studio compatible APIs +category: inference +inference_capable: true +credentials: [] +endpoints: + - host: host.openshell.internal + port: 1234 + protocol: rest + access: read-write + enforcement: enforce +binaries: + - /usr/bin/curl + - /usr/local/bin/curl + - /usr/bin/python3 + - /usr/local/bin/python + - /sandbox/.uv/python/** + - /sandbox/.venv/** ``` -Use this provider for clients that send OpenAI-compatible requests such as `POST /v1/chat/completions` or `POST /v1/responses`. - - - - - -Add a provider that points to LM Studio's Anthropic-compatible `POST /v1/messages` endpoint: - ```shell -openshell provider create \ - --name lmstudio-anthropic \ - --type anthropic \ - --credential ANTHROPIC_API_KEY=lmstudio \ - --config ANTHROPIC_BASE_URL=http://host.openshell.internal:1234 +openshell provider profile lint -f lmstudio.yaml +openshell provider profile import -f lmstudio.yaml +openshell provider create --name lmstudio --type lmstudio ``` -Use this provider for Anthropic-compatible `POST /v1/messages` requests. +The profile intentionally has no credential. LM Studio does not require one by +default, but the provider attachment still supplies its policy boundary. - +## Attach and Call the Native Endpoint - - -## Configure LM Studio as the local inference provider - -Set the managed inference route for the active gateway: - -

- - - +OpenAI-compatible request: ```shell -openshell inference set --provider lmstudio --model qwen/qwen3.5-2b -``` - -If the command succeeds, OpenShell has verified that the upstream is reachable and accepts the expected OpenAI-compatible request shape. - - - - - -```shell -openshell inference set --provider lmstudio-anthropic --model qwen/qwen3.5-2b -``` - -If the command succeeds, OpenShell has verified that the upstream is reachable and accepts the expected Anthropic-compatible request shape. - - - - - -
- -The active `inference.local` route is gateway-scoped, so only one provider and model pair is active at a time. Re-run `openshell inference set` whenever you want to switch between OpenAI-compatible and Anthropic-compatible clients. - -Confirm the saved config: - -```shell -openshell inference get -``` - -You should see either `Provider: lmstudio` or `Provider: lmstudio-anthropic`, along with `Model: qwen/qwen3.5-2b`. - -## Verify from Inside a Sandbox - -Run a simple request through `https://inference.local`: - - - - -```shell showLineNumbers={true} -openshell sandbox create -- \ - curl https://inference.local/v1/chat/completions \ - --json '{"messages":[{"role":"user","content":"hello"}],"max_tokens":10}' - -openshell sandbox create -- \ - curl https://inference.local/v1/responses \ +openshell sandbox create \ + --name lmstudio-client \ + --provider lmstudio \ + -- \ + curl http://host.openshell.internal:1234/v1/chat/completions \ --json '{ - "instructions": "You are a helpful assistant.", - "input": "hello", - "max_output_tokens": 10 + "model":"qwen/qwen3.5-2b", + "messages":[{"role":"user","content":"hello"}], + "max_tokens":10 }' ``` - - - +Anthropic-compatible request: ```shell -openshell sandbox create -- \ - curl https://inference.local/v1/messages \ - --json '{"messages":[{"role":"user","content":"hello"}],"max_tokens":10}' +openshell sandbox exec lmstudio-client -- \ + curl http://host.openshell.internal:1234/v1/messages \ + --json '{ + "model":"qwen/qwen3.5-2b", + "messages":[{"role":"user","content":"hello"}], + "max_tokens":10 + }' ``` - - - - - +Configure SDKs with the same native base URL. Some SDKs require a non-empty +API key even when LM Studio ignores it; use a literal non-secret value such as +`unused`. Keep the actual model ID in the client request. ## Troubleshooting -If setup fails, check these first: - -- LM Studio local server is running and reachable from the gateway host -- `OPENAI_BASE_URL` uses `http://host.openshell.internal:1234/v1` when you use an `openai` provider -- `ANTHROPIC_BASE_URL` uses `http://host.openshell.internal:1234` when you use an `anthropic` provider -- The gateway and LM Studio run on the same machine or a reachable network path -- The configured model name matches the model exposed by LM Studio - -Useful commands: - -```shell -openshell status -openshell inference get -openshell provider get lmstudio -openshell provider get lmstudio-anthropic -``` +- Enable **Serve on Local Network** in the LM Studio Developer tab, or use + `lms server start --bind 0.0.0.0`. +- Use `host.openshell.internal`, not `127.0.0.1` or `localhost`, in the + sandbox. +- Confirm the model is loaded with `lms ps`. +- Confirm the attachment with + `openshell sandbox provider list lmstudio-client`. +- Inspect endpoint and binary policy with + `openshell policy get lmstudio-client --full`. +- A remote gateway cannot reach an LM Studio process on your laptop without a + tunnel or shared network path. ## Next Steps -- To learn more about using the LM Studio CLI, refer to [LM Studio docs](https://lmstudio.ai/docs/cli) -- To learn more about managed inference, refer to [Inference Routing](/sandboxes/inference-routing). -- To configure a different self-hosted backend, refer to [Inference Routing](/sandboxes/inference-routing#configure-inference-routing). +- [Provider-backed Inference](/sandboxes/inference-routing) +- [Profiles](/providers/profiles) +- [LM Studio CLI](https://lmstudio.ai/docs/cli) diff --git a/docs/get-started/tutorials/microsoft-graph-provider-refresh.mdx b/docs/get-started/tutorials/microsoft-graph-provider-refresh.mdx index d3c4a7584..76adf55fd 100644 --- a/docs/get-started/tutorials/microsoft-graph-provider-refresh.mdx +++ b/docs/get-started/tutorials/microsoft-graph-provider-refresh.mdx @@ -1,14 +1,14 @@ --- # SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. # SPDX-License-Identifier: Apache-2.0 -title: "Refresh Microsoft Graph Credentials with Providers v2" +title: "Refresh Microsoft Graph Credentials with a Provider Profile" sidebar-title: "Microsoft Graph Provider Refresh" slug: "get-started/tutorials/microsoft-graph-provider-refresh" -description: "Configure a Providers v2 Microsoft Graph profile with gateway-managed OAuth2 refresh-token rotation." +description: "Configure a Microsoft Graph provider profile with gateway-managed OAuth2 refresh-token rotation." keywords: "Generative AI, Cybersecurity, Tutorial, Providers, Microsoft Graph, OAuth2, Credential Refresh, Sandbox" --- -Use Providers v2 to keep Microsoft Graph access tokens short lived while sandboxes receive a stable `MS_GRAPH_ACCESS_TOKEN` placeholder. OpenShell stores the non-injectable refresh material at the gateway, refreshes the Microsoft Graph access token before it expires, updates the provider record, and injects the current credential into newly launched sandbox processes. +Use a provider profile to keep Microsoft Graph access tokens short lived while sandboxes receive a stable `MS_GRAPH_ACCESS_TOKEN` placeholder. OpenShell stores the non-injectable refresh material at the gateway, refreshes the Microsoft Graph access token before it expires, updates the provider record, and injects the current credential into newly launched sandbox processes. After completing this tutorial, you have: @@ -43,14 +43,6 @@ Do not commit access tokens, refresh tokens, or local `.env` files. The commands -## Enable Providers v2 - -Enable provider profile policy composition on the active gateway: - -```shell -openshell settings set --global --key providers_v2_enabled --value true --yes -``` - ## Create a Microsoft Graph Provider Profile Create `microsoft-graph-mail.yaml` with this profile: diff --git a/docs/images/splunk-cim-v1.1-downgrade.png b/docs/images/splunk-cim-v1.1-downgrade.png new file mode 100644 index 000000000..feb8e2168 Binary files /dev/null and b/docs/images/splunk-cim-v1.1-downgrade.png differ diff --git a/docs/index.mdx b/docs/index.mdx index 3d774b1b4..b7623332e 100644 --- a/docs/index.mdx +++ b/docs/index.mdx @@ -3,7 +3,7 @@ # SPDX-License-Identifier: Apache-2.0 title: "NVIDIA OpenShell Developer Guide" description: "OpenShell is the safe, private runtime for autonomous AI agents. Run agents in sandboxed environments that protect your data, credentials, and infrastructure." -keywords: "Generative AI, Cybersecurity, AI Agents, Sandboxing, Security, Privacy, Inference Routing" +keywords: "Generative AI, Cybersecurity, AI Agents, Sandboxing, Security, Privacy, Provider Credentials" position: 1 --- @@ -91,9 +91,9 @@ Deploy gateways, create sandboxes, configure policies, providers, and community Concept - + -Keep inference traffic private by routing API calls to local or self-hosted backends. +Attach model providers to selected sandboxes and call their native endpoints without exposing credentials. Concept diff --git a/docs/kubernetes/access-control.mdx b/docs/kubernetes/access-control.mdx index d6cb2140b..3fd7a66ec 100644 --- a/docs/kubernetes/access-control.mdx +++ b/docs/kubernetes/access-control.mdx @@ -31,7 +31,7 @@ The gateway verifies supervisor JWT-SVIDs with JWT bundles fetched from the SPIF ## OIDC User Authentication -Set `server.oidc.issuer` to enable OIDC. The gateway validates the `Authorization: Bearer ` header on every request against the issuer's JWKS endpoint. +Set `server.oidc.issuer` to enable OIDC. The gateway validates the `Authorization: Bearer ` header on every request against the issuer's JWKS endpoint. It accepts JWTs signed with RS256, RS384, RS512, PS256, PS384, PS512, ES256, ES384, or EdDSA (Ed25519) keys published in the issuer JWKS. Okta tenants that sign access tokens with ES256 work without additional gateway configuration. ```shell helm upgrade openshell \ @@ -39,9 +39,12 @@ helm upgrade openshell \ --version \ --namespace openshell \ --set server.oidc.issuer=https://your-idp.example.com/realms/openshell \ - --set server.oidc.audience=openshell-cli + --set server.oidc.audience=openshell-cli \ + --set server.tls.clientCaSecretName="" ``` +Set `server.tls.clientCaSecretName=""` when the gateway terminates TLS directly and browsers or CLI clients connect without client certificates. The chart omits `client_ca_path` from `gateway.toml` and does not mount the client-CA volume, leaving HTTPS-only transport with OIDC for user authentication. Do not set the value to `null`; omit the key to use the chart default, or set it to `""` to disable client certificate verification. + The `audience` value must match the client ID configured in your identity provider for the OpenShell resource server. ### OIDC values reference @@ -50,7 +53,7 @@ The `audience` value must match the client ID configured in your identity provid |---|---|---| | `server.oidc.issuer` | `""` | OIDC issuer URL. Empty disables OIDC. | | `server.oidc.audience` | `openshell-cli` | Expected `aud` claim in the JWT. | -| `server.oidc.jwksTtl` | `3600` | JWKS key cache TTL in seconds. | +| `server.oidc.jwksTtl` | `3600` | JWKS key cache TTL in seconds. Must be greater than zero. | | `server.oidc.rolesClaim` | `""` | Dot-separated path to the roles array in JWT claims. | | `server.oidc.adminRole` | `""` | Role name that grants admin access. | | `server.oidc.userRole` | `""` | Role name that grants standard user access. | @@ -69,6 +72,7 @@ helm upgrade openshell \ --namespace openshell \ --set server.oidc.issuer=https://your-idp.example.com/realms/openshell \ --set server.oidc.audience=openshell-cli \ + --set server.tls.clientCaSecretName="" \ --set server.oidc.rolesClaim=realm_access.roles \ --set server.oidc.adminRole=openshell-admin \ --set server.oidc.userRole=openshell-user @@ -100,6 +104,8 @@ helm upgrade openshell \ The gateway still serves TLS and sandbox supervisors still authenticate with gateway-minted sandbox JWTs. User-facing CLI/API calls without OIDC or mTLS credentials are accepted as an unauthenticated local developer principal. The proxy is responsible for authenticating callers and forwarding only authorized traffic. +When the gateway terminates TLS directly and callers connect without client certificates, also set `server.tls.clientCaSecretName=""` as described in the OIDC section above. + To also disable TLS entirely (when the proxy terminates TLS before the request reaches the gateway): ```shell diff --git a/docs/kubernetes/setup.mdx b/docs/kubernetes/setup.mdx index 221f935eb..fb7881af2 100644 --- a/docs/kubernetes/setup.mdx +++ b/docs/kubernetes/setup.mdx @@ -99,6 +99,36 @@ helm upgrade --install openshell \ The chart automatically generates PKI secrets on first install using pre-install Helm hooks. No manual secret creation is required. +### Split gateway and workspace releases + +For a platform-managed namespace, install the gateway without namespace-scoped +sandbox resources, then install the workspace chart in the sandbox namespace: + +```shell +helm upgrade --install openshell \ + oci://ghcr.io/nvidia/openshell/helm-chart \ + --version \ + --namespace openshell \ + --set workspaceResources.enabled=false \ + --set server.sandboxNamespace=app-a + +helm upgrade --install openshell-workspace \ + oci://ghcr.io/nvidia/openshell/openshell-workspace \ + --version \ + --namespace app-a \ + --set gateway.serviceAccount.name=openshell \ + --set gateway.serviceAccount.namespace=openshell +``` + +The workspace chart does not create the namespace or deploy a gateway. It owns +only the sandbox ServiceAccount, Role, RoleBinding, and NetworkPolicy in its +release namespace. For one pre-provisioned namespace, keep +`server.drivers.kubernetes.workspaceMode=shared` and set +`server.sandboxNamespace=app-a`. To map multiple workspaces to separately +provisioned namespaces, use `workspaceMode=operator`, configure exactly one of +`operatorNamespaceLabel` or `operatorNamespaceFile`, and install the workspace +chart in every allowlisted namespace. + ## Wait for the gateway to be ready ```shell @@ -163,6 +193,7 @@ The most commonly changed values are: | `workload.kind` | Gateway workload controller. Use `statefulset` for SQLite or `deployment` with `server.externalDbSecret`. | | `workload.allowMultiReplicaStatefulSet` | Allow `replicaCount > 1` with `workload.kind=statefulset`. Prefer Deployment for external database-backed multi-replica gateways. | | `server.sandboxNamespace` | Namespace where sandbox pods are created. Defaults to the Helm release namespace when left empty. | +| `workspaceResources.enabled` | Create namespace-scoped sandbox prerequisites from the gateway chart. Disable when installing the workspace chart separately. | | `server.externalDbSecret` | Secret containing a PostgreSQL connection URI in the `uri` key. Use when the database is managed outside the chart. | | `server.telemetryEnabled` | Enable anonymous OpenShell telemetry from the gateway and its sandbox supervisors. Set to `false` to opt out. | | `server.sandboxImage` | Default sandbox image used when a sandbox does not specify one. | diff --git a/docs/observability/logging.mdx b/docs/observability/logging.mdx index bc8d54324..4bc4aad6d 100644 --- a/docs/observability/logging.mdx +++ b/docs/observability/logging.mdx @@ -24,7 +24,7 @@ These events cover startup plumbing, gRPC communication, and internal state tran ### OCSF structured events -Network, process, filesystem, configuration, and API activity events use the [Open Cybersecurity Schema Framework (OCSF)](https://ocsf.io) format. OCSF is an open standard for normalizing security telemetry across tools and platforms. OpenShell maps sandbox events to OCSF v1.8.0 event classes, including API Activity [6003] with the `ai_operation` profile for inference observability. +Network, process, filesystem, configuration, and API activity events use the [Open Cybersecurity Schema Framework (OCSF)](https://ocsf.io) format. OCSF is an open standard for normalizing security telemetry across tools and platforms. OpenShell maps sandbox events to OCSF v1.8.0 event classes. In the log file, OCSF events appear in a shorthand format with an `OCSF` level label, designed for quick human and agent scanning: @@ -59,9 +59,8 @@ OpenShell maps sandbox events to these OCSF classes: | `SSH:` | SSH Activity | 4007 | SSH handshakes, authentication, channel operations | | `PROC:` | Process Activity | 1007 | Process start, exit, timeout, signal failures | | `FINDING:` | Detection Finding | 2004 | Security findings (nonce replay, proxy bypass, unsafe policy) | -| `CONFIG:` | Device Config State Change | 5019 | Policy load/reload, Landlock, TLS setup, inference routes | +| `CONFIG:` | Device Config State Change | 5019 | Policy load/reload, Landlock, TLS setup, provider and settings changes | | `LIFECYCLE:` | Application Lifecycle | 6002 | Sandbox supervisor start, SSH server ready | -| `API:INFERENCE` | API Activity | 6003 | AI model inference calls through `inference.local` (model, provider, latency, tokens) | ## Reading the Shorthand Format diff --git a/docs/observability/ocsf-json-export.mdx b/docs/observability/ocsf-json-export.mdx index bb85ff6cf..82c4e2b8c 100644 --- a/docs/observability/ocsf-json-export.mdx +++ b/docs/observability/ocsf-json-export.mdx @@ -3,7 +3,7 @@ # SPDX-License-Identifier: Apache-2.0 title: "OCSF JSON Export" sidebar-title: "OCSF JSON Export" -description: "How to enable full OCSF JSON logging for SIEM integration, compliance, and structured analysis." +description: "How to enable full OCSF JSON logging for SIEM integration, compliance, and structured analysis. Includes schema version downgrade for AWS Security Lake, Splunk, and CrowdStrike compatibility." keywords: "Generative AI, Cybersecurity, OCSF, JSON, SIEM, Compliance, Observability" --- @@ -39,6 +39,8 @@ When enabled, OCSF JSON records are written to `/var/log/openshell-ocsf.YYYY-MM- ## JSON Record Structure +`metadata.uid` uniquely identifies an event and stays unchanged when that record is serialized again. Use `container.uid` to associate the event with its sandbox, not `metadata.uid`. Records without a sandbox association omit the container; unknown images are omitted rather than represented by an empty image name. + Each line is a complete OCSF v1.8.0 JSON object. Here is an example of a network connection event: ```json @@ -140,33 +142,57 @@ The `class_uid` field identifies the event type: | 2004 | Detection Finding | `FINDING:` | | 5019 | Device Config State Change | `CONFIG:` | | 6002 | Application Lifecycle | `LIFECYCLE:` | -| 6003 | API Activity | `API:INFERENCE` | -### API Activity [6003] — AI Inference +## SIEM Schema Version Compatibility -When the inference proxy routes a model call through `inference.local`, an API Activity event is emitted with the `ai_operation` profile: +OpenShell emits OCSF v1.8.0 events internally, but many SIEMs only support older schema versions. The `ocsf_schema_version` setting tells the JSONL layer to downgrade events before writing, stripping fields and profiles that don't exist in the target version. + +Set the target version globally: + +```shell +openshell settings set --global --key ocsf_schema_version --value "1.1" +``` + +Or per-sandbox: + +```shell +openshell settings set my-sandbox --key ocsf_schema_version --value "1.3" +``` + +The setting takes effect on the next poll cycle, by default every 10 seconds. No sandbox restart is required. + +Supported target versions are `1.1` and `1.3`. When set, the JSONL layer applies the following transformations: + +- Strips fields added after the target version: `ai_model`, `container`, `observation_point_id` +- Removes unknown profile names from `metadata.profiles`: `ai_operation`, `container` +- Rewrites `metadata.version` to match the target version + +The shorthand log output is unaffected. The internal event model stays at v1.8.0; only the serialized JSONL is transformed. + +When unset or empty, no downgrade is applied and events are written at the current schema version. + +| SIEM | Required OCSF Version | `ocsf_schema_version` Value | +|---|---|---| +| AWS Security Lake | v1.1.0 | `1.1` | +| Splunk CIM Add-On | v1.1-v1.3 | `1.1` or `1.3` | +| CrowdStrike FDR | v1.5.0 | Not yet supported | +| Datadog Cloud SIEM | v1.5.0 (selectable) | Not yet supported | + + +Downgrading is lossy. The `ai_model` field and `ai_operation` profile are stripped for v1.1 and v1.3 targets, so AI model attribution is not present in downgraded events. If your audit workflow requires knowing which model handled a request, use the full schema version (leave `ocsf_schema_version` unset). OpenShell supports OCSF v1.8.0 natively; upgrade your SIEM's OCSF schema if you need AI attribution alongside backward-compatible ingestion. + + +Downgraded events include a `downgraded_from` marker in the `unmapped` object so auditors can distinguish "no model was involved" from "model attribution was stripped": ```json { - "class_uid": 6003, - "class_name": "API Activity", - "activity_id": 99, - "activity_name": "Other", - "api": { "operation": "POST /v1/messages" }, - "actor": { "process": { "name": "openshell-supervisor", "pid": 1 } }, - "src_endpoint": { "ip": "127.0.0.1", "port": 3128 }, - "ai_model": { "name": "claude-haiku-4-5-20251001", "ai_provider": "https://api.anthropic.com/v1" }, - "metadata": { "version": "1.8.0", "profiles": ["container", "host", "ai_operation"] }, - "status": "Success", - "unmapped": { "latency_ms": 701 } + "unmapped": { + "downgraded_from": "1.8.0" + } } ``` -The shorthand log renders this as: - -```text -OCSF API:INFERENCE [INFO] Success claude-haiku-4-5-20251001 via https://api.anthropic.com/v1 701ms [POST /v1/messages] -``` +The core OCSF event structure (class UIDs, activity IDs, network and HTTP fields) is identical across v1.1 through v1.8. The stripped fields are all profile-gated additions. ## Integration with External Tools @@ -174,8 +200,9 @@ The JSONL file can be shipped to any tool that accepts OCSF-formatted data: | Tool | Integration Path | |---|---| -| Splunk | Use the [Splunk OCSF Add-on](https://splunkbase.splunk.com/app/6943) to ingest OCSF JSONL files. | -| Amazon Security Lake | OCSF is the native schema for Security Lake. | +| Splunk | Use the [Splunk OCSF Add-on](https://splunkbase.splunk.com/app/6943) to ingest OCSF JSONL files. Set `ocsf_schema_version` to `1.3` for CIM Add-On compatibility. | +| Amazon Security Lake | OCSF is the native schema for Security Lake. Set `ocsf_schema_version` to `1.1` for v1.1.0 compatibility. | +| CrowdStrike FDR | Ship JSONL files to Falcon Data Replicator. v1.5 downgrade target is not yet supported. | | Elastic | Use Filebeat to ship JSONL files with the OCSF field mappings. | | Custom pipelines | Parse the JSONL file with `jq`, Python, or any JSON-capable tool. | @@ -188,7 +215,9 @@ cat /var/log/openshell-ocsf.2026-04-01.log | \ ## Relationship to Shorthand Logs -The shorthand format in `openshell.YYYY-MM-DD.log` and the JSON format in `openshell-ocsf.YYYY-MM-DD.log` are derived from the same OCSF events. The shorthand is a human-readable projection; the JSON is the complete record. Both are generated at the same time from the same event data. +Sandbox events identify the sandbox environment with `device.type_id: 99` (Other) and `device.type: "Sandbox"`. The operating system is reported separately in `device.os.name`; the device type does not classify the underlying host or compute backend. + +The shorthand format in `openshell.YYYY-MM-DD.log` and the JSON format in `openshell-ocsf.YYYY-MM-DD.log` are derived from the same OCSF events. The shorthand is a human-readable projection; the JSON is the full structured record when no schema downgrade is configured. When `ocsf_schema_version` is set, the JSON export is a lossy projection of the internal event model. Both formats are generated at the same time from the same event data. The shorthand log is always active. The JSON export is opt-in through `ocsf_json_enabled`. diff --git a/docs/providers/google-vertex-ai.mdx b/docs/providers/google-vertex-ai.mdx index 21b9d4d37..781768b7e 100644 --- a/docs/providers/google-vertex-ai.mdx +++ b/docs/providers/google-vertex-ai.mdx @@ -3,28 +3,30 @@ # SPDX-License-Identifier: Apache-2.0 title: "Google Vertex AI" sidebar-title: "Google Vertex AI" -description: "Configure OpenShell to route inference traffic through Google Vertex AI, including Anthropic Claude and Gemini models." -keywords: "Generative AI, Cybersecurity, AI Agents, Sandboxing, Google Vertex AI, Anthropic Claude, Inference Routing" +description: "Attach a Google Vertex AI provider and call native Vertex endpoints with gateway-refreshed credentials." +keywords: "Generative AI, Cybersecurity, AI Agents, Sandboxing, Google Vertex AI, Anthropic Claude" --- -Google Vertex AI is a managed machine learning platform that hosts Anthropic Claude, Gemini, and third-party models through Google Cloud. OpenShell can route `inference.local` traffic to Vertex AI using gateway-managed credential refresh, so sandbox agents do not handle GCP credentials directly. +The `google-vertex-ai` provider gives selected sandboxes access to native +Google Vertex AI endpoints. OpenShell keeps refresh bootstrap material at the +gateway, rotates short-lived access tokens, and resolves token placeholders +only at endpoints authorized by the provider profile. + +OpenShell does not choose a model or transform a request. The workload uses the +native Vertex endpoint and request format for its selected model. ## Prerequisites -Before creating a Vertex AI provider, ensure you have: +- A GCP project with the Vertex AI API enabled. +- A service account with the Vertex AI User role and a downloaded JSON key for + production, or gcloud Application Default Credentials for local development. +- Access to the selected model in the intended Vertex region. -- A GCP project with the [Vertex AI API](https://console.cloud.google.com/apis/library/aiplatform.googleapis.com) enabled. -- One of the following: - - A GCP service account with the **Vertex AI User** role and a downloaded JSON key file, for production use. - - The `gcloud` CLI with Application Default Credentials configured, for local development. - -## Authentication - -The `google-vertex-ai` provider supports two credential sources. +## Create a Provider ### Service Account Key -Supply the JSON key file content as the `GOOGLE_SERVICE_ACCOUNT_KEY` credential. OpenShell persists that value only as gateway-side refresh bootstrap material until you update or delete it. The raw service-account JSON and private key are not sandbox runtime credentials and are not exposed to sandboxes. Runtime inference requests use short-lived access tokens minted by the gateway and stored under a separate credential key. +Create the provider with the JSON key as gateway-only bootstrap material: ```shell openshell provider create \ @@ -35,7 +37,7 @@ openshell provider create \ --config VERTEX_AI_REGION=us-central1 ``` -Then configure gateway-managed refresh so the gateway uses the private key as refresh bootstrap material and rotates access tokens: +Configure gateway-managed refresh: ```shell openshell provider refresh configure vertex-prod \ @@ -46,15 +48,16 @@ openshell provider refresh configure vertex-prod \ --secret-material-key private_key ``` +The private key remains in the gateway credential store. Sandboxes receive +only an opaque placeholder for the short-lived access token. + ### gcloud Application Default Credentials -For local development, configure ADC first, then pass `--from-gcloud-adc`: +For local development: ```shell gcloud auth application-default login -``` -```shell openshell provider create \ --name vertex-local \ --type google-vertex-ai \ @@ -63,162 +66,115 @@ openshell provider create \ --config VERTEX_AI_REGION=us-central1 ``` -`--from-gcloud-adc` reads `GOOGLE_APPLICATION_CREDENTIALS` first, then falls back to `$CLOUDSDK_CONFIG/application_default_credentials.json` when that environment variable is set, then to `~/.config/gcloud/application_default_credentials.json`. It configures an OAuth2 refresh token flow on the gateway and immediately mints the first access token before the command returns. If the command succeeds, the provider is ready for inference right away. It only works with user credentials generated by `gcloud auth application-default login`. If your ADC file is a service account key, the CLI returns an error and directs you to use the service account key method above. - -ADC-backed providers mint and rotate access tokens into `GOOGLE_VERTEX_AI_TOKEN`. - - -`--from-gcloud-adc` is valid for `google-vertex-ai` and `google-cloud` providers. - +`--from-gcloud-adc` reads authorized-user ADC, configures an OAuth2 refresh +grant at the gateway, and immediately mints `GOOGLE_VERTEX_AI_TOKEN`. The ADC +file and refresh token do not enter the sandbox. ## Configuration Keys -Pass these as `--config KEY=VALUE` when creating the provider, or set them as environment variables and use `--from-existing`. - | Key | Required | Default | Description | |---|---|---|---| -| `VERTEX_AI_PROJECT_ID` | Yes (unless `GOOGLE_VERTEX_AI_BASE_URL` or `VERTEX_AI_BASE_URL` is set) | — | GCP project ID. | -| `VERTEX_AI_REGION` | No | `us-central1` | Vertex location selector. Use a regional location such as `us-central1`, or `global`, `us`, or `eu` for the supported global and multi-region endpoints. | -| `GOOGLE_VERTEX_AI_BASE_URL` | No | — | Full base URL override for non-Anthropic routes. Must be an official Vertex AI HTTPS endpoint root. | -| `VERTEX_AI_BASE_URL` | No | — | Backward-compatible alias for `GOOGLE_VERTEX_AI_BASE_URL`. | -| `VERTEX_AI_PUBLISHER` | No | Inferred from model name | Set to `anthropic` to force Anthropic Messages API routing, or any other value for OpenAI-compatible routing. | +| `VERTEX_AI_PROJECT_ID` | Yes | — | GCP project ID exposed as non-secret workload configuration. | +| `VERTEX_AI_REGION` | No | `us-central1` | Vertex location exposed as non-secret workload configuration. | -When `VERTEX_AI_PROJECT_ID` is set and no base URL override is present, the gateway maps `VERTEX_AI_REGION` to the Vertex host automatically: +When the provider is attached, OpenShell also projects standard project and +location aliases such as `GOOGLE_CLOUD_PROJECT`, `ANTHROPIC_VERTEX_PROJECT_ID`, +`CLOUD_ML_REGION`, and `VERTEX_LOCATION`. -- Regional locations such as `us-central1` use `https://-aiplatform.googleapis.com`. -- `global` uses `https://aiplatform.googleapis.com`. -- `us` and `eu` use `https://aiplatform..rep.googleapis.com`. +## Attach the Provider -For Anthropic models, OpenShell builds the publisher-model Vertex path automatically and injects `anthropic_version` into the request body. Vertex rawPredict does not receive `anthropic-version` as a header, and OpenShell strips `anthropic-beta` for Vertex Claude routes. For non-Anthropic models, OpenShell uses Vertex's OpenAI-compatible Chat Completions route under `.../endpoints/openapi/chat/completions`. - - -Use `GOOGLE_VERTEX_AI_BASE_URL` or `VERTEX_AI_BASE_URL` only for non-Anthropic Vertex routes. OpenShell rejects Anthropic models when a base URL override is set because Anthropic routes require model-path shaping and `anthropic_version` body injection. Overrides must use `https://` and an official Vertex AI hostname such as `aiplatform.googleapis.com`, `aiplatform.us.rep.googleapis.com`, `aiplatform.eu.rep.googleapis.com`, or `-aiplatform.googleapis.com`. - - -## Supported Models - -Vertex AI hosts Anthropic Claude models (claude-3-5-sonnet, claude-3-opus, and others) through a native Messages API integration, and Gemini and other third-party models through Vertex's OpenAI-compatible Chat Completions endpoint. OpenShell infers the routing path from the model name. For the full list of available models and regions, refer to the [Google Cloud model garden documentation](https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/overview). - -Model names that match the `claude-*` pattern route through the Anthropic Messages API on Vertex. All other model names route through Vertex Chat Completions. Set `VERTEX_AI_PUBLISHER=anthropic` to force Anthropic routing when the model name does not follow the standard pattern. - -OpenShell exposes Anthropic Vertex routes for inference only. It does not advertise OpenAI-style model discovery for those routes, so use the Google Cloud docs or Model Garden to discover supported Anthropic model IDs. - -## Configure Inference Routing - -Before configuring inference routing, enable provider endpoint injection so the Vertex AI network endpoints are automatically included in sandbox policies: +Attach it while creating a sandbox: ```shell -openshell settings set --global --key providers_v2_enabled --value true --yes +openshell sandbox create \ + --name vertex-agent \ + --provider vertex-local ``` -Then point `inference.local` at the provider: +Or attach it to an existing sandbox: ```shell -openshell inference set \ - --provider vertex-prod \ - --model claude-sonnet-4-6 +openshell sandbox provider attach vertex-agent vertex-local ``` -Use `--no-verify` if the endpoint verification fails. This is common with the `global` region, where the validation probe may not match the actual rawPredict path: +Launch a new process after runtime attachment so it receives the provider +environment. Existing processes do not gain newly attached environment +variables. + +## Call the Native Vertex API + +Claude models use Vertex's publisher-model endpoint. Run a request from a new +sandbox process: ```shell -openshell inference set \ - --provider vertex-prod \ - --model claude-sonnet-4-6 \ - --no-verify +openshell sandbox exec vertex-agent -- sh -lc ' + token=${GOOGLE_VERTEX_AI_SERVICE_ACCOUNT_TOKEN:-$GOOGLE_VERTEX_AI_TOKEN} + curl -X POST \ + -H "Authorization: Bearer $token" \ + -H "Content-Type: application/json" \ + -d '\''{ + "anthropic_version":"vertex-2023-10-16", + "max_tokens":1024, + "messages":[{"role":"user","content":"Hello"}] + }'\'' \ + "https://${CLOUD_ML_REGION}-aiplatform.googleapis.com/v1/projects/${GOOGLE_CLOUD_PROJECT}/locations/${CLOUD_ML_REGION}/publishers/anthropic/models/claude-sonnet-4-6:rawPredict" +' ``` -Sandboxes on that gateway reach the model at `https://inference.local`. For full details on inference routing, refer to [Inference Routing](/sandboxes/inference-routing). +Use the model ID and location supported by your GCP project. For `global`, `us`, +or `eu`, use the corresponding Google-documented hostname instead of the +regional `-aiplatform.googleapis.com` form. -## Use from a Sandbox +Gemini and third-party models use their documented native or +OpenAI-compatible Vertex endpoints. Configure the model, URL, streaming mode, +and timeout in the client. OpenShell does not rewrite them. -Agents inside sandboxes should reach Vertex AI through `inference.local`, not by connecting to Vertex AI directly. The gateway manages GCP credential refresh and request translation; the agent only needs to point its SDK at the local endpoint. +## Verify and Troubleshoot -The complete setup from scratch: +Inspect the attachment and effective policy: ```shell -# 1. Enable provider endpoint injection -openshell settings set --global --key providers_v2_enabled --value true --yes - -# 2. Create the provider -openshell provider create \ - --name vertex-local \ - --type google-vertex-ai \ - --from-gcloud-adc \ - --config VERTEX_AI_PROJECT_ID=my-gcp-project \ - --config VERTEX_AI_REGION=us-central1 - -# 3. Configure inference routing -openshell inference set --provider vertex-local --model claude-sonnet-4-6 --no-verify - -# 4. Create a sandbox with the provider attached -openshell sandbox create --name my-sandbox --provider vertex-local +openshell sandbox provider list vertex-agent +openshell policy get vertex-agent --full +openshell provider refresh status vertex-local ``` -Then inside the sandbox, launch the agent as shown below. +Common failures: - - +- A missing token variable usually means the process started before provider + attachment. Launch a new process. +- `connection not allowed by policy` means the provider endpoint or caller + binary is absent from the effective policy. A gateway global policy override + suppresses provider-derived entries. +- `credential_endpoint_mismatch` means the request destination is outside the + provider profile's endpoint binding. +- A Vertex 400 or 404 usually means the model, location, publisher path, or + request body does not match the native API. +- A Vertex 401 or 403 can indicate an expired refresh grant or missing GCP IAM + permission. Check `provider refresh status` and the Vertex AI User role. -```shell -ANTHROPIC_BASE_URL="https://inference.local" ANTHROPIC_API_KEY=unused claude --bare -``` +Provider creation does not verify model access. The native request is the +end-to-end check. -`--bare` skips the OAuth login flow and uses `ANTHROPIC_API_KEY` directly for authentication. The key value does not reach Vertex AI — `inference.local` strips it and injects the real GCP access token before forwarding. +## Migrate an Existing Vertex Route - -Do not set `CLAUDE_CODE_USE_VERTEX=1` inside the sandbox. That flag makes Claude Code connect directly to Vertex AI and attempt GCP credential discovery (ADC file, metadata service), which fails because the sandbox does not expose GCP credentials. Use `inference.local` instead. - +An earlier managed route stored the provider and model separately and rewrote +requests for the workload. After upgrading, the provider and its refresh state +remain, but the route does not. - - +1. Attach the preserved Vertex provider to each intended sandbox. +2. Launch new workload processes. +3. Move the route's model and timeout into the client configuration. +4. Change the client to the native Vertex endpoint and request format. +5. Verify one non-streaming and one streaming native request before production + rollout. -```shell -ANTHROPIC_BASE_URL="https://inference.local/v1" ANTHROPIC_API_KEY=unused opencode -``` - -OpenCode requires `/v1` in the base URL. Without it, OpenCode sends `POST /messages` instead of `POST /v1/messages`, which does not match the inference pattern and is denied. - - - - -### Policy Proposals - -After running an agent, the TUI (`openshell term`) may show policy proposals for denied endpoints. Common ones for Vertex AI sandboxes: - -| Endpoint | Action | Reason | -|---|---|---| -| `metadata.google.internal:80` | **Reject** | Resolves to `169.254.169.254` (GCE metadata service). Always blocked regardless of policy — the proxy blocks the resolved IP unconditionally to prevent credential exfiltration. | -| `downloads.claude.ai:443` | Approve if desired | Claude Code update checking and asset loading. Not required for inference. | -| `storage.googleapis.com:443` | Approve if desired | Google Cloud Storage. Used by some Claude Code features. Not required for inference. | - -## From Existing Environment - -If one of these token env vars is already set in your shell, create the provider with `--from-existing`: - -- `GOOGLE_VERTEX_AI_TOKEN` or `VERTEX_AI_TOKEN` -- `GOOGLE_VERTEX_AI_SERVICE_ACCOUNT_TOKEN` or `VERTEX_AI_SERVICE_ACCOUNT_TOKEN` - -OpenShell also reads these config env vars during `--from-existing`: - -- `VERTEX_AI_PROJECT_ID` -- `VERTEX_AI_REGION` -- `GOOGLE_VERTEX_AI_BASE_URL` or `VERTEX_AI_BASE_URL` -- `VERTEX_AI_PUBLISHER` - -Then create the provider: - -```shell -openshell provider create \ - --name vertex-env \ - --type google-vertex-ai \ - --from-existing -``` - -This reads credentials and config from the environment variables listed in the configuration keys table above. +Do not attach the provider to every sandbox automatically. The old route was +workspace-global; the replacement intentionally grants access per sandbox. ## Next Steps -- To configure `inference.local` routing, refer to [Inference Routing](/sandboxes/inference-routing). -- To manage provider credentials and refresh, refer to [Providers](/sandboxes/manage-providers). -- To apply network policies to sandboxes using this provider, refer to [Policies](/sandboxes/policies). +- [Provider-backed Inference](/sandboxes/inference-routing) +- [Profiles](/providers/profiles) +- [Providers](/sandboxes/manage-providers) +- [Customize Sandbox Policies](/sandboxes/policies) diff --git a/docs/sandboxes/providers-v2.mdx b/docs/providers/profiles.mdx similarity index 92% rename from docs/sandboxes/providers-v2.mdx rename to docs/providers/profiles.mdx index 7a0f96b5a..e8f6ecc45 100644 --- a/docs/sandboxes/providers-v2.mdx +++ b/docs/providers/profiles.mdx @@ -1,24 +1,24 @@ --- # SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. # SPDX-License-Identifier: Apache-2.0 -title: "Providers v2" -sidebar-title: "Providers v2" +title: "Profiles" +sidebar-title: "Profiles" description: "Use provider profiles to attach credentials, network policy, and refresh metadata to OpenShell sandboxes." keywords: "Generative AI, Cybersecurity, Providers, Provider Profiles, Credentials, Policy, Sandbox" -position: 5 +position: 1 --- -Providers v2 turns providers from credential records into profile-backed access bundles. A provider profile describes the credentials, endpoints, binaries, policy rules, and refresh behavior for a provider type. A provider instance stores the concrete credential and config values for one gateway. +Provider profiles turn providers from credential records into profile-backed access bundles. A provider profile describes the credentials, endpoints, binaries, policy rules, and refresh behavior for a provider type. A provider instance stores the concrete credential and config values for one gateway. -Use Providers v2 when you want provider-owned policy rules to travel with provider credentials. For example, a GitHub provider can describe both `GITHUB_TOKEN` and the GitHub API endpoints that a sandbox needs, so users do not have to copy the same network policy into every sandbox. +Use provider profiles when you want provider-owned policy rules to travel with provider credentials. For example, a GitHub provider can describe both `GITHUB_TOKEN` and the GitHub API endpoints that a sandbox needs, so users do not have to copy the same network policy into every sandbox. -## Why Providers v2 Exists +## Why Provider Profiles Exist Provider credentials and network policy were previously configured through separate workflows. A user could create a GitHub provider that stored `GITHUB_TOKEN`, but the sandbox still needed a separate policy that allowed `api.github.com`, selected the right binaries, and configured REST enforcement. -Providers v2 keeps those pieces together: +Provider profiles keep those pieces together: -| Need | Providers v2 behavior | +| Need | Profile-backed behavior | |---|---| | Repeatable provider setup | Built-in and custom provider profiles define reusable provider types. | | Provider-aware policy | Attached providers contribute `_provider_*` network policy entries to the effective sandbox policy. | @@ -27,32 +27,13 @@ Providers v2 keeps those pieces together: | Credential rotation | Provider refresh metadata lets the gateway refresh short-lived access tokens and update provider records. | | Credential transport | Credential delivery uses environment placeholders and proxy rewrite. Static placeholders resolve only at profile endpoints or explicitly bound sandbox policy endpoints for endpointless profiles. | -## Enable Providers v2 - -Provider profile policy composition is controlled by the gateway-level `providers_v2_enabled` setting. Enable it on the active gateway: - -```shell -openshell settings set --global --key providers_v2_enabled --value true -``` - -When the setting is disabled or unset, attached provider profiles do not add network policy entries to the effective policy. Static provider credential placeholders still use the profile endpoints as a resolution boundary. - -To disable provider profile policy composition, delete the setting: - -```shell -openshell settings delete --global --key providers_v2_enabled -``` - - -The feature flag controls provider-derived policy layers. It does not disable endpoint binding for static credential placeholders. Provider profiles can also declare dynamic token grants that the sandbox proxy resolves on demand for matching HTTP endpoints. - - ## Available Features -Providers v2 currently includes these user-facing features: +Provider profiles include these user-facing features: - Built-in provider profiles loaded by the gateway by default. - Gateway configuration can compose built-in, user-managed, and interceptor-vended profile sources. Selecting only an interceptor makes its catalog authoritative by omission. +- A user-managed profile shadows a built-in profile with the same ID in its applicable workspace or platform scope. The built-in remains the fallback outside that scope. Source-managed interceptor collisions still fail closed. - `openshell provider list-profiles` with table, YAML, and JSON output. - `openshell provider profile export`, `import`, `update`, `lint`, and `delete` for custom profiles. - Provider instances created from built-in or imported profile IDs with `openshell provider create --type `. @@ -74,6 +55,12 @@ supply that boundary by default. For an endpointless profile, a sandbox policy endpoint can name the attached provider instance explicitly. The proxy checks the resulting association before it substitutes the real value. +The built-in `openai` and `anthropic` profiles bind credentials to their public +vendor endpoints. To use a proxy or compatible API at another host, import a +custom profile that declares that host and attach a provider created from that +profile. OpenShell never broadens a built-in profile's credential boundary from +a base URL environment variable alone. + A request can use a static credential only when all of these checks pass: | Check | Configuration source | @@ -209,7 +196,7 @@ Later access-token rotations do not require workload restarts. ## Roadmap -The following Providers v2 design items are not part of the current behavior: +The following provider profile design items are not part of the current behavior: | Roadmap item | Current behavior | |---|---| @@ -217,16 +204,15 @@ The following Providers v2 design items are not part of the current behavior: | Binary-scoped credential injection | Provider profile binaries affect policy composition but do not yet restrict placeholder resolution by calling binary. Static and dynamic credentials are endpoint-scoped. | | Credential verification on create | `openshell provider create` does not yet probe provider verification endpoints or expose `--no-verify`. | | Automatic credential scope extraction | OpenShell does not yet inspect upstream provider responses to discover credential scopes. | -| Inference mounting from attached providers | `inference_capable` is profile metadata. Attaching an inference-capable provider does not yet create `inference.local` routes. | -| Multi-provider inference routing | Path-based routing such as `inference.local/openai/...` and `inference.local/anthropic/...` is not yet wired to provider profiles. | | Policy prover integration | OpenShell does not yet run the policy prover automatically on sandbox startup or block startup based on prover findings. | | Refresh telemetry as OCSF events | Credential refresh logs are secret-safe gateway logs. OCSF refresh events and metrics are future work. | -Use [Inference Routing](/sandboxes/inference-routing) for the current `inference.local` model. +Use [Provider-backed Inference](/sandboxes/inference-routing) to attach an +inference provider and call its native endpoint. ## Provider Profiles -A provider profile defines a provider type. It contains metadata, credential declarations, endpoint policy, binary policy, inference metadata, and optional credential refresh metadata. +A provider profile defines a provider type. It contains metadata, credential declarations, endpoint policy, binary policy, an informational provider category, and optional credential refresh metadata. List available profiles: @@ -241,10 +227,11 @@ of truth: list, export, provider creation, policy composition, and sandbox provider environment resolution use the interceptor-vended profiles instead of built-in or user-imported profiles. -Built-in Providers v2 profiles currently include: +Built-in provider profiles currently include: | Profile ID | Category | Credential environment variables | |---|---|---| +| `anthropic` | `inference` | `ANTHROPIC_API_KEY` | | `claude-code` | `agent` | `ANTHROPIC_API_KEY`, `CLAUDE_API_KEY` | | `codex` | `agent` | `CODEX_AUTH_ACCESS_TOKEN`, `CODEX_AUTH_REFRESH_TOKEN`, `CODEX_AUTH_ACCOUNT_ID`, `CODEX_AUTH_ID_TOKEN` | | `copilot` | `agent` | `COPILOT_GITHUB_TOKEN`, `GH_TOKEN`, `GITHUB_TOKEN` | @@ -253,6 +240,7 @@ Built-in Providers v2 profiles currently include: | `github` | `source_control` | `GITHUB_TOKEN`, `GH_TOKEN` | | `google-vertex-ai` | `inference` | `GOOGLE_SERVICE_ACCOUNT_KEY`, `GOOGLE_VERTEX_AI_SERVICE_ACCOUNT_TOKEN`, `VERTEX_AI_SERVICE_ACCOUNT_TOKEN`, `GOOGLE_VERTEX_AI_TOKEN`, `VERTEX_AI_TOKEN` | | `nvidia` | `inference` | `NVIDIA_API_KEY` | +| `openai` | `inference` | `OPENAI_API_KEY` | | `pypi` | `data` | None | Export a built-in profile as YAML: @@ -440,8 +428,8 @@ binaries: `credentials` declares the credential names, environment variables, auth metadata, optional refresh metadata, and optional dynamic token grant metadata for the provider type. The `auth_style` field accepts `basic`, `bearer`, `header`, `query`, or `path`. When `auth_style` is `path`, set `path_template` to a URL path containing the `{credential}` placeholder exactly once (for example, `/v1/{credential}/resources`). Static credentials are exposed as placeholder environment variables and resolved in outbound HTTP requests only at their binding endpoints. Every static credential environment key receives the full profile endpoint set when the profile defines endpoints. An endpointless profile requires explicit sandbox policy bindings for each attached provider instance. Dynamic token grants are resolved by the sandbox proxy on demand for matching profile endpoints and support `bearer` or `header` placement. Credential environment variable names must not use the reserved `v_` prefix, such as `v10_GITHUB_TOKEN`, because OpenShell uses that namespace for revision-scoped placeholders. -`discovery` controls what `--from-existing` scans when -`providers_v2_enabled=true`. Each entry in `discovery.credentials` must name a +`discovery` controls what `--from-existing` scans. Each entry in +`discovery.credentials` must name a credential declared under `credentials`. OpenShell scans the referenced credential's `env_vars` in order and stores the first non-empty local environment value under the actual environment variable key. @@ -450,7 +438,10 @@ environment value under the actual environment variable key. `binaries` contains the executable paths allowed to reach the profile endpoints when the profile contributes policy to a sandbox. -`inference_capable` marks profiles that are intended to participate in inference workflows. It does not currently mount or configure `inference.local`. +`inference_capable` is informational metadata that marks profiles intended for +model and inference APIs. It does not grant access, select a model, configure a +client, or change routing. The sandbox must attach a provider instance, and the +workload calls the profile-authorized native endpoint. ### Refresh Metadata @@ -465,7 +456,7 @@ Profile YAML can declare these refresh strategies: | `oauth2_refresh_token` | The gateway exchanges a refresh token for a short-lived access token. | | `oauth2_client_credentials` | The gateway mints a short-lived access token with OAuth2 client credentials. | | `google_service_account_jwt` | The gateway signs a Google service account JWT and exchanges it for an access token. | -| `aws_sts_assume_role` | The gateway calls `sts:AssumeRole` and mints `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, and `AWS_SESSION_TOKEN` in one operation. Requires `providers_v2_enabled=true`. | +| `aws_sts_assume_role` | The gateway calls `sts:AssumeRole` and mints `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, and `AWS_SESSION_TOKEN` in one operation. | `openshell provider refresh configure` accepts only gateway-mintable strategies: `oauth2-refresh-token`, `oauth2-client-credentials`, `google-service-account-jwt`, and `aws-sts-assume-role`. Use `openshell provider update` for `static` and `external` refresh patterns. @@ -582,15 +573,11 @@ openshell provider create \ --from-existing ``` -When `providers_v2_enabled=true`, `--from-existing` uses the provider profile's -`discovery` section. If no profile exists for the requested type, the command -fails instead of falling back to the legacy provider registry. When -`providers_v2_enabled=false`, `--from-existing` uses the legacy provider -registry and ignores profile `discovery` metadata. - -For example, with Providers v2 enabled, `--type openai --from-existing` -requires an imported `openai` profile with a `discovery` section. Setting -`OPENAI_API_KEY` alone is not enough for v2 profile discovery. +`--from-existing` uses the provider profile's `discovery` section. If no +profile exists for the requested type, the command fails instead of falling +back to the retired provider registry. The built-in `openai` profile discovers +`OPENAI_API_KEY`; the built-in `anthropic` profile discovers +`ANTHROPIC_API_KEY`. Create a provider from an imported custom profile: @@ -699,7 +686,7 @@ cannot replace or delete its primary credential or any co-minted output. Use management. You can still update unrelated credentials, configuration, and credential expiry metadata. -For a complete Microsoft Graph OAuth2 refresh-token walkthrough, see [Refresh Microsoft Graph Credentials with Providers v2](/get-started/tutorials/microsoft-graph-provider-refresh). +For a complete Microsoft Graph OAuth2 refresh-token walkthrough, see [Refresh Microsoft Graph Credentials with a Provider Profile](/get-started/tutorials/microsoft-graph-provider-refresh). The profile YAML strategy values use underscores, while the CLI `--strategy` values use kebab-case: @@ -846,13 +833,14 @@ openshell sandbox create \ -- claude ``` -When `providers_v2_enabled=true`, each attached provider with a matching profile contributes a provider policy layer to the sandbox effective policy. The base policy is the user-authored sandbox policy that you can edit and apply. The effective policy is the composed policy that the sandbox enforces: base policy plus provider policy layers. When the setting is disabled, the sandbox still receives endpoint-bound provider credentials but not provider-derived policy entries. +Each attached provider with a matching profile contributes a provider policy layer to the sandbox effective policy. The base policy is the user-authored sandbox policy that you can edit and apply. The effective policy is the composed policy that the sandbox enforces: base policy plus provider policy layers. A gateway-global policy suppresses provider-derived policy layers. Updating a custom provider profile affects every provider instance whose `type` matches that profile ID. Provider instances are not rewritten, and sandbox-authored policies are not modified. Running sandboxes observe the updated provider-derived policy on their next config sync. If a gateway-global policy is active, provider-derived policy layers remain suppressed. -Providers v2 does not infer or auto-attach providers from the sandbox command. -Attach providers explicitly with `--provider` during sandbox creation, or use -`openshell sandbox provider attach` after creation. +The CLI can infer a provider profile from a recognized sandbox command and +auto-create the provider from profile discovery. Attach a different existing +provider explicitly with `--provider`, or use `openshell sandbox provider +attach` after creation. List providers attached to a sandbox: @@ -861,6 +849,9 @@ openshell sandbox provider list provider-demo ``` The list output includes provider name, provider type, credential key count, and config key count. +Add `--output json` or `--output yaml` to return the provider name and type plus +sorted `credential_keys` and `config_keys` arrays. Structured output never +includes credential values, opaque credential handles, or config values. ## Policy Composition diff --git a/docs/reference/gateway-auth.mdx b/docs/reference/gateway-auth.mdx index 5969c43e6..ac56e0fcf 100644 --- a/docs/reference/gateway-auth.mdx +++ b/docs/reference/gateway-auth.mdx @@ -91,7 +91,7 @@ The same settings are available through environment variables: |---|---|---| | `OPENSHELL_OIDC_ISSUER` | OIDC issuer URL. The gateway discovers `/.well-known/openid-configuration` from this URL. | None | | `OPENSHELL_OIDC_AUDIENCE` | Expected JWT `aud` claim. | `openshell-cli` | -| `OPENSHELL_OIDC_JWKS_TTL` | JWKS cache TTL in seconds. The gateway also refreshes on an unknown key ID. | `3600` | +| `OPENSHELL_OIDC_JWKS_TTL` | JWKS cache TTL in seconds. The gateway also refreshes on an unknown key ID. Must be greater than zero. | `3600` | | `OPENSHELL_OIDC_ROLES_CLAIM` | Dot-separated claim path containing roles. | `realm_access.roles` | | `OPENSHELL_OIDC_ADMIN_ROLE` | Role required for admin operations. | `openshell-admin` | | `OPENSHELL_OIDC_USER_ROLE` | Role required for standard user operations. | `openshell-user` | @@ -180,18 +180,37 @@ The connection flow: For a headless environment, set `OPENSHELL_NO_BROWSER=1` before registering or logging in to the gateway. When this variable is set and `OPENSHELL_OIDC_CLIENT_SECRET` is not configured, the CLI uses the Device Authorization Grant (RFC 8628) with S256 PKCE. This flow prompts the user to visit a verification URL on any device with a browser and enter a displayed code. The CLI polls the token endpoint until the user completes authorization. This requires the OIDC client to have the device authorization grant enabled on the identity provider. 1. The CLI loads the stored OIDC token bundle. -2. If the access token is expired and a refresh token is available, the CLI refreshes it with the OIDC scopes saved in the gateway metadata. +2. If the access token is expired or near expiry, the CLI refreshes it with the OIDC scopes saved in the gateway metadata. If refresh fails, the command stops before sending a protected request and directs you to run `openshell gateway login `. 3. The CLI connects to the gateway and attaches `authorization: Bearer ` metadata to each gRPC request. -4. The gateway validates the JWT signature, issuer, audience, expiration, and key ID against the issuer's JWKS. +4. The gateway validates the JWT signature, issuer, audience, expiration, subject, and key ID against the issuer's JWKS. 5. The gateway extracts roles and optional scopes from the configured claim paths. 6. The gateway authorizes the gRPC method. Platform-scoped methods require the configured admin role. Workspace-scoped methods require the configured user role and a sufficient membership in the target workspace. Admin role holders satisfy user-role checks and bypass workspace membership checks. For the Platform Admin, Workspace Admin, and Workspace User permissions, refer to [Manage Workspaces and Access](/sandboxes/manage-workspaces). +#### JWT validation + +The gateway validates OIDC access tokens against the issuer's JWKS endpoint. It derives the signing algorithm from each JWK's `kty` and `crv` fields, not from the JWT header: + +| JWK `kty` | `crv` | Algorithm | +|---|---|---| +| RSA | — | RS256, RS384, RS512, PS256, PS384, or PS512 | +| EC | P-256 | ES256 | +| EC | P-384 | ES384 | +| OKP | Ed25519 | EdDSA (Ed25519) | + +For EC and OKP keys, `crv` fully determines the algorithm. For RSA keys, the algorithm is selected from the JWK's declared `alg` field when it names one of the six RSA algorithms above; if `alg` is absent (as with Microsoft Entra ID, which omits it entirely), the gateway defaults to RS256. If `alg` names a non-RSA algorithm, the key is treated as contradictory and skipped (see below) rather than falling back to RS256. + +The gateway rejects tokens when the JWT header `alg` does not match the algorithm pinned from the JWK. Tokens must include `iss`, `aud`, `exp`, and `sub` claims. The `iss` and `aud` values must match the configured issuer and audience. + +JWKs with `use: enc`, an `alg` that genuinely contradicts the derived algorithm (for example an RSA key declaring `ES256`), or `key_ops` that excludes `verify` are skipped. The gateway refreshes the JWKS cache when it encounters an unknown key ID, throttled to at most once per second to bound how much an unknown-`kid` lookup can amplify requests against the issuer. If a refresh yields zero usable keys, the gateway keeps serving the previous key set rather than dropping all signing keys and failing every request, but only for up to three times the configured TTL. If the JWKS is still empty after that grace period, the cached keys are evicted and all tokens are rejected until the issuer recovers. + +Common identity providers such as Keycloak (RS256), Microsoft Entra ID (RSA), and Okta (often ES256) publish compatible signing keys. + If `OPENSHELL_OIDC_SCOPES_CLAIM` is set, the gateway also enforces scopes. It accepts space-delimited scope strings such as `scope: "openid sandbox:read"` and JSON arrays such as `scp: ["sandbox:read"]`. Standard OIDC scopes such as `openid`, `profile`, `email`, and `offline_access` are ignored for authorization. `openshell:all` grants access to all scoped methods. -Supervisor-to-gateway RPCs do not use user OIDC tokens or mTLS user identity. Each sandbox supervisor presents a gateway-minted `Authorization: Bearer` token scoped to its sandbox ID. On Kubernetes, the Kubernetes compute driver validates the projected ServiceAccount token with TokenReview, verifies the live pod UID and controlling `Sandbox` ownerReference, and returns the authenticated sandbox ID to the gateway. The gateway verifies that sandbox still exists before minting its JWT. Log upload, policy status, credential environment lookup, inference bundle lookup, and sandbox config sync run with sandbox-restricted scope, while CLI users authenticate with OIDC, edge auth, local mTLS user authentication, or an explicitly enabled unauthenticated local developer mode. `GetInferenceBundle` returns route material that includes provider credentials, so it requires a sandbox principal; user callers manage inference configuration through the user-facing inference APIs instead. +Supervisor-to-gateway RPCs do not use user OIDC tokens or mTLS user identity. Each sandbox supervisor presents a gateway-minted `Authorization: Bearer` token scoped to its sandbox ID. On Kubernetes, the Kubernetes compute driver validates the projected ServiceAccount token with TokenReview, verifies the live pod UID and controlling `Sandbox` ownerReference, and returns the authenticated sandbox ID to the gateway. The gateway verifies that sandbox still exists before minting its JWT. Log upload, policy status, provider environment lookup, and sandbox config sync run with sandbox-restricted scope, while CLI users authenticate with OIDC, edge auth, local mTLS user authentication, or an explicitly enabled unauthenticated local developer mode. Provider environment responses expose only the credentials and configuration attached to that sandbox, subject to endpoint binding and credential expiry checks. Re-authenticate an OIDC gateway with: diff --git a/docs/reference/gateway-config.mdx b/docs/reference/gateway-config.mdx index 22b64e9e9..14cdb05ec 100644 --- a/docs/reference/gateway-config.mdx +++ b/docs/reference/gateway-config.mdx @@ -105,7 +105,7 @@ disable_tls = false # Shared driver defaults. These inherit into [openshell.drivers.] tables # when the driver-specific table does not override them. -default_image = "ghcr.io/nvidia/openshell/sandbox:latest" +default_image = "ghcr.io/nvidia/openshell-community/sandboxes/base:latest" # Defaults to the gateway version; override to pin a specific build. # supervisor_image = "ghcr.io/nvidia/openshell/supervisor:" client_tls_secret_name = "openshell-client-tls" @@ -140,6 +140,8 @@ max_payload_bytes = 262144 timeout = "500ms" # Gateway listener TLS (distinct from the per-driver guest_tls_*). +# client_ca_path is optional; omit it for HTTPS-only listeners that do not +# verify client certificates. [openshell.gateway.tls] cert_path = "/etc/openshell/certs/gateway.pem" key_path = "/etc/openshell/certs/gateway-key.pem" @@ -172,7 +174,7 @@ service_name = "openshell-gateway" [openshell.gateway.oidc] issuer = "https://idp.example.com/realms/openshell" audience = "openshell-cli" -jwks_ttl_secs = 3600 +jwks_ttl_secs = 3600 # Must be greater than zero. roles_claim = "realm_access.roles" admin_role = "openshell-admin" user_role = "openshell-user" @@ -235,7 +237,7 @@ The OpenTelemetry SDK logs export failures after startup. Spans in a failed batc Only OpenTelemetry traces are exported. Inbound gRPC and HTTP requests produce server spans named for the RPC or HTTP method. Store and compute-driver operations appear as child spans. Internal reconciliation, credential-refresh, and driver-watch loops create operation roots for their store work because no inbound request supplies a parent. The gateway continues valid W3C `traceparent` context and starts a new trace when none is supplied. Request spans carry `method`, `path`, and the `request_id` that also appears in gateway logs. Health endpoint spans use DEBUG level and are not exported by the default INFO filter. -The gateway forwards the OTLP configuration, configured gateway name, and W3C trace context to managed external drivers. Built-in drivers also export their spans to the same collector through dedicated in-process providers. Driver spans retain the gateway trace context, use a distinct service name such as `openshell-driver-docker` or `openshell-driver-podman`, and carry the gateway name as the `openshell.gateway.name` resource attribute. Operator-run external drivers own their own telemetry configuration. +The gateway forwards the OTLP configuration, configured gateway name, configured compute driver, and W3C trace context to managed external drivers. Built-in drivers also export their spans to the same collector through dedicated in-process providers. Driver spans retain the gateway trace context, use a distinct service name such as `openshell-driver-docker` or `openshell-driver-podman`, and carry the same `openshell.gateway.name` and `openshell.gateway.compute_driver` resource attributes as gateway spans. Compute-driver client and server spans use the same fully qualified protobuf operation name, such as `openshell.compute.v1.ComputeDriver/CreateSandbox`, in both the span name and `rpc.method`. The service name and span kind distinguish each side. Backend-prefixed child spans identify implementation work. A streaming watch records a terminal status when observed; consumer teardown without a terminal status leaves the span status unset. Operator-run external drivers own their own telemetry configuration. For Helm deployments, set `server.otlp.endpoint` to render this table. The optional `server.otlp.serviceName` value overrides the gateway service name; @@ -479,7 +481,7 @@ workspace_mode = "shared" # gateway_id = "openshell" namespace = "agents" service_account_name = "openshell-sandbox" -default_image = "ghcr.io/nvidia/openshell/sandbox:latest" +default_image = "ghcr.io/nvidia/openshell-community/sandboxes/base:latest" image_pull_policy = "IfNotPresent" image_pull_secrets = ["regcred"] # Defaults to the gateway version; override to pin a specific build. @@ -602,7 +604,7 @@ compute_drivers = ["docker"] [openshell.drivers.docker] socket_path = "/var/run/docker.sock" -default_image = "ghcr.io/nvidia/openshell/sandbox:latest" +default_image = "ghcr.io/nvidia/openshell-community/sandboxes/base:latest" # Docker vocabulary: Always | IfNotPresent | Never. Empty behaves like IfNotPresent. image_pull_policy = "IfNotPresent" sandbox_namespace = "docker-dev" @@ -646,7 +648,7 @@ compute_drivers = ["podman"] # asks the podman CLI where its socket is, and fails to start if neither finds # one. Set this to pin a specific Podman machine instead. socket_path = "/run/user/1000/podman/podman.sock" -default_image = "ghcr.io/nvidia/openshell/sandbox:latest" +default_image = "ghcr.io/nvidia/openshell-community/sandboxes/base:latest" image_pull_policy = "missing" # always | missing | never | newer grpc_endpoint = "https://host.containers.internal:17670" # The gateway overwrites gateway_port from bind_address at runtime. @@ -779,10 +781,10 @@ compute_drivers = ["vm"] state_dir = "/var/lib/openshell/vm" # Where the gateway looks for the openshell-driver-vm subprocess binary. driver_dir = "/usr/local/libexec/openshell" -default_image = "ghcr.io/nvidia/openshell/sandbox:latest" +default_image = "ghcr.io/nvidia/openshell-community/sandboxes/base:latest" grpc_endpoint = "https://host.containers.internal:17670" # Empty falls back to default_image. -bootstrap_image = "ghcr.io/nvidia/openshell/sandbox:latest" +bootstrap_image = "ghcr.io/nvidia/openshell-community/sandboxes/base:latest" krun_log_level = 1 vcpus = 2 mem_mib = 2048 @@ -794,8 +796,67 @@ guest_tls_key = "/var/lib/openshell/guest-tls/client-key.pem" # Defaults to 10001 when unset; matching GID is used if sandbox_gid is empty. # Any non-root Linux UID/GID is valid. # sandbox_uid = 20001 +# Corporate forward proxy for sandbox egress. The keys, their semantics, and +# the fail-closed contract are identical to the Podman driver above: only TLS +# (CONNECT) egress is chained, plain-HTTP destination requests always dial +# directly, credentials must come from proxy_auth_file rather than the URL, +# an http:// proxy with credentials requires proxy_auth_allow_insecure, and +# any present-but-invalid value is rejected at gateway startup rather than +# degrading to a direct dial. proxy_auth_file and proxy_ca_bundle are paths on +# the gateway host. +# +# The sandbox cannot select or override these settings. They reach the guest +# supervisor on its command line through a per-sandbox file the driver writes +# into the overlay upperdir on every launch, so a sandbox image cannot supply +# its own values or disable the operator's by baking a file at that path. +# +# Reachability: a proxy on the corporate network needs no special address and +# works on every VM sandbox. The guest's callback to the gateway is unaffected +# and never traverses the proxy. +# +# A proxy on the gateway host itself is reachable only from libkrun-backed +# (non-GPU) sandboxes: their egress leaves through gvproxy, which NATs +# 192.168.127.254 to the host's 127.0.0.1, so address it as +# http://host.openshell.internal: rather than http://127.0.0.1:. +# GPU sandboxes run on the QEMU/TAP backend, which has no such NAT — +# host.openshell.internal resolves to the TAP host address, and the driver's +# nftables rules let the guest reach only the gateway port on the host. A +# gateway-host proxy URL is therefore rejected when the sandbox launches on +# QEMU, rather than timing out on every CONNECT; give GPU sandboxes a proxy +# address routable from the guest's masqueraded egress. +# +# Because a microVM has no bind mounts or container secrets, the driver stages +# the credential and the CA into the per-sandbox overlay disk: the credential +# root-only inside the guest, and both removed with the sandbox. The +# credential is therefore at rest in that overlay image on the gateway host — +# the same delivery the per-sandbox gateway token already uses, and a +# difference from the Podman secret model worth noting when choosing where to +# keep proxy credentials. +# https_proxy = "http://host.openshell.internal:8080" +# no_proxy = "10.0.0.0/8,.internal.example" +# proxy_auth_file = "/etc/openshell/secrets/proxy-auth" +# proxy_auth_allow_insecure = true +# Last resort for hostname-filtering proxy ACLs; see the Podman section above. +# proxy_connect_by_hostname = true +# Corporate CA trusted for an https:// proxy and TLS-intercepting proxies. +# proxy_ca_bundle = "/etc/openshell/tls/proxy-ca.pem" +# Where the gateway stages rootfs tar archives for `--from ./rootfs.tar`. +# Defaults to /rootfs-tar-staging. The gateway creates one +# request-scoped subdirectory per staging slot and removes it after use. +# rootfs_tar_staging_dir = "/var/lib/openshell/vm/rootfs-tar-staging" +# Largest rootfs tar archive the driver accepts, in bytes. Defaults to 10 GiB. +# Gzip archives are decompressed while staging, and the limit also bounds the +# expanded tar. +# rootfs_tar_max_bytes = 10737418240 ``` +Rootfs tar staging requires the gateway and the VM driver to share a filesystem +and run as the same user. That holds for the managed VM driver, which the +gateway starts as a subprocess. If you point `compute_driver_endpoints` at an +externally managed `vm` socket owned by another user, the driver cannot read the +gateway's staging directory and rootfs tar sources fail with a +`FAILED_PRECONDITION` error; use a registry image reference instead. + ### Extension Driver Extension drivers run outside the gateway and expose the diff --git a/docs/reference/policy-schema.mdx b/docs/reference/policy-schema.mdx index 64942a384..8f7baa313 100644 --- a/docs/reference/policy-schema.mdx +++ b/docs/reference/policy-schema.mdx @@ -92,14 +92,16 @@ Configures [Landlock LSM](https://docs.kernel.org/security/landlock.html) enforc **Compatibility modes:** -| Value | Kernel ABI unavailable | Individual path inaccessible | All paths inaccessible | -|---|---|---|---| -| `best_effort` | Warns and continues without Landlock. | Skips the path, applies remaining rules. | Warns and continues without Landlock (refuses to apply an empty ruleset). | -| `hard_requirement` | Aborts sandbox startup. | Aborts sandbox startup. | Aborts sandbox startup. | +| Value | No paths configured | Kernel ABI unavailable | Individual path inaccessible | All paths inaccessible | +|---|---|---|---|---| +| `best_effort` | Landlock skipped (no-op). | Warns and continues without Landlock. | Skips the path, applies remaining rules. | Warns and continues without Landlock (refuses to apply an empty ruleset). | +| `hard_requirement` | Aborts sandbox startup. | Aborts sandbox startup. | Aborts sandbox startup, except in Kubernetes sidecar (current-user) mode where the inaccessible path is skipped. | Aborts sandbox startup. | `best_effort` (the default) is appropriate for most deployments. It handles missing paths gracefully. For example, `/app` might not exist in every container image but is included in the baseline path set for containers that do have it. Individual missing paths are skipped while the remaining filesystem rules are still enforced. -`hard_requirement` is for environments where any gap in filesystem isolation is unacceptable. If a listed path cannot be opened for any reason (missing, permission denied, symlink loop), sandbox startup fails immediately rather than running with reduced protection. +`hard_requirement` is for environments where any gap in filesystem isolation is unacceptable. If a listed path cannot be opened for any reason (missing, permission denied, symlink loop), sandbox startup fails immediately rather than running with reduced protection. Configuring `hard_requirement` with no filesystem paths is also a startup error. + +In Kubernetes sidecar (current-user) mode the sandbox cannot distinguish an intentionally denied path from a misconfigured one, so an individual inaccessible path is skipped and the remaining rules are applied instead of aborting. The other `hard_requirement` failures (kernel ABI unavailable, no paths configured, all paths inaccessible) still abort startup. When a path is skipped under `best_effort`, the sandbox logs a warning that includes the path, the specific error, and a human-readable reason (for example, "path does not exist" or "permission denied"). @@ -182,7 +184,8 @@ Each endpoint defines a reachable destination and optional inspection rules. | `persisted_queries` | string | No | GraphQL hash-only behavior for `protocol: graphql` and GraphQL-over-WebSocket operation policy. Default is `deny`; use `allow_registered` only with `graphql_persisted_queries`. | | `graphql_persisted_queries` | map | No | Trusted GraphQL persisted-query registry keyed by hash or saved-query ID. Values contain `operation_type`, optional `operation_name`, and optional root `fields`. | | `graphql_max_body_bytes` | integer | No | Maximum GraphQL-over-HTTP request body bytes buffered for inspection. Defaults to `65536`. | -| `mcp` | object | No | MCP endpoint options for `protocol: mcp`. MCP endpoints must set a concrete `host` and `port` or `ports`; `protocol: mcp` alone is invalid and is not treated as a wildcard endpoint. | +| `mcp` | object | No | MCP endpoint options for `protocol: mcp`. Omit this key to use all MCP endpoint defaults, including the exact `2025-11-25` revision; `mcp: null` is invalid. The object is rejected on other protocols. Every MCP endpoint must still set a concrete `host` and `port` or `ports`; an entry containing only `protocol: mcp` is invalid and is not treated as a wildcard endpoint. | +| `mcp.versions` | list of string | No | Nonempty allowlist of exact supported MCP core revisions: `2025-03-26`, `2025-06-18`, and `2025-11-25`. Omission resolves to the exact allowlist `["2025-11-25"]`; it never means latest or all known revisions. The key must be absent to use this default; `versions: null` and `versions: []` are invalid. Values must be unique, contain no extra whitespace, and name a revision in the closed supported set. For every request except a valid standalone `initialize`, OpenShell selects the revision from one `MCP-Protocol-Version` header or, when the header is absent, the MCP specification's `2025-03-26` compatibility fallback. The selected revision must appear in this allowlist. Duplicate, empty, or unsupported header values receive `400 Bad Request`; a supported revision outside the allowlist receives `403 Forbidden`. The sessionless `2026-07-28` revision is not yet supported. For an unsupported revision, omit `protocol` and `mcp` only when deliberate uninspected L4 passthrough is an acceptable weaker boundary. | | `mcp.max_body_bytes` | integer | No | Maximum MCP JSON-RPC-over-HTTP request body bytes buffered for inspection. Defaults to `65536`. | | `mcp.strict_tool_names` | bool | No | Defaults to `true`. Requires `tools/call` `params.name` values to match `^[A-Za-z0-9_.-]{1,128}$` before policy evaluation. Set to `false` only for compatibility with MCP servers that intentionally use non-recommended tool names. Wildcard `tool` matchers require this to remain enabled. | | `mcp.allow_all_known_mcp_methods` | bool | No | Defaults to `false`. When `true`, enables the endpoint MCP method profile: omitted `rules` allow all MCP-family methods and all tools before `deny_rules`, and omitted rule `method` uses that profile. When unset or `false`, explicit MCP method rules are required; rules with `tool` or `params.name` must set `method: tools/call`. | @@ -193,12 +196,15 @@ Each endpoint defines a reachable destination and optional inspection rules. - `access` and `rules` are mutually exclusive; setting both is rejected. - `protocol: tcp` requires a valid DNS hostname. Hostless `allowed_ips`, IP-literal hosts, trailing-dot names, and malformed DNS selectors are rejected with a policy-validation error. - `protocol: tcp` requires at least one port and rejects L7-only fields, including `path`, `enforcement`, `access`, `rules`, `deny_rules`, request rewriting and credential signing fields, and GraphQL, JSON-RPC, or MCP options. +- A `protocol: tcp` hostname constrains connection routing, not application authority. OpenShell does not inspect TLS SNI, HTTP `Host`, or another protocol-level destination in the stream. Compatible shared infrastructure can therefore expose other tenants, virtual hosts, or services behind an allowed hostname. - A sandbox runtime must support policy DNS and transparent TCP capture before it can activate a policy containing `protocol: tcp`. Docker and Podman provide this runtime support. - Adding the first `protocol: tcp` endpoint to a running sandbox that started without one is rejected atomically because its DNS and capture substrate is startup infrastructure. Recreate the sandbox with a TCP endpoint. A sandbox that started with the substrate can remove and re-add TCP endpoints dynamically. +- Policy-advisor agent proposals cannot request `protocol: tcp` or `tls: skip`. Add native TCP or raw TLS access through an administrator-authored policy. Agent proposals may omit `protocol` to use the explicit proxy with its default TLS termination and HTTP authority checks. - When `protocol` is set, at least one of `access` or `rules` is required for `rest`, `websocket`, `graphql`, and `sql`. - `mcp` and `json-rpc` reject `access` presets; use explicit `rules`. - `json-rpc` requires explicit `rules` with `allow.method`. - `mcp` requires `rules` unless `mcp.allow_all_known_mcp_methods: true`. +- YAML defaulting requires the `mcp` or `mcp.versions` key to be absent. Explicit `mcp: null`, `versions: null`, and `versions: []` values are rejected. At protobuf ingress, an empty repeated `versions` field means omission and resolves to `["2025-11-25"]` because protobuf repeated fields do not preserve field presence. - `deny_rules` require `protocol`. For non-MCP protocols, `deny_rules` also require `rules` or `access` to define the base allow set. MCP `deny_rules` may omit both when `mcp.allow_all_known_mcp_methods: true` supplies the base allow set. - `rules: []` (empty list) is rejected; use `access: full` or remove `rules`. - Non-empty `rules` must contain at least one effective allow clause; rules where every entry lacks an allow are rejected as deny-all. @@ -216,7 +222,7 @@ global policies that use this field. Network policy admission does not expand the credential boundary unless the endpoint explicitly supplies this binding. OpenShell rejects a request mismatch with HTTP 403 and `credential_endpoint_mismatch`. Refer to [Static Credential Endpoint -Binding](/sandboxes/providers-v2#understand-static-credential-endpoint-binding). +Binding](/providers/profiles#understand-static-credential-endpoint-binding). This example allows the sandbox to reach Google Cloud Storage and binds the static credentials from the attached `work-gcp` provider to that endpoint: @@ -340,7 +346,7 @@ Do not combine `method`, `path`, or `query` with `operation_type`, `operation_na ##### MCP Allow And Deny Rules (`protocol: mcp`) -MCP rules match sandbox-to-server MCP Streamable HTTP request bodies by MCP method and optional tool selectors. OpenShell parses the underlying JSON-RPC 2.0 envelope, validates known MCP request and notification params, and preserves unknown extension methods as policy-addressable literal method strings. Until OpenShell exposes explicit MCP version profiles, `mcp.allow_all_known_mcp_methods` defaults to `false`, so endpoints require explicit MCP method rules. Set `mcp.allow_all_known_mcp_methods: true` to enable the endpoint method profile; in that mode, rules can omit `method`, and tool selectors are normalized to `tools/call` internally. By default, `tools/call` `params.name` must match the MCP-recommended tool-name pattern `^[A-Za-z0-9_.-]{1,128}$`; configure `mcp.strict_tool_names: false` on the endpoint only to allow a server that intentionally uses names outside that pattern. Wildcard `tool` matchers require `mcp.strict_tool_names` to remain enabled. JSON-RPC responses and server-to-client MCP messages on response bodies or SSE streams are relayed but are not currently parsed for policy enforcement. +MCP rules match sandbox-to-server MCP Streamable HTTP request bodies by MCP method and optional tool selectors. OpenShell parses the underlying JSON-RPC 2.0 envelope, validates known MCP request and notification params, and preserves unknown extension methods as policy-addressable literal method strings. An endpoint that omits `mcp.versions`, including one that omits the entire `mcp` object, immediately resolves to the exact `2025-11-25` allowlist. Canonical serialization and stored policy data contain that explicit materialized list, so adding another supported revision cannot widen the normalized policy. For every request except a valid standalone `initialize`, OpenShell checks exactly one `MCP-Protocol-Version` value against the allowlist before policy evaluation and repeats the check after middleware changes the request. An absent header selects the `2025-03-26` compatibility fallback; it does not select the policy default. The check stores no connection or session state. The current parser remains version-independent and does not yet apply the batch rules recorded in each revision's wire profile. `mcp.allow_all_known_mcp_methods` defaults to `false`, so endpoints require explicit MCP method rules. Set it to `true` to enable the endpoint method profile; in that mode, rules can omit `method`, and tool selectors are normalized to `tools/call` internally. By default, `tools/call` `params.name` must match the MCP-recommended tool-name pattern `^[A-Za-z0-9_.-]{1,128}$`; configure `mcp.strict_tool_names: false` on the endpoint only to allow a server that intentionally uses names outside that pattern. Wildcard `tool` matchers require `mcp.strict_tool_names` to remain enabled. JSON-RPC responses and server-to-client MCP messages on response bodies or SSE streams are relayed but are not currently parsed for policy enforcement. Use `rules` for MCP allow rules and `deny_rules` for MCP deny rules. Deny rules take precedence over allow rules. If an MCP endpoint sets `mcp.allow_all_known_mcp_methods: true` and omits `rules`, OpenShell allows all MCP-family methods and all tools, then applies any `deny_rules`. Otherwise, the endpoint must define explicit rules. A broad allow or deny rule whose method matcher includes `tools/call` cannot be combined with tool-specific allow rules because it would bypass or erase the tool filter; add `tool` or `params.name` to scope `tools/call`, or remove the tool-specific rules. In a batch request, one denied call denies the full batch. @@ -380,6 +386,15 @@ endpoints: tool: execute_code ``` +This example omits `mcp.versions`, so OpenShell materializes `["2025-11-25"]`. To authorize an intentional compatibility range for an older server, set an explicit nonempty allowlist: + +```yaml showLineNumbers={false} +mcp: + versions: ["2025-03-26", "2025-11-25"] +``` + +OpenShell canonicalizes an explicit list in semantic order. Later runtime negotiation selects one allowed revision and applies only that profile; it does not combine the profiles. + ##### JSON-RPC Allow Rule (`protocol: json-rpc`) JSON-RPC allow rules match sandbox-to-server JSON-RPC-over-HTTP request objects by RPC method. They apply to single JSON-RPC requests and batch requests. For a batch, OpenShell evaluates each call independently. Client-to-server JSON-RPC response frames in POST bodies are denied. Server-to-client messages on HTTP response bodies or MCP SSE streams are relayed but are not currently parsed for policy enforcement. diff --git a/docs/reference/sandbox-compute-drivers.mdx b/docs/reference/sandbox-compute-drivers.mdx index 03da703ee..e46ebf8a2 100644 --- a/docs/reference/sandbox-compute-drivers.mdx +++ b/docs/reference/sandbox-compute-drivers.mdx @@ -3,14 +3,21 @@ # SPDX-License-Identifier: Apache-2.0 title: "Sandbox Compute Drivers" sidebar-title: "Compute Drivers" -description: "Reference for Docker, Podman, MicroVM, and Kubernetes sandbox compute drivers." -keywords: "Generative AI, Cybersecurity, AI Agents, Sandboxing, Docker, Podman, MicroVM, Kubernetes, Reference" +description: "Reference for Docker, Podman, MicroVM, Kubernetes, and Windows MXC sandbox compute drivers." +keywords: "Generative AI, Cybersecurity, AI Agents, Sandboxing, Docker, Podman, MicroVM, Kubernetes, MXC, Reference" position: 4 --- The gateway's configured compute driver determines how OpenShell creates each sandbox. The CLI workflow stays the same across drivers: you create, connect to, inspect, stop, start, and delete sandboxes through the gateway API. -Every compute driver runs the OpenShell supervisor inside the sandbox workload. The supervisor launches the agent process, applies policy, routes egress through the proxy, injects configured credentials, and maintains the gateway session. +Most compute drivers run the OpenShell supervisor inside the sandbox workload. +The supervisor launches the agent process, applies policy, routes egress through +the proxy, injects configured credentials, and maintains the gateway session. +A driver may instead set `driver_reports_runtime_readiness`. In that mode, +driver-reported readiness does not require a supervisor session. The canonical +create-time policy is part of `DriverSandboxSpec`; a driver that enforces policy +outside the standard supervisor fetches later revisions through the existing +sandbox configuration API. The Windows MXC driver reports its own readiness. Stop stops compute but retains the sandbox record and the driver's persistent workspace boundary. Start reactivates the same driver resource. @@ -42,11 +49,12 @@ Configure the compute driver on the gateway. Current releases accept one driver compute_drivers = ["docker"] ``` -Reserved built-in values are `docker`, `podman`, `kubernetes`, and `vm`. +Reserved built-in values are `docker`, `podman`, `kubernetes`, `vm`, and `mxc`. +The `mxc` driver is available only in native Windows gateway builds. Non-reserved names select an extension driver and require a `socket_path` in `[openshell.drivers.]`. -When `compute_drivers` is unset, the gateway auto-detects Kubernetes, then Podman, then Docker. Local container runtimes must respond to an API probe before the gateway selects them. The VM driver is never auto-detected; configure it explicitly with `compute_drivers = ["vm"]` or set `OPENSHELL_DRIVERS=vm` in the launch environment. +When `compute_drivers` is unset, the gateway auto-detects Kubernetes, then Podman, then Docker. Docker must respond on a known API socket. Podman first probes known API sockets and then asks the `podman` CLI for the active native or machine-backed socket. The VM driver is never auto-detected; configure it explicitly with `compute_drivers = ["vm"]` or set `OPENSHELL_DRIVERS=vm` in the launch environment. Common gateway options: @@ -125,6 +133,20 @@ request. The VM driver accepts `gpu_device_ids`, for example accepts at most one entry and allows either `--gpu` or `--gpu 1` when `gpu_device_ids` is set. +## Resource Capability Reporting + +API clients can read each configured driver's static resource request support +from the protected `GetGatewayInfo` response. The capability response describes +which request forms a driver implements. It does not report live resource +inventory, availability, or scheduling capacity, so a supported request can +still fail when the selected runtime cannot provide the resource. + +CPU and memory capabilities report whether the driver enforces a resource +limit. GPU capabilities report whether the driver accepts a default GPU request +and an explicit GPU count. An omitted `resource_capabilities` message, or an +omitted CPU, memory, or GPU capability, means the driver did not report that +capability. A reported `false` value means that request form is unsupported. + For Kubernetes, `pod.runtime_class_name` maps to PodSpec `runtimeClassName`. It overrides the gateway's configured default runtime class for that sandbox, while a typed `SandboxTemplate.runtime_class_name` value from the API still @@ -135,7 +157,7 @@ the gateway. If the primary listener covers that address, the gateway reuses it and sandbox JWT authentication restricts the supervisor to its callback RPC allowlist. If the primary listener is not reachable through that address, the gateway creates an additional callback-only listener. Use the primary endpoint -for CLI, administrator, health, reflection, inference-route management, and +for CLI, administrator, health, reflection, provider management, and HTTP requests. A `PermissionDenied` response from an additional callback-only listener is expected for those requests. Do not broaden the primary listener to `0.0.0.0` solely to make sandbox callbacks reachable. @@ -144,7 +166,7 @@ to `0.0.0.0` solely to make sandbox callbacks reachable. [Docker](https://www.docker.com/get-started/)-backed sandboxes run as containers on the gateway host. Use Docker for local development, single-machine gateways, and hosts that already use Docker Desktop or Docker Engine. -The gateway talks to the Docker daemon to create sandbox containers. Docker is also required for local image builds from directories or Dockerfiles. +The gateway talks to the Docker daemon to create sandbox containers. Docker Desktop and compatible macOS runtimes route `host.openshell.internal` through an IPv4 host-gateway alias. The gateway reuses an IPv4 primary listener @@ -227,12 +249,32 @@ namespace roots. These checks do not make host bind mounts safe. [Podman](https://podman.io/)-backed sandboxes run as rootless containers on the gateway host. Use Podman for Linux workstation workflows that avoid a rootful Docker daemon. -The gateway talks to the Podman API socket. The Podman driver requires Podman 5.x, cgroups v2, rootless networking, and an active Podman user socket. When `socket_path` is not set, the driver probes for a responsive Podman socket and fails to start if none respond. +The gateway talks to the Podman API socket. The Podman driver requires Podman 5.x, cgroups v2, rootless networking, and an active Podman user socket. When `socket_path` is not set, the driver probes known socket paths, then uses the `podman` CLI to resolve the active native or machine-backed connection. It fails to start if neither method finds a socket. For maintainer-level implementation details, refer to the [Podman driver README](https://github.com/NVIDIA/OpenShell/blob/main/crates/openshell-driver-podman/README.md) and [Podman networking notes](https://github.com/NVIDIA/OpenShell/blob/main/crates/openshell-driver-podman/NETWORKING.md). Select Podman with `compute_drivers = ["podman"]` in `[openshell.gateway]`. Configure Podman driver values such as `socket_path`, `network_name`, `supervisor_image`, `stop_timeout_secs`, `image_pull_policy`, `grpc_endpoint`, `host_gateway_ip`, `sandbox_ssh_socket_path`, `sandbox_pids_limit`, and `guest_tls_*` in `[openshell.drivers.podman]`. +### macOS Podman Socket Path + +On macOS, Homebrew-installed Podman does not create the default socket path +that the driver probes (`~/.local/share/containers/podman/machine/podman.sock`). +The actual API socket lives under `/var/folders/` in a path that macOS can +rotate after a reboot. + +If the gateway fails with `Podman socket not found; is podman machine running?` +while `podman machine list` shows a running machine, set the +`OPENSHELL_PODMAN_SOCKET` environment variable to the dynamic socket path: + +```shell +export OPENSHELL_PODMAN_SOCKET="$(podman machine inspect --format '{{.ConnectionInfo.PodmanSocket.Path}}')" +``` + +Add this to your shell profile or gateway launch environment so it resolves +correctly after each reboot. Alternatively, set `socket_path` in +`[openshell.drivers.podman]` to the current path, but note that the path may +change when macOS rotates `/var/folders/`. + Podman sandboxes default to a 45-second graceful stop window before Podman escalates from `SIGTERM` to `SIGKILL`. Set `stop_timeout_secs` in gateway config, or `OPENSHELL_STOP_TIMEOUT` for the standalone driver, when a local runtime needs a different teardown window. Stop stops the existing Podman container while retaining its named workspace @@ -340,7 +382,7 @@ The gateway starts `openshell-driver-vm` over a private Unix socket and passes i ### Local image resolution -The VM driver resolves sandbox images from a local container engine before falling back to registry pulls. It tries Docker first, then falls back to the Podman socket (Docker-compatible API). On Linux with Podman, enable the API socket so the driver can find local images: +The VM driver resolves sandbox images from a local container engine before falling back to registry pulls. It tries Docker first, then uses the same Podman socket discovery as the Podman driver. On Linux with Podman, enable the API socket so the driver can find local images: ```shell systemctl --user start podman.socket @@ -352,6 +394,18 @@ The VM driver creates nftables rules on the host for each sandbox VM's TAP netwo On hosts with restrictive firewalls (e.g. firewalld), the host firewall may additionally block VM traffic that the driver's rules accept. If VM sandboxes cannot reach the network, verify that the host firewall allows forwarding and input for `vmtap-*` interfaces. See the [VM driver README](https://github.com/NVIDIA/OpenShell/blob/main/crates/openshell-driver-vm/README.md#host-side-nftables-rules) for details. +### Corporate Proxy Egress + +For proxy-required networks, the VM driver accepts the same corporate egress proxy keys as the Podman driver: `https_proxy`, `no_proxy`, `proxy_auth_file`, `proxy_auth_allow_insecure`, `proxy_connect_by_hostname`, and `proxy_ca_bundle`. The in-guest supervisor chains policy-approved TLS tunnels through the proxy with HTTP CONNECT instead of dialing destinations directly. + +The settings reach the guest supervisor on its command line through a per-sandbox argument file the driver writes into the overlay upperdir on every launch, so a sandbox cannot select, alter, or disable the proxy from inside the guest — including through image `ENV`, the sandbox environment, or files baked into the image at the paths the driver uses. + +A proxy on the corporate network needs no special address and works on every VM sandbox. The guest's callback to the gateway never traverses the proxy. + +A proxy on the gateway host itself works only for libkrun-backed (non-GPU) sandboxes, whose egress leaves through gvproxy: configure `https_proxy = "http://host.openshell.internal:"` rather than a `127.0.0.1` URL, because gvproxy NATs that alias to the host's `127.0.0.1`. GPU sandboxes use the QEMU/TAP backend, where `host.openshell.internal` resolves to the TAP host address and the driver's [host firewall rules](#host-firewall) allow the guest to reach only the gateway port on the host. The driver rejects a gateway-host proxy URL when a sandbox launches on QEMU instead of letting every CONNECT time out, so give GPU sandboxes a proxy address routable from the guest's masqueraded egress. + +Because a microVM has no bind mounts or container secrets, the driver stages the credential (root-only) and the CA bundle into the per-sandbox overlay disk and removes them with the sandbox. See the [Gateway Configuration File](./gateway-config) reference for the full contract, including the cleartext-credential acknowledgement and the validated-IP CONNECT behavior. + ## Kubernetes Driver Kubernetes-backed sandboxes run as pods in the configured sandbox namespace. Use Kubernetes for shared clusters, remote compute, GPU scheduling, and operator-managed environments. @@ -436,7 +490,9 @@ The Kubernetes driver creates namespaced `agents.x-k8s.io` `Sandbox` resources f Stop patches the existing resource rather than deleting it. For `v1beta1`, the driver sets `spec.operatingMode` to `Suspended` or `Running`. For `v1alpha1`, it sets `spec.replicas` to `0` or `1`. The Sandbox resource and its -workspace PVC keep their identity across both operations. +workspace PVC keep their identity across both operations. Stop returns only +after the controller reports suspension and deletes the old pod, so an +immediate start cannot race the prior pod's termination. If Agent Sandbox is upgraded in place, restart the OpenShell gateway after the controller and CRD rollout completes so the gateway can detect the served API versions again. diff --git a/docs/sandboxes/inference-routing.mdx b/docs/sandboxes/inference-routing.mdx index 2c065c6cb..902c09eea 100644 --- a/docs/sandboxes/inference-routing.mdx +++ b/docs/sandboxes/inference-routing.mdx @@ -1,339 +1,311 @@ --- # SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. # SPDX-License-Identifier: Apache-2.0 -title: "Inference Routing" -sidebar-title: "Inference Routing" -description: "Understand and configure OpenShell inference routing through inference.local and external endpoints." -keywords: "Generative AI, Cybersecurity, Inference Routing, Configuration, Privacy, LLM, Provider" +title: "Provider-backed Inference" +sidebar-title: "Provider-backed Inference" +description: "Grant a sandbox access to model providers through provider profiles, attachments, and native endpoints." +keywords: "Generative AI, Cybersecurity, Inference, Provider Profiles, Credentials, LLM" position: 8 --- -OpenShell handles inference traffic through two paths: external endpoints and `inference.local`. +OpenShell grants model-provider access through provider profiles and sandbox +attachments. The workload calls the provider's native API. OpenShell evaluates +the request against the profile-derived network policy and substitutes the real +credential only at an endpoint authorized by that profile. -| Path | How it works | +This keeps the complete access contract in one place: + +| Concern | Owner | |---|---| -| External endpoints | Traffic to hosts like `api.openai.com` or `api.anthropic.com` is treated like any other outbound request, allowed or denied by `network_policies`. Refer to [Policies](/sandboxes/policies). | -| `inference.local` | A sandbox-local HTTPS endpoint that routes model requests through the gateway. The privacy router strips sandbox-supplied credentials, forwards only approved inference headers, injects the configured backend credentials, and forwards to the managed model endpoint. | +| Credential and refresh lifecycle | Provider instance | +| Authorized hosts, ports, paths, and binaries | Provider profile | +| Which workload receives access | Sandbox provider attachment | +| Base URL, model, request shape, and timeout | Native client or workload configuration | -## How `inference.local` Works +## Define and Attach a Hosted Provider -When code inside a sandbox calls `https://inference.local`, the privacy router routes the request to the configured backend for that gateway. The configured model is applied to generation requests, provider credentials come from OpenShell rather than from code inside the sandbox, and only approved inference headers are forwarded upstream. +Start from an existing profile, review its access, and import it under a new +ID. For example, export the NVIDIA profile: -If code calls an external inference host directly, OpenShell evaluates that traffic only through `network_policies`. +```shell +openshell provider profile export nvidia -o yaml > nvidia-native.yaml +``` -| Property | Detail | -|---|---| -| Credentials | No sandbox API keys needed. Credentials come from the configured provider record. The router strips caller-supplied `Authorization` before forwarding the request. | -| Header forwarding | `inference.local` forwards only a per-provider header allowlist. OpenAI routes allow `openai-organization` and `x-model-id`. Anthropic routes allow `anthropic-version` and `anthropic-beta`. Vertex Claude rawPredict routes strip `anthropic-beta` and do not forward `anthropic-version` as a header because the router injects `anthropic_version` into the Vertex request body. NVIDIA routes allow `x-model-id`. AWS Bedrock routes have no passthrough headers today. All other caller headers are stripped. | -| Configuration | One provider and one model define sandbox inference for the active gateway. Every sandbox on that gateway sees the same `inference.local` backend. | -| Provider support | NVIDIA, Anthropic, Google Vertex AI, AWS Bedrock (via a translating bridge — direct AWS with SigV4 signing is a separate follow-up), and any OpenAI-compatible provider all work through the same endpoint. Vertex routes Claude models through `/v1/messages` and non-Anthropic models through `/v1/chat/completions`. The gateway resolves the upstream Vertex host from the provider config, including regional, global, and supported multi-region endpoints. | -| Streaming reliability | The router tolerates idle gaps of up to 120 seconds between streamed chunks so long reasoning responses are not cut off mid-stream. | -| Hot refresh | OpenShell picks up provider credential changes and inference updates without recreating sandboxes. Changes propagate within about 5 seconds by default. | +In `nvidia-native.yaml`, change `id` to `nvidia-native`, give the profile a +distinct `display_name`, and set `binaries` to the paths that may call the API. +Keep the credential and endpoint definitions you intend to grant. For a Python +workload, the edited fields can look like: -## Supported API Patterns +```yaml +id: nvidia-native +display_name: NVIDIA Native API +binaries: + - /usr/bin/python3 + - /usr/bin/python3.13 + - /usr/local/bin/python + - /sandbox/.venv/** +``` -Supported request patterns depend on the provider configured for `inference.local`. +Lint and import the complete edited profile, then create a provider from its +new ID: - - +```shell +openshell provider profile lint -f nvidia-native.yaml +openshell provider profile import -f nvidia-native.yaml -| Pattern | Method | Path | -|---|---|---| -| Chat Completions | `POST` | `/v1/chat/completions` | -| Completions | `POST` | `/v1/completions` | -| Responses | `POST` | `/v1/responses` | -| Embeddings | `POST` | `/v1/embeddings` | -| Model Discovery | `GET` | `/v1/models` | -| Model Discovery | `GET` | `/v1/models/*` | +openshell provider create \ + --name nvidia-prod \ + --type nvidia-native \ + --from-existing - +openshell sandbox create \ + --name inference-demo \ + --provider nvidia-prod \ + -- python app.py +``` - +The imported provider profile supplies `NVIDIA_API_KEY` as an opaque placeholder and +allows the profile's native endpoint. Configure the client with the real model +identifier: -| Pattern | Method | Path | -|---|---|---| -| Messages | `POST` | `/v1/messages` | +```python +import os +from openai import OpenAI - +client = OpenAI( + base_url="https://integrate.api.nvidia.com/v1", + api_key=os.environ["NVIDIA_API_KEY"], + timeout=300, +) - +response = client.responses.create( + model="nvidia/nemotron-3-nano-30b-a3b", + input="Hello!", +) +print(response.output_text) +``` -| Pattern | Method | Path | -|---|---|---| -| InvokeModel | `POST` | `/model/{modelId}/invoke` | +Use the same export, edit, and import flow for OpenAI, Anthropic, or another +hosted service. Set a new profile ID, retain only the endpoints and credentials +the workload needs, and name the actual client binaries. Provider attachment +does not select or rewrite a model. -The `{modelId}` segment is constrained to a single non-empty path segment to avoid path-traversal liabilities. `/model//invoke` and `/model/a/b/invoke` both no-match. +This release still loads built-in profiles for compatibility, so existing +providers continue to resolve their profiles. Treat those built-ins as starting +templates for new provider definitions rather than the primary setup workflow. - -Today the `aws-bedrock` provider type is bridge-fronted only. The router does not inject any auth header on outbound requests; the configured `BEDROCK_BASE_URL` is expected to point at a translating bridge or Bedrock-compatible proxy whose own pod holds operator-side credentials. SigV4 signing for direct AWS Bedrock is deferred to a follow-up release. +## Attach a Provider to a Running Sandbox -`InvokeModelWithResponseStream` is intentionally not advertised yet. The streaming path emits AWS event-stream framing, which our protocol-aware error path does not yet model; surfacing it without that work risks shipping responses the sandbox cannot interpret on failure. It will land alongside the streaming-error work in a follow-up. - +```shell +openshell sandbox provider attach inference-demo nvidia-prod +openshell sandbox provider list inference-demo +``` - - +Running sandboxes poll for provider and effective-policy changes. Launch a new +process after attachment so it receives the new credential placeholder: -Requests to `inference.local` that do not match the configured provider's supported patterns are denied. +```shell +openshell sandbox exec inference-demo -- python app.py +``` -Google Vertex AI does not expose every OpenAI-compatible path through `inference.local`. Vertex routes for Gemini and other non-Anthropic models currently support Chat Completions. Vertex routes for Claude models use the Anthropic Messages pattern. Base URL overrides are only supported for non-Anthropic Vertex routes. +An already-running process does not gain new environment variables. Credential +rotation and detach still take effect immediately at placeholder resolution. -## Configure Inference Routing +Detach the provider to revoke its policy and credential access: -The managed local inference endpoint uses three values: +```shell +openshell sandbox provider detach inference-demo nvidia-prod +``` -| Value | Description | -|---|---| -| Provider record | The credential backend OpenShell uses to authenticate with the upstream model host. | -| Model ID | The model to use for generation requests. | -| Timeout | Per-request timeout in seconds for upstream inference calls. Defaults to 60 seconds. | +## Use a Custom or Self-hosted Endpoint -For tested providers and base URLs, refer to [Supported Inference Providers](/sandboxes/manage-providers#supported-inference-providers). +Do not reuse the built-in `openai` or `anthropic` profile for an alternate +host. Those profiles bind credentials to the public vendor endpoints. Import a +profile that names the intended endpoint and the binaries that may call it. -## Create a Provider +For a credentialless Ollama server on the gateway host, save this profile as +`ollama-openai.yaml`: -Create a provider that holds the backend credentials you want OpenShell to use. +```yaml +id: ollama-openai +display_name: Ollama +description: Host-local Ollama OpenAI-compatible API +category: inference +inference_capable: true +credentials: [] +endpoints: + - host: host.openshell.internal + port: 11434 + protocol: rest + access: read-write + enforcement: enforce +binaries: + - /usr/bin/curl + - /usr/local/bin/curl + - /usr/bin/python3 + - /usr/local/bin/python + - /sandbox/.uv/python/** + - /sandbox/.venv/** +``` - - +Import the profile, create an instance, and attach it: + +```shell +openshell provider profile lint -f ollama-openai.yaml +openshell provider profile import -f ollama-openai.yaml +openshell provider create --name ollama --type ollama-openai + +openshell sandbox create \ + --name ollama-client \ + --provider ollama \ + --env OPENAI_BASE_URL=http://host.openshell.internal:11434/v1 \ + -- python app.py +``` + +Use any non-empty placeholder value if the client library requires an API key +for a server that does not authenticate requests: + +```python +import os +from openai import OpenAI + +client = OpenAI( + base_url=os.environ["OPENAI_BASE_URL"], + api_key="unused", +) +response = client.chat.completions.create( + model="qwen3.5:0.8b", + messages=[{"role": "user", "content": "Hello"}], +) +``` + +For an authenticated alternate endpoint, declare a credential in the custom +profile, bind it to that endpoint, and create the provider from the original +credential source. OpenShell never exports stored credential values. + +## Verify Access + +Inspect the effective policy, including provider-derived entries: + +```shell +openshell policy get inference-demo --full +``` + +Then run a native request from a new sandbox process. A successful request +confirms endpoint policy, binary attribution, credential substitution, DNS, +and upstream service behavior. + +If the request is denied: + +- Confirm the provider is attached with `openshell sandbox provider list`. +- Confirm the caller binary and native endpoint appear in + `openshell policy get --full`. +- If the provider was attached after the process started, launch a new process. +- If the sandbox uses a gateway global policy override, add the native endpoint + there because a global override suppresses provider-derived policy layers. +- Inspect logs for `credential_endpoint_mismatch`. That error means policy + admitted the request but the provider profile did not authorize its + credential at the requested endpoint. + +## Migrate from Managed Inference Routes + +OpenShell removed the workspace-global managed inference route and the +`openshell inference` commands. Upgrades remove stored route records. Provider +records, provider refresh configuration, and existing sandbox attachments are +preserved. Built-in profiles remain available during the transition, but new +setups should import an explicit profile under a new ID. + +The old route cannot be converted automatically. It applied one provider and +model to every sandbox in a workspace, while provider attachments intentionally +grant access to selected sandboxes. OpenShell cannot infer which sandboxes +should receive that authority. + +### Now + +Earlier releases configured one shared route: ```shell openshell provider create --name nvidia-prod --type nvidia --from-existing -``` - -This reads `NVIDIA_API_KEY` from your environment. - - - - - -Any cloud provider that exposes an OpenAI-compatible API works with the `openai` provider type. You need three values from the provider: the base URL, an API key, and a model name. - -```shell -openshell provider create \ - --name my-cloud-provider \ - --type openai \ - --credential OPENAI_API_KEY= \ - --config OPENAI_BASE_URL=https://api.example.com/v1 -``` - -Replace the base URL and API key with the values from your provider. For supported providers out of the box, refer to [Supported Inference Providers](/sandboxes/manage-providers#supported-inference-providers). For other providers, refer to your provider's documentation for the correct base URL, available models, and API key setup. - - - - - -```shell -openshell provider create \ - --name vertex-local \ - --type google-vertex-ai \ - --from-gcloud-adc \ - --config VERTEX_AI_PROJECT_ID=my-gcp-project \ - --config VERTEX_AI_REGION=us-central1 -``` - -Use [Google Vertex AI](/providers/google-vertex-ai) for the full auth flows, including the production service-account refresh path, ADC-backed providers that mint `GOOGLE_VERTEX_AI_TOKEN`, and `--from-existing` support. - - - - - -```shell -openshell provider create \ - --name my-local-model \ - --type openai \ - --credential OPENAI_API_KEY=empty-if-not-required \ - --config OPENAI_BASE_URL=http://host.openshell.internal:11434/v1 -``` - -Use `--config OPENAI_BASE_URL` to point to any OpenAI-compatible server running where the gateway runs. For host-backed local inference, use `host.openshell.internal` or the host's LAN IP. Avoid `127.0.0.1` and `localhost`. Set `OPENAI_API_KEY` to a dummy value if the server does not require authentication. - - -For a self-contained setup, the Ollama sandbox bundles Ollama inside the sandbox itself, so no host-level provider is needed. Refer to [Inference Ollama](/get-started/tutorials/inference-ollama) for details. - - - -Ollama also supports cloud-hosted models using the `:cloud` tag suffix, for example `qwen3.5:cloud`. - - - - - -```shell -openshell provider create --name anthropic-prod --type anthropic --from-existing -``` - -This reads `ANTHROPIC_API_KEY` from your environment. - - - - - -```shell -openshell provider create \ - --name bedrock-bridge \ - --type aws-bedrock \ - --credential AWS_ACCESS_KEY_ID=unused-bridge-fronted-shape \ - --config BEDROCK_BASE_URL=http://your-bedrock-bridge.your-ns.svc.cluster.local:8080 -``` - -Then set the inference route, passing `--no-verify` because the validation probe does not yet support Bedrock protocols: - -```shell openshell inference set \ - --provider bedrock-bridge \ - --model anthropic.claude-3-5-sonnet-20241022-v2:0 \ - --no-verify + --provider nvidia-prod \ + --model nvidia/nemotron-3-nano-30b-a3b \ + --timeout 300 ``` -**Why a placeholder credential?** `provider create` requires a non-empty `credentials` map even when the upstream auth scheme is `AuthHeader::None` — `aws-bedrock` falls into that bucket today because the router never injects a credential header on outbound requests; the bridge holds operator-side auth in its own pod. Any non-empty string value satisfies the structural requirement; `unused-bridge-fronted-shape` makes the intent obvious in `openshell provider get` output. The same pattern applies to any standalone-router profile that registers `AuthHeader::None`. When the SigV4 follow-up lands and the router begins signing requests itself, this becomes a real key. +Clients called `https://inference.local`, supplied a placeholder key and model, +and relied on OpenShell to rewrite the request. -**About the bridge-fronted shape.** The router does not inject any auth header on outbound requests. Point `BEDROCK_BASE_URL` at a translating bridge or Bedrock-compatible proxy that handles authentication in its own pod. The bridge is expected to accept Bedrock InvokeModel requests on the patterns listed above and forward to the operator's real upstream. +### After -**About `--no-verify`.** The default validation probe does not yet recognize the `aws_bedrock_invoke` protocol, so without `--no-verify` the `inference set` call would fail before it could mint a route. The first sandbox round-trip is the real verification today. - -**For direct AWS Bedrock**, refer to a future release that adds the SigV4 router-side signer. Until then, a `BEDROCK_BASE_URL` is required at provider-create time — the core profile sets `default_base_url: ""`, so route resolution rejects providers without it rather than silently forwarding prompts to AWS with no usable auth. - - - - -## Set Inference Routing - -Point `inference.local` at that provider and choose the model to use: +Export the old provider type's profile, edit its ID and access contract, import +it, and create a replacement provider from the original credential source: ```shell -openshell inference set \ - --provider nvidia-prod \ - --model nvidia/nemotron-3-nano-30b-a3b +openshell provider profile export nvidia -o yaml > nvidia-native.yaml +# Edit id, display_name, endpoints, and binaries in nvidia-native.yaml. +openshell provider profile lint -f nvidia-native.yaml +openshell provider profile import -f nvidia-native.yaml +openshell provider create \ + --name nvidia-native-prod \ + --type nvidia-native \ + --from-existing +openshell sandbox provider attach inference-demo nvidia-native-prod ``` -To override the default 60-second per-request timeout, add `--timeout`: +OpenShell does not export stored credential values or change a provider's +profile type in place. If the original credential source is unavailable, the +compatibility built-in lets you attach the preserved provider while you arrange +credential rotation into the replacement provider. -```shell -openshell inference set \ - --provider nvidia-prod \ - --model nvidia/nemotron-3-nano-30b-a3b \ - --timeout 300 -``` +Update each workload to: -The value is in seconds. When `--timeout` is omitted or set to `0`, the default of 60 seconds applies. Increase `--timeout` when you expect extended thinking phases so the full response completes before the request deadline. +1. Call the provider's native endpoint. +2. Read the credential variable declared by its profile. +3. Send the real provider model identifier. +4. Configure request timeouts in the client. +5. Use the provider's native request format. -## Inspect and Update the Config +Launch a new process after attachment and verify a native request before +upgrading production workloads. Code that still calls `inference.local` fails +DNS resolution because OpenShell no longer resolves or trusts that virtual +host. -Confirm that the provider and model are set correctly: +### Migration Checklist -```shell -openshell inference get -Gateway inference: +Before upgrading, record the old provider, model, and timeout with the previous +release's `openshell inference get`. Identify actual consumers, export and edit +the source profile, import it under a new ID, and create the replacement +provider from the original credential source. Attach it only to those +sandboxes, migrate their clients, and test the native path while the old route +is still available. Delete the old route on the previous release to expose +missed consumers. After upgrading the gateway, delete and recreate every +pre-upgrade sandbox so no old supervisor, DNS entry, trust material, or cached +route survives. Reapply only the provider attachments each replacement +sandbox needs. - Provider: nvidia-prod - Model: nvidia/nemotron-3-nano-30b-a3b - Timeout: 300s - Version: 1 -``` +Special cases require additional work: -Use `update` when you want to change only one field: +- A provider with an alternate `OPENAI_BASE_URL` or `ANTHROPIC_BASE_URL` needs + an endpoint-bearing custom profile for that host. +- Host-local services need `host.openshell.internal` or a reachable LAN/service + hostname, not `127.0.0.1` or `localhost`. +- Google Vertex AI clients must use the native Vertex endpoint and + authentication behavior. See [Google Vertex AI](/providers/google-vertex-ai). +- A bridge-fronted AWS Bedrock deployment needs a custom profile that declares + the bridge endpoint and allowed client binaries. -```shell -openshell inference update --model nvidia/nemotron-3-nano-30b-a3b -openshell inference update --provider openai-prod -openshell inference update --timeout 120 -``` +## Security Differences -## Use the Local Endpoint from a Sandbox - -After inference is configured, code inside any sandbox can call `https://inference.local` directly. The client-supplied `model` and `api_key` values are not sent upstream — the privacy router injects the real credentials from the configured provider and rewrites the model before forwarding. Some SDKs require a non-empty API key even though `inference.local` does not use the sandbox-provided value; pass any placeholder such as `unused`. - - - - -```shell -ANTHROPIC_BASE_URL="https://inference.local" ANTHROPIC_API_KEY=unused claude --bare -``` - -`--bare` skips the OAuth login flow and uses `ANTHROPIC_API_KEY` directly. The key is stripped by the proxy and never reaches the upstream provider. - - -Claude Code appends `/v1/messages` to `ANTHROPIC_BASE_URL`, so omit the `/v1` suffix from the base URL. - - - - - -```shell -ANTHROPIC_BASE_URL="https://inference.local/v1" ANTHROPIC_API_KEY=unused opencode -``` - - -OpenCode appends `/messages` directly to `ANTHROPIC_BASE_URL`. Include the `/v1` suffix so the full path becomes `/v1/messages`, which matches the inference pattern. - - - - - -```python -from openai import OpenAI - -client = OpenAI(base_url="https://inference.local/v1", api_key="unused") - -response = client.chat.completions.create( - model="anything", - messages=[{"role": "user", "content": "Hello"}], -) -``` - - - - -```python -import anthropic - -client = anthropic.Anthropic( - base_url="https://inference.local", - api_key="unused", -) - -message = client.messages.create( - model="anything", - max_tokens=1024, - messages=[{"role": "user", "content": "Hello"}], -) -``` - - - - -Use `inference.local` when inference should stay private and credentials should not be exposed inside the sandbox. External providers reached directly belong in `network_policies` instead. - -When the upstream runs on the same machine as the gateway, bind it to `0.0.0.0` and point the provider at `host.openshell.internal` or the host's LAN IP. `127.0.0.1` and `localhost` usually fail because the request originates from the gateway or sandbox runtime, not from your shell. - -If the gateway runs on a remote host or behind a cloud deployment, `host.openshell.internal` points to that remote machine, not to your laptop. A locally running Ollama or vLLM process is not reachable from a remote gateway unless you add your own tunnel or shared network path. - -## Verify from a Sandbox - -`openshell inference set` and `openshell inference update` verify the resolved upstream endpoint by default before saving the configuration. If the endpoint is not live yet, retry with `--no-verify` to persist the route without the probe. - -To confirm end-to-end connectivity from a sandbox, run: - -```shell -curl https://inference.local/v1/responses \ - -H "Content-Type: application/json" \ - -d '{ - "instructions": "You are a helpful assistant.", - "input": "Hello!" - }' -``` - -A successful response confirms the privacy router can reach the configured backend and the model is serving requests. - -- Gateway-scoped: Every sandbox using the active gateway sees the same `inference.local` backend. -- HTTPS only: `inference.local` is intercepted only for HTTPS traffic. -- Hot reload: Provider, model, and timeout changes are picked up by running sandboxes within about 5 seconds by default. No sandbox recreation is required. +Provider attachment preserves credential non-disclosure: workloads receive +opaque placeholders, and the proxy substitutes a real credential only after +network policy and endpoint binding both pass. Native requests are no longer +filtered or rewritten by a model-specific router. The provider profile's L7 +rules therefore define the allowed API surface, and the workload controls +headers, model selection, request shape, streaming, and timeout behavior. ## Next Steps -Explore related topics: - -- To follow a complete Ollama-based local setup, refer to [Inference Ollama](/get-started/tutorials/inference-ollama). -- To follow a complete LM Studio-based local setup, refer to [Local Inference LM Studio](/get-started/tutorials/local-inference-lmstudio). -- To control external endpoints, refer to [Policies](/sandboxes/policies). -- To manage provider records, refer to [Providers](/sandboxes/manage-providers). +- [Profiles](/providers/profiles) +- [Providers](/sandboxes/manage-providers) +- [Customize Sandbox Policies](/sandboxes/policies) +- [Google Vertex AI](/providers/google-vertex-ai) diff --git a/docs/sandboxes/manage-gateways.mdx b/docs/sandboxes/manage-gateways.mdx index 03e4bdfaa..731fd52c1 100644 --- a/docs/sandboxes/manage-gateways.mdx +++ b/docs/sandboxes/manage-gateways.mdx @@ -15,7 +15,7 @@ The gateway is responsible for: - Provisioning and managing sandboxes, including creation, deletion, and status monitoring. - Storing provider credentials and delivering them to sandboxes at startup. - Delivering network and filesystem policies to sandboxes. Policy enforcement itself happens inside each sandbox through the proxy, OPA, Landlock, and seccomp. -- Managing inference configuration and serving inference bundles so sandboxes can route requests to the correct backend. +- Resolving endpoint-bound provider environments for the providers attached to each sandbox. - Providing the SSH tunnel endpoint so you can connect to sandboxes without exposing them directly. OpenShell separates gateway access from the compute driver that runs sandboxes. Use [Installation](/about/installation) to install OpenShell, choose a compute driver, and start a gateway. This page covers working with gateway entries after a gateway exists. diff --git a/docs/sandboxes/manage-providers.mdx b/docs/sandboxes/manage-providers.mdx index 66fc41804..895d6761c 100644 --- a/docs/sandboxes/manage-providers.mdx +++ b/docs/sandboxes/manage-providers.mdx @@ -13,7 +13,9 @@ AI agents typically need credentials to access external services: an API key for Create and manage providers that supply credentials to sandboxes. -Providers v2 is available for profile-backed provider policy, provider-owned network rules, and gateway-managed credential refresh. This page remains the credential-focused provider command reference. For the new workflow, see [Providers v2](/sandboxes/providers-v2). +Provider profiles define provider credentials, policy, and refresh behavior. See +[Profiles](/providers/profiles) for the profile format and custom +profile workflow. Provider profiles include metadata for known endpoints and binaries. View @@ -89,23 +91,23 @@ the stored gateway access token is expired and a refresh token is available, the CLI refreshes it before storing the provider credential. It does not store the OIDC refresh token in the provider. -Provider profile metadata is available for known provider types. Provider profile -network policy is gateway opt-in: - -```shell -openshell settings set --global --key providers_v2_enabled --value true -``` - -Without `providers_v2_enabled=true`, attached provider profiles do not contribute -network policy to the sandbox. Static credential endpoint binding remains active -in either mode. - -When `providers_v2_enabled=true`, `--from-existing` uses profile-backed -discovery instead of the legacy provider registry. The requested `--type` must +`--from-existing` uses profile-backed discovery. The requested `--type` must have a built-in or imported provider profile with a `discovery` section. If no matching profile exists, the CLI returns an error instead of falling back to legacy discovery. +Create a provider whose profile requires no static credentials without a +credential source: + +```shell +openshell provider create --name public-pypi --type pypi +``` + +Provider creation rejects profileless types. For a custom GitLab deployment, +import a profile with the deployment's endpoints and then create an instance +using that profile ID. Existing profileless provider records remain usable, but +OpenShell does not create new ones. + Update a custom provider profile after exporting it, editing its endpoints, binaries, or credential metadata, and preserving the exported `resource_version`: @@ -117,9 +119,18 @@ openshell provider profile update my-api -f my-api-profile.yaml Import remains create-only and fails if the profile ID already exists. Use `provider profile update ` for existing custom profiles. Built-in profiles are read-only. The target ID must match the profile ID in the file. Update accepts -one file at a time and rejects stale resource versions. When -`providers_v2_enabled=true`, updated profile policy applies to all provider -instances of that type on the next sandbox config sync. +one file at a time and rejects stale resource versions. Updated profile policy +applies to all provider instances of that type on the next sandbox config sync. + +Delete one or more custom provider profiles by ID: + +```shell +openshell provider profile delete custom-api custom-alt +``` + +When a multi-profile delete fails for one entry, the CLI reports that profile's +failure and continues with the remaining IDs. The command exits with an error +after it attempts every requested deletion if any entry failed. Static credentials require a built-in or imported provider profile. Profiles @@ -218,17 +229,15 @@ openshell provider refresh rotate my-graph --credential-key MS_GRAPH_ACCESS_TOKE ### AWS STS -AWS STS refresh requires `providers_v2_enabled=true`. The gateway calls -`sts:AssumeRole` and writes three short-lived credentials (`AWS_ACCESS_KEY_ID`, -`AWS_SECRET_ACCESS_KEY`, `AWS_SESSION_TOKEN`) to the provider record atomically. +The gateway calls `sts:AssumeRole` and writes three short-lived credentials +(`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_SESSION_TOKEN`) to the +provider record atomically. The `aws` and `aws-s3` profiles declare `AWS_SECRET_ACCESS_KEY` and `AWS_SESSION_TOKEN` as `additional_outputs` of the `AWS_ACCESS_KEY_ID` refresh, so all three credentials are gateway-minted. Create the provider with `--runtime-credentials` — no placeholder credential is needed. ```shell -openshell settings set --global --key providers_v2_enabled --value true --yes - openshell provider create --name my-aws --type aws-s3 --runtime-credentials openshell provider refresh configure my-aws \ @@ -278,6 +287,16 @@ Delete a provider: openshell provider delete my-claude ``` +Delete multiple providers by listing their names in one command: + +```shell +openshell provider delete my-claude my-github +``` + +When a multi-provider delete fails for one entry, the CLI reports that +provider's failure and continues with the remaining names. The command exits +with an error after it attempts every requested deletion if any entry failed. + ## Attach Providers to Sandboxes Pass one or more `--provider` flags when creating a sandbox: @@ -291,25 +310,23 @@ credentials from every attached provider as placeholders at runtime. Each static credential resolves only for endpoints in that provider's profile, or for explicitly bound sandbox policy endpoints when the profile is endpointless. Profile-managed providers also contribute provider-generated network policy -entries when `providers_v2_enabled` is enabled at the gateway. When the setting -is disabled, endpoint binding still applies, but provider-generated policy does -not. +entries. Endpoint binding independently limits where each static credential can +be resolved. - -Legacy provider attachment is fixed at sandbox creation time. Providers v2 adds -`openshell sandbox provider attach` and `openshell sandbox provider detach` for -running sandboxes. See [Providers v2](/sandboxes/providers-v2#attach-and-detach-providers) -for runtime attach and detach behavior. + +Use `--provider` to attach providers when creating a sandbox. To change the +providers attached to a running sandbox, use `openshell sandbox provider attach` +and `openshell sandbox provider detach`. See +[Profiles](/providers/profiles#attach-and-detach-providers) for details. - + ### Auto-Discovery Shortcut -When `providers_v2_enabled=false` and the trailing command in -`openshell sandbox create` is a recognized tool name (`claude`, `codex`, or -`opencode`), the CLI auto-creates the required provider from your local -credentials if one does not already exist. You do not need to create the -provider separately: +When the trailing command in `openshell sandbox create` maps to an available +profile, the CLI auto-creates the required provider from local profile +discovery if one does not already exist. You do not need to create the provider +separately: ```shell openshell sandbox create -- claude @@ -318,9 +335,8 @@ openshell sandbox create -- claude This detects `claude` as a known tool, finds your `ANTHROPIC_API_KEY`, creates a provider, attaches it to the sandbox, and launches Claude Code. -Providers v2 disables command-derived provider inference. When -`providers_v2_enabled=true`, create or import the provider profile, create the -provider instance, and pass `--provider ` explicitly. +If the inferred type has no built-in or imported profile, creation fails. Import +a custom profile first or pass an existing provider with `--provider `. ## How Credential Injection Works @@ -343,15 +359,13 @@ request. A plain sandbox policy endpoint does not grant credential use unless the profile already covers it or the endpoint explicitly binds an endpointless provider. -Endpoint binding applies whether `providers_v2_enabled` is enabled or disabled. -The setting controls provider policy composition only. Every static credential -declared by a provider receives the complete endpoint set from that provider's +Every static credential declared by a provider receives the complete endpoint set from that provider's profile, or the explicitly bound sandbox policy endpoints for an endpointless profile. Credential resolution requires the proxy to handle the request as HTTP. Raw `tls: skip` and non-HTTP tunnels remain opaque and do not support credential -rewrite. Refer to [Providers v2](/sandboxes/providers-v2#understand-static-credential-endpoint-binding) +rewrite. Refer to [Profiles](/providers/profiles#understand-static-credential-endpoint-binding) for endpoint matching and migration guidance. ### Supported injection locations @@ -397,8 +411,8 @@ there returns `credential_endpoint_mismatch`. For a custom service, define the credential in a custom provider profile. Put stable endpoints in the profile, or leave the profile endpointless and bind -each concrete provider instance from sandbox policy. Refer to [Provider -Profiles](/sandboxes/providers-v2#provider-profiles) for the profile workflow +each concrete provider instance from sandbox policy. Refer to +[Profiles](/providers/profiles#provider-profiles) for the profile workflow and schema. ## Supported Provider Types @@ -408,17 +422,14 @@ The following provider types are supported. | Type | Environment Variables Injected | Typical Use | |---|---|---| | `anthropic` | `ANTHROPIC_API_KEY` | Anthropic API | -| `aws-bedrock` | `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_SESSION_TOKEN`, `AWS_REGION` | AWS Bedrock InvokeModel via a translating bridge. Today the router does not inject any auth header; the configured `BEDROCK_BASE_URL` upstream is expected to handle auth itself. Refer to [Inference Routing](/sandboxes/inference-routing). | +| `aws-bedrock` | `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_SESSION_TOKEN`, `AWS_REGION` | Declarative Bedrock credential shape. The built-in profile does not grant a bridge or AWS endpoint; use an endpoint-bearing profile for direct access. | | `claude` | `ANTHROPIC_API_KEY`, `CLAUDE_API_KEY` | Claude Code, Anthropic API | | `codex` | `OPENAI_API_KEY` | OpenAI Codex | | `copilot` | `COPILOT_GITHUB_TOKEN`, `GH_TOKEN`, `GITHUB_TOKEN` | GitHub Copilot CLI | | `deepinfra` | `DEEPINFRA_API_KEY` | DeepInfra inference API | -| `generic` | User-defined | Legacy credential storage. Import an endpoint-bearing custom profile for credentials attached to a sandbox. | | `github` | `GITHUB_TOKEN`, `GH_TOKEN` | GitHub API and `gh` CLI. Refer to [GitHub Sandbox](/get-started/tutorials/github-sandbox). | -| `gitlab` | `GITLAB_TOKEN`, `GLAB_TOKEN`, `CI_JOB_TOKEN` | GitLab API, `glab` CLI | | `nvidia` | `NVIDIA_API_KEY` | NVIDIA API Catalog | -| `openai` | `OPENAI_API_KEY` | Any OpenAI-compatible endpoint. Set `--config OPENAI_BASE_URL` to point to the provider. Refer to [Inference Routing](/sandboxes/inference-routing). | -| `opencode` | `OPENCODE_API_KEY`, `OPENROUTER_API_KEY`, `OPENAI_API_KEY` | OpenCode | +| `openai` | `OPENAI_API_KEY` | Public OpenAI API. Use a custom endpoint-bearing profile for another OpenAI-compatible host. | `ANTHROPIC_API_KEY` is an API key from [console.anthropic.com](https://console.anthropic.com), not a subscription token. Subscription users must generate a separate API key from the Anthropic Console. @@ -431,24 +442,26 @@ ID as the provider `--type`. -## Supported Inference Providers +## Inference Provider Access -The following providers have been tested with `inference.local`. Any provider that exposes an OpenAI-compatible API works with the `openai` type. Set `--config OPENAI_BASE_URL` to the provider's base URL and `--credential OPENAI_API_KEY` to your API key. +Attach an inference provider to each sandbox that needs it and configure the +workload to call the provider's native endpoint. The workload, not OpenShell, +selects the model and request timeout. -| Provider | Name | Type | Base URL | API Key Variable | -|---|---|---|---|---| -| AWS Bedrock (via bridge) | `bedrock-bridge` | `aws-bedrock` | Operator-supplied `BEDROCK_BASE_URL` | None at router level (bridge holds creds) | -| NVIDIA API Catalog | `nvidia-prod` | `nvidia` | `https://integrate.api.nvidia.com/v1` | `NVIDIA_API_KEY` | -| Anthropic | `anthropic-prod` | `anthropic` | `https://api.anthropic.com` | `ANTHROPIC_API_KEY` | -| Google Vertex AI | `vertex-prod` | `google-vertex-ai` | Regional, global, or multi-region Vertex endpoint | `GOOGLE_VERTEX_AI_TOKEN` or `GOOGLE_VERTEX_AI_SERVICE_ACCOUNT_TOKEN` | -| Baseten | `baseten` | `openai` | `https://inference.baseten.co/v1` | `OPENAI_API_KEY` | -| Bitdeer AI | `bitdeer` | `openai` | `https://api-inference.bitdeer.ai/v1` | `OPENAI_API_KEY` | -| DeepInfra | `deepinfra` | `deepinfra` | `https://api.deepinfra.com/v1/openai` | `DEEPINFRA_API_KEY` | -| Groq | `groq` | `openai` | `https://api.groq.com/openai/v1` | `OPENAI_API_KEY` | -| Ollama (local) | `ollama` | `openai` | `http://host.openshell.internal:11434/v1` | `OPENAI_API_KEY` | -| LM Studio (local) | `lmstudio` | `openai` | `http://host.openshell.internal:1234/v1` | `OPENAI_API_KEY` | +| Provider | Built-in profile | Native base URL | Credential variable | +|---|---|---|---| +| OpenAI | `openai` | `https://api.openai.com/v1` | `OPENAI_API_KEY` | +| Anthropic | `anthropic` | `https://api.anthropic.com` | `ANTHROPIC_API_KEY` | +| NVIDIA API Catalog | `nvidia` | `https://integrate.api.nvidia.com/v1` | `NVIDIA_API_KEY` | +| DeepInfra | `deepinfra` | `https://api.deepinfra.com/v1/openai` | `DEEPINFRA_API_KEY` | +| Google Vertex AI | `google-vertex-ai` | Region and model dependent | `GOOGLE_VERTEX_AI_TOKEN` or `GOOGLE_VERTEX_AI_SERVICE_ACCOUNT_TOKEN` | -Refer to your provider's documentation for the correct base URL, available models, and API key setup. For the Vertex-specific auth flows and config keys, refer to [Google Vertex AI](/providers/google-vertex-ai). To configure inference routing, refer to [Inference Routing](/sandboxes/inference-routing). +An OpenAI-compatible protocol does not make the built-in `openai` profile safe +for an arbitrary host. Baseten, Bitdeer, Groq, Ollama, LM Studio, self-hosted +NIM, and other alternate endpoints need a custom profile that declares the +actual host, port, credential, and allowed binaries. See +[Provider-backed Inference](/sandboxes/inference-routing) for complete examples +and migration guidance. ## Next Steps diff --git a/docs/sandboxes/manage-sandboxes.mdx b/docs/sandboxes/manage-sandboxes.mdx index d8fa185a8..dd192d40f 100644 --- a/docs/sandboxes/manage-sandboxes.mdx +++ b/docs/sandboxes/manage-sandboxes.mdx @@ -23,9 +23,11 @@ openshell sandbox create -- claude The trailing command is the sandbox's canonical main process. OpenShell starts it once, streams its output, and returns its exit status. Exit code 0 leaves a retained sandbox in `Completed`; a nonzero exit leaves it in `Error` with a -`MainProcessFailed` condition. With no trailing command, OpenShell -starts `/bin/bash -l` in a retained pseudo-terminal. Add `--detach` to create -the sandbox without attaching: +`MainProcessFailed` condition. With no trailing command, OpenShell starts a +login shell in a retained pseudo-terminal: `/bin/bash -l` when the image +provides bash, otherwise a shell detected in the image such as `/bin/sh` on +minimal bases like Alpine. Add `--detach` to create the sandbox without +attaching: ```shell openshell sandbox create --name worker --detach -- ./worker @@ -142,20 +144,123 @@ openshell sandbox create \ ### Custom Containers -Use `--from` to create a sandbox from the base image, another pre-built sandbox name, a local directory, or a container image: +Use `--from` to create a sandbox from the base image, another pre-built sandbox name, a rootfs tar archive, or a container image: ```shell openshell sandbox create --from base openshell sandbox create --from ollama -openshell sandbox create --from ./my-sandbox-dir +openshell sandbox create --from ./rootfs.tar openshell sandbox create --from my-registry.example.com/my-image:latest ``` Bare names such as `base` and `ollama` resolve to images under `ghcr.io/nvidia/openshell-community/sandboxes`. Set `OPENSHELL_COMMUNITY_REGISTRY` when you need to use an internal mirror. -Local directories and Dockerfiles require a local gateway because the CLI builds -through the local Docker daemon. Use a registry image reference for remote -gateways. +`--from` does not build local Dockerfiles or directories. Build and tag the +image with the container engine used by your local gateway, then pass the +resulting image reference: + +```shell +# Docker gateway +docker build -t my-image:latest . +openshell sandbox create --from my-image:latest + +# Podman gateway +podman build -t localhost/my-image:latest . +openshell sandbox create --from localhost/my-image:latest +``` + +For a remote gateway, push the image to a registry that the gateway can pull +from and use that registry image reference. + + +**Pre-0.1.0 breaking change:** `openshell sandbox create --from ./Dockerfile` +and directory sources no longer build images. Build and tag the image before +you create the sandbox. + + +#### Rootfs Tar Archives + +A rootfs tar archive (`.tar`, `.tar.gz`, `.tgz`) is a flat filesystem produced +by `docker export`, `podman export`, or `buildah mount` plus `tar`. It lets you +create a sandbox without a registry or a running image daemon: + +```shell +docker create --name export-me my-image:latest +docker export -o rootfs.tar export-me +docker rm export-me + +openshell sandbox create --from ./rootfs.tar +``` + +Rootfs tar sources require a local gateway running the VM compute driver. The +CLI asks the gateway for a staging slot, writes the archive to the location the +gateway allocates, and passes back a single-use token; the gateway resolves that +token to a path for the driver. Because the CLI writes the archive directly to +the gateway host's filesystem, the two must share a filesystem and run as the +same user. Gateways using the Docker, Podman, or Kubernetes drivers reject +rootfs tar sources. + +Gzip-compressed archives (`.tar.gz`, `.tgz`) are decompressed while the gateway +stages them, so the sandbox sees the same filesystem either way. Compression is +detected from the archive contents, not the file name. + +The gateway caps archive size (10 GiB by default, configurable with the VM +driver's `rootfs_tar_max_bytes`), and reclaims an unused staging slot after 30 +minutes. The cap applies to the expanded archive too: a compressed source that +decompresses past the limit is rejected. + +## Reuse Workload Templates + +Sandbox workload templates let workspace admins define reusable runtime shapes for a workspace. A template stores the image, environment, resource requests, and driver-specific configuration that sandboxes should inherit. When you create a sandbox from a template, the create request can still attach providers, labels, and policy, but the workload comes from the named template. + +Create a template: + +```shell +openshell sandbox template create gpu-kata \ + --image registry.example.com/agent:latest \ + --cpu 2 \ + --memory 4Gi \ + --gpu 1 \ + --label team=runtime \ + --env FEATURE_FLAG=on +``` + +Use `--gpu` without a count when the template should request the active +driver's default GPU assignment. Use `--gpu COUNT` when the template needs a +specific number of GPUs. + +Add driver-specific settings when the active compute driver needs them: + +```shell +openshell sandbox template create gpu-kata \ + --image registry.example.com/agent:latest \ + --driver-config-json '{"kubernetes":{"pod":{"runtime_class_name":"kata-containers","node_selector":{"pool":"gpu"}}}}' +``` + +If you omit `--image`, the gateway applies its default sandbox image when a sandbox is created from the template. Use this when the template should only define resource, environment, or driver settings. + +Create a sandbox from a template: + +```shell +openshell sandbox create --template gpu-kata --provider github -- claude +``` + +The `--template` flag cannot be combined with inline workload flags such as `--from`, `--cpu`, `--memory`, `--gpu`, `--env`, or `--driver-config-json`. Put those values on the template instead. Create-time policy and provider attachments remain part of the sandbox request, so each sandbox can keep its own access boundary. + +Inspect and manage templates: + +```shell +openshell sandbox template list +openshell sandbox template list --label-selector team=runtime +openshell sandbox template get gpu-kata +openshell sandbox template delete gpu-kata +``` + +Use `--all-workspaces` with `sandbox template list` when you need an admin view across workspaces: + +```shell +openshell sandbox template list --all-workspaces +``` ## Base Sandbox Container @@ -253,7 +358,7 @@ openshell sandbox create --env API_KEY=sk-test --env DEBUG=1 -- my-agent Variables set with `--env` are available to all processes in the sandbox, including interactive shells and exec commands. -When an `--env` key looks like a credential — a known provider variable, or a name whose underscore-separated segments include a credential word such as `TOKEN`, `SECRET`, `PASSWORD`, `CREDENTIAL`, `API_KEY`, `ACCESS_KEY`, or `SECRET_KEY` (for example `DB_TOKEN` or `MY_ACCESS_KEY`) — `sandbox create` prints a non-blocking warning. Matching is on whole segments, so unrelated names like `TOKENIZERS_PARALLELISM` or `PASSWORDLESS_LOGIN` do not warn. The agent inside the sandbox can read plain environment values directly, so to hide a secret from the agent, attach it through a [provider](/sandboxes/providers-v2) with `--provider` instead. Suppress the warning with `--no-credential-warnings`. Detection uses the key name only; values are never inspected or printed. +When an `--env` key looks like a credential — a known provider variable, or a name whose underscore-separated segments include a credential word such as `TOKEN`, `SECRET`, `PASSWORD`, `CREDENTIAL`, `API_KEY`, `ACCESS_KEY`, or `SECRET_KEY` (for example `DB_TOKEN` or `MY_ACCESS_KEY`) — `sandbox create` prints a non-blocking warning. Matching is on whole segments, so unrelated names like `TOKENIZERS_PARALLELISM` or `PASSWORDLESS_LOGIN` do not warn. The agent inside the sandbox can read plain environment values directly, so to hide a secret from the agent, attach it through a [profile-backed provider](/providers/profiles) with `--provider` instead. Suppress the warning with `--no-credential-warnings`. Detection uses the key name only; values are never inspected or printed. You can also set per-command environment variables with `sandbox exec`: @@ -295,6 +400,24 @@ with SandboxClient.from_active_cluster() as client: assert sandbox.id in {s.id for s in matches} ``` +Create reusable sandbox templates through the Python SDK when several runs +should share the same workload shape: + +```python +from openshell import SandboxClient + +with SandboxClient.from_active_cluster() as client: + templates = client.sandbox_templates() + + templates.create( + workspace="default", + name="python", + image="ghcr.io/nvidia/openshell-community/sandboxes/python:latest", + ) + + sandbox = client.create_from_template(workspace="default", template_name="python") +``` + For non-interactive automation, pass a renewable client-credentials provider. Omitted issuer, client ID, audience, and scopes are read from the active gateway's metadata. The client requires TLS for non-loopback gateways: @@ -335,6 +458,17 @@ List endpoints for one sandbox: openshell service list my-sandbox ``` +Use structured output for automation: + +```shell +openshell service list --output json +openshell service list my-sandbox --output yaml +``` + +Each record contains `workspace`, `sandbox`, `service`, `target_port`, and +`url`. The unnamed service uses an empty `service` string. An empty result is +an empty collection. + Show or delete one endpoint: ```shell @@ -411,7 +545,7 @@ OpenShell Terminal combines sandbox status and live logs in a single real-time d openshell term ``` -Use the terminal to spot blocked connections marked `action=deny` and inference-related proxy activity. If a connection is blocked unexpectedly, add the host to your network policy. Refer to [Policies](/sandboxes/policies) for the workflow. +Use the terminal to spot blocked connections marked `action=deny` and provider-related proxy activity. If a connection is blocked unexpectedly, add the host to your network policy or update the attached provider profile. Refer to [Policies](/sandboxes/policies) for the workflow. The dashboard has three panels stacked vertically: Gateways, Providers (or Global Settings), and Sandboxes. Navigate within a panel with `Up`/`Down` or `j`/`k`. At a list boundary the cursor overflows into the adjacent panel, skipping empty panels. Use `Tab`/`Shift+Tab` to cycle panels directly. Press `h`/`l` or `Left`/`Right` in the middle panel to switch between the Providers and Global Settings tabs. @@ -433,6 +567,20 @@ openshell forward list openshell forward stop 8000 my-sandbox ``` +Use JSON or YAML output when inspecting tracked forwards from automation: + +```shell +openshell forward list --output json +openshell forward list --output yaml +``` + +Structured output includes `sandbox`, `bind_address`, `port`, `pid`, and +`alive`. The `alive` boolean reports whether the tracked PID still matches the +expected OpenShell SSH forward; it does not probe the forwarded socket. When no +forwards are tracked, structured output returns an empty collection. + +The default table colorizes the `STATUS` column only when both standard output and standard error are capable ANSI terminals; piping or redirecting either stream, or running under `TERM=dumb`, gives a plain-text table. Other styled output—including `-v` log lines, progress spinners, prompts, and error messages—is decided per stream, so redirecting one stream leaves the other styled. Prefer `--output json` for automation rather than matching on the table. Set `NO_COLOR` to any non-empty value or pass `--color never` to suppress ANSI formatting, and `--color always` to force it when piping into a pager. `--color` applies to every `openshell` command. + You can also forward a port at creation time with `--forward`: @@ -532,6 +680,16 @@ Deleting a sandbox stops all processes, releases resources, and purges injected openshell sandbox delete my-sandbox ``` +Delete multiple sandboxes by listing their names in one command: + +```shell +openshell sandbox delete sandbox-a sandbox-b +``` + +When a multi-sandbox delete fails for one entry, the CLI reports that sandbox's +failure and continues with the remaining names. The command exits with an error +after it attempts every requested deletion if any entry failed. + ## Sandbox Lifecycle Every sandbox moves through a defined set of phases: diff --git a/docs/sandboxes/manage-workspaces.mdx b/docs/sandboxes/manage-workspaces.mdx index 5efbd4d09..003201ac2 100644 --- a/docs/sandboxes/manage-workspaces.mdx +++ b/docs/sandboxes/manage-workspaces.mdx @@ -9,8 +9,9 @@ position: 3 --- An OpenShell workspace is an access and resource isolation boundary. Sandboxes, -providers, services, policies, settings, and inference routes belong to a -workspace and are not visible to members of other workspaces. +sandbox workload templates, providers, provider profiles, services, policies, +and settings belong to a workspace and are not visible to members of other +workspaces. The CLI targets the `default` workspace unless you set `--workspace` or `OPENSHELL_WORKSPACE`. The logical OpenShell workspace described here is @@ -49,6 +50,8 @@ The following table summarizes common operations. | Add Workspace Users or remove members | Any workspace | Assigned workspace | No | | Assign the Workspace Admin role | Any workspace | No | No | | Create, use, or delete sandboxes and services | Any workspace | Assigned workspace | Assigned workspace | +| Create or delete sandbox workload templates | Any workspace | Assigned workspace | No | +| Read or list sandbox workload templates | Any workspace | Assigned workspace | Assigned workspace | | Create, update, or delete providers | Any workspace | Assigned workspace | No | | Change workspace policy or settings | Any workspace | Assigned workspace | No | | Manage platform profiles or global configuration | Yes | No | No | @@ -121,6 +124,10 @@ openshell workspace member remove \ --subject 'oidc-subject-for-user' ``` +Add `--output json` or `--output yaml` to list membership records for +automation. Each record contains `subject` and a normalized `role` of `admin`, +`user`, or `unknown`. An empty result is an empty collection. + To change a member's role, remove the existing membership and add it again with the new role. A Platform Admin must perform any change to `admin`. @@ -142,8 +149,9 @@ export OPENSHELL_WORKSPACE=team-ml openshell sandbox list ``` -An empty workspace value resolves to `default`. It never means all -workspaces. +When `--workspace` is omitted, the CLI intentionally selects the `default` +workspace. An empty workspace name is invalid and never means either `default` +or all workspaces. Platform Admins can opt into cross-workspace list operations: @@ -153,6 +161,29 @@ openshell provider list --all-workspaces openshell service list --all-workspaces ``` +The public API represents this choice with a `WorkspaceSelector` oneof. Set +`workspace` to a non-empty name, including the literal `default`, or set the +`all_workspaces` marker on list requests that support it. Omitting the selector +or sending an unset selector is invalid for workspace-scoped operations. The +all-workspaces variant is accepted only by sandbox, sandbox template, provider, +and service list requests, and it requires Platform Admin access. + +Clients migrating from the previous request fields should make the scope +explicit: + +| Previous request | Typed selector | +| --- | --- | +| `workspace: "team-ml"` | `workspace_scope.workspace: "team-ml"` | +| Empty or omitted `workspace` for the default | `workspace_scope.workspace: "default"` | +| `all_workspaces: true` | `workspace_scope.all_workspaces: {}` | +| `global: true` | Omit `workspace_scope` | + +The Rust and Python SDKs expose separate all-workspaces list methods. The Go +SDK uses `ListAll`, and the TypeScript SDK uses a discriminated option type, so +a caller cannot select a named workspace and all workspaces in one typed call. +The TUI starts in the `default` workspace and sends that named selector +explicitly. Its all-workspaces view sends the marker instead. + Provider profiles and policy also have explicit `--global` operations. Those operations target platform scope and require Platform Admin access. A Workspace Admin should use `--workspace` for workspace-scoped profiles and @@ -183,10 +214,10 @@ be deleted. openshell workspace delete team-ml ``` -A custom workspace must not contain sandboxes, providers, provider profiles, -services, SSH sessions, settings, policies, draft policy chunks, or credential -refresh state. Remove those resources before retrying deletion. OpenShell -removes membership records and inference routes as part of successful +A custom workspace must not contain sandboxes, sandbox workload templates, +providers, provider profiles, services, SSH sessions, settings, policies, draft +policy chunks, or credential refresh state. Remove those resources before +retrying deletion. OpenShell removes membership records as part of successful workspace deletion. ## Next Steps diff --git a/docs/sandboxes/policies.mdx b/docs/sandboxes/policies.mdx index a85bbd9b9..af25238a6 100644 --- a/docs/sandboxes/policies.mdx +++ b/docs/sandboxes/policies.mdx @@ -69,7 +69,7 @@ When a hot reload changes rules, the supervisor publishes a new policy generatio | `filesystem_policy` | Static | Controls which directories the agent can access on disk. Paths are split into `read_only` and `read_write` lists. Any path not listed in either list is inaccessible. Set `include_workdir: true` to automatically add the agent's working directory to `read_write`. [Landlock LSM](https://docs.kernel.org/security/landlock.html) enforces these restrictions at the kernel level. | | `landlock` | Static | Configures Landlock LSM enforcement behavior. Set `compatibility` to `best_effort` (skip individual inaccessible paths while applying remaining rules) or `hard_requirement` (fail if any path is inaccessible or the required kernel ABI is unavailable). Refer to the [Policy Schema Reference](/reference/policy-schema#landlock) for the full behavior table. | | `process` | Static | Optionally overrides the OS-level identity for the agent process. Explicit values must be `sandbox` or numeric UID/GID values from `1` through `4294967294`; root and the invalid identity sentinel are rejected. Docker and Podman may use named identities through per-field OCI `USER` fallback; Kubernetes uses its platform-selected numeric identity. The agent also runs with seccomp filters that block dangerous system calls. | -| `network_policies` | Dynamic | Controls network access for ordinary outbound traffic from the sandbox. Each block has a name, a list of endpoints (host, port, protocol, and optional rules), and a list of binaries allowed to use those endpoints.
Every outbound connection except `https://inference.local` passes through the network supervisor, which queries the [policy engine](/about/how-it-works#core-components) with the destination and calling binary. A connection is allowed only when both match an entry in the same policy block.
For endpoints with `protocol: rest`, the proxy auto-detects TLS and terminates it so each HTTP request can be checked against that endpoint's `rules` (method and path). For endpoints with `protocol: websocket`, the proxy validates the RFC 6455 upgrade and evaluates `GET` rules for the handshake plus either `WEBSOCKET_TEXT` rules for raw client text messages or GraphQL operation rules for GraphQL-over-WebSocket messages. Set `websocket_credential_rewrite: true` only when a WebSocket or REST compatibility endpoint must keep placeholder credentials in sandbox-owned text frames and resolve them at the OpenShell relay boundary.
Endpoints with `protocol: tcp` allow ordinary DNS resolution and native TCP connections without inspecting payloads. Endpoints without `protocol` retain L4 passthrough through an explicit proxy.
If no endpoint matches, the connection is denied. Configure managed inference separately through [Inference Routing](/sandboxes/inference-routing). | +| `network_policies` | Dynamic | Controls outbound traffic from the sandbox, including native model-provider endpoints. Each block has a name, a list of endpoints (host, port, protocol, and optional rules), and a list of binaries allowed to use those endpoints.
Every outbound connection passes through the network supervisor, which queries the [policy engine](/about/how-it-works#core-components) with the destination and calling binary. A connection is allowed only when both match an entry in the same policy block. Attached provider profiles can contribute endpoint and binary entries to the effective policy.
For endpoints with `protocol: rest`, the proxy auto-detects TLS and terminates it so each HTTP request can be checked against that endpoint's `rules` (method and path). For endpoints with `protocol: websocket`, the proxy validates the RFC 6455 upgrade and evaluates `GET` rules for the handshake plus either `WEBSOCKET_TEXT` rules for raw client text messages or GraphQL operation rules for GraphQL-over-WebSocket messages. Set `websocket_credential_rewrite: true` only when a WebSocket or REST compatibility endpoint must keep placeholder credentials in sandbox-owned text frames and resolve them at the OpenShell relay boundary.
Endpoints with `protocol: tcp` allow ordinary DNS resolution and native TCP connections without inspecting payloads. Endpoints without `protocol` retain L4 passthrough through an explicit proxy.
If no endpoint matches, the connection is denied. | | `network_middlewares` | Dynamic | Declares keyed HTTP and WebSocket middleware configs. After network and L7 policy admit a request or upgrade, OpenShell matches each config's host selectors independently and runs matching entries by their unique ascending `order` before credential injection. WebSocket-capable entries continue on complete client text messages. | ## Supervisor Middleware @@ -131,6 +131,8 @@ network_policies: OpenShell answers DNS only for hostnames eligible under an active `protocol: tcp` endpoint. It validates upstream answers against destination and SSRF controls, returns a supervisor-owned synthetic address, and records the validated real addresses. When the application connects to the synthetic address, OpenShell recovers the hostname and port, evaluates the calling process against the current policy generation, and dials only an address pinned by that DNS result. +Treat that hostname as a connection-routing constraint, not an application-authority boundary. OpenShell does not inspect TLS SNI, HTTP `Host`, or another protocol-level destination inside a `protocol: tcp` stream. If the approved hostname reaches compatible shared infrastructure, a client may be able to select another tenant, virtual host, or service behind the same front door. Use native TCP only when the trust boundary includes every destination that the shared infrastructure can expose; use an inspected protocol when application authority must remain constrained. + Applications must honor the returned DNS TTL and resolve the hostname again before reconnecting after that TTL expires. A client that caches the synthetic address indefinitely can receive a connection failure after the mapping expires. Docker and Podman currently advertise only IPv4 egress for this feature, so OpenShell returns an empty successful answer for AAAA queries and lets dual-stack clients use the working A record. DNS resolution does not authorize a connection by itself. Unknown names, wrong ports, stale mappings, disallowed destination addresses, and binaries outside the matching policy fail closed. Applications cannot inherit access by connecting directly to a real IP returned by an upstream resolver. @@ -232,10 +234,17 @@ The following steps outline the hot-reload policy update workflow. openshell policy list ``` + Add `--output json` or `--output yaml` for automation. Structured policy + history contains the scope, sandbox name when applicable, version, full + hash, status, and available revision timestamps, load error, and provenance. + Use `openshell policy list --global --output json` for global history. + ### Validation failures OpenShell validates a complete candidate policy before activating any part of it. Endpoints may overlap when their connection and request-processing metadata agree. For example, two `api.example.com:443` REST entries can contribute different allow and deny rules when they use the same TLS, destination, credential, parser, and enforcement settings. A plain L4 endpoint may overlap an L7 endpoint because it authorizes the destination without contributing request-processing metadata. A more-specific path endpoint may override request-processing metadata from a broader endpoint, such as a `/graphql` GraphQL endpoint alongside a general REST endpoint for the same host. OpenShell rejects the candidate when overlapping exact or wildcard host selectors can both contribute equally specific endpoint configuration and disagree on those fields. +Internal policy-advisor provenance does not make otherwise compatible endpoints ambiguous. This lets an advisor proposal extend a provider-covered host without modifying the provider rule. TLS, destination IP constraints, credential handling, protocol, parser, and equally specific enforcement settings must still agree. + When the gateway knows the affected sandbox scope, it validates the complete effective candidate before persistence. This covers direct policy replacement, incremental merges and proposal approvals, provider attachment, and @@ -367,7 +376,7 @@ Static provider placeholders resolve only when the request host, port, and path also match an endpoint in the provider profile. A sandbox policy allow does not expand that binding. A mismatch returns HTTP 403 with `credential_endpoint_mismatch`. Refer to [Static Credential Endpoint -Binding](/sandboxes/providers-v2#understand-static-credential-endpoint-binding). +Binding](/providers/profiles#understand-static-credential-endpoint-binding). For example: @@ -556,7 +565,7 @@ You can declare the scope across several `--add-endpoint` arguments. The update An endpoint's allow rules, deny rules, and allowed IPs apply to every port that endpoint carries, so an update that changes any of them has to name every one of those ports. Declaring `api.example.com:443` alone on an endpoint that also serves `8443` is rejected, because the change would reach `8443` as well. Declaring every existing binary does not lift this requirement; the two are separate axes of the same product. -The sandbox picks the parser for a request by most-specific path, but it authorizes the request against every endpoint that matches it. A broad REST endpoint and a narrower GraphQL endpoint on one host and port share the same method-and-path rule vocabulary, so that combination stays supported. MCP does not: its rules address JSON-RPC methods and tool names, so a plain REST rule on an overlapping path could authorize a tool call the MCP endpoint denies. An MCP endpoint therefore cannot share a host and port with a differently inspected endpoint, and two MCP endpoints there must agree on strict-tool-name, method-profile, and body-limit settings, even under different paths or in separate rules. An update creating either situation is rejected. A policy that already contains one is left alone so unrelated updates still apply, but it should be repaired with full YAML replacement. +The sandbox picks the parser for a request by most-specific path, but it authorizes the request against every endpoint that matches it. A broad REST endpoint and a narrower GraphQL endpoint on one host and port share the same method-and-path rule vocabulary, so that combination stays supported. MCP does not: its rules address JSON-RPC methods and tool names, so a plain REST rule on an overlapping path could authorize a tool call the MCP endpoint denies. An MCP endpoint therefore cannot share a host and port with a differently inspected endpoint, and two MCP endpoints there must agree on their exact revision allowlist, strict-tool-name, method-profile, and body-limit settings, even under different paths or in separate rules. An update creating either situation is rejected. A policy that already contains one is left alone so unrelated updates still apply, but it should be repaired with full YAML replacement. To grant one binary access to only part of an existing rule's endpoints, send it under its own `--rule-name`. The gateway normally folds an update into an existing rule that shares an endpoint, but it keeps your rule name whenever folding would grant authorization you did not declare, so the narrow grant lands as its own rule authorizing exactly what you asked for. The update reports that it kept your rule name and names the rule it would otherwise have folded into. An MCP contract conflict is the exception: one host and port carry a single MCP contract regardless of which rule holds them, so a conflicting update is rejected rather than moved to a separate rule. @@ -607,7 +616,7 @@ Do not fix `credential_endpoint_mismatch` by widening sandbox policy. Export the provider profile with `openshell provider profile export -o yaml`. Update the custom provider profile only when the destination is an intended credential recipient. Refer to [Static Credential Endpoint -Binding](/sandboxes/providers-v2#understand-static-credential-endpoint-binding) +Binding](/providers/profiles#understand-static-credential-endpoint-binding) for the complete authorization model. For small changes, prefer `openshell policy update` over rewriting the full YAML: @@ -729,10 +738,20 @@ REST rules can also constrain query parameter values: ### MCP and JSON-RPC matching -MCP endpoints use `protocol: mcp`. The proxy parses sandbox-to-server MCP Streamable HTTP request bodies, validates known MCP request and notification params, can evaluate the MCP method against rule `method`, and can match tool calls with the `tool` alias. Unknown extension methods stay addressable as literal method strings. Until OpenShell exposes MCP version profiles, `mcp.allow_all_known_mcp_methods` defaults to `false`, so endpoints require explicit MCP method rules. Set `mcp.allow_all_known_mcp_methods: true` to enable the endpoint method profile; in that mode, rules can omit `method`, and tool selectors are normalized to `tools/call` internally. By default, MCP `tools/call` tool names must match `^[A-Za-z0-9_.-]{1,128}$`; set `mcp.strict_tool_names: false` on that endpoint only when a server intentionally uses names outside the MCP-recommended pattern. Wildcard `tool` matchers require `mcp.strict_tool_names` to remain enabled. Generic JSON-RPC endpoints use `protocol: json-rpc` and evaluate `method`. +MCP endpoints use `protocol: mcp`. The proxy parses sandbox-to-server MCP Streamable HTTP request bodies, validates known MCP request and notification params, can evaluate the MCP method against rule `method`, and can match tool calls with the `tool` alias. Unknown extension methods stay addressable as literal method strings. You may omit the entire `mcp` stanza when using its defaults, or omit only `mcp.versions` when setting another MCP option. OpenShell immediately resolves either form to the exact `["2025-11-25"]` allowlist and stores the materialized list in canonical policy data. Adding another supported revision therefore never widens a normalized policy. Defaulting requires the `mcp` or `mcp.versions` key to be absent; explicit `mcp: null`, `versions: null`, and `versions: []` values are invalid. Use an explicit nonempty allowlist only for intentional compatibility or downgrade control. Supported revisions are `2025-03-26`, `2025-06-18`, and `2025-11-25`; the exact support floor is `2025-03-26`, and this is a closed set rather than a date range. Explicit values must be unique and contain no extra whitespace; OpenShell stores them in semantic order. Moving aliases such as `draft` or `latest` are rejected because they could change policy meaning without a policy edit; omission never means all known versions. The sessionless `2026-07-28` revision is not accepted until OpenShell supports its distinct per-request contract. A version identifies a core MCP revision only; there is no policy syntax for separately named SEP overlays. For every MCP HTTP request except a valid standalone `initialize`, OpenShell selects the revision from one `MCP-Protocol-Version` header or, when the header is absent, the MCP specification's `2025-03-26` compatibility fallback. The selected revision must appear in the allowlist. Duplicate, empty, and unsupported header values receive `400 Bad Request`; a supported revision outside the allowlist receives `403 Forbidden`. OpenShell checks the request again after middleware changes it and before forwarding. Generic JSON-RPC endpoints do not use this header and continue to evaluate only `method`. `mcp.allow_all_known_mcp_methods` defaults to `false`, so endpoints require explicit MCP method rules. Set it to `true` to enable the endpoint method profile; in that mode, rules can omit `method`, and tool selectors are normalized to `tools/call` internally. By default, MCP `tools/call` tool names must match `^[A-Za-z0-9_.-]{1,128}$`; set `mcp.strict_tool_names: false` on that endpoint only when a server intentionally uses names outside the MCP-recommended pattern. Wildcard `tool` matchers require `mcp.strict_tool_names` to remain enabled. + +The current registry declares these profile facts for later runtime enforcement: + +- `2025-03-26` permits nonempty same-side top-level JSON-RPC batches; OpenShell's planned enforcement caps them at 64 members. +- `2025-06-18` prohibits top-level JSON-RPC arrays. +- `2025-11-25` prohibits top-level JSON-RPC arrays. + +OpenShell uses the per-request header only to enforce the allowlist; the current parser does not yet apply these version-specific batch rules. OpenShell does not treat the client's `initialize.params.protocolVersion` as the selected revision, inspect the server's initialization response, or bind `MCP-Session-Id`. The version check is stateless and applies independently to each later request, including GET and DELETE. MCP endpoints must declare a concrete destination with `host` and `port` or `ports`. A policy entry that only sets `protocol: mcp` is invalid and is not treated as a wildcard MCP authorization. Use `path: /mcp` when the server's MCP endpoint is path-scoped; omitting `path` matches every HTTP path on that host and port. +Existing versionless inspected MCP policies remain valid and normalize to the exact `2025-11-25` default; they do not need a migration edit. This is a pinned default, not a moving latest or all-known selection. For an unsupported revision, omit `protocol` and the `mcp` stanza to authorize deliberate, uninspected TCP passthrough only when that weaker L4 boundary is acceptable; OpenShell does not silently downgrade to an older inspected profile. + MCP policy enforcement is directional. It applies to HTTP request bodies sent by the sandboxed process to the configured endpoint. JSON-RPC responses and server-to-client MCP messages carried on response bodies or SSE streams are relayed but are not currently parsed for policy enforcement. MCP and JSON-RPC endpoint policies currently require full policy YAML applied with `openshell policy set`; the incremental `openshell policy update --add-endpoint` parser does not accept `mcp` or `json-rpc` as protocols. @@ -748,8 +767,6 @@ An MCP client first sends `initialize`. After the server returns a successful re path: /mcp protocol: mcp enforcement: enforce - mcp: - max_body_bytes: 131072 rules: - allow: method: initialize @@ -769,7 +786,14 @@ An MCP client first sends `initialize`. After the server returns a successful re - { path: /usr/bin/python3 } ``` -`mcp.max_body_bytes` controls how many MCP-over-HTTP request body bytes OpenShell buffers for inspection. It defaults to `65536`. `mcp.strict_tool_names` defaults to `true` for each MCP endpoint. `mcp.allow_all_known_mcp_methods` defaults to `false`; when it is unset or `false`, the endpoint must define explicit MCP method rules. If an MCP endpoint sets `mcp.allow_all_known_mcp_methods: true` and omits `rules`, OpenShell allows all MCP-family methods and all tools, then applies any `deny_rules`. A broad allow or deny rule whose method matcher includes `tools/call` cannot be combined with tool-specific allow rules because it would bypass or erase the tool filter; add `tool` or `params.name` to scope `tools/call`, or remove the tool-specific rules. +The example omits the entire `mcp` stanza, so OpenShell uses the exact `2025-11-25` revision and the other MCP defaults. Canonical serialization still shows the materialized revision list. To allow an older server intentionally, add an explicit compatibility or downgrade allowlist: + +```yaml showLineNumbers={false} + mcp: + versions: ["2025-03-26", "2025-11-25"] +``` + +OpenShell canonicalizes this list in semantic order. Later runtime negotiation selects one allowed revision and applies only that profile; it does not combine the two profiles. `mcp.max_body_bytes` controls how many MCP-over-HTTP request body bytes OpenShell buffers for inspection and defaults to `65536`. `mcp.strict_tool_names` defaults to `true` for each MCP endpoint. `mcp.allow_all_known_mcp_methods` defaults to `false`; when it is unset or `false`, the endpoint must define explicit MCP method rules. If an MCP endpoint sets `mcp.allow_all_known_mcp_methods: true` and omits `rules`, OpenShell allows all MCP-family methods and all tools, then applies any `deny_rules`. A broad allow or deny rule whose method matcher includes `tools/call` cannot be combined with tool-specific allow rules because it would bypass or erase the tool filter; add `tool` or `params.name` to scope `tools/call`, or remove the tool-specific rules. Use `protocol: json-rpc` and `method` when you need generic JSON-RPC 2.0 matching for a non-MCP server. Generic JSON-RPC method rules accept exact method names, or `method: "*"` as the all-method sentinel; other wildcard or glob patterns are rejected. `json_rpc.max_body_bytes` controls the generic JSON-RPC inspection buffer. diff --git a/docs/sandboxes/policy-advisor.mdx b/docs/sandboxes/policy-advisor.mdx index b4d607cb3..80884620d 100644 --- a/docs/sandboxes/policy-advisor.mdx +++ b/docs/sandboxes/policy-advisor.mdx @@ -129,7 +129,26 @@ OpenShell has two proposal paths: | Path | Source | Typical rule shape | |---|---|---| | Mechanistic mapper | Aggregated denial summaries from the sandbox. | Groups by host, port, and binary. If L7 request samples are available, it can draft REST method and path rules. Otherwise it drafts an L4 endpoint. | -| Agent-authored proposal | The in-sandbox agent, using `policy.local`. | Usually a REST `addRule` with exact host, port, binary, method, and path from the structured denial. It can also propose L4 rules for opaque protocols. | +| Agent-authored proposal | The in-sandbox agent, using `policy.local`. | Usually a REST `addRule` with exact host, port, binary, method, and path from the structured denial. It can also omit `protocol` for endpoint-only access through the explicit proxy. | + +### How proposal provenance works + +OpenShell tracks whether an endpoint and binary came from policy advisor. This is internal provenance; it is not a policy YAML field that authors set. Think of each marker as answering “who introduced this identity?” rather than “what traffic does this allow?” + +| Marker | `false` | `true` | When both declarations meet | +|---|---|---|---| +| Endpoint provenance | A user or provider explicitly declared the endpoint. | Policy advisor proposed the endpoint. | The values do not conflict by themselves. The explicit declaration wins if the endpoints merge. | +| Binary provenance | A user or provider explicitly declared the binary path. | Policy advisor proposed the binary path. | The explicit declaration wins if the same path merges. | + +For example, a GitHub provider can explicitly declare `api.github.com:443` for read operations. An agent can then propose `PUT /repos/NVIDIA/OpenShell/contents/docs/**` for the same endpoint. The provider endpoint has provenance `false`; the proposal endpoint has provenance `true`. OpenShell allows that overlap, keeps the provider rule immutable, and stores an approved write rule in the sandbox policy layer. + +Provenance does not hide a real endpoint conflict. The same two declarations still fail validation if they disagree on connection or request-processing behavior that must have one value, such as TLS mode, `allowed_ips`, an equally specific L7 protocol or parser contract, credential binding, or enforcement mode. Authorization fields such as compatible allow and deny rules can combine. + +Exact-host SSRF trust requires an exact endpoint and the matching binary identity to be explicit in the same rule. An advisor-only endpoint or binary does not create that stronger trust. For example: + +- A provider rule that explicitly declares both `/usr/bin/gh` and `api.github.com:443` already establishes exact-host trust for that pair. A later advisor proposal does not create or broaden that trust. +- If the provider declares `api.github.com:443` for `/usr/bin/gh`, but the advisor proposes the endpoint for `/usr/bin/curl`, `curl` does not inherit the provider's binary identity. Its proposed rule remains subject to the normal SSRF checks. +- If an advisor proposes `internal-api.example:443` and it resolves to a private address, approval alone is not enough. A developer must explicitly authorize the intended address range with `allowed_ips`. For REST APIs, prefer L7 rules over broad L4 access. A good proposal allows one method and the smallest safe path: @@ -170,9 +189,9 @@ For REST APIs, prefer L7 rules over broad L4 access. A good proposal allows one } ``` -The current `policy.local` JSON shape covers L4 endpoints and REST method or path rules. Use [Customize Sandbox Policies](/sandboxes/policies) or [Policy Schema Reference](/reference/policy-schema) for policy fields that are not part of the agent-authored proposal surface, such as WebSocket credential rewrite, GraphQL operation matching, endpoint path scoping, and provider-owned policy bundles. +The current `policy.local` JSON shape covers explicit-proxy endpoints and REST method or path rules. Agent-authored proposals cannot set `protocol: tcp` or `tls: skip`, because those modes bypass application-authority inspection. When a task requires native TCP or a raw TLS tunnel, a developer must add the rule through the normal policy-authoring workflow. Omitting `protocol` remains supported and retains the explicit proxy's default TLS termination and HTTP authority checks. Use [Customize Sandbox Policies](/sandboxes/policies) or [Policy Schema Reference](/reference/policy-schema) for policy fields that are not part of the agent-authored proposal surface, such as WebSocket credential rewrite, GraphQL operation matching, endpoint path scoping, and provider-owned policy bundles. -Policy advisor proposals do not add `allowed_ips` automatically. If an advisor-proposed hostname resolves to an internal or private address, OpenShell's SSRF protections still block the connection until a developer explicitly adds the required `allowed_ips` entry. Exact hostname trust for user-declared policy endpoints does not apply to advisor-generated proposal binaries. +Policy advisor proposals do not add `allowed_ips` automatically. If an advisor-proposed hostname resolves to an internal or private address, OpenShell's SSRF protections still block the connection until a developer explicitly adds the required `allowed_ips` entry. Private RFC 1918, CGNAT, IPv6 ULA, and other special-use destinations classified as internal produce advisory security notes when they appear as literal endpoint IPs or in `allowed_ips`. CIDR intersections are included, and hostless `allowed_ips` rules receive an additional warning because they can match any hostname resolving into the configured range. @@ -195,7 +214,7 @@ Findings are categorical. There is no severity tier. The reviewer reads the cate Before approval, the gateway rebuilds the candidate token from the live base policy, immutable provider rules, and non-secret credential metadata. When that token is unchanged, it reuses the persisted prover result instead of rerunning the prover. When it changes, the gateway evaluates and persists the refreshed candidate, leaves the chunk pending, and requires the reviewer to inspect and approve the new token. Edits and deduplicated resubmissions follow the same path. Merge, policy-shape, provider-composition, credential, or prover failures are shown as application errors and cannot be approved. Security notes flag concerns such as internal or private destinations and `allowed_ips`, wildcard hosts, hostless `allowed_ips`, ephemeral ports, and well-known database or service ports. Any prover finding or security note keeps the chunk pending in auto mode. -The full reasoning model lives in [`crates/openshell-prover/README.md`](https://github.com/NVIDIA/OpenShell/blob/main/crates/openshell-prover/README.md). Provider profiles composed in via [Providers v2](/sandboxes/providers-v2) are part of the effective policy the prover reasons over. +The full reasoning model lives in [`crates/openshell-prover/README.md`](https://github.com/NVIDIA/OpenShell/blob/main/crates/openshell-prover/README.md). Provider profiles composed in via [Profiles](/providers/profiles) are part of the effective policy the prover reasons over. ## Review Proposals diff --git a/docs/security/best-practices.mdx b/docs/security/best-practices.mdx index 1c541f8f8..63aaf184a 100644 --- a/docs/security/best-practices.mdx +++ b/docs/security/best-practices.mdx @@ -9,7 +9,7 @@ keywords: "Generative AI, Cybersecurity, Security, Policy, Sandbox, Landlock, Se position: 1 --- -OpenShell enforces sandbox security across four layers: network, filesystem, process, and inference. +OpenShell enforces sandbox security across four layers: network, filesystem, process, and provider credentials. This page documents every configurable control, its default, what it protects, and the risk of relaxing it. For the full policy YAML schema, refer to the [Policy Schema](/reference/policy-schema). @@ -31,7 +31,7 @@ You can update dynamic controls on a running sandbox with `openshell policy upda | Network | Unauthorized outbound connections and data exfiltration. | CONNECT proxy + OPA policy engine | Yes. Use `openshell policy update`, `openshell policy set`, or operator approval in the TUI. | | Filesystem | System binary tampering, credential theft, config manipulation. | Landlock LSM (kernel level) | No. Requires sandbox re-creation. | | Process | Privilege escalation, fork bombs, dangerous syscalls. | Seccomp BPF + privilege drop (`setuid`/`setgid`) | No. Requires sandbox re-creation. | -| Inference | Credential exposure, unauthorized model access. | Proxy intercept of `inference.local` | Yes. Use `openshell inference set`. | +| Provider credentials | Credential exposure and unauthorized service access. | Network policy plus profile endpoint binding and proxy substitution | Yes. Attach or detach providers and update dynamic policy. | ## Network Controls @@ -233,22 +233,19 @@ This ordering is intentional: named network-namespace setup still relies on priv 6. Landlock filesystem restrictions. 7. Runtime seccomp socket domain and syscall filters. -## Inference Controls +## Provider-backed Inference Controls -OpenShell routes all inference traffic through the gateway to isolate provider credentials from the sandbox. - -### Routed Inference through `inference.local` - -The proxy intercepts HTTPS CONNECT requests to `inference.local` and routes matching inference API requests through the sandbox-local router. -The agent never receives the provider API key. +Inference providers use the same endpoint-bound credential mechanism as other +providers. Attach a profile-backed provider only to sandboxes that need it. +The agent receives an opaque placeholder, not the real API key or access token. | Aspect | Detail | |---|---| -| Default | Always active. The proxy handles `inference.local` before OPA policy evaluation. The gateway injects credentials on the host side. | -| Keep-alive isolation | If a sandbox reuses a keep-alive connection that previously carried a routed inference request for a subsequent non-inference request, the proxy denies the non-inference request with `connection not allowed by policy` and closes the connection. This prevents agents from reusing an inference-authorized connection for other destinations. | -| What you can change | Configure inference routes with `openshell inference set`. | -| Risk if bypassed | If an inference provider's host is added directly to `network_policies`, the agent could reach it with a stolen or hardcoded key, bypassing credential isolation. | -| Recommendation | Do not add inference provider hosts to `network_policies`. Use OpenShell inference routing instead. | +| Default | No model-provider access. A sandbox must attach a provider and policy must admit the native endpoint and calling binary. | +| Credential boundary | The placeholder resolves only for hosts, ports, and paths authorized by the provider profile. | +| What you can change | Attach or detach providers and adjust the profile or sandbox policy. Model and timeout configuration remain in the native client. | +| Risk if relaxed | Broad endpoint paths or binary globs can expose more of a provider's API than the workload needs. `allow_uninspected_credentials` removes L7 visibility and should be exceptional. | +| Recommendation | Use narrow endpoint paths and binaries, inspect the effective policy, and keep credentials in provider records rather than workload environment configuration. | ## Gateway Security diff --git a/e2e/mcp-conformance.sh b/e2e/mcp-conformance.sh index 1bbe5951f..a9e912e34 100755 --- a/e2e/mcp-conformance.sh +++ b/e2e/mcp-conformance.sh @@ -330,7 +330,8 @@ create_client_sandbox() { # for each scenario URL before executing the TypeScript client. python3 "${ROOT}/e2e/mcp-conformance/render-policy.py" \ "http://192.0.2.1:1/" "${policy_file}" \ - "${ROOT}/e2e/mcp-conformance/policy-template.yaml" >/dev/null + "${ROOT}/e2e/mcp-conformance/policy-template.yaml" \ + "${SPEC_VERSION}" >/dev/null echo "Creating MCP conformance client sandbox ${sandbox_name}..." >&2 if ! "${openshell}" sandbox create \ diff --git a/e2e/mcp-conformance/README.md b/e2e/mcp-conformance/README.md index 398decaa8..8d244ab61 100644 --- a/e2e/mcp-conformance/README.md +++ b/e2e/mcp-conformance/README.md @@ -35,12 +35,7 @@ bridge at `host.openshell.internal` (the alias `e2e/with-docker-gateway.sh` attaches to the CI job container on the e2e network), at `host.docker.internal` on local Docker Desktop, or via `--add-host ...:host-gateway` on local Linux. -The generated policy uses `protocol: mcp` and sets -`mcp.allow_all_known_mcp_methods: true` so omitted rule methods use the endpoint -MCP method profile. That keeps OpenShell deny-by-default at the network boundary -while allowing the upstream scenarios to exercise MCP behavior. The policy body -lives in `policy-template.yaml`; the wrapper renders its host, port, and path -placeholders from the upstream server URL. +The generated policy uses `protocol: mcp`, inserts the conformance runner's spec revision into the endpoint allowlist, and sets `mcp.allow_all_known_mcp_methods: true` so omitted rule methods use the endpoint MCP method profile. OpenShell enforces that allowlist on each non-initialize request using `MCP-Protocol-Version`, with `2025-03-26` as the missing-header fallback. The conformance runner selects the revision used by its client and server; OpenShell's request-version check does not yet provide complete revision-specific message parsing or response validation. The policy keeps OpenShell deny-by-default at the network boundary while allowing the upstream scenarios to exercise MCP behavior. The policy body lives in `policy-template.yaml`; the wrapper renders its MCP revision, host, port, and path placeholders from the upstream server URL. For local runs, the wrapper builds `openshell/supervisor:dev` automatically when no supervisor image override is set. Set `OPENSHELL_DOCKER_SUPERVISOR_IMAGE` diff --git a/e2e/mcp-conformance/client-through-openshell.sh b/e2e/mcp-conformance/client-through-openshell.sh index f0f9d1d4e..a3ab20c0c 100755 --- a/e2e/mcp-conformance/client-through-openshell.sh +++ b/e2e/mcp-conformance/client-through-openshell.sh @@ -51,13 +51,14 @@ POLICY_TEMPLATE="${ROOT}/e2e/mcp-conformance/policy-template.yaml" POLICY_WAIT="${OPENSHELL_MCP_CONFORMANCE_POLICY_WAIT:-0}" POLICY_WAIT_TIMEOUT="${OPENSHELL_MCP_CONFORMANCE_POLICY_WAIT_TIMEOUT:-60}" CLIENT_TIMEOUT_SECONDS="${OPENSHELL_MCP_CONFORMANCE_CLIENT_TIMEOUT_SECONDS:-120}" +MCP_VERSION="${MCP_CONFORMANCE_PROTOCOL_VERSION:-${OPENSHELL_MCP_CONFORMANCE_SPEC_VERSION:-2025-11-25}}" require_active_gateway POLICY_FILE="$(mktemp "${TMPDIR:-/tmp}/openshell-mcp-conformance-policy.XXXXXX.yaml")" trap 'rm -f "${POLICY_FILE}"' EXIT -CLIENT_SERVER_URL="$(python3 "${ROOT}/e2e/mcp-conformance/render-policy.py" "${SERVER_URL}" "${POLICY_FILE}" "${POLICY_TEMPLATE}")" +CLIENT_SERVER_URL="$(python3 "${ROOT}/e2e/mcp-conformance/render-policy.py" "${SERVER_URL}" "${POLICY_FILE}" "${POLICY_TEMPLATE}" "${MCP_VERSION}")" ENV_ARGS=() diff --git a/e2e/mcp-conformance/policy-template.yaml b/e2e/mcp-conformance/policy-template.yaml index 2344bff0c..471e6dd9d 100644 --- a/e2e/mcp-conformance/policy-template.yaml +++ b/e2e/mcp-conformance/policy-template.yaml @@ -44,6 +44,7 @@ ${port_spec} - "192.168.0.0/16" - "fc00::/7" mcp: + versions: [${mcp_version}] max_body_bytes: 131072 allow_all_known_mcp_methods: true rules: diff --git a/e2e/mcp-conformance/render-policy.py b/e2e/mcp-conformance/render-policy.py index 6f7a0db92..0d1b9e5c3 100644 --- a/e2e/mcp-conformance/render-policy.py +++ b/e2e/mcp-conformance/render-policy.py @@ -10,7 +10,7 @@ rewritten) URL the client should connect to. Used both to seed the reusable client sandbox with a placeholder policy and to install the per-scenario policy in client-through-openshell.sh. -Usage: render-policy.py +Usage: render-policy.py """ import json @@ -21,7 +21,18 @@ from ipaddress import ip_address from pathlib import Path from urllib.parse import urlparse, urlunparse -raw_url, policy_file, policy_template = sys.argv[1:4] +if len(sys.argv) != 5: + raise SystemExit( + "usage: render-policy.py " + " " + ) + +raw_url, policy_file, policy_template, mcp_version = sys.argv[1:5] +# Keep this boundary check synchronized with McpProtocolVersion::ALL. The +# renderer is standalone Python, so it cannot import the Rust registry. +supported_mcp_versions = {"2025-03-26", "2025-06-18", "2025-11-25"} +if mcp_version not in supported_mcp_versions: + raise SystemExit(f"unsupported MCP protocol version: {mcp_version!r}") parsed = urlparse(raw_url) if parsed.scheme not in ("http", "https"): @@ -69,6 +80,7 @@ policy = template.substitute( host_spec=f"host: {json.dumps(target_host)}", port_spec=f" port: {port}", path=json.dumps(path), + mcp_version=json.dumps(mcp_version), ) Path(policy_file).write_text(policy, encoding="utf-8") diff --git a/e2e/no-compute-driver-gateway.sh b/e2e/no-compute-driver-gateway.sh index 7ad3896ca..baa4a7d19 100755 --- a/e2e/no-compute-driver-gateway.sh +++ b/e2e/no-compute-driver-gateway.sh @@ -8,11 +8,13 @@ ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" cd "${ROOT}" echo "Building gateway without compiled compute drivers..." -cargo build -p openshell-server --bin openshell-gateway \ +cargo build -p openshell-gateway --bin openshell-gateway \ --no-default-features --features telemetry +cargo check -p openshell-core --no-default-features --all-targets -dependency_tree="$(cargo tree -p openshell-server \ +dependency_tree="$(cargo tree -p openshell-gateway \ --no-default-features --features telemetry --edges normal)" +server_dependency_tree="$(cargo tree -p openshell-server --edges normal)" for driver in \ openshell-driver-docker \ openshell-driver-kubernetes \ @@ -22,7 +24,18 @@ for driver in \ echo "ERROR: driver-free gateway dependency graph contains ${driver}" >&2 exit 1 fi + if grep -q "${driver} v" <<<"${server_dependency_tree}"; then + echo "ERROR: openshell-server dependency graph contains ${driver}" >&2 + exit 1 + fi done +if rg -n \ + 'ComputeDriverKind|openshell_driver_(docker|podman|kubernetes)([^_[:alnum:]]|$)|ComputeRuntime::new_(docker|podman|kubernetes)|VmComputeConfig|compute::vm|driver_config::builtin|libkrun|gvproxy|qemu' \ + crates/openshell-core crates/openshell-server; then + echo "ERROR: backend-specific compute-driver knowledge leaked into core/server" >&2 + exit 1 +fi + "${ROOT}/target/debug/openshell-gateway" --version echo "Driver-free gateway build passed." diff --git a/e2e/policy-advisor/mechanistic-smoke.sh b/e2e/policy-advisor/mechanistic-smoke.sh index d4f360128..4ba780856 100755 --- a/e2e/policy-advisor/mechanistic-smoke.sh +++ b/e2e/policy-advisor/mechanistic-smoke.sh @@ -76,9 +76,10 @@ create_sandbox() { "$OPENSHELL_BIN" sandbox delete "$SANDBOX" >/dev/null 2>&1 || true "$OPENSHELL_BIN" sandbox create \ --name "$SANDBOX" \ + --detach \ --no-auto-providers \ --no-tty \ - -- bash -lc "echo sandbox ready" \ + -- sh -c "exec sleep infinity" \ | sed 's/^/ /' "$OPENSHELL_BIN" sandbox ssh-config "$SANDBOX" > "$SSH_CONFIG" @@ -101,14 +102,20 @@ trigger_l4_deny() { } assert_pending_chunk() { - step "Waiting ${FLUSH_WAIT}s then checking for pending chunk" - sleep "$FLUSH_WAIT" - local output - output="$("$OPENSHELL_BIN" rule get "$SANDBOX" --status pending 2>&1)" + step "Waiting up to ${FLUSH_WAIT}s for pending chunk" + local output="" + local _i + for _i in $(seq 1 "$FLUSH_WAIT"); do + output="$("$OPENSHELL_BIN" rule get "$SANDBOX" --status pending 2>&1)" + if printf '%s\n' "$output" | grep -qi "blocked.invalid"; then + printf '%s\n' "$output" | sed 's/^/ /' + ok "pending mechanistic chunk present for blocked.invalid" + return + fi + sleep 1 + done printf '%s\n' "$output" | sed 's/^/ /' - printf '%s\n' "$output" | grep -qi "blocked.invalid" \ - || fail "no pending chunk for blocked.invalid" - ok "pending mechanistic chunk present for blocked.invalid" + fail "no pending chunk for blocked.invalid after ${FLUSH_WAIT}s" } main() { diff --git a/e2e/policy-advisor/test.sh b/e2e/policy-advisor/test.sh index 431079020..98854b52c 100755 --- a/e2e/policy-advisor/test.sh +++ b/e2e/policy-advisor/test.sh @@ -230,9 +230,7 @@ create_sandbox() { --policy "$POLICY_FILE" \ --upload "${RUNNER_SOURCE}:/sandbox/policy-validation-runner.sh" \ --no-git-ignore \ - --no-auto-providers \ - --no-tty \ - -- bash -lc "chmod +x /sandbox/policy-validation-runner.sh && echo sandbox ready" + --no-auto-providers } connect_ssh() { @@ -245,6 +243,8 @@ connect_ssh() { local i for i in $(seq 1 "$retries"); do if ssh -F "$SSH_CONFIG" "$SSH_HOST" true >/dev/null 2>&1; then + ssh -F "$SSH_CONFIG" "$SSH_HOST" chmod +x /sandbox/policy-validation-runner.sh \ + || fail "could not make uploaded policy runner executable" return fi sleep 2 diff --git a/e2e/policy-advisor/wait-smoke.sh b/e2e/policy-advisor/wait-smoke.sh index a1f2bcf73..006295a47 100755 --- a/e2e/policy-advisor/wait-smoke.sh +++ b/e2e/policy-advisor/wait-smoke.sh @@ -129,8 +129,6 @@ create_sandbox() { --upload "${RUNNER_SOURCE}:/sandbox/runner.sh" \ --no-git-ignore \ --no-auto-providers \ - --no-tty \ - -- bash -lc "chmod +x /sandbox/runner.sh && echo sandbox ready" \ | sed 's/^/ /' "$OPENSHELL_BIN" sandbox ssh-config "$SANDBOX" > "$SSH_CONFIG" @@ -140,6 +138,8 @@ create_sandbox() { local _i for _i in $(seq 1 30); do if ssh -F "$SSH_CONFIG" "$SSH_HOST" true >/dev/null 2>&1; then + ssh -F "$SSH_CONFIG" "$SSH_HOST" chmod +x /sandbox/runner.sh \ + || fail "could not make uploaded runner executable" ok "SSH up" return fi diff --git a/e2e/python/conftest.py b/e2e/python/conftest.py index 9ae1e6291..502f2540e 100644 --- a/e2e/python/conftest.py +++ b/e2e/python/conftest.py @@ -11,7 +11,7 @@ from typing import TYPE_CHECKING import grpc import pytest -from openshell import InferenceRouteClient, Sandbox, SandboxClient, WorkspaceClient +from openshell import Sandbox, SandboxClient, WorkspaceClient if TYPE_CHECKING: from collections.abc import Callable, Iterator @@ -101,11 +101,6 @@ def sandbox(cluster_name: str | None) -> Callable[..., Sandbox]: return _create -@pytest.fixture(scope="session") -def inference_client(sandbox_client: SandboxClient) -> InferenceRouteClient: - return InferenceRouteClient.from_sandbox_client(sandbox_client) - - @pytest.fixture(scope="session") def workspace_client(sandbox_client: SandboxClient) -> WorkspaceClient: return WorkspaceClient.from_sandbox_client(sandbox_client) diff --git a/e2e/python/oidc/oidc_auth_test.py b/e2e/python/oidc/oidc_auth_test.py index 1ed8d56fc..06b5199a6 100644 --- a/e2e/python/oidc/oidc_auth_test.py +++ b/e2e/python/oidc/oidc_auth_test.py @@ -51,11 +51,12 @@ class TestRbac: token = get_token("admin@test", "admin", scopes="openid openshell:all") stub, metadata = stub_with_token(token) req = openshell_pb2.CreateProviderRequest( + workspace_scope=datamodel_pb2.WorkspaceSelector(workspace="default"), provider=datamodel_pb2.Provider( metadata=datamodel_pb2.ObjectMeta(name="e2e-oidc-admin-test"), type="claude", - credentials={"API_KEY": "test-value"}, - ) + credentials={"ANTHROPIC_API_KEY": "test-value"}, + ), ) try: stub.CreateProvider(req, metadata=metadata) @@ -67,7 +68,12 @@ class TestRbac: finally: with contextlib.suppress(grpc.RpcError): stub.DeleteProvider( - openshell_pb2.DeleteProviderRequest(name="e2e-oidc-admin-test"), + openshell_pb2.DeleteProviderRequest( + workspace_scope=datamodel_pb2.WorkspaceSelector( + workspace="default" + ), + name="e2e-oidc-admin-test", + ), metadata=metadata, ) @@ -75,11 +81,12 @@ class TestRbac: token = get_token("user@test", "user", scopes="openid openshell:all") stub, metadata = stub_with_token(token) req = openshell_pb2.CreateProviderRequest( + workspace_scope=datamodel_pb2.WorkspaceSelector(workspace="default"), provider=datamodel_pb2.Provider( metadata=datamodel_pb2.ObjectMeta(name="e2e-oidc-user-blocked"), type="claude", - credentials={"API_KEY": "test-value"}, - ) + credentials={"ANTHROPIC_API_KEY": "test-value"}, + ), ) with pytest.raises(grpc.RpcError) as exc_info: stub.CreateProvider(req, metadata=metadata) @@ -103,7 +110,10 @@ class TestRbac: ) try: user_stub.ListSandboxes( - openshell_pb2.ListSandboxesRequest(), metadata=user_md + openshell_pb2.ListSandboxesRequest( + workspace_scope=datamodel_pb2.WorkspaceSelector(workspace="default") + ), + metadata=user_md, ) finally: with contextlib.suppress(grpc.RpcError): @@ -118,7 +128,11 @@ class TestRbac: channel = grpc_channel() stub = openshell_pb2_grpc.OpenShellStub(channel) with pytest.raises(grpc.RpcError) as exc_info: - stub.ListSandboxes(openshell_pb2.ListSandboxesRequest()) + stub.ListSandboxes( + openshell_pb2.ListSandboxesRequest( + workspace_scope=datamodel_pb2.WorkspaceSelector(workspace="default") + ) + ) assert exc_info.value.code() in ( grpc.StatusCode.UNAUTHENTICATED, grpc.StatusCode.PERMISSION_DENIED, @@ -151,7 +165,12 @@ class TestScopes: "admin@test", "admin", scopes="openid sandbox:read sandbox:write" ) stub, metadata = stub_with_token(token) - stub.ListSandboxes(openshell_pb2.ListSandboxesRequest(), metadata=metadata) + stub.ListSandboxes( + openshell_pb2.ListSandboxesRequest( + workspace_scope=datamodel_pb2.WorkspaceSelector(workspace="default") + ), + metadata=metadata, + ) def test_sandbox_scoped_token_cannot_list_providers(self) -> None: token = get_token( @@ -159,21 +178,41 @@ class TestScopes: ) stub, metadata = stub_with_token(token) with pytest.raises(grpc.RpcError) as exc_info: - stub.ListProviders(openshell_pb2.ListProvidersRequest(), metadata=metadata) + stub.ListProviders( + openshell_pb2.ListProvidersRequest( + workspace_scope=datamodel_pb2.WorkspaceSelector(workspace="default") + ), + metadata=metadata, + ) assert exc_info.value.code() == grpc.StatusCode.PERMISSION_DENIED assert "provider:read" in exc_info.value.details() def test_openshell_all_grants_full_access(self) -> None: token = get_token("admin@test", "admin", scopes="openid openshell:all") stub, metadata = stub_with_token(token) - stub.ListSandboxes(openshell_pb2.ListSandboxesRequest(), metadata=metadata) - stub.ListProviders(openshell_pb2.ListProvidersRequest(), metadata=metadata) + stub.ListSandboxes( + openshell_pb2.ListSandboxesRequest( + workspace_scope=datamodel_pb2.WorkspaceSelector(workspace="default") + ), + metadata=metadata, + ) + stub.ListProviders( + openshell_pb2.ListProvidersRequest( + workspace_scope=datamodel_pb2.WorkspaceSelector(workspace="default") + ), + metadata=metadata, + ) def test_no_openshell_scopes_denied(self) -> None: token = get_token("admin@test", "admin") stub, metadata = stub_with_token(token) with pytest.raises(grpc.RpcError) as exc_info: - stub.ListSandboxes(openshell_pb2.ListSandboxesRequest(), metadata=metadata) + stub.ListSandboxes( + openshell_pb2.ListSandboxesRequest( + workspace_scope=datamodel_pb2.WorkspaceSelector(workspace="default") + ), + metadata=metadata, + ) assert exc_info.value.code() == grpc.StatusCode.PERMISSION_DENIED diff --git a/e2e/python/oidc/workspace_authz_test.py b/e2e/python/oidc/workspace_authz_test.py index 8336b7907..fe8ad4bce 100644 --- a/e2e/python/oidc/workspace_authz_test.py +++ b/e2e/python/oidc/workspace_authz_test.py @@ -26,13 +26,11 @@ if TYPE_CHECKING: from openshell._proto import ( datamodel_pb2, - inference_pb2, - inference_pb2_grpc, openshell_pb2, openshell_pb2_grpc, ) -from .helpers import extract_sub, get_token, grpc_channel, stub_with_token +from .helpers import extract_sub, get_token, stub_with_token WS = "e2e-authz-test" @@ -178,7 +176,7 @@ def _workspace_rpcs() -> list[tuple[str, Callable]]: "CreateSandbox", lambda s, m: s.CreateSandbox( openshell_pb2.CreateSandboxRequest( - workspace=WS, + workspace_scope=datamodel_pb2.WorkspaceSelector(workspace=WS), spec=openshell_pb2.SandboxSpec( template=openshell_pb2.SandboxTemplate(image="ubuntu:24.04") ), @@ -189,20 +187,29 @@ def _workspace_rpcs() -> list[tuple[str, Callable]]: ( "GetSandbox", lambda s, m: s.GetSandbox( - openshell_pb2.GetSandboxRequest(name="nonexistent", workspace=WS), + openshell_pb2.GetSandboxRequest( + name="nonexistent", + workspace_scope=datamodel_pb2.WorkspaceSelector(workspace=WS), + ), metadata=m, ), ), ( "ListSandboxes", lambda s, m: s.ListSandboxes( - openshell_pb2.ListSandboxesRequest(workspace=WS), metadata=m + openshell_pb2.ListSandboxesRequest( + workspace_scope=datamodel_pb2.WorkspaceSelector(workspace=WS) + ), + metadata=m, ), ), ( "DeleteSandbox", lambda s, m: s.DeleteSandbox( - openshell_pb2.DeleteSandboxRequest(name="nonexistent", workspace=WS), + openshell_pb2.DeleteSandboxRequest( + name="nonexistent", + workspace_scope=datamodel_pb2.WorkspaceSelector(workspace=WS), + ), metadata=m, ), ), @@ -210,7 +217,8 @@ def _workspace_rpcs() -> list[tuple[str, Callable]]: "ListSandboxProviders", lambda s, m: s.ListSandboxProviders( openshell_pb2.ListSandboxProvidersRequest( - sandbox_name="nonexistent", workspace=WS + sandbox_name="nonexistent", + workspace_scope=datamodel_pb2.WorkspaceSelector(workspace=WS), ), metadata=m, ), @@ -221,7 +229,7 @@ def _workspace_rpcs() -> list[tuple[str, Callable]]: openshell_pb2.AttachSandboxProviderRequest( sandbox_name="nonexistent", provider_name="nonexistent", - workspace=WS, + workspace_scope=datamodel_pb2.WorkspaceSelector(workspace=WS), ), metadata=m, ), @@ -232,7 +240,7 @@ def _workspace_rpcs() -> list[tuple[str, Callable]]: openshell_pb2.DetachSandboxProviderRequest( sandbox_name="nonexistent", provider_name="nonexistent", - workspace=WS, + workspace_scope=datamodel_pb2.WorkspaceSelector(workspace=WS), ), metadata=m, ), @@ -242,13 +250,13 @@ def _workspace_rpcs() -> list[tuple[str, Callable]]: "CreateProvider", lambda s, m: s.CreateProvider( openshell_pb2.CreateProviderRequest( - workspace=WS, + workspace_scope=datamodel_pb2.WorkspaceSelector(workspace=WS), provider=datamodel_pb2.Provider( metadata=datamodel_pb2.ObjectMeta( name="authz-test", workspace=WS ), type="claude", - credentials={"K": "v"}, + credentials={"ANTHROPIC_API_KEY": "v"}, ), ), metadata=m, @@ -257,27 +265,33 @@ def _workspace_rpcs() -> list[tuple[str, Callable]]: ( "GetProvider", lambda s, m: s.GetProvider( - openshell_pb2.GetProviderRequest(name="nonexistent", workspace=WS), + openshell_pb2.GetProviderRequest( + name="nonexistent", + workspace_scope=datamodel_pb2.WorkspaceSelector(workspace=WS), + ), metadata=m, ), ), ( "ListProviders", lambda s, m: s.ListProviders( - openshell_pb2.ListProvidersRequest(workspace=WS), metadata=m + openshell_pb2.ListProvidersRequest( + workspace_scope=datamodel_pb2.WorkspaceSelector(workspace=WS) + ), + metadata=m, ), ), ( "UpdateProvider", lambda s, m: s.UpdateProvider( openshell_pb2.UpdateProviderRequest( - workspace=WS, + workspace_scope=datamodel_pb2.WorkspaceSelector(workspace=WS), provider=datamodel_pb2.Provider( metadata=datamodel_pb2.ObjectMeta( name="nonexistent", workspace=WS ), type="claude", - credentials={"K": "v"}, + credentials={"ANTHROPIC_API_KEY": "v"}, ), ), metadata=m, @@ -286,7 +300,10 @@ def _workspace_rpcs() -> list[tuple[str, Callable]]: ( "DeleteProvider", lambda s, m: s.DeleteProvider( - openshell_pb2.DeleteProviderRequest(name="nonexistent", workspace=WS), + openshell_pb2.DeleteProviderRequest( + name="nonexistent", + workspace_scope=datamodel_pb2.WorkspaceSelector(workspace=WS), + ), metadata=m, ), ), @@ -339,7 +356,8 @@ def _workspace_rpcs() -> list[tuple[str, Callable]]: "GetProviderRefreshStatus", lambda s, m: s.GetProviderRefreshStatus( openshell_pb2.GetProviderRefreshStatusRequest( - provider="nonexistent", workspace=WS + provider="nonexistent", + workspace_scope=datamodel_pb2.WorkspaceSelector(workspace=WS), ), metadata=m, ), @@ -351,7 +369,7 @@ def _workspace_rpcs() -> list[tuple[str, Callable]]: provider="nonexistent", credential_key="k", strategy=openshell_pb2.PROVIDER_CREDENTIAL_REFRESH_STRATEGY_STATIC, - workspace=WS, + workspace_scope=datamodel_pb2.WorkspaceSelector(workspace=WS), ), metadata=m, ), @@ -362,7 +380,7 @@ def _workspace_rpcs() -> list[tuple[str, Callable]]: openshell_pb2.RotateProviderCredentialRequest( provider="nonexistent", credential_key="k", - workspace=WS, + workspace_scope=datamodel_pb2.WorkspaceSelector(workspace=WS), ), metadata=m, ), @@ -373,7 +391,7 @@ def _workspace_rpcs() -> list[tuple[str, Callable]]: openshell_pb2.DeleteProviderRefreshRequest( provider="nonexistent", credential_key="k", - workspace=WS, + workspace_scope=datamodel_pb2.WorkspaceSelector(workspace=WS), ), metadata=m, ), @@ -386,7 +404,7 @@ def _workspace_rpcs() -> list[tuple[str, Callable]]: sandbox="nonexistent", service="svc", target_port=8080, - workspace=WS, + workspace_scope=datamodel_pb2.WorkspaceSelector(workspace=WS), ), metadata=m, ), @@ -395,7 +413,9 @@ def _workspace_rpcs() -> list[tuple[str, Callable]]: "GetService", lambda s, m: s.GetService( openshell_pb2.GetServiceRequest( - sandbox="nonexistent", service="svc", workspace=WS + sandbox="nonexistent", + service="svc", + workspace_scope=datamodel_pb2.WorkspaceSelector(workspace=WS), ), metadata=m, ), @@ -403,14 +423,19 @@ def _workspace_rpcs() -> list[tuple[str, Callable]]: ( "ListServices", lambda s, m: s.ListServices( - openshell_pb2.ListServicesRequest(workspace=WS), metadata=m + openshell_pb2.ListServicesRequest( + workspace_scope=datamodel_pb2.WorkspaceSelector(workspace=WS) + ), + metadata=m, ), ), ( "DeleteService", lambda s, m: s.DeleteService( openshell_pb2.DeleteServiceRequest( - sandbox="nonexistent", service="svc", workspace=WS + sandbox="nonexistent", + service="svc", + workspace_scope=datamodel_pb2.WorkspaceSelector(workspace=WS), ), metadata=m, ), @@ -420,7 +445,8 @@ def _workspace_rpcs() -> list[tuple[str, Callable]]: "GetSandboxPolicyStatus", lambda s, m: s.GetSandboxPolicyStatus( openshell_pb2.GetSandboxPolicyStatusRequest( - name="nonexistent", workspace=WS + name="nonexistent", + workspace_scope=datamodel_pb2.WorkspaceSelector(workspace=WS), ), metadata=m, ), @@ -429,7 +455,8 @@ def _workspace_rpcs() -> list[tuple[str, Callable]]: "ListSandboxPolicies", lambda s, m: s.ListSandboxPolicies( openshell_pb2.ListSandboxPoliciesRequest( - name="nonexistent", workspace=WS + name="nonexistent", + workspace_scope=datamodel_pb2.WorkspaceSelector(workspace=WS), ), metadata=m, ), @@ -437,7 +464,10 @@ def _workspace_rpcs() -> list[tuple[str, Callable]]: ( "GetDraftPolicy", lambda s, m: s.GetDraftPolicy( - openshell_pb2.GetDraftPolicyRequest(name="nonexistent", workspace=WS), + openshell_pb2.GetDraftPolicyRequest( + name="nonexistent", + workspace_scope=datamodel_pb2.WorkspaceSelector(workspace=WS), + ), metadata=m, ), ), @@ -445,7 +475,9 @@ def _workspace_rpcs() -> list[tuple[str, Callable]]: "ApproveDraftChunk", lambda s, m: s.ApproveDraftChunk( openshell_pb2.ApproveDraftChunkRequest( - name="nonexistent", chunk_id="x", workspace=WS + name="nonexistent", + chunk_id="x", + workspace_scope=datamodel_pb2.WorkspaceSelector(workspace=WS), ), metadata=m, ), @@ -454,7 +486,9 @@ def _workspace_rpcs() -> list[tuple[str, Callable]]: "RejectDraftChunk", lambda s, m: s.RejectDraftChunk( openshell_pb2.RejectDraftChunkRequest( - name="nonexistent", chunk_id="x", workspace=WS + name="nonexistent", + chunk_id="x", + workspace_scope=datamodel_pb2.WorkspaceSelector(workspace=WS), ), metadata=m, ), @@ -463,7 +497,8 @@ def _workspace_rpcs() -> list[tuple[str, Callable]]: "ApproveAllDraftChunks", lambda s, m: s.ApproveAllDraftChunks( openshell_pb2.ApproveAllDraftChunksRequest( - name="nonexistent", workspace=WS + name="nonexistent", + workspace_scope=datamodel_pb2.WorkspaceSelector(workspace=WS), ), metadata=m, ), @@ -472,7 +507,9 @@ def _workspace_rpcs() -> list[tuple[str, Callable]]: "EditDraftChunk", lambda s, m: s.EditDraftChunk( openshell_pb2.EditDraftChunkRequest( - name="nonexistent", chunk_id="x", workspace=WS + name="nonexistent", + chunk_id="x", + workspace_scope=datamodel_pb2.WorkspaceSelector(workspace=WS), ), metadata=m, ), @@ -481,7 +518,9 @@ def _workspace_rpcs() -> list[tuple[str, Callable]]: "UndoDraftChunk", lambda s, m: s.UndoDraftChunk( openshell_pb2.UndoDraftChunkRequest( - name="nonexistent", chunk_id="x", workspace=WS + name="nonexistent", + chunk_id="x", + workspace_scope=datamodel_pb2.WorkspaceSelector(workspace=WS), ), metadata=m, ), @@ -489,39 +528,23 @@ def _workspace_rpcs() -> list[tuple[str, Callable]]: ( "ClearDraftChunks", lambda s, m: s.ClearDraftChunks( - openshell_pb2.ClearDraftChunksRequest(name="nonexistent", workspace=WS), + openshell_pb2.ClearDraftChunksRequest( + name="nonexistent", + workspace_scope=datamodel_pb2.WorkspaceSelector(workspace=WS), + ), metadata=m, ), ), ( "GetDraftHistory", lambda s, m: s.GetDraftHistory( - openshell_pb2.GetDraftHistoryRequest(name="nonexistent", workspace=WS), - metadata=m, - ), - ), - # ── Inference domain ── - ( - "SetInferenceRoute", - lambda _s, m: _inference_stub().SetInferenceRoute( - inference_pb2.SetInferenceRouteRequest( - provider_name="nonexistent", workspace=WS + openshell_pb2.GetDraftHistoryRequest( + name="nonexistent", + workspace_scope=datamodel_pb2.WorkspaceSelector(workspace=WS), ), metadata=m, ), ), - ( - "GetInferenceRoute", - lambda _s, m: _inference_stub().GetInferenceRoute( - inference_pb2.GetInferenceRouteRequest(workspace=WS), metadata=m - ), - ), - ( - "DeleteInferenceRoute", - lambda _s, m: _inference_stub().DeleteInferenceRoute( - inference_pb2.DeleteInferenceRouteRequest(workspace=WS), metadata=m - ), - ), ] @@ -582,34 +605,20 @@ def _global_policy_read_rpcs() -> list[tuple[str, Callable]]: ( "GetSandboxPolicyStatus", lambda s, m: s.GetSandboxPolicyStatus( - openshell_pb2.GetSandboxPolicyStatusRequest( - workspace="", **{"global": True} - ), + openshell_pb2.GetSandboxPolicyStatusRequest(**{"global": True}), metadata=m, ), ), ( "ListSandboxPolicies", lambda s, m: s.ListSandboxPolicies( - openshell_pb2.ListSandboxPoliciesRequest( - workspace="", **{"global": True} - ), + openshell_pb2.ListSandboxPoliciesRequest(**{"global": True}), metadata=m, ), ), ] -_cached_inference_stub: inference_pb2_grpc.InferenceStub | None = None - - -def _inference_stub() -> inference_pb2_grpc.InferenceStub: - global _cached_inference_stub - if _cached_inference_stub is None: - _cached_inference_stub = inference_pb2_grpc.InferenceStub(grpc_channel()) - return _cached_inference_stub - - # ── Test class ─────────────────────────────────────────────────────────── @@ -660,13 +669,15 @@ class TestWorkspaceAuthorization: with contextlib.suppress(grpc.RpcError): stub.CreateProvider( openshell_pb2.CreateProviderRequest( - workspace=workspace, + workspace_scope=datamodel_pb2.WorkspaceSelector( + workspace=workspace + ), provider=datamodel_pb2.Provider( metadata=datamodel_pb2.ObjectMeta( name=prov_name, workspace=workspace ), type="claude", - credentials={"API_KEY": "test"}, + credentials={"ANTHROPIC_API_KEY": "test"}, ), ), metadata=metadata, @@ -675,7 +686,10 @@ class TestWorkspaceAuthorization: with contextlib.suppress(grpc.RpcError): stub.DeleteProvider( openshell_pb2.DeleteProviderRequest( - name=prov_name, workspace=workspace + name=prov_name, + workspace_scope=datamodel_pb2.WorkspaceSelector( + workspace=workspace + ), ), metadata=metadata, ) @@ -707,7 +721,10 @@ class TestWorkspaceAuthorization: stub, metadata, user_sub = user_ctx with pytest.raises(grpc.RpcError) as exc_info: stub.UpdateConfig( - openshell_pb2.UpdateConfigRequest(name="nonexistent", workspace=WS), + openshell_pb2.UpdateConfigRequest( + name="nonexistent", + workspace_scope=datamodel_pb2.WorkspaceSelector(workspace=WS), + ), metadata=metadata, ) _assert_non_member_denial( @@ -747,7 +764,9 @@ class TestWorkspaceAuthorization: response = admin_stub.CreateSandbox( openshell_pb2.CreateSandboxRequest( name=sandbox_name, - workspace=other_workspace, + workspace_scope=datamodel_pb2.WorkspaceSelector( + workspace=other_workspace + ), spec=openshell_pb2.SandboxSpec(), ), metadata=admin_md, @@ -758,7 +777,7 @@ class TestWorkspaceAuthorization: user_stub.GetSandboxLogs( openshell_pb2.GetSandboxLogsRequest( sandbox_id=sandbox_id, - workspace=WS, + workspace_scope=datamodel_pb2.WorkspaceSelector(workspace=WS), ), metadata=user_md, ) @@ -774,7 +793,9 @@ class TestWorkspaceAuthorization: admin_stub.DeleteSandbox( openshell_pb2.DeleteSandboxRequest( name=sandbox_name, - workspace=other_workspace, + workspace_scope=datamodel_pb2.WorkspaceSelector( + workspace=other_workspace + ), ), metadata=admin_md, ) @@ -807,7 +828,6 @@ class TestWorkspaceAuthorization: with pytest.raises(grpc.RpcError) as exc_info: user_stub.UpdateConfig( openshell_pb2.UpdateConfigRequest( - workspace="", setting_key="log_level", delete_setting=True, **{"global": True}, @@ -858,7 +878,9 @@ class TestWorkspaceAuthorization: ) -> None: stub, metadata = admin_ctx stub.ListSandboxes( - openshell_pb2.ListSandboxesRequest(workspace=WS), + openshell_pb2.ListSandboxesRequest( + workspace_scope=datamodel_pb2.WorkspaceSelector(workspace=WS) + ), metadata=metadata, ) @@ -869,7 +891,10 @@ class TestWorkspaceAuthorization: ) -> None: stub, metadata = admin_ctx resp = stub.GetProvider( - openshell_pb2.GetProviderRequest(name=seed_provider, workspace=WS), + openshell_pb2.GetProviderRequest( + name=seed_provider, + workspace_scope=datamodel_pb2.WorkspaceSelector(workspace=WS), + ), metadata=metadata, ) assert resp.provider.metadata.name == seed_provider @@ -880,7 +905,9 @@ class TestWorkspaceAuthorization: ) -> None: stub, metadata = admin_ctx stub.ListServices( - openshell_pb2.ListServicesRequest(workspace=WS), + openshell_pb2.ListServicesRequest( + workspace_scope=datamodel_pb2.WorkspaceSelector(workspace=WS) + ), metadata=metadata, ) @@ -892,7 +919,10 @@ class TestWorkspaceAuthorization: # May fail with NOT_FOUND for the sandbox name, but should not fail with PERMISSION_DENIED try: stub.GetDraftHistory( - openshell_pb2.GetDraftHistoryRequest(name="nonexistent", workspace=WS), + openshell_pb2.GetDraftHistoryRequest( + name="nonexistent", + workspace_scope=datamodel_pb2.WorkspaceSelector(workspace=WS), + ), metadata=metadata, ) except grpc.RpcError as e: @@ -922,26 +952,35 @@ class TestWorkspaceAuthorization: # ListSandboxes user_stub.ListSandboxes( - openshell_pb2.ListSandboxesRequest(workspace=WS), + openshell_pb2.ListSandboxesRequest( + workspace_scope=datamodel_pb2.WorkspaceSelector(workspace=WS) + ), metadata=user_md, ) # GetProvider resp = user_stub.GetProvider( - openshell_pb2.GetProviderRequest(name=seed_provider, workspace=WS), + openshell_pb2.GetProviderRequest( + name=seed_provider, + workspace_scope=datamodel_pb2.WorkspaceSelector(workspace=WS), + ), metadata=user_md, ) assert resp.provider.metadata.name == seed_provider # ListProviders user_stub.ListProviders( - openshell_pb2.ListProvidersRequest(workspace=WS), + openshell_pb2.ListProvidersRequest( + workspace_scope=datamodel_pb2.WorkspaceSelector(workspace=WS) + ), metadata=user_md, ) # ListServices user_stub.ListServices( - openshell_pb2.ListServicesRequest(workspace=WS), + openshell_pb2.ListServicesRequest( + workspace_scope=datamodel_pb2.WorkspaceSelector(workspace=WS) + ), metadata=user_md, ) @@ -971,13 +1010,13 @@ class TestWorkspaceAuthorization: with pytest.raises(grpc.RpcError) as exc_info: user_stub.CreateProvider( openshell_pb2.CreateProviderRequest( - workspace=WS, + workspace_scope=datamodel_pb2.WorkspaceSelector(workspace=WS), provider=datamodel_pb2.Provider( metadata=datamodel_pb2.ObjectMeta( name="user-blocked", workspace=WS ), type="claude", - credentials={"K": "v"}, + credentials={"ANTHROPIC_API_KEY": "v"}, ), ), metadata=user_md, @@ -1012,7 +1051,7 @@ class TestWorkspaceAuthorization: openshell_pb2.ApproveDraftChunkRequest( name="nonexistent", chunk_id="x", - workspace=WS, + workspace_scope=datamodel_pb2.WorkspaceSelector(workspace=WS), ), metadata=user_md, ) @@ -1042,11 +1081,11 @@ class TestWorkspaceAuthorization: try: user_stub.CreateProvider( openshell_pb2.CreateProviderRequest( - workspace=WS, + workspace_scope=datamodel_pb2.WorkspaceSelector(workspace=WS), provider=datamodel_pb2.Provider( metadata=datamodel_pb2.ObjectMeta(name=prov_name, workspace=WS), type="claude", - credentials={"K": "v"}, + credentials={"ANTHROPIC_API_KEY": "v"}, ), ), metadata=user_md, @@ -1065,7 +1104,10 @@ class TestWorkspaceAuthorization: finally: with contextlib.suppress(grpc.RpcError): admin_stub.DeleteProvider( - openshell_pb2.DeleteProviderRequest(name=prov_name, workspace=WS), + openshell_pb2.DeleteProviderRequest( + name=prov_name, + workspace_scope=datamodel_pb2.WorkspaceSelector(workspace=WS), + ), metadata=admin_md, ) _remove_member(admin_stub, admin_md, WS, user_sub) @@ -1086,21 +1128,33 @@ class TestWorkspaceAuthorization: try: with pytest.raises(grpc.RpcError) as exc_info: user_stub.ListSandboxes( - openshell_pb2.ListSandboxesRequest(all_workspaces=True), + openshell_pb2.ListSandboxesRequest( + workspace_scope=datamodel_pb2.WorkspaceSelector( + all_workspaces=datamodel_pb2.AllWorkspaces() + ) + ), metadata=user_md, ) assert exc_info.value.code() == grpc.StatusCode.PERMISSION_DENIED with pytest.raises(grpc.RpcError) as exc_info: user_stub.ListProviders( - openshell_pb2.ListProvidersRequest(all_workspaces=True), + openshell_pb2.ListProvidersRequest( + workspace_scope=datamodel_pb2.WorkspaceSelector( + all_workspaces=datamodel_pb2.AllWorkspaces() + ) + ), metadata=user_md, ) assert exc_info.value.code() == grpc.StatusCode.PERMISSION_DENIED with pytest.raises(grpc.RpcError) as exc_info: user_stub.ListServices( - openshell_pb2.ListServicesRequest(all_workspaces=True), + openshell_pb2.ListServicesRequest( + workspace_scope=datamodel_pb2.WorkspaceSelector( + all_workspaces=datamodel_pb2.AllWorkspaces() + ) + ), metadata=user_md, ) assert exc_info.value.code() == grpc.StatusCode.PERMISSION_DENIED diff --git a/e2e/python/test_inference_routing.py b/e2e/python/test_inference_routing.py deleted file mode 100644 index 0e971f5b8..000000000 --- a/e2e/python/test_inference_routing.py +++ /dev/null @@ -1,375 +0,0 @@ -# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -# SPDX-License-Identifier: Apache-2.0 - -"""E2E tests for explicit inference routing via `inference.local`. - -In the new model, sandbox traffic is routed only when the request targets -`inference.local`. There is no implicit catch-all interception for arbitrary -hosts like `api.openai.com`. -""" - -from __future__ import annotations - -import fcntl -from contextlib import contextmanager -from typing import TYPE_CHECKING - -import grpc -import pytest - -from openshell._proto import datamodel_pb2, openshell_pb2, sandbox_pb2 - -if TYPE_CHECKING: - from collections.abc import Callable, Iterator - - from openshell import ( - InferenceRouteClient, - InferenceRouteConfig, - Sandbox, - SandboxClient, - ) - - -_BASE_FILESYSTEM = sandbox_pb2.FilesystemPolicy( - include_workdir=True, - read_only=["/usr", "/lib", "/etc", "/app", "/var/log", "/proc", "/dev/urandom"], - read_write=["/sandbox", "/tmp"], -) -_BASE_LANDLOCK = sandbox_pb2.LandlockPolicy(compatibility="best_effort") -_BASE_PROCESS = sandbox_pb2.ProcessPolicy(run_as_user="sandbox", run_as_group="sandbox") - -pytestmark = pytest.mark.xdist_group("inference-routing") - -_MANAGED_OPENAI_MODEL_ID = "mock/e2e-openai-model" -_MANAGED_OPENAI_PROVIDER_NAME = "e2e-managed-openai" -_INFERENCE_CONFIG_LOCK = "/tmp/openshell-e2e-inference-config.lock" - - -def _baseline_policy() -> sandbox_pb2.SandboxPolicy: - return sandbox_pb2.SandboxPolicy( - version=1, - filesystem=_BASE_FILESYSTEM, - landlock=_BASE_LANDLOCK, - process=_BASE_PROCESS, - ) - - -def _upsert_managed_inference( - inference_client: InferenceRouteClient, - sandbox_client: SandboxClient, - *, - provider_name: str, - provider_type: str, - credential_key: str, - base_url_key: str, - model_id: str, - base_url: str, -) -> None: - provider = datamodel_pb2.Provider( - metadata=datamodel_pb2.ObjectMeta(name=provider_name), - type=provider_type, - credentials={credential_key: "mock"}, - config={ - base_url_key: base_url, - }, - ) - timeout = sandbox_client._timeout - - for _ in range(5): - try: - sandbox_client._stub.UpdateProvider( - openshell_pb2.UpdateProviderRequest( - provider=provider, workspace="default" - ), - timeout=timeout, - ) - break - except grpc.RpcError as exc: - if exc.code() != grpc.StatusCode.NOT_FOUND: - raise - - try: - sandbox_client._stub.CreateProvider( - openshell_pb2.CreateProviderRequest( - provider=provider, workspace="default" - ), - timeout=timeout, - ) - break - except grpc.RpcError as create_exc: - if create_exc.code() == grpc.StatusCode.ALREADY_EXISTS: - continue - raise - else: - raise RuntimeError("failed to upsert managed e2e provider after retries") - - inference_client.set_route( - workspace="default", - provider_name=provider_name, - model_id=model_id, - ) - - -def _current_route( - inference_client: InferenceRouteClient, -) -> InferenceRouteConfig | None: - try: - return inference_client.get_route(workspace="default") - except grpc.RpcError as exc: - if exc.code() == grpc.StatusCode.NOT_FOUND: - return None - raise - - -def _restore_route( - inference_client: InferenceRouteClient, - previous: InferenceRouteConfig | None, -) -> None: - if previous is None: - return - - inference_client.set_route( - workspace="default", - provider_name=previous.provider_name, - model_id=previous.model_id, - # Teardown restores prior shared state as-is, even if the previous - # route is intentionally unreachable or no longer verifiable. - no_verify=True, - ) - - -@contextmanager -def _cluster_config_lock() -> Iterator[None]: - with open(_INFERENCE_CONFIG_LOCK, "a+", encoding="utf-8") as lock_file: - fcntl.flock(lock_file.fileno(), fcntl.LOCK_EX) - try: - yield - finally: - fcntl.flock(lock_file.fileno(), fcntl.LOCK_UN) - - -@pytest.fixture -def managed_openai_route( - inference_client: InferenceRouteClient, - sandbox_client: SandboxClient, -) -> Iterator[str]: - with _cluster_config_lock(): - previous = _current_route(inference_client) - _upsert_managed_inference( - inference_client, - sandbox_client, - provider_name=_MANAGED_OPENAI_PROVIDER_NAME, - provider_type="openai", - credential_key="OPENAI_API_KEY", - base_url_key="OPENAI_BASE_URL", - model_id=_MANAGED_OPENAI_MODEL_ID, - base_url="mock://e2e-managed-openai", - ) - try: - yield _MANAGED_OPENAI_MODEL_ID - finally: - _restore_route(inference_client, previous) - - -def test_model_discovery_call_routed_to_backend( - sandbox: Callable[..., Sandbox], - managed_openai_route: str, -) -> None: - """Model discovery endpoint is treated as an inference protocol.""" - spec = datamodel_pb2.SandboxSpec(policy=_baseline_policy()) - - def call_models() -> str: - import ssl - import urllib.request - - ctx = ssl.create_default_context() - ctx.check_hostname = False - ctx.verify_mode = ssl.CERT_NONE - - req = urllib.request.Request("https://inference.local/v1/models", method="GET") - resp = urllib.request.urlopen(req, timeout=30, context=ctx) - return resp.read().decode() - - with sandbox(spec=spec, delete_on_exit=True) as sb: - result = sb.exec_python(call_models, timeout_seconds=60) - assert result.exit_code == 0, f"stderr: {result.stderr}" - output = result.stdout.strip() - assert "Hello from openshell mock backend" in output - assert managed_openai_route in output - - -def test_inference_call_routed_to_backend( - sandbox: Callable[..., Sandbox], - managed_openai_route: str, -) -> None: - """OpenAI chat request to `inference.local` is intercepted and routed.""" - spec = datamodel_pb2.SandboxSpec(policy=_baseline_policy()) - - def call_chat_completions() -> str: - import json - import ssl - import urllib.request - - body = json.dumps( - { - "model": "test-model", - "messages": [{"role": "user", "content": "hello"}], - } - ).encode() - - req = urllib.request.Request( - "https://inference.local/v1/chat/completions", - data=body, - headers={ - "Content-Type": "application/json", - "Authorization": "Bearer dummy-key", - }, - method="POST", - ) - # The proxy will TLS-terminate, so we need to accept its cert. - ctx = ssl.create_default_context() - ctx.check_hostname = False - ctx.verify_mode = ssl.CERT_NONE - - resp = urllib.request.urlopen(req, timeout=30, context=ctx) - return resp.read().decode() - - with sandbox(spec=spec, delete_on_exit=True) as sb: - result = sb.exec_python(call_chat_completions, timeout_seconds=60) - assert result.exit_code == 0, f"stderr: {result.stderr}" - output = result.stdout.strip() - assert "Hello from openshell mock backend" in output - assert managed_openai_route in output - - -def test_non_inference_request_denied( - sandbox: Callable[..., Sandbox], - managed_openai_route: str, -) -> None: - """Non-inference path on `inference.local` is denied with 403.""" - _ = managed_openai_route - spec = datamodel_pb2.SandboxSpec(policy=_baseline_policy()) - - def make_non_inference_request() -> str: - import ssl - import urllib.error - import urllib.request - - ctx = ssl.create_default_context() - ctx.check_hostname = False - ctx.verify_mode = ssl.CERT_NONE - - try: - req = urllib.request.Request("https://inference.local/v1/not-inference") - urllib.request.urlopen(req, timeout=10, context=ctx) - return "unexpected_success" - except urllib.error.HTTPError as e: - return f"http_error_{e.code}" - except Exception as e: - return f"error: {e}" - - with sandbox(spec=spec, delete_on_exit=True) as sb: - result = sb.exec_python(make_non_inference_request, timeout_seconds=30) - assert result.exit_code == 0, f"stderr: {result.stderr}" - assert result.stdout.strip() == "http_error_403" - - -def test_unsupported_protocol_returns_400( - sandbox: Callable[..., Sandbox], - managed_openai_route: str, -) -> None: - """Protocol mismatch returns 400 when no compatible route exists.""" - _ = managed_openai_route - spec = datamodel_pb2.SandboxSpec(policy=_baseline_policy()) - - def call_anthropic_messages() -> str: - import json - import ssl - import urllib.error - import urllib.request - - body = json.dumps( - { - "model": "claude-test", - "max_tokens": 64, - "messages": [{"role": "user", "content": "hello"}], - } - ).encode() - - req = urllib.request.Request( - "https://inference.local/v1/messages", - data=body, - headers={ - "Content-Type": "application/json", - "x-api-key": "dummy-key", - "anthropic-version": "2023-06-01", - }, - method="POST", - ) - ctx = ssl.create_default_context() - ctx.check_hostname = False - ctx.verify_mode = ssl.CERT_NONE - - try: - resp = urllib.request.urlopen(req, timeout=30, context=ctx) - return resp.read().decode() - except urllib.error.HTTPError as exc: - return ( - f"http_error_{exc.code}:{exc.read().decode('utf-8', errors='replace')}" - ) - - with sandbox(spec=spec, delete_on_exit=True) as sb: - result = sb.exec_python(call_anthropic_messages, timeout_seconds=60) - assert result.exit_code == 0, f"stderr: {result.stderr}" - output = result.stdout.strip() - assert output.startswith("http_error_400"), output - assert "no compatible inference route" in output - - -def test_non_inference_host_is_not_intercepted( - sandbox: Callable[..., Sandbox], - managed_openai_route: str, -) -> None: - """Requests to non-`inference.local` hosts do not get inference routing.""" - _ = managed_openai_route - spec = datamodel_pb2.SandboxSpec(policy=_baseline_policy()) - - def call_external_openai_endpoint() -> str: - import json - import ssl - import urllib.error - import urllib.request - - body = json.dumps( - { - "model": "test-model", - "messages": [{"role": "user", "content": "hello"}], - } - ).encode() - - req = urllib.request.Request( - "https://api.openai.com/v1/chat/completions", - data=body, - headers={ - "Content-Type": "application/json", - "Authorization": "Bearer dummy", - }, - method="POST", - ) - ctx = ssl.create_default_context() - ctx.check_hostname = False - ctx.verify_mode = ssl.CERT_NONE - - try: - resp = urllib.request.urlopen(req, timeout=30, context=ctx) - return resp.read().decode() - except urllib.error.URLError as exc: - return f"url_error:{exc}" - except Exception as exc: - return f"error:{type(exc).__name__}:{exc}" - - with sandbox(spec=spec, delete_on_exit=True) as sb: - result = sb.exec_python(call_external_openai_endpoint, timeout_seconds=60) - assert result.exit_code == 0, f"stderr: {result.stderr}" - output = result.stdout.strip() - assert "Tunnel connection failed: 403 Forbidden" in output diff --git a/e2e/python/test_policy_validation.py b/e2e/python/test_policy_validation.py index 7122d8061..7beb7e146 100644 --- a/e2e/python/test_policy_validation.py +++ b/e2e/python/test_policy_validation.py @@ -10,6 +10,7 @@ overly broad filesystem paths). from __future__ import annotations +import contextlib from typing import TYPE_CHECKING import grpc @@ -68,7 +69,13 @@ def test_create_sandbox_rejects_root_user( stub = sandbox_client._stub with pytest.raises(grpc.RpcError) as exc_info: - stub.CreateSandbox(openshell_pb2.CreateSandboxRequest(name="", spec=spec)) + stub.CreateSandbox( + openshell_pb2.CreateSandboxRequest( + workspace_scope=datamodel_pb2.WorkspaceSelector(workspace="default"), + name="", + spec=spec, + ) + ) assert exc_info.value.code() == grpc.StatusCode.INVALID_ARGUMENT assert "root" in exc_info.value.details().lower() @@ -92,7 +99,13 @@ def test_create_sandbox_rejects_path_traversal( stub = sandbox_client._stub with pytest.raises(grpc.RpcError) as exc_info: - stub.CreateSandbox(openshell_pb2.CreateSandboxRequest(name="", spec=spec)) + stub.CreateSandbox( + openshell_pb2.CreateSandboxRequest( + workspace_scope=datamodel_pb2.WorkspaceSelector(workspace="default"), + name="", + spec=spec, + ) + ) assert exc_info.value.code() == grpc.StatusCode.INVALID_ARGUMENT assert "traversal" in exc_info.value.details().lower() @@ -116,12 +129,72 @@ def test_create_sandbox_rejects_overly_broad_paths( stub = sandbox_client._stub with pytest.raises(grpc.RpcError) as exc_info: - stub.CreateSandbox(openshell_pb2.CreateSandboxRequest(name="", spec=spec)) + stub.CreateSandbox( + openshell_pb2.CreateSandboxRequest( + workspace_scope=datamodel_pb2.WorkspaceSelector(workspace="default"), + name="", + spec=spec, + ) + ) assert exc_info.value.code() == grpc.StatusCode.INVALID_ARGUMENT assert "broad" in exc_info.value.details().lower() +def test_create_sandbox_materializes_default_mcp_version( + sandbox_client: SandboxClient, +) -> None: + """An omitted MCP options stanza is stored with the pinned default version.""" + policy = _safe_policy() + policy.network_policies["mcp_default"].CopyFrom( + sandbox_pb2.NetworkPolicyRule( + name="mcp_default", + endpoints=[ + sandbox_pb2.NetworkEndpoint( + host="mcp.example.com", + port=443, + protocol="mcp", + rules=[ + sandbox_pb2.L7Rule( + allow=sandbox_pb2.L7Allow(method="initialize") + ) + ], + ) + ], + ) + ) + assert not policy.network_policies["mcp_default"].endpoints[0].HasField("mcp") + spec = datamodel_pb2.SandboxSpec(policy=policy) + + created = sandbox_client.create(workspace="default", spec=spec) + try: + stored = sandbox_client._stub.GetSandbox( + openshell_pb2.GetSandboxRequest( + name=created.name, + workspace_scope=datamodel_pb2.WorkspaceSelector(workspace="default"), + ) + ) + stored_endpoint = stored.sandbox.spec.policy.network_policies[ + "mcp_default" + ].endpoints[0] + + assert stored_endpoint.HasField("mcp") + assert list(stored_endpoint.mcp.versions) == ["2025-11-25"] + + config = sandbox_client._stub.GetSandboxConfig( + sandbox_pb2.GetSandboxConfigRequest(sandbox_id=created.id) + ) + endpoint = config.policy.network_policies["mcp_default"].endpoints[0] + + assert endpoint.HasField("mcp") + assert list(endpoint.mcp.versions) == ["2025-11-25"] + finally: + # Creation can complete before the compute driver reaches Ready, so + # cleanup targets the gateway resource directly by its canonical name. + with contextlib.suppress(grpc.RpcError): + sandbox_client.delete(created.name, workspace="default") + + def test_update_policy_rejects_immutable_fields( sandbox: Callable[..., Sandbox], sandbox_client: SandboxClient, @@ -155,6 +228,9 @@ def test_update_policy_rejects_immutable_fields( with pytest.raises(grpc.RpcError) as exc_info: stub.UpdateConfig( openshell_pb2.UpdateConfigRequest( + workspace_scope=datamodel_pb2.WorkspaceSelector( + workspace="default" + ), name=sandbox_name, policy=unsafe_policy, ) diff --git a/e2e/python/test_sandbox_providers.py b/e2e/python/test_sandbox_providers.py index 40fd05a12..0b3c9b0a8 100644 --- a/e2e/python/test_sandbox_providers.py +++ b/e2e/python/test_sandbox_providers.py @@ -11,6 +11,11 @@ persisted sandbox spec environment map. from __future__ import annotations +import json +import socket +import subprocess +import sys +import textwrap import time from contextlib import contextmanager from typing import TYPE_CHECKING @@ -70,16 +75,19 @@ def provider( name: str, provider_type: str, credentials: dict[str, str], + profile_workspace: str = "", ) -> Iterator[str]: """Create a provider for the duration of the block, then delete it.""" _delete_provider(stub, name) stub.CreateProvider( openshell_pb2.CreateProviderRequest( + workspace_scope=datamodel_pb2.WorkspaceSelector(workspace="default"), provider=datamodel_pb2.Provider( metadata=datamodel_pb2.ObjectMeta(name=name), type=provider_type, credentials=credentials, - ) + profile_workspace=profile_workspace, + ), ) ) try: @@ -91,7 +99,12 @@ def provider( def _delete_provider(stub: object, name: str) -> None: """Delete a provider, ignoring not-found errors.""" try: - stub.DeleteProvider(openshell_pb2.DeleteProviderRequest(name=name)) + stub.DeleteProvider( + openshell_pb2.DeleteProviderRequest( + workspace_scope=datamodel_pb2.WorkspaceSelector(workspace="default"), + name=name, + ) + ) except grpc.RpcError as exc: if hasattr(exc, "code") and exc.code() == grpc.StatusCode.NOT_FOUND: pass @@ -99,66 +112,194 @@ def _delete_provider(stub: object, name: str) -> None: raise -@pytest.fixture -def providers_v2_enabled( - sandbox_client: SandboxClient, - _gateway_config_guard: None, -) -> Iterator[None]: - """Enable the gateway-global ``providers_v2_enabled`` opt-in for one test. +def _delete_provider_profile(stub: object, profile_id: str) -> None: + """Delete a provider profile, ignoring not-found errors.""" + try: + stub.DeleteProviderProfile( + openshell_pb2.DeleteProviderProfileRequest( + id=profile_id, + workspace="default", + ) + ) + except grpc.RpcError as exc: + if hasattr(exc, "code") and exc.code() == grpc.StatusCode.NOT_FOUND: + pass + else: + raise - Composing a provider's network policy onto a sandbox is gated behind this - setting, which defaults off; the built-in github profile's git-transport - rules only reach the sandbox with it enabled. - The setting is gateway-global. Exclusivity against other xdist workers is - provided by the ``exclusive_gateway_config`` marker plus the autouse - ``_gateway_config_guard`` guard (see conftest): no concurrent worker is - mid-test while this fixture mutates and restores the setting, so none can - observe the transient value. Depending on the guard here also orders the - exclusive lock acquisition before the mutation. - - ``GetGatewayConfig`` returns known keys even when unset, with an empty - ``SettingValue`` (no populated oneof), so the setting is treated as present - only when its value oneof is set; otherwise restore is a delete. ``global`` - is a Python keyword, so it is passed through a dict expansion. - """ - stub = sandbox_client._stub - key = "providers_v2_enabled" - config = stub.GetGatewayConfig(sandbox_pb2.GetGatewayConfigRequest()) - prior_value = sandbox_pb2.SettingValue() - had_prior = ( - key in config.settings - and config.settings[key].WhichOneof("value") is not None - ) - if had_prior: - prior_value.CopyFrom(config.settings[key]) - - stub.UpdateConfig( - openshell_pb2.UpdateConfigRequest( - setting_key=key, - setting_value=sandbox_pb2.SettingValue(bool_value=True), - **{"global": True}, +@contextmanager +def imported_provider_profile( + stub: object, + *, + profile: openshell_pb2.ProviderProfile, + source: str, +) -> Iterator[str]: + """Import a workspace-scoped provider profile for the duration of the block.""" + _delete_provider_profile(stub, profile.id) + response = stub.ImportProviderProfiles( + openshell_pb2.ImportProviderProfilesRequest( + profiles=[ + openshell_pb2.ProviderProfileImportItem( + profile=profile, + source=source, + ) + ], + workspace="default", ) ) + assert response.imported, f"profile import failed: {response.diagnostics!r}" try: - yield + yield profile.id finally: - if had_prior: - stub.UpdateConfig( - openshell_pb2.UpdateConfigRequest( - setting_key=key, - setting_value=prior_value, - **{"global": True}, - ) + _delete_provider_profile(stub, profile.id) + + +def _native_inference_profile( + *, + profile_id: str, + env_var: str, + port: int, + rules: list[sandbox_pb2.L7Rule], + auth_style: str = "bearer", + header_name: str = "authorization", +) -> openshell_pb2.ProviderProfile: + return openshell_pb2.ProviderProfile( + id=profile_id, + display_name=f"{profile_id} display", + description="E2E imported inference profile fixture", + category=openshell_pb2.PROVIDER_PROFILE_CATEGORY_INFERENCE, + inference_capable=True, + credentials=[ + openshell_pb2.ProviderProfileCredential( + name="api_key", + description="API key", + env_vars=[env_var], + required=True, + auth_style=auth_style, + header_name=header_name, ) - else: - stub.UpdateConfig( - openshell_pb2.UpdateConfigRequest( - setting_key=key, - delete_setting=True, - **{"global": True}, - ) + ], + endpoints=[ + sandbox_pb2.NetworkEndpoint( + host="host.openshell.internal", + port=port, + protocol="rest", + tls="none", + enforcement="enforce", + rules=rules, + allowed_ips=[ + "10.0.0.0/8", + "172.16.0.0/12", + "192.168.0.0/16", + "fc00::/7", + ], ) + ], + binaries=[ + sandbox_pb2.NetworkBinary(path="/usr/bin/python*"), + sandbox_pb2.NetworkBinary(path="/usr/local/bin/python*"), + sandbox_pb2.NetworkBinary(path="/sandbox/.uv/python/**/python*"), + ], + ) + + +@contextmanager +def native_endpoint_server() -> Iterator[int]: + """Start a small host-side HTTP fixture that echoes auth and request data.""" + with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as listener: + listener.bind(("127.0.0.1", 0)) + port = listener.getsockname()[1] + + script = textwrap.dedent( + """ + import json + import sys + from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer + + PORT = int(sys.argv[1]) + + class Handler(BaseHTTPRequestHandler): + def _reply(self): + content_length = int(self.headers.get("Content-Length", "0")) + body = self.rfile.read(content_length) if content_length else b"" + payload = { + "method": self.command, + "path": self.path, + "authorization": self.headers.get("Authorization"), + "x_api_key": self.headers.get("x-api-key"), + "body": body.decode("utf-8"), + } + encoded = json.dumps(payload).encode("utf-8") + self.send_response(200) + self.send_header("Content-Type", "application/json") + self.send_header("Content-Length", str(len(encoded))) + self.end_headers() + self.wfile.write(encoded) + + def do_GET(self): + self._reply() + + def do_POST(self): + self._reply() + + def log_message(self, fmt, *args): + pass + + ThreadingHTTPServer(("0.0.0.0", PORT), Handler).serve_forever() + """ + ) + proc = subprocess.Popen( + [sys.executable, "-c", script, str(port)], + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + text=True, + ) + + deadline = time.monotonic() + 20 + while time.monotonic() < deadline: + if proc.poll() is not None: + stdout, stderr = proc.communicate(timeout=5) + raise RuntimeError( + "native endpoint fixture exited early: " + f"stdout={stdout!r} stderr={stderr!r}" + ) + try: + with socket.create_connection(("127.0.0.1", port), timeout=1): + break + except OSError: + time.sleep(0.2) + else: + proc.kill() + stdout, stderr = proc.communicate(timeout=5) + raise RuntimeError( + "native endpoint fixture did not become ready: " + f"stdout={stdout!r} stderr={stderr!r}" + ) + + try: + yield port + finally: + proc.kill() + proc.communicate(timeout=5) + + +def _proxy_connect(): + """Return a closure that sends a raw CONNECT and returns the status line.""" + + def fn(host, port): + import socket + + conn = socket.create_connection(("10.200.0.1", 3128), timeout=10) + try: + conn.sendall( + f"CONNECT {host}:{port} HTTP/1.1\r\nHost: {host}\r\n\r\n".encode() + ) + return conn.recv(256).decode("latin1") + finally: + conn.close() + + return fn # =========================================================================== @@ -195,38 +336,25 @@ def test_provider_credentials_available_as_env_vars( assert value != "sk-e2e-test-key-12345" -def test_profileless_provider_credentials_fail_closed( - sandbox: Callable[..., Sandbox], +def test_profileless_provider_creation_is_rejected( sandbox_client: SandboxClient, ) -> None: - """Profileless credentials are withheld because they have no endpoint binding.""" - with provider( - sandbox_client._stub, - name="e2e-test-generic-provider-env", - provider_type="generic", - credentials={ - "CUSTOM_SERVICE_TOKEN": "token-generic-123", - "CUSTOM_SERVICE_URL": "https://internal.example.test/api", - }, - ) as provider_name: - spec = datamodel_pb2.SandboxSpec( - policy=_default_policy(), - providers=[provider_name], + """New providers must reference a built-in or imported profile.""" + with pytest.raises(grpc.RpcError) as exc_info: + sandbox_client._stub.CreateProvider( + openshell_pb2.CreateProviderRequest( + workspace_scope=datamodel_pb2.WorkspaceSelector(workspace="default"), + provider=datamodel_pb2.Provider( + metadata=datamodel_pb2.ObjectMeta( + name="e2e-test-profileless-provider" + ), + type="generic", + credentials={"CUSTOM_SERVICE_TOKEN": "token-generic-123"}, + ), + ) ) - - def read_generic_env_vars() -> str: - import os - - token = os.environ.get("CUSTOM_SERVICE_TOKEN", "NOT_SET") - url = os.environ.get("CUSTOM_SERVICE_URL", "NOT_SET") - return f"{token}|{url}" - - with sandbox(spec=spec, delete_on_exit=True) as sb: - result = sb.exec_python(read_generic_env_vars) - assert result.exit_code == 0, result.stderr - token, url = result.stdout.strip().split("|") - assert token == "NOT_SET" - assert url == "NOT_SET" + assert exc_info.value.code() == grpc.StatusCode.INVALID_ARGUMENT + assert "provider profile 'generic' was not found" in exc_info.value.details() def test_endpointless_profile_credentials_fail_closed_without_policy_binding( @@ -326,9 +454,7 @@ def test_nvidia_provider_injects_nvidia_api_key_env_var( with sandbox(spec=spec, delete_on_exit=True) as sb: result = sb.exec_python(read_nvidia_key) assert result.exit_code == 0, result.stderr - assert _is_placeholder_for_env_key( - result.stdout.strip(), "NVIDIA_API_KEY" - ) + assert _is_placeholder_for_env_key(result.stdout.strip(), "NVIDIA_API_KEY") def test_attach_detach_updates_credentials_for_later_exec_launches( @@ -377,6 +503,9 @@ def test_attach_detach_updates_credentials_for_later_exec_launches( try: stub.AttachSandboxProvider( openshell_pb2.AttachSandboxProviderRequest( + workspace_scope=datamodel_pb2.WorkspaceSelector( + workspace="default" + ), sandbox_name=sb.sandbox.name, provider_name=provider_name, ) @@ -388,6 +517,9 @@ def test_attach_detach_updates_credentials_for_later_exec_launches( stub.DetachSandboxProvider( openshell_pb2.DetachSandboxProviderRequest( + workspace_scope=datamodel_pb2.WorkspaceSelector( + workspace="default" + ), sandbox_name=sb.sandbox.name, provider_name=provider_name, ) @@ -397,6 +529,9 @@ def test_attach_detach_updates_credentials_for_later_exec_launches( try: stub.DetachSandboxProvider( openshell_pb2.DetachSandboxProviderRequest( + workspace_scope=datamodel_pb2.WorkspaceSelector( + workspace="default" + ), sandbox_name=sb.sandbox.name, provider_name=provider_name, ) @@ -406,6 +541,193 @@ def test_attach_detach_updates_credentials_for_later_exec_launches( raise +def test_imported_openai_profile_allows_native_endpoint_with_attached_provider( + sandbox: Callable[..., Sandbox], + sandbox_client: SandboxClient, +) -> None: + """Imported fixture profiles should support native OpenAI-style access.""" + stub = sandbox_client._stub + profile_id = f"e2e-native-openai-{int(time.time() * 1000)}" + provider_name = f"{profile_id}-provider" + secret = "sk-native-openai-secret" + + profile = _native_inference_profile( + profile_id=profile_id, + env_var="OPENAI_API_KEY", + port=0, + rules=[ + sandbox_pb2.L7Rule( + allow=sandbox_pb2.L7Allow( + method="POST", + path="/v1/chat/completions", + ) + ) + ], + ) + + def call_native_openai(host: str, port: int) -> str: + import json + import os + import urllib.error + import urllib.request + + body = json.dumps( + { + "model": "fixture-openai-model", + "messages": [{"role": "user", "content": "hello"}], + } + ).encode() + request = urllib.request.Request( + f"http://{host}:{port}/v1/chat/completions", + data=body, + headers={ + "Content-Type": "application/json", + "Authorization": f"Bearer {os.environ['OPENAI_API_KEY']}", + }, + method="POST", + ) + try: + with urllib.request.urlopen(request, timeout=30) as response: + return response.read().decode() + except urllib.error.HTTPError as exc: + raise RuntimeError( + f"native OpenAI request failed with {exc.code}: " + f"{exc.read().decode(errors='replace')}" + ) from exc + + with native_endpoint_server() as port: + profile.endpoints[0].port = port + with imported_provider_profile( + stub, + profile=profile, + source=f"{profile_id}.yaml", + ): + with provider( + stub, + name=provider_name, + provider_type=profile_id, + credentials={"OPENAI_API_KEY": secret}, + profile_workspace="default", + ) as attached_provider: + spec = datamodel_pb2.SandboxSpec( + policy=_default_policy(), + providers=[attached_provider], + ) + with sandbox(spec=spec, delete_on_exit=True) as sb: + result = sb.exec_python( + call_native_openai, + args=("host.openshell.internal", port), + timeout_seconds=60, + ) + assert result.exit_code == 0, result.stderr + payload = json.loads(result.stdout) + body = json.loads(payload["body"]) + assert payload["method"] == "POST" + assert payload["path"] == "/v1/chat/completions" + assert payload["authorization"] == f"Bearer {secret}" + assert body["model"] == "fixture-openai-model" + + +def test_imported_anthropic_profile_uses_native_endpoint_and_inference_local_is_not_privileged( + sandbox: Callable[..., Sandbox], + sandbox_client: SandboxClient, +) -> None: + """Attached imported profiles should not resurrect `inference.local` routing.""" + stub = sandbox_client._stub + profile_id = f"e2e-native-anthropic-{int(time.time() * 1000)}" + provider_name = f"{profile_id}-provider" + secret = "sk-native-anthropic-secret" + + profile = _native_inference_profile( + profile_id=profile_id, + env_var="ANTHROPIC_API_KEY", + port=0, + rules=[ + sandbox_pb2.L7Rule( + allow=sandbox_pb2.L7Allow( + method="POST", + path="/v1/messages", + ) + ) + ], + auth_style="header", + header_name="x-api-key", + ) + + def call_native_anthropic(host: str, port: int) -> str: + import json + import os + import urllib.error + import urllib.request + + body = json.dumps( + { + "model": "fixture-anthropic-model", + "messages": [{"role": "user", "content": "hello"}], + } + ).encode() + request = urllib.request.Request( + f"http://{host}:{port}/v1/messages", + data=body, + headers={ + "Content-Type": "application/json", + "x-api-key": os.environ["ANTHROPIC_API_KEY"], + "anthropic-version": "2023-06-01", + }, + method="POST", + ) + try: + with urllib.request.urlopen(request, timeout=30) as response: + return response.read().decode() + except urllib.error.HTTPError as exc: + raise RuntimeError( + f"native Anthropic request failed with {exc.code}: " + f"{exc.read().decode(errors='replace')}" + ) from exc + + with native_endpoint_server() as port: + profile.endpoints[0].port = port + with imported_provider_profile( + stub, + profile=profile, + source=f"{profile_id}.yaml", + ): + with provider( + stub, + name=provider_name, + provider_type=profile_id, + credentials={"ANTHROPIC_API_KEY": secret}, + profile_workspace="default", + ) as attached_provider: + spec = datamodel_pb2.SandboxSpec( + policy=_default_policy(), + providers=[attached_provider], + ) + with sandbox(spec=spec, delete_on_exit=True) as sb: + native_result = sb.exec_python( + call_native_anthropic, + args=("host.openshell.internal", port), + timeout_seconds=60, + ) + assert native_result.exit_code == 0, native_result.stderr + payload = json.loads(native_result.stdout) + body = json.loads(payload["body"]) + assert payload["method"] == "POST" + assert payload["path"] == "/v1/messages" + assert payload["x_api_key"] == secret + assert body["model"] == "fixture-anthropic-model" + + denied = sb.exec_python( + _proxy_connect(), + args=("inference.local", 443), + timeout_seconds=30, + ) + assert denied.exit_code == 0, denied.stderr + status = denied.stdout.strip() + assert status.startswith("HTTP/1.1 "), status + assert " 200 " not in status, status + + # =========================================================================== # Tests: security & edge cases # =========================================================================== @@ -444,7 +766,12 @@ def test_credentials_not_in_persisted_spec_environment( with sandbox(spec=spec, delete_on_exit=True) as sb: fetched = sandbox_client._stub.GetSandbox( - openshell_pb2.GetSandboxRequest(name=sb.sandbox.name) + openshell_pb2.GetSandboxRequest( + workspace_scope=datamodel_pb2.WorkspaceSelector( + workspace="default" + ), + name=sb.sandbox.name, + ) ) persisted_env = dict(fetched.sandbox.spec.environment) assert "ANTHROPIC_API_KEY" not in persisted_env, ( @@ -468,26 +795,37 @@ def test_update_provider_preserves_unset_credentials_and_config( try: stub.CreateProvider( openshell_pb2.CreateProviderRequest( + workspace_scope=datamodel_pb2.WorkspaceSelector(workspace="default"), provider=datamodel_pb2.Provider( metadata=datamodel_pb2.ObjectMeta(name=name), - type="generic", - credentials={"KEY_A": "val-a", "KEY_B": "val-b"}, + type="codex", + credentials={ + "CODEX_AUTH_ACCESS_TOKEN": "val-a", + "CODEX_AUTH_REFRESH_TOKEN": "val-b", + "CODEX_AUTH_ACCOUNT_ID": "account-id", + }, config={"BASE_URL": "https://example.com"}, - ) + ), ) ) stub.UpdateProvider( openshell_pb2.UpdateProviderRequest( + workspace_scope=datamodel_pb2.WorkspaceSelector(workspace="default"), provider=datamodel_pb2.Provider( metadata=datamodel_pb2.ObjectMeta(name=name), type="", - credentials={"KEY_A": "rotated-a"}, - ) + credentials={"CODEX_AUTH_ACCESS_TOKEN": "rotated-a"}, + ), ) ) - got = stub.GetProvider(openshell_pb2.GetProviderRequest(name=name)) + got = stub.GetProvider( + openshell_pb2.GetProviderRequest( + workspace_scope=datamodel_pb2.WorkspaceSelector(workspace="default"), + name=name, + ) + ) p = got.provider # Credential keys are preserved but values are redacted. assert len(p.credentials) > 0, "credential keys should be preserved" @@ -513,25 +851,32 @@ def test_update_provider_empty_maps_preserves_all( try: stub.CreateProvider( openshell_pb2.CreateProviderRequest( + workspace_scope=datamodel_pb2.WorkspaceSelector(workspace="default"), provider=datamodel_pb2.Provider( metadata=datamodel_pb2.ObjectMeta(name=name), - type="generic", - credentials={"TOKEN": "secret"}, + type="openai", + credentials={"OPENAI_API_KEY": "secret"}, config={"URL": "https://api.example.com"}, - ) + ), ) ) stub.UpdateProvider( openshell_pb2.UpdateProviderRequest( + workspace_scope=datamodel_pb2.WorkspaceSelector(workspace="default"), provider=datamodel_pb2.Provider( metadata=datamodel_pb2.ObjectMeta(name=name), type="", - ) + ), ) ) - got = stub.GetProvider(openshell_pb2.GetProviderRequest(name=name)) + got = stub.GetProvider( + openshell_pb2.GetProviderRequest( + workspace_scope=datamodel_pb2.WorkspaceSelector(workspace="default"), + name=name, + ) + ) p = got.provider # Credential keys are preserved but values are redacted. assert len(p.credentials) > 0, "credential keys should be preserved" @@ -555,26 +900,33 @@ def test_update_provider_merges_config_preserves_credentials( try: stub.CreateProvider( openshell_pb2.CreateProviderRequest( + workspace_scope=datamodel_pb2.WorkspaceSelector(workspace="default"), provider=datamodel_pb2.Provider( metadata=datamodel_pb2.ObjectMeta(name=name), - type="generic", - credentials={"API_KEY": "original-key"}, + type="openai", + credentials={"OPENAI_API_KEY": "original-key"}, config={"ENDPOINT": "https://old.example.com"}, - ) + ), ) ) stub.UpdateProvider( openshell_pb2.UpdateProviderRequest( + workspace_scope=datamodel_pb2.WorkspaceSelector(workspace="default"), provider=datamodel_pb2.Provider( metadata=datamodel_pb2.ObjectMeta(name=name), type="", config={"ENDPOINT": "https://new.example.com"}, - ) + ), ) ) - got = stub.GetProvider(openshell_pb2.GetProviderRequest(name=name)) + got = stub.GetProvider( + openshell_pb2.GetProviderRequest( + workspace_scope=datamodel_pb2.WorkspaceSelector(workspace="default"), + name=name, + ) + ) p = got.provider # Credential keys are preserved but values are redacted. assert len(p.credentials) > 0, "credential keys should be preserved" @@ -598,21 +950,25 @@ def test_update_provider_rejects_type_change( try: stub.CreateProvider( openshell_pb2.CreateProviderRequest( + workspace_scope=datamodel_pb2.WorkspaceSelector(workspace="default"), provider=datamodel_pb2.Provider( metadata=datamodel_pb2.ObjectMeta(name=name), - type="generic", - credentials={"KEY": "val"}, - ) + type="openai", + credentials={"OPENAI_API_KEY": "val"}, + ), ) ) with pytest.raises(grpc.RpcError) as exc_info: stub.UpdateProvider( openshell_pb2.UpdateProviderRequest( + workspace_scope=datamodel_pb2.WorkspaceSelector( + workspace="default" + ), provider=datamodel_pb2.Provider( metadata=datamodel_pb2.ObjectMeta(name=name), type="nvidia", - ) + ), ) ) assert exc_info.value.code() == grpc.StatusCode.INVALID_ARGUMENT @@ -626,8 +982,6 @@ def test_update_provider_rejects_type_change( # =========================================================================== -@pytest.mark.exclusive_gateway_config -@pytest.mark.usefixtures("providers_v2_enabled") def test_github_provider_allows_https_git_clone( sandbox: Callable[..., Sandbox], sandbox_client: SandboxClient, @@ -640,9 +994,7 @@ def test_github_provider_allows_https_git_clone( the sandbox, exercising provider attachment, effective-policy composition, TLS interception, and real git behavior end to end. git delegates HTTPS to a ``git-remote-https`` helper whose ancestor is ``/usr/bin/git``, so the - profile's git binary covers it via ancestor matching. The - ``providers_v2_enabled`` fixture turns on the gateway-global gate that - composes the provider's network policy. + profile's git binary covers it via ancestor matching. """ with provider( sandbox_client._stub, diff --git a/e2e/run.sh b/e2e/run.sh index 0505730f0..875186394 100755 --- a/e2e/run.sh +++ b/e2e/run.sh @@ -250,7 +250,7 @@ guest_gateway_bin= if [ "${mode}" = host ]; then echo "==> Building native host openshell-gateway" mise x -- cargo build "${cargo_jobs[@]}" \ - -p openshell-server \ + -p openshell-gateway \ --bin openshell-gateway \ --features bundled-z3 host_gateway_bin="${target_dir}/debug/openshell-gateway" @@ -268,7 +268,7 @@ else mise x -- cargo zigbuild "${cargo_jobs[@]}" \ --release \ --target "${linux_gateway_zig_target}" \ - -p openshell-server \ + -p openshell-gateway \ --bin openshell-gateway \ --features bundled-z3 ) diff --git a/e2e/rust/Cargo.lock b/e2e/rust/Cargo.lock index bbccb7496..c5235dff8 100644 --- a/e2e/rust/Cargo.lock +++ b/e2e/rust/Cargo.lock @@ -31,6 +31,30 @@ version = "1.5.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f2032f911046de80f0a198e0901378627c33f59ea0ac00e363d481118bd70a53" +[[package]] +name = "aws-lc-rs" +version = "1.18.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b281d307588d634de920874890732659e2e7672f72b5e10e81badc1a8a83621e" +dependencies = [ + "aws-lc-sys", + "untrusted", + "zeroize", +] + +[[package]] +name = "aws-lc-sys" +version = "0.45.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9bff6c3b54fad79a2e60b8102caf565819711497c1f5f092f49508e2f5c31b27" +dependencies = [ + "cc", + "cmake", + "dunce", + "fs_extra", + "pkg-config", +] + [[package]] name = "axum" version = "0.8.9" @@ -157,6 +181,8 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "5add81bb678e6cb321aff7fa0dc7689ad82b112dbc032cea19f91d6b8e3582b9" dependencies = [ "find-msvc-tools", + "jobserver", + "libc", "shlex", ] @@ -172,6 +198,15 @@ version = "0.2.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f079e83a288787bcd14a6aea84cee5c87a67c5a3e660c30f557a3d24761b3527" +[[package]] +name = "cmake" +version = "0.1.58" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c0f78a02292a74a88ac736019ab962ece0bc380e3f977bf72e376c5d78ff0678" +dependencies = [ + "cc", +] + [[package]] name = "cpufeatures" version = "0.2.17" @@ -218,6 +253,12 @@ dependencies = [ "syn 3.0.3", ] +[[package]] +name = "dunce" +version = "1.0.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "92773504d58c093f6de2459af4af33faa518c13451eb8f2b5698ed3d36e7c813" + [[package]] name = "either" version = "1.17.0" @@ -237,7 +278,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "39cab71617ae0d63f51a36d69f866391735b51691dbda63cf6f96d042b63efeb" dependencies = [ "libc", - "windows-sys 0.61.2", + "windows-sys", ] [[package]] @@ -267,6 +308,12 @@ dependencies = [ "percent-encoding", ] +[[package]] +name = "fs_extra" +version = "1.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "42703706b716c37f96a77aea830392ad231f44c9e9a67872fa5548707e11b11c" + [[package]] name = "futures-channel" version = "0.3.33" @@ -346,10 +393,8 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ff2abc00be7fca6ebc474524697ae276ad847ad0a6b3faa4bcb027e9a4614ad0" dependencies = [ "cfg-if", - "js-sys", "libc", "wasi", - "wasm-bindgen", ] [[package]] @@ -646,6 +691,8 @@ checksum = "d466e9454f08e4a911e14806c24e16fba1b4c121d1ea474396f396069cf949d9" dependencies = [ "equivalent", "hashbrown", + "serde", + "serde_core", ] [[package]] @@ -663,6 +710,16 @@ version = "1.0.18" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "8f42a60cbdf9a97f5d2305f08a87dc4e09308d1276d28c869c684d7777685682" +[[package]] +name = "jobserver" +version = "0.1.35" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1c00acbd29eabad4a2392fa0e921c874934dbbf4194312ad20f04a0ed67a3cb3" +dependencies = [ + "getrandom 0.4.3", + "libc", +] + [[package]] name = "js-sys" version = "0.3.103" @@ -676,17 +733,20 @@ dependencies = [ [[package]] name = "jsonwebtoken" -version = "9.3.1" +version = "10.4.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5a87cc7a48537badeae96744432de36f4be2b4a34a05a5ef32e9dd8a1c169dde" +checksum = "eba32bfb4ffdeaca3e34431072faf01745c9b26d25504aa7a6cf5684334fc4fc" dependencies = [ + "aws-lc-rs", "base64", + "getrandom 0.2.17", "js-sys", "pem", - "ring", "serde", "serde_json", + "signature", "simple_asn1", + "zeroize", ] [[package]] @@ -696,14 +756,10 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "3eaf3ede3fee6db1a4c2ee091bf8a8b4dccdc6d17f656fb07896ee72867612f2" [[package]] -name = "libyml" -version = "0.0.5" +name = "libm" +version = "0.2.16" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3302702afa434ffa30847a83305f0a69d6abd74293b6554c18ec85c7ef30c980" -dependencies = [ - "anyhow", - "version_check", -] +checksum = "b6d2cec3eae94f9f509c767b45932f1ada8350c4bdb85af2fcab4a3c14807981" [[package]] name = "linux-raw-sys" @@ -758,7 +814,7 @@ checksum = "30d65c71f1ce40ab09135ce117d742b9f8a19ff91a41a8b57ed50bc2de59c427" dependencies = [ "libc", "wasi", - "windows-sys 0.61.2", + "windows-sys", ] [[package]] @@ -773,6 +829,22 @@ dependencies = [ "libc", ] +[[package]] +name = "noyalib" +version = "0.0.28" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9f075ef19fa3bcf8697c0ef96c37d5c435d339a40ab8081cae3aac3a4e7fee9a" +dependencies = [ + "hashbrown", + "indexmap", + "libm", + "memchr", + "rustc-hash", + "serde", + "serde_core", + "smallvec", +] + [[package]] name = "num-bigint" version = "0.4.8" @@ -827,11 +899,11 @@ dependencies = [ "hyper-util", "jsonwebtoken", "nix", + "noyalib", "prost", "rand", "serde", "serde_json", - "serde_yml", "serial_test", "sha1", "sha2", @@ -909,6 +981,12 @@ version = "0.2.17" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "a89322df9ebe1c1578d689c92318e070967d1042b512afbe49518723f4e6d5cd" +[[package]] +name = "pkg-config" +version = "0.3.34" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f6b464fbc74e149a392436b17d523f769e057cb6877f6a5c4618bc6f11800548" + [[package]] name = "potential_utf" version = "0.1.5" @@ -993,7 +1071,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b9ef1d0d795eb7d84685bca4f72f3649f064e6641543d3a8c415898726a57b41" dependencies = [ "rand_chacha", - "rand_core", + "rand_core 0.9.5", ] [[package]] @@ -1003,7 +1081,16 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d3022b5f1df60f26e1ffddd6c66e8aa15de382ae63b3a0c1bfc0e4d3e3f325cb" dependencies = [ "ppv-lite86", - "rand_core", + "rand_core 0.9.5", +] + +[[package]] +name = "rand_core" +version = "0.6.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ec0be4795e2f6a28069bec0b5ff3e2ac9bafc99e6a9a7dc3547996c5c816922c" +dependencies = [ + "getrandom 0.2.17", ] [[package]] @@ -1025,18 +1112,10 @@ dependencies = [ ] [[package]] -name = "ring" -version = "0.17.14" +name = "rustc-hash" +version = "2.1.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a4689e6c2294d81e88dc6261c768b63bc4fcdb852be6d1352498b114f61383b7" -dependencies = [ - "cc", - "cfg-if", - "getrandom 0.2.17", - "libc", - "untrusted", - "windows-sys 0.52.0", -] +checksum = "6b1e7f9a428571be2dc5bc0505c13fb6bf936822b894ec87abf8a08a4e51742d" [[package]] name = "rustix" @@ -1048,7 +1127,7 @@ dependencies = [ "errno", "libc", "linux-raw-sys", - "windows-sys 0.61.2", + "windows-sys", ] [[package]] @@ -1135,21 +1214,6 @@ dependencies = [ "serde", ] -[[package]] -name = "serde_yml" -version = "0.0.12" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "59e2dd588bf1597a252c3b920e0143eb99b0f76e4e082f4c92ce34fbc9e71ddd" -dependencies = [ - "indexmap", - "itoa", - "libyml", - "memchr", - "ryu", - "serde", - "version_check", -] - [[package]] name = "serial_test" version = "3.5.0" @@ -1213,6 +1277,15 @@ dependencies = [ "libc", ] +[[package]] +name = "signature" +version = "2.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "77549399552de45a898a580c1b41d445bf730df867cc44e6c0233bbc4b8329de" +dependencies = [ + "rand_core 0.6.4", +] + [[package]] name = "simple_asn1" version = "0.6.4" @@ -1244,7 +1317,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c3d1e2c7f27f8d4cb10542a02c49005dbd6e93095799d6f3be745fae9f8fedd4" dependencies = [ "libc", - "windows-sys 0.61.2", + "windows-sys", ] [[package]] @@ -1302,7 +1375,7 @@ dependencies = [ "getrandom 0.4.3", "once_cell", "rustix", - "windows-sys 0.61.2", + "windows-sys", ] [[package]] @@ -1379,7 +1452,7 @@ dependencies = [ "signal-hook-registry", "socket2", "tokio-macros", - "windows-sys 0.61.2", + "windows-sys", ] [[package]] @@ -1540,9 +1613,9 @@ checksum = "e6e4313cd5fcd3dad5cafa179702e2b244f760991f45397d14d4ebf38247da75" [[package]] name = "untrusted" -version = "0.9.0" +version = "0.7.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8ecb6da28b8a351d773b68d5825ac39017e680750f980f3a1a85cd8dd28a47c1" +checksum = "a156c684c91ea7d62626509bce3cb4e1d9ed5c4d978f7b4352658f96a4c26b4a" [[package]] name = "url" @@ -1665,15 +1738,6 @@ version = "0.2.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f0805222e57f7521d6a62e36fa9163bc891acd422f971defe97d64e70d0a4fe5" -[[package]] -name = "windows-sys" -version = "0.52.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "282be5f36a8ce781fad8c8ae18fa3f9beff57ec1b52cb3de0789201425d9a33d" -dependencies = [ - "windows-targets", -] - [[package]] name = "windows-sys" version = "0.61.2" @@ -1683,70 +1747,6 @@ dependencies = [ "windows-link", ] -[[package]] -name = "windows-targets" -version = "0.52.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9b724f72796e036ab90c1021d4780d4d3d648aca59e491e6b98e725b84e99973" -dependencies = [ - "windows_aarch64_gnullvm", - "windows_aarch64_msvc", - "windows_i686_gnu", - "windows_i686_gnullvm", - "windows_i686_msvc", - "windows_x86_64_gnu", - "windows_x86_64_gnullvm", - "windows_x86_64_msvc", -] - -[[package]] -name = "windows_aarch64_gnullvm" -version = "0.52.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "32a4622180e7a0ec044bb555404c800bc9fd9ec262ec147edd5989ccd0c02cd3" - -[[package]] -name = "windows_aarch64_msvc" -version = "0.52.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "09ec2a7bb152e2252b53fa7803150007879548bc709c039df7627cabbd05d469" - -[[package]] -name = "windows_i686_gnu" -version = "0.52.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8e9b5ad5ab802e97eb8e295ac6720e509ee4c243f69d781394014ebfe8bbfa0b" - -[[package]] -name = "windows_i686_gnullvm" -version = "0.52.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0eee52d38c090b3caa76c563b86c3a4bd71ef1a819287c19d586d7334ae8ed66" - -[[package]] -name = "windows_i686_msvc" -version = "0.52.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "240948bc05c5e7c6dabba28bf89d89ffce3e303022809e73deaefe4f6ec56c66" - -[[package]] -name = "windows_x86_64_gnu" -version = "0.52.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "147a5c80aabfbf0c7d901cb5895d1de30ef2907eb21fbbab29ca94c5b08b1a78" - -[[package]] -name = "windows_x86_64_gnullvm" -version = "0.52.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "24d5b23dc417412679681396f2b49f3de8c1473deb516bd34410872eff51ed0d" - -[[package]] -name = "windows_x86_64_msvc" -version = "0.52.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "589f6da84c646204747d1270a2a5661ea66ed1cced2631d546fdfb155959f9ec" - [[package]] name = "wit-bindgen" version = "0.57.1" @@ -1823,6 +1823,26 @@ dependencies = [ "synstructure", ] +[[package]] +name = "zeroize" +version = "1.9.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e13c156562582aa81c60cb29407084cdb54c4164760106ab78e6c5b0858cf64e" +dependencies = [ + "zeroize_derive", +] + +[[package]] +name = "zeroize_derive" +version = "1.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3c50655cbb0fe3fc43170059e702f1ce5e19b84cec58dc87b037a09935c2f328" +dependencies = [ + "proc-macro2", + "quote", + "syn 2.0.119", +] + [[package]] name = "zerotrie" version = "0.2.4" diff --git a/e2e/rust/Cargo.toml b/e2e/rust/Cargo.toml index 9ef36ac8f..7b76e0387 100644 --- a/e2e/rust/Cargo.toml +++ b/e2e/rust/Cargo.toml @@ -16,6 +16,7 @@ license = "Apache-2.0" publish = false [features] +# Selects the common E2E suite. e2e = [] # Selects tests that rely on `host.openshell.internal` (the sandbox's stable # alias to the host running test fixtures). docker, podman, and vm wire the @@ -48,11 +49,21 @@ name = "provider_refresh_keycloak" path = "tests/provider_refresh_keycloak.rs" required-features = ["e2e-provider-refresh-keycloak"] +[[test]] +name = "vm_overlay" +path = "tests/vm_overlay.rs" +required-features = ["e2e-vm"] + [[test]] name = "custom_image" path = "tests/custom_image.rs" required-features = ["e2e-docker"] +[[test]] +name = "rootfs_tar" +path = "tests/rootfs_tar.rs" +required-features = ["e2e-vm"] + [[test]] name = "docker_preflight" path = "tests/docker_preflight.rs" @@ -103,6 +114,11 @@ name = "vm_gateway_start" path = "tests/vm_gateway_start.rs" required-features = ["e2e-vm"] +[[test]] +name = "vm_corporate_proxy" +path = "tests/vm_corporate_proxy.rs" +required-features = ["e2e-vm"] + [[test]] name = "provider_token_exchange" path = "tests/provider_token_exchange.rs" @@ -163,6 +179,11 @@ name = "workspace_lifecycle" path = "tests/workspace_lifecycle.rs" required-features = ["e2e"] +[[test]] +name = "sandbox_templates" +path = "tests/sandbox_templates.rs" +required-features = ["e2e"] + [[test]] name = "proxy_egress_pipeline" path = "tests/proxy_egress_pipeline.rs" @@ -196,7 +217,7 @@ futures-util = "0.3" http-body-util = "0.1" hyper = { version = "1", features = ["client", "http1"] } hyper-util = { version = "0.1", features = ["tokio"] } -jsonwebtoken = "9" +jsonwebtoken = { version = "10", features = ["aws_lc_rs"] } prost = "0.14" tokio = { version = "1.43", features = ["full"] } tokio-stream = { version = "0.1", features = ["net"] } @@ -207,7 +228,7 @@ hex = "0.4" rand = "0.9" serde = { version = "1", features = ["derive"] } serde_json = "1" -serde_yml = "0.0.12" +serde_yml = { package = "noyalib", version = "0.0.28", default-features = false, features = ["std", "compat-serde-yaml"] } tonic = { version = "0.14", features = ["transport"] } tonic-prost = "0.14" tower = "0.5" diff --git a/e2e/rust/e2e-docker.sh b/e2e/rust/e2e-docker.sh index 6c2886808..ba97b85a5 100755 --- a/e2e/rust/e2e-docker.sh +++ b/e2e/rust/e2e-docker.sh @@ -2,23 +2,47 @@ # SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. # SPDX-License-Identifier: Apache-2.0 -# Run a Rust e2e test against a standalone gateway running the bundled Docker -# compute driver. Set OPENSHELL_GATEWAY_ENDPOINT=http://host:port to reuse an -# existing plaintext gateway instead of starting an ephemeral one. +# Run standalone CLI conformance, and optionally a focused Rust e2e test, +# against a gateway using the bundled Docker compute driver. Set +# OPENSHELL_GATEWAY_ENDPOINT=http://host:port to reuse an existing plaintext +# gateway instead of starting an ephemeral one. set -euo pipefail ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)" -E2E_TEST="${OPENSHELL_E2E_DOCKER_TEST:-smoke}" -E2E_FEATURES="${OPENSHELL_E2E_DOCKER_FEATURES:-e2e,e2e-docker}" +E2E_TEST="${OPENSHELL_E2E_DOCKER_TEST:-}" +E2E_FEATURES="${OPENSHELL_E2E_DOCKER_FEATURES-e2e-docker}" DEFAULT_WORKLOAD_MANIFEST="${ROOT}/e2e/gpu/images/.build/workloads.yaml" +RUN_WITH_GATEWAY_COMMAND="__openshell_run_docker_e2e" +# shellcheck source=e2e/support/conformance.sh +source "${ROOT}/e2e/support/conformance.sh" + +run_e2e() { + e2e_run_openshell_conformance "Docker" + + if [ -z "${E2E_FEATURES}" ]; then + return 0 + fi + + local cargo_args=( + cargo test --manifest-path "${ROOT}/e2e/rust/Cargo.toml" + --features "${E2E_FEATURES}" + ) + if [ -n "${E2E_TEST}" ]; then + cargo_args+=(--test "${E2E_TEST}") + fi + cargo_args+=(-- --nocapture) + "${cargo_args[@]}" +} if [ "${E2E_TEST}" = "gpu" ] && [ -z "${OPENSHELL_E2E_WORKLOAD_MANIFEST:-}" ] && [ ! -f "${DEFAULT_WORKLOAD_MANIFEST}" ]; then echo "note: running GPU e2e without a workload manifest; workload validation will log an explicit skip. Build one with 'mise run e2e:workloads:build' or set OPENSHELL_E2E_WORKLOAD_MANIFEST." fi +if [ "${1:-}" = "${RUN_WITH_GATEWAY_COMMAND}" ]; then + run_e2e + exit 0 +fi + exec "${ROOT}/e2e/with-docker-gateway.sh" \ - cargo test --manifest-path "${ROOT}/e2e/rust/Cargo.toml" \ - --features "${E2E_FEATURES}" \ - --test "${E2E_TEST}" \ - -- --nocapture + bash "${BASH_SOURCE[0]}" "${RUN_WITH_GATEWAY_COMMAND}" diff --git a/e2e/rust/e2e-kubernetes.sh b/e2e/rust/e2e-kubernetes.sh index cf28e3572..c54057bb9 100755 --- a/e2e/rust/e2e-kubernetes.sh +++ b/e2e/rust/e2e-kubernetes.sh @@ -6,7 +6,7 @@ # via Helm. Set OPENSHELL_E2E_KUBE_CONTEXT to target an existing cluster; # otherwise an ephemeral k3d cluster is created and torn down by # with-kube-gateway.sh. Set OPENSHELL_E2E_KUBE_TEST to scope to a single -# integration test (e.g. smoke) for local debugging. +# integration test for local debugging. # # Features: the default set includes `e2e-host-gateway` so tests that rely on # the sandbox-side `host.openshell.internal` alias compile and run. The @@ -18,14 +18,18 @@ set -euo pipefail ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)" +RUN_WITH_GATEWAY_COMMAND="__openshell_run_kubernetes_e2e" +# shellcheck source=e2e/support/conformance.sh +source "${ROOT}/e2e/support/conformance.sh" -E2E_FEATURES="${OPENSHELL_E2E_KUBERNETES_FEATURES:-e2e,e2e-host-gateway,e2e-kubernetes}" +E2E_FEATURES="${OPENSHELL_E2E_KUBERNETES_FEATURES-e2e,e2e-host-gateway,e2e-kubernetes}" # Docker and Podman build their local gateway and CLI together in the shared # gateway wrapper. Kubernetes consumes published gateway images, so only its # local CLI needs to be built when CI has not supplied a prebuilt one. if [ -z "${OPENSHELL_BIN:-}" ]; then cargo build -p openshell-cli + export OPENSHELL_BIN="${ROOT}/target/debug/openshell" fi test_filter=() @@ -33,15 +37,42 @@ if [ -n "${OPENSHELL_E2E_KUBE_TEST:-}" ]; then test_filter+=(--test "${OPENSHELL_E2E_KUBE_TEST}") fi +is_operator_workspace_mode() { + [[ ",${E2E_FEATURES}," == *",e2e-kubernetes-workspace-operator,"* ]] +} + +run_conformance() { + if is_operator_workspace_mode; then + echo "note: skipping standalone CLI conformance in Kubernetes operator workspace mode; default workspace is not operator-allowlisted (see #2971)." + return 0 + fi + + e2e_run_openshell_conformance "Kubernetes" +} + run_suite() { "${ROOT}/e2e/with-kube-gateway.sh" \ - cargo test --manifest-path "${ROOT}/e2e/rust/Cargo.toml" \ - --features "${E2E_FEATURES}" \ - --no-fail-fast \ - ${test_filter[@]+"${test_filter[@]}"} \ - -- --nocapture + bash "${BASH_SOURCE[0]}" "${RUN_WITH_GATEWAY_COMMAND}" } +run_e2e() { + run_conformance + if [ -z "${E2E_FEATURES}" ]; then + return 0 + fi + + cargo test --manifest-path "${ROOT}/e2e/rust/Cargo.toml" \ + --features "${E2E_FEATURES}" \ + --no-fail-fast \ + ${test_filter[@]+"${test_filter[@]}"} \ + -- --nocapture +} + +if [ "${1:-}" = "${RUN_WITH_GATEWAY_COMMAND}" ]; then + run_e2e + exit 0 +fi + if [ "${OPENSHELL_E2E_CREDENTIAL_DRIVERS:-0}" = "1" ] \ && [ -z "${OPENSHELL_E2E_CREDENTIAL_DRIVER:-}" ]; then OPENSHELL_E2E_CREDENTIAL_DRIVER=kubernetes-secrets run_suite @@ -50,8 +81,4 @@ if [ "${OPENSHELL_E2E_CREDENTIAL_DRIVERS:-0}" = "1" ] \ fi exec "${ROOT}/e2e/with-kube-gateway.sh" \ - cargo test --manifest-path "${ROOT}/e2e/rust/Cargo.toml" \ - --features "${E2E_FEATURES}" \ - --no-fail-fast \ - ${test_filter[@]+"${test_filter[@]}"} \ - -- --nocapture + bash "${BASH_SOURCE[0]}" "${RUN_WITH_GATEWAY_COMMAND}" diff --git a/e2e/rust/e2e-openshift.sh b/e2e/rust/e2e-openshift.sh deleted file mode 100755 index 639e6323a..000000000 --- a/e2e/rust/e2e-openshift.sh +++ /dev/null @@ -1,184 +0,0 @@ -#!/usr/bin/env bash -# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. -# SPDX-License-Identifier: Apache-2.0 -# -# Validates all OpenShift database-backend scenarios against a live cluster. -# -# Prerequisites: -# - oc CLI authenticated to an OpenShift cluster -# - helm 3.x installed -# -# Usage: -# mise run e2e:openshift -# e2e/rust/e2e-openshift.sh [--chart-path ./deploy/helm/openshell] [--image-tag dev] - -set -euo pipefail - -ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)" -# shellcheck source=e2e/support/gateway-common.sh -source "${ROOT}/e2e/support/gateway-common.sh" - -CHART_PATH="${CHART_PATH:-./deploy/helm/openshell}" -NAMESPACE="openshell" -RELEASE="openshell" -IMAGE_TAG="${IMAGE_TAG:-dev}" -WAIT_TIMEOUT="120s" -PASSED=0 -FAILED=0 -SCENARIOS=() -EXTERNAL_PG_SECRET="my-pg-credentials" -EXTERNAL_PG_SERVICE="openshell-e2e-postgres" -EXTERNAL_PG_SERVICE_ACCOUNT="openshell-e2e-postgres" -EXTERNAL_PG_PASSWORD="openshell-e2e-postgres" -EXTERNAL_PG_DATABASE="openshell" -EXTERNAL_PG_USERNAME="openshell" -EXTERNAL_PG_MANIFEST="${ROOT}/e2e/kubernetes/postgres-fixture.yaml" - -while [[ $# -gt 0 ]]; do - case $1 in - --chart-path) CHART_PATH="$2"; shift 2 ;; - --image-tag) IMAGE_TAG="$2"; shift 2 ;; - --namespace) NAMESPACE="$2"; shift 2 ;; - *) echo "Unknown option: $1" >&2; exit 1 ;; - esac -done - -# --- helpers ---------------------------------------------------------------- - -log() { echo "==> $*"; } -pass() { log "PASS: $1"; PASSED=$((PASSED + 1)); SCENARIOS+=("PASS $1"); } -fail() { log "FAIL: $1 — $2"; FAILED=$((FAILED + 1)); SCENARIOS+=("FAIL $1: $2"); } - -wait_for_ready() { - local label="$1" timeout="$2" - if oc wait pod -n "$NAMESPACE" -l "$label" --for=condition=Ready --timeout="$timeout" 2>/dev/null; then - return 0 - fi - return 1 -} - -cleanup_release() { - log "Cleaning up release $RELEASE" - helm uninstall "$RELEASE" -n "$NAMESPACE" --wait 2>/dev/null || true - # Wait for pods to terminate - for i in $(seq 1 30); do - if [ -z "$(oc get pods -n "$NAMESPACE" -l "app.kubernetes.io/instance=$RELEASE" --no-headers 2>/dev/null)" ]; then - break - fi - sleep 2 - done - # Clean up PVCs left by StatefulSets - oc delete pvc -n "$NAMESPACE" -l "app.kubernetes.io/instance=$RELEASE" --wait=false 2>/dev/null || true -} - -deploy_external_pg() { - local pg_uri - - log "Deploying standalone PostgreSQL as external database..." - oc create serviceaccount "$EXTERNAL_PG_SERVICE_ACCOUNT" -n "$NAMESPACE" 2>/dev/null || true - oc adm policy add-scc-to-user anyuid -z "$EXTERNAL_PG_SERVICE_ACCOUNT" -n "$NAMESPACE" >/dev/null - - oc apply -n "$NAMESPACE" -f "$EXTERNAL_PG_MANIFEST" - oc rollout status "deployment/${EXTERNAL_PG_SERVICE}" -n "$NAMESPACE" --timeout="$WAIT_TIMEOUT" - - pg_uri="postgresql://${EXTERNAL_PG_USERNAME}:${EXTERNAL_PG_PASSWORD}@${EXTERNAL_PG_SERVICE}.${NAMESPACE}.svc.cluster.local:5432/${EXTERNAL_PG_DATABASE}" - log "Creating existing Secret with PostgreSQL credentials..." - oc delete secret "$EXTERNAL_PG_SECRET" -n "$NAMESPACE" --ignore-not-found >/dev/null 2>&1 || true - oc create secret generic "$EXTERNAL_PG_SECRET" -n "$NAMESPACE" \ - --from-literal=uri="$pg_uri" -} - -cleanup_external_pg() { - oc delete -n "$NAMESPACE" -f "$EXTERNAL_PG_MANIFEST" --ignore-not-found 2>/dev/null || true - oc delete secret "$EXTERNAL_PG_SECRET" -n "$NAMESPACE" --ignore-not-found 2>/dev/null || true - oc adm policy remove-scc-from-user anyuid -z "$EXTERNAL_PG_SERVICE_ACCOUNT" \ - -n "$NAMESPACE" 2>/dev/null || true -} - -verify_gateway() { - local scenario="$1" - if wait_for_ready "app.kubernetes.io/name=openshell,app.kubernetes.io/instance=$RELEASE" "$WAIT_TIMEOUT"; then - # Check the pod is actually running (not CrashLoopBackOff) - local phase - phase=$(oc get pod -n "$NAMESPACE" -l "app.kubernetes.io/name=openshell,app.kubernetes.io/instance=$RELEASE" \ - -o jsonpath='{.items[0].status.phase}' 2>/dev/null) - if [ "$phase" = "Running" ]; then - pass "$scenario" - else - fail "$scenario" "pod phase is $phase, expected Running" - fi - else - local status - status=$(oc get pods -n "$NAMESPACE" -l "app.kubernetes.io/name=openshell" --no-headers 2>/dev/null || echo "no pods found") - fail "$scenario" "gateway pod not ready within $WAIT_TIMEOUT ($status)" - fi -} - -# --- setup ------------------------------------------------------------------ - -log "Setting up namespace $NAMESPACE" -oc create ns "$NAMESPACE" 2>/dev/null || true -oc adm policy add-scc-to-user privileged -z "${RELEASE}-sandbox" -n "$NAMESPACE" - -OPENSHIFT_FLAGS=( - --set server.disableTls=true - --set "server.telemetryEnabled=${OPENSHELL_TELEMETRY_ENABLED}" - --set podSecurityContext.fsGroup=null - --set securityContext.runAsUser=null - --set image.tag="$IMAGE_TAG" -) - -# --- scenario 1: SQLite (default, no postgres) ----------------------------- - -SCENARIO="SQLite (default)" -log "Testing: $SCENARIO" -cleanup_release - -helm install "$RELEASE" "$CHART_PATH" -n "$NAMESPACE" \ - "${OPENSHIFT_FLAGS[@]}" - -verify_gateway "$SCENARIO" -cleanup_release - -# --- scenario 2: External PostgreSQL with existing Secret ------------------- - -SCENARIO="External PostgreSQL (externalDbSecret)" -log "Testing: $SCENARIO" -cleanup_release - -deploy_external_pg - -# Install OpenShell pointing at the existing Secret -helm install "$RELEASE" "$CHART_PATH" -n "$NAMESPACE" \ - "${OPENSHIFT_FLAGS[@]}" \ - --set server.externalDbSecret="$EXTERNAL_PG_SECRET" - -verify_gateway "$SCENARIO" - -# Cleanup external postgres and secret -cleanup_release -cleanup_external_pg - -# --- teardown --------------------------------------------------------------- - -log "Removing SCC binding and namespace" -oc adm policy remove-scc-from-user privileged -z "${RELEASE}-sandbox" -n "$NAMESPACE" 2>/dev/null || true -cleanup_external_pg -oc delete ns "$NAMESPACE" --wait=false 2>/dev/null || true - -# --- summary ---------------------------------------------------------------- - -echo "" -echo "========================================" -echo " Test Summary" -echo "========================================" -for s in "${SCENARIOS[@]}"; do - echo " $s" -done -echo "----------------------------------------" -echo " Passed: $PASSED Failed: $FAILED" -echo "========================================" - -if [ "$FAILED" -gt 0 ]; then - exit 1 -fi diff --git a/e2e/rust/e2e-podman.sh b/e2e/rust/e2e-podman.sh index f5f1effd1..736519458 100755 --- a/e2e/rust/e2e-podman.sh +++ b/e2e/rust/e2e-podman.sh @@ -10,28 +10,43 @@ set -euo pipefail ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)" E2E_TEST="${OPENSHELL_E2E_PODMAN_TEST:-}" -E2E_FEATURES="${OPENSHELL_E2E_PODMAN_FEATURES:-e2e-podman}" +E2E_FEATURES="${OPENSHELL_E2E_PODMAN_FEATURES-e2e-podman}" DEFAULT_WORKLOAD_MANIFEST="${ROOT}/e2e/gpu/images/.build/workloads.yaml" +RUN_WITH_GATEWAY_COMMAND="__openshell_run_podman_e2e" +# shellcheck source=e2e/support/conformance.sh +source "${ROOT}/e2e/support/conformance.sh" if [ "${E2E_TEST}" = "gpu" ] && [ -z "${OPENSHELL_E2E_WORKLOAD_MANIFEST:-}" ] && [ ! -f "${DEFAULT_WORKLOAD_MANIFEST}" ]; then echo "note: running Podman GPU e2e without a workload manifest; workload validation will log an explicit skip. Build one with 'CONTAINER_ENGINE=podman mise run e2e:workloads:build' or set OPENSHELL_E2E_WORKLOAD_MANIFEST." fi +if [ "${1:-}" = "${RUN_WITH_GATEWAY_COMMAND}" ]; then + e2e_run_openshell_conformance "Podman" + if [ -z "${E2E_FEATURES}" ]; then + exit 0 + fi + + TEST_ARGS=( + cargo test --manifest-path "${ROOT}/e2e/rust/Cargo.toml" + --features "${E2E_FEATURES}" + ) + if [ -n "${E2E_TEST}" ]; then + TEST_ARGS+=(--test "${E2E_TEST}") + fi + TEST_ARGS+=(-- --nocapture) + "${TEST_ARGS[@]}" + exit 0 +fi + # An empty selector runs the full Podman suite, including provider_token_exchange. if [ -z "${E2E_TEST}" ] || [ "${E2E_TEST}" = "provider_token_exchange" ]; then export OPENSHELL_E2E_SPIFFE_FIXTURE="${OPENSHELL_E2E_SPIFFE_FIXTURE:-1}" fi -cargo build -p openshell-cli - -TEST_ARGS=( - cargo test --manifest-path "${ROOT}/e2e/rust/Cargo.toml" - --features "${E2E_FEATURES}" -) -if [ -n "${E2E_TEST}" ]; then - TEST_ARGS+=(--test "${E2E_TEST}") +if [ -n "${OPENSHELL_GATEWAY_ENDPOINT:-}" ] && [ -z "${OPENSHELL_BIN:-}" ]; then + cargo build -p openshell-cli + export OPENSHELL_BIN="${ROOT}/target/debug/openshell" fi -TEST_ARGS+=(-- --nocapture) exec "${ROOT}/e2e/with-podman-gateway.sh" \ - "${TEST_ARGS[@]}" + bash "${BASH_SOURCE[0]}" "${RUN_WITH_GATEWAY_COMMAND}" diff --git a/e2e/rust/e2e-vm.sh b/e2e/rust/e2e-vm.sh index 4acb7256b..96da2a879 100755 --- a/e2e/rust/e2e-vm.sh +++ b/e2e/rust/e2e-vm.sh @@ -46,13 +46,15 @@ set -euo pipefail ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)" source "${ROOT}/e2e/support/gateway-common.sh" +# shellcheck source=e2e/support/conformance.sh +source "${ROOT}/e2e/support/conformance.sh" COMPRESSED_DIR="${ROOT}/target/vm-runtime-compressed" GATEWAY_BIN="${OPENSHELL_GATEWAY_BIN:-${ROOT}/target/debug/openshell-gateway}" DRIVER_BIN="${OPENSHELL_VM_DRIVER_BIN:-${ROOT}/target/debug/openshell-driver-vm}" CLI_BIN="${OPENSHELL_BIN:-${ROOT}/target/debug/openshell}" E2E_TEST_OVERRIDE="${OPENSHELL_E2E_VM_TEST:-}" -E2E_FEATURES="${OPENSHELL_E2E_VM_FEATURES:-e2e-vm}" +E2E_FEATURES="${OPENSHELL_E2E_VM_FEATURES-e2e-vm}" SANDBOX_IMAGE="${OPENSHELL_SANDBOX_IMAGE:-${COMMUNITY_SANDBOX_IMAGE:-ghcr.io/nvidia/openshell-community/sandboxes/base:latest}}" # The VM driver places `compute-driver.sock` under `[openshell.drivers.vm].state_dir`. @@ -101,10 +103,10 @@ if [ -z "${OPENSHELL_GATEWAY_BIN:-}" ]; then if [ "${OPENSHELL_E2E_EXTERNAL_COMPUTE_DRIVER:-0}" = "1" ]; then echo "==> Building driver-free openshell-gateway" cargo build \ - -p openshell-server --bin openshell-gateway \ - --no-default-features + -p openshell-gateway --bin openshell-gateway \ + --no-default-features --features telemetry else - build_packages+=(-p openshell-server) + build_packages+=(-p openshell-gateway) fi else echo "==> Using prebuilt openshell-gateway at ${GATEWAY_BIN}" @@ -371,7 +373,6 @@ fi # The CLI uses the raw endpoint but still resolves matching metadata so it # can find the mTLS client bundle. -export OPENSHELL_E2E_EXPECT_VM_OVERLAY=1 export OPENSHELL_E2E_DRIVER="vm" export OPENSHELL_E2E_VM_STATE_DIR="${RUN_STATE_DIR}" e2e_export_gateway_restart_metadata \ @@ -386,6 +387,8 @@ e2e_export_gateway_restart_metadata \ # preparation; allow 180s for slower CI runners. export OPENSHELL_PROVISION_TIMEOUT="${SANDBOX_PROVISION_TIMEOUT}" +e2e_run_openshell_conformance "VM" + run_e2e_test() { local test_target="$1" shift @@ -403,7 +406,8 @@ run_e2e_test() { if [ -n "${E2E_TEST_OVERRIDE}" ]; then run_e2e_test "${E2E_TEST_OVERRIDE}" else - run_e2e_test smoke run_e2e_test host_gateway_alias + run_e2e_test vm_overlay run_e2e_test vm_gateway_start + run_e2e_test vm_corporate_proxy fi diff --git a/e2e/rust/src/harness/container.rs b/e2e/rust/src/harness/container.rs index 034cfca78..542b9db8e 100644 --- a/e2e/rust/src/harness/container.rs +++ b/e2e/rust/src/harness/container.rs @@ -72,6 +72,75 @@ impl ContainerEngine { } } +static NEXT_IMAGE_ID: std::sync::atomic::AtomicU64 = std::sync::atomic::AtomicU64::new(0); + +/// Builds a container image from a local Dockerfile via the active container +/// engine and removes it on drop. +/// +/// `openshell sandbox create --from` no longer builds local Dockerfiles +/// itself (see the pre-0.1.0 breaking-change notes), so e2e tests that need a +/// custom image build it out-of-band with this guard and pass the resulting +/// `tag()` to `--from`. +pub struct ImageGuard { + engine: ContainerEngine, + tag: String, +} + +impl ImageGuard { + pub fn build(label: &str, dockerfile: &Path, context: &Path) -> Result { + let engine = ContainerEngine::from_env()?; + let unique = NEXT_IMAGE_ID.fetch_add(1, std::sync::atomic::Ordering::Relaxed); + let timestamp = std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .unwrap_or_default() + .as_nanos(); + let tag = format!("localhost/openshell-e2e-{label}-{timestamp}-{unique}:latest"); + let output = engine + .command() + .args([ + "build", + "--file", + dockerfile + .to_str() + .ok_or_else(|| "Dockerfile path must be UTF-8".to_string())?, + "--tag", + &tag, + context + .to_str() + .ok_or_else(|| "image context path must be UTF-8".to_string())?, + ]) + .output() + .map_err(|err| format!("run {} build: {err}", engine.name()))?; + if !output.status.success() { + return Err(format!( + "{} build failed (exit {:?}):\n{}{}", + engine.name(), + output.status.code(), + String::from_utf8_lossy(&output.stdout), + String::from_utf8_lossy(&output.stderr) + )); + } + Ok(Self { engine, tag }) + } + + #[must_use] + pub fn tag(&self) -> &str { + &self.tag + } +} + +impl Drop for ImageGuard { + fn drop(&mut self) { + let _ = self + .engine + .command() + .args(["image", "rm", "--force", &self.tag]) + .stdout(std::process::Stdio::null()) + .stderr(std::process::Stdio::null()) + .status(); + } +} + #[must_use] pub fn e2e_network_name() -> Option { std::env::var("OPENSHELL_E2E_NETWORK_NAME") diff --git a/e2e/rust/src/harness/host_process.rs b/e2e/rust/src/harness/host_process.rs new file mode 100644 index 000000000..f9fefd75a --- /dev/null +++ b/e2e/rust/src/harness/host_process.rs @@ -0,0 +1,103 @@ +// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +//! Host-process TCP fixtures for e2e tests. +//! +//! [`HostSupportContainer`](super::container::HostSupportContainer) publishes +//! the same shape of fixture through a container engine. VM sandboxes reach +//! the host through gvproxy's `host.openshell.internal` alias and the VM e2e +//! lane has no container runtime of its own, so this variant runs the fixture +//! as a plain host process instead — keeping the lane free of a container +//! dependency it does not otherwise need. + +use std::io::Read as _; +use std::path::PathBuf; +use std::process::{Child, Command, Stdio}; +use std::time::Duration; + +use super::port::wait_for_port; + +/// A `python3` fixture listening on a host TCP port. +/// +/// Output is captured to a temp file rather than a pipe: these fixtures log +/// every request they serve, and a full pipe buffer would block the process +/// mid-test. [`logs`](Self::logs) reads the file, which is where a test finds +/// its evidence (the CONNECT targets a proxy saw, for example). +pub struct HostPythonFixture { + /// Host port the fixture listens on. + pub port: u16, + child: Child, + log_path: PathBuf, +} + +impl HostPythonFixture { + /// Start `python3 -c