trending/OpenShell
1
0
Fork 0
mirror of https://github.com/NVIDIA/OpenShell.git synced 2026-10-14 04:59:28 +08:00
Code Issues Packages Projects Releases Wiki Activity
Files
pull-request/2338
OpenShell/tasks/test.toml
T
Seth JenningsTaylor MutchVarsha Prasad Narsing
 5548405fcb feat(credentials): add provider credential storage drivers (#2437) 
* feat(credentials): add provider credential storage drivers

Signed-off-by: Taylor Mutch <taylormutch@gmail.com>

* fix(credentials): harden credential update handling

Signed-off-by: Taylor Mutch <taylormutch@gmail.com>

* fix(credentials): harden credential driver security, correctness, and performance

Address review findings from the credential storage drivers PR:

- Route additional_credentials through the driver on refresh to prevent
  silent data loss for multi-credential providers (e.g. AWS STS)
- Clean up stored credential handles on CAS failure during refresh to
  prevent orphaned secrets in external backends
- Enforce namespace validation in the Kubernetes Secrets driver to
  prevent cross-namespace credential access when allow_reference_namespace
  is not enabled
- Cache Vault Kubernetes auth tokens with 80% TTL to avoid re-authenticating
  on every credential operation
- Parallelize resolve_credentials in all three drivers using try_join_all
  for faster sandbox startup
- Add existingSecret support for the KEK Secret to fix helm template/GitOps
  workflows where lookup returns empty and regenerates the key
- Document RBAC blast radius for the Kubernetes Secrets credential driver
  and recommend a dedicated namespace

Signed-off-by: Varsha Prasad <varshaprasad96@gmail.com>
Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com>

* fix(credentials): add optimistic concurrency, fix thundering herd, parallelize operations

Use resourceVersion optimistic concurrency with retry loop for K8s
Secret ownership checks to prevent TOCTOU races. Switch Vault token
cache from RwLock to Mutex with double-check pattern to prevent
thundering herd on cache miss. Parallelize credential store and delete
operations across independent keys using try_join_all.

Signed-off-by: Varsha Prasad <varshaprasad96@gmail.com>
Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com>

* fix(credentials): handle partial failures, add delete retry, consolidate cleanup

Replace try_join_all with join_all in credential store/delete operations
to handle partial failures — successfully-stored handles are cleaned up
when another key fails. Add retry loop with conflict detection to
db-credstore delete_credential, matching the K8s driver pattern.
Consolidate 4 manual cleanup_pre_stored_provider_credentials call sites
into a single error handler using an async block. Remove inconsistent
.trim() from db-credstore validate_handle_owner.

Signed-off-by: Varsha Prasad <varshaprasad96@gmail.com>
Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com>

* fix(credentials): fix retry loop guard and remove unprotected validation

Remove attempt-count guard from 409/Aborted match arms in retry loops
so the post-loop Status::aborted error is reachable after exhausting
retries. Previously, last-attempt conflicts fell through to the
catch-all error arm, producing misleading Status::unavailable errors.

Remove duplicate validation calls that ran after
prepare_provider_credential_update but outside the cleanup-protected
async block, which would leak pre-stored handles on failure.

Signed-off-by: Varsha Prasad <varshaprasad96@gmail.com>
Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com>

* fix(credentials): add workspace/provider UUID to credential backend paths

Include workspace and provider ID in credential backend object paths to ensure
cross-workspace uniqueness and prevent credential collision (GATOR-1806c9be-01).

- Updated credential driver proto to include workspace and provider_id fields
- Modified Vault driver to include workspace/provider_id in managed_secret_path
- Modified Kubernetes Secrets driver to include workspace/provider_id in
  credential_owner_id and managed_secret_name
- Updated all credential runtime calls to pass workspace/provider_id
- Updated tests to use the new signatures

This prevents two workspaces sharing the same external credential store from
colliding on provider names, which was a critical security issue (CWE-639).

* fix(credentials): preserve provider-level expiration for handle-backed credentials

Compute effective expiration from both provider and driver values using the
earliest non-zero timestamp and skip expired values before insertion
(GATOR-1806c9be-02).

- Modified resolve_provider_handles to check provider credential_expires_at_ms
- Skip expired credentials during resolution instead of returning them
- Use effective expiration (min of provider and driver) in resolution results
- Fix inference.rs to preserve earliest expiration when merging

This ensures handle-backed credentials respect the same expiration semantics
as inline credentials.

* fix(credentials): stage refresh changes under new handles before validation

Stage credential replacements under new immutable handles instead of reusing
existing handles to prevent overwriting committed values before validation/CAS
(GATOR-1806c9be-03).

- Stage credentials with empty existing_handles map to force new handle creation
- Validate and CAS before the new values are committed to backend storage
- Delete old handles only after successful CAS
- On CAS failure, delete only the newly staged handles
- This prevents CWE-362/CWE-367 race conditions where failed refreshes could
  still modify or delete the active credential

The fix ensures that a rejected refresh cannot modify the backend object still
referenced by the committed provider record.

* fix(credentials): add timeouts to credential driver RPCs

Apply configured timeouts to both startup capability negotiation and runtime
RPCs to prevent indefinite hangs (GATOR-1806c9be-05).

- Add DEFAULT_CREDENTIAL_DRIVER_RPC_TIMEOUT_SECS constant (30s)
- Apply timeout to GetCapabilities during startup connection
- Apply timeout to all runtime RPCs (store, delete, resolve)
- Use tokio::time::timeout to bound the entire GetCapabilities operation
  during startup, not just the socket connection
- Return contextual deadline errors on timeout

This prevents a faulty or overloaded driver from hanging gateway operations
indefinitely.

* fix(credentials): fix test to use consistent workspace/provider identity

The Kubernetes auth Vault resolve test was constructing a managed path with
test-workspace/test-provider-id but sending default/prov-123 in the request,
causing validation to reject the request (GATOR-18e32351-01).

- Update test to use test-workspace and test-provider-id in the request to
  match the logical_path construction
- This ensures the test exercises the intended code path and validates
  Kubernetes auth resolution properly

The test now passes and correctly validates identity enforcement.

* fix(credentials): use unique staging ID for refresh to avoid overwrites

Stage refresh replacements under genuinely distinct immutable handles using
a unique staging ID to prevent overwriting committed values (GATOR-1806c9be-03).

- Generate a unique staging ID using UUID for each refresh operation
- Use this staging ID when storing credentials instead of the real provider ID
- Pass the same staging ID during cleanup on failure to delete only staged objects
- This ensures deterministic paths (Vault) and object names (K8s) don't collide
  with the committed provider's credentials

The fix prevents failed refreshes from silently replacing active credentials
or breaking providers by deleting still-referenced backend objects.

* fix(credentials): wrap credential driver RPCs in local timeouts

Add local tokio::time::timeout wrappers around credential driver RPCs to
bound non-compliant or stalled UDS peers (GATOR-1806c9be-05).

- Wrap StoreCredential, DeleteCredential, and ResolveCredentials in local timeouts
- Return contextual deadline_exceeded errors when timeouts occur
- Keep existing gRPC timeout metadata for compliant implementations
- GetCapabilities during startup was already wrapped in previous commit

This ensures a faulty local driver cannot hang gateway operations indefinitely,
even if it accepts the connection but never responds to the RPC.

* fix(credentials): preserve ownership for staged refreshes

Signed-off-by: Seth Jennings <sjenning@redhat.com>

* fix(credentials): bound startup capability probe

Signed-off-by: Seth Jennings <sjenning@redhat.com>

* test(provider): authenticate credential handler requests

Signed-off-by: Seth Jennings <sjenning@redhat.com>

* fix(ci): grant actions read to credential driver e2e

Signed-off-by: Seth Jennings <sjenning@redhat.com>

---------

Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
Signed-off-by: Varsha Prasad <varshaprasad96@gmail.com>
Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com>
Signed-off-by: Seth Jennings <sjenning@redhat.com>
Co-authored-by: Taylor Mutch <taylormutch@gmail.com>
Co-authored-by: Varsha Prasad Narsing <varshaprasad96@gmail.com>
2026-08-05 16:29:41 +00:00

182 lines
7.3 KiB
TOML
Raw Permalink Blame History

# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
# Test tasks (Rust + Python)
[test]
description = "Run all tests (Rust + Python)"
depends = ["test:rust", "test:python", "test:sbom", "test:install-sh", "test:build-env", "test:packaging-assets", "test:docs-website"]
["test:docs-website"]
description = "Test the docs-website sync script"
# --no-project skips the workspace (maturin) build; --with supplies pytest and
# the script's runtime dep (PyYAML), which live outside the project env.
run = "uv run --no-project --with pytest --with pyyaml pytest tasks/scripts/sync_docs_website_test.py"
["test:sbom"]
description = "Run SBOM tooling tests"
run = "uv run --no-project --with pytest pytest -o \"python_files=*_test.py\" deploy/sbom/"
hide = true
["test:install-sh"]
description = "Run focused install.sh shell tests"
run = "tasks/scripts/test-install-sh.sh"
hide = true
["test:build-env"]
description = "Run build-env.sh helper shell tests"
run = "tasks/scripts/test-build-env.sh"
hide = true
["test:packaging-assets"]
description = "Run static packaging asset tests"
run = "tasks/scripts/test-packaging-assets.sh"
hide = true
[e2e]
description = "Run all end-to-end tests (Rust + Python + MCP)"
depends = ["e2e:rust", "e2e:python", "e2e:mcp"]
["e2e:test"]
description = "Build the current checkout and run a named host or Nix test-guest E2E suite"
run = "e2e/run.sh"
["e2e:gpu"]
description = "Run Docker GPU end-to-end tests"
depends = ["e2e:docker:gpu"]
["e2e:workloads:build"]
description = "Build local GPU workload test images and manifest"
run = "bash tasks/scripts/e2e-gpu-build-images.sh"
["test:rust"]
description = "Run Rust tests"
run = [
# Run the workspace once without openshell-server so we can run that crate
# with test-only helpers enabled.
"cargo test --workspace --exclude openshell-server",
"cargo test -p openshell-server --features test-support",
]
hide = true
["test:python"]
description = "Run Python tests"
depends = ["python:proto"]
env = { UV_NO_SYNC = "1" }
run = "uv run pytest python/"
hide = true
["e2e:rust"]
description = "Run Rust CLI e2e tests against a Docker-backed gateway"
run = [
"e2e/with-docker-gateway.sh cargo test --manifest-path e2e/rust/Cargo.toml --features e2e-docker",
]
["e2e:websocket-conformance"]
description = "Run focused WebSocket conformance e2e tests against a Docker-backed gateway"
run = [
"e2e/with-docker-gateway.sh cargo test --manifest-path e2e/rust/Cargo.toml --features e2e-docker --test websocket_conformance",
]
["e2e:mcp"]
description = "Run MCP conformance e2e scenarios against one Docker-backed gateway (static defaults for spec 2025-11-25; set OPENSHELL_MCP_CONFORMANCE_SCENARIOS for a focused subset)"
run = "bash e2e/mcp-conformance.sh"
["e2e:nodejs"]
description = "Alias for e2e:mcp"
depends = ["e2e:mcp"]
["e2e:python"]
description = "Run Python e2e tests against a Docker-backed gateway (E2E_PARALLEL=N or 'auto'; default 5)"
depends = ["python:proto"]
env = { UV_NO_SYNC = "1", PYTHONPATH = "python" }
run = "e2e/with-docker-gateway.sh uv run pytest -o python_files='test_*.py *_test.py' -m 'not gpu' -n ${E2E_PARALLEL:-5} e2e/python"
["e2e:podman"]
description = "Run Rust CLI e2e tests against a Podman-backed gateway"
run = "e2e/rust/e2e-podman.sh"
["e2e:oidc-pkce"]
description = "Run Linux browser PKCE and RBAC e2e tests against Keycloak and a Podman gateway"
run = [
"CONTAINER_RUNTIME=podman e2e/with-keycloak.sh env OPENSHELL_E2E_OIDC_GATEWAY=1 e2e/with-podman-gateway.sh cargo test --manifest-path e2e/rust/Cargo.toml --features e2e-oidc-pkce --test oidc_pkce",
]
["e2e:oidc-pkce:docker"]
description = "Run Linux browser PKCE and RBAC e2e tests against Keycloak and a Docker gateway"
run = [
"CONTAINER_RUNTIME=docker e2e/with-keycloak.sh env OPENSHELL_E2E_OIDC_GATEWAY=1 e2e/with-docker-gateway.sh cargo test --manifest-path e2e/rust/Cargo.toml --features e2e-oidc-pkce --test oidc_pkce",
]
["e2e:oidc-python:docker"]
description = "Run Python OIDC and workspace authorization e2e tests against Keycloak and a Docker gateway"
depends = ["python:proto"]
env = { UV_NO_SYNC = "1", PYTHONPATH = "python" }
run = [
"CONTAINER_RUNTIME=docker e2e/with-keycloak.sh env OPENSHELL_E2E_OIDC_GATEWAY=1 e2e/with-docker-gateway.sh uv run pytest -m 'not gpu' e2e/python/oidc",
]
["e2e:podman:rootless"]
description = "Run Rust CLI e2e tests against a rootless Podman-backed gateway"
run = "e2e/rust/e2e-podman-rootless.sh"
["e2e:podman:gpu"]
description = "Run GPU e2e against a standalone gateway with the Podman compute driver"
env = { OPENSHELL_E2E_PODMAN_GPU = "1", OPENSHELL_E2E_PODMAN_TEST = "gpu", OPENSHELL_E2E_PODMAN_FEATURES = "e2e-podman-gpu" }
run = "e2e/rust/e2e-podman.sh"
["e2e:kubernetes"]
description = "Run Rust CLI e2e tests against an OpenShell gateway deployed on Kubernetes via Helm (set OPENSHELL_E2E_KUBE_CONTEXT to reuse a cluster; otherwise creates a local k3d cluster when k3d is installed; set OPENSHELL_E2E_KUBE_TEST=<name> to scope to one test)"
run = "e2e/rust/e2e-kubernetes.sh"
["e2e:kubernetes:v1alpha1"]
description = "Run Kubernetes e2e against Agent Sandbox v1alpha1"
env = { AGENT_SANDBOX_VERSION = "v0.4.6" }
run = "e2e/rust/e2e-kubernetes.sh"
["e2e:kubernetes:agent-sandbox-versions"]
description = "Run Kubernetes e2e against Agent Sandbox v1beta1 and v1alpha1"
run = [
"e2e/rust/e2e-kubernetes.sh",
"AGENT_SANDBOX_VERSION=v0.4.6 e2e/rust/e2e-kubernetes.sh",
]
["e2e:kubernetes:sidecar"]
description = "Run Kubernetes e2e with the supervisor sidecar topology overlay"
env = { OPENSHELL_E2E_KUBE_EXTRA_VALUES = "deploy/helm/openshell/ci/values-sidecar.yaml" }
run = "e2e/rust/e2e-kubernetes.sh"
["e2e:kubernetes:db"]
description = "Run Kubernetes e2e with all database backend scenarios (SQLite and external PostgreSQL with existingSecret)"
env = { OPENSHELL_E2E_KUBE_DB_SCENARIOS = "1" }
run = "e2e/rust/e2e-kubernetes.sh"
["e2e:kubernetes:credential-drivers"]
description = "Run Kubernetes e2e for provider credential storage backed by Kubernetes Secrets and Vault"
env = { OPENSHELL_E2E_CREDENTIAL_DRIVERS = "1", OPENSHELL_E2E_KUBE_TEST = "credential_drivers", OPENSHELL_E2E_KUBERNETES_FEATURES = "e2e,e2e-kubernetes,e2e-kubernetes-credential-drivers" }
run = "e2e/rust/e2e-kubernetes.sh"
["e2e:vm"]
description = "Start openshell-gateway with the VM compute driver and run VM e2e tests"
run = "e2e/rust/e2e-vm.sh"
["e2e:docker"]
description = "Run smoke e2e against a standalone gateway with the Docker compute driver"
run = "e2e/rust/e2e-docker.sh"
["e2e:mechanistic-smoke"]
description = "Run mechanistic L4 smoke against a Docker-backed gateway"
run = [
"cargo build -p openshell-cli",
"e2e/with-docker-gateway.sh bash -lc 'target/debug/openshell settings set --global --key agent_policy_proposals_enabled --value true --yes && OPENSHELL_BIN=$PWD/target/debug/openshell bash e2e/policy-advisor/mechanistic-smoke.sh'",
]
["e2e:docker:gpu"]
description = "Run GPU e2e against a standalone gateway with the Docker compute driver"
env = { OPENSHELL_E2E_DOCKER_GPU = "1", OPENSHELL_E2E_DOCKER_TEST = "gpu", OPENSHELL_E2E_DOCKER_FEATURES = "e2e-docker-gpu" }
run = "e2e/rust/e2e-docker.sh"
["e2e:openshift"]
description = "Run OpenShift database-backend integration scenarios against a live cluster (requires oc CLI authenticated to an OpenShift cluster)"
run = "e2e/rust/e2e-openshift.sh"
Reference in New Issue View Git Blame Copy Permalink