5548405fcb
* feat(credentials): add provider credential storage drivers Signed-off-by: Taylor Mutch <taylormutch@gmail.com> * fix(credentials): harden credential update handling Signed-off-by: Taylor Mutch <taylormutch@gmail.com> * fix(credentials): harden credential driver security, correctness, and performance Address review findings from the credential storage drivers PR: - Route additional_credentials through the driver on refresh to prevent silent data loss for multi-credential providers (e.g. AWS STS) - Clean up stored credential handles on CAS failure during refresh to prevent orphaned secrets in external backends - Enforce namespace validation in the Kubernetes Secrets driver to prevent cross-namespace credential access when allow_reference_namespace is not enabled - Cache Vault Kubernetes auth tokens with 80% TTL to avoid re-authenticating on every credential operation - Parallelize resolve_credentials in all three drivers using try_join_all for faster sandbox startup - Add existingSecret support for the KEK Secret to fix helm template/GitOps workflows where lookup returns empty and regenerates the key - Document RBAC blast radius for the Kubernetes Secrets credential driver and recommend a dedicated namespace Signed-off-by: Varsha Prasad <varshaprasad96@gmail.com> Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com> * fix(credentials): add optimistic concurrency, fix thundering herd, parallelize operations Use resourceVersion optimistic concurrency with retry loop for K8s Secret ownership checks to prevent TOCTOU races. Switch Vault token cache from RwLock to Mutex with double-check pattern to prevent thundering herd on cache miss. Parallelize credential store and delete operations across independent keys using try_join_all. Signed-off-by: Varsha Prasad <varshaprasad96@gmail.com> Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com> * fix(credentials): handle partial failures, add delete retry, consolidate cleanup Replace try_join_all with join_all in credential store/delete operations to handle partial failures — successfully-stored handles are cleaned up when another key fails. Add retry loop with conflict detection to db-credstore delete_credential, matching the K8s driver pattern. Consolidate 4 manual cleanup_pre_stored_provider_credentials call sites into a single error handler using an async block. Remove inconsistent .trim() from db-credstore validate_handle_owner. Signed-off-by: Varsha Prasad <varshaprasad96@gmail.com> Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com> * fix(credentials): fix retry loop guard and remove unprotected validation Remove attempt-count guard from 409/Aborted match arms in retry loops so the post-loop Status::aborted error is reachable after exhausting retries. Previously, last-attempt conflicts fell through to the catch-all error arm, producing misleading Status::unavailable errors. Remove duplicate validation calls that ran after prepare_provider_credential_update but outside the cleanup-protected async block, which would leak pre-stored handles on failure. Signed-off-by: Varsha Prasad <varshaprasad96@gmail.com> Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com> * fix(credentials): add workspace/provider UUID to credential backend paths Include workspace and provider ID in credential backend object paths to ensure cross-workspace uniqueness and prevent credential collision (GATOR-1806c9be-01). - Updated credential driver proto to include workspace and provider_id fields - Modified Vault driver to include workspace/provider_id in managed_secret_path - Modified Kubernetes Secrets driver to include workspace/provider_id in credential_owner_id and managed_secret_name - Updated all credential runtime calls to pass workspace/provider_id - Updated tests to use the new signatures This prevents two workspaces sharing the same external credential store from colliding on provider names, which was a critical security issue (CWE-639). * fix(credentials): preserve provider-level expiration for handle-backed credentials Compute effective expiration from both provider and driver values using the earliest non-zero timestamp and skip expired values before insertion (GATOR-1806c9be-02). - Modified resolve_provider_handles to check provider credential_expires_at_ms - Skip expired credentials during resolution instead of returning them - Use effective expiration (min of provider and driver) in resolution results - Fix inference.rs to preserve earliest expiration when merging This ensures handle-backed credentials respect the same expiration semantics as inline credentials. * fix(credentials): stage refresh changes under new handles before validation Stage credential replacements under new immutable handles instead of reusing existing handles to prevent overwriting committed values before validation/CAS (GATOR-1806c9be-03). - Stage credentials with empty existing_handles map to force new handle creation - Validate and CAS before the new values are committed to backend storage - Delete old handles only after successful CAS - On CAS failure, delete only the newly staged handles - This prevents CWE-362/CWE-367 race conditions where failed refreshes could still modify or delete the active credential The fix ensures that a rejected refresh cannot modify the backend object still referenced by the committed provider record. * fix(credentials): add timeouts to credential driver RPCs Apply configured timeouts to both startup capability negotiation and runtime RPCs to prevent indefinite hangs (GATOR-1806c9be-05). - Add DEFAULT_CREDENTIAL_DRIVER_RPC_TIMEOUT_SECS constant (30s) - Apply timeout to GetCapabilities during startup connection - Apply timeout to all runtime RPCs (store, delete, resolve) - Use tokio::time::timeout to bound the entire GetCapabilities operation during startup, not just the socket connection - Return contextual deadline errors on timeout This prevents a faulty or overloaded driver from hanging gateway operations indefinitely. * fix(credentials): fix test to use consistent workspace/provider identity The Kubernetes auth Vault resolve test was constructing a managed path with test-workspace/test-provider-id but sending default/prov-123 in the request, causing validation to reject the request (GATOR-18e32351-01). - Update test to use test-workspace and test-provider-id in the request to match the logical_path construction - This ensures the test exercises the intended code path and validates Kubernetes auth resolution properly The test now passes and correctly validates identity enforcement. * fix(credentials): use unique staging ID for refresh to avoid overwrites Stage refresh replacements under genuinely distinct immutable handles using a unique staging ID to prevent overwriting committed values (GATOR-1806c9be-03). - Generate a unique staging ID using UUID for each refresh operation - Use this staging ID when storing credentials instead of the real provider ID - Pass the same staging ID during cleanup on failure to delete only staged objects - This ensures deterministic paths (Vault) and object names (K8s) don't collide with the committed provider's credentials The fix prevents failed refreshes from silently replacing active credentials or breaking providers by deleting still-referenced backend objects. * fix(credentials): wrap credential driver RPCs in local timeouts Add local tokio::time::timeout wrappers around credential driver RPCs to bound non-compliant or stalled UDS peers (GATOR-1806c9be-05). - Wrap StoreCredential, DeleteCredential, and ResolveCredentials in local timeouts - Return contextual deadline_exceeded errors when timeouts occur - Keep existing gRPC timeout metadata for compliant implementations - GetCapabilities during startup was already wrapped in previous commit This ensures a faulty local driver cannot hang gateway operations indefinitely, even if it accepts the connection but never responds to the RPC. * fix(credentials): preserve ownership for staged refreshes Signed-off-by: Seth Jennings <sjenning@redhat.com> * fix(credentials): bound startup capability probe Signed-off-by: Seth Jennings <sjenning@redhat.com> * test(provider): authenticate credential handler requests Signed-off-by: Seth Jennings <sjenning@redhat.com> * fix(ci): grant actions read to credential driver e2e Signed-off-by: Seth Jennings <sjenning@redhat.com> --------- Signed-off-by: Taylor Mutch <taylormutch@gmail.com> Signed-off-by: Varsha Prasad <varshaprasad96@gmail.com> Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com> Signed-off-by: Seth Jennings <sjenning@redhat.com> Co-authored-by: Taylor Mutch <taylormutch@gmail.com> Co-authored-by: Varsha Prasad Narsing <varshaprasad96@gmail.com>
13 KiB
CI
This document describes how OpenShell's continuous integration works for pull requests, with a focus on what contributors need to do to get their PR tested.
For local test commands see TESTING.md. For PR conventions see CONTRIBUTING.md.
Overview
PR CI that runs on NVIDIA self-hosted runners uses NVIDIA's copy-pr-bot. The bot mirrors trusted PR commits to internal pull-request/<N> branches in this repository. The gated workflows trigger on pushes to those branches, not on the original PR.
Branch Checks run automatically after copy-pr-bot mirrors the PR. Required CI Gates posts PR-head statuses that verify the mirror exists, is current, and ran the expected push-based workflows. E2E suites are opt-in because they are more expensive and publish temporary images.
Merge queue validation is a second integration gate for main. After a PR has passed the required PR-head statuses, a maintainer adds it to the merge queue. GitHub creates a temporary merge-group branch that combines the latest main, the queued PR, and any earlier queued PRs. The same required OpenShell / ... status contexts are then published against the merge-group SHA before GitHub merges it.
Three opt-in labels enable the long-running E2E suites:
test:e2eruns the standard Docker, rootless Podman, Kubernetes, and VM E2E suites inBranch E2E Checkstest:e2e-gpuruns GPU E2E inBranch E2E Checkstest:e2e-kubernetesruns Kubernetes E2E with the HA Helm overlay (replicaCount: 2and bundled PostgreSQL) and the credential-driver suite (Kubernetes Secrets plus Vault) inBranch E2E Checks
When multiple labels are present, Branch E2E Checks builds the shared gateway and supervisor images once, builds one CLI artifact per runner architecture, builds the Linux VM driver artifact once, and fans out all enabled suites in parallel. Docker, Podman, GPU, Rust, Python, MCP, and VM E2E jobs reuse the matching prebuilt gateway and CLI binaries instead of compiling additional debug binaries in each job; Kubernetes E2E consumes the gateway image directly and reuses the prebuilt CLI. VM E2E also reuses the prebuilt VM driver artifact and falls back to local VM-driver/runtime preparation for local runs or workflow invocations that omit the artifact.
The OpenShell / E2E and OpenShell / GPU E2E required statuses are evaluated from separate suite result jobs inside that workflow. test:e2e-kubernetes is optional while Kubernetes HA and credential-driver behavior are under active iteration: failures are visible in the workflow run but do not publish a required CI gate status.
The GitHub ruleset should require the OpenShell / ... statuses published by Required CI Gates, not the push-triggered workflow jobs directly.
Commit signing
copy-pr-bot decides whether to mirror a PR automatically based on whether the author is trusted. For org members and collaborators, "trusted" means all commits in the PR are cryptographically signed. Unsigned commits, even from an org member, force the bot to wait for a maintainer's /ok to test <SHA>.
DCO sign-off (-s / Signed-off-by) is a separate requirement and does not count as commit signing. Dependabot-authored dependency update PRs are allowlisted in DCO Assistant because the bot cannot sign commits.
One-time setup with an SSH key
If you already use an SSH key for git push, you can reuse it as a signing key. (You can also generate a separate one - GitHub allows the same SSH key as both auth and signing.)
-
Generate a key (skip if reusing your existing SSH key):
ssh-keygen -t ed25519 -C "you@example.com" -f ~/.ssh/id_ed25519_signing -
Add the public key at https://github.com/settings/keys using New SSH key, and set Key type: Signing Key (not Authentication). Signing keys are managed separately from authentication keys, even when they reuse the same key material - you have to add the entry once for each role.
-
Configure git globally:
git config --global gpg.format ssh git config --global user.signingkey ~/.ssh/id_ed25519_signing.pub git config --global commit.gpgsign true git config --global tag.gpgsign true -
Verify on a test commit:
git commit --allow-empty -s -m "test: signing"Push the branch and confirm GitHub shows the commit as Verified.
Pull request flows
Internal contributor PR
Prerequisites:
- Org member or collaborator on the repo.
- All commits cryptographically signed (see Commit signing).
- All commits include a DCO sign-off (
git commit -s).
Flow:
- Open the PR. copy-pr-bot mirrors it to
pull-request/<N>automatically. - The mirror push runs
Branch Checksautomatically.Required CI Gateskeeps the PR blocked until the mirror exists, matches the PR head SHA, and the required push-based workflow succeeds. The firstBranch E2E Checksrun only resolves metadata and skips expensive jobs unless an E2E label is already set. - A maintainer applies
test:e2e,test:e2e-gpu, and/ortest:e2e-kubernetes.E2E Label Helpposts a comment with a link to the existing gated workflow run. - The maintainer opens that link and clicks Re-run all jobs. This time
pr_metadatasees the label and the build/E2E jobs run. - When the run finishes, the matching
OpenShell / ...gate status flips to green automatically. - New commits push to the mirror automatically and re-trigger
Branch Checksplus any labeled E2E jobs inBranch E2E Checks. - When the PR is ready to merge, use Add to merge queue instead of merging directly. The queue validates the final integration state before updating
main.
Forked PR
Prerequisites:
- DCO sign-off (
git commit -s) on every commit. Commit signing is not required for forks - copy-pr-bot trusts forks based on maintainer review, not signing. - A maintainer must vouch you. See the Vouch System.
Flow:
- Open the PR. The vouch check confirms you are vouched (otherwise the PR is auto-closed).
- copy-pr-bot does not mirror forks automatically. A maintainer reviews the diff and comments
/ok to test <SHA>with your latest commit SHA. - After
/ok to test, copy-pr-bot mirrors topull-request/<N>. From here the flow is identical to internal PRs:Required CI Gatesverifies the mirror and required push workflows, and maintainers apply the E2E label when the extra suites are needed. - When the PR is ready to merge, maintainers add it to the merge queue so the queued integration state is tested before it reaches
main.
Important: every new commit you push requires another /ok to test <new-SHA> from a maintainer before push-based CI will run on it. If a label is applied while the mirror is stale, E2E Label Help will post a comment explaining what's needed.
Merge queue
GitHub merge queue is required for main. Repository administrators must enable Require merge queue in the branch ruleset for main and keep these required status contexts aligned with the PR gates:
OpenShell / Branch ChecksOpenShell / E2EOpenShell / GPU E2EOpenShell / Helm Lint
Do not require the underlying workflow job names directly. Required CI Gates publishes stable commit statuses for both PR-head mirror commits and merge-group commits.
Merge-group runs use the merge_group event. The event is distinct from pull_request and push, and GitHub will not report required checks for queued PRs unless the workflows include it. In this repository:
Branch Checksruns the standard non-E2E gates on the merge-group SHA.Branch E2E Checksruns core E2E and GPU E2E for merge groups. Kubernetes HA E2E remains optional and label-driven on PRs.Helm Lintruns for merge groups without the PR diff optimization, because the merge-group branch is the final integration state.Required CI Gatesposts the sameOpenShell / ...statuses to the merge-group SHA and does not require apull-request/<N>mirror for merge-group events.
Maintainers should add ready PRs to the queue rather than pressing a direct merge button. GitHub removes a PR from the queue if the merge-group checks fail or time out.
copy-pr-bot
copy-pr-bot is a GitHub App maintained by NVIDIA that solves a specific GitHub Actions security problem: by default, pull_request-triggered workflows on a self-hosted runner can run an arbitrary contributor's code on hardware the project owns. For projects that need self-hosted runners (GPU access, ARM hardware, on-prem secrets), GitHub's recommended pattern is to never trigger workflows directly from external pull_request events.
copy-pr-bot enforces that pattern. When a PR is opened against this repository, the bot evaluates whether the change is trusted - by default, only commits authored by org members and signed with a verified key are trusted, and forks always need an explicit per-SHA approval. Once a change passes that check, the bot mirrors the PR head into a branch named pull-request/<N> inside this repository. Our self-hosted workflows then trigger on push to those mirror branches, never on the original pull_request event.
The user-visible consequences inside this repo:
- A PR cannot run E2E until copy-pr-bot has mirrored it. For trusted authors this happens within seconds of opening the PR; for forked PRs it requires a maintainer to comment
/ok to test <SHA>. - New commits to a fork need a fresh
/ok to test <new-SHA>before the mirror updates. - The
pull-request/<N>branches are not for humans to push to - they are managed by the bot.
The bot's full administrator documentation is internal to NVIDIA. The only command contributors may see in PR comments is /ok to test <SHA>, used by maintainers to approve a specific commit on a forked PR for testing.
Workflow files
| File | Role |
|---|---|
.github/workflows/branch-checks.yml |
Required non-E2E checks. Triggers on push: pull-request/[0-9]+ for PR mirrors and merge_group for queued merges. |
.github/workflows/branch-e2e.yml |
Standard, GPU, Kubernetes HA, and Kubernetes credential-driver E2E. PR mirror pushes use test:e2e, test:e2e-gpu, and test:e2e-kubernetes labels; merge groups run core and GPU E2E. |
.github/workflows/helm-lint.yml |
Helm chart validation. PR mirror pushes skip lint jobs unless Helm inputs changed; merge groups always validate Helm because they represent the final integration state. |
.github/actions/pr-gate/action.yml |
Composite action that resolves PR metadata and verifies the required label is set for PR mirror pushes. Non-push events are allowed through. |
.github/actions/pr-merge-base/action.yml |
Composite action that resolves and fetches the merge-base commit for pull-request/<N> push workflows. |
.github/workflows/required-ci-gates.yml |
Posts required PR-head and merge-group statuses for gated CI workflows. This is what branch protection and merge queue should require. |
.github/workflows/e2e-label-help.yml |
When a test:e2e* label is applied, posts a PR comment telling the maintainer the next manual step (re-run an existing workflow run, or /ok to test <SHA> to refresh the mirror). |
Release workflows
These workflows run after merge to publish dev/tagged artifacts and verify them. They are not PR-gated.
| File | Role |
|---|---|
.github/workflows/release-dev.yml |
Publishes the rolling dev build on every push to main. Builds gateway/supervisor images and binaries, packages, wheels, and pushes the Helm chart as oci://ghcr.io/nvidia/openshell/helm-chart:0.0.0-dev (plus an immutable 0.0.0-dev.<sha> pin). Also dispatchable manually. |
.github/workflows/release-tag.yml |
Publishes a tagged public release. |
.github/workflows/release-canary.yml |
Smoke-tests published artifacts on macos, ubuntu, fedora, and kubernetes (kind + Helm) runners. Triggers automatically when Release Dev succeeds, and via workflow_dispatch on any branch (gh workflow run release-canary.yml --ref <branch>). The kubernetes job pins to 0.0.0-dev artifacts; the other jobs install the latest tagged release via install.sh. See the test-release-canary skill for the manual-dispatch playbook and local kind reproduction. |
Required status contexts
Require these statuses in the branch ruleset for PR and merge-queue CI:
OpenShell / Branch ChecksOpenShell / E2EOpenShell / GPU E2EOpenShell / Helm Lint
Do not require the underlying workflow jobs directly. PR workflow jobs only appear after copy-pr-bot mirrors trusted code, and merge-group workflow jobs run on temporary queue branches. The stable OpenShell / ... contexts prove the expected workflow completed for the commit that GitHub is about to merge.