ShijuandJohn Myers
29a2b8e7d9
* feat(providers): report bounded token-grant failures Signed-off-by: Shiju <shiju@nvidia.com> * test(providers): align policy metadata and TLS lockfile Signed-off-by: Shiju <shiju@nvidia.com> * test(providers): fix trusted grant recovery fixture Signed-off-by: Shiju <shiju@nvidia.com> * test(providers): enable denial API in recovery fixture Signed-off-by: Shiju <shiju@nvidia.com> * refactor(providers): simplify token-grant failure reporting Remove duplication and dead paths from the token-grant failure reporting without changing classification, the workload-visible 502 response, or the wire format. - Build the gRPC failure envelope with tonic-types, store the typed failure in EndpointFailure, and share one request-ID rule and one endpoint-status authority check. - Merge the gateway exchange client wrapper, drop forwarding validators, and name the request field when a reported cause is rejected. - Inject fallback credentials only from live provider state and delete the unread per-connection credential copy. Keep one 502 writer per path and restore the base token-cache handling. - Check the token endpoint after the cache lookup, still before any credential acquisition. - Document 429 and 5xx classification as implemented, and drop tests that repeat retained assertions through the same code path. Signed-off-by: Shiju <shiju@nvidia.com> * refactor(providers): trim token-grant failure tests and supervisor API Cut the token-grant failure tests to one test per distinct behavior across core, gateway, supervisor, CLI and Go, and replace the 794-line closure E2E with status assertions in the existing route-recovery E2E: the refused grant appears in sandbox status with the response's request ID, code, stage and upstream status, and the matching grant's later success clears it. - Pin the live provider snapshot with one helper used by inspected and fallback forwarding, and inject selector-matched grants through inject_for_selectors, which borrows the pinned context. Move the test-only wrapper into test_support. - Drop the 35-second wrappers around the gateway grant RPC. The shared channel bounds the connection attempt and the gateway bounds its token-service calls. - Log the token service's OAuth error and error_description, without control characters and limited to 256 characters each, with the request ID in the gateway or supervisor log. Status and the workload response still carry only the bounded cause. - Keep one check that an untrusted token-service certificate cannot assert a cause. Signed-off-by: Shiju <shiju@nvidia.com> * fix(providers): restore test import and features lockfile The injection tests named EndpointObservationSender through the parent module, which no longer imports it. The features suite lockfile kept the tonic TLS dependency that the removed E2E fixture added. Signed-off-by: Shiju <shiju@nvidia.com> * test(providers): fix body-bound fixture width and table type Pass the response-size width as a named argument; an implicitly captured constant panics at runtime. Name the mutation type in the validation table for Clippy. Signed-off-by: Shiju <shiju@nvidia.com> * test(providers): pad exact-bound body without a format width MAX_TOKEN_RESPONSE_BYTES exceeds u16::MAX, the largest format width, so the padded fixture panicked. Pad with repeat_n as before. Signed-off-by: Shiju <shiju@nvidia.com> * test(server): update public RPC schema hash after rebase Main added the ListProvidersRequest label selector. The merged public schema hash comes from the Branch Checks storage_proto test output. Signed-off-by: Shiju <shiju@nvidia.com> * fix(providers): keep token-service error text out of grant logs A token service can echo a submitted credential in its OAuth error or error_description. Log only the classified failure code, HTTP status and request ID, and cover the canary echo in a dedicated test binary. Signed-off-by: Shiju <shiju@nvidia.com> * docs(providers): tighten token grant failure documentation Signed-off-by: Shiju <shiju@nvidia.com> * docs(providers): condense token grant failure guidance Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> --------- Signed-off-by: Shiju <shiju@nvidia.com> Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> Co-authored-by: John Myers <9696606+johntmyers@users.noreply.github.com>
OpenShell gateway server
The server owns control-plane RPCs, authentication, provider credential delivery,
and sandbox lifecycle orchestration. The openshell-gateway crate composes the
server with selected compute drivers.
Operator credential export
GetProviderCredentials exports selected profile-declared runtime environment
keys from one provider in an explicitly named workspace. It requires the distinct
Principal::Operator, established only from a verified TLS leaf certificate with
OU=operator when mtls_auth.operator_enabled is enabled. The TLS listener supplies
its certificate fingerprint separately from user identity; the ordinary
identity-only serving path cannot establish an operator.
Operators explicitly inherit user/platform-admin permissions, but not sandbox or peer identities. Descriptor authorization and a handler guard both protect the export RPC. Operator mutation admission uses a separate caller namespace from ordinary mTLS users with the same certificate subject. Credential responses never enter durable mutation replay or generic interceptor routing.
The export resolver validates the entire selection before minting, resolves only selected credential handles, and fences provider revisions, refresh epochs, and profile declarations before delivery. Driver expiration and provider expiration combine using the earlier timestamp. OAuth minting inputs, broker-only keys, and supervisor-bound dynamic grants are not exportable. A failed response can still have committed refresh effects. Missing refresh-owned stored outputs trigger refresh-on-read; missing static outputs and other credential-backend errors fail closed without falling back to inline values.
Refresh coordination
Explicit rotation, background refresh, and operator refresh-on-read share the same refresh coordination:
- A process-local weak keyed mutex coalesces waiters without retaining deleted providers indefinitely.
- A bounded admission queue and executor pool limit detached refresh work.
- PostgreSQL deployments also take a dedicated per-provider/per-credential advisory lock; SQLite uses the local mutex. Worker secret cleanup takes the same locks and reloads state before changing its resource version.
- A durable
refresh_in_progressmarker precedes upstream minting. Arefresh_committingmarker protects the interval between grant persistence and provider-handle persistence. Only a completed provider commit publishesrefreshedstate, with a unique completed-mint identity in metadata annotations. Waiters coalesce against that identity, not bookkeeping changes to the resource version. - Automatic callers recheck live recovery and retry deadlines under both locks. A failed mint blocks queued automatic exchanges without disabling scheduled worker retries or a separately requested manual rotation.
- Detached refresh tasks complete despite request cancellation, preserving replacement refresh tokens returned by the issuer.
- Existing generation/CAS fencing remains active for deletion and reconfiguration. An ambiguous rotating-token exchange is parked for recovery rather than automatically consuming the grant again.
A crash or lost database-lock session can leave a parked marker. Reconfigure or reauthorize the refresh grant after investigating its issuer state. Changing a marker directly is not a supported recovery workflow.
Focused verification
cargo test -p openshell-server --features test-support,prebuilt-z3
cargo clippy -p openshell-server --all-targets --features test-support,prebuilt-z3 -- -D warnings
The ignored postgres_refresh_locks_coordinate_independent_replica_sessions test
runs under mise run test:rust:postgres (see TESTING.md) and still accepts
OPENSHELL_REFRESH_TEST_DATABASE_URL pointing to a disposable PostgreSQL database.
It exercises independent sessions without the local mutex. Do not point this
fixture at an existing deployment database.