mirror of
https://github.com/NVIDIA/OpenShell.git
synced 2026-10-21 06:11:31 +08:00
* feat(api): return typed deletion outcomes with explicit missing-target semantics Implement phase 2 of #3051 across the public gateway API and first-party SDKs. Preserve asynchronous sandbox deletion and observed resource identities, reserve legacy wire fields, and document the coordinated migration. Signed-off-by: Mrunal Patel <mrunalp@gmail.com> * fix(sdk): bind deletion waits to sandbox identity Track accepted deletions by original sandbox identity in Rust and TypeScript. Preserve name-only waits and cover replacement races and lookup failures. Merge the phase-one cleanup fix and adapt its interceptor regressions to typed deletion outcomes. Refs #3051. Signed-off-by: Mrunal Patel <mrunalp@gmail.com> * test(e2e): accept asynchronous stopped sandbox deletion Allow either completed or accepted deletion output, then continue polling for actual sandbox absence. This preserves the lifecycle assertion under the phase-two deletion contract. Refs #3051. Signed-off-by: Mrunal Patel <mrunalp@gmail.com> --------- Signed-off-by: Mrunal Patel <mrunalp@gmail.com>
161 lines
9.3 KiB
Plaintext
161 lines
9.3 KiB
Plaintext
---
|
|
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
|
|
# SPDX-License-Identifier: Apache-2.0
|
|
title: "API Errors"
|
|
sidebar-title: "API Errors"
|
|
description: "Inspect structured gateway failures and choose an appropriate recovery action."
|
|
keywords: "OpenShell, SDK, gRPC, API errors, retries"
|
|
---
|
|
|
|
Gateway errors include a gRPC status code and a message. Shared request validation
|
|
and concurrency checks also return standard protobuf details in
|
|
`grpc-status-details-bin`. Use the status code and structured fields for decisions.
|
|
Treat messages as explanations whose text can change.
|
|
|
|
## Structured details
|
|
|
|
The SDKs decode these standard detail types while retaining the original failure.
|
|
Older gateways and checks that have not adopted structured details can return only
|
|
a code and message. Missing details do not change the meaning of the status code.
|
|
|
|
| Detail | Meaning |
|
|
|---|---|
|
|
| `google.rpc.BadRequest` | `field_violations` identifies rejected fields and explains each violation. Shared sandbox, exec, provider size, and workspace selector checks supply these details. |
|
|
| `google.rpc.ErrorInfo` | `reason`, `domain`, and `metadata` identify a failure without parsing its message. Gateway reasons use the `openshell.nvidia.com` domain. |
|
|
| `google.rpc.RetryInfo` | `retry_delay` gives a minimum delay before an otherwise safe retry. Its presence does not guarantee that a mutation has not already committed. |
|
|
|
|
Recognized gateway reasons include the following.
|
|
|
|
| Reason | Code | Recovery |
|
|
|---|---|---|
|
|
| `INVALID_ARGUMENT` | `INVALID_ARGUMENT` | Correct the fields listed in `BadRequest`. |
|
|
| `RESOURCE_VERSION_CONFLICT` | `ABORTED` | Read the resource again and construct a new conditional write. `metadata.recovery` is `REFRESH_STATE`; `current_resource_version` is included when known. |
|
|
| `PROFILE_SOURCE_UNAVAILABLE` | `UNAVAILABLE` | Retry a profile snapshot read after at least the supplied delay. |
|
|
|
|
## Status and retry guidance
|
|
|
|
A timeout or disconnected transport can occur after a mutation commits. Do not
|
|
automatically repeat creates, credential rotation, or command execution based
|
|
only on a transient status. Retry a mutation only when its documented operation
|
|
contract makes the repeated request safe. A correlation ID does not provide that
|
|
guarantee.
|
|
|
|
| Status | Recovery |
|
|
|---|---|
|
|
| `INVALID_ARGUMENT`, `OUT_OF_RANGE` | Correct the request. |
|
|
| `UNAUTHENTICATED` | Refresh or replace credentials before a new attempt. |
|
|
| `PERMISSION_DENIED` | Obtain the required authorization. |
|
|
| `NOT_FOUND` | Check the resource and workspace. Follow the operation's missing-resource contract. |
|
|
| `ALREADY_EXISTS` | Inspect the existing resource before deciding whether it satisfies the request. |
|
|
| `FAILED_PRECONDITION` | Resolve the reported state or configuration requirement. |
|
|
| `ABORTED` | Read fresh state before retrying a conditional operation. |
|
|
| `UNAVAILABLE`, `RESOURCE_EXHAUSTED` | A transient condition may recover. Apply backoff and any minimum retry delay only to a retry-safe operation. |
|
|
| `DEADLINE_EXCEEDED`, `CANCELLED` | A mutation's outcome can be unknown. Cancellation does not imply rollback. |
|
|
| `UNIMPLEMENTED` | Check gateway and SDK version compatibility. |
|
|
| `INTERNAL`, `UNKNOWN`, `DATA_LOSS` | Preserve the status and correlation metadata for diagnosis. Do not blindly retry mutations. |
|
|
|
|
## SDK access
|
|
|
|
Each SDK exposes decoded fields and an escape hatch for complete transport data.
|
|
Unknown details remain available through the original error even when the SDK
|
|
does not recognize their message type.
|
|
|
|
| SDK | Decoded details | Original failure |
|
|
|---|---|---|
|
|
| Rust | `SdkError::error_details()`, `retry_delay()` | `SdkError::grpc_status()` returns the original `tonic::Status`, including details bytes and metadata. |
|
|
| Go | `StatusError.FieldViolations`, `ErrorInfo`, `RetryDelay`, `GRPCCode` | `StatusError.Cause` retains the gRPC error. `status.FromError` can recover its details through error wrapping. |
|
|
| TypeScript | `SdkError.fieldViolations`, `errorInfo`, `retryDelayMs`, `connectCode` | `SdkError.cause` retains the `ConnectError`, including all details and metadata. Use `fromConnect` for raw calls. |
|
|
| Python | `GatewayError.field_violations`, `error_info`, `retry_delay` | `raw_error` retains the gRPC exception and metadata; `raw_status` retains the parsed envelope. Use `from_grpc_error` for raw calls. |
|
|
|
|
Retry delays are Rust and Go durations, TypeScript milliseconds, and Python
|
|
seconds. Absent or invalid delay details do not produce a suggested delay.
|
|
|
|
## Migration
|
|
|
|
Existing error codes and human-readable messages remain available. Python curated
|
|
clients now raise `GatewayError`, which remains a `grpc.RpcError`; existing
|
|
`except grpc.RpcError` handlers continue to work. `GatewayError` is not a
|
|
`grpc.Call`. If your handler checks that interface, inspect `raw_error` before
|
|
checking the status, or call `GatewayError.code()` directly. Python deletion waits
|
|
recognize `NOT_FOUND` through the wrapper; managed-sandbox cleanup requests
|
|
`allow_missing=True` and handles the typed deletion outcome. Other failures still
|
|
propagate. Rust error variants now retain
|
|
status fields; use `..` when destructuring variants that do not need those fields.
|
|
Go and TypeScript add typed detail fields to their existing error types.
|
|
|
|
No SDK automatically retries a mutation as a result of decoding these details.
|
|
|
|
## Deletion outcomes
|
|
|
|
Delete, membership-removal, and SSH-revocation RPCs return a typed outcome.
|
|
Transport success alone does not establish completion.
|
|
|
|
| Outcome | Meaning |
|
|
|---|---|
|
|
| `COMPLETED` | The gateway resource was removed, or the existing SSH session is revoked. Downstream platform garbage collection may still be finishing. |
|
|
| `ACCEPTED` | Sandbox deletion started, but the gateway sandbox record still exists. Observe its removal before assuming completion. |
|
|
| `ALREADY_ABSENT` | The target was missing when resolved, and the request set `allow_missing=true`. |
|
|
| `UNSPECIFIED` or an unknown value | Completion is not established. Check gateway and SDK compatibility. |
|
|
|
|
Requests default to `allow_missing=false`: an initially missing target returns
|
|
`NOT_FOUND`. Set it to `true` for cleanup that allows an absent target. This flag
|
|
does not suppress missing parents, authorization failures, invalid requests,
|
|
failed preconditions, or backend errors. Re-revoking an existing revoked session
|
|
returns `COMPLETED` after authorization without changing its resource version.
|
|
|
|
`DeleteSandbox` also returns the original `sandbox_id` when it found a target.
|
|
For an accepted deletion, poll or watch that identity. A same-name sandbox with a
|
|
different ID is not the sandbox being deleted. Cancellation or a disconnected
|
|
client does not stop the gateway's already-started deletion worker.
|
|
|
|
Pass the deletion result's ID to the SDK wait helper to avoid following a
|
|
same-name replacement.
|
|
|
|
| SDK | Identity-aware deletion wait |
|
|
|---|---|
|
|
| Rust | `client.wait_deleted(name, timeout, result.sandbox_id.as_deref()).await?` on either the default or workspace-scoped client. |
|
|
| Python | `client.wait_deleted(name, workspace=workspace, expected_sandbox_id=result.sandbox_id)`. Managed-sandbox cleanup supplies this automatically. |
|
|
| TypeScript | `client.sandbox.waitDeleted(name, timeoutSecs, { workspace, expectedSandboxId: result.sandboxId })`. |
|
|
|
|
These waits complete on `NOT_FOUND` or a different observed ID. Without an
|
|
expected ID, they wait for the name to be absent. Existing Rust callers must add
|
|
the third argument; use `None` to retain name-only behavior.
|
|
|
|
`allow_missing` is not request deduplication. A later request by name can delete
|
|
a newly created resource with that name. Do not blindly repeat a timed-out
|
|
deletion when names might be reused.
|
|
|
|
### SDK migration for deletion
|
|
|
|
Upgrade gateways and clients together. This pre-1.0 change removes the boolean
|
|
response fields, reserving their names and wire numbers. New clients reading an
|
|
old response see `UNSPECIFIED`; they must not interpret it as completion.
|
|
|
|
| RPC | Removed field | Replacement |
|
|
|---|---|---|
|
|
| `DeleteSandboxTemplate` | `deleted` | `outcome` |
|
|
| `DeleteSandbox` | `deleted` | `outcome`, `sandbox_id` |
|
|
| `DeleteService` | `deleted` | `outcome` |
|
|
| `DeleteProvider` | `deleted` | `outcome` |
|
|
| `DeleteProviderRefresh` | `deleted` | `outcome` |
|
|
| `DeleteProviderProfile` | `deleted` | `outcome` |
|
|
| `DeleteWorkspace` | `deleted` | `outcome` |
|
|
| `RemoveWorkspaceMember` | `removed` | `outcome` |
|
|
| `RevokeSshSession` | `revoked` | `outcome` |
|
|
|
|
These RPCs use a typed result rather than `Empty` because callers must
|
|
distinguish absence, asynchronous acceptance, and completion.
|
|
|
|
Curated SDK methods return `DeletionResult` instead of a boolean or no result.
|
|
Rust accepts `DeleteOptions { allow_missing: true }`; Python accepts the keyword
|
|
`allow_missing=True`; TypeScript accepts `{ allowMissing: true }`; Go accepts an
|
|
optional `DeleteOptions{AllowMissing: true}` and returns `(result, error)`.
|
|
Go and Python preserve unknown numeric outcomes, Rust uses `Unknown(i32)`, and
|
|
TypeScript returns `outcome: 'unknown'` with `rawOutcome`.
|
|
|
|
Other mutation responses remain unchanged: provider attach/detach also return
|
|
the resulting sandbox, profile import/update return resources and diagnostics,
|
|
and config updates return version/application information. Their supplementary
|
|
booleans are not the sole result. Internal compute-driver and supervisor
|
|
contracts are separate from this public API migration.
|