* feat(kubernetes): support HA gateway rebalancing Signed-off-by: Drew Newberry <anewberry@nvidia.com> * perf(server): cache peer connections, tokens, and owner lookups Every forwarded relay rebuilt its setup from scratch: an owner lookup, a blocking read of the peer token, a TLS connect to the owning replica, and a TokenReview plus Pod GET on the receiving side. Sandbox service routing does this per HTTP request, so the apiserver calls scaled with traffic. Cache all of it on ServerState: - peer channels pooled per endpoint, so relays multiplex over one connection instead of redialing - peer tokens keyed by SHA-256, expiring at min(ttl, token exp) so a hit cannot accept an expired token - owner records for 3s against a 45s ownership TTL, still freshness checked before use Entries are evicted when a relay fails. Also raise HTTP/2 max_concurrent_streams to 1024, since pooling funnels every relay between two replicas onto one connection and hyper's default of 200 sits below the 256 pending-relay budget. Signed-off-by: divesh <dgude@nvidia.com> * perf(server): pool upstream connections for sandbox services Each HTTP request to a sandbox service opened its own supervisor relay, paying a new TCP connection and HTTP/1 handshake every time. Worse, it counted against the 32 in-flight relay cap, so a service handling more than 32 concurrent requests failed outright. Pool idle upstreams per endpoint and port, up to 8 each for 15s. Reuse is safe because the pool only returns a connection hyper reports as ready, and HTTP/1 cannot start a request until the previous body has drained. Upgrades are never pooled since they take the connection over, and a failed send evicts that endpoint. Pruning is bounded per key, with the full sweep limited to once per 30s. Signed-off-by: divesh <dgude@nvidia.com> * fix(server): address HA gateway review findings (#3449) - Let a gateway own supervisor sessions without a peer endpoint. Requiring one whenever the store is PostgreSQL broke every single-instance PostgreSQL deployment, because no sandbox supervisor could connect. A cross-replica request to an owner that advertises no endpoint now fails immediately naming the cause, instead of retrying until the wait timeout. - Close a supervisor session on heartbeat only when another replica owns it, or after renewals fail for the ownership TTL. A database error no longer drops every session heartbeating during an outage. - Clamp owner record ages at zero so a skewed or corrupt stored timestamp cannot produce a negative age. - Bound the cross-object advisory lock with a lock timeout, so a stuck holder fails instead of blocking every mutation in the fleet. - Refuse to start when a peer endpoint is configured on a multi-replica backend but peer authentication is unavailable, and warn when a multi-replica backend has no peer endpoint at all. - Reject a plaintext peer endpoint when the gateway serves TLS. - Skip the sandbox watch poller on single-replica backends, where the local update bus already sees every write. - Rate-limit the peer owner cache sweep so an insert no longer scans the whole map under the lock. - Retry GET and HEAD on a pooled upstream the sandbox closed, instead of returning 502, and drop an emptied endpoint from the pool right away. - Document the gateway peer environment variables and the post-rollout ownership skew operators should expect. Signed-off-by: divesh <dgude@nvidia.com> * fix(server): harden HA supervisor ownership Signed-off-by: Drew Newberry <anewberry@nvidia.com> --------- Signed-off-by: Drew Newberry <anewberry@nvidia.com> Signed-off-by: divesh <dgude@nvidia.com> Co-authored-by: Drew Newberry <anewberry@nvidia.com> Co-authored-by: divesh <dgude@nvidia.com> Co-authored-by: Divesh Chowdary <47188680+FrostGod@users.noreply.github.com>
OpenShell SDK for Go
Important
Read the full documentation for guides, API reference with gRPC mapping, and testing patterns.
A Go SDK for interacting with OpenShell servers, providing idiomatic Go bindings for shell session management, command execution, provider configuration, and service exposure.
Why a Go SDK?
Go is the language of the Kubernetes ecosystem. If you want to build an operator, controller, or any automation that manages OpenShell resources as native Kubernetes objects, you need a Go client.
This SDK is modeled after
k8s.io/client-go, the standard
Kubernetes client library that every Go operator developer already knows. The
patterns will look familiar:
- Typed sub-clients per resource:
client.Sandboxes(),client.Providers(),client.Exec(), just likeclientset.CoreV1().Pods() - Domain types separated from wire formats: clean Go structs in a
typespackage, no proto leakage into the public API (likek8s.io/api) - Watch primitives: channel-based watchers with
ResultChan()andStop(), identical towatch.Interfacein client-go - Functional options: variadic option patterns for list filtering, pagination, and watch configuration. Nil options are silently ignored at every entry point, so conditional option lists are safe to pass without filtering out nil entries.
- Composable auth with token refresh: wraps
oauth2.TokenSourcefor automatic token caching and coalesced refresh, following the k8s client-gocachingTokenSourcepattern - Fake client for testing: an in-memory implementation of the full client
interface (like
k8s.io/client-go/kubernetes/fake), so operators can be tested without a real gateway
Quick Start
import v1 "github.com/NVIDIA/OpenShell/sdk/go/openshell/v1"
// Connect to a gateway
client, err := v1.NewClient(v1.Config{
Address: "gateway.example.com:443",
Auth: v1.StaticToken("my-token"),
})
if err != nil {
log.Fatal(err)
}
defer client.Close()
// Create a sandbox and wait until it's ready
sandbox, err := client.Sandboxes().Create(ctx, "default", "my-sandbox", &v1.SandboxSpec{
Template: &v1.SandboxTemplate{Image: "python:3.12"},
}, nil)
if err != nil {
log.Fatal(err)
}
sandbox, err = client.Sandboxes().WaitReady(ctx, "default", sandbox.Name)
if err != nil {
log.Fatal(err)
}
// Run a command
result, err := client.Exec().Run(ctx, "default", sandbox.Name,
[]string{"python3", "-c", "print('hello from sandbox')"},
v1.ExecOptions{},
)
if err != nil {
log.Fatal(err)
}
fmt.Println(string(result.Stdout))
Pagination
List methods return a lazy pager without issuing a request. NextPage fetches
one page with the supplied context, while ListAll explicitly exhausts every
page. PageSize is a per-request maximum and PageToken resumes a prior query.
pager, err := client.Sandboxes().List("default", v1.ListOptions{PageSize: 100})
if err != nil {
log.Fatal(err)
}
for {
page, err := pager.NextPage(ctx)
if err != nil {
log.Fatal(err)
}
if page == nil {
break
}
for _, sandbox := range page.Items {
fmt.Println(sandbox.Name)
}
}
With automatic token refresh
For OIDC gateways, use RefreshableToken to wrap any oauth2.TokenSource with
automatic caching and coalesced refresh:
import "golang.org/x/oauth2"
tokenSource := oauth2Config.TokenSource(ctx, initialToken)
auth, err := v1.RefreshableToken(tokenSource,
v1.WithLeeway(30*time.Second),
)
if err != nil {
log.Fatal(err)
}
client, err := v1.NewClient(v1.Config{
Address: "gateway.example.com:443",
Auth: auth,
})
if err != nil {
log.Fatal(err)
}
defer client.Close()
Concurrent callers share a single refresh call. If the token source fails, the SDK falls back to the cached token with a logged warning. See the Auth docs for details.
With edge proxy headers
When a gateway sits behind a zero-trust reverse proxy, use WithExtraHeaders to
attach proxy-specific headers alongside standard auth:
base := v1.StaticToken("my-gateway-token")
auth, err := v1.WithExtraHeaders(base, map[string]string{
"x-proxy-auth": "proxy-secret",
})
if err != nil {
log.Fatal(err)
}
client, err := v1.NewClient(v1.Config{
Address: "gateway.example.com:443",
Auth: auth,
})
For Cloudflare Access, use the convenience constructor in the edge package:
import "github.com/NVIDIA/OpenShell/sdk/go/openshell/v1/edge"
auth, err := edge.CloudflareAccess(base, os.Getenv("CF_ACCESS_TOKEN"))
For gRPC behind edge proxies that reject HTTP/2, use the WebSocket tunnel:
tunnel, err := edge.NewTunnelProxy(
"wss://gateway.example.com/ws",
os.Getenv("CF_ACCESS_TOKEN"),
)
if err != nil {
log.Fatal(err)
}
defer tunnel.Close()
client, err := v1.NewClient(v1.Config{
Address: tunnel.Addr(),
Auth: v1.StaticToken("my-token"),
TLS: &v1.TLSConfig{Insecure: true}, // local tunnel, no TLS
})
OIDC Login
The oidc package provides gateway-aware OIDC authentication with browser,
keyboard, device code, and client credentials flows:
import "github.com/NVIDIA/OpenShell/sdk/go/openshell/v1/oidc"
// Gateway-aware login: reads OIDC config from gateway metadata
token, err := oidc.Login(ctx, "my-gateway")
if err != nil {
log.Fatal(err)
}
// Use the token with the SDK client
client, err := v1.NewClient(v1.Config{
Address: "gateway.example.com:443",
Auth: v1.StaticToken(token.AccessToken),
})
For headless environments, use the device code flow:
token, err := oidc.DeviceLogin(ctx,
oidc.WithIssuer("https://auth.example.com"),
oidc.WithClientID("my-app"),
)
For a one-shot service-account token exchange, use ClientCredentials. For a
long-running SDK client, attach the renewable, memory-only auth provider:
auth, err := oidc.NewClientCredentialsAuth(
oidc.WithGateway("my-gateway"),
oidc.WithClientSecretProvider(func(context.Context) (string, error) {
return os.Getenv("OPENSHELL_OIDC_CLIENT_SECRET"), nil
}),
)
if err != nil {
log.Fatal(err)
}
client, err := v1.NewClient(v1.Config{
Address: "gateway.example.com:443",
Auth: auth,
TLS: tlsConfig,
})
if err != nil {
log.Fatal(err)
}
defer client.Close()
You can use explicit WithIssuer, WithClientID, WithScopes, and
WithAudience options instead of WithGateway. The provider repeats the grant
before expiry and never writes the secret or access token to disk.
See the oidc package docs for all options and flows.
See the Getting Started guide for the full walkthrough.
Migrating from v0.0.101
The pre-1.0 SDK intentionally includes source-incompatible API corrections:
TCP.Listenreturns aForwardListenerlifecycle handle. The SDK owns the accept loop; callers dialAddr()and callClose()instead of callingAccept()or passing the handle tohttp.Serve.- Resource operations take an explicit workspace, and workspace-bearing domain types preserve that scope.
- Several public struct field orders changed. Use keyed struct literals.
- Initialisms use Go spelling, including
JSONRPCMaxBodyBytes. - Provider profile durations use the exact
RefreshBefore,MaxLifetime, andCacheTTLfields. The legacy whole-second fields were removed.
These changes are intentional while the module remains below v1. Update callers as one migration rather than relying on the v0.0.101 API shape.
Architecture
Client
├── Sandboxes() → SandboxInterface (create, get, list, delete, watch, wait, logs)
├── Exec() → ExecInterface (run, stream, interactive)
├── Files() → FileInterface (upload, download)
├── Health() → HealthInterface (health check, gateway info, current user)
├── Services() → ServiceInterface (expose, get, list, delete)
├── Providers() → ProviderInterface (CRUD + ensure)
│ ├── Profiles() → ProfileInterface (list, get, import, update, lint, delete)
│ └── Refresh() → RefreshInterface (configure, status, rotate, delete)
├── Workspaces() → WorkspaceInterface (create, get, list, delete, members)
└── Policy() → PolicyInterface (draft review, approve, reject, merge, status)
All domain types live in openshell/v1/types/. Proto-to-Go conversions happen in
an internal converter layer. The public API surface uses type aliases so
consumers import a single package. See the Architecture overview for details.
Features
| Feature | Interface | Docs |
|---|---|---|
| Sandbox lifecycle (create, get, list, delete, watch, wait) | SandboxInterface |
Sandboxes |
| Command execution (collected, streamed, interactive PTY) | ExecInterface |
Exec |
| Provider management (CRUD + idempotent ensure) | ProviderInterface |
Providers |
| Provider profiles (list, import, lint, update) | ProfileInterface |
Profiles |
| Credential refresh (configure, rotate, status) | RefreshInterface |
Refresh |
| Service exposure (expose, list, delete) | ServiceInterface |
Services |
| File transfer API (transport capability-gated) | FileInterface |
Files |
| Policy management (draft review, approve, reject, merge, global policy) | PolicyInterface |
Policy |
| Sandbox logs (streaming retrieval) | SandboxInterface |
Sandboxes |
| Workspace management (create, get, list, delete, members) | WorkspaceInterface |
Workspaces |
| Sandbox provider attachment (attach, detach, list) | SandboxInterface |
Sandboxes |
| Gateway info and current user identity | HealthInterface |
Health |
| Health checking | HealthInterface |
Health |
| SSH tunneling and TCP forwarding | SSHInterface, TCPInterface |
SSH, TCP |
| Auth: static token, refreshable token (oauth2.TokenSource) | AuthProvider |
Auth |
| Edge auth: extra headers, Cloudflare Access, WebSocket tunnel | AuthProvider, edge.TunnelProxy |
Edge |
Typed errors (IsNotFound, IsAlreadyExists, IsConflict, ...) |
StatusError |
Error Handling |
| Real-time watch with auto-stop on terminal phase | WatchInterface[T] |
Sandboxes |
| Fake client for testing (no gRPC server needed) | fake.Client |
Testing |
| OIDC login and renewable service auth | oidc.Login, oidc.DeviceLogin, oidc.ClientCredentials, oidc.NewClientCredentialsAuth |
OIDC |
| Gateway config convenience (load CLI gateway configs, auto-wire auth) | gateway.NewClient, gateway.LoadConfig |
Gateway |
Prerequisites
- Go 1.25.13 or later
- mise (recommended for reproducible builds)
Build and Test
git clone https://github.com/NVIDIA/OpenShell.git
cd OpenShell/sdk/go
mise run test # Run tests with coverage
mise run lint # Run golangci-lint
mise run ci # Full CI pipeline (lint + build + test)
Build commands use mise for reproducible tool management.
Documentation
Full API documentation is available at the OpenShell Go SDK Docs site.
To build the docs locally:
cargo install mdbook
mdbook serve docs
License
Apache-2.0. See LICENSE for details.
Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.