Closes#3746
Use Homebrew atomic_write for exact legacy config migrations and keep the first-install write distinct in the formula test.
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* feat(sdk): add TypeScript SDK (@nvidia/openshell-sdk)
First native, per-language SDK for the OpenShell gateway: a thin, idiomatic
TypeScript client over proto-generated gRPC stubs (connect-es), no FFI. Covers
the v0.1 surface — sandbox lifecycle (create/get/list/delete + waitReady/
waitDeleted), health, and streamed exec.
- sdk/typescript/: package, client/transport/errors, protoc + protoc-gen-es
codegen (gen/ gitignored, absorbed into dist/ at build), committed lockfile.
- tasks/typescript.toml: sdk:ts install/proto/typecheck/build/ci/publish;
sdk:ts:typecheck wired into `check`; sdk-typescript job in branch-checks
(typecheck, build, and a --dry-run publish that validates the release path).
- Enforce SPDX headers on .ts/.tsx/.mts/.cts (skip node_modules and gen/);
back-fill docs/_components/jsx.d.ts and fern/components/CustomFooter.tsx.
- release.py gains an npm version format; release-tag.yml publishes to
GitHub Packages on tag, stamping the version (0.0.0 placeholder in git);
prerelease builds publish under the `next` dist-tag, not `latest`.
Ships as @nvidia/openshell-sdk on GitHub Packages pre-GA; public npm
(@openshell/sdk) follows at GA with an unchanged public API.
Signed-off-by: Max Dubrinsky <mdubrinsky@nvidia.com>
* chore(sdk): adopt TypeScript 6, tidy @types/node range
- typescript ^5.7.2 -> ^6.0.3 (6.0 is now `latest`; the old caret capped at 5.x)
- @types/node ^24.0.0 -> ^24 (same range, tidier)
No source changes; codegen, typecheck, and build pass on 6.0.3. Verified the
emitted d.ts still type-check for downstream consumers on TypeScript 5.0.4
through 5.9.3, so this does not raise the SDK's consumer TS floor.
Signed-off-by: Max Dubrinsky <mdubrinsky@nvidia.com>
* refactor(sdk): group operations under a composable SandboxClient
Reshape the client from flat methods (createSandbox, listSandboxes, exec) to a
scoped SandboxClient reached as `client.sandbox.create/get/list/delete/exec`
(+ waitReady/waitDeleted), mirroring the CLI's noun-verb model and the Python
SDK's SandboxClient.
SandboxClient is also usable standalone via SandboxClient.connect();
OpenShellClient composes it over a single shared transport, so future
service/provider clients reuse one connection. health() stays top-level as a
gateway call. No behavior change; types are unchanged.
Signed-off-by: Max Dubrinsky <mdubrinsky@nvidia.com>
* chore(sdk): generate TypeScript SDK stubs with buf
Replace the protoc gen.sh with `buf generate` + buf.gen.yaml. `buf`
(@bufbuild/buf) is a package devDependency and self-compiles the protos, so
the TS SDK no longer depends on the mise-pinned protoc; it drives the same
connect-es plugin. Generation stays limited to the client-surface closure
(openshell/sandbox/datamodel) via the input paths.
Output is byte-identical to the previous protoc + protoc-gen-es pipeline. Lays
the groundwork for a shared buf.yaml (lint/breaking/LSP) as a follow-up.
Signed-off-by: Max Dubrinsky <mdubrinsky@nvidia.com>
* build(proto): add repo-level buf module with lint
Declare proto/ as a single buf v2 module in a root buf.yaml so buf
generate, lint, breaking, and the editor LSP resolve imports the same
way. Lint uses STANDARD with six documented exceptions for deviations
the current protos intentionally make: the flat proto/ layout with
nested packages (DIRECTORY_SAME_PACKAGE, PACKAGE_DIRECTORY_MATCH) and
the established API shape with unsuffixed services and reused
request/response messages (RPC_REQUEST_RESPONSE_UNIQUE,
RPC_REQUEST_STANDARD_NAME, RPC_RESPONSE_STANDARD_NAME, SERVICE_SUFFIX).
Every other STANDARD rule now enforces on future protos. Breaking uses
FILE.
Code generation stays package-scoped in sdk/typescript/buf.gen.yaml
since it binds to that package's connect-es plugin and output dir;
its inputs are unchanged and regeneration is byte-identical.
Wire the check in via a proto:lint mise task that runs buf from the
SDK devDependencies. It is a dependency of both sdk:ts:ci (so the
TypeScript SDK CI job enforces it) and the top-level lint aggregate
(so local pre-commit covers it).
Signed-off-by: Max Dubrinsky <mdubrinsky@nvidia.com>
* chore(sdk): publish as unscoped openshell-sdk on public npm
Rename the package from @nvidia/openshell-sdk to the unscoped
openshell-sdk and target public npm (registry.npmjs.org) instead of
GitHub Packages. GitHub Packages requires a scope matching the owning
org, and the @openshell scope is blocked by an unrelated existing
package, so an unscoped name on public npm is the lowest-friction
distribution path and needs no org approval.
Rework the release-tag publish job to auth against registry.npmjs.org
with NPM_TOKEN (the job now only needs packages: read to pull the CI
image). Update the README install instructions and usage imports.
Signed-off-by: Max Dubrinsky <mdubrinsky@nvidia.com>
* chore(sdk): publish @nvidia/openshell-sdk to GitHub Packages
Revert the unscoped-name switch. GitHub Packages only accepts scoped
names matching the owning org, so shipping there first (which needs no
external npm org or NPM_TOKEN, just the repo's GITHUB_TOKEN) requires
the @nvidia scope. Keeping the @nvidia/openshell-sdk name also lets a
later public-npm release use the same install specifier, so adding
public npm becomes a second publish step rather than a rename.
Restore the GitHub Packages publish auth in the release-tag job and the
scoped install instructions in the README (keeping the buf codegen
note).
Signed-off-by: Max Dubrinsky <mdubrinsky@nvidia.com>
* feat(sdk-ts): add streaming exec, forward, ssh, provider, and config methods
Grow SandboxClient to the surface the first two consumers need. execStream
yields stdout/stderr chunks as they arrive and exec now drains it, keeping its
buffered ExecResult and signature unchanged. execInteractive is the TTY + stdin
transport primitive (start-first framing, output/write/resize/close/done, no
terminal glue). forward binds a local TCP listener that tunnels each accepted
connection into the sandbox for the process lifetime, minting and revoking a
per-socket SSH session token around a forwardTcp bidi. Adds createSshSession /
revokeSshSession, attach/detach/listProviders, and getConfig / setPolicy /
setSetting (sandbox-scoped, network-policy-only, with an optional wait poll).
Signed-off-by: Max Dubrinsky <mdubrinsky@nvidia.com>
* build(sdk-ts): add Biome and Vitest tooling
The TypeScript SDK had no formatter or linter and no test runner. Add Biome
(format + lint, generated src/gen excluded) enforcing 2-space indent, single
quotes, semicolons, and a 120-column width, and reformat the existing
hand-written sources accordingly. Add Vitest for unit tests. Wire sdk:ts:format,
sdk:ts:lint, and sdk:ts:test mise tasks into the fmt/lint aggregates, the root
test suite, and sdk:ts:ci so they run in CI.
Signed-off-by: Max Dubrinsky <mdubrinsky@nvidia.com>
* test(sdk-ts): cover the sandbox surface with in-memory transport tests
Exercise SandboxClient against an in-memory OpenShell service built with
createRouterTransport: request assembly and id resolution, u64/int64 rendered as
strings, enum lowercasing, fromConnect code mapping, the exec/execStream drain
plus a backward-compat check on exec, execInteractive start-first ordering and
done resolution, and a forward() byte relay against a loopback echo with close()
teardown.
Signed-off-by: Max Dubrinsky <mdubrinsky@nvidia.com>
* docs(sdk-ts): document the new surface and connect/upload/download boundaries
Document execStream, execInteractive, forward, ssh sessions, providers, and
config/policy in the SDK README, and record the intentional boundaries:
interactive connect / PTY ownership, upload/download (no file-transfer RPC), and
detached forwards stay out of scope. Note the Biome/Vitest dev commands.
Signed-off-by: Max Dubrinsky <mdubrinsky@nvidia.com>
* feat(sdk-ts): support mTLS client authentication
Add clientCert and clientKey to ConnectOptions so the SDK can
authenticate to the default local gateway, which uses mTLS user
authentication. Without a client certificate and key the SDK could
verify the server but never authenticate the caller, so it could not
connect to the standard Docker, VM, Homebrew, or Linux-package gateway.
Validate the pair as both-or-neither and pass cert and key through to
the Node TLS options for https gateways. The h2c path is unchanged.
Signed-off-by: Max Dubrinsky <mdubrinsky@nvidia.com>
* chore(sdk-ts): drop the demo script and its tsx dependency
Remove src/demo.ts, the demo npm script, the tsx devDependency, and the
tsconfig build exclude for the demo. The demo was never part of the
published package, and dropping it also removes the only place that
logged part of an SSH session token.
Signed-off-by: Max Dubrinsky <mdubrinsky@nvidia.com>
* fix(sdk-ts)!: harden exec streaming, waits, SSH, and forwarding
Address review feedback on the sandbox surface.
- Make the streamed command exit code observable from idiomatic
for-await: the terminal exit is now an in-band ExecStreamEvent
({ type: 'exit', exitCode }) rather than the async generator return
value, which for-await discards. A stream that ends without an exit
event now throws instead of reporting success.
- Bound waitReady, waitDeleted, and the setPolicy wait by their timeout:
each poll RPC carries a per-iteration deadline and the waits accept an
AbortSignal, so a stalled call can no longer leave a wait pending
forever. Add waitTimeoutSecs to SetPolicyOptions.
- Validate the CreateSshSession response against the proto charset and
range contract before returning it or using its token, since the
values feed an OpenSSH ProxyCommand.
- Respect socket backpressure when relaying forwarded responses: pause
reading the gRPC stream when the local socket buffer is full and
resume on drain so memory stays bounded.
- Expose create-time sandbox policy: add policy and an advanced rawSpec
passthrough to SandboxSpec so the safety boundary is expressible at
creation and new spec fields do not require an SDK change.
BREAKING CHANGE: execStream and the interactive exec output now yield a
terminal { type: 'exit', exitCode } event; consumers iterating the
stream must handle that arm. The exit code is no longer the async
generator return value.
Signed-off-by: Max Dubrinsky <mdubrinsky@nvidia.com>
* feat(sdk-ts): export error contract, enum unions, and caller cancellation
Address Tier-1 review feedback on the TypeScript SDK public surface (PR #2122).
- Errors: export SdkError and SdkErrorCode so callers can use instanceof and
exhaustively switch on .code. fromConnect preserves the originating
ConnectError as .cause and its status as .connectCode, maps Aborted to a new
'aborted' code for optimistic-concurrency conflicts, and maps Canceled and
DeadlineExceeded to 'canceled'. errorCode() behavior is unchanged.
- Enums: replace the string-typed phase, status, scope, and policySource fields
with lowercase literal unions (SandboxPhaseName, HealthStatus,
SettingScopeName, PolicySourceName) backed by exhaustive Record maps. The
unions are a hand-maintained mirror of the generated proto enums; a new drift
test pins each literal to its generated member name.
- Cancellation: accept an optional AbortSignal on exec, execInteractive, and
forward, threaded into both sandbox resolution and the streaming RPC. forward
tears down its local listener on abort.
Signed-off-by: Max Dubrinsky <mdubrinsky@nvidia.com>
* feat(sdk-ts): add raw escape hatch for uncurated gateway RPCs
The curated sub-clients reduce proto messages to ergonomic subsets (for
example get() drops created_at_ms, the full spec, conditions, runtime
endpoints, and current_policy_version), and not every gateway RPC has a
typed helper yet. Rather than ship methods that exist but throw, expose a
generated client for the full surface.
OpenShellClient.raw and SandboxClient.raw are generated clients covering
every gateway RPC, returning the verbatim wire messages so proto
distinctions the curated types smooth over are preserved. .transport
exposes the shared connection for building extra clients over one socket.
Generated request/response types are published at the new
@nvidia/openshell-sdk/raw subpath. Curated methods stay the default;
raw is the always-available floor.
Signed-off-by: Max Dubrinsky <mdubrinsky@nvidia.com>
* fix(sdk-ts): address review feedback on exec, forward, and auth transport
Settle exec's `done` promise before yielding the exit event so a consumer
that breaks on exit no longer leaves it pending forever, and give it a lone
rejection handler plus a finally-settle so a stream error or early abandon
can never surface as an unhandled rejection or a hang.
Attach an 'error' listener to each accepted forward socket synchronously,
before forwardConnection awaits CreateSshSession; a peer reset in that
window previously emitted an unhandled 'error' and crashed the process.
Reject ambiguous or unsafe transport configs at buildTransport: oidcToken
and edgeToken together (silently OIDC-only), and any auth token sent over
plaintext http:// to a non-loopback host unless allowInsecureAuth is set.
Wrap versionPin so a non-u64 expectedResourceVersion raises
SdkError('invalid_config') instead of a raw BigInt SyntaxError, and raise
the Node engine floor to >=20.3 for AbortSignal.any().
Signed-off-by: Max Dubrinsky <mdubrinsky@nvidia.com>
* fix(sdk-ts): address review feedback
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* docs(sdk-ts): defer published sdk guide
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
---------
Signed-off-by: Max Dubrinsky <mdubrinsky@nvidia.com>
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
Co-authored-by: Drew Newberry <anewberry@nvidia.com>
* feat(gateway): add TOML configuration file (RFC 0003)
Introduces an opt-in --config / OPENSHELL_GATEWAY_CONFIG flag that loads a
TOML file with gateway-wide settings and per-driver tables. Source
precedence is CLI > env > file > built-in default, implemented via clap's
ValueSource so existing flags and env vars keep their priority.
Driver crates (kubernetes, docker, podman, vm) now derive Deserialize on
their config structs. SupervisorSideloadMethod gains Deserialize with
kebab-case rename. A per-driver inheritance allowlist on the loader side
overlays [openshell.gateway] shared defaults (default_image,
supervisor_image, image_pull_policy, guest_tls_*, ssh_handshake_skew_secs,
client_tls_secret_name, host_gateway_ip, enable_user_namespaces) onto
each [openshell.drivers.<name>] table before deserialization.
The Helm chart renders a new gateway-config ConfigMap and mounts it at
/etc/openshell/gateway.toml. The migrated OPENSHELL_* env entries are
dropped from the StatefulSet — only the Secret-backed
OPENSHELL_SSH_HANDSHAKE_SECRET remains. database_url stays on --db-url.
Adds examples/gateway/gateway.example.toml and updates architecture/gateway.md
with the source precedence and inheritance rules.
* docs(gateway): drop ssh_handshake_skew_secs and ssh_handshake_secret from examples
Both fields are scheduled for removal. Remove the example values and the
env-only note so the gateway.toml example and the architecture doc stop
recommending settings that will not exist much longer.
* docs(rfc): correct OPENSHELL_CONFIG to OPENSHELL_GATEWAY_CONFIG in RFC 0003
* docs(gateway): add per-driver TOML example configurations
Adds focused single-driver examples next to the comprehensive
gateway.example.toml: kubernetes, docker, podman, and microvm. Each one
demonstrates the realistic settings for that driver plus how shared
[openshell.gateway] defaults inherit into the driver table.
A new unit test (`checked_in_examples_parse`) loads every example through
the config_file loader so schema drift fails CI rather than silently
shipping a broken example.
* refactor(gateway): drop image_pull_policy from shared inheritance
Kubernetes and Podman use mutually-incompatible vocabularies for the same
TOML key:
- Kubernetes: `Always | IfNotPresent | Never` (free-form string passed
verbatim to the K8s API).
- Podman: `always | missing | never | newer` (strict lowercase enum
deserialised into `ImagePullPolicy`).
No value means the same thing in both drivers. Sharing the key at
`[openshell.gateway]` scope and inheriting it into every active driver's
table meant any value safe for one driver was either wrong or silently
dropped for the other (`IfNotPresent` → `ImagePullPolicy::Missing` after
`.unwrap_or_default()`). Operators run one driver per gateway, so the
"shared default" never pays for itself.
Make `image_pull_policy` driver-local:
- Remove the field from `GatewayFileSection` and from
`inheritable_keys()` for both Kubernetes and Podman.
- Drop the file→`RunArgs` merge for the gateway-scope key.
- Stop unconditionally clobbering the driver value with
`config.sandbox_image_pull_policy` in the runtime wiring — only apply
the CLI/env override when it was set (and, for Podman, only when it
parses into the lowercase enum).
- Move the key under `[openshell.drivers.kubernetes]` and
`[openshell.drivers.podman]` in every example, the RFC, the
architecture doc, and the Helm-rendered gateway ConfigMap.
The supervisor pull policy follows the same shape: it is K8s-only and
moves into `[openshell.drivers.kubernetes]` alongside `image_pull_policy`
in the Helm template.
* fix(gateway): address review feedback on TOML configuration
Resolves the P1 and P2 issues raised in PR #1317:
- Helm gateway ConfigMap moves `grpc_endpoint` under
`[openshell.drivers.kubernetes]` so the default install no longer fails
the gateway's `deny_unknown_fields` schema check.
- `kubernetes_config_from_file` and `podman_config_from_file` only let
the gateway-wide CLI/env `grpc_endpoint` overwrite the driver-table
value when it was actually supplied, preserving file-only configs.
- Kubernetes driver default `image_pull_policy` is now empty (was Podman
vocabulary "missing"), so default deployments let the Kubernetes API
apply its own policy instead of being rejected.
- New `disable_tls` gateway field plumbs `.Values.server.disableTls`
through the TOML ConfigMap instead of relying on env vars dropped from
the StatefulSet.
- StatefulSet pod template now carries a `checksum/gateway-config`
annotation so `helm upgrade` rolls pods when the ConfigMap changes.
- Auxiliary listener resolution preserves the full `SocketAddr` from
`health_bind_address` / `metrics_bind_address`, so a loopback-pinned
health port is not silently relocated onto the public bind address.
- `ssh_session_ttl_secs` from the file is now applied to `Config` (it
was previously accepted by the loader but never read).
New regression coverage: cli-level merge tests for the new fields plus
helm-unittest assertions for the ConfigMap shape, checksum annotation,
and `disable_tls` rendering.
* docs(gateway): consolidate gateway TOML examples into docs reference
Replaces the per-driver example files under examples/gateway/ with a
single published reference page at docs/reference/gateway-config.mdx
covering source precedence, layout, the full example, and the four
per-driver examples (Kubernetes, Docker, Podman, microVM). Drew flagged
during PR #1317 review that the examples belong with the user-facing
docs rather than in a sibling examples/ directory.
The cross-references in architecture/gateway.md and RFC 0003 are updated
to point at the new docs page; the round-trip test in config_file.rs is
removed (schema coverage stays on the inline parses_full_example test
and per-field merge tests — doc-snippet drift belongs in a separate
docs-lint, not in a cross-tree Rust unit test).
* refactor(core): move DEFAULT_K8S_NAMESPACE into K8s driver
The constant is Kubernetes-specific (used only by KubernetesComputeConfig's
Default impl) and does not belong in openshell-core. Relocate it to the
driver crate that owns the K8s vocabulary; openshell-core retains only
truly cross-cutting defaults.
* refactor(core): move Podman bridge default into Podman driver
DEFAULT_NETWORK_NAME is Podman vocabulary, consumed only by the Podman
driver. Also drops the unused DEFAULT_IMAGE_PULL_POLICY constant.
* docs(auth): scrub remaining SSH handshake secret references
Sweeps the trailing mentions left after the rebase: the gateway
config-file module doc, the Helm gateway-config ConfigMap header,
the gateway-config.mdx env-only note, and the RPM systemd unit
comment for init-gateway-env.sh.
* docs(gateway): clarify OPENSHELL_GRPC_ENDPOINT applies to all drivers
The previous comment implied the callback endpoint was Kubernetes-only,
but the value is propagated to every compute driver (Kubernetes, Docker,
Podman, VM) and must be reachable from wherever the sandbox runs.
* refactor(gateway): move driver options into config (#1394)
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* fix(e2e): regenerate gateway config via TOML for docker + podman harnesses
The gateway CLI flags moved into TOML config tables in 560550d2 (#1394),
which made every existing e2e/with-{docker,podman}-gateway.sh invocation
fail with "unexpected argument '--sandbox-namespace'" (and a long tail
of similar driver-specific options) before the gateway could even bind.
Replace the obsolete CLI flags with a synthesized
`[openshell.drivers.<driver>]` table written to `${STATE_DIR}/gateway.toml`
and passed via the new `--config` flag. Only the gateway-wide flags
that survived 560550d2 (bind-address, port, drivers, db-url, tls-*,
disable-tls, log-level, health-port) stay on the command line.
Both scripts get a small `toml_string` helper to properly TOML-quote
the values (the previous `%q` printf format produced bash-escape, not
TOML-escape). The Docker harness also corrects two field names that
diverged from the driver schema: `docker_network_name` →
`network_name`, and the supervisor binary/image plumbing now reads
through to `supervisor_bin` / `supervisor_image` in the same table.
The Podman harness drops `--ssh-gateway-port` (deleted in 560550d2 —
gRPC + SSH are multiplexed on the same port now) and substitutes
`network_name` + `gateway_port` for the obsolete `--sandbox-namespace`
(which the Podman driver never had as a typed field).
* fix(core): swap bind-only 0.0.0.0 SSH gateway host for cluster URL host
CLI's resolve_ssh_gateway treated 0.0.0.0 as a loopback "keep as-is"
when the cluster URL was also loopback, so the SSH proxy connected to
0.0.0.0:port. The unspecified address is never a valid connect target
and is not present in any TLS cert SAN, which produced BadCertificate
TLS handshake failures during `openshell sandbox create -- ...` in
docker/podman e2e (e.g. bypass_detection).
Resolution: when the server returns 0.0.0.0 or :: as the gateway host
and both endpoints are loopback, fall back to the cluster URL's host
(which the CLI is already using to reach the gateway, so it must
resolve and match the cert).
* fix(e2e): repair podman harness on macOS
Podman 5.x with the applehv/libkrun provider no longer creates the legacy
~/.local/share/containers/podman/machine/podman.sock symlink, and
`podman system service` is a Linux-only subcommand — the macOS client
delegates the API service to the VM. Both assumptions in the harness
were stale, so the script tried to start a temporary service that podman
rejected with "unknown flag: --time".
- Discover the macOS socket via `podman machine inspect` instead of the
hardcoded path.
- On Darwin, fail fast with a "start podman machine" message rather than
attempting the Linux-only `podman system service` fallback.
- Write socket_path into [openshell.drivers.podman] so the in-process
driver picks up the discovered socket; the driver reads TOML only
after the config refactor (560550d2), so OPENSHELL_PODMAN_SOCKET alone
was no longer enough.
* fix(server): use clone_from for TLS client CA assignment
clippy 1.95.0 rejects assigning the result of `Clone::clone()` to an
existing variable under `-D warnings` (`assigning_clones`). Switch to
`clone_from(&...)` to satisfy the lint and avoid the redundant
allocation.
---------
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
Co-authored-by: Drew Newberry <anewberry@nvidia.com>
Oliver Calder