* feat(server): add sandbox workload templates
Signed-off-by: Gordon Sim <gsim@redhat.com>
* feat(go-sdk): add sandbox workload template support
Signed-off-by: Gordon Sim <gsim@redhat.com>
* feat(rust-sdk): add sandbox workload template support
Signed-off-by: Gordon Sim <gsim@redhat.com>
* feat(python-sdk): add sandbox workload template support
Signed-off-by: Gordon Sim <gsim@redhat.com>
* feat(typescript-sdk): add sandbox workload template support
Signed-off-by: Gordon Sim <gsim@redhat.com>
* docs(agents): document sandbox workload templates
Signed-off-by: Gordon Sim <gsim@redhat.com>
* fix(cli): support default GPU requests in sandbox templates
Signed-off-by: Gordon Sim <gsim@redhat.com>
* fix(server): cap sandbox templates per workspace
Signed-off-by: Gordon Sim <gsim@redhat.com>
* docs(architecture): document sandbox workload template boundaries
Signed-off-by: Gordon Sim <gsim@redhat.com>
* feat(cli+sdk): expose sandbox workload template provenance
Signed-off-by: Gordon Sim <gsim@redhat.com>
* fix(cli): include sandbox template annotations in output
Signed-off-by: Gordon Sim <gsim@redhat.com>
* feat(sandbox): add label selectors to template listing
Signed-off-by: Gordon Sim <gsim@redhat.com>
* test(e2e): cover sandbox template failure paths
Signed-off-by: Gordon Sim <gsim@redhat.com>
* fix(server): preserve command and ttl when creating sandbox from template
Signed-off-by: Gordon Sim <gsim@redhat.com>
* test(server): add field coverage test for template merge
Signed-off-by: Gordon Sim <gsim@redhat.com>
* fix(go-sdk): add pagination support to fake client
Signed-off-by: Gordon Sim <gsim@redhat.com>
* fix(cli): warn on env vars that looks like secrets
Signed-off-by: Gordon Sim <gsim@redhat.com>
* fix(sdk-ts): propagate sandbox workspace through lifecycle calls
Signed-off-by: Gordon Sim <gsim@redhat.com>
* fix(docs): update workspace management docs
Signed-off-by: Gordon Sim <gsim@redhat.com>
* fix(ts-sdk): support command and tty when creating from template
Signed-off-by: Gordon Sim <gsim@redhat.com>
* fix(go-sdk): support command and tty when creating from template
Signed-off-by: Gordon Sim <gsim@redhat.com>
* test(python-sdk): verify command and tty handling when creating from template
Signed-off-by: Gordon Sim <gsim@redhat.com>
* fix(go-sdk): update docs and ClientInterface
Signed-off-by: Gordon Sim <gsim@redhat.com>
* fix(server): validate sandbox create specs before I/O
Signed-off-by: Gordon Sim <gsim@redhat.com>
* fix(go-sdk): guard empty DNS-1123 label validation
Signed-off-by: Gordon Sim <gsim@redhat.com>
* fix(python-sdk): allow empty template builder mappings
Signed-off-by: Gordon Sim <gsim@redhat.com>
* fix(cli): align template GPU JSON default output
Signed-off-by: Gordon Sim <gsim@redhat.com>
---------
Signed-off-by: Gordon Sim <gsim@redhat.com>
Closes#3112
Reuse the dashboard workspace cycle action from the providers pane, advertise the shortcut, and cover the state transition and rendered hint.
Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>
* feat(helm): split gateway and workspace charts
Signed-off-by: Dhiraj Bokde <dbokde@nvidia.com>
* fix(helm): preserve split chart upgrade compatibility
Keep workspace manifests valid after value validation and default legacy reused values to the combined resource topology.
* fix(ci): preserve VM runtime for E2E
The Rust cache restores target/ after VM runtime artifacts are staged,
overwriting target/vm-runtime-compressed before openshell-driver-vm is built.
Stage the compressed runtime outside target and pass that location through
OPENSHELL_VM_RUNTIME_COMPRESSED_DIR so build.rs can embed the supervisor.
Also locate the Helm split-ownership test repository root from the script
path rather than git rev-parse. The test runs in a container where the
GitHub checkout can be owned by a different UID and rejected as dubious
ownership.
Signed-off-by: Dhiraj Bokde <dbokde@nvidia.com>
* fix(ci): install yq for Helm ownership test
The split-chart ownership regression uses yq to inspect rendered YAML,
but the Helm CI container installs only tools declared in mise.
Declare and lock yq so mise install --locked provides the test dependency.
Signed-off-by: Dhiraj Bokde <dbokde@nvidia.com>
---------
Signed-off-by: Dhiraj Bokde <dbokde@nvidia.com>
* fix(cli): suppress ANSI color when stdout is not a terminal
The CLI colorized output unconditionally. owo-colors is built without
its `supports-colors` feature, so `.green()` and friends emitted escape
sequences regardless of destination, and nothing in the CLI read
NO_COLOR. Piping any command through grep or awk matched against bytes
the caller could not see; `forward list` was the case that surfaced it,
where an escape sits immediately before the STATUS word and defeats a
pattern anchored on whitespace.
Add a `color` module holding a process-wide switch resolved once in
run_async, before any output. Command modules import its `Colorize`
trait in place of `OwoColorize`; the method names match, so the ~450
call sites are unchanged, but each consults the switch when it renders
and delegates to owo-colors so the escape bytes stay identical. The two
traits collide by design: importing both in one module is an ambiguity
error, which keeps unconditional coloring from returning.
owo-colors is not the only styled path, and the rest each carry their
own default, so the switch governs them too:
- tracing_subscriber formats with ANSI on, does no terminal detection,
and writes to stdout, so `openshell -v ... | ...` leaked escapes the
same way the tables did. It now takes the setting via with_ansi.
- indicatif and dialoguer both style through console, which has its
own detection but cannot learn about --color. Overriding console's
global switch covers every progress bar and prompt rather than the
specific ones constructed today. Both the stdout and stderr switches
are set, since prompts and progress bars draw to stderr.
- miette renders errors through its own handler, likewise unaware of
--color, so init installs one built from the setting.
Resolution order: `--color always|never`, then NO_COLOR, then
CLICOLOR_FORCE, then whether stdout is a terminal. The decision is made
against stdout even for stderr text, since stdout is what gets parsed;
`--color always` restores styling when redirecting.
Padding is unaffected — the format spec is forwarded to the inner
Display, so widths measure text rather than text plus escapes.
Signed-off-by: Mrunal Patel <mrunalp@gmail.com>
* fix(cli): resolve color per output stream
Review feedback on #3026.
Resolving one answer from stdout and handing it to every library meant a
redirected stream inherited the other stream's terminal check. Running
`openshell ... 2> build.log` from a terminal wrote escapes into the log,
because console's stderr switch and miette's handler were both given
stdout's answer. That is worse than the behavior before this branch,
where both libraries did their own per-stream detection.
Resolve `auto` separately for stdout and stderr and hand each library
the answer for the stream it writes to: tracing and console's stdout
switch get stdout, miette and console's stderr switch get stderr. The
owo-colors wrapper is the exception, since its call sites are split
across println! and eprintln! and a Painted value cannot tell which
macro will consume it; it styles only when both streams accept escapes,
erring toward plain text rather than risking a redirected stream.
Existing tests could not catch this: Command::output gives both streams
pipes, so a per-stream decision and a single stdout-derived one look
identical. Add a test that puts stdout on a pty and stderr on a pipe,
which fails when stderr is handed stdout's answer.
Replace CLICOLOR_FORCE with FORCE_COLOR. The clicolors spec does not say
how to treat `0`, and implementations that special-case it disagree with
force-color.org, which keys on presence and non-emptiness only. Using
FORCE_COLOR gives it the same rule as NO_COLOR: set and non-empty means
yes, whatever the value. Nothing depended on CLICOLOR_FORCE, which was
introduced earlier on this branch and never released.
Carry the whole style in an owo_colors::Style rather than dispatching a
local enum through a six-arm match, and merge styles when chaining so
`x.green().bold()` emits one `\x1b[32;1m...\x1b[0m` instead of nesting
two wrappers. No call site styles already-styled text, so merging is
safe; the emitted bytes are shorter and there is a single reset.
Signed-off-by: Mrunal Patel <mrunalp@gmail.com>
---------
Signed-off-by: Mrunal Patel <mrunalp@gmail.com>
* feat(build): add defaults-without-telemetry feature alias
Cargo cannot subtract a single default feature, so compiling telemetry out
meant `--no-default-features` plus a hand-maintained keep-list of the crate's
other defaults. That keep-list was already wrong for operators: telemetry is
the only default on openshell-server and openshell-driver-vm, but
openshell-sandbox also defaults to `bundled-ca-roots`, so a bare
`--no-default-features` silently swapped the supervisor onto the platform
trust store.
Add a `defaults-without-telemetry` alias to each of the three telemetry-
carrying binary crates, enumerating every default except `telemetry`.
Telemetry-free builds become `--no-default-features --features
defaults-without-telemetry` and stay correct as the default set grows.
The alias is a keep-list, not a switch. Enabling it on top of the defaults
would otherwise produce a telemetry-on binary that reads as telemetry-free, so
each crate root carries a `compile_error!` for the `telemetry` +
`defaults-without-telemetry` combination.
Add `rust:verify:defaults-without-telemetry` to guard both properties: each
alias still equals its crate's defaults minus `telemetry`, and the
mutual-exclusion error is wired up. The additive-misuse check matches on the
`compile_error!` text rather than a nonzero exit code so it cannot pass
vacuously on hosts where openshell-driver-vm fails to build for unrelated
reasons. `rust:verify:telemetry-off` now builds through the alias.
Signed-off-by: Russell Bryant <rbryant@redhat.com>
* fix feature alias for openshell-server
Signed-off-by: Russell Bryant <rbryant@redhat.com>
* fix(ci): run Rust verification in Nix shell
Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>
---------
Signed-off-by: Russell Bryant <rbryant@redhat.com>
Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>
Co-authored-by: John Myers <9696606+johntmyers@users.noreply.github.com>
* fix(server): release driver-owned sandbox resources on out-of-band removal
Neither the watch path nor the periodic prune sweep call the driver's
DeleteSandbox RPC when a sandbox's compute resource is discovered gone
out-of-band (container removed without going through OpenShell's own
delete path, or the gateway crashing mid-create). Only store-side
records were cleaned up, leaving driver-owned secrets and volumes
behind indefinitely — a credential leak for the Podman driver's
per-sandbox bearer-token and proxy-auth secrets.
DeleteSandbox is already idempotent and already reclaims these
resources when the underlying container is already gone, so no new
RPC is needed: this wires the existing call into the two paths that
discover a sandbox is gone without going through an explicit delete
request.
The watch path defers the actual call to a background task, gated by
a non-blocking check of the sandbox's lifecycle gate: watch events are
processed sequentially and must never block on a driver call, and
skipping the call entirely when a concurrent request-side operation
already holds the gate avoids racing that operation's own driver call.
The prune sweep calls it inline, since it already makes a blocking
GetSandbox call per sandbox as part of its normal operation.
Closes#2352
Signed-off-by: politerealism <burdcat17@gmail.com>
* fix(server): stop holding sync_lock across the prune sweep's driver call
The prune sweep's second sync_lock acquisition has no inner block
scope, so it's held through the rest of prune_missing_sandbox,
including the driver cleanup call added in the previous commit. That
call is a network RPC to the compute driver; holding the gateway-wide
sync_lock across it blocks every other sandbox operation on the
gateway for as long as that one driver call takes, not just the
sandbox being pruned.
Use the same backgrounded spawn_driver_sandbox_cleanup already used by
the watch path instead of the inline cleanup_driver_sandbox_resources,
which is now unused and removed. Add a regression test that blocks the
driver's delete call indefinitely and confirms the prune sweep still
completes and removes the store record promptly.
Addresses gator finding GATOR-9902b7bb-01 on #3042.
Signed-off-by: politerealism <burdcat17@gmail.com>
---------
Signed-off-by: politerealism <burdcat17@gmail.com>
`sandbox exec` seeded its exit code to 0 and only overwrote it on an
`Exit` event, so a stream that ended early, was cancelled, or was truncated
reported a successful run. The gateway already treats the same condition as
a relay failure (`Status::unavailable`); mirror that on the CLI side so exit
0 always means an observed exit status of 0.
Fixes#2732
Signed-off-by: rootkiller6788 <rootkiller6788@users.noreply.github.com>
Co-authored-by: rootkiller6788 <rootkiller6788@users.noreply.github.com>
Characterization tests for behavior that later work changes. No behavior
change.
- Pin the `openshell logs` output format, so a change to how that text is
produced shows up as a diff rather than passing silently.
- Cover the supervisor log push layer.
- Assert OcsfEvent survives a JSON round trip. Serialization is written
by hand and deserialization dispatches on class_uid into independent
per-variant paths, so a field can serialize correctly and still be
dropped or rejected on the way back in.
Refs #1055
Signed-off-by: Kris Hicks <khicks@nvidia.com>
OCSF_VERSION has been 1.8.0; two descriptions still said 1.7.0.
The default_gateway_id doc comment described a hostname fallback the
implementation never had. A per-replica default would be wrong here:
gateway_id is embedded in the JWT iss/aud, so it has to be stable across
replicas.
Signed-off-by: Kris Hicks <khicks@nvidia.com>
parse_duration_to_ms multiplied the parsed number by its unit
multiplier without a range check. A large --since value on
`openshell logs` overflows i64: a debug build panics with "attempt to
multiply with overflow", and a release build wraps to a nonsense log
cutoff.
Use checked_mul and return a diagnostic naming the offending input.
Every in-range result is unchanged, and the largest accepted duration
still leaves the caller's `now_ms - dur_ms` well inside i64.
Signed-off-by: Nathan DeMoss <ndemoss28@gmail.com>
* fix(compute): recover Error-phase sandboxes on gateway startup
Sandboxes whose container exited on its own (e.g. SIGTERM from a Podman
or Docker machine restart) were left stuck in the Error phase even though
the container could be restarted. The startup sweep skipped every
Error-phase sandbox unconditionally.
Include Error-phase sandboxes in the startup sweep when their Ready
condition reason marks a container exit or stop. If the driver restarts
the container, move the sandbox back to Provisioning with a Resumed
condition; if the container is gone or the start fails, leave the Error
state untouched. Genuine (non-container-exit) errors are still skipped.
Container-exit restart is handled by the drivers' existing idempotent
start_sandbox path, which already restarts stopped/exited containers.
The ContainerExited/ContainerStopped condition reasons are promoted to
shared constants in openshell-core so the gateway and drivers agree on
the recovery signal.
Fixes#2179
Signed-off-by: Ian Miller <milleryan2003@gmail.com>
* fix(compute): keep ordinary container exits terminal during startup recovery
Startup recovery treated every non-OOM container exit as recoverable,
lumping application crashes and non-zero exits together with
machine/daemon-restart signal kills under the shared `ContainerExited`
Ready-condition reason. On the next gateway startup it restarted those
crashed workloads and replaced the terminal error with `Resumed`,
erasing the failure signal and repeatedly reviving crash-prone
sandboxes.
Introduce a distinct `ContainerRuntimeRestart` reason for containers
terminated by an external signal (exit 137/143 = SIGKILL/SIGTERM), which
is the signature of a machine/daemon restart. The Podman inspect-based
classification emits it; the startup recovery gate now recovers only
`ContainerRuntimeRestart` and `ContainerStopped`. Ordinary
`ContainerExited` records stay terminal, so genuine failures keep their
error signal. Because only the new reason is recoverable and older
gateways never persisted it, exits recorded before this change also stay
terminal after an upgrade.
Add regression coverage: a generic `ContainerExited` error is not
relaunched, a signal-kill classifies as `ContainerRuntimeRestart`, and
the startup sweep recovers only the runtime-restart record.
Signed-off-by: Ian Miller <milleryan2003@gmail.com>
* fix(docker): reclassify signal-killed containers as runtime restart
Mirror the Podman classification in the Docker driver so that
externally signal-killed containers (exit 137/143, non-OOM) are
persisted with the ContainerRuntimeRestart ready reason and become
eligible for startup recovery, while ordinary exits and OOM kills stay
terminal.
current_snapshots now inspects EXITED containers to obtain the exit
code and OOM flag, then applies the shared exit classification. Without
the inspect step the list-summary path lacks an exit code, so the
recovery gate could never fire for Docker.
Adds unit tests covering signal-kill reclassification (137/143),
ordinary exit staying terminal (exit 1), and OOM staying terminal
despite exit 137.
Signed-off-by: Ian Miller <milleryan2003@gmail.com>
---------
Signed-off-by: Ian Miller <milleryan2003@gmail.com>
Closes#3038
Compress platform runtime inputs in Nix and stage the complete bundle outside
Cargo's target directory. Reject missing or empty embedding artifacts during
the driver build.
Signed-off-by: Simon Scatton <sscatton@nvidia.com>
This changes the DNS resolver test fixture to retry until TCP and UDP can share
an ephemeral port.
It fixes an issue seen in CI where a port that was successfully assigned for
one protocol failed for the other.
Signed-off-by: Kris Hicks <khicks@nvidia.com>
Ctrl-C closes gateway listeners while supervisor sessions are still current,
causing benign HTTP/2 broken pipes to be logged as warnings. Track gateway
shutdown so recognized transport closes are treated as expected.
Signed-off-by: Kris Hicks <khicks@nvidia.com>
* fix(helm): refresh kubeconfig for existing k3d clusters
Docker can recreate the k3d load balancer on a new API port. Start existing
clusters and prefer fresh k3d entries so create does not retain a stale
endpoint.
Signed-off-by: Kris Hicks <khicks@nvidia.com>
* fix(dev): conditionally enable local OTLP export
Probe port 4317 before adding OTLP configuration for the VM, Docker,
and Podman gateway tasks. Document the startup behavior and troubleshooting
for local collector availability.
Signed-off-by: Kris Hicks <khicks@nvidia.com>
---------
Signed-off-by: Kris Hicks <khicks@nvidia.com>
* feat(gateway): add installation name configuration
Add a first-class operator-assigned gateway name with TOML, CLI, environment,
and Helm configuration surfaces. Local gateways default to openshell, while
Helm defaults to the chart fullname; operators sharing a collector across
namespaces or clusters can set a globally distinct name.
Signed-off-by: Kris Hicks <khicks@nvidia.com>
* feat(gateway): identify gateways in exported traces
Attach the configured gateway installation name and compute driver to the
gateway OpenTelemetry resource so operators can filter traces from multiple
installations that share a collector. Forward the gateway name and OTLP
endpoint to managed external drivers so their distinct service resources carry
the same installation identity.
Keep service.name stable per process type, omit blank resource values, and
leave per-span operation names and request attributes unchanged.
Refs #2507
Signed-off-by: Kris Hicks <khicks@nvidia.com>
---------
Signed-off-by: Kris Hicks <khicks@nvidia.com>
Simon Scatton