* refactor(inference): remove managed inference routes
Closes#3172
Remove the inference route control plane, inference.local data path, built-in router crate, and SDK surface. Move inference workloads to explicitly imported provider profiles and native endpoints, with migration cleanup and updated tests and documentation.
Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>
* fix(policy): preserve alternate upstream isolation
Restore the provider policy activation guard so legacy OpenAI and Anthropic providers configured for alternate base URLs do not grant egress to the built-in public vendor endpoints.
Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>
---------
Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>
* chore(deps): replace ring with AWS-LC
Signed-off-by: Simon Scatton <sscatton@nvidia.com>
* fix(lint): address warnings after dependency upgrades
Signed-off-by: Simon Scatton <sscatton@nvidia.com>
* fix(tls): limit provider initialization to reqwest clients
Signed-off-by: Simon Scatton <sscatton@nvidia.com>
---------
Signed-off-by: Simon Scatton <sscatton@nvidia.com>
* feat(sandbox): support rootfs tar as --from source for VM driver
Accept flat rootfs tar archives (.tar, .tar.gz, .tgz) via the --from
flag for VM-backed gateways. The CLI detects the archive extension,
validates that the gateway uses the VM compute driver, and passes the
tar path through driver_config. The VM driver copies the tar into its
staging area and feeds it into the existing rootfs extraction and ext4
disk creation pipeline, skipping the container image pull/export steps.
Closes#2175
Signed-off-by: Philippe Martin <phmartin@redhat.com>
* fix(sandbox): validate rootfs tar path at the VM driver boundary
The rootfs_tar_path field in driver_config was passed from the API
caller directly to tokio::fs::copy without validation. An authenticated
user bypassing the CLI could supply arbitrary host paths (e.g.
/dev/zero for disk exhaustion, or readable host files for data
exfiltration).
Introduce a trusted staging directory that the VM driver creates on
startup and advertises via GetCapabilities. The CLI now copies the tar
into the staging directory before creating the sandbox, and the driver
validates that the received path is a regular file inside the staging
root and within a configurable size limit (default 10 GiB) before any
I/O.
New VmDriverConfig options:
- rootfs_tar_staging_dir: override the staging directory
(default: <state_dir>/rootfs-tar-staging)
- rootfs_tar_max_bytes: override the size limit (default: 10 GiB)
Addresses GATOR-28b5152e-01.
Signed-off-by: Philippe Martin <phmartin@redhat.com>
* fix(sandbox): request-scoped staging, size pre-check, and cleanup for rootfs tar
Tighten the rootfs tar staging flow to address the remaining GATOR-01
obligations:
- Request-scoped staging: the CLI creates a unique per-request
subdirectory (req-<pid>) under the staging root instead of placing
files directly in the shared directory. The driver enforces that the
tar path is at depth 2 (staging_root/<subdir>/<file>), preventing
cross-request path selection.
- Size pre-check: the driver advertises rootfs_tar_max_bytes via
GetCapabilities. The CLI reads this limit and rejects oversized files
before copying, avoiding disk exhaustion in the staging directory.
- Cleanup: the driver removes the request staging subdirectory after
consuming the tar (on cache hit, copy success, or copy failure),
ensuring staged data does not persist beyond the request.
Signed-off-by: Philippe Martin <phmartin@redhat.com>
* fix(vm): restore rootfs-tar sandboxes from persisted image identity
On restore or restart, the one-shot staged tar archive has already been
cleaned up. Reading the persisted image identity from the sandbox state
directory and resolving the cached disk path directly avoids re-accessing
the deleted staging path.
Addresses GATOR-168b9210-01.
Signed-off-by: Philippe Martin <phmartin@redhat.com>
* fix(cli): use random staging dirs and enforce byte limit during rootfs tar copy
Replace PID-based request staging directories with tempfile-generated
random names to prevent collisions and make paths unpredictable.
Replace bare tokio::fs::copy with a streaming copy loop that enforces
the advertised max_bytes limit during transfer, closing the TOCTOU gap
between the pre-copy size check and the actual copy.
Signed-off-by: Philippe Martin <phmartin@nvidia.com>
Signed-off-by: Philippe Martin <phmartin@redhat.com>
* fix(sandbox): issue rootfs tar staging slots from the gateway
A caller could name any host path in `driver_config.vm.rootfs_tar_path`,
which the privileged VM driver then read. The CLI-side locality check did
not apply to direct API requests.
The gateway now owns staging. `BeginRootfsTarStaging` allocates a
request-scoped directory under the driver-advertised staging root and
returns an opaque single-use token; `CreateSandbox` carries the token, and
the gateway substitutes the path it allocated before dispatching to the
driver. `template.driver_config.<driver>.rootfs_tar_path` is rejected
outright in request validation, so a caller-supplied path never reaches
privileged I/O.
Tokens are bound to the issuing workspace and subject, consumed once, and
expire after 30 minutes. Outstanding slots are capped per caller and
overall, so one caller can neither exhaust the staging filesystem nor
starve others. An RAII guard reclaims the directory on every failure path
after consumption, and an age-gated sweep runs at startup and on each
reconcile pass for directories whose driver died before its own cleanup.
The token is stripped from the public sandbox before persistence: the
stored copy is returned verbatim by GetSandbox, ListSandboxes and
WatchSandbox to every member of the workspace.
Also fixes two defects this exposed:
- The CLI wrote `rootfs_tar_path` at the top level of `driver_config`, but
the gateway forwards only `driver_config.<driver_name>`, silently
dropping unmatched keys. The archive never reached the VM driver, so the
documented `--from ./rootfs.tar` flow did not work at all. Config is now
nested under `vm` and deep-merged, so a caller's existing VM settings
survive instead of being clobbered by a shallow extend.
- Staging previously required `GetGatewayInfo`, which is restricted to
`platform_admin`, making the feature unusable for ordinary users on any
RBAC-enabled gateway. The new RPC matches CreateSandbox at
`sandbox:write` / `workspace_role: user`.
`compute_driver.proto` is unchanged; the gateway reads the staging root
from the capabilities it already stores.
Refs #2175
Signed-off-by: Philippe Martin <phmartin@redhat.com>
* fix(vm): derive rootfs tar cache identity from archive contents
The prepared-disk cache key combined the archive's full path with an mtime
truncated to seconds, then mapped punctuation to `-`. Distinct paths such as
`/tmp/a/b.tar` and `/tmp/a-b.tar` collapsed onto the same key and reused each
other's disk, a rewrite within the same second kept stale contents, and a long
path could exceed filesystem component limits.
Identity is now a SHA-256 of the archive contents. This is also what makes the
cache work at all now that the gateway allocates a fresh staging directory per
request: a path-derived key would miss on every create.
The archive is hashed, the cache checked, and only on a miss copied — so a hit
skips writing a multi-gigabyte file. The copy is hashed as it is written and
rejected if the digest differs from the first pass, which closes the window
where the source changes during staging rather than approximating it with a
re-stat.
Refs #2175
Signed-off-by: Philippe Martin <phmartin@redhat.com>
* fix(vm): decompress gzip rootfs tar archives during staging
`--from` accepts `.tar.gz` and `.tgz`, but the driver staged whatever bytes
it was given and the guest image-prep VM extracts the staged file with a
plain `tar -xpf`. Compressed sources therefore depended on the guest tar
auto-detecting gzip, and the prepared disk was sized from the compressed
length, which is far too small for the expanded rootfs.
Staging now detects gzip from the archive's magic bytes -- the driver only
ever sees a gateway-issued staging path, never the caller's file name -- and
writes an uncompressed tar. The digest still covers the source bytes, so the
"archive changed while staging" check is unaffected, and expansion is bounded
by `rootfs_tar_max_bytes` so a compression bomb cannot fill the host disk.
`extract_rootfs_archive_to` sniffs gzip as well, so the host-side extraction
path matches.
Adds unit coverage for gzip staging, bounded expansion, and gzip extraction,
plus an e2e sandbox created from a gzip-compressed export.
Signed-off-by: Philippe Martin <phmartin@redhat.com>
---------
Signed-off-by: Philippe Martin <phmartin@redhat.com>
Signed-off-by: Philippe Martin <phmartin@nvidia.com>
Keep a dedicated SQLite connection alive so pool connection replacement
retains the shared in-memory schema and objects.
Closes#3173
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* docs(gateway-config): fix stale community sandbox image path
Signed-off-by: Yuedong Wu <dwcn22@outlook.com>
* docs(sandbox-image): purge remaining stale image references
Rebasing onto main surfaced four more instances of the same dead
ghcr.io/nvidia/openshell/sandbox path, introduced by commits merged
after this branch was opened: three test fixtures (driver-docker,
openshell-ocsf, compute::mod) and one user-facing default in the
SPIFFE token-exchange Podman demo README. Correct all four to
ghcr.io/nvidia/openshell-community/sandboxes/base, consistent with
the rest of this fix.
Signed-off-by: Yuedong Wu <dwcn22@outlook.com>
---------
Signed-off-by: Yuedong Wu <dwcn22@outlook.com>
* fix(sandbox): detect an available login shell instead of hardcoding /bin/bash
The built-in default sandbox command and the interactive SSH session
hardcoded /bin/bash. Minimal images such as Alpine ship only /bin/sh
(BusyBox ash), so sandbox startup failed with an opaque "No such file or
directory (os error 2)" that never named the missing binary.
Add openshell-core::shell with shell-path constants and a runtime
detect_login_shell() that resolves a shell present in the sandbox image
($SHELL if executable, then bash, then /bin/sh). Use it for:
- the built-in default command (only the default is remapped; explicit
user commands are never rewritten), resolved in the supervisor so it
inspects the sandbox filesystem rather than the gateway's
- the SSH interactive shell
- the SHELL environment variable
Also name the program in the spawn error so a missing shell/binary is
diagnosable instead of a bare ENOENT.
Refs #3146
Signed-off-by: Akram <akram.benaissi@gmail.com>
* fix(sandbox): drop $SHELL preference in shell detection
$SHELL is image/user-controlled and the detected shell is later invoked
with `-lc`, so an executable that is not a compatible shell (e.g.
SHELL=/bin/false) would pass the executable check and then break command
execution even when /bin/sh is available. Resolve only from known shell
paths instead.
Also add a USR_BASH constant for /usr/bin/bash rather than a string
literal in SHELL_CANDIDATES.
Refs #3146
Signed-off-by: Akram <akram.benaissi@gmail.com>
* fix(sandbox): resolve the default login shell in the supervisor (empty command = default)
Addresses review: interactive PTY SSH now uses the detected shell, the shell
tests are portable across the Windows lane, and default-shell provenance is
carried without a new spec field.
An omitted command is left empty end to end and resolved in the supervisor,
which is the only place that sees the sandbox image:
- The CLI forwards the command as-is; the gateway persists an omitted command
as empty (no baked /bin/bash -l) and requests a TTY.
- MainProcessConfig carries the command empty (the transport now allows it);
the supervisor resolves a login shell that exists in the sandbox image (bash
when present, otherwise /bin/sh on minimal images like Alpine) and logs the
resolved shell.
- Interactive PTY SSH (spawn_pty_shell) uses the detected shell; a shared
build_ssh_shell_command helper covers the PTY and non-PTY paths, with a
deterministic sh-only regression test.
- Unix-only shell tests are gated with cfg(unix).
An explicit command is always run verbatim.
Refs #3146
Signed-off-by: Akram <akram.benaissi@gmail.com>
---------
Signed-off-by: Akram <akram.benaissi@gmail.com>
Centralize compute-driver RPC descriptors, stream instrumentation, provider
routing, and standalone installation in openshell-otel. Use typed RPC
constants so gateway and in-process driver paths cannot panic on unknown
operation strings or repeat runtime method parsing.
Emit semantic-convention rpc.service and rpc.method attributes, preserve
trace context and resource identity across deployment modes, and route both
RPC boundary and backend crate spans to each selected driver provider. Leave
consumer-dropped watch spans unset while recording observed terminal status,
and avoid reboxing untraced external-driver streams.
Derive each driver tracing identity from Cargo package and crate metadata and
attach its descriptor to the compute-driver registration, keeping provider
selection and target routing tied to the registered implementation. Share
tracing setup and round-trip test support across Docker, Podman, Kubernetes,
and VM, and update the gateway tracing documentation.
Signed-off-by: Kris Hicks <khicks@nvidia.com>
* feat(server): add sandbox workload templates
Signed-off-by: Gordon Sim <gsim@redhat.com>
* feat(go-sdk): add sandbox workload template support
Signed-off-by: Gordon Sim <gsim@redhat.com>
* feat(rust-sdk): add sandbox workload template support
Signed-off-by: Gordon Sim <gsim@redhat.com>
* feat(python-sdk): add sandbox workload template support
Signed-off-by: Gordon Sim <gsim@redhat.com>
* feat(typescript-sdk): add sandbox workload template support
Signed-off-by: Gordon Sim <gsim@redhat.com>
* docs(agents): document sandbox workload templates
Signed-off-by: Gordon Sim <gsim@redhat.com>
* fix(cli): support default GPU requests in sandbox templates
Signed-off-by: Gordon Sim <gsim@redhat.com>
* fix(server): cap sandbox templates per workspace
Signed-off-by: Gordon Sim <gsim@redhat.com>
* docs(architecture): document sandbox workload template boundaries
Signed-off-by: Gordon Sim <gsim@redhat.com>
* feat(cli+sdk): expose sandbox workload template provenance
Signed-off-by: Gordon Sim <gsim@redhat.com>
* fix(cli): include sandbox template annotations in output
Signed-off-by: Gordon Sim <gsim@redhat.com>
* feat(sandbox): add label selectors to template listing
Signed-off-by: Gordon Sim <gsim@redhat.com>
* test(e2e): cover sandbox template failure paths
Signed-off-by: Gordon Sim <gsim@redhat.com>
* fix(server): preserve command and ttl when creating sandbox from template
Signed-off-by: Gordon Sim <gsim@redhat.com>
* test(server): add field coverage test for template merge
Signed-off-by: Gordon Sim <gsim@redhat.com>
* fix(go-sdk): add pagination support to fake client
Signed-off-by: Gordon Sim <gsim@redhat.com>
* fix(cli): warn on env vars that looks like secrets
Signed-off-by: Gordon Sim <gsim@redhat.com>
* fix(sdk-ts): propagate sandbox workspace through lifecycle calls
Signed-off-by: Gordon Sim <gsim@redhat.com>
* fix(docs): update workspace management docs
Signed-off-by: Gordon Sim <gsim@redhat.com>
* fix(ts-sdk): support command and tty when creating from template
Signed-off-by: Gordon Sim <gsim@redhat.com>
* fix(go-sdk): support command and tty when creating from template
Signed-off-by: Gordon Sim <gsim@redhat.com>
* test(python-sdk): verify command and tty handling when creating from template
Signed-off-by: Gordon Sim <gsim@redhat.com>
* fix(go-sdk): update docs and ClientInterface
Signed-off-by: Gordon Sim <gsim@redhat.com>
* fix(server): validate sandbox create specs before I/O
Signed-off-by: Gordon Sim <gsim@redhat.com>
* fix(go-sdk): guard empty DNS-1123 label validation
Signed-off-by: Gordon Sim <gsim@redhat.com>
* fix(python-sdk): allow empty template builder mappings
Signed-off-by: Gordon Sim <gsim@redhat.com>
* fix(cli): align template GPU JSON default output
Signed-off-by: Gordon Sim <gsim@redhat.com>
---------
Signed-off-by: Gordon Sim <gsim@redhat.com>
* feat(build): add defaults-without-telemetry feature alias
Cargo cannot subtract a single default feature, so compiling telemetry out
meant `--no-default-features` plus a hand-maintained keep-list of the crate's
other defaults. That keep-list was already wrong for operators: telemetry is
the only default on openshell-server and openshell-driver-vm, but
openshell-sandbox also defaults to `bundled-ca-roots`, so a bare
`--no-default-features` silently swapped the supervisor onto the platform
trust store.
Add a `defaults-without-telemetry` alias to each of the three telemetry-
carrying binary crates, enumerating every default except `telemetry`.
Telemetry-free builds become `--no-default-features --features
defaults-without-telemetry` and stay correct as the default set grows.
The alias is a keep-list, not a switch. Enabling it on top of the defaults
would otherwise produce a telemetry-on binary that reads as telemetry-free, so
each crate root carries a `compile_error!` for the `telemetry` +
`defaults-without-telemetry` combination.
Add `rust:verify:defaults-without-telemetry` to guard both properties: each
alias still equals its crate's defaults minus `telemetry`, and the
mutual-exclusion error is wired up. The additive-misuse check matches on the
`compile_error!` text rather than a nonzero exit code so it cannot pass
vacuously on hosts where openshell-driver-vm fails to build for unrelated
reasons. `rust:verify:telemetry-off` now builds through the alias.
Signed-off-by: Russell Bryant <rbryant@redhat.com>
* fix feature alias for openshell-server
Signed-off-by: Russell Bryant <rbryant@redhat.com>
* fix(ci): run Rust verification in Nix shell
Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>
---------
Signed-off-by: Russell Bryant <rbryant@redhat.com>
Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>
Co-authored-by: John Myers <9696606+johntmyers@users.noreply.github.com>
* fix(server): release driver-owned sandbox resources on out-of-band removal
Neither the watch path nor the periodic prune sweep call the driver's
DeleteSandbox RPC when a sandbox's compute resource is discovered gone
out-of-band (container removed without going through OpenShell's own
delete path, or the gateway crashing mid-create). Only store-side
records were cleaned up, leaving driver-owned secrets and volumes
behind indefinitely — a credential leak for the Podman driver's
per-sandbox bearer-token and proxy-auth secrets.
DeleteSandbox is already idempotent and already reclaims these
resources when the underlying container is already gone, so no new
RPC is needed: this wires the existing call into the two paths that
discover a sandbox is gone without going through an explicit delete
request.
The watch path defers the actual call to a background task, gated by
a non-blocking check of the sandbox's lifecycle gate: watch events are
processed sequentially and must never block on a driver call, and
skipping the call entirely when a concurrent request-side operation
already holds the gate avoids racing that operation's own driver call.
The prune sweep calls it inline, since it already makes a blocking
GetSandbox call per sandbox as part of its normal operation.
Closes#2352
Signed-off-by: politerealism <burdcat17@gmail.com>
* fix(server): stop holding sync_lock across the prune sweep's driver call
The prune sweep's second sync_lock acquisition has no inner block
scope, so it's held through the rest of prune_missing_sandbox,
including the driver cleanup call added in the previous commit. That
call is a network RPC to the compute driver; holding the gateway-wide
sync_lock across it blocks every other sandbox operation on the
gateway for as long as that one driver call takes, not just the
sandbox being pruned.
Use the same backgrounded spawn_driver_sandbox_cleanup already used by
the watch path instead of the inline cleanup_driver_sandbox_resources,
which is now unused and removed. Add a regression test that blocks the
driver's delete call indefinitely and confirms the prune sweep still
completes and removes the store record promptly.
Addresses gator finding GATOR-9902b7bb-01 on #3042.
Signed-off-by: politerealism <burdcat17@gmail.com>
---------
Signed-off-by: politerealism <burdcat17@gmail.com>
* fix(compute): recover Error-phase sandboxes on gateway startup
Sandboxes whose container exited on its own (e.g. SIGTERM from a Podman
or Docker machine restart) were left stuck in the Error phase even though
the container could be restarted. The startup sweep skipped every
Error-phase sandbox unconditionally.
Include Error-phase sandboxes in the startup sweep when their Ready
condition reason marks a container exit or stop. If the driver restarts
the container, move the sandbox back to Provisioning with a Resumed
condition; if the container is gone or the start fails, leave the Error
state untouched. Genuine (non-container-exit) errors are still skipped.
Container-exit restart is handled by the drivers' existing idempotent
start_sandbox path, which already restarts stopped/exited containers.
The ContainerExited/ContainerStopped condition reasons are promoted to
shared constants in openshell-core so the gateway and drivers agree on
the recovery signal.
Fixes#2179
Signed-off-by: Ian Miller <milleryan2003@gmail.com>
* fix(compute): keep ordinary container exits terminal during startup recovery
Startup recovery treated every non-OOM container exit as recoverable,
lumping application crashes and non-zero exits together with
machine/daemon-restart signal kills under the shared `ContainerExited`
Ready-condition reason. On the next gateway startup it restarted those
crashed workloads and replaced the terminal error with `Resumed`,
erasing the failure signal and repeatedly reviving crash-prone
sandboxes.
Introduce a distinct `ContainerRuntimeRestart` reason for containers
terminated by an external signal (exit 137/143 = SIGKILL/SIGTERM), which
is the signature of a machine/daemon restart. The Podman inspect-based
classification emits it; the startup recovery gate now recovers only
`ContainerRuntimeRestart` and `ContainerStopped`. Ordinary
`ContainerExited` records stay terminal, so genuine failures keep their
error signal. Because only the new reason is recoverable and older
gateways never persisted it, exits recorded before this change also stay
terminal after an upgrade.
Add regression coverage: a generic `ContainerExited` error is not
relaunched, a signal-kill classifies as `ContainerRuntimeRestart`, and
the startup sweep recovers only the runtime-restart record.
Signed-off-by: Ian Miller <milleryan2003@gmail.com>
* fix(docker): reclassify signal-killed containers as runtime restart
Mirror the Podman classification in the Docker driver so that
externally signal-killed containers (exit 137/143, non-OOM) are
persisted with the ContainerRuntimeRestart ready reason and become
eligible for startup recovery, while ordinary exits and OOM kills stay
terminal.
current_snapshots now inspects EXITED containers to obtain the exit
code and OOM flag, then applies the shared exit classification. Without
the inspect step the list-summary path lacks an exit code, so the
recovery gate could never fire for Docker.
Adds unit tests covering signal-kill reclassification (137/143),
ordinary exit staying terminal (exit 1), and OOM staying terminal
despite exit 137.
Signed-off-by: Ian Miller <milleryan2003@gmail.com>
---------
Signed-off-by: Ian Miller <milleryan2003@gmail.com>
Ctrl-C closes gateway listeners while supervisor sessions are still current,
causing benign HTTP/2 broken pipes to be logged as warnings. Track gateway
shutdown so recognized transport closes are treated as expected.
Signed-off-by: Kris Hicks <khicks@nvidia.com>
* feat(gateway): add installation name configuration
Add a first-class operator-assigned gateway name with TOML, CLI, environment,
and Helm configuration surfaces. Local gateways default to openshell, while
Helm defaults to the chart fullname; operators sharing a collector across
namespaces or clusters can set a globally distinct name.
Signed-off-by: Kris Hicks <khicks@nvidia.com>
* feat(gateway): identify gateways in exported traces
Attach the configured gateway installation name and compute driver to the
gateway OpenTelemetry resource so operators can filter traces from multiple
installations that share a collector. Forward the gateway name and OTLP
endpoint to managed external drivers so their distinct service resources carry
the same installation identity.
Keep service.name stable per process type, omit blank resource values, and
leave per-span operation names and request attributes unchanged.
Refs #2507
Signed-off-by: Kris Hicks <khicks@nvidia.com>
---------
Signed-off-by: Kris Hicks <khicks@nvidia.com>
* feat(driver): add MXC compute driver for Windows isolation sessions
Introduces the openshell-driver-mxc crate implementing ComputeDriver
backed by Microsoft MXC isolation sessions (Windows only). Wires the
new driver into the server's build_compute_runtime dispatch and adds
the Mxc variant to ComputeDriverKind.
Also adds a local protobuf-src stub (tools/protobuf-src-local) to
unblock Windows builds that lack MSYS2/MinGW, and pins the zig
Windows x64 toolchain in mise.lock.
(cherry picked from commit 4f7012224efb18fbfeb47aa87e0cfd3f036f32f0)
Signed-off-by: Jamie King <jamiek@nvidia.com>
* wip(mxc): checkpoint hung-agent work (recon, policy_map embed, A1 wiring, demo artifacts)
Safety checkpoint of uncommitted work from the background agent run that stalled mid-Step-7. Includes: mxc-driver-recon.md (Step 0.5), policy_map.rs (~876L embedded mapper), A1 policy-threading edits across driver.rs/policy.rs/mxc.rs/compute/mod.rs, and examples/ (demo.yaml + mxc-gateway.toml). Not yet verified to compile end-to-end; to be reorganized into the skill's Step 11 commit sequence.
(cherry picked from commit 38e42c03870be3d10e984a54f17a3b61122ff510)
Signed-off-by: Jamie King <jamiek@nvidia.com>
* test(mxc): fix lifecycle and policy unit-test compile drift
- Bring futures::StreamExt into scope for the watch-stream `.next()` call in
driver::lifecycle_tests so the negative policy proof test compiles.
- Bind a local `mapper` and drop the unused/deprecated NetworkBinary in the
embedded-mapper network-policy rejection test.
Signed-off-by: Jamie King <jamiek@nvidia.com>
(cherry picked from commit 039b0baf98735ca672dae52be8d3af2417dc0c1a)
Signed-off-by: Jamie King <jamiek@nvidia.com>
* fix(mxc): downgrade missing sandbox_token to debug log
The gateway mints `sandbox_token` only when a sandbox-JWT issuer is
configured. There is no in-sandbox supervisor on MXC (supervisor-removal
design — D1/D4), so no component ever consumes the token; requiring it
on the driver side blocks the demo's `--disable-tls` smoke gateway with a
spurious `invalid_argument`. Log the absence and proceed instead.
Signed-off-by: Jamie King <jamiek@nvidia.com>
(cherry picked from commit cea209797d0edcb1d152251e748900b0a63cca62)
Signed-off-by: Jamie King <jamiek@nvidia.com>
* fix(mxc): keep sandbox Ready after a successful one-shot agent exec
monitor_exec demoted Ready->Error on exit 0 (reason ExecCompleted), so the positive demo (write hello.txt + exit) landed in Error phase. Keep Ready=True (reason AgentCompleted) on success; only non-zero exits go to ExecFailed. Tighten the positive lifecycle test to assert the terminal condition stays Ready=True/AgentCompleted. Verified live via gateway mock round-trip: phase now Provisioning->Ready with no demotion.
(cherry picked from commit 54ab030f03ca0f83d0050d8dd843b633651684ad)
Signed-off-by: Jamie King <jamiek@nvidia.com>
* feat(mxc): add processContainer backend for default-deny enforcement
Add a backend selector to the MXC driver (isolation_session default | process_container). process_container drives a one-shot AppContainer that is genuinely default-deny: a write to any ungranted path is denied by the OS, unlike isolation_session which is grant-only and cannot deny. The lifecycle forks on the flag - isolation_session keeps provision/start/exec, process_container runs a single ephemeral container via run_oneshot.
Also: run-demo.ps1 gains -Backend and hardens the CLI register/create calls; docs corrected to state isolation_session does NOT deny out-of-policy writes and that the negative proof requires process_container.
Verified end-to-end on a real demo box (gateway -> CLI -> driver -> MXC): in-policy write succeeds, out-of-policy write denied (PermissionDenied), OVERALL: PASS.
(cherry picked from commit c6cde3860bbe1b8edb3147d3e840f6bf0ece32d8)
Signed-off-by: Jamie King <jamiek@nvidia.com>
* refactor(driver-mxc): embed policy mapper as a module; remove standalone crate
Adopt the proto-based mapper (map_to_mxc) as the single source of truth,
embedded in openshell-driver-mxc as a Windows-gated `policy_map` module.
Rewire EmbeddedPolicyMapper to call it directly on the typed SandboxPolicy,
deleting the serde_yaml proto->YAML bridge. Move the CLI to a windows-gated
example and the parity tests into the crate; delete openshell-policy-mapper.
- gate policy_map + seam Windows-only (MXC is Windows-only)
- drop serde_yaml; add dev-deps openshell-policy, clap, anyhow
- normalize mapped paths to Windows form in the seam, in one place
- docs: add driver-mxc to AGENTS.md table; correct design doc section 17 test lane
Signed-off-by: Giedrius Burachas <gburachas@nvidia.com>
(cherry picked from commit f22f9c7a25b9a651c5c5cc73f62fb01c4d6c1a8d)
Signed-off-by: Jamie King <jamiek@nvidia.com>
* feat(driver-mxc): implement lossless split_policy for proxy-delegated egress
Signed-off-by: Giedrius Burachas <gburachas@nvidia.com>
(cherry picked from commit 96d6afa0e2dc6a1d54edd12c34a0ceb0a30dadd0)
Signed-off-by: Jamie King <jamiek@nvidia.com>
* feat(driver-mxc): implement Pattern-C governed-egress split through the policy seam
- split_policy: SocketAddr proxy_redirect (replaces bare port), processcontainer
containment guard naming MXC M1, version preserved in the trimmed proxy_policy,
delegation reported as an info loss item
- seam: MappedConfig carries trimmed_policy + proxy_addr; MapCtx.egress selects
the split path; coarse path unchanged when egress is disabled
- driver: [openshell.drivers.mxc] egress_proxy / egress_proxy_addr config,
validated at create (isolation_session rejected until M1); lifecycle threads
the redirect into provision and stores the trimmed policy per sandbox,
emitting an EgressRedirect platform event
- mxc: optional MxcNetwork block (defaultPolicy=block + proxy) in provision and
one-shot configs; mock records configs for test assertions
- tests: lossless-invariant suite over all example policies (validate +
serialize round-trip), split lifecycle proof, M1 rejection; example gains
--split --proxy-addr writing mxc-config.json / trimmed-policy.yaml /
loss-report.json
Signed-off-by: Giedrius Burachas <gburachas@nvidia.com>
(cherry picked from commit 34d54ad9f25dc6034c3ba15668555ff0d22cddd8)
Signed-off-by: Jamie King <jamiek@nvidia.com>
* fix(driver-mxc): emit MXC network.proxy as {localhost: port}
Verified against the real wxc-exec 0.6.0-alpha via --dry-run: MXC accepts
only the {localhost: N} proxy shape (the form the design doc specifies)
and rejects {host, port} with a parse error. Schema 0.6.0-alpha can
express only a loopback port, so non-127.0.0.1 redirect addresses are now
rejected: split_policy emits an error loss (no proxy block) and the driver
refuses egress_proxy_addr values off 127.0.0.1. Per-sandbox attribution
must use per-sandbox ports until the schema widens.
Signed-off-by: Giedrius Burachas <gburachas@nvidia.com>
(cherry picked from commit edde8d5434571fd5398409204fcf6862672c0793)
Signed-off-by: Jamie King <jamiek@nvidia.com>
* fix(driver-mxc): serialize isolation_session stop/deprovision as unit variants
Empirical contract finding from the real test lane (build 26300.8553,
wxc-exec 2026-06-10): the stop and deprovision experimental blocks are
unit variants in the wxc-exec schema and must serialize as null; sending
{} is rejected with malformed_request (invalid type: map, expected unit),
while provision/start accept maps. The production invoker, the real-lane
test, the probe script, and the e2e runner all sent {} - the driver could
provision and run an agent but never stop or delete an isolation-session
sandbox against this build. Pinned by a unit test.
Signed-off-by: Giedrius Burachas <gburachas@nvidia.com>
(cherry picked from commit 0df39ca0b22ebc21eb965b2b567a5b2cac26af32)
Signed-off-by: Jamie King <jamiek@nvidia.com>
* test(driver-mxc): add Tier-0 mapper coverage matrix with schema drift guard
Three-quadrant, table-driven matrix (38 tests): mappable fields assert
exact MXC output; every OpenShell field MXC cannot express asserts a loss
item with the expected severity (and seam rejection on error); an empty
policy asserts the restrictive default-deny posture for every MXC knob
OpenShell does not control. The handled_fields_inventory drift guard
serializes a fully-populated policy and compares its YAML keys against
the mapper-handled field lists, so a new openshell-policy field fails the
suite until consciously mapped, delegated, or reported as loss.
Re-exports the policy seam types for integration tests; adds serde_yml,
base64, serde_json as dev-dependencies.
Signed-off-by: Giedrius Burachas <gburachas@nvidia.com>
(cherry picked from commit 91807f984a3b16846e35d6ca0d5ec41057aafa3a)
Signed-off-by: Jamie King <jamiek@nvidia.com>
* feat(driver-mxc): inject agent_env into sandbox process.env
Add MxcComputeConfig.agent_env: each entry is either KEY=VALUE (verbatim) or a bare KEY resolved from the gateway host environment at launch, keeping secrets (e.g. inference API keys) out of the config file. Wire it into the agent process so gateway-launched agents can authenticate to cloud endpoints (process.env was previously hardcoded empty). Unit-tested via resolve_agent_env_passthrough_and_host_lookup.
Also add a gateway-driven cloud-inference (T1) test harness: mxc-inference.toml (agent_env + curl agent), inference.yaml policy, and run-inference-test.ps1 which starts the gateway, creates an isolation_session sandbox, runs an authenticated Nemotron call, and bundles redacted results. Documented agent_env in mxc-gateway.toml. Validated end-to-end on the test box (chat HTTP 200 + completion via the gateway).
(cherry picked from commit 94d9e827b8b77e0af9dc654943ea8e7d8400cc9f)
Signed-off-by: Jamie King <jamiek@nvidia.com>
* test(driver-mxc): avoid unsafe env mutation in resolve_agent_env test
Replace std::env::{set,remove}_var (unsafe + racy under parallel test
execution in edition 2024) with a read-only PATH lookup. Preserves all
three behaviors under test and drops the #[allow(unsafe_code)].
(cherry picked from commit ac5766eeab2db4e8cc6fcd8d8a97809edaf3df30)
Signed-off-by: Jamie King <jamiek@nvidia.com>
* fix(mxc): adapt MXC driver to current GitHub OpenShell API
The MXC driver crate was authored on GitLab against an earlier proto/core
API. Adapt it to the API on GitHub main:
- build_capabilities_response no longer takes supports_interactive_session
- DriverSandboxSpec.gpu (bool) is now resource_requirements; detect GPU via
effective_driver_gpu_count(driver_gpu_requirements(..))
- DriverSandbox gained a `workspace` field
- SandboxPolicy gained `network_middlewares`: pass it through the proxy split,
emit a loss item on the coarse MXC path, and account for it in the mapper
drift-guard test
Verified: cargo check + 75 mock-based tests pass (lib 27, examples 10,
policy_mapper_matrix 38).
Signed-off-by: Jamie King <jamiek@nvidia.com>
* feat(server): wire the MXC compute driver into the gateway on Windows
Register openshell-driver-mxc as the Windows-only in-process compute
backend so compute_driver = "mxc" resolves to a working runtime:
- ComputeRuntime::new_mxc, adapted to the current 11-arg from_driver
- mxc_policy_sink A1 side channel, staged in create_sandbox before dispatch
- mxc_config_from_context loader and the Mxc dispatch arm (Windows
constructs; other targets return an explicit "Windows-only" error)
- Windows-gated openshell-driver-mxc dependency
- Mxc arms for the telemetry, config-file required-fields, and CLI
reserved-builtin matches to keep them exhaustive/correct
Verified with cargo check --workspace --features openshell-prover/bundled-z3
on x86_64-pc-windows-msvc, stacked on PR #2496.
Signed-off-by: Jamie King <jamiek@nvidia.com>
* fix(driver-mxc): implement GetGatewayListenerRequirements for #2496 base
Signed-off-by: Jamie King <jamiek@nvidia.com>
* test(driver-mxc): add probe-gated real wxc-exec test lane (no mocks)
- tests/wxc_exec_real.rs: ignored-by-default integration tests against a
real wxc-exec. Six --dry-run contract tests run wherever the binary
exists (they caught the network.proxy shape mismatch); enforcement
tests (processcontainer default-deny positive/negative, isolation
session lifecycle round trip with a deprovision drop-guard) probe the
backend and SKIP with a recorded reason where it is not live.
- examples/probe-mxc-host.ps1: classifies a host (OS build, --probe,
per-backend trial) and emits a JSON capability verdict.
- examples/run-mxc-e2e.ps1 + e2e-policies/: scenario runner generalizing
run-demo.ps1 (fs-rw, fs-readonly, fs-default-deny-empty,
network-policy-rejected) with PASS/FAIL/SKIP gating and a stale
OPENSHELL_MXC_MOCK_WXC guard in real mode.
- tasks/windows.toml: windows:test:mxc-real:x64, windows:e2e:mxc,
windows:e2e:mxc:mock.
Signed-off-by: Giedrius Burachas <gburachas@nvidia.com>
(cherry picked from commit 49afafe892caded59c4df50a9b652011ad97f41c)
Signed-off-by: Jamie King <jamiek@nvidia.com>
* fix(driver-mxc): address PR review feedback
Signed-off-by: Shailendra Singh <shailendras@nvidia.com>
* docs: defer public MXC documentation
Signed-off-by: Drew Newberry <385+drew@users.noreply.github.com>
* fix(driver-mxc): build Windows capabilities response
Signed-off-by: Drew Newberry <385+drew@users.noreply.github.com>
* fix(server): gate in-tree tracing on Windows
Signed-off-by: Drew Newberry <385+drew@users.noreply.github.com>
* test(windows): fix cross-platform test assumptions
Signed-off-by: Drew Newberry <385+drew@users.noreply.github.com>
* test(windows): verify process and PEM portably
Signed-off-by: Drew Newberry <385+drew@users.noreply.github.com>
* test(windows): keep lifecycle command in policy
Signed-off-by: Drew Newberry <385+drew@users.noreply.github.com>
---------
Signed-off-by: Jamie King <jamiek@nvidia.com>
Signed-off-by: Giedrius Burachas <gburachas@nvidia.com>
Signed-off-by: Shailendra Singh <shailendras@nvidia.com>
Signed-off-by: Drew Newberry <385+drew@users.noreply.github.com>
Co-authored-by: Prashant K <pkhodade@nvidia.com>
Co-authored-by: Giedrius Burachas <gburachas@nvidia.com>
Co-authored-by: Shailendra Singh <shailendras@nvidia.com>
Co-authored-by: Drew Newberry <385+drew@users.noreply.github.com>
Mirror the VM, Podman, and Docker driver tracing setup for Kubernetes.
Export standalone driver spans through OTLP/gRPC as the distinct
openshell-driver-kubernetes service, preserve gateway trace context, record
lifecycle operations and gRPC failures, and flush spans on shutdown.
Kubernetes currently runs in-process when selected as a built-in gateway
driver. Use the temporary server-boundary shim shared with Podman and Docker
so traces retain the shape they will have when Kubernetes moves to a
separate process. Move the common ComputeDriver RPC tracing layer into
openshell-otel to keep all drivers aligned.
Propagate the active W3C context through the controller-reserved Sandbox
annotation and enable Agent Sandbox OTLP export in the local k3s workflow.
This connects asynchronous controller reconciliation spans to the originating
OpenShell create trace.
Expose gateway OTLP configuration through Helm and add an Aspire collector
to the local k3s workflow. Extend helm:k3s:forward with OTLP ingest and trace
UI forwarding for Kubernetes and local container gateway development.
Signed-off-by: Kris Hicks <khicks@nvidia.com>
On Agent Sandbox v1beta1, a `sandbox stop` then `sandbox start` left the
sandbox stuck at `Starting` and `start` timed out after 300s, even though
the workload pod was `Running 1/1` and the CR reported `Ready=True`.
The v1beta1 controller (v0.5.0) sets `Suspended=True (PodTerminated)` on
stop and does not clear it on resume, so a resumed CR carries both
`Ready=True (DependenciesReady)` and a stale `Suspended=True`. The gateway's
`derive_phase` treated any `Suspended=True` condition as `Stopped` before it
consulted `Ready`, so a resumed-but-Ready sandbox was read as not-running and
the `Starting` phase never advanced to `Ready`.
`Ready=True` means the sandbox is usable through this gateway and must win
over a `Suspended=True` condition. Only resolve `Stopped` from `Suspended`
when `Ready` is not `True`; a genuine stop leaves `Ready` unset or `False`,
so it still resolves to `Stopped`. v1alpha1 (which encodes desired state as
`spec.replicas` and emits no lingering `Suspended`) is unaffected.
The underlying stale condition is an upstream bug in the agent-sandbox
controller, fixed by kubernetes-sigs/agent-sandbox#1150 (in v0.5.6). This
gateway change keeps OpenShell resilient regardless of controller version.
Fixes#2932
Signed-off-by: Russell Bryant <rbryant@redhat.com>
Mirror the VM and Podman driver tracing setup for Docker. Export Docker
driver spans through OTLP/gRPC as the distinct openshell-driver-docker
service, preserve gateway trace context, record lifecycle and asynchronous
provisioning operations, and report gRPC failures.
Docker currently runs in-process when selected as a built-in gateway driver.
Add the same temporary server-boundary shim used by Podman so traces retain
the shape they will have when Docker moves to a separate process. Generalize
the gateway provider selection for both in-process drivers and share the OTLP
collector fixture across Docker, Podman, and VM tracing tests.
Signed-off-by: Kris Hicks <khicks@nvidia.com>
* feat(provider): add ability to request token exchange instead of client credentials as OAuth grant_type
Signed-off-by: Gordon Sim <gsim@redhat.com>
* test(proxy): add further tests for token exchange
Signed-off-by: Gordon Sim <gsim@redhat.com>
* test(provider): add runnable example for token exchange
Signed-off-by: Gordon Sim <gsim@redhat.com>
* test(e2e): cover Podman token exchange grants
Signed-off-by: Gordon Sim <gsim@redhat.com>
* refactor(oauth): extract duplicated functionality from server and supervisor
Signed-off-by: Gordon Sim <gsim@redhat.com>
* fix(provider): evict nearest-to-expiry entry from intermediate token cache
Signed-off-by: Gordon Sim <gsim@redhat.com>
* doc(supervisor): add podman example for token exchange
Signed-off-by: Gordon Sim <gsim@redhat.com>
* fix(provider): withhold token-exchange subject credentials
Signed-off-by: Gordon Sim <gsim@redhat.com>
---------
Signed-off-by: Gordon Sim <gsim@redhat.com>
* fix(policy): bind reviews to applicable candidates
Build and validate the exact effective-policy candidate before approval, bind review to live policy/provider/credential inputs, and preserve inspected endpoint contracts during mechanistic expansion.
Closes#2821
Signed-off-by: John Myers <johntmyers@users.noreply.github.com>
* fix(policy): canonicalize advisor review inputs
Serialize nested protobuf maps in stable key order for proposal review tokens and effective-policy hashes. Narrow reused multi-port endpoint contracts to the denied port so advisor proposals cannot widen binary access. Add regressions for both cases.
Signed-off-by: John Myers <johntmyers@users.noreply.github.com>
* test(e2e): keep advisor sandbox running
Create the issue 2821 regression sandbox detached with a durable canonical main process so policy denial, approval, and hot-reload checks run before lifecycle exit.
Signed-off-by: John Myers <johntmyers@users.noreply.github.com>
* fix(policy): apply reviewed draft batches atomically
Signed-off-by: John Myers <johntmyers@users.noreply.github.com>
---------
Signed-off-by: John Myers <johntmyers@users.noreply.github.com>
Co-authored-by: John Myers <johntmyers@users.noreply.github.com>
Closes#2877
Use the gateway's configured provider profile catalog for sandbox creation and provider attachment validation.
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
Mirror the VM driver tracing setup for Podman. Export standalone driver
spans through OTLP/gRPC as the distinct openshell-driver-podman service,
propagate W3C context across ComputeDriver RPCs, record bounded RPC names
and failures, and flush buffered spans during graceful shutdown.
Podman still runs in-process when selected as a built-in gateway driver.
Add a temporary tracing shim that partitions gateway and Podman spans by
target into separate tracer providers while preserving their shared trace
and parentage. The shim also emits the same ComputeDriver server boundary
that the tonic layer emits out of process, keeping the observable trace
shape stable when Podman is eventually extracted.
Trace container create preparation, image and storage setup, lifecycle
operations, and cleanup. Document the service boundary and cover it with
isolated and repeated tracing tests.
Signed-off-by: Kris Hicks <khicks@nvidia.com>
