Use ManagedGateway's per-process supervisor image override so the fixture
CA reaches the supervisor when runtime image environment variables take
precedence over gateway TOML. Restore the wrapper's original image after
each test, following the provider-readiness fixture.
Signed-off-by: Shiju <shiju@nvidia.com>
When the gateway exports OTLP traces, compute drivers pass the gateway's
endpoint to supervisors as OPENSHELL_OTLP_ENDPOINT, along with TRACEPARENT
from the operation that launched them. The Kubernetes, Docker, Podman, and
VM drivers all receive the endpoint from the gateway; driver TOML cannot
set it. On Podman Machine, the Podman driver points a loopback endpoint
at host.containers.internal, since supervisor loopback is the VM's.
The supervisor exports spans as openshell-supervisor, tagged with
the sandbox ID, and flushes them before exiting.
supervisor.startup joins the sandbox's creation trace and covers image
policy discovery, policy load, and boundary attach, confirm, agent start,
and access start. Supervisor calls to the gateway carry W3C trace context,
so the gateway's server spans nest under them.
Each egress connection emits supervisor.egress.connect with authorize,
resolve, and dial children. These spans use DEBUG level because every
outbound connection starts its own trace. OpenShell spans export at INFO,
or at the sandbox log level when it is debug or trace; spans from other
libraries export at INFO.
Signed-off-by: Kris Hicks <khicks@nvidia.com>
* feat(providers): compose dynamic credential grants per request
Select the most-specific grant independently for each protected header,
preserving each credential's issuer, audience and cache identity. Resolve
all selected grants before rewriting the request, replace agent-supplied
header copies, and fail closed on collisions or acquisition failures.
Allow distinct-header compositions in gateway validation and redact raw
issuer errors. Cover independent cache entries, atomic TLS relay failure,
header replacement and concurrent request isolation; document the profile
contract.
Fixes#3320
Signed-off-by: Shiju <shiju@nvidia.com>
* fix(providers): correct dynamic grant CI checks
Use if-let for a grant result whose error payload is deliberately ignored,
and name the empty query map type in the regression fixture. Preserve grant
acquisition, failure redaction and request atomicity. Update the token-exchange
failure test to require the sanitized error instead of raw issuer text.
Signed-off-by: Shiju <shiju@nvidia.com>
---------
Signed-off-by: Shiju <shiju@nvidia.com>
* refactor(supervisor): move backend setup behind the selected backend
Keep launch decoding, policy discovery and client construction with the
trusted backend while sharing supervisor admission and lifecycle handling.
Preserve VM policy identity validation and live credential handles.
Exercise alternate launch data through shared startup, readiness and
shutdown, including confirmation denial and resource cleanup.
Closes#4173
Signed-off-by: Shiju <shiju@nvidia.com>
* fix(supervisor): resolve backend setup lint failures
Box shared startup futures to keep callers' async state small.
Use the private module boundary for backend setup visibility and keep
test helpers and defaults explicit without changing their inputs.
Signed-off-by: Shiju <shiju@nvidia.com>
* refactor(supervisor): simplify selected backend startup
Signed-off-by: Shiju <shiju@nvidia.com>
* test(supervisor): keep imports before descriptor assertions
Signed-off-by: Shiju <shiju@nvidia.com>
* test(supervisor): preserve discovery failure startup coverage
Exercise backend discovery failure through shared startup before the
existing confirmation denial and successful shutdown cases. Assert that
startup preserves the error without launching a workload or readiness.
Signed-off-by: Shiju <shiju@nvidia.com>
---------
Signed-off-by: Shiju <shiju@nvidia.com>
Revert PR #4107 and restore binary installers for the Podman driver qualification lanes.
This reverts commit 8990656c13.
Signed-off-by: Simon Scatton <sscatton@nvidia.com>
* fix(podman): create managed workspace volumes owned by the workload identity
Podman now creates the managed /sandbox volume with uid/gid options for the
resolved workload identity, so the workload starts directly as that
identity. This fixes rootful sandboxes whose image USER or policy
run_as_user could not write to a root-owned /sandbox, and removes the
root-then-drop workspace chown start path.
Resource admission accepts the managed workspace volume when its options
match the workload container's final identity, or are empty for volumes
created by older gateways. The channel volume still requires empty options.
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* test(podman): cover managed volume reuse and workspace access
Signed-off-by: Evan Lezar <elezar@nvidia.com>
* refactor(podman): clarify managed volume creation and validation
Signed-off-by: Evan Lezar <elezar@nvidia.com>
* test(podman): verify workspace access across user namespaces
Signed-off-by: Evan Lezar <elezar@nvidia.com>
---------
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Evan Lezar <elezar@nvidia.com>
Co-authored-by: Evan Lezar <elezar@nvidia.com>
* feat(server): add gateway capacity metrics and optional HPA
Expose per-replica supervisor sessions, pending relay capacity, relay
rejections and claim latency, and outbound peer request outcomes and
latency. Use bounded labels and Prometheus histograms for the new latency
metrics while preserving existing summary metrics.
Add an optional Helm HPA with external-database and resource validation,
conservative scale-down defaults, and support for custom metrics. Keep
certificate hook pods outside gateway workload selectors.
Document per-pod scraping, scaling limits, upgrade behavior, and
PostgreSQL connection sizing.
Part of #3528
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* feat(server): unify routing metrics and clarify replica capacity
Combine local relay setup and outbound peer requests in one counter, labeled by operation, route, target, outcome, and gRPC status. Preserve peer latency metrics.
Rename the rejection reason from global_capacity to replica_capacity to reflect the per-replica relay budget. Keep capacity limits unchanged.
Update tests and documentation.
Signed-off-by: divesh <dgude@nvidia.com>
* fix(server): align routed request metric labels and outcomes
Count a local relay as successful only when its supervisor claims it, the
same event that answers a peer relay on the owner, and rename the outcomes
to local_error and remote_error so they say where an attempt failed. Label
the peer latency histogram by operation, like the routed request counter,
and rename the target label to relay_kind so it does not read as the
Prometheus scrape target.
Rename RelayCapacity.global to per_replica, drop the per-sandbox relay
capacity gauge, which no per-sandbox series can pair with, describe the
latency histogram as peer-only, and restore the note that unavailable
spikes are expected during rollouts.
Part of #3528
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
---------
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
Signed-off-by: divesh <dgude@nvidia.com>
Co-authored-by: divesh <dgude@nvidia.com>
Closes#3502
Resolve trusted OCI runtime images with driver TOML, process environment, and compiled-default precedence across Docker, Podman, and Kubernetes.
Signed-off-by: Evan Lezar <elezar@nvidia.com>
Add credentials[].stable_placeholder so a running client keeps using an externally rotated static credential through the reference it already holds. Squashed replay of the earlier four commits onto upstream main 71c3cd957; re-derives the frozen schema fingerprints and adapts the provider environment revision test to the added capability argument.
Signed-off-by: Shiju <shiju@nvidia.com>
* fix(vm): validate launch credentials before preparation
Negotiate a launch-authentication requirement and reject incomplete gateway
configuration before driver validation, archive consumption or persistence.
Validate replacement VM credentials before stopping active compute and
prefer explicit signer configuration over local discovery.
Closes#3949. Part of #3955.
Signed-off-by: Shiju <shiju@nvidia.com>
* fix(vm): validate saved launch credentials before restore side effects
Signed-off-by: Shiju <shiju@nvidia.com>
* chore(gateway): satisfy launch preflight lint checks
Signed-off-by: Shiju <shiju@nvidia.com>
* docs(gateway): complete the MicroVM launch-signing example
Include the gateway JWT paths in the standalone VM configuration and show
the output directory required by local certificate generation. Explain
the distinction between configuration preflight and launch-key validation.
Refs #3949. Part of #3955.
Signed-off-by: Shiju <shiju@nvidia.com>
* test(vm): account for preparation state in launch credential fixture
Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>
---------
Signed-off-by: Shiju <shiju@nvidia.com>
Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>
Co-authored-by: John Myers <9696606+johntmyers@users.noreply.github.com>
* feat(server): separate image preparation and admission deadlines
Give sandbox image preparation its own deadline and start the admission
deadline after preparation finishes. Persist both phases across gateway
restarts and show recovery guidance for the phase that expired.
Preserve the current service authorization schema and regenerate the Go
bindings with the preparation timestamps.
Fixes#3952
Related to #3955
Signed-off-by: Shiju <shiju@nvidia.com>
* fix(cli): simplify preparation timeout fallback selection
Use lazy Option fallbacks while preserving timeout messages and retained
sandbox behavior.
Signed-off-by: Shiju <shiju@nvidia.com>
* docs(server): clarify admission timer prerequisites
Signed-off-by: Shiju <shiju@nvidia.com>
* fix(compute): enforce deadlines during initial sandbox create
Release stalled create operations after preparation expires and preserve
the timeout diagnosis across late driver results. Keep failed-create cleanup
bound to its original attempt so another replica can retry safely.
Signed-off-by: Shiju <shiju@nvidia.com>
* test(server): satisfy deadline regression lints
Drop the create-error mutex guard before matching its cloned value and use
idiomatic iteration and timeout matching in the deadline fixtures.
Signed-off-by: Shiju <shiju@nvidia.com>
* fix(server): retain ownership of pending provisioning operations
Keep submitted create and start requests alive after caller cancellation,
monitor failure, or preparation timeout. Persist request ownership before
dispatch and retain staged uploads while the driver response is pending.
Require timeout cleanup to stop compute after driver settlement without
discarding an active cleanup claim. Fence late result handling against newer
operations, preserve failed-start recovery, and defer automatic restart while
another request owns the sandbox. Expose pending ownership in CLI JSON.
Add ordered multi-replica and cancellation regressions for the review findings.
Signed-off-by: Shiju <shiju@nvidia.com>
* fix(openshell): preserve tracing and accept ready create responses
Carry the request span into the detached provisioning worker so compute
driver calls remain attached to their parent trace after task handoff.
Update the compensation regression to require no backend DELETE when the
durable cleanup claim fails, matching the operation ownership requirement.
Accept the gateway's current Ready snapshot when a sandbox becomes ready
before CREATE returns. Do not require the client to observe an earlier
provisioning phase. Cover the Ready-only watch and command attachment.
Signed-off-by: Shiju <shiju@nvidia.com>
---------
Signed-off-by: Shiju <shiju@nvidia.com>
* fix(relay): close outbound relay stream when target closes first
Closes#3724
out_tx was cloned into the target-reading task, so the function's
own copy kept the outbound relay stream open until the client side
also ended. A target that closed a keep-alive connection never
reached the client as EOF, so reused connections hung forever.
Move out_tx into the target-reading task instead of cloning it, so
dropping it on target EOF ends the outbound stream right away.
Signed-off-by: Eric Curtin <eric.curtin@docker.com>
* fix(relay): keep client uploads after target half-close
Signed-off-by: Eric Curtin <eric.curtin@docker.com>
---------
Signed-off-by: Eric Curtin <eric.curtin@docker.com>
* feat(gateway): validate VM filesystem tools during preflight
Check required local VM tools through config preflight and share executable
resolution with VM image operations. Report selected paths and actionable
errors without creating gateway or sandbox state.
Bound probe output and execution time, and clean up probe descendants on
interruption. Preserve pure static validation and skip local tool checks
for remote driver endpoints and unrelated drivers.
Fixes#3951
Related to #3955
Signed-off-by: Shiju <shiju@nvidia.com>
* fix(gateway): stabilize filesystem preflight checks
Combine identical filesystem-tool error arms and normalize rendered
diagnostics in command tests so terminal wrapping preserves assertions.
Describe driver TLS validation without depending on removed guest fields.
Signed-off-by: Shiju <shiju@nvidia.com>
* test(gateway): serialize preflight fixture paths as TOML
Keep temporary paths quoted and escaped through the TOML serializer
instead of relying on Rust Debug formatting.
Signed-off-by: Shiju <shiju@nvidia.com>
---------
Signed-off-by: Shiju <shiju@nvidia.com>
Run image preparation in an owned worker process, reserve its process
identity until cleanup completes, and protect staging with leases so
cancellation and recovery cannot race with another preparation attempt.
Signed-off-by: Shiju <shiju@nvidia.com>
* fix(vm): enforce the configured workload identity
Reject conflicting policy users and groups before VM image preparation
and before guest attach or process startup changes state. Validate
supervisor policy updates against the protected VM workload identity.
Preserve the gateway CA transport and capability-free sandbox launcher.
Signed-off-by: Shiju <shiju@nvidia.com>
* test(sandbox): clarify VM identity rejection fixtures
Name invalid user and group fixtures distinctly and move the final
workload identity into its group mismatch test.
Signed-off-by: Shiju <shiju@nvidia.com>
* fix(supervisor): align VM identity startup with current APIs
Pass the optional rejection-log key for VM identity failures and keep
generic startup-write regressions free of VM identity constraints.
Repair the call sites after the branch rebase so the identity and cleanup
proposals compile against the current startup helpers.
Signed-off-by: Shiju <shiju@nvidia.com>
* fix(vm): restore inactive sandbox workload identity
Recover the persisted overlay owner before publishing stopped and terminal
sandboxes. Keep resources manageable when identity metadata is invalid.
Clarify fixed MicroVM ownership in policy-generation guidance.
Signed-off-by: Shiju <shiju@nvidia.com>
* test(vm): flush identity fixture before restart
Persist the canonical identity file before readiness and report the observed
exec, canonical and file-owner identities before comparing them.
Signed-off-by: Shiju <shiju@nvidia.com>
---------
Signed-off-by: Shiju <shiju@nvidia.com>
The Docker driver passed a bare reference as CreateImageOptions.from_image
with no tag. The daemon interprets a tagless fromImage as a request for
every tag in the repository and pulls them all (issue #4029).
Normalize a pull reference by appending ':latest' when it has neither an
explicit tag nor a digest, matching 'docker pull' and the Podman driver.
Parsing inspects only the final path component so a registry port (e.g.
'registry:5000/team/app') is not mistaken for a tag and a digest-pinned
reference ('...@sha256:...') is left untouched. Applied at both pull
sites (pull_image and pull_runtime_image).
Signed-off-by: Udaya Tejas <udayatejas2004@gmail.com>
* feat(cli): stream non-TTY exec input before EOF
Add --stream-stdin using the existing interactive exec RPC without a PTY. Preserve separate output streams and enforce the existing 4 MiB cumulative input cap while forwarding input.
Require explicit clean stdin EOF and drain the response through its final gRPC status. Cover held-open input, limits, cancellation, trailers, and default finite-input behavior with subprocess and live sandbox regressions.
Signed-off-by: Shiju <shiju@nvidia.com>
* test(cli): distinguish exec cancellation from stdin EOF
Treat transport termination and response cancellation as separate test observations. Verify explicit stdin EOF through the shared frame writer and cover cancellation in the pinned Tonic decoder.
Signed-off-by: Shiju <shiju@nvidia.com>
* style(cli): use lazy optional stdin dispatch
Signed-off-by: Shiju <shiju@nvidia.com>
* test(cli): use imported duration in stdin EOF regression
Signed-off-by: Shiju <shiju@nvidia.com>
---------
Signed-off-by: Shiju <shiju@nvidia.com>
The release tarball ships an unsigned binary that fails at runtime
when Hypervisor.framework rejects the caller. Sign with the existing
entitlements plist during the build, before artifact upload.
Closes#3506
Signed-off-by: Florent Benoit <fbenoit@redhat.com>
* fix(providers): restore supervisor-backed GCP metadata discovery
Relay the reserved metadata endpoint to the supervisor and restore project, account, and placeholder token responses from live provider state. Cover Google SDK discovery and repeated refresh with provider E2E tests.
Closes#3860
Signed-off-by: Philippe Martin <phmartin@redhat.com>
* fix(providers): preserve default metadata account without an email
Use the default account identifier when the optional service account email is missing or empty. Cover repeated SDK refresh for missing, empty, and configured email values with distinct providers for parallel E2E execution.
Signed-off-by: Philippe Martin <phmartin@redhat.com>
* test(sandbox): fix metadata relay lint and timeout
Signed-off-by: Philippe Martin <phmartin@redhat.com>
---------
Signed-off-by: Philippe Martin <phmartin@redhat.com>
* test(python): synchronize interactive exec TTY readiness
Wait for the complete readiness marker before streaming stdin so PTY echo cannot split the separately written TTY flags. Preserve pipe stream separation and verify consumed stdin and both output sentinels in TTY mode.
Fixes#4075
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* test(python): reuse interactive exec readiness marker
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
---------
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Since #2726 the canonical main process's stdout and stderr are captured
in pipes that feed only the in-memory replay buffer used by sandbox
connect. Agent output therefore never reaches the container's own stdout
and stderr, so it is missing from kubectl logs, docker logs, and podman
logs and from anything that collects container logs. Before #2726 the
entrypoint inherited the container's descriptors and its output appeared
there.
Copy the main process's output to the launcher's stdout and stderr in
addition to the replay buffer, restoring the earlier behavior:
- Output is copied byte for byte to the matching stream from a
forwarder thread per stream, after it is published to the replay
buffer. When the container runtime falls behind on a stream, that
stream's reader waits instead of dropping output, so backpressure
reaches the agent as it did with inherited descriptors, while the
other stream and attachments keep receiving output.
- Before the main process's exit is published, the output readers
finish and queued output is drained to the container log, so an
agent's final lines are not lost at shutdown. A 30 second deadline
covers both; when it expires, readers waiting on the container log
are released and drain the pipes into the replay buffer only, so a
stalled container log cannot block exit reporting.
- PTY-mode processes are not copied. The terminal stream carries escape
sequences and echoed input, and terminal commands never reached the
container log before #2726.
- Exec, SSH, and SFTP sessions are not copied.
Launcher log lines keep their existing format and remain in the
container's stderr. They are written as whole lines, and a newline is
inserted first when the agent left stderr mid-line, so launcher and
agent lines do not merge.
The Docker and VM drivers appended the tail of the workload's output to
failure messages: Docker the workload container's log, and the VM driver
the guest console, which carries the launcher's stdout and stderr. Those
messages land in the sandbox's Ready condition and in platform events
that the gateway republishes to the sandbox event stream. With agent
output in that log, those messages would carry arbitrary agent output,
including anything sensitive the agent prints, into gateway status and
events. The supervisor starts its health endpoint only after the agent
starts, so every Docker failure path could include agent output, and the
VM driver reports one whenever the VM or host supervisor exits. Forward
only the supervisor's log tail, matching the Podman driver, which reads
the workload log solely to match fixed launcher markers and never
forwards raw workload output. The workload's output remains available
through docker logs and the VM's rootfs-console.log.
Document where main process output appears in the logging docs and the
cluster debugging skill.
Closes#3928
Signed-off-by: Kris Hicks <khicks@nvidia.com>
* fix(policy): refresh pending proposals when the sandbox policy changes
Approving, removing, or undoing a rule, or updating the sandbox policy,
changes the inputs every other pending proposal was evaluated against.
Only proposals the new policy covered were reconciled; the rest kept
their old prover result and review token. The review surface
(GetDraftPolicy) therefore showed a stale evaluation, and the first
approval of the next proposal refreshed it and failed with
FAILED_PRECONDITION, so approving proposals one after another always
failed once.
Re-evaluate the remaining pending proposals at each policy change,
reusing the cached prover result unless the proposal's inputs changed.
Approval still rejects a review token that does not match the stored
evaluation, so a reviewer holding a pre-refresh evaluation must still
refetch it.
When a refresh does happen at approval time (inputs changed between
fetch and approve), the CLI now explains that the rule was re-evaluated
and how to review it, instead of printing the raw gRPC status.
Closes#3884
Signed-off-by: fede-kamel <fkamelhar@gmail.com>
* fix(policy): make pending proposal refresh race-safe and bounded
Store refreshed evaluations with a compare-and-swap: the store re-reads
the proposal, refuses when its rule name, proposed rule, or review token
changed since the evaluation read it, copies only the evaluation fields
onto the stored record, and updates only if the payload is still the one
it read. A refresh can no longer revert a concurrent edit or observation,
and the edit path uses the same guard against a concurrent refresh.
Bound each refresh to the 32 newest pending proposals; the rest keep the
approval-time recheck, which still refuses a stale review token. Operator
decisions (approve, approve-all, remove, undo) refresh before responding.
UpdateConfig, which holds the gateway-wide sandbox sync guard, and
agent-driven auto-approval refresh in a background task instead, one per
sandbox with later changes coalesced into a single rerun.
Refs #3884
Signed-off-by: fede-kamel <fkamelhar@gmail.com>
* docs(policy): describe proposal rechecks after approvals and approve-all
Explain that approving, removing, or undoing a rule rechecks the other
pending proposals so they can be approved one after another, when the
recheck is deferred or bounded, and what rule approve reports when a
proposal changed after it was listed. Show rule approve-all in Run Your
First Agent with its security-flag behavior.
Refs #3884
Signed-off-by: fede-kamel <fkamelhar@gmail.com>
* fix(policy): refresh pending proposals after a full policy replacement
A full policy UpdateConfig (openshell policy set) re-reads the latest
revision after its atomic write, finds the revision it just committed,
and returns before reaching the pending-proposal refresh at the end of
the handler. Pending proposals kept their stale evaluation, so rule get
showed the old candidate and the next approval failed with the refresh
precondition. Schedule the background refresh right after the commit.
Refs #3884
Signed-off-by: fede-kamel <fkamelhar@gmail.com>
---------
Signed-off-by: fede-kamel <fkamelhar@gmail.com>
With a non-terminal stdin, sandbox exec read stdin to EOF before it sent
the exec request. A pipe that never closes (CI runners, supervisors, agent
harnesses) blocked the CLI forever in read(2) without the gateway ever
seeing the request, and a slow producer delayed the command until EOF.
Collect piped stdin on a detached reader thread for at most 200 ms. Input
that reaches EOF within that window still travels in the single request
that older gateways need. If the pipe is still open, start the command
through the streaming RPC and forward the collected prefix plus the rest of
stdin as it arrives, closing remote stdin at EOF. The 4 MiB cap covers the
prefix and the streamed remainder together.
Closes#3993
Signed-off-by: Federico Kamelhar <federico.kamelhar@oracle.com>
* fix(supervisor): wait for repair when the gateway refuses a startup policy write
Startup writes the sandbox policy to the gateway in two cases: it
uploads a discovered image policy when the gateway has none, and it
writes the policy back after adding the proxy baseline filesystem paths.
When the gateway refused either write with FAILED_PRECONDITION or
INVALID_ARGUMENT, for example because the policy binds a provider that
is not attached, startup treated the refusal as a permanent error and
the supervisor exited. The sandbox never reached the ConfigurationInvalid
repair state that other startup rejections use.
Report such a refusal as a configuration rejection carrying the
gateway's message, log it once per write and error code, and keep
polling, so attaching the provider or replacing the policy completes
startup. Other error codes keep their current handling: transient codes
are retried, and permission, not-found and authentication failures
still end startup.
Skip the baseline-path write-back while a global policy is active. The
gateway refuses every sandbox policy write in that state, so startup
exited whenever a global policy lacked a baseline path. The supervisor
now adds the paths to its own copy of the policy without saving a
revision.
Signed-off-by: Shiju <shiju@nvidia.com>
* test(supervisor): stabilize startup refusal log capture
Keep a second tracing dispatcher alive while capturing startup refusal
logs. With only one dispatcher, a parallel test thread without a default
subscriber can cache Interest::never for the shared OCSF callsite after
the capture thread rebuilds the cache.
Preserve the exact log-count, diagnostic, configuration-generation and
repair assertions. Production startup behavior is unchanged.
Signed-off-by: Shiju <shiju@nvidia.com>
* fix(supervisor): reconcile stale startup rejection reports
Refetch desired configuration immediately when a rejection report is aborted because its generation changed. Preserve acknowledged rejection pacing and all other report error handling.
Signed-off-by: Shiju <shiju@nvidia.com>
* test(supervisor): box startup repair race futures
Keep the repair regressions below the large-future lint threshold without changing their inputs, scheduling, or assertions.
Signed-off-by: Shiju <shiju@nvidia.com>
---------
Signed-off-by: Shiju <shiju@nvidia.com>
Shiju