## Summary
Bump the pinned libkrunfw from v5.6.1 to v5.6.2, the latest upstream
release. The guest kernel stays on Linux 6.12.109 and the ABI stays at 5.
## Changes
- libkrunfw: v5.6.1 (617938cf) -> v5.6.2 (f14ac64c)
Upstream changes in v5.6.2: swap, zram (zstd), MGLRU and PSI enabled in
the aarch64 and x86_64 kernel configs, PAT/MTRR enabled on x86_64 for
write-combining mappings, and native macOS kernel builds. The Linux build
path of the Makefile ($(KERNEL_SOURCES) and kernel.c targets) used by
build-libkrun.sh is unchanged.
Signed-off-by: Florent Benoit <fbenoit@redhat.com>
## Summary
Bump the pinned libkrun from v1.19.4 to v1.19.6, the latest upstream
release. v1.19.5 includes libkrun/libkrun@d2d8dd6, which removes refused
vsock connections without waiting for the 5 s reaper, so VM sandbox
startup no longer stalls on "Waiting for VM supervisor".
## Related Issue
Closes#4259
## Changes
- libkrun: v1.19.4 (728df812) -> v1.19.6 (227b2de6)
- Update LIBKRUN_REF in pins.env and the fallback defaults in
build-libkrun.sh and build-libkrun-macos.sh
- libkrunfw stays on v5.6.1
Other notable upstream fixes (v1.19.5): vsock proxy release after clean
shutdown, macOS virtio-fs xattr handling, x86_64 initrd kept below the
MMIO gap, PIT missed-tick reinjection disabled, macOS VMs with >64GB
RAM. v1.19.6 adds an API to set the render server fd. No API removals.
## Testing
- libkrun v1.19.6 builds on macOS arm64 with `make BLK=1 NET=1`
Signed-off-by: Florent Benoit <fbenoit@redhat.com>
Reject host paths and malformed CDI selectors for Docker and Podman
using shared qualified-name validation. Preserve the generic GPU
request checks for drivers using other identifier formats, including
VM PCI addresses.
Validate explicit and resolved CDI selectors, cover vendor-agnostic
names and rejected paths, and update the related documentation.
Signed-off-by: Adrien Langou <alangou@nvidia.com>
Signed-off-by: Evan Lezar <elezar@nvidia.com>
* fix(ha): keep sandboxes ready across gateway pod rolls
A gateway pod roll left sandboxes not ready long enough for clients to
fail with "sandbox is not ready", which made the HA e2e test
sandbox_file_sync_survives_gateway_pod_rolls flaky. Two causes:
- The supervisor never reset its reconnect backoff after an accepted
session, so each gateway restart over a sandbox's lifetime doubled the
delay until every reconnect waited the full 30s maximum. Reset it
after an accepted session.
- A stopping gateway replica redirected its supervisors and then
immediately demoted their sandboxes to Provisioning, before the
redirected supervisor published its replacement session. Wait up to
5s for the replacement before demoting; the existing disconnect path
keeps the sandbox Ready once a replacement exists.
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
* fix(ha): wait for handoff only when a redirect went out
Address review feedback on the shutdown handoff:
- Wait for a replacement session only when the stopping replica queued a
SessionRedirect. Single-replica gateways, supervisors without redirect
support, and full outbound channels no longer pay the grace period.
- Close the session stream in both directions before cleanup so a
supervisor without a redirect sees EOF and reconnects at once.
- Bound the replacement wait with timeout_at and the shared session-wait
backoff, so the grace is a hard ceiling.
- Name the supervisor session shutdown timeout and assert at compile
time that it leaves room after the grace period.
- Demote the sandbox when session setup fails after the owner publish,
so a failed takeover cannot leave it Ready with no session.
- Move the supervisor reconnect delays into ReconnectBackoff so both
loop branches reset the same way, and test reconnect sequences.
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
* fix(ha): refresh gateway membership before shutdown redirects
A stopping replica redirected its supervisors using its cached ring,
which the membership worker refreshes only every 10s. During a rolling
update the replacement for the previous pod is often only seconds old,
so the cached ring missed it and still named the departed pod. The
redirect then went nowhere or to a dead pod, whose 10s connect timeout
outlasted the handoff grace, and the sandbox was demoted mid-roll.
Re-read live membership, bounded to 2s, just before session shutdown
computes redirects.
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
* fix(ha): hold ownership through the shutdown handoff
The stopping replica released its owner record before waiting for the
replacement session. A reconciliation sweep on another replica in that
window saw no owner, and with the Kubernetes driver leaving readiness
to the gateway it demoted the sandbox to Provisioning anyway. Keep the
record until a different session supersedes it or the grace expires,
then release it only if it is still ours.
Also from review:
- Keep polling the owner record through store errors until the deadline.
- Skip the handoff wait for sandboxes that are not Ready.
- Re-read membership in the membership worker's shutdown branch and
await the worker before session shutdown, so the worker stays the only
writer of the ring and peer map.
- Share the remove, release, and demote steps of failed session setup.
- Test redirect queuing directly, including unsupported supervisors and
a full outbound channel.
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
---------
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
* fix(sandbox): reject local control peers when the kernel denies the ingress filter
Binding the boundary control listener attaches a classic BPF filter that drops
loopback-interface ingress. The fix for CVE-2026-53236 gates SO_ATTACH_FILTER on
CAP_NET_ADMIN, and a sandbox pod holds no capabilities at all, so the attach
returns EPERM and every sandbox dies at startup with:
bind boundary control listener: Operation not permitted (os error 1)
Granting the capability is not an option: the native Linux audit evidence
requires an empty capability set, so a sandbox that could attach the filter
would fail its own qualification instead.
Treat that EPERM as "filter unavailable" and move the check to accept time. Any
peer the workload can reach the listener from arrives over `lo`, so one end is
loopback or the two ends share an address; a supervisor runs in a different pod
and presents a different address. Every non-loopback listener applies the check
regardless of whether the filter attached, so a sandbox behaves the same on any
kernel and the filter is an extra standing layer where it is available.
The peer address comes from accept() rather than getpeername(), which fails with
ENOTCONN when a peer resets while still queued; the serve loop also treats
NotConnected, ConnectionReset and ConnectionAborted as per-connection errors so a
reset cannot stop the listener. An unavailable filter is reported as an OCSF
ConfigStateChange so operators can see which sandboxes run in this mode.
The isolation-interface filter tests now skip when the kernel denies the attach,
since they fail on every patched kernel without CAP_NET_ADMIN.
Signed-off-by: divesh <dgude@nvidia.com>
* fix(sandbox): remove privileged control socket filters
Use the control listener's peer checks on every kernel instead of attaching a packet filter. Retain workload socket confinement, the broker's control-port reservation, and reset-safe acceptance. Cover local and supervisor address classification and require accept-time rejection in the listener test.
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* docs: remove support matrix changes from filter fix
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
---------
Signed-off-by: divesh <dgude@nvidia.com>
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
Co-authored-by: Drew Newberry <anewberry@nvidia.com>
* fix(providers): bind multi-route grants to admitted endpoints
Carry gateway-derived endpoint owners through policy and credential delivery. Select grants only from owners that admit the current request, refresh provider snapshots per request, and reject superseded installations before forwarding.
Cover persistent routes, overlapping owners, denial, refresh, cache expiry and recovery with focused tests and a Podman regression.
Signed-off-by: Shiju <shiju@nvidia.com>
* refactor(providers): simplify admitted-endpoint grant selection
Share admitted endpoints between forwarding and credential authorization. Select grants from the live snapshot without rebuilding a locked map, reuse profile policy construction, and preserve freshness guards when grants are removed.
Signed-off-by: Shiju <shiju@nvidia.com>
* fix(providers): preserve trusted owners across policy approval
Sanitize authored owner metadata before composing effective policies, preserve derived owners through proposal approval, and normalize advisor endpoint comparisons. Account for gateway-only owner fields in SDK coverage and schema inventory.
Signed-off-by: Shiju <shiju@nvidia.com>
* refactor(providers): share dynamic credential key layout and require a snapshot for admission
Move the endpoint-bound credential key layout into openshell-core. DynamicCredentialKey encodes the key the gateway builds, and the supervisor reads its endpoint selector, credential identity, and revision-scoped form through the same module instead of parsing tab-separated strings in three places. Revision scoping becomes ProviderCredentialSnapshot::scoped_key.
inject_for_admitted_owners now takes the pinned snapshot directly. The L7 admission path never acquires a grant from the live credential map, which production populates only alongside a snapshot. L4 forwarding keeps the selector-only path through inject_if_needed. Both share one acquire-and-rewrite step.
Delete two relay tests whose behavior the Rego admission tests already prove. Add a test that grants for different headers are acquired only for admitting owners.
Signed-off-by: Shiju <shiju@nvidia.com>
* fix(providers): share credential preparation across HTTP relays
Apply endpoint-owned grants after policy and request middleware in REST,
GraphQL, JSON-RPC/MCP and inspected plaintext forwarding. Preserve live
provider and policy checks through guarded writes, reject missing owner
metadata, and reject profiles that cannot inject dynamic HTTP credentials.
Cover production stream dispatch, multiple endpoints, failure handling and
plaintext ownership with focused regressions. Document upgrade ordering.
Closes#3657
Signed-off-by: Shiju <shiju@nvidia.com>
* fix(providers): validate dynamic credential key encoding
Reject control characters at profile import and key construction. Propagate
invalid key errors through gateway environment resolution and reject malformed
key layouts before token acquisition while preserving supported legacy keys.
Consolidate duplicate relay success tests and share forward-proxy setup without
removing ownership, canonicalization, refresh or guarded-write assertions.
Document token-grant endpoint authority.
Signed-off-by: Shiju <shiju@nvidia.com>
---------
Signed-off-by: Shiju <shiju@nvidia.com>
* fix(cli): preserve existing directories during single-file upload
Inspect remote destination types before choosing archive entry names.
Keep directory and source symlinks intact, recheck detected type changes,
and report the resulting path. Add real tar and live conformance coverage.
Closes#4175
Signed-off-by: Shiju <shiju@nvidia.com>
* fix(cli): box upload future in sandbox_create
Boxing the upload future keeps the sandbox_create future under the clippy::large_futures threshold after sandbox_upload_planned began returning the uploaded path.
Signed-off-by: Shiju <shiju@nvidia.com>
---------
Signed-off-by: Shiju <shiju@nvidia.com>
* fix(supervisor): report rejected OCI working directories as WorkspaceValidationFailed
The RFC 0012 split dropped the supervisor exit status that drivers map to
the WorkspaceValidationFailed condition. An image WORKDIR the sandbox
identity cannot use then surfaced as ControlSupervisorStartFailed on
Docker and as a signal kill on Podman.
The supervisor now exits with the reserved status when the sandbox rejects
the image working directory. Docker maps that supervisor exit during
readiness and monitoring, and Podman maps the supervisor companion's exit
instead of the workload's.
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* feat(podman): honor OCI image working directories
Podman sandboxes now use a custom OCI image WORKDIR as the workspace and
as the working directory for agent commands, matching Docker. Empty, /,
and /sandbox values keep the managed /sandbox workspace volume.
A custom workspace stays in the image's container filesystem: no
workspace volume, no archive upload, and no root setup step. Resolve the
image ID, user, environment, and working directory from one pinned
inspection, validate the workdir with the shared OCI rules, and reject
Podman control-path overlaps and image volumes or driver mounts that
cover it. The runtime starts from / and passes the resolved path to agent
commands with --workdir.
Add podman_oci_identity to the Podman CI test list.
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* fix(drivers): preserve custom workspace failures across supervisor cleanup
Check the Podman custom workspace rather than the absent managed volume in the OCI identity E2E. Classify Bollard wait errors by exit code and record Docker workspace failures before removing the supervisor so readiness retains the specific failure reason.
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* fix(docker): use sandbox ID for recorded startup failures
Keep the workload container ID for Docker inspection and use the sandbox ID to retrieve the monitor failure. Test with distinct identifiers so the lookup cannot accidentally pass with a container ID.
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* refactor(podman): keep WORKDIR change focused on workspace behavior
Remove the new cross-driver workspace failure classification path and its monitor/readiness workaround. Preserve Podman WORKDIR resolution, mount safety, and workspace rejection without requiring a distinct condition reason.
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* refactor(podman): simplify inspected image metadata and test fixtures
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* code review
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
---------
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* ci(release): add advisory compatibility review
Signed-off-by: Simon Scatton <sscatton@nvidia.com>
* test(ci): remove compatibility reviewer test harness
Signed-off-by: Simon Scatton <sscatton@nvidia.com>
* ci(release): run compatibility SDK in the dev shell
Signed-off-by: Simon Scatton <sscatton@nvidia.com>
* ci(release): trigger a one-shot compatibility review
Signed-off-by: Simon Scatton <sscatton@nvidia.com>
* ci(release): remove one-shot review trigger
Signed-off-by: Simon Scatton <sscatton@nvidia.com>
---------
Signed-off-by: Simon Scatton <sscatton@nvidia.com>
Adds the two stdlib-only helpers that back the maintainer approval gate,
ahead of the workflows that call them.
check_maintainer_approval.py parses the linked handles out of
MAINTAINERS.md, folds a review list to each reviewer's latest decisive
position, and exits non-zero unless a maintainer's latest position is an
approval. It fails closed when the list yields no handles.
alert_maintainer_change.py renders the approver-set delta between two
versions of MAINTAINERS.md, and prints nothing when the set is unchanged.
Landing these first lets the gate workflow, which reads both the list and
the decision logic from main rather than from the pull request ref,
actually execute on the pull request that introduces it.
Signed-off-by: Jim Meyer <jimeyer@nvidia.com>
* feat(server): place supervisor sessions by consistent hash and hand them off on shutdown
Signed-off-by: divesh <dgude@nvidia.com>
* feat: route long-lived sandbox connections to the owning gateway replica
Signed-off-by: divesh <dgude@nvidia.com>
* feat(helm): add opt-in per-replica GRPCRoute routing
Signed-off-by: divesh <dgude@nvidia.com>
---------
Signed-off-by: divesh <dgude@nvidia.com>
* docs(observability): correct supervisor log access and retention
Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>
* docs(observability): clarify runtime log sources and retention limits
Remove Docker copy guidance for the live supervisor tmpfs mount while retaining the Podman example. Distinguish sandbox-runtime security events from supervisor JSONL and gateway log streams, and qualify independent three-file retention.
Validation: mise run docs, Markdown lint, and git diff --check passed. Fern reported the same three existing unrelated warnings.
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
---------
Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Co-authored-by: Matthew Grossman <mgrossman@nvidia.com>
`openshell forward service` minted an SSH session token before every
forwarded TCP connection and revoked it afterwards: two store commits per
connection. The token added nothing on that path. `ForwardTcp` already
authenticates the caller and authorizes it against the sandbox's workspace
on every stream before it looks at the token, the relay to the supervisor
is opened with the sandbox id and target only, and the token is never
forwarded, audited, or visible to the target service. The mechanism exists
for `openshell sandbox ssh`, where the process that opens the stream is an
ssh ProxyCommand holding nothing but the token. Reusing it per TCP
connection put a store write on the connect path and serialized concurrent
forwards on commit latency: #3494 measured the symptom, and #3543 made the
commits cheaper, but each one still holds SQLite's writer lock for an
fsync, so connection setup under a burst stayed linear in the number of
concurrent connections.
Let `target.tcp` streams omit `authorization_token`. The gateway admits
them on the already-authorized principal, counts them against the same
per-sandbox connection cap, and touches no store. `target.ssh` streams keep
requiring the token. A token supplied with a TCP target is still validated
and counted per token, so an older CLI against a new gateway is unchanged.
The CLI stops minting and revoking a session per forwarded connection;
against a gateway that predates this change it recognizes the
`authorization_token is required` rejection once and falls back to
per-connection tokens for the rest of that forward.
Tests cover token-less TCP admission and slot release, SSH targets still
rejected without a token, a supplied token still validated, and the
per-sandbox cap for token-less forwards. CLI integration tests run
`service_forward_tcp` against a mock gateway: token-less inits echo data
with no CreateSshSession or RevokeSshSession call, and a gateway that
rejects the empty token is detected once, after which every connection in
that forward mints and revokes its own token. Architecture and security
docs describe which targets carry a token, and the per-token connection
limit now reads 3, matching the gateway.
Signed-off-by: Jason T. Greene <jason.greene@redhat.com>
* fix(ssh): persist sandbox host identities
Store each sandbox's Ed25519 host key in the gateway credential store and
deliver it only to the supervisor. Preserve identity across restarts,
delete owned credentials with the sandbox, and expose the public SHA256
fingerprint through sandbox and SSH-session APIs and client SDKs.
Cover credential ownership, cancellation, deletion retries, client
compatibility, and pinned SSH connections through lifecycle transitions.
Closes#3835
Signed-off-by: Mike Nguyen <miken@nvidia.com>
* fix(compute): clean up failed sandbox SSH identity creation
Signed-off-by: Mike Nguyen <miken@nvidia.com>
* test(ssh): wait for sandbox deletion before name reuse
Signed-off-by: Mike Nguyen <miken@nvidia.com>
---------
Signed-off-by: Mike Nguyen <miken@nvidia.com>
Drew Newberry