* fix(providers): bind multi-route grants to admitted endpoints
Carry gateway-derived endpoint owners through policy and credential delivery. Select grants only from owners that admit the current request, refresh provider snapshots per request, and reject superseded installations before forwarding.
Cover persistent routes, overlapping owners, denial, refresh, cache expiry and recovery with focused tests and a Podman regression.
Signed-off-by: Shiju <shiju@nvidia.com>
* refactor(providers): simplify admitted-endpoint grant selection
Share admitted endpoints between forwarding and credential authorization. Select grants from the live snapshot without rebuilding a locked map, reuse profile policy construction, and preserve freshness guards when grants are removed.
Signed-off-by: Shiju <shiju@nvidia.com>
* fix(providers): preserve trusted owners across policy approval
Sanitize authored owner metadata before composing effective policies, preserve derived owners through proposal approval, and normalize advisor endpoint comparisons. Account for gateway-only owner fields in SDK coverage and schema inventory.
Signed-off-by: Shiju <shiju@nvidia.com>
* refactor(providers): share dynamic credential key layout and require a snapshot for admission
Move the endpoint-bound credential key layout into openshell-core. DynamicCredentialKey encodes the key the gateway builds, and the supervisor reads its endpoint selector, credential identity, and revision-scoped form through the same module instead of parsing tab-separated strings in three places. Revision scoping becomes ProviderCredentialSnapshot::scoped_key.
inject_for_admitted_owners now takes the pinned snapshot directly. The L7 admission path never acquires a grant from the live credential map, which production populates only alongside a snapshot. L4 forwarding keeps the selector-only path through inject_if_needed. Both share one acquire-and-rewrite step.
Delete two relay tests whose behavior the Rego admission tests already prove. Add a test that grants for different headers are acquired only for admitting owners.
Signed-off-by: Shiju <shiju@nvidia.com>
* fix(providers): share credential preparation across HTTP relays
Apply endpoint-owned grants after policy and request middleware in REST,
GraphQL, JSON-RPC/MCP and inspected plaintext forwarding. Preserve live
provider and policy checks through guarded writes, reject missing owner
metadata, and reject profiles that cannot inject dynamic HTTP credentials.
Cover production stream dispatch, multiple endpoints, failure handling and
plaintext ownership with focused regressions. Document upgrade ordering.
Closes#3657
Signed-off-by: Shiju <shiju@nvidia.com>
* fix(providers): validate dynamic credential key encoding
Reject control characters at profile import and key construction. Propagate
invalid key errors through gateway environment resolution and reject malformed
key layouts before token acquisition while preserving supported legacy keys.
Consolidate duplicate relay success tests and share forward-proxy setup without
removing ownership, canonicalization, refresh or guarded-write assertions.
Document token-grant endpoint authority.
Signed-off-by: Shiju <shiju@nvidia.com>
---------
Signed-off-by: Shiju <shiju@nvidia.com>
* fix(supervisor): report rejected OCI working directories as WorkspaceValidationFailed
The RFC 0012 split dropped the supervisor exit status that drivers map to
the WorkspaceValidationFailed condition. An image WORKDIR the sandbox
identity cannot use then surfaced as ControlSupervisorStartFailed on
Docker and as a signal kill on Podman.
The supervisor now exits with the reserved status when the sandbox rejects
the image working directory. Docker maps that supervisor exit during
readiness and monitoring, and Podman maps the supervisor companion's exit
instead of the workload's.
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* feat(podman): honor OCI image working directories
Podman sandboxes now use a custom OCI image WORKDIR as the workspace and
as the working directory for agent commands, matching Docker. Empty, /,
and /sandbox values keep the managed /sandbox workspace volume.
A custom workspace stays in the image's container filesystem: no
workspace volume, no archive upload, and no root setup step. Resolve the
image ID, user, environment, and working directory from one pinned
inspection, validate the workdir with the shared OCI rules, and reject
Podman control-path overlaps and image volumes or driver mounts that
cover it. The runtime starts from / and passes the resolved path to agent
commands with --workdir.
Add podman_oci_identity to the Podman CI test list.
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* fix(drivers): preserve custom workspace failures across supervisor cleanup
Check the Podman custom workspace rather than the absent managed volume in the OCI identity E2E. Classify Bollard wait errors by exit code and record Docker workspace failures before removing the supervisor so readiness retains the specific failure reason.
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* fix(docker): use sandbox ID for recorded startup failures
Keep the workload container ID for Docker inspection and use the sandbox ID to retrieve the monitor failure. Test with distinct identifiers so the lookup cannot accidentally pass with a container ID.
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* refactor(podman): keep WORKDIR change focused on workspace behavior
Remove the new cross-driver workspace failure classification path and its monitor/readiness workaround. Preserve Podman WORKDIR resolution, mount safety, and workspace rejection without requiring a distinct condition reason.
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* refactor(podman): simplify inspected image metadata and test fixtures
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* code review
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
---------
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* fix(ssh): persist sandbox host identities
Store each sandbox's Ed25519 host key in the gateway credential store and
deliver it only to the supervisor. Preserve identity across restarts,
delete owned credentials with the sandbox, and expose the public SHA256
fingerprint through sandbox and SSH-session APIs and client SDKs.
Cover credential ownership, cancellation, deletion retries, client
compatibility, and pinned SSH connections through lifecycle transitions.
Closes#3835
Signed-off-by: Mike Nguyen <miken@nvidia.com>
* fix(compute): clean up failed sandbox SSH identity creation
Signed-off-by: Mike Nguyen <miken@nvidia.com>
* test(ssh): wait for sandbox deletion before name reuse
Signed-off-by: Mike Nguyen <miken@nvidia.com>
---------
Signed-off-by: Mike Nguyen <miken@nvidia.com>
* fix(sandbox): accept local connections natively on loopback-confined sockets
On kernels without SECCOMP_FILTER_FLAG_WAIT_KILLABLE_RECV (RHEL 9 / RHCOS
5.14), the broker cannot safely write an accepted peer address into
workload memory, so accept/accept4 with a peer-address buffer failed with
EOPNOTSUPP. Static binaries and Go servers, which issue the raw syscall,
could not accept connections at all.
Move local acceptance into the kernel and replace per-accept inspection
with standing kernel confinement:
- Bind every broker-created TCP/UDP socket to the loopback device before
injection and verify the binding. Accepted sockets inherit it, so they
can neither receive routed ingress nor emit routed egress.
- Stop notifying accept/accept4 and remove the accept workers, the
SIGUSR2 accept-interrupt monitor, and the 64-accept ceiling.
- Continue getpeername natively for every descriptor except relayed
connections.
- Deny interface-selection socket options and MSG_FASTOPEN sends from
scalar syscall arguments, so confinement does not depend on the
capability state of the user namespace that owns the network namespace
and covers unregistered descriptors.
- Drop loopback-interface ingress on the non-loopback TCP control
listener before it listens, so a workload cannot reach it by
reconnecting a natively accepted socket.
- Mark descriptors above stdio close-on-exec in every workload pre_exec
path.
- Require an active confinement probe at qualification and carry it as
required authenticated audit evidence.
Document OpenShift 4.19 as the minimum release: RHCOS kernels for 4.16
through 4.18 are built without Landlock.
Closes#4058
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* fix(sandbox): close review gaps in native local accept confinement
- Reject a TCP boundary control listener on a loopback address,
including IPv4-mapped loopback. Production drivers bind the
unspecified address; a loopback listener would be reachable from
workload sockets the broker does not track.
- Extend the confinement probe to IPv6 and prove that an accepted
socket keeps its loopback binding after an AF_UNSPEC disconnect.
- Pin the accepted-peer contract with tests: a loopback-bound listener
admits only clients in the sandbox network namespace, and workload
sockets cannot bind a non-loopback source address.
- Correct the support matrix: accepted connections are bounded by
per-process descriptor limits, the runtime PID limit, and the sandbox
memory limit, not by a broker limit.
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* refactor(sandbox): replace unsafe socket calls with socket2 and rustix
Socket confinement now uses socket2 for device binding and the ingress
filter, and rustix for interface lookup and AF_UNSPEC disconnect, so the
module contains no unsafe code. The control-listener filter is no longer
locked: no safe API exposes SO_LOCK_FILTER, and the listener descriptor
never leaves the trusted sandbox process, which marks every descriptor
above stdio close-on-exec before running workload code.
Tests added for native accept use rustix and socket2 instead of raw libc
calls. rustix issues accept4 and getpeername as raw syscalls, so the
direct-syscall test keeps its meaning. The close-on-exec sweep test runs
in a re-executed test process instead of a forked child.
The close_range(CLOSE_RANGE_CLOEXEC) syscall in the pre_exec hook remains
the only unsafe added by this branch; neither rustix nor nix wraps it.
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* fix(sandbox): allowlist workload socket families
The workload seccomp filter denied only AF_PACKET, AF_BLUETOOTH, and
AF_VSOCK, and the broker continued socket() for every non-INET family.
Several protocol families, including AF_RXRPC, AF_SMC, and AF_KCM, carry
traffic over kernel-owned sockets that the broker never creates and that
are not bound to loopback.
Allow only AF_UNIX, AF_NETLINK (still limited to NETLINK_ROUTE), and the
brokered AF_INET/AF_INET6 families, and restrict socketpair(2) to
AF_UNIX. Both decisions use the scalar domain argument. The broker
independently refuses non-INET families other than AF_UNIX and
AF_NETLINK with EAFNOSUPPORT.
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* refactor(sandbox): remove legacy read-only mode
After native local accept, only two broker paths still wrote into
workload memory: getpeername on relayed connections and the per-message
lengths of a first sendmmsg to the DNS relay. Legacy mode existed only to
refuse those writes on kernels without WAIT_KILLABLE_RECV, and the
getpeername refusal broke CPython TLS on RHEL 9.
The broker now never writes workload memory:
- getpeername is no longer mediated and reports the kernel peer on every
kernel; relayed connections report the loopback relay address.
- A first send to the DNS relay pins the broker's socket copy to the
relay and continues the syscall, so the kernel performs the send and
writes any per-message results.
Remove the listener mode, the task-memory write path and probe, and the
task_memory_write, cancellation, and task_memory_writes_disabled audit
evidence fields. WAIT_KILLABLE_RECV is still used when available and is
reported for diagnostics, but no mediation decision depends on it.
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* refactor(sandbox): drop WAIT_KILLABLE_RECV and make handlers restart-safe
The broker no longer writes workload memory, so killable notification
waits only reduced how often a signal restarts a notified syscall. RHEL 9
kernels never had them, so the handlers must tolerate restarts anyway.
Install the listener without the flag on every kernel instead of
special-casing newer ones.
Handlers now check that the notification is still live immediately
before each side effect (bind, connect of the retained socket, listen,
and the first DNS send), and answer a restarted operation the broker
already completed as the kernel would: a repeated TCP connect returns
EISCONN, repeating the same UDP association succeeds, a repeat of a
completed bind succeeds, and a failed relay reports its errno.
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* fix(sandbox): finish teardown only when every workload descendant has exited
Teardown waited only for registered process groups to disappear. A root
is unregistered once reaped, so a descendant that ignored SIGTERM could
outlive it while termination reported success and never sent SIGKILL,
violating the bounded-termination requirement.
The sandbox now becomes a child subreaper when it is not PID 1, so
orphaned descendants stay in its tree and are reaped. Termination waits
until no live descendant remains, and every scanned process is signalled
through a pidfd after confirming its start time, so a reused PID is never
signalled.
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* fix(sandbox): keep a frozen workload from resuming itself
Freezing stops every workload process with SIGSTOP. tgkill and
rt_tgsigqueueinfo were not mediated, and mediated kill, rt_sigqueueinfo,
and tkill passed SIGCONT through, so a workload process that was not yet
stopped could resume the others while the supervisor recovered.
Mediate tgkill and rt_tgsigqueueinfo like tkill: refuse targets in the
sandbox thread group, report a thread outside the named group as
missing, and continue otherwise. The boundary marks the broker frozen
before stopping the workload and clears it after resuming; while frozen,
workload requests to send SIGCONT fail with EPERM.
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* fix(sandbox): send mediated DNS datagrams from the broker socket
The native UDP send path re-ran the workload's own sendmsg, so
per-message ancillary data such as IP_PKTINFO rode along; only the
loopback destination contained a routing override.
The broker now reads the datagram and sends it from its retained,
relay-connected socket, which it builds without ancillary data, so a
per-message override cannot redirect the packet. It writes nothing back
into workload memory. A send carrying control data is refused with
EOPNOTSUPP rather than silently stripped.
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* fix(sandbox): keep sandbox control variables out of the workload environment
The canonical process inherits the sandbox environment so the image's
own ENV reaches the workload, then removed only a denylist of credential
variables. Other variables in the reserved OPENSHELL_ namespace, such as
the serialized user environment and the log level, still reached the
workload.
Remove every inherited OPENSHELL_ variable before applying the declared
environment, and restore OPENSHELL_SANDBOX=1. The image's ordinary ENV
and declared variables are unaffected; declared variables cannot use the
reserved namespace.
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* fix(sandbox): keep /proc read-only for GPU workloads
GPU mode granted read-write access to all of /proc so CUDA's cuInit could
write thread names to /proc/<pid>/task/<tid>/comm.
Keep /proc read-only. Open mediation now serves a writable open of the
caller's own thread comm file: the broker opens it and injects the
descriptor, with no syscall continued. The kernel accepts a comm write
only from the target's own thread group, so a substituted path or reused
thread ID cannot rename another process's thread. Any other /proc write
is left to Landlock, which denies it.
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* fix(sandbox): keep the broker socket for connected UDP DNS sockets
Sending DNS datagrams from the broker's socket required the broker to
keep its copy, but the connect paths to the relay and to loopback still
released it. glibc connects the resolver socket and then sends A and
AAAA together with sendmmsg, which is mediated, so resolution failed.
Release the copy on those connect paths only for TCP. Add a regression
test for connect followed by sendmmsg that fails fast rather than
hanging.
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* fix(sandbox): keep slow loopback connects from stalling mediation
The loopback connect path held the registry lock while polling a TCP
connect for up to five seconds on the single notification dispatcher.
A workload connecting to a busy local listener stalled every other
mediated syscall, including opens and signals.
Connect a duplicate of the retained socket without holding the lock. A
nonblocking socket gets the native EINPROGRESS and the kernel completes
the handshake on the shared socket; a blocking socket waits on a bounded
worker thread.
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* fix(sandbox): contain network broker handler panics
A panic in a notification handler unwound the single broker thread while
the health flag still read healthy, so every later blocked workload
syscall hung until the sandbox was killed.
Run each dispatch under catch_unwind: a panicking handler fails only that
syscall with EIO and the broker keeps mediating. If the broker thread
ever exits, mark it unhealthy so dependent operations fail closed instead
of blocking.
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* fix(sandbox): reject ancillary data on DNS sends instead of broker-sending
Sending mediated DNS datagrams from the broker's own socket required
keeping a broker handle for each DNS socket's whole life, which regressed
real name resolution through reclaim and resource accounting that the
mock-based unit tests did not exercise.
Revert to continuing the kernel send, but refuse a send that carries
ancillary control data (msg_controllen != 0) at read time, so a
per-message routing override such as IP_PKTINFO cannot ride a mediated
DNS send. A loopback destination contains an override that races the
check. Full broker-side UDP mediation is left to a separate change with
deployment e2e.
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* fix(sandbox): address review findings across the hardening changes
Fixes:
- The blocking local-connect worker no longer toggles O_NONBLOCK on the
open file it shares with the workload; it waits with a plain connect.
- tgkill and rt_tgsigqueueinfo are notified only when they send SIGCONT,
so ordinary thread signals such as Go preemption stay in the kernel.
The frozen flag is re-read immediately before delivery.
- A shell redirect to the caller's own thread comm file works again;
O_CREAT and O_TRUNC are no-ops there and O_EXCL returns EEXIST.
- The teardown scan is a linear walk and fails closed when /proc cannot
be read.
Simplifications:
- Remove tests that need infrastructure outside the repo or prove
nothing: the topology harness tests, the static-server benchmark, the
direct-syscall accept test, the comm rename test without Landlock, a
serde-default test, and the test-only panic hook in setsockopt.
- Drop the unused Failed connect outcome, the redundant read-back after
binding to loopback, redundant OPENSHELL_SANDBOX settings, and stale or
duplicated comments; reuse the reserved environment prefix constant.
- Tighten the support matrix and OpenShift wording.
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* fix(cli): wait for input when piped exec stdin is nonblocking
Processes that inherit the same stdin share one open file description,
so any of them can make it nonblocking for all. `sandbox exec` then read
no input yet, got EAGAIN, and failed with "Resource temporarily
unavailable (os error 11)". Parallel e2e tests inherit the runner's
stdin, which made credential_gating fail intermittently.
The stdin reader now blocks in poll() until input or end of file arrives
instead of treating EAGAIN as fatal, and it leaves the shared descriptor's
flags alone. The e2e exec helper also stops inheriting the runner's
stdin, since its commands take no input.
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* fix(sandbox): fix macOS lint and older-glibc test linking
Import HashSet only in the Linux-only process scan, and call gettid
through the raw syscall in a test, since the libc wrapper needs glibc
2.30 or newer.
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* docs(sandbox): clarify socket peer address reporting
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
---------
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* feat(providers): compose dynamic credential grants per request
Select the most-specific grant independently for each protected header,
preserving each credential's issuer, audience and cache identity. Resolve
all selected grants before rewriting the request, replace agent-supplied
header copies, and fail closed on collisions or acquisition failures.
Allow distinct-header compositions in gateway validation and redact raw
issuer errors. Cover independent cache entries, atomic TLS relay failure,
header replacement and concurrent request isolation; document the profile
contract.
Fixes#3320
Signed-off-by: Shiju <shiju@nvidia.com>
* fix(providers): correct dynamic grant CI checks
Use if-let for a grant result whose error payload is deliberately ignored,
and name the empty query map type in the regression fixture. Preserve grant
acquisition, failure redaction and request atomicity. Update the token-exchange
failure test to require the sanitized error instead of raw issuer text.
Signed-off-by: Shiju <shiju@nvidia.com>
---------
Signed-off-by: Shiju <shiju@nvidia.com>
* fix(podman): create managed workspace volumes owned by the workload identity
Podman now creates the managed /sandbox volume with uid/gid options for the
resolved workload identity, so the workload starts directly as that
identity. This fixes rootful sandboxes whose image USER or policy
run_as_user could not write to a root-owned /sandbox, and removes the
root-then-drop workspace chown start path.
Resource admission accepts the managed workspace volume when its options
match the workload container's final identity, or are empty for volumes
created by older gateways. The channel volume still requires empty options.
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* test(podman): cover managed volume reuse and workspace access
Signed-off-by: Evan Lezar <elezar@nvidia.com>
* refactor(podman): clarify managed volume creation and validation
Signed-off-by: Evan Lezar <elezar@nvidia.com>
* test(podman): verify workspace access across user namespaces
Signed-off-by: Evan Lezar <elezar@nvidia.com>
---------
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Evan Lezar <elezar@nvidia.com>
Co-authored-by: Evan Lezar <elezar@nvidia.com>
* fix(vm): enforce the configured workload identity
Reject conflicting policy users and groups before VM image preparation
and before guest attach or process startup changes state. Validate
supervisor policy updates against the protected VM workload identity.
Preserve the gateway CA transport and capability-free sandbox launcher.
Signed-off-by: Shiju <shiju@nvidia.com>
* test(sandbox): clarify VM identity rejection fixtures
Name invalid user and group fixtures distinctly and move the final
workload identity into its group mismatch test.
Signed-off-by: Shiju <shiju@nvidia.com>
* fix(supervisor): align VM identity startup with current APIs
Pass the optional rejection-log key for VM identity failures and keep
generic startup-write regressions free of VM identity constraints.
Repair the call sites after the branch rebase so the identity and cleanup
proposals compile against the current startup helpers.
Signed-off-by: Shiju <shiju@nvidia.com>
* fix(vm): restore inactive sandbox workload identity
Recover the persisted overlay owner before publishing stopped and terminal
sandboxes. Keep resources manageable when identity metadata is invalid.
Clarify fixed MicroVM ownership in policy-generation guidance.
Signed-off-by: Shiju <shiju@nvidia.com>
* test(vm): flush identity fixture before restart
Persist the canonical identity file before readiness and report the observed
exec, canonical and file-owner identities before comparing them.
Signed-off-by: Shiju <shiju@nvidia.com>
---------
Signed-off-by: Shiju <shiju@nvidia.com>
* fix(providers): restore supervisor-backed GCP metadata discovery
Relay the reserved metadata endpoint to the supervisor and restore project, account, and placeholder token responses from live provider state. Cover Google SDK discovery and repeated refresh with provider E2E tests.
Closes#3860
Signed-off-by: Philippe Martin <phmartin@redhat.com>
* fix(providers): preserve default metadata account without an email
Use the default account identifier when the optional service account email is missing or empty. Cover repeated SDK refresh for missing, empty, and configured email values with distinct providers for parallel E2E execution.
Signed-off-by: Philippe Martin <phmartin@redhat.com>
* test(sandbox): fix metadata relay lint and timeout
Signed-off-by: Philippe Martin <phmartin@redhat.com>
---------
Signed-off-by: Philippe Martin <phmartin@redhat.com>
* test(python): synchronize interactive exec TTY readiness
Wait for the complete readiness marker before streaming stdin so PTY echo cannot split the separately written TTY flags. Preserve pipe stream separation and verify consumed stdin and both output sentinels in TTY mode.
Fixes#4075
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* test(python): reuse interactive exec readiness marker
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
---------
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Start driver cleanup after terminal finalization and retain disconnect fallback. Add detached success and failure e2e coverage across supervisor-based drivers.
Closes#3938
Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>
679b19067 added global.image.registry (ghcr.io/nvidia) as the fallback for
empty per-image registries and split e2e image references into registry and
repository. Locally built images such as openshell/gateway:<tag> have no
registry host, so the chart rewrote them to ghcr.io/nvidia/openshell/* and
the k3d cluster could not pull them. Clear global.image.registry in the
Kubernetes e2e wrapper, which sets every image's registry explicitly.
Signed-off-by: Kris Hicks <khicks@nvidia.com>
* test(podman): move podman_preflight into driver-podman integration tests
podman_preflight verifies that openshell-driver-podman fails fast when
its Podman socket is unreachable. It only needs the standalone driver
binary, not a gateway, so it never fit the gateway-backed e2e-podman
harness it lived under and never ran anywhere in CI.
Move it into crates/openshell-driver-podman/tests/ as a plain Cargo
integration test. It now runs via the existing required workspace test
job with no special mise task, workflow step, or coverage exception.
Signed-off-by: politerealism <burdcat17@gmail.com>
* test(podman): make preflight diagnostics portable
Signed-off-by: Evan Lezar <elezar@nvidia.com>
---------
Signed-off-by: politerealism <burdcat17@gmail.com>
Signed-off-by: Evan Lezar <elezar@nvidia.com>
Co-authored-by: Evan Lezar <elezar@nvidia.com>
* fix(e2e): stop sandbox leaks from async Drop cleanup
Closes#2922
SandboxGuard::Drop spawned a detached thread to delete the sandbox.
The thread got killed with the test process before the delete
finished. Switch to a blocking command in Drop, like ManagedCleanup
already does. Also wrap two tests' manual cleanup in RAII guards so
a panic does not leak a sandbox.
Signed-off-by: Eric Curtin <eric.curtin@docker.com>
* test(e2e): arm sandbox guards before create
Address review: install guards with explicit names first.
Signed-off-by: Eric Curtin <eric.curtin@docker.com>
---------
Signed-off-by: Eric Curtin <eric.curtin@docker.com>
* fix(sandbox-backend): sort boundary request objects before hashing
Sort boundary request objects recursively before hashing so serde_json's
preserve_order feature cannot change digest identity. Cover canonical
bytes, envelope round trips, and rejection of modified provider values
and operations.
Signed-off-by: Shiju <shiju@nvidia.com>
* feat(mcp): upgrade tower-mcp-types to 0.22.2
Upgrade tower-mcp-types from 0.12.0 to an exact-pinned 0.22.2 and use its
inspection APIs to validate MCP requests against the selected revision.
Carry inspection metadata into policy evaluation and validate requests
after header rewriting, before forwarding.
Add explicit support for the sessionless 2026-07-28 revision while keeping
2025-11-25 as the default. Validate per-request metadata and standard HTTP
header mirrors, and support discovery, tools, and subscription requests.
Delegate batch availability and parameter schemas to Tower. Share typed
request names between policy and HTTP checks, retain the local batch
resource cap, and centralize MCP policy version parsing and ordering.
Keep supported MCP revisions and shared allowlist parsing in the canonical
policy schema; core re-exports those types. Tower owns wire-profile
semantics, and every supported policy revision must map to the matching
inspector profile.
Reject duplicate JSON keys, invalid known-method parameters, unavailable
methods, and unsupported batches. Keep exact extension allow rules and
deny precedence. Document request inspection boundaries and add unit,
forwarding, and sandbox coverage.
Refs #2174.
Signed-off-by: Shiju <shiju@nvidia.com>
* test(mcp): prove authorization at the forwarding boundary
Cover March batch denial in both member orders, valid and malformed
controls, and audit behavior across both relay entry paths. Exercise real
middleware tool rewrites with matching metadata and assert the exact
upstream representation or zero forwarded bytes.
Verify legacy bodyless SSE GET remains usable while GET tool bodies and
unsupported DELETE cleanup are rejected. Clarify request-selected profile
and middleware mutation comments without changing production behavior.
Signed-off-by: Shiju <shiju@nvidia.com>
* test(mcp): exercise permitted profiles through the sandbox proxy
Cover March and June singleton policies and select November and July
separately under one endpoint allowlist. Capture upstream tool receipts
to distinguish proxy policy denial from an upstream rejection.
Extend middleware rewrite coverage to June and multi-version policies,
and preserve the sessionless discovery and subscription checks through
the shared fixture helpers.
Signed-off-by: Shiju <shiju@nvidia.com>
* test(kubernetes): box the admission check future
Keep the admission test future below Clippy's size limit when the
workspace dependency features are unified.
Signed-off-by: Shiju <shiju@nvidia.com>
* test(mcp): reuse the forwarding fixture identity cache
Share the binary identity cache across protocol-profile cases, matching
the proxy lifecycle and avoiding repeated hashes of the test executable.
Keep procfs authorization and all forwarding assertions intact.
Signed-off-by: Shiju <shiju@nvidia.com>
---------
Signed-off-by: Shiju <shiju@nvidia.com>
The /_ws_tunnel endpoint pipes a WebSocket into the full gRPC service. On a
plaintext loopback gateway any web page could open it, since browsers do not
apply CORS to WebSocket upgrades. Mount the tunnel only when
enable_websocket_tunnel is set (config file, --enable-websocket-tunnel, or
OPENSHELL_ENABLE_WEBSOCKET_TUNNEL; server.enableWebsocketTunnel in Helm).
BREAKING CHANGE: gateways behind an authenticating edge proxy must set
enable_websocket_tunnel = true for CLI edge-tunnel connections.
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* fix(policy): propose rules for unknown DNS hosts
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
* docs(policy): clarify synthetic DNS use across protocols
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
* fix(policy): harden unknown-host DNS observations
- Emit the policy_dns_ineligible denial for every unknown name and
report observation staging failures as DNS failure events.
- Refuse unknown names during fail-closed quarantine and after the
observation budget, now a quarter of each address family's pool.
- Pin transparent TCP to the mapping of the deciding policy generation
so a reload between DNS and authorization fails closed.
- Stop Docker workloads from inheriting host DNS search domains, which
let the first expanded short name claim an observation address.
- Share mechanistic draft polling in conformance, register
new-hostname-proposal in the installed suite, and update docs.
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
* test(policy): build policy DNS proxy tests on every target
The proxy tests name PolicyEndpointId, which proxy.rs imported only on
Linux, so the macOS test build failed. Import it for test builds too.
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
* fix(policy): name DNS queries and mapped hosts in OCSF denials
DNS denial and failure events attached port 53 to the queried name,
which read as a connection to that host. They now carry only the name.
Transparent TCP denials for a policy DNS address show the mapped
hostname and keep the synthetic address in dst_endpoint.ip.
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
---------
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
* test(podman): run driver-podman userns suite against rootful Podman too
The driver-specific-integration job only ran the driver-podman testsuite
(default/auto/keep-id/private userns reference checks) against
fedora-podman-rootless, leaving rootful behavior for this scenario
unverified even though the compute driver auto-detects and explicitly
supports rootful Podman.
The default-userns-baseline and userns-profile playbooks hard-asserted a
rootless tmachine gateway user, so pointing them at a rootful environment
would have failed that assertion immediately rather than exercising
anything. They now detect rootful vs. rootless via the existing
tmachine_container_runtime role and branch the reference-capture user
accordingly, while keeping the captured reference file itself owned by
tmachine, since the archived test binary that reads it back always runs
unprivileged as tmachine regardless of daemon mode.
Signed-off-by: politerealism <burdcat17@gmail.com>
* test(podman): add real-daemon coverage for resource limits and daemon failure
Neither the Podman driver's resource-limit enforcement nor its behavior
when the Podman daemon is unreachable had any test coverage against a
real daemon; both were only exercised through unit tests against a
mocked Podman client.
podman_resource_limits.rs creates a sandbox with --cpu/--memory flags and
reads /sys/fs/cgroup/memory.max and cpu.max from inside the sandbox
itself, verifying the limit is actually enforced rather than just echoed
back by the template API. Expected values are cross-checked against the
driver's own parse_cpu_to_microseconds/parse_memory_to_bytes and against
a real local `podman run --cpus/--memory` container.
podman_preflight.rs spawns the standalone openshell-driver-podman binary
against a guaranteed-nonexistent Podman socket and asserts it exits
non-zero within its bounded retry window with an actionable error naming
the socket path, rather than hanging or failing silently.
Signed-off-by: politerealism <burdcat17@gmail.com>
* test(podman): make rootful userns and cgroup checks pass
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* test(podman): match lifecycle containers by isolation role label
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* test(podman): accept non-expiring bootstrap tokens
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
---------
Signed-off-by: politerealism <burdcat17@gmail.com>
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
Co-authored-by: Drew Newberry <anewberry@nvidia.com>
* fix(podman): restore host gateway alias mediation
Signed-off-by: Gordon Sim <gsim@redhat.com>
* fix(podman-e2e-tests): enable broader test podman e2e coverage
Signed-off-by: Gordon Sim <gsim@redhat.com>
* fix(tests): make test more reliable
Signed-off-by: Gordon Sim <gsim@redhat.com>
* fix(podman): fix macos linting error
Signed-off-by: Gordon Sim <gsim@redhat.com>
---------
Signed-off-by: Gordon Sim <gsim@redhat.com>
Replace Bash 4+ parameter expansion (${VAR,,}) with tr-based
lowercasing so the script works on systems with older shells.
Signed-off-by: Florent Benoit <fbenoit@redhat.com>
* fix(driver-kubernetes-secrets)!: store provider credentials in one namespace
The Kubernetes Secrets credential driver now stores every credential in its
configured namespace in all workspace modes and rejects handles that reference
any other namespace before contacting the Kubernetes API. The gateway reaches
credential Secrets through the Role in that namespace; this allows removing the
Secret rules from the ClusterRole.
- Remove the workspace_mode, gateway_id, and allow_reference_namespace driver
settings and stop rendering them from Helm. Configurations that set them fail
at startup. Existing credential state is not migrated.
- Add server.credentialDrivers.kubernetesSecrets.createNamespace to provision a
dedicated credential namespace. The namespace is kept on uninstall, adopted
by a reinstall of the same release, and left untouched when something else owns
it.
- Update the gateway config reference, Kubernetes setup docs, 0.1.0
upgrade guide, compute-runtime architecture, and cluster debugging skill.
Signed-off-by: Kris Hicks <khicks@nvidia.com>
* fix(helm): reduce gateway Secret permissions
Remove the gateway's Secret list permission in every workspace mode and
grant source Secret reads through a Role in the sandbox namespace.
Bootstrap Secret cleanup deletes Secrets by exact name instead of listing
them.
- Grant get on the copied client TLS and image-pull Secrets through a Role in
the sandbox namespace. The ClusterRole keeps get and patch on those names
for the ownership check and server-side apply into workspace namespaces.
- Delete sandbox and supervisor bootstrap Secrets by exact name, derived from
the runtime generation recorded on the Sandbox and, on restart, the target
generation, tolerating 404. The generation annotation is cleared only after
cleanup succeeds, and each bootstrap Secret has a Pod owner reference, so
garbage collection removes any generation the driver does not name.
- Drop Secret list from the ClusterRole and the shared-mode sandbox Role.
- Extend the managed e2e RBAC checks to Secret list.
- Update the Kubernetes setup and sandbox runtime docs, compute-runtime
architecture, and cluster debugging skill.
Signed-off-by: Kris Hicks <khicks@nvidia.com>
* fix(driver-kubernetes): stage workspace Secrets per runtime generation
Every Secret the Kubernetes driver writes into a workspace namespace is
now scoped to one sandbox runtime generation, immutable, and created
with create only, so the gateway never reads, patches, or adopts an
existing Secret there. This removes the gateway's cluster-wide get and
patch on the copied client TLS and image-pull Secret names.
- In managed mode, create an immutable copy of each configured
image-pull Secret per generation, named os-pull-<id>-<generation>-<n>
and owned by the generation's workload and supervisor Pods. Pods and
the restarted Sandbox template reference those names, and generation
cleanup deletes them by name. A Secret already holding a generation
name fails the create.
- Outside shared mode, stage the gateway client TLS material into the
supervisor bootstrap Secret instead of copying the client TLS Secret
into the workspace namespace.
- Remove the fixed-name TLS and image-pull copies, the target ownership
read, and the ClusterRole get and patch rule on the copied names.
Source reads stay in the sandbox-namespace Role.
- Update the managed e2e to expect generation image-pull Secrets and
client TLS material in the supervisor bootstrap Secret, and to check
that the gateway cannot read the copied names in workspace namespaces.
- Update the gateway config and compute driver references, Kubernetes
setup and sandbox runtime docs, compute-runtime architecture, driver
README, and cluster debugging skill.
Signed-off-by: Kris Hicks <khicks@nvidia.com>
* fix(helm)!: grant operator-mode Secret permissions through the workspace chart
The operator-mode gateway ClusterRole grants no Secret permissions.
The openshell-workspace chart Role, installed in each operator-managed
namespace, grants the gateway create and delete on Secrets for sandbox
runtime generations. Operator-managed namespaces require the workspace
chart.
- Fail the chart tests on any ClusterRole rule that includes Secrets in
operator and shared modes.
- Install the workspace chart when the operator e2e provisions a
namespace, and assert that the gateway has no Secret permissions in a
namespace without it.
- Update the Kubernetes setup docs, 0.1.0 upgrade guide, compute-runtime
architecture, and cluster debugging skill.
Signed-off-by: Kris Hicks <khicks@nvidia.com>
---------
Signed-off-by: Kris Hicks <khicks@nvidia.com>
* fix(security): pin supplied executable identity chains
Signed-off-by: Jim Meyer <jimeyer@nvidia.com>
* docs(security): document executable identity chain pinning
Signed-off-by: Jim Meyer <jimeyer@nvidia.com>
* fix(binary-identity): compile Linux ancestry hashing
Signed-off-by: Jim Meyer <jimeyer@nvidia.com>
* test(binary-identity): avoid cross-label ancestry fixture
Signed-off-by: Jim Meyer <jimeyer@nvidia.com>
* test(e2e): choose distinct denied TCP port
Fix flaky test due to sequential port assignment on MacOS
Signed-off-by: Jim Meyer <jimeyer@nvidia.com>
* fix(identity): bound executable evidence cache
Reject identity chains atomically when the supervisor cache reaches its hard limit, and preserve existing pins without eviction. Classify malformed or conflicting evidence separately from policy denials at the staged TCP boundary.
Signed-off-by: Jim Meyer <jimeyer@nvidia.com>
---------
Signed-off-by: Jim Meyer <jimeyer@nvidia.com>
The credential driver e2e has not passed end to end, and the disabled
kubernetes-credential-drivers CI lane hid three problems.
The test broke when the JSON format of provider list changed.
Continuation-token pagination (#3249) changed
provider list --output json from a bare array of providers to an object
with next_page_token and a providers array. The test still parsed the
output as an array, so it failed before checking either storage
backend. Read the providers array from the new object instead.
Its sandbox name was about 58 characters, but sandbox names are
DNS-routable and limited to 19, so sandbox creation was rejected. Build
a short unique name instead.
The sandbox guard deletes its sandbox from a detached thread on drop, so
the test deleted the provider while the sandbox still existed. The
gateway rejects deleting a provider that is attached to a sandbox, the
test ignored that error, and the credential Secret remained. Delete the
sandbox explicitly before returning from the sandbox check.
Signed-off-by: Kris Hicks <khicks@nvidia.com>
John T. Myers