Commit Graph
336 Commits
Author SHA1 Message Date
Jesse Jaggars 2c31ac7e60 feat(network): support additional destination CAs (#3292)
* feat(network): support additional destination CAs

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* fix(network): harden additional CA propagation

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* fix(network): narrow additional CA review scope

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* fix(network): harden additional CA rotation

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* feat(network): complete additional CA delivery

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* fix(docker): remove stale runtime fixture field

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* fix(network): adapt additional CA series to upstream main

Resolve rebase fallout against current upstream main after the series was
rebased 214 commits forward:

- follow upstream #3110: supervisor→gateway TLS is CA-only, so
  configure_gateway_tls and its isolation tests drop the client identity
- follow upstream tls_secret_names reduction to [String; 1] in the Podman
  driver while keeping the network trust bundle delivery
- port [openshell.supervisor.network] rendering into the
  openshell.gatewayConfigToml serializer in _toml.tpl
- adapt tests to upstream struct/function changes (DriverStartupContext,
  SandboxRunConfig, IsolationSpecInput, supervisor_pod, VM bootstrap image)
- drop conformance-artifact-prefix plumbing removed upstream

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* refactor(network): adopt upstream additional CA bundle surface

Unify the additional destination CA series on the mechanism merged to
main in #3996: one operator pre-assembled PEM bundle, the
--additional-ca-bundle supervisor flag, and the
/etc/openshell/tls/additional/ca-bundle.pem staging path.

- Replace the gateway-level additional_ca_cert_paths list with a single
  additional_ca_bundle file validated and staged at gateway startup.
- Restore the upstream supervisor argument path and fold-into-roots
  behavior; drop the --network-additional-ca-bundle and
  --network-additional-ca-digest contracts.
- Keep gateway normalization bounds, content-addressed staging,
  digest-based driver reconciliation, and stop/start rotation; the
  freshness check now runs at the gateway/driver boundary instead of
  the supervisor boundary.
- Reject additional_ca_bundle in operator driver JSON config; standalone
  drivers self-source the bundle from --sandbox-additional-ca-bundle
  with the same validation.
- Extend the Kubernetes ConfigMap name digest segment to keep the full
  63-byte budget after the prefix rename.
- Box the persisted-sandbox reconciliation futures so callers stay
  under the clippy::large_futures threshold.

Updates Docker, Podman, Kubernetes, and VM delivery, Helm chart and
tests, e2e wiring, gateway configuration docs, and the cluster
debugging skill to the single mechanism.

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* fix(ci): make gateway startup tests wrap-tolerant and refresh example lockfile

The network trust startup tests asserted on raw substrings of gateway
diagnostics, but the CLI renderer wraps output at a fixed width with
continuation markers and hard-breaks long paths. Compare assertions on
whitespace- and continuation-squashed text so they hold regardless of
wrap position.

Format the e2e workspace (separate cargo workspace, not covered by the
root cargo fmt run) and refresh examples/isolation-backend-v8/Cargo.lock
for the libc dependency added to the network trust crate.

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* fix(podman): reuse TLS CA secret during stopped-pair trust replacement

The additional CA e2e never ran in CI before the first maintainer test
dispatch, and its local container inspection was broken in two ways: it
located the supervisor container through e2e network membership (the
supervisor is host-networked or runtime-bridged, so the filter matched
the CI job container instead), and it required the gateway mTLS CA as a
bind mount (Podman delivers it as a container secret when user
namespaces remap UIDs). Select the supervisor by managed labels and
accept the secret delivery form.

The lifecycle leg then exposed a driver bug: stopped-pair trust
replacement re-created the per-sandbox TLS CA secret, colliding with
the secret still owned by the renamed stopped pair. Mirror the token
and resolver secret handling: require and reuse the existing secret
during replacement instead of creating it.

Validated end-to-end with the Podman additional CA e2e lane.

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* fix(server): restore upstream recovery semantics for start-request failures

Recovery no longer marks sandboxes Error/InvalidPersistedSandbox when
driver_start_request fails during the startup or lifecycle sweeps.
Gateway configuration problems (for example a missing launch signer)
must not permanently fail unrelated sandbox records; the sweep logs a
warning and skips them, matching the behavior before the additional CA
work. Sandbox-record failures surface through the normal start path.

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* refactor(podman): refresh stale supervisors without replacing workloads

Replace the stopped-pair trust reconciliation with supervisor-only
recreation. The destination-trust generation label moves from the
workload to the supervisor container, and start_sandbox recreates only
the supervisor when its generation is missing or stale. The workload
container, its volumes, its secrets, and its workspace data are never
replaced: the additional CA bundle is mounted only into the supervisor.

Running sandboxes are never gated on trust labels, and recovery without
fresh launch credentials keeps the existing supervisor. The intentional
-removal fence machinery in the watcher existed only for workload
replacement and is removed with it.

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* refactor(docker): drop workload replacement for destination-trust changes

Delete the stopped-workload trust reconciliation path and all of its
machinery: the trust-generation label on workload containers, the
generation match check in the start fence, the stopped-sandbox
replacement, the bounded workspace archive copy with its 512 MiB cap
and regular-file-only validation, and the boundary-state rollback
helpers.

No replacement machinery is needed: the trusted supervisor container
is already re-created from the gateway's current configuration on
every start via ensure_control_process_for_container, so stop/start
picks up the current additional CA bundle while the workload
container, its volumes, and its workspace data are never replaced.

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* docs(configuration): drop reconciliation wording from additional CA lifecycle

The stop/start lifecycle description already matches the
supervisor-only refresh model; remove the two remaining references to
restart reconciliation of workload containers.

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* fix(network): port additional CA test fixtures to the crypto interface

Route the gateway TLS isolation test PKI, the destination CA source
fixtures, and the gateway startup trust tests through
openshell_crypto::pki now that rcgen access is centralized there:
generate_keypair/self_signed/signed_by replace the direct rcgen calls
and CertifiedKey field access follows the wrapper's shape.

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* fix(e2e): repair additional CA lane for Docker and non-container builds

The first maintainer test dispatch revealed two latent issues in the
additional CA e2e that no local run had exercised: the Podman gateway
CA secret helper referenced ContainerEngine without the container
feature gate, breaking Kubernetes and VM builds of the test, and the
Docker branch asserted a client ca.crt bind mount that the Docker
driver never creates. Gate the helper like its callers, and assert
Docker's actual delivery path instead: the gateway mTLS CA staged as
tls/ca.pem inside the read-only supervisor state volume, selected
through the OPENSHELL_TLS_CA environment pointer.

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* fix(helm): restore alphabetical README ordering for supervisor keys

The helm:docs check regenerates the chart README with helm-docs, which
orders rows alphabetically; a conflict resolution had placed
supervisor.network.additionalCaConfigMapName after the
supervisor.podSecurityContext rows. Regenerate with mise run
helm:docs.

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* chore: refresh testkit lockfile for openshell-core libc dependency

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* fix(driver-podman): stage operator CA bundle as driver-owned artifact

Standalone Podman drivers accepting --sandbox-additional-ca-bundle
previously kept the operator's source path as the trust artifact path.
The immutable-artifact verification performed on every create/start
requires exactly mode 0444, which an operator-owned source path cannot
guarantee, so a readable-but-writable source failed reconciliation.

Read the operator bundle with the same bounded, no-symlink validation,
then stage the normalized material into a driver-owned artifact inside a
private tempdir held for the driver process lifetime. The artifact is
created 0600, synced, and only then restricted to 0444 so no observer
ever sees a writable artifact. The operator source and its permissions
are never referenced after startup.

Addresses GATOR-291f43b2-02.

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* fix(driver-podman): pin retained workload image during supervisor refresh

Supervisor-only trust reconciliation re-entered
create_sandbox_with_existing_token, which re-pulled and re-resolved the
workload image tag even though the workload container is retained. If
the tag moved to a different image between stop and start, identity
resolution and child-environment construction would reflect the new
image while the retained workload still runs the original one, leaving
mode-0600 bootstrap files the retained user cannot read.

During a refresh with a retained workload, inspect the stopped container
and pin to the immutable image ID it was created from (Config.Image);
never pull or resolve the request tag. The requested reference is still
used for child metadata so the retained workload's environment is
byte-identical to its original launch.

Addresses GATOR-291f43b2-01.

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* test(e2e): verify host-only VM trust delivery and drop curl dependency

The additional-CA lane's VM assertions read the destination CA out of the
guest overlay and a guest-side supervisor-argv marker — artifacts from
the guest-supervisor design that no longer exist now that the VM driver
stages the CA host-only and runs the supervisor on the host. Inspect the
driver-staged host artifact (digest-compared against the source), assert
the guest overlay contains no CA material at the old mount paths, and
keep the serial-console no-disclosure check.

The workload validation scripts also ran curl, which exists only in the
container-engine fixture image; the Kubernetes and VM lanes use the
uv-based workload image and died instantly with "canonical main process
already finished". Rewrite the main-process, exec, and lifecycle
validation scripts in python3 (present in every lane's workload image).
The network-policy binary identity is the resolved /proc/<pid>/exe
target, so the versioned python3.12 binaries are allowlisted rather
than the unversioned symlinks. The hostname mismatch check keeps its
semantics: same reachable fixture, mismatched TLS SNI, and the failure
must specifically be hostname verification.

Addresses GATOR-291f43b2-04.

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* fix(kubernetes): allow admission-added labels on destination-trust ConfigMaps

The destination-trust ConfigMap validation compared the whole label map
for equality. A mutating admission webhook that attaches an
organizational label to the created ConfigMap made the create response
fail validation, and the retained ConfigMap then failed every subsequent
create/start through the existing-object path, blocking sandbox launches
despite the ownership labels, immutable flag, digest, and certificate
bytes remaining correct.

Validate the gateway-owned label pairs individually and allow unrelated
labels; the exact trust-material checks are unchanged. Covered by a
regression test with an admission-added label and a missing
gateway-owned label.

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* fix(podman): close supervisor-refresh rollback gaps

The supervisor-only trust refresh renamed the previous supervisor to a
random rollback name and then performed the token-secret lookup. A
transient lookup failure returned with the supervisor still renamed,
where a normal delete could never find it, and the rollback rename and
final rollback removal ignored their own failures, leaving orphans with
attached control and channel resources.

Complete every fallible preparation before the rename. On a failed
replacement, restore the canonical name; if the restore itself fails,
return an error naming the rollback container so the orphan stays
identifiable — a later retry still succeeds because the canonical name
is absent and the replacement path recreates a fresh supervisor. Treat
not-found during the final rollback removal as success and name the
orphan in both the log and the error on other removal failures; a retry
observes the fresh canonical supervisor and resumes the normal start
path.

Covered by failure-injection tests for each transaction boundary:
preparation failure before any rename, replacement failure restoring
the canonical name, and restore failure naming the orphaned rollback
container.

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* docs: address destination-trust review corrections

The gateway-configuration RFC documented an additional_ca_cert_paths
list that the strict configuration parser never accepted; document the
additional_ca_bundle single-path setting the parser actually enforces.

The debug-openshell-cluster skill named the wrong generation annotation
(openshell.ai/network-supervisor-trust-generation) and described digest
argv and network-init canonicalization checks the implementation does
not perform. Correct it to openshell.ai/additional-ca-generation and
describe the actual delivery: the supervisor Pod mounts the immutable
ConfigMap read-only and receives --additional-ca-bundle for the mount
path, with the driver validating the digest annotation and exact bytes
per creation.

Classify the destination-trust generation hashes as deterministic
identity derivation rather than authenticity proofs, following the
openshell-crypto exclusions table: trust decisions come from the
gateway's normalization, immutable staging, exact-byte validation, and
TLS verification. Record the classification in the crypto README and at
each hashing site.

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* fix(network): reconcile additional CA tests with suspend/resume and auth changes

Upstream's suspend/resume lifecycle took StartSandboxRequest fields 7
and 8, so the durable-snapshot field moves from 7 to 9; update the
wire-format test to assert the field-9 key (0x4a) and initialize the
new resume controls. Repair the cli test module after the auth opt-in
test merged alongside the environment-guard helpers, and initialize the
gateway driver-startup context's network-trust fields in the shared
supervision test.

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

* fix(server): box start-recovery futures for clippy large-futures

The durable-snapshot field grows StartSandboxRequest enough that the
start-recovery match and the recovery-expiry test exceed the
large-futures limit under -D warnings. Box::pin both futures, matching
the pattern the recovery path used before the suspend/resume rework.

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>

---------

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>
2026-10-27 18:31:11 +00:00
Drew Newberry e5adf6f484 feat(supervisor): recover shared instances with hash-based HA (#4864)
* feat(supervisor): recover shared instances with hash-based HA

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* refactor(supervisor): keep HA coordination out of compute drivers

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(supervisor): fence pool admission and reconciliation races

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* feat(supervisor): keep pool mode stable across replica scaling

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* refactor(supervisor): default shared deployments to pool mode

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(supervisor): fence refused launches and remove legacy mode

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(supervisor): confirm live owners and fence stale observations

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(supervisor): preserve takeover cause log formatting

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* feat(helm): add memory-based supervisor autoscaling

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(supervisor): retain acknowledgements and fence streamed input

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* docs(supervisor): consolidate experimental multi-tenant guidance

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* docs(supervisor): simplify multi-tenant supervisor guide

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* refactor(supervisor): simplify pool member identity

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* feat(helm): make supervisor affinity configurable

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* feat(helm): make supervisor disruption budget optional

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* test(supervisor): align HA fixtures with suspend support

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

---------

Signed-off-by: Drew Newberry <anewberry@nvidia.com>
2026-10-27 04:28:39 +00:00
Simon Scatton 9c6c25204c fix(deps): update tokio and crypto-bigint security patches (#4924)
* fix(deps): update tokio and crypto-bigint security patches

Signed-off-by: Simon Scatton <sscatton@nvidia.com>

* chore(deps): document temporary crypto-bigint advisory exception

Signed-off-by: Simon Scatton <sscatton@nvidia.com>

* fix(sandbox): migrate owned descriptors to AsyncFd register

Signed-off-by: Simon Scatton <sscatton@nvidia.com>

* test(server): avoid deadlock in concurrent CAS retry test

Signed-off-by: Simon Scatton <sscatton@nvidia.com>

---------

Signed-off-by: Simon Scatton <sscatton@nvidia.com>
2026-10-26 14:04:16 +00:00
Simon Scatton bc4c8f0013 fix(deps): update ctutils to resolve timing advisory (#4867)
Signed-off-by: Simon Scatton <sscatton@nvidia.com>
2026-10-23 14:13:07 +00:00
Drew Newberry 9a85667d0d feat(kubernetes): host sandboxes on the shared supervisor (#4702)
* feat(sandbox-backend): route runtime connections to shared supervisor sessions

A supervisor shared by several sandboxes needs one listener for every
session's runtime-initiated Sandbox Protocol connections. The runtime
now sends a fixed, bounded routing preface (magic, version, length, and
an opaque route handle) before its pinned TLS handshake when its
bootstrap names a route. RuntimeRouter treats the handle only as a hint
that selects which generation-pinned TLS identity to demand, closes
unknown, removed, and replaced handles before any handshake, bounds
unauthenticated handshakes globally and per route, and queues an
authenticated connection only for its own session, so no peer can
displace a live channel. A routed session waits up to the bootstrap
grace for its first connection, because it is registered before its
workload Pod is scheduled.

The session_control module defines the trusted contract between a
compute driver and a shared supervisor: registering one generation's
runtime descriptor and gateway-issued credentials, reading its status,
and removing it, over mutually authenticated TLS 1.3.

Dedicated descriptors and bootstrap listeners keep their encodings.

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* feat(supervisor): host gateway-registered sessions in one process

openshell-supervisor experimental-host replaces the idle bootstrap
process with a session host. A mutually authenticated session-control
API accepts registrations only from the admission certificate holder,
the gateway. Each registration starts a SessionHost session that acts
only with the gateway credentials issued for its generation, keeps its
proxy CA, trust directory, and SSH socket under a private directory,
and receives its runtime connections through the routed listener.
Status reports each session's own readiness; the health socket keeps
reporting process health only.

Removing a session runs on a task the host owns, so it completes when
the caller disconnects, concurrent removers share its result, and the
route stays reserved until the session has ended and its files are
gone. An identical registration is accepted again and a different one
for a registered route is refused.

Sessions get no upstream proxy, extra CA bundle, or provider workload
identity; the Kubernetes driver refuses configurations that need them.

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* feat(server): issue a TLS-only private PKI from generate-certs

generate-certs --tls-only creates just a server and a client TLS Secret
from a fresh CA, with the same create-if-missing contract and a recovery
hint for partial state. The Helm certgen hook uses it for private PKIs
that must not carry a sandbox-JWT signing key.

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* feat(kubernetes): supervise gateway-topology generations as shared sessions

With experimental_supervisor_topology = "gateway", each new runtime
generation becomes a session on the gateway's shared supervisor instead
of a supervisor Pod and boundary Service of its own. The generation
records its placement and session route on its Sandbox resource when it
starts, and every later decision reads that record, so changing the
setting never reinterprets a running generation and unknown records fail
closed. The runtime identity ends with the session route, which no Pod
UID can equal, so Pod authentication cannot vouch for a shared
generation.

Create and start add a per-sandbox egress allowance to the Pods the
shared supervisor's Service selects, bind it into the boundary evidence,
and register the session over session control before releasing the
workload; registration does not wait for the runtime, so create still
returns before the gateway persists the generation. Readiness follows
the session's own status. Stop, restart, and delete remove only that
session, and reconciliation suspends a workload whose session is gone
and removes sessions that no current generation may run. Configurations
with an upstream proxy or provider SPIFFE identity are refused.

Closes #4700

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* feat(helm): deploy and wire the shared supervisor

With the gateway topology enabled, the chart runs the session host as a
single-replica Deployment with its Service and a NetworkPolicy that
admits session control only from the gateway and runtime connections
only from sandbox workload Pods. The certgen hook issues a private
session-control PKI whose client credential only the gateway mounts,
the gateway may read only that Service, and the chart renders the
matching experimental_shared_supervisor table. The supervisor Pod has no
service account token. Settings the shared supervisor cannot isolate,
and a hand-written connection table, fail at render time.

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* refactor(ocsf): attribute events to session-bound contexts

A supervisor process that hosts several sandboxes cannot use one
process-wide OCSF context: every event would carry the wrong sandbox
identity, and middleware requests read their sandbox identity from the
same context.

Builders now accept a ContextRef, either a borrowed context or a context
shared by a hosted session, so per-session contexts need not be leaked.
ctx() returns the context bound to the calling thread when present and
falls back to the process-wide context otherwise. OCSF JSON output
settings can likewise be bound per thread, and OpenTelemetry can parent
a span from an explicit W3C trace context instead of process
environment. A dedicated supervisor keeps using the process-wide
context.

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* test(e2e): run two sandboxes on one shared Kubernetes supervisor

The e2e:kubernetes:shared-supervisor lane enables the gateway topology
and creates two sandboxes with different policies and provider
credentials. It checks that one shared supervisor process hosts both,
that each session enforces its own policy and injects its own
credential, that stopping one leaves the other working, that the routed
listener refuses stale and unknown routes and drops a peer that names a
live route without its pinned identity, and that a restart registers a
fresh session on the same process.

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* refactor(supervisor): scope gateway credentials and network state to a session

The supervisor kept its gateway bearer, Sandbox Protocol bearer, and
renewal task in process-wide slots, and the network supervisor read
provider token grants, proxy CA files, and the gateway identity from
process environment. A second sandbox in the same process would have
replaced or rotated the first sandbox's credentials, reused its
provider access tokens, and exposed its denials through policy.local.

GatewayCredentials now holds one session's gateway and Sandbox Protocol
credentials and their renewal; every gateway client receives a
GatewayEndpoint that pairs the URL with them, including log push,
ConnectSupervisor and its redirects, and extension-credential rotation.
run_networking takes a NetworkSession with the sandbox identity, the
session's gateway access, its Workload API socket, its policy.local
denial source, and its proxy CA files. Provider token grants use a cache
and HTTP client owned by the session and fail closed without a Workload
API socket.

run_session supervises one generation from an explicit SessionConfig and
owner-provided SessionControls. run_sandbox_with_backend remains the
dedicated entry point: it keeps the process-wide OCSF context, reads the
same environment, binds readiness endpoints to session readiness, and
stops on SIGTERM or SIGINT. The binary shares one set of credentials
between log push and supervision, as before.

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* docs: describe the experimental shared supervisor

Document the gateway supervisor topology, its chart-owned connection
settings and limits, and how to debug and test it.

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* feat(supervisor): host concurrent sessions on session-bound runtimes

SessionHost runs several supervisor sessions in one process. Each
session gets its own Tokio runtime, and every thread that runtime starts,
plus the thread driving it, is bound to the session's OCSF context, OCSF
JSON settings, log sink, and denial log. Tasks, blocking work,
reconnects, and credential renewal spawned by a session therefore keep
its identity, while threads outside any session carry none and push no
logs. Gateway credentials stay explicit per session.

Removing a session stops its workload gracefully, or cancels it at once
while it is still starting, then shuts down only its runtime and revokes
its gateway credentials. The key stays registered until the session
thread has finished, so the same generation cannot gain a second owner
while one is stopping. A panicking session reports a failure, clears its
readiness, and is revoked the same way. Hosted sessions get a private
proxy trust directory and serve policy.local denials from their own log.

A focused test runs two sessions concurrently and checks that nested and
blocking tasks see their own identity, gateway requests carry their own
bearer, log lines reach only their own stream, and removing one leaves
the other supervising.

Closes #4699

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix: tighten shared supervisor naming, removal deadlines, and route probes

The shared supervisor's resource name appended its suffix before
truncating, so a release fullname of 51 or 52 characters produced the
gateway Service's own name. A fullname without room for the suffix now
keeps a prefix and a hash of the whole fullname.

Removing a session waits up to 20 s for its workload to stop and 5 s for
the session runtime to shut down, but the driver gave DELETE the same
15 s budget as other session-control calls and reported normal removals
as failures. The contract now defines the removal grace and a 45 s
removal deadline, and deleting a sandbox no longer waits on a session
whose workload is already going away.

The shared host runs no supervisor OTLP trace or OCSF exporter. The
driver warns at startup when the gateway configures that export, and the
docs state the limit. experimental-host keeps its Unix-only imports in
its Unix entry point.

The e2e route probe fails when the supervisor neither refuses nor
challenges a connection, or keeps a challenged connection open after a
malformed TLS answer, and it checks that every probe reached the
supervisor Pod.

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* refactor(supervisor): route shared supervisor crypto through openshell-crypto

Main now routes first-party TLS providers and digests through the
openshell-crypto facade. Build the session-control TLS configurations and
the routed-runtime test server from its configuration provider, install the
experimental host's default provider through it, and hash shared session
registrations with its SHA-256 instead of a direct sha2 dependency.

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* feat(supervisor): report shared supervisor health over TCP

experimental-host takes --health-port instead of a Unix health socket,
matching experimental-bootstrap. It binds the wildcard health listener
after the session-control and runtime listeners and accepts and closes
connections while the process runs. The chart probes the named container
port health (5502) with native kubelet tcpSocket checks; session control
stays on 5501, the Service exposes only runtime and session control, and
/run/openshell remains for per-session files.

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* refactor(supervisor): probe shared supervisor health on session control

The shared supervisor's kubelet tcpSocket probes now connect to the
mTLS session-control listener (5501) instead of a separate health
listener. Drop the experimental-host --health-port flag, its accept
loop, and the chart's 5502 container port. Probes open and close a
TCP connection; every session-control request still requires mTLS.

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* refactor(supervisor): use shared mode flags for the session host

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* refactor(supervisor): describe the shared host module by its mode

Drop the experimental label from the shared host module documentation
so it matches the --mode=shared entry point. No behavior changes.

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(test): preserve shared supervisor workspace overlays

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* refactor(supervisor): name hosted lifetimes sandbox instances

A sandbox instance is one running generation of a stable Sandbox
resource. Gateway transport connections and the gateway supervisor
sessions they carry can reconnect within one instance, so the hosted
object, its configuration, controls, host, and per-instance network,
denial, and token-grant state now use the sandbox instance name.

Rename session_host to sandbox_instance_host, run_session to
run_sandbox_instance, and SessionConfig, SessionControls, SessionHost,
and related hosted types to their SandboxInstance equivalents. Gateway
supervisor sessions, SSH sessions, and authentication session types
keep their names. No wire, configuration, or lifecycle behavior
changes.

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* refactor(supervisor): launch sandbox instances from one config and prepared bindings

Replace SandboxRunConfig and its field-by-field copy into
SandboxInstanceConfig with a single launch configuration. Live per-instance
state (gateway credentials, OCSF context and JSON settings, log sink, and
denial log) is now an explicit SandboxInstanceBindings value that the owner
prepares once, instead of placeholders the runner or host overwrote.

The dedicated binary loads launch inputs from its arguments and environment
and shares one GatewayCredentials with log push; run_sandbox_with_backend
keeps the process-wide OCSF context, health endpoints, and signal handling.
The instance host binds the same SandboxInstanceBindings to its runtime
threads and revokes their credentials when the instance ends.

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* refactor(supervisor): use session-control wire types for instance status and trace context

Drop the supervisor's duplicate SandboxInstancePhase, SandboxInstanceStatus,
and TraceContext families. The sandbox instance host reports
SandboxInstanceStatusReport directly, with its key as the route, so the
shared host no longer converts or re-keys status, lists, or removal
results. SandboxInstanceConfig carries RegistrationTraceContext and the
dedicated process reads it through trace_context_from_environment. No wire
definitions change.

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(supervisor): update Windows host proxy test config

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* refactor(supervisor): assign shared sandbox instances over a supervisor-initiated gRPC stream

Replace the private HTTP/JSON session-control API, which the gateway dialed
on the shared supervisor, with a supervisor-initiated stream to the gateway.
The gateway owns the desired set of sandbox instances; the shared supervisor
attaches with a gateway-scoped host certificate, receives complete assignment
snapshots with session-scoped revisions, reconciles its instance registry to
each one, and reports what it observes. Losing the stream never stops an
instance; only a later snapshot that omits it does.

- Add the private openshell.supervisor.host.v1 protocol, served on the gateway
  listener but authorized only for the supervisor host certificate, excluded
  from SDK generation and reflection, and classified in the schema inventory.
- Add SupervisorHostController in openshell-core: uninitialized until the
  driver derives the desired set once, compare-and-replace full resyncs,
  capacity bound, session fencing by connection epoch, revision-bounded acks,
  bounded sends.
- Classify TLS peers per connection against the ordinary client CA and the
  supervisor host CA from one configuration snapshot; a chain matching
  neither or both is refused, and host connections never become users.
- Persist each shared generation's launch material in its supervisor Secret so
  a restarted gateway rebuilds assignments from Sandbox resources, and
  withdraw instances on stop, delete, and suspend.
- Issue the host identity with `generate-certs --client-only`; the gateway
  mounts only the CA. Drop the control port, server certificate, and gateway
  ingress rule from the chart; require gateway TLS for the topology.

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* refactor(supervisor): authenticate the shared supervisor stream with a gateway-minted supervisor token

Replace the shared supervisor's host CA and client certificate with a
supervisor token: a bearer the gateway mints from its existing launch-signing
key under a profile of its own (typ openshell-supervisor+jwt, subject
openshell-supervisor:<gateway_id>, no sandbox, generation, or epoch claims).
The supervisor dials the gateway's Attach stream over ordinary
server-authenticated TLS and presents the token; the gateway verifies it
with the same key infrastructure, grants only the assignment stream, refuses
every other credential on it, and ends a stream when the token that opened
it expires.

The gateway renews the token at half its lifetime and publishes each renewal
into the supervisor host controller; the Kubernetes driver's only part in
authentication is delivery, replacing the token in the chart-created
<release>-supervisor-token Secret, which the supervisor mounts and re-reads
on every connection attempt. The gateway Role may get and patch only that
Secret.

Remove the host-only TLS realm classification, the supervisor host CA mount
and config, certgen --client-only, and the supervisor host PKI; gateway TLS
returns to its pre-feature behavior and remains required for the topology.

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* refactor(helm): remove redundant certgen condition

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* refactor(supervisor): extract gateway-owned shared supervision into openshell-supervisor-control

Move the shared supervisor control types out of openshell-core and
openshell-sandbox-backend into a new openshell-supervisor-control crate:
the SupervisorHostController (desired set, revision fencing, host
sessions, observations), the SandboxInstanceRegistration launch material
and SandboxInstanceStatusReport mapping, and a new SharedSupervision
coordinator that owns the control contracts a compute driver previously
implemented itself: validated persist-before-assign launches, recovery
from durable records with the compare-and-replace baseline captured before
the driver lists them, keep-previous-set on any failed or mismatched read,
and one interpretation of host observations for readiness and bootstrap.
Supervisor token delivery keeps its retry and newer-value policy in the
crate; the write itself stays with the driver.

The gateway now constructs the coordination: ComputeDriverBuildContext
exposes shared_supervision() and the server keeps the controller for the
Attach stream and token minting, replacing the driver-created controller
that was installed into the gateway. The Kubernetes factory injects it
through KubernetesComputeDriver::with_shared_supervision; new() stays the
dedicated-topology constructor and refuses a configuration that selects
the gateway topology without the gateway's coordination.

The Kubernetes driver keeps its platform work: placement annotations, the
shared supervisor Service and egress NetworkPolicy, launch Secret create
and read (bytes only), its own fence and Pod evidence, and the token
Secret patch. The supervisor process consumes the moved registration and
status types unchanged. Wire protocol, storage JSON, annotations, and the
supervisor token profile are unchanged; the private protobuf stays
generated in openshell-core and is re-exported by the new crate.

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

---------

Signed-off-by: Drew Newberry <anewberry@nvidia.com>
2026-10-22 22:20:02 +00:00
Taylor Mutch 39d95b42cb chore(e2e): remove obsolete OpenShift sandbox SCC grant (#4823)
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
2026-10-22 06:36:02 +00:00
Simon Scatton ceecf027c2 fix(deps): update webpki to distrust compromised emSign CAs (#4794)
Signed-off-by: Simon Scatton <sscatton@nvidia.com>
2026-10-21 13:30:22 +00:00
Drew Newberry 32f026442e feat(sandbox): initiate supervisor connections across compute drivers (#4656)
* feat(sandbox): support sandbox-initiated supervisor connections

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* refactor(sandbox): consolidate control transport setup

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* test(sandbox): remove prototype-specific unit tests

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* feat(kubernetes): migrate legacy control flow on restart

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* docs: remove prototype changes from published docs

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(sandbox): adapt reverse control to published backend protocol

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* feat(sandbox): initiate supervisor connections across compute drivers

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(docker): stage reverse control socket for the supervisor identity

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* refactor(sandbox): keep runtime transports behind the backend

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(docker): mount the runtime control volume read-only

The sandbox runtime only dials the supervisor-owned control socket, so
it does not need write access to the control volume. Mount it read-only
in the runtime container and stop staging the control directory through
the runtime bundle; the supervisor stager already initializes it through
its own writable mount.

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(kubernetes): separate legacy and runtime-initiated control services

Keep os-boundary selecting the workload and add os-supervisor for runtime-initiated connections. Bind the mode marker to its runtime generation so a restart by an older gateway does not inherit stale mode metadata. Preserve the legacy Service across creation and restart, validate the active endpoint, and drop the Service patch permission.

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

---------

Signed-off-by: Drew Newberry <anewberry@nvidia.com>
2026-10-20 18:25:45 +00:00
Varsha 1cb9ef8ab4 refactor(crypto): introduce a generic interface for crypto libraries and refactor AWS-LC (#3351)
* refactor(crypto): centralize AWS-LC behind backend interface

Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com>

* refactor(crypto): route new host proxy tests through facade

Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com>

* fix(crypto): make backend boundary check portable

Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com>

* fix(crypto): route Google JWT signing through facade and verify context dispatch

Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com>

* refactor(crypto): propagate digest backend failures

Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com>

* refactor(crypto): route Vault TLS fixtures through facade

Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com>

* refactor(crypto): adapt isolation runtime callers after main rebase

Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com>

* fix(crypto): declare sandbox dependency and remove boundary lint

Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com>

* refactor(crypto): support independent backends and backend-free CA import

Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com>

* chore(crypto): refresh feature suite lockfile after rebase

Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com>

* fix(crypto): route runtime integrity and entropy through backend

Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com>

* chore(crypto): refresh lockfile after upstream rebase

Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com>

* fix(crypto): route VM socket entropy through backend after rebase

Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com>

* docs(crypto): document architecture and contributor boundaries

Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com>

* test(crypto): adapt upstream fixtures to backend facade

Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com>

* test(crypto): route relay TLS fixture through facade

Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com>

* test(crypto): adapt service bearer TLS fixture after rebase

Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com>

* fix(crypto): route provider profile TLS initialization through facade

Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com>

* fix(crypto): adapt upstream VM and supervisor paths to facade

Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com>

* ci(crypto): enforce backend independence in branch checks

Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com>

* test(crypto): adapt upstream token grant TLS fixture

Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com>

* fix(crypto): avoid format collect in executable hashing

Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com>

* fix(crypto): address provider selection and decryption review

Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com>

* fix(crypto): adapt upstream operator TLS paths after rebase

Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com>

* fix(crypto): adapt upstream TLS fixtures and protocol tests

Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com>

---------

Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com>
2026-10-19 19:09:49 +00:00
Piotr Mlocekanddivesh 604b9f66f6 feat(config): deliver gateway configuration over the supervisor session (#4320)
Supervisors poll the gateway for configuration every 10 seconds. This PR adds an opt-in push mode (config_delivery_mode = "push"): supervisors still start by polling, and once their ConnectSupervisor session is accepted, the gateway delivers every later change over it and supervisors apply it and report the result. poll stays the default, and older supervisors and gateways keep polling.

---------

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
Signed-off-by: divesh <dgude@nvidia.com>
Co-authored-by: divesh <dgude@nvidia.com>
2026-10-16 14:48:04 -07:00
Evan LezarandMatthew Grossman a2982bf53d fix(ci): restore default e2e coverage with a Kubernetes recovery waiver (#4424)
* fix(ci): make conformance-only e2e selection explicit

Signed-off-by: Evan Lezar <elezar@nvidia.com>

* fix(ci): use mapped NodePort for kind gateway sessions

Signed-off-by: Evan Lezar <elezar@nvidia.com>

* fix(ci): waive SSH transport recovery only on Kubernetes

Signed-off-by: Evan Lezar <elezar@nvidia.com>

* fix(e2e): log a notice when runners skip Rust e2e tests

Emit a GitHub Actions notice from the Docker, Podman, Kubernetes, and VM
runners when e2e-conformance-only skips the Rust tests, so a narrowed lane
is visible in the job summary instead of silently reporting success.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

* docs(testing): move e2e conformance-only selector to shared section

The selector applies to all four driver runners, so document it with the
Rust CLI e2e suites instead of under the Kubernetes task variants.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

---------

Signed-off-by: Evan Lezar <elezar@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Co-authored-by: Matthew Grossman <mgrossman@nvidia.com>
2026-10-16 17:43:11 +00:00
Florent BENOIT a4c848adbd feat(vm): add host bind mount support via virtiofs (libkrun) (#3463)
* feat(vm): add host bind mount support via virtiofs (libkrun)

Introduce host directory bind mounts for libkrun VM sandboxes using
the virtiofs4 API with simplified permission semantics. The gateway
accepts `enable_bind_mounts` in the `[openshell.drivers.vm]` TOML
table and forwards it to the VM driver subprocess.

Mounts are specified per-sandbox in driver_config.mounts[] with
source, target, optional read_only (default: read-only), and an
optional `type` that must be `bind`, so Docker and Podman bind entries
(without `selinux_label`) parse unchanged.

At sandbox validation the driver rejects mounts unless bind mounts
are enabled and resource admission is disabled. It rejects GPU and
other QEMU launches, sources with control characters, and more mounts
than libkrun has interrupt lines for (32 on aarch64, 3 on x86_64), and
checks targets against reserved VM-internal directories and each other
(no nested targets). The QEMU check runs again once the launch backend
is resolved.

Host source directories are checked at provisioning, before image
preparation, so a briefly missing source fails provisioning visibly
instead of denying recovery of a persisted sandbox. Missing,
inaccessible, symlinked, non-directory, and unreadable sources are
rejected: libkrun does not follow a final symlink, and on macOS it
panics during guest boot when it cannot open the share root.

The driver generates virtiofs tags and always writes a guest-side
manifest into the overlay disk, together with the init drop-ins, so an
image-supplied manifest can never drive guest mounts. The launcher
parses its tab-separated --vm-mount arguments from the right, so a
source can never shift the tag or mode, and the driver logs each
shared directory at launch.

Guest init mounts the shares after every /sandbox ownership fixup
and the init drop-ins, so recursive chowns never reach host files.
It refuses targets that traverse symlinks in the image, hands newly
created mount point parents under /sandbox to the sandbox user, and
skips an empty manifest.

krun_add_virtiofs4 is resolved optionally, so older libkrun runtimes
still boot sandboxes without mounts.

The kernel config lists CONFIG_FUSE_FS and CONFIG_VIRTIO_FS
explicitly; the pinned libkrunfw base config already enables both.

A vm_bind_mount e2e test checks a read-write share round-trips data
with the host, accepts a new file, directory, and nested file that the
host then sees, and that a read-only share rejects writes. libkrun on
rootless Linux refuses file creation by a guest user whose UID/GID
differ from its own, so the VM e2e gateway runs the sandbox as the
gateway user there. It enables bind mounts like the Docker and Podman
e2e gateways.

The runtimes docs cover the mount limit, the symlink restriction, the
guest-kernel exposure, and the rootless Linux UID/GID note. The
debug-openshell-cluster skill lists VM bind mount checks.

Signed-off-by: Florent Benoit <fbenoit@redhat.com>

* fix(nix): refresh vm-runtime hashes to the libkrun v1.19.4 runtime

The Nix vm-runtime package fetches the rolling vm-runtime release with
fixed hashes pinned on 2026-09-16. The release was rebuilt on
2026-09-21 after the libkrun v1.19.4 / libkrunfw v5.6.1 bump (#3451),
but the hashes were not updated, so CI kept substituting the
2026-09-15 runtime (libkrun v1.17.4) from the binary cache.

That libkrun lacks krun_add_virtiofs4, which VM bind mounts need.
Point the hashes at the current release assets.

Signed-off-by: Florent Benoit <fbenoit@redhat.com>

---------

Signed-off-by: Florent Benoit <fbenoit@redhat.com>
2026-10-16 07:56:32 +00:00
Drew Newberry b2069a267c feat(docker): support host and bridge supervisor networking (#4016)
* feat(docker): select supervisor networking for ECI workloads

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* test(docker): use automatic networking in e2e

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* test(docker): rely on existing e2e networking coverage

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* docs(docker): shorten supervisor networking configuration

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(docker): let local gateway discover supervisor endpoint

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* refactor(docker): make supervisor networking operator-selected

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* refactor(docker): resolve host gateway in the driver

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* docs(docker): simplify host networking guidance

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* docs(docker): trim configuration guidance

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(docker): discover host gateway once at driver startup

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

---------

Signed-off-by: Drew Newberry <anewberry@nvidia.com>
2026-10-16 01:44:16 +00:00
ShijuandJohn Myers 29a2b8e7d9 feat(providers): report bounded token-grant failures (#4427)
* feat(providers): report bounded token-grant failures

Signed-off-by: Shiju <shiju@nvidia.com>

* test(providers): align policy metadata and TLS lockfile

Signed-off-by: Shiju <shiju@nvidia.com>

* test(providers): fix trusted grant recovery fixture

Signed-off-by: Shiju <shiju@nvidia.com>

* test(providers): enable denial API in recovery fixture

Signed-off-by: Shiju <shiju@nvidia.com>

* refactor(providers): simplify token-grant failure reporting

Remove duplication and dead paths from the token-grant failure
reporting without changing classification, the workload-visible
502 response, or the wire format.

- Build the gRPC failure envelope with tonic-types, store the typed
  failure in EndpointFailure, and share one request-ID rule and one
  endpoint-status authority check.
- Merge the gateway exchange client wrapper, drop forwarding
  validators, and name the request field when a reported cause is
  rejected.
- Inject fallback credentials only from live provider state and
  delete the unread per-connection credential copy. Keep one 502
  writer per path and restore the base token-cache handling.
- Check the token endpoint after the cache lookup, still before any
  credential acquisition.
- Document 429 and 5xx classification as implemented, and drop tests
  that repeat retained assertions through the same code path.

Signed-off-by: Shiju <shiju@nvidia.com>

* refactor(providers): trim token-grant failure tests and supervisor API

Cut the token-grant failure tests to one test per distinct behavior
across core, gateway, supervisor, CLI and Go, and replace the 794-line
closure E2E with status assertions in the existing route-recovery E2E:
the refused grant appears in sandbox status with the response's
request ID, code, stage and upstream status, and the matching grant's
later success clears it.

- Pin the live provider snapshot with one helper used by inspected and
  fallback forwarding, and inject selector-matched grants through
  inject_for_selectors, which borrows the pinned context. Move the
  test-only wrapper into test_support.
- Drop the 35-second wrappers around the gateway grant RPC. The shared
  channel bounds the connection attempt and the gateway bounds its
  token-service calls.
- Log the token service's OAuth error and error_description, without
  control characters and limited to 256 characters each, with the
  request ID in the gateway or supervisor log. Status and the workload
  response still carry only the bounded cause.
- Keep one check that an untrusted token-service certificate cannot
  assert a cause.

Signed-off-by: Shiju <shiju@nvidia.com>

* fix(providers): restore test import and features lockfile

The injection tests named EndpointObservationSender through the parent
module, which no longer imports it. The features suite lockfile kept the
tonic TLS dependency that the removed E2E fixture added.

Signed-off-by: Shiju <shiju@nvidia.com>

* test(providers): fix body-bound fixture width and table type

Pass the response-size width as a named argument; an implicitly captured
constant panics at runtime. Name the mutation type in the validation
table for Clippy.

Signed-off-by: Shiju <shiju@nvidia.com>

* test(providers): pad exact-bound body without a format width

MAX_TOKEN_RESPONSE_BYTES exceeds u16::MAX, the largest format width, so
the padded fixture panicked. Pad with repeat_n as before.

Signed-off-by: Shiju <shiju@nvidia.com>

* test(server): update public RPC schema hash after rebase

Main added the ListProvidersRequest label selector. The merged public
schema hash comes from the Branch Checks storage_proto test output.

Signed-off-by: Shiju <shiju@nvidia.com>

* fix(providers): keep token-service error text out of grant logs

A token service can echo a submitted credential in its OAuth error or
error_description. Log only the classified failure code, HTTP status and
request ID, and cover the canary echo in a dedicated test binary.

Signed-off-by: Shiju <shiju@nvidia.com>

* docs(providers): tighten token grant failure documentation

Signed-off-by: Shiju <shiju@nvidia.com>

* docs(providers): condense token grant failure guidance

Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>

---------

Signed-off-by: Shiju <shiju@nvidia.com>
Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>
Co-authored-by: John Myers <9696606+johntmyers@users.noreply.github.com>
2026-10-15 18:45:27 +00:00
John T. Myers fce4e5aea5 fix(sandbox): release broker-owned listening sockets (#4444)
Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>
2026-10-13 16:13:43 +00:00
Shiju 6a6401b9c3 fix(cli): backpressure sandbox uploads (#4448)
* fix(cli): backpressure sandbox uploads

Stream archive uploads over a bounded SSH subsystem whose credit follows extraction stdin progress. Preserve the existing input budget and tar destination semantics, validate completion, and clean up cancelled transfers. Add protocol, lifecycle, and large-upload regression coverage without changing russh.

Signed-off-by: Shiju <shiju@nvidia.com>

* fix(cli): preserve upload cancellation and errors

Keep cancellation active through upload setup and retry delays, and
terminate cancelled SSH probes. Preserve protocol failures through output
completion and report captured diagnostics on early extraction failure.
Remove redundant upload validation and extend the affected regressions.

Signed-off-by: Shiju <shiju@nvidia.com>

* refactor(cli): carry upload credit over SSH streams

Replace the framed upload protocol with SSH's own streams. The client
sends one JSON plan line, the raw archive and EOF. The supervisor's
stdout carries only cumulative totals of bytes written to extraction
stdin, extraction output goes to stderr, and the exit status reports
the result. Extraction starts inline through the ordinary exec path.

This removes the frame codec, SHA-256 digests, the CLI sha2 dependency
and the supervisor upload state machine while keeping the 1 MiB credit
window, the 4 MiB input guard, cancellation and error reporting. A
failed archive now stops SSH before closing stdin, so EOF never marks a
truncated archive complete.

Signed-off-by: Shiju <shiju@nvidia.com>

* fix(cli): isolate upload SSH connections and pin credit cadence

- Apply the forward multiplexing overrides to the upload ssh command, so
  a user ControlMaster cannot carry an upload over another session's
  master connection.
- Kill ssh before closing its stdin when a transfer fails locally, and
  include ssh stderr in transport errors.
- Share ACK_INTERVAL_BYTES from core. Assert at compile time that CLI
  chunks fit the credit left after an ack and that the window stays
  below the supervisor's pending-input guard.
- Log the real reason when the supervisor rejects an upload.
- Test that a single-packet upload reports its final total.

Signed-off-by: Shiju <shiju@nvidia.com>

* test(cli): assert upload ssh opens the subsystem without a TTY

Signed-off-by: Shiju <shiju@nvidia.com>

---------

Signed-off-by: Shiju <shiju@nvidia.com>
2026-10-13 14:55:45 +00:00
Matthew GrossmanandEvan Lezar 8c67c27e8b test(oci): add a shared Docker and Podman suite for image USER and WORKDIR (#3931)
* test(oci): share OCI image checks across Docker and Podman

Add an oci-image feature testsuite that runs against installed artifacts.
It covers custom WORKDIR placement, image content and ownership, workspace
writes from the main process and exec, file transfer including
directory-merge upload, OCI user identity, the managed /sandbox fallback,
and rejection of an unwritable WORKDIR.

CI runs the suite on the Docker rootful, Podman rootful, and Podman
rootless tmachine guests.

Replace the Docker-only custom_image e2e with the shared suite.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Evan Lezar <elezar@nvidia.com>

* test(oci): require WorkspaceValidationFailed and cover image USER fallback

The unwritable WORKDIR scenario now requires the WorkspaceValidationFailed
reason, so unrelated provisioning failures cannot pass it. The numeric
USER scenario supplies a policy without a process section, so the image
USER is the only source of the sandbox identity.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Evan Lezar <elezar@nvidia.com>

* test(podman): leave shared OCI identity checks to the oci-image suite

The oci-image feature suite covers the sandbox identity seen by the main
process and sandbox exec on Docker and Podman. The Podman e2e keeps the
checks only it can make: the pinned image ID and the isolated
workload/supervisor container pair.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Evan Lezar <elezar@nvidia.com>

* test(oci): accept existing workspace rejection diagnostics

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Evan Lezar <elezar@nvidia.com>

* test(oci): fix cleanup ordering and restore coverage parity

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Evan Lezar <elezar@nvidia.com>

* test(oci): remove migrated legacy workspace helpers and checks

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

* fix(drivers): report rejected OCI working directories as WorkspaceValidationFailed

The supervisor exits with the reserved status when the sandbox rejects the
image working directory. Docker maps that exit during readiness and
monitoring and records it before removing the supervisor container, and
Podman maps the supervisor companion's exit instead of the workload's.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

* test(oci): require WorkspaceValidationFailed for unwritable WORKDIR

Unrelated provisioning failures can no longer pass the rejection scenario.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

* fix(oci): keep exit 78 reserved and report fixed workspace diagnostics

The supervisor no longer passes on a workload's own exit status 78, so
drivers cannot misreport it as a workspace rejection after a runtime
restart. The rejection is recognized once at agent start and carried as a
typed error. Docker reports the same fixed condition message as Podman and
logs the supervisor output instead, and its monitor removes the supervisor
container when a stop overlaps diagnostics collection. The OCI image suite
registers images for cleanup only after they build.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

* refactor(oci): simplify workspace failure handling and OCI image tests

Report a recorded Docker workspace rejection once, where supervisor
readiness fails, and match its fixed message exactly. Read the Podman
supervisor exit status once per inspection. Choose the supervisor exit
status in one function and derive the workspace error type.

The OCI image suite shares step helpers, checks that a custom workspace
does not also create /sandbox, and its playbook waits for a connected
gateway and collects failure diagnostics through the shared role.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

* test(oci): replace podman_oci_identity with shared and driver suite checks

The oci-image feature suite now checks, on Docker and Podman, that the
image user has no capabilities, has no_new_privs set, and cannot read
the sandbox control credentials, and that a custom WORKDIR is not a
mount. The checks that need container inspection (pinned image ID,
network=none, supervisor user and capabilities) move to a small test in
the Podman driver suite. Remove the legacy podman_oci_identity e2e test.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

* test(oci): keep podman_oci_identity for Podman-only isolation checks

Restore the Podman e2e test removed in the previous commit and drop
only the checks the shared oci-image suite now runs on Docker and
Podman: workload capabilities, no_new_privs, credential readability,
and the workspace and /sandbox mount checks. The pinned image ID and
the workload/supervisor isolation checks stay, as does the Podman
driver suite unchanged.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

* refactor(oci): simplify workspace failure reporting and OCI image suite

Carry the Docker control supervisor's failure reason in a typed error
instead of matching status text, build the fixed workspace message where
the wait result is classified, and drop the fake Docker server test.

Inline image cleanup ordering in the suite, read the container engine
once, share sandbox naming, and stop granting a literal /sandbox in the
image identity policy. Add e2e:oci-image mise tasks and use them from the
ARM64 Docker workflow and TESTING.md.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

* test(oci): move shared container checks into the OCI image suite

Inspect the containers the driver created through the gateway's engine in
the shared suite: the workload runs the inspected image ID as the OCI
identity without a network or supervisor credential mounts, and the
supervisor is a separate non-root container. Keep only the Podman-specific
supervisor identity and capability checks in podman_oci_identity.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

* test(oci): check the workspace rejection reason from sandbox JSON

Read the WorkspaceValidationFailed condition reason from sandbox get
--output json instead of matching CLI output, and restore the Docker
driver's original workspace failure helpers.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

* test(oci): capture fixture image identity before sandbox creation

Signed-off-by: Evan Lezar <elezar@nvidia.com>

---------

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Evan Lezar <elezar@nvidia.com>
Co-authored-by: Evan Lezar <elezar@nvidia.com>
2026-10-13 13:00:59 +00:00
Taylor Mutch 5e414eadde test(e2e): remove stale policy retries (#4453)
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
2026-10-13 00:47:36 +00:00
Yuedong Wu 3495c8e92a fix(kubernetes): refresh sandbox runtime image on start (#4376)
Signed-off-by: Yuedong Wu <dwcn22@outlook.com>
2026-10-12 20:21:27 +00:00
Shiju e585b49173 feat(providers): support stable credential placeholders (#3339)
* feat(providers): support stable credential placeholders

Add credentials[].stable_placeholder so a running client keeps using an externally rotated static credential through the reference it already holds. Squashed replay of the earlier four commits onto upstream main 71c3cd957; re-derives the frozen schema fingerprints and adapts the provider environment revision test to the added capability argument.

Signed-off-by: Shiju <shiju@nvidia.com>

* fix(e2e): apply stable-placeholder supervisor image override

Use ManagedGateway's per-process supervisor image override so the fixture
CA reaches the supervisor when runtime image environment variables take
precedence over gateway TOML. Restore the wrapper's original image after
each test, following the provider-readiness fixture.

Signed-off-by: Shiju <shiju@nvidia.com>

* test(server): update public RPC schema fingerprint after merging main

Signed-off-by: Shiju <shiju@nvidia.com>

---------

Signed-off-by: Shiju <shiju@nvidia.com>
2026-10-12 16:39:31 +00:00
Evan Lezar c4dc30dc1b test(conformance): run driver suites with cargo (#3866)
* test(conformance): run driver suites with cargo

Signed-off-by: Evan Lezar <elezar@nvidia.com>

* fix(e2e): resolve conformance CLI path before cargo tests

Signed-off-by: Evan Lezar <elezar@nvidia.com>

* refactor(conformance): remove unused CLI scenario APIs

Signed-off-by: Evan Lezar <elezar@nvidia.com>

---------

Signed-off-by: Evan Lezar <elezar@nvidia.com>
2026-10-12 15:35:03 +00:00
Drew Newberry e1f3c82caa fix(deps): update noyalib and preserve policy null rejection (#4348)
* fix(deps): update noyalib and preserve policy null rejection

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(providers): preserve quoted YAML duration exports

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* test(providers): assert YAML duration semantics instead of quoting

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(yaml): preserve authored input and cross-reader string semantics

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* refactor(yaml): validate schema objects before standard decoding

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* docs(policy): remove YAML compatibility documentation changes

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

---------

Signed-off-by: Drew Newberry <anewberry@nvidia.com>
2026-10-09 05:04:49 +00:00
Daneyon Hansen 40e01424c1 fix(helm): improve GatewayClass overrides (#4345)
* fix(helm): improve GatewayClass overrides

Give chart-created Gateways a distinct default name and allow GRPCRoutes
to select a listener on an existing Gateway.

Fixes: #2469

Signed-off-by: Daneyon Hansen <daneyon.hansen@solo.io>

* fix(e2e): resolve rendered Gateway name

Signed-off-by: Daneyon Hansen <daneyon.hansen@solo.io>

* fix(helm): preserve Gateway name defaults

Signed-off-by: Daneyon Hansen <daneyon.hansen@solo.io>

---------

Signed-off-by: Daneyon Hansen <daneyon.hansen@solo.io>
2026-10-09 04:13:57 +00:00
cjagwani a991b8e369 fix(sandbox): preserve exec capacity under failure and load (#4324)
* fix(sandbox): reclaim failed exec waits

Signed-off-by: Charan Jagwani <cjagwani@nvidia.com>

* test(e2e): cover exec recovery after policy denial

Signed-off-by: Charan Jagwani <cjagwani@nvidia.com>

* fix(sandbox): preserve live descriptor headroom

Signed-off-by: Charan Jagwani <cjagwani@nvidia.com>

* fix(sandbox): reclaim descriptors before exec

Signed-off-by: Charan Jagwani <cjagwani@nvidia.com>

* fix(sandbox): separate exec and socket capacity

Signed-off-by: Charan Jagwani <cjagwani@nvidia.com>

* fix(sandbox): stream Landlock baseline entries

Signed-off-by: Charan Jagwani <cjagwani@nvidia.com>

---------

Signed-off-by: Charan Jagwani <cjagwani@nvidia.com>
2026-10-09 01:26:31 +00:00
Gaizka MenendezandTaylor Mutch fe3942f2da feat(helm): migrate gateway configuration to gatewayConfig (#3384)
* feat(helm): add generic gateway TOML serializer

Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com>

* docs(architecture): define Helm gateway configuration boundary

Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com>

* feat(helm): define default gateway configuration map

Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com>

* refactor(helm): render gateway config from values map

Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com>

* test(helm): cover generic gateway TOML rendering

Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com>

* feat(helm): protect gateway config secret boundary

Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com>

* docs(helm): classify legacy gateway configuration values

Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com>

* refactor(helm): derive dual-use resources from gateway config

Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com>

* test(helm): migrate gateway config scenarios

Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com>

* refactor(helm): derive credential resources from gateway config

Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com>

* refactor(helm): migrate gateway config overlays

Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com>

* refactor(helm): remove gateway configuration shadow values

Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com>

* refactor(helm): align runtime config with resources

Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com>

* docs(helm): add gateway config migration guide

Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com>

* test(helm): validate rendered gateway config

Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com>

* test(helm): cover config resource coherence

Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com>

* fix(helm): make Kubernetes E2E deployable

Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com>

* fix(e2e): pass host aliases through gateway config

Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com>

* docs(e2e): reference gateway config host alias

Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com>

* fix(helm): enforce gateway resource ownership

Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com>

* test(e2e): cover chart host gateway input

Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com>

* fix(helm): address gateway config review findings

Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com>

* fix(helm): repair RFC 0012 migration integration

Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com>

* fix(e2e): align Kubernetes parity with RFC 0012

Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com>

* fix(helm): complete gateway config migration

Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com>

* chore(helm): add SPDX header to TOML template

* fix(helm): preserve legacy gateway configuration aliases

Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com>

* fix(helm): complete legacy gateway config compatibility

Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com>

* feat(helm): support gateway config arrays of tables

* fix(helm): preserve sandbox identity aliases

* fix(helm): address gateway config review findings

Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com>

* fix(helm): preserve gateway config migration settings

Signed-off-by: Taylor Mutch <taylormutch@gmail.com>

* test(helm): validate extension arrays in CI values

Signed-off-by: Taylor Mutch <taylormutch@gmail.com>

* fix(helm): restore unconditional SPDX headers

Signed-off-by: Taylor Mutch <taylormutch@gmail.com>

* fix(helm): restore route SPDX header placement

Signed-off-by: Taylor Mutch <taylormutch@gmail.com>

* docs(helm): simplify configuration and migration examples

Signed-off-by: Taylor Mutch <taylormutch@gmail.com>

* docs(helm): remove redundant flag explanations

Signed-off-by: Taylor Mutch <taylormutch@gmail.com>

* docs(helm): move chart configuration guidance to Kubernetes setup

Signed-off-by: Taylor Mutch <taylormutch@gmail.com>

---------

Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com>
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
Co-authored-by: Taylor Mutch <taylormutch@gmail.com>
2026-10-08 18:39:59 +00:00
John T. Myers 67adcf1a3c feat(ocsf): emit full JSON records to supervisor stderr (#4323)
* feat(ocsf): emit full JSON records to supervisor stderr

Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>

* fix(ocsf): preserve console record boundaries

Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>

---------

Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>
2026-10-08 05:36:19 +00:00
Shiju ffb530671f fix(providers): prepare dynamic credentials across HTTP relay paths (#4152)
* fix(providers): bind multi-route grants to admitted endpoints

Carry gateway-derived endpoint owners through policy and credential delivery. Select grants only from owners that admit the current request, refresh provider snapshots per request, and reject superseded installations before forwarding.

Cover persistent routes, overlapping owners, denial, refresh, cache expiry and recovery with focused tests and a Podman regression.

Signed-off-by: Shiju <shiju@nvidia.com>

* refactor(providers): simplify admitted-endpoint grant selection

Share admitted endpoints between forwarding and credential authorization. Select grants from the live snapshot without rebuilding a locked map, reuse profile policy construction, and preserve freshness guards when grants are removed.

Signed-off-by: Shiju <shiju@nvidia.com>

* fix(providers): preserve trusted owners across policy approval

Sanitize authored owner metadata before composing effective policies, preserve derived owners through proposal approval, and normalize advisor endpoint comparisons. Account for gateway-only owner fields in SDK coverage and schema inventory.

Signed-off-by: Shiju <shiju@nvidia.com>

* refactor(providers): share dynamic credential key layout and require a snapshot for admission

Move the endpoint-bound credential key layout into openshell-core. DynamicCredentialKey encodes the key the gateway builds, and the supervisor reads its endpoint selector, credential identity, and revision-scoped form through the same module instead of parsing tab-separated strings in three places. Revision scoping becomes ProviderCredentialSnapshot::scoped_key.

inject_for_admitted_owners now takes the pinned snapshot directly. The L7 admission path never acquires a grant from the live credential map, which production populates only alongside a snapshot. L4 forwarding keeps the selector-only path through inject_if_needed. Both share one acquire-and-rewrite step.

Delete two relay tests whose behavior the Rego admission tests already prove. Add a test that grants for different headers are acquired only for admitting owners.

Signed-off-by: Shiju <shiju@nvidia.com>

* fix(providers): share credential preparation across HTTP relays

Apply endpoint-owned grants after policy and request middleware in REST,
GraphQL, JSON-RPC/MCP and inspected plaintext forwarding. Preserve live
provider and policy checks through guarded writes, reject missing owner
metadata, and reject profiles that cannot inject dynamic HTTP credentials.

Cover production stream dispatch, multiple endpoints, failure handling and
plaintext ownership with focused regressions. Document upgrade ordering.

Closes #3657

Signed-off-by: Shiju <shiju@nvidia.com>

* fix(providers): validate dynamic credential key encoding

Reject control characters at profile import and key construction. Propagate
invalid key errors through gateway environment resolution and reject malformed
key layouts before token acquisition while preserving supported legacy keys.

Consolidate duplicate relay success tests and share forward-proxy setup without
removing ownership, canonicalization, refresh or guarded-write assertions.
Document token-grant endpoint authority.

Signed-off-by: Shiju <shiju@nvidia.com>

---------

Signed-off-by: Shiju <shiju@nvidia.com>
2026-10-07 23:48:40 +00:00
Matthew Grossman 1fff8b97b9 feat(podman): honor OCI image working directories (#3982)
* fix(supervisor): report rejected OCI working directories as WorkspaceValidationFailed

The RFC 0012 split dropped the supervisor exit status that drivers map to
the WorkspaceValidationFailed condition. An image WORKDIR the sandbox
identity cannot use then surfaced as ControlSupervisorStartFailed on
Docker and as a signal kill on Podman.

The supervisor now exits with the reserved status when the sandbox rejects
the image working directory. Docker maps that supervisor exit during
readiness and monitoring, and Podman maps the supervisor companion's exit
instead of the workload's.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

* feat(podman): honor OCI image working directories

Podman sandboxes now use a custom OCI image WORKDIR as the workspace and
as the working directory for agent commands, matching Docker. Empty, /,
and /sandbox values keep the managed /sandbox workspace volume.

A custom workspace stays in the image's container filesystem: no
workspace volume, no archive upload, and no root setup step. Resolve the
image ID, user, environment, and working directory from one pinned
inspection, validate the workdir with the shared OCI rules, and reject
Podman control-path overlaps and image volumes or driver mounts that
cover it. The runtime starts from / and passes the resolved path to agent
commands with --workdir.

Add podman_oci_identity to the Podman CI test list.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

* fix(drivers): preserve custom workspace failures across supervisor cleanup

Check the Podman custom workspace rather than the absent managed volume in the OCI identity E2E. Classify Bollard wait errors by exit code and record Docker workspace failures before removing the supervisor so readiness retains the specific failure reason.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

* fix(docker): use sandbox ID for recorded startup failures

Keep the workload container ID for Docker inspection and use the sandbox ID to retrieve the monitor failure. Test with distinct identifiers so the lookup cannot accidentally pass with a container ID.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

* refactor(podman): keep WORKDIR change focused on workspace behavior

Remove the new cross-driver workspace failure classification path and its monitor/readiness workaround. Preserve Podman WORKDIR resolution, mount safety, and workspace rejection without requiring a distinct condition reason.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

* refactor(podman): simplify inspected image metadata and test fixtures

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

* code review

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

---------

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
2026-10-07 21:29:58 +00:00
Mike Nguyen 9fd41e6bfd fix(ssh): persist sandbox host identities (#4094)
* fix(ssh): persist sandbox host identities

Store each sandbox's Ed25519 host key in the gateway credential store and
deliver it only to the supervisor. Preserve identity across restarts,
delete owned credentials with the sandbox, and expose the public SHA256
fingerprint through sandbox and SSH-session APIs and client SDKs.

Cover credential ownership, cancellation, deletion retries, client
compatibility, and pinned SSH connections through lifecycle transitions.

Closes #3835

Signed-off-by: Mike Nguyen <miken@nvidia.com>

* fix(compute): clean up failed sandbox SSH identity creation

Signed-off-by: Mike Nguyen <miken@nvidia.com>

* test(ssh): wait for sandbox deletion before name reuse

Signed-off-by: Mike Nguyen <miken@nvidia.com>

---------

Signed-off-by: Mike Nguyen <miken@nvidia.com>
2026-10-06 20:09:20 +00:00
Simon Scatton 8760396975 fix(compute): revalidate late exit events after restart (#4235)
Signed-off-by: Simon Scatton <sscatton@nvidia.com>
2026-10-06 15:25:53 +00:00
Drew Newberry 12cec59bf4 fix(sandbox): accept local connections natively on loopback-confined sockets (#4150)
* fix(sandbox): accept local connections natively on loopback-confined sockets

On kernels without SECCOMP_FILTER_FLAG_WAIT_KILLABLE_RECV (RHEL 9 / RHCOS
5.14), the broker cannot safely write an accepted peer address into
workload memory, so accept/accept4 with a peer-address buffer failed with
EOPNOTSUPP. Static binaries and Go servers, which issue the raw syscall,
could not accept connections at all.

Move local acceptance into the kernel and replace per-accept inspection
with standing kernel confinement:

- Bind every broker-created TCP/UDP socket to the loopback device before
  injection and verify the binding. Accepted sockets inherit it, so they
  can neither receive routed ingress nor emit routed egress.
- Stop notifying accept/accept4 and remove the accept workers, the
  SIGUSR2 accept-interrupt monitor, and the 64-accept ceiling.
- Continue getpeername natively for every descriptor except relayed
  connections.
- Deny interface-selection socket options and MSG_FASTOPEN sends from
  scalar syscall arguments, so confinement does not depend on the
  capability state of the user namespace that owns the network namespace
  and covers unregistered descriptors.
- Drop loopback-interface ingress on the non-loopback TCP control
  listener before it listens, so a workload cannot reach it by
  reconnecting a natively accepted socket.
- Mark descriptors above stdio close-on-exec in every workload pre_exec
  path.
- Require an active confinement probe at qualification and carry it as
  required authenticated audit evidence.

Document OpenShift 4.19 as the minimum release: RHCOS kernels for 4.16
through 4.18 are built without Landlock.

Closes #4058

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(sandbox): close review gaps in native local accept confinement

- Reject a TCP boundary control listener on a loopback address,
  including IPv4-mapped loopback. Production drivers bind the
  unspecified address; a loopback listener would be reachable from
  workload sockets the broker does not track.
- Extend the confinement probe to IPv6 and prove that an accepted
  socket keeps its loopback binding after an AF_UNSPEC disconnect.
- Pin the accepted-peer contract with tests: a loopback-bound listener
  admits only clients in the sandbox network namespace, and workload
  sockets cannot bind a non-loopback source address.
- Correct the support matrix: accepted connections are bounded by
  per-process descriptor limits, the runtime PID limit, and the sandbox
  memory limit, not by a broker limit.

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* refactor(sandbox): replace unsafe socket calls with socket2 and rustix

Socket confinement now uses socket2 for device binding and the ingress
filter, and rustix for interface lookup and AF_UNSPEC disconnect, so the
module contains no unsafe code. The control-listener filter is no longer
locked: no safe API exposes SO_LOCK_FILTER, and the listener descriptor
never leaves the trusted sandbox process, which marks every descriptor
above stdio close-on-exec before running workload code.

Tests added for native accept use rustix and socket2 instead of raw libc
calls. rustix issues accept4 and getpeername as raw syscalls, so the
direct-syscall test keeps its meaning. The close-on-exec sweep test runs
in a re-executed test process instead of a forked child.

The close_range(CLOSE_RANGE_CLOEXEC) syscall in the pre_exec hook remains
the only unsafe added by this branch; neither rustix nor nix wraps it.

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(sandbox): allowlist workload socket families

The workload seccomp filter denied only AF_PACKET, AF_BLUETOOTH, and
AF_VSOCK, and the broker continued socket() for every non-INET family.
Several protocol families, including AF_RXRPC, AF_SMC, and AF_KCM, carry
traffic over kernel-owned sockets that the broker never creates and that
are not bound to loopback.

Allow only AF_UNIX, AF_NETLINK (still limited to NETLINK_ROUTE), and the
brokered AF_INET/AF_INET6 families, and restrict socketpair(2) to
AF_UNIX. Both decisions use the scalar domain argument. The broker
independently refuses non-INET families other than AF_UNIX and
AF_NETLINK with EAFNOSUPPORT.

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* refactor(sandbox): remove legacy read-only mode

After native local accept, only two broker paths still wrote into
workload memory: getpeername on relayed connections and the per-message
lengths of a first sendmmsg to the DNS relay. Legacy mode existed only to
refuse those writes on kernels without WAIT_KILLABLE_RECV, and the
getpeername refusal broke CPython TLS on RHEL 9.

The broker now never writes workload memory:
- getpeername is no longer mediated and reports the kernel peer on every
  kernel; relayed connections report the loopback relay address.
- A first send to the DNS relay pins the broker's socket copy to the
  relay and continues the syscall, so the kernel performs the send and
  writes any per-message results.

Remove the listener mode, the task-memory write path and probe, and the
task_memory_write, cancellation, and task_memory_writes_disabled audit
evidence fields. WAIT_KILLABLE_RECV is still used when available and is
reported for diagnostics, but no mediation decision depends on it.

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* refactor(sandbox): drop WAIT_KILLABLE_RECV and make handlers restart-safe

The broker no longer writes workload memory, so killable notification
waits only reduced how often a signal restarts a notified syscall. RHEL 9
kernels never had them, so the handlers must tolerate restarts anyway.
Install the listener without the flag on every kernel instead of
special-casing newer ones.

Handlers now check that the notification is still live immediately
before each side effect (bind, connect of the retained socket, listen,
and the first DNS send), and answer a restarted operation the broker
already completed as the kernel would: a repeated TCP connect returns
EISCONN, repeating the same UDP association succeeds, a repeat of a
completed bind succeeds, and a failed relay reports its errno.

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(sandbox): finish teardown only when every workload descendant has exited

Teardown waited only for registered process groups to disappear. A root
is unregistered once reaped, so a descendant that ignored SIGTERM could
outlive it while termination reported success and never sent SIGKILL,
violating the bounded-termination requirement.

The sandbox now becomes a child subreaper when it is not PID 1, so
orphaned descendants stay in its tree and are reaped. Termination waits
until no live descendant remains, and every scanned process is signalled
through a pidfd after confirming its start time, so a reused PID is never
signalled.

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(sandbox): keep a frozen workload from resuming itself

Freezing stops every workload process with SIGSTOP. tgkill and
rt_tgsigqueueinfo were not mediated, and mediated kill, rt_sigqueueinfo,
and tkill passed SIGCONT through, so a workload process that was not yet
stopped could resume the others while the supervisor recovered.

Mediate tgkill and rt_tgsigqueueinfo like tkill: refuse targets in the
sandbox thread group, report a thread outside the named group as
missing, and continue otherwise. The boundary marks the broker frozen
before stopping the workload and clears it after resuming; while frozen,
workload requests to send SIGCONT fail with EPERM.

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(sandbox): send mediated DNS datagrams from the broker socket

The native UDP send path re-ran the workload's own sendmsg, so
per-message ancillary data such as IP_PKTINFO rode along; only the
loopback destination contained a routing override.

The broker now reads the datagram and sends it from its retained,
relay-connected socket, which it builds without ancillary data, so a
per-message override cannot redirect the packet. It writes nothing back
into workload memory. A send carrying control data is refused with
EOPNOTSUPP rather than silently stripped.

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(sandbox): keep sandbox control variables out of the workload environment

The canonical process inherits the sandbox environment so the image's
own ENV reaches the workload, then removed only a denylist of credential
variables. Other variables in the reserved OPENSHELL_ namespace, such as
the serialized user environment and the log level, still reached the
workload.

Remove every inherited OPENSHELL_ variable before applying the declared
environment, and restore OPENSHELL_SANDBOX=1. The image's ordinary ENV
and declared variables are unaffected; declared variables cannot use the
reserved namespace.

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(sandbox): keep /proc read-only for GPU workloads

GPU mode granted read-write access to all of /proc so CUDA's cuInit could
write thread names to /proc/<pid>/task/<tid>/comm.

Keep /proc read-only. Open mediation now serves a writable open of the
caller's own thread comm file: the broker opens it and injects the
descriptor, with no syscall continued. The kernel accepts a comm write
only from the target's own thread group, so a substituted path or reused
thread ID cannot rename another process's thread. Any other /proc write
is left to Landlock, which denies it.

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(sandbox): keep the broker socket for connected UDP DNS sockets

Sending DNS datagrams from the broker's socket required the broker to
keep its copy, but the connect paths to the relay and to loopback still
released it. glibc connects the resolver socket and then sends A and
AAAA together with sendmmsg, which is mediated, so resolution failed.

Release the copy on those connect paths only for TCP. Add a regression
test for connect followed by sendmmsg that fails fast rather than
hanging.

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(sandbox): keep slow loopback connects from stalling mediation

The loopback connect path held the registry lock while polling a TCP
connect for up to five seconds on the single notification dispatcher.
A workload connecting to a busy local listener stalled every other
mediated syscall, including opens and signals.

Connect a duplicate of the retained socket without holding the lock. A
nonblocking socket gets the native EINPROGRESS and the kernel completes
the handshake on the shared socket; a blocking socket waits on a bounded
worker thread.

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(sandbox): contain network broker handler panics

A panic in a notification handler unwound the single broker thread while
the health flag still read healthy, so every later blocked workload
syscall hung until the sandbox was killed.

Run each dispatch under catch_unwind: a panicking handler fails only that
syscall with EIO and the broker keeps mediating. If the broker thread
ever exits, mark it unhealthy so dependent operations fail closed instead
of blocking.

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(sandbox): reject ancillary data on DNS sends instead of broker-sending

Sending mediated DNS datagrams from the broker's own socket required
keeping a broker handle for each DNS socket's whole life, which regressed
real name resolution through reclaim and resource accounting that the
mock-based unit tests did not exercise.

Revert to continuing the kernel send, but refuse a send that carries
ancillary control data (msg_controllen != 0) at read time, so a
per-message routing override such as IP_PKTINFO cannot ride a mediated
DNS send. A loopback destination contains an override that races the
check. Full broker-side UDP mediation is left to a separate change with
deployment e2e.

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(sandbox): address review findings across the hardening changes

Fixes:
- The blocking local-connect worker no longer toggles O_NONBLOCK on the
  open file it shares with the workload; it waits with a plain connect.
- tgkill and rt_tgsigqueueinfo are notified only when they send SIGCONT,
  so ordinary thread signals such as Go preemption stay in the kernel.
  The frozen flag is re-read immediately before delivery.
- A shell redirect to the caller's own thread comm file works again;
  O_CREAT and O_TRUNC are no-ops there and O_EXCL returns EEXIST.
- The teardown scan is a linear walk and fails closed when /proc cannot
  be read.

Simplifications:
- Remove tests that need infrastructure outside the repo or prove
  nothing: the topology harness tests, the static-server benchmark, the
  direct-syscall accept test, the comm rename test without Landlock, a
  serde-default test, and the test-only panic hook in setsockopt.
- Drop the unused Failed connect outcome, the redundant read-back after
  binding to loopback, redundant OPENSHELL_SANDBOX settings, and stale or
  duplicated comments; reuse the reserved environment prefix constant.
- Tighten the support matrix and OpenShift wording.

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(cli): wait for input when piped exec stdin is nonblocking

Processes that inherit the same stdin share one open file description,
so any of them can make it nonblocking for all. `sandbox exec` then read
no input yet, got EAGAIN, and failed with "Resource temporarily
unavailable (os error 11)". Parallel e2e tests inherit the runner's
stdin, which made credential_gating fail intermittently.

The stdin reader now blocks in poll() until input or end of file arrives
instead of treating EAGAIN as fatal, and it leaves the shared descriptor's
flags alone. The e2e exec helper also stops inheriting the runner's
stdin, since its commands take no input.

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(sandbox): fix macOS lint and older-glibc test linking

Import HashSet only in the Linux-only process scan, and call gettid
through the raw syscall in a test, since the libc wrapper needs glibc
2.30 or newer.

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* docs(sandbox): clarify socket peer address reporting

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

---------

Signed-off-by: Drew Newberry <anewberry@nvidia.com>
2026-10-06 04:08:04 +00:00
Shiju 8579dfb301 feat(providers): add multiple tokens to one request (#4047)
* feat(providers): compose dynamic credential grants per request

Select the most-specific grant independently for each protected header,
preserving each credential's issuer, audience and cache identity. Resolve
all selected grants before rewriting the request, replace agent-supplied
header copies, and fail closed on collisions or acquisition failures.

Allow distinct-header compositions in gateway validation and redact raw
issuer errors. Cover independent cache entries, atomic TLS relay failure,
header replacement and concurrent request isolation; document the profile
contract.

Fixes #3320

Signed-off-by: Shiju <shiju@nvidia.com>

* fix(providers): correct dynamic grant CI checks

Use if-let for a grant result whose error payload is deliberately ignored,
and name the empty query map type in the regression fixture. Preserve grant
acquisition, failure redaction and request atomicity. Update the token-exchange
failure test to require the sanitized error instead of raw issuer text.

Signed-off-by: Shiju <shiju@nvidia.com>

---------

Signed-off-by: Shiju <shiju@nvidia.com>
2026-10-05 23:15:46 +00:00
Simon Scatton be7af99ff2 fix(e2e): override provider readiness supervisor image via environment (#4207)
Signed-off-by: Simon Scatton <sscatton@nvidia.com>
2026-10-05 17:54:48 +00:00
Matthew GrossmanandEvan Lezar 13c248fd45 fix(podman): create managed workspace volumes owned by the workload identity (#3981)
* fix(podman): create managed workspace volumes owned by the workload identity

Podman now creates the managed /sandbox volume with uid/gid options for the
resolved workload identity, so the workload starts directly as that
identity. This fixes rootful sandboxes whose image USER or policy
run_as_user could not write to a root-owned /sandbox, and removes the
root-then-drop workspace chown start path.

Resource admission accepts the managed workspace volume when its options
match the workload container's final identity, or are empty for volumes
created by older gateways. The channel volume still requires empty options.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

* test(podman): cover managed volume reuse and workspace access

Signed-off-by: Evan Lezar <elezar@nvidia.com>

* refactor(podman): clarify managed volume creation and validation

Signed-off-by: Evan Lezar <elezar@nvidia.com>

* test(podman): verify workspace access across user namespaces

Signed-off-by: Evan Lezar <elezar@nvidia.com>

---------

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Evan Lezar <elezar@nvidia.com>
Co-authored-by: Evan Lezar <elezar@nvidia.com>
2026-10-05 16:42:43 +00:00
Shiju d676e036a4 fix(vm): reject conflicting workload identity selectors (#4036)
* fix(vm): enforce the configured workload identity

Reject conflicting policy users and groups before VM image preparation
and before guest attach or process startup changes state. Validate
supervisor policy updates against the protected VM workload identity.

Preserve the gateway CA transport and capability-free sandbox launcher.

Signed-off-by: Shiju <shiju@nvidia.com>

* test(sandbox): clarify VM identity rejection fixtures

Name invalid user and group fixtures distinctly and move the final
workload identity into its group mismatch test.

Signed-off-by: Shiju <shiju@nvidia.com>

* fix(supervisor): align VM identity startup with current APIs

Pass the optional rejection-log key for VM identity failures and keep
generic startup-write regressions free of VM identity constraints.

Repair the call sites after the branch rebase so the identity and cleanup
proposals compile against the current startup helpers.

Signed-off-by: Shiju <shiju@nvidia.com>

* fix(vm): restore inactive sandbox workload identity

Recover the persisted overlay owner before publishing stopped and terminal
sandboxes. Keep resources manageable when identity metadata is invalid.
Clarify fixed MicroVM ownership in policy-generation guidance.

Signed-off-by: Shiju <shiju@nvidia.com>

* test(vm): flush identity fixture before restart

Persist the canonical identity file before readiness and report the observed
exec, canonical and file-owner identities before comparing them.

Signed-off-by: Shiju <shiju@nvidia.com>

---------

Signed-off-by: Shiju <shiju@nvidia.com>
2026-10-03 20:03:39 +00:00
Drew Newberry 121930be07 fix(sandbox): replace lifetime exec cap with retry deadlines (#4105)
* fix(sandbox): replace lifetime exec cap with retry deadlines

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* docs(sandboxes): remove exec recovery overview change

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* docs(skills): remove exec recovery CLI skill change

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

---------

Signed-off-by: Drew Newberry <anewberry@nvidia.com>
2026-10-02 21:31:40 +00:00
Philippe Martin f7273e48f6 fix(providers): restore supervisor-backed GCP metadata discovery (#3973)
* fix(providers): restore supervisor-backed GCP metadata discovery

Relay the reserved metadata endpoint to the supervisor and restore project, account, and placeholder token responses from live provider state. Cover Google SDK discovery and repeated refresh with provider E2E tests.

Closes #3860

Signed-off-by: Philippe Martin <phmartin@redhat.com>

* fix(providers): preserve default metadata account without an email

Use the default account identifier when the optional service account email is missing or empty. Cover repeated SDK refresh for missing, empty, and configured email values with distinct providers for parallel E2E execution.

Signed-off-by: Philippe Martin <phmartin@redhat.com>

* test(sandbox): fix metadata relay lint and timeout

Signed-off-by: Philippe Martin <phmartin@redhat.com>

---------

Signed-off-by: Philippe Martin <phmartin@redhat.com>
2026-10-02 15:33:37 +00:00
Drew Newberry 8719fc9f37 fix(sandbox): restrict provider file mode (#4093)
* fix(sandbox): restrict provider file mode (fixes #4091)

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* refactor(sandbox): set provider file mode with safe API

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

---------

Signed-off-by: Drew Newberry <anewberry@nvidia.com>
2026-10-02 06:35:15 +00:00
Matthew Grossman 76cfd0e31d test(python): synchronize interactive exec TTY readiness (#4076)
* test(python): synchronize interactive exec TTY readiness

Wait for the complete readiness marker before streaming stdin so PTY echo cannot split the separately written TTY flags. Preserve pipe stream separation and verify consumed stdin and both output sentinels in TTY mode.

Fixes #4075

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

* test(python): reuse interactive exec readiness marker

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

---------

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
2026-10-01 21:47:40 +00:00
Evan Lezar 0e8d9e55f9 test(e2e): remove schema parity campaign (#3864)
Signed-off-by: Evan Lezar <elezar@nvidia.com>
2026-10-01 13:17:20 +00:00
Simon Scatton e21b7fd8cf chore(build): remove bundled Z3 support (#3275)
* chore(build): remove bundled Z3 support

Signed-off-by: Simon Scatton <sscatton@nvidia.com>
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>

* fix(build): preserve vendored Z3 for local gateway artifacts

Signed-off-by: Simon Scatton <sscatton@nvidia.com>

---------

Signed-off-by: Simon Scatton <sscatton@nvidia.com>
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
2026-10-01 12:08:33 +00:00
Drew Newberry 021400be8a refactor(auth): separate sandbox identity from TLS (#3110)
* refactor(auth): separate sandbox identity from TLS

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* docs(auth): clarify gateway mTLS behavior

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* test(auth): include workspace scope in TLS authorization checks

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* test(e2e): bound service auth sandbox names for large PIDs

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

---------

Signed-off-by: Drew Newberry <anewberry@nvidia.com>
2026-10-01 04:33:25 +00:00
John T. Myers 2935e9731b fix(gateway): delete finalized ephemeral sandboxes while connected (#3984)
Start driver cleanup after terminal finalization and retain disconnect fallback. Add detached success and failure e2e coverage across supervisor-based drivers.

Closes #3938

Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>
2026-10-01 00:09:08 +00:00
krishicks 9912d21d30 fix(e2e): keep locally built Kubernetes images off the chart's default registry (#3991)
679b19067 added global.image.registry (ghcr.io/nvidia) as the fallback for
empty per-image registries and split e2e image references into registry and
repository. Locally built images such as openshell/gateway:<tag> have no
registry host, so the chart rewrote them to ghcr.io/nvidia/openshell/* and
the k3d cluster could not pull them. Clear global.image.registry in the
Kubernetes e2e wrapper, which sets every image's registry explicitly.

Signed-off-by: Kris Hicks <khicks@nvidia.com>
2026-09-30 23:06:03 +00:00
Derek Carr 912a077bd6 feat(service): add bearer authorization passthrough (#3796)
* feat(service): add bearer authorization passthrough

Signed-off-by: Derek Carr <decarr@redhat.com>

* docs(sdk): add service authorization migration guide

Signed-off-by: Derek Carr <decarr@redhat.com>

* fix(server): remove stale version import

Signed-off-by: Derek Carr <decarr@redhat.com>

* docs(upgrade): remove service authorization SDK guide

Signed-off-by: Derek Carr <decarr@redhat.com>

* fix(e2e): relabel provider readiness TLS mount

Signed-off-by: Derek Carr <decarr@redhat.com>

* test(e2e): stabilize exposed service routing

Signed-off-by: Derek Carr <decarr@redhat.com>

* test(e2e): support HTTPS service routing

Signed-off-by: Derek Carr <decarr@redhat.com>

---------

Signed-off-by: Derek Carr <decarr@redhat.com>
2026-09-30 20:22:46 +00:00
Polite_realismandEvan Lezar b8ffe5244c test(podman): move podman_preflight into driver-podman integration tests (#3783)
* test(podman): move podman_preflight into driver-podman integration tests

podman_preflight verifies that openshell-driver-podman fails fast when
its Podman socket is unreachable. It only needs the standalone driver
binary, not a gateway, so it never fit the gateway-backed e2e-podman
harness it lived under and never ran anywhere in CI.

Move it into crates/openshell-driver-podman/tests/ as a plain Cargo
integration test. It now runs via the existing required workspace test
job with no special mise task, workflow step, or coverage exception.

Signed-off-by: politerealism <burdcat17@gmail.com>

* test(podman): make preflight diagnostics portable

Signed-off-by: Evan Lezar <elezar@nvidia.com>

---------

Signed-off-by: politerealism <burdcat17@gmail.com>
Signed-off-by: Evan Lezar <elezar@nvidia.com>
Co-authored-by: Evan Lezar <elezar@nvidia.com>
2026-09-30 06:05:18 +00:00
Drew Newberry 252882f37f feat(providers): serve sandbox config files on demand (#3832)
* feat(providers): serve sandbox config files on demand

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* docs(providers): defer managed file documentation

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* docs(providers): mark managed file api experimental

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* style(go): format provider profile fields

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(providers): preserve legacy environment with managed files

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

---------

Signed-off-by: Drew Newberry <anewberry@nvidia.com>
2026-09-30 01:41:32 +00:00
Eric Curtin cfcc3733bd fix(e2e): stop sandbox leaks from async Drop cleanup (#3750)
* fix(e2e): stop sandbox leaks from async Drop cleanup

Closes #2922

SandboxGuard::Drop spawned a detached thread to delete the sandbox.
The thread got killed with the test process before the delete
finished. Switch to a blocking command in Drop, like ManagedCleanup
already does. Also wrap two tests' manual cleanup in RAII guards so
a panic does not leak a sandbox.

Signed-off-by: Eric Curtin <eric.curtin@docker.com>

* test(e2e): arm sandbox guards before create

Address review: install guards with explicit names first.

Signed-off-by: Eric Curtin <eric.curtin@docker.com>

---------

Signed-off-by: Eric Curtin <eric.curtin@docker.com>
2026-09-29 10:10:29 +00:00
Philippe MartinandJohn Myers 9cb72baa2e feat(docker): support corporate proxy CA bundles (#3549)
* feat(docker): support corporate proxy CA bundles

Closes #3545

Validate and stage operator-owned proxy CA bundles for Docker supervisors, add corporate proxy E2E coverage, and document the trust contract.

Signed-off-by: Philippe Martin <phmartin@redhat.com>

* fix(docker): validate proxy config on startup

Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>

* test(docker): use the E2E workload image for proxy tests

Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>

* test(docker): generate strict corporate proxy certificates

Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>

* test(docker): surface intercepted TLS fixture errors

Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>

* test(docker): drain buffered TLS proxy data

Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>

* test(docker): relay intercepted HTTP deterministically

Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>

---------

Signed-off-by: Philippe Martin <phmartin@redhat.com>
Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>
Co-authored-by: John Myers <9696606+johntmyers@users.noreply.github.com>
2026-09-29 05:15:02 +00:00
Drew Newberry acbac9cb79 feat(sandbox): add main restart policy (#2798)
* feat(sandbox): add main restart policy

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(sandbox): harden policy-driven restarts

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(sandbox): address restart review feedback

Signed-off-by: Drew Newberry <385+drew@users.noreply.github.com>

* fix(sandbox): port restart policy to current runtime

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* perf(sandbox): restart promptly after terminal delivery

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

---------

Signed-off-by: Drew Newberry <anewberry@nvidia.com>
Signed-off-by: Drew Newberry <385+drew@users.noreply.github.com>
2026-09-29 02:18:30 +00:00