mirror of
https://github.com/NVIDIA/OpenShell.git
synced 2026-10-29 07:23:25 +08:00
fa2d381eaa9e917fd074b4bc49efea8a2ad3e43e
186
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Jesse Jaggars
|
2c31ac7e60 |
feat(network): support additional destination CAs (#3292)
* feat(network): support additional destination CAs Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(network): harden additional CA propagation Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(network): narrow additional CA review scope Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(network): harden additional CA rotation Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * feat(network): complete additional CA delivery Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(docker): remove stale runtime fixture field Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(network): adapt additional CA series to upstream main Resolve rebase fallout against current upstream main after the series was rebased 214 commits forward: - follow upstream #3110: supervisor→gateway TLS is CA-only, so configure_gateway_tls and its isolation tests drop the client identity - follow upstream tls_secret_names reduction to [String; 1] in the Podman driver while keeping the network trust bundle delivery - port [openshell.supervisor.network] rendering into the openshell.gatewayConfigToml serializer in _toml.tpl - adapt tests to upstream struct/function changes (DriverStartupContext, SandboxRunConfig, IsolationSpecInput, supervisor_pod, VM bootstrap image) - drop conformance-artifact-prefix plumbing removed upstream Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * refactor(network): adopt upstream additional CA bundle surface Unify the additional destination CA series on the mechanism merged to main in #3996: one operator pre-assembled PEM bundle, the --additional-ca-bundle supervisor flag, and the /etc/openshell/tls/additional/ca-bundle.pem staging path. - Replace the gateway-level additional_ca_cert_paths list with a single additional_ca_bundle file validated and staged at gateway startup. - Restore the upstream supervisor argument path and fold-into-roots behavior; drop the --network-additional-ca-bundle and --network-additional-ca-digest contracts. - Keep gateway normalization bounds, content-addressed staging, digest-based driver reconciliation, and stop/start rotation; the freshness check now runs at the gateway/driver boundary instead of the supervisor boundary. - Reject additional_ca_bundle in operator driver JSON config; standalone drivers self-source the bundle from --sandbox-additional-ca-bundle with the same validation. - Extend the Kubernetes ConfigMap name digest segment to keep the full 63-byte budget after the prefix rename. - Box the persisted-sandbox reconciliation futures so callers stay under the clippy::large_futures threshold. Updates Docker, Podman, Kubernetes, and VM delivery, Helm chart and tests, e2e wiring, gateway configuration docs, and the cluster debugging skill to the single mechanism. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(ci): make gateway startup tests wrap-tolerant and refresh example lockfile The network trust startup tests asserted on raw substrings of gateway diagnostics, but the CLI renderer wraps output at a fixed width with continuation markers and hard-breaks long paths. Compare assertions on whitespace- and continuation-squashed text so they hold regardless of wrap position. Format the e2e workspace (separate cargo workspace, not covered by the root cargo fmt run) and refresh examples/isolation-backend-v8/Cargo.lock for the libc dependency added to the network trust crate. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(podman): reuse TLS CA secret during stopped-pair trust replacement The additional CA e2e never ran in CI before the first maintainer test dispatch, and its local container inspection was broken in two ways: it located the supervisor container through e2e network membership (the supervisor is host-networked or runtime-bridged, so the filter matched the CI job container instead), and it required the gateway mTLS CA as a bind mount (Podman delivers it as a container secret when user namespaces remap UIDs). Select the supervisor by managed labels and accept the secret delivery form. The lifecycle leg then exposed a driver bug: stopped-pair trust replacement re-created the per-sandbox TLS CA secret, colliding with the secret still owned by the renamed stopped pair. Mirror the token and resolver secret handling: require and reuse the existing secret during replacement instead of creating it. Validated end-to-end with the Podman additional CA e2e lane. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(server): restore upstream recovery semantics for start-request failures Recovery no longer marks sandboxes Error/InvalidPersistedSandbox when driver_start_request fails during the startup or lifecycle sweeps. Gateway configuration problems (for example a missing launch signer) must not permanently fail unrelated sandbox records; the sweep logs a warning and skips them, matching the behavior before the additional CA work. Sandbox-record failures surface through the normal start path. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * refactor(podman): refresh stale supervisors without replacing workloads Replace the stopped-pair trust reconciliation with supervisor-only recreation. The destination-trust generation label moves from the workload to the supervisor container, and start_sandbox recreates only the supervisor when its generation is missing or stale. The workload container, its volumes, its secrets, and its workspace data are never replaced: the additional CA bundle is mounted only into the supervisor. Running sandboxes are never gated on trust labels, and recovery without fresh launch credentials keeps the existing supervisor. The intentional -removal fence machinery in the watcher existed only for workload replacement and is removed with it. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * refactor(docker): drop workload replacement for destination-trust changes Delete the stopped-workload trust reconciliation path and all of its machinery: the trust-generation label on workload containers, the generation match check in the start fence, the stopped-sandbox replacement, the bounded workspace archive copy with its 512 MiB cap and regular-file-only validation, and the boundary-state rollback helpers. No replacement machinery is needed: the trusted supervisor container is already re-created from the gateway's current configuration on every start via ensure_control_process_for_container, so stop/start picks up the current additional CA bundle while the workload container, its volumes, and its workspace data are never replaced. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * docs(configuration): drop reconciliation wording from additional CA lifecycle The stop/start lifecycle description already matches the supervisor-only refresh model; remove the two remaining references to restart reconciliation of workload containers. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(network): port additional CA test fixtures to the crypto interface Route the gateway TLS isolation test PKI, the destination CA source fixtures, and the gateway startup trust tests through openshell_crypto::pki now that rcgen access is centralized there: generate_keypair/self_signed/signed_by replace the direct rcgen calls and CertifiedKey field access follows the wrapper's shape. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(e2e): repair additional CA lane for Docker and non-container builds The first maintainer test dispatch revealed two latent issues in the additional CA e2e that no local run had exercised: the Podman gateway CA secret helper referenced ContainerEngine without the container feature gate, breaking Kubernetes and VM builds of the test, and the Docker branch asserted a client ca.crt bind mount that the Docker driver never creates. Gate the helper like its callers, and assert Docker's actual delivery path instead: the gateway mTLS CA staged as tls/ca.pem inside the read-only supervisor state volume, selected through the OPENSHELL_TLS_CA environment pointer. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(helm): restore alphabetical README ordering for supervisor keys The helm:docs check regenerates the chart README with helm-docs, which orders rows alphabetically; a conflict resolution had placed supervisor.network.additionalCaConfigMapName after the supervisor.podSecurityContext rows. Regenerate with mise run helm:docs. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * chore: refresh testkit lockfile for openshell-core libc dependency Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(driver-podman): stage operator CA bundle as driver-owned artifact Standalone Podman drivers accepting --sandbox-additional-ca-bundle previously kept the operator's source path as the trust artifact path. The immutable-artifact verification performed on every create/start requires exactly mode 0444, which an operator-owned source path cannot guarantee, so a readable-but-writable source failed reconciliation. Read the operator bundle with the same bounded, no-symlink validation, then stage the normalized material into a driver-owned artifact inside a private tempdir held for the driver process lifetime. The artifact is created 0600, synced, and only then restricted to 0444 so no observer ever sees a writable artifact. The operator source and its permissions are never referenced after startup. Addresses GATOR-291f43b2-02. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(driver-podman): pin retained workload image during supervisor refresh Supervisor-only trust reconciliation re-entered create_sandbox_with_existing_token, which re-pulled and re-resolved the workload image tag even though the workload container is retained. If the tag moved to a different image between stop and start, identity resolution and child-environment construction would reflect the new image while the retained workload still runs the original one, leaving mode-0600 bootstrap files the retained user cannot read. During a refresh with a retained workload, inspect the stopped container and pin to the immutable image ID it was created from (Config.Image); never pull or resolve the request tag. The requested reference is still used for child metadata so the retained workload's environment is byte-identical to its original launch. Addresses GATOR-291f43b2-01. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * test(e2e): verify host-only VM trust delivery and drop curl dependency The additional-CA lane's VM assertions read the destination CA out of the guest overlay and a guest-side supervisor-argv marker — artifacts from the guest-supervisor design that no longer exist now that the VM driver stages the CA host-only and runs the supervisor on the host. Inspect the driver-staged host artifact (digest-compared against the source), assert the guest overlay contains no CA material at the old mount paths, and keep the serial-console no-disclosure check. The workload validation scripts also ran curl, which exists only in the container-engine fixture image; the Kubernetes and VM lanes use the uv-based workload image and died instantly with "canonical main process already finished". Rewrite the main-process, exec, and lifecycle validation scripts in python3 (present in every lane's workload image). The network-policy binary identity is the resolved /proc/<pid>/exe target, so the versioned python3.12 binaries are allowlisted rather than the unversioned symlinks. The hostname mismatch check keeps its semantics: same reachable fixture, mismatched TLS SNI, and the failure must specifically be hostname verification. Addresses GATOR-291f43b2-04. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(kubernetes): allow admission-added labels on destination-trust ConfigMaps The destination-trust ConfigMap validation compared the whole label map for equality. A mutating admission webhook that attaches an organizational label to the created ConfigMap made the create response fail validation, and the retained ConfigMap then failed every subsequent create/start through the existing-object path, blocking sandbox launches despite the ownership labels, immutable flag, digest, and certificate bytes remaining correct. Validate the gateway-owned label pairs individually and allow unrelated labels; the exact trust-material checks are unchanged. Covered by a regression test with an admission-added label and a missing gateway-owned label. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(podman): close supervisor-refresh rollback gaps The supervisor-only trust refresh renamed the previous supervisor to a random rollback name and then performed the token-secret lookup. A transient lookup failure returned with the supervisor still renamed, where a normal delete could never find it, and the rollback rename and final rollback removal ignored their own failures, leaving orphans with attached control and channel resources. Complete every fallible preparation before the rename. On a failed replacement, restore the canonical name; if the restore itself fails, return an error naming the rollback container so the orphan stays identifiable — a later retry still succeeds because the canonical name is absent and the replacement path recreates a fresh supervisor. Treat not-found during the final rollback removal as success and name the orphan in both the log and the error on other removal failures; a retry observes the fresh canonical supervisor and resumes the normal start path. Covered by failure-injection tests for each transaction boundary: preparation failure before any rename, replacement failure restoring the canonical name, and restore failure naming the orphaned rollback container. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * docs: address destination-trust review corrections The gateway-configuration RFC documented an additional_ca_cert_paths list that the strict configuration parser never accepted; document the additional_ca_bundle single-path setting the parser actually enforces. The debug-openshell-cluster skill named the wrong generation annotation (openshell.ai/network-supervisor-trust-generation) and described digest argv and network-init canonicalization checks the implementation does not perform. Correct it to openshell.ai/additional-ca-generation and describe the actual delivery: the supervisor Pod mounts the immutable ConfigMap read-only and receives --additional-ca-bundle for the mount path, with the driver validating the digest annotation and exact bytes per creation. Classify the destination-trust generation hashes as deterministic identity derivation rather than authenticity proofs, following the openshell-crypto exclusions table: trust decisions come from the gateway's normalization, immutable staging, exact-byte validation, and TLS verification. Record the classification in the crypto README and at each hashing site. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(network): reconcile additional CA tests with suspend/resume and auth changes Upstream's suspend/resume lifecycle took StartSandboxRequest fields 7 and 8, so the durable-snapshot field moves from 7 to 9; update the wire-format test to assert the field-9 key (0x4a) and initialize the new resume controls. Repair the cli test module after the auth opt-in test merged alongside the environment-guard helpers, and initialize the gateway driver-startup context's network-trust fields in the shared supervision test. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(server): box start-recovery futures for clippy large-futures The durable-snapshot field grows StartSandboxRequest enough that the start-recovery match and the recovery-expiry test exceed the large-futures limit under -D warnings. Box::pin both futures, matching the pattern the recovery path used before the suspend/resume rework. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> --------- Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> |
||
|
Drew Newberry
|
e5adf6f484 |
feat(supervisor): recover shared instances with hash-based HA (#4864)
* feat(supervisor): recover shared instances with hash-based HA Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(supervisor): keep HA coordination out of compute drivers Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(supervisor): fence pool admission and reconciliation races Signed-off-by: Drew Newberry <anewberry@nvidia.com> * feat(supervisor): keep pool mode stable across replica scaling Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(supervisor): default shared deployments to pool mode Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(supervisor): fence refused launches and remove legacy mode Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(supervisor): confirm live owners and fence stale observations Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(supervisor): preserve takeover cause log formatting Signed-off-by: Drew Newberry <anewberry@nvidia.com> * feat(helm): add memory-based supervisor autoscaling Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(supervisor): retain acknowledgements and fence streamed input Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs(supervisor): consolidate experimental multi-tenant guidance Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs(supervisor): simplify multi-tenant supervisor guide Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(supervisor): simplify pool member identity Signed-off-by: Drew Newberry <anewberry@nvidia.com> * feat(helm): make supervisor affinity configurable Signed-off-by: Drew Newberry <anewberry@nvidia.com> * feat(helm): make supervisor disruption budget optional Signed-off-by: Drew Newberry <anewberry@nvidia.com> * test(supervisor): align HA fixtures with suspend support Signed-off-by: Drew Newberry <anewberry@nvidia.com> --------- Signed-off-by: Drew Newberry <anewberry@nvidia.com> |
||
|
Drew Newberry
|
9a85667d0d |
feat(kubernetes): host sandboxes on the shared supervisor (#4702)
* feat(sandbox-backend): route runtime connections to shared supervisor sessions A supervisor shared by several sandboxes needs one listener for every session's runtime-initiated Sandbox Protocol connections. The runtime now sends a fixed, bounded routing preface (magic, version, length, and an opaque route handle) before its pinned TLS handshake when its bootstrap names a route. RuntimeRouter treats the handle only as a hint that selects which generation-pinned TLS identity to demand, closes unknown, removed, and replaced handles before any handshake, bounds unauthenticated handshakes globally and per route, and queues an authenticated connection only for its own session, so no peer can displace a live channel. A routed session waits up to the bootstrap grace for its first connection, because it is registered before its workload Pod is scheduled. The session_control module defines the trusted contract between a compute driver and a shared supervisor: registering one generation's runtime descriptor and gateway-issued credentials, reading its status, and removing it, over mutually authenticated TLS 1.3. Dedicated descriptors and bootstrap listeners keep their encodings. Signed-off-by: Drew Newberry <anewberry@nvidia.com> * feat(supervisor): host gateway-registered sessions in one process openshell-supervisor experimental-host replaces the idle bootstrap process with a session host. A mutually authenticated session-control API accepts registrations only from the admission certificate holder, the gateway. Each registration starts a SessionHost session that acts only with the gateway credentials issued for its generation, keeps its proxy CA, trust directory, and SSH socket under a private directory, and receives its runtime connections through the routed listener. Status reports each session's own readiness; the health socket keeps reporting process health only. Removing a session runs on a task the host owns, so it completes when the caller disconnects, concurrent removers share its result, and the route stays reserved until the session has ended and its files are gone. An identical registration is accepted again and a different one for a registered route is refused. Sessions get no upstream proxy, extra CA bundle, or provider workload identity; the Kubernetes driver refuses configurations that need them. Signed-off-by: Drew Newberry <anewberry@nvidia.com> * feat(server): issue a TLS-only private PKI from generate-certs generate-certs --tls-only creates just a server and a client TLS Secret from a fresh CA, with the same create-if-missing contract and a recovery hint for partial state. The Helm certgen hook uses it for private PKIs that must not carry a sandbox-JWT signing key. Signed-off-by: Drew Newberry <anewberry@nvidia.com> * feat(kubernetes): supervise gateway-topology generations as shared sessions With experimental_supervisor_topology = "gateway", each new runtime generation becomes a session on the gateway's shared supervisor instead of a supervisor Pod and boundary Service of its own. The generation records its placement and session route on its Sandbox resource when it starts, and every later decision reads that record, so changing the setting never reinterprets a running generation and unknown records fail closed. The runtime identity ends with the session route, which no Pod UID can equal, so Pod authentication cannot vouch for a shared generation. Create and start add a per-sandbox egress allowance to the Pods the shared supervisor's Service selects, bind it into the boundary evidence, and register the session over session control before releasing the workload; registration does not wait for the runtime, so create still returns before the gateway persists the generation. Readiness follows the session's own status. Stop, restart, and delete remove only that session, and reconciliation suspends a workload whose session is gone and removes sessions that no current generation may run. Configurations with an upstream proxy or provider SPIFFE identity are refused. Closes #4700 Signed-off-by: Drew Newberry <anewberry@nvidia.com> * feat(helm): deploy and wire the shared supervisor With the gateway topology enabled, the chart runs the session host as a single-replica Deployment with its Service and a NetworkPolicy that admits session control only from the gateway and runtime connections only from sandbox workload Pods. The certgen hook issues a private session-control PKI whose client credential only the gateway mounts, the gateway may read only that Service, and the chart renders the matching experimental_shared_supervisor table. The supervisor Pod has no service account token. Settings the shared supervisor cannot isolate, and a hand-written connection table, fail at render time. Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(ocsf): attribute events to session-bound contexts A supervisor process that hosts several sandboxes cannot use one process-wide OCSF context: every event would carry the wrong sandbox identity, and middleware requests read their sandbox identity from the same context. Builders now accept a ContextRef, either a borrowed context or a context shared by a hosted session, so per-session contexts need not be leaked. ctx() returns the context bound to the calling thread when present and falls back to the process-wide context otherwise. OCSF JSON output settings can likewise be bound per thread, and OpenTelemetry can parent a span from an explicit W3C trace context instead of process environment. A dedicated supervisor keeps using the process-wide context. Signed-off-by: Drew Newberry <anewberry@nvidia.com> * test(e2e): run two sandboxes on one shared Kubernetes supervisor The e2e:kubernetes:shared-supervisor lane enables the gateway topology and creates two sandboxes with different policies and provider credentials. It checks that one shared supervisor process hosts both, that each session enforces its own policy and injects its own credential, that stopping one leaves the other working, that the routed listener refuses stale and unknown routes and drops a peer that names a live route without its pinned identity, and that a restart registers a fresh session on the same process. Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(supervisor): scope gateway credentials and network state to a session The supervisor kept its gateway bearer, Sandbox Protocol bearer, and renewal task in process-wide slots, and the network supervisor read provider token grants, proxy CA files, and the gateway identity from process environment. A second sandbox in the same process would have replaced or rotated the first sandbox's credentials, reused its provider access tokens, and exposed its denials through policy.local. GatewayCredentials now holds one session's gateway and Sandbox Protocol credentials and their renewal; every gateway client receives a GatewayEndpoint that pairs the URL with them, including log push, ConnectSupervisor and its redirects, and extension-credential rotation. run_networking takes a NetworkSession with the sandbox identity, the session's gateway access, its Workload API socket, its policy.local denial source, and its proxy CA files. Provider token grants use a cache and HTTP client owned by the session and fail closed without a Workload API socket. run_session supervises one generation from an explicit SessionConfig and owner-provided SessionControls. run_sandbox_with_backend remains the dedicated entry point: it keeps the process-wide OCSF context, reads the same environment, binds readiness endpoints to session readiness, and stops on SIGTERM or SIGINT. The binary shares one set of credentials between log push and supervision, as before. Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs: describe the experimental shared supervisor Document the gateway supervisor topology, its chart-owned connection settings and limits, and how to debug and test it. Signed-off-by: Drew Newberry <anewberry@nvidia.com> * feat(supervisor): host concurrent sessions on session-bound runtimes SessionHost runs several supervisor sessions in one process. Each session gets its own Tokio runtime, and every thread that runtime starts, plus the thread driving it, is bound to the session's OCSF context, OCSF JSON settings, log sink, and denial log. Tasks, blocking work, reconnects, and credential renewal spawned by a session therefore keep its identity, while threads outside any session carry none and push no logs. Gateway credentials stay explicit per session. Removing a session stops its workload gracefully, or cancels it at once while it is still starting, then shuts down only its runtime and revokes its gateway credentials. The key stays registered until the session thread has finished, so the same generation cannot gain a second owner while one is stopping. A panicking session reports a failure, clears its readiness, and is revoked the same way. Hosted sessions get a private proxy trust directory and serve policy.local denials from their own log. A focused test runs two sessions concurrently and checks that nested and blocking tasks see their own identity, gateway requests carry their own bearer, log lines reach only their own stream, and removing one leaves the other supervising. Closes #4699 Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix: tighten shared supervisor naming, removal deadlines, and route probes The shared supervisor's resource name appended its suffix before truncating, so a release fullname of 51 or 52 characters produced the gateway Service's own name. A fullname without room for the suffix now keeps a prefix and a hash of the whole fullname. Removing a session waits up to 20 s for its workload to stop and 5 s for the session runtime to shut down, but the driver gave DELETE the same 15 s budget as other session-control calls and reported normal removals as failures. The contract now defines the removal grace and a 45 s removal deadline, and deleting a sandbox no longer waits on a session whose workload is already going away. The shared host runs no supervisor OTLP trace or OCSF exporter. The driver warns at startup when the gateway configures that export, and the docs state the limit. experimental-host keeps its Unix-only imports in its Unix entry point. The e2e route probe fails when the supervisor neither refuses nor challenges a connection, or keeps a challenged connection open after a malformed TLS answer, and it checks that every probe reached the supervisor Pod. Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(supervisor): route shared supervisor crypto through openshell-crypto Main now routes first-party TLS providers and digests through the openshell-crypto facade. Build the session-control TLS configurations and the routed-runtime test server from its configuration provider, install the experimental host's default provider through it, and hash shared session registrations with its SHA-256 instead of a direct sha2 dependency. Signed-off-by: Drew Newberry <anewberry@nvidia.com> * feat(supervisor): report shared supervisor health over TCP experimental-host takes --health-port instead of a Unix health socket, matching experimental-bootstrap. It binds the wildcard health listener after the session-control and runtime listeners and accepts and closes connections while the process runs. The chart probes the named container port health (5502) with native kubelet tcpSocket checks; session control stays on 5501, the Service exposes only runtime and session control, and /run/openshell remains for per-session files. Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(supervisor): probe shared supervisor health on session control The shared supervisor's kubelet tcpSocket probes now connect to the mTLS session-control listener (5501) instead of a separate health listener. Drop the experimental-host --health-port flag, its accept loop, and the chart's 5502 container port. Probes open and close a TCP connection; every session-control request still requires mTLS. Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(supervisor): use shared mode flags for the session host Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(supervisor): describe the shared host module by its mode Drop the experimental label from the shared host module documentation so it matches the --mode=shared entry point. No behavior changes. Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(test): preserve shared supervisor workspace overlays Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(supervisor): name hosted lifetimes sandbox instances A sandbox instance is one running generation of a stable Sandbox resource. Gateway transport connections and the gateway supervisor sessions they carry can reconnect within one instance, so the hosted object, its configuration, controls, host, and per-instance network, denial, and token-grant state now use the sandbox instance name. Rename session_host to sandbox_instance_host, run_session to run_sandbox_instance, and SessionConfig, SessionControls, SessionHost, and related hosted types to their SandboxInstance equivalents. Gateway supervisor sessions, SSH sessions, and authentication session types keep their names. No wire, configuration, or lifecycle behavior changes. Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(supervisor): launch sandbox instances from one config and prepared bindings Replace SandboxRunConfig and its field-by-field copy into SandboxInstanceConfig with a single launch configuration. Live per-instance state (gateway credentials, OCSF context and JSON settings, log sink, and denial log) is now an explicit SandboxInstanceBindings value that the owner prepares once, instead of placeholders the runner or host overwrote. The dedicated binary loads launch inputs from its arguments and environment and shares one GatewayCredentials with log push; run_sandbox_with_backend keeps the process-wide OCSF context, health endpoints, and signal handling. The instance host binds the same SandboxInstanceBindings to its runtime threads and revokes their credentials when the instance ends. Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(supervisor): use session-control wire types for instance status and trace context Drop the supervisor's duplicate SandboxInstancePhase, SandboxInstanceStatus, and TraceContext families. The sandbox instance host reports SandboxInstanceStatusReport directly, with its key as the route, so the shared host no longer converts or re-keys status, lists, or removal results. SandboxInstanceConfig carries RegistrationTraceContext and the dedicated process reads it through trace_context_from_environment. No wire definitions change. Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(supervisor): update Windows host proxy test config Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(supervisor): assign shared sandbox instances over a supervisor-initiated gRPC stream Replace the private HTTP/JSON session-control API, which the gateway dialed on the shared supervisor, with a supervisor-initiated stream to the gateway. The gateway owns the desired set of sandbox instances; the shared supervisor attaches with a gateway-scoped host certificate, receives complete assignment snapshots with session-scoped revisions, reconciles its instance registry to each one, and reports what it observes. Losing the stream never stops an instance; only a later snapshot that omits it does. - Add the private openshell.supervisor.host.v1 protocol, served on the gateway listener but authorized only for the supervisor host certificate, excluded from SDK generation and reflection, and classified in the schema inventory. - Add SupervisorHostController in openshell-core: uninitialized until the driver derives the desired set once, compare-and-replace full resyncs, capacity bound, session fencing by connection epoch, revision-bounded acks, bounded sends. - Classify TLS peers per connection against the ordinary client CA and the supervisor host CA from one configuration snapshot; a chain matching neither or both is refused, and host connections never become users. - Persist each shared generation's launch material in its supervisor Secret so a restarted gateway rebuilds assignments from Sandbox resources, and withdraw instances on stop, delete, and suspend. - Issue the host identity with `generate-certs --client-only`; the gateway mounts only the CA. Drop the control port, server certificate, and gateway ingress rule from the chart; require gateway TLS for the topology. Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(supervisor): authenticate the shared supervisor stream with a gateway-minted supervisor token Replace the shared supervisor's host CA and client certificate with a supervisor token: a bearer the gateway mints from its existing launch-signing key under a profile of its own (typ openshell-supervisor+jwt, subject openshell-supervisor:<gateway_id>, no sandbox, generation, or epoch claims). The supervisor dials the gateway's Attach stream over ordinary server-authenticated TLS and presents the token; the gateway verifies it with the same key infrastructure, grants only the assignment stream, refuses every other credential on it, and ends a stream when the token that opened it expires. The gateway renews the token at half its lifetime and publishes each renewal into the supervisor host controller; the Kubernetes driver's only part in authentication is delivery, replacing the token in the chart-created <release>-supervisor-token Secret, which the supervisor mounts and re-reads on every connection attempt. The gateway Role may get and patch only that Secret. Remove the host-only TLS realm classification, the supervisor host CA mount and config, certgen --client-only, and the supervisor host PKI; gateway TLS returns to its pre-feature behavior and remains required for the topology. Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(helm): remove redundant certgen condition Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(supervisor): extract gateway-owned shared supervision into openshell-supervisor-control Move the shared supervisor control types out of openshell-core and openshell-sandbox-backend into a new openshell-supervisor-control crate: the SupervisorHostController (desired set, revision fencing, host sessions, observations), the SandboxInstanceRegistration launch material and SandboxInstanceStatusReport mapping, and a new SharedSupervision coordinator that owns the control contracts a compute driver previously implemented itself: validated persist-before-assign launches, recovery from durable records with the compare-and-replace baseline captured before the driver lists them, keep-previous-set on any failed or mismatched read, and one interpretation of host observations for readiness and bootstrap. Supervisor token delivery keeps its retry and newer-value policy in the crate; the write itself stays with the driver. The gateway now constructs the coordination: ComputeDriverBuildContext exposes shared_supervision() and the server keeps the controller for the Attach stream and token minting, replacing the driver-created controller that was installed into the gateway. The Kubernetes factory injects it through KubernetesComputeDriver::with_shared_supervision; new() stays the dedicated-topology constructor and refuses a configuration that selects the gateway topology without the gateway's coordination. The Kubernetes driver keeps its platform work: placement annotations, the shared supervisor Service and egress NetworkPolicy, launch Secret create and read (bytes only), its own fence and Pod evidence, and the token Secret patch. The supervisor process consumes the moved registration and status types unchanged. Wire protocol, storage JSON, annotations, and the supervisor token profile are unchanged; the private protobuf stays generated in openshell-core and is re-exported by the new crate. Signed-off-by: Drew Newberry <anewberry@nvidia.com> --------- Signed-off-by: Drew Newberry <anewberry@nvidia.com> |
||
|
Drew Newberry
|
32f026442e |
feat(sandbox): initiate supervisor connections across compute drivers (#4656)
* feat(sandbox): support sandbox-initiated supervisor connections Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(sandbox): consolidate control transport setup Signed-off-by: Drew Newberry <anewberry@nvidia.com> * test(sandbox): remove prototype-specific unit tests Signed-off-by: Drew Newberry <anewberry@nvidia.com> * feat(kubernetes): migrate legacy control flow on restart Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs: remove prototype changes from published docs Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(sandbox): adapt reverse control to published backend protocol Signed-off-by: Drew Newberry <anewberry@nvidia.com> * feat(sandbox): initiate supervisor connections across compute drivers Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(docker): stage reverse control socket for the supervisor identity Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(sandbox): keep runtime transports behind the backend Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(docker): mount the runtime control volume read-only The sandbox runtime only dials the supervisor-owned control socket, so it does not need write access to the control volume. Mount it read-only in the runtime container and stop staging the control directory through the runtime bundle; the supervisor stager already initializes it through its own writable mount. Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(kubernetes): separate legacy and runtime-initiated control services Keep os-boundary selecting the workload and add os-supervisor for runtime-initiated connections. Bind the mode marker to its runtime generation so a restart by an older gateway does not inherit stale mode metadata. Preserve the legacy Service across creation and restart, validate the active endpoint, and drop the Service patch permission. Signed-off-by: Drew Newberry <anewberry@nvidia.com> --------- Signed-off-by: Drew Newberry <anewberry@nvidia.com> |
||
|
Varsha
|
1cb9ef8ab4 |
refactor(crypto): introduce a generic interface for crypto libraries and refactor AWS-LC (#3351)
* refactor(crypto): centralize AWS-LC behind backend interface Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com> * refactor(crypto): route new host proxy tests through facade Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com> * fix(crypto): make backend boundary check portable Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com> * fix(crypto): route Google JWT signing through facade and verify context dispatch Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com> * refactor(crypto): propagate digest backend failures Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com> * refactor(crypto): route Vault TLS fixtures through facade Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com> * refactor(crypto): adapt isolation runtime callers after main rebase Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com> * fix(crypto): declare sandbox dependency and remove boundary lint Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com> * refactor(crypto): support independent backends and backend-free CA import Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com> * chore(crypto): refresh feature suite lockfile after rebase Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com> * fix(crypto): route runtime integrity and entropy through backend Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com> * chore(crypto): refresh lockfile after upstream rebase Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com> * fix(crypto): route VM socket entropy through backend after rebase Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com> * docs(crypto): document architecture and contributor boundaries Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com> * test(crypto): adapt upstream fixtures to backend facade Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com> * test(crypto): route relay TLS fixture through facade Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com> * test(crypto): adapt service bearer TLS fixture after rebase Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com> * fix(crypto): route provider profile TLS initialization through facade Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com> * fix(crypto): adapt upstream VM and supervisor paths to facade Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com> * ci(crypto): enforce backend independence in branch checks Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com> * test(crypto): adapt upstream token grant TLS fixture Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com> * fix(crypto): avoid format collect in executable hashing Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com> * fix(crypto): address provider selection and decryption review Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com> * fix(crypto): adapt upstream operator TLS paths after rebase Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com> * fix(crypto): adapt upstream TLS fixtures and protocol tests Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com> --------- Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com> |
||
|
Piotr Mlocekanddivesh
|
604b9f66f6 |
feat(config): deliver gateway configuration over the supervisor session (#4320)
Supervisors poll the gateway for configuration every 10 seconds. This PR adds an opt-in push mode (config_delivery_mode = "push"): supervisors still start by polling, and once their ConnectSupervisor session is accepted, the gateway delivers every later change over it and supervisors apply it and report the result. poll stays the default, and older supervisors and gateways keep polling. --------- Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> Signed-off-by: divesh <dgude@nvidia.com> Co-authored-by: divesh <dgude@nvidia.com> |
||
|
Florent BENOIT
|
a4c848adbd |
feat(vm): add host bind mount support via virtiofs (libkrun) (#3463)
* feat(vm): add host bind mount support via virtiofs (libkrun) Introduce host directory bind mounts for libkrun VM sandboxes using the virtiofs4 API with simplified permission semantics. The gateway accepts `enable_bind_mounts` in the `[openshell.drivers.vm]` TOML table and forwards it to the VM driver subprocess. Mounts are specified per-sandbox in driver_config.mounts[] with source, target, optional read_only (default: read-only), and an optional `type` that must be `bind`, so Docker and Podman bind entries (without `selinux_label`) parse unchanged. At sandbox validation the driver rejects mounts unless bind mounts are enabled and resource admission is disabled. It rejects GPU and other QEMU launches, sources with control characters, and more mounts than libkrun has interrupt lines for (32 on aarch64, 3 on x86_64), and checks targets against reserved VM-internal directories and each other (no nested targets). The QEMU check runs again once the launch backend is resolved. Host source directories are checked at provisioning, before image preparation, so a briefly missing source fails provisioning visibly instead of denying recovery of a persisted sandbox. Missing, inaccessible, symlinked, non-directory, and unreadable sources are rejected: libkrun does not follow a final symlink, and on macOS it panics during guest boot when it cannot open the share root. The driver generates virtiofs tags and always writes a guest-side manifest into the overlay disk, together with the init drop-ins, so an image-supplied manifest can never drive guest mounts. The launcher parses its tab-separated --vm-mount arguments from the right, so a source can never shift the tag or mode, and the driver logs each shared directory at launch. Guest init mounts the shares after every /sandbox ownership fixup and the init drop-ins, so recursive chowns never reach host files. It refuses targets that traverse symlinks in the image, hands newly created mount point parents under /sandbox to the sandbox user, and skips an empty manifest. krun_add_virtiofs4 is resolved optionally, so older libkrun runtimes still boot sandboxes without mounts. The kernel config lists CONFIG_FUSE_FS and CONFIG_VIRTIO_FS explicitly; the pinned libkrunfw base config already enables both. A vm_bind_mount e2e test checks a read-write share round-trips data with the host, accepts a new file, directory, and nested file that the host then sees, and that a read-only share rejects writes. libkrun on rootless Linux refuses file creation by a guest user whose UID/GID differ from its own, so the VM e2e gateway runs the sandbox as the gateway user there. It enables bind mounts like the Docker and Podman e2e gateways. The runtimes docs cover the mount limit, the symlink restriction, the guest-kernel exposure, and the rootless Linux UID/GID note. The debug-openshell-cluster skill lists VM bind mount checks. Signed-off-by: Florent Benoit <fbenoit@redhat.com> * fix(nix): refresh vm-runtime hashes to the libkrun v1.19.4 runtime The Nix vm-runtime package fetches the rolling vm-runtime release with fixed hashes pinned on 2026-09-16. The release was rebuilt on 2026-09-21 after the libkrun v1.19.4 / libkrunfw v5.6.1 bump (#3451), but the hashes were not updated, so CI kept substituting the 2026-09-15 runtime (libkrun v1.17.4) from the binary cache. That libkrun lacks krun_add_virtiofs4, which VM bind mounts need. Point the hashes at the current release assets. Signed-off-by: Florent Benoit <fbenoit@redhat.com> --------- Signed-off-by: Florent Benoit <fbenoit@redhat.com> |
||
|
ShijuandJohn Myers
|
29a2b8e7d9 |
feat(providers): report bounded token-grant failures (#4427)
* feat(providers): report bounded token-grant failures Signed-off-by: Shiju <shiju@nvidia.com> * test(providers): align policy metadata and TLS lockfile Signed-off-by: Shiju <shiju@nvidia.com> * test(providers): fix trusted grant recovery fixture Signed-off-by: Shiju <shiju@nvidia.com> * test(providers): enable denial API in recovery fixture Signed-off-by: Shiju <shiju@nvidia.com> * refactor(providers): simplify token-grant failure reporting Remove duplication and dead paths from the token-grant failure reporting without changing classification, the workload-visible 502 response, or the wire format. - Build the gRPC failure envelope with tonic-types, store the typed failure in EndpointFailure, and share one request-ID rule and one endpoint-status authority check. - Merge the gateway exchange client wrapper, drop forwarding validators, and name the request field when a reported cause is rejected. - Inject fallback credentials only from live provider state and delete the unread per-connection credential copy. Keep one 502 writer per path and restore the base token-cache handling. - Check the token endpoint after the cache lookup, still before any credential acquisition. - Document 429 and 5xx classification as implemented, and drop tests that repeat retained assertions through the same code path. Signed-off-by: Shiju <shiju@nvidia.com> * refactor(providers): trim token-grant failure tests and supervisor API Cut the token-grant failure tests to one test per distinct behavior across core, gateway, supervisor, CLI and Go, and replace the 794-line closure E2E with status assertions in the existing route-recovery E2E: the refused grant appears in sandbox status with the response's request ID, code, stage and upstream status, and the matching grant's later success clears it. - Pin the live provider snapshot with one helper used by inspected and fallback forwarding, and inject selector-matched grants through inject_for_selectors, which borrows the pinned context. Move the test-only wrapper into test_support. - Drop the 35-second wrappers around the gateway grant RPC. The shared channel bounds the connection attempt and the gateway bounds its token-service calls. - Log the token service's OAuth error and error_description, without control characters and limited to 256 characters each, with the request ID in the gateway or supervisor log. Status and the workload response still carry only the bounded cause. - Keep one check that an untrusted token-service certificate cannot assert a cause. Signed-off-by: Shiju <shiju@nvidia.com> * fix(providers): restore test import and features lockfile The injection tests named EndpointObservationSender through the parent module, which no longer imports it. The features suite lockfile kept the tonic TLS dependency that the removed E2E fixture added. Signed-off-by: Shiju <shiju@nvidia.com> * test(providers): fix body-bound fixture width and table type Pass the response-size width as a named argument; an implicitly captured constant panics at runtime. Name the mutation type in the validation table for Clippy. Signed-off-by: Shiju <shiju@nvidia.com> * test(providers): pad exact-bound body without a format width MAX_TOKEN_RESPONSE_BYTES exceeds u16::MAX, the largest format width, so the padded fixture panicked. Pad with repeat_n as before. Signed-off-by: Shiju <shiju@nvidia.com> * test(server): update public RPC schema hash after rebase Main added the ListProvidersRequest label selector. The merged public schema hash comes from the Branch Checks storage_proto test output. Signed-off-by: Shiju <shiju@nvidia.com> * fix(providers): keep token-service error text out of grant logs A token service can echo a submitted credential in its OAuth error or error_description. Log only the classified failure code, HTTP status and request ID, and cover the canary echo in a dedicated test binary. Signed-off-by: Shiju <shiju@nvidia.com> * docs(providers): tighten token grant failure documentation Signed-off-by: Shiju <shiju@nvidia.com> * docs(providers): condense token grant failure guidance Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> --------- Signed-off-by: Shiju <shiju@nvidia.com> Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> Co-authored-by: John Myers <9696606+johntmyers@users.noreply.github.com> |
||
|
John T. Myers
|
fce4e5aea5 |
fix(sandbox): release broker-owned listening sockets (#4444)
Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> |
||
|
Shiju
|
6a6401b9c3 |
fix(cli): backpressure sandbox uploads (#4448)
* fix(cli): backpressure sandbox uploads Stream archive uploads over a bounded SSH subsystem whose credit follows extraction stdin progress. Preserve the existing input budget and tar destination semantics, validate completion, and clean up cancelled transfers. Add protocol, lifecycle, and large-upload regression coverage without changing russh. Signed-off-by: Shiju <shiju@nvidia.com> * fix(cli): preserve upload cancellation and errors Keep cancellation active through upload setup and retry delays, and terminate cancelled SSH probes. Preserve protocol failures through output completion and report captured diagnostics on early extraction failure. Remove redundant upload validation and extend the affected regressions. Signed-off-by: Shiju <shiju@nvidia.com> * refactor(cli): carry upload credit over SSH streams Replace the framed upload protocol with SSH's own streams. The client sends one JSON plan line, the raw archive and EOF. The supervisor's stdout carries only cumulative totals of bytes written to extraction stdin, extraction output goes to stderr, and the exit status reports the result. Extraction starts inline through the ordinary exec path. This removes the frame codec, SHA-256 digests, the CLI sha2 dependency and the supervisor upload state machine while keeping the 1 MiB credit window, the 4 MiB input guard, cancellation and error reporting. A failed archive now stops SSH before closing stdin, so EOF never marks a truncated archive complete. Signed-off-by: Shiju <shiju@nvidia.com> * fix(cli): isolate upload SSH connections and pin credit cadence - Apply the forward multiplexing overrides to the upload ssh command, so a user ControlMaster cannot carry an upload over another session's master connection. - Kill ssh before closing its stdin when a transfer fails locally, and include ssh stderr in transport errors. - Share ACK_INTERVAL_BYTES from core. Assert at compile time that CLI chunks fit the credit left after an ack and that the window stays below the supervisor's pending-input guard. - Log the real reason when the supervisor rejects an upload. - Test that a single-packet upload reports its final total. Signed-off-by: Shiju <shiju@nvidia.com> * test(cli): assert upload ssh opens the subsystem without a TTY Signed-off-by: Shiju <shiju@nvidia.com> --------- Signed-off-by: Shiju <shiju@nvidia.com> |
||
|
Matthew GrossmanandEvan Lezar
|
8c67c27e8b |
test(oci): add a shared Docker and Podman suite for image USER and WORKDIR (#3931)
* test(oci): share OCI image checks across Docker and Podman Add an oci-image feature testsuite that runs against installed artifacts. It covers custom WORKDIR placement, image content and ownership, workspace writes from the main process and exec, file transfer including directory-merge upload, OCI user identity, the managed /sandbox fallback, and rejection of an unwritable WORKDIR. CI runs the suite on the Docker rootful, Podman rootful, and Podman rootless tmachine guests. Replace the Docker-only custom_image e2e with the shared suite. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> Signed-off-by: Evan Lezar <elezar@nvidia.com> * test(oci): require WorkspaceValidationFailed and cover image USER fallback The unwritable WORKDIR scenario now requires the WorkspaceValidationFailed reason, so unrelated provisioning failures cannot pass it. The numeric USER scenario supplies a policy without a process section, so the image USER is the only source of the sandbox identity. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> Signed-off-by: Evan Lezar <elezar@nvidia.com> * test(podman): leave shared OCI identity checks to the oci-image suite The oci-image feature suite covers the sandbox identity seen by the main process and sandbox exec on Docker and Podman. The Podman e2e keeps the checks only it can make: the pinned image ID and the isolated workload/supervisor container pair. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> Signed-off-by: Evan Lezar <elezar@nvidia.com> * test(oci): accept existing workspace rejection diagnostics Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> Signed-off-by: Evan Lezar <elezar@nvidia.com> * test(oci): fix cleanup ordering and restore coverage parity Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> Signed-off-by: Evan Lezar <elezar@nvidia.com> * test(oci): remove migrated legacy workspace helpers and checks Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * fix(drivers): report rejected OCI working directories as WorkspaceValidationFailed The supervisor exits with the reserved status when the sandbox rejects the image working directory. Docker maps that exit during readiness and monitoring and records it before removing the supervisor container, and Podman maps the supervisor companion's exit instead of the workload's. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * test(oci): require WorkspaceValidationFailed for unwritable WORKDIR Unrelated provisioning failures can no longer pass the rejection scenario. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * fix(oci): keep exit 78 reserved and report fixed workspace diagnostics The supervisor no longer passes on a workload's own exit status 78, so drivers cannot misreport it as a workspace rejection after a runtime restart. The rejection is recognized once at agent start and carried as a typed error. Docker reports the same fixed condition message as Podman and logs the supervisor output instead, and its monitor removes the supervisor container when a stop overlaps diagnostics collection. The OCI image suite registers images for cleanup only after they build. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * refactor(oci): simplify workspace failure handling and OCI image tests Report a recorded Docker workspace rejection once, where supervisor readiness fails, and match its fixed message exactly. Read the Podman supervisor exit status once per inspection. Choose the supervisor exit status in one function and derive the workspace error type. The OCI image suite shares step helpers, checks that a custom workspace does not also create /sandbox, and its playbook waits for a connected gateway and collects failure diagnostics through the shared role. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * test(oci): replace podman_oci_identity with shared and driver suite checks The oci-image feature suite now checks, on Docker and Podman, that the image user has no capabilities, has no_new_privs set, and cannot read the sandbox control credentials, and that a custom WORKDIR is not a mount. The checks that need container inspection (pinned image ID, network=none, supervisor user and capabilities) move to a small test in the Podman driver suite. Remove the legacy podman_oci_identity e2e test. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * test(oci): keep podman_oci_identity for Podman-only isolation checks Restore the Podman e2e test removed in the previous commit and drop only the checks the shared oci-image suite now runs on Docker and Podman: workload capabilities, no_new_privs, credential readability, and the workspace and /sandbox mount checks. The pinned image ID and the workload/supervisor isolation checks stay, as does the Podman driver suite unchanged. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * refactor(oci): simplify workspace failure reporting and OCI image suite Carry the Docker control supervisor's failure reason in a typed error instead of matching status text, build the fixed workspace message where the wait result is classified, and drop the fake Docker server test. Inline image cleanup ordering in the suite, read the container engine once, share sandbox naming, and stop granting a literal /sandbox in the image identity policy. Add e2e:oci-image mise tasks and use them from the ARM64 Docker workflow and TESTING.md. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * test(oci): move shared container checks into the OCI image suite Inspect the containers the driver created through the gateway's engine in the shared suite: the workload runs the inspected image ID as the OCI identity without a network or supervisor credential mounts, and the supervisor is a separate non-root container. Keep only the Podman-specific supervisor identity and capability checks in podman_oci_identity. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * test(oci): check the workspace rejection reason from sandbox JSON Read the WorkspaceValidationFailed condition reason from sandbox get --output json instead of matching CLI output, and restore the Docker driver's original workspace failure helpers. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * test(oci): capture fixture image identity before sandbox creation Signed-off-by: Evan Lezar <elezar@nvidia.com> --------- Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> Signed-off-by: Evan Lezar <elezar@nvidia.com> Co-authored-by: Evan Lezar <elezar@nvidia.com> |
||
|
Taylor Mutch
|
5e414eadde |
test(e2e): remove stale policy retries (#4453)
Signed-off-by: Taylor Mutch <taylormutch@gmail.com> |
||
|
Yuedong Wu
|
3495c8e92a |
fix(kubernetes): refresh sandbox runtime image on start (#4376)
Signed-off-by: Yuedong Wu <dwcn22@outlook.com> |
||
|
Shiju
|
e585b49173 |
feat(providers): support stable credential placeholders (#3339)
* feat(providers): support stable credential placeholders Add credentials[].stable_placeholder so a running client keeps using an externally rotated static credential through the reference it already holds. Squashed replay of the earlier four commits onto upstream main 71c3cd957; re-derives the frozen schema fingerprints and adapts the provider environment revision test to the added capability argument. Signed-off-by: Shiju <shiju@nvidia.com> * fix(e2e): apply stable-placeholder supervisor image override Use ManagedGateway's per-process supervisor image override so the fixture CA reaches the supervisor when runtime image environment variables take precedence over gateway TOML. Restore the wrapper's original image after each test, following the provider-readiness fixture. Signed-off-by: Shiju <shiju@nvidia.com> * test(server): update public RPC schema fingerprint after merging main Signed-off-by: Shiju <shiju@nvidia.com> --------- Signed-off-by: Shiju <shiju@nvidia.com> |
||
|
cjagwani
|
a991b8e369 |
fix(sandbox): preserve exec capacity under failure and load (#4324)
* fix(sandbox): reclaim failed exec waits Signed-off-by: Charan Jagwani <cjagwani@nvidia.com> * test(e2e): cover exec recovery after policy denial Signed-off-by: Charan Jagwani <cjagwani@nvidia.com> * fix(sandbox): preserve live descriptor headroom Signed-off-by: Charan Jagwani <cjagwani@nvidia.com> * fix(sandbox): reclaim descriptors before exec Signed-off-by: Charan Jagwani <cjagwani@nvidia.com> * fix(sandbox): separate exec and socket capacity Signed-off-by: Charan Jagwani <cjagwani@nvidia.com> * fix(sandbox): stream Landlock baseline entries Signed-off-by: Charan Jagwani <cjagwani@nvidia.com> --------- Signed-off-by: Charan Jagwani <cjagwani@nvidia.com> |
||
|
John T. Myers
|
67adcf1a3c |
feat(ocsf): emit full JSON records to supervisor stderr (#4323)
* feat(ocsf): emit full JSON records to supervisor stderr Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * fix(ocsf): preserve console record boundaries Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> --------- Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> |
||
|
Shiju
|
ffb530671f |
fix(providers): prepare dynamic credentials across HTTP relay paths (#4152)
* fix(providers): bind multi-route grants to admitted endpoints Carry gateway-derived endpoint owners through policy and credential delivery. Select grants only from owners that admit the current request, refresh provider snapshots per request, and reject superseded installations before forwarding. Cover persistent routes, overlapping owners, denial, refresh, cache expiry and recovery with focused tests and a Podman regression. Signed-off-by: Shiju <shiju@nvidia.com> * refactor(providers): simplify admitted-endpoint grant selection Share admitted endpoints between forwarding and credential authorization. Select grants from the live snapshot without rebuilding a locked map, reuse profile policy construction, and preserve freshness guards when grants are removed. Signed-off-by: Shiju <shiju@nvidia.com> * fix(providers): preserve trusted owners across policy approval Sanitize authored owner metadata before composing effective policies, preserve derived owners through proposal approval, and normalize advisor endpoint comparisons. Account for gateway-only owner fields in SDK coverage and schema inventory. Signed-off-by: Shiju <shiju@nvidia.com> * refactor(providers): share dynamic credential key layout and require a snapshot for admission Move the endpoint-bound credential key layout into openshell-core. DynamicCredentialKey encodes the key the gateway builds, and the supervisor reads its endpoint selector, credential identity, and revision-scoped form through the same module instead of parsing tab-separated strings in three places. Revision scoping becomes ProviderCredentialSnapshot::scoped_key. inject_for_admitted_owners now takes the pinned snapshot directly. The L7 admission path never acquires a grant from the live credential map, which production populates only alongside a snapshot. L4 forwarding keeps the selector-only path through inject_if_needed. Both share one acquire-and-rewrite step. Delete two relay tests whose behavior the Rego admission tests already prove. Add a test that grants for different headers are acquired only for admitting owners. Signed-off-by: Shiju <shiju@nvidia.com> * fix(providers): share credential preparation across HTTP relays Apply endpoint-owned grants after policy and request middleware in REST, GraphQL, JSON-RPC/MCP and inspected plaintext forwarding. Preserve live provider and policy checks through guarded writes, reject missing owner metadata, and reject profiles that cannot inject dynamic HTTP credentials. Cover production stream dispatch, multiple endpoints, failure handling and plaintext ownership with focused regressions. Document upgrade ordering. Closes #3657 Signed-off-by: Shiju <shiju@nvidia.com> * fix(providers): validate dynamic credential key encoding Reject control characters at profile import and key construction. Propagate invalid key errors through gateway environment resolution and reject malformed key layouts before token acquisition while preserving supported legacy keys. Consolidate duplicate relay success tests and share forward-proxy setup without removing ownership, canonicalization, refresh or guarded-write assertions. Document token-grant endpoint authority. Signed-off-by: Shiju <shiju@nvidia.com> --------- Signed-off-by: Shiju <shiju@nvidia.com> |
||
|
Matthew Grossman
|
1fff8b97b9 |
feat(podman): honor OCI image working directories (#3982)
* fix(supervisor): report rejected OCI working directories as WorkspaceValidationFailed The RFC 0012 split dropped the supervisor exit status that drivers map to the WorkspaceValidationFailed condition. An image WORKDIR the sandbox identity cannot use then surfaced as ControlSupervisorStartFailed on Docker and as a signal kill on Podman. The supervisor now exits with the reserved status when the sandbox rejects the image working directory. Docker maps that supervisor exit during readiness and monitoring, and Podman maps the supervisor companion's exit instead of the workload's. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * feat(podman): honor OCI image working directories Podman sandboxes now use a custom OCI image WORKDIR as the workspace and as the working directory for agent commands, matching Docker. Empty, /, and /sandbox values keep the managed /sandbox workspace volume. A custom workspace stays in the image's container filesystem: no workspace volume, no archive upload, and no root setup step. Resolve the image ID, user, environment, and working directory from one pinned inspection, validate the workdir with the shared OCI rules, and reject Podman control-path overlaps and image volumes or driver mounts that cover it. The runtime starts from / and passes the resolved path to agent commands with --workdir. Add podman_oci_identity to the Podman CI test list. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * fix(drivers): preserve custom workspace failures across supervisor cleanup Check the Podman custom workspace rather than the absent managed volume in the OCI identity E2E. Classify Bollard wait errors by exit code and record Docker workspace failures before removing the supervisor so readiness retains the specific failure reason. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * fix(docker): use sandbox ID for recorded startup failures Keep the workload container ID for Docker inspection and use the sandbox ID to retrieve the monitor failure. Test with distinct identifiers so the lookup cannot accidentally pass with a container ID. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * refactor(podman): keep WORKDIR change focused on workspace behavior Remove the new cross-driver workspace failure classification path and its monitor/readiness workaround. Preserve Podman WORKDIR resolution, mount safety, and workspace rejection without requiring a distinct condition reason. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * refactor(podman): simplify inspected image metadata and test fixtures Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * code review Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> --------- Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> |
||
|
Mike Nguyen
|
9fd41e6bfd |
fix(ssh): persist sandbox host identities (#4094)
* fix(ssh): persist sandbox host identities Store each sandbox's Ed25519 host key in the gateway credential store and deliver it only to the supervisor. Preserve identity across restarts, delete owned credentials with the sandbox, and expose the public SHA256 fingerprint through sandbox and SSH-session APIs and client SDKs. Cover credential ownership, cancellation, deletion retries, client compatibility, and pinned SSH connections through lifecycle transitions. Closes #3835 Signed-off-by: Mike Nguyen <miken@nvidia.com> * fix(compute): clean up failed sandbox SSH identity creation Signed-off-by: Mike Nguyen <miken@nvidia.com> * test(ssh): wait for sandbox deletion before name reuse Signed-off-by: Mike Nguyen <miken@nvidia.com> --------- Signed-off-by: Mike Nguyen <miken@nvidia.com> |
||
|
Simon Scatton
|
8760396975 |
fix(compute): revalidate late exit events after restart (#4235)
Signed-off-by: Simon Scatton <sscatton@nvidia.com> |
||
|
Shiju
|
8579dfb301 |
feat(providers): add multiple tokens to one request (#4047)
* feat(providers): compose dynamic credential grants per request Select the most-specific grant independently for each protected header, preserving each credential's issuer, audience and cache identity. Resolve all selected grants before rewriting the request, replace agent-supplied header copies, and fail closed on collisions or acquisition failures. Allow distinct-header compositions in gateway validation and redact raw issuer errors. Cover independent cache entries, atomic TLS relay failure, header replacement and concurrent request isolation; document the profile contract. Fixes #3320 Signed-off-by: Shiju <shiju@nvidia.com> * fix(providers): correct dynamic grant CI checks Use if-let for a grant result whose error payload is deliberately ignored, and name the empty query map type in the regression fixture. Preserve grant acquisition, failure redaction and request atomicity. Update the token-exchange failure test to require the sanitized error instead of raw issuer text. Signed-off-by: Shiju <shiju@nvidia.com> --------- Signed-off-by: Shiju <shiju@nvidia.com> |
||
|
Simon Scatton
|
be7af99ff2 |
fix(e2e): override provider readiness supervisor image via environment (#4207)
Signed-off-by: Simon Scatton <sscatton@nvidia.com> |
||
|
Matthew GrossmanandEvan Lezar
|
13c248fd45 |
fix(podman): create managed workspace volumes owned by the workload identity (#3981)
* fix(podman): create managed workspace volumes owned by the workload identity Podman now creates the managed /sandbox volume with uid/gid options for the resolved workload identity, so the workload starts directly as that identity. This fixes rootful sandboxes whose image USER or policy run_as_user could not write to a root-owned /sandbox, and removes the root-then-drop workspace chown start path. Resource admission accepts the managed workspace volume when its options match the workload container's final identity, or are empty for volumes created by older gateways. The channel volume still requires empty options. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * test(podman): cover managed volume reuse and workspace access Signed-off-by: Evan Lezar <elezar@nvidia.com> * refactor(podman): clarify managed volume creation and validation Signed-off-by: Evan Lezar <elezar@nvidia.com> * test(podman): verify workspace access across user namespaces Signed-off-by: Evan Lezar <elezar@nvidia.com> --------- Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> Signed-off-by: Evan Lezar <elezar@nvidia.com> Co-authored-by: Evan Lezar <elezar@nvidia.com> |
||
|
Shiju
|
d676e036a4 |
fix(vm): reject conflicting workload identity selectors (#4036)
* fix(vm): enforce the configured workload identity Reject conflicting policy users and groups before VM image preparation and before guest attach or process startup changes state. Validate supervisor policy updates against the protected VM workload identity. Preserve the gateway CA transport and capability-free sandbox launcher. Signed-off-by: Shiju <shiju@nvidia.com> * test(sandbox): clarify VM identity rejection fixtures Name invalid user and group fixtures distinctly and move the final workload identity into its group mismatch test. Signed-off-by: Shiju <shiju@nvidia.com> * fix(supervisor): align VM identity startup with current APIs Pass the optional rejection-log key for VM identity failures and keep generic startup-write regressions free of VM identity constraints. Repair the call sites after the branch rebase so the identity and cleanup proposals compile against the current startup helpers. Signed-off-by: Shiju <shiju@nvidia.com> * fix(vm): restore inactive sandbox workload identity Recover the persisted overlay owner before publishing stopped and terminal sandboxes. Keep resources manageable when identity metadata is invalid. Clarify fixed MicroVM ownership in policy-generation guidance. Signed-off-by: Shiju <shiju@nvidia.com> * test(vm): flush identity fixture before restart Persist the canonical identity file before readiness and report the observed exec, canonical and file-owner identities before comparing them. Signed-off-by: Shiju <shiju@nvidia.com> --------- Signed-off-by: Shiju <shiju@nvidia.com> |
||
|
Drew Newberry
|
121930be07 |
fix(sandbox): replace lifetime exec cap with retry deadlines (#4105)
* fix(sandbox): replace lifetime exec cap with retry deadlines Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs(sandboxes): remove exec recovery overview change Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs(skills): remove exec recovery CLI skill change Signed-off-by: Drew Newberry <anewberry@nvidia.com> --------- Signed-off-by: Drew Newberry <anewberry@nvidia.com> |
||
|
Drew Newberry
|
8719fc9f37 |
fix(sandbox): restrict provider file mode (#4093)
* fix(sandbox): restrict provider file mode (fixes #4091) Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(sandbox): set provider file mode with safe API Signed-off-by: Drew Newberry <anewberry@nvidia.com> --------- Signed-off-by: Drew Newberry <anewberry@nvidia.com> |
||
|
Drew Newberry
|
021400be8a |
refactor(auth): separate sandbox identity from TLS (#3110)
* refactor(auth): separate sandbox identity from TLS Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs(auth): clarify gateway mTLS behavior Signed-off-by: Drew Newberry <anewberry@nvidia.com> * test(auth): include workspace scope in TLS authorization checks Signed-off-by: Drew Newberry <anewberry@nvidia.com> * test(e2e): bound service auth sandbox names for large PIDs Signed-off-by: Drew Newberry <anewberry@nvidia.com> --------- Signed-off-by: Drew Newberry <anewberry@nvidia.com> |
||
|
John T. Myers
|
2935e9731b |
fix(gateway): delete finalized ephemeral sandboxes while connected (#3984)
Start driver cleanup after terminal finalization and retain disconnect fallback. Add detached success and failure e2e coverage across supervisor-based drivers. Closes #3938 Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> |
||
|
Derek Carr
|
912a077bd6 |
feat(service): add bearer authorization passthrough (#3796)
* feat(service): add bearer authorization passthrough Signed-off-by: Derek Carr <decarr@redhat.com> * docs(sdk): add service authorization migration guide Signed-off-by: Derek Carr <decarr@redhat.com> * fix(server): remove stale version import Signed-off-by: Derek Carr <decarr@redhat.com> * docs(upgrade): remove service authorization SDK guide Signed-off-by: Derek Carr <decarr@redhat.com> * fix(e2e): relabel provider readiness TLS mount Signed-off-by: Derek Carr <decarr@redhat.com> * test(e2e): stabilize exposed service routing Signed-off-by: Derek Carr <decarr@redhat.com> * test(e2e): support HTTPS service routing Signed-off-by: Derek Carr <decarr@redhat.com> --------- Signed-off-by: Derek Carr <decarr@redhat.com> |
||
|
Polite_realismandEvan Lezar
|
b8ffe5244c |
test(podman): move podman_preflight into driver-podman integration tests (#3783)
* test(podman): move podman_preflight into driver-podman integration tests podman_preflight verifies that openshell-driver-podman fails fast when its Podman socket is unreachable. It only needs the standalone driver binary, not a gateway, so it never fit the gateway-backed e2e-podman harness it lived under and never ran anywhere in CI. Move it into crates/openshell-driver-podman/tests/ as a plain Cargo integration test. It now runs via the existing required workspace test job with no special mise task, workflow step, or coverage exception. Signed-off-by: politerealism <burdcat17@gmail.com> * test(podman): make preflight diagnostics portable Signed-off-by: Evan Lezar <elezar@nvidia.com> --------- Signed-off-by: politerealism <burdcat17@gmail.com> Signed-off-by: Evan Lezar <elezar@nvidia.com> Co-authored-by: Evan Lezar <elezar@nvidia.com> |
||
|
Drew Newberry
|
252882f37f |
feat(providers): serve sandbox config files on demand (#3832)
* feat(providers): serve sandbox config files on demand Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs(providers): defer managed file documentation Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs(providers): mark managed file api experimental Signed-off-by: Drew Newberry <anewberry@nvidia.com> * style(go): format provider profile fields Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(providers): preserve legacy environment with managed files Signed-off-by: Drew Newberry <anewberry@nvidia.com> --------- Signed-off-by: Drew Newberry <anewberry@nvidia.com> |
||
|
Eric Curtin
|
cfcc3733bd |
fix(e2e): stop sandbox leaks from async Drop cleanup (#3750)
* fix(e2e): stop sandbox leaks from async Drop cleanup Closes #2922 SandboxGuard::Drop spawned a detached thread to delete the sandbox. The thread got killed with the test process before the delete finished. Switch to a blocking command in Drop, like ManagedCleanup already does. Also wrap two tests' manual cleanup in RAII guards so a panic does not leak a sandbox. Signed-off-by: Eric Curtin <eric.curtin@docker.com> * test(e2e): arm sandbox guards before create Address review: install guards with explicit names first. Signed-off-by: Eric Curtin <eric.curtin@docker.com> --------- Signed-off-by: Eric Curtin <eric.curtin@docker.com> |
||
|
Philippe MartinandJohn Myers
|
9cb72baa2e |
feat(docker): support corporate proxy CA bundles (#3549)
* feat(docker): support corporate proxy CA bundles Closes #3545 Validate and stage operator-owned proxy CA bundles for Docker supervisors, add corporate proxy E2E coverage, and document the trust contract. Signed-off-by: Philippe Martin <phmartin@redhat.com> * fix(docker): validate proxy config on startup Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * test(docker): use the E2E workload image for proxy tests Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * test(docker): generate strict corporate proxy certificates Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * test(docker): surface intercepted TLS fixture errors Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * test(docker): drain buffered TLS proxy data Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * test(docker): relay intercepted HTTP deterministically Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> --------- Signed-off-by: Philippe Martin <phmartin@redhat.com> Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> Co-authored-by: John Myers <9696606+johntmyers@users.noreply.github.com> |
||
|
Drew Newberry
|
acbac9cb79 |
feat(sandbox): add main restart policy (#2798)
* feat(sandbox): add main restart policy Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(sandbox): harden policy-driven restarts Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(sandbox): address restart review feedback Signed-off-by: Drew Newberry <385+drew@users.noreply.github.com> * fix(sandbox): port restart policy to current runtime Signed-off-by: Drew Newberry <anewberry@nvidia.com> * perf(sandbox): restart promptly after terminal delivery Signed-off-by: Drew Newberry <anewberry@nvidia.com> --------- Signed-off-by: Drew Newberry <anewberry@nvidia.com> Signed-off-by: Drew Newberry <385+drew@users.noreply.github.com> |
||
|
Shiju
|
1358941b81 |
feat(mcp): inspect requests with Tower-selected protocol profiles (#3335)
* fix(sandbox-backend): sort boundary request objects before hashing Sort boundary request objects recursively before hashing so serde_json's preserve_order feature cannot change digest identity. Cover canonical bytes, envelope round trips, and rejection of modified provider values and operations. Signed-off-by: Shiju <shiju@nvidia.com> * feat(mcp): upgrade tower-mcp-types to 0.22.2 Upgrade tower-mcp-types from 0.12.0 to an exact-pinned 0.22.2 and use its inspection APIs to validate MCP requests against the selected revision. Carry inspection metadata into policy evaluation and validate requests after header rewriting, before forwarding. Add explicit support for the sessionless 2026-07-28 revision while keeping 2025-11-25 as the default. Validate per-request metadata and standard HTTP header mirrors, and support discovery, tools, and subscription requests. Delegate batch availability and parameter schemas to Tower. Share typed request names between policy and HTTP checks, retain the local batch resource cap, and centralize MCP policy version parsing and ordering. Keep supported MCP revisions and shared allowlist parsing in the canonical policy schema; core re-exports those types. Tower owns wire-profile semantics, and every supported policy revision must map to the matching inspector profile. Reject duplicate JSON keys, invalid known-method parameters, unavailable methods, and unsupported batches. Keep exact extension allow rules and deny precedence. Document request inspection boundaries and add unit, forwarding, and sandbox coverage. Refs #2174. Signed-off-by: Shiju <shiju@nvidia.com> * test(mcp): prove authorization at the forwarding boundary Cover March batch denial in both member orders, valid and malformed controls, and audit behavior across both relay entry paths. Exercise real middleware tool rewrites with matching metadata and assert the exact upstream representation or zero forwarded bytes. Verify legacy bodyless SSE GET remains usable while GET tool bodies and unsupported DELETE cleanup are rejected. Clarify request-selected profile and middleware mutation comments without changing production behavior. Signed-off-by: Shiju <shiju@nvidia.com> * test(mcp): exercise permitted profiles through the sandbox proxy Cover March and June singleton policies and select November and July separately under one endpoint allowlist. Capture upstream tool receipts to distinguish proxy policy denial from an upstream rejection. Extend middleware rewrite coverage to June and multi-version policies, and preserve the sessionless discovery and subscription checks through the shared fixture helpers. Signed-off-by: Shiju <shiju@nvidia.com> * test(kubernetes): box the admission check future Keep the admission test future below Clippy's size limit when the workspace dependency features are unified. Signed-off-by: Shiju <shiju@nvidia.com> * test(mcp): reuse the forwarding fixture identity cache Share the binary identity cache across protocol-profile cases, matching the proxy lifecycle and avoiding repeated hashes of the test executable. Keep procfs authorization and all forwarding assertions intact. Signed-off-by: Shiju <shiju@nvidia.com> --------- Signed-off-by: Shiju <shiju@nvidia.com> |
||
|
Evan Lezar
|
eef8bec0c9 |
test(e2e): run podman suite with tmachine (#3637)
* test(e2e): remove superseded podman userns coverage Signed-off-by: Evan Lezar <elezar@nvidia.com> * test(tmachine): run podman e2e archive Signed-off-by: Evan Lezar <elezar@nvidia.com> * test(tmachine): generate podman e2e archive inventory Signed-off-by: Evan Lezar <elezar@nvidia.com> --------- Signed-off-by: Evan Lezar <elezar@nvidia.com> |
||
|
Evan Lezar
|
0c29d8e061 |
test(conformance): migrate file transfer scenarios (#3597)
Signed-off-by: Evan Lezar <elezar@nvidia.com> |
||
|
Evan Lezar
|
c63f8ce564 |
test(cli): migrate gateway-free smoke coverage (#3641)
* test(cli): migrate gateway-free smoke coverage Signed-off-by: Evan Lezar <elezar@nvidia.com> * fix(e2e): remove migrated tests from podman CI Signed-off-by: Evan Lezar <elezar@nvidia.com> --------- Signed-off-by: Evan Lezar <elezar@nvidia.com> |
||
|
Drew Newberry
|
6f00d5cacc |
fix(e2e): reserve distinct corporate proxy fixture ports (#3761)
Signed-off-by: Drew Newberry <anewberry@nvidia.com> |
||
|
Drew Newberry
|
7a50c0899f |
fix(cli): stream piped exec stdin beyond gRPC request limit (#3687)
* fix(cli): stream piped exec stdin across gRPC messages Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(cli): preserve small exec requests and surface stdin errors Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(cli): preserve exec stdin limit across gRPC streaming Signed-off-by: Drew Newberry <anewberry@nvidia.com> --------- Signed-off-by: Drew Newberry <anewberry@nvidia.com> |
||
|
Drew Newberry
|
4688061882 |
fix(sandbox): deliver complete exec output before success (#3688)
* fix(sandbox): preserve exec output through channel close Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(exec): propagate output delivery failures before exit Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs(cli): clarify exec output delivery failures Signed-off-by: Drew Newberry <anewberry@nvidia.com> --------- Signed-off-by: Drew Newberry <anewberry@nvidia.com> |
||
|
Polite_realismandDrew Newberry
|
d376c90755 |
test(podman): close rootful userns, resource-limit, and daemon-failure CI gaps (#3690)
* test(podman): run driver-podman userns suite against rootful Podman too The driver-specific-integration job only ran the driver-podman testsuite (default/auto/keep-id/private userns reference checks) against fedora-podman-rootless, leaving rootful behavior for this scenario unverified even though the compute driver auto-detects and explicitly supports rootful Podman. The default-userns-baseline and userns-profile playbooks hard-asserted a rootless tmachine gateway user, so pointing them at a rootful environment would have failed that assertion immediately rather than exercising anything. They now detect rootful vs. rootless via the existing tmachine_container_runtime role and branch the reference-capture user accordingly, while keeping the captured reference file itself owned by tmachine, since the archived test binary that reads it back always runs unprivileged as tmachine regardless of daemon mode. Signed-off-by: politerealism <burdcat17@gmail.com> * test(podman): add real-daemon coverage for resource limits and daemon failure Neither the Podman driver's resource-limit enforcement nor its behavior when the Podman daemon is unreachable had any test coverage against a real daemon; both were only exercised through unit tests against a mocked Podman client. podman_resource_limits.rs creates a sandbox with --cpu/--memory flags and reads /sys/fs/cgroup/memory.max and cpu.max from inside the sandbox itself, verifying the limit is actually enforced rather than just echoed back by the template API. Expected values are cross-checked against the driver's own parse_cpu_to_microseconds/parse_memory_to_bytes and against a real local `podman run --cpus/--memory` container. podman_preflight.rs spawns the standalone openshell-driver-podman binary against a guaranteed-nonexistent Podman socket and asserts it exits non-zero within its bounded retry window with an actionable error naming the socket path, rather than hanging or failing silently. Signed-off-by: politerealism <burdcat17@gmail.com> * test(podman): make rootful userns and cgroup checks pass Signed-off-by: Drew Newberry <anewberry@nvidia.com> * test(podman): match lifecycle containers by isolation role label Signed-off-by: Drew Newberry <anewberry@nvidia.com> * test(podman): accept non-expiring bootstrap tokens Signed-off-by: Drew Newberry <anewberry@nvidia.com> --------- Signed-off-by: politerealism <burdcat17@gmail.com> Signed-off-by: Drew Newberry <anewberry@nvidia.com> Co-authored-by: Drew Newberry <anewberry@nvidia.com> |
||
|
grs
|
8369bc11a5 |
fix(podman): restore host gateway alias mediation (#3606)
* fix(podman): restore host gateway alias mediation Signed-off-by: Gordon Sim <gsim@redhat.com> * fix(podman-e2e-tests): enable broader test podman e2e coverage Signed-off-by: Gordon Sim <gsim@redhat.com> * fix(tests): make test more reliable Signed-off-by: Gordon Sim <gsim@redhat.com> * fix(podman): fix macos linting error Signed-off-by: Gordon Sim <gsim@redhat.com> --------- Signed-off-by: Gordon Sim <gsim@redhat.com> |
||
|
Drew Newberry
|
0518bd4c83 |
fix(sandbox): preserve local sessions across host sleep (#3573)
* fix(cli): recover sandbox connect transport Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(auth): propagate non-expiring local sandbox sessions Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(cli): distinguish main exit from transport loss Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(cli): bound sandbox connect recovery Signed-off-by: Drew Newberry <anewberry@nvidia.com> --------- Signed-off-by: Drew Newberry <anewberry@nvidia.com> |
||
|
krishicks
|
0b351c4a9b |
fix(helm)!: reduce gateway Secret privileges (#3616)
* fix(driver-kubernetes-secrets)!: store provider credentials in one namespace The Kubernetes Secrets credential driver now stores every credential in its configured namespace in all workspace modes and rejects handles that reference any other namespace before contacting the Kubernetes API. The gateway reaches credential Secrets through the Role in that namespace; this allows removing the Secret rules from the ClusterRole. - Remove the workspace_mode, gateway_id, and allow_reference_namespace driver settings and stop rendering them from Helm. Configurations that set them fail at startup. Existing credential state is not migrated. - Add server.credentialDrivers.kubernetesSecrets.createNamespace to provision a dedicated credential namespace. The namespace is kept on uninstall, adopted by a reinstall of the same release, and left untouched when something else owns it. - Update the gateway config reference, Kubernetes setup docs, 0.1.0 upgrade guide, compute-runtime architecture, and cluster debugging skill. Signed-off-by: Kris Hicks <khicks@nvidia.com> * fix(helm): reduce gateway Secret permissions Remove the gateway's Secret list permission in every workspace mode and grant source Secret reads through a Role in the sandbox namespace. Bootstrap Secret cleanup deletes Secrets by exact name instead of listing them. - Grant get on the copied client TLS and image-pull Secrets through a Role in the sandbox namespace. The ClusterRole keeps get and patch on those names for the ownership check and server-side apply into workspace namespaces. - Delete sandbox and supervisor bootstrap Secrets by exact name, derived from the runtime generation recorded on the Sandbox and, on restart, the target generation, tolerating 404. The generation annotation is cleared only after cleanup succeeds, and each bootstrap Secret has a Pod owner reference, so garbage collection removes any generation the driver does not name. - Drop Secret list from the ClusterRole and the shared-mode sandbox Role. - Extend the managed e2e RBAC checks to Secret list. - Update the Kubernetes setup and sandbox runtime docs, compute-runtime architecture, and cluster debugging skill. Signed-off-by: Kris Hicks <khicks@nvidia.com> * fix(driver-kubernetes): stage workspace Secrets per runtime generation Every Secret the Kubernetes driver writes into a workspace namespace is now scoped to one sandbox runtime generation, immutable, and created with create only, so the gateway never reads, patches, or adopts an existing Secret there. This removes the gateway's cluster-wide get and patch on the copied client TLS and image-pull Secret names. - In managed mode, create an immutable copy of each configured image-pull Secret per generation, named os-pull-<id>-<generation>-<n> and owned by the generation's workload and supervisor Pods. Pods and the restarted Sandbox template reference those names, and generation cleanup deletes them by name. A Secret already holding a generation name fails the create. - Outside shared mode, stage the gateway client TLS material into the supervisor bootstrap Secret instead of copying the client TLS Secret into the workspace namespace. - Remove the fixed-name TLS and image-pull copies, the target ownership read, and the ClusterRole get and patch rule on the copied names. Source reads stay in the sandbox-namespace Role. - Update the managed e2e to expect generation image-pull Secrets and client TLS material in the supervisor bootstrap Secret, and to check that the gateway cannot read the copied names in workspace namespaces. - Update the gateway config and compute driver references, Kubernetes setup and sandbox runtime docs, compute-runtime architecture, driver README, and cluster debugging skill. Signed-off-by: Kris Hicks <khicks@nvidia.com> * fix(helm)!: grant operator-mode Secret permissions through the workspace chart The operator-mode gateway ClusterRole grants no Secret permissions. The openshell-workspace chart Role, installed in each operator-managed namespace, grants the gateway create and delete on Secrets for sandbox runtime generations. Operator-managed namespaces require the workspace chart. - Fail the chart tests on any ClusterRole rule that includes Secrets in operator and shared modes. - Install the workspace chart when the operator e2e provisions a namespace, and assert that the gateway has no Secret permissions in a namespace without it. - Update the Kubernetes setup docs, 0.1.0 upgrade guide, compute-runtime architecture, and cluster debugging skill. Signed-off-by: Kris Hicks <khicks@nvidia.com> --------- Signed-off-by: Kris Hicks <khicks@nvidia.com> |
||
|
Jim Meyer
|
62df64625b |
fix(identity): assess leaf and ancestor executable identities (#3633)
* fix(security): pin supplied executable identity chains Signed-off-by: Jim Meyer <jimeyer@nvidia.com> * docs(security): document executable identity chain pinning Signed-off-by: Jim Meyer <jimeyer@nvidia.com> * fix(binary-identity): compile Linux ancestry hashing Signed-off-by: Jim Meyer <jimeyer@nvidia.com> * test(binary-identity): avoid cross-label ancestry fixture Signed-off-by: Jim Meyer <jimeyer@nvidia.com> * test(e2e): choose distinct denied TCP port Fix flaky test due to sequential port assignment on MacOS Signed-off-by: Jim Meyer <jimeyer@nvidia.com> * fix(identity): bound executable evidence cache Reject identity chains atomically when the supervisor cache reaches its hard limit, and preserve existing pins without eviction. Classify malformed or conflicting evidence separately from policy denials at the staged TCP boundary. Signed-off-by: Jim Meyer <jimeyer@nvidia.com> --------- Signed-off-by: Jim Meyer <jimeyer@nvidia.com> |
||
|
Drew Newberry
|
52aac37866 |
fix(network): honor HTTP response connection closure (#3581)
* fix(network): honor HTTP response connection closure Signed-off-by: Drew Newberry <anewberry@nvidia.com> * test(network): fix EOF fixture socket setup Signed-off-by: Drew Newberry <anewberry@nvidia.com> * test(network): keep pipeline probe response reusable Signed-off-by: Drew Newberry <anewberry@nvidia.com> --------- Signed-off-by: Drew Newberry <anewberry@nvidia.com> |
||
|
Piotr Mlocek
|
bed9e5eafc |
fix(supervisor): use better error message when sandbox connect is not available (#3572)
* fix(supervisor): explain unavailable main terminal attachments Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> * fix(supervisor): reset cursor after terminal attachment errors Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> * fix(supervisor): format read-only warnings for terminal clients Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> * docs: drop terminal attachment documentation additions Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> * fix(supervisor): exit read-only viewers on Ctrl-C Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> * docs: simplify read-only viewer Ctrl-C guidance Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> --------- Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> |
||
|
Evan Lezar
|
6cb1140c66 |
fix(drivers): normalize label namespace (#3609)
Signed-off-by: Evan Lezar <elezar@nvidia.com> |
||
|
krishicks
|
f8002d19ad |
fix(e2e): repair the credential driver test (#3565)
The credential driver e2e has not passed end to end, and the disabled kubernetes-credential-drivers CI lane hid three problems. The test broke when the JSON format of provider list changed. Continuation-token pagination (#3249) changed provider list --output json from a bare array of providers to an object with next_page_token and a providers array. The test still parsed the output as an array, so it failed before checking either storage backend. Read the providers array from the new object instead. Its sandbox name was about 58 characters, but sandbox names are DNS-routable and limited to 19, so sandbox creation was rejected. Build a short unique name instead. The sandbox guard deletes its sandbox from a detached thread on drop, so the test deleted the provider while the sandbox still existed. The gateway rejects deleting a provider that is attached to a sandbox, the test ignored that error, and the credential Secret remained. Delete the sandbox explicitly before returning from the sandbox check. Signed-off-by: Kris Hicks <khicks@nvidia.com> |