mirror of
https://github.com/NVIDIA/OpenShell.git
synced 2026-11-07 08:55:37 +08:00
fa2d381eaa9e917fd074b4bc49efea8a2ad3e43e
1034
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Jesse Jaggars
|
2c31ac7e60 |
feat(network): support additional destination CAs (#3292)
* feat(network): support additional destination CAs Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(network): harden additional CA propagation Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(network): narrow additional CA review scope Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(network): harden additional CA rotation Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * feat(network): complete additional CA delivery Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(docker): remove stale runtime fixture field Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(network): adapt additional CA series to upstream main Resolve rebase fallout against current upstream main after the series was rebased 214 commits forward: - follow upstream #3110: supervisor→gateway TLS is CA-only, so configure_gateway_tls and its isolation tests drop the client identity - follow upstream tls_secret_names reduction to [String; 1] in the Podman driver while keeping the network trust bundle delivery - port [openshell.supervisor.network] rendering into the openshell.gatewayConfigToml serializer in _toml.tpl - adapt tests to upstream struct/function changes (DriverStartupContext, SandboxRunConfig, IsolationSpecInput, supervisor_pod, VM bootstrap image) - drop conformance-artifact-prefix plumbing removed upstream Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * refactor(network): adopt upstream additional CA bundle surface Unify the additional destination CA series on the mechanism merged to main in #3996: one operator pre-assembled PEM bundle, the --additional-ca-bundle supervisor flag, and the /etc/openshell/tls/additional/ca-bundle.pem staging path. - Replace the gateway-level additional_ca_cert_paths list with a single additional_ca_bundle file validated and staged at gateway startup. - Restore the upstream supervisor argument path and fold-into-roots behavior; drop the --network-additional-ca-bundle and --network-additional-ca-digest contracts. - Keep gateway normalization bounds, content-addressed staging, digest-based driver reconciliation, and stop/start rotation; the freshness check now runs at the gateway/driver boundary instead of the supervisor boundary. - Reject additional_ca_bundle in operator driver JSON config; standalone drivers self-source the bundle from --sandbox-additional-ca-bundle with the same validation. - Extend the Kubernetes ConfigMap name digest segment to keep the full 63-byte budget after the prefix rename. - Box the persisted-sandbox reconciliation futures so callers stay under the clippy::large_futures threshold. Updates Docker, Podman, Kubernetes, and VM delivery, Helm chart and tests, e2e wiring, gateway configuration docs, and the cluster debugging skill to the single mechanism. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(ci): make gateway startup tests wrap-tolerant and refresh example lockfile The network trust startup tests asserted on raw substrings of gateway diagnostics, but the CLI renderer wraps output at a fixed width with continuation markers and hard-breaks long paths. Compare assertions on whitespace- and continuation-squashed text so they hold regardless of wrap position. Format the e2e workspace (separate cargo workspace, not covered by the root cargo fmt run) and refresh examples/isolation-backend-v8/Cargo.lock for the libc dependency added to the network trust crate. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(podman): reuse TLS CA secret during stopped-pair trust replacement The additional CA e2e never ran in CI before the first maintainer test dispatch, and its local container inspection was broken in two ways: it located the supervisor container through e2e network membership (the supervisor is host-networked or runtime-bridged, so the filter matched the CI job container instead), and it required the gateway mTLS CA as a bind mount (Podman delivers it as a container secret when user namespaces remap UIDs). Select the supervisor by managed labels and accept the secret delivery form. The lifecycle leg then exposed a driver bug: stopped-pair trust replacement re-created the per-sandbox TLS CA secret, colliding with the secret still owned by the renamed stopped pair. Mirror the token and resolver secret handling: require and reuse the existing secret during replacement instead of creating it. Validated end-to-end with the Podman additional CA e2e lane. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(server): restore upstream recovery semantics for start-request failures Recovery no longer marks sandboxes Error/InvalidPersistedSandbox when driver_start_request fails during the startup or lifecycle sweeps. Gateway configuration problems (for example a missing launch signer) must not permanently fail unrelated sandbox records; the sweep logs a warning and skips them, matching the behavior before the additional CA work. Sandbox-record failures surface through the normal start path. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * refactor(podman): refresh stale supervisors without replacing workloads Replace the stopped-pair trust reconciliation with supervisor-only recreation. The destination-trust generation label moves from the workload to the supervisor container, and start_sandbox recreates only the supervisor when its generation is missing or stale. The workload container, its volumes, its secrets, and its workspace data are never replaced: the additional CA bundle is mounted only into the supervisor. Running sandboxes are never gated on trust labels, and recovery without fresh launch credentials keeps the existing supervisor. The intentional -removal fence machinery in the watcher existed only for workload replacement and is removed with it. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * refactor(docker): drop workload replacement for destination-trust changes Delete the stopped-workload trust reconciliation path and all of its machinery: the trust-generation label on workload containers, the generation match check in the start fence, the stopped-sandbox replacement, the bounded workspace archive copy with its 512 MiB cap and regular-file-only validation, and the boundary-state rollback helpers. No replacement machinery is needed: the trusted supervisor container is already re-created from the gateway's current configuration on every start via ensure_control_process_for_container, so stop/start picks up the current additional CA bundle while the workload container, its volumes, and its workspace data are never replaced. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * docs(configuration): drop reconciliation wording from additional CA lifecycle The stop/start lifecycle description already matches the supervisor-only refresh model; remove the two remaining references to restart reconciliation of workload containers. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(network): port additional CA test fixtures to the crypto interface Route the gateway TLS isolation test PKI, the destination CA source fixtures, and the gateway startup trust tests through openshell_crypto::pki now that rcgen access is centralized there: generate_keypair/self_signed/signed_by replace the direct rcgen calls and CertifiedKey field access follows the wrapper's shape. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(e2e): repair additional CA lane for Docker and non-container builds The first maintainer test dispatch revealed two latent issues in the additional CA e2e that no local run had exercised: the Podman gateway CA secret helper referenced ContainerEngine without the container feature gate, breaking Kubernetes and VM builds of the test, and the Docker branch asserted a client ca.crt bind mount that the Docker driver never creates. Gate the helper like its callers, and assert Docker's actual delivery path instead: the gateway mTLS CA staged as tls/ca.pem inside the read-only supervisor state volume, selected through the OPENSHELL_TLS_CA environment pointer. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(helm): restore alphabetical README ordering for supervisor keys The helm:docs check regenerates the chart README with helm-docs, which orders rows alphabetically; a conflict resolution had placed supervisor.network.additionalCaConfigMapName after the supervisor.podSecurityContext rows. Regenerate with mise run helm:docs. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * chore: refresh testkit lockfile for openshell-core libc dependency Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(driver-podman): stage operator CA bundle as driver-owned artifact Standalone Podman drivers accepting --sandbox-additional-ca-bundle previously kept the operator's source path as the trust artifact path. The immutable-artifact verification performed on every create/start requires exactly mode 0444, which an operator-owned source path cannot guarantee, so a readable-but-writable source failed reconciliation. Read the operator bundle with the same bounded, no-symlink validation, then stage the normalized material into a driver-owned artifact inside a private tempdir held for the driver process lifetime. The artifact is created 0600, synced, and only then restricted to 0444 so no observer ever sees a writable artifact. The operator source and its permissions are never referenced after startup. Addresses GATOR-291f43b2-02. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(driver-podman): pin retained workload image during supervisor refresh Supervisor-only trust reconciliation re-entered create_sandbox_with_existing_token, which re-pulled and re-resolved the workload image tag even though the workload container is retained. If the tag moved to a different image between stop and start, identity resolution and child-environment construction would reflect the new image while the retained workload still runs the original one, leaving mode-0600 bootstrap files the retained user cannot read. During a refresh with a retained workload, inspect the stopped container and pin to the immutable image ID it was created from (Config.Image); never pull or resolve the request tag. The requested reference is still used for child metadata so the retained workload's environment is byte-identical to its original launch. Addresses GATOR-291f43b2-01. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * test(e2e): verify host-only VM trust delivery and drop curl dependency The additional-CA lane's VM assertions read the destination CA out of the guest overlay and a guest-side supervisor-argv marker — artifacts from the guest-supervisor design that no longer exist now that the VM driver stages the CA host-only and runs the supervisor on the host. Inspect the driver-staged host artifact (digest-compared against the source), assert the guest overlay contains no CA material at the old mount paths, and keep the serial-console no-disclosure check. The workload validation scripts also ran curl, which exists only in the container-engine fixture image; the Kubernetes and VM lanes use the uv-based workload image and died instantly with "canonical main process already finished". Rewrite the main-process, exec, and lifecycle validation scripts in python3 (present in every lane's workload image). The network-policy binary identity is the resolved /proc/<pid>/exe target, so the versioned python3.12 binaries are allowlisted rather than the unversioned symlinks. The hostname mismatch check keeps its semantics: same reachable fixture, mismatched TLS SNI, and the failure must specifically be hostname verification. Addresses GATOR-291f43b2-04. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(kubernetes): allow admission-added labels on destination-trust ConfigMaps The destination-trust ConfigMap validation compared the whole label map for equality. A mutating admission webhook that attaches an organizational label to the created ConfigMap made the create response fail validation, and the retained ConfigMap then failed every subsequent create/start through the existing-object path, blocking sandbox launches despite the ownership labels, immutable flag, digest, and certificate bytes remaining correct. Validate the gateway-owned label pairs individually and allow unrelated labels; the exact trust-material checks are unchanged. Covered by a regression test with an admission-added label and a missing gateway-owned label. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(podman): close supervisor-refresh rollback gaps The supervisor-only trust refresh renamed the previous supervisor to a random rollback name and then performed the token-secret lookup. A transient lookup failure returned with the supervisor still renamed, where a normal delete could never find it, and the rollback rename and final rollback removal ignored their own failures, leaving orphans with attached control and channel resources. Complete every fallible preparation before the rename. On a failed replacement, restore the canonical name; if the restore itself fails, return an error naming the rollback container so the orphan stays identifiable — a later retry still succeeds because the canonical name is absent and the replacement path recreates a fresh supervisor. Treat not-found during the final rollback removal as success and name the orphan in both the log and the error on other removal failures; a retry observes the fresh canonical supervisor and resumes the normal start path. Covered by failure-injection tests for each transaction boundary: preparation failure before any rename, replacement failure restoring the canonical name, and restore failure naming the orphaned rollback container. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * docs: address destination-trust review corrections The gateway-configuration RFC documented an additional_ca_cert_paths list that the strict configuration parser never accepted; document the additional_ca_bundle single-path setting the parser actually enforces. The debug-openshell-cluster skill named the wrong generation annotation (openshell.ai/network-supervisor-trust-generation) and described digest argv and network-init canonicalization checks the implementation does not perform. Correct it to openshell.ai/additional-ca-generation and describe the actual delivery: the supervisor Pod mounts the immutable ConfigMap read-only and receives --additional-ca-bundle for the mount path, with the driver validating the digest annotation and exact bytes per creation. Classify the destination-trust generation hashes as deterministic identity derivation rather than authenticity proofs, following the openshell-crypto exclusions table: trust decisions come from the gateway's normalization, immutable staging, exact-byte validation, and TLS verification. Record the classification in the crypto README and at each hashing site. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(network): reconcile additional CA tests with suspend/resume and auth changes Upstream's suspend/resume lifecycle took StartSandboxRequest fields 7 and 8, so the durable-snapshot field moves from 7 to 9; update the wire-format test to assert the field-9 key (0x4a) and initialize the new resume controls. Repair the cli test module after the auth opt-in test merged alongside the environment-guard helpers, and initialize the gateway driver-startup context's network-trust fields in the shared supervision test. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(server): box start-recovery futures for clippy large-futures The durable-snapshot field grows StartSandboxRequest enough that the start-recovery match and the recovery-expiry test exceed the large-futures limit under -D warnings. Box::pin both futures, matching the pattern the recovery path used before the suspend/resume rework. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> --------- Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> |
||
|
Jesse Jaggars
|
26134dc735 |
feat(gateway): add hashed service-routing hostnames for multi-gateway clusters (#4842)
* feat(gateway): add hashed service-routing hostnames for multi-gateway clusters
Closes #4840
Add an optional service_routing_gateway_name gateway setting (TOML, CLI,
env, Helm) that disambiguates exposed service URLs when multiple gateways
share a cluster, ingress, and wildcard certificate. When configured,
exposed-service hostnames become a deterministic 16-character lowercase
base32 SHA-256 label of (gateway name, workspace, sandbox, service) under
the same base domain, keeping every hostname exactly one label deep for
wildcard-certificate compatibility. Unset gateways preserve the existing
readable hostname format byte-for-byte, and configured gateways still
accept legacy hosts during migration.
Hashed hosts resolve through an in-memory route index hydrated from
persisted ServiceEndpoint records at startup, refreshed on miss for
multi-replica deployments, maintained across expose, delete, and sandbox
cleanup, and enforced for collisions before endpoint persistence.
Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>
* fix(lint): satisfy clippy doc lints on main-landed sandbox_runtime code
The shared-supervisor landing (
|
||
|
Drew Newberry
|
e5adf6f484 |
feat(supervisor): recover shared instances with hash-based HA (#4864)
* feat(supervisor): recover shared instances with hash-based HA Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(supervisor): keep HA coordination out of compute drivers Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(supervisor): fence pool admission and reconciliation races Signed-off-by: Drew Newberry <anewberry@nvidia.com> * feat(supervisor): keep pool mode stable across replica scaling Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(supervisor): default shared deployments to pool mode Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(supervisor): fence refused launches and remove legacy mode Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(supervisor): confirm live owners and fence stale observations Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(supervisor): preserve takeover cause log formatting Signed-off-by: Drew Newberry <anewberry@nvidia.com> * feat(helm): add memory-based supervisor autoscaling Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(supervisor): retain acknowledgements and fence streamed input Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs(supervisor): consolidate experimental multi-tenant guidance Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs(supervisor): simplify multi-tenant supervisor guide Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(supervisor): simplify pool member identity Signed-off-by: Drew Newberry <anewberry@nvidia.com> * feat(helm): make supervisor affinity configurable Signed-off-by: Drew Newberry <anewberry@nvidia.com> * feat(helm): make supervisor disruption budget optional Signed-off-by: Drew Newberry <anewberry@nvidia.com> * test(supervisor): align HA fixtures with suspend support Signed-off-by: Drew Newberry <anewberry@nvidia.com> --------- Signed-off-by: Drew Newberry <anewberry@nvidia.com> |
||
|
Drew Newberry
|
f07127e5d7 |
feat(sandbox): add suspend and resume lifecycle (#4765)
* feat(sandbox): add suspend and resume lifecycle Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(sandbox): preserve V8 state across failed resume launches Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(sandbox): handle suspend and resume recovery races Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(example): move V8 suspend support to a dependent change Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(sandbox): align suspend tests with current supervisor APIs Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(sandbox): isolate suspend requests and resume compensation Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(supervisor): allow restoration to wait until agent admission Signed-off-by: Drew Newberry <anewberry@nvidia.com> --------- Signed-off-by: Drew Newberry <anewberry@nvidia.com> |
||
|
Taylor Mutch
|
7b0f8ae07c |
fix(gateway): require authentication or explicit development opt-in (#4821)
* fix(gateway): require authentication or explicit development opt-in Signed-off-by: Taylor Mutch <taylormutch@gmail.com> * fix(gateway): reject implicit unauthenticated admin requests Signed-off-by: Taylor Mutch <taylormutch@gmail.com> --------- Signed-off-by: Taylor Mutch <taylormutch@gmail.com> |
||
|
Shailendra SinghandDrew Newberry
|
17eda52d45 |
docs(windows): route main documentation to the Windows release branch (#4758)
* docs(windows): route main documentation to the Windows release branch Signed-off-by: Shailendra Singh <shailendras@nvidia.com> * docs(windows): include native MXC tests in full validation Signed-off-by: Shailendra Singh <shailendras@nvidia.com> * docs(windows): address documentation review feedback Signed-off-by: Shailendra Singh <shailendras@nvidia.com> * docs(windows): include E2E in full validation Signed-off-by: Shailendra Singh <shailendras@nvidia.com> * docs(windows): stop validation after any failed task Signed-off-by: Shailendra Singh <shailendras@nvidia.com> * docs(windows): simplify quickstart and installation guidance Signed-off-by: Drew Newberry <anewberry@nvidia.com> --------- Signed-off-by: Shailendra Singh <shailendras@nvidia.com> Signed-off-by: Drew Newberry <anewberry@nvidia.com> Co-authored-by: Drew Newberry <anewberry@nvidia.com> |
||
|
Free Wortley
|
f5d9223e08 |
feat(supervisor): let operators require specific middleware bindings (#4678)
* feat(middleware): authorize supervisor bindings at registration Let operators require and limit the operation and phase pairs that an external middleware may expose. Validate the same binding policy at gateway startup and when a supervisor discovers the service, and keep only authorized bindings in the registry used for dispatch. Preserve dynamic discovery and the existing wire configuration when no binding policy is set. Document that older supervisors must be upgraded before relying on binding restrictions. Signed-off-by: Free Wortley <freeqaz@openai.com> * docs(skills): check which supervisor middleware hooks the operator authorizes Signed-off-by: Free Wortley <freeqaz@openai.com> * fix(go): regenerate the sandbox protobuf binding Restore the comment separator emitted by the pinned generator. This keeps the checked-in binding byte-for-byte reproducible without changing the protobuf schema or Go behavior. Signed-off-by: Free Wortley <freeqaz@openai.com> --------- Signed-off-by: Free Wortley <freeqaz@openai.com> |
||
|
Simon Scatton
|
9c6c25204c |
fix(deps): update tokio and crypto-bigint security patches (#4924)
* fix(deps): update tokio and crypto-bigint security patches Signed-off-by: Simon Scatton <sscatton@nvidia.com> * chore(deps): document temporary crypto-bigint advisory exception Signed-off-by: Simon Scatton <sscatton@nvidia.com> * fix(sandbox): migrate owned descriptors to AsyncFd register Signed-off-by: Simon Scatton <sscatton@nvidia.com> * test(server): avoid deadlock in concurrent CAS retry test Signed-off-by: Simon Scatton <sscatton@nvidia.com> --------- Signed-off-by: Simon Scatton <sscatton@nvidia.com> |
||
|
krishicks
|
92434dcb2b |
test(windows): repair the Windows MSVC test suite (#4862)
* fix(isolation): validate Unix socket paths the same way on every host Prior to this change, SandboxTransport::validate checked Unix socket paths with Path::is_absolute(). On Windows that requires a drive or UNC prefix, so a POSIX path such as /run/backend.sock was rejected as "invalid boundary control transport endpoint". The isolation backend registration work in #3898 added tests that register Unix endpoints in the core registration types, gateway config, gateway config preflight, and supervisor backend setup; all of them failed on the Windows MSVC runners. The Windows test job only runs on PRs labeled test:windows, so the break went unnoticed on main. Unix transport is refused at runtime on non-Unix hosts, so these paths are always POSIX paths regardless of where they are validated. Now, both the core registration validator and the sandbox backend protocol validator use a new openshell_core::paths::is_posix_absolute helper, which applies the POSIX rule (the path starts with '/') on every host. On Unix this matches is_absolute() exactly. Signed-off-by: Kris Hicks <khicks@nvidia.com> * test(supervisor): run baseline write-back cases only on Unix hosts The startup policy write-refusal tests from #3785 rely on enrich_proto_baseline_paths adding the proxy baseline paths (/usr, /tmp, /dev/null, ...) to a policy, which triggers a write-back that the fake gateway then refuses. Enrichment only adds paths that exist on the host. Prior to this change, the tests assumed those paths exist, so on Windows enrichment added nothing, the precondition assertions failed, and the baseline case of permanent_codes_still_exit never saw a write to refuse. Now, the four baseline write-back tests are gated to Unix, and permanent_codes_still_exit checks the image policy upload case on every host and the baseline write-back case only on Unix. Signed-off-by: Kris Hicks <khicks@nvidia.com> * test(server): scale the SQLite concurrent-write test for Windows fsync sqlite_concurrent_writes_return_their_own_resource_version (#4408) runs 8 writers that each make 25 unconditional and 25 version-checked writes against one row of an on-disk WAL database with synchronous=FULL. Writers serialize on the SQLite write lock and wait out sqlx's default 5s busy_timeout. Prior to this change, the per-commit fsync cost on the Windows runners was high enough that a waiting writer starved past that timeout and failed with "database is locked" after about 7s. Now, each writer makes 5 writes on Windows and 25 elsewhere. The test still races every writer on the same row, which is what it checks. Signed-off-by: Kris Hicks <khicks@nvidia.com> * test(server): hold scope-mix locks for wall-clock time, not timer ticks random_scope_mix_never_deadlocks (#4147) runs 64 tasks of 50 random lock operations and holds each operation's locks for 0-2 ms, failing if the mix does not finish within 20s. Prior to this change, the hold was a tokio::time::sleep. Windows timers round a 1 ms sleep up to about 16 ms, and a quarter of the operations take the exclusive global lock, so the serialized holds alone took several seconds and the full mix exceeded the 20s deadline on the Windows runners. Now, occupy yields until a std::time::Instant deadline, so holds last 0-2 ms of wall-clock time on every platform and the deadline means the same thing everywhere. Signed-off-by: Kris Hicks <khicks@nvidia.com> * test(supervisor): run sandbox instance host tests only on Unix hosts The sandbox_instance_host tests added with per-instance supervisor state (#4701) admit hosted instances through SandboxInstanceHost::new, which prepares a private directory per instance. Hosted sandbox instances are Unix-only: on other hosts prepare_private_dir returns "hosted sandbox instances require a Unix host". Prior to this change, the test module still compiled and ran on Windows, so all three tests failed while creating their host. Now, the test module is gated to Unix, and the Unix-only permission check inside one test no longer needs its own cfg attribute. Signed-off-by: Kris Hicks <khicks@nvidia.com> --------- Signed-off-by: Kris Hicks <khicks@nvidia.com> |
||
|
Drew Newberry
|
9a85667d0d |
feat(kubernetes): host sandboxes on the shared supervisor (#4702)
* feat(sandbox-backend): route runtime connections to shared supervisor sessions A supervisor shared by several sandboxes needs one listener for every session's runtime-initiated Sandbox Protocol connections. The runtime now sends a fixed, bounded routing preface (magic, version, length, and an opaque route handle) before its pinned TLS handshake when its bootstrap names a route. RuntimeRouter treats the handle only as a hint that selects which generation-pinned TLS identity to demand, closes unknown, removed, and replaced handles before any handshake, bounds unauthenticated handshakes globally and per route, and queues an authenticated connection only for its own session, so no peer can displace a live channel. A routed session waits up to the bootstrap grace for its first connection, because it is registered before its workload Pod is scheduled. The session_control module defines the trusted contract between a compute driver and a shared supervisor: registering one generation's runtime descriptor and gateway-issued credentials, reading its status, and removing it, over mutually authenticated TLS 1.3. Dedicated descriptors and bootstrap listeners keep their encodings. Signed-off-by: Drew Newberry <anewberry@nvidia.com> * feat(supervisor): host gateway-registered sessions in one process openshell-supervisor experimental-host replaces the idle bootstrap process with a session host. A mutually authenticated session-control API accepts registrations only from the admission certificate holder, the gateway. Each registration starts a SessionHost session that acts only with the gateway credentials issued for its generation, keeps its proxy CA, trust directory, and SSH socket under a private directory, and receives its runtime connections through the routed listener. Status reports each session's own readiness; the health socket keeps reporting process health only. Removing a session runs on a task the host owns, so it completes when the caller disconnects, concurrent removers share its result, and the route stays reserved until the session has ended and its files are gone. An identical registration is accepted again and a different one for a registered route is refused. Sessions get no upstream proxy, extra CA bundle, or provider workload identity; the Kubernetes driver refuses configurations that need them. Signed-off-by: Drew Newberry <anewberry@nvidia.com> * feat(server): issue a TLS-only private PKI from generate-certs generate-certs --tls-only creates just a server and a client TLS Secret from a fresh CA, with the same create-if-missing contract and a recovery hint for partial state. The Helm certgen hook uses it for private PKIs that must not carry a sandbox-JWT signing key. Signed-off-by: Drew Newberry <anewberry@nvidia.com> * feat(kubernetes): supervise gateway-topology generations as shared sessions With experimental_supervisor_topology = "gateway", each new runtime generation becomes a session on the gateway's shared supervisor instead of a supervisor Pod and boundary Service of its own. The generation records its placement and session route on its Sandbox resource when it starts, and every later decision reads that record, so changing the setting never reinterprets a running generation and unknown records fail closed. The runtime identity ends with the session route, which no Pod UID can equal, so Pod authentication cannot vouch for a shared generation. Create and start add a per-sandbox egress allowance to the Pods the shared supervisor's Service selects, bind it into the boundary evidence, and register the session over session control before releasing the workload; registration does not wait for the runtime, so create still returns before the gateway persists the generation. Readiness follows the session's own status. Stop, restart, and delete remove only that session, and reconciliation suspends a workload whose session is gone and removes sessions that no current generation may run. Configurations with an upstream proxy or provider SPIFFE identity are refused. Closes #4700 Signed-off-by: Drew Newberry <anewberry@nvidia.com> * feat(helm): deploy and wire the shared supervisor With the gateway topology enabled, the chart runs the session host as a single-replica Deployment with its Service and a NetworkPolicy that admits session control only from the gateway and runtime connections only from sandbox workload Pods. The certgen hook issues a private session-control PKI whose client credential only the gateway mounts, the gateway may read only that Service, and the chart renders the matching experimental_shared_supervisor table. The supervisor Pod has no service account token. Settings the shared supervisor cannot isolate, and a hand-written connection table, fail at render time. Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(ocsf): attribute events to session-bound contexts A supervisor process that hosts several sandboxes cannot use one process-wide OCSF context: every event would carry the wrong sandbox identity, and middleware requests read their sandbox identity from the same context. Builders now accept a ContextRef, either a borrowed context or a context shared by a hosted session, so per-session contexts need not be leaked. ctx() returns the context bound to the calling thread when present and falls back to the process-wide context otherwise. OCSF JSON output settings can likewise be bound per thread, and OpenTelemetry can parent a span from an explicit W3C trace context instead of process environment. A dedicated supervisor keeps using the process-wide context. Signed-off-by: Drew Newberry <anewberry@nvidia.com> * test(e2e): run two sandboxes on one shared Kubernetes supervisor The e2e:kubernetes:shared-supervisor lane enables the gateway topology and creates two sandboxes with different policies and provider credentials. It checks that one shared supervisor process hosts both, that each session enforces its own policy and injects its own credential, that stopping one leaves the other working, that the routed listener refuses stale and unknown routes and drops a peer that names a live route without its pinned identity, and that a restart registers a fresh session on the same process. Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(supervisor): scope gateway credentials and network state to a session The supervisor kept its gateway bearer, Sandbox Protocol bearer, and renewal task in process-wide slots, and the network supervisor read provider token grants, proxy CA files, and the gateway identity from process environment. A second sandbox in the same process would have replaced or rotated the first sandbox's credentials, reused its provider access tokens, and exposed its denials through policy.local. GatewayCredentials now holds one session's gateway and Sandbox Protocol credentials and their renewal; every gateway client receives a GatewayEndpoint that pairs the URL with them, including log push, ConnectSupervisor and its redirects, and extension-credential rotation. run_networking takes a NetworkSession with the sandbox identity, the session's gateway access, its Workload API socket, its policy.local denial source, and its proxy CA files. Provider token grants use a cache and HTTP client owned by the session and fail closed without a Workload API socket. run_session supervises one generation from an explicit SessionConfig and owner-provided SessionControls. run_sandbox_with_backend remains the dedicated entry point: it keeps the process-wide OCSF context, reads the same environment, binds readiness endpoints to session readiness, and stops on SIGTERM or SIGINT. The binary shares one set of credentials between log push and supervision, as before. Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs: describe the experimental shared supervisor Document the gateway supervisor topology, its chart-owned connection settings and limits, and how to debug and test it. Signed-off-by: Drew Newberry <anewberry@nvidia.com> * feat(supervisor): host concurrent sessions on session-bound runtimes SessionHost runs several supervisor sessions in one process. Each session gets its own Tokio runtime, and every thread that runtime starts, plus the thread driving it, is bound to the session's OCSF context, OCSF JSON settings, log sink, and denial log. Tasks, blocking work, reconnects, and credential renewal spawned by a session therefore keep its identity, while threads outside any session carry none and push no logs. Gateway credentials stay explicit per session. Removing a session stops its workload gracefully, or cancels it at once while it is still starting, then shuts down only its runtime and revokes its gateway credentials. The key stays registered until the session thread has finished, so the same generation cannot gain a second owner while one is stopping. A panicking session reports a failure, clears its readiness, and is revoked the same way. Hosted sessions get a private proxy trust directory and serve policy.local denials from their own log. A focused test runs two sessions concurrently and checks that nested and blocking tasks see their own identity, gateway requests carry their own bearer, log lines reach only their own stream, and removing one leaves the other supervising. Closes #4699 Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix: tighten shared supervisor naming, removal deadlines, and route probes The shared supervisor's resource name appended its suffix before truncating, so a release fullname of 51 or 52 characters produced the gateway Service's own name. A fullname without room for the suffix now keeps a prefix and a hash of the whole fullname. Removing a session waits up to 20 s for its workload to stop and 5 s for the session runtime to shut down, but the driver gave DELETE the same 15 s budget as other session-control calls and reported normal removals as failures. The contract now defines the removal grace and a 45 s removal deadline, and deleting a sandbox no longer waits on a session whose workload is already going away. The shared host runs no supervisor OTLP trace or OCSF exporter. The driver warns at startup when the gateway configures that export, and the docs state the limit. experimental-host keeps its Unix-only imports in its Unix entry point. The e2e route probe fails when the supervisor neither refuses nor challenges a connection, or keeps a challenged connection open after a malformed TLS answer, and it checks that every probe reached the supervisor Pod. Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(supervisor): route shared supervisor crypto through openshell-crypto Main now routes first-party TLS providers and digests through the openshell-crypto facade. Build the session-control TLS configurations and the routed-runtime test server from its configuration provider, install the experimental host's default provider through it, and hash shared session registrations with its SHA-256 instead of a direct sha2 dependency. Signed-off-by: Drew Newberry <anewberry@nvidia.com> * feat(supervisor): report shared supervisor health over TCP experimental-host takes --health-port instead of a Unix health socket, matching experimental-bootstrap. It binds the wildcard health listener after the session-control and runtime listeners and accepts and closes connections while the process runs. The chart probes the named container port health (5502) with native kubelet tcpSocket checks; session control stays on 5501, the Service exposes only runtime and session control, and /run/openshell remains for per-session files. Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(supervisor): probe shared supervisor health on session control The shared supervisor's kubelet tcpSocket probes now connect to the mTLS session-control listener (5501) instead of a separate health listener. Drop the experimental-host --health-port flag, its accept loop, and the chart's 5502 container port. Probes open and close a TCP connection; every session-control request still requires mTLS. Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(supervisor): use shared mode flags for the session host Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(supervisor): describe the shared host module by its mode Drop the experimental label from the shared host module documentation so it matches the --mode=shared entry point. No behavior changes. Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(test): preserve shared supervisor workspace overlays Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(supervisor): name hosted lifetimes sandbox instances A sandbox instance is one running generation of a stable Sandbox resource. Gateway transport connections and the gateway supervisor sessions they carry can reconnect within one instance, so the hosted object, its configuration, controls, host, and per-instance network, denial, and token-grant state now use the sandbox instance name. Rename session_host to sandbox_instance_host, run_session to run_sandbox_instance, and SessionConfig, SessionControls, SessionHost, and related hosted types to their SandboxInstance equivalents. Gateway supervisor sessions, SSH sessions, and authentication session types keep their names. No wire, configuration, or lifecycle behavior changes. Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(supervisor): launch sandbox instances from one config and prepared bindings Replace SandboxRunConfig and its field-by-field copy into SandboxInstanceConfig with a single launch configuration. Live per-instance state (gateway credentials, OCSF context and JSON settings, log sink, and denial log) is now an explicit SandboxInstanceBindings value that the owner prepares once, instead of placeholders the runner or host overwrote. The dedicated binary loads launch inputs from its arguments and environment and shares one GatewayCredentials with log push; run_sandbox_with_backend keeps the process-wide OCSF context, health endpoints, and signal handling. The instance host binds the same SandboxInstanceBindings to its runtime threads and revokes their credentials when the instance ends. Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(supervisor): use session-control wire types for instance status and trace context Drop the supervisor's duplicate SandboxInstancePhase, SandboxInstanceStatus, and TraceContext families. The sandbox instance host reports SandboxInstanceStatusReport directly, with its key as the route, so the shared host no longer converts or re-keys status, lists, or removal results. SandboxInstanceConfig carries RegistrationTraceContext and the dedicated process reads it through trace_context_from_environment. No wire definitions change. Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(supervisor): update Windows host proxy test config Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(supervisor): assign shared sandbox instances over a supervisor-initiated gRPC stream Replace the private HTTP/JSON session-control API, which the gateway dialed on the shared supervisor, with a supervisor-initiated stream to the gateway. The gateway owns the desired set of sandbox instances; the shared supervisor attaches with a gateway-scoped host certificate, receives complete assignment snapshots with session-scoped revisions, reconciles its instance registry to each one, and reports what it observes. Losing the stream never stops an instance; only a later snapshot that omits it does. - Add the private openshell.supervisor.host.v1 protocol, served on the gateway listener but authorized only for the supervisor host certificate, excluded from SDK generation and reflection, and classified in the schema inventory. - Add SupervisorHostController in openshell-core: uninitialized until the driver derives the desired set once, compare-and-replace full resyncs, capacity bound, session fencing by connection epoch, revision-bounded acks, bounded sends. - Classify TLS peers per connection against the ordinary client CA and the supervisor host CA from one configuration snapshot; a chain matching neither or both is refused, and host connections never become users. - Persist each shared generation's launch material in its supervisor Secret so a restarted gateway rebuilds assignments from Sandbox resources, and withdraw instances on stop, delete, and suspend. - Issue the host identity with `generate-certs --client-only`; the gateway mounts only the CA. Drop the control port, server certificate, and gateway ingress rule from the chart; require gateway TLS for the topology. Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(supervisor): authenticate the shared supervisor stream with a gateway-minted supervisor token Replace the shared supervisor's host CA and client certificate with a supervisor token: a bearer the gateway mints from its existing launch-signing key under a profile of its own (typ openshell-supervisor+jwt, subject openshell-supervisor:<gateway_id>, no sandbox, generation, or epoch claims). The supervisor dials the gateway's Attach stream over ordinary server-authenticated TLS and presents the token; the gateway verifies it with the same key infrastructure, grants only the assignment stream, refuses every other credential on it, and ends a stream when the token that opened it expires. The gateway renews the token at half its lifetime and publishes each renewal into the supervisor host controller; the Kubernetes driver's only part in authentication is delivery, replacing the token in the chart-created <release>-supervisor-token Secret, which the supervisor mounts and re-reads on every connection attempt. The gateway Role may get and patch only that Secret. Remove the host-only TLS realm classification, the supervisor host CA mount and config, certgen --client-only, and the supervisor host PKI; gateway TLS returns to its pre-feature behavior and remains required for the topology. Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(helm): remove redundant certgen condition Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(supervisor): extract gateway-owned shared supervision into openshell-supervisor-control Move the shared supervisor control types out of openshell-core and openshell-sandbox-backend into a new openshell-supervisor-control crate: the SupervisorHostController (desired set, revision fencing, host sessions, observations), the SandboxInstanceRegistration launch material and SandboxInstanceStatusReport mapping, and a new SharedSupervision coordinator that owns the control contracts a compute driver previously implemented itself: validated persist-before-assign launches, recovery from durable records with the compare-and-replace baseline captured before the driver lists them, keep-previous-set on any failed or mismatched read, and one interpretation of host observations for readiness and bootstrap. Supervisor token delivery keeps its retry and newer-value policy in the crate; the write itself stays with the driver. The gateway now constructs the coordination: ComputeDriverBuildContext exposes shared_supervision() and the server keeps the controller for the Attach stream and token minting, replacing the driver-created controller that was installed into the gateway. The Kubernetes factory injects it through KubernetesComputeDriver::with_shared_supervision; new() stays the dedicated-topology constructor and refuses a configuration that selects the gateway topology without the gateway's coordination. The Kubernetes driver keeps its platform work: placement annotations, the shared supervisor Service and egress NetworkPolicy, launch Secret create and read (bytes only), its own fence and Pod evidence, and the token Secret patch. The supervisor process consumes the moved registration and status types unchanged. Wire protocol, storage JSON, annotations, and the supervisor token profile are unchanged; the private protobuf stays generated in openshell-core and is re-exported by the new crate. Signed-off-by: Drew Newberry <anewberry@nvidia.com> --------- Signed-off-by: Drew Newberry <anewberry@nvidia.com> |
||
|
Emilien Macchi
|
07b7a74b62 |
fix(server): bound the provisioning-deadline scan's lock wait (#4549)
The provisioning-deadline scan takes each provisioning sandbox's local S(global) S(workspace) X(sandbox) keys in ascending key order and waited for them with no deadline. When a sandbox's key sorts before its workspace's key, the scan held X(sandbox) while it waited behind a provider or profile writer, so start, stop and delete for that sandbox waited for the writer too. In the other order, the scan held S(workspace) while it waited for a busy sandbox, and provider writers on that workspace queued behind it. Either way the rest of the pass waited as well. Each pass now has a 500 ms budget for waiting on these keys. A sandbox whose keys stay busy is skipped and picked up on the next pass a second later. Once the budget is spent, the rest of the pass only takes keys that are free when it reaches them, so a busy workspace or global key costs one wait per pass rather than one per sandbox. The local acquisition is cancel-safe, so giving up releases the keys it already took. Skips log at debug and count in the new openshell_server_provisioning_deadline_skips_total counter, labeled by whether the sandbox was already past its deadline. A steady rate of delayed expiries can then be alerted on without a warning every pass. The gateway metrics page and the cluster debugging skill describe it. The test that pinned the old wait is replaced by tests for both key orders: a lifecycle lock or a provider writer proceeds and the scan finishes while the other side holds its key, the skipped sandbox is untouched and expires once the writer releases, and a skip doesn't end the pass for other sandboxes. Another test checks that sandboxes behind one busy workspace share a single wait. Part of #4307 Closes #4547 Signed-off-by: Emilien Macchi <emacchi@redhat.com> |
||
|
Simon Ruggier
|
eec00ad492 |
fix(podman): Increase the API timeout (#4820)
Experience has shown that ordinary podman container creation can take longer than 30 seconds for some users, so for now, this change doubles the timeout, until the source of the slowness is found and accounted for. Signed-off-by: Simon Ruggier <Simon.Ruggier@gmail.com> |
||
|
Fede Kamelhar
|
78376bd041 |
fix(installer): resolve the gateway config dir like the CLI does (#4063)
* fix(installer): resolve the gateway config dir like the CLI does install.sh hardcoded $TARGET_HOME/.config/openshell in two places: the readiness check, which reads the client certificates it expects the gateway registration to have written, and the cleanup that removes a stale openshell entry before re-adding it. The CLI instead keeps both under $XDG_CONFIG_HOME/openshell whenever that variable is set, so with the variable pointing anywhere other than ~/.config the installer and the CLI looked in different directories: the readiness check waited for certificates that were never going to appear there, and the cleanup left the real stale entry in place. Resolve the directory through as_target_user instead, which answers from the same environment the CLI is invoked in. That covers both branches without duplicating their logic: the same-user branch inherits the caller's XDG_CONFIG_HOME, while the sudo and runuser branches reset the environment and fall back to the target user's home. An empty XDG_CONFIG_HOME falls back to $HOME/.config, following the XDG base directory specification. The CLI diverges here: xdg_config_dir treats a set-but-empty value as set and joins onto it, yielding the relative path "openshell", which no installer path can usefully match. Worth fixing separately in the CLI. Signed-off-by: fede-kamel <fkamelhar@gmail.com> * fix(core): treat an empty XDG_CONFIG_HOME as unset xdg_config_dir() joined onto an empty XDG_CONFIG_HOME and resolved the config root to the relative path "openshell", while the installer falls back to $HOME/.config/openshell for the same value. Registration runs the CLI with the caller's environment, so the two could disagree on where the gateway entry lives. Treat an empty value as unset, as the XDG base directory spec requires, so the CLI and installer resolve the same directory. Add tests for a value with spaces and drive registration cleanup through the helper for the spaces and empty cases. Signed-off-by: fede-kamel <fkamelhar@gmail.com> --------- Signed-off-by: fede-kamel <fkamelhar@gmail.com> |
||
|
Drew Newberry
|
ea13dfce41 |
refactor(supervisor): isolate state per sandbox instance (#4701)
* refactor(ocsf): attribute events to session-bound contexts A supervisor process that hosts several sandboxes cannot use one process-wide OCSF context: every event would carry the wrong sandbox identity, and middleware requests read their sandbox identity from the same context. Builders now accept a ContextRef, either a borrowed context or a context shared by a hosted session, so per-session contexts need not be leaked. ctx() returns the context bound to the calling thread when present and falls back to the process-wide context otherwise. OCSF JSON output settings can likewise be bound per thread, and OpenTelemetry can parent a span from an explicit W3C trace context instead of process environment. A dedicated supervisor keeps using the process-wide context. Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(supervisor): scope gateway credentials and network state to a session The supervisor kept its gateway bearer, Sandbox Protocol bearer, and renewal task in process-wide slots, and the network supervisor read provider token grants, proxy CA files, and the gateway identity from process environment. A second sandbox in the same process would have replaced or rotated the first sandbox's credentials, reused its provider access tokens, and exposed its denials through policy.local. GatewayCredentials now holds one session's gateway and Sandbox Protocol credentials and their renewal; every gateway client receives a GatewayEndpoint that pairs the URL with them, including log push, ConnectSupervisor and its redirects, and extension-credential rotation. run_networking takes a NetworkSession with the sandbox identity, the session's gateway access, its Workload API socket, its policy.local denial source, and its proxy CA files. Provider token grants use a cache and HTTP client owned by the session and fail closed without a Workload API socket. run_session supervises one generation from an explicit SessionConfig and owner-provided SessionControls. run_sandbox_with_backend remains the dedicated entry point: it keeps the process-wide OCSF context, reads the same environment, binds readiness endpoints to session readiness, and stops on SIGTERM or SIGINT. The binary shares one set of credentials between log push and supervision, as before. Signed-off-by: Drew Newberry <anewberry@nvidia.com> * feat(supervisor): host concurrent sessions on session-bound runtimes SessionHost runs several supervisor sessions in one process. Each session gets its own Tokio runtime, and every thread that runtime starts, plus the thread driving it, is bound to the session's OCSF context, OCSF JSON settings, log sink, and denial log. Tasks, blocking work, reconnects, and credential renewal spawned by a session therefore keep its identity, while threads outside any session carry none and push no logs. Gateway credentials stay explicit per session. Removing a session stops its workload gracefully, or cancels it at once while it is still starting, then shuts down only its runtime and revokes its gateway credentials. The key stays registered until the session thread has finished, so the same generation cannot gain a second owner while one is stopping. A panicking session reports a failure, clears its readiness, and is revoked the same way. Hosted sessions get a private proxy trust directory and serve policy.local denials from their own log. A focused test runs two sessions concurrently and checks that nested and blocking tasks see their own identity, gateway requests carry their own bearer, log lines reach only their own stream, and removing one leaves the other supervising. Closes #4699 Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(supervisor): name hosted lifetimes sandbox instances A sandbox instance is one running generation of a stable Sandbox resource. Gateway transport connections and the gateway supervisor sessions they carry can reconnect within one instance, so the hosted object, its configuration, controls, host, and per-instance network, denial, and token-grant state now use the sandbox instance name. Rename session_host to sandbox_instance_host, run_session to run_sandbox_instance, and SessionConfig, SessionControls, SessionHost, and related hosted types to their SandboxInstance equivalents. Gateway supervisor sessions, SSH sessions, and authentication session types keep their names. No wire, configuration, or lifecycle behavior changes. Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(supervisor): launch sandbox instances from one config and prepared bindings Replace SandboxRunConfig and its field-by-field copy into SandboxInstanceConfig with a single launch configuration. Live per-instance state (gateway credentials, OCSF context and JSON settings, log sink, and denial log) is now an explicit SandboxInstanceBindings value that the owner prepares once, instead of placeholders the runner or host overwrote. The dedicated binary loads launch inputs from its arguments and environment and shares one GatewayCredentials with log push; run_sandbox_with_backend keeps the process-wide OCSF context, health endpoints, and signal handling. The instance host binds the same SandboxInstanceBindings to its runtime threads and revokes their credentials when the instance ends. Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(supervisor): update Windows host proxy test config Signed-off-by: Drew Newberry <anewberry@nvidia.com> --------- Signed-off-by: Drew Newberry <anewberry@nvidia.com> |
||
|
Balaji Ganesan
|
9152b6e4a1 |
fix(vm): boot a digest-pinned image from the cache without a registry call (#4582)
A registry-sourced sandbox boot authenticated with the registry, and for a tag resolved the manifest, before it looked at the root disk cache. It did so on every boot, even when the image reference carries a digest and the root disk for that digest is already cached. Each boot then waits on the registry, and a registry outage stops every boot of an image the host already has. A digest names the image's content, and since #4342 a cached root disk is published only after its manifest was verified against that digest. On a cache hit for a digest-pinned reference the registry cannot add anything, so return the hit before authenticating. Authentication and the pull stay on the miss path, and a tag still resolves through the registry. The hit emits the same Pulling, CacheHit and Pulled events as before. Behaviour change: a host whose registry credentials were revoked still boots digest-pinned images it has already cached, as a kubelet does with IfNotPresent and a digest-pinned image. Test: bootstrap_registry_cache_hit_for_a_digest_pin_makes_no_registry_request covers a miss (authenticates and fetches), a cached digest pin (no request at all) and a tag for the same content (still resolved). It fails without this change with ["GET /v2/"]. Closes #4581 Signed-off-by: Balaji Ganesan <bganesan@nvidia.com> |
||
|
Mrunal Patel
|
0a51652308 |
chore(server): remove unused server macros crate (#4818)
Signed-off-by: Mrunal Patel <mrunalp@gmail.com> |
||
|
Taylor Mutch
|
029fda684d |
refactor(cli): use Rust SDK transport (#4817)
Signed-off-by: Taylor Mutch <taylormutch@gmail.com> |
||
|
Yuedong Wu
|
20cb306d3a |
perf(docker): slow steady-state supervisor healthcheck (#4802)
Signed-off-by: Yuedong Wu <dwcn22@outlook.com> |
||
|
krishicks
|
4b7486e73d |
test(auth): share fixtures for gateway audit event tests (#4778)
Tests for the gateway's OCSF events each carried their own copy of the same fixtures: converting captured events to JSON, the request ID and peer address, running a handler inside a routed request, and checking event shapes. They now share them: - OcsfCapture::json_events returns captured events as their exported JSON. - auth::audit::test_support holds the test request origin, audited() for running a handler as a routed request, and assertions for authentication failures, logons, denials, boundary findings, the request origin, and redacted secrets. The event assertions also check each event against its OCSF class schema, which the handler tests did not do before. - auth::principal::test_support builds the OIDC user and gateway-JWT sandbox principals that several test modules defined identically. Refs #2912 Signed-off-by: Kris Hicks <khicks@nvidia.com> |
||
|
krishicks
|
19ae2b7ed7 |
test(cli): retry forward tests when ssh loses the port after the check (#4799)
Forward tests retried on a new port only when the CLI's port check found the port in use. Another test can also take the port after that check passes and before the fake ssh binds it. The fake then panicked with exit status 101, which the retry did not recognize, so the test failed. The fake ssh now exits 255 when its port is taken, as ssh -o ExitOnForwardFailure=yes does, and the tests retry on that exit status too. Signed-off-by: Kris Hicks <khicks@nvidia.com> |
||
|
krishicks
|
a4c27f9ca9 |
refactor(auth): share audit helpers for gateway denials and rejections (#4777)
Handlers recorded each refusal by emitting an OCSF event and then building the status by hand, so a new refusal could skip the event. These helpers do both in one call: - audit::deny emits an authorization denial for the current request and returns the permission-denied status. Guards, workspace authorization, and the token and current-user handlers use it. - audit::reject_credential emits an Authentication failure for a credential a handler verifies itself and returns the rejection's status. It replaces emit_credential_failure for SSH session tokens and IssueSandboxToken. RefreshSandboxToken now takes its path and origin from the request scope, as every other handler does, instead of rebuilding them from the request. - Failure and logon events share one Authentication builder. - AuthMechanism and AuthFailureReason carry their audit labels. The logon digest uses the label instead of the variant's Debug name. Events and statuses are unchanged. Refs #2912 Signed-off-by: Kris Hicks <khicks@nvidia.com> |
||
|
krishicks
|
321b93a898 |
feat(auth): emit OCSF events for SSH session and sandbox token checks (#4762)
Some credentials are checked by handlers rather than by the gateway's authenticators, so their rejections only reached the caller and plain logs. They now emit the same OCSF events as the router's checks: - SSH session tokens. A ForwardTcp SSH connection whose token is missing, unknown, revoked, minted for another sandbox, or expired emits an Authentication failure with protocol ssh_session_token and a reason category. An accepted token logs on once, like other credentials. The router has already authenticated the caller, so these events name that caller. - IssueSandboxToken. A compute-driver identity that no longer matches the sandbox's recorded runtime emits an Authentication failure (stale_identity); a sandbox with no record, or no usable recorded identity, emits unknown_identity. A sandbox calling it with a gateway JWT is refused with method_not_permitted and raises the sandbox boundary finding. - RefreshSandboxToken. A caller that is not a gateway-JWT sandbox is refused with method_not_permitted. A request for an extension service the sandbox policy does not select is refused with the new extension_service_not_selected reason, without a finding, because a policy update can race the supervisor's request. - Sandbox logons on IssueSandboxToken and RefreshSandboxToken. The router logged a compute-driver credential on IssueSandboxToken, or a gateway JWT on RefreshSandboxToken, before the handler finished checking the recorded runtime or the token lineage, so a credential that failed that check recorded a success and then a failure. Those handlers now record the logon once their check passes. Other credentials on these methods are fully verified by the router and still log on there, even when authorization then refuses them. - GetCurrentUser. A non-user principal is refused with method_not_permitted. The router already blocks such callers, so this is defense in depth. Callers receive the same status codes and messages as before, and the events never include the tokens. Refs #2912 Signed-off-by: Kris Hicks <khicks@nvidia.com> |
||
|
Drew Newberry
|
72ab4dd9ef |
feat(kubernetes): deploy shared supervisor mode (#4695)
* feat(kubernetes): bootstrap experimental gateway supervisor topology Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs: move the experimental supervisor topology under an Experimental section Signed-off-by: Drew Newberry <anewberry@nvidia.com> * feat(supervisor): report experimental bootstrap health over TCP experimental-bootstrap now takes --health-port instead of a Unix health socket. It binds the dedicated supervisor's wildcard readiness listener, reachable at the Pod IP, and accepts and closes connections while the process runs. The chart probes the named container port health (5502) with native kubelet tcpSocket checks and drops the health-only emptyDir. No filesystem state remains to clean up after an unclean exit. Dedicated supervisors keep --health-socket-path and health --socket. Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs: defer public experimental supervisor docs Remove the experimental supervisor topology additions from the published gateway configuration page and the public debug-openshell-cluster skill. Public documentation will be added once the experimental stack settles. Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(supervisor): replace bootstrap with shared host stub Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(supervisor): name shared supervisor subcommand shared Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(supervisor): select shared mode with explicit endpoint flags Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(supervisor): name the shared host module after its mode Rename the shared-mode host module from experimental to shared to match the --mode=shared flag. No behavior changes. Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(kubernetes): drop the experimental prefix from the supervisor topology Rename the Helm template to supervisor.yaml, its Kubernetes names and labels to the -supervisor suffix and supervisor component, and the driver setting to supervisor_topology. The setting remains opt-in and workspace remains reserved. Signed-off-by: Drew Newberry <anewberry@nvidia.com> --------- Signed-off-by: Drew Newberry <anewberry@nvidia.com> |
||
|
Shiju
|
465324a831 |
fix(vm): populate an empty guest /etc/hosts with loopback names (#4182)
OCI images commonly ship an empty /etc/hosts and rely on the container runtime to bind-mount a generated one. The VM driver does not, and guest init wrote /etc/hostname and /etc/resolv.conf but never /etc/hosts, so localhost did not resolve: the guest resolver is the loopback relay, which has no record for it. Add configure_hosts to guest init, called after configure_hostname. It writes IPv4 and IPv6 localhost entries when the file is missing or empty, keeps a non-empty file and leaves a symlink alone, because an absolute link target would resolve against init's own root. A write failure logs a warning and boot continues. Cover the file states and the boot-path call site with tests that run the shipped init function under bash. Closes #3934 Signed-off-by: Shiju <shiju@nvidia.com> |
||
|
krishicks
|
915a5d1553 |
feat(auth): emit OCSF logon events for accepted gateway credentials (#4761)
When a gateway replica accepts a credential for the first time, it now emits an Informational OCSF Authentication event with status Success. The caller's request proceeds exactly as before. The event records: - Who: the principal (user, mTLS operator, sandbox, or gateway peer), with the same user types as authorization denials. - How: the authentication protocol (OpenID, mtls, sandbox_jwt, compute_driver, or k8s_service_account). - Where: the gRPC method of the request that logged on. - Correlation: the caller's address and request ID, the same fields the failure and denial events carry. Every gRPC request authenticates, and an idle sandbox's supervisor polls the gateway several times a minute, so an event per request would bury user logons. Each credential therefore logs on once: later requests with the same bearer token or client certificate emit nothing until the credential has been in use for an hour, when it logs on again. The replica's connections share this record, which holds up to 16,384 credentials and forgets the oldest early when a new one arrives while full. It keys on a SHA-256 digest of the credential that authenticated the caller (the bearer token or the client certificate's fingerprint), so headers the authenticator ignored cannot force new logons, and a renewed certificate for the same subject logs on again. Digests stay in memory. Requests admitted by allow_unauthenticated_users, or by a gateway with no authentication configured, emit no success event, since no credential was verified. Refs #2912 Signed-off-by: Kris Hicks <khicks@nvidia.com> |
||
|
Eric Curtin
|
ea265a2f9a |
docs(cli): say --upload cannot take a command (#4502)
Signed-off-by: Eric Curtin <eric.curtin@docker.com> |
||
|
Drew Newberry
|
32f026442e |
feat(sandbox): initiate supervisor connections across compute drivers (#4656)
* feat(sandbox): support sandbox-initiated supervisor connections Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(sandbox): consolidate control transport setup Signed-off-by: Drew Newberry <anewberry@nvidia.com> * test(sandbox): remove prototype-specific unit tests Signed-off-by: Drew Newberry <anewberry@nvidia.com> * feat(kubernetes): migrate legacy control flow on restart Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs: remove prototype changes from published docs Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(sandbox): adapt reverse control to published backend protocol Signed-off-by: Drew Newberry <anewberry@nvidia.com> * feat(sandbox): initiate supervisor connections across compute drivers Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(docker): stage reverse control socket for the supervisor identity Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(sandbox): keep runtime transports behind the backend Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(docker): mount the runtime control volume read-only The sandbox runtime only dials the supervisor-owned control socket, so it does not need write access to the control volume. Mount it read-only in the runtime container and stop staging the control directory through the runtime bundle; the supervisor stager already initializes it through its own writable mount. Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(kubernetes): separate legacy and runtime-initiated control services Keep os-boundary selecting the workload and add os-supervisor for runtime-initiated connections. Bind the mode marker to its runtime generation so a restart by an older gateway does not inherit stale mode metadata. Preserve the legacy Service across creation and restart, validate the active endpoint, and drop the Service patch permission. Signed-off-by: Drew Newberry <anewberry@nvidia.com> --------- Signed-off-by: Drew Newberry <anewberry@nvidia.com> |
||
|
krishicks
|
a24340beba |
feat(auth): emit OCSF events for gateway authorization denials (#4728)
When the gateway refuses a caller that has already authenticated, it now emits an OCSF API Activity event. The caller gets exactly the same error as before. The event records: - Who: the principal (user, mTLS operator, sandbox, or gateway peer). - What: the gRPC method, plus the sandbox or workspace that was refused when there is one. - Why: a reason such as cross_sandbox, missing_role or not_workspace_member. - Correlation: the caller's address and request ID, the same fields the authentication events carry. That covers the denials made centrally by the router and the shared guards, plus the checks inside handlers that have no shared guard: a sandbox naming another sandbox (in either sandbox-name resolver), a non-user caller refused by mutation admission, a sandbox sending a config update it isn't allowed to, and a non-admin granting the workspace admin role. Denials raised in the mutation admission worker, which runs in a spawned task, keep the request's context. When a sandbox credential is refused for calling a method it may not call, touching another sandbox, or acting across workspaces, the gateway also emits a High-severity Detection Finding. It carries the same correlation_uid as the denial event. A real supervisor never makes these requests, so a finding means the credential is probably being misused. Two cases are recorded without a finding: a method the gateway doesn't recognize (unknown_method), so a newer supervisor doesn't look like a compromised one, and a supervisor whose own sandbox was deleted mid-request, which is refused but isn't misuse. The shorthand line names the refused sandbox or workspace, for example sandbox:<id>, so operators reading console logs can see the target without JSONL or OTLP export. To support this, the OCSF crate's Actor can now hold a user instead of a process, and the API Activity builder gained the fields gateway events need. Supervisor inference events come out unchanged. Gateway events declare the container profile only when they concern a sandbox, as MXC findings do. The OCSF 1.8.0 resource_details schema is vendored so the refused sandbox or workspace goes in the standard resources field. Refs #2912 Signed-off-by: Kris Hicks <khicks@nvidia.com> |
||
|
Simon Scatton
|
30492bc580 |
fix(deps): update russh to 0.64.1 (#4742)
Signed-off-by: Simon Scatton <sscatton@nvidia.com> |
||
|
ShijuandJohn Myers
|
292c9c8d48 |
fix(vm): flush guest filesystem before a planned sandbox stop (#4195)
* fix(vm): flush guest filesystem before a planned sandbox stop A planned stop of a MicroVM sandbox ended the VM before the guest wrote its dirty pages to disk, so files written shortly before the stop were missing after the next start. The driver closed the host supervisor's parent-liveness pipe before it sent SIGTERM. The supervisor exits at once on that end-of-file, so it never ran its orderly shutdown: terminate the workload, then ask the guest to terminate. Signal the supervisor first, keep the pipe open until it has exited, and give it a grace period derived from the supervisor's own bounds and kept below the gateway's reclaim deadline. In the guest runtime, syncfs the root filesystem after the process tree is terminated and before the termination is acknowledged, bounded to 3 seconds and best effort. Only the VM backend, which uses a vsock boundary listener, flushes; the container drivers share this runtime and are unchanged. A sandbox whose main process has already exited is not covered. Closes #4185 Signed-off-by: Shiju <shiju@nvidia.com> * fix(vm): warn about unconfirmed flush after supervisor kill Preserve the supervisor exit status so a planned stop warns when a signal killed the supervisor and guest writeback was not confirmed. Keep normal workload exit codes quiet, including 143 after a handled SIGTERM. Extend the existing process tests to check the warning after a forced kill and its absence after an orderly shutdown. Refs #4185 Signed-off-by: Shiju <shiju@nvidia.com> * fix(vm): satisfy shutdown timeout control-flow lint Use if let for the single successful timeout pattern while preserving the supervisor exit status and forced-kill error handling. Signed-off-by: Shiju <shiju@nvidia.com> * fix(vm): preserve shutdown merge resolutions during rebase Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * docs(vm): explain planned stop durability and flush limits Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> --------- Signed-off-by: Shiju <shiju@nvidia.com> Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> Co-authored-by: John Myers <9696606+johntmyers@users.noreply.github.com> |
||
|
Eric Curtin
|
d2a35084cd |
fix(server): reject reserved annotation keys (#4626)
Signed-off-by: Eric Curtin <eric.curtin@docker.com> |
||
|
Eric Curtin
|
2798108b3f |
fix(ocsf): report https scheme for TLS-terminated requests (#4684)
* fix(ocsf): report https scheme for TLS-terminated requests Closes #4562 Signed-off-by: Eric Curtin <eric.curtin@docker.com> * fix(ocsf): use transport scheme on passthrough relay Signed-off-by: Eric Curtin <eric.curtin@docker.com> --------- Signed-off-by: Eric Curtin <eric.curtin@docker.com> |
||
|
Eric Curtin
|
b01dd62572 |
fix(server): warn on dangling provider refs at startup (#4661)
* fix(server): warn on dangling provider refs at startup Signed-off-by: Eric Curtin <eric.curtin@docker.com> * fix(server): load providers concurrently at startup Signed-off-by: Eric Curtin <eric.curtin@docker.com> --------- Signed-off-by: Eric Curtin <eric.curtin@docker.com> |
||
|
Varsha
|
1cb9ef8ab4 |
refactor(crypto): introduce a generic interface for crypto libraries and refactor AWS-LC (#3351)
* refactor(crypto): centralize AWS-LC behind backend interface Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com> * refactor(crypto): route new host proxy tests through facade Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com> * fix(crypto): make backend boundary check portable Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com> * fix(crypto): route Google JWT signing through facade and verify context dispatch Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com> * refactor(crypto): propagate digest backend failures Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com> * refactor(crypto): route Vault TLS fixtures through facade Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com> * refactor(crypto): adapt isolation runtime callers after main rebase Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com> * fix(crypto): declare sandbox dependency and remove boundary lint Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com> * refactor(crypto): support independent backends and backend-free CA import Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com> * chore(crypto): refresh feature suite lockfile after rebase Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com> * fix(crypto): route runtime integrity and entropy through backend Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com> * chore(crypto): refresh lockfile after upstream rebase Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com> * fix(crypto): route VM socket entropy through backend after rebase Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com> * docs(crypto): document architecture and contributor boundaries Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com> * test(crypto): adapt upstream fixtures to backend facade Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com> * test(crypto): route relay TLS fixture through facade Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com> * test(crypto): adapt service bearer TLS fixture after rebase Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com> * fix(crypto): route provider profile TLS initialization through facade Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com> * fix(crypto): adapt upstream VM and supervisor paths to facade Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com> * ci(crypto): enforce backend independence in branch checks Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com> * test(crypto): adapt upstream token grant TLS fixture Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com> * fix(crypto): avoid format collect in executable hashing Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com> * fix(crypto): address provider selection and decryption review Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com> * fix(crypto): adapt upstream operator TLS paths after rebase Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com> * fix(crypto): adapt upstream TLS fixtures and protocol tests Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com> --------- Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com> |
||
|
Eric Curtin
|
b58e24c9da |
fix(supervisor): report plaintext scheme to middleware (#4397)
Closes #4253 Signed-off-by: Eric Curtin <eric.curtin@docker.com> |
||
|
krishicks
|
fdfbf84f8b |
fix(ocsf): report the required device.os.type_id (#4677)
OCSF 1.8 requires type_id on the os object, but device.os carried only a name. Events now report Linux (200), Windows (100), macOS (300), or Other (99) with the platform name as the type label. The os object schema is vendored so tests validate it. Refs #2912 Signed-off-by: Kris Hicks <khicks@nvidia.com> |
||
|
krishicks
|
f41f24b549 |
feat(auth): record peer address and request ID on authentication events (#4673)
Authentication events now carry the connection's immediate peer as src_endpoint, or edge-tunnel for WebSocket-tunneled requests, and the request's x-request-id as metadata.correlation_uid when it is a UUID. Refs #2912 Signed-off-by: Kris Hicks <khicks@nvidia.com> |
||
|
Piotr Mlocekanddivesh
|
604b9f66f6 |
feat(config): deliver gateway configuration over the supervisor session (#4320)
Supervisors poll the gateway for configuration every 10 seconds. This PR adds an opt-in push mode (config_delivery_mode = "push"): supervisors still start by polling, and once their ConnectSupervisor session is accepted, the gateway delivers every later change over it and supervisors apply it and report the result. poll stays the default, and older supervisors and gateways keep polling. --------- Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> Signed-off-by: divesh <dgude@nvidia.com> Co-authored-by: divesh <dgude@nvidia.com> |
||
|
krishicks
|
35a3390dbc |
feat(auth): classify rejected gateway credentials in authentication events (#4655)
Authenticators now return the mechanism and a reason category with the status they reject a request with, and the gateway router records both in the Authentication event. Callers still receive the original status. Refs #2912 Signed-off-by: Kris Hicks <khicks@nvidia.com> |
||
|
Drew Newberry
|
77929d9c14 |
feat(sandbox): publish and activate sandbox backend protocol (#3898)
* feat(isolation): publish backend protocol contracts independently of runtime Signed-off-by: Drew Newberry <anewberry@nvidia.com> * feat(isolation): expose trusted launch and registry data contracts Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(isolation): name the required capability sandbox Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(sandbox): align protocol and compute capability names Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(sandbox): finalize public protocol contract names Signed-off-by: Drew Newberry <anewberry@nvidia.com> * feat(isolation)!: activate backend contracts in sandbox runtime and driver selection Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(sandbox): simplify runtime and fix backend example Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(examples): organize WASI backend in Rust Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(sandbox): defer protocol conformance fixtures Signed-off-by: Drew Newberry <anewberry@nvidia.com> * feat(examples): add supervised V8 backend demo Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(examples): stream V8 guest output with default policy Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(examples): handle V8 rejections and roll back failed launches Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(examples): align V8 crypto dependencies with CI AWS-LC Signed-off-by: Drew Newberry <anewberry@nvidia.com> * feat(sandbox): represent Sandbox Protocol operations in protobuf Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(isolation): replace backend traits with RPC clients Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(proto): remove legacy isolation protocol reservations Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(proto): clarify Sandbox Protocol names Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs(isolation): clarify authentication bootstrap and supervisor APIs Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(isolation): simplify policy enforcement confirmation Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(sandbox): rotate launch generations and stabilize retry sessions Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(sandbox): remove redundant backend adapters and TCP flags Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(sandbox): repair Docker restarts and macOS lint Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(isolation): simplify RPC contracts and backend registration Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(sandbox): deliver TCP denials before closing the RPC Signed-off-by: Drew Newberry <anewberry@nvidia.com> --------- Signed-off-by: Drew Newberry <anewberry@nvidia.com> |
||
|
Matthew Grossman
|
633cc6dee6 |
perf(server): drop async_trait from mutation replay and supervisor gateway traits (#4619)
* perf(server): drop async_trait from mutation replay Mutation is never used as a trait object, so the boxed futures from #[tonic::async_trait] buy nothing. Declare its methods as -> impl Future + Send instead. This keeps the Send guarantee, removes a heap allocation per call, and cuts a cold openshell-server lib build from ~60s to ~36s. Box the approve_all_draft_chunks future at its call sites, since it is no longer boxed implicitly and trips clippy::large_futures. Closes #4618 Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * perf(supervisor): drop async_trait from generic-only gateway traits StartupGateway, PolicyGatewayClient, and ReportClient are only used through generic parameters, never as trait objects. Declare their methods as -> impl Future + Send instead of boxing each future. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> --------- Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> |
||
|
Eric CurtinandJohn Myers
|
8fcba288cf |
feat(cli): add --annotation to sandbox create (#4325)
* feat(cli): add --annotation to sandbox create Signed-off-by: Eric Curtin <eric.curtin@docker.com> * fix(cli): drop reserved annotation check Signed-off-by: Eric Curtin <eric.curtin@docker.com> * test(cli): adapt annotation tests to env helper Signed-off-by: Eric Curtin <eric.curtin@docker.com> --------- Signed-off-by: Eric Curtin <eric.curtin@docker.com> Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> Co-authored-by: John Myers <9696606+johntmyers@users.noreply.github.com> |
||
|
Emilien Macchi
|
b8526eb1d9 |
fix(server): install the metrics recorder before startup reconciliation (#4524)
The endpoint-status startup sweep ran before the gateway installed its Prometheus recorder, so the lock waits, timeouts, errors and hold times it recorded went to the no-op recorder and only its warnings reached the log. The sweep retries lock timeouts for up to three lock waits before startup fails, and a slow sweep that still succeeds is what operators want to see in the lock metrics. When metrics are enabled, the recorder is now installed before startup reconciliation, and the metrics listener still binds with the other listeners. A new pod's first scrape therefore already holds the sweep's samples, so they show in the raw counter and histogram count values, not in rate() or increase(). A sweep that fails still exits before any listener binds, so its timeouts remain log-only. The startup lock-timeout test now waits for the sweep's timeout counter instead of a fixed sleep and checks its timeout and wait samples, and the debug skill says which startup timeouts reach the metrics and how to read them. Part of #4307 Closes #4523 Signed-off-by: Emilien Macchi <emacchi@redhat.com> |
||
|
krishicks
|
9c6867ccec |
feat(auth): emit OCSF events for credential-less gateway requests (#4632)
Requests the gateway rejects before consulting any authenticator now emit an OCSF Authentication [3002] event: no credential presented, mTLS required without a client certificate, or an operator certificate combined with an authorization header. Each event carries a reason category in status_detail and never the presented credential. The response returned to the caller is unchanged. openshell-ocsf gains the Authentication class with user and service objects, and a shared OcsfCapture test layer. Schema validation now rejects attributes the class does not define and profile attributes whose profile the event does not declare. Refs #2912 Signed-off-by: Kris Hicks <khicks@nvidia.com> |
||
|
Florent BENOIT
|
a4c848adbd |
feat(vm): add host bind mount support via virtiofs (libkrun) (#3463)
* feat(vm): add host bind mount support via virtiofs (libkrun) Introduce host directory bind mounts for libkrun VM sandboxes using the virtiofs4 API with simplified permission semantics. The gateway accepts `enable_bind_mounts` in the `[openshell.drivers.vm]` TOML table and forwards it to the VM driver subprocess. Mounts are specified per-sandbox in driver_config.mounts[] with source, target, optional read_only (default: read-only), and an optional `type` that must be `bind`, so Docker and Podman bind entries (without `selinux_label`) parse unchanged. At sandbox validation the driver rejects mounts unless bind mounts are enabled and resource admission is disabled. It rejects GPU and other QEMU launches, sources with control characters, and more mounts than libkrun has interrupt lines for (32 on aarch64, 3 on x86_64), and checks targets against reserved VM-internal directories and each other (no nested targets). The QEMU check runs again once the launch backend is resolved. Host source directories are checked at provisioning, before image preparation, so a briefly missing source fails provisioning visibly instead of denying recovery of a persisted sandbox. Missing, inaccessible, symlinked, non-directory, and unreadable sources are rejected: libkrun does not follow a final symlink, and on macOS it panics during guest boot when it cannot open the share root. The driver generates virtiofs tags and always writes a guest-side manifest into the overlay disk, together with the init drop-ins, so an image-supplied manifest can never drive guest mounts. The launcher parses its tab-separated --vm-mount arguments from the right, so a source can never shift the tag or mode, and the driver logs each shared directory at launch. Guest init mounts the shares after every /sandbox ownership fixup and the init drop-ins, so recursive chowns never reach host files. It refuses targets that traverse symlinks in the image, hands newly created mount point parents under /sandbox to the sandbox user, and skips an empty manifest. krun_add_virtiofs4 is resolved optionally, so older libkrun runtimes still boot sandboxes without mounts. The kernel config lists CONFIG_FUSE_FS and CONFIG_VIRTIO_FS explicitly; the pinned libkrunfw base config already enables both. A vm_bind_mount e2e test checks a read-write share round-trips data with the host, accepts a new file, directory, and nested file that the host then sees, and that a read-only share rejects writes. libkrun on rootless Linux refuses file creation by a guest user whose UID/GID differ from its own, so the VM e2e gateway runs the sandbox as the gateway user there. It enables bind mounts like the Docker and Podman e2e gateways. The runtimes docs cover the mount limit, the symlink restriction, the guest-kernel exposure, and the rootless Linux UID/GID note. The debug-openshell-cluster skill lists VM bind mount checks. Signed-off-by: Florent Benoit <fbenoit@redhat.com> * fix(nix): refresh vm-runtime hashes to the libkrun v1.19.4 runtime The Nix vm-runtime package fetches the rolling vm-runtime release with fixed hashes pinned on 2026-09-16. The release was rebuilt on 2026-09-21 after the libkrun v1.19.4 / libkrunfw v5.6.1 bump (#3451), but the hashes were not updated, so CI kept substituting the 2026-09-15 runtime (libkrun v1.17.4) from the binary cache. That libkrun lacks krun_add_virtiofs4, which VM bind mounts need. Point the hashes at the current release assets. Signed-off-by: Florent Benoit <fbenoit@redhat.com> --------- Signed-off-by: Florent Benoit <fbenoit@redhat.com> |
||
|
Drew Newberry
|
b2069a267c |
feat(docker): support host and bridge supervisor networking (#4016)
* feat(docker): select supervisor networking for ECI workloads Signed-off-by: Drew Newberry <anewberry@nvidia.com> * test(docker): use automatic networking in e2e Signed-off-by: Drew Newberry <anewberry@nvidia.com> * test(docker): rely on existing e2e networking coverage Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs(docker): shorten supervisor networking configuration Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(docker): let local gateway discover supervisor endpoint Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(docker): make supervisor networking operator-selected Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(docker): resolve host gateway in the driver Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs(docker): simplify host networking guidance Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs(docker): trim configuration guidance Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(docker): discover host gateway once at driver startup Signed-off-by: Drew Newberry <anewberry@nvidia.com> --------- Signed-off-by: Drew Newberry <anewberry@nvidia.com> |
||
|
Polite_realism
|
10241ab47b |
fix(podman): trust an operator-supplied additional CA for direct egress (#3996)
* fix(podman): trust an operator-supplied additional CA for direct egress Since v0.1.2 the Podman supervisor runs as its own container, separate from the workload image, so it can no longer inherit a private CA baked into the workload's system trust store. Policy-allowed, inspected HTTPS requests to destinations signed by that CA fail upstream TLS establishment, and the only workaround is rebuilding the supervisor image with the CA baked in. Add a Podman-only additional_ca_bundle setting, independent of the existing proxy_ca_bundle (which requires https_proxy), that bind-mounts an operator PEM into the supervisor container and folds it into the upstream trust store alongside the system CA bundle. Validated on the gateway host at sandbox-create time, matching the existing corporate proxy CA bundle's validation, so a misconfigured path or certificate-free bundle fails closed immediately rather than producing a failed sandbox once the supervisor container starts. Scoped to Podman only, reusing the existing additive-bundle mechanism without new gateway.toml schema, proto fields, or cross-driver plumbing -- a narrower alternative to the in-flight cross-driver implementation in #3292, which remains the intended long-term direction. Fixes #3781. Signed-off-by: politerealism <burdcat17@gmail.com> * fix(podman): keep additional CA mount on the supervisor build_isolation_specs() keeps only trusted_mount() destinations on the supervisor, and the additional CA path was not in that list. The bind mount stayed on the workload while the supervisor received --additional-ca-bundle for a file it did not have, so startup failed. Add the path to trusted_mount() and cover the final supervisor and workload specs with a test. Correct the docs and a code comment to say the bundle is validated when the driver starts, and add Podman troubleshooting guidance to the debug-openshell-cluster skill. Signed-off-by: politerealism <burdcat17@gmail.com> --------- Signed-off-by: politerealism <burdcat17@gmail.com> |
||
|
ShijuandJohn Myers
|
29a2b8e7d9 |
feat(providers): report bounded token-grant failures (#4427)
* feat(providers): report bounded token-grant failures Signed-off-by: Shiju <shiju@nvidia.com> * test(providers): align policy metadata and TLS lockfile Signed-off-by: Shiju <shiju@nvidia.com> * test(providers): fix trusted grant recovery fixture Signed-off-by: Shiju <shiju@nvidia.com> * test(providers): enable denial API in recovery fixture Signed-off-by: Shiju <shiju@nvidia.com> * refactor(providers): simplify token-grant failure reporting Remove duplication and dead paths from the token-grant failure reporting without changing classification, the workload-visible 502 response, or the wire format. - Build the gRPC failure envelope with tonic-types, store the typed failure in EndpointFailure, and share one request-ID rule and one endpoint-status authority check. - Merge the gateway exchange client wrapper, drop forwarding validators, and name the request field when a reported cause is rejected. - Inject fallback credentials only from live provider state and delete the unread per-connection credential copy. Keep one 502 writer per path and restore the base token-cache handling. - Check the token endpoint after the cache lookup, still before any credential acquisition. - Document 429 and 5xx classification as implemented, and drop tests that repeat retained assertions through the same code path. Signed-off-by: Shiju <shiju@nvidia.com> * refactor(providers): trim token-grant failure tests and supervisor API Cut the token-grant failure tests to one test per distinct behavior across core, gateway, supervisor, CLI and Go, and replace the 794-line closure E2E with status assertions in the existing route-recovery E2E: the refused grant appears in sandbox status with the response's request ID, code, stage and upstream status, and the matching grant's later success clears it. - Pin the live provider snapshot with one helper used by inspected and fallback forwarding, and inject selector-matched grants through inject_for_selectors, which borrows the pinned context. Move the test-only wrapper into test_support. - Drop the 35-second wrappers around the gateway grant RPC. The shared channel bounds the connection attempt and the gateway bounds its token-service calls. - Log the token service's OAuth error and error_description, without control characters and limited to 256 characters each, with the request ID in the gateway or supervisor log. Status and the workload response still carry only the bounded cause. - Keep one check that an untrusted token-service certificate cannot assert a cause. Signed-off-by: Shiju <shiju@nvidia.com> * fix(providers): restore test import and features lockfile The injection tests named EndpointObservationSender through the parent module, which no longer imports it. The features suite lockfile kept the tonic TLS dependency that the removed E2E fixture added. Signed-off-by: Shiju <shiju@nvidia.com> * test(providers): fix body-bound fixture width and table type Pass the response-size width as a named argument; an implicitly captured constant panics at runtime. Name the mutation type in the validation table for Clippy. Signed-off-by: Shiju <shiju@nvidia.com> * test(providers): pad exact-bound body without a format width MAX_TOKEN_RESPONSE_BYTES exceeds u16::MAX, the largest format width, so the padded fixture panicked. Pad with repeat_n as before. Signed-off-by: Shiju <shiju@nvidia.com> * test(server): update public RPC schema hash after rebase Main added the ListProvidersRequest label selector. The merged public schema hash comes from the Branch Checks storage_proto test output. Signed-off-by: Shiju <shiju@nvidia.com> * fix(providers): keep token-service error text out of grant logs A token service can echo a submitted credential in its OAuth error or error_description. Log only the classified failure code, HTTP status and request ID, and cover the canary echo in a dedicated test binary. Signed-off-by: Shiju <shiju@nvidia.com> * docs(providers): tighten token grant failure documentation Signed-off-by: Shiju <shiju@nvidia.com> * docs(providers): condense token grant failure guidance Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> --------- Signed-off-by: Shiju <shiju@nvidia.com> Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> Co-authored-by: John Myers <9696606+johntmyers@users.noreply.github.com> |
||
|
John T. Myers
|
01087a4515 |
fix(sandbox): distinguish descriptor lookup outcomes (#4494)
* fix(sandbox): distinguish descriptor lookup outcomes Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * fix(sandbox): preserve bad-descriptor errors in native fallback Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> --------- Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> |
||
|
alangou
|
c5519ea8ba |
fix(network): bind hostname policy to supervisor DNS (#4602)
Signed-off-by: Adrien Langou <alangou@nvidia.com> |