mirror of
https://github.com/NVIDIA/OpenShell.git
synced 2026-10-29 07:23:25 +08:00
bd2ecaf1526c9738cf4dfcb05fa6f5dacbc98b4d
291
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Drew Newberry
|
e5adf6f484 |
feat(supervisor): recover shared instances with hash-based HA (#4864)
* feat(supervisor): recover shared instances with hash-based HA Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(supervisor): keep HA coordination out of compute drivers Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(supervisor): fence pool admission and reconciliation races Signed-off-by: Drew Newberry <anewberry@nvidia.com> * feat(supervisor): keep pool mode stable across replica scaling Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(supervisor): default shared deployments to pool mode Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(supervisor): fence refused launches and remove legacy mode Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(supervisor): confirm live owners and fence stale observations Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(supervisor): preserve takeover cause log formatting Signed-off-by: Drew Newberry <anewberry@nvidia.com> * feat(helm): add memory-based supervisor autoscaling Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(supervisor): retain acknowledgements and fence streamed input Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs(supervisor): consolidate experimental multi-tenant guidance Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs(supervisor): simplify multi-tenant supervisor guide Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(supervisor): simplify pool member identity Signed-off-by: Drew Newberry <anewberry@nvidia.com> * feat(helm): make supervisor affinity configurable Signed-off-by: Drew Newberry <anewberry@nvidia.com> * feat(helm): make supervisor disruption budget optional Signed-off-by: Drew Newberry <anewberry@nvidia.com> * test(supervisor): align HA fixtures with suspend support Signed-off-by: Drew Newberry <anewberry@nvidia.com> --------- Signed-off-by: Drew Newberry <anewberry@nvidia.com> |
||
|
Simon Scatton
|
cc14afcc15 |
fix(ci): exempt Dependabot PRs from issue acceptance (#4928)
Signed-off-by: Simon Scatton <sscatton@nvidia.com> |
||
|
Simon Scatton
|
72bc3d3668 |
ci(release): infer optional workflow tags (#4707)
* ci(release): infer optional workflow tags Signed-off-by: Simon Scatton <sscatton@nvidia.com> * fix(release): use resolved tag in success notifications Signed-off-by: Simon Scatton <sscatton@nvidia.com> * fix(release): pin prerelease notes to resolved candidate Signed-off-by: Simon Scatton <sscatton@nvidia.com> --------- Signed-off-by: Simon Scatton <sscatton@nvidia.com> |
||
|
Jim Meyer
|
567d36033b |
fix(build): raise file limit for native gateway builds (#3122)
Native gateway builds exhaust the default open-file soft limit. Raise the soft limit to the hard limit (capped) in the shared build environment and apply it from the gateway task scripts. Explicitly read the current soft limit and only raise it when needed. Signed-off-by: Jim Meyer <jimeyer@nvidia.com> |
||
|
Oliver Calder
|
72e4eaa89c |
feat(snap): publish prerelease builds to latest/beta channel (#4870)
* feat(snap): publish prerelease builds to latest/beta channel Signed-off-by: Oliver Calder <oliver.calder@canonical.com> * fixup! feat(snap): publish prerelease builds to latest/beta channel Signed-off-by: Oliver Calder <oliver.calder@canonical.com> * fix(snap): use concurrency group when checking version to publish snap Signed-off-by: Oliver Calder <oliver.calder@canonical.com> --------- Signed-off-by: Oliver Calder <oliver.calder@canonical.com> |
||
|
Drew Newberry
|
9a85667d0d |
feat(kubernetes): host sandboxes on the shared supervisor (#4702)
* feat(sandbox-backend): route runtime connections to shared supervisor sessions A supervisor shared by several sandboxes needs one listener for every session's runtime-initiated Sandbox Protocol connections. The runtime now sends a fixed, bounded routing preface (magic, version, length, and an opaque route handle) before its pinned TLS handshake when its bootstrap names a route. RuntimeRouter treats the handle only as a hint that selects which generation-pinned TLS identity to demand, closes unknown, removed, and replaced handles before any handshake, bounds unauthenticated handshakes globally and per route, and queues an authenticated connection only for its own session, so no peer can displace a live channel. A routed session waits up to the bootstrap grace for its first connection, because it is registered before its workload Pod is scheduled. The session_control module defines the trusted contract between a compute driver and a shared supervisor: registering one generation's runtime descriptor and gateway-issued credentials, reading its status, and removing it, over mutually authenticated TLS 1.3. Dedicated descriptors and bootstrap listeners keep their encodings. Signed-off-by: Drew Newberry <anewberry@nvidia.com> * feat(supervisor): host gateway-registered sessions in one process openshell-supervisor experimental-host replaces the idle bootstrap process with a session host. A mutually authenticated session-control API accepts registrations only from the admission certificate holder, the gateway. Each registration starts a SessionHost session that acts only with the gateway credentials issued for its generation, keeps its proxy CA, trust directory, and SSH socket under a private directory, and receives its runtime connections through the routed listener. Status reports each session's own readiness; the health socket keeps reporting process health only. Removing a session runs on a task the host owns, so it completes when the caller disconnects, concurrent removers share its result, and the route stays reserved until the session has ended and its files are gone. An identical registration is accepted again and a different one for a registered route is refused. Sessions get no upstream proxy, extra CA bundle, or provider workload identity; the Kubernetes driver refuses configurations that need them. Signed-off-by: Drew Newberry <anewberry@nvidia.com> * feat(server): issue a TLS-only private PKI from generate-certs generate-certs --tls-only creates just a server and a client TLS Secret from a fresh CA, with the same create-if-missing contract and a recovery hint for partial state. The Helm certgen hook uses it for private PKIs that must not carry a sandbox-JWT signing key. Signed-off-by: Drew Newberry <anewberry@nvidia.com> * feat(kubernetes): supervise gateway-topology generations as shared sessions With experimental_supervisor_topology = "gateway", each new runtime generation becomes a session on the gateway's shared supervisor instead of a supervisor Pod and boundary Service of its own. The generation records its placement and session route on its Sandbox resource when it starts, and every later decision reads that record, so changing the setting never reinterprets a running generation and unknown records fail closed. The runtime identity ends with the session route, which no Pod UID can equal, so Pod authentication cannot vouch for a shared generation. Create and start add a per-sandbox egress allowance to the Pods the shared supervisor's Service selects, bind it into the boundary evidence, and register the session over session control before releasing the workload; registration does not wait for the runtime, so create still returns before the gateway persists the generation. Readiness follows the session's own status. Stop, restart, and delete remove only that session, and reconciliation suspends a workload whose session is gone and removes sessions that no current generation may run. Configurations with an upstream proxy or provider SPIFFE identity are refused. Closes #4700 Signed-off-by: Drew Newberry <anewberry@nvidia.com> * feat(helm): deploy and wire the shared supervisor With the gateway topology enabled, the chart runs the session host as a single-replica Deployment with its Service and a NetworkPolicy that admits session control only from the gateway and runtime connections only from sandbox workload Pods. The certgen hook issues a private session-control PKI whose client credential only the gateway mounts, the gateway may read only that Service, and the chart renders the matching experimental_shared_supervisor table. The supervisor Pod has no service account token. Settings the shared supervisor cannot isolate, and a hand-written connection table, fail at render time. Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(ocsf): attribute events to session-bound contexts A supervisor process that hosts several sandboxes cannot use one process-wide OCSF context: every event would carry the wrong sandbox identity, and middleware requests read their sandbox identity from the same context. Builders now accept a ContextRef, either a borrowed context or a context shared by a hosted session, so per-session contexts need not be leaked. ctx() returns the context bound to the calling thread when present and falls back to the process-wide context otherwise. OCSF JSON output settings can likewise be bound per thread, and OpenTelemetry can parent a span from an explicit W3C trace context instead of process environment. A dedicated supervisor keeps using the process-wide context. Signed-off-by: Drew Newberry <anewberry@nvidia.com> * test(e2e): run two sandboxes on one shared Kubernetes supervisor The e2e:kubernetes:shared-supervisor lane enables the gateway topology and creates two sandboxes with different policies and provider credentials. It checks that one shared supervisor process hosts both, that each session enforces its own policy and injects its own credential, that stopping one leaves the other working, that the routed listener refuses stale and unknown routes and drops a peer that names a live route without its pinned identity, and that a restart registers a fresh session on the same process. Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(supervisor): scope gateway credentials and network state to a session The supervisor kept its gateway bearer, Sandbox Protocol bearer, and renewal task in process-wide slots, and the network supervisor read provider token grants, proxy CA files, and the gateway identity from process environment. A second sandbox in the same process would have replaced or rotated the first sandbox's credentials, reused its provider access tokens, and exposed its denials through policy.local. GatewayCredentials now holds one session's gateway and Sandbox Protocol credentials and their renewal; every gateway client receives a GatewayEndpoint that pairs the URL with them, including log push, ConnectSupervisor and its redirects, and extension-credential rotation. run_networking takes a NetworkSession with the sandbox identity, the session's gateway access, its Workload API socket, its policy.local denial source, and its proxy CA files. Provider token grants use a cache and HTTP client owned by the session and fail closed without a Workload API socket. run_session supervises one generation from an explicit SessionConfig and owner-provided SessionControls. run_sandbox_with_backend remains the dedicated entry point: it keeps the process-wide OCSF context, reads the same environment, binds readiness endpoints to session readiness, and stops on SIGTERM or SIGINT. The binary shares one set of credentials between log push and supervision, as before. Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs: describe the experimental shared supervisor Document the gateway supervisor topology, its chart-owned connection settings and limits, and how to debug and test it. Signed-off-by: Drew Newberry <anewberry@nvidia.com> * feat(supervisor): host concurrent sessions on session-bound runtimes SessionHost runs several supervisor sessions in one process. Each session gets its own Tokio runtime, and every thread that runtime starts, plus the thread driving it, is bound to the session's OCSF context, OCSF JSON settings, log sink, and denial log. Tasks, blocking work, reconnects, and credential renewal spawned by a session therefore keep its identity, while threads outside any session carry none and push no logs. Gateway credentials stay explicit per session. Removing a session stops its workload gracefully, or cancels it at once while it is still starting, then shuts down only its runtime and revokes its gateway credentials. The key stays registered until the session thread has finished, so the same generation cannot gain a second owner while one is stopping. A panicking session reports a failure, clears its readiness, and is revoked the same way. Hosted sessions get a private proxy trust directory and serve policy.local denials from their own log. A focused test runs two sessions concurrently and checks that nested and blocking tasks see their own identity, gateway requests carry their own bearer, log lines reach only their own stream, and removing one leaves the other supervising. Closes #4699 Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix: tighten shared supervisor naming, removal deadlines, and route probes The shared supervisor's resource name appended its suffix before truncating, so a release fullname of 51 or 52 characters produced the gateway Service's own name. A fullname without room for the suffix now keeps a prefix and a hash of the whole fullname. Removing a session waits up to 20 s for its workload to stop and 5 s for the session runtime to shut down, but the driver gave DELETE the same 15 s budget as other session-control calls and reported normal removals as failures. The contract now defines the removal grace and a 45 s removal deadline, and deleting a sandbox no longer waits on a session whose workload is already going away. The shared host runs no supervisor OTLP trace or OCSF exporter. The driver warns at startup when the gateway configures that export, and the docs state the limit. experimental-host keeps its Unix-only imports in its Unix entry point. The e2e route probe fails when the supervisor neither refuses nor challenges a connection, or keeps a challenged connection open after a malformed TLS answer, and it checks that every probe reached the supervisor Pod. Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(supervisor): route shared supervisor crypto through openshell-crypto Main now routes first-party TLS providers and digests through the openshell-crypto facade. Build the session-control TLS configurations and the routed-runtime test server from its configuration provider, install the experimental host's default provider through it, and hash shared session registrations with its SHA-256 instead of a direct sha2 dependency. Signed-off-by: Drew Newberry <anewberry@nvidia.com> * feat(supervisor): report shared supervisor health over TCP experimental-host takes --health-port instead of a Unix health socket, matching experimental-bootstrap. It binds the wildcard health listener after the session-control and runtime listeners and accepts and closes connections while the process runs. The chart probes the named container port health (5502) with native kubelet tcpSocket checks; session control stays on 5501, the Service exposes only runtime and session control, and /run/openshell remains for per-session files. Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(supervisor): probe shared supervisor health on session control The shared supervisor's kubelet tcpSocket probes now connect to the mTLS session-control listener (5501) instead of a separate health listener. Drop the experimental-host --health-port flag, its accept loop, and the chart's 5502 container port. Probes open and close a TCP connection; every session-control request still requires mTLS. Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(supervisor): use shared mode flags for the session host Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(supervisor): describe the shared host module by its mode Drop the experimental label from the shared host module documentation so it matches the --mode=shared entry point. No behavior changes. Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(test): preserve shared supervisor workspace overlays Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(supervisor): name hosted lifetimes sandbox instances A sandbox instance is one running generation of a stable Sandbox resource. Gateway transport connections and the gateway supervisor sessions they carry can reconnect within one instance, so the hosted object, its configuration, controls, host, and per-instance network, denial, and token-grant state now use the sandbox instance name. Rename session_host to sandbox_instance_host, run_session to run_sandbox_instance, and SessionConfig, SessionControls, SessionHost, and related hosted types to their SandboxInstance equivalents. Gateway supervisor sessions, SSH sessions, and authentication session types keep their names. No wire, configuration, or lifecycle behavior changes. Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(supervisor): launch sandbox instances from one config and prepared bindings Replace SandboxRunConfig and its field-by-field copy into SandboxInstanceConfig with a single launch configuration. Live per-instance state (gateway credentials, OCSF context and JSON settings, log sink, and denial log) is now an explicit SandboxInstanceBindings value that the owner prepares once, instead of placeholders the runner or host overwrote. The dedicated binary loads launch inputs from its arguments and environment and shares one GatewayCredentials with log push; run_sandbox_with_backend keeps the process-wide OCSF context, health endpoints, and signal handling. The instance host binds the same SandboxInstanceBindings to its runtime threads and revokes their credentials when the instance ends. Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(supervisor): use session-control wire types for instance status and trace context Drop the supervisor's duplicate SandboxInstancePhase, SandboxInstanceStatus, and TraceContext families. The sandbox instance host reports SandboxInstanceStatusReport directly, with its key as the route, so the shared host no longer converts or re-keys status, lists, or removal results. SandboxInstanceConfig carries RegistrationTraceContext and the dedicated process reads it through trace_context_from_environment. No wire definitions change. Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(supervisor): update Windows host proxy test config Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(supervisor): assign shared sandbox instances over a supervisor-initiated gRPC stream Replace the private HTTP/JSON session-control API, which the gateway dialed on the shared supervisor, with a supervisor-initiated stream to the gateway. The gateway owns the desired set of sandbox instances; the shared supervisor attaches with a gateway-scoped host certificate, receives complete assignment snapshots with session-scoped revisions, reconciles its instance registry to each one, and reports what it observes. Losing the stream never stops an instance; only a later snapshot that omits it does. - Add the private openshell.supervisor.host.v1 protocol, served on the gateway listener but authorized only for the supervisor host certificate, excluded from SDK generation and reflection, and classified in the schema inventory. - Add SupervisorHostController in openshell-core: uninitialized until the driver derives the desired set once, compare-and-replace full resyncs, capacity bound, session fencing by connection epoch, revision-bounded acks, bounded sends. - Classify TLS peers per connection against the ordinary client CA and the supervisor host CA from one configuration snapshot; a chain matching neither or both is refused, and host connections never become users. - Persist each shared generation's launch material in its supervisor Secret so a restarted gateway rebuilds assignments from Sandbox resources, and withdraw instances on stop, delete, and suspend. - Issue the host identity with `generate-certs --client-only`; the gateway mounts only the CA. Drop the control port, server certificate, and gateway ingress rule from the chart; require gateway TLS for the topology. Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(supervisor): authenticate the shared supervisor stream with a gateway-minted supervisor token Replace the shared supervisor's host CA and client certificate with a supervisor token: a bearer the gateway mints from its existing launch-signing key under a profile of its own (typ openshell-supervisor+jwt, subject openshell-supervisor:<gateway_id>, no sandbox, generation, or epoch claims). The supervisor dials the gateway's Attach stream over ordinary server-authenticated TLS and presents the token; the gateway verifies it with the same key infrastructure, grants only the assignment stream, refuses every other credential on it, and ends a stream when the token that opened it expires. The gateway renews the token at half its lifetime and publishes each renewal into the supervisor host controller; the Kubernetes driver's only part in authentication is delivery, replacing the token in the chart-created <release>-supervisor-token Secret, which the supervisor mounts and re-reads on every connection attempt. The gateway Role may get and patch only that Secret. Remove the host-only TLS realm classification, the supervisor host CA mount and config, certgen --client-only, and the supervisor host PKI; gateway TLS returns to its pre-feature behavior and remains required for the topology. Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(helm): remove redundant certgen condition Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(supervisor): extract gateway-owned shared supervision into openshell-supervisor-control Move the shared supervisor control types out of openshell-core and openshell-sandbox-backend into a new openshell-supervisor-control crate: the SupervisorHostController (desired set, revision fencing, host sessions, observations), the SandboxInstanceRegistration launch material and SandboxInstanceStatusReport mapping, and a new SharedSupervision coordinator that owns the control contracts a compute driver previously implemented itself: validated persist-before-assign launches, recovery from durable records with the compare-and-replace baseline captured before the driver lists them, keep-previous-set on any failed or mismatched read, and one interpretation of host observations for readiness and bootstrap. Supervisor token delivery keeps its retry and newer-value policy in the crate; the write itself stays with the driver. The gateway now constructs the coordination: ComputeDriverBuildContext exposes shared_supervision() and the server keeps the controller for the Attach stream and token minting, replacing the driver-created controller that was installed into the gateway. The Kubernetes factory injects it through KubernetesComputeDriver::with_shared_supervision; new() stays the dedicated-topology constructor and refuses a configuration that selects the gateway topology without the gateway's coordination. The Kubernetes driver keeps its platform work: placement annotations, the shared supervisor Service and egress NetworkPolicy, launch Secret create and read (bytes only), its own fence and Pod evidence, and the token Secret patch. The supervisor process consumes the moved registration and status types unchanged. Wire protocol, storage JSON, annotations, and the supervisor token profile are unchanged; the private protobuf stays generated in openshell-core and is re-exported by the new crate. Signed-off-by: Drew Newberry <anewberry@nvidia.com> --------- Signed-off-by: Drew Newberry <anewberry@nvidia.com> |
||
|
Fede Kamelhar
|
78376bd041 |
fix(installer): resolve the gateway config dir like the CLI does (#4063)
* fix(installer): resolve the gateway config dir like the CLI does install.sh hardcoded $TARGET_HOME/.config/openshell in two places: the readiness check, which reads the client certificates it expects the gateway registration to have written, and the cleanup that removes a stale openshell entry before re-adding it. The CLI instead keeps both under $XDG_CONFIG_HOME/openshell whenever that variable is set, so with the variable pointing anywhere other than ~/.config the installer and the CLI looked in different directories: the readiness check waited for certificates that were never going to appear there, and the cleanup left the real stale entry in place. Resolve the directory through as_target_user instead, which answers from the same environment the CLI is invoked in. That covers both branches without duplicating their logic: the same-user branch inherits the caller's XDG_CONFIG_HOME, while the sudo and runuser branches reset the environment and fall back to the target user's home. An empty XDG_CONFIG_HOME falls back to $HOME/.config, following the XDG base directory specification. The CLI diverges here: xdg_config_dir treats a set-but-empty value as set and joins onto it, yielding the relative path "openshell", which no installer path can usefully match. Worth fixing separately in the CLI. Signed-off-by: fede-kamel <fkamelhar@gmail.com> * fix(core): treat an empty XDG_CONFIG_HOME as unset xdg_config_dir() joined onto an empty XDG_CONFIG_HOME and resolved the config root to the relative path "openshell", while the installer falls back to $HOME/.config/openshell for the same value. Registration runs the CLI with the caller's environment, so the two could disagree on where the gateway entry lives. Treat an empty value as unset, as the XDG base directory spec requires, so the CLI and installer resolve the same directory. Add tests for a value with spaces and drive registration cleanup through the helper for the spaces and empty cases. Signed-off-by: fede-kamel <fkamelhar@gmail.com> --------- Signed-off-by: fede-kamel <fkamelhar@gmail.com> |
||
|
Oliver Calder
|
cb42f5d9e8 |
fix(snap): use openshell-prover instead of openshell.prover (#4835)
The alias `openshell-prover` has been granted for OpenShell: https://forum.snapcraft.io/t/alias-request-for-openshell/53658 Thus, we can remove guidance differentiating `openshell.prover` for the snap from `openshell-prover` for other packaging formats. It is still worth noting that installing a locally-built snap directly, rather than from the store, means there is no alias. Signed-off-by: Oliver Calder <oliver.calder@canonical.com> |
||
|
Matthew Grossman
|
fac82327b3 |
test(testkit): add the testkit library for end-to-end suites (#4724)
* test(testkit): add the testkit workspace for end-to-end suites Add tests/testkit as its own Cargo workspace: testkit, testkit-macros, and testkit-spec. testkit reads a target file, connects to the gateway through openshell-sdk, creates and cleans up sandboxes, and runs the installed CLI. #[testkit::test] requires a stable ID and a doc comment and declares capability requirements. testkit-ready waits for a target's gateway and CLI before tests run. Nothing uses testkit yet. The lockfile is seeded from the root lockfile so shared crates resolve to the same versions. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * ci(tests): format, lint and test testkit in Branch Checks Nothing else builds testkit, so check it in Branch Checks and the matching mise tasks to catch openshell-sdk changes that break it. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * docs(tests): describe how to write and run testkit tests Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * fix(testkit): give each sandbox its own cleanup timeout Cleanup shared one two-minute budget across all of a test's sandboxes, so a slow delete left the rest with no time and could leak them on a shared gateway. Delete them concurrently, each with the full timeout. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * docs(testkit): show how to exclude tests by capability Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * docs: list tests/testkit in the project structure tables Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * refactor(testkit): drop test IDs and capabilities for now Nothing reads test IDs or capability requirements until a runner, waivers, or result history exist, so remove the id and requires arguments and the target file's capabilities section. Both can return as optional attribute arguments without changing existing tests. With them gone, testkit-spec only checked the function shape and read the doc comment, so fold it into testkit-macros. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * docs(testkit): describe what testkit provides today Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * refactor(testkit): find the gateway the way the CLI does Replace the target file (OPENSHELL_TEST_TARGET) with the CLI's own gateway selection: OPENSHELL_GATEWAY_ENDPOINT, then OPENSHELL_GATEWAY, then the active gateway. A registered gateway's endpoint, auth mode, and mTLS material come from its CLI config directory. cx.cli runs OPENSHELL_BIN against the same gateway and workspace as the SDK. Suites now run unchanged under e2e/with-docker-gateway.sh and e2e/with-podman-gateway.sh, and environments need no extra file. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * fix(testkit): keep labels when creating a sandbox with a policy The policy path creates the sandbox through gRPC and dropped the spec's labels without an error. Pass them in the create request. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * fix(testkit): use the registered gateway's credentials for an endpoint When OPENSHELL_GATEWAY_ENDPOINT is set, the CLI takes credentials from the gateway named by OPENSHELL_GATEWAY, or else the registered gateway with that endpoint. testkit connected without a client certificate instead, so an mTLS endpoint such as the one e2e/rust/e2e-vm.sh exports worked for cx.cli but not for the SDK. Resolve it the way the CLI does, and pass both the name and the endpoint to cx.cli. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * docs(tests): list testkit's gateway variables in precedence order OPENSHELL_GATEWAY_ENDPOINT takes precedence over OPENSHELL_GATEWAY. Also mention OPENSHELL_BIN. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * fix(testkit): apply other attributes to the generated test #[testkit::test] left attributes such as #[ignore] and #[cfg] on the nested body function, where they have no effect on the test. Move them to the generated test, and reject a second test attribute such as Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> #[test] or #[tokio::test]. * fix(testkit): show usage for testkit-ready --help Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * test(testkit): cover cleanup after pass, failure, and panic Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * fix(testkit): reject #[should_panic] on testkit tests A testkit test panics on any failure, including an unreachable gateway or a failed cleanup, so #[should_panic] would let such a failure pass. Reject it and point to checking the expected failure in the body. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * test(testkit): keep only the macro's misuse checks as unit tests Suites built on testkit exercise gateway resolution, the CLI wrapper, polling, errors, and cleanup against a real gateway. Keep one unit test for the macro's compile errors, which a passing suite never reaches, and drop the test-only seams the other tests needed. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * refactor(testkit): connect through the SDK's mTLS support The SDK's ClientConfig now takes a client identity, so testkit no longer builds its own channel. It still resolves the registered gateway and reads its credential files, then hands the CA and client certificate to OpenShellClient::connect. That keeps TLS behavior, connect timeouts and keep-alive settings in the SDK, and drops testkit's direct tonic dependency. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * fix(testkit): bound readiness and cleanup with timeouts wait_ready checked elapsed time only after a health call returned, and cleanup started its timeout only after the delete returned, so a gateway that accepts a connection but never answers could hang either one. Wrap each whole operation in tokio::time::timeout: the readiness wait, including every connect and health call, and each sandbox's delete plus wait for deletion. A test covers a peer that accepts connections and never replies. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> --------- Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> |
||
|
Simon Scatton
|
220a1d5cda |
chore(ci): use GPT-6.1 Sol for compatibility review (#4796)
Signed-off-by: Simon Scatton <sscatton@nvidia.com> |
||
|
Matthew Grossman
|
bd0edb1941 |
refactor(tests): move tmachine support files under tests/tmachine (#4726)
* refactor(tests): move tmachine support files under tests/tmachine Ansible, cloud-init, and the tmachine Nix config belong to tmachine, so they now live in tests/tmachine/ alongside the crate, which moves to tests/tmachine/cli/. tests/suites/ and tests/support/ are unchanged. tmachine still runs from the directory holding ansible/, so playbook paths stay the same; only inputs outside that directory gain a ../ prefix. The Nix build of tmachine and the base images are identical to before. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * refactor(tests): name the tmachine crate directory orchestrator The crate boots VMs, provisions them with Ansible, runs suites, and caches disks; the other entries in tests/tmachine/ are its config. Name the directory for that role rather than for the command-line entry point. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * refactor(tests): keep the tmachine crate at tests/tmachine Put the crate back at tests/tmachine/, next to the Ansible and Nix config it uses, instead of in a subdirectory. Build tmachine from a fileset of Cargo.toml, Cargo.lock, and src/ so edits to that config do not rebuild the binary. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> --------- Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> |
||
|
Max Dubrinsky
|
a2b8022e07 |
fix(python): ship credential driver protocol bindings (#4760)
Closes #4759 Signed-off-by: Max Dubrinsky <mdubrinsky@nvidia.com> |
||
|
Matthew Grossman
|
0f5b4f0fbb |
ci(release): reuse the promoted prerelease's release notes preview (#4754)
* ci(release): reuse the promoted prerelease's release notes preview Stable releases now publish the selected prerelease's release-notes artifact with the prerelease tag replaced by the stable tag. Release falls back to generating notes when the preview is missing, expired, or malformed, or when dispatched with regenerate-notes=true. The selection script exposes the qualified Pre-release run ID for the download. Closes #4753 Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * ci(release): reuse release notes from the promotion job Download and retag the preview in Select Qualified Prerelease, which already knows the qualified Pre-release run, instead of a separate job. Drop the regenerate-notes input and preview format checks. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * ci(release): name the publish notes input after the caller's result Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> --------- Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> |
||
|
Oliver Calder
|
d4ad1bbd03 |
fix(snap): only enable gateway user service for one user (#4615)
* fix(snap): only enable gateway user service for one user Signed-off-by: Oliver Calder <oliver.calder@canonical.com> * docs(snap): correct snap docs for the gateway user service Signed-off-by: Oliver Calder <oliver.calder@canonical.com> * fix(snap): add documentation of global to per-user enablement migration Signed-off-by: Oliver Calder <oliver.calder@canonical.com> --------- Signed-off-by: Oliver Calder <oliver.calder@canonical.com> |
||
|
Varsha
|
1cb9ef8ab4 |
refactor(crypto): introduce a generic interface for crypto libraries and refactor AWS-LC (#3351)
* refactor(crypto): centralize AWS-LC behind backend interface Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com> * refactor(crypto): route new host proxy tests through facade Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com> * fix(crypto): make backend boundary check portable Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com> * fix(crypto): route Google JWT signing through facade and verify context dispatch Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com> * refactor(crypto): propagate digest backend failures Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com> * refactor(crypto): route Vault TLS fixtures through facade Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com> * refactor(crypto): adapt isolation runtime callers after main rebase Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com> * fix(crypto): declare sandbox dependency and remove boundary lint Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com> * refactor(crypto): support independent backends and backend-free CA import Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com> * chore(crypto): refresh feature suite lockfile after rebase Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com> * fix(crypto): route runtime integrity and entropy through backend Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com> * chore(crypto): refresh lockfile after upstream rebase Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com> * fix(crypto): route VM socket entropy through backend after rebase Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com> * docs(crypto): document architecture and contributor boundaries Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com> * test(crypto): adapt upstream fixtures to backend facade Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com> * test(crypto): route relay TLS fixture through facade Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com> * test(crypto): adapt service bearer TLS fixture after rebase Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com> * fix(crypto): route provider profile TLS initialization through facade Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com> * fix(crypto): adapt upstream VM and supervisor paths to facade Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com> * ci(crypto): enforce backend independence in branch checks Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com> * test(crypto): adapt upstream token grant TLS fixture Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com> * fix(crypto): avoid format collect in executable hashing Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com> * fix(crypto): address provider selection and decryption review Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com> * fix(crypto): adapt upstream operator TLS paths after rebase Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com> * fix(crypto): adapt upstream TLS fixtures and protocol tests Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com> --------- Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com> |
||
|
Simon Scatton
|
b1cfc91720 |
ci(release): promote qualified prereleases to stable (#4611)
* ci(release): promote qualified prereleases to stable Signed-off-by: Simon Scatton <sscatton@nvidia.com> * fix(release): preserve qualified images during stable publication Signed-off-by: Simon Scatton <sscatton@nvidia.com> * ci(release): run shared release jobs with Nix Signed-off-by: Simon Scatton <sscatton@nvidia.com> * fix(release): recheck latest before publishing shared channels Signed-off-by: Simon Scatton <sscatton@nvidia.com> --------- Signed-off-by: Simon Scatton <sscatton@nvidia.com> |
||
|
Piotr Mlocekanddivesh
|
604b9f66f6 |
feat(config): deliver gateway configuration over the supervisor session (#4320)
Supervisors poll the gateway for configuration every 10 seconds. This PR adds an opt-in push mode (config_delivery_mode = "push"): supervisors still start by polling, and once their ConnectSupervisor session is accepted, the gateway delivers every later change over it and supervisors apply it and report the result. poll stays the default, and older supervisors and gateways keep polling. --------- Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> Signed-off-by: divesh <dgude@nvidia.com> Co-authored-by: divesh <dgude@nvidia.com> |
||
|
Drew Newberry
|
77929d9c14 |
feat(sandbox): publish and activate sandbox backend protocol (#3898)
* feat(isolation): publish backend protocol contracts independently of runtime Signed-off-by: Drew Newberry <anewberry@nvidia.com> * feat(isolation): expose trusted launch and registry data contracts Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(isolation): name the required capability sandbox Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(sandbox): align protocol and compute capability names Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(sandbox): finalize public protocol contract names Signed-off-by: Drew Newberry <anewberry@nvidia.com> * feat(isolation)!: activate backend contracts in sandbox runtime and driver selection Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(sandbox): simplify runtime and fix backend example Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(examples): organize WASI backend in Rust Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(sandbox): defer protocol conformance fixtures Signed-off-by: Drew Newberry <anewberry@nvidia.com> * feat(examples): add supervised V8 backend demo Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(examples): stream V8 guest output with default policy Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(examples): handle V8 rejections and roll back failed launches Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(examples): align V8 crypto dependencies with CI AWS-LC Signed-off-by: Drew Newberry <anewberry@nvidia.com> * feat(sandbox): represent Sandbox Protocol operations in protobuf Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(isolation): replace backend traits with RPC clients Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(proto): remove legacy isolation protocol reservations Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(proto): clarify Sandbox Protocol names Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs(isolation): clarify authentication bootstrap and supervisor APIs Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(isolation): simplify policy enforcement confirmation Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(sandbox): rotate launch generations and stabilize retry sessions Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(sandbox): remove redundant backend adapters and TCP flags Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(sandbox): repair Docker restarts and macOS lint Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(isolation): simplify RPC contracts and backend registration Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(sandbox): deliver TCP denials before closing the RPC Signed-off-by: Drew Newberry <anewberry@nvidia.com> --------- Signed-off-by: Drew Newberry <anewberry@nvidia.com> |
||
|
Evan LezarandMatthew Grossman
|
a2982bf53d |
fix(ci): restore default e2e coverage with a Kubernetes recovery waiver (#4424)
* fix(ci): make conformance-only e2e selection explicit Signed-off-by: Evan Lezar <elezar@nvidia.com> * fix(ci): use mapped NodePort for kind gateway sessions Signed-off-by: Evan Lezar <elezar@nvidia.com> * fix(ci): waive SSH transport recovery only on Kubernetes Signed-off-by: Evan Lezar <elezar@nvidia.com> * fix(e2e): log a notice when runners skip Rust e2e tests Emit a GitHub Actions notice from the Docker, Podman, Kubernetes, and VM runners when e2e-conformance-only skips the Rust tests, so a narrowed lane is visible in the job summary instead of silently reporting success. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * docs(testing): move e2e conformance-only selector to shared section The selector applies to all four driver runners, so document it with the Rust CLI e2e suites instead of under the Kubernetes task variants. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> --------- Signed-off-by: Evan Lezar <elezar@nvidia.com> Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> Co-authored-by: Matthew Grossman <mgrossman@nvidia.com> |
||
|
Drew Newberry
|
b2069a267c |
feat(docker): support host and bridge supervisor networking (#4016)
* feat(docker): select supervisor networking for ECI workloads Signed-off-by: Drew Newberry <anewberry@nvidia.com> * test(docker): use automatic networking in e2e Signed-off-by: Drew Newberry <anewberry@nvidia.com> * test(docker): rely on existing e2e networking coverage Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs(docker): shorten supervisor networking configuration Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(docker): let local gateway discover supervisor endpoint Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(docker): make supervisor networking operator-selected Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(docker): resolve host gateway in the driver Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs(docker): simplify host networking guidance Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs(docker): trim configuration guidance Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(docker): discover host gateway once at driver startup Signed-off-by: Drew Newberry <anewberry@nvidia.com> --------- Signed-off-by: Drew Newberry <anewberry@nvidia.com> |
||
|
Simon Scatton
|
253884414a |
ci(release): decouple prerelease publication from qualification (#4420)
* ci(release): publish prereleases before qualification Signed-off-by: Simon Scatton <sscatton@nvidia.com> * ci(release): decouple prerelease publication from qualification Signed-off-by: Simon Scatton <sscatton@nvidia.com> --------- Signed-off-by: Simon Scatton <sscatton@nvidia.com> |
||
|
Simon Scatton
|
10c6319757 |
ci(compatibility): require explicit risk acceptance (#4595)
* ci(compatibility): require explicit risk acceptance Signed-off-by: Simon Scatton <sscatton@nvidia.com> * test(compatibility): remove report regression tests Signed-off-by: Simon Scatton <sscatton@nvidia.com> --------- Signed-off-by: Simon Scatton <sscatton@nvidia.com> |
||
|
krishicks
|
07fd2386ce |
feat(server): export gateway OCSF events as OTLP log records (#4500)
The gateway only persisted its OCSF events to the JSONL file, so collectors had to tail that file. Adding `[openshell.gateway.otlp.ocsf]` now also exports each gateway OCSF event as an OTLP log record, using the same resource identity as gateway traces. `schema_version` optionally downgrades the exported JSON to OCSF 1.1 or 1.3. OTLP endpoints are now configured per signal, so OCSF events can go to a collector that feeds a SIEM while spans go elsewhere. The existing `[openshell.gateway.otlp]` `endpoint` is the default for every signal and on its own still enables trace export. `[otlp.traces]` overrides the trace endpoint, and `[otlp.ocsf]` takes its own `endpoint` or falls back to the default. Each record carries the OCSF JSON as its body, the event time as its timestamp, a severity mapped from `severity_id`, and `ocsf.*` and `openshell.sandbox.id` attributes for filtering without parsing the body. Records flow through a queue bounded by count and bytes to the SDK OTLP log exporter in batches bounded the same way. Failed exports retry with backoff; records lost to a full queue, an oversized body, or exhausted retries are counted in metrics and reported to the collector in an `openshell.ocsf.gap` record on the next successful export. The exporter lives in `openshell-otel` so other processes can share it. Helm deployments set the `traces` and `ocsf` tables through `gatewayConfig`, and the local development gateway tasks enable OCSF export. Closes #4499 Refs #2892 Signed-off-by: Kris Hicks <khicks@nvidia.com> |
||
|
Polite_realism
|
36ee06f7b1 |
fix(rpm): build the Python SDK from older protoc and install every SDK module (#4439)
* fix(python): let proto stub generation skip .pyi for older protoc Add a --no-pyi option to generate_python_proto.py for toolchains whose protoc predates --pyi_out, such as Fedora's grpcio-tools 1.48, and list python3-googleapis-common-protos and python3-httpx in the spec's Recommends and dist-info Requires-Dist, which the SDK needs to import. Default behavior is unchanged. Contributes to #3510. Signed-off-by: politerealism <burdcat17@gmail.com> * fix(rpm): install every SDK module and record the license accounting The python3-openshell %install copied only __init__.py, sandbox.py and _proto/, but the package imports .errors, .mutations and .providers and ships a py.typed marker, so an installed RPM could not import openshell. Install the missing files and add a %check that fails if any relative import in the installed package has no matching installed module. Importing the package needs grpcio, which the build environment lacks, so the check is static. Set License to the minimal SPDX expression covering every crate linked into the shipped binaries, keep the pure-Python subpackage at Apache-2.0, and note how the non-crate content is covered. Signed-off-by: politerealism <burdcat17@gmail.com> --------- Signed-off-by: politerealism <burdcat17@gmail.com> |
||
|
Drew Newberry
|
6e69360d0e |
refactor(runtime): extract portable driver and isolation foundations (#4106)
* refactor(runtime): decouple supervisor access and boundary audit validation Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(runtime): keep supervisor process access boundary driven Signed-off-by: Drew Newberry <anewberry@nvidia.com> * feat(cli): support environment values from caller variables Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(runtime): extract driver-independent MXC foundations Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(runtime): defer UI policy support to MXC integration Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(runtime): integrate current policy and SSH launch contracts Signed-off-by: Drew Newberry <anewberry@nvidia.com> * test(runtime): reconcile fixtures with updated main Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(runtime): repair stacked branch CI contracts Signed-off-by: Drew Newberry <anewberry@nvidia.com> * test(runtime): update boundary fixtures for current contracts Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs(runtime): clarify scoped guards for launch inputs and delete Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(runtime): validate foreground attachment intent before launch Signed-off-by: Drew Newberry <anewberry@nvidia.com> --------- Signed-off-by: Drew Newberry <anewberry@nvidia.com> |
||
|
|
e30c938bb1 |
feat(server): scope mutation locks and bound lock waits (#4147)
* test(server): add a local PostgreSQL test runner
The openshell-server tests that need a real PostgreSQL server are
ignored by default and had no shared way to run. The mutation-replay
test reads its own OPENSHELL_REPLAY_TEST_DATABASE_URL, so every
contributor had to provision a database by hand, and the advisory-lock
tests that follow in this series need the same setup.
Add mise run test:rust:postgres. It runs every ignored postgres_* test
in openshell-server against OPENSHELL_TEST_POSTGRES_URL, or starts a
disposable PostgreSQL container with Docker or Podman (CONTAINER_ENGINE
selects one) on the image pinned by the Kubernetes e2e fixture and
removes it on exit. Tests run one at a time because advisory locks are
database-wide. The runner always points the legacy replay variable at
the selected database, so an inherited URL cannot send that test to a
different server. test:postgres-runner checks that selection with a fake
cargo and needs no database or container engine. CI does not run the
PostgreSQL tests; TESTING.md documents the task.
Part of #3528
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* fix(server): hold the mutation guard while deleting a provider
DeleteProvider checked that no sandbox referenced the provider and then
deleted the record without holding the sandbox mutation guard. Sandbox
create and provider attach take that guard while they write provider
references, so one of them could add a reference after the attached
sandbox check passed, and the delete then removed a provider that a
sandbox spec still named.
Take the guard before the attached sandbox check, as provider create and
update already do, so the check and the delete run against a stable
set of sandbox references, and reject an empty name after authorization
but before the guard. A new test holds the guard, attaches the provider
while the delete waits, and asserts that the delete fails with
FailedPrecondition and leaves the provider in place.
Part of #3528
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* fix(server): release staged credentials when a refresh fails early
A provider credential refresh stages the minted values under new
credential handles before it commits the provider update, and deletes
those handles when validation or persistence fails. Two earlier returns
skipped that cleanup. The minted expiry was converted to a protobuf
timestamp only after staging, so an expiry outside the timestamp range
failed the refresh and left the staged values in the credential driver.
A failure to acquire the sandbox mutation guard returned the same way.
Convert the expiry before staging anything, so a bad value fails before
any handle exists, and delete the staged handles when the guard cannot
be acquired. A new test refreshes a stored credential with an expiry of
i64::MAX and asserts that the call fails, the provider keeps its
original handles, and the credential driver holds the same number of
values as before. The guard failure path has no test here, because the
SQLite guard used by unit tests cannot fail.
Part of #3528
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* feat(server): acquire mutation locks on a dedicated PostgreSQL lock pool
The cross-object mutation guard took its advisory-lock session from the
10-connection data pool and closed it on every release, so each guarded
mutation opened a new database connection, and lock holders competed
with their own critical sections for data connections. A lock wait that
hit the 10 second lock_timeout surfaced as INTERNAL.
Add a lazy, dedicated 4-connection lock pool and a new
persistence::mutation_lock module for the lock key, mode and set types.
Store::acquire_distributed_mutation_guard now takes a lock set and a
deadline, and each lock statement sets lock_timeout to the time left
before that deadline, so a cancelled acquisition never waits past it on
the server. Returned connections are scrubbed with
pg_advisory_unlock_all() and reused, and each return is bounded by 5
seconds so a stalled session cannot keep a pool permit.
A lock statement that PostgreSQL ends with SQLSTATE 55P03 fails with a
new LockTimeout error, and so does a wait for a lock connection during
which every one was checked out at some point, or that started with less
than a second left, too little to open one. The API returns it as
UNAVAILABLE with reason MUTATION_LOCK_TIMEOUT and a 1 second retry
delay, because the mutation lock was not acquired, so the request's
guarded writes did not run. Only the guard acquisition classifies 55P03;
any other 55P03 stays a database error. A lock connection that
PostgreSQL does not open by the deadline despite at least a second to do
so (refused, out of connection slots, starting up, or not answering, all
of which SQLx retries silently) is a database error, not a lock timeout.
Mutations still take the legacy global key exclusively, so old and new
replicas keep excluding each other during a rolling upgrade. Each
PostgreSQL-backed gateway pod can now open up to 14 database
connections (10 data plus 4 lock) instead of 10, and the chart's
autoscaling.maxReplicas comment says what its default of 4 needs. The
SSH host identity lock stays on a data connection, because sandbox
creation takes it while holding its mutation guard and a second
lock-pool connection there could exhaust the pool. The provider
credential refresh lock stays there too, so a refresh still progresses
while every lock connection is in use.
Part of #3528
Co-authored-by: divesh <dgude@nvidia.com>
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* perf(server): partition mutation locks by workspace and sandbox
Every cross-object mutation, and every lifecycle and supervisor update,
took one process-wide mutex and, on PostgreSQL, one fleet-wide advisory
lock. Unrelated sandboxes and providers therefore serialized across all
replicas, and one slow critical section delayed every other mutation in
the fleet.
Replace that lock with hierarchical intention locks on a global key, one
key per workspace and one key per sandbox, each held shared or
exclusive. Global settings and policy writers and platform-scope profile
writers hold the global key exclusively. Provider and workspace-scoped
profile writers, including DeleteProvider and credential refresh, hold
their workspace key exclusively. Sandbox writers and supervisor reports
hold their sandbox key exclusively under shared global and workspace
keys. Lifecycle, driver-watch and reconcile paths take only
process-local keys and keep relying on compare-and-swap across replicas.
Workspace and sandbox keys are domain-separated SHA-256 prefixes. Each
guard takes its local keys and then its advisory locks in ascending
order under one 10 second deadline, and no path nests guards. The
settings mutex is removed, since the global and sandbox keys cover it.
Sandbox settings are keyed by sandbox name, which a sandbox key no
longer covers once another sandbox reuses the name, so a settings
update returns NOT_FOUND unless its sandbox still holds the name after
the read, and ABORTED if the settings row was recreated before the
write.
The global key keeps its legacy value, so older replicas, which hold it
exclusively for every mutation, still exclude new ones during a rolling
upgrade. Guards now also give up after 10 seconds on SQLite, and
provider refresh reports a guard timeout as UNAVAILABLE instead of
INTERNAL. A supervisor report for a missing sandbox returns NOT_FOUND.
Startup endpoint-status reconciliation still holds the global key for
its whole scan.
Part of #3528
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* fix(server): reconcile endpoint status on startup one sandbox at a time
Startup endpoint-status reconciliation held the global mutation key
exclusively for its whole scan. On PostgreSQL that blocked every guarded
mutation on every replica until the starting gateway had walked all
sandboxes, and the scan first waited behind every in-flight sandbox
mutation. It also paged with an offset, so sandboxes deleted mid-scan
shifted rows between pages, and a write that raced the scan (a Conflict
from a lifecycle update on another replica, or a sandbox deleted after
it was listed) failed gateway startup.
Walk the sandboxes with keyset paging instead, and reset each candidate
under its own sandbox-scoped guard, up to four at a time to match the
lock pool. A candidate with a fresh shared owner is skipped before any
guard is taken. Under the guard the reconciliation re-checks the owner,
re-reads the sandbox, skips it when it is gone or no longer carries
endpoint status, and writes against the version it just read. A Conflict
or a database error at write time, which is how a row deleted after the
re-read surfaces, is retried with a fresh read, at most five attempts
per sandbox. A lock timeout on the guard is retried the same way, so a
holder of the sandbox, its workspace or the global key that releases
within the retry window cannot stop a replica from starting. No
lock-timeout retry starts once the scan has run for three lock timeouts,
so a scan stuck behind a long holder fails within about 40 seconds,
inside the chart's 60-second startup probe, and logs the sandbox it gave
up on. Store::list_all_messages loses its only caller and is removed.
Reconciliation stays fail-closed: a holder that outlasts the retry
window, an owner lookup failure, or a sandbox that keeps changing still
fails startup, so stale endpoint success is never served. User-visible
change: a restarting gateway no longer stalls mutations of unrelated
sandboxes and providers on its peers while it reconciles.
Part of #3528
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* perf(server): skip the endpoint-status lock when a session was replaced
When a supervisor session ends, the gateway resets that sandbox's tool
server endpoint evidence unless a replacement session already owns
endpoint observation. The replacement check ran only after the reset
had read the sandbox and taken its sandbox mutation guard, so a reset
whose supervisor had already reconnected still queued behind other
mutations of that sandbox, and on PostgreSQL it used a lock-pool
connection, only to find it had nothing to do.
Check for a replacement first: a live local session, or a fresh owner
record on a peer. When one exists, return without reading the sandbox
or taking the guard. Otherwise take the guard and repeat the same check
under it, because a replacement's pre-acknowledgement reset runs under
the same sandbox key and resetting after it would wipe the
replacement's fresh evidence. Both checks share one helper, so they
cannot drift apart. When no replacement exists, the reset costs one
extra owner-row read.
This helps most when a replica notices a dead stream late, after the
supervisor has already reconnected elsewhere (a network partition or a
keepalive expiry).
Deferring to a replacement leaves the old session's evidence to the
replacement's pre-acknowledgement reset, which can time out on its
bounded mutation lock wait. That reset's failure path removed the
replacement and released its owner record but scheduled no
invalidation, so the old session's success evidence could outlive both
sessions. An old session that ended after its replacement registered
already deferred this way, and the unguarded check now makes every
disconnect that finds the replacement defer too. When the reset fails
while the replacement is still current, also schedule the retried
invalidation that follows a session end. That failure path keeps the
lifecycle demotion from abandon_session_setup, but runs it in the
spawned task before the retry and only while this session is still
current, so a timed-out reset returns promptly.
Part of #3528
Co-authored-by: divesh <dgude@nvidia.com>
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* test(server): cover staged credential cleanup on a lock timeout
A provider credential refresh deletes its staged handles when it cannot
acquire the provider mutation guard, but that path had no test: the
SQLite guard used by unit tests could not fail. Mutation locks now
acquire under a deadline that tests can shorten, so the timeout path
can be exercised directly.
The new test shortens the lock deadline to 50 ms, holds the workspace
mutation guard, and refreshes a provider that already stores an AWS
access key. The refresh returns UNAVAILABLE with the
MUTATION_LOCK_TIMEOUT reason, the provider keeps its original handle,
which still resolves to the old key, and the credential driver holds
the same number of values as before the refresh.
Part of #3528
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* feat(server): export mutation lock wait and timeout metrics
A handler mutation guard can now wait up to 10 seconds for its keys and
then fail with UNAVAILABLE, but nothing showed operators how long guards
waited or how often they gave up, short of reading warning logs on each
replica.
Export openshell_server_mutation_lock_wait_seconds, a histogram of the
time one guard took to acquire all of its keys (local registry plus
PostgreSQL advisory locks), recorded on success, and
openshell_server_mutation_lock_timeouts_total, a counter of guard
acquisitions that timed out. Both carry a bounded scope label: global,
workspace or sandbox. The platform scope, whose workspace is empty,
counts as global. The histogram uses the same 1 ms to 15 s buckets as
the relay and peer histograms, and the timeout counter starts at 0 for
every scope, so rate() works on an idle replica. The timeout warning
now also logs the scope. A guard that fails for another reason, such as
a lock connection PostgreSQL did not open, is not counted as a timeout
and logs a "mutation lock acquisition failed" warning instead.
Lifecycle, driver-watch and reconcile paths take only process-local
locks and never time out, so they are not recorded.
Part of #3528
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* test(server): add PostgreSQL mutation lock integration tests
The scoped mutation locks were only unit-tested on SQLite, where the
database layer is a no-op. Add ignored postgres_* tests that run the
advisory-lock layer against a real PostgreSQL through mise run
test:rust:postgres, each in its own disposable schema (TestSchema).
persistence/mutation_lock_pg_tests.rs uses two stores on one database
as two replicas. It covers disjoint sandbox scopes holding at once, a
workspace key blocking only sandboxes in that workspace, the global key
blocking every scope, and exclusion in both directions against a raw
holder of the legacy global key, whose 55P03 stays a database error
outside a guard acquisition. It checks that timed-out and cancelled
acquisitions release the keys they already took, that a released lock
connection returns to the pool without residual locks, that a release
stalled on the network closes its session and the pool recovers, and
that the lock pool never grows past its size. It also checks that a
full lock pool times out as lock contention, while a lock connection
that PostgreSQL never answers fails as a database error.
compute/mutation_guard.rs runs the random scope mix across two
PostgreSQL runtimes and checks that a by-id guard waits for a workspace
writer on the other replica. The provider refresh lock test moves onto
the runner too, and checks that a refresh proceeds while every lock
connection is held.
A bench_postgres_* capacity check measures a paced burst of 500
reconnects 12 ms apart into one replica and fails when the p99 lock
wait reaches a fifth of the lock timeout. Machine load moves those
waits, so mise run test:rust:postgres skips it and mise run
test:rust:postgres:bench runs it alone, through a new
OPENSHELL_TEST_POSTGRES_PREFIX setting in the runner.
The tests add test-only helpers: the lock pool's size and idle count
and the backend pid of a held lock session. Nothing changes outside
tests and the test runner.
Part of #3528
Co-authored-by: divesh <dgude@nvidia.com>
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* ci: run the PostgreSQL-backed server tests
Branch Checks now runs the postgres_* tests through the same runner
developers use, on an x86_64 Linux runner with Docker. They cover lock
scope exclusion, interop with replicas still on the old global key,
cancellation, lock-pool bounds and connection failures, which no other
CI lane checks directly. The load-sensitive bench_postgres_* capacity
checks stay local.
Part of #3528
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* docs: document scoped mutation locks and lock-pool sizing
In the High Availability guide, size PostgreSQL for 14 connections per
gateway pod, 10 for data and 4 for the new lock pool, so three or more
replicas now exceed what the PostgreSQL default leaves, and keep
headroom for lock sessions that cancelled requests leave behind.
Explain which changes lock a sandbox, a workspace, or the whole fleet,
and that a lock wait over 10 seconds returns a retryable UNAVAILABLE
with reason MUTATION_LOCK_TIMEOUT. A new upgrade subsection says to
raise max_connections to the 14-per-pod size before upgrading, because
the rollout already runs new pods at that count, and warns that older
replicas keep serializing every mutation until the rollout finishes.
Monitor Capacity gains the lock metrics and their queries, and names
slow driver, credential, middleware, or profile source calls as another
cause of lock waits.
List the lock wait histogram and timeout counter, with their scope
label, on the Gateway Metrics page. The counter also counts background
work and leaves out lock connections that PostgreSQL does not open
although at least a second of the wait remained.
In the cluster debugging skill, replace the single advisory-lock
paragraph with the lock levels and pool, what each timeout detail
means, how a lock connection that PostgreSQL does not open shows up,
when terminating a lock backend is safe, a pg_locks query that shows
holders and waiters, and how to spot the global key. The per-pod greps
also read the timeout counter, and new rows cover lock timeouts and
lock connections that PostgreSQL does not open.
Part of #3528
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* feat(server): check mutation lock ordering in debug builds
The mutation lock ordering rules were prose only, so a nested guard on
a different key passed every test and could only deadlock under
contention. Debug builds now register each held guard with the Tokio
task (or the thread, outside a task) that acquired it, and panic before
waiting when a local key, a blocking lifecycle gate or the SSH identity
lock would break rules 1, 4 or 5. Startup endpoint-status
reconciliation and the tests that hold a guard while driving guarded
code run under lock_order::branch(), which marks deliberate concurrency
within one task. Release builds compile the check out.
Part of #3528
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* fix(server): do not hold gateway shutdown for the endpoint-status retry
When a replacement supervisor session failed its endpoint-status reset,
the spawned cleanup task kept the session lifetime guard through
retry_endpoint_status_after_supervisor_disconnect. That retry only ends
once the reset is durable, so a gateway shutdown during a database or
lock outage waited the full 10 s and reported ownership cleanup as
incomplete. The guard now covers only the lifecycle demotion, as in
finish_supervisor_session, and a SQLite test checks that shutdown
completes while the retry is still blocked.
Part of #3528
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* refactor(server): make the platform profile lock scope explicit
Platform-scope provider profile writers (import, update, delete) pass an
empty workspace on purpose and need X(global), because they validate
sandboxes in every workspace. MutationScope::profiles() now names that
escalation and picks Global for the platform scope, instead of relying
on MutationScope::Workspace("") to mean it. An empty workspace name now
trips a debug assertion, and release builds keep X(global) as a
fail-safe that over-locks rather than under-locks.
Part of #3528
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* perf(server): amortize local mutation lock table sweeps
LocalMutationLocks swept its whole table on every key lookup, so a
burst of N concurrent acquisitions on one replica cost O(N^2) under a
blocking mutex. The table is now swept only when a new key grows it
past the larger of 64 and twice the live entries left by the last
sweep, which makes acquisition amortized O(1). Released entries,
including those of cancelled waiters, stay bounded at about twice the
peak live set. The lifecycle gate registry keeps its per-call sweep.
Part of #3528
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* feat(server): make the database pool ceiling configurable
Both stores hardcoded their pool ceiling: 10 connections for Postgres, 5
for on-disk SQLite. One pool is shared by every database-backed RPC, so
that number is also the gateway's ceiling on concurrent database work.
Once every connection is checked out, callers queue on acquire and sqlx
logs "time to acquire exceeded slow threshold"; sandbox creates then time
out and are retried, which adds load rather than shedding it.
The right ceiling is deployment-specific — it depends on how many gateway
replicas share the database and what max_connections the server itself
allows — so it cannot be a single number baked into the binary.
Expose it on the same config surface as the rest of the gateway's
settings: --db-max-connections, OPENSHELL_DB_MAX_CONNECTIONS, and the
TOML key database_max_connections, resolved in that precedence order.
Helm deployments set the key through gatewayConfig. Omitting it keeps
each backend's previous value, so existing deployments do not move.
Values below 1 are rejected rather than silently replaced by the default:
a zero pool would block every acquire, and quietly ignoring a typo would
reproduce the ceiling the operator is trying to lift. An in-memory SQLite
database stays pinned to one connection, since the database lives inside
that connection.
Refs: #2561
Signed-off-by: Bryce Wilkinson <22760097+bjw123@users.noreply.github.com>
(cherry picked from commit
|
||
|
Simon Scatton
|
d3e6a35e47 |
ci(compatibility): review PR breaks and acknowledge release risks (#4488)
* ci(compatibility): review PR breaks and acknowledge release risks Signed-off-by: Simon Scatton <sscatton@nvidia.com> * fix(ci): use matching compatibility review tooling Signed-off-by: Simon Scatton <sscatton@nvidia.com> * ci(compatibility): scope Rust SDK review to public exports Signed-off-by: Simon Scatton <sscatton@nvidia.com> * ci(compatibility): account for Go struct field additions Signed-off-by: Simon Scatton <sscatton@nvidia.com> * ci(compatibility): restrict egress and fix report retries Signed-off-by: Simon Scatton <sscatton@nvidia.com> --------- Signed-off-by: Simon Scatton <sscatton@nvidia.com> |
||
|
Matthew GrossmanandEvan Lezar
|
8c67c27e8b |
test(oci): add a shared Docker and Podman suite for image USER and WORKDIR (#3931)
* test(oci): share OCI image checks across Docker and Podman Add an oci-image feature testsuite that runs against installed artifacts. It covers custom WORKDIR placement, image content and ownership, workspace writes from the main process and exec, file transfer including directory-merge upload, OCI user identity, the managed /sandbox fallback, and rejection of an unwritable WORKDIR. CI runs the suite on the Docker rootful, Podman rootful, and Podman rootless tmachine guests. Replace the Docker-only custom_image e2e with the shared suite. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> Signed-off-by: Evan Lezar <elezar@nvidia.com> * test(oci): require WorkspaceValidationFailed and cover image USER fallback The unwritable WORKDIR scenario now requires the WorkspaceValidationFailed reason, so unrelated provisioning failures cannot pass it. The numeric USER scenario supplies a policy without a process section, so the image USER is the only source of the sandbox identity. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> Signed-off-by: Evan Lezar <elezar@nvidia.com> * test(podman): leave shared OCI identity checks to the oci-image suite The oci-image feature suite covers the sandbox identity seen by the main process and sandbox exec on Docker and Podman. The Podman e2e keeps the checks only it can make: the pinned image ID and the isolated workload/supervisor container pair. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> Signed-off-by: Evan Lezar <elezar@nvidia.com> * test(oci): accept existing workspace rejection diagnostics Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> Signed-off-by: Evan Lezar <elezar@nvidia.com> * test(oci): fix cleanup ordering and restore coverage parity Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> Signed-off-by: Evan Lezar <elezar@nvidia.com> * test(oci): remove migrated legacy workspace helpers and checks Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * fix(drivers): report rejected OCI working directories as WorkspaceValidationFailed The supervisor exits with the reserved status when the sandbox rejects the image working directory. Docker maps that exit during readiness and monitoring and records it before removing the supervisor container, and Podman maps the supervisor companion's exit instead of the workload's. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * test(oci): require WorkspaceValidationFailed for unwritable WORKDIR Unrelated provisioning failures can no longer pass the rejection scenario. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * fix(oci): keep exit 78 reserved and report fixed workspace diagnostics The supervisor no longer passes on a workload's own exit status 78, so drivers cannot misreport it as a workspace rejection after a runtime restart. The rejection is recognized once at agent start and carried as a typed error. Docker reports the same fixed condition message as Podman and logs the supervisor output instead, and its monitor removes the supervisor container when a stop overlaps diagnostics collection. The OCI image suite registers images for cleanup only after they build. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * refactor(oci): simplify workspace failure handling and OCI image tests Report a recorded Docker workspace rejection once, where supervisor readiness fails, and match its fixed message exactly. Read the Podman supervisor exit status once per inspection. Choose the supervisor exit status in one function and derive the workspace error type. The OCI image suite shares step helpers, checks that a custom workspace does not also create /sandbox, and its playbook waits for a connected gateway and collects failure diagnostics through the shared role. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * test(oci): replace podman_oci_identity with shared and driver suite checks The oci-image feature suite now checks, on Docker and Podman, that the image user has no capabilities, has no_new_privs set, and cannot read the sandbox control credentials, and that a custom WORKDIR is not a mount. The checks that need container inspection (pinned image ID, network=none, supervisor user and capabilities) move to a small test in the Podman driver suite. Remove the legacy podman_oci_identity e2e test. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * test(oci): keep podman_oci_identity for Podman-only isolation checks Restore the Podman e2e test removed in the previous commit and drop only the checks the shared oci-image suite now runs on Docker and Podman: workload capabilities, no_new_privs, credential readability, and the workspace and /sandbox mount checks. The pinned image ID and the workload/supervisor isolation checks stay, as does the Podman driver suite unchanged. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * refactor(oci): simplify workspace failure reporting and OCI image suite Carry the Docker control supervisor's failure reason in a typed error instead of matching status text, build the fixed workspace message where the wait result is classified, and drop the fake Docker server test. Inline image cleanup ordering in the suite, read the container engine once, share sandbox naming, and stop granting a literal /sandbox in the image identity policy. Add e2e:oci-image mise tasks and use them from the ARM64 Docker workflow and TESTING.md. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * test(oci): move shared container checks into the OCI image suite Inspect the containers the driver created through the gateway's engine in the shared suite: the workload runs the inspected image ID as the OCI identity without a network or supervisor credential mounts, and the supervisor is a separate non-root container. Keep only the Podman-specific supervisor identity and capability checks in podman_oci_identity. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * test(oci): check the workspace rejection reason from sandbox JSON Read the WorkspaceValidationFailed condition reason from sandbox get --output json instead of matching CLI output, and restore the Docker driver's original workspace failure helpers. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * test(oci): capture fixture image identity before sandbox creation Signed-off-by: Evan Lezar <elezar@nvidia.com> --------- Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> Signed-off-by: Evan Lezar <elezar@nvidia.com> Co-authored-by: Evan Lezar <elezar@nvidia.com> |
||
|
Florent BENOIT
|
cf6ec17785 |
refactor(vm): source runtime pins only from pins.env (#4373)
The VM runtime scripts sourced pins.env optionally and duplicated the LIBKRUN_REF and UMOCI_VERSION defaults, so every bump had to update the same value in several files. Require pins.env, drop the duplicated defaults, and fail on an invalid LIBKRUN_REF instead of silently building upstream HEAD. Environment overrides still take precedence through the defaults in pins.env. Signed-off-by: Florent Benoit <fbenoit@redhat.com> |
||
|
Evan Lezar
|
2ffc3d2b4d |
ci: require accepted issues for community pull requests (#4340)
Evaluate issue acceptance using trusted default-branch code and the PR author's repository permissions. Report the native workflow job result, protect enforcement files, and align PR creation and review guidance. Closes #4339 Signed-off-by: Evan Lezar <elezar@nvidia.com> |
||
|
Oliver Calder
|
16c8ca5c43 |
fix(docs): use lowercase "snap" as it is not a proper noun (#4387)
Signed-off-by: Oliver Calder <oliver.calder@canonical.com> |
||
|
Evan Lezar
|
c4dc30dc1b |
test(conformance): run driver suites with cargo (#3866)
* test(conformance): run driver suites with cargo Signed-off-by: Evan Lezar <elezar@nvidia.com> * fix(e2e): resolve conformance CLI path before cargo tests Signed-off-by: Evan Lezar <elezar@nvidia.com> * refactor(conformance): remove unused CLI scenario APIs Signed-off-by: Evan Lezar <elezar@nvidia.com> --------- Signed-off-by: Evan Lezar <elezar@nvidia.com> |
||
|
Evan Lezar
|
78975ea220 |
chore(test-guest): remove snap reproduction helper (#4368)
Signed-off-by: Evan Lezar <elezar@nvidia.com> |
||
|
Oliver Calder
|
82bc756545 |
feat(snap): run gateway as a user service by default (#4099)
* feat(snap): run gateway as a user service by default Replace the existing system `gateway` service with separate `user-gateway` and `system-gateway` services. For new installs, only the `user-gateway` service is enabled. For existing installs, only the `system-gateway` service is enabled. Existing installs continue to have a one-time migration which removes legacy insecure configurations. It is up to `install.sh` or users to manually copy mTLS credentials from the root-owned `$SNAP_COMMON/tls` to the invoking user's OpenShell snap directory. This is explained in the snap description in `snapcraft.yaml`, visible in the Snap Store listing and via `snap info openshell`. Service enablement is now managed by a new snap configuration option named `gateway-mode`, so users can switch from one mode to another via e.g. `sudo snap set openshell gateway-mode=user`. The `install` and `post-refresh` hooks select which service to start by setting this mode. If the `gateway-mode` is already set, then we know the one-time migration has already taken place. Signed-off-by: Oliver Calder <oliver.calder@canonical.com> * fix(snap): simplify the snap gateway wrapper Signed-off-by: Oliver Calder <oliver.calder@canonical.com> * fix(snap): use `snap start --user` when running as non-root Signed-off-by: Oliver Calder <oliver.calder@canonical.com> * fix(snap): ensure install.sh waits for the correct docker daemon Signed-off-by: Oliver Calder <oliver.calder@canonical.com> * feat(snap): document how to safely switch between gateway modes New installs run the gateway as a user service by default, while upgrades of existing installs from prior to the user gateway continue to use the existing system gateway. The user and system gateway are fundamentally different services and there is no way to migrate state between them. However, it may be desirable for a user to run the gateway as a system service after a fresh install. To support this, this commit adds documentation for how to safely transition from the user gateway mode to the system gateway mode, and vice versa. Also, clarifies when it is necessary to manually copy TLS credentials from the system gateway state directory to the user's home directory: when upgrading a pre-TLS snap installation, or when switching from user to system gateway mode. When upgrading an existing install with an HTTPS gateway running as a system service, copying credentials again is not necessary. Signed-off-by: Oliver Calder <oliver.calder@canonical.com> --------- Signed-off-by: Oliver Calder <oliver.calder@canonical.com> |
||
|
Gaizka MenendezandTaylor Mutch
|
fe3942f2da |
feat(helm): migrate gateway configuration to gatewayConfig (#3384)
* feat(helm): add generic gateway TOML serializer Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * docs(architecture): define Helm gateway configuration boundary Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * feat(helm): define default gateway configuration map Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * refactor(helm): render gateway config from values map Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * test(helm): cover generic gateway TOML rendering Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * feat(helm): protect gateway config secret boundary Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * docs(helm): classify legacy gateway configuration values Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * refactor(helm): derive dual-use resources from gateway config Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * test(helm): migrate gateway config scenarios Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * refactor(helm): derive credential resources from gateway config Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * refactor(helm): migrate gateway config overlays Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * refactor(helm): remove gateway configuration shadow values Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * refactor(helm): align runtime config with resources Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * docs(helm): add gateway config migration guide Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * test(helm): validate rendered gateway config Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * test(helm): cover config resource coherence Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * fix(helm): make Kubernetes E2E deployable Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * fix(e2e): pass host aliases through gateway config Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * docs(e2e): reference gateway config host alias Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * fix(helm): enforce gateway resource ownership Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * test(e2e): cover chart host gateway input Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * fix(helm): address gateway config review findings Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * fix(helm): repair RFC 0012 migration integration Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * fix(e2e): align Kubernetes parity with RFC 0012 Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * fix(helm): complete gateway config migration Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * chore(helm): add SPDX header to TOML template * fix(helm): preserve legacy gateway configuration aliases Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * fix(helm): complete legacy gateway config compatibility Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * feat(helm): support gateway config arrays of tables * fix(helm): preserve sandbox identity aliases * fix(helm): address gateway config review findings Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * fix(helm): preserve gateway config migration settings Signed-off-by: Taylor Mutch <taylormutch@gmail.com> * test(helm): validate extension arrays in CI values Signed-off-by: Taylor Mutch <taylormutch@gmail.com> * fix(helm): restore unconditional SPDX headers Signed-off-by: Taylor Mutch <taylormutch@gmail.com> * fix(helm): restore route SPDX header placement Signed-off-by: Taylor Mutch <taylormutch@gmail.com> * docs(helm): simplify configuration and migration examples Signed-off-by: Taylor Mutch <taylormutch@gmail.com> * docs(helm): remove redundant flag explanations Signed-off-by: Taylor Mutch <taylormutch@gmail.com> * docs(helm): move chart configuration guidance to Kubernetes setup Signed-off-by: Taylor Mutch <taylormutch@gmail.com> --------- Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> Signed-off-by: Taylor Mutch <taylormutch@gmail.com> Co-authored-by: Taylor Mutch <taylormutch@gmail.com> |
||
|
Florent BENOIT
|
616024f79e |
fix(vm): bump libkrun to v1.19.6 (#4282)
## Summary Bump the pinned libkrun from v1.19.4 to v1.19.6, the latest upstream release. v1.19.5 includes libkrun/libkrun@d2d8dd6, which removes refused vsock connections without waiting for the 5 s reaper, so VM sandbox startup no longer stalls on "Waiting for VM supervisor". ## Related Issue Closes #4259 ## Changes - libkrun: v1.19.4 (728df812) -> v1.19.6 (227b2de6) - Update LIBKRUN_REF in pins.env and the fallback defaults in build-libkrun.sh and build-libkrun-macos.sh - libkrunfw stays on v5.6.1 Other notable upstream fixes (v1.19.5): vsock proxy release after clean shutdown, macOS virtio-fs xattr handling, x86_64 initrd kept below the MMIO gap, PIT missed-tick reinjection disabled, macOS VMs with >64GB RAM. v1.19.6 adds an API to set the render server fd. No API removals. ## Testing - libkrun v1.19.6 builds on macOS arm64 with `make BLK=1 NET=1` Signed-off-by: Florent Benoit <fbenoit@redhat.com> |
||
|
alangou
|
249c6e727f |
fix(ci): retain sanitized Codex scan diagnostics on failure (#4304)
* fix(ci): preserve public-safe Codex scan diagnostics Signed-off-by: Adrien Langou <alangou@nvidia.com> * fix(ci): reject non-regular scanner reports without blocking Signed-off-by: Adrien Langou <alangou@nvidia.com> --------- Signed-off-by: Adrien Langou <alangou@nvidia.com> |
||
|
Simon Scatton
|
fa7dc26930 |
ci(release): add advisory compatibility review (#4200)
* ci(release): add advisory compatibility review Signed-off-by: Simon Scatton <sscatton@nvidia.com> * test(ci): remove compatibility reviewer test harness Signed-off-by: Simon Scatton <sscatton@nvidia.com> * ci(release): run compatibility SDK in the dev shell Signed-off-by: Simon Scatton <sscatton@nvidia.com> * ci(release): trigger a one-shot compatibility review Signed-off-by: Simon Scatton <sscatton@nvidia.com> * ci(release): remove one-shot review trigger Signed-off-by: Simon Scatton <sscatton@nvidia.com> --------- Signed-off-by: Simon Scatton <sscatton@nvidia.com> |
||
|
Jim Meyer
|
3fc93e2827 |
feat(ci): add the maintainer approval decision tools (#3791)
Adds the two stdlib-only helpers that back the maintainer approval gate, ahead of the workflows that call them. check_maintainer_approval.py parses the linked handles out of MAINTAINERS.md, folds a review list to each reviewer's latest decisive position, and exits non-zero unless a maintainer's latest position is an approval. It fails closed when the list yields no handles. alert_maintainer_change.py renders the approver-set delta between two versions of MAINTAINERS.md, and prints nothing when the set is unchanged. Landing these first lets the gate workflow, which reads both the list and the decision logic from main rather than from the pull request ref, actually execute on the pull request that introduces it. Signed-off-by: Jim Meyer <jimeyer@nvidia.com> |
||
|
Evan Lezar
|
e9271cb313 |
feat(gateway): support runtime image env overrides (#3504)
Closes #3502 Resolve trusted OCI runtime images with driver TOML, process environment, and compiled-default precedence across Docker, Podman, and Kubernetes. Signed-off-by: Evan Lezar <elezar@nvidia.com> |
||
|
Eric Curtin
|
adcd28b620 |
fix(install): stop waiting when the gateway service fails (#4143)
Fixes #4040 Signed-off-by: Eric Curtin <eric.curtin@docker.com> |
||
|
Oliver Calder
|
6e865df349 |
feat(snap): ship the standalone prover binary in the snap (#3717)
Signed-off-by: Oliver Calder <oliver.calder@canonical.com> Signed-off-by: Evan Lezar <elezar@nvidia.com> |
||
|
Oliver Calder
|
1ad4e428a6 |
fix(snap): simplify snap hooks (#3988)
* fix(snap): simplify snap hooks
The `post-refresh` hook runs after initial snap installation as well, so
there is no need to call the `install` hook from within the
`post-refresh` hook; instead, the logic can simply be moved into the
`post-refresh` hook directly, and the `install` hook removed.
Also, the existing `install` hook logic looked for an insecure
configuration, and if found, replaced the entire configuration file with
a minimal default in the current format. But OpenShell does that default
behavior without any config file, so we may as well simply remove the
configuration file entirely to keep up-to-date with the current default
behavior. Let OpenShell create a configuration file if it needs to,
rather than auto-create one via the packaging scripts.
Signed-off-by: Oliver Calder <oliver.calder@canonical.com>
* fix(snap): remove the connect-plug-docker hook
The `openshell:docker` is auto-connected to the system `:docker` slot,
so there should not be a need to separately restart the gateway service
when the interface is connected.
For locally-built test snaps which were not published to the store, the
autoconnection is not made, but when the snap is installed, the gateway
will attempt to start anyway and fail to find any available compute
driver, so quickly restart until it hits the systemd start-limit, after
which systemd prevents the service from being started again. If a user
tries to manually connect their locally-built `openshell` snap to the
`:docker` slot, then the `connect-plug-docker` hook runs and triggers a
restart of the gateway, which will usually fail because the start limit
has already been hit. An error in the hook will thus cause the interface
connection to be undone, which is undesirable.
Thus, we can remove this hook entirely, and instead allow interface
connections to succeed as intended. The user still needs to manually
restart the gateway service after making a manual connection (as was the
case previously) and probably needs to `systemctl reset-failed` first,
but at least connection will succeed beforehand so they can proceed with
these steps.
Signed-off-by: Oliver Calder <oliver.calder@canonical.com>
* fix(snap): set refresh-mode: endure again, with manual restart
Return to the previous behavior before commit
|
||
|
Simon ScattonandMrunal Patel
|
5698c4f746 |
fix(ci): qualify protobuf compatibility by release train (#4049)
* feat(ci): detect breaking protobuf changes Compare the proto module against the PR or merge-group base and report Buf violations in Branch Checks. Add local reproduction and fixture coverage. Closes #3794 Signed-off-by: Mrunal Patel <mrunalp@gmail.com> * fix(ci): pin protobuf check container image Signed-off-by: Mrunal Patel <mrunalp@gmail.com> * fix(ci): qualify protobuf compatibility by release train Signed-off-by: Simon Scatton <sscatton@nvidia.com> * refactor(ci): reuse protobuf compatibility action Signed-off-by: Simon Scatton <sscatton@nvidia.com> * refactor(ci): run protobuf checks as a Nix app with one ref Signed-off-by: Simon Scatton <sscatton@nvidia.com> --------- Signed-off-by: Mrunal Patel <mrunalp@gmail.com> Signed-off-by: Simon Scatton <sscatton@nvidia.com> Co-authored-by: Mrunal Patel <mrunalp@gmail.com> |
||
|
Evan Lezar
|
0e8d9e55f9 |
test(e2e): remove schema parity campaign (#3864)
Signed-off-by: Evan Lezar <elezar@nvidia.com> |
||
|
Simon Scatton
|
e21b7fd8cf |
chore(build): remove bundled Z3 support (#3275)
* chore(build): remove bundled Z3 support Signed-off-by: Simon Scatton <sscatton@nvidia.com> Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> * fix(build): preserve vendored Z3 for local gateway artifacts Signed-off-by: Simon Scatton <sscatton@nvidia.com> --------- Signed-off-by: Simon Scatton <sscatton@nvidia.com> Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> |
||
|
Philippe MartinandJohn Myers
|
9cb72baa2e |
feat(docker): support corporate proxy CA bundles (#3549)
* feat(docker): support corporate proxy CA bundles Closes #3545 Validate and stage operator-owned proxy CA bundles for Docker supervisors, add corporate proxy E2E coverage, and document the trust contract. Signed-off-by: Philippe Martin <phmartin@redhat.com> * fix(docker): validate proxy config on startup Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * test(docker): use the E2E workload image for proxy tests Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * test(docker): generate strict corporate proxy certificates Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * test(docker): surface intercepted TLS fixture errors Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * test(docker): drain buffered TLS proxy data Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * test(docker): relay intercepted HTTP deterministically Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> --------- Signed-off-by: Philippe Martin <phmartin@redhat.com> Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> Co-authored-by: John Myers <9696606+johntmyers@users.noreply.github.com> |
||
|
Evan Lezar
|
b77f5ddfc1 |
test(install): support Bash 3.2 mock capture (#3790)
Signed-off-by: Evan Lezar <elezar@nvidia.com> |
||
|
Johnny Greco
|
87929ad130 |
docs: keep page URLs aligned with file paths (#3713)
* docs: align page file names and nav labels with published URLs Signed-off-by: Johnny Greco <jogreco@nvidia.com> * docs: redirect moved dev pages and fix agent guide redirects Signed-off-by: Johnny Greco <jogreco@nvidia.com> * ci(docs): check that page URLs match their file paths Signed-off-by: Johnny Greco <jogreco@nvidia.com> * fix(docs): align navigation checks with repo conventions Signed-off-by: Johnny Greco <jogreco@nvidia.com> * docs: omit historical redirect aliases Signed-off-by: Johnny Greco <jogreco@nvidia.com> * fix(docs): preserve published overview and TypeScript URLs Signed-off-by: Johnny Greco <jogreco@nvidia.com> * fix(docs): redirect unversioned overview and TypeScript URLs Signed-off-by: Johnny Greco <jogreco@nvidia.com> --------- Signed-off-by: Johnny Greco <jogreco@nvidia.com> |
||
|
Johnny Greco
|
8934b74a85 |
fix(docs): sync redirects with versioned snapshots (#3754)
* fix(docs): sync redirects with versioned snapshots Signed-off-by: Johnny Greco <jogreco@nvidia.com> * fix(docs): make redirect validation types explicit Signed-off-by: Johnny Greco <jogreco@nvidia.com> --------- Signed-off-by: Johnny Greco <jogreco@nvidia.com> |