189 Commits

Include renames

Author	SHA1	Message	Date
Drew Newberry	6e69360d0e	refactor(runtime): extract portable driver and isolation foundations (#4106) ... * refactor(runtime): decouple supervisor access and boundary audit validation Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(runtime): keep supervisor process access boundary driven Signed-off-by: Drew Newberry <anewberry@nvidia.com> * feat(cli): support environment values from caller variables Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(runtime): extract driver-independent MXC foundations Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(runtime): defer UI policy support to MXC integration Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(runtime): integrate current policy and SSH launch contracts Signed-off-by: Drew Newberry <anewberry@nvidia.com> * test(runtime): reconcile fixtures with updated main Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(runtime): repair stacked branch CI contracts Signed-off-by: Drew Newberry <anewberry@nvidia.com> * test(runtime): update boundary fixtures for current contracts Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs(runtime): clarify scoped guards for launch inputs and delete Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(runtime): validate foreground attachment intent before launch Signed-off-by: Drew Newberry <anewberry@nvidia.com> --------- Signed-off-by: Drew Newberry <anewberry@nvidia.com>	2026-10-14 03:07:31 +00:00
3 people	e30c938bb1	feat(server): scope mutation locks and bound lock waits (#4147) ... * test(server): add a local PostgreSQL test runner The openshell-server tests that need a real PostgreSQL server are ignored by default and had no shared way to run. The mutation-replay test reads its own OPENSHELL_REPLAY_TEST_DATABASE_URL, so every contributor had to provision a database by hand, and the advisory-lock tests that follow in this series need the same setup. Add mise run test:rust:postgres. It runs every ignored postgres_* test in openshell-server against OPENSHELL_TEST_POSTGRES_URL, or starts a disposable PostgreSQL container with Docker or Podman (CONTAINER_ENGINE selects one) on the image pinned by the Kubernetes e2e fixture and removes it on exit. Tests run one at a time because advisory locks are database-wide. The runner always points the legacy replay variable at the selected database, so an inherited URL cannot send that test to a different server. test:postgres-runner checks that selection with a fake cargo and needs no database or container engine. CI does not run the PostgreSQL tests; TESTING.md documents the task. Part of #3528 Signed-off-by: Emilien Macchi <emacchi@redhat.com> * fix(server): hold the mutation guard while deleting a provider DeleteProvider checked that no sandbox referenced the provider and then deleted the record without holding the sandbox mutation guard. Sandbox create and provider attach take that guard while they write provider references, so one of them could add a reference after the attached sandbox check passed, and the delete then removed a provider that a sandbox spec still named. Take the guard before the attached sandbox check, as provider create and update already do, so the check and the delete run against a stable set of sandbox references, and reject an empty name after authorization but before the guard. A new test holds the guard, attaches the provider while the delete waits, and asserts that the delete fails with FailedPrecondition and leaves the provider in place. Part of #3528 Signed-off-by: Emilien Macchi <emacchi@redhat.com> * fix(server): release staged credentials when a refresh fails early A provider credential refresh stages the minted values under new credential handles before it commits the provider update, and deletes those handles when validation or persistence fails. Two earlier returns skipped that cleanup. The minted expiry was converted to a protobuf timestamp only after staging, so an expiry outside the timestamp range failed the refresh and left the staged values in the credential driver. A failure to acquire the sandbox mutation guard returned the same way. Convert the expiry before staging anything, so a bad value fails before any handle exists, and delete the staged handles when the guard cannot be acquired. A new test refreshes a stored credential with an expiry of i64::MAX and asserts that the call fails, the provider keeps its original handles, and the credential driver holds the same number of values as before. The guard failure path has no test here, because the SQLite guard used by unit tests cannot fail. Part of #3528 Signed-off-by: Emilien Macchi <emacchi@redhat.com> * feat(server): acquire mutation locks on a dedicated PostgreSQL lock pool The cross-object mutation guard took its advisory-lock session from the 10-connection data pool and closed it on every release, so each guarded mutation opened a new database connection, and lock holders competed with their own critical sections for data connections. A lock wait that hit the 10 second lock_timeout surfaced as INTERNAL. Add a lazy, dedicated 4-connection lock pool and a new persistence::mutation_lock module for the lock key, mode and set types. Store::acquire_distributed_mutation_guard now takes a lock set and a deadline, and each lock statement sets lock_timeout to the time left before that deadline, so a cancelled acquisition never waits past it on the server. Returned connections are scrubbed with pg_advisory_unlock_all() and reused, and each return is bounded by 5 seconds so a stalled session cannot keep a pool permit. A lock statement that PostgreSQL ends with SQLSTATE 55P03 fails with a new LockTimeout error, and so does a wait for a lock connection during which every one was checked out at some point, or that started with less than a second left, too little to open one. The API returns it as UNAVAILABLE with reason MUTATION_LOCK_TIMEOUT and a 1 second retry delay, because the mutation lock was not acquired, so the request's guarded writes did not run. Only the guard acquisition classifies 55P03; any other 55P03 stays a database error. A lock connection that PostgreSQL does not open by the deadline despite at least a second to do so (refused, out of connection slots, starting up, or not answering, all of which SQLx retries silently) is a database error, not a lock timeout. Mutations still take the legacy global key exclusively, so old and new replicas keep excluding each other during a rolling upgrade. Each PostgreSQL-backed gateway pod can now open up to 14 database connections (10 data plus 4 lock) instead of 10, and the chart's autoscaling.maxReplicas comment says what its default of 4 needs. The SSH host identity lock stays on a data connection, because sandbox creation takes it while holding its mutation guard and a second lock-pool connection there could exhaust the pool. The provider credential refresh lock stays there too, so a refresh still progresses while every lock connection is in use. Part of #3528 Co-authored-by: divesh <dgude@nvidia.com> Signed-off-by: Emilien Macchi <emacchi@redhat.com> * perf(server): partition mutation locks by workspace and sandbox Every cross-object mutation, and every lifecycle and supervisor update, took one process-wide mutex and, on PostgreSQL, one fleet-wide advisory lock. Unrelated sandboxes and providers therefore serialized across all replicas, and one slow critical section delayed every other mutation in the fleet. Replace that lock with hierarchical intention locks on a global key, one key per workspace and one key per sandbox, each held shared or exclusive. Global settings and policy writers and platform-scope profile writers hold the global key exclusively. Provider and workspace-scoped profile writers, including DeleteProvider and credential refresh, hold their workspace key exclusively. Sandbox writers and supervisor reports hold their sandbox key exclusively under shared global and workspace keys. Lifecycle, driver-watch and reconcile paths take only process-local keys and keep relying on compare-and-swap across replicas. Workspace and sandbox keys are domain-separated SHA-256 prefixes. Each guard takes its local keys and then its advisory locks in ascending order under one 10 second deadline, and no path nests guards. The settings mutex is removed, since the global and sandbox keys cover it. Sandbox settings are keyed by sandbox name, which a sandbox key no longer covers once another sandbox reuses the name, so a settings update returns NOT_FOUND unless its sandbox still holds the name after the read, and ABORTED if the settings row was recreated before the write. The global key keeps its legacy value, so older replicas, which hold it exclusively for every mutation, still exclude new ones during a rolling upgrade. Guards now also give up after 10 seconds on SQLite, and provider refresh reports a guard timeout as UNAVAILABLE instead of INTERNAL. A supervisor report for a missing sandbox returns NOT_FOUND. Startup endpoint-status reconciliation still holds the global key for its whole scan. Part of #3528 Signed-off-by: Emilien Macchi <emacchi@redhat.com> * fix(server): reconcile endpoint status on startup one sandbox at a time Startup endpoint-status reconciliation held the global mutation key exclusively for its whole scan. On PostgreSQL that blocked every guarded mutation on every replica until the starting gateway had walked all sandboxes, and the scan first waited behind every in-flight sandbox mutation. It also paged with an offset, so sandboxes deleted mid-scan shifted rows between pages, and a write that raced the scan (a Conflict from a lifecycle update on another replica, or a sandbox deleted after it was listed) failed gateway startup. Walk the sandboxes with keyset paging instead, and reset each candidate under its own sandbox-scoped guard, up to four at a time to match the lock pool. A candidate with a fresh shared owner is skipped before any guard is taken. Under the guard the reconciliation re-checks the owner, re-reads the sandbox, skips it when it is gone or no longer carries endpoint status, and writes against the version it just read. A Conflict or a database error at write time, which is how a row deleted after the re-read surfaces, is retried with a fresh read, at most five attempts per sandbox. A lock timeout on the guard is retried the same way, so a holder of the sandbox, its workspace or the global key that releases within the retry window cannot stop a replica from starting. No lock-timeout retry starts once the scan has run for three lock timeouts, so a scan stuck behind a long holder fails within about 40 seconds, inside the chart's 60-second startup probe, and logs the sandbox it gave up on. Store::list_all_messages loses its only caller and is removed. Reconciliation stays fail-closed: a holder that outlasts the retry window, an owner lookup failure, or a sandbox that keeps changing still fails startup, so stale endpoint success is never served. User-visible change: a restarting gateway no longer stalls mutations of unrelated sandboxes and providers on its peers while it reconciles. Part of #3528 Signed-off-by: Emilien Macchi <emacchi@redhat.com> * perf(server): skip the endpoint-status lock when a session was replaced When a supervisor session ends, the gateway resets that sandbox's tool server endpoint evidence unless a replacement session already owns endpoint observation. The replacement check ran only after the reset had read the sandbox and taken its sandbox mutation guard, so a reset whose supervisor had already reconnected still queued behind other mutations of that sandbox, and on PostgreSQL it used a lock-pool connection, only to find it had nothing to do. Check for a replacement first: a live local session, or a fresh owner record on a peer. When one exists, return without reading the sandbox or taking the guard. Otherwise take the guard and repeat the same check under it, because a replacement's pre-acknowledgement reset runs under the same sandbox key and resetting after it would wipe the replacement's fresh evidence. Both checks share one helper, so they cannot drift apart. When no replacement exists, the reset costs one extra owner-row read. This helps most when a replica notices a dead stream late, after the supervisor has already reconnected elsewhere (a network partition or a keepalive expiry). Deferring to a replacement leaves the old session's evidence to the replacement's pre-acknowledgement reset, which can time out on its bounded mutation lock wait. That reset's failure path removed the replacement and released its owner record but scheduled no invalidation, so the old session's success evidence could outlive both sessions. An old session that ended after its replacement registered already deferred this way, and the unguarded check now makes every disconnect that finds the replacement defer too. When the reset fails while the replacement is still current, also schedule the retried invalidation that follows a session end. That failure path keeps the lifecycle demotion from abandon_session_setup, but runs it in the spawned task before the retry and only while this session is still current, so a timed-out reset returns promptly. Part of #3528 Co-authored-by: divesh <dgude@nvidia.com> Signed-off-by: Emilien Macchi <emacchi@redhat.com> * test(server): cover staged credential cleanup on a lock timeout A provider credential refresh deletes its staged handles when it cannot acquire the provider mutation guard, but that path had no test: the SQLite guard used by unit tests could not fail. Mutation locks now acquire under a deadline that tests can shorten, so the timeout path can be exercised directly. The new test shortens the lock deadline to 50 ms, holds the workspace mutation guard, and refreshes a provider that already stores an AWS access key. The refresh returns UNAVAILABLE with the MUTATION_LOCK_TIMEOUT reason, the provider keeps its original handle, which still resolves to the old key, and the credential driver holds the same number of values as before the refresh. Part of #3528 Signed-off-by: Emilien Macchi <emacchi@redhat.com> * feat(server): export mutation lock wait and timeout metrics A handler mutation guard can now wait up to 10 seconds for its keys and then fail with UNAVAILABLE, but nothing showed operators how long guards waited or how often they gave up, short of reading warning logs on each replica. Export openshell_server_mutation_lock_wait_seconds, a histogram of the time one guard took to acquire all of its keys (local registry plus PostgreSQL advisory locks), recorded on success, and openshell_server_mutation_lock_timeouts_total, a counter of guard acquisitions that timed out. Both carry a bounded scope label: global, workspace or sandbox. The platform scope, whose workspace is empty, counts as global. The histogram uses the same 1 ms to 15 s buckets as the relay and peer histograms, and the timeout counter starts at 0 for every scope, so rate() works on an idle replica. The timeout warning now also logs the scope. A guard that fails for another reason, such as a lock connection PostgreSQL did not open, is not counted as a timeout and logs a "mutation lock acquisition failed" warning instead. Lifecycle, driver-watch and reconcile paths take only process-local locks and never time out, so they are not recorded. Part of #3528 Signed-off-by: Emilien Macchi <emacchi@redhat.com> * test(server): add PostgreSQL mutation lock integration tests The scoped mutation locks were only unit-tested on SQLite, where the database layer is a no-op. Add ignored postgres_* tests that run the advisory-lock layer against a real PostgreSQL through mise run test:rust:postgres, each in its own disposable schema (TestSchema). persistence/mutation_lock_pg_tests.rs uses two stores on one database as two replicas. It covers disjoint sandbox scopes holding at once, a workspace key blocking only sandboxes in that workspace, the global key blocking every scope, and exclusion in both directions against a raw holder of the legacy global key, whose 55P03 stays a database error outside a guard acquisition. It checks that timed-out and cancelled acquisitions release the keys they already took, that a released lock connection returns to the pool without residual locks, that a release stalled on the network closes its session and the pool recovers, and that the lock pool never grows past its size. It also checks that a full lock pool times out as lock contention, while a lock connection that PostgreSQL never answers fails as a database error. compute/mutation_guard.rs runs the random scope mix across two PostgreSQL runtimes and checks that a by-id guard waits for a workspace writer on the other replica. The provider refresh lock test moves onto the runner too, and checks that a refresh proceeds while every lock connection is held. A bench_postgres_* capacity check measures a paced burst of 500 reconnects 12 ms apart into one replica and fails when the p99 lock wait reaches a fifth of the lock timeout. Machine load moves those waits, so mise run test:rust:postgres skips it and mise run test:rust:postgres:bench runs it alone, through a new OPENSHELL_TEST_POSTGRES_PREFIX setting in the runner. The tests add test-only helpers: the lock pool's size and idle count and the backend pid of a held lock session. Nothing changes outside tests and the test runner. Part of #3528 Co-authored-by: divesh <dgude@nvidia.com> Signed-off-by: Emilien Macchi <emacchi@redhat.com> * ci: run the PostgreSQL-backed server tests Branch Checks now runs the postgres_* tests through the same runner developers use, on an x86_64 Linux runner with Docker. They cover lock scope exclusion, interop with replicas still on the old global key, cancellation, lock-pool bounds and connection failures, which no other CI lane checks directly. The load-sensitive bench_postgres_* capacity checks stay local. Part of #3528 Signed-off-by: Emilien Macchi <emacchi@redhat.com> * docs: document scoped mutation locks and lock-pool sizing In the High Availability guide, size PostgreSQL for 14 connections per gateway pod, 10 for data and 4 for the new lock pool, so three or more replicas now exceed what the PostgreSQL default leaves, and keep headroom for lock sessions that cancelled requests leave behind. Explain which changes lock a sandbox, a workspace, or the whole fleet, and that a lock wait over 10 seconds returns a retryable UNAVAILABLE with reason MUTATION_LOCK_TIMEOUT. A new upgrade subsection says to raise max_connections to the 14-per-pod size before upgrading, because the rollout already runs new pods at that count, and warns that older replicas keep serializing every mutation until the rollout finishes. Monitor Capacity gains the lock metrics and their queries, and names slow driver, credential, middleware, or profile source calls as another cause of lock waits. List the lock wait histogram and timeout counter, with their scope label, on the Gateway Metrics page. The counter also counts background work and leaves out lock connections that PostgreSQL does not open although at least a second of the wait remained. In the cluster debugging skill, replace the single advisory-lock paragraph with the lock levels and pool, what each timeout detail means, how a lock connection that PostgreSQL does not open shows up, when terminating a lock backend is safe, a pg_locks query that shows holders and waiters, and how to spot the global key. The per-pod greps also read the timeout counter, and new rows cover lock timeouts and lock connections that PostgreSQL does not open. Part of #3528 Signed-off-by: Emilien Macchi <emacchi@redhat.com> * feat(server): check mutation lock ordering in debug builds The mutation lock ordering rules were prose only, so a nested guard on a different key passed every test and could only deadlock under contention. Debug builds now register each held guard with the Tokio task (or the thread, outside a task) that acquired it, and panic before waiting when a local key, a blocking lifecycle gate or the SSH identity lock would break rules 1, 4 or 5. Startup endpoint-status reconciliation and the tests that hold a guard while driving guarded code run under lock_order::branch(), which marks deliberate concurrency within one task. Release builds compile the check out. Part of #3528 Signed-off-by: Emilien Macchi <emacchi@redhat.com> * fix(server): do not hold gateway shutdown for the endpoint-status retry When a replacement supervisor session failed its endpoint-status reset, the spawned cleanup task kept the session lifetime guard through retry_endpoint_status_after_supervisor_disconnect. That retry only ends once the reset is durable, so a gateway shutdown during a database or lock outage waited the full 10 s and reported ownership cleanup as incomplete. The guard now covers only the lifecycle demotion, as in finish_supervisor_session, and a SQLite test checks that shutdown completes while the retry is still blocked. Part of #3528 Signed-off-by: Emilien Macchi <emacchi@redhat.com> * refactor(server): make the platform profile lock scope explicit Platform-scope provider profile writers (import, update, delete) pass an empty workspace on purpose and need X(global), because they validate sandboxes in every workspace. MutationScope::profiles() now names that escalation and picks Global for the platform scope, instead of relying on MutationScope::Workspace("") to mean it. An empty workspace name now trips a debug assertion, and release builds keep X(global) as a fail-safe that over-locks rather than under-locks. Part of #3528 Signed-off-by: Emilien Macchi <emacchi@redhat.com> * perf(server): amortize local mutation lock table sweeps LocalMutationLocks swept its whole table on every key lookup, so a burst of N concurrent acquisitions on one replica cost O(N^2) under a blocking mutex. The table is now swept only when a new key grows it past the larger of 64 and twice the live entries left by the last sweep, which makes acquisition amortized O(1). Released entries, including those of cancelled waiters, stay bounded at about twice the peak live set. The lifecycle gate registry keeps its per-call sweep. Part of #3528 Signed-off-by: Emilien Macchi <emacchi@redhat.com> * feat(server): make the database pool ceiling configurable Both stores hardcoded their pool ceiling: 10 connections for Postgres, 5 for on-disk SQLite. One pool is shared by every database-backed RPC, so that number is also the gateway's ceiling on concurrent database work. Once every connection is checked out, callers queue on acquire and sqlx logs "time to acquire exceeded slow threshold"; sandbox creates then time out and are retried, which adds load rather than shedding it. The right ceiling is deployment-specific — it depends on how many gateway replicas share the database and what max_connections the server itself allows — so it cannot be a single number baked into the binary. Expose it on the same config surface as the rest of the gateway's settings: --db-max-connections, OPENSHELL_DB_MAX_CONNECTIONS, and the TOML key database_max_connections, resolved in that precedence order. Helm deployments set the key through gatewayConfig. Omitting it keeps each backend's previous value, so existing deployments do not move. Values below 1 are rejected rather than silently replaced by the default: a zero pool would block every acquire, and quietly ignoring a typo would reproduce the ceiling the operator is trying to lift. An in-memory SQLite database stays pinned to one connection, since the database lives inside that connection. Refs: #2561 Signed-off-by: Bryce Wilkinson <22760097+bjw123@users.noreply.github.com> (cherry picked from commit 33517b32bd) [emacchi: dropped the server.dbMaxConnections Helm value; operators set the key through gatewayConfig] Signed-off-by: Emilien Macchi <emacchi@redhat.com> * feat(server): make the mutation lock pool size configurable The mutation lock pool was fixed at 4 connections per replica. That number rests on one reconnect-burst bench rather than scale testing, so a gateway that serves many more sandboxes needs a way to raise it without a rebuild. Add --db-lock-max-connections, OPENSHELL_DB_LOCK_MAX_CONNECTIONS and the database_lock_max_connections gateway.toml key next to the data pool knob, resolved in the same order; Helm deployments set it through gatewayConfig. The default stays 4, zero is rejected, SQLite ignores it with a warning, and the gateway logs both pool sizes at startup. PostgreSQL now needs at least 2 data connections, because the SSH identity lock keeps one while it queries the pool. The connection sizing docs, the HA guide formula and the debug skill now count data plus lock connections. Part of #3528 Signed-off-by: Emilien Macchi <emacchi@redhat.com> * feat(server): export mutation lock errors and lock connections in use A mutation lock acquisition that failed without timing out, such as a lock connection PostgreSQL never opened or a failed lock statement, only logged a warning, and readiness stays healthy because it pings the data pool. Count those failures in openshell_server_mutation_lock_errors_total{scope}, kept apart from timeouts because they point at PostgreSQL capacity or connectivity rather than contention. Export openshell_server_mutation_lock_connections_in_use as a gauge slot held by each checked-out lock connection, next to openshell_server_mutation_lock_connections_capacity, which reads the configured lock pool size; both exist only with PostgreSQL. The metrics catalog, the HA guide's signals and PromQL examples, and the debug skill cover the new series. Part of #3528 Signed-off-by: Emilien Macchi <emacchi@redhat.com> * feat(server): time mutation lock holds and test a stalled holder Nothing bounds how long a mutation guard is held, so a holder stuck in a compute driver, credential backend or middleware call keeps its scope locked while waiters time out. Each guard now records its hold in openshell_server_mutation_lock_hold_seconds{scope} when it is released, and logs a warning when the hold outlasted the lock wait timeout, so operators can find the replica and scope behind those timeouts. A PostgreSQL test with a stalled holder pins the current contract: waiters on its sandbox, workspace and global scopes fail by their deadline, other scopes proceed, the holder keeps its key, and a full lock pool on one replica leaves the other working. The metrics catalog, the HA guide and the debug skill describe the new signals, which observe holds without bounding them. Part of #3528 Signed-off-by: Emilien Macchi <emacchi@redhat.com> --------- Signed-off-by: Emilien Macchi <emacchi@redhat.com> Co-authored-by: divesh <dgude@nvidia.com> Co-authored-by: Bryce Wilkinson <22760097+bjw123@users.noreply.github.com>	2026-10-13 23:34:26 +00:00
Simon Scatton	d3e6a35e47	ci(compatibility): review PR breaks and acknowledge release risks (#4488) ... * ci(compatibility): review PR breaks and acknowledge release risks Signed-off-by: Simon Scatton <sscatton@nvidia.com> * fix(ci): use matching compatibility review tooling Signed-off-by: Simon Scatton <sscatton@nvidia.com> * ci(compatibility): scope Rust SDK review to public exports Signed-off-by: Simon Scatton <sscatton@nvidia.com> * ci(compatibility): account for Go struct field additions Signed-off-by: Simon Scatton <sscatton@nvidia.com> * ci(compatibility): restrict egress and fix report retries Signed-off-by: Simon Scatton <sscatton@nvidia.com> --------- Signed-off-by: Simon Scatton <sscatton@nvidia.com>	2026-10-13 19:13:19 +00:00
Florent BENOIT	cf6ec17785	refactor(vm): source runtime pins only from pins.env (#4373) ... The VM runtime scripts sourced pins.env optionally and duplicated the LIBKRUN_REF and UMOCI_VERSION defaults, so every bump had to update the same value in several files. Require pins.env, drop the duplicated defaults, and fail on an invalid LIBKRUN_REF instead of silently building upstream HEAD. Environment overrides still take precedence through the defaults in pins.env. Signed-off-by: Florent Benoit <fbenoit@redhat.com>	2026-10-13 11:56:47 +00:00
Evan Lezar	2ffc3d2b4d	ci: require accepted issues for community pull requests (#4340) ... Evaluate issue acceptance using trusted default-branch code and the PR author's repository permissions. Report the native workflow job result, protect enforcement files, and align PR creation and review guidance. Closes #4339 Signed-off-by: Evan Lezar <elezar@nvidia.com>	2026-10-13 11:44:44 +00:00
Oliver Calder	16c8ca5c43	fix(docs): use lowercase "snap" as it is not a proper noun (#4387) ... Signed-off-by: Oliver Calder <oliver.calder@canonical.com>	2026-10-12 19:02:26 +00:00
Evan Lezar	78975ea220	chore(test-guest): remove snap reproduction helper (#4368) ... Signed-off-by: Evan Lezar <elezar@nvidia.com>	2026-10-09 12:38:45 +00:00
Oliver Calder	82bc756545	feat(snap): run gateway as a user service by default (#4099) ... * feat(snap): run gateway as a user service by default Replace the existing system `gateway` service with separate `user-gateway` and `system-gateway` services. For new installs, only the `user-gateway` service is enabled. For existing installs, only the `system-gateway` service is enabled. Existing installs continue to have a one-time migration which removes legacy insecure configurations. It is up to `install.sh` or users to manually copy mTLS credentials from the root-owned `$SNAP_COMMON/tls` to the invoking user's OpenShell snap directory. This is explained in the snap description in `snapcraft.yaml`, visible in the Snap Store listing and via `snap info openshell`. Service enablement is now managed by a new snap configuration option named `gateway-mode`, so users can switch from one mode to another via e.g. `sudo snap set openshell gateway-mode=user`. The `install` and `post-refresh` hooks select which service to start by setting this mode. If the `gateway-mode` is already set, then we know the one-time migration has already taken place. Signed-off-by: Oliver Calder <oliver.calder@canonical.com> * fix(snap): simplify the snap gateway wrapper Signed-off-by: Oliver Calder <oliver.calder@canonical.com> * fix(snap): use `snap start --user` when running as non-root Signed-off-by: Oliver Calder <oliver.calder@canonical.com> * fix(snap): ensure install.sh waits for the correct docker daemon Signed-off-by: Oliver Calder <oliver.calder@canonical.com> * feat(snap): document how to safely switch between gateway modes New installs run the gateway as a user service by default, while upgrades of existing installs from prior to the user gateway continue to use the existing system gateway. The user and system gateway are fundamentally different services and there is no way to migrate state between them. However, it may be desirable for a user to run the gateway as a system service after a fresh install. To support this, this commit adds documentation for how to safely transition from the user gateway mode to the system gateway mode, and vice versa. Also, clarifies when it is necessary to manually copy TLS credentials from the system gateway state directory to the user's home directory: when upgrading a pre-TLS snap installation, or when switching from user to system gateway mode. When upgrading an existing install with an HTTPS gateway running as a system service, copying credentials again is not necessary. Signed-off-by: Oliver Calder <oliver.calder@canonical.com> --------- Signed-off-by: Oliver Calder <oliver.calder@canonical.com>	2026-10-09 11:39:08 +00:00
Gaizka MenendezandTaylor Mutch	fe3942f2da	feat(helm): migrate gateway configuration to gatewayConfig (#3384) ... * feat(helm): add generic gateway TOML serializer Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * docs(architecture): define Helm gateway configuration boundary Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * feat(helm): define default gateway configuration map Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * refactor(helm): render gateway config from values map Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * test(helm): cover generic gateway TOML rendering Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * feat(helm): protect gateway config secret boundary Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * docs(helm): classify legacy gateway configuration values Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * refactor(helm): derive dual-use resources from gateway config Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * test(helm): migrate gateway config scenarios Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * refactor(helm): derive credential resources from gateway config Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * refactor(helm): migrate gateway config overlays Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * refactor(helm): remove gateway configuration shadow values Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * refactor(helm): align runtime config with resources Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * docs(helm): add gateway config migration guide Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * test(helm): validate rendered gateway config Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * test(helm): cover config resource coherence Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * fix(helm): make Kubernetes E2E deployable Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * fix(e2e): pass host aliases through gateway config Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * docs(e2e): reference gateway config host alias Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * fix(helm): enforce gateway resource ownership Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * test(e2e): cover chart host gateway input Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * fix(helm): address gateway config review findings Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * fix(helm): repair RFC 0012 migration integration Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * fix(e2e): align Kubernetes parity with RFC 0012 Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * fix(helm): complete gateway config migration Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * chore(helm): add SPDX header to TOML template * fix(helm): preserve legacy gateway configuration aliases Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * fix(helm): complete legacy gateway config compatibility Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * feat(helm): support gateway config arrays of tables * fix(helm): preserve sandbox identity aliases * fix(helm): address gateway config review findings Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * fix(helm): preserve gateway config migration settings Signed-off-by: Taylor Mutch <taylormutch@gmail.com> * test(helm): validate extension arrays in CI values Signed-off-by: Taylor Mutch <taylormutch@gmail.com> * fix(helm): restore unconditional SPDX headers Signed-off-by: Taylor Mutch <taylormutch@gmail.com> * fix(helm): restore route SPDX header placement Signed-off-by: Taylor Mutch <taylormutch@gmail.com> * docs(helm): simplify configuration and migration examples Signed-off-by: Taylor Mutch <taylormutch@gmail.com> * docs(helm): remove redundant flag explanations Signed-off-by: Taylor Mutch <taylormutch@gmail.com> * docs(helm): move chart configuration guidance to Kubernetes setup Signed-off-by: Taylor Mutch <taylormutch@gmail.com> --------- Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> Signed-off-by: Taylor Mutch <taylormutch@gmail.com> Co-authored-by: Taylor Mutch <taylormutch@gmail.com>	2026-10-08 18:39:59 +00:00
Florent BENOIT	616024f79e	fix(vm): bump libkrun to v1.19.6 (#4282) ... ## Summary Bump the pinned libkrun from v1.19.4 to v1.19.6, the latest upstream release. v1.19.5 includes libkrun/libkrun@d2d8dd6, which removes refused vsock connections without waiting for the 5 s reaper, so VM sandbox startup no longer stalls on "Waiting for VM supervisor". ## Related Issue Closes #4259 ## Changes - libkrun: v1.19.4 (728df812) -> v1.19.6 (227b2de6) - Update LIBKRUN_REF in pins.env and the fallback defaults in build-libkrun.sh and build-libkrun-macos.sh - libkrunfw stays on v5.6.1 Other notable upstream fixes (v1.19.5): vsock proxy release after clean shutdown, macOS virtio-fs xattr handling, x86_64 initrd kept below the MMIO gap, PIT missed-tick reinjection disabled, macOS VMs with >64GB RAM. v1.19.6 adds an API to set the render server fd. No API removals. ## Testing - libkrun v1.19.6 builds on macOS arm64 with `make BLK=1 NET=1` Signed-off-by: Florent Benoit <fbenoit@redhat.com>	2026-10-08 17:32:47 +00:00
alangou	249c6e727f	fix(ci): retain sanitized Codex scan diagnostics on failure (#4304) ... * fix(ci): preserve public-safe Codex scan diagnostics Signed-off-by: Adrien Langou <alangou@nvidia.com> * fix(ci): reject non-regular scanner reports without blocking Signed-off-by: Adrien Langou <alangou@nvidia.com> --------- Signed-off-by: Adrien Langou <alangou@nvidia.com>	2026-10-07 23:25:28 +00:00
Simon Scatton	fa7dc26930	ci(release): add advisory compatibility review (#4200) ... * ci(release): add advisory compatibility review Signed-off-by: Simon Scatton <sscatton@nvidia.com> * test(ci): remove compatibility reviewer test harness Signed-off-by: Simon Scatton <sscatton@nvidia.com> * ci(release): run compatibility SDK in the dev shell Signed-off-by: Simon Scatton <sscatton@nvidia.com> * ci(release): trigger a one-shot compatibility review Signed-off-by: Simon Scatton <sscatton@nvidia.com> * ci(release): remove one-shot review trigger Signed-off-by: Simon Scatton <sscatton@nvidia.com> --------- Signed-off-by: Simon Scatton <sscatton@nvidia.com>	2026-10-07 14:12:05 +00:00
Jim Meyer	3fc93e2827	feat(ci): add the maintainer approval decision tools (#3791) ... Adds the two stdlib-only helpers that back the maintainer approval gate, ahead of the workflows that call them. check_maintainer_approval.py parses the linked handles out of MAINTAINERS.md, folds a review list to each reviewer's latest decisive position, and exits non-zero unless a maintainer's latest position is an approval. It fails closed when the list yields no handles. alert_maintainer_change.py renders the approver-set delta between two versions of MAINTAINERS.md, and prints nothing when the set is unchanged. Landing these first lets the gate workflow, which reads both the list and the decision logic from main rather than from the pull request ref, actually execute on the pull request that introduces it. Signed-off-by: Jim Meyer <jimeyer@nvidia.com>	2026-10-07 09:37:05 +00:00
Evan Lezar	e9271cb313	feat(gateway): support runtime image env overrides (#3504) ... Closes #3502 Resolve trusted OCI runtime images with driver TOML, process environment, and compiled-default precedence across Docker, Podman, and Kubernetes. Signed-off-by: Evan Lezar <elezar@nvidia.com>	2026-10-05 12:23:13 +00:00
Eric Curtin	adcd28b620	fix(install): stop waiting when the gateway service fails (#4143) ... Fixes #4040 Signed-off-by: Eric Curtin <eric.curtin@docker.com>	2026-10-03 19:59:57 +00:00
Oliver Calder	6e865df349	feat(snap): ship the standalone prover binary in the snap (#3717) ... Signed-off-by: Oliver Calder <oliver.calder@canonical.com> Signed-off-by: Evan Lezar <elezar@nvidia.com>	2026-10-02 11:51:56 +00:00
Oliver Calder	1ad4e428a6	fix(snap): simplify snap hooks (#3988) ... * fix(snap): simplify snap hooks The `post-refresh` hook runs after initial snap installation as well, so there is no need to call the `install` hook from within the `post-refresh` hook; instead, the logic can simply be moved into the `post-refresh` hook directly, and the `install` hook removed. Also, the existing `install` hook logic looked for an insecure configuration, and if found, replaced the entire configuration file with a minimal default in the current format. But OpenShell does that default behavior without any config file, so we may as well simply remove the configuration file entirely to keep up-to-date with the current default behavior. Let OpenShell create a configuration file if it needs to, rather than auto-create one via the packaging scripts. Signed-off-by: Oliver Calder <oliver.calder@canonical.com> * fix(snap): remove the connect-plug-docker hook The `openshell:docker` is auto-connected to the system `:docker` slot, so there should not be a need to separately restart the gateway service when the interface is connected. For locally-built test snaps which were not published to the store, the autoconnection is not made, but when the snap is installed, the gateway will attempt to start anyway and fail to find any available compute driver, so quickly restart until it hits the systemd start-limit, after which systemd prevents the service from being started again. If a user tries to manually connect their locally-built `openshell` snap to the `:docker` slot, then the `connect-plug-docker` hook runs and triggers a restart of the gateway, which will usually fail because the start limit has already been hit. An error in the hook will thus cause the interface connection to be undone, which is undesirable. Thus, we can remove this hook entirely, and instead allow interface connections to succeed as intended. The user still needs to manually restart the gateway service after making a manual connection (as was the case previously) and probably needs to `systemctl reset-failed` first, but at least connection will succeed beforehand so they can proceed with these steps. Signed-off-by: Oliver Calder <oliver.calder@canonical.com> * fix(snap): set refresh-mode: endure again, with manual restart Return to the previous behavior before commit a67567e58, where the gateway is not stopped before refreshes. The `post-refresh` hook now restarts the gateway if the TLS configuration was corrected, so we don't have to enforce restarting the gateway on every refresh even when not necessary. Thus, set `refresh-mode: endure`, and let the hook decide when the gateway needs to be restarted. Signed-off-by: Oliver Calder <oliver.calder@canonical.com> * fix(snap): update docs and tests to reflect snap hook changes Signed-off-by: Oliver Calder <oliver.calder@canonical.com> * docs(snap): remove verbose explanation of snap gateway refresh behavior Signed-off-by: Oliver Calder <oliver.calder@canonical.com> --------- Signed-off-by: Oliver Calder <oliver.calder@canonical.com>	2026-10-01 15:10:38 +00:00
Simon ScattonandMrunal Patel	5698c4f746	fix(ci): qualify protobuf compatibility by release train (#4049) ... * feat(ci): detect breaking protobuf changes Compare the proto module against the PR or merge-group base and report Buf violations in Branch Checks. Add local reproduction and fixture coverage. Closes #3794 Signed-off-by: Mrunal Patel <mrunalp@gmail.com> * fix(ci): pin protobuf check container image Signed-off-by: Mrunal Patel <mrunalp@gmail.com> * fix(ci): qualify protobuf compatibility by release train Signed-off-by: Simon Scatton <sscatton@nvidia.com> * refactor(ci): reuse protobuf compatibility action Signed-off-by: Simon Scatton <sscatton@nvidia.com> * refactor(ci): run protobuf checks as a Nix app with one ref Signed-off-by: Simon Scatton <sscatton@nvidia.com> --------- Signed-off-by: Mrunal Patel <mrunalp@gmail.com> Signed-off-by: Simon Scatton <sscatton@nvidia.com> Co-authored-by: Mrunal Patel <mrunalp@gmail.com>	2026-10-01 14:00:45 +00:00
Evan Lezar	0e8d9e55f9	test(e2e): remove schema parity campaign (#3864) ... Signed-off-by: Evan Lezar <elezar@nvidia.com>	2026-10-01 13:17:20 +00:00
Simon Scatton	e21b7fd8cf	chore(build): remove bundled Z3 support (#3275) ... * chore(build): remove bundled Z3 support Signed-off-by: Simon Scatton <sscatton@nvidia.com> Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> * fix(build): preserve vendored Z3 for local gateway artifacts Signed-off-by: Simon Scatton <sscatton@nvidia.com> --------- Signed-off-by: Simon Scatton <sscatton@nvidia.com> Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>	2026-10-01 12:08:33 +00:00
Philippe MartinandJohn Myers	9cb72baa2e	feat(docker): support corporate proxy CA bundles (#3549) ... * feat(docker): support corporate proxy CA bundles Closes #3545 Validate and stage operator-owned proxy CA bundles for Docker supervisors, add corporate proxy E2E coverage, and document the trust contract. Signed-off-by: Philippe Martin <phmartin@redhat.com> * fix(docker): validate proxy config on startup Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * test(docker): use the E2E workload image for proxy tests Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * test(docker): generate strict corporate proxy certificates Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * test(docker): surface intercepted TLS fixture errors Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * test(docker): drain buffered TLS proxy data Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * test(docker): relay intercepted HTTP deterministically Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> --------- Signed-off-by: Philippe Martin <phmartin@redhat.com> Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> Co-authored-by: John Myers <9696606+johntmyers@users.noreply.github.com>	2026-09-29 05:15:02 +00:00
Evan Lezar	b77f5ddfc1	test(install): support Bash 3.2 mock capture (#3790) ... Signed-off-by: Evan Lezar <elezar@nvidia.com>	2026-09-28 20:41:52 +00:00
Johnny Greco	87929ad130	docs: keep page URLs aligned with file paths (#3713) ... * docs: align page file names and nav labels with published URLs Signed-off-by: Johnny Greco <jogreco@nvidia.com> * docs: redirect moved dev pages and fix agent guide redirects Signed-off-by: Johnny Greco <jogreco@nvidia.com> * ci(docs): check that page URLs match their file paths Signed-off-by: Johnny Greco <jogreco@nvidia.com> * fix(docs): align navigation checks with repo conventions Signed-off-by: Johnny Greco <jogreco@nvidia.com> * docs: omit historical redirect aliases Signed-off-by: Johnny Greco <jogreco@nvidia.com> * fix(docs): preserve published overview and TypeScript URLs Signed-off-by: Johnny Greco <jogreco@nvidia.com> * fix(docs): redirect unversioned overview and TypeScript URLs Signed-off-by: Johnny Greco <jogreco@nvidia.com> --------- Signed-off-by: Johnny Greco <jogreco@nvidia.com>	2026-09-28 02:08:54 +00:00
Johnny Greco	8934b74a85	fix(docs): sync redirects with versioned snapshots (#3754) ... * fix(docs): sync redirects with versioned snapshots Signed-off-by: Johnny Greco <jogreco@nvidia.com> * fix(docs): make redirect validation types explicit Signed-off-by: Johnny Greco <jogreco@nvidia.com> --------- Signed-off-by: Johnny Greco <jogreco@nvidia.com>	2026-09-28 00:27:56 +00:00
Drew Newberry	d1a19c70ee	fix(homebrew): overwrite generated gateway config during migration (#3752) ... Closes #3746 Use Homebrew atomic_write for exact legacy config migrations and keep the first-install write distinct in the formula test. Signed-off-by: Drew Newberry <anewberry@nvidia.com>	2026-09-27 18:56:08 +00:00
Drew Newberry	a67567e583	fix(snap): require mTLS for the snap gateway (#3726) ... * fix(snap): require mTLS for the snap gateway Replace the installer opt-in with an authenticated snap gateway. The wrapper no longer forces plaintext, so the gateway serves TLS from the bundle it already generates in $SNAP_COMMON/tls. The install hook writes a config that enables mTLS user auth instead of unauthenticated access, and a new post-refresh hook migrates the exact legacy default on existing installs. install.sh waits for the gateway, detects whether it serves TLS, copies the client bundle into the target user's snap state directory, and registers the gateway over HTTPS. Older plaintext snap revisions still register over HTTP with a warning. The release canary asserts mTLS auth and HTTPS registration. Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(snap): pass config preflight and detect the mTLS gateway reliably An explicit [openshell.gateway.mtls_auth] table fails config preflight, which validates mTLS auth before the local TLS bundle supplies the client CA. Write a default that pins the Docker driver instead; with the wrapper's TLS bundle the gateway requires client certificates and enables mTLS user auth automatically, as the native packages do. The mTLS gateway rejects TLS handshakes without a client certificate, and it still answers plaintext loopback HTTP for sandbox service routing, so the installer could misdetect it as a legacy plaintext gateway. Probe HTTPS with the root-owned client bundle, and treat a gateway as legacy only when a plaintext gRPC Health call succeeds. Signed-off-by: Drew Newberry <anewberry@nvidia.com> * chore(snap): simplify install hook comment Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(snap): migrate insecure gateway configs on refresh Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(snap): simplify mTLS detection and config migration Detect the mTLS snap from the installed revision's post-refresh hook instead of probing plaintext gRPC, and drop the scheme global. Remove the installer's pre-hook config fallback, which is dead now that every channel ships the install hook and which wrote the insecure default. Give the install hook a single write path with a simple backup name, and shorten the manual client certificate steps. Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(snap): stop keeping a copy of replaced insecure configs Signed-off-by: Drew Newberry <anewberry@nvidia.com> * feat(install): make the snap an opt-in install method Stop selecting the OpenShell snap just because the snap command exists. Linux installs default to the Debian or RPM package; OPENSHELL_INSTALL_METHOD=snap (or deb, rpm) selects the package explicitly. Hosts that already have the OpenShell snap keep refreshing it rather than gaining a second gateway on the same port. The release canary and snap repro script opt in explicitly. Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(snap): let the gateway auto-detect its compute driver Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(snap): restart the gateway after refresh Published revisions use refresh-mode: endure, and snapd honors the old revision's setting during a refresh, so the plaintext gateway kept running with the migrated config unused until a manual restart. Restart the gateway from the post-refresh hook so the mTLS config takes effect immediately. Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs(snap): drop refresh notes from the snap description Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs(snap): trim snap refresh notes from installation docs Signed-off-by: Drew Newberry <anewberry@nvidia.com> --------- Signed-off-by: Drew Newberry <anewberry@nvidia.com>	2026-09-26 01:50:02 +00:00
Piotr Mlocek	6f596828ad	docs(fern): publish ordered version snapshots (#3721) ... * docs(fern): publish and order versioned release docs Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> * docs(fern): remove version availability badges Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> * docs(fern): keep version badges optional Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> --------- Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>	2026-09-25 19:56:05 +00:00
Drew Newberry	08548713c5	fix(install): honor pinned releases and speed up prerelease discovery (#3681) ... * fix(install): use native packages for prereleases and speed up discovery Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(install): honor pinned versions and guard Snap migration Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(install): omit Snap migration guard Signed-off-by: Drew Newberry <anewberry@nvidia.com> --------- Signed-off-by: Drew Newberry <anewberry@nvidia.com>	2026-09-24 17:57:42 -07:00
krishicks	4cd1351e18	fix(test): use portable file mode checks in snap installer tests (#3670) ... The snap installer tests added in #3656 checked the gateway config mode with GNU stat -c, which BSD stat on macOS rejects, so mise run ci failed locally on macOS. Check the mode with find -perm instead, which matches the exact mode on both GNU and BSD systems. Signed-off-by: Kris Hicks <khicks@nvidia.com>	2026-09-24 21:17:12 +00:00
Oliver Calder	6c864ec9ab	fix(install): avoid installing incompatible docker snap (#3666) ... * fix(install): avoid installing incompatible docker snap The work to land RFC-0012 added new restrictions when interacting with Docker by setting `NoNewPrivs`. This prevents the `docker` snap from transitioning its AppArmor profile from `snap.docker.dockerd` to `docker-default` when it tries to launch a container. Thus, the `docker` snap is currently incompatible with OpenShell. This commit prevents `install.sh` from installing the `docker` snap before installing the `openshell` snap, and instead requires the user to install a non-snap Docker daemon before proceeding with installing the snap. Systems without the `snap` command are unaffected, since they install native packages without checking for the presence of Docker or other compute providers. Signed-off-by: Oliver Calder <oliver.calder@canonical.com> * fix(snap): only snapd 2.76 for openshell snap since store installs work Signed-off-by: Oliver Calder <oliver.calder@canonical.com> * fixup! fix(install): avoid installing incompatible docker snap Signed-off-by: Oliver Calder <oliver.calder@canonical.com> * fixup! fix(snap): only snapd 2.76 for openshell snap since store installs work Signed-off-by: Oliver Calder <oliver.calder@canonical.com> --------- Signed-off-by: Oliver Calder <oliver.calder@canonical.com>	2026-09-24 19:46:00 +00:00
Drew Newberry	0518bd4c83	fix(sandbox): preserve local sessions across host sleep (#3573) ... * fix(cli): recover sandbox connect transport Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(auth): propagate non-expiring local sandbox sessions Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(cli): distinguish main exit from transport loss Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(cli): bound sandbox connect recovery Signed-off-by: Drew Newberry <anewberry@nvidia.com> --------- Signed-off-by: Drew Newberry <anewberry@nvidia.com>	2026-09-24 17:22:30 +00:00
Oliver Calder	e60098d748	fix(snap): install openshell snap via install.sh when snap available (#3656) ... * fix(snap): update stale snap docs and tests Previously, the `openshell` snap required the `docker` snap. Now, it works with any Docker daemon running on the system. Furthermore, the `snap-declaration` assertion on the `openshell` snap when installed from the Snap Store causes the `openshell` snap to always connect to the system `:docker` slot, rather than a slot provided by the `docker` snap. This commit updates the documentation, including the `description` field in `snapcraft.yaml`, to ensure that all information is correct and up-to-date. Additionally, some tests connected the `openshell:docker` plug to the `docker` snap's `docker:docker-daemon` slot, which is inconsistent with how the `openshell` snap operates when installed from the store. Update those tests to connect to the system `:docker` slot as well. Signed-off-by: Oliver Calder <oliver.calder@canonical.com> * fix(snap): require snapd 2.76 for openshell snap Signed-off-by: Oliver Calder <oliver.calder@canonical.com> * fix(nix): align snap gateway reproducer timeout with release-canary Signed-off-by: Oliver Calder <oliver.calder@canonical.com> * fix(snap): require snapd 2.77 for openshell snap Signed-off-by: Oliver Calder <oliver.calder@canonical.com> * fixup! fix(snap): require snapd 2.77 for openshell snap Signed-off-by: Oliver Calder <oliver.calder@canonical.com> * feat(snap): install openshell snap via install.sh when snap available Change `install.sh` to install the `openshell` snap by default when snapd is installed on the host. This installs the snap from the `latest/stable` channel, which should match the most up-to-date release tag on github. If `OPENSHELL_VERSION=dev` is set for `install.sh`, then it will install the `openshell` snap from the `latest/edge` channel, which matches the latest dev release available on github. The `openshell` snap currently requires Docker in order to function. If a Docker daemon is already installed on the system, it will be used by the `openshell` snap. Otherwise, `install.sh` will install the `docker` snap first, wait for the Docker daemon to be ready, and then install the `openshell` snap. Also, update the `release-canary.yml` to split the `ubuntu-snap` job into `ubuntu-snap-system-docker` and `ubuntu-snap-provisions-docker`, which test the two aforementioned scenarios. Previously, `ubuntu-snap` manually installed a given snap artifact as built from CI, but with these new jobs, it instead uses `install.sh` to install the published `openshell` snap from the `latest/edge` track, thus matching the behavior of the other release canary jobs. Make corresponding changes to the `nix` guest reproducer, documentation, and tests. Signed-off-by: Oliver Calder <oliver.calder@canonical.com> * fix(snap): configure local gateway authentication The `openshell` snap runs the gateway as a systemd system service, which runs as root. Thus, the mTLS certs are generated by root and stored in a root-owned directory to which non-root users do not have access. For this reason, the snap's `openshell-gateway-wrapper` script sets `OPENSHELL_DISABLE_TLS=true`. This commit ensures that the `openshell` snap's gateway allows unauthenticated local access by writing a default `gateway.toml` configuration file during the install hook, which runs after the snap is first installed but before services are started. The config file contains sets `allow_unauthenticated_users = true`. Signed-off-by: Oliver Calder <oliver.calder@canonical.com> * fix(install): configure snap gateway authentication Recently, a new install hook was added which writes a default config file for the `openshell` snap to allow unauthenticated local access to the gateway. This is because the gateway service runs as root and the mTLS certificates are not accessible to non-root users. However, the `install.sh` script installs the `openshell` snap from the snap store, and the published version may not yet have that new install hook. Or, the user may already have the snap installed, in which case the install hook does not run. In either case, we need `install.sh` to ensure that the config file is written to set the gateway auth to `allow_unauthenticated_users = true`, and then restart the openshell gateway service. Signed-off-by: Oliver Calder <oliver.calder@canonical.com> * fixup! feat(snap): install openshell snap via install.sh when snap available Signed-off-by: Oliver Calder <oliver.calder@canonical.com> --------- Signed-off-by: Oliver Calder <oliver.calder@canonical.com>	2026-09-24 15:10:08 +00:00
Mark CampbellandKris Hicks	679b190677	feat(testing): support independent gateway and supervisor image overrides (#3341) ... * feat(testing): normalize configurable test images Signed-off-by: Bobbins228 <mcampbel@redhat.com> Signed-off-by: Kris Hicks <khicks@nvidia.com> * feat(helm): add global image overrides Signed-off-by: Bobbins228 <mcampbel@redhat.com> * feat(helm): support image registry overrides Signed-off-by: Bobbins228 <mcampbel@redhat.com> Signed-off-by: Kris Hicks <khicks@nvidia.com> * refactor(helm): simplify image configuration Signed-off-by: Bobbins228 <mcampbel@redhat.com> * fix(e2e): avoid reloading reused kind sandbox image Signed-off-by: Bobbins228 <mcampbel@redhat.com> * fix(helm): default sandbox image to nvcr.io/nvidia/base/ubuntu:24.04 Signed-off-by: Kris Hicks <khicks@nvidia.com> --------- Signed-off-by: Bobbins228 <mcampbel@redhat.com> Signed-off-by: Kris Hicks <khicks@nvidia.com> Co-authored-by: Kris Hicks <khicks@nvidia.com>	2026-09-23 23:35:30 +00:00
Mrunal Patel	a408f5dd08	chore(kubernetes): update Agent Sandbox to v1.0.3 (#3578) ... Signed-off-by: Mrunal Patel <mrunalp@gmail.com>	2026-09-23 18:30:24 +00:00
Evan Lezar	907f894ebc	ci(release): publish prereleases with qualification summary (#3593) ... Signed-off-by: Evan Lezar <elezar@nvidia.com>	2026-09-23 13:18:30 +00:00
Jim Meyer	88357775ad	docs(windows): align Z3 pin with z3-sys 0.13 (#3561) ... z3-sys 0.13 generates bindings for the Z3 5.x series and resolves GH_RELEASE_VERSION to 5.1.0 for the prebuilt-release path. The Windows wrapper still pinned Z3_SYS_Z3_VERSION to 4.16.0, so a prebuilt Windows build would fetch an archive that does not match the generated bindings. Move the wrapper pin to 5.1.0 and update the contributor, architecture, and skill references that still named 4.16.0. Signed-off-by: Jim Meyer <jimeyer@nvidia.com>	2026-09-22 19:29:24 +00:00
3 people	50230616d5	refactor(runtime): retire Community image dependencies (#3386) ... * feat(sandbox): default to official Alpine sandbox image default_sandbox_image() now returns docker.io/library/alpine:3.22, a generic version-qualified official image, so a fresh install no longer depends on the community sandbox image catalog. All compute drivers (docker, podman, kubernetes, vm) inherit this fallback. Part of #3116. Signed-off-by: Akram Signed-off-by: Akram <akram.benaissi@gmail.com> * feat(deploy): default deployment configs to the official Alpine sandbox image Update the shared gateway default_image, Helm chart values, the standalone Kubernetes manifest, and the dev gateway task scripts to use docker.io/library/alpine:3.22 instead of the community base image, consistent with default_sandbox_image(). GPU e2e image-build base is left unchanged (CUDA needs a glibc base). Part of #3116. Signed-off-by: Akram Signed-off-by: Akram <akram.benaissi@gmail.com> * feat(driver): default to numeric non-root identity for USER-less images With the default sandbox image now Alpine, images that declare no OCI USER must start instead of being rejected. When the image declares no USER and the policy requests none, the Podman and Docker drivers now supply a numeric non-root identity (DEFAULT_SANDBOX_UID/GID = 1000) instead of rejecting, matching the numeric-identity behavior of the Kubernetes and VM drivers. The supervisor's resolved-identity path runs the sandbox as a synthesized non-root account without the account existing in the image. Images that declare a USER keep the OCI resolution path unchanged. Part of #3116. Signed-off-by: Akram <akram.benaissi@gmail.com> Signed-off-by: Evan Lezar <elezar@nvidia.com> * test(conformance): use Alpine workload image Signed-off-by: Evan Lezar <elezar@nvidia.com> * refactor(policy): drop community image /app path from default policy The restrictive default policy granted read-only access to /app, a directory that only existed in the community base image. A generic Alpine default has no /app, so remove it. Landlock best-effort already ignores absent paths; this just stops advertising a community-specific layout in the default. Part of #3116. Signed-off-by: Akram Signed-off-by: Akram <akram.benaissi@gmail.com> * docs(config): document Alpine default images Signed-off-by: Evan Lezar <elezar@nvidia.com> * fix(podman): report early sandbox termination Signed-off-by: Evan Lezar <elezar@nvidia.com> * fix(podman): initialize rootless workspace ownership Signed-off-by: Evan Lezar <elezar@nvidia.com> * fix(sandbox): qualify NVIDIA Ubuntu default Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(podman): initialize rootful default workspace Signed-off-by: Drew Newberry <anewberry@nvidia.com> * feat(sftp): add native sandbox adapter Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(sftp): gate runtime helper support to Linux Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(sftp): support standard OpenSSH file operations Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(sftp): harden rename and special file handling Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(runtime): remove community image dependencies Signed-off-by: Drew Newberry <anewberry@nvidia.com> * test(e2e): build provider readiness tool fixture Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(e2e): use a dedicated Noble fixture for Docker tests Signed-off-by: Evan Lezar <elezar@nvidia.com> --------- Signed-off-by: Akram Signed-off-by: Akram <akram.benaissi@gmail.com> Signed-off-by: Evan Lezar <elezar@nvidia.com> Signed-off-by: Drew Newberry <anewberry@nvidia.com> Co-authored-by: Evan Lezar <elezar@nvidia.com> Co-authored-by: Drew Newberry <anewberry@nvidia.com>	2026-09-22 14:43:51 +02:00
Emilien Macchi	99ed6a9df0	chore(build): remove stale static-supervisor leftovers (#3520) ... The opt-in glibc-static supervisor variant from #2682 is gone: the Nix release builds (#2977) removed it from CI and the supervisor Dockerfile, and the RFC 0012 sandbox split (#2942) removed it from the staging script. The split also moved the binary that runs inside workload images to openshell-sandbox; the supervisor now runs from its own image and is dynamically linked. A few places still describe the old model: - skills/debug-openshell-cluster told operators to check that supervisor_image contains a static /openshell-supervisor. Ask instead for a supervisor from the matching release whose loader and shared libraries are available inside that image, and give a `--version` check that shows both. The static requirement for /openshell-sandbox is unchanged. - verify-static-binary.sh justified its check with the supervisor and the glibc-static variant. Describe the property it enforces instead, with the musl sandbox runtime as the main example. - stage-prebuilt-binaries.sh kept an unreachable gnu-static case in target_triple. No behavior change: resolve_component only ever selects gnu or musl. Signed-off-by: Emilien Macchi <emacchi@redhat.com>	2026-09-22 00:19:37 +00:00
Drew Newberry	dee4f98dda	chore(vm): refresh runtime defaults and hardening (#3446) ... Signed-off-by: Drew Newberry <anewberry@nvidia.com>	2026-09-21 10:19:30 +00:00
Drew Newberry	d6f3e1f5f2	fix(packaging): keep SPDX comments out of Debian control (#3483) ... Signed-off-by: Drew Newberry <anewberry@nvidia.com>	2026-09-19 18:55:33 -07:00
Drew Newberry	17ce738bfb	fix(ci)!: remove gateway callback listener dependency (#3365) ... * fix(ci): repair post-merge release canary Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(packaging): bootstrap canary runtime prerequisites Signed-off-by: Drew Newberry <anewberry@nvidia.com> * ci(canary): collect macOS VM diagnostics Signed-off-by: Drew Newberry <anewberry@nvidia.com> * ci(canary): pin libkrun-compatible macOS runner Signed-off-by: Drew Newberry <anewberry@nvidia.com> * ci(canary): limit macOS smoke test to package startup Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(compute)!: remove gateway callback listeners Run Docker supervisors on host networking so they use the operator-configured primary gateway endpoint. Remove the unused compute-driver callback listener negotiation and listener-scoped routing machinery. BREAKING CHANGE: The ComputeDriver API no longer exposes GetGatewayListenerRequirements or GatewayListenerRequirement. External drivers must regenerate bindings and connect supervisors to the configured primary gateway endpoint. Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(docker): use sandbox runtime image in launcher Signed-off-by: Drew Newberry <anewberry@nvidia.com> * test(podman): exercise production endpoint selection Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(e2e): route supervisors to reachable gateways Signed-off-by: Drew Newberry <anewberry@nvidia.com> * test(e2e): align Podman endpoint fixtures Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(docker): preserve host aliases for supervisors Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(docker): align sandbox host gateway pin Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(e2e): address Docker fixtures by bridge IP Signed-off-by: Drew Newberry <anewberry@nvidia.com> * test(e2e): serialize sandbox lifecycle cases Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(e2e): host Docker TCP fixture with gateway Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(e2e): use loopback for host-network supervisor Signed-off-by: Drew Newberry <anewberry@nvidia.com> --------- Signed-off-by: Drew Newberry <anewberry@nvidia.com>	2026-09-18 20:55:55 +00:00
Drew Newberry	d91b1999a0	feat(api)!: use sandbox names as canonical RPC references (#3272) ... * feat(api)!: use sandbox names as canonical references Signed-off-by: Drew Newberry <anewberry@nvidia.com> * test(cli): update forward color fixture for workspace scope Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(supervisor): use sandbox names for settings lookup Signed-off-by: Drew Newberry <anewberry@nvidia.com> * test(e2e): use canonical sandbox request fields Signed-off-by: Drew Newberry <anewberry@nvidia.com> * test(e2e): use canonical sandbox receipt field Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(server): harden sandbox mutation handling Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(ci): update rebased sandbox references Signed-off-by: Drew Newberry <anewberry@nvidia.com> * feat(api)!: standardize canonical entity references Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs(api): codify protobuf API conventions Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(api): preserve workspace selector semantics Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(api): restore workspace selector parity Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(api): preserve descriptive name fields Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(api): update e2e request fixtures Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(core): omit workspace selector during bootstrap Signed-off-by: Drew Newberry <anewberry@nvidia.com> * chore(api): remove proto convention checker Signed-off-by: Drew Newberry <anewberry@nvidia.com> * chore(api): refresh schema fingerprints after rebase Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(cli): use canonical provider receipt field Signed-off-by: Drew Newberry <anewberry@nvidia.com> --------- Signed-off-by: Drew Newberry <anewberry@nvidia.com>	2026-09-18 11:39:55 -07:00
Florent BENOIT	c5a8c4d220	chore(vm): bump libkrun to v1.19.4 and libkrunfw to v5.6.1 (#3451) ... The libkrun and libkrunfw projects moved from the containers/ GitHub org to libkrun/. Update all clone URLs and version pins accordingly. libkrun v1.19.4 introduces krun_add_virtiofs4 with a permissions semantics parameter, enabling correct file ownership mapping on host-to-guest bind mounts — required for issue #2585. - libkrun: v1.17.4 → v1.19.4 (libkrun/libkrun) - libkrunfw: 463f717b → v5.6.1 (libkrun/libkrunfw) - Centralise LIBKRUN_REF in pins.env instead of hardcoding in each build script - Source pins.env in the macOS build script for consistency with the Linux build script - Remove the init/init make target step — since libkrun/libkrun@05c4eb7 the init blob moved into the init_blob crate and is built by cargo Ref: https://github.com/NVIDIA/OpenShell/issues/2585 Signed-off-by: Florent Benoit <fbenoit@redhat.com>	2026-09-18 13:32:04 +00:00
Piotr Mlocek	c108c31696	docs(fern): sync announcement configuration (#3436) ... * docs(fern): sync announcement configuration Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> * fix(docs): scope announcements to synced channel Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> * docs(fern): use dev as source version Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> * docs(fern): use channel-neutral logo link Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> --------- Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>	2026-09-18 03:43:01 +00:00
Drew NewberryandPiotr Mlocek	8b77925eb7	feat(installer): support prerelease installations (#3364) ... * feat(installer): support prerelease installations Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs(readme): clarify 0.1.0 production readiness Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs(readme): refine production readiness message Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(release): publish rolling prerelease channels Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(release): keep prereleases out of GitHub releases Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs(readme): refine 0.1.0 notice Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs(readme): expand 0.1.0 notice Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs(readme): simplify 0.1.0 notice Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(installer): simplify prerelease alias Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(installer): require successful prerelease runs Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(release): reuse exact prerelease artifacts Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(installer): include prover in prerelease bundle Signed-off-by: Drew Newberry <anewberry@nvidia.com> * ci: remove prerelease post-publish validation Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs(fern): sync announcement configuration Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> * docs: remove stale prerelease canary claim Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs(fern): defer announcement synchronization Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs(fern): simplify release announcement Signed-off-by: Drew Newberry <anewberry@nvidia.com> --------- Signed-off-by: Drew Newberry <anewberry@nvidia.com> Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> Co-authored-by: Piotr Mlocek <pmlocek@nvidia.com>	2026-09-18 00:53:36 +00:00
Philippe MartinandJohn Myers	04146692d9	refactor(providers)!: make provider profiles import-only (#3383) ... * chore(providers): remove dead provider plugin modules Twelve modules under crates/openshell-providers/src/providers/ were never declared in providers/mod.rs, so they have not been compiled since the plugin registry was narrowed to the two adapters it still registers. Four of them (generic, gitlab, opencode, outlook) key off provider type identifiers that normalize_provider_type already retired. Keep google_cloud and vertex, which are the only plugins ProviderRegistry::new registers. Signed-off-by: Philippe Martin <phmartin@redhat.com> * test(providers): load example profiles from providers/ at test time Add an example_profiles module that reads the YAML under providers/ from the source checkout at run time and parses it with the existing profile loader. It is gated behind a new non-default example-profiles feature so it is available to this crate's own tests and, once wired into dev-dependencies, to gateway and CLI tests, while never reaching a release binary. Nothing consumes it yet; later changes move the test fixtures off the compiled catalog and onto these files. Signed-off-by: Philippe Martin <phmartin@redhat.com> * test: source profile fixtures from providers/ instead of the compiled catalog Unit and integration fixtures reached into builtin_profiles() to get a profile to work with, which ties the tests to the compiled catalog rather than to the files an operator would import. Point them at the example_profiles loader instead. The profiles.rs tests keep their coverage unchanged and become explicit golden tests over providers/*.yaml: they are what keeps those files valid once nothing compiles them. builtin_profiles_are_sorted_by_id widens into a test that the whole example set parses, sorts, and lints clean as one catalog. The CLI fake gateways serve the example profiles the way a real gateway serves what an operator imported, through a shared helper. No behavior change: the gateway still loads the built-in source by default and serves the same profiles. Signed-off-by: Philippe Martin <phmartin@redhat.com> * docs(providers): document the example provider profiles Each file under providers/ now opens with a header naming its expected client binary identities, the image layout those paths assume, the credential scope, the endpoint access it grants, and a smoke test. Add a README covering the import commands and why a profile should be copied and edited rather than imported unchanged. Several of these profiles bind network access to paths that only exist in the OpenShell Community image — /sandbox/.venv, /app/.venv, /sandbox/.cursor-server, /usr/lib/node_modules. Imported unchanged into another image the profile matches nothing: the catalog still advertises it, but the credential is never injected and the traffic is denied. The headers say so where it applies. Comments only; the profile schema has no documentation fields and all fifteen files still lint clean. Signed-off-by: Philippe Martin <phmartin@redhat.com> * test(server): seed unit-test state with the example provider profiles Server unit tests inherited the builtin + user source default from ServerState, so around a hundred and forty assertions about github, openai and the rest resolved against the compiled catalog. Point the test state at the user source alone and import the example profiles from providers/ into its store first, the way an operator would. Every one of those assertions keeps passing unchanged, which is the point: it proves the gateway behaves identically with an imported catalog before the default moves. Four tests asserted builtin-source semantics specifically. Under import-only the only profiles a gateway cannot edit are the ones a non-user source vends, so they now exercise a source-managed profile composed with the user source; the read-only guard they cover is the one that still applies to interceptor catalogs. The list test asserts the imported profile's user/platform identity instead of builtin with an empty scope. test_server_state_with_user_only_github_profile is gone: the default test state now is a user-only gateway with github imported. Signed-off-by: Philippe Martin <phmartin@redhat.com> * refactor(cli): resolve credential suggestions from the gateway catalog The --env credential warning scanned a profile table compiled into the CLI, so its suggestions described the binary rather than the gateway the user is talking to: a profile the gateway does not serve was suggested anyway, and an imported custom profile never was. Fetch the catalog from the connected gateway instead. The warning moves out of argument parsing and into sandbox create and sandbox template create, where a client already exists. A catalog fetch failure is not fatal — the warning degrades to its generic form rather than blocking sandbox creation. Extract the ListProviderProfiles paging loop from provider list-profiles into a shared fetch_provider_profile_catalog, which the profile-driven paths now share. Signed-off-by: Philippe Martin <phmartin@redhat.com> * refactor(cli)!: infer providers from the gateway catalog Command-to-provider inference went through a hardcoded alias table in openshell-providers, a second copy of the built-in catalog's identifiers. A custom profile could never be inferred no matter what binaries it declared, and the table drifted from the profiles it mirrored. Infer from the connected gateway's catalog instead: match the command's basename against each profile's ID and against the basenames of the binaries the profile authorizes. A profile that names /usr/bin/claude is the profile for running claude. The match must be unique — where several profiles claim a command, the user names one with --provider — and an empty catalog infers nothing. No command is special-cased. `binaries` is the operator's authorization statement, so a profile that declares a binary claims the command that runs it, whatever that binary is; narrowing that belongs in the profile rather than in a list compiled into the CLI, which could never cover an unbounded catalog anyway. Breaking: the retired aliases stop resolving, and commands the old table never listed can now infer. git, pip and uv are declared by the github and pypi example profiles, so they infer where they previously did not. Signed-off-by: Philippe Martin <phmartin@redhat.com> * refactor(providers,server)!: resolve provider profiles by exact ID normalize_provider_type was a hardcoded alias map — gh to github, claude to claude-code, vertex to google-vertex-ai — and a second copy of the built-in catalog's identifiers, independent of the YAML it mirrored. It let a provider type resolve to a profile the operator never named, and it made the built-in IDs behave as a reserved namespace. Remove it, along with detect_provider_from_command and the alias-normalizing ProviderRegistry::inject_env. A provider type now names a profile exactly: - the effective catalog resolves an ID or reports it absent, with no alias retry - plugins activate only for the ID of a profile the gateway resolved; a provider with no resolvable profile gets no plugin projection, instead of falling back to an alias guess - the CLI surfaces the gateway's not-found instead of retrying under an alias - telemetry buckets by profile ID, and the gitlab, opencode and outlook buckets go with the aliases that were their only source Breaking: `--type gh`, `--type claude` and the other aliases no longer resolve. Use the profile's own ID. Signed-off-by: Philippe Martin <phmartin@redhat.com> * feat(server): fail closed when a provider's profile is absent A sandbox composed from a provider whose profile the gateway cannot resolve started anyway: the credential and policy builders warned and skipped, so the sandbox came up carrying none of that provider's credentials or network policy. The operator learned about it later, as a denied connection or a missing environment variable, rather than as the configuration error it is. Check at the two composition boundaries — CreateSandbox and AttachSandboxProvider — right beside the catalog snapshot already taken there, and reject with a bounded diagnostic naming the provider, the profile it refers to, and the import command that supplies it. Scope-aware: a platform-scoped provider is told to import with --global. Read paths are untouched. ListProviders, GetProvider and profile export keep working so an operator can see and recover an affected provider, and the shared policy builders keep their warn-and-skip for the diagnostic paths that also reach them. Signed-off-by: Philippe Martin <phmartin@redhat.com> * refactor(server): scope the vendor base-URL pin to declared endpoints provider_profile_endpoints_are_active withheld a credential and the provider's policy layer when an openai or anthropic provider pointed its client somewhere other than the public vendor endpoint. The guard was keyed on profile.source == "builtin" and on those two profile IDs, so it protected only profiles OpenShell shipped. Once profiles are import-only no profile is ever builtin, and the guard would silently stop applying — including to an operator who imported providers/openai.yaml verbatim. Key it on the profile instead of on where the profile came from. A profile's endpoints are the boundary its credential is bound to, so if the provider configures a *_BASE_URL pointing at a host the profile does not declare, the profile no longer describes where that credential goes and is treated as endpointless. Host matching reuses the DNS-label-aware matcher in openshell-core, so wildcard endpoints such as Vertex's *-aiplatform.googleapis.com resolve correctly. A profile with no declared endpoints has no boundary to contradict, and a config value that names no host is not a redirect this can reason about. Both keep the profile active. The control now covers every endpoint-bearing profile, including an operator's own, and the last "builtin" string leaves the gateway. Signed-off-by: Philippe Martin <phmartin@redhat.com> * test(e2e): import example provider profiles during gateway bring-up The e2e suites create providers from github, openai, nvidia, claude-code and google-cloud, and the GitHub lane runs a real git clone through the profile's binary attribution. A gateway serves only the profiles an operator imported, so the lanes have to import them. Add e2e_import_example_provider_profiles to the shared bring-up helpers and call it from the Docker, Podman, Kubernetes and VM wrappers once the gateway is healthy and registered. It runs the same command the upgrade notes give operators, against the repository's own providers/ directory, so the lanes exercise the documented path rather than a test-only shortcut. Lands before the default changes: a same-ID user profile already shadows a built-in, so importing works today. Signed-off-by: Philippe Martin <phmartin@redhat.com> * refactor(providers)!: make provider profiles import-only OpenShell compiled fifteen provider profile YAML files into every release binary and selected them by default, so a fresh gateway published a catalog it was never configured with. Those profiles are not image-neutral: their binary selectors name paths that exist in the OpenShell Community image, so changing the sandbox image could make a profile inert while the catalog still advertised it. Every endpoint, credential name and binary path in providers/ was also effectively part of the 0.1.0 public contract. A gateway's catalog is now exactly what an operator imported: - providers/*.yaml is no longer include_str!'d, and builtin_profiles() is gone from the openshell-providers API - the default provider_profile_sources is [{ type = "user" }], and a gateway with nothing imported reaches ready and serves an empty catalog — an empty catalog is a valid state, not a startup failure - the builtin source type is removed from the configuration schema, and a gateway.toml that still names it is rejected at parse time with the import command rather than an unknown-variant error - nothing reserves the canonical identifiers any more, so github, pypi, anthropic and the rest import at their own IDs and the imported profile is the only definition for that ID; the static_fallback that kept a shipped definition resident behind an imported one is gone with the source that produced it A collision between an interceptor-vended profile and an imported one still fails closed, which is the behavior that shadowing quietly bypassed for built-ins. Operators upgrading should export the profiles their deployment relies on before upgrading, or copy them from the providers/ directory of the matching release tag, then import them at the scope their providers use. Signed-off-by: Philippe Martin <phmartin@redhat.com> * docs(providers): document import-only provider profiles Rewrite the provider profile documentation around a catalog the operator builds rather than one the gateway ships. - gateway-config: the default is [{ type = "user" }], the builtin source type is gone and rejected at startup, and an empty catalog is a valid ready state - profiles: replace the built-in profile table and its shadowing semantics with the import workflow, and say plainly that a profile whose binary paths do not match the image is inert - manage-providers: replace the two fixed provider-type tables with `provider list-profiles`, and describe catalog-driven command inference - inference-routing: drop the "still loads built-in profiles for compatibility" transition text; its migration walkthrough is now the normal path - quickstart and the Docker Compose, GitHub, AWS, Google Cloud and Vertex tutorials: import the profile before creating the provider, since nothing resolves without it - release notes: a 0.1.0 migration section covering export-before-upgrade, importing from the release tag's providers/ directory, what happens to a provider whose profile is missing, and the removal of the legacy type aliases - README, architecture, the openshell-cli and debug-openshell-cluster skills, and the governance interceptor example follow the same change; the example's smoke assertion now imports a profile to prove an authoritative interceptor hides it Signed-off-by: Philippe Martin <phmartin@redhat.com> * fix(cli,e2e): distinguish catalog failures and authenticate OIDC seeding Addresses two findings from review (GATOR-c1bd9867-02 and -03). The provider profile catalog lookup collapsed its error into an empty catalog, so an authorization, availability or transport failure on ListProviderProfiles was indistinguishable from a gateway that genuinely has no profiles. Command inference then resolved nothing and the sandbox was created without the provider it needed, deferring the failure to the workload. Keep the lookup's outcome instead of discarding it. The credential warning is advisory and still degrades to its generic form, but inference now consults the catalog only when there is a command to resolve and surfaces the lookup failure when there is, naming the gateway and pointing at explicit --provider selection. Two regression tests cover it through a fake gateway whose ListProviderProfiles returns UNAVAILABLE while every other RPC succeeds: sandbox creation fails without sending a provider-less create request, and a sandbox with no trailing command still succeeds because it needs no catalog. The e2e profile seeding also ran unauthenticated in the OIDC lanes. Those lanes deliberately skip gateway registration and start the gateway without a TLS client CA, so no mTLS identity exists and no token has been acquired when the import runs; the wrapper exited during setup. Skipping the import is not sufficient because the provider tests now require the claude-code profile. Add e2e_register_oidc_admin_session, which mints an administrator token with Keycloak's password grant — the same grant the OIDC test helpers use — and writes the gateway metadata and token bundle that an interactive login would have stored, so the import runs as an authenticated administrator. The mTLS lanes keep the direct import unchanged. Both affected wrappers are covered: with-podman-gateway.sh had the same defect as with-docker-gateway.sh. Signed-off-by: Philippe Martin <phmartin@redhat.com> * refactor(cli)!: remove command-derived provider attachment Addresses GATOR-c1bd9867-01. A profile's `binaries` list authorizes a binary to reach that profile's endpoints. It is not a statement that running the binary asks for the provider, and reading it as attachment intent let a command silently gain provider authority: the aws-s3 example declares /bin/bash, so `sandbox create -- bash` resolved an existing aws-s3 provider and attached its credential-backed capability and network policy to the shell and its descendants. Attachment of an already-created provider never prompted, so the confirmation flow did not guard it. The distinction that would make inference sound — whether a declared binary is a profile's client or merely a permitted runtime — cannot be expressed: NetworkBinary carries only a path, and the field that encoded it was removed in 0.1.0. Any substitute is a guess. Restricting the guess to a unique claimant does not help, because uniqueness measures how sparse the catalog is rather than what the user intended, and a compiled list of "generic" commands could never cover an unbounded operator catalog. Remove trailing-command inference rather than approximate it. A provider is attached only when named with --provider, which still creates a missing provider from local discovery when the name matches an imported profile ID. Sandboxes with no providers remain a normal, fully supported state. Removing inference also settles GATOR-c1bd9867-02: with no consumer deriving authority from the catalog, its only remaining use is the advisory credential warning, so a failed lookup degrades that warning instead of blocking creation. The regression test now asserts that an unreachable catalog still creates the sandbox and attaches nothing. While repurposing the deduplication test, a pre-existing defect surfaced: repeating a name in --provider auto-created it twice and the second attempt failed with "provider already exists", because the explicit pass never consulted the set of names it had already handled. Guard it. Signed-off-by: Philippe Martin <phmartin@redhat.com> * fix(e2e): load the OIDC token and trust the gateway when seeding profiles Addresses the carried finding GATOR-c1bd9867-03. e2e_register_oidc_admin_session established a session the CLI never used. The CLI decides whether to load a stored bearer token by matching on the auth_mode field of the gateway metadata alone; the helper omitted that field, so the metadata fell through to the default arm and oidc_token.json stayed on disk unread. The profile import went out unauthenticated exactly as it had before the helper existed. The helper also installed no trust anchor, leaving the CLI unable to verify the gateway's self-signed serving certificate. Write auth_mode = "oidc" so the stored token is loaded, and install the CA at <gateway>/mtls/ca.crt. Only the CA is installed: with no client certificate or key on disk the CLI falls back to CA-only server verification and authenticates with the bearer token, which is what these lanes need because they start the gateway without --tls-client-ca. Certificate verification stays on; no insecure transport override is introduced. Assert the session before anything depends on it. ListProviderProfiles is annotated auth_mode: "bearer", so it cannot succeed unless the token was loaded and accepted. The helper now fails at that point, naming the gateway config directory and echoing the CLI output, rather than letting the defect surface later as an opaque profile import error. Both affected wrappers pass the PKI directory and CLI binary the helper needs; the mTLS lanes are untouched. Signed-off-by: Philippe Martin <phmartin@redhat.com> * fix(e2e): request the OpenShell scopes when minting the admin token The OIDC lanes failed at the session assertion added in b25dd0298 with PERMISSION_DENIED and "scope 'provider:read' required". Authentication was working -- the gateway logged ListProviderProfiles at gRPC status 7, which it can only reach once the bearer token has been loaded and accepted. The token simply carried no OpenShell scope. sandbox:*, provider:*, config:*, workspace:* and openshell:all are optional client scopes on the openshell-cli client in scripts/keycloak-realm.json, so Keycloak mints them only when the request asks for them. The password grant here asked for nothing, leaving the realm defaults (openid, profile, email, roles, web-origins, acr) and an access token that authorizes no RPC. Request "openid openshell:all", as e2e/python/oidc/oidc_auth_test.py already does for its administrator tokens. openshell:all is SCOPE_ALL in crates/openshell-server/src/auth/authz.rs, so one scope covers the setup calls without enumerating them. Verified against the realm: the token goes from "email profile openid" to "email profile openshell:all openid". Record the same scopes in metadata.json. The helper writes the bundle openshell gateway login would have stored, and oidc_scopes is the field that login path reads back, so leaving it out would misdescribe the stored token. Signed-off-by: Philippe Martin <phmartin@redhat.com> * fix(e2e): seed provider profiles once per VM lane gateway The VM lanes failed on the second test target with "custom provider profile 'anthropic' already exists" for all fifteen example profiles, and the import exited non-zero. e2e_import_example_provider_profiles was called from run_e2e_test, so it ran once per target -- four times against one long-lived gateway. That was harmless while import overwrote silently, but profiles are now import-only: ImportProviderProfiles is create-only and reports an existing id as an error-severity diagnostic, with no overwrite flag on the request. The first import therefore succeeds and every later one fails. Hoist the call to just after the conformance run, which is where the docker, podman and kube lanes already seed their catalogs. The profiles persist for the gateway's lifetime, so every target still finds them, and the E2E_TEST_OVERRIDE path is covered by the same single call. Signed-off-by: Philippe Martin <phmartin@redhat.com> * test(e2e): name the provider explicitly in the OIDC workspace-user test user_can_create_sandbox_with_inferred_provider_command reached the gateway once the admin session was fixed, and then failed: it asserted a missing-provider error, but the sandbox was created and died provisioning with "failed to spawn sandbox entrypoint process 'claude-code'". The test drove provider resolution by passing claude-code as the trailing command and relying on the CLI to infer the provider type from it. That inference is what this branch removed, so the trailing word is now nothing but an entrypoint, and the image has no such binary. The regression the test guards is not inference itself: it is that a workspace user resolving a provider is not gated behind Platform Admin. Name the provider with --provider, the only remaining way to attach one. The CLI still has to fetch the claude-code profile before it can auto-create the provider, so the lookup a workspace user must be allowed to make still happens, and auto-creation still stops at the non-interactive branch with "missing required provider". Both assertions therefore keep their meaning. Rename the test and rework its comments to describe what it now exercises; the old name would otherwise outlive the behavior it was named for. Signed-off-by: Philippe Martin <phmartin@redhat.com> * fix(providers): reconcile import-only profiles with main Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> --------- Signed-off-by: Philippe Martin <phmartin@redhat.com> Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> Co-authored-by: John Myers <9696606+johntmyers@users.noreply.github.com>	2026-09-17 22:59:27 +00:00
John T. Myers	9b9f7905e1	fix(dev): extract Docker sandbox runtime from sandbox image (#3422) ... Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>	2026-09-17 18:21:11 +00:00
Johnny Greco	58b5f8f976	feat(prover): add standalone policy boundary checker (#3289) ... * feat(prover): add standalone policy maximum checker Signed-off-by: Johnny Greco <jogreco@nvidia.com> * refactor(prover): simplify check scope schema Signed-off-by: Johnny Greco <jogreco@nvidia.com> * fix(prover): align containment and cancellation with runtime Signed-off-by: Johnny Greco <jogreco@nvidia.com> * fix(prover): stabilize containment checks in CI Signed-off-by: Johnny Greco <jogreco@nvidia.com> * test(prover): avoid solver in fast-path guard test Signed-off-by: Johnny Greco <jogreco@nvidia.com> * fix(prover): align string containment with runtime Signed-off-by: Johnny Greco <jogreco@nvidia.com> * fix(prover): reject ambiguous z3 string escapes Signed-off-by: Johnny Greco <jogreco@nvidia.com> * fix(prover): align containment with runtime boundaries Signed-off-by: Johnny Greco <jogreco@nvidia.com> * fix(prover-cli): harden cancellation and invalid input Signed-off-by: Johnny Greco <jogreco@nvidia.com> * feat(packaging): install policy prover with OpenShell Signed-off-by: Johnny Greco <jogreco@nvidia.com> * docs(prover): clarify installation and check results Signed-off-by: Johnny Greco <jogreco@nvidia.com> * docs(build): describe prover distribution directly Signed-off-by: Johnny Greco <jogreco@nvidia.com> * refactor(prover): rename maximum policy to boundary Signed-off-by: Johnny Greco <jogreco@nvidia.com> * refactor(prover): localize fail-closed validation Signed-off-by: Johnny Greco <jogreco@nvidia.com> * test(prover): cover fail-closed CLI surfaces Signed-off-by: Johnny Greco <jogreco@nvidia.com> * test(prover): allow CI load for REST solver proof Signed-off-by: Johnny Greco <jogreco@nvidia.com> * fix(prover): use canonical policy schema for containment Signed-off-by: Johnny Greco <jogreco@nvidia.com> * fix(prover): preserve uncertainty for runtime binary globs Signed-off-by: Johnny Greco <jogreco@nvidia.com> * fix(prover): bound policy validation work Signed-off-by: Johnny Greco <jogreco@nvidia.com> * docs(prover): document validation resource limits Signed-off-by: Johnny Greco <jogreco@nvidia.com> * fix(prover): make containment API extensible Signed-off-by: Johnny Greco <jogreco@nvidia.com> * docs(prover): define containment API contract Signed-off-by: Johnny Greco <jogreco@nvidia.com> * fix(ci): integrate prover with consolidated builds Signed-off-by: Johnny Greco <jogreco@nvidia.com> * fix(ci): declare release packaging dependency Signed-off-by: Johnny Greco <jogreco@nvidia.com> --------- Signed-off-by: Johnny Greco <jogreco@nvidia.com>	2026-09-17 17:30:03 +00:00
Drew Newberry	b3e4ad4579	fix(ci): repair RFC 0012 post-merge checks (#3360) ... * fix(ci): lock nextest for Windows ARM64 Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(docker): preserve GPU access in sandbox runtime Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(windows): restore host proxy identity binding Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(windows): satisfy cross-platform network lint Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(windows): satisfy MXC lint Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(windows): exclude Unix supervisor runtime Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(mxc): satisfy Windows test lint Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(windows): escape proxy test policy paths Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(sandbox): accept admitted GPU runtime groups Signed-off-by: Drew Newberry <anewberry@nvidia.com> * test(docker): serialize proxy pipeline scenarios Signed-off-by: Drew Newberry <anewberry@nvidia.com> --------- Signed-off-by: Drew Newberry <anewberry@nvidia.com>	2026-09-16 04:08:00 +00:00
Prekshi Vyas	314c73343c	fix(ci): restore prebuilt Z3 on Windows (#3353) ... Signed-off-by: Prekshi Vyas <prekshiv@nvidia.com>	2026-09-16 01:41:53 +00:00