* feat(snap): run gateway as a user service by default
Replace the existing system `gateway` service with separate
`user-gateway` and `system-gateway` services. For new installs, only the
`user-gateway` service is enabled. For existing installs, only the
`system-gateway` service is enabled.
Existing installs continue to have a one-time migration which removes
legacy insecure configurations. It is up to `install.sh` or users to
manually copy mTLS credentials from the root-owned `$SNAP_COMMON/tls` to
the invoking user's OpenShell snap directory. This is explained in the
snap description in `snapcraft.yaml`, visible in the Snap Store listing
and via `snap info openshell`.
Service enablement is now managed by a new snap configuration option
named `gateway-mode`, so users can switch from one mode to another via
e.g. `sudo snap set openshell gateway-mode=user`. The `install` and
`post-refresh` hooks select which service to start by setting this mode.
If the `gateway-mode` is already set, then we know the one-time
migration has already taken place.
Signed-off-by: Oliver Calder <oliver.calder@canonical.com>
* fix(snap): simplify the snap gateway wrapper
Signed-off-by: Oliver Calder <oliver.calder@canonical.com>
* fix(snap): use `snap start --user` when running as non-root
Signed-off-by: Oliver Calder <oliver.calder@canonical.com>
* fix(snap): ensure install.sh waits for the correct docker daemon
Signed-off-by: Oliver Calder <oliver.calder@canonical.com>
* feat(snap): document how to safely switch between gateway modes
New installs run the gateway as a user service by default, while
upgrades of existing installs from prior to the user gateway continue to
use the existing system gateway. The user and system gateway are
fundamentally different services and there is no way to migrate state
between them.
However, it may be desirable for a user to run the gateway as a system
service after a fresh install. To support this, this commit adds
documentation for how to safely transition from the user gateway mode to
the system gateway mode, and vice versa.
Also, clarifies when it is necessary to manually copy TLS credentials
from the system gateway state directory to the user's home directory:
when upgrading a pre-TLS snap installation, or when switching from user
to system gateway mode. When upgrading an existing install with an HTTPS
gateway running as a system service, copying credentials again is not
necessary.
Signed-off-by: Oliver Calder <oliver.calder@canonical.com>
---------
Signed-off-by: Oliver Calder <oliver.calder@canonical.com>
## Summary
Bump the pinned libkrun from v1.19.4 to v1.19.6, the latest upstream
release. v1.19.5 includes libkrun/libkrun@d2d8dd6, which removes refused
vsock connections without waiting for the 5 s reaper, so VM sandbox
startup no longer stalls on "Waiting for VM supervisor".
## Related Issue
Closes#4259
## Changes
- libkrun: v1.19.4 (728df812) -> v1.19.6 (227b2de6)
- Update LIBKRUN_REF in pins.env and the fallback defaults in
build-libkrun.sh and build-libkrun-macos.sh
- libkrunfw stays on v5.6.1
Other notable upstream fixes (v1.19.5): vsock proxy release after clean
shutdown, macOS virtio-fs xattr handling, x86_64 initrd kept below the
MMIO gap, PIT missed-tick reinjection disabled, macOS VMs with >64GB
RAM. v1.19.6 adds an API to set the render server fd. No API removals.
## Testing
- libkrun v1.19.6 builds on macOS arm64 with `make BLK=1 NET=1`
Signed-off-by: Florent Benoit <fbenoit@redhat.com>
* ci(release): add advisory compatibility review
Signed-off-by: Simon Scatton <sscatton@nvidia.com>
* test(ci): remove compatibility reviewer test harness
Signed-off-by: Simon Scatton <sscatton@nvidia.com>
* ci(release): run compatibility SDK in the dev shell
Signed-off-by: Simon Scatton <sscatton@nvidia.com>
* ci(release): trigger a one-shot compatibility review
Signed-off-by: Simon Scatton <sscatton@nvidia.com>
* ci(release): remove one-shot review trigger
Signed-off-by: Simon Scatton <sscatton@nvidia.com>
---------
Signed-off-by: Simon Scatton <sscatton@nvidia.com>
Adds the two stdlib-only helpers that back the maintainer approval gate,
ahead of the workflows that call them.
check_maintainer_approval.py parses the linked handles out of
MAINTAINERS.md, folds a review list to each reviewer's latest decisive
position, and exits non-zero unless a maintainer's latest position is an
approval. It fails closed when the list yields no handles.
alert_maintainer_change.py renders the approver-set delta between two
versions of MAINTAINERS.md, and prints nothing when the set is unchanged.
Landing these first lets the gate workflow, which reads both the list and
the decision logic from main rather than from the pull request ref,
actually execute on the pull request that introduces it.
Signed-off-by: Jim Meyer <jimeyer@nvidia.com>
Closes#3502
Resolve trusted OCI runtime images with driver TOML, process environment, and compiled-default precedence across Docker, Podman, and Kubernetes.
Signed-off-by: Evan Lezar <elezar@nvidia.com>
* fix(snap): simplify snap hooks
The `post-refresh` hook runs after initial snap installation as well, so
there is no need to call the `install` hook from within the
`post-refresh` hook; instead, the logic can simply be moved into the
`post-refresh` hook directly, and the `install` hook removed.
Also, the existing `install` hook logic looked for an insecure
configuration, and if found, replaced the entire configuration file with
a minimal default in the current format. But OpenShell does that default
behavior without any config file, so we may as well simply remove the
configuration file entirely to keep up-to-date with the current default
behavior. Let OpenShell create a configuration file if it needs to,
rather than auto-create one via the packaging scripts.
Signed-off-by: Oliver Calder <oliver.calder@canonical.com>
* fix(snap): remove the connect-plug-docker hook
The `openshell:docker` is auto-connected to the system `:docker` slot,
so there should not be a need to separately restart the gateway service
when the interface is connected.
For locally-built test snaps which were not published to the store, the
autoconnection is not made, but when the snap is installed, the gateway
will attempt to start anyway and fail to find any available compute
driver, so quickly restart until it hits the systemd start-limit, after
which systemd prevents the service from being started again. If a user
tries to manually connect their locally-built `openshell` snap to the
`:docker` slot, then the `connect-plug-docker` hook runs and triggers a
restart of the gateway, which will usually fail because the start limit
has already been hit. An error in the hook will thus cause the interface
connection to be undone, which is undesirable.
Thus, we can remove this hook entirely, and instead allow interface
connections to succeed as intended. The user still needs to manually
restart the gateway service after making a manual connection (as was the
case previously) and probably needs to `systemctl reset-failed` first,
but at least connection will succeed beforehand so they can proceed with
these steps.
Signed-off-by: Oliver Calder <oliver.calder@canonical.com>
* fix(snap): set refresh-mode: endure again, with manual restart
Return to the previous behavior before commit a67567e58, where the
gateway is not stopped before refreshes. The `post-refresh` hook
now restarts the gateway if the TLS configuration was corrected, so we
don't have to enforce restarting the gateway on every refresh even when
not necessary. Thus, set `refresh-mode: endure`, and let the hook decide
when the gateway needs to be restarted.
Signed-off-by: Oliver Calder <oliver.calder@canonical.com>
* fix(snap): update docs and tests to reflect snap hook changes
Signed-off-by: Oliver Calder <oliver.calder@canonical.com>
* docs(snap): remove verbose explanation of snap gateway refresh behavior
Signed-off-by: Oliver Calder <oliver.calder@canonical.com>
---------
Signed-off-by: Oliver Calder <oliver.calder@canonical.com>
* feat(ci): detect breaking protobuf changes
Compare the proto module against the PR or merge-group base and report Buf violations in Branch Checks. Add local reproduction and fixture coverage.
Closes#3794
Signed-off-by: Mrunal Patel <mrunalp@gmail.com>
* fix(ci): pin protobuf check container image
Signed-off-by: Mrunal Patel <mrunalp@gmail.com>
* fix(ci): qualify protobuf compatibility by release train
Signed-off-by: Simon Scatton <sscatton@nvidia.com>
* refactor(ci): reuse protobuf compatibility action
Signed-off-by: Simon Scatton <sscatton@nvidia.com>
* refactor(ci): run protobuf checks as a Nix app with one ref
Signed-off-by: Simon Scatton <sscatton@nvidia.com>
---------
Signed-off-by: Mrunal Patel <mrunalp@gmail.com>
Signed-off-by: Simon Scatton <sscatton@nvidia.com>
Co-authored-by: Mrunal Patel <mrunalp@gmail.com>
* docs: align page file names and nav labels with published URLs
Signed-off-by: Johnny Greco <jogreco@nvidia.com>
* docs: redirect moved dev pages and fix agent guide redirects
Signed-off-by: Johnny Greco <jogreco@nvidia.com>
* ci(docs): check that page URLs match their file paths
Signed-off-by: Johnny Greco <jogreco@nvidia.com>
* fix(docs): align navigation checks with repo conventions
Signed-off-by: Johnny Greco <jogreco@nvidia.com>
* docs: omit historical redirect aliases
Signed-off-by: Johnny Greco <jogreco@nvidia.com>
* fix(docs): preserve published overview and TypeScript URLs
Signed-off-by: Johnny Greco <jogreco@nvidia.com>
* fix(docs): redirect unversioned overview and TypeScript URLs
Signed-off-by: Johnny Greco <jogreco@nvidia.com>
---------
Signed-off-by: Johnny Greco <jogreco@nvidia.com>
Closes#3746
Use Homebrew atomic_write for exact legacy config migrations and keep the first-install write distinct in the formula test.
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* fix(snap): require mTLS for the snap gateway
Replace the installer opt-in with an authenticated snap gateway. The wrapper
no longer forces plaintext, so the gateway serves TLS from the bundle it
already generates in $SNAP_COMMON/tls. The install hook writes a config that
enables mTLS user auth instead of unauthenticated access, and a new
post-refresh hook migrates the exact legacy default on existing installs.
install.sh waits for the gateway, detects whether it serves TLS, copies the
client bundle into the target user's snap state directory, and registers the
gateway over HTTPS. Older plaintext snap revisions still register over HTTP
with a warning. The release canary asserts mTLS auth and HTTPS registration.
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* fix(snap): pass config preflight and detect the mTLS gateway reliably
An explicit [openshell.gateway.mtls_auth] table fails config preflight,
which validates mTLS auth before the local TLS bundle supplies the client
CA. Write a default that pins the Docker driver instead; with the wrapper's
TLS bundle the gateway requires client certificates and enables mTLS user
auth automatically, as the native packages do.
The mTLS gateway rejects TLS handshakes without a client certificate, and
it still answers plaintext loopback HTTP for sandbox service routing, so the
installer could misdetect it as a legacy plaintext gateway. Probe HTTPS with
the root-owned client bundle, and treat a gateway as legacy only when a
plaintext gRPC Health call succeeds.
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* chore(snap): simplify install hook comment
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* fix(snap): migrate insecure gateway configs on refresh
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* refactor(snap): simplify mTLS detection and config migration
Detect the mTLS snap from the installed revision's post-refresh hook instead
of probing plaintext gRPC, and drop the scheme global. Remove the installer's
pre-hook config fallback, which is dead now that every channel ships the
install hook and which wrote the insecure default. Give the install hook a
single write path with a simple backup name, and shorten the manual client
certificate steps.
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* fix(snap): stop keeping a copy of replaced insecure configs
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* feat(install): make the snap an opt-in install method
Stop selecting the OpenShell snap just because the snap command exists. Linux
installs default to the Debian or RPM package; OPENSHELL_INSTALL_METHOD=snap
(or deb, rpm) selects the package explicitly. Hosts that already have the
OpenShell snap keep refreshing it rather than gaining a second gateway on the
same port. The release canary and snap repro script opt in explicitly.
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* fix(snap): let the gateway auto-detect its compute driver
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* fix(snap): restart the gateway after refresh
Published revisions use refresh-mode: endure, and snapd honors the old
revision's setting during a refresh, so the plaintext gateway kept running
with the migrated config unused until a manual restart. Restart the gateway
from the post-refresh hook so the mTLS config takes effect immediately.
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* docs(snap): drop refresh notes from the snap description
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* docs(snap): trim snap refresh notes from installation docs
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
---------
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* docs(fern): publish and order versioned release docs
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
* docs(fern): remove version availability badges
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
* docs(fern): keep version badges optional
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
---------
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
The snap installer tests added in #3656 checked the gateway config mode
with GNU stat -c, which BSD stat on macOS rejects, so mise run ci failed
locally on macOS. Check the mode with find -perm instead, which matches
the exact mode on both GNU and BSD systems.
Signed-off-by: Kris Hicks <khicks@nvidia.com>
* fix(install): avoid installing incompatible docker snap
The work to land RFC-0012 added new restrictions when interacting with
Docker by setting `NoNewPrivs`. This prevents the `docker` snap from
transitioning its AppArmor profile from `snap.docker.dockerd` to
`docker-default` when it tries to launch a container. Thus, the `docker`
snap is currently incompatible with OpenShell.
This commit prevents `install.sh` from installing the `docker` snap
before installing the `openshell` snap, and instead requires the user to
install a non-snap Docker daemon before proceeding with installing the
snap. Systems without the `snap` command are unaffected, since they
install native packages without checking for the presence of Docker or
other compute providers.
Signed-off-by: Oliver Calder <oliver.calder@canonical.com>
* fix(snap): only snapd 2.76 for openshell snap since store installs work
Signed-off-by: Oliver Calder <oliver.calder@canonical.com>
* fixup! fix(install): avoid installing incompatible docker snap
Signed-off-by: Oliver Calder <oliver.calder@canonical.com>
* fixup! fix(snap): only snapd 2.76 for openshell snap since store installs work
Signed-off-by: Oliver Calder <oliver.calder@canonical.com>
---------
Signed-off-by: Oliver Calder <oliver.calder@canonical.com>
* fix(snap): update stale snap docs and tests
Previously, the `openshell` snap required the `docker` snap. Now, it
works with any Docker daemon running on the system. Furthermore, the
`snap-declaration` assertion on the `openshell` snap when installed from
the Snap Store causes the `openshell` snap to always connect to the
system `:docker` slot, rather than a slot provided by the `docker` snap.
This commit updates the documentation, including the `description` field in
`snapcraft.yaml`, to ensure that all information is correct and
up-to-date.
Additionally, some tests connected the `openshell:docker` plug to the
`docker` snap's `docker:docker-daemon` slot, which is inconsistent with
how the `openshell` snap operates when installed from the store. Update
those tests to connect to the system `:docker` slot as well.
Signed-off-by: Oliver Calder <oliver.calder@canonical.com>
* fix(snap): require snapd 2.76 for openshell snap
Signed-off-by: Oliver Calder <oliver.calder@canonical.com>
* fix(nix): align snap gateway reproducer timeout with release-canary
Signed-off-by: Oliver Calder <oliver.calder@canonical.com>
* fix(snap): require snapd 2.77 for openshell snap
Signed-off-by: Oliver Calder <oliver.calder@canonical.com>
* fixup! fix(snap): require snapd 2.77 for openshell snap
Signed-off-by: Oliver Calder <oliver.calder@canonical.com>
* feat(snap): install openshell snap via install.sh when snap available
Change `install.sh` to install the `openshell` snap by default when
snapd is installed on the host. This installs the snap from the
`latest/stable` channel, which should match the most up-to-date release
tag on github.
If `OPENSHELL_VERSION=dev` is set for `install.sh`, then it will install
the `openshell` snap from the `latest/edge` channel, which matches the
latest dev release available on github.
The `openshell` snap currently requires Docker in order to function. If
a Docker daemon is already installed on the system, it will be used by
the `openshell` snap. Otherwise, `install.sh` will install the `docker`
snap first, wait for the Docker daemon to be ready, and then install the
`openshell` snap.
Also, update the `release-canary.yml` to split the `ubuntu-snap` job
into `ubuntu-snap-system-docker` and `ubuntu-snap-provisions-docker`,
which test the two aforementioned scenarios. Previously, `ubuntu-snap`
manually installed a given snap artifact as built from CI, but with
these new jobs, it instead uses `install.sh` to install the published
`openshell` snap from the `latest/edge` track, thus matching the
behavior of the other release canary jobs.
Make corresponding changes to the `nix` guest reproducer, documentation,
and tests.
Signed-off-by: Oliver Calder <oliver.calder@canonical.com>
* fix(snap): configure local gateway authentication
The `openshell` snap runs the gateway as a systemd system service, which
runs as root. Thus, the mTLS certs are generated by root and stored in a
root-owned directory to which non-root users do not have access. For
this reason, the snap's `openshell-gateway-wrapper` script sets
`OPENSHELL_DISABLE_TLS=true`.
This commit ensures that the `openshell` snap's gateway allows
unauthenticated local access by writing a default `gateway.toml`
configuration file during the install hook, which runs after the snap is
first installed but before services are started. The config file
contains sets `allow_unauthenticated_users = true`.
Signed-off-by: Oliver Calder <oliver.calder@canonical.com>
* fix(install): configure snap gateway authentication
Recently, a new install hook was added which writes a default config
file for the `openshell` snap to allow unauthenticated local access to
the gateway. This is because the gateway service runs as root and the
mTLS certificates are not accessible to non-root users.
However, the `install.sh` script installs the `openshell` snap from the
snap store, and the published version may not yet have that new install
hook. Or, the user may already have the snap installed, in which case
the install hook does not run. In either case, we need `install.sh` to
ensure that the config file is written to set the gateway auth to
`allow_unauthenticated_users = true`, and then restart the openshell
gateway service.
Signed-off-by: Oliver Calder <oliver.calder@canonical.com>
* fixup! feat(snap): install openshell snap via install.sh when snap available
Signed-off-by: Oliver Calder <oliver.calder@canonical.com>
---------
Signed-off-by: Oliver Calder <oliver.calder@canonical.com>
z3-sys 0.13 generates bindings for the Z3 5.x series and resolves
GH_RELEASE_VERSION to 5.1.0 for the prebuilt-release path. The Windows
wrapper still pinned Z3_SYS_Z3_VERSION to 4.16.0, so a prebuilt Windows
build would fetch an archive that does not match the generated bindings.
Move the wrapper pin to 5.1.0 and update the contributor, architecture,
and skill references that still named 4.16.0.
Signed-off-by: Jim Meyer <jimeyer@nvidia.com>
* feat(sandbox): default to official Alpine sandbox image
default_sandbox_image() now returns docker.io/library/alpine:3.22, a generic
version-qualified official image, so a fresh install no longer depends on the
community sandbox image catalog. All compute drivers (docker, podman,
kubernetes, vm) inherit this fallback.
Part of #3116.
Signed-off-by: Akram
Signed-off-by: Akram <akram.benaissi@gmail.com>
* feat(deploy): default deployment configs to the official Alpine sandbox image
Update the shared gateway default_image, Helm chart values, the standalone
Kubernetes manifest, and the dev gateway task scripts to use
docker.io/library/alpine:3.22 instead of the community base image, consistent
with default_sandbox_image(). GPU e2e image-build base is left unchanged (CUDA
needs a glibc base).
Part of #3116.
Signed-off-by: Akram
Signed-off-by: Akram <akram.benaissi@gmail.com>
* feat(driver): default to numeric non-root identity for USER-less images
With the default sandbox image now Alpine, images that declare no OCI USER
must start instead of being rejected. When the image declares no USER and
the policy requests none, the Podman and Docker drivers now supply a numeric
non-root identity (DEFAULT_SANDBOX_UID/GID = 1000) instead of rejecting,
matching the numeric-identity behavior of the Kubernetes and VM drivers. The
supervisor's resolved-identity path runs the sandbox as a synthesized
non-root account without the account existing in the image. Images that
declare a USER keep the OCI resolution path unchanged.
Part of #3116.
Signed-off-by: Akram <akram.benaissi@gmail.com>
Signed-off-by: Evan Lezar <elezar@nvidia.com>
* test(conformance): use Alpine workload image
Signed-off-by: Evan Lezar <elezar@nvidia.com>
* refactor(policy): drop community image /app path from default policy
The restrictive default policy granted read-only access to /app, a directory
that only existed in the community base image. A generic Alpine default has no
/app, so remove it. Landlock best-effort already ignores absent paths; this
just stops advertising a community-specific layout in the default.
Part of #3116.
Signed-off-by: Akram
Signed-off-by: Akram <akram.benaissi@gmail.com>
* docs(config): document Alpine default images
Signed-off-by: Evan Lezar <elezar@nvidia.com>
* fix(podman): report early sandbox termination
Signed-off-by: Evan Lezar <elezar@nvidia.com>
* fix(podman): initialize rootless workspace ownership
Signed-off-by: Evan Lezar <elezar@nvidia.com>
* fix(sandbox): qualify NVIDIA Ubuntu default
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* fix(podman): initialize rootful default workspace
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* feat(sftp): add native sandbox adapter
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* fix(sftp): gate runtime helper support to Linux
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* fix(sftp): support standard OpenSSH file operations
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* fix(sftp): harden rename and special file handling
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* refactor(runtime): remove community image dependencies
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* test(e2e): build provider readiness tool fixture
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* fix(e2e): use a dedicated Noble fixture for Docker tests
Signed-off-by: Evan Lezar <elezar@nvidia.com>
---------
Signed-off-by: Akram
Signed-off-by: Akram <akram.benaissi@gmail.com>
Signed-off-by: Evan Lezar <elezar@nvidia.com>
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
Co-authored-by: Evan Lezar <elezar@nvidia.com>
Co-authored-by: Drew Newberry <anewberry@nvidia.com>
The opt-in glibc-static supervisor variant from #2682 is gone: the Nix
release builds (#2977) removed it from CI and the supervisor Dockerfile,
and the RFC 0012 sandbox split (#2942) removed it from the staging script.
The split also moved the binary that runs inside workload images to
openshell-sandbox; the supervisor now runs from its own image and is
dynamically linked. A few places still describe the old model:
- skills/debug-openshell-cluster told operators to check that
supervisor_image contains a static /openshell-supervisor. Ask instead for
a supervisor from the matching release whose loader and shared libraries
are available inside that image, and give a `--version` check that shows
both. The static requirement for /openshell-sandbox is unchanged.
- verify-static-binary.sh justified its check with the supervisor and the
glibc-static variant. Describe the property it enforces instead, with
the musl sandbox runtime as the main example.
- stage-prebuilt-binaries.sh kept an unreachable gnu-static case in
target_triple.
No behavior change: resolve_component only ever selects gnu or musl.
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
The libkrun and libkrunfw projects moved from the containers/ GitHub
org to libkrun/. Update all clone URLs and version pins accordingly.
libkrun v1.19.4 introduces krun_add_virtiofs4 with a permissions
semantics parameter, enabling correct file ownership mapping on
host-to-guest bind mounts — required for issue #2585.
- libkrun: v1.17.4 → v1.19.4 (libkrun/libkrun)
- libkrunfw: 463f717b → v5.6.1 (libkrun/libkrunfw)
- Centralise LIBKRUN_REF in pins.env instead of hardcoding in each
build script
- Source pins.env in the macOS build script for consistency with the
Linux build script
- Remove the init/init make target step — since libkrun/libkrun@05c4eb7
the init blob moved into the init_blob crate and is built by cargo
Ref: https://github.com/NVIDIA/OpenShell/issues/2585
Signed-off-by: Florent Benoit <fbenoit@redhat.com>
* docs(fern): sync announcement configuration
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
* fix(docs): scope announcements to synced channel
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
* docs(fern): use dev as source version
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
* docs(fern): use channel-neutral logo link
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
---------
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
* chore(providers): remove dead provider plugin modules
Twelve modules under crates/openshell-providers/src/providers/ were never
declared in providers/mod.rs, so they have not been compiled since the plugin
registry was narrowed to the two adapters it still registers. Four of them
(generic, gitlab, opencode, outlook) key off provider type identifiers that
normalize_provider_type already retired.
Keep google_cloud and vertex, which are the only plugins ProviderRegistry::new
registers.
Signed-off-by: Philippe Martin <phmartin@redhat.com>
* test(providers): load example profiles from providers/ at test time
Add an example_profiles module that reads the YAML under providers/ from the
source checkout at run time and parses it with the existing profile loader. It
is gated behind a new non-default example-profiles feature so it is available to
this crate's own tests and, once wired into dev-dependencies, to gateway and CLI
tests, while never reaching a release binary.
Nothing consumes it yet; later changes move the test fixtures off the compiled
catalog and onto these files.
Signed-off-by: Philippe Martin <phmartin@redhat.com>
* test: source profile fixtures from providers/ instead of the compiled catalog
Unit and integration fixtures reached into builtin_profiles() to get a profile
to work with, which ties the tests to the compiled catalog rather than to the
files an operator would import. Point them at the example_profiles loader
instead.
The profiles.rs tests keep their coverage unchanged and become explicit golden
tests over providers/*.yaml: they are what keeps those files valid once nothing
compiles them. builtin_profiles_are_sorted_by_id widens into a test that the
whole example set parses, sorts, and lints clean as one catalog.
The CLI fake gateways serve the example profiles the way a real gateway serves
what an operator imported, through a shared helper.
No behavior change: the gateway still loads the built-in source by default and
serves the same profiles.
Signed-off-by: Philippe Martin <phmartin@redhat.com>
* docs(providers): document the example provider profiles
Each file under providers/ now opens with a header naming its expected client
binary identities, the image layout those paths assume, the credential scope,
the endpoint access it grants, and a smoke test. Add a README covering the
import commands and why a profile should be copied and edited rather than
imported unchanged.
Several of these profiles bind network access to paths that only exist in the
OpenShell Community image — /sandbox/.venv, /app/.venv, /sandbox/.cursor-server,
/usr/lib/node_modules. Imported unchanged into another image the profile matches
nothing: the catalog still advertises it, but the credential is never injected
and the traffic is denied. The headers say so where it applies.
Comments only; the profile schema has no documentation fields and all fifteen
files still lint clean.
Signed-off-by: Philippe Martin <phmartin@redhat.com>
* test(server): seed unit-test state with the example provider profiles
Server unit tests inherited the builtin + user source default from ServerState,
so around a hundred and forty assertions about github, openai and the rest
resolved against the compiled catalog. Point the test state at the user source
alone and import the example profiles from providers/ into its store first, the
way an operator would. Every one of those assertions keeps passing unchanged,
which is the point: it proves the gateway behaves identically with an imported
catalog before the default moves.
Four tests asserted builtin-source semantics specifically. Under import-only the
only profiles a gateway cannot edit are the ones a non-user source vends, so
they now exercise a source-managed profile composed with the user source; the
read-only guard they cover is the one that still applies to interceptor
catalogs. The list test asserts the imported profile's user/platform identity
instead of builtin with an empty scope.
test_server_state_with_user_only_github_profile is gone: the default test state
now is a user-only gateway with github imported.
Signed-off-by: Philippe Martin <phmartin@redhat.com>
* refactor(cli): resolve credential suggestions from the gateway catalog
The --env credential warning scanned a profile table compiled into the CLI, so
its suggestions described the binary rather than the gateway the user is talking
to: a profile the gateway does not serve was suggested anyway, and an imported
custom profile never was.
Fetch the catalog from the connected gateway instead. The warning moves out of
argument parsing and into sandbox create and sandbox template create, where a
client already exists. A catalog fetch failure is not fatal — the warning
degrades to its generic form rather than blocking sandbox creation.
Extract the ListProviderProfiles paging loop from provider list-profiles into a
shared fetch_provider_profile_catalog, which the profile-driven paths now share.
Signed-off-by: Philippe Martin <phmartin@redhat.com>
* refactor(cli)!: infer providers from the gateway catalog
Command-to-provider inference went through a hardcoded alias table in
openshell-providers, a second copy of the built-in catalog's identifiers. A
custom profile could never be inferred no matter what binaries it declared, and
the table drifted from the profiles it mirrored.
Infer from the connected gateway's catalog instead: match the command's basename
against each profile's ID and against the basenames of the binaries the profile
authorizes. A profile that names /usr/bin/claude is the profile for running
claude. The match must be unique — where several profiles claim a command, the
user names one with --provider — and an empty catalog infers nothing.
No command is special-cased. `binaries` is the operator's authorization
statement, so a profile that declares a binary claims the command that runs it,
whatever that binary is; narrowing that belongs in the profile rather than in a
list compiled into the CLI, which could never cover an unbounded catalog anyway.
Breaking: the retired aliases stop resolving, and commands the old table never
listed can now infer. git, pip and uv are declared by the github and pypi
example profiles, so they infer where they previously did not.
Signed-off-by: Philippe Martin <phmartin@redhat.com>
* refactor(providers,server)!: resolve provider profiles by exact ID
normalize_provider_type was a hardcoded alias map — gh to github, claude to
claude-code, vertex to google-vertex-ai — and a second copy of the built-in
catalog's identifiers, independent of the YAML it mirrored. It let a provider
type resolve to a profile the operator never named, and it made the built-in IDs
behave as a reserved namespace.
Remove it, along with detect_provider_from_command and the alias-normalizing
ProviderRegistry::inject_env. A provider type now names a profile exactly:
- the effective catalog resolves an ID or reports it absent, with no alias retry
- plugins activate only for the ID of a profile the gateway resolved; a provider
with no resolvable profile gets no plugin projection, instead of falling back
to an alias guess
- the CLI surfaces the gateway's not-found instead of retrying under an alias
- telemetry buckets by profile ID, and the gitlab, opencode and outlook buckets
go with the aliases that were their only source
Breaking: `--type gh`, `--type claude` and the other aliases no longer resolve.
Use the profile's own ID.
Signed-off-by: Philippe Martin <phmartin@redhat.com>
* feat(server): fail closed when a provider's profile is absent
A sandbox composed from a provider whose profile the gateway cannot resolve
started anyway: the credential and policy builders warned and skipped, so the
sandbox came up carrying none of that provider's credentials or network policy.
The operator learned about it later, as a denied connection or a missing
environment variable, rather than as the configuration error it is.
Check at the two composition boundaries — CreateSandbox and
AttachSandboxProvider — right beside the catalog snapshot already taken there,
and reject with a bounded diagnostic naming the provider, the profile it refers
to, and the import command that supplies it. Scope-aware: a platform-scoped
provider is told to import with --global.
Read paths are untouched. ListProviders, GetProvider and profile export keep
working so an operator can see and recover an affected provider, and the shared
policy builders keep their warn-and-skip for the diagnostic paths that also
reach them.
Signed-off-by: Philippe Martin <phmartin@redhat.com>
* refactor(server): scope the vendor base-URL pin to declared endpoints
provider_profile_endpoints_are_active withheld a credential and the provider's
policy layer when an openai or anthropic provider pointed its client somewhere
other than the public vendor endpoint. The guard was keyed on
profile.source == "builtin" and on those two profile IDs, so it protected only
profiles OpenShell shipped. Once profiles are import-only no profile is ever
builtin, and the guard would silently stop applying — including to an operator
who imported providers/openai.yaml verbatim.
Key it on the profile instead of on where the profile came from. A profile's
endpoints are the boundary its credential is bound to, so if the provider
configures a *_BASE_URL pointing at a host the profile does not declare, the
profile no longer describes where that credential goes and is treated as
endpointless. Host matching reuses the DNS-label-aware matcher in
openshell-core, so wildcard endpoints such as Vertex's
*-aiplatform.googleapis.com resolve correctly.
A profile with no declared endpoints has no boundary to contradict, and a config
value that names no host is not a redirect this can reason about. Both keep the
profile active. The control now covers every endpoint-bearing profile, including
an operator's own, and the last "builtin" string leaves the gateway.
Signed-off-by: Philippe Martin <phmartin@redhat.com>
* test(e2e): import example provider profiles during gateway bring-up
The e2e suites create providers from github, openai, nvidia, claude-code and
google-cloud, and the GitHub lane runs a real git clone through the profile's
binary attribution. A gateway serves only the profiles an operator imported, so
the lanes have to import them.
Add e2e_import_example_provider_profiles to the shared bring-up helpers and call
it from the Docker, Podman, Kubernetes and VM wrappers once the gateway is
healthy and registered. It runs the same command the upgrade notes give
operators, against the repository's own providers/ directory, so the lanes
exercise the documented path rather than a test-only shortcut.
Lands before the default changes: a same-ID user profile already shadows a
built-in, so importing works today.
Signed-off-by: Philippe Martin <phmartin@redhat.com>
* refactor(providers)!: make provider profiles import-only
OpenShell compiled fifteen provider profile YAML files into every release binary
and selected them by default, so a fresh gateway published a catalog it was
never configured with. Those profiles are not image-neutral: their binary
selectors name paths that exist in the OpenShell Community image, so changing
the sandbox image could make a profile inert while the catalog still advertised
it. Every endpoint, credential name and binary path in providers/ was also
effectively part of the 0.1.0 public contract.
A gateway's catalog is now exactly what an operator imported:
- providers/*.yaml is no longer include_str!'d, and builtin_profiles() is gone
from the openshell-providers API
- the default provider_profile_sources is [{ type = "user" }], and a gateway
with nothing imported reaches ready and serves an empty catalog — an empty
catalog is a valid state, not a startup failure
- the builtin source type is removed from the configuration schema, and a
gateway.toml that still names it is rejected at parse time with the import
command rather than an unknown-variant error
- nothing reserves the canonical identifiers any more, so github, pypi,
anthropic and the rest import at their own IDs and the imported profile is the
only definition for that ID; the static_fallback that kept a shipped
definition resident behind an imported one is gone with the source that
produced it
A collision between an interceptor-vended profile and an imported one still
fails closed, which is the behavior that shadowing quietly bypassed for
built-ins.
Operators upgrading should export the profiles their deployment relies on
before upgrading, or copy them from the providers/ directory of the matching
release tag, then import them at the scope their providers use.
Signed-off-by: Philippe Martin <phmartin@redhat.com>
* docs(providers): document import-only provider profiles
Rewrite the provider profile documentation around a catalog the operator builds
rather than one the gateway ships.
- gateway-config: the default is [{ type = "user" }], the builtin source type is
gone and rejected at startup, and an empty catalog is a valid ready state
- profiles: replace the built-in profile table and its shadowing semantics with
the import workflow, and say plainly that a profile whose binary paths do not
match the image is inert
- manage-providers: replace the two fixed provider-type tables with
`provider list-profiles`, and describe catalog-driven command inference
- inference-routing: drop the "still loads built-in profiles for compatibility"
transition text; its migration walkthrough is now the normal path
- quickstart and the Docker Compose, GitHub, AWS, Google Cloud and Vertex
tutorials: import the profile before creating the provider, since nothing
resolves without it
- release notes: a 0.1.0 migration section covering export-before-upgrade,
importing from the release tag's providers/ directory, what happens to a
provider whose profile is missing, and the removal of the legacy type aliases
- README, architecture, the openshell-cli and debug-openshell-cluster skills,
and the governance interceptor example follow the same change; the example's
smoke assertion now imports a profile to prove an authoritative interceptor
hides it
Signed-off-by: Philippe Martin <phmartin@redhat.com>
* fix(cli,e2e): distinguish catalog failures and authenticate OIDC seeding
Addresses two findings from review (GATOR-c1bd9867-02 and -03).
The provider profile catalog lookup collapsed its error into an empty
catalog, so an authorization, availability or transport failure on
ListProviderProfiles was indistinguishable from a gateway that genuinely has
no profiles. Command inference then resolved nothing and the sandbox was
created without the provider it needed, deferring the failure to the workload.
Keep the lookup's outcome instead of discarding it. The credential warning is
advisory and still degrades to its generic form, but inference now consults the
catalog only when there is a command to resolve and surfaces the lookup failure
when there is, naming the gateway and pointing at explicit --provider selection.
Two regression tests cover it through a fake gateway whose ListProviderProfiles
returns UNAVAILABLE while every other RPC succeeds: sandbox creation fails
without sending a provider-less create request, and a sandbox with no trailing
command still succeeds because it needs no catalog.
The e2e profile seeding also ran unauthenticated in the OIDC lanes. Those lanes
deliberately skip gateway registration and start the gateway without a TLS
client CA, so no mTLS identity exists and no token has been acquired when the
import runs; the wrapper exited during setup. Skipping the import is not
sufficient because the provider tests now require the claude-code profile.
Add e2e_register_oidc_admin_session, which mints an administrator token with
Keycloak's password grant — the same grant the OIDC test helpers use — and
writes the gateway metadata and token bundle that an interactive login would
have stored, so the import runs as an authenticated administrator. The mTLS
lanes keep the direct import unchanged. Both affected wrappers are covered:
with-podman-gateway.sh had the same defect as with-docker-gateway.sh.
Signed-off-by: Philippe Martin <phmartin@redhat.com>
* refactor(cli)!: remove command-derived provider attachment
Addresses GATOR-c1bd9867-01.
A profile's `binaries` list authorizes a binary to reach that profile's
endpoints. It is not a statement that running the binary asks for the provider,
and reading it as attachment intent let a command silently gain provider
authority: the aws-s3 example declares /bin/bash, so `sandbox create -- bash`
resolved an existing aws-s3 provider and attached its credential-backed
capability and network policy to the shell and its descendants. Attachment of
an already-created provider never prompted, so the confirmation flow did not
guard it.
The distinction that would make inference sound — whether a declared binary is
a profile's client or merely a permitted runtime — cannot be expressed:
NetworkBinary carries only a path, and the field that encoded it was removed in
0.1.0. Any substitute is a guess. Restricting the guess to a unique claimant
does not help, because uniqueness measures how sparse the catalog is rather
than what the user intended, and a compiled list of "generic" commands could
never cover an unbounded operator catalog.
Remove trailing-command inference rather than approximate it. A provider is
attached only when named with --provider, which still creates a missing
provider from local discovery when the name matches an imported profile ID.
Sandboxes with no providers remain a normal, fully supported state.
Removing inference also settles GATOR-c1bd9867-02: with no consumer deriving
authority from the catalog, its only remaining use is the advisory credential
warning, so a failed lookup degrades that warning instead of blocking creation.
The regression test now asserts that an unreachable catalog still creates the
sandbox and attaches nothing.
While repurposing the deduplication test, a pre-existing defect surfaced:
repeating a name in --provider auto-created it twice and the second attempt
failed with "provider already exists", because the explicit pass never
consulted the set of names it had already handled. Guard it.
Signed-off-by: Philippe Martin <phmartin@redhat.com>
* fix(e2e): load the OIDC token and trust the gateway when seeding profiles
Addresses the carried finding GATOR-c1bd9867-03.
e2e_register_oidc_admin_session established a session the CLI never used. The
CLI decides whether to load a stored bearer token by matching on the auth_mode
field of the gateway metadata alone; the helper omitted that field, so the
metadata fell through to the default arm and oidc_token.json stayed on disk
unread. The profile import went out unauthenticated exactly as it had before
the helper existed. The helper also installed no trust anchor, leaving the CLI
unable to verify the gateway's self-signed serving certificate.
Write auth_mode = "oidc" so the stored token is loaded, and install the CA at
<gateway>/mtls/ca.crt. Only the CA is installed: with no client certificate or
key on disk the CLI falls back to CA-only server verification and authenticates
with the bearer token, which is what these lanes need because they start the
gateway without --tls-client-ca. Certificate verification stays on; no insecure
transport override is introduced.
Assert the session before anything depends on it. ListProviderProfiles is
annotated auth_mode: "bearer", so it cannot succeed unless the token was loaded
and accepted. The helper now fails at that point, naming the gateway config
directory and echoing the CLI output, rather than letting the defect surface
later as an opaque profile import error.
Both affected wrappers pass the PKI directory and CLI binary the helper needs;
the mTLS lanes are untouched.
Signed-off-by: Philippe Martin <phmartin@redhat.com>
* fix(e2e): request the OpenShell scopes when minting the admin token
The OIDC lanes failed at the session assertion added in b25dd0298 with
PERMISSION_DENIED and "scope 'provider:read' required". Authentication was
working -- the gateway logged ListProviderProfiles at gRPC status 7, which it
can only reach once the bearer token has been loaded and accepted. The token
simply carried no OpenShell scope.
sandbox:*, provider:*, config:*, workspace:* and openshell:all are optional
client scopes on the openshell-cli client in scripts/keycloak-realm.json, so
Keycloak mints them only when the request asks for them. The password grant
here asked for nothing, leaving the realm defaults (openid, profile, email,
roles, web-origins, acr) and an access token that authorizes no RPC.
Request "openid openshell:all", as e2e/python/oidc/oidc_auth_test.py already
does for its administrator tokens. openshell:all is SCOPE_ALL in
crates/openshell-server/src/auth/authz.rs, so one scope covers the setup calls
without enumerating them. Verified against the realm: the token goes from
"email profile openid" to "email profile openshell:all openid".
Record the same scopes in metadata.json. The helper writes the bundle
openshell gateway login would have stored, and oidc_scopes is the field that
login path reads back, so leaving it out would misdescribe the stored token.
Signed-off-by: Philippe Martin <phmartin@redhat.com>
* fix(e2e): seed provider profiles once per VM lane gateway
The VM lanes failed on the second test target with "custom provider profile
'anthropic' already exists" for all fifteen example profiles, and the import
exited non-zero.
e2e_import_example_provider_profiles was called from run_e2e_test, so it ran
once per target -- four times against one long-lived gateway. That was
harmless while import overwrote silently, but profiles are now import-only:
ImportProviderProfiles is create-only and reports an existing id as an
error-severity diagnostic, with no overwrite flag on the request. The first
import therefore succeeds and every later one fails.
Hoist the call to just after the conformance run, which is where the docker,
podman and kube lanes already seed their catalogs. The profiles persist for
the gateway's lifetime, so every target still finds them, and the
E2E_TEST_OVERRIDE path is covered by the same single call.
Signed-off-by: Philippe Martin <phmartin@redhat.com>
* test(e2e): name the provider explicitly in the OIDC workspace-user test
user_can_create_sandbox_with_inferred_provider_command reached the gateway
once the admin session was fixed, and then failed: it asserted a
missing-provider error, but the sandbox was created and died provisioning
with "failed to spawn sandbox entrypoint process 'claude-code'".
The test drove provider resolution by passing claude-code as the trailing
command and relying on the CLI to infer the provider type from it. That
inference is what this branch removed, so the trailing word is now nothing
but an entrypoint, and the image has no such binary.
The regression the test guards is not inference itself: it is that a
workspace user resolving a provider is not gated behind Platform Admin.
Name the provider with --provider, the only remaining way to attach one.
The CLI still has to fetch the claude-code profile before it can auto-create
the provider, so the lookup a workspace user must be allowed to make still
happens, and auto-creation still stops at the non-interactive branch with
"missing required provider". Both assertions therefore keep their meaning.
Rename the test and rework its comments to describe what it now exercises;
the old name would otherwise outlive the behavior it was named for.
Signed-off-by: Philippe Martin <phmartin@redhat.com>
* fix(providers): reconcile import-only profiles with main
Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>
---------
Signed-off-by: Philippe Martin <phmartin@redhat.com>
Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>
Co-authored-by: John Myers <9696606+johntmyers@users.noreply.github.com>
Drew Newberry