A DNS64 answer like 64:ff9b::a00:5 is really 10.0.0.5, but the SSRF
checks treated it as a public IPv6 address. Addresses inside a NAT64
prefix are now checked as the IPv4 address they embed, in policy DNS
and in the CONNECT path. The well-known prefix is always known; other
prefixes come from the new nat64_prefixes driver setting, or from
ipv4only.arpa discovery when the supervisor starts.
policy_dns_ipv6_egress and nat64_prefixes can now be set in the Docker,
Podman, Kubernetes and VM driver configs. Before this, only auto was
reachable in practice.
The supervisor logs its IPv6 egress decision (requested mode, result,
default routes, route state) and the NAT64 prefixes it uses as OCSF
config events.
Tests: route detection fixtures per backend, a start_mediated test with
a fake mediation source and DNS64 upstream, driver arg tests, and an
e2e check of the decision event.
Signed-off-by: Joffref <mjoffre@blaxel.ai>
Mediated policy DNS always answered AAAA queries with NOERROR/NODATA and
resolved A queries upstream as A only. On IPv6-only hosts behind
NAT64/DNS64 the trusted resolver returns no A records, so every
policy-allowed name failed with policy_dns_upstream_no_data and native
TCP egress was unusable.
Add a driver-owned supervisor flag, --policy-dns-ipv6-egress
{auto,enabled,disabled}. The default auto mode enables AAAA answers only
when the supervisor network namespace has an IPv6 default route and no
IPv4 default route, so dual-stack and IPv4-only hosts keep the current
A-record fallback. AAAA answers use the existing epoch-scoped synthetic
IPv6 pool and are pinned and dialed through the same resolved-endpoint
store and destination validation as IPv4 answers.
Signed-off-by: Joffref <mjoffre@blaxel.ai>
Closes#3746
Use Homebrew atomic_write for exact legacy config migrations and keep the first-install write distinct in the formula test.
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* fix(snap): require mTLS for the snap gateway
Replace the installer opt-in with an authenticated snap gateway. The wrapper
no longer forces plaintext, so the gateway serves TLS from the bundle it
already generates in $SNAP_COMMON/tls. The install hook writes a config that
enables mTLS user auth instead of unauthenticated access, and a new
post-refresh hook migrates the exact legacy default on existing installs.
install.sh waits for the gateway, detects whether it serves TLS, copies the
client bundle into the target user's snap state directory, and registers the
gateway over HTTPS. Older plaintext snap revisions still register over HTTP
with a warning. The release canary asserts mTLS auth and HTTPS registration.
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* fix(snap): pass config preflight and detect the mTLS gateway reliably
An explicit [openshell.gateway.mtls_auth] table fails config preflight,
which validates mTLS auth before the local TLS bundle supplies the client
CA. Write a default that pins the Docker driver instead; with the wrapper's
TLS bundle the gateway requires client certificates and enables mTLS user
auth automatically, as the native packages do.
The mTLS gateway rejects TLS handshakes without a client certificate, and
it still answers plaintext loopback HTTP for sandbox service routing, so the
installer could misdetect it as a legacy plaintext gateway. Probe HTTPS with
the root-owned client bundle, and treat a gateway as legacy only when a
plaintext gRPC Health call succeeds.
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* chore(snap): simplify install hook comment
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* fix(snap): migrate insecure gateway configs on refresh
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* refactor(snap): simplify mTLS detection and config migration
Detect the mTLS snap from the installed revision's post-refresh hook instead
of probing plaintext gRPC, and drop the scheme global. Remove the installer's
pre-hook config fallback, which is dead now that every channel ships the
install hook and which wrote the insecure default. Give the install hook a
single write path with a simple backup name, and shorten the manual client
certificate steps.
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* fix(snap): stop keeping a copy of replaced insecure configs
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* feat(install): make the snap an opt-in install method
Stop selecting the OpenShell snap just because the snap command exists. Linux
installs default to the Debian or RPM package; OPENSHELL_INSTALL_METHOD=snap
(or deb, rpm) selects the package explicitly. Hosts that already have the
OpenShell snap keep refreshing it rather than gaining a second gateway on the
same port. The release canary and snap repro script opt in explicitly.
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* fix(snap): let the gateway auto-detect its compute driver
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* fix(snap): restart the gateway after refresh
Published revisions use refresh-mode: endure, and snapd honors the old
revision's setting during a refresh, so the plaintext gateway kept running
with the migrated config unused until a manual restart. Restart the gateway
from the post-refresh hook so the mTLS config takes effect immediately.
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* docs(snap): drop refresh notes from the snap description
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* docs(snap): trim snap refresh notes from installation docs
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
---------
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
The /_ws_tunnel endpoint pipes a WebSocket into the full gRPC service. On a
plaintext loopback gateway any web page could open it, since browsers do not
apply CORS to WebSocket upgrades. Mount the tunnel only when
enable_websocket_tunnel is set (config file, --enable-websocket-tunnel, or
OPENSHELL_ENABLE_WEBSOCKET_TUNNEL; server.enableWebsocketTunnel in Helm).
BREAKING CHANGE: gateways behind an authenticating edge proxy must set
enable_websocket_tunnel = true for CLI edge-tunnel connections.
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* docs(readme): streamline README and move reference detail to docs
Restructure the README as a short path from overview to quickstart to
further reading. Move prerelease install steps into the installation
guide and telemetry build flags into a new observability page. Fix
broken docs links and outdated runtime and credential descriptions.
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* docs(readme): describe 0.1.0 as adding new isolation primitives
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* docs(architecture): add policy prover as a gateway component
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* docs: describe OpenShell as a runtime for fleets of autonomous AI agents
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* docs(architecture): describe policy prover as formal verification
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* docs(architecture): name OpenShell Sandbox in component table
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* docs(architecture): fold isolation backend into supervisor row
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* docs(architecture): mention formal verification in overview
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* docs(run-agent): use the published OpenCode image in the first-agent guide
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* docs(inference): correct provider examples and readiness
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
* docs(providers): correct Google binding and provider selection
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
* docs(readme): sharpen value prop, how it works, and explore further
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* docs(run-agent): use example Anthropic profile and add policy advisor step
Import the example Anthropic profile, which now allows OpenCode, instead
of editing it with sed. Add a step that shows how to review and approve
mechanistic policy proposals as the agent needs more access.
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* feat(providers): add OpenRouter example for OpenCode
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* docs(run-agent): run OpenCode against OpenRouter with a free model
Add an example OpenRouter provider profile scoped to OpenCode and switch
the first-agent guide to it, using a free Nemotron model so readers do
not need OpenRouter credits. Revert the OpenCode binary added to the
example Anthropic profile.
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* docs(readme): link first-agent guide and add agent skills section
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
---------
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
Co-authored-by: Piotr Mlocek <pmlocek@nvidia.com>
* docs(fern): publish and order versioned release docs
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
* docs(fern): remove version availability badges
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
* docs(fern): keep version badges optional
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
---------
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
* fix(policy): propose rules for unknown DNS hosts
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
* docs(policy): clarify synthetic DNS use across protocols
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
* fix(policy): harden unknown-host DNS observations
- Emit the policy_dns_ineligible denial for every unknown name and
report observation staging failures as DNS failure events.
- Refuse unknown names during fail-closed quarantine and after the
observation budget, now a quarter of each address family's pool.
- Pin transparent TCP to the mapping of the deciding policy generation
so a reload between DNS and authorization fails closed.
- Stop Docker workloads from inheriting host DNS search domains, which
let the first expanded short name claim an observation address.
- Share mechanistic draft polling in conformance, register
new-hostname-proposal in the installed suite, and update docs.
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
* test(policy): build policy DNS proxy tests on every target
The proxy tests name PolicyEndpointId, which proxy.rs imported only on
Linux, so the macOS test build failed. Import it for test builds too.
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
* fix(policy): name DNS queries and mapped hosts in OCSF denials
DNS denial and failure events attached port 53 to the queried name,
which read as a connection to that host. They now carry only the name.
Transparent TCP denials for a policy DNS address show the mapped
hostname and keep the synthetic address in dst_endpoint.ip.
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
---------
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
* docs(extensibility): reorganize extensibility and middleware guides
Add an extensibility overview that introduces the extension protocol and
links every extension point, and move extension caller authentication to
a shared page used by middleware and gateway interceptors.
Split the supervisor middleware guide into an overview, a Configure and
Operate guide, and a Middleware Operations reference for service authors.
The overview explains when to use middleware, shows where it runs, lists
current limitations, and defines the service contract. Configure and
Operate covers policy attachment, service registration, failure behavior,
and observability. Middleware Operations describes HTTP request, HTTP
response, and WebSocket message operations with shared inputs and results,
per-operation diagrams, and detail accordions.
Pin page slugs so links resolve, redirect the replaced dev middleware
URL, and update reference and architecture links.
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
* docs(middleware): clarify navigation and service contracts
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
* docs(middleware): drop obsolete dev URL redirect
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
---------
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
* fix(install): avoid installing incompatible docker snap
The work to land RFC-0012 added new restrictions when interacting with
Docker by setting `NoNewPrivs`. This prevents the `docker` snap from
transitioning its AppArmor profile from `snap.docker.dockerd` to
`docker-default` when it tries to launch a container. Thus, the `docker`
snap is currently incompatible with OpenShell.
This commit prevents `install.sh` from installing the `docker` snap
before installing the `openshell` snap, and instead requires the user to
install a non-snap Docker daemon before proceeding with installing the
snap. Systems without the `snap` command are unaffected, since they
install native packages without checking for the presence of Docker or
other compute providers.
Signed-off-by: Oliver Calder <oliver.calder@canonical.com>
* fix(snap): only snapd 2.76 for openshell snap since store installs work
Signed-off-by: Oliver Calder <oliver.calder@canonical.com>
* fixup! fix(install): avoid installing incompatible docker snap
Signed-off-by: Oliver Calder <oliver.calder@canonical.com>
* fixup! fix(snap): only snapd 2.76 for openshell snap since store installs work
Signed-off-by: Oliver Calder <oliver.calder@canonical.com>
---------
Signed-off-by: Oliver Calder <oliver.calder@canonical.com>
* perf(server): enable WAL and NORMAL sync for the SQLite store
On-disk SQLite stores ran with sqlx defaults: rollback journal
(`journal_mode=delete`) and `synchronous=FULL`. Every autocommit write paid
several fsyncs and blocked readers while it held the lock, so gateway hot
paths made of many small writes serialized on disk latency. The clearest
case is `openshell forward service`, which mints and revokes an SSH session
token around every forwarded TCP connection: two commits per connection,
tens of milliseconds each on a virtual disk, wall clock linear in the
number of concurrent connections, and enough queueing that bursts hit the
per-sandbox connection cap and get refused.
Switch on-disk databases to WAL with `synchronous=NORMAL`. The mode change
runs once on a single connection before the pool opens: entering WAL needs
exclusive access to the file, so doing it up front means pool connections
only ever re-apply the pragma to a file already in WAL mode, and a failure
surfaces as one clear connect error. The first start after upgrading an
existing database therefore needs the file to be otherwise unopened.
`synchronous` is applied through the connect options on every pooled
connection. In-memory databases keep their defaults. A crash can now roll
back the most recent transactions without corrupting the database, which
is the standard WAL trade-off and fits the single-node scope of the SQLite
backend.
Tests cover a fresh store, an existing rollback-journal file that must be
switched on connect, sidecar permissions, and concurrent readers under a
burst of insert-then-update writes. Architecture, configuration and Helm
docs describe the durability trade-off, the sidecar files, and the local
filesystem requirement.
Signed-off-by: Jason T. Greene <jason.greene@redhat.com>
* fix(server): keep SQLite commits durable except SSH session issuance
WAL with synchronous=NORMAL can roll back acknowledged commits after a
power loss or kernel crash, including SSH session revocations and other
authorization-tightening writes. Run the main pool with synchronous=FULL
so every acknowledged write is durable; in WAL mode that is a single
fsync of the WAL per commit.
Add Store::create_relaxed for inserts that are safe to lose, and use it
only for SSH session issuance: a dropped token just fails validation.
On file-backed SQLite it runs on a dedicated single-connection pool with
synchronous=NORMAL. Both pools share one WAL, so the next FULL commit
also makes earlier relaxed commits durable. Postgres treats it as an
ordinary durable MustCreate insert.
Signed-off-by: Mrunal Patel <mrunalp@gmail.com>
---------
Signed-off-by: Jason T. Greene <jason.greene@redhat.com>
Signed-off-by: Mrunal Patel <mrunalp@gmail.com>
Co-authored-by: Mrunal Patel <mrunalp@gmail.com>
* fix(podman): restore host gateway alias mediation
Signed-off-by: Gordon Sim <gsim@redhat.com>
* fix(podman-e2e-tests): enable broader test podman e2e coverage
Signed-off-by: Gordon Sim <gsim@redhat.com>
* fix(tests): make test more reliable
Signed-off-by: Gordon Sim <gsim@redhat.com>
* fix(podman): fix macos linting error
Signed-off-by: Gordon Sim <gsim@redhat.com>
---------
Signed-off-by: Gordon Sim <gsim@redhat.com>
* fix(snap): update stale snap docs and tests
Previously, the `openshell` snap required the `docker` snap. Now, it
works with any Docker daemon running on the system. Furthermore, the
`snap-declaration` assertion on the `openshell` snap when installed from
the Snap Store causes the `openshell` snap to always connect to the
system `:docker` slot, rather than a slot provided by the `docker` snap.
This commit updates the documentation, including the `description` field in
`snapcraft.yaml`, to ensure that all information is correct and
up-to-date.
Additionally, some tests connected the `openshell:docker` plug to the
`docker` snap's `docker:docker-daemon` slot, which is inconsistent with
how the `openshell` snap operates when installed from the store. Update
those tests to connect to the system `:docker` slot as well.
Signed-off-by: Oliver Calder <oliver.calder@canonical.com>
* fix(snap): require snapd 2.76 for openshell snap
Signed-off-by: Oliver Calder <oliver.calder@canonical.com>
* fix(nix): align snap gateway reproducer timeout with release-canary
Signed-off-by: Oliver Calder <oliver.calder@canonical.com>
* fix(snap): require snapd 2.77 for openshell snap
Signed-off-by: Oliver Calder <oliver.calder@canonical.com>
* fixup! fix(snap): require snapd 2.77 for openshell snap
Signed-off-by: Oliver Calder <oliver.calder@canonical.com>
* feat(snap): install openshell snap via install.sh when snap available
Change `install.sh` to install the `openshell` snap by default when
snapd is installed on the host. This installs the snap from the
`latest/stable` channel, which should match the most up-to-date release
tag on github.
If `OPENSHELL_VERSION=dev` is set for `install.sh`, then it will install
the `openshell` snap from the `latest/edge` channel, which matches the
latest dev release available on github.
The `openshell` snap currently requires Docker in order to function. If
a Docker daemon is already installed on the system, it will be used by
the `openshell` snap. Otherwise, `install.sh` will install the `docker`
snap first, wait for the Docker daemon to be ready, and then install the
`openshell` snap.
Also, update the `release-canary.yml` to split the `ubuntu-snap` job
into `ubuntu-snap-system-docker` and `ubuntu-snap-provisions-docker`,
which test the two aforementioned scenarios. Previously, `ubuntu-snap`
manually installed a given snap artifact as built from CI, but with
these new jobs, it instead uses `install.sh` to install the published
`openshell` snap from the `latest/edge` track, thus matching the
behavior of the other release canary jobs.
Make corresponding changes to the `nix` guest reproducer, documentation,
and tests.
Signed-off-by: Oliver Calder <oliver.calder@canonical.com>
* fix(snap): configure local gateway authentication
The `openshell` snap runs the gateway as a systemd system service, which
runs as root. Thus, the mTLS certs are generated by root and stored in a
root-owned directory to which non-root users do not have access. For
this reason, the snap's `openshell-gateway-wrapper` script sets
`OPENSHELL_DISABLE_TLS=true`.
This commit ensures that the `openshell` snap's gateway allows
unauthenticated local access by writing a default `gateway.toml`
configuration file during the install hook, which runs after the snap is
first installed but before services are started. The config file
contains sets `allow_unauthenticated_users = true`.
Signed-off-by: Oliver Calder <oliver.calder@canonical.com>
* fix(install): configure snap gateway authentication
Recently, a new install hook was added which writes a default config
file for the `openshell` snap to allow unauthenticated local access to
the gateway. This is because the gateway service runs as root and the
mTLS certificates are not accessible to non-root users.
However, the `install.sh` script installs the `openshell` snap from the
snap store, and the published version may not yet have that new install
hook. Or, the user may already have the snap installed, in which case
the install hook does not run. In either case, we need `install.sh` to
ensure that the config file is written to set the gateway auth to
`allow_unauthenticated_users = true`, and then restart the openshell
gateway service.
Signed-off-by: Oliver Calder <oliver.calder@canonical.com>
* fixup! feat(snap): install openshell snap via install.sh when snap available
Signed-off-by: Oliver Calder <oliver.calder@canonical.com>
---------
Signed-off-by: Oliver Calder <oliver.calder@canonical.com>
* fix(sandbox): keep boundary connection live under stalled relays
Stalled relay streams could exhaust the shared HTTP/2 connection window and
freeze DNS, exec, and control traffic. Size flow control so the connection
window exceeds all streams' windows plus a reserve, and cap relay streams.
Recovery no longer replaces a live connection after a single stream failure,
force-closes a dead transport before reattaching, and retries while the
sandbox still reports the old same-epoch connection as active. The sandbox
now reports attach/confirm rejections instead of closing the stream.
Closes#3396
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* fix(sandbox): warn only on sustained relay stream waits
Package managers routinely hold more than the relay budget's worth of connections for short bursts; report only waits that last longer than five seconds.
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* fix(sandbox): keep aborted channel cached until reattach completes
Clearing the channel cache during the reattach retry window let concurrent requests cache and use a connection that never attached or confirmed. Keep the aborted generation installed so they fail fast and wait on the in-progress recovery.
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
---------
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* fix(driver-kubernetes-secrets)!: store provider credentials in one namespace
The Kubernetes Secrets credential driver now stores every credential in its
configured namespace in all workspace modes and rejects handles that reference
any other namespace before contacting the Kubernetes API. The gateway reaches
credential Secrets through the Role in that namespace; this allows removing the
Secret rules from the ClusterRole.
- Remove the workspace_mode, gateway_id, and allow_reference_namespace driver
settings and stop rendering them from Helm. Configurations that set them fail
at startup. Existing credential state is not migrated.
- Add server.credentialDrivers.kubernetesSecrets.createNamespace to provision a
dedicated credential namespace. The namespace is kept on uninstall, adopted
by a reinstall of the same release, and left untouched when something else owns
it.
- Update the gateway config reference, Kubernetes setup docs, 0.1.0
upgrade guide, compute-runtime architecture, and cluster debugging skill.
Signed-off-by: Kris Hicks <khicks@nvidia.com>
* fix(helm): reduce gateway Secret permissions
Remove the gateway's Secret list permission in every workspace mode and
grant source Secret reads through a Role in the sandbox namespace.
Bootstrap Secret cleanup deletes Secrets by exact name instead of listing
them.
- Grant get on the copied client TLS and image-pull Secrets through a Role in
the sandbox namespace. The ClusterRole keeps get and patch on those names
for the ownership check and server-side apply into workspace namespaces.
- Delete sandbox and supervisor bootstrap Secrets by exact name, derived from
the runtime generation recorded on the Sandbox and, on restart, the target
generation, tolerating 404. The generation annotation is cleared only after
cleanup succeeds, and each bootstrap Secret has a Pod owner reference, so
garbage collection removes any generation the driver does not name.
- Drop Secret list from the ClusterRole and the shared-mode sandbox Role.
- Extend the managed e2e RBAC checks to Secret list.
- Update the Kubernetes setup and sandbox runtime docs, compute-runtime
architecture, and cluster debugging skill.
Signed-off-by: Kris Hicks <khicks@nvidia.com>
* fix(driver-kubernetes): stage workspace Secrets per runtime generation
Every Secret the Kubernetes driver writes into a workspace namespace is
now scoped to one sandbox runtime generation, immutable, and created
with create only, so the gateway never reads, patches, or adopts an
existing Secret there. This removes the gateway's cluster-wide get and
patch on the copied client TLS and image-pull Secret names.
- In managed mode, create an immutable copy of each configured
image-pull Secret per generation, named os-pull-<id>-<generation>-<n>
and owned by the generation's workload and supervisor Pods. Pods and
the restarted Sandbox template reference those names, and generation
cleanup deletes them by name. A Secret already holding a generation
name fails the create.
- Outside shared mode, stage the gateway client TLS material into the
supervisor bootstrap Secret instead of copying the client TLS Secret
into the workspace namespace.
- Remove the fixed-name TLS and image-pull copies, the target ownership
read, and the ClusterRole get and patch rule on the copied names.
Source reads stay in the sandbox-namespace Role.
- Update the managed e2e to expect generation image-pull Secrets and
client TLS material in the supervisor bootstrap Secret, and to check
that the gateway cannot read the copied names in workspace namespaces.
- Update the gateway config and compute driver references, Kubernetes
setup and sandbox runtime docs, compute-runtime architecture, driver
README, and cluster debugging skill.
Signed-off-by: Kris Hicks <khicks@nvidia.com>
* fix(helm)!: grant operator-mode Secret permissions through the workspace chart
The operator-mode gateway ClusterRole grants no Secret permissions.
The openshell-workspace chart Role, installed in each operator-managed
namespace, grants the gateway create and delete on Secrets for sandbox
runtime generations. Operator-managed namespaces require the workspace
chart.
- Fail the chart tests on any ClusterRole rule that includes Secrets in
operator and shared modes.
- Install the workspace chart when the operator e2e provisions a
namespace, and assert that the gateway has no Secret permissions in a
namespace without it.
- Update the Kubernetes setup docs, 0.1.0 upgrade guide, compute-runtime
architecture, and cluster debugging skill.
Signed-off-by: Kris Hicks <khicks@nvidia.com>
---------
Signed-off-by: Kris Hicks <khicks@nvidia.com>
* fix(security): pin supplied executable identity chains
Signed-off-by: Jim Meyer <jimeyer@nvidia.com>
* docs(security): document executable identity chain pinning
Signed-off-by: Jim Meyer <jimeyer@nvidia.com>
* fix(binary-identity): compile Linux ancestry hashing
Signed-off-by: Jim Meyer <jimeyer@nvidia.com>
* test(binary-identity): avoid cross-label ancestry fixture
Signed-off-by: Jim Meyer <jimeyer@nvidia.com>
* test(e2e): choose distinct denied TCP port
Fix flaky test due to sequential port assignment on MacOS
Signed-off-by: Jim Meyer <jimeyer@nvidia.com>
* fix(identity): bound executable evidence cache
Reject identity chains atomically when the supervisor cache reaches its hard limit, and preserve existing pins without eviction. Classify malformed or conflicting evidence separately from policy denials at the staged TCP boundary.
Signed-off-by: Jim Meyer <jimeyer@nvidia.com>
---------
Signed-off-by: Jim Meyer <jimeyer@nvidia.com>
* fix(sandbox): fall back to plain seccomp listener when WAIT_KILLABLE_RECV is unavailable
SECCOMP_FILTER_FLAG_WAIT_KILLABLE_RECV was added in Linux 5.19. On older
kernels (for example RHEL 9.x / 5.14 nodes such as RHCOS on OpenShift) the
flag is rejected with EINVAL, which made the capability-free sandbox fail to
start during the notification probe with "notification launcher disappeared".
Install the notification listener with WAIT_KILLABLE_RECV when the kernel
supports it and fall back to a plain NEW_LISTENER on EINVAL. The fallback
listener records wait_killable_recv = false: its notification receive is
uninterruptible, but the sandbox is otherwise fully functional.
Signed-off-by: Akram <akram.benaissi@gmail.com>
* fix(sandbox): address review — legacy read-only listener mode + cancellation invariant
Follow-up to the PR review (GATOR-de00bfcc-01 / mrunalp): make the < 5.19
fallback cancellation-safe instead of racing broker writes.
- Record an explicit ListenerMode (Killable vs LegacyReadOnly); add
writes_disabled()/mode() and emit the selected mode in qualification output
(seccomp_listener_mode).
- Centralize task-memory output writes behind NotificationListener::
write_task_output; in LegacyReadOnly mode getpeername, accept/accept4 with a
non-null address, and sendmmsg length write-backs fail closed with EOPNOTSUPP.
accept with a null address, socket/connect/bind/listen/sendto/sendmsg keep
working (copied inputs, scalar responses, atomic ADDFD_SEND).
- Enforce the launch invariant `cancellation || task_memory_writes_disabled` in
SandboxConfirmEvidence::validate() rather than dropping cancellation
unconditionally; add task_memory_writes_disabled to SeccompEvidence.
- Add per-path fail-closed tests (write_task_output, write_socket_addr, a real
plain listener installed on a modern kernel) and confirmation-invariant tests.
- Correct the flag-semantics comments and document both modes plus the reduced
legacy syscall compatibility in architecture/sandbox.md.
Signed-off-by: Akram <akram.benaissi@gmail.com>
* docs: document legacy read-only sandbox mode on kernels before 5.19
Kernels without SECCOMP_FILTER_FLAG_WAIT_KILLABLE_RECV (< 5.19, e.g. RHEL 9.x /
RHCOS 5.14) run the sandbox in a legacy read-only cancellation mode where the
broker fails closed with EOPNOTSUPP on the mediated operations that write results
back into workload memory (getpeername, accept/accept4 with a non-null address,
sendmmsg length write-backs). Document this observable behavior and its syscall
limitations in the public Fern docs: the support-matrix kernel requirements and
the OpenShift runtime guidance.
Signed-off-by: Akram <akram.benaissi@gmail.com>
* style(sandbox): satisfy rustfmt and clippy doc_markdown
Match the pinned rustfmt (Rust 1.95.0) line-wrapping for the write_task_output
call, and backtick `legacy_read_only` in the qualification-report doc comment
so clippy::doc_markdown (-D warnings) passes.
Signed-off-by: Akram <akram.benaissi@gmail.com>
* fix(sandbox): migrate task_memory_writes_disabled into backend protocol
Add the `task_memory_writes_disabled` field to `SeccompEvidence` in the
backend protocol contract and relax the validation from requiring
`cancellation` to accepting `cancellation || task_memory_writes_disabled`.
This completes the rebase migration missed by the isolation-interface
refactor: the sandbox reports this field but the backend struct lacked
it, and the validator rejected every pre-5.19 legacy listener.
Signed-off-by: Akram <akram.benaissi@gmail.com>
---------
Signed-off-by: Akram <akram.benaissi@gmail.com>
* fix(api): emit warning on WatchSandbox broadcast lag instead of terminating
Broadcast lag on the status, log, and platform receivers was converted to a RESOURCE_EXHAUSTED status that terminated the whole watch stream. Lag is recoverable: the receiver resumes at the oldest surviving message. Emit a SandboxStreamWarning and continue streaming instead; keep terminating on Closed. Add helpers and unit tests covering the warning payload and receiver recovery after lag.
Partially addresses #3055 (cursor/resume follow up separately).
Signed-off-by: Artem Lytvyn <alytvyn@redhat.com>
* refactor(server): group per-sandbox log bus state and stamp sequence numbers
Signed-off-by: Artem Lytvyn <alytvyn@redhat.com>
* feat(proto): add resume cursor fields to sandbox watch API
Signed-off-by: Artem Lytvyn <alytvyn@redhat.com>
* feat(server): stamp watch cursors from a shared per-sandbox sequence
Allocate cursors from a single SeqAllocator shared by the log and
platform event buses, so a sandbox's merged watch stream carries
unique, strictly increasing cursors. A single resume_after_cursor can
then unambiguously locate a client's position across both sources.
Rewrite both publish paths to allocate the sequence, stamp
event.cursor, send, and append to the tail under one lock. This
removes the previous get_mut().expect() TOCTOU race where a concurrent
remove() between the two lock sections could panic.
Signed-off-by: Artem Lytvyn <alytvyn@redhat.com>
* feat(server): serve WatchSandbox resume from cursor with gap detection
Add tail_after() to the log and platform event buses, returning every
buffered event newer than a client's resume cursor. Each PerSandbox now
tracks last_trimmed_seq (the highest seq it has evicted) so a resume is
reported as an unrecoverable ResumeGap only when this bus dropped an
event the client still needs.
Judging gaps by evictions, not by the tail's oldest seq, is required
under the shared cursor space: each bus's tail is non-contiguous in the
global sequence because the other bus owns the missing seqs, so
comparing against tail.front() would flag false gaps.
Signed-off-by: Artem Lytvyn <alytvyn@redhat.com>
* feat(server): resume WatchSandbox from cursor across log and platform buses
Wire resume_after_cursor into the watch producer. On a non-zero cursor,
replay events strictly after it from both the log and platform buses,
merge by shared cursor, and emit in order before entering the live loop.
A trimmed range on either bus is an unrecoverable gap and terminates the
stream with OUT_OF_RANGE carrying the requested and earliest-available
cursors, distinct from recoverable lag which warns and continues.
Signed-off-by: Artem Lytvyn <alytvyn@redhat.com>
* test(server): cover WatchSandbox cursor resume paths
Add handler-level tests for the resumable watch stream: replay strictly
after the client cursor, merge log and platform events in shared-cursor
order, suppress duplicates when resuming at the latest cursor, and
terminate with OUT_OF_RANGE when the requested cursor has been trimmed.
Signed-off-by: Artem Lytvyn <alytvyn@redhat.com>
* docs(api): document WatchSandbox loss-awareness and resume
Signed-off-by: Artem Lytvyn <alytvyn@redhat.com>
* fix(server): deliver watch events once and harden cursor teardown
Signed-off-by: Artem Lytvyn <alytvyn@redhat.com>
* feat(sdk): add loss-aware resumable watch_logs to Rust SDK client
Signed-off-by: Artem Lytvyn <alytvyn@redhat.com>
* fix(server): keep watch cursors monotonic across teardown and restart
Signed-off-by: Artem Lytvyn <alytvyn@redhat.com>
* fix(server): merge live watch sources by cursor before emission
Signed-off-by: Artem Lytvyn <alytvyn@redhat.com>
* fix(api): bind watch cursors to a cursor space and merge tail sources
Signed-off-by: Artem Lytvyn <alytvyn@redhat.com>
* fix(server): revalidate the watch cursor space after collecting replay
Signed-off-by: Artem Lytvyn <alytvyn@redhat.com>
* test(server): update the public RPC schema fingerprint for the string cursor
Signed-off-by: Artem Lytvyn <alytvyn@redhat.com>
* fix(server): hold watch events above the publication watermark and emit the watch lag warning before its batch
Signed-off-by: Artem Lytvyn <alytvyn@redhat.com>
* test(server): synchronize the watch live-order test with the end of initialization
Signed-off-by: Artem Lytvyn <alytvyn@redhat.com>
* fix(sdk): use canonical sandbox name in watch_logs
Signed-off-by: Artem Lytvyn <alytvyn@redhat.com>
* test(sdk): guard canonical-name addressing in watch_logs
Signed-off-by: Artem Lytvyn <alytvyn@redhat.com>
* fix(server): fix public rpc schema
Signed-off-by: Artem Lytvyn <alytvyn@redhat.com>
* fix(api): reconcile watch resume rebase
Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>
* fix(server): bound interactive relay cleanup
Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>
---------
Signed-off-by: Artem Lytvyn <alytvyn@redhat.com>
Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>
Co-authored-by: John Myers <9696606+johntmyers@users.noreply.github.com>
z3-sys 0.13 generates bindings for the Z3 5.x series and resolves
GH_RELEASE_VERSION to 5.1.0 for the prebuilt-release path. The Windows
wrapper still pinned Z3_SYS_Z3_VERSION to 4.16.0, so a prebuilt Windows
build would fetch an archive that does not match the generated bindings.
Move the wrapper pin to 5.1.0 and update the contributor, architecture,
and skill references that still named 4.16.0.
Signed-off-by: Jim Meyer <jimeyer@nvidia.com>
* feat(kubernetes): support corporate proxy CA bundle
The Kubernetes driver had no way to supply a CA bundle for the corporate
egress proxy, so an `https://` proxy with a private CA, or a TLS-intercepting
proxy, could not be used. Podman and VM already expose `proxy_ca_bundle`.
Add `proxy_ca_bundle` to `[openshell.drivers.kubernetes]` as a path the
gateway Pod reads. The gateway stages the PEM into the existing
per-generation supervisor bootstrap Secret and passes
`--upstream-proxy-ca-bundle` on the supervisor argv. That Secret is already
immutable, owner-referenced and garbage-collected, and its volume mounts
every key at /.openshell/supervisor with no items filter, so this needs no
new object kind, volume, mount, or RBAC verb, and works in shared, managed
and operator workspace modes.
The bundle is deliberately read from the gateway's filesystem rather than
referenced as an object in the sandbox namespace. It becomes a trust anchor
for every upstream the sandbox reaches, so it must stay in the gateway's
trust domain; the immutable staging Secret also keeps the anchor from
changing underneath a running sandbox.
Bound the staged bundle at 256 KiB. The shared reader's limit is exactly the
apiserver's own Secret limit and the bootstrap Secret carries four other
keys, so a bundle between the two would pass gateway startup and then fail
every sandbox create with an opaque `data: Too long`.
Delegate the URL, no_proxy, connect_by_hostname and ca_bundle rules to the
shared validate_upstream_proxy_settings, keeping the Secret-specific
credential block local: this driver accepts an explicit
`proxy_auth_allow_insecure = false` without credentials, which the shared
rules reject. This also fixes the acknowledgement being demanded for an
`https://` proxy, where the credential travels inside the verified TLS
session. Add auth_setting_label so the inline-credential diagnostic names
the Secret keys instead of proxy_auth_file, which this driver rejects as an
unknown key.
Document that the bundle should carry only the CA that signs the proxy's
certificate, or that an intercepting proxy re-signs upstream certificates
with. Public roots already reach the sandbox through the supervisor image and
its TLS stack, and the bundle is concatenated with that system store into a
single boundary control frame, so a full merged trust bundle spends the frame
budget on duplicated roots. The frame, not the apiserver Secret limit, is the
tighter of the two ceilings in practice; raising the staging bound requires
checking it.
Closes#3443
Signed-off-by: Philippe Martin <phmartin@redhat.com>
* fix(helm): quote proxy CA ConfigMap references
Signed-off-by: Philippe Martin <phmartin@redhat.com>
---------
Signed-off-by: Philippe Martin <phmartin@redhat.com>
* feat(sandbox): default to official Alpine sandbox image
default_sandbox_image() now returns docker.io/library/alpine:3.22, a generic
version-qualified official image, so a fresh install no longer depends on the
community sandbox image catalog. All compute drivers (docker, podman,
kubernetes, vm) inherit this fallback.
Part of #3116.
Signed-off-by: Akram
Signed-off-by: Akram <akram.benaissi@gmail.com>
* feat(deploy): default deployment configs to the official Alpine sandbox image
Update the shared gateway default_image, Helm chart values, the standalone
Kubernetes manifest, and the dev gateway task scripts to use
docker.io/library/alpine:3.22 instead of the community base image, consistent
with default_sandbox_image(). GPU e2e image-build base is left unchanged (CUDA
needs a glibc base).
Part of #3116.
Signed-off-by: Akram
Signed-off-by: Akram <akram.benaissi@gmail.com>
* feat(driver): default to numeric non-root identity for USER-less images
With the default sandbox image now Alpine, images that declare no OCI USER
must start instead of being rejected. When the image declares no USER and
the policy requests none, the Podman and Docker drivers now supply a numeric
non-root identity (DEFAULT_SANDBOX_UID/GID = 1000) instead of rejecting,
matching the numeric-identity behavior of the Kubernetes and VM drivers. The
supervisor's resolved-identity path runs the sandbox as a synthesized
non-root account without the account existing in the image. Images that
declare a USER keep the OCI resolution path unchanged.
Part of #3116.
Signed-off-by: Akram <akram.benaissi@gmail.com>
Signed-off-by: Evan Lezar <elezar@nvidia.com>
* test(conformance): use Alpine workload image
Signed-off-by: Evan Lezar <elezar@nvidia.com>
* refactor(policy): drop community image /app path from default policy
The restrictive default policy granted read-only access to /app, a directory
that only existed in the community base image. A generic Alpine default has no
/app, so remove it. Landlock best-effort already ignores absent paths; this
just stops advertising a community-specific layout in the default.
Part of #3116.
Signed-off-by: Akram
Signed-off-by: Akram <akram.benaissi@gmail.com>
* docs(config): document Alpine default images
Signed-off-by: Evan Lezar <elezar@nvidia.com>
* fix(podman): report early sandbox termination
Signed-off-by: Evan Lezar <elezar@nvidia.com>
* fix(podman): initialize rootless workspace ownership
Signed-off-by: Evan Lezar <elezar@nvidia.com>
* fix(sandbox): qualify NVIDIA Ubuntu default
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* fix(podman): initialize rootful default workspace
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* feat(sftp): add native sandbox adapter
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* fix(sftp): gate runtime helper support to Linux
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* fix(sftp): support standard OpenSSH file operations
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* fix(sftp): harden rename and special file handling
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* refactor(runtime): remove community image dependencies
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* test(e2e): build provider readiness tool fixture
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* fix(e2e): use a dedicated Noble fixture for Docker tests
Signed-off-by: Evan Lezar <elezar@nvidia.com>
---------
Signed-off-by: Akram
Signed-off-by: Akram <akram.benaissi@gmail.com>
Signed-off-by: Evan Lezar <elezar@nvidia.com>
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
Co-authored-by: Evan Lezar <elezar@nvidia.com>
Co-authored-by: Drew Newberry <anewberry@nvidia.com>
Joffref