Experience has shown that ordinary podman container creation can take
longer than 30 seconds for some users, so for now, this change doubles
the timeout, until the source of the slowness is found and accounted
for.
Signed-off-by: Simon Ruggier <Simon.Ruggier@gmail.com>
* fix(installer): resolve the gateway config dir like the CLI does
install.sh hardcoded $TARGET_HOME/.config/openshell in two places: the
readiness check, which reads the client certificates it expects the
gateway registration to have written, and the cleanup that removes a
stale openshell entry before re-adding it. The CLI instead keeps both
under $XDG_CONFIG_HOME/openshell whenever that variable is set, so with
the variable pointing anywhere other than ~/.config the installer and
the CLI looked in different directories: the readiness check waited for
certificates that were never going to appear there, and the cleanup left
the real stale entry in place.
Resolve the directory through as_target_user instead, which answers from
the same environment the CLI is invoked in. That covers both branches
without duplicating their logic: the same-user branch inherits the
caller's XDG_CONFIG_HOME, while the sudo and runuser branches reset the
environment and fall back to the target user's home.
An empty XDG_CONFIG_HOME falls back to $HOME/.config, following the XDG
base directory specification. The CLI diverges here: xdg_config_dir
treats a set-but-empty value as set and joins onto it, yielding the
relative path "openshell", which no installer path can usefully match.
Worth fixing separately in the CLI.
Signed-off-by: fede-kamel <fkamelhar@gmail.com>
* fix(core): treat an empty XDG_CONFIG_HOME as unset
xdg_config_dir() joined onto an empty XDG_CONFIG_HOME and resolved the
config root to the relative path "openshell", while the installer falls
back to $HOME/.config/openshell for the same value. Registration runs the
CLI with the caller's environment, so the two could disagree on where the
gateway entry lives.
Treat an empty value as unset, as the XDG base directory spec requires,
so the CLI and installer resolve the same directory. Add tests for a value
with spaces and drive registration cleanup through the helper for the
spaces and empty cases.
Signed-off-by: fede-kamel <fkamelhar@gmail.com>
---------
Signed-off-by: fede-kamel <fkamelhar@gmail.com>
* refactor(ocsf): attribute events to session-bound contexts
A supervisor process that hosts several sandboxes cannot use one
process-wide OCSF context: every event would carry the wrong sandbox
identity, and middleware requests read their sandbox identity from the
same context.
Builders now accept a ContextRef, either a borrowed context or a context
shared by a hosted session, so per-session contexts need not be leaked.
ctx() returns the context bound to the calling thread when present and
falls back to the process-wide context otherwise. OCSF JSON output
settings can likewise be bound per thread, and OpenTelemetry can parent
a span from an explicit W3C trace context instead of process
environment. A dedicated supervisor keeps using the process-wide
context.
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* refactor(supervisor): scope gateway credentials and network state to a session
The supervisor kept its gateway bearer, Sandbox Protocol bearer, and
renewal task in process-wide slots, and the network supervisor read
provider token grants, proxy CA files, and the gateway identity from
process environment. A second sandbox in the same process would have
replaced or rotated the first sandbox's credentials, reused its
provider access tokens, and exposed its denials through policy.local.
GatewayCredentials now holds one session's gateway and Sandbox Protocol
credentials and their renewal; every gateway client receives a
GatewayEndpoint that pairs the URL with them, including log push,
ConnectSupervisor and its redirects, and extension-credential rotation.
run_networking takes a NetworkSession with the sandbox identity, the
session's gateway access, its Workload API socket, its policy.local
denial source, and its proxy CA files. Provider token grants use a cache
and HTTP client owned by the session and fail closed without a Workload
API socket.
run_session supervises one generation from an explicit SessionConfig and
owner-provided SessionControls. run_sandbox_with_backend remains the
dedicated entry point: it keeps the process-wide OCSF context, reads the
same environment, binds readiness endpoints to session readiness, and
stops on SIGTERM or SIGINT. The binary shares one set of credentials
between log push and supervision, as before.
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* feat(supervisor): host concurrent sessions on session-bound runtimes
SessionHost runs several supervisor sessions in one process. Each
session gets its own Tokio runtime, and every thread that runtime starts,
plus the thread driving it, is bound to the session's OCSF context, OCSF
JSON settings, log sink, and denial log. Tasks, blocking work,
reconnects, and credential renewal spawned by a session therefore keep
its identity, while threads outside any session carry none and push no
logs. Gateway credentials stay explicit per session.
Removing a session stops its workload gracefully, or cancels it at once
while it is still starting, then shuts down only its runtime and revokes
its gateway credentials. The key stays registered until the session
thread has finished, so the same generation cannot gain a second owner
while one is stopping. A panicking session reports a failure, clears its
readiness, and is revoked the same way. Hosted sessions get a private
proxy trust directory and serve policy.local denials from their own log.
A focused test runs two sessions concurrently and checks that nested and
blocking tasks see their own identity, gateway requests carry their own
bearer, log lines reach only their own stream, and removing one leaves
the other supervising.
Closes#4699
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* refactor(supervisor): name hosted lifetimes sandbox instances
A sandbox instance is one running generation of a stable Sandbox
resource. Gateway transport connections and the gateway supervisor
sessions they carry can reconnect within one instance, so the hosted
object, its configuration, controls, host, and per-instance network,
denial, and token-grant state now use the sandbox instance name.
Rename session_host to sandbox_instance_host, run_session to
run_sandbox_instance, and SessionConfig, SessionControls, SessionHost,
and related hosted types to their SandboxInstance equivalents. Gateway
supervisor sessions, SSH sessions, and authentication session types
keep their names. No wire, configuration, or lifecycle behavior
changes.
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* refactor(supervisor): launch sandbox instances from one config and prepared bindings
Replace SandboxRunConfig and its field-by-field copy into
SandboxInstanceConfig with a single launch configuration. Live per-instance
state (gateway credentials, OCSF context and JSON settings, log sink, and
denial log) is now an explicit SandboxInstanceBindings value that the owner
prepares once, instead of placeholders the runner or host overwrote.
The dedicated binary loads launch inputs from its arguments and environment
and shares one GatewayCredentials with log push; run_sandbox_with_backend
keeps the process-wide OCSF context, health endpoints, and signal handling.
The instance host binds the same SandboxInstanceBindings to its runtime
threads and revokes their credentials when the instance ends.
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* fix(supervisor): update Windows host proxy test config
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
---------
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
A registry-sourced sandbox boot authenticated with the registry, and for a
tag resolved the manifest, before it looked at the root disk cache. It did
so on every boot, even when the image reference carries a digest and the
root disk for that digest is already cached. Each boot then waits on the
registry, and a registry outage stops every boot of an image the host
already has.
A digest names the image's content, and since #4342 a cached root disk is
published only after its manifest was verified against that digest. On a
cache hit for a digest-pinned reference the registry cannot add anything,
so return the hit before authenticating. Authentication and the pull stay
on the miss path, and a tag still resolves through the registry. The hit
emits the same Pulling, CacheHit and Pulled events as before.
Behaviour change: a host whose registry credentials were revoked still
boots digest-pinned images it has already cached, as a kubelet does with
IfNotPresent and a digest-pinned image.
Test: bootstrap_registry_cache_hit_for_a_digest_pin_makes_no_registry_request
covers a miss (authenticates and fetches), a cached digest pin (no request
at all) and a tag for the same content (still resolved). It fails without
this change with ["GET /v2/"].
Closes#4581
Signed-off-by: Balaji Ganesan <bganesan@nvidia.com>
The alias `openshell-prover` has been granted for OpenShell:
https://forum.snapcraft.io/t/alias-request-for-openshell/53658
Thus, we can remove guidance differentiating `openshell.prover` for the
snap from `openshell-prover` for other packaging formats. It is still
worth noting that installing a locally-built snap directly, rather than
from the store, means there is no alias.
Signed-off-by: Oliver Calder <oliver.calder@canonical.com>
* test(testkit): add the testkit workspace for end-to-end suites
Add tests/testkit as its own Cargo workspace: testkit, testkit-macros,
and testkit-spec. testkit reads a target file, connects to the gateway
through openshell-sdk, creates and cleans up sandboxes, and runs the
installed CLI. #[testkit::test] requires a stable ID and a doc comment and
declares capability requirements. testkit-ready waits for a target's
gateway and CLI before tests run.
Nothing uses testkit yet. The lockfile is seeded from the root lockfile
so shared crates resolve to the same versions.
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* ci(tests): format, lint and test testkit in Branch Checks
Nothing else builds testkit, so check it in Branch Checks and the
matching mise tasks to catch openshell-sdk changes that break it.
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* docs(tests): describe how to write and run testkit tests
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* fix(testkit): give each sandbox its own cleanup timeout
Cleanup shared one two-minute budget across all of a test's sandboxes, so
a slow delete left the rest with no time and could leak them on a shared
gateway. Delete them concurrently, each with the full timeout.
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* docs(testkit): show how to exclude tests by capability
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* docs: list tests/testkit in the project structure tables
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* refactor(testkit): drop test IDs and capabilities for now
Nothing reads test IDs or capability requirements until a runner, waivers,
or result history exist, so remove the id and requires arguments and the
target file's capabilities section. Both can return as optional attribute
arguments without changing existing tests.
With them gone, testkit-spec only checked the function shape and read the
doc comment, so fold it into testkit-macros.
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* docs(testkit): describe what testkit provides today
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* refactor(testkit): find the gateway the way the CLI does
Replace the target file (OPENSHELL_TEST_TARGET) with the CLI's own gateway
selection: OPENSHELL_GATEWAY_ENDPOINT, then OPENSHELL_GATEWAY, then the
active gateway. A registered gateway's endpoint, auth mode, and mTLS
material come from its CLI config directory. cx.cli runs OPENSHELL_BIN
against the same gateway and workspace as the SDK.
Suites now run unchanged under e2e/with-docker-gateway.sh and
e2e/with-podman-gateway.sh, and environments need no extra file.
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* fix(testkit): keep labels when creating a sandbox with a policy
The policy path creates the sandbox through gRPC and dropped the spec's
labels without an error. Pass them in the create request.
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* fix(testkit): use the registered gateway's credentials for an endpoint
When OPENSHELL_GATEWAY_ENDPOINT is set, the CLI takes credentials from the
gateway named by OPENSHELL_GATEWAY, or else the registered gateway with
that endpoint. testkit connected without a client certificate instead, so
an mTLS endpoint such as the one e2e/rust/e2e-vm.sh exports worked for
cx.cli but not for the SDK. Resolve it the way the CLI does, and pass
both the name and the endpoint to cx.cli.
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* docs(tests): list testkit's gateway variables in precedence order
OPENSHELL_GATEWAY_ENDPOINT takes precedence over OPENSHELL_GATEWAY. Also
mention OPENSHELL_BIN.
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* fix(testkit): apply other attributes to the generated test
#[testkit::test] left attributes such as #[ignore] and #[cfg] on the
nested body function, where they have no effect on the test. Move them
to the generated test, and reject a second test attribute such as
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
#[test] or #[tokio::test].
* fix(testkit): show usage for testkit-ready --help
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* test(testkit): cover cleanup after pass, failure, and panic
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* fix(testkit): reject #[should_panic] on testkit tests
A testkit test panics on any failure, including an unreachable gateway or
a failed cleanup, so #[should_panic] would let such a failure pass. Reject
it and point to checking the expected failure in the body.
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* test(testkit): keep only the macro's misuse checks as unit tests
Suites built on testkit exercise gateway resolution, the CLI wrapper,
polling, errors, and cleanup against a real gateway. Keep one unit test
for the macro's compile errors, which a passing suite never reaches, and
drop the test-only seams the other tests needed.
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* refactor(testkit): connect through the SDK's mTLS support
The SDK's ClientConfig now takes a client identity, so testkit no longer builds
its own channel. It still resolves the registered gateway and reads its
credential files, then hands the CA and client certificate to
OpenShellClient::connect. That keeps TLS behavior, connect timeouts and
keep-alive settings in the SDK, and drops testkit's direct tonic dependency.
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* fix(testkit): bound readiness and cleanup with timeouts
wait_ready checked elapsed time only after a health call returned, and cleanup
started its timeout only after the delete returned, so a gateway that accepts a
connection but never answers could hang either one. Wrap each whole operation in
tokio::time::timeout: the readiness wait, including every connect and health
call, and each sandbox's delete plus wait for deletion. A test covers a peer
that accepts connections and never replies.
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
---------
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Tests for the gateway's OCSF events each carried their own copy of the
same fixtures: converting captured events to JSON, the request ID and
peer address, running a handler inside a routed request, and checking
event shapes. They now share them:
- OcsfCapture::json_events returns captured events as their exported
JSON.
- auth::audit::test_support holds the test request origin, audited()
for running a handler as a routed request, and assertions for
authentication failures, logons, denials, boundary findings, the
request origin, and redacted secrets. The event assertions also check
each event against its OCSF class schema, which the handler tests did
not do before.
- auth::principal::test_support builds the OIDC user and gateway-JWT
sandbox principals that several test modules defined identically.
Refs #2912
Signed-off-by: Kris Hicks <khicks@nvidia.com>
Workload images no longer need iproute2 or nftables since the supervisor
moved out of the workload (#2942). Docker and Podman run images without
USER as UID/GID 1000 (#3386), the driver creates /sandbox, and the CLI no
longer builds Dockerfiles (#3214). Update the bring-your-own-container
example and the runtimes page to match.
Fixes#3965
Signed-off-by: Adel Zaalouk <azaalouk@redhat.com>
Forward tests retried on a new port only when the CLI's port check
found the port in use. Another test can also take the port after that
check passes and before the fake ssh binds it. The fake then panicked
with exit status 101, which the retry did not recognize, so the test
failed.
The fake ssh now exits 255 when its port is taken, as
ssh -o ExitOnForwardFailure=yes does, and the tests retry on that exit
status too.
Signed-off-by: Kris Hicks <khicks@nvidia.com>
Handlers recorded each refusal by emitting an OCSF event and then
building the status by hand, so a new refusal could skip the event.
These helpers do both in one call:
- audit::deny emits an authorization denial for the current request and
returns the permission-denied status. Guards, workspace
authorization, and the token and current-user handlers use it.
- audit::reject_credential emits an Authentication failure for a
credential a handler verifies itself and returns the rejection's
status. It replaces emit_credential_failure for SSH session tokens and
IssueSandboxToken. RefreshSandboxToken now takes its path and origin
from the request scope, as every other handler does, instead of
rebuilding them from the request.
- Failure and logon events share one Authentication builder.
- AuthMechanism and AuthFailureReason carry their audit labels. The
logon digest uses the label instead of the variant's Debug name.
Events and statuses are unchanged.
Refs #2912
Signed-off-by: Kris Hicks <khicks@nvidia.com>
* feat(skills): add release-captain skill for the weekly release cadence
Add an internal skill with Monday pre-flight, Tuesday stable cut, and
recovery steps for the RFC 0014 release train. All tag, workflow, merge,
and Slack actions require explicit maintainer approval.
Closes#4433
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* docs(skills): document starting a release train after tag-push triggers
Pre-release tag pushes now start Release Tag, so the skill no longer
dispatches after pushing. Add seed-commit selection from Release Dev
results, a commit-type summary for the patch/minor decision, and a
recovery row for a tag push that starts no run.
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* docs(skills): create lightweight release tags regardless of signing config
With tag.gpgSign enabled, git tag creates an annotated tag and blocks in
an editor. Disable signing for the release tag commands so they create
the same lightweight tags as Auto-Tag.
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* docs(skills): capture release captain lessons from starting 0.1.5
Note that Auto-Tag skips report success and show how to tell from the
log, the UTC time of the schedule across DST, where pre-releases appear
instead of the Releases page, why tags need a forced fetch, reading
workflow triggers from origin/main, the rolling dev tag, and the common
compatibility-review false positive for unpublished internal crates.
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* docs(skills): add recovery for pre-releases failing on new RustSec advisories
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* docs(skills): recommend fixing forward to a green pre-release
A failed pre-release doesn't end the train. Document merging the fix and
tagging the next pre-release by hand once its Release Dev run passes,
letting the failed run finish for its remaining signal.
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* docs(skills): describe the triage channel and thread replies for pre-release failures
Name #openshell-triage, where TriageBot posts each failed Release Tag
attempt, and ask the captain to reply in the failure's thread so others
know a fix is underway. List common transient failure types and rerun
once when a failure looks unrelated to the commit.
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* docs(skills): drop Codex Security from release captain run matching
Codex Security no longer runs in release qualification (#4469), so the
stable release can't rely on a pre-release's Codex result and runs can't
be matched by its job name. Match dispatched runs by the qualification
artifact name instead.
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* docs(skills): describe where and how to post the release coordination thread
Name the CNCF Slack channel, note that the release ships from main with
no release branch or cherry-picks, and add an example post.
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* docs(skills): make seeding the next train the outgoing captain's handoff
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* docs(skills): update release captain for the Release workflow and add usage guide
Stable releases are now dispatched with release.yml, which selects the
qualified pre-release and creates the tag, and pre-releases run in
release-prerelease.yml. Replace the stable tag push steps and stale
recovery rows, add a preview of the selected candidate, add a section on
when to use each part of the skill, and trim redundant explanations.
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* docs(skills): explain what the release cutoff guarantees
The release ships the final pre-release's commit only if it passes, so
the coordination post must not promise that every PR merged by the
cutoff ships. Describe the Auto-Tag and hand-tagged cutoffs, and tag
the final pre-release by hand on Tuesday for a later cutoff.
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* docs(skills): add release notes review and v0.1.5 release day lessons
Review the release notes preview from the Pre-release run before
approval, and describe how Release regenerates them. Correct the thread
when the shipped pre-release was cut before the posted cutoff, require
maintainer agreement for accepted-risk breaks in a patch release, and
add recovery rows for startup failures and release notes failures.
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* docs(skills): make starting the next train the captain's final duty
Move starting the next train after closing the loop, so it is the last
Tuesday step. The captain's week ends when the next -pre.1 Pre-release
run passes or its failure is handed off. Also move open milestone items
when closing the loop.
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* docs(skills): trim the release captain skill to its durable steps
Remove repeated guidance, week-specific examples, fragile commands, and
debugging detail an agent can derive from the workflows. Keep the
approval guardrails, the weekly steps, and a short recovery table.
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* docs(ci): document Release Auto-Tag and pre-release notes previews
Add Release Auto-Tag to the workflow table, including that each train's
-pre.1 is pushed by hand, and note that every Pre-release run writes a
release notes preview.
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* docs(skills): add a template for the release announcement reply
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* docs(skills): watch pre-releases in the week before your turn
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* docs(skills): remove internal channel references from release captain
Check pre-release runs directly instead of relying on an internal
notification channel, and drop internal Slack channel IDs.
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* docs(skills): fix pre-release failures in a new number via Auto-Tag
A rerun repeats the same commit, so a failure that needs a fix ships
in the next -pre.N. Cut it by dispatching Release Auto-Tag from the tip
of main, and tag a specific commit by hand only when needed.
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
---------
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Some credentials are checked by handlers rather than by the gateway's
authenticators, so their rejections only reached the caller and plain
logs. They now emit the same OCSF events as the router's checks:
- SSH session tokens. A ForwardTcp SSH connection whose token is
missing, unknown, revoked, minted for another sandbox, or expired
emits an Authentication failure with protocol ssh_session_token and a
reason category. An accepted token logs on once, like other
credentials. The router has already authenticated the caller, so these
events name that caller.
- IssueSandboxToken. A compute-driver identity that no longer matches
the sandbox's recorded runtime emits an Authentication failure
(stale_identity); a sandbox with no record, or no usable recorded
identity, emits unknown_identity. A sandbox calling it with a gateway
JWT is refused with method_not_permitted and raises the sandbox
boundary finding.
- RefreshSandboxToken. A caller that is not a gateway-JWT sandbox is
refused with method_not_permitted. A request for an extension service
the sandbox policy does not select is refused with the new
extension_service_not_selected reason, without a finding, because a
policy update can race the supervisor's request.
- Sandbox logons on IssueSandboxToken and RefreshSandboxToken. The
router logged a compute-driver credential on IssueSandboxToken, or a
gateway JWT on RefreshSandboxToken, before the handler finished
checking the recorded runtime or the token lineage, so a credential
that failed that check recorded a success and then a failure. Those
handlers now record the logon once their check passes. Other
credentials on these methods are fully verified by the router and
still log on there, even when authorization then refuses them.
- GetCurrentUser. A non-user principal is refused with
method_not_permitted. The router already blocks such callers, so this
is defense in depth.
Callers receive the same status codes and messages as before, and the
events never include the tokens.
Refs #2912
Signed-off-by: Kris Hicks <khicks@nvidia.com>
* feat(kubernetes): bootstrap experimental gateway supervisor topology
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* docs: move the experimental supervisor topology under an Experimental section
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* feat(supervisor): report experimental bootstrap health over TCP
experimental-bootstrap now takes --health-port instead of a Unix health
socket. It binds the dedicated supervisor's wildcard readiness listener,
reachable at the Pod IP, and accepts and closes connections while the
process runs. The chart probes the named container port health (5502)
with native kubelet tcpSocket checks and drops the health-only emptyDir.
No filesystem state remains to clean up after an unclean exit.
Dedicated supervisors keep --health-socket-path and health --socket.
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* docs: defer public experimental supervisor docs
Remove the experimental supervisor topology additions from the published gateway configuration page and the public debug-openshell-cluster skill. Public documentation will be added once the experimental stack settles.
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* refactor(supervisor): replace bootstrap with shared host stub
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* refactor(supervisor): name shared supervisor subcommand shared
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* refactor(supervisor): select shared mode with explicit endpoint flags
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* refactor(supervisor): name the shared host module after its mode
Rename the shared-mode host module from experimental to shared to match
the --mode=shared flag. No behavior changes.
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* refactor(kubernetes): drop the experimental prefix from the supervisor topology
Rename the Helm template to supervisor.yaml, its Kubernetes names and
labels to the -supervisor suffix and supervisor component, and the
driver setting to supervisor_topology. The setting remains opt-in and
workspace remains reserved.
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
---------
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* refactor(tests): move tmachine support files under tests/tmachine
Ansible, cloud-init, and the tmachine Nix config belong to tmachine, so
they now live in tests/tmachine/ alongside the crate, which moves to
tests/tmachine/cli/. tests/suites/ and tests/support/ are unchanged.
tmachine still runs from the directory holding ansible/, so playbook
paths stay the same; only inputs outside that directory gain a ../
prefix. The Nix build of tmachine and the base images are identical to
before.
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* refactor(tests): name the tmachine crate directory orchestrator
The crate boots VMs, provisions them with Ansible, runs suites, and
caches disks; the other entries in tests/tmachine/ are its config.
Name the directory for that role rather than for the command-line
entry point.
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* refactor(tests): keep the tmachine crate at tests/tmachine
Put the crate back at tests/tmachine/, next to the Ansible and Nix config
it uses, instead of in a subdirectory. Build tmachine from a fileset of
Cargo.toml, Cargo.lock, and src/ so edits to that config do not rebuild
the binary.
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
---------
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
OCI images commonly ship an empty /etc/hosts and rely on the container runtime to bind-mount a generated one. The VM driver does not, and guest init wrote /etc/hostname and /etc/resolv.conf but never /etc/hosts, so localhost did not resolve: the guest resolver is the loopback relay, which has no record for it.
Add configure_hosts to guest init, called after configure_hostname. It writes IPv4 and IPv6 localhost entries when the file is missing or empty, keeps a non-empty file and leaves a symlink alone, because an absolute link target would resolve against init's own root. A write failure logs a warning and boot continues.
Cover the file states and the boot-path call site with tests that run the shipped init function under bash.
Closes#3934
Signed-off-by: Shiju <shiju@nvidia.com>
Go resolves the sdk/go submodule version from sdk/go/vX.Y.Z tags, so
without one pkg.go.dev only shows v0.0.0 pseudo-versions from main. Add
a Release Tag job that creates the prefixed tag at the release commit
and asks the Go module proxy to fetch it.
Signed-off-by: Jim Meyer <jimeyer@nvidia.com>
* docs(providers): close stdin in the openrouter smoke test
The sandbox main process keeps stdin open so attach and detach work,
and opencode run reads stdin until EOF, so the documented smoke test
never printed a result. Run it through sh -c with stdin redirected from
/dev/null and note why.
Closes#3883
Signed-off-by: fede-kamel <fkamelhar@gmail.com>
* Remove verbose comment in providers/openrouter.yaml
---------
Signed-off-by: fede-kamel <fkamelhar@gmail.com>
Co-authored-by: Taylor Mutch <taylormutch@gmail.com>
When a gateway replica accepts a credential for the first time, it now
emits an Informational OCSF Authentication event with status Success.
The caller's request proceeds exactly as before. The event records:
- Who: the principal (user, mTLS operator, sandbox, or gateway peer),
with the same user types as authorization denials.
- How: the authentication protocol (OpenID, mtls, sandbox_jwt,
compute_driver, or k8s_service_account).
- Where: the gRPC method of the request that logged on.
- Correlation: the caller's address and request ID, the same fields the
failure and denial events carry.
Every gRPC request authenticates, and an idle sandbox's supervisor polls
the gateway several times a minute, so an event per request would bury
user logons. Each credential therefore logs on once: later requests with
the same bearer token or client certificate emit nothing until the
credential has been in use for an hour, when it logs on again. The
replica's connections share this record, which holds up to 16,384
credentials and forgets the oldest early when a new one arrives while
full. It keys on a SHA-256
digest of the credential that authenticated the caller (the bearer
token or the client certificate's fingerprint), so headers the
authenticator ignored cannot force new logons, and a renewed certificate
for the same subject logs on again. Digests stay in memory.
Requests admitted by allow_unauthenticated_users, or by a gateway with
no authentication configured, emit no success event, since no credential
was verified.
Refs #2912
Signed-off-by: Kris Hicks <khicks@nvidia.com>
* ci(release): reuse the promoted prerelease's release notes preview
Stable releases now publish the selected prerelease's release-notes
artifact with the prerelease tag replaced by the stable tag. Release
falls back to generating notes when the preview is missing, expired, or
malformed, or when dispatched with regenerate-notes=true. The selection
script exposes the qualified Pre-release run ID for the download.
Closes#4753
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* ci(release): reuse release notes from the promotion job
Download and retag the preview in Select Qualified Prerelease, which
already knows the qualified Pre-release run, instead of a separate job.
Drop the regenerate-notes input and preview format checks.
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* ci(release): name the publish notes input after the caller's result
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
---------
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* feat(sandbox): support sandbox-initiated supervisor connections
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* refactor(sandbox): consolidate control transport setup
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* test(sandbox): remove prototype-specific unit tests
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* feat(kubernetes): migrate legacy control flow on restart
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* docs: remove prototype changes from published docs
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* fix(sandbox): adapt reverse control to published backend protocol
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* feat(sandbox): initiate supervisor connections across compute drivers
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* fix(docker): stage reverse control socket for the supervisor identity
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* refactor(sandbox): keep runtime transports behind the backend
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* fix(docker): mount the runtime control volume read-only
The sandbox runtime only dials the supervisor-owned control socket, so
it does not need write access to the control volume. Mount it read-only
in the runtime container and stop staging the control directory through
the runtime bundle; the supervisor stager already initializes it through
its own writable mount.
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* fix(kubernetes): separate legacy and runtime-initiated control services
Keep os-boundary selecting the workload and add os-supervisor for runtime-initiated connections. Bind the mode marker to its runtime generation so a restart by an older gateway does not inherit stale mode metadata. Preserve the legacy Service across creation and restart, validate the active endpoint, and drop the Service patch permission.
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
---------
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
When the gateway refuses a caller that has already authenticated, it
now emits an OCSF API Activity event. The caller gets exactly the same
error as before. The event records:
- Who: the principal (user, mTLS operator, sandbox, or gateway peer).
- What: the gRPC method, plus the sandbox or workspace that was
refused when there is one.
- Why: a reason such as cross_sandbox, missing_role or
not_workspace_member.
- Correlation: the caller's address and request ID, the same fields
the authentication events carry.
That covers the denials made centrally by the router and the shared
guards, plus the checks inside handlers that have no shared guard: a
sandbox naming another sandbox (in either sandbox-name resolver), a
non-user caller refused by mutation admission, a sandbox sending a
config update it isn't allowed to, and a non-admin granting the
workspace admin role. Denials raised in the mutation admission worker,
which runs in a spawned task, keep the request's context.
When a sandbox credential is refused for calling a method it may not
call, touching another sandbox, or acting across workspaces, the
gateway also emits a High-severity Detection Finding. It carries the
same correlation_uid as the denial event. A real supervisor never makes
these requests, so a finding means the credential is probably being
misused. Two cases are recorded without a finding: a method the
gateway doesn't recognize (unknown_method), so a newer supervisor
doesn't look like a compromised one, and a supervisor whose own sandbox
was deleted mid-request, which is refused but isn't misuse.
The shorthand line names the refused sandbox or workspace, for example
sandbox:<id>, so operators reading console logs can see the target
without JSONL or OTLP export.
To support this, the OCSF crate's Actor can now hold a user instead of
a process, and the API Activity builder gained the fields gateway
events need. Supervisor inference events come out unchanged. Gateway
events declare the container profile only when they concern a sandbox,
as MXC findings do. The OCSF
1.8.0 resource_details schema is vendored so the refused sandbox or
workspace goes in the standard resources field.
Refs #2912
Signed-off-by: Kris Hicks <khicks@nvidia.com>
* fix(vm): flush guest filesystem before a planned sandbox stop
A planned stop of a MicroVM sandbox ended the VM before the guest wrote its dirty pages to disk, so files written shortly before the stop were missing after the next start.
The driver closed the host supervisor's parent-liveness pipe before it sent SIGTERM. The supervisor exits at once on that end-of-file, so it never ran its orderly shutdown: terminate the workload, then ask the guest to terminate. Signal the supervisor first, keep the pipe open until it has exited, and give it a grace period derived from the supervisor's own bounds and kept below the gateway's reclaim deadline.
In the guest runtime, syncfs the root filesystem after the process tree is terminated and before the termination is acknowledged, bounded to 3 seconds and best effort. Only the VM backend, which uses a vsock boundary listener, flushes; the container drivers share this runtime and are unchanged.
A sandbox whose main process has already exited is not covered.
Closes#4185
Signed-off-by: Shiju <shiju@nvidia.com>
* fix(vm): warn about unconfirmed flush after supervisor kill
Preserve the supervisor exit status so a planned stop warns when a signal
killed the supervisor and guest writeback was not confirmed. Keep normal
workload exit codes quiet, including 143 after a handled SIGTERM.
Extend the existing process tests to check the warning after a forced kill
and its absence after an orderly shutdown.
Refs #4185
Signed-off-by: Shiju <shiju@nvidia.com>
* fix(vm): satisfy shutdown timeout control-flow lint
Use if let for the single successful timeout pattern while preserving
the supervisor exit status and forced-kill error handling.
Signed-off-by: Shiju <shiju@nvidia.com>
* fix(vm): preserve shutdown merge resolutions during rebase
Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>
* docs(vm): explain planned stop durability and flush limits
Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>
---------
Signed-off-by: Shiju <shiju@nvidia.com>
Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>
Co-authored-by: John Myers <9696606+johntmyers@users.noreply.github.com>
* fix(snap): only enable gateway user service for one user
Signed-off-by: Oliver Calder <oliver.calder@canonical.com>
* docs(snap): correct snap docs for the gateway user service
Signed-off-by: Oliver Calder <oliver.calder@canonical.com>
* fix(snap): add documentation of global to per-user enablement migration
Signed-off-by: Oliver Calder <oliver.calder@canonical.com>
---------
Signed-off-by: Oliver Calder <oliver.calder@canonical.com>
* fix(ocsf): report https scheme for TLS-terminated requests
Closes#4562
Signed-off-by: Eric Curtin <eric.curtin@docker.com>
* fix(ocsf): use transport scheme on passthrough relay
Signed-off-by: Eric Curtin <eric.curtin@docker.com>
---------
Signed-off-by: Eric Curtin <eric.curtin@docker.com>
* fix(server): warn on dangling provider refs at startup
Signed-off-by: Eric Curtin <eric.curtin@docker.com>
* fix(server): load providers concurrently at startup
Signed-off-by: Eric Curtin <eric.curtin@docker.com>
---------
Signed-off-by: Eric Curtin <eric.curtin@docker.com>
helm push uploads the chart manifest by digest, then fetches it back to
apply the version tag. GHCR sometimes returns 404 for a manifest it just
accepted, failing the push with 'failed to perform "Tag" on
destination: <digest>: not found'. This failed Release Dev on Oct 5 and
the v0.1.5-pre.2 Release Tag run after qualification passed.
Retry each chart push up to four times with a short backoff. A retry
finds the blobs and manifest already present and only applies the tag.
Closes#4577
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* ci(release): name the failed stage and failing jobs in pre-release failure notices
The pre-release failure notification said only that the run failed.
Name the failed stage (build, publication, or release qualification),
link the tag, and list each failing job of the run attempt with a link.
Skip the aggregate Release Qualification job unless it is the only
failure, and still send the message if the jobs lookup fails.
Closes#4718
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* ci(release): simplify the failure notice to a list of failed jobs
Drop the stage headline and the aggregate-job filtering. Keep the
existing message and append a link to each failed job.
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
---------
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Simon Ruggier