* ci(compatibility): review PR breaks and acknowledge release risks
Signed-off-by: Simon Scatton <sscatton@nvidia.com>
* fix(ci): use matching compatibility review tooling
Signed-off-by: Simon Scatton <sscatton@nvidia.com>
* ci(compatibility): scope Rust SDK review to public exports
Signed-off-by: Simon Scatton <sscatton@nvidia.com>
* ci(compatibility): account for Go struct field additions
Signed-off-by: Simon Scatton <sscatton@nvidia.com>
* ci(compatibility): restrict egress and fix report retries
Signed-off-by: Simon Scatton <sscatton@nvidia.com>
---------
Signed-off-by: Simon Scatton <sscatton@nvidia.com>
* fix(cli): backpressure sandbox uploads
Stream archive uploads over a bounded SSH subsystem whose credit follows extraction stdin progress. Preserve the existing input budget and tar destination semantics, validate completion, and clean up cancelled transfers. Add protocol, lifecycle, and large-upload regression coverage without changing russh.
Signed-off-by: Shiju <shiju@nvidia.com>
* fix(cli): preserve upload cancellation and errors
Keep cancellation active through upload setup and retry delays, and
terminate cancelled SSH probes. Preserve protocol failures through output
completion and report captured diagnostics on early extraction failure.
Remove redundant upload validation and extend the affected regressions.
Signed-off-by: Shiju <shiju@nvidia.com>
* refactor(cli): carry upload credit over SSH streams
Replace the framed upload protocol with SSH's own streams. The client
sends one JSON plan line, the raw archive and EOF. The supervisor's
stdout carries only cumulative totals of bytes written to extraction
stdin, extraction output goes to stderr, and the exit status reports
the result. Extraction starts inline through the ordinary exec path.
This removes the frame codec, SHA-256 digests, the CLI sha2 dependency
and the supervisor upload state machine while keeping the 1 MiB credit
window, the 4 MiB input guard, cancellation and error reporting. A
failed archive now stops SSH before closing stdin, so EOF never marks a
truncated archive complete.
Signed-off-by: Shiju <shiju@nvidia.com>
* fix(cli): isolate upload SSH connections and pin credit cadence
- Apply the forward multiplexing overrides to the upload ssh command, so
a user ControlMaster cannot carry an upload over another session's
master connection.
- Kill ssh before closing its stdin when a transfer fails locally, and
include ssh stderr in transport errors.
- Share ACK_INTERVAL_BYTES from core. Assert at compile time that CLI
chunks fit the credit left after an ack and that the window stays
below the supervisor's pending-input guard.
- Log the real reason when the supervisor rejects an upload.
- Test that a single-packet upload reports its final total.
Signed-off-by: Shiju <shiju@nvidia.com>
* test(cli): assert upload ssh opens the subsystem without a TTY
Signed-off-by: Shiju <shiju@nvidia.com>
---------
Signed-off-by: Shiju <shiju@nvidia.com>
* refactor(prover): remove legacy unused reachability constraints
Remove can_write_to_endpoint() and the solver variables and encoding
methods that exist solely to support it. No active query calls this
method; confirmed zero callers across the monorepo. Crate is not
published externally, so no public API concern.
Removed as root cause:
- can_write_to_endpoint(): zero callers in the codebase
Removed as direct consumers of can_write_to_endpoint():
- binary_can_write: only read inside can_write_to_endpoint()
- credential_has_write: only read inside can_write_to_endpoint()
Removed as directly dead (no query ever read them):
- credential_has_destructive: flagged by compiler, silenced with
#[allow(dead_code)] rather than removed
- filesystem_readable: flagged by compiler, silenced with
#[allow(dead_code)] rather than removed
Removed as now-empty encoding methods:
- encode_credentials(): exclusively populated credential_has_write
and credential_has_destructive
- encode_filesystem(): exclusively populated filesystem_readable
Also removed two #[allow(dead_code)] allowances and the corresponding
HashMap::new() initialisers and build() call sites.
Active query behavior is unchanged. All existing prover tests pass
without modification and cover all four finding categories end-to-end.
No new tests added as they would duplicate existing coverage exactly.
Note: removing encode_credentials() leaves write_actions_for_scopes()
and destructive_actions_for_scopes() in credentials.rs with no callers.
Left as a follow-up outside the scope of this change.
Closes#2412
Signed-off-by: Evie Howard <evhoward@redhat.com>
* refactor(prover): remove dead write-capability helpers from binary registry
BinaryProtocol::can_write(), BinaryCapability::can_write(), and
write_mechanisms() lost their only callers when can_write_to_endpoint()
and its supporting model fields were removed. Delete the now-dead methods.
write_mechanisms() had no callers prior to this series; the two can_write()
methods were made dead by the preceding model.rs cleanup.
Signed-off-by: Evie Howard <evhoward@redhat.com>
* docs(prover): remove stale write-bypass mention from module doc comment
The module doc incorrectly listed write-bypass violation detection as a
prover capability. The prover only checks reachability queries; write-bypass
detection was never implemented.
Signed-off-by: Evie Howard <evhoward@redhat.com>
---------
Signed-off-by: Evie Howard <evhoward@redhat.com>
* test(podman): run the e2e binaries that pass in the tmachine guest
Remove ten binaries from podmanE2eFollowUpBinaries so the rootless tmachine e2e-podman archive runs them: credential_gating, forward_proxy_graphql_l7, forward_proxy_jsonrpc_l7, forward_proxy_l7_bypass, host_gateway_alias, landlock, no_proxy, podman_oci_identity, provider_refresh_handles, and websocket_conformance. All ten passed in two consecutive probe runs. Update the exclusion comments and TESTING.md to match what the remaining entries still need. Contributes to #3712.
Signed-off-by: politerealism <burdcat17@gmail.com>
* test(podman): explain why each remaining e2e binary is excluded
Give every remaining podmanE2eFollowUpBinaries entry its own reason, and mark the five that the wrapper-based Podman lane still runs (driver_config_volume, local_driver_token_restart, podman_corporate_proxy, podman_gateway_start, provider_token_exchange) so it is clear they are covered there until tmachine provides what they need. Contributes to #3712.
Signed-off-by: politerealism <burdcat17@gmail.com>
---------
Signed-off-by: politerealism <burdcat17@gmail.com>
* fix(kubernetes): restart kube read timeout when each request is written (fixes#4451)
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* refactor(kubernetes): use a per-request tower timeout instead of a custom kube client
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
---------
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* test(oci): share OCI image checks across Docker and Podman
Add an oci-image feature testsuite that runs against installed artifacts.
It covers custom WORKDIR placement, image content and ownership, workspace
writes from the main process and exec, file transfer including
directory-merge upload, OCI user identity, the managed /sandbox fallback,
and rejection of an unwritable WORKDIR.
CI runs the suite on the Docker rootful, Podman rootful, and Podman
rootless tmachine guests.
Replace the Docker-only custom_image e2e with the shared suite.
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Evan Lezar <elezar@nvidia.com>
* test(oci): require WorkspaceValidationFailed and cover image USER fallback
The unwritable WORKDIR scenario now requires the WorkspaceValidationFailed
reason, so unrelated provisioning failures cannot pass it. The numeric
USER scenario supplies a policy without a process section, so the image
USER is the only source of the sandbox identity.
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Evan Lezar <elezar@nvidia.com>
* test(podman): leave shared OCI identity checks to the oci-image suite
The oci-image feature suite covers the sandbox identity seen by the main
process and sandbox exec on Docker and Podman. The Podman e2e keeps the
checks only it can make: the pinned image ID and the isolated
workload/supervisor container pair.
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Evan Lezar <elezar@nvidia.com>
* test(oci): accept existing workspace rejection diagnostics
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Evan Lezar <elezar@nvidia.com>
* test(oci): fix cleanup ordering and restore coverage parity
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Evan Lezar <elezar@nvidia.com>
* test(oci): remove migrated legacy workspace helpers and checks
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* fix(drivers): report rejected OCI working directories as WorkspaceValidationFailed
The supervisor exits with the reserved status when the sandbox rejects the
image working directory. Docker maps that exit during readiness and
monitoring and records it before removing the supervisor container, and
Podman maps the supervisor companion's exit instead of the workload's.
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* test(oci): require WorkspaceValidationFailed for unwritable WORKDIR
Unrelated provisioning failures can no longer pass the rejection scenario.
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* fix(oci): keep exit 78 reserved and report fixed workspace diagnostics
The supervisor no longer passes on a workload's own exit status 78, so
drivers cannot misreport it as a workspace rejection after a runtime
restart. The rejection is recognized once at agent start and carried as a
typed error. Docker reports the same fixed condition message as Podman and
logs the supervisor output instead, and its monitor removes the supervisor
container when a stop overlaps diagnostics collection. The OCI image suite
registers images for cleanup only after they build.
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* refactor(oci): simplify workspace failure handling and OCI image tests
Report a recorded Docker workspace rejection once, where supervisor
readiness fails, and match its fixed message exactly. Read the Podman
supervisor exit status once per inspection. Choose the supervisor exit
status in one function and derive the workspace error type.
The OCI image suite shares step helpers, checks that a custom workspace
does not also create /sandbox, and its playbook waits for a connected
gateway and collects failure diagnostics through the shared role.
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* test(oci): replace podman_oci_identity with shared and driver suite checks
The oci-image feature suite now checks, on Docker and Podman, that the
image user has no capabilities, has no_new_privs set, and cannot read
the sandbox control credentials, and that a custom WORKDIR is not a
mount. The checks that need container inspection (pinned image ID,
network=none, supervisor user and capabilities) move to a small test in
the Podman driver suite. Remove the legacy podman_oci_identity e2e test.
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* test(oci): keep podman_oci_identity for Podman-only isolation checks
Restore the Podman e2e test removed in the previous commit and drop
only the checks the shared oci-image suite now runs on Docker and
Podman: workload capabilities, no_new_privs, credential readability,
and the workspace and /sandbox mount checks. The pinned image ID and
the workload/supervisor isolation checks stay, as does the Podman
driver suite unchanged.
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* refactor(oci): simplify workspace failure reporting and OCI image suite
Carry the Docker control supervisor's failure reason in a typed error
instead of matching status text, build the fixed workspace message where
the wait result is classified, and drop the fake Docker server test.
Inline image cleanup ordering in the suite, read the container engine
once, share sandbox naming, and stop granting a literal /sandbox in the
image identity policy. Add e2e:oci-image mise tasks and use them from the
ARM64 Docker workflow and TESTING.md.
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* test(oci): move shared container checks into the OCI image suite
Inspect the containers the driver created through the gateway's engine in
the shared suite: the workload runs the inspected image ID as the OCI
identity without a network or supervisor credential mounts, and the
supervisor is a separate non-root container. Keep only the Podman-specific
supervisor identity and capability checks in podman_oci_identity.
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* test(oci): check the workspace rejection reason from sandbox JSON
Read the WorkspaceValidationFailed condition reason from sandbox get
--output json instead of matching CLI output, and restore the Docker
driver's original workspace failure helpers.
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* test(oci): capture fixture image identity before sandbox creation
Signed-off-by: Evan Lezar <elezar@nvidia.com>
---------
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Evan Lezar <elezar@nvidia.com>
Co-authored-by: Evan Lezar <elezar@nvidia.com>
The VM runtime scripts sourced pins.env optionally and duplicated the
LIBKRUN_REF and UMOCI_VERSION defaults, so every bump had to update the
same value in several files.
Require pins.env, drop the duplicated defaults, and fail on an invalid
LIBKRUN_REF instead of silently building upstream HEAD. Environment
overrides still take precedence through the defaults in pins.env.
Signed-off-by: Florent Benoit <fbenoit@redhat.com>
* fix(driver-mxc): pass gateway name to mocked backend
Refs #4191
Signed-off-by: Philippe Martin <phmartin@redhat.com>
* fix(windows): use native executable paths in network test fixtures
Signed-off-by: Philippe Martin <phmartin@redhat.com>
---------
Signed-off-by: Philippe Martin <phmartin@redhat.com>
Remove blanket review routing while preserving the explicit owners of the vouch list, workflows, actions, approval helpers, and SBOM check already on main.
Signed-off-by: Jim Meyer <jimeyer@nvidia.com>
* test(tmachine): add K3s PostgreSQL and mTLS conformance installer
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* fix(tmachine): reach the HA TLS gateway through its ClusterIP
Drop the port-forward restart removed by the ClusterIP change, resolve the Service DNS name covered by the server certificate to the ClusterIP, bound each authenticated API wait attempt, and collect diagnostics from every gateway Pod.
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* ci(tmachine): run K3s PostgreSQL and mTLS conformance
Add the k3s-ha-tls installer to every core conformance matrix so Branch E2E, Release Dev, Release Tag, and manual Integration Tests cover three gateway replicas sharing PostgreSQL behind mTLS.
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* refactor(tmachine): rename k3s-ha-tls installer to k3s-ha
TLS is part of the supported HA configuration once gateway replicas share PostgreSQL, so the installer name only needs to distinguish HA from the single-replica k3s lane.
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* refactor(tmachine): install K3s HA in a single Helm rollout
Add an optional Helm values overlay to the K3s installer so the k3s-ha
installer stages PostgreSQL first and installs the three-replica TLS
gateway directly, instead of installing a plaintext gateway and upgrading.
Drop the redundant unauthenticated-users override and the install-time
API wait, which the conformance playbook already performs. Share K3s
installer inputs in the tmachine configuration.
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* fix(tmachine): make K3s HA staging rerunnable and wait for PostgreSQL
Apply the OpenShell namespace instead of creating it so the staging play
tolerates an existing namespace, and wait for the PostgreSQL rollout so
gateway Pods do not start crash-looping against a missing database.
Reflow and pluralize the K3s lane notes in CI.md.
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
---------
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
The runtimes page examples omit the [openshell] version = 2 table, so a
snippet copied into a new gateway.toml fails preflight with
missing_version. Add the table to the first example and state that the
remaining examples show only the tables they change.
Closes#3881
Signed-off-by: fede-kamel <fkamelhar@gmail.com>
Spell out that auto-closed PRs keep their discussion and can be reopened or replaced, separate DCO sign-off from cryptographic commit signing, and say to mark PR checklist items complete only after verification.
Signed-off-by: politerealism <burdcat17@gmail.com>
* feat(python): expose sandbox status conditions in the SDK
Add SandboxConditionRef and surface status.conditions on SandboxStatusRef and SandboxRef. Callers need condition reasons and transition times to time terminal transitions and explain failures that happen before the main process starts, without reading the raw proto.
Signed-off-by: Max Dubrinsky <mdubrinsky@nvidia.com>
* test(python): use real gateway condition values in conditions fixtures
Signed-off-by: Max Dubrinsky <mdubrinsky@nvidia.com>
---------
Signed-off-by: Max Dubrinsky <mdubrinsky@nvidia.com>
The stop path wrote Stopping and Stopped with a single compare-and-swap.
In HA deployments every replica applies compute driver watch events for
the terminating runtime without the per-replica lifecycle gate, so a
watch write landing between the re-read and the CAS failed the stop RPC
with "update sandbox lifecycle failed due to concurrent modification"
and left the sandbox in Stopping.
Retry both writes on PersistenceError::Conflict up to
STOP_PHASE_CAS_RETRY_LIMIT, re-reading the sandbox on each attempt. The
Stopping write re-runs the stop preconditions and the name check; the
Stopped write proceeds only while the record is still Stopping, returns
an already Stopped record as-is, and aborts on any other phase. Gateway
startup recovery of retained Stopping sandboxes now shares the same
helper instead of writing against its pre-driver snapshot.
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
* feat(providers): support stable credential placeholders
Add credentials[].stable_placeholder so a running client keeps using an externally rotated static credential through the reference it already holds. Squashed replay of the earlier four commits onto upstream main 71c3cd957; re-derives the frozen schema fingerprints and adapts the provider environment revision test to the added capability argument.
Signed-off-by: Shiju <shiju@nvidia.com>
* fix(e2e): apply stable-placeholder supervisor image override
Use ManagedGateway's per-process supervisor image override so the fixture
CA reaches the supervisor when runtime image environment variables take
precedence over gateway TOML. Restore the wrapper's original image after
each test, following the provider-readiness fixture.
Signed-off-by: Shiju <shiju@nvidia.com>
* test(server): update public RPC schema fingerprint after merging main
Signed-off-by: Shiju <shiju@nvidia.com>
---------
Signed-off-by: Shiju <shiju@nvidia.com>
* test(k3s): connect tmachine clients directly to ClusterIP
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* test(conformance): wait for a connected gateway on every installer
Replace the K3s-specific boot readiness hook with one shared gateway wait in the conformance playbook. Skip conformance and collect the existing diagnostics when the gateway never connects. Discover the gateway Service port with its ClusterIP, and capture both iptables backends in K3s diagnostics.
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* test(tmachine): centralize gateway diagnostics and trim K3s changes
Move failure diagnostics into one openshell_diagnostics role that collects the systemd gateway journal and, when K3s is installed, the K3s diagnostics. Drop install-time diagnostics so the K3s installer change stays minimal, revert unrelated skill edits, and shorten the CI.md note.
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
---------
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Adds the OpenShell / Maintainer Approval status check. It passes only when
someone listed in MAINTAINERS.md has approved the pull request, and it
notifies no one: a required status check is the only native primitive that
enforces who must approve without requesting a review from them.
The job runs on pull_request_review and merge_group with read-only
permissions and posts nothing anywhere; its exit code is the check result.
Both MAINTAINERS.md and the decision helper are read from main via a
sparse checkout, never from the pull request ref, so a contributor cannot
add themselves to the list or rewrite the decision logic and self-approve.
Because pull_request_review runs the pull request's own copy of the
workflow file, CODEOWNERS now covers the whole enforcement surface --
workflows, actions, CODEOWNERS itself, the zizmor config, MAINTAINERS.md,
and the enforcement scripts -- naming the outside collaborators explicitly
alongside the team, since org policy keeps them off GitHub teams. The
ruleset's native code owner requirement is what blocks a merge that
disables the gate, regardless of what the tampered check reports.
Also adds a change-alert workflow that comments the approver-set delta on
pull requests touching MAINTAINERS.md, and documents both in CONTRIBUTING.
Signed-off-by: Jim Meyer <jimeyer@nvidia.com>
Jim Meyer