mirror of
https://github.com/NVIDIA/OpenShell.git
synced 2026-10-29 07:23:25 +08:00
v0.1.7-pre.1
288
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Jesse Jaggars
|
2c31ac7e60 |
feat(network): support additional destination CAs (#3292)
* feat(network): support additional destination CAs Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(network): harden additional CA propagation Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(network): narrow additional CA review scope Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(network): harden additional CA rotation Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * feat(network): complete additional CA delivery Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(docker): remove stale runtime fixture field Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(network): adapt additional CA series to upstream main Resolve rebase fallout against current upstream main after the series was rebased 214 commits forward: - follow upstream #3110: supervisor→gateway TLS is CA-only, so configure_gateway_tls and its isolation tests drop the client identity - follow upstream tls_secret_names reduction to [String; 1] in the Podman driver while keeping the network trust bundle delivery - port [openshell.supervisor.network] rendering into the openshell.gatewayConfigToml serializer in _toml.tpl - adapt tests to upstream struct/function changes (DriverStartupContext, SandboxRunConfig, IsolationSpecInput, supervisor_pod, VM bootstrap image) - drop conformance-artifact-prefix plumbing removed upstream Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * refactor(network): adopt upstream additional CA bundle surface Unify the additional destination CA series on the mechanism merged to main in #3996: one operator pre-assembled PEM bundle, the --additional-ca-bundle supervisor flag, and the /etc/openshell/tls/additional/ca-bundle.pem staging path. - Replace the gateway-level additional_ca_cert_paths list with a single additional_ca_bundle file validated and staged at gateway startup. - Restore the upstream supervisor argument path and fold-into-roots behavior; drop the --network-additional-ca-bundle and --network-additional-ca-digest contracts. - Keep gateway normalization bounds, content-addressed staging, digest-based driver reconciliation, and stop/start rotation; the freshness check now runs at the gateway/driver boundary instead of the supervisor boundary. - Reject additional_ca_bundle in operator driver JSON config; standalone drivers self-source the bundle from --sandbox-additional-ca-bundle with the same validation. - Extend the Kubernetes ConfigMap name digest segment to keep the full 63-byte budget after the prefix rename. - Box the persisted-sandbox reconciliation futures so callers stay under the clippy::large_futures threshold. Updates Docker, Podman, Kubernetes, and VM delivery, Helm chart and tests, e2e wiring, gateway configuration docs, and the cluster debugging skill to the single mechanism. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(ci): make gateway startup tests wrap-tolerant and refresh example lockfile The network trust startup tests asserted on raw substrings of gateway diagnostics, but the CLI renderer wraps output at a fixed width with continuation markers and hard-breaks long paths. Compare assertions on whitespace- and continuation-squashed text so they hold regardless of wrap position. Format the e2e workspace (separate cargo workspace, not covered by the root cargo fmt run) and refresh examples/isolation-backend-v8/Cargo.lock for the libc dependency added to the network trust crate. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(podman): reuse TLS CA secret during stopped-pair trust replacement The additional CA e2e never ran in CI before the first maintainer test dispatch, and its local container inspection was broken in two ways: it located the supervisor container through e2e network membership (the supervisor is host-networked or runtime-bridged, so the filter matched the CI job container instead), and it required the gateway mTLS CA as a bind mount (Podman delivers it as a container secret when user namespaces remap UIDs). Select the supervisor by managed labels and accept the secret delivery form. The lifecycle leg then exposed a driver bug: stopped-pair trust replacement re-created the per-sandbox TLS CA secret, colliding with the secret still owned by the renamed stopped pair. Mirror the token and resolver secret handling: require and reuse the existing secret during replacement instead of creating it. Validated end-to-end with the Podman additional CA e2e lane. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(server): restore upstream recovery semantics for start-request failures Recovery no longer marks sandboxes Error/InvalidPersistedSandbox when driver_start_request fails during the startup or lifecycle sweeps. Gateway configuration problems (for example a missing launch signer) must not permanently fail unrelated sandbox records; the sweep logs a warning and skips them, matching the behavior before the additional CA work. Sandbox-record failures surface through the normal start path. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * refactor(podman): refresh stale supervisors without replacing workloads Replace the stopped-pair trust reconciliation with supervisor-only recreation. The destination-trust generation label moves from the workload to the supervisor container, and start_sandbox recreates only the supervisor when its generation is missing or stale. The workload container, its volumes, its secrets, and its workspace data are never replaced: the additional CA bundle is mounted only into the supervisor. Running sandboxes are never gated on trust labels, and recovery without fresh launch credentials keeps the existing supervisor. The intentional -removal fence machinery in the watcher existed only for workload replacement and is removed with it. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * refactor(docker): drop workload replacement for destination-trust changes Delete the stopped-workload trust reconciliation path and all of its machinery: the trust-generation label on workload containers, the generation match check in the start fence, the stopped-sandbox replacement, the bounded workspace archive copy with its 512 MiB cap and regular-file-only validation, and the boundary-state rollback helpers. No replacement machinery is needed: the trusted supervisor container is already re-created from the gateway's current configuration on every start via ensure_control_process_for_container, so stop/start picks up the current additional CA bundle while the workload container, its volumes, and its workspace data are never replaced. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * docs(configuration): drop reconciliation wording from additional CA lifecycle The stop/start lifecycle description already matches the supervisor-only refresh model; remove the two remaining references to restart reconciliation of workload containers. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(network): port additional CA test fixtures to the crypto interface Route the gateway TLS isolation test PKI, the destination CA source fixtures, and the gateway startup trust tests through openshell_crypto::pki now that rcgen access is centralized there: generate_keypair/self_signed/signed_by replace the direct rcgen calls and CertifiedKey field access follows the wrapper's shape. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(e2e): repair additional CA lane for Docker and non-container builds The first maintainer test dispatch revealed two latent issues in the additional CA e2e that no local run had exercised: the Podman gateway CA secret helper referenced ContainerEngine without the container feature gate, breaking Kubernetes and VM builds of the test, and the Docker branch asserted a client ca.crt bind mount that the Docker driver never creates. Gate the helper like its callers, and assert Docker's actual delivery path instead: the gateway mTLS CA staged as tls/ca.pem inside the read-only supervisor state volume, selected through the OPENSHELL_TLS_CA environment pointer. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(helm): restore alphabetical README ordering for supervisor keys The helm:docs check regenerates the chart README with helm-docs, which orders rows alphabetically; a conflict resolution had placed supervisor.network.additionalCaConfigMapName after the supervisor.podSecurityContext rows. Regenerate with mise run helm:docs. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * chore: refresh testkit lockfile for openshell-core libc dependency Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(driver-podman): stage operator CA bundle as driver-owned artifact Standalone Podman drivers accepting --sandbox-additional-ca-bundle previously kept the operator's source path as the trust artifact path. The immutable-artifact verification performed on every create/start requires exactly mode 0444, which an operator-owned source path cannot guarantee, so a readable-but-writable source failed reconciliation. Read the operator bundle with the same bounded, no-symlink validation, then stage the normalized material into a driver-owned artifact inside a private tempdir held for the driver process lifetime. The artifact is created 0600, synced, and only then restricted to 0444 so no observer ever sees a writable artifact. The operator source and its permissions are never referenced after startup. Addresses GATOR-291f43b2-02. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(driver-podman): pin retained workload image during supervisor refresh Supervisor-only trust reconciliation re-entered create_sandbox_with_existing_token, which re-pulled and re-resolved the workload image tag even though the workload container is retained. If the tag moved to a different image between stop and start, identity resolution and child-environment construction would reflect the new image while the retained workload still runs the original one, leaving mode-0600 bootstrap files the retained user cannot read. During a refresh with a retained workload, inspect the stopped container and pin to the immutable image ID it was created from (Config.Image); never pull or resolve the request tag. The requested reference is still used for child metadata so the retained workload's environment is byte-identical to its original launch. Addresses GATOR-291f43b2-01. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * test(e2e): verify host-only VM trust delivery and drop curl dependency The additional-CA lane's VM assertions read the destination CA out of the guest overlay and a guest-side supervisor-argv marker — artifacts from the guest-supervisor design that no longer exist now that the VM driver stages the CA host-only and runs the supervisor on the host. Inspect the driver-staged host artifact (digest-compared against the source), assert the guest overlay contains no CA material at the old mount paths, and keep the serial-console no-disclosure check. The workload validation scripts also ran curl, which exists only in the container-engine fixture image; the Kubernetes and VM lanes use the uv-based workload image and died instantly with "canonical main process already finished". Rewrite the main-process, exec, and lifecycle validation scripts in python3 (present in every lane's workload image). The network-policy binary identity is the resolved /proc/<pid>/exe target, so the versioned python3.12 binaries are allowlisted rather than the unversioned symlinks. The hostname mismatch check keeps its semantics: same reachable fixture, mismatched TLS SNI, and the failure must specifically be hostname verification. Addresses GATOR-291f43b2-04. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(kubernetes): allow admission-added labels on destination-trust ConfigMaps The destination-trust ConfigMap validation compared the whole label map for equality. A mutating admission webhook that attaches an organizational label to the created ConfigMap made the create response fail validation, and the retained ConfigMap then failed every subsequent create/start through the existing-object path, blocking sandbox launches despite the ownership labels, immutable flag, digest, and certificate bytes remaining correct. Validate the gateway-owned label pairs individually and allow unrelated labels; the exact trust-material checks are unchanged. Covered by a regression test with an admission-added label and a missing gateway-owned label. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(podman): close supervisor-refresh rollback gaps The supervisor-only trust refresh renamed the previous supervisor to a random rollback name and then performed the token-secret lookup. A transient lookup failure returned with the supervisor still renamed, where a normal delete could never find it, and the rollback rename and final rollback removal ignored their own failures, leaving orphans with attached control and channel resources. Complete every fallible preparation before the rename. On a failed replacement, restore the canonical name; if the restore itself fails, return an error naming the rollback container so the orphan stays identifiable — a later retry still succeeds because the canonical name is absent and the replacement path recreates a fresh supervisor. Treat not-found during the final rollback removal as success and name the orphan in both the log and the error on other removal failures; a retry observes the fresh canonical supervisor and resumes the normal start path. Covered by failure-injection tests for each transaction boundary: preparation failure before any rename, replacement failure restoring the canonical name, and restore failure naming the orphaned rollback container. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * docs: address destination-trust review corrections The gateway-configuration RFC documented an additional_ca_cert_paths list that the strict configuration parser never accepted; document the additional_ca_bundle single-path setting the parser actually enforces. The debug-openshell-cluster skill named the wrong generation annotation (openshell.ai/network-supervisor-trust-generation) and described digest argv and network-init canonicalization checks the implementation does not perform. Correct it to openshell.ai/additional-ca-generation and describe the actual delivery: the supervisor Pod mounts the immutable ConfigMap read-only and receives --additional-ca-bundle for the mount path, with the driver validating the digest annotation and exact bytes per creation. Classify the destination-trust generation hashes as deterministic identity derivation rather than authenticity proofs, following the openshell-crypto exclusions table: trust decisions come from the gateway's normalization, immutable staging, exact-byte validation, and TLS verification. Record the classification in the crypto README and at each hashing site. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(network): reconcile additional CA tests with suspend/resume and auth changes Upstream's suspend/resume lifecycle took StartSandboxRequest fields 7 and 8, so the durable-snapshot field moves from 7 to 9; update the wire-format test to assert the field-9 key (0x4a) and initialize the new resume controls. Repair the cli test module after the auth opt-in test merged alongside the environment-guard helpers, and initialize the gateway driver-startup context's network-trust fields in the shared supervision test. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(server): box start-recovery futures for clippy large-futures The durable-snapshot field grows StartSandboxRequest enough that the start-recovery match and the recovery-expiry test exceed the large-futures limit under -D warnings. Box::pin both futures, matching the pattern the recovery path used before the suspend/resume rework. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> --------- Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> |
||
|
Jesse Jaggars
|
26134dc735 |
feat(gateway): add hashed service-routing hostnames for multi-gateway clusters (#4842)
* feat(gateway): add hashed service-routing hostnames for multi-gateway clusters
Closes #4840
Add an optional service_routing_gateway_name gateway setting (TOML, CLI,
env, Helm) that disambiguates exposed service URLs when multiple gateways
share a cluster, ingress, and wildcard certificate. When configured,
exposed-service hostnames become a deterministic 16-character lowercase
base32 SHA-256 label of (gateway name, workspace, sandbox, service) under
the same base domain, keeping every hostname exactly one label deep for
wildcard-certificate compatibility. Unset gateways preserve the existing
readable hostname format byte-for-byte, and configured gateways still
accept legacy hosts during migration.
Hashed hosts resolve through an in-memory route index hydrated from
persisted ServiceEndpoint records at startup, refreshed on miss for
multi-replica deployments, maintained across expose, delete, and sandbox
cleanup, and enforced for collisions before endpoint persistence.
Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>
* fix(lint): satisfy clippy doc lints on main-landed sandbox_runtime code
The shared-supervisor landing (
|
||
|
Drew Newberry
|
e5adf6f484 |
feat(supervisor): recover shared instances with hash-based HA (#4864)
* feat(supervisor): recover shared instances with hash-based HA Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(supervisor): keep HA coordination out of compute drivers Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(supervisor): fence pool admission and reconciliation races Signed-off-by: Drew Newberry <anewberry@nvidia.com> * feat(supervisor): keep pool mode stable across replica scaling Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(supervisor): default shared deployments to pool mode Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(supervisor): fence refused launches and remove legacy mode Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(supervisor): confirm live owners and fence stale observations Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(supervisor): preserve takeover cause log formatting Signed-off-by: Drew Newberry <anewberry@nvidia.com> * feat(helm): add memory-based supervisor autoscaling Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(supervisor): retain acknowledgements and fence streamed input Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs(supervisor): consolidate experimental multi-tenant guidance Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs(supervisor): simplify multi-tenant supervisor guide Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(supervisor): simplify pool member identity Signed-off-by: Drew Newberry <anewberry@nvidia.com> * feat(helm): make supervisor affinity configurable Signed-off-by: Drew Newberry <anewberry@nvidia.com> * feat(helm): make supervisor disruption budget optional Signed-off-by: Drew Newberry <anewberry@nvidia.com> * test(supervisor): align HA fixtures with suspend support Signed-off-by: Drew Newberry <anewberry@nvidia.com> --------- Signed-off-by: Drew Newberry <anewberry@nvidia.com> |
||
|
Brooke Storm
|
c4688aa871 |
feat(helm): add certificate hook pod labels and annotations (#4895)
Signed-off-by: Brooke Storm <brookes@nvidia.com> |
||
|
Eric Curtin
|
16250df210 |
feat(helm): add server.extraEnv for the gateway container (#4876)
Signed-off-by: Eric Curtin <eric.curtin@docker.com> |
||
|
Drew Newberry
|
9a85667d0d |
feat(kubernetes): host sandboxes on the shared supervisor (#4702)
* feat(sandbox-backend): route runtime connections to shared supervisor sessions A supervisor shared by several sandboxes needs one listener for every session's runtime-initiated Sandbox Protocol connections. The runtime now sends a fixed, bounded routing preface (magic, version, length, and an opaque route handle) before its pinned TLS handshake when its bootstrap names a route. RuntimeRouter treats the handle only as a hint that selects which generation-pinned TLS identity to demand, closes unknown, removed, and replaced handles before any handshake, bounds unauthenticated handshakes globally and per route, and queues an authenticated connection only for its own session, so no peer can displace a live channel. A routed session waits up to the bootstrap grace for its first connection, because it is registered before its workload Pod is scheduled. The session_control module defines the trusted contract between a compute driver and a shared supervisor: registering one generation's runtime descriptor and gateway-issued credentials, reading its status, and removing it, over mutually authenticated TLS 1.3. Dedicated descriptors and bootstrap listeners keep their encodings. Signed-off-by: Drew Newberry <anewberry@nvidia.com> * feat(supervisor): host gateway-registered sessions in one process openshell-supervisor experimental-host replaces the idle bootstrap process with a session host. A mutually authenticated session-control API accepts registrations only from the admission certificate holder, the gateway. Each registration starts a SessionHost session that acts only with the gateway credentials issued for its generation, keeps its proxy CA, trust directory, and SSH socket under a private directory, and receives its runtime connections through the routed listener. Status reports each session's own readiness; the health socket keeps reporting process health only. Removing a session runs on a task the host owns, so it completes when the caller disconnects, concurrent removers share its result, and the route stays reserved until the session has ended and its files are gone. An identical registration is accepted again and a different one for a registered route is refused. Sessions get no upstream proxy, extra CA bundle, or provider workload identity; the Kubernetes driver refuses configurations that need them. Signed-off-by: Drew Newberry <anewberry@nvidia.com> * feat(server): issue a TLS-only private PKI from generate-certs generate-certs --tls-only creates just a server and a client TLS Secret from a fresh CA, with the same create-if-missing contract and a recovery hint for partial state. The Helm certgen hook uses it for private PKIs that must not carry a sandbox-JWT signing key. Signed-off-by: Drew Newberry <anewberry@nvidia.com> * feat(kubernetes): supervise gateway-topology generations as shared sessions With experimental_supervisor_topology = "gateway", each new runtime generation becomes a session on the gateway's shared supervisor instead of a supervisor Pod and boundary Service of its own. The generation records its placement and session route on its Sandbox resource when it starts, and every later decision reads that record, so changing the setting never reinterprets a running generation and unknown records fail closed. The runtime identity ends with the session route, which no Pod UID can equal, so Pod authentication cannot vouch for a shared generation. Create and start add a per-sandbox egress allowance to the Pods the shared supervisor's Service selects, bind it into the boundary evidence, and register the session over session control before releasing the workload; registration does not wait for the runtime, so create still returns before the gateway persists the generation. Readiness follows the session's own status. Stop, restart, and delete remove only that session, and reconciliation suspends a workload whose session is gone and removes sessions that no current generation may run. Configurations with an upstream proxy or provider SPIFFE identity are refused. Closes #4700 Signed-off-by: Drew Newberry <anewberry@nvidia.com> * feat(helm): deploy and wire the shared supervisor With the gateway topology enabled, the chart runs the session host as a single-replica Deployment with its Service and a NetworkPolicy that admits session control only from the gateway and runtime connections only from sandbox workload Pods. The certgen hook issues a private session-control PKI whose client credential only the gateway mounts, the gateway may read only that Service, and the chart renders the matching experimental_shared_supervisor table. The supervisor Pod has no service account token. Settings the shared supervisor cannot isolate, and a hand-written connection table, fail at render time. Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(ocsf): attribute events to session-bound contexts A supervisor process that hosts several sandboxes cannot use one process-wide OCSF context: every event would carry the wrong sandbox identity, and middleware requests read their sandbox identity from the same context. Builders now accept a ContextRef, either a borrowed context or a context shared by a hosted session, so per-session contexts need not be leaked. ctx() returns the context bound to the calling thread when present and falls back to the process-wide context otherwise. OCSF JSON output settings can likewise be bound per thread, and OpenTelemetry can parent a span from an explicit W3C trace context instead of process environment. A dedicated supervisor keeps using the process-wide context. Signed-off-by: Drew Newberry <anewberry@nvidia.com> * test(e2e): run two sandboxes on one shared Kubernetes supervisor The e2e:kubernetes:shared-supervisor lane enables the gateway topology and creates two sandboxes with different policies and provider credentials. It checks that one shared supervisor process hosts both, that each session enforces its own policy and injects its own credential, that stopping one leaves the other working, that the routed listener refuses stale and unknown routes and drops a peer that names a live route without its pinned identity, and that a restart registers a fresh session on the same process. Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(supervisor): scope gateway credentials and network state to a session The supervisor kept its gateway bearer, Sandbox Protocol bearer, and renewal task in process-wide slots, and the network supervisor read provider token grants, proxy CA files, and the gateway identity from process environment. A second sandbox in the same process would have replaced or rotated the first sandbox's credentials, reused its provider access tokens, and exposed its denials through policy.local. GatewayCredentials now holds one session's gateway and Sandbox Protocol credentials and their renewal; every gateway client receives a GatewayEndpoint that pairs the URL with them, including log push, ConnectSupervisor and its redirects, and extension-credential rotation. run_networking takes a NetworkSession with the sandbox identity, the session's gateway access, its Workload API socket, its policy.local denial source, and its proxy CA files. Provider token grants use a cache and HTTP client owned by the session and fail closed without a Workload API socket. run_session supervises one generation from an explicit SessionConfig and owner-provided SessionControls. run_sandbox_with_backend remains the dedicated entry point: it keeps the process-wide OCSF context, reads the same environment, binds readiness endpoints to session readiness, and stops on SIGTERM or SIGINT. The binary shares one set of credentials between log push and supervision, as before. Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs: describe the experimental shared supervisor Document the gateway supervisor topology, its chart-owned connection settings and limits, and how to debug and test it. Signed-off-by: Drew Newberry <anewberry@nvidia.com> * feat(supervisor): host concurrent sessions on session-bound runtimes SessionHost runs several supervisor sessions in one process. Each session gets its own Tokio runtime, and every thread that runtime starts, plus the thread driving it, is bound to the session's OCSF context, OCSF JSON settings, log sink, and denial log. Tasks, blocking work, reconnects, and credential renewal spawned by a session therefore keep its identity, while threads outside any session carry none and push no logs. Gateway credentials stay explicit per session. Removing a session stops its workload gracefully, or cancels it at once while it is still starting, then shuts down only its runtime and revokes its gateway credentials. The key stays registered until the session thread has finished, so the same generation cannot gain a second owner while one is stopping. A panicking session reports a failure, clears its readiness, and is revoked the same way. Hosted sessions get a private proxy trust directory and serve policy.local denials from their own log. A focused test runs two sessions concurrently and checks that nested and blocking tasks see their own identity, gateway requests carry their own bearer, log lines reach only their own stream, and removing one leaves the other supervising. Closes #4699 Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix: tighten shared supervisor naming, removal deadlines, and route probes The shared supervisor's resource name appended its suffix before truncating, so a release fullname of 51 or 52 characters produced the gateway Service's own name. A fullname without room for the suffix now keeps a prefix and a hash of the whole fullname. Removing a session waits up to 20 s for its workload to stop and 5 s for the session runtime to shut down, but the driver gave DELETE the same 15 s budget as other session-control calls and reported normal removals as failures. The contract now defines the removal grace and a 45 s removal deadline, and deleting a sandbox no longer waits on a session whose workload is already going away. The shared host runs no supervisor OTLP trace or OCSF exporter. The driver warns at startup when the gateway configures that export, and the docs state the limit. experimental-host keeps its Unix-only imports in its Unix entry point. The e2e route probe fails when the supervisor neither refuses nor challenges a connection, or keeps a challenged connection open after a malformed TLS answer, and it checks that every probe reached the supervisor Pod. Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(supervisor): route shared supervisor crypto through openshell-crypto Main now routes first-party TLS providers and digests through the openshell-crypto facade. Build the session-control TLS configurations and the routed-runtime test server from its configuration provider, install the experimental host's default provider through it, and hash shared session registrations with its SHA-256 instead of a direct sha2 dependency. Signed-off-by: Drew Newberry <anewberry@nvidia.com> * feat(supervisor): report shared supervisor health over TCP experimental-host takes --health-port instead of a Unix health socket, matching experimental-bootstrap. It binds the wildcard health listener after the session-control and runtime listeners and accepts and closes connections while the process runs. The chart probes the named container port health (5502) with native kubelet tcpSocket checks; session control stays on 5501, the Service exposes only runtime and session control, and /run/openshell remains for per-session files. Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(supervisor): probe shared supervisor health on session control The shared supervisor's kubelet tcpSocket probes now connect to the mTLS session-control listener (5501) instead of a separate health listener. Drop the experimental-host --health-port flag, its accept loop, and the chart's 5502 container port. Probes open and close a TCP connection; every session-control request still requires mTLS. Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(supervisor): use shared mode flags for the session host Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(supervisor): describe the shared host module by its mode Drop the experimental label from the shared host module documentation so it matches the --mode=shared entry point. No behavior changes. Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(test): preserve shared supervisor workspace overlays Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(supervisor): name hosted lifetimes sandbox instances A sandbox instance is one running generation of a stable Sandbox resource. Gateway transport connections and the gateway supervisor sessions they carry can reconnect within one instance, so the hosted object, its configuration, controls, host, and per-instance network, denial, and token-grant state now use the sandbox instance name. Rename session_host to sandbox_instance_host, run_session to run_sandbox_instance, and SessionConfig, SessionControls, SessionHost, and related hosted types to their SandboxInstance equivalents. Gateway supervisor sessions, SSH sessions, and authentication session types keep their names. No wire, configuration, or lifecycle behavior changes. Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(supervisor): launch sandbox instances from one config and prepared bindings Replace SandboxRunConfig and its field-by-field copy into SandboxInstanceConfig with a single launch configuration. Live per-instance state (gateway credentials, OCSF context and JSON settings, log sink, and denial log) is now an explicit SandboxInstanceBindings value that the owner prepares once, instead of placeholders the runner or host overwrote. The dedicated binary loads launch inputs from its arguments and environment and shares one GatewayCredentials with log push; run_sandbox_with_backend keeps the process-wide OCSF context, health endpoints, and signal handling. The instance host binds the same SandboxInstanceBindings to its runtime threads and revokes their credentials when the instance ends. Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(supervisor): use session-control wire types for instance status and trace context Drop the supervisor's duplicate SandboxInstancePhase, SandboxInstanceStatus, and TraceContext families. The sandbox instance host reports SandboxInstanceStatusReport directly, with its key as the route, so the shared host no longer converts or re-keys status, lists, or removal results. SandboxInstanceConfig carries RegistrationTraceContext and the dedicated process reads it through trace_context_from_environment. No wire definitions change. Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(supervisor): update Windows host proxy test config Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(supervisor): assign shared sandbox instances over a supervisor-initiated gRPC stream Replace the private HTTP/JSON session-control API, which the gateway dialed on the shared supervisor, with a supervisor-initiated stream to the gateway. The gateway owns the desired set of sandbox instances; the shared supervisor attaches with a gateway-scoped host certificate, receives complete assignment snapshots with session-scoped revisions, reconciles its instance registry to each one, and reports what it observes. Losing the stream never stops an instance; only a later snapshot that omits it does. - Add the private openshell.supervisor.host.v1 protocol, served on the gateway listener but authorized only for the supervisor host certificate, excluded from SDK generation and reflection, and classified in the schema inventory. - Add SupervisorHostController in openshell-core: uninitialized until the driver derives the desired set once, compare-and-replace full resyncs, capacity bound, session fencing by connection epoch, revision-bounded acks, bounded sends. - Classify TLS peers per connection against the ordinary client CA and the supervisor host CA from one configuration snapshot; a chain matching neither or both is refused, and host connections never become users. - Persist each shared generation's launch material in its supervisor Secret so a restarted gateway rebuilds assignments from Sandbox resources, and withdraw instances on stop, delete, and suspend. - Issue the host identity with `generate-certs --client-only`; the gateway mounts only the CA. Drop the control port, server certificate, and gateway ingress rule from the chart; require gateway TLS for the topology. Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(supervisor): authenticate the shared supervisor stream with a gateway-minted supervisor token Replace the shared supervisor's host CA and client certificate with a supervisor token: a bearer the gateway mints from its existing launch-signing key under a profile of its own (typ openshell-supervisor+jwt, subject openshell-supervisor:<gateway_id>, no sandbox, generation, or epoch claims). The supervisor dials the gateway's Attach stream over ordinary server-authenticated TLS and presents the token; the gateway verifies it with the same key infrastructure, grants only the assignment stream, refuses every other credential on it, and ends a stream when the token that opened it expires. The gateway renews the token at half its lifetime and publishes each renewal into the supervisor host controller; the Kubernetes driver's only part in authentication is delivery, replacing the token in the chart-created <release>-supervisor-token Secret, which the supervisor mounts and re-reads on every connection attempt. The gateway Role may get and patch only that Secret. Remove the host-only TLS realm classification, the supervisor host CA mount and config, certgen --client-only, and the supervisor host PKI; gateway TLS returns to its pre-feature behavior and remains required for the topology. Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(helm): remove redundant certgen condition Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(supervisor): extract gateway-owned shared supervision into openshell-supervisor-control Move the shared supervisor control types out of openshell-core and openshell-sandbox-backend into a new openshell-supervisor-control crate: the SupervisorHostController (desired set, revision fencing, host sessions, observations), the SandboxInstanceRegistration launch material and SandboxInstanceStatusReport mapping, and a new SharedSupervision coordinator that owns the control contracts a compute driver previously implemented itself: validated persist-before-assign launches, recovery from durable records with the compare-and-replace baseline captured before the driver lists them, keep-previous-set on any failed or mismatched read, and one interpretation of host observations for readiness and bootstrap. Supervisor token delivery keeps its retry and newer-value policy in the crate; the write itself stays with the driver. The gateway now constructs the coordination: ComputeDriverBuildContext exposes shared_supervision() and the server keeps the controller for the Attach stream and token minting, replacing the driver-created controller that was installed into the gateway. The Kubernetes factory injects it through KubernetesComputeDriver::with_shared_supervision; new() stays the dedicated-topology constructor and refuses a configuration that selects the gateway topology without the gateway's coordination. The Kubernetes driver keeps its platform work: placement annotations, the shared supervisor Service and egress NetworkPolicy, launch Secret create and read (bytes only), its own fence and Pod evidence, and the token Secret patch. The supervisor process consumes the moved registration and status types unchanged. Wire protocol, storage JSON, annotations, and the supervisor token profile are unchanged; the private protobuf stays generated in openshell-core and is re-exported by the new crate. Signed-off-by: Drew Newberry <anewberry@nvidia.com> --------- Signed-off-by: Drew Newberry <anewberry@nvidia.com> |
||
|
Drew Newberry
|
d22394370d |
fix(helm): allow shared supervisor SCC-assigned IDs (#4785)
Signed-off-by: Drew Newberry <anewberry@nvidia.com> |
||
|
Drew Newberry
|
72ab4dd9ef |
feat(kubernetes): deploy shared supervisor mode (#4695)
* feat(kubernetes): bootstrap experimental gateway supervisor topology Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs: move the experimental supervisor topology under an Experimental section Signed-off-by: Drew Newberry <anewberry@nvidia.com> * feat(supervisor): report experimental bootstrap health over TCP experimental-bootstrap now takes --health-port instead of a Unix health socket. It binds the dedicated supervisor's wildcard readiness listener, reachable at the Pod IP, and accepts and closes connections while the process runs. The chart probes the named container port health (5502) with native kubelet tcpSocket checks and drops the health-only emptyDir. No filesystem state remains to clean up after an unclean exit. Dedicated supervisors keep --health-socket-path and health --socket. Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs: defer public experimental supervisor docs Remove the experimental supervisor topology additions from the published gateway configuration page and the public debug-openshell-cluster skill. Public documentation will be added once the experimental stack settles. Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(supervisor): replace bootstrap with shared host stub Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(supervisor): name shared supervisor subcommand shared Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(supervisor): select shared mode with explicit endpoint flags Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(supervisor): name the shared host module after its mode Rename the shared-mode host module from experimental to shared to match the --mode=shared flag. No behavior changes. Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(kubernetes): drop the experimental prefix from the supervisor topology Rename the Helm template to supervisor.yaml, its Kubernetes names and labels to the -supervisor suffix and supervisor component, and the driver setting to supervisor_topology. The setting remains opt-in and workspace remains reserved. Signed-off-by: Drew Newberry <anewberry@nvidia.com> --------- Signed-off-by: Drew Newberry <anewberry@nvidia.com> |
||
|
Yuedong Wu
|
44c07dab2a |
feat(helm): make gateway PVC size and StorageClass configurable (#3854)
Signed-off-by: Yuedong Wu <dwcn22@outlook.com> |
||
|
krishicks
|
07fd2386ce |
feat(server): export gateway OCSF events as OTLP log records (#4500)
The gateway only persisted its OCSF events to the JSONL file, so collectors had to tail that file. Adding `[openshell.gateway.otlp.ocsf]` now also exports each gateway OCSF event as an OTLP log record, using the same resource identity as gateway traces. `schema_version` optionally downgrades the exported JSON to OCSF 1.1 or 1.3. OTLP endpoints are now configured per signal, so OCSF events can go to a collector that feeds a SIEM while spans go elsewhere. The existing `[openshell.gateway.otlp]` `endpoint` is the default for every signal and on its own still enables trace export. `[otlp.traces]` overrides the trace endpoint, and `[otlp.ocsf]` takes its own `endpoint` or falls back to the default. Each record carries the OCSF JSON as its body, the event time as its timestamp, a severity mapped from `severity_id`, and `ocsf.*` and `openshell.sandbox.id` attributes for filtering without parsing the body. Records flow through a queue bounded by count and bytes to the SDK OTLP log exporter in batches bounded the same way. Failed exports retry with backoff; records lost to a full queue, an oversized body, or exhausted retries are counted in metrics and reported to the collector in an `openshell.ocsf.gap` record on the next successful export. The exporter lives in `openshell-otel` so other processes can share it. Helm deployments set the `traces` and `ocsf` tables through `gatewayConfig`, and the local development gateway tasks enable OCSF export. Closes #4499 Refs #2892 Signed-off-by: Kris Hicks <khicks@nvidia.com> |
||
|
Yuedong Wu
|
3f66c26af1 |
fix(helm): render Restricted-compatible security contexts (#3340)
Signed-off-by: Yuedong Wu <dwcn22@outlook.com> |
||
|
|
e30c938bb1 |
feat(server): scope mutation locks and bound lock waits (#4147)
* test(server): add a local PostgreSQL test runner
The openshell-server tests that need a real PostgreSQL server are
ignored by default and had no shared way to run. The mutation-replay
test reads its own OPENSHELL_REPLAY_TEST_DATABASE_URL, so every
contributor had to provision a database by hand, and the advisory-lock
tests that follow in this series need the same setup.
Add mise run test:rust:postgres. It runs every ignored postgres_* test
in openshell-server against OPENSHELL_TEST_POSTGRES_URL, or starts a
disposable PostgreSQL container with Docker or Podman (CONTAINER_ENGINE
selects one) on the image pinned by the Kubernetes e2e fixture and
removes it on exit. Tests run one at a time because advisory locks are
database-wide. The runner always points the legacy replay variable at
the selected database, so an inherited URL cannot send that test to a
different server. test:postgres-runner checks that selection with a fake
cargo and needs no database or container engine. CI does not run the
PostgreSQL tests; TESTING.md documents the task.
Part of #3528
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* fix(server): hold the mutation guard while deleting a provider
DeleteProvider checked that no sandbox referenced the provider and then
deleted the record without holding the sandbox mutation guard. Sandbox
create and provider attach take that guard while they write provider
references, so one of them could add a reference after the attached
sandbox check passed, and the delete then removed a provider that a
sandbox spec still named.
Take the guard before the attached sandbox check, as provider create and
update already do, so the check and the delete run against a stable
set of sandbox references, and reject an empty name after authorization
but before the guard. A new test holds the guard, attaches the provider
while the delete waits, and asserts that the delete fails with
FailedPrecondition and leaves the provider in place.
Part of #3528
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* fix(server): release staged credentials when a refresh fails early
A provider credential refresh stages the minted values under new
credential handles before it commits the provider update, and deletes
those handles when validation or persistence fails. Two earlier returns
skipped that cleanup. The minted expiry was converted to a protobuf
timestamp only after staging, so an expiry outside the timestamp range
failed the refresh and left the staged values in the credential driver.
A failure to acquire the sandbox mutation guard returned the same way.
Convert the expiry before staging anything, so a bad value fails before
any handle exists, and delete the staged handles when the guard cannot
be acquired. A new test refreshes a stored credential with an expiry of
i64::MAX and asserts that the call fails, the provider keeps its
original handles, and the credential driver holds the same number of
values as before. The guard failure path has no test here, because the
SQLite guard used by unit tests cannot fail.
Part of #3528
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* feat(server): acquire mutation locks on a dedicated PostgreSQL lock pool
The cross-object mutation guard took its advisory-lock session from the
10-connection data pool and closed it on every release, so each guarded
mutation opened a new database connection, and lock holders competed
with their own critical sections for data connections. A lock wait that
hit the 10 second lock_timeout surfaced as INTERNAL.
Add a lazy, dedicated 4-connection lock pool and a new
persistence::mutation_lock module for the lock key, mode and set types.
Store::acquire_distributed_mutation_guard now takes a lock set and a
deadline, and each lock statement sets lock_timeout to the time left
before that deadline, so a cancelled acquisition never waits past it on
the server. Returned connections are scrubbed with
pg_advisory_unlock_all() and reused, and each return is bounded by 5
seconds so a stalled session cannot keep a pool permit.
A lock statement that PostgreSQL ends with SQLSTATE 55P03 fails with a
new LockTimeout error, and so does a wait for a lock connection during
which every one was checked out at some point, or that started with less
than a second left, too little to open one. The API returns it as
UNAVAILABLE with reason MUTATION_LOCK_TIMEOUT and a 1 second retry
delay, because the mutation lock was not acquired, so the request's
guarded writes did not run. Only the guard acquisition classifies 55P03;
any other 55P03 stays a database error. A lock connection that
PostgreSQL does not open by the deadline despite at least a second to do
so (refused, out of connection slots, starting up, or not answering, all
of which SQLx retries silently) is a database error, not a lock timeout.
Mutations still take the legacy global key exclusively, so old and new
replicas keep excluding each other during a rolling upgrade. Each
PostgreSQL-backed gateway pod can now open up to 14 database
connections (10 data plus 4 lock) instead of 10, and the chart's
autoscaling.maxReplicas comment says what its default of 4 needs. The
SSH host identity lock stays on a data connection, because sandbox
creation takes it while holding its mutation guard and a second
lock-pool connection there could exhaust the pool. The provider
credential refresh lock stays there too, so a refresh still progresses
while every lock connection is in use.
Part of #3528
Co-authored-by: divesh <dgude@nvidia.com>
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* perf(server): partition mutation locks by workspace and sandbox
Every cross-object mutation, and every lifecycle and supervisor update,
took one process-wide mutex and, on PostgreSQL, one fleet-wide advisory
lock. Unrelated sandboxes and providers therefore serialized across all
replicas, and one slow critical section delayed every other mutation in
the fleet.
Replace that lock with hierarchical intention locks on a global key, one
key per workspace and one key per sandbox, each held shared or
exclusive. Global settings and policy writers and platform-scope profile
writers hold the global key exclusively. Provider and workspace-scoped
profile writers, including DeleteProvider and credential refresh, hold
their workspace key exclusively. Sandbox writers and supervisor reports
hold their sandbox key exclusively under shared global and workspace
keys. Lifecycle, driver-watch and reconcile paths take only
process-local keys and keep relying on compare-and-swap across replicas.
Workspace and sandbox keys are domain-separated SHA-256 prefixes. Each
guard takes its local keys and then its advisory locks in ascending
order under one 10 second deadline, and no path nests guards. The
settings mutex is removed, since the global and sandbox keys cover it.
Sandbox settings are keyed by sandbox name, which a sandbox key no
longer covers once another sandbox reuses the name, so a settings
update returns NOT_FOUND unless its sandbox still holds the name after
the read, and ABORTED if the settings row was recreated before the
write.
The global key keeps its legacy value, so older replicas, which hold it
exclusively for every mutation, still exclude new ones during a rolling
upgrade. Guards now also give up after 10 seconds on SQLite, and
provider refresh reports a guard timeout as UNAVAILABLE instead of
INTERNAL. A supervisor report for a missing sandbox returns NOT_FOUND.
Startup endpoint-status reconciliation still holds the global key for
its whole scan.
Part of #3528
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* fix(server): reconcile endpoint status on startup one sandbox at a time
Startup endpoint-status reconciliation held the global mutation key
exclusively for its whole scan. On PostgreSQL that blocked every guarded
mutation on every replica until the starting gateway had walked all
sandboxes, and the scan first waited behind every in-flight sandbox
mutation. It also paged with an offset, so sandboxes deleted mid-scan
shifted rows between pages, and a write that raced the scan (a Conflict
from a lifecycle update on another replica, or a sandbox deleted after
it was listed) failed gateway startup.
Walk the sandboxes with keyset paging instead, and reset each candidate
under its own sandbox-scoped guard, up to four at a time to match the
lock pool. A candidate with a fresh shared owner is skipped before any
guard is taken. Under the guard the reconciliation re-checks the owner,
re-reads the sandbox, skips it when it is gone or no longer carries
endpoint status, and writes against the version it just read. A Conflict
or a database error at write time, which is how a row deleted after the
re-read surfaces, is retried with a fresh read, at most five attempts
per sandbox. A lock timeout on the guard is retried the same way, so a
holder of the sandbox, its workspace or the global key that releases
within the retry window cannot stop a replica from starting. No
lock-timeout retry starts once the scan has run for three lock timeouts,
so a scan stuck behind a long holder fails within about 40 seconds,
inside the chart's 60-second startup probe, and logs the sandbox it gave
up on. Store::list_all_messages loses its only caller and is removed.
Reconciliation stays fail-closed: a holder that outlasts the retry
window, an owner lookup failure, or a sandbox that keeps changing still
fails startup, so stale endpoint success is never served. User-visible
change: a restarting gateway no longer stalls mutations of unrelated
sandboxes and providers on its peers while it reconciles.
Part of #3528
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* perf(server): skip the endpoint-status lock when a session was replaced
When a supervisor session ends, the gateway resets that sandbox's tool
server endpoint evidence unless a replacement session already owns
endpoint observation. The replacement check ran only after the reset
had read the sandbox and taken its sandbox mutation guard, so a reset
whose supervisor had already reconnected still queued behind other
mutations of that sandbox, and on PostgreSQL it used a lock-pool
connection, only to find it had nothing to do.
Check for a replacement first: a live local session, or a fresh owner
record on a peer. When one exists, return without reading the sandbox
or taking the guard. Otherwise take the guard and repeat the same check
under it, because a replacement's pre-acknowledgement reset runs under
the same sandbox key and resetting after it would wipe the
replacement's fresh evidence. Both checks share one helper, so they
cannot drift apart. When no replacement exists, the reset costs one
extra owner-row read.
This helps most when a replica notices a dead stream late, after the
supervisor has already reconnected elsewhere (a network partition or a
keepalive expiry).
Deferring to a replacement leaves the old session's evidence to the
replacement's pre-acknowledgement reset, which can time out on its
bounded mutation lock wait. That reset's failure path removed the
replacement and released its owner record but scheduled no
invalidation, so the old session's success evidence could outlive both
sessions. An old session that ended after its replacement registered
already deferred this way, and the unguarded check now makes every
disconnect that finds the replacement defer too. When the reset fails
while the replacement is still current, also schedule the retried
invalidation that follows a session end. That failure path keeps the
lifecycle demotion from abandon_session_setup, but runs it in the
spawned task before the retry and only while this session is still
current, so a timed-out reset returns promptly.
Part of #3528
Co-authored-by: divesh <dgude@nvidia.com>
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* test(server): cover staged credential cleanup on a lock timeout
A provider credential refresh deletes its staged handles when it cannot
acquire the provider mutation guard, but that path had no test: the
SQLite guard used by unit tests could not fail. Mutation locks now
acquire under a deadline that tests can shorten, so the timeout path
can be exercised directly.
The new test shortens the lock deadline to 50 ms, holds the workspace
mutation guard, and refreshes a provider that already stores an AWS
access key. The refresh returns UNAVAILABLE with the
MUTATION_LOCK_TIMEOUT reason, the provider keeps its original handle,
which still resolves to the old key, and the credential driver holds
the same number of values as before the refresh.
Part of #3528
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* feat(server): export mutation lock wait and timeout metrics
A handler mutation guard can now wait up to 10 seconds for its keys and
then fail with UNAVAILABLE, but nothing showed operators how long guards
waited or how often they gave up, short of reading warning logs on each
replica.
Export openshell_server_mutation_lock_wait_seconds, a histogram of the
time one guard took to acquire all of its keys (local registry plus
PostgreSQL advisory locks), recorded on success, and
openshell_server_mutation_lock_timeouts_total, a counter of guard
acquisitions that timed out. Both carry a bounded scope label: global,
workspace or sandbox. The platform scope, whose workspace is empty,
counts as global. The histogram uses the same 1 ms to 15 s buckets as
the relay and peer histograms, and the timeout counter starts at 0 for
every scope, so rate() works on an idle replica. The timeout warning
now also logs the scope. A guard that fails for another reason, such as
a lock connection PostgreSQL did not open, is not counted as a timeout
and logs a "mutation lock acquisition failed" warning instead.
Lifecycle, driver-watch and reconcile paths take only process-local
locks and never time out, so they are not recorded.
Part of #3528
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* test(server): add PostgreSQL mutation lock integration tests
The scoped mutation locks were only unit-tested on SQLite, where the
database layer is a no-op. Add ignored postgres_* tests that run the
advisory-lock layer against a real PostgreSQL through mise run
test:rust:postgres, each in its own disposable schema (TestSchema).
persistence/mutation_lock_pg_tests.rs uses two stores on one database
as two replicas. It covers disjoint sandbox scopes holding at once, a
workspace key blocking only sandboxes in that workspace, the global key
blocking every scope, and exclusion in both directions against a raw
holder of the legacy global key, whose 55P03 stays a database error
outside a guard acquisition. It checks that timed-out and cancelled
acquisitions release the keys they already took, that a released lock
connection returns to the pool without residual locks, that a release
stalled on the network closes its session and the pool recovers, and
that the lock pool never grows past its size. It also checks that a
full lock pool times out as lock contention, while a lock connection
that PostgreSQL never answers fails as a database error.
compute/mutation_guard.rs runs the random scope mix across two
PostgreSQL runtimes and checks that a by-id guard waits for a workspace
writer on the other replica. The provider refresh lock test moves onto
the runner too, and checks that a refresh proceeds while every lock
connection is held.
A bench_postgres_* capacity check measures a paced burst of 500
reconnects 12 ms apart into one replica and fails when the p99 lock
wait reaches a fifth of the lock timeout. Machine load moves those
waits, so mise run test:rust:postgres skips it and mise run
test:rust:postgres:bench runs it alone, through a new
OPENSHELL_TEST_POSTGRES_PREFIX setting in the runner.
The tests add test-only helpers: the lock pool's size and idle count
and the backend pid of a held lock session. Nothing changes outside
tests and the test runner.
Part of #3528
Co-authored-by: divesh <dgude@nvidia.com>
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* ci: run the PostgreSQL-backed server tests
Branch Checks now runs the postgres_* tests through the same runner
developers use, on an x86_64 Linux runner with Docker. They cover lock
scope exclusion, interop with replicas still on the old global key,
cancellation, lock-pool bounds and connection failures, which no other
CI lane checks directly. The load-sensitive bench_postgres_* capacity
checks stay local.
Part of #3528
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* docs: document scoped mutation locks and lock-pool sizing
In the High Availability guide, size PostgreSQL for 14 connections per
gateway pod, 10 for data and 4 for the new lock pool, so three or more
replicas now exceed what the PostgreSQL default leaves, and keep
headroom for lock sessions that cancelled requests leave behind.
Explain which changes lock a sandbox, a workspace, or the whole fleet,
and that a lock wait over 10 seconds returns a retryable UNAVAILABLE
with reason MUTATION_LOCK_TIMEOUT. A new upgrade subsection says to
raise max_connections to the 14-per-pod size before upgrading, because
the rollout already runs new pods at that count, and warns that older
replicas keep serializing every mutation until the rollout finishes.
Monitor Capacity gains the lock metrics and their queries, and names
slow driver, credential, middleware, or profile source calls as another
cause of lock waits.
List the lock wait histogram and timeout counter, with their scope
label, on the Gateway Metrics page. The counter also counts background
work and leaves out lock connections that PostgreSQL does not open
although at least a second of the wait remained.
In the cluster debugging skill, replace the single advisory-lock
paragraph with the lock levels and pool, what each timeout detail
means, how a lock connection that PostgreSQL does not open shows up,
when terminating a lock backend is safe, a pg_locks query that shows
holders and waiters, and how to spot the global key. The per-pod greps
also read the timeout counter, and new rows cover lock timeouts and
lock connections that PostgreSQL does not open.
Part of #3528
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* feat(server): check mutation lock ordering in debug builds
The mutation lock ordering rules were prose only, so a nested guard on
a different key passed every test and could only deadlock under
contention. Debug builds now register each held guard with the Tokio
task (or the thread, outside a task) that acquired it, and panic before
waiting when a local key, a blocking lifecycle gate or the SSH identity
lock would break rules 1, 4 or 5. Startup endpoint-status
reconciliation and the tests that hold a guard while driving guarded
code run under lock_order::branch(), which marks deliberate concurrency
within one task. Release builds compile the check out.
Part of #3528
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* fix(server): do not hold gateway shutdown for the endpoint-status retry
When a replacement supervisor session failed its endpoint-status reset,
the spawned cleanup task kept the session lifetime guard through
retry_endpoint_status_after_supervisor_disconnect. That retry only ends
once the reset is durable, so a gateway shutdown during a database or
lock outage waited the full 10 s and reported ownership cleanup as
incomplete. The guard now covers only the lifecycle demotion, as in
finish_supervisor_session, and a SQLite test checks that shutdown
completes while the retry is still blocked.
Part of #3528
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* refactor(server): make the platform profile lock scope explicit
Platform-scope provider profile writers (import, update, delete) pass an
empty workspace on purpose and need X(global), because they validate
sandboxes in every workspace. MutationScope::profiles() now names that
escalation and picks Global for the platform scope, instead of relying
on MutationScope::Workspace("") to mean it. An empty workspace name now
trips a debug assertion, and release builds keep X(global) as a
fail-safe that over-locks rather than under-locks.
Part of #3528
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* perf(server): amortize local mutation lock table sweeps
LocalMutationLocks swept its whole table on every key lookup, so a
burst of N concurrent acquisitions on one replica cost O(N^2) under a
blocking mutex. The table is now swept only when a new key grows it
past the larger of 64 and twice the live entries left by the last
sweep, which makes acquisition amortized O(1). Released entries,
including those of cancelled waiters, stay bounded at about twice the
peak live set. The lifecycle gate registry keeps its per-call sweep.
Part of #3528
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* feat(server): make the database pool ceiling configurable
Both stores hardcoded their pool ceiling: 10 connections for Postgres, 5
for on-disk SQLite. One pool is shared by every database-backed RPC, so
that number is also the gateway's ceiling on concurrent database work.
Once every connection is checked out, callers queue on acquire and sqlx
logs "time to acquire exceeded slow threshold"; sandbox creates then time
out and are retried, which adds load rather than shedding it.
The right ceiling is deployment-specific — it depends on how many gateway
replicas share the database and what max_connections the server itself
allows — so it cannot be a single number baked into the binary.
Expose it on the same config surface as the rest of the gateway's
settings: --db-max-connections, OPENSHELL_DB_MAX_CONNECTIONS, and the
TOML key database_max_connections, resolved in that precedence order.
Helm deployments set the key through gatewayConfig. Omitting it keeps
each backend's previous value, so existing deployments do not move.
Values below 1 are rejected rather than silently replaced by the default:
a zero pool would block every acquire, and quietly ignoring a typo would
reproduce the ceiling the operator is trying to lift. An in-memory SQLite
database stays pinned to one connection, since the database lives inside
that connection.
Refs: #2561
Signed-off-by: Bryce Wilkinson <22760097+bjw123@users.noreply.github.com>
(cherry picked from commit
|
||
|
Matthew Grossman
|
90161fbd7d |
feat(kubernetes): add operator allowlists for sandbox pod annotations and labels (#4497)
* feat(kubernetes): add operator allowlist for sandbox pod annotations Add allowed_pod_annotations to the Kubernetes driver config. Sandbox template annotations must match an exact key or a <domain>/* entry, and the default empty list rejects all of them. Keys in the openshell.ai and agents.x-k8s.io domains are always rejected. Expose the setting through the server.allowedPodAnnotations Helm value. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * test(kubernetes): cover pod annotation rejection on sandbox create Assert the exact rejected annotation keys, exercise create_sandbox_inner with a disallowed annotation, and point the rejection error at the Helm gatewayConfig path. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * fix(kubernetes): drop Helm path from pod annotation rejection error The rejection error reaches API callers, not operators, so name only the gateway config key. Spell the Helm path in the runtimes docs with the quoted table key to avoid an invalid nested gatewayConfig rendering. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * test(kubernetes): condense pod annotation allowlist tests Fold the allowlist matching and validation cases into table-driven tests, reuse the json_struct helper for templates, and document why the openshell.ai and agents.x-k8s.io domains are reserved. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * feat(kubernetes): add operator allowlist for sandbox pod labels Sandbox template labels were copied onto workload pods unfiltered. Labels select pods for NetworkPolicy rules and Services, so a caller could place a sandbox under rules meant for other workloads or set OpenShell-owned labels. Add allowed_pod_labels, sharing the allowed_pod_annotations entry format, validation, and reserved openshell.ai / agents.x-k8s.io domains. Empty by default, which rejects every template label. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> --------- Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> |
||
|
Oliver Calder
|
16c8ca5c43 |
fix(docs): use lowercase "snap" as it is not a proper noun (#4387)
Signed-off-by: Oliver Calder <oliver.calder@canonical.com> |
||
|
Seth Jennings
|
4c1b16a4a1 |
feat(server): add operator-only provider credential retrieval (#4357)
* feat(server): add operator-only provider credential retrieval Authenticate operators exclusively through verified direct gateway mTLS. Export selected runtime credentials with freshness validation and all-or-nothing delivery. Coordinate refresh across callers and replicas, preserve rotated grants across RPC cancellation, and fence credential delivery against reconfiguration. Add generated bindings, security and regression tests, and operator documentation. Signed-off-by: Seth Jennings <sjenning@redhat.com> * feat(sdk): add curated operator credential retrieval Expose provider credential clients in Python and TypeScript with explicit workspace and key selection, freshness options, optional expiration, and preserved gateway errors. Reuse existing mTLS transports without introducing retries or secret caching. Add generated-wire and error-mapping tests, SDK package validation, and operator usage documentation. Signed-off-by: Seth Jennings <sjenning@redhat.com> * fix(server): honor recovery state for queued automatic refreshes Signed-off-by: Seth Jennings <sjenning@redhat.com> --------- Signed-off-by: Seth Jennings <sjenning@redhat.com> |
||
|
Oliver Calder
|
82bc756545 |
feat(snap): run gateway as a user service by default (#4099)
* feat(snap): run gateway as a user service by default Replace the existing system `gateway` service with separate `user-gateway` and `system-gateway` services. For new installs, only the `user-gateway` service is enabled. For existing installs, only the `system-gateway` service is enabled. Existing installs continue to have a one-time migration which removes legacy insecure configurations. It is up to `install.sh` or users to manually copy mTLS credentials from the root-owned `$SNAP_COMMON/tls` to the invoking user's OpenShell snap directory. This is explained in the snap description in `snapcraft.yaml`, visible in the Snap Store listing and via `snap info openshell`. Service enablement is now managed by a new snap configuration option named `gateway-mode`, so users can switch from one mode to another via e.g. `sudo snap set openshell gateway-mode=user`. The `install` and `post-refresh` hooks select which service to start by setting this mode. If the `gateway-mode` is already set, then we know the one-time migration has already taken place. Signed-off-by: Oliver Calder <oliver.calder@canonical.com> * fix(snap): simplify the snap gateway wrapper Signed-off-by: Oliver Calder <oliver.calder@canonical.com> * fix(snap): use `snap start --user` when running as non-root Signed-off-by: Oliver Calder <oliver.calder@canonical.com> * fix(snap): ensure install.sh waits for the correct docker daemon Signed-off-by: Oliver Calder <oliver.calder@canonical.com> * feat(snap): document how to safely switch between gateway modes New installs run the gateway as a user service by default, while upgrades of existing installs from prior to the user gateway continue to use the existing system gateway. The user and system gateway are fundamentally different services and there is no way to migrate state between them. However, it may be desirable for a user to run the gateway as a system service after a fresh install. To support this, this commit adds documentation for how to safely transition from the user gateway mode to the system gateway mode, and vice versa. Also, clarifies when it is necessary to manually copy TLS credentials from the system gateway state directory to the user's home directory: when upgrading a pre-TLS snap installation, or when switching from user to system gateway mode. When upgrading an existing install with an HTTPS gateway running as a system service, copying credentials again is not necessary. Signed-off-by: Oliver Calder <oliver.calder@canonical.com> --------- Signed-off-by: Oliver Calder <oliver.calder@canonical.com> |
||
|
Daneyon Hansen
|
40e01424c1 |
fix(helm): improve GatewayClass overrides (#4345)
* fix(helm): improve GatewayClass overrides Give chart-created Gateways a distinct default name and allow GRPCRoutes to select a listener on an existing Gateway. Fixes: #2469 Signed-off-by: Daneyon Hansen <daneyon.hansen@solo.io> * fix(e2e): resolve rendered Gateway name Signed-off-by: Daneyon Hansen <daneyon.hansen@solo.io> * fix(helm): preserve Gateway name defaults Signed-off-by: Daneyon Hansen <daneyon.hansen@solo.io> --------- Signed-off-by: Daneyon Hansen <daneyon.hansen@solo.io> |
||
|
Gaizka MenendezandTaylor Mutch
|
fe3942f2da |
feat(helm): migrate gateway configuration to gatewayConfig (#3384)
* feat(helm): add generic gateway TOML serializer Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * docs(architecture): define Helm gateway configuration boundary Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * feat(helm): define default gateway configuration map Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * refactor(helm): render gateway config from values map Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * test(helm): cover generic gateway TOML rendering Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * feat(helm): protect gateway config secret boundary Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * docs(helm): classify legacy gateway configuration values Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * refactor(helm): derive dual-use resources from gateway config Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * test(helm): migrate gateway config scenarios Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * refactor(helm): derive credential resources from gateway config Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * refactor(helm): migrate gateway config overlays Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * refactor(helm): remove gateway configuration shadow values Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * refactor(helm): align runtime config with resources Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * docs(helm): add gateway config migration guide Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * test(helm): validate rendered gateway config Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * test(helm): cover config resource coherence Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * fix(helm): make Kubernetes E2E deployable Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * fix(e2e): pass host aliases through gateway config Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * docs(e2e): reference gateway config host alias Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * fix(helm): enforce gateway resource ownership Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * test(e2e): cover chart host gateway input Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * fix(helm): address gateway config review findings Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * fix(helm): repair RFC 0012 migration integration Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * fix(e2e): align Kubernetes parity with RFC 0012 Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * fix(helm): complete gateway config migration Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * chore(helm): add SPDX header to TOML template * fix(helm): preserve legacy gateway configuration aliases Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * fix(helm): complete legacy gateway config compatibility Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * feat(helm): support gateway config arrays of tables * fix(helm): preserve sandbox identity aliases * fix(helm): address gateway config review findings Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> * fix(helm): preserve gateway config migration settings Signed-off-by: Taylor Mutch <taylormutch@gmail.com> * test(helm): validate extension arrays in CI values Signed-off-by: Taylor Mutch <taylormutch@gmail.com> * fix(helm): restore unconditional SPDX headers Signed-off-by: Taylor Mutch <taylormutch@gmail.com> * fix(helm): restore route SPDX header placement Signed-off-by: Taylor Mutch <taylormutch@gmail.com> * docs(helm): simplify configuration and migration examples Signed-off-by: Taylor Mutch <taylormutch@gmail.com> * docs(helm): remove redundant flag explanations Signed-off-by: Taylor Mutch <taylormutch@gmail.com> * docs(helm): move chart configuration guidance to Kubernetes setup Signed-off-by: Taylor Mutch <taylormutch@gmail.com> --------- Signed-off-by: Gaizka Menendez Hernandez <gmenende@redhat.com> Signed-off-by: Taylor Mutch <taylormutch@gmail.com> Co-authored-by: Taylor Mutch <taylormutch@gmail.com> |
||
|
Divesh
|
834b79a8c2 |
feat(server): place supervisor sessions by consistent hash (#3661)
* feat(server): place supervisor sessions by consistent hash and hand them off on shutdown Signed-off-by: divesh <dgude@nvidia.com> * feat: route long-lived sandbox connections to the owning gateway replica Signed-off-by: divesh <dgude@nvidia.com> * feat(helm): add opt-in per-replica GRPCRoute routing Signed-off-by: divesh <dgude@nvidia.com> --------- Signed-off-by: divesh <dgude@nvidia.com> |
||
|
alangou
|
33869a1df5 |
fix(images): update gateway base and allow supervisor base override (#4236)
Signed-off-by: Adrien Langou <alangou@nvidia.com> |
||
|
Eric Curtin
|
dfef088bc3 |
fix(docker): generate gateway JWT keys in compose quickstart (#3838)
* fix(docker): generate gateway JWT keys in compose quickstart Fixes #2891 Signed-off-by: Eric Curtin <eric.curtin@docker.com> * docs(docker): add compose mTLS steps Signed-off-by: Eric Curtin <eric.curtin@docker.com> --------- Signed-off-by: Eric Curtin <eric.curtin@docker.com> |
||
|
Emilien Macchianddivesh
|
8b3cc3fdc0 |
feat(server): add gateway capacity metrics and optional HPA (#3978)
* feat(server): add gateway capacity metrics and optional HPA Expose per-replica supervisor sessions, pending relay capacity, relay rejections and claim latency, and outbound peer request outcomes and latency. Use bounded labels and Prometheus histograms for the new latency metrics while preserving existing summary metrics. Add an optional Helm HPA with external-database and resource validation, conservative scale-down defaults, and support for custom metrics. Keep certificate hook pods outside gateway workload selectors. Document per-pod scraping, scaling limits, upgrade behavior, and PostgreSQL connection sizing. Part of #3528 Signed-off-by: Emilien Macchi <emacchi@redhat.com> * feat(server): unify routing metrics and clarify replica capacity Combine local relay setup and outbound peer requests in one counter, labeled by operation, route, target, outcome, and gRPC status. Preserve peer latency metrics. Rename the rejection reason from global_capacity to replica_capacity to reflect the per-replica relay budget. Keep capacity limits unchanged. Update tests and documentation. Signed-off-by: divesh <dgude@nvidia.com> * fix(server): align routed request metric labels and outcomes Count a local relay as successful only when its supervisor claims it, the same event that answers a peer relay on the owner, and rename the outcomes to local_error and remote_error so they say where an attempt failed. Label the peer latency histogram by operation, like the routed request counter, and rename the target label to relay_kind so it does not read as the Prometheus scrape target. Rename RelayCapacity.global to per_replica, drop the per-sandbox relay capacity gauge, which no per-sandbox series can pair with, describe the latency histogram as peer-only, and restore the note that unavailable spikes are expected during rollouts. Part of #3528 Signed-off-by: Emilien Macchi <emacchi@redhat.com> --------- Signed-off-by: Emilien Macchi <emacchi@redhat.com> Signed-off-by: divesh <dgude@nvidia.com> Co-authored-by: divesh <dgude@nvidia.com> |
||
|
Evan Lezar
|
e9271cb313 |
feat(gateway): support runtime image env overrides (#3504)
Closes #3502 Resolve trusted OCI runtime images with driver TOML, process environment, and compiled-default precedence across Docker, Podman, and Kubernetes. Signed-off-by: Evan Lezar <elezar@nvidia.com> |
||
|
Shiju
|
1c123a4542 |
feat(gateway): check VM host tools during config preflight (#4037)
* feat(gateway): validate VM filesystem tools during preflight Check required local VM tools through config preflight and share executable resolution with VM image operations. Report selected paths and actionable errors without creating gateway or sandbox state. Bound probe output and execution time, and clean up probe descendants on interruption. Preserve pure static validation and skip local tool checks for remote driver endpoints and unrelated drivers. Fixes #3951 Related to #3955 Signed-off-by: Shiju <shiju@nvidia.com> * fix(gateway): stabilize filesystem preflight checks Combine identical filesystem-tool error arms and normalize rendered diagnostics in command tests so terminal wrapping preserves assertions. Describe driver TLS validation without depending on removed guest fields. Signed-off-by: Shiju <shiju@nvidia.com> * test(gateway): serialize preflight fixture paths as TOML Keep temporary paths quoted and escaped through the TOML serializer instead of relying on Rust Debug formatting. Signed-off-by: Shiju <shiju@nvidia.com> --------- Signed-off-by: Shiju <shiju@nvidia.com> |
||
|
Eric Curtin
|
a48920ac04 |
feat(helm): add sandbox UID and GID values (#3947)
* feat(helm): add sandbox UID and GID values Closes #2697 Signed-off-by: Eric Curtin <eric.curtin@docker.com> * fix(helm): reject boolean sandbox UID and GID Signed-off-by: Eric Curtin <eric.curtin@docker.com> --------- Signed-off-by: Eric Curtin <eric.curtin@docker.com> |
||
|
Simon Scatton
|
e21b7fd8cf |
chore(build): remove bundled Z3 support (#3275)
* chore(build): remove bundled Z3 support Signed-off-by: Simon Scatton <sscatton@nvidia.com> Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> * fix(build): preserve vendored Z3 for local gateway artifacts Signed-off-by: Simon Scatton <sscatton@nvidia.com> --------- Signed-off-by: Simon Scatton <sscatton@nvidia.com> Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> |
||
|
Drew Newberry
|
021400be8a |
refactor(auth): separate sandbox identity from TLS (#3110)
* refactor(auth): separate sandbox identity from TLS Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs(auth): clarify gateway mTLS behavior Signed-off-by: Drew Newberry <anewberry@nvidia.com> * test(auth): include workspace scope in TLS authorization checks Signed-off-by: Drew Newberry <anewberry@nvidia.com> * test(e2e): bound service auth sandbox names for large PIDs Signed-off-by: Drew Newberry <anewberry@nvidia.com> --------- Signed-off-by: Drew Newberry <anewberry@nvidia.com> |
||
|
krishicks
|
33a8eac196 |
feat(helm): configure gateway OCSF JSONL output (#3876)
Previously, Helm installations could not enable the gateway OCSF JSONL destination through chart values because generated `gateway.toml` omitted the `openshell.gateway.ocsf_log` table. Now, setting `server.ocsfLog.enabled` renders the path, optional schema version, rotation, retention, and queue limits into gateway configuration. Output is disabled by default. The default path, `/tmp/gateway-ocsf.jsonl`, is writable in the gateway container with either the StatefulSet or Deployment workload, so enabling output does not require persistent storage. Invalid schema versions, rotation values, non-positive limits, or an empty path while enabled fail chart rendering. Additionally, `server.extraVolumes` and `server.extraVolumeMounts` add operator-supplied volumes to the gateway pod, so operators who want records to survive restarts can place the OCSF path on persistent storage without replacing chart-generated configuration. The gateway pod's default termination grace period rises from 5 to 30 seconds. Gateway shutdown can spend up to 10 seconds on supervisor session cleanup before allowing 5 seconds to drain queued OCSF records, so the 5-second default risked a SIGKILL before the final records were written. The grace period is only an upper bound: the gateway exits as soon as its shutdown completes. Refs #2762 Signed-off-by: Kris Hicks <khicks@nvidia.com> |
||
|
ansjindal
|
d009f30121 |
feat(helm): make cluster-scoped RBAC optional (#3459)
The gateway chart always rendered the ClusterRole and ClusterRoleBinding, so
every install and upgrade required cluster-admin even when only namespaced
objects were needed. Installers that are namespace-admin GitOps or platform
controllers could not run the release at all, and clusters where cluster-scoped
RBAC is owned by a separate team had no supported way to split the install.
Add an rbac values block so a cluster-admin can apply the cluster-scoped objects
once and a namespace-admin can install and upgrade the release without
cluster-scoped permissions:
rbac:
create: true
clusterScoped:
create: true
clusterRoleName: ""
clusterRoleBindingName: ""
rbac.clusterScoped.create gates the ClusterRole and ClusterRoleBinding, and is
independent of the workspace mode. rbac.create additionally gates the namespaced
sandbox Role and RoleBinding, which matters because Kubernetes escalation
prevention stops an installer holding only the built-in admin role from creating
a Role that grants agents.x-k8s.io verbs it does not itself hold. The certgen
hook and credential driver RBAC keep their existing flags.
Both flags default to true, so current installs are unchanged. The helpers treat
a missing rbac block as enabled so upgrades with --reuse-values do not drop RBAC,
matching the existing workspaceResources pattern. The ClusterRoleBinding roleRef
follows clusterRoleName so a separately applied ClusterRole can carry a name the
cluster-admin chooses.
Document the migration for a release that already owns the cluster-scoped
objects: Helm deletes objects that leave the manifest, so annotate them with
helm.sh/resource-policy=keep before setting the flag, otherwise the gateway
loses TokenReview until a cluster-admin re-applies them.
Signed-off-by: ansjindal <ansjindal@nvidia.com>
|
||
|
Drew Newberry
|
d4f5034d7f |
fix(gateway)!: make the WebSocket tunnel opt-in (#3727)
The /_ws_tunnel endpoint pipes a WebSocket into the full gRPC service. On a plaintext loopback gateway any web page could open it, since browsers do not apply CORS to WebSocket upgrades. Mount the tunnel only when enable_websocket_tunnel is set (config file, --enable-websocket-tunnel, or OPENSHELL_ENABLE_WEBSOCKET_TUNNEL; server.enableWebsocketTunnel in Helm). BREAKING CHANGE: gateways behind an authenticating edge proxy must set enable_websocket_tunnel = true for CLI edge-tunnel connections. Signed-off-by: Drew Newberry <anewberry@nvidia.com> |
||
|
Drew Newberry
|
9244868056 |
docs: refresh architecture and agent guides (#3705)
* docs: refresh architecture and agent guides Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs: describe updated security architecture neutrally Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs: highlight new isolation primitives Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs(sandboxes): clarify how to disconnect Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs: align architecture and guides with current navigation Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs(extensibility): streamline extension authentication guidance Signed-off-by: Drew Newberry <anewberry@nvidia.com> --------- Signed-off-by: Drew Newberry <anewberry@nvidia.com> |
||
|
Drew Newberry
|
1374672967 |
fix(helm): restore Kubernetes e2e chart rendering (#3692)
Signed-off-by: Drew Newberry <anewberry@nvidia.com> |
||
|
Drew Newberry
|
52cb8ecee7 |
fix(kubernetes): remove NetworkPolicy acknowledgement (#3677)
Signed-off-by: Drew Newberry <anewberry@nvidia.com> |
||
|
Jason T. GreeneandMrunal Patel
|
7691f88e0e |
perf(server): enable WAL for the SQLite store; relax sync only for SSH session issuance (#3543)
* perf(server): enable WAL and NORMAL sync for the SQLite store On-disk SQLite stores ran with sqlx defaults: rollback journal (`journal_mode=delete`) and `synchronous=FULL`. Every autocommit write paid several fsyncs and blocked readers while it held the lock, so gateway hot paths made of many small writes serialized on disk latency. The clearest case is `openshell forward service`, which mints and revokes an SSH session token around every forwarded TCP connection: two commits per connection, tens of milliseconds each on a virtual disk, wall clock linear in the number of concurrent connections, and enough queueing that bursts hit the per-sandbox connection cap and get refused. Switch on-disk databases to WAL with `synchronous=NORMAL`. The mode change runs once on a single connection before the pool opens: entering WAL needs exclusive access to the file, so doing it up front means pool connections only ever re-apply the pragma to a file already in WAL mode, and a failure surfaces as one clear connect error. The first start after upgrading an existing database therefore needs the file to be otherwise unopened. `synchronous` is applied through the connect options on every pooled connection. In-memory databases keep their defaults. A crash can now roll back the most recent transactions without corrupting the database, which is the standard WAL trade-off and fits the single-node scope of the SQLite backend. Tests cover a fresh store, an existing rollback-journal file that must be switched on connect, sidecar permissions, and concurrent readers under a burst of insert-then-update writes. Architecture, configuration and Helm docs describe the durability trade-off, the sidecar files, and the local filesystem requirement. Signed-off-by: Jason T. Greene <jason.greene@redhat.com> * fix(server): keep SQLite commits durable except SSH session issuance WAL with synchronous=NORMAL can roll back acknowledged commits after a power loss or kernel crash, including SSH session revocations and other authorization-tightening writes. Run the main pool with synchronous=FULL so every acknowledged write is durable; in WAL mode that is a single fsync of the WAL per commit. Add Store::create_relaxed for inserts that are safe to lose, and use it only for SSH session issuance: a dropped token just fails validation. On file-backed SQLite it runs on a dedicated single-connection pool with synchronous=NORMAL. Both pools share one WAL, so the next FULL commit also makes earlier relaxed commits durable. Postgres treats it as an ordinary durable MustCreate insert. Signed-off-by: Mrunal Patel <mrunalp@gmail.com> --------- Signed-off-by: Jason T. Greene <jason.greene@redhat.com> Signed-off-by: Mrunal Patel <mrunalp@gmail.com> Co-authored-by: Mrunal Patel <mrunalp@gmail.com> |
||
|
Mark CampbellandKris Hicks
|
679b190677 |
feat(testing): support independent gateway and supervisor image overrides (#3341)
* feat(testing): normalize configurable test images Signed-off-by: Bobbins228 <mcampbel@redhat.com> Signed-off-by: Kris Hicks <khicks@nvidia.com> * feat(helm): add global image overrides Signed-off-by: Bobbins228 <mcampbel@redhat.com> * feat(helm): support image registry overrides Signed-off-by: Bobbins228 <mcampbel@redhat.com> Signed-off-by: Kris Hicks <khicks@nvidia.com> * refactor(helm): simplify image configuration Signed-off-by: Bobbins228 <mcampbel@redhat.com> * fix(e2e): avoid reloading reused kind sandbox image Signed-off-by: Bobbins228 <mcampbel@redhat.com> * fix(helm): default sandbox image to nvcr.io/nvidia/base/ubuntu:24.04 Signed-off-by: Kris Hicks <khicks@nvidia.com> --------- Signed-off-by: Bobbins228 <mcampbel@redhat.com> Signed-off-by: Kris Hicks <khicks@nvidia.com> Co-authored-by: Kris Hicks <khicks@nvidia.com> |
||
|
krishicks
|
0b351c4a9b |
fix(helm)!: reduce gateway Secret privileges (#3616)
* fix(driver-kubernetes-secrets)!: store provider credentials in one namespace The Kubernetes Secrets credential driver now stores every credential in its configured namespace in all workspace modes and rejects handles that reference any other namespace before contacting the Kubernetes API. The gateway reaches credential Secrets through the Role in that namespace; this allows removing the Secret rules from the ClusterRole. - Remove the workspace_mode, gateway_id, and allow_reference_namespace driver settings and stop rendering them from Helm. Configurations that set them fail at startup. Existing credential state is not migrated. - Add server.credentialDrivers.kubernetesSecrets.createNamespace to provision a dedicated credential namespace. The namespace is kept on uninstall, adopted by a reinstall of the same release, and left untouched when something else owns it. - Update the gateway config reference, Kubernetes setup docs, 0.1.0 upgrade guide, compute-runtime architecture, and cluster debugging skill. Signed-off-by: Kris Hicks <khicks@nvidia.com> * fix(helm): reduce gateway Secret permissions Remove the gateway's Secret list permission in every workspace mode and grant source Secret reads through a Role in the sandbox namespace. Bootstrap Secret cleanup deletes Secrets by exact name instead of listing them. - Grant get on the copied client TLS and image-pull Secrets through a Role in the sandbox namespace. The ClusterRole keeps get and patch on those names for the ownership check and server-side apply into workspace namespaces. - Delete sandbox and supervisor bootstrap Secrets by exact name, derived from the runtime generation recorded on the Sandbox and, on restart, the target generation, tolerating 404. The generation annotation is cleared only after cleanup succeeds, and each bootstrap Secret has a Pod owner reference, so garbage collection removes any generation the driver does not name. - Drop Secret list from the ClusterRole and the shared-mode sandbox Role. - Extend the managed e2e RBAC checks to Secret list. - Update the Kubernetes setup and sandbox runtime docs, compute-runtime architecture, and cluster debugging skill. Signed-off-by: Kris Hicks <khicks@nvidia.com> * fix(driver-kubernetes): stage workspace Secrets per runtime generation Every Secret the Kubernetes driver writes into a workspace namespace is now scoped to one sandbox runtime generation, immutable, and created with create only, so the gateway never reads, patches, or adopts an existing Secret there. This removes the gateway's cluster-wide get and patch on the copied client TLS and image-pull Secret names. - In managed mode, create an immutable copy of each configured image-pull Secret per generation, named os-pull-<id>-<generation>-<n> and owned by the generation's workload and supervisor Pods. Pods and the restarted Sandbox template reference those names, and generation cleanup deletes them by name. A Secret already holding a generation name fails the create. - Outside shared mode, stage the gateway client TLS material into the supervisor bootstrap Secret instead of copying the client TLS Secret into the workspace namespace. - Remove the fixed-name TLS and image-pull copies, the target ownership read, and the ClusterRole get and patch rule on the copied names. Source reads stay in the sandbox-namespace Role. - Update the managed e2e to expect generation image-pull Secrets and client TLS material in the supervisor bootstrap Secret, and to check that the gateway cannot read the copied names in workspace namespaces. - Update the gateway config and compute driver references, Kubernetes setup and sandbox runtime docs, compute-runtime architecture, driver README, and cluster debugging skill. Signed-off-by: Kris Hicks <khicks@nvidia.com> * fix(helm)!: grant operator-mode Secret permissions through the workspace chart The operator-mode gateway ClusterRole grants no Secret permissions. The openshell-workspace chart Role, installed in each operator-managed namespace, grants the gateway create and delete on Secrets for sandbox runtime generations. Operator-managed namespaces require the workspace chart. - Fail the chart tests on any ClusterRole rule that includes Secrets in operator and shared modes. - Install the workspace chart when the operator e2e provisions a namespace, and assert that the gateway has no Secret permissions in a namespace without it. - Update the Kubernetes setup docs, 0.1.0 upgrade guide, compute-runtime architecture, and cluster debugging skill. Signed-off-by: Kris Hicks <khicks@nvidia.com> --------- Signed-off-by: Kris Hicks <khicks@nvidia.com> |
||
|
Mrunal Patel
|
a408f5dd08 |
chore(kubernetes): update Agent Sandbox to v1.0.3 (#3578)
Signed-off-by: Mrunal Patel <mrunalp@gmail.com> |
||
|
Drew Newberry
|
84960e70a3 |
fix(kubernetes): scope resource admission RBAC (#3571)
* fix(kubernetes): scope resource admission RBAC Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(helm): gate PVC admission reads Signed-off-by: Drew Newberry <anewberry@nvidia.com> --------- Signed-off-by: Drew Newberry <anewberry@nvidia.com> |
||
|
Drew Newberry
|
1e34e8c576 |
fix(drivers): require admission labels for external resources (#3538)
* fix(drivers): require admission labels for external resources Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(drivers): address resource admission review findings Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(core): reserve driver-owned admission labels Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(core): clarify workspace admission label Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(drivers): clarify resource admission failures Signed-off-by: Drew Newberry <anewberry@nvidia.com> * test(e2e): configure resource admission fixtures Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(kubernetes): retry forbidden admission lookups Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(e2e): preserve external driver admission defaults Signed-off-by: Drew Newberry <anewberry@nvidia.com> --------- Signed-off-by: Drew Newberry <anewberry@nvidia.com> |
||
|
Philippe Martin
|
35e0a68e4a |
feat(kubernetes): support corporate proxy CA bundle (#3447)
* feat(kubernetes): support corporate proxy CA bundle The Kubernetes driver had no way to supply a CA bundle for the corporate egress proxy, so an `https://` proxy with a private CA, or a TLS-intercepting proxy, could not be used. Podman and VM already expose `proxy_ca_bundle`. Add `proxy_ca_bundle` to `[openshell.drivers.kubernetes]` as a path the gateway Pod reads. The gateway stages the PEM into the existing per-generation supervisor bootstrap Secret and passes `--upstream-proxy-ca-bundle` on the supervisor argv. That Secret is already immutable, owner-referenced and garbage-collected, and its volume mounts every key at /.openshell/supervisor with no items filter, so this needs no new object kind, volume, mount, or RBAC verb, and works in shared, managed and operator workspace modes. The bundle is deliberately read from the gateway's filesystem rather than referenced as an object in the sandbox namespace. It becomes a trust anchor for every upstream the sandbox reaches, so it must stay in the gateway's trust domain; the immutable staging Secret also keeps the anchor from changing underneath a running sandbox. Bound the staged bundle at 256 KiB. The shared reader's limit is exactly the apiserver's own Secret limit and the bootstrap Secret carries four other keys, so a bundle between the two would pass gateway startup and then fail every sandbox create with an opaque `data: Too long`. Delegate the URL, no_proxy, connect_by_hostname and ca_bundle rules to the shared validate_upstream_proxy_settings, keeping the Secret-specific credential block local: this driver accepts an explicit `proxy_auth_allow_insecure = false` without credentials, which the shared rules reject. This also fixes the acknowledgement being demanded for an `https://` proxy, where the credential travels inside the verified TLS session. Add auth_setting_label so the inline-credential diagnostic names the Secret keys instead of proxy_auth_file, which this driver rejects as an unknown key. Document that the bundle should carry only the CA that signs the proxy's certificate, or that an intercepting proxy re-signs upstream certificates with. Public roots already reach the sandbox through the supervisor image and its TLS stack, and the bundle is concatenated with that system store into a single boundary control frame, so a full merged trust bundle spends the frame budget on duplicated roots. The frame, not the apiserver Secret limit, is the tighter of the two ceilings in practice; raising the staging bound requires checking it. Closes #3443 Signed-off-by: Philippe Martin <phmartin@redhat.com> * fix(helm): quote proxy CA ConfigMap references Signed-off-by: Philippe Martin <phmartin@redhat.com> --------- Signed-off-by: Philippe Martin <phmartin@redhat.com> |
||
|
|
50230616d5 |
refactor(runtime): retire Community image dependencies (#3386)
* feat(sandbox): default to official Alpine sandbox image default_sandbox_image() now returns docker.io/library/alpine:3.22, a generic version-qualified official image, so a fresh install no longer depends on the community sandbox image catalog. All compute drivers (docker, podman, kubernetes, vm) inherit this fallback. Part of #3116. Signed-off-by: Akram Signed-off-by: Akram <akram.benaissi@gmail.com> * feat(deploy): default deployment configs to the official Alpine sandbox image Update the shared gateway default_image, Helm chart values, the standalone Kubernetes manifest, and the dev gateway task scripts to use docker.io/library/alpine:3.22 instead of the community base image, consistent with default_sandbox_image(). GPU e2e image-build base is left unchanged (CUDA needs a glibc base). Part of #3116. Signed-off-by: Akram Signed-off-by: Akram <akram.benaissi@gmail.com> * feat(driver): default to numeric non-root identity for USER-less images With the default sandbox image now Alpine, images that declare no OCI USER must start instead of being rejected. When the image declares no USER and the policy requests none, the Podman and Docker drivers now supply a numeric non-root identity (DEFAULT_SANDBOX_UID/GID = 1000) instead of rejecting, matching the numeric-identity behavior of the Kubernetes and VM drivers. The supervisor's resolved-identity path runs the sandbox as a synthesized non-root account without the account existing in the image. Images that declare a USER keep the OCI resolution path unchanged. Part of #3116. Signed-off-by: Akram <akram.benaissi@gmail.com> Signed-off-by: Evan Lezar <elezar@nvidia.com> * test(conformance): use Alpine workload image Signed-off-by: Evan Lezar <elezar@nvidia.com> * refactor(policy): drop community image /app path from default policy The restrictive default policy granted read-only access to /app, a directory that only existed in the community base image. A generic Alpine default has no /app, so remove it. Landlock best-effort already ignores absent paths; this just stops advertising a community-specific layout in the default. Part of #3116. Signed-off-by: Akram Signed-off-by: Akram <akram.benaissi@gmail.com> * docs(config): document Alpine default images Signed-off-by: Evan Lezar <elezar@nvidia.com> * fix(podman): report early sandbox termination Signed-off-by: Evan Lezar <elezar@nvidia.com> * fix(podman): initialize rootless workspace ownership Signed-off-by: Evan Lezar <elezar@nvidia.com> * fix(sandbox): qualify NVIDIA Ubuntu default Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(podman): initialize rootful default workspace Signed-off-by: Drew Newberry <anewberry@nvidia.com> * feat(sftp): add native sandbox adapter Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(sftp): gate runtime helper support to Linux Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(sftp): support standard OpenSSH file operations Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(sftp): harden rename and special file handling Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(runtime): remove community image dependencies Signed-off-by: Drew Newberry <anewberry@nvidia.com> * test(e2e): build provider readiness tool fixture Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(e2e): use a dedicated Noble fixture for Docker tests Signed-off-by: Evan Lezar <elezar@nvidia.com> --------- Signed-off-by: Akram Signed-off-by: Akram <akram.benaissi@gmail.com> Signed-off-by: Evan Lezar <elezar@nvidia.com> Signed-off-by: Drew Newberry <anewberry@nvidia.com> Co-authored-by: Evan Lezar <elezar@nvidia.com> Co-authored-by: Drew Newberry <anewberry@nvidia.com> |
||
|
|
0a770d9173 |
feat(kubernetes): support HA gateway rebalancing (#1868)
* feat(kubernetes): support HA gateway rebalancing Signed-off-by: Drew Newberry <anewberry@nvidia.com> * perf(server): cache peer connections, tokens, and owner lookups Every forwarded relay rebuilt its setup from scratch: an owner lookup, a blocking read of the peer token, a TLS connect to the owning replica, and a TokenReview plus Pod GET on the receiving side. Sandbox service routing does this per HTTP request, so the apiserver calls scaled with traffic. Cache all of it on ServerState: - peer channels pooled per endpoint, so relays multiplex over one connection instead of redialing - peer tokens keyed by SHA-256, expiring at min(ttl, token exp) so a hit cannot accept an expired token - owner records for 3s against a 45s ownership TTL, still freshness checked before use Entries are evicted when a relay fails. Also raise HTTP/2 max_concurrent_streams to 1024, since pooling funnels every relay between two replicas onto one connection and hyper's default of 200 sits below the 256 pending-relay budget. Signed-off-by: divesh <dgude@nvidia.com> * perf(server): pool upstream connections for sandbox services Each HTTP request to a sandbox service opened its own supervisor relay, paying a new TCP connection and HTTP/1 handshake every time. Worse, it counted against the 32 in-flight relay cap, so a service handling more than 32 concurrent requests failed outright. Pool idle upstreams per endpoint and port, up to 8 each for 15s. Reuse is safe because the pool only returns a connection hyper reports as ready, and HTTP/1 cannot start a request until the previous body has drained. Upgrades are never pooled since they take the connection over, and a failed send evicts that endpoint. Pruning is bounded per key, with the full sweep limited to once per 30s. Signed-off-by: divesh <dgude@nvidia.com> * fix(server): address HA gateway review findings (#3449) - Let a gateway own supervisor sessions without a peer endpoint. Requiring one whenever the store is PostgreSQL broke every single-instance PostgreSQL deployment, because no sandbox supervisor could connect. A cross-replica request to an owner that advertises no endpoint now fails immediately naming the cause, instead of retrying until the wait timeout. - Close a supervisor session on heartbeat only when another replica owns it, or after renewals fail for the ownership TTL. A database error no longer drops every session heartbeating during an outage. - Clamp owner record ages at zero so a skewed or corrupt stored timestamp cannot produce a negative age. - Bound the cross-object advisory lock with a lock timeout, so a stuck holder fails instead of blocking every mutation in the fleet. - Refuse to start when a peer endpoint is configured on a multi-replica backend but peer authentication is unavailable, and warn when a multi-replica backend has no peer endpoint at all. - Reject a plaintext peer endpoint when the gateway serves TLS. - Skip the sandbox watch poller on single-replica backends, where the local update bus already sees every write. - Rate-limit the peer owner cache sweep so an insert no longer scans the whole map under the lock. - Retry GET and HEAD on a pooled upstream the sandbox closed, instead of returning 502, and drop an emptied endpoint from the pool right away. - Document the gateway peer environment variables and the post-rollout ownership skew operators should expect. Signed-off-by: divesh <dgude@nvidia.com> * fix(server): harden HA supervisor ownership Signed-off-by: Drew Newberry <anewberry@nvidia.com> --------- Signed-off-by: Drew Newberry <anewberry@nvidia.com> Signed-off-by: divesh <dgude@nvidia.com> Co-authored-by: Drew Newberry <anewberry@nvidia.com> Co-authored-by: divesh <dgude@nvidia.com> Co-authored-by: Divesh Chowdary <47188680+FrostGod@users.noreply.github.com> |
||
|
Drew Newberry
|
d6f3e1f5f2 |
fix(packaging): keep SPDX comments out of Debian control (#3483)
Signed-off-by: Drew Newberry <anewberry@nvidia.com> |
||
|
Drew Newberry
|
17ce738bfb |
fix(ci)!: remove gateway callback listener dependency (#3365)
* fix(ci): repair post-merge release canary Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(packaging): bootstrap canary runtime prerequisites Signed-off-by: Drew Newberry <anewberry@nvidia.com> * ci(canary): collect macOS VM diagnostics Signed-off-by: Drew Newberry <anewberry@nvidia.com> * ci(canary): pin libkrun-compatible macOS runner Signed-off-by: Drew Newberry <anewberry@nvidia.com> * ci(canary): limit macOS smoke test to package startup Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(compute)!: remove gateway callback listeners Run Docker supervisors on host networking so they use the operator-configured primary gateway endpoint. Remove the unused compute-driver callback listener negotiation and listener-scoped routing machinery. BREAKING CHANGE: The ComputeDriver API no longer exposes GetGatewayListenerRequirements or GatewayListenerRequirement. External drivers must regenerate bindings and connect supervisors to the configured primary gateway endpoint. Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(docker): use sandbox runtime image in launcher Signed-off-by: Drew Newberry <anewberry@nvidia.com> * test(podman): exercise production endpoint selection Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(e2e): route supervisors to reachable gateways Signed-off-by: Drew Newberry <anewberry@nvidia.com> * test(e2e): align Podman endpoint fixtures Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(docker): preserve host aliases for supervisors Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(docker): align sandbox host gateway pin Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(e2e): address Docker fixtures by bridge IP Signed-off-by: Drew Newberry <anewberry@nvidia.com> * test(e2e): serialize sandbox lifecycle cases Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(e2e): host Docker TCP fixture with gateway Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(e2e): use loopback for host-network supervisor Signed-off-by: Drew Newberry <anewberry@nvidia.com> --------- Signed-off-by: Drew Newberry <anewberry@nvidia.com> |
||
|
John T. Myers
|
903d9a0e7a |
chore(license): align repository compliance text (#3467)
Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> |
||
|
Drew NewberryandPiotr Mlocek
|
8b77925eb7 |
feat(installer): support prerelease installations (#3364)
* feat(installer): support prerelease installations Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs(readme): clarify 0.1.0 production readiness Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs(readme): refine production readiness message Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(release): publish rolling prerelease channels Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(release): keep prereleases out of GitHub releases Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs(readme): refine 0.1.0 notice Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs(readme): expand 0.1.0 notice Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs(readme): simplify 0.1.0 notice Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(installer): simplify prerelease alias Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(installer): require successful prerelease runs Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(release): reuse exact prerelease artifacts Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(installer): include prover in prerelease bundle Signed-off-by: Drew Newberry <anewberry@nvidia.com> * ci: remove prerelease post-publish validation Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs(fern): sync announcement configuration Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> * docs: remove stale prerelease canary claim Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs(fern): defer announcement synchronization Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs(fern): simplify release announcement Signed-off-by: Drew Newberry <anewberry@nvidia.com> --------- Signed-off-by: Drew Newberry <anewberry@nvidia.com> Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> Co-authored-by: Piotr Mlocek <pmlocek@nvidia.com> |
||
|
Drew Newberry
|
07d4ac5474 |
fix(helm): grant secret cleanup permissions (#3363)
Signed-off-by: Drew Newberry <anewberry@nvidia.com> |
||
|
Johnny Greco
|
58b5f8f976 |
feat(prover): add standalone policy boundary checker (#3289)
* feat(prover): add standalone policy maximum checker Signed-off-by: Johnny Greco <jogreco@nvidia.com> * refactor(prover): simplify check scope schema Signed-off-by: Johnny Greco <jogreco@nvidia.com> * fix(prover): align containment and cancellation with runtime Signed-off-by: Johnny Greco <jogreco@nvidia.com> * fix(prover): stabilize containment checks in CI Signed-off-by: Johnny Greco <jogreco@nvidia.com> * test(prover): avoid solver in fast-path guard test Signed-off-by: Johnny Greco <jogreco@nvidia.com> * fix(prover): align string containment with runtime Signed-off-by: Johnny Greco <jogreco@nvidia.com> * fix(prover): reject ambiguous z3 string escapes Signed-off-by: Johnny Greco <jogreco@nvidia.com> * fix(prover): align containment with runtime boundaries Signed-off-by: Johnny Greco <jogreco@nvidia.com> * fix(prover-cli): harden cancellation and invalid input Signed-off-by: Johnny Greco <jogreco@nvidia.com> * feat(packaging): install policy prover with OpenShell Signed-off-by: Johnny Greco <jogreco@nvidia.com> * docs(prover): clarify installation and check results Signed-off-by: Johnny Greco <jogreco@nvidia.com> * docs(build): describe prover distribution directly Signed-off-by: Johnny Greco <jogreco@nvidia.com> * refactor(prover): rename maximum policy to boundary Signed-off-by: Johnny Greco <jogreco@nvidia.com> * refactor(prover): localize fail-closed validation Signed-off-by: Johnny Greco <jogreco@nvidia.com> * test(prover): cover fail-closed CLI surfaces Signed-off-by: Johnny Greco <jogreco@nvidia.com> * test(prover): allow CI load for REST solver proof Signed-off-by: Johnny Greco <jogreco@nvidia.com> * fix(prover): use canonical policy schema for containment Signed-off-by: Johnny Greco <jogreco@nvidia.com> * fix(prover): preserve uncertainty for runtime binary globs Signed-off-by: Johnny Greco <jogreco@nvidia.com> * fix(prover): bound policy validation work Signed-off-by: Johnny Greco <jogreco@nvidia.com> * docs(prover): document validation resource limits Signed-off-by: Johnny Greco <jogreco@nvidia.com> * fix(prover): make containment API extensible Signed-off-by: Johnny Greco <jogreco@nvidia.com> * docs(prover): define containment API contract Signed-off-by: Johnny Greco <jogreco@nvidia.com> * fix(ci): integrate prover with consolidated builds Signed-off-by: Johnny Greco <jogreco@nvidia.com> * fix(ci): declare release packaging dependency Signed-off-by: Johnny Greco <jogreco@nvidia.com> --------- Signed-off-by: Johnny Greco <jogreco@nvidia.com> |
||
|
John T. Myers
|
0fd3385726 |
fix(container): use distroless Debian 13 for supervisor (#3393)
* fix(container): use distroless Debian 13 for supervisor Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * fix(docker): preserve supervisor bootstrap ownership Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> --------- Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> |
||
|
krishicks
|
12a7a35910 |
chore(tools): upgrade mise to 2026.9.9 (#3385)
Signed-off-by: Kris Hicks <khicks@nvidia.com> |