mirror of
https://github.com/NVIDIA/OpenShell.git
synced 2026-10-29 07:23:25 +08:00
2e1e2c50a61375d7bd3bed912c87207bc6b3bd99
531
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Drew Newberry
|
e5adf6f484 |
feat(supervisor): recover shared instances with hash-based HA (#4864)
* feat(supervisor): recover shared instances with hash-based HA Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(supervisor): keep HA coordination out of compute drivers Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(supervisor): fence pool admission and reconciliation races Signed-off-by: Drew Newberry <anewberry@nvidia.com> * feat(supervisor): keep pool mode stable across replica scaling Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(supervisor): default shared deployments to pool mode Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(supervisor): fence refused launches and remove legacy mode Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(supervisor): confirm live owners and fence stale observations Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(supervisor): preserve takeover cause log formatting Signed-off-by: Drew Newberry <anewberry@nvidia.com> * feat(helm): add memory-based supervisor autoscaling Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(supervisor): retain acknowledgements and fence streamed input Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs(supervisor): consolidate experimental multi-tenant guidance Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs(supervisor): simplify multi-tenant supervisor guide Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(supervisor): simplify pool member identity Signed-off-by: Drew Newberry <anewberry@nvidia.com> * feat(helm): make supervisor affinity configurable Signed-off-by: Drew Newberry <anewberry@nvidia.com> * feat(helm): make supervisor disruption budget optional Signed-off-by: Drew Newberry <anewberry@nvidia.com> * test(supervisor): align HA fixtures with suspend support Signed-off-by: Drew Newberry <anewberry@nvidia.com> --------- Signed-off-by: Drew Newberry <anewberry@nvidia.com> |
||
|
Taylor Mutch
|
7b0f8ae07c |
fix(gateway): require authentication or explicit development opt-in (#4821)
* fix(gateway): require authentication or explicit development opt-in Signed-off-by: Taylor Mutch <taylormutch@gmail.com> * fix(gateway): reject implicit unauthenticated admin requests Signed-off-by: Taylor Mutch <taylormutch@gmail.com> --------- Signed-off-by: Taylor Mutch <taylormutch@gmail.com> |
||
|
Shailendra SinghandDrew Newberry
|
17eda52d45 |
docs(windows): route main documentation to the Windows release branch (#4758)
* docs(windows): route main documentation to the Windows release branch Signed-off-by: Shailendra Singh <shailendras@nvidia.com> * docs(windows): include native MXC tests in full validation Signed-off-by: Shailendra Singh <shailendras@nvidia.com> * docs(windows): address documentation review feedback Signed-off-by: Shailendra Singh <shailendras@nvidia.com> * docs(windows): include E2E in full validation Signed-off-by: Shailendra Singh <shailendras@nvidia.com> * docs(windows): stop validation after any failed task Signed-off-by: Shailendra Singh <shailendras@nvidia.com> * docs(windows): simplify quickstart and installation guidance Signed-off-by: Drew Newberry <anewberry@nvidia.com> --------- Signed-off-by: Shailendra Singh <shailendras@nvidia.com> Signed-off-by: Drew Newberry <anewberry@nvidia.com> Co-authored-by: Drew Newberry <anewberry@nvidia.com> |
||
|
Free Wortley
|
f5d9223e08 |
feat(supervisor): let operators require specific middleware bindings (#4678)
* feat(middleware): authorize supervisor bindings at registration Let operators require and limit the operation and phase pairs that an external middleware may expose. Validate the same binding policy at gateway startup and when a supervisor discovers the service, and keep only authorized bindings in the registry used for dispatch. Preserve dynamic discovery and the existing wire configuration when no binding policy is set. Document that older supervisors must be upgraded before relying on binding restrictions. Signed-off-by: Free Wortley <freeqaz@openai.com> * docs(skills): check which supervisor middleware hooks the operator authorizes Signed-off-by: Free Wortley <freeqaz@openai.com> * fix(go): regenerate the sandbox protobuf binding Restore the comment separator emitted by the pinned generator. This keeps the checked-in binding byte-for-byte reproducible without changing the protobuf schema or Go behavior. Signed-off-by: Free Wortley <freeqaz@openai.com> --------- Signed-off-by: Free Wortley <freeqaz@openai.com> |
||
|
Brooke Storm
|
c4688aa871 |
feat(helm): add certificate hook pod labels and annotations (#4895)
Signed-off-by: Brooke Storm <brookes@nvidia.com> |
||
|
Eric Curtin
|
16250df210 |
feat(helm): add server.extraEnv for the gateway container (#4876)
Signed-off-by: Eric Curtin <eric.curtin@docker.com> |
||
|
Oliver Calder
|
72e4eaa89c |
feat(snap): publish prerelease builds to latest/beta channel (#4870)
* feat(snap): publish prerelease builds to latest/beta channel Signed-off-by: Oliver Calder <oliver.calder@canonical.com> * fixup! feat(snap): publish prerelease builds to latest/beta channel Signed-off-by: Oliver Calder <oliver.calder@canonical.com> * fix(snap): use concurrency group when checking version to publish snap Signed-off-by: Oliver Calder <oliver.calder@canonical.com> --------- Signed-off-by: Oliver Calder <oliver.calder@canonical.com> |
||
|
Emilien Macchi
|
07b7a74b62 |
fix(server): bound the provisioning-deadline scan's lock wait (#4549)
The provisioning-deadline scan takes each provisioning sandbox's local S(global) S(workspace) X(sandbox) keys in ascending key order and waited for them with no deadline. When a sandbox's key sorts before its workspace's key, the scan held X(sandbox) while it waited behind a provider or profile writer, so start, stop and delete for that sandbox waited for the writer too. In the other order, the scan held S(workspace) while it waited for a busy sandbox, and provider writers on that workspace queued behind it. Either way the rest of the pass waited as well. Each pass now has a 500 ms budget for waiting on these keys. A sandbox whose keys stay busy is skipped and picked up on the next pass a second later. Once the budget is spent, the rest of the pass only takes keys that are free when it reaches them, so a busy workspace or global key costs one wait per pass rather than one per sandbox. The local acquisition is cancel-safe, so giving up releases the keys it already took. Skips log at debug and count in the new openshell_server_provisioning_deadline_skips_total counter, labeled by whether the sandbox was already past its deadline. A steady rate of delayed expiries can then be alerted on without a warning every pass. The gateway metrics page and the cluster debugging skill describe it. The test that pinned the old wait is replaced by tests for both key orders: a lifecycle lock or a provider writer proceeds and the scan finishes while the other side holds its key, the skipped sandbox is untouched and expires once the writer releases, and a skip doesn't end the pass for other sandboxes. Another test checks that sandboxes behind one busy workspace share a single wait. Part of #4307 Closes #4547 Signed-off-by: Emilien Macchi <emacchi@redhat.com> |
||
|
Taylor Mutch
|
132c582349 |
docs(extensibility): document external compute driver setup (#4826)
Signed-off-by: Taylor Mutch <taylormutch@gmail.com> |
||
|
Oliver Calder
|
cb42f5d9e8 |
fix(snap): use openshell-prover instead of openshell.prover (#4835)
The alias `openshell-prover` has been granted for OpenShell: https://forum.snapcraft.io/t/alias-request-for-openshell/53658 Thus, we can remove guidance differentiating `openshell.prover` for the snap from `openshell-prover` for other packaging formats. It is still worth noting that installing a locally-built snap directly, rather than from the store, means there is no alias. Signed-off-by: Oliver Calder <oliver.calder@canonical.com> |
||
|
Adel Zaalouk
|
229382e51d |
docs(sandbox): align workload image requirements with current runtime (#3967)
Workload images no longer need iproute2 or nftables since the supervisor moved out of the workload (#2942). Docker and Podman run images without USER as UID/GID 1000 (#3386), the driver creates /sandbox, and the CLI no longer builds Dockerfiles (#3214). Update the bring-your-own-container example and the runtimes page to match. Fixes #3965 Signed-off-by: Adel Zaalouk <azaalouk@redhat.com> |
||
|
krishicks
|
321b93a898 |
feat(auth): emit OCSF events for SSH session and sandbox token checks (#4762)
Some credentials are checked by handlers rather than by the gateway's authenticators, so their rejections only reached the caller and plain logs. They now emit the same OCSF events as the router's checks: - SSH session tokens. A ForwardTcp SSH connection whose token is missing, unknown, revoked, minted for another sandbox, or expired emits an Authentication failure with protocol ssh_session_token and a reason category. An accepted token logs on once, like other credentials. The router has already authenticated the caller, so these events name that caller. - IssueSandboxToken. A compute-driver identity that no longer matches the sandbox's recorded runtime emits an Authentication failure (stale_identity); a sandbox with no record, or no usable recorded identity, emits unknown_identity. A sandbox calling it with a gateway JWT is refused with method_not_permitted and raises the sandbox boundary finding. - RefreshSandboxToken. A caller that is not a gateway-JWT sandbox is refused with method_not_permitted. A request for an extension service the sandbox policy does not select is refused with the new extension_service_not_selected reason, without a finding, because a policy update can race the supervisor's request. - Sandbox logons on IssueSandboxToken and RefreshSandboxToken. The router logged a compute-driver credential on IssueSandboxToken, or a gateway JWT on RefreshSandboxToken, before the handler finished checking the recorded runtime or the token lineage, so a credential that failed that check recorded a success and then a failure. Those handlers now record the logon once their check passes. Other credentials on these methods are fully verified by the router and still log on there, even when authorization then refuses them. - GetCurrentUser. A non-user principal is refused with method_not_permitted. The router already blocks such callers, so this is defense in depth. Callers receive the same status codes and messages as before, and the events never include the tokens. Refs #2912 Signed-off-by: Kris Hicks <khicks@nvidia.com> |
||
|
Max Dubrinsky
|
a2b8022e07 |
fix(python): ship credential driver protocol bindings (#4760)
Closes #4759 Signed-off-by: Max Dubrinsky <mdubrinsky@nvidia.com> |
||
|
krishicks
|
915a5d1553 |
feat(auth): emit OCSF logon events for accepted gateway credentials (#4761)
When a gateway replica accepts a credential for the first time, it now emits an Informational OCSF Authentication event with status Success. The caller's request proceeds exactly as before. The event records: - Who: the principal (user, mTLS operator, sandbox, or gateway peer), with the same user types as authorization denials. - How: the authentication protocol (OpenID, mtls, sandbox_jwt, compute_driver, or k8s_service_account). - Where: the gRPC method of the request that logged on. - Correlation: the caller's address and request ID, the same fields the failure and denial events carry. Every gRPC request authenticates, and an idle sandbox's supervisor polls the gateway several times a minute, so an event per request would bury user logons. Each credential therefore logs on once: later requests with the same bearer token or client certificate emit nothing until the credential has been in use for an hour, when it logs on again. The replica's connections share this record, which holds up to 16,384 credentials and forgets the oldest early when a new one arrives while full. It keys on a SHA-256 digest of the credential that authenticated the caller (the bearer token or the client certificate's fingerprint), so headers the authenticator ignored cannot force new logons, and a renewed certificate for the same subject logs on again. Digests stay in memory. Requests admitted by allow_unauthenticated_users, or by a gateway with no authentication configured, emit no success event, since no credential was verified. Refs #2912 Signed-off-by: Kris Hicks <khicks@nvidia.com> |
||
|
Eric Curtin
|
ea265a2f9a |
docs(cli): say --upload cannot take a command (#4502)
Signed-off-by: Eric Curtin <eric.curtin@docker.com> |
||
|
krishicks
|
a24340beba |
feat(auth): emit OCSF events for gateway authorization denials (#4728)
When the gateway refuses a caller that has already authenticated, it now emits an OCSF API Activity event. The caller gets exactly the same error as before. The event records: - Who: the principal (user, mTLS operator, sandbox, or gateway peer). - What: the gRPC method, plus the sandbox or workspace that was refused when there is one. - Why: a reason such as cross_sandbox, missing_role or not_workspace_member. - Correlation: the caller's address and request ID, the same fields the authentication events carry. That covers the denials made centrally by the router and the shared guards, plus the checks inside handlers that have no shared guard: a sandbox naming another sandbox (in either sandbox-name resolver), a non-user caller refused by mutation admission, a sandbox sending a config update it isn't allowed to, and a non-admin granting the workspace admin role. Denials raised in the mutation admission worker, which runs in a spawned task, keep the request's context. When a sandbox credential is refused for calling a method it may not call, touching another sandbox, or acting across workspaces, the gateway also emits a High-severity Detection Finding. It carries the same correlation_uid as the denial event. A real supervisor never makes these requests, so a finding means the credential is probably being misused. Two cases are recorded without a finding: a method the gateway doesn't recognize (unknown_method), so a newer supervisor doesn't look like a compromised one, and a supervisor whose own sandbox was deleted mid-request, which is refused but isn't misuse. The shorthand line names the refused sandbox or workspace, for example sandbox:<id>, so operators reading console logs can see the target without JSONL or OTLP export. To support this, the OCSF crate's Actor can now hold a user instead of a process, and the API Activity builder gained the fields gateway events need. Supervisor inference events come out unchanged. Gateway events declare the container profile only when they concern a sandbox, as MXC findings do. The OCSF 1.8.0 resource_details schema is vendored so the refused sandbox or workspace goes in the standard resources field. Refs #2912 Signed-off-by: Kris Hicks <khicks@nvidia.com> |
||
|
ShijuandJohn Myers
|
292c9c8d48 |
fix(vm): flush guest filesystem before a planned sandbox stop (#4195)
* fix(vm): flush guest filesystem before a planned sandbox stop A planned stop of a MicroVM sandbox ended the VM before the guest wrote its dirty pages to disk, so files written shortly before the stop were missing after the next start. The driver closed the host supervisor's parent-liveness pipe before it sent SIGTERM. The supervisor exits at once on that end-of-file, so it never ran its orderly shutdown: terminate the workload, then ask the guest to terminate. Signal the supervisor first, keep the pipe open until it has exited, and give it a grace period derived from the supervisor's own bounds and kept below the gateway's reclaim deadline. In the guest runtime, syncfs the root filesystem after the process tree is terminated and before the termination is acknowledged, bounded to 3 seconds and best effort. Only the VM backend, which uses a vsock boundary listener, flushes; the container drivers share this runtime and are unchanged. A sandbox whose main process has already exited is not covered. Closes #4185 Signed-off-by: Shiju <shiju@nvidia.com> * fix(vm): warn about unconfirmed flush after supervisor kill Preserve the supervisor exit status so a planned stop warns when a signal killed the supervisor and guest writeback was not confirmed. Keep normal workload exit codes quiet, including 143 after a handled SIGTERM. Extend the existing process tests to check the warning after a forced kill and its absence after an orderly shutdown. Refs #4185 Signed-off-by: Shiju <shiju@nvidia.com> * fix(vm): satisfy shutdown timeout control-flow lint Use if let for the single successful timeout pattern while preserving the supervisor exit status and forced-kill error handling. Signed-off-by: Shiju <shiju@nvidia.com> * fix(vm): preserve shutdown merge resolutions during rebase Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * docs(vm): explain planned stop durability and flush limits Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> --------- Signed-off-by: Shiju <shiju@nvidia.com> Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> Co-authored-by: John Myers <9696606+johntmyers@users.noreply.github.com> |
||
|
Oliver Calder
|
d4ad1bbd03 |
fix(snap): only enable gateway user service for one user (#4615)
* fix(snap): only enable gateway user service for one user Signed-off-by: Oliver Calder <oliver.calder@canonical.com> * docs(snap): correct snap docs for the gateway user service Signed-off-by: Oliver Calder <oliver.calder@canonical.com> * fix(snap): add documentation of global to per-user enablement migration Signed-off-by: Oliver Calder <oliver.calder@canonical.com> --------- Signed-off-by: Oliver Calder <oliver.calder@canonical.com> |
||
|
Eric Curtin
|
2798108b3f |
fix(ocsf): report https scheme for TLS-terminated requests (#4684)
* fix(ocsf): report https scheme for TLS-terminated requests Closes #4562 Signed-off-by: Eric Curtin <eric.curtin@docker.com> * fix(ocsf): use transport scheme on passthrough relay Signed-off-by: Eric Curtin <eric.curtin@docker.com> --------- Signed-off-by: Eric Curtin <eric.curtin@docker.com> |
||
|
Simon Scatton
|
b1cfc91720 |
ci(release): promote qualified prereleases to stable (#4611)
* ci(release): promote qualified prereleases to stable Signed-off-by: Simon Scatton <sscatton@nvidia.com> * fix(release): preserve qualified images during stable publication Signed-off-by: Simon Scatton <sscatton@nvidia.com> * ci(release): run shared release jobs with Nix Signed-off-by: Simon Scatton <sscatton@nvidia.com> * fix(release): recheck latest before publishing shared channels Signed-off-by: Simon Scatton <sscatton@nvidia.com> --------- Signed-off-by: Simon Scatton <sscatton@nvidia.com> |
||
|
krishicks
|
fdfbf84f8b |
fix(ocsf): report the required device.os.type_id (#4677)
OCSF 1.8 requires type_id on the os object, but device.os carried only a name. Events now report Linux (200), Windows (100), macOS (300), or Other (99) with the platform name as the type label. The os object schema is vendored so tests validate it. Refs #2912 Signed-off-by: Kris Hicks <khicks@nvidia.com> |
||
|
krishicks
|
f41f24b549 |
feat(auth): record peer address and request ID on authentication events (#4673)
Authentication events now carry the connection's immediate peer as src_endpoint, or edge-tunnel for WebSocket-tunneled requests, and the request's x-request-id as metadata.correlation_uid when it is a UUID. Refs #2912 Signed-off-by: Kris Hicks <khicks@nvidia.com> |
||
|
Piotr Mlocekanddivesh
|
604b9f66f6 |
feat(config): deliver gateway configuration over the supervisor session (#4320)
Supervisors poll the gateway for configuration every 10 seconds. This PR adds an opt-in push mode (config_delivery_mode = "push"): supervisors still start by polling, and once their ConnectSupervisor session is accepted, the gateway delivers every later change over it and supervisors apply it and report the result. poll stays the default, and older supervisors and gateways keep polling. --------- Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> Signed-off-by: divesh <dgude@nvidia.com> Co-authored-by: divesh <dgude@nvidia.com> |
||
|
krishicks
|
35a3390dbc |
feat(auth): classify rejected gateway credentials in authentication events (#4655)
Authenticators now return the mechanism and a reason category with the status they reject a request with, and the gateway router records both in the Authentication event. Callers still receive the original status. Refs #2912 Signed-off-by: Kris Hicks <khicks@nvidia.com> |
||
|
Drew Newberry
|
77929d9c14 |
feat(sandbox): publish and activate sandbox backend protocol (#3898)
* feat(isolation): publish backend protocol contracts independently of runtime Signed-off-by: Drew Newberry <anewberry@nvidia.com> * feat(isolation): expose trusted launch and registry data contracts Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(isolation): name the required capability sandbox Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(sandbox): align protocol and compute capability names Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(sandbox): finalize public protocol contract names Signed-off-by: Drew Newberry <anewberry@nvidia.com> * feat(isolation)!: activate backend contracts in sandbox runtime and driver selection Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(sandbox): simplify runtime and fix backend example Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(examples): organize WASI backend in Rust Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(sandbox): defer protocol conformance fixtures Signed-off-by: Drew Newberry <anewberry@nvidia.com> * feat(examples): add supervised V8 backend demo Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(examples): stream V8 guest output with default policy Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(examples): handle V8 rejections and roll back failed launches Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(examples): align V8 crypto dependencies with CI AWS-LC Signed-off-by: Drew Newberry <anewberry@nvidia.com> * feat(sandbox): represent Sandbox Protocol operations in protobuf Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(isolation): replace backend traits with RPC clients Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(proto): remove legacy isolation protocol reservations Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(proto): clarify Sandbox Protocol names Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs(isolation): clarify authentication bootstrap and supervisor APIs Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(isolation): simplify policy enforcement confirmation Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(sandbox): rotate launch generations and stabilize retry sessions Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(sandbox): remove redundant backend adapters and TCP flags Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(sandbox): repair Docker restarts and macOS lint Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(isolation): simplify RPC contracts and backend registration Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(sandbox): deliver TCP denials before closing the RPC Signed-off-by: Drew Newberry <anewberry@nvidia.com> --------- Signed-off-by: Drew Newberry <anewberry@nvidia.com> |
||
|
Eric CurtinandJohn Myers
|
8fcba288cf |
feat(cli): add --annotation to sandbox create (#4325)
* feat(cli): add --annotation to sandbox create Signed-off-by: Eric Curtin <eric.curtin@docker.com> * fix(cli): drop reserved annotation check Signed-off-by: Eric Curtin <eric.curtin@docker.com> * test(cli): adapt annotation tests to env helper Signed-off-by: Eric Curtin <eric.curtin@docker.com> --------- Signed-off-by: Eric Curtin <eric.curtin@docker.com> Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> Co-authored-by: John Myers <9696606+johntmyers@users.noreply.github.com> |
||
|
krishicks
|
9c6867ccec |
feat(auth): emit OCSF events for credential-less gateway requests (#4632)
Requests the gateway rejects before consulting any authenticator now emit an OCSF Authentication [3002] event: no credential presented, mTLS required without a client certificate, or an operator certificate combined with an authorization header. Each event carries a reason category in status_detail and never the presented credential. The response returned to the caller is unchanged. openshell-ocsf gains the Authentication class with user and service objects, and a shared OcsfCapture test layer. Schema validation now rejects attributes the class does not define and profile attributes whose profile the event does not declare. Refs #2912 Signed-off-by: Kris Hicks <khicks@nvidia.com> |
||
|
Florent BENOIT
|
a4c848adbd |
feat(vm): add host bind mount support via virtiofs (libkrun) (#3463)
* feat(vm): add host bind mount support via virtiofs (libkrun) Introduce host directory bind mounts for libkrun VM sandboxes using the virtiofs4 API with simplified permission semantics. The gateway accepts `enable_bind_mounts` in the `[openshell.drivers.vm]` TOML table and forwards it to the VM driver subprocess. Mounts are specified per-sandbox in driver_config.mounts[] with source, target, optional read_only (default: read-only), and an optional `type` that must be `bind`, so Docker and Podman bind entries (without `selinux_label`) parse unchanged. At sandbox validation the driver rejects mounts unless bind mounts are enabled and resource admission is disabled. It rejects GPU and other QEMU launches, sources with control characters, and more mounts than libkrun has interrupt lines for (32 on aarch64, 3 on x86_64), and checks targets against reserved VM-internal directories and each other (no nested targets). The QEMU check runs again once the launch backend is resolved. Host source directories are checked at provisioning, before image preparation, so a briefly missing source fails provisioning visibly instead of denying recovery of a persisted sandbox. Missing, inaccessible, symlinked, non-directory, and unreadable sources are rejected: libkrun does not follow a final symlink, and on macOS it panics during guest boot when it cannot open the share root. The driver generates virtiofs tags and always writes a guest-side manifest into the overlay disk, together with the init drop-ins, so an image-supplied manifest can never drive guest mounts. The launcher parses its tab-separated --vm-mount arguments from the right, so a source can never shift the tag or mode, and the driver logs each shared directory at launch. Guest init mounts the shares after every /sandbox ownership fixup and the init drop-ins, so recursive chowns never reach host files. It refuses targets that traverse symlinks in the image, hands newly created mount point parents under /sandbox to the sandbox user, and skips an empty manifest. krun_add_virtiofs4 is resolved optionally, so older libkrun runtimes still boot sandboxes without mounts. The kernel config lists CONFIG_FUSE_FS and CONFIG_VIRTIO_FS explicitly; the pinned libkrunfw base config already enables both. A vm_bind_mount e2e test checks a read-write share round-trips data with the host, accepts a new file, directory, and nested file that the host then sees, and that a read-only share rejects writes. libkrun on rootless Linux refuses file creation by a guest user whose UID/GID differ from its own, so the VM e2e gateway runs the sandbox as the gateway user there. It enables bind mounts like the Docker and Podman e2e gateways. The runtimes docs cover the mount limit, the symlink restriction, the guest-kernel exposure, and the rootless Linux UID/GID note. The debug-openshell-cluster skill lists VM bind mount checks. Signed-off-by: Florent Benoit <fbenoit@redhat.com> * fix(nix): refresh vm-runtime hashes to the libkrun v1.19.4 runtime The Nix vm-runtime package fetches the rolling vm-runtime release with fixed hashes pinned on 2026-09-16. The release was rebuilt on 2026-09-21 after the libkrun v1.19.4 / libkrunfw v5.6.1 bump (#3451), but the hashes were not updated, so CI kept substituting the 2026-09-15 runtime (libkrun v1.17.4) from the binary cache. That libkrun lacks krun_add_virtiofs4, which VM bind mounts need. Point the hashes at the current release assets. Signed-off-by: Florent Benoit <fbenoit@redhat.com> --------- Signed-off-by: Florent Benoit <fbenoit@redhat.com> |
||
|
Roland Huss
|
5add9df2cd |
feat(sdk/go)!: replace positional labels with functional Create options (#3198)
* feat(sdk/go)!: replace positional labels with functional Create options
Move labels and annotations from separate parameter positions into
functional options (WithLabels, WithAnnotations) on Create() and
CreateFromTemplate().
Before: Create(ctx, ws, name, spec, labels, CreateOptions{Annotations: ann})
After: Create(ctx, ws, name, spec, WithLabels(labels), WithAnnotations(ann))
This unifies two resource metadata fields that were inconsistently
placed (labels as positional, annotations in a struct) into a single
extensible options pattern matching the existing LogOption convention.
BREAKING CHANGE: Create() and CreateFromTemplate() signatures changed.
All callers must use WithLabels() and WithAnnotations() functional
options instead of positional labels and CreateOptions struct.
Closes #2807
Signed-off-by: Roland Huß <rhuss@redhat.com>
* fix(sdk/go): ignore nil Create options
Callers of the previous signature passed nil for the positional labels
parameter to create an unlabeled sandbox:
Create(ctx, ws, name, spec, nil)
That call still compiles against the new variadic signature, because an
untyped nil converts to CreateOption. ApplyCreateOptions then invoked it
and panicked with a nil pointer dereference, so the migration was unsafe
for existing consumers.
Skip nil options in ApplyCreateOptions and cover the legacy trailing-nil
call with regression tests against ApplyCreateOptions directly, the fake
client (Create and CreateFromTemplate), and the gRPC client.
Signed-off-by: Roland Huß <rhuss@redhat.com>
* refactor(sdk/go): apply Create options through the shared options helper
Signed-off-by: Roland Huß <rhuss@redhat.com>
* docs(sdk/go): use WithLabels in CreateSandboxFromTemplate examples
Signed-off-by: Roland Huß <rhuss@redhat.com>
---------
Signed-off-by: Roland Huß <rhuss@redhat.com>
|
||
|
Drew Newberry
|
b2069a267c |
feat(docker): support host and bridge supervisor networking (#4016)
* feat(docker): select supervisor networking for ECI workloads Signed-off-by: Drew Newberry <anewberry@nvidia.com> * test(docker): use automatic networking in e2e Signed-off-by: Drew Newberry <anewberry@nvidia.com> * test(docker): rely on existing e2e networking coverage Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs(docker): shorten supervisor networking configuration Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(docker): let local gateway discover supervisor endpoint Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(docker): make supervisor networking operator-selected Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(docker): resolve host gateway in the driver Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs(docker): simplify host networking guidance Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs(docker): trim configuration guidance Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(docker): discover host gateway once at driver startup Signed-off-by: Drew Newberry <anewberry@nvidia.com> --------- Signed-off-by: Drew Newberry <anewberry@nvidia.com> |
||
|
Yuedong Wu
|
44c07dab2a |
feat(helm): make gateway PVC size and StorageClass configurable (#3854)
Signed-off-by: Yuedong Wu <dwcn22@outlook.com> |
||
|
Polite_realism
|
10241ab47b |
fix(podman): trust an operator-supplied additional CA for direct egress (#3996)
* fix(podman): trust an operator-supplied additional CA for direct egress Since v0.1.2 the Podman supervisor runs as its own container, separate from the workload image, so it can no longer inherit a private CA baked into the workload's system trust store. Policy-allowed, inspected HTTPS requests to destinations signed by that CA fail upstream TLS establishment, and the only workaround is rebuilding the supervisor image with the CA baked in. Add a Podman-only additional_ca_bundle setting, independent of the existing proxy_ca_bundle (which requires https_proxy), that bind-mounts an operator PEM into the supervisor container and folds it into the upstream trust store alongside the system CA bundle. Validated on the gateway host at sandbox-create time, matching the existing corporate proxy CA bundle's validation, so a misconfigured path or certificate-free bundle fails closed immediately rather than producing a failed sandbox once the supervisor container starts. Scoped to Podman only, reusing the existing additive-bundle mechanism without new gateway.toml schema, proto fields, or cross-driver plumbing -- a narrower alternative to the in-flight cross-driver implementation in #3292, which remains the intended long-term direction. Fixes #3781. Signed-off-by: politerealism <burdcat17@gmail.com> * fix(podman): keep additional CA mount on the supervisor build_isolation_specs() keeps only trusted_mount() destinations on the supervisor, and the additional CA path was not in that list. The bind mount stayed on the workload while the supervisor received --additional-ca-bundle for a file it did not have, so startup failed. Add the path to trusted_mount() and cover the final supervisor and workload specs with a test. Correct the docs and a code comment to say the bundle is validated when the driver starts, and add Podman troubleshooting guidance to the debug-openshell-cluster skill. Signed-off-by: politerealism <burdcat17@gmail.com> --------- Signed-off-by: politerealism <burdcat17@gmail.com> |
||
|
Drew Newberry
|
6546691d5e |
docs(kubernetes): document managed workspaces setup (#4622)
* docs(kubernetes): document managed workspaces setup Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs(kubernetes): rename workspace guide title Signed-off-by: Drew Newberry <anewberry@nvidia.com> --------- Signed-off-by: Drew Newberry <anewberry@nvidia.com> |
||
|
ShijuandJohn Myers
|
29a2b8e7d9 |
feat(providers): report bounded token-grant failures (#4427)
* feat(providers): report bounded token-grant failures Signed-off-by: Shiju <shiju@nvidia.com> * test(providers): align policy metadata and TLS lockfile Signed-off-by: Shiju <shiju@nvidia.com> * test(providers): fix trusted grant recovery fixture Signed-off-by: Shiju <shiju@nvidia.com> * test(providers): enable denial API in recovery fixture Signed-off-by: Shiju <shiju@nvidia.com> * refactor(providers): simplify token-grant failure reporting Remove duplication and dead paths from the token-grant failure reporting without changing classification, the workload-visible 502 response, or the wire format. - Build the gRPC failure envelope with tonic-types, store the typed failure in EndpointFailure, and share one request-ID rule and one endpoint-status authority check. - Merge the gateway exchange client wrapper, drop forwarding validators, and name the request field when a reported cause is rejected. - Inject fallback credentials only from live provider state and delete the unread per-connection credential copy. Keep one 502 writer per path and restore the base token-cache handling. - Check the token endpoint after the cache lookup, still before any credential acquisition. - Document 429 and 5xx classification as implemented, and drop tests that repeat retained assertions through the same code path. Signed-off-by: Shiju <shiju@nvidia.com> * refactor(providers): trim token-grant failure tests and supervisor API Cut the token-grant failure tests to one test per distinct behavior across core, gateway, supervisor, CLI and Go, and replace the 794-line closure E2E with status assertions in the existing route-recovery E2E: the refused grant appears in sandbox status with the response's request ID, code, stage and upstream status, and the matching grant's later success clears it. - Pin the live provider snapshot with one helper used by inspected and fallback forwarding, and inject selector-matched grants through inject_for_selectors, which borrows the pinned context. Move the test-only wrapper into test_support. - Drop the 35-second wrappers around the gateway grant RPC. The shared channel bounds the connection attempt and the gateway bounds its token-service calls. - Log the token service's OAuth error and error_description, without control characters and limited to 256 characters each, with the request ID in the gateway or supervisor log. Status and the workload response still carry only the bounded cause. - Keep one check that an untrusted token-service certificate cannot assert a cause. Signed-off-by: Shiju <shiju@nvidia.com> * fix(providers): restore test import and features lockfile The injection tests named EndpointObservationSender through the parent module, which no longer imports it. The features suite lockfile kept the tonic TLS dependency that the removed E2E fixture added. Signed-off-by: Shiju <shiju@nvidia.com> * test(providers): fix body-bound fixture width and table type Pass the response-size width as a named argument; an implicitly captured constant panics at runtime. Name the mutation type in the validation table for Clippy. Signed-off-by: Shiju <shiju@nvidia.com> * test(providers): pad exact-bound body without a format width MAX_TOKEN_RESPONSE_BYTES exceeds u16::MAX, the largest format width, so the padded fixture panicked. Pad with repeat_n as before. Signed-off-by: Shiju <shiju@nvidia.com> * test(server): update public RPC schema hash after rebase Main added the ListProvidersRequest label selector. The merged public schema hash comes from the Branch Checks storage_proto test output. Signed-off-by: Shiju <shiju@nvidia.com> * fix(providers): keep token-service error text out of grant logs A token service can echo a submitted credential in its OAuth error or error_description. Log only the classified failure code, HTTP status and request ID, and cover the canary echo in a dedicated test binary. Signed-off-by: Shiju <shiju@nvidia.com> * docs(providers): tighten token grant failure documentation Signed-off-by: Shiju <shiju@nvidia.com> * docs(providers): condense token grant failure guidance Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> --------- Signed-off-by: Shiju <shiju@nvidia.com> Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> Co-authored-by: John Myers <9696606+johntmyers@users.noreply.github.com> |
||
|
alangou
|
c5519ea8ba |
fix(network): bind hostname policy to supervisor DNS (#4602)
Signed-off-by: Adrien Langou <alangou@nvidia.com> |
||
|
Simon Scatton
|
253884414a |
ci(release): decouple prerelease publication from qualification (#4420)
* ci(release): publish prereleases before qualification Signed-off-by: Simon Scatton <sscatton@nvidia.com> * ci(release): decouple prerelease publication from qualification Signed-off-by: Simon Scatton <sscatton@nvidia.com> --------- Signed-off-by: Simon Scatton <sscatton@nvidia.com> |
||
|
John T. Myers
|
909f336b62 |
fix(kubernetes): validate runtime control mount paths (#4522)
Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> |
||
|
krishicks
|
d20c090a69 |
docs(observability): add observability overview and traces pages (#4560)
Add an Observability overview that lists the traces, metrics, and OCSF events OpenShell produces, where each goes, and the minimal configuration to enable it, linking to the detailed pages instead of repeating them. Add a Traces page for what is traced, span names, resource attributes, gateway names, and trace context, and point the OTLP Export reference to it instead of describing spans there. Move metric naming conventions into Gateway Metrics, and replace the stale denied-connection example in OCSF JSON Export with a real record. List every common OTEL_* environment variable in the gateway OTLP tuning reference, split into the variables the gateway honors and the ones it ignores, with the reasons. Closes #4548 Signed-off-by: Kris Hicks <khicks@nvidia.com> |
||
|
krishicks
|
6dbe4adc56 |
feat(supervisor): export OCSF events as OTLP log records (#4545)
Supervisor OCSF events reached only the sandbox's local JSONL file and the shorthand log lines sent to the gateway, so collectors had no structured copy of most of the security record. When the gateway has `[openshell.gateway.otlp.ocsf]` configured, compute drivers now pass its endpoint and schema version to supervisors, which export their own OCSF events directly as OTLP log records. Supervisors already have egress, so the gateway and its supervisor connection carry no extra load. Supervisors use the same `openshell-otel` exporter as the gateway, with a supervisor resource identity that includes `openshell.sandbox.id` and `openshell.gateway.name`, so telemetry from sandboxes of different gateways sharing a collector can be told apart. This applies to supervisor spans too. Drivers pass the settings to supervisors as `OPENSHELL_OTLP_OCSF_ENDPOINT`, `OPENSHELL_OTLP_OCSF_SCHEMA_VERSION`, and `OPENSHELL_GATEWAY_NAME`; the gateway passes them to the VM driver as arguments. Collector headers that the gateway uses for OCSF export (`OTEL_EXPORTER_OTLP_LOGS_HEADERS`, else `OTEL_EXPORTER_OTLP_HEADERS`) travel in each sandbox's supervisor authentication bundle, which only the supervisor reads, so authenticated collectors accept supervisor records without exposing credentials to the workload. The VM driver now sends SIGTERM and waits for the supervisor to exit before closing its liveness pipe. Closing the pipe first made the supervisor exit at once and drop queued records. Closes #4544 Refs #2892 Signed-off-by: Kris Hicks <khicks@nvidia.com> |
||
|
krishicks
|
07fd2386ce |
feat(server): export gateway OCSF events as OTLP log records (#4500)
The gateway only persisted its OCSF events to the JSONL file, so collectors had to tail that file. Adding `[openshell.gateway.otlp.ocsf]` now also exports each gateway OCSF event as an OTLP log record, using the same resource identity as gateway traces. `schema_version` optionally downgrades the exported JSON to OCSF 1.1 or 1.3. OTLP endpoints are now configured per signal, so OCSF events can go to a collector that feeds a SIEM while spans go elsewhere. The existing `[openshell.gateway.otlp]` `endpoint` is the default for every signal and on its own still enables trace export. `[otlp.traces]` overrides the trace endpoint, and `[otlp.ocsf]` takes its own `endpoint` or falls back to the default. Each record carries the OCSF JSON as its body, the event time as its timestamp, a severity mapped from `severity_id`, and `ocsf.*` and `openshell.sandbox.id` attributes for filtering without parsing the body. Records flow through a queue bounded by count and bytes to the SDK OTLP log exporter in batches bounded the same way. Failed exports retry with backoff; records lost to a full queue, an oversized body, or exhausted retries are counted in metrics and reported to the collector in an `openshell.ocsf.gap` record on the next successful export. The exporter lives in `openshell-otel` so other processes can share it. Helm deployments set the `traces` and `ocsf` tables through `gatewayConfig`, and the local development gateway tasks enable OCSF export. Closes #4499 Refs #2892 Signed-off-by: Kris Hicks <khicks@nvidia.com> |
||
|
Yuedong Wu
|
3f66c26af1 |
fix(helm): render Restricted-compatible security contexts (#3340)
Signed-off-by: Yuedong Wu <dwcn22@outlook.com> |
||
|
Matthew Grossman
|
42e0d00a1d |
fix(supervisor): rename OCSF log files from .log to .jsonl (#4263)
* fix(supervisor): use jsonl files for independent OCSF retention Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * test(supervisor): trim and separate log retention regressions Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * refactor(supervisor): drop legacy OCSF log migration Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * docs(observability): remove legacy OCSF filename guidance Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * docs(ocsf): remove stale supervisor log filename comments Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * docs(supervisor): clarify denial reader log source Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * docs(supervisor): retain OCSF JSON opt-in explanation Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> --------- Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> |
||
|
Drew Newberry
|
6e69360d0e |
refactor(runtime): extract portable driver and isolation foundations (#4106)
* refactor(runtime): decouple supervisor access and boundary audit validation Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(runtime): keep supervisor process access boundary driven Signed-off-by: Drew Newberry <anewberry@nvidia.com> * feat(cli): support environment values from caller variables Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(runtime): extract driver-independent MXC foundations Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(runtime): defer UI policy support to MXC integration Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(runtime): integrate current policy and SSH launch contracts Signed-off-by: Drew Newberry <anewberry@nvidia.com> * test(runtime): reconcile fixtures with updated main Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(runtime): repair stacked branch CI contracts Signed-off-by: Drew Newberry <anewberry@nvidia.com> * test(runtime): update boundary fixtures for current contracts Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs(runtime): clarify scoped guards for launch inputs and delete Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(runtime): validate foreground attachment intent before launch Signed-off-by: Drew Newberry <anewberry@nvidia.com> --------- Signed-off-by: Drew Newberry <anewberry@nvidia.com> |
||
|
Eric Curtin
|
39b13d120b |
docs(cli): document sandbox create size limits (#4447)
Signed-off-by: Eric Curtin <eric.curtin@docker.com> |
||
|
Andrew Whiteandandrewwhitecdw
|
58fdb40246 |
docs(sandboxes): add IPv6 loopback example to Port Forwarding (#2449)
The core IPv6 forwarding fix landed on main via #2552; this carries the docs example forward into the current Port Forwarding section of docs/how-it-works/sandboxes/overview.mdx after the page move in #3705. Remaining scope is documentation only. Signed-off-by: Andrew White <andrewh@cdw.com> Co-authored-by: andrewwhitecdw <andrewwhitecdw@users.noreply.github.com> |
||
|
Shiju
|
b4baf23db3 |
fix(compute): allow stopping errored sandboxes for recovery (#4231)
* fix(compute): allow stopping errored sandboxes for recovery Signed-off-by: Shiju <shiju@nvidia.com> * test(compute): focus sandbox error recovery coverage Remove duplicate cancellation and idempotency checks covered by existing lifecycle tests. Keep recovery guidance focused on the stop/start commands and timeout exception. Signed-off-by: Shiju <shiju@nvidia.com> --------- Signed-off-by: Shiju <shiju@nvidia.com> |
||
|
|
e30c938bb1 |
feat(server): scope mutation locks and bound lock waits (#4147)
* test(server): add a local PostgreSQL test runner
The openshell-server tests that need a real PostgreSQL server are
ignored by default and had no shared way to run. The mutation-replay
test reads its own OPENSHELL_REPLAY_TEST_DATABASE_URL, so every
contributor had to provision a database by hand, and the advisory-lock
tests that follow in this series need the same setup.
Add mise run test:rust:postgres. It runs every ignored postgres_* test
in openshell-server against OPENSHELL_TEST_POSTGRES_URL, or starts a
disposable PostgreSQL container with Docker or Podman (CONTAINER_ENGINE
selects one) on the image pinned by the Kubernetes e2e fixture and
removes it on exit. Tests run one at a time because advisory locks are
database-wide. The runner always points the legacy replay variable at
the selected database, so an inherited URL cannot send that test to a
different server. test:postgres-runner checks that selection with a fake
cargo and needs no database or container engine. CI does not run the
PostgreSQL tests; TESTING.md documents the task.
Part of #3528
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* fix(server): hold the mutation guard while deleting a provider
DeleteProvider checked that no sandbox referenced the provider and then
deleted the record without holding the sandbox mutation guard. Sandbox
create and provider attach take that guard while they write provider
references, so one of them could add a reference after the attached
sandbox check passed, and the delete then removed a provider that a
sandbox spec still named.
Take the guard before the attached sandbox check, as provider create and
update already do, so the check and the delete run against a stable
set of sandbox references, and reject an empty name after authorization
but before the guard. A new test holds the guard, attaches the provider
while the delete waits, and asserts that the delete fails with
FailedPrecondition and leaves the provider in place.
Part of #3528
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* fix(server): release staged credentials when a refresh fails early
A provider credential refresh stages the minted values under new
credential handles before it commits the provider update, and deletes
those handles when validation or persistence fails. Two earlier returns
skipped that cleanup. The minted expiry was converted to a protobuf
timestamp only after staging, so an expiry outside the timestamp range
failed the refresh and left the staged values in the credential driver.
A failure to acquire the sandbox mutation guard returned the same way.
Convert the expiry before staging anything, so a bad value fails before
any handle exists, and delete the staged handles when the guard cannot
be acquired. A new test refreshes a stored credential with an expiry of
i64::MAX and asserts that the call fails, the provider keeps its
original handles, and the credential driver holds the same number of
values as before. The guard failure path has no test here, because the
SQLite guard used by unit tests cannot fail.
Part of #3528
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* feat(server): acquire mutation locks on a dedicated PostgreSQL lock pool
The cross-object mutation guard took its advisory-lock session from the
10-connection data pool and closed it on every release, so each guarded
mutation opened a new database connection, and lock holders competed
with their own critical sections for data connections. A lock wait that
hit the 10 second lock_timeout surfaced as INTERNAL.
Add a lazy, dedicated 4-connection lock pool and a new
persistence::mutation_lock module for the lock key, mode and set types.
Store::acquire_distributed_mutation_guard now takes a lock set and a
deadline, and each lock statement sets lock_timeout to the time left
before that deadline, so a cancelled acquisition never waits past it on
the server. Returned connections are scrubbed with
pg_advisory_unlock_all() and reused, and each return is bounded by 5
seconds so a stalled session cannot keep a pool permit.
A lock statement that PostgreSQL ends with SQLSTATE 55P03 fails with a
new LockTimeout error, and so does a wait for a lock connection during
which every one was checked out at some point, or that started with less
than a second left, too little to open one. The API returns it as
UNAVAILABLE with reason MUTATION_LOCK_TIMEOUT and a 1 second retry
delay, because the mutation lock was not acquired, so the request's
guarded writes did not run. Only the guard acquisition classifies 55P03;
any other 55P03 stays a database error. A lock connection that
PostgreSQL does not open by the deadline despite at least a second to do
so (refused, out of connection slots, starting up, or not answering, all
of which SQLx retries silently) is a database error, not a lock timeout.
Mutations still take the legacy global key exclusively, so old and new
replicas keep excluding each other during a rolling upgrade. Each
PostgreSQL-backed gateway pod can now open up to 14 database
connections (10 data plus 4 lock) instead of 10, and the chart's
autoscaling.maxReplicas comment says what its default of 4 needs. The
SSH host identity lock stays on a data connection, because sandbox
creation takes it while holding its mutation guard and a second
lock-pool connection there could exhaust the pool. The provider
credential refresh lock stays there too, so a refresh still progresses
while every lock connection is in use.
Part of #3528
Co-authored-by: divesh <dgude@nvidia.com>
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* perf(server): partition mutation locks by workspace and sandbox
Every cross-object mutation, and every lifecycle and supervisor update,
took one process-wide mutex and, on PostgreSQL, one fleet-wide advisory
lock. Unrelated sandboxes and providers therefore serialized across all
replicas, and one slow critical section delayed every other mutation in
the fleet.
Replace that lock with hierarchical intention locks on a global key, one
key per workspace and one key per sandbox, each held shared or
exclusive. Global settings and policy writers and platform-scope profile
writers hold the global key exclusively. Provider and workspace-scoped
profile writers, including DeleteProvider and credential refresh, hold
their workspace key exclusively. Sandbox writers and supervisor reports
hold their sandbox key exclusively under shared global and workspace
keys. Lifecycle, driver-watch and reconcile paths take only
process-local keys and keep relying on compare-and-swap across replicas.
Workspace and sandbox keys are domain-separated SHA-256 prefixes. Each
guard takes its local keys and then its advisory locks in ascending
order under one 10 second deadline, and no path nests guards. The
settings mutex is removed, since the global and sandbox keys cover it.
Sandbox settings are keyed by sandbox name, which a sandbox key no
longer covers once another sandbox reuses the name, so a settings
update returns NOT_FOUND unless its sandbox still holds the name after
the read, and ABORTED if the settings row was recreated before the
write.
The global key keeps its legacy value, so older replicas, which hold it
exclusively for every mutation, still exclude new ones during a rolling
upgrade. Guards now also give up after 10 seconds on SQLite, and
provider refresh reports a guard timeout as UNAVAILABLE instead of
INTERNAL. A supervisor report for a missing sandbox returns NOT_FOUND.
Startup endpoint-status reconciliation still holds the global key for
its whole scan.
Part of #3528
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* fix(server): reconcile endpoint status on startup one sandbox at a time
Startup endpoint-status reconciliation held the global mutation key
exclusively for its whole scan. On PostgreSQL that blocked every guarded
mutation on every replica until the starting gateway had walked all
sandboxes, and the scan first waited behind every in-flight sandbox
mutation. It also paged with an offset, so sandboxes deleted mid-scan
shifted rows between pages, and a write that raced the scan (a Conflict
from a lifecycle update on another replica, or a sandbox deleted after
it was listed) failed gateway startup.
Walk the sandboxes with keyset paging instead, and reset each candidate
under its own sandbox-scoped guard, up to four at a time to match the
lock pool. A candidate with a fresh shared owner is skipped before any
guard is taken. Under the guard the reconciliation re-checks the owner,
re-reads the sandbox, skips it when it is gone or no longer carries
endpoint status, and writes against the version it just read. A Conflict
or a database error at write time, which is how a row deleted after the
re-read surfaces, is retried with a fresh read, at most five attempts
per sandbox. A lock timeout on the guard is retried the same way, so a
holder of the sandbox, its workspace or the global key that releases
within the retry window cannot stop a replica from starting. No
lock-timeout retry starts once the scan has run for three lock timeouts,
so a scan stuck behind a long holder fails within about 40 seconds,
inside the chart's 60-second startup probe, and logs the sandbox it gave
up on. Store::list_all_messages loses its only caller and is removed.
Reconciliation stays fail-closed: a holder that outlasts the retry
window, an owner lookup failure, or a sandbox that keeps changing still
fails startup, so stale endpoint success is never served. User-visible
change: a restarting gateway no longer stalls mutations of unrelated
sandboxes and providers on its peers while it reconciles.
Part of #3528
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* perf(server): skip the endpoint-status lock when a session was replaced
When a supervisor session ends, the gateway resets that sandbox's tool
server endpoint evidence unless a replacement session already owns
endpoint observation. The replacement check ran only after the reset
had read the sandbox and taken its sandbox mutation guard, so a reset
whose supervisor had already reconnected still queued behind other
mutations of that sandbox, and on PostgreSQL it used a lock-pool
connection, only to find it had nothing to do.
Check for a replacement first: a live local session, or a fresh owner
record on a peer. When one exists, return without reading the sandbox
or taking the guard. Otherwise take the guard and repeat the same check
under it, because a replacement's pre-acknowledgement reset runs under
the same sandbox key and resetting after it would wipe the
replacement's fresh evidence. Both checks share one helper, so they
cannot drift apart. When no replacement exists, the reset costs one
extra owner-row read.
This helps most when a replica notices a dead stream late, after the
supervisor has already reconnected elsewhere (a network partition or a
keepalive expiry).
Deferring to a replacement leaves the old session's evidence to the
replacement's pre-acknowledgement reset, which can time out on its
bounded mutation lock wait. That reset's failure path removed the
replacement and released its owner record but scheduled no
invalidation, so the old session's success evidence could outlive both
sessions. An old session that ended after its replacement registered
already deferred this way, and the unguarded check now makes every
disconnect that finds the replacement defer too. When the reset fails
while the replacement is still current, also schedule the retried
invalidation that follows a session end. That failure path keeps the
lifecycle demotion from abandon_session_setup, but runs it in the
spawned task before the retry and only while this session is still
current, so a timed-out reset returns promptly.
Part of #3528
Co-authored-by: divesh <dgude@nvidia.com>
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* test(server): cover staged credential cleanup on a lock timeout
A provider credential refresh deletes its staged handles when it cannot
acquire the provider mutation guard, but that path had no test: the
SQLite guard used by unit tests could not fail. Mutation locks now
acquire under a deadline that tests can shorten, so the timeout path
can be exercised directly.
The new test shortens the lock deadline to 50 ms, holds the workspace
mutation guard, and refreshes a provider that already stores an AWS
access key. The refresh returns UNAVAILABLE with the
MUTATION_LOCK_TIMEOUT reason, the provider keeps its original handle,
which still resolves to the old key, and the credential driver holds
the same number of values as before the refresh.
Part of #3528
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* feat(server): export mutation lock wait and timeout metrics
A handler mutation guard can now wait up to 10 seconds for its keys and
then fail with UNAVAILABLE, but nothing showed operators how long guards
waited or how often they gave up, short of reading warning logs on each
replica.
Export openshell_server_mutation_lock_wait_seconds, a histogram of the
time one guard took to acquire all of its keys (local registry plus
PostgreSQL advisory locks), recorded on success, and
openshell_server_mutation_lock_timeouts_total, a counter of guard
acquisitions that timed out. Both carry a bounded scope label: global,
workspace or sandbox. The platform scope, whose workspace is empty,
counts as global. The histogram uses the same 1 ms to 15 s buckets as
the relay and peer histograms, and the timeout counter starts at 0 for
every scope, so rate() works on an idle replica. The timeout warning
now also logs the scope. A guard that fails for another reason, such as
a lock connection PostgreSQL did not open, is not counted as a timeout
and logs a "mutation lock acquisition failed" warning instead.
Lifecycle, driver-watch and reconcile paths take only process-local
locks and never time out, so they are not recorded.
Part of #3528
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* test(server): add PostgreSQL mutation lock integration tests
The scoped mutation locks were only unit-tested on SQLite, where the
database layer is a no-op. Add ignored postgres_* tests that run the
advisory-lock layer against a real PostgreSQL through mise run
test:rust:postgres, each in its own disposable schema (TestSchema).
persistence/mutation_lock_pg_tests.rs uses two stores on one database
as two replicas. It covers disjoint sandbox scopes holding at once, a
workspace key blocking only sandboxes in that workspace, the global key
blocking every scope, and exclusion in both directions against a raw
holder of the legacy global key, whose 55P03 stays a database error
outside a guard acquisition. It checks that timed-out and cancelled
acquisitions release the keys they already took, that a released lock
connection returns to the pool without residual locks, that a release
stalled on the network closes its session and the pool recovers, and
that the lock pool never grows past its size. It also checks that a
full lock pool times out as lock contention, while a lock connection
that PostgreSQL never answers fails as a database error.
compute/mutation_guard.rs runs the random scope mix across two
PostgreSQL runtimes and checks that a by-id guard waits for a workspace
writer on the other replica. The provider refresh lock test moves onto
the runner too, and checks that a refresh proceeds while every lock
connection is held.
A bench_postgres_* capacity check measures a paced burst of 500
reconnects 12 ms apart into one replica and fails when the p99 lock
wait reaches a fifth of the lock timeout. Machine load moves those
waits, so mise run test:rust:postgres skips it and mise run
test:rust:postgres:bench runs it alone, through a new
OPENSHELL_TEST_POSTGRES_PREFIX setting in the runner.
The tests add test-only helpers: the lock pool's size and idle count
and the backend pid of a held lock session. Nothing changes outside
tests and the test runner.
Part of #3528
Co-authored-by: divesh <dgude@nvidia.com>
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* ci: run the PostgreSQL-backed server tests
Branch Checks now runs the postgres_* tests through the same runner
developers use, on an x86_64 Linux runner with Docker. They cover lock
scope exclusion, interop with replicas still on the old global key,
cancellation, lock-pool bounds and connection failures, which no other
CI lane checks directly. The load-sensitive bench_postgres_* capacity
checks stay local.
Part of #3528
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* docs: document scoped mutation locks and lock-pool sizing
In the High Availability guide, size PostgreSQL for 14 connections per
gateway pod, 10 for data and 4 for the new lock pool, so three or more
replicas now exceed what the PostgreSQL default leaves, and keep
headroom for lock sessions that cancelled requests leave behind.
Explain which changes lock a sandbox, a workspace, or the whole fleet,
and that a lock wait over 10 seconds returns a retryable UNAVAILABLE
with reason MUTATION_LOCK_TIMEOUT. A new upgrade subsection says to
raise max_connections to the 14-per-pod size before upgrading, because
the rollout already runs new pods at that count, and warns that older
replicas keep serializing every mutation until the rollout finishes.
Monitor Capacity gains the lock metrics and their queries, and names
slow driver, credential, middleware, or profile source calls as another
cause of lock waits.
List the lock wait histogram and timeout counter, with their scope
label, on the Gateway Metrics page. The counter also counts background
work and leaves out lock connections that PostgreSQL does not open
although at least a second of the wait remained.
In the cluster debugging skill, replace the single advisory-lock
paragraph with the lock levels and pool, what each timeout detail
means, how a lock connection that PostgreSQL does not open shows up,
when terminating a lock backend is safe, a pg_locks query that shows
holders and waiters, and how to spot the global key. The per-pod greps
also read the timeout counter, and new rows cover lock timeouts and
lock connections that PostgreSQL does not open.
Part of #3528
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* feat(server): check mutation lock ordering in debug builds
The mutation lock ordering rules were prose only, so a nested guard on
a different key passed every test and could only deadlock under
contention. Debug builds now register each held guard with the Tokio
task (or the thread, outside a task) that acquired it, and panic before
waiting when a local key, a blocking lifecycle gate or the SSH identity
lock would break rules 1, 4 or 5. Startup endpoint-status
reconciliation and the tests that hold a guard while driving guarded
code run under lock_order::branch(), which marks deliberate concurrency
within one task. Release builds compile the check out.
Part of #3528
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* fix(server): do not hold gateway shutdown for the endpoint-status retry
When a replacement supervisor session failed its endpoint-status reset,
the spawned cleanup task kept the session lifetime guard through
retry_endpoint_status_after_supervisor_disconnect. That retry only ends
once the reset is durable, so a gateway shutdown during a database or
lock outage waited the full 10 s and reported ownership cleanup as
incomplete. The guard now covers only the lifecycle demotion, as in
finish_supervisor_session, and a SQLite test checks that shutdown
completes while the retry is still blocked.
Part of #3528
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* refactor(server): make the platform profile lock scope explicit
Platform-scope provider profile writers (import, update, delete) pass an
empty workspace on purpose and need X(global), because they validate
sandboxes in every workspace. MutationScope::profiles() now names that
escalation and picks Global for the platform scope, instead of relying
on MutationScope::Workspace("") to mean it. An empty workspace name now
trips a debug assertion, and release builds keep X(global) as a
fail-safe that over-locks rather than under-locks.
Part of #3528
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* perf(server): amortize local mutation lock table sweeps
LocalMutationLocks swept its whole table on every key lookup, so a
burst of N concurrent acquisitions on one replica cost O(N^2) under a
blocking mutex. The table is now swept only when a new key grows it
past the larger of 64 and twice the live entries left by the last
sweep, which makes acquisition amortized O(1). Released entries,
including those of cancelled waiters, stay bounded at about twice the
peak live set. The lifecycle gate registry keeps its per-call sweep.
Part of #3528
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
* feat(server): make the database pool ceiling configurable
Both stores hardcoded their pool ceiling: 10 connections for Postgres, 5
for on-disk SQLite. One pool is shared by every database-backed RPC, so
that number is also the gateway's ceiling on concurrent database work.
Once every connection is checked out, callers queue on acquire and sqlx
logs "time to acquire exceeded slow threshold"; sandbox creates then time
out and are retried, which adds load rather than shedding it.
The right ceiling is deployment-specific — it depends on how many gateway
replicas share the database and what max_connections the server itself
allows — so it cannot be a single number baked into the binary.
Expose it on the same config surface as the rest of the gateway's
settings: --db-max-connections, OPENSHELL_DB_MAX_CONNECTIONS, and the
TOML key database_max_connections, resolved in that precedence order.
Helm deployments set the key through gatewayConfig. Omitting it keeps
each backend's previous value, so existing deployments do not move.
Values below 1 are rejected rather than silently replaced by the default:
a zero pool would block every acquire, and quietly ignoring a typo would
reproduce the ceiling the operator is trying to lift. An in-memory SQLite
database stays pinned to one connection, since the database lives inside
that connection.
Refs: #2561
Signed-off-by: Bryce Wilkinson <22760097+bjw123@users.noreply.github.com>
(cherry picked from commit
|
||
|
Matthew Grossman
|
90161fbd7d |
feat(kubernetes): add operator allowlists for sandbox pod annotations and labels (#4497)
* feat(kubernetes): add operator allowlist for sandbox pod annotations Add allowed_pod_annotations to the Kubernetes driver config. Sandbox template annotations must match an exact key or a <domain>/* entry, and the default empty list rejects all of them. Keys in the openshell.ai and agents.x-k8s.io domains are always rejected. Expose the setting through the server.allowedPodAnnotations Helm value. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * test(kubernetes): cover pod annotation rejection on sandbox create Assert the exact rejected annotation keys, exercise create_sandbox_inner with a disallowed annotation, and point the rejection error at the Helm gatewayConfig path. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * fix(kubernetes): drop Helm path from pod annotation rejection error The rejection error reaches API callers, not operators, so name only the gateway config key. Spell the Helm path in the runtimes docs with the quoted table key to avoid an invalid nested gatewayConfig rendering. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * test(kubernetes): condense pod annotation allowlist tests Fold the allowlist matching and validation cases into table-driven tests, reuse the json_struct helper for templates, and document why the openshell.ai and agents.x-k8s.io domains are reserved. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * feat(kubernetes): add operator allowlist for sandbox pod labels Sandbox template labels were copied onto workload pods unfiltered. Labels select pods for NetworkPolicy rules and Services, so a caller could place a sandbox under rules meant for other workloads or set OpenShell-owned labels. Add allowed_pod_labels, sharing the allowed_pod_annotations entry format, validation, and reserved openshell.ai / agents.x-k8s.io domains. Empty by default, which rejects every template label. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> --------- Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> |
||
|
Eric Curtin
|
82977e27a4 |
fix(otel): enable gzip and https OTLP export (#4399)
* fix(otel): enable gzip and https OTLP export Signed-off-by: Eric Curtin <eric.curtin@docker.com> * test(otel): reject untrusted OTLP collector certificates Signed-off-by: Eric Curtin <eric.curtin@docker.com> --------- Signed-off-by: Eric Curtin <eric.curtin@docker.com> |
||
|
Simon Scatton
|
b9dc1f6466 |
fix(sdk-ts): preserve rpc errors for out-of-range responses (#4495)
Signed-off-by: Simon Scatton <sscatton@nvidia.com> |