mirror of
https://github.com/NVIDIA/OpenShell.git
synced 2026-10-16 05:20:22 +08:00
pull-request/3673
22
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
0a770d9173 |
feat(kubernetes): support HA gateway rebalancing (#1868)
* feat(kubernetes): support HA gateway rebalancing Signed-off-by: Drew Newberry <anewberry@nvidia.com> * perf(server): cache peer connections, tokens, and owner lookups Every forwarded relay rebuilt its setup from scratch: an owner lookup, a blocking read of the peer token, a TLS connect to the owning replica, and a TokenReview plus Pod GET on the receiving side. Sandbox service routing does this per HTTP request, so the apiserver calls scaled with traffic. Cache all of it on ServerState: - peer channels pooled per endpoint, so relays multiplex over one connection instead of redialing - peer tokens keyed by SHA-256, expiring at min(ttl, token exp) so a hit cannot accept an expired token - owner records for 3s against a 45s ownership TTL, still freshness checked before use Entries are evicted when a relay fails. Also raise HTTP/2 max_concurrent_streams to 1024, since pooling funnels every relay between two replicas onto one connection and hyper's default of 200 sits below the 256 pending-relay budget. Signed-off-by: divesh <dgude@nvidia.com> * perf(server): pool upstream connections for sandbox services Each HTTP request to a sandbox service opened its own supervisor relay, paying a new TCP connection and HTTP/1 handshake every time. Worse, it counted against the 32 in-flight relay cap, so a service handling more than 32 concurrent requests failed outright. Pool idle upstreams per endpoint and port, up to 8 each for 15s. Reuse is safe because the pool only returns a connection hyper reports as ready, and HTTP/1 cannot start a request until the previous body has drained. Upgrades are never pooled since they take the connection over, and a failed send evicts that endpoint. Pruning is bounded per key, with the full sweep limited to once per 30s. Signed-off-by: divesh <dgude@nvidia.com> * fix(server): address HA gateway review findings (#3449) - Let a gateway own supervisor sessions without a peer endpoint. Requiring one whenever the store is PostgreSQL broke every single-instance PostgreSQL deployment, because no sandbox supervisor could connect. A cross-replica request to an owner that advertises no endpoint now fails immediately naming the cause, instead of retrying until the wait timeout. - Close a supervisor session on heartbeat only when another replica owns it, or after renewals fail for the ownership TTL. A database error no longer drops every session heartbeating during an outage. - Clamp owner record ages at zero so a skewed or corrupt stored timestamp cannot produce a negative age. - Bound the cross-object advisory lock with a lock timeout, so a stuck holder fails instead of blocking every mutation in the fleet. - Refuse to start when a peer endpoint is configured on a multi-replica backend but peer authentication is unavailable, and warn when a multi-replica backend has no peer endpoint at all. - Reject a plaintext peer endpoint when the gateway serves TLS. - Skip the sandbox watch poller on single-replica backends, where the local update bus already sees every write. - Rate-limit the peer owner cache sweep so an insert no longer scans the whole map under the lock. - Retry GET and HEAD on a pooled upstream the sandbox closed, instead of returning 502, and drop an emptied endpoint from the pool right away. - Document the gateway peer environment variables and the post-rollout ownership skew operators should expect. Signed-off-by: divesh <dgude@nvidia.com> * fix(server): harden HA supervisor ownership Signed-off-by: Drew Newberry <anewberry@nvidia.com> --------- Signed-off-by: Drew Newberry <anewberry@nvidia.com> Signed-off-by: divesh <dgude@nvidia.com> Co-authored-by: Drew Newberry <anewberry@nvidia.com> Co-authored-by: divesh <dgude@nvidia.com> Co-authored-by: Divesh Chowdary <47188680+FrostGod@users.noreply.github.com> |
||
|
John T. Myers
|
1d010f4187 |
feat(sandbox): validate configuration before workload activation (#3259)
* feat(sandbox): validate configuration before workload activation Closes #3145 Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * fix(sandbox): bound startup failures and preserve activation history Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * fix(sandbox): enforce deadlines on startup RPC attempts Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * test(sandbox): isolate provider auto-create policy fixture Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * test(sandbox): remove unnecessary fixture string delimiters Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * test(sandbox): capture startup logs before ephemeral cleanup Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * fix(sandbox): preserve credential revocation after rebase Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * fix(server): retain provider revision helper for endpoint reports Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * fix(server): import provider object trait in production Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * fix(supervisor): preserve admission across boundary extraction Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * fix(sandbox): use existing boundary discovery imports Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * test(sandbox): distinguish workload and supervisor containers Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * test(server): reconcile admission schema with timestamp migration Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * test(server): reconcile admission with typed deletion schema Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * test(server): reconcile admission with mutation request IDs Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * test(supervisor): adapt local startup fixture to admission state Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * fix(supervisor): deduplicate startup quarantine diagnostics Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * fix(tui): show configuration blockers in sandbox notes Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * feat(sandbox): expire provisioning repair attempts after five minutes Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * fix(supervisor): reconcile admission with provider readiness Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * fix(tui): separate configuration summaries from full diagnostics Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * fix(tui): shorten invalid configuration note Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * test(server): refresh schema inventory after rebase Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> --------- Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> |
||
|
Shiju
|
9708ba9999 |
feat(providers): report applied sandbox provider changes (#3391)
* feat(providers): report applied sandbox provider changes Record exact provider mutation targets in shared configuration operations. Require authenticated evidence that credentials, effective policy, and the workload launch environment have been installed before reporting readiness. Add bounded CLI and Rust SDK status and wait support, preserving ordinary revision-scoped references for existing processes. Verify new-client rotation and acknowledged detach revocation without external-stable resolver changes. Signed-off-by: Shiju <shiju@nvidia.com> * fix(providers): align readiness times with protobuf contracts Represent readiness receipts, status, and operation times with Timestamp and report intervals with Duration. Reserve the scalar field tags, update all consumers and generated bindings, and preserve timestamp presence and nanosecond identity through storage and client validation. Qualify both empty-map constructors in the Linux boundary test so its module compiles while retaining the explicit default required by Clippy. Signed-off-by: Shiju <shiju@nvidia.com> * fix(cli): preserve provider mutation storage uncertainty Recognize the gateway's exact structured storage-uncertainty reason for provider attach, detach, and update. Explain that the change may already be saved and must be reconciled before retrying, without exposing server messages or metadata. Preserve uncertainty ahead of generic retry hints. Exercise saved mutations through the CLI and verify single submission, redaction, missing receipt handling, and untrusted error-detail rejection. Document the recovery guidance for users and the public CLI skill. Signed-off-by: Shiju <shiju@nvidia.com> * fix(cli): explain denied provider profile lookups Report exact and alias profile lookup denials with fixed permission and workspace guidance. Keep backend details redacted and stop before provider mutations. Cover denied create and update calls through the CLI. Verify the complete provider list independently in the cross-workspace OIDC regression, extracting its JSON object from surrounding startup diagnostics. Signed-off-by: Shiju <shiju@nvidia.com> --------- Signed-off-by: Shiju <shiju@nvidia.com> |
||
|
Mrunal Patel
|
c502be9fd7 |
feat(api): return typed deletion outcomes with explicit missing-target semantics (#3317)
* feat(api): return typed deletion outcomes with explicit missing-target semantics Implement phase 2 of #3051 across the public gateway API and first-party SDKs. Preserve asynchronous sandbox deletion and observed resource identities, reserve legacy wire fields, and document the coordinated migration. Signed-off-by: Mrunal Patel <mrunalp@gmail.com> * fix(sdk): bind deletion waits to sandbox identity Track accepted deletions by original sandbox identity in Rust and TypeScript. Preserve name-only waits and cover replacement races and lookup failures. Merge the phase-one cleanup fix and adapt its interceptor regressions to typed deletion outcomes. Refs #3051. Signed-off-by: Mrunal Patel <mrunalp@gmail.com> * test(e2e): accept asynchronous stopped sandbox deletion Allow either completed or accepted deletion output, then continue polling for actual sandbox absence. This preserves the lifecycle assertion under the phase-two deletion contract. Refs #3051. Signed-off-by: Mrunal Patel <mrunalp@gmail.com> --------- Signed-off-by: Mrunal Patel <mrunalp@gmail.com> |
||
|
Shiju
|
fd3fd9cf74 |
feat(sandbox): explain failed calls to external tool servers (#3207)
Show configured tool server addresses and their last observed connection results together in sandbox status. Keep sandbox lifecycle readiness separate so an external connection failure does not mark the sandbox unready. Expose direct endpoint records through the CLI and SDKs, with plain-language failure explanations and gateway acceptance times. Keep observation tracking, runtime reporting, and gateway validation in dedicated endpoint status modules. Preserve bounded reporting, request attribution, retry ordering, and configuration and supervisor authority checks. Clear obsolete observations while retaining the configured addresses, and document the distinction between an observed HTTP response, current availability, and tool success. Signed-off-by: Shiju <shiju@nvidia.com> |
||
|
Simon Scatton
|
48c449d8c8 |
chore(deps): replace ring with AWS-LC (#3243)
* chore(deps): replace ring with AWS-LC Signed-off-by: Simon Scatton <sscatton@nvidia.com> * fix(lint): address warnings after dependency upgrades Signed-off-by: Simon Scatton <sscatton@nvidia.com> * fix(tls): limit provider initialization to reqwest clients Signed-off-by: Simon Scatton <sscatton@nvidia.com> --------- Signed-off-by: Simon Scatton <sscatton@nvidia.com> |
||
|
Philippe Martin
|
118b250f01 |
feat(sandbox): support rootfs tar as --from source for VM driver (#2863)
* feat(sandbox): support rootfs tar as --from source for VM driver Accept flat rootfs tar archives (.tar, .tar.gz, .tgz) via the --from flag for VM-backed gateways. The CLI detects the archive extension, validates that the gateway uses the VM compute driver, and passes the tar path through driver_config. The VM driver copies the tar into its staging area and feeds it into the existing rootfs extraction and ext4 disk creation pipeline, skipping the container image pull/export steps. Closes #2175 Signed-off-by: Philippe Martin <phmartin@redhat.com> * fix(sandbox): validate rootfs tar path at the VM driver boundary The rootfs_tar_path field in driver_config was passed from the API caller directly to tokio::fs::copy without validation. An authenticated user bypassing the CLI could supply arbitrary host paths (e.g. /dev/zero for disk exhaustion, or readable host files for data exfiltration). Introduce a trusted staging directory that the VM driver creates on startup and advertises via GetCapabilities. The CLI now copies the tar into the staging directory before creating the sandbox, and the driver validates that the received path is a regular file inside the staging root and within a configurable size limit (default 10 GiB) before any I/O. New VmDriverConfig options: - rootfs_tar_staging_dir: override the staging directory (default: <state_dir>/rootfs-tar-staging) - rootfs_tar_max_bytes: override the size limit (default: 10 GiB) Addresses GATOR-28b5152e-01. Signed-off-by: Philippe Martin <phmartin@redhat.com> * fix(sandbox): request-scoped staging, size pre-check, and cleanup for rootfs tar Tighten the rootfs tar staging flow to address the remaining GATOR-01 obligations: - Request-scoped staging: the CLI creates a unique per-request subdirectory (req-<pid>) under the staging root instead of placing files directly in the shared directory. The driver enforces that the tar path is at depth 2 (staging_root/<subdir>/<file>), preventing cross-request path selection. - Size pre-check: the driver advertises rootfs_tar_max_bytes via GetCapabilities. The CLI reads this limit and rejects oversized files before copying, avoiding disk exhaustion in the staging directory. - Cleanup: the driver removes the request staging subdirectory after consuming the tar (on cache hit, copy success, or copy failure), ensuring staged data does not persist beyond the request. Signed-off-by: Philippe Martin <phmartin@redhat.com> * fix(vm): restore rootfs-tar sandboxes from persisted image identity On restore or restart, the one-shot staged tar archive has already been cleaned up. Reading the persisted image identity from the sandbox state directory and resolving the cached disk path directly avoids re-accessing the deleted staging path. Addresses GATOR-168b9210-01. Signed-off-by: Philippe Martin <phmartin@redhat.com> * fix(cli): use random staging dirs and enforce byte limit during rootfs tar copy Replace PID-based request staging directories with tempfile-generated random names to prevent collisions and make paths unpredictable. Replace bare tokio::fs::copy with a streaming copy loop that enforces the advertised max_bytes limit during transfer, closing the TOCTOU gap between the pre-copy size check and the actual copy. Signed-off-by: Philippe Martin <phmartin@nvidia.com> Signed-off-by: Philippe Martin <phmartin@redhat.com> * fix(sandbox): issue rootfs tar staging slots from the gateway A caller could name any host path in `driver_config.vm.rootfs_tar_path`, which the privileged VM driver then read. The CLI-side locality check did not apply to direct API requests. The gateway now owns staging. `BeginRootfsTarStaging` allocates a request-scoped directory under the driver-advertised staging root and returns an opaque single-use token; `CreateSandbox` carries the token, and the gateway substitutes the path it allocated before dispatching to the driver. `template.driver_config.<driver>.rootfs_tar_path` is rejected outright in request validation, so a caller-supplied path never reaches privileged I/O. Tokens are bound to the issuing workspace and subject, consumed once, and expire after 30 minutes. Outstanding slots are capped per caller and overall, so one caller can neither exhaust the staging filesystem nor starve others. An RAII guard reclaims the directory on every failure path after consumption, and an age-gated sweep runs at startup and on each reconcile pass for directories whose driver died before its own cleanup. The token is stripped from the public sandbox before persistence: the stored copy is returned verbatim by GetSandbox, ListSandboxes and WatchSandbox to every member of the workspace. Also fixes two defects this exposed: - The CLI wrote `rootfs_tar_path` at the top level of `driver_config`, but the gateway forwards only `driver_config.<driver_name>`, silently dropping unmatched keys. The archive never reached the VM driver, so the documented `--from ./rootfs.tar` flow did not work at all. Config is now nested under `vm` and deep-merged, so a caller's existing VM settings survive instead of being clobbered by a shallow extend. - Staging previously required `GetGatewayInfo`, which is restricted to `platform_admin`, making the feature unusable for ordinary users on any RBAC-enabled gateway. The new RPC matches CreateSandbox at `sandbox:write` / `workspace_role: user`. `compute_driver.proto` is unchanged; the gateway reads the staging root from the capabilities it already stores. Refs #2175 Signed-off-by: Philippe Martin <phmartin@redhat.com> * fix(vm): derive rootfs tar cache identity from archive contents The prepared-disk cache key combined the archive's full path with an mtime truncated to seconds, then mapped punctuation to `-`. Distinct paths such as `/tmp/a/b.tar` and `/tmp/a-b.tar` collapsed onto the same key and reused each other's disk, a rewrite within the same second kept stale contents, and a long path could exceed filesystem component limits. Identity is now a SHA-256 of the archive contents. This is also what makes the cache work at all now that the gateway allocates a fresh staging directory per request: a path-derived key would miss on every create. The archive is hashed, the cache checked, and only on a miss copied — so a hit skips writing a multi-gigabyte file. The copy is hashed as it is written and rejected if the digest differs from the first pass, which closes the window where the source changes during staging rather than approximating it with a re-stat. Refs #2175 Signed-off-by: Philippe Martin <phmartin@redhat.com> * fix(vm): decompress gzip rootfs tar archives during staging `--from` accepts `.tar.gz` and `.tgz`, but the driver staged whatever bytes it was given and the guest image-prep VM extracts the staged file with a plain `tar -xpf`. Compressed sources therefore depended on the guest tar auto-detecting gzip, and the prepared disk was sized from the compressed length, which is far too small for the expanded rootfs. Staging now detects gzip from the archive's magic bytes -- the driver only ever sees a gateway-issued staging path, never the caller's file name -- and writes an uncompressed tar. The digest still covers the source bytes, so the "archive changed while staging" check is unaffected, and expansion is bounded by `rootfs_tar_max_bytes` so a compression bomb cannot fill the host disk. `extract_rootfs_archive_to` sniffs gzip as well, so the host-side extraction path matches. Adds unit coverage for gzip staging, bounded expansion, and gzip extraction, plus an e2e sandbox created from a gzip-compressed export. Signed-off-by: Philippe Martin <phmartin@redhat.com> --------- Signed-off-by: Philippe Martin <phmartin@redhat.com> Signed-off-by: Philippe Martin <phmartin@nvidia.com> |
||
|
grs
|
74960ebfae |
feat(server): add sandbox templates (#2833)
* feat(server): add sandbox workload templates Signed-off-by: Gordon Sim <gsim@redhat.com> * feat(go-sdk): add sandbox workload template support Signed-off-by: Gordon Sim <gsim@redhat.com> * feat(rust-sdk): add sandbox workload template support Signed-off-by: Gordon Sim <gsim@redhat.com> * feat(python-sdk): add sandbox workload template support Signed-off-by: Gordon Sim <gsim@redhat.com> * feat(typescript-sdk): add sandbox workload template support Signed-off-by: Gordon Sim <gsim@redhat.com> * docs(agents): document sandbox workload templates Signed-off-by: Gordon Sim <gsim@redhat.com> * fix(cli): support default GPU requests in sandbox templates Signed-off-by: Gordon Sim <gsim@redhat.com> * fix(server): cap sandbox templates per workspace Signed-off-by: Gordon Sim <gsim@redhat.com> * docs(architecture): document sandbox workload template boundaries Signed-off-by: Gordon Sim <gsim@redhat.com> * feat(cli+sdk): expose sandbox workload template provenance Signed-off-by: Gordon Sim <gsim@redhat.com> * fix(cli): include sandbox template annotations in output Signed-off-by: Gordon Sim <gsim@redhat.com> * feat(sandbox): add label selectors to template listing Signed-off-by: Gordon Sim <gsim@redhat.com> * test(e2e): cover sandbox template failure paths Signed-off-by: Gordon Sim <gsim@redhat.com> * fix(server): preserve command and ttl when creating sandbox from template Signed-off-by: Gordon Sim <gsim@redhat.com> * test(server): add field coverage test for template merge Signed-off-by: Gordon Sim <gsim@redhat.com> * fix(go-sdk): add pagination support to fake client Signed-off-by: Gordon Sim <gsim@redhat.com> * fix(cli): warn on env vars that looks like secrets Signed-off-by: Gordon Sim <gsim@redhat.com> * fix(sdk-ts): propagate sandbox workspace through lifecycle calls Signed-off-by: Gordon Sim <gsim@redhat.com> * fix(docs): update workspace management docs Signed-off-by: Gordon Sim <gsim@redhat.com> * fix(ts-sdk): support command and tty when creating from template Signed-off-by: Gordon Sim <gsim@redhat.com> * fix(go-sdk): support command and tty when creating from template Signed-off-by: Gordon Sim <gsim@redhat.com> * test(python-sdk): verify command and tty handling when creating from template Signed-off-by: Gordon Sim <gsim@redhat.com> * fix(go-sdk): update docs and ClientInterface Signed-off-by: Gordon Sim <gsim@redhat.com> * fix(server): validate sandbox create specs before I/O Signed-off-by: Gordon Sim <gsim@redhat.com> * fix(go-sdk): guard empty DNS-1123 label validation Signed-off-by: Gordon Sim <gsim@redhat.com> * fix(python-sdk): allow empty template builder mappings Signed-off-by: Gordon Sim <gsim@redhat.com> * fix(cli): align template GPU JSON default output Signed-off-by: Gordon Sim <gsim@redhat.com> --------- Signed-off-by: Gordon Sim <gsim@redhat.com> |
||
|
Drew Newberry
|
69a05ebb3b | fix(sandbox): complete successful main processes (#2884) | ||
|
grs
|
40d1b48666 |
feat(provider): support for SPIFFE backed token exchange (#1970)
* feat(provider): add ability to request token exchange instead of client credentials as OAuth grant_type Signed-off-by: Gordon Sim <gsim@redhat.com> * test(proxy): add further tests for token exchange Signed-off-by: Gordon Sim <gsim@redhat.com> * test(provider): add runnable example for token exchange Signed-off-by: Gordon Sim <gsim@redhat.com> * test(e2e): cover Podman token exchange grants Signed-off-by: Gordon Sim <gsim@redhat.com> * refactor(oauth): extract duplicated functionality from server and supervisor Signed-off-by: Gordon Sim <gsim@redhat.com> * fix(provider): evict nearest-to-expiry entry from intermediate token cache Signed-off-by: Gordon Sim <gsim@redhat.com> * doc(supervisor): add podman example for token exchange Signed-off-by: Gordon Sim <gsim@redhat.com> * fix(provider): withhold token-exchange subject credentials Signed-off-by: Gordon Sim <gsim@redhat.com> --------- Signed-off-by: Gordon Sim <gsim@redhat.com> |
||
|
Drew Newberry
|
ef296806f5 |
feat(sandbox): add canonical main process (#2726)
* feat(sandbox): add canonical main process Closes #2710 Persist and supervise one canonical workload per sandbox, attach sandbox connect to its retained session, and make every unexpected main-process exit terminal. Signed-off-by: Drew Newberry <anewberry@nvidia.com> * feat(sandbox): simplify canonical main process contract Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(sandbox): preserve legacy VM main compatibility Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(sandbox): preserve main status across driver updates Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(sandbox): satisfy macOS process lint Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(sandbox): gate Linux exit acknowledgement publisher Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(sandbox): simplify main process plumbing Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(supervisor): make controlling tty ioctl portable Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(supervisor): initialize canonical process environment Signed-off-by: Drew Newberry <anewberry@nvidia.com> * perf(supervisor): optimize retained main session Signed-off-by: Drew Newberry <anewberry@nvidia.com> * feat(sandbox): detach main session on ctrl-c Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(sandbox): use explicit main detach keys Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(dev): atomically stage Docker supervisor Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(test): align Docker main environment assertion Signed-off-by: Drew Newberry <anewberry@nvidia.com> * feat(sdk): expose canonical main process fields Signed-off-by: Drew Newberry <anewberry@nvidia.com> --------- Signed-off-by: Drew Newberry <anewberry@nvidia.com> |
||
|
Seth Jennings
|
0f8fad23c4 |
feat(sandbox): add stop and start operations (#2653)
* feat(sandbox): add suspend and resume operations Signed-off-by: Seth Jennings <sjenning@redhat.com> * fix(server): preserve lifecycle work after cancellation Signed-off-by: Seth Jennings <sjenning@redhat.com> * fix(server): reconcile ambiguous lifecycle outcomes Signed-off-by: Seth Jennings <sjenning@redhat.com> * fix(server): complete suspended session cleanup Signed-off-by: Seth Jennings <sjenning@redhat.com> * fix(vm): preserve suspension state on resume failure Signed-off-by: Seth Jennings <sjenning@redhat.com> * fix(server): retry retained lifecycle transitions Signed-off-by: Seth Jennings <sjenning@redhat.com> * fix(server): clean sessions after suspend reconciliation Signed-off-by: Seth Jennings <sjenning@redhat.com> * test(sandbox): cover deleting suspended sandbox Signed-off-by: Seth Jennings <sjenning@redhat.com> * fix(kubernetes): preserve progressing sandbox suspension Signed-off-by: Seth Jennings <sjenning@redhat.com> * fix(kubernetes): bound suspend status polling Signed-off-by: Seth Jennings <sjenning@redhat.com> * fix(kubernetes): detect legacy sandbox suspension Signed-off-by: Seth Jennings <sjenning@redhat.com> * fix(tui): render suspended sandbox phases Signed-off-by: Seth Jennings <sjenning@redhat.com> * refactor(sandbox): rename suspend and resume lifecycle Signed-off-by: Seth Jennings <sjenning@redhat.com> * perf(server): clean stopped sessions on transition Signed-off-by: Seth Jennings <sjenning@redhat.com> * fix(kubernetes): fail fast on rejected stop Signed-off-by: Seth Jennings <sjenning@redhat.com> * fix(compute): fence stale restart lifecycle events Signed-off-by: Seth Jennings <sjenning@redhat.com> --------- Signed-off-by: Seth Jennings <sjenning@redhat.com> |
||
|
Derek Carr
|
9c019a93f5 |
Wire authorization into workspace model (#2445)
* feat(auth): implement RFC 0011 Phase 2 workspace authorization Signed-off-by: Derek Carr <decarr@redhat.com> * fix(auth): address PR review feedback on workspace authorization - Docker e2e: add --health-port and switch readiness probe from `openshell status` to `curl /healthz`, fixing a false-positive readiness check in OIDC mode where the CLI exited 0 without actually contacting the gateway - ListWorkspaces: move membership filtering from post-query N+1 lookups into a SQL EXISTS subquery so pagination applies to the visible set, not the global ordering. Add generic list_with_membership to the persistence layer. - Descriptor validator: reject role/scope fields on unauthenticated and sandbox auth modes, and allow-list workspace_role as user/admin and global_role as platform_admin to catch typos at startup Signed-off-by: Derek Carr <decarr@redhat.com> * fix(server): use authed request in delete telemetry test The workspace authorization added by the Phase 2 auth changes requires a Principal on every delete request. The delete-telemetry test was still using a bare Request::new, so extract_principal failed before the handler could acquire the delete gate, causing a 5-second timeout flake. Signed-off-by: Derek Carr <decarr@redhat.com> * fix(auth): address gator review findings for workspace authorization - Inject unauthenticated-local-dev principal in no-auth gateway mode so handlers that call extract_principal() always find one. - Cap label-selector membership query at MAX_PAGE_SIZE instead of u32::MAX to bound the in-memory read. - Authorize workspace membership before resolving workspace existence in all sandbox RPCs to prevent workspace-name enumeration by non-members. - Remove dead_code allow on AuthorizedWorkspace.workspace now that callers use the normalized name from the authz result. Signed-off-by: Derek Carr <decarr@redhat.com> * fix(auth): close workspace-name oracle and label-selector truncation Swap authorize-before-resolve ordering in 27 handlers across provider.rs, service.rs, policy.rs, and workspace.rs to prevent CWE-203 workspace-name enumeration by non-members. Add combined membership+label SQL query (list_with_membership_and_selector) to both persistence backends so ListWorkspaces with label selectors no longer silently drops results beyond the first page of membership matches. Signed-off-by: Derek Carr <decarr@redhat.com> * test(auth): add non-member rejection and membership+label persistence tests Add comprehensive test coverage for workspace authorization changes: - Non-member rejection tests across all 44 workspace-scoped handlers (sandbox, provider, service, policy, workspace, inference) verifying PERMISSION_DENIED is returned instead of NOT_FOUND to prevent CWE-203 workspace-name oracle - Persistence test for list_with_membership_and_selector verifying SQL-level membership EXISTS + label filtering, multiple predicates, no-match cases, and pagination Signed-off-by: Derek Carr <decarr@redhat.com> * fix(auth): format merged import line in sandbox tests Signed-off-by: Derek Carr <decarr@redhat.com> * fix(auth): address gator re-review findings on workspace authorization - Fix TUI unconditionally setting providers_v2_enabled after provider refresh; read the actual gateway setting via GetGatewayConfig at startup instead - Fix SQLite json_extract with dotted label keys (e.g. example.com/env) by quoting the key in the JSON path - Add authed_request wrappers to upstream OCI identity tests that were missing a principal after rebase - Add test proving GetGatewayConfig is accessible without Platform Admin - Add test for dotted/prefixed Kubernetes-style label key filtering Signed-off-by: Derek Carr <decarr@redhat.com> * fix(auth): address second gator re-review findings - Loosen GetGatewayConfig from platform_admin to scope-only so workspace users can discover providers_v2_enabled during sandbox creation with inferred-provider commands; update proto descriptor, descriptor validation, and RFC 0011 access table - Add validate_label_selector to handle_list_workspaces and escape single quotes in SQLite json_extract interpolation (CWE-89 defense-in-depth) - Re-fetch providers_v2_enabled after TUI gateway switch so the new gateway's capability is reflected - Add e2e test for workspace user with inferred-provider command - Add persistence test for adversarial label keys with SQL injection attempts - Add handler test for invalid label selector rejection in ListWorkspaces Signed-off-by: Derek Carr <decarr@redhat.com> * fix(auth): address third gator review findings - Cap label selector pairs at 64 (CWE-400) to bound SQLite dynamic SQL - Add SCOPE_ONLY_METHODS allowlist for scope-without-role RPCs (CWE-863) - Normalize ID-based data-plane handlers to return NOT_FOUND for unauthorized sandboxes, closing the cross-workspace oracle (CWE-203) - Fix TUI provider profile cache lookup key mismatch for legacy providers with empty profile_workspace - Add whoami to CLI skill reference command tree - Update TUI skill doc with workspace, provider, and settings coverage - Document scope/workspace orthogonality on GetGatewayConfig proto Signed-off-by: Derek Carr <decarr@redhat.com> * fix(auth): extend CWE-203 normalization to policy.rs sandbox handlers GetSandboxConfig and GetSandboxLogs in policy.rs had the same fetch-before-authorize pattern that leaked cross-workspace sandbox existence. Promote fetch_and_authorize_sandbox to pub(super) and use it from both sandbox.rs and policy.rs handlers. Signed-off-by: Derek Carr <decarr@redhat.com> * test(auth): update assertions for CWE-203 sandbox ID normalization Cross-workspace sandbox access via ID-based handlers now returns NOT_FOUND instead of PERMISSION_DENIED to prevent existence inference. Update the unit test and OIDC e2e assertion to match. Signed-off-by: Derek Carr <decarr@redhat.com> * fix(auth): narrow CWE-203 error mapping and correct whoami output formats Only remap PERMISSION_DENIED to NOT_FOUND in fetch_and_authorize_sandbox and RevokeSshSession, letting INTERNAL and UNAUTHENTICATED propagate as-is. Fix whoami --output format values in cli-reference.md to match the actual CLI (table/json/yaml, not text/json). Signed-off-by: Derek Carr <decarr@redhat.com> * fix(ci): share network namespace with Keycloak in containerized CI In GitHub Actions job containers, Docker port publishing lands on the host, not inside the job container. Detect this environment and attach Keycloak to the job container's network namespace instead, with hardened defaults (cap-drop ALL, no-new-privileges, loopback-only listener). Signed-off-by: Derek Carr <decarr@redhat.com> --------- Signed-off-by: Derek Carr <decarr@redhat.com> |
||
|
Derek Carr
|
5952a5a23f |
feat(workspace): add workspace resource model with scoping, membershi… (#2243)
* feat(workspace): implement workspace model (Phase 1 of RFC 0011) Implements workspace and membership model providing hard isolation boundaries for multi-player OpenShell deployments. Workspace CRUD with Kubernetes-style Terminating phase for graceful deletion. All resources scoped by workspace via ObjectMeta. Membership RPCs for workspace access control. Persistence migration shifts name uniqueness to (object_type, workspace, name). Provider profiles support platform and workspace scoping. Service routing uses workspace-prefixed DNS labels. Inference routes renamed and workspace-scoped with DeleteInferenceRoute RPC. Python SDK with WorkspaceClient, two-method list pattern (workspace-scoped and for_all_workspaces), and workspace parameter on all methods. CLI workspace flags, TUI workspace cycling. K8s driver filters unmanaged CRs and uses delete preconditions. Podman driver uses immutable container IDs. Label serialization fixed across all put_if call sites. Signed-off-by: Derek Carr <decarr@redhat.com> * fix(cli): delegate sandbox upload command to existing upload function The standalone `sandbox upload` command reimplemented upload logic inline with two bugs: it used `Path::exists()` which follows symlinks (rejecting dangling symlinks), and it ran git-aware filtering on symlink sources. The `run::sandbox_upload()` function already handles both cases correctly via `sandbox_upload_plan()`. Replace the inline logic with a call to the existing function. Signed-off-by: Derek Carr <decarr@redhat.com> * fix(e2e): shorten sandbox names and fix test compatibility Shorten the sandbox name in initial_sparse_policy_is_acknowledged_as_loaded from 'e2e-2159-sparse-enrich' (22 chars) to 'e2e-sparse-enrich' (17 chars) to comply with MAX_ROUTABLE_NAME_LEN (19 chars). Also capture stderr in create_keep_with_args so future sandbox creation failures include the actual CLI error instead of reporting empty output. Signed-off-by: Derek Carr <decarr@redhat.com> * test(workspace): add test coverage for workspace CRUD and persistence isolation Add unit tests for workspace create happy path, get round-trip, get not-found, get empty-name rejection, already-exists error, and resolve_workspace not-found. Add persistence test proving cross-workspace name uniqueness (same name in different workspaces produces separate records). Add workspace name max-length boundary tests. Fix e2e harness to include stderr in name-parse-failure error path. Align Python e2e test_workspace_crud with try/finally pattern. Document provider profile catalog workspace scoping gap in RFC 0011. Signed-off-by: Derek Carr <decarr@redhat.com> * fix(examples): update examples for workspace model compatibility Shorten sandbox names in demo scripts to fit the 19-character MAX_ROUTABLE_NAME_LEN limit: policy-demo prefix to pd-, multi-agent notepad derives a short SANDBOX_TAG from the run ID, governance interceptor uses gs-PID-RANDOM. Update vscode-remote-sandbox.md SSH host aliases from openshell-{name} to openshell-{name}.{workspace} format. Signed-off-by: Derek Carr <decarr@redhat.com> * feat(sdk): add workspace-scoped client and workspace CRUD Add WorkspaceScopedClient modeled after kube::Api::namespaced — captures workspace once and injects it into every sandbox request. Add workspace CRUD methods (create, get, list, delete) and list_sandboxes_all_workspaces on OpenShellClient. Extend SandboxRef with workspace field and add WorkspaceRef type. Include mock tests for all new operations. Signed-off-by: Derek Carr <decarr@redhat.com> * fix(lint): resolve clippy warnings in workspace test assertions Signed-off-by: Derek Carr <decarr@redhat.com> * fix(docs): convert indented code blocks to fenced in RFC 0011 Signed-off-by: Derek Carr <decarr@redhat.com> * fix(lint): resolve clippy warnings and apply cargo fmt across workspace Auto-format with cargo fmt and fix clippy warnings exposed by the reformat: unnecessary qualifications, map_unwrap_or, identical match arms, unused variable prefix, dead code annotations, and let-unit-value in e2e harness. Signed-off-by: Derek Carr <decarr@redhat.com> * fix(workspace): address workspace scoping issues from review - Add workspace field to settings JSON output (CLI) - Skip Podman containers missing workspace label instead of defaulting to empty string, matching K8s driver behavior - Add resource_version to list_by_scope SELECT in both SQLite and Postgres backends, with regression test - Gate PolicyLocalContext proposal/lookup routes on workspace readiness, returning 503 when workspace is not yet discovered - Block sandbox and provider creation in TUI all-workspaces mode - Clear workspace vectors in TUI reset_sandbox_state Signed-off-by: Derek Carr <decarr@redhat.com> * fix(workspace): make provider profile catalog workspace-aware Thread workspace through snapshot_catalog so the EffectiveProviderProfileCatalog enforces workspace boundaries on both read and write paths. UserProviderProfileSource now loads platform-scoped profiles (workspace "") plus the target workspace's profiles, preventing cross-workspace duplicate profile ID collisions that previously caused global catalog failures. Update RFC 0011 to reflect catalog scoping is implemented in Phase 1 rather than deferred to future work. Signed-off-by: Derek Carr <decarr@redhat.com> * fix(persistence): include workspace column in atomic policy revision INSERT put_policy_revision_atomic omitted the workspace column from the INSERT into the objects table in both SQLite and Postgres backends, causing atomically-written policy revisions to lose their workspace association. Add workspace field to AtomicPolicyRevisionWrite and thread it through both backend INSERT statements, matching the non-atomic put_policy_revision path which already included it. Signed-off-by: Derek Carr <decarr@redhat.com> * fix(proxy): skip ancestor walk when socket owner is the entrypoint collect_ancestor_identities walked the entire process tree above the entrypoint when the connecting process was the entrypoint itself, SHA256-hashing every ancestor binary (IDE, shell, container runtime). On dev machines with large binaries in the ancestor chain this exceeded the 30-second test timeout. When start_pid == stop_pid there are no intermediate ancestors to verify, so return an empty list immediately. Signed-off-by: Derek Carr <decarr@redhat.com> * fix(workspace): make provider profile catalog scope-aware Allow the same profile ID at platform and workspace scopes by introducing layered catalog entries where workspace profiles shadow platform profiles. Add source and scope fields to the ProviderProfile proto and CLI output. Migrate List/Get handlers to the catalog, fixing divergence with runtime profile resolution. Signed-off-by: Derek Carr <decarr@redhat.com> * fix(e2e): align podman e2e labels with centralized driver constants The podman driver moved its container labels to the centralized openshell.ai/ prefix, but the e2e test harness and cleanup script still referenced the old openshell.sandbox-* keys, causing the local_driver_token_restart test to fail on container lookup. Signed-off-by: Derek Carr <decarr@redhat.com> * fix(e2e): align python profile isolation test with scope-aware catalog Platform profiles are now visible in workspace listings as fallbacks per the layered catalog design. Update the assertion to match. Signed-off-by: Derek Carr <decarr@redhat.com> * fix(workspace): honor profile_workspace in runtime profile resolution Runtime profile lookups now consult provider.profile_workspace via get_type_profile_for_scope. Providers created with --global-profile (profile_workspace="") resolve to the platform profile even when a workspace profile shadows the same ID. All 6 runtime call sites updated; type-only call sites remain scope-agnostic. Signed-off-by: Derek Carr <decarr@redhat.com> --------- Signed-off-by: Derek Carr <decarr@redhat.com> |
||
|
Evan Lezar
|
21aaa89522 |
feat(gateway): add elevated gateway info (#2202)
* feat(cli)!: fold gateway metadata into list BREAKING CHANGE: openshell gateway info no longer shows local gateway registration metadata. Use openshell gateway list or openshell gateway list -o json for local registration details. Signed-off-by: Evan Lezar <elezar@nvidia.com> * feat(gateway): add elevated gateway info Signed-off-by: Evan Lezar <elezar@nvidia.com> --------- Signed-off-by: Evan Lezar <elezar@nvidia.com> |
||
|
John T. Myers
|
48a7d09e82 |
feat(providers): support profile updates (#1914)
* feat(providers): support profile updates Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * fix(providers): require CAS for profile updates Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * fix(providers): serialize profile update validation Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * fix(providers): serialize profile import invariants Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * fix(providers): require profile update target id Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> --------- Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> |
||
|
Yuedong Wu
|
ff028ce0df |
feat(server): support TLS certificate hot-reload (#1870)
* feat(server): support TLS certificate hot-reload Signed-off-by: Yuedong Wu <dwcn22@outlook.com> * refactor(server): extract shared TLS test utilities Signed-off-by: Yuedong Wu <dwcn22@outlook.com> --------- Signed-off-by: Yuedong Wu <dwcn22@outlook.com> |
||
|
Eric Curtin
|
d5b79e5ba1 |
refactor(server): deduplicate test helpers and grpc utilities (#1708)
Remove three groups of copy-pasted code in openshell-server: 1. grpc/mod.rs had a private current_time_ms() wrapper identical to the one already exported from persistence/mod.rs. Remove the duplicate and update the three grpc sub-modules (policy, sandbox, service) to import directly from crate::persistence. 2. test_store() was repeated verbatim in seven #[cfg(test)] blocks. Promote a single canonical version to persistence/mod.rs (cfg-gated) and replace all copies with crate::persistence::test_store() calls or a thin Arc wrapper in supervisor_session. 3. grpc_client_mtls() and build_tls_root() were copy-pasted across edge_tunnel_auth.rs and multiplex_tls_integration.rs. Move both into the existing tests/common/mod.rs shared module and import from there. |
||
|
alangou
|
2bdc968ed9 |
fix(gateway): make readiness health checks dependency-aware (#1328)
* feat(gateway): add readiness probe metrics and test-only store close Emit Prometheus readiness metrics for database probes (healthy gauge and outcome-labeled latency histogram) with coverage in health HTTP tests. Restrict Store::close behind test support cfg to prevent accidental runtime pool shutdown under live traffic. Signed-off-by: Adrien Langou <alangou@nvidia.com> * test(e2e): add simple e2e test with kubernetes to test /readyz Signed-off-by: Adrien Langou <alangou@nvidia.com> --------- Signed-off-by: Adrien Langou <alangou@nvidia.com> |
||
|
Taylor Mutch
|
a3b16c18ab | feat(auth): per-sandbox authentication to gateway (#1404) | ||
|
Eric Curtin
|
5620c8b229 |
refactor: deduplicate repeated patterns across crates (#1499)
Remove ~280 lines of duplicated code across 30 files in 5 areas: - centered_rect: consolidate 5 identical TUI layout helpers into a single pub fn in openshell-tui/src/ui/mod.rs - server test helpers: replace ~100 inline Store::connect() calls with local test_store() helpers; deduplicate test_server_state() in grpc/service.rs to use the shared test_support version - rogue PKI: extract 20-line rogue CA+client cert generation block (duplicated in two integration tests) into generate_rogue_pki() in tests/common/mod.rs - provider tests: replace 8 identical 28-line test modules with a single macro_rules! test_discovers_env_credential! invocation - label constants: centralize openshell.ai/ container label keys in openshell-core::driver_utils; update Docker and Kubernetes drivers to import from there instead of redefining them locally |
||
|
Eric Curtin
|
10af3e609f |
refactor: deduplicate shared test helpers (#1399)
Extract four categories of copy-pasted test code into single canonical locations: - POLICY_OBJECT_TYPE / DRAFT_CHUNK_OBJECT_TYPE constants that were defined identically in both persistence/sqlite.rs and persistence/postgres.rs are now owned by persistence/mod.rs. - test_server_state() was duplicated across grpc/provider.rs, grpc/policy.rs, and grpc/sandbox.rs. A shared grpc::test_support module in grpc/mod.rs now owns the single implementation; each submodule imports it. - The TestOpenShell stub (full OpenShell trait impl), install_rustls_provider(), PkiBundle, generate_pki(), and start_test_server() were copied across up to five openshell-server integration test files. A new tests/common/mod.rs module owns them; each test file uses mod common. - StubResponse, unique_socket_path(), and spawn_podman_stub() were duplicated between openshell-driver-podman/src/driver.rs and openshell-driver-podman/src/grpc.rs. A new src/test_utils.rs (cfg(test)-gated) owns the shared helpers. |