Commit Graph
191 Commits
Author SHA1 Message Date
alangou 0c7e59a953 fix(deps): bump russh, jsonwebtoken, tar and npm lint deps (#2617)
Signed-off-by: Adrien Langou <alangou@nvidia.com>
2026-08-06 09:39:43 +00:00
5548405fcb feat(credentials): add provider credential storage drivers (#2437)
* feat(credentials): add provider credential storage drivers

Signed-off-by: Taylor Mutch <taylormutch@gmail.com>

* fix(credentials): harden credential update handling

Signed-off-by: Taylor Mutch <taylormutch@gmail.com>

* fix(credentials): harden credential driver security, correctness, and performance

Address review findings from the credential storage drivers PR:

- Route additional_credentials through the driver on refresh to prevent
  silent data loss for multi-credential providers (e.g. AWS STS)
- Clean up stored credential handles on CAS failure during refresh to
  prevent orphaned secrets in external backends
- Enforce namespace validation in the Kubernetes Secrets driver to
  prevent cross-namespace credential access when allow_reference_namespace
  is not enabled
- Cache Vault Kubernetes auth tokens with 80% TTL to avoid re-authenticating
  on every credential operation
- Parallelize resolve_credentials in all three drivers using try_join_all
  for faster sandbox startup
- Add existingSecret support for the KEK Secret to fix helm template/GitOps
  workflows where lookup returns empty and regenerates the key
- Document RBAC blast radius for the Kubernetes Secrets credential driver
  and recommend a dedicated namespace

Signed-off-by: Varsha Prasad <varshaprasad96@gmail.com>
Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com>

* fix(credentials): add optimistic concurrency, fix thundering herd, parallelize operations

Use resourceVersion optimistic concurrency with retry loop for K8s
Secret ownership checks to prevent TOCTOU races. Switch Vault token
cache from RwLock to Mutex with double-check pattern to prevent
thundering herd on cache miss. Parallelize credential store and delete
operations across independent keys using try_join_all.

Signed-off-by: Varsha Prasad <varshaprasad96@gmail.com>
Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com>

* fix(credentials): handle partial failures, add delete retry, consolidate cleanup

Replace try_join_all with join_all in credential store/delete operations
to handle partial failures — successfully-stored handles are cleaned up
when another key fails. Add retry loop with conflict detection to
db-credstore delete_credential, matching the K8s driver pattern.
Consolidate 4 manual cleanup_pre_stored_provider_credentials call sites
into a single error handler using an async block. Remove inconsistent
.trim() from db-credstore validate_handle_owner.

Signed-off-by: Varsha Prasad <varshaprasad96@gmail.com>
Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com>

* fix(credentials): fix retry loop guard and remove unprotected validation

Remove attempt-count guard from 409/Aborted match arms in retry loops
so the post-loop Status::aborted error is reachable after exhausting
retries. Previously, last-attempt conflicts fell through to the
catch-all error arm, producing misleading Status::unavailable errors.

Remove duplicate validation calls that ran after
prepare_provider_credential_update but outside the cleanup-protected
async block, which would leak pre-stored handles on failure.

Signed-off-by: Varsha Prasad <varshaprasad96@gmail.com>
Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com>

* fix(credentials): add workspace/provider UUID to credential backend paths

Include workspace and provider ID in credential backend object paths to ensure
cross-workspace uniqueness and prevent credential collision (GATOR-1806c9be-01).

- Updated credential driver proto to include workspace and provider_id fields
- Modified Vault driver to include workspace/provider_id in managed_secret_path
- Modified Kubernetes Secrets driver to include workspace/provider_id in
  credential_owner_id and managed_secret_name
- Updated all credential runtime calls to pass workspace/provider_id
- Updated tests to use the new signatures

This prevents two workspaces sharing the same external credential store from
colliding on provider names, which was a critical security issue (CWE-639).

* fix(credentials): preserve provider-level expiration for handle-backed credentials

Compute effective expiration from both provider and driver values using the
earliest non-zero timestamp and skip expired values before insertion
(GATOR-1806c9be-02).

- Modified resolve_provider_handles to check provider credential_expires_at_ms
- Skip expired credentials during resolution instead of returning them
- Use effective expiration (min of provider and driver) in resolution results
- Fix inference.rs to preserve earliest expiration when merging

This ensures handle-backed credentials respect the same expiration semantics
as inline credentials.

* fix(credentials): stage refresh changes under new handles before validation

Stage credential replacements under new immutable handles instead of reusing
existing handles to prevent overwriting committed values before validation/CAS
(GATOR-1806c9be-03).

- Stage credentials with empty existing_handles map to force new handle creation
- Validate and CAS before the new values are committed to backend storage
- Delete old handles only after successful CAS
- On CAS failure, delete only the newly staged handles
- This prevents CWE-362/CWE-367 race conditions where failed refreshes could
  still modify or delete the active credential

The fix ensures that a rejected refresh cannot modify the backend object still
referenced by the committed provider record.

* fix(credentials): add timeouts to credential driver RPCs

Apply configured timeouts to both startup capability negotiation and runtime
RPCs to prevent indefinite hangs (GATOR-1806c9be-05).

- Add DEFAULT_CREDENTIAL_DRIVER_RPC_TIMEOUT_SECS constant (30s)
- Apply timeout to GetCapabilities during startup connection
- Apply timeout to all runtime RPCs (store, delete, resolve)
- Use tokio::time::timeout to bound the entire GetCapabilities operation
  during startup, not just the socket connection
- Return contextual deadline errors on timeout

This prevents a faulty or overloaded driver from hanging gateway operations
indefinitely.

* fix(credentials): fix test to use consistent workspace/provider identity

The Kubernetes auth Vault resolve test was constructing a managed path with
test-workspace/test-provider-id but sending default/prov-123 in the request,
causing validation to reject the request (GATOR-18e32351-01).

- Update test to use test-workspace and test-provider-id in the request to
  match the logical_path construction
- This ensures the test exercises the intended code path and validates
  Kubernetes auth resolution properly

The test now passes and correctly validates identity enforcement.

* fix(credentials): use unique staging ID for refresh to avoid overwrites

Stage refresh replacements under genuinely distinct immutable handles using
a unique staging ID to prevent overwriting committed values (GATOR-1806c9be-03).

- Generate a unique staging ID using UUID for each refresh operation
- Use this staging ID when storing credentials instead of the real provider ID
- Pass the same staging ID during cleanup on failure to delete only staged objects
- This ensures deterministic paths (Vault) and object names (K8s) don't collide
  with the committed provider's credentials

The fix prevents failed refreshes from silently replacing active credentials
or breaking providers by deleting still-referenced backend objects.

* fix(credentials): wrap credential driver RPCs in local timeouts

Add local tokio::time::timeout wrappers around credential driver RPCs to
bound non-compliant or stalled UDS peers (GATOR-1806c9be-05).

- Wrap StoreCredential, DeleteCredential, and ResolveCredentials in local timeouts
- Return contextual deadline_exceeded errors when timeouts occur
- Keep existing gRPC timeout metadata for compliant implementations
- GetCapabilities during startup was already wrapped in previous commit

This ensures a faulty local driver cannot hang gateway operations indefinitely,
even if it accepts the connection but never responds to the RPC.

* fix(credentials): preserve ownership for staged refreshes

Signed-off-by: Seth Jennings <sjenning@redhat.com>

* fix(credentials): bound startup capability probe

Signed-off-by: Seth Jennings <sjenning@redhat.com>

* test(provider): authenticate credential handler requests

Signed-off-by: Seth Jennings <sjenning@redhat.com>

* fix(ci): grant actions read to credential driver e2e

Signed-off-by: Seth Jennings <sjenning@redhat.com>

---------

Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
Signed-off-by: Varsha Prasad <varshaprasad96@gmail.com>
Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com>
Signed-off-by: Seth Jennings <sjenning@redhat.com>
Co-authored-by: Taylor Mutch <taylormutch@gmail.com>
Co-authored-by: Varsha Prasad Narsing <varshaprasad96@gmail.com>
2026-08-05 16:29:41 +00:00
Jim Meyer 8c7dd148a9 perf(net): set TCP_NODELAY on latency-sensitive TCP hops (#2220)
* perf(net): set TCP_NODELAY on all tunnel and proxy TCP hops

The sandbox tunnel added ~44 ms of latency to every small
request/response because no socket in the path disabled Nagle's
algorithm, so sub-MSS writes waited on delayed ACKs at each hop.

Set TCP_NODELAY on every latency-sensitive TCP socket:

- gateway: accepted connections on the public listener (gRPC relay
  frames and WS tunnel writes)
- CLI: edge tunnel local accept + underlying WebSocket TCP stream,
  insecure TLS connector (tonic's default connector already does
  this), and service-forward accepted sockets
- supervisor: direct-tcpip connect into the sandbox netns, TCP relay
  target dials, egress proxy accepted connections, and all upstream
  CONNECT/HTTP dials (via a new connect_upstream helper)

Setting TCP_NODELAY on connect is best-effort: a failure only costs
latency, so we log and continue rather than fail the connection.

The sandbox SSH transport rides a unix domain socket and gRPC client
channels use tonic defaults (nodelay on), so no change is needed there.

Fixes #2219

Signed-off-by: Jim Meyer <jim@meyer4hire.com>

* refactor(net): house TCP_NODELAY helper in a shared net module

Address review feedback on the TCP_NODELAY change:

- Move the shared set-nodelay helper out of supervisor_session into a
  new crate-private `net` module in openshell-supervisor-process, so
  ssh and supervisor_session no longer reach across modules through a
  pub(crate) item.
- Make the best-effort comments at each call site terse and consistent.

No behavior change; the benchmark ladder reproduces the same numbers.

Signed-off-by: Jim Meyer <jim@meyer4hire.com>

* refactor(net): consolidate TCP_NODELAY helpers into openshell_core::net

Move the best-effort TCP_NODELAY helpers into the shared openshell_core::net
module so every crate dials and configures sockets the same way:

- Add set_tcp_nodelay_best_effort (accepted/existing streams) and
  connect_tcp_nodelay_best_effort (dial + set) with unit tests.
- Migrate all call sites in openshell-cli, openshell-server, and the
  supervisor crates to the shared helpers.
- Remove the crate-private net module from openshell-supervisor-process.
- Document socket guidance in AGENTS.md (Network Sockets).

Signed-off-by: Jim Meyer <jim@meyer4hire.com>

* perf(net): set TCP_NODELAY on exec bridge and metadata server

The gateway-side single-use SSH-over-relay loopback bridge and the
sandbox IMDS metadata server were missed latency-sensitive TCP hops.
Set TCP_NODELAY on the accepted client connection and both russh client
dials of the exec bridge — interactive keystrokes and line-buffered PTY
output are the most tinygram-heavy traffic in the system — and on the
metadata server's accepted connections.

Also log unrecognized MaybeTlsStream variants in the edge tunnel so a
future TLS-backend change surfaces a silent TCP_NODELAY miss instead of
skipping it quietly.

Signed-off-by: Jim Meyer <jim@meyer4hire.com>

* perf(net): set TCP_NODELAY on openshell-sdk socket paths

The openshell-sdk crate landed on main with its own copies of the CLI's
hand-rolled sockets, which the CLI and TUI are meant to consume. Give
them the same treatment as the CLI equivalents:

- edge_tunnel: the accepted local tunnel connection and the WebSocket's
  underlying TCP socket (plain and rustls variants).
- transport: the dial in InsecureTlsConnector, tonic's custom-connector
  path.

Only these hand-rolled sockets need it. Tonic's own connector defaults
tcp_nodelay to true and applies it itself, so plain Endpoint::connect
callers were already covered.

Signed-off-by: Jim Meyer <jim@meyer4hire.com>

---------

Signed-off-by: Jim Meyer <jim@meyer4hire.com>
2026-08-04 17:59:13 +00:00
krishicks 4d55265f12 test(server): close traced handler before span assertion (#2604)
Keep the completed instrumented handler future in an explicit pinned box and
drop it after the await. This releases the handler-side request span clone
before the disconnect test checks producer ownership, avoiding compiler- and
platform-dependent retention of an unfinished span.

Signed-off-by: Kris Hicks <khicks@nvidia.com>
2026-08-04 17:00:56 +00:00
Polite_realismandAdam Miller 0e9a44cfa9 feat(build): add system CA root mode (#2324)
* feat(build): add system CA root mode

Allow distro builds to use native trust stores for supervisor upstream TLS while keeping bundled Mozilla roots as the default. Avoid bundled root crates in system-ca-roots builds by using native-root TLS features and z3 0.20.

Signed-off-by: Adam Miller <admiller@redhat.com>

* fix(build): keep CA root feature in telemetry-off verification

The telemetry-off task uses --no-default-features which now disables
bundled-ca-roots in addition to telemetry, triggering the compile_error
guard. Re-enable bundled-ca-roots explicitly so the task verifies only
telemetry compilation.

Signed-off-by: Scott Burdine <sburdine@nvidia.com>
Signed-off-by: politerealism <burdcat17@gmail.com>

* fix(sdk): disable oauth2 default features to prevent webpki-roots leak

The bare `oauth2 = "5"` dependency re-enabled default features
(rustls-tls → reqwest/rustls-tls → webpki-roots), defeating the
system-ca-roots feature gate. Mirror the CLI fix: disable defaults
and enable only the `reqwest` feature.

Signed-off-by: Quinn Burdine <sburdine@redhat.com>
Signed-off-by: politerealism <burdcat17@gmail.com>

* refactor(build): simplify CA root selection to single feature toggle

Replace mutually exclusive bundled-ca-roots / system-ca-roots features
with a single bundled-ca-roots toggle. Disabling it implies system roots
via rustls-native-certs, which is now a regular (non-optional) dependency.
This fixes cargo --all-features and simplifies the distro build interface
from --no-default-features --features system-ca-roots to just
--no-default-features.

Signed-off-by: Quinn Burdine <sburdine@redhat.com>
Signed-off-by: politerealism <burdcat17@gmail.com>

* feat(build): add system-ca-roots convenience alias and fix verify task

Add a system-ca-roots feature alias on openshell-sandbox that includes
all other defaults (telemetry) except bundled-ca-roots, so distro
builds can use --no-default-features --features system-ca-roots without
manually re-adding unrelated defaults. Update the verify CI task to use
the alias and scope checks to the sandbox package. Fix task description
to use "build mode" terminology instead of implying a Cargo feature.

Signed-off-by: Quinn Burdine <sburdine@redhat.com>
Signed-off-by: politerealism <burdcat17@gmail.com>

* ci: fix system CA roots step name to use build mode terminology

Signed-off-by: Quinn Burdine <sburdine@redhat.com>
Signed-off-by: politerealism <burdcat17@gmail.com>

* refactor(sandbox): reorder features to place system-ca-roots alias near default

Signed-off-by: Quinn Burdine <sburdine@redhat.com>
Signed-off-by: politerealism <burdcat17@gmail.com>

* fix(proxy): unwrap Result from build_upstream_client_config in tests

The function signature changed to return Result but the test call sites
were not updated, causing type mismatch compilation errors in CI.

Signed-off-by: Quinn Burdine <sburdine@redhat.com>
Signed-off-by: politerealism <burdcat17@gmail.com>

---------

Signed-off-by: Adam Miller <admiller@redhat.com>
Signed-off-by: Scott Burdine <sburdine@nvidia.com>
Signed-off-by: politerealism <burdcat17@gmail.com>
Signed-off-by: Quinn Burdine <sburdine@redhat.com>
Co-authored-by: Adam Miller <admiller@redhat.com>
2026-08-04 15:55:44 +00:00
Simon Scatton b9818619b8 test: disable tests flaky under parallel stress (#2611)
Repeated Bazel test targets with --runs_per_test at 10, 20, 50, 100, and
200 runs. The tests failed intermittently when multiple instances ran
concurrently. This indicates timing, shared tracing state, socket readiness,
or parallel-safety issues that need focused follow-up before re-enabling the
tests.

A normal non-repeated bazel test //... run passes with these tests ignored.

Disabled tests:
- sandbox_forward_foreground_fails_when_ssh_exits_before_listener_opens
- sandbox_forward_background_terminates_owned_child_when_listener_never_opens
- podman_socket_probe_accepts_successful_ping_response
- podman_socket_probe_rejects_docker_ping_response
- docker_socket_probe_accepts_successful_ping_response
- docker_socket_probe_rejects_podman_ping_response
- docker_socket_detection_returns_the_responsive_candidate
- podman_socket_detection_returns_the_responsive_candidate
- driver_watch_events_are_roots_and_store_operations_have_parents
- reconcile_sweeps_are_roots_and_operations_have_parents
- gateway_listeners_bind_ipv6_wildcard_and_ipv4_callback_on_same_port
- watch_producer_releases_request_span_when_client_disconnects
- expected_conflicts_leave_the_span_unmarked
- store_spans_record_what_they_touched_as_attributes
- store_operations_export_spans_with_parents
- refresh_worker_ticks_are_roots_and_store_operations_have_parents

Signed-off-by: Simon Scatton <sscatton@nvidia.com>
2026-08-04 15:38:50 +00:00
krishicks 8328412959 feat(vm): export driver traces over OTLP (#2564)
Continue distributed traces across the gateway-to-driver process boundary and
export VM driver spans to the same OTLP/gRPC collector. The driver reports as
the distinct openshell-driver-vm service.

Updated the gateway architecture and configuration reference with a generic
external-driver forwarding contract.

Instrumented:

- Every RemoteComputeDriver RPC injects the active W3C trace context into
  tonic metadata. Managed VM readiness and runtime initialization give startup
  capability probes stable parent operations rather than isolated root spans.
- A tonic service layer creates fixed, low-cardinality server spans for every
  ComputeDriver RPC. New handlers inherit tracing automatically; failures record
  OpenTelemetry error status and the gRPC status code.
- Background provisioning remains attached to CreateSandbox after the RPC
  returns without extending the RPC span lifetime.
- Provisioning records image preparation, bootstrap image resolution, overlay
  preparation, lifecycle configuration, pre-launch hooks, guest preparation,
  and launcher spawn as child spans.
- VM startup reconciliation roots one trace for the persisted-sandbox scan,
  with per-sandbox restore and provision operations beneath it. The root remains
  open until all spawned restore tasks finish.
- Delete cleanup records its own child operation.

Design notes:

- The gateway forwards its configured OTLP endpoint to managed external
  drivers. SDK `OTEL_*` variables continue to own sampling, batching, limits,
  headers, and transport tuning.
- The VM driver has its own tracer provider and service resource so trace
  backends preserve the service boundary.
- RPC operation names come from an explicit method mapping, keeping cardinality
  bounded without parsing the protobuf descriptor set at runtime.
- Propagation uses a remote SpanContext for spawned provisioning. This keeps
  one trace while allowing the CreateSandbox server span to finish when the
  RPC response is sent.
- Startup restoration is independent of gateway requests. It begins at the VM
  driver reconciliation span rather than attaching to an unrelated RPC.
- Existing tracing events remain on the logging path. The OpenTelemetry layer
  exports spans only and excludes the SDK exporter callsites to avoid recursive
  traces.
- Export configuration failures do not prevent the driver from serving, and
  buffered spans are drained during graceful shutdown.
- Trace fields identify drivers, sandboxes, images, lifecycle phases, and gRPC
  outcomes without recording credentials, sandbox tokens, or request query
  parameters.

Refs #2507

Signed-off-by: Kris Hicks <khicks@nvidia.com>
2026-08-03 20:52:22 +00:00
Simon Scatton 0a3ec7a112 feat(bazel): add rustfmt checks to tests (#2599) 2026-08-03 16:37:15 +00:00
Simon Scatton 1959ea19be build(bazel): establish RFC 0012 Rust reference graph (#2414)
Establish the Phase 1 reference implementation proposed by RFC 0012
while retaining the existing Cargo and Mise workflows during evaluation.

- pin Bazel 9.1.1 and configure Bzlmod, rules_rs, LLVM, protobuf, and
  Rust 1.95 toolchains
- import third-party crates from Cargo metadata and propagate the
  workspace version into Bazel targets
- add library, binary, proc-macro, unit-test, and integration-test
  targets across the supported Rust workspace crates and drivers
- generate protobuf Rust sources and descriptor sets under Bazel while
  preserving Cargo-compatible generated-code imports
- annotate aws-lc-sys and zstd-sys native dependencies, build Z3 4.15.2
  from source, and generate z3-sys bindings
- make CLI and procfs test fixtures available as explicit Bazel inputs
  without relying on fixed host binary paths
- define optimized release targets for Linux x86_64 and aarch64 CLI,
  sandbox, and gateway binaries, plus macOS aarch64 artifacts
- add Bazel, buildifier, and lcov to the Nix development environment

RFC: 0012 (rfc12 branch)
Refs: #2491

Signed-off-by: Simon Scatton <sscatton@nvidia.com>
2026-08-03 09:05:40 +00:00
Piotr Mlocek e75331772b test(server): stabilize watch span cancellation test (#2582)
* test(server): stabilize watch span cancellation test

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>

* test(server): poll span exporter at fixed interval

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>

---------

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
2026-08-01 00:40:48 +00:00
John T. MyersandJohn Myers 905b554c7c refactor(network): consolidate proxy egress pipeline (#2373)
* refactor(network): introduce shared egress pipeline

Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>

* test(network): cover shared proxy egress paths

Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>

* refactor(network): make destination authorization explicit

Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>

* refactor(network): pin proxy relay policy context

Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>

* test(network): lock relay generation contracts

Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>

* test(network): establish phase zero compatibility baseline

Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>

* feat(policy): detect ambiguous network endpoints

Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>

* feat(sandbox): fail closed on invalid policy updates

Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>

* refactor(network): invalidate relays on policy changes

Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>

* docs(policy): document validation failure posture

Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>

* test(network): cover validation and middleware egress

Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>

* fix(config): move policy failure mode to gateway toml

Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>

* test(network): name proxy contracts by behavior

Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>

* fix(network): align overlap validation with endpoint selection

Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>

* test(network): expect hard loopback denial

Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>

* test(network): match declared endpoint denial

Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>

* fix(policy): preserve path-specific endpoint overrides

Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>

* test(network): respect hard-blocked host gateways

Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>

* fix(network): reconcile proxy refactor with main

Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>

* fix(network): preserve CONNECT policy generation

Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>

* fix(policy): cover runtime endpoint glob semantics

Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>

* fix(server): reject ambiguous policies before persistence

Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>

* docs(policy): explain ambiguity preflight behavior

Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>

* fix(policy): compare body limits within protocol

Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>

* test(proxy): avoid global tracing capture race

Signed-off-by: John Myers <johntmyers@users.noreply.github.com>

* chore(server): format rebased provider tests

Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>

* test(server): authenticate rebased policy requests

Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>

* fix(sandbox): retain runtime on middleware outage

Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>

* fix(server): preflight provider composition activation

Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>

* fix(sandbox): distinguish runtime failure transitions

Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>

---------

Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>
Signed-off-by: John Myers <johntmyers@users.noreply.github.com>
Co-authored-by: John Myers <johntmyers@users.noreply.github.com>
2026-07-31 20:59:00 +00:00
krishicks 489bb0d5dd fix(server): isolate otel tracing test exporters (#2579)
Give each span-assertion test its own thread-scoped in-memory exporter so
parallel tests cannot contaminate or reset captured spans. Keep a bare global
tracing registry only to preserve callsite interest, and serialize scoped
subscriber changes because tracing caches that interest process-wide.

Remove the test-only OTLP collector, polling delivery barrier,
transport-specific test, and direct opentelemetry-proto dependency. Seed the
expected persistence conflict before tracing begins so its assertion window
contains only the operation under test.

Signed-off-by: Kris Hicks <khicks@nvidia.com>
2026-07-31 18:56:28 +00:00
Evan LezarandDrew Newberry d220d89468 feat(compute): negotiate gateway callback listeners (#2492)
* feat(compute): query gateway listener requirements

Signed-off-by: Evan Lezar <elezar@nvidia.com>

* feat(compute): add Podman listener requirements

Signed-off-by: Evan Lezar <elezar@nvidia.com>

* test(docker): use default gateway bind address

Signed-off-by: Evan Lezar <elezar@nvidia.com>

* fix(gateway): avoid wildcard primary listener

Signed-off-by: Evan Lezar <elezar@nvidia.com>

* fix(podman): validate callback listener discovery

Signed-off-by: Evan Lezar <elezar@nvidia.com>

* fix(server): support split dual-stack listeners

Signed-off-by: Evan Lezar <elezar@nvidia.com>

* fix(podman): support legacy rootless listener discovery

Signed-off-by: Evan Lezar <elezar@nvidia.com>

* test(e2e): accept loopback plaintext rejection

Signed-off-by: Evan Lezar <elezar@nvidia.com>

* docs(agent): add callback listener diagnostics

Signed-off-by: Evan Lezar <elezar@nvidia.com>

* fix(server): restrict compute callback listeners

Signed-off-by: Evan Lezar <elezar@nvidia.com>

* fix(podman): validate local callback port

Signed-off-by: Evan Lezar <elezar@nvidia.com>

* test(server): clarify callback listener contract

Signed-off-by: Evan Lezar <elezar@nvidia.com>

* fix(podman): require pasta for local callbacks

Signed-off-by: Evan Lezar <elezar@nvidia.com>

* docs(gateway): document RPM listener default

Signed-off-by: Evan Lezar <elezar@nvidia.com>

* refactor(server): keep listener provenance diagnostic-only

Signed-off-by: Evan Lezar <elezar@nvidia.com>

* fix(compute): preserve callback listener isolation

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* test(e2e): remove Podman callback relay

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(packaging): preserve Podman callback loopback

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* ci(e2e): run VM smoke on nested-virt runner

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* ci(e2e): gate VM smoke on usable KVM

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* ci(e2e): probe KVM through VM driver

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* ci(e2e): tolerate hosted KVM denial

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* test(server): close traced futures before assertions

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* revert: remove tracing test stabilization

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

---------

Signed-off-by: Evan Lezar <elezar@nvidia.com>
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
Co-authored-by: Drew Newberry <anewberry@nvidia.com>
2026-07-31 16:41:06 +00:00
krishicks 1a25439cab refactor(otel): share OTLP trace provider setup (#2567)
Extract common OpenTelemetry provider construction into openshell-otel so
OpenShell services share one OTLP/gRPC export implementation.

The shared crate owns:

- typed exporter setup errors and non-fatal provider enablement;
- endpoint trimming and URI validation before lazy exporter connection;
- fixed and environment-or-default service-name policies;
- service version and caller-supplied resource attributes;
- batch tracer-provider construction; and
- span-only tracing layers that exclude OpenTelemetry exporter callsites.

Migrate the gateway to the shared provider while retaining its configurable
service name, error marking, and tracing test collector. Add the shared crate
to the architecture inventory and document the tracing boundary.

Refs #2507

Signed-off-by: Kris Hicks <khicks@nvidia.com>
2026-07-31 15:22:13 +00:00
alangou 5541398ccb fix(deps): update russh dompurify and base image of the gateway (#2575)
Signed-off-by: Adrien Langou <alangou@nvidia.com>
2026-07-31 12:54:03 +00:00
krishicks fa24299092 feat(gateway): export traces over OTLP (#2534)
Add an opt-in OTLP/gRPC trace exporter to the gateway. Export is enabled
by the presence of an `[openshell.gateway.otlp]` table with an endpoint;
there is no separate toggle.

Instrumented:

- Inbound request server spans, named for the RPC (`$service/$method`) or
  `{method} {path}` for plain HTTP. They continue valid W3C `traceparent`
  context when present and start a new trace otherwise. gRPC spans also
  carry `rpc.system`, `rpc.service`, `rpc.method`, and trailer-derived
  `rpc.grpc.status_code`.
- Compute driver calls (create, delete, list, get, validate, watch) as
  client spans anchored on the `ComputeDriver` contract.
- Store reads and writes as children of the current request or loop span.
- Work with no inbound request: compute driver initialization, the sandbox
  reconcile sweep, provider credential refresh tick, and driver watch events.
  Each roots one operation trace so its child work does not arrive as anonymous
  single-span traces.

This is deliberately not exhaustive. Auth, policy evaluation, and
middleware remain uninstrumented, as do store lifecycle calls (`ping`,
`close`) that a readiness poll would turn into a span per tick. The aim is
a useful trace tree at a reviewable size; coverage can grow against real
traces.

Design notes:

- The TOML table owns whether and where to export. The SDK `OTEL_*`
  variables own how; sampling, batching, and limits are not mirrored into
  gateway config.
- The OpenTelemetry layer exports spans only. Existing `tracing` events
  remain on the stdout and sandbox-log paths and are not copied into trace
  payloads.
- Telemetry never blocks the gateway. A malformed endpoint logs an error
  and disables export rather than failing startup, and buffered spans are
  drained during graceful shutdown.
- Failed spans carry error status without a separate `error.type` attribute.
  Request spans use HTTP status and gRPC response trailers; driver spans use
  the returned gRPC status; autonomous loop spans record failed results
  explicitly. Store spans exempt `UniqueViolation` and `Conflict`, because
  those errors report expected contention such as a held lease or an
  optimistic-concurrency retry.
- The compute driver is reachable only through `TracedDriver::call`, so a
  call cannot skip its span. This is the client half of a client/server pair
  and the single place to inject context if drivers move out of process.
- Tests share one process-wide subscriber and in-memory exporter because
  `tracing` caches callsite interest globally.

Inbound W3C trace context is propagated into gateway request spans. Context
is not yet injected into outbound driver calls, so a future out-of-process
driver would still need propagation at the `TracedDriver` seam.

The Helm chart is intentionally unchanged, so OTLP export cannot yet be
enabled on a chart-deployed gateway.

Refs #2507

Signed-off-by: Kris Hicks <khicks@nvidia.com>
2026-07-30 20:29:42 +00:00
Nave Cohen 28f3bee0ce fix(server): notify watchers after atomic policy commits (#2556)
Signed-off-by: Nave Cohen <nave@monday.com>
2026-07-30 17:05:48 +00:00
Evan LezarandDrew Newberry df698042d0 refactor(server): isolate gateway listener context (#2542)
* refactor(server): isolate gateway listener context

Signed-off-by: Evan Lezar <elezar@nvidia.com>

* refactor(server): preserve gateway listener binding

Signed-off-by: Evan Lezar <elezar@nvidia.com>

* fix(server): preserve covered listener callback scope

Signed-off-by: Evan Lezar <elezar@nvidia.com>

* refactor(server): reuse gateway listener spec

Signed-off-by: Evan Lezar <elezar@nvidia.com>

* ci(e2e): pin rootless podman packages

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

---------

Signed-off-by: Evan Lezar <elezar@nvidia.com>
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
Co-authored-by: Drew Newberry <anewberry@nvidia.com>
2026-07-30 05:38:22 +00:00
Derek Carr 9c019a93f5 Wire authorization into workspace model (#2445)
* feat(auth): implement RFC 0011 Phase 2 workspace authorization

Signed-off-by: Derek Carr <decarr@redhat.com>

* fix(auth): address PR review feedback on workspace authorization

- Docker e2e: add --health-port and switch readiness probe from
  `openshell status` to `curl /healthz`, fixing a false-positive
  readiness check in OIDC mode where the CLI exited 0 without
  actually contacting the gateway

- ListWorkspaces: move membership filtering from post-query N+1
  lookups into a SQL EXISTS subquery so pagination applies to the
  visible set, not the global ordering. Add generic
  list_with_membership to the persistence layer.

- Descriptor validator: reject role/scope fields on unauthenticated
  and sandbox auth modes, and allow-list workspace_role as
  user/admin and global_role as platform_admin to catch typos at
  startup

Signed-off-by: Derek Carr <decarr@redhat.com>

* fix(server): use authed request in delete telemetry test

The workspace authorization added by the Phase 2 auth changes requires
a Principal on every delete request. The delete-telemetry test was still
using a bare Request::new, so extract_principal failed before the handler
could acquire the delete gate, causing a 5-second timeout flake.

Signed-off-by: Derek Carr <decarr@redhat.com>

* fix(auth): address gator review findings for workspace authorization

- Inject unauthenticated-local-dev principal in no-auth gateway mode so
  handlers that call extract_principal() always find one.
- Cap label-selector membership query at MAX_PAGE_SIZE instead of
  u32::MAX to bound the in-memory read.
- Authorize workspace membership before resolving workspace existence in
  all sandbox RPCs to prevent workspace-name enumeration by non-members.
- Remove dead_code allow on AuthorizedWorkspace.workspace now that
  callers use the normalized name from the authz result.

Signed-off-by: Derek Carr <decarr@redhat.com>

* fix(auth): close workspace-name oracle and label-selector truncation

Swap authorize-before-resolve ordering in 27 handlers across
provider.rs, service.rs, policy.rs, and workspace.rs to prevent
CWE-203 workspace-name enumeration by non-members.

Add combined membership+label SQL query (list_with_membership_and_selector)
to both persistence backends so ListWorkspaces with label selectors no
longer silently drops results beyond the first page of membership matches.

Signed-off-by: Derek Carr <decarr@redhat.com>

* test(auth): add non-member rejection and membership+label persistence tests

Add comprehensive test coverage for workspace authorization changes:
- Non-member rejection tests across all 44 workspace-scoped handlers
  (sandbox, provider, service, policy, workspace, inference) verifying
  PERMISSION_DENIED is returned instead of NOT_FOUND to prevent
  CWE-203 workspace-name oracle
- Persistence test for list_with_membership_and_selector verifying
  SQL-level membership EXISTS + label filtering, multiple predicates,
  no-match cases, and pagination

Signed-off-by: Derek Carr <decarr@redhat.com>

* fix(auth): format merged import line in sandbox tests

Signed-off-by: Derek Carr <decarr@redhat.com>

* fix(auth): address gator re-review findings on workspace authorization

- Fix TUI unconditionally setting providers_v2_enabled after provider
  refresh; read the actual gateway setting via GetGatewayConfig at
  startup instead
- Fix SQLite json_extract with dotted label keys (e.g. example.com/env)
  by quoting the key in the JSON path
- Add authed_request wrappers to upstream OCI identity tests that were
  missing a principal after rebase
- Add test proving GetGatewayConfig is accessible without Platform Admin
- Add test for dotted/prefixed Kubernetes-style label key filtering

Signed-off-by: Derek Carr <decarr@redhat.com>

* fix(auth): address second gator re-review findings

- Loosen GetGatewayConfig from platform_admin to scope-only so workspace
  users can discover providers_v2_enabled during sandbox creation with
  inferred-provider commands; update proto descriptor, descriptor
  validation, and RFC 0011 access table
- Add validate_label_selector to handle_list_workspaces and escape
  single quotes in SQLite json_extract interpolation (CWE-89
  defense-in-depth)
- Re-fetch providers_v2_enabled after TUI gateway switch so the new
  gateway's capability is reflected
- Add e2e test for workspace user with inferred-provider command
- Add persistence test for adversarial label keys with SQL injection
  attempts
- Add handler test for invalid label selector rejection in
  ListWorkspaces

Signed-off-by: Derek Carr <decarr@redhat.com>

* fix(auth): address third gator review findings

- Cap label selector pairs at 64 (CWE-400) to bound SQLite dynamic SQL
- Add SCOPE_ONLY_METHODS allowlist for scope-without-role RPCs (CWE-863)
- Normalize ID-based data-plane handlers to return NOT_FOUND for
  unauthorized sandboxes, closing the cross-workspace oracle (CWE-203)
- Fix TUI provider profile cache lookup key mismatch for legacy
  providers with empty profile_workspace
- Add whoami to CLI skill reference command tree
- Update TUI skill doc with workspace, provider, and settings coverage
- Document scope/workspace orthogonality on GetGatewayConfig proto

Signed-off-by: Derek Carr <decarr@redhat.com>

* fix(auth): extend CWE-203 normalization to policy.rs sandbox handlers

GetSandboxConfig and GetSandboxLogs in policy.rs had the same
fetch-before-authorize pattern that leaked cross-workspace sandbox
existence. Promote fetch_and_authorize_sandbox to pub(super) and
use it from both sandbox.rs and policy.rs handlers.

Signed-off-by: Derek Carr <decarr@redhat.com>

* test(auth): update assertions for CWE-203 sandbox ID normalization

Cross-workspace sandbox access via ID-based handlers now returns
NOT_FOUND instead of PERMISSION_DENIED to prevent existence inference.
Update the unit test and OIDC e2e assertion to match.

Signed-off-by: Derek Carr <decarr@redhat.com>

* fix(auth): narrow CWE-203 error mapping and correct whoami output formats

Only remap PERMISSION_DENIED to NOT_FOUND in fetch_and_authorize_sandbox
and RevokeSshSession, letting INTERNAL and UNAUTHENTICATED propagate
as-is. Fix whoami --output format values in cli-reference.md to match
the actual CLI (table/json/yaml, not text/json).

Signed-off-by: Derek Carr <decarr@redhat.com>

* fix(ci): share network namespace with Keycloak in containerized CI

In GitHub Actions job containers, Docker port publishing lands on the
host, not inside the job container. Detect this environment and attach
Keycloak to the job container's network namespace instead, with
hardened defaults (cap-drop ALL, no-new-privileges, loopback-only
listener).

Signed-off-by: Derek Carr <decarr@redhat.com>

---------

Signed-off-by: Derek Carr <decarr@redhat.com>
2026-07-30 00:31:32 +00:00
Evan LezarandDrew Newberry 662dee68eb refactor(compute): make sandbox readiness gateway-owned across all drivers (#2153)
* refactor(compute): extract create_sandbox_record and update_sandbox_record helpers

Split apply_sandbox_update_locked into named helpers to make the two
distinct paths explicit: create_sandbox_record for first-observation events
and update_sandbox_record for subsequent driver snapshots on existing sandboxes.
The dispatcher now uses a match on the existing record rather than an
early-return guard.

No behavior change.

Signed-off-by: Evan Lezar <elezar@nvidia.com>

* refactor(compute): make sandbox readiness gateway-owned across all drivers

Introduce compute_phase_components and apply_readiness_conditions to
centralise the gateway's phase composition logic. The public SandboxPhase
is now determined by combining the backend phase reported by the driver with
supervisor session presence, independent of the driver implementation.

Remove SupervisorReadiness from the driver contract. Running containers
always report BackendReady; the gateway owns the Ready decision. Rename
the dispatcher match to three arms so that status-less events for existing
sandboxes are a documented no-op rather than a silent pass-through.

Drop backend_ready_no_session and the SupervisorNotConnected condition.
The BackendReady driver condition plus the Provisioning phase already
communicates that the backend is up but the supervisor has not connected.
The redundant condition added noise without new information.

Closes #1951

Signed-off-by: Evan Lezar <elezar@nvidia.com>

* fix(compute): expose disconnected supervisor readiness

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* docs(sandboxes): clarify supervisor readiness lifecycle

Signed-off-by: Evan Lezar <elezar@nvidia.com>

---------

Signed-off-by: Evan Lezar <elezar@nvidia.com>
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
Co-authored-by: Drew Newberry <anewberry@nvidia.com>
2026-07-29 10:10:53 +00:00
Matthew Grossman bc14018cad feat(sandbox): use policy-first OCI image identity (#2509)
* feat(sandbox): use policy-first OCI image identity

Closes #2331

Preserve per-field policy omission, derive Docker and Podman fallbacks from the inspected immutable image, and resolve the final numeric identity before starting agent children.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

* fix(sandbox): preserve declared process identities

Keep explicit policy values and OCI-declared names intact, defer passwd lookup until a primary GID is required, and refresh stale policy examples.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

* fix(supervisor): reuse resolved OCI identity

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

* fix(supervisor): allow Linux pre-exec arguments

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

* fix(kubernetes): protect resolved sandbox identity

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

* fix(sandbox): prepare workspace for OCI identity

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

* refactor(sandbox): own only workspace root

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

* fix(sandbox): harden partial identity drops

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

* test(sandbox): scope OCI image e2e to Docker

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

* fix(sandbox): narrow OCI identity fallback scope

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

* test(podman): cover OCI identity launch

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

* fix(podman): exercise OCI fallback in E2E

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

---------

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
2026-07-29 05:27:21 +00:00
LR90 7955c8309b feat(k8s): support configuring workspace PVC storageClassName (#2463)
The Kubernetes driver's default workspace PVC never set storageClassName,
so on clusters with no default StorageClass the PVC stayed Pending and
sandbox creation failed.

Add a workspace_storage_class option to KubernetesComputeConfig, wired
through SandboxPodParams into the generated volumeClaimTemplates. When
non-empty it sets storageClassName; empty preserves the current behavior
of relying on the cluster default StorageClass.

Expose it via the OPENSHELL_K8S_WORKSPACE_STORAGE_CLASS env var on both
the standalone driver and the embedded gateway runtime defaults, and via
the server.workspaceStorageClass Helm value.

Closes #2442

Signed-off-by: lr90 <qiuweimin@matrixorigin.cn>
2026-07-29 03:35:00 +00:00
Derek Carr b78c8615e3 fix(server): stabilize flaky delete telemetry unit test (#2521)
Replace yield_now() spin-loop with sleep(10ms) polling in
delete_handler_ends_telemetry_for_the_resolved_sandbox_id. The
single-threaded tokio runtime starves the spawn_blocking threads
used by SQLite when yield_now() burns 100% CPU waiting for the
delete gate entry count. Increase the timeout from 1s to 5s for
consistency with similar guard tests.

Signed-off-by: Derek Carr <decarr@redhat.com>
2026-07-28 15:56:41 +00:00
Evan Lezar f00ad23a26 fix(podman): tolerate shutdown transport closes (#2498)
Signed-off-by: Evan Lezar <elezar@nvidia.com>
2026-07-28 08:00:37 +00:00
Evan Lezar 39bf94e528 fix(server): bind gateway listeners before sandbox resume (#2495)
* refactor(server): extract gateway listener binding

Signed-off-by: Evan Lezar <elezar@nvidia.com>

* fix(server): bind gateway listeners before sandbox resume

Signed-off-by: Evan Lezar <elezar@nvidia.com>

---------

Signed-off-by: Evan Lezar <elezar@nvidia.com>
2026-07-27 15:02:04 +00:00
Andrew White 76a5397e6f fix: assorted arithmetic and indexing robustness fixes (#2451)
* fix(core): avoid i64 underflow on expired GCP token

When a cached GCP access token was already expired (expires_at_ms < now),
the remaining lifetime calculation used plain subtraction:
(expires_at_ms - now) / 1000. This underflows in debug builds and wraps
in release builds.

Use saturating_sub so an expired token simply yields a non-positive
remaining lifetime and is skipped. Add a regression test.

Signed-off-by: Andrew White <andrewh@cdw.com>

* fix(server): avoid i64 underflow in lease age calculation

The lease steal path computed age as now_ms() - record.updated_at_ms.
If the stored updated_at timestamp is in the future (e.g. clock skew),
the subtraction underflows. Use saturating_sub to treat a future
timestamp as age 0, keeping the lease considered fresh.

Signed-off-by: Andrew White <andrewh@cdw.com>

* fix(server): avoid i64 overflow on large credential max lifetime

max_lifetime_seconds is only validated as >= 0. The code multiplied the
raw i64 value by 1000 to compute max_expires, which overflows for
values above i64::MAX / 1000. The capped i32 value used for the STS
request was already computed but not reused.

Compute max_lifetime_ms from the capped value with saturating_mul and
use it for both expires_at fallback and max_expires.

Signed-off-by: Andrew White <andrewh@cdw.com>

* fix(server): avoid hardcoded indexing into base_url_config_keys

resolve_vertex_ai_route assumed every inference provider profile has at
least two base URL config keys by indexing [0] and [1]. Only the Vertex
profile satisfies that today; other profiles would panic here.

Iterate base_url_config_keys with find_map instead, and add a test with
an empty key list to ensure no panic.

Signed-off-by: Andrew White <andrewh@cdw.com>

* fix(server): clamp lease age at zero for future timestamps

Address review feedback: add regression coverage for future-timestamp
lease clock skew. Extract lease_is_expired() and clamp the age at zero —
i64::saturating_sub saturates at i64::MIN, not zero, so a future
updated_at_ms must be clamped explicitly to be treated as age zero and
remain unstealable with a positive TTL.

Signed-off-by: Andrew White <andrewh@cdw.com>

* fix(server): fall back past blank preferred Vertex base URL

Address review feedback: find_map stopped at the first present key even
when its value was blank, and the outer filter discarded it without
considering lower-priority aliases. Filter blank values inside the
closure so a blank preferred key falls back to a valid alias, matching
the documented priority order. Add a regression test with a blank
preferred key and a valid fallback.

Signed-off-by: Andrew White <andrewh@cdw.com>

* fix(server): saturate AWS STS expiry cap on saturated clock

Address review feedback: now_ms + max_lifetime_ms could still overflow
when current_time_ms() saturates near i64::MAX. Compute
max_expires = now_ms.saturating_add(max_lifetime_ms) once and use it for
both the conversion fallback and the expiry cap. Add a boundary unit
test covering a saturated clock.

Signed-off-by: Andrew White <andrewh@cdw.com>

---------

Signed-off-by: Andrew White <andrewh@cdw.com>
2026-07-27 14:39:40 +00:00
Grace Smith deced8716b refactor(policy): extract shared L7 endpoint validation (#2389)
Move L7 endpoint semantic checks into the openshell-policy crate so both
profile lint and the runtime validator share one implementation. This
eliminates drift between the two validation paths.

The shared validator covers 9 checks: unknown protocol, rules/access
mutual exclusivity, JSON-RPC family access rejection, json-rpc requires
rules, non-JSON-RPC protocol requires rules or access, MCP requires
rules when allow_all is false, rules-would-deny-all detection,
deny_rules require protocol, and deny_rules require base allow set.

Changes rules/deny_rules fields to Option<Vec<...>> so absent vs empty
is distinguishable at lint time. Adds is_effectively_empty() to
L7AllowProfile for deny-all detection of allow: {} objects. Makes
rules_would_deny_all MCP-aware by checking tool/params.name selectors
before classifying a rule as deny-all. Adds params field to
L7AllowProfile so MCP tool selectors survive proto round-trip.

Signed-off-by: Grace Smith <gsmith@redhat.com>
Signed-off-by: Grace Smith <grasmith@redhat.com>
2026-07-24 17:48:24 +00:00
Piotr MlocekandEvan Lezar f7cd910829 fix: eliminate parallel Rust test flakes (#2434)
* fix: eliminate parallel Rust test flakes

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>

* refactor(supervisor-network): name workload proxy TCP connection

Signed-off-by: Evan Lezar <elezar@nvidia.com>

* refactor(supervisor-network): name procfs address family

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>

---------

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
Signed-off-by: Evan Lezar <elezar@nvidia.com>
Co-authored-by: Evan Lezar <elezar@nvidia.com>
2026-07-24 15:47:44 +00:00
alangou 1d4ac708f1 fix(policy): keep internal allowed IP proposals pending (#2416)
Signed-off-by: Adrien Langou <alangou@nvidia.com>
2026-07-23 12:17:21 +00:00
Piotr Mlocek cbdeb4d537 fix(server): prevent unrelated sandbox deletes from blocking deletion events (#2340)
* fix(server): unblock sandbox deletion events

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>

* refactor(server): simplify sandbox settings cleanup

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>

* refactor(server): make delete lock ordering explicit

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>

* fix(server): bind delete worker to stable sandbox id

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>

* fix(server): disarm canceled queued deletes

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>

* test: harden sandbox lifecycle cleanup coverage

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>

---------

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
2026-07-22 12:02:37 +00:00
Mesut Oezdil 744a65d52e fix(driver-podman): avoid panic when HOME is unset on macOS (#2327)
* fix(driver-podman): resolve Podman socket via auto-detection

Align Podman socket selection with the existing Docker model: explicit
config wins, otherwise probe openshell-core for a responsive socket.
This also fixes the original HOME-unset panic, since resolution no
longer hardcodes a per-OS default path.

- add detect_podman_socket() in openshell-core, mirroring
  detect_docker_socket
- PodmanComputeConfig.socket_path is now Option<PathBuf>, no default
- remove default_socket_path() (podman driver) and podman_socket_path()
  (vm driver), both replaced by the shared detector
- update server env override and CLI for the new Option type
- add tests: responsive-candidate detection in openshell-core, and
  config-error (not panic) when no socket is configured or reachable

* fix(driver-podman): address review feedback on socket resolution

Extract socket resolution into resolve_socket_path, taking the
detector as a parameter so tests do not depend on real env vars or
the host's actual Podman state. Replace the flaky env-mutating test
with three deterministic cases: explicit wins, detected is used when
absent, and neither source errors.

Fix the socket_path doc comment to describe it from a config user's
point of view, matching DockerComputeConfig's docstring. Drop a
comment that only made sense next to the Docker driver code.

Update the Podman README and gateway docs: they described a fixed
per-OS default path that no longer exists, replace with the actual
probe-then-fail behavior.

* docs(driver-podman): simplify socket default description

Previous wording was self-contradictory (says auto-detect on unset,
then lists the same var as a probed candidate) and omitted the Linux
/run/user/uid/podman/podman.sock candidate.
2026-07-21 14:01:19 +00:00
Derek Carr 5952a5a23f feat(workspace): add workspace resource model with scoping, membershi… (#2243)
* feat(workspace): implement workspace model (Phase 1 of RFC 0011)

Implements workspace and membership model providing hard isolation
boundaries for multi-player OpenShell deployments.

Workspace CRUD with Kubernetes-style Terminating phase for graceful
deletion. All resources scoped by workspace via ObjectMeta. Membership
RPCs for workspace access control. Persistence migration shifts name
uniqueness to (object_type, workspace, name). Provider profiles support
platform and workspace scoping. Service routing uses workspace-prefixed
DNS labels. Inference routes renamed and workspace-scoped with
DeleteInferenceRoute RPC. Python SDK with WorkspaceClient, two-method
list pattern (workspace-scoped and for_all_workspaces), and workspace
parameter on all methods. CLI workspace flags, TUI workspace cycling.
K8s driver filters unmanaged CRs and uses delete preconditions. Podman
driver uses immutable container IDs. Label serialization fixed across
all put_if call sites.

Signed-off-by: Derek Carr <decarr@redhat.com>

* fix(cli): delegate sandbox upload command to existing upload function

The standalone `sandbox upload` command reimplemented upload logic
inline with two bugs: it used `Path::exists()` which follows symlinks
(rejecting dangling symlinks), and it ran git-aware filtering on
symlink sources. The `run::sandbox_upload()` function already handles
both cases correctly via `sandbox_upload_plan()`. Replace the inline
logic with a call to the existing function.

Signed-off-by: Derek Carr <decarr@redhat.com>

* fix(e2e): shorten sandbox names and fix test compatibility

Shorten the sandbox name in initial_sparse_policy_is_acknowledged_as_loaded
from 'e2e-2159-sparse-enrich' (22 chars) to 'e2e-sparse-enrich' (17 chars)
to comply with MAX_ROUTABLE_NAME_LEN (19 chars).

Also capture stderr in create_keep_with_args so future sandbox creation
failures include the actual CLI error instead of reporting empty output.

Signed-off-by: Derek Carr <decarr@redhat.com>

* test(workspace): add test coverage for workspace CRUD and persistence isolation

Add unit tests for workspace create happy path, get round-trip, get
not-found, get empty-name rejection, already-exists error, and
resolve_workspace not-found. Add persistence test proving cross-workspace
name uniqueness (same name in different workspaces produces separate
records). Add workspace name max-length boundary tests. Fix e2e harness
to include stderr in name-parse-failure error path. Align Python e2e
test_workspace_crud with try/finally pattern. Document provider profile
catalog workspace scoping gap in RFC 0011.

Signed-off-by: Derek Carr <decarr@redhat.com>

* fix(examples): update examples for workspace model compatibility

Shorten sandbox names in demo scripts to fit the 19-character
MAX_ROUTABLE_NAME_LEN limit: policy-demo prefix to pd-, multi-agent
notepad derives a short SANDBOX_TAG from the run ID, governance
interceptor uses gs-PID-RANDOM. Update vscode-remote-sandbox.md SSH
host aliases from openshell-{name} to openshell-{name}.{workspace}
format.

Signed-off-by: Derek Carr <decarr@redhat.com>

* feat(sdk): add workspace-scoped client and workspace CRUD

Add WorkspaceScopedClient modeled after kube::Api::namespaced — captures
workspace once and injects it into every sandbox request. Add workspace
CRUD methods (create, get, list, delete) and list_sandboxes_all_workspaces
on OpenShellClient. Extend SandboxRef with workspace field and add
WorkspaceRef type. Include mock tests for all new operations.

Signed-off-by: Derek Carr <decarr@redhat.com>

* fix(lint): resolve clippy warnings in workspace test assertions

Signed-off-by: Derek Carr <decarr@redhat.com>

* fix(docs): convert indented code blocks to fenced in RFC 0011

Signed-off-by: Derek Carr <decarr@redhat.com>

* fix(lint): resolve clippy warnings and apply cargo fmt across workspace

Auto-format with cargo fmt and fix clippy warnings exposed by the
reformat: unnecessary qualifications, map_unwrap_or, identical match
arms, unused variable prefix, dead code annotations, and let-unit-value
in e2e harness.

Signed-off-by: Derek Carr <decarr@redhat.com>

* fix(workspace): address workspace scoping issues from review

- Add workspace field to settings JSON output (CLI)
- Skip Podman containers missing workspace label instead of defaulting
  to empty string, matching K8s driver behavior
- Add resource_version to list_by_scope SELECT in both SQLite and
  Postgres backends, with regression test
- Gate PolicyLocalContext proposal/lookup routes on workspace readiness,
  returning 503 when workspace is not yet discovered
- Block sandbox and provider creation in TUI all-workspaces mode
- Clear workspace vectors in TUI reset_sandbox_state

Signed-off-by: Derek Carr <decarr@redhat.com>

* fix(workspace): make provider profile catalog workspace-aware

Thread workspace through snapshot_catalog so the
EffectiveProviderProfileCatalog enforces workspace boundaries on both
read and write paths. UserProviderProfileSource now loads platform-scoped
profiles (workspace "") plus the target workspace's profiles, preventing
cross-workspace duplicate profile ID collisions that previously caused
global catalog failures.

Update RFC 0011 to reflect catalog scoping is implemented in Phase 1
rather than deferred to future work.

Signed-off-by: Derek Carr <decarr@redhat.com>

* fix(persistence): include workspace column in atomic policy revision INSERT

put_policy_revision_atomic omitted the workspace column from the INSERT
into the objects table in both SQLite and Postgres backends, causing
atomically-written policy revisions to lose their workspace association.
Add workspace field to AtomicPolicyRevisionWrite and thread it through
both backend INSERT statements, matching the non-atomic put_policy_revision
path which already included it.

Signed-off-by: Derek Carr <decarr@redhat.com>

* fix(proxy): skip ancestor walk when socket owner is the entrypoint

collect_ancestor_identities walked the entire process tree above the
entrypoint when the connecting process was the entrypoint itself,
SHA256-hashing every ancestor binary (IDE, shell, container runtime).
On dev machines with large binaries in the ancestor chain this exceeded
the 30-second test timeout. When start_pid == stop_pid there are no
intermediate ancestors to verify, so return an empty list immediately.

Signed-off-by: Derek Carr <decarr@redhat.com>

* fix(workspace): make provider profile catalog scope-aware

Allow the same profile ID at platform and workspace scopes by
introducing layered catalog entries where workspace profiles shadow
platform profiles. Add source and scope fields to the ProviderProfile
proto and CLI output. Migrate List/Get handlers to the catalog,
fixing divergence with runtime profile resolution.

Signed-off-by: Derek Carr <decarr@redhat.com>

* fix(e2e): align podman e2e labels with centralized driver constants

The podman driver moved its container labels to the centralized
openshell.ai/ prefix, but the e2e test harness and cleanup script
still referenced the old openshell.sandbox-* keys, causing the
local_driver_token_restart test to fail on container lookup.

Signed-off-by: Derek Carr <decarr@redhat.com>

* fix(e2e): align python profile isolation test with scope-aware catalog

Platform profiles are now visible in workspace listings as fallbacks
per the layered catalog design. Update the assertion to match.

Signed-off-by: Derek Carr <decarr@redhat.com>

* fix(workspace): honor profile_workspace in runtime profile resolution

Runtime profile lookups now consult provider.profile_workspace via
get_type_profile_for_scope. Providers created with --global-profile
(profile_workspace="") resolve to the platform profile even when a
workspace profile shadows the same ID. All 6 runtime call sites
updated; type-only call sites remain scope-agnostic.

Signed-off-by: Derek Carr <decarr@redhat.com>

---------

Signed-off-by: Derek Carr <decarr@redhat.com>
2026-07-21 00:59:18 +00:00
Russell Bryant 9377e0d5fe fix(providers): allow git clone/fetch via default GitHub provider (#2317)
* fix(providers): allow git clone/fetch via default GitHub provider

The github.com:443 git-transport endpoint used the read-only access
preset, which expands to GET/HEAD/OPTIONS only. Git smart HTTP requires
a POST to */git-upload-pack for clone and fetch, so the L7 proxy denied
those operations and `gh repo clone` / `git clone https://...` failed.

Replace the preset with explicit rules that permit the read-only methods
plus POST */git-upload-pack, so clone/fetch work while push
(git-receive-pack) stays blocked. Enabling push still requires an
explicit policy proposal.

Why allowing this POST is still read-only: in git's smart HTTP protocol
POST is an RPC transport, not a write. A clone/fetch does GET
*/info/refs (ref discovery) followed by POST */git-upload-pack, whose
body is only the client's want/have negotiation; the server responds
with a packfile and nothing on the server is modified (data flows
server -> client). The service names are from the server's perspective:
git-upload-pack = the server uploads a pack to the client (a read/
download), while git-receive-pack = the server receives a pack from the
client (the actual write/push). The new rule is scoped to
*/git-upload-pack only, so push (git-receive-pack) and arbitrary POSTs
to github.com remain denied.

Add a provider-profile regression test and a rego enforcement test
covering ref discovery, upload-pack (allowed), and receive-pack (denied).

Closes #1769

Signed-off-by: Russell Bryant <rbryant@redhat.com>

* test(providers): strengthen git-transport regression and add clone e2e

Pin the exact allowed rule set for the built-in github git-transport
endpoint in both the provider-profile and composed-policy tests, so a
broader or additional POST rule (e.g. POST **) that could enable push
via git-receive-pack fails the test instead of passing a substring
check. Add an e2e test that attaches the built-in github provider and
clones a public repo over HTTPS, exercising provider attachment,
effective-policy composition, TLS interception, and real git behavior.

Update the Providers V2 docs so the github.com git-transport endpoint
shows explicit clone/fetch rules instead of the stale read-only preset.

Refs #1769

Signed-off-by: Russell Bryant <rbryant@redhat.com>

* test(providers): isolate providers_v2 mutation in clone e2e

The clone e2e enables the gateway-global providers_v2_enabled setting.
Restore its exact prior value (or absence) captured via GetGatewayConfig
instead of unconditionally deleting it, and serialize the mutation
across xdist workers with an exclusive file lock on the run's shared
base temp dir, so a shared or pre-configured gateway is left untouched
and parallel workers cannot race the read-modify-restore.

Refs #1769

Signed-off-by: Russell Bryant <rbryant@redhat.com>

* test(providers): serialize providers_v2 mutation with a suite-wide guard

The clone e2e's per-fixture lock only coordinated fixtures that acquired
it; other xdist workers hit the same gateway without it and could
observe the transiently-enabled providers_v2_enabled global during their
own sandbox creation (CWE-362).

Add an autouse readers-writer guard in conftest: every test holds a
shared lock on the gateway config, and a test marked
exclusive_gateway_config holds an exclusive lock. Mark the clone test
exclusive so no other worker is mid-test while it enables and restores
the gateway-global setting. Exact prior-value restoration is retained.

Refs #1769

Signed-off-by: Russell Bryant <rbryant@redhat.com>

---------

Signed-off-by: Russell Bryant <rbryant@redhat.com>
2026-07-20 21:07:17 +00:00
emonq a2cd5f8eda fix(gateway): honor tty flag for interactive exec (#2315)
* fix(gateway): honor tty flag for interactive exec

Pass the requested TTY mode through the interactive SSH relay.
Skip PTY allocation and resize forwarding when TTY is disabled,
and add regression coverage for both modes.

Signed-off-by: emonq <emonq@outlook.com>

* test(gateway): improve `test_sandbox_interactive_exec_honors_tty` to test streamed stdin and stdout/stderr

Signed-off-by: emonq <emonq@outlook.com>

---------

Signed-off-by: emonq <emonq@outlook.com>
2026-07-20 17:47:37 +00:00
Piotr Mlocek d556748771 feat(supervisor-middleware): add network egress middleware (#2027)
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
2026-07-16 17:47:49 -07:00
Russell Bryant aa483ecb9a feat(providers): AWS STS AssumeRole refresh strategy and aws-s3 profile (#1782)
Add gateway-managed AWS STS credential refresh (provider-v2, #1576). The
gateway calls sts:AssumeRole and writes three short-lived credentials
(AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN) to the
provider record; the proxy re-signs requests with SigV4. Adds the aws and
aws-s3 provider profiles and a declarative multi-output refresh model
(additional_outputs) so one AssumeRole co-mints all three credentials.

Signed-off-by: Russell Bryant <rbryant@redhat.com>
2026-07-16 13:28:01 -07:00
krishicks cf4deccd37 fix(gateway): probe Docker socket during driver auto-detection (#2303)
Previously, Docker was auto-detected when the CLI was installed or a candidate
Unix socket existed. Neither check verified that the Docker API was responsive.
A similar check was done when auto-detecting Podman in the past, but was
replaced in 1f07bf04 with a probe of candidate Podman sockets instead.

This change applies the functional API probing approach introduced for Podman
in 1f07bf04 to Docker. It also makes Docker driver initialization use the same
socket-selection mechanism as Docker auto-detection instead of Bollard’s local
defaults. This means the previously auto-detectable Docker socket paths
$HOME/.docker/run/docker.sock and $XDG_RUNTIME_DIR/docker.sock will actually be
usable.

When no working compute driver can be auto-detected, the gateway exits early
with a message saying as much:

> configuration error: no compute driver configured and auto-detection found no
> suitable driver; set --drivers or OPENSHELL_DRIVERS to kubernetes, podman,
> docker, or vm

This makes for a better user experience when installing OpenShell without an
available supported compute driver.

Signed-off-by: Kris Hicks <khicks@nvidia.com>
2026-07-16 10:02:24 -07:00
Matthew Grossman 077adb7907 fix(server): persist sandbox labels on create (#2306)
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
2026-07-16 09:13:23 -07:00
Evan Lezar 21aaa89522 feat(gateway): add elevated gateway info (#2202)
* feat(cli)!: fold gateway metadata into list

BREAKING CHANGE: openshell gateway info no longer shows local gateway registration metadata. Use openshell gateway list or openshell gateway list -o json for local registration details.

Signed-off-by: Evan Lezar <elezar@nvidia.com>

* feat(gateway): add elevated gateway info

Signed-off-by: Evan Lezar <elezar@nvidia.com>

---------

Signed-off-by: Evan Lezar <elezar@nvidia.com>
2026-07-15 22:04:57 +00:00
Drew Newberry 83003e80fc feat(interceptors): initial gateway interceptor implementation and reference example (#2005)
* feat(gateway): add descriptor-driven interceptors

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* feat(gateway): add service-reflected interceptors

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* wip

* fix(gateway): harden interceptor evaluation

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* feat(interceptors): label metrics and harden governance smoke

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* remove on_error: ignore

* feat(gateway-interceptors): emit log annotations

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* feat(examples): govern provider profiles in interceptor

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(gateway): preserve update config annotations

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* feat(providers): support interceptor profile catalogs

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* wip

* feat(governance-interceptor): sign provider profiles

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(providers): use configured profile sources for refresh updates

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* feat(gateway-interceptors): add phase-specific evaluation payloads

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* feat(providers): compose provider profile sources

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(gateway-interceptors): preserve committed responses

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(gateway-interceptors): reject ambiguous protobuf oneofs

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(gateway-interceptors): validate patch candidates per binding

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* refactor(gateway-interceptors): use reflected protobuf codec

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(governance-example): canonicalize signed protobuf hashes

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(gateway): commit policy provenance atomically

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(gateway): close signed governance bypasses

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(gateway): isolate interceptor secrets and authority

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(gateway): snapshot provider profiles per request

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* chore(gateway): resolve server clippy warnings

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(server): satisfy provider source clippy lint

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(server): initialize policy test annotations

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(gateway-interceptors): require explicit route allowlist

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* docs(proto): clarify update annotation semantics

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

---------

Signed-off-by: Drew Newberry <anewberry@nvidia.com>
2026-07-14 20:58:36 -07:00
Tinson Lai e3d26dd3ae fix(policy): keep approved chunk when a mechanistic denial resubmits its endpoint (#2242)
* fix(policy): keep approved chunk when a mechanistic denial resubmits its endpoint

A mechanistic denial flush for an endpoint already covered by an
auto-approved mechanistic chunk flipped that chunk approved -> rejected
with no human action. The dedup upsert in put_draft_chunk returns the
existing row's id, which aliases onto the approved chunk; the self-reject
scan then matched the row against itself and rejected it, while the merged
rule stayed enforced — the governance ledger disagreed with the live policy.

Guard self_reject_mechanistic_if_already_covered to act only on a still
pending effective chunk, and exclude the incoming id from the covering
scan. Add a regression test and document the dedup/self-reject invariant.

Fixes #2165
Refs NVIDIA/NemoClaw#6329

Signed-off-by: Tinson Lai <tinsonl@nvidia.com>

* fix(policy): reject mechanistic chunk via atomic pending compare-and-set

The self-reject-when-covered path read a chunk, confirmed it was pending,
then issued an unconditional status update to rejected. An approval that
committed between the read and the write flipped an already-approved chunk
to rejected while its rule stayed merged, recreating the ledger/enforcement
mismatch through a concurrent path.

Add conditionally_reject_draft_chunk to the policy store: the pending->rejected
transition carries a status = 'pending' predicate on the final write and
reports whether a row changed. Zero changed rows is a benign no-op, meaning
another operation already decided the chunk. Implemented for both SQLite and
PostgreSQL. The pending pre-read and the self-exclusion guard stay as
defense-in-depth. Adds persistence-level regression tests proving an approved
row cannot be conditionally rejected and that an approval racing the reject
wins.

Signed-off-by: Tinson Lai <tinsonl@nvidia.com>

---------

Signed-off-by: Tinson Lai <tinsonl@nvidia.com>
2026-07-13 16:30:32 -07:00
Grace Smith 4e1ffef84b fix(certgen): stage temp dir inside output dir to fix cross-device rename (#2241)
generate-certs stages temp files beside --output-dir using
dir.with_file_name(). When --output-dir is a container or WSL volume
mount, the staging dir lands on a different filesystem and
std::fs::rename fails with EXDEV.

Move staging inside the output dir so the rename always stays on the
same filesystem, preserving atomic replacement.

Fixes #2173

Signed-off-by: Grace Smith <grasmith@redhat.com>
2026-07-13 09:20:25 -07:00
Adel Zaalouk bb72d0123c fix(server): allow newlines in exec command arguments (#1965) 2026-07-12 17:28:57 -07:00
Evan Lezar 45614a3fb6 refactor(api): remove SandboxTemplate.volume_claim_templates (#2088)
The field was added during the Kubernetes driver extraction refactor
(#817) as a pass-through mechanism, but was never wired up to a CLI
flag, Python SDK helper, or any documentation. The only reachable
user path was raw gRPC construction.

The Kubernetes driver now always injects the default workspace PVC,
removing the branching logic that checked for a user-supplied VCT.
Field number 9 is reserved in the proto to prevent reuse.

Signed-off-by: Evan Lezar <elezar@nvidia.com>
2026-07-01 16:04:47 +02:00
Evan Lezar 0a25fdf525 refactor(core): remove unused extra bind addresses (#2059)
Signed-off-by: Evan Lezar <elezar@nvidia.com>
2026-06-30 10:37:10 -07:00
Evan Lezar a5161d0bc3 refactor(server): normalize compute driver config acquisition (#1974)
* refactor(server): remove unused compute runtime constructor parameter

Signed-off-by: Evan Lezar <elezar@nvidia.com>

* refactor(server): normalize compute driver type imports

Signed-off-by: Evan Lezar <elezar@nvidia.com>

* refactor(server): key driver config tables by name

Signed-off-by: Evan Lezar <elezar@nvidia.com>

* refactor(server): normalize compute driver config acquisition

Signed-off-by: Evan Lezar <elezar@nvidia.com>

---------

Signed-off-by: Evan Lezar <elezar@nvidia.com>
2026-06-29 18:59:08 +02:00
krishicksandddurst 7bce1223dc feat(policy): add JSON-RPC and MCP L7 policies (#1865)
Add policy schema, proto, provider profile, OPA, and L7 proxy support for
`protocol: json-rpc` and `protocol: mcp`. Generic JSON-RPC endpoints match
exact method names only, with `method: "*"` as the all-method sentinel;
wildcard/glob methods and params matchers are rejected.

Parse JSON-RPC request bodies and batches in the forward proxy, deny
response-shaped client frames, limit receive-stream GET allowance to MCP
endpoints, and redact params in decision logs. Preserve L7 rule params on the
proto load path so MCP `tools/call` tool filters behave like YAML-loaded
policies.

Add MCP conformance coverage, JSON-RPC L7 e2e coverage, and docs for the new
protocols and current matcher limitations.

Signed-off-by: Kris Hicks <khicks@nvidia.com>
Co-authored-by: ddurst <267424412+ddurst-nvidia@users.noreply.github.com>
2026-06-26 15:52:16 -07:00
Gal Zaidman 45e5a5d128 fix(server): prevent exec relays from hanging on idle connections (#1992)
Add HTTP/2 keepalive on supervisor multiplex connections so half-dead
sessions cannot leave in-flight exec relays parked indefinitely. Configure
SSH keepalive on exec relay clients so long silent commands are not timed
out on stdout idle alone; wedged or orphaned relays fail after missed
keepalives instead.

After a command reports exit status, bound how long the gateway waits for
the trailing channel close. Return UNAVAILABLE when a relay closes before
reporting exit status rather than defaulting to exit code 1.

Signed-off-by: Gal Zaidman <gzaidman@nvidia.com>
2026-06-26 13:33:32 -07:00
Taylor Mutch ba21bb32a2 feat(kubernetes): support agent-sandbox v1beta1 (#2009)
* feat(kubernetes): support agent-sandbox v1beta1

Signed-off-by: Taylor Mutch <taylormutch@gmail.com>

* ci(kubernetes): test agent-sandbox api versions

Signed-off-by: Taylor Mutch <taylormutch@gmail.com>

* fix(kubernetes): retry agent-sandbox raw 404s

Signed-off-by: Taylor Mutch <taylormutch@gmail.com>

* fix(kubernetes): harden agent-sandbox api setup

Signed-off-by: Taylor Mutch <taylormutch@gmail.com>

* fix(kubernetes): cache agent-sandbox api version

Signed-off-by: Taylor Mutch <taylormutch@gmail.com>

* docs(kubernetes): document agent-sandbox upgrade behavior

Signed-off-by: Taylor Mutch <taylormutch@gmail.com>

* refactor(kubernetes): collapse agent-sandbox api selection

Signed-off-by: Taylor Mutch <taylormutch@gmail.com>

---------

Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
2026-06-26 11:44:52 -07:00
John T. Myers b855d8d872 fix(policy): reserve provider rule namespace (#1991)
Closes #1982

Reject user-authored _provider_* network policy keys at gateway boundaries, strip provider-derived rules from sandbox sync, and document the round-trip workflow.

Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>
2026-06-26 08:28:35 -07:00