mirror of
https://github.com/NVIDIA/OpenShell.git
synced 2026-10-16 05:20:22 +08:00
pull-request/2592
590
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 alangou
|
0c7e59a953 |
fix(deps): bump russh, jsonwebtoken, tar and npm lint deps (#2617)
Signed-off-by: Adrien Langou <alangou@nvidia.com> |
||
|
Artem Lytvyn
|
d2c44b0e53 |
fix(supervisor-middleware): configure HTTP/2 keepalive on middleware gRPC channel (#2608)
Signed-off-by: Artem Lytvyn <alytvyn@redhat.com> |
||
|
|
5548405fcb |
feat(credentials): add provider credential storage drivers (#2437)
* feat(credentials): add provider credential storage drivers Signed-off-by: Taylor Mutch <taylormutch@gmail.com> * fix(credentials): harden credential update handling Signed-off-by: Taylor Mutch <taylormutch@gmail.com> * fix(credentials): harden credential driver security, correctness, and performance Address review findings from the credential storage drivers PR: - Route additional_credentials through the driver on refresh to prevent silent data loss for multi-credential providers (e.g. AWS STS) - Clean up stored credential handles on CAS failure during refresh to prevent orphaned secrets in external backends - Enforce namespace validation in the Kubernetes Secrets driver to prevent cross-namespace credential access when allow_reference_namespace is not enabled - Cache Vault Kubernetes auth tokens with 80% TTL to avoid re-authenticating on every credential operation - Parallelize resolve_credentials in all three drivers using try_join_all for faster sandbox startup - Add existingSecret support for the KEK Secret to fix helm template/GitOps workflows where lookup returns empty and regenerates the key - Document RBAC blast radius for the Kubernetes Secrets credential driver and recommend a dedicated namespace Signed-off-by: Varsha Prasad <varshaprasad96@gmail.com> Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com> * fix(credentials): add optimistic concurrency, fix thundering herd, parallelize operations Use resourceVersion optimistic concurrency with retry loop for K8s Secret ownership checks to prevent TOCTOU races. Switch Vault token cache from RwLock to Mutex with double-check pattern to prevent thundering herd on cache miss. Parallelize credential store and delete operations across independent keys using try_join_all. Signed-off-by: Varsha Prasad <varshaprasad96@gmail.com> Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com> * fix(credentials): handle partial failures, add delete retry, consolidate cleanup Replace try_join_all with join_all in credential store/delete operations to handle partial failures — successfully-stored handles are cleaned up when another key fails. Add retry loop with conflict detection to db-credstore delete_credential, matching the K8s driver pattern. Consolidate 4 manual cleanup_pre_stored_provider_credentials call sites into a single error handler using an async block. Remove inconsistent .trim() from db-credstore validate_handle_owner. Signed-off-by: Varsha Prasad <varshaprasad96@gmail.com> Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com> * fix(credentials): fix retry loop guard and remove unprotected validation Remove attempt-count guard from 409/Aborted match arms in retry loops so the post-loop Status::aborted error is reachable after exhausting retries. Previously, last-attempt conflicts fell through to the catch-all error arm, producing misleading Status::unavailable errors. Remove duplicate validation calls that ran after prepare_provider_credential_update but outside the cleanup-protected async block, which would leak pre-stored handles on failure. Signed-off-by: Varsha Prasad <varshaprasad96@gmail.com> Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com> * fix(credentials): add workspace/provider UUID to credential backend paths Include workspace and provider ID in credential backend object paths to ensure cross-workspace uniqueness and prevent credential collision (GATOR-1806c9be-01). - Updated credential driver proto to include workspace and provider_id fields - Modified Vault driver to include workspace/provider_id in managed_secret_path - Modified Kubernetes Secrets driver to include workspace/provider_id in credential_owner_id and managed_secret_name - Updated all credential runtime calls to pass workspace/provider_id - Updated tests to use the new signatures This prevents two workspaces sharing the same external credential store from colliding on provider names, which was a critical security issue (CWE-639). * fix(credentials): preserve provider-level expiration for handle-backed credentials Compute effective expiration from both provider and driver values using the earliest non-zero timestamp and skip expired values before insertion (GATOR-1806c9be-02). - Modified resolve_provider_handles to check provider credential_expires_at_ms - Skip expired credentials during resolution instead of returning them - Use effective expiration (min of provider and driver) in resolution results - Fix inference.rs to preserve earliest expiration when merging This ensures handle-backed credentials respect the same expiration semantics as inline credentials. * fix(credentials): stage refresh changes under new handles before validation Stage credential replacements under new immutable handles instead of reusing existing handles to prevent overwriting committed values before validation/CAS (GATOR-1806c9be-03). - Stage credentials with empty existing_handles map to force new handle creation - Validate and CAS before the new values are committed to backend storage - Delete old handles only after successful CAS - On CAS failure, delete only the newly staged handles - This prevents CWE-362/CWE-367 race conditions where failed refreshes could still modify or delete the active credential The fix ensures that a rejected refresh cannot modify the backend object still referenced by the committed provider record. * fix(credentials): add timeouts to credential driver RPCs Apply configured timeouts to both startup capability negotiation and runtime RPCs to prevent indefinite hangs (GATOR-1806c9be-05). - Add DEFAULT_CREDENTIAL_DRIVER_RPC_TIMEOUT_SECS constant (30s) - Apply timeout to GetCapabilities during startup connection - Apply timeout to all runtime RPCs (store, delete, resolve) - Use tokio::time::timeout to bound the entire GetCapabilities operation during startup, not just the socket connection - Return contextual deadline errors on timeout This prevents a faulty or overloaded driver from hanging gateway operations indefinitely. * fix(credentials): fix test to use consistent workspace/provider identity The Kubernetes auth Vault resolve test was constructing a managed path with test-workspace/test-provider-id but sending default/prov-123 in the request, causing validation to reject the request (GATOR-18e32351-01). - Update test to use test-workspace and test-provider-id in the request to match the logical_path construction - This ensures the test exercises the intended code path and validates Kubernetes auth resolution properly The test now passes and correctly validates identity enforcement. * fix(credentials): use unique staging ID for refresh to avoid overwrites Stage refresh replacements under genuinely distinct immutable handles using a unique staging ID to prevent overwriting committed values (GATOR-1806c9be-03). - Generate a unique staging ID using UUID for each refresh operation - Use this staging ID when storing credentials instead of the real provider ID - Pass the same staging ID during cleanup on failure to delete only staged objects - This ensures deterministic paths (Vault) and object names (K8s) don't collide with the committed provider's credentials The fix prevents failed refreshes from silently replacing active credentials or breaking providers by deleting still-referenced backend objects. * fix(credentials): wrap credential driver RPCs in local timeouts Add local tokio::time::timeout wrappers around credential driver RPCs to bound non-compliant or stalled UDS peers (GATOR-1806c9be-05). - Wrap StoreCredential, DeleteCredential, and ResolveCredentials in local timeouts - Return contextual deadline_exceeded errors when timeouts occur - Keep existing gRPC timeout metadata for compliant implementations - GetCapabilities during startup was already wrapped in previous commit This ensures a faulty local driver cannot hang gateway operations indefinitely, even if it accepts the connection but never responds to the RPC. * fix(credentials): preserve ownership for staged refreshes Signed-off-by: Seth Jennings <sjenning@redhat.com> * fix(credentials): bound startup capability probe Signed-off-by: Seth Jennings <sjenning@redhat.com> * test(provider): authenticate credential handler requests Signed-off-by: Seth Jennings <sjenning@redhat.com> * fix(ci): grant actions read to credential driver e2e Signed-off-by: Seth Jennings <sjenning@redhat.com> --------- Signed-off-by: Taylor Mutch <taylormutch@gmail.com> Signed-off-by: Varsha Prasad <varshaprasad96@gmail.com> Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com> Signed-off-by: Seth Jennings <sjenning@redhat.com> Co-authored-by: Taylor Mutch <taylormutch@gmail.com> Co-authored-by: Varsha Prasad Narsing <varshaprasad96@gmail.com> |
||
|
Jim Meyer
|
8c7dd148a9 |
perf(net): set TCP_NODELAY on latency-sensitive TCP hops (#2220)
* perf(net): set TCP_NODELAY on all tunnel and proxy TCP hops The sandbox tunnel added ~44 ms of latency to every small request/response because no socket in the path disabled Nagle's algorithm, so sub-MSS writes waited on delayed ACKs at each hop. Set TCP_NODELAY on every latency-sensitive TCP socket: - gateway: accepted connections on the public listener (gRPC relay frames and WS tunnel writes) - CLI: edge tunnel local accept + underlying WebSocket TCP stream, insecure TLS connector (tonic's default connector already does this), and service-forward accepted sockets - supervisor: direct-tcpip connect into the sandbox netns, TCP relay target dials, egress proxy accepted connections, and all upstream CONNECT/HTTP dials (via a new connect_upstream helper) Setting TCP_NODELAY on connect is best-effort: a failure only costs latency, so we log and continue rather than fail the connection. The sandbox SSH transport rides a unix domain socket and gRPC client channels use tonic defaults (nodelay on), so no change is needed there. Fixes #2219 Signed-off-by: Jim Meyer <jim@meyer4hire.com> * refactor(net): house TCP_NODELAY helper in a shared net module Address review feedback on the TCP_NODELAY change: - Move the shared set-nodelay helper out of supervisor_session into a new crate-private `net` module in openshell-supervisor-process, so ssh and supervisor_session no longer reach across modules through a pub(crate) item. - Make the best-effort comments at each call site terse and consistent. No behavior change; the benchmark ladder reproduces the same numbers. Signed-off-by: Jim Meyer <jim@meyer4hire.com> * refactor(net): consolidate TCP_NODELAY helpers into openshell_core::net Move the best-effort TCP_NODELAY helpers into the shared openshell_core::net module so every crate dials and configures sockets the same way: - Add set_tcp_nodelay_best_effort (accepted/existing streams) and connect_tcp_nodelay_best_effort (dial + set) with unit tests. - Migrate all call sites in openshell-cli, openshell-server, and the supervisor crates to the shared helpers. - Remove the crate-private net module from openshell-supervisor-process. - Document socket guidance in AGENTS.md (Network Sockets). Signed-off-by: Jim Meyer <jim@meyer4hire.com> * perf(net): set TCP_NODELAY on exec bridge and metadata server The gateway-side single-use SSH-over-relay loopback bridge and the sandbox IMDS metadata server were missed latency-sensitive TCP hops. Set TCP_NODELAY on the accepted client connection and both russh client dials of the exec bridge — interactive keystrokes and line-buffered PTY output are the most tinygram-heavy traffic in the system — and on the metadata server's accepted connections. Also log unrecognized MaybeTlsStream variants in the edge tunnel so a future TLS-backend change surfaces a silent TCP_NODELAY miss instead of skipping it quietly. Signed-off-by: Jim Meyer <jim@meyer4hire.com> * perf(net): set TCP_NODELAY on openshell-sdk socket paths The openshell-sdk crate landed on main with its own copies of the CLI's hand-rolled sockets, which the CLI and TUI are meant to consume. Give them the same treatment as the CLI equivalents: - edge_tunnel: the accepted local tunnel connection and the WebSocket's underlying TCP socket (plain and rustls variants). - transport: the dial in InsecureTlsConnector, tonic's custom-connector path. Only these hand-rolled sockets need it. Tonic's own connector defaults tcp_nodelay to true and applies it itself, so plain Endpoint::connect callers were already covered. Signed-off-by: Jim Meyer <jim@meyer4hire.com> --------- Signed-off-by: Jim Meyer <jim@meyer4hire.com> |
||
|
krishicks
|
490f66f47a |
docs(cli): recommend providers for secrets (#2603)
Update the docs for `sandbox create` and `exec` to dissuade use of `--env` for secrets, and enhance the docs for `--provider` to explain what it's for. Signed-off-by: Kris Hicks <khicks@nvidia.com> |
||
|
Matthew Grossman
|
537805568d |
feat(sandbox): honor OCI image working directories (#2530)
* feat(sandbox): honor Docker OCI working directories Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * fix(sandbox): honor effective workspace access Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * test(sandbox): cover enforced workspace denial Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * docs(docker): explain effective workdir checks Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * fix(sandbox): validate effective workspace writes Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * fix(sandbox): reserve supervisor control roots Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * refactor(sandbox): centralize control paths Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * fix(sandbox): reserve OCI runtime mount roots Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> --------- Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> |
||
|
krishicks
|
4d55265f12 |
test(server): close traced handler before span assertion (#2604)
Keep the completed instrumented handler future in an explicit pinned box and drop it after the await. This releases the handler-side request span clone before the disconnect test checks producer ownership, avoiding compiler- and platform-dependent retention of an unfinished span. Signed-off-by: Kris Hicks <khicks@nvidia.com> |
||
|
Polite_realismandAdam Miller
|
0e9a44cfa9 |
feat(build): add system CA root mode (#2324)
* feat(build): add system CA root mode Allow distro builds to use native trust stores for supervisor upstream TLS while keeping bundled Mozilla roots as the default. Avoid bundled root crates in system-ca-roots builds by using native-root TLS features and z3 0.20. Signed-off-by: Adam Miller <admiller@redhat.com> * fix(build): keep CA root feature in telemetry-off verification The telemetry-off task uses --no-default-features which now disables bundled-ca-roots in addition to telemetry, triggering the compile_error guard. Re-enable bundled-ca-roots explicitly so the task verifies only telemetry compilation. Signed-off-by: Scott Burdine <sburdine@nvidia.com> Signed-off-by: politerealism <burdcat17@gmail.com> * fix(sdk): disable oauth2 default features to prevent webpki-roots leak The bare `oauth2 = "5"` dependency re-enabled default features (rustls-tls → reqwest/rustls-tls → webpki-roots), defeating the system-ca-roots feature gate. Mirror the CLI fix: disable defaults and enable only the `reqwest` feature. Signed-off-by: Quinn Burdine <sburdine@redhat.com> Signed-off-by: politerealism <burdcat17@gmail.com> * refactor(build): simplify CA root selection to single feature toggle Replace mutually exclusive bundled-ca-roots / system-ca-roots features with a single bundled-ca-roots toggle. Disabling it implies system roots via rustls-native-certs, which is now a regular (non-optional) dependency. This fixes cargo --all-features and simplifies the distro build interface from --no-default-features --features system-ca-roots to just --no-default-features. Signed-off-by: Quinn Burdine <sburdine@redhat.com> Signed-off-by: politerealism <burdcat17@gmail.com> * feat(build): add system-ca-roots convenience alias and fix verify task Add a system-ca-roots feature alias on openshell-sandbox that includes all other defaults (telemetry) except bundled-ca-roots, so distro builds can use --no-default-features --features system-ca-roots without manually re-adding unrelated defaults. Update the verify CI task to use the alias and scope checks to the sandbox package. Fix task description to use "build mode" terminology instead of implying a Cargo feature. Signed-off-by: Quinn Burdine <sburdine@redhat.com> Signed-off-by: politerealism <burdcat17@gmail.com> * ci: fix system CA roots step name to use build mode terminology Signed-off-by: Quinn Burdine <sburdine@redhat.com> Signed-off-by: politerealism <burdcat17@gmail.com> * refactor(sandbox): reorder features to place system-ca-roots alias near default Signed-off-by: Quinn Burdine <sburdine@redhat.com> Signed-off-by: politerealism <burdcat17@gmail.com> * fix(proxy): unwrap Result from build_upstream_client_config in tests The function signature changed to return Result but the test call sites were not updated, causing type mismatch compilation errors in CI. Signed-off-by: Quinn Burdine <sburdine@redhat.com> Signed-off-by: politerealism <burdcat17@gmail.com> --------- Signed-off-by: Adam Miller <admiller@redhat.com> Signed-off-by: Scott Burdine <sburdine@nvidia.com> Signed-off-by: politerealism <burdcat17@gmail.com> Signed-off-by: Quinn Burdine <sburdine@redhat.com> Co-authored-by: Adam Miller <admiller@redhat.com> |
||
|
Simon Scatton
|
b9818619b8 |
test: disable tests flaky under parallel stress (#2611)
Repeated Bazel test targets with --runs_per_test at 10, 20, 50, 100, and 200 runs. The tests failed intermittently when multiple instances ran concurrently. This indicates timing, shared tracing state, socket readiness, or parallel-safety issues that need focused follow-up before re-enabling the tests. A normal non-repeated bazel test //... run passes with these tests ignored. Disabled tests: - sandbox_forward_foreground_fails_when_ssh_exits_before_listener_opens - sandbox_forward_background_terminates_owned_child_when_listener_never_opens - podman_socket_probe_accepts_successful_ping_response - podman_socket_probe_rejects_docker_ping_response - docker_socket_probe_accepts_successful_ping_response - docker_socket_probe_rejects_podman_ping_response - docker_socket_detection_returns_the_responsive_candidate - podman_socket_detection_returns_the_responsive_candidate - driver_watch_events_are_roots_and_store_operations_have_parents - reconcile_sweeps_are_roots_and_operations_have_parents - gateway_listeners_bind_ipv6_wildcard_and_ipv4_callback_on_same_port - watch_producer_releases_request_span_when_client_disconnects - expected_conflicts_leave_the_span_unmarked - store_spans_record_what_they_touched_as_attributes - store_operations_export_spans_with_parents - refresh_worker_ticks_are_roots_and_store_operations_have_parents Signed-off-by: Simon Scatton <sscatton@nvidia.com> |
||
|
krishicks
|
8328412959 |
feat(vm): export driver traces over OTLP (#2564)
Continue distributed traces across the gateway-to-driver process boundary and export VM driver spans to the same OTLP/gRPC collector. The driver reports as the distinct openshell-driver-vm service. Updated the gateway architecture and configuration reference with a generic external-driver forwarding contract. Instrumented: - Every RemoteComputeDriver RPC injects the active W3C trace context into tonic metadata. Managed VM readiness and runtime initialization give startup capability probes stable parent operations rather than isolated root spans. - A tonic service layer creates fixed, low-cardinality server spans for every ComputeDriver RPC. New handlers inherit tracing automatically; failures record OpenTelemetry error status and the gRPC status code. - Background provisioning remains attached to CreateSandbox after the RPC returns without extending the RPC span lifetime. - Provisioning records image preparation, bootstrap image resolution, overlay preparation, lifecycle configuration, pre-launch hooks, guest preparation, and launcher spawn as child spans. - VM startup reconciliation roots one trace for the persisted-sandbox scan, with per-sandbox restore and provision operations beneath it. The root remains open until all spawned restore tasks finish. - Delete cleanup records its own child operation. Design notes: - The gateway forwards its configured OTLP endpoint to managed external drivers. SDK `OTEL_*` variables continue to own sampling, batching, limits, headers, and transport tuning. - The VM driver has its own tracer provider and service resource so trace backends preserve the service boundary. - RPC operation names come from an explicit method mapping, keeping cardinality bounded without parsing the protobuf descriptor set at runtime. - Propagation uses a remote SpanContext for spawned provisioning. This keeps one trace while allowing the CreateSandbox server span to finish when the RPC response is sent. - Startup restoration is independent of gateway requests. It begins at the VM driver reconciliation span rather than attaching to an unrelated RPC. - Existing tracing events remain on the logging path. The OpenTelemetry layer exports spans only and excludes the SDK exporter callsites to avoid recursive traces. - Export configuration failures do not prevent the driver from serving, and buffered spans are drained during graceful shutdown. - Trace fields identify drivers, sandboxes, images, lifecycle phases, and gRPC outcomes without recording credentials, sandbox tokens, or request query parameters. Refs #2507 Signed-off-by: Kris Hicks <khicks@nvidia.com> |
||
|
Simon Scatton
|
0a3ec7a112 | feat(bazel): add rustfmt checks to tests (#2599) | ||
|
krishicks
|
704880e5c4 |
test(podman): gate gateway discovery test on Linux (#2580)
Run the constructor gateway-discovery error test only on Linux, matching the production branch that inspects the Podman bridge gateway. macOS uses its Podman machine callback path and correctly skips that inspection. Signed-off-by: Kris Hicks <khicks@nvidia.com> |
||
|
Simon Scatton
|
fde96f0488 |
build(bazel): add OpenTelemetry crate targets (#2595)
* build(bazel): add OpenTelemetry crate targets Signed-off-by: Simon Scatton <sscatton@nvidia.com> * fix(bazel): provide sandbox test policy input Signed-off-by: Simon Scatton <sscatton@nvidia.com> --------- Signed-off-by: Simon Scatton <sscatton@nvidia.com> |
||
|
Simon Scatton
|
1959ea19be |
build(bazel): establish RFC 0012 Rust reference graph (#2414)
Establish the Phase 1 reference implementation proposed by RFC 0012 while retaining the existing Cargo and Mise workflows during evaluation. - pin Bazel 9.1.1 and configure Bzlmod, rules_rs, LLVM, protobuf, and Rust 1.95 toolchains - import third-party crates from Cargo metadata and propagate the workspace version into Bazel targets - add library, binary, proc-macro, unit-test, and integration-test targets across the supported Rust workspace crates and drivers - generate protobuf Rust sources and descriptor sets under Bazel while preserving Cargo-compatible generated-code imports - annotate aws-lc-sys and zstd-sys native dependencies, build Z3 4.15.2 from source, and generate z3-sys bindings - make CLI and procfs test fixtures available as explicit Bazel inputs without relying on fixed host binary paths - define optimized release targets for Linux x86_64 and aarch64 CLI, sandbox, and gateway binaries, plus macOS aarch64 artifacts - add Bazel, buildifier, and lcov to the Nix development environment RFC: 0012 (rfc12 branch) Refs: #2491 Signed-off-by: Simon Scatton <sscatton@nvidia.com> |
||
|
Grace Smith
|
736e431d45 |
fix(supervisor): quote nft log prefix in bypass rules (#2555)
The log prefix value passed to nft contains colons (e.g. openshell:bypass:sandbox-cc817378:) but was not wrapped in double quotes. Since nft concatenates argv entries and parses the result, the bare colons cause a syntax error that silently prevents all bypass-attempt LOG rules from installing. Wrap log prefix values in nft-quoted strings via a new nft_quote() helper that strips embedded double-quotes (nft quoted strings don't support escape sequences). All four log-rule generation sites (TCP and UDP, for both per-sandbox and sidecar rulesets) are updated. Fixes #2470 Signed-off-by: Grace Smith <grasmith@redhat.com> |
||
|
Piotr Mlocek
|
e75331772b |
test(server): stabilize watch span cancellation test (#2582)
* test(server): stabilize watch span cancellation test Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> * test(server): poll span exporter at fixed interval Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> --------- Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> |
||
|
Piotr Mlocek
|
06c2db75dc |
fix(podman): combine sandbox stop and removal (#2570)
* fix(podman): combine sandbox stop and removal Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> * fix(podman): use libpod volume removal parameter Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> * fix(podman): preserve removal timeout margin Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> * test(podman): use virtual time for removal delay Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> --------- Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> |
||
|
John T. MyersandJohn Myers
|
905b554c7c |
refactor(network): consolidate proxy egress pipeline (#2373)
* refactor(network): introduce shared egress pipeline Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * test(network): cover shared proxy egress paths Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * refactor(network): make destination authorization explicit Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * refactor(network): pin proxy relay policy context Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * test(network): lock relay generation contracts Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * test(network): establish phase zero compatibility baseline Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * feat(policy): detect ambiguous network endpoints Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * feat(sandbox): fail closed on invalid policy updates Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * refactor(network): invalidate relays on policy changes Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * docs(policy): document validation failure posture Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * test(network): cover validation and middleware egress Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * fix(config): move policy failure mode to gateway toml Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * test(network): name proxy contracts by behavior Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * fix(network): align overlap validation with endpoint selection Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * test(network): expect hard loopback denial Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * test(network): match declared endpoint denial Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * fix(policy): preserve path-specific endpoint overrides Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * test(network): respect hard-blocked host gateways Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * fix(network): reconcile proxy refactor with main Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * fix(network): preserve CONNECT policy generation Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * fix(policy): cover runtime endpoint glob semantics Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * fix(server): reject ambiguous policies before persistence Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * docs(policy): explain ambiguity preflight behavior Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * fix(policy): compare body limits within protocol Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * test(proxy): avoid global tracing capture race Signed-off-by: John Myers <johntmyers@users.noreply.github.com> * chore(server): format rebased provider tests Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * test(server): authenticate rebased policy requests Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * fix(sandbox): retain runtime on middleware outage Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * fix(server): preflight provider composition activation Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * fix(sandbox): distinguish runtime failure transitions Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> --------- Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> Signed-off-by: John Myers <johntmyers@users.noreply.github.com> Co-authored-by: John Myers <johntmyers@users.noreply.github.com> |
||
|
krishicks
|
489bb0d5dd |
fix(server): isolate otel tracing test exporters (#2579)
Give each span-assertion test its own thread-scoped in-memory exporter so parallel tests cannot contaminate or reset captured spans. Keep a bare global tracing registry only to preserve callsite interest, and serialize scoped subscriber changes because tracing caches that interest process-wide. Remove the test-only OTLP collector, polling delivery barrier, transport-specific test, and direct opentelemetry-proto dependency. Seed the expected persistence conflict before tracing begins so its assertion window contains only the operation under test. Signed-off-by: Kris Hicks <khicks@nvidia.com> |
||
|
Evan LezarandDrew Newberry
|
d220d89468 |
feat(compute): negotiate gateway callback listeners (#2492)
* feat(compute): query gateway listener requirements Signed-off-by: Evan Lezar <elezar@nvidia.com> * feat(compute): add Podman listener requirements Signed-off-by: Evan Lezar <elezar@nvidia.com> * test(docker): use default gateway bind address Signed-off-by: Evan Lezar <elezar@nvidia.com> * fix(gateway): avoid wildcard primary listener Signed-off-by: Evan Lezar <elezar@nvidia.com> * fix(podman): validate callback listener discovery Signed-off-by: Evan Lezar <elezar@nvidia.com> * fix(server): support split dual-stack listeners Signed-off-by: Evan Lezar <elezar@nvidia.com> * fix(podman): support legacy rootless listener discovery Signed-off-by: Evan Lezar <elezar@nvidia.com> * test(e2e): accept loopback plaintext rejection Signed-off-by: Evan Lezar <elezar@nvidia.com> * docs(agent): add callback listener diagnostics Signed-off-by: Evan Lezar <elezar@nvidia.com> * fix(server): restrict compute callback listeners Signed-off-by: Evan Lezar <elezar@nvidia.com> * fix(podman): validate local callback port Signed-off-by: Evan Lezar <elezar@nvidia.com> * test(server): clarify callback listener contract Signed-off-by: Evan Lezar <elezar@nvidia.com> * fix(podman): require pasta for local callbacks Signed-off-by: Evan Lezar <elezar@nvidia.com> * docs(gateway): document RPM listener default Signed-off-by: Evan Lezar <elezar@nvidia.com> * refactor(server): keep listener provenance diagnostic-only Signed-off-by: Evan Lezar <elezar@nvidia.com> * fix(compute): preserve callback listener isolation Signed-off-by: Drew Newberry <anewberry@nvidia.com> * test(e2e): remove Podman callback relay Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(packaging): preserve Podman callback loopback Signed-off-by: Drew Newberry <anewberry@nvidia.com> * ci(e2e): run VM smoke on nested-virt runner Signed-off-by: Drew Newberry <anewberry@nvidia.com> * ci(e2e): gate VM smoke on usable KVM Signed-off-by: Drew Newberry <anewberry@nvidia.com> * ci(e2e): probe KVM through VM driver Signed-off-by: Drew Newberry <anewberry@nvidia.com> * ci(e2e): tolerate hosted KVM denial Signed-off-by: Drew Newberry <anewberry@nvidia.com> * test(server): close traced futures before assertions Signed-off-by: Drew Newberry <anewberry@nvidia.com> * revert: remove tracing test stabilization Signed-off-by: Drew Newberry <anewberry@nvidia.com> --------- Signed-off-by: Evan Lezar <elezar@nvidia.com> Signed-off-by: Drew Newberry <anewberry@nvidia.com> Co-authored-by: Drew Newberry <anewberry@nvidia.com> |
||
|
krishicks
|
1a25439cab |
refactor(otel): share OTLP trace provider setup (#2567)
Extract common OpenTelemetry provider construction into openshell-otel so OpenShell services share one OTLP/gRPC export implementation. The shared crate owns: - typed exporter setup errors and non-fatal provider enablement; - endpoint trimming and URI validation before lazy exporter connection; - fixed and environment-or-default service-name policies; - service version and caller-supplied resource attributes; - batch tracer-provider construction; and - span-only tracing layers that exclude OpenTelemetry exporter callsites. Migrate the gateway to the shared provider while retaining its configurable service name, error marking, and tracing test collector. Add the shared crate to the architecture inventory and document the tracing boundary. Refs #2507 Signed-off-by: Kris Hicks <khicks@nvidia.com> |
||
|
alangou
|
5541398ccb |
fix(deps): update russh dompurify and base image of the gateway (#2575)
Signed-off-by: Adrien Langou <alangou@nvidia.com> |
||
|
krishicks
|
fa24299092 |
feat(gateway): export traces over OTLP (#2534)
Add an opt-in OTLP/gRPC trace exporter to the gateway. Export is enabled
by the presence of an `[openshell.gateway.otlp]` table with an endpoint;
there is no separate toggle.
Instrumented:
- Inbound request server spans, named for the RPC (`$service/$method`) or
`{method} {path}` for plain HTTP. They continue valid W3C `traceparent`
context when present and start a new trace otherwise. gRPC spans also
carry `rpc.system`, `rpc.service`, `rpc.method`, and trailer-derived
`rpc.grpc.status_code`.
- Compute driver calls (create, delete, list, get, validate, watch) as
client spans anchored on the `ComputeDriver` contract.
- Store reads and writes as children of the current request or loop span.
- Work with no inbound request: compute driver initialization, the sandbox
reconcile sweep, provider credential refresh tick, and driver watch events.
Each roots one operation trace so its child work does not arrive as anonymous
single-span traces.
This is deliberately not exhaustive. Auth, policy evaluation, and
middleware remain uninstrumented, as do store lifecycle calls (`ping`,
`close`) that a readiness poll would turn into a span per tick. The aim is
a useful trace tree at a reviewable size; coverage can grow against real
traces.
Design notes:
- The TOML table owns whether and where to export. The SDK `OTEL_*`
variables own how; sampling, batching, and limits are not mirrored into
gateway config.
- The OpenTelemetry layer exports spans only. Existing `tracing` events
remain on the stdout and sandbox-log paths and are not copied into trace
payloads.
- Telemetry never blocks the gateway. A malformed endpoint logs an error
and disables export rather than failing startup, and buffered spans are
drained during graceful shutdown.
- Failed spans carry error status without a separate `error.type` attribute.
Request spans use HTTP status and gRPC response trailers; driver spans use
the returned gRPC status; autonomous loop spans record failed results
explicitly. Store spans exempt `UniqueViolation` and `Conflict`, because
those errors report expected contention such as a held lease or an
optimistic-concurrency retry.
- The compute driver is reachable only through `TracedDriver::call`, so a
call cannot skip its span. This is the client half of a client/server pair
and the single place to inject context if drivers move out of process.
- Tests share one process-wide subscriber and in-memory exporter because
`tracing` caches callsite interest globally.
Inbound W3C trace context is propagated into gateway request spans. Context
is not yet injected into outbound driver calls, so a future out-of-process
driver would still need propagation at the `TracedDriver` seam.
The Helm chart is intentionally unchanged, so OTLP export cannot yet be
enabled on a chart-deployed gateway.
Refs #2507
Signed-off-by: Kris Hicks <khicks@nvidia.com>
|
||
|
Nave Cohen
|
28f3bee0ce |
fix(server): notify watchers after atomic policy commits (#2556)
Signed-off-by: Nave Cohen <nave@monday.com> |
||
|
Evan LezarandDrew Newberry
|
df698042d0 |
refactor(server): isolate gateway listener context (#2542)
* refactor(server): isolate gateway listener context Signed-off-by: Evan Lezar <elezar@nvidia.com> * refactor(server): preserve gateway listener binding Signed-off-by: Evan Lezar <elezar@nvidia.com> * fix(server): preserve covered listener callback scope Signed-off-by: Evan Lezar <elezar@nvidia.com> * refactor(server): reuse gateway listener spec Signed-off-by: Evan Lezar <elezar@nvidia.com> * ci(e2e): pin rootless podman packages Signed-off-by: Drew Newberry <anewberry@nvidia.com> --------- Signed-off-by: Evan Lezar <elezar@nvidia.com> Signed-off-by: Drew Newberry <anewberry@nvidia.com> Co-authored-by: Drew Newberry <anewberry@nvidia.com> |
||
|
Derek Carr
|
9c019a93f5 |
Wire authorization into workspace model (#2445)
* feat(auth): implement RFC 0011 Phase 2 workspace authorization Signed-off-by: Derek Carr <decarr@redhat.com> * fix(auth): address PR review feedback on workspace authorization - Docker e2e: add --health-port and switch readiness probe from `openshell status` to `curl /healthz`, fixing a false-positive readiness check in OIDC mode where the CLI exited 0 without actually contacting the gateway - ListWorkspaces: move membership filtering from post-query N+1 lookups into a SQL EXISTS subquery so pagination applies to the visible set, not the global ordering. Add generic list_with_membership to the persistence layer. - Descriptor validator: reject role/scope fields on unauthenticated and sandbox auth modes, and allow-list workspace_role as user/admin and global_role as platform_admin to catch typos at startup Signed-off-by: Derek Carr <decarr@redhat.com> * fix(server): use authed request in delete telemetry test The workspace authorization added by the Phase 2 auth changes requires a Principal on every delete request. The delete-telemetry test was still using a bare Request::new, so extract_principal failed before the handler could acquire the delete gate, causing a 5-second timeout flake. Signed-off-by: Derek Carr <decarr@redhat.com> * fix(auth): address gator review findings for workspace authorization - Inject unauthenticated-local-dev principal in no-auth gateway mode so handlers that call extract_principal() always find one. - Cap label-selector membership query at MAX_PAGE_SIZE instead of u32::MAX to bound the in-memory read. - Authorize workspace membership before resolving workspace existence in all sandbox RPCs to prevent workspace-name enumeration by non-members. - Remove dead_code allow on AuthorizedWorkspace.workspace now that callers use the normalized name from the authz result. Signed-off-by: Derek Carr <decarr@redhat.com> * fix(auth): close workspace-name oracle and label-selector truncation Swap authorize-before-resolve ordering in 27 handlers across provider.rs, service.rs, policy.rs, and workspace.rs to prevent CWE-203 workspace-name enumeration by non-members. Add combined membership+label SQL query (list_with_membership_and_selector) to both persistence backends so ListWorkspaces with label selectors no longer silently drops results beyond the first page of membership matches. Signed-off-by: Derek Carr <decarr@redhat.com> * test(auth): add non-member rejection and membership+label persistence tests Add comprehensive test coverage for workspace authorization changes: - Non-member rejection tests across all 44 workspace-scoped handlers (sandbox, provider, service, policy, workspace, inference) verifying PERMISSION_DENIED is returned instead of NOT_FOUND to prevent CWE-203 workspace-name oracle - Persistence test for list_with_membership_and_selector verifying SQL-level membership EXISTS + label filtering, multiple predicates, no-match cases, and pagination Signed-off-by: Derek Carr <decarr@redhat.com> * fix(auth): format merged import line in sandbox tests Signed-off-by: Derek Carr <decarr@redhat.com> * fix(auth): address gator re-review findings on workspace authorization - Fix TUI unconditionally setting providers_v2_enabled after provider refresh; read the actual gateway setting via GetGatewayConfig at startup instead - Fix SQLite json_extract with dotted label keys (e.g. example.com/env) by quoting the key in the JSON path - Add authed_request wrappers to upstream OCI identity tests that were missing a principal after rebase - Add test proving GetGatewayConfig is accessible without Platform Admin - Add test for dotted/prefixed Kubernetes-style label key filtering Signed-off-by: Derek Carr <decarr@redhat.com> * fix(auth): address second gator re-review findings - Loosen GetGatewayConfig from platform_admin to scope-only so workspace users can discover providers_v2_enabled during sandbox creation with inferred-provider commands; update proto descriptor, descriptor validation, and RFC 0011 access table - Add validate_label_selector to handle_list_workspaces and escape single quotes in SQLite json_extract interpolation (CWE-89 defense-in-depth) - Re-fetch providers_v2_enabled after TUI gateway switch so the new gateway's capability is reflected - Add e2e test for workspace user with inferred-provider command - Add persistence test for adversarial label keys with SQL injection attempts - Add handler test for invalid label selector rejection in ListWorkspaces Signed-off-by: Derek Carr <decarr@redhat.com> * fix(auth): address third gator review findings - Cap label selector pairs at 64 (CWE-400) to bound SQLite dynamic SQL - Add SCOPE_ONLY_METHODS allowlist for scope-without-role RPCs (CWE-863) - Normalize ID-based data-plane handlers to return NOT_FOUND for unauthorized sandboxes, closing the cross-workspace oracle (CWE-203) - Fix TUI provider profile cache lookup key mismatch for legacy providers with empty profile_workspace - Add whoami to CLI skill reference command tree - Update TUI skill doc with workspace, provider, and settings coverage - Document scope/workspace orthogonality on GetGatewayConfig proto Signed-off-by: Derek Carr <decarr@redhat.com> * fix(auth): extend CWE-203 normalization to policy.rs sandbox handlers GetSandboxConfig and GetSandboxLogs in policy.rs had the same fetch-before-authorize pattern that leaked cross-workspace sandbox existence. Promote fetch_and_authorize_sandbox to pub(super) and use it from both sandbox.rs and policy.rs handlers. Signed-off-by: Derek Carr <decarr@redhat.com> * test(auth): update assertions for CWE-203 sandbox ID normalization Cross-workspace sandbox access via ID-based handlers now returns NOT_FOUND instead of PERMISSION_DENIED to prevent existence inference. Update the unit test and OIDC e2e assertion to match. Signed-off-by: Derek Carr <decarr@redhat.com> * fix(auth): narrow CWE-203 error mapping and correct whoami output formats Only remap PERMISSION_DENIED to NOT_FOUND in fetch_and_authorize_sandbox and RevokeSshSession, letting INTERNAL and UNAUTHENTICATED propagate as-is. Fix whoami --output format values in cli-reference.md to match the actual CLI (table/json/yaml, not text/json). Signed-off-by: Derek Carr <decarr@redhat.com> * fix(ci): share network namespace with Keycloak in containerized CI In GitHub Actions job containers, Docker port publishing lands on the host, not inside the job container. Detect this environment and attach Keycloak to the job container's network namespace instead, with hardened defaults (cap-drop ALL, no-new-privileges, loopback-only listener). Signed-off-by: Derek Carr <decarr@redhat.com> --------- Signed-off-by: Derek Carr <decarr@redhat.com> |
||
|
Russell Bryant
|
0cecb54249 |
fix(cli): bracket IPv6 bind literals in SSH forwards (#2552)
ForwardSpec accepts IPv6 bind addresses, but ssh_forward_arg() emitted them unbracketed (e.g. ::1:8080:127.0.0.1:8080), which OpenSSH rejects as a bad local forwarding specification. access_url() likewise produced invalid URLs like http://::1:8080/. Extract a shared bracket_ipv6_host() helper and use it in ssh_forward_arg(), access_url() (via format_gateway_url), and format_gateway_url() so IPv6 literals are bracketed consistently. Fixes #2279 Signed-off-by: Russell Bryant <rbryant@redhat.com> |
||
|
Roland Huss
|
1221b5868a |
fix(cli): isolate subprocess tests from host OPENSHELL_ env vars (#2523)
The subprocess-based integration tests inherit the full parent environment. If the developer has OPENSHELL_GATEWAY_INSECURE=true set in their shell, it leaks into the spawned CLI process and causes it to connect with .with_no_client_auth(), skipping the mTLS client certificate. The test server requires mTLS, so it responds with CertificateRequired and the test fails. Strip OPENSHELL_GATEWAY_INSECURE, OPENSHELL_GATEWAY, OPENSHELL_GATEWAY_ENDPOINT, and OPENSHELL_WORKSPACE from the subprocess environment. The test already sets --gateway and --gateway-endpoint explicitly via CLI args, so these env vars should not influence the subprocess behavior. Reported-by: Seth Jennings Signed-off-by: Roland Huß <rhuss@redhat.com> |
||
|
Evan LezarandDrew Newberry
|
662dee68eb |
refactor(compute): make sandbox readiness gateway-owned across all drivers (#2153)
* refactor(compute): extract create_sandbox_record and update_sandbox_record helpers Split apply_sandbox_update_locked into named helpers to make the two distinct paths explicit: create_sandbox_record for first-observation events and update_sandbox_record for subsequent driver snapshots on existing sandboxes. The dispatcher now uses a match on the existing record rather than an early-return guard. No behavior change. Signed-off-by: Evan Lezar <elezar@nvidia.com> * refactor(compute): make sandbox readiness gateway-owned across all drivers Introduce compute_phase_components and apply_readiness_conditions to centralise the gateway's phase composition logic. The public SandboxPhase is now determined by combining the backend phase reported by the driver with supervisor session presence, independent of the driver implementation. Remove SupervisorReadiness from the driver contract. Running containers always report BackendReady; the gateway owns the Ready decision. Rename the dispatcher match to three arms so that status-less events for existing sandboxes are a documented no-op rather than a silent pass-through. Drop backend_ready_no_session and the SupervisorNotConnected condition. The BackendReady driver condition plus the Provisioning phase already communicates that the backend is up but the supervisor has not connected. The redundant condition added noise without new information. Closes #1951 Signed-off-by: Evan Lezar <elezar@nvidia.com> * fix(compute): expose disconnected supervisor readiness Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs(sandboxes): clarify supervisor readiness lifecycle Signed-off-by: Evan Lezar <elezar@nvidia.com> --------- Signed-off-by: Evan Lezar <elezar@nvidia.com> Signed-off-by: Drew Newberry <anewberry@nvidia.com> Co-authored-by: Drew Newberry <anewberry@nvidia.com> |
||
|
Andrew White
|
101cbc9780 |
fix(cli): avoid panic on multi-byte UTF-8 in --since duration (#2446)
parse_duration_to_ms was moved to commands/common.rs since the original PR was opened, but it still split the last byte of the input with split_at(s.len() - 1), which panics when the final character is multi-byte UTF-8 (e.g. 'openshell logs my-sandbox --since 5€'). Split off the last character using its UTF-8 length instead, so invalid units surface the intended 'unknown duration unit' error. Add regression tests in commands/common.rs. Signed-off-by: Andrew White <andrewh@cdw.com> |
||
|
Matthew Grossman
|
bc14018cad |
feat(sandbox): use policy-first OCI image identity (#2509)
* feat(sandbox): use policy-first OCI image identity Closes #2331 Preserve per-field policy omission, derive Docker and Podman fallbacks from the inspected immutable image, and resolve the final numeric identity before starting agent children. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * fix(sandbox): preserve declared process identities Keep explicit policy values and OCI-declared names intact, defer passwd lookup until a primary GID is required, and refresh stale policy examples. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * fix(supervisor): reuse resolved OCI identity Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * fix(supervisor): allow Linux pre-exec arguments Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * fix(kubernetes): protect resolved sandbox identity Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * fix(sandbox): prepare workspace for OCI identity Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * refactor(sandbox): own only workspace root Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * fix(sandbox): harden partial identity drops Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * test(sandbox): scope OCI image e2e to Docker Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * fix(sandbox): narrow OCI identity fallback scope Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * test(podman): cover OCI identity launch Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * fix(podman): exercise OCI fallback in E2E Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> --------- Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> |
||
|
LR90
|
7955c8309b |
feat(k8s): support configuring workspace PVC storageClassName (#2463)
The Kubernetes driver's default workspace PVC never set storageClassName, so on clusters with no default StorageClass the PVC stayed Pending and sandbox creation failed. Add a workspace_storage_class option to KubernetesComputeConfig, wired through SandboxPodParams into the generated volumeClaimTemplates. When non-empty it sets storageClassName; empty preserves the current behavior of relying on the cluster default StorageClass. Expose it via the OPENSHELL_K8S_WORKSPACE_STORAGE_CLASS env var on both the standalone driver and the embedded gateway runtime defaults, and via the server.workspaceStorageClass Helm value. Closes #2442 Signed-off-by: lr90 <qiuweimin@matrixorigin.cn> |
||
|
Derek Carr
|
b78c8615e3 |
fix(server): stabilize flaky delete telemetry unit test (#2521)
Replace yield_now() spin-loop with sleep(10ms) polling in delete_handler_ends_telemetry_for_the_resolved_sandbox_id. The single-threaded tokio runtime starves the spawn_blocking threads used by SQLite when yield_now() burns 100% CPU waiting for the delete gate entry count. Increase the timeout from 1s to 5s for consistency with similar guard tests. Signed-off-by: Derek Carr <decarr@redhat.com> |
||
|
Evan Lezar
|
7e9a7f5110 |
fix(sandbox): gate Linux-only ordering import (#2513)
Signed-off-by: Evan Lezar <elezar@nvidia.com> |
||
|
Evan Lezar
|
f00ad23a26 |
fix(podman): tolerate shutdown transport closes (#2498)
Signed-off-by: Evan Lezar <elezar@nvidia.com> |
||
|
Varsha
|
24d491a080 |
refactor(cli): extract gateway commands into commands/gateway module (#2506)
Move gateway management functions (gateway_status, gateway_info, gateway_add, gateway_select, gateway_login, gateway_logout, gateway_list, gateway_remove, gateway_use, gateway_info_not_configured) and their private helpers from run.rs into commands/gateway.rs. Add pub use re-exports in run.rs so main.rs call sites remain unchanged. Extract shared test utilities (EnvVarGuard, with_tmp_xdg) into a crate-level test_utils module to avoid duplication between gateway tests and the remaining run.rs tests. Part of #2304 (PR 2/6). Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com> |
||
|
Roland Huss
|
52f9e9e900 |
fix(cli): eliminate flaky subprocess integration tests (#2504)
Remove vm_slow_progress_before_ready from the subprocess-based structured output tests. The 1.8s artificial delay (3x 600ms sleeps) left only ~0.7s headroom within the 5s OPENSHELL_PROVISION_TIMEOUT, causing sporadic TLS handshake failures on loaded Linux machines. Root cause analysis: - The CertificateRequired TLS alert occurs when the provision timeout fires mid-handshake, causing the client to drop the connection before presenting its certificate. - tokio::process::Command is NOT the cause: it is a thin async wrapper around std::process::Command that delegates environment setup, process spawning, and I/O identically. - The single-threaded tokio runtime is NOT the cause: testing with multi_thread flavor produces identical timing. - TempDir lifetime is sound: the directory lives until .output().await returns, so certs exist for the subprocess's entire lifetime. - Cert path resolution is deterministic: the subprocess resolves XDG_CONFIG_HOME/openshell/gateways/openshell/mtls/ correctly. The subprocess tests exist to verify that the real CLI binary produces clean stdout (no ANSI chrome, spinners, or progress text leaking into structured output). This invariant does not depend on slow progress events; the ProgressOutput::Silent branch that swallows progress events is already covered by the in-process test sandbox_create_keeps_waiting_while_vm_progress_arrives. Fixes #2501 Signed-off-by: Roland Huß <rhuss@redhat.com> |
||
|
Andrew White
|
0d5e5c5342 |
fix(cli): write exec stderr events to stderr in interactive mode (#2447)
The Stderr arm of the interactive exec event loop locked stdout, so stderr payloads were written to stdout. The non-interactive exec path already handles the same event with stderr. Write stderr payloads to stderr so shell redirection (e.g. 'openshell exec ... 2>err.txt') works in interactive mode. Signed-off-by: Andrew White <andrewh@cdw.com> |
||
|
Polite_realism
|
79bcf296cd |
fix(proxy): retry with backoff on transient accept errors instead of exiting (#2369)
* fix(proxy): retry with backoff on transient accept errors instead of exiting The proxy accept loop unconditionally broke on any accept() error, permanently killing the proxy while the sandbox continued to report Ready. Replace the break with a sleep-and-continue pattern: EMFILE/ENFILE errors get exponential backoff (100ms to 3.2s) to let file descriptors drain, and all other accept errors get a fixed 100ms backoff matching the existing metadata_server and edge_tunnel patterns. Closes #2337 Signed-off-by: Sean Burdine <sburdine@nvidia.com> Signed-off-by: politerealism <burdcat17@gmail.com> * fix(proxy): address review feedback on accept retry backoff - Use libc::EMFILE / libc::ENFILE instead of raw errno values; promote libc from dev-dependency to regular dependency - Extract accept_backoff() and is_fd_exhaustion_error() into testable helpers - Fix unreachable 5s cap: bump exponent limit from min(6) to min(7) so the 5_000ms ceiling is reachable (100 * 2^6 = 6400, capped to 5000) - Add 6 unit tests covering exponential progression, counter reset, saturation at cap, EMFILE/ENFILE detection, and non-FD error rejection Signed-off-by: Sean Burdine <sburdine@nvidia.com> Signed-off-by: politerealism <burdcat17@gmail.com> * fix(proxy): classify accept errors and exit on terminal failures Three-way error classification for the accept loop: transient errors (EMFILE, ECONNABORTED) retry with backoff, terminal errors (EBADF, EINVAL, ENOTSOCK) exit immediately, and unknown errors exit after 5 consecutive failures. Adds unit tests for the classifier and error handler, plus a subprocess-isolated integration test that validates EMFILE recovery by lowering RLIMIT_NOFILE. Signed-off-by: Quinn Burdine <sburdine@redhat.com> Signed-off-by: politerealism <burdcat17@gmail.com> * fix(proxy): use nested or-patterns to satisfy clippy lint Signed-off-by: Quinn Burdine <sburdine@redhat.com> Signed-off-by: politerealism <burdcat17@gmail.com> * fix(proxy): expand transient error allowlist and fix clippy/test issues Widen the accept-error classification to cover all Linux accept(2) transient errnos (ENETDOWN, EPROTO, ENOPROTOOPT, EHOSTDOWN, EHOSTUNREACH, EOPNOTSUPP, ENETUNREACH, ENONET behind cfg gate). Rename fd-exhaustion helpers to resource-pressure to reflect ENOBUFS and ENOMEM coverage. Fix clippy lints in the integration test (borrow_as_ptr, collection_is_never_read, while_let_loop) and make post-EMFILE recovery verification cross-platform by handling Linux's stale backlog behavior. Signed-off-by: Quinn Burdine <sburdine@redhat.com> Signed-off-by: politerealism <burdcat17@gmail.com> * fix(proxy): add remaining Linux accept(2) transient errnos and make test skip explicit Classify ENOSR, ESOCKTNOSUPPORT, EPROTONOSUPPORT, and ETIMEDOUT as transient accept errors per Linux accept(2) documentation. Treat ENOSR as resource pressure (exponential backoff). Make the EMFILE integration test skip path emit a diagnostic instead of silently returning success. Signed-off-by: Quinn Burdine <sburdine@redhat.com> Signed-off-by: politerealism <burdcat17@gmail.com> * fix(proxy): panic instead of silent skip when EMFILE is not triggered The integration test child runs inside Command::output() which captures stderr. A silent return on the Ok path meant CI would report a passing test without any diagnostic. Panic instead so the parent test fails visibly if the platform cannot induce EMFILE. Signed-off-by: Quinn Burdine <sburdine@redhat.com> Signed-off-by: politerealism <burdcat17@gmail.com> --------- Signed-off-by: Sean Burdine <sburdine@nvidia.com> Signed-off-by: politerealism <burdcat17@gmail.com> Signed-off-by: Quinn Burdine <sburdine@redhat.com> |
||
|
Evan Lezar
|
39bf94e528 |
fix(server): bind gateway listeners before sandbox resume (#2495)
* refactor(server): extract gateway listener binding Signed-off-by: Evan Lezar <elezar@nvidia.com> * fix(server): bind gateway listeners before sandbox resume Signed-off-by: Evan Lezar <elezar@nvidia.com> --------- Signed-off-by: Evan Lezar <elezar@nvidia.com> |
||
|
Andrew White
|
2d108818f8 |
fix(policy): avoid panic truncating multi-byte UTF-8 paths for display (#2448)
truncate_for_display sliced at a fixed byte index (&s[..77]), which panics when the index is not a char boundary. A policy with an over-long filesystem path containing multi-byte characters crashed the sandbox supervisor / OPA policy loader instead of producing the intended FieldTooLong violation. Back off to the nearest char boundary before slicing, and add regression tests. Signed-off-by: Andrew White <andrewh@cdw.com> |
||
|
Andrew White
|
76a5397e6f |
fix: assorted arithmetic and indexing robustness fixes (#2451)
* fix(core): avoid i64 underflow on expired GCP token When a cached GCP access token was already expired (expires_at_ms < now), the remaining lifetime calculation used plain subtraction: (expires_at_ms - now) / 1000. This underflows in debug builds and wraps in release builds. Use saturating_sub so an expired token simply yields a non-positive remaining lifetime and is skipped. Add a regression test. Signed-off-by: Andrew White <andrewh@cdw.com> * fix(server): avoid i64 underflow in lease age calculation The lease steal path computed age as now_ms() - record.updated_at_ms. If the stored updated_at timestamp is in the future (e.g. clock skew), the subtraction underflows. Use saturating_sub to treat a future timestamp as age 0, keeping the lease considered fresh. Signed-off-by: Andrew White <andrewh@cdw.com> * fix(server): avoid i64 overflow on large credential max lifetime max_lifetime_seconds is only validated as >= 0. The code multiplied the raw i64 value by 1000 to compute max_expires, which overflows for values above i64::MAX / 1000. The capped i32 value used for the STS request was already computed but not reused. Compute max_lifetime_ms from the capped value with saturating_mul and use it for both expires_at fallback and max_expires. Signed-off-by: Andrew White <andrewh@cdw.com> * fix(server): avoid hardcoded indexing into base_url_config_keys resolve_vertex_ai_route assumed every inference provider profile has at least two base URL config keys by indexing [0] and [1]. Only the Vertex profile satisfies that today; other profiles would panic here. Iterate base_url_config_keys with find_map instead, and add a test with an empty key list to ensure no panic. Signed-off-by: Andrew White <andrewh@cdw.com> * fix(server): clamp lease age at zero for future timestamps Address review feedback: add regression coverage for future-timestamp lease clock skew. Extract lease_is_expired() and clamp the age at zero — i64::saturating_sub saturates at i64::MIN, not zero, so a future updated_at_ms must be clamped explicitly to be treated as age zero and remain unstealable with a positive TTL. Signed-off-by: Andrew White <andrewh@cdw.com> * fix(server): fall back past blank preferred Vertex base URL Address review feedback: find_map stopped at the first present key even when its value was blank, and the outer filter discarded it without considering lower-priority aliases. Filter blank values inside the closure so a blank preferred key falls back to a valid alias, matching the documented priority order. Add a regression test with a blank preferred key and a valid fallback. Signed-off-by: Andrew White <andrewh@cdw.com> * fix(server): saturate AWS STS expiry cap on saturated clock Address review feedback: now_ms + max_lifetime_ms could still overflow when current_time_ms() saturates near i64::MAX. Compute max_expires = now_ms.saturating_add(max_lifetime_ms) once and use it for both the conversion fallback and the expiry cap. Add a boundary unit test covering a saturated clock. Signed-off-by: Andrew White <andrewh@cdw.com> --------- Signed-off-by: Andrew White <andrewh@cdw.com> |
||
|
Andrew White
|
516be602d8 |
fix: assorted byte-index slicing safety fixes (#2452)
* fix(supervisor-process): use slice patterns when rewriting passwd/group lines rewrite_passwd_at and rewrite_group_at indexed into a Vec of fields with fields[N] after checking fields.len(). Use slice patterns so a malformed line falls through to the no-op branch instead of risking a panic if the guard is ever changed. Add regression tests for malformed sandbox entries. Signed-off-by: Andrew White <andrewh@cdw.com> * fix(bootstrap): avoid byte-index slice after @ in SSH destination extract_host_from_ssh_destination sliced dest[at_pos + 1..] after finding '@'. '@' is ASCII so this is currently safe, but it is a latent panic surface if the split logic changes. Use get() instead and add a multi-byte hostname test. Signed-off-by: Andrew White <andrewh@cdw.com> * fix(bootstrap): avoid byte-index slice in .dockerignore glob matcher glob_match sliced path[idx + 1..] after matching '/'. '/' is ASCII so this is currently safe, but it is a latent UTF-8 panic surface. Use get() instead. Signed-off-by: Andrew White <andrewh@cdw.com> * fix(router): avoid literal byte offset when stripping /v1 prefix build_backend_url sliced &path[3..] after verifying the /v1 prefix. Use strip_prefix() so the code stays correct if the prefix length ever changes. Signed-off-by: Andrew White <andrewh@cdw.com> * fix(supervisor-network): avoid byte-index slice in inference path matching The Bedrock-style /*/ path matcher sliced rest[slash_at + 1..] after finding '/'. '/' is ASCII so this is currently safe, but it is a latent UTF-8 panic surface. Use get() instead. Signed-off-by: Andrew White <andrewh@cdw.com> --------- Signed-off-by: Andrew White <andrewh@cdw.com> |
||
|
Adel Zaalouk
|
afb462f336 |
fix(router): strip unsupported Anthropic beta fields from Vertex rawPredict body (#2465)
Claude Code v2.1.156+ sends `context_management` on every request. The direct Anthropic API accepts it, but Vertex AI rawPredict rejects it with HTTP 400 "Extra inputs are not permitted". The inference proxy was only stripping the `model` field for Vertex routes; extend it to also strip Anthropic-SDK-only beta fields listed in VERTEX_UNSUPPORTED_BODY_FIELDS. Fixes #2444 Signed-off-by: Adel Zaalouk <azaalouk@redhat.com> |
||
|
Grace Smith
|
deced8716b |
refactor(policy): extract shared L7 endpoint validation (#2389)
Move L7 endpoint semantic checks into the openshell-policy crate so both
profile lint and the runtime validator share one implementation. This
eliminates drift between the two validation paths.
The shared validator covers 9 checks: unknown protocol, rules/access
mutual exclusivity, JSON-RPC family access rejection, json-rpc requires
rules, non-JSON-RPC protocol requires rules or access, MCP requires
rules when allow_all is false, rules-would-deny-all detection,
deny_rules require protocol, and deny_rules require base allow set.
Changes rules/deny_rules fields to Option<Vec<...>> so absent vs empty
is distinguishable at lint time. Adds is_effectively_empty() to
L7AllowProfile for deny-all detection of allow: {} objects. Makes
rules_would_deny_all MCP-aware by checking tool/params.name selectors
before classifying a rule as deny-all. Adds params field to
L7AllowProfile so MCP tool selectors survive proto round-trip.
Signed-off-by: Grace Smith <gsmith@redhat.com>
Signed-off-by: Grace Smith <grasmith@redhat.com>
|
||
|
Philippe Martin
|
77e5c32217 |
feat(sandbox,gateway): route sandbox egress through corporate HTTP proxy (#2245)
* feat(sandbox,gateway): route sandbox egress through corporate HTTP proxy - Chain sandbox egress through a corporate HTTP proxy so outbound traffic from within the sandbox respects the host proxy settings - Forward sandbox proxy environment variables to the generated Podman config so the proxy is applied consistently to Podman-managed workloads Signed-off-by: Philippe Martin <phmartin@redhat.com> * fix(sandbox,podman): make corporate proxy routing operator-owned The operator-configured corporate egress proxy was injected under the conventional HTTPS_PROXY/HTTP_PROXY/NO_PROXY names as defaults beneath sandbox spec/template environment, so a sandbox creator could redirect egress at an arbitrary proxy or disable proxying with NO_PROXY=*. Route the boundary through reserved, supervisor-only variables (OPENSHELL_UPSTREAM_HTTPS_PROXY/HTTP_PROXY/NO_PROXY) written in the Podman driver's required-variable tier. Any sandbox-supplied value under a reserved name is stripped before the operator value is applied, so the supervisor never observes a reserved proxy variable the operator did not set. The supervisor now reads only the reserved names and ignores the conventional proxy variables the sandbox controls. Add the reserved proxy variables to the supervisor-only child-environment denylist so the corporate proxy URL and any embedded credentials are not inherited by the sandbox workload, which reaches egress through the local policy proxy and never needs them. Signed-off-by: Philippe Martin <phmartin@redhat.com> * feat(sandbox,podman): deliver corporate proxy credentials via secret file Proxy credentials were embedded inline in the proxy URL, so they were stored in gateway.toml and exposed in container metadata via 'podman inspect'. Reject inline 'user:pass@' credentials in https_proxy/http_proxy at startup (parsed with the url crate rather than hand-splitting), and add a proxy_auth_file option pointing at a 'user:pass' file. The driver stages that file as a per-sandbox root-only Podman secret, mounts it at a fixed path, and exports only the path in the reserved OPENSHELL_UPSTREAM_PROXY_AUTH_FILE variable, so the credential never appears in config, environment, or container metadata. Reading the file fails closed on a missing, empty, or control-character-bearing value. The supervisor reads the credential from the mounted file and builds the Proxy-Authorization: Basic header, rejecting control characters, and no longer derives credentials from URL userinfo. The auth-file path is added to the child-environment strip list, and the generated gateway.toml is written owner-only (mode 0600). Signed-off-by: Philippe Martin <phmartin@redhat.com> * fix(sandbox,podman): fail closed on invalid upstream proxy configuration The reserved OPENSHELL_UPSTREAM_* variables are an operator-owned egress boundary, but the supervisor treated present-but-invalid values as unset: an unsupported or malformed proxy URL was ignored with a warning, an unreadable auth file proceeded without credentials, and a malformed credential silently became unauthenticated. Any of these could quietly downgrade the corporate proxy boundary to direct dialing or unauthenticated proxy access. Make every configured-but-invalid proxy or auth setting fatal to supervisor proxy startup, emit an OCSF ConfigStateChange failure event before refusing, and share URL validation semantics between the Podman driver and the supervisor through a single validator in openshell-core (parse_upstream_proxy_url), so a value accepted at sandbox-create time can never be rejected in-container or vice versa. Inline user:pass@ URL credentials are now fatal in the supervisor too (previously warn-and-strip), matching the driver. Unset or empty variables still mean no proxy; only present-but-invalid values fail. Error paths never include credential content. Addresses the fail-closed review item on #2245. Signed-off-by: Philippe Martin <phmartin@redhat.com> * fix(sandbox,podman): finish the fail-closed upstream proxy credential contract The upstream proxy URL already had a single shared validator, but the credential did not: the Podman driver rejected only CR/LF/NUL while the supervisor rejected every control character, so a credential accepted at sandbox-create time (e.g. one containing a tab) could still be rejected in-container. Present-but-whitespace reserved OPENSHELL_UPSTREAM_* values were also silently treated as unset, quietly downgrading the operator's egress boundary to direct dialing. Add parse_upstream_proxy_credential to openshell-core as the single source of truth for the documented user:pass credential form (non-empty user, no control characters, trimmed) and use it in both the Podman driver's secret staging and the supervisor's Proxy-Authorization header construction. Error variants carry no payload so credential content can never leak into messages. Make a present-but-empty reserved variable fatal to supervisor proxy startup instead of meaning "unset"; only fully unset variables disable the proxy. The driver correspondingly rejects an empty no_proxy at config time so it can never inject a value the supervisor refuses. Addresses the remaining fail-closed credential/config review item on Signed-off-by: Philippe Martin <phmartin@redhat.com> #2245. * fix(sandbox,podman): close remaining fail-open upstream proxy config paths Two configuration paths could still silently run without the proxy boundary the operator believed was in effect. A no_proxy bypass list configured without any https_proxy/http_proxy was accepted by both the driver and the supervisor and simply meant "dial everything directly". Reject it on both sides, exactly like the existing proxy_auth_file-without-proxy rule: an operator who wrote a bypass list assumed proxying was active, so accepting it hides a fail-open state. The gateway.sh dev script guarded proxy settings with [[ -n "${VAR:-}" ]], which conflates unset with explicitly-empty and dropped the latter before the gateway's validation could see it. Use ${VAR+x} instead so a set-but-empty variable is written into gateway.toml and rejected at startup by validate_proxy_config rather than silently discarded. Addresses the remaining fail-open configuration review item on #2245. Signed-off-by: Philippe Martin <phmartin@redhat.com> * fix(sandbox,podman): reject upstream proxy URLs with path, query, or fragment parse_upstream_proxy_url accepted URLs like http://proxy.corp.com:8080/some/path and silently discarded everything after host:port, a lenience inherited from the original supervisor parser. A forward proxy is addressed by host:port only, so extra components indicate a misconfiguration (for example a pasted endpoint URL) and silently truncating them violates the present-but-invalid-is-fatal contract enforced everywhere else in this configuration surface. Reject a path, query, or fragment in the shared validator with a new UnexpectedComponent error. A bare trailing slash remains accepted because the url crate normalizes an absent http path to "/", making the two indistinguishable. Both the Podman driver (gateway startup) and the supervisor (sandbox startup) inherit the rule through the shared parser, keeping their semantics identical by construction. Addresses the proxy URL component review item on #2245. Signed-off-by: Philippe Martin <phmartin@redhat.com> * fix(sandbox,podman): remove plain-HTTP upstream proxy support The http_proxy path tunneled plain-HTTP requests through the corporate proxy with CONNECT to port 80 and then sent origin-form requests down the tunnel. Conventional enterprise forward proxies expect plain HTTP as absolute-form requests sent directly over the proxy connection, and commonly refuse CONNECT to port 80, so the setting looked supported but failed against typical deployments. Tunneling also blinds the proxy to the one protocol it could inspect. Narrow the feature to TLS (CONNECT) egress only, which is the conventional and already-correct case: plain-HTTP requests now always dial the destination directly, and only client CONNECT tunnels chain through the corporate proxy. Remove the http_proxy config field, the --sandbox-http-proxy / OPENSHELL_SANDBOX_HTTP_PROXY driver surface, the reserved OPENSHELL_UPSTREAM_HTTP_PROXY variable, and the UpstreamScheme plumbing. The feature never shipped, so this is a clean removal; a stray http_proxy key in gateway.toml still fails loudly through the config's deny_unknown_fields. Removing the plain-HTTP proxy branch also removes its host-gateway special case; the architecture doc now documents the real host-gateway behavior (add driver-injected host aliases to the reserved NO_PROXY list) instead of an invariant the HTTPS path never implemented. Plain-HTTP forwarding through a corporate proxy can return later as absolute-form forwarding behind its own design review. Addresses the plain-HTTP forwarding review item on #2245. Signed-off-by: Philippe Martin <phmartin@redhat.com> * fix(sandbox,podman): escape generated TOML and require explicit proxy URL form Escape backslashes, quotes, and control characters when gateway.sh writes proxy values into gateway.toml, so a hostile or unusual environment value cannot corrupt the config or inject extra keys. Restrict the upstream proxy URL grammar to the documented http://host:port form: a scheme-less value is no longer normalized to http:// and a missing port is no longer silently defaulted to 80. Docs, README, and CLI help now state the explicit-form requirement consistently. Also fix a test-only call of handle_tcp_connection that was missing the upstream_proxy argument added in an earlier commit. Signed-off-by: Philippe Martin <phmartin@redhat.com> * fix(sandbox,podman): preserve tunneled bytes read with the CONNECT response The CONNECT handshake reads the corporate proxy's response in chunks, so the read that completes the header block can also contain the first tunneled payload bytes. Those bytes were discarded, silently corrupting the start of the tunnel for server-speaks-first destinations or proxies that coalesce writes. connect_via now returns a PrefixedStream that replays any bytes received past the response terminator before reading from the socket again; writes pass through unchanged. Direct dials wrap the stream with an empty prefix so downstream relay and TLS paths keep a single stream type, and tls_connect_upstream is generalized to any AsyncRead + AsyncWrite stream. Adds regression coverage for a combined response/payload read and for prefix replay ordering. Signed-off-by: Philippe Martin <phmartin@redhat.com> * fix(sandbox,podman): gate cleartext proxy Basic auth behind an explicit opt-in Proxy-Authorization: Basic is base64 over the plain-TCP connection to the http:// corporate proxy, so anyone on the network path between the sandbox host and the proxy can recover the credential. Sending it is now an explicit operator decision instead of an implicit side effect of configuring proxy_auth_file. Add a proxy_auth_allow_insecure driver setting (CLI --sandbox-proxy-auth-allow-insecure, env OPENSHELL_SANDBOX_PROXY_AUTH_ALLOW_INSECURE), delivered to the supervisor as the reserved OPENSHELL_UPSTREAM_PROXY_AUTH_ALLOW_INSECURE variable. Fail-closed pairing on both sides: an auth file without the acknowledgement is rejected at gateway startup and at supervisor startup, as is the acknowledgement without an auth file or any value other than 'true'. gateway.sh writes the key only as a TOML boolean; a non-boolean value is emitted as a quoted string so the gateway rejects it at startup instead of risking injection. Documents the exposure prominently in the gateway config reference, the driver README, and the sandbox architecture doc. Signed-off-by: Philippe Martin <phmartin@redhat.com> * fix(sandbox,podman): honor port qualifiers and resolved addresses in NO_PROXY Two divergences from the documented NO_PROXY contract: - A port-qualified entry (internal.corp:8443) was silently stripped to its hostname and bypassed the proxy for every port, excluding traffic the operator never listed. Entries now keep the optional :port qualifier (also on IP and CIDR entries) and only apply to that destination port; a trailing qualifier that is not a valid port stays part of the pattern instead of widening the entry. - IP and CIDR entries only matched IP-literal hosts, so a bypass like 10.0.0.0/8 never applied to hostnames resolving into that range. NO_PROXY evaluation now sees the validated resolved addresses: an IP/CIDR entry matching through resolution authorizes a direct dial of only the addresses it contains, so a bypass scoped to an internal range cannot widen into a direct dial of addresses outside it. Hostname-level matches (loopback, wildcard, domain entries, IP-literal hosts) keep authorizing all validated addresses. proxy_for is replaced by decision(host, port, resolved) returning either the proxy endpoint or the permitted direct-dial subset, and dial_upstream restricts the direct connect to that subset. Adds regression coverage for port-scoped bypasses on domain, IP, and CIDR entries, invalid port qualifiers, resolved-address matching, and split-resolution subset dialing. Signed-off-by: Philippe Martin <phmartin@redhat.com> * fix(sandbox,podman): bind proxied CONNECT tunnels to validated addresses The CONNECT request sent to the corporate proxy carried the destination hostname, so the proxy resolved the name itself and the addresses that had passed SSRF and allowed_ips validation were discarded. Split-horizon DNS or rebinding at the proxy could then reach internal or otherwise unapproved destinations through a tunnel the supervisor logged as validated, and IP-range policy could not be enforced at all on proxied dials. CONNECT now targets a validated resolved address by default: the proxy performs no DNS resolution and the tunnel stays bound to the answer the supervisor checked. The hostname still travels inside the tunnel (TLS SNI, application Host), so destination servers behave normally. In split-horizon networks, operators point the gateway host at the corporate resolver so internal names validate to their internal addresses. For proxies whose ACLs filter on hostnames and reject IP CONNECT targets, a new proxy_connect_by_hostname opt-in (CLI --sandbox-proxy-connect-by-hostname, env OPENSHELL_SANDBOX_PROXY_CONNECT_BY_HOSTNAME, reserved OPENSHELL_UPSTREAM_PROXY_CONNECT_BY_HOSTNAME) restores hostname CONNECT, documented as re-opening proxy-side resolution and making the proxy's ACLs the effective egress control. Fail-closed pairing on both sides: the opt-in without a proxy, or any value other than 'true', is fatal. Adds regression coverage for the IP CONNECT request line (including IPv6 bracketing and hostname non-leakage), the hostname opt-in, and the config pairing rules in driver and supervisor. Signed-off-by: Philippe Martin <phmartin@redhat.com> * fix(sandbox,podman): reject empty port after bracketed IPv6 proxy host http://[fd00::1]: passed the explicit-port check because the bracketed branch only tested for a colon after the bracket, then fell back to port 80 — violating the fail-closed http://host:port contract and potentially sending configured Basic credentials to an unintended service. Require a non-empty suffix after ]:, matching the unbracketed branch, and cover the case in the shared parser tests. Signed-off-by: Philippe Martin <phmartin@redhat.com> * fix(sandbox,podman): fall back across validated addresses in proxied CONNECT The direct path hands TcpStream::connect the whole validated address list and it falls back across them, but the validated-IP CONNECT path attempted only the first address, so a dual-stack destination could fail through the corporate proxy even when a later validated address was reachable. connect_via_validated tries each validated address in order under one aggregate CONNECT_HANDSHAKE_TIMEOUT budget, returning the first success; when every attempt fails the error names the attempt count and carries the last failure. An empty address list is rejected up front. Adds regressions for first-fails/second-succeeds fallback (asserting both CONNECT request lines), the aggregate all-addresses failure message, and the empty-list rejection. Signed-off-by: Philippe Martin <phmartin@redhat.com> * fix(sandbox,podman): strip new reserved proxy vars and complete docs/tests Follow-ups from review: - Add OPENSHELL_UPSTREAM_PROXY_AUTH_ALLOW_INSECURE and OPENSHELL_UPSTREAM_PROXY_CONNECT_BY_HOSTNAME to the supervisor-only strip list so workload child processes never inherit them, matching the documented contract for the other reserved proxy variables, and cover both in the supervisor-only variable test. - Document all five OPENSHELL_SANDBOX_* proxy variables in the mise run gateway help text, marked Podman-only and stating the auth-file/acknowledgement pairing, and complete the gateway-key list in the Podman README. - Add NO_PROXY composition coverage for bracketed IPv6 entries with port qualifiers, bare IPv6 entries, and IPv6 CIDR matching against an IPv6-literal host and against a hostname's resolved addresses, including the port-qualified CIDR form. Signed-off-by: Philippe Martin <phmartin@redhat.com> * fix(sandbox,podman): cap each proxied CONNECT attempt within the shared budget A proxy that accepted the first CONNECT request but never responded consumed the entire aggregate handshake timeout, so later validated addresses were never tried and the hang defeated the multi-address fallback. Each attempt is now time-boxed to its fair share of the time remaining before the shared deadline (remaining / attempts_left): a hanging attempt is cut off with enough budget left for every remaining address, while time a fast failure does not use rolls over to later attempts and the total never exceeds CONNECT_HANDSHAKE_TIMEOUT. A timed-out attempt is recorded like any other failure, and the aggregate error distinguishes all-attempted from budget-exhausted runs. Adds a first-hangs/second-succeeds regression driven through a test-visible budget parameter so it runs in about a second instead of a real 30s window. Signed-off-by: Philippe Martin <phmartin@redhat.com> * fix(sandbox,podman): deliver corporate proxy config on the supervisor argv The proxy settings were injected as reserved OPENSHELL_UPSTREAM_* container environment variables. The driver only wrote names the operator configured, but container runtimes layer the spec environment over ENV values baked into the sandbox image, so an image could supply NO_PROXY=*, enable hostname CONNECT, or point an unconfigured deployment at an attacker-controlled proxy whenever the operator left a field unset. The settings now travel as supervisor command-line arguments (--upstream-proxy, --upstream-no-proxy, --upstream-proxy-auth-file, --upstream-proxy-auth-allow-insecure, --upstream-proxy-connect-by-hostname) built by the driver from operator config. The driver sets the container entrypoint and command explicitly, so neither sandbox spec/template environment nor image ENV can influence argv, and an omitted flag genuinely means unconfigured — in every supervisor topology, since the supervisor no longer consults its environment for these settings at all. Credentials stay on the root-only secret mount; only the mount path appears on argv. The reserved environment names, their strip-list entries, and the env-based validation surface are removed. UpstreamProxyConfig::from_args replaces from_env, reusing the same shared fail-closed validation and pairing rules keyed by the CLI flag names. Driver tests now assert the argv contract, including that sandbox-supplied environment cannot add, remove, or redirect proxy flags; supervisor tests cover from_args mapping and its pairing rules. Signed-off-by: Philippe Martin <phmartin@redhat.com> * docs(sandbox): align proxy comments with the argv transport The argv migration left comments describing the configuration as reserved environment variables ("reserved value", "present-but-empty variable", "reserved upstream proxy variables"). Rephrase them as driver-supplied arguments and operator settings so the documented trust boundary matches the implementation. Comment-only change. Signed-off-by: Philippe Martin <phmartin@redhat.com> * fix(sandbox): parse bracketed IPv6 authorities in client CONNECT targets parse_target split the CONNECT authority at the first colon, so an IPv6-literal target like [2001:db8::1]:443 always failed port parsing and IPv6-literal clients could never reach policy evaluation; a regression test even locked in that failure. Parse the RFC 3986 bracketed form and return the host bracket-free, matching what DNS resolution, SSRF validation, NO_PROXY matching, and the upstream CONNECT builder expect. Unclosed brackets, a missing or empty port after the bracket, and non-numeric ports are rejected; unbracketed behavior is unchanged. Replaces the failure-locking test with success coverage for bracketed targets and adds malformed-bracket rejection cases. Signed-off-by: Philippe Martin <phmartin@redhat.com> * test(podman): cover proxy-auth secret cleanup across lifecycle failures The per-sandbox proxy-auth credential secret is staged before the container is created and removed on cleanup, but no test proved the cleanup paths actually issue the secret removal. Add Podman-stub tests that drive create_sandbox to a container-create failure and to a start failure, and delete_sandbox for an out-of-band deletion, asserting each path issues the DELETE for the per-sandbox proxy-auth secret so a credential can never outlive the sandbox that owned it. Signed-off-by: Philippe Martin <phmartin@redhat.com> * docs: list corporate proxy keys in the Podman compute-driver overview The Fern Podman driver section enumerated its gateway.toml keys but omitted the corporate egress proxy settings. Add https_proxy, no_proxy, proxy_auth_file, proxy_auth_allow_insecure, and proxy_connect_by_hostname with a pointer to the gateway configuration reference for the full contract. No navigation change: the reference folder already includes the gateway configuration page. Signed-off-by: Philippe Martin <phmartin@redhat.com> * test(sandbox): cover the SSRF-to-TLS composition across the proxy tunnel Existing tests exercised validated-IP CONNECT and the upstream-TLS helper independently, but not the full boundary. Add an end-to-end regression that stands up a fake corporate proxy tunneling to a fake TLS server and drives the real path: connect_via_validated CONNECTs to the validated address, the proxy splices the tunnel, and tls_connect_upstream verifies the upstream certificate against the original hostname carried in SNI. It asserts the CONNECT authority is the validated IP and never the hostname, that verification succeeds for the matching hostname, and that a mismatched hostname is rejected — proving a rebinding or split-horizon substitution behind the proxy cannot pass certificate verification. Signed-off-by: Philippe Martin <phmartin@redhat.com> * fix(sandbox,podman): bound proxy-auth reads, reject port 0, fix stale comment Three review findings: - CWE-400: the proxy-auth credential file was read with an unbounded read_to_string on both the driver (sandbox-create) and supervisor (startup) paths, so a huge file or a special file such as /dev/zero could exhaust memory. Add a shared bounded reader in openshell-core that rejects non-regular files, caps the size at 4 KiB, and reads at most that many bytes; the driver runs it via spawn_blocking. Covers oversized, special-file, and missing-path cases on both sides. - Reject an upstream proxy URL with port 0: it passed the explicit-port check and startup validation but is not a connectable TCP port, so every proxied dial would fail later. Add a typed ZeroPort error with shared-validator and Podman-config tests. - Reword a driver-config comment that still described a 'reserved variable' to match the argv transport. Signed-off-by: Philippe Martin <phmartin@redhat.com> * fix(sandbox): open proxy-auth file non-blocking to reject FIFOs promptly read_upstream_proxy_credential_file opened the path with a blocking File::open before the regular-file check, so a configured FIFO with no writer would block open() indefinitely — hanging sandbox creation on the driver and supervisor startup. Open with O_NONBLOCK on Unix so the open returns immediately, then reject the non-regular file as before; O_NONBLOCK has no effect on the later read of a regular file. Adds a mkfifo regression asserting the reader returns promptly with a non-regular-file error instead of hanging. Signed-off-by: Philippe Martin <phmartin@redhat.com> * test(podman): cover corporate proxy egress across driver and supervisor The existing corporate-proxy tests construct config structs or call CONNECT helpers directly, so none of them detect a break in the wiring between layers: gateway TOML deserialization, the Podman argv and secret-mount semantics, supervisor CLI parsing, or policy denial before proxy contact. Add a Podman e2e that drives the whole chain against a fake authenticated forward proxy and asserts that an approved TLS request traverses it with a validated-IP CONNECT, a policy-denied destination is refused with 403 without ever reaching the proxy, credentials arrive through the mounted per-sandbox secret, and deleting the sandbox removes that secret. SupportContainer is a new harness fixture. Unlike ContainerHttpServer it probes readiness with a TCP connect rather than an HTTP GET, so it can host a forward proxy and TLS servers, and it exposes container logs and network IP for assertions. Signed-off-by: Philippe Martin <phmartin@redhat.com> * test(podman): restart the gateway on the proxy-config panic path The panic cleanup for the temporary corporate proxy configuration restored the gateway TOML but left the gateway process running with the temporary configuration still loaded, which could poison later test binaries in the same run. Nothing restarted it: the only ManagedGateway is the short-lived one inside restart_gateway, and its Drop only calls start, which does not reload config for an already-running gateway. Restore and synchronously stop/start the gateway in Drop, and set restored only after the normal restore and restart both succeed so a failed restart no longer suppresses the fallback. Signed-off-by: Philippe Martin <phmartin@redhat.com> * fix(supervisor-network): derive the loopback proxy bypass from resolved IPs The automatic bypass treated the host string "localhost" as proof that the destination was loopback and returned every resolved address. A sandbox controls its own /etc/hosts and resolve_socket_addrs consults it before DNS, so a workload could map localhost to any policy-allowed address and dial it directly, escaping the operator proxy and the inspection and audit boundary it exists to provide. Check the resolved addresses instead: the name bypasses only when the resolution is non-empty and every address is loopback. A mixed answer is not partially honored, and an IP literal is still authoritative for itself. A spoofed localhost falls through to the entries below, so an explicit operator NO_PROXY entry is still honored. This matches the trust model detect_trusted_host_gateway already applies to the same hosts file, which validates the mapped address rather than the alias. Signed-off-by: Philippe Martin <phmartin@redhat.com> * fix(podman): clean up proxy-auth secret when container already deleted The delete_sandbox early-return path cleaned up the token secret but skipped the proxy-auth secret, leaking it on disk. Also update tests for recent API changes (Optional socket_path, workspace field, list-based container lookup). Signed-off-by: Philippe Martin <philippe@openshell.dev> Signed-off-by: Philippe Martin <phmartin@redhat.com> --------- Signed-off-by: Philippe Martin <phmartin@redhat.com> Signed-off-by: Philippe Martin <philippe@openshell.dev> |
||
|
Evan Lezar
|
850bd42e8a |
refactor(tui): default create form state (#2458)
Signed-off-by: Evan Lezar <elezar@nvidia.com> |
||
|
Roland HussandEvan Lezar
|
b422b6783a |
feat(cli): add --output json/yaml to sandbox get, status, and sandbox create (#1989)
* feat(cli): add --output json/yaml to sandbox get, status, and sandbox create Add structured output support (JSON and YAML) to sandbox lifecycle commands: sandbox get, status, and sandbox create. Introduce ProgressOutput enum to cleanly separate interactive, plain, and silent display modes during sandbox provisioning. Signed-off-by: Roland Huß <rhuss@redhat.com> * fix(cli): reject structured create output with side effects Signed-off-by: Evan Lezar <elezar@nvidia.com> * fix(cli): suppress upload ssh stdout Signed-off-by: Evan Lezar <elezar@nvidia.com> --------- Signed-off-by: Roland Huß <rhuss@redhat.com> Signed-off-by: Evan Lezar <elezar@nvidia.com> Co-authored-by: Evan Lezar <elezar@nvidia.com> |
||
|
Piotr MlocekandEvan Lezar
|
f7cd910829 |
fix: eliminate parallel Rust test flakes (#2434)
* fix: eliminate parallel Rust test flakes Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> * refactor(supervisor-network): name workload proxy TCP connection Signed-off-by: Evan Lezar <elezar@nvidia.com> * refactor(supervisor-network): name procfs address family Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> --------- Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> Signed-off-by: Evan Lezar <elezar@nvidia.com> Co-authored-by: Evan Lezar <elezar@nvidia.com> |
||
|
Evan Lezar
|
21da343c9f |
refactor(supervisor): pass agent proposal state explicitly (#2421)
* test(supervisor-network): default L7 eval context in tests Signed-off-by: Evan Lezar <elezar@nvidia.com> * refactor(supervisor): pass agent proposal state explicitly Signed-off-by: Evan Lezar <elezar@nvidia.com> --------- Signed-off-by: Evan Lezar <elezar@nvidia.com> |