* perf(net): set TCP_NODELAY on all tunnel and proxy TCP hops
The sandbox tunnel added ~44 ms of latency to every small
request/response because no socket in the path disabled Nagle's
algorithm, so sub-MSS writes waited on delayed ACKs at each hop.
Set TCP_NODELAY on every latency-sensitive TCP socket:
- gateway: accepted connections on the public listener (gRPC relay
frames and WS tunnel writes)
- CLI: edge tunnel local accept + underlying WebSocket TCP stream,
insecure TLS connector (tonic's default connector already does
this), and service-forward accepted sockets
- supervisor: direct-tcpip connect into the sandbox netns, TCP relay
target dials, egress proxy accepted connections, and all upstream
CONNECT/HTTP dials (via a new connect_upstream helper)
Setting TCP_NODELAY on connect is best-effort: a failure only costs
latency, so we log and continue rather than fail the connection.
The sandbox SSH transport rides a unix domain socket and gRPC client
channels use tonic defaults (nodelay on), so no change is needed there.
Fixes#2219
Signed-off-by: Jim Meyer <jim@meyer4hire.com>
* refactor(net): house TCP_NODELAY helper in a shared net module
Address review feedback on the TCP_NODELAY change:
- Move the shared set-nodelay helper out of supervisor_session into a
new crate-private `net` module in openshell-supervisor-process, so
ssh and supervisor_session no longer reach across modules through a
pub(crate) item.
- Make the best-effort comments at each call site terse and consistent.
No behavior change; the benchmark ladder reproduces the same numbers.
Signed-off-by: Jim Meyer <jim@meyer4hire.com>
* refactor(net): consolidate TCP_NODELAY helpers into openshell_core::net
Move the best-effort TCP_NODELAY helpers into the shared openshell_core::net
module so every crate dials and configures sockets the same way:
- Add set_tcp_nodelay_best_effort (accepted/existing streams) and
connect_tcp_nodelay_best_effort (dial + set) with unit tests.
- Migrate all call sites in openshell-cli, openshell-server, and the
supervisor crates to the shared helpers.
- Remove the crate-private net module from openshell-supervisor-process.
- Document socket guidance in AGENTS.md (Network Sockets).
Signed-off-by: Jim Meyer <jim@meyer4hire.com>
* perf(net): set TCP_NODELAY on exec bridge and metadata server
The gateway-side single-use SSH-over-relay loopback bridge and the
sandbox IMDS metadata server were missed latency-sensitive TCP hops.
Set TCP_NODELAY on the accepted client connection and both russh client
dials of the exec bridge — interactive keystrokes and line-buffered PTY
output are the most tinygram-heavy traffic in the system — and on the
metadata server's accepted connections.
Also log unrecognized MaybeTlsStream variants in the edge tunnel so a
future TLS-backend change surfaces a silent TCP_NODELAY miss instead of
skipping it quietly.
Signed-off-by: Jim Meyer <jim@meyer4hire.com>
* perf(net): set TCP_NODELAY on openshell-sdk socket paths
The openshell-sdk crate landed on main with its own copies of the CLI's
hand-rolled sockets, which the CLI and TUI are meant to consume. Give
them the same treatment as the CLI equivalents:
- edge_tunnel: the accepted local tunnel connection and the WebSocket's
underlying TCP socket (plain and rustls variants).
- transport: the dial in InsecureTlsConnector, tonic's custom-connector
path.
Only these hand-rolled sockets need it. Tonic's own connector defaults
tcp_nodelay to true and applies it itself, so plain Endpoint::connect
callers were already covered.
Signed-off-by: Jim Meyer <jim@meyer4hire.com>
---------
Signed-off-by: Jim Meyer <jim@meyer4hire.com>
Update the docs for `sandbox create` and `exec` to dissuade use of `--env` for
secrets, and enhance the docs for `--provider` to explain what it's for.
Signed-off-by: Kris Hicks <khicks@nvidia.com>
Document Bazel targets alongside existing mise commands with a prominent
experimental notice linking to RFC 0012. Includes Bazelisk install
instructions, .bazelignore guidance for Cargo coexistence, and a mapping
of available build and test targets.
Signed-off-by: Roland Huß <rhuss@redhat.com>
Keep the completed instrumented handler future in an explicit pinned box and
drop it after the await. This releases the handler-side request span clone
before the disconnect test checks producer ownership, avoiding compiler- and
platform-dependent retention of an unfinished span.
Signed-off-by: Kris Hicks <khicks@nvidia.com>
* feat(build): add system CA root mode
Allow distro builds to use native trust stores for supervisor upstream TLS while keeping bundled Mozilla roots as the default. Avoid bundled root crates in system-ca-roots builds by using native-root TLS features and z3 0.20.
Signed-off-by: Adam Miller <admiller@redhat.com>
* fix(build): keep CA root feature in telemetry-off verification
The telemetry-off task uses --no-default-features which now disables
bundled-ca-roots in addition to telemetry, triggering the compile_error
guard. Re-enable bundled-ca-roots explicitly so the task verifies only
telemetry compilation.
Signed-off-by: Scott Burdine <sburdine@nvidia.com>
Signed-off-by: politerealism <burdcat17@gmail.com>
* fix(sdk): disable oauth2 default features to prevent webpki-roots leak
The bare `oauth2 = "5"` dependency re-enabled default features
(rustls-tls → reqwest/rustls-tls → webpki-roots), defeating the
system-ca-roots feature gate. Mirror the CLI fix: disable defaults
and enable only the `reqwest` feature.
Signed-off-by: Quinn Burdine <sburdine@redhat.com>
Signed-off-by: politerealism <burdcat17@gmail.com>
* refactor(build): simplify CA root selection to single feature toggle
Replace mutually exclusive bundled-ca-roots / system-ca-roots features
with a single bundled-ca-roots toggle. Disabling it implies system roots
via rustls-native-certs, which is now a regular (non-optional) dependency.
This fixes cargo --all-features and simplifies the distro build interface
from --no-default-features --features system-ca-roots to just
--no-default-features.
Signed-off-by: Quinn Burdine <sburdine@redhat.com>
Signed-off-by: politerealism <burdcat17@gmail.com>
* feat(build): add system-ca-roots convenience alias and fix verify task
Add a system-ca-roots feature alias on openshell-sandbox that includes
all other defaults (telemetry) except bundled-ca-roots, so distro
builds can use --no-default-features --features system-ca-roots without
manually re-adding unrelated defaults. Update the verify CI task to use
the alias and scope checks to the sandbox package. Fix task description
to use "build mode" terminology instead of implying a Cargo feature.
Signed-off-by: Quinn Burdine <sburdine@redhat.com>
Signed-off-by: politerealism <burdcat17@gmail.com>
* ci: fix system CA roots step name to use build mode terminology
Signed-off-by: Quinn Burdine <sburdine@redhat.com>
Signed-off-by: politerealism <burdcat17@gmail.com>
* refactor(sandbox): reorder features to place system-ca-roots alias near default
Signed-off-by: Quinn Burdine <sburdine@redhat.com>
Signed-off-by: politerealism <burdcat17@gmail.com>
* fix(proxy): unwrap Result from build_upstream_client_config in tests
The function signature changed to return Result but the test call sites
were not updated, causing type mismatch compilation errors in CI.
Signed-off-by: Quinn Burdine <sburdine@redhat.com>
Signed-off-by: politerealism <burdcat17@gmail.com>
---------
Signed-off-by: Adam Miller <admiller@redhat.com>
Signed-off-by: Scott Burdine <sburdine@nvidia.com>
Signed-off-by: politerealism <burdcat17@gmail.com>
Signed-off-by: Quinn Burdine <sburdine@redhat.com>
Co-authored-by: Adam Miller <admiller@redhat.com>
Repeated Bazel test targets with --runs_per_test at 10, 20, 50, 100, and
200 runs. The tests failed intermittently when multiple instances ran
concurrently. This indicates timing, shared tracing state, socket readiness,
or parallel-safety issues that need focused follow-up before re-enabling the
tests.
A normal non-repeated bazel test //... run passes with these tests ignored.
Disabled tests:
- sandbox_forward_foreground_fails_when_ssh_exits_before_listener_opens
- sandbox_forward_background_terminates_owned_child_when_listener_never_opens
- podman_socket_probe_accepts_successful_ping_response
- podman_socket_probe_rejects_docker_ping_response
- docker_socket_probe_accepts_successful_ping_response
- docker_socket_probe_rejects_podman_ping_response
- docker_socket_detection_returns_the_responsive_candidate
- podman_socket_detection_returns_the_responsive_candidate
- driver_watch_events_are_roots_and_store_operations_have_parents
- reconcile_sweeps_are_roots_and_operations_have_parents
- gateway_listeners_bind_ipv6_wildcard_and_ipv4_callback_on_same_port
- watch_producer_releases_request_span_when_client_disconnects
- expected_conflicts_leave_the_span_unmarked
- store_spans_record_what_they_touched_as_attributes
- store_operations_export_spans_with_parents
- refresh_worker_ticks_are_roots_and_store_operations_have_parents
Signed-off-by: Simon Scatton <sscatton@nvidia.com>
Continue distributed traces across the gateway-to-driver process boundary and
export VM driver spans to the same OTLP/gRPC collector. The driver reports as
the distinct openshell-driver-vm service.
Updated the gateway architecture and configuration reference with a generic
external-driver forwarding contract.
Instrumented:
- Every RemoteComputeDriver RPC injects the active W3C trace context into
tonic metadata. Managed VM readiness and runtime initialization give startup
capability probes stable parent operations rather than isolated root spans.
- A tonic service layer creates fixed, low-cardinality server spans for every
ComputeDriver RPC. New handlers inherit tracing automatically; failures record
OpenTelemetry error status and the gRPC status code.
- Background provisioning remains attached to CreateSandbox after the RPC
returns without extending the RPC span lifetime.
- Provisioning records image preparation, bootstrap image resolution, overlay
preparation, lifecycle configuration, pre-launch hooks, guest preparation,
and launcher spawn as child spans.
- VM startup reconciliation roots one trace for the persisted-sandbox scan,
with per-sandbox restore and provision operations beneath it. The root remains
open until all spawned restore tasks finish.
- Delete cleanup records its own child operation.
Design notes:
- The gateway forwards its configured OTLP endpoint to managed external
drivers. SDK `OTEL_*` variables continue to own sampling, batching, limits,
headers, and transport tuning.
- The VM driver has its own tracer provider and service resource so trace
backends preserve the service boundary.
- RPC operation names come from an explicit method mapping, keeping cardinality
bounded without parsing the protobuf descriptor set at runtime.
- Propagation uses a remote SpanContext for spawned provisioning. This keeps
one trace while allowing the CreateSandbox server span to finish when the
RPC response is sent.
- Startup restoration is independent of gateway requests. It begins at the VM
driver reconciliation span rather than attaching to an unrelated RPC.
- Existing tracing events remain on the logging path. The OpenTelemetry layer
exports spans only and excludes the SDK exporter callsites to avoid recursive
traces.
- Export configuration failures do not prevent the driver from serving, and
buffered spans are drained during graceful shutdown.
- Trace fields identify drivers, sandboxes, images, lifecycle phases, and gRPC
outcomes without recording credentials, sandbox tokens, or request query
parameters.
Refs #2507
Signed-off-by: Kris Hicks <khicks@nvidia.com>
Run the constructor gateway-discovery error test only on Linux, matching the
production branch that inspects the Podman bridge gateway. macOS uses its
Podman machine callback path and correctly skips that inspection.
Signed-off-by: Kris Hicks <khicks@nvidia.com>
Establish the Phase 1 reference implementation proposed by RFC 0012
while retaining the existing Cargo and Mise workflows during evaluation.
- pin Bazel 9.1.1 and configure Bzlmod, rules_rs, LLVM, protobuf, and
Rust 1.95 toolchains
- import third-party crates from Cargo metadata and propagate the
workspace version into Bazel targets
- add library, binary, proc-macro, unit-test, and integration-test
targets across the supported Rust workspace crates and drivers
- generate protobuf Rust sources and descriptor sets under Bazel while
preserving Cargo-compatible generated-code imports
- annotate aws-lc-sys and zstd-sys native dependencies, build Z3 4.15.2
from source, and generate z3-sys bindings
- make CLI and procfs test fixtures available as explicit Bazel inputs
without relying on fixed host binary paths
- define optimized release targets for Linux x86_64 and aarch64 CLI,
sandbox, and gateway binaries, plus macOS aarch64 artifacts
- add Bazel, buildifier, and lcov to the Nix development environment
RFC: 0012 (rfc12 branch)
Refs: #2491
Signed-off-by: Simon Scatton <sscatton@nvidia.com>
The log prefix value passed to nft contains colons (e.g.
openshell:bypass:sandbox-cc817378:) but was not wrapped in double
quotes. Since nft concatenates argv entries and parses the result,
the bare colons cause a syntax error that silently prevents all
bypass-attempt LOG rules from installing.
Wrap log prefix values in nft-quoted strings via a new nft_quote()
helper that strips embedded double-quotes (nft quoted strings don't
support escape sequences). All four log-rule generation sites (TCP
and UDP, for both per-sandbox and sidecar rulesets) are updated.
Fixes#2470
Signed-off-by: Grace Smith <grasmith@redhat.com>
* test(server): stabilize watch span cancellation test
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
* test(server): poll span exporter at fixed interval
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
---------
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
* fix(podman): combine sandbox stop and removal
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
* fix(podman): use libpod volume removal parameter
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
* fix(podman): preserve removal timeout margin
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
* test(podman): use virtual time for removal delay
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
---------
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
Give each span-assertion test its own thread-scoped in-memory exporter so
parallel tests cannot contaminate or reset captured spans. Keep a bare global
tracing registry only to preserve callsite interest, and serialize scoped
subscriber changes because tracing caches that interest process-wide.
Remove the test-only OTLP collector, polling delivery barrier,
transport-specific test, and direct opentelemetry-proto dependency. Seed the
expected persistence conflict before tracing begins so its assertion window
contains only the operation under test.
Signed-off-by: Kris Hicks <khicks@nvidia.com>
Extract common OpenTelemetry provider construction into openshell-otel so
OpenShell services share one OTLP/gRPC export implementation.
The shared crate owns:
- typed exporter setup errors and non-fatal provider enablement;
- endpoint trimming and URI validation before lazy exporter connection;
- fixed and environment-or-default service-name policies;
- service version and caller-supplied resource attributes;
- batch tracer-provider construction; and
- span-only tracing layers that exclude OpenTelemetry exporter callsites.
Migrate the gateway to the shared provider while retaining its configurable
service name, error marking, and tracing test collector. Add the shared crate
to the architecture inventory and document the tracing boundary.
Refs #2507
Signed-off-by: Kris Hicks <khicks@nvidia.com>
Add a published Issue Triage and Lifecycle page and align AGENTS.md,
CONTRIBUTING.md, README.md, the PR template, and the issue-handling
skills on the state:*/agent:* label model.
Signed-off-by: Kris Hicks <khicks@nvidia.com>
Add an opt-in OTLP/gRPC trace exporter to the gateway. Export is enabled
by the presence of an `[openshell.gateway.otlp]` table with an endpoint;
there is no separate toggle.
Instrumented:
- Inbound request server spans, named for the RPC (`$service/$method`) or
`{method} {path}` for plain HTTP. They continue valid W3C `traceparent`
context when present and start a new trace otherwise. gRPC spans also
carry `rpc.system`, `rpc.service`, `rpc.method`, and trailer-derived
`rpc.grpc.status_code`.
- Compute driver calls (create, delete, list, get, validate, watch) as
client spans anchored on the `ComputeDriver` contract.
- Store reads and writes as children of the current request or loop span.
- Work with no inbound request: compute driver initialization, the sandbox
reconcile sweep, provider credential refresh tick, and driver watch events.
Each roots one operation trace so its child work does not arrive as anonymous
single-span traces.
This is deliberately not exhaustive. Auth, policy evaluation, and
middleware remain uninstrumented, as do store lifecycle calls (`ping`,
`close`) that a readiness poll would turn into a span per tick. The aim is
a useful trace tree at a reviewable size; coverage can grow against real
traces.
Design notes:
- The TOML table owns whether and where to export. The SDK `OTEL_*`
variables own how; sampling, batching, and limits are not mirrored into
gateway config.
- The OpenTelemetry layer exports spans only. Existing `tracing` events
remain on the stdout and sandbox-log paths and are not copied into trace
payloads.
- Telemetry never blocks the gateway. A malformed endpoint logs an error
and disables export rather than failing startup, and buffered spans are
drained during graceful shutdown.
- Failed spans carry error status without a separate `error.type` attribute.
Request spans use HTTP status and gRPC response trailers; driver spans use
the returned gRPC status; autonomous loop spans record failed results
explicitly. Store spans exempt `UniqueViolation` and `Conflict`, because
those errors report expected contention such as a held lease or an
optimistic-concurrency retry.
- The compute driver is reachable only through `TracedDriver::call`, so a
call cannot skip its span. This is the client half of a client/server pair
and the single place to inject context if drivers move out of process.
- Tests share one process-wide subscriber and in-memory exporter because
`tracing` caches callsite interest globally.
Inbound W3C trace context is propagated into gateway request spans. Context
is not yet injected into outbound driver calls, so a future out-of-process
driver would still need propagation at the `TracedDriver` seam.
The Helm chart is intentionally unchanged, so OTLP export cannot yet be
enabled on a chart-deployed gateway.
Refs #2507
Signed-off-by: Kris Hicks <khicks@nvidia.com>
* feat(auth): implement RFC 0011 Phase 2 workspace authorization
Signed-off-by: Derek Carr <decarr@redhat.com>
* fix(auth): address PR review feedback on workspace authorization
- Docker e2e: add --health-port and switch readiness probe from
`openshell status` to `curl /healthz`, fixing a false-positive
readiness check in OIDC mode where the CLI exited 0 without
actually contacting the gateway
- ListWorkspaces: move membership filtering from post-query N+1
lookups into a SQL EXISTS subquery so pagination applies to the
visible set, not the global ordering. Add generic
list_with_membership to the persistence layer.
- Descriptor validator: reject role/scope fields on unauthenticated
and sandbox auth modes, and allow-list workspace_role as
user/admin and global_role as platform_admin to catch typos at
startup
Signed-off-by: Derek Carr <decarr@redhat.com>
* fix(server): use authed request in delete telemetry test
The workspace authorization added by the Phase 2 auth changes requires
a Principal on every delete request. The delete-telemetry test was still
using a bare Request::new, so extract_principal failed before the handler
could acquire the delete gate, causing a 5-second timeout flake.
Signed-off-by: Derek Carr <decarr@redhat.com>
* fix(auth): address gator review findings for workspace authorization
- Inject unauthenticated-local-dev principal in no-auth gateway mode so
handlers that call extract_principal() always find one.
- Cap label-selector membership query at MAX_PAGE_SIZE instead of
u32::MAX to bound the in-memory read.
- Authorize workspace membership before resolving workspace existence in
all sandbox RPCs to prevent workspace-name enumeration by non-members.
- Remove dead_code allow on AuthorizedWorkspace.workspace now that
callers use the normalized name from the authz result.
Signed-off-by: Derek Carr <decarr@redhat.com>
* fix(auth): close workspace-name oracle and label-selector truncation
Swap authorize-before-resolve ordering in 27 handlers across
provider.rs, service.rs, policy.rs, and workspace.rs to prevent
CWE-203 workspace-name enumeration by non-members.
Add combined membership+label SQL query (list_with_membership_and_selector)
to both persistence backends so ListWorkspaces with label selectors no
longer silently drops results beyond the first page of membership matches.
Signed-off-by: Derek Carr <decarr@redhat.com>
* test(auth): add non-member rejection and membership+label persistence tests
Add comprehensive test coverage for workspace authorization changes:
- Non-member rejection tests across all 44 workspace-scoped handlers
(sandbox, provider, service, policy, workspace, inference) verifying
PERMISSION_DENIED is returned instead of NOT_FOUND to prevent
CWE-203 workspace-name oracle
- Persistence test for list_with_membership_and_selector verifying
SQL-level membership EXISTS + label filtering, multiple predicates,
no-match cases, and pagination
Signed-off-by: Derek Carr <decarr@redhat.com>
* fix(auth): format merged import line in sandbox tests
Signed-off-by: Derek Carr <decarr@redhat.com>
* fix(auth): address gator re-review findings on workspace authorization
- Fix TUI unconditionally setting providers_v2_enabled after provider
refresh; read the actual gateway setting via GetGatewayConfig at
startup instead
- Fix SQLite json_extract with dotted label keys (e.g. example.com/env)
by quoting the key in the JSON path
- Add authed_request wrappers to upstream OCI identity tests that were
missing a principal after rebase
- Add test proving GetGatewayConfig is accessible without Platform Admin
- Add test for dotted/prefixed Kubernetes-style label key filtering
Signed-off-by: Derek Carr <decarr@redhat.com>
* fix(auth): address second gator re-review findings
- Loosen GetGatewayConfig from platform_admin to scope-only so workspace
users can discover providers_v2_enabled during sandbox creation with
inferred-provider commands; update proto descriptor, descriptor
validation, and RFC 0011 access table
- Add validate_label_selector to handle_list_workspaces and escape
single quotes in SQLite json_extract interpolation (CWE-89
defense-in-depth)
- Re-fetch providers_v2_enabled after TUI gateway switch so the new
gateway's capability is reflected
- Add e2e test for workspace user with inferred-provider command
- Add persistence test for adversarial label keys with SQL injection
attempts
- Add handler test for invalid label selector rejection in
ListWorkspaces
Signed-off-by: Derek Carr <decarr@redhat.com>
* fix(auth): address third gator review findings
- Cap label selector pairs at 64 (CWE-400) to bound SQLite dynamic SQL
- Add SCOPE_ONLY_METHODS allowlist for scope-without-role RPCs (CWE-863)
- Normalize ID-based data-plane handlers to return NOT_FOUND for
unauthorized sandboxes, closing the cross-workspace oracle (CWE-203)
- Fix TUI provider profile cache lookup key mismatch for legacy
providers with empty profile_workspace
- Add whoami to CLI skill reference command tree
- Update TUI skill doc with workspace, provider, and settings coverage
- Document scope/workspace orthogonality on GetGatewayConfig proto
Signed-off-by: Derek Carr <decarr@redhat.com>
* fix(auth): extend CWE-203 normalization to policy.rs sandbox handlers
GetSandboxConfig and GetSandboxLogs in policy.rs had the same
fetch-before-authorize pattern that leaked cross-workspace sandbox
existence. Promote fetch_and_authorize_sandbox to pub(super) and
use it from both sandbox.rs and policy.rs handlers.
Signed-off-by: Derek Carr <decarr@redhat.com>
* test(auth): update assertions for CWE-203 sandbox ID normalization
Cross-workspace sandbox access via ID-based handlers now returns
NOT_FOUND instead of PERMISSION_DENIED to prevent existence inference.
Update the unit test and OIDC e2e assertion to match.
Signed-off-by: Derek Carr <decarr@redhat.com>
* fix(auth): narrow CWE-203 error mapping and correct whoami output formats
Only remap PERMISSION_DENIED to NOT_FOUND in fetch_and_authorize_sandbox
and RevokeSshSession, letting INTERNAL and UNAUTHENTICATED propagate
as-is. Fix whoami --output format values in cli-reference.md to match
the actual CLI (table/json/yaml, not text/json).
Signed-off-by: Derek Carr <decarr@redhat.com>
* fix(ci): share network namespace with Keycloak in containerized CI
In GitHub Actions job containers, Docker port publishing lands on the
host, not inside the job container. Detect this environment and attach
Keycloak to the job container's network namespace instead, with
hardened defaults (cap-drop ALL, no-new-privileges, loopback-only
listener).
Signed-off-by: Derek Carr <decarr@redhat.com>
---------
Signed-off-by: Derek Carr <decarr@redhat.com>
ForwardSpec accepts IPv6 bind addresses, but ssh_forward_arg() emitted
them unbracketed (e.g. ::1:8080:127.0.0.1:8080), which OpenSSH rejects
as a bad local forwarding specification. access_url() likewise produced
invalid URLs like http://::1:8080/.
Extract a shared bracket_ipv6_host() helper and use it in
ssh_forward_arg(), access_url() (via format_gateway_url), and
format_gateway_url() so IPv6 literals are bracketed consistently.
Fixes#2279
Signed-off-by: Russell Bryant <rbryant@redhat.com>
The subprocess-based integration tests inherit the full parent
environment. If the developer has OPENSHELL_GATEWAY_INSECURE=true
set in their shell, it leaks into the spawned CLI process and causes
it to connect with .with_no_client_auth(), skipping the mTLS client
certificate. The test server requires mTLS, so it responds with
CertificateRequired and the test fails.
Strip OPENSHELL_GATEWAY_INSECURE, OPENSHELL_GATEWAY,
OPENSHELL_GATEWAY_ENDPOINT, and OPENSHELL_WORKSPACE from the
subprocess environment. The test already sets --gateway and
--gateway-endpoint explicitly via CLI args, so these env vars
should not influence the subprocess behavior.
Reported-by: Seth Jennings
Signed-off-by: Roland Huß <rhuss@redhat.com>
* refactor(compute): extract create_sandbox_record and update_sandbox_record helpers
Split apply_sandbox_update_locked into named helpers to make the two
distinct paths explicit: create_sandbox_record for first-observation events
and update_sandbox_record for subsequent driver snapshots on existing sandboxes.
The dispatcher now uses a match on the existing record rather than an
early-return guard.
No behavior change.
Signed-off-by: Evan Lezar <elezar@nvidia.com>
* refactor(compute): make sandbox readiness gateway-owned across all drivers
Introduce compute_phase_components and apply_readiness_conditions to
centralise the gateway's phase composition logic. The public SandboxPhase
is now determined by combining the backend phase reported by the driver with
supervisor session presence, independent of the driver implementation.
Remove SupervisorReadiness from the driver contract. Running containers
always report BackendReady; the gateway owns the Ready decision. Rename
the dispatcher match to three arms so that status-less events for existing
sandboxes are a documented no-op rather than a silent pass-through.
Drop backend_ready_no_session and the SupervisorNotConnected condition.
The BackendReady driver condition plus the Provisioning phase already
communicates that the backend is up but the supervisor has not connected.
The redundant condition added noise without new information.
Closes#1951
Signed-off-by: Evan Lezar <elezar@nvidia.com>
* fix(compute): expose disconnected supervisor readiness
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
* docs(sandboxes): clarify supervisor readiness lifecycle
Signed-off-by: Evan Lezar <elezar@nvidia.com>
---------
Signed-off-by: Evan Lezar <elezar@nvidia.com>
Signed-off-by: Drew Newberry <anewberry@nvidia.com>
Co-authored-by: Drew Newberry <anewberry@nvidia.com>
parse_duration_to_ms was moved to commands/common.rs since the
original PR was opened, but it still split the last byte of the
input with split_at(s.len() - 1), which panics when the final
character is multi-byte UTF-8 (e.g. 'openshell logs my-sandbox
--since 5€').
Split off the last character using its UTF-8 length instead, so
invalid units surface the intended 'unknown duration unit' error.
Add regression tests in commands/common.rs.
Signed-off-by: Andrew White <andrewh@cdw.com>
The Kubernetes driver's default workspace PVC never set storageClassName,
so on clusters with no default StorageClass the PVC stayed Pending and
sandbox creation failed.
Add a workspace_storage_class option to KubernetesComputeConfig, wired
through SandboxPodParams into the generated volumeClaimTemplates. When
non-empty it sets storageClassName; empty preserves the current behavior
of relying on the cluster default StorageClass.
Expose it via the OPENSHELL_K8S_WORKSPACE_STORAGE_CLASS env var on both
the standalone driver and the embedded gateway runtime defaults, and via
the server.workspaceStorageClass Helm value.
Closes#2442
Signed-off-by: lr90 <qiuweimin@matrixorigin.cn>
Replace yield_now() spin-loop with sleep(10ms) polling in
delete_handler_ends_telemetry_for_the_resolved_sandbox_id. The
single-threaded tokio runtime starves the spawn_blocking threads
used by SQLite when yield_now() burns 100% CPU waiting for the
delete gate entry count. Increase the timeout from 1s to 5s for
consistency with similar guard tests.
Signed-off-by: Derek Carr <decarr@redhat.com>
Move gateway management functions (gateway_status, gateway_info,
gateway_add, gateway_select, gateway_login, gateway_logout, gateway_list,
gateway_remove, gateway_use, gateway_info_not_configured) and their
private helpers from run.rs into commands/gateway.rs. Add pub use
re-exports in run.rs so main.rs call sites remain unchanged.
Extract shared test utilities (EnvVarGuard, with_tmp_xdg) into a
crate-level test_utils module to avoid duplication between gateway
tests and the remaining run.rs tests.
Part of #2304 (PR 2/6).
Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com>
Remove vm_slow_progress_before_ready from the subprocess-based
structured output tests. The 1.8s artificial delay (3x 600ms sleeps)
left only ~0.7s headroom within the 5s OPENSHELL_PROVISION_TIMEOUT,
causing sporadic TLS handshake failures on loaded Linux machines.
Root cause analysis:
- The CertificateRequired TLS alert occurs when the provision timeout
fires mid-handshake, causing the client to drop the connection before
presenting its certificate.
- tokio::process::Command is NOT the cause: it is a thin async wrapper
around std::process::Command that delegates environment setup, process
spawning, and I/O identically.
- The single-threaded tokio runtime is NOT the cause: testing with
multi_thread flavor produces identical timing.
- TempDir lifetime is sound: the directory lives until .output().await
returns, so certs exist for the subprocess's entire lifetime.
- Cert path resolution is deterministic: the subprocess resolves
XDG_CONFIG_HOME/openshell/gateways/openshell/mtls/ correctly.
The subprocess tests exist to verify that the real CLI binary produces
clean stdout (no ANSI chrome, spinners, or progress text leaking into
structured output). This invariant does not depend on slow progress
events; the ProgressOutput::Silent branch that swallows progress events
is already covered by the in-process test
sandbox_create_keeps_waiting_while_vm_progress_arrives.
Fixes#2501
Signed-off-by: Roland Huß <rhuss@redhat.com>
The Stderr arm of the interactive exec event loop locked stdout,
so stderr payloads were written to stdout. The non-interactive exec
path already handles the same event with stderr. Write stderr
payloads to stderr so shell redirection (e.g. 'openshell exec ...
2>err.txt') works in interactive mode.
Signed-off-by: Andrew White <andrewh@cdw.com>
Jim Meyer