* chore(ci): disable telemetry in internal test runs
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* test(ci): remove brittle telemetry wiring test
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* docs: trim CI telemetry guidance
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* test(e2e): share telemetry default with OpenShift
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
---------
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
* fix(policy): prevent implicit authorization inheritance
A network rule authorizes every listed binary to reach every listed
endpoint, so unioning an AddRule operation's binaries and endpoints
independently grants binary-by-endpoint pairs the operation never
declared. Require AddRule to declare the complete product before
merging, and reject an operation that would give one host and port two
different MCP inspection contracts. The rejection names the binaries the
operation still has to declare.
Fix proposal coverage on the same surface. An any-binary proposal was
vacuously covered by a binary-restricted loaded rule, and a complete
product split across several loaded rules was reported as uncovered.
Coverage compares merge-widened endpoint fields by containment and
exact-matches only the fields the merge never widens, so a policy the
gateway just merged always reads back as covered and the sandbox
policy.local /wait long-poll cannot spin to its deadline.
Signed-off-by: Shiju <shiju@nvidia.com>
* fix(policy): close authorization-inheritance gaps in merge and coverage
Coverage treated an unset proposal value for a field the merge retains as a
request for the default, so a proposal that merged cleanly into an endpoint
carrying enforcement, protocol, or tls read back as uncovered and left the
policy.local /wait long poll spinning. Unset now means unspecified.
Ports are a set on the wire but each port is an independent authorization.
Coverage and the inheritance check both resolve one binary, host, path, and
port at a time, so ports spread across loaded rules resolve and a complete
declaration split across incoming endpoints is accepted.
An incoming empty binary list means any binary. It now has to declare every
merged endpoint like a new concrete path does, and once declared the promotion
is applied instead of appending an empty list and leaving the restricted scope
in place.
The endpoint-overlap fallback folded a new rule name into an existing rule
where inheritance validation then rejected it, leaving no way to grant a binary
part of a rule. Folding now keeps the requested rule name when it would widen,
and reports that it did. An MCP contract conflict still propagates because one
host and port carry a single inspection contract.
AddAllowRules and AddDenyRules select an endpoint by host and port alone and
now reject a target that resolves more than once, including two paths on one
rule. RemoveBinary rejects an any-binary rule rather than reporting a success
that leaves the binary authorized.
Signed-off-by: Shiju <shiju@nvidia.com>
* fix(policy): gate port and MCP-contract widening independently of binary scope
The changed-port check only rejected when the operation also left an existing
binary undeclared. An operation listing every existing binary could therefore
declare one port of a multi-port endpoint and have its widened fields land on
the endpoint the merge shares across all of them, authorizing L7 rules on a
port it never named. Each changed port must now be named by the operation on
its own, independently of binary-scope coverage.
MCP contract compatibility was checked inside the endpoint fold, which only
compares endpoints agreeing on host, path, and a shared port. The sandbox
resolves one extended configuration per host and port and never consults the
path, so a second MCP endpoint under another path, or in another rule, left the
effective strict-tool-name, method-profile, and body-limit contract decided by
match order. Contract agreement is now enforced across the whole merged policy,
including provider-composed rules. A conflict already present in the baseline
is left alone so unrelated updates still apply.
An empty binary list authorizes any binary. Appending an incoming named list
made it non-empty and revoked every process the operation did not name, turning
an additive update into a silent mass revocation. An already-empty scope is now
kept and reported; only a restricted scope is replaced by an incoming
any-binary scope.
Warnings raised during a fold now name the rule that was actually modified
rather than the rule name the operation requested, which differ when the
endpoint-overlap fallback redirects the operation.
Signed-off-by: Shiju <shiju@nvidia.com>
* fix(policy): route undeclared-port conflicts through the separate-rule fallback
The fold-only classifier decides which merge errors disappear when the incoming
authorization stays on its own rule. UndeclaredPortWouldChange was added ahead
of the existing-binary conflict but never classified, so a differently named
narrow update against a multi-port endpoint failed outright instead of landing
separately. A same-key update still returns the error, because there the
operation chose the target.
The classifier is now an exhaustive match rather than a matches! with an
implicit false. A new variant defaulting to "not fold-only" is what withdrew the
separate-rule remedy here, so adding one has to be an explicit decision.
Inspection-contract agreement now covers protocol, not only MCP options. The
sandbox resolves one extended configuration per host and port and never
consults the path, so an MCP endpoint and a REST endpoint on the same host and
port left the effective inspection protocol decided by match order. Endpoints
with no protocol carry no contract and are skipped.
Signed-off-by: Shiju <shiju@nvidia.com>
* fix(policy): compare only MCP contracts when detecting endpoint conflicts
The post-merge conflict scan was broadened to compare inspection protocol as
well as MCP options on one host and port. The supervisor selects among matching
endpoint configs by most-specific path, so a broad REST endpoint and a narrower
GraphQL endpoint on the same host and port are unambiguous and supported. The
broader comparison rejected those updates even though the equivalent full policy
loads and serves correctly.
MCP options are not selected that way, so the scan keeps comparing them: two MCP
endpoints on one host and port still have to agree on strict tool names, method
profile, and body limit, whatever paths or rules hold them.
The policy page returns to describing the MCP-specific rule and the path-aware
selection it sits alongside.
Signed-off-by: Shiju <shiju@nvidia.com>
* fix(policy): keep MCP off a host and port shared with other inspection
Narrowing the conflict scan back to MCP options let an MCP endpoint sit under a
path already covered by a broader REST endpoint. The supervisor picks the parser
by most-specific path, so the MCP endpoint parses the request, but
_policy_allows_l7 is existential over every endpoint matching it. A plain REST
rule on the overlapping path can therefore make allow_request true for a
JSON-RPC tool call the MCP endpoint never allowed, and the relay forwards it.
The scan now records every inspected protocol on a host and port. MCP may not
share one with a differently inspected endpoint, and two MCP endpoints there
still have to agree on one contract. Endpoints that are not inspected carry no
contract and never compete, and two non-MCP endpoints stay supported because
they share one method-and-path rule vocabulary.
Signed-off-by: Shiju <shiju@nvidia.com>
---------
Signed-off-by: Shiju <shiju@nvidia.com>
* build(bazel): stage VM runtime bundle
Signed-off-by: Simon Scatton <sscatton@nvidia.com>
* build(bazel): add VM driver targets
Signed-off-by: Simon Scatton <sscatton@nvidia.com>
---------
Signed-off-by: Simon Scatton <sscatton@nvidia.com>
* feat(sdk/go): add Go SDK foundation, types, and sandbox client (A)
Add the Go SDK module with the full API contract and a working sandbox
client as the first vertical slice. All other resource clients are present
as stubs returning Unimplemented errors, to be replaced with real
implementations in subsequent PRs.
Contents:
- Module setup (go.mod, Makefile, mise.toml)
- All domain types (types/ package)
- Full ClientInterface with all sub-client accessors
- Shared infrastructure (errors, auth, gRPC connection, logging)
- Sandbox client with converter and tests (fully functional)
- Stub clients for remaining resources (exec, file, health, provider,
profile, config, refresh, policy, service, ssh, tcp)
Part of the Go SDK decomposition plan (#2270).
Implements #2044.
* fix(sdk/go): address review feedback on PR #2271
- Make scheme parsing drive transport selection: http:// uses plaintext
gRPC, https:// or no scheme uses TLS. Add regression tests.
- Add Resources and DriverConfig fields to SandboxTemplate and update
both converter directions (SandboxFromProto/SandboxSpecToProto).
- Regenerate proto bindings from current canonical proto sources to
eliminate drift (SigV4/MCP fields, params matchers, reserved fields).
- Run gofmt/goimports on all handwritten Go files.
Signed-off-by: Roland Huß <rhuss@redhat.com>
* fix(sdk/go): address principal engineer review findings
- Remove dead boolCount function that would fail golangci-lint (#1)
- Emit EventAdded for the first watch event instead of EventModified,
matching k8s watch semantics (#7)
- Add mutex locking to all mock server methods that access the shared
sandboxes map, fixing latent race conditions (#12)
- Skip HealthCheck integration test that calls an unimplemented stub (#13)
- Scope doc.go examples: mark sections for sub-clients not yet available
in this PR with "available in a future release" (#4)
- Document Config.Timeout/RetryPolicy/Logger and WatchOptions fields
as reserved for future use (#2, #6)
Signed-off-by: Roland Huß <rhuss@redhat.com>
* refactor(sdk/go): migrate mise config to centralized task include
Move Go SDK mise configuration from standalone sdk/go/mise.toml into
the project's centralized pattern:
- Add Go tools (go, golangci-lint, protoc-gen-go, protoc-gen-go-grpc)
to root mise.toml [tools] section
- Create tasks/go.toml with all SDK tasks using go: namespace prefix
and dir=sdk/go for working directory
- Update sdk/go/Makefile to reference namespaced task names
- Update proto:sync default path for monorepo layout
Addresses review feedback from drew on PR #2271 regarding mise
convention alignment.
Signed-off-by: Roland Huß <rhuss@redhat.com>
* refactor(sdk/go): remove UPSTREAM_VERSION standalone repo artifact
Remove sdk/go/proto/UPSTREAM_VERSION file and its exclusion from
proto:check. This was a leftover from the standalone repo prototype.
In a monorepo, proto drift is detectable via git diff between
sdk/go/proto/ and proto/ directly.
Signed-off-by: Roland Huß <rhuss@redhat.com>
* refactor(sdk/go): switch proto generation from protoc to buf
Replace raw protoc invocations with buf for Go SDK proto code generation,
aligning with the TS SDK approach (PR #2122).
- Add repo-level buf.yaml declaring proto/ as the buf module with lint
and breaking change detection config
- Add sdk/go/buf.gen.yaml configuring buf to generate Go code directly
from root proto/ (no more vendored .proto copies)
- Delete vendored .proto source files from sdk/go/proto/
- Rewrite go:proto:gen and go:proto:check mise tasks to use buf
- Remove go:proto:sync and go:proto:clean tasks (no longer needed)
- Add proto target to sdk/go/Makefile
- Add buf 1.72.0 to root mise.toml tool dependencies
- Include options.proto in generation (was stripped from vendored copies)
- Regenerate all .pb.go files via the new buf pipeline
Signed-off-by: Roland Huß <rhuss@redhat.com>
* test(sdk/go): add proto-converter field coverage detection
Use protobuf reflection to enumerate all fields on key proto messages
(SandboxSpec, SandboxTemplate, SandboxStatus, SandboxCondition,
SandboxPolicy) and compare against explicit handled/skipped sets in the
converter tests.
Unhandled fields produce warnings (t.Log), not failures, so proto
contributors are not forced to fix SDK converters in the same PR. Stale
entries in the handled set (removed proto fields) do fail, since they
indicate the converter references something that no longer exists.
A follow-up CI workflow will create GitHub issues when converter drift
lands on main.
Signed-off-by: Roland Huß <rhuss@redhat.com>
* fix(sdk/go): bump Go to 1.26 and fix errcheck lint violations
The upstream go.mod now has `toolchain go1.26.4`, which requires Go 1.26
to build golangci-lint. Bump the mise.toml Go version from 1.25 to 1.26
and wrap deferred Close() calls in test helpers to satisfy errcheck.
Assisted-By: 🤖 Claude Code
* feat(sdk/go): add ObjectMeta fields (annotations, workspace, deletion_timestamp)
Add three new proto ObjectMeta fields to Sandbox and Provider domain
types: Annotations (map), Workspace (string), and DeletionTimestamp
(*time.Time). Update converters in both directions, deep-copy maps at
the proto/SDK boundary, and add TimeFromMillisPtr/MillisFromTimePtr
helper functions.
Assisted-By: 🤖 Claude Code
* chore(sdk/go): regenerate proto bindings after rebase
Pick up workspace fields from upstream PR #2445 (Wire authorization
into workspace model). All request messages now include workspace
parameter in the generated Go bindings.
Assisted-By: 🤖 Claude Code
* feat(sdk/go): add workspace scoping to all RPC interfaces
Add workspace parameter to every sandbox-scoped RPC method across all
interfaces (Sandbox, Exec, File, Service, SSH, TCP, Config, Policy,
Provider, Profile, Refresh). The workspace string is passed as the
second parameter after ctx, following the convention workspace then
resource-name.
Key changes:
- SandboxInterface: all 10 methods gain workspace parameter
- sandbox_client.go: passes Workspace field in every proto request
- ListOptions: add AllWorkspaces field for cross-workspace queries
- All stub interfaces updated to match new signatures
- All sandbox client tests updated with "default" workspace
Assisted-By: 🤖 Claude Code
* chore(sdk/go): remove coverage.out from tracking
Assisted-By: 🤖 Claude Code
* fix(sdk/go): address review feedback from mrunalp
- Add RefreshStrategyAWSStsAssumeRole to match proto enum value 6,
fulfilling the "all domain types upfront" contract
- Wrap context.DeadlineExceeded and context.Canceled in StatusError
so IsDeadlineExceeded() and IsCancelled() helpers work correctly
- Return error from mapToStruct/SandboxSpecToProto instead of silently
discarding structpb.NewStruct failures on invalid template maps
Signed-off-by: Roland Huss <rhuss@redhat.com>
* fix(sdk/go): address remaining review items
- Wire go:ci into root ci task so SDK is tested in repository CI
- Fix gofmt formatting on converter files
- Add goimports to mise.toml tools
- Add coverage.out to .gitignore
- Add Go SDK section to AGENTS.md and CONTRIBUTING.md
- Add regression tests for context-error wrapping (IsDeadlineExceeded,
IsCancelled) and invalid template map rejection
- Remove panic from SandboxToProto, return error instead
Signed-off-by: Roland Huss <rhuss@redhat.com>
* fix(sdk/go): pin goimports version and update lockfile
Pin goimports to 0.48.0 instead of "latest" and regenerate mise.lock
to include the new entry.
Signed-off-by: Roland Huss <rhuss@redhat.com>
* fix(sdk/go): TLS.Insecure means skip-verify, not plaintext
Align TLS.Insecure semantics with the Rust SDK: Insecure: true now
uses TLS with InsecureSkipVerify (skip cert verification) instead of
switching to plaintext. Only the http:// scheme triggers plaintext.
This fixes token auth against dev/k3d gateways: StaticToken and
RefreshableToken require transport security, which real TLS (even
with InsecureSkipVerify) satisfies, but plaintext does not.
For http:// + token auth (dev gateways without TLS), wrap the auth
provider to override RequireTransportSecurity, matching the Rust
SDK's behavior where http:// accepts any auth mode.
Transport decision table (matches Rust SDK crates/openshell-sdk):
http:// + any TLS config -> plaintext (TLS config ignored)
https:// + Insecure: true -> TLS, skip cert verify
https:// + Insecure: false -> TLS, full verification
no scheme -> same as https://
Signed-off-by: Roland Huss <rhuss@redhat.com>
* feat(sdk/go): add missing policy proto fields
Add 6 previously silently dropped fields to the network policy types
and converters, preventing security-relevant data loss on round-trip:
NetworkEndpoint fields 19-23:
- CredentialSigning: SigV4 re-signing mode
- SigningService: AWS service name for SigV4
- SigningRegion: AWS region override for SigV4
- JsonRpcMaxBodyBytes: JSON-RPC body inspection limit
- Mcp: MCP-specific policy options (new McpOptions type)
L7Allow and L7DenyRule field 9:
- Params: MCP params matcher map for tools/call filtering
New type McpOptions with StrictToolNames and AllowAllKnownMcpMethods
optional booleans matching the proto definitions.
Signed-off-by: Roland Huss <rhuss@redhat.com>
* fix(sdk/go): enforce coverage test and extend to policy messages
Change coverage_test.go from t.Logf (silent) to t.Errorf so that
unhandled proto fields fail the test immediately. Add coverage tests
for NetworkEndpoint (23 fields), L7Allow (8 fields), L7DenyRule
(8 fields), and McpOptions (2 fields).
Any new proto field that is not in the handled set or explicitly
skipped now breaks the build, closing the silent-drift gap.
Signed-off-by: Roland Huss <rhuss@redhat.com>
* ci(sdk/go): add Go SDK job to branch-checks workflow
Add a Go SDK job to branch-checks.yml that runs mise run go:ci
(lint, build, test, proto-check, docs-check) on every PR. This
ensures the SDK is tested in CI, not just locally.
Signed-off-by: Roland Huss <rhuss@redhat.com>
* fix(sdk/go): address should-fix review items
#6 Fix broken godoc examples: add workspace parameter to all method
calls in doc.go that were broken after workspace scoping.
#7 Add Err field to Event[T]: Watch error events now carry the
underlying error instead of discarding it.
#8 Separate Unauthenticated from PermissionDenied: add
ErrorUnauthenticated code and IsUnauthenticated() helper. gRPC
Unauthenticated (401) now maps to its own code instead of
collapsing into PermissionDenied (403).
#9 Add Unwrap to StatusError: replace dead Details field with Cause
error field. StatusError.Unwrap() returns Cause, enabling
errors.Is/As unwrapping. FromGRPCError and contextError both
populate Cause.
Signed-off-by: Roland Huss <rhuss@redhat.com>
* ci(sdk/go): add go:format:check to CI pipeline
Add gofmt format verification to go:ci. Catches unformatted Go files
before they reach the PR. Fix formatting on coverage_test.go.
Signed-off-by: Roland Huss <rhuss@redhat.com>
* chore(sdk/go): remove Makefile in favor of mise tasks
All build, lint, test, and proto-gen tasks are already defined in
tasks/go.toml and invoked via mise. The Makefile was a leftover
that duplicated this and raised questions in review.
Signed-off-by: Roland Huß <rhuss@redhat.com>
* feat(sdk/go): sync proto bindings and add credential handle support
Regenerate Go proto bindings after rebase to pick up new
CredentialHandle message and Provider.credential_handles and
profile_workspace fields from upstream. Add domain types, converter
support, and proto field coverage tests for Provider and
CredentialHandle.
Signed-off-by: Roland Huß <rhuss@redhat.com>
* fix(sdk/go): reject plaintext auth leak and fix watch error handling
Reject http:// addresses when the auth provider requires transport
security instead of silently stripping the requirement. Remove the
insecureAuthWrapper that overrode RequireTransportSecurity.
Fix watch stream error handling: use blocking send for terminal
errors so they are never silently dropped when the channel is full,
and wrap mid-stream errors with converter.FromGRPCError so SDK error
helpers like IsUnavailable work on watch Event.Err.
Signed-off-by: Roland Huß <rhuss@redhat.com>
* fix(sdk/go): address review findings from multi-agent code review
- WaitReady now detects SandboxDeleting phase and returns immediately
instead of polling indefinitely
- Watch goroutine defers streamCancel() to prevent context leaks
- Fix StopOnTerminal=false test to keep stream open (was wrong-reason
pass due to stream ending, not StopOnTerminal logic)
- Add EventDeleted test covering the Deleting phase branch
- Add provider converter unit tests for CredentialHandle round-trip,
nil handling, and empty maps
Signed-off-by: Roland Huß <rhuss@redhat.com>
---------
Signed-off-by: Roland Huß <rhuss@redhat.com>
Signed-off-by: Roland Huss <rhuss@redhat.com>
* docs(readme): add theme-aware banner
Signed-off-by: Johnny Greco <jogreco@nvidia.com>
* docs(readme): exclude preview screenshot from tree
Signed-off-by: Johnny Greco <jogreco@nvidia.com>
---------
Signed-off-by: Johnny Greco <jogreco@nvidia.com>
* feat(credentials): add provider credential storage drivers
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
* fix(credentials): harden credential update handling
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
* fix(credentials): harden credential driver security, correctness, and performance
Address review findings from the credential storage drivers PR:
- Route additional_credentials through the driver on refresh to prevent
silent data loss for multi-credential providers (e.g. AWS STS)
- Clean up stored credential handles on CAS failure during refresh to
prevent orphaned secrets in external backends
- Enforce namespace validation in the Kubernetes Secrets driver to
prevent cross-namespace credential access when allow_reference_namespace
is not enabled
- Cache Vault Kubernetes auth tokens with 80% TTL to avoid re-authenticating
on every credential operation
- Parallelize resolve_credentials in all three drivers using try_join_all
for faster sandbox startup
- Add existingSecret support for the KEK Secret to fix helm template/GitOps
workflows where lookup returns empty and regenerates the key
- Document RBAC blast radius for the Kubernetes Secrets credential driver
and recommend a dedicated namespace
Signed-off-by: Varsha Prasad <varshaprasad96@gmail.com>
Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com>
* fix(credentials): add optimistic concurrency, fix thundering herd, parallelize operations
Use resourceVersion optimistic concurrency with retry loop for K8s
Secret ownership checks to prevent TOCTOU races. Switch Vault token
cache from RwLock to Mutex with double-check pattern to prevent
thundering herd on cache miss. Parallelize credential store and delete
operations across independent keys using try_join_all.
Signed-off-by: Varsha Prasad <varshaprasad96@gmail.com>
Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com>
* fix(credentials): handle partial failures, add delete retry, consolidate cleanup
Replace try_join_all with join_all in credential store/delete operations
to handle partial failures — successfully-stored handles are cleaned up
when another key fails. Add retry loop with conflict detection to
db-credstore delete_credential, matching the K8s driver pattern.
Consolidate 4 manual cleanup_pre_stored_provider_credentials call sites
into a single error handler using an async block. Remove inconsistent
.trim() from db-credstore validate_handle_owner.
Signed-off-by: Varsha Prasad <varshaprasad96@gmail.com>
Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com>
* fix(credentials): fix retry loop guard and remove unprotected validation
Remove attempt-count guard from 409/Aborted match arms in retry loops
so the post-loop Status::aborted error is reachable after exhausting
retries. Previously, last-attempt conflicts fell through to the
catch-all error arm, producing misleading Status::unavailable errors.
Remove duplicate validation calls that ran after
prepare_provider_credential_update but outside the cleanup-protected
async block, which would leak pre-stored handles on failure.
Signed-off-by: Varsha Prasad <varshaprasad96@gmail.com>
Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com>
* fix(credentials): add workspace/provider UUID to credential backend paths
Include workspace and provider ID in credential backend object paths to ensure
cross-workspace uniqueness and prevent credential collision (GATOR-1806c9be-01).
- Updated credential driver proto to include workspace and provider_id fields
- Modified Vault driver to include workspace/provider_id in managed_secret_path
- Modified Kubernetes Secrets driver to include workspace/provider_id in
credential_owner_id and managed_secret_name
- Updated all credential runtime calls to pass workspace/provider_id
- Updated tests to use the new signatures
This prevents two workspaces sharing the same external credential store from
colliding on provider names, which was a critical security issue (CWE-639).
* fix(credentials): preserve provider-level expiration for handle-backed credentials
Compute effective expiration from both provider and driver values using the
earliest non-zero timestamp and skip expired values before insertion
(GATOR-1806c9be-02).
- Modified resolve_provider_handles to check provider credential_expires_at_ms
- Skip expired credentials during resolution instead of returning them
- Use effective expiration (min of provider and driver) in resolution results
- Fix inference.rs to preserve earliest expiration when merging
This ensures handle-backed credentials respect the same expiration semantics
as inline credentials.
* fix(credentials): stage refresh changes under new handles before validation
Stage credential replacements under new immutable handles instead of reusing
existing handles to prevent overwriting committed values before validation/CAS
(GATOR-1806c9be-03).
- Stage credentials with empty existing_handles map to force new handle creation
- Validate and CAS before the new values are committed to backend storage
- Delete old handles only after successful CAS
- On CAS failure, delete only the newly staged handles
- This prevents CWE-362/CWE-367 race conditions where failed refreshes could
still modify or delete the active credential
The fix ensures that a rejected refresh cannot modify the backend object still
referenced by the committed provider record.
* fix(credentials): add timeouts to credential driver RPCs
Apply configured timeouts to both startup capability negotiation and runtime
RPCs to prevent indefinite hangs (GATOR-1806c9be-05).
- Add DEFAULT_CREDENTIAL_DRIVER_RPC_TIMEOUT_SECS constant (30s)
- Apply timeout to GetCapabilities during startup connection
- Apply timeout to all runtime RPCs (store, delete, resolve)
- Use tokio::time::timeout to bound the entire GetCapabilities operation
during startup, not just the socket connection
- Return contextual deadline errors on timeout
This prevents a faulty or overloaded driver from hanging gateway operations
indefinitely.
* fix(credentials): fix test to use consistent workspace/provider identity
The Kubernetes auth Vault resolve test was constructing a managed path with
test-workspace/test-provider-id but sending default/prov-123 in the request,
causing validation to reject the request (GATOR-18e32351-01).
- Update test to use test-workspace and test-provider-id in the request to
match the logical_path construction
- This ensures the test exercises the intended code path and validates
Kubernetes auth resolution properly
The test now passes and correctly validates identity enforcement.
* fix(credentials): use unique staging ID for refresh to avoid overwrites
Stage refresh replacements under genuinely distinct immutable handles using
a unique staging ID to prevent overwriting committed values (GATOR-1806c9be-03).
- Generate a unique staging ID using UUID for each refresh operation
- Use this staging ID when storing credentials instead of the real provider ID
- Pass the same staging ID during cleanup on failure to delete only staged objects
- This ensures deterministic paths (Vault) and object names (K8s) don't collide
with the committed provider's credentials
The fix prevents failed refreshes from silently replacing active credentials
or breaking providers by deleting still-referenced backend objects.
* fix(credentials): wrap credential driver RPCs in local timeouts
Add local tokio::time::timeout wrappers around credential driver RPCs to
bound non-compliant or stalled UDS peers (GATOR-1806c9be-05).
- Wrap StoreCredential, DeleteCredential, and ResolveCredentials in local timeouts
- Return contextual deadline_exceeded errors when timeouts occur
- Keep existing gRPC timeout metadata for compliant implementations
- GetCapabilities during startup was already wrapped in previous commit
This ensures a faulty local driver cannot hang gateway operations indefinitely,
even if it accepts the connection but never responds to the RPC.
* fix(credentials): preserve ownership for staged refreshes
Signed-off-by: Seth Jennings <sjenning@redhat.com>
* fix(credentials): bound startup capability probe
Signed-off-by: Seth Jennings <sjenning@redhat.com>
* test(provider): authenticate credential handler requests
Signed-off-by: Seth Jennings <sjenning@redhat.com>
* fix(ci): grant actions read to credential driver e2e
Signed-off-by: Seth Jennings <sjenning@redhat.com>
---------
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
Signed-off-by: Varsha Prasad <varshaprasad96@gmail.com>
Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com>
Signed-off-by: Seth Jennings <sjenning@redhat.com>
Co-authored-by: Taylor Mutch <taylormutch@gmail.com>
Co-authored-by: Varsha Prasad Narsing <varshaprasad96@gmail.com>
* perf(net): set TCP_NODELAY on all tunnel and proxy TCP hops
The sandbox tunnel added ~44 ms of latency to every small
request/response because no socket in the path disabled Nagle's
algorithm, so sub-MSS writes waited on delayed ACKs at each hop.
Set TCP_NODELAY on every latency-sensitive TCP socket:
- gateway: accepted connections on the public listener (gRPC relay
frames and WS tunnel writes)
- CLI: edge tunnel local accept + underlying WebSocket TCP stream,
insecure TLS connector (tonic's default connector already does
this), and service-forward accepted sockets
- supervisor: direct-tcpip connect into the sandbox netns, TCP relay
target dials, egress proxy accepted connections, and all upstream
CONNECT/HTTP dials (via a new connect_upstream helper)
Setting TCP_NODELAY on connect is best-effort: a failure only costs
latency, so we log and continue rather than fail the connection.
The sandbox SSH transport rides a unix domain socket and gRPC client
channels use tonic defaults (nodelay on), so no change is needed there.
Fixes#2219
Signed-off-by: Jim Meyer <jim@meyer4hire.com>
* refactor(net): house TCP_NODELAY helper in a shared net module
Address review feedback on the TCP_NODELAY change:
- Move the shared set-nodelay helper out of supervisor_session into a
new crate-private `net` module in openshell-supervisor-process, so
ssh and supervisor_session no longer reach across modules through a
pub(crate) item.
- Make the best-effort comments at each call site terse and consistent.
No behavior change; the benchmark ladder reproduces the same numbers.
Signed-off-by: Jim Meyer <jim@meyer4hire.com>
* refactor(net): consolidate TCP_NODELAY helpers into openshell_core::net
Move the best-effort TCP_NODELAY helpers into the shared openshell_core::net
module so every crate dials and configures sockets the same way:
- Add set_tcp_nodelay_best_effort (accepted/existing streams) and
connect_tcp_nodelay_best_effort (dial + set) with unit tests.
- Migrate all call sites in openshell-cli, openshell-server, and the
supervisor crates to the shared helpers.
- Remove the crate-private net module from openshell-supervisor-process.
- Document socket guidance in AGENTS.md (Network Sockets).
Signed-off-by: Jim Meyer <jim@meyer4hire.com>
* perf(net): set TCP_NODELAY on exec bridge and metadata server
The gateway-side single-use SSH-over-relay loopback bridge and the
sandbox IMDS metadata server were missed latency-sensitive TCP hops.
Set TCP_NODELAY on the accepted client connection and both russh client
dials of the exec bridge — interactive keystrokes and line-buffered PTY
output are the most tinygram-heavy traffic in the system — and on the
metadata server's accepted connections.
Also log unrecognized MaybeTlsStream variants in the edge tunnel so a
future TLS-backend change surfaces a silent TCP_NODELAY miss instead of
skipping it quietly.
Signed-off-by: Jim Meyer <jim@meyer4hire.com>
* perf(net): set TCP_NODELAY on openshell-sdk socket paths
The openshell-sdk crate landed on main with its own copies of the CLI's
hand-rolled sockets, which the CLI and TUI are meant to consume. Give
them the same treatment as the CLI equivalents:
- edge_tunnel: the accepted local tunnel connection and the WebSocket's
underlying TCP socket (plain and rustls variants).
- transport: the dial in InsecureTlsConnector, tonic's custom-connector
path.
Only these hand-rolled sockets need it. Tonic's own connector defaults
tcp_nodelay to true and applies it itself, so plain Endpoint::connect
callers were already covered.
Signed-off-by: Jim Meyer <jim@meyer4hire.com>
---------
Signed-off-by: Jim Meyer <jim@meyer4hire.com>
Update the docs for `sandbox create` and `exec` to dissuade use of `--env` for
secrets, and enhance the docs for `--provider` to explain what it's for.
Signed-off-by: Kris Hicks <khicks@nvidia.com>
Document Bazel targets alongside existing mise commands with a prominent
experimental notice linking to RFC 0012. Includes Bazelisk install
instructions, .bazelignore guidance for Cargo coexistence, and a mapping
of available build and test targets.
Signed-off-by: Roland Huß <rhuss@redhat.com>
Keep the completed instrumented handler future in an explicit pinned box and
drop it after the await. This releases the handler-side request span clone
before the disconnect test checks producer ownership, avoiding compiler- and
platform-dependent retention of an unfinished span.
Signed-off-by: Kris Hicks <khicks@nvidia.com>
* feat(build): add system CA root mode
Allow distro builds to use native trust stores for supervisor upstream TLS while keeping bundled Mozilla roots as the default. Avoid bundled root crates in system-ca-roots builds by using native-root TLS features and z3 0.20.
Signed-off-by: Adam Miller <admiller@redhat.com>
* fix(build): keep CA root feature in telemetry-off verification
The telemetry-off task uses --no-default-features which now disables
bundled-ca-roots in addition to telemetry, triggering the compile_error
guard. Re-enable bundled-ca-roots explicitly so the task verifies only
telemetry compilation.
Signed-off-by: Scott Burdine <sburdine@nvidia.com>
Signed-off-by: politerealism <burdcat17@gmail.com>
* fix(sdk): disable oauth2 default features to prevent webpki-roots leak
The bare `oauth2 = "5"` dependency re-enabled default features
(rustls-tls → reqwest/rustls-tls → webpki-roots), defeating the
system-ca-roots feature gate. Mirror the CLI fix: disable defaults
and enable only the `reqwest` feature.
Signed-off-by: Quinn Burdine <sburdine@redhat.com>
Signed-off-by: politerealism <burdcat17@gmail.com>
* refactor(build): simplify CA root selection to single feature toggle
Replace mutually exclusive bundled-ca-roots / system-ca-roots features
with a single bundled-ca-roots toggle. Disabling it implies system roots
via rustls-native-certs, which is now a regular (non-optional) dependency.
This fixes cargo --all-features and simplifies the distro build interface
from --no-default-features --features system-ca-roots to just
--no-default-features.
Signed-off-by: Quinn Burdine <sburdine@redhat.com>
Signed-off-by: politerealism <burdcat17@gmail.com>
* feat(build): add system-ca-roots convenience alias and fix verify task
Add a system-ca-roots feature alias on openshell-sandbox that includes
all other defaults (telemetry) except bundled-ca-roots, so distro
builds can use --no-default-features --features system-ca-roots without
manually re-adding unrelated defaults. Update the verify CI task to use
the alias and scope checks to the sandbox package. Fix task description
to use "build mode" terminology instead of implying a Cargo feature.
Signed-off-by: Quinn Burdine <sburdine@redhat.com>
Signed-off-by: politerealism <burdcat17@gmail.com>
* ci: fix system CA roots step name to use build mode terminology
Signed-off-by: Quinn Burdine <sburdine@redhat.com>
Signed-off-by: politerealism <burdcat17@gmail.com>
* refactor(sandbox): reorder features to place system-ca-roots alias near default
Signed-off-by: Quinn Burdine <sburdine@redhat.com>
Signed-off-by: politerealism <burdcat17@gmail.com>
* fix(proxy): unwrap Result from build_upstream_client_config in tests
The function signature changed to return Result but the test call sites
were not updated, causing type mismatch compilation errors in CI.
Signed-off-by: Quinn Burdine <sburdine@redhat.com>
Signed-off-by: politerealism <burdcat17@gmail.com>
---------
Signed-off-by: Adam Miller <admiller@redhat.com>
Signed-off-by: Scott Burdine <sburdine@nvidia.com>
Signed-off-by: politerealism <burdcat17@gmail.com>
Signed-off-by: Quinn Burdine <sburdine@redhat.com>
Co-authored-by: Adam Miller <admiller@redhat.com>
Repeated Bazel test targets with --runs_per_test at 10, 20, 50, 100, and
200 runs. The tests failed intermittently when multiple instances ran
concurrently. This indicates timing, shared tracing state, socket readiness,
or parallel-safety issues that need focused follow-up before re-enabling the
tests.
A normal non-repeated bazel test //... run passes with these tests ignored.
Disabled tests:
- sandbox_forward_foreground_fails_when_ssh_exits_before_listener_opens
- sandbox_forward_background_terminates_owned_child_when_listener_never_opens
- podman_socket_probe_accepts_successful_ping_response
- podman_socket_probe_rejects_docker_ping_response
- docker_socket_probe_accepts_successful_ping_response
- docker_socket_probe_rejects_podman_ping_response
- docker_socket_detection_returns_the_responsive_candidate
- podman_socket_detection_returns_the_responsive_candidate
- driver_watch_events_are_roots_and_store_operations_have_parents
- reconcile_sweeps_are_roots_and_operations_have_parents
- gateway_listeners_bind_ipv6_wildcard_and_ipv4_callback_on_same_port
- watch_producer_releases_request_span_when_client_disconnects
- expected_conflicts_leave_the_span_unmarked
- store_spans_record_what_they_touched_as_attributes
- store_operations_export_spans_with_parents
- refresh_worker_ticks_are_roots_and_store_operations_have_parents
Signed-off-by: Simon Scatton <sscatton@nvidia.com>
Continue distributed traces across the gateway-to-driver process boundary and
export VM driver spans to the same OTLP/gRPC collector. The driver reports as
the distinct openshell-driver-vm service.
Updated the gateway architecture and configuration reference with a generic
external-driver forwarding contract.
Instrumented:
- Every RemoteComputeDriver RPC injects the active W3C trace context into
tonic metadata. Managed VM readiness and runtime initialization give startup
capability probes stable parent operations rather than isolated root spans.
- A tonic service layer creates fixed, low-cardinality server spans for every
ComputeDriver RPC. New handlers inherit tracing automatically; failures record
OpenTelemetry error status and the gRPC status code.
- Background provisioning remains attached to CreateSandbox after the RPC
returns without extending the RPC span lifetime.
- Provisioning records image preparation, bootstrap image resolution, overlay
preparation, lifecycle configuration, pre-launch hooks, guest preparation,
and launcher spawn as child spans.
- VM startup reconciliation roots one trace for the persisted-sandbox scan,
with per-sandbox restore and provision operations beneath it. The root remains
open until all spawned restore tasks finish.
- Delete cleanup records its own child operation.
Design notes:
- The gateway forwards its configured OTLP endpoint to managed external
drivers. SDK `OTEL_*` variables continue to own sampling, batching, limits,
headers, and transport tuning.
- The VM driver has its own tracer provider and service resource so trace
backends preserve the service boundary.
- RPC operation names come from an explicit method mapping, keeping cardinality
bounded without parsing the protobuf descriptor set at runtime.
- Propagation uses a remote SpanContext for spawned provisioning. This keeps
one trace while allowing the CreateSandbox server span to finish when the
RPC response is sent.
- Startup restoration is independent of gateway requests. It begins at the VM
driver reconciliation span rather than attaching to an unrelated RPC.
- Existing tracing events remain on the logging path. The OpenTelemetry layer
exports spans only and excludes the SDK exporter callsites to avoid recursive
traces.
- Export configuration failures do not prevent the driver from serving, and
buffered spans are drained during graceful shutdown.
- Trace fields identify drivers, sandboxes, images, lifecycle phases, and gRPC
outcomes without recording credentials, sandbox tokens, or request query
parameters.
Refs #2507
Signed-off-by: Kris Hicks <khicks@nvidia.com>
Run the constructor gateway-discovery error test only on Linux, matching the
production branch that inspects the Podman bridge gateway. macOS uses its
Podman machine callback path and correctly skips that inspection.
Signed-off-by: Kris Hicks <khicks@nvidia.com>
Establish the Phase 1 reference implementation proposed by RFC 0012
while retaining the existing Cargo and Mise workflows during evaluation.
- pin Bazel 9.1.1 and configure Bzlmod, rules_rs, LLVM, protobuf, and
Rust 1.95 toolchains
- import third-party crates from Cargo metadata and propagate the
workspace version into Bazel targets
- add library, binary, proc-macro, unit-test, and integration-test
targets across the supported Rust workspace crates and drivers
- generate protobuf Rust sources and descriptor sets under Bazel while
preserving Cargo-compatible generated-code imports
- annotate aws-lc-sys and zstd-sys native dependencies, build Z3 4.15.2
from source, and generate z3-sys bindings
- make CLI and procfs test fixtures available as explicit Bazel inputs
without relying on fixed host binary paths
- define optimized release targets for Linux x86_64 and aarch64 CLI,
sandbox, and gateway binaries, plus macOS aarch64 artifacts
- add Bazel, buildifier, and lcov to the Nix development environment
RFC: 0012 (rfc12 branch)
Refs: #2491
Signed-off-by: Simon Scatton <sscatton@nvidia.com>
The log prefix value passed to nft contains colons (e.g.
openshell:bypass:sandbox-cc817378:) but was not wrapped in double
quotes. Since nft concatenates argv entries and parses the result,
the bare colons cause a syntax error that silently prevents all
bypass-attempt LOG rules from installing.
Wrap log prefix values in nft-quoted strings via a new nft_quote()
helper that strips embedded double-quotes (nft quoted strings don't
support escape sequences). All four log-rule generation sites (TCP
and UDP, for both per-sandbox and sidecar rulesets) are updated.
Fixes#2470
Signed-off-by: Grace Smith <grasmith@redhat.com>
* test(server): stabilize watch span cancellation test
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
* test(server): poll span exporter at fixed interval
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
---------
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
* fix(podman): combine sandbox stop and removal
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
* fix(podman): use libpod volume removal parameter
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
* fix(podman): preserve removal timeout margin
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
* test(podman): use virtual time for removal delay
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
---------
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
Matthew Grossman