mirror of
https://github.com/NVIDIA/OpenShell.git
synced 2026-10-31 07:44:57 +08:00
def7da876cb5665d1d9b1acde5fc31eda79bf93f
538
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Adel Zaalouk
|
def7da876c |
fix(ocsf): emit schema-conformant OCSF 1.8 events (#4288)
Native events failed the official OCSF 1.8.0 validator: HTTP Activity emitted is_src_dst_assignment_known, sandbox Process Activity events had no actor, connection_info lacked direction_id, and SSH Activity events named no endpoint. Schema tests missed these because they checked only top-level, non-profile attributes. Add a connection direction enum, require an actor on Process Activity and an endpoint on SSH Activity at compile time, and name the supervisor's SSH server with svc_name and its socket path. Validate against OCSF 1.8.0 only. validate_required_fields and validate_known_fields share one recursive walker that rejects undefined attributes at every level and, for events, attributes from undeclared profiles; objects are validated as objects. Records labelled with another version fail. Vendor tactic and technique so ATT&CK-mapped findings validate, drop the unused v1.7.0 schemas, and make the embedded-schema drift test tolerate CRLF checkouts. Keep it fixed with a per-builder conformance test and a Podman e2e test that runs an L7 policy against a local server and validates the supervisor's real Network, HTTP and SSH Activity JSONL. Top-level container stays unchanged; #4915 tracks it. The schema downgrade is unchanged. Closes #4283 Signed-off-by: Adel Zaalouk <azaalouk@redhat.com> |
||
|
Roland Huss
|
fdce6ef260 |
feat(supervisor): relay agent OTLP traces to the collector (#4276)
* feat(supervisor): relay agent OTLP traces to the collector Serve the reserved endpoint 192.0.0.8:4318 inside the supervisor, attribute each batch with openshell.sandbox.id and openshell.telemetry.source=agent at the protobuf wire level, buffer it with fixed bounds, and forward it over OTLP/gRPC to the collector the supervisor already uses, flushing before the exit report and keeping the relay up for exec sessions. Signed-off-by: Roland Huß <rhuss@redhat.com> * refactor(supervisor): let the OTLP relay exporter take a prepared channel Expose Exporter::with_channel as the seam for a collector transport built once in openshell-otel and shared by the gateway, the supervisor's own spans, and the relay. Exporter::new keeps building the plaintext channel for now. Signed-off-by: Roland Huß <rhuss@redhat.com> * feat(sandbox): leave caller-set OTLP endpoints alone Inject the relay variables only when the sandbox creation request names no OTLP endpoint, generic or traces-specific. A caller who names one has chosen where the agent exports; overwriting it would silently redirect an agent that deliberately exports elsewhere. A protocol or compression setting alone does not suppress injection. Applies to both spawn paths and to exec children, with the docs updated to match. Signed-off-by: Roland Huß <rhuss@redhat.com> * fix(supervisor): address OTLP relay review feedback Inject only the generic OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_EXPORTER_OTLP_PROTOCOL as lowest-precedence defaults, and only while the relay is active. The supervisor now starts the relay before attach and signals it to the sandbox runtime through AgentSpec::otlp_relay_active. An endpoint in the creation request, the image ENV, or an exec environment suppresses the defaults, and a provider-supplied traces endpoint is removed. Harden the relay itself: - skip the exit flush and shutdown wait when the last export failed - count batches still pending on Drop as discarded and log them - validate the collector endpoint with the shared openshell-otel helper - accept Content-Encoding: identity - fix the 413 and shutdown 503 messages and the memory bound docs - narrow the attribution claim to resource attributes Document best-effort delivery and the 404 for metrics and logs. Signed-off-by: Roland Huß <rhuss@redhat.com> * feat(supervisor): gate the agent trace relay behind [otlp.agent] The relay started whenever a sandbox supervisor had a trace endpoint, so enabling gateway trace export also changed every workload's environment. Make it opt-in: `[openshell.gateway.otlp.agent]` with `enabled = true` turns it on, and its optional `endpoint` falls back to the shared `[openshell.gateway.otlp]` endpoint, like the OCSF table. The gateway validates the endpoint and drivers pass it to supervisors as OPENSHELL_OTLP_AGENT_ENDPOINT, separate from OPENSHELL_OTLP_ENDPOINT. Docker, Podman (including the Podman Machine host alias), Kubernetes and VM drivers carry it, in-process and as standalone binaries. The shared Kubernetes supervisor still does not relay. The relay now acknowledges OTLP metrics and logs exports with an empty success and discards them, so agent SDKs that export every signal to the generic endpoint stop retrying. Discards are counted and logged once. Local dev gateway tasks, the Skaffold values and the e2e harness enable the table; OPENSHELL_E2E_OTLP_AGENT=0 runs the inert case with trace export still on. Signed-off-by: Roland Huß <rhuss@redhat.com> * docs(supervisor): describe the agent trace relay architecture Add an "Agent traces" section to the architecture page and document the relay in the crate READMEs: the reserved-destination hook in the network supervisor, the OTEL defaults the sandbox runtime injects, and how the Docker, Podman, Kubernetes, and VM drivers pass the agent endpoint. Signed-off-by: Roland Huß <rhuss@redhat.com> * fix(sandbox): disable OTLP compression in the relay env and cover exec children The relay refuses compressed bodies, so an image or creation request that sets gzip compression without naming an endpoint received the relay default and lost every trace export to a 415. The relay default now also sets OTEL_EXPORTER_OTLP_COMPRESSION and OTEL_EXPORTER_OTLP_TRACES_COMPRESSION to none, as #4275 asks. Boundary exec children start from a cleared environment and never see the image ENV, yet the relay default was suppressed when the sandbox's own environment named an endpoint, which left exec children with no endpoint at all. apply_otlp_relay_env now takes a ChildEnvironment so only children that inherit the sandbox environment consult it. Add a probe test for both TLS environment modes together with the relay default, now that the mode-aware TLS layering lives in compose_canonical_process_environment, and describe the whole composition at its call sites. Signed-off-by: Roland Huß <rhuss@redhat.com> * fix(supervisor): flush the agent trace relay before suspending The suspend lifecycle from #4765 returned without shutting the relay down, so Drop discarded batches the relay had already acknowledged to the agent, which does not resend them after resume. Shut the relay down, bounded by FLUSH_TIMEOUT, once the workload is frozen and before networking is dropped. Also tidy what the rebases left behind: drop a stray blank line inside the upstream startup block, rename SandboxInstanceConfig::agent_otlp_endpoint to otlp_agent_endpoint to match every other identifier, serialize the OCSF table through the generic serialize_enabled, correct the UnsupportedScheme doc comment (the supervisor's own exporter does speak TLS), and build the reserved-destination proxy test identities with host-native paths. Signed-off-by: Roland Huß <rhuss@redhat.com> * test(e2e): scope the Podman OTLP default and restore upstream lockfiles The Podman wrapper enabled OTLP export and the agent relay for every e2e run, including single-target CI jobs such as additional_ca. Enable it only for the full suite or a selection that includes otlp_relay, following the SPIFFE fixture pattern; the Docker wrapper stays opt-in. The e2e and features lockfiles had downgraded libsqlite3-sys, crossbeam-queue and hybrid-array against upstream with no manifest change requiring it. Restore upstream's lockfiles and add only the packages the new e2e dependencies pull in. Signed-off-by: Roland Huß <rhuss@redhat.com> --------- Signed-off-by: Roland Huß <rhuss@redhat.com> |
||
|
Polite_realism
|
a137abbf99 |
docs(gateway): list every cause of the Podman gateway --tls-cert error (#5001)
The Podman container gateway page said the `--tls-cert is required when TLS is enabled` startup error means OPENSHELL_LOCAL_TLS_DIR is unset or points at an empty directory. The same error also appears when the directory exists but the container cannot read it, which happens without --userns=keep-id or, on SELinux hosts, without the container_runtime_t label. Name those causes and the settings to check, and say that the SELinux type is also needed to read the data directory. Follows up #4977. Signed-off-by: politerealism <burdcat17@gmail.com> |
||
|
DiveshandDrew Newberry
|
44ef984b58 |
fix(supervisor): guard shared supervisor pool configuration (#4988)
* fix(supervisor): guard shared supervisor pool configuration Raise the gateway's minimum supervisor pool recovery timeout from 10 to 30 seconds. A failed pool member takes up to about 24 seconds to detect, so a shorter deadline ended workloads another member would have adopted. The Sandbox Protocol range stays 10 to 300 seconds for compatibility. Refuse an [openshell.gateway.supervisor_pool] table during config preflight, and at startup before the compute driver is built, unless the selected driver hosts sandboxes on the shared supervisor pool. A new ComputeDriverFactory::uses_shared_supervision hook reports this; the Kubernetes factory returns true with supervisor_topology = "gateway". Widen the readiness deadline of a shared-startup supervisor test that timed out under full-workspace load. Signed-off-by: divesh <dgude@nvidia.com> * fix(gateway): require driver selection for pool preflight Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs(gateway): remove pool preflight note Signed-off-by: Drew Newberry <anewberry@nvidia.com> --------- Signed-off-by: divesh <dgude@nvidia.com> Signed-off-by: Drew Newberry <anewberry@nvidia.com> Co-authored-by: Drew Newberry <anewberry@nvidia.com> |
||
|
Polite_realism
|
fa2d381eaa |
docs(gateway): make the Podman container gateway example work (#4977)
The "Using Podman" example no longer produces a working gateway on rootless Podman. Replace the single command with a tested procedure that runs the gateway with mTLS, and document the five settings it needs: a writable state directory, a same-path data directory, generated signing keys and TLS certificates, the SELinux type for the Podman socket, and matching host and container ports. Refs #4972 Signed-off-by: politerealism <burdcat17@gmail.com> |
||
|
Jesse Jaggars
|
2c31ac7e60 |
feat(network): support additional destination CAs (#3292)
* feat(network): support additional destination CAs Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(network): harden additional CA propagation Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(network): narrow additional CA review scope Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(network): harden additional CA rotation Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * feat(network): complete additional CA delivery Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(docker): remove stale runtime fixture field Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(network): adapt additional CA series to upstream main Resolve rebase fallout against current upstream main after the series was rebased 214 commits forward: - follow upstream #3110: supervisor→gateway TLS is CA-only, so configure_gateway_tls and its isolation tests drop the client identity - follow upstream tls_secret_names reduction to [String; 1] in the Podman driver while keeping the network trust bundle delivery - port [openshell.supervisor.network] rendering into the openshell.gatewayConfigToml serializer in _toml.tpl - adapt tests to upstream struct/function changes (DriverStartupContext, SandboxRunConfig, IsolationSpecInput, supervisor_pod, VM bootstrap image) - drop conformance-artifact-prefix plumbing removed upstream Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * refactor(network): adopt upstream additional CA bundle surface Unify the additional destination CA series on the mechanism merged to main in #3996: one operator pre-assembled PEM bundle, the --additional-ca-bundle supervisor flag, and the /etc/openshell/tls/additional/ca-bundle.pem staging path. - Replace the gateway-level additional_ca_cert_paths list with a single additional_ca_bundle file validated and staged at gateway startup. - Restore the upstream supervisor argument path and fold-into-roots behavior; drop the --network-additional-ca-bundle and --network-additional-ca-digest contracts. - Keep gateway normalization bounds, content-addressed staging, digest-based driver reconciliation, and stop/start rotation; the freshness check now runs at the gateway/driver boundary instead of the supervisor boundary. - Reject additional_ca_bundle in operator driver JSON config; standalone drivers self-source the bundle from --sandbox-additional-ca-bundle with the same validation. - Extend the Kubernetes ConfigMap name digest segment to keep the full 63-byte budget after the prefix rename. - Box the persisted-sandbox reconciliation futures so callers stay under the clippy::large_futures threshold. Updates Docker, Podman, Kubernetes, and VM delivery, Helm chart and tests, e2e wiring, gateway configuration docs, and the cluster debugging skill to the single mechanism. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(ci): make gateway startup tests wrap-tolerant and refresh example lockfile The network trust startup tests asserted on raw substrings of gateway diagnostics, but the CLI renderer wraps output at a fixed width with continuation markers and hard-breaks long paths. Compare assertions on whitespace- and continuation-squashed text so they hold regardless of wrap position. Format the e2e workspace (separate cargo workspace, not covered by the root cargo fmt run) and refresh examples/isolation-backend-v8/Cargo.lock for the libc dependency added to the network trust crate. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(podman): reuse TLS CA secret during stopped-pair trust replacement The additional CA e2e never ran in CI before the first maintainer test dispatch, and its local container inspection was broken in two ways: it located the supervisor container through e2e network membership (the supervisor is host-networked or runtime-bridged, so the filter matched the CI job container instead), and it required the gateway mTLS CA as a bind mount (Podman delivers it as a container secret when user namespaces remap UIDs). Select the supervisor by managed labels and accept the secret delivery form. The lifecycle leg then exposed a driver bug: stopped-pair trust replacement re-created the per-sandbox TLS CA secret, colliding with the secret still owned by the renamed stopped pair. Mirror the token and resolver secret handling: require and reuse the existing secret during replacement instead of creating it. Validated end-to-end with the Podman additional CA e2e lane. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(server): restore upstream recovery semantics for start-request failures Recovery no longer marks sandboxes Error/InvalidPersistedSandbox when driver_start_request fails during the startup or lifecycle sweeps. Gateway configuration problems (for example a missing launch signer) must not permanently fail unrelated sandbox records; the sweep logs a warning and skips them, matching the behavior before the additional CA work. Sandbox-record failures surface through the normal start path. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * refactor(podman): refresh stale supervisors without replacing workloads Replace the stopped-pair trust reconciliation with supervisor-only recreation. The destination-trust generation label moves from the workload to the supervisor container, and start_sandbox recreates only the supervisor when its generation is missing or stale. The workload container, its volumes, its secrets, and its workspace data are never replaced: the additional CA bundle is mounted only into the supervisor. Running sandboxes are never gated on trust labels, and recovery without fresh launch credentials keeps the existing supervisor. The intentional -removal fence machinery in the watcher existed only for workload replacement and is removed with it. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * refactor(docker): drop workload replacement for destination-trust changes Delete the stopped-workload trust reconciliation path and all of its machinery: the trust-generation label on workload containers, the generation match check in the start fence, the stopped-sandbox replacement, the bounded workspace archive copy with its 512 MiB cap and regular-file-only validation, and the boundary-state rollback helpers. No replacement machinery is needed: the trusted supervisor container is already re-created from the gateway's current configuration on every start via ensure_control_process_for_container, so stop/start picks up the current additional CA bundle while the workload container, its volumes, and its workspace data are never replaced. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * docs(configuration): drop reconciliation wording from additional CA lifecycle The stop/start lifecycle description already matches the supervisor-only refresh model; remove the two remaining references to restart reconciliation of workload containers. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(network): port additional CA test fixtures to the crypto interface Route the gateway TLS isolation test PKI, the destination CA source fixtures, and the gateway startup trust tests through openshell_crypto::pki now that rcgen access is centralized there: generate_keypair/self_signed/signed_by replace the direct rcgen calls and CertifiedKey field access follows the wrapper's shape. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(e2e): repair additional CA lane for Docker and non-container builds The first maintainer test dispatch revealed two latent issues in the additional CA e2e that no local run had exercised: the Podman gateway CA secret helper referenced ContainerEngine without the container feature gate, breaking Kubernetes and VM builds of the test, and the Docker branch asserted a client ca.crt bind mount that the Docker driver never creates. Gate the helper like its callers, and assert Docker's actual delivery path instead: the gateway mTLS CA staged as tls/ca.pem inside the read-only supervisor state volume, selected through the OPENSHELL_TLS_CA environment pointer. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(helm): restore alphabetical README ordering for supervisor keys The helm:docs check regenerates the chart README with helm-docs, which orders rows alphabetically; a conflict resolution had placed supervisor.network.additionalCaConfigMapName after the supervisor.podSecurityContext rows. Regenerate with mise run helm:docs. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * chore: refresh testkit lockfile for openshell-core libc dependency Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(driver-podman): stage operator CA bundle as driver-owned artifact Standalone Podman drivers accepting --sandbox-additional-ca-bundle previously kept the operator's source path as the trust artifact path. The immutable-artifact verification performed on every create/start requires exactly mode 0444, which an operator-owned source path cannot guarantee, so a readable-but-writable source failed reconciliation. Read the operator bundle with the same bounded, no-symlink validation, then stage the normalized material into a driver-owned artifact inside a private tempdir held for the driver process lifetime. The artifact is created 0600, synced, and only then restricted to 0444 so no observer ever sees a writable artifact. The operator source and its permissions are never referenced after startup. Addresses GATOR-291f43b2-02. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(driver-podman): pin retained workload image during supervisor refresh Supervisor-only trust reconciliation re-entered create_sandbox_with_existing_token, which re-pulled and re-resolved the workload image tag even though the workload container is retained. If the tag moved to a different image between stop and start, identity resolution and child-environment construction would reflect the new image while the retained workload still runs the original one, leaving mode-0600 bootstrap files the retained user cannot read. During a refresh with a retained workload, inspect the stopped container and pin to the immutable image ID it was created from (Config.Image); never pull or resolve the request tag. The requested reference is still used for child metadata so the retained workload's environment is byte-identical to its original launch. Addresses GATOR-291f43b2-01. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * test(e2e): verify host-only VM trust delivery and drop curl dependency The additional-CA lane's VM assertions read the destination CA out of the guest overlay and a guest-side supervisor-argv marker — artifacts from the guest-supervisor design that no longer exist now that the VM driver stages the CA host-only and runs the supervisor on the host. Inspect the driver-staged host artifact (digest-compared against the source), assert the guest overlay contains no CA material at the old mount paths, and keep the serial-console no-disclosure check. The workload validation scripts also ran curl, which exists only in the container-engine fixture image; the Kubernetes and VM lanes use the uv-based workload image and died instantly with "canonical main process already finished". Rewrite the main-process, exec, and lifecycle validation scripts in python3 (present in every lane's workload image). The network-policy binary identity is the resolved /proc/<pid>/exe target, so the versioned python3.12 binaries are allowlisted rather than the unversioned symlinks. The hostname mismatch check keeps its semantics: same reachable fixture, mismatched TLS SNI, and the failure must specifically be hostname verification. Addresses GATOR-291f43b2-04. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(kubernetes): allow admission-added labels on destination-trust ConfigMaps The destination-trust ConfigMap validation compared the whole label map for equality. A mutating admission webhook that attaches an organizational label to the created ConfigMap made the create response fail validation, and the retained ConfigMap then failed every subsequent create/start through the existing-object path, blocking sandbox launches despite the ownership labels, immutable flag, digest, and certificate bytes remaining correct. Validate the gateway-owned label pairs individually and allow unrelated labels; the exact trust-material checks are unchanged. Covered by a regression test with an admission-added label and a missing gateway-owned label. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(podman): close supervisor-refresh rollback gaps The supervisor-only trust refresh renamed the previous supervisor to a random rollback name and then performed the token-secret lookup. A transient lookup failure returned with the supervisor still renamed, where a normal delete could never find it, and the rollback rename and final rollback removal ignored their own failures, leaving orphans with attached control and channel resources. Complete every fallible preparation before the rename. On a failed replacement, restore the canonical name; if the restore itself fails, return an error naming the rollback container so the orphan stays identifiable — a later retry still succeeds because the canonical name is absent and the replacement path recreates a fresh supervisor. Treat not-found during the final rollback removal as success and name the orphan in both the log and the error on other removal failures; a retry observes the fresh canonical supervisor and resumes the normal start path. Covered by failure-injection tests for each transaction boundary: preparation failure before any rename, replacement failure restoring the canonical name, and restore failure naming the orphaned rollback container. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * docs: address destination-trust review corrections The gateway-configuration RFC documented an additional_ca_cert_paths list that the strict configuration parser never accepted; document the additional_ca_bundle single-path setting the parser actually enforces. The debug-openshell-cluster skill named the wrong generation annotation (openshell.ai/network-supervisor-trust-generation) and described digest argv and network-init canonicalization checks the implementation does not perform. Correct it to openshell.ai/additional-ca-generation and describe the actual delivery: the supervisor Pod mounts the immutable ConfigMap read-only and receives --additional-ca-bundle for the mount path, with the driver validating the digest annotation and exact bytes per creation. Classify the destination-trust generation hashes as deterministic identity derivation rather than authenticity proofs, following the openshell-crypto exclusions table: trust decisions come from the gateway's normalization, immutable staging, exact-byte validation, and TLS verification. Record the classification in the crypto README and at each hashing site. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(network): reconcile additional CA tests with suspend/resume and auth changes Upstream's suspend/resume lifecycle took StartSandboxRequest fields 7 and 8, so the durable-snapshot field moves from 7 to 9; update the wire-format test to assert the field-9 key (0x4a) and initialize the new resume controls. Repair the cli test module after the auth opt-in test merged alongside the environment-guard helpers, and initialize the gateway driver-startup context's network-trust fields in the shared supervision test. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> * fix(server): box start-recovery futures for clippy large-futures The durable-snapshot field grows StartSandboxRequest enough that the start-recovery match and the recovery-expiry test exceed the large-futures limit under -D warnings. Box::pin both futures, matching the pattern the recovery path used before the suspend/resume rework. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> --------- Signed-off-by: Jesse Jaggars <jjaggars@redhat.com> |
||
|
Jesse Jaggars
|
26134dc735 |
feat(gateway): add hashed service-routing hostnames for multi-gateway clusters (#4842)
* feat(gateway): add hashed service-routing hostnames for multi-gateway clusters
Closes #4840
Add an optional service_routing_gateway_name gateway setting (TOML, CLI,
env, Helm) that disambiguates exposed service URLs when multiple gateways
share a cluster, ingress, and wildcard certificate. When configured,
exposed-service hostnames become a deterministic 16-character lowercase
base32 SHA-256 label of (gateway name, workspace, sandbox, service) under
the same base domain, keeping every hostname exactly one label deep for
wildcard-certificate compatibility. Unset gateways preserve the existing
readable hostname format byte-for-byte, and configured gateways still
accept legacy hosts during migration.
Hashed hosts resolve through an in-memory route index hydrated from
persisted ServiceEndpoint records at startup, refreshed on miss for
multi-replica deployments, maintained across expose, delete, and sandbox
cleanup, and enforced for collisions before endpoint persistence.
Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>
* fix(lint): satisfy clippy doc lints on main-landed sandbox_runtime code
The shared-supervisor landing (
|
||
|
Drew Newberry
|
e5adf6f484 |
feat(supervisor): recover shared instances with hash-based HA (#4864)
* feat(supervisor): recover shared instances with hash-based HA Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(supervisor): keep HA coordination out of compute drivers Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(supervisor): fence pool admission and reconciliation races Signed-off-by: Drew Newberry <anewberry@nvidia.com> * feat(supervisor): keep pool mode stable across replica scaling Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(supervisor): default shared deployments to pool mode Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(supervisor): fence refused launches and remove legacy mode Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(supervisor): confirm live owners and fence stale observations Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(supervisor): preserve takeover cause log formatting Signed-off-by: Drew Newberry <anewberry@nvidia.com> * feat(helm): add memory-based supervisor autoscaling Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(supervisor): retain acknowledgements and fence streamed input Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs(supervisor): consolidate experimental multi-tenant guidance Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs(supervisor): simplify multi-tenant supervisor guide Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(supervisor): simplify pool member identity Signed-off-by: Drew Newberry <anewberry@nvidia.com> * feat(helm): make supervisor affinity configurable Signed-off-by: Drew Newberry <anewberry@nvidia.com> * feat(helm): make supervisor disruption budget optional Signed-off-by: Drew Newberry <anewberry@nvidia.com> * test(supervisor): align HA fixtures with suspend support Signed-off-by: Drew Newberry <anewberry@nvidia.com> --------- Signed-off-by: Drew Newberry <anewberry@nvidia.com> |
||
|
Taylor Mutch
|
7b0f8ae07c |
fix(gateway): require authentication or explicit development opt-in (#4821)
* fix(gateway): require authentication or explicit development opt-in Signed-off-by: Taylor Mutch <taylormutch@gmail.com> * fix(gateway): reject implicit unauthenticated admin requests Signed-off-by: Taylor Mutch <taylormutch@gmail.com> --------- Signed-off-by: Taylor Mutch <taylormutch@gmail.com> |
||
|
Shailendra SinghandDrew Newberry
|
17eda52d45 |
docs(windows): route main documentation to the Windows release branch (#4758)
* docs(windows): route main documentation to the Windows release branch Signed-off-by: Shailendra Singh <shailendras@nvidia.com> * docs(windows): include native MXC tests in full validation Signed-off-by: Shailendra Singh <shailendras@nvidia.com> * docs(windows): address documentation review feedback Signed-off-by: Shailendra Singh <shailendras@nvidia.com> * docs(windows): include E2E in full validation Signed-off-by: Shailendra Singh <shailendras@nvidia.com> * docs(windows): stop validation after any failed task Signed-off-by: Shailendra Singh <shailendras@nvidia.com> * docs(windows): simplify quickstart and installation guidance Signed-off-by: Drew Newberry <anewberry@nvidia.com> --------- Signed-off-by: Shailendra Singh <shailendras@nvidia.com> Signed-off-by: Drew Newberry <anewberry@nvidia.com> Co-authored-by: Drew Newberry <anewberry@nvidia.com> |
||
|
Free Wortley
|
f5d9223e08 |
feat(supervisor): let operators require specific middleware bindings (#4678)
* feat(middleware): authorize supervisor bindings at registration Let operators require and limit the operation and phase pairs that an external middleware may expose. Validate the same binding policy at gateway startup and when a supervisor discovers the service, and keep only authorized bindings in the registry used for dispatch. Preserve dynamic discovery and the existing wire configuration when no binding policy is set. Document that older supervisors must be upgraded before relying on binding restrictions. Signed-off-by: Free Wortley <freeqaz@openai.com> * docs(skills): check which supervisor middleware hooks the operator authorizes Signed-off-by: Free Wortley <freeqaz@openai.com> * fix(go): regenerate the sandbox protobuf binding Restore the comment separator emitted by the pinned generator. This keeps the checked-in binding byte-for-byte reproducible without changing the protobuf schema or Go behavior. Signed-off-by: Free Wortley <freeqaz@openai.com> --------- Signed-off-by: Free Wortley <freeqaz@openai.com> |
||
|
Brooke Storm
|
c4688aa871 |
feat(helm): add certificate hook pod labels and annotations (#4895)
Signed-off-by: Brooke Storm <brookes@nvidia.com> |
||
|
Eric Curtin
|
16250df210 |
feat(helm): add server.extraEnv for the gateway container (#4876)
Signed-off-by: Eric Curtin <eric.curtin@docker.com> |
||
|
Oliver Calder
|
72e4eaa89c |
feat(snap): publish prerelease builds to latest/beta channel (#4870)
* feat(snap): publish prerelease builds to latest/beta channel Signed-off-by: Oliver Calder <oliver.calder@canonical.com> * fixup! feat(snap): publish prerelease builds to latest/beta channel Signed-off-by: Oliver Calder <oliver.calder@canonical.com> * fix(snap): use concurrency group when checking version to publish snap Signed-off-by: Oliver Calder <oliver.calder@canonical.com> --------- Signed-off-by: Oliver Calder <oliver.calder@canonical.com> |
||
|
Emilien Macchi
|
07b7a74b62 |
fix(server): bound the provisioning-deadline scan's lock wait (#4549)
The provisioning-deadline scan takes each provisioning sandbox's local S(global) S(workspace) X(sandbox) keys in ascending key order and waited for them with no deadline. When a sandbox's key sorts before its workspace's key, the scan held X(sandbox) while it waited behind a provider or profile writer, so start, stop and delete for that sandbox waited for the writer too. In the other order, the scan held S(workspace) while it waited for a busy sandbox, and provider writers on that workspace queued behind it. Either way the rest of the pass waited as well. Each pass now has a 500 ms budget for waiting on these keys. A sandbox whose keys stay busy is skipped and picked up on the next pass a second later. Once the budget is spent, the rest of the pass only takes keys that are free when it reaches them, so a busy workspace or global key costs one wait per pass rather than one per sandbox. The local acquisition is cancel-safe, so giving up releases the keys it already took. Skips log at debug and count in the new openshell_server_provisioning_deadline_skips_total counter, labeled by whether the sandbox was already past its deadline. A steady rate of delayed expiries can then be alerted on without a warning every pass. The gateway metrics page and the cluster debugging skill describe it. The test that pinned the old wait is replaced by tests for both key orders: a lifecycle lock or a provider writer proceeds and the scan finishes while the other side holds its key, the skipped sandbox is untouched and expires once the writer releases, and a skip doesn't end the pass for other sandboxes. Another test checks that sandboxes behind one busy workspace share a single wait. Part of #4307 Closes #4547 Signed-off-by: Emilien Macchi <emacchi@redhat.com> |
||
|
Taylor Mutch
|
132c582349 |
docs(extensibility): document external compute driver setup (#4826)
Signed-off-by: Taylor Mutch <taylormutch@gmail.com> |
||
|
Oliver Calder
|
cb42f5d9e8 |
fix(snap): use openshell-prover instead of openshell.prover (#4835)
The alias `openshell-prover` has been granted for OpenShell: https://forum.snapcraft.io/t/alias-request-for-openshell/53658 Thus, we can remove guidance differentiating `openshell.prover` for the snap from `openshell-prover` for other packaging formats. It is still worth noting that installing a locally-built snap directly, rather than from the store, means there is no alias. Signed-off-by: Oliver Calder <oliver.calder@canonical.com> |
||
|
Adel Zaalouk
|
229382e51d |
docs(sandbox): align workload image requirements with current runtime (#3967)
Workload images no longer need iproute2 or nftables since the supervisor moved out of the workload (#2942). Docker and Podman run images without USER as UID/GID 1000 (#3386), the driver creates /sandbox, and the CLI no longer builds Dockerfiles (#3214). Update the bring-your-own-container example and the runtimes page to match. Fixes #3965 Signed-off-by: Adel Zaalouk <azaalouk@redhat.com> |
||
|
krishicks
|
321b93a898 |
feat(auth): emit OCSF events for SSH session and sandbox token checks (#4762)
Some credentials are checked by handlers rather than by the gateway's authenticators, so their rejections only reached the caller and plain logs. They now emit the same OCSF events as the router's checks: - SSH session tokens. A ForwardTcp SSH connection whose token is missing, unknown, revoked, minted for another sandbox, or expired emits an Authentication failure with protocol ssh_session_token and a reason category. An accepted token logs on once, like other credentials. The router has already authenticated the caller, so these events name that caller. - IssueSandboxToken. A compute-driver identity that no longer matches the sandbox's recorded runtime emits an Authentication failure (stale_identity); a sandbox with no record, or no usable recorded identity, emits unknown_identity. A sandbox calling it with a gateway JWT is refused with method_not_permitted and raises the sandbox boundary finding. - RefreshSandboxToken. A caller that is not a gateway-JWT sandbox is refused with method_not_permitted. A request for an extension service the sandbox policy does not select is refused with the new extension_service_not_selected reason, without a finding, because a policy update can race the supervisor's request. - Sandbox logons on IssueSandboxToken and RefreshSandboxToken. The router logged a compute-driver credential on IssueSandboxToken, or a gateway JWT on RefreshSandboxToken, before the handler finished checking the recorded runtime or the token lineage, so a credential that failed that check recorded a success and then a failure. Those handlers now record the logon once their check passes. Other credentials on these methods are fully verified by the router and still log on there, even when authorization then refuses them. - GetCurrentUser. A non-user principal is refused with method_not_permitted. The router already blocks such callers, so this is defense in depth. Callers receive the same status codes and messages as before, and the events never include the tokens. Refs #2912 Signed-off-by: Kris Hicks <khicks@nvidia.com> |
||
|
Max Dubrinsky
|
a2b8022e07 |
fix(python): ship credential driver protocol bindings (#4760)
Closes #4759 Signed-off-by: Max Dubrinsky <mdubrinsky@nvidia.com> |
||
|
krishicks
|
915a5d1553 |
feat(auth): emit OCSF logon events for accepted gateway credentials (#4761)
When a gateway replica accepts a credential for the first time, it now emits an Informational OCSF Authentication event with status Success. The caller's request proceeds exactly as before. The event records: - Who: the principal (user, mTLS operator, sandbox, or gateway peer), with the same user types as authorization denials. - How: the authentication protocol (OpenID, mtls, sandbox_jwt, compute_driver, or k8s_service_account). - Where: the gRPC method of the request that logged on. - Correlation: the caller's address and request ID, the same fields the failure and denial events carry. Every gRPC request authenticates, and an idle sandbox's supervisor polls the gateway several times a minute, so an event per request would bury user logons. Each credential therefore logs on once: later requests with the same bearer token or client certificate emit nothing until the credential has been in use for an hour, when it logs on again. The replica's connections share this record, which holds up to 16,384 credentials and forgets the oldest early when a new one arrives while full. It keys on a SHA-256 digest of the credential that authenticated the caller (the bearer token or the client certificate's fingerprint), so headers the authenticator ignored cannot force new logons, and a renewed certificate for the same subject logs on again. Digests stay in memory. Requests admitted by allow_unauthenticated_users, or by a gateway with no authentication configured, emit no success event, since no credential was verified. Refs #2912 Signed-off-by: Kris Hicks <khicks@nvidia.com> |
||
|
Eric Curtin
|
ea265a2f9a |
docs(cli): say --upload cannot take a command (#4502)
Signed-off-by: Eric Curtin <eric.curtin@docker.com> |
||
|
krishicks
|
a24340beba |
feat(auth): emit OCSF events for gateway authorization denials (#4728)
When the gateway refuses a caller that has already authenticated, it now emits an OCSF API Activity event. The caller gets exactly the same error as before. The event records: - Who: the principal (user, mTLS operator, sandbox, or gateway peer). - What: the gRPC method, plus the sandbox or workspace that was refused when there is one. - Why: a reason such as cross_sandbox, missing_role or not_workspace_member. - Correlation: the caller's address and request ID, the same fields the authentication events carry. That covers the denials made centrally by the router and the shared guards, plus the checks inside handlers that have no shared guard: a sandbox naming another sandbox (in either sandbox-name resolver), a non-user caller refused by mutation admission, a sandbox sending a config update it isn't allowed to, and a non-admin granting the workspace admin role. Denials raised in the mutation admission worker, which runs in a spawned task, keep the request's context. When a sandbox credential is refused for calling a method it may not call, touching another sandbox, or acting across workspaces, the gateway also emits a High-severity Detection Finding. It carries the same correlation_uid as the denial event. A real supervisor never makes these requests, so a finding means the credential is probably being misused. Two cases are recorded without a finding: a method the gateway doesn't recognize (unknown_method), so a newer supervisor doesn't look like a compromised one, and a supervisor whose own sandbox was deleted mid-request, which is refused but isn't misuse. The shorthand line names the refused sandbox or workspace, for example sandbox:<id>, so operators reading console logs can see the target without JSONL or OTLP export. To support this, the OCSF crate's Actor can now hold a user instead of a process, and the API Activity builder gained the fields gateway events need. Supervisor inference events come out unchanged. Gateway events declare the container profile only when they concern a sandbox, as MXC findings do. The OCSF 1.8.0 resource_details schema is vendored so the refused sandbox or workspace goes in the standard resources field. Refs #2912 Signed-off-by: Kris Hicks <khicks@nvidia.com> |
||
|
ShijuandJohn Myers
|
292c9c8d48 |
fix(vm): flush guest filesystem before a planned sandbox stop (#4195)
* fix(vm): flush guest filesystem before a planned sandbox stop A planned stop of a MicroVM sandbox ended the VM before the guest wrote its dirty pages to disk, so files written shortly before the stop were missing after the next start. The driver closed the host supervisor's parent-liveness pipe before it sent SIGTERM. The supervisor exits at once on that end-of-file, so it never ran its orderly shutdown: terminate the workload, then ask the guest to terminate. Signal the supervisor first, keep the pipe open until it has exited, and give it a grace period derived from the supervisor's own bounds and kept below the gateway's reclaim deadline. In the guest runtime, syncfs the root filesystem after the process tree is terminated and before the termination is acknowledged, bounded to 3 seconds and best effort. Only the VM backend, which uses a vsock boundary listener, flushes; the container drivers share this runtime and are unchanged. A sandbox whose main process has already exited is not covered. Closes #4185 Signed-off-by: Shiju <shiju@nvidia.com> * fix(vm): warn about unconfirmed flush after supervisor kill Preserve the supervisor exit status so a planned stop warns when a signal killed the supervisor and guest writeback was not confirmed. Keep normal workload exit codes quiet, including 143 after a handled SIGTERM. Extend the existing process tests to check the warning after a forced kill and its absence after an orderly shutdown. Refs #4185 Signed-off-by: Shiju <shiju@nvidia.com> * fix(vm): satisfy shutdown timeout control-flow lint Use if let for the single successful timeout pattern while preserving the supervisor exit status and forced-kill error handling. Signed-off-by: Shiju <shiju@nvidia.com> * fix(vm): preserve shutdown merge resolutions during rebase Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> * docs(vm): explain planned stop durability and flush limits Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> --------- Signed-off-by: Shiju <shiju@nvidia.com> Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> Co-authored-by: John Myers <9696606+johntmyers@users.noreply.github.com> |
||
|
Oliver Calder
|
d4ad1bbd03 |
fix(snap): only enable gateway user service for one user (#4615)
* fix(snap): only enable gateway user service for one user Signed-off-by: Oliver Calder <oliver.calder@canonical.com> * docs(snap): correct snap docs for the gateway user service Signed-off-by: Oliver Calder <oliver.calder@canonical.com> * fix(snap): add documentation of global to per-user enablement migration Signed-off-by: Oliver Calder <oliver.calder@canonical.com> --------- Signed-off-by: Oliver Calder <oliver.calder@canonical.com> |
||
|
Eric Curtin
|
2798108b3f |
fix(ocsf): report https scheme for TLS-terminated requests (#4684)
* fix(ocsf): report https scheme for TLS-terminated requests Closes #4562 Signed-off-by: Eric Curtin <eric.curtin@docker.com> * fix(ocsf): use transport scheme on passthrough relay Signed-off-by: Eric Curtin <eric.curtin@docker.com> --------- Signed-off-by: Eric Curtin <eric.curtin@docker.com> |
||
|
Simon Scatton
|
b1cfc91720 |
ci(release): promote qualified prereleases to stable (#4611)
* ci(release): promote qualified prereleases to stable Signed-off-by: Simon Scatton <sscatton@nvidia.com> * fix(release): preserve qualified images during stable publication Signed-off-by: Simon Scatton <sscatton@nvidia.com> * ci(release): run shared release jobs with Nix Signed-off-by: Simon Scatton <sscatton@nvidia.com> * fix(release): recheck latest before publishing shared channels Signed-off-by: Simon Scatton <sscatton@nvidia.com> --------- Signed-off-by: Simon Scatton <sscatton@nvidia.com> |
||
|
krishicks
|
fdfbf84f8b |
fix(ocsf): report the required device.os.type_id (#4677)
OCSF 1.8 requires type_id on the os object, but device.os carried only a name. Events now report Linux (200), Windows (100), macOS (300), or Other (99) with the platform name as the type label. The os object schema is vendored so tests validate it. Refs #2912 Signed-off-by: Kris Hicks <khicks@nvidia.com> |
||
|
krishicks
|
f41f24b549 |
feat(auth): record peer address and request ID on authentication events (#4673)
Authentication events now carry the connection's immediate peer as src_endpoint, or edge-tunnel for WebSocket-tunneled requests, and the request's x-request-id as metadata.correlation_uid when it is a UUID. Refs #2912 Signed-off-by: Kris Hicks <khicks@nvidia.com> |
||
|
Piotr Mlocekanddivesh
|
604b9f66f6 |
feat(config): deliver gateway configuration over the supervisor session (#4320)
Supervisors poll the gateway for configuration every 10 seconds. This PR adds an opt-in push mode (config_delivery_mode = "push"): supervisors still start by polling, and once their ConnectSupervisor session is accepted, the gateway delivers every later change over it and supervisors apply it and report the result. poll stays the default, and older supervisors and gateways keep polling. --------- Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com> Signed-off-by: divesh <dgude@nvidia.com> Co-authored-by: divesh <dgude@nvidia.com> |
||
|
krishicks
|
35a3390dbc |
feat(auth): classify rejected gateway credentials in authentication events (#4655)
Authenticators now return the mechanism and a reason category with the status they reject a request with, and the gateway router records both in the Authentication event. Callers still receive the original status. Refs #2912 Signed-off-by: Kris Hicks <khicks@nvidia.com> |
||
|
Drew Newberry
|
77929d9c14 |
feat(sandbox): publish and activate sandbox backend protocol (#3898)
* feat(isolation): publish backend protocol contracts independently of runtime Signed-off-by: Drew Newberry <anewberry@nvidia.com> * feat(isolation): expose trusted launch and registry data contracts Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(isolation): name the required capability sandbox Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(sandbox): align protocol and compute capability names Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(sandbox): finalize public protocol contract names Signed-off-by: Drew Newberry <anewberry@nvidia.com> * feat(isolation)!: activate backend contracts in sandbox runtime and driver selection Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(sandbox): simplify runtime and fix backend example Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(examples): organize WASI backend in Rust Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(sandbox): defer protocol conformance fixtures Signed-off-by: Drew Newberry <anewberry@nvidia.com> * feat(examples): add supervised V8 backend demo Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(examples): stream V8 guest output with default policy Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(examples): handle V8 rejections and roll back failed launches Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(examples): align V8 crypto dependencies with CI AWS-LC Signed-off-by: Drew Newberry <anewberry@nvidia.com> * feat(sandbox): represent Sandbox Protocol operations in protobuf Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(isolation): replace backend traits with RPC clients Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(proto): remove legacy isolation protocol reservations Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(proto): clarify Sandbox Protocol names Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs(isolation): clarify authentication bootstrap and supervisor APIs Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(isolation): simplify policy enforcement confirmation Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(sandbox): rotate launch generations and stabilize retry sessions Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(sandbox): remove redundant backend adapters and TCP flags Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(sandbox): repair Docker restarts and macOS lint Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(isolation): simplify RPC contracts and backend registration Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(sandbox): deliver TCP denials before closing the RPC Signed-off-by: Drew Newberry <anewberry@nvidia.com> --------- Signed-off-by: Drew Newberry <anewberry@nvidia.com> |
||
|
Eric CurtinandJohn Myers
|
8fcba288cf |
feat(cli): add --annotation to sandbox create (#4325)
* feat(cli): add --annotation to sandbox create Signed-off-by: Eric Curtin <eric.curtin@docker.com> * fix(cli): drop reserved annotation check Signed-off-by: Eric Curtin <eric.curtin@docker.com> * test(cli): adapt annotation tests to env helper Signed-off-by: Eric Curtin <eric.curtin@docker.com> --------- Signed-off-by: Eric Curtin <eric.curtin@docker.com> Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> Co-authored-by: John Myers <9696606+johntmyers@users.noreply.github.com> |
||
|
krishicks
|
9c6867ccec |
feat(auth): emit OCSF events for credential-less gateway requests (#4632)
Requests the gateway rejects before consulting any authenticator now emit an OCSF Authentication [3002] event: no credential presented, mTLS required without a client certificate, or an operator certificate combined with an authorization header. Each event carries a reason category in status_detail and never the presented credential. The response returned to the caller is unchanged. openshell-ocsf gains the Authentication class with user and service objects, and a shared OcsfCapture test layer. Schema validation now rejects attributes the class does not define and profile attributes whose profile the event does not declare. Refs #2912 Signed-off-by: Kris Hicks <khicks@nvidia.com> |
||
|
Florent BENOIT
|
a4c848adbd |
feat(vm): add host bind mount support via virtiofs (libkrun) (#3463)
* feat(vm): add host bind mount support via virtiofs (libkrun) Introduce host directory bind mounts for libkrun VM sandboxes using the virtiofs4 API with simplified permission semantics. The gateway accepts `enable_bind_mounts` in the `[openshell.drivers.vm]` TOML table and forwards it to the VM driver subprocess. Mounts are specified per-sandbox in driver_config.mounts[] with source, target, optional read_only (default: read-only), and an optional `type` that must be `bind`, so Docker and Podman bind entries (without `selinux_label`) parse unchanged. At sandbox validation the driver rejects mounts unless bind mounts are enabled and resource admission is disabled. It rejects GPU and other QEMU launches, sources with control characters, and more mounts than libkrun has interrupt lines for (32 on aarch64, 3 on x86_64), and checks targets against reserved VM-internal directories and each other (no nested targets). The QEMU check runs again once the launch backend is resolved. Host source directories are checked at provisioning, before image preparation, so a briefly missing source fails provisioning visibly instead of denying recovery of a persisted sandbox. Missing, inaccessible, symlinked, non-directory, and unreadable sources are rejected: libkrun does not follow a final symlink, and on macOS it panics during guest boot when it cannot open the share root. The driver generates virtiofs tags and always writes a guest-side manifest into the overlay disk, together with the init drop-ins, so an image-supplied manifest can never drive guest mounts. The launcher parses its tab-separated --vm-mount arguments from the right, so a source can never shift the tag or mode, and the driver logs each shared directory at launch. Guest init mounts the shares after every /sandbox ownership fixup and the init drop-ins, so recursive chowns never reach host files. It refuses targets that traverse symlinks in the image, hands newly created mount point parents under /sandbox to the sandbox user, and skips an empty manifest. krun_add_virtiofs4 is resolved optionally, so older libkrun runtimes still boot sandboxes without mounts. The kernel config lists CONFIG_FUSE_FS and CONFIG_VIRTIO_FS explicitly; the pinned libkrunfw base config already enables both. A vm_bind_mount e2e test checks a read-write share round-trips data with the host, accepts a new file, directory, and nested file that the host then sees, and that a read-only share rejects writes. libkrun on rootless Linux refuses file creation by a guest user whose UID/GID differ from its own, so the VM e2e gateway runs the sandbox as the gateway user there. It enables bind mounts like the Docker and Podman e2e gateways. The runtimes docs cover the mount limit, the symlink restriction, the guest-kernel exposure, and the rootless Linux UID/GID note. The debug-openshell-cluster skill lists VM bind mount checks. Signed-off-by: Florent Benoit <fbenoit@redhat.com> * fix(nix): refresh vm-runtime hashes to the libkrun v1.19.4 runtime The Nix vm-runtime package fetches the rolling vm-runtime release with fixed hashes pinned on 2026-09-16. The release was rebuilt on 2026-09-21 after the libkrun v1.19.4 / libkrunfw v5.6.1 bump (#3451), but the hashes were not updated, so CI kept substituting the 2026-09-15 runtime (libkrun v1.17.4) from the binary cache. That libkrun lacks krun_add_virtiofs4, which VM bind mounts need. Point the hashes at the current release assets. Signed-off-by: Florent Benoit <fbenoit@redhat.com> --------- Signed-off-by: Florent Benoit <fbenoit@redhat.com> |
||
|
Roland Huss
|
5add9df2cd |
feat(sdk/go)!: replace positional labels with functional Create options (#3198)
* feat(sdk/go)!: replace positional labels with functional Create options
Move labels and annotations from separate parameter positions into
functional options (WithLabels, WithAnnotations) on Create() and
CreateFromTemplate().
Before: Create(ctx, ws, name, spec, labels, CreateOptions{Annotations: ann})
After: Create(ctx, ws, name, spec, WithLabels(labels), WithAnnotations(ann))
This unifies two resource metadata fields that were inconsistently
placed (labels as positional, annotations in a struct) into a single
extensible options pattern matching the existing LogOption convention.
BREAKING CHANGE: Create() and CreateFromTemplate() signatures changed.
All callers must use WithLabels() and WithAnnotations() functional
options instead of positional labels and CreateOptions struct.
Closes #2807
Signed-off-by: Roland Huß <rhuss@redhat.com>
* fix(sdk/go): ignore nil Create options
Callers of the previous signature passed nil for the positional labels
parameter to create an unlabeled sandbox:
Create(ctx, ws, name, spec, nil)
That call still compiles against the new variadic signature, because an
untyped nil converts to CreateOption. ApplyCreateOptions then invoked it
and panicked with a nil pointer dereference, so the migration was unsafe
for existing consumers.
Skip nil options in ApplyCreateOptions and cover the legacy trailing-nil
call with regression tests against ApplyCreateOptions directly, the fake
client (Create and CreateFromTemplate), and the gRPC client.
Signed-off-by: Roland Huß <rhuss@redhat.com>
* refactor(sdk/go): apply Create options through the shared options helper
Signed-off-by: Roland Huß <rhuss@redhat.com>
* docs(sdk/go): use WithLabels in CreateSandboxFromTemplate examples
Signed-off-by: Roland Huß <rhuss@redhat.com>
---------
Signed-off-by: Roland Huß <rhuss@redhat.com>
|
||
|
Drew Newberry
|
b2069a267c |
feat(docker): support host and bridge supervisor networking (#4016)
* feat(docker): select supervisor networking for ECI workloads Signed-off-by: Drew Newberry <anewberry@nvidia.com> * test(docker): use automatic networking in e2e Signed-off-by: Drew Newberry <anewberry@nvidia.com> * test(docker): rely on existing e2e networking coverage Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs(docker): shorten supervisor networking configuration Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(docker): let local gateway discover supervisor endpoint Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(docker): make supervisor networking operator-selected Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(docker): resolve host gateway in the driver Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs(docker): simplify host networking guidance Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs(docker): trim configuration guidance Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(docker): discover host gateway once at driver startup Signed-off-by: Drew Newberry <anewberry@nvidia.com> --------- Signed-off-by: Drew Newberry <anewberry@nvidia.com> |
||
|
Yuedong Wu
|
44c07dab2a |
feat(helm): make gateway PVC size and StorageClass configurable (#3854)
Signed-off-by: Yuedong Wu <dwcn22@outlook.com> |
||
|
Polite_realism
|
10241ab47b |
fix(podman): trust an operator-supplied additional CA for direct egress (#3996)
* fix(podman): trust an operator-supplied additional CA for direct egress Since v0.1.2 the Podman supervisor runs as its own container, separate from the workload image, so it can no longer inherit a private CA baked into the workload's system trust store. Policy-allowed, inspected HTTPS requests to destinations signed by that CA fail upstream TLS establishment, and the only workaround is rebuilding the supervisor image with the CA baked in. Add a Podman-only additional_ca_bundle setting, independent of the existing proxy_ca_bundle (which requires https_proxy), that bind-mounts an operator PEM into the supervisor container and folds it into the upstream trust store alongside the system CA bundle. Validated on the gateway host at sandbox-create time, matching the existing corporate proxy CA bundle's validation, so a misconfigured path or certificate-free bundle fails closed immediately rather than producing a failed sandbox once the supervisor container starts. Scoped to Podman only, reusing the existing additive-bundle mechanism without new gateway.toml schema, proto fields, or cross-driver plumbing -- a narrower alternative to the in-flight cross-driver implementation in #3292, which remains the intended long-term direction. Fixes #3781. Signed-off-by: politerealism <burdcat17@gmail.com> * fix(podman): keep additional CA mount on the supervisor build_isolation_specs() keeps only trusted_mount() destinations on the supervisor, and the additional CA path was not in that list. The bind mount stayed on the workload while the supervisor received --additional-ca-bundle for a file it did not have, so startup failed. Add the path to trusted_mount() and cover the final supervisor and workload specs with a test. Correct the docs and a code comment to say the bundle is validated when the driver starts, and add Podman troubleshooting guidance to the debug-openshell-cluster skill. Signed-off-by: politerealism <burdcat17@gmail.com> --------- Signed-off-by: politerealism <burdcat17@gmail.com> |
||
|
Drew Newberry
|
6546691d5e |
docs(kubernetes): document managed workspaces setup (#4622)
* docs(kubernetes): document managed workspaces setup Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs(kubernetes): rename workspace guide title Signed-off-by: Drew Newberry <anewberry@nvidia.com> --------- Signed-off-by: Drew Newberry <anewberry@nvidia.com> |
||
|
ShijuandJohn Myers
|
29a2b8e7d9 |
feat(providers): report bounded token-grant failures (#4427)
* feat(providers): report bounded token-grant failures Signed-off-by: Shiju <shiju@nvidia.com> * test(providers): align policy metadata and TLS lockfile Signed-off-by: Shiju <shiju@nvidia.com> * test(providers): fix trusted grant recovery fixture Signed-off-by: Shiju <shiju@nvidia.com> * test(providers): enable denial API in recovery fixture Signed-off-by: Shiju <shiju@nvidia.com> * refactor(providers): simplify token-grant failure reporting Remove duplication and dead paths from the token-grant failure reporting without changing classification, the workload-visible 502 response, or the wire format. - Build the gRPC failure envelope with tonic-types, store the typed failure in EndpointFailure, and share one request-ID rule and one endpoint-status authority check. - Merge the gateway exchange client wrapper, drop forwarding validators, and name the request field when a reported cause is rejected. - Inject fallback credentials only from live provider state and delete the unread per-connection credential copy. Keep one 502 writer per path and restore the base token-cache handling. - Check the token endpoint after the cache lookup, still before any credential acquisition. - Document 429 and 5xx classification as implemented, and drop tests that repeat retained assertions through the same code path. Signed-off-by: Shiju <shiju@nvidia.com> * refactor(providers): trim token-grant failure tests and supervisor API Cut the token-grant failure tests to one test per distinct behavior across core, gateway, supervisor, CLI and Go, and replace the 794-line closure E2E with status assertions in the existing route-recovery E2E: the refused grant appears in sandbox status with the response's request ID, code, stage and upstream status, and the matching grant's later success clears it. - Pin the live provider snapshot with one helper used by inspected and fallback forwarding, and inject selector-matched grants through inject_for_selectors, which borrows the pinned context. Move the test-only wrapper into test_support. - Drop the 35-second wrappers around the gateway grant RPC. The shared channel bounds the connection attempt and the gateway bounds its token-service calls. - Log the token service's OAuth error and error_description, without control characters and limited to 256 characters each, with the request ID in the gateway or supervisor log. Status and the workload response still carry only the bounded cause. - Keep one check that an untrusted token-service certificate cannot assert a cause. Signed-off-by: Shiju <shiju@nvidia.com> * fix(providers): restore test import and features lockfile The injection tests named EndpointObservationSender through the parent module, which no longer imports it. The features suite lockfile kept the tonic TLS dependency that the removed E2E fixture added. Signed-off-by: Shiju <shiju@nvidia.com> * test(providers): fix body-bound fixture width and table type Pass the response-size width as a named argument; an implicitly captured constant panics at runtime. Name the mutation type in the validation table for Clippy. Signed-off-by: Shiju <shiju@nvidia.com> * test(providers): pad exact-bound body without a format width MAX_TOKEN_RESPONSE_BYTES exceeds u16::MAX, the largest format width, so the padded fixture panicked. Pad with repeat_n as before. Signed-off-by: Shiju <shiju@nvidia.com> * test(server): update public RPC schema hash after rebase Main added the ListProvidersRequest label selector. The merged public schema hash comes from the Branch Checks storage_proto test output. Signed-off-by: Shiju <shiju@nvidia.com> * fix(providers): keep token-service error text out of grant logs A token service can echo a submitted credential in its OAuth error or error_description. Log only the classified failure code, HTTP status and request ID, and cover the canary echo in a dedicated test binary. Signed-off-by: Shiju <shiju@nvidia.com> * docs(providers): tighten token grant failure documentation Signed-off-by: Shiju <shiju@nvidia.com> * docs(providers): condense token grant failure guidance Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> --------- Signed-off-by: Shiju <shiju@nvidia.com> Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> Co-authored-by: John Myers <9696606+johntmyers@users.noreply.github.com> |
||
|
alangou
|
c5519ea8ba |
fix(network): bind hostname policy to supervisor DNS (#4602)
Signed-off-by: Adrien Langou <alangou@nvidia.com> |
||
|
Simon Scatton
|
253884414a |
ci(release): decouple prerelease publication from qualification (#4420)
* ci(release): publish prereleases before qualification Signed-off-by: Simon Scatton <sscatton@nvidia.com> * ci(release): decouple prerelease publication from qualification Signed-off-by: Simon Scatton <sscatton@nvidia.com> --------- Signed-off-by: Simon Scatton <sscatton@nvidia.com> |
||
|
John T. Myers
|
909f336b62 |
fix(kubernetes): validate runtime control mount paths (#4522)
Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com> |
||
|
krishicks
|
d20c090a69 |
docs(observability): add observability overview and traces pages (#4560)
Add an Observability overview that lists the traces, metrics, and OCSF events OpenShell produces, where each goes, and the minimal configuration to enable it, linking to the detailed pages instead of repeating them. Add a Traces page for what is traced, span names, resource attributes, gateway names, and trace context, and point the OTLP Export reference to it instead of describing spans there. Move metric naming conventions into Gateway Metrics, and replace the stale denied-connection example in OCSF JSON Export with a real record. List every common OTEL_* environment variable in the gateway OTLP tuning reference, split into the variables the gateway honors and the ones it ignores, with the reasons. Closes #4548 Signed-off-by: Kris Hicks <khicks@nvidia.com> |
||
|
krishicks
|
6dbe4adc56 |
feat(supervisor): export OCSF events as OTLP log records (#4545)
Supervisor OCSF events reached only the sandbox's local JSONL file and the shorthand log lines sent to the gateway, so collectors had no structured copy of most of the security record. When the gateway has `[openshell.gateway.otlp.ocsf]` configured, compute drivers now pass its endpoint and schema version to supervisors, which export their own OCSF events directly as OTLP log records. Supervisors already have egress, so the gateway and its supervisor connection carry no extra load. Supervisors use the same `openshell-otel` exporter as the gateway, with a supervisor resource identity that includes `openshell.sandbox.id` and `openshell.gateway.name`, so telemetry from sandboxes of different gateways sharing a collector can be told apart. This applies to supervisor spans too. Drivers pass the settings to supervisors as `OPENSHELL_OTLP_OCSF_ENDPOINT`, `OPENSHELL_OTLP_OCSF_SCHEMA_VERSION`, and `OPENSHELL_GATEWAY_NAME`; the gateway passes them to the VM driver as arguments. Collector headers that the gateway uses for OCSF export (`OTEL_EXPORTER_OTLP_LOGS_HEADERS`, else `OTEL_EXPORTER_OTLP_HEADERS`) travel in each sandbox's supervisor authentication bundle, which only the supervisor reads, so authenticated collectors accept supervisor records without exposing credentials to the workload. The VM driver now sends SIGTERM and waits for the supervisor to exit before closing its liveness pipe. Closing the pipe first made the supervisor exit at once and drop queued records. Closes #4544 Refs #2892 Signed-off-by: Kris Hicks <khicks@nvidia.com> |
||
|
krishicks
|
07fd2386ce |
feat(server): export gateway OCSF events as OTLP log records (#4500)
The gateway only persisted its OCSF events to the JSONL file, so collectors had to tail that file. Adding `[openshell.gateway.otlp.ocsf]` now also exports each gateway OCSF event as an OTLP log record, using the same resource identity as gateway traces. `schema_version` optionally downgrades the exported JSON to OCSF 1.1 or 1.3. OTLP endpoints are now configured per signal, so OCSF events can go to a collector that feeds a SIEM while spans go elsewhere. The existing `[openshell.gateway.otlp]` `endpoint` is the default for every signal and on its own still enables trace export. `[otlp.traces]` overrides the trace endpoint, and `[otlp.ocsf]` takes its own `endpoint` or falls back to the default. Each record carries the OCSF JSON as its body, the event time as its timestamp, a severity mapped from `severity_id`, and `ocsf.*` and `openshell.sandbox.id` attributes for filtering without parsing the body. Records flow through a queue bounded by count and bytes to the SDK OTLP log exporter in batches bounded the same way. Failed exports retry with backoff; records lost to a full queue, an oversized body, or exhausted retries are counted in metrics and reported to the collector in an `openshell.ocsf.gap` record on the next successful export. The exporter lives in `openshell-otel` so other processes can share it. Helm deployments set the `traces` and `ocsf` tables through `gatewayConfig`, and the local development gateway tasks enable OCSF export. Closes #4499 Refs #2892 Signed-off-by: Kris Hicks <khicks@nvidia.com> |
||
|
Yuedong Wu
|
3f66c26af1 |
fix(helm): render Restricted-compatible security contexts (#3340)
Signed-off-by: Yuedong Wu <dwcn22@outlook.com> |
||
|
Matthew Grossman
|
42e0d00a1d |
fix(supervisor): rename OCSF log files from .log to .jsonl (#4263)
* fix(supervisor): use jsonl files for independent OCSF retention Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * test(supervisor): trim and separate log retention regressions Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * refactor(supervisor): drop legacy OCSF log migration Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * docs(observability): remove legacy OCSF filename guidance Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * docs(ocsf): remove stale supervisor log filename comments Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * docs(supervisor): clarify denial reader log source Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * docs(supervisor): retain OCSF JSON opt-in explanation Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> --------- Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> |
||
|
Drew Newberry
|
6e69360d0e |
refactor(runtime): extract portable driver and isolation foundations (#4106)
* refactor(runtime): decouple supervisor access and boundary audit validation Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(runtime): keep supervisor process access boundary driven Signed-off-by: Drew Newberry <anewberry@nvidia.com> * feat(cli): support environment values from caller variables Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(runtime): extract driver-independent MXC foundations Signed-off-by: Drew Newberry <anewberry@nvidia.com> * refactor(runtime): defer UI policy support to MXC integration Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(runtime): integrate current policy and SSH launch contracts Signed-off-by: Drew Newberry <anewberry@nvidia.com> * test(runtime): reconcile fixtures with updated main Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(runtime): repair stacked branch CI contracts Signed-off-by: Drew Newberry <anewberry@nvidia.com> * test(runtime): update boundary fixtures for current contracts Signed-off-by: Drew Newberry <anewberry@nvidia.com> * docs(runtime): clarify scoped guards for launch inputs and delete Signed-off-by: Drew Newberry <anewberry@nvidia.com> * fix(runtime): validate foreground attachment intent before launch Signed-off-by: Drew Newberry <anewberry@nvidia.com> --------- Signed-off-by: Drew Newberry <anewberry@nvidia.com> |