* feat(k8s): support ImageVolumeSource for supervisor sideload
Add a config-driven switch between two supervisor binary delivery
methods: image-volume (default) mounts the supervisor OCI image
directly as a read-only volume (K8s >= v1.33), while init-container
preserves the existing emptyDir + copy-self pattern for older clusters.
Configurable via --supervisor-sideload-method CLI arg,
OPENSHELL_SUPERVISOR_SIDELOAD_METHOD env var, or Helm
supervisor.sideloadMethod value.
* feat(helm): auto-detect supervisor sideload method from cluster version
Use Helm .Capabilities.KubeVersion to choose the supervisor sideload
method automatically: image-volume on K8s >= v1.33, init-container on
older clusters. An explicit supervisor.sideloadMethod value overrides
auto-detection.
* fix(helm): raise auto-detect threshold to K8s v1.35
The ImageVolume feature gate is only enabled by default starting in
K8s v1.35, not v1.33 (where it is beta but off by default). Clusters
on v1.33-v1.34 can still opt in with an explicit sideloadMethod value.
* fix(helm): derive grpcEndpoint from chart context
The chart hardcoded server.grpcEndpoint to
https://openshell.openshell.svc.cluster.local:8080, which only matched the
in-cluster Service DNS for the standard release name and namespace. A new
helper now builds <scheme>://<fullname>.<namespace>.svc.cluster.local:<port>
from chart context, picking the scheme from server.disableTls. An explicit
server.grpcEndpoint override is passed through verbatim.
* chore(scripts): validate k3d cluster name length early
helm-k3s-local.sh derives the cluster name from the current branch suffix.
Long branch names produced names exceeding k3d's 32-char cap and failed
deep inside k3d cluster create with a confusing validation error. cmd_create
now bails out before invoking docker/k3d with a copy-pasteable
HELM_K3S_CLUSTER_NAME override hint. Status, start, stop, delete, and help
remain unaffected so an over-long derived name does not block diagnostics.
Add opt-in support for Kubernetes user namespace isolation on sandbox
pods. When enabled, container UID 0 maps to an unprivileged host UID
and capabilities become namespaced, providing defense-in-depth for the
supervisor process.
Configuration is two-layered: a cluster-wide default via
OPENSHELL_ENABLE_USER_NAMESPACES (default false) and a per-sandbox
override via the new `user_namespaces` field on SandboxTemplate.
When user namespaces are active, the pod security context is extended
with SETUID, SETGID, and DAC_READ_SEARCH capabilities to match the
bounding-set requirements inside a user namespace.
Introduces SandboxPodParams struct to replace long argument lists on
sandbox_to_k8s_spec and sandbox_template_to_k8s.
Validated end-to-end on OCP 4.22 (K8s 1.35.3, CRI-O 1.35, RHEL
CoreOS, kernel 5.14) with full SSH tunnel and non-identity UID mapping.
* docs(helm): add agent sandbox prerequisite to Helm README
Add a Prerequisites section requiring the Kubernetes Agent Sandbox
CRDs and controller (v0.4.3) to be installed before deploying OpenShell.
* docs(helm): use latest release URL for agent sandbox manifest
* ci(helm): add OCI chart release workflow
Publishes the Helm chart to ghcr.io/nvidia/openshell/helm-chart via helm push on every tag (versioned + :latest) and main push (:0.0.0-dev overwrite + :0.0.0-dev.<sha> per-commit pin). Renames Chart.yaml name to helm-chart to match the target OCI path.
* ci(helm): use full sha in dev pinned chart version
The container images pushed by docker-build.yml are tagged with the
full $GITHUB_SHA, not a 7-char prefix. Match the chart's pinned
version (0.0.0-dev.<full-sha>) so the chart tag and the image tag
the chart resolves to are identical.
* docs(readme): add helm chart install instructions
Document the OCI chart at ghcr.io/nvidia/openshell/helm-chart with
examples for tagged, floating dev, and SHA-pinned dev installs, and
flag the Kubernetes deployment path as experimental.
* docs(helm): split chart details into chart README
Move the dev tag conventions and configuration pointers into a new
README under deploy/helm/openshell/ and shorten the top-level README
section to the install command (no --version, defaults to latest
tagged semver) plus a link to the chart README.
* fix(helm): grant node read access for GPU capacity checks
Signed-off-by: Jonas Toelke <jtoelke@nvidia.com>
* ci: add WSL GPU failure diagnostics
Signed-off-by: Jonas Toelke <jtoelke@nvidia.com>
* fix(gpu): bump device plugin for WSL CDI
* fix(sandbox): allow WSL GPU paths in Landlock
Allow /dev/dxg and /usr/lib/wsl as GPU baseline paths so WSL CDI GPU sandboxes can initialize NVML. Native Linux skips these entries when the paths do not exist.
* ci: remove temporary WSL GPU diagnostics
---------
Signed-off-by: Jonas Toelke <jtoelke@nvidia.com>
* feat(auth): add OIDC/Keycloak authentication with RBAC
Add OAuth2/OIDC authentication to the gateway server with role-based
access control, CLI login flows, and full deployment plumbing.
Server: JWT validation against configurable OIDC issuer (oidc.rs),
JWKS key caching with TTL and rotation handling, method classification
(unauthenticated/sandbox-secret/dual-auth/bearer), identity extraction
with provider-agnostic Identity type, and RBAC enforcement via
AuthzPolicy with configurable admin/user roles and auth-only mode.
CLI: browser-based Authorization Code + PKCE flow, Client Credentials
flow for CI/automation, token storage with refresh, gateway add/login/
logout commands, OIDC bearer token injection over mTLS transport,
discovery endpoint for auto-configuration.
Security: sandbox-secret scope restriction on UpdateConfig (policy
sync only), anti-spoofing header stripping, dual-auth fallthrough
from sandbox-secret to Bearer token.
Deployment: OIDC config wired through DeployOptions, Docker env vars,
Helm values/templates, HelmChart manifest, cluster-entrypoint.sh, and
bootstrap scripts. Keycloak dev server script with pre-configured
realm (test users, roles, PKCE client, CI client).
Tested with Keycloak. The roles claim path and role names are
configurable to support other OIDC providers.
* feat(auth): add OAuth2 scope-based fine-grained permissions
Add opt-in scope enforcement on top of existing OIDC role-based access
control. When --oidc-scopes-claim is set, the server extracts scopes
from the JWT and checks them per-method against an exhaustive scope map.
Scopes: sandbox:read, sandbox:write, provider:read, provider:write,
config:read, config:write, inference:read, inference:write, and
openshell:all (wildcard). Methods not in the scope map require
openshell:all. Scopes layer on top of roles and cannot escalate
privilege. Auth-only mode (empty role names) still enforces scopes
when enabled.
Server: scopes_claim in OidcConfig, scope extraction from JWT
(space-delimited and JSON array formats), standard OIDC scope
filtering, scope check in AuthzPolicy after role check.
CLI: --oidc-scopes on gateway add/start stored in metadata and
consumed by gateway login, --oidc-scopes-claim on gateway start
forwarded to server, scopes parameter in browser and client
credentials OAuth2 flows with openid deduplication.
Deployment: oidc_scopes_claim wired through DeployOptions, docker.rs,
Helm, bootstrap scripts, and cluster entrypoint.
Keycloak: realm config updated with built-in OIDC scopes and 9
OpenShell client scopes as optional on openshell-cli and openshell:all
as default on openshell-ci.
* fix(auth): address branch review findings
Add GetInferenceBundle to sandbox-secret methods so sandbox inference
route refresh works under OIDC. Make GetSandboxConfig dual-auth so CLI
users can read sandbox settings with Bearer tokens.
Preserve OIDC gateway metadata on restart — a bare gateway start
without --oidc-* flags no longer erases the stored OIDC registration.
Document CI client ID requirement (openshell-ci vs openshell-cli) in
the testing guide. Add security note about auth-only mode blast radius
for GitHub Actions.
* fix(auth): complete review findings for OIDC auth boundary
Move OpenShell/GetSandboxConfig from sandbox-secret-only to dual-auth
so CLI users can read sandbox settings with Bearer tokens while sandbox
supervisors continue using the shared secret.
Add sandbox secret interceptor to the inference bundle fetch path so
GetInferenceBundle works under OIDC-enabled gateways. Extract shared
interceptor constructor to avoid duplication.
Add GetSandboxConfig to the config:read scope map so scope enforcement
applies consistently when scopes are enabled.
Refactor OIDC metadata preservation into apply_oidc_gateway_metadata()
with explicit resume semantics — only preserve existing OIDC metadata
on real resume paths, not on fresh deployments.
Update architecture docs and testing guide to reflect the corrected
method classifications and add new test coverage for interceptor
injection, scope requirements, metadata preservation, and dual-auth
classification.
* refactor(auth): use oauth2 crate for CLI OIDC flows
Replace hand-written PKCE generation, authorization URL construction,
token exchange, client credentials, and token refresh with the oauth2
crate's typed API.
Eliminates sha2, hex, and getrandom dependencies from the CLI. The
custom urlencoded() helper and manual form POST logic are replaced by
BasicClient methods with proper type-state safety.
Discovery and the callback server remain custom since the oauth2 crate
does not provide OIDC discovery or a localhost redirect listener.
* refactor(auth): move server auth modules into auth/ directory
Group oidc.rs, authz.rs, identity.rs, and the auth HTTP endpoints
under src/auth/ module directory. No behavioral changes.
auth/mod.rs — module root, re-exports HTTP router
auth/oidc.rs — JWT validation, JWKS caching, method classification
auth/authz.rs — role and scope authorization policy
auth/identity.rs — provider-agnostic Identity type
auth/http.rs — /auth/connect and /auth/oidc-config endpoints
* fix(auth): use RequestBody auth type for client credentials flow
The oauth2 crate defaults to BasicAuth (HTTP Basic header) but Keycloak
and most OIDC providers expect client_secret_post (credentials in the
request body). Set AuthType::RequestBody explicitly to match the
pre-refactor behavior.
Also re-export Identity, IdentityProvider, and JwksCache from the auth
module so ServerState's public API remains nameable by external consumers.
* fix(auth): forward OPENSHELL_OIDC_SCOPES through cluster bootstrap
Pass --oidc-scopes to gateway start so the metadata includes requested
scopes after cluster bootstrap. Without this, users had to manually
edit metadata.json to set scopes for gateway login.
Usage: OPENSHELL_OIDC_SCOPES="openshell:all" mise run cluster
* test(auth): add OIDC e2e tests for RBAC, scopes, and client credentials
Add 10 end-to-end tests covering OIDC authentication against a live
K3s cluster with Keycloak:
RBAC (5 tests): admin can create providers, user cannot, user can list
sandboxes, unauthenticated requests rejected, health probe works
without auth.
Scopes (4 tests): sandbox-scoped token can list sandboxes but not
providers, openshell:all grants full access, no-scopes token denied.
Client credentials (1 test): CI token via client_credentials grant.
Tests are opt-in via OPENSHELL_E2E_OIDC=1 and OPENSHELL_E2E_OIDC_SCOPES=1
env vars. They derive the Keycloak URL from gateway metadata to match
the server's configured issuer.
Run with:
OPENSHELL_E2E_OIDC=1 OPENSHELL_E2E_OIDC_SCOPES=1 \
PYTHONPATH=python uv run pytest e2e/python/oidc/ -v
* fix(docs): fix markdown lint errors in OIDC architecture docs
Add blank lines before lists and fenced code blocks to satisfy
markdownlint MD031 and MD032 rules.
* feat(server): add Prometheus metrics infrastructure and gRPC/HTTP request metrics
Add metrics exposition via a dedicated server port (--metrics-port,
default disabled) following the same optional-listener pattern as the
health port. The metrics crate facade records counters and histograms
in the MultiplexedService hot path, and a PrometheusHandle renders them
at GET /metrics on the dedicated port.
Metrics added:
- openshell_server_grpc_requests_total (counter: method, code)
- openshell_server_grpc_request_duration_seconds (histogram: method, code)
- openshell_server_http_requests_total (counter: path, status)
- openshell_server_http_request_duration_seconds (histogram: path, status)
* feat(helm): add metrics port to openshell-server helm chart
Expose the Prometheus metrics endpoint in the helm chart by adding
service.metricsPort (default 9090) to values, the --metrics-port arg
and container port to the statefulset, and a metrics service port.
Set metricsPort to 0 to disable.
Move /health, /healthz, and /readyz to a dedicated plaintext HTTP port
(default 8081) so Kubernetes probes work without mTLS client certificates.
- Add health_bind_address to Config with --health-port CLI arg
- Spawn standalone axum::serve for health_router on the health port
- Remove health routes from the main multiplexed HTTP router
- Update Helm statefulset probes from tcpSocket to httpGet on health port
- Fix cluster-healthcheck.sh to open/close TCP without sending data,
avoiding InvalidContentType TLS errors in the gateway log
Closes#273
Verify inference endpoints synchronously on the server during set/update, expose a --no-verify escape hatch in the CLI and Python helper, and return actionable failures when validation does not pass.
* feat(sandbox): inject host gateway hostAliases into sandbox pods
Sandbox pods running in the k3s cluster cannot resolve host.docker.internal
by default, preventing them from reaching services on the Docker host.
Detect the host gateway IP (default route) in the cluster entrypoint,
thread it through the Helm chart to the gateway server, and inject
hostAliases entries (host.docker.internal, host.openshell.internal)
into every sandbox pod spec. The injection is conditional -- when the
IP is empty (non-Docker deployments), no hostAliases are added.
---------
Co-authored-by: Piotr Mlocek <pmlocek@nvidia.com>
* feat(sandbox): add gpu sandbox scheduling support
Allow sandbox creation to request GPU resources explicitly or infer them from GPU image names. This wires GPU intent through bootstrap, validates gateway support, and adds dedicated GPU E2E coverage for follow-up cluster testing.
Add OPENSHELL_SANDBOX_IMAGE_PULL_POLICY env/CLI flag that propagates
through Config -> SandboxClient -> pod spec generation. When set, the
policy is applied to both inline and pod-template sandbox containers.
Helm chart exposes server.sandboxImagePullPolicy (default empty = K8s
default behavior). cluster-deploy-fast sets it to Always so dev
clusters always pull fresh sandbox images.
* refactor(docker): rename server image to gateway
Rename Dockerfile.server to Dockerfile.gateway and update all image
references from openshell/server to openshell/gateway across Helm
charts, Kubernetes manifests, mise tasks, build/deploy scripts, CI
workflows, and documentation.
The underlying Rust binary (navigator-server) is unchanged -- this
rename only affects the Docker image name and Dockerfile.
* fix: catch remaining server->gateway references in docs and comments
Mrunal Patel