Files
OpenShell/e2e/python/oidc/oidc_auth_test.py
Mrunal Patel 084505425b feat(auth): add OIDC/Keycloak authentication with RBAC and scope-based permissions (#935)
* feat(auth): add OIDC/Keycloak authentication with RBAC

Add OAuth2/OIDC authentication to the gateway server with role-based
access control, CLI login flows, and full deployment plumbing.

Server: JWT validation against configurable OIDC issuer (oidc.rs),
JWKS key caching with TTL and rotation handling, method classification
(unauthenticated/sandbox-secret/dual-auth/bearer), identity extraction
with provider-agnostic Identity type, and RBAC enforcement via
AuthzPolicy with configurable admin/user roles and auth-only mode.

CLI: browser-based Authorization Code + PKCE flow, Client Credentials
flow for CI/automation, token storage with refresh, gateway add/login/
logout commands, OIDC bearer token injection over mTLS transport,
discovery endpoint for auto-configuration.

Security: sandbox-secret scope restriction on UpdateConfig (policy
sync only), anti-spoofing header stripping, dual-auth fallthrough
from sandbox-secret to Bearer token.

Deployment: OIDC config wired through DeployOptions, Docker env vars,
Helm values/templates, HelmChart manifest, cluster-entrypoint.sh, and
bootstrap scripts. Keycloak dev server script with pre-configured
realm (test users, roles, PKCE client, CI client).

Tested with Keycloak. The roles claim path and role names are
configurable to support other OIDC providers.

* feat(auth): add OAuth2 scope-based fine-grained permissions

Add opt-in scope enforcement on top of existing OIDC role-based access
control. When --oidc-scopes-claim is set, the server extracts scopes
from the JWT and checks them per-method against an exhaustive scope map.

Scopes: sandbox:read, sandbox:write, provider:read, provider:write,
config:read, config:write, inference:read, inference:write, and
openshell:all (wildcard). Methods not in the scope map require
openshell:all. Scopes layer on top of roles and cannot escalate
privilege. Auth-only mode (empty role names) still enforces scopes
when enabled.

Server: scopes_claim in OidcConfig, scope extraction from JWT
(space-delimited and JSON array formats), standard OIDC scope
filtering, scope check in AuthzPolicy after role check.

CLI: --oidc-scopes on gateway add/start stored in metadata and
consumed by gateway login, --oidc-scopes-claim on gateway start
forwarded to server, scopes parameter in browser and client
credentials OAuth2 flows with openid deduplication.

Deployment: oidc_scopes_claim wired through DeployOptions, docker.rs,
Helm, bootstrap scripts, and cluster entrypoint.

Keycloak: realm config updated with built-in OIDC scopes and 9
OpenShell client scopes as optional on openshell-cli and openshell:all
as default on openshell-ci.

* fix(auth): address branch review findings

Add GetInferenceBundle to sandbox-secret methods so sandbox inference
route refresh works under OIDC. Make GetSandboxConfig dual-auth so CLI
users can read sandbox settings with Bearer tokens.

Preserve OIDC gateway metadata on restart — a bare gateway start
without --oidc-* flags no longer erases the stored OIDC registration.

Document CI client ID requirement (openshell-ci vs openshell-cli) in
the testing guide. Add security note about auth-only mode blast radius
for GitHub Actions.

* fix(auth): complete review findings for OIDC auth boundary

Move OpenShell/GetSandboxConfig from sandbox-secret-only to dual-auth
so CLI users can read sandbox settings with Bearer tokens while sandbox
supervisors continue using the shared secret.

Add sandbox secret interceptor to the inference bundle fetch path so
GetInferenceBundle works under OIDC-enabled gateways. Extract shared
interceptor constructor to avoid duplication.

Add GetSandboxConfig to the config:read scope map so scope enforcement
applies consistently when scopes are enabled.

Refactor OIDC metadata preservation into apply_oidc_gateway_metadata()
with explicit resume semantics — only preserve existing OIDC metadata
on real resume paths, not on fresh deployments.

Update architecture docs and testing guide to reflect the corrected
method classifications and add new test coverage for interceptor
injection, scope requirements, metadata preservation, and dual-auth
classification.

* refactor(auth): use oauth2 crate for CLI OIDC flows

Replace hand-written PKCE generation, authorization URL construction,
token exchange, client credentials, and token refresh with the oauth2
crate's typed API.

Eliminates sha2, hex, and getrandom dependencies from the CLI. The
custom urlencoded() helper and manual form POST logic are replaced by
BasicClient methods with proper type-state safety.

Discovery and the callback server remain custom since the oauth2 crate
does not provide OIDC discovery or a localhost redirect listener.

* refactor(auth): move server auth modules into auth/ directory

Group oidc.rs, authz.rs, identity.rs, and the auth HTTP endpoints
under src/auth/ module directory. No behavioral changes.

  auth/mod.rs      — module root, re-exports HTTP router
  auth/oidc.rs     — JWT validation, JWKS caching, method classification
  auth/authz.rs    — role and scope authorization policy
  auth/identity.rs — provider-agnostic Identity type
  auth/http.rs     — /auth/connect and /auth/oidc-config endpoints

* fix(auth): use RequestBody auth type for client credentials flow

The oauth2 crate defaults to BasicAuth (HTTP Basic header) but Keycloak
and most OIDC providers expect client_secret_post (credentials in the
request body). Set AuthType::RequestBody explicitly to match the
pre-refactor behavior.

Also re-export Identity, IdentityProvider, and JwksCache from the auth
module so ServerState's public API remains nameable by external consumers.

* fix(auth): forward OPENSHELL_OIDC_SCOPES through cluster bootstrap

Pass --oidc-scopes to gateway start so the metadata includes requested
scopes after cluster bootstrap. Without this, users had to manually
edit metadata.json to set scopes for gateway login.

Usage: OPENSHELL_OIDC_SCOPES="openshell:all" mise run cluster

* test(auth): add OIDC e2e tests for RBAC, scopes, and client credentials

Add 10 end-to-end tests covering OIDC authentication against a live
K3s cluster with Keycloak:

RBAC (5 tests): admin can create providers, user cannot, user can list
sandboxes, unauthenticated requests rejected, health probe works
without auth.

Scopes (4 tests): sandbox-scoped token can list sandboxes but not
providers, openshell:all grants full access, no-scopes token denied.

Client credentials (1 test): CI token via client_credentials grant.

Tests are opt-in via OPENSHELL_E2E_OIDC=1 and OPENSHELL_E2E_OIDC_SCOPES=1
env vars. They derive the Keycloak URL from gateway metadata to match
the server's configured issuer.

Run with:

  OPENSHELL_E2E_OIDC=1 OPENSHELL_E2E_OIDC_SCOPES=1 \
  PYTHONPATH=python uv run pytest e2e/python/oidc/ -v

* fix(docs): fix markdown lint errors in OIDC architecture docs

Add blank lines before lists and fenced code blocks to satisfy
markdownlint MD031 and MD032 rules.
2026-04-30 10:37:23 -07:00

280 lines
10 KiB
Python

# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
"""End-to-end tests for OIDC authentication, RBAC, and scope enforcement.
These tests require:
- A running K3s cluster with OIDC enabled (OPENSHELL_OIDC_ISSUER set)
- A running Keycloak instance with the openshell realm
- The cluster started with OPENSHELL_OIDC_SCOPES_CLAIM=scope
Skip condition: set OPENSHELL_E2E_OIDC=1 to enable these tests.
"""
from __future__ import annotations
import contextlib
import json
import os
import urllib.parse
import urllib.request
from pathlib import Path
import grpc
import pytest
from openshell._proto import datamodel_pb2, openshell_pb2, openshell_pb2_grpc
KEYCLOAK_REALM = "openshell"
def _xdg_config_home() -> Path:
return Path(os.environ.get("XDG_CONFIG_HOME", Path.home() / ".config"))
def _keycloak_url() -> str:
"""Derive the Keycloak URL from the gateway's stored OIDC issuer.
The server validates the issuer claim in JWTs, so the token must be
requested from the same base URL the server was configured with
(typically the host IP, not localhost).
"""
if url := os.environ.get("OPENSHELL_KEYCLOAK_URL"):
return url
cluster_name = os.environ.get("OPENSHELL_GATEWAY", "openshell")
metadata_path = (
_xdg_config_home() / "openshell" / "gateways" / cluster_name / "metadata.json"
)
if metadata_path.exists():
metadata = json.loads(metadata_path.read_text())
issuer = metadata.get("oidc_issuer", "")
if issuer:
# issuer is like "http://192.168.4.172:8180/realms/openshell"
# extract base URL before /realms/
idx = issuer.find("/realms/")
if idx > 0:
return issuer[:idx]
return "http://localhost:8180"
TOKEN_ENDPOINT = (
f"{_keycloak_url()}/realms/{KEYCLOAK_REALM}/protocol/openid-connect/token"
)
pytestmark = pytest.mark.skipif(
os.environ.get("OPENSHELL_E2E_OIDC") != "1",
reason="OIDC e2e tests disabled (set OPENSHELL_E2E_OIDC=1)",
)
def _gateway_endpoint() -> tuple[str, bool]:
"""Read the active gateway endpoint from metadata."""
cluster_name = os.environ.get("OPENSHELL_GATEWAY", "openshell")
metadata_path = (
_xdg_config_home() / "openshell" / "gateways" / cluster_name / "metadata.json"
)
metadata = json.loads(metadata_path.read_text())
endpoint = metadata["gateway_endpoint"]
is_tls = endpoint.startswith("https://")
return endpoint, is_tls
def _mtls_dir() -> Path:
cluster_name = os.environ.get("OPENSHELL_GATEWAY", "openshell")
return _xdg_config_home() / "openshell" / "gateways" / cluster_name / "mtls"
def _token_request(data: dict[str, str]) -> str:
"""POST to the Keycloak token endpoint and return the access token."""
encoded = urllib.parse.urlencode(data).encode()
req = urllib.request.Request(TOKEN_ENDPOINT, data=encoded)
with urllib.request.urlopen(req, timeout=10) as resp:
body = json.loads(resp.read())
return body["access_token"]
def _get_token(
username: str,
password: str,
*,
client_id: str = "openshell-cli",
scopes: str | None = None,
) -> str:
"""Get an access token from Keycloak via password grant."""
data = {
"grant_type": "password",
"client_id": client_id,
"username": username,
"password": password,
}
if scopes:
data["scope"] = scopes
return _token_request(data)
def _get_ci_token(
*,
client_id: str = "openshell-ci",
client_secret: str = "ci-test-secret",
) -> str:
"""Get an access token via client credentials grant."""
return _token_request(
{
"grant_type": "client_credentials",
"client_id": client_id,
"client_secret": client_secret,
}
)
def _grpc_channel() -> grpc.Channel:
"""Create a gRPC channel to the gateway with mTLS transport."""
endpoint, is_tls = _gateway_endpoint()
parsed = urllib.parse.urlparse(endpoint)
host = parsed.hostname or "127.0.0.1"
port = parsed.port or (443 if is_tls else 80)
target = f"{host}:{port}"
if is_tls:
mtls = _mtls_dir()
ca_cert = (mtls / "ca.crt").read_bytes()
client_cert = (mtls / "tls.crt").read_bytes()
client_key = (mtls / "tls.key").read_bytes()
creds = grpc.ssl_channel_credentials(
root_certificates=ca_cert,
private_key=client_key,
certificate_chain=client_cert,
)
return grpc.secure_channel(target, creds)
return grpc.insecure_channel(target)
def _stub_with_token(token: str) -> openshell_pb2_grpc.OpenShellStub:
"""Create a gRPC stub that injects a Bearer token."""
channel = _grpc_channel()
return openshell_pb2_grpc.OpenShellStub(channel), [
("authorization", f"Bearer {token}")
]
# ── RBAC Tests ────────────────────────────────────────────────────────
class TestRbac:
"""Test role-based access control."""
def test_admin_can_create_provider(self) -> None:
token = _get_token("admin@test", "admin", scopes="openid openshell:all")
stub, metadata = _stub_with_token(token)
req = openshell_pb2.CreateProviderRequest(
provider=datamodel_pb2.Provider(
name="e2e-oidc-admin-test",
type="claude",
credentials={"API_KEY": "test-value"},
)
)
try:
stub.CreateProvider(req, metadata=metadata)
except grpc.RpcError as e:
if e.code() == grpc.StatusCode.ALREADY_EXISTS:
pass # fine, provider exists from a previous run
else:
raise
finally:
with contextlib.suppress(grpc.RpcError):
stub.DeleteProvider(
openshell_pb2.DeleteProviderRequest(name="e2e-oidc-admin-test"),
metadata=metadata,
)
def test_user_cannot_create_provider(self) -> None:
token = _get_token("user@test", "user", scopes="openid openshell:all")
stub, metadata = _stub_with_token(token)
req = openshell_pb2.CreateProviderRequest(
provider=datamodel_pb2.Provider(
name="e2e-oidc-user-blocked",
type="claude",
credentials={"API_KEY": "test-value"},
)
)
with pytest.raises(grpc.RpcError) as exc_info:
stub.CreateProvider(req, metadata=metadata)
assert exc_info.value.code() == grpc.StatusCode.PERMISSION_DENIED
assert "openshell-admin" in exc_info.value.details()
def test_user_can_list_sandboxes(self) -> None:
token = _get_token("user@test", "user", scopes="openid openshell:all")
stub, metadata = _stub_with_token(token)
stub.ListSandboxes(openshell_pb2.ListSandboxesRequest(), metadata=metadata)
def test_unauthenticated_request_rejected(self) -> None:
channel = _grpc_channel()
stub = openshell_pb2_grpc.OpenShellStub(channel)
with pytest.raises(grpc.RpcError) as exc_info:
stub.ListSandboxes(openshell_pb2.ListSandboxesRequest())
assert exc_info.value.code() == grpc.StatusCode.UNAUTHENTICATED
def test_health_does_not_require_auth(self) -> None:
channel = _grpc_channel()
stub = openshell_pb2_grpc.OpenShellStub(channel)
resp = stub.Health(openshell_pb2.HealthRequest())
assert resp.status == openshell_pb2.SERVICE_STATUS_HEALTHY
# ── Scope Enforcement Tests ──────────────────────────────────────────
class TestScopes:
"""Test scope-based fine-grained permissions.
These tests require the server to be started with
OPENSHELL_OIDC_SCOPES_CLAIM=scope.
"""
pytestmark = pytest.mark.skipif(
os.environ.get("OPENSHELL_E2E_OIDC_SCOPES") != "1",
reason="Scope e2e tests disabled (set OPENSHELL_E2E_OIDC_SCOPES=1)",
)
def test_sandbox_scoped_token_can_list_sandboxes(self) -> None:
token = _get_token(
"admin@test", "admin", scopes="openid sandbox:read sandbox:write"
)
stub, metadata = _stub_with_token(token)
stub.ListSandboxes(openshell_pb2.ListSandboxesRequest(), metadata=metadata)
def test_sandbox_scoped_token_cannot_list_providers(self) -> None:
token = _get_token(
"admin@test", "admin", scopes="openid sandbox:read sandbox:write"
)
stub, metadata = _stub_with_token(token)
with pytest.raises(grpc.RpcError) as exc_info:
stub.ListProviders(openshell_pb2.ListProvidersRequest(), metadata=metadata)
assert exc_info.value.code() == grpc.StatusCode.PERMISSION_DENIED
assert "provider:read" in exc_info.value.details()
def test_openshell_all_grants_full_access(self) -> None:
token = _get_token("admin@test", "admin", scopes="openid openshell:all")
stub, metadata = _stub_with_token(token)
stub.ListSandboxes(openshell_pb2.ListSandboxesRequest(), metadata=metadata)
stub.ListProviders(openshell_pb2.ListProvidersRequest(), metadata=metadata)
def test_no_openshell_scopes_denied(self) -> None:
token = _get_token("admin@test", "admin")
stub, metadata = _stub_with_token(token)
with pytest.raises(grpc.RpcError) as exc_info:
stub.ListSandboxes(openshell_pb2.ListSandboxesRequest(), metadata=metadata)
assert exc_info.value.code() == grpc.StatusCode.PERMISSION_DENIED
# ── Client Credentials Tests ─────────────────────────────────────────
class TestClientCredentials:
"""Test CI/automation client credentials flow."""
def test_ci_token_can_list_sandboxes(self) -> None:
token = _get_ci_token()
stub, metadata = _stub_with_token(token)
stub.ListSandboxes(openshell_pb2.ListSandboxesRequest(), metadata=metadata)