fix(helm): scope OIDC CA trust to the OIDC client

Signed-off-by: Gordon Sim <gsim@redhat.com>

.agents/skills/helm-dev-environment/SKILL.md

@@ -339,7 +339,9 @@ development TLS certificate, and publishes its trust anchor as the
 `openshell-keycloak-ca` ConfigMap in the OpenShell namespace. The command prints a
 port-forward command for acquiring tokens from the CLI. Rerunning setup rotates the
 development certificate and trust anchor; redeploy the gateway afterward so it reloads
-the mounted CA bundle.
+the mounted CA bundle. The chart renders the mount path as
+`[openshell.gateway.oidc] ca_bundle`; the gateway adds that issuer CA to native roots
+for OIDC discovery and JWKS requests without changing trust for other HTTPS clients.
 
 Then activate OIDC in the OpenShell Helm chart:
 1. Uncomment `#- ci/values-keycloak.yaml` in `skaffold.yaml`

architecture/gateway.md

@@ -291,7 +291,7 @@ Supported auth modes:
 | Plaintext | Local development or a trusted reverse proxy boundary. |
 | Unauthenticated local users | Trusted Kubernetes dev or fully trusted proxy deployments only. |
 | Cloudflare JWT | Edge-authenticated deployments where Cloudflare Access supplies identity. |
-| OIDC | Bearer-token auth for users, with browser or device-code PKCE and client credentials login. Discovery and JWKS retrieval require HTTPS, reject redirects, and pin JWKS to the issuer origin or an explicit origin allowlist. JWKS validation accepts RS256, RS384, RS512, PS256, PS384, PS512, ES256, ES384, and EdDSA (Ed25519) signing keys. |
+| OIDC | Bearer-token auth for users, with browser or device-code PKCE and client credentials login. Discovery and JWKS retrieval require HTTPS, reject redirects, and pin JWKS to the issuer origin or an explicit origin allowlist. An optional private issuer CA augments native roots for the OIDC client only. JWKS validation accepts RS256, RS384, RS512, PS256, PS384, PS512, ES256, ES384, and EdDSA (Ed25519) signing keys. |
 
 The CLI persists the scopes requested during OIDC login in gateway metadata and
 reuses them when refreshing an access token. This preserves the intended API

crates/openshell-core/src/config.rs

@@ -334,6 +334,12 @@ pub struct OidcConfig {
     /// OIDC issuer URL (e.g., `https://idp.example.com/realms/openshell`).
     pub issuer: String,
 
+    /// Optional PEM CA bundle for an issuer signed by a private CA. These
+    /// certificates augment the platform trust roots for OIDC discovery and
+    /// JWKS requests only.
+    #[serde(default)]
+    pub ca_bundle: Option<PathBuf>,
+
     /// Permit cleartext OIDC metadata and JWKS requests to numeric loopback
     /// addresses. This is a development-only escape hatch and never permits
     /// cleartext requests to hostnames or non-loopback addresses.

crates/openshell-server/src/auth/oidc.rs

@@ -576,9 +576,36 @@ impl JwksCache {
     /// initial key set.
     pub async fn new(config: &OidcConfig) -> Result<Self, String> {
         let _ = rustls::crypto::aws_lc_rs::default_provider().install_default();
-        let http = Client::builder()
+        let mut http_builder = Client::builder()
+            // A configured issuer CA augments these roots; it must not replace
+            // trust for unrelated public HTTPS endpoints used by the gateway.
+            .tls_built_in_root_certs(true)
             .timeout(Duration::from_secs(10))
-            .redirect(reqwest::redirect::Policy::none())
+            .redirect(reqwest::redirect::Policy::none());
+        if let Some(ca_bundle) = config.ca_bundle.as_deref() {
+            let pem = std::fs::read(ca_bundle).map_err(|error| {
+                format!(
+                    "failed to read OIDC CA bundle '{}': {error}",
+                    ca_bundle.display()
+                )
+            })?;
+            let certificates = reqwest::Certificate::from_pem_bundle(&pem).map_err(|error| {
+                format!(
+                    "failed to parse OIDC CA bundle '{}': {error}",
+                    ca_bundle.display()
+                )
+            })?;
+            if certificates.is_empty() {
+                return Err(format!(
+                    "OIDC CA bundle '{}' contains no certificates",
+                    ca_bundle.display()
+                ));
+            }
+            for certificate in certificates {
+                http_builder = http_builder.add_root_certificate(certificate);
+            }
+        }
+        let http = http_builder
             .build()
             .map_err(|e| format!("failed to create HTTP client: {e}"))?;
 
@@ -911,6 +938,7 @@ mod tests {
     fn transport_test_config(issuer: impl Into<String>) -> OidcConfig {
         OidcConfig {
             issuer: issuer.into(),
+            ca_bundle: None,
             dangerously_allow_insecure_http: false,
             jwks_allowed_origins: Vec::new(),
             audience: "test-audience".to_string(),
@@ -934,6 +962,22 @@ mod tests {
         );
     }
 
+    #[tokio::test]
+    async fn oidc_rejects_an_empty_ca_bundle() {
+        let ca_bundle = tempfile::NamedTempFile::new().unwrap();
+        let mut config = transport_test_config("https://issuer.example.com");
+        config.ca_bundle = Some(ca_bundle.path().to_path_buf());
+
+        let error = JwksCache::new(&config)
+            .await
+            .expect_err("an empty CA bundle must fail before discovery");
+
+        assert!(
+            error.contains("contains no certificates"),
+            "unexpected error: {error}"
+        );
+    }
+
     #[tokio::test]
     async fn oidc_rejects_non_loopback_http_even_when_acknowledged() {
         let mut config = transport_test_config("http://192.0.2.1/issuer");
@@ -1099,13 +1143,11 @@ mod tests {
 
         // Initialization succeeds after the client receives the rotated trust
         // anchor, and both discovery and JWKS stay on the authenticated channel.
-        let client = Client::builder()
-            .add_root_certificate(reqwest::Certificate::from_pem(ca_cert.pem().as_bytes()).unwrap())
-            .redirect(reqwest::redirect::Policy::none())
-            .build()
-            .unwrap();
-        let config = transport_test_config(issuer.clone());
-        let cache = JwksCache::new_with_client(&config, client)
+        let mut ca_bundle = tempfile::NamedTempFile::new().unwrap();
+        std::io::Write::write_all(&mut ca_bundle, ca_cert.pem().as_bytes()).unwrap();
+        let mut config = transport_test_config(issuer.clone());
+        config.ca_bundle = Some(ca_bundle.path().to_path_buf());
+        let cache = JwksCache::new(&config)
             .await
             .expect("TLS discovery and JWKS should accept the rotated CA");
 
@@ -1424,6 +1466,7 @@ mod tests {
 
         JwksCache::new(&OidcConfig {
             issuer,
+            ca_bundle: None,
             dangerously_allow_insecure_http: true,
             jwks_allowed_origins: Vec::new(),
             audience: TEST_AUDIENCE.to_owned(),
@@ -1763,6 +1806,7 @@ mod tests {
         fn test_oidc_config(issuer: &str) -> OidcConfig {
             OidcConfig {
                 issuer: issuer.to_string(),
+                ca_bundle: None,
                 dangerously_allow_insecure_http: true,
                 jwks_allowed_origins: Vec::new(),
                 audience: "test-audience".to_string(),

crates/openshell-server/src/cli.rs

@@ -162,6 +162,11 @@ struct RunArgs {
     #[arg(long, env = "OPENSHELL_OIDC_ISSUER")]
     oidc_issuer: Option<String>,
 
+    /// Path to a PEM CA bundle for an OIDC issuer signed by a private CA.
+    /// The certificates augment platform trust roots for OIDC requests only.
+    #[arg(long, env = "OPENSHELL_OIDC_CA_BUNDLE")]
+    oidc_ca_bundle: Option<PathBuf>,
+
     /// Development only: permit OIDC metadata and JWKS over HTTP when the
     /// endpoint uses a numeric loopback address.
     #[arg(
@@ -553,6 +558,7 @@ fn prepare_server_config_with_drivers(
     if let Some(issuer) = args.oidc_issuer.clone() {
         config = config.with_oidc(openshell_core::OidcConfig {
             issuer,
+            ca_bundle: args.oidc_ca_bundle.clone(),
             dangerously_allow_insecure_http: args.oidc_dangerously_allow_insecure_http,
             jwks_allowed_origins: args.oidc_jwks_allowed_origins.clone(),
             audience: args.oidc_audience.clone(),
@@ -1116,6 +1122,9 @@ fn merge_file_into_args(args: &mut RunArgs, file: &GatewayFileSection, matches:
         if args.oidc_issuer.is_none() && arg_defaulted(matches, "oidc_issuer") {
             args.oidc_issuer = Some(oidc.issuer.clone());
         }
+        if args.oidc_ca_bundle.is_none() && arg_defaulted(matches, "oidc_ca_bundle") {
+            args.oidc_ca_bundle.clone_from(&oidc.ca_bundle);
+        }
         if arg_defaulted(matches, "oidc_dangerously_allow_insecure_http") {
             args.oidc_dangerously_allow_insecure_http = oidc.dangerously_allow_insecure_http;
         }
@@ -2877,6 +2886,7 @@ compute_driver = "podman"
         let _g2 = EnvVarGuard::remove("OPENSHELL_OIDC_AUDIENCE");
         let _g3 = EnvVarGuard::remove("OPENSHELL_OIDC_DANGEROUSLY_ALLOW_INSECURE_HTTP");
         let _g4 = EnvVarGuard::remove("OPENSHELL_OIDC_JWKS_ALLOWED_ORIGINS");
+        let _g5 = EnvVarGuard::remove("OPENSHELL_OIDC_CA_BUNDLE");
 
         let (mut args, matches) =
             parse_with_args(&["openshell-gateway", "--db-url", "sqlite::memory:"]);
@@ -2884,6 +2894,7 @@ compute_driver = "podman"
             r#"
 [openshell.gateway.oidc]
 issuer = "https://idp.example.com"
+ca_bundle = "/etc/openshell/oidc-ca.pem"
 audience = "openshell-cli"
 dangerously_allow_insecure_http = true
 jwks_allowed_origins = ["https://keys.example.com"]
@@ -2892,6 +2903,10 @@ jwks_allowed_origins = ["https://keys.example.com"]
         merge_file_into_args(&mut args, &file.openshell.gateway, &matches);
 
         assert_eq!(args.oidc_issuer.as_deref(), Some("https://idp.example.com"));
+        assert_eq!(
+            args.oidc_ca_bundle.as_deref(),
+            Some(std::path::Path::new("/etc/openshell/oidc-ca.pem"))
+        );
         assert_eq!(args.oidc_audience, "openshell-cli");
         assert!(args.oidc_dangerously_allow_insecure_http);
         assert_eq!(

crates/openshell-server/src/grpc/mutation_replay/tests.rs

@@ -937,6 +937,7 @@ async fn unrelated_oidc_configuration_does_not_reset_mtls_admission_identity() {
     state.store.put_message(&template).await.unwrap();
     Arc::get_mut(&mut state).unwrap().config.oidc = Some(openshell_core::OidcConfig {
         issuer: "https://new.example.com".into(),
+        ca_bundle: None,
         dangerously_allow_insecure_http: false,
         jwks_allowed_origins: Vec::new(),
         audience: "openshell-cli".into(),

deploy/helm/openshell/README.md

@@ -315,7 +315,7 @@ discovery endpoint or its TLS CA.
 | server.name | string | `""` | Operator-facing gateway name. Defaults to the chart fullname so all replicas in one installation share an identity. Set explicitly when one telemetry collector receives spans from multiple namespaces or clusters. |
 | server.oidc.adminRole | string | `""` | Role name for admin access. Leave empty (with userRole also empty) for authentication-only mode. Both must be set or both empty. |
 | server.oidc.audience | string | `"openshell-cli"` | Expected audience claim for the API resource server. This should match the server's --oidc-audience, NOT the CLI client ID. |
-| server.oidc.caConfigMapName | string | `""` | Name of a ConfigMap containing a CA certificate bundle (key: ca.crt) for verifying the OIDC issuer's TLS certificate. Required when the issuer uses a non-public CA (e.g. OpenShift ingress, private PKI). |
+| server.oidc.caConfigMapName | string | `""` | Name of a ConfigMap containing a CA certificate bundle (key: ca.crt) for verifying the OIDC issuer's TLS certificate. These certificates augment platform trust roots for OIDC requests only. Required when the issuer uses a non-public CA (e.g. OpenShift ingress, private PKI). |
 | server.oidc.dangerouslyAllowInsecureHttp | bool | `false` | Development only: permit cleartext OIDC requests to numeric loopback addresses. This never permits HTTP to hostnames or non-loopback addresses. |
 | server.oidc.issuer | string | `""` | OIDC issuer URL (e.g. https://keycloak.example.com/realms/openshell). |
 | server.oidc.jwksAllowedOrigins | list | `[]` | Additional trusted HTTPS origins allowed to serve JWKS. The issuer origin is always allowed. Entries must not include a path or query. |

deploy/helm/openshell/templates/_gateway-workload.tpl

@@ -112,16 +112,7 @@ spec:
         {{- end }}
         # Most gateway settings live in the ConfigMap-backed TOML file
         # mounted at /etc/openshell/gateway.toml. Secret-bearing settings use
-        # env vars that the TOML references by name. Some process-level
-        # settings consumed by libraries outside gateway code also remain here.
-        {{- if and .Values.server.oidc.issuer .Values.server.oidc.caConfigMapName }}
-        # OIDC issuer custom-CA: rustls/reqwest read SSL_CERT_FILE for
-        # outbound TLS verification. This is a process-level env var
-        # consumed by the TLS stack itself, not by gateway code, so it
-        # cannot be represented in the gateway TOML schema.
-        - name: SSL_CERT_FILE
-          value: /etc/openshell-tls/oidc-ca/ca.crt
-        {{- end }}
+        # env vars that the TOML references by name.
         - name: OPENSHELL_TELEMETRY_ENABLED
           value: {{ .Values.server.telemetryEnabled | quote }}
         {{- if .Values.server.providerTokenGrants.spiffe.enabled }}

deploy/helm/openshell/templates/gateway-config.yaml

@@ -119,6 +119,9 @@ data:
 
     [openshell.gateway.oidc]
     issuer                           = {{ .Values.server.oidc.issuer | quote }}
+    {{- if .Values.server.oidc.caConfigMapName }}
+    ca_bundle                        = "/etc/openshell-tls/oidc-ca/ca.crt"
+    {{- end }}
     dangerously_allow_insecure_http = {{ .Values.server.oidc.dangerouslyAllowInsecureHttp }}
     jwks_allowed_origins             = {{ .Values.server.oidc.jwksAllowedOrigins | toJson }}
     audience                         = {{ .Values.server.oidc.audience | quote }}

deploy/helm/openshell/tests/gateway_config_test.yaml

@@ -278,6 +278,25 @@ tests:
           path: data["gateway.toml"]
           pattern: '(?m)^jwks_allowed_origins\s*=\s*\["https://keys.example.com"\]$'
 
+  - it: scopes the OIDC CA bundle to the OIDC client
+    template: templates/gateway-config.yaml
+    set:
+      server.oidc.issuer: https://issuer.example.com
+      server.oidc.caConfigMapName: openshell-oidc-ca
+    asserts:
+      - matchRegex:
+          path: data["gateway.toml"]
+          pattern: '(?ms)\[openshell\.gateway\.oidc\].*?ca_bundle\s*=\s*"/etc/openshell-tls/oidc-ca/ca\.crt"'
+
+  - it: omits the OIDC CA bundle path when no ConfigMap is configured
+    template: templates/gateway-config.yaml
+    set:
+      server.oidc.issuer: https://issuer.example.com
+    asserts:
+      - notMatchRegex:
+          path: data["gateway.toml"]
+          pattern: '(?m)^ca_bundle\s*='
+
   - it: treats a null OTLP map as disabled
     template: templates/gateway-config.yaml
     set:
@@ -306,6 +325,11 @@ tests:
       - equal:
           path: spec.template.spec.volumes[3].configMap.name
           value: openshell-oidc-ca
+      - notContains:
+          path: spec.template.spec.containers[0].env
+          content:
+            name: SSL_CERT_FILE
+          any: true
 
   # Regression for the P1 bug Drew flagged: grpc_endpoint MUST live in the
   # Kubernetes driver table, not in [openshell.gateway]. The gateway-side

deploy/helm/openshell/values.yaml

@@ -479,7 +479,8 @@ server:
     # -- Dot-separated path to the scopes array in the JWT claims.
     scopesClaim: ""
     # -- Name of a ConfigMap containing a CA certificate bundle (key: ca.crt)
-    # for verifying the OIDC issuer's TLS certificate. Required when the
+    # for verifying the OIDC issuer's TLS certificate. These certificates
+    # augment platform trust roots for OIDC requests only. Required when the
     # issuer uses a non-public CA (e.g. OpenShift ingress, private PKI).
     caConfigMapName: ""
 

docs/how-it-works/gateways/configuration.mdx

@@ -216,6 +216,7 @@ service_name = "openshell-gateway"
 
 [openshell.gateway.oidc]
 issuer                           = "https://idp.example.com/realms/openshell"
+ca_bundle                        = "/etc/openshell/oidc-ca.pem" # Optional private issuer CA.
 audience                         = "openshell-cli"
 jwks_ttl_secs                    = 3600 # Must be greater than zero.
 jwks_allowed_origins             = ["https://keys.example.com"] # Default: issuer origin only.
@@ -266,6 +267,8 @@ Local Docker, Podman, and VM gateways can also set `[openshell.gateway.mtls_auth
 
 The client-certificate handshake policy is derived and has no `require_client_auth` TOML field. This preserves bearer-only OIDC clients and prevents a file setting from silently weakening CA-only gateways.
 
+`[openshell.gateway.oidc] ca_bundle` points to a certificate-only PEM bundle for an issuer signed by a private CA. The bundle augments platform trust roots for OIDC discovery and JWKS requests only; it does not replace native CA discovery or change trust for provider refresh, token exchange, telemetry, Vault, or other gateway HTTPS clients. Set the same value with `--oidc-ca-bundle` or `OPENSHELL_OIDC_CA_BUNDLE`. For Helm deployments, `server.oidc.caConfigMapName` mounts the ConfigMap's `ca.crt` key and renders this path automatically.
+
 `[openshell.gateway.tls]` supports optional SNI-based dual-certificate mode for deployments that need separate internal and external server certificates. Set `external_cert_path` and `external_key_path` to point at the external (e.g. ACME/publicly-trusted) certificate and key. List the hostnames that should be served with the external certificate in `external_server_names`. Connections whose TLS SNI hostname matches one of those names receive the external certificate; all other connections (including those with no SNI) receive the primary internal certificate from `cert_path`/`key_path`. Both fields must be set together — providing only one is a configuration error. On Kubernetes with the Helm chart, the external certificate is managed automatically when `certManager.serverIssuerRef.name` is set; the chart populates these fields from the cert-manager-issued external server certificate.
 
 `[openshell.gateway] policy_validation_failure_mode` controls what sandbox supervisors do when a complete candidate policy fails runtime validation. The default, `fail_closed`, deactivates the previous network policy, closes relays pinned to it, and denies new egress until a valid generation loads. `retain_last_valid` leaves the previous valid generation active. Both modes reject the candidate atomically; startup keeps the workload unstarted until the effective policy and matching provider configuration pass admission. A rejected startup exposes `ConfigurationInvalid` and remains available for policy/provider repair in either mode. Gateway mutation paths that can preflight a known effective scope reject invalid candidates before persistence and leave the active policy unchanged regardless of this setting. Changing the value requires restarting the gateway so it can reload `gateway.toml` and distribute the new posture to sandbox supervisors.

docs/kubernetes/access-control.mdx

@@ -54,7 +54,7 @@ The `audience` value must match the client ID configured in your identity provid
 | `server.oidc.issuer` | `""` | OIDC issuer URL. Empty disables OIDC. |
 | `server.oidc.dangerouslyAllowInsecureHttp` | `false` | Development-only acknowledgement for numeric-loopback HTTP. It does not allow cluster-service or remote HTTP issuers. |
 | `server.oidc.jwksAllowedOrigins` | `[]` | Additional trusted HTTPS origins allowed to serve JWKS. |
-| `server.oidc.caConfigMapName` | `""` | ConfigMap containing the private issuer CA in `ca.crt`. |
+| `server.oidc.caConfigMapName` | `""` | ConfigMap containing the private issuer CA in `ca.crt`; it augments platform roots for OIDC requests only. |
 | `server.oidc.audience` | `openshell-cli` | Expected `aud` claim in the JWT. |
 | `server.oidc.jwksTtl` | `3600` | JWKS key cache TTL in seconds. Must be greater than zero. |
 | `server.oidc.rolesClaim` | `""` | Dot-separated path to the roles array in JWT claims. |
@@ -65,7 +65,9 @@ The `audience` value must match the client ID configured in your identity provid
 The issuer must use HTTPS. The gateway rejects discovery and JWKS redirects,
 limits response sizes, requires a JSON media type, and rejects a `jwks_uri` on
 a different origin unless that origin appears in `jwksAllowedOrigins`. Use
-`server.oidc.caConfigMapName` for issuers signed by a private CA.
+`server.oidc.caConfigMapName` for issuers signed by a private CA. The mounted
+CA does not replace the gateway's platform roots, so unrelated HTTPS clients
+such as provider token refresh continue to trust public services.
 
 ### Auth-only mode vs. RBAC mode
 

skills/debug-openshell-cluster/SKILL.md

@@ -915,7 +915,7 @@ credential failures.
 | Vault credential driver returns HTTP 403 / `Vault Kubernetes auth denied the configured role` on provider create | Vault's `auth/kubernetes` method or the gateway login role is not provisioned, or the role is not bound to the gateway service account and namespace | In Vault: `bao auth enable kubernetes` and `bao write auth/kubernetes/config kubernetes_host=... kubernetes_ca_cert=@...`; ensure the login role's `bound_service_account_names`/`bound_service_account_namespaces` match the gateway SA and namespace and its policy grants the credential paths |
 | CLI TLS error | Local mTLS bundle does not match server cert/CA | Check `~/.config/openshell/gateways/<name>/mtls/` |
 | Edge or OIDC gateway returns `Unauthenticated` | Stored login expired, audience/scopes mismatch, or gateway auth configuration changed | `openshell gateway info`, `openshell gateway login <name>`, gateway auth logs |
-| Gateway exits during OIDC initialization | Issuer is not HTTPS, discovery redirected, metadata used a non-JSON media type or exceeded its size limit, or `jwks_uri` uses an untrusted origin | Use an HTTPS issuer; mount a private CA with `server.oidc.caConfigMapName`; keep JWKS on the issuer origin or explicitly add its HTTPS origin to `server.oidc.jwksAllowedOrigins`. Numeric-loopback HTTP is development-only and also requires `server.oidc.dangerouslyAllowInsecureHttp=true` |
+| Gateway exits during OIDC initialization | Issuer is not HTTPS, discovery redirected, metadata used a non-JSON media type or exceeded its size limit, the configured `ca_bundle` is missing or invalid, or `jwks_uri` uses an untrusted origin | Use an HTTPS issuer; mount a private CA with `server.oidc.caConfigMapName` and confirm `[openshell.gateway.oidc] ca_bundle` names the mounted `ca.crt`; keep JWKS on the issuer origin or explicitly add its HTTPS origin to `server.oidc.jwksAllowedOrigins`. The issuer CA augments platform roots for OIDC only. Numeric-loopback HTTP is development-only and also requires `server.oidc.dangerouslyAllowInsecureHttp=true` |
 | Gateway fails before serving health after enabling an interceptor | Interceptor endpoint unavailable or manifest/binding validation failed | Gateway and interceptor logs; interceptor socket; `binding_policy`, phases, and failure policy |
 | Authenticated interceptor or middleware rejects gateway calls | Private CA or hostname mismatch, expected audience or issuer mismatch, stale/unknown `kid`, or malformed extension token | `tls_ca_cert_path`, registration `audience`, service verifier config and logs; fetch well-known metadata only through the already-trusted gateway TLS endpoint |
 | Provider profiles disappear after enabling an interceptor catalog | `provider_profile_sources` selected only an authoritative interceptor or returned invalid/duplicate IDs | Inspect source list and interceptor `Describe`/catalog logs; include `user` when composition with imported profiles is intended |