trending/OpenShell
1
0
Fork 0
mirror of https://github.com/NVIDIA/OpenShell.git synced 2026-10-02 07:34:45 +08:00
Code Issues Packages Projects Releases Wiki Activity

feat(kubernetes): support corporate proxy CA bundle (#3447)

Browse Source 
* feat(kubernetes): support corporate proxy CA bundle

The Kubernetes driver had no way to supply a CA bundle for the corporate
egress proxy, so an `https://` proxy with a private CA, or a TLS-intercepting
proxy, could not be used. Podman and VM already expose `proxy_ca_bundle`.

Add `proxy_ca_bundle` to `[openshell.drivers.kubernetes]` as a path the
gateway Pod reads. The gateway stages the PEM into the existing
per-generation supervisor bootstrap Secret and passes
`--upstream-proxy-ca-bundle` on the supervisor argv. That Secret is already
immutable, owner-referenced and garbage-collected, and its volume mounts
every key at /.openshell/supervisor with no items filter, so this needs no
new object kind, volume, mount, or RBAC verb, and works in shared, managed
and operator workspace modes.

The bundle is deliberately read from the gateway's filesystem rather than
referenced as an object in the sandbox namespace. It becomes a trust anchor
for every upstream the sandbox reaches, so it must stay in the gateway's
trust domain; the immutable staging Secret also keeps the anchor from
changing underneath a running sandbox.

Bound the staged bundle at 256 KiB. The shared reader's limit is exactly the
apiserver's own Secret limit and the bootstrap Secret carries four other
keys, so a bundle between the two would pass gateway startup and then fail
every sandbox create with an opaque `data: Too long`.

Delegate the URL, no_proxy, connect_by_hostname and ca_bundle rules to the
shared validate_upstream_proxy_settings, keeping the Secret-specific
credential block local: this driver accepts an explicit
`proxy_auth_allow_insecure = false` without credentials, which the shared
rules reject. This also fixes the acknowledgement being demanded for an
`https://` proxy, where the credential travels inside the verified TLS
session. Add auth_setting_label so the inline-credential diagnostic names
the Secret keys instead of proxy_auth_file, which this driver rejects as an
unknown key.

Document that the bundle should carry only the CA that signs the proxy's
certificate, or that an intercepting proxy re-signs upstream certificates
with. Public roots already reach the sandbox through the supervisor image and
its TLS stack, and the bundle is concatenated with that system store into a
single boundary control frame, so a full merged trust bundle spends the frame
budget on duplicated roots. The frame, not the apiserver Secret limit, is the
tighter of the two ceilings in practice; raising the staging bound requires
checking it.

Closes #3443

Signed-off-by: Philippe Martin <phmartin@redhat.com>

* fix(helm): quote proxy CA ConfigMap references

Signed-off-by: Philippe Martin <phmartin@redhat.com>

---------

Signed-off-by: Philippe Martin <phmartin@redhat.com>
This commit is contained in:
Philippe Martin
2026-09-22 17:57:40 +00:00
committed by GitHub
parent 3107ff1f82
commit 35e0a68e4a
18 changed files with 890 additions and 69 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/reference/gateway-config.mdx
+20 -6
View File
@@ -583,10 +583,11 @@ sandbox_runtime_image_pull_policy = "if_not_present"
# supervisor_image = "ghcr.io/nvidia/openshell/supervisor:<version>"
supervisor_image_pull_policy = "if_not_present"
# Optional corporate HTTP forward proxy for policy-approved TLS egress. The
# sandbox workload cannot select or override these settings. Only http:// proxy
# endpoints and TLS CONNECT traffic are supported; plain HTTP egress remains
# direct. `no_proxy` bypasses only the corporate proxy, never OpenShell policy.
# Optional corporate forward proxy for policy-approved TLS egress. The sandbox
# workload cannot select or override these settings. http:// and https:// proxy
# endpoints are supported, for TLS CONNECT traffic only; plain HTTP egress
# remains direct. `no_proxy` bypasses only the corporate proxy, never OpenShell
# policy.
# https_proxy = "http://proxy.corp.example:8080"
# no_proxy = ".svc,.svc.cluster.local,10.96.0.0/12,10.244.0.0/16"
# Proxy credentials must be an existing Secret in the sandbox namespace. The
@@ -601,12 +602,25 @@ supervisor_image_pull_policy = "if_not_present"
# content is validated fail-closed by the supervisor at startup and never
# falls back to direct egress.
# Proxy credentials mount only in the separately scheduled supervisor Pod.
# Required with a credential Secret: Basic authentication to an http:// proxy
# is cleartext on the connection to that proxy.
# Required with a credential Secret when the proxy URL is http://, because
# Basic authentication is then cleartext on the connection to that proxy. An
# https:// proxy carries the credential inside the verified TLS session and
# needs no acknowledgement.
# proxy_auth_allow_insecure = true
# Last resort for hostname-filtering proxy ACLs. The proxy resolves the target,
# so its ACL becomes part of the egress boundary for proxied connections.
# proxy_connect_by_hostname = true
# CA bundle trusted for the corporate proxy, as a path on the gateway Pod's
# filesystem. Needed for an https:// proxy whose certificate is not publicly
# trusted, and for a TLS-intercepting proxy that re-signs upstream
# certificates. The gateway reads the file and stages it into each sandbox's
# immutable supervisor bootstrap Secret, so the anchor stays in the gateway's
# trust domain and cannot change underneath a running sandbox. Helm mounts it
# from `upstreamProxy.caBundle.configMapName` and renders this path. Supply
# only the proxy's own CA: public roots already come from the supervisor image
# and its TLS stack, so a full merged trust bundle wastes the sandbox
# boundary's control-frame budget on duplicated roots.
# proxy_ca_bundle = "/etc/openshell-tls/proxy-ca/ca.crt"
# Required in raw gateway TOML because `namespace` identifies sandbox
# placement, not the gateway Service. Helm renders this from the release's
# gateway Service name and namespace.
docs/reference/sandbox-compute-drivers.mdx
+23 -2
View File
@@ -437,16 +437,37 @@ For maintainer-level implementation details, refer to the [Kubernetes driver REA
| `supervisor_image_pull_policy` | `supervisor.image.pullPolicy` | Set the Kubernetes image pull policy for the supervisor image. |
| `sandbox_runtime.network_policy_enforced` | `supervisor.sandboxRuntime.networkPolicyEnforced` | Acknowledge that the cluster CNI enforces ingress and egress `NetworkPolicy` in sandbox namespaces. This must be `true`. |
| `sandbox_runtime.boundary_port` | `supervisor.sandboxRuntime.boundaryPort` | Set the non-privileged TLS port used between the paired supervisor and sandbox Pods. |
| `https_proxy` | `upstreamProxy.url` | Set the operator-owned `http://host:port` corporate forward proxy used for policy-approved TLS CONNECT egress. |
| `https_proxy` | `upstreamProxy.url` | Set the operator-owned `http://host:port` or `https://host:port` corporate forward proxy used for policy-approved TLS CONNECT egress. |
| `no_proxy` | `upstreamProxy.noProxy` | Set destinations that bypass only the corporate proxy. OpenShell policy evaluation still applies. |
| `proxy_auth_secret_name` | `upstreamProxy.authSecret.name` | Set the existing Secret name in the sandbox namespace that contains the proxy credential. The Secret mounts only in the supervisor Pod. |
| `proxy_auth_secret_key` | `upstreamProxy.authSecret.key` | Set the Secret key containing the `user:pass` credential. |
| `proxy_auth_allow_insecure` | `upstreamProxy.authAllowInsecure` | Set `true` to acknowledge that Basic authentication to an HTTP proxy is cleartext. Required with a proxy credential Secret. |
| `proxy_auth_allow_insecure` | `upstreamProxy.authAllowInsecure` | Set `true` to acknowledge that Basic authentication to an HTTP proxy is cleartext. Required with a proxy credential Secret and an `http://` proxy; an `https://` proxy carries the credential inside the verified TLS session and needs no acknowledgement. |
| `proxy_connect_by_hostname` | `upstreamProxy.connectByHostname` | Send hostnames rather than validated IPs in CONNECT requests. Use only when proxy ACLs require hostname targets. |
| `proxy_ca_bundle` | `upstreamProxy.caBundle.configMapName` / `upstreamProxy.caBundle.key` | Trust a PEM CA bundle for the corporate proxy. Required for an `https://` proxy with a private CA, and for a TLS-intercepting proxy that re-signs upstream certificates. Helm mounts the ConfigMap into the gateway Pod; the gateway stages the bundle into each sandbox's immutable supervisor bootstrap Secret. |
| `workspace_default_storage_size` | `server.workspaceDefaultStorageSize` | Set the default workspace PVC size for new sandboxes. |
| `workspace_storage_class` | `server.workspaceStorageClass` | Set the `StorageClass` for the workspace PVC. Empty (default) omits `storageClassName` and uses the cluster's default `StorageClass`. Set this on clusters with no default `StorageClass`, otherwise the workspace PVC stays `Pending` and the sandbox never starts. |
| `sa_token_ttl_secs` | `server.sandboxJwt.k8sSaTokenTtlSecs` | Set the projected ServiceAccount token TTL used for the bootstrap token exchange. |
`proxy_ca_bundle` needs only the CA that signs the proxy's certificate, or that
a TLS-intercepting proxy re-signs upstream certificates with. Public roots
already come from the supervisor image and its TLS stack, so supplying a full
merged trust bundle adds hundreds of kilobytes of duplicated roots for no
benefit.
On OpenShift, copy the cluster proxy's trusted-CA ConfigMap into the gateway's
release namespace rather than pointing at an injected trusted-CA bundle. A copy
is required in any case, because ConfigMaps cannot be referenced across
namespaces:
```shell
CORP_CA=$(oc get proxy/cluster -o jsonpath='{.spec.trustedCA.name}')
oc -n openshift-config get cm "$CORP_CA" -o jsonpath='{.data.ca-bundle\.crt}' > corp-ca.pem
oc -n openshell create configmap corporate-proxy-ca --from-file=ca.crt=corp-ca.pem
```
Then set `upstreamProxy.caBundle.configMapName` to `corporate-proxy-ca` and
leave `upstreamProxy.caBundle.key` at its `ca.crt` default.
Managed-mode Secret copying requires the gateway ServiceAccount to create
Secrets. Kubernetes RBAC cannot restrict Secret `create` by resource name, so
the Helm chart grants cluster-wide Secret `create`; Secret `get` and `patch`