mirror of
https://github.com/NVIDIA/OpenShell.git
synced 2026-10-20 06:00:06 +08:00
0a770d9173
* feat(kubernetes): support HA gateway rebalancing Signed-off-by: Drew Newberry <anewberry@nvidia.com> * perf(server): cache peer connections, tokens, and owner lookups Every forwarded relay rebuilt its setup from scratch: an owner lookup, a blocking read of the peer token, a TLS connect to the owning replica, and a TokenReview plus Pod GET on the receiving side. Sandbox service routing does this per HTTP request, so the apiserver calls scaled with traffic. Cache all of it on ServerState: - peer channels pooled per endpoint, so relays multiplex over one connection instead of redialing - peer tokens keyed by SHA-256, expiring at min(ttl, token exp) so a hit cannot accept an expired token - owner records for 3s against a 45s ownership TTL, still freshness checked before use Entries are evicted when a relay fails. Also raise HTTP/2 max_concurrent_streams to 1024, since pooling funnels every relay between two replicas onto one connection and hyper's default of 200 sits below the 256 pending-relay budget. Signed-off-by: divesh <dgude@nvidia.com> * perf(server): pool upstream connections for sandbox services Each HTTP request to a sandbox service opened its own supervisor relay, paying a new TCP connection and HTTP/1 handshake every time. Worse, it counted against the 32 in-flight relay cap, so a service handling more than 32 concurrent requests failed outright. Pool idle upstreams per endpoint and port, up to 8 each for 15s. Reuse is safe because the pool only returns a connection hyper reports as ready, and HTTP/1 cannot start a request until the previous body has drained. Upgrades are never pooled since they take the connection over, and a failed send evicts that endpoint. Pruning is bounded per key, with the full sweep limited to once per 30s. Signed-off-by: divesh <dgude@nvidia.com> * fix(server): address HA gateway review findings (#3449) - Let a gateway own supervisor sessions without a peer endpoint. Requiring one whenever the store is PostgreSQL broke every single-instance PostgreSQL deployment, because no sandbox supervisor could connect. A cross-replica request to an owner that advertises no endpoint now fails immediately naming the cause, instead of retrying until the wait timeout. - Close a supervisor session on heartbeat only when another replica owns it, or after renewals fail for the ownership TTL. A database error no longer drops every session heartbeating during an outage. - Clamp owner record ages at zero so a skewed or corrupt stored timestamp cannot produce a negative age. - Bound the cross-object advisory lock with a lock timeout, so a stuck holder fails instead of blocking every mutation in the fleet. - Refuse to start when a peer endpoint is configured on a multi-replica backend but peer authentication is unavailable, and warn when a multi-replica backend has no peer endpoint at all. - Reject a plaintext peer endpoint when the gateway serves TLS. - Skip the sandbox watch poller on single-replica backends, where the local update bus already sees every write. - Rate-limit the peer owner cache sweep so an insert no longer scans the whole map under the lock. - Retry GET and HEAD on a pooled upstream the sandbox closed, instead of returning 502, and drop an emptied endpoint from the pool right away. - Document the gateway peer environment variables and the post-rollout ownership skew operators should expect. Signed-off-by: divesh <dgude@nvidia.com> * fix(server): harden HA supervisor ownership Signed-off-by: Drew Newberry <anewberry@nvidia.com> --------- Signed-off-by: Drew Newberry <anewberry@nvidia.com> Signed-off-by: divesh <dgude@nvidia.com> Co-authored-by: Drew Newberry <anewberry@nvidia.com> Co-authored-by: divesh <dgude@nvidia.com> Co-authored-by: Divesh Chowdary <47188680+FrostGod@users.noreply.github.com>
142 lines
4.9 KiB
YAML
142 lines
4.9 KiB
YAML
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
|
|
# SPDX-License-Identifier: Apache-2.0
|
|
|
|
suite: statefulset client CA volume
|
|
templates:
|
|
- templates/gateway-config.yaml
|
|
- templates/statefulset.yaml
|
|
release:
|
|
name: openshell
|
|
namespace: my-namespace
|
|
|
|
tests:
|
|
- it: mounts the server TLS secret ca.crt as client CA for built-in PKI
|
|
template: templates/statefulset.yaml
|
|
set:
|
|
pkiInitJob.enabled: true
|
|
certManager.enabled: false
|
|
asserts:
|
|
- equal:
|
|
path: spec.template.spec.volumes[5].name
|
|
value: tls-client-ca
|
|
- equal:
|
|
path: spec.template.spec.volumes[5].secret.secretName
|
|
value: openshell-server-tls
|
|
- equal:
|
|
path: spec.template.spec.volumes[5].secret.items[0].key
|
|
value: ca.crt
|
|
|
|
- it: shares the cert-manager server TLS ca.crt when clientCaFromServerTlsSecret is true
|
|
template: templates/statefulset.yaml
|
|
set:
|
|
certManager.enabled: true
|
|
certManager.clientCaFromServerTlsSecret: true
|
|
asserts:
|
|
- equal:
|
|
path: spec.template.spec.volumes[5].name
|
|
value: tls-client-ca
|
|
- equal:
|
|
path: spec.template.spec.volumes[5].secret.secretName
|
|
value: openshell-server-tls
|
|
- equal:
|
|
path: spec.template.spec.volumes[5].secret.items[0].key
|
|
value: ca.crt
|
|
|
|
# Regression: with cert-manager enabled and pkiInitJob left at its default
|
|
# `true`, the client CA condition must honor certManager precedence and NOT
|
|
# treat pkiInitJob.enabled as built-in TLS. With clientCaFromServerTlsSecret=false
|
|
# the gateway must mount the separate clientCaSecretName, not the server TLS
|
|
# cert secret.
|
|
- it: uses clientCaSecretName under cert-manager even when pkiInitJob stays enabled
|
|
template: templates/statefulset.yaml
|
|
set:
|
|
certManager.enabled: true
|
|
certManager.clientCaFromServerTlsSecret: false
|
|
pkiInitJob.enabled: true
|
|
server.tls.clientCaSecretName: openshell-ca-tls
|
|
asserts:
|
|
- equal:
|
|
path: spec.template.spec.volumes[5].name
|
|
value: tls-client-ca
|
|
- equal:
|
|
path: spec.template.spec.volumes[5].secret.secretName
|
|
value: openshell-ca-tls
|
|
- notExists:
|
|
path: spec.template.spec.volumes[5].secret.items
|
|
|
|
- it: omits client CA volume and mount when mTLS is disabled
|
|
template: templates/statefulset.yaml
|
|
set:
|
|
pkiInitJob.enabled: true
|
|
certManager.enabled: false
|
|
server.tls.enableMtls: false
|
|
asserts:
|
|
- notContains:
|
|
path: spec.template.spec.volumes
|
|
content:
|
|
name: tls-client-ca
|
|
any: true
|
|
- notContains:
|
|
path: spec.template.spec.containers[0].volumeMounts
|
|
content:
|
|
name: tls-client-ca
|
|
mountPath: /etc/openshell-tls/client-ca
|
|
readOnly: true
|
|
|
|
# When cert-manager owns TLS, does not share its CA, and no separate client CA
|
|
# secret is configured, there is no client CA to mount: the volume must not
|
|
# render rather than mounting an empty secret name.
|
|
- it: omits the client CA volume when cert-manager owns TLS and no client CA secret is set
|
|
template: templates/statefulset.yaml
|
|
set:
|
|
certManager.enabled: true
|
|
certManager.clientCaFromServerTlsSecret: false
|
|
pkiInitJob.enabled: true
|
|
server.tls.clientCaSecretName: ""
|
|
asserts:
|
|
- lengthEqual:
|
|
path: spec.template.spec.volumes
|
|
count: 4
|
|
- notContains:
|
|
path: spec.template.spec.containers[0].volumeMounts
|
|
content:
|
|
name: tls-client-ca
|
|
mountPath: /etc/openshell-tls/client-ca
|
|
readOnly: true
|
|
|
|
# Explicit HTTPS-only opt-out must suppress built-in PKI client-CA wiring even
|
|
# when pkiInitJob remains enabled (the default install path from #2095).
|
|
- it: omits the client CA volume for built-in PKI when clientCaSecretName is empty
|
|
template: templates/statefulset.yaml
|
|
set:
|
|
pkiInitJob.enabled: true
|
|
certManager.enabled: false
|
|
server.tls.clientCaSecretName: ""
|
|
asserts:
|
|
- lengthEqual:
|
|
path: spec.template.spec.volumes
|
|
count: 4
|
|
- notContains:
|
|
path: spec.template.spec.containers[0].volumeMounts
|
|
content:
|
|
name: tls-client-ca
|
|
mountPath: /etc/openshell-tls/client-ca
|
|
readOnly: true
|
|
|
|
- it: omits the client CA volume when cert-manager shares server CA and clientCaSecretName is empty
|
|
template: templates/statefulset.yaml
|
|
set:
|
|
certManager.enabled: true
|
|
certManager.clientCaFromServerTlsSecret: true
|
|
server.tls.clientCaSecretName: ""
|
|
asserts:
|
|
- lengthEqual:
|
|
path: spec.template.spec.volumes
|
|
count: 4
|
|
- notContains:
|
|
path: spec.template.spec.containers[0].volumeMounts
|
|
content:
|
|
name: tls-client-ca
|
|
mountPath: /etc/openshell-tls/client-ca
|
|
readOnly: true
|