mirror of
https://github.com/NVIDIA/OpenShell.git
synced 2026-10-20 06:00:06 +08:00
0a770d9173
* feat(kubernetes): support HA gateway rebalancing Signed-off-by: Drew Newberry <anewberry@nvidia.com> * perf(server): cache peer connections, tokens, and owner lookups Every forwarded relay rebuilt its setup from scratch: an owner lookup, a blocking read of the peer token, a TLS connect to the owning replica, and a TokenReview plus Pod GET on the receiving side. Sandbox service routing does this per HTTP request, so the apiserver calls scaled with traffic. Cache all of it on ServerState: - peer channels pooled per endpoint, so relays multiplex over one connection instead of redialing - peer tokens keyed by SHA-256, expiring at min(ttl, token exp) so a hit cannot accept an expired token - owner records for 3s against a 45s ownership TTL, still freshness checked before use Entries are evicted when a relay fails. Also raise HTTP/2 max_concurrent_streams to 1024, since pooling funnels every relay between two replicas onto one connection and hyper's default of 200 sits below the 256 pending-relay budget. Signed-off-by: divesh <dgude@nvidia.com> * perf(server): pool upstream connections for sandbox services Each HTTP request to a sandbox service opened its own supervisor relay, paying a new TCP connection and HTTP/1 handshake every time. Worse, it counted against the 32 in-flight relay cap, so a service handling more than 32 concurrent requests failed outright. Pool idle upstreams per endpoint and port, up to 8 each for 15s. Reuse is safe because the pool only returns a connection hyper reports as ready, and HTTP/1 cannot start a request until the previous body has drained. Upgrades are never pooled since they take the connection over, and a failed send evicts that endpoint. Pruning is bounded per key, with the full sweep limited to once per 30s. Signed-off-by: divesh <dgude@nvidia.com> * fix(server): address HA gateway review findings (#3449) - Let a gateway own supervisor sessions without a peer endpoint. Requiring one whenever the store is PostgreSQL broke every single-instance PostgreSQL deployment, because no sandbox supervisor could connect. A cross-replica request to an owner that advertises no endpoint now fails immediately naming the cause, instead of retrying until the wait timeout. - Close a supervisor session on heartbeat only when another replica owns it, or after renewals fail for the ownership TTL. A database error no longer drops every session heartbeating during an outage. - Clamp owner record ages at zero so a skewed or corrupt stored timestamp cannot produce a negative age. - Bound the cross-object advisory lock with a lock timeout, so a stuck holder fails instead of blocking every mutation in the fleet. - Refuse to start when a peer endpoint is configured on a multi-replica backend but peer authentication is unavailable, and warn when a multi-replica backend has no peer endpoint at all. - Reject a plaintext peer endpoint when the gateway serves TLS. - Skip the sandbox watch poller on single-replica backends, where the local update bus already sees every write. - Rate-limit the peer owner cache sweep so an insert no longer scans the whole map under the lock. - Retry GET and HEAD on a pooled upstream the sandbox closed, instead of returning 502, and drop an emptied endpoint from the pool right away. - Document the gateway peer environment variables and the post-rollout ownership skew operators should expect. Signed-off-by: divesh <dgude@nvidia.com> * fix(server): harden HA supervisor ownership Signed-off-by: Drew Newberry <anewberry@nvidia.com> --------- Signed-off-by: Drew Newberry <anewberry@nvidia.com> Signed-off-by: divesh <dgude@nvidia.com> Co-authored-by: Drew Newberry <anewberry@nvidia.com> Co-authored-by: divesh <dgude@nvidia.com> Co-authored-by: Divesh Chowdary <47188680+FrostGod@users.noreply.github.com>
207 lines
7.5 KiB
YAML
207 lines
7.5 KiB
YAML
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
|
|
# SPDX-License-Identifier: Apache-2.0
|
|
|
|
suite: credential drivers
|
|
templates:
|
|
- templates/gateway-config.yaml
|
|
- templates/credential-storage-key-encryption-key-secret.yaml
|
|
- templates/statefulset.yaml
|
|
- templates/deployment.yaml
|
|
- templates/credential-secrets-role.yaml
|
|
- templates/credential-secrets-rolebinding.yaml
|
|
release:
|
|
name: openshell
|
|
namespace: my-namespace
|
|
|
|
tests:
|
|
- it: renders default encrypted credential storage by default
|
|
template: templates/gateway-config.yaml
|
|
asserts:
|
|
- notMatchRegex:
|
|
path: data["gateway.toml"]
|
|
pattern: 'credential_drivers\s*='
|
|
- matchRegex:
|
|
path: data["gateway.toml"]
|
|
pattern: '(?ms)\[openshell\.gateway\.credential_storage\].*?key_encryption_key_env\s*=\s*"OPENSHELL_GATEWAY_CREDENTIAL_KEY_ENCRYPTION_KEY"'
|
|
- notMatchRegex:
|
|
path: data["gateway.toml"]
|
|
pattern: 'key_encryption_key_path\s*='
|
|
|
|
- it: creates a retained default credential storage key-encryption key Secret by default
|
|
template: templates/credential-storage-key-encryption-key-secret.yaml
|
|
asserts:
|
|
- equal:
|
|
path: kind
|
|
value: Secret
|
|
- matchRegex:
|
|
path: metadata.name
|
|
pattern: 'credential-storage-key-encryption-key$'
|
|
- equal:
|
|
path: metadata.annotations["helm.sh/resource-policy"]
|
|
value: keep
|
|
- matchRegex:
|
|
path: data["key-encryption-key"]
|
|
pattern: '.+'
|
|
|
|
- it: injects the default credential storage key-encryption key Secret into the gateway pod by default
|
|
template: templates/statefulset.yaml
|
|
asserts:
|
|
- contains:
|
|
path: spec.template.spec.containers[0].env
|
|
content:
|
|
name: OPENSHELL_GATEWAY_CREDENTIAL_KEY_ENCRYPTION_KEY
|
|
valueFrom:
|
|
secretKeyRef:
|
|
name: openshell-credential-storage-key-encryption-key
|
|
key: key-encryption-key
|
|
|
|
- it: renders Kubernetes Secrets credential driver config
|
|
template: templates/gateway-config.yaml
|
|
set:
|
|
server.credentialDrivers.kubernetesSecrets.enabled: true
|
|
server.credentialDrivers.kubernetesSecrets.namespace: provider-secrets
|
|
asserts:
|
|
- matchRegex:
|
|
path: data["gateway.toml"]
|
|
pattern: 'credential_drivers\s*=\s*\["kubernetes-secrets"\]'
|
|
- matchRegex:
|
|
path: data["gateway.toml"]
|
|
pattern: '(?ms)\[openshell\.credential_drivers\.kubernetes-secrets\].*?namespace\s*=\s*"provider-secrets".*?allow_reference_namespace\s*=\s*false'
|
|
- notMatchRegex:
|
|
path: data["gateway.toml"]
|
|
pattern: 'transport\s*=\s*"in_tree"'
|
|
- notMatchRegex:
|
|
path: data["gateway.toml"]
|
|
pattern: '\[openshell\.gateway\.credential_storage\]'
|
|
|
|
- it: renders Vault credential driver config
|
|
template: templates/gateway-config.yaml
|
|
set:
|
|
server.credentialDrivers.vault.enabled: true
|
|
server.credentialDrivers.vault.address: https://vault.vault.svc.cluster.local:8200
|
|
server.credentialDrivers.vault.caConfigMapName: vault-ca
|
|
server.credentialDrivers.vault.role: openshell-gateway
|
|
asserts:
|
|
- matchRegex:
|
|
path: data["gateway.toml"]
|
|
pattern: 'credential_drivers\s*=\s*\["vault"\]'
|
|
- matchRegex:
|
|
path: data["gateway.toml"]
|
|
pattern: '(?ms)\[openshell\.credential_drivers\.vault\].*?address\s*=\s*"https://vault\.vault\.svc\.cluster\.local:8200".*?ca_bundle\s*=\s*"/etc/openshell-tls/vault-ca/ca\.crt".*?auth_method\s*=\s*"kubernetes".*?role\s*=\s*"openshell-gateway"'
|
|
- notMatchRegex:
|
|
path: data["gateway.toml"]
|
|
pattern: 'transport\s*=\s*"in_tree"'
|
|
|
|
- it: rejects multiple enabled credential drivers
|
|
template: templates/statefulset.yaml
|
|
set:
|
|
server.credentialDrivers.kubernetesSecrets.enabled: true
|
|
server.credentialDrivers.vault.enabled: true
|
|
server.credentialDrivers.vault.address: https://vault.vault.svc.cluster.local:8200
|
|
server.credentialDrivers.vault.role: openshell-gateway
|
|
asserts:
|
|
- failedTemplate:
|
|
errorPattern: "only one external server.credentialDrivers backend can be enabled at a time"
|
|
|
|
- it: mounts the Vault CA ConfigMap into the gateway
|
|
template: templates/statefulset.yaml
|
|
set:
|
|
server.credentialDrivers.vault.enabled: true
|
|
server.credentialDrivers.vault.address: https://vault.vault.svc.cluster.local:8200
|
|
server.credentialDrivers.vault.caConfigMapName: vault-ca
|
|
server.credentialDrivers.vault.role: openshell-gateway
|
|
asserts:
|
|
- contains:
|
|
path: spec.template.spec.containers[0].volumeMounts
|
|
content:
|
|
name: vault-ca
|
|
mountPath: /etc/openshell-tls/vault-ca
|
|
readOnly: true
|
|
- contains:
|
|
path: spec.template.spec.volumes
|
|
content:
|
|
name: vault-ca
|
|
configMap:
|
|
name: vault-ca
|
|
items:
|
|
- key: ca.crt
|
|
path: ca.crt
|
|
|
|
- it: creates namespaced Kubernetes Secret manager RBAC
|
|
template: templates/credential-secrets-role.yaml
|
|
set:
|
|
server.credentialDrivers.kubernetesSecrets.enabled: true
|
|
server.credentialDrivers.kubernetesSecrets.namespace: provider-secrets
|
|
asserts:
|
|
- equal:
|
|
path: metadata.namespace
|
|
value: provider-secrets
|
|
- equal:
|
|
path: rules[0].resources[0]
|
|
value: secrets
|
|
- equal:
|
|
path: rules[0].verbs[0]
|
|
value: get
|
|
- contains:
|
|
path: rules[0].verbs
|
|
content: create
|
|
- contains:
|
|
path: rules[0].verbs
|
|
content: patch
|
|
- contains:
|
|
path: rules[0].verbs
|
|
content: delete
|
|
|
|
- it: binds Kubernetes Secret manager RBAC to the gateway ServiceAccount
|
|
template: templates/credential-secrets-rolebinding.yaml
|
|
set:
|
|
server.credentialDrivers.kubernetesSecrets.enabled: true
|
|
server.credentialDrivers.kubernetesSecrets.namespace: provider-secrets
|
|
asserts:
|
|
- equal:
|
|
path: metadata.namespace
|
|
value: provider-secrets
|
|
- equal:
|
|
path: subjects[0].name
|
|
value: openshell
|
|
- equal:
|
|
path: subjects[0].namespace
|
|
value: my-namespace
|
|
|
|
- it: allows default credential storage on a Deployment with an external database
|
|
template: templates/deployment.yaml
|
|
set:
|
|
workload.kind: deployment
|
|
server.externalDbSecret: openshell-pg
|
|
asserts:
|
|
- equal:
|
|
path: kind
|
|
value: Deployment
|
|
- contains:
|
|
path: spec.template.spec.containers[0].env
|
|
content:
|
|
name: OPENSHELL_GATEWAY_CREDENTIAL_KEY_ENCRYPTION_KEY
|
|
valueFrom:
|
|
secretKeyRef:
|
|
name: openshell-credential-storage-key-encryption-key
|
|
key: key-encryption-key
|
|
|
|
- it: allows default credential storage with multiple replicas and an external database
|
|
template: templates/statefulset.yaml
|
|
set:
|
|
replicaCount: 2
|
|
server.externalDbSecret: openshell-pg
|
|
workload.allowMultiReplicaStatefulSet: true
|
|
asserts:
|
|
- equal:
|
|
path: spec.replicas
|
|
value: 2
|
|
- contains:
|
|
path: spec.template.spec.containers[0].env
|
|
content:
|
|
name: OPENSHELL_GATEWAY_CREDENTIAL_KEY_ENCRYPTION_KEY
|
|
valueFrom:
|
|
secretKeyRef:
|
|
name: openshell-credential-storage-key-encryption-key
|
|
key: key-encryption-key
|