mirror of
https://github.com/NVIDIA/OpenShell.git
synced 2026-10-30 07:33:59 +08:00
docs(website): sync dev docs from e30c938bb1
This commit is contained in:
github-actions[bot]
@@ -1,8 +1,8 @@
|
||||
snapshots:
|
||||
dev:
|
||||
source-ref: 90161fbd7d87b6c07b88183b801b30b4925d84d7
|
||||
source-sha: 90161fbd7d87b6c07b88183b801b30b4925d84d7
|
||||
version: 0.1.5.dev3
|
||||
source-ref: e30c938bb159322d5018b9c261c001c658f40071
|
||||
source-sha: e30c938bb159322d5018b9c261c001c658f40071
|
||||
version: 0.1.5.dev5
|
||||
latest:
|
||||
source-ref: 5a7728c304b5528e954e9ae900cf21f84e1a12b1
|
||||
source-sha: 5a7728c304b5528e954e9ae900cf21f84e1a12b1
|
||||
|
||||
@@ -163,6 +163,13 @@ metrics_bind_address = "0.0.0.0:9090"
|
||||
|
||||
log_level = "info"
|
||||
|
||||
# Connection ceiling for the gateway's database pool. Omit to use the backend
|
||||
# default: 10 for Postgres, 5 for on-disk SQLite.
|
||||
database_max_connections = 10
|
||||
|
||||
# Connection ceiling for the Postgres mutation lock pool. Omit to use 4.
|
||||
database_lock_max_connections = 4
|
||||
|
||||
# When omitted, the gateway auto-detects Kubernetes, then Podman, then Docker.
|
||||
# VM is never auto-detected and requires an explicit entry here.
|
||||
compute_driver = "kubernetes"
|
||||
@@ -329,6 +336,26 @@ External compute drivers declare this requirement by adding `openshell.compute.l
|
||||
|
||||
`[openshell.gateway.auth] allow_unauthenticated_users = true` is an unsafe local-development and trusted-proxy escape hatch. It accepts user-facing CLI/API calls without OIDC or mTLS credentials while sandbox supervisors still authenticate with gateway-minted sandbox JWTs. Leave it false for shared and production gateways.
|
||||
|
||||
## Database Pool
|
||||
|
||||
`[openshell.gateway] database_max_connections` sets the connection ceiling for the gateway's database pool. One pool is shared by every database-backed RPC, so this value is also the gateway's ceiling on concurrent database work. When the pool is saturated, callers queue on connection acquisition and sqlx logs that the time to acquire a connection exceeded its slow threshold. The background readiness monitor draws from the same pool, so sustained saturation can push its database ping past the check timeout and flip `/readyz` to not-ready while the gateway is otherwise serving.
|
||||
|
||||
Omit the key to use the backend default, which is unchanged from earlier releases:
|
||||
|
||||
| Backend | Default ceiling |
|
||||
|---|---|
|
||||
| Postgres | 10 |
|
||||
| SQLite (on disk) | 5 |
|
||||
| SQLite (in memory) | 1, always. The database lives inside its single connection, so a configured value is ignored and the gateway logs that it was. |
|
||||
|
||||
The equivalent flag and environment variable are `--db-max-connections` and `OPENSHELL_DB_MAX_CONNECTIONS`, and Helm deployments set the key under `gatewayConfig.openshell.gateway`. Values below `1` are rejected at startup: a zero-sized pool would block every acquisition rather than degrade. Postgres needs at least `2`, because the SSH identity lock keeps one data connection while it queries the pool, so the gateway rejects `1` there too.
|
||||
|
||||
On Postgres, the gateway holds its mutation advisory locks on a separate lock pool, one connection per guarded mutation while it runs or waits. `[openshell.gateway] database_lock_max_connections` sets that pool's ceiling, `4` by default, so it bounds how many guarded mutations a replica runs or waits on in Postgres at once. When every lock connection is in use, further guarded mutations queue for one, and fail with `MUTATION_LOCK_TIMEOUT` only if none frees up before their 10-second lock wait runs out. The equivalent flag and environment variable are `--db-lock-max-connections` and `OPENSHELL_DB_LOCK_MAX_CONNECTIONS`, and Helm deployments set the key under `gatewayConfig.openshell.gateway`. Values below `1` are rejected at startup. SQLite has no lock pool, so the gateway ignores the key there and logs that it did.
|
||||
|
||||
On Postgres, each replica opens both pools, so plan for `(database_max_connections + database_lock_max_connections) × replicaCount` connections, 14 per replica with the defaults, and keep that total below the server's own `max_connections` (and below any connection-pooler limit in front of it). Rollouts run extra pods; refer to [Size PostgreSQL Connections](/kubernetes/high-availability#size-postgresql-connections). Raising either ceiling past what the server admits moves the failure from a slow acquire to a refused connection.
|
||||
|
||||
On SQLite, writers are serialized regardless of pool size, so a larger ceiling buys read concurrency only. A gateway that needs more write throughput wants an external Postgres database (`server.externalDbSecret`), not a larger SQLite pool.
|
||||
|
||||
## OCSF JSONL Output
|
||||
|
||||
`[openshell.gateway.ocsf_log]` enables a gateway-local file containing native OCSF events emitted by the gateway, including events without a sandbox association. Omit the table to disable it. Restart the gateway after changing this configuration.
|
||||
|
||||
@@ -212,6 +212,9 @@ configuration, and access control.
|
||||
| Relay rejections | `openshell_server_relay_rejected_total` | Alert on any increase. |
|
||||
| Relay claims | `openshell_server_relay_claim_duration_seconds`, `openshell_server_relay_expired_total` | Supervisor connect-back time for exec, SSH, forwarding, and service traffic, including relays requested through peers. A high 99th percentile while rejections stay flat points at the supervisor or its node, not at relay capacity. Relays not claimed within 10 seconds are missing from the histogram and count as expired, so alert on any increase. |
|
||||
| Routed requests | `openshell_server_routed_request_attempts_total`, `openshell_server_peer_request_duration_seconds` | Local/peer relay setup mix and outbound peer rate, failures, and latency. Counts completed or cancelled attempts, including retries. Spikes of `grpc_code="unavailable"` during rollouts are expected. Do not use as an HPA target. |
|
||||
| Lock contention | `openshell_server_mutation_lock_wait_seconds`, `openshell_server_mutation_lock_hold_seconds`, `openshell_server_mutation_lock_timeouts_total` | Alert on any timeout. Long holds on a scope make its waiters wait, and holds longer than 10 seconds make them time out. |
|
||||
| Lock errors | `openshell_server_mutation_lock_errors_total` | Alert on any increase, separately from timeouts. PostgreSQL ran out of connection slots, restarted, or became unreachable. Readiness can stay healthy, because it uses established data connections. |
|
||||
| Lock pool | `openshell_server_mutation_lock_connections_in_use`, `openshell_server_mutation_lock_connections_capacity` | PostgreSQL only. Graph in use over capacity per replica. Values that stay near 1 mean that guarded operations on this replica hold their lock connections, or wait in PostgreSQL for a lock held elsewhere, often by another replica, and lock waits and timeouts follow. |
|
||||
|
||||
These queries assume that Prometheus labels each series with `namespace` and
|
||||
`pod`:
|
||||
@@ -241,6 +244,24 @@ sum by (operation) (rate(openshell_server_routed_request_attempts_total{route="p
|
||||
|
||||
# 99th percentile peer request latency across the fleet.
|
||||
histogram_quantile(0.99, sum by (le, operation) (rate(openshell_server_peer_request_duration_seconds_bucket{outcome="success"}[5m])))
|
||||
|
||||
# 99th percentile mutation lock wait by scope. Watch scope="sandbox" during rollouts.
|
||||
histogram_quantile(0.99, sum by (le, scope) (rate(openshell_server_mutation_lock_wait_seconds_bucket[5m])))
|
||||
|
||||
# Mutation lock timeouts by scope. Alert on any increase.
|
||||
sum by (scope) (increase(openshell_server_mutation_lock_timeouts_total[10m]))
|
||||
|
||||
# 99th percentile time a mutation lock is held, by replica and scope.
|
||||
# Conflicting acquisitions time out after 10 seconds.
|
||||
histogram_quantile(0.99, sum by (namespace, pod, le, scope) (rate(openshell_server_mutation_lock_hold_seconds_bucket[5m])))
|
||||
|
||||
# Mutation lock acquisitions that failed without timing out. Timeouts do not
|
||||
# include them, so alert on any increase here as well.
|
||||
sum by (namespace, pod, scope) (increase(openshell_server_mutation_lock_errors_total[10m]))
|
||||
|
||||
# Lock-pool connections in use on each replica, PostgreSQL only. Values that
|
||||
# stay near 1 mean the pool is close to full.
|
||||
max by (namespace, pod) (max_over_time(openshell_server_mutation_lock_connections_in_use[5m]) / openshell_server_mutation_lock_connections_capacity)
|
||||
```
|
||||
|
||||
Relay capacity is used on the replica that owns the sandbox's supervisor
|
||||
@@ -250,6 +271,20 @@ stays counted for up to about 40 seconds, the 10-second claim timeout plus the
|
||||
30-second cleanup interval. Routed request counts include retries during
|
||||
rollouts.
|
||||
|
||||
Each supervisor that reconnects takes one or two short sandbox-scoped locks on
|
||||
the replica that receives it. When a gateway pod stops during a rollout, its
|
||||
supervisors reconnect to the other replicas, so watch
|
||||
`openshell_server_mutation_lock_wait_seconds` and
|
||||
`openshell_server_mutation_lock_timeouts_total` with `scope="sandbox"` on the
|
||||
receiving replicas. Rising waits or any timeout mean that those replicas
|
||||
receive reconnects faster than they can absorb them, or that slow compute
|
||||
driver, credential backend, middleware, or provider profile source calls during
|
||||
sandbox and provider changes hold their locks longer. Guarded work has no time
|
||||
limit of its own, so such a call holds its lock until it returns:
|
||||
`openshell_server_mutation_lock_hold_seconds` shows how long, and the replica
|
||||
logs `mutation lock held longer than the lock wait timeout` when it releases a
|
||||
lock held longer than 10 seconds.
|
||||
|
||||
## Scale the Gateway
|
||||
|
||||
Scale the gateway by changing a fixed replica count or by letting a
|
||||
@@ -380,9 +415,14 @@ kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/openshell/pods
|
||||
|
||||
### Size PostgreSQL Connections
|
||||
|
||||
Each gateway pod opens up to 10 PostgreSQL connections on demand. The chart
|
||||
uses the Kubernetes default rolling update strategy for each workload kind, so
|
||||
the number of pods that run during a rollout depends on the kind.
|
||||
With the default settings, each gateway pod opens up to 14 PostgreSQL
|
||||
connections on demand: 10 for data access (`database_max_connections`) and 4
|
||||
for mutation locks (`database_lock_max_connections`), both set under
|
||||
`gatewayConfig.openshell.gateway`. A request cancelled during
|
||||
a lock wait can leave its lock session open for up to 10 seconds, so keep
|
||||
headroom. The chart uses the Kubernetes default rolling update strategy for
|
||||
each workload kind, so the number of pods that run during a rollout depends on
|
||||
the kind.
|
||||
|
||||
A Deployment rollout adds up to 25 percent of the replica count, rounded up,
|
||||
as surge pods. Kubernetes does not count terminating pods against that surge,
|
||||
@@ -390,18 +430,20 @@ and each replaced pod can keep its connections throughout its termination
|
||||
grace period. A rollout can therefore run up to twice the replica count at
|
||||
once, and an eviction or an autoscaler scale-down during the rollout adds more
|
||||
terminating pods. Set PostgreSQL `max_connections` to at least
|
||||
`(2 × replicas + surge) × 10`, which leaves one surge of margin for those
|
||||
pods, plus headroom for your other clients and administration. Use
|
||||
`autoscaling.maxReplicas` as the replica count when autoscaling is enabled.
|
||||
`(2 × replicas + surge) × (data + lock connections)`, which leaves one surge
|
||||
of margin for those pods, plus headroom for your other clients and
|
||||
administration. Use `autoscaling.maxReplicas` as the replica count when
|
||||
autoscaling is enabled. With the defaults, that is
|
||||
`(2 × replicas + surge) × 14`:
|
||||
|
||||
| Replicas | Surge | Minimum `max_connections` |
|
||||
|---|---|---|
|
||||
| 2 | 1 | `(4 + 1) × 10 = 50` |
|
||||
| 3 | 1 | `(6 + 1) × 10 = 70` |
|
||||
| 4 | 1 | `(8 + 1) × 10 = 90` |
|
||||
| 6 | 2 | `(12 + 2) × 10 = 140` |
|
||||
| 2 | 1 | `(4 + 1) × 14 = 70` |
|
||||
| 3 | 1 | `(6 + 1) × 14 = 98` |
|
||||
| 4 | 1 | `(8 + 1) × 14 = 126` |
|
||||
| 6 | 2 | `(12 + 2) × 14 = 196` |
|
||||
|
||||
With a Deployment, five or more replicas exceed what the PostgreSQL default of
|
||||
With a Deployment, three or more replicas exceed what the PostgreSQL default of
|
||||
100 leaves for the gateway, because PostgreSQL reserves 3 connections for
|
||||
superusers. Raise `max_connections` or use a larger managed instance. Let one
|
||||
rollout finish before you start another, because each overlapping rollout adds
|
||||
@@ -410,14 +452,58 @@ its own terminating pods.
|
||||
If you run several replicas as a StatefulSet with
|
||||
`workload.allowMultiReplicaStatefulSet`, a rollout adds no surge pods. It
|
||||
replaces one pod at a time and creates each replacement only after the old pod
|
||||
exits. Set `max_connections` to at least `replicas × 10`, using
|
||||
`autoscaling.maxReplicas` when autoscaling is enabled, plus headroom for your
|
||||
other clients and administration.
|
||||
exits. Set `max_connections` to at least
|
||||
`replicas × (data + lock connections)`, 14 per replica with the defaults,
|
||||
using `autoscaling.maxReplicas` when autoscaling is enabled, plus headroom for
|
||||
your other clients and administration.
|
||||
|
||||
Raise `database_lock_max_connections` only when mutation lock timeouts on a
|
||||
replica come from a full lock pool: the gateway logs them with the detail
|
||||
`waiting for a mutation lock connection`, and
|
||||
`openshell_server_mutation_lock_connections_in_use` stays at
|
||||
`openshell_server_mutation_lock_connections_capacity`. Also check that the
|
||||
replica's own acquisitions are not mostly waiting behind another holder: few
|
||||
of its timeouts carry the detail `waiting for a PostgreSQL advisory lock` or
|
||||
`canceling statement due to lock timeout`, and `pg_locks` shows the pod's lock
|
||||
sessions granted rather than waiting. Otherwise, find the slow holder first,
|
||||
because a larger pool only queues more waiters behind it. Each extra lock
|
||||
connection adds `2 × replicas + surge` to the minimum `max_connections` for a
|
||||
Deployment.
|
||||
|
||||
If a connection pooler sits between the gateway and PostgreSQL, use session
|
||||
pooling. The gateway holds session-level advisory locks, which transaction
|
||||
pooling breaks.
|
||||
|
||||
Mutations lock only what they change. Sandbox operations lock their own
|
||||
sandbox, provider and workspace-profile changes lock their workspace, and
|
||||
gateway-global policy, settings, and platform-profile changes lock the whole
|
||||
fleet. Sandbox start, stop, and delete lock only on the replica that runs them,
|
||||
and only a gateway-global change on that replica blocks them directly.
|
||||
Provider, workspace-profile, and other replicas' gateway-global changes delay
|
||||
them only while another operation on the same sandbox waits behind the change.
|
||||
Sandbox create, start, restart, and delete also take one fleet-wide lock while
|
||||
they provision or remove the sandbox's SSH host identity, so that step runs for
|
||||
one sandbox at a time across all replicas. For other
|
||||
mutations, a lock wait longer than 10 seconds returns `UNAVAILABLE` with the
|
||||
reason `MUTATION_LOCK_TIMEOUT`, and clients can retry the request. A request
|
||||
that carried a `request_id` leaves its admission unresolved, like any other
|
||||
error, so a retry with the same ID returns `REQUEST_OUTCOME_UNCERTAIN`. Observe
|
||||
resource state and reconcile effects, then send a new request with a new
|
||||
`request_id`. Refer to
|
||||
[Durable Request Admission](/sdk/api-errors#durable-request-admission).
|
||||
|
||||
### Upgrade from an Earlier Release
|
||||
|
||||
With the default settings, each gateway pod now opens up to 14 PostgreSQL
|
||||
connections instead of 10. The upgrade rollout already runs new pods at that
|
||||
count, so raise `max_connections` to the minimum in
|
||||
[Size PostgreSQL Connections](#size-postgresql-connections) before you upgrade.
|
||||
|
||||
While gateways from the earlier release and the new release run together
|
||||
during the upgrade, the older replicas serialize every mutation across the
|
||||
fleet. Mutations can be slower and can briefly fail under load until the
|
||||
rollout finishes.
|
||||
|
||||
## Next Steps
|
||||
|
||||
- To expose the gateway through a highly available data path, refer to
|
||||
|
||||
@@ -3,7 +3,7 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
title: "Gateway Metrics"
|
||||
sidebar-title: "Gateway Metrics"
|
||||
description: "Scrape Prometheus metrics from the OpenShell gateway, including supervisor session, relay capacity, and peer routing signals for multi-replica deployments."
|
||||
description: "Scrape Prometheus metrics from the OpenShell gateway, including supervisor session, relay capacity, peer routing, and mutation lock signals for multi-replica deployments."
|
||||
keywords: "Generative AI, Cybersecurity, Observability, Metrics, Prometheus, Gateway, Kubernetes, High Availability"
|
||||
---
|
||||
|
||||
@@ -175,6 +175,17 @@ that owns a sandbox's supervisor session:
|
||||
| `openshell_server_routed_request_attempts_total` | Counter | `operation`, `route`, `relay_kind`, `outcome`, `grpc_code` | Local relay setup and outbound peer attempts, counted once when they finish or are cancelled. Each retry counts separately. |
|
||||
| `openshell_server_peer_request_duration_seconds` | Histogram | `operation`, `outcome` | Latency of outbound peer requests only (`route="peer"`). For relays, until the owner's supervisor claimed the relay. |
|
||||
|
||||
Mutation locks:
|
||||
|
||||
| Metric | Type | Labels | Description |
|
||||
|---|---|---|---|
|
||||
| `openshell_server_mutation_lock_wait_seconds` | Histogram | `scope` | Time to acquire a mutation lock, recorded for successful acquisitions. |
|
||||
| `openshell_server_mutation_lock_hold_seconds` | Histogram | `scope` | Time a mutation lock was held, recorded when it is released, including requests cancelled while they held it. A conflicting acquisition times out after 10 seconds, so the gateway also logs `mutation lock held longer than the lock wait timeout` with `scope` and `held_ms` when it releases a lock held longer than that. |
|
||||
| `openshell_server_mutation_lock_timeouts_total` | Counter | `scope` | Acquisitions that waited longer than 10 seconds, including background work such as supervisor disconnect handling and provider credential refresh. API requests fail with `UNAVAILABLE` and reason `MUTATION_LOCK_TIMEOUT`, and [API Errors](/sdk/api-errors) describes how to retry them. A lock connection that PostgreSQL does not open, although at least a second of the wait remained, counts in `openshell_server_mutation_lock_errors_total` instead. |
|
||||
| `openshell_server_mutation_lock_errors_total` | Counter | `scope` | Acquisitions that failed without timing out: PostgreSQL did not open a lock connection, or a lock statement failed. API requests fail with `INTERNAL`. |
|
||||
| `openshell_server_mutation_lock_connections_in_use` | Gauge | None | PostgreSQL lock-pool connections checked out on this replica: holding mutation locks, waiting for a PostgreSQL lock, or being returned to the pool, which can take up to 5 seconds. Exported only with PostgreSQL. |
|
||||
| `openshell_server_mutation_lock_connections_capacity` | Gauge | None | Lock-pool size per replica, `4` unless [`database_lock_max_connections`](/how-it-works/gateways/configuration#database-pool) sets it. Exported only with PostgreSQL. |
|
||||
|
||||
The labels take these values:
|
||||
|
||||
- `operation` is `relay`, `report_provider_readiness`, `report_endpoint_status`,
|
||||
@@ -210,9 +221,12 @@ The labels take these values:
|
||||
unchanged. The label is not named `code` because
|
||||
`openshell_server_grpc_requests_total` uses that name for the numeric status
|
||||
code.
|
||||
- `scope` is `global`, `workspace`, or `sandbox`.
|
||||
|
||||
Gauges and counters exist from startup. They start at `0`, except
|
||||
`openshell_server_relay_pending_capacity`, which starts at its limit. `openshell_server_routed_request_attempts_total` starts with seven
|
||||
`openshell_server_relay_pending_capacity` and
|
||||
`openshell_server_mutation_lock_connections_capacity`, which start at their
|
||||
limits. `openshell_server_routed_request_attempts_total` starts with seven
|
||||
`outcome="success", grpc_code="ok"` series: four relay kind and route
|
||||
combinations, and three other peer operations with `relay_kind="none"`. Other outcomes and codes,
|
||||
and every histogram, appear after their first sample, so write alert expressions
|
||||
|
||||
@@ -31,6 +31,7 @@ Recognized gateway reasons include the following.
|
||||
| `INVALID_ARGUMENT` | `INVALID_ARGUMENT` | Correct the fields listed in `BadRequest`. |
|
||||
| `RESOURCE_VERSION_CONFLICT` | `ABORTED` | Read the resource again and construct a new conditional write. `metadata.recovery` is `REFRESH_STATE`; `current_resource_version` is included when known. |
|
||||
| `PROFILE_SOURCE_UNAVAILABLE` | `UNAVAILABLE` | Retry a profile snapshot read after at least the supplied delay. |
|
||||
| `MUTATION_LOCK_TIMEOUT` | `UNAVAILABLE` | The request waited more than 10 seconds for a concurrent mutation to release a sandbox, workspace, or gateway-wide lock. Retry after at least the supplied delay. A request that carried a `request_id` leaves its admission unresolved, like any other error, so a retry with that ID returns `REQUEST_OUTCOME_UNCERTAIN`. Observe resource state and reconcile effects before you start a new request with a new ID. |
|
||||
| `REQUEST_ID_PAYLOAD_MISMATCH` | `FAILED_PRECONDITION` | Keep the original payload for that request ID. Inspect the original operation before submitting a different one. |
|
||||
| `REQUEST_OUTCOME_UNCERTAIN` | `FAILED_PRECONDITION` | An attempt is admitted but has no confirmed replayable success. Observe resource state and reconcile effects. Do not switch to a new ID to bypass the admission. |
|
||||
| `REQUEST_REPLAY_UNAVAILABLE` | `FAILED_PRECONDITION` | The original scope, resource, interceptor transformation, or fingerprint key is no longer replayable. Reconcile effects; the gateway does not execute the request again. Missing private-key material can also reject admission before work starts. |
|
||||
|
||||
Reference in New Issue
Block a user