OpenShell/docs/observability/gateway-metrics.mdx

---
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
title: "Gateway Metrics"
sidebar-title: "Gateway Metrics"
description: "Scrape Prometheus metrics from the OpenShell gateway, including supervisor session, relay capacity, and peer routing signals for multi-replica deployments."
keywords: "Generative AI, Cybersecurity, Observability, Metrics, Prometheus, Gateway, Kubernetes, High Availability"
---

The gateway exports Prometheus metrics for API requests, database readiness,
gateway interceptors, the OCSF log file, and multi-replica capacity. Use them
to alert on saturation, to follow rollouts, and to choose autoscaling signals.

## Enable the Metrics Endpoint

The gateway serves `GET /metrics` in the Prometheus text format on its own
listener, separate from the API and health ports. When the listener is
disabled, the gateway records no metrics. Enable it in one of these ways:

- In `gateway.toml`, set `metrics_bind_address` under `[openshell.gateway]`,
  for example `metrics_bind_address = "0.0.0.0:9090"`. Refer to
  [Gateway Configuration](/how-it-works/gateways/configuration).
- On the command line, pass `--metrics-port` or set `OPENSHELL_METRICS_PORT`.
  The standalone gateway binary defaults to `0`, which disables the endpoint.
- In the Helm chart, set `service.metricsPort`, which defaults to `9090`. The
  chart renders the TOML key, a container port named `metrics`, and a
  `metrics` port on the gateway Service. Set it to `0` to disable the
  endpoint.

A stopping gateway keeps serving metrics until the process exits.

<Warning>
The metrics endpoint uses plaintext HTTP with no authentication. Anyone who
can reach the port can read the metrics. The Helm chart publishes the port on
the gateway Service, so a `LoadBalancer` or `NodePort` Service exposes it
outside the cluster. Keep the Service internal, and restrict the port to your
monitoring system.
</Warning>

The following NetworkPolicy keeps the API and health ports open and accepts
metrics scrapes only from the `monitoring` namespace. Adjust the ports if you
changed `service.port`, `service.healthPort`, or `service.metricsPort`.

```yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: openshell-gateway-metrics
  namespace: openshell
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: openshell
      app.kubernetes.io/instance: openshell
  policyTypes:
    - Ingress
  ingress:
    - ports:
        - port: 8080
        - port: 8081
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: monitoring
      ports:
        - port: 9090
```

## Scrape Each Replica

Capacity metrics describe one gateway replica, so scrape every gateway pod
rather than the Service, and keep the `namespace` and `pod` labels on each
series. With the Prometheus Operator, a PodMonitor adds those labels
automatically:

```yaml
apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: openshell-gateway
  namespace: openshell
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: openshell
      app.kubernetes.io/instance: openshell
  podMetricsEndpoints:
    - port: metrics
      path: /metrics
      interval: 30s
```

With a plain Prometheus configuration, use pod discovery and copy the labels
yourself:

```yaml
scrape_configs:
  - job_name: openshell-gateway
    kubernetes_sd_configs:
      - role: pod
        namespaces:
          names: [openshell]
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_label_app_kubernetes_io_instance]
        regex: openshell
        action: keep
      - source_labels: [__meta_kubernetes_pod_container_port_name]
        regex: metrics
        action: keep
      - source_labels: [__meta_kubernetes_namespace]
        target_label: namespace
      - source_labels: [__meta_kubernetes_pod_name]
        target_label: pod
```

To check one pod without Prometheus, read its metrics through the Kubernetes
API server proxy:

```shell
kubectl get --raw "/api/v1/namespaces/openshell/pods/<pod>:9090/proxy/metrics" \
  | grep '^openshell_server_supervisor_sessions'
```

The API server connects to the pod from the control plane. A NetworkPolicy
like the one above blocks that connection unless it also allows the control
plane on the metrics port. `kubectl port-forward` reaches the pod through the
kubelet instead, and NetworkPolicy does not block it. Start the port-forward:

```shell
kubectl -n openshell port-forward pod/<pod> 9090:9090
```

Then read the metrics from another terminal:

```shell
curl -s http://localhost:9090/metrics | grep '^openshell_server_supervisor_sessions'
```

## Histograms and Summaries

The multi-replica latency metrics are Prometheus histograms with `_bucket`,
`_sum`, and `_count` series and buckets from 1 millisecond to 15 seconds, so
`histogram_quantile` can aggregate them across replicas. The older
`*_duration_seconds` metrics and the interceptor latency metric are summaries
with `quantile` labels that each replica computes on its own. Do not average
summary quantiles across pods.

## Multi-Replica Capacity Metrics

These metrics describe how much work one replica holds and how the replicas
interact. No metric carries a sandbox, channel, endpoint, or replica
identifier, because the scrape target already identifies the replica.

Supervisor sessions:

| Metric | Type | Labels | Description |
|---|---|---|---|
| `openshell_server_supervisor_sessions` | Gauge | None | Supervisor control sessions held by this replica. After a rollout settles, the sum across pods equals the number of connected sandboxes. |

Relays:

| Metric | Type | Labels | Description |
|---|---|---|---|
| `openshell_server_relay_pending` | Gauge | None | Relay channels on this replica waiting for the supervisor to connect back, including channels opened for peer replicas. A channel whose caller gave up stays counted until it is claimed or cleaned up, up to about 40 seconds. |
| `openshell_server_relay_pending_capacity` | Gauge | None | Pending relay limit per replica, `256`. |
| `openshell_server_relay_rejected_total` | Counter | `reason` | Relay opens rejected at a limit. Rejected relays fail the client request, so use this counter, not client error codes, to detect saturation. |
| `openshell_server_relay_expired_total` | Counter | None | Pending relays dropped because the supervisor did not connect back within 10 seconds. |
| `openshell_server_relay_claim_duration_seconds` | Histogram | None | Time from opening a relay to the supervisor claiming it. |

Routed requests include local relay setup and outbound requests to the replica
that owns a sandbox's supervisor session:

| Metric | Type | Labels | Description |
|---|---|---|---|
| `openshell_server_routed_request_attempts_total` | Counter | `operation`, `route`, `relay_kind`, `outcome`, `grpc_code` | Local relay setup and outbound peer attempts, counted once when they finish or are cancelled. Each retry counts separately. |
| `openshell_server_peer_request_duration_seconds` | Histogram | `operation`, `outcome` | Latency of outbound peer requests only (`route="peer"`). For relays, until the owner's supervisor claimed the relay. |

The labels take these values:

- `operation` is `relay`, `report_provider_readiness`, `report_endpoint_status`,
  or `get_sandbox_provider_status`. Only relay setup includes the local route;
  the other operations count outbound peer calls, not local API handling. On the
  owning replica, `openshell_server_grpc_requests_total` records the matching
  `method`, such as `PeerRelay` for `relay`.
- `relay_kind` is `ssh` or `tcp` for relays, and `none` for other operations. An
  omitted relay target counts as `ssh`, matching the protocol's compatibility
  behavior. TCP hostnames and ports are not labels.
- `route` is `local` when this replica tries its own supervisor session, or
  `peer` when it tries another replica. A routed retry counts again, including a
  change from local to peer routing. Waiting for an owner does not count. Incoming
  `PeerRelay` requests do not increment this counter on the receiving replica;
  they appear in that replica's gRPC request counter instead. This counter does
  not measure active connections, unique client requests, or unfinished attempts.
- `reason` is `replica_capacity` when the gateway replica's pending-relay budget
  is full, or `sandbox_capacity` when one sandbox's budget on that replica, 32
  pending relays, is full.
- `outcome` is `success`, `local_error`, or `remote_error`. A relay succeeds
  when the supervisor claims it; this does not measure the data stream.
  `local_error` means the attempt failed on this replica, and `remote_error`
  means the owning replica returned an error or the open connection to it
  failed. On `route="local"` this replica is the owner, so failures that appear
  as `remote_error` on the peer route, such as an unclaimed relay
  (`deadline_exceeded`) or a full relay budget (`resource_exhausted`), appear as
  `local_error`. A cancelled attempt records `local_error` with
  `grpc_code="cancelled"`, even when the owner was already handling it.
- `grpc_code` is the lowercase gRPC status name, such as `ok`, `unavailable`,
  or `resource_exhausted`, from local setup, the local supervisor claim, or the
  peer RPC. For a peer relay, the gateway then reports a failure to the client
  as `UNAVAILABLE`. The other RPCs return the owner's status to the client
  unchanged. The label is not named `code` because
  `openshell_server_grpc_requests_total` uses that name for the numeric status
  code.

Gauges and counters exist from startup. They start at `0`, except
`openshell_server_relay_pending_capacity`, which starts at its limit. `openshell_server_routed_request_attempts_total` starts with seven
`outcome="success", grpc_code="ok"` series: four relay kind and route
combinations, and three other peer operations with `relay_kind="none"`. Other outcomes and codes,
and every histogram, appear after their first sample, so write alert expressions
that tolerate absent series.

## Request and Readiness Metrics

The gateway also records every API request and its background database check:

| Metric | Type | Labels | Description |
|---|---|---|---|
| `openshell_server_grpc_requests_total` | Counter | `method`, `code` | gRPC requests by method name and numeric status code, including internal peer requests. Streaming RPCs record the status known when the response headers are sent. |
| `openshell_server_grpc_request_duration_seconds` | Summary | `method`, `code` | Time until the response headers are sent. For streaming RPCs this covers setup only. |
| `openshell_server_http_requests_total` | Counter | `path`, `status` | HTTP requests on the API port. `path` is `/_ws_tunnel`, `/auth`, or `unknown`. |
| `openshell_server_http_request_duration_seconds` | Summary | `path`, `status` | Duration of those HTTP requests. |
| `openshell_server_readiness_database_healthy` | Gauge | None | `1` when the last background database check succeeded, otherwise `0`. |
| `openshell_server_readiness_database_probe_duration_seconds` | Summary | `outcome` | Duration of the background database check. `outcome` is `success`, `db_error`, or `timeout`. |

## Gateway Interceptor Metrics

Gateways with [gateway interceptors](/extensibility/gateway-interceptors)
record their evaluations:

| Metric | Type | Labels | Description |
|---|---|---|---|
| `openshell_gateway_interceptor_latency_seconds` | Summary | None | Duration of interceptor calls. |
| `openshell_gateway_interceptor_evaluations_total` | Counter | `decision`, `interceptor`, `binding_id` | Interceptor evaluations by decision. |
| `openshell_gateway_interceptor_patches_total` | Counter | `interceptor`, `binding_id` | Patches that interceptors applied. |
| `openshell_gateway_interceptor_fail_open_total` | Counter | None | Interceptor failures that let the request continue. |
| `openshell_gateway_interceptor_fail_closed_total` | Counter | None | Interceptor failures that rejected the request. |
| `openshell_gateway_interceptor_post_commit_observation_failures_total` | Counter | `stage` | Failed post-commit observations. `stage` is `response_body` or `evaluation`. |

## OCSF Log Metrics

Gateways that write an
[OCSF JSONL file](/how-it-works/gateways/configuration#ocsf-jsonl-output)
record its queue and writer:

| Metric | Type | Labels | Description |
|---|---|---|---|
| `openshell_ocsf_log_queued_total` | Counter | None | Records queued for the file. |
| `openshell_ocsf_log_written_total` | Counter | None | Records written to the file. |
| `openshell_ocsf_log_dropped_total` | Counter | `reason` | Records discarded, by bounded reason. |
| `openshell_ocsf_log_writer_errors_total` | Counter | None | Failed writes, reopens, and retention cleanups. |
| `openshell_ocsf_log_recovery_discarded_bytes_total` | Counter | None | Bytes of an incomplete trailing line removed when the writer opens the file. |
| `openshell_ocsf_log_queue_records` | Gauge | None | Records waiting to be written. |
| `openshell_ocsf_log_queue_bytes` | Gauge | None | Encoded bytes waiting to be written. |

## Next Steps

- To monitor capacity and choose autoscaling signals for multiple gateway
  replicas, refer to
  [High Availability](/kubernetes/high-availability#monitor-capacity).
- To configure the metrics listener in `gateway.toml`, refer to
  [Gateway Configuration](/how-it-works/gateways/configuration).