mirror of
https://github.com/agent-substrate/substrate.git
synced 2026-11-12 10:35:06 +08:00
Jeff LuoandKrisztian F
9bfc67e815
Part of #563 Substrate had no way to measure how much telemetry it sends. This PR adds one, and builds it from the parts the benchmark suite already has. The managed collector sends its self-metrics to Cloud Monitoring only, and gives no data for each source. Thus you cannot see which component makes the volume. A second collector, the telemetry meter, counts the spans and the datapoints by `service.name`. ## Contents - **`benchmarking/telemetry/meter.yaml`** — the meter. A tee: it counts the telemetry by service, then sends it on to the managed collector. Thus one run gives the volume for each service and the cost of the load together. - **`benchmarking/telemetry/README.md`** — the method: install, measure, read, remove. - **`hack/install-ate.sh --otlp-endpoint`** — sends all control plane telemetry to the meter. Each component reads the endpoint from the `ate-otel-config` ConfigMap through `envFrom`, and `ate-controller` copies the value to the ateom worker pods, thus one patch is sufficient for all of them. - **`benchmarking/workloads/deploy.sh --otlp-endpoint`** — the same for actor containers, which substrate configures separately. - **`benchmarking/monitoring.yaml`** — scrapes the meter and cAdvisor. - **`benchmarking/observability.md`** — prerequisites and the scenario ladder. - **`docs/dev/best-practices/otel-collector.md`** — the volume model and the size guidance, corrected for the `ParentBased` defaults from #711. ## Correction The `glutton` ActorTemplate had no `env`. Substrate puts no OTLP configuration in an actor container, thus the exporter used the SDK default `localhost:4317`, no process listens on that address in the sandbox, and substrate discarded each actor span with no error message. The template now sets `OTEL_EXPORTER_OTLP_ENDPOINT` and `OTEL_RESOURCE_ATTRIBUTES`. `ActorTemplate.spec` is immutable, thus `deploy.sh --deploy` deletes the templates before it applies them. ## How I verified this On a GKE cluster with managed OTel. The sequence: ```bash # 1. Start from a clean state. ./hack/install-ate.sh --delete-benchmarks ./hack/install-ate.sh --delete-ate-system # 2. Install substrate from this branch. ./hack/install-ate.sh --deploy-ate-system # 3. Install the meter and Prometheus. kubectl apply -f benchmarking/telemetry/meter.yaml kubectl apply -f benchmarking/monitoring.yaml # 4. Send the control plane to the meter. METER=http://telemetry-meter.benchmarking.svc.cluster.local:4317 ./hack/install-ate.sh --deploy-ate-system --otlp-endpoint "${METER}" # 5. Install the workloads and locust, with the actors on the meter also. ./benchmarking/deploy_locust.sh --deploy --worker-count 10 \ --otlp-endpoint "${METER}" # 6. Read the volume. kubectl port-forward -n benchmarking svc/prometheus 9090:9090 ``` ## Result The idle floor, from one measurement on 10 workers: | Service | datapoints/min | |---|---| | `ate-controller` | 234.8 | | `ateapi` | 54.3 | | `atelet` | 44.9 | | `atenet-router` | 10.4 | | `ateom-gvisor` | 5.2 | | `glutton-actor` | 4.2 | | **Total** | **353.7** | The `glutton-actor` row is the correction above. Before it, actor telemetry could not arrive at all. Positive control for the same window: `otelcol_receiver_accepted_metric_points` 10.5/s, and each `otelcol_receiver_refused_*` counter at zero. Thus the zero values in the table are true zeros, and not data that did not arrive. Two values differ from the earlier manual measurement. `ate-controller` is now the largest source, and `ateom-gvisor` is no longer zero. The new imagecache counters from #831 are a probable cause. This difference is the reason to keep results out of the repository. ### Traces Trace volume follows the root sample decision, not the rate of each component. `atelet` and `ateom-gvisor` sent zero spans in each run, because they inherit the decision that `ateapi` makes and start no trace of their own. At idle each component is near zero: a trace needs an operation, but a metric pushes on a timer. This run gives no span volume for a usual load. The actors crashed on suspend, and the failed retries made an artificial rate of 13.7k spans/s. That rate is not a measurement of substrate. It does show that the meter accepted the spans with no refusals. ## Follow-up A separate PR adds the ladder to `automation/tests.yaml` and writes the results as artifacts of each run. The open items are at the end of `observability.md`. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR --------- Co-authored-by: Krisztian F <103492698+krisztianfekete@users.noreply.github.com>
659 lines
24 KiB
YAML
659 lines
24 KiB
YAML
# Copyright 2026 Google LLC
|
|
#
|
|
# Licensed under the Apache License, Version 2.0 (the "License");
|
|
# you may not use this file except in compliance with the License.
|
|
# You may obtain a copy of the License at
|
|
#
|
|
# http://www.apache.org/licenses/LICENSE-2.0
|
|
#
|
|
# Unless required by applicable law or agreed to in writing, software
|
|
# distributed under the License is distributed on an "AS IS" BASIS,
|
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
# See the License for the specific language governing permissions and
|
|
# limitations under the License.
|
|
|
|
apiVersion: v1
|
|
kind: Namespace
|
|
metadata:
|
|
name: benchmarking
|
|
---
|
|
apiVersion: v1
|
|
kind: ServiceAccount
|
|
metadata:
|
|
name: prometheus
|
|
namespace: benchmarking
|
|
---
|
|
apiVersion: rbac.authorization.k8s.io/v1
|
|
kind: Role
|
|
metadata:
|
|
name: prometheus
|
|
namespace: benchmarking
|
|
rules:
|
|
- apiGroups: [""]
|
|
resources: ["nodes", "nodes/proxy", "services", "endpoints", "pods"]
|
|
verbs: ["get", "list", "watch"]
|
|
- apiGroups: ["extensions", "networking.k8s.io"]
|
|
resources: ["ingresses"]
|
|
verbs: ["get", "list", "watch"]
|
|
---
|
|
apiVersion: rbac.authorization.k8s.io/v1
|
|
kind: RoleBinding
|
|
metadata:
|
|
name: prometheus
|
|
namespace: benchmarking
|
|
roleRef:
|
|
apiGroup: rbac.authorization.k8s.io
|
|
kind: Role
|
|
name: prometheus
|
|
subjects:
|
|
- kind: ServiceAccount
|
|
name: prometheus
|
|
namespace: benchmarking
|
|
---
|
|
# A node is a cluster-scoped resource. Thus the namespaced Role above cannot
|
|
# give access to node discovery or to the kubelet proxy. The cadvisor scrape
|
|
# job needs the two permissions.
|
|
apiVersion: rbac.authorization.k8s.io/v1
|
|
kind: ClusterRole
|
|
metadata:
|
|
name: prometheus-benchmarking-nodes
|
|
rules:
|
|
- apiGroups: [""]
|
|
resources: ["nodes", "nodes/proxy", "nodes/metrics"]
|
|
verbs: ["get", "list", "watch"]
|
|
- nonResourceURLs: ["/metrics"]
|
|
verbs: ["get"]
|
|
---
|
|
apiVersion: rbac.authorization.k8s.io/v1
|
|
kind: ClusterRoleBinding
|
|
metadata:
|
|
name: prometheus-benchmarking-nodes
|
|
roleRef:
|
|
apiGroup: rbac.authorization.k8s.io
|
|
kind: ClusterRole
|
|
name: prometheus-benchmarking-nodes
|
|
subjects:
|
|
- kind: ServiceAccount
|
|
name: prometheus
|
|
namespace: benchmarking
|
|
---
|
|
apiVersion: v1
|
|
kind: ConfigMap
|
|
metadata:
|
|
name: prometheus-config
|
|
namespace: benchmarking
|
|
data:
|
|
prometheus.yml: |
|
|
global:
|
|
scrape_interval: 10s
|
|
scrape_configs:
|
|
- job_name: 'kubernetes-pods'
|
|
kubernetes_sd_configs:
|
|
- role: pod
|
|
namespaces:
|
|
names:
|
|
- benchmarking
|
|
relabel_configs:
|
|
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
|
|
action: keep
|
|
regex: "true"
|
|
- source_labels: [__meta_kubernetes_pod_name]
|
|
action: replace
|
|
target_label: pod
|
|
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
|
|
action: replace
|
|
target_label: __metrics_path__
|
|
regex: (.+)
|
|
- source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
|
|
action: replace
|
|
regex: ([^:]+)(?::\d+)?;(\d+)
|
|
replacement: $1:$2
|
|
target_label: __address__
|
|
|
|
# The telemetry-meter gives its substrate_* counts on port 8889, which
|
|
# the annotation job above collects. It gives its own otelcol_* counters
|
|
# on port 8888. The annotation relabel can name only one port, thus
|
|
# scrape port 8888 here. When a service reports zero volume,
|
|
# otelcol_receiver_accepted_* shows the difference between "sent nothing"
|
|
# and "arrived nowhere".
|
|
- job_name: 'telemetry-meter-self'
|
|
kubernetes_sd_configs:
|
|
- role: pod
|
|
namespaces:
|
|
names:
|
|
- benchmarking
|
|
relabel_configs:
|
|
- source_labels:
|
|
- __meta_kubernetes_pod_label_app
|
|
- __meta_kubernetes_pod_container_port_number
|
|
action: keep
|
|
regex: telemetry-meter;8888
|
|
- source_labels: [__meta_kubernetes_pod_name]
|
|
action: replace
|
|
target_label: pod
|
|
|
|
# The container memory directly from the cAdvisor endpoint of the
|
|
# kubelet. `kubectl top` reads metrics-server, which reports an average
|
|
# across its own window and hides short peaks.
|
|
# container_memory_working_set_bytes is the value that the OOM killer
|
|
# uses.
|
|
- job_name: 'kubernetes-cadvisor'
|
|
scheme: https
|
|
tls_config:
|
|
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
|
|
insecure_skip_verify: true
|
|
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
|
|
kubernetes_sd_configs:
|
|
- role: node
|
|
relabel_configs:
|
|
- action: labelmap
|
|
regex: __meta_kubernetes_node_label_(.+)
|
|
- target_label: __address__
|
|
replacement: kubernetes.default.svc:443
|
|
- source_labels: [__meta_kubernetes_node_name]
|
|
regex: (.+)
|
|
target_label: __metrics_path__
|
|
replacement: /api/v1/nodes/$1/proxy/metrics/cadvisor
|
|
metric_relabel_configs:
|
|
# cAdvisor sends a very large number of metrics. Keep only the
|
|
# metrics that the benchmarks read.
|
|
- source_labels: [__name__]
|
|
action: keep
|
|
regex: container_(memory_working_set_bytes|cpu_usage_seconds_total|spec_memory_limit_bytes)
|
|
# Drop the series that have an empty container label, not the label
|
|
# itself. cAdvisor gives a rollup for the whole pod cgroup next to
|
|
# the series for each container. Keeping both counts the memory of
|
|
# each pod two times.
|
|
- source_labels: [container]
|
|
action: drop
|
|
regex: ""
|
|
---
|
|
apiVersion: apps/v1
|
|
kind: Deployment
|
|
metadata:
|
|
name: prometheus
|
|
namespace: benchmarking
|
|
labels:
|
|
app: prometheus
|
|
spec:
|
|
replicas: 1
|
|
selector:
|
|
matchLabels:
|
|
app: prometheus
|
|
template:
|
|
metadata:
|
|
labels:
|
|
app: prometheus
|
|
spec:
|
|
serviceAccountName: prometheus
|
|
containers:
|
|
- name: prometheus
|
|
image: prom/prometheus:v2.45.0
|
|
args:
|
|
- "--config.file=/etc/prometheus/prometheus.yml"
|
|
- "--storage.tsdb.path=/prometheus/"
|
|
ports:
|
|
- containerPort: 9090
|
|
volumeMounts:
|
|
- name: config-volume
|
|
mountPath: /etc/prometheus/
|
|
volumes:
|
|
- name: config-volume
|
|
configMap:
|
|
name: prometheus-config
|
|
---
|
|
apiVersion: v1
|
|
kind: Service
|
|
metadata:
|
|
name: prometheus
|
|
namespace: benchmarking
|
|
spec:
|
|
selector:
|
|
app: prometheus
|
|
ports:
|
|
- protocol: TCP
|
|
port: 9090
|
|
targetPort: 9090
|
|
---
|
|
apiVersion: v1
|
|
kind: ConfigMap
|
|
metadata:
|
|
name: grafana-datasources
|
|
namespace: benchmarking
|
|
data:
|
|
prometheus.yaml: |-
|
|
apiVersion: 1
|
|
datasources:
|
|
- name: Prometheus
|
|
type: prometheus
|
|
url: http://prometheus:9090
|
|
access: proxy
|
|
isDefault: true
|
|
---
|
|
apiVersion: v1
|
|
kind: ConfigMap
|
|
metadata:
|
|
name: grafana-dashboards-provider
|
|
namespace: benchmarking
|
|
data:
|
|
provider.yaml: |-
|
|
apiVersion: 1
|
|
providers:
|
|
- name: 'default'
|
|
orgId: 1
|
|
folder: 'Locust'
|
|
type: file
|
|
disableDeletion: false
|
|
updateIntervalSeconds: 10
|
|
editable: true
|
|
options:
|
|
path: /var/lib/grafana/dashboards
|
|
---
|
|
apiVersion: v1
|
|
kind: ConfigMap
|
|
metadata:
|
|
name: grafana-dashboards
|
|
namespace: benchmarking
|
|
data:
|
|
locust-dashboard.json: |-
|
|
{
|
|
"uid": "locust-load-test-dashboard",
|
|
"title": "ATE API",
|
|
"panels": [
|
|
{
|
|
"title": "GetActor QPS",
|
|
"type": "timeseries",
|
|
"gridPos": { "h": 8, "w": 12, "x": 0, "y": 0 },
|
|
"targets": [
|
|
{ "expr": "sum(rate(locust_requests_total_total{name=\"GetActor\", user_class=\"AteAPIUser\"}[1m])) or sum(rate(locust_requests_total{name=\"GetActor\", user_class=\"AteAPIUser\"}[1m]))", "refId": "A", "legendFormat": "Total" },
|
|
{ "expr": "sum(rate(locust_requests_total_total{name=\"GetActor\", user_class=\"AteAPIUser\"}[1m])) by (status) or sum(rate(locust_requests_total{name=\"GetActor\", user_class=\"AteAPIUser\"}[1m])) by (status)", "refId": "B", "legendFormat": "{{status}}" }
|
|
],
|
|
"fieldConfig": { "defaults": { "unit": "req/sec" } }
|
|
},
|
|
{
|
|
"title": "GetActor Latency 99th Pct",
|
|
"type": "timeseries",
|
|
"gridPos": { "h": 8, "w": 12, "x": 12, "y": 0 },
|
|
"targets": [
|
|
{ "expr": "histogram_quantile(0.99, sum(rate(locust_request_duration_milliseconds_bucket{name=\"GetActor\", user_class=\"AteAPIUser\"}[1m])) by (le))", "refId": "A", "legendFormat": "Total" },
|
|
{ "expr": "histogram_quantile(0.99, sum(rate(locust_request_duration_milliseconds_bucket{name=\"GetActor\", user_class=\"AteAPIUser\"}[1m])) by (le, status))", "refId": "B", "legendFormat": "{{status}}" }
|
|
],
|
|
"fieldConfig": { "defaults": { "unit": "ms" } }
|
|
},
|
|
{
|
|
"title": "GetActor Latency Heatmap",
|
|
"type": "heatmap",
|
|
"gridPos": { "h": 8, "w": 24, "x": 0, "y": 8 },
|
|
"targets": [
|
|
{
|
|
"expr": "sum(rate(locust_request_duration_milliseconds_bucket{name=\"GetActor\", user_class=\"AteAPIUser\"}[1m])) by (le)",
|
|
"refId": "A"
|
|
}
|
|
],
|
|
|
|
"transformations": [
|
|
{
|
|
"id": "labelsToFields",
|
|
"options": {
|
|
"valueLabel": "le"
|
|
}
|
|
}
|
|
],
|
|
"options": {
|
|
"calculate": false,
|
|
"yAxis": {
|
|
"unit": "ms"
|
|
}
|
|
}
|
|
},
|
|
{
|
|
"title": "ResumeActor QPS",
|
|
"type": "timeseries",
|
|
"gridPos": { "h": 8, "w": 12, "x": 0, "y": 16 },
|
|
"targets": [
|
|
{ "expr": "sum(rate(locust_requests_total_total{name=\"ResumeActor\", user_class=\"AteAPIUser\"}[1m])) or sum(rate(locust_requests_total{name=\"ResumeActor\", user_class=\"AteAPIUser\"}[1m]))", "refId": "A", "legendFormat": "Total" },
|
|
{ "expr": "sum(rate(locust_requests_total_total{name=\"ResumeActor\", user_class=\"AteAPIUser\"}[1m])) by (status) or sum(rate(locust_requests_total{name=\"ResumeActor\", user_class=\"AteAPIUser\"}[1m])) by (status)", "refId": "B", "legendFormat": "{{status}}" }
|
|
],
|
|
"fieldConfig": { "defaults": { "unit": "req/sec" } }
|
|
},
|
|
{
|
|
"title": "ResumeActor Latency 99th Pct",
|
|
"type": "timeseries",
|
|
"gridPos": { "h": 8, "w": 12, "x": 12, "y": 16 },
|
|
"targets": [
|
|
{ "expr": "histogram_quantile(0.99, sum(rate(locust_request_duration_milliseconds_bucket{name=\"ResumeActor\", user_class=\"AteAPIUser\"}[1m])) by (le))", "refId": "A", "legendFormat": "Total" },
|
|
{ "expr": "histogram_quantile(0.99, sum(rate(locust_request_duration_milliseconds_bucket{name=\"ResumeActor\", user_class=\"AteAPIUser\"}[1m])) by (le, status))", "refId": "B", "legendFormat": "{{status}}" }
|
|
],
|
|
"fieldConfig": { "defaults": { "unit": "ms" } }
|
|
},
|
|
{
|
|
"title": "ResumeActor Latency Heatmap",
|
|
"type": "heatmap",
|
|
"gridPos": { "h": 8, "w": 24, "x": 0, "y": 24 },
|
|
"targets": [
|
|
{
|
|
"expr": "sum(rate(locust_request_duration_milliseconds_bucket{name=\"ResumeActor\", user_class=\"AteAPIUser\"}[1m])) by (le)",
|
|
"refId": "A"
|
|
}
|
|
],
|
|
|
|
"transformations": [
|
|
{
|
|
"id": "labelsToFields",
|
|
"options": {
|
|
"valueLabel": "le"
|
|
}
|
|
}
|
|
],
|
|
"options": {
|
|
"calculate": false,
|
|
"yAxis": {
|
|
"unit": "ms"
|
|
}
|
|
}
|
|
},
|
|
{
|
|
"title": "SuspendActor QPS",
|
|
"type": "timeseries",
|
|
"gridPos": { "h": 8, "w": 12, "x": 0, "y": 32 },
|
|
"targets": [
|
|
{ "expr": "sum(rate(locust_requests_total_total{name=\"SuspendActor\", user_class=\"AteAPIUser\"}[1m])) or sum(rate(locust_requests_total{name=\"SuspendActor\", user_class=\"AteAPIUser\"}[1m]))", "refId": "A", "legendFormat": "Total" },
|
|
{ "expr": "sum(rate(locust_requests_total_total{name=\"SuspendActor\", user_class=\"AteAPIUser\"}[1m])) by (status) or sum(rate(locust_requests_total{name=\"SuspendActor\", user_class=\"AteAPIUser\"}[1m])) by (status)", "refId": "B", "legendFormat": "{{status}}" }
|
|
],
|
|
"fieldConfig": { "defaults": { "unit": "req/sec" } }
|
|
},
|
|
{
|
|
"title": "SuspendActor Latency 99th Pct",
|
|
"type": "timeseries",
|
|
"gridPos": { "h": 8, "w": 12, "x": 12, "y": 32 },
|
|
"targets": [
|
|
{ "expr": "histogram_quantile(0.99, sum(rate(locust_request_duration_milliseconds_bucket{name=\"SuspendActor\", user_class=\"AteAPIUser\"}[1m])) by (le))", "refId": "A", "legendFormat": "Total" },
|
|
{ "expr": "histogram_quantile(0.99, sum(rate(locust_request_duration_milliseconds_bucket{name=\"SuspendActor\", user_class=\"AteAPIUser\"}[1m])) by (le, status))", "refId": "B", "legendFormat": "{{status}}" }
|
|
],
|
|
"fieldConfig": { "defaults": { "unit": "ms" } }
|
|
},
|
|
{
|
|
"title": "SuspendActor Latency Heatmap",
|
|
"type": "heatmap",
|
|
"gridPos": { "h": 8, "w": 24, "x": 0, "y": 40 },
|
|
"targets": [
|
|
{
|
|
"expr": "sum(rate(locust_request_duration_milliseconds_bucket{name=\"SuspendActor\", user_class=\"AteAPIUser\"}[1m])) by (le)",
|
|
"refId": "A"
|
|
}
|
|
],
|
|
|
|
"transformations": [
|
|
{
|
|
"id": "labelsToFields",
|
|
"options": {
|
|
"valueLabel": "le"
|
|
}
|
|
}
|
|
],
|
|
"options": {
|
|
"calculate": false,
|
|
"yAxis": {
|
|
"unit": "ms"
|
|
}
|
|
}
|
|
},
|
|
{
|
|
"title": "Locust Requesters (Active Users)",
|
|
"type": "timeseries",
|
|
"gridPos": { "h": 8, "w": 24, "x": 0, "y": 48 },
|
|
"targets": [
|
|
{ "expr": "sum(locust_users{user_class=\"AteAPIUser\"}) by (pod)", "refId": "A" }
|
|
]
|
|
}
|
|
],
|
|
|
|
"schemaVersion": 38,
|
|
"version": 1
|
|
}
|
|
locust-counter-dashboard.json: |-
|
|
{
|
|
"uid": "locust-counter-demo-dashboard",
|
|
"title": "Counter Demo",
|
|
"panels": [
|
|
{
|
|
"title": "ResumeActor QPS",
|
|
"type": "timeseries",
|
|
"gridPos": { "h": 8, "w": 12, "x": 0, "y": 0 },
|
|
"targets": [
|
|
{ "expr": "sum(rate(locust_requests_total_total{name=\"ResumeActor\", user_class=\"CounterUser\"}[1m])) or sum(rate(locust_requests_total{name=\"ResumeActor\", user_class=\"CounterUser\"}[1m]))", "refId": "A", "legendFormat": "Total" },
|
|
{ "expr": "sum(rate(locust_requests_total_total{name=\"ResumeActor\", user_class=\"CounterUser\"}[1m])) by (status) or sum(rate(locust_requests_total{name=\"ResumeActor\", user_class=\"CounterUser\"}[1m])) by (status)", "refId": "B", "legendFormat": "{{status}}" }
|
|
],
|
|
"fieldConfig": { "defaults": { "unit": "req/sec" } }
|
|
},
|
|
{
|
|
"title": "ResumeActor Latency 99th Pct",
|
|
"type": "timeseries",
|
|
"gridPos": { "h": 8, "w": 12, "x": 12, "y": 0 },
|
|
"targets": [
|
|
{ "expr": "histogram_quantile(0.99, sum(rate(locust_request_duration_milliseconds_bucket{name=\"ResumeActor\", user_class=\"CounterUser\"}[1m])) by (le))", "refId": "A", "legendFormat": "Total" },
|
|
{ "expr": "histogram_quantile(0.99, sum(rate(locust_request_duration_milliseconds_bucket{name=\"ResumeActor\", user_class=\"CounterUser\"}[1m])) by (le, status))", "refId": "B", "legendFormat": "{{status}}" }
|
|
],
|
|
"fieldConfig": { "defaults": { "unit": "ms" } }
|
|
},
|
|
{
|
|
"title": "ResumeActor Latency Heatmap",
|
|
"type": "heatmap",
|
|
"gridPos": { "h": 8, "w": 24, "x": 0, "y": 8 },
|
|
"targets": [
|
|
{
|
|
"expr": "sum(rate(locust_request_duration_milliseconds_bucket{name=\"ResumeActor\", user_class=\"CounterUser\"}[1m])) by (le)",
|
|
"refId": "A"
|
|
}
|
|
],
|
|
"transformations": [
|
|
{
|
|
"id": "labelsToFields",
|
|
"options": {
|
|
"valueLabel": "le"
|
|
}
|
|
}
|
|
],
|
|
"options": {
|
|
"calculate": false,
|
|
"yAxis": {
|
|
"unit": "ms"
|
|
}
|
|
}
|
|
},
|
|
{
|
|
"title": "RunCounter QPS",
|
|
"type": "timeseries",
|
|
"gridPos": { "h": 8, "w": 12, "x": 0, "y": 16 },
|
|
"targets": [
|
|
{ "expr": "sum(rate(locust_requests_total_total{name=\"RunCounter\", user_class=\"CounterUser\"}[1m])) or sum(rate(locust_requests_total{name=\"RunCounter\", user_class=\"CounterUser\"}[1m]))", "refId": "A", "legendFormat": "Total" },
|
|
{ "expr": "sum(rate(locust_requests_total_total{name=\"RunCounter\", user_class=\"CounterUser\"}[1m])) by (status) or sum(rate(locust_requests_total{name=\"RunCounter\", user_class=\"CounterUser\"}[1m])) by (status)", "refId": "B", "legendFormat": "{{status}}" }
|
|
],
|
|
"fieldConfig": { "defaults": { "unit": "req/sec" } }
|
|
},
|
|
{
|
|
"title": "RunCounter Latency 99th Pct",
|
|
"type": "timeseries",
|
|
"gridPos": { "h": 8, "w": 12, "x": 12, "y": 16 },
|
|
"targets": [
|
|
{ "expr": "histogram_quantile(0.99, sum(rate(locust_request_duration_milliseconds_bucket{name=\"RunCounter\", user_class=\"CounterUser\"}[1m])) by (le))", "refId": "A", "legendFormat": "Total" },
|
|
{ "expr": "histogram_quantile(0.99, sum(rate(locust_request_duration_milliseconds_bucket{name=\"RunCounter\", user_class=\"CounterUser\"}[1m])) by (le, status))", "refId": "B", "legendFormat": "{{status}}" }
|
|
],
|
|
"fieldConfig": { "defaults": { "unit": "ms" } }
|
|
},
|
|
{
|
|
"title": "RunCounter Latency Heatmap",
|
|
"type": "heatmap",
|
|
"gridPos": { "h": 8, "w": 24, "x": 0, "y": 24 },
|
|
"targets": [
|
|
{
|
|
"expr": "sum(rate(locust_request_duration_milliseconds_bucket{name=\"RunCounter\", user_class=\"CounterUser\"}[1m])) by (le)",
|
|
"refId": "A"
|
|
}
|
|
],
|
|
"transformations": [
|
|
{
|
|
"id": "labelsToFields",
|
|
"options": {
|
|
"valueLabel": "le"
|
|
}
|
|
}
|
|
],
|
|
"options": {
|
|
"calculate": false,
|
|
"yAxis": {
|
|
"unit": "ms"
|
|
}
|
|
}
|
|
},
|
|
{
|
|
"title": "SuspendActor QPS",
|
|
"type": "timeseries",
|
|
"gridPos": { "h": 8, "w": 12, "x": 0, "y": 32 },
|
|
"targets": [
|
|
{ "expr": "sum(rate(locust_requests_total_total{name=\"SuspendActor\", user_class=\"CounterUser\"}[1m])) or sum(rate(locust_requests_total{name=\"SuspendActor\", user_class=\"CounterUser\"}[1m]))", "refId": "A", "legendFormat": "Total" },
|
|
{ "expr": "sum(rate(locust_requests_total_total{name=\"SuspendActor\", user_class=\"CounterUser\"}[1m])) by (status) or sum(rate(locust_requests_total{name=\"SuspendActor\", user_class=\"CounterUser\"}[1m])) by (status)", "refId": "B", "legendFormat": "{{status}}" }
|
|
],
|
|
"fieldConfig": { "defaults": { "unit": "req/sec" } }
|
|
},
|
|
{
|
|
"title": "SuspendActor Latency 99th Pct",
|
|
"type": "timeseries",
|
|
"gridPos": { "h": 8, "w": 12, "x": 12, "y": 32 },
|
|
"targets": [
|
|
{ "expr": "histogram_quantile(0.99, sum(rate(locust_request_duration_milliseconds_bucket{name=\"SuspendActor\", user_class=\"CounterUser\"}[1m])) by (le))", "refId": "A", "legendFormat": "Total" },
|
|
{ "expr": "histogram_quantile(0.99, sum(rate(locust_request_duration_milliseconds_bucket{name=\"SuspendActor\", user_class=\"CounterUser\"}[1m])) by (le, status))", "refId": "B", "legendFormat": "{{status}}" }
|
|
],
|
|
"fieldConfig": { "defaults": { "unit": "ms" } }
|
|
},
|
|
{
|
|
"title": "SuspendActor Latency Heatmap",
|
|
"type": "heatmap",
|
|
"gridPos": { "h": 8, "w": 24, "x": 0, "y": 40 },
|
|
"targets": [
|
|
{
|
|
"expr": "sum(rate(locust_request_duration_milliseconds_bucket{name=\"SuspendActor\", user_class=\"CounterUser\"}[1m])) by (le)",
|
|
"refId": "A"
|
|
}
|
|
],
|
|
"transformations": [
|
|
{
|
|
"id": "labelsToFields",
|
|
"options": {
|
|
"valueLabel": "le"
|
|
}
|
|
}
|
|
],
|
|
"options": {
|
|
"calculate": false,
|
|
"yAxis": {
|
|
"unit": "ms"
|
|
}
|
|
}
|
|
},
|
|
{
|
|
"title": "Locust Requesters (Active Users)",
|
|
"type": "timeseries",
|
|
"gridPos": { "h": 8, "w": 24, "x": 0, "y": 48 },
|
|
"targets": [
|
|
{ "expr": "sum(locust_users{user_class=\"CounterUser\"}) by (pod)", "refId": "A" }
|
|
]
|
|
}
|
|
],
|
|
"schemaVersion": 38,
|
|
"version": 1
|
|
}
|
|
locust-workloads-dashboard.json: |-
|
|
{
|
|
"uid": "locust-workloads-dashboard",
|
|
"title": "Workloads Benchmarking",
|
|
"panels": [
|
|
{
|
|
"title": "QPS by Workload",
|
|
"type": "timeseries",
|
|
"gridPos": { "h": 8, "w": 12, "x": 0, "y": 0 },
|
|
"targets": [
|
|
{ "expr": "sum(rate(locust_requests_total_total{user_class=~\"SleepUser|UserMemUser|KernelMemUser\"}[1m])) by (user_class, name) or sum(rate(locust_requests_total{user_class=~\"SleepUser|UserMemUser|KernelMemUser\"}[1m])) by (user_class, name)", "refId": "A", "legendFormat": "{{user_class}} - {{name}}" }
|
|
],
|
|
"fieldConfig": { "defaults": { "unit": "req/sec" } }
|
|
},
|
|
{
|
|
"title": "Latency 99th Pct by Workload",
|
|
"type": "timeseries",
|
|
"gridPos": { "h": 8, "w": 12, "x": 12, "y": 0 },
|
|
"targets": [
|
|
{ "expr": "histogram_quantile(0.99, sum(rate(locust_request_duration_milliseconds_bucket{user_class=~\"SleepUser|UserMemUser|KernelMemUser\"}[1m])) by (le, user_class, name))", "refId": "A", "legendFormat": "{{user_class}} - {{name}}" }
|
|
],
|
|
"fieldConfig": { "defaults": { "unit": "ms" } }
|
|
},
|
|
{
|
|
"title": "Active Users by Workload",
|
|
"type": "timeseries",
|
|
"gridPos": { "h": 8, "w": 24, "x": 0, "y": 8 },
|
|
"targets": [
|
|
{ "expr": "sum(locust_users{user_class=~\"SleepUser|UserMemUser|KernelMemUser\"}) by (user_class)", "refId": "A", "legendFormat": "{{user_class}}" }
|
|
]
|
|
}
|
|
],
|
|
"schemaVersion": 38,
|
|
"version": 1
|
|
}
|
|
---
|
|
apiVersion: apps/v1
|
|
kind: Deployment
|
|
metadata:
|
|
name: grafana
|
|
namespace: benchmarking
|
|
spec:
|
|
replicas: 1
|
|
selector:
|
|
matchLabels:
|
|
app: grafana
|
|
template:
|
|
metadata:
|
|
labels:
|
|
app: grafana
|
|
spec:
|
|
containers:
|
|
- name: grafana
|
|
image: grafana/grafana:10.0.0
|
|
env:
|
|
- name: GF_AUTH_ANONYMOUS_ENABLED
|
|
value: "true"
|
|
- name: GF_AUTH_ANONYMOUS_ORG_ROLE
|
|
value: "Admin"
|
|
- name: GF_AUTH_DISABLE_LOGIN_FORM
|
|
value: "true"
|
|
ports:
|
|
- containerPort: 3000
|
|
volumeMounts:
|
|
- name: grafana-datasources
|
|
mountPath: /etc/grafana/provisioning/datasources
|
|
- name: grafana-dashboards-provider
|
|
mountPath: /etc/grafana/provisioning/dashboards
|
|
- name: grafana-dashboards
|
|
mountPath: /var/lib/grafana/dashboards
|
|
volumes:
|
|
- name: grafana-datasources
|
|
configMap:
|
|
name: grafana-datasources
|
|
- name: grafana-dashboards-provider
|
|
configMap:
|
|
name: grafana-dashboards-provider
|
|
- name: grafana-dashboards
|
|
configMap:
|
|
name: grafana-dashboards
|
|
---
|
|
apiVersion: v1
|
|
kind: Service
|
|
metadata:
|
|
name: grafana
|
|
namespace: benchmarking
|
|
spec:
|
|
selector:
|
|
app: grafana
|
|
ports:
|
|
- protocol: TCP
|
|
port: 3000
|
|
targetPort: 3000
|