mirror of
https://github.com/agent-substrate/substrate.git
synced 2026-11-12 10:35:06 +08:00
Haven Xia
a325d4118b
This PR introduces a new OpenTelemetry histogram `"atenet.router.route.duration"` from `atenet-router` over the existing OTLP path. It measures substrate overhead -- from envoy receiving a request to envoy sending the request to the the resolved worker endpoint, excluding actor compute and the response. This is the `Substrate E2E latency` that under our control. Add a well-defined dashboard for E2E metrics that contains 6 charts. - Substrate routing latency — P50/P95/P99 - Substrate routing latency P99 - by stages - Substrate routing latency P99 — by ActorTemplate - Substrate routing QPS - by Status - Substrate E2E (full round-trip) Latency — P50/P95/P99 - Substrate E2E (full round-trip) QPS — by Response Code <img width="1601" height="894" alt="image" src="https://github.com/user-attachments/assets/be6fc572-dfd8-4ef0-a0a9-a77f85f2ecf8" /> Fixes b/508613998 > It's a good idea to open an issue first for discussion. - [ ] Tests pass - [x] Appropriate changes to documentation are included in the PR
Cloud Monitoring dashboards
Google Cloud Monitoring dashboard definitions for ATE. They turn the raw
prometheus.googleapis.com/... metrics that ATE emits into readable
per-method / per-stage latency / throughput / error views.
| File | Shows |
|---|---|
ate-grpc-dashboard.json |
ateapi & atelet gRPC latency (p50/p95/p99), request rate, and error rate, by method |
ate-e2e-latency-dashboard.json |
"Substrate Routing & E2E Latency". Substrate routing P50/P95/P99, routing P99 by stage (routing / ateapi ResumeActor / atelet Restore), routing P99 by ActorTemplate, routing QPS by status, plus the E2E full round-trip from Envoy (ms — includes actor compute + response, so it's context, not our overhead): P99 and QPS by response class. Needs the atenet-router-envoy PodMonitoring for the round-trip lines. |
Applying
Dashboards are created/updated (idempotently) by setup:
go run ./tools/setup-gcp --create-monitoring-dashboards # also part of: --all
Or apply any single file by hand:
gcloud monitoring dashboards create --config-from-file=monitoring/dashboards/<file>.json