mirror of
https://github.com/agent-substrate/substrate.git
synced 2026-11-11 10:23:17 +08:00
Haven Xia
67bb44d090
This PR introduces a new OpenTelemetry histogram `"atelet.snapshot.size"` from `atelet` over the existing OTLP path. It records the uncompressed size in bytes of each gVisor snapshot image written during checkpoint -- `checkpoint.img` , `pages.img` and `pages_meta.img` -- tagged by image `kind` and `ActorTemplate`. This surfaces how large actor snapshots are and which `ActorTemplate`s are expensive to checkpoint/restore. Add a well-defined dashboard for snapshot metrics that contains 3 charts. - snapshot image size p99 by `ActorTemplate` - snapshot image size p50/p95/p99 (all kinds) - snapshot QPS by `ActorTemplate` <img width="996" height="717" alt="image" src="https://github.com/user-attachments/assets/99bbab46-37dc-476e-bfd9-cdffeb174b4b" /> > It's a good idea to open an issue first for discussion. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR
Cloud Monitoring dashboards
Google Cloud Monitoring dashboard definitions for ATE. They turn the raw
prometheus.googleapis.com/... metrics that ATE emits into readable
per-method / per-stage latency / throughput / error views.
| File | Shows |
|---|---|
ate-grpc-dashboard.json |
ateapi & atelet gRPC latency (p50/p95/p99), request rate, and error rate, by method |
ate-e2e-latency-dashboard.json |
"Substrate Routing & E2E Latency". Substrate routing P50/P95/P99, routing P99 by stage (routing / ateapi ResumeActor / atelet Restore), routing P99 by ActorTemplate, routing QPS by status, plus the E2E full round-trip from Envoy (ms — includes actor compute + response, so it's context, not our overhead): P99 and QPS by response class. Needs the atenet-router-envoy PodMonitoring for the round-trip lines. |
ate-snapshot-dashboard.json |
Substrate snapshot image size and throughput ("Substrate Snapshot Size & QPS"): snapshot (pages) image size P99 by ActorTemplate, P50/P95/P99 by image kind, and snapshot QPS. Checkpoint/Restore latency is not here — it's the atelet gRPC Checkpoint/Restore methods in ate-grpc-dashboard.json. |
Applying
Dashboards are created/updated (idempotently) by setup:
go run ./tools/setup-gcp --create-monitoring-dashboards # also part of: --all
Or apply any single file by hand:
gcloud monitoring dashboards create --config-from-file=monitoring/dashboards/<file>.json