This PR introduces a new OpenTelemetry histogram
`"atelet.snapshot.size"` from `atelet` over the existing
OTLP path. It records the uncompressed size in bytes of each gVisor
snapshot image written during checkpoint -- `checkpoint.img` ,
`pages.img` and `pages_meta.img` -- tagged by image `kind` and
`ActorTemplate`. This surfaces how large actor snapshots are and which
`ActorTemplate`s are expensive to checkpoint/restore.
Add a well-defined dashboard for snapshot metrics that contains 3
charts.
- snapshot image size p99 by `ActorTemplate`
- snapshot image size p50/p95/p99 (all kinds)
- snapshot QPS by `ActorTemplate`
<img width="996" height="717" alt="image"
src="https://github.com/user-attachments/assets/99bbab46-37dc-476e-bfd9-cdffeb174b4b"
/>
> It's a good idea to open an issue first for discussion.
- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR
This PR introduces a new OpenTelemetry histogram
`"atenet.router.route.duration"` from `atenet-router` over the existing
OTLP path. It measures substrate overhead -- from envoy receiving a
request to envoy sending the request to the the resolved worker
endpoint, excluding actor compute and the response. This is the
`Substrate E2E latency` that under our control.
Add a well-defined dashboard for E2E metrics that contains 6 charts.
- Substrate routing latency — P50/P95/P99
- Substrate routing latency P99 - by stages
- Substrate routing latency P99 — by ActorTemplate
- Substrate routing QPS - by Status
- Substrate E2E (full round-trip) Latency — P50/P95/P99
- Substrate E2E (full round-trip) QPS — by Response Code
<img width="1601" height="894" alt="image"
src="https://github.com/user-attachments/assets/be6fc572-dfd8-4ef0-a0a9-a77f85f2ecf8"
/>
Fixes b/508613998
> It's a good idea to open an issue first for discussion.
- [ ] Tests pass
- [x] Appropriate changes to documentation are included in the PR
This commit was automatically generated by the tools/reorg tool.
```
go run ./tools/reorg
go generate ./...
bash benchmarking/locust/generate_protos.sh
```
Haven Xia