mirror of
https://github.com/agent-substrate/substrate.git
synced 2026-11-10 10:13:50 +08:00
Krisztian F
0dbe1523e4
This PR implements the last two metrics from #433. Right now, we can see that a resume was slow but not where. `ate.actor.lifecycle.operation.duration` covers the whole ateapi operation, and `atenet.router.route.duration` covers the edge, but everything between ateapi-atelet-actors is one block that contains fetching the manifest, downloading the snapshot, unpack the OCI image, call to ateom. We have `rpc.server.call.duration` that gives us the atelet restore total time, but template, kind, and scope labels are missing, so today we cannot really pinpoint why/where we have a regressions in latency. In this PR I am adding per-phase histograms, here's an example of what we can know after these changes: ```console ateom_restore 522 ms ############################### download 8.9 ms # manifest_fetch 3.2 ms oci_unpack 2.6 ms --------------------------------------------------------- total 535 ms ``` It also fixes a gap #683 opened where a `data_on_golden` resume was labeled identically to a plain one on the lifecycle histogram. Things folks might want to argue with: - total as a phase value. Partly duplicates `rpc.server.call.duration`, but that one has no domain labels and gRPC-specifc. We can drop it, but it's an inferior operational UX, so I'd rather have it here. - Phases overlap, they are not a partition of total, because download runs concurrently with the asset fetch and unpack. I called this out in the metric description. Do not sum across phases. - New `ate.snapshot.scope` key rather than a new `ate.snapshot.kind` value for `data_on_golden`. A new value would collapse local and external into one bucket, which is the biggest latency difference there is. This does add a label to the already shipped lifecycle histogram. Verified on kind, and all e2e suites pass, and the emitted series cover every kind (golden, latest, local) on both metrics with no unknown values. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR