mirror of
https://github.com/agent-substrate/substrate.git
synced 2026-10-26 07:23:59 +08:00
Tim Bai
632e66221f
The ateoms now write the actor usage records and serve their latest sample (#2194, #2195), so atelet stops writing its own and counts CPU per activation. - **atelet CPU.** Baselines are keyed by actor and epoch. A new activation counts in full if it began after atelet started, and a worker that does not answer keeps its baselines. This fixes the undercount on micro-VM restores and the double charge when a silent worker's baselines were dropped. - **atelet records removed.** The ateoms' `ate.actor.usage_sampled` records replace them, ending the period with two usage records per actor. - **Timeout.** `GetActiveWorkloadStats` answers from memory, so atelet's `statsRPCTimeout` drops from 55s to 5s. The 50s poll floor stays. - **Cleanup.** `UsageEmitter.EmitFinal` and its flush, atelet's stdout `SyncedWriter`, and the exported `actorlog.LabelsKey` go; nothing uses them now. - **Docs.** The proto comments describe the cached discovery read. The registry and the guide say when the final record is written: whenever the ateom tears an activation down (checkpoint, terminate, graceful shutdown, or a micro-VM re-host), read just before the teardown. #### Known limitation and future work atelet's CPU counter only sees samples of actors still hosted, so it loses each activation's tail: the CPU from the last sample atelet read to the teardown, up to one poll interval plus one sample interval. An activation shorter than that, the common checkpoint-when-idle case, can lose most or all of its CPU in the metric. The usage records keep it, through the final record. To fix it later, the ateoms can own the counter: each ateom adds an activation's CPU increase at every reading, initial, periodic, and final, and exports it through the existing OTLP relay. No gRPC change is needed. atelet then drops the baseline map, its start time, and the silent-worker rule; this also removes the baselines a crashed worker's leftover directory keeps. The cost is cardinality: the series become one per worker pod per label set instead of one per node, still bounded by pods rather than actors, plus a short-lived series per replaced pod. A collector can drop the pod attributes if that matters. #### Breaking change The stdout usage record moves from atelet to the ateoms, with flat `ate.*` keys and the message `Actor usage sampled`. Log filters on atelet's `Actor usage sample` record stop matching: filter on `jsonPayload.msg="Actor usage sampled"` and the `ate.*` keys instead. Identity is no longer in the label group, so on GKE it does not land in `LogEntry.labels`. Part of #1748 (steps 4 to 6). - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR