mirror of
https://github.com/agent-substrate/substrate.git
synced 2026-10-02 03:24:42 +08:00
Tim Bai
721152f14c
Part of #896 (Phase 1 of #550): the events channel — per-actor usage samples as structured log events. With this, both halves of #174's cardinality split exist: template-level metrics in the TSDB (#961), and everything carrying actor/atespace identity here, in the log store. ## The events One JSON record per executing actor per sweep on atelet's stdout, riding the poller's existing probe — no extra RPC load, and one knob governs both channels (`--actor-stats-poll-interval 0` disables the subsystem). Idle workers emit nothing: an idle fleet is silent by design. Records use the same label vocabulary as actorlog's lifecycle events (`ateattr.ActorLogLabels`, including the GCE `logging.googleapis.com/labels` spelling) — so one Cloud Logging filter on `labels."ate.actor.uid"` returns an actor's container logs, lifecycle transitions, and usage samples interleaved. Identity comes solely from each sample's echo, per the stats RPCs' attribution contract. Measurements ride as payload fields (`kind`, class, source, the four numbers, `observed_at_unix_nano`); the `kind` field (today always `periodic`) lets future kinds join without reshaping the record. Events also carry the `ate.workerpool.namespace/name` pair the metric labels already carry, resolved by the sweep's own pod list, so a pool-level metric spike can pivot to the actors behind it — pool membership lives on the worker pod and is unrecoverable from logs once the pod is gone, so it must be stamped at emission. An unresolved pod emits without the pair, following the metric channel's rule. ## Scope Earlier revisions also took first/final lifecycle samples from the Run/Restore/Checkpoint handlers. Per the review discussion on suspend/resume latency, those are descoped from this PR: it now touches no lifecycle path at all. The bracket design (including taking the final sample inside ateom's CheckpointWorkload and echoing it in the response) moves to a follow-up under #896. ## Isolation The poller dials its own short-lived connection per probe (`dialAteomStats`) and never touches the lifecycle RPCs' cached clients — after review on #961 the isolation is structural. ## Validated live on ate-dev Periodic events read back from Cloud Logging via `labels."ate.actor.name"`, one per executing actor per minute, with `ate.actor.uid` confirmed promoted into `LogEntry.labels` (filterable) — the claim only production could prove. Consumer note: aggregate freely in the log store, but a log-based metric built over these events must label only by the bounded set (template, class, source, pool) — promoting actor identity into a metric label would reintroduce exactly the cardinality #174 keeps out of the TSDB. Part of #896. Part of #550.