atenet: report "unknown" when a resume does not complete (#1482)

Fixes #1474

`atenet.router.route.duration` carries two labels that the router filled
in wrongly on a failure path.

## `ate.router.resume` said "none" for a failed resume

`none` means "the resume found the actor already running" — the warm
route. The router also gave `none` to every failed resume, every
canceled caller, and every caller shed by a full parking lot. Those
failures landed in the warm-route series, so they did not show in the
cold-start rate and they polluted the warm-route latency.

A gRPC code cannot recover the fact. A canceled leader's flight outlives
its request and keeps restoring the actor, and a `DeadlineExceeded` can
land in the middle of a restore. So the PR does not classify the error.
It reserves `none`, `triggered` and `joined` for a resume that
completed, and adds a fourth value, `unknown`, for every resume that did
not:

| Case | Before | After |
|---|---|---|
| Actor already running | `none` | `none` |
| Leader completed a cold activation | `triggered` | `triggered` |
| Joiner waited on a completed cold activation | `joined` | `joined` |
| Resume failed (leader and joiners) | `none` | `unknown` |
| Caller canceled before the flight finished | `none` | `unknown` |
| Caller shed by a full parking lot | `none` | `unknown` |
| Egress, which never resumes an actor | `none` | `unknown` |

## `ate.template.*` was the empty string on a failure

The registry marks `ate.template.atespace` and `ate.template.name`
required on this metric, but the router sent an empty string when it
failed before it resolved a template.
`ateattr.NormalizeTemplateDimension` now sends the constant `"unknown"`
instead. The constant does not come from the request, thus it adds no
caller-controlled label value — the cardinality rule in
`docs/metrics/substrate.yaml` still holds, and the PR records the
exception there.

## How to read the change on a dashboard

A query that already splits by `ate.router.resume` gains an `unknown`
series and loses the failures that used to hide inside `none`. A `none`
rate reads lower after the change and a cold-start rate is unaffected.
Do not read the time in the `unknown` series as an activation time.

## Documentation

- `docs/metrics/registry/metrics.yaml`: the `unknown` member of
`ate.router.resume`, the `"unknown"` fallback on the two template
labels, and a note on the metric.
- `docs/metrics/substrate.yaml`: the cardinality-rule exception.
- `docs/observability.md`: the label list for the metric.

## Tests

- `resumer_test.go`: a failed flight gives `unknown` to the leader and
to all nine joiners; a canceled caller gives `unknown`; a shed caller
and a shed joiner give `unknown`.
- `metrics_test.go`: empty template dimensions record as `"unknown"`;
`Result.resume()` defaults to `unknown`.
- `ateattr_test.go`: `NormalizeTemplateDimension` and the four
`RouterResume*` values.

- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR

docs/metrics/registry/metrics.yaml

@@ -160,18 +160,28 @@ groups:
     brief: >
       The identity of an ActorTemplate. The values are not limited to a list.
       But an operator makes each template. No value comes from a request. Thus
-      the number of values stays small.
+      the number of values stays small. One metric,
+      atenet.router.route.duration, also reports the literal "unknown" when it
+      has no template to name.
     attributes:
       - id: ate.template.atespace
         stability: development
         type: string
-        brief: The atespace of the ActorTemplate of the actor.
-        examples: [ate-demo-counter]
+        brief: >
+          The atespace of the ActorTemplate of the actor. On
+          atenet.router.route.duration only, the value is "unknown" when the
+          direction has no template, or the request failed before the router
+          resolved one.
+        examples: [ate-demo-counter, unknown]
       - id: ate.template.name
         stability: development
         type: string
-        brief: The name of the ActorTemplate of the actor.
-        examples: [counter]
+        brief: >
+          The name of the ActorTemplate of the actor. On
+          atenet.router.route.duration only, the value is "unknown" when the
+          direction has no template, or the request failed before the router
+          resolved one.
+        examples: [counter, unknown]
 
   - id: registry.ate.workerpool
     type: attribute_group
@@ -504,16 +514,26 @@ groups:
               stability: development
               value: triggered
               brief: >
-                This request got the singleflight lock and did the resume. Its
-                time is the activation time. Its rate is the number of resumes
-                each second.
+                This request got the singleflight lock and completed the resume.
+                Its time is the activation time. Its rate is the number of
+                resumes each second.
             - id: joined
               stability: development
               value: joined
               brief: >
-                A resume was already in operation. This request waited for it.
-                This value is separate because it is not a correct sample. One
-                cold start with 50 requests is one slow activation and not 51.
+                A resume was already in operation. This request waited for it,
+                and the resume activated the actor. This value is separate
+                because it is not a correct sample. One cold start with 50
+                requests is one slow activation and not 51.
+            - id: unknown
+              stability: development
+              value: unknown
+              brief: >
+                The resume did not complete, thus the router cannot tell whether
+                an activation ran. The resume failed, or the request stopped
+                before the resume gave an answer, or the direction does not
+                resume an actor. Do not read the time in this series as an
+                activation time.
       - id: ate.router.outcome
         stability: development
         brief: The result of the route attempt.
@@ -1191,7 +1211,9 @@ groups:
       This is the measurement at the user boundary. Divide the data by
       ate.router.resume before you read it. The triggered series is the
       activation time. The none series is the usual warm route. A sum across the
-      two series puts milliseconds and tens of seconds in one distribution.
+      two series puts milliseconds and tens of seconds in one distribution. The
+      unknown series holds the resumes that did not complete, thus it is neither
+      an activation time nor a warm route.
     annotations:
       substrate:
         emitted_by: [atenet-router]

docs/metrics/substrate.yaml

@@ -151,7 +151,11 @@ cardinality_rules:
     brief: >
       Each ate.* metric label has a list of permitted values, or it names an
       object that an operator made. The names of the templates and the pools
-      never come from a request.
+      never come from a request. One exception: on
+      atenet.router.route.duration, ate.template.atespace and ate.template.name
+      hold the literal "unknown" when the router has no template to name. The
+      router sends that constant in place of the empty string. It does not send
+      a value that the request gave.
     enforced: false
     could_be_enforced_by: >
       A Weaver Rego policy can make each ate.* attribute of type string give a

docs/observability.md

@@ -267,7 +267,7 @@ For `ate.workerpool.desired_workers` and `ate.workerpool.ready_workers`:
 
 For `atenet.router.route.duration`:
 * `ate.router.outcome` categorizes the route attempt result: `ok`, `cancelled`, `timeout`, `no_capacity`, `failed_precondition`, `lock_conflict`, `not_found`, `unavailable`, `rate_limited`, or `resume_error`.
-* `ate.router.resume` indicates the singleflight execution state of actor resumption: `none` (actor already running), `triggered` (initiated cold activation), or `joined` (parked on in-flight activation).
+* `ate.router.resume` indicates the singleflight execution state of actor resumption: `none` (the resume found the actor already running), `triggered` (this request completed a cold activation), `joined` (this request waited on another request's resume, which activated the actor), or `unknown` (the resume did not complete, so whether an activation ran is unknown). `ate.template.atespace` and `ate.template.name` hold `unknown` when the router has no template to name.
 
 For `ate.scheduler.eligible_workers`:
 * `ate.scheduling.constraint` categorizes the scheduling request constraint type: `none` (unconstrained), `selector` (actor or template label selectors specified), or `required_nodes` (pinned to specific node VMs).