mirror of
https://github.com/agent-substrate/substrate.git
synced 2026-11-12 10:35:06 +08:00
Jeff LuoandKrisztian F
9bfc67e815
Part of #563 Substrate had no way to measure how much telemetry it sends. This PR adds one, and builds it from the parts the benchmark suite already has. The managed collector sends its self-metrics to Cloud Monitoring only, and gives no data for each source. Thus you cannot see which component makes the volume. A second collector, the telemetry meter, counts the spans and the datapoints by `service.name`. ## Contents - **`benchmarking/telemetry/meter.yaml`** — the meter. A tee: it counts the telemetry by service, then sends it on to the managed collector. Thus one run gives the volume for each service and the cost of the load together. - **`benchmarking/telemetry/README.md`** — the method: install, measure, read, remove. - **`hack/install-ate.sh --otlp-endpoint`** — sends all control plane telemetry to the meter. Each component reads the endpoint from the `ate-otel-config` ConfigMap through `envFrom`, and `ate-controller` copies the value to the ateom worker pods, thus one patch is sufficient for all of them. - **`benchmarking/workloads/deploy.sh --otlp-endpoint`** — the same for actor containers, which substrate configures separately. - **`benchmarking/monitoring.yaml`** — scrapes the meter and cAdvisor. - **`benchmarking/observability.md`** — prerequisites and the scenario ladder. - **`docs/dev/best-practices/otel-collector.md`** — the volume model and the size guidance, corrected for the `ParentBased` defaults from #711. ## Correction The `glutton` ActorTemplate had no `env`. Substrate puts no OTLP configuration in an actor container, thus the exporter used the SDK default `localhost:4317`, no process listens on that address in the sandbox, and substrate discarded each actor span with no error message. The template now sets `OTEL_EXPORTER_OTLP_ENDPOINT` and `OTEL_RESOURCE_ATTRIBUTES`. `ActorTemplate.spec` is immutable, thus `deploy.sh --deploy` deletes the templates before it applies them. ## How I verified this On a GKE cluster with managed OTel. The sequence: ```bash # 1. Start from a clean state. ./hack/install-ate.sh --delete-benchmarks ./hack/install-ate.sh --delete-ate-system # 2. Install substrate from this branch. ./hack/install-ate.sh --deploy-ate-system # 3. Install the meter and Prometheus. kubectl apply -f benchmarking/telemetry/meter.yaml kubectl apply -f benchmarking/monitoring.yaml # 4. Send the control plane to the meter. METER=http://telemetry-meter.benchmarking.svc.cluster.local:4317 ./hack/install-ate.sh --deploy-ate-system --otlp-endpoint "${METER}" # 5. Install the workloads and locust, with the actors on the meter also. ./benchmarking/deploy_locust.sh --deploy --worker-count 10 \ --otlp-endpoint "${METER}" # 6. Read the volume. kubectl port-forward -n benchmarking svc/prometheus 9090:9090 ``` ## Result The idle floor, from one measurement on 10 workers: | Service | datapoints/min | |---|---| | `ate-controller` | 234.8 | | `ateapi` | 54.3 | | `atelet` | 44.9 | | `atenet-router` | 10.4 | | `ateom-gvisor` | 5.2 | | `glutton-actor` | 4.2 | | **Total** | **353.7** | The `glutton-actor` row is the correction above. Before it, actor telemetry could not arrive at all. Positive control for the same window: `otelcol_receiver_accepted_metric_points` 10.5/s, and each `otelcol_receiver_refused_*` counter at zero. Thus the zero values in the table are true zeros, and not data that did not arrive. Two values differ from the earlier manual measurement. `ate-controller` is now the largest source, and `ateom-gvisor` is no longer zero. The new imagecache counters from #831 are a probable cause. This difference is the reason to keep results out of the repository. ### Traces Trace volume follows the root sample decision, not the rate of each component. `atelet` and `ateom-gvisor` sent zero spans in each run, because they inherit the decision that `ateapi` makes and start no trace of their own. At idle each component is near zero: a trace needs an operation, but a metric pushes on a timer. This run gives no span volume for a usual load. The actors crashed on suspend, and the failed retries made an artificial rate of 13.7k spans/s. That rate is not a measurement of substrate. It does show that the meter accepted the spans with no refusals. ## Follow-up A separate PR adds the ladder to `automation/tests.yaml` and writes the results as artifacts of each run. The open items are at the end of `observability.md`. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR --------- Co-authored-by: Krisztian F <103492698+krisztianfekete@users.noreply.github.com>