mirror of
https://github.com/agent-substrate/substrate.git
synced 2026-10-02 03:24:42 +08:00
Da Huang
7a9bb4b1f2
OTEL_EXPORTER_OTLP_ENDPOINT is written in 9 places: hardcoded inline in 4 base workload manifests, then re-patched in 5 spots across the kind overlays. Changing the collector address means editing all 9 and knowing which install path renders which. Replace them with one checked-in ConfigMap per environment, consumed by every component via envFrom: manifests/ate-install/ate-otel-config.yaml (GKE) manifests/ate-install/kind/ate-otel-config.yaml (kind, same name) The GKE copy is listed in manifests/ate-install/base, so token-client, agentgateway and agentgateway-token-client all inherit it; the kind overlay lists its own copy of the same ConfigMap name. No overlay builds on both, so the two never collide. A kustomize-only fix does not work here. The GKE path applies the base directory raw and hack/install-ate.sh's targeted redeploys apply single files with no Kustomize, so the mechanism has to survive `kubectl apply -f <one-file>` -- which rules out configMapGenerator (hash-suffixed names) and replacements. envFrom on a stable name reaches every path. deploy_ate_system applies the ConfigMap before the rendered bundle, the same way it already applies the namespace. A container whose envFrom target is missing does not start, and a raw directory apply is ordered by filename, so ate-api-server.yaml and ate-controller.yaml would otherwise be created first and sit in CreateContainerConfigError until the ConfigMap caught up. This also fixes a latent bug in the targeted redeploys. deploy_ate_apiserver, deploy_atelet and deploy_atenet apply raw files even in kind mode, silently reverting the endpoint to the GKE value. They now apply the environment's ConfigMap via a new apply_otel_config helper, which picks the file by ATE_INSTALL_KIND rather than applying the base copy unconditionally -- applying the base one on kind would break telemetry for every component at once. atenet-router needs no special handling: sincea98f85b1its --otlp-collector-address defaults to $OTEL_EXPORTER_OTLP_ENDPOINT, so Envoy's own spans follow the ConfigMap along with everything else. OTEL_TRACES_SAMPLER deliberately stays an inline kind patch on ate-api-server rather than moving into the ConfigMap. The ConfigMap is shared by every component via envFrom, so putting parentbased_always_on there would pin the router and the rest of the control plane to 100% and undo the per-component ratios from15eecd02. Note that a ConfigMap edit does not roll the consuming pods the way an inline env change did, since the pod template is unchanged. Callers must follow a change with `kubectl rollout restart`. This is documented in both ConfigMaps, in docs/observability.md, and in the tracing best practices, which previously told authors to hardcode the variable. Fixes #745 > It's a good idea to open an issue first for discussion. - [ ] Tests pass - [ ] Appropriate changes to documentation are included in the PR