mirror of https://github.com/agent-substrate/substrate.git synced 2026-10-02 03:24:42 +08:00

Files

T

Krisztian F 5f7c690108 (feat): Actor lifecycle events over otlp (#1658 )

## What this does

ateapi writes a record every time an actor changes state. Until now
those records only went to the pod's stdout, and nothing reads stdout.
This sends the same records to a collector as OTLP log events.

Actor name and uid cannot be metric labels (too many values), and traces
are sampled at 1%. So these records are the only way to answer "what
state is this actor in, and since when".

## Changes

- `serverboot.InitLogging` sets up a LoggerProvider, next to the
existing tracer and meter ones.
- New `internal/actorevent` package builds the log records.
- ateapi emits at the two places that already write the stdout records.
- Two event names: `ate.actor.state_changed` and `ate.actor.crashed`.
- Both names are registered in `docs/metrics/registry/events.yaml`, so
`make verify` checks them.
- kind gets a logs pipeline and a count connector. The e2e suite reads
the counts back.
- Docs updated. `otel-collector.md` said substrate has no
LoggerProvider, which is no longer true.

## Opt-in

`OTEL_LOGS_EXPORTER` defaults to `none`. Only the kind overlay sets it
to `otlp`. The base ConfigMap is untouched, so no deployed environment
changes when this merges.

## Notes on the design

- **No slog bridge.** Only two call sites emit these records, so
emitting twice costs two lines. A bridge would also send every ateapi
log over the wire, could not set the event name, and would loop, because
SDK export errors are logged through slog.
- **Batching processor, not the simple one.** These records sit on the
actor resume path. A processor that exports inside the emit call would
add a blocking gRPC call there, so a slow collector would become control
plane latency.
- **Two event names, not one per state.** `ate.actor.state` already says
which transition happened. A crash gets its own name because it carries
two extra attributes and a higher severity.
- **Both copies are kept on purpose.** No collector in this repo reads
pod stdout, so nothing is duplicated today. `kubectl logs` keeps
working. If a filelog agent is ever added, drop one of the two. The
escape hatch is written down in `docs/metrics/substrate.yaml`.

## Dependencies

Adds `otel/log`, `otel/sdk/log` and `otlploggrpc`, all pinned at
v0.20.0. That is the release that matches the pinned `otel v1.44.0`.
v0.21.0 would pull the core modules to v1.45.0, which this change does
not need. The logs API has a
v1.47.0 release candidate upstream, so it is on its way to stable.

## Testing

- Unit tests for the exporter resolver, the record builder, and both
ateapi emit sites.
- The record builder test checks the attribute set matches what the
event name declares, in both directions.
- The ateapi tests check the OTLP record carries the same attributes as
the stdout record.
- Ran end to end on kind. Records arrive with the right event name,
severity, attributes, and with trace context on the record's own fields
rather than as attributes.
- Checked the off state too. With `OTEL_LOGS_EXPORTER` removed, the
collector receives no log records and stdout is unchanged.

- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR

2026-09-18 18:11:12 +00:00

5.9 KiB

Raw Blame History

Agent Substrate

Project Overview

Agent Substrate is a system built on top of Kubernetes which manages agent-like workloads to achieve higher scale and efficiency than Kubernetes alone can offer, with lower latency. It takes the Kubernetes control-plane out of the critical path to achieve lower latency by mapping a larger set of “actors” (applications such as agents) onto a smaller set of ready “workers” (Kubernetes Pods). Agent Substrate relies on the fact that agent-like applications tend to be idle most of the time to achieve heavy multiplexing.

For development, it's recommended to read the README.md and CONTRIBUTING.md in the root folder. See hack/install-ate.sh and tools/setup-gcp for provisioning and deploying clusters and GCP resources.

Repository Layout

cmd/          # One subdirectory per binary (ateapi, atelet, atenet, …)
internal/     # Shared packages, internal to this module only
pkg/          # Shared packages intended for external import
docs/         # Design docs and developer guides
hack/         # Dev/CI scripts and code generators
manifests/    # Kubernetes YAML for deploying Agent Substrate
demos/        # Self-contained example applications
benchmarking/ # Load-testing tools and workloads
tools/        # Standalone Go tools (go run ./tools/<name>) for Dev/CI

Where to put new Go code, quick rules:

Situation	Location
Only used by one binary	`cmd/<binary>/internal/<pkg>`
Shared across binaries, not for external import	`internal/<pkg>`
Public API for external consumers	`pkg/<pkg>`
Public proto (control-plane gRPC API)	`pkg/proto/<name>`
Internal proto (atelet / ateom)	`internal/proto/<name>`
Dev/CI scripts	`hack/`
Standalone Go dev/CI tools	`tools/<name>` with its own `go.mod`

See docs/dev/code-layout.md for the full rationale and per-directory details.

Build and Test Commands

Agent Substrate uses a Makefile for its build and test tasks.

Building

Binaries: make build (builds images and kubectl-ate) or make build-atectl
Images: make build-images (uses ko to build container images)
Demos: make build-demos

Testing and Verification

Run Unit Tests: make test
Run E2E Tests: make e2e (Requires GCP cluster setup and built images)
Run Linters and Verifiers: make verify (Includes go vet and checks for formatting, boilerplate headers, licenses, and go modules)

Code Style Guidelines

Go Formatting: Code must be formatted with gofmt. Run make fmt to automatically format all files before submitting changes.
Copyright Headers: All files must contain appropriate copyright and license headers. See templates in hack/boilerplate/.
Modularity: Submit small, focused Pull Requests that touch a limited part of the codebase for easier reviews and rebasing.
Go Modules: Ensure go.mod is clean. Run go mod tidy if adding or removing dependencies.
Comments: Keep them brief and to the point. Comment the final state of the code, not the path taken to it — a problem that only existed partway through writing the change is noise to the next reader, as is a pointer to a scratch or planning file that isn't in the repository.
Spelling: American English. golangci-lint runs misspell with locale: US, so British spellings fail lint.

Commit Messages

Describe the change and why it was needed: The message is read on main long after the PR branch is gone, so it should stand on its own.
No issue or PR references: Leave out #1234, Fixes #1234, and GitHub URLs. GitHub renders them as cross-references on the linked thread, and rebases or force-pushes repeat them. Put that context in the pull request description instead, where it belongs to the review rather than to the permanent history.

Metrics

docs/metrics/registry/ is an OpenTelemetry Weaver registry. metrics.yaml defines every metric instrument the ate system components emit, plus every attribute any signal uses and the permitted values of each. events.yaml defines the log events emitted over OTLP. Read them to find an instrument, an event, or their attributes.

If you add or rename an instrument or an event, or add an attribute, follow docs/dev/best-practices/metrics.md, update the registry, and run hack/verify/metrics.sh. make verify runs the same check. Weaver reads the whole directory, so a new file in it is checked straight away.

Some attributes are defined in the registry but barred from metric labels: actor identity is the main one. The note on each says so, and cardinality_rules in docs/metrics/substrate.yaml is where the bar lives.

docs/metrics/substrate.yaml holds the rules Weaver cannot express: the cardinality rules, the known exceptions, and the subsystems that emit no metrics. Read blind_spots before you attribute a fault to a component.

See the metric registry section of the observability guide.

Testing Instructions

Write tests for all new code. We will not merge code that lacks tests.
Ensure changes do not break existing tests.
Run make verify locally before requesting a code review to catch common issues like missed copyright headers or formatting drift.
For end-to-end tests involving the actual infrastructure, ensure you have a running cluster (setup via hack/ate-dev-env.sh.example and go run ./tools/setup-gcp bootstrap).

Security Considerations

The security story for Substrate is very early and many features are missing. However! Take care to respect security best practices when writing code in order to improve Substrate's security over time. The following is what Substrate currently offers. Keep this up to date when updating AGENTS.md.

Workload Isolation: The project uses gVisor (runsc) for sandboxing and security isolation of workloads on pods.

For future plans for security, reference docs/roadmap.md.

5.9 KiB Raw Blame History