mirror of https://github.com/agent-substrate/substrate.git synced 2026-11-12 10:35:06 +08:00

Files

T

Sairaj Pokale ba45517fa0 benchmarks: durdir benchmark that evaluates the efficacy and performance of DurDir (#907 )

# Benchmark DurDir

Fixes #673

Adds a `DurDirUser` load-generation workload that exercises the
DurableDir
suspend/resume loop end to end, verifies every served byte against a
SHA-256
digest, and emits separable latency percentiles for each step.

## The loop

per VU, first iteration: create -> resume -> WriteDisk ->
ReadDisk+verify
steady state: suspend -> resume -> ReadDisk+verify (cold)
                                      -> ReadDisk+verify (warm)
                                      -> WriteDisk (overwrite, TRUNCATE)

Under `onCommit: Data` the container cold-boots from the OCI image and
process
memory is discarded, so a matching digest after resume can only have
come from
the restored DurableDir. That is the durability assertion.

## Results

All six scenarios ran on GKE/gvisor at 1 VU for 1m each: `Data` vs
`Full`
snapshot scope, explicit vs implicit resume, and a 5/10/64 MiB size
sweep.
**Zero failures across every run**, so every served byte matched its
digest in
all six.

**Snapshot growth over repeated overwrites:** `SuspendActor` latency
stayed flat
across consecutive overwrite-and-suspend cycles on the same DurableDir
volume.
The loop overwrites with `WRITE_MODE_TRUNCATE`, so the file is exactly X
bytes
after every write and the captured directory contents are the same size
every
cycle. Nothing accumulates across suspends. That is the growth question
the
issue asks about.

Latency percentiles per step are in the run artifacts.

## Change surface

- **glutton:** `WriteDisk` returns size + sha256; new `ReadDisk` with a
`READ_MODE_DIGEST_ONLY` mode for measuring restore cost without paying
wire
  transfer; disk RPCs exposed over HTTP mode.
- **manifests:** two new ActorTemplates, `glutton-durdir-{data,full}`,
with a
  `durableDir` volume and `onPause: Full` / `onCommit: {Data,Full}`.
- **boomer:** shared actor-lifecycle plumbing extracted from the ping
task, then
  a `DurDirUser` task on top of it; a general `resume_mode` knob.
- **harnesses:** `--workload` selects the task at deploy time;
`durdir.py` stub,
  typed dynconfig flags, six nightly scenarios.

The two boomer commits above are incremental extractions made as the
second
workload landed. The final package layout for
`internal/benchmarking/boomer`
lands as a follow-up PR.

## Testing

- `go build ./...`, `go test -race ./cmd/benchmarking/...
./internal/benchmarking/...`
  clean, no race warnings.
- `hack/verify-all.sh`: all nine checks pass. Python protos regenerated,
tree
  clean.
- Both ActorTemplates reach `Ready`; golden snapshots confirmed in the
bucket.
- **Manual durability proof:** 200 MiB payload written, suspended,
resumed, and
re-read with matching sha256 across every read. Process memory discarded
and
  container cold-booted in between.
- **Scale validation:** DurableDir persistence verified up to 1 GiB with
0
  failures.
- **Regression gate:** `glutton_baseline_5_users` ran 919 requests with
0
failures and 7 ms ping latency post-rebase. No regression on the
existing
  benchmark.

## Deliberately out of scope

- **Image size over time:** `ActorSnapshot` has no size field, so there
is
nothing for a client to read. The issue permits deferring this.
`SuspendActor`
latency is the available proxy; `atelet.snapshot.size` is the
server-side one.
- **boomer package restructure:** landing as a follow-up so this PR's
files stay
  reviewable in place.
- **RAM-backed variant:** glutton already has `WriteRAM`; small
follow-up.

2026-08-18 20:40:08 -07:00

5.2 KiB

Raw Blame History

Substrate Benchmarking

This is the nascent suite for benchmarking Substrate's performance at scale.

The suite also measures the telemetry volume and the capacity of the OTel collector: how much trace data and metric data substrate and its actors send, and if the collector can accept it. To make a measurement, read telemetry/README.md. For the prerequisites and the scenario ladder, read observability.md.

Deploy benchmarks

Important

Source the environment configuration file (e.g., source .ate-dev-env.sh) first so PROJECT_ID, BUCKET_NAME, etc. are set.

Note that deploying the benchmarks does not run them. You must visit Locust's web UI to start a test.

A single wrapper deploys the scale workloads, builds and pushes the Locust image, then deploys the Locust workers:

./benchmarking/deploy_locust.sh --deploy

Useful flags:

--worker-count N — number of WorkerPool replicas (default 1).
--skip-build — reuse the existing :latest locust image (skip the docker build && docker push step).

To tear everything down (locust then workloads, in reverse order):

./benchmarking/deploy_locust.sh --delete

The same operations are also reachable from the top-level installer for convenience:

./hack/install-ate.sh --deploy-benchmarks
./hack/install-ate.sh --delete-benchmarks

The installer accepts --benchmark-worker-count N (default 1). --skip-build is only available when invoking benchmarking/deploy_locust.sh directly.

Running Tests

Locust Web UI

Run kubectl port-forward svc/locust -n benchmarking 8089:8089
Visit http://localhost:8089 in your browser to configure and start the load test.

The different user classes you can select are different types of load behaviors you can throw at the system. Note that the "CounterUser" load type requires that the counter demo be installed.

You can also configure things like the number of users, how quickly those users are spawned, the frequency with which requests are made and whether or not tracing is enabled.

User classes implemented in boomer rather than Python are selected at deploy time — the stack runs one per deployment:

./benchmarking/locust/deploy.sh --deploy --user-class durdir

Headless (automation only)

runner.py runs a test without the web UI, writing CSVs, logs and traces to --dest. The nightly automation submits it as a Job on the test cluster; it is not a local entry point. See automation/README.md.

python3 runner.py -f tests/<user-class>.py -t 1m -u 1 --name <run-name> --dest /tmp/bench

Test-specific flags are appended to the same command; see the sections below.

DurDir Benchmark

The DurDir benchmark evaluates actor suspend/resume performance, disk persistence overhead, and state restoration latency when a durable directory is attached to the actor.

DurDir Configuration Knobs

--durdir-file-size-bytes: Size in bytes of the data file (default 8388608 = 8 MiB).
--resume-mode: Resume trigger mode:
- explicit (default): Client invokes the ResumeActor RPC before sending traffic.
- implicit: Client sends traffic through the router without an explicit wake RPC, testing traffic-triggered resume.
--durdir-read-mode: Verification read mode:
- data (default): Server returns full payload bytes for client-side SHA-256 verification.
- digest: Server hashes the file and returns size and digest, reducing network transfer.
--durdir-template: ActorTemplate name:
- glutton-durdir-data (default): Attaches a durable data directory without memory snapshot restore.
- glutton-durdir-full: Attaches a durable data directory and performs a full memory snapshot restore.

DurDir Reported Metrics

DurDirWrite: Initial truncate-write creating the data file.
DurDirServeInitial: First read immediately following file creation.
SuspendActor: Actor suspend latency (snapshot creation + persistence upload).
ResumeActor: Actor resume latency.
DurDirServeAfterResume: First read after resume (measures page faults / lazy load overhead on restored volume).
DurDirServeWarm: Subsequent read within the same active cycle (cached state baseline).
DurDirOverwrite: In-place file overwrite with checksum verification.

Viewing Traces

You must have enabled otel tracing for your cluster to view traces.

You can find trace IDs by viewing the logs tab in the Locust UI

Optional: Prometheus + Grafana

Locust provides graphs, statistics, etc. via the UI. However, you can install Prometheus/Grafana if you want richer details or the ability to perform deeper analysis. Skip this section if you're only using the Locust web UI.

kubectl apply -f benchmarking/monitoring.yaml

Once installed:

Run kubectl port-forward svc/grafana -n benchmarking 3000:3000
Visit http://localhost:3000 in your browser.

Development

Rebuilding gRPC Python clients

Make sure you have a virtual environment created (python3 -m venv venv) and activated (source venv/bin/activate).

Install project requirements: pip install -r requirements.txt

Then run generate_protos.sh to generate the Python proto clients.

5.2 KiB Raw Blame History