## Summary The CLI benchmarks now let each checker choose how many workers to use. On the 16-core benchmark Mac, mypy goes from 16 external workers to zero, and Pyright checks in one process. ty, Pyrefly, and Zuban still select 16 threads. `--single-threaded` still limits checker parallelism. The editor benchmarks keep their existing settings. The [18-project default-concurrency results](https://github.com/astral-sh/ty/pull/4930) were collected with the file-selection audit.
3.0 KiB
Getting started
- Install
uv- Unix:
curl -LsSf https://astral.sh/uv/install.sh | sh - Windows:
powershell -c "irm https://astral.sh/uv/install.ps1 | iex"
- Unix:
- Build ty:
cargo build --bin ty --release cdinto the benchmark directory:cd scripts/ty_benchmark- Install npm 11.10.0 or newer, which supports the dependency cooldown in
.npmrc - Install Pyright:
npm ci --ignore-scripts - Run benchmarks:
uv run benchmark
Requires hyperfine 2.0 or newer.
Benchmarks
Cold check time
Run with:
uv run --python 3.15 benchmark
Measures elapsed time and peak resident set size (RSS) when type checking a project without a pre-existing cache.
Each checker discovers files from the project's configured directories and uses its
default worker settings. Pass --single-threaded to limit checker parallelism.
Warm check time
Run with:
uv run --python 3.15 benchmark --warm
Measures how long it takes to recheck a project if there were no changes.
Note
: Of the benchmarked type checkers, only mypy supports caching.
LSP: Time to first diagnostic
Measures how long it takes for a newly started LSP to return its first diagnostic response for each open file. A server can continue checking files in the background.
Run with:
uv run --python 3.15 pytest src/benchmark/test_lsp_diagnostics.py::test_fetch_diagnostics
Note: Use -v -s to see the set of diagnostics returned by each type checker.
LSP: Re-check time
Measure how long it takes to recheck all open files after making a single change in a file.
Run with:
uv run --python 3.15 pytest src/benchmark/test_lsp_diagnostics.py::test_incremental_edit
Note
: This benchmark uses pull diagnostics for type checkers that support this operation (ty, Zuban), and falls back to publish diagnostics otherwise (Pyright, Pyrefly).
Known limitations
The tested type checkers implement Python's type system to varying degrees and some projects only successfully pass type checking using a specific type checker. We benchmark against the latest version of each type checker, but some projects may use another version in practice, leading to different diagnostic results between benchmarking and reality.
Updating the benchmark
The benchmark script supports snapshotting the results when running with --snapshot and --accept.
The goal of those snapshots is to catch accidental regressions. For example, if a project adds
new dependencies that we fail to install. They are not intended as a testing tool. E.g. the snapshot runner doesn't account for platform differences so that
you might see differences when running the snapshots on your machine.