Summary -- This PR updates our ecosystem report to run unknown rule names through [renamed-rule (RUF202)](https://docs.astral.sh/ruff/rules/renamed-rule/) before normalizing them back to codes. In my first rule rename PR, the stable ecosystem report failed on typeshed because its configuration uses the human-readable name of one of the renamed rules. Our current normalization process relies on `ruff rule --all --output-format=json`, which no longer contains the old rule name, causing the human-readable name not to be mapped back to a code, which fails on stable: ``` ruff failed Cause: Invalid selector `blank-line-after-function` in `ignore` from `/home/runner/work/_temp/ruff-ecosystem-checkouts/python:typeshed/pyproject.toml`. Selecting rules by name requires preview mode ``` https://github.com/astral-sh/ruff/actions/runs/43917728573/job/133476621427 Instead of running `RUF202` on every config file as I initially pictured, we synthesize a config file containing only any unknown rule names and send that to Ruff via stdin. This both avoids the performance cost of creating a subprocess for every project/config and retains some ecosystem coverage for redirected prefixes and codes. Test Plan -- CI on #30465, which I'll stack on this PR
ruff-ecosystem
Compare lint and format results for two different ruff versions (e.g. main and a PR) on real world projects.
Installation
From the Ruff project root, install with uv:
uv tool install -e ./python/ruff-ecosystem
Usage
ruff-ecosystem <check | format> <baseline executable> <comparison executable>
Note executable paths may be absolute, relative to the current working directory, or will be looked up in the current Python environment and PATH.
Run ruff check ecosystem checks comparing your debug build to your system Ruff:
ruff-ecosystem check ruff "./target/debug/ruff"
Run ruff format ecosystem checks comparing your debug build to your system Ruff:
ruff-ecosystem format ruff "./target/debug/ruff"
Run ruff format ecosystem checks comparing with changes to code that is already formatted:
ruff-ecosystem format ruff "./target/debug/ruff" --format-comparison ruff-then-ruff
Run ruff format ecosystem checks comparing with the Black formatter:
ruff-ecosystem format black ruff -v --cache python/checkouts --format-comparison black-and-ruff
The default output format is markdown, which includes nice summaries of the changes. You can use --output-format json to display the raw data — this is
particularly useful when making changes to the ecosystem checks.
Add --html-output report.html to also write a searchable HTML report containing all changes.
Development
When developing, it can be useful to set the --pdb flag to drop into a debugger on failure:
ruff-ecosystem check ruff "./target/debug/ruff" --pdb
You can also provide a path to cache checkouts to speed up repeated runs:
ruff-ecosystem check ruff "./target/debug/ruff" --cache ./repos