Summary
--
This PR updates our ecosystem report to run unknown rule names through
[renamed-rule (RUF202)](https://docs.astral.sh/ruff/rules/renamed-rule/)
before normalizing them back to codes. In my first rule rename PR, the
stable ecosystem report failed on typeshed because its configuration
uses the human-readable name of one of the renamed rules. Our current
normalization process relies on `ruff rule --all --output-format=json`,
which no longer contains the old rule name, causing the human-readable
name not to be mapped back to a code, which fails on stable:
```
ruff failed
Cause: Invalid selector `blank-line-after-function` in `ignore` from `/home/runner/work/_temp/ruff-ecosystem-checkouts/python:typeshed/pyproject.toml`. Selecting rules by name requires preview mode
```
https://github.com/astral-sh/ruff/actions/runs/43917728573/job/133476621427
Instead of running `RUF202` on every config file as I initially
pictured, we synthesize a config file containing only any unknown rule
names and send that to Ruff via stdin. This both avoids the performance
cost of creating a subprocess for every project/config and retains some
ecosystem coverage for redirected prefixes and codes.
Test Plan
--
CI on #30465, which I'll stack on this PR
Summary
--
Now that we allow rule names in selectors in preview, we need to
normalize them back to codes to
allow running on stable. This PR implements this normalization as one of
our config overrides,
using the comparison executable to extract the rule mapping from `ruff
rule`.
Test Plan
--
CI on this PR and then #27154
Summary
--
This is the first in a series of migration rules to help users move from
`noqa` comments with rule
codes to `ruff:ignore` comments with names. This rule simply replaces
`noqa: codes` with
`ruff:ignore[codes]` (and the file-level variants), deferring the
transformation of codes to names
to a follow-up rule. This rule may be useful on its own without the
codes -> names rule, assuming we
stabilize `ruff:ignore` in the next minor release without stabilizing
human-readable names.
The rule skips any comment consisting entirely of known `external`
selectors, while still emitting diagnostics for totally unknown codes.
We also skip file-level `flake8: noqa` directives assuming they are
intended to be shared with flake8.
I initially wrote a small, standalone implementation in the rule file
itself, but it had to be
combined with the `RUF100` implementation to leave unused `noqa` codes
out of `RUF105` to be cleaned
up by `RUF100` instead, as I note in the mdtest for this case.
Test Plan
--
New mdtests
I also expect a huge number of ecosystem results on this PR, possibly so
many that the check times out.
---------
Co-authored-by: Micha Reiser <micha@reiser.io>
## Summary
Follow up to #26050 to clean up the ecosystem check script now that
diagnostics without severity are the baseline again. This reverts both
the temporary normalization code from that PR and the earlier regex
change from https://github.com/astral-sh/ruff/pull/23845.
## Test Plan
CI on this PR
Summary
--
This reverts #23845 since we're now prioritizing the human-readable
names and category work before getting back to severity.
I had to revert (the revert of) the changes to the ecosystem check
script and then add another normalization pass again to get CI to pass.
I can follow up immediately to revert these changes once this is the
baseline.
Test Plan
--
Existing tests updated following #25937
I said I didn't love diagnostics like this:
```
fib.py:1:8: unused-import: [*] `os` imported but unused
```
with both a human-readable name and a fix indicator on that PR, but I
think it's not actually that
bad, and I haven't thought of a better alternative, so I think it's fine
for now.
Summary
--
Reverts the name normalization part of #25937 now that human-readable
names are also present in the
baseline for future ecosystem check runs.
Test Plan
--
CI on this PR
## Summary
This PR updates all of our CLI output formats to use human-readable
names instead of rule codes in preview.
## Test Plan
The first commit adds a new CLI test and then each following commit
updates one output format and the associated snapshots.
The last commit also updates the ecosystem check to normalize rule names
back to codes to avoid a huge diff that causes the check to time out.
Alternatively, we may just want to accept the time out on this PR so
that we can see the un-normalized report in the future.
Summary
--
I noticed while extending the regex to match human-readable names that
it also didn't cover the
ASYNC rules, which have five letters in the rule code. For example,
#24644 revealed no new
diagnostics in the ecosystem, but a local run on my cached ecosystem
repos revealed many
diagnostics. This won't show up as a diff in CI on this PR but should
help with ecosystem checks on
the ASYNC rules in the future.
This is just the "aesthetic" part of implementing `warning` severities.
Configuration changes are in
https://github.com/astral-sh/ruff/pull/23846
It's expected that the `preview` ecosystem check times out here.
Summary
--
The huge number of changes in
https://github.com/astral-sh/ruff/pull/22205#issuecomment-3696660639
should have
obviously been a red flag, but I think it would be nice if CI failed
when new
ecosystem panics were introduced. This PR adds a check for diagnostic
lines that
start with `panic: Panicked at crates/`, raises a `ToolError` if any are
found
in the results from the comparison executable, and then ~~also exits
non-zero if
any errors are returned~~ fails the CI run if the corresponding error
message
was printed.
After trying this out in CI, I opted not to change the script's exit
code itself
because that suppressed the ecosystem comment. It feels a little hackier
this way but preserves the behavior I wanted of both failing CI and
still getting
the ecosystem comment to help with debugging.
Test Plan
--
Local testing on the 0.15.3 tag showing that ruff-ecosystem exited
non-zero and
some manual testing in CI, as you can see below.
## Summary
This PR is a spin-off from https://github.com/astral-sh/ruff/pull/19415.
It enables replacing the severity and lint name in a ty-style
diagnostic:
```
error[unused-import]: `os` imported but unused
```
with the noqa code and optional fix availability icon for a Ruff
diagnostic:
```
F401 [*] `os` imported but unused
F821 Undefined name `a`
```
or nothing at all for a Ruff syntax error:
```
SyntaxError: Expected one or more symbol names after import
```
Ruff adds the `SyntaxError` prefix to these messages manually.
Initially (d912458), I just passed a `hide_severity` flag through a
bunch of calls to get it into `annotate-snippets`, but after looking at
it again today, I think reusing the `None` severity/level gave a nicer
result. As I note in a lengthy code comment, I think all of this code
should be temporary and reverted when Ruff gets real severities, so
hopefully it's okay if it feels a little hacky.
I think the main visible downside of this approach is that we can't
style the asterisk in the fix availabilty icon in cyan, as in Ruff's
current output. It's part of the message in this PR and any styling gets
overwritten in `annotate-snippets`.
<img width="400" height="342" alt="image"
src="https://github.com/user-attachments/assets/57542ec9-a81c-4a01-91c7-bd6d7ec99f99"
/>
Hmm, I guess reusing `Level::None` also means the `F401` isn't red
anymore. Maybe my initial approach was better after all. In any case,
the rest of the PR should be basically the same, it just depends how we
want to toggle the severity.
## Test Plan
New `ruff_db` tests. These snapshots should be compared to the two tests
just above them (`hide_severity_output` vs `output` and
`hide_severity_syntax_errors` against `syntax_errors`).
Summary
--
This updates the regex in `ruff-ecosystem` to catch syntax errors in an
effort to prevent bugs like #16874. This should catch `ParseError`s,
`UnsupportedSyntaxError`s, and the upcoming `SemanticSyntaxError`s.
Test Plan
--
I ran the ecosystem check locally comparing v0.11.0 and v0.11.1 and saw
a large number (2757!) of new syntax errors. I also manually tested the
regex on a few lines before that.
If we merge this before #16878, I'd expect to see that number decrease
substantially in that PR too, as another test.
Adds the ability to override `ruff.toml` or `pyproject.toml` settings
per-project during ecosystem checks.
Exploring this as a fix for the `setuptools` project error.
Also useful for including Jupyter Notebooks in the ecosystem checks, see
#9293
Note the remaining `sphinx` project error is resolved in #9294
Instead of displaying the total completed project count in the "changed"
section of a header, we now separately calculated the changed and
unchanged count to make the header message nice and clear.
e.g.
> ℹ️ ecosystem check **detected format changes**. (+1772 -1859 lines in
239 files in 26 projects; 6 project errors; 9 projects unchanged)
and
> ℹ️ ecosystem check **detected linter changes**. (+4598 -5023
violations, +0 -40 fixes in 13 projects; 4 project errors; 24 projects
unchanged)
Previously, it would have included the unchanged count in the first
project count.
Extends https://github.com/astral-sh/ruff/pull/8416 activating the
`black-and-ruff` and `black-then-ruff` formatter comparison modes for
ecosystem checks allowing us to compare changes to Black across the
ecosystem.
Until https://github.com/astral-sh/ruff/issues/8076 is ready, it seems
beneficial to get feedback on preview mode changes.
Tested locally, updated logs to output the flags passed to `ruff` and
verified `--preview` is used.
Fixes bug where `total_affected_rules` is empty, a division by zero
error can occur if there are only errors and no rule changes. Calculates
the maximum display per rule with the calculated project maximum as the
upper bound instead of 50, this should show more rule variety when
project maximums are lower.
This commit was meant to be in #8223 but I missed it.
Closes#7239
- Refactors `scripts/check_ecosystem.py` into a new Python project at
`python/ruff-ecosystem`
- Includes
[documentation](https://github.com/astral-sh/ruff/blob/zanie/ecosystem-format/python/ruff-ecosystem/README.md)
now
- Provides a `ruff-ecosystem` CLI
- Fixes bug where `ruff check` report included "fixable" summary line
- Adds truncation to `ruff check` reports
- Otherwise we often won't see the `ruff format` reports
- The truncation uses some very simple heuristics and could be improved
in the future
- Identifies diagnostic changes that occur just because a violation's
fix available changes
- We still show the diff for the line because it's could matter _where_
this changes, but we could improve this
- Similarly, we could improve detection of diagnostic changes where just
the message changes
- Adds support for JSON ecosystem check output
- I added this primarily for development purposes
- If there are no changes, only errors while processing projects, we
display a different summary message
- When caching repositories, we now checkout the requested ref
- Adds `ruff format` reports, which format with the baseline then the
use `format --diff` to generate a report
- Runs all CI jobs when the CI workflow is changed
## Known problems
- Since we must format the project to get a baseline, the permalink line
numbers do not exactly correspond to the correct range
- This looks... hard. I tried using `git diff` and some wonky hunk
matching to recover the original line numbers but it doesn't seem worth
it. I think we should probably commit the formatted changes to a fork or
something if we want great results here. Consequently, I've just used
the start line instead of a range for now.
- I don't love the comment structure — it'd be nice, perhaps, to have
separate headings for the linter and formatter.
- However, the `pr-comment` workflow is an absolute pain to change
because it runs _separately_ from this pull request so I if I want to
make edits to it I can only test it via manual workflow dispatch.
- Lines are not printed "as we go" which means they're all held in
memory, presumably this would be a problem for large-scale ecosystem
checks
- We are encountering a hard limit with the maximum comment length
supported by GitHub. We will need to move the bulk of the report
elsewhere.
## Future work
- Update `ruff-ecosystem` to support non-default projects and
`check_ecosystem_all.py` behavior
- Remove existing ecosystem check scripts
- Add preview mode toggle (#8076)
- Add a toggle for truncation
- Add hints for quick reproduction of runs locally
- Consider parsing JSON output of Ruff instead of using regex to parse
the text output
- Links to project repositories should use the commit hash we checked
against
- When caching repositories, we should pull the latest changes for the
ref
- Sort check diffs by path and rule code only (changes in messages
should not change order)
- Update check diffs to distinguish between new violations and changes
in messages
- Add "fix" diffs
- Remove existing formatter similarity reports
- On release pull request, compare to the previous tag instead
---------
Co-authored-by: konsti <konstin@mailbox.org>