Summary
--
This PR updates our ecosystem report to run unknown rule names through
[renamed-rule (RUF202)](https://docs.astral.sh/ruff/rules/renamed-rule/)
before normalizing them back to codes. In my first rule rename PR, the
stable ecosystem report failed on typeshed because its configuration
uses the human-readable name of one of the renamed rules. Our current
normalization process relies on `ruff rule --all --output-format=json`,
which no longer contains the old rule name, causing the human-readable
name not to be mapped back to a code, which fails on stable:
```
ruff failed
Cause: Invalid selector `blank-line-after-function` in `ignore` from `/home/runner/work/_temp/ruff-ecosystem-checkouts/python:typeshed/pyproject.toml`. Selecting rules by name requires preview mode
```
https://github.com/astral-sh/ruff/actions/runs/43917728573/job/133476621427
Instead of running `RUF202` on every config file as I initially
pictured, we synthesize a config file containing only any unknown rule
names and send that to Ruff via stdin. This both avoids the performance
cost of creating a subprocess for every project/config and retains some
ecosystem coverage for redirected prefixes and codes.
Test Plan
--
CI on #30465, which I'll stack on this PR
Summary
--
This PR updates our ruff-pre-commit dependency to 0.16.5, which supports
the new category selectors,
and then enables `lint.preview` and the new default categories so that
we can dogfood them on our
code. There were 66 diagnostics initially, which I also fixed or
ignored:
```
28 FURB113 [ ] repeated-append
12 ASYNC240 [ ] blocking-path-method-in-async-function
6 RUF105 [*] noqa-comments
5 PLW1510 [ ] subprocess-run-without-check
3 PLW0108 [ ] unnecessary-lambda
2 EXE001 [ ] shebang-not-executable
2 RUF039 [*] unraw-re-pattern
1 B024 [ ] abstract-base-class-without-abstract-method
1 FURB142 [*] for-loop-set-mutations
1 PERF102 [ ] incorrect-dict-iterator
1 PLC0207 [*] missing-maxsplit-arg
1 PLW0127 [ ] self-assigning-variable
1 TRY004 [ ] type-check-without-type-error
1 PLW0120 [*] useless-else-on-loop
1 TRY201 [ ] verbose-raise
```
Of these, I ignored `repeated-append` and `blocking-path-method-in-async-function`. I think they are
still reasonable defaults, just less useful/important in our uses.
0.16.5 also includes https://github.com/astral-sh/ruff-pre-commit/pull/159, so I removed `markdown` from our main `ruff-format` hook, leaving it for the later mdtest formatting stage.
I also had to make a few changes related to config discovery in tests:
- move our cached ecosystem checkouts to `$RUNNER_TEMP` instead of inside of our checkout
- convert some CLI tests to use the `CliTest` helpers
- add `configurationPreference: editorOnly` to some notebook tests
In these cases, our own Ruff settings were being inherited by test code. This likely affects many other tests, but I only updated the obviously failing ones for now. I'm least confident about the notebook tests, but as far as I can tell from the history (https://github.com/astral-sh/ruff/pull/11944, https://github.com/astral-sh/ruff/pull/22810), the specific diagnostics are not the important parts being asserted.
Test Plan
--
CI on this PR
Summary
--
This PR allows adding rules that only belong to categories and only have
names rather than requiring
a linter group and code.
The first non-empty commit uses `Rule::name` instead of
`Rule::noqa_code` throughout the linter tests, causing a huge number of
file renames without changes.
The second commit updates `FixTable` from a mapping of `SecondaryCode ->
(name, fix_count)` to
`DiagnosticId -> (Option<SecondaryCode>, fix_count)`. This doesn't
really change its usage but
involves minor diffs to a substantial number of lines. It also allows us
to drop some of the special
`hashbrown` handling and the `hashbrown` dependency in `ruff_linter`,
which is nice.
The third commit contains the bulk of the change, including the macro
changes needed to support
`map_codes` entries like that for the `pytest-fixture-autouse` rule in
the fourth commit:
```rust
() => rules::ruff::rules::PytestFixtureAutouse,
```
These changes are pretty mechanical overall, just handling cases where
the linter or noqa code could
be missing, which fortunately seemed pretty straightforward in every
case.
The fourth commit takes advantage of these changes to reinstate `RUF076`
as our first purely
`pedantic` rule. I'm happy to drop that if it's too controversial or
opens too many questions about
the previously-removed rule and its code, but I thought it was nice to
have a motivating example. We
moved this rule from `preview` to `removed` because it was too pedantic
for the `RUF` category, even
in preview, so it's exactly the kind of rule we can support now with the
categories.
Finally, the fifth commit updates the contributing docs to reflect that
linters and codes are
optional. This may need to be adjusted if we back out the `RUF076`
change.
Test Plan
--
Existing tests, plus a few new ones for `pytest-fixture-autouse`
Summary
--
Now that we allow rule names in selectors in preview, we need to
normalize them back to codes to
allow running on stable. This PR implements this normalization as one of
our config overrides,
using the comparison executable to extract the rule mapping from `ruff
rule`.
Test Plan
--
CI on this PR and then #27154
## Summary
@zanieb noticed while we were discussing #12595 that this flag is now
unnecessary, so remove it and the flags which reference it.
## Test Plan
Question for maintainers: is there a test to add *or* remove here? (I’ve
opened this as a draft PR with that in view!)
Fixes a regression in the ecosystem checks from
https://github.com/astral-sh/ruff/pull/9687 which was causing them to
run for multiple hours due to the size of the output.
We need the concise format for comparisons.
We should probably update the ecosystem checks to actually diff the full
output in the future because that'd be nice.
# Conflicts:
# python/ruff-ecosystem/ruff_ecosystem/projects.py
Updated implementation of https://github.com/astral-sh/ruff/pull/7369
which was left out in the cold.
This was motivated again following changes in #9691 and #9689 where we
could not test the changes without actually deprecating or removing
rules.
---
Follow-up to discussion in https://github.com/astral-sh/ruff/pull/7210
Moves integration tests from using rules that are transitively in
nursery / preview groups to dedicated test rules that only exist during
development. These rules always raise violations (they do not require
specific file behavior). The rules are not available in production or in
the documentation.
Uses features instead of `cfg(test)` for cross-crate support per
https://github.com/rust-lang/cargo/issues/8379
Uses our existing configuration overrides to unset the
`required-version` option in ecosystem projects during checks.
The downside to this approach, is we will now update the config file of
_every_ project (with a config file). This roughly normalizes the configuration file, as we
don't preserve comments and such. We could instead do a more targeted
approach applying this override to projects which we know use this
setting 🤷♀️
Adds the ability to override `ruff.toml` or `pyproject.toml` settings
per-project during ecosystem checks.
Exploring this as a fix for the `setuptools` project error.
Also useful for including Jupyter Notebooks in the ecosystem checks, see
#9293
Note the remaining `sphinx` project error is resolved in #9294
Extends https://github.com/astral-sh/ruff/pull/8416 activating the
`black-and-ruff` and `black-then-ruff` formatter comparison modes for
ecosystem checks allowing us to compare changes to Black across the
ecosystem.
Previously, the ecosystem checks formatted with the baseline then
formatted again with `--diff` to get the changed files.
Now, the ecosystem checks support a new mode where we:
- Format with the baseline
- Commit the changes
- Reset to the target ref
- Format again
- Check the diff from the baseline commit
This effectively tests Ruff changes on unformatted code rather than
changes in previously formatted code (unless, of course, the project is
already using Ruff).
While this mode is the new default, I've retained the old one for local
checks. The mode can be toggled with `--format-comparison <type>`.
Includes some more aggressive resetting of the GitHub repositories when
cached.
Here, I've also stubbed comparison modes in which `black` is used as the
baseline. While these do nothing here, #8419 adds support.
I tested this with the commit from #8216 and ecosystem changes appear
https://gist.github.com/zanieb/a982ec8c392939043613267474471a6e
Until https://github.com/astral-sh/ruff/issues/8076 is ready, it seems
beneficial to get feedback on preview mode changes.
Tested locally, updated logs to output the flags passed to `ruff` and
verified `--preview` is used.
Closes#7239
- Refactors `scripts/check_ecosystem.py` into a new Python project at
`python/ruff-ecosystem`
- Includes
[documentation](https://github.com/astral-sh/ruff/blob/zanie/ecosystem-format/python/ruff-ecosystem/README.md)
now
- Provides a `ruff-ecosystem` CLI
- Fixes bug where `ruff check` report included "fixable" summary line
- Adds truncation to `ruff check` reports
- Otherwise we often won't see the `ruff format` reports
- The truncation uses some very simple heuristics and could be improved
in the future
- Identifies diagnostic changes that occur just because a violation's
fix available changes
- We still show the diff for the line because it's could matter _where_
this changes, but we could improve this
- Similarly, we could improve detection of diagnostic changes where just
the message changes
- Adds support for JSON ecosystem check output
- I added this primarily for development purposes
- If there are no changes, only errors while processing projects, we
display a different summary message
- When caching repositories, we now checkout the requested ref
- Adds `ruff format` reports, which format with the baseline then the
use `format --diff` to generate a report
- Runs all CI jobs when the CI workflow is changed
## Known problems
- Since we must format the project to get a baseline, the permalink line
numbers do not exactly correspond to the correct range
- This looks... hard. I tried using `git diff` and some wonky hunk
matching to recover the original line numbers but it doesn't seem worth
it. I think we should probably commit the formatted changes to a fork or
something if we want great results here. Consequently, I've just used
the start line instead of a range for now.
- I don't love the comment structure — it'd be nice, perhaps, to have
separate headings for the linter and formatter.
- However, the `pr-comment` workflow is an absolute pain to change
because it runs _separately_ from this pull request so I if I want to
make edits to it I can only test it via manual workflow dispatch.
- Lines are not printed "as we go" which means they're all held in
memory, presumably this would be a problem for large-scale ecosystem
checks
- We are encountering a hard limit with the maximum comment length
supported by GitHub. We will need to move the bulk of the report
elsewhere.
## Future work
- Update `ruff-ecosystem` to support non-default projects and
`check_ecosystem_all.py` behavior
- Remove existing ecosystem check scripts
- Add preview mode toggle (#8076)
- Add a toggle for truncation
- Add hints for quick reproduction of runs locally
- Consider parsing JSON output of Ruff instead of using regex to parse
the text output
- Links to project repositories should use the commit hash we checked
against
- When caching repositories, we should pull the latest changes for the
ref
- Sort check diffs by path and rule code only (changes in messages
should not change order)
- Update check diffs to distinguish between new violations and changes
in messages
- Add "fix" diffs
- Remove existing formatter similarity reports
- On release pull request, compare to the previous tag instead
---------
Co-authored-by: konsti <konstin@mailbox.org>