Commit Graph
621 Commits
Author SHA1 Message Date
Micha Reiser f5c67e3a91 Inline the expression walker into its visitors in release builds
A visitor that checks for a yield and then walks the expression can
dispatch on its kind twice. Inlining walk_expr lets the compiler combine
the checks and removes the stack copy of the classified expression on
recursive calls.

Leave inlining to the compiler with debug assertions enabled. At
opt-level=0, forcing the walker into the visitor enlarges its stack
frame enough to overflow the ty server thread on 2,000 nested additions.
2026-11-20 17:49:39 +00:00
Micha Reiser c554af9e42 Keep node storage first in the AST header
Resolving a child ID loads the row pointer from NodeStorage. Preserve
the field order of Ast to keep NodeStorage at offset zero and the row
pointer close to the start of the header.

In the x86-64 profiling build, the load in
Arguments::as_single_positional moved from offset 168 to offset zero.
That shrank the instruction from seven bytes to three. The same load
occurs throughout the generated child accessors.
2026-11-20 17:38:12 +00:00
Micha Reiser 8e8bb225a2 Inline conversions between handles for the same arena node
Every variant of a classified expression carries the same row and arena
pointers. Expose the generated conversions across crate boundaries so
the compiler can return those pointers directly in ExprKind::view.
2026-11-20 17:37:48 +00:00
Micha Reiser 6dde97cbce Document Python arena storage, traversal and node identity
Describe how borrowed handles, retained references, rewriters and
parser allocation work together.
2026-11-20 16:16:30 +00:00
Micha Reiser 90f29abdf3 Reserve arena capacity as the parser consumes source
Start with a bounded reservation based on source length. When the row
buffer fills, use the observed node density to estimate how much space
the rest of the source will need.
2026-11-20 16:16:30 +00:00
Micha Reiser 676eb21f0a Represent Python syntax with arena rows and typed node handles
Store child IDs in the arena. Expose borrowed typed nodes, fields and
source-order lists through generated accessors. The walkers accept an
already classified expression or statement. Resolve child IDs with
checked indexing.
2026-11-20 16:16:30 +00:00
Micha Reiser 504a6b153d Use source-ordered arguments for call checks and rewriting 2026-11-20 16:16:30 +00:00
Micha Reiser 455e8a66f6 Reuse evaluation-order traversal and compare aligned call arguments directly
Existence searches over calls and class headers can inspect each slot
once in written order. For f(key=a, *b), the predicate sees a first.
2026-11-20 16:16:30 +00:00
Micha Reiser 649d08f711 Read source-ordered call arguments once in Ruff and ty 2026-11-20 16:16:30 +00:00
Micha Reiser d6a10a5ac1 Consume source-ordered arguments directly in Ruff and ty 2026-11-20 16:16:30 +00:00
Micha Reiser 462f7ef367 Use the stored call shape for arity and unpacking checks 2026-11-20 16:16:30 +00:00
Micha Reiser 6d26f0396a Find a positional or named call argument in one pass 2026-11-20 16:16:30 +00:00
Micha Reiser 97e1369fc4 Store call arguments in source order
Store positional and keyword arguments in one list for calls and class headers. Source-order visitors can walk the list directly, and evaluation-order visitors visit starred arguments before keywords.

Box keyword values so a positional argument does not carry the full size of a keyword entry.
2026-11-20 16:16:30 +00:00
Micha Reiser 31bd638de3 Read parameters in written order for syntax checks and searches
Traverse source order when each parameter is checked independently,
and inspect roles directly when locating the positional-only slash.

Expression searches can check annotations and defaults in the same
written order, including for recovered signatures.
2026-11-20 16:16:30 +00:00
Micha Reiser d6dfc0920e Consume parameter roles in binding and linter checks 2026-11-20 16:16:30 +00:00
Micha Reiser 0835f5fe98 Consume parameter roles directly in signature checks and fixes 2026-11-20 16:16:30 +00:00
Micha Reiser 35e592cec3 Filter keyword parameters in a single pass 2026-11-20 16:16:30 +00:00
Micha Reiser 4c25f9b780 Consume parameter roles together in the formatter, linter and ty 2026-11-20 16:16:30 +00:00
Micha Reiser 97ac08bf0c Store function and lambda parameters in source order 2026-11-20 16:16:30 +00:00
Micha Reiser f2ab2a7d31 Read class pattern arguments directly for bindings, arity and searches
Expression searches inspect each argument’s pattern in written order.
On recovered syntax this can visit a keyword before a positional pattern.
2026-11-20 16:16:30 +00:00
Micha Reiser a9d8532b29 Store class pattern arguments in source order 2026-11-20 16:16:30 +00:00
Micha Reiser 863c499231 Share pattern capture lookup across binding visitors
Match the binding introduced by an as, star, or mapping pattern
in one place. Cover the order of recovered mapping captures in
structural comparisons.
2026-11-20 16:16:30 +00:00
Micha Reiser ddbd91227c Visit mapping pattern entries once for captures and searches
For expression searches, test each key followed by its pattern. This lets
the search stop within an entry and avoids a second pass over the list.
2026-11-20 16:16:30 +00:00
Micha Reiser 7a1b37b5d4 Store mapping pattern entries and captures in source order 2026-11-20 15:17:27 +00:00
Micha Reiser 86378a0010 Use dictionary entry children directly in the formatter and rules 2026-11-20 15:17:27 +00:00
Micha Reiser 191c8ba42b Represent dictionary entries as Python AST nodes 2026-11-20 14:28:04 +00:00
Micha Reiser 49597b86fe Remove unused source-based parentheses helper (#30614)
## Summary

Today, `ruff_python_ast` has two implementations of
`parenthesized_range`, but only tests use the source-tokenizing one.

This PR removes the unused implementation and moves the remaining
distinct cases into the token-based implementation's tests.

Testing: All AST integration tests pass, and retired instruction counts
show no regression in Ruff.
2026-11-20 10:31:33 +00:00
astral-automations-bot[bot]andBrent Westbrook a7ba68b491 Bump version to 0.17.6 (#30578)
Co-authored-by: astral-automations-bot[bot] <305554984+astral-automations-bot[bot]@users.noreply.github.com>
Co-authored-by: Brent Westbrook <36778786+ntBre@users.noreply.github.com>
2026-11-19 10:32:47 -05:00
Shivansh ShuklaandAlex Waygood 7aae7a5312 Treat relative imports from .__future__ as ordinary imports (#30260)
Co-authored-by: Alex Waygood <alex.waygood@gmail.com>
2026-11-19 15:03:01 +00:00
Micha Reiser c6c706a877 Avoid repeated scans of unclosed script metadata blocks (#30571)
## Summary

Extracting PEP 723 metadata can scan the same suffix for every unclosed
opening delimiter, leading to quadratic work. Resume at the end of a
failed scan and accept already decoded Python source to avoid redundant
UTF-8 validation.

Testing: Parser tests pass and Ruff and ty compile.
2026-11-19 14:48:11 +01:00
Micha ReiserandAlex Waygood 0222c19dd8 Share ty configuration for Ruff’s standalone scripts (#30463)
## Summary

Use a shared ty override for Ruff’s standalone scripts now that PEP 723
scripts can inherit project settings. Fix the diagnostics in the two PR
automation scripts so they can use the same checks.

## Notes to reviewers

All 28 standalone scripts passed ty with locked dependencies, and the
changed-file hooks and mocked workflow checks passed.

---------

Co-authored-by: Alex Waygood <alex.waygood@gmail.com>
2026-11-16 17:35:08 +00:00
Brent Westbrook 244668339a Adopt new APIs from Rust 1.98 (#30419)
Summary
--

- The first commit uses [strip_circumfix] to remove paired delimiters in
one call.
- The second commit uses [map_or_default] to combine many `map` and
`unwrap_or_default` calls.
- The third commit uses [bool::ok_or] to turn boolean checks into
`Result<(), E>` values.
- The fourth commit uses [str::substr_range] to get the byte range of a
TODO directive directly from the comment.

The last commit is slightly more involved than the others, as Codex
found a panic in the old code if
a `HACK` TODO comment used the Kelvin sign as its `K` and added a test
showing that `substr_range`
helped to fix it. That seems absurdly niche, but the new code is simpler
anyway.

There's also a new [format_into] API for formatting integers into a
`NumBuffer` and bypassing some
`Display` overhead. We could use this in the `UP031` fix, but it makes
the code a bit more
complicated and seemed worth trying and benchmarking separately to see
if it's worth the trouble.

[strip_circumfix]:
https://doc.rust-lang.org/stable/std/primitive.str.html#method.strip_circumfix
[map_or_default]:
https://doc.rust-lang.org/stable/std/option/enum.Option.html#method.map_or_default
[bool::ok_or]:
https://doc.rust-lang.org/stable/std/primitive.bool.html#method.ok_or
[str::substr_range]:
https://doc.rust-lang.org/stable/std/primitive.str.html#method.substr_range
[format_into]:
https://doc.rust-lang.org/stable/core/primitive.usize.html#method.format_into
2026-11-16 10:04:15 -05:00
Brent Westbrook e517fdc45f Update Rust toolchain to 1.100 and MSRV to 1.98 (#30416)
Summary
--

There were several new cargo and clippy lints here, so I tried to break
up the changes into several
commits:

- deleting some unused workspace dependencies, these looked kind of
intentional as they also had
  some shear ignores, but it seems okay to delete those too?
- allowing the [redundant_homepage] lint. We could delete a lot of
these, but the ones in the `ruff`
  and `ty` crates seem necessary for maturin, as I noted in a comment.
- fixing the [missing_lints_inheritance] lint. It seems that several of
our crates don't inherit our
workspace lints. For now I just included the `allow` for
`redundant_homepage`, but we could follow
up by inheriting the full table and fixing all the additional lints that
produces. (This commit is
  actually combined with the previous one)
- allowing cargo's new [unused_dependencies] lint, which seems redundant
with `shear` and had some
false positives on our benchmark dependencies. Codex tried to work
around this with some
`cfg`-gated imports in the benchmark crate, but ignoring this seemed
better for now, at least
  while we're still using `shear`.
- running `clippy --fix` on the remaining lint errors, with one manual
intervention where Clippy
dropped some of our `cfg` handling. These new diagnostics were mostly
around returning boolean
  expressions.
- renaming `NodeIndex::_NONE` to avoid [used_underscore_items]

There are also some new APIs in 1.98 that we can use now. I'll follow up
with those separately.

[redundant_homepage]:
https://doc.rust-lang.org/cargo/reference/lints.html#redundant_homepage
[missing_lints_inheritance]:
https://doc.rust-lang.org/cargo/reference/lints.html#missing_lints_inheritance
[used_underscore_items]:
https://clippy.rust-lang.org/master/#used_underscore_items
[unused_dependencies]:
https://doc.rust-lang.org/cargo/reference/lints.html#unused_dependencies
2026-11-16 10:04:14 -05:00
0ebb77d35c Bump version to 0.17.5 (#30380)
Co-authored-by: astral-automations-bot[bot] <305554984+astral-automations-bot[bot]@users.noreply.github.com>
Co-authored-by: Brent Westbrook <36778786+ntBre@users.noreply.github.com>
Co-authored-by: Brent Westbrook <brentrwestbrook@gmail.com>
2026-11-12 17:26:23 +00:00
Micha Reiser 8c1d3c34e4 Match docstring section headings against the whole line (#30354)
## Summary

Previously, Ruff parsed the leading words of each docstring line as a
possible section name. Both `Returns:` and `Returns: the new value.`
matched `Returns`, so Ruff extracted the name again and checked the
remaining text to reject the second line.

Ruff now skips the indentation already measured by `leading_space`,
trims trailing whitespace, and strips one optional trailing colon. It
passes the rest of the line to the existing section-name parser.
`Returns:` yields `Returns`. The inline description in `Returns: the new
value.` makes the match fail immediately. On a match,
`suspected_as_section` returns both the section kind and the borrowed
name so the caller can record its source range.

Each measurement counts instructions for one uncached lint of the NumPy
`globals.py` benchmark after initialization.

| Rules | Before | After | Reduction |
| --- | ---: | ---: | ---: |
| All stable | 819,973 | 767,658 | 6.4% |
| All with preview | 951,580 | 899,315 | 5.5% |

## Test Plan

Ran the linter and Python AST tests and compared diagnostics and edits
across the benchmark files and docstring conventions.
2026-11-12 11:26:26 +01:00
Alex WaygoodandMicha Reiser 22e77392fb [ty] Split invalid-return-type into 3 focussed rules; rename empty-body and unsound-return (#29947)
Co-authored-by: Micha Reiser <micha@reiser.io>
2026-11-11 16:59:57 +00:00
Micha Reiser 434e35024d Make ComparableExpr comparison lazy (#30259)
## Summary

Ruff uses `ComparableExpr` to compare Python syntax while ignoring
details such as source ranges and expression contexts. Previously,
`ComparableExpr::from(&expr)` recursively built a second tree containing
the fields used for comparison. For `factory(item, *items, limit=2,
**options)`, it boxed the comparable callee and the operand of `*items`,
and collected the positional arguments and keywords into two vectors.
Equality and hashing then walked that tree. Statements, match patterns,
and modules built similar trees.

`ComparableExpr<'a>` now holds a single `ast::ExprRef<'a>`. The
statement, pattern, and module keys likewise borrow their AST nodes.
Constructing a key takes constant time and allocates nothing. When
comparing two calls, equality checks the callees first. If one is
`factory` and the other is `other`, it can return before visiting any
arguments. If the callees match, it walks the original argument slices,
comparing each child as needed. Hashing walks the same structural
fields, and `Debug` displays them. Borrowing the expression also removes
the need to run `drop` after comparing.


The field helpers exhaustively destructure each node so that adding an
AST field requires deciding whether it participates in comparison.

| Workload | `main` | This PR |
| --- | ---: | ---: |
| Size of a `ComparableExpr` on x86-64 | 128 bytes | 16 bytes |
| Construct the call key | 4 allocations, 800 bytes | 0 allocations |
| Construct the key for an `if` statement containing the call | 8
allocations, 1,864 bytes | 0 allocations |
| Construct two equal call keys and compare them | 2,745 instructions |
731 instructions |
| Construct one call key and hash it | 1,621 instructions | 589
instructions |

Codspeed reports a 3% improvement on most linter benchmarks.


## Considerations

Repeatedly comparing or hashing the same key now becomes slightly more
expensive because the implementation repeats the normalization in each
hash and equal call. However, the most common case is that we compare
nodes that are unequal, where the early return outweights building the
full normalized tree. If hashing becomes a problem, we should cache the
hash key (ideally only at call sites where it matters because this
increases instruction counts again).

## Test Plan

Structural comparison and formatter tests passed, and all five lint
fixtures produced identical diagnostics on both builds.
2026-11-11 11:14:40 +01:00
astral-automations-bot[bot]andBrent Westbrook 05cdc36a45 Bump version to 0.17.4 (#30073)
Co-authored-by: astral-automations-bot[bot] <305554984+astral-automations-bot[bot]@users.noreply.github.com>
Co-authored-by: Brent Westbrook <36778786+ntBre@users.noreply.github.com>
2026-11-05 13:22:15 -05:00
Micha Reiser d91a55f1fa Collect return statements with a statement-only visitor (#29911)
## Summary

Eleven lint checks collect `return` statements from the current
function. Their visitor now skips expressions as well as nested
functions and classes. The annotation fixer keeps its existing visitor
because it also needs to detect `yield` and `yield from` before
suggesting a return type.

For PLR0911 alone, 196 calls with all stable rules enabled, this cuts
instructions from 323,004 to 64,650 (−79.98%)

## Test Plan

The existing Ruff linter unit suite passes with no snapshot changes.
2026-11-02 09:27:21 -08:00
astral-automations-bot[bot]andBrent Westbrook ba65abf8ba Bump version to 0.17.3 (#29745)
Co-authored-by: astral-automations-bot[bot] <305554984+astral-automations-bot[bot]@users.noreply.github.com>
Co-authored-by: Brent Westbrook <36778786+ntBre@users.noreply.github.com>
2026-10-26 20:14:27 -04:00
astral-automations-bot[bot]andBrent Westbrook 30516b42ab Bump version to 0.17.2 (#29637)
Co-authored-by: astral-automations-bot[bot] <305554984+astral-automations-bot[bot]@users.noreply.github.com>
Co-authored-by: Brent Westbrook <36778786+ntBre@users.noreply.github.com>
2026-10-22 13:54:24 -04:00
Micha Reiser 2983c7c529 Avoid intermediate names in lint name resolution (#29506) 2026-10-20 10:35:07 -07:00
David Peter c64734bcea [ty] Enable return type inference in scripts and tools (#29439)
## Summary

Enable return type inference for all 24 standalone Python scripts
checked by ty in CI, along with `py-fuzzer` and `ty_benchmark`.
2026-10-16 12:41:30 +00:00
Zanie Blueandzaniebot c79af46103 Require hashes for build dependencies (#29336)
Enable `require-build-hashes` for the Python projects and inline scripts
so isolated builds require declared hashes for their build dependencies.
Pin the build backends and their dependencies with hashes.

---------

Co-authored-by: zaniebot <242828183+zaniebot@users.noreply.github.com>
2026-10-15 20:15:23 +01:00
astral-automations-bot[bot] 4683270c69 Bump version to 0.17.1 (#29420)
Co-authored-by: astral-automations-bot[bot] <305554984+astral-automations-bot[bot]@users.noreply.github.com>
2026-10-15 10:56:41 -04:00
astral-automations-bot[bot]andBrent Westbrook 5b27c1ccfa Bump version to 0.17.0 (#29259)
Co-authored-by: astral-automations-bot[bot] <305554984+astral-automations-bot[bot]@users.noreply.github.com>
Co-authored-by: Brent Westbrook <brentrwestbrook@gmail.com>
2026-10-09 13:45:29 -04:00
Brent Westbrook ef23c9c39e Update default and latest Python versions for 3.15 (#28792)
Summary
--

This PR updates our default Python version to 3.11, in line with 3.10
reaching [EOL] next month, and
updates our latest Python version to 3.15. I followed last year's PR
(#20725) to find everywhere
that needed to change. Fortunately there were not as many test changes
needed as last year's
deferred annotation issues.

I also updated the language classifier in our pyproject.toml.

[EOL]: https://devguide.python.org/versions/#supported-versions

Test Plan
--

Existing tests with updated snapshots. Most of the updates were to
snapshots for TC001, TC002, and
TC003, which now default to the lazy-import fix.

After updating the snapshots, I tried to make sure we still had coverage
for whatever changed. That
meant adding a few more mdtests for the TC rules, and one small mdtest
for RUF017 to restore
coverage for the `functools.reduce` fix.
2026-10-09 12:26:57 -04:00
Alex Waygood f52de0dcbd [ty] Enable additional ty checks for standalone scripts (#29093) 2026-10-03 18:51:56 +00:00
Alex Waygood 8ecfc92734 Check script lockfiles in CI (#29021) 2026-10-02 14:28:56 +01:00
Charlie Marsh 598f4bdfd5 Adopt Rust 1.97 integer APIs and Cargo warning policy (#29048)
## Summary

- Use the integer bit APIs stabilized in Rust 1.97 for packed node
indices, token capacity, rule iteration, and type-variable support sets.
- Use Cargo's warning policy in our native and WASM Clippy checks,
keeping warnings fatal without changing compiler flags and invalidating
the build cache. Update the contributing guides to match.

Preserves the Rust 1.97 MSRV. The warning policy change follows
https://github.com/astral-sh/uv/pull/22122.
2026-10-01 14:01:44 -04:00