Commit Graph
18316 Commits
Author SHA1 Message Date
Micha Reiser f5c67e3a91 Inline the expression walker into its visitors in release builds
A visitor that checks for a yield and then walks the expression can
dispatch on its kind twice. Inlining walk_expr lets the compiler combine
the checks and removes the stack copy of the classified expression on
recursive calls.

Leave inlining to the compiler with debug assertions enabled. At
opt-level=0, forcing the walker into the visitor enlarges its stack
frame enough to overflow the ty server thread on 2,000 nested additions.
2026-11-20 17:49:39 +00:00
Micha Reiser c554af9e42 Keep node storage first in the AST header
Resolving a child ID loads the row pointer from NodeStorage. Preserve
the field order of Ast to keep NodeStorage at offset zero and the row
pointer close to the start of the header.

In the x86-64 profiling build, the load in
Arguments::as_single_positional moved from offset 168 to offset zero.
That shrank the instruction from seven bytes to three. The same load
occurs throughout the generated child accessors.
2026-11-20 17:38:12 +00:00
Micha Reiser 8e8bb225a2 Inline conversions between handles for the same arena node
Every variant of a classified expression carries the same row and arena
pointers. Expose the generated conversions across crate boundaries so
the compiler can return those pointers directly in ExprKind::view.
2026-11-20 17:37:48 +00:00
Micha Reiser 6dde97cbce Document Python arena storage, traversal and node identity
Describe how borrowed handles, retained references, rewriters and
parser allocation work together.
2026-11-20 16:16:30 +00:00
Micha Reiser a9447d7df1 [ty] Use typed arena nodes in semantic and IDE queries
Use parser row IDs for stored AST references. Read the corresponding
borrowed node in inference, reference search and IDE queries.
2026-11-20 16:16:30 +00:00
Micha Reiser 37684d51ce Format Python using the arena and source-order visitor
Read children directly from the stored argument and parameter lists.
Use the syntax tree to locate comments and construct formatted output.
2026-11-20 16:16:30 +00:00
Micha Reiser f74b58fffe Construct replacement syntax for lint fixes in an arena
Build expressions and statements for fixes in a destination arena.
Copy the children each fix retains into that arena before printing
the replacement syntax.
2026-11-20 16:16:30 +00:00
Micha Reiser 4d8784b920 Read typed arena nodes in lint rules
Match on the classified node and read its fields through typed
accessors. Use the stored lists when inspecting child nodes.
2026-11-20 16:16:30 +00:00
Micha Reiser d8666fbbda Traverse typed arena nodes in Ruff's checker
Forward the classified expression or statement to the walker and
pass the borrowed node through rule dispatch. Visit call arguments in
evaluation order when processing type definitions.
2026-11-20 16:16:30 +00:00
Micha Reiser d2846dda15 Use arena nodes in the semantic model and source edits
Keep borrowed node handles in the semantic model. Produce replacement
syntax in a builder and generate Python source from the resulting tree.
2026-11-20 16:16:30 +00:00
Micha Reiser 90f29abdf3 Reserve arena capacity as the parser consumes source
Start with a bounded reservation based on source length. When the row
buffer fills, use the observed node density to estimate how much space
the rest of the source will need.
2026-11-20 16:16:30 +00:00
Micha Reiser 043802b0d8 Build typed arena nodes in the Python parser
Carry the typed IDs returned by node constructors through parsing and
recovery. Parse source-ordered argument and parameter lists directly.
Keep ownership of the tree with the parsed module.
2026-11-20 16:16:30 +00:00
Micha Reiser 676eb21f0a Represent Python syntax with arena rows and typed node handles
Store child IDs in the arena. Expose borrowed typed nodes, fields and
source-order lists through generated accessors. The walkers accept an
already classified expression or statement. Resolve child IDs with
checked indexing.
2026-11-20 16:16:30 +00:00
Micha Reiser b6352c04b1 Use stored argument order in call inference and linting
Check the stored arguments directly when matching a call shape and
reserve class base space from the argument count.
2026-11-20 16:16:30 +00:00
Micha Reiser c8003c1533 Check call arity and argument kinds together in lint rules
Check the stored number of arguments before matching a positional
argument list. Inspect the sole item when checking an ABC base or
metaclass, and count positionals only after identifying a setter.
2026-11-20 16:16:30 +00:00
Micha Reiser a9d03facaa Visit ordinary call arguments in evaluation order 2026-11-20 16:16:30 +00:00
Micha Reiser 2766b66062 Validate positional and keyword arguments as the parser reads them 2026-11-20 16:16:30 +00:00
Micha Reiser 504a6b153d Use source-ordered arguments for call checks and rewriting 2026-11-20 16:16:30 +00:00
Micha Reiser 455e8a66f6 Reuse evaluation-order traversal and compare aligned call arguments directly
Existence searches over calls and class headers can inspect each slot
once in written order. For f(key=a, *b), the predicate sees a first.
2026-11-20 16:16:30 +00:00
Micha Reiser 649d08f711 Read source-ordered call arguments once in Ruff and ty 2026-11-20 16:16:30 +00:00
Micha Reiser d6a10a5ac1 Consume source-ordered arguments directly in Ruff and ty 2026-11-20 16:16:30 +00:00
Micha Reiser 462f7ef367 Use the stored call shape for arity and unpacking checks 2026-11-20 16:16:30 +00:00
Micha Reiser 67aff7f88d Check URL arguments and unpacking in one pass 2026-11-20 16:16:30 +00:00
Micha Reiser 305c41149a Check source-ordered call arguments without repeated scans 2026-11-20 16:16:30 +00:00
Micha Reiser 6d26f0396a Find a positional or named call argument in one pass 2026-11-20 16:16:30 +00:00
Micha Reiser 97e1369fc4 Store call arguments in source order
Store positional and keyword arguments in one list for calls and class headers. Source-order visitors can walk the list directly, and evaluation-order visitors visit starred arguments before keywords.

Box keyword values so a positional argument does not carry the full size of a keyword entry.
2026-11-20 16:16:30 +00:00
Micha Reiser 31bd638de3 Read parameters in written order for syntax checks and searches
Traverse source order when each parameter is checked independently,
and inspect roles directly when locating the positional-only slash.

Expression searches can check annotations and defaults in the same
written order, including for recovered signatures.
2026-11-20 16:16:30 +00:00
Micha Reiser 5ba9e21582 Use explicit LF in recovered parameter regression test 2026-11-20 16:16:30 +00:00
Micha Reiser d6dfc0920e Consume parameter roles in binding and linter checks 2026-11-20 16:16:30 +00:00
Micha Reiser c5299723d0 Read route parameter names from the stored parameter roles 2026-11-20 16:16:30 +00:00
Micha Reiser 0835f5fe98 Consume parameter roles directly in signature checks and fixes 2026-11-20 16:16:30 +00:00
Micha Reiser 35e592cec3 Filter keyword parameters in a single pass 2026-11-20 16:16:30 +00:00
Micha Reiser 4c25f9b780 Consume parameter roles together in the formatter, linter and ty 2026-11-20 16:16:30 +00:00
Micha Reiser 97ac08bf0c Store function and lambda parameters in source order 2026-11-20 16:16:30 +00:00
Micha Reiser 347af7ab05 Cover class pattern arguments in code generation
Round-trip a class pattern with a nested positional pattern and a
keyword pattern.
2026-11-20 16:16:30 +00:00
Micha Reiser f2ab2a7d31 Read class pattern arguments directly for bindings, arity and searches
Expression searches inspect each argument’s pattern in written order.
On recovered syntax this can visit a keyword before a positional pattern.
2026-11-20 16:16:30 +00:00
Micha Reiser a9d8532b29 Store class pattern arguments in source order 2026-11-20 16:16:30 +00:00
Micha Reiser 863c499231 Share pattern capture lookup across binding visitors
Match the binding introduced by an as, star, or mapping pattern
in one place. Cover the order of recovered mapping captures in
structural comparisons.
2026-11-20 16:16:30 +00:00
Micha Reiser 42500393ae Skip the mapping capture lookup when there are no dangling comments 2026-11-20 16:16:30 +00:00
Micha Reiser ddbd91227c Visit mapping pattern entries once for captures and searches
For expression searches, test each key followed by its pattern. This lets
the search stop within an entry and avoids a second pass over the list.
2026-11-20 16:16:30 +00:00
Micha Reiser 7a1b37b5d4 Store mapping pattern entries and captures in source order 2026-11-20 15:17:27 +00:00
Micha Reiser a6634d87fc Use dictionary entry ranges for repeated keys and unpacked arguments
In `f(**{**(options := defaults)})`, the inner entry spans
`**(options := defaults)`. PIE804 can copy that slice into the call
directly, preserving the parentheses covered by #30629.

For a repeated key, the ends of adjacent entries give both boundaries
of the deletion, including any parentheses around the values.
2026-11-20 15:17:27 +00:00
Micha Reiser 86378a0010 Use dictionary entry children directly in the formatter and rules 2026-11-20 15:17:27 +00:00
Micha Reiser fb4570d984 Use dictionary entry ranges to remove nested unpacking 2026-11-20 14:28:04 +00:00
Micha Reiser 191c8ba42b Represent dictionary entries as Python AST nodes 2026-11-20 14:28:04 +00:00
Micha Reiser e905eb561c [ty] Fix a panic when reporting invalid type parameter order in unpacked class bases (#30630)
## Summary

Today, ty panics on `class C(*(int, Generic[T, U]))` when `T` has a
default and `U` is required. To locate the error, ty looks for the
second base in the class header. The header contains a single starred
expression that expands to two bases.

This PR changes the diagnostic to use the source expression ty recorded
for each base during expansion, following #24695 and #24699. The
diagnostic now underlines `T, U` and reports that a required type
parameter cannot follow one with a default.

Added an mdtest that checks the diagnostic on the unpacked generic base.
2026-11-20 15:43:20 +01:00
Micha Reiser 7e70fa6611 Preserve parentheses when simplifying nested dictionary unpacking (PIE804) (#30629)
## Summary

Today, PIE804 suggests rewriting `foo(**{**(options := defaults)})` as
`foo(**options := defaults)`. The edit drops the parentheses required
around the assignment expression, so `ruff check --fix` rejects it as
invalid syntax.

This PR changes PIE804 to preserve parentheses when copying the unpacked
value, using the same lookup as #11372. Ruff can then simplify the call
to `foo(**(options := defaults))`.

Added a regression fixture that failed before the change and now passes.
2026-11-20 15:35:13 +01:00
Brent Westbrook a866aff503 Include rule names in warning and error messages (#30575)
Summary
--

Fixes #27012 by finding the remaining places where we used rule codes
rather than also
including/preferring rule names.

Most of these places used `RuleSelector`s, so I added
`RuleSelector::name_and_code`, which uses
`Rule::name_and_code` for rule selectors and falls back on the existing
`prefix_and_code` for
non-rule selectors.

Test Plan
--

Updated snapshots
2026-11-20 09:25:57 -05:00
Alex Waygood d8c6e237e7 [ty] Split mismatched-type-name by type constructor (#29957)
## Summary

This PR splits `mismatched-type-name` into eight independently
configurable warnings for `NewType`, functional enums, `TypeVar`,
`ParamSpec`, `TypeVarTuple`, `namedtuple`/`NamedTuple`, `TypedDict`, and
`TypeAliasType`.

~~This PR is actually quite a bit less breaking for users than other
rule splits we've done over the last few days. Because
`mismatched-type-name` is no longer the canonical name for _any_ rule on
this branch, we're able to make it a deprecated alias for _all_ the new
rules, meaning that it will continue to work on the CLI, in ty
configuration files, and ty suppression comments (we'll just emit a
warning diagnostic telling users to upgrade their configuration
file/suppression comment).~~

~~A more specific rule takes precedence over the deprecated name if both
are given within the same configuration table. On the command line, the
last matching flag wins (e.g. `--error=mismatched-enum-name
--ignore=mismatched-type-name` will suppress diagnostics for mismatched
enum names).~~

(After review from @MichaReiser, I backed out these changes. It's now a
somewhat breaking change, because the rule is now marked as "removed"
rather than "deprecated".)

As well as splitting the rule into eight separate rules, it also cleans
up our error recovery for several of these type factories. For example,
in the following snippet, we fail to recognize `T` as a `TypeVar` due to
bad error recovery from the non-literal name. But we don't really need a
nonliteral name in this case to determine that it's a validly
constructed TypeVar. It's probably a mistake, but there's no _soundness_
issue here:

```py
from typing import TypeVar

def f(x: str):
    # main emits an error: `invalid-typevar`
    # this PR emits a warning: `mismatched-typevar-name`
    T = TypeVar(x)

    def g(x: T) -> T:
        return x
    
    reveal_type(g(42))  # main: Todo. This PR: `Literal[42]`.
```

## Motivation

The main motivation here is that having a namedtuple or enum with the
"wrong name", for example, just seems much less "obviously wrong" than
having a typevar or paramspec with the "wrong name". The [ecosystem
report](https://github.com/astral-sh/ruff/pull/29957#issuecomment-6603964612)
is evidence of this. namedtuples and enums define runtime classes that
can be used in runtime contexts and instantiated; these constructs
aren't _just_ for type checkers.
TypeVars/ParamSpecs/TypeVarTuples/NewTypes/TypedDicts are all _only_ for
type checkers and there's no plausible reason why you'd want to give the
type a different name to the variable it's assigned to.

`mismatched-namedtuple-name` and `mismatched-enum-name` are therefore
now disabled by default, matching the default severity we already
settled on for `mistmatched-sentinel-name` in
https://github.com/astral-sh/ruff/pull/27206. `mismatched-typevar-name`,
`mismatched-paramspec-name`, `mismatched-typevartuple-name`,
`mismatched-enum-name`, `mismatched-namedtuple-name` and
`mismatched-newtype-name` all remain enabled by default with `warning`
severity.
2026-11-20 12:41:39 +00:00
Alex Waygood 157cae3c06 [ty] Honor inline ignores on mismatched ParamSpec components (#30595)
When `**kwargs` is annotated with `P.kwargs`, the matching `*args`
annotation must be `P.args`. We emit a diagnostic on `P.args` as a
result -- but if you add a suppression comment to that line... a new
diagnostic pops up on `P.kwargs`! That's not how suppression comments
are really meant to work. This PR fixes that.

## Test plan

### Before (`aa46aae60`)

Adding `# ty: ignore[invalid-paramspec]` on the `*args` line moves the
diagnostic to `**kwargs`.


https://github.com/user-attachments/assets/50ee1858-8126-45bb-896f-2517b98950fe

### After (`684c14273`)

Adding the same ignore clears the diagnostic.



https://github.com/user-attachments/assets/0a7fc2b0-ffd0-49b4-87ae-714835b7421f
2026-11-20 12:13:58 +00:00