Commit Graph
443 Commits
Author SHA1 Message Date
Charlie Marsh 574e10752f Expand semantic syntax errors for invalid walruses (#25415)
## Summary

Python rejects assignment expressions in several comprehension contexts,
but we previously only reported the case in which a walrus rebound a
comprehension variable.

As a result, we accepted invalid walruses in the iterable and class-body
portions clause of a comprehension, and missed some rebinding cases in
filter clauses and nested comprehensions. It turns out the rules are
really complicated!

For example, we now report the two newly modeled syntax errors:

```py
# error: [invalid-syntax] "assignment expression cannot be used in a comprehension iterable expression"
[x for x in (values := [1])]

class C:
    # error: [invalid-syntax] "assignment expression within a comprehension cannot be used in a class body"
    [(value := item) for item in [1]]
```

We also catch previously missed instances of the existing rebinding
error:

```py
# error: [invalid-syntax] "assignment expression cannot rebind comprehension variable"
[x for x in [1] if (x := 0)]
```

In general, I decided not to worry about emitting multiple diagnostics
for a single error. These are rare, they're fatal, and it added a lot of
complexity (so in some cases we may emit two errors for a walrus that is
invalid for two reasons).
2026-05-27 21:39:41 +01:00
Micha ReiserandDhruv Manilawala 38562889b9 Reduce memory usage by dropping token-excess capacity and improve performance by approximating the initial tokens vec size (#25354)
Co-authored-by: Dhruv Manilawala <dhruvmanila@gmail.com>
2026-05-26 18:08:12 +01:00
Micha Reiser 6ab8b737c5 Use ThinVec in AST to shrink Stmt (#25361) 2026-05-24 16:23:36 +01:00
Micha Reiser 5f284d1f52 Recover incomplete list comprehension elements (#25326) 2026-05-23 10:03:56 +01:00
Charlie Marsh 2fd7ed0a16 Avoid redundant TokenValue drops in the lexer (#25300)
## Summary

If we avoid resetting the lexer value slot when it's already empty, this
apparently improves performance by 1-2%. (In optimized builds, this
keeps the common case to a discriminant branch instead of re-entering
the `TokenValue` drop path.)
2026-05-21 19:30:48 +01:00
Charlie Marsh aea5ed4d27 Avoid unnecessary parser lookahead for operators (#25290)
## Summary

We only need to peek ahead two tokens if the first token is `not` or
`in`. We can avoid the `peek` in the majority of cases, which apparently
speeds up the parser by 20-30%.
2026-05-21 14:13:20 +01:00
Samuel ColvinandCharlie Marsh c423054dc0 Add a recursion limit to the parser (#24810)
## Summary

Partial fix for #22930.

Without this malicious or machine generated code could cause a stack
overflow with something as simple as `'(' * 5000 + '1' + ')' * 5000`.

I decided to do the simplest thing and have a limit that's always
applied with a reasonable default. Since:
* the overhead of this check will be tiny
* it seems inconceivable that anyone will want to have no limit 

## Test Plan

PR includes tests.

---------

Co-authored-by: Charlie Marsh <charlie.r.marsh@gmail.com>
2026-05-21 09:34:53 +01:00
Jelle Zijlstra 409c13f3ec Add full PEP 798 support (#25104)
Fixes #25098.

This adds full support for PEP 798 through the parser, linter,
formatter, and type checker.
There are a lot of changes but they are mostly straightforward. Let me
know though if it's better to split this up.
2026-05-16 16:32:57 +01:00
Eyüp Can AkmanandMicha Reiser f32733c063 [flake8-pyi] Fix PYI016 false positive for f-string debug specifier (#24098)
Co-authored-by: Micha Reiser <micha@reiser.io>
2026-04-30 11:14:25 +02:00
David Peter cf14481f44 Semantic syntax errors: better error message for global vs parameter (#24902)
## Summary

This is an attempt to improve the error message for something like
```py
a = 1

def f(a):
    global a
```

I realize that the previous wording ("name 'a' is parameter and global")
is used by CPython itself, but unless we are trying to be consistent
with CPython, I think we can improve upon this? The previous version
seemed a bit cryptic to me when I first saw it.

## Test Plan

Updated snapshot tests
2026-04-29 08:46:57 +02:00
Alex Waygood 581b3fa230 Update Rust toolchain to 1.95 and MSRV to 1.93 (#24677) 2026-04-17 11:39:14 +01:00
Micha Reiser ad30af4cd0 Force update insta snapshots (#24495) 2026-04-08 17:33:19 +02:00
Charlie Marsh 9a55bc6568 Reject multi-line f-string elements before Python 3.12 (#24355)
## Summary

Before Python 3.12, a replacement field in an f-string can span multiple
lines only if the outer f-string is triple-quoted. This was relaxed in
Python 3.12, but we weren't rejecting these as syntactically invalid on
earlier versions.

Closes https://github.com/astral-sh/ruff/issues/24348.
2026-04-05 15:09:26 +00:00
Dylan 5f88756ee1 Disallow starred expressions as values of starred expressions (#24280)
Part of #19077
2026-04-02 14:57:28 +00:00
Charlie Marsh fcef46ccdf Fix %foo? parsing in IPython assignment expressions (#24152)
## Summary

We now distinguish between IPython escape commands lexed after an `=`
sign from those at the start of a logical line, which allows `x = %foo?`
to be interpreted as "assign the result of running a line magic named
`foo?`", matching the IPython assignment-magic transform.

See:
https://github.com/astral-sh/ruff/issues/21705#issuecomment-4085099342.
2026-03-24 13:14:53 +00:00
Charlie Marsh 8bc437ef90 Add support for star-unpacking of comprehensions (PEP 798) (#23788)
## Summary

In Python 3.15, you can star-unpack comprehensions (e.g., `[*x for x in
y]`) via [PEP
798](https://docs.python.org/3.15/whatsnew/3.15.html#whatsnew315-unpacking-in-comprehensions).
This PR adds support to the parser.
2026-03-09 10:23:52 -04:00
Alex Waygood 0d84a0654a [ty] Add validation for type parameters with defaults after TypeVarTuple (#23807) 2026-03-08 18:28:16 +00:00
Charlie Marsh 735da8b415 Reject semantic syntax errors for lazy imports (#23757)
## Summary

We now reject cases like:

- `lazy import ...` inside functions, including `async def`
- `lazy from ... import ...` inside functions, including `async def`
- `lazy import ...` and `lazy from ... import ...` inside class bodies
- `lazy import ...` and `lazy from ... import ...` anywhere inside a
`try` statement, including `except` / `except*`
- `lazy from ... import *` anywhere
- `lazy from __future__ import ...` anywhere

A follow-up to #23755.

See: https://github.com/astral-sh/ruff/issues/21305.
2026-03-06 14:25:13 +00:00
Charlie Marsh 8a10b6743c Add support for lazy import parsing (#23755)
## Summary

This PR adds `is_lazy` to the AST and parser, and supports `lazy` import
formatting in the formatter. The name `is_lazy` matches the [CPython
AST](https://github.com/python/cpython/blob/c3fb0d9d96902774c08b199dda0479a8d31398a5/Parser/Python.asdl#L48),
though we could modify it if there's demand.

There are no further changes to Ruff rules or even to Ruff import
sorting, since those deserve separate design discussions.

I've also omitted support for flagging semantic syntax errors (e.g.,
`lazy import` inside `def`).

See: https://github.com/astral-sh/ruff/issues/21305.
2026-03-06 09:12:21 -05:00
Dhruv ManilawalaandKartik Ganapathi 6551438942 [parser] Fix indentation tracking after line continuations (#23417)
## Summary

fixes: https://github.com/astral-sh/ruff/issues/19301

## Test Plan

Add new lexer and parser test cases.

---------

Co-authored-by: Kartik Ganapathi <kartikganapathi@icloud.com>
2026-02-19 12:52:14 +00:00
Charlie Marsh 8fb1123c54 [ty] Visit parameters in source order (#23403)
## Summary

Previously, `Parameters::visit_source_order` used `ParametersIterator`
which visits in type-based order. For well-formed Python code, this
matches source order; but error recovery can produce ASTs like `def
foo(**kwargs, a):`.
2026-02-19 02:46:06 +00:00
kar-ganap 038235c6db [parser] Fix false syntax error for match-like annotated assignments (#23297)
## Summary

Fixes #22528.

The parser incorrectly treats `match[0]: int` and `match [x, y, z]:
{dict}` as the start of a `match` statement, producing a false syntax
error. The issue is that the parser checks only for a colon after the
subject expression to confirm a match statement, but annotated
assignments with `match` as a variable name also have a colon.

This PR adds a `self.peek() == TokenKind::Newline` guard to the colon
check, so that `match` is only treated as a keyword when the colon is
followed by a newline (which is required for a match statement's body).
When the colon is followed by something else (like a type annotation
value), it falls through to the soft-keyword-as-identifier path.

## Test Plan

All 42 existing match-related parser tests pass. No snapshot changes
needed. Clippy clean.

Reproduction from the issue:

```python
# These should parse without error:
match[0]: int
match [x, y, z]: {0: "a", 1: "b"}
```
2026-02-18 15:15:35 +05:30
Brent Westbrook 6ded4bed16 Fix f-string middle panic when parsing t-strings (#23232)
Summary
--

This PR fixes #23198 by copying Dylan's changes from #19183 to the other
`InterpolatedStringElementsKind` variant, in line with Micha's prophetic
comment
(https://github.com/astral-sh/ruff/pull/19183/changes#r2217325217).

Test Plan
--

Two new tests derived from the issue
2026-02-12 14:27:53 +00:00
Jack O'Connor 4712503c6d [ty] cargo insta test --force-update-snapshots (#22313)
Snapshot tests recently started reporting this warning:

> Snapshot test passes but the existing value is in a legacy format.
> Please run cargo insta test --force-update-snapshots to update to a
> newer format.

This PR is the result of that forced update.

One file (crates/ruff_db/src/diagnostic/render/full.rs) seems to get
corrupted, because it contains strings with unprintable characters that
trigger some bug in cargo-insta. I've manually reverted that file, and
also manually reverted the `input_file:` lines, which we like.
2026-01-05 07:55:47 -08:00
Matthew Mckee a2e0ff57c3 Run cargo sort (#22310) 2026-01-02 19:58:15 +00:00
Micha Reiser 014abe1ee1 [ty] Fix completion in decorators with missing declaration (#22177) 2025-12-25 15:05:47 +00:00
Micha Reiser eef403f6cf Revert "[ty] Fix completion in decorator without class or function definition" (#22176) 2025-12-24 17:27:21 +01:00
Micha Reiser adef89eb7c [ty] Fix completion in decorator without class or function definition 2025-12-24 16:57:36 +01:00
Bhuminjay SoniandBrent Westbrook 52849a5e68 [syntax-errors] Annotated name cannot be global (#20868)
## Summary

<!-- What's the purpose of the change? What does it do, and why? -->
This PR implements a new semantic syntax error where annotated name
can't be global
example
```
x: int = 1

def f():
    global x
    x: str = "foo"  # SyntaxError: annotated name 'x' can't be global
 ```

## Test Plan

<!-- How was it tested? -->
I have written tests as directed in #17412

---------

Signed-off-by: 11happy <soni5happy@gmail.com>
Signed-off-by: 11happy <bhuminjaysoni@gmail.com>
Co-authored-by: Brent Westbrook <brentrwestbrook@gmail.com>
2025-12-17 08:39:47 -05:00
Micha Reiser e2ec2bc306 Use datatest for formatter tests (#21933) 2025-12-13 08:02:22 +00:00
Micha Reiser aa27925e87 Skip over trivia tokens after re-lexing (#21895) 2025-12-11 10:45:18 +00:00
c722f498fe [flake8-bugbear] Catch yield expressions within other statements (B901) (#21200)
## Summary

This PR re-implements [return-in-generator
(B901)](https://docs.astral.sh/ruff/rules/return-in-generator/#return-in-generator-b901)
for async generators as a semantic syntax error. This is not a syntax
error for sync generators, so we'll need to preserve both the lint rule
and the syntax error in this case.

It also updates B901 and the new implementation to catch cases where the
generator's `yield` or `yield from` expression is part of another
statement, as in:

```py
def foo():
    return (yield)
```

These were previously not caught because we only looked for
`Stmt::Expr(Expr::Yield)` in `visit_stmt` instead of visiting `yield`
expressions directly. I think this modification is within the spirit of
the rule and safe to try out since the rule is in preview.

## Test Plan

<!-- How was it tested? -->
I have written tests as directed in #17412

---------

Signed-off-by: 11happy <soni5happy@gmail.com>
Signed-off-by: 11happy <bhuminjaysoni@gmail.com>
Co-authored-by: Brent Westbrook <brentrwestbrook@gmail.com>
Co-authored-by: Brent Westbrook <36778786+ntBre@users.noreply.github.com>
2025-12-03 12:05:15 -05:00
Bhuminjay Soni f68080b55e [syntax-error] Default type parameter followed by non-default type parameter (#21657)
## Summary

This PR implements syntax error where a default type parameter is
followed by a non-default type parameter.
https://github.com/astral-sh/ruff/issues/17412#issuecomment-3584088217


## Test Plan

I have written inline tests as directed in #17412

---------

Signed-off-by: 11happy <bhuminjaysoni@gmail.com>
Signed-off-by: 11happy <soni5happy@gmail.com>
2025-12-03 12:01:31 +05:30
Brent Westbrook 2250fa6f98 Fix syntax error false positives for await outside functions (#21763)
## Summary

Fixes #21750 and a related bug in `PLE1142`. We were not properly
considering generators to be valid `await` contexts, which caused the
`F704` issue. One of the tests I added for this also uncovered an issue
in `PLE1142` for comprehensions nested within async generators because
we were only checking the current scope rather than traversing the
nested context.

## Test Plan

Both of these rules are implemented as semantic syntax errors, so I
added tests (and fixes) in both Ruff and ty.
2025-12-02 21:02:02 +00:00
Micha Reiser 515de2d062 Move Token, TokenKind and Tokens to ruff-python-ast (#21760) 2025-12-02 20:10:46 +01:00
Dan ParizherandDhruv Manilawala 474b00568a [parser] Fix panic when parsing IPython escape command expressions (#21480)
## Summary

Fixes a panic when parsing IPython escape commands with `Help` kind
(`?`) in expression contexts. The parser now reports an error instead of
panicking.

Fixes #21465.

## Problem

The parser panicked with `unreachable!()` in
`parse_ipython_escape_command_expression` when encountering escape
commands with `Help` kind (`?`) in expression contexts, where only
`Magic` (`%`) and `Shell` (`!`) are allowed.

## Approach

Replaced the `unreachable!()` panic with error handling that adds a
`ParseErrorType::OtherError` and continues parsing, returning a valid
AST node with the error attached.

## Test Plan

Added `test_ipython_escape_command_in_with_statement` and
`test_ipython_help_escape_command_as_expression` to verify the fix.

---------

Co-authored-by: Dhruv Manilawala <dhruvmanila@gmail.com>
2025-11-24 05:40:27 +00:00
8529d79a70 [ty] name is parameter and global is a syntax error (#21312)
Co-authored-by: Alex Waygood <alex.waygood@gmail.com>
Co-authored-by: Brent Westbrook <36778786+ntBre@users.noreply.github.com>
2025-11-14 18:15:34 +00:00
Brent Westbrook 835e31b3ff Fix syntax error false positive on alternative match patterns (#21362)
Summary
--

Fixes #21360 by using the union of names instead of overwriting them, as
Micha suggested originally on #21104.

This avoids overwriting the `n` name in the `Subscript` by the empty set
of names visited in the nested OR pattern before visiting the other arm
of the outer OR pattern.

Test Plan
--

A new inline test case taken from the issue
2025-11-10 10:51:51 -05:00
cddc0fedc2 [syntax-error]: no binding for nonlocal PLE0117 as a semantic syntax error (#21032)
<!--
Thank you for contributing to Ruff/ty! To help us out with reviewing,
please consider the following:

- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title? (Please prefix
with `[ty]` for ty pull
  requests.)
- Does this pull request include references to any relevant issues?
-->

## Summary

<!-- What's the purpose of the change? What does it do, and why? -->

This PR ports PLE0117 as a semantic syntax error.

## Test Plan

<!-- How was it tested? -->
Tests previously written

---------

Signed-off-by: 11happy <soni5happy@gmail.com>
Co-authored-by: Brent Westbrook <36778786+ntBre@users.noreply.github.com>
Co-authored-by: Brent Westbrook <brentrwestbrook@gmail.com>
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
2025-11-05 19:13:28 +00:00
Luca Chiodini 69b4c29924 Consistently wrap tokens in parser diagnostics in backticks instead of 'quotes' (#21163)
The parser currently uses single quotes to wrap tokens. This is
inconsistent with the rest of ruff/ty, which use backticks.

For example, see the inconsistent diagnostics produced in this simple
example: https://play.ty.dev/0a9d6eab-6599-4a1d-8e40-032091f7f50f

Consistently wrapping tokens in backticks produces uniform diagnostics.
Following the style decision of #723, in #2889 some quotes were already
switched into backticks.

This is also in line with Rust's guide on diagnostics
(https://rustc-dev-guide.rust-lang.org/diagnostics.html#diagnostic-structure):

> When code or an identifier must appear in a message or label, it
should be surrounded with backticks
2025-10-31 11:59:11 -04:00
Brent Westbrook f0fe6d62fb Fix syntax error false positive on nested alternative patterns (#21104)
## Summary

Fixes #21101 by storing the child visitor's names in the parent visitor.
This makes sure that `visitor.names` on line 1818 isn't empty after we
visit a nested OR pattern.

## Test Plan

New inline test cases derived from the issue,
[playground](https://play.ruff.rs/7b6439ac-ee8f-4593-9a3e-c2aa34a595d0)
2025-10-30 13:40:03 -04:00
Takayuki Maeda 2c9433796a [ruff] Autogenerate TypeParam nodes (#21028) 2025-10-22 14:06:24 +02:00
Takayuki Maeda 6271fba1e1 [ruff] Auto generate ast Pattern nodes (#21024) 2025-10-22 08:24:34 +02:00
Micha ReiserandAlex Waygood 9d1ffd605c [ty] Implement go-to for binary and unary operators (#21001)
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
2025-10-21 19:25:41 +02:00
Bhuminjay SoniandBrent Westbrook 3dd78e711e [syntax-errors] Name is parameter and global (#20426)
## Summary

<!-- What's the purpose of the change? What does it do, and why? -->
This PR implements a new semantic syntax error where name is parameter &
global.

## Test Plan

<!-- How was it tested? -->
I have written inline test as directed in #17412

---------

Signed-off-by: 11happy <soni5happy@gmail.com>
Co-authored-by: Brent Westbrook <36778786+ntBre@users.noreply.github.com>
2025-10-21 16:51:16 +00:00
Micha Reiser 69ce064569 [ty] Fix completions at end of file (#20993) 2025-10-21 09:24:31 +00:00
Brent Westbrook 38c074e67d Catch syntax errors in nested interpolations before Python 3.12 (#20949)
Summary
--

This PR fixes the issue I added in #20867 and noticed in #20930. Cases
like this
cause an error on any Python version:

```py
f"{1:""}"
```

which gave me a false sense of security before. Cases like this are
still
invalid only before 3.12 and weren't flagged after the changes in
#20867:

```py
f'{1: abcd "{'aa'}" }'
           # ^  reused quote
f'{1: abcd "{"\n"}" }'
            # ^  backslash
```

I didn't recognize these as nested interpolations that also need to be
checked
for invalid expressions, so filtering out the whole format spec wasn't
quite
right. And `elements.interpolations()` only iterates over the outermost 
interpolations, not the nested ones.

There's basically no code change in this PR, I just moved the existing
check
from `parse_interpolated_string`, which parses the entire string, to
`parse_interpolated_element`. This kind of seems more natural anyway and
avoids
having to try to recursively visit nested elements after the fact in
`parse_interpolated_string`. So viewing the diff with something like

```
git diff --color-moved --ignore-space-change --color-moved-ws=allow-indentation-change main
```

should make this more clear.

Test Plan
--

New tests
2025-10-20 09:03:13 -04:00
Bhuminjay SoniandBrent Westbrook 7198e53182 [syntax-errors] Alternative match patterns bind different names (#20682)
<!--
Thank you for contributing to Ruff/ty! To help us out with reviewing,
please consider the following:

- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title? (Please prefix
with `[ty]` for ty pull
  requests.)
- Does this pull request include references to any relevant issues?
-->

## Summary

<!-- What's the purpose of the change? What does it do, and why? -->
This PR implements semantic syntax error where alternative patterns bind
different names

## Test Plan

<!-- How was it tested? -->
I have written inline tests as directed in #17412

---------

Signed-off-by: 11happy <soni5happy@gmail.com>
Co-authored-by: Brent Westbrook <brentrwestbrook@gmail.com>
2025-10-17 21:35:48 +00:00
Alex Waygood c424007645 Update usage instructions and lockfile for py-fuzzer script (#20940) 2025-10-17 15:57:17 +01:00
Bhuminjay Soni 73520e4acd [syntax-errors]: implement F702 as semantic syntax error (#20869)
<!--
Thank you for contributing to Ruff/ty! To help us out with reviewing,
please consider the following:

- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title? (Please prefix
with `[ty]` for ty pull
  requests.)
- Does this pull request include references to any relevant issues?
-->

## Summary

<!-- What's the purpose of the change? What does it do, and why? -->

This PR implements `F702`
https://docs.astral.sh/ruff/rules/continue-outside-loop/ as semantic
syntax error.

## Test Plan

<!-- How was it tested? -->
Tests are already previously written in F702

---------

Signed-off-by: 11happy <soni5happy@gmail.com>
2025-10-15 19:27:15 +00:00