Commit Graph
443 Commits
Author SHA1 Message Date
Micha Reiser 90f29abdf3 Reserve arena capacity as the parser consumes source
Start with a bounded reservation based on source length. When the row
buffer fills, use the observed node density to estimate how much space
the rest of the source will need.
2026-11-20 16:16:30 +00:00
Micha Reiser 043802b0d8 Build typed arena nodes in the Python parser
Carry the typed IDs returned by node constructors through parsing and
recovery. Parse source-ordered argument and parameter lists directly.
Keep ownership of the tree with the parsed module.
2026-11-20 16:16:30 +00:00
Micha Reiser 2766b66062 Validate positional and keyword arguments as the parser reads them 2026-11-20 16:16:30 +00:00
Micha Reiser 455e8a66f6 Reuse evaluation-order traversal and compare aligned call arguments directly
Existence searches over calls and class headers can inspect each slot
once in written order. For f(key=a, *b), the predicate sees a first.
2026-11-20 16:16:30 +00:00
Micha Reiser 97e1369fc4 Store call arguments in source order
Store positional and keyword arguments in one list for calls and class headers. Source-order visitors can walk the list directly, and evaluation-order visitors visit starred arguments before keywords.

Box keyword values so a positional argument does not carry the full size of a keyword entry.
2026-11-20 16:16:30 +00:00
Micha Reiser 31bd638de3 Read parameters in written order for syntax checks and searches
Traverse source order when each parameter is checked independently,
and inspect roles directly when locating the positional-only slash.

Expression searches can check annotations and defaults in the same
written order, including for recovered signatures.
2026-11-20 16:16:30 +00:00
Micha Reiser 0835f5fe98 Consume parameter roles directly in signature checks and fixes 2026-11-20 16:16:30 +00:00
Micha Reiser 35e592cec3 Filter keyword parameters in a single pass 2026-11-20 16:16:30 +00:00
Micha Reiser 4c25f9b780 Consume parameter roles together in the formatter, linter and ty 2026-11-20 16:16:30 +00:00
Micha Reiser 97ac08bf0c Store function and lambda parameters in source order 2026-11-20 16:16:30 +00:00
Micha Reiser a9d8532b29 Store class pattern arguments in source order 2026-11-20 16:16:30 +00:00
Micha Reiser ddbd91227c Visit mapping pattern entries once for captures and searches
For expression searches, test each key followed by its pattern. This lets
the search stop within an entry and avoids a second pass over the list.
2026-11-20 16:16:30 +00:00
Micha Reiser 7a1b37b5d4 Store mapping pattern entries and captures in source order 2026-11-20 15:17:27 +00:00
Micha Reiser 191c8ba42b Represent dictionary entries as Python AST nodes 2026-11-20 14:28:04 +00:00
astral-automations-bot[bot]andBrent Westbrook a7ba68b491 Bump version to 0.17.6 (#30578)
Co-authored-by: astral-automations-bot[bot] <305554984+astral-automations-bot[bot]@users.noreply.github.com>
Co-authored-by: Brent Westbrook <36778786+ntBre@users.noreply.github.com>
2026-11-19 10:32:47 -05:00
Shivansh ShuklaandAlex Waygood 7aae7a5312 Treat relative imports from .__future__ as ordinary imports (#30260)
Co-authored-by: Alex Waygood <alex.waygood@gmail.com>
2026-11-19 15:03:01 +00:00
0ebb77d35c Bump version to 0.17.5 (#30380)
Co-authored-by: astral-automations-bot[bot] <305554984+astral-automations-bot[bot]@users.noreply.github.com>
Co-authored-by: Brent Westbrook <36778786+ntBre@users.noreply.github.com>
Co-authored-by: Brent Westbrook <brentrwestbrook@gmail.com>
2026-11-12 17:26:23 +00:00
Micha Reiser 0b4359778d Parse hex escapes one ASCII byte at a time (#30374)
## Summary

On main, parsing `b"\x41"` calls `parse_unicode_literal(2)`. The x86-64
profiling build uses one function for widths 2, 4, and 8, so the loop
bound is a runtime argument. For each digit (`4`, then `1`), the loop
saves its source position and calls `next_char()` to decode and consume
a UTF-8 character before checking whether it is a hex digit.

This PR calls `parse_unicode_literal::<2>()` for `\x`, which lets the
compiler specialize the loop for two digits. This is the largest
measured improvement. The new loop also peeks at the next byte and
advances by one when it sees a valid ASCII hex digit. Only if that check
fails does it save the source position and call `next_char()`. For an
invalid escape such as `"\x4é"`, both versions consume `é` and report
its two-byte range.

Callgrind instruction counts for a whole uncached `ruff check` on
[JAX](https://github.com/jax-ml/jax/tree/55029b8f7e65adb79764103871ec0357fa440227/jax/_src/internal_test_util/export_back_compat_test_data).
On the compatibility fixtures, changing only the width to `const N`
saves 8.36 million of the 13.19 million instructions saved by the full
change.

| Workload | Main | `const N` only | `const N` + deferred error position
| Full change | Total reduction |
| --- | ---: | ---: | ---: | ---: | ---: |
| JAX export compatibility fixtures | 131,434,592 | 123,077,887 |
120,580,490 | 118,244,820 | 10.04% |
| JAX export implementation, compatibility fixtures, and export tests |
226,412,243 | 218,062,626 | 215,565,059 | 213,219,152 | 5.83% |

## Test Plan

The parser tests passed, and diagnostics matched main on valid,
malformed, and non-ASCII escapes.
2026-11-12 17:26:00 +01:00
astral-automations-bot[bot]andBrent Westbrook 05cdc36a45 Bump version to 0.17.4 (#30073)
Co-authored-by: astral-automations-bot[bot] <305554984+astral-automations-bot[bot]@users.noreply.github.com>
Co-authored-by: Brent Westbrook <36778786+ntBre@users.noreply.github.com>
2026-11-05 13:22:15 -05:00
SacrimentoandBrent Westbrook 888976c6ab [syntax-errors] Extend __debug__ write check to attribute targets (#27908)
## Summary

Ruff currently flags assigning to or deleting `__debug__` when it's a
bare name (`__debug__ = False`, `del __debug__`), but not when it's an
attribute target (`x.__debug__ = False`, `del x.__debug__`).

CPython forbids both assigning to `x.__debug__` (`SyntaxError` on every
supported Python version) and deleting `x.__debug__` only became a
`SyntaxError` in Python 3.14.

This PR extends the existing `__debug__` check to `Expr::Attribute`
targets, gated on the target Python version for the deletion case.

Part of #17412.

## Test Plan

Added inline test_err/test_ok fixtures alongside the new
`Expr::Attribute` check
Cross-checked the version gating logic and errors against real CPython
3.12/3.13/3.14.

---------

Co-authored-by: Brent Westbrook <brentrwestbrook@gmail.com>
2026-11-03 13:06:40 -05:00
Charlie Marsh 48630fbedd Allow match identifiers after a semicolon (#29760)
## Summary

`match` is a soft keyword. It can name a variable, a function, or the
target of an annotated assignment, including after a semicolon.

```python
first(); match = value
first(); match(value)
first(); match[0]: int
```

The parser reported a compound-statement error as soon as it saw the
`match` token. We now check the preceding separator when parsing a match
statement. If the token turns out to be an identifier, speculative
parsing discards the diagnostic. A genuine `match` statement on the same
logical line still produces the original error.
2026-10-27 17:10:45 -07:00
Charlie Marsh abee0da3f2 Accept a bare raise before a semicolon (#29761)
## Summary

A bare `raise` can end at a semicolon. For example, Python accepts this
handler and re-raises the current exception.

```python
try:
    work()
except ValueError:
    raise; report()
```

The parser tried to read the semicolon as the start of an exception
expression and reported a syntax error. We now recognize both a newline
and a semicolon as terminators for a bare `raise`.
2026-10-27 09:18:53 -07:00
astral-automations-bot[bot]andBrent Westbrook ba65abf8ba Bump version to 0.17.3 (#29745)
Co-authored-by: astral-automations-bot[bot] <305554984+astral-automations-bot[bot]@users.noreply.github.com>
Co-authored-by: Brent Westbrook <36778786+ntBre@users.noreply.github.com>
2026-10-26 20:14:27 -04:00
Charlie Marsh 5e90790d11 Compute the lexer keyword hash at compile time (#29677)
## Summary

Changing the lexer's keyword set now takes one edit to the Rust list.
The compiler finds a collision-free multiplier and builds the 64-slot
table, so we can remove the Python generator and its lockfile.

A two-byte keyword like `if` packs to `0x6669_0000_0000_0000`. Only the
low two bytes of the multiplier affect its slot. We place keywords from
shortest to longest, choosing each new byte of the multiplier without
disturbing the shorter keywords. The lexer still selects a slot with one
multiply and shift, then compares the full word.
2026-10-23 15:29:49 -04:00
Charlie Marsh e3304857ec Specialize ASCII identifier lexing (#29503)
## Summary

We give ASCII identifiers a dedicated path, avoiding Unicode bookkeeping
and checking string prefixes only when a one- or two-letter name reaches
a quote. Both paths share the existing batched ASCII scanner, preserving
fast scanning for Unicode names with long ASCII suffixes.

Stacked on #29463 to measure the incremental improvement. [CodSpeed
reports](https://github.com/astral-sh/ruff/pull/29503/checks?check_run_id=118300735426)
6.6–9.8% greater efficiency across all five lexer benchmarks, with no
reported regressions. We preserve the existing scanner and add no
inlining attributes or unsafe code.
2026-10-23 13:58:34 -04:00
Charlie Marsh de9d86e245 Speed up identifier and keyword lexing (#29463)
## Summary

We scan ASCII identifier continuations in sixteen-byte batches using a
byte lookup table, avoiding per-character decoding. Keyword lookup
filters by length and initial byte, then uses a compact hash table with
a full-word comparison to reject collisions. Non-ASCII suffixes retain
the existing Unicode path.
2026-10-23 13:58:33 -04:00
astral-automations-bot[bot]andBrent Westbrook 30516b42ab Bump version to 0.17.2 (#29637)
Co-authored-by: astral-automations-bot[bot] <305554984+astral-automations-bot[bot]@users.noreply.github.com>
Co-authored-by: Brent Westbrook <36778786+ntBre@users.noreply.github.com>
2026-10-22 13:54:24 -04:00
Micha Reiser 3e3f22dcc4 Make stacker an opt-in feature (#29505) 2026-10-19 18:02:19 -07:00
Charlie Marsh 85430a958b Share postfix parsing and reduce expression copies (#29480)
## Summary

We skip prefix-operator dispatch for identifiers and parse atoms together with their postfix chains. Keeping this work in `parse_primary_expression` makes the grouping-parenthesis flag local and avoids passing an intermediate `ParsedExpr` between atom and postfix parsing.
2026-10-19 20:35:16 +00:00
astral-automations-bot[bot] 4683270c69 Bump version to 0.17.1 (#29420)
Co-authored-by: astral-automations-bot[bot] <305554984+astral-automations-bot[bot]@users.noreply.github.com>
2026-10-15 10:56:41 -04:00
Charlie Marsh 2e696c7c86 [ty] Fix panic on completions past the end of a notebook's last cell (#29306)
## Summary

This PR fixes a completion panic in a notebook with an empty last cell.

To analyze a notebook, we combine its Python cells into one source
string, adding a newline after each cell:

```text
First cell:       "x = 1"       at offsets 0..5
Empty last cell:  ""            at offsets 6..6
Combined source:  "x = 1\n\n"  at offsets 0..7
```

We parse the cells separately, but the resulting module describes the
combined source. Previously, we ended that module at offset 6, where the
empty cell's content ends. Position conversion uses the combined source,
so a request for character 1 of the empty cell can resolve to offset 7.
That is the end of the source, but it was outside the module's range.
Looking up the completion position then panicked with `Range is not
contained within root`.

We now give the module the range 0..7, as we would when parsing the same
source as a whole. This makes positions at the end of the combined
source safe to look up. #29307 separately corrects position conversion
so the same request resolves to offset 6, the end of the requested
cell's content.

Closes https://github.com/astral-sh/ty/issues/4687.
2026-10-13 09:15:13 -04:00
astral-automations-bot[bot]andBrent Westbrook 5b27c1ccfa Bump version to 0.17.0 (#29259)
Co-authored-by: astral-automations-bot[bot] <305554984+astral-automations-bot[bot]@users.noreply.github.com>
Co-authored-by: Brent Westbrook <brentrwestbrook@gmail.com>
2026-10-09 13:45:29 -04:00
Brent Westbrook 8f362bf4cb Integrate vendored Unicode names into Ruff
Use the vendored runtime and generator instead of the registry dependency,
providing Unicode 17 names with Python-compatible matching. Update workspace
dependencies and tool configuration, add parser regressions, and document how
to update Unicode data when supporting a new Python version.
2026-10-09 12:26:57 -04:00
Charlie Marsh 598f4bdfd5 Adopt Rust 1.97 integer APIs and Cargo warning policy (#29048)
## Summary

- Use the integer bit APIs stabilized in Rust 1.97 for packed node
indices, token capacity, rule iteration, and type-variable support sets.
- Use Cargo's warning policy in our native and WASM Clippy checks,
keeping warnings fatal without changing compiler flags and invalidating
the build cache. Update the contributing guides to match.

Preserves the Rust 1.97 MSRV. The warning policy change follows
https://github.com/astral-sh/uv/pull/22122.
2026-10-01 14:01:44 -04:00
astral-automations-bot[bot]andBrent Westbrook 3265ed1f94 Bump version to 0.16.10 (#29055)
Co-authored-by: astral-automations-bot[bot] <305554984+astral-automations-bot[bot]@users.noreply.github.com>
Co-authored-by: Brent Westbrook <brentrwestbrook@gmail.com>
2026-10-01 13:40:09 -04:00
astral-automations-bot[bot]andBrent Westbrook 0be08a206f Bump version to 0.16.9 (#28882)
Co-authored-by: astral-automations-bot[bot] <305554984+astral-automations-bot[bot]@users.noreply.github.com>
Co-authored-by: Brent Westbrook <36778786+ntBre@users.noreply.github.com>
2026-09-24 16:21:50 -04:00
Micha Reiser 689dc812f3 Rename TokenKind::Name to Identifier (#28667) 2026-09-17 15:47:10 +02:00
astral-automations-bot[bot]andBrent Westbrook 62914c4b9b Bump version to 0.16.8 (#28648)
Co-authored-by: astral-automations-bot[bot] <305554984+astral-automations-bot[bot]@users.noreply.github.com>
Co-authored-by: Brent Westbrook <36778786+ntBre@users.noreply.github.com>
2026-09-16 11:31:02 -04:00
Charlie Marsh 7b13f01ba1 [ty] Reduce retained AST memory by shrinking expressions (#28335)
## Summary

We reduce `Expr` from 64 to 56 bytes on 64-bit targets. The largest
variants determine the enum's size, so shrinking those variants saves
eight bytes in every `Expr` value, including expressions in the parsed
modules retained by ty.

The changes are:

- Store positional call arguments in `ThinVec`, reducing `Arguments` and
`ExprCall` by eight bytes.
- Store all comparison operands in one boxed slice. This removes the
separate left-operand allocation and reduces `ExprCompare` by eight
bytes while keeping operators in a boxed slice.
- Store dictionary-comprehension generators in a boxed slice, reducing
`ExprDictComp` by eight bytes.
- Store f-string and t-string elements in boxed slices, and represent a
single f-string directly as `FString`. This keeps single parts inline
and reduces `ExprFString` and `ExprTString` by eight bytes.

The parser, AST visitors, formatter, lint rules, and type inference use
the updated layouts. Size assertions cover the 56-byte enum and its
affected variants.
2026-09-10 15:39:14 -04:00
astral-automations-bot[bot]andBrent Westbrook b5dba861cc Bump version to 0.16.7 (#28496)
Co-authored-by: astral-automations-bot[bot] <305554984+astral-automations-bot[bot]@users.noreply.github.com>
Co-authored-by: Brent Westbrook <36778786+ntBre@users.noreply.github.com>
2026-09-10 13:47:30 -04:00
Micha Reiser a99b96b9ac Expose hidden lexer token metadata accessors (#28443) 2026-09-09 09:02:54 +02:00
Charlie Marsh e7adf82ff0 Reuse parser name lookups when interning (#28399)
Use hashbrown's borrowed-entry insertion for long parser names so a
missing name reuses its lookup when it is inserted. Construct the
heap-backed `Name` only on a miss, while preserving the inline bypass
and shared allocations for repeated long names.
2026-09-07 19:00:20 -04:00
Brent Westbrook 22f65a2ab5 Bump 0.16.6 (#28280) 2026-09-03 12:36:42 -04:00
Brent Westbrook 0ecff87790 Validate unary expressions when parsing (#28233)
Summary
--

While reviewing #28231, Codex found a preexisting bug shared with our
unary minus handling. Namely, CPython emits an error for code like this,
where the unary expression is part of a longer expression:

```pycon
>>> match foo:
...     case -1**2: ...
...
  File "<python-input-0>", line 2
    case -1**2: ...
           ^^
SyntaxError: invalid syntax
```

but Ruff does not:

```console
$ ruff check --ignore ALL - <<EOF
match foo:
    case -1**2: ...
EOF
All checks passed!
```

We now verify that the parsed unary expression is a number literal and
emit an error if not.

Test Plan
--

New inline snapshots
2026-09-02 09:34:51 -04:00
Brent Westbrook eff1c88d70 Allow unary plus in match patterns on Python 3.15 (#28231)
Summary
--

As added in https://github.com/python/cpython/pull/148566, CPython now
allows unary plus in match
patterns, mirroring the support for unary minus. We already had a
special case for this for improved
error reporting and recovery, so I just swapped out the `add_error` call
for
`add_unsupported_syntax_error` with a new error kind.

Part of https://github.com/astral-sh/ruff/issues/28206

Test Plan
--

New inline parser tests
2026-09-02 09:16:08 -04:00
Brent Westbrook 9e4938c4a6 Bump 0.16.5 (#28110) 2026-08-27 12:09:07 -04:00
Alex Waygood d5a6394425 Fix compiler warnings seen in CI jobs (#28066) 2026-08-26 14:21:42 +01:00
Charlie Marsh 88d56e5feb Use stabilized Rust pattern assertion macros (#27955)
## Summary

- Use Rust 1.96's stabilized `assert_matches!` and
`debug_assert_matches!` macros throughout Ruff and ty.
- Include mismatched values in assertion failures and remove redundant
manual debug formatting.
- Preserve existing assertions for values that do not implement `Debug`.
2026-08-21 09:29:39 -04:00
Brent Westbrook 11c76bf48f Bump 0.16.4 (#27937) 2026-08-20 13:14:29 -04:00
Augustin BouverotandMicha Reiser 3ea20f6161 Align t-string/bytes mix error with CPython 3.14 (#27766)
Co-authored-by: Micha Reiser <micha@reiser.io>
2026-08-19 12:35:23 +02:00