Store positional and keyword arguments in one list for calls and class headers. Source-order visitors can walk the list directly, and evaluation-order visitors visit starred arguments before keywords.
Box keyword values so a positional argument does not carry the full size of a keyword entry.
## Summary
Today, the formatter tokenizes source forwards and backwards to check
whether a match pattern is parenthesized, while expression formatting
uses an existing index of parenthesized ranges.
This PR uses the same index for patterns, removing the last production
use of `BackwardsTokenizer`.
## Summary
Today, range formatting tokenizes source backwards before a clause body
to find the header's colon.
This PR finds the colon in the parser's existing tokens instead.
Range-format output is unchanged. Retired instructions decrease by 0.36%
on 10,000 clauses and 1.72% on 10,000 commented clauses, with no
meaningful regression in single-header or whole-file formatting.
## Summary
Today, the formatter tokenizes source forwards and backwards to place
comments immediately after the opening bracket of subscripts and slices.
This PR shares a direct check for those comments across both cases.
Testing: All formatter tests pass, and paired instruction counts improve
for ordinary subscriptions while remaining near baseline for slices and
controls.
Summary
--
There were several new cargo and clippy lints here, so I tried to break
up the changes into several
commits:
- deleting some unused workspace dependencies, these looked kind of
intentional as they also had
some shear ignores, but it seems okay to delete those too?
- allowing the [redundant_homepage] lint. We could delete a lot of
these, but the ones in the `ruff`
and `ty` crates seem necessary for maturin, as I noted in a comment.
- fixing the [missing_lints_inheritance] lint. It seems that several of
our crates don't inherit our
workspace lints. For now I just included the `allow` for
`redundant_homepage`, but we could follow
up by inheriting the full table and fixing all the additional lints that
produces. (This commit is
actually combined with the previous one)
- allowing cargo's new [unused_dependencies] lint, which seems redundant
with `shear` and had some
false positives on our benchmark dependencies. Codex tried to work
around this with some
`cfg`-gated imports in the benchmark crate, but ignoring this seemed
better for now, at least
while we're still using `shear`.
- running `clippy --fix` on the remaining lint errors, with one manual
intervention where Clippy
dropped some of our `cfg` handling. These new diagnostics were mostly
around returning boolean
expressions.
- renaming `NodeIndex::_NONE` to avoid [used_underscore_items]
There are also some new APIs in 1.98 that we can use now. I'll follow up
with those separately.
[redundant_homepage]:
https://doc.rust-lang.org/cargo/reference/lints.html#redundant_homepage
[missing_lints_inheritance]:
https://doc.rust-lang.org/cargo/reference/lints.html#missing_lints_inheritance
[used_underscore_items]:
https://clippy.rust-lang.org/master/#used_underscore_items
[unused_dependencies]:
https://doc.rust-lang.org/cargo/reference/lints.html#unused_dependencies
## Summary
Formatting and printing the top-level modules in CPython’s standard
library took 3.9% less time in a local benchmark (125.7 ms to 120.8 ms),
or 2.4% less with parsing included. CodSpeed reports a 5.1% efficiency
improvement on the [`large/dataset.py` formatter
benchmark](https://github.com/astral-sh/ruff/pull/29841#issuecomment-6515803489).
Today the formatter emits a source slice for `a`, a dot token, and a
source slice for `b` just to print `a.b`. We now emit the whole
expression as one slice when the dot touches both names, the first name
is unparenthesized, and there are no comments or call-chain breaks to
handle. Reading directly from the source also preserves Unicode
spellings like `𝓪.𝓫`.
For `factory().attr`, we still format the call and handle any line break
before the dot, but we can emit `.attr` as one slice. This follows the
approach in [Oxc #28462](https://github.com/oxc-project/oxc/pull/28462).
## Summary
When source maps are disabled and the Bloom filter rules out comments on
a node, the formatter can call `fmt_fields` directly. This skips cloning
the comment handle and writing empty leading and trailing comments.
Nodes with possible comments or source maps still take the existing
path.
Compared with #29852, the fast path used 4.44% fewer instructions across
Ruff’s five formatter benchmarks and 2.79% fewer across ten Python
standard-library modules. Synthetic assignments with leading and
trailing comments used 0.13% more. These are medians from 21 paired
arm64 trials that include trivia construction and printing.
## Test Plan
The formatter tests passed, and the parent and this change produced
identical formatted text and source-map markers across the comparison
corpora.
## Summary
Most nodes have no attached comments. A 256-byte Bloom filter lets the
formatter skip the comment map lookup when a node's address maps to a
clear bit. On a hit, the formatter still checks the map.
Building and printing pre-parsed documents on arm64 took 4.82% fewer
instructions across five Ruff benchmark fixtures and 3.40% fewer across
ten Python standard-library modules, measured as the median of 21 paired
runs against `main`. A synthetic workload with leading and trailing
comments on every assignment took 0.38% more instructions.
## Test Plan
The formatter tests passed, including a forced filter collision, and all
benchmark inputs produced byte-for-byte identical output.
Summary
--
This addresses the suggestion from [#29635]. `DisplayDiagnosticConfig`
previously determined the
`DiagnosticFormat` for diagnostics coming from both Ruff and ty, which
meant that it had to have a
shared enum between the two, despite each program supporting a different
subset of the available
formats. This PR eliminates the shared `DiagnosticFormat` and dispatch
through `DisplayDiagnostics`,
leaving Ruff and ty to construct `Renderer`s directly and do their own
dispatch on the output
formats they actually support. This will be especially helpful in
#29635, where we want to provide
slightly different arguments to each new `SarifRenderer`.
The `Renderer` trait itself isn't strictly necessary, but it made the
calls on the Ruff side a lot
neater since the trait provides a default `write` method based on the
required `render` method.
Codex also threw in a default `display` method that helps in a couple of
places too.
Test Plan
--
Existing tests
[#29635]:
https://github.com/astral-sh/ruff/pull/29635#discussion_r4352505588
Summary
--
This PR updates our default Python version to 3.11, in line with 3.10
reaching [EOL] next month, and
updates our latest Python version to 3.15. I followed last year's PR
(#20725) to find everywhere
that needed to change. Fortunately there were not as many test changes
needed as last year's
deferred annotation issues.
I also updated the language classifier in our pyproject.toml.
[EOL]: https://devguide.python.org/versions/#supported-versions
Test Plan
--
Existing tests with updated snapshots. Most of the updates were to
snapshots for TC001, TC002, and
TC003, which now default to the lazy-import fix.
After updating the snapshots, I tried to make sure we still had coverage
for whatever changed. That
meant adding a few more mdtests for the TC rules, and one small mdtest
for RUF017 to restore
coverage for the `functools.reduce` fix.
Summary
--
I wasn't really planning to do a full stabilization pass, but I quickly skimmed through `preview.rs`
and remembered that the isort pragma change was kind a bug fix but technically breaking and thought
we might want to stabilize it.
This PR stabilizes the isort pragma comment width handling introduced in #27313, along with the
related pragma handling for `E501` and `W505` from #24071. There was a comment about also
stabilizing the related formatter preview feature on the function from #24071, so that is stabilized
here too.
I'm fine holding off on this, especially if the formatter change is a better fit for the 2027 style
guide. Like I said above, I mostly wanted to stabilize the isort change, but that snowballed a bit
since the helpers were shared.
Test Plan
--
Updated existing snapshots
## Summary
We add accessors and iteration methods to `ExprCompare` and use them
throughout AST visitors, formatting, lint rules, and type inference. For
`a < b <= c`, `iter` yields `(a, Lt, b)` and `(b, LtE, c)`. `as_single`
returns a triple only when there is exactly one operator and two
operands, so callers can match an operator and both operands together
while rejecting comparison chains. Rules that only inspect operands use
the existing operand slice directly.
The `first_operand`, `first_operator`, and `second_operand` accessors
assert the parser's existing guarantee that a comparison has at least
one operator and two operands. Error recovery synthesizes missing
operands; manually constructed ASTs must uphold the same invariant.
## Summary
We reduce `Expr` from 64 to 56 bytes on 64-bit targets. The largest
variants determine the enum's size, so shrinking those variants saves
eight bytes in every `Expr` value, including expressions in the parsed
modules retained by ty.
The changes are:
- Store positional call arguments in `ThinVec`, reducing `Arguments` and
`ExprCall` by eight bytes.
- Store all comparison operands in one boxed slice. This removes the
separate left-operand allocation and reduces `ExprCompare` by eight
bytes while keeping operators in a boxed slice.
- Store dictionary-comprehension generators in a boxed slice, reducing
`ExprDictComp` by eight bytes.
- Store f-string and t-string elements in boxed slices, and represent a
single f-string directly as `FString`. This keeps single parts inline
and reduces `ExprFString` and `ExprTString` by eight bytes.
The parser, AST visitors, formatter, lint rules, and type inference use
the updated layouts. Size assertions cover the 56-byte enum and its
affected variants.
Summary
--
This PR updates our ruff-pre-commit dependency to 0.16.5, which supports
the new category selectors,
and then enables `lint.preview` and the new default categories so that
we can dogfood them on our
code. There were 66 diagnostics initially, which I also fixed or
ignored:
```
28 FURB113 [ ] repeated-append
12 ASYNC240 [ ] blocking-path-method-in-async-function
6 RUF105 [*] noqa-comments
5 PLW1510 [ ] subprocess-run-without-check
3 PLW0108 [ ] unnecessary-lambda
2 EXE001 [ ] shebang-not-executable
2 RUF039 [*] unraw-re-pattern
1 B024 [ ] abstract-base-class-without-abstract-method
1 FURB142 [*] for-loop-set-mutations
1 PERF102 [ ] incorrect-dict-iterator
1 PLC0207 [*] missing-maxsplit-arg
1 PLW0127 [ ] self-assigning-variable
1 TRY004 [ ] type-check-without-type-error
1 PLW0120 [*] useless-else-on-loop
1 TRY201 [ ] verbose-raise
```
Of these, I ignored `repeated-append` and `blocking-path-method-in-async-function`. I think they are
still reasonable defaults, just less useful/important in our uses.
0.16.5 also includes https://github.com/astral-sh/ruff-pre-commit/pull/159, so I removed `markdown` from our main `ruff-format` hook, leaving it for the later mdtest formatting stage.
I also had to make a few changes related to config discovery in tests:
- move our cached ecosystem checkouts to `$RUNNER_TEMP` instead of inside of our checkout
- convert some CLI tests to use the `CliTest` helpers
- add `configurationPreference: editorOnly` to some notebook tests
In these cases, our own Ruff settings were being inherited by test code. This likely affects many other tests, but I only updated the obviously failing ones for now. I'm least confident about the notebook tests, but as far as I can tell from the history (https://github.com/astral-sh/ruff/pull/11944, https://github.com/astral-sh/ruff/pull/22810), the specific diagnostics are not the important parts being asserted.
Test Plan
--
CI on this PR
## Summary
- Use Rust 1.96's stabilized `assert_matches!` and
`debug_assert_matches!` macros throughout Ruff and ty.
- Include mismatched values in assertion failures and remove redundant
manual debug formatting.
- Preserve existing assertions for values that do not implement `Debug`.
## Summary
This PR reduces the size of `Expr` from 72 to 64 bytes by storing only
the start offset on `ExprCall` (64 to 56 bytes) and boxing the uncommon
implicitly concatenated string representation in `ExprStringLiteral` (64
to 48 bytes). Both changes are required to reduce the size of the
containing `Expr` enum.
`ExprCall` derives its end offset from its argument list; parser and
error-recovery paths assert that their ranges share the same end.
---------
Co-authored-by: Charlie Marsh <charliemarsh@openai.com>
## Summary
This is the first PR out of 3. It extends `ty_python_semantic` and lower
crates to support checking files that use different Python versions
while using a single Salsa db.
The motivation for supporting different Python versions is that scrolls
need to be checked independently. Initially, I planned to implement
scroll support in a similar way to how we support multiple workspace
folders in VS Code by using multiple dbs, but it turned out, that
they're different enough. The main difference is that scrolls are
checked as part of their enclosing project, and the diagnostics from
scrolls are merged with the diagnostics from the project. If we used
separate scroll dbs, then the scroll diagnostics would use `File`s
pointing into the scroll db and all other diagnostics would use `File`s
pointing into the main db. There's nothing that would allow us to tell
to which db any such `File`s belonged, and `File`s from different dbs
can even share the same value. There's also no lifetime that binds them
to one particular db because they're inputs. We could have decided to
rewrite the `Diagnostic`s for scrolls and convert the scroll db `File`s
to project db `File`s, but that felt very brittle. The other
disadvantage was that it became very difficult to tell which query is
cached because you now also need to reason about whether the current
query is cached for a specific argument and db (and not just the
argument).
This is why I decided to explore whether we could support checking files
with different Python version, search paths, and platforms, and this PR
is the first step towards this. It adds support for checking files with
different Python versions. Specifically, this PR removes the
`db.python_version` from `ruff_db::Db` and lifts it up to
`ty_project::Db`, proving that no semantic crate depends on it.
The first step is to make `parsed_module` Python version aware. That is,
instead of calling `db.python_version`, parametrize the query by file
and Python version. For this, I introduced a new `PythonFile` which is a
`(File, PythonVersion)` tuple. We now attach this `PythonFile` to
`Scope` (instead of `File`), because `::node` accessors need it to
retrieve their node. That in turn required `semantic_index` to take a
`PythonFile` instead of `File` as well. This was a mostly smooth ride.
Supporting multiple Python versions in the module resolver required
adding a `PythonVersion` argument to most queries. The next PR will
replace the `PythonVersion` with a new `ResolverSettings` (or similar
name) interned struct that wraps both Python version and search paths.
This was also mostly straightforward.
The hard part was to remove all `db.python_version` calls from type
inference and type operations. Today, we call `db.python_version` where
we need it, but we can't do this anymore. Instead, we have to pass the
Python version down to each function that needs it.
Now, this is not true for all functions. Functions that have access to a
scope (via `Definition`, `Expression` or similar) could retrieve the
Python version from there (because `Scope` stores a `PythonFile`). I
decided against that and, instead established the following policy:
* Explicitly pass the Python version to all semantic functions.
* The only exception to this are Salsa queries that have exactly one
argument other than `db`. For those, retrieving the Python version from
their `Scope` is allowed for performance reasons.
Ultimately what that means: The context should be initialized at the
query boundary and then not change, unless we cross into another Salsa
query (but assert that the python version does not change).
There are two motivations for this:
* I think it's easier to reason about what Python version a type
operation uses if there's a consistent way of retrieving it (always as
argument)
* At query boundaries, we can assert that the explicitly passed Python
version and the `Scope`'s Python version are identical, before calling
into the query (debug assert).
However, supporting different Python versions now requires that we
parametrize all type operations with the Python version under which the
operation should be performed (in the future, this will be Python
version, search paths, and platforms).
Now, passing `db` and `PythonVersion` as separate arguments not only
results in a huge diff (add another 7k changes on top of the current
diff), is cumbersome, it also is slower. That's why I this PR introduces
a new `SemanticContext` that wraps a `db` and a `PythonVersion` (Happy
to discuss names). The variable is consistently referred to as `ctx` to
reduce the diff (using `context` makes it more likely that lines break).
## Next steps
* Change `SemanticContext` to store a `db` and `SearchPathSettings` to
support different search paths
* After that, change `SemanticContext` to store a `db` and a
`ProgramFile` (which is a `(File, Program)` tuple.
* Use the new infrastructure for scroll support
## Performance
There's currently a 2-3% walltime regression. Codex reports a lower
local regression.
There are some options that we can explore if we think this regression
is too big. However, it would make the model a little more muddy. The
most promising is probably to have two versions of very hot queries.
E.g. `is_redundant_with` could have a `is_redundant_with_project` and
`is_redundant_with_script` query. `is_redundant_with_project` would use
`db.python_version` internally (yes, we'd have to add that back). I
don't like that. Not only does it reintroduce `db.python_version` and
make it very easy that we use it somewhere where we should not.
## Review
I, honestly, tried to split this PR into smaller, reviewable commits.
But I don't think I succeeded. But I also don't think it makes sense to
review this commit line by line. I'm sure there are places where we
still pass `db` where we should pass a `ctx`. I'm sure there are some
call sites where we should retrieve the `python_version` in a different
way. But I'm not sure it's worth finding all these places in a review.
Instead, I'd focus on the overall change. What's your reaction on
`SemanticContext`? That we now need to pass it everywhere? What about my
policy that we use a single `SemanticContext` within a query? Should we
relax this constraint, so that `SemanticIndex` must be passed in fewer
places, if the `PythonVersion` can be retrieved at the relevant call
site (because it has `Scope`)? The downside of that is that it becomes
more difficult to assert that we don't mix programs in a single
inference, but is this something we should really be concerned about? Do
you think the performance regression is to big for this feature?
## Notes
* Make associated tracked methods that don't take `python_version`
normal methods with an inner query instead. The method should take a
`python_version`, and we assert (before calling the query), that the
target's (self's) `python_version` matches the passed python version
## Summary
This PR uses Hawk 0.1.11 to narrow internal declarations to the
visibility they actually need.
Here, I've intentionally _only_ made visibility changes; unused API
removals are split into follow-up PRs. In total, 1,201 declaration
visibility modifiers are narrowed. Public generated formatter APIs and
other declarations whose narrower visibility would activate dead-code or
API-shape lints remain unchanged.