Fixed a link to `frozen.discard` in the Python docs this was a valid
link in older Python docs but has been broken.
While I was there I noticed a few other uncanonical links so replaced
them with more canonical versions and removed some links to specific
Python versions that are unnecessary.
## Summary
This PR reduces the size of `Expr` from 72 to 64 bytes by storing only
the start offset on `ExprCall` (64 to 56 bytes) and boxing the uncommon
implicitly concatenated string representation in `ExprStringLiteral` (64
to 48 bytes). Both changes are required to reduce the size of the
containing `Expr` enum.
`ExprCall` derives its end offset from its argument list; parser and
error-recovery paths assert that their ranges share the same end.
---------
Co-authored-by: Charlie Marsh <charliemarsh@openai.com>
## Summary
This PR uses Hawk 0.1.11 to narrow internal declarations to the
visibility they actually need.
Here, I've intentionally _only_ made visibility changes; unused API
removals are split into follow-up PRs. In total, 1,201 declaration
visibility modifiers are narrowed. Public generated formatter APIs and
other declarations whose narrower visibility would activate dead-code or
API-shape lints remain unchanged.
## Summary
This is a first step towards #20411. After this, we would work to either
upstream changes or adjust things within Ruff.
Performance improvements have already been upstreamed
(https://github.com/rust-lang/annotate-snippets-rs/pull/437,
https://github.com/rust-lang/annotate-snippets-rs/pull/440)
Re-implemented the following from the fork:
- Hidden snippets
(https://github.com/rust-lang/annotate-snippets-rs/issues/448)
- Renderer-level disabling of hyperlinks
- Jupyter cell indices
(https://github.com/rust-lang/annotate-snippets-rs/issues/449)
- Fixable mark
(https://github.com/rust-lang/annotate-snippets-rs/issues/450)
- `lineno_offset` (maybe not needed if #20648 is taken care of)
- anonymized lines affects origin
(https://github.com/rust-lang/annotate-snippets-rs/issues/451)
- cut indicator (maybe not needed if we can switch to unicode renderer)
Additional hacks to minimize behavior change include
- Adjust upstream's hidden severity behavior to match Ruff's
(https://github.com/rust-lang/annotate-snippets-rs/issues/452)
- How newlines are annotated
(https://github.com/rust-lang/annotate-snippets-rs/issues/454)
- To not indent for line numbers when none will be present
(https://github.com/rust-lang/annotate-snippets-rs/issues/453)
- Enabling `simd` by default
All of the above represents work that needs to be resolved to address
#20411.
`lib.name` hack is used to simplify maintenance with the assumption that
we'll continue working towards #20411.
Behavior changes that were included
- Tweak when implicit padding lines are present
- Fix Origin's line/column is now derived from the primary annotation
- Make consistent the rendering of primary/context annotations
- Fix snippets to show up for empty source (but may degrade #24528)
- Fix handling of term width
- Fix rendering of annotations between newline and first character
(granted, maybe the caller is wrong)
- Tweak multiline annotation markers
New or upcoming features annotate-snippets features that we will be able
to leverage by moving this forward
- `AnnotationKind::Visible` to improve the quality of built-in
code-folding if Ruff were to switch to it
- `DecorStyle::Unicode`
- *(upcoming)* No-graphiscs mode for screen readers (may also be useful
for AIs)
Deferred:
- Moving related `Group`s into the same `Report`
- Using `secondary_title` for any `Group` after the first in a `Report`
- Using a `Snippet`, instead of titles, for "virtual code" like
https://github.com/astral-sh/ruff/pull/27033#discussion_r3622727236
- Switching hidden snippets to `Origin`
## Test Plan
Within cpython, I ran
```console
$ hyperfine -i "../../ruff/target-baseline/release/ruff check" "../../ruff/target-update/release/ruff check" -N -m20 --output=null
Benchmark 1: ../../ruff/target-baseline/release/ruff check
Time (mean ± σ): 27.5 ms ± 1.1 ms [User: 33.0 ms, System: 108.4 ms]
Range (min … max): 25.0 ms … 31.7 ms 94 runs
Warning: Ignoring non-zero exit code.
Benchmark 2: ../../ruff/target-update/release/ruff check
Time (mean ± σ): 28.1 ms ± 2.0 ms [User: 33.4 ms, System: 105.7 ms]
Range (min … max): 24.8 ms … 38.9 ms 102 runs
Warning: Ignoring non-zero exit code.
Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet system without any interferences from other programs. It might help to use t
he '--warmup' or '--prepare' options.
Summary
../../ruff/target-baseline/release/ruff check ran
1.03 ± 0.08 times faster than ../../ruff/target-update/release/ruff check
```
Before:
<img width="486" height="583" alt="image"
src="https://github.com/user-attachments/assets/2d94dabf-b28d-48c1-b17c-52aac59fd57e"
/>
After:
<img width="486" height="583" alt="image"
src="https://github.com/user-attachments/assets/b388ada3-e3d7-4ba0-b90b-1330eb81f4be"
/>
## Summary
This PR changes `ruff_python_ast::Name` from a 24-byte `CompactString`
to a 16-byte immutable `CharStr` from
[`astral-sh/char_str`](https://github.com/astral-sh/char_str).
`char_str` is an Astral-maintained fork of
[`lean_string`](https://github.com/ryota2357/lean_string) that provides
compact owned strings with mutable and immutable variants: growable
`CharString` for construction and immutable, exactly allocated `CharStr`
for storage. (It also provides Salsa and get-size features.)
Names up to 16 UTF-8 bytes remain inline, static names remain
allocation-free, and longer names use an exactly sized,
reference-counted allocation with no unused capacity.
Removing eight bytes from every `Name` also shrinks common AST nodes.
For example, `Identifier` and `ExprName` drop from 40 to 32 bytes, while
`ExprAttribute`, `StmtFunctionDef`, `StmtClassDef`, and several other
variants each drop by eight bytes.
The parser interns names that exceed the inline capacity, so repeated
longer names share one backing allocation within a parser invocation.
## Why 16 bytes?
Across the full `mypy_primer` corpus, there 20.8 million name
occurrences:
| UTF-8 length | Occurrences | Share | Unique name/file pairs | Share |
|---|---:|---:|---:|---:|
| 1–16 bytes | 18,955,757 | 91.02% | 3,192,424 | 79.42% |
| 17–24 bytes | 1,348,894 | 6.48% | 529,826 | 13.18% |
| 25+ bytes | 522,376 | 2.51% | 297,575 | 7.40% |
| Total | 20,827,027 | 100% | 4,019,825 | 100% |
The 16-byte cutoff keeps 91% of occurrences inline while saving eight
bytes for every occurrence. The 1–16-byte bucket alone removes 151.6 MB
of handle storage across the corpus.
The tradeoff is the 17–24-byte bucket: `CompactString` stored those
names inline, while `CharStr` allocates once per unique name within a
file. But the interning amortizes that cost when a name repeats. Names
longer than 24 bytes already allocated on `main`; for those names, this
change both reduces the handle size and deduplicates repeated backing
allocations.
## Memory impact
Accounting for the observed per-file multiplicities and structural
sharing, modeled `Name` storage across the corpus changes as follows:
| | `main` | This PR | Difference |
|---|---:|---:|---:|
| `Name` handles and backing storage | 526.18 MB | 366.27 MB | -159.91
MB (-30.4%) |
The current [instrumented memory
report](https://github.com/astral-sh/ruff/pull/26594#issuecomment-4906725175)
shows a consistent reduction in total retained memory:
| Project | `main` | This PR | Difference |
|---|---:|---:|---:|
| flake8 | 29.21 MB | 28.71 MB | -1.69% |
| trio | 71.01 MB | 70.01 MB | -1.41% |
| sphinx | 168.81 MB | 166.82 MB | -1.18% |
| prefect | 454.75 MB | 449.06 MB | -1.25% |
Within those totals, `parsed_module` decreases by roughly 2.1% and
`semantic_index` by 2.4–2.7%. Several name-heavy collections decrease by
considerably more.
## Summary
Use scratch buffers instead of allocating new `Vec`s and shrinking them
when constructing the AST node. While this doesn't remove the need for
copying the elements, it reduces the allocations within our parser
(`shrink_to_fit` often needs to allocate a new buffer and move all
elements). One shared scratch buffer per node type is sufficient,
because our parser is recursive (more nested parse rule always push to
the end, and parse function drain from the end as they complete).
## Performance
Very consistent perf improvement
## Codspeed memory regression
This can cause the parser's peak memory to increase, because we retain
the scratch buffer allocation after parsing a clause completed. However,
that memory is only temporary and released as soon as the parser
completes. Most project-level memory reports show a memory reduction.
It's only specific parser benchmarks that show a memory increase
## Summary
The lexer checks `pending_indentation` for every token. `Option::take`
writes `None` even on the common path where no indentation is pending,
so we now read the field first and clear it only when a value is
present. This avoids a redundant write in the hot lexer loop while
preserving the existing indentation behavior.
This PR is stacked on #26765 so that CodSpeed can measure the
pending-indentation optimization independently.
---------
Co-authored-by: Micha Reiser <micha@reiser.io>
## Summary
`Lexer::next_token` previously marked the token start before every
token, and the common lexing path marked it again after optional
whitespace handling even when the cursor had not moved. We now track
token starts only on paths that actually lex a token:
interpolated-string and indentation paths mark their ranges explicitly,
while the common path marks the start immediately before consuming the
token. Whitespace-at-EOF errors use an explicit start offset instead of
depending on the current token range.
<!--
Thank you for contributing to Ruff/ty! To help us out with reviewing,
please consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title? (Please prefix
with `[ty]` for ty pull
requests.)
- Does this pull request include references to any relevant issues?
- Does this PR follow our AI policy
(https://github.com/astral-sh/.github/blob/main/AI_POLICY.md)?
-->
## Summary
Following up on [prior review
feedback](https://github.com/astral-sh/ruff/pull/26679#discussion_r3557123788),
this introduces a shared helper (`ruff_python_trivia::tab_offset`) that
we use to consolidate the calculation for advancing a column to the next
tab stop.
## Test Plan
See included tests for the new helper. Otherwise, since this is a pure
refactor, we rely on existing test coverage.
<!-- How was it tested? -->
## Summary
Identifier lexing previously inspected the following one or two
characters for a quote before checking whether the identifier could be a
string prefix. Gate those lookaheads behind a shared string-prefix
check, so ordinary identifiers avoid unnecessary cursor reads while
string-prefix flags continue through the existing path.
## Summary
Publish Ruff's production Rust crates to crates.io as part of the
existing release workflow. This follows uv's setup: workspace
dependencies carry registry-compatible version pins, internal crates
receive generated READMEs and an explicit versioning policy, CI dry-runs
the full workspace publish, and release jobs publish missing crate
versions idempotently through crates.io trusted publishing.
The publishing set contains 32 packages: Ruff's CLI, WASM package,
server, linter, formatter, parser, and the internal Ruff component
crates required by them. It also includes `ty_module_resolver`,
`ty_site_packages`, and `ty_static`, because those three crates are part
of Ruff's normal dependency closure.
The publishing set deliberately excludes:
- Test-only crates: `mdtest`, `ruff_mdtest`,
`ruff_python_ast_integration_tests`, and
`ruff_python_trivia_integration_tests`.
- Benchmark and development crates: `ruff_benchmark` and `ruff_dev`.
- All remaining `ty_*` crates.
These packages retain `publish = false`, so bootstrap, version bumping,
README generation, CI dry-runs, and release publishing skip them through
Cargo metadata instead of a separate hard-coded allowlist.
The crates.io job is intentionally independent of release announcement,
so a crates.io outage cannot block the existing binary, PyPI, npm, or
GitHub release outputs. A bootstrap script reserves new crate names,
configures the release workflow as each crate's trusted publisher,
enables trusted-publishing-only releases, and records configured crates
for idempotent reruns.
The full workspace publish dry-run packages and verifies all 32 selected
crates.
Closes https://github.com/astral-sh/ruff/issues/43.
## Summary
Reject unparenthesized `yield` expressions after the first element of an
expression list:
```python
def generator():
1, yield 2
```
The parser previously reused the yield-enabled expression context for
every tuple element. Python's grammar only allows the initial
simple-statement alternative to be a `yield` expression; later
comma-separated elements use the ordinary starred-expression grammar.
This clears the yield allowance for subsequent elements while preserving
valid parenthesized nested yields.
## Summary
Reject bare starred targets in comprehensions:
```python
[item for *items in source]
```
The semantic syntax checker already rejects a bare starred target in an
ordinary `for` statement. This applies the same validation to
comprehension clauses while preserving valid tuple and list unpacking
targets such as `for *items, in source`.
## Summary
Reject `__debug__` as a lambda parameter name:
```python
lambda __debug__: 0
```
Function parameters already use the semantic syntax check that prevents
assignments to `__debug__`, but lambda parameters only checked for
duplicate names. This applies the existing identifier validation
consistently to both forms.
## Summary
Reject parenthesized star imports such as:
```python
from module import (*)
```
Python only permits a bare star in `from` imports. The parser previously
accepted the parenthesized form because star aliases were parsed
independently of the surrounding import-list delimiters.
This adds the missing parser validation while preserving ordinary
parenthesized imports and bare star imports.
## Summary
Mapping-pattern duplicate detection and F601 compare keys through
`HashableExpr`, but numeric normalization previously only recognized
values equivalent to `True` or `False`. This meant that other values
Python treats as the same dictionary key were considered distinct:
```python
match value:
case {2: first, 2.0: second}: ...
```
This adds a dedicated numeric representation for integer, float,
boolean, and complex literals so that exactly integral floats compare
with equivalent integers, including signed forms and zero-imaginary
complex values. Tuple normalization is limited to statically known
values so dynamic tuple elements do not introduce false-positive F601
diagnostics.
Numeric normalization uses the existing `u64` integer representation.
Larger integers retain structural comparison, avoiding
arbitrary-precision conversion in parser and linter hot paths.
## Summary
Python reserves `_` for wildcard patterns and does not allow it as the
binding target in either of these forms:
```python
match value:
case 1 as _:
pass
case {**_}:
pass
```
We currently recover an AST for both forms without reporting a parser
error. Both are grammar errors in CPython, so this change validates the
capture target while parsing `as` and mapping patterns and reports
`cannot use '_' as a target` on the underscore.
We previously tried to address the mapping-pattern case in
[#6838](https://github.com/astral-sh/ruff/pull/6838) by representing the
rest target as a general `PatternMatchAs` node. This keeps the existing
`Identifier` representation and recovered AST, including statements
after the invalid match, while rejecting the invalid target at the
parser boundary.
## Summary
Class definitions do not allow a bare generator expression as a base,
even though the same syntax is valid as the sole argument in a function
call. We currently parse the class form without an error:
```python
class C(base for base in bases):
pass
```
This change reuses the existing unparenthesized-generator diagnostic for
class arguments while continuing to accept explicitly parenthesized or
unpacked generator expressions:
```python
class C((base for base in bases)): ...
class C(*(base for base in bases)): ...
```
## Summary
Mapping patterns reject duplicate literal keys, but duplicate detection
compared AST shapes instead of Python hash equality. As a result, values
that Python treats as the same key were accepted:
```python
match value:
case {1.0: first, True: second}: ...
```
This switches mapping-key detection to `HashableExpr` and extends
boolean-equivalent normalization through signed numeric expressions and
complex values with a zero imaginary part. We now report duplicate-key
syntax errors for `0`/`False`, `1.0`/`True`, `-0`/`False`, and `1 +
0j`/`True`, matching CPython.
## Summary
Using ThinVec here is impactful because it reduces the size of `Expr`:
```
Arguments: 48 → 40 bytes
ExprCall: 72 → 64 bytes
Expr: 80 → 72 bytes
```
In profiling, it was empty about 80% of the time.
## Summary
Function type parameter lists did not run the same default-order
validation as class and type-alias parameter lists. As a result, we
accepted functions with a required type parameter after one with a
default:
```python
def f[T = int, U](): ...
```
This reuses the existing type-parameter ordering check for function
definitions, so we now report the same semantic syntax error that we
already emit for equivalent classes and type aliases.
## Summary
Python permits at most one starred name in a sequence pattern, but our
parser currently accepts multiple:
```python
match value:
case [*head, middle, *tail]:
pass
```
This tracks whether a sequence pattern already contains a star and
reports `invalid-syntax` on each later starred name. The diagnostic
points to the additional starred pattern and uses CPython's error
message, while keeping the recovered AST available to downstream tools.
## Summary
Stacks on #25726 and enables Clippy's `iter_over_hash_type` restriction
lint across the workspace, so new hash-map and hash-set iterations must
make their ordering assumptions explicit.
Most existing loops are order-independent and use a site-local
`#[expect(clippy::iter_over_hash_type)]`. At the order-sensitive sites,
this sorts before iteration. (In practice, that should stabilize some
parser fixtures, configuration diagnostics, union normalization, etc.).
This lint is kind of annoying, but given how much we want to crack down
on non-determinism in ty, it seems worthwhile to enable IMO.
## Summary
Shrink Suite vectors to drop the excess capacity during parsing.
I excluded other nodes for now to get a better sense of the performance
and memory impact.
I'm a bit conflicted on this. This is a pretty huge memory improvement
for ty, but there are a few linter benchmarks that regress by 1-2%
because some vectors need to be copied to smaller allocations, which
hurts performance. For the linter and formatter, it's also not important
to shrink the vectors, because the AST is never stored for long. But
this is different for ty where we cache the AST.
I'm curious to hear what others think on this. I could also try to
reduce the places where we call `shrink_to_fit`, e.g., only for
statements?
## Summary
Fixes https://github.com/astral-sh/ty/issues/1627.
Here's an example diagnostic with the current release of ty:
<img width="2286" height="286" alt="image"
src="https://github.com/user-attachments/assets/46ab9154-aaf0-4451-b301-5ff12fcd8507"
/>
On this branch, this diagnostic becomes:
<img width="1464" height="408" alt="image"
src="https://github.com/user-attachments/assets/449e0b6c-58d2-4501-91df-c267db6f48ca"
/>
The exact span of the node that creates the syntax error is now retained
and highlighted in the diagnostic.
## Implementation
Propagating the range of the node inside the string annotation into the
diagnostic is trivial. However, naively implementing that quickly
revealed that this would make diagnostics like this unsuppressable:
```py
x: """list[
yield from range(42)
]"""
```
The primary range of the diagnostic is now specifically the `yield from
range(42)` part of the string rather than the string node as a whole.
But I cannot add a `ty: ignore` comment that is either on or above the
`yield from range(42)` part of the string -- the "comment" there would
just become part of the string.
This PR also improves the consistency of our parser error messages in
general when it comes to capitalization.
## Test Plan
Mdtests extended and updated
---------
Co-authored-by: Micha Reiser <micha@reiser.io>