Commit Graph
443 Commits
Author SHA1 Message Date
Eduardo Rittner CoelhoandBrent Westbrook e96caa26b7 [syntax-errors] Detect duplicate keyword arguments (#17804)
## Summary

Part of https://github.com/astral-sh/ruff/issues/17412

Detects duplicate keyword arguments in function call.

## Test Plan

Added inline tests to validate intended behavior

---------

Co-authored-by: Brent Westbrook <brentrwestbrook@gmail.com>
2026-08-18 13:39:19 -04:00
Ross Titmarsh 4d813d53cd Fix broken link to Python docs (#27757)
Fixed a link to `frozen.discard` in the Python docs this was a valid
link in older Python docs but has been broken.

While I was there I noticed a few other uncanonical links so replaced
them with more canonical versions and removed some links to specific
Python versions that are unnecessary.
2026-08-14 16:08:37 +00:00
Micha Reiser 7a9aed24ff Guarantee minimum stack size when parsing a module, standalone expression, and suites (#25464) 2026-08-14 14:01:30 +02:00
Swayam Mhaskar d332d20d3f [syntax-errors] Name is parameter and nonlocal (#27628)
## Summary
Part of #17412
Detects semantic syntax error where name is parameter and nonlocal

## Test Plan
Added tests in `nonlocal_parameter.py`
2026-08-13 13:48:40 -04:00
Brent Westbrook b0e47022cf Bump 0.16.3 (#27723) 2026-08-13 09:39:16 -04:00
Alex Waygood c0824de94a Enable rustfmt on more files (#27630) 2026-08-10 15:30:41 +01:00
Charlie MarshandCharlie Marsh c56aad17fd [ty] Reduce Expr size to 64 bytes (#27591)
## Summary

This PR reduces the size of `Expr` from 72 to 64 bytes by storing only
the start offset on `ExprCall` (64 to 56 bytes) and boxing the uncommon
implicitly concatenated string representation in `ExprStringLiteral` (64
to 48 bytes). Both changes are required to reduce the size of the
containing `Expr` enum.

`ExprCall` derives its end offset from its argument list; parser and
error-recovery paths assert that their ranges share the same end.

---------

Co-authored-by: Charlie Marsh <charliemarsh@openai.com>
2026-08-09 13:41:24 +00:00
Brent Westbrook 5b48a04097 Bump 0.16.2 (#27555) 2026-08-07 09:12:50 -04:00
Charlie Marsh 6cfeb8aa07 Narrow internal visibility with Hawk (#27339)
## Summary

This PR uses Hawk 0.1.11 to narrow internal declarations to the
visibility they actually need.

Here, I've intentionally _only_ made visibility changes; unused API
removals are split into follow-up PRs. In total, 1,201 declaration
visibility modifiers are narrowed. Public generated formatter APIs and
other declarations whose narrower visibility would activate dead-code or
API-shape lints remain unchanged.
2026-07-31 22:08:13 -04:00
Brent Westbrook 80790b348b Bump 0.16.1 (#27330) 2026-07-30 13:54:42 -04:00
Ed Page 958efb1ff4 Vendor latest annotate-snippets (#27033)
## Summary

This is a first step towards #20411. After this, we would work to either
upstream changes or adjust things within Ruff.

Performance improvements have already been upstreamed
(https://github.com/rust-lang/annotate-snippets-rs/pull/437,
https://github.com/rust-lang/annotate-snippets-rs/pull/440)

Re-implemented the following from the fork:
- Hidden snippets
(https://github.com/rust-lang/annotate-snippets-rs/issues/448)
- Renderer-level disabling of hyperlinks
- Jupyter cell indices
(https://github.com/rust-lang/annotate-snippets-rs/issues/449)
- Fixable mark
(https://github.com/rust-lang/annotate-snippets-rs/issues/450)
- `lineno_offset` (maybe not needed if #20648 is taken care of)
- anonymized lines affects origin
(https://github.com/rust-lang/annotate-snippets-rs/issues/451)
- cut indicator (maybe not needed if we can switch to unicode renderer)

Additional hacks to minimize behavior change include
- Adjust upstream's hidden severity behavior to match Ruff's
(https://github.com/rust-lang/annotate-snippets-rs/issues/452)
- How newlines are annotated
(https://github.com/rust-lang/annotate-snippets-rs/issues/454)
- To not indent for line numbers when none will be present
(https://github.com/rust-lang/annotate-snippets-rs/issues/453)
- Enabling `simd` by default

All of the above represents work that needs to be resolved to address
#20411.

`lib.name` hack is used to simplify maintenance with the assumption that
we'll continue working towards #20411.

Behavior changes that were included
- Tweak when implicit padding lines are present
- Fix Origin's line/column is now derived from the primary annotation
- Make consistent the rendering of primary/context annotations
- Fix snippets to show up for empty source (but may degrade #24528)
- Fix handling of term width
- Fix rendering of annotations between newline and first character
(granted, maybe the caller is wrong)
- Tweak multiline annotation markers

New or upcoming features annotate-snippets features that we will be able
to leverage by moving this forward
- `AnnotationKind::Visible` to improve the quality of built-in
code-folding if Ruff were to switch to it
- `DecorStyle::Unicode`
- *(upcoming)* No-graphiscs mode for screen readers (may also be useful
for AIs)

Deferred:
- Moving related `Group`s into the same `Report`
- Using `secondary_title` for any `Group` after the first in a `Report`
- Using a `Snippet`, instead of titles, for "virtual code" like
https://github.com/astral-sh/ruff/pull/27033#discussion_r3622727236
- Switching hidden snippets to `Origin`

## Test Plan

Within cpython, I ran
```console
$ hyperfine -i "../../ruff/target-baseline/release/ruff check" "../../ruff/target-update/release/ruff check" -N -m20 --output=null
Benchmark 1: ../../ruff/target-baseline/release/ruff check
  Time (mean ± σ):      27.5 ms ±   1.1 ms    [User: 33.0 ms, System: 108.4 ms]
  Range (min … max):    25.0 ms …  31.7 ms    94 runs

  Warning: Ignoring non-zero exit code.

Benchmark 2: ../../ruff/target-update/release/ruff check
  Time (mean ± σ):      28.1 ms ±   2.0 ms    [User: 33.4 ms, System: 105.7 ms]
  Range (min … max):    24.8 ms …  38.9 ms    102 runs

  Warning: Ignoring non-zero exit code.
  Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet system without any interferences from other programs. It might help to use t
he '--warmup' or '--prepare' options.

Summary
  ../../ruff/target-baseline/release/ruff check ran
    1.03 ± 0.08 times faster than ../../ruff/target-update/release/ruff check
```

Before:
<img width="486" height="583" alt="image"
src="https://github.com/user-attachments/assets/2d94dabf-b28d-48c1-b17c-52aac59fd57e"
/>

After:
<img width="486" height="583" alt="image"
src="https://github.com/user-attachments/assets/b388ada3-e3d7-4ba0-b90b-1330eb81f4be"
/>
2026-07-28 11:57:09 -05:00
Ed Page 34d0944f3b Reduce syntax error noise by swallowing dedents like indents 2026-07-24 14:42:57 -05:00
Ed Page 57fb39b364 Test more indentation cases 2026-07-24 14:42:57 -05:00
Brent Westbrook a2635fd8f3 Bump 0.16.0 (#27136) 2026-07-23 14:37:34 -04:00
Alex Waygood c5aa9063e2 Prefer let guards and direct pattern matching (#26975) 2026-07-19 15:07:15 +01:00
Charlie Marsh 301b7ed4d6 [ty] Share heap-backed parser names with CharStr (#26594)
## Summary

This PR changes `ruff_python_ast::Name` from a 24-byte `CompactString`
to a 16-byte immutable `CharStr` from
[`astral-sh/char_str`](https://github.com/astral-sh/char_str).

`char_str` is an Astral-maintained fork of
[`lean_string`](https://github.com/ryota2357/lean_string) that provides
compact owned strings with mutable and immutable variants: growable
`CharString` for construction and immutable, exactly allocated `CharStr`
for storage. (It also provides Salsa and get-size features.)

Names up to 16 UTF-8 bytes remain inline, static names remain
allocation-free, and longer names use an exactly sized,
reference-counted allocation with no unused capacity.

Removing eight bytes from every `Name` also shrinks common AST nodes.
For example, `Identifier` and `ExprName` drop from 40 to 32 bytes, while
`ExprAttribute`, `StmtFunctionDef`, `StmtClassDef`, and several other
variants each drop by eight bytes.

The parser interns names that exceed the inline capacity, so repeated
longer names share one backing allocation within a parser invocation.

## Why 16 bytes?

Across the full `mypy_primer` corpus, there 20.8 million name
occurrences:

| UTF-8 length | Occurrences | Share | Unique name/file pairs | Share |
|---|---:|---:|---:|---:|
| 1–16 bytes | 18,955,757 | 91.02% | 3,192,424 | 79.42% |
| 17–24 bytes | 1,348,894 | 6.48% | 529,826 | 13.18% |
| 25+ bytes | 522,376 | 2.51% | 297,575 | 7.40% |
| Total | 20,827,027 | 100% | 4,019,825 | 100% |

The 16-byte cutoff keeps 91% of occurrences inline while saving eight
bytes for every occurrence. The 1–16-byte bucket alone removes 151.6 MB
of handle storage across the corpus.

The tradeoff is the 17–24-byte bucket: `CompactString` stored those
names inline, while `CharStr` allocates once per unique name within a
file. But the interning amortizes that cost when a name repeats. Names
longer than 24 bytes already allocated on `main`; for those names, this
change both reduces the handle size and deduplicates repeated backing
allocations.

## Memory impact

Accounting for the observed per-file multiplicities and structural
sharing, modeled `Name` storage across the corpus changes as follows:

| | `main` | This PR | Difference |
|---|---:|---:|---:|
| `Name` handles and backing storage | 526.18 MB | 366.27 MB | -159.91
MB (-30.4%) |

The current [instrumented memory
report](https://github.com/astral-sh/ruff/pull/26594#issuecomment-4906725175)
shows a consistent reduction in total retained memory:

| Project | `main` | This PR | Difference |
|---|---:|---:|---:|
| flake8 | 29.21 MB | 28.71 MB | -1.69% |
| trio | 71.01 MB | 70.01 MB | -1.41% |
| sphinx | 168.81 MB | 166.82 MB | -1.18% |
| prefect | 454.75 MB | 449.06 MB | -1.25% |

Within those totals, `parsed_module` decreases by roughly 2.1% and
`semantic_index` by 2.4–2.7%. Several name-heavy collections decrease by
considerably more.
2026-07-16 14:53:48 -04:00
Brent Westbrook 0177a7e0d2 Bump 0.15.22 (#26884) 2026-07-16 10:54:43 -04:00
Micha Reiser 9a5220ecdb Reuse parser scratch buffers (#26798)
## Summary

Use scratch buffers instead of allocating new `Vec`s and shrinking them
when constructing the AST node. While this doesn't remove the need for
copying the elements, it reduces the allocations within our parser
(`shrink_to_fit` often needs to allocate a new buffer and move all
elements). One shared scratch buffer per node type is sufficient,
because our parser is recursive (more nested parse rule always push to
the end, and parse function drain from the end as they complete).


## Performance

Very consistent perf improvement

## Codspeed memory regression

This can cause the parser's peak memory to increase, because we retain
the scratch buffer allocation after parsing a clause completed. However,
that memory is only temporary and released as soon as the parser
completes. Most project-level memory reports show a memory reduction.
It's only specific parser benchmarks that show a memory increase
2026-07-15 17:09:07 -07:00
Charlie Marsh bdf1320bba Update Rust toolchain to 1.97 and MSRV to 1.95 (#26844) 2026-07-15 08:12:16 +02:00
Charlie MarshandMicha Reiser 6e29838d4b Avoid redundant pending-indentation writes (#26774)
## Summary

The lexer checks `pending_indentation` for every token. `Option::take`
writes `None` even on the common path where no indentation is pending,
so we now read the field first and clear it only when a value is
present. This avoids a redundant write in the hot lexer loop while
preserving the existing indentation behavior.

This PR is stacked on #26765 so that CodSpeed can measure the
pending-indentation optimization independently.

---------

Co-authored-by: Micha Reiser <micha@reiser.io>
2026-07-14 17:47:07 +00:00
Charlie Marsh 25b10f9d01 Avoid redundant lexer token bookkeeping (#26765)
## Summary

`Lexer::next_token` previously marked the token start before every
token, and the common lexing path marked it again after optional
whitespace handling even when the cursor had not moved. We now track
token starts only on paths that actually lex a token:
interpolated-string and indentation paths mark their ranges explicitly,
while the common path marks the start immediately before consuming the
token. Whitespace-at-EOF errors use an explicit start offset instead of
depending on the current token range.
2026-07-14 10:40:42 -07:00
Lérè c87ba51052 [ty] Consolidate tab stop offset calculations (#26707)
<!--
Thank you for contributing to Ruff/ty! To help us out with reviewing,
please consider the following:

- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title? (Please prefix
with `[ty]` for ty pull
  requests.)
- Does this pull request include references to any relevant issues?
- Does this PR follow our AI policy
(https://github.com/astral-sh/.github/blob/main/AI_POLICY.md)?
-->

## Summary

Following up on [prior review
feedback](https://github.com/astral-sh/ruff/pull/26679#discussion_r3557123788),
this introduces a shared helper (`ruff_python_trivia::tab_offset`) that
we use to consolidate the calculation for advancing a column to the next
tab stop.

## Test Plan

See included tests for the new helper. Otherwise, since this is a pure
refactor, we rely on existing test coverage.
<!-- How was it tested? -->
2026-07-14 16:34:34 +00:00
Charlie Marsh 6fd508aa62 Avoid unnecessary identifier lookahead (#26525)
## Summary

Identifier lexing previously inspected the following one or two
characters for a quote before checking whether the identifier could be a
string prefix. Gate those lookaheads behind a shared string-prefix
check, so ordinary identifiers avoid unnecessary cursor reads while
string-prefix flags continue through the existing path.
2026-07-13 09:52:50 -04:00
Brent Westbrook 3e1f636451 Bump 0.15.21 (#26676) 2026-07-09 15:42:27 -04:00
Omar Yusuf AbdiandMicha Reiser 311158e260 [ruff] Detect syntax errors in individual notebook cells (#26419)
Co-authored-by: Micha Reiser <micha@reiser.io>
2026-07-07 11:28:10 +02:00
Brent Westbrook f82a36b6ba Bump 0.15.20 (#26376) 2026-06-25 10:00:55 -07:00
Dhruv Manilawala c24ded8ef6 Update ruff_python_parser README for crates.io (#26315)
## Summary

This PR updates the parser README to:
1. Link the parser README's AST reference to the published
`ruff_python_ast` crate
2. Remove the obsolete "Python version support" section now that version
specific syntax errors are implemented

Additionally, I've also updated the README generator to add a
"Versioning" section while preserving the rest of the handwritten README
and updated the generator to refresh marker-delimited versioning
sections. Maybe this is not worth doing, let me know if that's the case.

### Review

-
https://github.com/astral-sh/ruff/pull/26315/commits/5a4145a37436872c197c7de9771a04e91f4f0500
- updates the README manually as mentioned in (1) and (2) above
-
https://github.com/astral-sh/ruff/pull/26315/commits/dcb59253c41ecbebacd919defcbb725461a95478
- updates the README generator to add the versioning section

## Test plan

Here's the [Rendered
README](https://github.com/astral-sh/ruff/blob/dcb59253c41ecbebacd919defcbb725461a95478/crates/ruff_python_parser/README.md)
for preview.

Ran `uv run --script scripts/generate-crate-readmes.py` twice and
verified idempotent output
2026-06-24 12:07:14 +05:30
Charlie Marsh 7f0436561d Bump version to 0.15.19 (#26291) 2026-06-23 20:42:26 -04:00
Charlie Marsh dff016b7a4 Publish Ruff crates to crates.io (#26271)
## Summary

Publish Ruff's production Rust crates to crates.io as part of the
existing release workflow. This follows uv's setup: workspace
dependencies carry registry-compatible version pins, internal crates
receive generated READMEs and an explicit versioning policy, CI dry-runs
the full workspace publish, and release jobs publish missing crate
versions idempotently through crates.io trusted publishing.

The publishing set contains 32 packages: Ruff's CLI, WASM package,
server, linter, formatter, parser, and the internal Ruff component
crates required by them. It also includes `ty_module_resolver`,
`ty_site_packages`, and `ty_static`, because those three crates are part
of Ruff's normal dependency closure.

The publishing set deliberately excludes:

- Test-only crates: `mdtest`, `ruff_mdtest`,
`ruff_python_ast_integration_tests`, and
`ruff_python_trivia_integration_tests`.
- Benchmark and development crates: `ruff_benchmark` and `ruff_dev`.
- All remaining `ty_*` crates.

These packages retain `publish = false`, so bootstrap, version bumping,
README generation, CI dry-runs, and release publishing skip them through
Cargo metadata instead of a separate hard-coded allowlist.

The crates.io job is intentionally independent of release announcement,
so a crates.io outage cannot block the existing binary, PyPI, npm, or
GitHub release outputs. A bootstrap script reserves new crate names,
configures the release workflow as each crate's trusted publisher,
enables trusted-publishing-only releases, and records configured crates
for idempotent reruns.

The full workspace publish dry-run packages and verifies all 32 selected
crates.

Closes https://github.com/astral-sh/ruff/issues/43.
2026-06-23 12:49:23 -04:00
Micha Reiser b8ae5978c2 Avoid reallocating singleton call arguments (#26223) 2026-06-22 17:53:33 +02:00
Micha Reiser ae306a059e Avoid allocating when parsing single string literals (#26200) 2026-06-22 07:53:17 +02:00
Micha Reiser 4fda309ba2 Inline parser recovery context checks (#26038) 2026-06-16 15:43:32 +02:00
Micha Reiser bcae1b70fd Match parser keywords as bytes (#26037) 2026-06-16 15:13:33 +02:00
Charlie Marsh 3cef3416c5 Reject yield expressions after commas (#26024)
## Summary

Reject unparenthesized `yield` expressions after the first element of an
expression list:

```python
def generator():
    1, yield 2
```

The parser previously reused the yield-enabled expression context for
every tuple element. Python's grammar only allows the initial
simple-statement alternative to be a `yield` expression; later
comma-separated elements use the ordinary starred-expression grammar.

This clears the yield allowance for subsequent elements while preserving
valid parenthesized nested yields.
2026-06-16 09:28:12 +00:00
Charlie Marsh 841fb339d3 Reject starred comprehension targets (#26023)
## Summary

Reject bare starred targets in comprehensions:

```python
[item for *items in source]
```

The semantic syntax checker already rejects a bare starred target in an
ordinary `for` statement. This applies the same validation to
comprehension clauses while preserving valid tuple and list unpacking
targets such as `for *items, in source`.
2026-06-16 05:20:45 -04:00
Charlie Marsh 4b93e63a9d Reject __debug__ lambda parameters (#26022)
## Summary

Reject `__debug__` as a lambda parameter name:

```python
lambda __debug__: 0
```

Function parameters already use the semantic syntax check that prevents
assignments to `__debug__`, but lambda parameters only checked for
duplicate names. This applies the existing identifier validation
consistently to both forms.
2026-06-15 17:02:00 -04:00
Charlie Marsh a50dc58628 Reject parenthesized star imports (#26021)
## Summary

Reject parenthesized star imports such as:

```python
from module import (*)
```

Python only permits a bare star in `from` imports. The parser previously
accepted the parenthesized form because star aliases were parsed
independently of the surrounding import-list delimiters.

This adds the missing parser validation while preserving ordinary
parenthesized imports and bare star imports.
2026-06-15 17:01:37 -04:00
Charlie Marsh 8de183b907 Detect equivalent numeric mapping keys (#26009)
## Summary

Mapping-pattern duplicate detection and F601 compare keys through
`HashableExpr`, but numeric normalization previously only recognized
values equivalent to `True` or `False`. This meant that other values
Python treats as the same dictionary key were considered distinct:

```python
match value:
    case {2: first, 2.0: second}: ...
```

This adds a dedicated numeric representation for integer, float,
boolean, and complex literals so that exactly integral floats compare
with equivalent integers, including signed forms and zero-imaginary
complex values. Tuple normalization is limited to statically known
values so dynamic tuple elements do not introduce false-positive F601
diagnostics.

Numeric normalization uses the existing `u64` integer representation.
Larger integers retain structural comparison, avoiding
arbitrary-precision conversion in parser and linter hot paths.
2026-06-15 19:17:13 +00:00
Charlie Marsh b6ab2b32fb Reject _ as a match-pattern target (#25977)
## Summary

Python reserves `_` for wildcard patterns and does not allow it as the
binding target in either of these forms:

```python
match value:
    case 1 as _:
        pass
    case {**_}:
        pass
```

We currently recover an AST for both forms without reporting a parser
error. Both are grammar errors in CPython, so this change validates the
capture target while parsing `as` and mapping patterns and reports
`cannot use '_' as a target` on the underscore.

We previously tried to address the mapping-pattern case in
[#6838](https://github.com/astral-sh/ruff/pull/6838) by representing the
rest target as a general `PatternMatchAs` node. This keeps the existing
`Identifier` representation and recovered AST, including statements
after the invalid match, while rejecting the invalid target at the
parser boundary.
2026-06-15 13:47:21 +00:00
Charlie Marsh 7a7bfd84c4 Reject unparenthesized generator expressions in class bases (#25978)
## Summary

Class definitions do not allow a bare generator expression as a base,
even though the same syntax is valid as the sole argument in a function
call. We currently parse the class form without an error:

```python
class C(base for base in bases):
    pass
```

This change reuses the existing unparenthesized-generator diagnostic for
class arguments while continuing to accept explicitly parenthesized or
unpacked generator expressions:

```python
class C((base for base in bases)): ...
class C(*(base for base in bases)): ...
```
2026-06-15 13:34:24 +00:00
Charlie Marsh a9de595e40 Detect mapping keys equivalent to booleans (#25982)
## Summary

Mapping patterns reject duplicate literal keys, but duplicate detection
compared AST shapes instead of Python hash equality. As a result, values
that Python treats as the same key were accepted:

```python
match value:
    case {1.0: first, True: second}: ...
```

This switches mapping-key detection to `HashableExpr` and extends
boolean-equivalent normalization through signed numeric expressions and
complex values with a zero imaginary part. We now report duplicate-key
syntax errors for `0`/`False`, `1.0`/`True`, `-0`/`False`, and `1 +
0j`/`True`, matching CPython.
2026-06-15 09:30:48 -04:00
Charlie Marsh b0a887e367 Use ThinVec for call keywords (#25999)
## Summary

Using ThinVec here is impactful because it reduces the size of `Expr`:

```
Arguments: 48 → 40 bytes
ExprCall:  72 → 64 bytes
Expr:      80 → 72 bytes
```

In profiling, it was empty about 80% of the time.
2026-06-15 09:29:14 -04:00
Micha Reiser 32a86842a6 Move value parsing out of lexing (#25360) 2026-06-15 13:15:35 +02:00
Charlie Marsh 39b09a9d69 Validate function type parameter default order (#25981)
## Summary

Function type parameter lists did not run the same default-order
validation as class and type-alias parameter lists. As a result, we
accepted functions with a required type parameter after one with a
default:

```python
def f[T = int, U](): ...
```

This reuses the existing type-parameter ordering check for function
definitions, so we now report the same semantic syntax error that we
already emit for equivalent classes and type aliases.
2026-06-14 18:54:47 -04:00
Charlie Marsh ded6c002f9 Reject multiple starred names in sequence patterns (#25976)
## Summary

Python permits at most one starred name in a sequence pattern, but our
parser currently accepts multiple:

```python
match value:
    case [*head, middle, *tail]:
        pass
```

This tracks whether a sequence pattern already contains a star and
reports `invalid-syntax` on each later starred name. The diagnostic
points to the additional starred pattern and uses CPython's error
message, while keeping the recovered AST available to downstream tools.
2026-06-14 14:10:13 -04:00
Charlie Marsh 948359fd97 Enable Clippy's iter_over_hash_type lint (#25732)
## Summary

Stacks on #25726 and enables Clippy's `iter_over_hash_type` restriction
lint across the workspace, so new hash-map and hash-set iterations must
make their ordering assumptions explicit.

Most existing loops are order-independent and use a site-local
`#[expect(clippy::iter_over_hash_type)]`. At the order-sensitive sites,
this sorts before iteration. (In practice, that should stabilize some
parser fixtures, configuration diagnostics, union normalization, etc.).

This lint is kind of annoying, but given how much we want to crack down
on non-determinism in ty, it seems worthwhile to enable IMO.
2026-06-09 16:35:09 +00:00
Micha Reiser aa894ffa41 Reserve AST Vec's with correct capacity for common cases (#25451) 2026-06-05 07:00:32 +01:00
Micha Reiser 10ccd511e9 Shrink additional parser AST collections (#25465) 2026-06-04 10:14:51 +01:00
Micha Reiser 1518f1d5fd Drop excess capacity from Suites during parsing (#25368)
## Summary

Shrink Suite vectors to drop the excess capacity during parsing. 

I excluded other nodes for now to get a better sense of the performance
and memory impact.

I'm a bit conflicted on this. This is a pretty huge memory improvement
for ty, but there are a few linter benchmarks that regress by 1-2%
because some vectors need to be copied to smaller allocations, which
hurts performance. For the linter and formatter, it's also not important
to shrink the vectors, because the AST is never stored for long. But
this is different for ty where we cache the AST.

I'm curious to hear what others think on this. I could also try to
reduce the places where we call `shrink_to_fit`, e.g., only for
statements?
2026-05-29 18:17:16 +00:00
Alex WaygoodandMicha Reiser 366fe21ba3 [ty] Improve diagnostics for syntax errors in forward annotations (#25158)
## Summary

Fixes https://github.com/astral-sh/ty/issues/1627.

Here's an example diagnostic with the current release of ty:

<img width="2286" height="286" alt="image"
src="https://github.com/user-attachments/assets/46ab9154-aaf0-4451-b301-5ff12fcd8507"
/>

On this branch, this diagnostic becomes:

<img width="1464" height="408" alt="image"
src="https://github.com/user-attachments/assets/449e0b6c-58d2-4501-91df-c267db6f48ca"
/>

The exact span of the node that creates the syntax error is now retained
and highlighted in the diagnostic.

## Implementation

Propagating the range of the node inside the string annotation into the
diagnostic is trivial. However, naively implementing that quickly
revealed that this would make diagnostics like this unsuppressable:

```py
x: """list[
    yield from range(42)
]"""
```

The primary range of the diagnostic is now specifically the `yield from
range(42)` part of the string rather than the string node as a whole.
But I cannot add a `ty: ignore` comment that is either on or above the
`yield from range(42)` part of the string -- the "comment" there would
just become part of the string.

This PR also improves the consistency of our parser error messages in
general when it comes to capitalization.

## Test Plan

Mdtests extended and updated

---------

Co-authored-by: Micha Reiser <micha@reiser.io>
2026-05-28 10:51:20 +00:00