Commit Graph
105 Commits
Author SHA1 Message Date
Micha Reiser 807c913a66 Remove backwards tokenizer (#30622)
## Summary

The parser now stores the tokens. This removes the need for
`BackwardsTokenizer`, which was often difficult to reason about.
This PR removes `BackwardsTokenizer`, its unused cursor helper, and its
backwards-only tests and snapshots.
2026-11-20 11:25:56 +00:00
Micha Reiser 3f44d52da5 Fix bogus token range in forward tokenizer (#30611)
## Summary

Today, when tokenizing part of a source file, `SimpleTokenizer` can
report a final `Bogus` token extending past the requested range. This
can also cause the backwards tokenizer to panic (#28113).

This PR limits the final `Bogus` token to the requested range and
corrects the documentation for bogus tokens.

Testing: Added integration coverage, and all tokenizer integration tests
pass.
2026-11-20 09:38:39 +00:00
astral-automations-bot[bot]andBrent Westbrook a7ba68b491 Bump version to 0.17.6 (#30578)
Co-authored-by: astral-automations-bot[bot] <305554984+astral-automations-bot[bot]@users.noreply.github.com>
Co-authored-by: Brent Westbrook <36778786+ntBre@users.noreply.github.com>
2026-11-19 10:32:47 -05:00
Micha Reiser d3e454ea7c [ty] Speed-up find references in workspaces with many unicode files (#30402)
## Summary

A find-references search for `C` currently visits files just because
they contain punctuation like `…`. After an exact-name search misses,
`NameMatcher` jumps to the next non-ASCII character. It skips
punctuation and compares whole identifiers after NFKC normalization, so
it finds `obj.C` and skips `obj.Cache`. Related to astral-sh/ty#5006.

I replayed 243 find-references requests on
[Django](https://github.com/django/django/tree/a05e16a7230fe33d3f43f4a3aaed103cebb42efc)
(2,918 Python files, two CPUs). Timings are medians of three runs per
build. Parse counts are from a separate run.

| | ty 0.0.81 | PR base | This PR |
| --- | ---: | ---: | ---: |
| Median request | 2.55 ms | 28.32 ms | 2.66 ms |
| All 243 requests | 13.77 s | 20.20 s | 13.43 s |
| Distinct files parsed | 1,327 | 1,219 | 1,189 |
| Reparses | 2,032 | 20,029 | 6,912 |

Checking non-ASCII identifiers took 0.14 s of worker CPU time across all
243 requests.

## Test Plan

The matcher tests passed, and all three builds returned the same 343
reference locations.
2026-11-16 10:02:55 +01:00
0ebb77d35c Bump version to 0.17.5 (#30380)
Co-authored-by: astral-automations-bot[bot] <305554984+astral-automations-bot[bot]@users.noreply.github.com>
Co-authored-by: Brent Westbrook <36778786+ntBre@users.noreply.github.com>
Co-authored-by: Brent Westbrook <brentrwestbrook@gmail.com>
2026-11-12 17:26:23 +00:00
Micha Reiser 5a784dfbd4 Inline public Cursor methods across crates (#30368)
## Summary

Ruff’s suppression-comment scanners call
`ruff_python_trivia::Cursor::first` and `bump` for each character. Add
`#[inline]` to the public cursor methods and the forward and backward
predicate loops so callers can inline these operations.

Callgrind instruction counts for uncached Ruff lint on x86-64 with the
profiling build (without fat LTO):

| Workload | Main | With this change | Reduction |
| --- | ---: | ---: | ---: |
| JAX docs | 871,822,822 | 747,817,978 | 14.22% |
| JAX shard_map.py | 210,767,215 | 98,080,629 | 53.46% |
| Jinja | 204,158,352 | 201,518,289 | 1.29% |

## Test Plan

The trivia unit tests passed, and the benchmark runs produced identical
diagnostics.
2026-11-12 15:50:54 +01:00
astral-automations-bot[bot]andBrent Westbrook 05cdc36a45 Bump version to 0.17.4 (#30073)
Co-authored-by: astral-automations-bot[bot] <305554984+astral-automations-bot[bot]@users.noreply.github.com>
Co-authored-by: Brent Westbrook <36778786+ntBre@users.noreply.github.com>
2026-11-05 13:22:15 -05:00
astral-automations-bot[bot]andBrent Westbrook ba65abf8ba Bump version to 0.17.3 (#29745)
Co-authored-by: astral-automations-bot[bot] <305554984+astral-automations-bot[bot]@users.noreply.github.com>
Co-authored-by: Brent Westbrook <36778786+ntBre@users.noreply.github.com>
2026-10-26 20:14:27 -04:00
astral-automations-bot[bot]andBrent Westbrook 30516b42ab Bump version to 0.17.2 (#29637)
Co-authored-by: astral-automations-bot[bot] <305554984+astral-automations-bot[bot]@users.noreply.github.com>
Co-authored-by: Brent Westbrook <36778786+ntBre@users.noreply.github.com>
2026-10-22 13:54:24 -04:00
astral-automations-bot[bot] 4683270c69 Bump version to 0.17.1 (#29420)
Co-authored-by: astral-automations-bot[bot] <305554984+astral-automations-bot[bot]@users.noreply.github.com>
2026-10-15 10:56:41 -04:00
Marcos MaxandBrent Westbrook 83a3afc315 Fix dedent with multiple leading comments (#29382)
Fix unsafe dedent behavior when multiple leading comments are present.
Added regression coverage for the affected indentation cases.

Fixes https://github.com/astral-sh/ruff/issues/29381.

---------

Co-authored-by: Brent Westbrook <brentrwestbrook@gmail.com>
2026-10-14 20:28:17 +00:00
astral-automations-bot[bot]andBrent Westbrook 5b27c1ccfa Bump version to 0.17.0 (#29259)
Co-authored-by: astral-automations-bot[bot] <305554984+astral-automations-bot[bot]@users.noreply.github.com>
Co-authored-by: Brent Westbrook <brentrwestbrook@gmail.com>
2026-10-09 13:45:29 -04:00
astral-automations-bot[bot]andBrent Westbrook 3265ed1f94 Bump version to 0.16.10 (#29055)
Co-authored-by: astral-automations-bot[bot] <305554984+astral-automations-bot[bot]@users.noreply.github.com>
Co-authored-by: Brent Westbrook <brentrwestbrook@gmail.com>
2026-10-01 13:40:09 -04:00
astral-automations-bot[bot]andBrent Westbrook 0be08a206f Bump version to 0.16.9 (#28882)
Co-authored-by: astral-automations-bot[bot] <305554984+astral-automations-bot[bot]@users.noreply.github.com>
Co-authored-by: Brent Westbrook <36778786+ntBre@users.noreply.github.com>
2026-09-24 16:21:50 -04:00
Micha Reiser f240bf8498 [ty] Preserve fast name filtering for normalized Unicode source (#28701) 2026-09-18 18:28:51 +00:00
Micha Reiser bee9b1c3f6 [ty] Reuse identifier matcher in LSP operations (#28663) 2026-09-17 13:14:48 +00:00
astral-automations-bot[bot]andBrent Westbrook 62914c4b9b Bump version to 0.16.8 (#28648)
Co-authored-by: astral-automations-bot[bot] <305554984+astral-automations-bot[bot]@users.noreply.github.com>
Co-authored-by: Brent Westbrook <36778786+ntBre@users.noreply.github.com>
2026-09-16 11:31:02 -04:00
astral-automations-bot[bot]andBrent Westbrook b5dba861cc Bump version to 0.16.7 (#28496)
Co-authored-by: astral-automations-bot[bot] <305554984+astral-automations-bot[bot]@users.noreply.github.com>
Co-authored-by: Brent Westbrook <36778786+ntBre@users.noreply.github.com>
2026-09-10 13:47:30 -04:00
Brent Westbrook 22f65a2ab5 Bump 0.16.6 (#28280) 2026-09-03 12:36:42 -04:00
Brent Westbrook 9e4938c4a6 Bump 0.16.5 (#28110) 2026-08-27 12:09:07 -04:00
Charlie Marsh 88d56e5feb Use stabilized Rust pattern assertion macros (#27955)
## Summary

- Use Rust 1.96's stabilized `assert_matches!` and
`debug_assert_matches!` macros throughout Ruff and ty.
- Include mismatched values in assertion failures and remove redundant
manual debug formatting.
- Preserve existing assertions for values that do not implement `Debug`.
2026-08-21 09:29:39 -04:00
Brent Westbrook 11c76bf48f Bump 0.16.4 (#27937) 2026-08-20 13:14:29 -04:00
Brent Westbrook b0e47022cf Bump 0.16.3 (#27723) 2026-08-13 09:39:16 -04:00
Alex Waygood c0824de94a Enable rustfmt on more files (#27630) 2026-08-10 15:30:41 +01:00
Brent Westbrook 5b48a04097 Bump 0.16.2 (#27555) 2026-08-07 09:12:50 -04:00
Charlie Marsh 6cfeb8aa07 Narrow internal visibility with Hawk (#27339)
## Summary

This PR uses Hawk 0.1.11 to narrow internal declarations to the
visibility they actually need.

Here, I've intentionally _only_ made visibility changes; unused API
removals are split into follow-up PRs. In total, 1,201 declaration
visibility modifiers are narrowed. Public generated formatter APIs and
other declarations whose narrower visibility would activate dead-code or
API-shape lints remain unchanged.
2026-07-31 22:08:13 -04:00
Brent Westbrook 80790b348b Bump 0.16.1 (#27330) 2026-07-30 13:54:42 -04:00
Brent Westbrook a2635fd8f3 Bump 0.16.0 (#27136) 2026-07-23 14:37:34 -04:00
Brent Westbrook 0177a7e0d2 Bump 0.15.22 (#26884) 2026-07-16 10:54:43 -04:00
Lérè c87ba51052 [ty] Consolidate tab stop offset calculations (#26707)
<!--
Thank you for contributing to Ruff/ty! To help us out with reviewing,
please consider the following:

- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title? (Please prefix
with `[ty]` for ty pull
  requests.)
- Does this pull request include references to any relevant issues?
- Does this PR follow our AI policy
(https://github.com/astral-sh/.github/blob/main/AI_POLICY.md)?
-->

## Summary

Following up on [prior review
feedback](https://github.com/astral-sh/ruff/pull/26679#discussion_r3557123788),
this introduces a shared helper (`ruff_python_trivia::tab_offset`) that
we use to consolidate the calculation for advancing a column to the next
tab stop.

## Test Plan

See included tests for the new helper. Otherwise, since this is a pure
refactor, we rely on existing test coverage.
<!-- How was it tested? -->
2026-07-14 16:34:34 +00:00
Lérè 9b3d7d7ebb [ty] Correct how we expand tabs in docstrings (#26679)
<!--
Thank you for contributing to Ruff/ty! To help us out with reviewing,
please consider the following:

- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title? (Please prefix
with `[ty]` for ty pull
  requests.)
- Does this pull request include references to any relevant issues?
- Does this PR follow our AI policy
(https://github.com/astral-sh/.github/blob/main/AI_POLICY.md)?
-->

## Summary

This corrects how we expand tabs as part of PEP-257 normalization of
docstrings.

Previously, we incorrectly replaced each tab in the docstring with 8
spaces. However, as per [the reference
implementation](https://peps.python.org/pep-0257/#handling-docstring-indentation)
(and the [`str.expandtabs`
docs](https://docs.python.org/3/library/stdtypes.html#str.expandtabs)),
we should instead treat a tab as a directive to advance to the next tab
stop (the next column at a multiple of 8).

This matters for Google-style parameter extraction when a tab follows
spaces. For example, consider an `Args:` section whose first item is
indented with two spaces followed by a tab, while the second uses eight
spaces:

```python
def example(first: str, second: str):
    """Summary.

    Args:
      \tfirst: First parameter.
        second: Second parameter.
    """
```

The previous normalization produced this docstring. Because the
parameter items have different indentation, `second` is treated as
continuation text for `first` rather than as a separate parameter:

```text
Summary.

Args:
          first: First parameter.
    second: Second parameter.
```

Whereas the correct interpretation according to PEP 257 aligns the
parameter items, allowing documentation to be extracted for both `first`
and `second`:

```text
Summary.

Args:
    first: First parameter.
    second: Second parameter.
```

## Test Plan

See included tests.
<!-- How was it tested? -->
2026-07-10 09:46:12 -07:00
Brent Westbrook 3e1f636451 Bump 0.15.21 (#26676) 2026-07-09 15:42:27 -04:00
Charlie Marsh 2cd74f0050 Cache parenthesized expression boundaries in the formatter (#26344)
## Summary

Formatting frequently asks whether an expression was parenthesized in
the source. Prior to this change, each check re-tokenized the
surrounding source and skipped trivia independently, even though the
parser token stream is already available.

This PR builds `TriviaRanges` in the same token traversal that collects
comment ranges. It stores those ranges alongside a
`ParenthesizedExpressions` index backed by a single
`FxHashSet<TextRange>`, so comment placement and `PyFormatContext` share
the same source-boundary data. Hot-path checks become one lookup without
an additional token traversal or source-scanning fallback.

Formatter paths that need the full parenthesized range continue to use
the parsed-token `parentheses_iterator`.
2026-06-26 15:04:42 -04:00
Brent Westbrook f82a36b6ba Bump 0.15.20 (#26376) 2026-06-25 10:00:55 -07:00
Charlie Marsh 7f0436561d Bump version to 0.15.19 (#26291) 2026-06-23 20:42:26 -04:00
Charlie Marsh dff016b7a4 Publish Ruff crates to crates.io (#26271)
## Summary

Publish Ruff's production Rust crates to crates.io as part of the
existing release workflow. This follows uv's setup: workspace
dependencies carry registry-compatible version pins, internal crates
receive generated READMEs and an explicit versioning policy, CI dry-runs
the full workspace publish, and release jobs publish missing crate
versions idempotently through crates.io trusted publishing.

The publishing set contains 32 packages: Ruff's CLI, WASM package,
server, linter, formatter, parser, and the internal Ruff component
crates required by them. It also includes `ty_module_resolver`,
`ty_site_packages`, and `ty_static`, because those three crates are part
of Ruff's normal dependency closure.

The publishing set deliberately excludes:

- Test-only crates: `mdtest`, `ruff_mdtest`,
`ruff_python_ast_integration_tests`, and
`ruff_python_trivia_integration_tests`.
- Benchmark and development crates: `ruff_benchmark` and `ruff_dev`.
- All remaining `ty_*` crates.

These packages retain `publish = false`, so bootstrap, version bumping,
README generation, CI dry-runs, and release publishing skip them through
Cargo metadata instead of a separate hard-coded allowlist.

The crates.io job is intentionally independent of release announcement,
so a crates.io outage cannot block the existing binary, PyPI, npm, or
GitHub release outputs. A bootstrap script reserves new crate names,
configures the release workflow as each crate's trusted publisher,
enables trusted-publishing-only releases, and records configured crates
for idempotent reruns.

The full workspace publish dry-run packages and verifies all 32 selected
crates.

Closes https://github.com/astral-sh/ruff/issues/43.
2026-06-23 12:49:23 -04:00
Alex Waygood 581b3fa230 Update Rust toolchain to 1.95 and MSRV to 1.93 (#24677) 2026-04-17 11:39:14 +01:00
Dylan 7fdb556189 Strip form feeds from indent passed to dedent_to (#24381)
When adjusting "simple" indentation in the formation of edits, we
attempt to dedent manually rather than deferring to LibCST. To do so we
must provide a desired indentation, in the form of a string. We often
grab this from source code by slicing the text in a range beginning at
the start of a line.

In Python, the start of a line may contain form feeds but these do not
contribute to the indentation. In this PR, we strip the provided
indentation of its leading form feeds in order to get the correct
indentation amount for use in `dedent_to`. This avoids the introduction
of a syntax error in the edit
[`adjust_indentation`](https://github.com/astral-sh/ruff/blob/1f430e68af6e627569776dfbcd03b98ac7c29eb6/crates/ruff_linter/src/fix/edits.rs#L429).

Note: We could try to stay closer to the user's intent by prepending the
form feed prefix of the provided indentation _everywhere_ in the
resulting edit, but that seems a little unwieldy and this is a bit of an
edge case anyway.

Closes #24373
2026-04-02 20:14:59 -05:00
Ed CussandMicha Reiser 37f5d61595 RUF072: skip formfeeds on dedent (#24308)
Co-authored-by: Micha Reiser <micha@reiser.io>
2026-04-02 14:49:38 +02:00
Daniil SivakandMicha Reiser 9d2b16029a E501/W505/formatter: Exclude nested pragma comments from line width calculation (#24071)
Closes #18470

## Summary

Exclude nested pragma comments like the `# noqa` in `test # comment 1 # noqa` from `E501`, `W505` and the formatter's line width calculation. 


## Test Plan

- Added linter test fixture `E501_5.py` with preview test cases for
nested pragmas (`# comment #noqa`, `## noqa`, `# comment # type:
ignore`)
- Added formatter test fixture `trailing_pragma_nested.py` demonstrating
stable vs preview reserved width behavior
- Doctests for `find_trailing_pragma_offset` in `pragmas.rs`

---------

Co-authored-by: Micha Reiser <micha@reiser.io>
2026-03-24 10:08:00 +00:00
tmimmanuelandMicha Reiser 7255e362e4 [pycodestyle] Recognize pyrefly: as a pragma comment (E501) (#24019)
Co-authored-by: Micha Reiser <micha@reiser.io>
2026-03-17 18:58:50 +01:00
Charlie Marsh 8a10b6743c Add support for lazy import parsing (#23755)
## Summary

This PR adds `is_lazy` to the AST and parser, and supports `lazy` import
formatting in the formatter. The name `is_lazy` matches the [CPython
AST](https://github.com/python/cpython/blob/c3fb0d9d96902774c08b199dda0479a8d31398a5/Parser/Python.asdl#L48),
though we could modify it if there's demand.

There are no further changes to Ruff rules or even to Ruff import
sorting, since those deserve separate design discussions.

I've also omitted support for flagging semantic syntax errors (e.g.,
`lazy import` inside `def`).

See: https://github.com/astral-sh/ruff/issues/21305.
2026-03-06 09:12:21 -05:00
Matthew Mckee a2e0ff57c3 Run cargo sort (#22310) 2026-01-02 19:58:15 +00:00
Dylan f2b7c82534 Handle t-string prefixes in SimpleTokenizer (#20578)
The simple tokenizer is meant to skip strings, but it was recording a
`Name` token for t-strings (from the `t`). This PR fixes that.
2025-09-25 14:33:37 -05:00
89ca493fd9 [ruff] Preserve relative whitespace in multi-line expressions (RUF033) (#19647)
## Summary

Fixes #19581

I decided to add in a `indent_first_line` function into
[`textwrap.rs`](https://github.com/astral-sh/ruff/blob/main/crates/ruff_python_trivia/src/textwrap.rs),
as it solely focuses on text manipulation utilities. It follows the same
design as `indent()`, and there may be situations in the future where it
can be reused as well.

---------

Co-authored-by: Brent Westbrook <36778786+ntBre@users.noreply.github.com>
Co-authored-by: Brent Westbrook <brentrwestbrook@gmail.com>
2025-08-27 19:15:44 +00:00
Micha Reiser b3b900dc1e Treat ty: comments as pragma comments (#18532)
## Summary

Add support for ty's `ty:` pragma comments to ruff's formatter and E501

Fixes https://github.com/astral-sh/ruff/issues/18529

## Test Plan

Added test
2025-06-07 16:02:43 +02:00
Alex Waygood 7df79cfb70 Add offset method to ruff_python_trivia::Cursor (#18371) 2025-05-29 16:08:15 +01:00
Micha Reiser 9ae698fe30 Switch to Rust 2024 edition (#18129) 2025-05-16 13:25:28 +02:00
Micha Reiser fa628018b2 Use #[expect(lint)] over #[allow(lint)] where possible (#17822) 2025-05-03 21:20:31 +02:00
Alex Waygood 040071bbc5 [red-knot] Ignore surrounding whitespace when looking for <!-- snapshot-diagnostics --> directives in mdtests (#16380) 2025-02-27 13:25:31 +00:00