## Summary
The parser now stores the tokens. This removes the need for
`BackwardsTokenizer`, which was often difficult to reason about.
This PR removes `BackwardsTokenizer`, its unused cursor helper, and its
backwards-only tests and snapshots.
## Summary
Today, when tokenizing part of a source file, `SimpleTokenizer` can
report a final `Bogus` token extending past the requested range. This
can also cause the backwards tokenizer to panic (#28113).
This PR limits the final `Bogus` token to the requested range and
corrects the documentation for bogus tokens.
Testing: Added integration coverage, and all tokenizer integration tests
pass.
## Summary
A find-references search for `C` currently visits files just because
they contain punctuation like `…`. After an exact-name search misses,
`NameMatcher` jumps to the next non-ASCII character. It skips
punctuation and compares whole identifiers after NFKC normalization, so
it finds `obj.C` and skips `obj.Cache`. Related to astral-sh/ty#5006.
I replayed 243 find-references requests on
[Django](https://github.com/django/django/tree/a05e16a7230fe33d3f43f4a3aaed103cebb42efc)
(2,918 Python files, two CPUs). Timings are medians of three runs per
build. Parse counts are from a separate run.
| | ty 0.0.81 | PR base | This PR |
| --- | ---: | ---: | ---: |
| Median request | 2.55 ms | 28.32 ms | 2.66 ms |
| All 243 requests | 13.77 s | 20.20 s | 13.43 s |
| Distinct files parsed | 1,327 | 1,219 | 1,189 |
| Reparses | 2,032 | 20,029 | 6,912 |
Checking non-ASCII identifiers took 0.14 s of worker CPU time across all
243 requests.
## Test Plan
The matcher tests passed, and all three builds returned the same 343
reference locations.
## Summary
Ruff’s suppression-comment scanners call
`ruff_python_trivia::Cursor::first` and `bump` for each character. Add
`#[inline]` to the public cursor methods and the forward and backward
predicate loops so callers can inline these operations.
Callgrind instruction counts for uncached Ruff lint on x86-64 with the
profiling build (without fat LTO):
| Workload | Main | With this change | Reduction |
| --- | ---: | ---: | ---: |
| JAX docs | 871,822,822 | 747,817,978 | 14.22% |
| JAX shard_map.py | 210,767,215 | 98,080,629 | 53.46% |
| Jinja | 204,158,352 | 201,518,289 | 1.29% |
## Test Plan
The trivia unit tests passed, and the benchmark runs produced identical
diagnostics.
## Summary
- Use Rust 1.96's stabilized `assert_matches!` and
`debug_assert_matches!` macros throughout Ruff and ty.
- Include mismatched values in assertion failures and remove redundant
manual debug formatting.
- Preserve existing assertions for values that do not implement `Debug`.
## Summary
This PR uses Hawk 0.1.11 to narrow internal declarations to the
visibility they actually need.
Here, I've intentionally _only_ made visibility changes; unused API
removals are split into follow-up PRs. In total, 1,201 declaration
visibility modifiers are narrowed. Public generated formatter APIs and
other declarations whose narrower visibility would activate dead-code or
API-shape lints remain unchanged.
<!--
Thank you for contributing to Ruff/ty! To help us out with reviewing,
please consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title? (Please prefix
with `[ty]` for ty pull
requests.)
- Does this pull request include references to any relevant issues?
- Does this PR follow our AI policy
(https://github.com/astral-sh/.github/blob/main/AI_POLICY.md)?
-->
## Summary
Following up on [prior review
feedback](https://github.com/astral-sh/ruff/pull/26679#discussion_r3557123788),
this introduces a shared helper (`ruff_python_trivia::tab_offset`) that
we use to consolidate the calculation for advancing a column to the next
tab stop.
## Test Plan
See included tests for the new helper. Otherwise, since this is a pure
refactor, we rely on existing test coverage.
<!-- How was it tested? -->
<!--
Thank you for contributing to Ruff/ty! To help us out with reviewing,
please consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title? (Please prefix
with `[ty]` for ty pull
requests.)
- Does this pull request include references to any relevant issues?
- Does this PR follow our AI policy
(https://github.com/astral-sh/.github/blob/main/AI_POLICY.md)?
-->
## Summary
This corrects how we expand tabs as part of PEP-257 normalization of
docstrings.
Previously, we incorrectly replaced each tab in the docstring with 8
spaces. However, as per [the reference
implementation](https://peps.python.org/pep-0257/#handling-docstring-indentation)
(and the [`str.expandtabs`
docs](https://docs.python.org/3/library/stdtypes.html#str.expandtabs)),
we should instead treat a tab as a directive to advance to the next tab
stop (the next column at a multiple of 8).
This matters for Google-style parameter extraction when a tab follows
spaces. For example, consider an `Args:` section whose first item is
indented with two spaces followed by a tab, while the second uses eight
spaces:
```python
def example(first: str, second: str):
"""Summary.
Args:
\tfirst: First parameter.
second: Second parameter.
"""
```
The previous normalization produced this docstring. Because the
parameter items have different indentation, `second` is treated as
continuation text for `first` rather than as a separate parameter:
```text
Summary.
Args:
first: First parameter.
second: Second parameter.
```
Whereas the correct interpretation according to PEP 257 aligns the
parameter items, allowing documentation to be extracted for both `first`
and `second`:
```text
Summary.
Args:
first: First parameter.
second: Second parameter.
```
## Test Plan
See included tests.
<!-- How was it tested? -->
## Summary
Formatting frequently asks whether an expression was parenthesized in
the source. Prior to this change, each check re-tokenized the
surrounding source and skipped trivia independently, even though the
parser token stream is already available.
This PR builds `TriviaRanges` in the same token traversal that collects
comment ranges. It stores those ranges alongside a
`ParenthesizedExpressions` index backed by a single
`FxHashSet<TextRange>`, so comment placement and `PyFormatContext` share
the same source-boundary data. Hot-path checks become one lookup without
an additional token traversal or source-scanning fallback.
Formatter paths that need the full parenthesized range continue to use
the parsed-token `parentheses_iterator`.
## Summary
Publish Ruff's production Rust crates to crates.io as part of the
existing release workflow. This follows uv's setup: workspace
dependencies carry registry-compatible version pins, internal crates
receive generated READMEs and an explicit versioning policy, CI dry-runs
the full workspace publish, and release jobs publish missing crate
versions idempotently through crates.io trusted publishing.
The publishing set contains 32 packages: Ruff's CLI, WASM package,
server, linter, formatter, parser, and the internal Ruff component
crates required by them. It also includes `ty_module_resolver`,
`ty_site_packages`, and `ty_static`, because those three crates are part
of Ruff's normal dependency closure.
The publishing set deliberately excludes:
- Test-only crates: `mdtest`, `ruff_mdtest`,
`ruff_python_ast_integration_tests`, and
`ruff_python_trivia_integration_tests`.
- Benchmark and development crates: `ruff_benchmark` and `ruff_dev`.
- All remaining `ty_*` crates.
These packages retain `publish = false`, so bootstrap, version bumping,
README generation, CI dry-runs, and release publishing skip them through
Cargo metadata instead of a separate hard-coded allowlist.
The crates.io job is intentionally independent of release announcement,
so a crates.io outage cannot block the existing binary, PyPI, npm, or
GitHub release outputs. A bootstrap script reserves new crate names,
configures the release workflow as each crate's trusted publisher,
enables trusted-publishing-only releases, and records configured crates
for idempotent reruns.
The full workspace publish dry-run packages and verifies all 32 selected
crates.
Closes https://github.com/astral-sh/ruff/issues/43.
When adjusting "simple" indentation in the formation of edits, we
attempt to dedent manually rather than deferring to LibCST. To do so we
must provide a desired indentation, in the form of a string. We often
grab this from source code by slicing the text in a range beginning at
the start of a line.
In Python, the start of a line may contain form feeds but these do not
contribute to the indentation. In this PR, we strip the provided
indentation of its leading form feeds in order to get the correct
indentation amount for use in `dedent_to`. This avoids the introduction
of a syntax error in the edit
[`adjust_indentation`](https://github.com/astral-sh/ruff/blob/1f430e68af6e627569776dfbcd03b98ac7c29eb6/crates/ruff_linter/src/fix/edits.rs#L429).
Note: We could try to stay closer to the user's intent by prepending the
form feed prefix of the provided indentation _everywhere_ in the
resulting edit, but that seems a little unwieldy and this is a bit of an
edge case anyway.
Closes#24373
Closes#18470
## Summary
Exclude nested pragma comments like the `# noqa` in `test # comment 1 # noqa` from `E501`, `W505` and the formatter's line width calculation.
## Test Plan
- Added linter test fixture `E501_5.py` with preview test cases for
nested pragmas (`# comment #noqa`, `## noqa`, `# comment # type:
ignore`)
- Added formatter test fixture `trailing_pragma_nested.py` demonstrating
stable vs preview reserved width behavior
- Doctests for `find_trailing_pragma_offset` in `pragmas.rs`
---------
Co-authored-by: Micha Reiser <micha@reiser.io>