## Summary
Add unsafe fixes to diagnostics that suggest `Literal`: wrap integer,
bytes, and boolean literals and strings that cannot be parsed as forward
annotations in `Literal[...]`, and replace parameterized `LiteralString`
with `Literal` when its arguments are valid literal parameters.
The fixes preserve literal spelling and reuse unshadowed import aliases.
They import from `typing` on Python 3.8+, or from `typing_extensions` on
Python 3.7 when it is a declared direct dependency and the installed
module exports `Literal`. The typing-module availability checks are
shared with the `reveal_type` and `assert_never` fixes.
Valid calls such as `collections.abc.Mapping.register(...)` currently
produce missing-attribute errors. Using `register` as a class decorator
can also hide the registered class’s members and generic parameters,
causing further false positives and negatives.
This PR gives protocol classes declared in source or ordinary `.pyi`
files the metaclass `typing._ProtocolMeta`, which derives from
`ABCMeta`. That makes ABC methods available and preserves the types of
registered classes. It also lets ty report incompatible custom
metaclasses.
Typeshed’s standard-library stubs need a narrow compatibility exception:
they sometimes list `Protocol` bases that the runtime class does not
inherit. When those bases do not select a custom metaclass, ty uses
`ABCMeta` for attribute lookup but does not let that inferred metaclass
create artificial inheritance conflicts. This applies to bundled or
configured typeshed, not to other `.pyi` files. An explicitly chosen
custom metaclass continues to constrain subclasses. Known built-ins
retain their actual `type` metaclass.
Mypy applies a similar permissive fallback more broadly; this PR limits
the exception to typeshed.
Fixes https://github.com/astral-sh/ty/issues/1204.
## Ecosystem results
The [current ecosystem
run](https://github.com/astral-sh/ruff/actions/runs/32385816515/attempts/1)
removes missing-`register` errors in pandas, ppb-vector, Jinja, Spack,
and Ibis, plus two spurious metaclass conflicts in Spack’s legacy
`typing_extensions`. All 294 Ibis changes follow from recovering
registered classes’ members and generic parameters. The added
diagnostics expose existing annotation gaps, deliberately invalid tests,
and the known [virtual ABC subclass
limitation](https://github.com/astral-sh/ty/issues/2391).
The beartype library has an [old compatibility
workaround](https://github.com/beartype/beartype/blob/d6c056225f2e5632ea72006b43ffb066bd9a2c6b/beartype/typing/_typingpep544.py#L73-L79)
for the fact that `_ProtocolMeta` used to be missing from typeshed; it
substitutes `ABCMeta` under `if TYPE_CHECKING`, so it now produces a
metaclass conflict. We accept this false positive: typeshed now declares
`_ProtocolMeta`, and beartype can remove the workaround and derive from
`type(Protocol)` directly.
## Test plan
- Cover protocol metaclasses, bundled and custom typeshed, and
registration decorators that preserve generic classes.
- Check compatible and conflicting metaclasses, built-in collections,
and inheritance through ordinary and dynamically created classes.
- Protect subclass-type bounds, class-namespace attributes, quoted
unions, and dataclass-transform inheritance.
## Summary
This is a first step towards #20411. After this, we would work to either
upstream changes or adjust things within Ruff.
Performance improvements have already been upstreamed
(https://github.com/rust-lang/annotate-snippets-rs/pull/437,
https://github.com/rust-lang/annotate-snippets-rs/pull/440)
Re-implemented the following from the fork:
- Hidden snippets
(https://github.com/rust-lang/annotate-snippets-rs/issues/448)
- Renderer-level disabling of hyperlinks
- Jupyter cell indices
(https://github.com/rust-lang/annotate-snippets-rs/issues/449)
- Fixable mark
(https://github.com/rust-lang/annotate-snippets-rs/issues/450)
- `lineno_offset` (maybe not needed if #20648 is taken care of)
- anonymized lines affects origin
(https://github.com/rust-lang/annotate-snippets-rs/issues/451)
- cut indicator (maybe not needed if we can switch to unicode renderer)
Additional hacks to minimize behavior change include
- Adjust upstream's hidden severity behavior to match Ruff's
(https://github.com/rust-lang/annotate-snippets-rs/issues/452)
- How newlines are annotated
(https://github.com/rust-lang/annotate-snippets-rs/issues/454)
- To not indent for line numbers when none will be present
(https://github.com/rust-lang/annotate-snippets-rs/issues/453)
- Enabling `simd` by default
All of the above represents work that needs to be resolved to address
#20411.
`lib.name` hack is used to simplify maintenance with the assumption that
we'll continue working towards #20411.
Behavior changes that were included
- Tweak when implicit padding lines are present
- Fix Origin's line/column is now derived from the primary annotation
- Make consistent the rendering of primary/context annotations
- Fix snippets to show up for empty source (but may degrade #24528)
- Fix handling of term width
- Fix rendering of annotations between newline and first character
(granted, maybe the caller is wrong)
- Tweak multiline annotation markers
New or upcoming features annotate-snippets features that we will be able
to leverage by moving this forward
- `AnnotationKind::Visible` to improve the quality of built-in
code-folding if Ruff were to switch to it
- `DecorStyle::Unicode`
- *(upcoming)* No-graphiscs mode for screen readers (may also be useful
for AIs)
Deferred:
- Moving related `Group`s into the same `Report`
- Using `secondary_title` for any `Group` after the first in a `Report`
- Using a `Snippet`, instead of titles, for "virtual code" like
https://github.com/astral-sh/ruff/pull/27033#discussion_r3622727236
- Switching hidden snippets to `Origin`
## Test Plan
Within cpython, I ran
```console
$ hyperfine -i "../../ruff/target-baseline/release/ruff check" "../../ruff/target-update/release/ruff check" -N -m20 --output=null
Benchmark 1: ../../ruff/target-baseline/release/ruff check
Time (mean ± σ): 27.5 ms ± 1.1 ms [User: 33.0 ms, System: 108.4 ms]
Range (min … max): 25.0 ms … 31.7 ms 94 runs
Warning: Ignoring non-zero exit code.
Benchmark 2: ../../ruff/target-update/release/ruff check
Time (mean ± σ): 28.1 ms ± 2.0 ms [User: 33.4 ms, System: 105.7 ms]
Range (min … max): 24.8 ms … 38.9 ms 102 runs
Warning: Ignoring non-zero exit code.
Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet system without any interferences from other programs. It might help to use t
he '--warmup' or '--prepare' options.
Summary
../../ruff/target-baseline/release/ruff check ran
1.03 ± 0.08 times faster than ../../ruff/target-update/release/ruff check
```
Before:
<img width="486" height="583" alt="image"
src="https://github.com/user-attachments/assets/2d94dabf-b28d-48c1-b17c-52aac59fd57e"
/>
After:
<img width="486" height="583" alt="image"
src="https://github.com/user-attachments/assets/b388ada3-e3d7-4ba0-b90b-1330eb81f4be"
/>
## Summary
Fixes https://github.com/astral-sh/ty/issues/1627.
Here's an example diagnostic with the current release of ty:
<img width="2286" height="286" alt="image"
src="https://github.com/user-attachments/assets/46ab9154-aaf0-4451-b301-5ff12fcd8507"
/>
On this branch, this diagnostic becomes:
<img width="1464" height="408" alt="image"
src="https://github.com/user-attachments/assets/449e0b6c-58d2-4501-91df-c267db6f48ca"
/>
The exact span of the node that creates the syntax error is now retained
and highlighted in the diagnostic.
## Implementation
Propagating the range of the node inside the string annotation into the
diagnostic is trivial. However, naively implementing that quickly
revealed that this would make diagnostics like this unsuppressable:
```py
x: """list[
yield from range(42)
]"""
```
The primary range of the diagnostic is now specifically the `yield from
range(42)` part of the string rather than the string node as a whole.
But I cannot add a `ty: ignore` comment that is either on or above the
`yield from range(42)` part of the string -- the "comment" there would
just become part of the string.
This PR also improves the consistency of our parser error messages in
general when it comes to capitalization.
## Test Plan
Mdtests extended and updated
---------
Co-authored-by: Micha Reiser <micha@reiser.io>
## Summary
Like Mypy and Pyright, we now ignore annotations on non-name (i.e.,
subscript and attribute) targets, and emit a diagnostic for the invalid
annotation -- with the exception of annotated assignments on `self` and
class attributes (i.e., receivers).
Closes https://github.com/astral-sh/ty/issues/509.
## Summary
Currently we call `infer_annotation_expression` on return annotations
and parameter annotations, and then after that we do some ad-hoc checks
to make sure that the `TypeAndQualifiers` returned doesn't actually have
any qualifiers in it. A simpler way of checking that an annotation
expression doesn't contain type qualifiers is by simply calling
`infer_type_expression` rather than `infer_annotation_expression`, since
type qualifiers are always banned in type expressions -- they're only
valid in annotation expressions.
A naive way of doing this would regress our error messages -- rather
than saying "Type qualifiers are not valid in parameter annotations", we
would end up with the vaguer and more jargon-y "Type qualifiers are not
valid in type expressions" error message. That's easily fixed, however,
by plumbing some more context through our existing `InferenceFlags`
bitflag. Doing this allows us to improve many of our existing error
messages as well.
## Test Plan
mdtests updated
## Summary
In lots of places in our type-expression parsing, we continue to call
`self.infer_type_expression()` on sub-expressions in the AST even after
we've already determined that the type expression is invalid. I think
that's generally a mistake; it often leads to us emitting many
diagnostics on a single type expression when one would really be
sufficient. This PR switches many callsites from `infer_type_expression`
to `infer_expression`, to avoid this phenomenon of cascading diagnostics
in error cases.
## Test Plan
Mdtests and snapshots updated.
## Summary
Two small followups to d9daf6d430:
- A minor fix for a bug I introduced in that commit. We should reset
`context.multi_inference` to the state it was previously in rather than
hardcoding `false` here. This has real-world impact on `altair` (see the
mypy_primer report)
- Also detect invalid PEP-604 unions involving `types.GenericAlias`
instances and objects that don't have `__or__` methods at runtime. These
need to be special-cased in the same way as class-literal types, because
of typeshed's very forgiving `types.GenericAlias.__or__` annotation:
https://github.com/python/typeshed/blob/07ffb67b924d5f532f8b8b72a4902f58faae6aca/stdlib/types.pyi#L700-L702
## Test Plan
mdtests extended
- Updated the `prek` hook to replace `blacken-docs` with ruff
- Added some formatting directives to preserve obvious intentional
formatting choices
- Made some choice edits to content for better readability with new format
- Formatted md tests and updated snapshots
This multiplication was always using only using the larger sub-ast size
value, which didn't do anything unsound but made sub-string annotations
become exhausted on files 32x smaller than expected (16k nodes instead
of 512k nodes).
Also I decided to do a cleanup pass on the diagnostics to make them more
precise and helpful.
I ultimately hunted down the core issue with my previous approach to "if
you give string annotation sub-AST nodes any kind of NodeIndex, a bunch
of random places in the code will start thinking it's ok to store info
about them, which is a problem because they all count up from index 0
which creates conflicts/crashes".
Rather than trying to play whackamole I decided to create a scheme to
give string annotation nodes a unique NodeIndex by shifting up the
parent node's index -- so 0xAB's sub-AST nodes look like 0xAB00...0000,
0xAB00..0001, etc. This scheme avoids any collisions for any reasonable
AST (most string annotations are like, a dozen sub-nodes, so they need
maybe 4 or 5 bits, which would require hundreds of MB of python code to
run out of bits...).
As a bonus, this admits an extremely simple implementation of recording
and fetching sub-AST types... they just are stored now, and you can just
pass in their NodeIndex and get back actual results.
* Fixes https://github.com/astral-sh/ty/issues/1640
* Fixes https://github.com/astral-sh/ty/issues/2028
## Summary
This PR adds diagnostic for invalid binary operators in type
expressions. It should close https://github.com/astral-sh/ty/issues/706
if merged.
Please feel free to suggest better wordings for the diagnostic message.
## Test Plan
I modified `mdtest/annotations/invalid.md` and added a test for each
binary operator, and fixed tests that was broken by the new diagnostic.