Store positional and keyword arguments in one list for calls and class headers. Source-order visitors can walk the list directly, and evaluation-order visitors visit starred arguments before keywords.
Box keyword values so a positional argument does not carry the full size of a keyword entry.
## Summary
Today, ty panics on `class C(*(int, Generic[T, U]))` when `T` has a
default and `U` is required. To locate the error, ty looks for the
second base in the class header. The header contains a single starred
expression that expands to two bases.
This PR changes the diagnostic to use the source expression ty recorded
for each base during expansion, following #24695 and #24699. The
diagnostic now underlines `T, U` and reports that a required type
parameter cannot follow one with a default.
Added an mdtest that checks the diagnostic on the unpacked generic base.
## Summary
This PR splits `mismatched-type-name` into eight independently
configurable warnings for `NewType`, functional enums, `TypeVar`,
`ParamSpec`, `TypeVarTuple`, `namedtuple`/`NamedTuple`, `TypedDict`, and
`TypeAliasType`.
~~This PR is actually quite a bit less breaking for users than other
rule splits we've done over the last few days. Because
`mismatched-type-name` is no longer the canonical name for _any_ rule on
this branch, we're able to make it a deprecated alias for _all_ the new
rules, meaning that it will continue to work on the CLI, in ty
configuration files, and ty suppression comments (we'll just emit a
warning diagnostic telling users to upgrade their configuration
file/suppression comment).~~
~~A more specific rule takes precedence over the deprecated name if both
are given within the same configuration table. On the command line, the
last matching flag wins (e.g. `--error=mismatched-enum-name
--ignore=mismatched-type-name` will suppress diagnostics for mismatched
enum names).~~
(After review from @MichaReiser, I backed out these changes. It's now a
somewhat breaking change, because the rule is now marked as "removed"
rather than "deprecated".)
As well as splitting the rule into eight separate rules, it also cleans
up our error recovery for several of these type factories. For example,
in the following snippet, we fail to recognize `T` as a `TypeVar` due to
bad error recovery from the non-literal name. But we don't really need a
nonliteral name in this case to determine that it's a validly
constructed TypeVar. It's probably a mistake, but there's no _soundness_
issue here:
```py
from typing import TypeVar
def f(x: str):
# main emits an error: `invalid-typevar`
# this PR emits a warning: `mismatched-typevar-name`
T = TypeVar(x)
def g(x: T) -> T:
return x
reveal_type(g(42)) # main: Todo. This PR: `Literal[42]`.
```
## Motivation
The main motivation here is that having a namedtuple or enum with the
"wrong name", for example, just seems much less "obviously wrong" than
having a typevar or paramspec with the "wrong name". The [ecosystem
report](https://github.com/astral-sh/ruff/pull/29957#issuecomment-6603964612)
is evidence of this. namedtuples and enums define runtime classes that
can be used in runtime contexts and instantiated; these constructs
aren't _just_ for type checkers.
TypeVars/ParamSpecs/TypeVarTuples/NewTypes/TypedDicts are all _only_ for
type checkers and there's no plausible reason why you'd want to give the
type a different name to the variable it's assigned to.
`mismatched-namedtuple-name` and `mismatched-enum-name` are therefore
now disabled by default, matching the default severity we already
settled on for `mistmatched-sentinel-name` in
https://github.com/astral-sh/ruff/pull/27206. `mismatched-typevar-name`,
`mismatched-paramspec-name`, `mismatched-typevartuple-name`,
`mismatched-enum-name`, `mismatched-namedtuple-name` and
`mismatched-newtype-name` all remain enabled by default with `warning`
severity.
When `**kwargs` is annotated with `P.kwargs`, the matching `*args`
annotation must be `P.args`. We emit a diagnostic on `P.args` as a
result -- but if you add a suppression comment to that line... a new
diagnostic pops up on `P.kwargs`! That's not how suppression comments
are really meant to work. This PR fixes that.
## Test plan
### Before (`aa46aae60`)
Adding `# ty: ignore[invalid-paramspec]` on the `*args` line moves the
diagnostic to `**kwargs`.
https://github.com/user-attachments/assets/50ee1858-8126-45bb-896f-2517b98950fe
### After (`684c14273`)
Adding the same ignore clears the diagnostic.
https://github.com/user-attachments/assets/0a7fc2b0-ffd0-49b4-87ae-714835b7421f
When a generic function expects a callback that returns `Id[R]`, where
`type Id[T] = T`, ty can lose the relationship between `R` and a type
variable from the caller.
```py
from collections.abc import Callable
from typing import reveal_type
type Id[T] = T
def run[R](callback: Callable[[], Id[R]]) -> R:
return callback()
def check[T](callback: Callable[[], T]):
reveal_type(run(callback)) # Before: Unknown. After: T@check.
```
When an alias bound expands to a type variable, record the relationship
between the two variables. In this example, the solver can infer `R`
from the callback's return type `T`.
This is just an edge case Codex noticed, not something that showed up in
real code, so it's quite low priority; it just seemed like a very simple
and clearly correct fix.
Disabling `invalid-generic-class` currently also erroneously suppresses
`shadowed-type-variable` diagnostics on nested classes, even when
`shadowed-type-variable` is enabled. With this fix,
`shadowed-type-variable` controls those diagnostics independently.
## Test plan
Save this example as `example.py`.
```python
class Outer[T]:
class Inner[T]:
pass
```
Run `ty check --python-version 3.12 --ignore invalid-generic-class
--error shadowed-type-variable example.py` with a build of ty from each
branch. Before the fix, ty reports no diagnostics. With the fix, it
reports `shadowed-type-variable` for `Inner`’s `T`. Replacing `--error
shadowed-type-variable` with `--ignore shadowed-type-variable`
suppresses the diagnostic.
## Summary
In this code on ty 0.0.107, `wrap(lambda n: n + 1)` alone works fine,
but passing that same result straight to `register` makes ty report an
error on `n + 1`.
```python
from collections.abc import Callable
def wrap[**P](f: Callable[P, int]) -> Callable[P, int]:
return f
def register[**P](f: Callable[P, int]) -> None: ...
wrap(lambda n: n + 1) # OK
register(wrap(lambda n: n + 1)) # Error in ty 0.0.107
```
`P` stands for all the parameters of the callback. Neither `wrap` nor
`register` says what type `n` has. In the nested call, we wrongly gave
`n` the type of the whole parameter list, `(*args: object, **kwargs:
object)`, and then rejected the addition.
With this fix, `n` stays `Unknown` in both calls. If the callback
signature does give a concrete type for a parameter, ty still uses it.
Fixesastral-sh/ty#5075.
ty could infer `Unknown` for a call when one overload left a parameter
unannotated and another supplied a type for it. In this example,
`first()` has no arguments to infer `T` from, so its return type
defaults to `int` unless the caller supplies an expected type.
```python
from typing import overload, reveal_type
from typing_extensions import Never, TypeVar
T = TypeVar("T", default=int)
def first(*values: T) -> T:
return values[0]
@overload
def choose(value, index: int) -> int: ...
@overload
def choose(value: Never, index: object) -> object: ...
def choose(value, index):
return index
reveal_type(choose(first(), 1)) # previously Unknown, now int
```
Both overloads initially match. The first gives `first()` no expected
type, so it infers `int`. The second infers `Never` for the same
expression. Previously, ty reused `Never` while checking the first
overload and treated the call as ambiguous. It now uses the argument
type inferred for each overload, so the first overload wins.
## Test plan
Checked the original example with a nested `create()` whose return type
is a type variable bounded by `int`. ty reveals `int` for
`choose(create(), 1)`, including when `create()` is replaced by
`next(iter([]))`.
On main, we hang on this snippet:
```python
class Property[T]:
value: T
class Either[T](Property[T]):
def __init__[A, B, C](
self: "Either[A | B | C]",
first: type[Property[A]],
second: type[Property[B]],
third: type[Property[C]],
) -> None: ...
result = Either(Property[float], Property[str | None], Property[bytes])
```
The three arguments determine `A`, `B`, and `C`, so `result` should have
type `Either[float | str | None | bytes]`. Previously, we kept
rebuilding the same union in different orders, treating each order as a
new result, and cycling forever.
ty now substitutes one type variable at a time. It lets other bindings
resolve a cycle before choosing a fixed point or falling back to
`Unknown`. When two variables refer back to each other, ty also checks
their bounds. For example, a constructor’s `int` bound still applies
when the class parameter itself is unbounded.
Fixesastral-sh/ty#5057
I tried all six orders of `A`, `B`, and `C` in the `__init__`
declaration above. With the call unchanged, ty inferred `Either[float |
str | None | bytes]` every time.
## Ecosystem report
Bokeh finished in 20 seconds after timing out at 180 seconds on the
base. Across five other projects, 58 existing diagnostics changed which
unresolved type variables ty displayed. Both revisions rejected the same
operations, and typed array inputs retained their concrete dtype. No
stable diagnostics were added or removed. [Full report for
`baf1faa7`](https://github.com/astral-sh/ruff/pull/30510#issuecomment-6965510325)
When inferring a specialization for a generic call, we try each element
of a union return context individually before falling back to the entire
union. If a nested call fails during one of those attempts, its error
recovery type can make the proposed specialization appear valid even
though it isn't.
There is an open constraint solver bug (astral-sh/ty#4696) where we
don't verify that all solutions collectively "cover" all of the valid
specializations of any _non-inferable_ typevars. This can cause us to
infer a provisional specialization that is incompatible with the actual
arguments of a nested call. We then incorrectly accept the narrowed
specialization because the nested call's recovery type satisfies the
expected parameter type.
For now, we work around this by recording when call binding fails, and
threading that information through `InferContext` so that `try_narrow`
can reject speculative inference attempts that depend on error recovery.
We track these failures even when their diagnostics are suppressed, but
ignore failures in unreachable code. For calls with a single candidate
signature, we also allow narrowing to succeed when the failed argument
corresponds to a parameter whose type contains no typevars, since that
argument cannot influence the proposed specialization.
There are still limitations around callable unions and overloads, which
#27942 will address.
Closesastral-sh/ty#4971
## Summary
This PR adds a new `project-manager` table:
```toml
[project-manager]
project = "uv" | "auto" | "none"
project-mode = "sync" | "metadata"
script = "uv" | "none"
```
This PR only implements the script part, but I listed all options to
outline where we want to go with this in the future.
`script = "uv"` opts in to installing PEP 723 script dependencies with
uv (to reduce the need for `TY_UV`). For now, `project-manager.script`
defaults to `"none"`, but I plan to change this to `"uv"` in the near
future. The new option follows the same configuration precedence as
other script settings:
* A script with its own `tool.ty` section uses the value from its own
table to control whether it uses uv to sync its dependencies or not.
* Scripts without a `tool.ty` section use the project level
`project-manager.script`
For now, overriding the setting based on file patterns isn't supported
(I have no strong reason against it, it's mainly to cut scope).
`TY_UV` takes precedence over any setting in the configuration (similar
to other CLI options that support environment variables).
The recommended way to check untrusted code is to set `TY_UV=off`, which
overrides `project-manager.script`.
For now, we keep supporting `experimental.useUv` for scripts, but the
in-editor `configuration.project-manager.script` takes precedence over
`experimental.useUv`.
## Why is the setting not under `environment.
For projects, setting `project-manager.project = "metadata"` also
changes how we discover the project (we prefer the workspace over the
closest ty project), and `metadata` doesn't initialize the Python
environment.
## LSP watch mode
Changing `project-manager.script` at the project level requires saving
any open script (or closing and reopening) for the sync to start. This
simplified the implementation a fair bit and seems an okay trade off.
The "gradual parameters" `*args: Any, **kwargs: Any` are special-cased
to match any possible set of arguments, even when they follow other
"normal" parameters in a signature. But they do not make the _entire_
signature gradual -- calls still must satisfy the other normal
parameters present.
We currently are "too gradual" in our treatment of these signatures,
allowing a callable with an initial typed parameter plus gradual
parameters to be assigned to a callable type that has the wrong type for
its first parameter:
```python
from typing import Any, Callable
def lowercase(value: str, *args: Any, **kwargs: Any) -> str:
return value.lower()
callback: Callable[[int], str] = lowercase # Previously accepted by ty
callback(1) # AttributeError
```
This PR rewrites callable assignability (with a significant net
reduction in Rust LOC!) and now correctly reports the error on this
assignment. The first positional argument binds to `value`, which
requires a `str`. The gradual `*args` and `**kwargs` only describe the
remaining arguments.
Closesastral-sh/ty#5052.
## Ecosystem report
All 104 public ecosystem changes on `8681ef368f` are expected. Most new
errors identify callbacks or overrides that cannot accept an argument
their types promise. [Full ecosystem
report](https://github.com/astral-sh/ruff/pull/30509#issuecomment-6966654399)
## Summary
Fixesastral-sh/ty#5035.
When targeting Python 3.12 or earlier, infer covariance for read-only
fields on Pydantic models frozen through a class keyword or
`ConfigDict`. Use the same setter lookup as assignment checking so a
custom `__setattr__` keeps the model invariant. On Python 3.13 and
later, the synthesized `__replace__` still makes these models invariant.
On main, `functools.partial` rejects a union-valued keyword when
different overloads accept its members and a required argument is left
for the caller. For example, ty accepts binding `option="text"` or
`option=None` separately, but reports `Expected str, found str | None`
here.
```python
from functools import partial
from typing import overload
@overload
def f(value: int, /, *, option: str) -> str: ...
@overload
def f(value: int, /, *, option: None) -> int: ...
def f(value: int, /, *, option: str | None) -> str | int:
return value if option is None else option
def apply(option: str | None) -> str | int:
bound = partial(f, option=option)
return bound(1)
```
When overload resolution retries with each union member, it matches
parameters again. This used to eliminate both overloads because `value`
was missing. The fix defers the missing-argument check on every match
and keeps all compatible overloads for the eventual call. In the
example, ty infers `str | int` for `bound(1)` because the result depends
on the stored keyword. An explicit override selects the corresponding
overload, so `bound(1, option=None)` returns `int`.
For an overloaded function whose two-positional-argument form requires
`option=None`, a partial `p` that stores an `option` of type `str |
None` rejects `p(1, 2)`. It accepts `p(1, 2, option=None)` because the
caller replaces the stored value.
## Test plan
Checked `partial(datetime.replace, tzinfo=tz)` in Pydantic with the
proposed typeshed sync from #30443, which splits `tzinfo` and `None`
into separate overloads. The fix removed the diagnostic on this call. A
direct check confirmed that the partial accepts a datetime receiver and
a `tzinfo=None` override, and rejects a missing or integer receiver.
[Pydantic call
site](https://github.com/pydantic/pydantic/blob/560f0f96a37d6913afdfddb2de22030ef08e68c3/pydantic/experimental/pipeline.py#L313)
ty could spend minutes inferring the type of a Bokeh `Either`
constructed from a property with 28 literal marker names and a property
accepting `str`. To rule out a candidate type, the solver explored
combinations of bounds for every marker. Rejecting both bounds for just
one marker is already a counterexample.
Before searching for a counterexample, ty now tries one path that makes
each atomic constraint on it false. It follows that path only while the
diagram already has the next node, so it never joins branches or
backtracks. If the path reaches false and its assignments are
consistent, ty can rule out the candidate immediately. Otherwise it uses
the existing exact search.
Fixesastral-sh/ty#5046.
## Test plan
I checked [Bokeh’s original
`dataspec.py`](https://github.com/bokeh/bokeh/blob/1c2309202d8623ede74a996a367f3b60eaf79411/src/bokeh/core/property/dataspec.py#L408-L409)
with the base and PR Linux binaries, the same checker configuration, and
Bokeh’s dependencies in a Python 3.12 environment. The base binary
exceeded a 20-second timeout. Removing just the reported constructor
call let it finish in 0.69 seconds. The PR binary checked the unmodified
file in 0.80 seconds with no diagnostics. Using Bokeh’s classes
directly, it also accepted `assert_type(Either(MarkerType,
Regex("^@.*$")), Either[str, str])`.
## Summary
ty previously accepted `call_with_int(callback)` below, even though a
caller can pass `forward` a function that takes no arguments.
```python
from typing import Callable, Concatenate
def call_with_int(callback: Callable[Concatenate[int, ...], None]) -> None:
callback(1)
def forward[**P](callback: Callable[P, None]) -> None:
call_with_int(callback) # now an error
```
Check the fixed positional parameters before the caller's `ParamSpec`
against the prefix required by `Concatenate`. This also rules out
overloads that require more positional parameters than the callback
guarantees.
This fixes a bug that otherwise caused new false-positive regressions on
https://github.com/astral-sh/ruff/pull/30423.
## Ecosystem report
The public run for `deb2cd0da5` found three expected changes across 165
projects. ty now reports a missing class receiver in a
pytest-autoprofile wrapper annotation and removes two incorrect
overload-related errors in discord.py. [Full ecosystem
report](https://github.com/astral-sh/ruff/pull/30470#issuecomment-6927397479).
ty used to abort with a stack overflow while checking this
`super().__init__` call. It now reports `invalid-argument-type` on
`value` in `super().__init__(value)`.
```python
class Box[T]:
def __init__[A, B](self: "Box[A]", value: A | tuple[B]) -> None: ...
class Child[T](Box[bytes]):
def __init__(self, value: T | tuple[T]) -> None:
super().__init__(value)
```
The receiver for that call is a `Child[T]`, which is already a
`Box[bytes]`. That fixes `A` to `bytes`. A one-element tuple is valid
because the initializer can infer `B = T`. But `Child[str]("hello")`
would forward a plain string where the initializer expects `bytes` or a
tuple.
ty now uses the receiver’s existing class type arguments when checking a
constructor method. Looking up the method on a bare generic class still
allows the call to infer those arguments.
The stack overflow happened while ty was trying to explain the failed
call. It combined type-variable bindings inferred from individual
arguments, then repeatedly substituted them into each other. A cycle
through a tuple element can add another layer of `tuple[...]` on every
substitution. The original reproducer had the same problem with a
callable’s return type. ty now drops the parts of a binding that would
keep growing and uses the remaining evidence to report the error.
Fixes https://github.com/astral-sh/ty/issues/5047
## Test plan
Compared the CLI on the merge base and this change using the example
above and the original reproducer in both PEP 695 and legacy `TypeVar`
syntax. All three aborted on the merge base and produced argument-type
diagnostics on this change. The example above passes with `Box[T]` as
the base.
## Ecosystem report
On `3c94d4a654a3c2cc6a05b4845d4057271599ebb0`, six new diagnostics
identify constructors that accept arguments incompatible with the
receiver’s specialization. The change also removes a false return
warning in PyMongo and simplifies two existing setuptools diagnostics.
All nine changes were reproduced, with no new regressions found. [Full
ecosystem
report](https://github.com/astral-sh/ruff/pull/30473#issuecomment-6931764426)
## Summary
The `ty-extensions` package lets Python evaluate and inspect
`Intersection[A, B]` and `Not[A]` at runtime. Update the stub docstrings
and the `experimental-syntax` documentation to point readers to the
package.
## Summary
Use a shared ty override for Ruff’s standalone scripts now that PEP 723
scripts can inherit project settings. Fix the diagnostics in the two PR
automation scripts so they can use the same checks.
## Notes to reviewers
All 28 standalone scripts passed ty with locked dependencies, and the
changed-file hooks and mocked workflow checks passed.
---------
Co-authored-by: Alex Waygood <alex.waygood@gmail.com>
Checking nested calls to a generic overload can spend most of its time
materializing the same type arguments repeatedly. For a type such as
`Row[Row[Any]]` with a covariant parameter, each layer could visit its
argument twice to compute the same top materialization. This doubles the
work with each layer.
Compute the top materialization lazily and reuse it when comparing the
argument with its materialization or constructing the result. This also
skips the top traversal when only the bottom materialization is needed.
Improves astral-sh/ty#5041.
## Test plan
I timed the reproducer from the issue on macOS arm64 using an optimized
debug build. Checking eight nested overloaded calls went from 29.05 s to
1.55 s. To isolate materialization from overload resolution, I also
checked `Top` applied to 23 nested `Row` types. That went from 3.08 s to
0.03 s. Both examples passed before and after the change.