Store positional and keyword arguments in one list for calls and class headers. Source-order visitors can walk the list directly, and evaluation-order visitors visit starred arguments before keywords.
Box keyword values so a positional argument does not carry the full size of a keyword entry.
## Summary
This PR splits `mismatched-type-name` into eight independently
configurable warnings for `NewType`, functional enums, `TypeVar`,
`ParamSpec`, `TypeVarTuple`, `namedtuple`/`NamedTuple`, `TypedDict`, and
`TypeAliasType`.
~~This PR is actually quite a bit less breaking for users than other
rule splits we've done over the last few days. Because
`mismatched-type-name` is no longer the canonical name for _any_ rule on
this branch, we're able to make it a deprecated alias for _all_ the new
rules, meaning that it will continue to work on the CLI, in ty
configuration files, and ty suppression comments (we'll just emit a
warning diagnostic telling users to upgrade their configuration
file/suppression comment).~~
~~A more specific rule takes precedence over the deprecated name if both
are given within the same configuration table. On the command line, the
last matching flag wins (e.g. `--error=mismatched-enum-name
--ignore=mismatched-type-name` will suppress diagnostics for mismatched
enum names).~~
(After review from @MichaReiser, I backed out these changes. It's now a
somewhat breaking change, because the rule is now marked as "removed"
rather than "deprecated".)
As well as splitting the rule into eight separate rules, it also cleans
up our error recovery for several of these type factories. For example,
in the following snippet, we fail to recognize `T` as a `TypeVar` due to
bad error recovery from the non-literal name. But we don't really need a
nonliteral name in this case to determine that it's a validly
constructed TypeVar. It's probably a mistake, but there's no _soundness_
issue here:
```py
from typing import TypeVar
def f(x: str):
# main emits an error: `invalid-typevar`
# this PR emits a warning: `mismatched-typevar-name`
T = TypeVar(x)
def g(x: T) -> T:
return x
reveal_type(g(42)) # main: Todo. This PR: `Literal[42]`.
```
## Motivation
The main motivation here is that having a namedtuple or enum with the
"wrong name", for example, just seems much less "obviously wrong" than
having a typevar or paramspec with the "wrong name". The [ecosystem
report](https://github.com/astral-sh/ruff/pull/29957#issuecomment-6603964612)
is evidence of this. namedtuples and enums define runtime classes that
can be used in runtime contexts and instantiated; these constructs
aren't _just_ for type checkers.
TypeVars/ParamSpecs/TypeVarTuples/NewTypes/TypedDicts are all _only_ for
type checkers and there's no plausible reason why you'd want to give the
type a different name to the variable it's assigned to.
`mismatched-namedtuple-name` and `mismatched-enum-name` are therefore
now disabled by default, matching the default severity we already
settled on for `mistmatched-sentinel-name` in
https://github.com/astral-sh/ruff/pull/27206. `mismatched-typevar-name`,
`mismatched-paramspec-name`, `mismatched-typevartuple-name`,
`mismatched-enum-name`, `mismatched-namedtuple-name` and
`mismatched-newtype-name` all remain enabled by default with `warning`
severity.
## Summary
Python assigns unpacking targets from left to right. By the time it
tries to unpack `pair` below, `value` already holds `"bad"`.
```python
def f(pair: tuple[int, ...]) -> None:
value: int | str = 1
try:
value, (a, b) = "bad", pair
except ValueError:
value.bit_length() # Raises AttributeError when pair has the wrong length.
```
ty accepted this call because it took a checkpoint only before the outer
unpack, when `value` was still `1`. We now take a checkpoint before
unpacking each tuple or list target. The handler sees `Literal[1,
"bad"]`, so ty flags the call.
We also take the nested checkpoint when the source is an exact-length
literal. If we replace `pair` with `(1, 2)`, the unpack always succeeds,
but ty can still include `"bad"` in the handler even if a later
assignment overwrites it before anything raises.
Closes https://github.com/astral-sh/ty/issues/4958.
## Summary
ty treats distinct bytes keys as the same key when decoding them as
UTF-8 produces the same replacement character. In this example, checking
one entry convinces ty that the other is an `int`.
```python
def increment(counts: dict[bytes, int | None]) -> int:
if counts[b"\xaa"] is not None:
return counts[b"\xff"] + 1 # The second entry can still be None.
return 0
```
We now escape the bytes when building the key for narrowing. This
preserves the value of each bytes literal, including literal
backslashes.
Closes https://github.com/astral-sh/ty/issues/2579.
## Summary
A nested function can see a later assignment to a name it closes over.
When ty indexed an assignment to `x`, it walked every enclosing-scope
snapshot in the file to find the ones that could see the new value of
`x`.
We now group the snapshots for lazy scopes by name. An assignment to `x`
visits only snapshots for `x`, and a binding in an intervening scope
stops the update from reaching a shadowed name.
A cold `ty check` of NumPy used 8.2–9.1% fewer instructions.
---------
Co-authored-by: Charlie Marsh <charliemarsh@openai.com>
## Summary
Consider a file that imports `x` from this module:
```python
x: int = 1
list
dict
list
```
Changing the last `list` to `dict` changes which name that expression
reads. Since #29956 stores the place ID of each read in the use-def map,
the edit also invalidates the importer's lookup of `x`, even though its
declaration and bindings are the same.
We now store the place ID alongside the use ID in the AST index. The
use-def map records the bindings that reach each use, so ty can reuse
the cached lookup of `x` across this edit.
With an unannotated `x = 1`, the end-of-scope bindings query also stays
cached. The full import lookup for that case still reruns because it
reads the file’s semantic index to check the visibility of the scope.
Follow-up to [Carl’s review of
#29956](https://github.com/astral-sh/ruff/pull/29956#pullrequestreview-5777183605).
A condition saved before a loop can keep narrowing a value after the
loop body has reassigned it. For example, after `is_none = x is None`,
ty can treat `x + 1` as safe even though a previous iteration set `x` to
`None`.
Invalidate the saved condition when a reachable loop backedge can carry
a write to a place it narrows. This uses the same whole-condition
invalidation as a reassignment in straight-line code.
Fixesastral-sh/ty#4934
## Test plan
Compared a guarded `x += 1` in a loop with a type-preserving `x += 1`
between saving and testing the condition in straight-line code. Ty
invalidates the saved narrowing in both cases.
## Ecosystem report
The public ecosystem run for `aa1cb33eeb` found one expected change.
StaticFrame no longer receives a false `redundant-condition-strict`
warning when a previous loop iteration can initialize `dtype`. The
result matched the reproduction with the workflow binaries.
[Full ecosystem
analysis](https://github.com/astral-sh/ruff/pull/30074#issuecomment-6666621923)
## Summary
When ty indexes a read like `obj.child`, it already knows which place
the expression refers to and which bindings can reach that read. Type
inference then reconstructs the place from the syntax and looks it up
again. We now record the place ID alongside the reaching bindings and
reuse it when inferring a load in the same scope. Separately parsed
string annotations still resolve from syntax.
We store use IDs in node order, with a bitmap marking which nodes have
uses. We use two-byte records when the IDs for the place and the set of
reaching bindings each fit in a byte for every use in a scope.
Otherwise, each record in that scope takes eight bytes. Across the
source files we checked in Home Assistant, Prefect, Sphinx, Trio, and
Flake8, only 28 of 135,894 scopes with recorded uses (0.021%) needed
eight-byte records. Those scopes accounted for 0.99% of all recorded
uses.
On Home Assistant, `ty check` on macOS arm64 executed 2.49% fewer
instructions and used 7.8 MiB less peak memory than `main`.
## Summary
At the end of this `if`, ty merges the narrowing and reachability
constraints for `number`, then repeats the same work for `text`. Both
parameters reach the join through the same paths, and each keeps its
original binding and declaration in both branches.
```python
def pair(flag: bool, number: int, text: str):
if flag:
result = (number, text)
else:
result = (text, number)
reveal_type(number) # int
reveal_type(text) # str
return result
```
We now remember the last merge at the join for which both branches agree
on a single binding and a single declaration. If another name has the
same input constraints and pending paths in both branches, we reuse the
merged constraints and keep that name’s own definitions. We also inline
the simple cases and cache lookups for constraint `AND` and `OR`
operations.
(We used 2.11% fewer instructions for a cold check of PyTorch.
Instruction counts also fell on Black (3.31%) and Jinja (0.90%).)
---------
Co-authored-by: Charlie Marsh <charliemarsh@openai.com>
A generic type alias defined in a class could report a class-local name
as undefined or silently resolve it to a module-level name. For example,
in:
```py
class Outer:
class Inner:
type Pair[T] = tuple[T, Inner]
```
ty skipped the class scope that defined `Inner` because the alias's
type-parameter scope sat between its value and the class.
Name lookup now sees the enclosing class through consecutive annotation
scopes. The semantic index and type inference use the same visibility
rule.
Fixes https://github.com/astral-sh/ty/issues/4886.
## Summary
A cold `ty check` of SQLAlchemy executes 4.75% fewer instructions in a
release build (26.24 billion to 25.00 billion) and produces identical
diagnostics. Four-worker runs suggested a 2.34% speedup, though the
confidence interval included no change.
On `main`, looking up a generated member such as a dataclass’s
`__init__` can classify the class twice, once before searching its body
and again when that search misses. Across the SQLAlchemy check, ty
classified classes 3.19 million times. We now read the relevant
class-header fields together and reuse the classification after a miss.
A lookup like `obj.x` can find `x` in the class body or in a method that
assigns to `self.x`. We now fetch the class-body symbol table and the
names assigned to method receivers in one Salsa query per class. For
instance lookups, we also skip the `NamedTuple` field check when the
class body has no symbol named `x`, since every field is declared there.
Sharing the two inputs means that adding an unrelated class attribute
can invalidate a lookup that only needed receiver names. Applying
`dataclass` to the same class with different options can also cache
those names once per variant.
---------
Co-authored-by: Charlie Marsh <charliemarsh@openai.com>
## Summary
Ty could panic on `assert predicate` inside a loop where `predicate`
starts as a lambda and a later iteration rebinds it to a class. When
cycle recovery added the class to the operand type, ty looked up its
metaclass to determine its truthiness. That lookup could introduce a new
inference cycle while Salsa was recovering the first one.
Reachability now evaluates the predicate from the operand types that
expression inference has already normalized. Outside loops, it caches
the result so repeated walks of the same condition can reuse it. Inside
a loop, a condition can depend on an assignment it guards, so the caller
reads the inferred operands directly. Expression inference no longer
stores predicate truthiness or recomputes it during recovery.
The 163-project ecosystem run produced identical diagnostics. Checking
isort took 291 ms, up from 248 ms on the base revision, which gives back
some of the speedup from #29706. The average timing factor across
projects was 1.02×. CodSpeed found no significant changes among the 214
benchmarks it compared. [Ecosystem
diff](https://dqhfo45wdv0ek.cloudfront.net/ruff/pull-requests/29759/ty-ecosystem/40467940782/1/diff.html)
·
[Timings](https://dqhfo45wdv0ek.cloudfront.net/ruff/pull-requests/29759/ty-ecosystem/40467940782/1/timing.html)
·
[CodSpeed](https://github.com/astral-sh/ruff/pull/29759#issuecomment-6435864937)
Closes https://github.com/astral-sh/ty/issues/4834.
## Summary
Checking Home Assistant’s Anthropic integration took 283.1 ms before
this change and 252.6 ms after (10.8% less time, with a 95% paired
interval of 7.8–13.7% less). Each of 24 blocks ran `ty check
homeassistant/components/anthropic` twice with the baseline build and
once with the changed build, in balanced, randomized order. All runs
emitted the same 41 lines of diagnostics. The full Home Assistant
timings did not resolve this patch’s individual contribution.
The Anthropic SDK’s synchronous and asynchronous streaming functions
each test 53 event names in one `or` expression. At each operand, ty
used to reapply every condition leading up to it, recording 1,378
conditions per expression. The continuation already carries those
earlier conditions, so adding just the preceding operand’s condition
brings the count down to 52.
Checking a file that imports the two original function bodies went from
95.9 ms to 19.3 ms. Changing just those two `or` chains to tuple
membership checks made both builds take about 12 ms. Toggling prefix
replay in a single release executable also reproduced the integration
improvement (270.1 ms to 237.5 ms).
An assignment expression (`:=`) can introduce a new binding partway
through a chain. Those chains keep the earlier conditions so the
reassigned name gets the same reachability information.
## Summary
Cold CLI checks of Home Assistant’s KNX integration took 14% less time,
and checks of its Reolink integration took 4.5% less time. Each row
averages 12 paired runs of `ty check
homeassistant/components/<integration>` with the project’s dependencies
installed. Both integrations emitted the same diagnostics before and
after.
| Integration | Before | After |
| --- | ---: | ---: |
| KNX | 389.5 ms | 334.9 ms |
| Reolink | 396.0 ms | 378.1 ms |
The KNX decoder has a long sequence of branches inside a `try` block.
Reading a name like `GroupValueRead` for the first time used to trigger
another snapshot of the bindings visible to the exception handler.
Restoring an earlier snapshot already gives that name the same undefined
state. We now leave the checkpoint’s binding revision alone on that
first read and advance it when the place gets a definition.
In isolation, a cold CLI check importing the original
`xknx.APCI.from_knx` body went from 84.5 ms to 36.4 ms (57%). For
`reolink_aio.Host.map_channel_json_response`, it went from 41.9 ms to
32.4 ms (23%).
When a match case captures the subject under its own name and its guard
fails, later cases now narrow that capture. Previously, those cases
could retain a wider type because narrowing targeted only the binding
read before the match.
Retain bindings that capture the entire subject under its original name,
including `as` patterns and OR alternatives. Captures of extracted
values and assignments made by guards remain distinct from the saved
subject.
Fixesastral-sh/ty#4761.
## Summary
A full ty check of isort went from 204 to 172 ms in the ecosystem run.
Across all 163 projects, the average timing factor was 1.00×. Local
checks of two isort revisions on macOS took 16–23% less time. [Ecosystem
timings](https://dqhfo45wdv0ek.cloudfront.net/ruff/pull-requests/29706/ty-ecosystem/40342132072/1/timing.html)
In `if bool(stop()):`, the call to `bool` has type `bool`, but neither
branch can run if `stop()` returns `Never`. Ty uses the operand types to
work this out. In a loop, those types can depend on an assignment
guarded by the same condition. Computing types and truthiness in
separate queries can make ty revisit that loop repeatedly.
We now record each predicate’s evaluation context when we index it.
Expression inference computes the predicate’s truthiness alongside its
operand types and recomputes it if cycle recovery widens them. Ordinary
conditions and match guards both read that result.
The only warning that changed consistently across ecosystem runs was in
psycopg. It now lists the `int` class separately even though `type[str |
Any | int]` already includes it. [Ecosystem
diff](https://dqhfo45wdv0ek.cloudfront.net/ruff/pull-requests/29706/ty-ecosystem/40342132072/1/diff.html)
## Summary
Home Assistant’s `Entity` class reads or writes 106 different `self`
attributes across 70 methods, but none of those methods annotate one.
When checking overrides, ty could search all 70 methods for each name
just to determine that there was no declaration.
Collect receiver annotations once per class and reuse that list when
checking overrides. When inferring an implicit attribute, skip names
that the class only reads: a read of `self.x` does not define `x`.
On a class with 256 methods that each read eight different attributes,
median CPU time dropped from 896 ms to 126 ms over six runs. ty still
reported the one incompatible annotation. Two full Home Assistant checks
on Linux went from 47.0 to 45.4 s and from 49.4 to 49.2 s, with about 21
MiB less peak memory and the same 13,419 diagnostics.
Canonical equivalence can make distinct string values look the same. ty
escapes their displays to distinguish them, but the current approach
relies on ICU4X's deprecated `NfcInert` property, which [ICU4X plans to
remove](https://github.com/unicode-org/icu4x/issues/7892), and can
escape characters that do not need escaping.
Replace inertness checks with NFC checks on runs of unescaped text. When
adding a character would make a run non-NFC, escape that character and
continue checking after it. The resulting string literal is in NFC and
evaluates to the original value, so canonically equivalent values remain
distinguishable.
For example, `\u212b` (the Angstrom sign) is canonically equivalent to
`Å`. ty now keeps the unrelated `é` visible while still distinguishing
the two values:
```py
reveal_type("éÅ")
# main: Literal["éÅ"]
# this PR: Literal["éÅ"]
reveal_type("é\u212b")
# main: Literal["\xe9\u212b"]
# this PR: Literal["é\u212b"]
```
The change also uses Unicode grapheme-break properties to keep
characters visually separate from the quotes and escape sequences around
them. It escapes the braille blank because it can look like an ordinary
space.
Extract `is_end_of_scope_reachable(db, ScopeId)` into the reachability
module and use it when checking ordinary and generator functions for
implicit `None` returns.
This will allow us to use this helper in the return type inference
implementation (#29253).
## Summary
We now mark both branches of a condition with type `Never` as
unreachable:
```python
from typing import Never
def f(value: Never):
if value:
result = 1 # Unreachable.
else:
result = 2 # Also unreachable.
```
Previously, we treated `Never` as having ambiguous truthiness, allowing
either assignment to affect inference. In the reported loop, the
variables depend on each other, so including these impossible
assignments made their inferred types depend on which file we checked
first.
We add `Truthiness::Uninhabited` to represent the absence of either
boolean outcome. `Type::bool` and `Type::try_bool` return this variant
for `Never`, replacing the opt-in `bool_if_inhabited` API and making
callers handle the distinction explicitly.
Closes https://github.com/astral-sh/ty/issues/4577.
`Project::dependency_metadata` resolves Python environments inside a
Salsa query without tracking those inputs. This can leave
invalid-environment, missing-environment, and environment-mismatch
diagnostics stale after an environment changes.
Store the selected environment's resolution result in `ProgramSettings`
and track uv's environment directory through Salsa. Resolve uv's
canonical path whenever the query runs so it observes symlink changes
even when uv metadata is unchanged.
After a failed selection, retain the last usable environment for
watching and recovery, and preserve the checking program's search paths.
Dependency metadata reports the current environment error.
Fixesastral-sh/ty#4604.
## Summary
We now narrow unions of tuples when an element is checked for truthiness
or with `isinstance`, `issubclass`, or `TypeIs`. Eliminating an
alternative also narrows the tuple’s other elements, preserving the
relationship between a tag and its associated value. For example,
checking a boolean literal tag selects the corresponding callable
signature in either branch.
We reuse the tuple-union filtering used by equality checks and register
the containing tuple as a possible narrowing target for call predicates.
We keep uncertain alternatives and skip filtering for non-literal or
out-of-bounds indices. `TypeGuard` checks retain their existing
behavior.
For truthiness checks and calls, we skip the additional tuple narrowing
if the expression contains an assignment (`:=`). Evaluating the
expression can replace the tuple after it has been read. This
conservatively includes harmless assignments to other names; a TODO test
records that limitation.
Closes https://github.com/astral-sh/ty/issues/4674.
---------
Co-authored-by: Charlie Marsh <charlie.r.marsh@gmail.com>
## Summary
When a module imports a large namespace and then defines a small
`__all__`, we currently include every imported name when another module
uses a wildcard import. Packages with many such modules can create
hundreds of thousands of unnecessary definitions and reachability
constraints.
This PR instead recognizes an unchanged literal `__all__` before
semantic indexing and limits the possible exports to its names. Other
uses of `__all__`, including mutations and later wildcard imports, fall
back to the existing analysis. So this fixes narrow (but non-contrived)
cases.
For the `pulumi-cloudflare` reproduction in
[astral-sh/ty#4606](https://github.com/astral-sh/ty/issues/4606), peak
memory goes from 5,649-6,344 MiB to 453-459 MiB.
This also changes some diagnostics by avoiding cycles during import
resolution. Previously, determining the contents of `__all__` could
require type inference that depended on imports we were still resolving,
leaving some types as `Divergent`. Reading the literal directly breaks
those cycles, so we resolve more types successfully. This removes
spurious warnings and enables checks that were previously skipped
because the relevant types were unresolved.