This PR replaces #25027. It adds an opt-in check for imports of
installed distributions that the current project does not declare
directly in `pyproject.toml`.
## Overview
The check uses ty's existing uv workspace integration (`TY_UV=1`) to
call `uv workspace metadata --frozen --active`. uv now provides module
ownership without synchronizing the environment, so ty can read the
metadata without installing packages or updating `uv.lock`.
The metadata provides:
1. Each workspace project's runtime and optional dependencies.
2. Dependencies declared in dependency groups.
3. Importable modules and their owning distributions, including source
paths for editable installs.
During import inference, ty maps each resolved import to its
distribution and compares it against the importing project's direct
declarations. If the declaration is missing, ty emits
`missing-direct-dependency`, naming the distribution to add. Each
distribution is reported once per file, at its first unsuppressed
import.
The rule is disabled by default and requires an existing, synchronized
environment. Unavailable or incompatible dependency metadata disables
the check; rejected dependency metadata produces a debug message.
## Nuances
- Editable installs may lack entries in uv's module-owner map. ty uses
their source paths to attribute resolved imports.
- Files identified as package code require runtime or optional
dependencies. Non-package files, such as tests and scripts, can also use
dependency groups, but those groups' transitive dependencies still need
direct declarations.
- For `from package import child`, a resolved child module is checked
against its own distribution rather than the parent's. Imports with
unknown or ambiguous ownership are skipped.
- Ownership metadata is used only when the explicitly selected Python
environment matches uv's environment.
- The current integration covers directory checks. Explicit file
arguments, `--config-file`, and standalone PEP 723 scripts do not use
the workspace's dependency declarations.
The schema and rule documentation are regenerated. Most of the
rule-documentation diff consists of updated source-line links.
## Summary
`PythonVersionWithSource` contains from where ty derived the
`PythonVersion`. If the Python version comes from the project's
`pyproject.toml`, then the source includes the configuration file name
and the text range of the `requires-python` or `python-version`
key-value pair. Any change to that `pyproject.toml` would result in a
new and different `PythonVersionWithSource`, which, in turn, results in
a distinct interned `Program`. This results in invalidating every single
type inference query (more accurately, we now get a new instance of each
type inference query, but we loose all cached results).
Storing `PythonVersionWithSource` also has the downside that it prevents
scripts that otherwise have identical `Program` settings to share the
same `Program`, only because their `python-version` comes from different
script metadata blocks.
This PR moves `PythonVersionWithSource` out of `Program`. `Program`
should only contain information that changes type inference.
`PythonVersionWithSource` does not. `Db` now has a new
`python_version_with_source` method that, given a file, returns the
source for it (A script can return a different source than a project
file).
## Test Plan
Testing: Semantic, project, core, server, Markdown, and corpus tests
pass; repository hooks pass.
## Summary
This is the last PR that adds support for type checking multiple
programs with different Python Version, search paths, and platform using
a single Salsa DB.
This PR changes `Program` from a singleton input to an interned struct.
`ProgramFile` changes from `(File, ResolverEnvironment)` to `(File,
Program)`.
Changing `Program` from an input to an interned means that it now has a
`'db` lifetime. This requires changing how we construct and store
`Program`. That's why some of the db initialization code changes.
I doubt that it will be noticeable. But this should speed up mdtests
that contain tests with different Python versions (e.g. Py312, Py311,
Py312, Py311), because we now reuse parsed modules, and even type
inference results because the inference is now keyed by Program (instead
of updating an existing Program)
## Memory regression
I had codex look into the CodSpeed reported memory regression. The
reason is that `Program` now being an interned and being initialized
later changes the assigned `File` ids, which in turn changes in which
order we iterate over the project files. And this change the max peak
memory usage. It can also change the usage reported on the memory usage
report (we enter fixpoint cycles from different heads, etc).
The changes in the memory report are mainly from the new
`ProgramFile::python_version` (needed to mitigate a larger regression,
but we can reconsider), and that `Project` now stores `ProgramSettings`.
## Summary
This PR continues the refactor to make type inference and module
resolution support scripts that may have different settings than the
main project.
This PR introduces:
* `ResolverEnvironment(SearchPaths, PythonVersion)`: Uniquely identifies
the settings used to resolve a module
* `ResolverFile(File, ResolverEnvironment)`: A file with its
`ResolverEnvironment`. Used for desperate module resolution and
`module_to_file` (and all other places where the operation doesn't
depend on type-inference specific settings).
This PR also `ProgramFile` and updates the `Program` alias to point to
`ResolverEnvironment`. These changes are in preparation to reduce the
diff of the third PR (we don't want to rewrite all `python_file` call
sites to `resolver_file` to then rename them to `program_file`).
## Memory regression
This PR changes `Program` from `PythonVersion` (u16) to
`ResolverEnvironment` (u64). This increases the size of interned structs
and interned query arguments that contain `Program`. Memory usage is
still lower than before we made db `Program` aware (the base PR reduced
memory usage by about 1%).
## Test Plan
Testing: Ran workspace checks, resolver/core/semantic/IDE tests, and
repository hooks, including cross-version inference coverage.
## Summary
This is the first PR out of 3. It extends `ty_python_semantic` and lower
crates to support checking files that use different Python versions
while using a single Salsa db.
The motivation for supporting different Python versions is that scrolls
need to be checked independently. Initially, I planned to implement
scroll support in a similar way to how we support multiple workspace
folders in VS Code by using multiple dbs, but it turned out, that
they're different enough. The main difference is that scrolls are
checked as part of their enclosing project, and the diagnostics from
scrolls are merged with the diagnostics from the project. If we used
separate scroll dbs, then the scroll diagnostics would use `File`s
pointing into the scroll db and all other diagnostics would use `File`s
pointing into the main db. There's nothing that would allow us to tell
to which db any such `File`s belonged, and `File`s from different dbs
can even share the same value. There's also no lifetime that binds them
to one particular db because they're inputs. We could have decided to
rewrite the `Diagnostic`s for scrolls and convert the scroll db `File`s
to project db `File`s, but that felt very brittle. The other
disadvantage was that it became very difficult to tell which query is
cached because you now also need to reason about whether the current
query is cached for a specific argument and db (and not just the
argument).
This is why I decided to explore whether we could support checking files
with different Python version, search paths, and platforms, and this PR
is the first step towards this. It adds support for checking files with
different Python versions. Specifically, this PR removes the
`db.python_version` from `ruff_db::Db` and lifts it up to
`ty_project::Db`, proving that no semantic crate depends on it.
The first step is to make `parsed_module` Python version aware. That is,
instead of calling `db.python_version`, parametrize the query by file
and Python version. For this, I introduced a new `PythonFile` which is a
`(File, PythonVersion)` tuple. We now attach this `PythonFile` to
`Scope` (instead of `File`), because `::node` accessors need it to
retrieve their node. That in turn required `semantic_index` to take a
`PythonFile` instead of `File` as well. This was a mostly smooth ride.
Supporting multiple Python versions in the module resolver required
adding a `PythonVersion` argument to most queries. The next PR will
replace the `PythonVersion` with a new `ResolverSettings` (or similar
name) interned struct that wraps both Python version and search paths.
This was also mostly straightforward.
The hard part was to remove all `db.python_version` calls from type
inference and type operations. Today, we call `db.python_version` where
we need it, but we can't do this anymore. Instead, we have to pass the
Python version down to each function that needs it.
Now, this is not true for all functions. Functions that have access to a
scope (via `Definition`, `Expression` or similar) could retrieve the
Python version from there (because `Scope` stores a `PythonFile`). I
decided against that and, instead established the following policy:
* Explicitly pass the Python version to all semantic functions.
* The only exception to this are Salsa queries that have exactly one
argument other than `db`. For those, retrieving the Python version from
their `Scope` is allowed for performance reasons.
Ultimately what that means: The context should be initialized at the
query boundary and then not change, unless we cross into another Salsa
query (but assert that the python version does not change).
There are two motivations for this:
* I think it's easier to reason about what Python version a type
operation uses if there's a consistent way of retrieving it (always as
argument)
* At query boundaries, we can assert that the explicitly passed Python
version and the `Scope`'s Python version are identical, before calling
into the query (debug assert).
However, supporting different Python versions now requires that we
parametrize all type operations with the Python version under which the
operation should be performed (in the future, this will be Python
version, search paths, and platforms).
Now, passing `db` and `PythonVersion` as separate arguments not only
results in a huge diff (add another 7k changes on top of the current
diff), is cumbersome, it also is slower. That's why I this PR introduces
a new `SemanticContext` that wraps a `db` and a `PythonVersion` (Happy
to discuss names). The variable is consistently referred to as `ctx` to
reduce the diff (using `context` makes it more likely that lines break).
## Next steps
* Change `SemanticContext` to store a `db` and `SearchPathSettings` to
support different search paths
* After that, change `SemanticContext` to store a `db` and a
`ProgramFile` (which is a `(File, Program)` tuple.
* Use the new infrastructure for scroll support
## Performance
There's currently a 2-3% walltime regression. Codex reports a lower
local regression.
There are some options that we can explore if we think this regression
is too big. However, it would make the model a little more muddy. The
most promising is probably to have two versions of very hot queries.
E.g. `is_redundant_with` could have a `is_redundant_with_project` and
`is_redundant_with_script` query. `is_redundant_with_project` would use
`db.python_version` internally (yes, we'd have to add that back). I
don't like that. Not only does it reintroduce `db.python_version` and
make it very easy that we use it somewhere where we should not.
## Review
I, honestly, tried to split this PR into smaller, reviewable commits.
But I don't think I succeeded. But I also don't think it makes sense to
review this commit line by line. I'm sure there are places where we
still pass `db` where we should pass a `ctx`. I'm sure there are some
call sites where we should retrieve the `python_version` in a different
way. But I'm not sure it's worth finding all these places in a review.
Instead, I'd focus on the overall change. What's your reaction on
`SemanticContext`? That we now need to pass it everywhere? What about my
policy that we use a single `SemanticContext` within a query? Should we
relax this constraint, so that `SemanticIndex` must be passed in fewer
places, if the `PythonVersion` can be retrieved at the relevant call
site (because it has `Scope`)? The downside of that is that it becomes
more difficult to assert that we don't mix programs in a single
inference, but is this something we should really be concerned about? Do
you think the performance regression is to big for this feature?
## Notes
* Make associated tracked methods that don't take `python_version`
normal methods with an inner query instead. The method should take a
`python_version`, and we assert (before calling the query), that the
target's (self's) `python_version` matches the passed python version
## Summary
Each ty mdtest currently constructs a new single-threaded Rayon pool to
prevent fixture-level work from competing with concurrently running
tests. With 339 fixtures, this repeatedly creates and tears down worker
threads during a full `cargo test` run.
This change caches one single-threaded pool per test-harness worker.
Fixture execution remains limited to one Rayon thread, while harness
workers can reuse their pool across fixtures. In paired benchmarks at
default concurrency, this reduced median mdtest runtime by 7.6%; the
improvement was 8.4% at 12 harness threads and 16.2% in single-threaded
harness execution.
Summary
--
This is a first step toward adding mdtests for Ruff. I actually wrote
the code
in the opposite order, first copy-pasting `ty_test` to a `ruff_test`
crate, and then
factoring out the shared code, but I figured it would be easier to
review in
this order. I also opened a stacked PR with the `ruff_test` changes
(#24617)
to show that the API works well for that too.
The main change here is moving several of the modules from `ty_test` to
a new
`mdtest` crate:
- `assertion`
- `diagnostic`
- `matcher`
- `parser`
Beyond moving these files to the new crate, I made `Matcher` functions
take a
`&dyn Db` to support passing a different concrete type from `ruff_test`,
and I
also made the parser generic over an `MdtestConfig` trait to allow Ruff
to use a
separate config struct. I also introduced new `TestConfig` and `TestDb`
types to allow
testing the `matcher` and `parser` within the `mdtest` crate without
depending
on either the real ty `Db` or `ty_test` config type.
The lib.rs file from `ty_test` was essentially split in half, with the
shared
code moved to the `mdtest` crate and the ty-specific parts kept in
`ty_test`.
Test Plan
--
All existing mdtests and the unit tests from `ty_test` should still
pass, and
the stacked branch with the `ruff_test` crate tests the split API
## Summary
This is in preparation for https://github.com/astral-sh/ruff/pull/24097
I want to integrate fixes into mdtests, but `ty_test` doesn't depend on
`ty_project`, and adding this new dependency would create more cyclic
dependencies between crates (not targets, just crates), which r-a
sometimes struggles with.
The solution in this PR is to move `fixes` to `ty_python_semantic`. This
is mostly a trivial change, except that it requires adding a
`check_file` method to `db.`
## Test Plan
`cargo test`
## Summary
Derived from #17371Fixesastral-sh/ty#256
Fixes https://github.com/astral-sh/ty/issues/1415
Fixes https://github.com/astral-sh/ty/issues/1433
Fixes https://github.com/astral-sh/ty/issues/1524
Properly handles any kind of recursive inference and prevents panics.
---
Let me explain techniques for converging fixed-point iterations during
recursive type inference.
There are two types of type inference that naively don't converge
(causing salsa to panic): divergent type inference and oscillating type
inference.
### Divergent type inference
Divergent type inference occurs when eagerly expanding a recursive type.
A typical example is this:
```python
class C:
def f(self, other: "C"):
self.x = (other.x, 1)
reveal_type(C().x) # revealed: Unknown | tuple[Unknown | tuple[Unknown | tuple[..., Literal[1]], Literal[1]], Literal[1]]
```
To solve this problem, we have already introduced `Divergent` types
(https://github.com/astral-sh/ruff/pull/20312). `Divergent` types are
treated as a kind of dynamic type [^1].
```python
Unknown | tuple[Unknown | tuple[Unknown | tuple[..., Literal[1]], Literal[1]], Literal[1]]
=> Unknown | tuple[Divergent, Literal[1]]
```
When a query function that returns a type enters a cycle, it sets
`Divergent` as the cycle initial value (instead of `Never`). Then, in
the cycle recovery function, it reduces the nesting of types containing
`Divergent` to converge.
```python
0th: Divergent
1st: Unknown | tuple[Divergent, Literal[1]]
2nd: Unknown | tuple[Unknown | tuple[Divergent, Literal[1]], Literal[1]]
=> Unknown | tuple[Divergent, Literal[1]]
```
Each cycle recovery function for each query should operate only on the
`Divergent` type originating from that query.
For this reason, while `Divergent` appears the same as `Any` to the
user, it internally carries some information: the location where the
cycle occurred. Previously, we roughly identified this by having the
scope where the cycle occurred, but with the update to salsa, functions
that create cycle initial values can now receive a `salsa::Id`
(https://github.com/salsa-rs/salsa/pull/1012). This is an opaque ID that
uniquely identifies the cycle head (the query that is the starting point
for the fixed-point iteration). `Divergent` now has this `salsa::Id`.
### Oscillating type inference
Now, another thing to consider is oscillating type inference.
Oscillating type inference arises from the fact that monotonicity is
broken. Monotonicity here means that for a query function, if it enters
a cycle, the calculation must start from a "bottom value" and progress
towards the final result with each cycle. Monotonicity breaks down in
type systems that have features like overloading and overriding.
```python
class Base:
def flip(self) -> "Sub":
return Sub()
class Sub(Base):
def flip(self) -> "Base":
return Base()
class C:
def __init__(self, x: Sub):
self.x = x
def replace_with(self, other: "C"):
self.x = other.x.flip()
reveal_type(C(Sub()).x)
```
Naive fixed-point iteration results in `Divergent -> Sub -> Base -> Sub
-> ...`, which oscillates forever without diverging or converging. To
address this, the salsa API has been modified so that the cycle recovery
function receives the value of the previous cycle
(https://github.com/salsa-rs/salsa/pull/1012).
The cycle recovery function returns the union type of the current cycle
and the previous cycle. In the above example, the result type for each
cycle is `Divergent -> Sub -> Base (= Sub | Base) -> Base`, which
converges.
The final result of oscillating type inference does not contain
`Divergent` because `Divergent` that appears in a union type can be
removed, as is clear from the expansion. This simplification is
performed at the same time as nesting reduction.
```
T | Divergent = T | (T | (T | ...)) = T
```
[^1]: In theory, it may be possible to strictly treat types containing
`Divergent` types as recursive types, but we probably shouldn't go that
deep yet. (AFAIK, there are no PEPs that specify how to handle
implicitly recursive types that aren't named by type aliases)
## Performance analysis
A happy side effect of this PR is that we've observed widespread
performance improvements!
This is likely due to the removal of the `ITERATIONS_BEFORE_FALLBACK`
and max-specialization depth trick
(https://github.com/astral-sh/ty/issues/1433,
https://github.com/astral-sh/ty/issues/1415), which means we reach a
fixed point much sooner.
## Ecosystem analysis
The changes look good overall.
You may notice changes in the converged values for recursive types,
this is because the way recursive types are normalized has been changed.
Previously, types containing `Divergent` types were normalized by
replacing them with the `Divergent` type itself, but in this PR, types
with a nesting level of 2 or more that contain `Divergent` types are
normalized by replacing them with a type with a nesting level of 1. This
means that information about the non-divergent parts of recursive types
is no longer lost.
```python
# previous
tuple[tuple[Divergent, int], int] => Divergent
# now
tuple[tuple[Divergent, int], int] => tuple[Divergent, int]
```
The false positive error introduced in this PR occurs in class
definitions with self-referential base classes, such as the one below.
```python
from typing_extensions import Generic, TypeVar
T = TypeVar("T")
U = TypeVar("U")
class Base2(Generic[T, U]): ...
# TODO: no error
# error: [unsupported-base] "Unsupported class base with type `<class 'Base2[Sub2, U@Sub2]'> | <class 'Base2[Sub2[Unknown], U@Sub2]'>`"
class Sub2(Base2["Sub2", U]): ...
```
This is due to the lack of support for unions of MROs, or because cyclic
legacy generic types are not inferred as generic types early in the
query cycle.
## Test Plan
All samples listed in astral-sh/ty#256 are tested and passed without any
panic!
## Acknowledgments
Thanks to @MichaReiser for working on bug fixes and improvements to
salsa for this PR. @carljm also contributed early on to the discussion
of the query convergence mechanism proposed in this PR.
---------
Co-authored-by: Carl Meyer <carl@astral.sh>
## Summary
This PR updates the server to keep track of open files both system and
virtual files.
This is done by updating the project by adding the file in the open file
set in `didOpen` notification and removing it in `didClose`
notification.
This does mean that for workspace diagnostics, ty will only check open
files because the behavior of different diagnostic builder is to first
check `is_file_open` and only add diagnostics for open files. So, this
required updating the `is_file_open` model to be `should_check_file`
model which validates whether the file needs to be checked based on the
`CheckMode`. If the check mode is open files only then it will check
whether the file is open. If it's all files then it'll return `true` by
default.
Closes: astral-sh/ty#619
## Test Plan
### Before
There are two files in the project: `__init__.py` and `diagnostics.py`.
In the video, I'm demonstrating the old behavior where making changes to
the (open) `diagnostics.py` file results in re-parsing the file:
https://github.com/user-attachments/assets/c2ac0ecd-9c77-42af-a924-c3744b146045
### After
Same setup as above.
In the video, I'm demonstrating the new behavior where making changes to
the (open) `diagnostics.py` file doesn't result in re-parting the file:
https://github.com/user-attachments/assets/7b82fe92-f330-44c7-b527-c841c4545f8f