## Summary
This PR changes `ruff_python_ast::Name` from a 24-byte `CompactString`
to a 16-byte immutable `CharStr` from
[`astral-sh/char_str`](https://github.com/astral-sh/char_str).
`char_str` is an Astral-maintained fork of
[`lean_string`](https://github.com/ryota2357/lean_string) that provides
compact owned strings with mutable and immutable variants: growable
`CharString` for construction and immutable, exactly allocated `CharStr`
for storage. (It also provides Salsa and get-size features.)
Names up to 16 UTF-8 bytes remain inline, static names remain
allocation-free, and longer names use an exactly sized,
reference-counted allocation with no unused capacity.
Removing eight bytes from every `Name` also shrinks common AST nodes.
For example, `Identifier` and `ExprName` drop from 40 to 32 bytes, while
`ExprAttribute`, `StmtFunctionDef`, `StmtClassDef`, and several other
variants each drop by eight bytes.
The parser interns names that exceed the inline capacity, so repeated
longer names share one backing allocation within a parser invocation.
## Why 16 bytes?
Across the full `mypy_primer` corpus, there 20.8 million name
occurrences:
| UTF-8 length | Occurrences | Share | Unique name/file pairs | Share |
|---|---:|---:|---:|---:|
| 1–16 bytes | 18,955,757 | 91.02% | 3,192,424 | 79.42% |
| 17–24 bytes | 1,348,894 | 6.48% | 529,826 | 13.18% |
| 25+ bytes | 522,376 | 2.51% | 297,575 | 7.40% |
| Total | 20,827,027 | 100% | 4,019,825 | 100% |
The 16-byte cutoff keeps 91% of occurrences inline while saving eight
bytes for every occurrence. The 1–16-byte bucket alone removes 151.6 MB
of handle storage across the corpus.
The tradeoff is the 17–24-byte bucket: `CompactString` stored those
names inline, while `CharStr` allocates once per unique name within a
file. But the interning amortizes that cost when a name repeats. Names
longer than 24 bytes already allocated on `main`; for those names, this
change both reduces the handle size and deduplicates repeated backing
allocations.
## Memory impact
Accounting for the observed per-file multiplicities and structural
sharing, modeled `Name` storage across the corpus changes as follows:
| | `main` | This PR | Difference |
|---|---:|---:|---:|
| `Name` handles and backing storage | 526.18 MB | 366.27 MB | -159.91
MB (-30.4%) |
The current [instrumented memory
report](https://github.com/astral-sh/ruff/pull/26594#issuecomment-4906725175)
shows a consistent reduction in total retained memory:
| Project | `main` | This PR | Difference |
|---|---:|---:|---:|
| flake8 | 29.21 MB | 28.71 MB | -1.69% |
| trio | 71.01 MB | 70.01 MB | -1.41% |
| sphinx | 168.81 MB | 166.82 MB | -1.18% |
| prefect | 454.75 MB | 449.06 MB | -1.25% |
Within those totals, `parsed_module` decreases by roughly 2.1% and
`semantic_index` by 2.4–2.7%. Several name-heavy collections decrease by
considerably more.
## Summary
Error if `tab-size` is set to zero (it is used as a divisor). Closes
#6423.
Also fixes a typo.
## Test Plan
Running ruff with a config
```toml
[tool.ruff]
tab-size = 0
```
returns an error message to the user saying that `tab-size` must be
greater than zero.
## Summary
This changes the caching design from one cache file per source file, to
one cache file per package. This greatly reduces the amount of cache
files that are opened and written, while maintaining roughly the same
(combined) size as bincode is very compact.
Below are some very much not scientific performance tests. It uses
projects/sources to check:
* small.py: single, 31 bytes Python file with 2 errors.
* test.py: single, 43k Python file with 8 errors.
* fastapi: FastAPI repo, 1134 files checked, 0 errors.
Source | Before # files | After # files | Before size | After size
-------|-------|-------|-------|-------
small.py | 1 | 1 | 20 K | 20 K
test.py | 1 | 1 | 60 K | 60 K
fastapi | 1134 | 518 | 4.5 M | 2.3 M
One question that might come up is why fastapi still has 518 cache files
and not 1? That is because this is using the existing package
resolution, which sees examples, docs, etc. as separate from the "main"
source code (in the fastapi directory in the repo). In this future it
might be worth consider switching to a one cache file per repo strategy.
This new design is not perfect and does have a number of known issues.
First, like the old design it doesn't remove the cache for a source file
that has been (re)moved until `ruff clean` is called.
Second, this currently uses a large mutex around the mutation of the
package cache (e.g. inserting result). This could be (or become) a
bottleneck. It's future work to test and improve this (if needed).
Third, currently the packages and opened and stored in a sequential
loop, this could be done parallel. This is also future work.
## Test Plan
Run `ruff check` (with caching enabled) twice on any Python source code
and it should produce the same results.
## Summary
This PR runs `rustfmt` with a few nightly options as a one-time fix to
catch some malformatted comments. I ended up just running with:
```toml
condense_wildcard_suffixes = true
edition = "2021"
max_width = 100
normalize_comments = true
normalize_doc_attributes = true
reorder_impl_items = true
unstable_features = true
use_field_init_shorthand = true
```
Since these all seem like reasonable things to fix, so may as well while
I'm here.
This PR introduces a new `CacheKey` trait for types that can be used as a cache key.
I'm not entirely sure if this is worth the "overhead", but I was surprised to find `HashableHashSet` and got scared when I looked at the time complexity of the `hash` function. These implementations must be extremely slow in hashed collections.
I then searched for usages and quickly realized that only the cache uses these `Hash` implementations, where performance is less sensitive.
This PR introduces a new `CacheKey` trait to communicate the difference between a hash and computing a key for the cache. The new trait can be implemented for types that don't implement `Hash` for performance reasons, and we can define additional constraints on the implementation: For example, we'll want to enforce portability when we add remote caching support. Using a different trait further allows us not to implement it for types without stable identities (e.g. pointers) or use other implementations than the standard hash function.