## Summary
This PR changes `ruff_python_ast::Name` from a 24-byte `CompactString`
to a 16-byte immutable `CharStr` from
[`astral-sh/char_str`](https://github.com/astral-sh/char_str).
`char_str` is an Astral-maintained fork of
[`lean_string`](https://github.com/ryota2357/lean_string) that provides
compact owned strings with mutable and immutable variants: growable
`CharString` for construction and immutable, exactly allocated `CharStr`
for storage. (It also provides Salsa and get-size features.)
Names up to 16 UTF-8 bytes remain inline, static names remain
allocation-free, and longer names use an exactly sized,
reference-counted allocation with no unused capacity.
Removing eight bytes from every `Name` also shrinks common AST nodes.
For example, `Identifier` and `ExprName` drop from 40 to 32 bytes, while
`ExprAttribute`, `StmtFunctionDef`, `StmtClassDef`, and several other
variants each drop by eight bytes.
The parser interns names that exceed the inline capacity, so repeated
longer names share one backing allocation within a parser invocation.
## Why 16 bytes?
Across the full `mypy_primer` corpus, there 20.8 million name
occurrences:
| UTF-8 length | Occurrences | Share | Unique name/file pairs | Share |
|---|---:|---:|---:|---:|
| 1–16 bytes | 18,955,757 | 91.02% | 3,192,424 | 79.42% |
| 17–24 bytes | 1,348,894 | 6.48% | 529,826 | 13.18% |
| 25+ bytes | 522,376 | 2.51% | 297,575 | 7.40% |
| Total | 20,827,027 | 100% | 4,019,825 | 100% |
The 16-byte cutoff keeps 91% of occurrences inline while saving eight
bytes for every occurrence. The 1–16-byte bucket alone removes 151.6 MB
of handle storage across the corpus.
The tradeoff is the 17–24-byte bucket: `CompactString` stored those
names inline, while `CharStr` allocates once per unique name within a
file. But the interning amortizes that cost when a name repeats. Names
longer than 24 bytes already allocated on `main`; for those names, this
change both reduces the handle size and deduplicates repeated backing
allocations.
## Memory impact
Accounting for the observed per-file multiplicities and structural
sharing, modeled `Name` storage across the corpus changes as follows:
| | `main` | This PR | Difference |
|---|---:|---:|---:|
| `Name` handles and backing storage | 526.18 MB | 366.27 MB | -159.91
MB (-30.4%) |
The current [instrumented memory
report](https://github.com/astral-sh/ruff/pull/26594#issuecomment-4906725175)
shows a consistent reduction in total retained memory:
| Project | `main` | This PR | Difference |
|---|---:|---:|---:|
| flake8 | 29.21 MB | 28.71 MB | -1.69% |
| trio | 71.01 MB | 70.01 MB | -1.41% |
| sphinx | 168.81 MB | 166.82 MB | -1.18% |
| prefect | 454.75 MB | 449.06 MB | -1.25% |
Within those totals, `parsed_module` decreases by roughly 2.1% and
`semantic_index` by 2.4–2.7%. Several name-heavy collections decrease by
considerably more.
## Summary
Publish Ruff's production Rust crates to crates.io as part of the
existing release workflow. This follows uv's setup: workspace
dependencies carry registry-compatible version pins, internal crates
receive generated READMEs and an explicit versioning policy, CI dry-runs
the full workspace publish, and release jobs publish missing crate
versions idempotently through crates.io trusted publishing.
The publishing set contains 32 packages: Ruff's CLI, WASM package,
server, linter, formatter, parser, and the internal Ruff component
crates required by them. It also includes `ty_module_resolver`,
`ty_site_packages`, and `ty_static`, because those three crates are part
of Ruff's normal dependency closure.
The publishing set deliberately excludes:
- Test-only crates: `mdtest`, `ruff_mdtest`,
`ruff_python_ast_integration_tests`, and
`ruff_python_trivia_integration_tests`.
- Benchmark and development crates: `ruff_benchmark` and `ruff_dev`.
- All remaining `ty_*` crates.
These packages retain `publish = false`, so bootstrap, version bumping,
README generation, CI dry-runs, and release publishing skip them through
Cargo metadata instead of a separate hard-coded allowlist.
The crates.io job is intentionally independent of release announcement,
so a crates.io outage cannot block the existing binary, PyPI, npm, or
GitHub release outputs. A bootstrap script reserves new crate names,
configures the release workflow as each crate's trusted publisher,
enables trusted-publishing-only releases, and records configured crates
for idempotent reruns.
The full workspace publish dry-run packages and verifies all 32 selected
crates.
Closes https://github.com/astral-sh/ruff/issues/43.
## Summary
This PR modifies our `Cargo.toml` files to use workspace dependencies
for _all_ dependencies, rather than the status quo of sporadically
trying to use workspace dependencies for those dependencies that are
used across multiple crates. I find the current situation more confusing
and harder to manage, since we have a mix of workspace and crate-local
dependencies, whereas this setup consistently uses the same approach for
all dependencies.
Update to [Rust
1.74](https://blog.rust-lang.org/2023/11/16/Rust-1.74.0.html) and use
the new clippy lints table.
The update itself introduced a new clippy lint about superfluous hashes
in raw strings, which got removed.
I moved our lint config from `rustflags` to the newly stabilized
[workspace.lints](https://doc.rust-lang.org/stable/cargo/reference/workspaces.html#the-lints-table).
One consequence is that we have to `unsafe_code = "warn"` instead of
"forbid" because the latter now actually bans unsafe code:
```
error[E0453]: allow(unsafe_code) incompatible with previous forbid
--> crates/ruff_source_file/src/newlines.rs:62:17
|
62 | #[allow(unsafe_code)]
| ^^^^^^^^^^^ overruled by previous forbid
|
= note: `forbid` lint level was set on command line
```
---------
Co-authored-by: Charlie Marsh <charlie.r.marsh@gmail.com>
## Summary
Error if `tab-size` is set to zero (it is used as a divisor). Closes
#6423.
Also fixes a typo.
## Test Plan
Running ruff with a config
```toml
[tool.ruff]
tab-size = 0
```
returns an error message to the user saying that `tab-size` must be
greater than zero.
## Summary
This changes the caching design from one cache file per source file, to
one cache file per package. This greatly reduces the amount of cache
files that are opened and written, while maintaining roughly the same
(combined) size as bincode is very compact.
Below are some very much not scientific performance tests. It uses
projects/sources to check:
* small.py: single, 31 bytes Python file with 2 errors.
* test.py: single, 43k Python file with 8 errors.
* fastapi: FastAPI repo, 1134 files checked, 0 errors.
Source | Before # files | After # files | Before size | After size
-------|-------|-------|-------|-------
small.py | 1 | 1 | 20 K | 20 K
test.py | 1 | 1 | 60 K | 60 K
fastapi | 1134 | 518 | 4.5 M | 2.3 M
One question that might come up is why fastapi still has 518 cache files
and not 1? That is because this is using the existing package
resolution, which sees examples, docs, etc. as separate from the "main"
source code (in the fastapi directory in the repo). In this future it
might be worth consider switching to a one cache file per repo strategy.
This new design is not perfect and does have a number of known issues.
First, like the old design it doesn't remove the cache for a source file
that has been (re)moved until `ruff clean` is called.
Second, this currently uses a large mutex around the mutation of the
package cache (e.g. inserting result). This could be (or become) a
bottleneck. It's future work to test and improve this (if needed).
Third, currently the packages and opened and stored in a sequential
loop, this could be done parallel. This is also future work.
## Test Plan
Run `ruff check` (with caching enabled) twice on any Python source code
and it should produce the same results.
## Summary
This PR runs `rustfmt` with a few nightly options as a one-time fix to
catch some malformatted comments. I ended up just running with:
```toml
condense_wildcard_suffixes = true
edition = "2021"
max_width = 100
normalize_comments = true
normalize_doc_attributes = true
reorder_impl_items = true
unstable_features = true
use_field_init_shorthand = true
```
Since these all seem like reasonable things to fix, so may as well while
I'm here.
This PR introduces a new `CacheKey` trait for types that can be used as a cache key.
I'm not entirely sure if this is worth the "overhead", but I was surprised to find `HashableHashSet` and got scared when I looked at the time complexity of the `hash` function. These implementations must be extremely slow in hashed collections.
I then searched for usages and quickly realized that only the cache uses these `Hash` implementations, where performance is less sensitive.
This PR introduces a new `CacheKey` trait to communicate the difference between a hash and computing a key for the cache. The new trait can be implemented for types that don't implement `Hash` for performance reasons, and we can define additional constraints on the implementation: For example, we'll want to enforce portability when we add remote caching support. Using a different trait further allows us not to implement it for types without stable identities (e.g. pointers) or use other implementations than the standard hash function.