mirror of
https://github.com/astral-sh/ruff.git
synced 2026-11-21 12:16:15 +08:00
## Summary This PR changes `ruff_python_ast::Name` from a 24-byte `CompactString` to a 16-byte immutable `CharStr` from [`astral-sh/char_str`](https://github.com/astral-sh/char_str). `char_str` is an Astral-maintained fork of [`lean_string`](https://github.com/ryota2357/lean_string) that provides compact owned strings with mutable and immutable variants: growable `CharString` for construction and immutable, exactly allocated `CharStr` for storage. (It also provides Salsa and get-size features.) Names up to 16 UTF-8 bytes remain inline, static names remain allocation-free, and longer names use an exactly sized, reference-counted allocation with no unused capacity. Removing eight bytes from every `Name` also shrinks common AST nodes. For example, `Identifier` and `ExprName` drop from 40 to 32 bytes, while `ExprAttribute`, `StmtFunctionDef`, `StmtClassDef`, and several other variants each drop by eight bytes. The parser interns names that exceed the inline capacity, so repeated longer names share one backing allocation within a parser invocation. ## Why 16 bytes? Across the full `mypy_primer` corpus, there 20.8 million name occurrences: | UTF-8 length | Occurrences | Share | Unique name/file pairs | Share | |---|---:|---:|---:|---:| | 1–16 bytes | 18,955,757 | 91.02% | 3,192,424 | 79.42% | | 17–24 bytes | 1,348,894 | 6.48% | 529,826 | 13.18% | | 25+ bytes | 522,376 | 2.51% | 297,575 | 7.40% | | Total | 20,827,027 | 100% | 4,019,825 | 100% | The 16-byte cutoff keeps 91% of occurrences inline while saving eight bytes for every occurrence. The 1–16-byte bucket alone removes 151.6 MB of handle storage across the corpus. The tradeoff is the 17–24-byte bucket: `CompactString` stored those names inline, while `CharStr` allocates once per unique name within a file. But the interning amortizes that cost when a name repeats. Names longer than 24 bytes already allocated on `main`; for those names, this change both reduces the handle size and deduplicates repeated backing allocations. ## Memory impact Accounting for the observed per-file multiplicities and structural sharing, modeled `Name` storage across the corpus changes as follows: | | `main` | This PR | Difference | |---|---:|---:|---:| | `Name` handles and backing storage | 526.18 MB | 366.27 MB | -159.91 MB (-30.4%) | The current [instrumented memory report](https://github.com/astral-sh/ruff/pull/26594#issuecomment-4906725175) shows a consistent reduction in total retained memory: | Project | `main` | This PR | Difference | |---|---:|---:|---:| | flake8 | 29.21 MB | 28.71 MB | -1.69% | | trio | 71.01 MB | 70.01 MB | -1.41% | | sphinx | 168.81 MB | 166.82 MB | -1.18% | | prefect | 454.75 MB | 449.06 MB | -1.25% | Within those totals, `parsed_module` decreases by roughly 2.1% and `semantic_index` by 2.4–2.7%. Several name-heavy collections decrease by considerably more.