Files
ruff/crates/ruff_python_parser
Micha Reiser 0b4359778d Parse hex escapes one ASCII byte at a time (#30374)
## Summary

On main, parsing `b"\x41"` calls `parse_unicode_literal(2)`. The x86-64
profiling build uses one function for widths 2, 4, and 8, so the loop
bound is a runtime argument. For each digit (`4`, then `1`), the loop
saves its source position and calls `next_char()` to decode and consume
a UTF-8 character before checking whether it is a hex digit.

This PR calls `parse_unicode_literal::<2>()` for `\x`, which lets the
compiler specialize the loop for two digits. This is the largest
measured improvement. The new loop also peeks at the next byte and
advances by one when it sees a valid ASCII hex digit. Only if that check
fails does it save the source position and call `next_char()`. For an
invalid escape such as `"\x4é"`, both versions consume `é` and report
its two-byte range.

Callgrind instruction counts for a whole uncached `ruff check` on
[JAX](https://github.com/jax-ml/jax/tree/55029b8f7e65adb79764103871ec0357fa440227/jax/_src/internal_test_util/export_back_compat_test_data).
On the compatibility fixtures, changing only the width to `const N`
saves 8.36 million of the 13.19 million instructions saved by the full
change.

| Workload | Main | `const N` only | `const N` + deferred error position
| Full change | Total reduction |
| --- | ---: | ---: | ---: | ---: | ---: |
| JAX export compatibility fixtures | 131,434,592 | 123,077,887 |
120,580,490 | 118,244,820 | 10.04% |
| JAX export implementation, compatibility fixtures, and export tests |
226,412,243 | 218,062,626 | 215,565,059 | 213,219,152 | 5.83% |

## Test Plan

The parser tests passed, and diagnostics matched main on valid,
malformed, and non-ASCII escapes.
2026-11-12 17:26:00 +01:00
..
2026-11-05 13:22:15 -05:00
2026-11-05 13:22:15 -05:00

Ruff Python Parser

Ruff's Python parser is a hand-written recursive descent parser which can parse Python source code into an Abstract Syntax Tree (AST). It also utilizes the Pratt parsing technique to parse expressions with different precedence.

Try out the parser in the playground.

Contributing

Refer to the contributing guidelines to get started and GitHub issues with the parser label for issues that need help.

Versioning

This crate is an internal component of Ruff. The Rust API exposed here is unstable and will have frequent breaking changes.

This version (0.0.21) is a component of Ruff 0.17.4. The source can be found here.

See Ruff's crate versioning policy for details on versioning.