Commit Graph
5 Commits
Author SHA1 Message Date
Alex Waygood 2e34f5ba22 Use a shared time budget for fuzzing (#29280)
Our `py-fuzzer` script currently limits each failing seed to have a
maximum of 60 seconds for minimization, even when the CI job has time
remaining. For example, in [this fuzz job on
#29025](https://github.com/astral-sh/ruff/actions/runs/37632264340/job/112829497328),
minimization timed out for seed 57 even though there would have been
another 15 minutes before GitHub cancelled the CI job. Running the
script locally demonstrated that pysource_minimize was [easily
able](https://github.com/astral-sh/ruff/pull/29025#issuecomment-6064567417)
to minimize the panic if only given slightly more time to do so.

This PR replaces the per-seed limit in CI with a global, shared budget
for checking seeds and minimizing failures. Local runs have no overall
budget by default. CI, however, records the job start time in its first
step and passes it and the job timeout to the fuzzer, which subtracts
elapsed setup and build time and reserves an additional 90 seconds for
other job overhead and reporting. Individual calls to ty or Ruff still
have a five-second timeout, allowing us to minimize hangs or
pathological performance issues.

By default, the fuzzer reports failing seeds as they are found and
prints completed minimizations. When the deadline is reached, it stops
starting new seed and minimizer checks, reports the original source for
unfinished minimizations, and identifies unchecked seeds. Checks already
in progress can finish after the deadline. The logs and, when requested,
the GitHub issue body include a command to rerun failing and unchecked
seeds; the script exits with status 2 if any seeds were left unchecked.

Here are some mockups of what bot-opened issues could look like, using
fictional data:

<details>
<summary>Issue with minimized, unminimized, and unchecked
seeds</summary>

Run listed here: https://github.com/astral-sh/ty/actions/runs/1234567890

Each snippet caused ty to exit with a status other than 0, 1, or 2, or
time out after five seconds during fuzzing.

Build ty from commit `0123456789abcdef0123456789abcdef01234567` with
`cargo build --locked --bin=ty`. Save a snippet to `repro.py`, then run:

```
target/debug/ty check repro.py --python-version=3.10 --python-platform=linux
```


### Seed: 78

```python
class Example:
    pass
```

<details>
<summary>Seed: 93 (original source; minimization did not
complete)</summary>

```python
class Original:
    value = [1, 2, 3]
    def method(self):
        return self.value
```

</details>

### Unchecked seeds

The time budget expired before 2 seeds could be checked.

- `123`
- `124`

To check all seeds that failed or were left unchecked, check out Ruff
commit `0123456789abcdef0123456789abcdef01234567` and run from the
repository root:

```shell
uv run --project=./python/py-fuzzer fuzz --bin=ty 78 93 123 124
```

</details>

<details>
<summary>Issue with unchecked seeds and no failures</summary>

Run listed here:
https://github.com/astral-sh/ruff/actions/runs/1234567890

Test executable: ruff at commit
`0123456789abcdef0123456789abcdef01234567`.


### Unchecked seeds

The time budget expired before 3 seeds could be checked.

- `10`
- `11`
- `12`

To check all seeds that failed or were left unchecked, check out Ruff
commit `0123456789abcdef0123456789abcdef01234567` and run from the
repository root:

```shell
uv run --project=./python/py-fuzzer fuzz --bin=ruff 10 11 12
```

</details>

## Rejected alternatives

1. Keep a per-minimization budget, but just increase it (by a lot?).

We could do this, and it fixes the theoretical problem that codex raised
where a couple of seeds that take a very long time to minimize stop the
script from even discovering bugs in later seeds (because it never even
gets to those later seeds).

I decided against this approach, however, because in practice I think it
will only be hangs that take a _long_ time to minimize. We've seen in
the past that a ty hang can cause the script as a whole to timeout in
CI, but whenever it's done so in the past, it's never caused the script
to fail to get onto non-hanging seeds. It's always been the case that
the script has completed all seeds except the problematic ones that were
causing it to hang. So I decided that per-minimization budgets were a
solution looking for a problem.

2. Get rid of all budgets altogether, go back to something more similar
to what the script was like prior to
https://github.com/astral-sh/ruff/pull/29045.

This is very attractive, because it avoids a lot of the new complexity
added in this PR to do with calculating budgets, interrupting
subprocesses that have already been started, etc.
   
I decided against this, however, because I do think the fact that the
script now minimizes hangs substantially increases the risk that the
script might finish without having finished minimizing all seeds. In the
event that this happens, I _do_ think it's quite useful (though not
essential) for the script to report which seeds it didn't get to, before
GitHub cancels the CI job altogether.
2026-10-14 16:03:48 +01:00
Alex Waygood c79d773317 Include minimized reproducers in daily fuzz issues (#29100) 2026-10-05 14:41:20 +01:00
Alex Waygood c424007645 Update usage instructions and lockfile for py-fuzzer script (#20940) 2025-10-17 15:57:17 +01:00
Alex Waygood f1b2e85339 py-fuzzer: recommend using uvx rather than uv run to run the fuzzer (#14645) 2024-11-27 22:19:52 +00:00
Alex WaygoodandDavid Peter e0f3eaf1dd Turn the fuzz-parser script into a properly packaged Python project (#14606)
## Summary

This PR gets rid of the `requirements.in` and `requirements.txt` files
in the `scripts/fuzz-parser` directory, and replaces them with
`pyproject.toml` and `uv.lock` files. The script is renamed from
`fuzz-parser` to `py-fuzzer` (since it can now also be used to fuzz
red-knot as well as the parser, following
https://github.com/astral-sh/ruff/pull/14566), and moved from the
`scripts/` directory to the `python/` directory, since it's now a
(uv)-pip-installable project in its own right.

I've been resisting this for a while, because conceptually this script
just doesn't feel "complicated" enough to me for it to be a full-blown
package. However, I think it's time to do this. Making it a proper
package has several advantages:
- It means we can run it from the project root using `uv run` without
having to activate a virtual environment and ensure that all required
dependencies are installed into that environment
- Using a `pyproject.toml` file means that we can express that the
project requires Python 3.12+ to run properly; this wasn't possible
before
- I've been running mypy on the project locally when I've been working
on it or reviewing other people's PRs; now I can put the mypy config for
the project in the `pyproject.toml` file

## Test Plan

I manually tested that all the commands detailed in
`python/py-fuzzer/README.md` work for me locally.

---------

Co-authored-by: David Peter <sharkdp@users.noreply.github.com>
2024-11-27 08:09:04 +00:00