mirror of
https://github.com/tester-army/e2e.git
synced 2026-11-21 03:05:26 +08:00
131 lines
4.1 KiB
Plaintext
131 lines
4.1 KiB
Plaintext
---
|
|
title: Bug bashes
|
|
description: Run many explore sessions at the same time, then prove each bug with a failing test.
|
|
"og:image": "https://e2e.com/docs/images/og/bug-bash.png?v=eb59f8290c"
|
|
"twitter:image": "https://e2e.com/docs/images/og/bug-bash.png?v=eb59f8290c"
|
|
---
|
|
|
|
import { ShowMore } from './snippets/show-more.jsx';
|
|
import { Video } from './snippets/video.jsx';
|
|
|
|
A bug bash runs many `e2e explore` sessions at the same time, each on one
|
|
area of the app. A coding agent then writes a test for each finding and
|
|
reports it as a bug only when the test fails.
|
|
|
|
<Video>
|
|
<video src="/images/bug-bash.mp4" aria-label="A coding agent runs a bug bash in headed browsers." muted loop playsInline controls />
|
|
|
|
Each browser is one explore session.
|
|
</Video>
|
|
|
|
## Before you start
|
|
|
|
- Make sure the app can serve many explorers at the same time. Start the
|
|
app one time and set `reuseExisting: true` on the target's `app.command`,
|
|
or set port `0` in the app URL so each run starts its own copy. When `CI`
|
|
is set, e2e ignores `reuseExisting`. Use port `0`. See
|
|
[Starting your app](/web#starting-your-app).
|
|
- On mobile, declare one target for each simulator or emulator. Each
|
|
explorer needs its own device.
|
|
- Each explore session uses model calls. Each run ends with its tokens,
|
|
cost, and duration.
|
|
- On a deployed site that other people use, do not change data.
|
|
|
|
The agent does the rest of the setup, such as seeded accounts and sign-in,
|
|
from the skill.
|
|
|
|
## Install the skill
|
|
|
|
The coding agent runs the bug bash from the [`e2e` skill](/coding-agents),
|
|
which `e2e init` installs. To install it without `init`:
|
|
|
|
<CodeGroup>
|
|
```bash npm
|
|
npx skills add e2e-inc/e2e
|
|
```
|
|
|
|
```bash pnpm
|
|
pnpm dlx skills add e2e-inc/e2e
|
|
```
|
|
|
|
```bash bun
|
|
bunx skills add e2e-inc/e2e
|
|
```
|
|
</CodeGroup>
|
|
|
|
## Start a bug bash
|
|
|
|
Give a prompt to your coding agent:
|
|
|
|
```text
|
|
Bug bash the checkout and account settings on this branch.
|
|
```
|
|
|
|
## What the agent does
|
|
|
|
<ShowMore id="bug-bash-steps">
|
|
|
|
1. Reads the routes of the app. On a branch, it also reads the diff.
|
|
2. Writes 5 to 10 charters. A charter is a one-sentence goal for one area,
|
|
with one posture: first-time user, numbers and copy, edge input, state
|
|
after reload, or error paths.
|
|
3. Runs one `e2e explore` for each charter, 4 at the same time, each with
|
|
its own persona (`--agent`) and output directory (`--output`).
|
|
4. Merges the findings of all charters.
|
|
5. Rejects findings that have a known cause. The cause can be a limit of
|
|
the explorer (for example, a link that opens a new tab), the local
|
|
environment, the intended design, or the seed data.
|
|
6. Writes a repro test for each remaining finding: the finding's steps as
|
|
`agent.act` calls and its expected behavior as the closing
|
|
`agent.assert`. It can start one subagent per area.
|
|
7. Runs the repro tests. The finding is a bug only when its test fails on
|
|
the reported defect: `ASSERTION_FAILED` from the closing check, or an
|
|
`act` that fails on the defect itself.
|
|
8. Reports, in this order: confirmed bugs, production risks that it cannot
|
|
verify locally, rejected findings by reason, and warnings.
|
|
|
|
</ShowMore>
|
|
|
|
Each confirmed bug has its repro test and steps. It also has the
|
|
explorer's screenshot and video when the engine records them.
|
|
|
|
<Warning>
|
|
The video is not masked. If the explorer typed a password, check the video
|
|
before you share it.
|
|
</Warning>
|
|
|
|
To read the full procedure, run `npx e2e guide bug-bash`.
|
|
|
|
## Repro tests
|
|
|
|
A repro test fails until the bug is fixed. The agent tags each one
|
|
`bugbash`. Keep them out of the run that gates merges:
|
|
|
|
<CodeGroup>
|
|
```bash npm
|
|
npx e2e run --exclude-tag bugbash
|
|
```
|
|
|
|
```bash pnpm
|
|
pnpm exec e2e run --exclude-tag bugbash
|
|
```
|
|
|
|
```bash bun
|
|
bunx e2e run --exclude-tag bugbash
|
|
```
|
|
</CodeGroup>
|
|
|
|
After the fix, the test passes. Remove the tag and keep the test as a
|
|
regression test.
|
|
|
|
## Next
|
|
|
|
<CardGroup cols={2}>
|
|
<Card title="Exploring without a test" icon="compass" href="/explore">
|
|
What one `e2e explore` run does, its budgets, and its report.
|
|
</Card>
|
|
<Card title="MCP server" icon="plug" href="/mcp-server">
|
|
Live sessions for coding agents, several at once.
|
|
</Card>
|
|
</CardGroup>
|