Files
e2e/docs/bug-bash.mdx

131 lines
4.1 KiB
Plaintext

---
title: Bug bashes
description: Run many explore sessions at the same time, then prove each bug with a failing test.
"og:image": "https://e2e.com/docs/images/og/bug-bash.png?v=eb59f8290c"
"twitter:image": "https://e2e.com/docs/images/og/bug-bash.png?v=eb59f8290c"
---
import { ShowMore } from './snippets/show-more.jsx';
import { Video } from './snippets/video.jsx';
A bug bash runs many `e2e explore` sessions at the same time, each on one
area of the app. A coding agent then writes a test for each finding and
reports it as a bug only when the test fails.
<Video>
<video src="/images/bug-bash.mp4" aria-label="A coding agent runs a bug bash in headed browsers." muted loop playsInline controls />
Each browser is one explore session.
</Video>
## Before you start
- Make sure the app can serve many explorers at the same time. Start the
app one time and set `reuseExisting: true` on the target's `app.command`,
or set port `0` in the app URL so each run starts its own copy. When `CI`
is set, e2e ignores `reuseExisting`. Use port `0`. See
[Starting your app](/web#starting-your-app).
- On mobile, declare one target for each simulator or emulator. Each
explorer needs its own device.
- Each explore session uses model calls. Each run ends with its tokens,
cost, and duration.
- On a deployed site that other people use, do not change data.
The agent does the rest of the setup, such as seeded accounts and sign-in,
from the skill.
## Install the skill
The coding agent runs the bug bash from the [`e2e` skill](/coding-agents),
which `e2e init` installs. To install it without `init`:
<CodeGroup>
```bash npm
npx skills add e2e-inc/e2e
```
```bash pnpm
pnpm dlx skills add e2e-inc/e2e
```
```bash bun
bunx skills add e2e-inc/e2e
```
</CodeGroup>
## Start a bug bash
Give a prompt to your coding agent:
```text
Bug bash the checkout and account settings on this branch.
```
## What the agent does
<ShowMore id="bug-bash-steps">
1. Reads the routes of the app. On a branch, it also reads the diff.
2. Writes 5 to 10 charters. A charter is a one-sentence goal for one area,
with one posture: first-time user, numbers and copy, edge input, state
after reload, or error paths.
3. Runs one `e2e explore` for each charter, 4 at the same time, each with
its own persona (`--agent`) and output directory (`--output`).
4. Merges the findings of all charters.
5. Rejects findings that have a known cause. The cause can be a limit of
the explorer (for example, a link that opens a new tab), the local
environment, the intended design, or the seed data.
6. Writes a repro test for each remaining finding: the finding's steps as
`agent.act` calls and its expected behavior as the closing
`agent.assert`. It can start one subagent per area.
7. Runs the repro tests. The finding is a bug only when its test fails on
the reported defect: `ASSERTION_FAILED` from the closing check, or an
`act` that fails on the defect itself.
8. Reports, in this order: confirmed bugs, production risks that it cannot
verify locally, rejected findings by reason, and warnings.
</ShowMore>
Each confirmed bug has its repro test and steps. It also has the
explorer's screenshot and video when the engine records them.
<Warning>
The video is not masked. If the explorer typed a password, check the video
before you share it.
</Warning>
To read the full procedure, run `npx e2e guide bug-bash`.
## Repro tests
A repro test fails until the bug is fixed. The agent tags each one
`bugbash`. Keep them out of the run that gates merges:
<CodeGroup>
```bash npm
npx e2e run --exclude-tag bugbash
```
```bash pnpm
pnpm exec e2e run --exclude-tag bugbash
```
```bash bun
bunx e2e run --exclude-tag bugbash
```
</CodeGroup>
After the fix, the test passes. Remove the tag and keep the test as a
regression test.
## Next
<CardGroup cols={2}>
<Card title="Exploring without a test" icon="compass" href="/explore">
What one `e2e explore` run does, its budgets, and its report.
</Card>
<Card title="MCP server" icon="plug" href="/mcp-server">
Live sessions for coding agents, several at once.
</Card>
</CardGroup>