mirror of
https://github.com/tester-army/e2e.git
synced 2026-11-21 03:05:26 +08:00
Co-authored-by: Oskar Kwaśniewski <oskar@okwasniewski.com>
221 lines
7.6 KiB
Plaintext
221 lines
7.6 KiB
Plaintext
---
|
||
title: Exploring without a test
|
||
description: Give the agent a goal, not a test file. It plans its steps, operates the app, and reports what it finds.
|
||
"og:image": "https://e2e.com/docs/images/og/explore.png?v=8969611dd1"
|
||
"twitter:image": "https://e2e.com/docs/images/og/explore.png?v=8969611dd1"
|
||
---
|
||
|
||
import { Video } from './snippets/video.jsx';
|
||
|
||
`e2e explore` runs the agent against your app with a goal instead of a test
|
||
file. The agent plans a step, does it, and records findings until the goal
|
||
is covered or a budget ends, then writes an assessment. It uses the same
|
||
model, tools, and action rules as `agent.act()`.
|
||
|
||
<Video>
|
||
<video src="/images/explore.mp4" aria-label="e2e explore drives the app in a headed browser and reports findings in the terminal." muted loop playsInline controls />
|
||
|
||
`e2e explore --headed` drives the app and reports findings as it goes.
|
||
</Video>
|
||
|
||
## Start an exploration
|
||
|
||
Name the flow and the user goal, and add limits if necessary:
|
||
|
||
<CodeGroup>
|
||
```bash npm
|
||
npx e2e explore 'Explore the checkout flow like a first-time buyer and report anything off'
|
||
```
|
||
|
||
```bash pnpm
|
||
pnpm exec e2e explore 'Explore the checkout flow like a first-time buyer and report anything off'
|
||
```
|
||
|
||
```bash bun
|
||
bunx e2e explore 'Explore the checkout flow like a first-time buyer and report anything off'
|
||
```
|
||
</CodeGroup>
|
||
|
||
A limit can be "stop before placing the order." With no goal, explore uses
|
||
`Explore the app and find bugs`. A failed step does not stop the run.
|
||
|
||
With no `e2e.config.ts`, explore runs the
|
||
[`e2e init`](/reference/cli#e2e-init) wizard first. The exploration starts
|
||
right after when nothing is left to do by hand. Otherwise init lists the
|
||
steps left, such as `npm install` or setting the model key.
|
||
Without a terminal, as in CI, explore fails with `CONFIG_NOT_FOUND`.
|
||
|
||
## Read the results
|
||
|
||
The terminal shows each finding and each finished step as they happen:
|
||
|
||
```text
|
||
❯ web Exploring Explore this bookshop like a careful first-time buyer: browse the catalog, ...
|
||
⚑ medium issue Home greeting exposes an unresolved userName template (/)
|
||
⚑ high issue Neuromancer shows an impossible negative stock availability (/catalog)
|
||
✓ 1 Catalog browsing 31.46s · 7 actions · 4 findings
|
||
⚑ high issue Cart total does not update when Hyperion quantity increases (/cart)
|
||
⚑ trivial warning Checkout newsletter label misspells Receive (/checkout)
|
||
✓ 2 Cart quantities and removal 19.47s · 4 actions · 3 findings
|
||
ended: the agent covered the goal
|
||
```
|
||
|
||
An `issue` fails the run with exit code 1. A `warning` does not. The final
|
||
report puts the most severe issues first. Each finding has the steps to
|
||
reproduce it and, when possible, a screenshot. Compare each finding with
|
||
its evidence before you act on it. For the other exit codes, the JSON
|
||
report, and artifact paths, see the [CLI reference](/reference/cli#e2e-explore).
|
||
|
||
## Check other screen sizes
|
||
|
||
When the target's engine can resize the viewport (the web engine can), a
|
||
goal can ask for layouts at other widths:
|
||
|
||
<CodeGroup>
|
||
```bash npm
|
||
npx e2e explore 'Check the pricing page at desktop, tablet, and phone widths for broken layouts'
|
||
```
|
||
|
||
```bash pnpm
|
||
pnpm exec e2e explore 'Check the pricing page at desktop, tablet, and phone widths for broken layouts'
|
||
```
|
||
|
||
```bash bun
|
||
bunx e2e explore 'Check the pricing page at desktop, tablet, and phone widths for broken layouts'
|
||
```
|
||
</CodeGroup>
|
||
|
||
## Start signed in
|
||
|
||
Add `--session <name>`, the name of a session that a
|
||
[setup test](/authentication) saves:
|
||
|
||
<CodeGroup>
|
||
```bash npm
|
||
npx e2e explore --session admin 'Explore the admin settings and find bugs'
|
||
```
|
||
|
||
```bash pnpm
|
||
pnpm exec e2e explore --session admin 'Explore the admin settings and find bugs'
|
||
```
|
||
|
||
```bash bun
|
||
bunx e2e explore --session admin 'Explore the admin settings and find bugs'
|
||
```
|
||
</CodeGroup>
|
||
|
||
Explore runs only that setup test and restores the session. The explorer
|
||
knows the app starts signed in and plans no sign-in step.
|
||
|
||
Screenshots mask the password field. A value the app shows anywhere else
|
||
reaches the model. See
|
||
[screenshots after a secret fill](/reference/security#screenshots-after-a-secret-fill).
|
||
|
||
## Explore a pull request's change
|
||
|
||
Add `--changed-since <ref>` to explore what the branch changed. The planner
|
||
reads the commit messages and the committed diff since the merge base with
|
||
the ref. It spends its steps on the screens that the change can affect.
|
||
|
||
<CodeGroup>
|
||
```bash npm
|
||
npx e2e explore --changed-since origin/main
|
||
```
|
||
|
||
```bash pnpm
|
||
pnpm exec e2e explore --changed-since origin/main
|
||
```
|
||
|
||
```bash bun
|
||
bunx e2e explore --changed-since origin/main
|
||
```
|
||
</CodeGroup>
|
||
|
||
Only a defect the change can cause is an `issue`. A defect elsewhere in the
|
||
app is a `warning` and doesn't fail the run.
|
||
|
||
With no goal, the goal is `Explore what this change affects and find bugs in
|
||
it`. The planner can then skip a change no user can see, such as docs or
|
||
tests only. The run ends `skipped` and passes. It skips only when it read
|
||
the whole change. A large diff, a binary file, or an uncommitted file rules
|
||
a skip out.
|
||
|
||
The branch's own commit messages and code can sway the call. To tell a skip
|
||
from a clean exploration, read `run.explore.ended`.
|
||
|
||
For what the model reads, see
|
||
[Security](/reference/security#mcp-and-explore).
|
||
|
||
On GitHub Actions, run it in its own job on `pull_request`. With the
|
||
[GitHub reporter](/github) in the config, the findings go to their own pull
|
||
request comment. A fork's run can't post; see
|
||
[when nothing is posted](/github#when-nothing-is-posted).
|
||
|
||
```yaml title=".github/workflows/explore.yml"
|
||
on: pull_request
|
||
|
||
jobs:
|
||
explore:
|
||
runs-on: ubuntu-latest
|
||
permissions:
|
||
contents: read
|
||
pull-requests: write
|
||
# An issue exits 1. Keep the findings as a review aid, not a merge gate.
|
||
continue-on-error: true
|
||
steps:
|
||
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
||
with:
|
||
fetch-depth: 0
|
||
# ...install, browsers
|
||
- run: npx e2e explore --changed-since origin/${{ github.base_ref }} --max-steps 4
|
||
env:
|
||
AI_GATEWAY_API_KEY: ${{ secrets.AI_GATEWAY_API_KEY }}
|
||
GITHUB_TOKEN: ${{ github.token }}
|
||
```
|
||
|
||
`fetch-depth: 0` puts the base branch in the checkout. Without it, the run
|
||
fails with `INVALID_CHANGED_SINCE`.
|
||
|
||
Keep the working tree clean after install and build, or the run is never
|
||
skipped. Ignore `.e2e/` and build output. Run e2e with `npx`, not
|
||
`pnpm exec`, which can rewrite `pnpm-lock.yaml`.
|
||
|
||
<Warning>
|
||
Never run it on `pull_request_target`. That run holds your secrets, and the
|
||
model reads the diff and commit messages a fork wrote.
|
||
</Warning>
|
||
|
||
## If the run stops too early
|
||
|
||
A run stops after 8 steps or 10 minutes. To reach further into the flow,
|
||
raise `--max-steps <n>` (up to 100) or `--timeout <ms>` (up to `7200000`, 2
|
||
hours).
|
||
|
||
A run with time left after the main flows returns to them with edge-case
|
||
inputs, combined filters, reloads, and, with more than one account configured
|
||
and no `--session`, another account. It finishes early only when nothing new
|
||
is reachable.
|
||
|
||
## Choose the target and agent
|
||
|
||
Explore uses the targets, app command, models, and agents from
|
||
`e2e.config.ts`. Select them with `--target <name>` and `--agent <name>`.
|
||
Without `--target`, explore uses the first target.
|
||
|
||
## Explore from code
|
||
|
||
To start an exploration from your own code, with a config value,
|
||
cancellation, and progress events, call
|
||
[`explore` from `e2e/runner`](/reference/runner#explore).
|
||
|
||
## Next
|
||
|
||
<CardGroup cols={2}>
|
||
<Card title="Bug bashes" icon="bug" href="/bug-bash">
|
||
Explore a whole app or branch with many explorations, and prove each bug with a test.
|
||
</Card>
|
||
<Card title="Agents and personas" icon="users" href="/agents">
|
||
The agents explore can run as.
|
||
</Card>
|
||
</CardGroup>
|