Files
e2e/docs/reference/cli.mdx
T

396 lines
20 KiB
Plaintext
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
---
title: CLI
description: e2e init, guide, mcp, run, explore, list, cache, and telemetry, flag by flag.
---
```bash
npx e2e init [directory]
npx e2e guide [topic]
npx e2e mcp [options]
npx e2e run [files...] [options]
npx e2e explore [goal] [options]
npx e2e list [files...] [options]
npx e2e cache ls|clear|stats [--config <path>]
npx e2e telemetry [status|enable|disable]
npx e2e --help
npx e2e --version
```
The CLI needs Node.js 22.12 or newer.
## Help and version
| Flag | Behavior |
| --- | --- |
| `-h`, `--help` | Prints the help of the command it follows and exits 0. `e2e help run` is the same as `e2e run --help`. |
| `-v`, `--version` | Prints the installed `e2e` version and exits 0. |
```console
$ npx e2e --version
0.11.0
```
Output is colored on a terminal or in CI, plain when piped or when `NO_COLOR`
is set. `FORCE_COLOR` keeps the colors. `e2e` without a command prints the
help on stderr and exits 2.
## e2e init
Scaffolds a project and adds `e2e` to `devDependencies`. Existing
config and test files are skipped. With a `directory` argument the project is
scaffolded there.
The wizard asks three things when it creates a config:
- The engine. **Web** (Playwright, the default) adds `@e2edev/playwright`.
**Mobile (iOS/Android)** adds `@e2edev/agent-device`, defaults to iOS on
macOS and Android elsewhere, and opens Settings. **None** adds nothing.
- The model gateway agent steps use. **Vercel AI Gateway** (the default),
**OpenRouter**, an **OpenAI-compatible endpoint** (prompts for an HTTPS or
loopback base URL), or **None**. A gateway adds `ai@^7.0.0`, the gateway's
provider package when it has one, and an `agents` block that constructs
the model.
- Where to install the agent skill (see [e2e guide](#e2e-guide)):
`.agents/skills/` for Codex, Cursor, Copilot, Gemini CLI, OpenCode, and
Zed, and `.claude/skills/` for Claude Code. Both are preselected.
After confirming the file changes, choose whether to install the selected
dependencies. The installer follows the project's `packageManager` field or
lockfile, then the invoking package manager, then npm.
| Argument or flag | Type | Default | Behavior |
| --- | --- | --- | --- |
| `[directory]` | path | the working directory | Where to scaffold; created when missing |
| `-y`, `--yes` | boolean | `false` | Skips prompts: Web engine, Vercel AI Gateway, skill in both directories, nothing installed |
| Creates | Content |
| --- | --- |
| `package.json` | Created as a private ESM package when missing; otherwise only missing dependencies and scripts are added. Engine packages get the caret range of the version released with the CLI. A `test:e2e` script runs `e2e run` unless one exists |
| `e2e.config.ts` | One target for the selected engine. Playwright reads `APP_URL` (default `http://localhost:3000`) and shows the `command` option in a comment. With a gateway, `agents.default` is `createAgent({ model, system })` |
| `tests/example.e2e.ts` | An app-open test that passes against any page, plus a commented agent test. Device setup gets a Settings check instead |
| `.gitignore` | Appends `node_modules/`, `.e2e/artifacts/`, `.e2e/cache/`, `.e2e/sessions/`, `.e2e/report.json`, `.e2e/ai-trace.json`, `.e2e/junit.xml`, `.e2e/summary.md`, `.e2e/failures/`, and `.e2e/logs/`. Committing the cache is opt-in; see [commit your traces](/reference/config#commit-your-traces) |
| `.agents/skills/e2e/`, `.claude/skills/e2e/` | `SKILL.md` and one `references/<topic>.md` per `e2e guide` topic |
```bash
npx e2e init --yes
```
Re-running `init` keeps an existing config and adds no packages for it. Once
a skill directory exists, later runs refresh copies that differ from the
installed version and never add a directory the project did not choose.
Cancellation leaves files untouched and exits 0. An invalid `package.json`
exits 2 before any write. Without a terminal on stdin and stdout, `init`
without `--yes` exits 2 and names the flag.
## e2e guide
```bash
npx e2e guide # the overview
npx e2e guide setup # one topic
```
Prints the skill that `init` installs. `guide` alone prints the overview;
`guide <topic>` prints one of `setup`, `writing-tests`, `agent`, `running`,
`explore`, `debugging`, or `mcp`. An unknown topic exits 2 and names the
valid ones. The same skill installs with `npx skills add tester-army/e2e`.
See [Coding agents](/coding-agents).
## e2e mcp
```bash
npx e2e mcp [--config <path>] [--target <name>] [--headless]
```
An MCP server on stdio that gives a coding agent a live session on the app.
Flags, tools, and session rules are on the [MCP server](/reference/mcp) page.
## e2e run
### Positional arguments
```bash
e2e run [files...]
```
Test files, directories, or globs relative to the project root:
- a file path (`tests/signup.e2e.ts`) selects that file;
- a directory (`tests/agent`, `.`) selects every discovered test file beneath
it;
- a bare name selects every discovered file whose path ends with it at a
segment boundary. `signup.e2e.ts` and `agent/signup.e2e.ts` both select
`tests/agent/signup.e2e.ts`. Without a `/` or `.` it matches the file name
with extensions dropped, so `signup` selects it too;
- a glob (`'tests/**/*.smoke.e2e.ts'`, quoted) uses the grammar of the config
`tests` globs: `*`, `?`, and a whole `**` segment. A malformed glob is
`INVALID_GLOB`.
Positionals narrow the config globs and the tag, platform, and capability
filters; they never bypass them. Names and globs are case-sensitive on every
OS. A path outside the project root is `COLLECTION_ERROR`.
A positional starting with `-` that names no file is a usage error, exit 2.
This is what `pnpm test:e2e -- --headed` produces: the `--` reaches e2e and
`--headed` would otherwise be taken for a file. The message names the direct
form, `pnpm exec e2e run --headed`.
When nothing is left to run, `NO_TESTS` names the cause: no file matched the
globs (with look-alike files such as `tests/login.test.ts`), a positional
matched nothing (with the nearest path), a matched file registered no tests,
or every test was filtered out.
### Flags
| Flag | Type | Default | Behavior |
| --- | --- | --- | --- |
| `--config <path>` | string | discovered | Explicit config path. A missing file is `CONFIG_NOT_FOUND`. |
| `--target <ids>` | comma-separated list | every configured target | Selects targets by name. Only selected targets start app commands and services. An unknown ID is `UNKNOWN_TARGET`. |
| `--tag <tag>` | string, repeatable | none | Tag filter. Repeat to accumulate. |
| `--tag-mode <mode>` | `any` or `all` | `any` | Tag composition. |
| `--headed` | boolean | `false` | Visible UI when the engine supports it. |
| `--agent <names>` | comma-separated or repeated | `default` | The configured agents unpinned tests run as, one result per name. A test pinned with `{ agent }` runs as the pinned names the flag also gives, or as its whole pin when the flag names none of them. An unknown name is `INVALID_CONFIG`. |
| `--retries <n>` | integer 0-10 | config | Replaces `config.retries`. |
| `--workers <n>` | integer 1-1024 | config | Replaces `config.workers`. |
| `--reporter <ids>` | comma-separated list | config | Replaces the built-in IDs in `config.reporters`: `list`, `json`, `junit`, `markdown`. `json` with `list` is `INVALID_CONFIG`. [Reporter objects](/reference/reporters) in the config keep running. |
| `--artifacts <dir>` | string | `.e2e/artifacts` | Relocates the report and artifact tree. |
| `--no-cache` | boolean | `false` | Trace cache off for this run, whatever `config.cache` says. |
| `--pass-with-no-tests` | boolean | `false` | Exit 0 instead of `NO_TESTS` when nothing is selected. |
| `--debug` | boolean | `false` | Phase timings and an agent step table on stderr after the run. |
| `--ai-trace` | boolean | `false` | Records every model call to `.e2e/ai-trace.json`, an AI SDK devtools database. Open it with [unbox-ai](https://github.com/tester-army/unbox-ai). |
| `--video` | boolean | `false` | Records every attempt (WebM on a browser engine, MP4 on a device engine). Adds the `video` kind to `config.artifacts` for this run; an engine that cannot record fails with `UNSUPPORTED_ARTIFACT`. See [video](/reference/config#video). |
Out-of-range `--retries` and `--workers` fail with `INVALID_CONFIG` before
anything starts, the same as the config value would.
`--debug` appends two tables to stderr. Phase timings are summed across the
runner and every worker. The agent step table lists every agent step in
execution order with its per-phase split, model calls, tokens, and cost when
the provider reported one.
```console
$ npx e2e run --debug
...
[e2e debug] phase timings (wall 14.2s)
phase count total avg max
scheduler 1 11.8s 11.8s 11.8s
test.body 6 10.1s 1.7s 3.9s
session.launch 6 1.1s 183ms 410ms
engine.prepare 1 612ms 612ms 612ms
collect 1 420ms 420ms 420ms
config.load 1 118ms 118ms 118ms
realm.import 1 96ms 96ms 96ms
session.close 6 88ms 15ms 31ms
engine.init 1 74ms 74ms 74ms
[e2e debug] agent steps (execution order, model anthropic/claude-sonnet-4.5, total $0.0312)
step total model observe action calls tokens in/out cached cost
act "add a todo named groceries" 3.9s 2.8s 410ms 620ms 3 9412/388 61% (5740) $0.0198
assert "the list shows groceries" 1.2s 1.1s 88ms 0ms 1 3120/41 0% (0) $0.0114
```
`cached` is the share of input tokens the provider served from its prompt
cache, `-` when it reports no split. The report records the same per step
(`model.cacheReadTokens`, `model.cacheWriteTokens`) and per run
(`usage.modelCachedTokens`).
```bash
npx e2e run
npx e2e run tests/signup.e2e.ts --tag smoke
npx e2e run tests/agent
npx e2e run 'tests/**/*.smoke.e2e.ts'
npx e2e run --target web,webkit --tag smoke --tag billing --tag-mode all
npx e2e run --reporter json --workers 4 --retries 2
npx e2e run --reporter list,junit
```
### Output
| Reporter | Writes |
| --- | --- |
| `list` | Progress to the terminal: a `RUN` banner, one line per setup step, one line per test file and target with failed and agent-driven tests listed beneath, a `Failed Tests` section with each error and a code frame, and a summary (`Test Files`, `Tests`, `AI`, `Cache`, `Start at`, `Duration`, `Report`). On a TTY a live window shows the running tests and the current agent step with its model turns, tool calls, and an excerpt of the model's reasoning. Untrusted fields are capped at 8 KiB with control characters stripped. |
| `json` | The report document to stdout and nothing else there, so `e2e run --reporter json \| jq` works. Same document as `.e2e/report.json`, described by `schema/report-v1.schema.json` in the package. The shape changes only with the `schemaVersion` field (`report-1` today). Cannot be combined with `list`. |
| `junit` | `.e2e/junit.xml` beside the report. One `<testsuite>` per file, one `<testcase>` per test-target pair named `title > path [target]`. A test-class error is a `<failure>`, any other an `<error>`, a skip a `<skipped>` with its reason. Run-level errors form a `run` suite. |
| `markdown` | `.e2e/summary.md` beside the report: a headline with counts and spend, run-level errors, one block per failed or flaky test with its error and its facts (expected and observed, what a locator asked for), the step it went wrong at, whether every attempt failed alike, the last model turns, the screen's location and closest nodes, evidence paths, and a link to the test's own page; and every test folded away by file. Each failed or flaky test also gets a page under `.e2e/failures/` with every step, every kept model turn, and the screen at failure inline. An `e2e explore` run renders its findings instead. This is the page [`@e2edev/github`](/github) posts as the pull request comment. |
Every run writes `.e2e/report.json` under the artifact parent whatever the
reporters are. A `junit.xml` or `summary.md` that could not be written is a
line on stderr, never a run error.
With `--ai-trace`, the run also writes `.e2e/ai-trace.json`: one entry per
model round trip with the exact prompt, tool definitions, response, usage,
and provider metadata. Observations in it are already redacted. Inline image
and file bytes are replaced with byte counts. Open it with
`npx unbox-ai .e2e/ai-trace.json`; see [Debugging a run](/debugging#agent-steps).
### Signals
Signals escalate.
1. The first `SIGINT` or `SIGTERM` interrupts. The running test ends with
status `interrupted`, its `afterEach` hooks and session close run within
the cleanup budget, every worker disposes its engine, and the runner writes
a partial report and exits 130.
2. A second signal forces. Workers dispose their engines at once and are
killed when the cleanup budget is spent. The report is still written.
3. A third signal exits on the spot, after killing the process groups of every
app command and service the run started. A process attached through
`reuseExisting` is left alone.
A worker whose runner disappears disposes its engine and exits within the
cleanup budget.
## e2e explore
```bash
e2e explore [goal]
```
A run with no test file. The agent gets a goal, plans one exploration step
at a time, reports defects through its `report_finding` tool, and ends with an
assessment. The default goal is `Explore the app and find bugs`. See
[Exploring without a test](/explore).
| Flag | Type | Default | Behavior |
| --- | --- | --- | --- |
| `--config <path>` | string | discovered | Explicit config path, with the same errors as `run`. |
| `--target <id>` | string | the first configured target | The one target to explore. Several targets and no flag explores the first and says so on stderr. An unknown name is `UNKNOWN_TARGET`. |
| `--agent <name>` | string | `default` | The configured agent the exploration runs as. Its model, tools, and context are reused; a hand-rolled `StepExecutor` is replaced by the built-in agent, with a notice. An unknown name is `INVALID_CONFIG`. |
| `--max-steps <n>` | integer 1-12 | `8` | Exploration steps at most. Out of range exits 2. |
| `--timeout <ms>` | integer 180000-900000 | `600000` | The run's wall clock. The last minute is kept for the closing assessment. Out of range exits 2. |
| `--headed` | boolean | `false` | Requests visible UI when the engine supports it. |
| `--reporter <ids>` | comma-separated list | resolved config value | As for `run`. The exploration summary rows print under the `list` summary; `json` carries the record in the report. |
| `--artifacts <dir>` | string | `.e2e/artifacts` | Relocates the report and the artifact tree; a finding's evidence screenshot is one of the attempt's artifacts there. |
| `--debug` | boolean | `false` | As for `run`: phase timings and the agent step table. |
| `--ai-trace` | boolean | `false` | As for `run`. |
| `--video` | boolean | `false` | As for `run`: one video of the whole exploration (WebM on a browser engine, MP4 on a device engine) when the engine can record; otherwise the run fails with `UNSUPPORTED_ARTIFACT` before exploration starts. |
The run has one result, under the virtual file `explore`, titled by the goal.
The trace cache is off and retries are zero whatever the config says.
`.e2e/report.json` gains `run.explore`: the goal, the budgets, why the run
ended (`finished`, `step-limit`, `time`, `stuck`, or `aborted`), the
assessment, the steps, and the findings.
Exit codes: `0` when steps ran and no finding of kind `issue` was reported,
`1` when one was or when no step ran at all, `2` and `3` as for `run`, `130`
on Ctrl-C.
```bash
npx e2e explore
npx e2e explore 'Explore the checkout flow like a first-time buyer and report anything off'
npx e2e explore --target web --max-steps 4 --headed
npx e2e explore 'Hunt for broken forms and dead links' --agent thorough --video
```
## e2e list
```bash
e2e list [files...]
```
A dry run. `list` selects tests exactly as `run` does, prints one line per
test-target pair, and exits. No app process, engine, or worker starts, and
nothing is written under `.e2e/`.
Positionals and the selection flags (`--config`, `--target`, `--tag`,
`--tag-mode`, `--pass-with-no-tests`) are those of `run`. `--reporter`
accepts `list` (the default) or `json`.
| Reporter | Writes |
| --- | --- |
| `list` | One line per pair, `file › title [target]`, in selection order, with the describe path joined by ` › ` (`tests/billing.e2e.ts › checkout › pays [web]`). A skipped test keeps its line with `(skipped: <reason>)` appended. Setup tests are listed like any other. |
| `json` | `{ "pairs": [...] }` to stdout. Each pair carries `file`, `title`, `titlePath`, `kind` (`test` or `setup`), `target`, `disposition` (`run` or `skip`), and `skipReason` when skipped. |
Pairs a filter removed are not listed.
```bash
npx e2e list
npx e2e list tests/signup.e2e.ts
npx e2e list --tag smoke --target web
npx e2e list --reporter json
```
## e2e cache
Reads and empties the trace cache under `.e2e/cache/`. Entries are files
named after the digest of their key, so these commands are the only way to
see what is in there. All three resolve the store through the discovered
config or `--config <path>`, honor `cache.dir`, and exit 0 on an empty store.
| Command | Prints |
| --- | --- |
| `e2e cache ls` | One row per entry: the test and target it was recorded for, the first 12 characters of its instruction digest, its age, and how many actions a replay would run (`(truncated)` when recording hit the action cap). Rows are grouped by test, then target, oldest first. |
| `e2e cache stats` | The store directory, the number of readable entries, their total size, and the number of unreadable files when there are any. |
| `e2e cache clear` | Deletes every cache file, then the directory when nothing else is left in it, and reports how many files went. |
```bash
npx e2e cache ls
npx e2e cache stats
npx e2e cache clear
npx e2e cache ls --config packages/web/e2e.config.ts
```
```text
TEST TARGET INSTRUCTION AGE ACTIONS
tests/billing.e2e.ts::upgrades web 4f1c0a9d3b62 2h 4
tests/signup.e2e.ts::signs%20up web 9ab7715ce004 3d 7 (truncated)
```
An entry recorded before entries carried provenance prints `-` in those
columns and still replays. A file the store cannot read (truncated, over the
1 MiB ceiling, or from a newer schema) is counted separately on stderr, treated
as a miss by a run, and removed by `clear`. Files the runner never wrote are
left alone.
<Note>
These commands read the file store only. With a custom `cache.store` they
exit 2 and say so.
</Note>
## e2e telemetry
Shows whether anonymous usage telemetry is on, and switches it.
[Telemetry](/telemetry) lists every property sent.
| Command | Effect |
| --- | --- |
| `e2e telemetry`, `e2e telemetry status` | Prints `Status: enabled`, or `Status: disabled` with the reason: the variable that is set, the saved choice, or a preferences directory that cannot be written. |
| `e2e telemetry disable` | Saves the opt-out to `~/.config/e2e/telemetry.json` (`$XDG_CONFIG_HOME/e2e/` when set, `%APPDATA%\e2e\` on Windows) and prints the status. |
| `e2e telemetry enable` | Saves the opt-in and prints the status. A variable set in the environment still wins, and the status says so. |
```bash
npx e2e telemetry
npx e2e telemetry disable
E2E_TELEMETRY_DEBUG=1 npx e2e run
```
`E2E_TELEMETRY_DISABLED=1` and `DO_NOT_TRACK=1` turn telemetry off without
touching the file. `E2E_TELEMETRY_DEBUG=1` prints every event to stderr and
sends nothing. The first command on a machine prints a notice once; in CI
nothing is printed and no file is written.
## Exit codes
| Code | Meaning |
| ---: | --- |
| 0 | All selected tests passed, were flaky, or were explicitly skipped |
| 1 | Final test or setup failure, or test timeout |
| 2 | CLI, config, collection, dependency, credential, model-config, or policy error |
| 3 | Engine, app process, model provider, artifact, or cleanup infrastructure failure |
| 4 | Internal runner invariant or unhandled runner error |
| 130 | Interrupted by a user or CI signal |
Precedence for a mixed run is `130 > 4 > 3 > 2 > 1 > 0`. The report keeps every
individual result; precedence affects only the process exit code.
Argument parsing failures exit 2 with a one-line error on stderr.
## Diagnostics
A message that rejects a name suggests the nearest valid one. Keys from other
runners' configs (`testDir`, `baseURL`, `webServer`, `projects`) say where
that fact lives here. A failing `import` in the config or a test file ends
with the install or add command for the project's package manager, or the
subpaths the package exports. A model credential the provider rejects names
the variable the provider reads.