mirror of
https://github.com/tester-army/e2e.git
synced 2026-11-22 18:19:48 +08:00
396 lines
20 KiB
Plaintext
396 lines
20 KiB
Plaintext
---
|
||
title: CLI
|
||
description: e2e init, guide, mcp, run, explore, list, cache, and telemetry, flag by flag.
|
||
---
|
||
|
||
```bash
|
||
npx e2e init [directory]
|
||
npx e2e guide [topic]
|
||
npx e2e mcp [options]
|
||
npx e2e run [files...] [options]
|
||
npx e2e explore [goal] [options]
|
||
npx e2e list [files...] [options]
|
||
npx e2e cache ls|clear|stats [--config <path>]
|
||
npx e2e telemetry [status|enable|disable]
|
||
npx e2e --help
|
||
npx e2e --version
|
||
```
|
||
|
||
The CLI needs Node.js 22.12 or newer.
|
||
|
||
## Help and version
|
||
|
||
| Flag | Behavior |
|
||
| --- | --- |
|
||
| `-h`, `--help` | Prints the help of the command it follows and exits 0. `e2e help run` is the same as `e2e run --help`. |
|
||
| `-v`, `--version` | Prints the installed `e2e` version and exits 0. |
|
||
|
||
```console
|
||
$ npx e2e --version
|
||
0.11.0
|
||
```
|
||
|
||
Output is colored on a terminal or in CI, plain when piped or when `NO_COLOR`
|
||
is set. `FORCE_COLOR` keeps the colors. `e2e` without a command prints the
|
||
help on stderr and exits 2.
|
||
|
||
## e2e init
|
||
|
||
Scaffolds a project and adds `e2e` to `devDependencies`. Existing
|
||
config and test files are skipped. With a `directory` argument the project is
|
||
scaffolded there.
|
||
|
||
The wizard asks three things when it creates a config:
|
||
|
||
- The engine. **Web** (Playwright, the default) adds `@e2edev/playwright`.
|
||
**Mobile (iOS/Android)** adds `@e2edev/agent-device`, defaults to iOS on
|
||
macOS and Android elsewhere, and opens Settings. **None** adds nothing.
|
||
- The model gateway agent steps use. **Vercel AI Gateway** (the default),
|
||
**OpenRouter**, an **OpenAI-compatible endpoint** (prompts for an HTTPS or
|
||
loopback base URL), or **None**. A gateway adds `ai@^7.0.0`, the gateway's
|
||
provider package when it has one, and an `agents` block that constructs
|
||
the model.
|
||
- Where to install the agent skill (see [e2e guide](#e2e-guide)):
|
||
`.agents/skills/` for Codex, Cursor, Copilot, Gemini CLI, OpenCode, and
|
||
Zed, and `.claude/skills/` for Claude Code. Both are preselected.
|
||
|
||
After confirming the file changes, choose whether to install the selected
|
||
dependencies. The installer follows the project's `packageManager` field or
|
||
lockfile, then the invoking package manager, then npm.
|
||
|
||
| Argument or flag | Type | Default | Behavior |
|
||
| --- | --- | --- | --- |
|
||
| `[directory]` | path | the working directory | Where to scaffold; created when missing |
|
||
| `-y`, `--yes` | boolean | `false` | Skips prompts: Web engine, Vercel AI Gateway, skill in both directories, nothing installed |
|
||
|
||
| Creates | Content |
|
||
| --- | --- |
|
||
| `package.json` | Created as a private ESM package when missing; otherwise only missing dependencies and scripts are added. Engine packages get the caret range of the version released with the CLI. A `test:e2e` script runs `e2e run` unless one exists |
|
||
| `e2e.config.ts` | One target for the selected engine. Playwright reads `APP_URL` (default `http://localhost:3000`) and shows the `command` option in a comment. With a gateway, `agents.default` is `createAgent({ model, system })` |
|
||
| `tests/example.e2e.ts` | An app-open test that passes against any page, plus a commented agent test. Device setup gets a Settings check instead |
|
||
| `.gitignore` | Appends `node_modules/`, `.e2e/artifacts/`, `.e2e/cache/`, `.e2e/sessions/`, `.e2e/report.json`, `.e2e/ai-trace.json`, `.e2e/junit.xml`, `.e2e/summary.md`, `.e2e/failures/`, and `.e2e/logs/`. Committing the cache is opt-in; see [commit your traces](/reference/config#commit-your-traces) |
|
||
| `.agents/skills/e2e/`, `.claude/skills/e2e/` | `SKILL.md` and one `references/<topic>.md` per `e2e guide` topic |
|
||
|
||
```bash
|
||
npx e2e init --yes
|
||
```
|
||
|
||
Re-running `init` keeps an existing config and adds no packages for it. Once
|
||
a skill directory exists, later runs refresh copies that differ from the
|
||
installed version and never add a directory the project did not choose.
|
||
|
||
Cancellation leaves files untouched and exits 0. An invalid `package.json`
|
||
exits 2 before any write. Without a terminal on stdin and stdout, `init`
|
||
without `--yes` exits 2 and names the flag.
|
||
|
||
## e2e guide
|
||
|
||
```bash
|
||
npx e2e guide # the overview
|
||
npx e2e guide setup # one topic
|
||
```
|
||
|
||
Prints the skill that `init` installs. `guide` alone prints the overview;
|
||
`guide <topic>` prints one of `setup`, `writing-tests`, `agent`, `running`,
|
||
`explore`, `debugging`, or `mcp`. An unknown topic exits 2 and names the
|
||
valid ones. The same skill installs with `npx skills add tester-army/e2e`.
|
||
See [Coding agents](/coding-agents).
|
||
|
||
## e2e mcp
|
||
|
||
```bash
|
||
npx e2e mcp [--config <path>] [--target <name>] [--headless]
|
||
```
|
||
|
||
An MCP server on stdio that gives a coding agent a live session on the app.
|
||
Flags, tools, and session rules are on the [MCP server](/reference/mcp) page.
|
||
|
||
## e2e run
|
||
|
||
### Positional arguments
|
||
|
||
```bash
|
||
e2e run [files...]
|
||
```
|
||
|
||
Test files, directories, or globs relative to the project root:
|
||
|
||
- a file path (`tests/signup.e2e.ts`) selects that file;
|
||
- a directory (`tests/agent`, `.`) selects every discovered test file beneath
|
||
it;
|
||
- a bare name selects every discovered file whose path ends with it at a
|
||
segment boundary. `signup.e2e.ts` and `agent/signup.e2e.ts` both select
|
||
`tests/agent/signup.e2e.ts`. Without a `/` or `.` it matches the file name
|
||
with extensions dropped, so `signup` selects it too;
|
||
- a glob (`'tests/**/*.smoke.e2e.ts'`, quoted) uses the grammar of the config
|
||
`tests` globs: `*`, `?`, and a whole `**` segment. A malformed glob is
|
||
`INVALID_GLOB`.
|
||
|
||
Positionals narrow the config globs and the tag, platform, and capability
|
||
filters; they never bypass them. Names and globs are case-sensitive on every
|
||
OS. A path outside the project root is `COLLECTION_ERROR`.
|
||
|
||
A positional starting with `-` that names no file is a usage error, exit 2.
|
||
This is what `pnpm test:e2e -- --headed` produces: the `--` reaches e2e and
|
||
`--headed` would otherwise be taken for a file. The message names the direct
|
||
form, `pnpm exec e2e run --headed`.
|
||
|
||
When nothing is left to run, `NO_TESTS` names the cause: no file matched the
|
||
globs (with look-alike files such as `tests/login.test.ts`), a positional
|
||
matched nothing (with the nearest path), a matched file registered no tests,
|
||
or every test was filtered out.
|
||
|
||
### Flags
|
||
|
||
| Flag | Type | Default | Behavior |
|
||
| --- | --- | --- | --- |
|
||
| `--config <path>` | string | discovered | Explicit config path. A missing file is `CONFIG_NOT_FOUND`. |
|
||
| `--target <ids>` | comma-separated list | every configured target | Selects targets by name. Only selected targets start app commands and services. An unknown ID is `UNKNOWN_TARGET`. |
|
||
| `--tag <tag>` | string, repeatable | none | Tag filter. Repeat to accumulate. |
|
||
| `--tag-mode <mode>` | `any` or `all` | `any` | Tag composition. |
|
||
| `--headed` | boolean | `false` | Visible UI when the engine supports it. |
|
||
| `--agent <names>` | comma-separated or repeated | `default` | The configured agents unpinned tests run as, one result per name. A test pinned with `{ agent }` runs as the pinned names the flag also gives, or as its whole pin when the flag names none of them. An unknown name is `INVALID_CONFIG`. |
|
||
| `--retries <n>` | integer 0-10 | config | Replaces `config.retries`. |
|
||
| `--workers <n>` | integer 1-1024 | config | Replaces `config.workers`. |
|
||
| `--reporter <ids>` | comma-separated list | config | Replaces the built-in IDs in `config.reporters`: `list`, `json`, `junit`, `markdown`. `json` with `list` is `INVALID_CONFIG`. [Reporter objects](/reference/reporters) in the config keep running. |
|
||
| `--artifacts <dir>` | string | `.e2e/artifacts` | Relocates the report and artifact tree. |
|
||
| `--no-cache` | boolean | `false` | Trace cache off for this run, whatever `config.cache` says. |
|
||
| `--pass-with-no-tests` | boolean | `false` | Exit 0 instead of `NO_TESTS` when nothing is selected. |
|
||
| `--debug` | boolean | `false` | Phase timings and an agent step table on stderr after the run. |
|
||
| `--ai-trace` | boolean | `false` | Records every model call to `.e2e/ai-trace.json`, an AI SDK devtools database. Open it with [unbox-ai](https://github.com/tester-army/unbox-ai). |
|
||
| `--video` | boolean | `false` | Records every attempt (WebM on a browser engine, MP4 on a device engine). Adds the `video` kind to `config.artifacts` for this run; an engine that cannot record fails with `UNSUPPORTED_ARTIFACT`. See [video](/reference/config#video). |
|
||
|
||
Out-of-range `--retries` and `--workers` fail with `INVALID_CONFIG` before
|
||
anything starts, the same as the config value would.
|
||
|
||
`--debug` appends two tables to stderr. Phase timings are summed across the
|
||
runner and every worker. The agent step table lists every agent step in
|
||
execution order with its per-phase split, model calls, tokens, and cost when
|
||
the provider reported one.
|
||
|
||
```console
|
||
$ npx e2e run --debug
|
||
...
|
||
[e2e debug] phase timings (wall 14.2s)
|
||
phase count total avg max
|
||
scheduler 1 11.8s 11.8s 11.8s
|
||
test.body 6 10.1s 1.7s 3.9s
|
||
session.launch 6 1.1s 183ms 410ms
|
||
engine.prepare 1 612ms 612ms 612ms
|
||
collect 1 420ms 420ms 420ms
|
||
config.load 1 118ms 118ms 118ms
|
||
realm.import 1 96ms 96ms 96ms
|
||
session.close 6 88ms 15ms 31ms
|
||
engine.init 1 74ms 74ms 74ms
|
||
|
||
[e2e debug] agent steps (execution order, model anthropic/claude-sonnet-4.5, total $0.0312)
|
||
step total model observe action calls tokens in/out cached cost
|
||
act "add a todo named groceries" 3.9s 2.8s 410ms 620ms 3 9412/388 61% (5740) $0.0198
|
||
assert "the list shows groceries" 1.2s 1.1s 88ms 0ms 1 3120/41 0% (0) $0.0114
|
||
```
|
||
|
||
`cached` is the share of input tokens the provider served from its prompt
|
||
cache, `-` when it reports no split. The report records the same per step
|
||
(`model.cacheReadTokens`, `model.cacheWriteTokens`) and per run
|
||
(`usage.modelCachedTokens`).
|
||
|
||
```bash
|
||
npx e2e run
|
||
npx e2e run tests/signup.e2e.ts --tag smoke
|
||
npx e2e run tests/agent
|
||
npx e2e run 'tests/**/*.smoke.e2e.ts'
|
||
npx e2e run --target web,webkit --tag smoke --tag billing --tag-mode all
|
||
npx e2e run --reporter json --workers 4 --retries 2
|
||
npx e2e run --reporter list,junit
|
||
```
|
||
|
||
### Output
|
||
|
||
| Reporter | Writes |
|
||
| --- | --- |
|
||
| `list` | Progress to the terminal: a `RUN` banner, one line per setup step, one line per test file and target with failed and agent-driven tests listed beneath, a `Failed Tests` section with each error and a code frame, and a summary (`Test Files`, `Tests`, `AI`, `Cache`, `Start at`, `Duration`, `Report`). On a TTY a live window shows the running tests and the current agent step with its model turns, tool calls, and an excerpt of the model's reasoning. Untrusted fields are capped at 8 KiB with control characters stripped. |
|
||
| `json` | The report document to stdout and nothing else there, so `e2e run --reporter json \| jq` works. Same document as `.e2e/report.json`, described by `schema/report-v1.schema.json` in the package. The shape changes only with the `schemaVersion` field (`report-1` today). Cannot be combined with `list`. |
|
||
| `junit` | `.e2e/junit.xml` beside the report. One `<testsuite>` per file, one `<testcase>` per test-target pair named `title > path [target]`. A test-class error is a `<failure>`, any other an `<error>`, a skip a `<skipped>` with its reason. Run-level errors form a `run` suite. |
|
||
| `markdown` | `.e2e/summary.md` beside the report: a headline with counts and spend, run-level errors, one block per failed or flaky test with its error and its facts (expected and observed, what a locator asked for), the step it went wrong at, whether every attempt failed alike, the last model turns, the screen's location and closest nodes, evidence paths, and a link to the test's own page; and every test folded away by file. Each failed or flaky test also gets a page under `.e2e/failures/` with every step, every kept model turn, and the screen at failure inline. An `e2e explore` run renders its findings instead. This is the page [`@e2edev/github`](/github) posts as the pull request comment. |
|
||
|
||
Every run writes `.e2e/report.json` under the artifact parent whatever the
|
||
reporters are. A `junit.xml` or `summary.md` that could not be written is a
|
||
line on stderr, never a run error.
|
||
|
||
With `--ai-trace`, the run also writes `.e2e/ai-trace.json`: one entry per
|
||
model round trip with the exact prompt, tool definitions, response, usage,
|
||
and provider metadata. Observations in it are already redacted. Inline image
|
||
and file bytes are replaced with byte counts. Open it with
|
||
`npx unbox-ai .e2e/ai-trace.json`; see [Debugging a run](/debugging#agent-steps).
|
||
|
||
### Signals
|
||
|
||
Signals escalate.
|
||
|
||
1. The first `SIGINT` or `SIGTERM` interrupts. The running test ends with
|
||
status `interrupted`, its `afterEach` hooks and session close run within
|
||
the cleanup budget, every worker disposes its engine, and the runner writes
|
||
a partial report and exits 130.
|
||
2. A second signal forces. Workers dispose their engines at once and are
|
||
killed when the cleanup budget is spent. The report is still written.
|
||
3. A third signal exits on the spot, after killing the process groups of every
|
||
app command and service the run started. A process attached through
|
||
`reuseExisting` is left alone.
|
||
|
||
A worker whose runner disappears disposes its engine and exits within the
|
||
cleanup budget.
|
||
|
||
## e2e explore
|
||
|
||
```bash
|
||
e2e explore [goal]
|
||
```
|
||
|
||
A run with no test file. The agent gets a goal, plans one exploration step
|
||
at a time, reports defects through its `report_finding` tool, and ends with an
|
||
assessment. The default goal is `Explore the app and find bugs`. See
|
||
[Exploring without a test](/explore).
|
||
|
||
| Flag | Type | Default | Behavior |
|
||
| --- | --- | --- | --- |
|
||
| `--config <path>` | string | discovered | Explicit config path, with the same errors as `run`. |
|
||
| `--target <id>` | string | the first configured target | The one target to explore. Several targets and no flag explores the first and says so on stderr. An unknown name is `UNKNOWN_TARGET`. |
|
||
| `--agent <name>` | string | `default` | The configured agent the exploration runs as. Its model, tools, and context are reused; a hand-rolled `StepExecutor` is replaced by the built-in agent, with a notice. An unknown name is `INVALID_CONFIG`. |
|
||
| `--max-steps <n>` | integer 1-12 | `8` | Exploration steps at most. Out of range exits 2. |
|
||
| `--timeout <ms>` | integer 180000-900000 | `600000` | The run's wall clock. The last minute is kept for the closing assessment. Out of range exits 2. |
|
||
| `--headed` | boolean | `false` | Requests visible UI when the engine supports it. |
|
||
| `--reporter <ids>` | comma-separated list | resolved config value | As for `run`. The exploration summary rows print under the `list` summary; `json` carries the record in the report. |
|
||
| `--artifacts <dir>` | string | `.e2e/artifacts` | Relocates the report and the artifact tree; a finding's evidence screenshot is one of the attempt's artifacts there. |
|
||
| `--debug` | boolean | `false` | As for `run`: phase timings and the agent step table. |
|
||
| `--ai-trace` | boolean | `false` | As for `run`. |
|
||
| `--video` | boolean | `false` | As for `run`: one video of the whole exploration (WebM on a browser engine, MP4 on a device engine) when the engine can record; otherwise the run fails with `UNSUPPORTED_ARTIFACT` before exploration starts. |
|
||
|
||
The run has one result, under the virtual file `explore`, titled by the goal.
|
||
The trace cache is off and retries are zero whatever the config says.
|
||
`.e2e/report.json` gains `run.explore`: the goal, the budgets, why the run
|
||
ended (`finished`, `step-limit`, `time`, `stuck`, or `aborted`), the
|
||
assessment, the steps, and the findings.
|
||
|
||
Exit codes: `0` when steps ran and no finding of kind `issue` was reported,
|
||
`1` when one was or when no step ran at all, `2` and `3` as for `run`, `130`
|
||
on Ctrl-C.
|
||
|
||
```bash
|
||
npx e2e explore
|
||
npx e2e explore 'Explore the checkout flow like a first-time buyer and report anything off'
|
||
npx e2e explore --target web --max-steps 4 --headed
|
||
npx e2e explore 'Hunt for broken forms and dead links' --agent thorough --video
|
||
```
|
||
|
||
## e2e list
|
||
|
||
```bash
|
||
e2e list [files...]
|
||
```
|
||
|
||
A dry run. `list` selects tests exactly as `run` does, prints one line per
|
||
test-target pair, and exits. No app process, engine, or worker starts, and
|
||
nothing is written under `.e2e/`.
|
||
|
||
Positionals and the selection flags (`--config`, `--target`, `--tag`,
|
||
`--tag-mode`, `--pass-with-no-tests`) are those of `run`. `--reporter`
|
||
accepts `list` (the default) or `json`.
|
||
|
||
| Reporter | Writes |
|
||
| --- | --- |
|
||
| `list` | One line per pair, `file › title [target]`, in selection order, with the describe path joined by ` › ` (`tests/billing.e2e.ts › checkout › pays [web]`). A skipped test keeps its line with `(skipped: <reason>)` appended. Setup tests are listed like any other. |
|
||
| `json` | `{ "pairs": [...] }` to stdout. Each pair carries `file`, `title`, `titlePath`, `kind` (`test` or `setup`), `target`, `disposition` (`run` or `skip`), and `skipReason` when skipped. |
|
||
|
||
Pairs a filter removed are not listed.
|
||
|
||
```bash
|
||
npx e2e list
|
||
npx e2e list tests/signup.e2e.ts
|
||
npx e2e list --tag smoke --target web
|
||
npx e2e list --reporter json
|
||
```
|
||
|
||
## e2e cache
|
||
|
||
Reads and empties the trace cache under `.e2e/cache/`. Entries are files
|
||
named after the digest of their key, so these commands are the only way to
|
||
see what is in there. All three resolve the store through the discovered
|
||
config or `--config <path>`, honor `cache.dir`, and exit 0 on an empty store.
|
||
|
||
| Command | Prints |
|
||
| --- | --- |
|
||
| `e2e cache ls` | One row per entry: the test and target it was recorded for, the first 12 characters of its instruction digest, its age, and how many actions a replay would run (`(truncated)` when recording hit the action cap). Rows are grouped by test, then target, oldest first. |
|
||
| `e2e cache stats` | The store directory, the number of readable entries, their total size, and the number of unreadable files when there are any. |
|
||
| `e2e cache clear` | Deletes every cache file, then the directory when nothing else is left in it, and reports how many files went. |
|
||
|
||
```bash
|
||
npx e2e cache ls
|
||
npx e2e cache stats
|
||
npx e2e cache clear
|
||
npx e2e cache ls --config packages/web/e2e.config.ts
|
||
```
|
||
|
||
```text
|
||
TEST TARGET INSTRUCTION AGE ACTIONS
|
||
tests/billing.e2e.ts::upgrades web 4f1c0a9d3b62 2h 4
|
||
tests/signup.e2e.ts::signs%20up web 9ab7715ce004 3d 7 (truncated)
|
||
```
|
||
|
||
An entry recorded before entries carried provenance prints `-` in those
|
||
columns and still replays. A file the store cannot read (truncated, over the
|
||
1 MiB ceiling, or from a newer schema) is counted separately on stderr, treated
|
||
as a miss by a run, and removed by `clear`. Files the runner never wrote are
|
||
left alone.
|
||
|
||
<Note>
|
||
These commands read the file store only. With a custom `cache.store` they
|
||
exit 2 and say so.
|
||
</Note>
|
||
|
||
## e2e telemetry
|
||
|
||
Shows whether anonymous usage telemetry is on, and switches it.
|
||
[Telemetry](/telemetry) lists every property sent.
|
||
|
||
| Command | Effect |
|
||
| --- | --- |
|
||
| `e2e telemetry`, `e2e telemetry status` | Prints `Status: enabled`, or `Status: disabled` with the reason: the variable that is set, the saved choice, or a preferences directory that cannot be written. |
|
||
| `e2e telemetry disable` | Saves the opt-out to `~/.config/e2e/telemetry.json` (`$XDG_CONFIG_HOME/e2e/` when set, `%APPDATA%\e2e\` on Windows) and prints the status. |
|
||
| `e2e telemetry enable` | Saves the opt-in and prints the status. A variable set in the environment still wins, and the status says so. |
|
||
|
||
```bash
|
||
npx e2e telemetry
|
||
npx e2e telemetry disable
|
||
E2E_TELEMETRY_DEBUG=1 npx e2e run
|
||
```
|
||
|
||
`E2E_TELEMETRY_DISABLED=1` and `DO_NOT_TRACK=1` turn telemetry off without
|
||
touching the file. `E2E_TELEMETRY_DEBUG=1` prints every event to stderr and
|
||
sends nothing. The first command on a machine prints a notice once; in CI
|
||
nothing is printed and no file is written.
|
||
|
||
## Exit codes
|
||
|
||
| Code | Meaning |
|
||
| ---: | --- |
|
||
| 0 | All selected tests passed, were flaky, or were explicitly skipped |
|
||
| 1 | Final test or setup failure, or test timeout |
|
||
| 2 | CLI, config, collection, dependency, credential, model-config, or policy error |
|
||
| 3 | Engine, app process, model provider, artifact, or cleanup infrastructure failure |
|
||
| 4 | Internal runner invariant or unhandled runner error |
|
||
| 130 | Interrupted by a user or CI signal |
|
||
|
||
Precedence for a mixed run is `130 > 4 > 3 > 2 > 1 > 0`. The report keeps every
|
||
individual result; precedence affects only the process exit code.
|
||
|
||
Argument parsing failures exit 2 with a one-line error on stderr.
|
||
|
||
## Diagnostics
|
||
|
||
A message that rejects a name suggests the nearest valid one. Keys from other
|
||
runners' configs (`testDir`, `baseURL`, `webServer`, `projects`) say where
|
||
that fact lives here. A failing `import` in the config or a test file ends
|
||
with the install or add command for the project's package manager, or the
|
||
subpaths the package exports. A model credential the provider rejects names
|
||
the variable the provider reads.
|