mirror of
https://github.com/tester-army/e2e.git
synced 2026-11-21 03:05:26 +08:00
* feat(mcp): attach to a test held at screen.pause() or --step
A test held at `screen.pause()` or at an `e2e run --step` hold now prints
an ID, such as `Paused (p1) in pays with a card [web]:`. You can pass it to
`open_session {attach: "p1"}` in `e2e mcp` to look at the held test's app
and act on it.
The tools run in the worker, on the engine session the held attempt already
has. Requests go from the MCP server to the runner over the directory's
step socket, and the runner relays them to the worker. Replies are redacted
with the attempt's secret ledger before they're sent.
- `close_session` detaches and leaves the test held.
- When the test resumes, the session ends right away and a call still in
progress is cancelled.
- Attached calls aren't recorded as steps in the report or the trace.
- `run_code`, the recording tools, and `type_secret` aren't available in an
attached session.
- `e2e run --step` in a terminal now listens on the step socket like a pause
does, so you can attach at its holds.
With several workers paused, each prompt is now drawn from the hold it
names. Before, it could show one test's title next to another test's screen.
* fix(mcp): print output a tool's timer writes after its attached call ends
Output is muted only while the attached call is running. The call scope
lives in its own module, so process-output.ts doesn't load the attach tools.
* feat(mcp): record in an attached session when the run doesn't record
start_recording and stop_recording join the attached catalog, which now
lists exactly the normal session's tools. If the run already records the
test, start_recording fails with the run's video directory. A recording
still running when the session ends is saved.
The docs no longer describe what a pause ID looks like. The examples and
sample output show one.
* fix(mcp): don't hold a resumed test while an attached recording saves
On a resume or an interrupt the worker sends the stop for the agent's
recording and lets the test continue. The file finishes in the background
and its path goes to the log. Teardown waits for a save still in progress
for at most cleanupTimeout.
A project tool's attachScreenshot works in an attached session and stays
out of the test's report. In the live list a long title ends in an
ellipsis, so the pause ID stays visible.
561 lines
24 KiB
Plaintext
561 lines
24 KiB
Plaintext
---
|
||
title: Debugging a run
|
||
description: Read the trace page, open the evidence, watch the browser, and see what the model was sent.
|
||
"og:image": "https://e2e.com/docs/images/og/debugging.png?v=38a51086e1"
|
||
"twitter:image": "https://e2e.com/docs/images/og/debugging.png?v=38a51086e1"
|
||
---
|
||
|
||
A failed test prints its error and the path to its [trace page](#traces). For
|
||
a summary of the whole run, add the `markdown` reporter. It writes
|
||
`.e2e/summary.md`.
|
||
|
||
## Read the failure
|
||
|
||
The `list` reporter's `Failed Tests` section shows the error, the failing
|
||
line, nearby matches for a failed locator, what the app logged, and the
|
||
trace page. It quotes the app's first 3 errors (console errors, uncaught
|
||
exceptions, failed requests, 5xx responses), redacted like the trace page.
|
||
`.e2e/summary.md` quotes them too:
|
||
|
||
```text
|
||
FAIL |web| tests/todos.e2e.ts > shows the remaining count
|
||
LOCATOR_NOT_FOUND: locator did not become visible: getByRole("heading", name: "Todos")
|
||
❯ tests/todos.e2e.ts:31:3
|
||
❯ at http://127.0.0.1:4271/nope
|
||
❯ app 2 errors, 2 warnings logged
|
||
× console error: Error occurred prerendering page "/todos"
|
||
× network error: GET /api/todos 500 Internal Server Error
|
||
❯ trace .e2e/results/todos-shows-the-remaining-count-796fac5f12ab34cd/trace.md
|
||
```
|
||
|
||
The exit code tells the type of failure:
|
||
|
||
| Exit | Meaning | Retry the job? |
|
||
| ---: | --- | --- |
|
||
| 1 | A test failed | Only if the test is flaky |
|
||
| 2 | Configuration, collection, or policy | Fix the reported problem first |
|
||
| 3 | Engine, app process, model provider, artifact, or cleanup infrastructure failure | If the cause is temporary |
|
||
| 4 | An internal runner error | Report it |
|
||
| 130 | The run was interrupted | |
|
||
|
||
## Traces
|
||
|
||
By default, each failed, timed out, interrupted, or flaky test gets a trace
|
||
page, `trace.md`, in its [results directory](#artifacts). Open it in an
|
||
editor's markdown preview so its links work. After the browser or device
|
||
the test ran on, it shows:
|
||
|
||
- **Steps.** The node an action landed on, the values an assertion read
|
||
while it waited, the replay cache's decision, what the app logged, where
|
||
the page went, and how the screen changed since the step before.
|
||
- **App log.** Console output, uncaught exceptions, failed requests, and
|
||
4xx and 5xx responses, each with its step: up to 50 lines, errors and
|
||
warnings first. An attempt keeps at most 200 errors and warnings and 100
|
||
other lines.
|
||
- **The agent's last turns.**
|
||
- **Screen at failure.** The screenshot and the semantic tree, captured in
|
||
at most 5 s. An app that doesn't answer in time gets no screen section.
|
||
|
||
<Accordion title="Example trace page">
|
||
|
||
````markdown
|
||
# ✗ todos › shows the remaining count
|
||
|
||
`tests/todos.e2e.ts` · failed · 2.2s
|
||
|
||
**ASSERTION_FAILED**
|
||
|
||
```text
|
||
expect.toHaveText failed
|
||
locator: getByRole("status", name: "Remaining")
|
||
expected: text "1 remaining"
|
||
observed: text "0 remaining" (match count 1)
|
||
```
|
||
|
||
Ran on: browser `chromium 141.0.7390.37` · user agent `Mozilla/5.0 (Macintosh; …) HeadlessChrome/141.0.7390.37 Safari/537.36`
|
||
|
||
Look at: `tests/todos.e2e.ts:15`
|
||
|
||
## Steps
|
||
|
||
1. ✓ `app.open` `/todos` (37ms in beforeEach) — `tests/todos.e2e.ts:4`
|
||
2. ✓ `agent.act` "Add a todo named Milk." (429ms, 2 model calls) — `tests/todos.e2e.ts:12`
|
||
- cache: no replay: nothing recorded for this step yet; recording saved (`.e2e/cache/1227d5a6….json`)
|
||
- type "Milk" into textbox "New todo" (17ms)
|
||
- press "Enter" on the focused field (5ms)
|
||
- ✗ network error: `POST /api/todos 500 Internal Server Error`
|
||
- screen: 6 changes since step 1 at `/todos`, 38 nodes
|
||
- `added listitem "Milk Delete Milk" testid="todo"`
|
||
- `added checkbox "Milk"`
|
||
- `changed status "Remaining" text="1 remaining" (was: status "Remaining" text="0 remaining")`
|
||
- 3 more
|
||
3. ✓ `locator.check` `getByLabel("Milk")` (13ms) — `tests/todos.e2e.ts:14`
|
||
- check checkbox "Milk" (13ms)
|
||
- screen: 3 changes since step 2 at `/todos`, 38 nodes
|
||
- `added checkbox "Milk" [checked]`
|
||
- `changed status "Remaining" text="0 remaining" (was: status "Remaining" text="1 remaining")`
|
||
- `removed checkbox "Milk"`
|
||
4. ✗ `expect.toHaveText` `getByRole("status", name: "Remaining")` (1.6s) — **ASSERTION_FAILED** — `tests/todos.e2e.ts:15`
|
||
- expect gave up after 16 reads in 1.6s: text "0 remaining" (1 match) x16
|
||
|
||
## App log
|
||
|
||
What the app logged, oldest first, with the step it happened in.
|
||
|
||
- step 2 · ✗ network error: `POST /api/todos 500 Internal Server Error`
|
||
|
||
## Screen at failure
|
||
`http://127.0.0.1:4271/todos` · viewport 1280x720 · 38 nodes
|
||
|
||

|
||
|
||
```text
|
||
#n29 textbox "New todo"
|
||
#n30 button "Add"
|
||
#n35 listitem "Milk Delete Milk" testid="todo"
|
||
#n36 checkbox "Milk" [checked]
|
||
#n40 status "Remaining" text="0 remaining"
|
||
```
|
||
````
|
||
|
||
</Accordion>
|
||
|
||
An assertion that read one value the whole time has a wrong expectation, or
|
||
the app never reached that state: look in the app log for a failed request.
|
||
An assertion whose readings changed (`"2 remaining" x3 -> at 410ms "1
|
||
remaining"`) raced the app. `screen: unchanged` after an action means the
|
||
action had no visible effect. On web, a step's screen is the one an action
|
||
left. On a device, it's the screen the step acted on.
|
||
|
||
The [`trace`](/reference/config#trace) mode sets which tests keep a page.
|
||
`--trace` keeps one for every test of a run, passing ones included.
|
||
`--trace off` writes none. A test's own `trace` option wins over both.
|
||
|
||
## Artifacts
|
||
|
||
Each test gets one directory under `.e2e/results/` (or
|
||
[`<output>`](/reference/config#output)`/results/`), with its trace page and
|
||
one `attempt-<n>` per attempt. A serial group's tests share one directory:
|
||
|
||
```text
|
||
.e2e/results/todos-adds-a-todo-1a2b3c4d5e6f7a8b/trace.md
|
||
.e2e/results/todos-adds-a-todo-1a2b3c4d5e6f7a8b/attempt-1/screen-at-failure.txt
|
||
.e2e/results/todos-adds-a-todo-1a2b3c4d5e6f7a8b/attempt-1/screenshots/001-failure.png
|
||
```
|
||
|
||
Each run that reaches its tests empties `results/` first. A `--last-failed`
|
||
rerun keeps the files that the report it reruns names, and writes its own
|
||
attempts under each test's `rerun-<n>/`.
|
||
|
||
| Artifact | When |
|
||
| --- | --- |
|
||
| Screenshots | A failed attempt captures one when the engine can. The built-in `agent.assert` captures one after either verdict, unless `screenshot: false`. `app.screenshot('label')` captures one on request. They mask secure fields only. See [screenshots after a secret fill](/reference/security#screenshots-after-a-secret-fill). |
|
||
| Downloads | Files saved through `browser.waitForDownload`, kind `download`, as the app served them. [Downloads](/reference/security#downloads) says when the runner masks secrets in them. |
|
||
| Videos | When you [record them](#watch-it-happen). Videos mask nothing. See [where redaction stops](/reference/security#where-redaction-stops). |
|
||
| Agent transcripts | With [`--debug`](#agent-steps). |
|
||
|
||
## For scripts: report.json
|
||
|
||
Every run that reaches its tests also writes `report.json` to the
|
||
[output directory](/reference/config#output), `.e2e` by default. It holds
|
||
every result of the run in a versioned schema. `--reporter json` prints it to
|
||
stdout:
|
||
|
||
```bash
|
||
jq '.run | {status, exitCode, errors}' .e2e/report.json
|
||
jq '.run.results[] | select(.status != "passed") | .attempts[-1]
|
||
| {status, error, failure, steps: [.steps[] | select(.status != "passed") | {api, label, source, status, error}], artifacts}' .e2e/report.json
|
||
```
|
||
|
||
`run.errors` holds run-level failures. The last attempt of each failed
|
||
result holds:
|
||
|
||
| Field | Holds |
|
||
| --- | --- |
|
||
| `error` | The code, message, source line, and assertion details |
|
||
| `failure` | The screen and screenshot artifacts, and any locator candidates |
|
||
| `steps` | The calls that led to the failure. Each step's `events` hold what it did (`engine`), what it polled (`poll`), and where the app went (`navigation`). Agent steps add `cache` and recent model turns. `observationTruncated: true` means at least one semantic observation was incomplete. |
|
||
| `appLog` | What the app logged. Each line's `step` is the `index` of its step, absent before the first step. |
|
||
| `artifacts` | The captured files |
|
||
|
||
A serial group's tests have no `attempts` of their own. Their results are in
|
||
`run.serialGroups`, one member for each test in each attempt. A retry the run
|
||
interrupted or that never reached its members has `cutRetry: true` and does
|
||
not hold the verdicts, so skip it:
|
||
|
||
```bash
|
||
jq '.run.serialGroups[] | (.attempts | map(select(.cutRetry != true)) | last).members[]? | select(.status != "passed")
|
||
| {testId, status, error, failure}' .e2e/report.json
|
||
```
|
||
|
||
## Watch it happen
|
||
|
||
<CodeGroup>
|
||
```bash npm
|
||
npx e2e run --headed # a visible browser
|
||
npx e2e run --video # record every attempt
|
||
npx e2e run --video=retain-on-failure # record every attempt, keep the failures
|
||
```
|
||
|
||
```bash pnpm
|
||
pnpm exec e2e run --headed # a visible browser
|
||
pnpm exec e2e run --video # record every attempt
|
||
pnpm exec e2e run --video=retain-on-failure # record every attempt, keep the failures
|
||
```
|
||
|
||
```bash bun
|
||
bunx e2e run --headed # a visible browser
|
||
bunx e2e run --video # record every attempt
|
||
bunx e2e run --video=retain-on-failure # record every attempt, keep the failures
|
||
```
|
||
</CodeGroup>
|
||
|
||
`--video` takes an optional mode. Put test files before it, or write
|
||
`--video=<mode>`.
|
||
|
||
Each recording goes into its attempt's directory: WebM per page on web, MP4
|
||
on a device. The failure recap prints the path, or the URL when a hosted
|
||
browser keeps the recording. Subtract the recording's `startedAt` from a
|
||
step's `startedAt` to find that step in the video.
|
||
|
||
In CI, set `video: 'on-first-retry'` in the config. A test that passes the
|
||
first time records nothing, and a flaky test leaves a recording of its
|
||
retry.
|
||
|
||
A target's `video` wins over the config's, and `--video` wins over both. A
|
||
test's own option, such as `test('checkout', { video: 'on' }, ...)`, wins
|
||
over all of them.
|
||
|
||
The config's `video` and `--video` apply only to targets whose engine can
|
||
record, and one notice names the others. When a target or a test asks for a
|
||
recording its engine can't make, the run fails with `UNSUPPORTED_ARTIFACT`
|
||
before any test starts.
|
||
|
||
[Recording](/reference/config#recording) has every mode and file.
|
||
|
||
## Step through a test
|
||
|
||
`--step` runs the first step, then holds each later step until you press a
|
||
key. Add `--headed` to watch the browser between steps.
|
||
|
||
<CodeGroup>
|
||
```bash npm
|
||
npx e2e run tests/checkout.e2e.ts --step --headed
|
||
```
|
||
|
||
```bash pnpm
|
||
pnpm exec e2e run tests/checkout.e2e.ts --step --headed
|
||
```
|
||
|
||
```bash bun
|
||
bunx e2e run tests/checkout.e2e.ts --step --headed
|
||
```
|
||
</CodeGroup>
|
||
|
||
The held step shows under its test, below the steps that already ran:
|
||
|
||
```text
|
||
❯ |web| tests/checkout.e2e.ts 0/1
|
||
└── pays with a card (k3f9)
|
||
✓ app.open "/checkout" 412ms
|
||
▶ step 2 locator.fill "getByLabel("Email")" tests/checkout.e2e.ts:9
|
||
press enter to run this step, press h to show help, press q to quit
|
||
```
|
||
|
||
The keys follow [`--watch`](/reference/cli#watch-mode). These work at a prompt:
|
||
|
||
| Key | Does |
|
||
| --- | --- |
|
||
| Enter | runs the held step and holds the next one |
|
||
| `n` | runs the rest of this test, retries included, and holds at the next test |
|
||
| `a` | runs the rest of the run without holding steps |
|
||
| `h` | shows every key under the prompt, or hides them |
|
||
| `q`, Esc | stops the run as Ctrl-C does: the test ends `interrupted`, the rest are skipped, exit 130 |
|
||
|
||
- A key pressed within 150 ms of a prompt is ignored, so a key meant for the
|
||
last step never runs this one. Ctrl-C works at any time, also while a step
|
||
runs.
|
||
- The time a step is held counts toward nothing: not its duration, not the
|
||
test's `timeout`, not `maxDuration`. A test that hangs after a step still
|
||
times out on the time it had left. The test's own duration in the report
|
||
includes the holds.
|
||
- Setup tests, retries, `afterEach` hooks, fixture teardowns, and steps inside
|
||
an `expect.poll` read run without stopping. An `agent.act` is one step.
|
||
- Steps started together, as in `Promise.all`, share one prompt and run on one
|
||
Enter, so a `waitForResponse` beside the click that triggers it never waits
|
||
for a key. A step already running when another is held keeps running on its
|
||
own timeout.
|
||
- A hosted browser or device can end a session that sits idle too long while
|
||
a step is held.
|
||
|
||
`--step` runs one test at a time. It is refused in CI, with `--watch`, with
|
||
`--workers` above 1, and with the json reporter.
|
||
|
||
## Step through a test without a terminal
|
||
|
||
When stdin or stdout is not a terminal, as in a coding agent's shell,
|
||
`--step` runs the test in the background. The command returns at the first
|
||
held step, with the screen that step waits on. Run each later command from
|
||
the same directory. A shell that gives commands a terminal gets the key
|
||
prompt instead; redirect stdin from the null device (`< /dev/null`, or
|
||
`< NUL` in Windows `cmd`).
|
||
|
||
<CodeGroup>
|
||
```bash npm
|
||
npx e2e run tests/checkout.e2e.ts --step
|
||
npx e2e run --step next
|
||
```
|
||
|
||
```bash pnpm
|
||
pnpm exec e2e run tests/checkout.e2e.ts --step
|
||
pnpm exec e2e run --step next
|
||
```
|
||
|
||
```bash bun
|
||
bunx e2e run tests/checkout.e2e.ts --step
|
||
bunx e2e run --step next
|
||
```
|
||
</CodeGroup>
|
||
|
||
```text
|
||
✓ app.open "/checkout" 124ms
|
||
Paused (k3f9) before step 2 of pays with a card [web]:
|
||
locator.fill "getByLabel("Email")" (tests/checkout.e2e.ts:9)
|
||
Screen:
|
||
http://localhost:3000/checkout
|
||
heading "Checkout"
|
||
textbox "Email"
|
||
button "Pay"
|
||
Next: e2e run --step next to run it, e2e run --step rest to run the rest, or e2e run --step stop to end the run.
|
||
```
|
||
|
||
| Command | Does |
|
||
| --- | --- |
|
||
| `e2e run --step next` | runs the held step and returns at the next one |
|
||
| `e2e run --step rest` | runs the rest of the run without holding steps |
|
||
| `e2e run --step stop` | stops the run as `q` does, exit 130, even while another command waits |
|
||
|
||
- Each answer starts with what the run printed since the last command: a
|
||
line per step that ran, as the first line above, with the error under a
|
||
failed one. The last answer ends with the run summary and
|
||
`The stepped run has ended.`, and the command exits with the run's exit
|
||
code.
|
||
- The screen lists the app's nodes as a trace page does, with secrets
|
||
redacted.
|
||
- One directory has one stepped run. A second `--step` there exits 2 until
|
||
the first ends.
|
||
- `next` and `rest` only act on a step a command has printed. When a
|
||
command is killed while it waits, the next one prints what it missed: the
|
||
step, without running it, or the end of the run. A `next` or `rest` sent while a step runs waits for
|
||
the next held step without acting. Either way, the answer opens with a line
|
||
saying the action was not applied.
|
||
- A step held for 30 minutes with no command stops the run.
|
||
- With `--step`, a file or directory named `next`, `rest`, or `stop` needs
|
||
a path, such as `./next`.
|
||
|
||
## Pause a test
|
||
|
||
With [`screen.pause()`](/reference/screen#screen-pause), you can pause the
|
||
test at a line, with the app open, until you resume it. You can use it in
|
||
any `e2e run`, with or without [`--step`](#step-through-a-test). The test
|
||
is paused each time the line runs, such as on every pass of a loop.
|
||
|
||
```ts title="tests/checkout.e2e.ts"
|
||
import { test } from '@e2e-dev/web';
|
||
|
||
test('pays with a card', async ({ app, screen }) => {
|
||
await app.open('/checkout');
|
||
await screen.pause();
|
||
});
|
||
```
|
||
|
||
Add `--headed` to look at the browser while the test is paused:
|
||
|
||
<CodeGroup>
|
||
```bash npm
|
||
npx e2e run tests/checkout.e2e.ts --headed
|
||
```
|
||
|
||
```bash pnpm
|
||
pnpm exec e2e run tests/checkout.e2e.ts --headed
|
||
```
|
||
|
||
```bash bun
|
||
bunx e2e run tests/checkout.e2e.ts --headed
|
||
```
|
||
</CodeGroup>
|
||
|
||
In a terminal, the pause is shown under its test:
|
||
|
||
```text
|
||
❯ |web| tests/checkout.e2e.ts 0/1
|
||
└── pays with a card (k3f9)
|
||
▶ paused screen.pause() tests/checkout.e2e.ts:5
|
||
press enter to resume, press h to show help, press q to quit
|
||
```
|
||
|
||
Press Enter to resume the test, `h` to list the keys, or `q` to stop the run
|
||
with exit 130, as at a [`--step` prompt](#step-through-a-test).
|
||
|
||
Without a terminal, as in a coding agent's shell, the pause is printed with
|
||
the screen it waits on:
|
||
|
||
```text
|
||
Paused (k3f9) in pays with a card [web]:
|
||
screen.pause() (tests/checkout.e2e.ts:5)
|
||
Screen:
|
||
http://localhost:3000/checkout
|
||
heading "Checkout"
|
||
Next: e2e run --step next to resume the test, or e2e run --step stop to end the run.
|
||
```
|
||
|
||
To resume the test, run `e2e run --step next` from another shell in the
|
||
same directory. To end the run with exit 130, run `e2e run --step stop`. You
|
||
can use these commands with or without a terminal. At this pause, you can
|
||
send `rest` in place of `next`.
|
||
|
||
To [attach over MCP](/reference/mcp#attaching-to-a-held-test) from the same
|
||
directory, pass the ID from the pause header, as in
|
||
`open_session {attach: "k3f9"}`.
|
||
|
||
You get an answer once the test is paused again or the run ends. At the
|
||
end, you get `The run has ended.` and the run's exit code. See
|
||
[Step through a test without a terminal](#step-through-a-test-without-a-terminal)
|
||
for what each answer holds.
|
||
|
||
- Paused time doesn't count toward any timeout. But without a terminal, the
|
||
run is stopped after 30 minutes at a pause without a command.
|
||
- The test is paused on every attempt, retries and setup tests included.
|
||
- With several workers, pauses are shown one at a time, oldest first, while
|
||
the other workers keep running.
|
||
- Call `screen.pause()` with `await` in a
|
||
[`beforeEach`](/reference/test#beforeeach) hook,
|
||
[fixture](/fixtures) setup, or the test body. See
|
||
[`screen.pause()`](/reference/screen#screen-pause) for what happens
|
||
anywhere else.
|
||
- You can control one run per directory at a time with `e2e run --step`
|
||
commands. A `--step` run started without a terminal is controlled from its
|
||
start. A run without `--step` is controlled from its first pause. If
|
||
another run in the directory is already controlled this way:
|
||
- Without a terminal, the test fails at the pause with `RUNNER_BUSY`. End
|
||
the other run with `e2e run --step stop`, or run this one in a terminal.
|
||
- In a terminal, you can still use the keys. A line is printed to say
|
||
that the commands go to the other run.
|
||
- With `CI` set, the test fails at the pause with `PAUSE_UNAVAILABLE`.
|
||
Remove the call before you commit.
|
||
|
||
## Console output
|
||
|
||
`console.log` and `console.error` in a test print while it runs, under a
|
||
heading that names the stream, target, file, and test:
|
||
|
||
```text
|
||
stdout | |web| tests/todos.e2e.ts > adds a todo
|
||
hello from the test
|
||
```
|
||
|
||
Output from outside a test, such as a file's top level or a reporter,
|
||
prints under `stdout | runner` or `stderr | runner`. Both streams print
|
||
where `list` does: stdout, or stderr beside `json`. Without the `list`
|
||
reporter, or another reporter that prints to the terminal with
|
||
[`onOutput`](/reference/reporters#onoutput), output goes to its own stream
|
||
as written, except with the `json` reporter (`--reporter json` or in
|
||
`reporters`), which keeps stdout for the report and prints output on
|
||
stderr. Output from the config load prints before the
|
||
run header, or a warning replaces it if the load fails.
|
||
|
||
The runner [redacts](/reference/security#redaction) secret values. The
|
||
report doesn't include console output, so use `expect` or an artifact for
|
||
anything a later reader needs.
|
||
|
||
## Agent steps
|
||
|
||
These flags show what the agent did:
|
||
|
||
<CodeGroup>
|
||
```bash npm
|
||
npx e2e run --debug # timings and a step table on stderr
|
||
npx e2e run --ai-trace # every model call, for a trace viewer
|
||
npx e2e run --no-cache # run the agent live instead of replaying
|
||
```
|
||
|
||
```bash pnpm
|
||
pnpm exec e2e run --debug # timings and a step table on stderr
|
||
pnpm exec e2e run --ai-trace # every model call, for a trace viewer
|
||
pnpm exec e2e run --no-cache # run the agent live instead of replaying
|
||
```
|
||
|
||
```bash bun
|
||
bunx e2e run --debug # timings and a step table on stderr
|
||
bunx e2e run --ai-trace # every model call, for a trace viewer
|
||
bunx e2e run --no-cache # run the agent live instead of replaying
|
||
```
|
||
</CodeGroup>
|
||
|
||
`--debug` adds the run's phase timings and a table with one row per agent
|
||
step: duration, model calls, tokens, prompt-cache share, and cost when the
|
||
provider reports one. Each planned step also saves its model transcript as
|
||
an artifact.
|
||
|
||
```console
|
||
[e2e debug] agent steps (execution order, model openai/gpt-6-luna-fast, total $0.0312)
|
||
step total model observe action calls tokens in/out cached cost
|
||
act "add a todo named groceries" 3.9s 2.8s 410ms 620ms 3 9412/388 61% (5740) $0.0198
|
||
assert "the list shows groceries" 1.2s 1.1s 88ms 0ms 1 3120/41 0% (0) $0.0114
|
||
```
|
||
|
||
`--ai-trace` writes `.e2e/ai-trace.json`: one run per agent step, one entry
|
||
per model call, with the redacted input and image bytes replaced by counts.
|
||
Read it with [unbox-ai](https://github.com/e2e-inc/unbox-ai) to see
|
||
prompts, tool calls, and token usage:
|
||
|
||
<CodeGroup>
|
||
```bash npm
|
||
npx unbox-ai runs .e2e/ai-trace.json
|
||
npx unbox-ai summary .e2e/ai-trace.json --run 0
|
||
```
|
||
|
||
```bash pnpm
|
||
pnpm dlx unbox-ai runs .e2e/ai-trace.json
|
||
pnpm dlx unbox-ai summary .e2e/ai-trace.json --run 0
|
||
```
|
||
|
||
```bash bun
|
||
bunx unbox-ai runs .e2e/ai-trace.json
|
||
bunx unbox-ai summary .e2e/ai-trace.json --run 0
|
||
```
|
||
</CodeGroup>
|
||
|
||
A replayed step makes no model call and leaves no AI trace. Add `--no-cache`
|
||
to trace the whole flow or rule out a stale [recording](/cache).
|
||
|
||
## Common errors
|
||
|
||
| Code | Usual cause | Fix |
|
||
| --- | --- | --- |
|
||
| `APP_UNREACHABLE` | Nothing answers at `app.url`, or `app.command` never became ready. On iOS, the message may name the automation runner instead | Set `app.command.log` and read it; check the port; raise `app.command.startupTimeout`. For the iOS runner, follow the message (see [Mobile](/mobile#troubleshooting)) |
|
||
| `APP_ALREADY_RUNNING` | Something already serves `app.url` when the runner wanted to start `app.command` | Stop it, or set `reuseExisting: true` for local runs |
|
||
| `LOCATOR_NOT_FOUND` | Wrong role or name, text not exact, element inside an iframe, page not open | Read the markup for the accessible name; try `exact: false`; `browser.frameLocator` for iframes; `--headed` to look |
|
||
| `LOCATOR_AMBIGUOUS` | Two matches: a hidden duplicate, a repeated label | Add `{ name }`, scope under a container, `filter`, `first()`, or `{ visible: true }` |
|
||
| `ASSERTION_FAILED` | The expectation is wrong, or the state settles after 5 s; for `agent.assert`, the judgment was false | Compare with the screenshot; `{ timeout }` on the matcher; rewrite the question |
|
||
| `ASSERTION_INCONCLUSIVE` | `agent.assert` or `agent.extract` asked about something the screen doesn't show: on another page, still loading, or only in pixels | Open or wait for the right screen first; add `vision: true` when the answer is in pixels (the message says so) |
|
||
| `ACTION_FAILED` | Element covered, disabled, or detached, or an operation timed out | Wait on the right condition with `expect` first; close overlays |
|
||
| `STEP_BUDGET_EXHAUSTED`, `STEP_TIMEOUT` | The agent ran out of actions, model calls, or time | Scope the goal to one step. Raise the agent entry's `maxSteps` or `maxModelCalls` for the budget, the config `timeout` for an `act` step, or the agent's `judgmentTimeout` for `assert`, `waitFor`, and `extract` |
|
||
| `MODEL_UNAVAILABLE` | No model constructed in the config, or GitHub Copilot doesn't serve the configured model to your account | See [Models](/models); for Copilot, [Pick a model](/subscriptions#pick-a-model) |
|
||
| `MODEL_PROVIDER_FAILED` | Bad key, rate limit, no credits, or every try of one request went 120 s without an answer | Check the key the provider reads and the quota; for a slow local model, a smaller screen (`agents.<name>.maxObservationBytes`) or a faster model |
|
||
| `MODEL_REFUSED` | The model refused the prompt (finish reason `content-filter`). Exit 1 | Rephrase the instruction, or switch to another model |
|
||
|
||
The [errors reference](/reference/errors) lists every code, its class, and
|
||
its exit code.
|
||
|
||
## Next
|
||
|
||
<CardGroup cols={2}>
|
||
<Card title="CLI reference" icon="terminal" href="/reference/cli#e2e-run">
|
||
Every flag of `e2e run`.
|
||
</Card>
|
||
<Card title="Errors" icon="triangle-exclamation" href="/reference/errors">
|
||
Every error code.
|
||
</Card>
|
||
</CardGroup>
|