mirror of
https://github.com/tester-army/e2e.git
synced 2026-11-22 05:00:35 +08:00
468 lines
20 KiB
Plaintext
468 lines
20 KiB
Plaintext
---
|
||
title: Debugging a run
|
||
description: Read the trace page, open the evidence, watch the browser, and see what the model was sent.
|
||
"og:image": "https://e2e.com/docs/images/og/debugging.png?v=38a51086e1"
|
||
"twitter:image": "https://e2e.com/docs/images/og/debugging.png?v=38a51086e1"
|
||
---
|
||
|
||
A failed test prints its error and the path to its [trace page](#traces). For
|
||
a summary of the whole run, add the `markdown` reporter. It writes
|
||
`.e2e/summary.md`.
|
||
|
||
## Read the failure
|
||
|
||
The `list` reporter's `Failed Tests` section shows the error, the failing
|
||
line, nearby matches for a failed locator, what the app logged, and the
|
||
trace page. It quotes the app's first 3 errors (console errors, uncaught
|
||
exceptions, failed requests, 5xx responses), redacted like the trace page.
|
||
`.e2e/summary.md` quotes them too:
|
||
|
||
```text
|
||
FAIL |web| tests/todos.e2e.ts > shows the remaining count
|
||
LOCATOR_NOT_FOUND: locator did not become visible: getByRole("heading", name: "Todos")
|
||
❯ tests/todos.e2e.ts:31:3
|
||
❯ at http://127.0.0.1:4271/nope
|
||
❯ app 2 errors, 2 warnings logged
|
||
× console error: Error occurred prerendering page "/todos"
|
||
× network error: GET /api/todos 500 Internal Server Error
|
||
❯ trace .e2e/results/todos-shows-the-remaining-count-796fac5f12ab34cd/trace.md
|
||
```
|
||
|
||
The exit code tells the type of failure:
|
||
|
||
| Exit | Meaning | Retry the job? |
|
||
| ---: | --- | --- |
|
||
| 1 | A test failed | Only if the test is flaky |
|
||
| 2 | Configuration, collection, or policy | Fix the reported problem first |
|
||
| 3 | Engine, app process, model provider, artifact, or cleanup infrastructure failure | If the cause is temporary |
|
||
| 4 | An internal runner error | Report it |
|
||
| 130 | The run was interrupted | |
|
||
|
||
## Traces
|
||
|
||
By default, each failed, timed out, interrupted, or flaky test gets a trace
|
||
page, `trace.md`, in its [results directory](#artifacts). Open it in an
|
||
editor's markdown preview so its links work. After the browser or device
|
||
the test ran on, it shows:
|
||
|
||
- **Steps.** The node an action landed on, the values an assertion read
|
||
while it waited, the replay cache's decision, what the app logged, where
|
||
the page went, and how the screen changed since the step before.
|
||
- **App log.** Console output, uncaught exceptions, failed requests, and
|
||
4xx and 5xx responses, each with its step: up to 50 lines, errors and
|
||
warnings first. An attempt keeps at most 200 errors and warnings and 100
|
||
other lines.
|
||
- **The agent's last turns.**
|
||
- **Screen at failure.** The screenshot and the semantic tree, captured in
|
||
at most 5 s. An app that doesn't answer in time gets no screen section.
|
||
|
||
<Accordion title="Example trace page">
|
||
|
||
````markdown
|
||
# ✗ todos › shows the remaining count
|
||
|
||
`tests/todos.e2e.ts` · failed · 2.2s
|
||
|
||
**ASSERTION_FAILED**
|
||
|
||
```text
|
||
expect.toHaveText failed
|
||
locator: getByRole("status", name: "Remaining")
|
||
expected: text "1 remaining"
|
||
observed: text "0 remaining" (match count 1)
|
||
```
|
||
|
||
Ran on: browser `chromium 141.0.7390.37` · user agent `Mozilla/5.0 (Macintosh; …) HeadlessChrome/141.0.7390.37 Safari/537.36`
|
||
|
||
Look at: `tests/todos.e2e.ts:15`
|
||
|
||
## Steps
|
||
|
||
1. ✓ `app.open` `/todos` (37ms in beforeEach) — `tests/todos.e2e.ts:4`
|
||
2. ✓ `agent.act` "Add a todo named Milk." (429ms, 2 model calls) — `tests/todos.e2e.ts:12`
|
||
- cache: no replay: nothing recorded for this step yet; recording saved (`.e2e/cache/1227d5a6….json`)
|
||
- type "Milk" into textbox "New todo" (17ms)
|
||
- press "Enter" on the focused field (5ms)
|
||
- ✗ network error: `POST /api/todos 500 Internal Server Error`
|
||
- screen: 6 changes since step 1 at `/todos`, 38 nodes
|
||
- `added listitem "Milk Delete Milk" testid="todo"`
|
||
- `added checkbox "Milk"`
|
||
- `changed status "Remaining" text="1 remaining" (was: status "Remaining" text="0 remaining")`
|
||
- 3 more
|
||
3. ✓ `locator.check` `getByLabel("Milk")` (13ms) — `tests/todos.e2e.ts:14`
|
||
- check checkbox "Milk" (13ms)
|
||
- screen: 3 changes since step 2 at `/todos`, 38 nodes
|
||
- `added checkbox "Milk" [checked]`
|
||
- `changed status "Remaining" text="0 remaining" (was: status "Remaining" text="1 remaining")`
|
||
- `removed checkbox "Milk"`
|
||
4. ✗ `expect.toHaveText` `getByRole("status", name: "Remaining")` (1.6s) — **ASSERTION_FAILED** — `tests/todos.e2e.ts:15`
|
||
- expect gave up after 16 reads in 1.6s: text "0 remaining" (1 match) x16
|
||
|
||
## App log
|
||
|
||
What the app logged, oldest first, with the step it happened in.
|
||
|
||
- step 2 · ✗ network error: `POST /api/todos 500 Internal Server Error`
|
||
|
||
## Screen at failure
|
||
`http://127.0.0.1:4271/todos` · viewport 1280x720 · 38 nodes
|
||
|
||

|
||
|
||
```text
|
||
#n29 textbox "New todo"
|
||
#n30 button "Add"
|
||
#n35 listitem "Milk Delete Milk" testid="todo"
|
||
#n36 checkbox "Milk" [checked]
|
||
#n40 status "Remaining" text="0 remaining"
|
||
```
|
||
````
|
||
|
||
</Accordion>
|
||
|
||
An assertion that read one value the whole time has a wrong expectation, or
|
||
the app never reached that state: look in the app log for a failed request.
|
||
An assertion whose readings changed (`"2 remaining" x3 -> at 410ms "1
|
||
remaining"`) raced the app. `screen: unchanged` after an action means the
|
||
action had no visible effect. On web, a step's screen is the one an action
|
||
left. On a device, it's the screen the step acted on.
|
||
|
||
The [`trace`](/reference/config#trace) mode sets which tests keep a page.
|
||
`--trace` keeps one for every test of a run, passing ones included.
|
||
`--trace off` writes none. A test's own `trace` option wins over both.
|
||
|
||
## Artifacts
|
||
|
||
Each test gets one directory under `.e2e/results/` (or
|
||
[`<output>`](/reference/config#output)`/results/`), with its trace page and
|
||
one `attempt-<n>` per attempt. A serial group's tests share one directory:
|
||
|
||
```text
|
||
.e2e/results/todos-adds-a-todo-1a2b3c4d5e6f7a8b/trace.md
|
||
.e2e/results/todos-adds-a-todo-1a2b3c4d5e6f7a8b/attempt-1/screen-at-failure.txt
|
||
.e2e/results/todos-adds-a-todo-1a2b3c4d5e6f7a8b/attempt-1/screenshots/001-failure.png
|
||
```
|
||
|
||
Each run that reaches its tests empties `results/` first. A `--last-failed`
|
||
rerun keeps the files that the report it reruns names, and writes its own
|
||
attempts under each test's `rerun-<n>/`.
|
||
|
||
| Artifact | When |
|
||
| --- | --- |
|
||
| Screenshots | A failed attempt captures one when the engine can. The built-in `agent.assert` captures one after either verdict, unless `screenshot: false`. `app.screenshot('label')` captures one on request. They mask secure fields only. See [screenshots after a secret fill](/reference/security#screenshots-after-a-secret-fill). |
|
||
| Downloads | Files saved through `browser.waitForDownload`, kind `download`, as the app served them. [Downloads](/reference/security#downloads) says when the runner masks secrets in them. |
|
||
| Videos | When you [record them](#watch-it-happen). Videos mask nothing. See [where redaction stops](/reference/security#where-redaction-stops). |
|
||
| Agent transcripts | With [`--debug`](#agent-steps). |
|
||
|
||
## For scripts: report.json
|
||
|
||
Every run that reaches its tests also writes `report.json` to the
|
||
[output directory](/reference/config#output), `.e2e` by default. It holds
|
||
every result of the run in a versioned schema. `--reporter json` prints it to
|
||
stdout:
|
||
|
||
```bash
|
||
jq '.run | {status, exitCode, errors}' .e2e/report.json
|
||
jq '.run.results[] | select(.status != "passed") | .attempts[-1]
|
||
| {status, error, failure, steps: [.steps[] | select(.status != "passed") | {api, label, source, status, error}], artifacts}' .e2e/report.json
|
||
```
|
||
|
||
`run.errors` holds run-level failures. The last attempt of each failed
|
||
result holds:
|
||
|
||
| Field | Holds |
|
||
| --- | --- |
|
||
| `error` | The code, message, source line, and assertion details |
|
||
| `failure` | The screen and screenshot artifacts, and any locator candidates |
|
||
| `steps` | The calls that led to the failure. Each step's `events` hold what it did (`engine`), what it polled (`poll`), and where the app went (`navigation`). Agent steps add `cache` and recent model turns. `observationTruncated: true` means at least one semantic observation was incomplete. |
|
||
| `appLog` | What the app logged. Each line's `step` is the `index` of its step, absent before the first step. |
|
||
| `artifacts` | The captured files |
|
||
|
||
A serial group's tests have no `attempts` of their own. Their results are in
|
||
`run.serialGroups`, one member for each test in each attempt. A retry the run
|
||
interrupted or that never reached its members has `cutRetry: true` and does
|
||
not hold the verdicts, so skip it:
|
||
|
||
```bash
|
||
jq '.run.serialGroups[] | (.attempts | map(select(.cutRetry != true)) | last).members[]? | select(.status != "passed")
|
||
| {testId, status, error, failure}' .e2e/report.json
|
||
```
|
||
|
||
## Watch it happen
|
||
|
||
<CodeGroup>
|
||
```bash npm
|
||
npx e2e run --headed # a visible browser
|
||
npx e2e run --video # record every attempt
|
||
npx e2e run --video=retain-on-failure # record every attempt, keep the failures
|
||
```
|
||
|
||
```bash pnpm
|
||
pnpm exec e2e run --headed # a visible browser
|
||
pnpm exec e2e run --video # record every attempt
|
||
pnpm exec e2e run --video=retain-on-failure # record every attempt, keep the failures
|
||
```
|
||
|
||
```bash bun
|
||
bunx e2e run --headed # a visible browser
|
||
bunx e2e run --video # record every attempt
|
||
bunx e2e run --video=retain-on-failure # record every attempt, keep the failures
|
||
```
|
||
</CodeGroup>
|
||
|
||
`--video` takes an optional mode. Put test files before it, or write
|
||
`--video=<mode>`.
|
||
|
||
Each recording goes into its attempt's directory: WebM per page on web, MP4
|
||
on a device. The failure recap prints the path, or the URL when a hosted
|
||
browser keeps the recording. Subtract the recording's `startedAt` from a
|
||
step's `startedAt` to find that step in the video.
|
||
|
||
In CI, set `video: 'on-first-retry'` in the config. A test that passes the
|
||
first time records nothing, and a flaky test leaves a recording of its
|
||
retry.
|
||
|
||
A target's `video` wins over the config's, and `--video` wins over both. A
|
||
test's own option, such as `test('checkout', { video: 'on' }, ...)`, wins
|
||
over all of them.
|
||
|
||
The config's `video` and `--video` apply only to targets whose engine can
|
||
record, and one notice names the others. When a target or a test asks for a
|
||
recording its engine can't make, the run fails with `UNSUPPORTED_ARTIFACT`
|
||
before any test starts.
|
||
|
||
[Recording](/reference/config#recording) has every mode and file.
|
||
|
||
## Step through a test
|
||
|
||
`--step` runs the first step, then holds each later step until you press a
|
||
key. Add `--headed` to watch the browser between steps.
|
||
|
||
<CodeGroup>
|
||
```bash npm
|
||
npx e2e run tests/checkout.e2e.ts --step --headed
|
||
```
|
||
|
||
```bash pnpm
|
||
pnpm exec e2e run tests/checkout.e2e.ts --step --headed
|
||
```
|
||
|
||
```bash bun
|
||
bunx e2e run tests/checkout.e2e.ts --step --headed
|
||
```
|
||
</CodeGroup>
|
||
|
||
The held step shows under its test, below the steps that already ran:
|
||
|
||
```text
|
||
❯ |web| tests/checkout.e2e.ts 0/1
|
||
└── pays with a card paused
|
||
✓ app.open "/checkout" 412ms
|
||
▶ step 2 locator.fill "getByLabel("Email")" tests/checkout.e2e.ts:9
|
||
press enter to run this step, press h to show help, press q to quit
|
||
```
|
||
|
||
The keys follow [`--watch`](/reference/cli#watch-mode). These work at a prompt:
|
||
|
||
| Key | Does |
|
||
| --- | --- |
|
||
| Enter | runs the held step and holds the next one |
|
||
| `n` | runs the rest of this test, retries included, and holds at the next test |
|
||
| `a` | runs the rest of the run without stopping |
|
||
| `h` | shows every key under the prompt, or hides them |
|
||
| `q`, Esc | stops the run as Ctrl-C does: the test ends `interrupted`, the rest are skipped, exit 130 |
|
||
|
||
- A key pressed within 150 ms of a prompt is ignored, so a key meant for the
|
||
last step never runs this one. Ctrl-C works at any time, also while a step
|
||
runs.
|
||
- The time a step is held counts toward nothing: not its duration, not the
|
||
test's `timeout`, not `maxDuration`. A test that hangs after a step still
|
||
times out on the time it had left. The test's own duration in the report
|
||
includes the holds.
|
||
- Setup tests, retries, `afterEach` hooks, fixture teardowns, and steps inside
|
||
an `expect.poll` read run without stopping. An `agent.act` is one step.
|
||
- Steps started together, as in `Promise.all`, share one prompt and run on one
|
||
Enter, so a `waitForResponse` beside the click that triggers it never waits
|
||
for a key. A step already running when another is held keeps running on its
|
||
own timeout.
|
||
- A hosted browser or device can end a session that sits idle too long while
|
||
a step is held.
|
||
|
||
`--step` runs one test at a time. It is refused in CI, with `--watch`, with
|
||
`--workers` above 1, and with the json reporter.
|
||
|
||
## Step through a test without a terminal
|
||
|
||
When stdin or stdout is not a terminal, as in a coding agent's shell,
|
||
`--step` runs the test in the background. The command returns at the first
|
||
held step, with the screen that step waits on. Run each later command from
|
||
the same directory. A shell that gives commands a terminal gets the key
|
||
prompt instead; redirect stdin from the null device (`< /dev/null`, or
|
||
`< NUL` in Windows `cmd`).
|
||
|
||
<CodeGroup>
|
||
```bash npm
|
||
npx e2e run tests/checkout.e2e.ts --step
|
||
npx e2e run --step next
|
||
```
|
||
|
||
```bash pnpm
|
||
pnpm exec e2e run tests/checkout.e2e.ts --step
|
||
pnpm exec e2e run --step next
|
||
```
|
||
|
||
```bash bun
|
||
bunx e2e run tests/checkout.e2e.ts --step
|
||
bunx e2e run --step next
|
||
```
|
||
</CodeGroup>
|
||
|
||
```text
|
||
✓ app.open "/checkout" 124ms
|
||
Paused before step 2 of pays with a card [web]:
|
||
locator.fill "getByLabel("Email")" (tests/checkout.e2e.ts:9)
|
||
Screen:
|
||
http://localhost:3000/checkout
|
||
heading "Checkout"
|
||
textbox "Email"
|
||
button "Pay"
|
||
Next: e2e run --step next runs it, e2e run --step rest runs the rest, e2e run --step stop ends the run.
|
||
```
|
||
|
||
| Command | Does |
|
||
| --- | --- |
|
||
| `e2e run --step next` | runs the held step and returns at the next one |
|
||
| `e2e run --step rest` | runs the rest of the run without stopping |
|
||
| `e2e run --step stop` | stops the run as `q` does, exit 130, even while another command waits |
|
||
|
||
- Each answer starts with what the run printed since the last command: a
|
||
line per step that ran, as the first line above, with the error under a
|
||
failed one. The last answer ends with the run summary and
|
||
`The stepped run has ended.`, and the command exits with the run's exit
|
||
code.
|
||
- The screen lists the app's nodes as a trace page does, with secrets
|
||
redacted.
|
||
- One directory has one stepped run. A second `--step` there exits 2 until
|
||
the first ends.
|
||
- `next` and `rest` only act on a step a command has printed. When a
|
||
command is killed while it waits, the next one prints what it missed: the
|
||
step, without running it, or the end of the run. A `next` or `rest` sent while a step runs waits for
|
||
the next pause without acting. Either way, the answer opens with a line
|
||
saying the action was not applied.
|
||
- A step held for 30 minutes with no command stops the run.
|
||
- With `--step`, a file or directory named `next`, `rest`, or `stop` needs
|
||
a path, such as `./next`.
|
||
|
||
## Console output
|
||
|
||
`console.log` and `console.error` in a test print while it runs, under a
|
||
heading that names the stream, target, file, and test:
|
||
|
||
```text
|
||
stdout | |web| tests/todos.e2e.ts > adds a todo
|
||
hello from the test
|
||
```
|
||
|
||
Output from outside a test, such as a file's top level or a reporter,
|
||
prints under `stdout | runner` or `stderr | runner`. Both streams print on
|
||
stdout. Without the `list` reporter, output goes to its own stream as
|
||
written, except with the `json` reporter (`--reporter json` or in
|
||
`reporters`), which keeps stdout for the report and prints output on
|
||
stderr. Output from the config load prints before the
|
||
run header, or a warning replaces it if the load fails.
|
||
|
||
The runner [redacts](/reference/security#redaction) secret values. The
|
||
report doesn't include console output, so use `expect` or an artifact for
|
||
anything a later reader needs.
|
||
|
||
## Agent steps
|
||
|
||
These flags show what the agent did:
|
||
|
||
<CodeGroup>
|
||
```bash npm
|
||
npx e2e run --debug # timings and a step table on stderr
|
||
npx e2e run --ai-trace # every model call, for a trace viewer
|
||
npx e2e run --no-cache # run the agent live instead of replaying
|
||
```
|
||
|
||
```bash pnpm
|
||
pnpm exec e2e run --debug # timings and a step table on stderr
|
||
pnpm exec e2e run --ai-trace # every model call, for a trace viewer
|
||
pnpm exec e2e run --no-cache # run the agent live instead of replaying
|
||
```
|
||
|
||
```bash bun
|
||
bunx e2e run --debug # timings and a step table on stderr
|
||
bunx e2e run --ai-trace # every model call, for a trace viewer
|
||
bunx e2e run --no-cache # run the agent live instead of replaying
|
||
```
|
||
</CodeGroup>
|
||
|
||
`--debug` adds the run's phase timings and a table with one row per agent
|
||
step: duration, model calls, tokens, prompt-cache share, and cost when the
|
||
provider reports one. Each planned step also saves its model transcript as
|
||
an artifact.
|
||
|
||
```console
|
||
[e2e debug] agent steps (execution order, model openai/gpt-6-luna-fast, total $0.0312)
|
||
step total model observe action calls tokens in/out cached cost
|
||
act "add a todo named groceries" 3.9s 2.8s 410ms 620ms 3 9412/388 61% (5740) $0.0198
|
||
assert "the list shows groceries" 1.2s 1.1s 88ms 0ms 1 3120/41 0% (0) $0.0114
|
||
```
|
||
|
||
`--ai-trace` writes `.e2e/ai-trace.json`: one run per agent step, one entry
|
||
per model call, with the redacted input and image bytes replaced by counts.
|
||
Read it with [unbox-ai](https://github.com/e2e-inc/unbox-ai) to see
|
||
prompts, tool calls, and token usage:
|
||
|
||
<CodeGroup>
|
||
```bash npm
|
||
npx unbox-ai runs .e2e/ai-trace.json
|
||
npx unbox-ai summary .e2e/ai-trace.json --run 0
|
||
```
|
||
|
||
```bash pnpm
|
||
pnpm dlx unbox-ai runs .e2e/ai-trace.json
|
||
pnpm dlx unbox-ai summary .e2e/ai-trace.json --run 0
|
||
```
|
||
|
||
```bash bun
|
||
bunx unbox-ai runs .e2e/ai-trace.json
|
||
bunx unbox-ai summary .e2e/ai-trace.json --run 0
|
||
```
|
||
</CodeGroup>
|
||
|
||
A replayed step makes no model call and leaves no AI trace. Add `--no-cache`
|
||
to trace the whole flow or rule out a stale [recording](/cache).
|
||
|
||
## Common errors
|
||
|
||
| Code | Usual cause | Fix |
|
||
| --- | --- | --- |
|
||
| `APP_UNREACHABLE` | Nothing answers at `app.url`, or `app.command` never became ready. On iOS, the message may name the automation runner instead | Set `app.command.log` and read it; check the port; raise `app.command.startupTimeout`. For the iOS runner, follow the message (see [Mobile](/mobile#troubleshooting)) |
|
||
| `APP_ALREADY_RUNNING` | Something already serves `app.url` when the runner wanted to start `app.command` | Stop it, or set `reuseExisting: true` for local runs |
|
||
| `LOCATOR_NOT_FOUND` | Wrong role or name, text not exact, element inside an iframe, page not open | Read the markup for the accessible name; try `exact: false`; `browser.frameLocator` for iframes; `--headed` to look |
|
||
| `LOCATOR_AMBIGUOUS` | Two matches: a hidden duplicate, a repeated label | Add `{ name }`, scope under a container, `filter`, `first()`, or `{ visible: true }` |
|
||
| `ASSERTION_FAILED` | The expectation is wrong, or the state settles after 5 s; for `agent.assert`, the judgment was false | Compare with the screenshot; `{ timeout }` on the matcher; rewrite the question |
|
||
| `ASSERTION_INCONCLUSIVE` | `agent.assert` or `agent.extract` asked about something the screen doesn't show: on another page, still loading, or only in pixels | Open or wait for the right screen first; add `vision: true` when the answer is in pixels (the message says so) |
|
||
| `ACTION_FAILED` | Element covered, disabled, or detached, or an operation timed out | Wait on the right condition with `expect` first; close overlays |
|
||
| `STEP_BUDGET_EXHAUSTED`, `STEP_TIMEOUT` | The agent ran out of actions, model calls, or time | Scope the goal to one step. Raise the agent entry's `maxSteps` or `maxModelCalls` for the budget, the config `timeout` for an `act` step, or the agent's `judgmentTimeout` for `assert`, `waitFor`, and `extract` |
|
||
| `MODEL_UNAVAILABLE` | No model constructed in the config, or GitHub Copilot doesn't serve the configured model to your account | See [Models](/models); for Copilot, [Pick a model](/subscriptions#pick-a-model) |
|
||
| `MODEL_PROVIDER_FAILED` | Bad key, rate limit, no credits, or every try of one request went 120 s without an answer | Check the key the provider reads and the quota; for a slow local model, a smaller screen (`agents.<name>.maxObservationBytes`) or a faster model |
|
||
| `MODEL_REFUSED` | The model refused the prompt (finish reason `content-filter`). Exit 1 | Rephrase the instruction, or switch to another model |
|
||
|
||
The [errors reference](/reference/errors) lists every code, its class, and
|
||
its exit code.
|
||
|
||
## Next
|
||
|
||
<CardGroup cols={2}>
|
||
<Card title="CLI reference" icon="terminal" href="/reference/cli#e2e-run">
|
||
Every flag of `e2e run`.
|
||
</Card>
|
||
<Card title="Errors" icon="triangle-exclamation" href="/reference/errors">
|
||
Every error code.
|
||
</Card>
|
||
</CardGroup>
|