mirror of
https://github.com/tester-army/e2e.git
synced 2026-11-22 18:19:48 +08:00
* docs(ci): rename the Vercel page to Vercel previews * docs(ci): move the Vercel previews page to /ci/vercel-previews * docs(ci): capitalize Vercel Previews * docs(ci): drop the redirect, the page is not shipped
294 lines
12 KiB
Plaintext
294 lines
12 KiB
Plaintext
---
|
|
title: Continuous integration
|
|
sidebarTitle: Setting up CI
|
|
description: Run e2e on GitHub Actions, Vercel deployments, GitLab CI/CD, Expo Workflows, Bitrise, Codemagic, Norrix, or any CI service that runs Node.
|
|
"og:image": "https://e2e.com/docs/images/og/ci.png?v=01be716140"
|
|
"twitter:image": "https://e2e.com/docs/images/og/ci.png?v=01be716140"
|
|
---
|
|
|
|
e2e runs on each CI service that can run Node 24.8 or later (22.22.3 or
|
|
later on Node 22), or [Bun](/reference/cli#bun) 1.4.3 or later for web tests.
|
|
Each job installs the dependencies, starts the app or a device, runs the
|
|
tests, and uploads `.e2e/`.
|
|
|
|
## Pick your CI service
|
|
|
|
<CardGroup cols={2}>
|
|
<Card title="GitHub Actions" icon="/images/integrations/github-actions.svg" href="/ci/github-actions">
|
|
Web and mobile workflows. Post the results as a pull request comment.
|
|
</Card>
|
|
<Card title="Vercel Previews" icon="/images/integrations/vercel.svg" href="/ci/vercel-previews">
|
|
Test each deployment and hold production releases until the tests pass.
|
|
</Card>
|
|
<Card title="GitLab CI/CD" icon="/images/integrations/gitlab.svg" href="/ci/gitlab">
|
|
Web and mobile jobs. Show the results in the merge request.
|
|
</Card>
|
|
<Card title="Expo Workflows" icon="/images/integrations/expo.svg" href="/ci/eas-workflows">
|
|
Build an Expo app with Expo Build, then run iOS and Android tests on it.
|
|
</Card>
|
|
<Card title="Bitrise" icon="/images/integrations/bitrise.svg" href="/ci/bitrise">
|
|
iOS and Android workflows with the Bitrise steps for simulators and emulators.
|
|
</Card>
|
|
<Card title="Codemagic" icon="/images/integrations/codemagic.svg" href="/ci/codemagic">
|
|
iOS and Android workflows on Mac mini and Linux instances.
|
|
</Card>
|
|
<Card title="Norrix" icon="/images/integrations/norrix.png" href="/ci/norrix">
|
|
Build a NativeScript app with Norrix, then run its suite on a Norrix simulator or emulator.
|
|
</Card>
|
|
</CardGroup>
|
|
|
|
For another CI service, copy the [GitHub Actions](/ci/github-actions)
|
|
workflow. To lease browsers or devices from a hosted service, see
|
|
[Hosted browsers and devices](/integrations).
|
|
|
|
## Upload the reports
|
|
|
|
Upload `.e2e/` at the end of each job, also when a test fails. It holds:
|
|
|
|
| Path | What it holds |
|
|
| --- | --- |
|
|
| `report.json` | The run report, from each run that reaches its tests |
|
|
| `junit.xml` | With the `junit` reporter (`--reporter list,junit`): the results, the tests a [test filter](/filters) left out as skipped, and startup failures such as `APP_UNREACHABLE` in a `run` suite |
|
|
| `results/` | The trace pages, screenshots, and screen dumps of each test. With `--video`, also a video for each target whose engine can record |
|
|
|
|
Point the test-report integration of your CI service at `.e2e/junit.xml`.
|
|
On GitLab, `junit.xml` also links each failure screenshot, so it shows in the test report.
|
|
The [`output`](/reference/config#output) option moves `.e2e/`.
|
|
|
|
## Model credentials
|
|
|
|
Give the job the credential your provider needs, such as
|
|
`AI_GATEWAY_API_KEY` (see [Models](/models)). Tests that never use `agent`
|
|
need none.
|
|
|
|
<Warning>
|
|
Each job that has the key gives it to the code it runs: tests, config, and
|
|
dependencies. Give the key only to builds of branches you trust. Run
|
|
untrusted pull request code with no secrets and no write tokens. See
|
|
[Security](/security).
|
|
</Warning>
|
|
|
|
Keep keys in CI secrets, and set spending limits where you can. GitHub does
|
|
not give secrets to pull requests from forks.
|
|
|
|
## What CI changes
|
|
|
|
CI mode is on when `CI` has a value other than an empty string, `0`, or
|
|
`false`. To get these defaults locally, run `CI=1 npx e2e run`:
|
|
|
|
| Setting | Local | CI |
|
|
| --- | --- | --- |
|
|
| `retries` | `0` | `1` |
|
|
| `workers` | half your cores | `1` |
|
|
| `cache` when unset | `read-write` | `read-only` |
|
|
| `.only` | allowed | rejected with `ONLY_IN_CI` |
|
|
| `command.reuseExisting` | honored | ignored, with a notice |
|
|
|
|
To change the worker and retry defaults, use flags:
|
|
|
|
<CodeGroup>
|
|
```bash npm
|
|
npx e2e run --workers 4 --retries 0
|
|
```
|
|
|
|
```bash pnpm
|
|
pnpm exec e2e run --workers 4 --retries 0
|
|
```
|
|
|
|
```bash bun
|
|
bunx e2e run --workers 4 --retries 0
|
|
```
|
|
</CodeGroup>
|
|
|
|
To send no [telemetry](/telemetry#in-ci) from CI, add
|
|
`E2E_TELEMETRY_DISABLED: '1'` to the `env` of the job.
|
|
|
|
## Mobile jobs
|
|
|
|
Without a [hosted device service](/mobile#hosted-devices), each platform
|
|
needs its own machine, build, and device:
|
|
|
|
| | iOS | Android |
|
|
| --- | --- | --- |
|
|
| Machine | macOS with Xcode and a simulator runtime | Linux with KVM (nested virtualization) |
|
|
| Build | Release simulator `.app`, no signing | Release `.apk` |
|
|
| Device | A booted simulator | A booted emulator |
|
|
|
|
Make a Release build on both platforms. A React Native or Expo debug build
|
|
loads its JavaScript from Metro, which these jobs don't start, so it opens
|
|
on an error screen. The examples use a React Native project with `ios/` and
|
|
`android/` in the repository. For an Expo project without them, run
|
|
`npx expo prebuild` first.
|
|
|
|
Install the build, then run the suite:
|
|
|
|
<CodeGroup>
|
|
```bash npm
|
|
# iOS
|
|
xcrun simctl install booted path/to/MyApp.app
|
|
npx e2e run --target ios --reporter list,junit
|
|
|
|
# Android
|
|
adb install -r path/to/app.apk
|
|
npx e2e run --target android --reporter list,junit
|
|
```
|
|
|
|
```bash pnpm
|
|
# iOS
|
|
xcrun simctl install booted path/to/MyApp.app
|
|
pnpm exec e2e run --target ios --reporter list,junit
|
|
|
|
# Android
|
|
adb install -r path/to/app.apk
|
|
pnpm exec e2e run --target android --reporter list,junit
|
|
```
|
|
|
|
```bash bun
|
|
# iOS
|
|
xcrun simctl install booted path/to/MyApp.app
|
|
bunx e2e run --target ios --reporter list,junit
|
|
|
|
# Android
|
|
adb install -r path/to/app.apk
|
|
bunx e2e run --target android --reporter list,junit
|
|
```
|
|
</CodeGroup>
|
|
|
|
The commands assume two targets named `ios` and `android`, each a
|
|
`mobile()` engine with the app's `bundleId`. See
|
|
[Point at your app](/mobile#point-at-your-app). To let the suite install the
|
|
build, set `app.appPath` and call `device.installApp()`.
|
|
|
|
These rules apply on each service:
|
|
|
|
- **One job for each platform.** Run `--target ios` on a Mac and
|
|
`--target android` on Linux with KVM. Hosted Macs use Apple Silicon,
|
|
which can't nest virtualization, so the Android emulator doesn't start
|
|
on them.
|
|
- **Boot the device in its own step.** With no device booted, the engine
|
|
asks agent-device to boot one, but a new Linux runner has no emulator
|
|
image. A separate step also tells a failed boot from a failed test.
|
|
- **Pin the iOS runtime.** A simulator picked by name (`iPhone 17`) runs
|
|
the image's default iOS, and the accessibility tree changes between iOS
|
|
releases. A runner image update can then fail a locator assertion with
|
|
no change in your code. The examples create a simulator on a named
|
|
runtime and boot it by UDID. Replace `iOS-26-5` with a runtime the image
|
|
has (`xcrun simctl list runtimes`).
|
|
- **Cache `~/.agent-device/apple-runner/derived`.** agent-device builds
|
|
its XCUITest runner there the first time it drives a simulator. Don't
|
|
cache all of `~/.agent-device`. The rest is the state of the last
|
|
daemon, and a restored copy can stop the next run with
|
|
`Failed to communicate with daemon`.
|
|
- **Replay local recordings.** The replay cache key holds the app, not the
|
|
device, so a step recorded on your simulator replays on the CI one.
|
|
[Commit the replay cache](/cache#commit-the-replay-cache).
|
|
|
|
## Gate pull requests
|
|
|
|
Each non-zero exit fails the job, a failed agent step included. To block
|
|
merges on a failed run, make the e2e job a required check.
|
|
[Debugging a run](/debugging#read-the-failure) lists each exit code and
|
|
whether rerunning the job can help.
|
|
|
|
To keep the checks predictable:
|
|
|
|
- Give each `agent.act` one [atomic goal](/core-concepts#goals). Then
|
|
check its result with a locator assertion or `agent.assert`.
|
|
- [Commit the replay cache](/cache#commit-the-replay-cache). A replayed
|
|
`agent.act` makes no model call. With the built-in agent, a cache miss, a
|
|
hand-off, and every agent judgment (`agent.assert`, `agent.waitFor`,
|
|
`agent.extract`) call the model.
|
|
- Watch for flaky tests. The report marks a test that passes only on the
|
|
CI retry as flaky.
|
|
- Add `--max-failures <n>` to stop the run after that many failed tests, so
|
|
a broken deploy doesn't spend a model call on each remaining step. The job
|
|
exits 1.
|
|
- Add `--max-duration <ms>` to limit test-running time; the clock starts
|
|
when tests start, not during app startup. The job exits 1.
|
|
|
|
## Run only affected tests
|
|
|
|
On a pull request, pass `--changed-since` so a [test filter](/filters) can
|
|
leave out the tests the change is unlikely to affect. When the filter scores
|
|
tests, as the decision filter does, each target starts the files of the
|
|
likeliest to fail first. Without the flag, as on `main`, the whole suite
|
|
runs. See [Add the decision filter](/filters#add-the-decision-filter).
|
|
|
|
With the CI default of one worker, only the relevance bands order files and
|
|
collection order breaks ties: e2e reads the
|
|
[timings history](#cache-the-timings-history) only with more than one worker.
|
|
See [Start the likeliest first](/filters#start-the-likeliest-first).
|
|
|
|
## Cache the timings history
|
|
|
|
With more than one worker, `e2e run` starts the longest files first, after
|
|
any file it has no time for, ordered by `<output>/timings.json` (`.e2e/timings.json` by default): a few KB that a
|
|
run updates when it ran at least one file whole. A fresh CI checkout has none,
|
|
so restore it from your CI cache before the run and save it after.
|
|
[GitHub Actions](/ci/github-actions#cache-the-timings-history) has the step.
|
|
|
|
- The file is advisory. One that is missing, corrupt, or invalid is no history,
|
|
and the run falls back to the previous `report.json`. A restored file can
|
|
change the order of files, never a result.
|
|
- A run times a file only when it ran all of it. A `--grep` or
|
|
`--last-failed` run leaves the entry of a file it ran in part as it was.
|
|
- CI defaults to one worker. Set [`workers`](/reference/config#workers) or
|
|
pass `--workers` to benefit.
|
|
- Key the cache by branch, with `main` as the fallback, so a new pull
|
|
request starts from the history of `main`. With [sharding](#sharding), add
|
|
the shard index to the key.
|
|
- `--timings <file>` reads and writes another path.
|
|
|
|
## Sharding
|
|
|
|
`--shard <index/total>` runs one slice of the suite. Every job runs the
|
|
same command with its own index, and every machine slices the suite the
|
|
same way. A shard with no test fails with `NO_TESTS`. If a matrix can have
|
|
more shards than tests, add `--pass-with-no-tests`. The
|
|
[CLI reference](/reference/cli#flags) has the slicing rules.
|
|
|
|
On GitHub Actions, a matrix runs one job for each shard:
|
|
|
|
```yaml
|
|
jobs:
|
|
e2e:
|
|
strategy:
|
|
fail-fast: false
|
|
matrix:
|
|
shard: [1, 2, 3]
|
|
steps:
|
|
# ...
|
|
- run: npx e2e run --shard ${{ matrix.shard }}/3 --reporter list,junit
|
|
```
|
|
|
|
Upload each shard's `report.json` and `junit.xml` under a name with the
|
|
shard index. e2e can't merge shard reports yet.
|
|
|
|
## Rerunning failures
|
|
|
|
`--last-failed` runs only the tests that didn't pass in the previous run,
|
|
read from `report.json` in the output directory. Run it in the same step as
|
|
the first run:
|
|
|
|
```yaml
|
|
- run: npx e2e run || npx e2e run --last-failed
|
|
```
|
|
|
|
The rerun keeps the first run's artifacts and writes its own under
|
|
`rerun-<n>/` in each test's directory, so one upload holds both. The
|
|
[CLI reference](/reference/cli#flags) lists what counts as not passed.
|
|
|
|
A separate job has no report on its runner. Download it into the output
|
|
directory first, or rerun the failed job.
|
|
|
|
## Next
|
|
|
|
<CardGroup cols={2}>
|
|
<Card title="Pull request comments" icon="code-pull-request" href="/github">
|
|
One comment per run, edited in place on a rerun.
|
|
</Card>
|
|
<Card title="CLI reference" icon="terminal" href="/reference/cli">
|
|
Flags, reporters, and every exit code.
|
|
</Card>
|
|
</CardGroup>
|