Files
e2e/docs/visual-testing.mdx
Kacper KapuściakandOskar Kwaśniewski d593863b41 feat(cli): write a GitHub Actions or GitLab CI/CD pipeline in e2e init (#1647)
* feat(cli): write a GitHub Actions or GitLab CI/CD pipeline in e2e init

e2e init asks for a CI pipeline in a project with the web engine (--ci github-actions|gitlab|none). It writes one job for the project's package manager and Node version that installs Chromium, runs e2e run, and keeps the report, and names the secret or CI/CD variable for the model key. It never overwrites a pipeline file; beside an existing .gitlab-ci.yml it writes a file to include.

* fix(cli): handle cubic's findings on the init CI step

Exact GitLab stage names, a distinct job name beside an existing e2e job, loopback baseURL in an existing config, no writes through a symlinked .github or .gitlab, --ci none in the skill, gitlab in the telemetry comment, and the GitLab Node image's major version in the docs.

* refactor(cli): keep the init CI pipelines as YAML templates

The GitHub Actions workflow and the GitLab job live in packages/e2e/templates/ci/ as complete pipelines for npm at the repository root, linted as they are. init loads them with the yaml package's Document API and edits a copy for the project's package manager, folders, Node version, secrets, and target, keeping the comments. The pinned actions now live only in the templates.

* fix(cli): point setup-bun at the package.json of a project in a subfolder

* docs(ci): link the GitLab page from init and the GitLab template

* feat(cli): write the iOS and Android CI jobs in e2e init

init writes one job per platform the config's targets run on: web targets in the web job, iOS and Android targets in their own jobs, which build the app, boot a simulator or an emulator, and install the app. Targets come from the new config, or from loading an existing one after the confirmation and the install. Expo and bare React Native apps get their build; any other app gets a step to replace. GitHub's mobile jobs follow e2e-inc/e2e-react-native's workflow; GitLab's run on self-managed runners.

* fix(cli): reinstall an emulator system image that holds no image in the Android CI job

* fix(cli): handle cubic's findings on the mobile CI jobs

Upload the emulator log from the GitLab Android job, count peer and optional engine dependencies, ask nothing for a new config with no engine, key caches on Bun's bun.lockb when that is the lockfile, and run the symlink test with a Windows junction.

* fix(cli): put current Android command-line tools in place before the emulator image

The runner image's cmdline-tools can be older than the system image's package format, and android-emulator-runner uses them to create the AVD, which then fails with 'contains no system images'. The Android job downloads command-line tools 23.0, checks its SHA-256, and installs the image with it.

* fix(cli): import moved core modules from @e2e-dev/core in init

* fix(cli): count a pipeline as set up only when a command runs e2e run

init matched the text 'e2e run' anywhere in a workflow, so a comment or a step name such as 'our old e2e runner' made --ci write nothing. It now reads the commands (run, script, before_script, after_script) from the parsed YAML and matches e2e run as words of their own.

Also: --ci help and the InitOptions JSDoc say tests on every platform, and yaml is pinned exactly like the other dependencies.

* chore(ci): bump setup-node and upload-artifact pins in the init templates, docs, and skill

docs/ci/vercel-previews.mdx on main pins setup-node v7.1.0 and upload-artifact v7.0.2, the pins this repository's own workflows run. The templates, three docs pages, and the skill still had v7.0.0 and v7.0.1, so the pin test failed once main was merged in.

---------

Co-authored-by: Oskar Kwaśniewski <oskar@okwasniewski.com>
2026-11-16 17:21:49 +00:00

151 lines
5.2 KiB
Plaintext

---
title: Visual comparison
description: Compare the screen, or one element, against a stored screenshot on the web, iOS, and Android with the same test.
"og:image": "https://e2e.com/docs/images/og/visual-testing.png?v=4929216b19"
"twitter:image": "https://e2e.com/docs/images/og/visual-testing.png?v=4929216b19"
---
import { Video } from './snippets/video.jsx';
`toHaveScreenshot` compares what the app shows with a PNG stored next to
the test file. It works on every engine that captures pixels, so one test
checks a browser, an iOS simulator, and an Android emulator.
<Video>
<video src="/images/visual-testing.mp4" aria-label="A failed toHaveScreenshot assertion in the terminal, then the expected, actual, and diff images side by side." muted loop playsInline controls />
A failed comparison, then its expected, actual, and diff images.
</Video>
```ts title="tests/checkout.e2e.ts"
import { test, expect } from 'e2e';
test('checkout looks right', async ({ app, screen }) => {
await app.open('/checkout');
await expect(screen).toHaveScreenshot('checkout.png');
await expect(screen.getByRole('button', 'Pay')).toHaveScreenshot('pay-button.png');
});
```
`expect(screen)` compares the whole screen. `expect(locator)` scrolls its
one match into view and compares that box.
## Stored screenshots
Commit the screenshots with the tests. The first run has nothing to compare
with, so it writes the screenshot and fails. Look at the image, commit it,
and run again.
```text
tests/checkout.e2e.ts
tests/checkout.e2e.ts-snapshots/checkout-web-linux.png
tests/checkout.e2e.ts-snapshots/checkout-ios-darwin.png
tests/checkout.e2e.ts-snapshots/pay-button-web-linux.png
```
Each target and operating system keeps its own file, since fonts and
rendering differ. Without a name, the file takes the test's name:
`checkout-looks-right-1-web-linux.png`.
When the app changes on purpose, rewrite the screenshots that differ and
write the missing ones. The run then passes:
<CodeGroup>
```bash npm
npx e2e run --update-snapshots
```
```bash pnpm
pnpm exec e2e run --update-snapshots
```
```bash bun
bunx e2e run --update-snapshots
```
</CodeGroup>
## In CI
Take the screenshots on the machine that renders them in CI: a Linux
runner for the web and Android, a macOS runner with the same simulator for
iOS.
Without `--update-snapshots`, a CI run never writes into the project. A
missing screenshot fails the test, and the run saves it in the test's
results at `snapshots/<path in the project>` for you to download and
commit. To write all of them at one time, run `--update-snapshots` in a job
like the test job:
```yaml title=".github/workflows/screenshots.yml"
name: Update screenshots
on: workflow_dispatch
jobs:
screenshots:
runs-on: ubuntu-latest
steps:
# the same checkout, setup, and install steps as the test job
- run: npx e2e run --update-snapshots
- uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
with:
name: screenshots
path: tests/**/*-snapshots/
```
Download the artifact into the project, check the images, and commit
them.
## When it fails
The matcher takes screenshots until one matches, or until the assertion
timeout (`assertionTimeout`, 5 s) ends. On a mismatch, the failure gives the
number of different pixels and three images in the test's results: `-diff`
with the changed pixels in red, `-actual`, and `-expected`.
```text
2173 pixels (0.24% of the image) differ from tests/home.e2e.ts-snapshots/screen-web-darwin.png; run with --update-snapshots to keep this run's
diff: .e2e/results/home-1a2b3c4d5e6f7a8b/attempt-1/screenshots/screen-diff.png
actual: .e2e/results/home-1a2b3c4d5e6f7a8b/attempt-1/screenshots/screen-actual.png
expected: .e2e/results/home-1a2b3c4d5e6f7a8b/attempt-1/screenshots/screen-expected.png
```
To write a screenshot, the matcher waits for two in a row that are the
same. A screen that never holds still fails, with the last two screenshots
and their diff.
## Tolerance and masks
```ts
await expect(screen).toHaveScreenshot('dashboard.png', {
maxDiffPixelRatio: 0.01,
mask: [screen.getByTestId('last-updated')],
});
```
| Option | Default | Meaning |
| --- | --- | --- |
| `threshold` | `0.2` | How far one pixel's color can drift and still count as the same, 0 to 1 |
| `maxDiffPixels` | `0` | How many pixels can differ |
| `maxDiffPixelRatio` | `0` | What share of the pixels can differ, 0 to 1 |
| `mask` | none | Locators painted over before the comparison, for content that changes between runs |
| `maskColor` | `#ff00ff` | Color of the mask boxes |
| `timeout` | `assertionTimeout` | Assertion budget in milliseconds |
On the web, e2e stops animations and hides the text caret. On iOS and
Android, it holds the status bar fixed, so the clock and battery match on
every run.
e2e masks secure fields in every screenshot, after a secret fill too. The
matcher fails with `POLICY_DENIED` when the engine can't prove every secure
field on screen is masked.
## Next
<CardGroup cols={2}>
<Card title="Screenshot matchers" icon="circle-check" href="/reference/expect#screenshot-matchers">
`toHaveScreenshot` and `.not`, with every option.
</Card>
<Card title="Security" icon="shield-halved" href="/security">
How secrets stay out of screenshots, reports, and model input.
</Card>
</CardGroup>