mirror of
https://github.com/tester-army/e2e.git
synced 2026-11-22 06:16:50 +08:00
* feat(cli): write a GitHub Actions or GitLab CI/CD pipeline in e2e init e2e init asks for a CI pipeline in a project with the web engine (--ci github-actions|gitlab|none). It writes one job for the project's package manager and Node version that installs Chromium, runs e2e run, and keeps the report, and names the secret or CI/CD variable for the model key. It never overwrites a pipeline file; beside an existing .gitlab-ci.yml it writes a file to include. * fix(cli): handle cubic's findings on the init CI step Exact GitLab stage names, a distinct job name beside an existing e2e job, loopback baseURL in an existing config, no writes through a symlinked .github or .gitlab, --ci none in the skill, gitlab in the telemetry comment, and the GitLab Node image's major version in the docs. * refactor(cli): keep the init CI pipelines as YAML templates The GitHub Actions workflow and the GitLab job live in packages/e2e/templates/ci/ as complete pipelines for npm at the repository root, linted as they are. init loads them with the yaml package's Document API and edits a copy for the project's package manager, folders, Node version, secrets, and target, keeping the comments. The pinned actions now live only in the templates. * fix(cli): point setup-bun at the package.json of a project in a subfolder * docs(ci): link the GitLab page from init and the GitLab template * feat(cli): write the iOS and Android CI jobs in e2e init init writes one job per platform the config's targets run on: web targets in the web job, iOS and Android targets in their own jobs, which build the app, boot a simulator or an emulator, and install the app. Targets come from the new config, or from loading an existing one after the confirmation and the install. Expo and bare React Native apps get their build; any other app gets a step to replace. GitHub's mobile jobs follow e2e-inc/e2e-react-native's workflow; GitLab's run on self-managed runners. * fix(cli): reinstall an emulator system image that holds no image in the Android CI job * fix(cli): handle cubic's findings on the mobile CI jobs Upload the emulator log from the GitLab Android job, count peer and optional engine dependencies, ask nothing for a new config with no engine, key caches on Bun's bun.lockb when that is the lockfile, and run the symlink test with a Windows junction. * fix(cli): put current Android command-line tools in place before the emulator image The runner image's cmdline-tools can be older than the system image's package format, and android-emulator-runner uses them to create the AVD, which then fails with 'contains no system images'. The Android job downloads command-line tools 23.0, checks its SHA-256, and installs the image with it. * fix(cli): import moved core modules from @e2e-dev/core in init * fix(cli): count a pipeline as set up only when a command runs e2e run init matched the text 'e2e run' anywhere in a workflow, so a comment or a step name such as 'our old e2e runner' made --ci write nothing. It now reads the commands (run, script, before_script, after_script) from the parsed YAML and matches e2e run as words of their own. Also: --ci help and the InitOptions JSDoc say tests on every platform, and yaml is pinned exactly like the other dependencies. * chore(ci): bump setup-node and upload-artifact pins in the init templates, docs, and skill docs/ci/vercel-previews.mdx on main pins setup-node v7.1.0 and upload-artifact v7.0.2, the pins this repository's own workflows run. The templates, three docs pages, and the skill still had v7.0.0 and v7.0.1, so the pin test failed once main was merged in. --------- Co-authored-by: Oskar Kwaśniewski <oskar@okwasniewski.com>
218 lines
7.9 KiB
Plaintext
218 lines
7.9 KiB
Plaintext
---
|
||
title: Pull request comments
|
||
sidebarTitle: GitHub PR comments
|
||
description: The @e2e-dev/github reporter posts one comment per run from GitHub Actions and keeps it current.
|
||
"og:image": "https://e2e.com/docs/images/og/github.png?v=be9fcc65d9"
|
||
"twitter:image": "https://e2e.com/docs/images/og/github.png?v=be9fcc65d9"
|
||
---
|
||
|
||
`@e2e-dev/github` is a reporter that posts the test results as a pull
|
||
request comment from GitHub Actions, and writes them to the job summary. A
|
||
rerun updates the same comment.
|
||
|
||
## Setup
|
||
|
||
<CodeGroup>
|
||
```bash npm
|
||
npm install --save-dev @e2e-dev/github
|
||
```
|
||
|
||
```bash pnpm
|
||
pnpm add -D @e2e-dev/github
|
||
```
|
||
|
||
```bash bun
|
||
bun add -d @e2e-dev/github
|
||
```
|
||
</CodeGroup>
|
||
|
||
```ts title="e2e.config.ts"
|
||
import type { E2EConfig } from 'e2e';
|
||
import { web } from '@e2e-dev/web';
|
||
import { github } from '@e2e-dev/github';
|
||
|
||
export default {
|
||
targets: [{ engine: web(), app: { url: 'http://localhost:3000' } }],
|
||
reporters: ['list', github()],
|
||
} satisfies E2EConfig;
|
||
```
|
||
|
||
Give the workflow `pull-requests: write`. Pass `GITHUB_TOKEN` to the test
|
||
step:
|
||
|
||
```yaml title=".github/workflows/e2e.yml"
|
||
permissions:
|
||
contents: read
|
||
pull-requests: write
|
||
|
||
jobs:
|
||
e2e:
|
||
steps:
|
||
# ...checkout, install, browsers
|
||
- name: Run e2e
|
||
run: npx e2e run --reporter list,junit
|
||
env:
|
||
GITHUB_TOKEN: ${{ github.token }}
|
||
- name: Upload artifacts
|
||
if: ${{ !cancelled() }}
|
||
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
|
||
with:
|
||
name: e2e-artifacts
|
||
path: .e2e/results
|
||
```
|
||
|
||
Upload the artifacts also when a retry passes: each failure's evidence
|
||
links open the run's Artifacts section. Keep `path: .e2e/results` so the
|
||
download has the same per-test directories as the report.
|
||
|
||
Source links start at the nearest directory with `.git` inside
|
||
`GITHUB_WORKSPACE`. A project below it, such as `packages/e2e-tests`, gets
|
||
links under that path. `actions/checkout` with a `path` input works too.
|
||
|
||
## The comment
|
||
|
||
```markdown expandable
|
||
### 🔴 e2e: 1 failed, 1 flaky, 3 passed, 1 skipped
|
||
9 agent steps · 4 replayed from cache · 31 model calls · 128k tokens (62% cached) · $0.31
|
||
|
||
**🔴 billing › upgrades to Pro**
|
||
[tests/billing.e2e.ts:12](https://github.com/acme/shop/blob/1a2b3c/tests/billing.e2e.ts#L12)
|
||
|
||
**ASSERTION_FAILED** at step 4 of 6: `agent.act` "Pick the Pro plan and pay", after 38.0s and 12 model calls
|
||
|
||
> expected heading "Your cart" to be visible
|
||
> The Pay button opened a page that still shows the Free plan.
|
||
|
||
- Failed the same way on both attempts: **ASSERTION_FAILED** at step 4.
|
||
- Turn 9: `tap({"target":"n41"})` → Screen changed: /checkout/plan. New: #n50 heading "Free plan"
|
||
- Screen: `/checkout/plan`
|
||
|
||
Evidence: [screenshot](https://github.com/acme/shop/actions/runs/123#artifacts), [video](https://github.com/acme/shop/actions/runs/123#artifacts)
|
||
|
||
<details>
|
||
<summary>⚠️ 1 flaky test passed on a retry</summary>
|
||
|
||
**⚠️ invite a member by email**
|
||
[tests/team.e2e.ts:21](https://github.com/acme/shop/blob/1a2b3c/tests/team.e2e.ts#L21)
|
||
|
||
**STEP_TIMEOUT** at step 3 of 3: `agent.waitFor` "the invitation email", after 30.0s
|
||
|
||
> the invitation email took too long
|
||
|
||
Evidence: [screenshot](https://github.com/acme/shop/actions/runs/123#artifacts)
|
||
</details>
|
||
|
||
<details>
|
||
<summary>All 6 tests in 5 files</summary>
|
||
|
||
| | Test | Agent | Time |
|
||
| --- | --- | --- | --- |
|
||
| 🔴 | **tests/billing.e2e.ts** · 1 failed | 3 steps · 19 calls | 41.2s |
|
||
| 🔴 | billing › upgrades to Pro | 3 steps · 19 calls | 41.2s |
|
||
| ⚠️ | **tests/team.e2e.ts** · 1 flaky | 1 step | 6.4s |
|
||
| ⚠️ | invite a member by email (1 failed attempt first) | 1 step | 6.4s |
|
||
...
|
||
</details>
|
||
|
||
<sub>Tested by [e2e](https://e2e.com) 0.13.0 · 2m 15s · web · [run artifacts](https://github.com/acme/shop/actions/runs/123#artifacts)</sub>
|
||
```
|
||
|
||
Run-level errors, when there are any, come before the failures. The test
|
||
list has an agent column only when the run used the agent.
|
||
|
||
A run whose [test filters](/filters) left tests out says so in the
|
||
headline, such as `e2e: 7 passed, 35 left out by filters`, so it never
|
||
reads as the whole suite. Above the test list, a line names the change the filters judged,
|
||
such as `12 files changed since abc1234`, then one line per filter says
|
||
what it kept, and a folded table lists each test it left out with the
|
||
reason. When no filter ran, such as when no file changed, a line says
|
||
why.
|
||
|
||
The reporter shortens a large report to fit GitHub's comment limit. App text
|
||
that GitHub would turn into a mention, an issue link, or a URL shows as
|
||
code, so a screen that reads `@someone` notifies no one. The built-in
|
||
`markdown` reporter writes the same summary to `.e2e/summary.md`.
|
||
|
||
## Rerunning the failed tests
|
||
|
||
A `--last-failed` rerun updates the comment as one run with the first pass,
|
||
covering the whole suite:
|
||
|
||
```yaml
|
||
- name: Run e2e
|
||
run: npx e2e run || npx e2e run --last-failed
|
||
env:
|
||
GITHUB_TOKEN: ${{ github.token }}
|
||
```
|
||
|
||
| First pass | Rerun | Comment shows |
|
||
| --- | --- | --- |
|
||
| failed | passed | flaky |
|
||
| failed | failed | failed |
|
||
| interrupted | passed | passed |
|
||
| any other result | not run | the first-pass result |
|
||
|
||
The comment keeps the first pass's filter lines and left-out tests.
|
||
|
||
The comment stays red while the rerun left out something it owed: a failed
|
||
test, a test that `--max-failures` or `--max-duration` stopped before it ran, or a failed
|
||
`beforeAll` or `afterAll` whose tests another filter kept out. It shows ⏹️
|
||
when the rerun left out a test that the first pass never finished. The
|
||
rerun's own exit code counts only what it ran. Each attempt's evidence links
|
||
to its own files.
|
||
|
||
## Matrix jobs
|
||
|
||
The reporter finds its comment by project, workflow, and job, which matrix
|
||
jobs share. Give each matrix entry its own `key`:
|
||
|
||
```ts
|
||
reporters: ['list', github({ key: process.env.MATRIX_BROWSER })],
|
||
```
|
||
|
||
Set `MATRIX_BROWSER: ${{ matrix.browser }}` in the step's `env`. The key
|
||
shows in the headline, such as `e2e chromium: 77 passed`. Keep it stable
|
||
across runs so a rerun updates the existing comment.
|
||
|
||
## Explorations
|
||
|
||
An [`e2e explore`](/explore) run posts a comment of its own, so a job can run
|
||
the tests and then explore. Give the exploration its own `--output`, such as
|
||
`npx e2e explore --output .e2e-explore`, and upload that directory too.
|
||
Otherwise it replaces the test run's results, and the evidence links in the
|
||
test run's comment break.
|
||
|
||
## When nothing is posted
|
||
|
||
The reporter never changes the run's status or exit code. When it can't
|
||
post, one row under the summary gives the reason:
|
||
|
||
```
|
||
GitHub not posted: set GITHUB_TOKEN in the step's env (GITHUB_TOKEN: ${{ github.token }}); written to the job summary
|
||
```
|
||
|
||
| Situation | What happens |
|
||
| --- | --- |
|
||
| Not on GitHub Actions | Nothing is written. The row says so. |
|
||
| A `push`, `schedule`, or `workflow_dispatch` run | The job summary is written. There is no pull request to comment on. |
|
||
| No `GITHUB_TOKEN` in the step's `env` | The job summary is written. The row shows the line to add. The reporter also reads `GH_TOKEN`. |
|
||
| A `pull_request` event from a fork | The token is read-only, so posting fails. The row says why. |
|
||
| An `issue_comment` event | On a pull request's thread, the comment has no source links, because the checked-out commit is on the default branch. |
|
||
| Missing `pull-requests: write` | Posting fails, and the reporter names the permission. |
|
||
| GitHub API rate limit | Posting stops and reports throttling. Wait for the reset or `Retry-After` before rerunning the job. |
|
||
|
||
The reporter reads the token when the run finishes, and the report doesn't
|
||
include it. On GitHub Enterprise Server, the reporter uses
|
||
`GITHUB_SERVER_URL` and `GITHUB_API_URL`.
|
||
|
||
## Next
|
||
|
||
<CardGroup cols={2}>
|
||
<Card title="GitHub Actions" icon="/images/integrations/github-actions.svg" href="/ci/github-actions">
|
||
The full workflow: install, browsers, cache, and artifacts.
|
||
</Card>
|
||
<Card title="Debugging a run" icon="magnifying-glass" href="/debugging">
|
||
Read the trace page that each failure links to.
|
||
</Card>
|
||
</CardGroup>
|