mirror of
https://github.com/tester-army/e2e.git
synced 2026-11-22 07:55:36 +08:00
* fix(run): keep --last-failed reruns honest: failure-limit skips, carried debts, kept evidence --last-failed now reruns tests --max-failures skipped. A test or failed suite hook another filter leaves out stays owed in the report's new run.carried until a rerun runs it; the github fold shows carried rows and errors and stays red while anything is carried. A rerun prunes artifacts/ to what the previous report names and writes its attempts under artifacts/rerun-<n>/ instead of wiping the tree and reusing attempt paths. * fix(run): carry a setup's hook debt when no selected test needs it The report has no row for an unneeded setup, so carryForward now asks the collection whether a test still exists instead of the report's rows. * fix(run): reject malformed artifacts in --last-failed reports, tighten rerun docs Distinct ids in the carried fixture row. * fix(run): keep carrying tests of a file a narrowed rerun could not collect * fix(github): a carried interrupted test reads interrupted, not failed, in the folded page
193 lines
7.2 KiB
Plaintext
193 lines
7.2 KiB
Plaintext
---
|
||
title: Pull request comments
|
||
description: The @e2e-dev/github reporter posts one comment per run from GitHub Actions and keeps it current.
|
||
---
|
||
|
||
`@e2e-dev/github` posts test results as a pull request comment, including
|
||
failures, source links, and links to workflow artifacts. Reruns update the
|
||
same comment, and a `--last-failed` rerun updates it as one run. The reporter
|
||
also writes the results to the job summary.
|
||
|
||
## Setup
|
||
|
||
<CodeGroup>
|
||
```bash npm
|
||
npm install --save-dev @e2e-dev/github
|
||
```
|
||
|
||
```bash pnpm
|
||
pnpm add -D @e2e-dev/github
|
||
```
|
||
|
||
```bash bun
|
||
bun add -d @e2e-dev/github
|
||
```
|
||
</CodeGroup>
|
||
|
||
```ts title="e2e.config.ts"
|
||
import type { E2EConfig } from 'e2e';
|
||
import { web } from '@e2e-dev/web';
|
||
import { github } from '@e2e-dev/github';
|
||
|
||
export default {
|
||
targets: [{ engine: web(), app: { url: 'http://localhost:3000' } }],
|
||
reporters: ['list', github()],
|
||
} satisfies E2EConfig;
|
||
```
|
||
|
||
Give the workflow permission to write pull request comments and pass
|
||
`GITHUB_TOKEN` to the test step:
|
||
|
||
```yaml title=".github/workflows/e2e.yml"
|
||
permissions:
|
||
contents: read
|
||
pull-requests: write
|
||
|
||
jobs:
|
||
e2e:
|
||
steps:
|
||
# ...checkout, install, browsers
|
||
- name: Run e2e
|
||
run: npx e2e run --reporter list,junit
|
||
env:
|
||
GITHUB_TOKEN: ${{ github.token }}
|
||
- name: Upload artifacts
|
||
if: ${{ !cancelled() }}
|
||
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
|
||
with:
|
||
name: e2e-artifacts
|
||
path: .e2e/artifacts
|
||
```
|
||
|
||
Upload artifacts even when a retry passes. Each failure's evidence links open
|
||
the run's Artifacts section; inside the download, every test has a directory
|
||
named after it and the agent that ran it
|
||
(`web/tests_billing.e2e.ts__upgrades/default/attempt-0/`), so keep
|
||
`path: .e2e/artifacts` and the download lines up with the report. A
|
||
`--last-failed` rerun keeps the first run's directories and puts its own
|
||
under `rerun-1/`, so a flaky test's failed attempt still links to its own
|
||
files. A project
|
||
below the checkout root, such as `packages/e2e-tests`, gets source links under
|
||
that path. The checkout root is the nearest directory holding `.git` inside
|
||
`GITHUB_WORKSPACE`, so `actions/checkout` with a `path` input works too.
|
||
|
||
## The comment
|
||
|
||
```markdown
|
||
### 🔴 e2e: 1 failed, 1 flaky, 3 passed, 1 skipped
|
||
9 agent steps · 4 replayed from cache · 31 model calls · 128k tokens (62% cached) · $0.31
|
||
|
||
**🔴 billing › upgrades to Pro**
|
||
[tests/billing.e2e.ts:12](https://github.com/acme/shop/blob/1a2b3c/tests/billing.e2e.ts#L12)
|
||
|
||
**ASSERTION_FAILED** at step 4 of 6: `agent.act` "Pick the Pro plan and pay", after 38.0s and 12 model calls
|
||
|
||
> expected heading "Your cart" to be visible
|
||
> The Pay button opened a page that still shows the Free plan.
|
||
|
||
- Failed the same way on both attempts: **ASSERTION_FAILED** at step 4.
|
||
- Turn 9: `tap({"target":"n41"})` → Screen changed: /checkout/plan. New: #n50 heading "Free plan"
|
||
- Screen: `/checkout/plan`
|
||
|
||
Evidence: [screenshot](https://github.com/acme/shop/actions/runs/123#artifacts), [video](https://github.com/acme/shop/actions/runs/123#artifacts), [trace](https://github.com/acme/shop/actions/runs/123#artifacts)
|
||
|
||
<details>
|
||
<summary>⚠️ 1 flaky test passed on a retry</summary>
|
||
|
||
**⚠️ invite a member by email**
|
||
[tests/team.e2e.ts:21](https://github.com/acme/shop/blob/1a2b3c/tests/team.e2e.ts#L21)
|
||
|
||
**STEP_TIMEOUT** at step 3 of 3: `agent.waitFor` "the invitation email", after 30.0s
|
||
|
||
> the invitation email took too long
|
||
|
||
Evidence: [screenshot](https://github.com/acme/shop/actions/runs/123#artifacts), [trace](https://github.com/acme/shop/actions/runs/123#artifacts)
|
||
</details>
|
||
|
||
<details>
|
||
<summary>All 6 tests in 5 files</summary>
|
||
|
||
| | Test | Agent | Time |
|
||
| --- | --- | --- | --- |
|
||
| 🔴 | **tests/billing.e2e.ts** · 1 failed | 3 steps · 19 calls | 41.2s |
|
||
| 🔴 | billing › upgrades to Pro | 3 steps · 19 calls | 41.2s |
|
||
| ⚠️ | **tests/team.e2e.ts** · 1 flaky | 1 step | 6.4s |
|
||
| ⚠️ | invite a member by email (1 failed attempt first) | 1 step | 6.4s |
|
||
...
|
||
</details>
|
||
|
||
<sub>e2e 0.13.0 · 2m 15s · web · [run artifacts](https://github.com/acme/shop/actions/runs/123#artifacts)</sub>
|
||
```
|
||
|
||
The comment shows test counts and agent usage, followed by each failure's
|
||
error, source line, and evidence links. Flaky tests and the full test list
|
||
are collapsed; the list is one table with a row per file above its tests,
|
||
and its agent column appears only when the run used the agent. Run-level
|
||
errors appear above the test failures. Large
|
||
reports are shortened to fit GitHub's comment limit. Text from the app under
|
||
test that GitHub would turn into a mention, an issue link, or a URL is shown
|
||
as code, so a screen reading `@someone` or `https://...` notifies no one and
|
||
links nowhere.
|
||
|
||
The built-in `markdown` reporter writes the same summary to
|
||
`.e2e/summary.md`.
|
||
|
||
## Rerunning the failed tests
|
||
|
||
A step that gives failed tests a second chance keeps one comment for both
|
||
passes:
|
||
|
||
```yaml
|
||
- name: Run e2e
|
||
run: npx e2e run || npx e2e run --last-failed
|
||
env:
|
||
GITHUB_TOKEN: ${{ github.token }}
|
||
```
|
||
|
||
The comment covers the whole suite, not only the tests the rerun ran again.
|
||
A test that failed and then passed shows as flaky, a test that failed twice
|
||
stays failed, one that was interrupted and then passed shows as passed, and
|
||
the rest keep their first-pass results. The comment stays red while the
|
||
rerun left out something it owed: a failed test, one `--max-failures` stopped
|
||
before it ran, or a failed `beforeAll` or `afterAll` whose tests another
|
||
filter kept out of the rerun. It shows ⏹️ when the rerun left out a test the
|
||
first pass never finished. The rerun's own exit code counts only what it ran.
|
||
A failed attempt's evidence links to the first run's files, which the rerun
|
||
keeps.
|
||
|
||
## Matrix jobs
|
||
|
||
The reporter identifies a comment by project, workflow, and job. Matrix
|
||
jobs share those identifiers, so give each matrix entry a distinct `key`:
|
||
|
||
```ts
|
||
reporters: ['list', github({ key: process.env.MATRIX_BROWSER })],
|
||
```
|
||
|
||
Set `MATRIX_BROWSER: ${{ matrix.browser }}` in the step's `env`. The key also
|
||
appears in the headline, such as `e2e chromium: 77 passed`. Keep it stable
|
||
across runs so reruns update the existing comment.
|
||
|
||
## When nothing is posted
|
||
|
||
The reporter never changes the run's status or exit code. When it cannot
|
||
post, it says why in one row under the summary:
|
||
|
||
```
|
||
Report .e2e/report.json
|
||
|
||
GitHub not posted: set GITHUB_TOKEN in the step's env (GITHUB_TOKEN: ${{ github.token }}); written to the job summary
|
||
```
|
||
|
||
| Situation | What happens |
|
||
| --- | --- |
|
||
| Not on GitHub Actions | Nothing is written; the row says so. |
|
||
| A `push`, `schedule`, or `workflow_dispatch` run | The job summary is written; there is no pull request to comment on. |
|
||
| No `GITHUB_TOKEN` in the step's `env` | The job summary is written; the row shows the line to add. `GH_TOKEN` is read too. |
|
||
| A `pull_request` event from a fork | Its token is read-only, so posting fails and the reporter explains why. |
|
||
| An `issue_comment` event | On a pull request's thread, the comment is posted without source links, since the checked-out commit is the default branch. |
|
||
| Missing `pull-requests: write` | The same failure, with the permission named. |
|
||
|
||
The token is read when the run finishes and is excluded from the report.
|
||
GitHub Enterprise Server uses `GITHUB_SERVER_URL` and `GITHUB_API_URL`.
|