mirror of
https://github.com/tester-army/e2e.git
synced 2026-11-22 05:00:35 +08:00
* docs: retire sidebar page icons The docs sidebar no longer shows icons. Drop icon: from every page's frontmatter, delete the grey silhouettes in docs/images/icons/, and make check-docs-logos fail on any frontmatter icon instead of requiring one on vendor pages. Card logos stay required. The authoring-docs skill trades its Page icons section for a short Vendor logos one. * docs: require a migrate overview card for each migration guide * docs: match only top-level icon keys in check-docs-logos
218 lines
7.9 KiB
Plaintext
218 lines
7.9 KiB
Plaintext
---
|
||
title: Pull request comments
|
||
sidebarTitle: GitHub PR comments
|
||
description: The @e2e-dev/github reporter posts one comment per run from GitHub Actions and keeps it current.
|
||
"og:image": "https://e2e.com/docs/images/og/github.png?v=be9fcc65d9"
|
||
"twitter:image": "https://e2e.com/docs/images/og/github.png?v=be9fcc65d9"
|
||
---
|
||
|
||
`@e2e-dev/github` is a reporter that posts the test results as a pull
|
||
request comment from GitHub Actions, and writes them to the job summary. A
|
||
rerun updates the same comment.
|
||
|
||
## Setup
|
||
|
||
<CodeGroup>
|
||
```bash npm
|
||
npm install --save-dev @e2e-dev/github
|
||
```
|
||
|
||
```bash pnpm
|
||
pnpm add -D @e2e-dev/github
|
||
```
|
||
|
||
```bash bun
|
||
bun add -d @e2e-dev/github
|
||
```
|
||
</CodeGroup>
|
||
|
||
```ts title="e2e.config.ts"
|
||
import type { E2EConfig } from 'e2e';
|
||
import { web } from '@e2e-dev/web';
|
||
import { github } from '@e2e-dev/github';
|
||
|
||
export default {
|
||
targets: [{ engine: web(), app: { url: 'http://localhost:3000' } }],
|
||
reporters: ['list', github()],
|
||
} satisfies E2EConfig;
|
||
```
|
||
|
||
Give the workflow `pull-requests: write`. Pass `GITHUB_TOKEN` to the test
|
||
step:
|
||
|
||
```yaml title=".github/workflows/e2e.yml"
|
||
permissions:
|
||
contents: read
|
||
pull-requests: write
|
||
|
||
jobs:
|
||
e2e:
|
||
steps:
|
||
# ...checkout, install, browsers
|
||
- name: Run e2e
|
||
run: npx e2e run --reporter list,junit
|
||
env:
|
||
GITHUB_TOKEN: ${{ github.token }}
|
||
- name: Upload artifacts
|
||
if: ${{ !cancelled() }}
|
||
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
|
||
with:
|
||
name: e2e-artifacts
|
||
path: .e2e/results
|
||
```
|
||
|
||
Upload the artifacts also when a retry passes: each failure's evidence
|
||
links open the run's Artifacts section. Keep `path: .e2e/results` so the
|
||
download has the same per-test directories as the report.
|
||
|
||
Source links start at the nearest directory with `.git` inside
|
||
`GITHUB_WORKSPACE`. A project below it, such as `packages/e2e-tests`, gets
|
||
links under that path. `actions/checkout` with a `path` input works too.
|
||
|
||
## The comment
|
||
|
||
```markdown expandable
|
||
### 🔴 e2e: 1 failed, 1 flaky, 3 passed, 1 skipped
|
||
9 agent steps · 4 replayed from cache · 31 model calls · 128k tokens (62% cached) · $0.31
|
||
|
||
**🔴 billing › upgrades to Pro**
|
||
[tests/billing.e2e.ts:12](https://github.com/acme/shop/blob/1a2b3c/tests/billing.e2e.ts#L12)
|
||
|
||
**ASSERTION_FAILED** at step 4 of 6: `agent.act` "Pick the Pro plan and pay", after 38.0s and 12 model calls
|
||
|
||
> expected heading "Your cart" to be visible
|
||
> The Pay button opened a page that still shows the Free plan.
|
||
|
||
- Failed the same way on both attempts: **ASSERTION_FAILED** at step 4.
|
||
- Turn 9: `tap({"target":"n41"})` → Screen changed: /checkout/plan. New: #n50 heading "Free plan"
|
||
- Screen: `/checkout/plan`
|
||
|
||
Evidence: [screenshot](https://github.com/acme/shop/actions/runs/123#artifacts), [video](https://github.com/acme/shop/actions/runs/123#artifacts)
|
||
|
||
<details>
|
||
<summary>⚠️ 1 flaky test passed on a retry</summary>
|
||
|
||
**⚠️ invite a member by email**
|
||
[tests/team.e2e.ts:21](https://github.com/acme/shop/blob/1a2b3c/tests/team.e2e.ts#L21)
|
||
|
||
**STEP_TIMEOUT** at step 3 of 3: `agent.waitFor` "the invitation email", after 30.0s
|
||
|
||
> the invitation email took too long
|
||
|
||
Evidence: [screenshot](https://github.com/acme/shop/actions/runs/123#artifacts)
|
||
</details>
|
||
|
||
<details>
|
||
<summary>All 6 tests in 5 files</summary>
|
||
|
||
| | Test | Agent | Time |
|
||
| --- | --- | --- | --- |
|
||
| 🔴 | **tests/billing.e2e.ts** · 1 failed | 3 steps · 19 calls | 41.2s |
|
||
| 🔴 | billing › upgrades to Pro | 3 steps · 19 calls | 41.2s |
|
||
| ⚠️ | **tests/team.e2e.ts** · 1 flaky | 1 step | 6.4s |
|
||
| ⚠️ | invite a member by email (1 failed attempt first) | 1 step | 6.4s |
|
||
...
|
||
</details>
|
||
|
||
<sub>Tested by [e2e](https://e2e.com) 0.13.0 · 2m 15s · web · [run artifacts](https://github.com/acme/shop/actions/runs/123#artifacts)</sub>
|
||
```
|
||
|
||
Run-level errors, when there are any, come before the failures. The test
|
||
list has an agent column only when the run used the agent.
|
||
|
||
A run whose [test filters](/filters) left tests out says so in the
|
||
headline, such as `e2e: 7 passed, 35 left out by filters`, so it never
|
||
reads as the whole suite. Above the test list, a line names the change the filters judged,
|
||
such as `12 files changed since abc1234`, then one line per filter says
|
||
what it kept, and a folded table lists each test it left out with the
|
||
reason. When no filter ran, such as when no file changed, a line says
|
||
why.
|
||
|
||
The reporter shortens a large report to fit GitHub's comment limit. App text
|
||
that GitHub would turn into a mention, an issue link, or a URL shows as
|
||
code, so a screen that reads `@someone` notifies no one. The built-in
|
||
`markdown` reporter writes the same summary to `.e2e/summary.md`.
|
||
|
||
## Rerunning the failed tests
|
||
|
||
A `--last-failed` rerun updates the comment as one run with the first pass,
|
||
covering the whole suite:
|
||
|
||
```yaml
|
||
- name: Run e2e
|
||
run: npx e2e run || npx e2e run --last-failed
|
||
env:
|
||
GITHUB_TOKEN: ${{ github.token }}
|
||
```
|
||
|
||
| First pass | Rerun | Comment shows |
|
||
| --- | --- | --- |
|
||
| failed | passed | flaky |
|
||
| failed | failed | failed |
|
||
| interrupted | passed | passed |
|
||
| any other result | not run | the first-pass result |
|
||
|
||
The comment keeps the first pass's filter lines and left-out tests.
|
||
|
||
The comment stays red while the rerun left out something it owed: a failed
|
||
test, a test that `--max-failures` or `--max-duration` stopped before it ran, or a failed
|
||
`beforeAll` or `afterAll` whose tests another filter kept out. It shows ⏹️
|
||
when the rerun left out a test that the first pass never finished. The
|
||
rerun's own exit code counts only what it ran. Each attempt's evidence links
|
||
to its own files.
|
||
|
||
## Matrix jobs
|
||
|
||
The reporter finds its comment by project, workflow, and job, which matrix
|
||
jobs share. Give each matrix entry its own `key`:
|
||
|
||
```ts
|
||
reporters: ['list', github({ key: process.env.MATRIX_BROWSER })],
|
||
```
|
||
|
||
Set `MATRIX_BROWSER: ${{ matrix.browser }}` in the step's `env`. The key
|
||
shows in the headline, such as `e2e chromium: 77 passed`. Keep it stable
|
||
across runs so a rerun updates the existing comment.
|
||
|
||
## Explorations
|
||
|
||
An [`e2e explore`](/explore) run posts a comment of its own, so a job can run
|
||
the tests and then explore. Give the exploration its own `--output`, such as
|
||
`npx e2e explore --output .e2e-explore`, and upload that directory too.
|
||
Otherwise it replaces the test run's results, and the evidence links in the
|
||
test run's comment break.
|
||
|
||
## When nothing is posted
|
||
|
||
The reporter never changes the run's status or exit code. When it can't
|
||
post, one row under the summary gives the reason:
|
||
|
||
```
|
||
GitHub not posted: set GITHUB_TOKEN in the step's env (GITHUB_TOKEN: ${{ github.token }}); written to the job summary
|
||
```
|
||
|
||
| Situation | What happens |
|
||
| --- | --- |
|
||
| Not on GitHub Actions | Nothing is written. The row says so. |
|
||
| A `push`, `schedule`, or `workflow_dispatch` run | The job summary is written. There is no pull request to comment on. |
|
||
| No `GITHUB_TOKEN` in the step's `env` | The job summary is written. The row shows the line to add. The reporter also reads `GH_TOKEN`. |
|
||
| A `pull_request` event from a fork | The token is read-only, so posting fails. The row says why. |
|
||
| An `issue_comment` event | On a pull request's thread, the comment has no source links, because the checked-out commit is on the default branch. |
|
||
| Missing `pull-requests: write` | Posting fails, and the reporter names the permission. |
|
||
| GitHub API rate limit | Posting stops and reports throttling. Wait for the reset or `Retry-After` before rerunning the job. |
|
||
|
||
The reporter reads the token when the run finishes, and the report doesn't
|
||
include it. On GitHub Enterprise Server, the reporter uses
|
||
`GITHUB_SERVER_URL` and `GITHUB_API_URL`.
|
||
|
||
## Next
|
||
|
||
<CardGroup cols={2}>
|
||
<Card title="GitHub Actions" icon="/images/integrations/github-actions.svg" href="/ci/github-actions">
|
||
The full workflow: install, browsers, cache, and artifacts.
|
||
</Card>
|
||
<Card title="Debugging a run" icon="magnifying-glass" href="/debugging">
|
||
Read the trace page that each failure links to.
|
||
</Card>
|
||
</CardGroup>
|