Files
e2e/docs/github.mdx
T
Kacper Kapuściak f185ca8ba2 docs: retire sidebar page icons (#1582)
* docs: retire sidebar page icons

The docs sidebar no longer shows icons. Drop icon: from every page's frontmatter, delete the grey silhouettes in docs/images/icons/, and make check-docs-logos fail on any frontmatter icon instead of requiring one on vendor pages. Card logos stay required. The authoring-docs skill trades its Page icons section for a short Vendor logos one.

* docs: require a migrate overview card for each migration guide

* docs: match only top-level icon keys in check-docs-logos
2026-11-05 18:23:59 +00:00

218 lines
7.9 KiB
Plaintext
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
---
title: Pull request comments
sidebarTitle: GitHub PR comments
description: The @e2e-dev/github reporter posts one comment per run from GitHub Actions and keeps it current.
"og:image": "https://e2e.com/docs/images/og/github.png?v=be9fcc65d9"
"twitter:image": "https://e2e.com/docs/images/og/github.png?v=be9fcc65d9"
---
`@e2e-dev/github` is a reporter that posts the test results as a pull
request comment from GitHub Actions, and writes them to the job summary. A
rerun updates the same comment.
## Setup
<CodeGroup>
```bash npm
npm install --save-dev @e2e-dev/github
```
```bash pnpm
pnpm add -D @e2e-dev/github
```
```bash bun
bun add -d @e2e-dev/github
```
</CodeGroup>
```ts title="e2e.config.ts"
import type { E2EConfig } from 'e2e';
import { web } from '@e2e-dev/web';
import { github } from '@e2e-dev/github';
export default {
targets: [{ engine: web(), app: { url: 'http://localhost:3000' } }],
reporters: ['list', github()],
} satisfies E2EConfig;
```
Give the workflow `pull-requests: write`. Pass `GITHUB_TOKEN` to the test
step:
```yaml title=".github/workflows/e2e.yml"
permissions:
contents: read
pull-requests: write
jobs:
e2e:
steps:
# ...checkout, install, browsers
- name: Run e2e
run: npx e2e run --reporter list,junit
env:
GITHUB_TOKEN: ${{ github.token }}
- name: Upload artifacts
if: ${{ !cancelled() }}
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: e2e-artifacts
path: .e2e/results
```
Upload the artifacts also when a retry passes: each failure's evidence
links open the run's Artifacts section. Keep `path: .e2e/results` so the
download has the same per-test directories as the report.
Source links start at the nearest directory with `.git` inside
`GITHUB_WORKSPACE`. A project below it, such as `packages/e2e-tests`, gets
links under that path. `actions/checkout` with a `path` input works too.
## The comment
```markdown expandable
### 🔴 e2e: 1 failed, 1 flaky, 3 passed, 1 skipped
9 agent steps · 4 replayed from cache · 31 model calls · 128k tokens (62% cached) · $0.31
**🔴 billing › upgrades to Pro**
[tests/billing.e2e.ts:12](https://github.com/acme/shop/blob/1a2b3c/tests/billing.e2e.ts#L12)
**ASSERTION_FAILED** at step 4 of 6: `agent.act` "Pick the Pro plan and pay", after 38.0s and 12 model calls
> expected heading "Your cart" to be visible
> The Pay button opened a page that still shows the Free plan.
- Failed the same way on both attempts: **ASSERTION_FAILED** at step 4.
- Turn 9: `tap({"target":"n41"})` → Screen changed: /checkout/plan. New: #n50 heading "Free plan"
- Screen: `/checkout/plan`
Evidence: [screenshot](https://github.com/acme/shop/actions/runs/123#artifacts), [video](https://github.com/acme/shop/actions/runs/123#artifacts)
<details>
<summary>⚠️ 1 flaky test passed on a retry</summary>
**⚠️ invite a member by email**
[tests/team.e2e.ts:21](https://github.com/acme/shop/blob/1a2b3c/tests/team.e2e.ts#L21)
**STEP_TIMEOUT** at step 3 of 3: `agent.waitFor` "the invitation email", after 30.0s
> the invitation email took too long
Evidence: [screenshot](https://github.com/acme/shop/actions/runs/123#artifacts)
</details>
<details>
<summary>All 6 tests in 5 files</summary>
| | Test | Agent | Time |
| --- | --- | --- | --- |
| 🔴 | **tests/billing.e2e.ts** · 1 failed | 3 steps · 19 calls | 41.2s |
| 🔴 | billing › upgrades to Pro | 3 steps · 19 calls | 41.2s |
| ⚠️ | **tests/team.e2e.ts** · 1 flaky | 1 step | 6.4s |
| ⚠️ | invite a member by email (1 failed attempt first) | 1 step | 6.4s |
...
</details>
<sub>Tested by [e2e](https://e2e.com) 0.13.0 · 2m 15s · web · [run artifacts](https://github.com/acme/shop/actions/runs/123#artifacts)</sub>
```
Run-level errors, when there are any, come before the failures. The test
list has an agent column only when the run used the agent.
A run whose [test filters](/filters) left tests out says so in the
headline, such as `e2e: 7 passed, 35 left out by filters`, so it never
reads as the whole suite. Above the test list, a line names the change the filters judged,
such as `12 files changed since abc1234`, then one line per filter says
what it kept, and a folded table lists each test it left out with the
reason. When no filter ran, such as when no file changed, a line says
why.
The reporter shortens a large report to fit GitHub's comment limit. App text
that GitHub would turn into a mention, an issue link, or a URL shows as
code, so a screen that reads `@someone` notifies no one. The built-in
`markdown` reporter writes the same summary to `.e2e/summary.md`.
## Rerunning the failed tests
A `--last-failed` rerun updates the comment as one run with the first pass,
covering the whole suite:
```yaml
- name: Run e2e
run: npx e2e run || npx e2e run --last-failed
env:
GITHUB_TOKEN: ${{ github.token }}
```
| First pass | Rerun | Comment shows |
| --- | --- | --- |
| failed | passed | flaky |
| failed | failed | failed |
| interrupted | passed | passed |
| any other result | not run | the first-pass result |
The comment keeps the first pass's filter lines and left-out tests.
The comment stays red while the rerun left out something it owed: a failed
test, a test that `--max-failures` or `--max-duration` stopped before it ran, or a failed
`beforeAll` or `afterAll` whose tests another filter kept out. It shows ⏹️
when the rerun left out a test that the first pass never finished. The
rerun's own exit code counts only what it ran. Each attempt's evidence links
to its own files.
## Matrix jobs
The reporter finds its comment by project, workflow, and job, which matrix
jobs share. Give each matrix entry its own `key`:
```ts
reporters: ['list', github({ key: process.env.MATRIX_BROWSER })],
```
Set `MATRIX_BROWSER: ${{ matrix.browser }}` in the step's `env`. The key
shows in the headline, such as `e2e chromium: 77 passed`. Keep it stable
across runs so a rerun updates the existing comment.
## Explorations
An [`e2e explore`](/explore) run posts a comment of its own, so a job can run
the tests and then explore. Give the exploration its own `--output`, such as
`npx e2e explore --output .e2e-explore`, and upload that directory too.
Otherwise it replaces the test run's results, and the evidence links in the
test run's comment break.
## When nothing is posted
The reporter never changes the run's status or exit code. When it can't
post, one row under the summary gives the reason:
```
GitHub not posted: set GITHUB_TOKEN in the step's env (GITHUB_TOKEN: ${{ github.token }}); written to the job summary
```
| Situation | What happens |
| --- | --- |
| Not on GitHub Actions | Nothing is written. The row says so. |
| A `push`, `schedule`, or `workflow_dispatch` run | The job summary is written. There is no pull request to comment on. |
| No `GITHUB_TOKEN` in the step's `env` | The job summary is written. The row shows the line to add. The reporter also reads `GH_TOKEN`. |
| A `pull_request` event from a fork | The token is read-only, so posting fails. The row says why. |
| An `issue_comment` event | On a pull request's thread, the comment has no source links, because the checked-out commit is on the default branch. |
| Missing `pull-requests: write` | Posting fails, and the reporter names the permission. |
| GitHub API rate limit | Posting stops and reports throttling. Wait for the reset or `Retry-After` before rerunning the job. |
The reporter reads the token when the run finishes, and the report doesn't
include it. On GitHub Enterprise Server, the reporter uses
`GITHUB_SERVER_URL` and `GITHUB_API_URL`.
## Next
<CardGroup cols={2}>
<Card title="GitHub Actions" icon="/images/integrations/github-actions.svg" href="/ci/github-actions">
The full workflow: install, browsers, cache, and artifacts.
</Card>
<Card title="Debugging a run" icon="magnifying-glass" href="/debugging">
Read the trace page that each failure links to.
</Card>
</CardGroup>