A coverage check in the pnpm check chain and its own step in spec.yml's check job: every web slug keeps its tests/<slug>.e2e.ts twin, and every mobile scenario its tests-agent entry outside the documented Control Inventory exclusion. Both catalogs parse in either quote style; a catalog that parses to no scenarios fails the check, a `slug:` or `name:` property the parser cannot read (a template literal, a bare identifier) fails it too, and a commented-out `name:` no longer counts as an entry, so a format change or a disabled entry cannot pass the check by hiding a scenario. The web agentic side stays unchecked on purpose: Refs #1777 leaves that part to the issue. file-round-trip gains the twin test the check was missing, opening the page in beforeEach, asserting the input resets after a rejected upload, and covering the voucher that then verifies. Co-authored-by: Donny/강동윤 <kdy.1997.dev@gmail.com>
@e2e-dev/web-benchmark
A Next.js app of self-contained scenarios that are hard to automate, and the
e2e suites written against them. This is where we test writing tests with
e2e: every scenario isolates one awkward surface (shadow roots, a canvas-only
UI, nested iframes, native dialogs, a second tab, drag and drop, a DOM that
remounts every second, an accessibility tree that lies) so a test either
handles that surface or fails for exactly that reason.
The app under app/ is copied from apps/web-benchmark in the e2e-inc
monorepo, where the same scenarios gate the autonomous QA agent. Keep diffs
against that source minimal so scenarios port both ways, and never fix a
planted bug.
Layout
app/: the Next.js project.app/src/examples.tsis the catalog: every scenario's slug, name, description, and, for bug-book entries, theplantedBug. One component per scenario inapp/src/Examples/, served at/e/<slug>; the home page lists them all.e2e.config.ts+tests/: the deterministic suite. Gates every PR (benchmark.yml);@e2e-dev/githubposts each run as one pull request comment. Every scenario has a deterministic twin,tests/<slug>.e2e.ts, driven with locators andexpectonly: the floor the agentic suite must match, so a red agentic run is the model, not the tools. A bug-book scenario asserts what the page truly shows, planted bug included and named in the title. A scenario the deterministic API cannot finish keeps its test declared with{ skip: '<reason>' }, so the gap stays visible.e2e.agent.config.ts+tests-agent/: the agentic suite, derived from the deterministic config. It gates every PR too. Each step spends real model calls (cents per run) and needsAI_GATEWAY_API_KEY.scenarios.e2e.tsdrives every task scenario the accessibility tree can carry with oneagent.actper scenario and checks the success message deterministically; scenarios the grammar cannot finish yet are declared and skipped with the reason.planted-bugs.e2e.tsasserts that anagent.assertof the correct behavior fails on a bug-book scenario, so a passing judgment there is a missed bug. Its recordings under.e2e/cache/are committed: CI replays them read-only and calls the model only for a step with no recording, so the gate runs at deterministic speed. When a scenario or a step changes, re-record locally with thetest:agentcommand below and commit the changed entries in the same pull request.
Commands
Build the runner first; the suites run the built e2e CLI by path, like
the testbed does: pnpm links no e2e bin for a workspace package whose dist
does not exist yet at install time, and CI installs before it builds.
pnpm build # from the repo root, once
pnpm --filter @e2e-dev/web-benchmark test # build the app, run tests/
pnpm --filter @e2e-dev/web-benchmark test:headed
AI_GATEWAY_API_KEY=... pnpm --filter @e2e-dev/web-benchmark test:agent
pnpm --filter @e2e-dev/web-benchmark dev # browse the scenarios on :4280
Writing tests
test rebuilds the production app on every run. While authoring, keep the dev
server up instead: the config sets reuseExisting, so outside CI a server
already answering on port 4280 is used as is and nothing is started or
stopped. Run one file at a time from the package directory:
pnpm --filter @e2e-dev/web-benchmark dev
cd apps/web-benchmark
node node_modules/e2e/dist/cli/bin.js run tests/login-form.e2e.ts --headed
Two things bite here:
- Role names match exactly by default. The home page links are named after the
scenario name and its description, so
getByRole('link', { name: 'Login Form', exact: false })is the query that finds one. Seetests/smoke.e2e.ts. Elsewhere a partial text match is a RegExp,getByText(/Order #10482/), andselectOptionnames its option by{ value }, one spelling across the suite. reuseExistingtrusts whatever answers on the port. A dev server left running from another checkout serves that checkout's scenarios.
Reports land in .e2e/report.json, artifacts and trace pages under .e2e/results/, both
ignored.
Scenario contract
Task scenarios end in a deterministic success state. Where the DOM semantics
are intact, the terminal element carries data-testid="success-message"; the
adversarial scenarios (Canvas Only, Canvas Whiteboard, Div Soup, Aria Hidden
Flow, Image Only UI, CSS Content UI, Onboarding Wizard) omit test ids and
semantics on purpose, so success is only visible in pixels.
Bug-book scenarios (registry entries with plantedBug) contain exactly one
deterministic product bug, marked PLANTED BUG (do not fix) in source, and
the rest of the flow works. A test asserting the correct behavior must fail
there, which is how we check that an assertion or an agent.assert judgment
catches a real bug instead of passing over it. Promo Storefront and Gift Card
Purchase are traps: bug-looking artifacts (overlays, a chat widget, aria labels
that read "undefined") that are not defects, so a correct test passes.
The Login Form account is declared as the benchmark credential in
e2e.config.ts, so tests reach it through credentials.user('benchmark') and
agent steps fill the password through the secret tool without the model ever
seeing it.
Adding a scenario
Create app/src/Examples/YourCase.tsx, register it in app/src/examples.ts
(the doc comment there states the rules for task and bug-book entries), and
write its tests in tests/ or tests-agent/. Pair every agentic step with a
deterministic check so a wrong judgment cannot pass silently.