Files
..

@e2e-dev/testbed

Dogfood workspace for the e2e runner: a real project consuming the built e2e package exactly like a user would, against playground apps we control. This is where every runner feature has a deterministic test. Hard UI surfaces live in the benchmarks (web-benchmark, mobile-benchmark), not here.

Layout

  • app/server.mjs — dependency-free playground app. The runner starts and stops it via the web engine's command option. The pages live under app/pages/ by group, each module exporting its routes and its nav entries: basics (home, todos, forms, login/session, dashboard, wizard), interaction (network, dialogs, board, pointer pad, iframes), canvas (four pixels-only surfaces), downloads (a file and a long page), and controls (control states, scrolling, the browser fixture's page, the speed counter, about). app/pages/index.mjs merges them and app/layout.mjs renders the nav from that registry.
  • app/playwright/: Playwright's test assets (v1.64.1, Apache-2.0, with its LICENSE and NOTICE) and the dynamic routes its test server sets, served by the playground for any path it has no page for.
  • app/bug-garden.mjs — a bookshop with planted defects, for e2e explore.
  • dogfood/server.mjs — an expense-claims app with a test API, for the executor dogfoods.
  • e2e.config.ts + tests/ — the local suite, gating every PR: queries, actions, polling assertions, sessions (test.setup + session:), serial groups, routes, dialogs, frames, downloads, uploads, keyboard input, credentials and secrets, plus one file per surface added since: control states and the failure codes (controls), pointer coordinates and path swipes (pointer), viewport and node scrolling (scroll), the browser fixture's own verbs (browser), and the per-action speed floor (speed). The files tagged playwright are Playwright's own tests/page specs (and tests/library/download.spec.ts) at v1.64.1, ported to the public API: each header names its source and the tests left out. fail: 'reason' marks a gap to close; skip: 'reason' one that fails with an error fail does not cover. A difference e2e keeps on purpose is left out in the header and stated in docs/migrate/playwright.mdx. --tag playwright runs only them, and CI runs them on every pull request. tests/helpers.ts holds the shared failure, boxOf, and centerOf, and the ported tests' content, asset, PREFIX, and CROSS.
  • e2e.notte.config.ts: the local suite on @e2e-dev/notte cloud browsers, run by hand with NOTTE_API_KEY and E2E_NOTTE_APP_URL, a public tunnel to the playground (pnpm app). Tests that open 127.0.0.1 or localhost directly fail there, since the browser runs in Notte's cloud.
  • e2e.lightpanda.config.ts and e2e.lightpanda-agent.config.ts: the local suite and the agent suite on @e2e-dev/lightpanda, which starts Lightpanda from LIGHTPANDA_PATH (or PATH), run by hand to see which features its nightly covers; the docs page integrations/lightpanda lists the result.
  • e2e.moli.config.ts: the full local suite on @e2e-dev/moli, which starts and stops Moli for the run, with one worker. Reports and artifacts land in .e2e-moli/.
  • e2e.scroll.config.ts + tests-scroll/ — scroll_to by text over the windowed /scroll page, driven by a scripted executor, so it needs no model. Its own config, since it makes the default agent the scripted executor; it runs as test:scroll, its own CI step, and never inside test, whose forwarded arguments (--changed-since) would not reach it.
  • e2e.agent.config.ts + tests-agent/ — opt-in agentic suite against the playground: agent.act flows, assisted polling, judgments, schema-validated extraction with zod, and the pixel tools on the canvas pages. e2e.mixed.config.ts adds the local suite to it for watching the list reporter interleave the two.
  • e2e.dogfood.config.ts + tests-dogfood/ — the built-in agent extended with project tools (seed and reset over the expense app's test API).
  • e2e.dogfood-brain.config.ts + tests-dogfood-brain/ — a hand-rolled StepExecutor with its own tools and transport; the engine is never touched.
  • e2e.dogfood-edge.config.ts + tests-dogfood-edge/ — steps expected to conclude blocked or failed; the report, not the exit code, is the output.
  • e2e.stress.config.ts + tests-stress/ — failures, timeouts, skips, flakes, and hostile titles for the reporters.
  • e2e.explore.config.ts + scripts/explore-bench.mjs — e2e explore against the bug garden, and a scorer that runs it across models.
  • e2e.scratch.config.ts + tests-scratch/ — a scratch agent config for one-off investigations with the AI SDK devtools recorder.

Commands

pnpm --filter e2e build               # the testbed runs the built runner
pnpm --filter @e2e-dev/testbed test    # typecheck + local suite (starts the app itself)
pnpm --filter @e2e-dev/testbed test:headed
pnpm --filter @e2e-dev/testbed app     # run the playground manually
AI_GATEWAY_API_KEY=... pnpm --filter @e2e-dev/testbed test:agent     # real model calls
AI_GATEWAY_API_KEY=... pnpm --filter @e2e-dev/testbed test:dogfood
AI_GATEWAY_API_KEY=... pnpm --filter @e2e-dev/testbed explore:garden
pnpm --filter @e2e-dev/testbed test:stress                            # reporters only

The local suite runs in CI on every push. Reports land in .e2e/report.json; artifacts and trace pages under .e2e/results/.

Browser Run live checks

e2e.browser-run.config.ts runs against a public HTTPS deployment of the playground pages. Set BROWSER_RUN_TEST_URL, CLOUDFLARE_ACCOUNT_ID, and CLOUDFLARE_API_TOKEN with Browser Rendering - Edit permission, then build the workspace packages and run from this directory:

node ../../packages/e2e/dist/cli/bin.js run --config e2e.browser-run.config.ts
BROWSER_RUN_SCOPE=attempt node ../../packages/e2e/dist/cli/bin.js run --config e2e.browser-run.config.ts

Both scopes exercise forms, browser operations, cookie and storage isolation, network routes, dialogs, and frames, with traces and WebM video enabled. Worker scope also runs the todos suite, including session persistence. These checks use a real Cloudflare account and run by hand.

History navigation currently fails on Browser Run while waiting for load after browser.back(). To check the supported flows separately, add --grep-invert 'navigation verbs'; keep the unfiltered result when reporting compatibility.

Run on Moli

Install the Moli binary on PATH, or set MOLI_PATH. From the repository root:

pnpm install --frozen-lockfile
pnpm build
CI=1 pnpm --filter @e2e-dev/testbed test:moli

For a smoke check:

cd apps/testbed
node node_modules/e2e/dist/cli/bin.js run tests/smoke.e2e.ts tests/dashboard.e2e.ts --config e2e.moli.config.ts --output .e2e-moli-smoke

Set MOLI_CDP_URL to attach to an existing server, which the run leaves running. Reports and artifacts land in .e2e-moli/. The config uses 5-second action and 30-second test timeouts, with no retries. Compare runs with the same limits. CI=1 prevents creating screenshot baselines.

Agentic suites

test:agent and test:dogfood spend real model calls, so they never gate a PR: they run on the weekly .github/workflows/agent.yml schedule, by manual dispatch, or by hand. Each config builds its model with the AI SDK's gateway(), pins openai/gpt-6-luna-fast, and honours the testbed's own E2E_MODEL variable so the same suite can be replayed across providers:

AI_GATEWAY_API_KEY=...  pnpm --filter @e2e-dev/testbed test:agent
E2E_MODEL=openai/gpt-6-luna-fast AI_GATEWAY_API_KEY=... pnpm --filter @e2e-dev/testbed test:agent

Agentic assertions are structurally comparable across models, not textually identical, so these tests assert on meaning (toContain) and pair every agentic step with a deterministic locator check.

To see where the tokens went, record the run and open the trace in unbox-ai:

AI_GATEWAY_API_KEY=... pnpm --filter @e2e-dev/testbed test:agent -- --ai-trace
npx unbox-ai .e2e/ai-trace.json          # viewer: treemap, waterfall, diffed turns
npx unbox-ai runs .e2e/ai-trace.json     # one line per agent step, from the terminal

To watch calls land while the suite runs, e2e.agent.config.ts registers the AI SDK devtools recorder when E2E_DEVTOOLS is set; start the live viewer first, in this directory:

npx unbox-ai devtools                    # live viewer on http://localhost:4983
E2E_DEVTOOLS=1 AI_GATEWAY_API_KEY=... pnpm --filter @e2e-dev/testbed test:agent -- --workers 1

The devtools recorder names runs after their first prompt and keeps one database per process, hence --workers 1; the --ai-trace file names runs after the test and step and merges every worker, so use it for anything you want to keep or compare.

Hazards worth knowing

  • fill on a rich-text host resolves successfully and changes nothing (the editor reverts it): a silent no-op only a paired deterministic assertion catches.
  • instanceof across a frame boundary is always false: an element inside an iframe belongs to that frame's realm, so field instanceof HTMLInputElement in a browser.evaluate silently takes the else branch.
  • Agentic assertions must be answerable from one observation. A screenshot cannot show recurrence, so assert state, not history.
  • String text matching is exact by default, inverting the Playwright and Testing-Library default.