@e2e-dev/testbed
Dogfood workspace for the e2e runner: a real project consuming the
built e2e package exactly like a user would, against playground apps we
control. This is where every runner feature has a deterministic test. Hard UI
surfaces live in the benchmarks (web-benchmark, mobile-benchmark), not
here.
Layout
app/server.mjs— dependency-free playground app. The runner starts and stops it via the web engine'scommandoption. The pages live underapp/pages/by group, each module exporting its routes and its nav entries:basics(home, todos, forms, login/session, dashboard, wizard),interaction(network, dialogs, board, pointer pad, iframes),canvas(four pixels-only surfaces),downloads(a file and a long page), andcontrols(control states, scrolling, the browser fixture's page, the speed counter, about).app/pages/index.mjsmerges them andapp/layout.mjsrenders the nav from that registry.app/playwright/: Playwright's test assets (v1.64.1, Apache-2.0, with itsLICENSEandNOTICE) and the dynamic routes its test server sets, served by the playground for any path it has no page for.app/bug-garden.mjs— a bookshop with planted defects, fore2e explore.dogfood/server.mjs— an expense-claims app with a test API, for the executor dogfoods.e2e.config.ts+tests/— the local suite, gating every PR: queries, actions, polling assertions, sessions (test.setup+session:), serial groups, routes, dialogs, frames, downloads, uploads, keyboard input, credentials and secrets, plus one file per surface added since: control states and the failure codes (controls), pointer coordinates and path swipes (pointer), viewport and node scrolling (scroll), thebrowserfixture's own verbs (browser), and the per-action speed floor (speed). The files taggedplaywrightare Playwright's owntests/pagespecs (andtests/library/download.spec.ts) at v1.64.1, ported to the public API: each header names its source and the tests left out.fail: 'reason'marks a gap to close;skip: 'reason'one that fails with an errorfaildoes not cover. A difference e2e keeps on purpose is left out in the header and stated indocs/migrate/playwright.mdx.--tag playwrightruns only them, and CI runs them on every pull request.tests/helpers.tsholds the sharedfailure,boxOf, andcenterOf, and the ported tests'content,asset,PREFIX, andCROSS.e2e.notte.config.ts: the local suite on@e2e-dev/nottecloud browsers, run by hand withNOTTE_API_KEYandE2E_NOTTE_APP_URL, a public tunnel to the playground (pnpm app). Tests that open127.0.0.1orlocalhostdirectly fail there, since the browser runs in Notte's cloud.e2e.lightpanda.config.tsande2e.lightpanda-agent.config.ts: the local suite and the agent suite on@e2e-dev/lightpanda, which starts Lightpanda fromLIGHTPANDA_PATH(orPATH), run by hand to see which features its nightly covers; the docs pageintegrations/lightpandalists the result.e2e.moli.config.ts: the full local suite on@e2e-dev/moli, which starts and stops Moli for the run, with one worker. Reports and artifacts land in.e2e-moli/.e2e.scroll.config.ts+tests-scroll/—scroll_toby text over the windowed/scrollpage, driven by a scripted executor, so it needs no model. Its own config, since it makes the default agent the scripted executor; it runs astest:scroll, its own CI step, and never insidetest, whose forwarded arguments (--changed-since) would not reach it.e2e.agent.config.ts+tests-agent/— opt-in agentic suite against the playground:agent.actflows, assisted polling, judgments, schema-validated extraction with zod, and the pixel tools on the canvas pages.e2e.mixed.config.tsadds the local suite to it for watching the list reporter interleave the two.e2e.dogfood.config.ts+tests-dogfood/— the built-in agent extended with project tools (seed and reset over the expense app's test API).e2e.dogfood-brain.config.ts+tests-dogfood-brain/— a hand-rolledStepExecutorwith its own tools and transport; the engine is never touched.e2e.dogfood-edge.config.ts+tests-dogfood-edge/— steps expected to conclude blocked or failed; the report, not the exit code, is the output.e2e.stress.config.ts+tests-stress/— failures, timeouts, skips, flakes, and hostile titles for the reporters.e2e.explore.config.ts+scripts/explore-bench.mjs—e2e exploreagainst the bug garden, and a scorer that runs it across models.e2e.scratch.config.ts+tests-scratch/— a scratch agent config for one-off investigations with the AI SDK devtools recorder.
Commands
pnpm --filter e2e build # the testbed runs the built runner
pnpm --filter @e2e-dev/testbed test # typecheck + local suite (starts the app itself)
pnpm --filter @e2e-dev/testbed test:headed
pnpm --filter @e2e-dev/testbed app # run the playground manually
AI_GATEWAY_API_KEY=... pnpm --filter @e2e-dev/testbed test:agent # real model calls
AI_GATEWAY_API_KEY=... pnpm --filter @e2e-dev/testbed test:dogfood
AI_GATEWAY_API_KEY=... pnpm --filter @e2e-dev/testbed explore:garden
pnpm --filter @e2e-dev/testbed test:stress # reporters only
The local suite runs in CI on every push. Reports land in .e2e/report.json;
artifacts and trace pages under .e2e/results/.
Browser Run live checks
e2e.browser-run.config.ts runs against a public HTTPS deployment of the
playground pages. Set BROWSER_RUN_TEST_URL, CLOUDFLARE_ACCOUNT_ID, and
CLOUDFLARE_API_TOKEN with Browser Rendering - Edit permission, then build
the workspace packages and run from this directory:
node ../../packages/e2e/dist/cli/bin.js run --config e2e.browser-run.config.ts
BROWSER_RUN_SCOPE=attempt node ../../packages/e2e/dist/cli/bin.js run --config e2e.browser-run.config.ts
Both scopes exercise forms, browser operations, cookie and storage isolation, network routes, dialogs, and frames, with traces and WebM video enabled. Worker scope also runs the todos suite, including session persistence. These checks use a real Cloudflare account and run by hand.
History navigation currently fails on Browser Run while waiting for load
after browser.back(). To check the supported flows separately, add
--grep-invert 'navigation verbs'; keep the unfiltered result when reporting
compatibility.
Run on Moli
Install the Moli binary
on PATH, or set MOLI_PATH. From the repository root:
pnpm install --frozen-lockfile
pnpm build
CI=1 pnpm --filter @e2e-dev/testbed test:moli
For a smoke check:
cd apps/testbed
node node_modules/e2e/dist/cli/bin.js run tests/smoke.e2e.ts tests/dashboard.e2e.ts --config e2e.moli.config.ts --output .e2e-moli-smoke
Set MOLI_CDP_URL to attach to an existing server, which the run leaves running.
Reports and artifacts land in .e2e-moli/. The config uses 5-second action
and 30-second test timeouts, with no retries. Compare runs with the same limits.
CI=1 prevents creating screenshot baselines.
Agentic suites
test:agent and test:dogfood spend real model calls, so they never gate a
PR: they run on the weekly .github/workflows/agent.yml schedule, by manual
dispatch, or by hand. Each config builds its model with the AI SDK's
gateway(), pins openai/gpt-6-luna-fast, and honours the testbed's own
E2E_MODEL variable so the same suite can be replayed across providers:
AI_GATEWAY_API_KEY=... pnpm --filter @e2e-dev/testbed test:agent
E2E_MODEL=openai/gpt-6-luna-fast AI_GATEWAY_API_KEY=... pnpm --filter @e2e-dev/testbed test:agent
Agentic assertions are structurally comparable across models, not textually
identical, so these tests assert on meaning (toContain) and pair every
agentic step with a deterministic locator check.
To see where the tokens went, record the run and open the trace in unbox-ai:
AI_GATEWAY_API_KEY=... pnpm --filter @e2e-dev/testbed test:agent -- --ai-trace
npx unbox-ai .e2e/ai-trace.json # viewer: treemap, waterfall, diffed turns
npx unbox-ai runs .e2e/ai-trace.json # one line per agent step, from the terminal
To watch calls land while the suite runs, e2e.agent.config.ts registers the
AI SDK devtools recorder when E2E_DEVTOOLS is set; start the live viewer
first, in this directory:
npx unbox-ai devtools # live viewer on http://localhost:4983
E2E_DEVTOOLS=1 AI_GATEWAY_API_KEY=... pnpm --filter @e2e-dev/testbed test:agent -- --workers 1
The devtools recorder names runs after their first prompt and keeps one
database per process, hence --workers 1; the --ai-trace file names runs
after the test and step and merges every worker, so use it for anything you
want to keep or compare.
Hazards worth knowing
fillon a rich-text host resolves successfully and changes nothing (the editor reverts it): a silent no-op only a paired deterministic assertion catches.instanceofacross a frame boundary is always false: an element inside an iframe belongs to that frame's realm, sofield instanceof HTMLInputElementin abrowser.evaluatesilently takes the else branch.- Agentic assertions must be answerable from one observation. A screenshot cannot show recurrence, so assert state, not history.
- String text matching is exact by default, inverting the Playwright and Testing-Library default.