* feat(reporters): build the built-in reporters on the public Reporter API
The built-in list, dot, json, junit, and markdown reporters now use only
the public Reporter contract: factories on e2e/reporters, printsTo for the
terminal stream a reporter prints to (one reporter per stream), live
onOutput, onStepPrompt, and onSummary hooks for any number of reporters,
and reportFiles for files the runner writes atomically. The list and dot
ids beside json print to stderr, so list,json now works as dot,json does.
@e2e-dev/html-reporter writes through reportFiles.
* docs(reporters): bring the Japanese and Chinese reporter pages up to date
Translates the reporter reference, the config reporters table, the
--reporter row, and the debugging output paragraph. In English, the name
section now says a reporter may share a built-in's name, and two lines
read less ambiguously.
With `await screen.pause()`, a test is paused at that line, with the app
open, until someone resumes it.
- In a terminal, press Enter to resume the test.
- From another shell in the same directory, run `e2e run --step next`.
A run without a terminal is resumed this way.
- Paused time doesn't count toward the test's timeout.
- With `CI` set, the test fails with `PAUSE_IN_CI`.
- Under `--step`, a test is still paused after `n` or `a`.
- A test isn't paused in an MCP `open_session {test}` session, under
`openHost` or `run()`, or in a core program.
- A test isn't paused under `e2e run --watch` yet.
* feat(cli): write a GitHub Actions or GitLab CI/CD pipeline in e2e init
e2e init asks for a CI pipeline in a project with the web engine (--ci github-actions|gitlab|none). It writes one job for the project's package manager and Node version that installs Chromium, runs e2e run, and keeps the report, and names the secret or CI/CD variable for the model key. It never overwrites a pipeline file; beside an existing .gitlab-ci.yml it writes a file to include.
* fix(cli): handle cubic's findings on the init CI step
Exact GitLab stage names, a distinct job name beside an existing e2e job, loopback baseURL in an existing config, no writes through a symlinked .github or .gitlab, --ci none in the skill, gitlab in the telemetry comment, and the GitLab Node image's major version in the docs.
* refactor(cli): keep the init CI pipelines as YAML templates
The GitHub Actions workflow and the GitLab job live in packages/e2e/templates/ci/ as complete pipelines for npm at the repository root, linted as they are. init loads them with the yaml package's Document API and edits a copy for the project's package manager, folders, Node version, secrets, and target, keeping the comments. The pinned actions now live only in the templates.
* fix(cli): point setup-bun at the package.json of a project in a subfolder
* docs(ci): link the GitLab page from init and the GitLab template
* feat(cli): write the iOS and Android CI jobs in e2e init
init writes one job per platform the config's targets run on: web targets in the web job, iOS and Android targets in their own jobs, which build the app, boot a simulator or an emulator, and install the app. Targets come from the new config, or from loading an existing one after the confirmation and the install. Expo and bare React Native apps get their build; any other app gets a step to replace. GitHub's mobile jobs follow e2e-inc/e2e-react-native's workflow; GitLab's run on self-managed runners.
* fix(cli): reinstall an emulator system image that holds no image in the Android CI job
* fix(cli): handle cubic's findings on the mobile CI jobs
Upload the emulator log from the GitLab Android job, count peer and optional engine dependencies, ask nothing for a new config with no engine, key caches on Bun's bun.lockb when that is the lockfile, and run the symlink test with a Windows junction.
* fix(cli): put current Android command-line tools in place before the emulator image
The runner image's cmdline-tools can be older than the system image's package format, and android-emulator-runner uses them to create the AVD, which then fails with 'contains no system images'. The Android job downloads command-line tools 23.0, checks its SHA-256, and installs the image with it.
* fix(cli): import moved core modules from @e2e-dev/core in init
* fix(cli): count a pipeline as set up only when a command runs e2e run
init matched the text 'e2e run' anywhere in a workflow, so a comment or a step name such as 'our old e2e runner' made --ci write nothing. It now reads the commands (run, script, before_script, after_script) from the parsed YAML and matches e2e run as words of their own.
Also: --ci help and the InitOptions JSDoc say tests on every platform, and yaml is pinned exactly like the other dependencies.
* chore(ci): bump setup-node and upload-artifact pins in the init templates, docs, and skill
docs/ci/vercel-previews.mdx on main pins setup-node v7.1.0 and upload-artifact v7.0.2, the pins this repository's own workflows run. The templates, three docs pages, and the skill still had v7.0.0 and v7.0.1, so the pin test failed once main was merged in.
---------
Co-authored-by: Oskar Kwaśniewski <oskar@okwasniewski.com>
* refactor: move e2e's core into @e2e-dev/core, and engines peer on it
The engine contract, locators, agent, replay cache, secrets, and the
session half of a worker move from packages/e2e to packages/core, the new
@e2e-dev/core package, at the same paths. e2e depends on it at its exact
version (a changesets fixed group) and reaches its modules through
@e2e-dev/core/lib/*, which is not public API; e2e/engine and e2e/agent
re-export it. @e2e-dev/web and @e2e-dev/mobile peer on @e2e-dev/core and
import the contract from @e2e-dev/core/engine. Install smoke checks that one
copy of @e2e-dev/core is installed.
Most of this diff is git mv and import specifiers rewritten to
@e2e-dev/core/lib/<path>; no behavior changes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ci: build and pack @e2e-dev/core where e2e is built or packed alone
The Bun job built e2e before the @e2e-dev/core it compiles against, and the
examples nightly installs e2e from a tarball whose exact @e2e-dev/core is
not on npm; it now packs @e2e-dev/core and overrides it in each example.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* docs(engines): name @e2e-dev/core/engine as the contract's source
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(e2e): list @e2e-dev/core beside an engine in init and upgrade
Engines peer on @e2e-dev/core. Yarn installs no peer the project does
not list, and under Plug'n'Play the engine then fails to load.
- init adds @e2e-dev/core with any engine
- upgrade adds a required @e2e-dev peer the project lacks, and keeps a
package e2e depends on at the version e2e names
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Oskar Kwaśniewski <oskar@okwasniewski.com>
The --output row in the CLI reference and the bundled running guide never said the directory must be inside the project root and not the root itself, so a run with an output outside it failed INVALID_CONFIG with no warning in the docs. Fixes ENG-976.
* feat(runner): start the tests a change is likeliest to break first
A test filter can score the tests it keeps: `relevance` in its result maps test ids to a number from 0 to 1. With `--changed-since`, each target starts the files of the likeliest tests first, in bands at 75%, 50%, and 25%, and the target with the likeliest file goes first. Inside a band the longest-first order from the previous run still decides, with more than one worker.
`decisionFilter()` scores every test it hands back: 1 for a test whose file, setup file, or stored screenshots changed, the model's answer for one it judged, and 0.5 for one it did not. It requires e2e 0.23.1 or newer, since an older runner refuses the unknown `relevance` key and drops the whole filter.
The run prints `ranked N of M tests by relevance, likeliest first` when the scores change the order.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
* test(runner): write the failing fixture as a constant, not built from a parameter
CodeQL flagged `failingFile(title)` for putting `JSON.stringify` output into generated code. Only `gamma` used it, so the file is a constant now.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
* docs(runner): say how targets, bands, and the ranking line are decided
Review of #1589 found the docs looser than the code: targets go by the band of their best file, not its exact score, with ties in config order; a rating the model did not give counts as 50% and the test takes the higher of its two ratings; the bands are half-open; the ranking line follows when the scores put tests in more than one band; and a filter that returns no scores leaves the order alone. One test title said an empty target goes last, but it keeps its place.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
* docs(runner): score known areas on the fallback path and name files, not tests
The areas example now scores the tests whose area a change reaches even when another changed file maps to no area, keeping the no-exclusion fallback. The README, docs, and skill say the run starts the files of the likeliest tests, ordered by relevance band with collection order breaking ties on one worker. The timings test makes beta and gamma wait for each other before they sleep, so a slow second worker cannot start alpha ahead of gamma.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
* fix(runner): keep the scores of a filter that fails
A filter that threw left every test unscored, so a run with no AI Gateway
key (a pull request from a fork) or a model outage fell back to collection
order and lost the one thing the decision filter knows without the model:
which tests the change touches.
The error a filter throws may now carry `relevance`. The runner checks it
the way it checks a result's, merges it by the highest score, and still
warns, leaves nothing out, and records `outcome: failed`. Scores that fail
the check are dropped, and the note says why. A call past its deadline is
abandoned, and nothing it throws is read.
decisionFilter() puts the scores the change alone gives (1 for a test the
change touches, 0.5 for the rest) on the error it throws, as a property
that is not enumerable, so printing the error does not list every test.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
* docs(runner): point the ordering prose at the timings history
The timings history (#1639) replaced the previous report as where file
durations come from, so the prose that says what orders files inside a
relevance band now names it: the filters guide, the `workers` bullet in
the config reference, the CI guide, the skill's running reference, and the
wording of the integration test that pins the one-worker gate.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
* feat(run): keep a timings history that CI can cache
`e2e run` already starts the longest files first, but it read the
previous run's durations out of report.json, which a fresh CI checkout
never has. Write a small versioned `<output>/timings.json` (timings-1)
at the end of every run instead: per target::file milliseconds, smoothed
with an exponential moving average, with files that are gone and targets
the config dropped pruned.
The file is advisory. A missing, corrupt, or schema-invalid file is no
history, and an absent one falls back to report.json as before, so a
restored cache can reorder files and never change a result.
`--timings <file>` moves it. `--last-failed` still reads report.json.
Documents the actions/cache recipe for GitHub Actions and the generic
restore/save rule in the CI guide. Duration-balanced --shard is left for
a separate decision.
Refs ENG-1111
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
* fix(run): time a file only when the run carried out all of it
A `--grep`, `--last-failed`, interrupted, or `--repeat-each` run summed
only part of a file's tests, or several repeats of them, into the
timings history, so a 60 s file could drop to about 40 s after a rerun
of one failing test. A file now has a timing only when none of its tests
was left out, cut short, or never reached, and repeats past the first do
not count. Adds the integration test for the runner wiring.
Refs ENG-1111
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
* fix(run): tighten the timings keys, count an interrupted retry as partial, and say when ordering applies
- A retry cut short after a failure leaves the result under the earlier
verdict; read the attempts, and a serial group's, for an interrupt.
- timings-1 keys: a target name, then a project-relative POSIX file with
no `..` segment, in the schema and the reader alike.
- Rerun cache key adds github.run_attempt, since a rerun shares run_id
and actions/cache never saves over an existing key.
- Docs and help say the history orders files only with more than one
worker, and that the file is `<output>/timings.json`.
Refs ENG-1111
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
* docs(run): say files with no recorded time start before the timed ones
Refs ENG-1111
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
* fix(run): time a file whose other tests a platforms list keeps off the target
A test a `platforms` list excludes is `selected: false` with cause
`platform-unavailable`, the same flag a `--grep` or `--last-failed` sets, so a
file holding both an iOS-only and an Android-only test never got a time on
either target. Only a filter makes a file partial; a permanent platform
exclusion does not.
Refs ENG-1111
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
* docs: retire sidebar page icons
The docs sidebar no longer shows icons. Drop icon: from every page's frontmatter, delete the grey silhouettes in docs/images/icons/, and make check-docs-logos fail on any frontmatter icon instead of requiring one on vendor pages. Card logos stay required. The authoring-docs skill trades its Page icons section for a short Vendor logos one.
* docs: require a migrate overview card for each migration guide
* docs: match only top-level icon keys in check-docs-logos
* feat(cli): draw the new e2e logo in init and help
* docs(cli): soften the bar-to-gap comment
* docs(cli): state the init logo's width limit
* feat(cli): load the init logo top to bottom like a scan
* feat(cli): draw the init logo's bars as full-block rows
* feat(cli): scan the init logo in as thin lines, then fill the bars
* feat(cli): scan the init logo in row by row
* test(cli): require the logo scan to add one row at a time
* feat(cli): draw each logo bar as one full-block row
* feat(cli): reveal the init logo with a sweeping scan line
* feat(cli): shade the init logo in
The engine question is a multiselect and --engine takes a list. With both engines the config gets a target per engine and the examples split into tests/web.e2e.ts and tests/mobile.e2e.ts, each requiring its engine's fixture.
`--reporter ''`, `--reporter ','`, or `--reporter "$REPORTERS"` with the
variable unset parsed to an empty list that replaced every configured
reporter, so a failing run printed nothing and exited 1. It is now a usage
error, exit 2, like an empty --tag, --target, or --agent. Applies to run
and explore, which share the parser.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* docs(reference): rewrite the API reference in the Next.js shape
Every reference page now opens with a lede and a minimal example, groups its
API under ## sections that start with a summary table linking to one ###
entry per member, and keeps the TypeScript declarations under ## Types.
* docs(reference): fix claims the review caught
Narrow the security summary rows to their sections, generate the screenshot
file name in the engine example instead of building it from the label, and
correct defaults, exit codes, and examples on config, cli, screen, web,
mobile, test, runner, and engine.
* docs(reference): address review findings
Scope error classes to runner errors, guard a zero-slot prepare in the engine
example, keep the cli #help-and-version anchor, and correct poll cancellation,
same-run screenshots, openLink schemes, type_secret, longPress durations,
runner signals and close failures, empty usernames, and the feedback exit
code. Make the reporter and extract examples self-contained.
* docs(reference): trim the summary tables to the minimum
Each group's summary table keeps the linked name and one or two short
columns (type, default, or a few-word description). Examples and details
live in the entries; facts that only a table cell held moved into its entry.
The expect overview is headed Reference instead of Entry points, and the
security rules table names what each section covers instead of restating
its claims.
* docs(reference): open every page's API under one Reference heading
Each page's first section after the lede is now ## Reference. A page with
one group lists its entries at ### right under it; a page with several keeps
each group as a ### with its summary table and its entries at ####.
Explanatory sections (behavior, budgets, setup, types) follow it. Linked
sub-parts that moved to ##### keep their anchors through explicit ids.
* docs(skills): describe the reference page shape in authoring-docs
Reference pages open their API under ## Reference, keep summary tables to
the name and one or two short columns, and nest entries at ### or ####
because Mintlify drops anchors below ####. ship-pr points reference edits
at these rules instead of the guide shortening pass.
* docs(reference): address cubic review on tables and skill
Keep readonly in the mobile device type, show the keyboard and mouse
object shapes, say a locator assertion times out as ASSERTION_FAILED, and
give reference pages a narrower shortening pass in authoring-docs.
* docs(reference): tighten the reference pages
Merge facts said twice into one place, move each declaration's JSDoc to a
single home, link timeout defaults, key grammar, and secure-field rules to
the pages that own them, and split long sentences. No fact is cut.
* docs(reference): address cubic review of the tightening pass
Qualify the GitHub token for fork pull requests, provider recording URLs,
duplicate device entries, MODEL_UNAVAILABLE timing, the APP_UNREACHABLE log
quote, UNKNOWN_TOOL versus UNSUPPORTED_CAPABILITY, node-less agent tools,
which model receives screenshots, passing assert evidence, and config-path
hosts; fix three garbled sentences.
* docs(reference): say device, not simulator, for worker isolation
* feat(cli): add e2e run --max-duration and maxDuration config
* fix(cli): keep e2e explore on its own --timeout, not maxDuration
* fix(cli): skip max-duration timer past setTimeout cap, cover config and explore paths
* refactor(cli): address max-duration review
- explore drops maxDuration from its config; runner no longer special-cases it
- scheduler records RUN_TIMEOUT from the skip reason, naming --max-duration or maxDuration
- stopEarly wakes the loop itself
- reject maxDuration above 2147483647 ms instead of skipping the timer
- trim cli docs, clarify ci wording, sync plugin
* feat(feedback): send bugs without a repro to GitHub, refuse outdated reports
* fix(feedback): redact error code, qualify version check docs, valid skill example
* feat(cli): flag per e2e init wizard question
--engine, --gateway, --endpoint, --skills, --mcp, --install/--no-install. A flag skips its prompt; omitted keeps the --yes default. Closes#1114.
* refactor(cli): address init flags review
- move init flags to cli/init/flags.ts, share parseNames
- --endpoint implies --gateway openai-compatible
- fold --yes defaults into one answers object
- --skills/--mcp name exact set on re-run, test .agents untouched
- docs: non-TTY needs --yes; hold migrate prompt edits until release
* feat(config): E2E_CACHE_DIR and --cache-dir override cache.dir
* fix(config): refuse --cache-dir beside cache.store, keep shared cache out of tests
* fix(config): resolve cache dir override with store, expand ~, share path display
- E2E_CACHE_DIR ignored beside cache.store (was rejecting default output)
- leading ~ in E2E_CACHE_DIR and --cache-dir is home dir
- displayPath helper for stale advice and trace pages
- move e2e cache registration to cli/cache.ts, help helpers to cli/help.ts
- docs: shared-cache caveats, file store only
* docs(cache): qualify shared-dir overwrite and ~ expansion