Commit Graph
506 Commits
Author SHA1 Message Date
Satyajit Sahoo 7e3205461b feat(expect): expect(fn).toPass() and expect.poll intervals (#2013)
* feat(expect)!: expect(fn).toPass() and expect.poll intervals

A block of assertions can be retried with `expect(fn).toPass({ timeout,
intervals })`, including the `.not.toPass()` and `expect.soft(fn).toPass()`
forms. Defaults are set with a top-level `toPass` config key. Both APIs
run on one retry loop.

BREAKING CHANGE: `expect.poll` no longer takes `interval`. It takes
`intervals`, a list of pauses whose last entry repeats. The default pauses
are `[100, 250, 500, 1000]` ms where they were a flat 100 ms.

Steps still running in a poll read or a toPass try are cancelled when the
try fails or is cut at its deadline. Later step calls from that try are
refused.

When a retry reaches the phase deadline, the runner's timeout is the
reported error. The last retry error is kept under `secondaryErrors`.

In `beforeAll`, `afterAll`, and worker fixture teardown, both APIs are
bounded by the hook deadline and use the config defaults.

A call `timeout` above the timer limit is `INVALID_CONFIG`.

* fix(expect): let skips through a retry and follow a moved deadline

A `test.skip()` or `test.fixme()` thrown inside a toPass block or a poll
read ends the retry right away and propagates unchanged.

The phase deadline is read on every pass, so a `test.setTimeout()` call
during a retry takes effect. The call's timeout stays fixed.

Options for `expect.poll` and `toPass` must be a plain object with known
keys.

* fix(expect): deprecate the interval option of expect.poll

`interval` works again as an alias for `intervals: [interval]`, following
the deprecation window in CONTRIBUTING.md. A warning that names `intervals`
is printed the first time it is used in a process. Passing both options is
`INVALID_CONFIG`.

* fix(expect): take a final sample at the deadline and refuse late ones

When a pause is cut short by the deadline, one more sample is taken. Its
result counts only if it settles without waiting past the deadline. A
sample that overran the time it had when it started no longer passes.

Call options for `intervals` use the config's rule, positive integers.

* fix(expect): drop the final sample at the deadline

The extra read counted only when it settled in its starting turn, so it
helped a synchronous read and never one that awaits the app or the
network. When the next pause would cross the deadline, the retry ends
without another read, as Playwright's loop does.

* fix(expect): keep a late try's skip and error

A try that settles after its time still does not pass. A `test.skip()` it
threw now reports `skipped`, and an error it threw is the `last:` line of
the timeout message.
2026-11-22 02:37:02 +00:00
Kunal Roy ChoudhuryandKunal Roy Choudhury 851127396c fix(web): refuse an unknown browser name or a malformed viewport in web() (#2037)
* fix(web): refuse an unknown browser name or a malformed viewport in web()

* docs(skills): note the browser and viewport config errors

* refactor(web): resolve the browser option in one place and sync the plugin copy

---------

Co-authored-by: Kunal Roy Choudhury <71250874+krockxz@users.noreply.github.com>
2026-11-22 02:29:32 +00:00
Vishwas Tiwari c30354ff84 feat(test): add test.describe.fail and the fail option on describe (#1925)
* feat(test): add test.describe.fail and the fail option on describe

A test now reads fail from its groups as well as itself. Any truthy layer marks it, the innermost reason wins, and fail: false does not unmark. Refused on, inside, or around a serial group. Part of #1855.

* test(collect): check an inner group's fail reason wins over an outer one

* fix(test): type describe without a cast and point the test.fail error at describe.fail

The collection error for test.fail(condition) outside a body now names describe.fail and the describe option, since a migrating Playwright user hits it first. The Playwright migrate row gains the conditional form, and the ja/zh rows take cubic's translations.

* fix(test): validate describe.fail options before adding the mark

describe.fail copied its options with a spread before validation, so an array or a non-enumerable bad key passed where describe() rejects it. The caller's options are now validated first, then again with fail: true for the serial rules. The fail docs now say true or any string marks, since a blank string marks too.

* docs(test): say fail is refused around a serial group too

* refactor(test): validate describe options in one pass

describe.fail no longer validates twice: the serial check takes the mark into account, and registerDescribe takes api and markFail without defaults.
2026-11-21 16:25:49 +01:00
f8aa3131cb fix(web): read content hidden under a closed details across shadow roots (#2007)
* fix(web): read content hidden under a closed details across shadow roots

isInClosedDetails walked up with closest('details'), which stops at a
shadow boundary, so a shadow tree under a closed <details>, or an element
slotted into one, read as visible and passed toBeVisible(). It now walks
the flat tree with flatParentOf, the change from #1503's f57eeb7e3. It
knows a <details> and its first <summary> by name and the HTML namespace,
so a <details> adopted from another frame still folds.

isSkippedContent stepped up with assignedSlot, which is null for a slot in
a closed root, so content slotted into a closed root under
content-visibility: hidden read as visible. It now steps with flatParentOf
too.

slotOf, which the visibility walk now asks for every light child of a
closed root's host, scanned every slot's assigned elements per element.
It now reads a closed root's HTML slots once per read into a map, by exact
slot name or, for a root that assigns by hand, by element.

Co-authored-by: Oskar Kwaśniewski <oskar@okwasniewski.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(web): fold and skip only what the browser hides, once per read

A closed <details> folds its body when its ::details-content computes
content-visibility: hidden, so a page whose CSS shows a closed body reads
visible; without that pseudo element the open attribute decides. Its
first HTML <summary> child is looked up once per read, so a wide
<details> is walked once instead of once per child.

content-visibility: hidden skips nothing on display: contents, an inline
element, a table row or row group, or ruby text, which checkVisibility()
reads as visible in Chromium. One predicate, skipsContents, answers that
for isSkippedContent, the tree walk and directTextOf, so a control that
reads visible is also in the agent's tree.

The e2e-shown, e2e-read and e2e-label selector engines read each
candidate in its own reader call. They now pass the reader one closed-slot
map for a synchronous selector operation and drop it in a microtask, so a
locator reads a closed root's slots once per operation, not per match.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor(web): drop the selector-operation slot cache, read slots plainly

The e2e-shown, e2e-read and e2e-label selector engines are back to main:
they no longer share a closed-slot map across the reads of one selector
operation, which only paid off on contrived pages. Within one read,
slotOf now maps each element a closed root's HTML slots hold to its slot
from assignedElements, the browser's own assignment, instead of
reimplementing named assignment. Comments are shorter.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(web): skip content only where the box skips it, and find closed slots by name

content-visibility: hidden skips nothing on a table, an inline table,
ruby, or an inline list item, as Chromium paints them, and a closed
<details> body laid out with ::details-content { display: contents }
renders, so both read visible. An HTML <button> or <fieldset> makes a box
of its own whatever its display, short of display: contents, so content
under one reads hidden. In SVG only an outermost <svg> has a box for the
property to act on: it skips whatever its display, and other SVG
elements, a nested <svg> included, ignore it.

In a closed root that assigns by name, slotOf finds an element's slot
from its slot attribute, the first HTML slot of that name, both read
outside any namespace, so a locator's read of one match never lists what
the slots hold. A root that assigns by hand still reads assignedElements().

isInClosedDetails asks whether a <details> folds its body before looking
for its summary, so an open one never looks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: michal-e2e <357875193+michal-e2e@users.noreply.github.com>
Co-authored-by: Oskar Kwaśniewski <oskar@okwasniewski.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-21 16:14:17 +01:00
jaidev 003c18a4ee feat(web): match route and waitForResponse by method (#1924)
`browser.route`, `browser.unroute`, and `browser.waitForResponse` accept a `{ method, url }` RouteMatcher. The method is case-insensitive, `url` keeps the string or RegExp grammar, and a route narrowed to another method steps aside for older routes and the network. `unroute` removes routes registered with the same pattern, method included.
2026-11-21 14:45:01 +00:00
Donny/강동윤andClaude Opus 5.5 a2d0363128 feat(desktop): add the @e2e-dev/desktop engine on agent-desktop (#1953)
* feat(desktop): add the @e2e-dev/desktop engine on agent-desktop

A desktop engine for native and Electron apps, built on agent-desktop
(vercel-labs, Apache-2.0), which reads and drives apps through the OS
accessibility tree. macOS is the tested platform; this is rollout phase 1
of the ENG-1352 proposal.

- The engine downloads the pinned agent-desktop 1.3.1 release in prepare
  and checks its SHA-256; the npm package is not a dependency because its
  postinstall would fail check:install-scripts.
- Fill text reaches agent-desktop on stdin, never in its arguments.
  Screenshots are cropped to the window and paint every secure field black,
  and are withheld when a secure field has no bounds or the snapshot was
  partial.
- Node ids stay stable across observations, so the agent gets diffs.
- The engine quits only the process it launched, checked by its process
  instance.
- apps/desktop-testbed holds an AppKit sign-in form with deterministic and
  agentic suites.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(desktop): accept the remaining budget in the fill arguments

The timeout agent-desktop gets is what is left of the operation's
deadline, so it can be a millisecond under the full budget on a slow
runner.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(desktop): report field values whole and keep a typed-into field's id

locate answers single-node reads, which the contract says carry the whole
value; the harness cuts what the agent sees. An unnamed field's identity no
longer includes its value, so typing into it is a change, not a remove and
an add. An image is projected as the contract role image, which both
locator spellings resolve to.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(desktop): withhold a capture with a border instead of masking it in place

A capture within four pixels of the window was taken as the window, so a
two-pixel border shifted every mask. Only a pixel of rounding is accepted
now, any other capture is cropped to its opaque box, and each mask reaches
two pixels past its field.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(desktop): give a read exactly its budget

The 500 ms floor exists so a click that landed is not reported as cut off;
a read has no such reason and now stops at the operation's deadline.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(desktop): keep agent-desktop out of the replay cache and report install failures

E2E_CACHE_DIR names the replay cache, which a project may commit, so the
binary goes to the OS cache directory only, found from the run's
environment. A network or tar failure in prepare is now
AGENT_DESKTOP_INSTALL_FAILED, as the docs say.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(desktop): offer doubleTap only with headed, and say what app.open and settle do

agent-desktop double-clicks with the real pointer only, so doubleTap is
declared with headed: true and sent with --headed. The docs said app.open()
quits the app if it runs; it quits only the copy the engine launched. settle
is a fixed wait, not a check that the UI is still.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(desktop): drive only the launched process, and keep ids and the select ladder honest

- Every read and scoped command goes to a window of the process the engine
  launched; before app.open() or after the app quit, the engine reads and
  drives nothing, so a developer's own copy is never touched.
- With appPath, the launched executable must come from that bundle; another
  copy registered under the same bundle id fails the open.
- quit keeps the process as the engine's own until it is known to be gone,
  so a failed lookup is reported and dispose tries again.
- Node identities are scoped to their window and retired once gone or
  ambiguous, so neither a twin nor another window's control inherits an id.
- The select ladder passes the label after --, reports a stale ref as
  NODE_STALE, and stops once any command of a way reached the app.
- A press on a control that refuses focus is refused unless it holds focus.
- A root swipe after a restart reads the window again.
- app.open() waits at most 30 s for a readable window, reports any other
  read failure as it is, and is cancelled with its operation.
- A screenshot is withheld when the window read after the capture differs
  in its secure fields.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(desktop): read placeholders, mixed checkboxes, and depth-cut trees as agent-desktop reports them

- A placeholder comes from agent-desktop's own placeholder field; the hint
  is kept as an attribute of its own.
- A mixed checkbox reports no checked state, so check() and uncheck() always
  reach the control instead of returning early.
- A node that counts children it did not list (the depth limit) marks the
  tree as cut short, so a screenshot from it is withheld.
- Siblings that share a developer id (AppKit menu items) tell themselves
  apart by name and keep their ids; a status with a developer id keeps its
  id while its text changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(desktop): launch an appPath by path and keep every process a launch started

- An appPath bundle is opened by its path (open -n), and the new process
  running from it is taken as the engine's own; launching by bundle id let
  LaunchServices pick another copy with the same id.
- A launch that failed after the app started still leaves the engine owning
  the new process, so the end of the attempt quits it.
- A window is the engine's only when its process instance matches too, so a
  later process given the same pid is never read or driven.
- An app list that is incomplete keeps the process the engine's, to quit
  again in dispose.
- A point action finds the window where it is now, so it works right after
  app.open() and after the window moved.
- A type or fill after a clear that changed the field fails as
  ACTION_MAY_HAVE_COMMITTED, never as a retryable stale ref.
- A screen larger than the binding cap keeps every node it just read.
- The launch wait is 10 s, and each read in it gets what is left of it.
- A bundle id that names no app says so, instead of asking for app.open().

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(desktop): refuse an archive with the wrong SHA-256 and cache nothing

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(desktop): say app.open brings the app forward, and that only macOS is tested

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(desktop): never take another copy as the launched one, and aim point actions at their window

- A launch agent-desktop refused takes no process as its own, and a copy is
  attributed to a launch by id only against a whole app list from before it,
  so cleanup never kills the developer's copy.
- A launch whose outcome is unknown is kept pending and settled by the next
  quit under its own budget, so a timed-out launch still has its app quit.
- A point action goes to the window it was observed in, wherever it is now,
  refuses a point outside that window, and brings it forward before the
  physical click.
- In the select ladder, ELEMENT_NOT_FOUND from select (no such option) goes
  on to the next way; from a command on the list itself it is NODE_STALE.
- A root swipe scrolls the window as it is now.
- With both appPath and bundleId, the appPath build is launched and its
  bundle id must match.
- An open that runs out of its own budget is OPERATION_TIMEOUT; only the full
  launch wait reports an unreadable window.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(desktop-testbed): check the saved screenshot for the password field's mask

The test checked only that the file was a PNG, which an unmasked capture
also is. It now decodes the image and requires the solid black bar the
password field's mask paints.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(desktop): say which app launches when appPath and bundleId are both set

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(desktop): report a read agent-desktop timed out as OPERATION_TIMEOUT

TIMEOUT was NOT_ACTIONABLE for reads as well as inputs, so a snapshot that
ran out of time skipped the launch wait's timeout handling.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(desktop): tell the launched process apart by a token, and measure points in their window

- Every launch passes a one-time token in the app's environment
  (launch --env, open --env), and a process is the engine's own only when
  ps -E shows that token. Guessing from the app list before and after a
  launch could take, and kill, a copy the developer or another worker
  started at the same time.
- A launch whose outcome is unknown is settled by the next quit, which looks
  for the token for up to 3 s and reports a failure when it finds none.
- A point is measured in the window of the newest screen observe or locate
  handed out, and sent unrounded, so it never lands past the window's edge.
- check and uncheck always go to agent-desktop, which reads the live state.
- One window's node ids stay while another window is read.
- The launch wait's commands share its deadline.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(desktop): pin option and app validation and the headed-only actions

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(desktop-testbed): measure the password mask at the capture's scale

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(desktop): keep identities from colliding, and never drop a secure node

- A node identity is a tagged tuple, so a developer id and a name can never
  spell another control's identity (`shared` + `Edit:` against `shared:Edit`),
  which let one control take another's binding.
- A secure node is never dropped as a wrapper, so its bounds are masked.
- A binding records the window its node was read in.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(desktop): press keys only into a focused control, type keystrokes, and check the launched process's token

- A press on a control needs its focus to succeed and to hold, and goes into
  the control's own window; before, a refused focus sent the key to
  whatever window had focus.
- The keyboard capability is declared with headed only and types with
  --headed: without physical input agent-desktop sets the text, and the
  contract asks for keystrokes the app's handlers see.
- The process agent-desktop reports for a launch must carry the launch's
  token; a launcher that handed over to a copy already running (a Linux
  desktop file) fails the open and never drives or quits that copy.
- A select that finds no such option never falls through to set-value,
  which would type the label in as free text; a menu left open after a
  missing option is reported as a changed app.
- A read tries the launched app's windows in turn, past a panel
  accessibility cannot read.
- No app bundle is opened once the operation's budget is spent.
- doctor fails an appPath with no Contents/Info.plist.
- The unused latest-screen field is gone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(desktop-testbed): require a mask the size of the password field, not one black row

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* chore(desktop): ship a NOTICE like every other published package

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(desktop): say typing into whatever has focus needs headed

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(desktop): withhold masks with no area, and give controls their title as text

- A secure field whose box has no area masks nothing, so the screenshot is
  withheld.
- A control reports its title as its text, as on a device, so
  textContent() reads a button's title; an input's name is a label, so it
  gets none.
- A binding keeps its node's box, to find a look-alike again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(desktop): find a pressed look-alike by its place, keep unresolved launches, and test on the host's ps

- A press checks focus on the control found again by role and box, since a
  look-alike has a new id in every read; the key is resolved before focus
  moves, and a node read in no window is refused.
- An environment that cannot be read (a macOS system app under SIP, a
  failed lookup) is no evidence: agent-desktop's report of the process it
  launched stands, so system apps open again and are quit at the end.
- A pending launch stays until it is claimed or a whole wait finds none.
- A window row is the engine's only with the launched process instance.
- A screenshot finds secure fields in whole reads of the window, even with
  interactive snapshots, which leave out a secure node with no action.
- A menu read in part never reports an option missing.
- The unit tests read process environments the host's way (ps -E or
  /proc), so they pass on the Linux CI runner, and remove the fake's
  scenario directories.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(desktop): dispose every surface a test creates, so no state directory is left behind

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor(desktop): drop Electron, so the engine targets native apps only

Electron apps are out of scope for this engine. Remove the paths that
existed only for them: the headed type fallback for a field the
accessibility API cannot set (Electron on Linux), and the expand and
menu-click steps of the select ladder (Electron on Windows and macOS).
A select now tries `select`, then sets the value of a list that takes
none (WinForms), and reports an option `select` cannot find as missing.

The Electron snapshot fixtures go too; the wrapper and shared-id tests
use the AppKit sign-in form and an inline AppKit menu. Docs, the
README, package metadata, and the skill now say native apps.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* chore(desktop): match the workspace's vitest 5.1.1

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* fix(desktop): apply the review of #1953

macOS only. prepare and init refuse any other OS: on Windows and Linux
some actions need physical input and GTK's password field is not
reported as secure, so masking would fail open. A config with a desktop
target still loads elsewhere.

The app under test and agent-desktop get only HOME, USER, LOGNAME,
SHELL, PATH, TMPDIR, LANG, LC_*, and __CF_USER_TEXT_ENCODING of the
run's environment, so secrets, model keys, and CI tokens never reach
them. `open` and NSWorkspace both handed the caller's whole environment
to the app.

The app launches in the background (`open -g`, `launch` without
`--activate`) unless headed, so typing at the Mac no longer lands in it.
An app that shows no window after 2 s is brought forward once.
Every launch passes -ApplePersistenceIgnoreState YES.

An appPath app whose environment cannot be read (a system app, a build
signed with `restrict`) is now claimed as the single new copy that was
not running before the launch, instead of polling until the test timed
out and leaving the app running. A launch reported for a copy that ran
before it is refused. An open that finds no process fails within 10 s.

select no longer falls back to set-value, which could type a missing
option into an editable combo box. press sends a key only when the
control's window is the app's key window, and finds the control again by
its stable id. locate fails on a truncated read with no match instead of
proving absence. The reads a screenshot's mask takes hand out no ids.
The window's own Close, Minimize, Zoom, and Full Screen buttons are left
out of the tree. desktop.useWindow() picks the window screen reads.

Stale state directories of killed workers are swept at init. The cached
agent-desktop is reused only when a verified unpack stamped it and its
executable and helper are intact. doctor says the permission checks need
the first run. A pending launch whose lookup timed out stays pending.

Tests: the screenshot-moved test now uses the window the fake lists, and
a steady capture is checked to succeed. Each fix has a test that fails
without it (checked by mutation). The testbed checks the password
field's own box is black and the field beside it is not, and
test:desktop-testbed builds the app.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(desktop): keep launches pending after open, and fix four review gaps

A fresh-context review of f0d1bb69f found these, each now with a test:

- openBundle: a failure after `open` ran (a list-apps error, or the
  budget running out while looking) was read as "nothing was sent", so
  the copy `open` started was never quit. Any failure after `open` is
  now ACTION_MAY_HAVE_COMMITTED and leaves the launch pending.
- An app that quits right after launch now fails the open at once,
  before the engine brings its bundle forward, which would have started
  a copy the engine does not own.
- A node inside a look-alike container is no longer marked stable: its
  id is built on its parent's, which is new in every snapshot, so press
  finds it by place again.
- Only a title-bar-sized button (at most 20 points) with a window
  control's name is left out; an app's own Close button stays.
- A window chosen with desktop.useWindow() that closed is now
  INVALID_STATE, not an empty screen a negative assertion would pass on.
- The release cache is swapped by rename instead of deleted first, and
  doctor checks the stamped copy, not only that the file exists.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(desktop): send keys and close select menus in the background, and quit apps killed workers left

Found in live runs on an unlocked Mac:

- A key press never reached an app in the background: agent-desktop
  sends a key to a named window only while that window is active, and
  answered STALE_REF, which the harness retried until the step timed out.
  The key now goes to the process by its app name, once the engine saw
  its copy is the only one with that name; with another copy running,
  press fails at once with NOT_ACTIONABLE and says to use headed.
- selectOption of a missing option on a pop-up button opened its menu,
  waited out the whole action budget, and left the menu open. select now
  waits at most 3 s; a menu it left open is closed with Escape and the
  option reported missing (NOT_ACTIONABLE), the list unchanged.
- A worker that was killed left its app running. Each worker records
  the app it launched and its token; the next run's sweep quits that app
  while its environment still shows the token.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(desktop): sweep a state directory whose launch record was cut short

A worker killed while writing app.json left half a JSON record. The sweep
parsed it inside the promise's success handler, so the SyntaxError escaped
and every later init failed on the same stale directory. A record that does
not parse now names no process and the directory still goes, and the record
is written to a side file and renamed into place.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(desktop): keep the state directory when dispose could not quit the app

dispose removed the state directory even when its quit failed, and app.json
went with it, so the next run's sweep had no pid and token to quit the
leaked app by. The directory now stays while the engine still owns a process
it could not quit; the owner file names a worker that is gone by then, so the
sweep removes the directory after it quits the app.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(desktop): claim a copy with no readable token only from whole app lists

Two paths took a process as the launch's own on evidence an incomplete app
list cannot give, and the engine later kills what it owns:

- launch by bundle id accepted agent-desktop's reported pid when its
  environment could not be read and the list before the launch was partial,
  so a copy already running could not be ruled out;
- claim() took the single new unreadable copy from a list after the launch
  that left copies out, where a second new one could hide.

The first now refuses the launch and leaves the process alone. The second
reports that it cannot tell; quit keeps the launch pending as it does for a
lookup that ran out of time.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(desktop): give a background key only what is left of its budget

sendKey built its command options once, so the key sent by the app's name,
after a refused window press and a list of the apps, still carried the budget
the operation had at its start and could run past the deadline. The options
are now built for each command from the time left.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(desktop): keep a select whose menu was left open as possibly committed

After a select that may have reached the app, closing a menu the app had
open was taken as proof that the option was missing and nothing was chosen,
and reported as NOT_ACTIONABLE. An open menu shows neither: it does not say
which control opened it. The failure now stays ACTION_MAY_HAVE_COMMITTED,
naming the option the list may lack, after the menu is closed. A failure to
close the menu (Escape refused while another copy runs) no longer replaces
the select's own error.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(desktop): measure a point from the window useWindow() chose

performAt measured a point from the window the last read used. After
desktop.useWindow() picked another window, the next tapAt still brought the
old window forward and clicked there. Choosing a window, or going back to the
focused one, now drops that window, so the next point action measures from
the window in use.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(desktop): read no empty screen from an app that cannot be read

capture() skipped every window it could not snapshot and returned no screen,
so observe showed an empty window and locate found nothing. While the screen
is locked, or after the app crashed, not.toBeVisible() then passed. Opus
reproduced it against the testbed app on a locked Mac.

Now:
- listed windows none of which could be read are a retryable NODE_STALE,
  so a locator reads again until its deadline;
- a chosen window whose snapshot says it is gone is INVALID_STATE, like
  one missing from the list;
- an app whose process is gone is INVALID_STATE.

An app with no window at all still reads as an empty screen. The launch's
readiness wait treats the new errors as a window not readable yet.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(desktop): count no matches in a window read in part

locate() refused a partial read only when nothing matched, and then with a
non-retryable ENGINE_FAILURE. One visible match in a read cut short still
counted as exactly one, though a second could sit in the part left out, so
a strict action or toHaveCount(1) could pass on it. And a locator for a
control not there yet failed at once instead of polling.

A partial read is now a retryable NODE_STALE whatever it matched, so the
locator reads again until its deadline. The docs say so for a window deeper
than the 24 levels a snapshot reads.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(desktop): correct getByText on buttons, and macOS-only metadata

- The desktop page said getByText cannot find a button by its title. The
  projection gives a button its title as text too, so it can; getByRole is
  what limits a match to the button.
- The package keywords listed windows and linux, which the engine refuses.
- CONTRIBUTING said a contract change bumps both engines; there are three.
- @e2e-dev/desktop moves to its alphabetical place in the docs package.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(desktop): cover the appPath launch and a successful install

Every surface test launched by bundle id, so a change to the open -g -n
arguments, the claim of the copy carrying the token, or its quit would not
fail a test; and the binary tests covered only failed installs.

- An appPath test puts a fake open first on PATH: it records its arguments
  and starts a process with the variable --env names, which the engine
  claims, reads, and quits.
- An install test serves an archive of its own through a digests table
  installPinned now takes (the pinned one by default), installs it twice
  at once, and checks the stamp, that nothing is left beside the entry, and
  that a third install fetches nothing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(desktop): find the app's running copies the way agent-desktop launches it

runningCopies() and claim() listed every app and kept the ones whose
bundle id or name equalled the target exactly. agent-desktop reports the
localized name (Calculator is 계산기 on a Korean Mac) and matches a launch
target in any case, so a copy the launch could hand over to was left out of
the list before it. A copy whose environment cannot be read was then taken
as the launch's own and killed at the end of the test; and a copy a launch
of unknown outcome started was never found, and left running.

Both now ask list-apps --app <target>, which matches the way the launch and
quit() do, and count every process it returns.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(desktop): stop the fake apps before removing the list of them

afterEach ran cleanupFakes() first, which deletes the file fake.spawned()
reads, so the processes a test left running were never killed and stayed
for their 60 s timer. They are now killed first; no stand-in process is left
after the suite.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(desktop): report an open LaunchServices refused as starting nothing

openBundle wrapped every failure of open as ACTION_MAY_HAVE_COMMITTED, so a
bundle LaunchServices refused (a missing executable, a bad signature) left the
launch pending. The end of the attempt then looked for its process for 3 s
and failed a second time, telling the user to quit an app that never
started. The error also quoted the whole open command line.

An open that exits with an error, or cannot run, is now ENGINE_FAILURE with
open's own message, and nothing is left to look for. Only an open the budget
or a cancellation stopped stays possibly committed. The appPath tests share
a helper that puts a stand-in open on PATH.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(desktop): record the launch token before the launch, and sweep by it

app.json was written only once the launch's process id was known. A worker
killed during the launch (which can wait up to 10 s for its process), or a
launch whose process was never found, left no record, and dispose then
deleted the state directory too. The next run could not find the app.

The record now holds the launch token and is written before anything is
sent. dispose keeps it while a launch is pending, as it does for an app that
would not quit. The sweep quits every process whose environment carries the
token, found by listing the processes, so it needs no process id and
cannot reach a process that took one over.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(desktop): act on no node read outside the window useWindow() chose

useWindow() changed the window every read used but kept the nodes read
before, so an action on a ref from the old window still reached its control
after a test chose another. Such a ref is now NODE_STALE, which the harness
answers by locating the node again in the window in use.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(desktop): read no empty screen from an app whose process id was reused

With no window of the launched process listed, capture() checked only that
its process id was alive. Once the app quits, macOS can give that id to
another process, so the read showed an empty screen and a negative assertion
could pass. The app list must now show the same process instance; a list
that left copies out is a retryable NODE_STALE.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(desktop): make doctor check an appPath bundle the way init does

doctor passed any bundle with a Contents/Info.plist, while init also needs a
CFBundleIdentifier in it, equal to app.bundleId when that is set. A bundle
doctor passed could then fail init, and a plist with no bundle id failed init
with plutil's raw error. Both now use one check, which reports a missing or
mismatched bundle id as the problem.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(desktop): never move another install's verified release out of the cache

download() checked whether the cache entry was usable and, if not, moved it
aside before renaming its own unpack in. Two installs that both found no
usable entry could interleave: the second moved aside the release the first
had just put in place, and a worker starting between its two renames found
no executable. The unpack is now renamed into place first. A rename onto an
existing entry fails, and then the entry is used if it checks out, and moved
aside only if it does not.

The race has no deterministic test; a new test covers replacing a damaged
entry, the path this reorders.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(desktop): say when two desktop targets share a run, and what reads empty

- The workers line said two desktop targets can run at the same time
  while telling headed runs to set workers: 1, which is what stops them.
  It now says they do, and why headed runs set workers: 1.
- The Reading row and capture()'s comment said a window is never read as
  an empty screen, but an app with no window open still is, on purpose.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* chore(desktop-testbed): pin ai to the version every other workspace uses

The testbed pinned ai 7.0.173 while the rest of the repo pins 7.0.177, which
put a second copy of ai and its gateway and provider packages in the
lockfile.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(desktop): report an open cancelled mid-launch as CANCELLED

An abort while open ran, or while the engine looked for the process it
started, came out as ACTION_MAY_HAVE_COMMITTED, where every other path of
the engine reports CANCELLED. It is now CANCELLED; the launch stays pending
as before, so the end of the attempt still looks for the app.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(desktop): settle a pending launch by its last lookup and by its token

quit() dropped a pending launch, and dispose its record, once any lookup in
its wait had a whole answer. Two cases lost an app that was still starting:

- an early complete list with no match, followed by partial ones that could
  hide the app once it registered;
- a process carrying the launch's token that the app list did not show yet.

Now only the wait's last answer counts; a lookup that ran out of time is no
answer. Before giving up, quit looks for processes carrying the token and
quits them. The launch stays pending, and its record kept, unless the last
answer was whole and found nothing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(desktop): run launches and quits of one surface one at a time

Two overlapping app.open() calls could both quit, then both launch, writing
into the one process id and launch record the engine keeps; they even shared
the app.json.partial file. The end of the attempt then quit only one copy.
restart, closeApp, endAttempt, and dispose now wait for the launch or quit
running before them. restart's budget counts from the call, so the wait is
part of it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(desktop): measure a headed click from where the window is after focus

performAt read the window's bounds, brought the window forward, and clicked
at the bounds read before. A window that moved as it came forward (onto
another space, out from under the menu bar) took the click somewhere else,
possibly another app. The window is now read again after focus-window, and
the point is checked against and measured from its bounds then.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(desktop): repair a damaged cache entry under a lock

12c8f64d0 renamed a fresh unpack into place first, which keeps a run from
moving aside a release another run just installed only when there was no
entry. Two runs that both found an entry damaged could still interleave: the
second moved aside the copy the first had just put in place. Its commit
message and comment claimed more than that.

Replacing an entry now holds a lock directory that names its owner's process
id, and checks the entry again under it. A lock whose owner is gone, or that
got no owner and is 10 s old, is taken over; one held for 60 s fails the
install with AGENT_DESKTOP_INSTALL_FAILED.

Tests repair one damaged entry from three runs at once and take over a lock a
dead process left. The interleaving itself has no deterministic test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(desktop): refuse getByPlaceholder, which macOS gives nothing to match

agent-desktop's macOS adapter reports no placeholder for a field; AppKit
folds it into the accessible name of a field with no label. A placeholder
query therefore matched nothing, and not.toBeVisible() or toHaveCount(0) on
it passed with the field on screen. locate() now refuses any expression that
uses a placeholder query with UNSUPPORTED_CAPABILITY, naming getByRole as
the way to find the field, and the docs say so.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(desktop): let desktop({ maxDepth }) set how deep a read goes

A read cut short at the fixed 24-level depth makes every locator on that
window read again until its timeout, since a match could sit below it. An
app nested deeper could not be tested with locators at all. maxDepth, from
1 to 255 (agent-desktop's limit), defaults to 24 and raises it; the partial
read error names the option.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(desktop): accept the verified cache copy a run published mid-repair, and keep a launch record when its process lookup fails

A run that finds no cache entry publishes without the lock, so it can fill the gap a locked repair makes between its two renames; the repair then failed with ENOTEMPTY although the cache held a verified release. The repair now accepts a usable copy there.

A failed process listing was read as an empty one, so the stale-worker sweep deleted the record that names the app a killed worker left running. The listing now reports failure apart from none found: the sweep keeps the owner and launch records (removing only the cached field text beside them), and an attempt's quit keeps its pending launch and retries.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* fix(desktop): publish the agent-desktop cache without a lock, keeping a verified winner

The fast path renamed a fresh unpack into place without the lock that the
repair of a damaged entry held. A run that published while a repair had
moved the damaged entry aside made the repair's own rename fail with
ENOTEMPTY, though the cache now held a checked release (seen on the Node
22.22.3 and Node 26 CI jobs). The lock's stale-owner takeover could also
remove a lock another waiter had just taken.

Every run publishes a copy of the same checked release, so the lock is
gone: a rename that collides with an entry accepts it once it is checked,
a damaged entry is moved aside by one rename that only one run wins, and a
whole copy moved aside by a run that saw the entry damaged earlier goes
back. Two tests hold one run at each of those points while another installs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(desktop): show a whole cache entry without the right stamp is downloaded again

The stamp test's entry had no executable or macOS helper, so it was
downloaded again even with the stamp check removed. The entry now has
both, and a missing stamp and one for another release each force a
download.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(desktop): take a launched process as the engine's own only by its launch token

When no environment could be read, a launch took the one new copy of the
app as its own, and a cleanup after an uncertain launch could quit it.
Being new does not show this launch started it: a copy the developer
opened at the same moment is new too, and was killed in a scripted probe.

A copy is now claimed only when it carries the launch token. agent-desktop's
launch, which reports the process it started, still stands for an app
launched by bundleId. An appPath to a bundle whose environment macOS hides
(an Apple platform app, a build signed with the restrict flag) is refused
by init and doctor, which name the bundleId to use instead, so open never
starts a copy the engine could not find again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(desktop): keep a launch record until cleanup confirms its process is gone

A process lookup that failed read as an empty list: the sweep then
deleted a killed worker's launch record, and the next sweep left the app
running though the lookup worked again. The record held only the token,
so a process the worker had identified, whose environment cannot be read,
could not be found by a later run. A dispose whose quit failed kept the
whole state directory, with the text agent-desktop keeps of the fields.

A failed lookup is now no answer: the pending quit fails and keeps the
launch, and the sweep keeps the record. A kill waits for the process to
exit before the record goes. Once a process is shown to be the launch's,
the record also names its id and start time, so a later run quits it
without reading its environment, and never a later process given that id.
Dispose and the sweep keep only the owner file and the record, and remove
the field text.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(desktop): never bring a background app forward to ask it for a window

An app launched in the background that showed no window for 2 s was
brought forward with a second open by path, bundle id, or name. That open
did not name the process the engine launched, so LaunchServices could
bring another copy forward, and it took the keyboard the guide promised
to leave alone.

The background launch now stays in the background. An app that opens no
window there fails the open with a message that it needs
desktop({ headed: true }), whose launch brings the app it starts forward.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(desktop): send a key only to the launched process's window, never by app name

agent-desktop refuses a key for a named window that is not active, which
an app in the background never is. The engine then sent the key by the
app's name once a list of the apps showed its copy alone, even a list
agent-desktop marked incomplete. The name does not pin the process, so a
copy that started between the list and the key could get it.

The fallback is gone: a refused key fails as NOT_ACTIONABLE, and nothing
is sent. Since a key then reaches an app in the background only by chance,
press joins the actions only headed offers, so a locator press without
headed fails as unsupported before it moves focus.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(desktop): never retry a press whose focus already moved

A locator press focuses the control, reads its window to check the
control kept focus, then sends the key. A stale read after the focus was
retryable, so the locator ran the whole press again and focused the
control a second time, which an app's focus handler sees. A retryable
failure after the focus is now ACTION_MAY_HAVE_COMMITTED; one before it
stays retryable.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(desktop): show a select that may have reached the app stays possibly committed

Removing the delivery check from the select's ELEMENT_NOT_FOUND mapping
left every select test green. A select whose failure agent-desktop marks
delivered_unverified must stay ACTION_MAY_HAVE_COMMITTED, not "no option".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-21 14:34:19 +00:00
Natasha TranandOskar Kwaśniewski abb40648ed feat(mobile): add device.viewport() (#2033)
* feat(mobile): add device.viewport()

* fix(mobile): preserve device refs after viewport reads

* fix(mobile): clarify viewport rotation fallbacks

* fix(mobile): simplify viewport reads and export ViewportSize

---------

Co-authored-by: Oskar Kwaśniewski <oskar@okwasniewski.com>
2026-11-21 13:58:50 +00:00
rentezzy ed48b0b455 feat(screen): accept percent points in tapAt, doubleTapAt, and swipe (#1963)
* feat(screen): accept percent points in tapAt, doubleTapAt, and swipe

A coordinate may be a percentage of the viewport ('0%' to '100%') as well as
CSS pixels, as in Maestro's `point: "50%,50%"`. Numbers stay pixels, and
`Point` stays pixels for `tap({ position })`, the agent, and the engine
contract.

A percent point costs one screen read inside the action's timeout, then
resolves to whole pixels, '100%' landing on the last one. Pixel points skip
the read. The step label keeps the point as written.

* test(screen): cover percent points in tapAt, doubleTapAt, and swipe

* docs(screen): document percent points for tapAt, doubleTapAt, and swipe

* chore: add a changeset for percent points

* docs(screen): say percent points round to the nearest pixel

* refactor(screen): resolve percent points through one helper

* test(screen): check a pixel swipe skips the screen read

* docs(screen): say a percent point reads the screen once
2026-11-21 14:39:52 +01:00
5b61f221aa fix(core): stop launch() hosts from driving one mobile device (#2008)
* fix(core): give each launch() host a target name of its own

Every launch() host named its target after its platform and started as
worker slot 0, so every mobile host drove the agent-device session
e2e-ios-0: the session of another host, or of a runner worker on an
unnamed ios target. The mobile pool keeps a device its own session name
holds, so the host took that device and both tapped one screen.

A host's target is now <platform>-launch-<pid>-<n>. An agent's replay
cache keys to the platform, so recordings still replay on any host.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(core): draw a random token per launch() host, renamed after resolution

The pid and a module counter repeated across worker threads and across
two copies of the package in one process, so two hosts could still share
a device session. A random token per host does not. The platform part is
cut to 40 characters, so a long engine name still fits in the artifact
folder name, and the target is renamed only after its config resolved,
so config errors name the target as the program wrote it. The replay
cache keys on the resolved name.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(core): leave a long platform out of a launch() host's name, and forget finished bindings

A platform cut to 40 characters could keep part of a secret the
redactor only knows whole, so a longer platform is left out of the name.
The token is 96 random bits. The mobile engine's finish now forgets the
target's bindings, so one engine reused by host after host no longer
keeps every finished host's device and daemon address.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(core): say how several launch() hosts share a machine's devices

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(core): say when appPath hosts can share a device, and test the 40-character name boundary

A host holds its device once its app opens, so two appPath-only hosts
launched back to back can bind one device. The recovery command now
takes the session name from the error, quoted, and names daemon stop.
The changeset said "the token alone" where the name is host-<token>.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(core): name a launch() host's artifact folders after its token, not its platform

An engine's platform can be a secret's value, and the folder name
core/<target>-<attempt> put it on disk under artifactsDir, where
redaction never reaches. The folders are now core/host-<token>-<attempt>.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(core): install an appPath build before opening it, and keep recovery to what the error names

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(core): pass launch() hosts to engines as hostId, keep target name

The random token no longer renames the host's target. Core hands it to
prepare, init and finish as hostId, and the mobile pool adds it to the
session name: e2e-<target>-<token>-<slot>. Fixtures, notices, errors,
provider tags and the replay cache key see the target name the program
wrote, so the cacheTarget indirection and the 40-character rule go.

* test(mobile): check a launch() host's init resumes the session its prepare warmed

The pool test now also has the host keep a device its own session holds
and runs init with the host's id: a surface whose init named the session
without hostId passed the old test and fails this one. Also moves the
SessionNamer doc comment back above its type.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(mobile): hand a launch() host's device bindings over a variable of its own

With the target keeping its name, a host opened inside an `e2e run` test
inherits the run's environment, where the run already left its bindings
for a target of the same name. A host's worker without the bindings in
memory (an engine served in another process) read the run's and drove
the device the test holds. The variable's digest now includes hostId.

Also names the host form of the session in `DeviceSession.session` and
the reference's `session` default, and mirrors the session option's
comment into the ja and zh references.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(mobile): fail a launch() host's close when its session stays open

A host's session name carries a random hostId, so no later host or run
takes its device back by name. finish now reports a session that will
not close, with its name, and counts one that is already gone as closed.
A run's worker and a session option keep their names and stay quiet.

Also from review: raise the @e2e-dev/core peer floor to 0.24.1, the
first release that sends hostId; bump e2e and web in the changeset and
say what a crashed host leaves behind; use pnpm dlx for agent-device in
the docs; drop a test that passed on main.

---------

Co-authored-by: michal-e2e <357875193+michal-e2e@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Oskar Kwaśniewski <oskar@okwasniewski.com>
2026-11-21 13:33:12 +00:00
Satyajit Sahoo 23aa697e26 feat(cli): show --step keys under the summary and view the screen and app log (#2015)
* feat(cli): show step keys under the summary and view the screen and app log

The --step and screen.pause() prompt now lists its keys below the live
summary, as --watch does, and the held step row uses a pause glyph.

Press s for the screen's accessibility tree or l for the attempt's app
log. Both open in a scrolling viewer on the alternate screen. The app log
is read from the worker during the hold through a new logs-request and
logs-reply message pair.

The mode without a terminal keeps its text.

* feat(cli): stop the terminal's Duration row while a step is held

The list reporter's Duration row counted wall time, so it kept ticking at
a --step prompt or a screen.pause(). The scheduler now reports when the
first hold opens and the last closes through a run-hold event, and the
row excludes that time, live and in the final summary.

report.json and the reports built from it keep wall time.

* feat(report): add run.durationMs without held time

The report only had the run's start and finish timestamps, so a run that
waited at a --step prompt or a screen.pause() looked as long as the wait.

run.durationMs is the run's duration with held time left out. The
terminal's final Duration row prints the same value. Markdown, the GitHub
comment, the HTML report, and telemetry read it in place of the timestamp
difference. startedAt and finishedAt still record wall time.

A simplification review also removed a few flags and options from the
step viewer and the hold path, and the docs keep each new fact in one
place.

* docs: trim the step viewer changeset and docs

The changeset is down to two sentences. The docs no longer say where the
keys are, since the terminal sample shows it, and the json reporter
section matches main again.

* fix(cli): print the viewer's text before the prompt is repainted

On p, the viewer closed first, so the live window was repainted and then
erased again for the printed text. That left a blank frame. The text is
now queued before the viewer closes, and the window is repainted once.

* fix(cli): number the step prompt by its trace position

The app log view numbered its lines by the recorder's step index, which
counts nested steps, while the prompt counted held steps only. A line
logged inside a nested step could show a different number than the
prompt.

The worker now sends each held step's position in the trace, so the
prompt, the app log view, and the trace page use one number.

The viewer's try blocks are removed too. Nothing received their rethrows,
and the guarded calls don't throw.

* fix(cli): count held time only while every running worker is held

With several workers, a screen.pause() in one test took the time other
tests spent running out of the run's duration. A run could read shorter
than one of its tests. An interval now counts as held only while every
running worker is held. The time limit keeps its rule.

Also from review:
- A routing state an older installed copy made is upgraded in place,
  so claimProcessOutput() no longer throws.
- The viewer holds writes only for terminal streams.
- Ctrl-Z closes the viewer and suspends.
- A failed screen read can be retried during the same hold.
- p is ignored while a read is pending.
- The app log view shows plain text, not Markdown code spans.
- The viewer's help row starts with q, and its headings use the
  redacted title and target.

* test(cli): cover an older dispatcher installed on the streams

The upgrade test seeded a state with no dispatchers. This one installs
an older copy's dispatchers first, then checks that writes are held
while the viewer is open.

The comment on that branch loses its FIXME prefix. It describes what the
code does.
2026-11-21 14:00:20 +01:00
Łukasz LechandClaude Opus 5.5 1d28ac0fea feat(mobile): add device.isKeyboardShown() (#1893)
* feat(mobile): add device.isKeyboardShown()

Returns whether the soft keyboard is on screen from one capture: the
keyboard agent-device reports (a measured or derived frame, or a
visibility probe), else visible keyboard elements in the tree, the same
band observations hide covered nodes under. A capture that cannot answer
(no keyboard in a sparse or cut tree, or keyboard elements agent-device
rejected with no probe) fails with ENGINE_FAILURE instead of guessing.

Refs #1855

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(i18n): mirror the isKeyboardShown heading and rows in ja and zh

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(mobile): note that isKeyboardShown reads the screen as it is now

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(mobile): answer isKeyboardShown only from what the capture can prove

- An unmeasurable keyboard reading answers false only for the reasons that
  say agent-device read the whole tree; a capture that could not see the
  keyboard's window fails with UNSUPPORTED_CAPABILITY.
- A rejected docked frame with no probe reads as shown: agent-device
  rejects one only for a tree that lists keyboard elements, as a keyboard
  still sliding in does on an iPhone.
- The incomplete-tree failure no longer claims a retryable flag that
  EngineError drops for ENGINE_FAILURE.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(mobile): read the keyboard by one rule and answer false only on proof

Review follow-up for isKeyboardShown:

- keyboardShownIn moves to keyboard.ts and is the one rule:
  dismissKeyboard's check is keyboardShownIn === true.
- Only agent-device's `absent` (or a probe that says hidden) answers
  false. keyboard-not-observed and keyboard-visibility-probe-unsupported
  now fail with UNSUPPORTED_CAPABILITY.
- Tree answers count keyboard elements drawn on screen only, so a
  keyboard left in the tree off screen does not read as shown.
- A tree-rejected reading with no on-screen key is the keyboard moving:
  both isKeyboardShown and dismissKeyboard capture again, briefly, before
  they answer.
- The benchmark test taps the field first, since a fill on Android opens
  no input method, and polls both reads.
- The ja and zh Throws tables are translated, and the Appium guide notes
  that the call reads the screen as it is now.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(mobile): say a keyboard reading can take several captures

isKeyboardShown's JSDoc and the reference said one capture, but a
keyboard still moving is captured again. keyboardShownIn's comment now
names what it decides instead of every keyboard check (observations
report measured state only), and the unsupported message says the tree
shows no keyboard drawn on screen.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(mobile-benchmark): read the keyboard on iOS only

The Android emulator in CI draws no soft keyboard, even for a tapped
field, so isKeyboardShown polled false until it timed out. The keyboard
reads move to their own iOS-only test, and the onboarding test is back
to what main has.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(mobile): start the keyboard drag above every element that says the keyboard is up

The drag's top edge came from keys alone, while isKeyboardShown also counts
the suggestion strip, so a capture that lists only the strip (a hardware
keyboard) dragged from under it. The reference example now polls the read
after dismissKeyboard, and the changeset notes the transition wait follows
the same rule.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(mobile): say the strip drag starts above the strip

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(mobile): pin the drag above the strip when keys sit under it

The strip-only test could not tell the highest keyboard element from any
of them. It now also lists the strip above the keys, with the strip holding
only text so nothing but the keyboard's top keeps the drag off it. The
drawn-keyboard predicate is one helper that keyboardBand and
keyboardElementsTop both read.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(mobile): refuse a keyboard drag when nothing bounds the keyboard

A visibility probe that says the keyboard is up, with no frame and no
keyboard element on screen, left keyboardTop at the bottom of the screen, so
the drag could start on the keyboard. dismissKeyboard now fails with
UNSUPPORTED_CAPABILITY there instead of guessing. The strip test's
assertion names the failing y.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(mobile): read the soft keyboard from keys, and keep each look in the budget

The suggestion strip alone, as XCTest lists it with a hardware keyboard, no
longer reads as the soft keyboard shown; it still bounds where the
dismissing drag may start, now also above a measured or derived frame. The
settle loop reserves the last capture's time along with its backoff, so a
slow capture after the drag no longer times the dismissal out. A capture
with no viewport and no window places no keyboard on screen, since the
tree's own extent holds the keyboard it is asked about.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(mobile): say expect.poll reads isKeyboardShown again after any error

A lasting UNSUPPORTED_CAPABILITY makes the poll wait out its timeout; the
docs implied only ENGINE_FAILURE was read again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-21 13:58:41 +01:00
Samanyu DasandClaude Opus 5.5 7a8e546e26 fix(mobile): fail toHaveScreenshot with POLICY_DENIED when a secure field can't be masked (#2014)
* test(mobile): expect a screenshot mobile withheld to be refused for masking

On mobile, a screenshot the engine cannot mask is withheld with no sign it
was ever taken, so toHaveScreenshot waits out its timeout and fails with
ASSERTION_FAILED, and the agent is told the engine captures no pixels. These
tests expect the engine to report the regions it masked (none) beside the
missing pixels, toHaveScreenshot to fail at once with POLICY_DENIED, and the
agent to read MASKING_UNPROVEN. A screenshot that was never taken, or one
withheld with every secure field masked, still keeps the matcher waiting.

Refs #1996

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(mobile): fail toHaveScreenshot with POLICY_DENIED when a secure field can't be masked

When a secure node on the mobile screen has no bounds, the engine cannot
mask it, so it drops the screenshot and reports nothing else. The session
could not tell that from a screenshot that failed, and the matcher checked
for missing pixels before it checked masking, so it polled until its timeout
and failed with ASSERTION_FAILED and "the engine returned no screenshot". On
the web the same case fails at once with POLICY_DENIED, which is what the
docs and AGENTS.md promise. The agent had the same gap: it was told the
engine captures no pixels (UNSUPPORTED_CAPABILITY) instead of
MASKING_UNPROVEN.

The engine contract now lets an engine withhold a screenshot it could not
mask by reporting maskedRegionCount without pixels. The mobile engine does
so for a secure node without bounds. An image it cannot read or decode is
still no screenshot, so the matcher takes another. When the count falls
short of the secure nodes, the session marks the observation
pixelsWithheld: 'MASKING_UNPROVEN', the token the agent already uses for
unproven pixels. toHaveScreenshot fails with POLICY_DENIED on it, and the
agent passes it on. A screenshot that was never taken (the web engine
mid-navigation, a failed device capture) reports no count, so the matcher
keeps waiting for it as before.

app.screenshot() on the same screen failed with ENGINE_FAILURE (exit 3)
for the same cause. It now fails with POLICY_DENIED too, as AGENTS.md
asks for withheld pixels. An image that cannot be decoded keeps
ENGINE_FAILURE, since there is no screenshot to withhold. Failure evidence
and agent evidence screenshots catch every error, so neither changes.

Fixes #1996

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor(core): decide withheld pixels once, in the session

The session now keeps pixels on an observation only when the masked regions
cover every secure node, and marks any screenshot the engine took without
that proof, sent or withheld, as `pixelsWithheld: 'MASKING_UNPROVEN'`.
`toHaveScreenshot` and the agent's `clearPixels` read that one field and no
longer check the masking themselves.

The mobile engine reports a secure node without bounds as unbounded before
it reads the image, since no image makes that screen maskable, and drops
`decodesAsPng`. `redactSecure` returns a tagged result, and the
`ENGINE_FAILURE` message for an image it cannot read says so.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(core): keep fallback pixels only when their masking is proven

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(core): isolate the unproven fallback pixels case

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(mobile): withhold pixels on an unbounded screen without a screenshot

A secure field without bounds makes the screen unmaskable from the tree
alone, so the mobile engine now checks the bounds before it takes a
device screenshot. `observe` reports no regions masked and
`app.screenshot()` fails with `POLICY_DENIED`, even when the device
screenshot would have failed, and neither pays a device round trip.

Lists `POLICY_DENIED` under `app.screenshot()` in the app reference, and
says `maskedRegionCount` is reported when the harness asks for pixels.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(core): name pixelFallback as a pixels request in the count contract

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(core): mark pixels withheld only for a pixels request

The session now marks an observation `MASKING_UNPROVEN` only when the
caller asked for pixels, so the engine contract needs no rule about when
an engine reports `maskedRegionCount`. The masking check on fallback
pixels runs after the bare-root check, so a malformed unavailable tree
keeps `INVALID_STATE`. Tests pin the masked count the agent passes on.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(core): name pixels: true in the withheld count contract

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(mobile): pin the error category of a withheld app.screenshot()

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(core): leave fallback pixels to the engine's own masking proof

An unavailable tree is a bare root, so the session has no secure node to
count, and checking the masked-region count against zero proves nothing.
The engine that sends fallback pixels proves their masking itself.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-21 13:53:30 +01:00
Samanyu DasandClaude Opus 5.5 34d5d744e7 docs(security): document pages that hide a password field from e2e (#2034)
* docs(security): document pages that hide a password field from e2e

On the web engine, e2e decides in the page's main world whether a field is
secure. A page script that replaces `Element.prototype.matches` makes a
password field count as a plain field. The observed screen then shows its
value, `inputValue` and `getAttribute` return it, and a failed
`toHaveValue` or `toHaveAttribute` prints it. Configured secrets are still
redacted, and screenshots still mask the field outside a closed shadow
root.

The places that say password fields are masked or never reach the model
link the limit: the security reference's model-input table, the
agent-steps and decision-models guides, and the e2e skill's agent
reference.

Refs #2010

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(security): limit the attribute reads to a password in the value attribute

`getAttribute` and `toHaveAttribute` read the element's HTML attributes,
not its live value, so they expose a hidden password field's value only
when the `value` attribute holds it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs: name secure fields where a read is denied, and what a hidden field leaks

- The MCP `POLICY_DENIED` row and the e2e skill's writing-tests reference
  say a read of a secure field is denied, as `screen.mdx` does, since a
  password field a page hides from e2e is not one.
- decision-models says the text model sees a hidden field's value and
  `type` can fill it, and that only `typeSecret` fills a configured secret.
- The security reference names the limit in its model-input table link,
  and says a closed shadow root loses its screenshot mask only when the
  page also replaces `querySelectorAll`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-21 13:49:07 +01:00
Daniel PetroandOskar Kwaśniewski e88153b2f3 feat(locator): add blur for web elements (#1919)
* feat(locator): add blur for web elements

* fix(locator): declare blur against the required core version

* fix(locator): cover MathML blur and engine capabilities

* docs(locator): list blur in action references

* fix(web): require core and e2e 0.24.1 for locator actions

---------

Co-authored-by: Oskar Kwaśniewski <oskar@okwasniewski.com>
2026-11-20 15:58:39 +00:00
Samanyu DasandClaude Opus 5.5 198d25f038 feat(test): add test.fixme (#1887)
* feat(test): add test.fixme

test.fixme takes the four call shapes test.skip takes. test.fixme(title, fn)
registers a skipped test whose body never runs. test.fixme(), test.fixme(reason),
and test.fixme(condition, reason) skip the running test from a body or hook.
Both report the test skipped with cause explicit and the reason "fixme", or
"fixme: <reason>" when a reason that is not blank is given.

The runtime form shares skipRunningTest with test.skip, which now takes the
API name so every error names test.fixme(). The registration form rejects a
serial group with an API-named message, as test.skipIf and test.runIf do.

Refs #1855

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(test): say test.fixme's reason and setup error need a non-blank reason and a true condition

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(test): route test.skip(title, fn) through registrar and mirror test.fixme docs in zh

test.skip(title, fn) in a serial group now names test.skip(), as test.fixme,
test.skipIf, and test.runIf do. docs/zh, which landed on main after this
branch was cut, gets the test.fixme section, the serial-group rows, the Types
block, and the Playwright guide rows.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* chore(changeset): keep the test.fixme changeset to one sentence

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(report): report test.fixme with its own skip cause, fixme

A test.fixme test, registered or skipped from its body, now reports
skip.cause fixme in report-v1, beside explicit. The reason text is
unchanged. Every consumer that counted an explicit skip as one the test
gave itself counts fixme too: serial member accounting, Skipped After
Failure, the GitHub merge summary, and NO_TESTS, which says marked fixme.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(test): document the fixme skip cause, and pin the GitHub merge count

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(report): keep fixme out of --last-failed, and cover the fixme cause's other paths

A SelfSkipCause type names the causes a test gives itself, and
RUNTIME_SKIP_APIS carries each runtime skip API's cause, so a new one
cannot report half-wired. Tests now pin --last-failed leaving fixme out,
the fixme cause beside a failure that outranks it, and the explicit
fallback for a RuntimeSkip from an older copy. The docs align the
test.skip setup row and say the Playwright guide's { skip: 'fixme' }
workaround reports cause explicit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* chore(changeset): tell users to update @e2e-dev/github with e2e for the fixme cause

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(types): give test.fixme(title, fn) worker fixtures, and name it in the executed docs

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ci: rerun checks after a flaky app-command-exit test on Node 24.8.0

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(migrate): name the serial-group exception for skip and fixme registrations

Playwright accepts test.fixme(title, fn) and test.skip(title, fn) inside a
serial describe, and e2e rejects them, so the "the same" cell names it.
Type a caught runtime skip's cause as unknown, so every reader goes through
runtimeSkipCause, and keep RuntimeSkip module-private. State the github
changeset as the symptom, and name test.fixme in the skip JSDoc.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs: shorten the migrate cell, and name test.fixme in failureBeforeSkip

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs: list test.fixme(title, details, body) as missing, and fix the github changeset

The PR comment never shows the executed count, so the changeset names the
merged report instead. The Playwright guide lists the details form under
the APIs e2e does not have yet.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(migrate): list Playwright's skip and fixme callback forms as missing

Playwright's test.skip(callback, description) and
test.fixme(callback, description) fail collection in e2e with
COLLECTION_ERROR. The guide now lists them, with the conditional form
inside the body or platforms on the test as the workaround, in en, ja,
and zh. The ja test reference says 直列実行 for serial groups, like the
rest of that page.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(schema): put the fixme skip cause in both report-1 fixtures

The valid fixture's runtime skip, on the result and its attempt, now has
cause fixme. The invalid fixture's skip that names a filter now has cause
fixme, and a test checks that the schema rejects exactly that pair: the
rule that only a filtered skip names a filter fails at that cause, and
deleting the filter clears it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(test): skip as fixme after a timeout test.setTimeout raised

A test that raises its timeout from 300 ms to 10 s, waits 900 ms, and then
calls test.fixme() is reported skipped with cause fixme, on the result and
its attempt.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-20 15:30:09 +00:00
Satyajit Sahoo 25eb2f8c5a feat(mcp): attach to a test held at screen.pause() or --step (#1972)
* feat(mcp): attach to a test held at screen.pause() or --step

A test held at `screen.pause()` or at an `e2e run --step` hold now prints
an ID, such as `Paused (p1) in pays with a card [web]:`. You can pass it to
`open_session {attach: "p1"}` in `e2e mcp` to look at the held test's app
and act on it.

The tools run in the worker, on the engine session the held attempt already
has. Requests go from the MCP server to the runner over the directory's
step socket, and the runner relays them to the worker. Replies are redacted
with the attempt's secret ledger before they're sent.

- `close_session` detaches and leaves the test held.
- When the test resumes, the session ends right away and a call still in
  progress is cancelled.
- Attached calls aren't recorded as steps in the report or the trace.
- `run_code`, the recording tools, and `type_secret` aren't available in an
  attached session.
- `e2e run --step` in a terminal now listens on the step socket like a pause
  does, so you can attach at its holds.

With several workers paused, each prompt is now drawn from the hold it
names. Before, it could show one test's title next to another test's screen.

* fix(mcp): print output a tool's timer writes after its attached call ends

Output is muted only while the attached call is running. The call scope
lives in its own module, so process-output.ts doesn't load the attach tools.

* feat(mcp): record in an attached session when the run doesn't record

start_recording and stop_recording join the attached catalog, which now
lists exactly the normal session's tools. If the run already records the
test, start_recording fails with the run's video directory. A recording
still running when the session ends is saved.

The docs no longer describe what a pause ID looks like. The examples and
sample output show one.

* fix(mcp): don't hold a resumed test while an attached recording saves

On a resume or an interrupt the worker sends the stop for the agent's
recording and lets the test continue. The file finishes in the background
and its path goes to the log. Teardown waits for a save still in progress
for at most cleanupTimeout.

A project tool's attachScreenshot works in an attached session and stays
out of the test's report. In the live list a long title ends in an
ellipsis, so the pause ID stays visible.
2026-11-20 15:06:42 +00:00
VarshithandOskar Kwaśniewski a06b931b8d fix(web): launch Chromium with its sandbox on where it can start (#1621)
* fix(web): launch Chromium with its sandbox on by default

The web engine launched Chromium without chromiumSandbox, so Playwright added --no-sandbox to every browser process. The agent can follow the app to another origin, so a renderer exploit on a page it reaches ran unsandboxed as the user running e2e run, with the checkout, .env, and model keys in reach.

Chromium now launches with chromiumSandbox: true unless the config opts out with web({ chromiumSandbox: false }) or the run environment sets E2E_CHROMIUM_SANDBOX=0, for hosts where the sandbox cannot start (root, containers, Ubuntu 23.10+ with the AppArmor user-namespace restriction). Firefox and WebKit never receive the option. The flag resolves from the option, then the run's environment, then on; the process environment never decides a launch, per the EngineInitInfo.env contract. CI opts out explicitly because the ubuntu-2404 runners restrict unprivileged user namespaces.

Docs updated next to the no-origin-allowlist note (docs/reference/web.mdx, docs/security.mdx, SECURITY.md, skills/e2e/references/setup.md) with a .changeset entry. The smol VM's --no-sandbox is unchanged: the VM is the boundary there.

Verified: pnpm check green; web unit 354/354; web integration 379 passed with 3 known load flakes that pass in isolation; testbed CLI 6/6 fail with "Chromium sandboxing failed!" by default and 6/6 pass with the opt-out on this host.

Fixes #1613

Signed-off-by: Varshith Puli <pulivarshit@gmail.com>

* docs(web): qualify chromiumSandbox coverage as local-launch-only

Cubic review on the sandbox-default PR found three doc overstatements: SECURITY.md presented the sandbox as universal while connect() and browser providers bypass launch, the skill reference omitted that E2E_CHROMIUM_SANDBOX=0 only applies when the option is omitted, and the reference page implied any container cannot run the sandbox. All three are corrected in the SECURITY note, the reference page, the option JSDoc, and both skill copies.

Signed-off-by: Varshith Puli <pulivarshit@gmail.com>

* fix(web): friendly Chromium sandbox failure and unified sandbox resolution

A sandboxed launch on a host whose sandbox cannot start failed with Playwright's raw error and its full call log, which only hints at the option. acquireBrowser now catches that one failure on a local launch and throws an ENGINE_FAILURE naming both opt-outs, E2E_CHROMIUM_SANDBOX=0 and web({ chromiumSandbox: false }), with no call log and no cause chain to leak it back.

The sandbox flag resolves in one place now: resolveChromiumSandbox(configured, env) in browser-connection.ts replaces the split readChromiumSandboxFlag plus surface resolver, and the surface keeps a single field that init resolves. BrowserConnection.acquire takes an options object, so a relaunch retries with the same options. The option JSDoc shrinks to two lines pointing at the security docs.

Tests follow the restructure: the cdp-attach edits are gone (the flag never applied to attach), screenshot-caret and stalled-frame run sandboxed by default, resolved from the ambient environment so the CI opt-out still reaches a direct launch, and the workflow comments shrink to one line. New tests pin the trim and lowercase handling, the opt-out surviving a failed relaunch, and the friendly error text.

Verified: web unit 358/358 with the opt-out; a real launch on this host fails with the friendly error by default and passes with E2E_CHROMIUM_SANDBOX=0.

Signed-off-by: Varshith Puli <pulivarshit@gmail.com>

* docs(web): minor bump for the sandbox default, CI opt-out, one home for the why

The sandbox default breaks Docker and Linux CI hosts, and the changeset said nothing about it: it is now a minor bump, opened as a maintainer-called release, telling users to set E2E_CHROMIUM_SANDBOX=0.

The CI docs name the opt-out where CI users find it: the GitHub Actions web workflow and the visual-testing screenshots workflow both set E2E_CHROMIUM_SANDBOX=0 on ubuntu-latest, matching this repo's own CI, with a note pointing at the security docs. The other CI pages run mobile targets only, so there is nothing to add there.

The why now lives in docs/security.mdx alone: the reference page, SECURITY.md, the option JSDoc, and the skill reference link to it instead of repeating it.

Verified: changeset-bumps, the docs check chain (anchors, MDX, examples, package managers, logos), and the skill tests all pass.

Signed-off-by: Varshith Puli <pulivarshit@gmail.com>

* docs(web): qualify the Chromium sandbox opt-out as ubuntu-latest only

The visual-testing guide said Linux runners generally block the user
namespaces Chromium needs. The limitation is specific to the
ubuntu-latest runners used here, so name them and tell readers to leave
the sandbox on where it starts.

* fix(web): address review follow-ups on the Chromium sandbox default

Move the sandbox launch error out of the surface and into
BrowserConnection.launchLocal, where the browser name and the resolved
sandbox flag are already known, and drop the untested verb guard. The
error now names the environment opt-out only when the option did not set
the value, since an explicit option wins over the variable. The surface
keeps the option as a readonly field and resolves it into a private
field defaulting to true, the same pattern as headed, so startAttempt
without init no longer needs the fallback. The skill GitHub Actions
recipe runs ubuntu-latest with E2E_CHROMIUM_SANDBOX 0, the web reference
points at the new ENGINE_FAILURE instead of Playwright error text, and
the resolve helper loses its dead branches.

Web unit suite: 362/363 pass; the one failure is artifact-path, which
fails identically on the base head because a real Chromium cannot start
sandboxed in this container.

* fix(web): address re-review follow-ups on the Chromium sandbox default

Resolve the sandbox into one { enabled, fromOption } object and pass it through acquire into launchLocal, which now takes the options object instead of five positional args, so the launch and its error cannot disagree about which opt-out set the value. Add the waiter-joins-a-pending-relaunch case, where the re-acquire runs once and a flip cannot cancel out, and assert retryable is false on the friendly error. Document E2E_CHROMIUM_SANDBOX in the environment reference, point the dead [Security](/security) links at https://e2e.com/docs/security, and fix the changeset wording, since the error no longer always names both opt-outs. The acquire JSDoc now states the actual guarantee: a launch takes the value its own call passes.

Web unit suite: 364/364 pass with E2E_CHROMIUM_SANDBOX=0 (artifact-path fails without it, identically on the base head, because a real Chromium cannot start sandboxed in this container). Web typecheck clean, oxlint clean. The new waiter test fails against a flip mutant injected on the re-acquire and passes without it.

Signed-off-by: Varshith Puli <pulivarshit@gmail.com>

* docs(web): mark E2E_CHROMIUM_SANDBOX as a value in the env reference

The entry was marked flag, but on this page 0 and false mean unset for flags, while 0 opts out of sandboxing here. Mark it value so readers do not apply the wrong flag semantics.

Signed-off-by: Varshith Puli <pulivarshit@gmail.com>

* docs(ci): restore the timings-history section lost in the rebase

The rebase conflict resolution in docs/ci/github-actions.mdx kept only the tail of the new Cache the timings history section, dropping its heading, intro, and example, which broke the docs anchor check (docs/ci.mdx links to #cache-the-timings-history). Restored in full; the sandbox opt-out paragraph follows it unchanged.

Verified: the docs check chain passes locally (anchors, examples, package managers, logos, MDX).

Signed-off-by: Varshith Puli <pulivarshit@gmail.com>

* docs(security): soften sandbox containment wording

The security page stated a renderer exploit stays contained by the
sandbox, which reads as a guarantee. Say the sandbox helps contain it
and note it does not prevent sandbox escapes or Chromium
vulnerabilities, matching the review feedback on the Chromium sandbox
default.

Signed-off-by: Varshith Puli <pulivarshit@gmail.com>

* docs(security): wrap the sandbox containment sentence at 80 columns

Signed-off-by: Varshith Puli <pulivarshit@gmail.com>

* fix(web): import EngineError from @e2e-dev/core/engine

browser-connection imported EngineError from e2e/engine, so loading the
web engine without the e2e package failed with ERR_MODULE_NOT_FOUND.
That broke the core-only install smoke check, which loads each engine
main entry without e2e installed. Import from @e2e-dev/core/engine
instead, like every other file in the web engine.

Signed-off-by: Varshith Puli <pulivarshit@gmail.com>

* docs(security): re-wrap sandbox paragraph and soften changeset wording

Re-wrap the sandbox containment paragraph in docs/security.mdx at 80
columns so vulnerabilities; no longer sits on a line of its own, and
soften the changeset to match: the sandbox helps contain a renderer
exploit but does not prevent sandbox escapes. The changeset keeps only
the user-facing part: what changed and the two opt-outs.

Signed-off-by: Varshith Puli <pulivarshit@gmail.com>

* fix(web): default the Chromium sandbox by platform, drop the env opt-out

On for macOS and Windows, opt-in on Linux where root, most containers
and Ubuntu 23.10+ cannot start it. The option alone decides, so CI,
the test env forwarding and the docs recipes need no opt-out.

* test(web): drop the relaunch cases the waiter test already covers

---------

Signed-off-by: Varshith Puli <pulivarshit@gmail.com>
Co-authored-by: Oskar Kwaśniewski <oskar@okwasniewski.com>
2026-11-20 11:25:39 +00:00
Karol Kulesza b180433047 feat(mobile): device.uninstallApp(id) and device.isAppInstalled(id) (#1906) 2026-11-20 13:03:34 +02:00
Parbez a5d178897a feat(test): change the running test's timeout with test.setTimeout (#1921) 2026-11-20 06:03:25 +00:00
Oskar Kwaśniewski 8eb3cc27ea fix(web,mobile): name engine handle types for inferred configs (#1976) 2026-11-19 18:09:30 +02:00
Oskar Kwaśniewski 1676981ab7 chore(mobile): bump agent-device to 0.21.40 (#1975) 2026-11-19 16:06:49 +00:00
Krzysztof Moch eddf577526 feat(reporters): build the built-in reporters on the public Reporter API (#1936)
* feat(reporters): build the built-in reporters on the public Reporter API

The built-in list, dot, json, junit, and markdown reporters now use only
the public Reporter contract: factories on e2e/reporters, printsTo for the
terminal stream a reporter prints to (one reporter per stream), live
onOutput, onStepPrompt, and onSummary hooks for any number of reporters,
and reportFiles for files the runner writes atomically. The list and dot
ids beside json print to stderr, so list,json now works as dot,json does.
@e2e-dev/html-reporter writes through reportFiles.

* docs(reporters): bring the Japanese and Chinese reporter pages up to date

Translates the reporter reference, the config reporters table, the
--reporter row, and the debugging output paragraph. In English, the name
section now says a reporter may share a built-in's name, and two lines
read less ambiguously.
2026-11-19 15:58:33 +00:00
Oskar Kwaśniewski 0e9d58ec98 feat(mcp)!: remove the run_tests tool (#1967) 2026-11-19 13:31:59 +02:00
Satyajit Sahoo 34322c5416 feat(cli): pause at screen.pause() under --watch (#1905)
Under `e2e run --watch`, a test is now paused at each `screen.pause()` on
every run, as in `e2e run`.

While a pause is shown, the keys are read by the pause prompt and not by
watch mode:

- Press Enter to resume the test.
- Press `q` to cancel that run and stay in watch mode.

`e2e run --step next` and `stop` can be used from another shell, as for a
run without `--watch`.
2026-11-19 11:15:49 +00:00
Satyajit Sahoo 4893d1dd75 feat: add screen.pause() (#1904)
With `await screen.pause()`, a test is paused at that line, with the app
open, until someone resumes it.

- In a terminal, press Enter to resume the test.
- From another shell in the same directory, run `e2e run --step next`.
  A run without a terminal is resumed this way.
- Paused time doesn't count toward the test's timeout.
- With `CI` set, the test fails with `PAUSE_IN_CI`.
- Under `--step`, a test is still paused after `n` or `a`.
- A test isn't paused in an MCP `open_session {test}` session, under
  `openHost` or `run()`, or in a core program.
- A test isn't paused under `e2e run --watch` yet.
2026-11-19 11:00:39 +00:00
Milosz JankiewiczandOskar Kwaśniewski 5e9b82e6ae feat(oauth): name the SuperGrok provider spacexsi, with spacexai and grok as aliases (#1943)
Co-authored-by: Oskar Kwaśniewski <oskar@okwasniewski.com>
2026-11-19 12:07:52 +02:00
bardropandOskar Kwaśniewski a213247170 feat(test): add test.each and test.for (#1882)
* feat(test): add test.each and test.for

* fix(test): read %i and %f titles like parseInt/parseFloat, as Vitest does

* fix(test): keep each titles on one line and spread only all-array tables

* fix(test): type unknown each rows loosely and keep only real cycles as [Circular]

An unknown row may be spread or passed whole at run time, so EachArgs
now types only the fixtures, last. %j only reads Node's circular-structure
TypeError as a cycle; a toJSON that throws names the row. Docs say %d
keeps BigInt's n suffix and shows Symbol as NaN, and that a string title
value keeps its line breaks.

* docs(test): note that %i keeps the BigInt n suffix

* fix(test): print undefined for %j of a function or Symbol, read Unicode $keys

- %j prints undefined for what JSON leaves out, as Vitest does, so a
  function's source never lands in a title
- $name and its path read Unicode identifier characters
- inline the test mode back into registrar
- docs: note that a Vitest test.each body gets no fixtures

* fix(test): read test.each fixtures from the parameter after the row

Fixtures are now set up on demand from what the body destructures (#1891).
A test.each row runs a wrapper, (fixtures) => body(...row, fixtures), so
registration read the wrapper's own parameter and refused every row with
"must destructure its first parameter". registerTest now takes the body
and the position of its fixtures parameter (the row's length when it is
spread, else 1), and fixtureNames reads that parameter.

* fix(test): keep Error rows on one line and locate rows through a helper

- An Error in a title shows as its name and message, not its stack, and
  whatever inspect still breaks is joined, so a title is always one line
- The source location reads the whole stack: the default 10 frames cut the
  test file off a row declared through a helper and a describe
- A tagged template table is a COLLECTION_ERROR instead of one test per
  template chunk
- A bound row body is told how to wrap itself with the row first
- tests: a run sets up only the fixtures a row body names; fixtureNames at
  a position
- docs, skill, changeset: a row body names its fixtures after the row

* fix(test): restore the stack limit on a throw, read only an own raw

- stackLocations puts Error.stackTraceLimit back in a finally
- Only an own raw property marks a template table
- The wrap hint for a bound row body names one argument per row item
- skill: rows are spread only when every row is an array

---------

Co-authored-by: Oskar Kwaśniewski <oskar@okwasniewski.com>
2026-11-19 08:56:16 +00:00
Asheesh Kumar 243eaa40ec fix(html-reporter): keep the previous page when a run stops before its tests (#1915)
* fix(html-reporter): keep the previous page when a run stops before its tests

writeReport never checked run.reportPath, so a run that stopped before
its tests (NO_TESTS, a config error, an app that never started) replaced
report.html with an empty page beside the previous run's report.json and
printed an HTML report row under "Report (not written)". It now returns
early with no row, as the built-in junit and markdown reporters do, and
the FinishedRun.reportPath docs say it is also absent when the run stops
before its tests.

Fixes #1897.

* docs(runner): say run()'s reportPath is also undefined when a run stops before its tests

The same field FinishedRun.reportPath documents, in reference/runner.mdx
and the ja and zh pages. The changeset is one line.
2026-11-19 02:36:26 +00:00
Oskar Kwaśniewski 9d240ac954 feat(core): report launch() sessions in the test that opens them (#1935)
A session a test opens with launch() from @e2e-dev/core was invisible to
the test's report, and a secret passed only to launch() could reach it.

- secrets: every owner of values (config, open host, open session) shares
  them through one refcounted map on globalThis; ledgers redact what is
  shared beside them without storing it
- report: a launch session's steps are mirrored into the test's recorder
  under an actor name (launch-1), with model usage and video
- lifecycle: the attempt owns hosts and sessions the test opened, cuts
  their steps when the body ends, and closes them at its end
- reporters: list, trace page, markdown, junit and html name the session
- errors a session throws no longer carry a secret in details
2026-11-18 16:37:55 +00:00
Krzysztof MochandOskar Kwaśniewski 847a09e6db feat(test)!: set fixtures up on demand, once per worker, and in beforeAll (#1891)
* feat(test): set fixtures up once per worker and pass them to beforeAll

test.extend() takes [fn, { scope: 'worker' }]: one instance per worker and
target, torn down when the worker ends. Suite hooks receive worker fixtures.
Engines can declare isolation: 'worker'; @e2e-dev/mobile does, so its device,
app, and screen reach beforeAll, afterAll, and worker fixtures.

Refs ENG-1297

* feat(e2e)!: set up fixtures on demand

A test now sets up only the test.extend() fixtures its body and its
beforeEach and afterEach hooks name in their first parameter, the fixtures
those name, and the auto ones. A suite hook sets up the worker fixtures it
names and the auto worker fixtures. Names are read from the function's
source, as Playwright and Vitest read them.

- [fn, { auto: true }] and [fn, { scope: 'worker', auto: true }] run a
  fixture nothing names.
- A fixture can name the fixtures declared beside it in the same extend()
  call. Fixtures run in declaration order, each after the ones it names.
- The attempt is resolved as one unit before the session launches: a
  fixture only an afterEach names is set up before the first beforeEach.
  An afterEach that names a fixture whose setup failed is skipped.
- Fixture rules fail as COLLECTION_ERROR with the file and line of the
  code: a parameter that is not destructured or takes a rest property, and
  a cycle, when the file is collected; a name the target lacks, a test
  fixture named by a suite hook or a worker fixture, and a fixture named
  like an engine's, at selection, for the pairs that run on that target.
- On an engine with isolation: 'worker', a suite hook or a worker fixture
  teardown that names no engine-backed fixture opens no engine attempt.

BREAKING CHANGE: a fixture nothing names no longer runs unless it is
declared with auto: true, and a test body, hook, or fixture function that
takes the whole fixture object (async (fixtures) => {}) no longer loads.

Refs ENG-1297

* refactor(core): declare core fixtures in one table

The five core fixtures (agent, app, screen, platform, session) are one
table in @e2e-dev/core: name, scope, and factory. createFixtures builds
the built-ins object through one loop over that table and the engine's
factories, every fixture a lazy accessor behind the same gate as before.

The runner's fixture pool, test.extend() validation, and launch read the
names and scopes from the table, so the name lists and the per-name scope
checks are gone. A launch session exposes the core fixtures that do not
belong to a test: app, screen, platform.

No behaviour change. app and screen were built with the attempt and are
now built on first read, like agent and the engine's fixtures.

Refs ENG-1297

* feat(e2e): extend a built-in fixture through test.extend

A test.extend() fixture named like one the test already has extends it,
by Playwright's rule: inside its function its own name is the fixture it
extends (the built-in, or an earlier fixture of that name), and tests,
hooks, and other fixtures that name it receive what it hands to use().

  base.extend({ device: async ({ device }, use) => {
    await device.setPermission('camera', 'deny');
    await use(device);
  } })

- A plain function takes the scope the extended fixture has on the
  target: device is a worker fixture on mobile, so its override runs once
  per worker and reaches beforeAll; app is a test fixture on web and a
  worker fixture on mobile. A declared scope that differs is a
  COLLECTION_ERROR naming both.
- A worker-scoped override receives what a worker fixture receives for
  that built-in, the forwarder to the attempt that is running.
- On a target without the built-in nothing fails until code that runs
  there names it; then the target has no such fixture.
- Only user code sees the override: the attempt's own built-ins, which the
  runner and the worker fixtures' forwarders read, are untouched.
- Types: the override keeps the extended fixture's type and scope, through
  an extend() overload that takes overrides without type arguments.

A fixture named like a core fixture, an engine's, or an earlier one of the
chain was refused before.

Refs ENG-1297

* docs(fixtures): name built-in fixtures and give their scope per engine

The docs now use one term, built-in fixtures, for agent, app, screen,
platform, session, and the ones an engine contributes (browser, device),
where pages said core fixtures, contributed fixtures, or engine fixtures.

The test reference gets one table with each built-in fixture's scope on
web and on mobile, and says what the scope decides: a test fixture reaches
test bodies, beforeEach, and afterEach; a worker fixture also reaches
beforeAll, afterAll, and worker fixtures. The hooks, worker fixtures, and
override sections link to it, as do the fixtures guide and the engine
pages. The Japanese and Chinese pages carry the table and the same terms.

Refs ENG-1297

* docs(ja): translate the fixture scope and on-demand changes

* docs(zh): translate the fixture scope and on-demand changes

* fix(e2e): let a worker fixture name an extended app or screen

A fixture that extends `app` or `screen` was taken for a test fixture when
its test.extend() call was checked, before any target is known. A worker
fixture declared after it that names `app` then failed at import on every
target, though the same fixture is accepted without the override and the
plan resolves both to worker scope on an engine whose state outlives a
test.

The scope of a plain function is now unresolved until a target is known,
whatever it extends, as it already was for an engine's own fixtures.

Refs ENG-1297

* fix(e2e): fail a suite hook that left an engine step running

On an engine whose state outlives a test, a beforeAll or afterAll hook, or
a worker fixture's teardown, can start a device step and not await it. The
hook's attempt then closed under the step, and its failure arrived later
as a stray rejection charged to whichever test ran then.

The suite attempt now does what a test attempt does once its body settled:
the step is failed at its call and cancelled, its outcome is awaited within
the cleanup budget, and the hook fails with STEP_NOT_AWAITED (HOOK_FAILED
for a suite hook).

The suite attempt keeps no step records, and StepRecorder dropped the
record of the step it had just opened before marking it running, so there
was nothing to abandon: a step now counts as running before the retained
window is trimmed.

Refs ENG-1297

* refactor(e2e): keep the fixtures a function names beside it

Registration parsed each test body, hook, and fixture function for the
fixtures it names, dropped the result, and left it in a WeakMap that the
planner and the pool read back with placeholder subjects. A cache miss
there would have named the wrong code and no line.

The parse result is now data: `names` on the registered test, the hook,
and the fixture definition. The planner and the pool read it, the WeakMap
and the per-function wrappers are gone, and one describeUser() gives an
error its subject. The explore test, registered by hand, parses its body
where it builds the record.

Unit tests state the names instead of generating source text for them.

Refs ENG-1297

* refactor(core): resolve built-in fixtures in one place

The core table declared the core fixtures, but what a target's built-in
fixtures are was still assembled around it: core names plus engine names in
three files, the 'engine' scope resolved in e2e through a cast into the
table, `platform` hard-coded as the one fixture that needs no attempt, and
two structural copies of the target.

core now exports builtInFixtures(target): names, scope(name),
needsAttempt(name), attemptless(name), and the unknown-fixture message,
over one BuiltInTarget. `platform` declares in the table that the target
alone decides it. launch, the planner, and the pool read from it.

DECLARED_BUILT_INS is the same view before a target is known: a scope the
engine decides is 'unresolved'. The planner takes either one, so its
no-target mode and the `target === undefined` branches through scopeOf and
read are gone. The scope of a fixture that extends `app` before a target
is known is 'unresolved' by construction.

Refs ENG-1297

* refactor(e2e): hand each function the fixtures it names

Every test body, hook, and fixture function must destructure its first
parameter, so what it reads is known before it runs. bindFixtures still
built a Proxy-gated bag for arbitrary property reads: one value in three
stores, the gate run twice (once over a faked plan), a third derivation of
scope, and an own-name shadow that re-derived what the planner had resolved.
No test code could reach the gate's throwing branches; only unit tests did,
by reading the bag directly.

The plan now records, per function, what each name it reads resolved to: a
chain fixture or a built-in. The pool hands each function a plain object
with exactly those names, the value a chain fixture handed over, or a
built-in read off the attempt when the function destructures it, so
built-ins stay lazy. An override that names itself receives the fixture it
extends because that is what its own name resolved to.

- bound.call(user) replaces user.fn(bound.fixtures); ready(user) asks
  whether every chain fixture it names was provided.
- attemptlessBuiltIns is a function from a name to a value that throws the
  reason for a built-in that needs an attempt.
- The planner runs once: it records what each fixture names, and the order
  is one declaration-order walk over that record. A second validating pass
  only reordered.

Behaviour: a bound or native function, whose parameters cannot be read, is
a COLLECTION_ERROR; it used to parse as naming nothing and still got lazy
built-ins. The docs no longer promise hooks and the body one shared object.

Refs ENG-1297

* refactor(e2e): move the suite attempt out of TargetExecutor

dispose, openSuiteHooks, and openSuiteAttempt were 177 lines of a second
attempt lifecycle inside TargetExecutor, written as closures in one method.

run/suite-attempt.ts now owns it: a SuiteAttempt class with open, acquire,
within, and close, over a SuiteAttemptHost interface the executor satisfies
(the way serial.ts borrows it through SerialHost), plus the worker
fixtures' teardown pass. It is the realm's SuiteHookSession itself, so the
executor keeps a context object and two call sites.

The three branches that built an attemptless reader with a "not available
to the <label>" error are one NoAttempt value ('isolated', 'unnamed',
'interrupted', or the launch failure) and one mapper to the message.

execute.ts: 1244 lines to 1052.

Refs ENG-1297

* refactor(e2e): share the attempt's fixture sequence

A run's attempt and an `e2e mcp` session's test each planned the test's
fixtures, bound them, entered the worker pool, set them up, ran the
beforeEach hooks and the body, filtered the afterEach hooks that can still
run, and labelled the teardowns. This PR widened that copy, and
standalone-test.ts asked the realm for the same hooks four times.

planAttemptFixtures(target, test, hooks) plans before any session exists
and returns open(), which binds the plan to the attempt: enter(), run()
for setup, hooks, and body, and cleanup() for the labelled afterEach hooks
and teardowns. Both call sites keep their own phases, budgets, and error
handling around it.

execute.ts: 1052 lines to 1031, its size on main.

Refs ENG-1297

* refactor(e2e): open the suite-hook pass without a lazy thunk

withSuiteHooks handed its work a thunk that opened the pass on first use.
Opening one does no I/O: the suite attempt launches nothing until a hook
that names an engine fixture runs. The pass is now opened up front and
closed in finally, and the `opened` slot and the `hooks()` call form go.

Refs ENG-1297

* refactor(e2e): ask the registry which hooks apply to a test

The selection-time fixture check rebuilt the scope chain of a test and
filtered the file's hooks by it, which the realm already did its own way.
registry.ts now exports scopeChain(test) and hooksFor(registration, test),
and both use them.

Refs ENG-1297

* refactor(core): move the fixture types into types/fixtures.ts

types.ts grew from 1679 to 1798 lines in this PR. The fixture scope, the
worker fixtures, and what test.extend() takes (definitions and overrides)
are one coherent group that nothing else in the file defines against, so
they move to types/fixtures.ts and types.ts re-exports them. The public
surface is the same.

types.ts: 1798 lines to 1715.

Refs ENG-1297

* refactor(core): drop the redundant override branch in FixtureDefinitions

A fixture that extends one the base already has was typed three ways: a
conditional branch inside the `Extra` map of FixtureDefinitions, the
`& FixtureOverrides` intersection, and the extend(overrides) overload.

Compiling the type tests without each in turn: the intersection types an
override written beside new fixtures with type arguments, and the overload
types an override-only call without them; both fail the tests when
removed. The conditional branch does not: with it gone the tests pass, and
a base name in `Extra` is `never` again, as it was before overrides. It is
removed, and the reference's copy of the declaration follows.

Refs ENG-1297

* refactor(e2e): split fixture errors from the parser, share hookLog

fixture-names.ts parses a parameter list, and also held FixtureUseError and
declarationSite, which format an error. They move to fixture-errors.ts.
Its identifier pattern held two raw zero-width characters; they are
written as escapes now.

hookLog and LOG_MODULE were pasted into three integration tests; they are
one helper in tests/helpers/hook-log.ts.

Refs ENG-1297

* docs(e2e): name the scope rule in the selection check's summary

The summary of assertFixtureUse still listed a chain fixture named like an
engine's as an error. Such a fixture extends the built-in now; the rule
left is a declared scope that differs from the extended fixture's.

Refs ENG-1297

* refactor(e2e): take execute.ts under 1000 lines

Two more pieces that are not the attempt's own sequence leave the file.

- Holding steps for `e2e run --step`: reading the screen a hold starts on
  and wiring AttemptHolds to the runner's hook move to step-hold.ts, beside
  the holds they configure (holdAttemptSteps, readHeldScreen).
- The result of a finished test or setup unit is a pure function of the
  pair, the verdict, the attempts, and a beforeAll failure: unitResult in
  units.ts, beside pairResult.

execute.ts: 1031 lines to 956. It was 1031 on main and 1226 at the review.

Refs ENG-1297

* test(e2e): match the not-awaited message without a step label

Main's #1880 names an unlabeled step by its api alone ("gadget.hang
finished"), where the message used to quote an empty label. The suite-hook
test's pattern required a label between the two, so it failed on the merge
with main.

* feat(e2e): an override inherits auto from the fixture it extends

A fixture that extends another and does not state `auto` now has the
extended fixture's, the way it has its scope: an override of an auto
fixture is set up for every test, and one of a built-in fixture on demand.
An `auto` that differs from the extended fixture's is a COLLECTION_ERROR
naming both, worded like the scope mismatch. This matches Playwright.

It also fixes which fixtures the plan starts from. An auto fixture was a
root by its own flag, so with an auto base and its override in one chain
the base was set up and the override skipped, while the name resolved to
the override; an override that replaced the base without naming it still
left the base running. An auto fixture is now a root only where its name
stands for it: the base runs once through the override's own name, or not
at all when the override replaces it.

Refs ENG-1297

* fix(core): name the hook or fixture that left a step running

STEP_NOT_AWAITED said "the test body returned before ..." for a step a
beforeAll, afterAll or beforeEach hook, a fixture's setup or a worker
fixture's teardown left running. The recorder now keeps the owner each
step was started under, and the runner says which code runs: the pool
for fixtures, hooks and the body, the suite attempt for suite hooks, and
a hosted session for itself.

---------

Co-authored-by: Oskar Kwaśniewski <oskar@okwasniewski.com>
2026-11-18 13:09:51 +00:00
Natasha Tran 902f044b49 feat: add a dot reporter (#1911)
* feat: add a dot reporter

* fix: flush dot output and separate list formatting

* docs: clarify buffered dot reporter output
2026-11-18 12:58:46 +00:00
Dery Ferdika f5cda16313 feat(html-reporter): filter a run's results by target (#1793) 2026-11-18 14:49:06 +02:00
Krzysztof Moch 6fbcb66f02 docs: show how to connect e2e mcp to OpenCode (#1926)
Refs ENG-1317
2026-11-18 10:02:37 +00:00
Kacper K d16ad2c2d5 feat(cli): add e2e report command to open HTML reports (#1892) 2026-11-18 08:39:35 +02:00
ParbezandOskar Kwaśniewski fabb0c07fb feat(expect): take an optional message on locator expectations (#1865)
* feat(expect): take an optional message on locator expectations

* docs(expect): name expect.soft(locator, message) and scope the message to assertion failures

* feat(expect): open an ambiguous locator's error and expect(screen) failures with the message

* fix(expect): open an ambiguous locator's toHaveScreenshot error with the message

labelAmbiguous moves to expect/heading.ts and wraps the screenshot capture, so
expect(locator, message).toHaveScreenshot() labels LOCATOR_AMBIGUOUS as the
polling matchers do. Tests cover it, a screen .not and the CI no-stored path
with a message. The reference drops a sentence about step names.

* docs(expect): update the ja and zh pages for the locator message

The translations landed on main after this branch was cut, so the English edits left docs/ja and docs/zh stale and docs:check failed on the code blocks of reference/expect.mdx. Bring both languages level with the English pages, the Playwright guide rows included, and rewrap one line of the English soft assertions paragraph.

---------

Co-authored-by: Oskar Kwaśniewski <oskar@okwasniewski.com>
2026-11-17 21:30:02 +00:00
Samanyu DasandClaude Opus 5.5 d960d2c448 fix(engine): wait for a crashed engine server's exit before its connection closes (#1863)
* fix(engine): wait for a crashed server's exit before its connection closes

spawnTransport handed the child's stdout to the connection directly. When
a server exits mid-request, its stdout can end before Node delivers the
exit and the rest of its stderr, and the connection reads both for its
message the moment it closes. The request then failed with only "the
peer closed the connection", which made the engine-protocol crash test
flaky on CI (Node 22).

The connection now reads stdout through a stream that ends once the
child has closed, or 2 s after stdout ended, for a server that stays up
or a grandchild that holds its stderr.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(engine): pin the wait on close, and time only the failing request

A crashed server's child that shares its stderr now has a test, which
fails when the transport waits for the exit instead of the close. The
stdout-first test measures the request instead of a 1.5 s test timeout,
which failed under load. The cap is a cleared timer, the changeset and
docs name the signal case, and the fixture's modes are one option.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(engine): end a server's stdout from a generator, and test the signal and the cap

The input is now an async generator over the server's stdout that waits
for the close or the 2 s cap, so a stdout error reaches the connection as
an error event and the cap's timer is aborted, by construction. New tests
pin the signal case the docs name and the cap's length, and the crash
helper takes the fixture's mode type.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(engine): say what a crash error holds after the cap, and what keeps a request open

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(engine): cap the stdout wait with an unref'd delay, and pin the cap from both sides

The aborted delay's .catch could never run and nothing tested the
abort. An unref'd delay needs neither, the pattern step-control.ts
uses. The stays-up test now fails a cap under 1.9 s or over 3 s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 21:01:20 +00:00
9602daa960 feat(notte): add @e2e-dev/notte, a browser provider for Notte cloud browsers (#1726)
* feat(notte): add @e2e-dev/notte, a browser provider for Notte cloud browsers

notte() runs each test attempt in its own Notte cloud browser session and
stops it when the attempt ends. It leases in attempt scope only: a Notte
session refuses Target.createBrowserContext, which worker scope needs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(notte): downloads, sweep, and console links, verified on the testbed

- downloads.read copies each download from the session's file storage,
  found by the site's suggested name. @e2e-dev/web now passes that name to
  providers as BrowserDownloadContext.suggestedFilename (optional,
  additive).
- sweep stops the sessions of workers that died, found through per-session
  marker files under the system temp directory, since Notte sessions have
  no tags.
- The progress line links the session's console page instead of the
  token-bearing viewer URL.
- A start response without cdp_url falls back to the session's debug info.
- apps/testbed gets e2e.notte.config.ts and test:notte.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(notte): address review: fail-closed ledger, required suggestedFilename, paged downloads, shared sweep helper

* docs(notte): add notte() to the security table and skill setup

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Oskar Kwaśniewski <oskar@okwasniewski.com>
2026-11-17 19:38:50 +00:00
Oskar Kwaśniewski 51d304e736 fix(core): cancel running calls by name when a session closes (#1880) 2026-11-17 20:58:39 +02:00
Kayaan TharaniandOskar Kwaśniewski f659d64d80 feat(revyl): add @e2e-dev/revyl for mobile tests on Revyl devices (#1708)
Co-authored-by: Oskar Kwaśniewski <oskar@okwasniewski.com>
2026-11-17 20:55:14 +02:00
Matt Van HornandMatt Van Horn 95c574adec fix(agent): strip unknown fields on no-input agent tools (#1835)
* fix(agent): strip unknown fields on no-input agent tools

A tool with an empty schema shape still uses the strict schema, which
keeps the provider-facing input closed, but a plain object is reduced to
an empty object before that check, so extra keys are dropped and the
call runs. Values that are not plain objects still fail validation.
Tools that declare fields still reject unknown keys, and screenshot does
too once a path field is present. The security and reference docs and a
patch changeset describe that split.

Fixes #1812

Signed-off-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>

* fix(agent): trim schemaTool comment and cover array input

Cut the schemaTool JSDoc to why an empty schema strips, drop the
changeset's provider-schema sentence, and add an array to the values a
no-input tool rejects.

* docs(security): drop provider schema detail and tighten tool rules

---------

Signed-off-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
2026-11-17 11:23:33 +00:00
Samanyu DasandClaude Opus 5.5 1d0271b306 feat(test): add test.skipIf and test.runIf (#1870)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 13:19:26 +02:00
Oskar Kwaśniewski 6c4eddd0f9 feat(cli): ask for a GitHub star after init, upgrade, and update notice (#1874) 2026-11-17 13:17:51 +02:00
Asheesh Kumar d4fb5ccd49 feat(expect): add toHaveRole locator matcher (#1857) 2026-11-17 10:24:13 +02:00
Donny/강동윤andClaude Opus 5.5 aadf95506d fix(report): keep \r\n line breaks in serialized errors and the JUnit report (#1860)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 10:20:56 +02:00
Donough Liu 29a0261775 feat(moli): add managed local browser provider (#1843)
Manage local Moli CDP browsers through the web engine's BrowserProvider API.
Each worker gets an available loopback port, layout and resource loading,
startup cancellation, and process cleanup when its lease ends.

Wire the testbed, documentation, consumer skill, build and test commands,
and a patch changeset for @e2e-dev/moli.

Keep process-group shutdown active after an executable wrapper exits,
and surface process-inspection failures in the provider tests.

Verified locally:
- pnpm check
- 16 Moli provider unit tests
- Testbed on two workers: 16 passed, 1 expected skip; both browsers released
- Empty and whitespace CDP URLs use managed browsers; padded URLs attach
  without stopping the existing server
- MCP login, secret fill, assertions, and screenshot
- Packed packages installed, typechecked, and run in an isolated project
2026-11-17 07:03:27 +00:00
Oskar Kwaśniewski 705c049205 fix(web): leave descendants with their own role out of a name from content (#1853) 2026-11-16 20:33:09 +02:00
Donny/강동윤andClaude Opus 5.5 0cd14a0dfa perf(core): read a wait again at 20 and 50 ms, then every 100 ms from its start (#1834)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 19:29:29 +02:00
Donny/강동윤andClaude Opus 5.5 1b7c3a99d6 perf(agent): keep the act tool set the same across steps for the prompt cache (#1823)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 19:27:52 +02:00
Kacper KapuściakandOskar Kwaśniewski d593863b41 feat(cli): write a GitHub Actions or GitLab CI/CD pipeline in e2e init (#1647)
* feat(cli): write a GitHub Actions or GitLab CI/CD pipeline in e2e init

e2e init asks for a CI pipeline in a project with the web engine (--ci github-actions|gitlab|none). It writes one job for the project's package manager and Node version that installs Chromium, runs e2e run, and keeps the report, and names the secret or CI/CD variable for the model key. It never overwrites a pipeline file; beside an existing .gitlab-ci.yml it writes a file to include.

* fix(cli): handle cubic's findings on the init CI step

Exact GitLab stage names, a distinct job name beside an existing e2e job, loopback baseURL in an existing config, no writes through a symlinked .github or .gitlab, --ci none in the skill, gitlab in the telemetry comment, and the GitLab Node image's major version in the docs.

* refactor(cli): keep the init CI pipelines as YAML templates

The GitHub Actions workflow and the GitLab job live in packages/e2e/templates/ci/ as complete pipelines for npm at the repository root, linted as they are. init loads them with the yaml package's Document API and edits a copy for the project's package manager, folders, Node version, secrets, and target, keeping the comments. The pinned actions now live only in the templates.

* fix(cli): point setup-bun at the package.json of a project in a subfolder

* docs(ci): link the GitLab page from init and the GitLab template

* feat(cli): write the iOS and Android CI jobs in e2e init

init writes one job per platform the config's targets run on: web targets in the web job, iOS and Android targets in their own jobs, which build the app, boot a simulator or an emulator, and install the app. Targets come from the new config, or from loading an existing one after the confirmation and the install. Expo and bare React Native apps get their build; any other app gets a step to replace. GitHub's mobile jobs follow e2e-inc/e2e-react-native's workflow; GitLab's run on self-managed runners.

* fix(cli): reinstall an emulator system image that holds no image in the Android CI job

* fix(cli): handle cubic's findings on the mobile CI jobs

Upload the emulator log from the GitLab Android job, count peer and optional engine dependencies, ask nothing for a new config with no engine, key caches on Bun's bun.lockb when that is the lockfile, and run the symlink test with a Windows junction.

* fix(cli): put current Android command-line tools in place before the emulator image

The runner image's cmdline-tools can be older than the system image's package format, and android-emulator-runner uses them to create the AVD, which then fails with 'contains no system images'. The Android job downloads command-line tools 23.0, checks its SHA-256, and installs the image with it.

* fix(cli): import moved core modules from @e2e-dev/core in init

* fix(cli): count a pipeline as set up only when a command runs e2e run

init matched the text 'e2e run' anywhere in a workflow, so a comment or a step name such as 'our old e2e runner' made --ci write nothing. It now reads the commands (run, script, before_script, after_script) from the parsed YAML and matches e2e run as words of their own.

Also: --ci help and the InitOptions JSDoc say tests on every platform, and yaml is pinned exactly like the other dependencies.

* chore(ci): bump setup-node and upload-artifact pins in the init templates, docs, and skill

docs/ci/vercel-previews.mdx on main pins setup-node v7.1.0 and upload-artifact v7.0.2, the pins this repository's own workflows run. The templates, three docs pages, and the skill still had v7.0.0 and v7.0.1, so the pin test failed once main was merged in.

---------

Co-authored-by: Oskar Kwaśniewski <oskar@okwasniewski.com>
2026-11-16 17:21:49 +00:00
Donny/강동윤andClaude Opus 5.5 3422abfb6a fix(agent): judge a waitFor screen change without waiting out the interval (#1819)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 19:22:00 +02:00