mirror of
https://github.com/tester-army/e2e.git
synced 2026-11-22 18:19:48 +08:00
main
110
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
7e3205461b |
feat(expect): expect(fn).toPass() and expect.poll intervals (#2013)
* feat(expect)!: expect(fn).toPass() and expect.poll intervals
A block of assertions can be retried with `expect(fn).toPass({ timeout,
intervals })`, including the `.not.toPass()` and `expect.soft(fn).toPass()`
forms. Defaults are set with a top-level `toPass` config key. Both APIs
run on one retry loop.
BREAKING CHANGE: `expect.poll` no longer takes `interval`. It takes
`intervals`, a list of pauses whose last entry repeats. The default pauses
are `[100, 250, 500, 1000]` ms where they were a flat 100 ms.
Steps still running in a poll read or a toPass try are cancelled when the
try fails or is cut at its deadline. Later step calls from that try are
refused.
When a retry reaches the phase deadline, the runner's timeout is the
reported error. The last retry error is kept under `secondaryErrors`.
In `beforeAll`, `afterAll`, and worker fixture teardown, both APIs are
bounded by the hook deadline and use the config defaults.
A call `timeout` above the timer limit is `INVALID_CONFIG`.
* fix(expect): let skips through a retry and follow a moved deadline
A `test.skip()` or `test.fixme()` thrown inside a toPass block or a poll
read ends the retry right away and propagates unchanged.
The phase deadline is read on every pass, so a `test.setTimeout()` call
during a retry takes effect. The call's timeout stays fixed.
Options for `expect.poll` and `toPass` must be a plain object with known
keys.
* fix(expect): deprecate the interval option of expect.poll
`interval` works again as an alias for `intervals: [interval]`, following
the deprecation window in CONTRIBUTING.md. A warning that names `intervals`
is printed the first time it is used in a process. Passing both options is
`INVALID_CONFIG`.
* fix(expect): take a final sample at the deadline and refuse late ones
When a pause is cut short by the deadline, one more sample is taken. Its
result counts only if it settles without waiting past the deadline. A
sample that overran the time it had when it started no longer passes.
Call options for `intervals` use the config's rule, positive integers.
* fix(expect): drop the final sample at the deadline
The extra read counted only when it settled in its starting turn, so it
helped a synchronous read and never one that awaits the app or the
network. When the next pause would cross the deadline, the retry ends
without another read, as Playwright's loop does.
* fix(expect): keep a late try's skip and error
A try that settles after its time still does not pass. A `test.skip()` it
threw now reports `skipped`, and an error it threw is the `last:` line of
the timeout message.
|
||
|
|
a2d0363128 |
feat(desktop): add the @e2e-dev/desktop engine on agent-desktop (#1953)
* feat(desktop): add the @e2e-dev/desktop engine on agent-desktop A desktop engine for native and Electron apps, built on agent-desktop (vercel-labs, Apache-2.0), which reads and drives apps through the OS accessibility tree. macOS is the tested platform; this is rollout phase 1 of the ENG-1352 proposal. - The engine downloads the pinned agent-desktop 1.3.1 release in prepare and checks its SHA-256; the npm package is not a dependency because its postinstall would fail check:install-scripts. - Fill text reaches agent-desktop on stdin, never in its arguments. Screenshots are cropped to the window and paint every secure field black, and are withheld when a secure field has no bounds or the snapshot was partial. - Node ids stay stable across observations, so the agent gets diffs. - The engine quits only the process it launched, checked by its process instance. - apps/desktop-testbed holds an AppKit sign-in form with deterministic and agentic suites. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(desktop): accept the remaining budget in the fill arguments The timeout agent-desktop gets is what is left of the operation's deadline, so it can be a millisecond under the full budget on a slow runner. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(desktop): report field values whole and keep a typed-into field's id locate answers single-node reads, which the contract says carry the whole value; the harness cuts what the agent sees. An unnamed field's identity no longer includes its value, so typing into it is a change, not a remove and an add. An image is projected as the contract role image, which both locator spellings resolve to. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(desktop): withhold a capture with a border instead of masking it in place A capture within four pixels of the window was taken as the window, so a two-pixel border shifted every mask. Only a pixel of rounding is accepted now, any other capture is cropped to its opaque box, and each mask reaches two pixels past its field. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(desktop): give a read exactly its budget The 500 ms floor exists so a click that landed is not reported as cut off; a read has no such reason and now stops at the operation's deadline. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(desktop): keep agent-desktop out of the replay cache and report install failures E2E_CACHE_DIR names the replay cache, which a project may commit, so the binary goes to the OS cache directory only, found from the run's environment. A network or tar failure in prepare is now AGENT_DESKTOP_INSTALL_FAILED, as the docs say. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(desktop): offer doubleTap only with headed, and say what app.open and settle do agent-desktop double-clicks with the real pointer only, so doubleTap is declared with headed: true and sent with --headed. The docs said app.open() quits the app if it runs; it quits only the copy the engine launched. settle is a fixed wait, not a check that the UI is still. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(desktop): drive only the launched process, and keep ids and the select ladder honest - Every read and scoped command goes to a window of the process the engine launched; before app.open() or after the app quit, the engine reads and drives nothing, so a developer's own copy is never touched. - With appPath, the launched executable must come from that bundle; another copy registered under the same bundle id fails the open. - quit keeps the process as the engine's own until it is known to be gone, so a failed lookup is reported and dispose tries again. - Node identities are scoped to their window and retired once gone or ambiguous, so neither a twin nor another window's control inherits an id. - The select ladder passes the label after --, reports a stale ref as NODE_STALE, and stops once any command of a way reached the app. - A press on a control that refuses focus is refused unless it holds focus. - A root swipe after a restart reads the window again. - app.open() waits at most 30 s for a readable window, reports any other read failure as it is, and is cancelled with its operation. - A screenshot is withheld when the window read after the capture differs in its secure fields. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(desktop): read placeholders, mixed checkboxes, and depth-cut trees as agent-desktop reports them - A placeholder comes from agent-desktop's own placeholder field; the hint is kept as an attribute of its own. - A mixed checkbox reports no checked state, so check() and uncheck() always reach the control instead of returning early. - A node that counts children it did not list (the depth limit) marks the tree as cut short, so a screenshot from it is withheld. - Siblings that share a developer id (AppKit menu items) tell themselves apart by name and keep their ids; a status with a developer id keeps its id while its text changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(desktop): launch an appPath by path and keep every process a launch started - An appPath bundle is opened by its path (open -n), and the new process running from it is taken as the engine's own; launching by bundle id let LaunchServices pick another copy with the same id. - A launch that failed after the app started still leaves the engine owning the new process, so the end of the attempt quits it. - A window is the engine's only when its process instance matches too, so a later process given the same pid is never read or driven. - An app list that is incomplete keeps the process the engine's, to quit again in dispose. - A point action finds the window where it is now, so it works right after app.open() and after the window moved. - A type or fill after a clear that changed the field fails as ACTION_MAY_HAVE_COMMITTED, never as a retryable stale ref. - A screen larger than the binding cap keeps every node it just read. - The launch wait is 10 s, and each read in it gets what is left of it. - A bundle id that names no app says so, instead of asking for app.open(). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(desktop): refuse an archive with the wrong SHA-256 and cache nothing Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(desktop): say app.open brings the app forward, and that only macOS is tested Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(desktop): never take another copy as the launched one, and aim point actions at their window - A launch agent-desktop refused takes no process as its own, and a copy is attributed to a launch by id only against a whole app list from before it, so cleanup never kills the developer's copy. - A launch whose outcome is unknown is kept pending and settled by the next quit under its own budget, so a timed-out launch still has its app quit. - A point action goes to the window it was observed in, wherever it is now, refuses a point outside that window, and brings it forward before the physical click. - In the select ladder, ELEMENT_NOT_FOUND from select (no such option) goes on to the next way; from a command on the list itself it is NODE_STALE. - A root swipe scrolls the window as it is now. - With both appPath and bundleId, the appPath build is launched and its bundle id must match. - An open that runs out of its own budget is OPERATION_TIMEOUT; only the full launch wait reports an unreadable window. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(desktop-testbed): check the saved screenshot for the password field's mask The test checked only that the file was a PNG, which an unmasked capture also is. It now decodes the image and requires the solid black bar the password field's mask paints. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(desktop): say which app launches when appPath and bundleId are both set Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(desktop): report a read agent-desktop timed out as OPERATION_TIMEOUT TIMEOUT was NOT_ACTIONABLE for reads as well as inputs, so a snapshot that ran out of time skipped the launch wait's timeout handling. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(desktop): tell the launched process apart by a token, and measure points in their window - Every launch passes a one-time token in the app's environment (launch --env, open --env), and a process is the engine's own only when ps -E shows that token. Guessing from the app list before and after a launch could take, and kill, a copy the developer or another worker started at the same time. - A launch whose outcome is unknown is settled by the next quit, which looks for the token for up to 3 s and reports a failure when it finds none. - A point is measured in the window of the newest screen observe or locate handed out, and sent unrounded, so it never lands past the window's edge. - check and uncheck always go to agent-desktop, which reads the live state. - One window's node ids stay while another window is read. - The launch wait's commands share its deadline. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(desktop): pin option and app validation and the headed-only actions Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(desktop-testbed): measure the password mask at the capture's scale Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(desktop): keep identities from colliding, and never drop a secure node - A node identity is a tagged tuple, so a developer id and a name can never spell another control's identity (`shared` + `Edit:` against `shared:Edit`), which let one control take another's binding. - A secure node is never dropped as a wrapper, so its bounds are masked. - A binding records the window its node was read in. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(desktop): press keys only into a focused control, type keystrokes, and check the launched process's token - A press on a control needs its focus to succeed and to hold, and goes into the control's own window; before, a refused focus sent the key to whatever window had focus. - The keyboard capability is declared with headed only and types with --headed: without physical input agent-desktop sets the text, and the contract asks for keystrokes the app's handlers see. - The process agent-desktop reports for a launch must carry the launch's token; a launcher that handed over to a copy already running (a Linux desktop file) fails the open and never drives or quits that copy. - A select that finds no such option never falls through to set-value, which would type the label in as free text; a menu left open after a missing option is reported as a changed app. - A read tries the launched app's windows in turn, past a panel accessibility cannot read. - No app bundle is opened once the operation's budget is spent. - doctor fails an appPath with no Contents/Info.plist. - The unused latest-screen field is gone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(desktop-testbed): require a mask the size of the password field, not one black row Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * chore(desktop): ship a NOTICE like every other published package Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(desktop): say typing into whatever has focus needs headed Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(desktop): withhold masks with no area, and give controls their title as text - A secure field whose box has no area masks nothing, so the screenshot is withheld. - A control reports its title as its text, as on a device, so textContent() reads a button's title; an input's name is a label, so it gets none. - A binding keeps its node's box, to find a look-alike again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(desktop): find a pressed look-alike by its place, keep unresolved launches, and test on the host's ps - A press checks focus on the control found again by role and box, since a look-alike has a new id in every read; the key is resolved before focus moves, and a node read in no window is refused. - An environment that cannot be read (a macOS system app under SIP, a failed lookup) is no evidence: agent-desktop's report of the process it launched stands, so system apps open again and are quit at the end. - A pending launch stays until it is claimed or a whole wait finds none. - A window row is the engine's only with the launched process instance. - A screenshot finds secure fields in whole reads of the window, even with interactive snapshots, which leave out a secure node with no action. - A menu read in part never reports an option missing. - The unit tests read process environments the host's way (ps -E or /proc), so they pass on the Linux CI runner, and remove the fake's scenario directories. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(desktop): dispose every surface a test creates, so no state directory is left behind Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(desktop): drop Electron, so the engine targets native apps only Electron apps are out of scope for this engine. Remove the paths that existed only for them: the headed type fallback for a field the accessibility API cannot set (Electron on Linux), and the expand and menu-click steps of the select ladder (Electron on Windows and macOS). A select now tries `select`, then sets the value of a list that takes none (WinForms), and reports an option `select` cannot find as missing. The Electron snapshot fixtures go too; the wrapper and shared-id tests use the AppKit sign-in form and an inline AppKit menu. Docs, the README, package metadata, and the skill now say native apps. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * chore(desktop): match the workspace's vitest 5.1.1 Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * fix(desktop): apply the review of #1953 macOS only. prepare and init refuse any other OS: on Windows and Linux some actions need physical input and GTK's password field is not reported as secure, so masking would fail open. A config with a desktop target still loads elsewhere. The app under test and agent-desktop get only HOME, USER, LOGNAME, SHELL, PATH, TMPDIR, LANG, LC_*, and __CF_USER_TEXT_ENCODING of the run's environment, so secrets, model keys, and CI tokens never reach them. `open` and NSWorkspace both handed the caller's whole environment to the app. The app launches in the background (`open -g`, `launch` without `--activate`) unless headed, so typing at the Mac no longer lands in it. An app that shows no window after 2 s is brought forward once. Every launch passes -ApplePersistenceIgnoreState YES. An appPath app whose environment cannot be read (a system app, a build signed with `restrict`) is now claimed as the single new copy that was not running before the launch, instead of polling until the test timed out and leaving the app running. A launch reported for a copy that ran before it is refused. An open that finds no process fails within 10 s. select no longer falls back to set-value, which could type a missing option into an editable combo box. press sends a key only when the control's window is the app's key window, and finds the control again by its stable id. locate fails on a truncated read with no match instead of proving absence. The reads a screenshot's mask takes hand out no ids. The window's own Close, Minimize, Zoom, and Full Screen buttons are left out of the tree. desktop.useWindow() picks the window screen reads. Stale state directories of killed workers are swept at init. The cached agent-desktop is reused only when a verified unpack stamped it and its executable and helper are intact. doctor says the permission checks need the first run. A pending launch whose lookup timed out stays pending. Tests: the screenshot-moved test now uses the window the fake lists, and a steady capture is checked to succeed. Each fix has a test that fails without it (checked by mutation). The testbed checks the password field's own box is black and the field beside it is not, and test:desktop-testbed builds the app. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(desktop): keep launches pending after open, and fix four review gaps A fresh-context review of f0d1bb69f found these, each now with a test: - openBundle: a failure after `open` ran (a list-apps error, or the budget running out while looking) was read as "nothing was sent", so the copy `open` started was never quit. Any failure after `open` is now ACTION_MAY_HAVE_COMMITTED and leaves the launch pending. - An app that quits right after launch now fails the open at once, before the engine brings its bundle forward, which would have started a copy the engine does not own. - A node inside a look-alike container is no longer marked stable: its id is built on its parent's, which is new in every snapshot, so press finds it by place again. - Only a title-bar-sized button (at most 20 points) with a window control's name is left out; an app's own Close button stays. - A window chosen with desktop.useWindow() that closed is now INVALID_STATE, not an empty screen a negative assertion would pass on. - The release cache is swapped by rename instead of deleted first, and doctor checks the stamped copy, not only that the file exists. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(desktop): send keys and close select menus in the background, and quit apps killed workers left Found in live runs on an unlocked Mac: - A key press never reached an app in the background: agent-desktop sends a key to a named window only while that window is active, and answered STALE_REF, which the harness retried until the step timed out. The key now goes to the process by its app name, once the engine saw its copy is the only one with that name; with another copy running, press fails at once with NOT_ACTIONABLE and says to use headed. - selectOption of a missing option on a pop-up button opened its menu, waited out the whole action budget, and left the menu open. select now waits at most 3 s; a menu it left open is closed with Escape and the option reported missing (NOT_ACTIONABLE), the list unchanged. - A worker that was killed left its app running. Each worker records the app it launched and its token; the next run's sweep quits that app while its environment still shows the token. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(desktop): sweep a state directory whose launch record was cut short A worker killed while writing app.json left half a JSON record. The sweep parsed it inside the promise's success handler, so the SyntaxError escaped and every later init failed on the same stale directory. A record that does not parse now names no process and the directory still goes, and the record is written to a side file and renamed into place. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(desktop): keep the state directory when dispose could not quit the app dispose removed the state directory even when its quit failed, and app.json went with it, so the next run's sweep had no pid and token to quit the leaked app by. The directory now stays while the engine still owns a process it could not quit; the owner file names a worker that is gone by then, so the sweep removes the directory after it quits the app. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(desktop): claim a copy with no readable token only from whole app lists Two paths took a process as the launch's own on evidence an incomplete app list cannot give, and the engine later kills what it owns: - launch by bundle id accepted agent-desktop's reported pid when its environment could not be read and the list before the launch was partial, so a copy already running could not be ruled out; - claim() took the single new unreadable copy from a list after the launch that left copies out, where a second new one could hide. The first now refuses the launch and leaves the process alone. The second reports that it cannot tell; quit keeps the launch pending as it does for a lookup that ran out of time. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(desktop): give a background key only what is left of its budget sendKey built its command options once, so the key sent by the app's name, after a refused window press and a list of the apps, still carried the budget the operation had at its start and could run past the deadline. The options are now built for each command from the time left. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(desktop): keep a select whose menu was left open as possibly committed After a select that may have reached the app, closing a menu the app had open was taken as proof that the option was missing and nothing was chosen, and reported as NOT_ACTIONABLE. An open menu shows neither: it does not say which control opened it. The failure now stays ACTION_MAY_HAVE_COMMITTED, naming the option the list may lack, after the menu is closed. A failure to close the menu (Escape refused while another copy runs) no longer replaces the select's own error. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(desktop): measure a point from the window useWindow() chose performAt measured a point from the window the last read used. After desktop.useWindow() picked another window, the next tapAt still brought the old window forward and clicked there. Choosing a window, or going back to the focused one, now drops that window, so the next point action measures from the window in use. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(desktop): read no empty screen from an app that cannot be read capture() skipped every window it could not snapshot and returned no screen, so observe showed an empty window and locate found nothing. While the screen is locked, or after the app crashed, not.toBeVisible() then passed. Opus reproduced it against the testbed app on a locked Mac. Now: - listed windows none of which could be read are a retryable NODE_STALE, so a locator reads again until its deadline; - a chosen window whose snapshot says it is gone is INVALID_STATE, like one missing from the list; - an app whose process is gone is INVALID_STATE. An app with no window at all still reads as an empty screen. The launch's readiness wait treats the new errors as a window not readable yet. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(desktop): count no matches in a window read in part locate() refused a partial read only when nothing matched, and then with a non-retryable ENGINE_FAILURE. One visible match in a read cut short still counted as exactly one, though a second could sit in the part left out, so a strict action or toHaveCount(1) could pass on it. And a locator for a control not there yet failed at once instead of polling. A partial read is now a retryable NODE_STALE whatever it matched, so the locator reads again until its deadline. The docs say so for a window deeper than the 24 levels a snapshot reads. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(desktop): correct getByText on buttons, and macOS-only metadata - The desktop page said getByText cannot find a button by its title. The projection gives a button its title as text too, so it can; getByRole is what limits a match to the button. - The package keywords listed windows and linux, which the engine refuses. - CONTRIBUTING said a contract change bumps both engines; there are three. - @e2e-dev/desktop moves to its alphabetical place in the docs package. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(desktop): cover the appPath launch and a successful install Every surface test launched by bundle id, so a change to the open -g -n arguments, the claim of the copy carrying the token, or its quit would not fail a test; and the binary tests covered only failed installs. - An appPath test puts a fake open first on PATH: it records its arguments and starts a process with the variable --env names, which the engine claims, reads, and quits. - An install test serves an archive of its own through a digests table installPinned now takes (the pinned one by default), installs it twice at once, and checks the stamp, that nothing is left beside the entry, and that a third install fetches nothing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(desktop): find the app's running copies the way agent-desktop launches it runningCopies() and claim() listed every app and kept the ones whose bundle id or name equalled the target exactly. agent-desktop reports the localized name (Calculator is 계산기 on a Korean Mac) and matches a launch target in any case, so a copy the launch could hand over to was left out of the list before it. A copy whose environment cannot be read was then taken as the launch's own and killed at the end of the test; and a copy a launch of unknown outcome started was never found, and left running. Both now ask list-apps --app <target>, which matches the way the launch and quit() do, and count every process it returns. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(desktop): stop the fake apps before removing the list of them afterEach ran cleanupFakes() first, which deletes the file fake.spawned() reads, so the processes a test left running were never killed and stayed for their 60 s timer. They are now killed first; no stand-in process is left after the suite. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(desktop): report an open LaunchServices refused as starting nothing openBundle wrapped every failure of open as ACTION_MAY_HAVE_COMMITTED, so a bundle LaunchServices refused (a missing executable, a bad signature) left the launch pending. The end of the attempt then looked for its process for 3 s and failed a second time, telling the user to quit an app that never started. The error also quoted the whole open command line. An open that exits with an error, or cannot run, is now ENGINE_FAILURE with open's own message, and nothing is left to look for. Only an open the budget or a cancellation stopped stays possibly committed. The appPath tests share a helper that puts a stand-in open on PATH. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(desktop): record the launch token before the launch, and sweep by it app.json was written only once the launch's process id was known. A worker killed during the launch (which can wait up to 10 s for its process), or a launch whose process was never found, left no record, and dispose then deleted the state directory too. The next run could not find the app. The record now holds the launch token and is written before anything is sent. dispose keeps it while a launch is pending, as it does for an app that would not quit. The sweep quits every process whose environment carries the token, found by listing the processes, so it needs no process id and cannot reach a process that took one over. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(desktop): act on no node read outside the window useWindow() chose useWindow() changed the window every read used but kept the nodes read before, so an action on a ref from the old window still reached its control after a test chose another. Such a ref is now NODE_STALE, which the harness answers by locating the node again in the window in use. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(desktop): read no empty screen from an app whose process id was reused With no window of the launched process listed, capture() checked only that its process id was alive. Once the app quits, macOS can give that id to another process, so the read showed an empty screen and a negative assertion could pass. The app list must now show the same process instance; a list that left copies out is a retryable NODE_STALE. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(desktop): make doctor check an appPath bundle the way init does doctor passed any bundle with a Contents/Info.plist, while init also needs a CFBundleIdentifier in it, equal to app.bundleId when that is set. A bundle doctor passed could then fail init, and a plist with no bundle id failed init with plutil's raw error. Both now use one check, which reports a missing or mismatched bundle id as the problem. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(desktop): never move another install's verified release out of the cache download() checked whether the cache entry was usable and, if not, moved it aside before renaming its own unpack in. Two installs that both found no usable entry could interleave: the second moved aside the release the first had just put in place, and a worker starting between its two renames found no executable. The unpack is now renamed into place first. A rename onto an existing entry fails, and then the entry is used if it checks out, and moved aside only if it does not. The race has no deterministic test; a new test covers replacing a damaged entry, the path this reorders. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(desktop): say when two desktop targets share a run, and what reads empty - The workers line said two desktop targets can run at the same time while telling headed runs to set workers: 1, which is what stops them. It now says they do, and why headed runs set workers: 1. - The Reading row and capture()'s comment said a window is never read as an empty screen, but an app with no window open still is, on purpose. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * chore(desktop-testbed): pin ai to the version every other workspace uses The testbed pinned ai 7.0.173 while the rest of the repo pins 7.0.177, which put a second copy of ai and its gateway and provider packages in the lockfile. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(desktop): report an open cancelled mid-launch as CANCELLED An abort while open ran, or while the engine looked for the process it started, came out as ACTION_MAY_HAVE_COMMITTED, where every other path of the engine reports CANCELLED. It is now CANCELLED; the launch stays pending as before, so the end of the attempt still looks for the app. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(desktop): settle a pending launch by its last lookup and by its token quit() dropped a pending launch, and dispose its record, once any lookup in its wait had a whole answer. Two cases lost an app that was still starting: - an early complete list with no match, followed by partial ones that could hide the app once it registered; - a process carrying the launch's token that the app list did not show yet. Now only the wait's last answer counts; a lookup that ran out of time is no answer. Before giving up, quit looks for processes carrying the token and quits them. The launch stays pending, and its record kept, unless the last answer was whole and found nothing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(desktop): run launches and quits of one surface one at a time Two overlapping app.open() calls could both quit, then both launch, writing into the one process id and launch record the engine keeps; they even shared the app.json.partial file. The end of the attempt then quit only one copy. restart, closeApp, endAttempt, and dispose now wait for the launch or quit running before them. restart's budget counts from the call, so the wait is part of it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(desktop): measure a headed click from where the window is after focus performAt read the window's bounds, brought the window forward, and clicked at the bounds read before. A window that moved as it came forward (onto another space, out from under the menu bar) took the click somewhere else, possibly another app. The window is now read again after focus-window, and the point is checked against and measured from its bounds then. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(desktop): repair a damaged cache entry under a lock 12c8f64d0 renamed a fresh unpack into place first, which keeps a run from moving aside a release another run just installed only when there was no entry. Two runs that both found an entry damaged could still interleave: the second moved aside the copy the first had just put in place. Its commit message and comment claimed more than that. Replacing an entry now holds a lock directory that names its owner's process id, and checks the entry again under it. A lock whose owner is gone, or that got no owner and is 10 s old, is taken over; one held for 60 s fails the install with AGENT_DESKTOP_INSTALL_FAILED. Tests repair one damaged entry from three runs at once and take over a lock a dead process left. The interleaving itself has no deterministic test. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(desktop): refuse getByPlaceholder, which macOS gives nothing to match agent-desktop's macOS adapter reports no placeholder for a field; AppKit folds it into the accessible name of a field with no label. A placeholder query therefore matched nothing, and not.toBeVisible() or toHaveCount(0) on it passed with the field on screen. locate() now refuses any expression that uses a placeholder query with UNSUPPORTED_CAPABILITY, naming getByRole as the way to find the field, and the docs say so. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(desktop): let desktop({ maxDepth }) set how deep a read goes A read cut short at the fixed 24-level depth makes every locator on that window read again until its timeout, since a match could sit below it. An app nested deeper could not be tested with locators at all. maxDepth, from 1 to 255 (agent-desktop's limit), defaults to 24 and raises it; the partial read error names the option. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(desktop): accept the verified cache copy a run published mid-repair, and keep a launch record when its process lookup fails A run that finds no cache entry publishes without the lock, so it can fill the gap a locked repair makes between its two renames; the repair then failed with ENOTEMPTY although the cache held a verified release. The repair now accepts a usable copy there. A failed process listing was read as an empty one, so the stale-worker sweep deleted the record that names the app a killed worker left running. The listing now reports failure apart from none found: the sweep keeps the owner and launch records (removing only the cached field text beside them), and an attempt's quit keeps its pending launch and retries. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> * fix(desktop): publish the agent-desktop cache without a lock, keeping a verified winner The fast path renamed a fresh unpack into place without the lock that the repair of a damaged entry held. A run that published while a repair had moved the damaged entry aside made the repair's own rename fail with ENOTEMPTY, though the cache now held a checked release (seen on the Node 22.22.3 and Node 26 CI jobs). The lock's stale-owner takeover could also remove a lock another waiter had just taken. Every run publishes a copy of the same checked release, so the lock is gone: a rename that collides with an entry accepts it once it is checked, a damaged entry is moved aside by one rename that only one run wins, and a whole copy moved aside by a run that saw the entry damaged earlier goes back. Two tests hold one run at each of those points while another installs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(desktop): show a whole cache entry without the right stamp is downloaded again The stamp test's entry had no executable or macOS helper, so it was downloaded again even with the stamp check removed. The entry now has both, and a missing stamp and one for another release each force a download. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(desktop): take a launched process as the engine's own only by its launch token When no environment could be read, a launch took the one new copy of the app as its own, and a cleanup after an uncertain launch could quit it. Being new does not show this launch started it: a copy the developer opened at the same moment is new too, and was killed in a scripted probe. A copy is now claimed only when it carries the launch token. agent-desktop's launch, which reports the process it started, still stands for an app launched by bundleId. An appPath to a bundle whose environment macOS hides (an Apple platform app, a build signed with the restrict flag) is refused by init and doctor, which name the bundleId to use instead, so open never starts a copy the engine could not find again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(desktop): keep a launch record until cleanup confirms its process is gone A process lookup that failed read as an empty list: the sweep then deleted a killed worker's launch record, and the next sweep left the app running though the lookup worked again. The record held only the token, so a process the worker had identified, whose environment cannot be read, could not be found by a later run. A dispose whose quit failed kept the whole state directory, with the text agent-desktop keeps of the fields. A failed lookup is now no answer: the pending quit fails and keeps the launch, and the sweep keeps the record. A kill waits for the process to exit before the record goes. Once a process is shown to be the launch's, the record also names its id and start time, so a later run quits it without reading its environment, and never a later process given that id. Dispose and the sweep keep only the owner file and the record, and remove the field text. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(desktop): never bring a background app forward to ask it for a window An app launched in the background that showed no window for 2 s was brought forward with a second open by path, bundle id, or name. That open did not name the process the engine launched, so LaunchServices could bring another copy forward, and it took the keyboard the guide promised to leave alone. The background launch now stays in the background. An app that opens no window there fails the open with a message that it needs desktop({ headed: true }), whose launch brings the app it starts forward. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(desktop): send a key only to the launched process's window, never by app name agent-desktop refuses a key for a named window that is not active, which an app in the background never is. The engine then sent the key by the app's name once a list of the apps showed its copy alone, even a list agent-desktop marked incomplete. The name does not pin the process, so a copy that started between the list and the key could get it. The fallback is gone: a refused key fails as NOT_ACTIONABLE, and nothing is sent. Since a key then reaches an app in the background only by chance, press joins the actions only headed offers, so a locator press without headed fails as unsupported before it moves focus. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(desktop): never retry a press whose focus already moved A locator press focuses the control, reads its window to check the control kept focus, then sends the key. A stale read after the focus was retryable, so the locator ran the whole press again and focused the control a second time, which an app's focus handler sees. A retryable failure after the focus is now ACTION_MAY_HAVE_COMMITTED; one before it stays retryable. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(desktop): show a select that may have reached the app stays possibly committed Removing the delivery check from the select's ELEMENT_NOT_FOUND mapping left every select test green. A select whose failure agent-desktop marks delivered_unverified must stay ACTION_MAY_HAVE_COMMITTED, not "no option". Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
0e9d58ec98 | feat(mcp)!: remove the run_tests tool (#1967) | ||
|
|
4893d1dd75 |
feat: add screen.pause() (#1904)
With `await screen.pause()`, a test is paused at that line, with the app
open, until someone resumes it.
- In a terminal, press Enter to resume the test.
- From another shell in the same directory, run `e2e run --step next`.
A run without a terminal is resumed this way.
- Paused time doesn't count toward the test's timeout.
- With `CI` set, the test fails with `PAUSE_IN_CI`.
- Under `--step`, a test is still paused after `n` or `a`.
- A test isn't paused in an MCP `open_session {test}` session, under
`openHost` or `run()`, or in a core program.
- A test isn't paused under `e2e run --watch` yet.
|
||
|
|
9d240ac954 |
feat(core): report launch() sessions in the test that opens them (#1935)
A session a test opens with launch() from @e2e-dev/core was invisible to the test's report, and a secret passed only to launch() could reach it. - secrets: every owner of values (config, open host, open session) shares them through one refcounted map on globalThis; ledgers redact what is shared beside them without storing it - report: a launch session's steps are mirrored into the test's recorder under an actor name (launch-1), with model usage and video - lifecycle: the attempt owns hosts and sessions the test opened, cuts their steps when the body ends, and closes them at its end - reporters: list, trace page, markdown, junit and html name the session - errors a session throws no longer carry a secret in details |
||
|
|
847a09e6db |
feat(test)!: set fixtures up on demand, once per worker, and in beforeAll (#1891)
* feat(test): set fixtures up once per worker and pass them to beforeAll
test.extend() takes [fn, { scope: 'worker' }]: one instance per worker and
target, torn down when the worker ends. Suite hooks receive worker fixtures.
Engines can declare isolation: 'worker'; @e2e-dev/mobile does, so its device,
app, and screen reach beforeAll, afterAll, and worker fixtures.
Refs ENG-1297
* feat(e2e)!: set up fixtures on demand
A test now sets up only the test.extend() fixtures its body and its
beforeEach and afterEach hooks name in their first parameter, the fixtures
those name, and the auto ones. A suite hook sets up the worker fixtures it
names and the auto worker fixtures. Names are read from the function's
source, as Playwright and Vitest read them.
- [fn, { auto: true }] and [fn, { scope: 'worker', auto: true }] run a
fixture nothing names.
- A fixture can name the fixtures declared beside it in the same extend()
call. Fixtures run in declaration order, each after the ones it names.
- The attempt is resolved as one unit before the session launches: a
fixture only an afterEach names is set up before the first beforeEach.
An afterEach that names a fixture whose setup failed is skipped.
- Fixture rules fail as COLLECTION_ERROR with the file and line of the
code: a parameter that is not destructured or takes a rest property, and
a cycle, when the file is collected; a name the target lacks, a test
fixture named by a suite hook or a worker fixture, and a fixture named
like an engine's, at selection, for the pairs that run on that target.
- On an engine with isolation: 'worker', a suite hook or a worker fixture
teardown that names no engine-backed fixture opens no engine attempt.
BREAKING CHANGE: a fixture nothing names no longer runs unless it is
declared with auto: true, and a test body, hook, or fixture function that
takes the whole fixture object (async (fixtures) => {}) no longer loads.
Refs ENG-1297
* refactor(core): declare core fixtures in one table
The five core fixtures (agent, app, screen, platform, session) are one
table in @e2e-dev/core: name, scope, and factory. createFixtures builds
the built-ins object through one loop over that table and the engine's
factories, every fixture a lazy accessor behind the same gate as before.
The runner's fixture pool, test.extend() validation, and launch read the
names and scopes from the table, so the name lists and the per-name scope
checks are gone. A launch session exposes the core fixtures that do not
belong to a test: app, screen, platform.
No behaviour change. app and screen were built with the attempt and are
now built on first read, like agent and the engine's fixtures.
Refs ENG-1297
* feat(e2e): extend a built-in fixture through test.extend
A test.extend() fixture named like one the test already has extends it,
by Playwright's rule: inside its function its own name is the fixture it
extends (the built-in, or an earlier fixture of that name), and tests,
hooks, and other fixtures that name it receive what it hands to use().
base.extend({ device: async ({ device }, use) => {
await device.setPermission('camera', 'deny');
await use(device);
} })
- A plain function takes the scope the extended fixture has on the
target: device is a worker fixture on mobile, so its override runs once
per worker and reaches beforeAll; app is a test fixture on web and a
worker fixture on mobile. A declared scope that differs is a
COLLECTION_ERROR naming both.
- A worker-scoped override receives what a worker fixture receives for
that built-in, the forwarder to the attempt that is running.
- On a target without the built-in nothing fails until code that runs
there names it; then the target has no such fixture.
- Only user code sees the override: the attempt's own built-ins, which the
runner and the worker fixtures' forwarders read, are untouched.
- Types: the override keeps the extended fixture's type and scope, through
an extend() overload that takes overrides without type arguments.
A fixture named like a core fixture, an engine's, or an earlier one of the
chain was refused before.
Refs ENG-1297
* docs(fixtures): name built-in fixtures and give their scope per engine
The docs now use one term, built-in fixtures, for agent, app, screen,
platform, session, and the ones an engine contributes (browser, device),
where pages said core fixtures, contributed fixtures, or engine fixtures.
The test reference gets one table with each built-in fixture's scope on
web and on mobile, and says what the scope decides: a test fixture reaches
test bodies, beforeEach, and afterEach; a worker fixture also reaches
beforeAll, afterAll, and worker fixtures. The hooks, worker fixtures, and
override sections link to it, as do the fixtures guide and the engine
pages. The Japanese and Chinese pages carry the table and the same terms.
Refs ENG-1297
* docs(ja): translate the fixture scope and on-demand changes
* docs(zh): translate the fixture scope and on-demand changes
* fix(e2e): let a worker fixture name an extended app or screen
A fixture that extends `app` or `screen` was taken for a test fixture when
its test.extend() call was checked, before any target is known. A worker
fixture declared after it that names `app` then failed at import on every
target, though the same fixture is accepted without the override and the
plan resolves both to worker scope on an engine whose state outlives a
test.
The scope of a plain function is now unresolved until a target is known,
whatever it extends, as it already was for an engine's own fixtures.
Refs ENG-1297
* fix(e2e): fail a suite hook that left an engine step running
On an engine whose state outlives a test, a beforeAll or afterAll hook, or
a worker fixture's teardown, can start a device step and not await it. The
hook's attempt then closed under the step, and its failure arrived later
as a stray rejection charged to whichever test ran then.
The suite attempt now does what a test attempt does once its body settled:
the step is failed at its call and cancelled, its outcome is awaited within
the cleanup budget, and the hook fails with STEP_NOT_AWAITED (HOOK_FAILED
for a suite hook).
The suite attempt keeps no step records, and StepRecorder dropped the
record of the step it had just opened before marking it running, so there
was nothing to abandon: a step now counts as running before the retained
window is trimmed.
Refs ENG-1297
* refactor(e2e): keep the fixtures a function names beside it
Registration parsed each test body, hook, and fixture function for the
fixtures it names, dropped the result, and left it in a WeakMap that the
planner and the pool read back with placeholder subjects. A cache miss
there would have named the wrong code and no line.
The parse result is now data: `names` on the registered test, the hook,
and the fixture definition. The planner and the pool read it, the WeakMap
and the per-function wrappers are gone, and one describeUser() gives an
error its subject. The explore test, registered by hand, parses its body
where it builds the record.
Unit tests state the names instead of generating source text for them.
Refs ENG-1297
* refactor(core): resolve built-in fixtures in one place
The core table declared the core fixtures, but what a target's built-in
fixtures are was still assembled around it: core names plus engine names in
three files, the 'engine' scope resolved in e2e through a cast into the
table, `platform` hard-coded as the one fixture that needs no attempt, and
two structural copies of the target.
core now exports builtInFixtures(target): names, scope(name),
needsAttempt(name), attemptless(name), and the unknown-fixture message,
over one BuiltInTarget. `platform` declares in the table that the target
alone decides it. launch, the planner, and the pool read from it.
DECLARED_BUILT_INS is the same view before a target is known: a scope the
engine decides is 'unresolved'. The planner takes either one, so its
no-target mode and the `target === undefined` branches through scopeOf and
read are gone. The scope of a fixture that extends `app` before a target
is known is 'unresolved' by construction.
Refs ENG-1297
* refactor(e2e): hand each function the fixtures it names
Every test body, hook, and fixture function must destructure its first
parameter, so what it reads is known before it runs. bindFixtures still
built a Proxy-gated bag for arbitrary property reads: one value in three
stores, the gate run twice (once over a faked plan), a third derivation of
scope, and an own-name shadow that re-derived what the planner had resolved.
No test code could reach the gate's throwing branches; only unit tests did,
by reading the bag directly.
The plan now records, per function, what each name it reads resolved to: a
chain fixture or a built-in. The pool hands each function a plain object
with exactly those names, the value a chain fixture handed over, or a
built-in read off the attempt when the function destructures it, so
built-ins stay lazy. An override that names itself receives the fixture it
extends because that is what its own name resolved to.
- bound.call(user) replaces user.fn(bound.fixtures); ready(user) asks
whether every chain fixture it names was provided.
- attemptlessBuiltIns is a function from a name to a value that throws the
reason for a built-in that needs an attempt.
- The planner runs once: it records what each fixture names, and the order
is one declaration-order walk over that record. A second validating pass
only reordered.
Behaviour: a bound or native function, whose parameters cannot be read, is
a COLLECTION_ERROR; it used to parse as naming nothing and still got lazy
built-ins. The docs no longer promise hooks and the body one shared object.
Refs ENG-1297
* refactor(e2e): move the suite attempt out of TargetExecutor
dispose, openSuiteHooks, and openSuiteAttempt were 177 lines of a second
attempt lifecycle inside TargetExecutor, written as closures in one method.
run/suite-attempt.ts now owns it: a SuiteAttempt class with open, acquire,
within, and close, over a SuiteAttemptHost interface the executor satisfies
(the way serial.ts borrows it through SerialHost), plus the worker
fixtures' teardown pass. It is the realm's SuiteHookSession itself, so the
executor keeps a context object and two call sites.
The three branches that built an attemptless reader with a "not available
to the <label>" error are one NoAttempt value ('isolated', 'unnamed',
'interrupted', or the launch failure) and one mapper to the message.
execute.ts: 1244 lines to 1052.
Refs ENG-1297
* refactor(e2e): share the attempt's fixture sequence
A run's attempt and an `e2e mcp` session's test each planned the test's
fixtures, bound them, entered the worker pool, set them up, ran the
beforeEach hooks and the body, filtered the afterEach hooks that can still
run, and labelled the teardowns. This PR widened that copy, and
standalone-test.ts asked the realm for the same hooks four times.
planAttemptFixtures(target, test, hooks) plans before any session exists
and returns open(), which binds the plan to the attempt: enter(), run()
for setup, hooks, and body, and cleanup() for the labelled afterEach hooks
and teardowns. Both call sites keep their own phases, budgets, and error
handling around it.
execute.ts: 1052 lines to 1031, its size on main.
Refs ENG-1297
* refactor(e2e): open the suite-hook pass without a lazy thunk
withSuiteHooks handed its work a thunk that opened the pass on first use.
Opening one does no I/O: the suite attempt launches nothing until a hook
that names an engine fixture runs. The pass is now opened up front and
closed in finally, and the `opened` slot and the `hooks()` call form go.
Refs ENG-1297
* refactor(e2e): ask the registry which hooks apply to a test
The selection-time fixture check rebuilt the scope chain of a test and
filtered the file's hooks by it, which the realm already did its own way.
registry.ts now exports scopeChain(test) and hooksFor(registration, test),
and both use them.
Refs ENG-1297
* refactor(core): move the fixture types into types/fixtures.ts
types.ts grew from 1679 to 1798 lines in this PR. The fixture scope, the
worker fixtures, and what test.extend() takes (definitions and overrides)
are one coherent group that nothing else in the file defines against, so
they move to types/fixtures.ts and types.ts re-exports them. The public
surface is the same.
types.ts: 1798 lines to 1715.
Refs ENG-1297
* refactor(core): drop the redundant override branch in FixtureDefinitions
A fixture that extends one the base already has was typed three ways: a
conditional branch inside the `Extra` map of FixtureDefinitions, the
`& FixtureOverrides` intersection, and the extend(overrides) overload.
Compiling the type tests without each in turn: the intersection types an
override written beside new fixtures with type arguments, and the overload
types an override-only call without them; both fail the tests when
removed. The conditional branch does not: with it gone the tests pass, and
a base name in `Extra` is `never` again, as it was before overrides. It is
removed, and the reference's copy of the declaration follows.
Refs ENG-1297
* refactor(e2e): split fixture errors from the parser, share hookLog
fixture-names.ts parses a parameter list, and also held FixtureUseError and
declarationSite, which format an error. They move to fixture-errors.ts.
Its identifier pattern held two raw zero-width characters; they are
written as escapes now.
hookLog and LOG_MODULE were pasted into three integration tests; they are
one helper in tests/helpers/hook-log.ts.
Refs ENG-1297
* docs(e2e): name the scope rule in the selection check's summary
The summary of assertFixtureUse still listed a chain fixture named like an
engine's as an error. Such a fixture extends the built-in now; the rule
left is a declared scope that differs from the extended fixture's.
Refs ENG-1297
* refactor(e2e): take execute.ts under 1000 lines
Two more pieces that are not the attempt's own sequence leave the file.
- Holding steps for `e2e run --step`: reading the screen a hold starts on
and wiring AttemptHolds to the runner's hook move to step-hold.ts, beside
the holds they configure (holdAttemptSteps, readHeldScreen).
- The result of a finished test or setup unit is a pure function of the
pair, the verdict, the attempts, and a beforeAll failure: unitResult in
units.ts, beside pairResult.
execute.ts: 1031 lines to 956. It was 1031 on main and 1226 at the review.
Refs ENG-1297
* test(e2e): match the not-awaited message without a step label
Main's #1880 names an unlabeled step by its api alone ("gadget.hang
finished"), where the message used to quote an empty label. The suite-hook
test's pattern required a label between the two, so it failed on the merge
with main.
* feat(e2e): an override inherits auto from the fixture it extends
A fixture that extends another and does not state `auto` now has the
extended fixture's, the way it has its scope: an override of an auto
fixture is set up for every test, and one of a built-in fixture on demand.
An `auto` that differs from the extended fixture's is a COLLECTION_ERROR
naming both, worded like the scope mismatch. This matches Playwright.
It also fixes which fixtures the plan starts from. An auto fixture was a
root by its own flag, so with an auto base and its override in one chain
the base was set up and the override skipped, while the name resolved to
the override; an override that replaced the base without naming it still
left the base running. An auto fixture is now a root only where its name
stands for it: the base runs once through the override's own name, or not
at all when the override replaces it.
Refs ENG-1297
* fix(core): name the hook or fixture that left a step running
STEP_NOT_AWAITED said "the test body returned before ..." for a step a
beforeAll, afterAll or beforeEach hook, a fixture's setup or a worker
fixture's teardown left running. The recorder now keeps the owner each
step was started under, and the runner says which code runs: the pool
for fixtures, hooks and the body, the suite attempt for suite hooks, and
a hosted session for itself.
---------
Co-authored-by: Oskar Kwaśniewski <oskar@okwasniewski.com>
|
||
|
|
95c574adec |
fix(agent): strip unknown fields on no-input agent tools (#1835)
* fix(agent): strip unknown fields on no-input agent tools A tool with an empty schema shape still uses the strict schema, which keeps the provider-facing input closed, but a plain object is reduced to an empty object before that check, so extra keys are dropped and the call runs. Values that are not plain objects still fail validation. Tools that declare fields still reject unknown keys, and screenshot does too once a path field is present. The security and reference docs and a patch changeset describe that split. Fixes #1812 Signed-off-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com> * fix(agent): trim schemaTool comment and cover array input Cut the schemaTool JSDoc to why an empty schema strips, drop the changeset's provider-schema sentence, and add an array to the values a no-input tool rejects. * docs(security): drop provider schema detail and tighten tool rules --------- Signed-off-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com> Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com> |
||
|
|
40cb356a1e |
fix(web,mobile): load the main entry without e2e installed (#1828)
`@e2e-dev/web` and `@e2e-dev/mobile` declare `e2e` an optional peer, but
their main entry imported it statically to build `test`. A program on
`@e2e-dev/core` alone that wrote `import { web } from '@e2e-dev/web'`
failed with ERR_MODULE_NOT_FOUND before any of its code ran.
The main entry now requires `e2e` only when it resolves. Without it, the
engine factory works and `test`, `describe`, and the hooks throw
RUNNER_NOT_INSTALLED when called. It uses createRequire, not a top-level
await, so a CommonJS file can still require() the entry.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
dbe3dc54a9 |
feat(core): add launch, SessionState, and createAgent to @e2e-dev/core (#1734)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Oskar Kwaśniewski <oskar@okwasniewski.com> |
||
|
|
2359860783 | feat(cli): step through a test without a terminal (#1750) | ||
|
|
5ff74812a8 | docs(skills): give AUTOMATION_UNSUPPORTED an example a device really lacks (#1732) | ||
|
|
17a2e464f0 |
feat(explore): explore what a pull request changed with --changed-since (#1711)
Co-authored-by: Oskar Kwaśniewski <oskar@okwasniewski.com> |
||
|
|
11dea90ddc |
fix(core): fail runs when app commands exit after readiness (#1541)
* test(core): reproduce app commands exiting after readiness * fix(core): fail runs when app commands exit after readiness * test(core): tighten app exit outcomes and diagnostic docs * fix(core): scope app command exits to dependent targets Retain all owners of deduplicated commands and route ready app exits through the scheduler. Ignore owners with no unfinished work, preserve healthy targets, and classify forced app cancellation before watchdog timeout synthesis. Cover shared ownership, completed targets, teardown, readiness and interrupt ordering, navigation advice, and both watchdog/app-grace orderings. * fix(core): clarify app readiness and output guidance * fix(core): share app command interruption and cover buffered exits Use the same per-worker interruption path for app exits and run interrupts. Iterate snapshots so a synchronous transport exit cannot skip another worker. Cover exits buffered between readiness and scheduling, serial-group exit status, and returned-unit skip reasons. Keep the aggregate process hook separate from the individual managed-process hook. * refactor(core): return app-exit target set directly The scheduler result contains only the target names used to distinguish app cancellation from a user interrupt. Return that set directly instead of wrapping it in a named single-field result and a caller temporary. Keep the same completion and drain order. * test(core): cover app-exit guards and process liveness * test(core): expect app exits to quote short logs whole Main's #1577 quotes a log of up to 40 lines whole and keeps the first lines of a longer one, so an app exit after readiness now quotes the test server's first line too. The test asserts that instead of the old tail-only quote, and docs/web.mdx says unexpected exits quote the log. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(changeset): note that background launchers now fail Apply suggestion from @kdy1 Co-authored-by: Donny/강동윤 <kdy.1997.dev@gmail.com> * fix(core): keep reused app hints from promising command output A refused navigation on a target whose app.command sets reuseExisting now says the runner may use a process it did not start, whose output app.command.log never holds. The docs scope the exit quote to e2e run, list spawn and log-open failures under APP_UNREACHABLE, and name the first declaring target in the shared-command error label. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Donny/강동윤 <kdy.1997.dev@gmail.com> |
||
|
|
3c9de80d9e |
feat(mcp): open a session without a config by naming the engine (#1665)
* feat(mcp): open a session without a config by naming the engine
open_session takes engine, a package name or the factory call a config
would write as {package, export, options}, and app. An option can be
another such call, which is how an engine gets a device or browser
provider. Packages are found in the project, beside e2e, or in
~/.e2e/engines; ENGINE_NOT_INSTALLED names the install command.
@e2e-dev/web and @e2e-dev/mobile default-export their factory.
* fix(mcp): load import-only packages and refuse builtins as engines
Review fixes: resolve a package by its node_modules directory, so an
installed package without a loadable entry is reported as such; fall
back to the import condition; refuse Node builtin names; never read
the options object itself as a factory call; mobile() names the
platforms it takes when called without one.
* docs(mcp): say "no e2e config file", split the config error rows, pin the __proto__ key
Review fixes: a concrete app.url in the skill example, the string
shorthand in AGENTS.md, the engine-only wording in the security note,
and a test assertion that the __proto__ key survives as an own property.
|
||
|
|
5847701fb5 | feat(e2e): run the CLI on Bun 1.4.3 and newer, without Node.js (#1616) | ||
|
|
93c93f5e25 | feat(cli): e2e run --step holds each step until a key (#1540) | ||
|
|
9289ff4ae7 |
fix(run): quote a long app log's first lines in APP_UNREACHABLE (#1577)
A process that fails early and then keeps logging - a file watcher, say - buried the startup error under its own tail: the message quoted only the last 20 lines appended since the command started. It now quotes the first and last 20 lines with `[startup output omitted]` between them when the log is longer, both windows still byte-bounded and passed through the same redactor, so the error that explains the failure survives. Fixes #1527 |
||
|
|
34ccc5f388 |
perf: decode and encode PNGs with @napi-rs/image (#1587)
Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com> |
||
|
|
f185ca8ba2 |
docs: retire sidebar page icons (#1582)
* docs: retire sidebar page icons The docs sidebar no longer shows icons. Drop icon: from every page's frontmatter, delete the grey silhouettes in docs/images/icons/, and make check-docs-logos fail on any frontmatter icon instead of requiring one on vendor pages. Card logos stay required. The authoring-docs skill trades its Page icons section for a short Vendor logos one. * docs: require a migrate overview card for each migration guide * docs: match only top-level icon keys in check-docs-logos |
||
|
|
9267248c4c | fix(web): browser.evaluate survives clobbered page globals (#1553) | ||
|
|
d2556bc23f | feat(e2e)!: inputValue and isChecked throw on the wrong element, uncheck clicks ARIA radios (#1552) | ||
|
|
d2953bb9bb | feat(e2e): timeout 0 means no step deadline; settle by-design Playwright gaps (#1554) | ||
|
|
0c0469605a | chore: replace e2e.tester.army with e2e.com (#1572) | ||
|
|
732ea8af9a |
feat(e2e): test.fail for tests expected to fail (#1539)
* feat(e2e): test.fail for tests expected to fail Register, runtime, and option forms. A product failure passes the test and keeps the error under expectedFailure; a pass fails with UNEXPECTED_PASS and is not retried. Infrastructure errors, interrupts, and blocked agent verdicts are not covered; a skip wins. * fix(e2e): test.fail review fixes Suite hook and API misuse errors not covered, reason redacted and bounded, verdict settles before recordings, beforeEach may mark, afterEach refused, reporters show the covered error, schema forbids marks on skipped, setup, and serial results. * fix(e2e): retry an unexpected pass of test.fail like Playwright * docs(test): shorten test.fail |
||
|
|
50fa7afc1a | feat(mcp): run_tests tool runs e2e run from the MCP server (#1520) | ||
|
|
79e2e13d33 | feat(locator): show how many candidates each filter step left (#1416) | ||
|
|
91bd61490d | feat(cli): e2e doctor preflight with engine doctor hook (#1493) | ||
|
|
94e6e76548 | feat(agent)!: keep model screenshots after a secret fill (#1437) | ||
|
|
d2ffb319cf |
feat(e2e): record one span of the screen with app.startRecording and app.stopRecording (#1381)
* feat(e2e): record one span of the screen with app.startRecording and app.stopRecording * docs(e2e): name the video modes app.startRecording refuses under |
||
|
|
34287cfe78 |
fix(run): exit 3 when a passed test's cleanup failed (#1233)
Signed-off-by: Varshith Puli <pulivarshit@gmail.com> |
||
|
|
3d2c3627ec | feat(mcp): run a test in a session and stop on the failure screen (#1315) | ||
|
|
6b41857530 | feat(mcp): run test code in a live session with run_code (#1316) | ||
|
|
da13624d31 | feat(runner): run only the tests a pull request can affect (#1206) | ||
|
|
9ceb0710c3 | feat(mcp): show the test line for each action in a session (#1317) | ||
|
|
93f77d81e2 |
fix(agent): classify a content-filter refusal as MODEL_REFUSED (#973)
Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: Oskar Kwaśniewski <oskar@okwasniewski.com> |
||
|
|
1071933813 | feat(config): discover e2e.config.js and document a JavaScript setup (#1280) | ||
|
|
c304924e53 |
docs(reference): rewrite the API reference in the Next.js shape (#1197)
* docs(reference): rewrite the API reference in the Next.js shape Every reference page now opens with a lede and a minimal example, groups its API under ## sections that start with a summary table linking to one ### entry per member, and keeps the TypeScript declarations under ## Types. * docs(reference): fix claims the review caught Narrow the security summary rows to their sections, generate the screenshot file name in the engine example instead of building it from the label, and correct defaults, exit codes, and examples on config, cli, screen, web, mobile, test, runner, and engine. * docs(reference): address review findings Scope error classes to runner errors, guard a zero-slot prepare in the engine example, keep the cli #help-and-version anchor, and correct poll cancellation, same-run screenshots, openLink schemes, type_secret, longPress durations, runner signals and close failures, empty usernames, and the feedback exit code. Make the reporter and extract examples self-contained. * docs(reference): trim the summary tables to the minimum Each group's summary table keeps the linked name and one or two short columns (type, default, or a few-word description). Examples and details live in the entries; facts that only a table cell held moved into its entry. The expect overview is headed Reference instead of Entry points, and the security rules table names what each section covers instead of restating its claims. * docs(reference): open every page's API under one Reference heading Each page's first section after the lede is now ## Reference. A page with one group lists its entries at ### right under it; a page with several keeps each group as a ### with its summary table and its entries at ####. Explanatory sections (behavior, budgets, setup, types) follow it. Linked sub-parts that moved to ##### keep their anchors through explicit ids. * docs(skills): describe the reference page shape in authoring-docs Reference pages open their API under ## Reference, keep summary tables to the name and one or two short columns, and nest entries at ### or #### because Mintlify drops anchors below ####. ship-pr points reference edits at these rules instead of the guide shortening pass. * docs(reference): address cubic review on tables and skill Keep readonly in the mobile device type, show the keyboard and mouse object shapes, say a locator assertion times out as ASSERTION_FAILED, and give reference pages a narrower shortening pass in authoring-docs. * docs(reference): tighten the reference pages Merge facts said twice into one place, move each declaration's JSDoc to a single home, link timeout defaults, key grammar, and secure-field rules to the pages that own them, and split long sentences. No fact is cut. * docs(reference): address cubic review of the tightening pass Qualify the GitHub token for fork pull requests, provider recording URLs, duplicate device entries, MODEL_UNAVAILABLE timing, the APP_UNREACHABLE log quote, UNKNOWN_TOOL versus UNSUPPORTED_CAPABILITY, node-less agent tools, which model receives screenshots, passing assert evidence, and config-path hosts; fix three garbled sentences. * docs(reference): say device, not simulator, for worker isolation |
||
|
|
bf3c8e8ac1 | fix(agent): stop the run once when the provider does not serve the model (#1153) | ||
|
|
236611fafb |
docs: fix packaged docs against source, ship snippets, check anchors (#1168)
* docs: fix packaged docs against source, ship snippets, check anchors * docs: tighten review wording, harden anchor check fences and ids * docs: address review on tool errors, CLI failure output, anchor check * docs(security): base64 is decoded in text downloads * docs(telemetry): shorten the telemetry page (#1174) * docs(telemetry): shorten the telemetry page * docs(telemetry): engine versions are semver or unversioned * docs(migrate): shorten the Playwright migration guide (#1172) * docs(debugging): shorten the debugging guide (#1170) * docs(auth): merge repeated secret facts on the signing-in page (#1169) |
||
|
|
2bbeb32d4c |
feat(runner): execute suites through e2e/runner (#1152)
* feat(runner): execute suites through e2e/runner * fix(runner): handle review edge cases and deterministic verification * fix(runner): isolate suite output and narrow public execution options * docs(runner): note RUNNER_BUSY on openHost and cached helper modules * chore(changeset): patch bump while e2e is 0.x * feat(runner): send e2e_run_completed telemetry from run() * docs(telemetry): scope run() event to completed runs, mention notice --------- Co-authored-by: Oskar Kwaśniewski <oskar@okwasniewski.com> |
||
|
|
9351d12387 |
fix(cli): reject --max-duration over the timer cap as a flag error, name the limit on interrupted tests (#1148)
* fix(cli): reject --max-duration over the timer cap as a flag error, name the limit on interrupted tests * docs(errors): note INTERRUPTED from --max-failures and the time limit |
||
|
|
babae32bda |
feat(cli): add e2e run --max-duration and maxDuration config (#1131)
* feat(cli): add e2e run --max-duration and maxDuration config * fix(cli): keep e2e explore on its own --timeout, not maxDuration * fix(cli): skip max-duration timer past setTimeout cap, cover config and explore paths * refactor(cli): address max-duration review - explore drops maxDuration from its config; runner no longer special-cases it - scheduler records RUN_TIMEOUT from the skip reason, naming --max-duration or maxDuration - stopEarly wakes the loop itself - reject maxDuration above 2147483647 ms instead of skipping the timer - trim cli docs, clarify ci wording, sync plugin |
||
|
|
3a53fbd8e5 |
feat(runner): host e2e sessions in another test runner (#980)
* feat(runner): open an e2e session from another test runner Adds openSession() to e2e/runner: the standalone attempt e2e mcp already opens, with its fixtures, live step progress (onStep), the step records (steps()), and cleanup handed to the calling runner. config takes a path or an E2EConfig value, as list() does; a config file is loaded fresh with its imports, so each session gets engine instances of its own. close() aborts the session's operation budget before teardown, so a fixture call still running is cancelled; e2e mcp gets the same fix. Disposing a session throws its cleanup failures as an AggregateError. A session reserves a free port only for the target it opens. * feat(runner): split hosted worker and test lifetimes * fix(runner): release host engine claims after clean teardown * fix(runner): enforce the hosted session deadline and keep the host after a failed launch A hosted session's timeout now cuts every running step when it passes, the way a run's body race does: an engine that ignores its operation timeout or a model call no longer outlives it, the operation fails with TEST_TIMEOUT, and close() reports the timeout. A timeout of 0 or less is refused, as the config's is; the Playwright example maps Playwright's 0 to the config default. A launch that fails or is cancelled leaves the host usable, as a run keeps its worker for the retry; only failed engine cleanup retires it. Split standalone.ts: the worker stays, standalone-attempt.ts owns one attempt's reservation, launch, fixtures, deadline, and close, host-trace.ts its trace.md. The attempt's kind is explicit instead of inferred from a name. Port allocation for one target is a helper shared with MCP. * fix(report): hosted trace pages stop pointing at a report.json that does not exist * refactor(runner): simplify the hosted worker after review The worker is an explicit state machine (idle, opening, live, retired) instead of four promises and a flag. A live attempt holds its registry through holdSecretRegistry, outranking a host opened meanwhile, so secrets no longer import from the expect layer and the published attempt carries no registry. Target selection is one helper shared with MCP; port allocation for one target is direct. The attempt's verdict uses the run's classifyAttemptStatus. The one-shot video branch was dead: MCP records off. The fixture-recording deadline cap is gone: the session timer and the step signal already cut every operation, and e2e run's body race covers runs. withAbort's third argument is the fallback the signal's own error outranks. The open-host suite is five files by concern on one shared harness. * chore: refresh the runner docs card and drop an unused test helper export * test(runner): pin the session verdict, not which deadline reporter won A poll bounded by the session deadline and the session's own cut fire at the same moment; on a slow host either can report first. Both are true. --------- Co-authored-by: bheemreddy-samsara <bheem.reddy@samsara.com> Co-authored-by: Oskar Kwaśniewski <oskar@okwasniewski.com> |
||
|
|
5d4053e8fa | docs: render an Open Graph card for every docs page with @vercel/og (#1062) | ||
|
|
521988496e |
docs(security): split the security page into a short guide and a rules reference (#1008)
The guide answers what most readers ask: how secrets stay out of the model, what can still leak, and what runs with your permissions. Every rule moves to docs/reference/security.mdx. smol host-loopback access moves to the smol page's warning. Fixes facts that no longer matched src/: defineTool errors are INVALID_CONFIG, the CI cache default skips a cache.store, start-screen replay matches routes, and outbound connections now list login, models, init, and the smol image pull. |
||
|
|
b92658be6e | feat(decision): full action grammar, none fallback, and vision through the OpenAI Decisions API (#965) | ||
|
|
c10dc00992 | fix(agent): name the status and the body of a provider error the SDK could not parse (#922) | ||
|
|
99d5ab5336 | fix(cache): --strict-cache replays on retries and never writes the cache (#923) | ||
|
|
1b92a6ddfa | feat: e2e trace pages replace the Playwright trace (#869) | ||
|
|
9fb70b9e99 | fix(config): reject a non-string credential username with INVALID_CONFIG (#918) |