Commit Graph
103 Commits
Author SHA1 Message Date
Donny/강동윤andClaude Opus 5.5 40cb356a1e fix(web,mobile): load the main entry without e2e installed (#1828)
`@e2e-dev/web` and `@e2e-dev/mobile` declare `e2e` an optional peer, but
their main entry imported it statically to build `test`. A program on
`@e2e-dev/core` alone that wrote `import { web } from '@e2e-dev/web'`
failed with ERR_MODULE_NOT_FOUND before any of its code ran.

The main entry now requires `e2e` only when it resolves. Without it, the
engine factory works and `test`, `describe`, and the hooks throw
RUNNER_NOT_INSTALLED when called. It uses createRequire, not a top-level
await, so a CommonJS file can still require() the entry.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 09:50:17 +00:00
dbe3dc54a9 feat(core): add launch, SessionState, and createAgent to @e2e-dev/core (#1734)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Oskar Kwaśniewski <oskar@okwasniewski.com>
2026-11-16 09:38:07 +01:00
Krzysztof Moch 2359860783 feat(cli): step through a test without a terminal (#1750) 2026-11-15 17:47:21 +01:00
Karol Kulesza 5ff74812a8 docs(skills): give AUTOMATION_UNSUPPORTED an example a device really lacks (#1732) 2026-11-13 08:49:29 +00:00
Krzysztof MochandOskar Kwaśniewski 17a2e464f0 feat(explore): explore what a pull request changed with --changed-since (#1711)
Co-authored-by: Oskar Kwaśniewski <oskar@okwasniewski.com>
2026-11-12 17:42:01 +00:00
11dea90ddc fix(core): fail runs when app commands exit after readiness (#1541)
* test(core): reproduce app commands exiting after readiness

* fix(core): fail runs when app commands exit after readiness

* test(core): tighten app exit outcomes and diagnostic docs

* fix(core): scope app command exits to dependent targets

Retain all owners of deduplicated commands and route ready app exits through the scheduler. Ignore owners with no unfinished work, preserve healthy targets, and classify forced app cancellation before watchdog timeout synthesis.

Cover shared ownership, completed targets, teardown, readiness and interrupt ordering, navigation advice, and both watchdog/app-grace orderings.

* fix(core): clarify app readiness and output guidance

* fix(core): share app command interruption and cover buffered exits

Use the same per-worker interruption path for app exits and run interrupts.
Iterate snapshots so a synchronous transport exit cannot skip another worker.

Cover exits buffered between readiness and scheduling, serial-group exit
status, and returned-unit skip reasons. Keep the aggregate process hook
separate from the individual managed-process hook.

* refactor(core): return app-exit target set directly

The scheduler result contains only the target names used to distinguish
app cancellation from a user interrupt. Return that set directly instead
of wrapping it in a named single-field result and a caller temporary.

Keep the same completion and drain order.

* test(core): cover app-exit guards and process liveness

* test(core): expect app exits to quote short logs whole

Main's #1577 quotes a log of up to 40 lines whole and keeps the first
lines of a longer one, so an app exit after readiness now quotes the
test server's first line too. The test asserts that instead of the old
tail-only quote, and docs/web.mdx says unexpected exits quote the log.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(changeset): note that background launchers now fail

Apply suggestion from @kdy1

Co-authored-by: Donny/강동윤 <kdy.1997.dev@gmail.com>

* fix(core): keep reused app hints from promising command output

A refused navigation on a target whose app.command sets reuseExisting
now says the runner may use a process it did not start, whose output
app.command.log never holds. The docs scope the exit quote to e2e run,
list spawn and log-open failures under APP_UNREACHABLE, and name the
first declaring target in the shared-command error label.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Donny/강동윤 <kdy.1997.dev@gmail.com>
2026-11-11 10:31:26 +00:00
Krzysztof Moch 3c9de80d9e feat(mcp): open a session without a config by naming the engine (#1665)
* feat(mcp): open a session without a config by naming the engine

open_session takes engine, a package name or the factory call a config
would write as {package, export, options}, and app. An option can be
another such call, which is how an engine gets a device or browser
provider. Packages are found in the project, beside e2e, or in
~/.e2e/engines; ENGINE_NOT_INSTALLED names the install command.
@e2e-dev/web and @e2e-dev/mobile default-export their factory.

* fix(mcp): load import-only packages and refuse builtins as engines

Review fixes: resolve a package by its node_modules directory, so an
installed package without a loadable entry is reported as such; fall
back to the import condition; refuse Node builtin names; never read
the options object itself as a factory call; mobile() names the
platforms it takes when called without one.

* docs(mcp): say "no e2e config file", split the config error rows, pin the __proto__ key

Review fixes: a concrete app.url in the skill example, the string
shorthand in AGENTS.md, the engine-only wording in the security note,
and a test assertion that the __proto__ key survives as an own property.
2026-11-10 11:41:37 +00:00
Oskar Kwaśniewski 5847701fb5 feat(e2e): run the CLI on Bun 1.4.3 and newer, without Node.js (#1616) 2026-11-09 18:25:18 +01:00
Krzysztof Moch 93c93f5e25 feat(cli): e2e run --step holds each step until a key (#1540) 2026-11-09 18:06:53 +01:00
Dery Ferdika 9289ff4ae7 fix(run): quote a long app log's first lines in APP_UNREACHABLE (#1577)
A process that fails early and then keeps logging - a file watcher, say -
buried the startup error under its own tail: the message quoted only the last
20 lines appended since the command started. It now quotes the first and last
20 lines with `[startup output omitted]` between them when the log is longer,
both windows still byte-bounded and passed through the same redactor, so the
error that explains the failure survives.

Fixes #1527
2026-11-09 13:44:55 +00:00
Donny/강동윤andClaude Sonnet 5.5 34ccc5f388 perf: decode and encode PNGs with @napi-rs/image (#1587)
Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-11-06 10:20:15 +01:00
Kacper Kapuściak f185ca8ba2 docs: retire sidebar page icons (#1582)
* docs: retire sidebar page icons

The docs sidebar no longer shows icons. Drop icon: from every page's frontmatter, delete the grey silhouettes in docs/images/icons/, and make check-docs-logos fail on any frontmatter icon instead of requiring one on vendor pages. Card logos stay required. The authoring-docs skill trades its Page icons section for a short Vendor logos one.

* docs: require a migrate overview card for each migration guide

* docs: match only top-level icon keys in check-docs-logos
2026-11-05 18:23:59 +00:00
Oskar Kwaśniewski 9267248c4c fix(web): browser.evaluate survives clobbered page globals (#1553) 2026-11-05 11:23:38 +01:00
Oskar Kwaśniewski d2556bc23f feat(e2e)!: inputValue and isChecked throw on the wrong element, uncheck clicks ARIA radios (#1552) 2026-11-05 09:47:58 +01:00
Oskar Kwaśniewski d2953bb9bb feat(e2e): timeout 0 means no step deadline; settle by-design Playwright gaps (#1554) 2026-11-05 09:40:25 +01:00
Oskar Kwaśniewski 0c0469605a chore: replace e2e.tester.army with e2e.com (#1572) 2026-11-05 09:23:45 +01:00
Oskar Kwaśniewski 732ea8af9a feat(e2e): test.fail for tests expected to fail (#1539)
* feat(e2e): test.fail for tests expected to fail

Register, runtime, and option forms. A product failure passes the test and
keeps the error under expectedFailure; a pass fails with UNEXPECTED_PASS and
is not retried. Infrastructure errors, interrupts, and blocked agent verdicts
are not covered; a skip wins.

* fix(e2e): test.fail review fixes

Suite hook and API misuse errors not covered, reason redacted and bounded,
verdict settles before recordings, beforeEach may mark, afterEach refused,
reporters show the covered error, schema forbids marks on skipped, setup,
and serial results.

* fix(e2e): retry an unexpected pass of test.fail like Playwright

* docs(test): shorten test.fail
2026-11-04 15:08:28 +00:00
Oskar Kwaśniewski 50fa7afc1a feat(mcp): run_tests tool runs e2e run from the MCP server (#1520) 2026-11-03 18:23:58 +01:00
Dery Ferdika 79e2e13d33 feat(locator): show how many candidates each filter step left (#1416) 2026-11-03 14:22:09 +01:00
Oskar Kwaśniewski 91bd61490d feat(cli): e2e doctor preflight with engine doctor hook (#1493) 2026-11-03 10:27:52 +01:00
Oskar Kwaśniewski 94e6e76548 feat(agent)!: keep model screenshots after a secret fill (#1437) 2026-10-28 19:11:24 -07:00
Oskar Kwaśniewski d2ffb319cf feat(e2e): record one span of the screen with app.startRecording and app.stopRecording (#1381)
* feat(e2e): record one span of the screen with app.startRecording and app.stopRecording

* docs(e2e): name the video modes app.startRecording refuses under
2026-10-28 03:06:09 +00:00
Varshith 34287cfe78 fix(run): exit 3 when a passed test's cleanup failed (#1233)
Signed-off-by: Varshith Puli <pulivarshit@gmail.com>
2026-10-27 19:48:43 -07:00
Kacper Kapuściak 3d2c3627ec feat(mcp): run a test in a session and stop on the failure screen (#1315) 2026-10-27 19:12:01 -07:00
Kacper Kapuściak 6b41857530 feat(mcp): run test code in a live session with run_code (#1316) 2026-10-27 16:29:24 -07:00
Oskar Kwaśniewski da13624d31 feat(runner): run only the tests a pull request can affect (#1206) 2026-10-23 12:29:15 -05:00
Kacper Kapuściak 9ceb0710c3 feat(mcp): show the test line for each action in a session (#1317) 2026-10-21 13:45:05 -04:00
93f77d81e2 fix(agent): classify a content-filter refusal as MODEL_REFUSED (#973)
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Oskar Kwaśniewski <oskar@okwasniewski.com>
2026-10-20 12:09:49 -04:00
Kacper Kapuściak 1071933813 feat(config): discover e2e.config.js and document a JavaScript setup (#1280) 2026-10-20 08:53:36 -04:00
Kacper Kapuściak c304924e53 docs(reference): rewrite the API reference in the Next.js shape (#1197)
* docs(reference): rewrite the API reference in the Next.js shape

Every reference page now opens with a lede and a minimal example, groups its
API under ## sections that start with a summary table linking to one ###
entry per member, and keeps the TypeScript declarations under ## Types.

* docs(reference): fix claims the review caught

Narrow the security summary rows to their sections, generate the screenshot
file name in the engine example instead of building it from the label, and
correct defaults, exit codes, and examples on config, cli, screen, web,
mobile, test, runner, and engine.

* docs(reference): address review findings

Scope error classes to runner errors, guard a zero-slot prepare in the engine
example, keep the cli #help-and-version anchor, and correct poll cancellation,
same-run screenshots, openLink schemes, type_secret, longPress durations,
runner signals and close failures, empty usernames, and the feedback exit
code. Make the reporter and extract examples self-contained.

* docs(reference): trim the summary tables to the minimum

Each group's summary table keeps the linked name and one or two short
columns (type, default, or a few-word description). Examples and details
live in the entries; facts that only a table cell held moved into its entry.
The expect overview is headed Reference instead of Entry points, and the
security rules table names what each section covers instead of restating
its claims.

* docs(reference): open every page's API under one Reference heading

Each page's first section after the lede is now ## Reference. A page with
one group lists its entries at ### right under it; a page with several keeps
each group as a ### with its summary table and its entries at ####.
Explanatory sections (behavior, budgets, setup, types) follow it. Linked
sub-parts that moved to ##### keep their anchors through explicit ids.

* docs(skills): describe the reference page shape in authoring-docs

Reference pages open their API under ## Reference, keep summary tables to
the name and one or two short columns, and nest entries at ### or ####
because Mintlify drops anchors below ####. ship-pr points reference edits
at these rules instead of the guide shortening pass.

* docs(reference): address cubic review on tables and skill

Keep readonly in the mobile device type, show the keyboard and mouse
object shapes, say a locator assertion times out as ASSERTION_FAILED, and
give reference pages a narrower shortening pass in authoring-docs.

* docs(reference): tighten the reference pages

Merge facts said twice into one place, move each declaration's JSDoc to a
single home, link timeout defaults, key grammar, and secure-field rules to
the pages that own them, and split long sentences. No fact is cut.

* docs(reference): address cubic review of the tightening pass

Qualify the GitHub token for fork pull requests, provider recording URLs,
duplicate device entries, MODEL_UNAVAILABLE timing, the APP_UNREACHABLE log
quote, UNKNOWN_TOOL versus UNSUPPORTED_CAPABILITY, node-less agent tools,
which model receives screenshots, passing assert evidence, and config-path
hosts; fix three garbled sentences.

* docs(reference): say device, not simulator, for worker isolation
2026-10-16 14:41:30 +00:00
Piotr Pasztor bf3c8e8ac1 fix(agent): stop the run once when the provider does not serve the model (#1153) 2026-10-16 14:44:25 +02:00
Oskar Kwaśniewski 236611fafb docs: fix packaged docs against source, ship snippets, check anchors (#1168)
* docs: fix packaged docs against source, ship snippets, check anchors

* docs: tighten review wording, harden anchor check fences and ids

* docs: address review on tool errors, CLI failure output, anchor check

* docs(security): base64 is decoded in text downloads

* docs(telemetry): shorten the telemetry page (#1174)

* docs(telemetry): shorten the telemetry page

* docs(telemetry): engine versions are semver or unversioned

* docs(migrate): shorten the Playwright migration guide (#1172)

* docs(debugging): shorten the debugging guide (#1170)

* docs(auth): merge repeated secret facts on the signing-in page (#1169)
2026-10-15 17:56:13 +02:00
axon_dendriteandOskar Kwaśniewski 2bbeb32d4c feat(runner): execute suites through e2e/runner (#1152)
* feat(runner): execute suites through e2e/runner

* fix(runner): handle review edge cases and deterministic verification

* fix(runner): isolate suite output and narrow public execution options

* docs(runner): note RUNNER_BUSY on openHost and cached helper modules

* chore(changeset): patch bump while e2e is 0.x

* feat(runner): send e2e_run_completed telemetry from run()

* docs(telemetry): scope run() event to completed runs, mention notice

---------

Co-authored-by: Oskar Kwaśniewski <oskar@okwasniewski.com>
2026-10-15 13:41:36 +02:00
Oskar Kwaśniewski 9351d12387 fix(cli): reject --max-duration over the timer cap as a flag error, name the limit on interrupted tests (#1148)
* fix(cli): reject --max-duration over the timer cap as a flag error, name the limit on interrupted tests

* docs(errors): note INTERRUPTED from --max-failures and the time limit
2026-10-15 10:12:00 +02:00
Oskar Kwaśniewski babae32bda feat(cli): add e2e run --max-duration and maxDuration config (#1131)
* feat(cli): add e2e run --max-duration and maxDuration config

* fix(cli): keep e2e explore on its own --timeout, not maxDuration

* fix(cli): skip max-duration timer past setTimeout cap, cover config and explore paths

* refactor(cli): address max-duration review

- explore drops maxDuration from its config; runner no longer special-cases it
- scheduler records RUN_TIMEOUT from the skip reason, naming --max-duration or maxDuration
- stopEarly wakes the loop itself
- reject maxDuration above 2147483647 ms instead of skipping the timer
- trim cli docs, clarify ci wording, sync plugin
2026-10-14 19:09:31 +02:00
3a53fbd8e5 feat(runner): host e2e sessions in another test runner (#980)
* feat(runner): open an e2e session from another test runner

Adds openSession() to e2e/runner: the standalone attempt e2e mcp already opens,
with its fixtures, live step progress (onStep), the step records (steps()), and
cleanup handed to the calling runner. config takes a path or an E2EConfig value,
as list() does; a config file is loaded fresh with its imports, so each session
gets engine instances of its own.

close() aborts the session's operation budget before teardown, so a fixture call
still running is cancelled; e2e mcp gets the same fix. Disposing a session
throws its cleanup failures as an AggregateError. A session reserves a free port
only for the target it opens.

* feat(runner): split hosted worker and test lifetimes

* fix(runner): release host engine claims after clean teardown

* fix(runner): enforce the hosted session deadline and keep the host after a failed launch

A hosted session's timeout now cuts every running step when it passes, the
way a run's body race does: an engine that ignores its operation timeout
or a model call no longer outlives it, the operation fails with
TEST_TIMEOUT, and close() reports the timeout. A timeout of 0 or less is
refused, as the config's is; the Playwright example maps Playwright's 0
to the config default.

A launch that fails or is cancelled leaves the host usable, as a run keeps
its worker for the retry; only failed engine cleanup retires it.

Split standalone.ts: the worker stays, standalone-attempt.ts owns one
attempt's reservation, launch, fixtures, deadline, and close, host-trace.ts
its trace.md. The attempt's kind is explicit instead of inferred from a
name. Port allocation for one target is a helper shared with MCP.

* fix(report): hosted trace pages stop pointing at a report.json that does not exist

* refactor(runner): simplify the hosted worker after review

The worker is an explicit state machine (idle, opening, live, retired)
instead of four promises and a flag. A live attempt holds its registry
through holdSecretRegistry, outranking a host opened meanwhile, so secrets
no longer import from the expect layer and the published attempt carries no
registry. Target selection is one helper shared with MCP; port allocation
for one target is direct. The attempt's verdict uses the run's
classifyAttemptStatus. The one-shot video branch was dead: MCP records off.
The fixture-recording deadline cap is gone: the session timer and the step
signal already cut every operation, and e2e run's body race covers runs.
withAbort's third argument is the fallback the signal's own error outranks.

The open-host suite is five files by concern on one shared harness.

* chore: refresh the runner docs card and drop an unused test helper export

* test(runner): pin the session verdict, not which deadline reporter won

A poll bounded by the session deadline and the session's own cut fire at
the same moment; on a slow host either can report first. Both are true.

---------

Co-authored-by: bheemreddy-samsara <bheem.reddy@samsara.com>
Co-authored-by: Oskar Kwaśniewski <oskar@okwasniewski.com>
2026-10-13 09:54:52 +02:00
Kacper Kapuściak 5d4053e8fa docs: render an Open Graph card for every docs page with @vercel/og (#1062) 2026-10-12 15:10:32 +02:00
Kacper Kapuściak 521988496e docs(security): split the security page into a short guide and a rules reference (#1008)
The guide answers what most readers ask: how secrets stay out of the model,
what can still leak, and what runs with your permissions. Every rule moves to
docs/reference/security.mdx. smol host-loopback access moves to the smol
page's warning. Fixes facts that no longer matched src/: defineTool errors are
INVALID_CONFIG, the CI cache default skips a cache.store, start-screen replay
matches routes, and outbound connections now list login, models, init, and the
smol image pull.
2026-10-09 13:13:28 +00:00
Oskar Kwaśniewski b92658be6e feat(decision): full action grammar, none fallback, and vision through the OpenAI Decisions API (#965) 2026-10-09 14:27:37 +02:00
Oskar Kwaśniewski c10dc00992 fix(agent): name the status and the body of a provider error the SDK could not parse (#922) 2026-10-08 08:43:50 +02:00
Oskar Kwaśniewski 99d5ab5336 fix(cache): --strict-cache replays on retries and never writes the cache (#923) 2026-10-08 08:43:25 +02:00
Oskar Kwaśniewski 1b92a6ddfa feat: e2e trace pages replace the Playwright trace (#869) 2026-10-07 11:28:27 +02:00
Oskar Kwaśniewski 9fb70b9e99 fix(config): reject a non-string credential username with INVALID_CONFIG (#918) 2026-10-06 15:35:05 +02:00
Oskar Kwaśniewski 13da94b759 fix(decision): decide through experimental_decide and reject unsupported models at config load (#916) 2026-10-06 14:16:05 +02:00
Kacper Kapuściak 63482bd6fe docs: reorganize the sidebar into platforms and a deep dive (#870) 2026-10-05 20:39:06 +02:00
Oskar Kwaśniewski 37806d7fc3 fix(run): kill a worker that blocks its event loop past the test timeout (#824) 2026-10-05 17:06:39 +02:00
Oskar Kwaśniewski 338378ed48 fix(run): a worker crash during a retry keeps the attempts before it (#837) 2026-10-05 16:09:43 +02:00
Oskar Kwaśniewski 6f528be1a8 fix(agent): fail tainted fallback-only steps with POLICY_DENIED (#827) 2026-10-05 16:05:55 +02:00
Oskar Kwaśniewski 87e75ae808 fix(run): serial group failures reach the exit code and the reporters (#813)
* fix(run): serial group failures reach the exit code and the reporters

- exit code folds serial groups: members carry no attempts, so a group's
  infrastructure error exited 1, and a launch failure (all members skipped) exited 0
- group attempt that fails before any member runs (launch, file load) fails
  the first member with the error, rest skip as serial-predecessor-failed;
  an interrupt there skips every member instead
- a retry that never reached its members keeps the verdicts of the attempt before it
- worker crash during a serial member fails that member with WORKER_CRASH
  instead of skipping it as never started

* docs(test): an interrupted serial launch skips every member
2026-10-04 10:35:21 +02:00
Oskar Kwaśniewski 78e135a372 fix(secrets): redact plain-string secret values from traces and text downloads (#814)
* fix(secrets): rewrite traces and text downloads whenever the run has a secret value

A registered value a test passed as a plain string (an app.open URL, a fill)
was redacted from titles, labels, and agent input but kept verbatim in
trace.zip: trace and download rewriting was gated on the exposure level,
which only a fill or an engine-held secret raises. Rewriting now depends on
the session ledger holding a value. The engine exposure level had no other
consumer and is removed; exposure now only decides pixels and saved-session
taint.

* fix(secrets): cover a plain-string fill, scope docs to the session's known values

* docs(secrets): download redaction is whole-value; TRACE_WITHHELD outside-path case

* docs(secrets): name the download rewrite failure path
2026-10-04 10:34:34 +02:00