Commit Graph
49 Commits
Author SHA1 Message Date
Oskar Kwaśniewski 8657c8b710 test(mobile): fake clock in engine tests, drop the simulator suite CI never ran (#808)
* test(mobile): fake the clock and shrink PNGs in engine tests, trim duplicates

engine.test.ts 6.4s -> 0.4s: retry backoff and transition budgets on fake timers, small screenshots. Merge or drop tests restating constants or covered elsewhere; prose asserts in errors.test check the headline and recovery only.

* test(mobile): delete the simulator suite no workflow runs

E2E_AGENT_DEVICE_SIMULATOR was never set in CI; the mobile benchmark drives real simulators and emulators. Drops test:simulator and the integration project.

* test(mobile): cover duplicate refind, clipboard step label, unreadable secure screenshot

* test(eas): restore Date.now in afterEach, unique run ids, drop constant asserts

* test(github): drop the factory tautology and markdown asserts core owns

* test(mobile): keep the videoTouches and interactive snapshot option checks

* test(mobile): name the openApp permission refusal by what it checks
2026-10-04 10:10:57 +02:00
Oskar Kwaśniewski 0d1ed15871 fix(e2e)!: load TypeScript with e2e's own oxc loader, so pnpm 11+ installs pass (#790)
* fix(e2e): load TypeScript with an oxc loader instead of tsx

* fix(e2e): close loader edge cases: sourceless CommonJS, export maps without extensions, unreadable tsconfig, chained source maps, Windows require paths

* fix(e2e): refuse Bun and Deno runtimes, name oxc load failures, follow Yarn PnP export targets

* perf(e2e): skip the syntax check for files whose only @ is in a package import

* fix(e2e): allow type-only exports in .cts, warn from runner only, Bun check first, docs

* fix(e2e): workers warn about tsconfigs runner did not, FILE: urls, foreign infra errors, review tests
2026-10-04 09:22:50 +02:00
github-actions[bot] 09733be70a chore: version packages (#774) 2026-10-04 09:12:59 +02:00
Oskar Kwaśniewski 3a9667c046 fix(run): keep --last-failed reruns honest: failure-limit skips, carried debts, kept evidence (#724)
* fix(run): keep --last-failed reruns honest: failure-limit skips, carried debts, kept evidence

--last-failed now reruns tests --max-failures skipped. A test or failed
suite hook another filter leaves out stays owed in the report's new
run.carried until a rerun runs it; the github fold shows carried rows and
errors and stays red while anything is carried. A rerun prunes artifacts/
to what the previous report names and writes its attempts under
artifacts/rerun-<n>/ instead of wiping the tree and reusing attempt paths.

* fix(run): carry a setup's hook debt when no selected test needs it

The report has no row for an unneeded setup, so carryForward now asks the
collection whether a test still exists instead of the report's rows.

* fix(run): reject malformed artifacts in --last-failed reports, tighten rerun docs

Distinct ids in the carried fixture row.

* fix(run): keep carrying tests of a file a narrowed rerun could not collect

* fix(github): a carried interrupted test reads interrupted, not failed, in the folded page
2026-10-03 16:36:53 +02:00
Oskar Kwaśniewski 7ab80bc610 fix(report)!: count interrupted tests apart from failures (#713)
* fix(report)!: count interrupted tests apart from failures

* fix(report): tell a failed test from its last failed attempt, whatever the cut retry left
2026-10-03 09:30:44 +02:00
github-actions[bot] a0ee3e9061 chore: version packages (#761)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2026-10-02 19:15:15 +02:00
Oskar Kwaśniewski 62bfe0a896 fix(github): long marker fields keep a digest so distinct keys never share a comment (#720)
* fix(github): long marker fields keep a digest so distinct keys never share a comment

* docs(changeset): note one-time duplicate for long marker fields
2026-10-02 18:17:26 +02:00
Oskar Kwaśniewski b573756818 fix(e2e): slim the install - split MCP server SDK, no sourcemaps (#763) 2026-10-02 15:17:58 +02:00
Oskar Kwaśniewski 6fa2bc2222 ci(release): publish through npm trusted publishing (OIDC) (#707)
Drops NPM_TOKEN from release.yml; each package grants release.yml publish
rights via an npm trusted publisher. Release job moves to ubuntu-latest
because npm rejects OIDC from self-hosted runners (Blacksmith included).
publishConfig.provenance pins removed so provenance attaches automatically
once the repo is public. Setup recipe documented in CONTRIBUTING.md.
2026-10-01 11:35:22 +02:00
github-actions[bot] ba9a8b6727 chore: version packages (#714)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2026-10-01 09:58:06 +02:00
Oskar Kwaśniewski 1a80c23d87 chore(npm): e2e ships the repo README, revised keywords for every package (#706)
* chore(npm): e2e ships the repo README, revised keywords for every package

* test(skill): follow the skill copy into prepare-build.ts

* fix(npm): keep README links relative on GitHub, rewrite them for the npm copy
2026-10-01 09:41:33 +02:00
Oskar Kwaśniewski 079a50e36d chore: release 2026-09-30 21:03:02 +02:00
Oskar Kwaśniewski 4d488f8286 feat!: the app is declared on its target; engines only drive it (#698)
* feat!: the app is declared on its target; engines only drive it

The app under test moves from the engine's options to `targets: [{ engine,
app }]`: `url`, `bundleId`, `appPath`, `identity`, `environment`,
`launchArguments`, `permissions`, `command`, and `readyUrl`. `web()` and
`mobile()` refuse every moved option in one `INVALID_CONFIG` naming the
target block; `defineEngine({ app })` is gone and engines check what their
platform needs in `validateApp`. `requires: ['browser' | 'native-app']` is
derived from the target's app. Replay cache and session keys are byte for
byte what they were, pinned by target-identity.test.ts.

Dependency services are not part of this version: `services` anywhere, and
`web({ services })`, fail saying so. `app.command` on port 0 without a
command, and `reuseExisting` on a free port, are refused. An `e2e mcp`
open cancelled or disconnected mid-start stops what it started, and a hung
shared start is abandoned once no session waits on it.

* fix: review follow-ups: cancelled opens, abandoned starts, null targets, stale docs

* refactor: drop the moved-option messages; old app keys are unknown keys

* test: give the replaced-nodes suite a budget that survives a loaded runner

* test: size the replaced-nodes hook budget to its seven tests
2026-09-30 19:48:00 +02:00
Oskar Kwaśniewski e4fca9debd chore(release): versioned releases publish to latest, ship the github reporter with the stable runner (#682)
* chore(release): versioned releases publish to latest, ship the github reporter with the stable runner

Versioned releases went to the beta dist-tag while latest stayed on a July placeholder, so a bare npx e2e init installed 0.1.0. The release script now passes no --tag and publishConfig.tag reads latest on every package.

@e2e-dev/github had no pending changeset; its only published builds peer on one exact canary runner and refuse a stable e2e. A patch changeset ships it alongside 0.15.0 with the wide peer range.

Quickstart and the skill setup reference install plain e2e instead of @canary. CONTRIBUTING and AGENTS describe the latest channel.

* chore(changeset): word the github reporter entry for the changelog
2026-09-30 16:38:52 +02:00
Oskar Kwaśniewski a5b3eaccfd fix(release): restore the wide e2e peer range the canary release pinned (#628) 2026-09-29 10:05:21 +02:00
Oskar Kwaśniewski 640a73ef75 chore: release 2026-09-28 20:50:21 +02:00
Oskar Kwaśniewski 67c29fc6b2 chore!: publish engines and reporter under the @e2e-dev npm scope (#602) 2026-09-28 19:05:05 +02:00
Oskar Kwaśniewski 61603e6c2c fix(github): a --last-failed rerun updates the pull request comment as one run (#537) 2026-09-25 17:09:09 +02:00
Oskar Kwaśniewski 60cd81f70e chore: restore the engine peer ranges the canary release pinned (#518) 2026-09-25 10:54:35 +02:00
Oskar Kwaśniewski 9a8e1cfc8d chore: release 2026-09-25 09:56:08 +02:00
Oskar Kwaśniewski 48c9e10d9b chore: release 2026-09-24 16:13:16 +02:00
Oskar Kwaśniewski cdc41098c7 fix(report): markdown cells cannot mention, autolink, or start a block, and all-dot names are rejected (#448) 2026-09-23 10:53:49 +02:00
dependabot[bot] 2e1a3107e1 chore(deps): bump the dev-tooling group across 1 directory with 7 updates (#464)
Signed-off-by: dependabot[bot] <support@github.com>
2026-09-23 10:48:59 +02:00
Oskar Kwaśniewski 9614834797 fix(release): engine and reporter peers on e2e stay >=0.15.0 <1 after a canary, and pnpm check enforces it (#432) 2026-09-23 08:24:35 +02:00
Oskar Kwaśniewski b84fefb5d2 chore: release 2026-09-22 18:14:46 +02:00
Oskar Kwaśniewski b71ecc0f09 fix(report): leave unselected results off the page and JUnit, link sources under the project path (#422) 2026-09-22 18:14:28 +02:00
Oskar Kwaśniewski a44d932418 chore: release 2026-09-22 15:50:19 +02:00
dependabot[bot] 500b7e3bc3 chore(deps): bump the dev-tooling group across 1 directory with 11 updates (#402)
Bumps the dev-tooling group with 11 updates in the / directory:

| Package | From | To |
| --- | --- | --- |
| [@changesets/cli](https://github.com/changesets/changesets/tree/HEAD/packages/cli) | `3.0.2` | `3.0.3` |
| [@types/node](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/node) | `26.4.1` | `26.6.1` |
| [fallow](https://github.com/fallow-rs/fallow) | `3.24.1` | `3.27.0` |
| [oxlint](https://github.com/oxc-project/oxc/tree/HEAD/npm/oxlint) | `1.82.0` | `1.83.0` |
| [@ai-sdk/openai](https://github.com/vercel/ai/tree/HEAD/packages/openai) | `4.0.69` | `4.0.70` |
| [@ai-sdk/openai-compatible](https://github.com/vercel/ai/tree/HEAD/packages/openai-compatible) | `3.0.51` | `3.0.52` |
| [@ai-sdk/xai](https://github.com/vercel/ai/tree/HEAD/packages/xai) | `5.0.2` | `5.0.3` |
| [ai](https://github.com/vercel/ai/tree/HEAD/packages/ai) | `7.0.105` | `7.0.106` |
| [vitest](https://github.com/vitest-dev/vitest/tree/HEAD/packages/vitest) | `5.0.0` | `5.0.1` |
| [@ai-sdk/devtools](https://github.com/vercel/ai/tree/HEAD/packages/devtools) | `1.0.18` | `1.0.22` |
| [mint](https://github.com/mintlify/mint/tree/HEAD/packages/mint) | `4.2.883` | `4.2.903` |



Updates `@changesets/cli` from 3.0.2 to 3.0.3
- [Release notes](https://github.com/changesets/changesets/releases)
- [Changelog](https://github.com/changesets/changesets/blob/main/packages/cli/CHANGELOG.md)
- [Commits](https://github.com/changesets/changesets/commits/@changesets/cli@3.0.3/packages/cli)

Updates `@types/node` from 26.4.1 to 26.6.1
- [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases)
- [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/node)

Updates `fallow` from 3.24.1 to 3.27.0
- [Release notes](https://github.com/fallow-rs/fallow/releases)
- [Changelog](https://github.com/fallow-rs/fallow/blob/main/release.toml)
- [Commits](https://github.com/fallow-rs/fallow/compare/v3.24.1...v3.27.0)

Updates `oxlint` from 1.82.0 to 1.83.0
- [Release notes](https://github.com/oxc-project/oxc/releases)
- [Changelog](https://github.com/oxc-project/oxc/blob/main/npm/oxlint/CHANGELOG.md)
- [Commits](https://github.com/oxc-project/oxc/commits/oxlint_v1.83.0/npm/oxlint)

Updates `@ai-sdk/openai` from 4.0.69 to 4.0.70
- [Release notes](https://github.com/vercel/ai/releases)
- [Changelog](https://github.com/vercel/ai/blob/main/packages/openai/CHANGELOG.md)
- [Commits](https://github.com/vercel/ai/commits/@ai-sdk/openai@4.0.70/packages/openai)

Updates `@ai-sdk/openai-compatible` from 3.0.51 to 3.0.52
- [Release notes](https://github.com/vercel/ai/releases)
- [Changelog](https://github.com/vercel/ai/blob/main/packages/openai-compatible/CHANGELOG.md)
- [Commits](https://github.com/vercel/ai/commits/@ai-sdk/openai-compatible@3.0.52/packages/openai-compatible)

Updates `@ai-sdk/xai` from 5.0.2 to 5.0.3
- [Release notes](https://github.com/vercel/ai/releases)
- [Changelog](https://github.com/vercel/ai/blob/main/packages/xai/CHANGELOG.md)
- [Commits](https://github.com/vercel/ai/commits/@ai-sdk/xai@5.0.3/packages/xai)

Updates `ai` from 7.0.105 to 7.0.106
- [Release notes](https://github.com/vercel/ai/releases)
- [Changelog](https://github.com/vercel/ai/blob/main/packages/ai/CHANGELOG.md)
- [Commits](https://github.com/vercel/ai/commits/ai@7.0.106/packages/ai)

Updates `vitest` from 5.0.0 to 5.0.1
- [Release notes](https://github.com/vitest-dev/vitest/releases)
- [Changelog](https://github.com/vitest-dev/vitest/blob/main/docs/releases.md)
- [Commits](https://github.com/vitest-dev/vitest/commits/v5.0.1/packages/vitest)

Updates `@ai-sdk/devtools` from 1.0.18 to 1.0.22
- [Release notes](https://github.com/vercel/ai/releases)
- [Changelog](https://github.com/vercel/ai/blob/main/packages/devtools/CHANGELOG.md)
- [Commits](https://github.com/vercel/ai/commits/@ai-sdk/devtools@1.0.22/packages/devtools)

Updates `mint` from 4.2.883 to 4.2.903
- [Commits](https://github.com/mintlify/mint/commits/HEAD/packages/mint)

---
updated-dependencies:
- dependency-name: "@ai-sdk/devtools"
  dependency-version: 1.0.22
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: dev-tooling
- dependency-name: "@ai-sdk/openai"
  dependency-version: 4.0.70
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: dev-tooling
- dependency-name: "@ai-sdk/openai-compatible"
  dependency-version: 3.0.52
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: dev-tooling
- dependency-name: "@ai-sdk/xai"
  dependency-version: 5.0.3
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: dev-tooling
- dependency-name: "@changesets/cli"
  dependency-version: 3.0.3
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: dev-tooling
- dependency-name: "@types/node"
  dependency-version: 26.6.1
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: dev-tooling
- dependency-name: ai
  dependency-version: 7.0.106
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: dev-tooling
- dependency-name: fallow
  dependency-version: 3.27.0
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: dev-tooling
- dependency-name: mint
  dependency-version: 4.2.903
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: dev-tooling
- dependency-name: oxlint
  dependency-version: 1.83.0
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: dev-tooling
- dependency-name: vitest
  dependency-version: 5.0.1
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: dev-tooling
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-09-22 11:56:23 +02:00
Oskar Kwaśniewski ec52066c70 chore: release 2026-09-21 20:01:38 +02:00
Oskar Kwaśniewski f5e96d0cbe feat!: rename engine packages to @e2edev/web and @e2edev/mobile (#392) 2026-09-21 19:52:32 +02:00
Oskar Kwaśniewski ca5e6196b6 chore: relicense from MIT to Apache-2.0 (#373) 2026-09-21 14:29:22 +02:00
Szymon Rybczak e9c2914c6d fix(docs): repair links to the published documentation (#367) 2026-09-20 14:36:38 +02:00
Oskar Kwaśniewski 3d0fe00ef2 chore: release 2026-09-17 23:41:56 +02:00
Oskar Kwaśniewski b733218f55 chore: release 2026-09-17 23:37:54 +02:00
Oskar Kwaśniewski 2e593dfb46 fix(agent): recover semantic timeouts with masked pixels (#317)
* fix(agent): recover semantic timeouts with masked pixels

Distinguish unavailable semantics from an empty tree so a proven fresh screenshot can remain useful without supporting an absence judgment. Reserve time for independent masking, reject stale references, and stop trace replay or recording after evidence degrades.

Respect judgment vision settings and attempt pixel taint. Keep device capture closed when accessibility bounds cannot prove masking. Report pre-model capture failures without inventing a revision or verdict.

* fix(agent): isolate fallback capture and evidence ownership

Reserve document node IDs before readers run and bound the entire semantic transaction, including metadata and handle readback. Quarantine late handles and publish each capture only while its page and generation remain current.

Carry semantic or pixel evidence through the internal session and agent paths. Preserve completed cache captures for one executor handoff, retire semantic presentation history on fallback, and keep public interfaces extendable.

* test(agent): align capture assertions with cache handoff

Count the reused cache observation once and wait for the fixture delayed control before asserting an unchanged screen after batched actions.

* docs(agent): clarify fallback without act vision modes

* test(agent): cover requested pixels after semantic fallback

* test(playwright): wait for recorded paint in lifecycle coverage

* test(playwright): flush a frame before checking video paint
2026-09-15 13:47:30 +02:00
Oskar Kwaśniewski 4c76360cb6 feat(report): the run page is paragraphs a reader can skim; flaky tests and the file list fold away (#310) 2026-09-14 17:50:53 +02:00
Oskar Kwaśniewski 9c835ba64e docs: links point at docs.e2e.army instead of the Mintlify preview host (#308) 2026-09-14 16:32:10 +02:00
Oskar Kwaśniewski d520071925 chore: release 2026-09-14 16:25:54 +02:00
Oskar Kwaśniewski 775941cdac fix(release): canary publishes in its own step and refuses a stale dist (#307) 2026-09-14 16:09:55 +02:00
Oskar Kwaśniewski 9820854c7d chore: release 2026-09-14 14:57:07 +02:00
Oskar Kwaśniewski 906e0b357c chore: release 2026-09-14 11:54:54 +02:00
Oskar Kwaśniewski c2c5df7df9 feat(e2e)!: the runner publishes as e2e; @e2edev/e2e is retired (#294) 2026-09-14 08:56:22 +02:00
Oskar Kwaśniewski e301105408 chore(release): publish the packages public and add a hand-run canary channel (#293) 2026-09-14 08:44:10 +02:00
Oskar Kwaśniewski 0746c305f9 chore: release 2026-09-12 18:59:03 +02:00
Oskar Kwaśniewski 8811ba7de9 feat(report): a markdown reporter writes the run as summary.md, with an exploration's findings, and the GitHub comment renders from it (#284) 2026-09-12 15:54:10 +02:00
Oskar Kwaśniewski 8287bf930b feat(run)!: run one test as several agents; --agent takes a list (#262) 2026-09-11 10:03:35 +02:00
Oskar Kwaśniewski 4cc86dcfaf fix(cli): init lists engines by platform, runs through the package manager, and links the live docs (#256) 2026-09-10 18:17:03 +02:00
Oskar Kwaśniewski d00b4ab50d chore: release 2026-09-10 17:51:59 +02:00
Oskar Kwaśniewski a0a4df01b4 feat(github): the GitHub reporter comments on the pull request from Actions (#246) 2026-09-10 14:47:36 +02:00