* docs: add a Next.js guide page
One page takes a Next.js developer from `e2e init` to CI: `next dev`
locally and `next start` on the build in CI from one config, the port,
`reuseExisting`, and the Next.js problems (`allowedDevOrigins` on
127.0.0.1, the route announcer that `getByRole('alert')` also matches, a
webpack watcher that reads `.e2e/`), plus the `create-next-app` starter
and the Vercel template. The hydration accordion moves here from web.mdx;
web, cache, and example-projects link to the page, with ja and zh
translations, and the e2e skill's setup reference carries the same
guidance.
Refs ENG-1378
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* docs: move the Next.js page into a Frameworks group
Next.js is not a platform like web, mobile, or desktop, so the page moves
to docs/frameworks/nextjs.mdx under a new Frameworks navigation group. The
web page no longer points at it from the app URL paragraph.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* feat(test): set fixtures up once per worker and pass them to beforeAll
test.extend() takes [fn, { scope: 'worker' }]: one instance per worker and
target, torn down when the worker ends. Suite hooks receive worker fixtures.
Engines can declare isolation: 'worker'; @e2e-dev/mobile does, so its device,
app, and screen reach beforeAll, afterAll, and worker fixtures.
Refs ENG-1297
* feat(e2e)!: set up fixtures on demand
A test now sets up only the test.extend() fixtures its body and its
beforeEach and afterEach hooks name in their first parameter, the fixtures
those name, and the auto ones. A suite hook sets up the worker fixtures it
names and the auto worker fixtures. Names are read from the function's
source, as Playwright and Vitest read them.
- [fn, { auto: true }] and [fn, { scope: 'worker', auto: true }] run a
fixture nothing names.
- A fixture can name the fixtures declared beside it in the same extend()
call. Fixtures run in declaration order, each after the ones it names.
- The attempt is resolved as one unit before the session launches: a
fixture only an afterEach names is set up before the first beforeEach.
An afterEach that names a fixture whose setup failed is skipped.
- Fixture rules fail as COLLECTION_ERROR with the file and line of the
code: a parameter that is not destructured or takes a rest property, and
a cycle, when the file is collected; a name the target lacks, a test
fixture named by a suite hook or a worker fixture, and a fixture named
like an engine's, at selection, for the pairs that run on that target.
- On an engine with isolation: 'worker', a suite hook or a worker fixture
teardown that names no engine-backed fixture opens no engine attempt.
BREAKING CHANGE: a fixture nothing names no longer runs unless it is
declared with auto: true, and a test body, hook, or fixture function that
takes the whole fixture object (async (fixtures) => {}) no longer loads.
Refs ENG-1297
* refactor(core): declare core fixtures in one table
The five core fixtures (agent, app, screen, platform, session) are one
table in @e2e-dev/core: name, scope, and factory. createFixtures builds
the built-ins object through one loop over that table and the engine's
factories, every fixture a lazy accessor behind the same gate as before.
The runner's fixture pool, test.extend() validation, and launch read the
names and scopes from the table, so the name lists and the per-name scope
checks are gone. A launch session exposes the core fixtures that do not
belong to a test: app, screen, platform.
No behaviour change. app and screen were built with the attempt and are
now built on first read, like agent and the engine's fixtures.
Refs ENG-1297
* feat(e2e): extend a built-in fixture through test.extend
A test.extend() fixture named like one the test already has extends it,
by Playwright's rule: inside its function its own name is the fixture it
extends (the built-in, or an earlier fixture of that name), and tests,
hooks, and other fixtures that name it receive what it hands to use().
base.extend({ device: async ({ device }, use) => {
await device.setPermission('camera', 'deny');
await use(device);
} })
- A plain function takes the scope the extended fixture has on the
target: device is a worker fixture on mobile, so its override runs once
per worker and reaches beforeAll; app is a test fixture on web and a
worker fixture on mobile. A declared scope that differs is a
COLLECTION_ERROR naming both.
- A worker-scoped override receives what a worker fixture receives for
that built-in, the forwarder to the attempt that is running.
- On a target without the built-in nothing fails until code that runs
there names it; then the target has no such fixture.
- Only user code sees the override: the attempt's own built-ins, which the
runner and the worker fixtures' forwarders read, are untouched.
- Types: the override keeps the extended fixture's type and scope, through
an extend() overload that takes overrides without type arguments.
A fixture named like a core fixture, an engine's, or an earlier one of the
chain was refused before.
Refs ENG-1297
* docs(fixtures): name built-in fixtures and give their scope per engine
The docs now use one term, built-in fixtures, for agent, app, screen,
platform, session, and the ones an engine contributes (browser, device),
where pages said core fixtures, contributed fixtures, or engine fixtures.
The test reference gets one table with each built-in fixture's scope on
web and on mobile, and says what the scope decides: a test fixture reaches
test bodies, beforeEach, and afterEach; a worker fixture also reaches
beforeAll, afterAll, and worker fixtures. The hooks, worker fixtures, and
override sections link to it, as do the fixtures guide and the engine
pages. The Japanese and Chinese pages carry the table and the same terms.
Refs ENG-1297
* docs(ja): translate the fixture scope and on-demand changes
* docs(zh): translate the fixture scope and on-demand changes
* fix(e2e): let a worker fixture name an extended app or screen
A fixture that extends `app` or `screen` was taken for a test fixture when
its test.extend() call was checked, before any target is known. A worker
fixture declared after it that names `app` then failed at import on every
target, though the same fixture is accepted without the override and the
plan resolves both to worker scope on an engine whose state outlives a
test.
The scope of a plain function is now unresolved until a target is known,
whatever it extends, as it already was for an engine's own fixtures.
Refs ENG-1297
* fix(e2e): fail a suite hook that left an engine step running
On an engine whose state outlives a test, a beforeAll or afterAll hook, or
a worker fixture's teardown, can start a device step and not await it. The
hook's attempt then closed under the step, and its failure arrived later
as a stray rejection charged to whichever test ran then.
The suite attempt now does what a test attempt does once its body settled:
the step is failed at its call and cancelled, its outcome is awaited within
the cleanup budget, and the hook fails with STEP_NOT_AWAITED (HOOK_FAILED
for a suite hook).
The suite attempt keeps no step records, and StepRecorder dropped the
record of the step it had just opened before marking it running, so there
was nothing to abandon: a step now counts as running before the retained
window is trimmed.
Refs ENG-1297
* refactor(e2e): keep the fixtures a function names beside it
Registration parsed each test body, hook, and fixture function for the
fixtures it names, dropped the result, and left it in a WeakMap that the
planner and the pool read back with placeholder subjects. A cache miss
there would have named the wrong code and no line.
The parse result is now data: `names` on the registered test, the hook,
and the fixture definition. The planner and the pool read it, the WeakMap
and the per-function wrappers are gone, and one describeUser() gives an
error its subject. The explore test, registered by hand, parses its body
where it builds the record.
Unit tests state the names instead of generating source text for them.
Refs ENG-1297
* refactor(core): resolve built-in fixtures in one place
The core table declared the core fixtures, but what a target's built-in
fixtures are was still assembled around it: core names plus engine names in
three files, the 'engine' scope resolved in e2e through a cast into the
table, `platform` hard-coded as the one fixture that needs no attempt, and
two structural copies of the target.
core now exports builtInFixtures(target): names, scope(name),
needsAttempt(name), attemptless(name), and the unknown-fixture message,
over one BuiltInTarget. `platform` declares in the table that the target
alone decides it. launch, the planner, and the pool read from it.
DECLARED_BUILT_INS is the same view before a target is known: a scope the
engine decides is 'unresolved'. The planner takes either one, so its
no-target mode and the `target === undefined` branches through scopeOf and
read are gone. The scope of a fixture that extends `app` before a target
is known is 'unresolved' by construction.
Refs ENG-1297
* refactor(e2e): hand each function the fixtures it names
Every test body, hook, and fixture function must destructure its first
parameter, so what it reads is known before it runs. bindFixtures still
built a Proxy-gated bag for arbitrary property reads: one value in three
stores, the gate run twice (once over a faked plan), a third derivation of
scope, and an own-name shadow that re-derived what the planner had resolved.
No test code could reach the gate's throwing branches; only unit tests did,
by reading the bag directly.
The plan now records, per function, what each name it reads resolved to: a
chain fixture or a built-in. The pool hands each function a plain object
with exactly those names, the value a chain fixture handed over, or a
built-in read off the attempt when the function destructures it, so
built-ins stay lazy. An override that names itself receives the fixture it
extends because that is what its own name resolved to.
- bound.call(user) replaces user.fn(bound.fixtures); ready(user) asks
whether every chain fixture it names was provided.
- attemptlessBuiltIns is a function from a name to a value that throws the
reason for a built-in that needs an attempt.
- The planner runs once: it records what each fixture names, and the order
is one declaration-order walk over that record. A second validating pass
only reordered.
Behaviour: a bound or native function, whose parameters cannot be read, is
a COLLECTION_ERROR; it used to parse as naming nothing and still got lazy
built-ins. The docs no longer promise hooks and the body one shared object.
Refs ENG-1297
* refactor(e2e): move the suite attempt out of TargetExecutor
dispose, openSuiteHooks, and openSuiteAttempt were 177 lines of a second
attempt lifecycle inside TargetExecutor, written as closures in one method.
run/suite-attempt.ts now owns it: a SuiteAttempt class with open, acquire,
within, and close, over a SuiteAttemptHost interface the executor satisfies
(the way serial.ts borrows it through SerialHost), plus the worker
fixtures' teardown pass. It is the realm's SuiteHookSession itself, so the
executor keeps a context object and two call sites.
The three branches that built an attemptless reader with a "not available
to the <label>" error are one NoAttempt value ('isolated', 'unnamed',
'interrupted', or the launch failure) and one mapper to the message.
execute.ts: 1244 lines to 1052.
Refs ENG-1297
* refactor(e2e): share the attempt's fixture sequence
A run's attempt and an `e2e mcp` session's test each planned the test's
fixtures, bound them, entered the worker pool, set them up, ran the
beforeEach hooks and the body, filtered the afterEach hooks that can still
run, and labelled the teardowns. This PR widened that copy, and
standalone-test.ts asked the realm for the same hooks four times.
planAttemptFixtures(target, test, hooks) plans before any session exists
and returns open(), which binds the plan to the attempt: enter(), run()
for setup, hooks, and body, and cleanup() for the labelled afterEach hooks
and teardowns. Both call sites keep their own phases, budgets, and error
handling around it.
execute.ts: 1052 lines to 1031, its size on main.
Refs ENG-1297
* refactor(e2e): open the suite-hook pass without a lazy thunk
withSuiteHooks handed its work a thunk that opened the pass on first use.
Opening one does no I/O: the suite attempt launches nothing until a hook
that names an engine fixture runs. The pass is now opened up front and
closed in finally, and the `opened` slot and the `hooks()` call form go.
Refs ENG-1297
* refactor(e2e): ask the registry which hooks apply to a test
The selection-time fixture check rebuilt the scope chain of a test and
filtered the file's hooks by it, which the realm already did its own way.
registry.ts now exports scopeChain(test) and hooksFor(registration, test),
and both use them.
Refs ENG-1297
* refactor(core): move the fixture types into types/fixtures.ts
types.ts grew from 1679 to 1798 lines in this PR. The fixture scope, the
worker fixtures, and what test.extend() takes (definitions and overrides)
are one coherent group that nothing else in the file defines against, so
they move to types/fixtures.ts and types.ts re-exports them. The public
surface is the same.
types.ts: 1798 lines to 1715.
Refs ENG-1297
* refactor(core): drop the redundant override branch in FixtureDefinitions
A fixture that extends one the base already has was typed three ways: a
conditional branch inside the `Extra` map of FixtureDefinitions, the
`& FixtureOverrides` intersection, and the extend(overrides) overload.
Compiling the type tests without each in turn: the intersection types an
override written beside new fixtures with type arguments, and the overload
types an override-only call without them; both fail the tests when
removed. The conditional branch does not: with it gone the tests pass, and
a base name in `Extra` is `never` again, as it was before overrides. It is
removed, and the reference's copy of the declaration follows.
Refs ENG-1297
* refactor(e2e): split fixture errors from the parser, share hookLog
fixture-names.ts parses a parameter list, and also held FixtureUseError and
declarationSite, which format an error. They move to fixture-errors.ts.
Its identifier pattern held two raw zero-width characters; they are
written as escapes now.
hookLog and LOG_MODULE were pasted into three integration tests; they are
one helper in tests/helpers/hook-log.ts.
Refs ENG-1297
* docs(e2e): name the scope rule in the selection check's summary
The summary of assertFixtureUse still listed a chain fixture named like an
engine's as an error. Such a fixture extends the built-in now; the rule
left is a declared scope that differs from the extended fixture's.
Refs ENG-1297
* refactor(e2e): take execute.ts under 1000 lines
Two more pieces that are not the attempt's own sequence leave the file.
- Holding steps for `e2e run --step`: reading the screen a hold starts on
and wiring AttemptHolds to the runner's hook move to step-hold.ts, beside
the holds they configure (holdAttemptSteps, readHeldScreen).
- The result of a finished test or setup unit is a pure function of the
pair, the verdict, the attempts, and a beforeAll failure: unitResult in
units.ts, beside pairResult.
execute.ts: 1031 lines to 956. It was 1031 on main and 1226 at the review.
Refs ENG-1297
* test(e2e): match the not-awaited message without a step label
Main's #1880 names an unlabeled step by its api alone ("gadget.hang
finished"), where the message used to quote an empty label. The suite-hook
test's pattern required a label between the two, so it failed on the merge
with main.
* feat(e2e): an override inherits auto from the fixture it extends
A fixture that extends another and does not state `auto` now has the
extended fixture's, the way it has its scope: an override of an auto
fixture is set up for every test, and one of a built-in fixture on demand.
An `auto` that differs from the extended fixture's is a COLLECTION_ERROR
naming both, worded like the scope mismatch. This matches Playwright.
It also fixes which fixtures the plan starts from. An auto fixture was a
root by its own flag, so with an auto base and its override in one chain
the base was set up and the override skipped, while the name resolved to
the override; an override that replaced the base without naming it still
left the base running. An auto fixture is now a root only where its name
stands for it: the base runs once through the override's own name, or not
at all when the override replaces it.
Refs ENG-1297
* fix(core): name the hook or fixture that left a step running
STEP_NOT_AWAITED said "the test body returned before ..." for a step a
beforeAll, afterAll or beforeEach hook, a fixture's setup or a worker
fixture's teardown left running. The recorder now keeps the owner each
step was started under, and the runner says which code runs: the pool
for fixtures, hooks and the body, the suite attempt for suite hooks, and
a hosted session for itself.
---------
Co-authored-by: Oskar Kwaśniewski <oskar@okwasniewski.com>
Manage local Moli CDP browsers through the web engine's BrowserProvider API.
Each worker gets an available loopback port, layout and resource loading,
startup cancellation, and process cleanup when its lease ends.
Wire the testbed, documentation, consumer skill, build and test commands,
and a patch changeset for @e2e-dev/moli.
Keep process-group shutdown active after an executable wrapper exits,
and surface process-inspection failures in the provider tests.
Verified locally:
- pnpm check
- 16 Moli provider unit tests
- Testbed on two workers: 16 passed, 1 expected skip; both browsers released
- Empty and whitespace CDP URLs use managed browsers; padded URLs attach
without stopping the existing server
- MCP login, secret fill, assertions, and screenshot
- Packed packages installed, typechecked, and run in an isolated project
* feat(runner): start the tests a change is likeliest to break first
A test filter can score the tests it keeps: `relevance` in its result maps test ids to a number from 0 to 1. With `--changed-since`, each target starts the files of the likeliest tests first, in bands at 75%, 50%, and 25%, and the target with the likeliest file goes first. Inside a band the longest-first order from the previous run still decides, with more than one worker.
`decisionFilter()` scores every test it hands back: 1 for a test whose file, setup file, or stored screenshots changed, the model's answer for one it judged, and 0.5 for one it did not. It requires e2e 0.23.1 or newer, since an older runner refuses the unknown `relevance` key and drops the whole filter.
The run prints `ranked N of M tests by relevance, likeliest first` when the scores change the order.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
* test(runner): write the failing fixture as a constant, not built from a parameter
CodeQL flagged `failingFile(title)` for putting `JSON.stringify` output into generated code. Only `gamma` used it, so the file is a constant now.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
* docs(runner): say how targets, bands, and the ranking line are decided
Review of #1589 found the docs looser than the code: targets go by the band of their best file, not its exact score, with ties in config order; a rating the model did not give counts as 50% and the test takes the higher of its two ratings; the bands are half-open; the ranking line follows when the scores put tests in more than one band; and a filter that returns no scores leaves the order alone. One test title said an empty target goes last, but it keeps its place.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
* docs(runner): score known areas on the fallback path and name files, not tests
The areas example now scores the tests whose area a change reaches even when another changed file maps to no area, keeping the no-exclusion fallback. The README, docs, and skill say the run starts the files of the likeliest tests, ordered by relevance band with collection order breaking ties on one worker. The timings test makes beta and gamma wait for each other before they sleep, so a slow second worker cannot start alpha ahead of gamma.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
* fix(runner): keep the scores of a filter that fails
A filter that threw left every test unscored, so a run with no AI Gateway
key (a pull request from a fork) or a model outage fell back to collection
order and lost the one thing the decision filter knows without the model:
which tests the change touches.
The error a filter throws may now carry `relevance`. The runner checks it
the way it checks a result's, merges it by the highest score, and still
warns, leaves nothing out, and records `outcome: failed`. Scores that fail
the check are dropped, and the note says why. A call past its deadline is
abandoned, and nothing it throws is read.
decisionFilter() puts the scores the change alone gives (1 for a test the
change touches, 0.5 for the rest) on the error it throws, as a property
that is not enumerable, so printing the error does not list every test.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
* docs(runner): point the ordering prose at the timings history
The timings history (#1639) replaced the previous report as where file
durations come from, so the prose that says what orders files inside a
relevance band now names it: the filters guide, the `workers` bullet in
the config reference, the CI guide, the skill's running reference, and the
wording of the integration test that pins the one-worker gate.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
* docs: add a Writing fixtures guide
* docs: list Writing fixtures after Project tools and fold its long example
* docs: show fixtures creating test data and leave sign-in to saved sessions
* docs: create fixture data with agent.act instead of a test API
* docs: cut the Writing fixtures page to the essentials
* docs: import expect from e2e in the fixtures example
* docs(skills): define an atomic agent.act goal with a bad and good example
* docs(skills): check each todo goal in the act example
* docs: define an atomic agent.act goal and split compound examples
* docs(agent-steps): rewrite with the authoring-docs skill
* docs(agents): rewrite with the authoring-docs skill
* docs(api-testing): rewrite with the authoring-docs skill
* docs(authentication): rewrite with the authoring-docs skill
INVALID_CONFIG also fires for a password shorter than 6 characters.
* docs(browser): rewrite with the authoring-docs skill
* docs(bug-bash): rewrite with the authoring-docs skill
* docs(cache): rewrite with the authoring-docs skill
* docs(ci): rewrite with the authoring-docs skill
Rerun folders are rerun-<n>/, not rerun-1/. Merge exit codes into Gate pull requests.
* docs(ci/bitrise): rewrite with the authoring-docs skill
* docs(ci/codemagic): rewrite with the authoring-docs skill
* docs(ci/eas-workflows): rewrite with the authoring-docs skill
* docs(ci/github-actions): rewrite with the authoring-docs skill
* docs(coding-agents): rewrite with the authoring-docs skill
* docs(core-concepts): rewrite with the authoring-docs skill
* docs(debugging): rewrite with the authoring-docs skill
* docs(decision-models): rewrite with the authoring-docs skill
A point tap costs three decision calls and one text call, not four.
* docs(example-projects): rewrite with the authoring-docs skill
* docs(executors): rewrite with the authoring-docs skill
* docs(explore): rewrite with the authoring-docs skill
* docs(github): rewrite with the authoring-docs skill
* docs(index): rewrite with the authoring-docs skill
* docs(integrations/eas): rewrite with the authoring-docs skill
* docs(integrations/index): rewrite with the authoring-docs skill
* docs(integrations/kernel): rewrite with the authoring-docs skill
* docs(integrations/smol): rewrite with the authoring-docs skill
* docs(migrate/cypress): rewrite with the authoring-docs skill
* docs(migrate/detox): rewrite with the authoring-docs skill
* docs(migrate/maestro): rewrite with the authoring-docs skill
Fold the command map tables into accordions.
* docs(migrate/playwright): rewrite with the authoring-docs skill
List use and testMatch among the Playwright keys e2e rejects. Fold the mapping tables into accordions.
* docs(migrate/selenium): rewrite with the authoring-docs skill
* docs(mobile): rewrite with the authoring-docs skill
Fix the Android installApp troubleshooting entry: agent-device 0.21.22 parses the manifest itself and searches ~/Library/Android/sdk on macOS.
* docs(models): rewrite with the authoring-docs skill
* docs(quickstart): rewrite with the authoring-docs skill
* docs(security): rewrite with the authoring-docs skill
Name smol beside Kernel and EAS Simulators among the integrations that make outbound connections.
* docs(subscriptions): rewrite with the authoring-docs skill
npx e2e login with no provider opens a picker.
* docs(telemetry): rewrite with the authoring-docs skill
Add the source-checkout, unwritable-preferences, and CI opt-out rules.
* docs(tools): rewrite with the authoring-docs skill
Config load fails only when an entry sets both tools and a custom executor.
* docs(visual-testing): rewrite with the authoring-docs skill
* docs(web): rewrite with the authoring-docs skill
Fold the Next.js dev-origins section, merge the basicAuth examples, and fix the reuseExisting row: in CI e2e ignores it and only a server that already answers is APP_ALREADY_RUNNING. Drop the unrelated agents block from the command config example.
* docs(writing-an-engine): rewrite with the authoring-docs skill
Remove the origin-check claim for location; PR #290 removed origin policy.
* docs(models): keep process.loadEnvFile() as a way to load the pulled OIDC token
* docs(mobile): restore the imports in the several-devices and agent-tools configs
* docs(agent-steps): shorten
Prose words 841 -> 764.
* docs(agents): shorten
Prose words 789 -> 675.
* docs(api-testing): shorten
Prose words 270 -> 266.
* docs(authentication): shorten
Prose words 1000 -> 849.
* docs(browser): shorten
Prose words 1934 -> 1697.
* docs(bug-bash): shorten
Prose words 615 -> 585.
* docs(cache): shorten
Prose words 2793 -> 2374.
* docs(ci): shorten
Prose words 1338 -> 1182.
* docs(ci/bitrise): shorten
Prose words 321 -> 306.
* docs(ci/codemagic): shorten
Prose words 259 -> 250.
* docs(ci/eas-workflows): shorten
Prose words 431 -> 394.
* docs(ci/github-actions): shorten
Prose words 544 -> 453.
* docs(coding-agents): shorten
Prose words 841 -> 735.
* docs(core-concepts): shorten
Prose words 686 -> 620.
* docs(debugging): shorten
Prose words 1826 -> 1604.
* docs(decision-models): shorten
Prose words 1003 -> 942.
* docs(example-projects): shorten
Prose words 347 -> 288.
* docs(executors): shorten
Prose words 1297 -> 1114.
* docs(explore): shorten
Prose words 451 -> 413.
* docs(github): shorten
Prose words 723 -> 626.
* docs(index): shorten
Prose words 387 -> 333.
* docs(integrations/eas): shorten
Prose words 1293 -> 1119.
* docs(integrations/index): shorten
Prose words 174 -> 163.
* docs(integrations/kernel): shorten
Prose words 794 -> 620.
* docs(integrations/smol): shorten
Prose words 924 -> 855.
* docs(migrate/cypress): shorten
Prose words 957 -> 890.
* docs(migrate/detox): shorten
Prose words 1073 -> 1021.
* docs(migrate/maestro): shorten
Prose words 2464 -> 2246.
* docs(migrate/playwright): shorten
Prose words 4835 -> 4546.
* docs(migrate/selenium): shorten
Prose words 933 -> 887.
* docs(mobile): shorten
Prose words 2912 -> 2595.
* docs(models): shorten
Prose words 1187 -> 1048.
* docs(quickstart): shorten
Prose words 980 -> 946.
* docs(security): shorten
Prose words 587 -> 559.
* docs(subscriptions): shorten
Prose words 694 -> 651.
* docs(telemetry): shorten
Prose words 2158 -> 1999.
* docs(tools): shorten
Prose words 460 -> 397.
* docs(visual-testing): shorten
Prose words 583 -> 566.
* docs(web): shorten
Prose words 1262 -> 1104.
* docs(writing-an-engine): shorten
Prose words 1731 -> 1522.
* docs(agent-steps): address review findings
* docs(agents): address review findings
* docs(authentication): address review findings
* docs(cache): address review findings
* docs(ci): address review findings
* docs(coding-agents): address review findings
* docs(debugging): address review findings
* docs(example-projects): address review findings
* docs(executors): address review findings
* docs(index): address review findings
* docs(integrations/eas): address review findings
* docs(migrate/maestro): address review findings
* docs(subscriptions): address review findings
* docs(telemetry): address review findings
* docs(tools): address review findings
* docs(web): address review findings
* docs(writing-an-engine): address review findings
* feat(decision): add Clef and Jev executors on the System One choice API
* feat(decision): default Clef transport to clef-flash
* feat(decision): generalize to decision models, address review
* fix(decision): reject non-string decision choices
* fix(decision): skip empty select labels and blank destinations
An empty option label or whitespace-only destination offered as a choice fails at dispatch with INVALID_ARGUMENT and aborts the step instead of letting the model re-aim. Omit them at enumeration. Also qualify the uncertain-action docs: blocking stops the selected action, while earlier confident actions in the step have already run.
* feat(decision): add opt-in vision for Clef, Jev stays text-only
The executor stays text-only by default. With vision true and a transport that declares it, masked viewport pixels ride along as System One images data URLs. Clef declares vision; Jev never receives pixels. Withheld or tainted viewports fall back to semantic text.
* docs(decision): qualify vision cost to calls carrying pixels
* fix(decision): untainted pixel fixtures and honest skill pixels line
The test helper takes a tainted flag so the image-send tests use the only state the runner returns pixels in; tainted captures stay withheld observations. The skill no longer claims pixels are unavailable: screenshots may inform node choices with vision true, without adding point actions.
* bench(testbed): luna vs clef-flash harness and first numbers
Same todo flow under two configs: default agent on gpt-6-luna via OpenCodeX and decisionExecutor on clef-flash. Luna side passed in 23s with 6 model calls; clef-flash pending credentials. Includes the probe script, the docs section, and the flow screenshot.
* docs(decision): fill clef-flash benchmark row
Two no-cache runs both blocked uncertain on the first action (a25 at p=0.619, conf=0.387 against the default 0.9 gates), answering in about a second with 1,543 input tokens.
* docs(decision): benchmark aggregates over 5 runs per side
Luna completed 5/5 flows at ~14s and ~25k input tokens per run; clef-flash blocked uncertain 5/5 at ~1s and ~1.5k tokens per run.
* bench(testbed): parametrized flow, gate sweep, wire-shape finding
Params fix the missing type choices; gates 0.6/0.5 and 0.51/0.3 still block. Direct experiment shows hierarchical structured questions take clef-flash from 0.57 to 0.89 confidence on the same screen.
* feat(decision): hierarchical operation-then-target on structured transports
Structured transports (Clef, Jev) now decide the operation first and the target within it, with object instructions and element records. Flat endpoints keep the single string choice. Live benchmark: first operation 0.909/0.802, first target 0.940/0.775, up from 0.747/0.558 flat.
* docs(decision): compare luna vs clef-flash vs clef with measured cost
* feat(decision): run decisions through AI SDK evaluate, gates opt-in
decisionExecutor now accepts any AI SDK evaluation model that answers
choice questions, such as typeSafeAi.evaluationModel('jev-latest'), and
decides through experimental_evaluate (one choice question per decision,
no retries, so the call budget stays exact). ai is an optional peer that
loads only on that path. evaluationModel() exposes clef(), jev(), and
systemOne() transports to experimental_evaluate, which covers Clef until
it has an AI SDK provider.
The official TypeSafe provider rounds probabilities to two places, so a
long distribution can sum to 0.99: validation now allows half a unit per
rounded option, the AI SDK rule, and jev() declares the two places.
Probability and confidence gates are off by default and stay
configurable. Measuring clef-flash without them surfaced three executor
bugs, each fixed here:
- a lone target is dispatched without a one-option question, which the
choice API rejects with HTTP 422;
- acting decisions receive the step's action history, and the repeat
guard counts the same action on the same screen across the whole step,
so an A-B oscillation blocks instead of burning the call budget;
- the operation and target questions state that typed text is not saved
until it is submitted.
The structured path also checks the call budget before the target
request (review feedback).
Testbed todo flow, 5 no-cache runs each: clef-flash 5/5 (9.3s mean, 12
calls), clef 5/5 (11.5s, 12 calls), luna unchanged at 5/5 (14.9s).
* bench(testbed): Laya through systemOne
Laya serves the System One API and accepts structured values, so it runs
through systemOne({ structured: true }) unchanged. On the todo flow its
first operation question chose unsupported at 0.29 / 0.10 on four runs,
answered by the multilingual checkpoint its router picked for an English
prompt; the fifth run and further probes hit insufficient_credits.
* fix(decision): address review on rounding, history, and evaluation ids
- validateDecision accepts a rounded distribution only when 1 lies between
the sums of each value's rounding interval, clipped to [0, 1]; a summed
per-option tolerance exceeded 1 with many options and admitted two
options at probability 1.
- Decisions send the 20 most recent history entries, each clipped to 240
characters, so long steps and large params stay within request limits.
- evaluationModel() builds its answer, confidence, and criteria maps with
null prototypes, so an id such as __proto__ stays an own entry.
- @ai-sdk/provider moves to dependencies: the public declarations import
its evaluation types. ai stays an optional peer.
* bench(testbed): pin Laya's English checkpoint
Laya's router sent the full English prompt to its multilingual checkpoint.
The API accepts model 'english' to pin the English one (its routing
metadata reports the explicit model), so the bench config sends it.
* docs(decision): benchmark Laya locally on M1, both checkpoints 0/5
* docs(decision): reconcile laya narrative with run evidence, money not tokens
* feat(decision): caller-overridable transport flags on clef() and jev()
* feat(decision): AI SDK evaluate executor with element table and text model
* fix(decision): empty-tree check and provider pin
* fix(decision): address review on secrets, options, errors, and docs
Answers the cubic threads on the evaluate-only executor: terminal checks reject an incomplete screen instead of judging it, secret markers are stripped at any depth, hidden/disabled options are never offered, select options share the 255-choice cap, select criteria name the option, out-of-range probabilities fail, text-model errors mirror the decision mapping, and the README/docs describe the evaluate-only API.
* fix(decision): count dropped select options and fix README
Counts enabled options inside rows dropped past the element cap as omitted, adds the E2EConfig type import to the README example, and qualifies when type is offered.
* fix(decision): settle terminal claims by the check and match codebase style
The second rejected done/failed claim now ends by the check verdict (fails -> ACTION_FAILED, anything else -> blocked), and the completion check honors the gates. The decision state carries the last 10 actions, check targets show their checked state, checked radios are never toggled, and options dropped by the per-question cap stay out of omitted. The docs example opens /todos in test code since navigate is not offered. Single quotes, template literals, and split long lines across the package.
* fix(decision): show filled fields to verdicts and pin the history window
Verdict views (assert and the completion check) build the element table with typing enabled, so a field the step filled keeps its row and value even when the step itself cannot type. The history test now asserts the window holds the newest ten actions.
* fix(decision): let completion checks see results and name the text key
Page text keeps a named node's text when it differs from the name, as formatNode does, so a status named Greeting that says Welcome back, admin! reaches the verdict. The completion check now asks whether the task is complete and sees the non-secret params and the actions with their typed values; rejected claims stay out. The text prompt names the single text key, asks for no code fence, and prefixes the request so JSON-object response modes accept it.
* fix(decision): keep secret handles out of replayed history and mark check inputs as data
Replayed summaries name a filled secret handle; seeding the history now replaces declared secret names with <secret>, so names reach the model only as secret question criteria. The completion check treats action descriptions, typed values, and params as data, and the guide says the check sees only non-secret values.
* fix(decision): redact only the handle in replayed secret-fill summaries
Matching the declared secret name anywhere in a replayed summary also erased labels and values equal to it, and missed handles the runner cut at 40 characters. Only the quoted handle after fill secret at the start of the summary becomes <secret> now.
* docs(decision): describe the one-request fan-out in the guide intro
* fix(decision): count a scroll that changes what is in view as progress
The fingerprint behind the stall guard hashed node content but not position, so a scroll that only moved the viewport read as page unchanged and the third one blocked the step. It now also hashes whether each node meets the viewport (membership, not coordinates), so a scroll that brings other nodes into view resets the count while one that moves nothing, as at the page bottom, still trips the guard.
* chore: prune @pnpm/exe from the lockfile after merging main
* fix(decision): match the supported Node range in engines
---------
Co-authored-by: Oskar Kwaśniewski <oskar@okwasniewski.com>
* fix(skill): declare the app on the target in skill and docs config examples
The bundled skill and docs/authentication.mdx still taught web({ url, command }), removed in #698. The skill's two config examples now live in docs/examples/skill/, typechecked and mirrored by check-docs-examples, so the old shape fails pnpm check.
* fix(skill): run the docs drift check on skill changes, drop stale service wording
* feat!: browser fixture, getByRole name arg, toMatchSchema, top-level hooks, separate credential namespace
API review feedback before launch:
- web fixture renamed browser (Web -> Browser, WebExpectation ->
BrowserExpectation); requires: ['browser']; destructuring web hints the
new name. web() factory and other Web* types unchanged.
- getByRole(role, name, options?) beside the object form; MCP locate
writes the short form.
- expect(value).toMatchSchema(standardSchema): sync, returns the typed
output; poll resolves to it, soft returns undefined after a kept failure.
- describe, beforeEach, afterEach, beforeAll, afterAll exported from e2e;
web and mobile export typed beforeEach/afterEach.
- credentials and secrets are separate namespaces: a password handle is
<name>.password, secrets.get() never returns it. Committed recordings
migrated offline (old key hash verified, then recomputed).
- runtime test.skip keeps Playwright semantics; docs say it throws.
* docs(web): browser acts on one active tab; cookies, routes, dialogs cover the whole browser
* fix(secrets): judge a fill's purpose by the config, not the handle; no old-name hints
A config-time secrets.get() handle reads its purpose from the run's
registry again, and authorizeSecretFill checks the registered purpose, so a
handle claiming generic-secret cannot put a password into a plain field.
Drop the web fixture migration hint. Changesets name the on-screen marker
change for replay entries.
* docs(skill): browser fixture, getByRole name arg, toMatchSchema, top-level hooks, credential namespace on the rewritten skill
* fix(expect): negated toMatchSchema returns nothing, as its type says
* fix(expect,docs): soft double negation stays soft, null ~standard is INVALID_ARGUMENT, Playwright samples keep Playwright's getByRole
Also documents SoftExpectCall, SoftValueExpectation, and NegatedPollExpectation
on the expect reference, requires: ['browser'] in the skill's options table,
and tightens the poll and secret-purpose tests.
* docs(babysit): triage pattern for scripted renames that rewrite another library's code
* docs: engine options take a secrets entry, web package hook exports, vault example imports
* refactor(secrets)!: credentials hold their password, allSecrets feeds fills; engines export describe and every hook; BrowserRoute, BrowserResponse
From the maintainability review:
- ResolvedConfig.secrets is the config.secrets entries only; a credential
carries its password; allSecrets is the one map fills, the ledger, and
redaction read. The per-consumer purpose filters and the .password
suffix parsing are gone, and e2e mcp builds handles from config.
- The config load and a running test name the credential a missing
secrets.get() most likely meant.
- @e2e-dev/web and @e2e-dev/mobile export describe and all four hooks
beside test, so a test file registers from one import.
- WebRoute and WebResponse are BrowserRoute and BrowserResponse.
- getByRole normalizes its arguments once; isTextMatch is the one
string-or-RegExp check; poll reuses ValueMatcherName.
* revert(web): keep WebRoute and WebResponse; Browser names the fixture, Web the network and option types
* feat!: the app is declared on its target; engines only drive it
The app under test moves from the engine's options to `targets: [{ engine,
app }]`: `url`, `bundleId`, `appPath`, `identity`, `environment`,
`launchArguments`, `permissions`, `command`, and `readyUrl`. `web()` and
`mobile()` refuse every moved option in one `INVALID_CONFIG` naming the
target block; `defineEngine({ app })` is gone and engines check what their
platform needs in `validateApp`. `requires: ['browser' | 'native-app']` is
derived from the target's app. Replay cache and session keys are byte for
byte what they were, pinned by target-identity.test.ts.
Dependency services are not part of this version: `services` anywhere, and
`web({ services })`, fail saying so. `app.command` on port 0 without a
command, and `reuseExisting` on a free port, are refused. An `e2e mcp`
open cancelled or disconnected mid-start stops what it started, and a hung
shared start is abandoned once no session waits on it.
* fix: review follow-ups: cancelled opens, abandoned starts, null targets, stale docs
* refactor: drop the moved-option messages; old app keys are unknown keys
* test: give the replaced-nodes suite a budget that survives a loaded runner
* test: size the replaced-nodes hook budget to its seven tests
* feat(agents)!: plain-object agents entries, createAgent and limits removed
Each agents entry is one AgentConfig object: model, judge, system, context,
tools, maxSteps, maxModelCalls, judgmentTimeout, maxObservationBytes,
maxInputTokens, providerOptions, or executor for a custom brain (system and
tools rejected beside it). No inheritance between agents.
- createAgent() kept as a stub throwing INVALID_CONFIG; bare StepExecutor or
legacy createAgent value as an entry fails naming { executor } / the object
- timeout -> judgmentTimeout, maxTurns removed (entry and
createToolLoopExecutor), unknown keys get did-you-mean
- top-level limits removed; maxModelTokensPerCall -> per-agent
maxInputTokens, the rest internal constants; report-1 run.limits keeps
its shape, filled from max across agents plus the constants
- tools validated at config load; built-in agent built from the entry at
fixture time; explore and the MCP catalog read the entry's tools
- migrate apps, docs, docs examples, init scaffold, skills, mobile README
* docs: AGENTS.md names the agent entry's model, not createAgent
* refactor(agents)!: delete createAgent instead of a throwing stub
* docs: MODEL_UNAVAILABLE scoped to the built-in agent, fixture-acquisition timing
* feat(agent)!: act takes one options bag and resolves with what the step did
* fix(agent): a legacy act call fails loudly; a retry names itself in step.cache
`act(instruction, params)` and `act(instruction, params, options)` from 0.7
arrived as an options bag of foreign keys, or with a third argument the
fixture dropped, and ran with their parameters silently ignored. Both now
fail with `INVALID_ARGUMENT` naming the move to `{ params }`, from a
JavaScript test as from a typed one; `schema` and `vision` keep their
`UNSUPPORTED_CAPABILITY` message. Unit-tested in act-validation.test.ts.
A step in a retry attempt records a trace but never replays one, and its
`step.cache` was absent, indistinguishable from caching being off. It now
reads `missed` with the new reason `retry`, in the report schema too.
The reference page's opening `Agent` interface catches up with the act
signature, the summary is described as the step's (the replay writes it when
the cache finishes the step), and the quickstart example passes its values
under `params`, which is what failed the docs typecheck in CI.