* feat(test): set fixtures up once per worker and pass them to beforeAll
test.extend() takes [fn, { scope: 'worker' }]: one instance per worker and
target, torn down when the worker ends. Suite hooks receive worker fixtures.
Engines can declare isolation: 'worker'; @e2e-dev/mobile does, so its device,
app, and screen reach beforeAll, afterAll, and worker fixtures.
Refs ENG-1297
* feat(e2e)!: set up fixtures on demand
A test now sets up only the test.extend() fixtures its body and its
beforeEach and afterEach hooks name in their first parameter, the fixtures
those name, and the auto ones. A suite hook sets up the worker fixtures it
names and the auto worker fixtures. Names are read from the function's
source, as Playwright and Vitest read them.
- [fn, { auto: true }] and [fn, { scope: 'worker', auto: true }] run a
fixture nothing names.
- A fixture can name the fixtures declared beside it in the same extend()
call. Fixtures run in declaration order, each after the ones it names.
- The attempt is resolved as one unit before the session launches: a
fixture only an afterEach names is set up before the first beforeEach.
An afterEach that names a fixture whose setup failed is skipped.
- Fixture rules fail as COLLECTION_ERROR with the file and line of the
code: a parameter that is not destructured or takes a rest property, and
a cycle, when the file is collected; a name the target lacks, a test
fixture named by a suite hook or a worker fixture, and a fixture named
like an engine's, at selection, for the pairs that run on that target.
- On an engine with isolation: 'worker', a suite hook or a worker fixture
teardown that names no engine-backed fixture opens no engine attempt.
BREAKING CHANGE: a fixture nothing names no longer runs unless it is
declared with auto: true, and a test body, hook, or fixture function that
takes the whole fixture object (async (fixtures) => {}) no longer loads.
Refs ENG-1297
* refactor(core): declare core fixtures in one table
The five core fixtures (agent, app, screen, platform, session) are one
table in @e2e-dev/core: name, scope, and factory. createFixtures builds
the built-ins object through one loop over that table and the engine's
factories, every fixture a lazy accessor behind the same gate as before.
The runner's fixture pool, test.extend() validation, and launch read the
names and scopes from the table, so the name lists and the per-name scope
checks are gone. A launch session exposes the core fixtures that do not
belong to a test: app, screen, platform.
No behaviour change. app and screen were built with the attempt and are
now built on first read, like agent and the engine's fixtures.
Refs ENG-1297
* feat(e2e): extend a built-in fixture through test.extend
A test.extend() fixture named like one the test already has extends it,
by Playwright's rule: inside its function its own name is the fixture it
extends (the built-in, or an earlier fixture of that name), and tests,
hooks, and other fixtures that name it receive what it hands to use().
base.extend({ device: async ({ device }, use) => {
await device.setPermission('camera', 'deny');
await use(device);
} })
- A plain function takes the scope the extended fixture has on the
target: device is a worker fixture on mobile, so its override runs once
per worker and reaches beforeAll; app is a test fixture on web and a
worker fixture on mobile. A declared scope that differs is a
COLLECTION_ERROR naming both.
- A worker-scoped override receives what a worker fixture receives for
that built-in, the forwarder to the attempt that is running.
- On a target without the built-in nothing fails until code that runs
there names it; then the target has no such fixture.
- Only user code sees the override: the attempt's own built-ins, which the
runner and the worker fixtures' forwarders read, are untouched.
- Types: the override keeps the extended fixture's type and scope, through
an extend() overload that takes overrides without type arguments.
A fixture named like a core fixture, an engine's, or an earlier one of the
chain was refused before.
Refs ENG-1297
* docs(fixtures): name built-in fixtures and give their scope per engine
The docs now use one term, built-in fixtures, for agent, app, screen,
platform, session, and the ones an engine contributes (browser, device),
where pages said core fixtures, contributed fixtures, or engine fixtures.
The test reference gets one table with each built-in fixture's scope on
web and on mobile, and says what the scope decides: a test fixture reaches
test bodies, beforeEach, and afterEach; a worker fixture also reaches
beforeAll, afterAll, and worker fixtures. The hooks, worker fixtures, and
override sections link to it, as do the fixtures guide and the engine
pages. The Japanese and Chinese pages carry the table and the same terms.
Refs ENG-1297
* docs(ja): translate the fixture scope and on-demand changes
* docs(zh): translate the fixture scope and on-demand changes
* fix(e2e): let a worker fixture name an extended app or screen
A fixture that extends `app` or `screen` was taken for a test fixture when
its test.extend() call was checked, before any target is known. A worker
fixture declared after it that names `app` then failed at import on every
target, though the same fixture is accepted without the override and the
plan resolves both to worker scope on an engine whose state outlives a
test.
The scope of a plain function is now unresolved until a target is known,
whatever it extends, as it already was for an engine's own fixtures.
Refs ENG-1297
* fix(e2e): fail a suite hook that left an engine step running
On an engine whose state outlives a test, a beforeAll or afterAll hook, or
a worker fixture's teardown, can start a device step and not await it. The
hook's attempt then closed under the step, and its failure arrived later
as a stray rejection charged to whichever test ran then.
The suite attempt now does what a test attempt does once its body settled:
the step is failed at its call and cancelled, its outcome is awaited within
the cleanup budget, and the hook fails with STEP_NOT_AWAITED (HOOK_FAILED
for a suite hook).
The suite attempt keeps no step records, and StepRecorder dropped the
record of the step it had just opened before marking it running, so there
was nothing to abandon: a step now counts as running before the retained
window is trimmed.
Refs ENG-1297
* refactor(e2e): keep the fixtures a function names beside it
Registration parsed each test body, hook, and fixture function for the
fixtures it names, dropped the result, and left it in a WeakMap that the
planner and the pool read back with placeholder subjects. A cache miss
there would have named the wrong code and no line.
The parse result is now data: `names` on the registered test, the hook,
and the fixture definition. The planner and the pool read it, the WeakMap
and the per-function wrappers are gone, and one describeUser() gives an
error its subject. The explore test, registered by hand, parses its body
where it builds the record.
Unit tests state the names instead of generating source text for them.
Refs ENG-1297
* refactor(core): resolve built-in fixtures in one place
The core table declared the core fixtures, but what a target's built-in
fixtures are was still assembled around it: core names plus engine names in
three files, the 'engine' scope resolved in e2e through a cast into the
table, `platform` hard-coded as the one fixture that needs no attempt, and
two structural copies of the target.
core now exports builtInFixtures(target): names, scope(name),
needsAttempt(name), attemptless(name), and the unknown-fixture message,
over one BuiltInTarget. `platform` declares in the table that the target
alone decides it. launch, the planner, and the pool read from it.
DECLARED_BUILT_INS is the same view before a target is known: a scope the
engine decides is 'unresolved'. The planner takes either one, so its
no-target mode and the `target === undefined` branches through scopeOf and
read are gone. The scope of a fixture that extends `app` before a target
is known is 'unresolved' by construction.
Refs ENG-1297
* refactor(e2e): hand each function the fixtures it names
Every test body, hook, and fixture function must destructure its first
parameter, so what it reads is known before it runs. bindFixtures still
built a Proxy-gated bag for arbitrary property reads: one value in three
stores, the gate run twice (once over a faked plan), a third derivation of
scope, and an own-name shadow that re-derived what the planner had resolved.
No test code could reach the gate's throwing branches; only unit tests did,
by reading the bag directly.
The plan now records, per function, what each name it reads resolved to: a
chain fixture or a built-in. The pool hands each function a plain object
with exactly those names, the value a chain fixture handed over, or a
built-in read off the attempt when the function destructures it, so
built-ins stay lazy. An override that names itself receives the fixture it
extends because that is what its own name resolved to.
- bound.call(user) replaces user.fn(bound.fixtures); ready(user) asks
whether every chain fixture it names was provided.
- attemptlessBuiltIns is a function from a name to a value that throws the
reason for a built-in that needs an attempt.
- The planner runs once: it records what each fixture names, and the order
is one declaration-order walk over that record. A second validating pass
only reordered.
Behaviour: a bound or native function, whose parameters cannot be read, is
a COLLECTION_ERROR; it used to parse as naming nothing and still got lazy
built-ins. The docs no longer promise hooks and the body one shared object.
Refs ENG-1297
* refactor(e2e): move the suite attempt out of TargetExecutor
dispose, openSuiteHooks, and openSuiteAttempt were 177 lines of a second
attempt lifecycle inside TargetExecutor, written as closures in one method.
run/suite-attempt.ts now owns it: a SuiteAttempt class with open, acquire,
within, and close, over a SuiteAttemptHost interface the executor satisfies
(the way serial.ts borrows it through SerialHost), plus the worker
fixtures' teardown pass. It is the realm's SuiteHookSession itself, so the
executor keeps a context object and two call sites.
The three branches that built an attemptless reader with a "not available
to the <label>" error are one NoAttempt value ('isolated', 'unnamed',
'interrupted', or the launch failure) and one mapper to the message.
execute.ts: 1244 lines to 1052.
Refs ENG-1297
* refactor(e2e): share the attempt's fixture sequence
A run's attempt and an `e2e mcp` session's test each planned the test's
fixtures, bound them, entered the worker pool, set them up, ran the
beforeEach hooks and the body, filtered the afterEach hooks that can still
run, and labelled the teardowns. This PR widened that copy, and
standalone-test.ts asked the realm for the same hooks four times.
planAttemptFixtures(target, test, hooks) plans before any session exists
and returns open(), which binds the plan to the attempt: enter(), run()
for setup, hooks, and body, and cleanup() for the labelled afterEach hooks
and teardowns. Both call sites keep their own phases, budgets, and error
handling around it.
execute.ts: 1052 lines to 1031, its size on main.
Refs ENG-1297
* refactor(e2e): open the suite-hook pass without a lazy thunk
withSuiteHooks handed its work a thunk that opened the pass on first use.
Opening one does no I/O: the suite attempt launches nothing until a hook
that names an engine fixture runs. The pass is now opened up front and
closed in finally, and the `opened` slot and the `hooks()` call form go.
Refs ENG-1297
* refactor(e2e): ask the registry which hooks apply to a test
The selection-time fixture check rebuilt the scope chain of a test and
filtered the file's hooks by it, which the realm already did its own way.
registry.ts now exports scopeChain(test) and hooksFor(registration, test),
and both use them.
Refs ENG-1297
* refactor(core): move the fixture types into types/fixtures.ts
types.ts grew from 1679 to 1798 lines in this PR. The fixture scope, the
worker fixtures, and what test.extend() takes (definitions and overrides)
are one coherent group that nothing else in the file defines against, so
they move to types/fixtures.ts and types.ts re-exports them. The public
surface is the same.
types.ts: 1798 lines to 1715.
Refs ENG-1297
* refactor(core): drop the redundant override branch in FixtureDefinitions
A fixture that extends one the base already has was typed three ways: a
conditional branch inside the `Extra` map of FixtureDefinitions, the
`& FixtureOverrides` intersection, and the extend(overrides) overload.
Compiling the type tests without each in turn: the intersection types an
override written beside new fixtures with type arguments, and the overload
types an override-only call without them; both fail the tests when
removed. The conditional branch does not: with it gone the tests pass, and
a base name in `Extra` is `never` again, as it was before overrides. It is
removed, and the reference's copy of the declaration follows.
Refs ENG-1297
* refactor(e2e): split fixture errors from the parser, share hookLog
fixture-names.ts parses a parameter list, and also held FixtureUseError and
declarationSite, which format an error. They move to fixture-errors.ts.
Its identifier pattern held two raw zero-width characters; they are
written as escapes now.
hookLog and LOG_MODULE were pasted into three integration tests; they are
one helper in tests/helpers/hook-log.ts.
Refs ENG-1297
* docs(e2e): name the scope rule in the selection check's summary
The summary of assertFixtureUse still listed a chain fixture named like an
engine's as an error. Such a fixture extends the built-in now; the rule
left is a declared scope that differs from the extended fixture's.
Refs ENG-1297
* refactor(e2e): take execute.ts under 1000 lines
Two more pieces that are not the attempt's own sequence leave the file.
- Holding steps for `e2e run --step`: reading the screen a hold starts on
and wiring AttemptHolds to the runner's hook move to step-hold.ts, beside
the holds they configure (holdAttemptSteps, readHeldScreen).
- The result of a finished test or setup unit is a pure function of the
pair, the verdict, the attempts, and a beforeAll failure: unitResult in
units.ts, beside pairResult.
execute.ts: 1031 lines to 956. It was 1031 on main and 1226 at the review.
Refs ENG-1297
* test(e2e): match the not-awaited message without a step label
Main's #1880 names an unlabeled step by its api alone ("gadget.hang
finished"), where the message used to quote an empty label. The suite-hook
test's pattern required a label between the two, so it failed on the merge
with main.
* feat(e2e): an override inherits auto from the fixture it extends
A fixture that extends another and does not state `auto` now has the
extended fixture's, the way it has its scope: an override of an auto
fixture is set up for every test, and one of a built-in fixture on demand.
An `auto` that differs from the extended fixture's is a COLLECTION_ERROR
naming both, worded like the scope mismatch. This matches Playwright.
It also fixes which fixtures the plan starts from. An auto fixture was a
root by its own flag, so with an auto base and its override in one chain
the base was set up and the override skipped, while the name resolved to
the override; an override that replaced the base without naming it still
left the base running. An auto fixture is now a root only where its name
stands for it: the base runs once through the override's own name, or not
at all when the override replaces it.
Refs ENG-1297
* fix(core): name the hook or fixture that left a step running
STEP_NOT_AWAITED said "the test body returned before ..." for a step a
beforeAll, afterAll or beforeEach hook, a fixture's setup or a worker
fixture's teardown left running. The recorder now keeps the owner each
step was started under, and the runner says which code runs: the pool
for fixtures, hooks and the body, the suite attempt for suite hooks, and
a hosted session for itself.
---------
Co-authored-by: Oskar Kwaśniewski <oskar@okwasniewski.com>
* refactor: move e2e's core into @e2e-dev/core, and engines peer on it
The engine contract, locators, agent, replay cache, secrets, and the
session half of a worker move from packages/e2e to packages/core, the new
@e2e-dev/core package, at the same paths. e2e depends on it at its exact
version (a changesets fixed group) and reaches its modules through
@e2e-dev/core/lib/*, which is not public API; e2e/engine and e2e/agent
re-export it. @e2e-dev/web and @e2e-dev/mobile peer on @e2e-dev/core and
import the contract from @e2e-dev/core/engine. Install smoke checks that one
copy of @e2e-dev/core is installed.
Most of this diff is git mv and import specifiers rewritten to
@e2e-dev/core/lib/<path>; no behavior changes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* ci: build and pack @e2e-dev/core where e2e is built or packed alone
The Bun job built e2e before the @e2e-dev/core it compiles against, and the
examples nightly installs e2e from a tarball whose exact @e2e-dev/core is
not on npm; it now packs @e2e-dev/core and overrides it in each example.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* docs(engines): name @e2e-dev/core/engine as the contract's source
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(e2e): list @e2e-dev/core beside an engine in init and upgrade
Engines peer on @e2e-dev/core. Yarn installs no peer the project does
not list, and under Plug'n'Play the engine then fails to load.
- init adds @e2e-dev/core with any engine
- upgrade adds a required @e2e-dev peer the project lacks, and keeps a
package e2e depends on at the version e2e names
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Oskar Kwaśniewski <oskar@okwasniewski.com>
* docs: retire sidebar page icons
The docs sidebar no longer shows icons. Drop icon: from every page's frontmatter, delete the grey silhouettes in docs/images/icons/, and make check-docs-logos fail on any frontmatter icon instead of requiring one on vendor pages. Card logos stay required. The authoring-docs skill trades its Page icons section for a short Vendor logos one.
* docs: require a migrate overview card for each migration guide
* docs: match only top-level icon keys in check-docs-logos
* feat(engine): engine protocol for out-of-process engines
The engine contract as LSP-style JSON-RPC: connectEngine turns a server
into an engine handle, serveEngine serves any defineEngine handle, and
spawn, socket, and loopback transports reach it. Abort reasons, errors,
and pixels cross intact; the manifest has a JSON schema for servers in
other languages. E2E_ENGINE_LOOPBACK=1 runs every engine through the
protocol in process; the testbed runs that way in CI.
* feat(engine): run every in-process engine over the protocol, drop E2E_ENGINE_LOOPBACK
* refactor(engine): drive engine protocol from one method table
- one METHODS table feeds describeEngine, server and client
- client builds the engine from the manifest, defineEngine validates it
- harness holds only protocol clients (EngineClientHandle)
- signals scoped to attempts, drop $/release
- worker connections queue for MCP session handover
- track attempts for servers without startAttempt
- locate/stopVideo return arrays, keyboard/type takes options
* feat(engine): let connectEngine answer session hooks on the client
Host-side relaunch (simctl) without wrapping the client in defineEngine,
which made the harness serve it over loopback a second time.
* docs(engine): group engine docs into one Engines section
New Engines overview and Engine servers guide; the protocol reference
keeps signatures, messages and errors.
* docs(engine): complete config examples, drop leftover OG card
* fix(engine): harden protocol from review
- bound finish/dispose hooks by their budget, always close transport
- late endAttempt retires only its own attempt (client and server)
- reject malformed error responses instead of throwing from stream
- resolve a secret once; server derives redacted forms via attempt/deriveSecret
- carry error stack, errno, and causes across the wire
- serveEngine always answers startAttempt/endAttempt, so signals are freed without hooks
- detach signal listeners when a connection closes
- refuse a server speaking another protocol version
- docs: platform in connectEngine examples, message and error updates
* fix(engine): fixed diagnostic for malformed error response
* feat(e2e): record one span of the screen with app.startRecording and app.stopRecording
* docs(e2e): name the video modes app.startRecording refuses under
* docs(reference): rewrite the API reference in the Next.js shape
Every reference page now opens with a lede and a minimal example, groups its
API under ## sections that start with a summary table linking to one ###
entry per member, and keeps the TypeScript declarations under ## Types.
* docs(reference): fix claims the review caught
Narrow the security summary rows to their sections, generate the screenshot
file name in the engine example instead of building it from the label, and
correct defaults, exit codes, and examples on config, cli, screen, web,
mobile, test, runner, and engine.
* docs(reference): address review findings
Scope error classes to runner errors, guard a zero-slot prepare in the engine
example, keep the cli #help-and-version anchor, and correct poll cancellation,
same-run screenshots, openLink schemes, type_secret, longPress durations,
runner signals and close failures, empty usernames, and the feedback exit
code. Make the reporter and extract examples self-contained.
* docs(reference): trim the summary tables to the minimum
Each group's summary table keeps the linked name and one or two short
columns (type, default, or a few-word description). Examples and details
live in the entries; facts that only a table cell held moved into its entry.
The expect overview is headed Reference instead of Entry points, and the
security rules table names what each section covers instead of restating
its claims.
* docs(reference): open every page's API under one Reference heading
Each page's first section after the lede is now ## Reference. A page with
one group lists its entries at ### right under it; a page with several keeps
each group as a ### with its summary table and its entries at ####.
Explanatory sections (behavior, budgets, setup, types) follow it. Linked
sub-parts that moved to ##### keep their anchors through explicit ids.
* docs(skills): describe the reference page shape in authoring-docs
Reference pages open their API under ## Reference, keep summary tables to
the name and one or two short columns, and nest entries at ### or ####
because Mintlify drops anchors below ####. ship-pr points reference edits
at these rules instead of the guide shortening pass.
* docs(reference): address cubic review on tables and skill
Keep readonly in the mobile device type, show the keyboard and mouse
object shapes, say a locator assertion times out as ASSERTION_FAILED, and
give reference pages a narrower shortening pass in authoring-docs.
* docs(reference): tighten the reference pages
Merge facts said twice into one place, move each declaration's JSDoc to a
single home, link timeout defaults, key grammar, and secure-field rules to
the pages that own them, and split long sentences. No fact is cut.
* docs(reference): address cubic review of the tightening pass
Qualify the GitHub token for fork pull requests, provider recording URLs,
duplicate device entries, MODEL_UNAVAILABLE timing, the APP_UNREACHABLE log
quote, UNKNOWN_TOOL versus UNSUPPORTED_CAPABILITY, node-less agent tools,
which model receives screenshots, passing assert evidence, and config-path
hosts; fix three garbled sentences.
* docs(reference): say device, not simulator, for worker isolation
* feat(engine): carry a measured keyboard state from the engine to executors
EngineSnapshot and ExecutorObservation gain an optional keyboardVisible:
whether an on-screen keyboard is showing, as the engine measured it apart
from the tree. false means it measured none; the field is omitted only
when the engine did not measure it. The session copies it onto the
semantic observation and the observation feed hands it to the executor, so
an executor no longer has to infer the keyboard from keyboard and key
nodes that a platform may list only in part.
It is a boolean, not the producer's band: dismissing the keyboard only
needs to know whether one is up, and the band's frame is in the producer's
own coordinate space.
* feat(mobile): report the keyboard band agent-device measured on each observation
observe() sets keyboardVisible from the capture's keyboard fact: true for a
visible band, false for a proven absence, and nothing when the producer
could not measure it or published nothing. agent-device's iOS runner
publishes the fact with every capture; Android captures carry none, so
Android observations leave the field out.
The gesture fallback for dismissing the keyboard reads the same fact
through the same helper.
* feat(decision): offer dismiss_keyboard from the measured keyboard state
The action space reads the observation's keyboardVisible when the engine
measured one and falls back to the tree's keyboard and key nodes when it
did not. A keyboard the tree leaves out still offers the control, and key
nodes left behind after the keyboard closed no longer do.
The page fingerprint hashes the measured state beside the pixels, so a
dismissal that closes the keyboard without changing the tree no longer
reads as unchanged and cannot complete the three-action stall streak. An
unmeasured state hashes apart from a measured false.
* refactor: address review on measured keyboard state
- fingerprint takes the observation
- keyboardVisible checks use === undefined in every layer
- trim docs to one user-facing sentence, drop engine.mdx paragraph
- merge changesets into one patch changeset
---------
Co-authored-by: Oskar Kwaśniewski <oskar@okwasniewski.com>
* fix(mobile): bound each operation by its budget
The mobile engine bounded agent-device calls only by the operation's
signal, and the harness aborts that signal with the attempt, not at
timeoutMs. A device command that outlived actionTimeout held the step until
the test timeout. Each engine entry now runs under withinBudget, as the web
engine bounds its own: the call sees a signal that also aborts at
timeoutMs, whatever it throws after that is the budget's OPERATION_TIMEOUT,
and the timer ends with the call. Input verbs report the timeout as
ACTION_MAY_HAVE_COMMITTED so the runner never repeats a tap that may have
landed. The OperationContext.signal docs now say it does not abort at
timeoutMs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(mobile): bound inputs only, each device command under the budget left
Bounding every entry broke a locator poll: its last snapshot, cut off at
the deadline, read as a timeout instead of a missing node, and an abandoned
read overlapped the next command. Only the device commands that send input
now run under the budget, through InputBudget.send: with no budget left an
input is not sent (OPERATION_TIMEOUT), and one that does not answer in time
is ACTION_MAY_HAVE_COMMITTED. Reads, including the relocation before a tap,
stay unbounded. An input sent with almost no budget left still gets 250 ms
to answer, and each command of a multi-command input is sent on its own, so
no keystroke follows a focus tap that used the budget up.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(mobile): refuse a hold the budget cannot cover, and document inputs that time out
A long press or timed swipe that holds at least as long as the budget left
could only be cut off mid-gesture, so it is refused before it is sent, as
NOT_ACTIONABLE naming the hold and the time left. An input counts as sent
only once its command was dispatched, so one that throws first is not. The
mobile guide says what an input that outlives its timeout does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Every engine minor re-keyed all committed recordings, so each `chore:
version packages` merge failed the strict web agent job with REPLAY_STALE
on main (#825 after #774). The key keeps the engine's name and SPI
version; a release of the same engine is caught at replay, where each
action re-finds its node. The 30 web benchmark entries move to their new
hashes by a re-key verified against the old ones.
* fix(secrets): rewrite traces and text downloads whenever the run has a secret value
A registered value a test passed as a plain string (an app.open URL, a fill)
was redacted from titles, labels, and agent input but kept verbatim in
trace.zip: trace and download rewriting was gated on the exposure level,
which only a fill or an engine-held secret raises. Rewriting now depends on
the session ledger holding a value. The engine exposure level had no other
consumer and is removed; exposure now only decides pixels and saved-session
taint.
* fix(secrets): cover a plain-string fill, scope docs to the session's known values
* docs(secrets): download redaction is whole-value; TRACE_WITHHELD outside-path case
* docs(secrets): name the download rewrite failure path
* fix(security): allow only http(s) navigation, refuse wrapped schemes on device links
view-source:file:// passed the file:/data:/javascript: denylist and loaded local files via app.open, browser.goto, and the navigate verb. resolveNavigationUrl now admits http: and https: only; mobile links also refuse view-source:, blob:, filesystem:.
* fix(security): admit exact about:blank, sync rule text in AGENTS.md and SECURITY.md
* fix(web): refuse about:blank cookie URL, qualify scheme rule text
* fix(expect): matcher options are honored or rejected, never silently ignored
* refactor(web): reuse core rejectUnknownOptions, presence toHaveAttribute reads third arg
- e2e/engine exports rejectUnknownOptions; web drops its timeout-only copy
- toHaveAttribute(name, undefined, options) validates and times out on options
- waitForURL waitUntil refusal tested and noted in migration table
* docs(engine): rejectUnknownOptions takes code, options must be plain object
* fix(expect): toHaveAttribute refuses a null value or options; tests use the @e2e-dev scope
* fix(expect): use the browser fixture name after the web rename
* feat!: the app is declared on its target; engines only drive it
The app under test moves from the engine's options to `targets: [{ engine,
app }]`: `url`, `bundleId`, `appPath`, `identity`, `environment`,
`launchArguments`, `permissions`, `command`, and `readyUrl`. `web()` and
`mobile()` refuse every moved option in one `INVALID_CONFIG` naming the
target block; `defineEngine({ app })` is gone and engines check what their
platform needs in `validateApp`. `requires: ['browser' | 'native-app']` is
derived from the target's app. Replay cache and session keys are byte for
byte what they were, pinned by target-identity.test.ts.
Dependency services are not part of this version: `services` anywhere, and
`web({ services })`, fail saying so. `app.command` on port 0 without a
command, and `reuseExisting` on a free port, are refused. An `e2e mcp`
open cancelled or disconnected mid-start stops what it started, and a hung
shared start is abandoned once no session waits on it.
* fix: review follow-ups: cancelled opens, abandoned starts, null targets, stale docs
* refactor: drop the moved-option messages; old app keys are unknown keys
* test: give the replaced-nodes suite a budget that survives a loaded runner
* test: size the replaced-nodes hook budget to its seven tests
* test(skill): fail a skill or docs paragraph that passes the removed --artifacts
* fix(config): name the replacement for a removed createAgent import
* fix(config): keep the code of a config error raised while the config evaluates
* fix(secrets): refuse a secrets.get() reference where a string is needed
* fix(config): reject unknown keys in engine, command, test, and query options
* fix(config): refuse a tests entry naming a directory, name the ! entry excluding a positional
* refactor(config): derive option key lists from their types
* fix(config): one shared unknown-key check; drop undefined env values
Setup and describe option keys typed from their interfaces; a directory tests entry is INVALID_GLOB like a trailing slash.
* fix(collect): name the excluding ! entry when exclusions left nothing, and in on-disk casing
* fix(config): address review - exclusion hint for bare names, array env, qualified hints
* test(select): use a missing tmp root, not /root, which CI cannot stat
* fix(mobile): accept videoTouches, added on main by #671, in the mobile() key list
* fix(mobile): reject unknown permission names and states in mobile() and device.openApp
Names and states are checked against DevicePermission and PermissionState
through satisfies-typed key lists. INVALID_CONFIG at load, INVALID_ARGUMENT
from device.openApp, before any device command. rejectUnknownKeys takes an
optional code for the test-time case.
* fix(eas): reject unknown easSimulators() options with INVALID_CONFIG
Its options are its own, mapped onto EAS's create input, not a passed-through
SDK body, so a typo such as buildID was dropped silently. Its two other option
refusals are INVALID_CONFIG too. e2e is now a peer, for e2e/engine.
* fix(mobile): require a plain object for permissions; clarify eas and permission notes
A Map passed the object check with no own keys, and the preset applied none
of it.
* docs: fix stale and wrong claims in guides
- output: --artifacts -> --output in bug-bash guide and skill; sessions and
cache locations follow output / cache.dir in security; artifact path
gains the agent segment in github
- ci: upload artifacts on !cancelled() in ci guide and skill
- mcp: several sessions (--max-sessions) in security
- replay cache naming: "Commit the replay cache" heading and anchors
(old anchor kept, e2e init links the new one),
trace replay -> replay cache in writing-an-engine and executors
- timeouts: STEP_TIMEOUT / STEP_BUDGET_EXHAUSTED advice names config
timeout, judgmentTimeout, maxSteps, maxModelCalls (debugging, skill)
- mobile: app.open() in device examples and quickstart, mobileTools pack
all mutating, Android toolbar anchor, drop agent-device release note
- executors: verdict code rules, missing ctx members and scrollUntil,
a replay cache section; agent-steps passed+code rule
- migrate: swipe direction is the content's, `!` test exclusions,
video row status, Kernel for Selenium Grid
- secrets: an unset env string fails config load, a provider's value is
redacted only from its first fill; three secret sinks
- misc: Models link text, telemetry links outbound connections, integration
config wording, web-only expect(web) in a cross-platform block,
capabilities heading in engine reference, swipe direction in screen reference
* docs(skill): separate the credential redaction paragraph
* docs(reference): KERNEL_API_KEY names @e2e-dev/kernel
* feat(config)!: remove specVersion from the config
The runner version is the format version. A config that still sets
specVersion fails INVALID_CONFIG telling the author to remove it. The
report's run.specVersion and the replay cache key keep their hard-coded
copies.
* feat(collect): "!" exclusions in the tests globs
An entry starting with "!" removes files an including glob matched,
in any order. Discovery never reads a directory an exclusion takes
whole, unless an including glob could reach a dot name beneath it.
A list of only exclusions is INVALID_CONFIG.
* refactor!: CacheStore and web({ screencast }) renames
TraceCacheStore is CacheStore, and the docs, skill, and CLI call the
cache the replay cache. web({ video: { size, quality } }) is
web({ screencast }) with WebScreencastOptions; web({ video }) fails
INVALID_CONFIG naming screencast.
* feat(secrets): secrets.get() in engine options, resolved per attempt
At config evaluation, before any run exists, secrets.get(name) returns
a reference by name. An engine declares the handles its options hold
in defineEngine({ secrets }); the config load fails INVALID_CONFIG for
a name no secret or credential has. The engine reads the value through
EngineAttemptContext.resolveSecret, which runs a provider fresh and
registers the value in the session and process ledgers before it
returns, without tainting the viewport. e2e/engine exports isSecret.
web({ basicAuth: { password } }) takes a Secret and declares it. Docs
show process.loadEnvFile('.env') in the config.
* feat(run)!: trace and video share recording modes; artifacts holds only store
One RecordingMode ('off' | 'on' | 'retain-on-failure' | 'on-first-retry' |
'on-all-retries') for trace and video, set on the config, a target, the CLI
(--trace [mode], --video [mode]) and a test, one precedence rule: config,
target, CLI, test. Trace defaults to on locally, on-first-retry in CI, with
a plan-time notice when a retry mode meets retries: 0. Serial members cannot
set trace.
Capability rule: a mode from the config root or a flag applies to targets
whose engine can record it, one notice naming the rest; a mode set on a
target or a test is required and fails with UNSUPPORTED_ARTIFACT.
artifacts.kinds (both forms) and artifacts.trace are removed and fail
INVALID_CONFIG naming the trace mode they meant. Failure screenshot and
screen text are always captured when the engine can. MCP sessions keep the
target's trace mode and trace only under 'on'.
* feat(run)!: output directory for every run result; ArtifactStore.putLink
output (default .e2e) and --output <dir> on run and explore hold report.json,
junit.xml, summary.md, ai-trace.json, artifacts/, sessions/ and the e2e mcp
session files; --last-failed reads <output>/report.json. The reportSibling
derivation from the artifacts dir is gone. --artifacts stays as a hidden
flag that fails with the --output it maps to.
A run clears <output>/artifacts at start. cache.dir stays independent. An
output that is the project root or outside it, is or sits in cache.dir,
would hold cache.dir under artifacts/, sessions/ or videos/, or holds a
tests glob root fails INVALID_CONFIG with the reason.
ArtifactStore gains optional putLink(link) for provider-hosted video links;
its ref lands beside the url. A passed retain-on-failure attempt's link is
never handed over, and a failing putLink never fails the run.
bench-ab and the mobile workflow move to --output.
* test(secrets): a hold's registry is probed by an unknown name, now that get() defers without one
* test(web): the download trigger's measured wait tolerates a timer firing early
* fix(secrets): rewrite the trace of an attempt whose engine holds a secret
A Playwright trace records the context options, basic-auth password in
plaintext, but a trace was rewritten only after a fill. resolveSecret now
marks the session so its trace goes through the redactor, screencast
frames kept since the value never reached the screen. The fake engine
writes a real trace archive, and the basic-auth fixture escapes its echo
and refuses a malformed header.
* fix(web): race the basic-auth secret with the attempt's signal
A provider still resolving after the launch timeout set the session after
the runner ended the attempt, so the retry failed with an attempt already
running. Resolution now runs first, raced with the signal.
* fix(mcp): a session notes a configured trace its engine cannot record
* test: a shadow-free fake trace archive and an ungarbled exclusion title
* docs: output paths, replay cache wording, and review corrections
--last-failed, --ai-trace, and the json reporter name <output> paths;
the skill says output never moves cache/ or the app's log; the
INVALID_GLOB row states the check the code makes; serial members list
only; the engine vocabulary names Secret and isSecret; the basic-auth
example uses a provider; the last trace cache wording is replay cache.
* chore(changeset): leave released notes alone; pending notes match the renames
Released CHANGELOG entries keep TraceCacheStore, the name they shipped
with; cache-store-rename.md announces the rename. The pending video and
provider recording changesets name on-all-retries and web({ screencast }).
* chore(bench): refuse a forwarded --output the harness would not clean up
* fix(secrets): a rewritten trace drops its screencast frames, engine-held secrets included
* feat(run)!: video modes on the config, targets, the CLI, and tests, with step captions
video: off | on | retain-on-failure | on-first-retry at the config root, per
target, --video [mode], and per test or describe, innermost winning.
artifacts.kinds: ['video'] and artifacts.video are removed. Kept recordings
get WebVTT step captions; a video can be a link to a hosted recording. The
engine contract's VideoSegment is a file or a link, and e2e/engine exports
the provider recording seam.
* feat(web): a browser provider records the attempts it serves, and the screencast takes a size and quality
BrowserProvider.record starts the service's own recording of the browser in
place of the page screencast. The session drives one AttemptVideo, the
screencast or the provider's. web({ video: { size, quality } }) tunes the
screencast.
* feat(mobile): a device provider records the attempts it serves
DeviceProvider.record starts the service's own recording of the leased
device in place of agent-device's. The worker learns its slot's lease id,
so record gets the lease as it traveled.
* feat(integrations): Kernel replays record the attempts that record video
* fix(run): captions never fail a video, provider values are normalized, and one list of modes
- Step captions create their directory and are best-effort.
- A provider's recording is checked at start in both engines and its URL
and start time are stored in the shapes the report schema admits; the
list reporter bounds a hosted URL.
- --video no longer hides an invalid target mode.
- A describe inside a serial group cannot set video.
- A mobile provider may keep its own leaseId field on a lease.
* fix(mobile): keep the lease type narrowed when checking its declared fields
* fix(run): MCP recordings get their step captions, a step spanning a restart is captioned in both recordings, and video.size refuses unknown keys
* fix: address review on the video API
- One stop in flight per provider recording, shared by a retry; mobile
dispose stops it, and a cancelled operation never starts one.
- Kernel retries a failed download without stopping the replay twice.
- Links from any engine are held to an http(s) URL with a host; a file a
provider names must be a regular file; media types are trimmed.
- web({ video }) takes a plain object only; the markdown page labels a
hosted artifact by its kind.
- Tests: fixture byte totals, captions linked by id, temp directories
removed, signals asserted, empty sizes, non-object options.
- Docs: engine support qualified, the explore row, the standalone preflight,
caption status text, a checked stop in the provider example.
* fix(run): a bad engine video link costs no other recording, captions path per recording in debugging guide
* test: fake engine video links doc matches segment order
* fix(mcp): session recordings follow video modes and list a provider's link
* test(web): provider test's fake page reports its viewport
* refactor(run): drop step captions from the video API, tracked in #636
* fix(web): a late provider start never replaces a newer recording, abandon keeps a failed stop's handle, video.size must be a plain object; provider recording errors and hosted URLs bounded