mirror of
https://github.com/tester-army/e2e.git
synced 2026-11-22 07:55:36 +08:00
* fix(secrets): rewrite traces and text downloads whenever the run has a secret value A registered value a test passed as a plain string (an app.open URL, a fill) was redacted from titles, labels, and agent input but kept verbatim in trace.zip: trace and download rewriting was gated on the exposure level, which only a fill or an engine-held secret raises. Rewriting now depends on the session ledger holding a value. The engine exposure level had no other consumer and is removed; exposure now only decides pixels and saved-session taint. * fix(secrets): cover a plain-string fill, scope docs to the session's known values * docs(secrets): download redaction is whole-value; TRACE_WITHHELD outside-path case * docs(secrets): name the download rewrite failure path
1364 lines
63 KiB
Plaintext
1364 lines
63 KiB
Plaintext
---
|
|
title: e2e/engine
|
|
description: Build an engine with observations, actions, lifecycle hooks, and fixtures.
|
|
---
|
|
|
|
Import engine types and helpers from `e2e/engine`. `defineEngine` validates
|
|
and freezes your implementation, derives its capabilities, and returns a
|
|
handle for `targets` in the config.
|
|
|
|
```ts
|
|
import { defineEngine, type EngineHandle } from 'e2e/engine';
|
|
|
|
export function defineEngine(spec: Engine): EngineHandle;
|
|
export function isEngineHandle(value: unknown): value is EngineHandle;
|
|
```
|
|
|
|
An engine drives a user interface, such as a browser or simulator. It
|
|
implements shared types for nodes, locators, actions, and errors. Use
|
|
`test.extend` fixtures for resources outside the UI, such as APIs or databases.
|
|
|
|
Validation fails with `INVALID_CONFIG` for:
|
|
|
|
- a spec that is not an object, or a missing or empty `name`;
|
|
- a missing or empty `version`, or a `spiVersion` other than `1`;
|
|
- an empty `platform`, or a `workers` that is not a positive safe integer;
|
|
- a hook (`observe`, `locate`, `perform`, `performAt`, `prepare`, `finish`,
|
|
`init`, `startAttempt`, `settleAttempt`, `endAttempt`, `dispose`,
|
|
`validateApp`) that is not a function;
|
|
- an unknown top-level key on an object literal (a class instance's own
|
|
fields are not checked);
|
|
- `perform`, `locate`, `performAt`, or `keyboard` without `observe`;
|
|
- `perform` without `actions` or the reverse; an `actions` list that is
|
|
empty, repeats a kind, or names one outside `LOCATOR_ACTION_KINDS`;
|
|
- `performAt` without `pointerActions` or the reverse; a `pointerActions`
|
|
list that is empty, repeats a kind, or names one outside
|
|
`POINTER_ACTION_KINDS`;
|
|
- a `tapModifiers` that is not a boolean, or `true` without `perform`;
|
|
- a `keyboard`, `state`, `artifacts`, or `session` manifest with an unknown
|
|
key or a non-function member. `keyboard` needs `type` and `press`; `state`
|
|
needs both `capture` and `restore`; `artifacts` needs `screenshot`;
|
|
`startTrace`/`stopTrace` and `startVideo`/`stopVideo` come in pairs;
|
|
- a `secrets` that is not an array of `secrets.get()` handles;
|
|
- an `app` member: the app under test is declared on the target, and an
|
|
engine checks what it needs in `validateApp`;
|
|
- a `fixtures` that is not an object, a fixture that is not a factory
|
|
function, or a fixture name that is not a lower-camel identifier or shadows
|
|
`agent`, `app`, `screen`, `platform`, or `session`.
|
|
|
|
Every function on the handle is bound to the spec it came from, so a class
|
|
instance is as valid a body as an object literal.
|
|
|
|
## Engine
|
|
|
|
```ts
|
|
/** The body of one target: typed, model-free, capability-graded. */
|
|
export interface Engine {
|
|
/** Engine name, for diagnostics. */
|
|
readonly name: string;
|
|
/**
|
|
* Required implementation version, recorded in reports and cache keys.
|
|
* Cached traces cannot replay across engine versions.
|
|
*/
|
|
readonly version: string;
|
|
/**
|
|
* Contract version literal. Additive optional members never bump it;
|
|
* changed required semantics do.
|
|
*/
|
|
readonly spiVersion: EngineSpiVersion;
|
|
/**
|
|
* Platform label this engine drives (`web`, `ios`, `android`, or a label of
|
|
* the engine's own). Tests filter on it through `platforms` and tool packs
|
|
* are scoped by it; the harness attaches no meaning to any value. A target
|
|
* inherits it; a target that names a platform of its own must agree with it.
|
|
*/
|
|
readonly platform?: string;
|
|
/**
|
|
* Maximum workers for this target, such as the number of devices in a
|
|
* pool. The scheduler applies this cap even when `config.workers` is
|
|
* higher. Omit when the engine has no concurrency limit.
|
|
*/
|
|
readonly workers?: number;
|
|
/** capability: observation. */
|
|
observe?(context: OperationContext, options?: EngineObserveOptions): Promise<EngineSnapshot>;
|
|
/**
|
|
* Performs one action on a ref from the newest observation or `locate`,
|
|
* with the platform's actionability checks. Both agent tools and locator
|
|
* actions call this method. A `swipe` on the observation root swipes the
|
|
* viewport.
|
|
*
|
|
* For safe trace replay, throw a retryable
|
|
* `NODE_STALE`-coded error when a ref no longer binds, and an
|
|
* `ACTION_MAY_HAVE_COMMITTED`-coded error when input may have reached the
|
|
* app. The runner never repeats an action that may have committed.
|
|
*/
|
|
perform?(ref: NodeRef, action: LocatorAction, context: OperationContext): Promise<void>;
|
|
/**
|
|
* The action kinds `perform` honors, required with it. The harness offers
|
|
* the agent and the `screen` tier exactly these: a kind left out is never
|
|
* presented as a tool that can only decline, and `screen`/`Locator` methods
|
|
* for it fail with `UNSUPPORTED_CAPABILITY` before reaching the engine.
|
|
* `perform` still throws `UNSUPPORTED_CAPABILITY` for a declared kind one
|
|
* particular node cannot take (a `check` on a toggle whose state is unknown).
|
|
*/
|
|
readonly actions?: readonly LocatorActionKind[];
|
|
/**
|
|
* capability: location - requires observation. Deterministic locator
|
|
* resolution for the `screen`/`expect` tier: resolve one expression to the
|
|
* nodes it currently matches. The harness owns polling, strictness, and
|
|
* staleness; an engine resolves once, immediately.
|
|
*/
|
|
locate?(
|
|
expression: LocatorExpression,
|
|
context: OperationContext,
|
|
): Promise<readonly SemanticNode[]>;
|
|
/**
|
|
* capability: pointer - requires observation. Performs one pointer action
|
|
* at a viewport point, in the CSS pixels of `SemanticNode.rect`, with no
|
|
* node behind it: the agent's `tap_at` verb lands here when the point the
|
|
* model named in a screenshot sits on nothing the tree lists (a shape on a
|
|
* canvas, a pin on a map, a control in a system sheet), and so do a test's
|
|
* `screen.tapAt`, `screen.swipe({ from, to })`, and `tap({ position })`.
|
|
* Dispatch the pointer at the point as given; the harness has clamped an
|
|
* agent's point to the viewport and validated a test's. Throw
|
|
* `UNSUPPORTED_CAPABILITY` for a declared kind the surface cannot deliver
|
|
* at this particular point.
|
|
*/
|
|
performAt?(point: ViewportPoint, action: PointerAction, context: OperationContext): Promise<void>;
|
|
/**
|
|
* The pointer action kinds `performAt` honors, required with it. The
|
|
* harness routes a point-addressed action to the engine only for a kind
|
|
* listed here; an undeclared kind fails with `UNSUPPORTED_CAPABILITY`
|
|
* before reaching the engine.
|
|
*/
|
|
readonly pointerActions?: readonly PointerActionKind[];
|
|
/**
|
|
* The surface honors `modifiers` on the `tap`, `doubleTap`, and
|
|
* `secondaryTap` actions `perform` receives. Without it the harness fails
|
|
* an action that carries modifiers with `UNSUPPORTED_CAPABILITY` before it
|
|
* reaches the engine.
|
|
*/
|
|
readonly tapModifiers?: boolean;
|
|
/** capability: keyboard - requires observation. Input to whatever holds focus; see `EngineKeyboard`. */
|
|
readonly keyboard?: EngineKeyboard;
|
|
/**
|
|
* Named deterministic surfaces this engine contributes to TestFixtures
|
|
* (a device fixture, a document fixture - anything). Keys become fixture and
|
|
* capability names; `requires: ['<name>']` gates at selection.
|
|
*/
|
|
readonly fixtures?: Readonly<Record<string, EngineFixtureFactory>>;
|
|
/** capability: state - opaque snapshot capture/restore for session reuse. */
|
|
readonly state?: EngineStateCapability;
|
|
/** capability: artifacts - screenshots and traces under the attempt directory. */
|
|
readonly artifacts?: EngineArtifacts;
|
|
/**
|
|
* Checks the app a target declares against what this engine can drive, at
|
|
* config load, synchronously: throw `INVALID_CONFIG` naming
|
|
* `info.targetName` for a field the platform needs and the target left out
|
|
* (a URL, a bundle id) or one it cannot drive. The harness has already
|
|
* checked every field's shape.
|
|
*/
|
|
validateApp?(app: EngineAppDeclaration, info: EngineAppCheckInfo): void;
|
|
/**
|
|
* The `secrets.get()` handles the engine's options hold, such as a
|
|
* basic-auth password. The config load checks each names a configured
|
|
* secret (`INVALID_CONFIG` otherwise), and only these resolve through
|
|
* `EngineAttemptContext.resolveSecret`. The handles carry names only, so
|
|
* no value reaches the config digest.
|
|
*/
|
|
readonly secrets?: readonly Secret[];
|
|
/** Steering hooks: open, back, restart, reset. */
|
|
readonly session?: EngineSession;
|
|
/**
|
|
* Runs once per target in the runner, after collection and before workers,
|
|
* app commands, and the run's clock start. Download dependencies or boot
|
|
* devices here; report progress through `info.log`. No launch timeout
|
|
* applies. A failure ends the run as an infrastructure error.
|
|
* The result can lower the target's worker cap, for example after
|
|
* discovering how many devices are available.
|
|
*/
|
|
prepare?(info: EnginePrepareInfo): Promise<void | EnginePrepareResult>;
|
|
/**
|
|
* Once per run and target in the runner process, after every worker of the
|
|
* target is gone, on every exit path, whenever `prepare` was called.
|
|
* Release what `prepare` acquired for the run (a leased cloud device).
|
|
* Runs within the cleanup budget, never cancelled by an interrupt; a
|
|
* failure is a cleanup-phase run error.
|
|
*/
|
|
finish?(info: EngineFinishInfo): Promise<void>;
|
|
/**
|
|
* Once per worker, before the first step; boot devices here, not in a step
|
|
* budget. The same handle can be booted again after `dispose`: a config-held
|
|
* handle outlives an in-process worker, so init MUST work on a disposed
|
|
* engine as it does on a fresh one.
|
|
*/
|
|
init?(info: EngineInitInfo): Promise<void>;
|
|
/** Before each attempt: set up per-test isolation. */
|
|
startAttempt?(context: EngineAttemptContext): Promise<void>;
|
|
/**
|
|
* After the body has settled, and again after teardown, before the verdict,
|
|
* within the cleanup budget: report a failure that landed on a path no step
|
|
* awaited (a callback the surface ran for the test: a request interceptor,
|
|
* a native dialog handler). Wait for one still running, then throw its
|
|
* error; the attempt fails with it the way a step would, an error already
|
|
* classified keeping its code. MUST report each failure once.
|
|
*/
|
|
settleAttempt?(context: EngineCleanupContext): Promise<void>;
|
|
/**
|
|
* After each attempt, within the cleanup budget: tear that isolation down.
|
|
* MUST be idempotent, and safe to call after a failed `startAttempt`.
|
|
*/
|
|
endAttempt?(context: EngineCleanupContext): Promise<void>;
|
|
/**
|
|
* Worker shutdown, within the cleanup budget; failure is a run error. Runs
|
|
* whether or not `init` ran, so it MUST tolerate a cold engine.
|
|
*/
|
|
dispose?(context: EngineCleanupContext): Promise<void>;
|
|
}
|
|
```
|
|
|
|
```ts
|
|
/** What every engine call carries: budget, ids, and who is acting. */
|
|
export interface OperationContext {
|
|
/** Aborts with the attempt or when the operation budget is spent. */
|
|
readonly signal: AbortSignal;
|
|
/** Remaining operation budget when the call starts. */
|
|
readonly timeoutMs: number;
|
|
/** The run id. */
|
|
readonly runId: string;
|
|
/** The attempt id. */
|
|
readonly attemptId: string;
|
|
/**
|
|
* Who asked: a test's own deterministic step, or the agent acting on a
|
|
* screen it observed. A deterministic step verifies its outcome with
|
|
* `expect`, so an engine may act as soon as the target holds still; an
|
|
* agent reads the screen right after acting, so an engine may wait for the
|
|
* transition to end first.
|
|
*/
|
|
readonly origin: 'test' | 'agent';
|
|
}
|
|
```
|
|
|
|
### Capabilities
|
|
|
|
| Member | Capability | Requires | Unlocks |
|
|
| --- | --- | --- | --- |
|
|
| `observe` | `observation` | - | judgment tier, prompt snapshots, trace replay, the viewport swipe's root |
|
|
| `perform` + `actions` | `actions` | `observe` | the agent's grammar verbs (`tap`, `double_tap`, `long_press`, `right_click`, `hover`, `type`, `press`, `select`, `check`, `type_secret`, `scroll`, `scroll_to`, `drag`, `upload`, and `tap_at` or `hover_at` when the point lands on a listed node) and the `screen`/`Locator` action set with platform actionability checks, each only for a kind listed in `actions` |
|
|
| `locate` | `location` | `observe` | `screen`, `Locator`, `expect(locator)` |
|
|
| `tapModifiers: true` | - | `perform` | `modifiers` on `tap`, `doubleTap`, and `secondaryTap` (`row.click({ modifiers: ['Shift'] })`), held for the click; without it such a call is `UNSUPPORTED_CAPABILITY` |
|
|
| `performAt` + `pointerActions` | `pointer` | `observe` | the agent's `tap_at` and `hover_at` verbs when the point lands on nothing the tree lists (the `tap` and `hover` pointer kinds), and the test's `screen.tapAt` and `tap({ position })` (the `tap` kind) and `screen.swipe({ from, to })` (the `swipeTo` kind): one pointer action at a viewport point in the CSS pixels of `SemanticNode.rect`, with no node behind it |
|
|
| `keyboard.{type,press,dismiss?}` | `keyboard` | `observe` | the agent's `type` and `press` without a target, the keyboard fallback of `type_at` and `press_at` on a point the tree lists nothing at, and `dismiss_keyboard` when `dismiss` exists: input to whatever holds focus, for a field drawn on a canvas or flattened out of a platform's tree; with the `focus` action, `locator.pressSequentially`, which the harness composes as `focus` then `keyboard.type` |
|
|
| `fixtures.<name>` | `<name>` | - | a contributed fixture; `requires: ['<name>']` gates at selection |
|
|
| `state` | `state` | - | `test.setup` sessions and the `session:` option |
|
|
| `artifacts` | `artifacts` | - | `app.screenshot()`, assertion evidence, the failure screenshot, and every [`trace` and `video` mode](/reference/config#recording) but `off`, from the config, a target, a test, `--trace`, or `--video` |
|
|
| `validateApp` | - | - | a config-load check of the target's `app` |
|
|
| `session` | - | - | `app.open()` and the agent's `navigate` verb (`open`), `app.back()` and the agent's `back` verb (`back`), `app.restart()` (`restart`), `app.clearState()` (`reset`) |
|
|
|
|
The agent offers the model only what `actions` declares: `tap` needs the
|
|
`tap` kind, `double_tap`, `long_press`, and `right_click` need `doubleTap`,
|
|
`longPress`, and `secondaryTap`; `hover` needs `hover`; `type` and `type_secret` need `fill`;
|
|
`press` needs `press`; `select` needs `selectOption`; `check` needs both
|
|
`check` and `uncheck`, since it sets a state rather than flipping one;
|
|
`scroll` and `scroll_to({ text })` need `swipe`; `scroll_to({ target })` needs `scrollIntoView`; `drag` needs
|
|
`dragTo`; `upload` needs `setInputFiles`; `navigate` needs `session.open`;
|
|
`back` needs `session.back`; `tap_at` and `hover_at` need the `tap` or
|
|
`hover` kind in `actions` or in `pointerActions`. `clear` and `focus` unlock
|
|
no verb. An undeclared verb is absent from the toolset, and a
|
|
`screen`/`Locator` method for an undeclared kind fails with
|
|
`UNSUPPORTED_CAPABILITY` before reaching the engine.
|
|
|
|
`engine.capabilities` is the computed `ReadonlySet` of capability names, and
|
|
is what selection, the report, and `UNSUPPORTED_CAPABILITY` messages read.
|
|
Selection also grades a target by the kind of app it declares, not by its
|
|
engine: `browser` for a target with `app.url`, `native-app` for one with
|
|
`app.bundleId` or `app.appPath`, so a test can `requires: ['native-app']`.
|
|
|
|
```ts
|
|
/**
|
|
* Capability names: the closed harness capabilities plus one name per
|
|
* contributed fixture. `actions` is `perform`; `location` is `locate`; both
|
|
* require observation, because their refs live in the observation's id space.
|
|
* `pointer` is `performAt`, an action addressed by a viewport point rather
|
|
* than a node, and requires observation because the point is read off its
|
|
* pixels.
|
|
*/
|
|
export type EngineCapability =
|
|
| 'observation'
|
|
| 'actions'
|
|
| 'location'
|
|
| 'pointer'
|
|
| 'keyboard'
|
|
| 'state'
|
|
| 'artifacts'
|
|
| (string & {});
|
|
|
|
/**
|
|
* capability: keyboard - requires observation. Input to whatever holds focus,
|
|
* with no node behind it: the agent's `type` and `press` without a target, and
|
|
* the fallback of its point-addressed `type_at` and `press_at` when the point
|
|
* lands on nothing the tree lists (a field drawn on a canvas, an input a
|
|
* platform flattens out of its accessibility tree). Focus is the app's: the
|
|
* harness tapped first, or the model said the field already has it.
|
|
*/
|
|
export interface EngineKeyboard {
|
|
/**
|
|
* Types `text` into the focused field as a person typing would, through
|
|
* the platform's keyboard path: a browser dispatches the key events of each
|
|
* character, a device sends the text through its keyboard. Where the
|
|
* platform has key events the app must receive them; setting the value
|
|
* without keystrokes is not a compliant implementation, since
|
|
* `locator.pressSequentially` and the agent's `type` rely on the app
|
|
* reacting as it does to a user. With `replace`, clears the field first
|
|
* (select-all and delete on a keyboard surface; the platform's clear on a
|
|
* device). Throw `NOT_ACTIONABLE` when nothing that accepts text has focus,
|
|
* so the keystrokes are never silently discarded into the document body.
|
|
*/
|
|
type(text: string, options: { readonly replace: boolean }, context: OperationContext): Promise<void>;
|
|
/**
|
|
* Sends one key of the contract grammar (`Enter`, `Escape`, `Tab`,
|
|
* `Shift+Tab`, a character) to whatever holds focus. The harness has
|
|
* already validated the key's shape; refuse with `UNSUPPORTED_CAPABILITY`
|
|
* a key the surface cannot deliver.
|
|
*/
|
|
press(key: string, context: OperationContext): Promise<void>;
|
|
/** Hides an on-screen keyboard. Platforms without one leave it undefined. */
|
|
dismiss?(context: OperationContext): Promise<void>;
|
|
}
|
|
```
|
|
|
|
## Lifecycle
|
|
|
|
| Hook | Budget |
|
|
| --- | --- |
|
|
| `prepare(info)` | none; only `info.signal` (the run's interrupt) cuts it short. Failure ends the run before any test: a runner error class (a `ConfigurationError` for an option the engine cannot honour) keeps its code, an `EngineError` `CANCELLED` stays a cancellation, and anything else, any other `EngineError` or a plain `Error`, is infrastructure `ENGINE_FAILURE` |
|
|
| `finish(info)` | `cleanupTimeout`, shared by every prepared target, never cut short by an interrupt; failure is a cleanup run error |
|
|
| `init(info)` | `launchTimeout`; failure is infrastructure |
|
|
| `startAttempt(context)` | `launchTimeout`; on failure or timeout `context.signal` aborts and `endAttempt` runs |
|
|
| `settleAttempt(context)` | `cleanupTimeout`, after the body and again after teardown; a thrown error is the attempt's failure and keeps its code |
|
|
| `endAttempt(context)` | `cleanupTimeout`; failure marks cleanup failed, the test keeps its status |
|
|
| `dispose(context)` | `cleanupTimeout`; failure is a run error |
|
|
|
|
```ts
|
|
/**
|
|
* What `prepare` learned that the run must honour. `workers` replaces the
|
|
* engine's declared cap for this target and run: the surfaces the engine
|
|
* actually provisioned, `1` to `info.slots`. Declared `workers` stays the
|
|
* static answer for engines that know it up front.
|
|
*/
|
|
export interface EnginePrepareResult {
|
|
/** Lowers the target's worker cap for this run, `1` to `info.slots`. */
|
|
readonly workers?: number;
|
|
/**
|
|
* Variables every worker of this target is spawned with, on top of the
|
|
* run's environment: how a runner-side `prepare` hands what it provisioned
|
|
* (a device pool it discovered) to each worker's `init`. The run's own
|
|
* variables win on a clash.
|
|
*/
|
|
readonly env?: Readonly<Record<string, string>>;
|
|
}
|
|
|
|
/**
|
|
* Facts handed to `prepare`, once per run and target in the runner process,
|
|
* before any worker exists.
|
|
*/
|
|
export interface EnginePrepareInfo {
|
|
/** The run id. */
|
|
readonly runId: string;
|
|
/** The target being prepared. */
|
|
readonly targetName: string;
|
|
/** Directory relative paths in the config resolve against; see `EngineInitInfo.projectRoot`. */
|
|
readonly projectRoot: string;
|
|
/** The app the target declares, as `init` receives it: what a device installs or warms up here. */
|
|
readonly app: EngineAppInfo;
|
|
/**
|
|
* The worker slots the run will start for this target, `0` to `slots - 1`:
|
|
* the run's worker cap, the engine's declared `workers`, and the work units
|
|
* selected for the target, whichever is smallest; `0` when nothing runs on
|
|
* it. Provision for exactly these slots (boot that many devices of a pool);
|
|
* `init` then finds them ready.
|
|
*/
|
|
readonly slots: number;
|
|
/**
|
|
* The run's environment: what every worker is started with. A host may hand
|
|
* the run an environment other than the runner process's own, so anything
|
|
* provisioned here that a worker later looks up by environment (a browser
|
|
* cache location) must be resolved and spawned against this, not
|
|
* `process.env`.
|
|
*/
|
|
readonly env: Readonly<Record<string, string | undefined>>;
|
|
/**
|
|
* Aborts on interrupt only. Provisioning has no budget: a first-run
|
|
* download is as long as the network makes it, and cutting it short would
|
|
* fail every test behind it.
|
|
*/
|
|
readonly signal: AbortSignal;
|
|
/**
|
|
* Reports one line of progress. The runner streams it as a `notice` run
|
|
* event, so it reaches the reporter and every host sink instead of being
|
|
* written to a worker's stderr underneath the live status block.
|
|
*/
|
|
readonly log: (line: string) => void;
|
|
}
|
|
|
|
/** Handed to `finish`, once per run and target after the last worker is gone, with the cleanup budget of one hook. */
|
|
export interface EngineFinishInfo extends EngineCleanupContext {
|
|
/** The run id. */
|
|
readonly runId: string;
|
|
/** The target being finished. */
|
|
readonly targetName: string;
|
|
/** The run's environment, the same `prepare` saw. */
|
|
readonly env: Readonly<Record<string, string | undefined>>;
|
|
/** Reports one line of progress, streamed as a `notice` run event like `prepare`'s. */
|
|
readonly log: (line: string) => void;
|
|
}
|
|
|
|
/** Run identity and harness-resolved facts handed to `init`, once per worker before the first step. */
|
|
export interface EngineInitInfo {
|
|
/** The run id. */
|
|
readonly runId: string;
|
|
/** The target this worker serves. */
|
|
readonly targetName: string;
|
|
/**
|
|
* Directory relative paths in the config resolve against: the config file's
|
|
* directory, or the run's `cwd` for a programmatic config. An engine option
|
|
* naming a file (a build to install) resolves here, never against
|
|
* `process.cwd()`, which an in-process run does not change.
|
|
*/
|
|
readonly projectRoot: string;
|
|
/** The app the target declares: the site policy of its URL, and what a device launches. */
|
|
readonly app: EngineAppInfo;
|
|
/**
|
|
* This worker's environment: the run's, plus what this target's `prepare`
|
|
* returned in its result's `env`. Read what `prepare` provisioned from
|
|
* here, never from `process.env`: an in-process worker shares its process
|
|
* with the runner and every other target.
|
|
*/
|
|
readonly env: Readonly<Record<string, string | undefined>>;
|
|
/** Whether the run asked for a visible surface (`--headed`). */
|
|
readonly headed: boolean;
|
|
/**
|
|
* This worker's 0-based slot among the target's workers: the lowest slot
|
|
* free when the worker was spawned, so a replacement worker takes over the
|
|
* slot of the one that exited. An engine with several devices hands each
|
|
* slot its own; `config.workers` bounds the slots a target can reach.
|
|
*/
|
|
readonly workerSlot: number;
|
|
/** Aborts on interrupt and when init exceeds the launch timeout; init must stop promptly. */
|
|
readonly signal: AbortSignal;
|
|
/**
|
|
* Reports one line of progress. The runner streams it as a `notice` run
|
|
* event, prefixed with the target and worker slot, so it reaches the
|
|
* reporter and every host sink instead of being written to a worker's
|
|
* stderr underneath the live status block. This is where a fact that only
|
|
* exists once the worker is up belongs: the URL a hosted browser or device
|
|
* can be watched at, which simulator a slot got, a version it detected.
|
|
* Meant for `init` itself; a line logged later is still delivered while
|
|
* the worker lives and dropped once it has exited. The line is shown as is
|
|
* in reporters and host sinks, so keep tokens and other secrets out of it
|
|
* (a viewer URL without its credential).
|
|
*/
|
|
readonly log: (line: string) => void;
|
|
}
|
|
|
|
/**
|
|
* The app the harness resolved from the target's declaration, handed to
|
|
* `prepare` and `init`: the site policy for the decisions only a surface can
|
|
* make (which nested documents enter an observation, which requests carry
|
|
* injected credentials), and what a device launches.
|
|
*/
|
|
export interface EngineAppInfo {
|
|
/**
|
|
* The app URL's registrable domain, as read by `siteOf`. Use `sameSite`
|
|
* to scope secret fills, injected headers, and observed child frames.
|
|
* Absent when the app has no URL. The runner passes navigation URLs
|
|
* separately through `session.open`.
|
|
*/
|
|
readonly site?: string;
|
|
/** The installed app to launch, as the target declared it. */
|
|
readonly bundleId?: string;
|
|
/** The build to install, as the target declared it; relative to `projectRoot`. */
|
|
readonly appPath?: string;
|
|
/** Arguments every fresh launch passes the app. */
|
|
readonly launchArguments?: readonly string[];
|
|
/** Permissions the app holds on every fresh launch, by name. */
|
|
readonly permissions?: Readonly<Record<string, AppPermissionState>>;
|
|
}
|
|
|
|
/**
|
|
* Per-attempt isolation context. An engine that must give each test a fresh
|
|
* surface (a fresh isolated context per test, a reset device) sets it up in
|
|
* `startAttempt` and tears it down in `endAttempt`.
|
|
*/
|
|
export interface EngineAttemptContext {
|
|
/** The attempt id. */
|
|
readonly attemptId: string;
|
|
/** Absolute directory every artifact of this attempt is written under. */
|
|
readonly artifactsDir: string;
|
|
/**
|
|
* The plaintext of a secret the engine declared in `secrets`, for an
|
|
* option the engine hands the app itself (basic-auth credentials). A
|
|
* provider runs again on every call. The value, and every value
|
|
* `options.derived` computes from it, join the attempt's redaction before
|
|
* this resolves, so reports, logs, and every observation redact them, and
|
|
* the attempt's trace and text downloads are rewritten. The protection is
|
|
* text only: unlike a fill, it withholds no screenshot or model pixels.
|
|
* A secret the engine did not declare is `SECRET_UNAVAILABLE`.
|
|
*/
|
|
readonly resolveSecret: (secret: Secret, options?: ResolveSecretOptions) => Promise<string>;
|
|
/**
|
|
* Aborts with the attempt, and the moment `startAttempt` fails or exceeds
|
|
* the launch timeout: a hook still running then must stop, because the
|
|
* harness ends the attempt's isolation right behind it and may retry.
|
|
*/
|
|
readonly signal: AbortSignal;
|
|
}
|
|
|
|
/** How `EngineAttemptContext.resolveSecret` treats what the engine makes of the value. */
|
|
export interface ResolveSecretOptions {
|
|
/**
|
|
* The forms of the value the app sees instead of the value itself, for the
|
|
* same redaction under the secret's name: the base64 `user:password` an
|
|
* `Authorization: Basic` header carries, which a page echoing its request
|
|
* headers would otherwise show.
|
|
*/
|
|
readonly derived?: (plaintext: string) => readonly string[];
|
|
}
|
|
|
|
/**
|
|
* Budget of one cleanup hook (`settleAttempt`, `endAttempt`, `dispose`). `signal` aborts when
|
|
* the budget is exhausted, so a hook that cannot finish in time stops instead
|
|
* of running on into the next attempt's setup.
|
|
*/
|
|
export interface EngineCleanupContext {
|
|
/** Aborts when the cleanup budget is spent. */
|
|
readonly signal: AbortSignal;
|
|
/** Remaining cleanup budget when the call starts. */
|
|
readonly timeoutMs: number;
|
|
}
|
|
```
|
|
|
|
URLs arrive through `session.open` only; `init` never learns the app URL.
|
|
|
|
## Observation
|
|
|
|
```ts
|
|
/**
|
|
* One fresh semantic snapshot of the surface under test. The harness owns
|
|
* everything downstream: revision minting, secret redaction, and the
|
|
* observation byte budget apply to every engine equally.
|
|
*/
|
|
export interface EngineSnapshot {
|
|
/**
|
|
* Where the surface is, as an opaque address the platform understands: a
|
|
* URL on a document platform, the foreground screen or activity on a
|
|
* device, the front window on a desktop. The harness shows it to the model
|
|
* and to the report, anchors trace replay on it, and applies the origin
|
|
* policy to it only when it parses as a URL. Omit when the platform has no
|
|
* notion of a current location.
|
|
*/
|
|
readonly location?: string;
|
|
/**
|
|
* The observation tree under one root the engine minted. The root's id MUST
|
|
* be stable for the surface across observations: `perform(root, swipe)` is
|
|
* the viewport swipe, and the harness addresses it from an earlier
|
|
* observation, possibly one taken before a `restart` or `reset`, so the
|
|
* root swipe MUST NOT depend on a live handle from that observation. A
|
|
* platform with several top-level elements (a device's windows) wraps them
|
|
* in a root of its own.
|
|
*/
|
|
readonly root: SemanticNode;
|
|
/** The viewport `SemanticNode.rect` and `ViewportPoint` are measured in, in CSS pixels. */
|
|
readonly viewport: ViewportSize;
|
|
/**
|
|
* True when `root` leaves out nodes that are on the surface: the engine
|
|
* stopped reading at a node cap or a depth limit, or could not enter a
|
|
* child document. The harness tells the model the listing is incomplete
|
|
* and never calls such a screen unchanged. Omit or set false for a tree
|
|
* read whole.
|
|
*/
|
|
readonly truncated?: boolean;
|
|
/** Semantic capture failed; only fresh, independently masked pixels are evidence. */
|
|
readonly treeUnavailable?: true;
|
|
/** Masked pixels, when requested and producible; omitted otherwise. */
|
|
readonly pixels?: ObservationPixels;
|
|
/** Regions masked in `pixels`; the harness checks it covers every secure node. */
|
|
readonly maskedRegionCount?: number;
|
|
}
|
|
|
|
export interface EngineObserveOptions {
|
|
/** Requests masked viewport pixels for the same revision as the tree. */
|
|
readonly pixels?: boolean;
|
|
/** Permits independently masked pixels to replace a failed semantic capture. */
|
|
readonly pixelFallback?: boolean;
|
|
}
|
|
```
|
|
|
|
Return the surface as it is right now. The runner mints the revision, stamps
|
|
it onto every ref, redacts registered secret values, bounds the byte size, and
|
|
checks that `maskedRegionCount` covers every `secure` node when `pixels` are
|
|
present.
|
|
|
|
Set `truncated` when a node limit or inaccessible subtree leaves content out.
|
|
The runner warns the model that the tree is incomplete and never treats a
|
|
truncated observation as unchanged. Omit it for a complete tree.
|
|
|
|
A valid empty tree may have a childless root and include `pixels`.
|
|
|
|
If semantic capture times out, an engine may recover only when `pixelFallback`
|
|
is true and it can prove screenshot masking without that capture. Return
|
|
`treeUnavailable: true`, fresh `pixels`, and a `root` containing only its
|
|
stable `ref`. That reference supports viewport actions; it supplies no
|
|
semantic node evidence. Retire earlier node references. The flag distinguishes
|
|
missing semantic evidence from a valid empty tree; never infer absence from
|
|
it. The runner validates this shape at the engine boundary and rejects a
|
|
fallback without permitted pixels as `UNSUPPORTED_CAPABILITY`, or one with
|
|
semantic root fields as `INVALID_STATE`. `EngineSnapshot` remains an
|
|
extendable interface; these dependent requirements are checked at runtime.
|
|
|
|
Without permission, pixels, or independent masking proof, propagate the
|
|
capture failure. Cancellation, policy denial, and hard engine failures must
|
|
also propagate.
|
|
|
|
The runner permits fallback for `act` and for judgments with `vision: true`
|
|
or `'only'`, before any secret fill. It includes the screenshot even if the
|
|
initial observation requested only a tree. A step that loses semantic
|
|
evidence cannot record or finish by replaying a trace. Engines whose masks
|
|
depend on their semantic capture cannot provide this fallback.
|
|
|
|
## Actions
|
|
|
|
```ts
|
|
/**
|
|
* One deterministic action `perform` carries out on a node. The `swipe`
|
|
* kind on the observation's root node is the viewport swipe: the agent's
|
|
* `scroll` verb and `screen.swipe()` both arrive as `perform(root, swipe)`,
|
|
* so a surface implements scrolling once.
|
|
*
|
|
* `press` takes one key in the `Key` grammar below.
|
|
*/
|
|
export type LocatorAction =
|
|
| {
|
|
readonly kind: 'tap' | 'doubleTap' | 'secondaryTap';
|
|
/** Keys held for the click; only an engine that declares `tapModifiers` receives it. */
|
|
readonly modifiers?: readonly KeyModifier[];
|
|
}
|
|
| {
|
|
readonly kind:
|
|
| 'check'
|
|
| 'uncheck'
|
|
| 'clear'
|
|
| 'focus'
|
|
| 'hover'
|
|
| 'scrollIntoView';
|
|
}
|
|
| { readonly kind: 'longPress'; readonly durationMs?: number }
|
|
| { readonly kind: 'fill'; readonly value: string; readonly sensitive: boolean }
|
|
| { readonly kind: 'press'; readonly key: Key }
|
|
| { readonly kind: 'selectOption'; readonly value: SelectOption }
|
|
| { readonly kind: 'setInputFiles'; readonly paths: readonly string[] }
|
|
| { readonly kind: 'dragTo'; readonly target: NodeRef }
|
|
| {
|
|
readonly kind: 'swipe';
|
|
readonly direction: ScrollDirection;
|
|
readonly momentum?: Momentum;
|
|
};
|
|
|
|
/** Every action kind; what an engine declares in `actions`. */
|
|
export type LocatorActionKind = LocatorAction['kind'];
|
|
|
|
/** The closed list of action kinds, in contract order; the type is derived from it. */
|
|
export const LOCATOR_ACTION_KINDS = [
|
|
'tap',
|
|
'doubleTap',
|
|
'secondaryTap',
|
|
'longPress',
|
|
'fill',
|
|
'clear',
|
|
'press',
|
|
'check',
|
|
'uncheck',
|
|
'focus',
|
|
'hover',
|
|
'scrollIntoView',
|
|
'selectOption',
|
|
'setInputFiles',
|
|
'dragTo',
|
|
'swipe',
|
|
] as const satisfies readonly LocatorActionKind[];
|
|
```
|
|
|
|
The grammar is closed. Anything beyond it is an agent-side `defineTool`.
|
|
|
|
### Pointer actions
|
|
|
|
Pointer actions use viewport coordinates when no semantic node is available.
|
|
`dragTo` ends at another point; `swipe` starts a scroll gesture at the given
|
|
point. `swipeTo` is the one kind with no node form: the finger's path from
|
|
the point to `target`, a touch swipe on a device and a pointer drag on a
|
|
document platform, distinct from `dragTo` where a platform holds before it
|
|
drags. Declare supported kinds in `pointerActions`.
|
|
|
|
```ts
|
|
export type PointerAction =
|
|
| { readonly kind: 'tap' | 'doubleTap' | 'secondaryTap' | 'hover' }
|
|
| { readonly kind: 'longPress'; readonly durationMs?: number }
|
|
| { readonly kind: 'dragTo'; readonly target: ViewportPoint }
|
|
| {
|
|
readonly kind: 'swipe';
|
|
readonly direction: ScrollDirection;
|
|
readonly momentum?: Momentum;
|
|
}
|
|
| { readonly kind: 'swipeTo'; readonly target: ViewportPoint };
|
|
|
|
/** Every pointer action kind; what an engine declares in `pointerActions`. */
|
|
export type PointerActionKind = PointerAction['kind'];
|
|
|
|
/** The closed list of pointer action kinds, in contract order; the type is derived from it. */
|
|
export const POINTER_ACTION_KINDS = [
|
|
'tap',
|
|
'doubleTap',
|
|
'secondaryTap',
|
|
'longPress',
|
|
'hover',
|
|
'dragTo',
|
|
'swipe',
|
|
'swipeTo',
|
|
] as const satisfies readonly PointerActionKind[];
|
|
```
|
|
|
|
### Keys
|
|
|
|
```ts
|
|
/**
|
|
* The key grammar of `press`, shared by every engine so a test's key names are
|
|
* portable: zero or more modifiers and one key, joined by `+`.
|
|
*
|
|
* - Modifiers: `Shift`, `Control`, `Alt`, `Meta`, `ControlOrMeta` (Control on
|
|
* Windows and Linux, Meta on macOS).
|
|
* - Named keys: those in `KEY_NAMES` (`Enter`, `Escape`, `Tab`, `Backspace`,
|
|
* `Delete`, `ArrowUp`..., `Home`, `End`, `PageUp`, `PageDown`, `Insert`,
|
|
* `Space`, `F1`..`F12`).
|
|
* - Any single printable character (`a`, `A`, `1`, `$`), typed as itself.
|
|
*
|
|
* `Control+a`, `Shift+Tab`, `Enter`, `$`. A platform without a key maps it
|
|
* or throws `UNSUPPORTED_CAPABILITY`; it never reinterprets the spelling.
|
|
* `parseKey` is the one parser, exported so no engine carries its own.
|
|
*/
|
|
export type Key = string;
|
|
|
|
export const KEY_MODIFIERS = ['Shift', 'Control', 'Alt', 'Meta', 'ControlOrMeta'] as const;
|
|
|
|
export const KEY_NAMES = [
|
|
'Enter',
|
|
'Escape',
|
|
'Tab',
|
|
'Backspace',
|
|
'Delete',
|
|
'Insert',
|
|
'Space',
|
|
'ArrowUp',
|
|
'ArrowDown',
|
|
'ArrowLeft',
|
|
'ArrowRight',
|
|
'Home',
|
|
'End',
|
|
'PageUp',
|
|
'PageDown',
|
|
'F1',
|
|
'F2',
|
|
'F3',
|
|
'F4',
|
|
'F5',
|
|
'F6',
|
|
'F7',
|
|
'F8',
|
|
'F9',
|
|
'F10',
|
|
'F11',
|
|
'F12',
|
|
] as const;
|
|
|
|
/** One parsed `press` key: its modifiers and either a named key or a single character. */
|
|
export interface ParsedKey {
|
|
/** Modifiers, in the order written. */
|
|
readonly modifiers: readonly KeyModifier[];
|
|
/** The key itself. */
|
|
readonly key: { readonly kind: 'named'; readonly name: KeyName } | { readonly kind: 'char'; readonly char: string };
|
|
}
|
|
|
|
export function parseKey(key: string): ParsedKey | undefined;
|
|
```
|
|
|
|
## App
|
|
|
|
The app under test is declared on the target (`targets: [{ engine, app }]`,
|
|
see [the app under test](/reference/config#the-app-under-test)); an engine
|
|
only drives it. The harness checks the shape of every field, then hands the
|
|
declaration to `validateApp` at config load, so an engine can require what
|
|
its platform needs and refuse what it cannot drive. `@e2e-dev/web` requires
|
|
`url`; `@e2e-dev/mobile` requires `bundleId` or `appPath` and refuses `url`.
|
|
|
|
```ts
|
|
/**
|
|
* The app a target declares, as its engine reads it at config load: what it
|
|
* is and where it is served. `EngineAppInfo` (below) is what `prepare` and
|
|
* `init` receive; the declaration adds the URL, which a surface never gets.
|
|
*/
|
|
export type EngineAppDeclaration = EngineAppInfo & {
|
|
/** Base URL the target opens, as declared; see `TargetApp.url`. */
|
|
readonly url?: string;
|
|
};
|
|
|
|
/** Handed to `Engine.validateApp` beside the declaration. */
|
|
export interface EngineAppCheckInfo {
|
|
/** The target declaring the app, for the message of a refusal. */
|
|
readonly targetName: string;
|
|
}
|
|
|
|
/** A permission's state when the app launches: held, refused, or not asked for yet, so the OS asks again. */
|
|
export type AppPermissionState = 'grant' | 'deny' | 'reset';
|
|
```
|
|
|
|
```ts
|
|
validateApp(app, { targetName }) {
|
|
if (app.url === undefined) {
|
|
throw new ConfigurationError('INVALID_CONFIG', `target "${targetName}" needs app.url`);
|
|
}
|
|
},
|
|
```
|
|
|
|
`validateApp` runs for every target the engine serves, with `{}` for a
|
|
target that declares no `app`. The resolved app comes back to the engine as
|
|
`EnginePrepareInfo.app` and `EngineInitInfo.app`: the site of the URL, which
|
|
`sameSite` from `e2e/engine` applies to a URL, and the device fields as the
|
|
target declared them. The URL itself stays with the harness, which opens it
|
|
through `session.open`. The target's `command` and `readyUrl` never reach
|
|
the engine: the runner starts it. `CommandConfig` is exported from `e2e`, not
|
|
`e2e/engine`.
|
|
|
|
## Session
|
|
|
|
```ts
|
|
/**
|
|
* Steering hooks behind the universal `app` fixture and the agent's
|
|
* `navigate` verb. The app itself is the target's `app`, handed over as data;
|
|
* node actions never live here, they are `perform`.
|
|
*
|
|
* `restart` and `reset` open nothing on an addressable surface: they end at
|
|
* a fresh surface with no location shown, and the harness reopens the app
|
|
* through `open` when the target has an address. A surface therefore never
|
|
* needs to know the app's URL, and an unreachable app after a restart is
|
|
* reported the same way as on first open. A surface without `open` (a
|
|
* device with a pinned app) relaunches the app inside `restart` and `reset`
|
|
* itself, since nothing else will.
|
|
*/
|
|
export interface EngineSession {
|
|
/**
|
|
* Opens one URL the harness already resolved against the base URL and the
|
|
* origin policy. Absent on a surface without addressable locations.
|
|
*/
|
|
open?(url: string, context: OperationContext): Promise<void>;
|
|
/** Navigates back once in the surface's history. */
|
|
back?(context: OperationContext): Promise<void>;
|
|
/** Recreates the execution context, keeping persisted state. */
|
|
restart?(context: OperationContext): Promise<void>;
|
|
/** Clears persisted client state and recreates the execution context. */
|
|
reset?(context: OperationContext): Promise<void>;
|
|
}
|
|
```
|
|
|
|
`open` is behind `app.open()`, the agent's `navigate` verb, and the reopen
|
|
after `restart` and `reset`; `back`, `restart`, and `reset` are behind
|
|
`app.back()`, `app.restart()`, and `app.clearState()`. An absent hook makes
|
|
its method fail with `UNSUPPORTED_CAPABILITY`.
|
|
|
|
## Errors
|
|
|
|
```ts
|
|
export class EngineError extends Error {
|
|
constructor(code: EngineErrorCode, message: string, options: { retryable: boolean; cause?: unknown });
|
|
/** One of `ENGINE_ERROR_CODES`. */
|
|
readonly code: EngineErrorCode;
|
|
/** Whether the runner may re-resolve and retry; only `RETRYABLE_ENGINE_ERROR_CODES` can be. */
|
|
readonly retryable: boolean;
|
|
}
|
|
|
|
/** The closed engine error code set; the type is derived from it, so the two cannot drift. */
|
|
export const ENGINE_ERROR_CODES = [
|
|
'NODE_STALE',
|
|
'FRAME_NOT_FOUND',
|
|
'FRAME_AMBIGUOUS',
|
|
'NOT_ACTIONABLE',
|
|
'ACTION_MAY_HAVE_COMMITTED',
|
|
'OPERATION_TIMEOUT',
|
|
'CANCELLED',
|
|
'UNSUPPORTED_CAPABILITY',
|
|
'INVALID_STATE',
|
|
'ENGINE_FAILURE',
|
|
] as const;
|
|
|
|
/** One of `ENGINE_ERROR_CODES`. */
|
|
export type EngineErrorCode = (typeof ENGINE_ERROR_CODES)[number];
|
|
|
|
/**
|
|
* The only codes that may be retryable: both describe a repeatable read. Any
|
|
* other retryable claim, from this module's class or a foreign copy of it, is
|
|
* coerced to a non-retryable `ENGINE_FAILURE`.
|
|
*/
|
|
export const RETRYABLE_ENGINE_ERROR_CODES: ReadonlySet<EngineErrorCode> = new Set([
|
|
'NODE_STALE',
|
|
'FRAME_NOT_FOUND',
|
|
]);
|
|
```
|
|
|
|
| Code | Runner mapping | Retryable |
|
|
| --- | --- | --- |
|
|
| `NODE_STALE` | re-resolve, then `LOCATOR_NOT_FOUND` | yes |
|
|
| `FRAME_NOT_FOUND` | re-resolve, then `LOCATOR_NOT_FOUND` | yes |
|
|
| `FRAME_AMBIGUOUS` | `LOCATOR_AMBIGUOUS` | no |
|
|
| `NOT_ACTIONABLE`, `ACTION_MAY_HAVE_COMMITTED`, `OPERATION_TIMEOUT` | `ACTION_FAILED` | no |
|
|
| `CANCELLED` | infrastructure `CANCELLED` | no |
|
|
| `UNSUPPORTED_CAPABILITY` | configuration `UNSUPPORTED_CAPABILITY` | no |
|
|
| `INVALID_STATE` | `APP_NOT_OPEN` | no |
|
|
| `ENGINE_FAILURE`, any non-`EngineError` | infrastructure `ENGINE_FAILURE` | no |
|
|
|
|
An `EngineError` a contributed fixture method throws is mapped by the table
|
|
above, so a fixture over a session that ended reports `APP_NOT_OPEN` the way
|
|
an observation would. A method can also fail a step with the runner's own
|
|
error classes, exported from the same entry:
|
|
|
|
```ts
|
|
/** The runner's error base: `category` sets the exit code, `code` names the cause. */
|
|
declare class E2EError extends Error {
|
|
readonly category: 'test' | 'configuration' | 'infrastructure' | 'internal' | 'interrupted';
|
|
readonly code: string;
|
|
readonly retryable: boolean;
|
|
}
|
|
|
|
/** A test failure: ASSERTION_FAILED, INVALID_LOCATOR, ACTION_FAILED, ... */
|
|
export class TestError extends E2EError {
|
|
constructor(code: string, message: string, options?: { retryable?: boolean; cause?: unknown });
|
|
}
|
|
/** POLICY_DENIED, APP_URL_REQUIRED, UNSUPPORTED_CAPABILITY, ... */
|
|
export class ConfigurationError extends E2EError {
|
|
constructor(code: string, message: string, options?: { cause?: unknown });
|
|
}
|
|
/** The surface itself failed: BROWSER_INSTALL_FAILED, ... */
|
|
export class InfrastructureError extends E2EError {
|
|
constructor(code: string, message: string, options?: { cause?: unknown });
|
|
}
|
|
```
|
|
|
|
`E2EError` itself is not exported; construct one of the three.
|
|
|
|
## Shared semantics
|
|
|
|
Use these helpers to keep behavior consistent across engines:
|
|
|
|
| Export | Use |
|
|
| --- | --- |
|
|
| `matchesText(actual, pattern)`, `toTextPattern(match, { exact? })`, `describePattern(pattern)` | `TextPattern` matching for `locate` and text matchers; `toTextPattern` throws `INVALID_ARGUMENT` for a `match` that is neither a string nor a `RegExp` |
|
|
| `resolveExpression(expression, nodes, { selector? })` | The reference `locate`. Resolves one `LocatorExpression` over a semantic tree in document order, with `text` and `label` answering the innermost match when a container echoes a descendant's text. An engine whose tree is the whole truth (a device) calls it; a browser engine reproduces it. A `selector` needs the hook, a `frame` is `FRAME_NOT_FOUND` |
|
|
| `urlMatches(current, expected, baseHref, ignoreCase?)` | `toHaveURL`-style comparison, `ignoreCase` as on the locator matchers; an `expected` that is neither a string nor a `RegExp` throws `INVALID_ARGUMENT`, as in `toTextPattern` |
|
|
| `pollCondition({ deadline, signal, negated, evaluate, onTimeout })`, `Deadline` | Assertion polling and the negation grace window |
|
|
| `validateJsonValue(value, label, rules?)` | Checks a value is plain JSON: a top-level `undefined` passes; a non-finite number, a cycle, an object that is not a plain object or array, or a function, symbol, bigint, or `undefined` inside is `INVALID_ARGUMENT` naming `label`. A `Secret` is `POLICY_DENIED` unless `rules.onSecret` throws its own; `rules.maxDepth` caps the nesting |
|
|
| `rejectUnknownKeys(label, value, keys, code?)` | Throws `INVALID_CONFIG` for the first own key of `value` outside `keys`, naming the nearest known key when one is a plausible typo (`web() has unknown key "viewprt"; did you mean "viewport"?`) and the known keys otherwise: the check the runner runs on its own config objects, for an engine factory's options. Pass `'INVALID_ARGUMENT'` as `code` for an object a test passes to a fixture method |
|
|
| `rejectUnknownOptions(api, options, known, code?)` | `INVALID_ARGUMENT` (or `code`, `INVALID_LOCATOR` for a locator option) for an option key a fixture method or matcher does not take, so a JavaScript caller's unsupported option fails instead of running on the default. `options` must be a plain object or `undefined`; a class instance or array is refused, and non-enumerable own keys count as keys |
|
|
| `obj`, `raceAbort`, `withTimeout`, `withinCleanupBudget` | See [Contributed fixtures](#contributed-fixtures) |
|
|
| `isProviderRecording(value)` | Whether what a provider's `record` resolved to is a recording: a parseable `startedAt` and a `stop`. Check it before keeping one, so a malformed recording fails the start, named, instead of the stop |
|
|
| `stopProviderRecording(recording, { artifactsDir, provider, leaseId, signal })` | Ends a hosted service's recording and checks it into a `VideoSegment`; see [Provider recordings](#provider-recordings) |
|
|
|
|
## State
|
|
|
|
```ts
|
|
/**
|
|
* An opaque, restorable snapshot of an engine's state. `data` is JSON the
|
|
* harness never inspects - a document platform's storage state, a device's app state, a
|
|
* desktop's window state all satisfy it identically. `format`/`version` let a
|
|
* engine reject a snapshot it can no longer read.
|
|
*/
|
|
export interface EngineState {
|
|
/** Engine-defined format name. */
|
|
readonly format: string;
|
|
/** Format version, for rejecting a snapshot the engine can no longer read. */
|
|
readonly version: number;
|
|
/** Opaque JSON the runner never inspects. */
|
|
readonly data: unknown;
|
|
/** Earliest moment the state is known to be invalid, if the engine knows one. */
|
|
readonly expiresAt?: string;
|
|
}
|
|
|
|
/**
|
|
* Platform-neutral state capture and restore. A setup test captures a
|
|
* snapshot; an ordinary test restores it at launch.
|
|
*
|
|
* Snapshots are credentials: cookies, tokens, and storage that authenticate
|
|
* the app under test. The duties split at the seam. The harness owns the
|
|
* snapshot from the moment `capture` returns: it encrypts it at rest with a
|
|
* per-run AES-256-GCM key that never touches disk, stores it under the run's
|
|
* private session directory, binds it to the run, target, engine, and app
|
|
* identity, and deletes it when the run ends; it never logs it, reports it,
|
|
* digests it, or sends it to a model. An engine MUST NOT persist, cache, or
|
|
* log a snapshot on its own, MUST NOT echo snapshot contents in an error
|
|
* message, and MUST treat `restore` as a replacement of the surface's whole
|
|
* persisted state, never a merge, so a restored session cannot leak into the
|
|
* next attempt.
|
|
*/
|
|
export interface EngineStateCapability {
|
|
/** Captures the surface's persisted state. */
|
|
capture(context: OperationContext): Promise<EngineState>;
|
|
/** Replaces the surface's persisted state with `state`. */
|
|
restore(state: EngineState, context: OperationContext): Promise<void>;
|
|
}
|
|
```
|
|
|
|
## Artifacts
|
|
|
|
```ts
|
|
/**
|
|
* Evidence capture. Paths are relative to the attempt artifact directory the
|
|
* engine received in `startAttempt`.
|
|
*/
|
|
export interface EngineArtifacts {
|
|
/** Captures a redacted screenshot; secure fields are masked at the source. */
|
|
screenshot(label: string | undefined, context: OperationContext): Promise<string>;
|
|
/** Starts recording an execution trace for the attempt. */
|
|
startTrace?(context: OperationContext): Promise<void>;
|
|
/**
|
|
* Stops the trace and returns its relative path, or every archive written,
|
|
* in order, when the trace had to be cut: a trace bound to one context
|
|
* closes as a segment when a restart or a state reset replaces the
|
|
* context, and a new one records on from there. The harness registers and
|
|
* redacts each returned archive.
|
|
*/
|
|
stopTrace?(context: OperationContext): Promise<string | readonly string[]>;
|
|
/**
|
|
* Starts recording the surface for the attempt. Declared together with
|
|
* `stopVideo`. A surface that has nothing to show yet (no page open) may
|
|
* defer the actual capture to the moment it does; the segments returned by
|
|
* `stopVideo` say when each one began.
|
|
*/
|
|
startVideo?(context: OperationContext): Promise<void>;
|
|
/**
|
|
* Stops recording and returns every segment written, in order. One segment
|
|
* is the common case; a surface whose recording is bound to a page returns
|
|
* one per page the attempt showed (a restart or a state reset opens a new
|
|
* one). An attempt that never showed anything returns none.
|
|
*/
|
|
stopVideo?(context: OperationContext): Promise<readonly VideoSegment[]>;
|
|
}
|
|
|
|
/**
|
|
* One recorded video of an attempt: a file the engine wrote into the attempt
|
|
* artifact directory, or a recording a hosted service keeps and links to.
|
|
*/
|
|
export type VideoSegment = VideoFile | VideoLink;
|
|
|
|
/** A video file in the attempt artifact directory. */
|
|
export interface VideoFile {
|
|
/** Path relative to the attempt artifact directory. */
|
|
readonly path: string;
|
|
/** When the segment started recording, as an ISO timestamp; its first frame is at or just after it. */
|
|
readonly startedAt: string;
|
|
}
|
|
|
|
/** A recording a hosted service keeps, by URL: the report links to it and the runner never downloads it. */
|
|
export interface VideoLink {
|
|
/** `http(s)` URL of the recording. */
|
|
readonly url: string;
|
|
/** Media type of what the URL serves: `video/mp4`, or `text/html` for a player page. */
|
|
readonly mediaType: string;
|
|
/** When the recording started, as an ISO timestamp; its first frame is at or just after it. */
|
|
readonly startedAt: string;
|
|
}
|
|
```
|
|
|
|
The runner calls `startVideo` only for an attempt whose
|
|
[video mode](/reference/config#video) records, then `startTrace` for one whose
|
|
[trace mode](/reference/config#trace) records, after
|
|
`startAttempt` and any session restore. Both use the launch budget and stop
|
|
before `endAttempt`, within the cleanup budget; a `startVideo` that fails
|
|
fails the launch. Return segments in recording order from `stopVideo` and
|
|
`stopTrace`. The runner decides which recordings are kept and deletes a file an
|
|
attempt does not keep.
|
|
|
|
### Provider recordings
|
|
|
|
An engine that leases its browser or device from a hosted service can let the
|
|
service record the attempt in place of its own capture, the way
|
|
[`BrowserProvider.record`](/browser#recording-on-the-provider) and
|
|
[`DeviceProvider.record`](/mobile#hosted-devices) do. `e2e/engine` exports
|
|
the shared shape and `stopProviderRecording`, so every engine turns a
|
|
provider's recording into a `VideoSegment` the same way.
|
|
|
|
```ts
|
|
/** Handed to a provider's `record`, once per attempt that records video. */
|
|
export interface ProviderRecordContext {
|
|
readonly runId: string;
|
|
readonly targetName: string;
|
|
/** The attempt the recording covers. */
|
|
readonly attemptId: string;
|
|
/** The run's environment, the same the provider's `acquire` saw. */
|
|
readonly env: Readonly<Record<string, string | undefined>>;
|
|
/** Aborts when the attempt is cancelled or its launch budget is spent. */
|
|
readonly signal: AbortSignal;
|
|
}
|
|
|
|
/** A recording a provider's service makes of the browser or device it leased, as `record` started it. */
|
|
export interface ProviderRecording {
|
|
/** When the recording started, as an ISO timestamp; its first frame is at or just after it. */
|
|
readonly startedAt: string;
|
|
stop(context: ProviderRecordingStopContext): Promise<ProviderRecordingResult>;
|
|
}
|
|
|
|
/** Handed to `ProviderRecording.stop`, once per recording. */
|
|
export interface ProviderRecordingStopContext {
|
|
/** Absolute directory, already created, a recording written as a file goes into. */
|
|
readonly dir: string;
|
|
/** Aborts when the cleanup budget is spent. */
|
|
readonly signal: AbortSignal;
|
|
}
|
|
|
|
/**
|
|
* What a stopped recording became: a file the provider wrote into the
|
|
* directory it was given, by name, or a recording its service keeps, by
|
|
* `http(s)` URL and media type.
|
|
*/
|
|
export type ProviderRecordingResult =
|
|
| { readonly file: string }
|
|
| { readonly url: string; readonly mediaType: string };
|
|
|
|
export function stopProviderRecording(
|
|
recording: ProviderRecording,
|
|
target: {
|
|
/** Absolute attempt artifact directory; a file recording goes into its `video` directory. */
|
|
readonly artifactsDir: string;
|
|
/** The provider as error messages name it: `browser provider "kernel"`. */
|
|
readonly provider: string;
|
|
/** The lease the recording covers, named in error messages. */
|
|
readonly leaseId: string;
|
|
readonly signal: AbortSignal;
|
|
},
|
|
): Promise<VideoSegment>;
|
|
```
|
|
|
|
`stopProviderRecording` creates the attempt's `video` directory, calls
|
|
`stop` with it, and checks what came back: a file must be a plain name that
|
|
exists in that directory, a link an `http(s)` URL with a media type. It
|
|
returns `{ path: 'video/<file>', startedAt }` or `{ url, mediaType,
|
|
startedAt }` for `stopVideo` to hand back. A `stop` that throws or returns
|
|
anything else is `ENGINE_FAILURE`, naming the provider and the lease. The
|
|
target argument's type is exported as `ProviderRecordingTarget`.
|
|
|
|
When the session knows a secret value, the runner redacts every text entry in
|
|
each trace archive, so a value a test spelled into a URL or the options the
|
|
engine opened the attempt with (`resolveSecret`) never stay in it. After a
|
|
secret fill it also drops the screencast frames: every `screencast/` entry,
|
|
each `screencast-frame` record in a `.trace` entry, and any image a record
|
|
names.
|
|
If redaction fails, it deletes the archives and records
|
|
`TRACE_WITHHELD`. Videos are unmasked and carry `redaction: 'incomplete'`.
|
|
A `download` an engine attaches is the app's bytes: when the session knows
|
|
a secret value, the runner rewrites a text one (`.txt`,
|
|
`.csv`, `.json`, `.html`) through the ledger and labels it `complete`; every
|
|
other download keeps its bytes and carries `redaction: 'incomplete'`.
|
|
|
|
## Contributed fixtures
|
|
|
|
```ts
|
|
/**
|
|
* A contributed fixture: any record of async methods, sync accessors, and
|
|
* nested namespaces, declared through context.fixture so its metadata controls
|
|
* labels, deadlines and verification. The factory must return the declared
|
|
* surface; a plain one is rejected with INVALID_CONFIG.
|
|
*/
|
|
export type EngineFixtureFactory = (context: EngineFixtureContext) => object;
|
|
|
|
/** Context handed to a contributed fixture factory, once per attempt. */
|
|
export interface EngineFixtureContext {
|
|
/** The target the fixture serves. */
|
|
readonly targetName: string;
|
|
/** Records the declared operations before invoking them; undeclared accessors retain their identity. */
|
|
fixture<T extends object>(name: string, surface: T, operations: FixtureOperations<T>): T;
|
|
readonly app: EngineAppInfo & {
|
|
/**
|
|
* Resolves a navigation target against the base URL. Throws
|
|
* `APP_URL_REQUIRED` when the target declares no URL and `POLICY_DENIED`
|
|
* for any scheme but `http:` and `https:` (`about:blank` aside), so a
|
|
* fixture never re-implements the rule the harness owns.
|
|
*/
|
|
resolveUrl(url: string): string;
|
|
};
|
|
/** Harness budgets, for fixtures that poll or navigate. */
|
|
readonly timeouts: {
|
|
readonly test: number;
|
|
readonly action: number;
|
|
readonly assertion: number;
|
|
};
|
|
/**
|
|
* The running phase's signal: the attempt's through `beforeEach` and the
|
|
* body, then a fresh one per `afterEach` hook so teardown can still drive
|
|
* the app after a body timeout. Read it per call; a fixture factory that
|
|
* captures it once keeps a signal that is dead by teardown.
|
|
*/
|
|
readonly signal: AbortSignal;
|
|
/** Per-call operation budget: the action timeout (or an explicit one) plus the attempt signal. */
|
|
operation(timeoutMs?: number): OperationContext;
|
|
/** Registers a file the current step produced under the attempt artifact directory. */
|
|
attachArtifact(
|
|
kind: 'screenshot' | 'trace' | 'video' | 'download' | 'log',
|
|
relativePath: string,
|
|
): void;
|
|
/** Records the viewport the current step established. */
|
|
attachViewport(viewport: { width: number; height: number; scale: number }): void;
|
|
/** Mints a public locator from a raw expression (a platform selector, say). */
|
|
locator(expression: LocatorExpression): Locator;
|
|
/** Mints a screen scope whose every query is wrapped by `scope` (a nested document, say). */
|
|
screen(scope: (expression: LocatorExpression) => LocatorExpression): Screen;
|
|
/**
|
|
* Attaches an expectation surface to a fixture object so `expect(fixture)`
|
|
* returns it. The harness records every matcher call as an assertion step
|
|
* named `expect.<matcher>`; the factory only decides what the matchers mean.
|
|
*/
|
|
expectable<T extends object, E extends object>(target: T, factory: () => E): T & Expectable<E>;
|
|
}
|
|
```
|
|
|
|
`context.fixture(name, surface, operations)` declares which methods are
|
|
recorded steps. Each operation names a `kind: 'resource' | 'assertion'`, an
|
|
optional `label(...args)`, an optional `timeout` (milliseconds, a function of
|
|
the arguments, or `false` when the method owns its deadline), and optional
|
|
`verifies`. The default timeout is `actionTimeout`; assertions verify by
|
|
default. Nested maps describe namespaces. Recording decorates the object in
|
|
place, so create a fresh surface in each factory invocation.
|
|
|
|
The runner opens the step before calling a declared method, so attachments
|
|
and failures from synchronous and asynchronous work land on the same step.
|
|
Work arriving after its step has closed cannot attach to a later one.
|
|
|
|
A factory must return the surface it declared through `context.fixture`; a
|
|
plain surface is `INVALID_CONFIG` the first time a test reaches for it.
|
|
`expectable` attaches an expectation surface so `expect(fixture)` returns it;
|
|
declare matcher recording with `context.fixture('expect', matchers,
|
|
operations)`. Type the fixture with `test.extend<{ name: Type }>()`.
|
|
|
|
The entrypoint also exports `obj(value)`, which drops `undefined`-valued keys
|
|
so a result built from optional inputs satisfies exact optional property
|
|
types, plus `raceAbort(work, signal, label)`, `withTimeout(promise,
|
|
timeoutMs, onTimeout)`, and `withinCleanupBudget(promise, { signal,
|
|
timeoutMs })`. Pass a thunk to `raceAbort` to check cancellation before
|
|
dispatch. `isSecret(value)` tells a `secrets.get()` handle from a string, for
|
|
an option that takes either.
|
|
|
|
## Vocabulary
|
|
|
|
`OperationContext`, `TextPattern`, `SemanticQuery`, `LocatorExpression`,
|
|
`NodeRef`, `SemanticNode`, `ViewportPoint`, `ViewportSize`, `ObservationPixels`,
|
|
`LocatorAction`, `LocatorActionKind`, `PointerAction`, `PointerActionKind`,
|
|
`ScrollDirection`, `Momentum`, `SelectOption`, `Key`,
|
|
`KeyModifier`, `KeyName`, `ParsedKey`, `ENGINE_SPI_VERSION`,
|
|
`LOCATOR_ACTION_KINDS`, `POINTER_ACTION_KINDS`, `KEY_MODIFIERS`,
|
|
`KEY_NAMES`, `parseKey`, `ENGINE_ERROR_CODES`, `RETRYABLE_ENGINE_ERROR_CODES`,
|
|
`Secret`, `isSecret`, `TextMatch`, `JsonValue`, `JsonValueRules`,
|
|
`ResolveExpressionOptions`, `PollConditionOptions`, `WithoutUndefined`,
|
|
`FixtureOperation`, `OBSERVED_NAME_LIMIT` (256) and `OBSERVED_TEXT_LIMIT`
|
|
(512) are exported from `e2e/engine`. An engine that cuts an observed field cuts it at exactly its
|
|
limit: `name` at the name limit, `text` and any `value` or `selection` it
|
|
cuts at the text limit. So a field exactly at its limit may have been cut; a
|
|
shorter one, or a longer one the engine left whole, is complete. Keep the start and cut the end: the
|
|
runner redacts the leading part of a secret that a field exactly at its limit
|
|
ends with, which a whole-value match cannot see. Single-node reads are
|
|
unbounded and always carry the full value. The `selector` expression kind carries a
|
|
platform-native selector string; `frame` scopes a query into a nested
|
|
document.
|
|
|
|
```ts
|
|
/** One node of an observation tree. */
|
|
export interface SemanticNode {
|
|
/** Node reference, valid against the observation it came from. */
|
|
readonly ref: NodeRef;
|
|
/** Semantic role. */
|
|
readonly role?: string;
|
|
/** Accessible name, cut at `OBSERVED_NAME_LIMIT`. */
|
|
readonly name?: string;
|
|
/** Visible text, cut at `OBSERVED_TEXT_LIMIT`. */
|
|
readonly text?: string;
|
|
/** Current input value; never reported for a secure field. */
|
|
readonly value?: string;
|
|
/**
|
|
* Text selected inside the node, for the focused field or editing host
|
|
* whose selection the platform reports; absent when the selection is
|
|
* collapsed, and never reported for a secure field. Cut at `OBSERVED_TEXT_LIMIT`.
|
|
*/
|
|
readonly selection?: string;
|
|
/**
|
|
* The node's test id, when the platform gives it one: the value of the
|
|
* project's test-id attribute on a document platform, an accessibility
|
|
* identifier on iOS, a resource id on Android, an automation id on a
|
|
* desktop. The `testId` query resolves against exactly this field, so the
|
|
* harness never knows where a platform keeps it.
|
|
*/
|
|
readonly testId?: string;
|
|
/** What a field takes, deciding which secrets may fill it. */
|
|
readonly inputPurpose?: 'username' | 'password' | 'one-time-code' | 'generic-secret' | 'none';
|
|
/**
|
|
* Boolean states the platform reports. `pressed` is a toggle button's
|
|
* pressed state; `secure` marks a field whose value is never observed.
|
|
*/
|
|
readonly states?: Readonly<
|
|
Partial<
|
|
Record<
|
|
'checked' | 'disabled' | 'selected' | 'expanded' | 'pressed' | 'focused' | 'hidden' | 'secure',
|
|
boolean
|
|
>
|
|
>
|
|
>;
|
|
/** Heading level of a heading node (`1` through `6` on a document platform); absent elsewhere. */
|
|
readonly level?: number;
|
|
/** Platform attributes, what `getAttribute` reads. */
|
|
readonly attributes?: Readonly<Record<string, string>>;
|
|
/**
|
|
* The node's box in the top-level viewport's CSS pixels (`ViewportPoint`
|
|
* space), for every node, including those inside nested documents: an
|
|
* engine that measures a child document against its own viewport shifts
|
|
* the boxes by the boundary element's before reporting them.
|
|
*/
|
|
readonly rect?: {
|
|
readonly x: number;
|
|
readonly y: number;
|
|
readonly width: number;
|
|
readonly height: number;
|
|
};
|
|
/**
|
|
* Platform selector that addresses this node within its own document, when
|
|
* the platform has one. It is structural, not semantic: it survives the
|
|
* content changes that rename a node, and a runner may store it to re-find
|
|
* the node cheaply on a later run. It is never part of the node's identity,
|
|
* so whatever it resolves to is still checked before it is used.
|
|
*
|
|
* It SHOULD be anchored on an attribute naming the node or one of its
|
|
* ancestors, and SHOULD be absent rather than positional all the way to the
|
|
* document root: such a path is shifted by anything inserted above the node,
|
|
* so it does not survive to the later run it exists for.
|
|
*/
|
|
readonly selector?: string;
|
|
/**
|
|
* Enclosing nested-document chain as selectors of each boundary node,
|
|
* outermost first. Absent for nodes in the main document.
|
|
*/
|
|
readonly framePath?: readonly string[];
|
|
/** Child nodes. */
|
|
readonly children?: readonly SemanticNode[];
|
|
}
|
|
|
|
/** The `screen` query kinds. */
|
|
export type QueryKind = 'role' | 'label' | 'placeholder' | 'text' | 'displayValue' | 'testId';
|
|
```
|
|
|
|
A `SemanticQuery` with `visible: true` must match only nodes whose
|
|
`states.hidden` is false, by the same predicate the engine reports on
|
|
`SemanticNode`, so `getByText('x', { visible: true })` and `toBeVisible()`
|
|
never disagree. Apply it wherever the query sits in the expression, under a
|
|
scope, filter, or index too.
|