mirror of
https://github.com/tester-army/e2e.git
synced 2026-11-22 18:19:48 +08:00
* feat(expect)!: expect(fn).toPass() and expect.poll intervals
A block of assertions can be retried with `expect(fn).toPass({ timeout,
intervals })`, including the `.not.toPass()` and `expect.soft(fn).toPass()`
forms. Defaults are set with a top-level `toPass` config key. Both APIs
run on one retry loop.
BREAKING CHANGE: `expect.poll` no longer takes `interval`. It takes
`intervals`, a list of pauses whose last entry repeats. The default pauses
are `[100, 250, 500, 1000]` ms where they were a flat 100 ms.
Steps still running in a poll read or a toPass try are cancelled when the
try fails or is cut at its deadline. Later step calls from that try are
refused.
When a retry reaches the phase deadline, the runner's timeout is the
reported error. The last retry error is kept under `secondaryErrors`.
In `beforeAll`, `afterAll`, and worker fixture teardown, both APIs are
bounded by the hook deadline and use the config defaults.
A call `timeout` above the timer limit is `INVALID_CONFIG`.
* fix(expect): let skips through a retry and follow a moved deadline
A `test.skip()` or `test.fixme()` thrown inside a toPass block or a poll
read ends the retry right away and propagates unchanged.
The phase deadline is read on every pass, so a `test.setTimeout()` call
during a retry takes effect. The call's timeout stays fixed.
Options for `expect.poll` and `toPass` must be a plain object with known
keys.
* fix(expect): deprecate the interval option of expect.poll
`interval` works again as an alias for `intervals: [interval]`, following
the deprecation window in CONTRIBUTING.md. A warning that names `intervals`
is printed the first time it is used in a process. Passing both options is
`INVALID_CONFIG`.
* fix(expect): take a final sample at the deadline and refuse late ones
When a pause is cut short by the deadline, one more sample is taken. Its
result counts only if it settles without waiting past the deadline. A
sample that overran the time it had when it started no longer passes.
Call options for `intervals` use the config's rule, positive integers.
* fix(expect): drop the final sample at the deadline
The extra read counted only when it settled in its starting turn, so it
helped a synchronous read and never one that awaits the app or the
network. When the next pause would cross the deadline, the retry ends
without another read, as Playwright's loop does.
* fix(expect): keep a late try's skip and error
A try that settles after its time still does not pass. A `test.skip()` it
threw now reports `skipped`, and an error it threw is the `last:` line of
the timeout message.
1960 lines
73 KiB
Plaintext
1960 lines
73 KiB
Plaintext
---
|
|
title: E2EConfig
|
|
description: Every configuration key, its default, its valid range, and the environment variables.
|
|
"og:image": "https://e2e.com/docs/images/og/reference/config.png?v=6c4dead1fc"
|
|
"twitter:image": "https://e2e.com/docs/images/og/reference/config.png?v=6c4dead1fc"
|
|
---
|
|
|
|
`e2e.config.ts` configures the runner: the targets tests run on, the app each
|
|
one tests, the agents, and the replay cache. Export an object with
|
|
`satisfies E2EConfig`, which checks its shape and keeps its inferred types.
|
|
The runner validates the config again at load, including ranges and rules
|
|
TypeScript cannot check.
|
|
For the options most projects change, see [Configure the project](/quickstart#configure-the-project).
|
|
|
|
```ts title="e2e.config.ts"
|
|
import type { E2EConfig } from 'e2e';
|
|
import { web } from '@e2e-dev/web';
|
|
|
|
export default {
|
|
targets: [
|
|
{
|
|
name: 'web',
|
|
engine: web({ browser: 'chromium' }),
|
|
app: { url: process.env.APP_URL ?? 'http://localhost:3000' },
|
|
},
|
|
],
|
|
} satisfies E2EConfig;
|
|
```
|
|
|
|
## Reference
|
|
|
|
### Top level
|
|
|
|
| Option | Type | Default |
|
|
| --- | --- | --- |
|
|
| [`projectId`](#projectid) | `string` | `package.json` name |
|
|
| [`targets`](#targets) | `Target[]` | required |
|
|
| [`tests`](#tests) | `string \| string[]` | `'tests/**/*.e2e.ts'` |
|
|
| [`timeout`](#timeout) | `number` | `120000` |
|
|
| [`launchTimeout`](#launchtimeout) | `number` | `60000` |
|
|
| [`actionTimeout`](#actiontimeout) | `number` | `30000` |
|
|
| [`assertionTimeout`](#assertiontimeout) | `number` | `5000` |
|
|
| [`toPass`](#topass) | `PassOptions` | [See defaults](#topass) |
|
|
| [`cleanupTimeout`](#cleanuptimeout) | `number` | `30000` |
|
|
| [`retries`](#retries) | `number` | `0`, `1` in CI |
|
|
| [`failOnSkippedFailure`](#failonskippedfailure) | `boolean` | `false` |
|
|
| [`maxDuration`](#maxduration) | `number` | - |
|
|
| [`workers`](#workers) | `number` | half the cores, `1` in CI |
|
|
| [`reporters`](#reporters) | `(BuiltinReporter \| Reporter)[]` | `['list']` |
|
|
| [`output`](#output) | `string` | `'.e2e'` |
|
|
| [`artifacts`](#artifacts) | `ArtifactsConfig` | local only |
|
|
| [`trace`](#trace) | `RecordingMode` | `'retain-on-failure'` |
|
|
| [`video`](#video) | `RecordingMode` | `'off'` |
|
|
| [`agents`](#agents) | `Record<string, AgentConfig>` | `default`, no model |
|
|
| [`filters`](#filters) | `TestFilter[]` | - |
|
|
| [`cache`](#cache) | `CacheMode \| CacheConfig` | `'read-write'`, `'read-only'` in CI |
|
|
| [`credentials`](#credentials) | `Record<string, CredentialConfig>` | - |
|
|
| [`secrets`](#secrets) | `Record<string, SecretConfig>` | - |
|
|
|
|
The full declaration is [`E2EConfig`](#e2econfig).
|
|
|
|
- An unknown key at any level is `INVALID_CONFIG`. There is no top-level
|
|
`app` key: each target declares [the app it tests](#the-app-under-test).
|
|
- Every numeric value must be a safe integer.
|
|
- Every timeout in milliseconds, here and in `app.command` and `agents`, is at
|
|
most 2147483647 (about 24 days). A larger one is `INVALID_CONFIG`.
|
|
- An optional key takes a value or is left out. Under
|
|
`exactOptionalPropertyTypes`, `undefined` is a type error, so
|
|
`store: process.env.CACHE_URL && createStore(...)` fails to compile. At
|
|
runtime, `undefined` counts as unset.
|
|
- The target's `app` fields also accept `undefined` in their types, so they
|
|
can read `process.env` directly. Spread any other key that depends on the
|
|
environment in only when it has a value:
|
|
|
|
```ts
|
|
export default {
|
|
targets: [{ engine: web(), app: { url: 'http://localhost:3000' } }],
|
|
...(process.env.CACHE_URL === undefined ? {} : { cache: { store: createStore(process.env.CACHE_URL) } }),
|
|
} satisfies E2EConfig;
|
|
```
|
|
|
|
#### `projectId`
|
|
|
|
A stable id for the project.
|
|
|
|
```ts
|
|
projectId: 'acme-web', // default: the root package.json name
|
|
```
|
|
|
|
- 1 through 256 characters.
|
|
- Without a `name` in the root `package.json`, it falls back to
|
|
`unportable-<sha256 of the project root, 32 hex>`.
|
|
|
|
#### `tests`
|
|
|
|
The test file globs, relative to the project root.
|
|
|
|
```ts
|
|
tests: ['tests/**/*.e2e.ts', '!tests/wip/**'], // default: 'tests/**/*.e2e.ts'
|
|
```
|
|
|
|
- A glob takes `*`, `?`, and a whole `**` segment. `?` matches one Unicode
|
|
code point, including an emoji.
|
|
- An entry that starts with `!` excludes. The example selects every test file
|
|
except those under `tests/wip/`.
|
|
- A file is selected when an including glob matches it and no exclusion does,
|
|
whatever the order of the list.
|
|
- Discovery visits only directories that could match. It never enters a
|
|
directory an exclusion takes whole, skips `node_modules`, and ignores
|
|
symlinks.
|
|
- A dot directory is included only when a glob segment starts with a dot.
|
|
- The default selects TypeScript only. For JavaScript tests, set
|
|
`'tests/**/*.e2e.js'` ([JavaScript](#javascript)).
|
|
|
|
| Throws | When |
|
|
| --- | --- |
|
|
| `INVALID_CONFIG` | The list holds only exclusions. |
|
|
| `INVALID_GLOB` | An entry with no wildcard names a directory (`'!tests/wip'`). A glob names files; the error names the glob to write (`'!tests/wip/**'`). |
|
|
| `NO_TESTS` | A file passed to `e2e run` is taken out by an exclusion. The error names that entry. |
|
|
|
|
#### `timeout`
|
|
|
|
The deadline of one test attempt, in milliseconds.
|
|
|
|
```ts
|
|
timeout: 180_000, // default: 120000
|
|
```
|
|
|
|
#### `launchTimeout`
|
|
|
|
The budget for engine init and attempt start, in milliseconds.
|
|
|
|
```ts
|
|
launchTimeout: 90_000, // default: 60000
|
|
```
|
|
|
|
#### `actionTimeout`
|
|
|
|
The budget of every engine operation, in milliseconds.
|
|
|
|
```ts
|
|
actionTimeout: 10_000, // default: 30000
|
|
```
|
|
|
|
- A targeted agent action is bounded by the smaller of `actionTimeout` and 15 s.
|
|
- A call's own `timeout` option wins over it. `timeout: 0` sets no deadline
|
|
for that call; the test timeout, or `cleanupTimeout` in `afterEach`, still
|
|
bounds it.
|
|
- `--action-timeout <ms>` overrides it for one run.
|
|
|
|
#### `assertionTimeout`
|
|
|
|
The default `expect` deadline, in milliseconds.
|
|
|
|
```ts
|
|
assertionTimeout: 10_000, // default: 5000
|
|
```
|
|
|
|
- A matcher's own `timeout` option wins over it. `timeout: 0` sets no deadline
|
|
for that matcher; the test timeout, or `cleanupTimeout` in `afterEach`,
|
|
still bounds it.
|
|
- `--assertion-timeout <ms>` overrides it for one run.
|
|
|
|
#### `toPass`
|
|
|
|
The default timeout and pauses for [retrying a block](/reference/expect#retrying-a-block).
|
|
Each call option takes precedence over the corresponding config field.
|
|
|
|
```ts
|
|
toPass: { timeout: 10_000, intervals: [100, 250, 500] },
|
|
```
|
|
|
|
| Option | Type | Default |
|
|
| --- | --- | --- |
|
|
| `timeout` | `number` | `0` |
|
|
| `intervals` | `readonly number[]` | `[100, 250, 500, 1000]` |
|
|
|
|
- `timeout` is an integer in milliseconds from 0 through 2147483647.
|
|
At `0`, retries are bounded only by the test or hook deadline.
|
|
- `intervals` is a non-empty list of positive integers in milliseconds.
|
|
The last pause is repeated.
|
|
|
|
Invalid values or unknown keys are reported as `INVALID_CONFIG` at config load.
|
|
|
|
In `beforeAll`, `afterAll`, and worker fixture teardown, the configured
|
|
defaults apply to `expect.poll` and `toPass`. Retries are capped by the
|
|
hook deadline.
|
|
|
|
#### `cleanupTimeout`
|
|
|
|
The budget of each `afterEach` hook, fixture teardown, and engine cleanup, in
|
|
milliseconds.
|
|
|
|
```ts
|
|
cleanupTimeout: 60_000, // default: 30000
|
|
```
|
|
|
|
#### `retries`
|
|
|
|
Retries per test.
|
|
|
|
```ts
|
|
retries: 2, // default: 1 in CI, else 0
|
|
```
|
|
|
|
- 0 through 10.
|
|
|
|
#### `failOnSkippedFailure`
|
|
|
|
Fails the run with exit code 1 when a test skips a retry after an earlier
|
|
failed attempt.
|
|
|
|
```ts
|
|
failOnSkippedFailure: true, // default: false
|
|
```
|
|
|
|
- The test keeps its `skipped` status and reason.
|
|
- Its failures remain in the report, and the CLI shows them under
|
|
`Skipped After Failure` with either setting.
|
|
- Clean skips, successful retries, and engine cleanup diagnostics do not
|
|
trigger it.
|
|
|
|
<Note>
|
|
A skip after a failed `expect.soft` needs no option: it
|
|
[fails the attempt](/reference/test#members).
|
|
</Note>
|
|
|
|
#### `maxDuration`
|
|
|
|
The wall-clock budget for running the tests, in milliseconds. It stops a run
|
|
that takes too long.
|
|
|
|
```ts
|
|
export default { targets, maxDuration: 900_000 } satisfies E2EConfig; // default: none
|
|
```
|
|
|
|
- At most 2147483647 (about 24 days).
|
|
- The clock starts when the tests start. The app command's
|
|
[`startupTimeout`](#command) bounds app startup instead.
|
|
- Once the time is spent, the runner dispatches nothing more. The running
|
|
tests end as `interrupted`, and the tests not started are skipped with cause
|
|
`time-limit`. The run records a `RUN_TIMEOUT` error and exits 1.
|
|
- `--last-failed` runs the interrupted and skipped tests next.
|
|
- `--max-duration <ms>` overrides it for one run.
|
|
- `e2e explore` ignores it and keeps its own `--timeout`.
|
|
|
|
#### `workers`
|
|
|
|
Parallel workers.
|
|
|
|
```ts
|
|
workers: 4, // default: 1 in CI, else half the cores
|
|
```
|
|
|
|
- 1 through 1024.
|
|
- Locally the default is `max(1, floor(availableParallelism() / 2))`.
|
|
- An engine may cap it lower: `workers` never exceeds what a target's engine
|
|
declares.
|
|
- With more than one worker, each target's files start longest first by the
|
|
[timings history](/ci#cache-the-timings-history). A file with no time starts
|
|
before the timed ones. With `--changed-since`, the likeliest files start
|
|
before that, as
|
|
[Start the likeliest first](/filters#start-the-likeliest-first) says.
|
|
|
|
#### `reporters`
|
|
|
|
The reporters of the run: built-in ids, and reporter objects such as the
|
|
factories in `e2e/reporters`.
|
|
|
|
```ts
|
|
reporters: ['list', 'junit'], // default: ['list']
|
|
```
|
|
|
|
| Value | Does |
|
|
| --- | --- |
|
|
| `'list'`, `list()` | Terminal progress, failures, and a run summary. |
|
|
| `'dot'`, `dot()` | Compact terminal progress, failures, and a run summary. Excludes `list`: the two together are `INVALID_CONFIG`. |
|
|
| `'junit'`, `junit()` | Writes `<output>/junit.xml`, or `outputFile`. |
|
|
| `'markdown'`, `markdown()` | Writes `<output>/summary.md`, or `outputFile`. |
|
|
| `'json'`, `json()` | Prints the report on stdout. Beside it, `list` or `dot` prints to stderr unless given a `stream`. |
|
|
| another `Reporter` object | Runs beside the built-in ones. `--reporter` never removes it. |
|
|
|
|
An id is its factory called with no options. An unknown id is
|
|
`INVALID_CONFIG`. See [Reporters](/reference/reporters).
|
|
|
|
### targets
|
|
|
|
A target is one named surface tests run on: the app under test and the
|
|
[engine](#engine) that drives it. `targets` is required and holds at least
|
|
one.
|
|
|
|
```ts title="e2e.config.ts"
|
|
targets: [
|
|
{ name: 'chromium', engine: web(), app: { url: 'http://localhost:3000' } },
|
|
{ name: 'firefox', engine: web({ browser: 'firefox' }), app: { url: 'http://localhost:3000' } },
|
|
],
|
|
```
|
|
|
|
| Option | Type | Default |
|
|
| --- | --- | --- |
|
|
| [`name`](#name) | `string` | the platform |
|
|
| [`platform`](#platform) | `string` | the engine's platform |
|
|
| [`engine`](#engine) | `EngineHandle` | - |
|
|
| [`app`](#the-app-under-test) | `TargetApp` | - |
|
|
| [`trace`](#trace) | `RecordingMode` | the config's `trace` |
|
|
| [`video`](#video) | `RecordingMode` | the config's `video` |
|
|
|
|
- `{ engine: web(), app: { url } }` is a complete target.
|
|
- A test whose `requires` names a capability the target lacks is skipped for
|
|
it. A `test.setup` that a selected test depends on must be runnable there,
|
|
or selection fails with `COLLECTION_ERROR`.
|
|
|
|
The full declaration is [`Target`](#target).
|
|
|
|
#### `name`
|
|
|
|
The id `--target`, reports, sessions, and artifact paths use.
|
|
|
|
```ts
|
|
name: 'chromium', // default: the platform
|
|
```
|
|
|
|
- ASCII letters, digits, `_`, `-`, and `.`, not only dots (`.` and `..` are
|
|
path segments).
|
|
- Unique across targets. Two targets on one platform need names.
|
|
|
|
#### `platform`
|
|
|
|
The platform label.
|
|
|
|
```ts
|
|
platform: 'web', // default: inherited from the engine
|
|
```
|
|
|
|
- Required for a target without an engine.
|
|
- When the target and its engine both name one, they must agree.
|
|
|
|
#### `engine`
|
|
|
|
The engine driving the surface: `web(...)`, `mobile(...)`, or any
|
|
`defineEngine` handle.
|
|
|
|
```ts
|
|
engine: web({ browser: 'firefox' }),
|
|
```
|
|
|
|
A target without an engine runs through agent tools only. A target with `app`
|
|
and no engine is `INVALID_CONFIG`.
|
|
|
|
### The app under test
|
|
|
|
A target declares the app it tests in `app`; the engine only drives it. A
|
|
browser target names the URL it opens, a device target the installed app or
|
|
the build it launches. The runner resolves `app` at config load.
|
|
|
|
```ts title="e2e.config.ts"
|
|
targets: [
|
|
{
|
|
name: 'web',
|
|
engine: web({ viewport: { width: 1440, height: 900 } }),
|
|
app: {
|
|
url: 'http://localhost:3000',
|
|
command: { executable: 'npm', args: ['run', 'dev'] },
|
|
readyUrl: 'http://localhost:3000/health',
|
|
},
|
|
},
|
|
{
|
|
name: 'ios',
|
|
engine: mobile({ platform: 'ios' }),
|
|
app: {
|
|
bundleId: 'com.example.app',
|
|
appPath: './build/App.app',
|
|
launchArguments: ['-e2e', 'YES'],
|
|
permissions: { camera: 'grant' },
|
|
},
|
|
},
|
|
],
|
|
```
|
|
|
|
| Option | Type | Default |
|
|
| --- | --- | --- |
|
|
| [`url`](#url) | `string` | - |
|
|
| [`bundleId`](#bundleid) | `string` | - |
|
|
| [`appPath`](#apppath) | `string` | - |
|
|
| [`linkBase`](#linkbase) | `string` | - |
|
|
| [`identity`](#identity) | `string` | derived from the app |
|
|
| [`environment`](#environment) | `'test' \| 'staging' \| 'production'` | `'test'` or `'production'` |
|
|
| [`launchArguments`](#launcharguments) | `string[]` | - |
|
|
| [`permissions`](#permissions) | `Record<string, AppPermissionState>` | - |
|
|
| [`command`](#command) | `CommandConfig` | - |
|
|
| [`readyUrl`](#readyurl) | `string` | `url` |
|
|
|
|
Each engine checks the keys at config load, with `INVALID_CONFIG` naming the
|
|
target. A target with `app` needs an [engine](#engine).
|
|
|
|
- A `web()` target needs `app.url` and refuses `bundleId`, `appPath`, and
|
|
`launchArguments`.
|
|
- A `mobile()` target needs `app.bundleId` or `app.appPath`. `app.url` on a
|
|
mobile target is not supported yet.
|
|
- `app.linkBase` needs an engine that opens links, such as `mobile()`.
|
|
- An unknown key in `app` or `app.command` fails with a did-you-mean.
|
|
|
|
<Note>
|
|
The old spellings are unknown keys and fail at config load with
|
|
`INVALID_CONFIG`: `web({ url })` and the other app options of `web()` and
|
|
`mobile()`, `mobile({ app })` (now `app.bundleId`), an app key directly on the
|
|
target, a top-level `app` or `url`, and [`services`](#services) anywhere.
|
|
</Note>
|
|
|
|
#### `url`
|
|
|
|
The base URL: `app.open()` opens it and relative navigation resolves against
|
|
it.
|
|
|
|
```ts
|
|
url: 'http://localhost:3000',
|
|
```
|
|
|
|
- WHATWG-normalized; no userinfo, query, or fragment.
|
|
- A missing scheme becomes `https://`, or `http://` for a loopback host.
|
|
- Plain HTTP is accepted for loopback hosts only.
|
|
- Port `0` on `127.0.0.1` or `[::1]` asks for a [free port](#free-ports).
|
|
- Its origin scopes engine request credentials, and its site scopes observed
|
|
frames: see [the app's site](#the-apps-site).
|
|
|
|
| Throws | When |
|
|
| --- | --- |
|
|
| `INVALID_APP_URL` | Port `0` on any host other than `127.0.0.1` or `[::1]`, `localhost` included. |
|
|
| `INVALID_CONFIG` | A port-0 `url` with nothing to start on it (no `app.command`). |
|
|
|
|
#### `bundleId`
|
|
|
|
The installed app a device launches: a bundle id, an Android package name, or
|
|
a display name the device resolves (`Settings`).
|
|
|
|
```ts
|
|
bundleId: 'com.example.app',
|
|
```
|
|
|
|
A link is `INVALID_CONFIG`: a test opens one with `device.openLink`.
|
|
|
|
#### `appPath`
|
|
|
|
The build the suite runs against, an iOS `.app` bundle or an Android `.apk`,
|
|
resolved against the project root.
|
|
|
|
```ts
|
|
appPath: './build/App.app',
|
|
```
|
|
|
|
Without `bundleId`, the app it installs is the one launched. See
|
|
[the device engine](/reference/mobile#engine) for when it is installed.
|
|
|
|
#### `linkBase`
|
|
|
|
Where `app.open(path)` opens on a device target: the path joins the link base,
|
|
and the link opens in the pinned app.
|
|
|
|
```ts
|
|
linkBase: 'https://shop.example.com', // app.open('/orders/42') opens https://shop.example.com/orders/42
|
|
linkBase: 'shopapp://', // app.open('/orders/42') opens shopapp://orders/42
|
|
```
|
|
|
|
- One slash joins the base and the path; none follows a bare scheme
|
|
(`shopapp://`).
|
|
- An absolute link passed to `app.open` opens as given, a custom app scheme
|
|
included; `file:`, `data:`, `javascript:`, `view-source:`, `blob:`, and
|
|
`filesystem:` links are `POLICY_DENIED`.
|
|
- Without `linkBase`, `app.open()` on a device launches the app and takes no
|
|
path (`APP_URL_REQUIRED`).
|
|
|
|
| Throws | When |
|
|
| --- | --- |
|
|
| `INVALID_CONFIG` | The engine opens no links (a `web()` target), the value is not an absolute URL or a scheme, it has a query or fragment (an empty `?` or `#` included), it has leading or trailing whitespace, or it names `file:`, `data:`, `javascript:`, `view-source:`, `blob:`, or `filesystem:`. |
|
|
| `POLICY_DENIED` | `app.open` got an absolute link with one of those schemes. |
|
|
|
|
#### `identity`
|
|
|
|
The stable identity keying replay cache and session entries.
|
|
|
|
```ts
|
|
identity: 'checkout-web', // default: the URL's origin and base path, else bundleId, else appPath
|
|
```
|
|
|
|
Set a stable value to reuse entries across preview URLs or versioned build
|
|
paths. With a [free port](#free-ports), the default derives from the declared
|
|
`:0`, so cache and session entries do not churn with the port.
|
|
|
|
<Note>
|
|
Different apps must have different identities. A shared identity replays
|
|
recordings across them.
|
|
</Note>
|
|
|
|
#### `environment`
|
|
|
|
Labels the target in the report and joins the cache and session identity
|
|
digest. It never gates a run.
|
|
|
|
```ts
|
|
environment: 'staging', // default: 'test' for a local host or no URL, else 'production'
|
|
```
|
|
|
|
- `'test'`, `'staging'`, or `'production'`.
|
|
- A local host is a loopback, `.localhost`, or `.test` host.
|
|
|
|
#### `launchArguments`
|
|
|
|
Arguments every fresh launch passes the app: `app.open()`, `app.restart()`,
|
|
and `app.clearState()`.
|
|
|
|
```ts
|
|
launchArguments: ['-e2e', 'YES'],
|
|
```
|
|
|
|
Arguments that select a build mode change what the app is: give each mode its
|
|
own `identity`.
|
|
|
|
#### `permissions`
|
|
|
|
Permissions the app holds, by name.
|
|
|
|
```ts
|
|
permissions: { camera: 'grant' },
|
|
```
|
|
|
|
Each is `'grant'`, `'deny'`, or `'reset'`.
|
|
|
|
- A `mobile()` target sets each one before every fresh launch. See
|
|
[permissions](/reference/mobile#engine).
|
|
- A `web()` target takes `'grant'` only and grants it to every page of the
|
|
attempt, on any origin. Headless Chromium refuses a permission left out. See
|
|
[Granting browser permissions](/web#granting-browser-permissions).
|
|
|
|
| Browser | Names |
|
|
| --- | --- |
|
|
| `chromium` | `accelerometer`, `ambient-light-sensor`, `background-sync`, `camera`, `clipboard-read`, `clipboard-write`, `geolocation`, `gyroscope`, `local-fonts`, `local-network-access`, `magnetometer`, `microphone`, `midi`, `midi-sysex`, `notifications`, `payment-handler`, `screen-wake-lock`, `storage-access` |
|
|
| `firefox` | `geolocation`, `local-network-access`, `notifications`, `persistent-storage`, `push`, `screen-wake-lock` |
|
|
| `webkit` | `camera`, `clipboard-read`, `geolocation`, `microphone`, `notifications`, `screen-wake-lock` |
|
|
|
|
A browser provider or `connect` takes the `chromium` names: the engine drives
|
|
it over CDP, Lightpanda included.
|
|
|
|
#### `command`
|
|
|
|
The process the runner starts before the first test and stops on every exit
|
|
path: a dev server, or Metro for a debug build.
|
|
|
|
```ts
|
|
command: {
|
|
executable: 'pnpm',
|
|
args: ['dev', '--port', '{port}'],
|
|
inheritEnv: ['STRIPE_*'],
|
|
startupTimeout: 120_000, // default: 60000
|
|
},
|
|
```
|
|
|
|
| Field | Example | Default |
|
|
| --- | --- | --- |
|
|
| `executable` | `'npm'` | required |
|
|
| `args` | `['run', 'dev']` | none |
|
|
| `cwd` | `'apps/web'` | the project root |
|
|
| `env` | `{ PORT: '{port}' }` | none |
|
|
| `inheritEnv` | `['STRIPE_*']` | none |
|
|
| `startupTimeout` | `120_000` | `60000` |
|
|
| `shutdownTimeout` | `5_000` | `10000` |
|
|
| `log` | `'.e2e/logs/app.log'` | output discarded |
|
|
| `reuseExisting` | `true` | `false` |
|
|
|
|
- `executable` is resolved with `PATH` and never shell-interpreted.
|
|
- `args` pass verbatim. `{port}` in `args` and `env` values expands to the
|
|
port of `url`.
|
|
- `cwd` resolves from the project root.
|
|
- The child inherits only `PATH`, `HOME`, the temp-directory variables, the
|
|
variables `inheritEnv` matches, and `command.env`. Pass what the app needs
|
|
explicitly.
|
|
- `inheritEnv` takes exact names or prefixes ending in `*` (`'STRIPE_*'`). It
|
|
copies a variable only when it is set. `env` wins over an inherited value.
|
|
- Every `args` entry and `env` value is a string. An `undefined` `env` value
|
|
is dropped.
|
|
- `startupTimeout` is the budget of the [`readyUrl`](#readyurl) probe.
|
|
- `shutdownTimeout` is the grace period before force-kill.
|
|
- In `e2e run`, a command that exits after readiness fails the run with `APP_UNREACHABLE` if a dependent target has unfinished work.
|
|
This includes exit code 0 and termination by a signal.
|
|
The runner interrupts active tests and skips queued tests only for those targets.
|
|
Targets that have finished their work ignore the exit.
|
|
- `log` receives the process's stdout and stderr, appended, resolved from the
|
|
project root. It must stay inside the project root.
|
|
- With a `log`, startup timeouts and command exits quote the lines appended
|
|
since the command started. A log of up to 40 lines inside 8 KiB is quoted
|
|
whole. Otherwise the quote keeps up to 20 lines from each end, with
|
|
`[startup output omitted]` between them, and for a log past 8 KiB those
|
|
lines come from its first and last 4 KiB.
|
|
An empty or unreadable log produces `no output in <log>`.
|
|
A shared log can include lines from other processes.
|
|
The quote replaces `command.env` and `inheritEnv` values of at least 4 characters with `<secret:NAME>` markers.
|
|
Without `log`, these errors ask you to set one.
|
|
A command that fails to spawn reports the spawn error instead.
|
|
- `reuseExisting`: when the readiness URL already answers before the command
|
|
starts, the runner uses that process. It starts nothing and stops nothing.
|
|
CI ignores it, with a notice, and always starts the command.
|
|
- Targets declaring the same command share one process, probed at the first
|
|
declaring target's `readyUrl`.
|
|
- The runner never terminates a process it did not start.
|
|
|
|
| Throws | When |
|
|
| --- | --- |
|
|
| `INVALID_CONFIG` | A bare `'*'` in `inheritEnv`. |
|
|
| `INVALID_CONFIG` | An `args` entry or `env` value that is not a string, a number or a `secrets.get()` handle included. |
|
|
| `INVALID_CONFIG` | `reuseExisting` on an app whose `url` asks for port 0: a process already running cannot serve the port the run assigns. |
|
|
| `APP_URL_REQUIRED` | An `app.command` with neither `app.url` nor `app.readyUrl`, at config load. |
|
|
| `APP_ALREADY_RUNNING` | Without `reuseExisting`, a `readyUrl` that already answers. |
|
|
| `APP_UNREACHABLE` | The command fails to spawn, its `log` cannot be opened, `startupTimeout` expires, the command exits before readiness, or it exits unexpectedly during `e2e run`. |
|
|
|
|
<Warning>
|
|
`log` holds the raw, unredacted output. Keep it in an ignored directory.
|
|
</Warning>
|
|
|
|
The full declaration is [`CommandConfig`](#commandconfig).
|
|
|
|
#### `readyUrl`
|
|
|
|
The URL polled until `command` is ready, at a 200-499 status.
|
|
|
|
```ts
|
|
readyUrl: 'http://localhost:3000/health', // default: url
|
|
```
|
|
|
|
- Required without a `url`.
|
|
- `{port}` in it is the port of `url`.
|
|
|
|
#### The app's site
|
|
|
|
A test or the agent may open any http(s) URL, and `about:blank`. Every other
|
|
scheme (`file:`, `data:`, `javascript:`, `view-source:`, `blob:`) is
|
|
`POLICY_DENIED`. The site of `url` is its registrable domain:
|
|
`app.example.com` and `auth.example.com` are one site. Child frames off the
|
|
site stay out of observations.
|
|
|
|
The browser engine's [`headers`](/reference/web#headers) and
|
|
[`basicAuth`](/reference/web#basicauth) are scoped tighter, to the origin of
|
|
`url` (scheme, host, and port, the run's free port included) plus the
|
|
origins you list. See
|
|
[request credentials](/reference/security#request-credentials).
|
|
|
|
#### Free ports
|
|
|
|
A `url` on a literal loopback address with port `0` (`http://127.0.0.1:0`,
|
|
`http://[::1]:0`) asks the run for a free TCP port. The runner picks one when
|
|
the config loads. It substitutes the port wherever the declaration used it:
|
|
the base URL, the default `readyUrl`, and the report. The command must take
|
|
the port through `{port}` in `args` or `env`.
|
|
|
|
```ts title="e2e.config.ts"
|
|
engine: web(),
|
|
app: {
|
|
url: 'http://127.0.0.1:0',
|
|
command: { executable: 'pnpm', args: ['dev', '--port', '{port}'], env: { PORT: '{port}' } },
|
|
},
|
|
```
|
|
|
|
- `localhost:0` is [`INVALID_APP_URL`](#url): it may resolve to an address
|
|
other than the one the command binds. Write the address the command will
|
|
bind.
|
|
- Each target asking for port `0` gets its own port and its own process. Two
|
|
targets that declare the same command on a fixed port share one.
|
|
- The default [`identity`](#identity) derives from the declared `:0`.
|
|
- Tests read the allocated URL from [`app.baseUrl`](/reference/app).
|
|
|
|
#### services
|
|
|
|
<Note>
|
|
This version starts no dependency processes (a database, an API mock, a dev
|
|
server several targets share), and `services` is an unknown key wherever it
|
|
appears. Start them before the run, or have `app.command` start a script that
|
|
brings them up and serves the app. A services API returns in a later release.
|
|
</Note>
|
|
|
|
### output
|
|
|
|
The directory everything a run writes about itself goes under, resolved from
|
|
the project root.
|
|
|
|
```ts
|
|
output: 'test-results/e2e', // default: '.e2e'
|
|
```
|
|
|
|
`--output <dir>` on [`e2e run`](/reference/cli#e2e-run) and `e2e explore`
|
|
overrides it for one run.
|
|
|
|
| Path under the output | What |
|
|
| --- | --- |
|
|
| `report.json` | The canonical report; `--last-failed` reads the one here |
|
|
| `junit.xml`, `summary.md` | The `junit` and `markdown` reporters' files |
|
|
| `ai-trace.json` | `--ai-trace` |
|
|
| `timings.json` | How long each file took, smoothed over recent runs. A run with more than one worker starts the longest files first by it; see [Cache the timings history](/ci#cache-the-timings-history) |
|
|
| `results/` | One directory per test: its [trace page](/debugging#traces), `trace.md`, when the [`trace`](#trace) mode keeps one, and its screenshots, videos, downloads, and logs, one `attempt-<n>/` per attempt. Cleared once a run's tests start, so it holds the latest run's evidence and nothing its report no longer names. A `--last-failed` rerun keeps the files the report it reruns names and writes its own attempts under each test's `rerun-<n>/` |
|
|
| `sessions/` | The run's encrypted `test.setup` sessions, deleted when it ends |
|
|
| `videos/<session>/` | [`e2e mcp`](/reference/mcp#recording) session recordings |
|
|
|
|
- A run that stops before its tests start writes nothing here. Such stops are
|
|
a selection that matches no test (`NO_TESTS`, a mistyped `--grep`), a
|
|
collection error, a target that cannot record what it asks for
|
|
(`UNSUPPORTED_ARTIFACT`), `NO_LAST_RUN`, an app command that fails to start,
|
|
or an interrupt. The previous run's `report.json`, `ai-trace.json`, and
|
|
`results/` stay as they were, so `--last-failed` still reads the last run
|
|
that executed.
|
|
- An `e2e mcp` session writes its attempt's artifacts under `results/mcp/` and
|
|
never clears it.
|
|
- The replay cache stays in [`cache.dir`](#dir) whatever `output` says, so
|
|
moving the results never moves the recordings.
|
|
- `e2e init` writes `.gitignore` entries for the default `.e2e/` paths only.
|
|
With `output` set elsewhere, add that directory to `.gitignore` yourself.
|
|
|
|
A run clears and owns part of the output directory. So the config load fails
|
|
with `INVALID_CONFIG`, naming the reason, when the directory:
|
|
|
|
- is the project root, or outside it;
|
|
- is a file, or a path under one (a missing directory is created);
|
|
- is the cache directory (`cache.dir`, `--cache-dir`, or `E2E_CACHE_DIR`) or
|
|
inside it, or would hold it under `results/`, `sessions/`, or `videos/`;
|
|
- holds a directory a `tests` glob scans (`output: 'tests'` with the default
|
|
`tests/**/*.e2e.ts`).
|
|
|
|
Paths are compared with symlinks resolved. An absolute path spelled through a
|
|
symlink (`/tmp` for `/private/tmp` on macOS) names the same directory as its
|
|
target. A symlink whose target does not exist yet is followed too, so an
|
|
output through one that points outside the project root is refused.
|
|
|
|
<Note>
|
|
The removed `--artifacts <dir>` flag fails with the `--output` it maps to. The
|
|
report used to go to the directory's parent, so `--artifacts out/artifacts` is
|
|
`--output out`. A parent an output cannot be (the project root, or outside it)
|
|
maps to `--output .e2e`.
|
|
</Note>
|
|
|
|
### artifacts
|
|
|
|
`artifacts` says where artifacts go, not which are made. `{ store }` hands
|
|
every artifact to a host store as it is produced; without one, files stay
|
|
local only.
|
|
|
|
```ts
|
|
artifacts: {
|
|
store: {
|
|
async put(artifact) {
|
|
const key = await bucket.upload(artifact.path, artifact.bytes);
|
|
return { ref: key };
|
|
},
|
|
},
|
|
},
|
|
```
|
|
|
|
| Member | Returns |
|
|
| --- | --- |
|
|
| [`store.put`](#put) | `Promise<{ ref }>` |
|
|
| [`store.putLink`](#putlink) | `Promise<{ ref }>` |
|
|
|
|
- [`trace` and `video`](#recording) choose the recordings. The runner
|
|
captures a failure's screenshot and screen text whenever the engine can.
|
|
- Each worker constructs its own store from the config module.
|
|
- A failed `put` or `putLink` never fails the run.
|
|
|
|
The removed forms fail with `INVALID_CONFIG` naming what replaced them:
|
|
|
|
| Written | Now |
|
|
| --- | --- |
|
|
| a kinds list (`artifacts: ['screenshot', 'trace']` or `artifacts.kinds`) | the `trace` mode it meant |
|
|
| `artifacts.trace` with `record: 'retries'` | `trace: 'on-all-retries'`, read together with a kinds list that holds `trace`, so the pair still means retries only |
|
|
| `artifacts.video` | the `video` option |
|
|
|
|
The declarations are under [`ArtifactsConfig`](#artifactsconfig).
|
|
|
|
#### `put`
|
|
|
|
Receives every artifact once it is complete on disk and returns a reference
|
|
the report records as `ref` beside the local path.
|
|
|
|
```ts
|
|
async put(artifact) {
|
|
return { ref: await bucket.upload(artifact.path, artifact.bytes) };
|
|
},
|
|
```
|
|
|
|
- Required on every store.
|
|
- `artifact.redaction` is the report's label. A store that exports only what
|
|
the runner vouches for keeps back an `incomplete` artifact: a video or a
|
|
download the runner did not rewrite.
|
|
|
|
#### `putLink`
|
|
|
|
Receives a video a hosted service keeps (a
|
|
[provider's own recording](#what-a-video-writes)). Such a video is a link with
|
|
no bytes, so `put` never receives it.
|
|
|
|
```ts
|
|
async putLink(link) {
|
|
return { ref: await index.record(link.url, link.attemptId) };
|
|
},
|
|
```
|
|
|
|
- Optional. Called for each link once the attempt stopped the recording.
|
|
- Its `ref` lands on the report's artifact beside `url`.
|
|
- A passed attempt's link under `retain-on-failure` is never handed over,
|
|
since the report leaves it out too.
|
|
|
|
### Recording
|
|
|
|
`trace` and `video` choose what is recorded, with one set of modes.
|
|
|
|
```ts
|
|
trace: 'on',
|
|
video: 'retain-on-failure',
|
|
targets: [
|
|
{ name: 'chromium', engine: web(), app: { url } },
|
|
{ name: 'firefox', engine: web({ browser: 'firefox' }), app: { url }, video: 'on-first-retry' },
|
|
],
|
|
```
|
|
|
|
| Option | Type | Default |
|
|
| --- | --- | --- |
|
|
| [`trace`](#trace) | `RecordingMode` | `'retain-on-failure'` |
|
|
| [`video`](#video) | `RecordingMode` | `'off'` |
|
|
|
|
```ts
|
|
/** Which tests keep a trace page, or which attempts record a video. */
|
|
export type RecordingMode = 'off' | 'on' | 'retain-on-failure' | 'on-first-retry' | 'on-all-retries';
|
|
```
|
|
|
|
| Mode | Trace page | Video |
|
|
| --- | --- | --- |
|
|
| `off` | none | none |
|
|
| `on` | every test, passing ones included | every attempt |
|
|
| `retain-on-failure` | each failed, timed-out, interrupted, or flaky test | every attempt, keeping those that did not pass |
|
|
| `on-first-retry` | each test whose first retry ran | the first retry only, the second attempt |
|
|
| `on-all-retries` | each test that was retried | every attempt after the first |
|
|
|
|
The names are Playwright's `use.trace` and `use.video` modes. Under a retry
|
|
mode, a test with `retries: 0` records nothing, and neither does a failure
|
|
the runner does not retry, such as `LAUNCH_TIMEOUT`. When such a test runs,
|
|
the run prints one notice per kind at plan time: no traces (or videos) will
|
|
be recorded for it.
|
|
|
|
Four places set a mode, lowest first. Each replaces the one before it, for
|
|
`trace` and `video` alike:
|
|
|
|
1. `trace` or `video` at the top of the config.
|
|
2. A target's `trace` or `video`.
|
|
3. `--trace [mode]` or `--video [mode]` on [`e2e run`](/reference/cli#e2e-run),
|
|
over the config and every target. A bare flag is `on`.
|
|
4. A test's or describe's own [`trace` or `video` option](/reference/test#options),
|
|
innermost wins: `test('checkout', { video: 'on' }, ...)`.
|
|
|
|
- A serial group records as one unit, so its members cannot set `trace` or
|
|
`video` (a `COLLECTION_ERROR`, like `retries`). The value on the serial
|
|
`describe`, or on a group around it, applies to every member.
|
|
- `e2e explore` is one attempt and uses the target's mode, `--trace` and
|
|
`--video` included, so a retry mode records nothing there.
|
|
- An `e2e mcp` session writes no trace and never records video from the
|
|
configured mode. Its recordings are the ones `start_recording` and
|
|
`stop_recording` make.
|
|
- `trace` and `video`, at the top or on a target, enter neither the config
|
|
digest nor a replay's [cache key](/cache#what-must-match). Turning one on or
|
|
off never invalidates cached replays.
|
|
|
|
<Note>
|
|
The old trace spellings fail with `INVALID_CONFIG` naming the mode they
|
|
meant, wherever a mode goes (the config, a target, a test, `--trace`):
|
|
`'all'` and `{ record: 'all' }` are `'on'`, and `'retries'` and `{ record:
|
|
'retries' }` are `'on-all-retries'`.
|
|
</Note>
|
|
|
|
#### `trace`
|
|
|
|
Which tests keep a trace page.
|
|
|
|
```ts
|
|
trace: 'on', // default: 'retain-on-failure'
|
|
```
|
|
|
|
- The default is the same locally and in CI.
|
|
- A trace page is markdown, `trace.md` in the test's directory under
|
|
`<output>/results/`, such as
|
|
`.e2e/results/checkout-applies-the-coupon-1a2b3c4d5e6f7a8b/trace.md`: every
|
|
step with what it did, the cache's decisions, what the app logged, and the
|
|
screen at failure.
|
|
- It works on every engine. [Debugging](/debugging#traces) shows one.
|
|
|
|
#### `video`
|
|
|
|
Which attempts record a video.
|
|
|
|
```ts
|
|
video: 'retain-on-failure', // default: 'off'
|
|
```
|
|
|
|
- Where the mode is set decides what an engine that cannot record does. See
|
|
[Engines that cannot record](#engines-that-cannot-record).
|
|
- A video costs CPU and holds what the screen showed, secrets included, so it
|
|
is always asked for.
|
|
|
|
#### Engines that cannot record
|
|
|
|
Before any test starts, the run grades every target. Where a `video` mode
|
|
was set decides what an engine that cannot record video does:
|
|
|
|
- Set at the top of the config or with `--video`: the mode applies to the
|
|
targets whose engine can record it. The run prints one notice at plan time
|
|
naming the targets that cannot, and runs them without video.
|
|
- Set on a target, or on a test that will run on that target: the mode is
|
|
required. A target whose engine cannot record it fails the run with
|
|
`UNSUPPORTED_ARTIFACT`, naming the target and, for a test's, the test.
|
|
|
|
Only a mode that records on some attempt counts: a run whose tests all
|
|
resolve to `off`, or to a retry mode with `retries: 0`, asks nothing of the
|
|
engine.
|
|
|
|
#### What a video writes
|
|
|
|
| Recorder | Files under the attempt's artifact directory |
|
|
| --- | --- |
|
|
| Browser engine, Playwright screencast | `video/video.webm`, one segment per page the attempt showed, later ones as `video/video-part<n>.webm`. [`web({ screencast })`](/reference/web#options) sets the frame size and quality |
|
|
| Mobile engine, agent-device | `video/video.mp4` |
|
|
| A provider's own recording ([`BrowserProvider.record`](/browser#recording-on-the-provider), [`DeviceProvider.record`](/mobile#hosted-devices)) | the file the provider wrote, as `video/<file>`: Kernel's is `video/replay.mp4`. Or a link to a recording the service keeps, recorded with `url` in place of `path`; the runner never downloads it |
|
|
|
|
- Each segment carries `startedAt`, so `step.startedAt - artifact.startedAt`
|
|
is a step's offset into it.
|
|
- A recording masks nothing and is recorded with `redaction: 'incomplete'`.
|
|
|
|
<Note>
|
|
Under `retain-on-failure` a link is left out of the report of a passed
|
|
attempt, but the service still holds the recording.
|
|
</Note>
|
|
|
|
### agents
|
|
|
|
Named agents, each one plain object: the built-in agent's options, or
|
|
`{ executor }` for a custom brain.
|
|
|
|
```ts title="e2e.config.ts"
|
|
import type { E2EConfig } from 'e2e';
|
|
import { web } from '@e2e-dev/web';
|
|
import { gateway } from 'ai';
|
|
import { houseBrain } from './house-brain.ts';
|
|
|
|
export default {
|
|
targets: [{ engine: web(), app: { url: 'http://localhost:3000' } }],
|
|
agents: {
|
|
default: { model: gateway('openai/gpt-6-luna-fast') },
|
|
ux: { model: gateway('openai/gpt-6-luna-fast'), system: 'Check labels, contrast, and keyboard access.' },
|
|
thorough: { model: gateway('anthropic/claude-opus-5'), context: 'Verify every claim on screen.' },
|
|
buyer: {
|
|
model: gateway('openai/gpt-6-luna-fast'),
|
|
system: 'You are a first-time buyer.',
|
|
context: 'Plans are called tiers. Billing lives under Settings.',
|
|
maxSteps: 40,
|
|
},
|
|
house: { executor: houseBrain, model: gateway('openai/gpt-6-luna-fast'), maxModelCalls: 10 },
|
|
},
|
|
} satisfies E2EConfig;
|
|
```
|
|
|
|
| Option | Type | Default |
|
|
| --- | --- | --- |
|
|
| [`model`](#model) | `ModelInstance` | - |
|
|
| [`judge`](#judge) | `ModelInstance` | `model` |
|
|
| [`context`](#context) | `string` | - |
|
|
| [`system`](#system) | `string` | - |
|
|
| [`tools`](#tools) | `Record<string, AgentTool>` | - |
|
|
| [`executor`](#executor) | `StepExecutor` | the built-in agent |
|
|
| [`maxSteps`](#maxsteps) | `number` | `25` |
|
|
| [`maxModelCalls`](#maxmodelcalls) | `number` | `25` |
|
|
| [`judgmentTimeout`](#judgmenttimeout) | `number` | `30000` |
|
|
| [`maxObservationBytes`](#maxobservationbytes) | `number` | `262144` |
|
|
| [`maxInputTokens`](#maxinputtokens) | `number` | `64000` |
|
|
| [`providerOptions`](#provideroptions) | `ProviderOptions` | - |
|
|
|
|
- Tests use the `default` agent unless you select another: with
|
|
`e2e run --agent <name>`, a test or group option, or an `agent.*` call. A
|
|
test with `{ agent: ['buyer', 'admin'] }` runs once for each agent.
|
|
- `default` exists even when the config declares no agents, but has no model.
|
|
- Every entry starts from the built-in defaults. Agents do not inherit from
|
|
each other: `ux` above gets no `context` from `default`. A key left out of
|
|
an entry is the default value, never another entry's.
|
|
- Each worker constructs its own agents from this config module.
|
|
- Names may contain ASCII letters, numbers, `_`, `-`, or `.`, and are never
|
|
only dots: a name is an artifact path segment.
|
|
|
|
| Throws | When |
|
|
| --- | --- |
|
|
| `INVALID_CONFIG` | An unknown agent name on the `--agent` flag. |
|
|
| `COLLECTION_ERROR` | An unknown agent name on a test or group. |
|
|
| `INVALID_ARGUMENT` | An unknown agent name on a call. |
|
|
|
|
See [Agents and personas](/agents). The declarations are under
|
|
[`AgentConfig`](#agentconfig).
|
|
|
|
#### `model`
|
|
|
|
An AI SDK model instance. There is no implicit default.
|
|
|
|
```ts
|
|
model: gateway('openai/gpt-6-luna-fast'),
|
|
```
|
|
|
|
Construct the model in your config with an AI SDK provider. Every provider
|
|
that supports tool calls works, see
|
|
[AI SDK providers](/models#ai-sdk-providers). For subscription sign-in, use
|
|
the [`e2e/oauth/*` models](/subscriptions).
|
|
|
|
| Gateway | Constructor | Package | Reads |
|
|
| --- | --- | --- | --- |
|
|
| [Vercel AI Gateway](https://vercel.com/docs/ai-gateway) | `gateway('openai/gpt-6-luna-fast')` | `ai` | `AI_GATEWAY_API_KEY`, or a Vercel OIDC token without it |
|
|
| [OpenRouter](https://openrouter.ai/models) | `openrouter('openai/gpt-6-luna-fast')` | `@openrouter/ai-sdk-provider` | `OPENROUTER_API_KEY` |
|
|
| Any OpenAI-compatible endpoint | `createOpenAICompatible({ name, baseURL, apiKey? }).chatModel('llama3.2')` | `@ai-sdk/openai-compatible` | `apiKey`, when the endpoint needs one |
|
|
| A provider directly | `openai('gpt-6-luna')`, `anthropic(...)`, ... | `@ai-sdk/openai`, `@ai-sdk/anthropic`, ... | the provider's own variable |
|
|
|
|
- Reports record `provider` and `modelId` per step.
|
|
- Each worker constructs its own instance from the config module. A digest
|
|
that differs between the runner and a worker is
|
|
`CONFIG_NOT_DETERMINISTIC`.
|
|
- A string is `INVALID_CONFIG` naming the constructor to write.
|
|
- An agent never falls back to `default`'s model. See
|
|
[Resolution order](#resolution-order).
|
|
|
|
See [Models](/models) and [`ModelInstance`](#modelinstance).
|
|
|
|
#### `judge`
|
|
|
|
The model for non-exploring `assert`, `waitFor`, and `extract`. Built-in
|
|
exploring assertions use `agents.<name>.model`, even when `judge` is set.
|
|
|
|
```ts
|
|
judge: gateway('anthropic/claude-opus-5'), // default: model
|
|
```
|
|
|
|
- Judgments allow 8192 output tokens per call.
|
|
- A judgment sees the instruction, its params, the agent's context, and the current screen with its location.
|
|
|
|
See [A judge of its own](/models#a-judge-of-its-own).
|
|
|
|
#### `context`
|
|
|
|
What the app calls things, told to every model call this agent makes, act
|
|
turns and judgments alike.
|
|
|
|
```ts
|
|
context: 'Plans are called tiers. Billing lives under Settings.',
|
|
```
|
|
|
|
At most 16384 bytes.
|
|
|
|
#### `system`
|
|
|
|
How the acting agent should work, appended to the built-in rules. Only the
|
|
act loop reads it.
|
|
|
|
```ts
|
|
system: 'You are a first-time buyer.',
|
|
```
|
|
|
|
With `executor` it is `INVALID_CONFIG`: the executor brings its own prompt.
|
|
|
|
#### `tools`
|
|
|
|
Project tools from `defineTool` (`e2e/agent`), offered beside the built-in
|
|
toolset.
|
|
|
|
```ts
|
|
tools: { lookupOrder },
|
|
```
|
|
|
|
With `executor` it is `INVALID_CONFIG`: the executor brings its own tools.
|
|
|
|
#### `executor`
|
|
|
|
A custom brain `agent.act()` dispatches to instead of the built-in agent.
|
|
|
|
```ts
|
|
house: { executor: houseBrain, model: gateway('openai/gpt-6-luna-fast'), maxModelCalls: 10 },
|
|
```
|
|
|
|
- `model`, `judge`, `context`, and the budgets still apply; judgments need a
|
|
model. [`system`](#system) and [`tools`](#tools) beside it are
|
|
`INVALID_CONFIG`.
|
|
- `maxModelCalls` bounds the model turns of one step.
|
|
- A `model` or `judge` that differs from the one the executor brought is
|
|
`INVALID_CONFIG`.
|
|
|
|
See [Custom executors](/executors).
|
|
|
|
#### `maxSteps`
|
|
|
|
Committed actions per agent call.
|
|
|
|
```ts
|
|
maxSteps: 40, // default: 25
|
|
```
|
|
|
|
1 through 100.
|
|
|
|
#### `maxModelCalls`
|
|
|
|
Model requests per agent call.
|
|
|
|
```ts
|
|
maxModelCalls: 10, // default: 25
|
|
```
|
|
|
|
1 through 100.
|
|
|
|
#### `judgmentTimeout`
|
|
|
|
The deadline of one `assert`, `waitFor`, or `extract` call, in milliseconds.
|
|
|
|
```ts
|
|
judgmentTimeout: 60_000, // default: 30000
|
|
```
|
|
|
|
Raise it for a slow judge. Engine operations keep `actionTimeout`.
|
|
|
|
#### `maxObservationBytes`
|
|
|
|
The observation payload ceiling for act turns and judgments.
|
|
|
|
```ts
|
|
maxObservationBytes: 524_288, // default: 262144
|
|
```
|
|
|
|
1024 through 16777216.
|
|
|
|
#### `maxInputTokens`
|
|
|
|
Input tokens per model request.
|
|
|
|
```ts
|
|
maxInputTokens: 128_000, // default: 64000
|
|
```
|
|
|
|
- 1 through 1000000.
|
|
- A dense screen is cut to fit under it.
|
|
|
|
#### `providerOptions`
|
|
|
|
Provider options every model call carries, such as a reasoning effort.
|
|
|
|
```ts
|
|
providerOptions: { openai: { reasoningEffort: 'low' } },
|
|
```
|
|
|
|
OpenAI and Azure OpenAI Responses calls also carry `store: false` and a prompt
|
|
cache key unless set here.
|
|
|
|
#### Removed options
|
|
|
|
An unknown key (the error names the nearest known one), a removed key, or a
|
|
value in an old shape fails the config load with `INVALID_CONFIG`:
|
|
|
|
| Written | Now |
|
|
| --- | --- |
|
|
| a bare `StepExecutor` as the entry | `{ executor }` |
|
|
| `timeout` | `judgmentTimeout` |
|
|
| `maxTurns` (on the entry or `createToolLoopExecutor`) | `maxModelCalls` |
|
|
| top-level `limits.maxModelTokensPerCall` | `maxInputTokens` on each agent |
|
|
| top-level `limits.maxAgentContextBytes`, `maxLedgerBytes`, `maxEventsPerStep` | removed; the runner fixes them at 16384 bytes, 8192 bytes, and 1000 events |
|
|
|
|
### filters
|
|
|
|
[Test filters](/filters) that leave out the tests a change can't affect,
|
|
applied in order when the run passes `--changed-since`.
|
|
|
|
```ts title="e2e.config.ts"
|
|
import type { E2EConfig } from 'e2e';
|
|
import { web } from '@e2e-dev/web';
|
|
import { gateway } from 'ai';
|
|
import { decisionFilter } from '@e2e-dev/decision';
|
|
|
|
export default {
|
|
targets: [{ engine: web(), app: { url: 'http://localhost:3000' } }],
|
|
filters: [decisionFilter({ model: gateway.decisionModel('typesafe-ai/jev') })],
|
|
} satisfies E2EConfig;
|
|
```
|
|
|
|
Which tests always run is on
|
|
[Test filters](/filters#what-always-runs). A `name` follows the target name
|
|
rules, is at most 256 characters, and is unique among `filters`.
|
|
|
|
| Throws | When |
|
|
| --- | --- |
|
|
| `INVALID_CONFIG` | A bad `name`, an item that isn't `{ name, filter }`, or a bare function such as `decisionFilter` in place of `decisionFilter({ model })`. |
|
|
|
|
The declarations are under [`TestFilter`](#testfilter), exported from `e2e`.
|
|
|
|
### credentials
|
|
|
|
Named accounts. Each entry is a username and a password the model never sees.
|
|
|
|
```ts
|
|
credentials: {
|
|
admin: { username: 'admin@example.com', password: () => readFile('.secrets/admin', 'utf8') },
|
|
},
|
|
```
|
|
|
|
| Field | Type |
|
|
| --- | --- |
|
|
| [`username`](#username) | `string` |
|
|
| [`password`](#password) | `string \| SecretProvider` |
|
|
|
|
- Credentials and secrets are separate namespaces. `secrets.get(name)`
|
|
returns a `secrets` entry only. A credential's password is reachable only
|
|
as `credentials.user(name).password`.
|
|
- The agent, the replay cache, and the reports know that password as a secret
|
|
named `<name>.password` (`<secret:admin.password>`). So `credentials.admin`
|
|
and `secrets.admin` can both exist.
|
|
|
|
| Throws | When |
|
|
| --- | --- |
|
|
| `INVALID_CONFIG` | A `secrets` entry named exactly `admin.password` beside an `admin` credential. |
|
|
| `INVALID_CONFIG` | Two credentials whose names map to one `E2E_USER_<NAME>_*` override, like `svc-a` and `svc_a` (see [Environment variables](/reference/environment)). |
|
|
|
|
See [Authentication](/authentication). The declarations are under
|
|
[`CredentialConfig`](#credentialconfig).
|
|
|
|
#### `username`
|
|
|
|
The account's plain username.
|
|
|
|
```ts
|
|
username: 'admin@example.com',
|
|
```
|
|
|
|
`E2E_USER_<NAME>_USERNAME` wins over the config value. With neither, the
|
|
entry is `INVALID_CONFIG`.
|
|
|
|
#### `password`
|
|
|
|
The account's password: a static value, or a `SecretProvider` function.
|
|
|
|
```ts
|
|
password: () => readFile('.secrets/admin', 'utf8'),
|
|
```
|
|
|
|
- A static password is at least 6 characters (code points), like a
|
|
[secret](#secrets), or `INVALID_CONFIG`.
|
|
- A `SecretProvider` runs on every authorized fill (a vault lookup, a freshly
|
|
computed TOTP). The config digest records it by name only. A provider
|
|
returning fewer than 6 characters fails the fill with
|
|
`AUTH_CREDENTIAL_UNAVAILABLE`.
|
|
- `E2E_USER_<NAME>_PASSWORD` wins over both.
|
|
|
|
#### `credentials.user()`
|
|
|
|
Resolves a configured entry into a `Credential` in a test.
|
|
|
|
```ts
|
|
const admin = credentials.user('admin');
|
|
```
|
|
|
|
- The username is a plain string. The password is an opaque `Secret` that
|
|
test code cannot read, accepted only by `locator.fill` and `agent.act`
|
|
params.
|
|
- An unconfigured name throws `AUTH_CREDENTIAL_UNAVAILABLE`.
|
|
|
|
### secrets
|
|
|
|
Named values the model must never see: API keys, tokens, anything sourced
|
|
from the environment.
|
|
|
|
```ts
|
|
secrets: {
|
|
apiKey: () => readFile('.secrets/api-key', 'utf8'),
|
|
},
|
|
```
|
|
|
|
| Entry | Example | Checked |
|
|
| --- | --- | --- |
|
|
| a static value | `'sk_test_51Hx…'` | at config load |
|
|
| a provider | `() => readFile('.secrets/api-key', 'utf8')` | on every authorized fill |
|
|
|
|
- A provider runs on every authorized fill (a vault lookup, a freshly
|
|
computed TOTP). The config digest records it by name only.
|
|
- An `E2E_SECRET_<NAME>` override wins over both.
|
|
- Every entry must resolve to a string of at least 6 characters (code points)
|
|
or a function; anything else is `INVALID_CONFIG`. A provider's value is held
|
|
to the same minimum at fill time, with `SECRET_UNAVAILABLE`.
|
|
- The minimum exists for redaction. Every occurrence of a value becomes
|
|
`<secret:name>`, so a shorter value would take ordinary text in reports and
|
|
output with it.
|
|
- Two entries whose names map to one variable, like `api-key` and `api_key`,
|
|
are `INVALID_CONFIG`.
|
|
|
|
The declarations are under [`SecretConfig`](#secretconfig). See
|
|
[Secrets that are not passwords](/authentication#secrets-that-are-not-passwords).
|
|
|
|
#### `secrets.get()`
|
|
|
|
Returns the opaque `Secret` for a configured entry.
|
|
|
|
```ts
|
|
await screen.getByLabel('API key').fill(secrets.get('apiKey'));
|
|
```
|
|
|
|
- It has no plaintext accessor and is accepted only by `locator.fill` and
|
|
`agent.act` params.
|
|
- The runner fills the field, masks the value in every observation, and
|
|
redacts it from logs, traces, and the report.
|
|
- An unconfigured name throws `SECRET_UNAVAILABLE`.
|
|
|
|
#### Secrets in engine options
|
|
|
|
While `e2e.config.ts` evaluates, before any run exists, `secrets.get(name)`
|
|
returns a reference by name that an engine option can hold.
|
|
|
|
```ts
|
|
engine: web({ basicAuth: { username, password: secrets.get('previewPassword') } }),
|
|
```
|
|
|
|
- The engine declares it. When no `secrets` entry has the name, the config
|
|
load fails with `INVALID_CONFIG`. A credential's password is not one:
|
|
declare a value an engine option needs under `secrets`.
|
|
- The engine resolves the value when each attempt starts, so a provider runs
|
|
again every time. The value then joins the attempt's redaction. It goes to
|
|
the engine, not onto the page, so it brings none of a fill's screenshot
|
|
restrictions.
|
|
- The digest records the name only.
|
|
- The reference is not the value. Turned into a string (a template literal,
|
|
`String()`, `+`, `JSON.stringify`), it throws `INVALID_CONFIG`. So does a
|
|
reference in an `args` entry or `env` value of `app.command`, which takes
|
|
strings and placeholders only. Read a value a string needs, such as an
|
|
app's API key, from `process.env` yourself.
|
|
|
|
### cache
|
|
|
|
Replay cache mode or options. In `read-write` mode the replay cache records an
|
|
`agent.act()` only after a later verification passes, such as a locator
|
|
assertion or `agent.assert()`.
|
|
|
|
```ts
|
|
cache: { mode: 'read-write', dir: '.e2e/cache', strict: true },
|
|
```
|
|
|
|
| Option | Type | Default |
|
|
| --- | --- | --- |
|
|
| [`mode`](#mode) | `CacheMode` | `'read-write'`, `'read-only'` in CI |
|
|
| [`store`](#store) | `CacheStore` | the file store at `dir` |
|
|
| [`dir`](#dir) | `string` | `'.e2e/cache'` |
|
|
| [`strict`](#strict) | `boolean` | `false` |
|
|
|
|
A bare string is shorthand for `{ mode }`: `cache: 'read-only'`. See
|
|
[Caching](/cache). The declarations are under [`CacheConfig`](#cacheconfig).
|
|
|
|
#### `mode`
|
|
|
|
What the replay cache reads and writes.
|
|
|
|
```ts
|
|
cache: 'read-write', // default: 'read-write' locally, 'read-only' in CI
|
|
```
|
|
|
|
| Mode | Does |
|
|
| --- | --- |
|
|
| `'read-write'` | Replays entries and records new ones. |
|
|
| `'read-only'` | Replays existing entries and writes none. |
|
|
| `'off'` | Disables both. |
|
|
|
|
- An explicit mode overrides the default. Set `'read-write'` explicitly when
|
|
CI restores the cache from its own cache service.
|
|
- `--no-cache` wins over both and disables caching for the run.
|
|
|
|
#### `store`
|
|
|
|
A custom entry store that replaces the file store.
|
|
|
|
```ts
|
|
cache: { store: createStore(cacheUrl) },
|
|
```
|
|
|
|
- Each worker constructs its own.
|
|
- Exempt from the CI demotion: it states its own trust through `writable`.
|
|
- `--cache-dir` beside it is `INVALID_CONFIG`, and `E2E_CACHE_DIR` is ignored.
|
|
|
|
`CacheStore`, `CacheReadResult`, `TraceEntry`, and `ActionTrace` are exported
|
|
from `e2e`.
|
|
|
|
#### `dir`
|
|
|
|
The file store directory, resolved against the project root.
|
|
|
|
```ts
|
|
cache: { dir: '.cache/e2e' }, // default: '.e2e/cache'
|
|
```
|
|
|
|
- It stays where it is whatever `output` says.
|
|
- `--cache-dir` and `E2E_CACHE_DIR` replace it.
|
|
|
|
#### `strict`
|
|
|
|
Fails a step whose recording exists but no longer replays with
|
|
`REPLAY_STALE`, instead of handing it to the agent.
|
|
|
|
```ts
|
|
cache: { strict: true }, // default: false
|
|
```
|
|
|
|
- `--strict-cache` turns it on for one run.
|
|
- The run then never writes the cache.
|
|
|
|
See [Fail on a stale recording](/cache#fail-on-a-stale-recording).
|
|
|
|
#### Commit the replay cache
|
|
|
|
`e2e init` adds `.e2e/cache/` to `.gitignore`, so committing entries is
|
|
opt-in. Remove that line to share replays with CI and teammates. CI stays
|
|
`read-only` unless `cache: 'read-write'` is set. See
|
|
[Caching](/cache#commit-the-replay-cache).
|
|
|
|
## Run limits
|
|
|
|
The report's `run.limits` block records the ceilings a run enforced. There is
|
|
no `limits` key: two fields come from the agents, and the runner fixes the
|
|
rest.
|
|
|
|
| Report field | Value |
|
|
| --- | --- |
|
|
| `maxAgentContextBytes` | 16384: an agent's `context` and a test's `agentContext` |
|
|
| `maxLedgerBytes` | 8192: the prior-step ledger an act step reads |
|
|
| `maxEventsPerStep` | 1000: events recorded per step |
|
|
| `maxObservationBytes` | the largest `agents.<name>.maxObservationBytes` |
|
|
| `maxModelTokensPerCall` | the largest `agents.<name>.maxInputTokens` |
|
|
|
|
Per-call options may only lower an agent's budgets.
|
|
|
|
## Environment variables
|
|
|
|
The runner reads a closed set, listed on the
|
|
[Environment variables](/reference/environment) page. `E2E_USER_<NAME>_USERNAME`
|
|
and `E2E_USER_<NAME>_PASSWORD` override a credential, `E2E_SECRET_<NAME>` a
|
|
secret, and `CI` switches the defaults below.
|
|
|
|
### Resolution order
|
|
|
|
| Setting | Order |
|
|
| --- | --- |
|
|
| Base URL | the target's `app.url`, else `APP_URL_REQUIRED` at the first navigation. The runner reads no `APP_URL`; the scaffolded config reads it itself |
|
|
| Built-in agent model | `agents.<name>.model`, then `MODEL_UNAVAILABLE` at `agent` acquisition; an agent never falls back to `default`'s model |
|
|
| Custom executor model | `agents.<name>.model`, else the model the executor brought; two different values is `INVALID_CONFIG` |
|
|
| Model credential | the provider package's own variable or the key passed at construction, read by the provider on the first model call |
|
|
| Credential username and password | the `E2E_USER_*` variable, then the config value |
|
|
|
|
The built-in agent checks for a missing model once per run, when the `agent`
|
|
fixture is first acquired, and stops the run with exit 2. A custom
|
|
[executor](#executor) can implement `act` and `assert` without a model.
|
|
`waitFor` and `extract` still require one.
|
|
|
|
### CI
|
|
|
|
CI mode is on unless `CI` is unset, blank, `0`, or `false`.
|
|
|
|
| Setting | Local | CI |
|
|
| --- | --- | --- |
|
|
| `retries` | `0` | `1` |
|
|
| `workers` | `max(1, floor(availableParallelism() / 2))` | `1` |
|
|
| `cache` | `read-write` | `read-only` when unset |
|
|
| `test.only` | allowed | `ONLY_IN_CI`, exit code 2 |
|
|
| `command.reuseExisting` | honored | ignored, with a notice |
|
|
|
|
## Loading
|
|
|
|
How the runner finds the config and loads it and the files it imports.
|
|
|
|
### Finding the config file
|
|
|
|
- `--config <path>` selects an explicit file.
|
|
- Otherwise the runner searches the working directory and each parent for
|
|
`e2e.config.ts`, `e2e.config.mts`, `e2e.config.js`, or `e2e.config.mjs`,
|
|
stopping at the repository root.
|
|
- A TypeScript config wins over a JavaScript one in the same directory, so
|
|
the `e2e.config.js` that `tsc` emits beside `e2e.config.ts` is ignored.
|
|
- Both TypeScript names, or both JavaScript names, in one directory is
|
|
`CONFIG_AMBIGUOUS`.
|
|
- The selected file's directory is the project root.
|
|
|
|
### `.env` files
|
|
|
|
e2e loads no `.env` file itself. Load one at the top of `e2e.config.ts` with
|
|
Node's `process.loadEnvFile()`. Every worker imports the config again, so the
|
|
variables reach the workers too:
|
|
|
|
```ts title="e2e.config.ts"
|
|
process.loadEnvFile('.env');
|
|
```
|
|
|
|
It throws when the file is missing; guard it with `existsSync` from
|
|
`node:fs` when only some machines have one. The path resolves against the
|
|
directory e2e runs from, but workers start in the config's directory. To run
|
|
from a subdirectory, pass `new URL('.env', import.meta.url)` instead.
|
|
|
|
### Module format
|
|
|
|
- Config and test files run as ESM whatever the nearest `package.json`
|
|
`type` says. So does every `.ts`, `.mts`, `.tsx`, and `.jsx` file they
|
|
reach: a helper by path, a package that exports its `.ts` source by name, or
|
|
a `.ts` file a CommonJS file requires.
|
|
- A `.cts` file is CommonJS, written with `require`, `import x = require()`,
|
|
and `export =`; its `import` statements are not turned into `require`
|
|
calls.
|
|
- A `.ts` file written with `require` or `module.exports` fails to load: name
|
|
it `.cts`.
|
|
- JavaScript keeps the format its package declares.
|
|
|
|
### JavaScript
|
|
|
|
A config and tests can be plain JavaScript, with no `tsconfig.json`:
|
|
|
|
```js title="e2e.config.js"
|
|
import { web } from '@e2e-dev/web';
|
|
import { gateway } from 'ai';
|
|
|
|
/** @type {import('e2e').E2EConfig} */
|
|
export default {
|
|
tests: 'tests/**/*.e2e.js',
|
|
agents: { default: { model: gateway('openai/gpt-6-luna-fast') } },
|
|
targets: [{ engine: web(), app: { url: 'http://localhost:3000' } }],
|
|
};
|
|
```
|
|
|
|
- Name the config `e2e.config.js` or `e2e.config.mjs`
|
|
([Finding the config file](#finding-the-config-file)).
|
|
- Write ES modules. Set `"type": "module"` in `package.json`, or name the
|
|
files `.mjs`. Without either, Node.js warns on every load that it reparsed
|
|
the file as ESM.
|
|
- Set [`tests`](#tests): the default selects only `*.e2e.ts`. Use
|
|
`'tests/**/*.e2e.mjs'` for `.mjs` tests. List `'tests/**/*.e2e.ts'` as
|
|
well to run both kinds while you move a suite.
|
|
- The `@type` comment does the job of `satisfies E2EConfig`: an editor
|
|
completes keys and shows their documentation. Test fixtures get their
|
|
types from the `test` import, with no annotation.
|
|
- The runner never type-checks. For editor errors, add `// @ts-check` to a
|
|
file, or `"checkJs": true` to a `jsconfig.json`.
|
|
- A JavaScript file imports a TypeScript helper by its `.ts` name
|
|
(`'./helpers.ts'`), not `.js`; see [Import resolution](#import-resolution).
|
|
Under `checkJs`, that import also needs `"allowImportingTsExtensions": true`
|
|
and `"noEmit": true`.
|
|
|
|
### Import resolution
|
|
|
|
e2e compiles TypeScript and JSX itself. An import in such a file resolves the
|
|
way TypeScript resolves it:
|
|
|
|
- `./helper.js` and `./helper` load `helper.ts`, and `./view.jsx` loads
|
|
`view.tsx`.
|
|
- `./lib` loads the file `lib/package.json` names as `main`, else
|
|
`lib/index.ts`.
|
|
- A bare import that `tsconfig.json` maps through `paths` or `baseUrl` loads
|
|
the mapped file.
|
|
- A `#` import or package export that names a missing `./x.js`, `./x.jsx`, or
|
|
`./x` loads its `x.ts` or `x.tsx` source when imported.
|
|
- A `#` import or package export that names a directory, or a subpath of a
|
|
package without `exports` that is a directory (`@acme/db/enums`), loads it
|
|
the way `./lib` loads. Under Yarn PnP that package subpath still fails.
|
|
- A `require()` of any of the last two fails as Node.js reports it.
|
|
- An import written in a JavaScript file resolves as Node.js resolves it.
|
|
|
|
### `tsconfig.json`
|
|
|
|
- The `tsconfig.json` is the nearest one above the file. When that one has
|
|
`references` and does not include the file itself (as Vite writes it), it
|
|
is the transitively referenced one that does.
|
|
- References are visited in declaration order, and cycles are skipped. If
|
|
none owns the file, the nearest config remains the fallback.
|
|
- e2e reads its `paths`, `baseUrl`, `jsx`, `jsxFactory`,
|
|
`jsxFragmentFactory`, `jsxImportSource`, `experimentalDecorators`,
|
|
`emitDecoratorMetadata`, `useDefineForClassFields` (and `target`, which
|
|
sets its default), and `verbatimModuleSyntax`, once per process and again
|
|
for each session `e2e mcp` opens.
|
|
- A `tsconfig.json` e2e cannot read, such as one that `extends` a package that
|
|
is not installed, is skipped with a warning.
|
|
- TypeScript inside an installed package compiles without the project's
|
|
`tsconfig.json`.
|
|
|
|
### What fails to compile
|
|
|
|
- e2e refuses, naming the file and line, two things it cannot compile:
|
|
auto-accessors (`accessor x = 1`) and decorators without
|
|
`experimentalDecorators`.
|
|
- Each file compiles on its own, without type information. So a type
|
|
imported without `import type` fails to load when the import stays in the
|
|
output (a decorated member's type under `emitDecoratorMetadata`, a
|
|
re-exported type). The error names the import to change.
|
|
- A JSON file imported without `with { type: 'json' }` loads with its parsed
|
|
contents as the default export.
|
|
- Stack traces and failure locations point at the TypeScript source, through
|
|
another loader's source map too (`NODE_OPTIONS=--import tsx`).
|
|
|
|
## Types
|
|
|
|
### `E2EConfig`
|
|
|
|
```ts
|
|
export interface E2EConfig {
|
|
/** Stable project id, 1 through 256 characters; defaults to the root `package.json` name. */
|
|
projectId?: string;
|
|
/** The surfaces tests run on; required, at least one, each naming its engine. */
|
|
targets: readonly Target[];
|
|
/** Test file globs relative to the project root (`*`, `?`, and a whole `**` segment; a leading `!` excludes); default every `*.e2e.ts` under `tests/`. */
|
|
tests?: string | readonly string[];
|
|
/** Test attempt deadline in milliseconds; default 120000. */
|
|
timeout?: number;
|
|
/** Engine init and attempt start budget in milliseconds; default 60000. */
|
|
launchTimeout?: number;
|
|
/** Budget of every engine operation in milliseconds; default 30000. */
|
|
actionTimeout?: number;
|
|
/** Default `expect` deadline in milliseconds; default 5000. */
|
|
assertionTimeout?: number;
|
|
/** Default timeout and pauses for `expect(fn).toPass()`. */
|
|
toPass?: PassOptions;
|
|
/** Budget of each `afterEach` hook, fixture teardown, and engine cleanup in milliseconds; default 30000. */
|
|
cleanupTimeout?: number;
|
|
/** Retries per test, 0 through 10; default 1 in CI, else 0. */
|
|
retries?: number;
|
|
/** Fail the run when a test skips a retry after an earlier failed attempt; default false. */
|
|
failOnSkippedFailure?: boolean;
|
|
/** Wall-clock budget for running the tests, in milliseconds; default none. */
|
|
maxDuration?: number;
|
|
/** Parallel workers, 1 through 1024; default 1 in CI, else half the cores. An engine may cap it lower. */
|
|
workers?: number;
|
|
/** `{ store }` hands every artifact to a host store as it is produced. */
|
|
artifacts?: ArtifactsConfig;
|
|
/** Which tests keep a trace page; default `retain-on-failure`. */
|
|
trace?: RecordingMode;
|
|
/** Which attempts record a video; default `off`. */
|
|
video?: RecordingMode;
|
|
/** The results directory, relative to the project root; default `.e2e`. */
|
|
output?: string;
|
|
/** Output renderers and reporter objects; default `['list']`. */
|
|
reporters?: readonly (BuiltinReporter | Reporter)[];
|
|
/** Named agents: the built-in agent's options, or `{ executor }` for a custom brain. */
|
|
agents?: Readonly<Record<string, AgentConfig>>;
|
|
/** Test filters, in order, run with `--changed-since`: each judges the tests the one before it kept. */
|
|
filters?: readonly TestFilter[];
|
|
/** Replay cache mode or options; default `read-write`, `read-only` in CI. */
|
|
cache?: CacheMode | CacheConfig;
|
|
/** Named accounts: a username and a password the model never sees. */
|
|
credentials?: Readonly<Record<string, CredentialConfig>>;
|
|
/** Named values the model must never see: API keys, tokens, anything sourced from the environment. */
|
|
secrets?: Readonly<Record<string, SecretConfig>>;
|
|
}
|
|
```
|
|
|
|
### `Target`
|
|
|
|
```ts
|
|
/**
|
|
* One named test target: the app under test and the engine that drives it.
|
|
* Without an engine, the target supports agent tools only.
|
|
*/
|
|
export interface Target {
|
|
/** Label in reports and for `--target`; defaults to the platform. */
|
|
name?: string;
|
|
/** Platform label; inherited from the engine when omitted. */
|
|
platform?: string;
|
|
/** The engine driving the surface. */
|
|
engine?: EngineHandle;
|
|
/** The app under test: what it is, where it is served, and the command that starts it. */
|
|
app?: TargetApp;
|
|
/** Which tests on this target keep a trace page, in place of the config's `trace`. */
|
|
trace?: RecordingMode;
|
|
/** Which attempts on this target record a video, in place of the config's `video`. */
|
|
video?: RecordingMode;
|
|
}
|
|
```
|
|
|
|
### `CommandConfig`
|
|
|
|
```ts
|
|
export interface CommandConfig {
|
|
/** Resolved with `PATH`; never shell-interpreted. */
|
|
executable: string;
|
|
/** Passed verbatim; `{port}` expands to the app's port. */
|
|
args?: readonly string[];
|
|
/** Working directory, resolved from the project root. */
|
|
cwd?: string;
|
|
/** Added to the inherited set; `{port}` expands in values. */
|
|
env?: Readonly<Record<string, string>>;
|
|
/** Variables copied from the runner's environment when set: exact names, or prefixes ending in `*`. */
|
|
inheritEnv?: readonly string[];
|
|
/** Ready-probe budget in milliseconds; default 60000. */
|
|
startupTimeout?: number;
|
|
/** Grace period before force-kill in milliseconds; default 10000. */
|
|
shutdownTimeout?: number;
|
|
/** File that receives the process's stdout and stderr; omitted discards output. */
|
|
log?: string;
|
|
/** Use a process already answering the readiness URL instead of spawning; default `false`. */
|
|
reuseExisting?: boolean;
|
|
}
|
|
```
|
|
|
|
### `ArtifactsConfig`
|
|
|
|
```ts
|
|
export interface ArtifactsConfig {
|
|
/** Host store every produced artifact is handed to; omit it to keep files local only. */
|
|
store?: ArtifactStore;
|
|
}
|
|
|
|
export interface ArtifactStore {
|
|
put(artifact: StoredArtifact): Promise<{ readonly ref: string }>;
|
|
/** Receives a provider-hosted video as a link; optional. */
|
|
putLink?(link: StoredArtifactLink): Promise<{ readonly ref: string }>;
|
|
}
|
|
|
|
/** One produced artifact, handed to `store.put` once it is complete on disk. */
|
|
export interface StoredArtifact {
|
|
readonly kind: 'screenshot' | 'video' | 'download' | 'log';
|
|
readonly mediaType: string;
|
|
readonly bytes: Uint8Array;
|
|
readonly size: number;
|
|
readonly sha256: string;
|
|
/** Report-relative path with `/` separators, stable across runs of the same test. */
|
|
readonly path: string;
|
|
/** How much of the file the runner masked, as the report records it. */
|
|
readonly redaction: 'complete' | 'incomplete';
|
|
readonly runId: string;
|
|
readonly testId: string;
|
|
readonly attemptId: string;
|
|
/** The step that produced it, when one was running. */
|
|
readonly stepId?: string;
|
|
/** When a time-based artifact (a video segment) began recording. */
|
|
readonly startedAt?: string;
|
|
}
|
|
|
|
/** One provider-hosted recording, handed to `store.putLink` once the attempt stopped it. */
|
|
export interface StoredArtifactLink {
|
|
readonly kind: 'video';
|
|
/** The `http(s)` URL the service serves the recording from. */
|
|
readonly url: string;
|
|
readonly mediaType: string;
|
|
/** A recording masks nothing, so a link is always `incomplete`. */
|
|
readonly redaction: 'incomplete';
|
|
readonly runId: string;
|
|
readonly testId: string;
|
|
readonly attemptId: string;
|
|
/** When the recording started, as an ISO timestamp. */
|
|
readonly startedAt: string;
|
|
/** The step that was running when the recording was registered, if any. */
|
|
readonly stepId?: string;
|
|
}
|
|
```
|
|
|
|
### `AgentConfig`
|
|
|
|
```ts
|
|
/** A project tool as `defineTool` from `e2e/agent` returns it. */
|
|
export interface AgentTool {
|
|
readonly tool: object;
|
|
readonly annotations: { readonly mutates: boolean; readonly platforms?: readonly string[] };
|
|
}
|
|
|
|
/** The options every agent takes, whichever brain runs its `act` steps. */
|
|
export interface AgentOptions {
|
|
/** An AI SDK model instance; no implicit default. */
|
|
model?: ModelInstance;
|
|
/** Model for `assert`, `waitFor`, and `extract`; defaults to `model`. */
|
|
judge?: ModelInstance;
|
|
/** What the app calls things, told to every model call; at most 16384 bytes. */
|
|
context?: string;
|
|
/** Committed actions per agent call, 1 through 100; default 25. */
|
|
maxSteps?: number;
|
|
/** Model requests per agent call, 1 through 100; default 25. */
|
|
maxModelCalls?: number;
|
|
/** Deadline of one `assert`, `waitFor`, or `extract` call in milliseconds; default 30000. */
|
|
judgmentTimeout?: number;
|
|
/** Observation payload ceiling for act turns and judgments, 1024 through 16777216; default 262144. */
|
|
maxObservationBytes?: number;
|
|
/** Input tokens per model request, 1 through 1000000; default 64000. */
|
|
maxInputTokens?: number;
|
|
/** Provider options every model call carries, such as a reasoning effort. */
|
|
providerOptions?: ProviderOptions;
|
|
}
|
|
|
|
/** One entry of `agents`: the built-in agent with its options, or a custom brain under `executor`. */
|
|
export type AgentConfig = AgentOptions &
|
|
(
|
|
| {
|
|
/** How the acting agent should work, appended to the built-in rules. */
|
|
system?: string;
|
|
/** Project tools from `defineTool`, offered beside the built-in toolset. */
|
|
tools?: Readonly<Record<string, AgentTool>>;
|
|
}
|
|
| {
|
|
/** A custom brain `agent.act()` dispatches to instead of the built-in agent. */
|
|
executor: StepExecutor;
|
|
}
|
|
);
|
|
```
|
|
|
|
### `ModelInstance`
|
|
|
|
```ts
|
|
/**
|
|
* An AI SDK LanguageModelV2+ instance. Its provider handles transport
|
|
* and credentials.
|
|
*/
|
|
export interface ModelInstance {
|
|
/** The AI SDK specification the instance implements. */
|
|
readonly specificationVersion: string;
|
|
/** Provider id, recorded per step in the report. */
|
|
readonly provider: string;
|
|
/** Model id, recorded per step in the report. */
|
|
readonly modelId: string;
|
|
/** The generate call every AI SDK language model implements; what tells a live instance from its three id strings. */
|
|
readonly doGenerate: (...args: never[]) => unknown;
|
|
}
|
|
```
|
|
|
|
### `TestFilter`
|
|
|
|
```ts
|
|
interface TestFilter {
|
|
/** Names the filter in the report and in errors. */
|
|
readonly name: string;
|
|
readonly filter: (tests: readonly TestFilterCandidate[], context: TestFilterContext) => TestFilterResult | Promise<TestFilterResult>;
|
|
}
|
|
|
|
interface TestFilterCandidate {
|
|
/** The report's `testId`, and the key `exclude` and `relevance` take. */
|
|
readonly id: string;
|
|
/** Project-relative path of the test file. */
|
|
readonly file: string;
|
|
/** The describe titles, then the test title. */
|
|
readonly titlePath: readonly string[];
|
|
/** The describe chain's tags, then the test's own. */
|
|
readonly tags: readonly string[];
|
|
/** Project-relative path of the setup test file that saves the test's session; absent without one. */
|
|
readonly setupFile?: string;
|
|
}
|
|
|
|
interface TestFilterContext {
|
|
/** Absolute project root, spelled as on disk like `changes.root`. */
|
|
readonly projectRoot: string;
|
|
/** The change every filter of the run judges. */
|
|
readonly changes: TestFilterChanges;
|
|
/** The config file the run loaded, relative to `changes.root` like `changes.files`; `undefined` for a config object or one outside the repository. */
|
|
readonly configFile?: string | undefined;
|
|
/** Aborts when the run is interrupted, or with a `TimeoutError` after 5 minutes. */
|
|
readonly signal: AbortSignal;
|
|
/** Prints one progress line. */
|
|
log(line: string): void;
|
|
/** Replaces every fixed-value secret and credential password in `text`, and any run of 8 or more consecutive characters of one. */
|
|
redact(text: string): string;
|
|
}
|
|
|
|
interface TestFilterChanges {
|
|
/** The merge base of the `--changed-since` ref and `HEAD`. */
|
|
readonly base: string;
|
|
/** Absolute path of the repository root, spelled as on disk like `projectRoot`. */
|
|
readonly root: string;
|
|
/** Repository-relative POSIX paths of every changed file, uncommitted and untracked files included. Sorted, never empty. */
|
|
readonly files: readonly string[];
|
|
}
|
|
|
|
interface TestFilterResult {
|
|
/** The tests to leave out, by `id`, each with a non-empty reason the report shows. An `id` of a collected test this filter wasn't given is ignored; any other fails the filter. */
|
|
readonly exclude: Readonly<Record<string, string>>;
|
|
/** How likely each test is to fail on this change, by `id` as for `exclude`, from 0 to 1; any other score fails the filter. Each target starts the files of the likeliest first, in bands at 75%, 50%, and 25%, and targets go by the band of their best file, ties in config order. A test no filter scores counts as 0, and a test several filters score keeps its highest. */
|
|
readonly relevance?: Readonly<Record<string, number>>;
|
|
/** One line about the whole decision. */
|
|
readonly note?: string;
|
|
}
|
|
```
|
|
|
|
`TestFilter`, `TestFilterCandidate`, `TestFilterContext`,
|
|
`TestFilterChanges`, and `TestFilterResult` are exported from `e2e`.
|
|
|
|
A filter that throws, returns an invalid result, or runs past 5 minutes
|
|
leaves nothing out, and a warning says so. An error it throws can carry
|
|
`relevance`, shaped as in `TestFilterResult`: the run still starts the
|
|
likeliest first. See [Test filters](/filters#write-your-own).
|
|
|
|
### `CredentialConfig`
|
|
|
|
```ts
|
|
/** One `config.credentials` entry: an account whose password is a secret named `<entry>.password`. */
|
|
export interface CredentialConfig {
|
|
username: string;
|
|
password: string | SecretProvider;
|
|
}
|
|
|
|
type SecretProvider = () => string | Promise<string>;
|
|
|
|
/** Named test identity. The password remains an opaque Secret. */
|
|
export interface Credential {
|
|
/** The entry's name in `credentials`. */
|
|
readonly name: string;
|
|
/** Plain username. */
|
|
readonly username: string;
|
|
/** Opaque; test code cannot read it. */
|
|
readonly password: Secret;
|
|
readonly [credentialBrand]: true;
|
|
}
|
|
```
|
|
|
|
### `SecretConfig`
|
|
|
|
```ts
|
|
/** One `config.secrets` entry: the value itself, or a provider computing it at fill time. */
|
|
export type SecretConfig = string | SecretProvider;
|
|
|
|
/**
|
|
* What a secret is for, which decides where it may be filled: a `password`
|
|
* goes only into a password field; a `generic-secret` (an API key, a token,
|
|
* any value from the environment) goes into any editable input.
|
|
*/
|
|
export type SecretPurpose = 'password' | 'generic-secret';
|
|
|
|
/** Opaque host-side value accepted only by sensitive input sinks. */
|
|
export interface Secret {
|
|
/** The entry's name in `secrets`, or `<credential>.password` for a credential's password. */
|
|
readonly name: string;
|
|
/** Where the value may be filled. */
|
|
readonly purpose: SecretPurpose;
|
|
readonly [secretBrand]: true;
|
|
}
|
|
```
|
|
|
|
### `CacheConfig`
|
|
|
|
```ts
|
|
/** Replay cache options. */
|
|
export interface CacheConfig {
|
|
/** Default `read-write`; `read-only` in CI when unset. */
|
|
mode?: CacheMode;
|
|
/** Custom entry store; omit it to use the file store at `dir`. */
|
|
store?: CacheStore;
|
|
/** File store directory, resolved against the project root; default `.e2e/cache`. */
|
|
dir?: string;
|
|
/** Fail a step whose recording no longer replays with `REPLAY_STALE`. Default `false`. */
|
|
strict?: boolean;
|
|
}
|
|
|
|
/** Where the replay cache keeps its entries: the file store by default, or a host's own through `cache.store`. */
|
|
export interface CacheStore {
|
|
readonly writable: boolean;
|
|
read(keyHash: string): Promise<CacheReadResult>;
|
|
/** Persists one trace, returning its size, or undefined when not written. */
|
|
write(keyHash: string, payload: ActionTrace): Promise<{ bytes: number } | undefined>;
|
|
/**
|
|
* Evicts one entry, best-effort. Called when a replayed or just-recorded
|
|
* flow is implicated in a failed attempt: the entry re-records on the next
|
|
* pass instead of replaying a poisoned flow forever. Optional: a store
|
|
* without eviction merely stays stale until the next confirmed write.
|
|
*/
|
|
delete?(keyHash: string): Promise<void>;
|
|
}
|
|
|
|
export type CacheReadResult =
|
|
| { readonly status: 'hit'; readonly entry: TraceEntry; readonly bytes: number }
|
|
| { readonly status: 'miss' }
|
|
| { readonly status: 'invalid'; readonly reason: string; readonly bytes?: number };
|
|
```
|