mirror of
https://github.com/tester-army/e2e.git
synced 2026-11-22 18:19:48 +08:00
* fix(e2e): load TypeScript with an oxc loader instead of tsx * fix(e2e): close loader edge cases: sourceless CommonJS, export maps without extensions, unreadable tsconfig, chained source maps, Windows require paths * fix(e2e): refuse Bun and Deno runtimes, name oxc load failures, follow Yarn PnP export targets * perf(e2e): skip the syntax check for files whose only @ is in a package import * fix(e2e): allow type-only exports in .cts, warn from runner only, Bun check first, docs * fix(e2e): workers warn about tsconfigs runner did not, FILE: urls, foreign infra errors, review tests
1039 lines
47 KiB
Plaintext
1039 lines
47 KiB
Plaintext
---
|
|
title: E2EConfig
|
|
description: Every configuration key, its default, its valid range, and the environment variables.
|
|
---
|
|
|
|
Export an object from `e2e.config.ts` and use `satisfies E2EConfig` to check
|
|
its shape without widening its inferred types. The runner also validates the
|
|
config when it loads, including ranges and rules TypeScript cannot check.
|
|
|
|
```ts title="e2e.config.ts"
|
|
import type { E2EConfig } from 'e2e';
|
|
import { web } from '@e2e-dev/web';
|
|
|
|
export default {
|
|
targets: [
|
|
{
|
|
name: 'web',
|
|
engine: web({ browser: 'chromium' }),
|
|
app: { url: process.env.APP_URL ?? 'http://localhost:3000' },
|
|
},
|
|
],
|
|
} satisfies E2EConfig;
|
|
```
|
|
|
|
Unknown keys are rejected with `INVALID_CONFIG` at every level. There is no
|
|
top-level `app` key: each target declares the app it tests (see
|
|
[the app under test](#the-app-under-test)).
|
|
|
|
An optional key takes a value or is left out; `undefined` is not a value, so
|
|
`store: process.env.CACHE_URL && createStore(...)` fails to compile. Spread a
|
|
key that depends on the environment in only when it has a value:
|
|
|
|
```ts
|
|
export default {
|
|
targets: [{ engine: web(), app: { url: 'http://localhost:3000' } }],
|
|
...(process.env.CACHE_URL === undefined ? {} : { cache: { store: createStore(process.env.CACHE_URL) } }),
|
|
} satisfies E2EConfig;
|
|
```
|
|
|
|
## Loading
|
|
|
|
`--config <path>` selects an explicit file. Otherwise the runner searches the
|
|
working directory and each parent for `e2e.config.ts` or `e2e.config.mts`,
|
|
stopping at the repository root. Both names in one directory is
|
|
`CONFIG_AMBIGUOUS`. The selected file's directory is the project root.
|
|
|
|
The `tests` globs select files relative to the project root. An entry that
|
|
starts with `!` excludes: `tests: ['tests/**/*.e2e.ts', '!tests/wip/**']`
|
|
selects every test file except those under `tests/wip/`. A file is selected
|
|
when an including glob matches it and no exclusion does, whatever the order of
|
|
the list, and a list of only exclusions is `INVALID_CONFIG`. A glob names
|
|
files, so an entry with no wildcard that names a directory (`'!tests/wip'`)
|
|
is `INVALID_GLOB` naming the glob to write (`'!tests/wip/**'`). A file
|
|
passed to `e2e run` that an exclusion takes out is `NO_TESTS` naming that
|
|
entry. Discovery visits
|
|
only directories that could match, never enters one an exclusion takes whole,
|
|
skips `node_modules`, and ignores symlinks. A dot directory is included only
|
|
when a glob segment explicitly starts with a dot.
|
|
|
|
e2e loads no `.env` file itself. Load one at the top of `e2e.config.ts` with
|
|
Node's `process.loadEnvFile()`; every worker imports the config again, so the
|
|
variables reach the workers too:
|
|
|
|
```ts title="e2e.config.ts"
|
|
process.loadEnvFile('.env');
|
|
```
|
|
|
|
It throws when the file is missing; guard it with `existsSync` from
|
|
`node:fs` when only some machines have one.
|
|
|
|
Config and test files run as ESM whatever the nearest `package.json` `type`
|
|
says, and so does every `.ts`, `.mts`, `.tsx`, and `.jsx` file they reach: a
|
|
helper by path, a package that exports its `.ts` source by name, or a `.ts`
|
|
file a CommonJS file requires. A `.cts` file is CommonJS, written with
|
|
`require`, `import x = require()`, and `export =`; its `import` statements are
|
|
not turned into `require` calls. A `.ts` file written with `require` or
|
|
`module.exports` fails to load: name it `.cts`. JavaScript keeps the format
|
|
its package declares.
|
|
|
|
e2e compiles TypeScript and JSX itself, and an import written in such a file
|
|
resolves the way TypeScript resolves it: `./helper.js` and `./helper` load
|
|
`helper.ts`, `./lib` loads the file `lib/package.json` names as `main`, else
|
|
`lib/index.ts`, a bare import that `tsconfig.json` maps through `paths` or
|
|
`baseUrl` loads the mapped file, and a `#` import or package export that names
|
|
a missing `./x.js` or `./x` loads `x.ts` when imported (a `require()` of it
|
|
fails as Node.js reports it). An import written in a JavaScript file resolves
|
|
as Node.js resolves it.
|
|
|
|
The `tsconfig.json` is the nearest one above the file, or, when that one has
|
|
`references` and does not include the file itself (as Vite writes it), the
|
|
directly referenced one that does. e2e reads its `paths`, `baseUrl`, `jsx`, `jsxFactory`,
|
|
`jsxFragmentFactory`, `jsxImportSource`, `experimentalDecorators`,
|
|
`emitDecoratorMetadata`, `useDefineForClassFields` (and `target`, which sets
|
|
its default), and `verbatimModuleSyntax`, once per process and again for each
|
|
session `e2e mcp` opens. A `tsconfig.json` e2e cannot read, such as one that
|
|
`extends` a package that is not installed, is skipped with a warning.
|
|
TypeScript inside an installed package compiles without the project's
|
|
`tsconfig.json`.
|
|
|
|
e2e refuses, naming the file and line, two things it cannot compile:
|
|
auto-accessors (`accessor x = 1`) and decorators without
|
|
`experimentalDecorators`. Each file compiles on its own, without type
|
|
information, so a type imported without `import type` fails to load when the
|
|
import stays in the output (a decorated member's type under
|
|
`emitDecoratorMetadata`, a re-exported type); the error names the import to
|
|
change. A JSON file imported without `with { type: 'json' }` loads with its
|
|
parsed contents as the default export. Stack traces and failure locations
|
|
point at the TypeScript source, through another loader's source map too
|
|
(`NODE_OPTIONS=--import tsx`).
|
|
|
|
## Top level
|
|
|
|
```ts
|
|
export interface E2EConfig {
|
|
/** Stable project id, 1 through 256 characters; defaults to the root `package.json` name. */
|
|
projectId?: string;
|
|
/** The surfaces tests run on; required, at least one, each naming its engine. */
|
|
targets: readonly Target[];
|
|
/** Test file globs relative to the project root (`*`, `?`, and a whole `**` segment; a leading `!` excludes); default every `*.e2e.ts` under `tests/`. */
|
|
tests?: string | readonly string[];
|
|
/** Test attempt deadline in milliseconds; default 120000. */
|
|
timeout?: number;
|
|
/** Engine init and attempt start budget in milliseconds; default 60000. */
|
|
launchTimeout?: number;
|
|
/** Budget of every engine operation in milliseconds; default 30000. */
|
|
actionTimeout?: number;
|
|
/** Default `expect` deadline in milliseconds; default 5000. */
|
|
assertionTimeout?: number;
|
|
/** Budget of each `afterEach` hook, fixture teardown, and engine cleanup in milliseconds; default 30000. */
|
|
cleanupTimeout?: number;
|
|
/** Retries per test, 0 through 10; default 1 in CI, else 0. */
|
|
retries?: number;
|
|
/** Fail the run when a test skips itself after a soft failure or an earlier failed attempt; default false. */
|
|
failOnSkippedFailure?: boolean;
|
|
/** Parallel workers, 1 through 1024; default 1 in CI, else half the cores. An engine may cap it lower. */
|
|
workers?: number;
|
|
/** `{ store }` hands every artifact to a host store as it is produced. */
|
|
artifacts?: ArtifactsConfig;
|
|
/**
|
|
* Which attempts record a trace; default `on`, `on-first-retry` in CI. A
|
|
* target's `trace` wins over it, `--trace [mode]` over both, and a test's
|
|
* own `trace` over all. Applies to the targets whose engine can trace.
|
|
*/
|
|
trace?: RecordingMode;
|
|
/**
|
|
* Which attempts record a video; default `off`. A target's `video` wins
|
|
* over it, `--video [mode]` over both, and a test's own `video` over all.
|
|
* Applies to the targets whose engine can record.
|
|
*/
|
|
video?: RecordingMode;
|
|
/**
|
|
* The results directory, relative to the project root; default `.e2e`.
|
|
* `--output <dir>` overrides it for one run.
|
|
*/
|
|
output?: string;
|
|
/**
|
|
* Output renderers and reporter objects. `junit` writes `<output>/junit.xml`,
|
|
* `markdown` writes `<output>/summary.md`, `json` prints the report and
|
|
* excludes `list`; a `Reporter` object runs
|
|
* beside them and `--reporter` never removes it.
|
|
*/
|
|
reporters?: readonly (BuiltinReporter | Reporter)[];
|
|
/**
|
|
* Named agents, each one plain object: the built-in agent's options, or
|
|
* `{ executor }` for a custom brain. Tests use `default` unless selected
|
|
* otherwise. No agent inherits another's values. Each worker constructs
|
|
* its own agents from this config module.
|
|
*/
|
|
agents?: Readonly<Record<string, AgentConfig>>;
|
|
/**
|
|
* Replay cache mode or options. Defaults to `read-write` locally and
|
|
* `read-only` in CI. An explicit mode overrides the default;
|
|
* `--no-cache` disables caching for the run.
|
|
*/
|
|
cache?: CacheMode | CacheConfig;
|
|
/**
|
|
* Named accounts. `credentials.user(name).password` is the handle of a
|
|
* credential's password, a secret named `<name>.password`; `secrets.get()`
|
|
* never returns it, so a credential and a secret may share a name.
|
|
*/
|
|
credentials?: Readonly<Record<string, CredentialConfig>>;
|
|
/**
|
|
* Named values the model must never see: API keys, tokens, anything sourced
|
|
* from the environment. `secrets.get(name)` hands a test the opaque handle;
|
|
* the value is filled by the runner, masked in every observation, and
|
|
* redacted from logs, traces, and the report.
|
|
*/
|
|
secrets?: Readonly<Record<string, SecretConfig>>;
|
|
}
|
|
```
|
|
|
|
Every numeric value must be a safe integer. `projectId` falls back to
|
|
`unportable-<sha256 of the project root, 32 hex>` when there is no
|
|
`package.json`. `workers` never exceeds what a target's engine declares. A
|
|
targeted agent action is bounded by the smaller of `actionTimeout` and 15 s.
|
|
|
|
## failOnSkippedFailure
|
|
|
|
Set `failOnSkippedFailure: true` to fail the run with exit code 1 when a test
|
|
skips after a soft assertion failure or skips a retry after an earlier failed
|
|
attempt. The default is `false`.
|
|
|
|
The test keeps its `skipped` status and reason. Its failures remain in the
|
|
report, and the CLI shows them under `Skipped After Failure` with either
|
|
setting. Clean skips, successful retries, and engine cleanup diagnostics do
|
|
not trigger this policy. A skip still stops the body and does not retry.
|
|
|
|
## targets
|
|
|
|
```ts
|
|
/**
|
|
* One named test target: the app under test and the engine that drives it.
|
|
* Without an engine, the target supports agent tools only.
|
|
*/
|
|
export interface Target {
|
|
/** Label in reports and for `--target`; defaults to the platform. */
|
|
name?: string;
|
|
/**
|
|
* Platform label, inherited from the engine when omitted. Required for a
|
|
* target without an engine; when both name one, they must agree.
|
|
*/
|
|
platform?: string;
|
|
/** The engine driving the surface: `web(...)`, `mobile(...)`, or any `defineEngine` handle. */
|
|
engine?: EngineHandle;
|
|
/** The app under test: what it is, where it is served, and the command that starts it. */
|
|
app?: TargetApp;
|
|
/**
|
|
* Which attempts on this target record a trace, in place of the config's
|
|
* `trace`; `--trace` and a test's own `trace` win over it. A mode set here
|
|
* is required of the engine: one that cannot trace fails the run with
|
|
* `UNSUPPORTED_ARTIFACT`.
|
|
*/
|
|
trace?: RecordingMode;
|
|
/**
|
|
* Which attempts on this target record a video, in place of the config's
|
|
* `video`; `--video` and a test's own `video` win over it. A mode set here
|
|
* is required of the engine: one that cannot record fails the run with
|
|
* `UNSUPPORTED_ARTIFACT`.
|
|
*/
|
|
video?: RecordingMode;
|
|
}
|
|
```
|
|
|
|
`name` is the id `--target`, reports, sessions, and artifact paths use:
|
|
ASCII letters, digits, `_`, `-`, `.`, not only dots (`.` and `..` are path
|
|
segments), unique across targets.
|
|
|
|
`{ engine: web(), app: { url } }` is a complete target. Two targets on one
|
|
platform need names. A target without an engine runs through agent tools
|
|
only. Tests whose `requires` names a capability the target lacks are skipped
|
|
for it; a `test.setup` a selected test depends on must be runnable there or
|
|
selection fails with `COLLECTION_ERROR`.
|
|
|
|
## The app under test
|
|
|
|
A target declares the app it tests in `app`; the engine only drives it. A
|
|
browser target names the URL it opens, a device target the installed app or
|
|
the build it launches. The runner resolves `app` at config load.
|
|
|
|
```ts title="e2e.config.ts"
|
|
targets: [
|
|
{
|
|
name: 'web',
|
|
engine: web({ viewport: { width: 1440, height: 900 } }),
|
|
app: {
|
|
url: 'http://localhost:3000',
|
|
command: { executable: 'npm', args: ['run', 'dev'] },
|
|
readyUrl: 'http://localhost:3000/health',
|
|
},
|
|
},
|
|
{
|
|
name: 'ios',
|
|
engine: mobile({ platform: 'ios' }),
|
|
app: {
|
|
bundleId: 'com.example.app',
|
|
appPath: './build/App.app',
|
|
launchArguments: ['-e2e', 'YES'],
|
|
permissions: { camera: 'grant' },
|
|
},
|
|
},
|
|
],
|
|
```
|
|
|
|
| Key | Targets | Meaning |
|
|
| --- | --- | --- |
|
|
| `url` | web | Base URL: `app.open()` opens it and relative navigation resolves against it. WHATWG-normalized; no userinfo, query, or fragment; a missing scheme becomes `https://`, or `http://` for a loopback host. Plain HTTP is accepted for loopback hosts only. Port `0` on `127.0.0.1` or `[::1]` asks for a [free port](#free-ports) |
|
|
| `bundleId` | mobile | The installed app a device launches: a bundle id, an Android package name, or a display name the device resolves (`Settings`). A link is `INVALID_CONFIG`: a test opens one with `device.openLink` |
|
|
| `appPath` | mobile | The build the suite runs against, an iOS `.app` bundle or an Android `.apk`, resolved against the project root. Without `bundleId`, the app it installs is the one launched. See [the device engine](/reference/mobile#engine) for when it is installed |
|
|
| `identity` | any | Stable identity keying replay cache and session entries. Defaults to the URL's origin and base path, else `bundleId`, else `appPath`. Set a stable value to reuse entries across preview URLs or versioned build paths. Different apps must have different identities to prevent replay across them |
|
|
| `environment` | any | `test`, `staging`, or `production`. Labels the target in the report and joins the cache and session identity digest; never gates a run. Defaults to `test` for loopback, `.localhost`, and `.test` hosts and for an app without a URL, `production` otherwise |
|
|
| `launchArguments` | mobile | Arguments every fresh launch passes the app: `app.open()`, `app.restart()`, and `app.clearState()`. Arguments that select a build mode change what the app is: give each mode its own `identity` |
|
|
| `permissions` | mobile | Permissions the app holds on every fresh launch, by name, each `grant`, `deny`, or `reset`; see [permissions](/reference/mobile#engine) |
|
|
| `command` | any | Process the runner starts before the first test and stops on every exit path (a dev server, Metro for a debug build); see [command](#command). Targets declaring the same command share one process, probed at the first declaring target's `readyUrl` |
|
|
| `readyUrl` | any | URL polled until `command` is ready (a 200-499 status); defaults to `url`, and is required without one. `{port}` in it is the port of `url` |
|
|
|
|
Each engine checks the keys at config load, with `INVALID_CONFIG` naming
|
|
the target. A `web()` target needs `app.url` and refuses `bundleId`,
|
|
`appPath`, `launchArguments`, and `permissions`. A `mobile()` target needs
|
|
`app.bundleId` or `app.appPath`; `app.url` on a mobile target is not
|
|
supported yet. An unknown key in `app` or `app.command` fails with a
|
|
did-you-mean. A target with `app` and no engine is
|
|
`INVALID_CONFIG`.
|
|
|
|
The old spellings are unknown keys and fail at config load with
|
|
`INVALID_CONFIG`: `web({ url })` and the other app options of `web()` and
|
|
`mobile()`, `mobile({ app })` (now `app.bundleId`), an app key directly on the
|
|
target, a top-level `app` or `url`, and `services` anywhere, which this
|
|
version does not have.
|
|
|
|
Port `0` on any host other than `127.0.0.1` or `[::1]`, `localhost`
|
|
included, is `INVALID_APP_URL`, and a port-0 `url` with nothing to start on
|
|
it (no `app.command`) is `INVALID_CONFIG`. An `app.command` with neither
|
|
`app.url` nor `app.readyUrl` is `APP_URL_REQUIRED` at config load.
|
|
|
|
### The app's site
|
|
|
|
A test or the agent may open any http(s) URL, and `about:blank`; every
|
|
other scheme (`file:`, `data:`, `javascript:`, `view-source:`, `blob:`) is
|
|
`POLICY_DENIED`. The site of `url`, its registrable domain
|
|
(`app.example.com` and `auth.example.com` are one site), scopes two things:
|
|
the browser engine's [`headers`](/reference/web#options) reach the
|
|
site and no other host, and child frames off the site stay out of
|
|
observations.
|
|
|
|
### Free ports
|
|
|
|
A `url` on a literal loopback address with port `0` (`http://127.0.0.1:0`,
|
|
`http://[::1]:0`) asks the run for a free TCP port. The runner picks one when
|
|
the config loads and substitutes it wherever the declaration used the port:
|
|
the base URL, the default `readyUrl`, and the report. The command must take
|
|
the port through `{port}` in `args` or `env`.
|
|
|
|
```ts title="e2e.config.ts"
|
|
engine: web(),
|
|
app: {
|
|
url: 'http://127.0.0.1:0',
|
|
command: { executable: 'pnpm', args: ['dev', '--port', '{port}'], env: { PORT: '{port}' } },
|
|
},
|
|
```
|
|
|
|
`localhost:0` is `INVALID_APP_URL`: it may resolve to an address other than
|
|
the one the command binds. Write the address the command will bind.
|
|
|
|
Each target asking for port `0` gets its own port and its own process; two
|
|
targets that declare the same command on a fixed port share one. `{port}` in
|
|
`app.command` and `app.readyUrl` is the port of `url`. The default `identity`
|
|
derives from the declared `:0`, so cache and session entries do not churn
|
|
with the port. Tests read the allocated URL from
|
|
[`app.baseUrl`](/reference/app).
|
|
|
|
### command
|
|
|
|
```ts
|
|
export interface CommandConfig {
|
|
/** Resolved with `PATH`; never shell-interpreted. */
|
|
executable: string;
|
|
/** Passed verbatim; `{port}` expands to the app's port. */
|
|
args?: readonly string[];
|
|
/** Working directory, resolved from the project root. */
|
|
cwd?: string;
|
|
/** Added to the inherited set; `{port}` expands in values. */
|
|
env?: Readonly<Record<string, string>>;
|
|
/** Ready-probe budget in milliseconds; default 60000. Expiry is `APP_UNREACHABLE`. */
|
|
startupTimeout?: number;
|
|
/** Grace period before force-kill in milliseconds; default 10000. */
|
|
shutdownTimeout?: number;
|
|
/** File that receives the process's stdout and stderr, appended, resolved from the project root. Omitted discards output. */
|
|
log?: string;
|
|
/**
|
|
* When the readiness URL already answers before the command starts, use that
|
|
* process instead of spawning: nothing is started and nothing is stopped.
|
|
* Off by default; CI ignores it and always starts the command. A URL on a
|
|
* free port (port 0) can never already answer, so the two together are
|
|
* `INVALID_CONFIG`.
|
|
*/
|
|
reuseExisting?: boolean;
|
|
}
|
|
```
|
|
|
|
`log` must stay inside the project root and holds the raw, unredacted
|
|
output, so keep it in an ignored directory. A startup failure quotes the last
|
|
20 lines appended since the command started, with `command.env` values
|
|
replaced by `<secret:NAME>` markers. Without `reuseExisting`, a `readyUrl`
|
|
that already answers is `APP_ALREADY_RUNNING`. `reuseExisting` on an app
|
|
whose `app.url` asks for port 0 is `INVALID_CONFIG`: a process already
|
|
running cannot serve the port the run assigns.
|
|
|
|
The child inherits only `PATH`, `HOME`, the temp-directory variables, and
|
|
`command.env`, so pass what the app needs explicitly. Every `args` entry and
|
|
`env` value is a string; anything else, a number or a `secrets.get()` handle
|
|
included, is `INVALID_CONFIG`, except an `undefined` `env` value, which is
|
|
dropped. The runner never terminates a process it did not start.
|
|
|
|
### services
|
|
|
|
<Note>
|
|
Dependency processes (a database, an API mock, a dev server several targets
|
|
share) are not started by this version, and `services` is an unknown key
|
|
wherever it appears. Start them before the run, or have `app.command` start
|
|
a script that brings them up and serves the app. A services API returns in a
|
|
later release.
|
|
</Note>
|
|
|
|
## output
|
|
|
|
```ts
|
|
output: 'test-results/e2e',
|
|
```
|
|
|
|
Everything a run writes about itself goes under one directory, `.e2e` by
|
|
default, resolved from the project root. `--output <dir>` on
|
|
[`e2e run`](/reference/cli#e2e-run) and `e2e explore` overrides it for one
|
|
run.
|
|
|
|
| Path under the output | What |
|
|
| --- | --- |
|
|
| `report.json` | The canonical report; `--last-failed` reads the one here |
|
|
| `junit.xml`, `summary.md`, `failures/` | The `junit` and `markdown` reporters' files |
|
|
| `ai-trace.json` | `--ai-trace` |
|
|
| `artifacts/` | Screenshots, traces, videos, downloads, and logs, one directory per attempt. Cleared once a run's tests start, so it holds the latest run's evidence and nothing its report no longer names. A `--last-failed` rerun keeps the files the report it reruns names and writes its own attempts under `artifacts/rerun-<n>/` |
|
|
| `sessions/` | The run's encrypted `test.setup` sessions, deleted when it ends |
|
|
| `videos/<session>/` | [`e2e mcp`](/reference/mcp#recording) session recordings |
|
|
|
|
A run that stops before its tests start writes nothing here: a selection
|
|
that matches no test (`NO_TESTS`, a mistyped `--grep`), a collection error,
|
|
a target that cannot record what it asks for (`UNSUPPORTED_ARTIFACT`),
|
|
`NO_LAST_RUN`, an app or service that fails to start, or an interrupt
|
|
leaves the previous run's `report.json`, `ai-trace.json`, and `artifacts/`
|
|
as they were, so `--last-failed` still reads the last run that executed. An `e2e mcp` session writes its attempt's artifacts under the same
|
|
`artifacts/` and never clears it.
|
|
|
|
The replay cache is independent: [`cache.dir`](#cache) defaults to
|
|
`.e2e/cache` and resolves from the project root whatever `output` says, so
|
|
moving the results never moves the recordings. Because a run clears and
|
|
owns part of it, the output directory fails the config load with
|
|
`INVALID_CONFIG`, naming the reason, when it:
|
|
|
|
- is the project root, or outside it;
|
|
- is a file, or a path under one (a missing directory is created);
|
|
- is `cache.dir` or inside it, or would hold `cache.dir` under `artifacts/`,
|
|
`sessions/`, or `videos/`;
|
|
- holds a directory a `tests` glob scans (`output: 'tests'` with the default
|
|
`tests/**/*.e2e.ts`).
|
|
|
|
Paths are compared with symlinks resolved, so an absolute path spelled
|
|
through a symlink (`/tmp` for `/private/tmp` on macOS) names the same
|
|
directory as its target. A symlink whose target does not exist yet is
|
|
followed too: an output through one that points outside the project root is
|
|
refused.
|
|
|
|
`e2e init` writes `.gitignore` entries for the default `.e2e/` paths only.
|
|
With `output` set elsewhere, add that directory to `.gitignore` yourself.
|
|
|
|
The removed `--artifacts <dir>` flag fails with the `--output` it maps to:
|
|
the report used to go to the directory's parent, so `--artifacts
|
|
out/artifacts` is `--output out`. A parent an output cannot be (the project
|
|
root, or outside it) maps to `--output .e2e`.
|
|
|
|
## artifacts
|
|
|
|
```ts
|
|
artifacts: {
|
|
store: {
|
|
async put(artifact) {
|
|
const key = await bucket.upload(artifact.path, artifact.bytes);
|
|
return { ref: key };
|
|
},
|
|
},
|
|
},
|
|
```
|
|
|
|
`artifacts` says where artifacts go, not which are made: [`trace` and
|
|
`video`](#recording) choose the recordings, and a failure's screenshot and
|
|
screen text are captured whenever the engine can (never after a secret
|
|
fill, see [Security](/security)). The removed forms fail with
|
|
`INVALID_CONFIG` naming what replaced them: a kinds list (`artifacts:
|
|
['screenshot', 'trace']` or `artifacts.kinds`) names the `trace` mode it
|
|
meant, `artifacts.trace` names `trace: 'on-all-retries'` for `record:
|
|
'retries'` (read together with a kinds list that holds `trace`, so the pair
|
|
still means retries only), and `artifacts.video` names the `video` option.
|
|
|
|
`store` receives every artifact once it is complete on disk and returns a
|
|
reference the report records as `ref` beside the local path. A failed `put`
|
|
never fails the run. Each worker constructs its own store from the config
|
|
module. `redaction` is the report's label: a store that exports only what
|
|
the runner vouches for keeps back an `incomplete` artifact, a video or a
|
|
download the runner did not rewrite.
|
|
|
|
A video a hosted service keeps (a [provider's own recording](#what-a-video-writes))
|
|
is a link with no bytes, so `put` never receives it. A store that implements
|
|
the optional `putLink` gets each one instead, once the attempt stopped the
|
|
recording, and its `ref` lands on the report's artifact beside `url`. A
|
|
passed attempt's link under `retain-on-failure` is never handed over, since
|
|
the report leaves it out too, and a failed `putLink` never fails the run.
|
|
|
|
```ts
|
|
artifacts: {
|
|
store: {
|
|
async put(artifact) {
|
|
return { ref: await bucket.upload(artifact.path, artifact.bytes) };
|
|
},
|
|
async putLink(link) {
|
|
return { ref: await index.record(link.url, link.attemptId) };
|
|
},
|
|
},
|
|
},
|
|
```
|
|
|
|
```ts
|
|
export interface ArtifactsConfig {
|
|
/** Host store every produced artifact is handed to; omit it to keep files local only. */
|
|
store?: ArtifactStore;
|
|
}
|
|
|
|
export interface ArtifactStore {
|
|
put(artifact: StoredArtifact): Promise<{ readonly ref: string }>;
|
|
/** Receives a provider-hosted video as a link; optional. */
|
|
putLink?(link: StoredArtifactLink): Promise<{ readonly ref: string }>;
|
|
}
|
|
|
|
/** One produced artifact, handed to `store.put` once it is complete on disk. */
|
|
export interface StoredArtifact {
|
|
readonly kind: 'screenshot' | 'trace' | 'video' | 'download' | 'log';
|
|
readonly mediaType: string;
|
|
readonly bytes: Uint8Array;
|
|
readonly size: number;
|
|
readonly sha256: string;
|
|
/** Report-relative path with `/` separators, stable across runs of the same test. */
|
|
readonly path: string;
|
|
/** How much of the file the runner masked, as the report records it. */
|
|
readonly redaction: 'complete' | 'not-required' | 'incomplete';
|
|
readonly runId: string;
|
|
readonly testId: string;
|
|
readonly attemptId: string;
|
|
/** The step that produced it, when one was running. */
|
|
readonly stepId?: string;
|
|
/** When a time-based artifact (a video segment) began recording. */
|
|
readonly startedAt?: string;
|
|
}
|
|
|
|
/** One provider-hosted recording, handed to `store.putLink` once the attempt stopped it. */
|
|
export interface StoredArtifactLink {
|
|
readonly kind: 'video';
|
|
/** The `http(s)` URL the service serves the recording from. */
|
|
readonly url: string;
|
|
readonly mediaType: string;
|
|
/** A recording masks nothing, so a link is always `incomplete`. */
|
|
readonly redaction: 'incomplete';
|
|
readonly runId: string;
|
|
readonly testId: string;
|
|
readonly attemptId: string;
|
|
/** When the recording started, as an ISO timestamp. */
|
|
readonly startedAt: string;
|
|
/** The step that was running when the recording was registered, if any. */
|
|
readonly stepId?: string;
|
|
}
|
|
```
|
|
|
|
## Recording
|
|
|
|
```ts
|
|
trace: 'on-first-retry',
|
|
video: 'retain-on-failure',
|
|
targets: [
|
|
{ name: 'chromium', engine: web(), app: { url } },
|
|
{ name: 'firefox', engine: web({ browser: 'firefox' }), app: { url }, video: 'on-first-retry' },
|
|
],
|
|
```
|
|
|
|
`trace` and `video` choose which attempts record, with one set of modes:
|
|
|
|
```ts
|
|
/** Which attempts record a trace or a video, and which recordings are kept. */
|
|
export type RecordingMode = 'off' | 'on' | 'retain-on-failure' | 'on-first-retry' | 'on-all-retries';
|
|
```
|
|
|
|
| Mode | Records | Keeps |
|
|
| --- | --- | --- |
|
|
| `off` | nothing | nothing |
|
|
| `on` | every attempt | every recording |
|
|
| `retain-on-failure` | every attempt | the recordings of attempts that did not pass; a passed attempt's files are deleted once its verdict is in |
|
|
| `on-first-retry` | the first retry only, the second attempt | that recording. A test that passes first time records nothing |
|
|
| `on-all-retries` | every attempt after the first | every recording. A test that passes first time records nothing |
|
|
|
|
The names are Playwright's `use.trace` and `use.video` modes. Under a retry
|
|
mode a test with `retries: 0` records nothing, and neither does a failure the
|
|
runner does not retry, such as `LAUNCH_TIMEOUT`. When a test whose engine can
|
|
record runs that way, the run prints one notice per kind at plan time saying
|
|
no traces (or videos) will be recorded for it.
|
|
|
|
The old trace spellings fail with `INVALID_CONFIG` naming the mode they
|
|
meant, wherever a mode goes (the config, a target, a test, `--trace`):
|
|
`'all'` and `{ record: 'all' }` are `'on'`, and `'retries'` and `{ record:
|
|
'retries' }` are `'on-all-retries'`.
|
|
|
|
Four places set a mode, lowest first, and each replaces the one before it.
|
|
The same rule holds for `trace` and `video`:
|
|
|
|
1. `trace` or `video` at the top of the config.
|
|
2. A target's `trace` or `video`.
|
|
3. `--trace [mode]` or `--video [mode]` on [`e2e run`](/reference/cli#e2e-run),
|
|
over the config and every target. A bare flag is `on`.
|
|
4. A test's or describe's own [`trace` or `video` option](/reference/test#options),
|
|
innermost wins: `test('checkout', { video: 'on' }, ...)`.
|
|
|
|
A serial group records one trace and one video per group attempt, so its
|
|
members cannot set `trace` or `video` (a `COLLECTION_ERROR`, like `retries`).
|
|
The value on the serial `describe`, or on a group around it, applies to every
|
|
member.
|
|
|
|
`e2e explore` and an `e2e mcp` session are one attempt each and use the
|
|
target's mode, `--trace` and `--video` included for `explore`, so a retry
|
|
mode records nothing there. That includes the CI default: `CI=1 e2e explore`
|
|
records no trace unless you pass `--trace on`, and its plan-time notice says
|
|
so. An `e2e mcp` session always closes as passed, so
|
|
it keeps a trace only under `on`. It never records video from the configured
|
|
mode: its recordings are the ones `start_recording` and `stop_recording`
|
|
make.
|
|
|
|
Neither `trace` nor `video`, at the top or on a target, enters the config
|
|
digest, and neither is part of a replay's
|
|
[cache key](/cache#what-must-match): turning one on or off never
|
|
invalidates cached replays.
|
|
|
|
### trace
|
|
|
|
The default is `on` locally and `on-first-retry` in CI (the same `CI`
|
|
detection that makes `retries` default to 1 there). Recording a trace costs
|
|
CPU for the whole attempt: on a 141-test suite, every-attempt traces added
|
|
about a quarter to the run's wall time, which is why CI records only the
|
|
retry a flaky test leaves behind.
|
|
|
|
The web engine writes a Playwright trace per attempt, `trace/trace.zip` under
|
|
the attempt's artifact directory, which the runner redacts after a secret
|
|
fill; [Debugging](/debugging#artifacts) shows how to open it.
|
|
|
|
### video
|
|
|
|
The default is `off`. A video costs more than a trace and holds what the
|
|
screen showed, secrets included, so it is always asked for.
|
|
|
|
### Engines that cannot record
|
|
|
|
Before any test starts, the run grades every target. Where a mode was set
|
|
decides what an engine that cannot record that kind does:
|
|
|
|
- Set at the top of the config or with `--trace` / `--video`: the mode applies
|
|
to the targets whose engine can record it. The run prints one notice at
|
|
plan time naming the targets that cannot, and runs them without.
|
|
- Set on a target, or on a test that will run on that target: the mode is
|
|
required. A target whose engine cannot record it fails the run with
|
|
`UNSUPPORTED_ARTIFACT`, naming the target and, for a test's, the test.
|
|
|
|
Only a mode that records on some attempt counts: a run whose tests all
|
|
resolve to `off`, or to a retry mode with `retries: 0`, asks nothing of the
|
|
engine.
|
|
|
|
### What a video writes
|
|
|
|
| Recorder | Files under the attempt's artifact directory |
|
|
| --- | --- |
|
|
| Browser engine, Playwright screencast | `video/video.webm`, one segment per page the attempt showed, later ones as `video/video-part<n>.webm`. [`web({ screencast })`](/reference/web#options) sets the frame size and quality |
|
|
| Mobile engine, agent-device | `video/video.mp4` |
|
|
| A provider's own recording ([`BrowserProvider.record`](/browser#recording-on-the-provider), [`DeviceProvider.record`](/mobile#hosted-devices)) | the file the provider wrote, as `video/<file>`: Kernel's is `video/replay.mp4`. Or a link to a recording the service keeps, recorded with `url` in place of `path`; the runner never downloads it |
|
|
|
|
Each segment carries `startedAt`, so `step.startedAt - artifact.startedAt` is
|
|
a step's offset into it.
|
|
|
|
A recording masks nothing and is recorded with `redaction: 'incomplete'`.
|
|
Under `retain-on-failure` a link is left out of the report of a passed
|
|
attempt, but the service still holds the recording.
|
|
|
|
## agents
|
|
|
|
```ts
|
|
/** A project tool as `defineTool` from `e2e/agent` returns it. */
|
|
export interface AgentTool {
|
|
readonly tool: object;
|
|
readonly annotations: { readonly mutates: boolean; readonly platforms?: readonly string[] };
|
|
}
|
|
|
|
/** The options every agent takes, whichever brain runs its `act` steps. */
|
|
export interface AgentOptions {
|
|
/** An AI SDK model instance; no implicit default. */
|
|
model?: ModelInstance;
|
|
/** Model for `assert`, `waitFor`, and `extract`; defaults to `model`. */
|
|
judge?: ModelInstance;
|
|
/**
|
|
* What the app calls things, told to every model call this agent makes,
|
|
* act turns and judgments alike. At most 16384 bytes.
|
|
*/
|
|
context?: string;
|
|
/** Committed actions per agent call, 1 through 100; default 25. */
|
|
maxSteps?: number;
|
|
/** Model requests per agent call, 1 through 100; default 25. */
|
|
maxModelCalls?: number;
|
|
/** Deadline of one `assert`, `waitFor`, or `extract` call in milliseconds; default 30000. Raise it for a slow judge; engine operations keep `actionTimeout`. */
|
|
judgmentTimeout?: number;
|
|
/** Observation payload ceiling for act turns and judgments, 1024 through 16777216; default 262144. */
|
|
maxObservationBytes?: number;
|
|
/** Input tokens per model request, 1 through 1000000; default 64000. A dense screen is cut to fit under it. */
|
|
maxInputTokens?: number;
|
|
/**
|
|
* Provider options every model call carries, e.g. a reasoning effort.
|
|
* OpenAI and Azure OpenAI calls also carry `store: false` and a prompt
|
|
* cache key unless set here.
|
|
*/
|
|
providerOptions?: ProviderOptions;
|
|
}
|
|
|
|
/** One entry of `agents`: the built-in agent with its options, or a custom brain under `executor`. */
|
|
export type AgentConfig = AgentOptions &
|
|
(
|
|
| {
|
|
/** How the acting agent should work, appended to the built-in rules; only the act loop reads it. */
|
|
system?: string;
|
|
/** Project tools from `defineTool`, offered beside the built-in toolset. */
|
|
tools?: Readonly<Record<string, AgentTool>>;
|
|
}
|
|
| {
|
|
/** A custom brain `agent.act()` dispatches to instead of the built-in agent. */
|
|
executor: StepExecutor;
|
|
}
|
|
);
|
|
```
|
|
|
|
Tests use the `default` agent unless you select another with
|
|
`e2e run --agent <name>`, a test or group option, or an individual `agent.*`
|
|
call. A test with `{ agent: ['buyer', 'admin'] }` runs once for each agent.
|
|
`default` exists even when the config declares no agents, but has no model.
|
|
|
|
Names may contain ASCII letters, numbers, `_`, `-`, or `.`, and are never
|
|
only dots: a name is an artifact path segment. An unknown name
|
|
is `INVALID_CONFIG` on the CLI flag, `COLLECTION_ERROR` on a test or group,
|
|
and `INVALID_ARGUMENT` on a call. See [Agents and personas](/agents).
|
|
|
|
```ts title="e2e.config.ts"
|
|
import type { E2EConfig } from 'e2e';
|
|
import { web } from '@e2e-dev/web';
|
|
import { gateway } from 'ai';
|
|
import { houseBrain } from './house-brain.ts';
|
|
|
|
export default {
|
|
targets: [{ engine: web(), app: { url: 'http://localhost:3000' } }],
|
|
agents: {
|
|
default: { model: gateway('openai/gpt-6-luna-fast') },
|
|
ux: { model: gateway('openai/gpt-6-luna-fast'), system: 'Check labels, contrast, and keyboard access.' },
|
|
thorough: { model: gateway('anthropic/claude-opus-5'), context: 'Verify every claim on screen.' },
|
|
buyer: {
|
|
model: gateway('openai/gpt-6-luna-fast'),
|
|
system: 'You are a first-time buyer.',
|
|
context: 'Plans are called tiers. Billing lives under Settings.',
|
|
maxSteps: 40,
|
|
},
|
|
house: { executor: houseBrain, model: gateway('openai/gpt-6-luna-fast'), maxModelCalls: 10 },
|
|
},
|
|
} satisfies E2EConfig;
|
|
```
|
|
|
|
Every entry starts from the built-in defaults. Agents do not inherit from
|
|
each other: `ux` above gets no `context` from `default`, and a key left out of
|
|
an entry is the default value, never another entry's.
|
|
|
|
With `executor`, `model`, `judge`, `context`, and the budgets still apply
|
|
(judgments need a model), while `system` and `tools` are `INVALID_CONFIG`: the
|
|
executor brings its own prompt and tools. `maxModelCalls` bounds the model
|
|
turns of one step. A string in `model` is `INVALID_CONFIG` naming the
|
|
constructor to write, and so is a `model` or `judge` that differs from the one
|
|
the executor brought. See [Custom executors](/executors) and
|
|
[A judge of its own](/models#a-judge-of-its-own).
|
|
|
|
Loading the config fails with `INVALID_CONFIG` for an unknown key (naming the
|
|
nearest known one), a removed key, and a value in the old shapes:
|
|
|
|
| Written | Now |
|
|
| --- | --- |
|
|
| a bare `StepExecutor` as the entry | `{ executor }` |
|
|
| `timeout` | `judgmentTimeout` |
|
|
| `maxTurns` (on the entry or `createToolLoopExecutor`) | `maxModelCalls` |
|
|
| top-level `limits.maxModelTokensPerCall` | `maxInputTokens` on each agent |
|
|
| top-level `limits.maxAgentContextBytes`, `maxLedgerBytes`, `maxEventsPerStep` | removed; the runner fixes them at 16384 bytes, 8192 bytes, and 1000 events |
|
|
|
|
Judgments (`assert`, `waitFor`, `extract`) allow 8192 output tokens per call
|
|
and see the instruction and the current screen only.
|
|
|
|
### model
|
|
|
|
```ts
|
|
/**
|
|
* An AI SDK LanguageModelV2+ instance. Its provider handles transport
|
|
* and credentials.
|
|
*/
|
|
export interface ModelInstance {
|
|
/** The AI SDK specification the instance implements. */
|
|
readonly specificationVersion: string;
|
|
/** Provider id, recorded per step in the report. */
|
|
readonly provider: string;
|
|
/** Model id, recorded per step in the report. */
|
|
readonly modelId: string;
|
|
/** The generate call every AI SDK language model implements; what tells a live instance from its three id strings. */
|
|
readonly doGenerate: (...args: never[]) => unknown;
|
|
}
|
|
```
|
|
|
|
Construct the model in your config with an AI SDK provider. You can use all
|
|
providers that support tool calls, see [AI SDK providers](/models#ai-sdk-providers).
|
|
For subscription sign-in, use the [`e2e/oauth/*` models](/subscriptions).
|
|
|
|
| Gateway | Constructor | Package | Reads |
|
|
| --- | --- | --- | --- |
|
|
| [Vercel AI Gateway](https://vercel.com/docs/ai-gateway) | `gateway('openai/gpt-6-luna-fast')` | `ai` | `AI_GATEWAY_API_KEY`, or a Vercel OIDC token without it |
|
|
| [OpenRouter](https://openrouter.ai/models) | `openrouter('openai/gpt-6-luna-fast')` | `@openrouter/ai-sdk-provider` | `OPENROUTER_API_KEY` |
|
|
| Any OpenAI-compatible endpoint | `createOpenAICompatible({ name, baseURL, apiKey? }).chatModel('llama3.2')` | `@ai-sdk/openai-compatible` | `apiKey`, when the endpoint needs one |
|
|
| A provider directly | `openai('gpt-6-luna')`, `anthropic(...)`, ... | `@ai-sdk/openai`, `@ai-sdk/anthropic`, ... | the provider's own variable |
|
|
|
|
Reports record `provider` and `modelId` per step. Each worker constructs its
|
|
own instance from the config module; a digest that differs between the
|
|
runner and a worker is `CONFIG_NOT_DETERMINISTIC`. See
|
|
[Models](/models).
|
|
|
|
## credentials
|
|
|
|
```ts
|
|
/** One `config.credentials` entry: an account whose password is a secret named `<entry>.password`. */
|
|
export interface CredentialConfig {
|
|
username: string;
|
|
password: string | SecretProvider;
|
|
}
|
|
|
|
type SecretProvider = () => string | Promise<string>;
|
|
```
|
|
|
|
A `SecretProvider` is called on every authorized fill (a vault lookup, a
|
|
freshly computed TOTP) and recorded in the config digest by name only. An
|
|
`E2E_USER_*` override wins over it. A static password is at least 6
|
|
characters (code points), like a secret, or `INVALID_CONFIG`; a provider
|
|
returning fewer fails the fill with `AUTH_CREDENTIAL_UNAVAILABLE`.
|
|
|
|
`credentials.user(name)` resolves a configured entry into a `Credential`.
|
|
The username is a plain string; the password is an opaque `Secret` that test
|
|
code cannot read, accepted only by `locator.fill` and `agent.act` params. An
|
|
unconfigured name throws `AUTH_CREDENTIAL_UNAVAILABLE`.
|
|
|
|
Credentials and secrets are separate namespaces: `secrets.get(name)` returns a
|
|
`secrets` entry only, and a credential's password is reachable only as
|
|
`credentials.user(name).password`. The agent, the replay cache, and the
|
|
reports know that handle as `<name>.password` (`<secret:admin.password>`), so
|
|
`credentials.admin` and `secrets.admin` can both exist; a `secrets` entry
|
|
named exactly `admin.password` beside an `admin` credential is
|
|
`INVALID_CONFIG`. So are two credentials whose names map to one
|
|
`E2E_USER_<NAME>_*` override, like `svc-a` and `svc_a` (see
|
|
[Environment variables](/reference/environment)).
|
|
|
|
```ts
|
|
/** Named test identity. The password remains an opaque Secret. */
|
|
export interface Credential {
|
|
/** The entry's name in `credentials`. */
|
|
readonly name: string;
|
|
/** Plain username. */
|
|
readonly username: string;
|
|
/** Opaque; test code cannot read it. */
|
|
readonly password: Secret;
|
|
readonly [credentialBrand]: true;
|
|
}
|
|
```
|
|
|
|
See [Signing in](/authentication).
|
|
|
|
## secrets
|
|
|
|
```ts
|
|
/** One `config.secrets` entry: the value itself, or a provider computing it at fill time. */
|
|
export type SecretConfig = string | SecretProvider;
|
|
```
|
|
|
|
An entry is a static value, or a provider called on every authorized fill (a
|
|
vault lookup, a freshly computed TOTP). It is recorded in the config digest by
|
|
name only, and an `E2E_SECRET_<NAME>` override wins over both. Two entries whose
|
|
names map to one variable, like `api-key` and `api_key`, are `INVALID_CONFIG`. Every entry
|
|
must resolve to a string of at least 6 characters (code points) or a
|
|
function; anything else is `INVALID_CONFIG`. The minimum exists for redaction: every occurrence
|
|
of a value becomes `<secret:name>`, and a shorter value would take ordinary
|
|
text in reports and output with it. A provider's value is held to the same
|
|
minimum at fill time, with `SECRET_UNAVAILABLE`.
|
|
|
|
`secrets.get(name)` returns the opaque `Secret` for a configured entry. It has
|
|
no plaintext accessor and is accepted only by `locator.fill` and `agent.act`
|
|
params; the runner fills the field, masks the value in every observation, and
|
|
redacts it from logs, traces, and the report. An unconfigured name throws
|
|
`SECRET_UNAVAILABLE`.
|
|
|
|
Called while `e2e.config.ts` evaluates, before any run exists, `secrets.get(name)`
|
|
returns a reference by name that an engine option can hold, such as
|
|
`web({ basicAuth: { username, password: secrets.get('previewPassword') } })`.
|
|
The engine declares it, the config load fails with `INVALID_CONFIG` when no
|
|
`secrets` entry has the name (a credential's password is not one: declare a
|
|
value an engine option needs under `secrets`), and the engine resolves the value
|
|
when each attempt starts (a provider runs again every time). The value joins
|
|
the attempt's redaction then, without the screenshot restrictions a fill
|
|
brings, since it goes to the engine rather than onto the page. The digest
|
|
records the name only. The reference is not the value: turned into a string
|
|
(a template literal, `String()`, `+`, `JSON.stringify`) it throws
|
|
`INVALID_CONFIG`, and so does one placed in an `args` entry or `env` value of
|
|
`app.command`, which takes strings and placeholders only. Read a value a string needs, such as an app's API
|
|
key, from `process.env` yourself.
|
|
|
|
```ts
|
|
/**
|
|
* What a secret is for, which decides where it may be filled: a `password`
|
|
* goes only into a password field; a `generic-secret` (an API key, a token,
|
|
* any value from the environment) goes into any editable input.
|
|
*/
|
|
export type SecretPurpose = 'password' | 'generic-secret';
|
|
|
|
/** Opaque host-side value accepted only by sensitive input sinks. */
|
|
export interface Secret {
|
|
/** The entry's name in `secrets`, or `<credential>.password` for a credential's password. */
|
|
readonly name: string;
|
|
/** Where the value may be filled. */
|
|
readonly purpose: SecretPurpose;
|
|
readonly [secretBrand]: true;
|
|
}
|
|
```
|
|
|
|
See [Secrets that are not passwords](/authentication#secrets-that-are-not-passwords).
|
|
|
|
## cache
|
|
|
|
In `read-write` mode the replay cache records an `agent.act()` only after a
|
|
later verification passes, such as a locator assertion or `agent.assert()`.
|
|
`read-only` replays existing entries and writes none; `off` disables both.
|
|
See [Caching agent steps](/cache).
|
|
|
|
```ts
|
|
/**
|
|
* Replay cache options. Defaults to files under `.e2e/cache/`.
|
|
* A custom store replaces file storage; each worker constructs its own.
|
|
*/
|
|
export interface CacheConfig {
|
|
/** Default `read-write`; `read-only` in CI when unset. */
|
|
mode?: CacheMode;
|
|
/** Custom entry store; omit it to use the file store at `dir`. */
|
|
store?: CacheStore;
|
|
/** File store directory, resolved against the project root; default `.e2e/cache`, whatever `output` says. */
|
|
dir?: string;
|
|
/** Fail a step whose recording no longer replays with `REPLAY_STALE`. Default `false`. */
|
|
strict?: boolean;
|
|
}
|
|
|
|
/** Where the replay cache keeps its entries: the file store by default, or a host's own through `cache.store`. */
|
|
export interface CacheStore {
|
|
readonly writable: boolean;
|
|
read(keyHash: string): Promise<CacheReadResult>;
|
|
/** Persists one trace, returning its size, or undefined when not written. */
|
|
write(keyHash: string, payload: ActionTrace): Promise<{ bytes: number } | undefined>;
|
|
/**
|
|
* Evicts one entry, best-effort. Called when a replayed or just-recorded
|
|
* flow is implicated in a failed attempt: the entry re-records on the next
|
|
* pass instead of replaying a poisoned flow forever. Optional: a store
|
|
* without eviction merely stays stale until the next confirmed write.
|
|
*/
|
|
delete?(keyHash: string): Promise<void>;
|
|
}
|
|
|
|
export type CacheReadResult =
|
|
| { readonly status: 'hit'; readonly entry: TraceEntry; readonly bytes: number }
|
|
| { readonly status: 'miss' }
|
|
| { readonly status: 'invalid'; readonly reason: string; readonly bytes?: number };
|
|
```
|
|
|
|
`CacheStore`, `CacheReadResult`, `TraceEntry`, and `ActionTrace` are
|
|
exported from `e2e`.
|
|
|
|
A bare string is shorthand for `{ mode }`. Set `'read-write'` explicitly when
|
|
CI restores the cache from its own cache service; `--no-cache` wins over
|
|
both. A custom `store` is exempt from the CI demotion and states its own
|
|
trust through `writable`. `strict` (or `--strict-cache` for one run) turns a
|
|
recording that exists but no longer replays into a `REPLAY_STALE` failure
|
|
instead of a hand-off to the agent; see
|
|
[Fail on a stale recording](/cache#fail-on-a-stale-recording).
|
|
|
|
### Commit the replay cache
|
|
|
|
`e2e init` adds `.e2e/cache/` to `.gitignore`, so committing entries is
|
|
opt-in. Remove that line to share replays with CI and teammates; CI stays
|
|
`read-only` unless `cache: 'read-write'` is set. See
|
|
[Caching agent steps](/cache#commit-the-replay-cache).
|
|
|
|
## Run limits
|
|
|
|
The report's `run.limits` block records the ceilings a run enforced. There is
|
|
no `limits` key: `maxObservationBytes` and `maxModelTokensPerCall` are the
|
|
largest `maxObservationBytes` and `maxInputTokens` any configured agent
|
|
sets, and the rest are fixed by the runner.
|
|
|
|
| Report field | Value |
|
|
| --- | --- |
|
|
| `maxAgentContextBytes` | 16384: an agent's `context` and a test's `agentContext` |
|
|
| `maxLedgerBytes` | 8192: the prior-step ledger an act step reads |
|
|
| `maxEventsPerStep` | 1000: events recorded per step |
|
|
| `maxObservationBytes` | the largest `agents.<name>.maxObservationBytes` |
|
|
| `maxModelTokensPerCall` | the largest `agents.<name>.maxInputTokens` |
|
|
|
|
Per-call options may only lower an agent's budgets.
|
|
|
|
## Environment variables
|
|
|
|
The runner reads a closed set, listed on the
|
|
[Environment variables](/reference/environment) page. `E2E_USER_<NAME>_USERNAME`
|
|
and `E2E_USER_<NAME>_PASSWORD` override a credential, `E2E_SECRET_<NAME>` a
|
|
secret, and `CI` switches the defaults below.
|
|
|
|
### Resolution order
|
|
|
|
| Setting | Order |
|
|
| --- | --- |
|
|
| Base URL | the target's `app.url`, else `APP_URL_REQUIRED` at the first navigation. The runner reads no `APP_URL`; the scaffolded config reads it itself |
|
|
| Built-in agent model | `agents.<name>.model`, then `MODEL_UNAVAILABLE` at `agent` acquisition; an agent never falls back to `default`'s model |
|
|
| Custom executor model | `agents.<name>.model`, else the model the executor brought; two different values is `INVALID_CONFIG` |
|
|
| Model credential | the provider package's own variable or the key passed at construction, read by the provider on the first model call |
|
|
| Credential username and password | the `E2E_USER_*` variable, then the config value |
|
|
|
|
The built-in agent checks for a missing model once per run, when the `agent`
|
|
fixture is first acquired, and stops the run with exit 2. A custom executor
|
|
can implement `act` and `assert` without a model. `waitFor` and `extract`
|
|
still require one.
|
|
|
|
### CI
|
|
|
|
CI mode is on unless `CI` is unset, blank, `0`, or `false`.
|
|
|
|
| Setting | Local | CI |
|
|
| --- | --- | --- |
|
|
| `retries` | `0` | `1` |
|
|
| `workers` | `max(1, floor(availableParallelism() / 2))` | `1` |
|
|
| `cache` | `read-write` | `read-only` when unset |
|
|
| `test.only` | allowed | `ONLY_IN_CI`, exit code 2 |
|
|
| `command.reuseExisting` | honored | ignored, with a notice |
|