mirror of
https://github.com/tester-army/e2e.git
synced 2026-11-21 03:05:26 +08:00
* docs: add a Next.js guide page
One page takes a Next.js developer from `e2e init` to CI: `next dev`
locally and `next start` on the build in CI from one config, the port,
`reuseExisting`, and the Next.js problems (`allowedDevOrigins` on
127.0.0.1, the route announcer that `getByRole('alert')` also matches, a
webpack watcher that reads `.e2e/`), plus the `create-next-app` starter
and the Vercel template. The hydration accordion moves here from web.mdx;
web, cache, and example-projects link to the page, with ja and zh
translations, and the e2e skill's setup reference carries the same
guidance.
Refs ENG-1378
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* docs: move the Next.js page into a Frameworks group
Next.js is not a platform like web, mobile, or desktop, so the page moves
to docs/frameworks/nextjs.mdx under a new Frameworks navigation group. The
web page no longer points at it from the app URL paragraph.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
454 lines
21 KiB
Plaintext
454 lines
21 KiB
Plaintext
---
|
|
title: Caching
|
|
description: Replay verified agent actions without model calls, and let the agent take over when the app changes.
|
|
"og:image": "https://e2e.com/docs/images/og/cache.png?v=d035986b13"
|
|
"twitter:image": "https://e2e.com/docs/images/og/cache.png?v=d035986b13"
|
|
---
|
|
|
|
The replay cache records the actions of an `agent.act()` step after a later
|
|
check verifies the result. On the next run, the runner replays those
|
|
actions without a model call. When a control or the expected end state no
|
|
longer matches, the agent takes over from the current screen. See
|
|
[Self-healing tests](/self-healing).
|
|
|
|
`agent.assert`, `agent.waitFor`, and `agent.extract` always run live. The
|
|
run summary counts the outcomes:
|
|
|
|
```text
|
|
AI 4.1k tokens · 2 model calls · openai/gpt-6-luna-fast
|
|
Cache 4 replayed · 1 handed off · 1 missed · 1 not saved
|
|
```
|
|
|
|
`not saved` counts the steps of passing and flaky tests whose
|
|
[`step.cache.write`](#when-a-recording-is-written) is `unconfirmed`,
|
|
`no-change`, or `not-written` in the attempt that passed.
|
|
|
|
## When a recording is written
|
|
|
|
End the test with a check of its outcome. In `read-write` mode, the runner
|
|
saves the recording of each `act` at the end of the attempt when a later
|
|
verification step passed. One check at the end confirms every `act` before
|
|
it.
|
|
|
|
```ts title="tests/cart.e2e.ts"
|
|
import { expect, test } from 'e2e';
|
|
|
|
test('adds an item', async ({ app, agent, screen }) => {
|
|
await app.open('/cart');
|
|
|
|
await agent.act('add one item to the cart');
|
|
|
|
await expect(screen.getByRole('status')).toHaveText('1 item');
|
|
});
|
|
```
|
|
|
|
| Counts as verification | Does not count |
|
|
| --- | --- |
|
|
| `expect(locator).toHaveText(...)` and other locator matchers | `expect(value).toBe(...)` on a plain value |
|
|
| `expect(browser).toHaveURL(...)` and other engine assertions | `expect.poll(...)` |
|
|
| `locator.waitFor()`, `browser.waitForURL()` | `locator.textContent()` and other reads |
|
|
| `agent.assert(...)`, `agent.waitFor(...)` | `agent.extract(...)`, another `act`, the attempt passing |
|
|
|
|
`step.cache.write`, in the report only, says what happened to the recording
|
|
when the attempt ended:
|
|
|
|
| `step.cache.write` | Meaning |
|
|
| --- | --- |
|
|
| `saved` | Recorded for the next run. |
|
|
| `kept` | The entry stays as it is. |
|
|
| `unconfirmed` | Not saved: no check passed after the step. End the test with a locator or engine assertion or `agent.assert`, or wait for the state with `locator.waitFor()` or `agent.waitFor`. |
|
|
| `evicted` | The entry was deleted: its replay ended in a failure or a repair, or no check confirmed it before the attempt failed. |
|
|
| `no-change` | Nothing recorded: no control appeared, went away, or changed state, and the route stayed the same. That includes a step whose only change is inside a [frame from another site](/browser#reach-into-an-iframe). Such a step runs live on every run. |
|
|
| `not-written` | Not saved: the store rejected the write, or refused the recording, for example one over the file store's size limit. |
|
|
|
|
A step that took no action records nothing and reports no `write`.
|
|
|
|
A failing attempt still saves the steps verified before the failure. It
|
|
deletes unverified entries and the replays involved in the failure. To keep
|
|
setup goals recorded while you fix a failing assertion, end the setup with
|
|
a wait for its state, such as `locator.waitFor()`. A custom store needs a `delete` method for this. Cancelling
|
|
a run leaves the cache unchanged.
|
|
|
|
<Accordion title="Failures where no model answered">
|
|
An attempt whose only failures are ones where no model answered evicts
|
|
nothing, since these say nothing about the app:
|
|
|
|
- `MODEL_UNAVAILABLE`: no model is configured, or GitHub Copilot doesn't
|
|
serve it to the account.
|
|
- `MODEL_PROVIDER_FAILED`: the provider returned an error, such as a
|
|
missing or rejected key, a rate limit, or a 5xx, or a request got no
|
|
response within 120 seconds.
|
|
|
|
So a `read-write` run without model access keeps its recordings, including
|
|
a step that replayed before an `agent.assert` that could not reach its
|
|
model.
|
|
|
|
These count as a failure of the flow: an unusable model answer
|
|
(`MODEL_OUTPUT_INVALID`, `CONTEXT_OVERFLOW`), a refusal (`MODEL_REFUSED`),
|
|
`STEP_TIMEOUT` (even with a
|
|
request still open), and any other failure in the same attempt, such as an
|
|
`afterEach` assertion.
|
|
</Accordion>
|
|
|
|
## What must match
|
|
|
|
A cache entry belongs to one test, target, instruction, set of params, and
|
|
agent. Each of these changes causes a miss:
|
|
|
|
- renaming the test, its `describe` group, or the target, or moving the
|
|
test file,
|
|
- changing the instruction or an ordinary param,
|
|
- running with another configured agent (`agent: ['buyer', 'admin']`
|
|
records one entry each), or changing the agent's `context` or the test's
|
|
`agentContext` (a secret in it enters the key as its name only),
|
|
- switching to another engine, or changing the project id or the app's
|
|
identity.
|
|
|
|
Changing the model doesn't cause a miss, and neither does upgrading the
|
|
engine: a node the new version resolves differently misses at replay.
|
|
|
|
Use `--no-cache` to evaluate a new model. Replays expose
|
|
`step.cache.recordedBy` when the entry has provenance; older or unattributed
|
|
entries omit it.
|
|
|
|
A timestamp or a fresh email address would miss on every run. Wrap such
|
|
values in `unique()`, and the runner puts in the current value at replay:
|
|
|
|
```ts
|
|
import { unique } from 'e2e';
|
|
|
|
const name = `E2E ${Date.now()} Company`;
|
|
await agent.act('create a company named {name}', { params: { name: unique(name) } });
|
|
await expect(screen.getByRole('heading', { name })).toBeVisible();
|
|
```
|
|
|
|
Use `unique()` for data the flow enters or checks, and ordinary params for
|
|
choices that change the flow, such as a plan name. The substitution also
|
|
covers the value as a URL spells it (`%20`, `+`) and a multi-line value as
|
|
the screen shows it, on one line. When a `unique()` value
|
|
equals or sits inside another param's value, the step isn't recorded:
|
|
`step.cache.notRecorded` reads `param-collision`, and the next run records
|
|
it.
|
|
|
|
For preview URLs that change on each deploy, set
|
|
[`app.identity`](/reference/config#the-app-under-test) to keep one cache
|
|
identity. On devices, replay uses the
|
|
[screen title](/reference/mobile#replay-cache-on-a-device) as the path.
|
|
|
|
## Rerun a test while you fix it
|
|
|
|
While you fix a failing test, keep the cache on in `read-write` mode, the
|
|
local default, and outside [strict mode](#fail-on-a-stale-recording),
|
|
which never writes. With `CI` set, an unset mode is `read-only`. A failed
|
|
run keeps the recordings a check
|
|
[confirmed](#when-a-recording-is-written) before the failure. The next run
|
|
replays those goals without model calls, as long as each replay still
|
|
matches the screen.
|
|
|
|
1. Run only that test, by the line its `test(` opens on:
|
|
`npx e2e run tests/cart.e2e.ts:3`.
|
|
2. Don't rename the test or reword a goal that passes. Each
|
|
[causes a miss](#what-must-match): a rename for every goal in the test,
|
|
a reworded goal for itself.
|
|
3. When retries are on (`retries` in the config, or `CI` set), add
|
|
`--retries 0`. A `retries` option on the test or its `describe` wins
|
|
over the flag. A retry doesn't replay, so it runs every goal live.
|
|
4. When the test passes, run it one time with `--no-cache`. Every goal
|
|
runs live, which shows the agent still reaches it without a recording.
|
|
|
|
## How a replay works
|
|
|
|
1. Check the starting screen: it must be the same route.
|
|
2. Find each recorded control by role, name, test id, and surrounding
|
|
context, and repeat its action. The runner waits for missing controls and
|
|
stops when a match is ambiguous.
|
|
3. Check the final route and what the original step changed: the controls
|
|
that appeared, with their checked, selected, expanded, or pressed state,
|
|
and the ones that went away.
|
|
4. Finish without a model call, or let the agent continue with a record of
|
|
what ran and why replay stopped.
|
|
|
|
Typed values replay as recorded. A secret fill resolves the current value
|
|
by name and goes through authorization again.
|
|
|
|
A replay finishes as soon as the recorded changes show, which can be before
|
|
content below them has loaded. To check content that loads after the step,
|
|
use `agent.waitFor`, which waits for it, rather than `agent.assert`, which
|
|
judges the screen once.
|
|
|
|
<Accordion title="What counts as the same route">
|
|
A route is the origin, the path, and the query. The record id and the
|
|
fragment don't count. Ids, tokens, timestamps, and cache busters in the
|
|
path or the query read as an id placeholder, so `/orders/42?t=1727780000`
|
|
and `/orders/7?t=1727780999` are one route.
|
|
|
|
Every other query value is part of the route: `?mode=safe` and
|
|
`?mode=unsafe` differ, and so do `/companies` and `/companies?tab=notes`, or
|
|
a new tracking parameter. `/products/summer-sneaker` and
|
|
`/products/winter-boot` are different routes too.
|
|
|
|
A location on the app's own origin is compared by its path, so a recording
|
|
follows the app to a preview deployment that shares its `app.identity`. A
|
|
page on any other origin is another route.
|
|
</Accordion>
|
|
|
|
<Accordion title="How the end check reads the screen">
|
|
- At least one recorded change must happen during the replay. An outcome
|
|
already on screen before the actions proves nothing. When the step's last
|
|
action moved to another route, that move is the evidence.
|
|
- An alert that the replay raised and the recording never saw stops the
|
|
replay. The web engine does not observe the
|
|
[Next.js](/frameworks/nextjs#fix-next-js-problems) App Router's route announcer, so
|
|
its title echo never counts.
|
|
- Text that differs on every run, such as a date, a duration, or an id,
|
|
is skipped while a stable change exists on either side of the step's delta,
|
|
whether something appeared or disappeared. With nothing else, it is kept,
|
|
and a changed value hands the replay off. In an alert, it reads as a placeholder.
|
|
A node known only by a test id the app generates (`node-<uuid>`) is
|
|
skipped the same way. When kept, a node that appeared may carry any id
|
|
with that pattern; one that went away is still checked by its exact id.
|
|
- A count that names what it counts, such as `3 records imported` or
|
|
`42件`, must read the same on replay when the step made it appear on the
|
|
screen it acted on. A count that only moved (`41 items` to `42 items`),
|
|
or one on a screen the step navigated to, is skipped only when its own
|
|
side of the delta has a stable change. Otherwise the count stays, even
|
|
when the opposite side has a stable change. A bare number such as a badge
|
|
follows the text rule above.
|
|
- A control with no name is found by its place among the unnamed controls
|
|
of its kind inside the same named row, list item, or group. A control
|
|
known only by a generated test id is found by the id pattern the same
|
|
way, or when it was the only one with that pattern.
|
|
</Accordion>
|
|
|
|
## Why a step was not replayed
|
|
|
|
Each `act` step reports `step.cache`, absent when caching is off for the
|
|
step. `agent.act()` returns it as `result.cache`, without `write`, and the
|
|
[trace page](/debugging#traces) shows it under the step.
|
|
|
|
| Run summary | `step.cache.mode` | Meaning |
|
|
| --- | --- | --- |
|
|
| `replayed` | `self-finalized` | The recording completed without a model call. |
|
|
| `handed off` | `agent-concluded` | Replay started, then the agent took over. |
|
|
| `missed` | `missed` | The agent ran the step from the beginning. |
|
|
|
|
Both carry `step.cache.reason`, and `step.cache.detail` says where the
|
|
replay stopped and why (`at action 2 of 3, tap button "Apply": ...`). After
|
|
an `end-mismatch` it names the recorded nodes that did not show, the ones
|
|
that did not go away, and any alert the recording never saw, or says the
|
|
outcome already showed before the replay acted.
|
|
`step.cache.entry` is the entry's key and its file name (`<entry>.json`) in
|
|
the file store.
|
|
|
|
| Reason | What happened | What to do |
|
|
| --- | --- | --- |
|
|
| `no-entry` | No recording matches this step and its params. | The next verified pass records it, unless the step's `step.cache.write` is `no-change`. |
|
|
| `retry` | Retries never replay, except under `--strict-cache` or `cache.strict`. | Nothing. |
|
|
| `invalid-entry` | The file is unreadable, over 1 MiB, or from another schema version. | `npx e2e cache clear`. |
|
|
| `truncated` | The recording hit a limit, see [below](#limits-and-gaps). | Split the step, or pass the secret as a `Secret`. |
|
|
| `wrong-context` | The app was on another route than when the step was recorded. See [How a replay works](#how-a-replay-works). | Open the same screen before the step. |
|
|
| `target-not-found` | A recorded control did not appear within 15 seconds, or it has no name and no named container to tell it from its twins. | The UI changed: the next passing run records it again. For an unnamed control, give it a label or put it in a named row or group. |
|
|
| `target-ambiguous` | The number of matching controls changed. | Give repeated controls distinct names or use unique test data. |
|
|
| `gap` | An action can't be replayed, see [below](#limits-and-gaps). | The agent does this part live. Put stable values in `params`. |
|
|
| `action-failed` | The app rejected a replayed action. | Check the app. |
|
|
| `action-uncertain` | A replayed action failed, and it's unknown whether it took effect. The agent is told to check before it acts again. | Check the app. |
|
|
| `viewport-changed` | A bare-point tap, hover, or drag was recorded on a viewport of another size. | Keep the viewport fixed. |
|
|
| `end-mismatch` | Every action ran, but the [end check](#how-a-replay-works) failed. | Leftover data from an earlier run, a banner that comes and goes, or a regression. Unique test data and cleanup in `afterEach` help. |
|
|
|
|
To rule the cache out of a failure, run one time without it:
|
|
|
|
<CodeGroup>
|
|
```bash npm
|
|
npx e2e run --no-cache
|
|
```
|
|
|
|
```bash pnpm
|
|
pnpm exec e2e run --no-cache
|
|
```
|
|
|
|
```bash bun
|
|
bunx e2e run --no-cache
|
|
```
|
|
</CodeGroup>
|
|
|
|
### Limits and gaps
|
|
|
|
A recording is `truncated` when the step:
|
|
|
|
- took more than 50 actions, which is possible only with `maxSteps` above 50,
|
|
- typed a value or URL over 4,096 characters, or one that holds a
|
|
registered secret,
|
|
- acted on a control with no usable description.
|
|
|
|
A `gap` is an action the cache can't repeat: a mutating
|
|
[project tool](/tools), or a value the agent read off the screen, typed
|
|
after it saw a screenshot, or worked out from the date. `step.cache.derived`
|
|
names the rule: `whole-node` (a control's whole name or text),
|
|
`minted-token` (a code or id the app showed as its own word, or any value
|
|
typed into a field marked `autocomplete="one-time-code"`), `date`, or
|
|
`pixels`. Values from the instruction or `params`, and values the agent
|
|
composed itself, replay. In a one-time-code field, only a whole code the
|
|
instruction or `params` spell out replays.
|
|
|
|
## What a recording holds
|
|
|
|
Each entry is a JSON file under `.e2e/cache/`, named by a hash of its key.
|
|
It holds actions, target descriptions, typed text, a step summary, and the
|
|
end checks: the route and up to eight controls that appeared and eight that
|
|
went away. Entries hold no prompts, conversations, screenshots, observed
|
|
trees, or secret values. A secret fill stores only the secret's name.
|
|
|
|
<Warning>
|
|
Other typed values are stored as typed. Review recordings before you share
|
|
them. See [cache trust](/reference/security#cache-trust).
|
|
</Warning>
|
|
|
|
## Modes
|
|
|
|
| Mode | Replays | Records | Default |
|
|
| --- | --- | --- | --- |
|
|
| `'read-write'` | yes | yes | locally |
|
|
| `'read-only'` | yes | no | in CI when `cache` is unset |
|
|
| `'off'` | no | no | `--no-cache` for one run |
|
|
|
|
```ts title="e2e.config.ts fragment"
|
|
cache: 'off',
|
|
// or
|
|
cache: { mode: 'read-write', dir: '.e2e/cache' },
|
|
```
|
|
|
|
`--no-cache` overrides the config. In `read-only` mode, stale entries stay
|
|
until a `read-write` run replaces them. A custom `cache.store` replaces the
|
|
file store and doesn't default to `read-only` in CI. `e2e explore` and
|
|
`e2e mcp` sessions don't use the cache. A test a session runs
|
|
([`open_session {test}`](/reference/mcp#running-a-test)) uses it as a run's
|
|
first attempt does.
|
|
|
|
## Share the cache across checkouts
|
|
|
|
Point every checkout of a project at one directory with `E2E_CACHE_DIR`, or
|
|
`--cache-dir` for one run. Either replaces `cache.dir`, resolves from the
|
|
project root (`~` or a leading `~/` is your home directory), and leaves the mode
|
|
alone. With a custom `cache.store`, `--cache-dir` is an error and
|
|
`E2E_CACHE_DIR` is ignored. Parallel runs can share the directory: each write
|
|
replaces its file atomically. Entries are keyed by project, so give every
|
|
checkout the same `projectId` (or `package.json` `name`); without either,
|
|
each checkout's path becomes its identity and nothing replays across them.
|
|
|
|
- Checkouts share entries: a failing run in any of them evicts the entry, and
|
|
checkouts with the same app identity but different app code overwrite each
|
|
other's entries (the last write wins).
|
|
- While the override is set, a cache committed under `cache.dir` isn't read.
|
|
- Use one directory per project. Two projects in one directory can fail with
|
|
a false `REPLAY_STALE` under `cache.strict`.
|
|
|
|
```bash
|
|
E2E_CACHE_DIR=~/.cache/my-app/e2e npx e2e run
|
|
```
|
|
|
|
## Commit the replay cache
|
|
|
|
<span id="commit-your-traces" />
|
|
|
|
`e2e init` adds `.e2e/cache/` to `.gitignore`, so each machine records its
|
|
own and CI starts with none. To share replays, remove that line and commit
|
|
the directory. CI stays `read-only` unless you set `cache: 'read-write'`.
|
|
Treat committed entries as test data, and review them in the pull request
|
|
that recorded them again.
|
|
|
|
Local runs leave a committed directory clean unless something changed. A
|
|
replayed step never rewrites its entry, and a live recording with the same
|
|
actions, paths, and anchors isn't written. So an entry for a UI that drifts
|
|
is repaired only when a control can no longer be found. A file changes when:
|
|
|
|
- a step records for the first time, or after a hand-off,
|
|
- a step fails after a replay, ends in an `end-mismatch` the agent had to
|
|
repair, passes with nothing to record, or isn't confirmed by the end of the attempt, which deletes the
|
|
entry,
|
|
- an `e2e` upgrade changes how entries are matched, which gives every entry
|
|
a new key, so the next run records them again.
|
|
|
|
## Fail on a stale recording
|
|
|
|
By default, a committed recording that stops replaying fails nothing: the
|
|
agent takes the step over, and every CI run pays for model calls until
|
|
someone records it again. With `--strict-cache` (or
|
|
`cache: { strict: true }`), the step fails with `REPLAY_STALE` and makes no
|
|
model call:
|
|
|
|
<CodeGroup>
|
|
```bash npm
|
|
CI=1 npx e2e run --strict-cache
|
|
```
|
|
|
|
```bash pnpm
|
|
CI=1 pnpm exec e2e run --strict-cache
|
|
```
|
|
|
|
```bash bun
|
|
CI=1 bunx e2e run --strict-cache
|
|
```
|
|
</CodeGroup>
|
|
|
|
`REPLAY_STALE` exits 2 and is not retried.
|
|
|
|
| Fails under strict | Still runs live |
|
|
| --- | --- |
|
|
| `invalid-entry`, `wrong-context`, `target-not-found`, `target-ambiguous`, `viewport-changed`, `action-failed`, `action-uncertain`, `end-mismatch`: the app or the entry changed | A step never recorded (a new or edited instruction or params) |
|
|
| `no-entry`, when the cache directory holds a recording of the same step under another key | A `gap` or a `truncated` recording |
|
|
| | A step whose store read fails |
|
|
| | A step whose test's `testId` in `report.json`, target name, or agent name is over 300 characters, when an earlier `e2e` version recorded it under another key and no `read-write` run of this version has replayed it since |
|
|
|
|
A recording under another key comes from an `e2e` upgrade that changed the
|
|
keys, or a change to the project id, the app's identity, or the agent's
|
|
context. The message names that file and, for a recording made by a
|
|
current version, what changed. Delete the file when nothing replays it. This check doesn't cover a
|
|
custom `cache.store`.
|
|
|
|
A strict run never writes or deletes an entry, even in `read-write` mode, so
|
|
a new step records only on a run without strict. Prefer `--strict-cache` in
|
|
CI over `cache.strict` in the config. To fix a stale entry:
|
|
|
|
1. Run the test in `read-write` mode without the flag. Turn `cache.strict`
|
|
off if the config sets it.
|
|
2. Commit the changed entry. A custom `cache.store` gets the new entry from
|
|
that run instead.
|
|
|
|
## Inspect the cache
|
|
|
|
<CodeGroup>
|
|
```bash npm
|
|
npx e2e cache ls # one row per entry: test, target, instruction digest, age, actions
|
|
npx e2e cache stats # entry count and size
|
|
npx e2e cache clear # delete every entry
|
|
```
|
|
|
|
```bash pnpm
|
|
pnpm exec e2e cache ls # one row per entry: test, target, instruction digest, age, actions
|
|
pnpm exec e2e cache stats # entry count and size
|
|
pnpm exec e2e cache clear # delete every entry
|
|
```
|
|
|
|
```bash bun
|
|
bunx e2e cache ls # one row per entry: test, target, instruction digest, age, actions
|
|
bunx e2e cache stats # entry count and size
|
|
bunx e2e cache clear # delete every entry
|
|
```
|
|
</CodeGroup>
|
|
|
|
<Note>
|
|
The `cached` percentage in AI usage is the provider's prompt cache. It
|
|
lowers the cost of a model call. The replay cache skips the call.
|
|
</Note>
|
|
|
|
## Next
|
|
|
|
<CardGroup cols={2}>
|
|
<Card title="Config reference" icon="sliders" href="/reference/config#cache">
|
|
`mode`, `store`, `dir`.
|
|
</Card>
|
|
<Card title="CLI reference" icon="terminal" href="/reference/cli#e2e-cache">
|
|
`e2e cache ls`, `stats`, `clear`.
|
|
</Card>
|
|
</CardGroup>
|