Files
e2e/docs/cache.mdx
Samanyu DasandClaude Opus 5.5 3761bb9b94 fix(cache): tell long-id steps apart under --strict-cache with an identity digest (#1810)
* fix(cache): tell long-id steps apart by an identity digest under --strict-cache

An entry keeps the test id, target id, and agent name cut to 300
characters. Two steps whose names agree in their first 299 characters
stored the same provenance, so the strict rekey check took a step that
was never recorded for one whose key changed and failed it with
REPLAY_STALE.

When a name is longer than an entry keeps, the entry now also stores
identityDigest, the SHA-256 of the whole redacted names, and stepKey
compares it. A confirmed replay writes the digest into a kept entry
that predates it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(cache): key by the identity digest a name that ends like a cut

bound() cuts a name to 299 characters and an ellipsis (298 after a
surrogate pair). A whole target or agent name of that shape stored no
digest, so it matched an entry recorded before the digest for another
step whose name was cut, and strict failed it with REPLAY_STALE.
boundIsAmbiguous() names both cases, and recordedProvenance writes the
digest for either.

Also pins that the digest is taken over the redacted names.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(cache): measure the long-id row by the test id, and say where a name ends in an ellipsis

The test id percent-encodes the titles, so the limit is reached well
before 300 characters of title text. The row names the test id again.
The digest JSDoc names the createAgent cache key as the source of a
name that ends like a cut, and the provenance tests leave the
settlement block.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(cache): refuse an identity digest that is an array or a number

RegExp#test stringifies its argument, so ['<64 hex>'] passes the hex
pattern; the typeof check is what refuses it, and now a test pins it.
Also rewords the stepKey JSDoc.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(cache): pin the no-rewrite side of the identity digest backfill

- Add a settleStagedTraces test: a confirmed kept entry that already
  keeps the step's identity digest gets no store write.
- The StagedTrace JSDoc names the identity digest beside the
  occurrence fields.
- The docs row no longer says "test id", which this page uses for the
  app's element test ids, and names the 299 or 300 characters ending
  in an ellipsis case.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(cache): pin that ambiguity is measured on the redacted names

- Test that a name a secret makes longer than an entry keeps gets an
  identity digest.
- The docs row drops the ellipsis case: target and agent names are
  ASCII only, and runner test ids are percent-encoded, so neither can
  end in one. It says the test's file path and percent-encoded titles,
  and that a confirmed read-write replay ends the exception.
- The identityDigest JSDoc names the secret-value case.
- One isOptionalDigest check serves paramsDigest and identityDigest.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(cache): pin that the backfill replaces or drops a stale identity digest

- A confirmed kept entry whose names redact differently now gets the
  step's new digest, and one whose names no longer need a digest drops
  it.
- Shorten the docs row to the maintainer's wording.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(cache): say the backfill rewrites differing provenance, and translate the row

- The StagedTrace, completeProvenance, and settleStagedTraces JSDoc say
  an entry is rewritten when its stored provenance differs from the
  step's, which now includes replacing or dropping a digest.
- Test that an invalid paramsDigest is refused too, in a describe for
  reading entries.
- Add the long-name row to the ja and zh cache tables.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(cache): pin that target and agent names are stored and digested redacted

- Test both fields with a registered secret: the stored name holds no
  secret value, and the digest is the same for two values.
- The StagedTrace and settleStagedTraces JSDoc point at
  completeProvenance instead of restating when it rewrites an entry.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(cache): name each identity digest case completeProvenance rewrites

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(cache): measure the long-name row by the testId in report.json

- A title is percent-encoded into the test id, so a 34-character
  Japanese title already passes 300 characters. Name the `testId`
  field users can see instead of "a very long title".
- Only a read-write run of this version writes the digest in, so say
  so.
- Drop a test assertion on a constant, and a stray blank line.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 04:28:59 +00:00

451 lines
20 KiB
Plaintext

---
title: Caching
description: Replay verified agent actions without model calls, and let the agent take over when the app changes.
"og:image": "https://e2e.com/docs/images/og/cache.png?v=d035986b13"
"twitter:image": "https://e2e.com/docs/images/og/cache.png?v=d035986b13"
---
The replay cache records the actions of an `agent.act()` step after a later
check verifies the result. On the next run, the runner replays those
actions without a model call. When a control or the expected end state no
longer matches, the agent takes over from the current screen. See
[Self-healing tests](/self-healing).
`agent.assert`, `agent.waitFor`, and `agent.extract` always run live. The
run summary counts the outcomes:
```text
AI 4.1k tokens · 2 model calls · openai/gpt-6-luna-fast
Cache 4 replayed · 1 handed off · 1 missed · 1 not saved
```
`not saved` counts the steps of passing and flaky tests whose
[`step.cache.write`](#when-a-recording-is-written) is `unconfirmed`,
`no-change`, or `not-written` in the attempt that passed.
## When a recording is written
End the test with a check of its outcome. In `read-write` mode, the runner
saves the recording of each `act` at the end of the attempt when a later
verification step passed. One check at the end confirms every `act` before
it.
```ts title="tests/cart.e2e.ts"
import { expect, test } from 'e2e';
test('adds an item', async ({ app, agent, screen }) => {
await app.open('/cart');
await agent.act('add one item to the cart');
await expect(screen.getByRole('status')).toHaveText('1 item');
});
```
| Counts as verification | Does not count |
| --- | --- |
| `expect(locator).toHaveText(...)` and other locator matchers | `expect(value).toBe(...)` on a plain value |
| `expect(browser).toHaveURL(...)` and other engine assertions | `expect.poll(...)` |
| `locator.waitFor()`, `browser.waitForURL()` | `locator.textContent()` and other reads |
| `agent.assert(...)`, `agent.waitFor(...)` | `agent.extract(...)`, another `act`, the attempt passing |
`step.cache.write`, in the report only, says what happened to the recording
when the attempt ended:
| `step.cache.write` | Meaning |
| --- | --- |
| `saved` | Recorded for the next run. |
| `kept` | The entry stays as it is. |
| `unconfirmed` | Not saved: no check passed after the step. End the test with a locator or engine assertion or `agent.assert`, or wait for the state with `locator.waitFor()` or `agent.waitFor`. |
| `evicted` | The entry was deleted: its replay ended in a failure or a repair, or no check confirmed it before the attempt failed. |
| `no-change` | Nothing recorded: no control appeared, went away, or changed state, and the route stayed the same. That includes a step whose only change is inside a [frame from another site](/browser#reach-into-an-iframe). Such a step runs live on every run. |
| `not-written` | Not saved: the store rejected the write, or refused the recording, for example one over the file store's size limit. |
A step that took no action records nothing and reports no `write`.
A failing attempt still saves the steps verified before the failure. It
deletes unverified entries and the replays involved in the failure. To keep
setup goals recorded while you fix a failing assertion, end the setup with
a wait for its state, such as `locator.waitFor()`. A custom store needs a `delete` method for this. Cancelling
a run leaves the cache unchanged.
<Accordion title="Failures where no model answered">
An attempt whose only failures are ones where no model answered evicts
nothing, since these say nothing about the app:
- `MODEL_UNAVAILABLE`: no model is configured, or GitHub Copilot doesn't
serve it to the account.
- `MODEL_PROVIDER_FAILED`: the provider returned an error, such as a
missing or rejected key, a rate limit, or a 5xx, or a request got no
response within 120 seconds.
So a `read-write` run without model access keeps its recordings, including
a step that replayed before an `agent.assert` that could not reach its
model.
These count as a failure of the flow: an unusable model answer
(`MODEL_OUTPUT_INVALID`, `CONTEXT_OVERFLOW`), a refusal (`MODEL_REFUSED`),
`STEP_TIMEOUT` (even with a
request still open), and any other failure in the same attempt, such as an
`afterEach` assertion.
</Accordion>
## What must match
A cache entry belongs to one test, target, instruction, set of params, and
agent. Each of these changes causes a miss:
- renaming the test, its `describe` group, or the target, or moving the
test file,
- changing the instruction or an ordinary param,
- running with another configured agent (`agent: ['buyer', 'admin']`
records one entry each), or changing the agent's `context` or the test's
`agentContext` (a secret in it enters the key as its name only),
- switching to another engine, or changing the project id or the app's
identity.
Changing the model doesn't cause a miss, and neither does upgrading the
engine: a node the new version resolves differently misses at replay.
Use `--no-cache` to evaluate a new model. Replays expose
`step.cache.recordedBy` when the entry has provenance; older or unattributed
entries omit it.
A timestamp or a fresh email address would miss on every run. Wrap such
values in `unique()`, and the runner puts in the current value at replay:
```ts
import { unique } from 'e2e';
const name = `E2E ${Date.now()} Company`;
await agent.act('create a company named {name}', { params: { name: unique(name) } });
await expect(screen.getByRole('heading', { name })).toBeVisible();
```
Use `unique()` for data the flow enters or checks, and ordinary params for
choices that change the flow, such as a plan name. The substitution also
covers the value as a URL spells it (`%20`, `+`) and a multi-line value as
the screen shows it, on one line. When a `unique()` value
equals or sits inside another param's value, the step isn't recorded:
`step.cache.notRecorded` reads `param-collision`, and the next run records
it.
For preview URLs that change on each deploy, set
[`app.identity`](/reference/config#the-app-under-test) to keep one cache
identity. On devices, replay uses the
[screen title](/reference/mobile#replay-cache-on-a-device) as the path.
## Rerun a test while you fix it
While you fix a failing test, keep the cache on in `read-write` mode, the
local default, and outside [strict mode](#fail-on-a-stale-recording),
which never writes. With `CI` set, an unset mode is `read-only`. A failed
run keeps the recordings a check
[confirmed](#when-a-recording-is-written) before the failure. The next run
replays those goals without model calls, as long as each replay still
matches the screen.
1. Run only that test, by the line its `test(` opens on:
`npx e2e run tests/cart.e2e.ts:3`.
2. Don't rename the test or reword a goal that passes. Each
[causes a miss](#what-must-match): a rename for every goal in the test,
a reworded goal for itself.
3. When retries are on (`retries` in the config, or `CI` set), add
`--retries 0`. A `retries` option on the test or its `describe` wins
over the flag. A retry doesn't replay, so it runs every goal live.
4. When the test passes, run it one time with `--no-cache`. Every goal
runs live, which shows the agent still reaches it without a recording.
## How a replay works
1. Check the starting screen: it must be the same route.
2. Find each recorded control by role, name, test id, and surrounding
context, and repeat its action. The runner waits for missing controls and
stops when a match is ambiguous.
3. Check the final route and what the original step changed: the controls
that appeared, with their checked, selected, expanded, or pressed state,
and the ones that went away.
4. Finish without a model call, or let the agent continue with a record of
what ran and why replay stopped.
Typed values replay as recorded. A secret fill resolves the current value
by name and goes through authorization again.
A replay finishes as soon as the recorded changes show, which can be before
content below them has loaded. To check content that loads after the step,
use `agent.waitFor`, which waits for it, rather than `agent.assert`, which
judges the screen once.
<Accordion title="What counts as the same route">
A route is the origin, the path, and the query. The record id and the
fragment don't count. Ids, tokens, timestamps, and cache busters in the
path or the query read as an id placeholder, so `/orders/42?t=1727780000`
and `/orders/7?t=1727780999` are one route.
Every other query value is part of the route: `?mode=safe` and
`?mode=unsafe` differ, and so do `/companies` and `/companies?tab=notes`, or
a new tracking parameter. `/products/summer-sneaker` and
`/products/winter-boot` are different routes too.
A location on the app's own origin is compared by its path, so a recording
follows the app to a preview deployment that shares its `app.identity`. A
page on any other origin is another route.
</Accordion>
<Accordion title="How the end check reads the screen">
- At least one recorded change must happen during the replay. An outcome
already on screen before the actions proves nothing. When the step's last
action moved to another route, that move is the evidence.
- An alert that the replay raised and the recording never saw stops the
replay. The web engine does not observe Next.js App Router's route
announcer, so its title echo never counts.
- Text that differs on every run, such as a date, a duration, or an id,
is skipped while a stable change exists on either side of the step's delta,
whether something appeared or disappeared. With nothing else, it is kept,
and a changed value hands the replay off. In an alert, it reads as a placeholder.
A node known only by a test id the app generates (`node-<uuid>`) is
skipped the same way. When kept, a node that appeared may carry any id
with that pattern; one that went away is still checked by its exact id.
- A count that names what it counts, such as `3 records imported` or
`42件`, must read the same on replay when the step made it appear on the
screen it acted on. A count that only moved (`41 items` to `42 items`),
or one on a screen the step navigated to, is skipped only when its own
side of the delta has a stable change. Otherwise the count stays, even
when the opposite side has a stable change. A bare number such as a badge
follows the text rule above.
- A control with no name is found by its place among the unnamed controls
of its kind inside the same named row, list item, or group. A control
known only by a generated test id is found by the id pattern the same
way, or when it was the only one with that pattern.
</Accordion>
## Why a step was not replayed
Each `act` step reports `step.cache`, absent when caching is off for the
step. `agent.act()` returns it as `result.cache`, without `write`, and the
[trace page](/debugging#traces) shows it under the step.
| Run summary | `step.cache.mode` | Meaning |
| --- | --- | --- |
| `replayed` | `self-finalized` | The recording completed without a model call. |
| `handed off` | `agent-concluded` | Replay started, then the agent took over. |
| `missed` | `missed` | The agent ran the step from the beginning. |
Both carry `step.cache.reason`, and `step.cache.detail` says where the
replay stopped and why (`at action 2 of 3, tap button "Apply": ...`). After
an `end-mismatch` it names the recorded nodes that did not show, the ones
that did not go away, and any alert the recording never saw, or says the
outcome already showed before the replay acted.
`step.cache.entry` is the entry's key and its file name (`<entry>.json`) in
the file store.
| Reason | What happened | What to do |
| --- | --- | --- |
| `no-entry` | No recording matches this step and its params. | The next verified pass records it, unless the step's `step.cache.write` is `no-change`. |
| `retry` | Retries never replay, except under `--strict-cache` or `cache.strict`. | Nothing. |
| `invalid-entry` | The file is unreadable, over 1 MiB, or from another schema version. | `npx e2e cache clear`. |
| `truncated` | The recording hit a limit, see [below](#limits-and-gaps). | Split the step, or pass the secret as a `Secret`. |
| `wrong-context` | The app was on another route than when the step was recorded. See [How a replay works](#how-a-replay-works). | Open the same screen before the step. |
| `target-not-found` | A recorded control did not appear within 15 seconds, or it has no name and no named container to tell it from its twins. | The UI changed: the next passing run records it again. For an unnamed control, give it a label or put it in a named row or group. |
| `target-ambiguous` | The number of matching controls changed. | Give repeated controls distinct names or use unique test data. |
| `gap` | An action can't be replayed, see [below](#limits-and-gaps). | The agent does this part live. Put stable values in `params`. |
| `action-failed` | The app rejected a replayed action. | Check the app. |
| `action-uncertain` | A replayed action failed, and it's unknown whether it took effect. The agent is told to check before it acts again. | Check the app. |
| `viewport-changed` | A bare-point tap, hover, or drag was recorded on a viewport of another size. | Keep the viewport fixed. |
| `end-mismatch` | Every action ran, but the [end check](#how-a-replay-works) failed. | Leftover data from an earlier run, a banner that comes and goes, or a regression. Unique test data and cleanup in `afterEach` help. |
To rule the cache out of a failure, run one time without it:
<CodeGroup>
```bash npm
npx e2e run --no-cache
```
```bash pnpm
pnpm exec e2e run --no-cache
```
```bash bun
bunx e2e run --no-cache
```
</CodeGroup>
### Limits and gaps
A recording is `truncated` when the step:
- took more than 50 actions, which is possible only with `maxSteps` above 50,
- typed a value or URL over 4,096 characters, or one that holds a
registered secret,
- acted on a control with no usable description.
A `gap` is an action the cache can't repeat: a mutating
[project tool](/tools), or a value the agent read off the screen, typed
after it saw a screenshot, or worked out from the date. `step.cache.derived`
names the rule: `whole-node` (a control's whole name or text),
`minted-token` (a code or id the app showed as its own word), `date`, or
`pixels`. Values from the instruction or `params`, and values the agent
composed itself, replay.
## What a recording holds
Each entry is a JSON file under `.e2e/cache/`, named by a hash of its key.
It holds actions, target descriptions, typed text, a step summary, and the
end checks: the route and up to eight controls that appeared and eight that
went away. Entries hold no prompts, conversations, screenshots, observed
trees, or secret values. A secret fill stores only the secret's name.
<Warning>
Other typed values are stored as typed. Review recordings before you share
them. See [cache trust](/reference/security#cache-trust).
</Warning>
## Modes
| Mode | Replays | Records | Default |
| --- | --- | --- | --- |
| `'read-write'` | yes | yes | locally |
| `'read-only'` | yes | no | in CI when `cache` is unset |
| `'off'` | no | no | `--no-cache` for one run |
```ts title="e2e.config.ts fragment"
cache: 'off',
// or
cache: { mode: 'read-write', dir: '.e2e/cache' },
```
`--no-cache` overrides the config. In `read-only` mode, stale entries stay
until a `read-write` run replaces them. A custom `cache.store` replaces the
file store and doesn't default to `read-only` in CI. `e2e explore` and
`e2e mcp` sessions don't use the cache. A test a session runs
([`open_session {test}`](/reference/mcp#running-a-test)) uses it as a run's
first attempt does.
## Share the cache across checkouts
Point every checkout of a project at one directory with `E2E_CACHE_DIR`, or
`--cache-dir` for one run. Either replaces `cache.dir`, resolves from the
project root (`~` or a leading `~/` is your home directory), and leaves the mode
alone. With a custom `cache.store`, `--cache-dir` is an error and
`E2E_CACHE_DIR` is ignored. Parallel runs can share the directory: each write
replaces its file atomically. Entries are keyed by project, so give every
checkout the same `projectId` (or `package.json` `name`); without either,
each checkout's path becomes its identity and nothing replays across them.
- Checkouts share entries: a failing run in any of them evicts the entry, and
checkouts with the same app identity but different app code overwrite each
other's entries (the last write wins).
- While the override is set, a cache committed under `cache.dir` isn't read.
- Use one directory per project. Two projects in one directory can fail with
a false `REPLAY_STALE` under `cache.strict`.
```bash
E2E_CACHE_DIR=~/.cache/my-app/e2e npx e2e run
```
## Commit the replay cache
<span id="commit-your-traces" />
`e2e init` adds `.e2e/cache/` to `.gitignore`, so each machine records its
own and CI starts with none. To share replays, remove that line and commit
the directory. CI stays `read-only` unless you set `cache: 'read-write'`.
Treat committed entries as test data, and review them in the pull request
that recorded them again.
Local runs leave a committed directory clean unless something changed. A
replayed step never rewrites its entry, and a live recording with the same
actions, paths, and anchors isn't written. So an entry for a UI that drifts
is repaired only when a control can no longer be found. A file changes when:
- a step records for the first time, or after a hand-off,
- a step fails after a replay, ends in an `end-mismatch` the agent had to
repair, passes with nothing to record, or isn't confirmed by the end of the attempt, which deletes the
entry,
- an `e2e` upgrade changes how entries are matched, which gives every entry
a new key, so the next run records them again.
## Fail on a stale recording
By default, a committed recording that stops replaying fails nothing: the
agent takes the step over, and every CI run pays for model calls until
someone records it again. With `--strict-cache` (or
`cache: { strict: true }`), the step fails with `REPLAY_STALE` and makes no
model call:
<CodeGroup>
```bash npm
CI=1 npx e2e run --strict-cache
```
```bash pnpm
CI=1 pnpm exec e2e run --strict-cache
```
```bash bun
CI=1 bunx e2e run --strict-cache
```
</CodeGroup>
`REPLAY_STALE` exits 2 and is not retried.
| Fails under strict | Still runs live |
| --- | --- |
| `invalid-entry`, `wrong-context`, `target-not-found`, `target-ambiguous`, `viewport-changed`, `action-failed`, `action-uncertain`, `end-mismatch`: the app or the entry changed | A step never recorded (a new or edited instruction or params) |
| `no-entry`, when the cache directory holds a recording of the same step under another key | A `gap` or a `truncated` recording |
| | A step whose store read fails |
| | A step whose test's `testId` in `report.json`, target name, or agent name is over 300 characters, when an earlier `e2e` version recorded it under another key and no `read-write` run of this version has replayed it since |
A recording under another key comes from an `e2e` upgrade that changed the
keys, or a change to the project id, the app's identity, or the agent's
context. The message names that file and, for a recording made by a
current version, what changed. Delete the file when nothing replays it. This check doesn't cover a
custom `cache.store`.
A strict run never writes or deletes an entry, even in `read-write` mode, so
a new step records only on a run without strict. Prefer `--strict-cache` in
CI over `cache.strict` in the config. To fix a stale entry:
1. Run the test in `read-write` mode without the flag. Turn `cache.strict`
off if the config sets it.
2. Commit the changed entry. A custom `cache.store` gets the new entry from
that run instead.
## Inspect the cache
<CodeGroup>
```bash npm
npx e2e cache ls # one row per entry: test, target, instruction digest, age, actions
npx e2e cache stats # entry count and size
npx e2e cache clear # delete every entry
```
```bash pnpm
pnpm exec e2e cache ls # one row per entry: test, target, instruction digest, age, actions
pnpm exec e2e cache stats # entry count and size
pnpm exec e2e cache clear # delete every entry
```
```bash bun
bunx e2e cache ls # one row per entry: test, target, instruction digest, age, actions
bunx e2e cache stats # entry count and size
bunx e2e cache clear # delete every entry
```
</CodeGroup>
<Note>
The `cached` percentage in AI usage is the provider's prompt cache. It
lowers the cost of a model call. The replay cache skips the call.
</Note>
## Next
<CardGroup cols={2}>
<Card title="Config reference" icon="sliders" href="/reference/config#cache">
`mode`, `store`, `dir`.
</Card>
<Card title="CLI reference" icon="terminal" href="/reference/cli#e2e-cache">
`e2e cache ls`, `stats`, `clear`.
</Card>
</CardGroup>