Files
e2e/docs/cache.mdx
T
Oskar Kwaśniewski b5dc0f87bb fix(cache): --strict-cache fails a step whose key changed under its recording (#800)
* fix(cache): --strict-cache fails a step whose key changed under its recording

A cache key change (REPLAY_POLICY_VERSION, a new key field, an engine minor, the agent's context) turned every committed entry into a no-entry miss, which strict ran live. Strict now lists the file store once per run and fails a no-entry step with REPLAY_STALE when an entry recorded for the same step sits under another key. Entries now record the step's params digest, occurrence, and agent so a repeat or another agent is not mistaken for it.

* fix(cache): never take an entry the attempt claimed for a rekeyed step

An entry from before the occurrence fields matches by test, target, and instruction; an earlier occurrence's entry is now excluded, so an unrecorded repeat runs live. The message no longer says to delete the old entry, which may still serve another app identity.

* fix(cache): match a rekeyed recording only by its whole step

An entry without the params, occurrence, and agent could be another call of the same instruction, so strict now ignores it instead of guessing. A confirmed whole replay in read-write mode writes the missing provenance in, since it proved which step the entry belongs to.

* test(web-benchmark): complete the provenance of 25 agent recordings

Written by a local read-write replay (no model calls); only recordedFor and createdAt change. The 4 gap entries need a live run with a key.

* test(cache): assert the provenance backfill comes from a replay
2026-10-03 22:48:11 +02:00

353 lines
16 KiB
Plaintext

---
title: Caching agent steps
description: Replay verified agent actions without model calls, and let the agent take over when the app changes.
---
The replay cache records the actions in an `agent.act()` step after a later
check verifies the result. On the next run, the runner replays those actions
without calling the model. If a control or the expected end state no longer
matches, the agent takes over from the current screen.
`agent.assert`, `agent.waitFor`, and `agent.extract` always run live.
The run summary reports the outcomes:
```text
AI 4.1k tokens · 2 model calls · anthropic/claude-sonnet-4.5
Cache 4 replayed · 1 handed off · 1 missed
```
| Run summary | `step.cache.mode` | Meaning |
| --- | --- | --- |
| `replayed` | `self-finalized` | The recording completed without a model call. |
| `handed off` | `agent-concluded` | Replay started, then the agent took over. |
| `missed` | `missed` | The agent ran the step from the beginning. |
Misses and hand-offs also include `step.cache.reason`. `agent.act()` returns
the same cache object as `result.cache`. The cache object is absent when
caching is off for the step.
To rule the cache out of a failure, run once without it:
<CodeGroup>
```bash npm
npx e2e run --no-cache
```
```bash pnpm
pnpm exec e2e run --no-cache
```
```bash bun
bunx e2e run --no-cache
```
</CodeGroup>
## How a replay works
1. Check the starting screen: the same route, whatever the record id or the
fragment. Retries always run live.
2. Find each recorded control and repeat its action. The runner matches
controls by their role, name, test id, and surrounding context. It waits
for missing controls and stops if a match is ambiguous. A control with no
name is found by its place among the unnamed controls of its kind inside
the same named row, list item, or group; with no such container to tell
it from its twins, the agent takes the step.
3. Check the final route and what the original step changed: the controls
that appeared, with their checked, selected, expanded, or pressed state,
and the ones that went away. At least one of those changes must happen
during the replay, counted from the first screen the replay saw on the
route it ended on, so an outcome already on screen before the actions
that should produce it proves nothing; a step whose last action moved to
another route has that move as its evidence. An alert the replay raised
that the recording never saw stops it too. Text that reads differently on
every run, a date, a time, a duration, an id, is not used, except in an
alert, where it is read as a placeholder.
4. Finish without a model call, or let the agent continue with a record of
what ran and why replay stopped.
A route is the origin, the path, and the query. Ids, tokens, timestamps,
and cache busters in either the path or the query are read as an id
placeholder, so `/orders/42?t=1727780000` and `/orders/7?t=1727780999` are
one route. Every other query value is part of it: `?mode=safe` and
`?mode=unsafe`, or `/companies` and `/companies?tab=notes`, are different
screens, and so is a tracking parameter that was not there when the step
was recorded. Two routes that differ in one word, `/products/summer-sneaker`
and `/products/winter-boot`, are different screens too, even when they look
alike. A location on the app's own origin is compared by its path, so a
recording follows the app to a preview deployment that shares its
`app.identity`; a page on any other origin is another route.
A count that names what it counts, `3 records imported` or `Showing 1 to 9
of 9 results`, is part of the effect when the step made it appear on the
screen it acted on, and must read the same on replay. A count that was
already on screen and only moved, `41 items` to `42 items`, or one on a
screen the step navigated to, is not used, since leftover data from earlier
runs moves it again. A bare number such as a badge is never used.
Typed values replay as recorded, even when an earlier run left the same
value on screen. Secret fills resolve the current value by name and run
through authorization again.
## What must match
A cache entry belongs to a specific test, target, instruction, set of
params, and agent. Renaming the test or target, changing the instruction or
an ordinary param, running with another configured agent, changing the
agent's `context` or the test's `agentContext`, or upgrading the engine's
major or minor version causes a miss. A test that runs once per agent
(`agent: ['buyer', 'admin']`) records once per agent. Changing the model
does not. The agent context enters the key as a hash of its redacted text,
so a secret in it contributes only its name.
A timestamp or a fresh email address would cause a miss on every run. Wrap
values like these in `unique()` so the runner substitutes the current value
when replaying:
```ts
import { unique } from 'e2e';
const name = `E2E ${Date.now()} Company`;
await agent.act('create a company named {name}', { params: { name: unique(name) } });
await expect(screen.getByRole('heading', { name })).toBeVisible();
```
Use `unique()` for data the flow enters or checks. Keep choices that change
the flow, such as a plan name, as ordinary params. The substitution covers
the value as a URL spells it too, `%20` in a path or `+` in a search query.
When a `unique()` value equals another param's value or appears inside one,
the step passes but is not recorded, and `step.cache.notRecorded` reads
`param-collision`. The next run, with a fresh value, records it.
For preview URLs that change on each deploy, set
[`app.identity`](/reference/config#the-app-under-test) to keep the same cache
identity. On devices, replay uses the
[screen title](/reference/mobile#replay-cache-on-a-device) as the path.
## When a recording is written
In `read-write` mode, the runner saves a recording at the end of the
attempt when a later verification step passed. A step that took no action
records nothing, and neither does one that changed nothing the replay could
check: no control appeared, went away, or changed state, and the route
stayed the same.
| Counts as verification | Does not count |
| --- | --- |
| `expect(locator).toHaveText(...)` and other locator matchers | `expect(value).toBe(...)` on a plain value |
| `expect(browser).toHaveURL(...)` and other engine assertions | `expect.poll(...)` |
| `locator.waitFor()`, `browser.waitForURL()` | `locator.textContent()` and other reads |
| `agent.assert(...)`, `agent.waitFor(...)` | `agent.extract(...)`, another `act`, the attempt passing |
Follow each `act` with a check of its outcome:
```ts title="tests/cart.e2e.ts"
import { expect, test } from 'e2e';
test('adds an item', async ({ app, agent, screen }) => {
await app.open('/cart');
await agent.act('add one item to the cart');
await expect(screen.getByRole('status')).toHaveText('1 item');
});
```
A failing attempt can still save steps verified before the failure. It
deletes unverified entries and replays implicated in the failure, including
an `end-mismatch`. Custom stores need a `delete` method for this cleanup.
Cancelling a run leaves the cache unchanged. An attempt whose only failures
are ones where no model answered leaves every entry it did not confirm
unchanged: `MODEL_UNAVAILABLE`
(no model configured) and `MODEL_PROVIDER_FAILED` (the provider returned an
error such as a missing or rejected key, a rate limit, or a 5xx, or a request
got no response within 120 seconds). Those say nothing about the app, so a
read-write run without model access keeps its recordings, including one a
step replayed before an `agent.assert` that could not reach its model, and
still saves steps verified before the failure. A model that answered with
something unusable (`MODEL_OUTPUT_INVALID`, `CONTEXT_OVERFLOW`) or a step that
ran out of time (`STEP_TIMEOUT`, even while a request was still open) counts
as a failure of the flow, and so does any other failure in the same attempt,
such as an `afterEach` assertion.
## Why a step was not replayed
`step.cache.reason` in the report says why:
| Reason | What happened | What to do |
| --- | --- | --- |
| `no-entry` | No recording matches this step and its params. | The next verified pass records it. |
| `retry` | Retries never replay. | Nothing. |
| `invalid-entry` | The file is unreadable, over 1 MiB, or from another schema version. | `npx e2e cache clear`. |
| `truncated` | The recording hit a limit, see [below](#limits-and-gaps). | Split the step, or pass the secret as a `Secret`. |
| `wrong-context` | The app was on another screen than when the step was recorded: a different origin, path, or query, not a different record id or fragment. | Open the same screen before the step. |
| `target-not-found` | A recorded control did not appear within 15 seconds, or it has no name and no named container to tell it from its twins. | The UI changed: the next passing run re-records. For an unnamed control, give it a label or put it in a named row or group. |
| `target-ambiguous` | The number of matching controls changed. | Give repeated controls distinct names or use unique test data. |
| `gap` | An action cannot be replayed, see [below](#limits-and-gaps). | The agent continues this part live. Put stable values in `params`. |
| `action-failed` | The app rejected a replayed action. | Check the app. |
| `action-uncertain` | A replayed action failed with an unknown commit state. The agent is told to check before acting again. | Check the app. |
| `viewport-changed` | A bare-point tap or hover was recorded on a viewport of another size. | Keep the viewport fixed. |
| `end-mismatch` | Every action ran, but the recorded route, a control that should have appeared or gone away, or a recorded state did not come back; the outcome was already on screen before the replay; or the replay raised an alert the recording never saw. | Leftover data from an earlier run, a banner that comes and goes, or a regression. |
A stop before any action ran is a miss. After replay starts, it is a
hand-off. Unique test data and cleanup in `afterEach` help avoid stale
state between runs.
### Limits and gaps
A recording is `truncated` when the step took more than 50 actions (only
possible with `maxSteps` raised above 50), typed a value or URL over 4,096
characters or one holding a registered secret, or acted on a control with no
usable description.
A `gap` is an action the cache cannot repeat: a mutating
[project tool](/tools), or a value the agent read off the screen, typed after
seeing a screenshot, or worked out from the date. `step.cache.derived` names
the rule: `whole-node` (a control's whole name or text), `minted-token` (a
code or id the app showed as its own word), `date`, or `pixels`. Values from
the instruction or `params`, or ones the agent composed itself, replay.
## What a recording holds
Each entry is a JSON file under `.e2e/cache/`, named by a hash of its key.
It contains actions, target descriptions, typed text, a step summary, and
checks for the final state. Those checks include the route, up to eight
controls that appeared during the step, alerts and status messages first,
and up to eight that went away.
Entries contain no prompts, conversations, screenshots, observed trees, or
secret values. A secret fill stores only the secret's name. Other typed
values are stored verbatim, so review recordings before sharing them; see
[cache trust](/security#cache-trust).
## Modes
| Mode | Replays | Records | Default |
| --- | --- | --- | --- |
| `'read-write'` | yes | yes | locally |
| `'read-only'` | yes | no | in CI when `cache` is unset |
| `'off'` | no | no | `--no-cache` for one run |
```ts title="e2e.config.ts fragment"
cache: 'off',
// or
cache: { mode: 'read-write', dir: '.e2e/cache' },
```
`--no-cache` overrides the config. To let CI update recordings, explicitly
set `cache: 'read-write'`. In `read-only` mode, stale entries remain until
a read-write run replaces them.
A custom `cache.store` replaces the file store and does not default to
read-only in CI. `e2e explore` and `e2e mcp` do not use the cache. See the
[config reference](/reference/config#cache).
## Commit the replay cache
<span id="commit-your-traces" />
`e2e init` adds `.e2e/cache/` to `.gitignore`, so every machine records its
own and CI starts with none. To share replays, remove that line and commit
the directory. CI stays `read-only` unless you set `cache: 'read-write'`.
Treat committed entries as test data and review them in the pull request
that re-recorded them.
Local runs leave a committed directory clean unless something changed. A
replayed step never rewrites its entry, and a live re-recording with the same
actions, paths, and anchors is not written: the summary, the measured end
wait, and the rule that flagged a typed run-time value differ between two
live runs of the same flow and do not count. Upgrading `e2e` to a version
that changes how entries are matched re-keys every entry, so the first run
after it records them again. A file changes when:
- a step records for the first time,
- the agent finishes a step after a hand-off, re-recording what it saw,
- a step fails after replaying, or ends in an `end-mismatch` the agent had
to repair, which deletes the entry,
- a step with an entry passes without leaving anything to record, which
deletes the entry,
- no verification step confirmed the entry by the end of the attempt, which
deletes it.
A replay that still finds every control refreshes nothing, so an entry for a
drifting UI is repaired only once a control can no longer be found.
## Fail on a stale recording
A committed recording that stops replaying is quiet by default: the agent
takes the step over, the test passes, and every CI run pays for the model
calls until someone re-records. `--strict-cache` (or `cache: { strict: true }`)
makes it loud. A step whose recording exists but no longer replays fails
with `REPLAY_STALE` and makes no model call:
<CodeGroup>
```bash npm
CI=1 npx e2e run --strict-cache
```
```bash pnpm
CI=1 pnpm exec e2e run --strict-cache
```
```bash bun
CI=1 bunx e2e run --strict-cache
```
</CodeGroup>
The reasons that fail are the ones that mean the app or the entry changed:
`invalid-entry`, `wrong-context`, `target-not-found`, `target-ambiguous`,
`viewport-changed`, `action-failed`, `action-uncertain`, and `end-mismatch`.
A `no-entry` miss fails too when the cache directory holds a recording of the
same step under another key, from before an `e2e` or engine upgrade or a
change to the project id, the app's identity, or the agent's context. The
message names that file; delete it once nothing replays it. A custom
`cache.store` is not checked this way. Entries recorded before this check
existed are covered once a `read-write` run replays or re-records them. A
step never recorded (a new or edited instruction or params), a retry, a
`gap`, and a `truncated` recording still run live. `REPLAY_STALE` exits 2 and is not retried, since a
retry never replays and would pass live. The entry stays where it is, even
in `read-write` mode, so every strict run fails on it until it is re-recorded:
run the test in `read-write` mode without the flag, and with `cache.strict`
off when the config set it, then commit the changed entry; a custom
`cache.store` receives the new entry from that run instead. The message
names the knob that turned strict on and where the entry lives: the cache
directory, or the configured `cache.store`. A store whose read fails is not
a stale recording; the step runs live.
## Inspect the cache
<CodeGroup>
```bash npm
npx e2e cache ls # one row per entry: test, target, instruction digest, age, actions
npx e2e cache stats # entry count and size
npx e2e cache clear # delete every entry
```
```bash pnpm
pnpm exec e2e cache ls # one row per entry: test, target, instruction digest, age, actions
pnpm exec e2e cache stats # entry count and size
pnpm exec e2e cache clear # delete every entry
```
```bash bun
bunx e2e cache ls # one row per entry: test, target, instruction digest, age, actions
bunx e2e cache stats # entry count and size
bunx e2e cache clear # delete every entry
```
</CodeGroup>
<Note>
The `cached` percentage in AI usage is the provider's prompt cache. It
reduces the cost of a model call. The replay cache described here skips the
call entirely.
</Note>
<CardGroup cols={2}>
<Card title="Config reference" icon="sliders" href="/reference/config#cache">
`mode`, `store`, `dir`.
</Card>
<Card title="CLI reference" icon="terminal" href="/reference/cli#e2e-cache">
`e2e cache ls`, `stats`, `clear`.
</Card>
</CardGroup>