mirror of
https://github.com/tester-army/e2e.git
synced 2026-11-22 07:55:36 +08:00
* fix(cache): --strict-cache fails a step whose key changed under its recording A cache key change (REPLAY_POLICY_VERSION, a new key field, an engine minor, the agent's context) turned every committed entry into a no-entry miss, which strict ran live. Strict now lists the file store once per run and fails a no-entry step with REPLAY_STALE when an entry recorded for the same step sits under another key. Entries now record the step's params digest, occurrence, and agent so a repeat or another agent is not mistaken for it. * fix(cache): never take an entry the attempt claimed for a rekeyed step An entry from before the occurrence fields matches by test, target, and instruction; an earlier occurrence's entry is now excluded, so an unrecorded repeat runs live. The message no longer says to delete the old entry, which may still serve another app identity. * fix(cache): match a rekeyed recording only by its whole step An entry without the params, occurrence, and agent could be another call of the same instruction, so strict now ignores it instead of guessing. A confirmed whole replay in read-write mode writes the missing provenance in, since it proved which step the entry belongs to. * test(web-benchmark): complete the provenance of 25 agent recordings Written by a local read-write replay (no model calls); only recordedFor and createdAt change. The 4 gap entries need a live run with a key. * test(cache): assert the provenance backfill comes from a replay
353 lines
16 KiB
Plaintext
353 lines
16 KiB
Plaintext
---
|
|
title: Caching agent steps
|
|
description: Replay verified agent actions without model calls, and let the agent take over when the app changes.
|
|
---
|
|
|
|
The replay cache records the actions in an `agent.act()` step after a later
|
|
check verifies the result. On the next run, the runner replays those actions
|
|
without calling the model. If a control or the expected end state no longer
|
|
matches, the agent takes over from the current screen.
|
|
|
|
`agent.assert`, `agent.waitFor`, and `agent.extract` always run live.
|
|
|
|
The run summary reports the outcomes:
|
|
|
|
```text
|
|
AI 4.1k tokens · 2 model calls · anthropic/claude-sonnet-4.5
|
|
Cache 4 replayed · 1 handed off · 1 missed
|
|
```
|
|
|
|
| Run summary | `step.cache.mode` | Meaning |
|
|
| --- | --- | --- |
|
|
| `replayed` | `self-finalized` | The recording completed without a model call. |
|
|
| `handed off` | `agent-concluded` | Replay started, then the agent took over. |
|
|
| `missed` | `missed` | The agent ran the step from the beginning. |
|
|
|
|
Misses and hand-offs also include `step.cache.reason`. `agent.act()` returns
|
|
the same cache object as `result.cache`. The cache object is absent when
|
|
caching is off for the step.
|
|
|
|
To rule the cache out of a failure, run once without it:
|
|
|
|
<CodeGroup>
|
|
```bash npm
|
|
npx e2e run --no-cache
|
|
```
|
|
|
|
```bash pnpm
|
|
pnpm exec e2e run --no-cache
|
|
```
|
|
|
|
```bash bun
|
|
bunx e2e run --no-cache
|
|
```
|
|
</CodeGroup>
|
|
|
|
## How a replay works
|
|
|
|
1. Check the starting screen: the same route, whatever the record id or the
|
|
fragment. Retries always run live.
|
|
2. Find each recorded control and repeat its action. The runner matches
|
|
controls by their role, name, test id, and surrounding context. It waits
|
|
for missing controls and stops if a match is ambiguous. A control with no
|
|
name is found by its place among the unnamed controls of its kind inside
|
|
the same named row, list item, or group; with no such container to tell
|
|
it from its twins, the agent takes the step.
|
|
3. Check the final route and what the original step changed: the controls
|
|
that appeared, with their checked, selected, expanded, or pressed state,
|
|
and the ones that went away. At least one of those changes must happen
|
|
during the replay, counted from the first screen the replay saw on the
|
|
route it ended on, so an outcome already on screen before the actions
|
|
that should produce it proves nothing; a step whose last action moved to
|
|
another route has that move as its evidence. An alert the replay raised
|
|
that the recording never saw stops it too. Text that reads differently on
|
|
every run, a date, a time, a duration, an id, is not used, except in an
|
|
alert, where it is read as a placeholder.
|
|
4. Finish without a model call, or let the agent continue with a record of
|
|
what ran and why replay stopped.
|
|
|
|
A route is the origin, the path, and the query. Ids, tokens, timestamps,
|
|
and cache busters in either the path or the query are read as an id
|
|
placeholder, so `/orders/42?t=1727780000` and `/orders/7?t=1727780999` are
|
|
one route. Every other query value is part of it: `?mode=safe` and
|
|
`?mode=unsafe`, or `/companies` and `/companies?tab=notes`, are different
|
|
screens, and so is a tracking parameter that was not there when the step
|
|
was recorded. Two routes that differ in one word, `/products/summer-sneaker`
|
|
and `/products/winter-boot`, are different screens too, even when they look
|
|
alike. A location on the app's own origin is compared by its path, so a
|
|
recording follows the app to a preview deployment that shares its
|
|
`app.identity`; a page on any other origin is another route.
|
|
|
|
A count that names what it counts, `3 records imported` or `Showing 1 to 9
|
|
of 9 results`, is part of the effect when the step made it appear on the
|
|
screen it acted on, and must read the same on replay. A count that was
|
|
already on screen and only moved, `41 items` to `42 items`, or one on a
|
|
screen the step navigated to, is not used, since leftover data from earlier
|
|
runs moves it again. A bare number such as a badge is never used.
|
|
|
|
Typed values replay as recorded, even when an earlier run left the same
|
|
value on screen. Secret fills resolve the current value by name and run
|
|
through authorization again.
|
|
|
|
## What must match
|
|
|
|
A cache entry belongs to a specific test, target, instruction, set of
|
|
params, and agent. Renaming the test or target, changing the instruction or
|
|
an ordinary param, running with another configured agent, changing the
|
|
agent's `context` or the test's `agentContext`, or upgrading the engine's
|
|
major or minor version causes a miss. A test that runs once per agent
|
|
(`agent: ['buyer', 'admin']`) records once per agent. Changing the model
|
|
does not. The agent context enters the key as a hash of its redacted text,
|
|
so a secret in it contributes only its name.
|
|
|
|
A timestamp or a fresh email address would cause a miss on every run. Wrap
|
|
values like these in `unique()` so the runner substitutes the current value
|
|
when replaying:
|
|
|
|
```ts
|
|
import { unique } from 'e2e';
|
|
|
|
const name = `E2E ${Date.now()} Company`;
|
|
await agent.act('create a company named {name}', { params: { name: unique(name) } });
|
|
await expect(screen.getByRole('heading', { name })).toBeVisible();
|
|
```
|
|
|
|
Use `unique()` for data the flow enters or checks. Keep choices that change
|
|
the flow, such as a plan name, as ordinary params. The substitution covers
|
|
the value as a URL spells it too, `%20` in a path or `+` in a search query.
|
|
|
|
When a `unique()` value equals another param's value or appears inside one,
|
|
the step passes but is not recorded, and `step.cache.notRecorded` reads
|
|
`param-collision`. The next run, with a fresh value, records it.
|
|
|
|
For preview URLs that change on each deploy, set
|
|
[`app.identity`](/reference/config#the-app-under-test) to keep the same cache
|
|
identity. On devices, replay uses the
|
|
[screen title](/reference/mobile#replay-cache-on-a-device) as the path.
|
|
|
|
## When a recording is written
|
|
|
|
In `read-write` mode, the runner saves a recording at the end of the
|
|
attempt when a later verification step passed. A step that took no action
|
|
records nothing, and neither does one that changed nothing the replay could
|
|
check: no control appeared, went away, or changed state, and the route
|
|
stayed the same.
|
|
|
|
| Counts as verification | Does not count |
|
|
| --- | --- |
|
|
| `expect(locator).toHaveText(...)` and other locator matchers | `expect(value).toBe(...)` on a plain value |
|
|
| `expect(browser).toHaveURL(...)` and other engine assertions | `expect.poll(...)` |
|
|
| `locator.waitFor()`, `browser.waitForURL()` | `locator.textContent()` and other reads |
|
|
| `agent.assert(...)`, `agent.waitFor(...)` | `agent.extract(...)`, another `act`, the attempt passing |
|
|
|
|
Follow each `act` with a check of its outcome:
|
|
|
|
```ts title="tests/cart.e2e.ts"
|
|
import { expect, test } from 'e2e';
|
|
|
|
test('adds an item', async ({ app, agent, screen }) => {
|
|
await app.open('/cart');
|
|
await agent.act('add one item to the cart');
|
|
await expect(screen.getByRole('status')).toHaveText('1 item');
|
|
});
|
|
```
|
|
|
|
A failing attempt can still save steps verified before the failure. It
|
|
deletes unverified entries and replays implicated in the failure, including
|
|
an `end-mismatch`. Custom stores need a `delete` method for this cleanup.
|
|
|
|
Cancelling a run leaves the cache unchanged. An attempt whose only failures
|
|
are ones where no model answered leaves every entry it did not confirm
|
|
unchanged: `MODEL_UNAVAILABLE`
|
|
(no model configured) and `MODEL_PROVIDER_FAILED` (the provider returned an
|
|
error such as a missing or rejected key, a rate limit, or a 5xx, or a request
|
|
got no response within 120 seconds). Those say nothing about the app, so a
|
|
read-write run without model access keeps its recordings, including one a
|
|
step replayed before an `agent.assert` that could not reach its model, and
|
|
still saves steps verified before the failure. A model that answered with
|
|
something unusable (`MODEL_OUTPUT_INVALID`, `CONTEXT_OVERFLOW`) or a step that
|
|
ran out of time (`STEP_TIMEOUT`, even while a request was still open) counts
|
|
as a failure of the flow, and so does any other failure in the same attempt,
|
|
such as an `afterEach` assertion.
|
|
|
|
## Why a step was not replayed
|
|
|
|
`step.cache.reason` in the report says why:
|
|
|
|
| Reason | What happened | What to do |
|
|
| --- | --- | --- |
|
|
| `no-entry` | No recording matches this step and its params. | The next verified pass records it. |
|
|
| `retry` | Retries never replay. | Nothing. |
|
|
| `invalid-entry` | The file is unreadable, over 1 MiB, or from another schema version. | `npx e2e cache clear`. |
|
|
| `truncated` | The recording hit a limit, see [below](#limits-and-gaps). | Split the step, or pass the secret as a `Secret`. |
|
|
| `wrong-context` | The app was on another screen than when the step was recorded: a different origin, path, or query, not a different record id or fragment. | Open the same screen before the step. |
|
|
| `target-not-found` | A recorded control did not appear within 15 seconds, or it has no name and no named container to tell it from its twins. | The UI changed: the next passing run re-records. For an unnamed control, give it a label or put it in a named row or group. |
|
|
| `target-ambiguous` | The number of matching controls changed. | Give repeated controls distinct names or use unique test data. |
|
|
| `gap` | An action cannot be replayed, see [below](#limits-and-gaps). | The agent continues this part live. Put stable values in `params`. |
|
|
| `action-failed` | The app rejected a replayed action. | Check the app. |
|
|
| `action-uncertain` | A replayed action failed with an unknown commit state. The agent is told to check before acting again. | Check the app. |
|
|
| `viewport-changed` | A bare-point tap or hover was recorded on a viewport of another size. | Keep the viewport fixed. |
|
|
| `end-mismatch` | Every action ran, but the recorded route, a control that should have appeared or gone away, or a recorded state did not come back; the outcome was already on screen before the replay; or the replay raised an alert the recording never saw. | Leftover data from an earlier run, a banner that comes and goes, or a regression. |
|
|
|
|
A stop before any action ran is a miss. After replay starts, it is a
|
|
hand-off. Unique test data and cleanup in `afterEach` help avoid stale
|
|
state between runs.
|
|
|
|
### Limits and gaps
|
|
|
|
A recording is `truncated` when the step took more than 50 actions (only
|
|
possible with `maxSteps` raised above 50), typed a value or URL over 4,096
|
|
characters or one holding a registered secret, or acted on a control with no
|
|
usable description.
|
|
|
|
A `gap` is an action the cache cannot repeat: a mutating
|
|
[project tool](/tools), or a value the agent read off the screen, typed after
|
|
seeing a screenshot, or worked out from the date. `step.cache.derived` names
|
|
the rule: `whole-node` (a control's whole name or text), `minted-token` (a
|
|
code or id the app showed as its own word), `date`, or `pixels`. Values from
|
|
the instruction or `params`, or ones the agent composed itself, replay.
|
|
|
|
## What a recording holds
|
|
|
|
Each entry is a JSON file under `.e2e/cache/`, named by a hash of its key.
|
|
It contains actions, target descriptions, typed text, a step summary, and
|
|
checks for the final state. Those checks include the route, up to eight
|
|
controls that appeared during the step, alerts and status messages first,
|
|
and up to eight that went away.
|
|
|
|
Entries contain no prompts, conversations, screenshots, observed trees, or
|
|
secret values. A secret fill stores only the secret's name. Other typed
|
|
values are stored verbatim, so review recordings before sharing them; see
|
|
[cache trust](/security#cache-trust).
|
|
|
|
## Modes
|
|
|
|
| Mode | Replays | Records | Default |
|
|
| --- | --- | --- | --- |
|
|
| `'read-write'` | yes | yes | locally |
|
|
| `'read-only'` | yes | no | in CI when `cache` is unset |
|
|
| `'off'` | no | no | `--no-cache` for one run |
|
|
|
|
```ts title="e2e.config.ts fragment"
|
|
cache: 'off',
|
|
// or
|
|
cache: { mode: 'read-write', dir: '.e2e/cache' },
|
|
```
|
|
|
|
`--no-cache` overrides the config. To let CI update recordings, explicitly
|
|
set `cache: 'read-write'`. In `read-only` mode, stale entries remain until
|
|
a read-write run replaces them.
|
|
|
|
A custom `cache.store` replaces the file store and does not default to
|
|
read-only in CI. `e2e explore` and `e2e mcp` do not use the cache. See the
|
|
[config reference](/reference/config#cache).
|
|
|
|
## Commit the replay cache
|
|
|
|
<span id="commit-your-traces" />
|
|
|
|
`e2e init` adds `.e2e/cache/` to `.gitignore`, so every machine records its
|
|
own and CI starts with none. To share replays, remove that line and commit
|
|
the directory. CI stays `read-only` unless you set `cache: 'read-write'`.
|
|
Treat committed entries as test data and review them in the pull request
|
|
that re-recorded them.
|
|
|
|
Local runs leave a committed directory clean unless something changed. A
|
|
replayed step never rewrites its entry, and a live re-recording with the same
|
|
actions, paths, and anchors is not written: the summary, the measured end
|
|
wait, and the rule that flagged a typed run-time value differ between two
|
|
live runs of the same flow and do not count. Upgrading `e2e` to a version
|
|
that changes how entries are matched re-keys every entry, so the first run
|
|
after it records them again. A file changes when:
|
|
|
|
- a step records for the first time,
|
|
- the agent finishes a step after a hand-off, re-recording what it saw,
|
|
- a step fails after replaying, or ends in an `end-mismatch` the agent had
|
|
to repair, which deletes the entry,
|
|
- a step with an entry passes without leaving anything to record, which
|
|
deletes the entry,
|
|
- no verification step confirmed the entry by the end of the attempt, which
|
|
deletes it.
|
|
|
|
A replay that still finds every control refreshes nothing, so an entry for a
|
|
drifting UI is repaired only once a control can no longer be found.
|
|
|
|
## Fail on a stale recording
|
|
|
|
A committed recording that stops replaying is quiet by default: the agent
|
|
takes the step over, the test passes, and every CI run pays for the model
|
|
calls until someone re-records. `--strict-cache` (or `cache: { strict: true }`)
|
|
makes it loud. A step whose recording exists but no longer replays fails
|
|
with `REPLAY_STALE` and makes no model call:
|
|
|
|
<CodeGroup>
|
|
```bash npm
|
|
CI=1 npx e2e run --strict-cache
|
|
```
|
|
|
|
```bash pnpm
|
|
CI=1 pnpm exec e2e run --strict-cache
|
|
```
|
|
|
|
```bash bun
|
|
CI=1 bunx e2e run --strict-cache
|
|
```
|
|
</CodeGroup>
|
|
|
|
The reasons that fail are the ones that mean the app or the entry changed:
|
|
`invalid-entry`, `wrong-context`, `target-not-found`, `target-ambiguous`,
|
|
`viewport-changed`, `action-failed`, `action-uncertain`, and `end-mismatch`.
|
|
A `no-entry` miss fails too when the cache directory holds a recording of the
|
|
same step under another key, from before an `e2e` or engine upgrade or a
|
|
change to the project id, the app's identity, or the agent's context. The
|
|
message names that file; delete it once nothing replays it. A custom
|
|
`cache.store` is not checked this way. Entries recorded before this check
|
|
existed are covered once a `read-write` run replays or re-records them. A
|
|
step never recorded (a new or edited instruction or params), a retry, a
|
|
`gap`, and a `truncated` recording still run live. `REPLAY_STALE` exits 2 and is not retried, since a
|
|
retry never replays and would pass live. The entry stays where it is, even
|
|
in `read-write` mode, so every strict run fails on it until it is re-recorded:
|
|
run the test in `read-write` mode without the flag, and with `cache.strict`
|
|
off when the config set it, then commit the changed entry; a custom
|
|
`cache.store` receives the new entry from that run instead. The message
|
|
names the knob that turned strict on and where the entry lives: the cache
|
|
directory, or the configured `cache.store`. A store whose read fails is not
|
|
a stale recording; the step runs live.
|
|
|
|
## Inspect the cache
|
|
|
|
<CodeGroup>
|
|
```bash npm
|
|
npx e2e cache ls # one row per entry: test, target, instruction digest, age, actions
|
|
npx e2e cache stats # entry count and size
|
|
npx e2e cache clear # delete every entry
|
|
```
|
|
|
|
```bash pnpm
|
|
pnpm exec e2e cache ls # one row per entry: test, target, instruction digest, age, actions
|
|
pnpm exec e2e cache stats # entry count and size
|
|
pnpm exec e2e cache clear # delete every entry
|
|
```
|
|
|
|
```bash bun
|
|
bunx e2e cache ls # one row per entry: test, target, instruction digest, age, actions
|
|
bunx e2e cache stats # entry count and size
|
|
bunx e2e cache clear # delete every entry
|
|
```
|
|
</CodeGroup>
|
|
|
|
<Note>
|
|
The `cached` percentage in AI usage is the provider's prompt cache. It
|
|
reduces the cost of a model call. The replay cache described here skips the
|
|
call entirely.
|
|
</Note>
|
|
|
|
<CardGroup cols={2}>
|
|
<Card title="Config reference" icon="sliders" href="/reference/config#cache">
|
|
`mode`, `store`, `dir`.
|
|
</Card>
|
|
<Card title="CLI reference" icon="terminal" href="/reference/cli#e2e-cache">
|
|
`e2e cache ls`, `stats`, `clear`.
|
|
</Card>
|
|
</CardGroup>
|