mirror of
https://github.com/tester-army/e2e.git
synced 2026-11-21 03:05:26 +08:00
Co-authored-by: Oskar Kwaśniewski <oskar@okwasniewski.com>
276 lines
13 KiB
Plaintext
276 lines
13 KiB
Plaintext
---
|
|
title: Custom executors
|
|
description: Customize the agent loop or implement your own step executor.
|
|
"og:image": "https://e2e.com/docs/images/og/executors.png?v=283f7962bb"
|
|
"twitter:image": "https://e2e.com/docs/images/og/executors.png?v=283f7962bb"
|
|
---
|
|
|
|
A step executor runs the `agent.act` and `agent.assert` steps of an agent.
|
|
Write your own when a `system` prompt and [project tools](/tools) are not
|
|
enough.
|
|
|
|
| Option | You write | The runner keeps |
|
|
| --- | --- | --- |
|
|
| `createToolLoopExecutor` from `e2e/agent` | The tools and the prompt. The loop adds only `complete_step`. | Budgets, repeated-call detection, verdicts, usage accounting, and debug transcripts |
|
|
| `StepExecutor` | `runStep(ctx)`. A model is optional, and the `e2e` and `e2e/engine` types compile without the `ai` package. | The step deadline, redacted observations, checked actions, and the verdict record |
|
|
|
|
Set either one as `executor` in an agents entry. The entry still reads
|
|
`model`, `judge`, `context`, and the budgets, and rejects `system` and
|
|
`tools`: pass those to `createToolLoopExecutor`. For a provider that returns
|
|
bounded choices, see the [decision executor](/decision-models).
|
|
|
|
## Keep the loop, change the vocabulary
|
|
|
|
This agent has two tools: one reads the screen, one taps a node by its id.
|
|
|
|
```ts title="e2e.config.ts"
|
|
import type { E2EConfig } from 'e2e';
|
|
import { createToolLoopExecutor } from 'e2e/agent';
|
|
import { web } from '@e2e-dev/web';
|
|
import { gateway, tool } from 'ai';
|
|
import { z } from 'zod';
|
|
|
|
export default {
|
|
targets: [{ engine: web(), app: { url: 'http://localhost:3000' } }],
|
|
agents: {
|
|
default: {
|
|
model: gateway('openai/gpt-6-luna-fast'),
|
|
maxModelCalls: 15,
|
|
executor: createToolLoopExecutor({
|
|
name: 'tap-only',
|
|
system: 'Read the screen before acting. Tap controls by their id from the latest read.',
|
|
buildPrompt: (ctx) => ctx.step.instruction,
|
|
tools: (ctx, { guard }) => ({
|
|
read_screen: tool({
|
|
description: 'Read the screen as text, one node per line as #id role "name"',
|
|
inputSchema: z.object({}),
|
|
execute: () => guard(async () => (await ctx.observe()).text),
|
|
}),
|
|
tap: tool({
|
|
description: 'Tap a node by its id from the latest read_screen',
|
|
inputSchema: z.object({ id: z.string() }),
|
|
execute: ({ id }) =>
|
|
guard(async () => {
|
|
await ctx.actions.tap({ id: id.replace(/^#/, '') });
|
|
return `Tapped ${id}.`;
|
|
}),
|
|
}),
|
|
}),
|
|
}),
|
|
},
|
|
},
|
|
} satisfies E2EConfig;
|
|
```
|
|
|
|
The loop runs at most `maxModelCalls` model turns per step with the entry's
|
|
`model`, or its own from `createToolLoopExecutor({ model })`. An executor can
|
|
also bring a `judge`. When the entry sets the same option, both must name the
|
|
same provider and model, or the config load fails with `INVALID_CONFIG`.
|
|
|
|
`buildPrompt` returns a string or a message history. `prepareMessages` runs
|
|
between turns to shorten the history, or returns `{ messages, stop }` when the
|
|
step makes no progress. A message it rewrites, in place or by replacement,
|
|
edits the request prefix, and on Anthropic models, whose thinking is bound
|
|
to it, the loop then stops sending the reasoning from that message on.
|
|
When the provider refuses a request as too large, the loop shortens the
|
|
history and retries once if that makes the request smaller, else fails with
|
|
`CONTEXT_OVERFLOW`; `inconclusiveOnOverflow: true` fails the step with
|
|
`ASSERTION_INCONCLUSIVE` instead, for an assertion loop whose earlier screens
|
|
are evidence. `guard()` stops new tool calls when the step is
|
|
concluding and returns a failure to the model as text. `ctx.actions`
|
|
authorizes, counts, and records each action, so these steps replay from the
|
|
[cache](/cache) and run on any engine that supports `tap`.
|
|
|
|
To keep the built-in actions beside your own tools, spread
|
|
`createGrammarTools(ctx, { guard })` into the toolset. It returns the
|
|
built-in agent's action tools for the verbs the target's engine supports,
|
|
and each result reports what changed on screen. `BASE_RULES` is the
|
|
built-in agent's guidance for those tools; start your `system` with it.
|
|
|
|
```ts
|
|
import { BASE_RULES, createGrammarTools, createToolLoopExecutor } from 'e2e/agent';
|
|
|
|
createToolLoopExecutor({
|
|
name: 'with-inbox',
|
|
system: `${BASE_RULES}\n\nCheck the inbox with read_inbox when a step mentions an email.`,
|
|
buildPrompt: (ctx) => ctx.step.instruction,
|
|
tools: (ctx, { guard }) => ({ ...createGrammarTools(ctx, { guard }), read_inbox: readInbox }),
|
|
});
|
|
```
|
|
|
|
Outside the loop, a grammar tool returns a `ScreenOutput`: text, or text
|
|
with the screenshot. When a tool throws an error that
|
|
`isRuntimeHardStop(error)` accepts, end the step and rethrow it.
|
|
|
|
A tool that works outside `ctx.actions`, such as a call to your test API,
|
|
wraps its body in `ctx.budgets.runTool({ name, mutates })`, which takes an
|
|
action slot and records the outcome. Don't call `ctx.actions` or
|
|
`ctx.observe` inside `runTool`: the call waits behind itself and deadlocks.
|
|
|
|
## Replace the executor
|
|
|
|
This executor uses no model and supports one assertion,
|
|
`agent.assert('the screen says Ready')`. Any other step returns a blocked
|
|
verdict.
|
|
|
|
```ts title="e2e.config.ts"
|
|
import type { E2EConfig, StepExecutor } from 'e2e';
|
|
import { web } from '@e2e-dev/web';
|
|
|
|
const scripted: StepExecutor = {
|
|
name: 'scripted',
|
|
version: '1',
|
|
cache: 'off',
|
|
async runStep(ctx) {
|
|
if (ctx.step.kind !== 'assert' || ctx.step.instruction !== 'the screen says Ready') {
|
|
return {
|
|
status: 'blocked',
|
|
errorCode: 'AUTOMATION_UNSUPPORTED',
|
|
summary: 'This executor only checks whether the screen says Ready.',
|
|
};
|
|
}
|
|
const screen = await ctx.observe();
|
|
const ready = screen.text.includes('Ready');
|
|
return {
|
|
status: ready ? 'passed' : 'failed',
|
|
summary: ready ? 'The screen says Ready.' : 'Ready is absent from the screen.',
|
|
};
|
|
},
|
|
};
|
|
|
|
export default {
|
|
targets: [{ engine: web(), app: { url: 'http://localhost:3000' } }],
|
|
agents: { default: { executor: scripted } },
|
|
} satisfies E2EConfig;
|
|
```
|
|
|
|
## What the context holds
|
|
|
|
| Member | What it is |
|
|
| --- | --- |
|
|
| `ctx.step` | The `kind`, instruction, params, and `index` in the attempt's timeline. A configured secret's value in the instruction or a param reads `<secret:name>`. A `secrets.get()` handle arrives as a name-only placeholder that `typeSecret` fills |
|
|
| `ctx.observe({ tree, pixels })` | Redacted screen text, with an optional node tree and masked screenshot |
|
|
| `ctx.read({ id }, { style })` | Secret-redacted computed styles of a currently listed node. See [Project tools](/tools#read-a-nodes-computed-style) |
|
|
| `ctx.actions` | Checked and recorded actions: `tap`, `doubleTap`, `longPress`, `secondaryTap`, `hover`, `type`, `typeSecret`, `press`, `select`, `check`, `drag`, `scrollTo`, `scrollUntil`, `upload`, `scroll`, `navigate`, `back`, `resize`, `switchTab`, `closeTab`, and the point and keyboard actions below |
|
|
| `ctx.target` | The target's `name`, `platform`, and `verbs`, the actions its engine supports. Offer only these to the model. `tapModifiers` is true when `tap`, `doubleTap`, and `secondaryTap` take `{ modifiers }` |
|
|
| `ctx.signal` | Aborts on cancellation, at the step deadline, or on any other hard stop. The deadline can't stop work that ignores it |
|
|
| `ctx.replayedPrefix` | Set when a cache replay ran part of the step first. See [Replay cache](#replay-cache) |
|
|
| `ctx.model`, `ctx.providerOptions` | The agent's configured model and `providerOptions`, when set |
|
|
| `ctx.agentContext` | The agent's `context` plus the test's `agentContext` |
|
|
| `ctx.ledger` | The completed steps of the test |
|
|
| `ctx.attempt` | Test and attempt ids, the test's `titlePath` (empty outside `e2e run`), retry index, abort signal, and a `memory` map that is never reported |
|
|
| `ctx.budgets` | Accounting for custom tools (`runTool`) and model calls (`recordModelCall`) |
|
|
| `ctx.attachTranscript(text)` | Under `--debug`, attaches the model transcript as a `log` artifact. Call it one time, at the end |
|
|
| `ctx.attachTurns(turns)` | Keeps a clipped record of the model turns (`{ index, calls, outcome }`) on the step. The report shows the last ones under a failed step |
|
|
| `ctx.attachScreenshot(pixels, label)` | Keeps observed pixels as a `screenshot` artifact of the step and resolves with its report id |
|
|
|
|
`tapAt(point)` and `hoverAt(point)` call the engine's coordinate action, or
|
|
act on the listed control at the point when the engine has none.
|
|
`dragAt(from, to)` drags between the points through the engine's `dragTo`
|
|
pointer kind. Without it, an engine with the `dragTo` action drags by id when
|
|
the points land on two different listed controls. Any other `dragAt` fails
|
|
with `UNSUPPORTED_CAPABILITY`. `hitTest(point)` finds the control at a point
|
|
without acting. Engines with a keyboard add `typeText`, `pressKey`, and
|
|
`dismissKeyboard` for the focused control. `upload(target, paths)` takes
|
|
project-relative paths and the `sample:` files listed under
|
|
[the upload policy](/reference/agent#action-tools), refuses a path outside the project
|
|
root or a hidden one with `POLICY_DENIED`, and a missing file with
|
|
`INVALID_ARGUMENT`.
|
|
|
|
The observation's `keyboardVisible` says whether an on-screen keyboard is
|
|
showing, and is absent when the engine can't tell.
|
|
|
|
When semantic capture fails, the observation sets `treeUnavailable: true`,
|
|
has no `tree`, and includes a screenshot if policy allows. Earlier node ids
|
|
are invalid, and a missing tree doesn't prove a control is absent.
|
|
|
|
## What the executor handles
|
|
|
|
`act` and `assert` reach `runStep` with the agent's `maxSteps`,
|
|
`maxModelCalls`, and `timeout`. On `assert`, `vision` arrives as
|
|
`ctx.step.vision` only when the executor declares `vision: true`; otherwise
|
|
the call fails with `UNSUPPORTED_CAPABILITY`. `screenshot` still attaches
|
|
evidence after your verdict.
|
|
|
|
`waitFor` and `extract` still use the built-in judgments. Without a
|
|
configured model they fail with `MODEL_UNAVAILABLE`.
|
|
|
|
## The verdict
|
|
|
|
Return `passed`, `failed`, or `blocked` with a summary. A blocked verdict
|
|
needs an error code, and a passed one has none.
|
|
[How agent steps work](/agent-steps#how-a-step-ends) lists the codes.
|
|
Return `STEP_BUDGET_EXHAUSTED`, `STEP_TIMEOUT`, or `CANCELLED` only when that
|
|
budget is spent, the deadline passed, or the test was cancelled. The runner
|
|
refuses any other use, and a passed verdict with a code, with
|
|
`MODEL_OUTPUT_INVALID`.
|
|
|
|
For a model your executor drives itself, `createVerdictTool(ctx.step.kind)`
|
|
from `e2e/agent` is the built-in agent's `complete_step`. Give the model its
|
|
`tool` and `VERDICT_RULES`, then return `verdict()`. On an `act` step it drops
|
|
`ASSERTION_INCONCLUSIVE` from a failed verdict.
|
|
|
|
## Replay cache
|
|
|
|
`cache` on a `StepExecutor` sets how its `act` steps use the
|
|
[replay cache](/cache). `assert` steps are never cached.
|
|
|
|
`'inherit'`, the default, follows the configured cache mode: a recorded step
|
|
replays without calling `runStep`. `'off'` never replays or records. Use it
|
|
when the executor's own history or memory is the source of truth.
|
|
|
|
When a replay stops partway, the runner calls `runStep` with
|
|
`ctx.replayedPrefix`: the replayed actions and why replay stopped. Continue
|
|
from the current screen without repeating them.
|
|
|
|
The cache records what a step does through `ctx.actions`. A mutating tool
|
|
call leaves a gap that the executor fills on replay. Entries are keyed by the
|
|
configured agent and its context, not the executor, so changing an agent's
|
|
executor keeps its entries.
|
|
|
|
## Driving the page yourself
|
|
|
|
On the web engine, `surfaceOf(handle)` from `@e2e-dev/web` gives you the
|
|
current attempt's `Page` and `BrowserContext`. Both accessors throw
|
|
`INVALID_STATE` outside an active attempt, and `page()` also when no page is
|
|
open.
|
|
|
|
```ts
|
|
import type { E2EConfig, StepExecutor } from 'e2e';
|
|
import { web, surfaceOf } from '@e2e-dev/web';
|
|
|
|
const engine = web();
|
|
const own: StepExecutor = {
|
|
name: 'own-tools',
|
|
version: '1',
|
|
cache: 'off',
|
|
async runStep(ctx) {
|
|
const page = surfaceOf(engine)!.page(); // the page app.open() navigated
|
|
await ctx.budgets.runTool({ name: 'click', mutates: true }, () =>
|
|
page.getByRole('button', { name: 'Save' }).click(),
|
|
);
|
|
return { status: 'passed', summary: 'saved' };
|
|
},
|
|
};
|
|
|
|
export default {
|
|
targets: [{ engine, app: { url: 'http://localhost:3000' } }],
|
|
agents: { default: { executor: own } },
|
|
} satisfies E2EConfig;
|
|
```
|
|
|
|
Direct Playwright calls skip action authorization and the replay cache,
|
|
and leave no artifacts. The step deadline and the verdict record still
|
|
apply.
|
|
|
|
## Next
|
|
|
|
<CardGroup cols={2}>
|
|
<Card title="agent reference" icon="wand-magic-sparkles" href="/reference/agent">
|
|
Every method, option, and budget.
|
|
</Card>
|
|
<Card title="Writing an engine" icon="plug" href="/writing-an-engine">
|
|
Add support for another platform.
|
|
</Card>
|
|
</CardGroup>
|