mirror of
https://github.com/tester-army/e2e.git
synced 2026-11-22 18:19:48 +08:00
An action the model sent in the same turn as complete_step was skipped as "already concluding", and the step still passed. The verdict now waits for the calls the model sent before it in the turn, read from the reply's own order, and is refused when one of them did not run or failed, or the last of them changed nothing. A call sent after complete_step does not run. The agent is told that a step which only asks for actions may end in the turn of its last action (ENG-928). Live runs show the models tested rarely do so yet. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
279 lines
13 KiB
Plaintext
279 lines
13 KiB
Plaintext
---
|
|
title: Custom executors
|
|
description: Customize the agent loop or implement your own step executor.
|
|
"og:image": "https://e2e.com/docs/images/og/executors.png?v=283f7962bb"
|
|
"twitter:image": "https://e2e.com/docs/images/og/executors.png?v=283f7962bb"
|
|
---
|
|
|
|
A step executor runs the `agent.act` and `agent.assert` steps of an agent.
|
|
Write your own when a `system` prompt and [project tools](/tools) are not
|
|
enough.
|
|
|
|
| Option | You write | The runner keeps |
|
|
| --- | --- | --- |
|
|
| `createToolLoopExecutor` from `e2e/agent` | The tools and the prompt. The loop adds only `complete_step`. | Budgets, repeated-call detection, verdicts, usage accounting, and debug transcripts |
|
|
| `StepExecutor` | `runStep(ctx)`. A model is optional, and the `e2e` and `e2e/engine` types compile without the `ai` package. | The step deadline, redacted observations, checked actions, and the verdict record |
|
|
|
|
Set either one as `executor` in an agents entry. The entry still reads
|
|
`model`, `judge`, `context`, and the budgets, and rejects `system` and
|
|
`tools`: pass those to `createToolLoopExecutor`. For a provider that returns
|
|
bounded choices, see the [decision executor](/decision-models).
|
|
|
|
## Keep the loop, change the vocabulary
|
|
|
|
This agent has two tools: one reads the screen, one taps a node by its id.
|
|
|
|
```ts title="e2e.config.ts"
|
|
import type { E2EConfig } from 'e2e';
|
|
import { createToolLoopExecutor } from 'e2e/agent';
|
|
import { web } from '@e2e-dev/web';
|
|
import { gateway, tool } from 'ai';
|
|
import { z } from 'zod';
|
|
|
|
export default {
|
|
targets: [{ engine: web(), app: { url: 'http://localhost:3000' } }],
|
|
agents: {
|
|
default: {
|
|
model: gateway('openai/gpt-6-luna-fast'),
|
|
maxModelCalls: 15,
|
|
executor: createToolLoopExecutor({
|
|
name: 'tap-only',
|
|
system: 'Read the screen before acting. Tap controls by their id from the latest read.',
|
|
buildPrompt: (ctx) => ctx.step.instruction,
|
|
tools: (ctx, { guard }) => ({
|
|
read_screen: tool({
|
|
description: 'Read the screen as text, one node per line as #id role "name"',
|
|
inputSchema: z.object({}),
|
|
execute: () => guard(async () => (await ctx.observe()).text),
|
|
}),
|
|
tap: tool({
|
|
description: 'Tap a node by its id from the latest read_screen',
|
|
inputSchema: z.object({ id: z.string() }),
|
|
execute: ({ id }) =>
|
|
guard(async () => {
|
|
await ctx.actions.tap({ id: id.replace(/^#/, '') });
|
|
return `Tapped ${id}.`;
|
|
}),
|
|
}),
|
|
}),
|
|
}),
|
|
},
|
|
},
|
|
} satisfies E2EConfig;
|
|
```
|
|
|
|
The loop runs at most `maxModelCalls` model turns per step with the entry's
|
|
`model`, or its own from `createToolLoopExecutor({ model })`. An executor can
|
|
also bring a `judge`. When the entry sets the same option, both must name the
|
|
same provider and model, or the config load fails with `INVALID_CONFIG`.
|
|
|
|
`buildPrompt` returns a string or a message history. `prepareMessages` runs
|
|
between turns to shorten the history, or returns `{ messages, stop }` when the
|
|
step makes no progress. A message it rewrites, in place or by replacement,
|
|
edits the request prefix, and on Anthropic models, whose thinking is bound
|
|
to it, the loop then stops sending the reasoning from that message on.
|
|
When the provider refuses a request as too large, the loop shortens the
|
|
history and retries once if that makes the request smaller, else fails with
|
|
`CONTEXT_OVERFLOW`; `inconclusiveOnOverflow: true` fails the step with
|
|
`ASSERTION_INCONCLUSIVE` instead, for an assertion loop whose earlier screens
|
|
are evidence. `guard()` stops new tool calls when the step is
|
|
concluding and returns a failure to the model as text. A `complete_step`
|
|
sent in the same turn as other calls waits for the calls before it, and is
|
|
refused when one of them failed or the last one reported no change; calls
|
|
after it don't run. `ctx.actions`
|
|
authorizes, counts, and records each action, so these steps replay from the
|
|
[cache](/cache) and run on any engine that supports `tap`.
|
|
|
|
To keep the built-in actions beside your own tools, spread
|
|
`createGrammarTools(ctx, { guard })` into the toolset. It returns the
|
|
built-in agent's action tools for the verbs the target's engine supports,
|
|
and each result reports what changed on screen. `BASE_RULES` is the
|
|
built-in agent's guidance for those tools; start your `system` with it.
|
|
|
|
```ts
|
|
import { BASE_RULES, createGrammarTools, createToolLoopExecutor } from 'e2e/agent';
|
|
|
|
createToolLoopExecutor({
|
|
name: 'with-inbox',
|
|
system: `${BASE_RULES}\n\nCheck the inbox with read_inbox when a step mentions an email.`,
|
|
buildPrompt: (ctx) => ctx.step.instruction,
|
|
tools: (ctx, { guard }) => ({ ...createGrammarTools(ctx, { guard }), read_inbox: readInbox }),
|
|
});
|
|
```
|
|
|
|
Outside the loop, a grammar tool returns a `ScreenOutput`: text, or text
|
|
with the screenshot. When a tool throws an error that
|
|
`isRuntimeHardStop(error)` accepts, end the step and rethrow it.
|
|
|
|
A tool that works outside `ctx.actions`, such as a call to your test API,
|
|
wraps its body in `ctx.budgets.runTool({ name, mutates })`, which takes an
|
|
action slot and records the outcome. Don't call `ctx.actions` or
|
|
`ctx.observe` inside `runTool`: the call waits behind itself and deadlocks.
|
|
|
|
## Replace the executor
|
|
|
|
This executor uses no model and supports one assertion,
|
|
`agent.assert('the screen says Ready')`. Any other step returns a blocked
|
|
verdict.
|
|
|
|
```ts title="e2e.config.ts"
|
|
import type { E2EConfig, StepExecutor } from 'e2e';
|
|
import { web } from '@e2e-dev/web';
|
|
|
|
const scripted: StepExecutor = {
|
|
name: 'scripted',
|
|
version: '1',
|
|
cache: 'off',
|
|
async runStep(ctx) {
|
|
if (ctx.step.kind !== 'assert' || ctx.step.instruction !== 'the screen says Ready') {
|
|
return {
|
|
status: 'blocked',
|
|
errorCode: 'AUTOMATION_UNSUPPORTED',
|
|
summary: 'This executor only checks whether the screen says Ready.',
|
|
};
|
|
}
|
|
const screen = await ctx.observe();
|
|
const ready = screen.text.includes('Ready');
|
|
return {
|
|
status: ready ? 'passed' : 'failed',
|
|
summary: ready ? 'The screen says Ready.' : 'Ready is absent from the screen.',
|
|
};
|
|
},
|
|
};
|
|
|
|
export default {
|
|
targets: [{ engine: web(), app: { url: 'http://localhost:3000' } }],
|
|
agents: { default: { executor: scripted } },
|
|
} satisfies E2EConfig;
|
|
```
|
|
|
|
## What the context holds
|
|
|
|
| Member | What it is |
|
|
| --- | --- |
|
|
| `ctx.step` | The `kind`, instruction, params, and `index` in the attempt's timeline. A configured secret's value in the instruction or a param reads `<secret:name>`. A `secrets.get()` handle arrives as a name-only placeholder that `typeSecret` fills |
|
|
| `ctx.observe({ tree, pixels })` | Redacted screen text, with an optional node tree and masked screenshot |
|
|
| `ctx.read({ id }, { style })` | Secret-redacted computed styles of a currently listed node. See [Project tools](/tools#read-a-nodes-computed-style) |
|
|
| `ctx.actions` | Checked and recorded actions: `tap`, `doubleTap`, `longPress`, `secondaryTap`, `hover`, `type`, `typeSecret`, `press`, `select`, `check`, `drag`, `scrollTo`, `scrollUntil`, `upload`, `scroll`, `navigate`, `back`, `resize`, `switchTab`, `closeTab`, and the point and keyboard actions below |
|
|
| `ctx.target` | The target's `name`, `platform`, and `verbs`, the actions its engine supports. Offer only these to the model. `tapModifiers` is true when `tap`, `doubleTap`, and `secondaryTap` take `{ modifiers }` |
|
|
| `ctx.signal` | Aborts on cancellation, at the step deadline, or on any other hard stop. The deadline can't stop work that ignores it |
|
|
| `ctx.replayedPrefix` | Set when a cache replay ran part of the step first. See [Replay cache](#replay-cache) |
|
|
| `ctx.model`, `ctx.providerOptions` | The agent's configured model and `providerOptions`, when set |
|
|
| `ctx.agentContext` | The agent's `context` plus the test's `agentContext` |
|
|
| `ctx.ledger` | The completed steps of the test |
|
|
| `ctx.attempt` | Test and attempt ids, the test's `titlePath` (empty outside `e2e run`), retry index, abort signal, and a `memory` map that is never reported |
|
|
| `ctx.budgets` | Accounting for custom tools (`runTool`) and model calls (`recordModelCall`) |
|
|
| `ctx.attachTranscript(text)` | Under `--debug`, attaches the model transcript as a `log` artifact. Call it one time, at the end |
|
|
| `ctx.attachTurns(turns)` | Keeps a clipped record of the model turns (`{ index, calls, outcome }`) on the step. The report shows the last ones under a failed step |
|
|
| `ctx.attachScreenshot(pixels, label)` | Keeps observed pixels as a `screenshot` artifact of the step and resolves with its report id |
|
|
|
|
`tapAt(point)` and `hoverAt(point)` call the engine's coordinate action, or
|
|
act on the listed control at the point when the engine has none.
|
|
`dragAt(from, to)` drags between the points through the engine's `dragTo`
|
|
pointer kind. Without it, an engine with the `dragTo` action drags by id when
|
|
the points land on two different listed controls. Any other `dragAt` fails
|
|
with `UNSUPPORTED_CAPABILITY`. `hitTest(point)` finds the control at a point
|
|
without acting. Engines with a keyboard add `typeText`, `pressKey`, and
|
|
`dismissKeyboard` for the focused control. `upload(target, paths)` takes
|
|
project-relative paths and the `sample:` files listed under
|
|
[the upload policy](/reference/agent#action-tools), refuses a path outside the project
|
|
root or a hidden one with `POLICY_DENIED`, and a missing file with
|
|
`INVALID_ARGUMENT`.
|
|
|
|
The observation's `keyboardVisible` says whether an on-screen keyboard is
|
|
showing, and is absent when the engine can't tell.
|
|
|
|
When semantic capture fails, the observation sets `treeUnavailable: true`,
|
|
has no `tree`, and includes a screenshot if policy allows. Earlier node ids
|
|
are invalid, and a missing tree doesn't prove a control is absent.
|
|
|
|
## What the executor handles
|
|
|
|
`act` and `assert` reach `runStep` with the agent's `maxSteps`,
|
|
`maxModelCalls`, and `timeout`. On `assert`, `vision` arrives as
|
|
`ctx.step.vision` only when the executor declares `vision: true`; otherwise
|
|
the call fails with `UNSUPPORTED_CAPABILITY`. `screenshot` still attaches
|
|
evidence after your verdict.
|
|
|
|
`waitFor` and `extract` still use the built-in judgments. Without a
|
|
configured model they fail with `MODEL_UNAVAILABLE`.
|
|
|
|
## The verdict
|
|
|
|
Return `passed`, `failed`, or `blocked` with a summary. A blocked verdict
|
|
needs an error code, and a passed one has none.
|
|
[How agent steps work](/agent-steps#how-a-step-ends) lists the codes.
|
|
Return `STEP_BUDGET_EXHAUSTED`, `STEP_TIMEOUT`, or `CANCELLED` only when that
|
|
budget is spent, the deadline passed, or the test was cancelled. The runner
|
|
refuses any other use, and a passed verdict with a code, with
|
|
`MODEL_OUTPUT_INVALID`.
|
|
|
|
For a model your executor drives itself, `createVerdictTool(ctx.step.kind)`
|
|
from `e2e/agent` is the built-in agent's `complete_step`. Give the model its
|
|
`tool` and `VERDICT_RULES`, then return `verdict()`. On an `act` step it drops
|
|
`ASSERTION_INCONCLUSIVE` from a failed verdict.
|
|
|
|
## Replay cache
|
|
|
|
`cache` on a `StepExecutor` sets how its `act` steps use the
|
|
[replay cache](/cache). `assert` steps are never cached.
|
|
|
|
`'inherit'`, the default, follows the configured cache mode: a recorded step
|
|
replays without calling `runStep`. `'off'` never replays or records. Use it
|
|
when the executor's own history or memory is the source of truth.
|
|
|
|
When a replay stops partway, the runner calls `runStep` with
|
|
`ctx.replayedPrefix`: the replayed actions and why replay stopped. Continue
|
|
from the current screen without repeating them.
|
|
|
|
The cache records what a step does through `ctx.actions`. A mutating tool
|
|
call leaves a gap that the executor fills on replay. Entries are keyed by the
|
|
configured agent and its context, not the executor, so changing an agent's
|
|
executor keeps its entries.
|
|
|
|
## Driving the page yourself
|
|
|
|
On the web engine, `surfaceOf(handle)` from `@e2e-dev/web` gives you the
|
|
current attempt's `Page` and `BrowserContext`. Both accessors throw
|
|
`INVALID_STATE` outside an active attempt, and `page()` also when no page is
|
|
open.
|
|
|
|
```ts
|
|
import type { E2EConfig, StepExecutor } from 'e2e';
|
|
import { web, surfaceOf } from '@e2e-dev/web';
|
|
|
|
const engine = web();
|
|
const own: StepExecutor = {
|
|
name: 'own-tools',
|
|
version: '1',
|
|
cache: 'off',
|
|
async runStep(ctx) {
|
|
const page = surfaceOf(engine)!.page(); // the page app.open() navigated
|
|
await ctx.budgets.runTool({ name: 'click', mutates: true }, () =>
|
|
page.getByRole('button', { name: 'Save' }).click(),
|
|
);
|
|
return { status: 'passed', summary: 'saved' };
|
|
},
|
|
};
|
|
|
|
export default {
|
|
targets: [{ engine, app: { url: 'http://localhost:3000' } }],
|
|
agents: { default: { executor: own } },
|
|
} satisfies E2EConfig;
|
|
```
|
|
|
|
Direct Playwright calls skip action authorization and the replay cache,
|
|
and leave no artifacts. The step deadline and the verdict record still
|
|
apply.
|
|
|
|
## Next
|
|
|
|
<CardGroup cols={2}>
|
|
<Card title="agent reference" icon="wand-magic-sparkles" href="/reference/agent">
|
|
Every method, option, and budget.
|
|
</Card>
|
|
<Card title="Writing an engine" icon="plug" href="/writing-an-engine">
|
|
Add support for another platform.
|
|
</Card>
|
|
</CardGroup>
|