mirror of
https://github.com/tester-army/e2e.git
synced 2026-11-22 18:19:48 +08:00
* feat(decision): add Clef and Jev executors on the System One choice API
* feat(decision): default Clef transport to clef-flash
* feat(decision): generalize to decision models, address review
* fix(decision): reject non-string decision choices
* fix(decision): skip empty select labels and blank destinations
An empty option label or whitespace-only destination offered as a choice fails at dispatch with INVALID_ARGUMENT and aborts the step instead of letting the model re-aim. Omit them at enumeration. Also qualify the uncertain-action docs: blocking stops the selected action, while earlier confident actions in the step have already run.
* feat(decision): add opt-in vision for Clef, Jev stays text-only
The executor stays text-only by default. With vision true and a transport that declares it, masked viewport pixels ride along as System One images data URLs. Clef declares vision; Jev never receives pixels. Withheld or tainted viewports fall back to semantic text.
* docs(decision): qualify vision cost to calls carrying pixels
* fix(decision): untainted pixel fixtures and honest skill pixels line
The test helper takes a tainted flag so the image-send tests use the only state the runner returns pixels in; tainted captures stay withheld observations. The skill no longer claims pixels are unavailable: screenshots may inform node choices with vision true, without adding point actions.
* bench(testbed): luna vs clef-flash harness and first numbers
Same todo flow under two configs: default agent on gpt-6-luna via OpenCodeX and decisionExecutor on clef-flash. Luna side passed in 23s with 6 model calls; clef-flash pending credentials. Includes the probe script, the docs section, and the flow screenshot.
* docs(decision): fill clef-flash benchmark row
Two no-cache runs both blocked uncertain on the first action (a25 at p=0.619, conf=0.387 against the default 0.9 gates), answering in about a second with 1,543 input tokens.
* docs(decision): benchmark aggregates over 5 runs per side
Luna completed 5/5 flows at ~14s and ~25k input tokens per run; clef-flash blocked uncertain 5/5 at ~1s and ~1.5k tokens per run.
* bench(testbed): parametrized flow, gate sweep, wire-shape finding
Params fix the missing type choices; gates 0.6/0.5 and 0.51/0.3 still block. Direct experiment shows hierarchical structured questions take clef-flash from 0.57 to 0.89 confidence on the same screen.
* feat(decision): hierarchical operation-then-target on structured transports
Structured transports (Clef, Jev) now decide the operation first and the target within it, with object instructions and element records. Flat endpoints keep the single string choice. Live benchmark: first operation 0.909/0.802, first target 0.940/0.775, up from 0.747/0.558 flat.
* docs(decision): compare luna vs clef-flash vs clef with measured cost
* feat(decision): run decisions through AI SDK evaluate, gates opt-in
decisionExecutor now accepts any AI SDK evaluation model that answers
choice questions, such as typeSafeAi.evaluationModel('jev-latest'), and
decides through experimental_evaluate (one choice question per decision,
no retries, so the call budget stays exact). ai is an optional peer that
loads only on that path. evaluationModel() exposes clef(), jev(), and
systemOne() transports to experimental_evaluate, which covers Clef until
it has an AI SDK provider.
The official TypeSafe provider rounds probabilities to two places, so a
long distribution can sum to 0.99: validation now allows half a unit per
rounded option, the AI SDK rule, and jev() declares the two places.
Probability and confidence gates are off by default and stay
configurable. Measuring clef-flash without them surfaced three executor
bugs, each fixed here:
- a lone target is dispatched without a one-option question, which the
choice API rejects with HTTP 422;
- acting decisions receive the step's action history, and the repeat
guard counts the same action on the same screen across the whole step,
so an A-B oscillation blocks instead of burning the call budget;
- the operation and target questions state that typed text is not saved
until it is submitted.
The structured path also checks the call budget before the target
request (review feedback).
Testbed todo flow, 5 no-cache runs each: clef-flash 5/5 (9.3s mean, 12
calls), clef 5/5 (11.5s, 12 calls), luna unchanged at 5/5 (14.9s).
* bench(testbed): Laya through systemOne
Laya serves the System One API and accepts structured values, so it runs
through systemOne({ structured: true }) unchanged. On the todo flow its
first operation question chose unsupported at 0.29 / 0.10 on four runs,
answered by the multilingual checkpoint its router picked for an English
prompt; the fifth run and further probes hit insufficient_credits.
* fix(decision): address review on rounding, history, and evaluation ids
- validateDecision accepts a rounded distribution only when 1 lies between
the sums of each value's rounding interval, clipped to [0, 1]; a summed
per-option tolerance exceeded 1 with many options and admitted two
options at probability 1.
- Decisions send the 20 most recent history entries, each clipped to 240
characters, so long steps and large params stay within request limits.
- evaluationModel() builds its answer, confidence, and criteria maps with
null prototypes, so an id such as __proto__ stays an own entry.
- @ai-sdk/provider moves to dependencies: the public declarations import
its evaluation types. ai stays an optional peer.
* bench(testbed): pin Laya's English checkpoint
Laya's router sent the full English prompt to its multilingual checkpoint.
The API accepts model 'english' to pin the English one (its routing
metadata reports the explicit model), so the bench config sends it.
* docs(decision): benchmark Laya locally on M1, both checkpoints 0/5
* docs(decision): reconcile laya narrative with run evidence, money not tokens
* feat(decision): caller-overridable transport flags on clef() and jev()
* feat(decision): AI SDK evaluate executor with element table and text model
* fix(decision): empty-tree check and provider pin
* fix(decision): address review on secrets, options, errors, and docs
Answers the cubic threads on the evaluate-only executor: terminal checks reject an incomplete screen instead of judging it, secret markers are stripped at any depth, hidden/disabled options are never offered, select options share the 255-choice cap, select criteria name the option, out-of-range probabilities fail, text-model errors mirror the decision mapping, and the README/docs describe the evaluate-only API.
* fix(decision): count dropped select options and fix README
Counts enabled options inside rows dropped past the element cap as omitted, adds the E2EConfig type import to the README example, and qualifies when type is offered.
* fix(decision): settle terminal claims by the check and match codebase style
The second rejected done/failed claim now ends by the check verdict (fails -> ACTION_FAILED, anything else -> blocked), and the completion check honors the gates. The decision state carries the last 10 actions, check targets show their checked state, checked radios are never toggled, and options dropped by the per-question cap stay out of omitted. The docs example opens /todos in test code since navigate is not offered. Single quotes, template literals, and split long lines across the package.
* fix(decision): show filled fields to verdicts and pin the history window
Verdict views (assert and the completion check) build the element table with typing enabled, so a field the step filled keeps its row and value even when the step itself cannot type. The history test now asserts the window holds the newest ten actions.
* fix(decision): let completion checks see results and name the text key
Page text keeps a named node's text when it differs from the name, as formatNode does, so a status named Greeting that says Welcome back, admin! reaches the verdict. The completion check now asks whether the task is complete and sees the non-secret params and the actions with their typed values; rejected claims stay out. The text prompt names the single text key, asks for no code fence, and prefixes the request so JSON-object response modes accept it.
* fix(decision): keep secret handles out of replayed history and mark check inputs as data
Replayed summaries name a filled secret handle; seeding the history now replaces declared secret names with <secret>, so names reach the model only as secret question criteria. The completion check treats action descriptions, typed values, and params as data, and the guide says the check sees only non-secret values.
* fix(decision): redact only the handle in replayed secret-fill summaries
Matching the declared secret name anywhere in a replayed summary also erased labels and values equal to it, and missed handles the runner cut at 40 characters. Only the quoted handle after fill secret at the start of the summary becomes <secret> now.
* docs(decision): describe the one-request fan-out in the guide intro
* fix(decision): count a scroll that changes what is in view as progress
The fingerprint behind the stall guard hashed node content but not position, so a scroll that only moved the viewport read as page unchanged and the third one blocked the step. It now also hashes whether each node meets the viewport (membership, not coordinates), so a scroll that brings other nodes into view resets the count while one that moves nothing, as at the page bottom, still trips the guard.
* chore: prune @pnpm/exe from the lockfile after merging main
* fix(decision): match the supported Node range in engines
---------
Co-authored-by: Oskar Kwaśniewski <oskar@okwasniewski.com>
248 lines
11 KiB
Plaintext
248 lines
11 KiB
Plaintext
---
|
|
title: Custom executors
|
|
description: Customize the agent loop or implement your own step executor.
|
|
---
|
|
|
|
Use a custom executor when a `system` prompt and [project tools](/tools) are
|
|
not enough. There are two options:
|
|
|
|
1. `createToolLoopExecutor` keeps the built-in loop: budgets, repeated-call
|
|
detection, verdicts, usage accounting, and debug transcripts. Your tools
|
|
replace the built-in actions; the loop adds only `complete_step`.
|
|
2. `StepExecutor` lets you implement `runStep(ctx)` yourself. A model and
|
|
the AI SDK are optional; the `e2e` and `e2e/engine` types compile with no
|
|
`ai` package installed.
|
|
|
|
Both receive a deadline, redacted observations, and checked action methods.
|
|
For a provider that returns bounded choices instead of generated text, see
|
|
the [Clef and Jev decision executor](/decision-models).
|
|
The runner records their verdicts. Either goes under `executor` in an agents
|
|
entry, beside the options that still apply: `model`, `judge`, `context`, and
|
|
the budgets. An entry with `executor` rejects `system` and `tools`; pass
|
|
them to `createToolLoopExecutor` instead.
|
|
|
|
## Keep the loop, change the vocabulary
|
|
|
|
This example gives the agent two tools: one reads the screen, one taps a
|
|
node by its id.
|
|
|
|
```ts title="e2e.config.ts"
|
|
import type { E2EConfig } from 'e2e';
|
|
import { createToolLoopExecutor } from 'e2e/agent';
|
|
import { web } from '@e2e-dev/web';
|
|
import { gateway, tool } from 'ai';
|
|
import { z } from 'zod';
|
|
|
|
export default {
|
|
targets: [{ engine: web(), app: { url: 'http://localhost:3000' } }],
|
|
agents: {
|
|
default: {
|
|
model: gateway('openai/gpt-6-luna-fast'),
|
|
maxModelCalls: 15,
|
|
executor: createToolLoopExecutor({
|
|
name: 'tap-only',
|
|
system: 'Read the screen before acting. Tap controls by their id from the latest read.',
|
|
buildPrompt: (ctx) => ctx.step.instruction,
|
|
tools: (ctx, { guard }) => ({
|
|
read_screen: tool({
|
|
description: 'Read the screen as text, one node per line as #id role "name"',
|
|
inputSchema: z.object({}),
|
|
execute: () => guard(async () => (await ctx.observe()).text),
|
|
}),
|
|
tap: tool({
|
|
description: 'Tap a node by its id from the latest read_screen',
|
|
inputSchema: z.object({ id: z.string() }),
|
|
execute: ({ id }) =>
|
|
guard(async () => {
|
|
await ctx.actions.tap({ id: id.replace(/^#/, '') });
|
|
return `Tapped ${id}.`;
|
|
}),
|
|
}),
|
|
}),
|
|
}),
|
|
},
|
|
},
|
|
} satisfies E2EConfig;
|
|
```
|
|
|
|
The loop reads the entry's `model` unless `createToolLoopExecutor({ model })`
|
|
brings its own, and its model turns per step are the entry's `maxModelCalls`.
|
|
`buildPrompt` returns a string or message history. Use `prepareMessages`
|
|
between turns to shorten the history or return `{ messages, stop }` when
|
|
the step is not progressing.
|
|
|
|
`guard()` stops new tool calls once the step is concluding and hands a
|
|
failure back to the model as text. `ctx.actions` authorizes, counts, and
|
|
records each action, so these steps replay from the [cache](/cache) like the
|
|
built-in agent's and run on any engine that supports `tap`.
|
|
|
|
A tool that works outside `ctx.actions`, such as a call to your test API,
|
|
wraps its body in `ctx.budgets.runTool({ name, mutates })` to take an action
|
|
slot and record the outcome. Never call `ctx.actions` or `ctx.observe` inside
|
|
`runTool`: the call queues behind its own operation and deadlocks. Keep tool
|
|
bodies short and support cancellation. The step deadline cannot cancel an
|
|
underlying operation that ignores its signal.
|
|
|
|
A custom executor can bring its own `model` or `judge`, as
|
|
`createToolLoopExecutor({ model })` does. If the agents entry sets one too,
|
|
they must name the same provider and model, or the config load fails with
|
|
`INVALID_CONFIG`.
|
|
|
|
## Replace the executor
|
|
|
|
This model-free executor supports one assertion,
|
|
`agent.assert('the screen says Ready')`. Other instructions return a blocked
|
|
verdict so the executor cannot claim to have checked something it does not
|
|
understand.
|
|
|
|
```ts title="e2e.config.ts"
|
|
import type { E2EConfig, StepExecutor } from 'e2e';
|
|
import { web } from '@e2e-dev/web';
|
|
|
|
const scripted: StepExecutor = {
|
|
name: 'scripted',
|
|
version: '1',
|
|
cache: 'off',
|
|
async runStep(ctx) {
|
|
if (ctx.step.kind !== 'assert' || ctx.step.instruction !== 'the screen says Ready') {
|
|
return {
|
|
status: 'blocked',
|
|
errorCode: 'AUTOMATION_UNSUPPORTED',
|
|
summary: 'This executor only checks whether the screen says Ready.',
|
|
};
|
|
}
|
|
const screen = await ctx.observe();
|
|
const ready = screen.text.includes('Ready');
|
|
return {
|
|
status: ready ? 'passed' : 'failed',
|
|
summary: ready ? 'The screen says Ready.' : 'Ready is absent from the screen.',
|
|
};
|
|
},
|
|
};
|
|
|
|
export default {
|
|
targets: [{ engine: web(), app: { url: 'http://localhost:3000' } }],
|
|
agents: { default: { executor: scripted } },
|
|
} satisfies E2EConfig;
|
|
```
|
|
|
|
The executor receives this context:
|
|
|
|
| Member | What it is |
|
|
| --- | --- |
|
|
| `ctx.step` | The instruction, params, and `index` in the attempt's timeline. A registered secret value the test spelled into the instruction or a param string reads `<secret:name>`; a `secrets.get()` handle is a placeholder you fill with `typeSecret` |
|
|
| `ctx.observe({ tree, pixels })` | Redacted screen text, with an optional node tree and masked screenshot |
|
|
| `ctx.actions` | Checked and recorded actions: `tap`, `doubleTap`, `longPress`, `secondaryTap`, `hover`, `type`, `typeSecret`, `press`, `select`, `check`, `drag`, `scrollTo`, `scrollUntil`, `upload`, `scroll`, `navigate`, `back`, and the point and keyboard verbs below |
|
|
| `ctx.target` | The target's `name` and `platform`, and `verbs`, the actions its engine supports. Offer only these to the model. |
|
|
| `ctx.pixelsTainted` | True after a [secret fill](/security#screenshots-after-a-secret-fill), when no screenshots are available. A secret an engine option holds (a basic-auth password) is redacted from text only and does not set it |
|
|
| `ctx.signal` | Aborts on cancellation, at the step deadline, or on any other hard stop. Stop promptly when it fires |
|
|
| `ctx.replayedPrefix` | Set when a cache replay ran part of the step before handing it over: the replayed actions and why replay stopped |
|
|
| `ctx.model`, `ctx.providerOptions` | The agent's configured model and `providerOptions`, when set |
|
|
| `ctx.agentContext` | The agent's `context` plus the test's `agentContext` |
|
|
| `ctx.ledger` | Completed steps in the test |
|
|
| `ctx.attempt` | Test and attempt ids, retry index, abort signal, and a `memory` map that is never reported |
|
|
| `ctx.budgets` | Accounting for custom tools and model calls |
|
|
| `ctx.attachTranscript(text)` | Attaches the model transcript to the step. Kept as a `log` artifact under `--debug`, a no-op otherwise; call once, at conclusion |
|
|
| `ctx.attachTurns(turns)` | Keeps a clipped record of the step's model turns (`{ index, calls, outcome }`) on the step record whatever the debug setting; the report shows the last ones under a failed step |
|
|
| `ctx.attachScreenshot(pixels, label)` | Keeps observed pixels as a `screenshot` artifact of the step and resolves with its report id |
|
|
|
|
`tapAt(point)` and `hoverAt(point)` use a node id when a listed control
|
|
occupies the point and otherwise call the engine's coordinate action.
|
|
`hitTest(point)` finds the listed control without acting. Keyboard-capable
|
|
engines also provide `typeText`, `pressKey`, and `dismissKeyboard` for the
|
|
focused control. `upload(target, paths)` authorizes every path against the
|
|
project root before the engine sees it: outside the project or hidden is
|
|
`POLICY_DENIED`, a missing file `INVALID_ARGUMENT`.
|
|
|
|
If semantic capture fails, an observation sets `treeUnavailable: true` and
|
|
omits `tree`. It includes a screenshot when policy allows one, even if pixels
|
|
were not requested. Earlier node ids are invalid, and a missing tree does
|
|
not prove a control is absent. Use fresh ids once the tree returns.
|
|
|
|
Secrets in params arrive as name-only placeholders.
|
|
`ctx.actions.typeSecret(target, name)` fills an authorized field without
|
|
passing the plaintext through your executor.
|
|
|
|
### What the executor handles
|
|
|
|
A custom executor handles `act` and `assert` through `runStep`. Both use the
|
|
agent's `maxSteps` and `maxModelCalls` and default to the configured
|
|
`timeout`. On `assert`, `vision` raises `UNSUPPORTED_CAPABILITY`, since your
|
|
executor chooses what it observes; `screenshot` still attaches evidence after
|
|
your verdict.
|
|
|
|
`waitFor` and `extract` still use the built-in judgments. They require a
|
|
configured model and raise `MODEL_UNAVAILABLE` without one.
|
|
|
|
### Driving the page yourself
|
|
|
|
Use `surfaceOf(handle)` from `@e2e-dev/web` to access the current
|
|
attempt's `Page` and `BrowserContext`. Both accessors throw `INVALID_STATE`
|
|
outside an active attempt.
|
|
|
|
```ts
|
|
import type { E2EConfig, StepExecutor } from 'e2e';
|
|
import { web, surfaceOf } from '@e2e-dev/web';
|
|
|
|
const engine = web();
|
|
const own: StepExecutor = {
|
|
name: 'own-tools',
|
|
version: '1',
|
|
cache: 'off',
|
|
async runStep(ctx) {
|
|
const page = surfaceOf(engine)!.page(); // the page app.open() navigated
|
|
await ctx.budgets.runTool({ name: 'click', mutates: true }, () =>
|
|
page.getByRole('button', { name: 'Save' }).click(),
|
|
);
|
|
return { status: 'passed', summary: 'saved' };
|
|
},
|
|
};
|
|
|
|
export default {
|
|
targets: [{ engine, app: { url: 'http://localhost:3000' } }],
|
|
agents: { default: { executor: own } },
|
|
} satisfies E2EConfig;
|
|
```
|
|
|
|
Direct Playwright calls bypass the runner's action authorization and the
|
|
replay cache. The runner still enforces the step deadline and records the
|
|
verdict, but these calls have no automatic replay or artifacts. Use
|
|
`ctx.budgets` to count custom tool and model calls.
|
|
|
|
### Replay cache
|
|
|
|
`cache` on a `StepExecutor` sets how its `act` steps use the
|
|
[replay cache](/cache); `assert` steps are never cached. `'inherit'`, the
|
|
default, follows the configured cache mode. A recorded step replays without
|
|
calling `runStep`. A replay that stops partway calls `runStep` with
|
|
`ctx.replayedPrefix`: continue from the current screen and do not redo the
|
|
replayed actions. What a step does through `ctx.actions` is recorded; a
|
|
mutating tool call leaves a gap the executor fills on replay. Entries are
|
|
keyed by the configured agent and its context, so an entry one agent
|
|
recorded never replays for another; the executor itself is not part of the
|
|
key, so changing an agent's executor keeps its entries.
|
|
|
|
`'off'` never replays or records the executor's steps, so every step reaches
|
|
`runStep`. Use it for a brain whose own history or memory is the source of
|
|
truth, or one that must see every step run. The two `StepExecutor` samples on
|
|
this page set it so their code runs on every call.
|
|
|
|
## The verdict
|
|
|
|
Return `passed`, `failed`, or `blocked` with a summary. A blocked verdict
|
|
also needs an error code, and a passed one carries none.
|
|
[How agent steps work](/agent-steps#how-a-step-ends) lists the codes and
|
|
resulting exits. `STEP_BUDGET_EXHAUSTED`, `STEP_TIMEOUT`, and `CANCELLED`
|
|
belong to the runner: return one only when that budget is spent, the step
|
|
deadline has passed, or the test was cancelled. Returning one otherwise, or
|
|
a passed verdict with any code, is refused with `MODEL_OUTPUT_INVALID`.
|
|
|
|
<CardGroup cols={2}>
|
|
<Card title="agent reference" icon="wand-magic-sparkles" href="/reference/agent">
|
|
Every method, option, and budget.
|
|
</Card>
|
|
<Card title="Writing an engine" icon="plug" href="/writing-an-engine">
|
|
Add support for another platform.
|
|
</Card>
|
|
</CardGroup>
|