mirror of
https://github.com/tester-army/e2e.git
synced 2026-11-21 03:05:26 +08:00
830 lines
39 KiB
Plaintext
830 lines
39 KiB
Plaintext
---
|
|
title: agent
|
|
description: Run agent actions, check conditions, and extract structured data.
|
|
"og:image": "https://e2e.com/docs/images/og/reference/agent.png?v=9a239b0466"
|
|
"twitter:image": "https://e2e.com/docs/images/og/reference/agent.png?v=9a239b0466"
|
|
---
|
|
|
|
The `agent` fixture runs natural-language steps against the app: `act`
|
|
interacts with it, `assert` checks the current screen, `waitFor` waits for a
|
|
condition, and `extract` reads structured data. Each call has its own timeout
|
|
and model-call budget.
|
|
|
|
```ts title="tests/team.e2e.ts"
|
|
import { test } from 'e2e';
|
|
|
|
test('invite a teammate', async ({ agent }) => {
|
|
await agent.act('invite {email} as an editor', { params: { email: 'ada@example.test' } });
|
|
await agent.waitFor('the members list shows ada@example.test');
|
|
await agent.assert('ada@example.test is listed as an editor');
|
|
});
|
|
```
|
|
|
|
The built-in agent needs a configured model. Without one, the first
|
|
acquisition of the `agent` fixture records one run-level `MODEL_UNAVAILABLE`
|
|
and stops the run (exit 2). [Custom executors](/executors) can run `act` and
|
|
`assert` without a model; `waitFor` and `extract` still need one.
|
|
|
|
## Reference
|
|
|
|
| Method | Returns |
|
|
| --- | --- |
|
|
| [`agent.act`](#agent-act) | `Promise<ActResult>` |
|
|
| [`agent.assert`](#agent-assert) | `Promise<void>` |
|
|
| [`agent.waitFor`](#agent-waitfor) | `Promise<void>` |
|
|
| [`agent.extract`](#agent-extract) | `Promise<InferOutput<Schema>>` |
|
|
|
|
[Types](#types) holds the signatures. [Budgets](#budgets) lists each
|
|
method's default timeout, model calls, and action steps, and the rules for
|
|
lowering them. Every method takes `agent`: the
|
|
name of a configured agent (`agents.<name>`) that runs that one call in place
|
|
of the test's.
|
|
|
|
### agent.act
|
|
|
|
Plans and executes one bounded multi-action flow. A passing verdict resolves
|
|
with an `ActResult`. A failing or blocked one throws an
|
|
[`AgentError`](#agenterror) whose `code` and `blocked` flag say why.
|
|
|
|
```ts
|
|
await agent.act('invite {email} as an editor', { params: { email: 'ada@example.test' } });
|
|
|
|
const member = credentials.user('member');
|
|
const { summary, cache } = await agent.act('Sign in', {
|
|
params: { username: member.username, password: member.password },
|
|
});
|
|
```
|
|
|
|
- `instruction` is 1 through 8192 UTF-8 bytes after NFC normalization.
|
|
- The [`ActResult`](#types) carries the step's `summary`, the `modelCalls`
|
|
and `actions` it spent, and `cache`: how the [replay cache](/cache) took
|
|
part.
|
|
- The agent acts through the [action tools](#action-tools) and requests
|
|
screenshots itself through the
|
|
[pixel tools](#pixel-tools-screenshot-and-the-point-verbs).
|
|
|
|
| Option | Type | Default |
|
|
| --- | --- | --- |
|
|
| `params` | `AgentParams` | none |
|
|
| `timeout` | `number` | `config.timeout`, 120000 |
|
|
| `maxSteps` | `number` | `agents.<name>.maxSteps`, 25 |
|
|
| `maxModelCalls` | `number` | `agents.<name>.maxModelCalls`, 25 |
|
|
| `agent` | `string` | the test's agent |
|
|
| `onAction` | [`ActionHook`](#types) | none |
|
|
| `onBeforeAction` | [`BeforeActionHook`](#types) | none |
|
|
|
|
`params` values come from your code, never from the model. Wrap a value that
|
|
differs on every run in [`unique()`](#unique). The model sees a `Secret` as
|
|
its name and purpose only. Before the runner fills it, it checks the field
|
|
the model picked: an enabled editable input, and a password field for a
|
|
password. `locator.fill` runs no such check, since your code chose the field.
|
|
|
|
`onAction` runs after each action the step completes, once the screen
|
|
settles, and before the next action. That includes actions the
|
|
[replay cache](/cache) replays. The step waits for it, and its time counts
|
|
toward `timeout`. Use it to observe, never to drive the app. If it throws,
|
|
the step fails with a message that starts with `agent.act onAction failed`.
|
|
A check inside it never confirms the step's recording.
|
|
|
|
```ts
|
|
await agent.act('open settings and turn on dark mode', {
|
|
onAction: ({ index, action }) => app.screenshot(`${index} ${action}`),
|
|
});
|
|
```
|
|
|
|
Narrow on `event.action` to read an action's details: `point` for `tapAt`
|
|
and `hoverAt`, `from` and `to` for `dragAt`. Points are in the same space as
|
|
`screen.tapAt` and `boundingBox()`, rounded and clamped to the viewport.
|
|
Replay reports the points it dispatches on the current screen. Mobile
|
|
engines have no `hoverAt` or `dragAt`.
|
|
|
|
```ts
|
|
await agent.act('move the pin to Warsaw', {
|
|
onAction: (event) => {
|
|
if (event.action === 'dragAt') console.log(event.from, event.to);
|
|
},
|
|
});
|
|
```
|
|
|
|
`onBeforeAction` runs before each action, replayed ones included, with the
|
|
same `index` that action's `onAction` gets. Use it to capture the screen an
|
|
action started on. The step waits for it, and it does not wait for the screen
|
|
to settle first. If it throws, the step fails before the action runs, with a
|
|
message that starts with `agent.act onBeforeAction failed`. An action can still
|
|
fail after it, with no `onAction` call.
|
|
|
|
```ts
|
|
await agent.act('open settings and turn on dark mode', {
|
|
onBeforeAction: ({ index }) => app.screenshot(`before ${index}`),
|
|
});
|
|
```
|
|
|
|
A failed verdict is `ACTION_FAILED` unless the executor names a more specific
|
|
code, such as `LOCATOR_NOT_FOUND` or `AUTHENTICATION_FAILED`. A blocked
|
|
verdict means the step could not reach a product verdict. It carries one of
|
|
these codes:
|
|
|
|
| Blocked on | Codes |
|
|
| --- | --- |
|
|
| credentials | `AUTH_CREDENTIAL_UNAVAILABLE`, `AUTH_CREDENTIAL_INVALID`, `SECRET_UNAVAILABLE` |
|
|
| environment | `ENVIRONMENT_UNAVAILABLE`, `APP_UNREACHABLE`, `APP_ALREADY_RUNNING` |
|
|
| seed data | `SEED_DATA_MISSING` |
|
|
| test setup | `TEST_SETUP_FAILED`, `APP_NOT_OPEN`, `POLICY_DENIED` |
|
|
| automation | `MODEL_UNAVAILABLE`, `AUTOMATION_UNSUPPORTED`, `STEP_BUDGET_EXHAUSTED`, `STEP_TIMEOUT`, `CONTEXT_OVERFLOW`, `MODEL_REFUSED` |
|
|
|
|
The runner's own stops, and the checks on the call itself:
|
|
|
|
| Throws | When |
|
|
| --- | --- |
|
|
| `STEP_BUDGET_EXHAUSTED`, `STEP_TIMEOUT` | A budget or the deadline ran out. |
|
|
| `CANCELLED` | The runner cancelled the step. |
|
|
| `STEP_NO_CONCLUSION` | The turns ran out without a conclusion. |
|
|
| `CONTEXT_OVERFLOW` | A request exceeded the model's context window. |
|
|
| `MODEL_REFUSED` | The model refused the request. |
|
|
| `MODEL_PROVIDER_FAILED` | The provider's transport failed after its retries. |
|
|
| `MODEL_OUTPUT_INVALID` | A verdict outside the grammar. |
|
|
| `REPLAY_STALE` | Under a strict cache, the step's recording no longer replays. |
|
|
| `INVALID_ARGUMENT` | A malformed instruction, oversized `params`, an unknown option, an `onAction` or `onBeforeAction` that is not a function, or a budget above the configured limit. An unknown `agent` name fails before the step opens. |
|
|
| `UNSUPPORTED_CAPABILITY` | `schema` or `vision`, which `act` does not take. |
|
|
|
|
### agent.assert
|
|
|
|
Checks one fresh observation without polling by default.
|
|
|
|
```ts
|
|
await agent.assert('the dashboard shows a trial badge');
|
|
await agent.assert('the table lists {name}', { params: { name } });
|
|
```
|
|
|
|
- The judge sees the assertion, its `params`, the current observation with
|
|
its location, and the agent's `context`, never prior steps or `act` summaries. It answers `holds`,
|
|
`fails`, or `inconclusive`.
|
|
- When the subject appears more than once, such as a total in a summary and
|
|
on a pay button, every instance must agree: one that contradicts it fails.
|
|
Name the place (`the order summary total is $42.00`) to judge only that one.
|
|
- A claim about some item, such as `a todo is marked done`, holds when one
|
|
item matches.
|
|
|
|
| Option | Type | Default |
|
|
| --- | --- | --- |
|
|
| `explore` | `boolean` | `false` |
|
|
| `maxSteps` | `number`, requires `explore` | min(10, configured limit) |
|
|
| `maxModelCalls` | `number`, requires `explore` | min(6, configured limit) |
|
|
| `params` | `AgentParams` | none |
|
|
| `timeout` | `number` | `agents.<name>.judgmentTimeout`, 30000 |
|
|
| `screenshot` | `boolean` | `true` |
|
|
| `vision` | [`VisionMode`](#vision) | `false` |
|
|
| `agent` | `string` | the test's agent |
|
|
|
|
`screenshot` controls the evidence attached after the judgment, passing or
|
|
failing.
|
|
`vision` controls what the model is shown before it. No evidence is captured
|
|
with `screenshot: false`.
|
|
|
|
`params` follow the [`agent.act`](#agent-act) rules, and invalid ones throw
|
|
`INVALID_ARGUMENT`. A `Secret` param reaches the judge as its name, never its
|
|
value. To check that a secret's value is on screen, use a locator assertion.
|
|
|
|
| Throws | When |
|
|
| --- | --- |
|
|
| `ASSERTION_FAILED` | The judgment is false. |
|
|
| `ASSERTION_INCONCLUSIVE` | The screen did not show enough to decide. |
|
|
|
|
Both carry the explanation and, when permitted, a screenshot path. An
|
|
inconclusive judgment made from the tree alone adds a vision hint to its
|
|
message:
|
|
|
|
- The hint reads `pass vision: true when the answer is in pixels`. Until a
|
|
step of the attempt has received pixels, it is worded as a condition
|
|
(`if the engine captures pixels, ...`).
|
|
- There is no hint when the engine declares no screenshot capture, or an
|
|
earlier step's pixel request was degraded: `vision: true` could add no
|
|
evidence then.
|
|
|
|
<Note>
|
|
A custom `StepExecutor` judges the assertion itself, and `assert` carries the
|
|
executor's explanation alone, with no vision hint. It honors `screenshot`,
|
|
since the runner takes that evidence after the verdict. It receives `vision`
|
|
as `step.vision` when it declares `vision: true`; otherwise `vision` is
|
|
`UNSUPPORTED_CAPABILITY`. Its budget and timeout are under
|
|
[Budgets](#budgets).
|
|
</Note>
|
|
|
|
#### Viewport exploration
|
|
|
|
With `explore: true`, `agent.assert` can scroll the current screen and combine
|
|
evidence from several viewports. This helps when a list fits on a tablet but
|
|
needs scrolling on a phone.
|
|
|
|
```ts
|
|
await agent.assert('Settings lists HOS Settings, Account, Language and Theme', {
|
|
explore: true,
|
|
maxSteps: 10,
|
|
maxModelCalls: 6,
|
|
});
|
|
```
|
|
|
|
Exploration offers `observe`, `scroll`, `scroll_to`, and project tools declared
|
|
`mutates: false`. The built-in executor also offers `screenshot` when `vision`
|
|
is `true` or `'only'`; the default is tree-only. Other actions are rejected. The viewport stays
|
|
where exploration ends.
|
|
|
|
Defaults are 10 scroll actions and 6 model calls, each capped at the selected
|
|
agent's configured limit. Overrides may exceed the 10/6 defaults, up to the
|
|
configured limits. The deadline
|
|
defaults to `judgmentTimeout`. `maxSteps` or `maxModelCalls` without
|
|
`explore: true` throws `INVALID_ARGUMENT`. The built-in executor uses
|
|
`agents.<name>.model`, even when `judge` is configured.
|
|
|
|
A contradiction is `ASSERTION_FAILED`; insufficient evidence within the time
|
|
or action/model budget is `ASSERTION_INCONCLUSIVE`.
|
|
|
|
Use `waitFor` for a condition that changes over time, and `explore` for content
|
|
elsewhere on the current screen.
|
|
|
|
### agent.waitFor
|
|
|
|
Polls a natural-language condition until a judgment holds.
|
|
|
|
```ts
|
|
await agent.waitFor('the upload progress reports completion', { interval: 500 }); // default: 3000
|
|
```
|
|
|
|
- It observes and judges at once, and succeeds on the first judgment that
|
|
holds.
|
|
- After a false or inconclusive judgment it observes every 500 ms (or every
|
|
`interval`, when shorter) without a model call. It judges again only once
|
|
the screen or its location differs from the one last judged, pixels
|
|
included under `vision`. Under `vision: 'only'` a location change alone
|
|
does not count, since that mode is not shown the location.
|
|
- A change is judged as soon as it is seen when the screen had not changed
|
|
in the `interval` before it, or not at all since the wait began. While the
|
|
screen keeps changing (a progress counter, a clock), it judges at most once
|
|
per `interval` after the previous judgment.
|
|
- Judgments follow the [`agent.assert`](#agent-assert) rules: a value shown
|
|
in more than one place holds only once every instance agrees.
|
|
|
|
| Option | Type | Default |
|
|
| --- | --- | --- |
|
|
| `params` | `AgentParams` | none |
|
|
| `timeout` | `number` | `agents.<name>.judgmentTimeout`, 30000 |
|
|
| `interval` | `number` | 3000 |
|
|
| `maxModelCalls` | `number` | `agents.<name>.maxModelCalls`, 25 |
|
|
| `vision` | [`VisionMode`](#vision) | `false` |
|
|
| `agent` | `string` | the test's agent |
|
|
|
|
`interval` is the least time between two judgments of a screen that keeps
|
|
changing, 100 through 60000 ms.
|
|
`params` work as in [`agent.assert`](#agent-assert).
|
|
|
|
| Throws | When |
|
|
| --- | --- |
|
|
| `STEP_TIMEOUT` | The deadline expired. |
|
|
| `STEP_BUDGET_EXHAUSTED` | The budget ran out. |
|
|
| `INVALID_ARGUMENT` | An out-of-range option, invalid `params`, or an option `waitFor` does not take. |
|
|
|
|
`STEP_TIMEOUT` and `STEP_BUDGET_EXHAUSTED` carry the last judgment. When that
|
|
round was inconclusive, they add the same vision hint as `agent.assert`.
|
|
|
|
### agent.extract
|
|
|
|
Reads structured output from one fresh observation, validates it against
|
|
`schema`, and returns the validated value. It performs no app actions.
|
|
|
|
```ts
|
|
import { z } from 'zod';
|
|
|
|
const summary = await agent.extract('the workspace name and plan', {
|
|
schema: z.object({ name: z.string(), plan: z.string() }),
|
|
});
|
|
```
|
|
|
|
- When the schema has a JSON Schema projection (zod does), the model receives
|
|
its shape: types, fields, required fields, and allowed literals. Value rules
|
|
such as `min`, `max`, lengths, patterns, and formats stay with your schema,
|
|
which judges the value the model read.
|
|
- The model can answer that the screen does not show the data. That throws
|
|
`ASSERTION_INCONCLUSIVE` with what was missing, instead of returning a
|
|
placeholder such as `""` or `0`. To accept absence as an answer, say so in
|
|
the instruction and the schema (`'the phone number, or null when none is
|
|
shown'` with `z.string().nullable()`).
|
|
|
|
| Option | Type | Default |
|
|
| --- | --- | --- |
|
|
| `schema` | `StandardSchemaV1` | required |
|
|
| `timeout` | `number` | `agents.<name>.judgmentTimeout`, 30000 |
|
|
| `vision` | [`VisionMode`](#vision) | `false` |
|
|
| `agent` | `string` | the test's agent |
|
|
|
|
| Throws | When |
|
|
| --- | --- |
|
|
| `INVALID_ARGUMENT` | The schema is not Standard Schema v1. |
|
|
| `ASSERTION_INCONCLUSIVE` | The screen does not show the data. |
|
|
| `MODEL_OUTPUT_INVALID` | Validation still fails after the repair round. |
|
|
|
|
## Budgets
|
|
|
|
Model calls count every model request, repair rounds included. Action steps
|
|
count committed engine actions. The remaining test timeout caps every
|
|
timeout.
|
|
|
|
| Method | Model calls | Action steps | Default timeout |
|
|
| --- | ---: | ---: | --- |
|
|
| `act` | `agents.<name>.maxModelCalls`, 25 | `agents.<name>.maxSteps`, 25 | `config.timeout`, 120000 ms |
|
|
| `waitFor` | `agents.<name>.maxModelCalls`, 25 | 0 | `agents.<name>.judgmentTimeout`, 30000 ms |
|
|
| `extract` | 2 | 0 | `agents.<name>.judgmentTimeout`, 30000 ms |
|
|
| `assert` without exploration | 2 | 0 | `agents.<name>.judgmentTimeout`, 30000 ms |
|
|
| `assert` with exploration | min(6, configured limit) | min(10, configured limit) | `agents.<name>.judgmentTimeout`, 30000 ms |
|
|
|
|
- `actionTimeout` limits engine operations, not model calls.
|
|
- Per-call budgets (`maxSteps`, `maxModelCalls`) can only lower the
|
|
configured limits. They must be positive integers no larger than those
|
|
limits; a larger one is `INVALID_ARGUMENT`.
|
|
- All durations are in milliseconds.
|
|
|
|
Without exploration, the `assert` row describes the built-in agent. A custom
|
|
`executor` runs non-exploring assertions with the agent's `maxSteps` and
|
|
`maxModelCalls`, and `config.timeout` as the default deadline. Exploring
|
|
assertions use the capped defaults and `judgmentTimeout` with any executor.
|
|
For `vision` and `screenshot` there, see the Note under
|
|
[`agent.assert`](#agent-assert).
|
|
|
|
## Action tools
|
|
|
|
The built-in agent offers one tool per verb the target's engine declares, and
|
|
none for a verb it does not. A tool that targets a node takes its id from the
|
|
newest observation. Every tool returns what changed on screen and is
|
|
recorded for the [replay cache](/cache), with its node when it has one. On an engine that declares
|
|
`tapModifiers`, `tap`, `double_tap`, and `right_click` take `modifiers`
|
|
(`Shift`, `Control`, `Alt`, `Meta`, `ControlOrMeta`) to hold keys for the
|
|
click, as a Shift-click extends a selection.
|
|
|
|
| Tool | Engine action | What it does |
|
|
| --- | --- | --- |
|
|
| `tap({ target, modifiers? })` | `tap` | Taps or clicks one node: the gesture for a button, link, menu item, tab, checkbox, row, or field. |
|
|
| `double_tap({ target, modifiers? })` | `doubleTap` | Double-clicks one node, for an item that opens or enters editing on the second click. |
|
|
| `long_press({ target })` | `longPress` | Presses one node and holds, for a long-press menu or action. |
|
|
| `right_click({ target, modifiers? })` | `secondaryTap` | Right-clicks one node to open its context menu. |
|
|
| `hover({ target })` | `hover` | Moves the pointer over one node without pressing: hover menus, flyouts, tooltips, controls revealed on mouse-over. |
|
|
| `type({ target?, value, replace? })` | `fill`; without `target`, `keyboard` | Replaces the node's value; without a target, types into whatever has focus. |
|
|
| `type_secret({ target, name })` | `fill` | Fills a secret declared in the step's params. On a step that declares none, it fills nothing. |
|
|
| `press({ target?, key, times? })` | `press`; without `target`, `keyboard` | Sends one key to a node or to the focused control, up to 20 times in one call. Each press is one action; with a target, the first press addresses the node and the rest go to whatever has focus, so a repeated `Tab` walks on. |
|
|
| `select({ target, value })` | `selectOption` | Picks an option by its visible label. |
|
|
| `check({ target, checked? })` | `check` and `uncheck` | Sets a checkbox, switch, or radio to a state (`true` by default). A control already in that state is left alone, unlike a tap, which flips it. |
|
|
| `scroll({ direction, target?, times? })` | `swipe` | Scrolls the viewport or one scrollable node, up to 20 screens per call. |
|
|
| `scroll_to({ target })` or `scroll_to({ text, direction?, target? })` | `scrollIntoView`; with `text`, `swipe` | With `target` alone, brings one listed node into the viewport, for a node the tree lists below or above the fold. With `text`, pages the list (`target`, else the viewport) screen by screen in `direction` (down by default) until a node reading that text shows, then brings it in when the engine can. That is for a row deep in a long or windowed list the tree has not rendered yet. The node's label must be the text, or hold it with no part of a word on either side, so `Row 12` never stops at `Row 120`. When `target` names a list, the node must be inside it, so a heading elsewhere reading the same does not end the paging. One action however many screens, up to 800; a screen that stops moving or the cap is `LOCATOR_NOT_FOUND`. Each form is offered only with its action kind. |
|
|
| `drag({ target, to })` | `dragTo` | Drags one node and drops it on another: reordering, kanban columns, drop zones. |
|
|
| `upload({ target, files })` | `setInputFiles` | Attaches files to a file input, or to a visible control that opens a native file chooser. Paths are project-relative or name a `sample:` file; see the policy below. |
|
|
| `navigate({ url })` | `session.open` | Opens a URL or an app-relative path. |
|
|
| `back()` | `session.back` | Goes back once: browser history on a page, the in-app back on a device. |
|
|
| `resize_viewport({ width, height })` | `session.resize` | Sets the viewport size in CSS pixels, 200 to 4096 per side, to check a layout at another width. The size holds until the next resize. Engines with `session.resize`, such as web. |
|
|
| `switch_tab({ tab })` | `session.switchTab` | Shows another open tab, numbered from 1 in the order the tabs opened. Engines with `session.switchTab`, such as web. |
|
|
| `close_tab({ tab })` | `session.closeTab` | Closes an open tab, numbered as `switch_tab` counts them. The last open tab stays open. Engines with `session.closeTab`, such as web. |
|
|
| `observe()` | | Looks again and reports what changed since the last screen the model received. |
|
|
| `dismiss_keyboard()` | `keyboard.dismiss` | Hides an on-screen keyboard. Device engines only. |
|
|
|
|
`scroll_to({ text })` reads each screen once when it has visibly scrolled, and
|
|
otherwise re-reads it for up to a second, the same beat a held-still look
|
|
uses, before that screen counts as stopped. A list that appends when it
|
|
reaches the end keeps being paged. A single changing time value
|
|
(`12:05`) beside other text does not count as the list moving. On a device, the next observation
|
|
checks stability without waiting for a second scroll effect. Engines with
|
|
`scrollIntoView` still wait for the final adjustment that brings the matched
|
|
node into view. Replay follows the same rule.
|
|
|
|
`upload` takes the model's path, but the runner decides:
|
|
|
|
- Each path resolves against the project root, as in
|
|
`locator.setInputFiles`.
|
|
- A path outside the project (through a symlink too), or one that names a
|
|
hidden file or a file under a hidden directory such as `.env`, `.git`, or
|
|
`.e2e`, is refused with `POLICY_DENIED`.
|
|
- A path that names no regular file is `INVALID_ARGUMENT`, an ordinary action
|
|
failure the model reads and corrects.
|
|
- `sample:photo.png`, `sample:photo.jpg`, `sample:document.pdf`,
|
|
`sample:notes.txt`, `sample:table.csv`, and `sample:empty.txt` name small
|
|
files e2e ships, so the agent can try an upload when the project has no
|
|
file to give it. The tool lists them.
|
|
- Every refusal and the final allow is a `policy` event named `upload.path`
|
|
on the step. Nothing reaches the engine until every path passes.
|
|
|
|
On engines with a keyboard, including the browser and device engines, `type`
|
|
and `press` can omit `target` to act on the focused field. `type` then
|
|
inserts at the caret; `replace: true` clears first. The browser engine
|
|
rejects typing when no editable field has focus. It checks the focused
|
|
element inside shadow roots and frames too, so focus on a button there is
|
|
refused. When an action closes an on-screen keyboard, the result says so. On
|
|
a touch screen that tap was often spent on closing it, and the model acts on
|
|
the control again.
|
|
|
|
When the engine reports text selected inside the focused field, the screen
|
|
the model reads lists it as `selection="..."` after the value. The web engine
|
|
does for an input, a textarea, and a `contenteditable` editor; a secure field
|
|
reports none. The runner drops it from a field whose value or text holds a
|
|
registered secret. It also drops it from a field cut at its length limit when
|
|
the selection is not in the part shown. So selected characters of a secret
|
|
never reach the model. A `press` of `Shift+ArrowLeft` shows what it selected,
|
|
and `times` repeats it in one call. So the agent extends a selection to
|
|
exactly the word it wants in one more turn, before it formats or replaces it.
|
|
|
|
The tap variants are separate tools, not an option on `tap`, on purpose.
|
|
Offered as an optional `kind`, a model filled it on ordinary buttons (a
|
|
double-click on Save), and a double-click on a submit button submits twice.
|
|
A tool has to be chosen, and each one says what takes a plain tap instead.
|
|
|
|
Mobile engines declare `hover`, `doubleTap`, `longPress`, `check`,
|
|
`uncheck`, `dragTo`, and `session.back`. So a device offers `hover`,
|
|
`double_tap`, `long_press`, `check`, `drag`, and `back`. They declare no
|
|
`secondaryTap`, `selectOption`, `scrollIntoView`, or `setInputFiles`.
|
|
|
|
A tab the app opens on the web stays in the background. The result of the
|
|
action that opened it says `The app opened a new tab (tab 2 of 2). The
|
|
screen still shows tab 1; use switch_tab to see it.` While more than one
|
|
tab is open, every screen header names the tab (`tab 1 of 2`). `switch_tab`
|
|
shows another tab and sends its screen whole. `close_tab` closes one; when
|
|
it was the tab on screen, the screen returns to the tab that opened it, or
|
|
to the tab shown most recently when that one is closed too. Both replay
|
|
from the cache by the same tab number. A tab takes its number when its
|
|
first page loads, so when one action opens two tabs whose pages load in
|
|
another order on a later run, the two numbers swap.
|
|
|
|
## Pixel tools: screenshot and the point verbs
|
|
|
|
`act` reads the semantic tree and requests screenshots when it needs them;
|
|
the same model reads both. A screen with no listed interactive nodes, such
|
|
as a canvas, opens with a screenshot.
|
|
|
|
The built-in agent offers these tools:
|
|
|
|
| Tool | What it does |
|
|
| --- | --- |
|
|
| `screenshot()` | Returns a masked viewport screenshot and includes a fresh screenshot after every later action in the step. |
|
|
| `tap_at({ x, y })` | Taps the coordinates. On an engine that cannot, taps the listed control at the point by node id. |
|
|
| `hover_at({ x, y })` | Moves the pointer to the coordinates. On an engine that cannot, hovers the listed control at the point by node id. Needs the `hover` action or pointer kind. |
|
|
| `drag_at({ x, y, to_x, to_y })` | Drags from one point to the other, for a shape on a canvas or a pin on a map. On an engine that cannot, drags by node id when the points land on two different listed controls. Needs the `dragTo` action or pointer kind. |
|
|
| `type_at({ x, y, value, replace? })` | Replaces a listed input's value. For an unlisted input, taps to focus and types at the caret unless `replace` is set. Unlisted inputs require keyboard support. |
|
|
| `press_at({ x, y, key })` | Sends a key to the control at the point. If no control is listed, taps to focus and uses the engine's keyboard. |
|
|
| `select_at({ x, y, value })` | Selects an option by visible label in the control at the point. |
|
|
|
|
- Coordinates refer to the latest screenshot. The model must take a
|
|
screenshot before it uses a point tool, and should prefer node ids for
|
|
listed controls.
|
|
- `type_at`, `press_at`, and `select_at` check each point against the tree.
|
|
A point on a listed control gets the same policy checks and
|
|
[replay cache](/cache) recording as an action by id. A `tap_at` or
|
|
`hover_at` point replays inside the node under it.
|
|
- `tap_at` needs an engine whose `actions` or `pointerActions` include `tap`,
|
|
`hover_at` one whose include `hover`, and `drag_at` one whose include
|
|
`dragTo`. Without the pointer kind, the point must land on a listed
|
|
control, and the two `drag_at` points on two different listed controls.
|
|
Otherwise the call fails with `UNSUPPORTED_CAPABILITY`.
|
|
- A bare-point tap, hover, or drag replays only on a viewport of the recorded
|
|
size, or at the same place inside the listed node that contained it. A
|
|
drag places each of its two points that way.
|
|
- Once screenshots are in use, the post-action wait also detects pixel
|
|
changes, including canvas redraws.
|
|
- Screenshots have a maximum long side of 768 px. Older images are removed in
|
|
batches, keeping the newest two.
|
|
- A step that sent pixels records `visionInput: true` and
|
|
`metrics.pixelBytes`.
|
|
|
|
If semantic capture times out, an engine that proves masking independently
|
|
can return a fresh screenshot with an explicit unavailable-tree warning.
|
|
Every `act` observation with unavailable semantics includes that image and
|
|
warning. Earlier node ids are then invalid, and the next successful semantic
|
|
capture presents the whole tree again. The screenshot shows the current
|
|
viewport; an unavailable tree does not prove that a control or message is
|
|
absent.
|
|
|
|
If semantic capture fails, trace replay stops. The step cannot record a new
|
|
trace entry, even if the tree recovers later.
|
|
|
|
## Vision
|
|
|
|
`assert`, `waitFor`, and `extract` accept `vision`; `act` requests its own
|
|
screenshots through the [pixel tools](#pixel-tools-screenshot-and-the-point-verbs).
|
|
A screenshot goes to the model that asked for it, so it must accept images:
|
|
the judge (else `model`) for a judgment, `model` for `act`.
|
|
|
|
```ts
|
|
await agent.assert('no overlay covers the checkout form', { vision: 'only' }); // default: false
|
|
```
|
|
|
|
| `vision` | Evidence sent to the judge |
|
|
| --- | --- |
|
|
| `false`, the default | Semantic tree |
|
|
| `true` | Semantic tree and a masked screenshot |
|
|
| `'only'` | Masked screenshot only, no tree or location |
|
|
|
|
Use `'only'` for visual checks, such as whether an overlay covers a form or a
|
|
chart looks correct. Screenshots show the current viewport, so scroll to the
|
|
relevant content first.
|
|
|
|
The browser engine's semantic tree leaves out the content of a
|
|
[frame from another site](/browser#reach-into-an-iframe). To judge that
|
|
content, use `true` or `'only'`.
|
|
|
|
Pixels are withheld, and the tree is sent alone, when:
|
|
|
|
- the engine masked fewer regions than the secure fields it observed
|
|
(`MASKING_UNPROVEN`), or
|
|
- the engine cannot capture pixels (`UNSUPPORTED_CAPABILITY`).
|
|
|
|
In `'only'` mode those cases fail with `POLICY_DENIED` instead.
|
|
|
|
With `vision: true` or `'only'`, a recoverable semantic-capture failure can
|
|
use a fresh screenshot when the engine proves masking independently. With
|
|
`vision: false`, the capture failure stops the judgment before any model
|
|
call: a missing tree cannot support a judgment about an empty screen.
|
|
|
|
If capture fails before a judgment, the report records zero model calls and
|
|
omits `observationRevision` and `explanation`. The step error describes the
|
|
capture failure. An answer the closed grammar rejects is a `schema` event on
|
|
the step and earns one repair round. When the repaired answer is rejected
|
|
too, the step fails with `MODEL_OUTPUT_INVALID`, and its `explanation` is the
|
|
rejection.
|
|
|
|
## Types
|
|
|
|
```ts
|
|
export interface Agent {
|
|
/** Plans and executes a bounded multi-action flow; resolves with what the step did. */
|
|
act(instruction: string, options?: ActOptions): Promise<ActResult>;
|
|
/** Polls a natural-language condition until true or timed out. */
|
|
waitFor(condition: string, options?: WaitForOptions): Promise<void>;
|
|
/** Extracts and validates structured screen data. */
|
|
extract<Schema extends StandardSchemaV1>(
|
|
instruction: string,
|
|
options: ExtractOptions<Schema>,
|
|
): Promise<StandardSchemaV1.InferOutput<Schema>>;
|
|
/** Judges a natural-language assertion against a fresh observation. */
|
|
assert(assertion: string, options?: AssertOptions): Promise<void>;
|
|
}
|
|
|
|
/**
|
|
* Parameters and budgets for one `act` call.
|
|
*/
|
|
export interface ActOptions extends AgentOption {
|
|
/**
|
|
* JSON-safe values the instruction refers to, at most 64 KiB and 32 levels
|
|
* deep. A `Secret` reaches the model by name only; the runner fills it. A
|
|
* value wrapped in `unique()` is different on every run, and the replay
|
|
* cache records a slot for it instead of the value.
|
|
*/
|
|
params?: AgentParams;
|
|
/** Step deadline in milliseconds; defaults to `config.timeout`. */
|
|
timeout?: number;
|
|
/** Action budget; defaults to `agents.<name>.maxSteps` and can only lower it. */
|
|
maxSteps?: number;
|
|
/** Model-call budget; defaults to `agents.<name>.maxModelCalls` and can only lower it. */
|
|
maxModelCalls?: number;
|
|
/**
|
|
* Runs after each action the step completes, once the screen settles, and
|
|
* before the next one. Awaited, and its time counts toward `timeout`.
|
|
* Observe only: drive nothing from it. A throw fails the step.
|
|
*/
|
|
onAction?: ActionHook;
|
|
/**
|
|
* Runs before each action, replayed ones included, and the step waits for
|
|
* it. Observe only: drive nothing from it. A throw fails the step before
|
|
* the action runs.
|
|
*/
|
|
onBeforeAction?: BeforeActionHook;
|
|
}
|
|
|
|
/** The `onBeforeAction` hook of an `act` call. */
|
|
export type BeforeActionHook = (event: BeforeActionEvent) => void | Promise<void>;
|
|
|
|
/** The `onAction` hook of an `act` call. */
|
|
export type ActionHook = (event: ActionEvent) => void | Promise<void>;
|
|
|
|
/** The configured agent (`agents.<name>`) one call runs with, in place of the test's. */
|
|
export interface AgentOption {
|
|
agent?: string;
|
|
}
|
|
|
|
/** `assert` options: one judgment, or opt-in bounded viewport exploration. */
|
|
export interface AssertOptions extends VisionOption, AgentOption {
|
|
/** Look around by scrolling before judging; defaults to false. */
|
|
explore?: boolean;
|
|
/** Scroll budget; requires explore, defaults to min(10, configured maxSteps). */
|
|
maxSteps?: number;
|
|
/** Model-call budget; requires explore, defaults to min(6, configured maxModelCalls). */
|
|
maxModelCalls?: number;
|
|
/**
|
|
* Values the assertion refers to as `{name}`, with the same rules as `act`
|
|
* params. A `Secret` reaches the judge by name only, never its value.
|
|
*/
|
|
params?: AgentParams;
|
|
/** Deadline in milliseconds; defaults to the agent's `judgmentTimeout`, 30000. */
|
|
timeout?: number;
|
|
/** Attach a redacted screenshot to the step; on by default. */
|
|
screenshot?: boolean;
|
|
}
|
|
|
|
/** `waitFor` options: a judgment on each screen change, at most once per `interval` while it keeps changing, until `timeout`. */
|
|
export interface WaitForOptions extends VisionOption, AgentOption {
|
|
/**
|
|
* Values the condition refers to as `{name}`, with the same rules as `act`
|
|
* params. A `Secret` reaches the judge by name only, never its value.
|
|
*/
|
|
params?: AgentParams;
|
|
/** Deadline in milliseconds; defaults to the agent's `judgmentTimeout`, 30000. */
|
|
timeout?: number;
|
|
/** Least time between two judgments of a screen that keeps changing, in milliseconds; 100 through 60000, default 3000. */
|
|
interval?: number;
|
|
/** Judgment budget; defaults to `agents.<name>.maxModelCalls` and can only lower it. */
|
|
maxModelCalls?: number;
|
|
}
|
|
|
|
/** `extract` options: one extraction plus one repair round, validated against `schema`. */
|
|
export interface ExtractOptions<Schema extends StandardSchemaV1> extends VisionOption, AgentOption {
|
|
/** Any Standard Schema v1 validator; the output is validated against it, with one repair round. */
|
|
schema: Schema;
|
|
/** Deadline in milliseconds; defaults to the agent's `judgmentTimeout`, 30000. */
|
|
timeout?: number;
|
|
}
|
|
|
|
export interface VisionOption {
|
|
/** What the judge is shown; `false` by default. */
|
|
vision?: VisionMode;
|
|
}
|
|
|
|
/**
|
|
* Evidence for `assert`, `waitFor`, and `extract`.
|
|
*
|
|
* - `false`: the tree.
|
|
* - `true`: the tree and a screenshot, on every call.
|
|
* - `'only'`: the screenshot, and not the tree.
|
|
*
|
|
* When pixels cannot be safely captured, `true` falls back to the tree;
|
|
* `'only'` fails with `POLICY_DENIED`.
|
|
*/
|
|
export type VisionMode = boolean | 'only';
|
|
|
|
/** What one passing `act` step did, as the report records it. */
|
|
export interface ActResult {
|
|
/** The executor's one-line account of the step, or the replay's when the cache finished it. */
|
|
readonly summary: string;
|
|
/** How the replay cache took part; absent when caching is off for the step. */
|
|
readonly cache?: StepCacheInfo;
|
|
/** Model calls the step spent; 0 when a cached replay finished it. */
|
|
readonly modelCalls: number;
|
|
/** Grammar actions and mutating tool calls the step performed. */
|
|
readonly actions: number;
|
|
}
|
|
|
|
/** The name `onAction` reports a committed grammar action under, e.g. `tap`, `navigate`, or `dragAt`. */
|
|
export type ActionName =
|
|
| 'tap' | 'doubleTap' | 'longPress' | 'secondaryTap' | 'hover' | 'scrollTo'
|
|
| 'type' | 'typeSecret' | 'press' | 'select' | 'check' | 'upload' | 'drag'
|
|
| 'scroll' | 'scrollUntil' | 'navigate' | 'back' | 'resize' | 'typeText'
|
|
| 'pressKey' | 'dismissKeyboard' | 'tapAt' | 'hoverAt' | 'dragAt';
|
|
|
|
/**
|
|
* The fields `onAction` adds for an action, by action name; other actions add
|
|
* none. Points are in the same space as `screen.tapAt` and `boundingBox()`,
|
|
* rounded and clamped to the viewport.
|
|
*/
|
|
export interface ActionDetails {
|
|
/** The dispatched point. */
|
|
readonly tapAt: { readonly point: Point };
|
|
/** The dispatched point. Mobile engines have no `hoverAt`. */
|
|
readonly hoverAt: { readonly point: Point };
|
|
/** The dispatched start and end. Mobile engines have no `dragAt`. */
|
|
readonly dragAt: { readonly from: Point; readonly to: Point };
|
|
}
|
|
|
|
/** What every `onAction` event carries. */
|
|
interface ActionEventBase {
|
|
/** 1-based position in the step's action budget; a failed attempt uses a number too. */
|
|
readonly index: number;
|
|
/** What the action did, secrets redacted. */
|
|
readonly summary: string;
|
|
/** `model` when the executor chose the action, `replay` when the cache replayed it. */
|
|
readonly source: 'model' | 'replay';
|
|
}
|
|
|
|
/**
|
|
* One completed action of an `act` step, as `onAction` receives it. Narrow on
|
|
* `action` to read its {@link ActionDetails}; a mutating project tool reads
|
|
* `tool:<name>`.
|
|
*/
|
|
export type ActionEvent = ActionEventBase &
|
|
(
|
|
| { readonly action: 'tapAt' | 'hoverAt'; readonly point: Point }
|
|
| { readonly action: 'dragAt'; readonly from: Point; readonly to: Point }
|
|
| { readonly action: Exclude<ActionName, keyof ActionDetails> | `tool:${string}` }
|
|
);
|
|
|
|
/** One action of an `act` step about to run, as `onBeforeAction` receives it. */
|
|
export interface BeforeActionEvent {
|
|
/** The same `index` the action's `onAction` event carries. */
|
|
readonly index: number;
|
|
/** `model` when the executor chose the action, `replay` when the cache replays it. */
|
|
readonly source: 'model' | 'replay';
|
|
}
|
|
|
|
/**
|
|
* How the replay cache took part in one agent step. `self-finalized` means
|
|
* the full trace replayed and the step passed with zero model calls;
|
|
* `agent-concluded` means a replayed prefix handed the step to the executor;
|
|
* `missed` means the executor ran the step from the top.
|
|
*/
|
|
export interface StepCacheInfo {
|
|
mode: 'self-finalized' | 'agent-concluded' | 'missed';
|
|
/** The miss or hand-off reason token; absent on `self-finalized`. */
|
|
reason?: TraceReplayMissReason | ReplayHandOffReason;
|
|
/**
|
|
* On a `gap` at a typed value: the rule that made the value this run's
|
|
* data rather than the flow's. Absent for a gap at a project tool.
|
|
*/
|
|
derived?: DerivedReason;
|
|
/**
|
|
* Why a passing step that ran live was still not recorded, when the cause
|
|
* is the call itself: `param-collision` when a `unique()` value equals, is
|
|
* spelled inside, or is the encoded form of another param's value, so the
|
|
* recording could not tell which param a recorded input came from.
|
|
*/
|
|
notRecorded?: 'param-collision';
|
|
replayedActions: number;
|
|
totalActions: number;
|
|
/** The step's cache key, which names its entry file (`<cache.dir>/<entry>.json` in the file store). */
|
|
entry?: string;
|
|
/**
|
|
* Why the replay missed or handed off, past what the reason token says:
|
|
* the recorded action it stopped at and what went wrong there, or the
|
|
* screen the recording started on against the one the step did.
|
|
*/
|
|
detail?: string;
|
|
/** Configured model when the consulted entry was recorded; absent on older entries. */
|
|
recordedBy?: { readonly provider: string; readonly model: string };
|
|
}
|
|
|
|
type TraceReplayMissReason = 'retry' | 'no-entry' | 'invalid-entry' | 'truncated' | 'wrong-context';
|
|
export type ReplayHandOffReason =
|
|
| 'gap'
|
|
| 'target-not-found'
|
|
| 'target-ambiguous'
|
|
| 'viewport-changed'
|
|
| 'action-failed'
|
|
| 'action-uncertain'
|
|
| 'end-mismatch';
|
|
export type DerivedReason = 'pixels' | 'whole-node' | 'minted-token' | 'date';
|
|
|
|
/** Values an `act` instruction refers to. */
|
|
export type AgentParams = Readonly<Record<string, AgentParam>>;
|
|
|
|
/** A JSON-safe value, a `Secret`, or a `Unique` string. */
|
|
export type AgentParam =
|
|
| JsonPrimitive
|
|
| Secret
|
|
| Unique
|
|
| readonly AgentParam[]
|
|
| { readonly [key: string]: AgentParam };
|
|
```
|
|
|
|
Each `StepCacheInfo.reason` is explained under
|
|
[Why a step was not replayed](/cache#why-a-step-was-not-replayed).
|
|
|
|
`Unique` is what [`unique()`](#unique) returns.
|
|
|
|
## unique
|
|
|
|
Marks an `act` param whose value differs on every run. The
|
|
[replay cache](/cache) records a slot instead of the value, and substitutes
|
|
this run's value on replay.
|
|
|
|
```ts
|
|
await agent.act('create a company named {name}', {
|
|
params: { name: unique(`Acme ${Date.now()}`) },
|
|
});
|
|
```
|
|
|
|
```ts
|
|
import { unique } from 'e2e';
|
|
|
|
/** A string param that differs on every run; the model sees `value`, the replay cache a slot. */
|
|
export interface Unique {
|
|
readonly value: string;
|
|
}
|
|
|
|
/**
|
|
* Marks a value that is different on every run. Mark values the flow types
|
|
* or looks for, not choices that steer it: two runs whose `unique()` values
|
|
* differ replay the same recording.
|
|
*/
|
|
export function unique(value: string): Unique;
|
|
```
|
|
|
|
| Throws | When |
|
|
| --- | --- |
|
|
| `INVALID_ARGUMENT` | The value is not a string, is empty or only whitespace, or contains the placeholder text `{{param:`. |
|
|
|
|
## AgentError
|
|
|
|
Every failure is an `AgentError` with a closed `code`, an `explanation`, an
|
|
optional `screenshot` path, and a `blocked` flag. Check it with
|
|
`isAgentError(error)` from `e2e`, not `instanceof`. The class and
|
|
every code are in [Errors](/reference/errors#agenterror).
|