mirror of
https://github.com/tester-army/e2e.git
synced 2026-11-22 06:16:50 +08:00
Co-authored-by: Oskar Kwaśniewski <oskar@okwasniewski.com>
169 lines
6.3 KiB
Plaintext
169 lines
6.3 KiB
Plaintext
---
|
|
title: Project tools
|
|
description: Add tools for test APIs, data setup, and device operations.
|
|
"og:image": "https://e2e.com/docs/images/og/tools.png?v=8747047102"
|
|
"twitter:image": "https://e2e.com/docs/images/og/tools.png?v=8747047102"
|
|
---
|
|
|
|
A project tool is an AI SDK tool that the agent can call during a goal, such
|
|
as a request to your test API or a data setup step.
|
|
|
|
## Add a tool
|
|
|
|
Wrap the tool with `defineTool` and put it in the agent's `tools`:
|
|
|
|
```ts title="e2e.config.ts"
|
|
import type { E2EConfig } from 'e2e';
|
|
import { defineTool } from 'e2e/agent';
|
|
import { web } from '@e2e-dev/web';
|
|
import { gateway, tool } from 'ai';
|
|
import { z } from 'zod';
|
|
|
|
const APP_URL = process.env.APP_URL ?? 'http://localhost:3000';
|
|
|
|
const seedCart = defineTool(
|
|
tool({
|
|
description: 'Seed the cart with a SKU via the store test API',
|
|
inputSchema: z.object({ sku: z.string() }),
|
|
execute: async ({ sku }) => {
|
|
const response = await fetch(`${APP_URL}/api/test/cart?sku=${encodeURIComponent(sku)}`);
|
|
if (!response.ok) throw new Error(`Cart seeding failed: HTTP ${response.status}`);
|
|
return response.text();
|
|
},
|
|
}),
|
|
{ mutates: true },
|
|
);
|
|
|
|
export default {
|
|
targets: [{ engine: web(), app: { url: APP_URL } }],
|
|
agents: { default: { model: gateway('openai/gpt-6-luna-fast'), tools: { seedCart } } },
|
|
} satisfies E2EConfig;
|
|
```
|
|
|
|
The agent can now seed a cart before it checks out:
|
|
|
|
```ts
|
|
await agent.act('seed the cart with SKU {sku}, then check out', {
|
|
params: { sku: 'SHIRT-BLUE-M' },
|
|
});
|
|
await expect(screen.getByText('Order confirmed')).toBeVisible();
|
|
```
|
|
|
|
<Note>
|
|
The config load fails with `INVALID_CONFIG` for a tool named like a
|
|
grammar verb (`tap`, `screenshot`) or a tool the harness adds:
|
|
`complete_step`, `locate`, `run_code`, `test_lines`, `start_recording`,
|
|
`stop_recording`, or `report_finding`. It also fails for a value
|
|
`defineTool` didn't build, and for `tools` on an entry with a custom
|
|
`executor`.
|
|
</Note>
|
|
|
|
## Declare what a tool does
|
|
|
|
`defineTool` takes the AI SDK tool and these annotations:
|
|
|
|
| Annotation | Meaning |
|
|
| --- | --- |
|
|
| `mutates: true` | Changes state. The call takes a slot of the step's action budget before it runs, failed or not, and runs in order with the other actions. The [replay cache](/cache) can't replay it, so the agent runs that part of the step live. |
|
|
| `mutates: false` | Read-only. Runs without an action slot and may observe the screen or read node properties. |
|
|
| `platforms: ['ios', 'android']` | Offers the tool only on those platforms. Omit it to offer the tool on every target. |
|
|
|
|
## Observe from a tool
|
|
|
|
A read-only tool can observe the redacted screen through `getToolContext`.
|
|
This example saves a masked screenshot as report evidence when pixels are
|
|
available:
|
|
|
|
```ts
|
|
import { randomUUID } from 'node:crypto';
|
|
import { defineTool, getToolContext } from 'e2e/agent';
|
|
import { tool } from 'ai';
|
|
import { z } from 'zod';
|
|
|
|
const captureEvidence = defineTool(
|
|
tool({
|
|
description: 'Save a screenshot as evidence in the report',
|
|
inputSchema: z.object({}),
|
|
execute: async (_input, executionOptions) => {
|
|
const context = getToolContext(executionOptions);
|
|
const observation = await context.observe({ pixels: true });
|
|
const pixels = observation.pixels;
|
|
if (pixels === undefined) {
|
|
return { withheld: observation.pixelsWithheld ?? 'UNSUPPORTED_CAPABILITY' };
|
|
}
|
|
const id = await context.attachScreenshot(pixels, `evidence-${randomUUID()}`);
|
|
return { evidence: id };
|
|
},
|
|
}),
|
|
{ mutates: false },
|
|
);
|
|
```
|
|
|
|
- `observe()` returns a redacted text snapshot. `tree: true` adds the node
|
|
tree and `pixels: true` a masked screenshot, under the built-in agent's
|
|
rules.
|
|
- In a `mutates: true` tool, `observe()` fails with `POLICY_DENIED`.
|
|
Observe in a separate read-only tool.
|
|
- `attachScreenshot(pixels, label)` adds the image to the step's artifacts
|
|
and returns its artifact id. A reused label overwrites the earlier image.
|
|
|
|
## Read a node's computed style
|
|
|
|
In a read-only tool, use `context.read` to measure a listed node's rendered
|
|
style on the web engine:
|
|
|
|
```ts
|
|
const context = getToolContext(executionOptions);
|
|
await context.observe();
|
|
const result = await context.read({ id: nodeId }, {
|
|
style: ['color', 'font-weight', 'visibility', 'opacity'],
|
|
});
|
|
return result.style;
|
|
```
|
|
|
|
Pass the node's ID from the newest observation. The result contains only
|
|
requested properties, with secret values redacted as in the tree.
|
|
Values use the browser's computed CSS format, such as `rgb(255, 0, 0)` for red.
|
|
|
|
Request properties from the [computed-style allow-list](/reference/engine#rendered-node-properties).
|
|
Custom properties, generated content, selectors, and page code are rejected.
|
|
|
|
Reads use the step's deadline and run in order with observations and actions,
|
|
without taking an action slot. They measure the current value on the original
|
|
element, and do not refresh the observation or wait for a value to change.
|
|
Computed `visibility` and `opacity` describe the element's CSS, not whether
|
|
another element covers it or an ancestor makes it invisible.
|
|
|
|
| Error | When |
|
|
| --- | --- |
|
|
| `POLICY_DENIED` | A `mutates: true` tool calls `read`, or the engine has no `readStyle` method, as with the mobile engine. |
|
|
| `INVALID_ARGUMENT` | The options or property names are invalid. |
|
|
| `LOCATOR_NOT_FOUND` | No semantic observation exists, the latest one does not list the ID, or the element detached. Observe again before reading. |
|
|
| `STEP_TIMEOUT` | The read exhausted the step's remaining time, including an engine timeout at the deadline. Catching the read error cannot make the step pass. |
|
|
|
|
Use `requires: ['computed-style']` on a test to skip targets without style reads.
|
|
|
|
## Device toolkits
|
|
|
|
`@e2e-dev/mobile` has a ready tool pack, `mobileTools`, with `open_app`,
|
|
`swipe`, and `alert`. Spread it to add your own tools beside it:
|
|
|
|
```ts
|
|
tools: { ...mobileTools(iphone, pixel), seedCart },
|
|
```
|
|
|
|
See [Agent tools](/mobile#agent-tools) for the setup. A tool of your own
|
|
that needs a raw driver session gets the running device's client from
|
|
[`unstable_deviceSession`](/reference/mobile#unstable_devicesession).
|
|
|
|
## Next
|
|
|
|
<CardGroup cols={2}>
|
|
<Card title="Custom executors" icon="gears" href="/executors">
|
|
Replace the prompt or decision loop.
|
|
</Card>
|
|
<Card title="How agent steps work" icon="stairs" href="/agent-steps">
|
|
What the model sees and what it may do.
|
|
</Card>
|
|
</CardGroup>
|