mirror of
https://github.com/tester-army/e2e.git
synced 2026-11-22 18:19:48 +08:00
* feat!: the app is declared on its target; engines only drive it
The app under test moves from the engine's options to `targets: [{ engine,
app }]`: `url`, `bundleId`, `appPath`, `identity`, `environment`,
`launchArguments`, `permissions`, `command`, and `readyUrl`. `web()` and
`mobile()` refuse every moved option in one `INVALID_CONFIG` naming the
target block; `defineEngine({ app })` is gone and engines check what their
platform needs in `validateApp`. `requires: ['browser' | 'native-app']` is
derived from the target's app. Replay cache and session keys are byte for
byte what they were, pinned by target-identity.test.ts.
Dependency services are not part of this version: `services` anywhere, and
`web({ services })`, fail saying so. `app.command` on port 0 without a
command, and `reuseExisting` on a free port, are refused. An `e2e mcp`
open cancelled or disconnected mid-start stops what it started, and a hung
shared start is abandoned once no session waits on it.
* fix: review follow-ups: cancelled opens, abandoned starts, null targets, stale docs
* refactor: drop the moved-option messages; old app keys are unknown keys
* test: give the replaced-nodes suite a budget that survives a loaded runner
* test: size the replaced-nodes hook budget to its seven tests
151 lines
5.3 KiB
Plaintext
151 lines
5.3 KiB
Plaintext
---
|
|
title: Project tools
|
|
description: Add tools for test APIs, data setup, and device operations.
|
|
---
|
|
|
|
Put tools in an agent's `tools` so the agent can call your test API or perform
|
|
other project-specific operations during a goal. Wrap each AI SDK tool with
|
|
`defineTool` to declare whether it changes state and which platforms support it.
|
|
|
|
```ts title="e2e.config.ts"
|
|
import type { E2EConfig } from 'e2e';
|
|
import { defineTool } from 'e2e/agent';
|
|
import { web } from '@e2e-dev/web';
|
|
import { gateway, tool } from 'ai';
|
|
import { z } from 'zod';
|
|
|
|
const APP_URL = process.env.APP_URL ?? 'http://localhost:3000';
|
|
|
|
const seedCart = defineTool(
|
|
tool({
|
|
description: 'Seed the cart with a SKU via the store test API',
|
|
inputSchema: z.object({ sku: z.string() }),
|
|
execute: async ({ sku }) => {
|
|
const response = await fetch(`${APP_URL}/api/test/cart?sku=${encodeURIComponent(sku)}`);
|
|
if (!response.ok) throw new Error(`Cart seeding failed: HTTP ${response.status}`);
|
|
return response.text();
|
|
},
|
|
}),
|
|
{ mutates: true },
|
|
);
|
|
|
|
export default {
|
|
targets: [{ engine: web(), app: { url: APP_URL } }],
|
|
agents: { default: { model: gateway('openai/gpt-6-luna-fast'), tools: { seedCart } } },
|
|
} satisfies E2EConfig;
|
|
```
|
|
|
|
The agent can now seed a cart before checking out:
|
|
|
|
```ts
|
|
await agent.act('seed the cart with SKU {sku}, then check out', {
|
|
params: { sku: 'SHIRT-BLUE-M' },
|
|
});
|
|
```
|
|
|
|
Built-in tool names are reserved. Loading the config fails with
|
|
`INVALID_CONFIG` for a project tool named like a grammar verb (`tap`,
|
|
`screenshot`), a tool the harness adds (`complete_step` in every step,
|
|
`locate`, `start_recording`, and `stop_recording` in an `e2e mcp` session,
|
|
`report_finding` under `e2e explore`), and for a value `defineTool` did not
|
|
build. Tools belong to
|
|
the built-in agent: an entry with a custom `executor` cannot also set
|
|
`tools`.
|
|
|
|
## Declare what a tool does
|
|
|
|
`defineTool` takes the AI SDK tool and these annotations:
|
|
|
|
| Annotation | Meaning |
|
|
| --- | --- |
|
|
| `mutates: true` | Changes state. The call counts against the step's action budget and runs in order with the other actions. The [replay cache](/cache) cannot replay it, so the agent runs that part of the step live. |
|
|
| `mutates: false` | Read-only. Runs without an action slot and may observe the screen. |
|
|
| `platforms: ['ios', 'android']` | Offers the tool only on those platforms. Omit this to offer it on every target. |
|
|
|
|
A failed call still consumes its action slot. The runner checks the budget
|
|
before dispatch, including when the model requests several calls at once.
|
|
|
|
## Observe from a tool
|
|
|
|
A read-only tool can observe the redacted screen. This example saves a
|
|
masked screenshot as report evidence when pixels are available:
|
|
|
|
```ts
|
|
import { randomUUID } from 'node:crypto';
|
|
import { defineTool, getToolContext } from 'e2e/agent';
|
|
import { tool } from 'ai';
|
|
import { z } from 'zod';
|
|
|
|
const captureEvidence = defineTool(
|
|
tool({
|
|
description: 'Save a screenshot as evidence in the report',
|
|
inputSchema: z.object({}),
|
|
execute: async (_input, executionOptions) => {
|
|
const context = getToolContext(executionOptions);
|
|
const observation = await context.observe({ pixels: true });
|
|
const pixels = observation.pixels;
|
|
if (pixels === undefined) {
|
|
return { withheld: observation.pixelsWithheld ?? 'UNSUPPORTED_CAPABILITY' };
|
|
}
|
|
const id = await context.attachScreenshot(pixels, `evidence-${randomUUID()}`);
|
|
return { evidence: id };
|
|
},
|
|
}),
|
|
{ mutates: false },
|
|
);
|
|
```
|
|
|
|
`observe()` returns a redacted text snapshot. Request `tree: true` for the
|
|
node tree or `pixels: true` for a masked screenshot. The built-in agent's
|
|
restrictions apply, including no pixels
|
|
[after a secret fill](/security#screenshots-after-a-secret-fill).
|
|
|
|
Observe in a separate read-only tool. A mutating tool already occupies the
|
|
action queue, so observing from inside it would wait on its own operation.
|
|
|
|
`attachScreenshot(pixels, label)` adds the image to the step's artifacts and
|
|
returns its artifact id. Use a unique label for each capture to avoid
|
|
overwriting an earlier image.
|
|
|
|
## Device toolkits
|
|
|
|
`@e2e-dev/mobile` ships a ready tool pack for its engine. Pass every device
|
|
engine to one `mobileTools` call, and spread the pack to add your own tools
|
|
beside it:
|
|
|
|
```ts
|
|
import type { E2EConfig } from 'e2e';
|
|
import { mobile } from '@e2e-dev/mobile';
|
|
import { mobileTools } from '@e2e-dev/mobile/tools';
|
|
import { gateway } from 'ai';
|
|
|
|
const iphone = mobile({ platform: 'ios' });
|
|
const pixel = mobile({ platform: 'android' });
|
|
|
|
export default {
|
|
targets: [
|
|
{ name: 'iphone', engine: iphone, app: { bundleId: 'Settings' } },
|
|
{ name: 'pixel', engine: pixel, app: { bundleId: 'com.android.settings' } },
|
|
],
|
|
agents: {
|
|
default: {
|
|
model: gateway('openai/gpt-6-luna-fast'),
|
|
tools: { ...mobileTools(iphone, pixel) /* , ...your own tools */ },
|
|
},
|
|
},
|
|
} satisfies E2EConfig;
|
|
```
|
|
|
|
The pack adds `open_app`, `swipe`, and `alert` on device targets.
|
|
Screenshots and point actions come from the engine's built-in tools.
|
|
See the [Mobile engine reference](/reference/mobile#agent-tools).
|
|
|
|
<CardGroup cols={2}>
|
|
<Card title="Custom executors" icon="gears" href="/executors">
|
|
Replace the prompt or decision loop.
|
|
</Card>
|
|
<Card title="How agent steps work" icon="stairs" href="/agent-steps">
|
|
What the model sees and what it may do.
|
|
</Card>
|
|
</CardGroup>
|