mirror of
https://github.com/tester-army/e2e.git
synced 2026-11-22 18:19:48 +08:00
190 lines
7.2 KiB
Plaintext
190 lines
7.2 KiB
Plaintext
---
|
|
title: Agents and personas
|
|
description: Configure several agents in one suite, pin tests to them, and run one flow as each persona.
|
|
"og:image": "https://e2e.com/docs/images/og/agents.png?v=94dc88725f"
|
|
"twitter:image": "https://e2e.com/docs/images/og/agents.png?v=94dc88725f"
|
|
---
|
|
|
|
import { Video } from './snippets/video.jsx';
|
|
|
|
An agent is a named entry in `agents` with its own model, instructions,
|
|
and tools. Tests use `default` unless you select another agent. Name more
|
|
agents to compare models or to run one flow as several personas.
|
|
|
|
## Define named agents
|
|
|
|
```ts title="e2e.config.ts"
|
|
import type { E2EConfig } from 'e2e';
|
|
import { web } from '@e2e-dev/web';
|
|
import { gateway } from 'ai';
|
|
|
|
const vocabulary = 'Plans are called tiers. Billing lives under Settings. "Approve" on an order refunds it.';
|
|
|
|
export default {
|
|
targets: [{ engine: web(), app: { url: 'http://localhost:3000' } }],
|
|
agents: {
|
|
default: { model: gateway('openai/gpt-6-luna-fast'), context: vocabulary },
|
|
thorough: {
|
|
model: gateway('anthropic/claude-opus-5'),
|
|
system: 'Verify every claim on screen before concluding a step.',
|
|
context: vocabulary,
|
|
},
|
|
buyer: { model: gateway('openai/gpt-6-luna-fast'), system: 'You are a first-time buyer.', context: vocabulary },
|
|
admin: { model: gateway('openai/gpt-6-luna-fast'), system: 'You manage orders and approve refunds.', context: vocabulary },
|
|
},
|
|
} satisfies E2EConfig;
|
|
```
|
|
|
|
An entry holds the `model`, an optional [`judge`](/models#a-judge-of-its-own),
|
|
`system`, `context`, [tools](/tools), budgets, and `providerOptions`.
|
|
`maxSteps` caps the actions of one step and `maxModelCalls` its model
|
|
calls: 25 by default, 100 at most. See the
|
|
[config reference](/reference/config#agents).
|
|
|
|
Entries don't inherit from `default`. Give each one the model and context it needs,
|
|
for example from a shared constant as above.
|
|
|
|
A [custom executor](/executors) goes under `executor`. The model, judge,
|
|
context, and budgets still apply. `system` and `tools` are rejected, since
|
|
the executor brings its own prompt and tools.
|
|
|
|
## Instructions and vocabulary
|
|
|
|
| | `system` | `context` |
|
|
|---|---|---|
|
|
| Answers | How should the acting agent work? | What does this app call things? |
|
|
| Who reads it | The act loop only | Every model call: act turns and the judges behind `assert`, `waitFor`, and `extract` |
|
|
| Belongs here | Persona, caution, what to verify before finishing, how to treat unexpected dialogs | Names of screens and menus, where a feature lives, which button submits a form, domain terms |
|
|
| Set on | `agents.<name>.system` | `agents.<name>.context`, at most 16 KiB |
|
|
|
|
A judge never sees `system`, so "a step is done once the form closes"
|
|
can't lower the bar for a verdict. Rule of thumb: true for any tester on
|
|
this app goes in `context`, true for this tester on any app goes in
|
|
`system`.
|
|
|
|
A test or group appends its own vocabulary to `context` with
|
|
`agentContext`. Use it for facts only that test knows, such as seeded data
|
|
or a feature flag that changes a label:
|
|
|
|
```ts title="tests/billing.e2e.ts"
|
|
import { describe, test } from 'e2e';
|
|
|
|
describe('billing', { agentContext: 'The seeded workspace is called "Acme Trial".' }, () => {
|
|
test('upgrades the tier', async ({ agent }) => { /* ... */ });
|
|
});
|
|
```
|
|
|
|
Keep instructions out of `agentContext`: it reaches the judges too.
|
|
|
|
Under `e2e run`, every prompt also carries the test's title path, such as
|
|
`Test: billing > upgrades the tier`. The judges read it too, so name a
|
|
test for what it covers, not for the verdict you expect.
|
|
|
|
## Pick an agent per run
|
|
|
|
`--agent` chooses the agent for tests that don't select one. Several names
|
|
run each test one time per agent:
|
|
|
|
<CodeGroup>
|
|
```bash npm
|
|
npx e2e run --agent thorough tests/checkout.e2e.ts
|
|
npx e2e run --agent buyer,admin
|
|
```
|
|
|
|
```bash pnpm
|
|
pnpm exec e2e run --agent thorough tests/checkout.e2e.ts
|
|
pnpm exec e2e run --agent buyer,admin
|
|
```
|
|
|
|
```bash bun
|
|
bunx e2e run --agent thorough tests/checkout.e2e.ts
|
|
bunx e2e run --agent buyer,admin
|
|
```
|
|
</CodeGroup>
|
|
|
|
## Pin an agent in the suite
|
|
|
|
A test, a describe block, or one call can select an agent. The closest
|
|
setting wins: call, then test, then enclosing groups, then run.
|
|
|
|
```ts title="tests/checkout.e2e.ts"
|
|
import { describe, test } from 'e2e';
|
|
|
|
describe('checkout as a first-time buyer', { agent: 'buyer' }, () => {
|
|
test('pays for one item', async ({ app, agent }) => { /* ... */ });
|
|
|
|
describe('refunds', { agent: 'admin' }, () => {
|
|
test('refunds the order', async ({ agent }) => { /* ... */ });
|
|
});
|
|
|
|
test('reorders, and the admin approves', async ({ app, agent }) => {
|
|
await app.open('/orders');
|
|
await agent.act('reorder the most recent order');
|
|
// The admin's act call sees what the earlier steps of this test did.
|
|
await agent.act('approve the pending reorder', { agent: 'admin' });
|
|
await agent.assert('the reorder shows as approved');
|
|
});
|
|
});
|
|
```
|
|
|
|
`--agent` doesn't replace an agent that a test selects. An unknown agent on
|
|
a test or group fails during collection. An unknown agent on a call raises
|
|
`INVALID_ARGUMENT`. Reports record the agent of each step.
|
|
|
|
## One flow, several personas
|
|
|
|
<Video>
|
|
<video src="/images/personas.mp4" aria-label="Four agents, each a persona who reads one language, run the same localization tests. The Arabic run fails on a button left in English." muted loop playsInline controls />
|
|
|
|
Four personas, one per language, run the same localization suite.
|
|
</Video>
|
|
|
|
Select several agents to run the same test one time with each:
|
|
|
|
```ts title="tests/checkout.e2e.ts"
|
|
import { describe, test } from 'e2e';
|
|
|
|
describe('checkout', { agent: ['buyer', 'admin'] }, () => {
|
|
test('pays for one item', async ({ app, agent }) => { /* runs once per persona */ });
|
|
test('applies a promo code', async ({ app, agent }) => { /* ... */ });
|
|
});
|
|
```
|
|
|
|
Each agent gets its own result and artifact directory. Terminal and JUnit
|
|
test names include the agent, such as `[admin]`.
|
|
|
|
`--agent` selects among the pinned agents and never adds one. Here,
|
|
`--agent admin` runs only `admin`. `--agent thorough` matches neither, so
|
|
both still run.
|
|
|
|
A serial group runs as one unit per agent, and its members share the
|
|
group's pin. A setup test pins at most one agent.
|
|
|
|
Each agent records its own [replay cache](/cache) entries, keyed by its name
|
|
and context. `buyer` never replays the steps of `admin`, and a changed
|
|
`agentContext` records again.
|
|
|
|
An agent is not a signed-in user. For signed-in personas, pair each agent
|
|
with a `session` in a separate describe block. See
|
|
[Authentication](/authentication#several-signed-in-users).
|
|
|
|
## Make the agent yours
|
|
|
|
When a step fails or uses too many calls, open its report, or run with
|
|
`--debug` to see the transcript. Fix the instruction first, using the labels
|
|
on screen. When the problem affects several tests, fix the agent:
|
|
vocabulary in `context`, instructions in `system`, or a [tool](/tools) for an
|
|
operation such as seeding a cart through a test API. To compare models or
|
|
`providerOptions`, run the same tests with `--agent <name>`.
|
|
|
|
## Next
|
|
|
|
<CardGroup cols={2}>
|
|
<Card title="Project tools" icon="wrench" href="/tools">
|
|
Give the agent a test API or a device toolkit.
|
|
</Card>
|
|
<Card title="Custom executors" icon="gears" href="/executors">
|
|
Change how the agent chooses and performs actions.
|
|
</Card>
|
|
</CardGroup>
|