mirror of
https://github.com/tester-army/e2e.git
synced 2026-11-21 03:05:26 +08:00
Add opt-in read-only assertion exploration with bounded scrolling and model calls. Keep every viewport for the judgment and classify exhausted assertions as inconclusive without changing default model selection. The built-in agent picks a dedicated assertion loop by step kind: its own rules, opening, vision policy, and inconclusiveOnOverflow. Tool removal lives in the grammar's assertion filter. createVerdictTool(kind) drops ASSERTION_INCONCLUSIVE from an act verdict instead of rejecting it, so ACP serves one complete_step without validating by hand. Co-authored-by: Ubuntu <ubuntu@ip-10-48-38-144.us-west-2.compute.internal>
304 lines
11 KiB
Plaintext
304 lines
11 KiB
Plaintext
---
|
|
title: Models
|
|
description: Configure a model with your own subscription, an API key, or a local server.
|
|
"og:image": "https://e2e.com/docs/images/og/models.png?v=5d360f2457"
|
|
"twitter:image": "https://e2e.com/docs/images/og/models.png?v=5d360f2457"
|
|
---
|
|
|
|
import { ModelProvider } from './snippets/model-provider.jsx';
|
|
|
|
Agent steps use the model you set in `e2e.config.ts`. There is no default.
|
|
The model comes from an [AI SDK](https://ai-sdk.dev/providers/ai-sdk-providers) provider: a
|
|
gateway, a direct API key, a local server, or your
|
|
[subscription](/subscriptions). For a first setup, `e2e init` writes the
|
|
config. See the [quickstart](/quickstart#use-a-subscription-or-api-key).
|
|
|
|
Tests without agent steps need no model. The built-in agent needs a model
|
|
that supports tool calls and images. [Decision models](/decision-models)
|
|
such as Clef or Jev run through a custom executor instead.
|
|
|
|
## Pick a gateway or a provider
|
|
|
|
The built-in agent needs the `ai` package, whatever the provider:
|
|
|
|
<CodeGroup>
|
|
```bash npm
|
|
npm install -D ai
|
|
```
|
|
|
|
```bash pnpm
|
|
pnpm add -D ai
|
|
```
|
|
|
|
```bash bun
|
|
bun add -d ai
|
|
```
|
|
</CodeGroup>
|
|
|
|
The provider reads its own key from the environment. A gateway reaches
|
|
many providers with one key. To switch, replace the model import and the
|
|
`model` value. Where a snippet defines `agent`, use it as `agents.default`:
|
|
|
|
<Tabs>
|
|
<Tab title="Vercel AI Gateway">
|
|
|
|
`gateway` comes with the `ai` package. Set `AI_GATEWAY_API_KEY`, or use a
|
|
[Vercel OIDC token](#vercel-oidc-instead-of-a-key) with no key.
|
|
|
|
```ts title="e2e.config.ts"
|
|
import type { E2EConfig } from 'e2e';
|
|
import { web } from '@e2e-dev/web';
|
|
import { gateway } from 'ai';
|
|
|
|
export default {
|
|
targets: [{ engine: web(), app: { url: 'http://localhost:3000' } }],
|
|
agents: { default: { model: gateway('openai/gpt-6-luna-fast') } },
|
|
} satisfies E2EConfig;
|
|
```
|
|
</Tab>
|
|
<Tab title="OpenRouter">
|
|
|
|
Install `@openrouter/ai-sdk-provider` and set `OPENROUTER_API_KEY`:
|
|
|
|
```ts title="Model setup"
|
|
import { openrouter } from '@openrouter/ai-sdk-provider';
|
|
|
|
const agent = { model: openrouter('openai/gpt-6-luna-fast') };
|
|
```
|
|
</Tab>
|
|
<Tab title="Provider API key">
|
|
|
|
Install the provider's AI SDK package and set its API key. For OpenAI,
|
|
install `@ai-sdk/openai` and set `OPENAI_API_KEY`:
|
|
|
|
```ts title="Model setup"
|
|
import { openai } from '@ai-sdk/openai';
|
|
|
|
const agent = { model: openai('gpt-6-luna') };
|
|
```
|
|
|
|
Other providers work the same way, with their own package, model IDs, and
|
|
credentials. See [AI SDK providers](#ai-sdk-providers).
|
|
</Tab>
|
|
<Tab title="Local or self-hosted">
|
|
|
|
Install `@ai-sdk/openai-compatible`. Set the endpoint URL and a model ID the
|
|
endpoint serves. Add `apiKey` if the server needs one. Set
|
|
`supportsStructuredOutputs: true` when the server accepts a JSON schema
|
|
response format, as Ollama does. Without it, the AI SDK drops the schema with a
|
|
`responseFormat` warning and agent steps may take more model calls:
|
|
|
|
```ts title="Model setup"
|
|
import { createOpenAICompatible } from '@ai-sdk/openai-compatible';
|
|
|
|
const local = createOpenAICompatible({
|
|
name: 'local',
|
|
baseURL: 'http://127.0.0.1:11434/v1',
|
|
supportsStructuredOutputs: true,
|
|
// apiKey: process.env.LLM_API_KEY,
|
|
});
|
|
|
|
const agent = { model: local.chatModel('your-model-id') };
|
|
```
|
|
</Tab>
|
|
</Tabs>
|
|
|
|
Gateways usually use `provider/model` IDs. Direct providers use their own.
|
|
|
|
### Vercel OIDC instead of a key
|
|
|
|
Without `AI_GATEWAY_API_KEY`, `gateway()` signs in with a short-lived Vercel
|
|
OIDC token, which it refreshes. Sign in to the Vercel CLI and link the
|
|
project one time:
|
|
|
|
<CodeGroup>
|
|
```bash npm
|
|
npx vercel login
|
|
npx vercel link
|
|
npx e2e run
|
|
```
|
|
|
|
```bash pnpm
|
|
pnpm dlx vercel login
|
|
pnpm dlx vercel link
|
|
pnpm exec e2e run
|
|
```
|
|
|
|
```bash bun
|
|
bunx vercel login
|
|
bunx vercel link
|
|
bunx e2e run
|
|
```
|
|
</CodeGroup>
|
|
|
|
The gateway uses a valid `VERCEL_OIDC_TOKEN`. Else it gets a token for the
|
|
project in `.vercel/project.json`, found from the working directory upward.
|
|
e2e loads no `.env` file, so a token from `npx vercel env pull` works only
|
|
when your shell exports it, or when the config loads the file with
|
|
[`process.loadEnvFile()`](/reference/config#loading).
|
|
|
|
## AI SDK providers
|
|
|
|
Every AI SDK provider that supports tool calls works. Each provider's
|
|
package and variables are on its [AI SDK provider page](https://ai-sdk.dev/providers/ai-sdk-providers). Select yours:
|
|
|
|
<ModelProvider show="select" />
|
|
|
|
A provider that is not in the list also works if it implements the AI SDK
|
|
language model specification v2 or later.
|
|
|
|
<Steps>
|
|
<Step title="Install the provider package">
|
|
Install `ai` and the provider package:
|
|
|
|
<ModelProvider show="install" />
|
|
</Step>
|
|
<Step title="Set the credentials">
|
|
Export the variables in your shell or CI, or load a `.env` file in the
|
|
config with [`process.loadEnvFile()`](/reference/config#loading).
|
|
|
|
<ModelProvider show="env" />
|
|
</Step>
|
|
<Step title="Add the model to the config">
|
|
Set the model in `agents.default`. The model ID is an example.
|
|
|
|
<ModelProvider show="config" />
|
|
</Step>
|
|
</Steps>
|
|
|
|
<Accordion title="Models that refuse a forced tool choice">
|
|
Agent steps ask every model for a tool call on every turn. Claude Opus 5.5,
|
|
Sonnet 5.5, and Fable 5.1 reject that request shape, DeepSeek's V4 models
|
|
(`@ai-sdk/deepseek`) refuse it because they think by default, and Muse Spark
|
|
models accept only `auto`. e2e continues with `auto`, and later steps on that
|
|
model start that way. Thinking can stay on.
|
|
</Accordion>
|
|
|
|
<Accordion title="Claude Opus 5.5, Sonnet 5.5, and Fable 5.1">
|
|
These models bind each thinking block to the system prompt and the messages
|
|
before it, and reject a request that replays the block after either changed.
|
|
An agent step keeps its prompt fixed, and once it elides a superseded screen
|
|
or screenshot from an earlier message it stops sending the thinking from that
|
|
message on. The step's `--debug` transcript notes it.
|
|
</Accordion>
|
|
|
|
## Several models in one config
|
|
|
|
Give each model its own agent name, then pick one with `--agent` or in a
|
|
test. See [Agents and personas](/agents). To switch a model without editing
|
|
the config, read an environment variable:
|
|
|
|
```ts title="Agent setup"
|
|
const agent = {
|
|
model: gateway(process.env.MODEL ?? 'openai/gpt-6-luna-fast'),
|
|
};
|
|
```
|
|
|
|
### Pick a model by the key that is set
|
|
|
|
When teammates hold different keys, choose the model by the variable that
|
|
is set. Keep one agent name, so everyone replays the same recordings:
|
|
|
|
```ts title="Agent setup"
|
|
import { anthropic } from '@ai-sdk/anthropic';
|
|
import { gateway } from 'ai';
|
|
|
|
const agent = {
|
|
model: process.env.ANTHROPIC_API_KEY
|
|
? anthropic('claude-sonnet-5-5')
|
|
: gateway('openai/gpt-6-luna-fast'),
|
|
};
|
|
```
|
|
|
|
Make `gateway()` the fallback: [Vercel OIDC](#vercel-oidc-instead-of-a-key)
|
|
signs it in with no variable to test. A key in `.env` counts only when
|
|
[`process.loadEnvFile()`](/reference/config#loading) runs before this code.
|
|
|
|
A step recorded with the other model still replays and names that model. See
|
|
[What must match](/cache#what-must-match).
|
|
|
|
### A judge of its own
|
|
|
|
By default, one model drives `agent.act` and answers `agent.assert`,
|
|
`agent.waitFor`, and `agent.extract`. Set `judge` to use another model for
|
|
those checks:
|
|
|
|
```ts title="Agent setup"
|
|
const agent = {
|
|
model: gateway('openai/gpt-6-luna-fast'),
|
|
judge: gateway('anthropic/claude-sonnet-5'),
|
|
};
|
|
```
|
|
|
|
Built-in `agent.assert` with `explore: true` runs the tool loop on
|
|
`agents.<name>.model`, even when `judge` is set.
|
|
|
|
Reports name the model of each judgment. `judge` takes an AI SDK language
|
|
model. For Clef or Jev, use the [decision executor](/decision-models). See
|
|
[what a judgment sees](/agent-steps#what-the-model-sees).
|
|
|
|
## Data retention and compliance
|
|
|
|
Model input is the redacted screen, the step instruction, and the
|
|
screenshots the agent asks for. e2e reads nothing back from the provider
|
|
after a response, so a run needs no state stored by the provider.
|
|
|
|
Every model request names e2e: a `User-Agent` that starts with
|
|
`e2e/<version> (<platform>; <arch>)`, plus `HTTP-Referer:
|
|
https://tester.army/e2e` and `X-Title: e2e` for Vercel AI Gateway and
|
|
OpenRouter. They replace the same headers set on the provider instance.
|
|
|
|
<AccordionGroup>
|
|
<Accordion title="OpenAI and Azure OpenAI: zero data retention">
|
|
Every request to a Responses model (`openai(...)`, `azure(...)`, or
|
|
`openai/*` on the gateway) carries `store: false` and a prompt cache key.
|
|
Chat Completions models and OpenAI-compatible providers get neither. The
|
|
provider keeps no copy of the conversation, and the prompt cache still
|
|
works. Organizations with zero data retention need no configuration.
|
|
|
|
To keep stored responses, set `store` in `providerOptions`. For Azure
|
|
OpenAI, the key is `azure`:
|
|
|
|
```ts title="Agent setup"
|
|
const agent = {
|
|
model: openai('gpt-6-luna'),
|
|
providerOptions: { openai: { store: true } },
|
|
};
|
|
```
|
|
</Accordion>
|
|
<Accordion title="Anthropic under a BAA">
|
|
Agent steps use the Messages API with tool use, prompt caching, images, and
|
|
structured outputs. All are eligible under Anthropic's HIPAA readiness.
|
|
|
|
The agreement covers `api.anthropic.com` only. Call it with
|
|
`@ai-sdk/anthropic`, not through a gateway or OpenRouter. Leave `judge`
|
|
unset, so one agreement covers every request. Anthropic's Covered Models
|
|
need 30-day retention, so organizations with zero data retention can't use
|
|
them.
|
|
</Accordion>
|
|
</AccordionGroup>
|
|
|
|
## What fails, and where
|
|
|
|
| Problem | Code | Fix |
|
|
| --- | --- | --- |
|
|
| A model is a string instead of an AI SDK instance | `INVALID_CONFIG` | Construct it, for example `gateway('openai/gpt-6-luna-fast')`. |
|
|
| The built-in agent has no model | `MODEL_UNAVAILABLE` | Set `agents.<name>.model`. |
|
|
| GitHub Copilot doesn't serve the model to your account | `MODEL_UNAVAILABLE` | Pick an id your plan serves. See [Pick a model](/subscriptions#pick-a-model). |
|
|
| A provider that reads its key at config load has none | `CONFIG_LOAD_FAILED` | Set the key. |
|
|
| Authentication fails on the first model call | `MODEL_PROVIDER_FAILED` | Check the provider's API key, or run `npx e2e login` for a subscription. For Vercel OIDC the message names only `AI_GATEWAY_API_KEY`. Check `vercel login` and `.vercel/project.json`. |
|
|
| The `ai` package is missing | `MODEL_UNAVAILABLE` | Install `ai`. See [Pick a gateway or a provider](#pick-a-gateway-or-a-provider). |
|
|
| A step's second turn fails with `Item with id 'rs_…' not found` | `MODEL_PROVIDER_FAILED` | Leave `store` off in `providerOptions`, the default, or call OpenAI directly instead of through a gateway. |
|
|
| A custom executor's own `model` or `judge` differs from the agents entry's | `INVALID_CONFIG` | See [Custom executors](/executors#keep-the-loop-change-the-vocabulary). |
|
|
|
|
## Next
|
|
|
|
<CardGroup cols={2}>
|
|
<Card title="Subscriptions" icon="key" href="/subscriptions">
|
|
Sign in with ChatGPT, Copilot, OpenCode Console, or SuperGrok.
|
|
</Card>
|
|
<Card title="Agents and personas" icon="users" href="/agents">
|
|
Choose agents for tests and signed-in users.
|
|
</Card>
|
|
</CardGroup>
|