mirror of
https://github.com/tester-army/e2e.git
synced 2026-11-22 18:19:48 +08:00
* feat(decision): add Clef and Jev executors on the System One choice API
* feat(decision): default Clef transport to clef-flash
* feat(decision): generalize to decision models, address review
* fix(decision): reject non-string decision choices
* fix(decision): skip empty select labels and blank destinations
An empty option label or whitespace-only destination offered as a choice fails at dispatch with INVALID_ARGUMENT and aborts the step instead of letting the model re-aim. Omit them at enumeration. Also qualify the uncertain-action docs: blocking stops the selected action, while earlier confident actions in the step have already run.
* feat(decision): add opt-in vision for Clef, Jev stays text-only
The executor stays text-only by default. With vision true and a transport that declares it, masked viewport pixels ride along as System One images data URLs. Clef declares vision; Jev never receives pixels. Withheld or tainted viewports fall back to semantic text.
* docs(decision): qualify vision cost to calls carrying pixels
* fix(decision): untainted pixel fixtures and honest skill pixels line
The test helper takes a tainted flag so the image-send tests use the only state the runner returns pixels in; tainted captures stay withheld observations. The skill no longer claims pixels are unavailable: screenshots may inform node choices with vision true, without adding point actions.
* bench(testbed): luna vs clef-flash harness and first numbers
Same todo flow under two configs: default agent on gpt-6-luna via OpenCodeX and decisionExecutor on clef-flash. Luna side passed in 23s with 6 model calls; clef-flash pending credentials. Includes the probe script, the docs section, and the flow screenshot.
* docs(decision): fill clef-flash benchmark row
Two no-cache runs both blocked uncertain on the first action (a25 at p=0.619, conf=0.387 against the default 0.9 gates), answering in about a second with 1,543 input tokens.
* docs(decision): benchmark aggregates over 5 runs per side
Luna completed 5/5 flows at ~14s and ~25k input tokens per run; clef-flash blocked uncertain 5/5 at ~1s and ~1.5k tokens per run.
* bench(testbed): parametrized flow, gate sweep, wire-shape finding
Params fix the missing type choices; gates 0.6/0.5 and 0.51/0.3 still block. Direct experiment shows hierarchical structured questions take clef-flash from 0.57 to 0.89 confidence on the same screen.
* feat(decision): hierarchical operation-then-target on structured transports
Structured transports (Clef, Jev) now decide the operation first and the target within it, with object instructions and element records. Flat endpoints keep the single string choice. Live benchmark: first operation 0.909/0.802, first target 0.940/0.775, up from 0.747/0.558 flat.
* docs(decision): compare luna vs clef-flash vs clef with measured cost
* feat(decision): run decisions through AI SDK evaluate, gates opt-in
decisionExecutor now accepts any AI SDK evaluation model that answers
choice questions, such as typeSafeAi.evaluationModel('jev-latest'), and
decides through experimental_evaluate (one choice question per decision,
no retries, so the call budget stays exact). ai is an optional peer that
loads only on that path. evaluationModel() exposes clef(), jev(), and
systemOne() transports to experimental_evaluate, which covers Clef until
it has an AI SDK provider.
The official TypeSafe provider rounds probabilities to two places, so a
long distribution can sum to 0.99: validation now allows half a unit per
rounded option, the AI SDK rule, and jev() declares the two places.
Probability and confidence gates are off by default and stay
configurable. Measuring clef-flash without them surfaced three executor
bugs, each fixed here:
- a lone target is dispatched without a one-option question, which the
choice API rejects with HTTP 422;
- acting decisions receive the step's action history, and the repeat
guard counts the same action on the same screen across the whole step,
so an A-B oscillation blocks instead of burning the call budget;
- the operation and target questions state that typed text is not saved
until it is submitted.
The structured path also checks the call budget before the target
request (review feedback).
Testbed todo flow, 5 no-cache runs each: clef-flash 5/5 (9.3s mean, 12
calls), clef 5/5 (11.5s, 12 calls), luna unchanged at 5/5 (14.9s).
* bench(testbed): Laya through systemOne
Laya serves the System One API and accepts structured values, so it runs
through systemOne({ structured: true }) unchanged. On the todo flow its
first operation question chose unsupported at 0.29 / 0.10 on four runs,
answered by the multilingual checkpoint its router picked for an English
prompt; the fifth run and further probes hit insufficient_credits.
* fix(decision): address review on rounding, history, and evaluation ids
- validateDecision accepts a rounded distribution only when 1 lies between
the sums of each value's rounding interval, clipped to [0, 1]; a summed
per-option tolerance exceeded 1 with many options and admitted two
options at probability 1.
- Decisions send the 20 most recent history entries, each clipped to 240
characters, so long steps and large params stay within request limits.
- evaluationModel() builds its answer, confidence, and criteria maps with
null prototypes, so an id such as __proto__ stays an own entry.
- @ai-sdk/provider moves to dependencies: the public declarations import
its evaluation types. ai stays an optional peer.
* bench(testbed): pin Laya's English checkpoint
Laya's router sent the full English prompt to its multilingual checkpoint.
The API accepts model 'english' to pin the English one (its routing
metadata reports the explicit model), so the bench config sends it.
* docs(decision): benchmark Laya locally on M1, both checkpoints 0/5
* docs(decision): reconcile laya narrative with run evidence, money not tokens
* feat(decision): caller-overridable transport flags on clef() and jev()
* feat(decision): AI SDK evaluate executor with element table and text model
* fix(decision): empty-tree check and provider pin
* fix(decision): address review on secrets, options, errors, and docs
Answers the cubic threads on the evaluate-only executor: terminal checks reject an incomplete screen instead of judging it, secret markers are stripped at any depth, hidden/disabled options are never offered, select options share the 255-choice cap, select criteria name the option, out-of-range probabilities fail, text-model errors mirror the decision mapping, and the README/docs describe the evaluate-only API.
* fix(decision): count dropped select options and fix README
Counts enabled options inside rows dropped past the element cap as omitted, adds the E2EConfig type import to the README example, and qualifies when type is offered.
* fix(decision): settle terminal claims by the check and match codebase style
The second rejected done/failed claim now ends by the check verdict (fails -> ACTION_FAILED, anything else -> blocked), and the completion check honors the gates. The decision state carries the last 10 actions, check targets show their checked state, checked radios are never toggled, and options dropped by the per-question cap stay out of omitted. The docs example opens /todos in test code since navigate is not offered. Single quotes, template literals, and split long lines across the package.
* fix(decision): show filled fields to verdicts and pin the history window
Verdict views (assert and the completion check) build the element table with typing enabled, so a field the step filled keeps its row and value even when the step itself cannot type. The history test now asserts the window holds the newest ten actions.
* fix(decision): let completion checks see results and name the text key
Page text keeps a named node's text when it differs from the name, as formatNode does, so a status named Greeting that says Welcome back, admin! reaches the verdict. The completion check now asks whether the task is complete and sees the non-secret params and the actions with their typed values; rejected claims stay out. The text prompt names the single text key, asks for no code fence, and prefixes the request so JSON-object response modes accept it.
* fix(decision): keep secret handles out of replayed history and mark check inputs as data
Replayed summaries name a filled secret handle; seeding the history now replaces declared secret names with <secret>, so names reach the model only as secret question criteria. The completion check treats action descriptions, typed values, and params as data, and the guide says the check sees only non-secret values.
* fix(decision): redact only the handle in replayed secret-fill summaries
Matching the declared secret name anywhere in a replayed summary also erased labels and values equal to it, and missed handles the runner cut at 40 characters. Only the quoted handle after fill secret at the start of the summary becomes <secret> now.
* docs(decision): describe the one-request fan-out in the guide intro
* fix(decision): count a scroll that changes what is in view as progress
The fingerprint behind the stall guard hashed node content but not position, so a scroll that only moved the viewport read as page unchanged and the third one blocked the step. It now also hashes whether each node meets the viewport (membership, not coordinates), so a scroll that brings other nodes into view resets the count while one that moves nothing, as at the page bottom, still trips the guard.
* chore: prune @pnpm/exe from the lockfile after merging main
* fix(decision): match the supported Node range in engines
---------
Co-authored-by: Oskar Kwaśniewski <oskar@okwasniewski.com>
292 lines
10 KiB
Plaintext
292 lines
10 KiB
Plaintext
---
|
|
title: Models
|
|
description: Configure a model with your own subscription, an API key, or a local server.
|
|
---
|
|
|
|
import { ModelProvider } from './snippets/model-provider.jsx';
|
|
|
|
Agent steps use the model configured in `e2e.config.ts`. You can use an
|
|
[existing subscription](/subscriptions), an API key, or a local server.
|
|
e2e uses the [AI SDK](https://ai-sdk.dev/providers). You can use
|
|
[all AI SDK providers that support tool calls](#ai-sdk-providers).
|
|
`e2e init` offers these choices and writes the config for you. For the first
|
|
setup, follow the [quickstart](/quickstart#use-a-subscription-or-api-key).
|
|
|
|
Tests without agent steps do not need a model. [Decision models](/decision-models), hosted or locally
|
|
hosted, such as Clef or Jev, can drive bounded actions and judge assertions through a custom executor. For the built-in agent,
|
|
choose a model that supports tool calls and images. Steps using the built-in agent need the `ai`
|
|
package installed, whichever provider you use:
|
|
|
|
<CodeGroup>
|
|
```bash npm
|
|
npm install -D ai
|
|
```
|
|
|
|
```bash pnpm
|
|
pnpm add -D ai
|
|
```
|
|
|
|
```bash bun
|
|
bun add -d ai
|
|
```
|
|
</CodeGroup>
|
|
|
|
There is no default model or shared API-key variable. The config selects the
|
|
model, and the provider reads its own key from the environment, such as
|
|
`AI_GATEWAY_API_KEY`. For a subscription, run `npx e2e login` instead.
|
|
|
|
## Pick a gateway or a provider
|
|
|
|
A gateway gives you access to several providers with one API key. A provider
|
|
package connects to one provider directly. To change an existing config,
|
|
replace its model import and `model` value:
|
|
|
|
<Tabs>
|
|
<Tab title="Vercel AI Gateway">
|
|
|
|
`gateway` is included in the `ai` package. Set `AI_GATEWAY_API_KEY`, or use
|
|
a [Vercel OIDC token](#vercel-oidc-instead-of-a-key) without a key.
|
|
|
|
```ts title="e2e.config.ts"
|
|
import type { E2EConfig } from 'e2e';
|
|
import { web } from '@e2e-dev/web';
|
|
import { gateway } from 'ai';
|
|
|
|
export default {
|
|
targets: [{ engine: web(), app: { url: 'http://localhost:3000' } }],
|
|
agents: { default: { model: gateway('openai/gpt-6-luna-fast') } },
|
|
} satisfies E2EConfig;
|
|
```
|
|
</Tab>
|
|
<Tab title="OpenRouter">
|
|
|
|
Install `@openrouter/ai-sdk-provider` and set `OPENROUTER_API_KEY`.
|
|
|
|
```ts title="Model setup"
|
|
import { openrouter } from '@openrouter/ai-sdk-provider';
|
|
|
|
const agent = { model: openrouter('openai/gpt-6-luna-fast') };
|
|
```
|
|
|
|
Use `agent` as `agents.default` in your config.
|
|
</Tab>
|
|
<Tab title="Provider API key">
|
|
|
|
Install the provider's AI SDK package and set its API key. For OpenAI,
|
|
install `@ai-sdk/openai` and set `OPENAI_API_KEY`:
|
|
|
|
```ts title="Model setup"
|
|
import { openai } from '@ai-sdk/openai';
|
|
|
|
const agent = { model: openai('gpt-6-luna') };
|
|
```
|
|
|
|
Use `agent` as `agents.default`. Other AI SDK providers work the same way,
|
|
with their own package, model IDs, and authentication settings. See
|
|
[AI SDK providers](#ai-sdk-providers).
|
|
</Tab>
|
|
<Tab title="Local or self-hosted">
|
|
|
|
Install `@ai-sdk/openai-compatible`. Set the endpoint URL and a model ID
|
|
served by that endpoint:
|
|
|
|
```ts title="Model setup"
|
|
import { createOpenAICompatible } from '@ai-sdk/openai-compatible';
|
|
|
|
const local = createOpenAICompatible({
|
|
name: 'local',
|
|
baseURL: 'http://127.0.0.1:11434/v1',
|
|
// apiKey: process.env.LLM_API_KEY,
|
|
});
|
|
|
|
const agent = { model: local.chatModel('your-model-id') };
|
|
```
|
|
|
|
Use `agent` as `agents.default`. Add `apiKey` if the server requires one.
|
|
</Tab>
|
|
</Tabs>
|
|
|
|
Model IDs depend on the provider. Gateways usually use `provider/model`;
|
|
direct providers use their own model IDs.
|
|
|
|
### Vercel OIDC instead of a key
|
|
|
|
Without `AI_GATEWAY_API_KEY`, `gateway()` authenticates with a Vercel OIDC
|
|
token. Sign in to the Vercel CLI and link the project once; the gateway then
|
|
mints a short-lived token for that project and refreshes it when it expires:
|
|
|
|
<CodeGroup>
|
|
```bash npm
|
|
npx vercel login
|
|
npx vercel link
|
|
npx e2e run
|
|
```
|
|
|
|
```bash pnpm
|
|
pnpm dlx vercel login
|
|
pnpm dlx vercel link
|
|
pnpm exec e2e run
|
|
```
|
|
|
|
```bash bun
|
|
bunx vercel login
|
|
bunx vercel link
|
|
bunx e2e run
|
|
```
|
|
</CodeGroup>
|
|
|
|
The token is a valid `VERCEL_OIDC_TOKEN`, otherwise one minted for the
|
|
project in `.vercel/project.json`, found by walking up from the working
|
|
directory. e2e loads no `.env` files, so a token pulled with
|
|
`npx vercel env pull` reaches the gateway only when your shell exports it or
|
|
the config loads the file with
|
|
[`process.loadEnvFile()`](/reference/config#loading).
|
|
|
|
An OIDC failure is the same `MODEL_PROVIDER_FAILED` as a bad key, and the AI
|
|
SDK's message names only `AI_GATEWAY_API_KEY`. With no key set, check that the
|
|
Vercel CLI is signed in and that `.vercel/project.json` exists.
|
|
|
|
## AI SDK providers
|
|
|
|
Select a provider. The three steps below show the setup for that provider.
|
|
|
|
<ModelProvider show="select" />
|
|
|
|
The list shows the AI SDK providers that support tool calls. A provider that
|
|
is not in the list also works if it uses the AI SDK language model
|
|
specification v2 or later. Its model must support tool calls and images.
|
|
|
|
### Install the provider package
|
|
|
|
Install `ai` and the provider package.
|
|
|
|
<ModelProvider show="install" />
|
|
|
|
### Set the credentials
|
|
|
|
The provider reads its credentials from the environment. e2e does not load
|
|
`.env` files. Export the variables in your shell or CI, or load the file in
|
|
the config with [`process.loadEnvFile()`](/reference/config#loading).
|
|
|
|
<ModelProvider show="env" />
|
|
|
|
If a key is missing or wrong, the first agent step fails with
|
|
`MODEL_PROVIDER_FAILED`. Some providers read the key when the config loads.
|
|
For these providers, a missing key stops all runs with `CONFIG_LOAD_FAILED`.
|
|
|
|
### Add the model to the config
|
|
|
|
Set the model in `agents.default`. The model ID is an example. Use a model
|
|
that supports tool calls and images.
|
|
|
|
<ModelProvider show="config" />
|
|
|
|
DeepSeek's V4 models (`@ai-sdk/deepseek`) think by default, and in that mode
|
|
they refuse the forced tool choice agent steps ask for. The step is sent again
|
|
with `auto` and an instruction to reply with tool calls only, and later steps
|
|
on that model start that way, so thinking can stay on.
|
|
|
|
## Several models in one config
|
|
|
|
Give agents different names to use different models:
|
|
|
|
```ts title="e2e.config.ts"
|
|
export default {
|
|
targets: [{ engine: web(), app: { url: 'http://localhost:3000' } }],
|
|
agents: {
|
|
default: { model: gateway('openai/gpt-6-luna-fast') },
|
|
careful: { model: gateway('anthropic/claude-sonnet-5') },
|
|
},
|
|
} satisfies E2EConfig;
|
|
```
|
|
|
|
Run with `npx e2e run --agent careful`, or
|
|
[choose an agent in a test](/agents). Each model can use a different provider.
|
|
Agents do not inherit from each other: an entry without a `model` does not
|
|
use `default`'s, see [Agents and personas](/agents).
|
|
|
|
### Switch a model with an env var
|
|
|
|
To switch a model without editing the config, read your own environment
|
|
variable:
|
|
|
|
```ts title="Agent setup"
|
|
const agent = {
|
|
model: gateway(process.env.MODEL ?? 'openai/gpt-6-luna-fast'),
|
|
};
|
|
```
|
|
|
|
### A judge of its own
|
|
|
|
By default, the same model drives `agent.act` and answers `agent.assert`,
|
|
`agent.waitFor`, and `agent.extract`. Set `judge` to use a separate model
|
|
for those checks:
|
|
|
|
```ts title="Agent setup"
|
|
const agent = {
|
|
model: gateway('openai/gpt-6-luna-fast'),
|
|
judge: gateway('anthropic/claude-sonnet-5'),
|
|
};
|
|
```
|
|
|
|
Judgments use the current screen and the question, without the acting
|
|
agent's transcript or summaries. Reports include the model used for each
|
|
judgment. See [Custom executors](/executors) if you replace the built-in agent.
|
|
|
|
`judge` takes an AI SDK language model. Clef and Jev use a different API;
|
|
use the [decision executor](/decision-models) for `agent.act` and `agent.assert`.
|
|
|
|
## Data retention and compliance
|
|
|
|
Model input is the redacted screen, the step instruction, and the
|
|
screenshots the agent asks for. e2e reads nothing back from the provider
|
|
after a response, so a run needs no state kept on the provider's side.
|
|
|
|
Every model request names e2e: the `User-Agent` starts with
|
|
`e2e/<version> (<platform>; <arch>)`, and `HTTP-Referer:
|
|
https://tester.army/e2e` with `X-Title: e2e` attribute the traffic to e2e on
|
|
the Vercel AI Gateway and OpenRouter. These replace the same headers set on
|
|
the provider instance.
|
|
|
|
**OpenAI and Azure OpenAI.** Every request carries `store: false` and a
|
|
prompt cache key. The provider keeps no copy of the conversation, a
|
|
reasoning model's earlier turns travel inline as encrypted content, and the
|
|
prompt cache still serves the repeated prefix. Organizations with zero data
|
|
retention need no configuration. To keep stored responses, set `store` in
|
|
`providerOptions`; the key is `azure` for Azure OpenAI:
|
|
|
|
```ts title="Agent setup"
|
|
const agent = {
|
|
model: openai('gpt-6-luna'),
|
|
providerOptions: { openai: { store: true } },
|
|
};
|
|
```
|
|
|
|
**Anthropic under a BAA.** Agent steps use the Messages API with tool use,
|
|
prompt caching, images, and structured outputs, all eligible under
|
|
Anthropic's HIPAA readiness. The agreement covers `api.anthropic.com` only:
|
|
call it directly with `@ai-sdk/anthropic` rather than through a gateway or
|
|
OpenRouter, and leave `judge` unset so one agreement covers every request.
|
|
Anthropic's Covered Models require 30-day retention and are not offered to
|
|
organizations with zero data retention.
|
|
|
|
## What fails, and where
|
|
|
|
| Problem | Code | Fix |
|
|
| --- | --- | --- |
|
|
| A model is a string instead of an AI SDK instance | `INVALID_CONFIG` | Construct it, for example `gateway('openai/gpt-6-luna-fast')`. |
|
|
| The built-in agent has no model | `MODEL_UNAVAILABLE` | Set `agents.<name>.model`. |
|
|
| Authentication fails on the first model call | `MODEL_PROVIDER_FAILED` | Check the provider's API key or the linked Vercel project, or run `npx e2e login` for a subscription. |
|
|
| The `ai` package is missing | `MODEL_UNAVAILABLE` | Run `npm install -D ai`. |
|
|
| A step's second turn fails with `Item with id 'rs_…' not found` | `MODEL_PROVIDER_FAILED` | Leave `store` off in `providerOptions`, the default, or call OpenAI directly instead of through a gateway. |
|
|
| A custom executor's own `model` or `judge` differs from the agents entry's | `INVALID_CONFIG` | See [Custom executors](/executors#keep-the-loop-change-the-vocabulary). |
|
|
|
|
<CardGroup cols={2}>
|
|
<Card title="Subscriptions" icon="key" href="/subscriptions">
|
|
Sign in with ChatGPT, Copilot, OpenCode Console, or SuperGrok.
|
|
</Card>
|
|
<Card title="Agents and personas" icon="users" href="/agents">
|
|
Choose agents for tests and signed-in users.
|
|
</Card>
|
|
</CardGroup>
|