Commit Graph
20 Commits
Author SHA1 Message Date
Gustavo CostaandOskar Kwaśniewski d1f2089081 feat(decision): act and assert through an AI SDK decision model (#775)
* feat(decision): add Clef and Jev executors on the System One choice API

* feat(decision): default Clef transport to clef-flash

* feat(decision): generalize to decision models, address review

* fix(decision): reject non-string decision choices

* fix(decision): skip empty select labels and blank destinations

An empty option label or whitespace-only destination offered as a choice fails at dispatch with INVALID_ARGUMENT and aborts the step instead of letting the model re-aim. Omit them at enumeration. Also qualify the uncertain-action docs: blocking stops the selected action, while earlier confident actions in the step have already run.

* feat(decision): add opt-in vision for Clef, Jev stays text-only

The executor stays text-only by default. With vision true and a transport that declares it, masked viewport pixels ride along as System One images data URLs. Clef declares vision; Jev never receives pixels. Withheld or tainted viewports fall back to semantic text.

* docs(decision): qualify vision cost to calls carrying pixels

* fix(decision): untainted pixel fixtures and honest skill pixels line

The test helper takes a tainted flag so the image-send tests use the only state the runner returns pixels in; tainted captures stay withheld observations. The skill no longer claims pixels are unavailable: screenshots may inform node choices with vision true, without adding point actions.

* bench(testbed): luna vs clef-flash harness and first numbers

Same todo flow under two configs: default agent on gpt-6-luna via OpenCodeX and decisionExecutor on clef-flash. Luna side passed in 23s with 6 model calls; clef-flash pending credentials. Includes the probe script, the docs section, and the flow screenshot.

* docs(decision): fill clef-flash benchmark row

Two no-cache runs both blocked uncertain on the first action (a25 at p=0.619, conf=0.387 against the default 0.9 gates), answering in about a second with 1,543 input tokens.

* docs(decision): benchmark aggregates over 5 runs per side

Luna completed 5/5 flows at ~14s and ~25k input tokens per run; clef-flash blocked uncertain 5/5 at ~1s and ~1.5k tokens per run.

* bench(testbed): parametrized flow, gate sweep, wire-shape finding

Params fix the missing type choices; gates 0.6/0.5 and 0.51/0.3 still block. Direct experiment shows hierarchical structured questions take clef-flash from 0.57 to 0.89 confidence on the same screen.

* feat(decision): hierarchical operation-then-target on structured transports

Structured transports (Clef, Jev) now decide the operation first and the target within it, with object instructions and element records. Flat endpoints keep the single string choice. Live benchmark: first operation 0.909/0.802, first target 0.940/0.775, up from 0.747/0.558 flat.

* docs(decision): compare luna vs clef-flash vs clef with measured cost

* feat(decision): run decisions through AI SDK evaluate, gates opt-in

decisionExecutor now accepts any AI SDK evaluation model that answers
choice questions, such as typeSafeAi.evaluationModel('jev-latest'), and
decides through experimental_evaluate (one choice question per decision,
no retries, so the call budget stays exact). ai is an optional peer that
loads only on that path. evaluationModel() exposes clef(), jev(), and
systemOne() transports to experimental_evaluate, which covers Clef until
it has an AI SDK provider.

The official TypeSafe provider rounds probabilities to two places, so a
long distribution can sum to 0.99: validation now allows half a unit per
rounded option, the AI SDK rule, and jev() declares the two places.

Probability and confidence gates are off by default and stay
configurable. Measuring clef-flash without them surfaced three executor
bugs, each fixed here:
- a lone target is dispatched without a one-option question, which the
  choice API rejects with HTTP 422;
- acting decisions receive the step's action history, and the repeat
  guard counts the same action on the same screen across the whole step,
  so an A-B oscillation blocks instead of burning the call budget;
- the operation and target questions state that typed text is not saved
  until it is submitted.
The structured path also checks the call budget before the target
request (review feedback).

Testbed todo flow, 5 no-cache runs each: clef-flash 5/5 (9.3s mean, 12
calls), clef 5/5 (11.5s, 12 calls), luna unchanged at 5/5 (14.9s).

* bench(testbed): Laya through systemOne

Laya serves the System One API and accepts structured values, so it runs
through systemOne({ structured: true }) unchanged. On the todo flow its
first operation question chose unsupported at 0.29 / 0.10 on four runs,
answered by the multilingual checkpoint its router picked for an English
prompt; the fifth run and further probes hit insufficient_credits.

* fix(decision): address review on rounding, history, and evaluation ids

- validateDecision accepts a rounded distribution only when 1 lies between
  the sums of each value's rounding interval, clipped to [0, 1]; a summed
  per-option tolerance exceeded 1 with many options and admitted two
  options at probability 1.
- Decisions send the 20 most recent history entries, each clipped to 240
  characters, so long steps and large params stay within request limits.
- evaluationModel() builds its answer, confidence, and criteria maps with
  null prototypes, so an id such as __proto__ stays an own entry.
- @ai-sdk/provider moves to dependencies: the public declarations import
  its evaluation types. ai stays an optional peer.

* bench(testbed): pin Laya's English checkpoint

Laya's router sent the full English prompt to its multilingual checkpoint.
The API accepts model 'english' to pin the English one (its routing
metadata reports the explicit model), so the bench config sends it.

* docs(decision): benchmark Laya locally on M1, both checkpoints 0/5

* docs(decision): reconcile laya narrative with run evidence, money not tokens

* feat(decision): caller-overridable transport flags on clef() and jev()

* feat(decision): AI SDK evaluate executor with element table and text model

* fix(decision): empty-tree check and provider pin

* fix(decision): address review on secrets, options, errors, and docs

Answers the cubic threads on the evaluate-only executor: terminal checks reject an incomplete screen instead of judging it, secret markers are stripped at any depth, hidden/disabled options are never offered, select options share the 255-choice cap, select criteria name the option, out-of-range probabilities fail, text-model errors mirror the decision mapping, and the README/docs describe the evaluate-only API.

* fix(decision): count dropped select options and fix README

Counts enabled options inside rows dropped past the element cap as omitted, adds the E2EConfig type import to the README example, and qualifies when type is offered.

* fix(decision): settle terminal claims by the check and match codebase style

The second rejected done/failed claim now ends by the check verdict (fails -> ACTION_FAILED, anything else -> blocked), and the completion check honors the gates. The decision state carries the last 10 actions, check targets show their checked state, checked radios are never toggled, and options dropped by the per-question cap stay out of omitted. The docs example opens /todos in test code since navigate is not offered. Single quotes, template literals, and split long lines across the package.

* fix(decision): show filled fields to verdicts and pin the history window

Verdict views (assert and the completion check) build the element table with typing enabled, so a field the step filled keeps its row and value even when the step itself cannot type. The history test now asserts the window holds the newest ten actions.

* fix(decision): let completion checks see results and name the text key

Page text keeps a named node's text when it differs from the name, as formatNode does, so a status named Greeting that says Welcome back, admin! reaches the verdict. The completion check now asks whether the task is complete and sees the non-secret params and the actions with their typed values; rejected claims stay out. The text prompt names the single text key, asks for no code fence, and prefixes the request so JSON-object response modes accept it.

* fix(decision): keep secret handles out of replayed history and mark check inputs as data

Replayed summaries name a filled secret handle; seeding the history now replaces declared secret names with <secret>, so names reach the model only as secret question criteria. The completion check treats action descriptions, typed values, and params as data, and the guide says the check sees only non-secret values.

* fix(decision): redact only the handle in replayed secret-fill summaries

Matching the declared secret name anywhere in a replayed summary also erased labels and values equal to it, and missed handles the runner cut at 40 characters. Only the quoted handle after fill secret at the start of the summary becomes <secret> now.

* docs(decision): describe the one-request fan-out in the guide intro

* fix(decision): count a scroll that changes what is in view as progress

The fingerprint behind the stall guard hashed node content but not position, so a scroll that only moved the viewport read as page unchanged and the third one blocked the step. It now also hashes whether each node meets the viewport (membership, not coordinates), so a scroll that brings other nodes into view resets the count while one that moves nothing, as at the page bottom, still trips the guard.

* chore: prune @pnpm/exe from the lockfile after merging main

* fix(decision): match the supported Node range in engines

---------

Co-authored-by: Oskar Kwaśniewski <oskar@okwasniewski.com>
2026-10-04 10:08:55 +02:00
Oskar Kwaśniewski 9f62d9bdce fix(skill): declare the app on the target in skill and docs config examples (#709)
* fix(skill): declare the app on the target in skill and docs config examples

The bundled skill and docs/authentication.mdx still taught web({ url, command }), removed in #698. The skill's two config examples now live in docs/examples/skill/, typechecked and mirrored by check-docs-examples, so the old shape fails pnpm check.

* fix(skill): run the docs drift check on skill changes, drop stale service wording
2026-10-01 09:43:09 +02:00
Oskar Kwaśniewski 3dc4ec9614 feat!: browser fixture, getByRole name arg, toMatchSchema, top-level hooks, separate credential namespace (#704)
* feat!: browser fixture, getByRole name arg, toMatchSchema, top-level hooks, separate credential namespace

API review feedback before launch:

- web fixture renamed browser (Web -> Browser, WebExpectation ->
  BrowserExpectation); requires: ['browser']; destructuring web hints the
  new name. web() factory and other Web* types unchanged.
- getByRole(role, name, options?) beside the object form; MCP locate
  writes the short form.
- expect(value).toMatchSchema(standardSchema): sync, returns the typed
  output; poll resolves to it, soft returns undefined after a kept failure.
- describe, beforeEach, afterEach, beforeAll, afterAll exported from e2e;
  web and mobile export typed beforeEach/afterEach.
- credentials and secrets are separate namespaces: a password handle is
  <name>.password, secrets.get() never returns it. Committed recordings
  migrated offline (old key hash verified, then recomputed).
- runtime test.skip keeps Playwright semantics; docs say it throws.

* docs(web): browser acts on one active tab; cookies, routes, dialogs cover the whole browser

* fix(secrets): judge a fill's purpose by the config, not the handle; no old-name hints

A config-time secrets.get() handle reads its purpose from the run's
registry again, and authorizeSecretFill checks the registered purpose, so a
handle claiming generic-secret cannot put a password into a plain field.
Drop the web fixture migration hint. Changesets name the on-screen marker
change for replay entries.

* docs(skill): browser fixture, getByRole name arg, toMatchSchema, top-level hooks, credential namespace on the rewritten skill

* fix(expect): negated toMatchSchema returns nothing, as its type says

* fix(expect,docs): soft double negation stays soft, null ~standard is INVALID_ARGUMENT, Playwright samples keep Playwright's getByRole

Also documents SoftExpectCall, SoftValueExpectation, and NegatedPollExpectation
on the expect reference, requires: ['browser'] in the skill's options table,
and tightens the poll and secret-purpose tests.

* docs(babysit): triage pattern for scripted renames that rewrite another library's code

* docs: engine options take a secrets entry, web package hook exports, vault example imports

* refactor(secrets)!: credentials hold their password, allSecrets feeds fills; engines export describe and every hook; BrowserRoute, BrowserResponse

From the maintainability review:
- ResolvedConfig.secrets is the config.secrets entries only; a credential
  carries its password; allSecrets is the one map fills, the ledger, and
  redaction read. The per-consumer purpose filters and the .password
  suffix parsing are gone, and e2e mcp builds handles from config.
- The config load and a running test name the credential a missing
  secrets.get() most likely meant.
- @e2e-dev/web and @e2e-dev/mobile export describe and all four hooks
  beside test, so a test file registers from one import.
- WebRoute and WebResponse are BrowserRoute and BrowserResponse.
- getByRole normalizes its arguments once; isTextMatch is the one
  string-or-RegExp check; poll reuses ValueMatcherName.

* revert(web): keep WebRoute and WebResponse; Browser names the fixture, Web the network and option types
2026-09-30 20:58:52 +02:00
Oskar Kwaśniewski 4d488f8286 feat!: the app is declared on its target; engines only drive it (#698)
* feat!: the app is declared on its target; engines only drive it

The app under test moves from the engine's options to `targets: [{ engine,
app }]`: `url`, `bundleId`, `appPath`, `identity`, `environment`,
`launchArguments`, `permissions`, `command`, and `readyUrl`. `web()` and
`mobile()` refuse every moved option in one `INVALID_CONFIG` naming the
target block; `defineEngine({ app })` is gone and engines check what their
platform needs in `validateApp`. `requires: ['browser' | 'native-app']` is
derived from the target's app. Replay cache and session keys are byte for
byte what they were, pinned by target-identity.test.ts.

Dependency services are not part of this version: `services` anywhere, and
`web({ services })`, fail saying so. `app.command` on port 0 without a
command, and `reuseExisting` on a free port, are refused. An `e2e mcp`
open cancelled or disconnected mid-start stops what it started, and a hung
shared start is abandoned once no session waits on it.

* fix: review follow-ups: cancelled opens, abandoned starts, null targets, stale docs

* refactor: drop the moved-option messages; old app keys are unknown keys

* test: give the replaced-nodes suite a budget that survives a loaded runner

* test: size the replaced-nodes hook budget to its seven tests
2026-09-30 19:48:00 +02:00
Oskar Kwaśniewski d228e22ea3 feat(agents)!: plain-object agents entries; createAgent and limits removed (#646)
* feat(agents)!: plain-object agents entries, createAgent and limits removed

Each agents entry is one AgentConfig object: model, judge, system, context,
tools, maxSteps, maxModelCalls, judgmentTimeout, maxObservationBytes,
maxInputTokens, providerOptions, or executor for a custom brain (system and
tools rejected beside it). No inheritance between agents.

- createAgent() kept as a stub throwing INVALID_CONFIG; bare StepExecutor or
  legacy createAgent value as an entry fails naming { executor } / the object
- timeout -> judgmentTimeout, maxTurns removed (entry and
  createToolLoopExecutor), unknown keys get did-you-mean
- top-level limits removed; maxModelTokensPerCall -> per-agent
  maxInputTokens, the rest internal constants; report-1 run.limits keeps
  its shape, filled from max across agents plus the constants
- tools validated at config load; built-in agent built from the entry at
  fixture time; explore and the MCP catalog read the entry's tools
- migrate apps, docs, docs examples, init scaffold, skills, mobile README

* docs: AGENTS.md names the agent entry's model, not createAgent

* refactor(agents)!: delete createAgent instead of a throwing stub

* docs: MODEL_UNAVAILABLE scoped to the built-in agent, fixture-acquisition timing
2026-09-29 19:51:02 +02:00
Oskar Kwaśniewski 67c29fc6b2 chore!: publish engines and reporter under the @e2e-dev npm scope (#602) 2026-09-28 19:05:05 +02:00
Oskar Kwaśniewski 920ac4ebf0 ci(mobile): iOS simulator and Android emulator gates for the mobile benchmark, with a Control Inventory scenario (#499) 2026-09-24 16:08:05 +02:00
Oskar Kwaśniewski 8802b0f0dc chore: pin openai/gpt-6-luna-fast as the example and suite model (#463) 2026-09-23 10:29:52 +02:00
Oskar KwaśniewskiandOskar Kwaśniewski bda000cedf feat(web): a browser provider leases hosted browsers per worker or per attempt (#418)
Co-authored-by: Oskar Kwaśniewski <oskar@tester.army>
2026-09-22 15:49:14 +02:00
Oskar KwaśniewskiandOskar Kwaśniewski 41f3b3211b feat(mobile): a device lease carries agent-device client configuration (#416)
Co-authored-by: Oskar Kwaśniewski <oskar@tester.army>
2026-09-22 14:48:23 +02:00
Oskar Kwaśniewski f5e96d0cbe feat!: rename engine packages to @e2edev/web and @e2edev/mobile (#392) 2026-09-21 19:52:32 +02:00
Oskar Kwaśniewski f7f86d91e8 feat(agent-device): lease hosted devices through a DeviceProvider and release them in a new engine finish hook (#365) 2026-09-21 18:05:29 +02:00
Oskar Kwaśniewski f09b69e90a docs: simplify onboarding and clarify model access (#346) 2026-09-18 11:52:06 +02:00
Oskar Kwaśniewski c2c5df7df9 feat(e2e)!: the runner publishes as e2e; @e2edev/e2e is retired (#294) 2026-09-14 08:56:22 +02:00
Oskar Kwaśniewski ff33218345 docs: e2e is agentic testing first; quickstart for web and mobile; icons everywhere (#282) 2026-09-12 13:52:59 +02:00
Oskar Kwaśniewski fcf4fe6ff7 docs: split guides and references by platform (#266) 2026-09-11 17:32:13 +02:00
Oskar Kwaśniewski 2d0c6931c4 feat(agent)!: the model is an AI SDK instance the config constructs; init asks which gateway (#255) 2026-09-10 17:26:25 +02:00
Oskar Kwaśniewski c72b05e57f feat(config)!: agents by name replace agent, and --agent picks one per run (#244) 2026-09-10 11:59:30 +02:00
Oskar Kwaśniewski 17cde5a128 feat(agent)!: act takes one options bag and resolves with what the step did (#219)
* feat(agent)!: act takes one options bag and resolves with what the step did

* fix(agent): a legacy act call fails loudly; a retry names itself in step.cache

`act(instruction, params)` and `act(instruction, params, options)` from 0.7
arrived as an options bag of foreign keys, or with a third argument the
fixture dropped, and ran with their parameters silently ignored. Both now
fail with `INVALID_ARGUMENT` naming the move to `{ params }`, from a
JavaScript test as from a typed one; `schema` and `vision` keep their
`UNSUPPORTED_CAPABILITY` message. Unit-tested in act-validation.test.ts.

A step in a retry attempt records a trace but never replays one, and its
`step.cache` was absent, indistinguishable from caching being off. It now
reads `missed` with the new reason `retry`, in the report schema too.

The reference page's opening `Agent` interface catches up with the act
signature, the summary is described as the step's (the replay writes it when
the cache finishes the step), and the quickstart example passes its values
under `params`, which is what failed the docs typecheck in CI.
2026-09-09 12:23:55 +02:00
devin-ai-integration[bot]andoskar d8a2cfd3fe docs(quickstart): three commands to green with no key, typechecked snippets (#213)
Co-authored-by: oskar <oskar@okwasniewski.com>
2026-09-09 11:27:48 +02:00