Commit Graph
51 Commits
Author SHA1 Message Date
Satyajit Sahoo 25eb2f8c5a feat(mcp): attach to a test held at screen.pause() or --step (#1972)
* feat(mcp): attach to a test held at screen.pause() or --step

A test held at `screen.pause()` or at an `e2e run --step` hold now prints
an ID, such as `Paused (p1) in pays with a card [web]:`. You can pass it to
`open_session {attach: "p1"}` in `e2e mcp` to look at the held test's app
and act on it.

The tools run in the worker, on the engine session the held attempt already
has. Requests go from the MCP server to the runner over the directory's
step socket, and the runner relays them to the worker. Replies are redacted
with the attempt's secret ledger before they're sent.

- `close_session` detaches and leaves the test held.
- When the test resumes, the session ends right away and a call still in
  progress is cancelled.
- Attached calls aren't recorded as steps in the report or the trace.
- `run_code`, the recording tools, and `type_secret` aren't available in an
  attached session.
- `e2e run --step` in a terminal now listens on the step socket like a pause
  does, so you can attach at its holds.

With several workers paused, each prompt is now drawn from the hold it
names. Before, it could show one test's title next to another test's screen.

* fix(mcp): print output a tool's timer writes after its attached call ends

Output is muted only while the attached call is running. The call scope
lives in its own module, so process-output.ts doesn't load the attach tools.

* feat(mcp): record in an attached session when the run doesn't record

start_recording and stop_recording join the attached catalog, which now
lists exactly the normal session's tools. If the run already records the
test, start_recording fails with the run's video directory. A recording
still running when the session ends is saved.

The docs no longer describe what a pause ID looks like. The examples and
sample output show one.

* fix(mcp): don't hold a resumed test while an attached recording saves

On a resume or an interrupt the worker sends the stop for the agent's
recording and lets the test continue. The file finishes in the background
and its path goes to the log. Teardown waits for a save still in progress
for at most cleanupTimeout.

A project tool's attachScreenshot works in an attached session and stays
out of the test's report. In the live list a long title ends in an
ellipsis, so the pause ID stays visible.
2026-11-20 15:06:42 +00:00
Krzysztof Moch eddf577526 feat(reporters): build the built-in reporters on the public Reporter API (#1936)
* feat(reporters): build the built-in reporters on the public Reporter API

The built-in list, dot, json, junit, and markdown reporters now use only
the public Reporter contract: factories on e2e/reporters, printsTo for the
terminal stream a reporter prints to (one reporter per stream), live
onOutput, onStepPrompt, and onSummary hooks for any number of reporters,
and reportFiles for files the runner writes atomically. The list and dot
ids beside json print to stderr, so list,json now works as dot,json does.
@e2e-dev/html-reporter writes through reportFiles.

* docs(reporters): bring the Japanese and Chinese reporter pages up to date

Translates the reporter reference, the config reporters table, the
--reporter row, and the debugging output paragraph. In English, the name
section now says a reporter may share a built-in's name, and two lines
read less ambiguously.
2026-11-19 15:58:33 +00:00
Satyajit Sahoo 34322c5416 feat(cli): pause at screen.pause() under --watch (#1905)
Under `e2e run --watch`, a test is now paused at each `screen.pause()` on
every run, as in `e2e run`.

While a pause is shown, the keys are read by the pause prompt and not by
watch mode:

- Press Enter to resume the test.
- Press `q` to cancel that run and stay in watch mode.

`e2e run --step next` and `stop` can be used from another shell, as for a
run without `--watch`.
2026-11-19 11:15:49 +00:00
Satyajit Sahoo 4893d1dd75 feat: add screen.pause() (#1904)
With `await screen.pause()`, a test is paused at that line, with the app
open, until someone resumes it.

- In a terminal, press Enter to resume the test.
- From another shell in the same directory, run `e2e run --step next`.
  A run without a terminal is resumed this way.
- Paused time doesn't count toward the test's timeout.
- With `CI` set, the test fails with `PAUSE_IN_CI`.
- Under `--step`, a test is still paused after `n` or `a`.
- A test isn't paused in an MCP `open_session {test}` session, under
  `openHost` or `run()`, or in a core program.
- A test isn't paused under `e2e run --watch` yet.
2026-11-19 11:00:39 +00:00
Krzysztof Moch 2359860783 feat(cli): step through a test without a terminal (#1750) 2026-11-15 17:47:21 +01:00
Oskar Kwaśniewski 3735887569 chore: move from tester-army to e2e-inc GitHub org (#1672) 2026-11-10 20:11:53 +01:00
Krzysztof Moch 93c93f5e25 feat(cli): e2e run --step holds each step until a key (#1540) 2026-11-09 18:06:53 +01:00
Kacper Kapuściak f185ca8ba2 docs: retire sidebar page icons (#1582)
* docs: retire sidebar page icons

The docs sidebar no longer shows icons. Drop icon: from every page's frontmatter, delete the grey silhouettes in docs/images/icons/, and make check-docs-logos fail on any frontmatter icon instead of requiring one on vendor pages. Card logos stay required. The authoring-docs skill trades its Page icons section for a short Vendor logos one.

* docs: require a migrate overview card for each migration guide

* docs: match only top-level icon keys in check-docs-logos
2026-11-05 18:23:59 +00:00
Oskar Kwaśniewski 0c0469605a chore: replace e2e.tester.army with e2e.com (#1572) 2026-11-05 09:23:45 +01:00
Oskar Kwaśniewski 50fa7afc1a feat(mcp): run_tests tool runs e2e run from the MCP server (#1520) 2026-11-03 18:23:58 +01:00
Oskar Kwaśniewski 723c1af84e feat(report): quote the app's first errors in the failure block (#1480) 2026-10-30 20:28:07 +01:00
dc8f56d205 fix(report): retain serial duration with interrupted retry verdicts (#1056)
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Oskar Kwaśniewski <oskar@okwasniewski.com>
Signed-off-by: Rudy Celekli <rudy@gradiahq.com>
Signed-off-by: Rudy Celekli <47457359+rudycelekli@users.noreply.github.com>
2026-10-30 13:01:10 +01:00
Oskar Kwaśniewski e62486d668 docs(quickstart): list the common config options (#1332) 2026-10-22 12:20:53 -04:00
Nguyễn Văn Hiếu 82039ad6d6 fix(report): note truncated agent observations (#1301) 2026-10-21 13:48:57 -04:00
93f77d81e2 fix(agent): classify a content-filter refusal as MODEL_REFUSED (#973)
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Oskar Kwaśniewski <oskar@okwasniewski.com>
2026-10-20 12:09:49 -04:00
Oskar Kwaśniewski d7ec0ecf44 feat(e2e): add screen.tree and screen.debug, print test output without list reporter (#1239) 2026-10-19 15:47:02 +02:00
Oskar Kwaśniewski 58a5cec116 feat(e2e): keep artifact screenshots after a secret fill (#1218)
* feat(e2e): keep artifact screenshots after a secret fill

Screenshots stay on disk and in CI, so a fill no longer turns off
app.screenshot, toHaveScreenshot, assert and failure evidence. Model
pixels stay withheld; the taint getter is renamed withholdsModelPixels.

* docs(e2e): drop remaining post-fill screenshot denial, tighten tests
2026-10-19 07:47:30 +00:00
Piotr Pasztor bf3c8e8ac1 fix(agent): stop the run once when the provider does not serve the model (#1153) 2026-10-16 14:44:25 +02:00
Oskar Kwaśniewski 236611fafb docs: fix packaged docs against source, ship snippets, check anchors (#1168)
* docs: fix packaged docs against source, ship snippets, check anchors

* docs: tighten review wording, harden anchor check fences and ids

* docs: address review on tool errors, CLI failure output, anchor check

* docs(security): base64 is decoded in text downloads

* docs(telemetry): shorten the telemetry page (#1174)

* docs(telemetry): shorten the telemetry page

* docs(telemetry): engine versions are semver or unversioned

* docs(migrate): shorten the Playwright migration guide (#1172)

* docs(debugging): shorten the debugging guide (#1170)

* docs(auth): merge repeated secret facts on the signing-in page (#1169)
2026-10-15 17:56:13 +02:00
Kacper Kapuściak 5d4053e8fa docs: render an Open Graph card for every docs page with @vercel/og (#1062) 2026-10-12 15:10:32 +02:00
Kacper Kapuściak 449fa93670 docs: rewrite and shorten the guide pages with the authoring-docs skill (#1019)
* docs(agent-steps): rewrite with the authoring-docs skill

* docs(agents): rewrite with the authoring-docs skill

* docs(api-testing): rewrite with the authoring-docs skill

* docs(authentication): rewrite with the authoring-docs skill

INVALID_CONFIG also fires for a password shorter than 6 characters.

* docs(browser): rewrite with the authoring-docs skill

* docs(bug-bash): rewrite with the authoring-docs skill

* docs(cache): rewrite with the authoring-docs skill

* docs(ci): rewrite with the authoring-docs skill

Rerun folders are rerun-<n>/, not rerun-1/. Merge exit codes into Gate pull requests.

* docs(ci/bitrise): rewrite with the authoring-docs skill

* docs(ci/codemagic): rewrite with the authoring-docs skill

* docs(ci/eas-workflows): rewrite with the authoring-docs skill

* docs(ci/github-actions): rewrite with the authoring-docs skill

* docs(coding-agents): rewrite with the authoring-docs skill

* docs(core-concepts): rewrite with the authoring-docs skill

* docs(debugging): rewrite with the authoring-docs skill

* docs(decision-models): rewrite with the authoring-docs skill

A point tap costs three decision calls and one text call, not four.

* docs(example-projects): rewrite with the authoring-docs skill

* docs(executors): rewrite with the authoring-docs skill

* docs(explore): rewrite with the authoring-docs skill

* docs(github): rewrite with the authoring-docs skill

* docs(index): rewrite with the authoring-docs skill

* docs(integrations/eas): rewrite with the authoring-docs skill

* docs(integrations/index): rewrite with the authoring-docs skill

* docs(integrations/kernel): rewrite with the authoring-docs skill

* docs(integrations/smol): rewrite with the authoring-docs skill

* docs(migrate/cypress): rewrite with the authoring-docs skill

* docs(migrate/detox): rewrite with the authoring-docs skill

* docs(migrate/maestro): rewrite with the authoring-docs skill

Fold the command map tables into accordions.

* docs(migrate/playwright): rewrite with the authoring-docs skill

List use and testMatch among the Playwright keys e2e rejects. Fold the mapping tables into accordions.

* docs(migrate/selenium): rewrite with the authoring-docs skill

* docs(mobile): rewrite with the authoring-docs skill

Fix the Android installApp troubleshooting entry: agent-device 0.21.22 parses the manifest itself and searches ~/Library/Android/sdk on macOS.

* docs(models): rewrite with the authoring-docs skill

* docs(quickstart): rewrite with the authoring-docs skill

* docs(security): rewrite with the authoring-docs skill

Name smol beside Kernel and EAS Simulators among the integrations that make outbound connections.

* docs(subscriptions): rewrite with the authoring-docs skill

npx e2e login with no provider opens a picker.

* docs(telemetry): rewrite with the authoring-docs skill

Add the source-checkout, unwritable-preferences, and CI opt-out rules.

* docs(tools): rewrite with the authoring-docs skill

Config load fails only when an entry sets both tools and a custom executor.

* docs(visual-testing): rewrite with the authoring-docs skill

* docs(web): rewrite with the authoring-docs skill

Fold the Next.js dev-origins section, merge the basicAuth examples, and fix the reuseExisting row: in CI e2e ignores it and only a server that already answers is APP_ALREADY_RUNNING. Drop the unrelated agents block from the command config example.

* docs(writing-an-engine): rewrite with the authoring-docs skill

Remove the origin-check claim for location; PR #290 removed origin policy.

* docs(models): keep process.loadEnvFile() as a way to load the pulled OIDC token

* docs(mobile): restore the imports in the several-devices and agent-tools configs

* docs(agent-steps): shorten

Prose words 841 -> 764.

* docs(agents): shorten

Prose words 789 -> 675.

* docs(api-testing): shorten

Prose words 270 -> 266.

* docs(authentication): shorten

Prose words 1000 -> 849.

* docs(browser): shorten

Prose words 1934 -> 1697.

* docs(bug-bash): shorten

Prose words 615 -> 585.

* docs(cache): shorten

Prose words 2793 -> 2374.

* docs(ci): shorten

Prose words 1338 -> 1182.

* docs(ci/bitrise): shorten

Prose words 321 -> 306.

* docs(ci/codemagic): shorten

Prose words 259 -> 250.

* docs(ci/eas-workflows): shorten

Prose words 431 -> 394.

* docs(ci/github-actions): shorten

Prose words 544 -> 453.

* docs(coding-agents): shorten

Prose words 841 -> 735.

* docs(core-concepts): shorten

Prose words 686 -> 620.

* docs(debugging): shorten

Prose words 1826 -> 1604.

* docs(decision-models): shorten

Prose words 1003 -> 942.

* docs(example-projects): shorten

Prose words 347 -> 288.

* docs(executors): shorten

Prose words 1297 -> 1114.

* docs(explore): shorten

Prose words 451 -> 413.

* docs(github): shorten

Prose words 723 -> 626.

* docs(index): shorten

Prose words 387 -> 333.

* docs(integrations/eas): shorten

Prose words 1293 -> 1119.

* docs(integrations/index): shorten

Prose words 174 -> 163.

* docs(integrations/kernel): shorten

Prose words 794 -> 620.

* docs(integrations/smol): shorten

Prose words 924 -> 855.

* docs(migrate/cypress): shorten

Prose words 957 -> 890.

* docs(migrate/detox): shorten

Prose words 1073 -> 1021.

* docs(migrate/maestro): shorten

Prose words 2464 -> 2246.

* docs(migrate/playwright): shorten

Prose words 4835 -> 4546.

* docs(migrate/selenium): shorten

Prose words 933 -> 887.

* docs(mobile): shorten

Prose words 2912 -> 2595.

* docs(models): shorten

Prose words 1187 -> 1048.

* docs(quickstart): shorten

Prose words 980 -> 946.

* docs(security): shorten

Prose words 587 -> 559.

* docs(subscriptions): shorten

Prose words 694 -> 651.

* docs(telemetry): shorten

Prose words 2158 -> 1999.

* docs(tools): shorten

Prose words 460 -> 397.

* docs(visual-testing): shorten

Prose words 583 -> 566.

* docs(web): shorten

Prose words 1262 -> 1104.

* docs(writing-an-engine): shorten

Prose words 1731 -> 1522.

* docs(agent-steps): address review findings

* docs(agents): address review findings

* docs(authentication): address review findings

* docs(cache): address review findings

* docs(ci): address review findings

* docs(coding-agents): address review findings

* docs(debugging): address review findings

* docs(example-projects): address review findings

* docs(executors): address review findings

* docs(index): address review findings

* docs(integrations/eas): address review findings

* docs(migrate/maestro): address review findings

* docs(subscriptions): address review findings

* docs(telemetry): address review findings

* docs(tools): address review findings

* docs(web): address review findings

* docs(writing-an-engine): address review findings
2026-10-10 12:55:15 +00:00
Kacper Kapuściak 521988496e docs(security): split the security page into a short guide and a rules reference (#1008)
The guide answers what most readers ask: how secrets stay out of the model,
what can still leak, and what runs with your permissions. Every rule moves to
docs/reference/security.mdx. smol host-loopback access moves to the smol
page's warning. Fixes facts that no longer matched src/: defineTool errors are
INVALID_CONFIG, the CI cache default skips a cache.store, start-screen replay
matches routes, and outbound connections now list login, models, init, and the
smol image pull.
2026-10-09 13:13:28 +00:00
Oskar Kwaśniewski 1b92a6ddfa feat: e2e trace pages replace the Playwright trace (#869) 2026-10-07 11:28:27 +02:00
Kacper Kapuściak 63482bd6fe docs: reorganize the sidebar into platforms and a deep dive (#870) 2026-10-05 20:39:06 +02:00
Oskar Kwaśniewski 01b88d3afd fix(run): redact secrets from what the config, collected test files, and reporters print (#829) 2026-10-05 16:25:45 +02:00
Oskar Kwaśniewski 78e135a372 fix(secrets): redact plain-string secret values from traces and text downloads (#814)
* fix(secrets): rewrite traces and text downloads whenever the run has a secret value

A registered value a test passed as a plain string (an app.open URL, a fill)
was redacted from titles, labels, and agent input but kept verbatim in
trace.zip: trace and download rewriting was gated on the exposure level,
which only a fill or an engine-held secret raises. Rewriting now depends on
the session ledger holding a value. The engine exposure level had no other
consumer and is removed; exposure now only decides pixels and saved-session
taint.

* fix(secrets): cover a plain-string fill, scope docs to the session's known values

* docs(secrets): download redaction is whole-value; TRACE_WITHHELD outside-path case

* docs(secrets): name the download rewrite failure path
2026-10-04 10:34:34 +02:00
Oskar Kwaśniewski aeaca4fa46 fix(run): bound failure evidence to one 5 s capture per attempt (#819)
A capture that came back empty was retried at each later capture point
(body, settle, finally), so a hung app cost 5 s per point. The engine
call was also awaited without racing the budget, so an engine ignoring
cancellation hung the attempt. Capture once per attempt and abandon the
engine call when the budget ends.
2026-10-04 10:34:18 +02:00
Oskar Kwaśniewski 02ad48a94e feat(web)!: pin playwright-core and add a browser install command (#798)
* feat(web)!: pin playwright-core and add an install command

@e2e-dev/web depends on playwright-core 1.63.0 instead of peering on
playwright. New e2e-web bin installs browsers for the pinned version:
npx @e2e-dev/web install chromium --with-deps. e2e init no longer adds
playwright.

* test(web): run the e2e-web bin for real, and in CI

* test(web): spawn the built e2e-web bin so Node 22.12 runs it
2026-10-03 22:22:20 +02:00
Oskar Kwaśniewski 3a9667c046 fix(run): keep --last-failed reruns honest: failure-limit skips, carried debts, kept evidence (#724)
* fix(run): keep --last-failed reruns honest: failure-limit skips, carried debts, kept evidence

--last-failed now reruns tests --max-failures skipped. A test or failed
suite hook another filter leaves out stays owed in the report's new
run.carried until a rerun runs it; the github fold shows carried rows and
errors and stays red while anything is carried. A rerun prunes artifacts/
to what the previous report names and writes its attempts under
artifacts/rerun-<n>/ instead of wiping the tree and reusing attempt paths.

* fix(run): carry a setup's hook debt when no selected test needs it

The report has no row for an unneeded setup, so carryForward now asks the
collection whether a test still exists instead of the report's rows.

* fix(run): reject malformed artifacts in --last-failed reports, tighten rerun docs

Distinct ids in the carried fixture row.

* fix(run): keep carrying tests of a file a narrowed rerun could not collect

* fix(github): a carried interrupted test reads interrupted, not failed, in the folded page
2026-10-03 16:36:53 +02:00
Oskar Kwaśniewski 3dc4ec9614 feat!: browser fixture, getByRole name arg, toMatchSchema, top-level hooks, separate credential namespace (#704)
* feat!: browser fixture, getByRole name arg, toMatchSchema, top-level hooks, separate credential namespace

API review feedback before launch:

- web fixture renamed browser (Web -> Browser, WebExpectation ->
  BrowserExpectation); requires: ['browser']; destructuring web hints the
  new name. web() factory and other Web* types unchanged.
- getByRole(role, name, options?) beside the object form; MCP locate
  writes the short form.
- expect(value).toMatchSchema(standardSchema): sync, returns the typed
  output; poll resolves to it, soft returns undefined after a kept failure.
- describe, beforeEach, afterEach, beforeAll, afterAll exported from e2e;
  web and mobile export typed beforeEach/afterEach.
- credentials and secrets are separate namespaces: a password handle is
  <name>.password, secrets.get() never returns it. Committed recordings
  migrated offline (old key hash verified, then recomputed).
- runtime test.skip keeps Playwright semantics; docs say it throws.

* docs(web): browser acts on one active tab; cookies, routes, dialogs cover the whole browser

* fix(secrets): judge a fill's purpose by the config, not the handle; no old-name hints

A config-time secrets.get() handle reads its purpose from the run's
registry again, and authorizeSecretFill checks the registered purpose, so a
handle claiming generic-secret cannot put a password into a plain field.
Drop the web fixture migration hint. Changesets name the on-screen marker
change for replay entries.

* docs(skill): browser fixture, getByRole name arg, toMatchSchema, top-level hooks, credential namespace on the rewritten skill

* fix(expect): negated toMatchSchema returns nothing, as its type says

* fix(expect,docs): soft double negation stays soft, null ~standard is INVALID_ARGUMENT, Playwright samples keep Playwright's getByRole

Also documents SoftExpectCall, SoftValueExpectation, and NegatedPollExpectation
on the expect reference, requires: ['browser'] in the skill's options table,
and tightens the poll and secret-purpose tests.

* docs(babysit): triage pattern for scripted renames that rewrite another library's code

* docs: engine options take a secrets entry, web package hook exports, vault example imports

* refactor(secrets)!: credentials hold their password, allSecrets feeds fills; engines export describe and every hook; BrowserRoute, BrowserResponse

From the maintainability review:
- ResolvedConfig.secrets is the config.secrets entries only; a credential
  carries its password; allSecrets is the one map fills, the ledger, and
  redaction read. The per-consumer purpose filters and the .password
  suffix parsing are gone, and e2e mcp builds handles from config.
- The config load and a running test name the credential a missing
  secrets.get() most likely meant.
- @e2e-dev/web and @e2e-dev/mobile export describe and all four hooks
  beside test, so a test file registers from one import.
- WebRoute and WebResponse are BrowserRoute and BrowserResponse.
- getByRole normalizes its arguments once; isTextMatch is the one
  string-or-RegExp check; poll reuses ValueMatcherName.

* revert(web): keep WebRoute and WebResponse; Browser names the fixture, Web the network and option types
2026-09-30 20:58:52 +02:00
Oskar Kwaśniewski 4d488f8286 feat!: the app is declared on its target; engines only drive it (#698)
* feat!: the app is declared on its target; engines only drive it

The app under test moves from the engine's options to `targets: [{ engine,
app }]`: `url`, `bundleId`, `appPath`, `identity`, `environment`,
`launchArguments`, `permissions`, `command`, and `readyUrl`. `web()` and
`mobile()` refuse every moved option in one `INVALID_CONFIG` naming the
target block; `defineEngine({ app })` is gone and engines check what their
platform needs in `validateApp`. `requires: ['browser' | 'native-app']` is
derived from the target's app. Replay cache and session keys are byte for
byte what they were, pinned by target-identity.test.ts.

Dependency services are not part of this version: `services` anywhere, and
`web({ services })`, fail saying so. `app.command` on port 0 without a
command, and `reuseExisting` on a free port, are refused. An `e2e mcp`
open cancelled or disconnected mid-start stops what it started, and a hung
shared start is abandoned once no session waits on it.

* fix: review follow-ups: cancelled opens, abandoned starts, null targets, stale docs

* refactor: drop the moved-option messages; old app keys are unknown keys

* test: give the replaced-nodes suite a budget that survives a loaded runner

* test: size the replaced-nodes hook budget to its seven tests
2026-09-30 19:48:00 +02:00
Oskar Kwaśniewski a53472cc42 fix(agent): extract stops inventing data, clearer unchanged-screen feedback, stable committed recordings (#668)
* fix(agent): extract reports absent data instead of inventing it

Send the caller schema's shape only (no min/max/length/pattern/format) inside
an agent-extract-2 envelope { found, value, missing }. found: false fails the
step ASSERTION_INCONCLUSIVE naming what was missing; the repair round may
decline instead of bending a value to fit. Explore covers a declined plan like
an invalid one.

* fix(cache): ignore a gap's derived rule when comparing re-recorded flows

derived follows how the agent read the value on that run (pixels vs node),
never what a replay does, so read-write runs rewrote committed entries.

* fix(agent): stop blaming a control for an unchanged tree without pixel proof

Tree-only unchanged result now says no listed node changed and a drawn
change needs a screenshot; "no visible effect" only when an identical
screenshot backs it. With no screenshot to be had (secret filled, engine
captures none), an unchanged listing still says to try another way.

* fix(cache): REPLAY_STALE names the strict knob and configured cache dir

Was hardcoded to --strict-cache and .e2e/cache even when cache.strict or
cache.dir came from config.
2026-09-30 14:45:53 +02:00
devin-ai-integration[bot]andoskar 256513dc0b docs: sync the package manager choice across every command snippet (#679)
Co-authored-by: oskar <oskar@okwasniewski.com>
2026-09-30 14:40:08 +02:00
Oskar Kwaśniewski 10d392ea3c fix(config): follow-ups to the config redesign: output data loss, engine-held secrets, migration messages, agents (#662) 2026-09-30 14:04:23 +02:00
Oskar Kwaśniewski 3842b908ce fix(agent): resend a stalled model request; keep turns of a timed-out step (#666)
* fix(agent): resend a stalled model request; keep turns of a timed-out step

One provider request with no response held the step until its own timeout
(an explore step spent 220s of 240s on a single call). Each generate
request is now bounded at 120s and fails as a retryable provider error, so
the SDK's transport retries resend it; the turn notes it. Turns attach as
each ends, so a step the clock settles keeps them, and loop notes survive
the report's per-turn clip.

* fix(agent): abandon a stalled request whose provider ignores the abort

* docs(changeset): say where a first-turn stall note shows

* fix(agent): clear the stall timer when a provider throws synchronously
2026-09-30 09:20:41 +02:00
Oskar Kwaśniewski bb51ec1fdb docs: trim guides to what a test author needs (#656)
* docs: trim guides to what a test author needs

Cut runner internals, give each repeated topic one owner page, rewrite
the densest paragraphs.

Owners now:
- screenshots after a secret fill: security#screenshots-after-a-secret-fill
  (auth, explore, debugging, agent-steps, tools, executors, engine link it)
- redaction rules and limits: security#redaction, #where-redaction-stops
- what the model sees/does in a step: agent-steps; security keeps the
  trust-boundary and POLICY_DENIED tables
- budgets and timeouts: reference/agent#budgets (agent-steps keeps cost levers)
- assert under a custom executor, executor model conflicts: executors
- exit codes: debugging; telemetry in CI: telemetry; explore credentials:
  explore; cache trust: security; personas: agents + authentication
- shard/last-failed edge cases: reference/cli; engine capabilities and
  lifecycle detail: reference/engine
- locator strictness: reference/screen; no sessions on a device: mobile

Other changes:
- security: redactor internals -> short user-facing summary
- cache: timing internals cut, reason table one line per cell plus a
  "Limits and gaps" subsection, committed-file changes as a list
- executors: tool-loop sample drives ctx.actions (portable, recorded)
- tools: simpler seed sample, mobileTools spread beside own tools
- writing-an-engine: locate in skeleton, minimal capability table, one-line
  lifecycle, own Keyboard heading, tool-pack pointer, GitHub test link
- models: `ai` prerequisite and key source up front, env-var switch moved
- migrate: shared rules to one line + link, framework-specific "migrate
  first", drop "Lives in" column, US spelling
- kernel: minimal setup sample, neutral replay size wording
- telemetry: source-checkout paragraph cut

* docs: address review on the guide trims

- tools: the seed sample reads APP_URL again
- ci: --last-failed reads report.json under output
- cypress, detox: strictness is for actions and single-node reads
- explore: artifacts, not evidence, land in the goal directory
2026-09-29 23:46:03 +02:00
Oskar Kwaśniewski 6b851bb604 docs: fix stale and wrong claims in guides (#653)
* docs: fix stale and wrong claims in guides

- output: --artifacts -> --output in bug-bash guide and skill; sessions and
  cache locations follow output / cache.dir in security; artifact path
  gains the agent segment in github
- ci: upload artifacts on !cancelled() in ci guide and skill
- mcp: several sessions (--max-sessions) in security
- replay cache naming: "Commit the replay cache" heading and anchors
  (old anchor kept, e2e init links the new one),
  trace replay -> replay cache in writing-an-engine and executors
- timeouts: STEP_TIMEOUT / STEP_BUDGET_EXHAUSTED advice names config
  timeout, judgmentTimeout, maxSteps, maxModelCalls (debugging, skill)
- mobile: app.open() in device examples and quickstart, mobileTools pack
  all mutating, Android toolbar anchor, drop agent-device release note
- executors: verdict code rules, missing ctx members and scrollUntil,
  a replay cache section; agent-steps passed+code rule
- migrate: swipe direction is the content's, `!` test exclusions,
  video row status, Kernel for Selenium Grid
- secrets: an unset env string fails config load, a provider's value is
  redacted only from its first fill; three secret sinks
- misc: Models link text, telemetry links outbound connections, integration
  config wording, web-only expect(web) in a cross-platform block,
  capabilities heading in engine reference, swipe direction in screen reference

* docs(skill): separate the credential redaction paragraph

* docs(reference): KERNEL_API_KEY names @e2e-dev/kernel
2026-09-29 23:38:52 +02:00
Oskar Kwaśniewski d7a218af07 feat(config)!: config shape, part 1: recording modes, output root, test-glob exclusions, renames, secrets in engine options (#645)
* feat(config)!: remove specVersion from the config

The runner version is the format version. A config that still sets
specVersion fails INVALID_CONFIG telling the author to remove it. The
report's run.specVersion and the replay cache key keep their hard-coded
copies.

* feat(collect): "!" exclusions in the tests globs

An entry starting with "!" removes files an including glob matched,
in any order. Discovery never reads a directory an exclusion takes
whole, unless an including glob could reach a dot name beneath it.
A list of only exclusions is INVALID_CONFIG.

* refactor!: CacheStore and web({ screencast }) renames

TraceCacheStore is CacheStore, and the docs, skill, and CLI call the
cache the replay cache. web({ video: { size, quality } }) is
web({ screencast }) with WebScreencastOptions; web({ video }) fails
INVALID_CONFIG naming screencast.

* feat(secrets): secrets.get() in engine options, resolved per attempt

At config evaluation, before any run exists, secrets.get(name) returns
a reference by name. An engine declares the handles its options hold
in defineEngine({ secrets }); the config load fails INVALID_CONFIG for
a name no secret or credential has. The engine reads the value through
EngineAttemptContext.resolveSecret, which runs a provider fresh and
registers the value in the session and process ledgers before it
returns, without tainting the viewport. e2e/engine exports isSecret.

web({ basicAuth: { password } }) takes a Secret and declares it. Docs
show process.loadEnvFile('.env') in the config.

* feat(run)!: trace and video share recording modes; artifacts holds only store

One RecordingMode ('off' | 'on' | 'retain-on-failure' | 'on-first-retry' |
'on-all-retries') for trace and video, set on the config, a target, the CLI
(--trace [mode], --video [mode]) and a test, one precedence rule: config,
target, CLI, test. Trace defaults to on locally, on-first-retry in CI, with
a plan-time notice when a retry mode meets retries: 0. Serial members cannot
set trace.

Capability rule: a mode from the config root or a flag applies to targets
whose engine can record it, one notice naming the rest; a mode set on a
target or a test is required and fails with UNSUPPORTED_ARTIFACT.

artifacts.kinds (both forms) and artifacts.trace are removed and fail
INVALID_CONFIG naming the trace mode they meant. Failure screenshot and
screen text are always captured when the engine can. MCP sessions keep the
target's trace mode and trace only under 'on'.

* feat(run)!: output directory for every run result; ArtifactStore.putLink

output (default .e2e) and --output <dir> on run and explore hold report.json,
junit.xml, summary.md, ai-trace.json, artifacts/, sessions/ and the e2e mcp
session files; --last-failed reads <output>/report.json. The reportSibling
derivation from the artifacts dir is gone. --artifacts stays as a hidden
flag that fails with the --output it maps to.

A run clears <output>/artifacts at start. cache.dir stays independent. An
output that is the project root or outside it, is or sits in cache.dir,
would hold cache.dir under artifacts/, sessions/ or videos/, or holds a
tests glob root fails INVALID_CONFIG with the reason.

ArtifactStore gains optional putLink(link) for provider-hosted video links;
its ref lands beside the url. A passed retain-on-failure attempt's link is
never handed over, and a failing putLink never fails the run.

bench-ab and the mobile workflow move to --output.

* test(secrets): a hold's registry is probed by an unknown name, now that get() defers without one

* test(web): the download trigger's measured wait tolerates a timer firing early

* fix(secrets): rewrite the trace of an attempt whose engine holds a secret

A Playwright trace records the context options, basic-auth password in
plaintext, but a trace was rewritten only after a fill. resolveSecret now
marks the session so its trace goes through the redactor, screencast
frames kept since the value never reached the screen. The fake engine
writes a real trace archive, and the basic-auth fixture escapes its echo
and refuses a malformed header.

* fix(web): race the basic-auth secret with the attempt's signal

A provider still resolving after the launch timeout set the session after
the runner ended the attempt, so the retry failed with an attempt already
running. Resolution now runs first, raced with the signal.

* fix(mcp): a session notes a configured trace its engine cannot record

* test: a shadow-free fake trace archive and an ungarbled exclusion title

* docs: output paths, replay cache wording, and review corrections

--last-failed, --ai-trace, and the json reporter name <output> paths;
the skill says output never moves cache/ or the app's log; the
INVALID_GLOB row states the check the code makes; serial members list
only; the engine vocabulary names Secret and isSecret; the basic-auth
example uses a provider; the last trace cache wording is replay cache.

* chore(changeset): leave released notes alone; pending notes match the renames

Released CHANGELOG entries keep TraceCacheStore, the name they shipped
with; cache-store-rename.md announces the rename. The pending video and
provider recording changesets name on-all-retries and web({ screencast }).

* chore(bench): refuse a forwarded --output the harness would not clean up

* fix(secrets): a rewritten trace drops its screencast frames, engine-held secrets included
2026-09-29 19:51:02 +02:00
Oskar Kwaśniewski aadcb5dddb feat(run)!: video modes on the config, targets, the CLI, and tests, with provider recordings (#610)
* feat(run)!: video modes on the config, targets, the CLI, and tests, with step captions

video: off | on | retain-on-failure | on-first-retry at the config root, per
target, --video [mode], and per test or describe, innermost winning.
artifacts.kinds: ['video'] and artifacts.video are removed. Kept recordings
get WebVTT step captions; a video can be a link to a hosted recording. The
engine contract's VideoSegment is a file or a link, and e2e/engine exports
the provider recording seam.

* feat(web): a browser provider records the attempts it serves, and the screencast takes a size and quality

BrowserProvider.record starts the service's own recording of the browser in
place of the page screencast. The session drives one AttemptVideo, the
screencast or the provider's. web({ video: { size, quality } }) tunes the
screencast.

* feat(mobile): a device provider records the attempts it serves

DeviceProvider.record starts the service's own recording of the leased
device in place of agent-device's. The worker learns its slot's lease id,
so record gets the lease as it traveled.

* feat(integrations): Kernel replays record the attempts that record video

* fix(run): captions never fail a video, provider values are normalized, and one list of modes

- Step captions create their directory and are best-effort.
- A provider's recording is checked at start in both engines and its URL
  and start time are stored in the shapes the report schema admits; the
  list reporter bounds a hosted URL.
- --video no longer hides an invalid target mode.
- A describe inside a serial group cannot set video.
- A mobile provider may keep its own leaseId field on a lease.

* fix(mobile): keep the lease type narrowed when checking its declared fields

* fix(run): MCP recordings get their step captions, a step spanning a restart is captioned in both recordings, and video.size refuses unknown keys

* fix: address review on the video API

- One stop in flight per provider recording, shared by a retry; mobile
  dispose stops it, and a cancelled operation never starts one.
- Kernel retries a failed download without stopping the replay twice.
- Links from any engine are held to an http(s) URL with a host; a file a
  provider names must be a regular file; media types are trimmed.
- web({ video }) takes a plain object only; the markdown page labels a
  hosted artifact by its kind.
- Tests: fixture byte totals, captions linked by id, temp directories
  removed, signals asserted, empty sizes, non-object options.
- Docs: engine support qualified, the explore row, the standalone preflight,
  caption status text, a checked stop in the provider example.

* fix(run): a bad engine video link costs no other recording, captions path per recording in debugging guide

* test: fake engine video links doc matches segment order

* fix(mcp): session recordings follow video modes and list a provider's link

* test(web): provider test's fake page reports its viewport

* refactor(run): drop step captions from the video API, tracked in #636

* fix(web): a late provider start never replaces a newer recording, abandon keeps a failed stop's handle, video.size must be a plain object; provider recording errors and hosted URLs bounded
2026-09-29 15:24:26 +02:00
Oskar Kwaśniewski bad98ea2eb fix(runner): a trace from a secret-filled attempt drops its screencast frames (#572) 2026-09-28 17:00:46 +02:00
Oskar Kwaśniewski c089fa6cc2 fix(runner): a download is redaction incomplete unless the runner rewrote it (#561)
* fix(runner): a download is redaction incomplete unless the runner rewrote it

Every download artifact was recorded redaction: 'complete' without the
runner reading a byte of it. Fill a field through a secret, have the app
serve an export that contains the same value, and the file landed in the
attempt's artifacts as complete, and an artifacts.store received the
plaintext with metadata that said it was clean. A store exporting only
complete artifacts would have exported the secret. Screenshots on the same
session were already withheld by the pixel taint.

A download now defaults to 'incomplete', the label a video carries: bytes
the app served that the runner did not rewrite. After a secret fill on the
session, a text-like download (.txt, .csv, .json, .html) is rewritten
through the session's secret ledger before it is hashed or stored and is
labelled 'complete', the way trace text entries are. A binary or a file
that is not UTF-8 stays as served and 'incomplete'. StoredArtifact carries
the redaction the report records, so a store can act on it.

* fix(runner): a rewritten download keeps its byte order mark

TextDecoder drops a leading UTF-8 BOM by default, so a CSV served with one
and rewritten through the ledger lost its encoding marker. Decoding with
ignoreBOM keeps it as part of the text and the write puts it back.

* fix(runner): redactor matches a value csv-quoted

A CSV field doubles each double quote, so a secret holding one survived the ledger in a text download that was then labelled complete. The ledger now spells a double quote doubled too.
2026-09-26 20:59:50 +02:00
Oskar Kwaśniewski ed151e0420 fix(agent): a keyboard fallback is not blamed for an unlisted change, and an inconclusive judgment points at vision (#449) 2026-09-23 14:19:57 +02:00
Oskar Kwaśniewski 8bfe1d84f4 fix(mobile): device timeouts keep their message, a busy or wedged runner names the recovery (#443) 2026-09-23 13:14:09 +02:00
Oskar Kwaśniewski fb6e336327 fix(report): the live block fills the terminal and console output prints above it (#366) 2026-09-20 14:15:05 +02:00
Oskar Kwaśniewski f09b69e90a docs: simplify onboarding and clarify model access (#346) 2026-09-18 11:52:06 +02:00
Oskar Kwaśniewski 5908a107f9 feat(report): a failure hands off what the screen held, a page per failed test, and the line to look at (#299) 2026-09-14 15:05:20 +02:00
Oskar Kwaśniewski abf1958867 feat(engine)!: lock the engine contract as a UI-only, cross-platform SPI (#291) 2026-09-14 08:44:46 +02:00
Oskar Kwaśniewski aa3b05bbbd feat(agent): a judgment sees the screen and nothing else, can run on its own judge model, and answers inconclusive instead of guessing (#288) 2026-09-12 18:08:48 +02:00
Oskar Kwaśniewski ff33218345 docs: e2e is agentic testing first; quickstart for web and mobile; icons everywhere (#282) 2026-09-12 13:52:59 +02:00
Oskar Kwaśniewski 07cae8787c docs(cli): run the CLI as npx e2e; drop --no-install everywhere (#275) 2026-09-12 11:07:20 +02:00