Commit Graph
68 Commits
Author SHA1 Message Date
Krzysztof Moch dc2afca0c0 feat(mcp)!: e2e mcp is headless by default, and open_session takes headed (#851)
* feat(mcp)!: e2e mcp runs headless by default and takes --headed, like run

Sessions were headed outside CI and took --headless, the opposite of
e2e run and e2e explore. --headless stays accepted and hidden: it now
asks for the default, and a refused flag would reach an MCP host as
nothing but a failed connection.

* docs(skill): --headed shows the UI when the engine supports it

* feat(mcp): open_session takes headed, so the agent picks per session

The server's --headed sets the default for sessions that do not say,
and headed: true or false overrides it, so a user who asks to watch
needs no change to the MCP client config.
2026-10-05 12:58:11 +02:00
ammann 2e26ec6551 fix(cli): forward flags in npm exec hints (#847) 2026-10-05 12:57:09 +02:00
Oskar Kwaśniewski 0d1ed15871 fix(e2e)!: load TypeScript with e2e's own oxc loader, so pnpm 11+ installs pass (#790)
* fix(e2e): load TypeScript with an oxc loader instead of tsx

* fix(e2e): close loader edge cases: sourceless CommonJS, export maps without extensions, unreadable tsconfig, chained source maps, Windows require paths

* fix(e2e): refuse Bun and Deno runtimes, name oxc load failures, follow Yarn PnP export targets

* perf(e2e): skip the syntax check for files whose only @ is in a package import

* fix(e2e): allow type-only exports in .cts, warn from runner only, Bun check first, docs

* fix(e2e): workers warn about tsconfigs runner did not, FILE: urls, foreign infra errors, review tests
2026-10-04 09:22:50 +02:00
Filip c66176bd04 feat(oauth): sign in with OpenCode Console for Zen and Go models (#794)
* feat(oauth): sign in with OpenCode Console for Zen and Go models

* fix(oauth): keep the Console refresh token, isolate tests from OPENCODE_API_KEY

A refresh grant without a refresh_token now keeps the previous one, as
the ChatGPT and SpaceXAI providers do. The vendor test helper blanks
OPENCODE_API_KEY so a key in the shell no longer replaces the stored
login under test. Docs and changeset now say the models listing covers
Zen and Go only, and that MISCONFIGURED needs a readable workspace config.
2026-10-03 18:39:18 +02:00
Oskar Kwaśniewski 3a9667c046 fix(run): keep --last-failed reruns honest: failure-limit skips, carried debts, kept evidence (#724)
* fix(run): keep --last-failed reruns honest: failure-limit skips, carried debts, kept evidence

--last-failed now reruns tests --max-failures skipped. A test or failed
suite hook another filter leaves out stays owed in the report's new
run.carried until a rerun runs it; the github fold shows carried rows and
errors and stays red while anything is carried. A rerun prunes artifacts/
to what the previous report names and writes its attempts under
artifacts/rerun-<n>/ instead of wiping the tree and reusing attempt paths.

* fix(run): carry a setup's hook debt when no selected test needs it

The report has no row for an unneeded setup, so carryForward now asks the
collection whether a test still exists instead of the report's rows.

* fix(run): reject malformed artifacts in --last-failed reports, tighten rerun docs

Distinct ids in the carried fixture row.

* fix(run): keep carrying tests of a file a narrowed rerun could not collect

* fix(github): a carried interrupted test reads interrupted, not failed, in the folded page
2026-10-03 16:36:53 +02:00
6639bfad71 fix(oauth): reach Copilot models served only over the Responses API (#779)
* fix(oauth): reach Copilot models served only over the Responses API

copilot() reads the plan's model listing on the first call and picks
chat completions or the Responses API for each model, falling back to
chat whenever the listing cannot be read so nothing that worked changes.
sendCopilotRequest reads the initiator and vision headers from a
Responses body (input[]/input_image) as well as a chat one, and the model
listing marks an uncallable model 'no chat or responses'.

* fix(changeset): schedule a minor release for the Responses endpoint support

The changeset asked for a patch while the change adds a call path a model could not
reach before; a minor bump is what the release intent states.

Addresses the cubic review comment on .changeset/copilot-responses-endpoints.md.

* fix(oauth): cancel and never cache a Copilot endpoint lookup that was aborted

The first call read the plan's model listing with no signal, and the delegate cached
whatever that produced. A listing that stalled therefore left `pending` unresolved and
every retry waited on the same promise instead of reaching the model. The signal now
reaches the lookup, an aborted lookup rejects instead of being read as a missing
endpoint, and a choice that could not be made clears `pending` so the next attempt
asks again. A test pins the aborted lookup and the missing-endpoint read as distinct.

Addresses the cubic review comment on packages/e2e/src/oauth/copilot.ts.

* docs(skill): separate Copilot models served over Responses from uncallable ones

The setup reference read as if every non-chat model is reached over the Responses API.
Only an enabled model the plan serves solely over `/responses` is; a model served over
an API `copilot()` does not speak, or one the plan has not enabled, still cannot be
called, and the listing marks those.

Addresses the cubic review comment on skills/e2e/references/setup.md.

* fix(oauth): pick Copilot endpoint from supported_endpoints alone, retry unreadable listing

- drop policy gate: grok-4.5, gpt-5.3-codex serve /responses with no policy
- unreadable listing: call over chat, not cached, asked again next call
- listing has own 10s timeout; one caller abort no longer fails others
- revoked login fails on listing with sign-in hint, no second 401
- missing @ai-sdk/openai names the package to install

* fix(init): install @ai-sdk/openai for the Copilot gateway

* docs(copilot): unreadable listing retry, init installs both peers

---------

Co-authored-by: Hermes Agent <agent@localhost>
Co-authored-by: Feco Linhares <fecolinhares@users.noreply.github.com>
Co-authored-by: Oskar Kwaśniewski <oskar@okwasniewski.com>
2026-10-03 13:34:41 +02:00
Oskar Kwaśniewski 7ab80bc610 fix(report)!: count interrupted tests apart from failures (#713)
* fix(report)!: count interrupted tests apart from failures

* fix(report): tell a failed test from its last failed attempt, whatever the cut retry left
2026-10-03 09:30:44 +02:00
Oskar Kwaśniewski 5609f5e5f0 fix(oauth): mark Copilot models copilot() cannot call (#767)
* fix(oauth): mark Copilot models copilot() cannot call

* test(oauth): pin both Copilot marks on one model
2026-10-02 16:58:04 +02:00
Oskar Kwaśniewski 4d488f8286 feat!: the app is declared on its target; engines only drive it (#698)
* feat!: the app is declared on its target; engines only drive it

The app under test moves from the engine's options to `targets: [{ engine,
app }]`: `url`, `bundleId`, `appPath`, `identity`, `environment`,
`launchArguments`, `permissions`, `command`, and `readyUrl`. `web()` and
`mobile()` refuse every moved option in one `INVALID_CONFIG` naming the
target block; `defineEngine({ app })` is gone and engines check what their
platform needs in `validateApp`. `requires: ['browser' | 'native-app']` is
derived from the target's app. Replay cache and session keys are byte for
byte what they were, pinned by target-identity.test.ts.

Dependency services are not part of this version: `services` anywhere, and
`web({ services })`, fail saying so. `app.command` on port 0 without a
command, and `reuseExisting` on a free port, are refused. An `e2e mcp`
open cancelled or disconnected mid-start stops what it started, and a hung
shared start is abandoned once no session waits on it.

* fix: review follow-ups: cancelled opens, abandoned starts, null targets, stale docs

* refactor: drop the moved-option messages; old app keys are unknown keys

* test: give the replaced-nodes suite a budget that survives a loaded runner

* test: size the replaced-nodes hook budget to its seven tests
2026-09-30 19:48:00 +02:00
devin-ai-integration[bot]andoskar 256513dc0b docs: sync the package manager choice across every command snippet (#679)
Co-authored-by: oskar <oskar@okwasniewski.com>
2026-09-30 14:40:08 +02:00
Oskar Kwaśniewski 10d392ea3c fix(config): follow-ups to the config redesign: output data loss, engine-held secrets, migration messages, agents (#662) 2026-09-30 14:04:23 +02:00
Oskar Kwaśniewski f575e5dec7 feat(cli): e2e feedback sends a report about e2e to the team (#678)
* feat(cli): e2e feedback sends a report about e2e to the team

One PostHog event with the report and anonymous machine facts; secret-named
env values and token shapes redacted first. --dry-run prints it. A saved
telemetry disable does not stop it; E2E_TELEMETRY_DISABLED and DO_NOT_TRACK do.

* docs(cli): feedback redacts secret-named values of six characters or more

* fix(cli): feedback counts limits in characters, never claims an unconfirmed report was lost
2026-09-30 13:04:08 +02:00
Oskar Kwaśniewski f67c49a355 fix: Ctrl-C, --max-failures, and e2e mcp client lifecycle (#664)
* fix(web): leave Ctrl-C to the runner, not Playwright's signal handlers

* fix(run): a worker killed by Ctrl-C while starting is not a boot failure

* fix(run): stop a running file at --max-failures without waiting for the interrupt

* fix(mcp): claim a session's config before evaluating it

* fix(mcp): close sessions and exit 0 when the client goes away

* fix(agent): lead a failed action with the attempt and its code; MCP isError

* fix(run): excuse only the workers one Ctrl-C reached, so a signalled boot loop still fails

* fix(web): keep Playwright's SIGHUP handler; it closes the browser on a hangup
2026-09-30 09:16:47 +02:00
Oskar Kwaśniewski 6b851bb604 docs: fix stale and wrong claims in guides (#653)
* docs: fix stale and wrong claims in guides

- output: --artifacts -> --output in bug-bash guide and skill; sessions and
  cache locations follow output / cache.dir in security; artifact path
  gains the agent segment in github
- ci: upload artifacts on !cancelled() in ci guide and skill
- mcp: several sessions (--max-sessions) in security
- replay cache naming: "Commit the replay cache" heading and anchors
  (old anchor kept, e2e init links the new one),
  trace replay -> replay cache in writing-an-engine and executors
- timeouts: STEP_TIMEOUT / STEP_BUDGET_EXHAUSTED advice names config
  timeout, judgmentTimeout, maxSteps, maxModelCalls (debugging, skill)
- mobile: app.open() in device examples and quickstart, mobileTools pack
  all mutating, Android toolbar anchor, drop agent-device release note
- executors: verdict code rules, missing ctx members and scrollUntil,
  a replay cache section; agent-steps passed+code rule
- migrate: swipe direction is the content's, `!` test exclusions,
  video row status, Kernel for Selenium Grid
- secrets: an unset env string fails config load, a provider's value is
  redacted only from its first fill; three secret sinks
- misc: Models link text, telemetry links outbound connections, integration
  config wording, web-only expect(web) in a cross-platform block,
  capabilities heading in engine reference, swipe direction in screen reference

* docs(skill): separate the credential redaction paragraph

* docs(reference): KERNEL_API_KEY names @e2e-dev/kernel
2026-09-29 23:38:52 +02:00
Oskar Kwaśniewski 8904b02ab5 docs(reference): fix stale and missing claims, sync JSDoc (#655)
* docs(reference): fix stale and missing claims, sync JSDoc

* docs(reference): only the built-in agent needs a model to acquire the fixture
2026-09-29 22:53:18 +02:00
Oskar Kwaśniewski d228e22ea3 feat(agents)!: plain-object agents entries; createAgent and limits removed (#646)
* feat(agents)!: plain-object agents entries, createAgent and limits removed

Each agents entry is one AgentConfig object: model, judge, system, context,
tools, maxSteps, maxModelCalls, judgmentTimeout, maxObservationBytes,
maxInputTokens, providerOptions, or executor for a custom brain (system and
tools rejected beside it). No inheritance between agents.

- createAgent() kept as a stub throwing INVALID_CONFIG; bare StepExecutor or
  legacy createAgent value as an entry fails naming { executor } / the object
- timeout -> judgmentTimeout, maxTurns removed (entry and
  createToolLoopExecutor), unknown keys get did-you-mean
- top-level limits removed; maxModelTokensPerCall -> per-agent
  maxInputTokens, the rest internal constants; report-1 run.limits keeps
  its shape, filled from max across agents plus the constants
- tools validated at config load; built-in agent built from the entry at
  fixture time; explore and the MCP catalog read the entry's tools
- migrate apps, docs, docs examples, init scaffold, skills, mobile README

* docs: AGENTS.md names the agent entry's model, not createAgent

* refactor(agents)!: delete createAgent instead of a throwing stub

* docs: MODEL_UNAVAILABLE scoped to the built-in agent, fixture-acquisition timing
2026-09-29 19:51:02 +02:00
Oskar Kwaśniewski d7a218af07 feat(config)!: config shape, part 1: recording modes, output root, test-glob exclusions, renames, secrets in engine options (#645)
* feat(config)!: remove specVersion from the config

The runner version is the format version. A config that still sets
specVersion fails INVALID_CONFIG telling the author to remove it. The
report's run.specVersion and the replay cache key keep their hard-coded
copies.

* feat(collect): "!" exclusions in the tests globs

An entry starting with "!" removes files an including glob matched,
in any order. Discovery never reads a directory an exclusion takes
whole, unless an including glob could reach a dot name beneath it.
A list of only exclusions is INVALID_CONFIG.

* refactor!: CacheStore and web({ screencast }) renames

TraceCacheStore is CacheStore, and the docs, skill, and CLI call the
cache the replay cache. web({ video: { size, quality } }) is
web({ screencast }) with WebScreencastOptions; web({ video }) fails
INVALID_CONFIG naming screencast.

* feat(secrets): secrets.get() in engine options, resolved per attempt

At config evaluation, before any run exists, secrets.get(name) returns
a reference by name. An engine declares the handles its options hold
in defineEngine({ secrets }); the config load fails INVALID_CONFIG for
a name no secret or credential has. The engine reads the value through
EngineAttemptContext.resolveSecret, which runs a provider fresh and
registers the value in the session and process ledgers before it
returns, without tainting the viewport. e2e/engine exports isSecret.

web({ basicAuth: { password } }) takes a Secret and declares it. Docs
show process.loadEnvFile('.env') in the config.

* feat(run)!: trace and video share recording modes; artifacts holds only store

One RecordingMode ('off' | 'on' | 'retain-on-failure' | 'on-first-retry' |
'on-all-retries') for trace and video, set on the config, a target, the CLI
(--trace [mode], --video [mode]) and a test, one precedence rule: config,
target, CLI, test. Trace defaults to on locally, on-first-retry in CI, with
a plan-time notice when a retry mode meets retries: 0. Serial members cannot
set trace.

Capability rule: a mode from the config root or a flag applies to targets
whose engine can record it, one notice naming the rest; a mode set on a
target or a test is required and fails with UNSUPPORTED_ARTIFACT.

artifacts.kinds (both forms) and artifacts.trace are removed and fail
INVALID_CONFIG naming the trace mode they meant. Failure screenshot and
screen text are always captured when the engine can. MCP sessions keep the
target's trace mode and trace only under 'on'.

* feat(run)!: output directory for every run result; ArtifactStore.putLink

output (default .e2e) and --output <dir> on run and explore hold report.json,
junit.xml, summary.md, ai-trace.json, artifacts/, sessions/ and the e2e mcp
session files; --last-failed reads <output>/report.json. The reportSibling
derivation from the artifacts dir is gone. --artifacts stays as a hidden
flag that fails with the --output it maps to.

A run clears <output>/artifacts at start. cache.dir stays independent. An
output that is the project root or outside it, is or sits in cache.dir,
would hold cache.dir under artifacts/, sessions/ or videos/, or holds a
tests glob root fails INVALID_CONFIG with the reason.

ArtifactStore gains optional putLink(link) for provider-hosted video links;
its ref lands beside the url. A passed retain-on-failure attempt's link is
never handed over, and a failing putLink never fails the run.

bench-ab and the mobile workflow move to --output.

* test(secrets): a hold's registry is probed by an unknown name, now that get() defers without one

* test(web): the download trigger's measured wait tolerates a timer firing early

* fix(secrets): rewrite the trace of an attempt whose engine holds a secret

A Playwright trace records the context options, basic-auth password in
plaintext, but a trace was rewritten only after a fill. resolveSecret now
marks the session so its trace goes through the redactor, screencast
frames kept since the value never reached the screen. The fake engine
writes a real trace archive, and the basic-auth fixture escapes its echo
and refuses a malformed header.

* fix(web): race the basic-auth secret with the attempt's signal

A provider still resolving after the launch timeout set the session after
the runner ended the attempt, so the retry failed with an attempt already
running. Resolution now runs first, raced with the signal.

* fix(mcp): a session notes a configured trace its engine cannot record

* test: a shadow-free fake trace archive and an ungarbled exclusion title

* docs: output paths, replay cache wording, and review corrections

--last-failed, --ai-trace, and the json reporter name <output> paths;
the skill says output never moves cache/ or the app's log; the
INVALID_GLOB row states the check the code makes; serial members list
only; the engine vocabulary names Secret and isSecret; the basic-auth
example uses a provider; the last trace cache wording is replay cache.

* chore(changeset): leave released notes alone; pending notes match the renames

Released CHANGELOG entries keep TraceCacheStore, the name they shipped
with; cache-store-rename.md announces the rename. The pending video and
provider recording changesets name on-all-retries and web({ screencast }).

* chore(bench): refuse a forwarded --output the harness would not clean up

* fix(secrets): a rewritten trace drops its screencast frames, engine-held secrets included
2026-09-29 19:51:02 +02:00
Oskar Kwaśniewski ec1ea1e4c3 feat(mcp): parallel sessions and a bug-bash skill topic (#634) 2026-09-29 17:17:07 +02:00
Oskar Kwaśniewski 8a65a9093d feat(explore): --session starts the exploration signed in (#641)
* test(testbed): admin session saved from a cookie, no secret fill

A setup that signs in without filling a secret carries no pixel taint, so tests and explorations restoring it keep screenshots.

* feat(explore): --session starts the exploration signed in

Explore collects the config's test files unselected, so the runner's own session selection runs exactly the setup that declares the name, then restores it into the exploration. In-process runs adopt the in-memory registration only for its virtual file; setup files import as usual. Planner and explorer are told they start signed in. A missing session producer now lists declared sessions with did-you-mean. Docs: a restored session keeps its setup's taint (authentication.mdx claimed otherwise).

* fix(explore): a --session setup runs as the project's agents

The explorer replaced the configured agent for the whole run, so an agentic setup pulled in by --session got the exploration rules, report_finding, explorer budgets, and cache off. In-memory tests now carry their own agents (InMemoryAttempts) and run with the trace cache off; setups keep the config's agents, cache, and retries. Uncollected-file notice no longer mentions positionals under explore; consumesSession drops the unreachable group walk.

* docs(explore): qualify app.open and screenshot claims, fix errors.mdx clause
2026-09-29 16:31:02 +02:00
Oskar Kwaśniewski aadcb5dddb feat(run)!: video modes on the config, targets, the CLI, and tests, with provider recordings (#610)
* feat(run)!: video modes on the config, targets, the CLI, and tests, with step captions

video: off | on | retain-on-failure | on-first-retry at the config root, per
target, --video [mode], and per test or describe, innermost winning.
artifacts.kinds: ['video'] and artifacts.video are removed. Kept recordings
get WebVTT step captions; a video can be a link to a hosted recording. The
engine contract's VideoSegment is a file or a link, and e2e/engine exports
the provider recording seam.

* feat(web): a browser provider records the attempts it serves, and the screencast takes a size and quality

BrowserProvider.record starts the service's own recording of the browser in
place of the page screencast. The session drives one AttemptVideo, the
screencast or the provider's. web({ video: { size, quality } }) tunes the
screencast.

* feat(mobile): a device provider records the attempts it serves

DeviceProvider.record starts the service's own recording of the leased
device in place of agent-device's. The worker learns its slot's lease id,
so record gets the lease as it traveled.

* feat(integrations): Kernel replays record the attempts that record video

* fix(run): captions never fail a video, provider values are normalized, and one list of modes

- Step captions create their directory and are best-effort.
- A provider's recording is checked at start in both engines and its URL
  and start time are stored in the shapes the report schema admits; the
  list reporter bounds a hosted URL.
- --video no longer hides an invalid target mode.
- A describe inside a serial group cannot set video.
- A mobile provider may keep its own leaseId field on a lease.

* fix(mobile): keep the lease type narrowed when checking its declared fields

* fix(run): MCP recordings get their step captions, a step spanning a restart is captioned in both recordings, and video.size refuses unknown keys

* fix: address review on the video API

- One stop in flight per provider recording, shared by a retry; mobile
  dispose stops it, and a cancelled operation never starts one.
- Kernel retries a failed download without stopping the replay twice.
- Links from any engine are held to an http(s) URL with a host; a file a
  provider names must be a regular file; media types are trimmed.
- web({ video }) takes a plain object only; the markdown page labels a
  hosted artifact by its kind.
- Tests: fixture byte totals, captions linked by id, temp directories
  removed, signals asserted, empty sizes, non-object options.
- Docs: engine support qualified, the explore row, the standalone preflight,
  caption status text, a checked stop in the provider example.

* fix(run): a bad engine video link costs no other recording, captions path per recording in debugging guide

* test: fake engine video links doc matches segment order

* fix(mcp): session recordings follow video modes and list a provider's link

* test(web): provider test's fake page reports its viewport

* refactor(run): drop step captions from the video API, tracked in #636

* fix(web): a late provider start never replaces a newer recording, abandon keeps a failed stop's handle, video.size must be a plain object; provider recording errors and hosted URLs bounded
2026-09-29 15:24:26 +02:00
Oskar Kwaśniewski 968b05545d fix(collect): a narrowed run skips an unselected file that fails to import (#615) 2026-09-29 11:26:12 +02:00
Oskar Kwaśniewski 04a126191e feat(mcp): record video of a live session on demand (#612)
* feat(mcp): record video of a live session on demand

start_recording and stop_recording in the session catalog when the
engine records video. Files move to .e2e/videos/<session>/ on stop;
close_session saves one still running. Sessions no longer record from
launch when the config asks for video.

* fix(mcp): serialize recorder calls, retry a failed stop safely

Review: close could race an in-flight start or stop, and a failed file
move lost the recording. Calls now queue; a stopped recording stays
until its files move. Recording tools lose the read-only mark.

* fix(mcp): queue the close's save behind a start in flight

Review: close checked isRecording first, which is false while a start
is still pending, so that recording ran past the close.
2026-09-29 10:36:39 +02:00
Oskar Kwaśniewski 67c29fc6b2 chore!: publish engines and reporter under the @e2e-dev npm scope (#602) 2026-09-28 19:05:05 +02:00
Oskar Kwaśniewski 90356f17fc fix(e2e): --last-failed reruns the scope of a failed suite hook (#576) 2026-09-28 16:22:44 +02:00
Oskar Kwaśniewski 75e414536d feat(report): --repeat-each ends with a pass rate per test (#589) 2026-09-28 16:05:39 +02:00
Oskar Kwaśniewski d6947bc66d feat(cache): --strict-cache fails a recording that no longer replays (#586) 2026-09-28 16:00:45 +02:00
Oskar Kwaśniewski cef69d2609 feat(cli): the e2e wordmark drops in on init and opens the help (#505) 2026-09-28 15:17:26 +02:00
Szymon Rybczak c1547c9deb fix(cli): quieter e2e init: no telemetry notice, planned changes listed one per line (#520) 2026-09-25 13:09:53 +02:00
Oskar Kwaśniewski d6a1a30519 feat(cli): init writes the skill once, .claude/skills/e2e links to the .agents/skills copy (#519) 2026-09-25 11:24:13 +02:00
Oskar Kwaśniewski 943de73404 fix(run): an interrupt reports every unstarted test once, so --last-failed can resume it (#431) 2026-09-23 09:54:36 +02:00
Oskar Kwaśniewski edbb84f6a1 docs: the migration tables, mobile README, and skill match what ships (#436) 2026-09-23 09:53:37 +02:00
Oskar Kwaśniewski 7aa0ffe0b2 fix(cli): init never writes the agent skill through a symlink (#453) 2026-09-23 08:27:05 +02:00
Oskar Kwaśniewski f01c01fbe3 feat(run): run every test n times with --repeat-each (#391)
`e2e run --repeat-each <n>` (`RunOptions.repeatEach`) runs every selected
test that many times. The repeat is a dimension of a pair's identity beside
the test, the target, and the agent: `TestTargetPair.repeat`, the wire pair,
the pair-start and progress messages, the scheduler's keys, and the report
result and serial group all carry it, 0 for the run every test has. The
first run's result id is unchanged; later runs get their own, the list
reporter, JUnit, and the failure pages title them `(repeat #n)`, and their
artifacts go under `repeat-<n>`. Setup tests run once; a filtered or skipped
pair is reported once. Run events `test-started` and `step` carry `repeat`.
`--last-failed` now keys the previous run's results by test, target, and
agent, so a test is named once whichever of its repeats did not pass.
2026-09-22 13:12:51 +02:00
Oskar Kwaśniewski 2e4d40eb07 feat(run): stop a run at --max-failures (#390)
`e2e run --max-failures <n>` (`RunOptions.maxFailures`) stops the run once
that many tests have failed or timed out. The scheduler counts every result
it reports, skips its queues with the new cause `failure-limit` and the
reason `run stopped after 3 failures (--max-failures 3)`, tells the runner,
and interrupts its workers the way the interrupt signal would; the interrupt
message now carries that skip as the abort reason, so the pairs a worker had
not started say why instead of `run interrupted before execution`. The
running tests end as `interrupted`, and the exit code is the failures' own,
never 130. Reporters get a `run-stopped` event, which the list reporter
prints. The report schema's skip causes gain `failure-limit`.
2026-09-22 13:12:51 +02:00
Oskar Kwaśniewski ae930bbf9a feat(cli): rerun with --last-failed and split a run with --shard (#389)
`--last-failed` selects only the tests the previous run did not pass, read
from the `report.json` this run overwrites (beside the artifact root):
failed, timed out, interrupted, or skipped because a setup, a serial
predecessor, a hook, or the worker failed, each named per target and agent
by its result id. No report there is `NO_LAST_RUN`; a run with nothing
failed selects nothing. `--shard <index/total>` runs one contiguous slice of
the selected tests, cut once every other filter and the serial closure
applied, the first shards taking the remainder, so a serial group is never
split and each shard brings only the setup tests it needs. `e2e list` takes
both. `RunOptions` and `ListOptions` carry them as `lastFailed` and `shard`.
2026-09-22 13:12:50 +02:00
Oskar Kwaśniewski dcfe53a4b0 feat(cli): select one test with a file:line positional (#388)
Any positional form may end in `:line` (`tests/signup.e2e.ts:12`,
`signup:12`) to select only the test whose `test(` call opens on that line
of each file it names. Several lines of one file add up, a file named
without a line is selected whole, and a serial group runs whole when a line
names one member. A positional that exists on disk as written stays a path.
When a named line declares no test, `NO_TESTS` names the positional and the
lines the file declares tests at. The mapped-line check spawns the built
CLI, as the source test does, since a vitest fork reports raw positions.
2026-09-22 13:12:50 +02:00
Oskar Kwaśniewski d043035236 feat(cli): select tests with --exclude-tag, --grep, and --grep-invert (#387)
`e2e run` and `e2e list` take three more selection flags, shared through one
option group. `--exclude-tag` leaves out tests carrying any of the tags
whatever `--tag` or a positional selected; a serial group still runs whole
when any member is selected. `--grep` keeps tests whose title matches one of
the patterns and `--grep-invert` leaves out the ones that match; the text is
the describe titles and the test title joined by one space, a bare pattern or
`/pattern/flags`, repeated for alternatives, and one that does not compile is
a usage error. `RunOptions` and `ListOptions` carry them as `excludeTags`,
`grep`, and `grepInvert`. `NO_TESTS` names the tags and patterns that left
nothing to run.
2026-09-22 13:12:49 +02:00
Oskar Kwaśniewski f5e96d0cbe feat!: rename engine packages to @e2edev/web and @e2edev/mobile (#392) 2026-09-21 19:52:32 +02:00
Oskar Kwaśniewski 9f2b73f929 feat(oauth)!: fold @e2edev/oauth into e2e as e2e/oauth, add e2e models, fix Codex second turns (#377) 2026-09-21 19:09:57 +02:00
Szymon Rybczak e9c2914c6d fix(docs): repair links to the published documentation (#367) 2026-09-20 14:36:38 +02:00
Oskar Kwaśniewski f09b69e90a docs: simplify onboarding and clarify model access (#346) 2026-09-18 11:52:06 +02:00
Oskar Kwaśniewski 15081f3068 feat(oauth): sign in with a ChatGPT, GitHub Copilot, or SuperGrok subscription (#335) 2026-09-17 23:34:10 +02:00
Oskar Kwaśniewski ba8d9ba81d fix(collect): glob forms that match nothing are INVALID_GLOB (#330)
* fix(collect): glob forms that match nothing are INVALID_GLOB

A tests glob or a positional glob that could never match a file now fails with INVALID_GLOB and a hint: an absolute path, a trailing slash, braces, a backslash separator, or a .. segment. A leading ./, a . segment, and a doubled slash are dropped, so ./tests/**/*.e2e.ts selects what tests/**/*.e2e.ts does instead of matching nothing, walking .git and .e2e, and pointing the NO_TESTS message at a missing example file.

* fix(collect): review fixes for the glob grammar

Segments without wildcards compile to names, so the near-miss scan reads a glob's leading directory off its segments (literalPrefix) and the positional glob test is the grammar's own GLOB_SYNTAX; the pattern string seam is gone. Character classes and extglobs are rejected with hints like braces. A trailing '.' and a normalized leading '!' are caught. An existing path wins over glob classification for positionals, so a file whose name has braces is still selectable by path. Messages name the fix; docs and the changeset describe positionals as the code treats them; the near-miss regression has a test.
2026-09-17 22:44:09 +02:00
Oskar Kwaśniewski ed3999e893 fix(cli): --tag takes comma lists and NO_TESTS names the nearest tag (#332)
* fix(cli): --tag takes comma lists and NO_TESTS names the nearest tag

--tag accepts a comma-separated list as well as repeats, like --target and --agent, and an empty value is a usage error instead of a run with the filter dropped. When the filter leaves nothing to run, NO_TESTS reads correctly under --tag-mode all and marks each tag no test declares with the nearest declared tag.

* fix(cli): one parser for the list-valued flags, tag blame only where it applies

Review fixes: --tag, --target, and --agent share parseNames (comma lists or repeats, each name once, empty is a usage error); NO_TESTS counts positional and .only exclusions as such instead of tag mismatches, computes the tag description once, and reads mode-neutral help; TagMode is one exported type; the did-you-mean note is one helper.
2026-09-17 22:41:20 +02:00
Oskar Kwaśniewski 95b3d7f419 feat(report): tags travel with a test into e2e list and the report (#333)
* feat(report): tags travel with a test into e2e list and the report

The tags a test declares are derived once at collection onto its identity, so selection, the wire, e2e list (text and json), and the report all read the same list. e2e list prints them as #tag after the target; run.results[] carry tags when a test declares any, so a reporter can group by tag. Report schema: an optional tags array of distinct non-empty strings on a result.

* feat(report): review fixes for tags on the identity

Registration derives a test's tags beside its title path, so collect.ts no longer walks the option layers a second time; ResolvedTestOptions drops its copy and the filter reads the identity. Results always carry tags, an empty list when none, like every other array in report-1; the schema pins the tag-name rule with a pattern and a targeted test. e2e list bounds titles and tags like every untrusted field. Stacked on the validation change so the schema promise holds.
2026-09-17 22:41:19 +02:00
Oskar Kwaśniewski d3afa6bacb feat(runner): single-file setup selection, runtime skip, more matchers (#325)
* feat(runner): single-file setup selection, runtime skip, more matchers, templated cache params

Four gaps a real suite hit on its first day:

- `e2e run login.e2e.ts` failed with COLLECTION_ERROR when the session it
  consumes is produced by a setup test in another file. Collection now
  imports every discovered file and positionals narrow the selection
  instead of the import set; tests in unnamed files are report-only
  unselected results, and a consumed session promotes its setup wherever
  it lives.
- `test.skip(condition, reason)` inside a body skips the running test for
  a fact only the app can tell. Steps that ran stay in the report, teardown
  runs, no retry is spent, and a setup test cannot skip. The signal is
  recognized by brand so a test module realm with its own copy of `e2e`
  still skips instead of failing. Report schema: a skipped attempt may
  carry steps and a skip reason.
- `expect(actual, message)` opens value failures with the caller's label;
  `toBeGreaterThanOrEqual`, `toBeLessThanOrEqual`, and `toBeCloseTo` join
  the value matchers; `getByRole` takes `level` for headings.
- Run-unique string params no longer defeat the trace cache. String params
  of three or more characters are templated in the recording (typed text,
  target descriptors, end anchors, summaries) and filled at replay from the
  current call, and the key digests the params' shape rather than their
  values. Numbers, booleans, short strings, and secret names stay literal.

* chore: changesets, type assertions, and level scope note for the runner fixes

* fix(runner): review follow-ups for skip, focus, and templated params

- A runtime skip keeps the suite realm: only failed, timed-out, and
  interrupted attempts discard it, so beforeAll runs once per suite.
- A serial member that skipped itself stays skipped when the group passes.
- Focus (`.only`) is decided among the tests positionals selected; a `.only`
  in an unnamed file neither runs nor silences the named files, and CI's
  ONLY_IN_CI applies to the same selection.
- A templatable param the recorded text never contained is pinned in the
  entry (`literalParams`) and a replay with another value misses, so a
  param that steered the flow without appearing in it cannot replay the
  wrong actions under a shape-only key.

* refactor(runner): one status vocabulary, one trace text walk, explicit cache key shape

Structural follow-ups from review, behavior unchanged except where noted:

- `records.ts` owns `AttemptStatus` and `isFailedStatus`; the realm,
  serial, retry, and video-retain decisions all ask it instead of spelling
  the status list by hand. `execute.ts` emits unit results through one
  helper and stays under 1000 lines.
- The whole runtime-skip API lives in `internal/skip.ts`: `parseSkipCall`
  tells the three call shapes apart, `skipRunningTest` refuses the cases
  that may not skip (outside a body, inside a setup), and the executor's
  catch is one uniform line. The published attempt carries `testKind`.
- `trace.ts` owns `mapTraceText`, the single list of text-bearing trace
  fields; `template.ts` is now the placeholder grammar and the params walk
  only. Location paths are templated too, so a value in the URL follows
  the run instead of handing off forever.
- Placeholders name params by JSON Pointer, so `{ 'a.b' }` and
  `{ a: { b } }` can no longer collide; the key digests an explicit
  `{ literal, templated }` model instead of an in-band marker; templating
  is one pass that records what it bound; a recording that already spells
  a placeholder is not written rather than mislabelled `truncated`.
- Collection stamps `selected` on files and tests; `Collection` drops the
  redundant `discovered` and `selectedFiles`.
- `getByRole({ level })` fails closed on the device engine, which reports
  no heading level, and the contract says so.
- The four ordering matchers share one body; `isProjectedSecret` is owned
  by the params projection that writes it.

* chore: rebase on #324, keep main's heading-level support

#324 landed level on the contract, the locator expression, the Playwright mapping, and the device engine, so this branch drops its own copies and their changesets; the matcher, skip, and expect changes to types.ts are re-applied on main's file.

* chore: split trace-cache param templating out of this PR

The templated params work moves to its own branch so the selection, skip,
and matcher changes can land on their own. This restores cache/identity.ts,
cache/trace.ts, agent/step-cache.ts, and agent/act-validation.ts to main.
2026-09-16 19:04:11 +02:00
Oskar Kwaśniewski 4c76360cb6 feat(report): the run page is paragraphs a reader can skim; flaky tests and the file list fold away (#310) 2026-09-14 17:50:53 +02:00
Oskar Kwaśniewski 5908a107f9 feat(report): a failure hands off what the screen held, a page per failed test, and the line to look at (#299) 2026-09-14 15:05:20 +02:00
Oskar Kwaśniewski 5d1ae7a7f5 docs(reference): cut the slop from the Reference tab (#295) 2026-09-14 09:42:41 +02:00
Oskar Kwaśniewski c2c5df7df9 feat(e2e)!: the runner publishes as e2e; @e2edev/e2e is retired (#294) 2026-09-14 08:56:22 +02:00