Commit Graph
21 Commits
Author SHA1 Message Date
Oskar Kwaśniewski 3a9667c046 fix(run): keep --last-failed reruns honest: failure-limit skips, carried debts, kept evidence (#724)
* fix(run): keep --last-failed reruns honest: failure-limit skips, carried debts, kept evidence

--last-failed now reruns tests --max-failures skipped. A test or failed
suite hook another filter leaves out stays owed in the report's new
run.carried until a rerun runs it; the github fold shows carried rows and
errors and stays red while anything is carried. A rerun prunes artifacts/
to what the previous report names and writes its attempts under
artifacts/rerun-<n>/ instead of wiping the tree and reusing attempt paths.

* fix(run): carry a setup's hook debt when no selected test needs it

The report has no row for an unneeded setup, so carryForward now asks the
collection whether a test still exists instead of the report's rows.

* fix(run): reject malformed artifacts in --last-failed reports, tighten rerun docs

Distinct ids in the carried fixture row.

* fix(run): keep carrying tests of a file a narrowed rerun could not collect

* fix(github): a carried interrupted test reads interrupted, not failed, in the folded page
2026-10-03 16:36:53 +02:00
Oskar Kwaśniewski 7ab80bc610 fix(report)!: count interrupted tests apart from failures (#713)
* fix(report)!: count interrupted tests apart from failures

* fix(report): tell a failed test from its last failed attempt, whatever the cut retry left
2026-10-03 09:30:44 +02:00
Oskar Kwaśniewski 4d488f8286 feat!: the app is declared on its target; engines only drive it (#698)
* feat!: the app is declared on its target; engines only drive it

The app under test moves from the engine's options to `targets: [{ engine,
app }]`: `url`, `bundleId`, `appPath`, `identity`, `environment`,
`launchArguments`, `permissions`, `command`, and `readyUrl`. `web()` and
`mobile()` refuse every moved option in one `INVALID_CONFIG` naming the
target block; `defineEngine({ app })` is gone and engines check what their
platform needs in `validateApp`. `requires: ['browser' | 'native-app']` is
derived from the target's app. Replay cache and session keys are byte for
byte what they were, pinned by target-identity.test.ts.

Dependency services are not part of this version: `services` anywhere, and
`web({ services })`, fail saying so. `app.command` on port 0 without a
command, and `reuseExisting` on a free port, are refused. An `e2e mcp`
open cancelled or disconnected mid-start stops what it started, and a hung
shared start is abandoned once no session waits on it.

* fix: review follow-ups: cancelled opens, abandoned starts, null targets, stale docs

* refactor: drop the moved-option messages; old app keys are unknown keys

* test: give the replaced-nodes suite a budget that survives a loaded runner

* test: size the replaced-nodes hook budget to its seven tests
2026-09-30 19:48:00 +02:00
devin-ai-integration[bot]andoskar 256513dc0b docs: sync the package manager choice across every command snippet (#679)
Co-authored-by: oskar <oskar@okwasniewski.com>
2026-09-30 14:40:08 +02:00
Oskar Kwaśniewski bb51ec1fdb docs: trim guides to what a test author needs (#656)
* docs: trim guides to what a test author needs

Cut runner internals, give each repeated topic one owner page, rewrite
the densest paragraphs.

Owners now:
- screenshots after a secret fill: security#screenshots-after-a-secret-fill
  (auth, explore, debugging, agent-steps, tools, executors, engine link it)
- redaction rules and limits: security#redaction, #where-redaction-stops
- what the model sees/does in a step: agent-steps; security keeps the
  trust-boundary and POLICY_DENIED tables
- budgets and timeouts: reference/agent#budgets (agent-steps keeps cost levers)
- assert under a custom executor, executor model conflicts: executors
- exit codes: debugging; telemetry in CI: telemetry; explore credentials:
  explore; cache trust: security; personas: agents + authentication
- shard/last-failed edge cases: reference/cli; engine capabilities and
  lifecycle detail: reference/engine
- locator strictness: reference/screen; no sessions on a device: mobile

Other changes:
- security: redactor internals -> short user-facing summary
- cache: timing internals cut, reason table one line per cell plus a
  "Limits and gaps" subsection, committed-file changes as a list
- executors: tool-loop sample drives ctx.actions (portable, recorded)
- tools: simpler seed sample, mobileTools spread beside own tools
- writing-an-engine: locate in skeleton, minimal capability table, one-line
  lifecycle, own Keyboard heading, tool-pack pointer, GitHub test link
- models: `ai` prerequisite and key source up front, env-var switch moved
- migrate: shared rules to one line + link, framework-specific "migrate
  first", drop "Lives in" column, US spelling
- kernel: minimal setup sample, neutral replay size wording
- telemetry: source-checkout paragraph cut

* docs: address review on the guide trims

- tools: the seed sample reads APP_URL again
- ci: --last-failed reads report.json under output
- cypress, detox: strictness is for actions and single-node reads
- explore: artifacts, not evidence, land in the goal directory
2026-09-29 23:46:03 +02:00
Oskar Kwaśniewski 6b851bb604 docs: fix stale and wrong claims in guides (#653)
* docs: fix stale and wrong claims in guides

- output: --artifacts -> --output in bug-bash guide and skill; sessions and
  cache locations follow output / cache.dir in security; artifact path
  gains the agent segment in github
- ci: upload artifacts on !cancelled() in ci guide and skill
- mcp: several sessions (--max-sessions) in security
- replay cache naming: "Commit the replay cache" heading and anchors
  (old anchor kept, e2e init links the new one),
  trace replay -> replay cache in writing-an-engine and executors
- timeouts: STEP_TIMEOUT / STEP_BUDGET_EXHAUSTED advice names config
  timeout, judgmentTimeout, maxSteps, maxModelCalls (debugging, skill)
- mobile: app.open() in device examples and quickstart, mobileTools pack
  all mutating, Android toolbar anchor, drop agent-device release note
- executors: verdict code rules, missing ctx members and scrollUntil,
  a replay cache section; agent-steps passed+code rule
- migrate: swipe direction is the content's, `!` test exclusions,
  video row status, Kernel for Selenium Grid
- secrets: an unset env string fails config load, a provider's value is
  redacted only from its first fill; three secret sinks
- misc: Models link text, telemetry links outbound connections, integration
  config wording, web-only expect(web) in a cross-platform block,
  capabilities heading in engine reference, swipe direction in screen reference

* docs(skill): separate the credential redaction paragraph

* docs(reference): KERNEL_API_KEY names @e2e-dev/kernel
2026-09-29 23:38:52 +02:00
Oskar Kwaśniewski 67c29fc6b2 chore!: publish engines and reporter under the @e2e-dev npm scope (#602) 2026-09-28 19:05:05 +02:00
Oskar Kwaśniewski dceae62730 fix(report): run page reads top down: source under the title, evidence kinds linked without paths, folded attempt runs, one target named once (#538) 2026-09-25 19:13:45 +02:00
Oskar Kwaśniewski 61603e6c2c fix(github): a --last-failed rerun updates the pull request comment as one run (#537) 2026-09-25 17:09:09 +02:00
Oskar Kwaśniewski cdc41098c7 fix(report): markdown cells cannot mention, autolink, or start a block, and all-dot names are rejected (#448) 2026-09-23 10:53:49 +02:00
Oskar Kwaśniewski b71ecc0f09 fix(report): leave unselected results off the page and JUnit, link sources under the project path (#422) 2026-09-22 18:14:28 +02:00
Oskar Kwaśniewski 2b9361ffea feat(report): the folded test list is one table, a row per file above its tests (#413) 2026-09-22 13:11:39 +02:00
Oskar Kwaśniewski f5e96d0cbe feat!: rename engine packages to @e2edev/web and @e2edev/mobile (#392) 2026-09-21 19:52:32 +02:00
Oskar Kwaśniewski f09b69e90a docs: simplify onboarding and clarify model access (#346) 2026-09-18 11:52:06 +02:00
Oskar Kwaśniewski 4c76360cb6 feat(report): the run page is paragraphs a reader can skim; flaky tests and the file list fold away (#310) 2026-09-14 17:50:53 +02:00
Oskar Kwaśniewski c2c5df7df9 feat(e2e)!: the runner publishes as e2e; @e2edev/e2e is retired (#294) 2026-09-14 08:56:22 +02:00
Oskar Kwaśniewski 8811ba7de9 feat(report): a markdown reporter writes the run as summary.md, with an exploration's findings, and the GitHub comment renders from it (#284) 2026-09-12 15:54:10 +02:00
Oskar Kwaśniewski 3de7471f77 fix(collect): a test declared through a helper reports the helper call in the test file as its source, never a runner or node_modules frame (#277) 2026-09-12 11:12:58 +02:00
Oskar Kwaśniewski 07cae8787c docs(cli): run the CLI as npx e2e; drop --no-install everywhere (#275) 2026-09-12 11:07:20 +02:00
Oskar Kwaśniewski fcf4fe6ff7 docs: split guides and references by platform (#266) 2026-09-11 17:32:13 +02:00
Oskar Kwaśniewski a0a4df01b4 feat(github): the GitHub reporter comments on the pull request from Actions (#246) 2026-09-10 14:47:36 +02:00