* fix(secrets): rewrite traces and text downloads whenever the run has a secret value
A registered value a test passed as a plain string (an app.open URL, a fill)
was redacted from titles, labels, and agent input but kept verbatim in
trace.zip: trace and download rewriting was gated on the exposure level,
which only a fill or an engine-held secret raises. Rewriting now depends on
the session ledger holding a value. The engine exposure level had no other
consumer and is removed; exposure now only decides pixels and saved-session
taint.
* fix(secrets): cover a plain-string fill, scope docs to the session's known values
* docs(secrets): download redaction is whole-value; TRACE_WITHHELD outside-path case
* docs(secrets): name the download rewrite failure path
A capture that came back empty was retried at each later capture point
(body, settle, finally), so a hung app cost 5 s per point. The engine
call was also awaited without racing the budget, so an engine ignoring
cancellation hung the attempt. Capture once per attempt and abandon the
engine call when the budget ends.
* feat(web)!: pin playwright-core and add an install command
@e2e-dev/web depends on playwright-core 1.63.0 instead of peering on
playwright. New e2e-web bin installs browsers for the pinned version:
npx @e2e-dev/web install chromium --with-deps. e2e init no longer adds
playwright.
* test(web): run the e2e-web bin for real, and in CI
* test(web): spawn the built e2e-web bin so Node 22.12 runs it
* fix(run): keep --last-failed reruns honest: failure-limit skips, carried debts, kept evidence
--last-failed now reruns tests --max-failures skipped. A test or failed
suite hook another filter leaves out stays owed in the report's new
run.carried until a rerun runs it; the github fold shows carried rows and
errors and stays red while anything is carried. A rerun prunes artifacts/
to what the previous report names and writes its attempts under
artifacts/rerun-<n>/ instead of wiping the tree and reusing attempt paths.
* fix(run): carry a setup's hook debt when no selected test needs it
The report has no row for an unneeded setup, so carryForward now asks the
collection whether a test still exists instead of the report's rows.
* fix(run): reject malformed artifacts in --last-failed reports, tighten rerun docs
Distinct ids in the carried fixture row.
* fix(run): keep carrying tests of a file a narrowed rerun could not collect
* fix(github): a carried interrupted test reads interrupted, not failed, in the folded page
* feat!: browser fixture, getByRole name arg, toMatchSchema, top-level hooks, separate credential namespace
API review feedback before launch:
- web fixture renamed browser (Web -> Browser, WebExpectation ->
BrowserExpectation); requires: ['browser']; destructuring web hints the
new name. web() factory and other Web* types unchanged.
- getByRole(role, name, options?) beside the object form; MCP locate
writes the short form.
- expect(value).toMatchSchema(standardSchema): sync, returns the typed
output; poll resolves to it, soft returns undefined after a kept failure.
- describe, beforeEach, afterEach, beforeAll, afterAll exported from e2e;
web and mobile export typed beforeEach/afterEach.
- credentials and secrets are separate namespaces: a password handle is
<name>.password, secrets.get() never returns it. Committed recordings
migrated offline (old key hash verified, then recomputed).
- runtime test.skip keeps Playwright semantics; docs say it throws.
* docs(web): browser acts on one active tab; cookies, routes, dialogs cover the whole browser
* fix(secrets): judge a fill's purpose by the config, not the handle; no old-name hints
A config-time secrets.get() handle reads its purpose from the run's
registry again, and authorizeSecretFill checks the registered purpose, so a
handle claiming generic-secret cannot put a password into a plain field.
Drop the web fixture migration hint. Changesets name the on-screen marker
change for replay entries.
* docs(skill): browser fixture, getByRole name arg, toMatchSchema, top-level hooks, credential namespace on the rewritten skill
* fix(expect): negated toMatchSchema returns nothing, as its type says
* fix(expect,docs): soft double negation stays soft, null ~standard is INVALID_ARGUMENT, Playwright samples keep Playwright's getByRole
Also documents SoftExpectCall, SoftValueExpectation, and NegatedPollExpectation
on the expect reference, requires: ['browser'] in the skill's options table,
and tightens the poll and secret-purpose tests.
* docs(babysit): triage pattern for scripted renames that rewrite another library's code
* docs: engine options take a secrets entry, web package hook exports, vault example imports
* refactor(secrets)!: credentials hold their password, allSecrets feeds fills; engines export describe and every hook; BrowserRoute, BrowserResponse
From the maintainability review:
- ResolvedConfig.secrets is the config.secrets entries only; a credential
carries its password; allSecrets is the one map fills, the ledger, and
redaction read. The per-consumer purpose filters and the .password
suffix parsing are gone, and e2e mcp builds handles from config.
- The config load and a running test name the credential a missing
secrets.get() most likely meant.
- @e2e-dev/web and @e2e-dev/mobile export describe and all four hooks
beside test, so a test file registers from one import.
- WebRoute and WebResponse are BrowserRoute and BrowserResponse.
- getByRole normalizes its arguments once; isTextMatch is the one
string-or-RegExp check; poll reuses ValueMatcherName.
* revert(web): keep WebRoute and WebResponse; Browser names the fixture, Web the network and option types
* feat!: the app is declared on its target; engines only drive it
The app under test moves from the engine's options to `targets: [{ engine,
app }]`: `url`, `bundleId`, `appPath`, `identity`, `environment`,
`launchArguments`, `permissions`, `command`, and `readyUrl`. `web()` and
`mobile()` refuse every moved option in one `INVALID_CONFIG` naming the
target block; `defineEngine({ app })` is gone and engines check what their
platform needs in `validateApp`. `requires: ['browser' | 'native-app']` is
derived from the target's app. Replay cache and session keys are byte for
byte what they were, pinned by target-identity.test.ts.
Dependency services are not part of this version: `services` anywhere, and
`web({ services })`, fail saying so. `app.command` on port 0 without a
command, and `reuseExisting` on a free port, are refused. An `e2e mcp`
open cancelled or disconnected mid-start stops what it started, and a hung
shared start is abandoned once no session waits on it.
* fix: review follow-ups: cancelled opens, abandoned starts, null targets, stale docs
* refactor: drop the moved-option messages; old app keys are unknown keys
* test: give the replaced-nodes suite a budget that survives a loaded runner
* test: size the replaced-nodes hook budget to its seven tests
* fix(agent): extract reports absent data instead of inventing it
Send the caller schema's shape only (no min/max/length/pattern/format) inside
an agent-extract-2 envelope { found, value, missing }. found: false fails the
step ASSERTION_INCONCLUSIVE naming what was missing; the repair round may
decline instead of bending a value to fit. Explore covers a declined plan like
an invalid one.
* fix(cache): ignore a gap's derived rule when comparing re-recorded flows
derived follows how the agent read the value on that run (pixels vs node),
never what a replay does, so read-write runs rewrote committed entries.
* fix(agent): stop blaming a control for an unchanged tree without pixel proof
Tree-only unchanged result now says no listed node changed and a drawn
change needs a screenshot; "no visible effect" only when an identical
screenshot backs it. With no screenshot to be had (secret filled, engine
captures none), an unchanged listing still says to try another way.
* fix(cache): REPLAY_STALE names the strict knob and configured cache dir
Was hardcoded to --strict-cache and .e2e/cache even when cache.strict or
cache.dir came from config.
* fix(agent): resend a stalled model request; keep turns of a timed-out step
One provider request with no response held the step until its own timeout
(an explore step spent 220s of 240s on a single call). Each generate
request is now bounded at 120s and fails as a retryable provider error, so
the SDK's transport retries resend it; the turn notes it. Turns attach as
each ends, so a step the clock settles keeps them, and loop notes survive
the report's per-turn clip.
* fix(agent): abandon a stalled request whose provider ignores the abort
* docs(changeset): say where a first-turn stall note shows
* fix(agent): clear the stall timer when a provider throws synchronously
* docs: trim guides to what a test author needs
Cut runner internals, give each repeated topic one owner page, rewrite
the densest paragraphs.
Owners now:
- screenshots after a secret fill: security#screenshots-after-a-secret-fill
(auth, explore, debugging, agent-steps, tools, executors, engine link it)
- redaction rules and limits: security#redaction, #where-redaction-stops
- what the model sees/does in a step: agent-steps; security keeps the
trust-boundary and POLICY_DENIED tables
- budgets and timeouts: reference/agent#budgets (agent-steps keeps cost levers)
- assert under a custom executor, executor model conflicts: executors
- exit codes: debugging; telemetry in CI: telemetry; explore credentials:
explore; cache trust: security; personas: agents + authentication
- shard/last-failed edge cases: reference/cli; engine capabilities and
lifecycle detail: reference/engine
- locator strictness: reference/screen; no sessions on a device: mobile
Other changes:
- security: redactor internals -> short user-facing summary
- cache: timing internals cut, reason table one line per cell plus a
"Limits and gaps" subsection, committed-file changes as a list
- executors: tool-loop sample drives ctx.actions (portable, recorded)
- tools: simpler seed sample, mobileTools spread beside own tools
- writing-an-engine: locate in skeleton, minimal capability table, one-line
lifecycle, own Keyboard heading, tool-pack pointer, GitHub test link
- models: `ai` prerequisite and key source up front, env-var switch moved
- migrate: shared rules to one line + link, framework-specific "migrate
first", drop "Lives in" column, US spelling
- kernel: minimal setup sample, neutral replay size wording
- telemetry: source-checkout paragraph cut
* docs: address review on the guide trims
- tools: the seed sample reads APP_URL again
- ci: --last-failed reads report.json under output
- cypress, detox: strictness is for actions and single-node reads
- explore: artifacts, not evidence, land in the goal directory
* docs: fix stale and wrong claims in guides
- output: --artifacts -> --output in bug-bash guide and skill; sessions and
cache locations follow output / cache.dir in security; artifact path
gains the agent segment in github
- ci: upload artifacts on !cancelled() in ci guide and skill
- mcp: several sessions (--max-sessions) in security
- replay cache naming: "Commit the replay cache" heading and anchors
(old anchor kept, e2e init links the new one),
trace replay -> replay cache in writing-an-engine and executors
- timeouts: STEP_TIMEOUT / STEP_BUDGET_EXHAUSTED advice names config
timeout, judgmentTimeout, maxSteps, maxModelCalls (debugging, skill)
- mobile: app.open() in device examples and quickstart, mobileTools pack
all mutating, Android toolbar anchor, drop agent-device release note
- executors: verdict code rules, missing ctx members and scrollUntil,
a replay cache section; agent-steps passed+code rule
- migrate: swipe direction is the content's, `!` test exclusions,
video row status, Kernel for Selenium Grid
- secrets: an unset env string fails config load, a provider's value is
redacted only from its first fill; three secret sinks
- misc: Models link text, telemetry links outbound connections, integration
config wording, web-only expect(web) in a cross-platform block,
capabilities heading in engine reference, swipe direction in screen reference
* docs(skill): separate the credential redaction paragraph
* docs(reference): KERNEL_API_KEY names @e2e-dev/kernel
* feat(config)!: remove specVersion from the config
The runner version is the format version. A config that still sets
specVersion fails INVALID_CONFIG telling the author to remove it. The
report's run.specVersion and the replay cache key keep their hard-coded
copies.
* feat(collect): "!" exclusions in the tests globs
An entry starting with "!" removes files an including glob matched,
in any order. Discovery never reads a directory an exclusion takes
whole, unless an including glob could reach a dot name beneath it.
A list of only exclusions is INVALID_CONFIG.
* refactor!: CacheStore and web({ screencast }) renames
TraceCacheStore is CacheStore, and the docs, skill, and CLI call the
cache the replay cache. web({ video: { size, quality } }) is
web({ screencast }) with WebScreencastOptions; web({ video }) fails
INVALID_CONFIG naming screencast.
* feat(secrets): secrets.get() in engine options, resolved per attempt
At config evaluation, before any run exists, secrets.get(name) returns
a reference by name. An engine declares the handles its options hold
in defineEngine({ secrets }); the config load fails INVALID_CONFIG for
a name no secret or credential has. The engine reads the value through
EngineAttemptContext.resolveSecret, which runs a provider fresh and
registers the value in the session and process ledgers before it
returns, without tainting the viewport. e2e/engine exports isSecret.
web({ basicAuth: { password } }) takes a Secret and declares it. Docs
show process.loadEnvFile('.env') in the config.
* feat(run)!: trace and video share recording modes; artifacts holds only store
One RecordingMode ('off' | 'on' | 'retain-on-failure' | 'on-first-retry' |
'on-all-retries') for trace and video, set on the config, a target, the CLI
(--trace [mode], --video [mode]) and a test, one precedence rule: config,
target, CLI, test. Trace defaults to on locally, on-first-retry in CI, with
a plan-time notice when a retry mode meets retries: 0. Serial members cannot
set trace.
Capability rule: a mode from the config root or a flag applies to targets
whose engine can record it, one notice naming the rest; a mode set on a
target or a test is required and fails with UNSUPPORTED_ARTIFACT.
artifacts.kinds (both forms) and artifacts.trace are removed and fail
INVALID_CONFIG naming the trace mode they meant. Failure screenshot and
screen text are always captured when the engine can. MCP sessions keep the
target's trace mode and trace only under 'on'.
* feat(run)!: output directory for every run result; ArtifactStore.putLink
output (default .e2e) and --output <dir> on run and explore hold report.json,
junit.xml, summary.md, ai-trace.json, artifacts/, sessions/ and the e2e mcp
session files; --last-failed reads <output>/report.json. The reportSibling
derivation from the artifacts dir is gone. --artifacts stays as a hidden
flag that fails with the --output it maps to.
A run clears <output>/artifacts at start. cache.dir stays independent. An
output that is the project root or outside it, is or sits in cache.dir,
would hold cache.dir under artifacts/, sessions/ or videos/, or holds a
tests glob root fails INVALID_CONFIG with the reason.
ArtifactStore gains optional putLink(link) for provider-hosted video links;
its ref lands beside the url. A passed retain-on-failure attempt's link is
never handed over, and a failing putLink never fails the run.
bench-ab and the mobile workflow move to --output.
* test(secrets): a hold's registry is probed by an unknown name, now that get() defers without one
* test(web): the download trigger's measured wait tolerates a timer firing early
* fix(secrets): rewrite the trace of an attempt whose engine holds a secret
A Playwright trace records the context options, basic-auth password in
plaintext, but a trace was rewritten only after a fill. resolveSecret now
marks the session so its trace goes through the redactor, screencast
frames kept since the value never reached the screen. The fake engine
writes a real trace archive, and the basic-auth fixture escapes its echo
and refuses a malformed header.
* fix(web): race the basic-auth secret with the attempt's signal
A provider still resolving after the launch timeout set the session after
the runner ended the attempt, so the retry failed with an attempt already
running. Resolution now runs first, raced with the signal.
* fix(mcp): a session notes a configured trace its engine cannot record
* test: a shadow-free fake trace archive and an ungarbled exclusion title
* docs: output paths, replay cache wording, and review corrections
--last-failed, --ai-trace, and the json reporter name <output> paths;
the skill says output never moves cache/ or the app's log; the
INVALID_GLOB row states the check the code makes; serial members list
only; the engine vocabulary names Secret and isSecret; the basic-auth
example uses a provider; the last trace cache wording is replay cache.
* chore(changeset): leave released notes alone; pending notes match the renames
Released CHANGELOG entries keep TraceCacheStore, the name they shipped
with; cache-store-rename.md announces the rename. The pending video and
provider recording changesets name on-all-retries and web({ screencast }).
* chore(bench): refuse a forwarded --output the harness would not clean up
* fix(secrets): a rewritten trace drops its screencast frames, engine-held secrets included
* feat(run)!: video modes on the config, targets, the CLI, and tests, with step captions
video: off | on | retain-on-failure | on-first-retry at the config root, per
target, --video [mode], and per test or describe, innermost winning.
artifacts.kinds: ['video'] and artifacts.video are removed. Kept recordings
get WebVTT step captions; a video can be a link to a hosted recording. The
engine contract's VideoSegment is a file or a link, and e2e/engine exports
the provider recording seam.
* feat(web): a browser provider records the attempts it serves, and the screencast takes a size and quality
BrowserProvider.record starts the service's own recording of the browser in
place of the page screencast. The session drives one AttemptVideo, the
screencast or the provider's. web({ video: { size, quality } }) tunes the
screencast.
* feat(mobile): a device provider records the attempts it serves
DeviceProvider.record starts the service's own recording of the leased
device in place of agent-device's. The worker learns its slot's lease id,
so record gets the lease as it traveled.
* feat(integrations): Kernel replays record the attempts that record video
* fix(run): captions never fail a video, provider values are normalized, and one list of modes
- Step captions create their directory and are best-effort.
- A provider's recording is checked at start in both engines and its URL
and start time are stored in the shapes the report schema admits; the
list reporter bounds a hosted URL.
- --video no longer hides an invalid target mode.
- A describe inside a serial group cannot set video.
- A mobile provider may keep its own leaseId field on a lease.
* fix(mobile): keep the lease type narrowed when checking its declared fields
* fix(run): MCP recordings get their step captions, a step spanning a restart is captioned in both recordings, and video.size refuses unknown keys
* fix: address review on the video API
- One stop in flight per provider recording, shared by a retry; mobile
dispose stops it, and a cancelled operation never starts one.
- Kernel retries a failed download without stopping the replay twice.
- Links from any engine are held to an http(s) URL with a host; a file a
provider names must be a regular file; media types are trimmed.
- web({ video }) takes a plain object only; the markdown page labels a
hosted artifact by its kind.
- Tests: fixture byte totals, captions linked by id, temp directories
removed, signals asserted, empty sizes, non-object options.
- Docs: engine support qualified, the explore row, the standalone preflight,
caption status text, a checked stop in the provider example.
* fix(run): a bad engine video link costs no other recording, captions path per recording in debugging guide
* test: fake engine video links doc matches segment order
* fix(mcp): session recordings follow video modes and list a provider's link
* test(web): provider test's fake page reports its viewport
* refactor(run): drop step captions from the video API, tracked in #636
* fix(web): a late provider start never replaces a newer recording, abandon keeps a failed stop's handle, video.size must be a plain object; provider recording errors and hosted URLs bounded
* fix(runner): a download is redaction incomplete unless the runner rewrote it
Every download artifact was recorded redaction: 'complete' without the
runner reading a byte of it. Fill a field through a secret, have the app
serve an export that contains the same value, and the file landed in the
attempt's artifacts as complete, and an artifacts.store received the
plaintext with metadata that said it was clean. A store exporting only
complete artifacts would have exported the secret. Screenshots on the same
session were already withheld by the pixel taint.
A download now defaults to 'incomplete', the label a video carries: bytes
the app served that the runner did not rewrite. After a secret fill on the
session, a text-like download (.txt, .csv, .json, .html) is rewritten
through the session's secret ledger before it is hashed or stored and is
labelled 'complete', the way trace text entries are. A binary or a file
that is not UTF-8 stays as served and 'incomplete'. StoredArtifact carries
the redaction the report records, so a store can act on it.
* fix(runner): a rewritten download keeps its byte order mark
TextDecoder drops a leading UTF-8 BOM by default, so a CSV served with one
and rewritten through the ledger lost its encoding marker. Decoding with
ignoreBOM keeps it as part of the text and the write puts it back.
* fix(runner): redactor matches a value csv-quoted
A CSV field doubles each double quote, so a secret holding one survived the ledger in a text download that was then labelled complete. The ledger now spells a double quote doubled too.