* feat(mcp)!: e2e mcp runs headless by default and takes --headed, like run
Sessions were headed outside CI and took --headless, the opposite of
e2e run and e2e explore. --headless stays accepted and hidden: it now
asks for the default, and a refused flag would reach an MCP host as
nothing but a failed connection.
* docs(skill): --headed shows the UI when the engine supports it
* feat(mcp): open_session takes headed, so the agent picks per session
The server's --headed sets the default for sessions that do not say,
and headed: true or false overrides it, so a user who asks to watch
needs no change to the MCP client config.
* fix(e2e): load TypeScript with an oxc loader instead of tsx
* fix(e2e): close loader edge cases: sourceless CommonJS, export maps without extensions, unreadable tsconfig, chained source maps, Windows require paths
* fix(e2e): refuse Bun and Deno runtimes, name oxc load failures, follow Yarn PnP export targets
* perf(e2e): skip the syntax check for files whose only @ is in a package import
* fix(e2e): allow type-only exports in .cts, warn from runner only, Bun check first, docs
* fix(e2e): workers warn about tsconfigs runner did not, FILE: urls, foreign infra errors, review tests
* feat(oauth): sign in with OpenCode Console for Zen and Go models
* fix(oauth): keep the Console refresh token, isolate tests from OPENCODE_API_KEY
A refresh grant without a refresh_token now keeps the previous one, as
the ChatGPT and SpaceXAI providers do. The vendor test helper blanks
OPENCODE_API_KEY so a key in the shell no longer replaces the stored
login under test. Docs and changeset now say the models listing covers
Zen and Go only, and that MISCONFIGURED needs a readable workspace config.
* fix(run): keep --last-failed reruns honest: failure-limit skips, carried debts, kept evidence
--last-failed now reruns tests --max-failures skipped. A test or failed
suite hook another filter leaves out stays owed in the report's new
run.carried until a rerun runs it; the github fold shows carried rows and
errors and stays red while anything is carried. A rerun prunes artifacts/
to what the previous report names and writes its attempts under
artifacts/rerun-<n>/ instead of wiping the tree and reusing attempt paths.
* fix(run): carry a setup's hook debt when no selected test needs it
The report has no row for an unneeded setup, so carryForward now asks the
collection whether a test still exists instead of the report's rows.
* fix(run): reject malformed artifacts in --last-failed reports, tighten rerun docs
Distinct ids in the carried fixture row.
* fix(run): keep carrying tests of a file a narrowed rerun could not collect
* fix(github): a carried interrupted test reads interrupted, not failed, in the folded page
* fix(oauth): reach Copilot models served only over the Responses API
copilot() reads the plan's model listing on the first call and picks
chat completions or the Responses API for each model, falling back to
chat whenever the listing cannot be read so nothing that worked changes.
sendCopilotRequest reads the initiator and vision headers from a
Responses body (input[]/input_image) as well as a chat one, and the model
listing marks an uncallable model 'no chat or responses'.
* fix(changeset): schedule a minor release for the Responses endpoint support
The changeset asked for a patch while the change adds a call path a model could not
reach before; a minor bump is what the release intent states.
Addresses the cubic review comment on .changeset/copilot-responses-endpoints.md.
* fix(oauth): cancel and never cache a Copilot endpoint lookup that was aborted
The first call read the plan's model listing with no signal, and the delegate cached
whatever that produced. A listing that stalled therefore left `pending` unresolved and
every retry waited on the same promise instead of reaching the model. The signal now
reaches the lookup, an aborted lookup rejects instead of being read as a missing
endpoint, and a choice that could not be made clears `pending` so the next attempt
asks again. A test pins the aborted lookup and the missing-endpoint read as distinct.
Addresses the cubic review comment on packages/e2e/src/oauth/copilot.ts.
* docs(skill): separate Copilot models served over Responses from uncallable ones
The setup reference read as if every non-chat model is reached over the Responses API.
Only an enabled model the plan serves solely over `/responses` is; a model served over
an API `copilot()` does not speak, or one the plan has not enabled, still cannot be
called, and the listing marks those.
Addresses the cubic review comment on skills/e2e/references/setup.md.
* fix(oauth): pick Copilot endpoint from supported_endpoints alone, retry unreadable listing
- drop policy gate: grok-4.5, gpt-5.3-codex serve /responses with no policy
- unreadable listing: call over chat, not cached, asked again next call
- listing has own 10s timeout; one caller abort no longer fails others
- revoked login fails on listing with sign-in hint, no second 401
- missing @ai-sdk/openai names the package to install
* fix(init): install @ai-sdk/openai for the Copilot gateway
* docs(copilot): unreadable listing retry, init installs both peers
---------
Co-authored-by: Hermes Agent <agent@localhost>
Co-authored-by: Feco Linhares <fecolinhares@users.noreply.github.com>
Co-authored-by: Oskar Kwaśniewski <oskar@okwasniewski.com>
* fix(report)!: count interrupted tests apart from failures
* fix(report): tell a failed test from its last failed attempt, whatever the cut retry left
* feat!: the app is declared on its target; engines only drive it
The app under test moves from the engine's options to `targets: [{ engine,
app }]`: `url`, `bundleId`, `appPath`, `identity`, `environment`,
`launchArguments`, `permissions`, `command`, and `readyUrl`. `web()` and
`mobile()` refuse every moved option in one `INVALID_CONFIG` naming the
target block; `defineEngine({ app })` is gone and engines check what their
platform needs in `validateApp`. `requires: ['browser' | 'native-app']` is
derived from the target's app. Replay cache and session keys are byte for
byte what they were, pinned by target-identity.test.ts.
Dependency services are not part of this version: `services` anywhere, and
`web({ services })`, fail saying so. `app.command` on port 0 without a
command, and `reuseExisting` on a free port, are refused. An `e2e mcp`
open cancelled or disconnected mid-start stops what it started, and a hung
shared start is abandoned once no session waits on it.
* fix: review follow-ups: cancelled opens, abandoned starts, null targets, stale docs
* refactor: drop the moved-option messages; old app keys are unknown keys
* test: give the replaced-nodes suite a budget that survives a loaded runner
* test: size the replaced-nodes hook budget to its seven tests
* feat(cli): e2e feedback sends a report about e2e to the team
One PostHog event with the report and anonymous machine facts; secret-named
env values and token shapes redacted first. --dry-run prints it. A saved
telemetry disable does not stop it; E2E_TELEMETRY_DISABLED and DO_NOT_TRACK do.
* docs(cli): feedback redacts secret-named values of six characters or more
* fix(cli): feedback counts limits in characters, never claims an unconfirmed report was lost
* fix(web): leave Ctrl-C to the runner, not Playwright's signal handlers
* fix(run): a worker killed by Ctrl-C while starting is not a boot failure
* fix(run): stop a running file at --max-failures without waiting for the interrupt
* fix(mcp): claim a session's config before evaluating it
* fix(mcp): close sessions and exit 0 when the client goes away
* fix(agent): lead a failed action with the attempt and its code; MCP isError
* fix(run): excuse only the workers one Ctrl-C reached, so a signalled boot loop still fails
* fix(web): keep Playwright's SIGHUP handler; it closes the browser on a hangup
* docs: fix stale and wrong claims in guides
- output: --artifacts -> --output in bug-bash guide and skill; sessions and
cache locations follow output / cache.dir in security; artifact path
gains the agent segment in github
- ci: upload artifacts on !cancelled() in ci guide and skill
- mcp: several sessions (--max-sessions) in security
- replay cache naming: "Commit the replay cache" heading and anchors
(old anchor kept, e2e init links the new one),
trace replay -> replay cache in writing-an-engine and executors
- timeouts: STEP_TIMEOUT / STEP_BUDGET_EXHAUSTED advice names config
timeout, judgmentTimeout, maxSteps, maxModelCalls (debugging, skill)
- mobile: app.open() in device examples and quickstart, mobileTools pack
all mutating, Android toolbar anchor, drop agent-device release note
- executors: verdict code rules, missing ctx members and scrollUntil,
a replay cache section; agent-steps passed+code rule
- migrate: swipe direction is the content's, `!` test exclusions,
video row status, Kernel for Selenium Grid
- secrets: an unset env string fails config load, a provider's value is
redacted only from its first fill; three secret sinks
- misc: Models link text, telemetry links outbound connections, integration
config wording, web-only expect(web) in a cross-platform block,
capabilities heading in engine reference, swipe direction in screen reference
* docs(skill): separate the credential redaction paragraph
* docs(reference): KERNEL_API_KEY names @e2e-dev/kernel
* feat(agents)!: plain-object agents entries, createAgent and limits removed
Each agents entry is one AgentConfig object: model, judge, system, context,
tools, maxSteps, maxModelCalls, judgmentTimeout, maxObservationBytes,
maxInputTokens, providerOptions, or executor for a custom brain (system and
tools rejected beside it). No inheritance between agents.
- createAgent() kept as a stub throwing INVALID_CONFIG; bare StepExecutor or
legacy createAgent value as an entry fails naming { executor } / the object
- timeout -> judgmentTimeout, maxTurns removed (entry and
createToolLoopExecutor), unknown keys get did-you-mean
- top-level limits removed; maxModelTokensPerCall -> per-agent
maxInputTokens, the rest internal constants; report-1 run.limits keeps
its shape, filled from max across agents plus the constants
- tools validated at config load; built-in agent built from the entry at
fixture time; explore and the MCP catalog read the entry's tools
- migrate apps, docs, docs examples, init scaffold, skills, mobile README
* docs: AGENTS.md names the agent entry's model, not createAgent
* refactor(agents)!: delete createAgent instead of a throwing stub
* docs: MODEL_UNAVAILABLE scoped to the built-in agent, fixture-acquisition timing
* feat(config)!: remove specVersion from the config
The runner version is the format version. A config that still sets
specVersion fails INVALID_CONFIG telling the author to remove it. The
report's run.specVersion and the replay cache key keep their hard-coded
copies.
* feat(collect): "!" exclusions in the tests globs
An entry starting with "!" removes files an including glob matched,
in any order. Discovery never reads a directory an exclusion takes
whole, unless an including glob could reach a dot name beneath it.
A list of only exclusions is INVALID_CONFIG.
* refactor!: CacheStore and web({ screencast }) renames
TraceCacheStore is CacheStore, and the docs, skill, and CLI call the
cache the replay cache. web({ video: { size, quality } }) is
web({ screencast }) with WebScreencastOptions; web({ video }) fails
INVALID_CONFIG naming screencast.
* feat(secrets): secrets.get() in engine options, resolved per attempt
At config evaluation, before any run exists, secrets.get(name) returns
a reference by name. An engine declares the handles its options hold
in defineEngine({ secrets }); the config load fails INVALID_CONFIG for
a name no secret or credential has. The engine reads the value through
EngineAttemptContext.resolveSecret, which runs a provider fresh and
registers the value in the session and process ledgers before it
returns, without tainting the viewport. e2e/engine exports isSecret.
web({ basicAuth: { password } }) takes a Secret and declares it. Docs
show process.loadEnvFile('.env') in the config.
* feat(run)!: trace and video share recording modes; artifacts holds only store
One RecordingMode ('off' | 'on' | 'retain-on-failure' | 'on-first-retry' |
'on-all-retries') for trace and video, set on the config, a target, the CLI
(--trace [mode], --video [mode]) and a test, one precedence rule: config,
target, CLI, test. Trace defaults to on locally, on-first-retry in CI, with
a plan-time notice when a retry mode meets retries: 0. Serial members cannot
set trace.
Capability rule: a mode from the config root or a flag applies to targets
whose engine can record it, one notice naming the rest; a mode set on a
target or a test is required and fails with UNSUPPORTED_ARTIFACT.
artifacts.kinds (both forms) and artifacts.trace are removed and fail
INVALID_CONFIG naming the trace mode they meant. Failure screenshot and
screen text are always captured when the engine can. MCP sessions keep the
target's trace mode and trace only under 'on'.
* feat(run)!: output directory for every run result; ArtifactStore.putLink
output (default .e2e) and --output <dir> on run and explore hold report.json,
junit.xml, summary.md, ai-trace.json, artifacts/, sessions/ and the e2e mcp
session files; --last-failed reads <output>/report.json. The reportSibling
derivation from the artifacts dir is gone. --artifacts stays as a hidden
flag that fails with the --output it maps to.
A run clears <output>/artifacts at start. cache.dir stays independent. An
output that is the project root or outside it, is or sits in cache.dir,
would hold cache.dir under artifacts/, sessions/ or videos/, or holds a
tests glob root fails INVALID_CONFIG with the reason.
ArtifactStore gains optional putLink(link) for provider-hosted video links;
its ref lands beside the url. A passed retain-on-failure attempt's link is
never handed over, and a failing putLink never fails the run.
bench-ab and the mobile workflow move to --output.
* test(secrets): a hold's registry is probed by an unknown name, now that get() defers without one
* test(web): the download trigger's measured wait tolerates a timer firing early
* fix(secrets): rewrite the trace of an attempt whose engine holds a secret
A Playwright trace records the context options, basic-auth password in
plaintext, but a trace was rewritten only after a fill. resolveSecret now
marks the session so its trace goes through the redactor, screencast
frames kept since the value never reached the screen. The fake engine
writes a real trace archive, and the basic-auth fixture escapes its echo
and refuses a malformed header.
* fix(web): race the basic-auth secret with the attempt's signal
A provider still resolving after the launch timeout set the session after
the runner ended the attempt, so the retry failed with an attempt already
running. Resolution now runs first, raced with the signal.
* fix(mcp): a session notes a configured trace its engine cannot record
* test: a shadow-free fake trace archive and an ungarbled exclusion title
* docs: output paths, replay cache wording, and review corrections
--last-failed, --ai-trace, and the json reporter name <output> paths;
the skill says output never moves cache/ or the app's log; the
INVALID_GLOB row states the check the code makes; serial members list
only; the engine vocabulary names Secret and isSecret; the basic-auth
example uses a provider; the last trace cache wording is replay cache.
* chore(changeset): leave released notes alone; pending notes match the renames
Released CHANGELOG entries keep TraceCacheStore, the name they shipped
with; cache-store-rename.md announces the rename. The pending video and
provider recording changesets name on-all-retries and web({ screencast }).
* chore(bench): refuse a forwarded --output the harness would not clean up
* fix(secrets): a rewritten trace drops its screencast frames, engine-held secrets included
* test(testbed): admin session saved from a cookie, no secret fill
A setup that signs in without filling a secret carries no pixel taint, so tests and explorations restoring it keep screenshots.
* feat(explore): --session starts the exploration signed in
Explore collects the config's test files unselected, so the runner's own session selection runs exactly the setup that declares the name, then restores it into the exploration. In-process runs adopt the in-memory registration only for its virtual file; setup files import as usual. Planner and explorer are told they start signed in. A missing session producer now lists declared sessions with did-you-mean. Docs: a restored session keeps its setup's taint (authentication.mdx claimed otherwise).
* fix(explore): a --session setup runs as the project's agents
The explorer replaced the configured agent for the whole run, so an agentic setup pulled in by --session got the exploration rules, report_finding, explorer budgets, and cache off. In-memory tests now carry their own agents (InMemoryAttempts) and run with the trace cache off; setups keep the config's agents, cache, and retries. Uncollected-file notice no longer mentions positionals under explore; consumesSession drops the unreachable group walk.
* docs(explore): qualify app.open and screenshot claims, fix errors.mdx clause
* feat(run)!: video modes on the config, targets, the CLI, and tests, with step captions
video: off | on | retain-on-failure | on-first-retry at the config root, per
target, --video [mode], and per test or describe, innermost winning.
artifacts.kinds: ['video'] and artifacts.video are removed. Kept recordings
get WebVTT step captions; a video can be a link to a hosted recording. The
engine contract's VideoSegment is a file or a link, and e2e/engine exports
the provider recording seam.
* feat(web): a browser provider records the attempts it serves, and the screencast takes a size and quality
BrowserProvider.record starts the service's own recording of the browser in
place of the page screencast. The session drives one AttemptVideo, the
screencast or the provider's. web({ video: { size, quality } }) tunes the
screencast.
* feat(mobile): a device provider records the attempts it serves
DeviceProvider.record starts the service's own recording of the leased
device in place of agent-device's. The worker learns its slot's lease id,
so record gets the lease as it traveled.
* feat(integrations): Kernel replays record the attempts that record video
* fix(run): captions never fail a video, provider values are normalized, and one list of modes
- Step captions create their directory and are best-effort.
- A provider's recording is checked at start in both engines and its URL
and start time are stored in the shapes the report schema admits; the
list reporter bounds a hosted URL.
- --video no longer hides an invalid target mode.
- A describe inside a serial group cannot set video.
- A mobile provider may keep its own leaseId field on a lease.
* fix(mobile): keep the lease type narrowed when checking its declared fields
* fix(run): MCP recordings get their step captions, a step spanning a restart is captioned in both recordings, and video.size refuses unknown keys
* fix: address review on the video API
- One stop in flight per provider recording, shared by a retry; mobile
dispose stops it, and a cancelled operation never starts one.
- Kernel retries a failed download without stopping the replay twice.
- Links from any engine are held to an http(s) URL with a host; a file a
provider names must be a regular file; media types are trimmed.
- web({ video }) takes a plain object only; the markdown page labels a
hosted artifact by its kind.
- Tests: fixture byte totals, captions linked by id, temp directories
removed, signals asserted, empty sizes, non-object options.
- Docs: engine support qualified, the explore row, the standalone preflight,
caption status text, a checked stop in the provider example.
* fix(run): a bad engine video link costs no other recording, captions path per recording in debugging guide
* test: fake engine video links doc matches segment order
* fix(mcp): session recordings follow video modes and list a provider's link
* test(web): provider test's fake page reports its viewport
* refactor(run): drop step captions from the video API, tracked in #636
* fix(web): a late provider start never replaces a newer recording, abandon keeps a failed stop's handle, video.size must be a plain object; provider recording errors and hosted URLs bounded
* feat(mcp): record video of a live session on demand
start_recording and stop_recording in the session catalog when the
engine records video. Files move to .e2e/videos/<session>/ on stop;
close_session saves one still running. Sessions no longer record from
launch when the config asks for video.
* fix(mcp): serialize recorder calls, retry a failed stop safely
Review: close could race an in-flight start or stop, and a failed file
move lost the recording. Calls now queue; a stopped recording stays
until its files move. Recording tools lose the read-only mark.
* fix(mcp): queue the close's save behind a start in flight
Review: close checked isRecording first, which is false while a start
is still pending, so that recording ran past the close.
`e2e run --repeat-each <n>` (`RunOptions.repeatEach`) runs every selected
test that many times. The repeat is a dimension of a pair's identity beside
the test, the target, and the agent: `TestTargetPair.repeat`, the wire pair,
the pair-start and progress messages, the scheduler's keys, and the report
result and serial group all carry it, 0 for the run every test has. The
first run's result id is unchanged; later runs get their own, the list
reporter, JUnit, and the failure pages title them `(repeat #n)`, and their
artifacts go under `repeat-<n>`. Setup tests run once; a filtered or skipped
pair is reported once. Run events `test-started` and `step` carry `repeat`.
`--last-failed` now keys the previous run's results by test, target, and
agent, so a test is named once whichever of its repeats did not pass.
`e2e run --max-failures <n>` (`RunOptions.maxFailures`) stops the run once
that many tests have failed or timed out. The scheduler counts every result
it reports, skips its queues with the new cause `failure-limit` and the
reason `run stopped after 3 failures (--max-failures 3)`, tells the runner,
and interrupts its workers the way the interrupt signal would; the interrupt
message now carries that skip as the abort reason, so the pairs a worker had
not started say why instead of `run interrupted before execution`. The
running tests end as `interrupted`, and the exit code is the failures' own,
never 130. Reporters get a `run-stopped` event, which the list reporter
prints. The report schema's skip causes gain `failure-limit`.
`--last-failed` selects only the tests the previous run did not pass, read
from the `report.json` this run overwrites (beside the artifact root):
failed, timed out, interrupted, or skipped because a setup, a serial
predecessor, a hook, or the worker failed, each named per target and agent
by its result id. No report there is `NO_LAST_RUN`; a run with nothing
failed selects nothing. `--shard <index/total>` runs one contiguous slice of
the selected tests, cut once every other filter and the serial closure
applied, the first shards taking the remainder, so a serial group is never
split and each shard brings only the setup tests it needs. `e2e list` takes
both. `RunOptions` and `ListOptions` carry them as `lastFailed` and `shard`.
Any positional form may end in `:line` (`tests/signup.e2e.ts:12`,
`signup:12`) to select only the test whose `test(` call opens on that line
of each file it names. Several lines of one file add up, a file named
without a line is selected whole, and a serial group runs whole when a line
names one member. A positional that exists on disk as written stays a path.
When a named line declares no test, `NO_TESTS` names the positional and the
lines the file declares tests at. The mapped-line check spawns the built
CLI, as the source test does, since a vitest fork reports raw positions.
`e2e run` and `e2e list` take three more selection flags, shared through one
option group. `--exclude-tag` leaves out tests carrying any of the tags
whatever `--tag` or a positional selected; a serial group still runs whole
when any member is selected. `--grep` keeps tests whose title matches one of
the patterns and `--grep-invert` leaves out the ones that match; the text is
the describe titles and the test title joined by one space, a bare pattern or
`/pattern/flags`, repeated for alternatives, and one that does not compile is
a usage error. `RunOptions` and `ListOptions` carry them as `excludeTags`,
`grep`, and `grepInvert`. `NO_TESTS` names the tags and patterns that left
nothing to run.
* fix(collect): glob forms that match nothing are INVALID_GLOB
A tests glob or a positional glob that could never match a file now fails with INVALID_GLOB and a hint: an absolute path, a trailing slash, braces, a backslash separator, or a .. segment. A leading ./, a . segment, and a doubled slash are dropped, so ./tests/**/*.e2e.ts selects what tests/**/*.e2e.ts does instead of matching nothing, walking .git and .e2e, and pointing the NO_TESTS message at a missing example file.
* fix(collect): review fixes for the glob grammar
Segments without wildcards compile to names, so the near-miss scan reads a glob's leading directory off its segments (literalPrefix) and the positional glob test is the grammar's own GLOB_SYNTAX; the pattern string seam is gone. Character classes and extglobs are rejected with hints like braces. A trailing '.' and a normalized leading '!' are caught. An existing path wins over glob classification for positionals, so a file whose name has braces is still selectable by path. Messages name the fix; docs and the changeset describe positionals as the code treats them; the near-miss regression has a test.
* fix(cli): --tag takes comma lists and NO_TESTS names the nearest tag
--tag accepts a comma-separated list as well as repeats, like --target and --agent, and an empty value is a usage error instead of a run with the filter dropped. When the filter leaves nothing to run, NO_TESTS reads correctly under --tag-mode all and marks each tag no test declares with the nearest declared tag.
* fix(cli): one parser for the list-valued flags, tag blame only where it applies
Review fixes: --tag, --target, and --agent share parseNames (comma lists or repeats, each name once, empty is a usage error); NO_TESTS counts positional and .only exclusions as such instead of tag mismatches, computes the tag description once, and reads mode-neutral help; TagMode is one exported type; the did-you-mean note is one helper.
* feat(report): tags travel with a test into e2e list and the report
The tags a test declares are derived once at collection onto its identity, so selection, the wire, e2e list (text and json), and the report all read the same list. e2e list prints them as #tag after the target; run.results[] carry tags when a test declares any, so a reporter can group by tag. Report schema: an optional tags array of distinct non-empty strings on a result.
* feat(report): review fixes for tags on the identity
Registration derives a test's tags beside its title path, so collect.ts no longer walks the option layers a second time; ResolvedTestOptions drops its copy and the filter reads the identity. Results always carry tags, an empty list when none, like every other array in report-1; the schema pins the tag-name rule with a pattern and a targeted test. e2e list bounds titles and tags like every untrusted field. Stacked on the validation change so the schema promise holds.
* feat(runner): single-file setup selection, runtime skip, more matchers, templated cache params
Four gaps a real suite hit on its first day:
- `e2e run login.e2e.ts` failed with COLLECTION_ERROR when the session it
consumes is produced by a setup test in another file. Collection now
imports every discovered file and positionals narrow the selection
instead of the import set; tests in unnamed files are report-only
unselected results, and a consumed session promotes its setup wherever
it lives.
- `test.skip(condition, reason)` inside a body skips the running test for
a fact only the app can tell. Steps that ran stay in the report, teardown
runs, no retry is spent, and a setup test cannot skip. The signal is
recognized by brand so a test module realm with its own copy of `e2e`
still skips instead of failing. Report schema: a skipped attempt may
carry steps and a skip reason.
- `expect(actual, message)` opens value failures with the caller's label;
`toBeGreaterThanOrEqual`, `toBeLessThanOrEqual`, and `toBeCloseTo` join
the value matchers; `getByRole` takes `level` for headings.
- Run-unique string params no longer defeat the trace cache. String params
of three or more characters are templated in the recording (typed text,
target descriptors, end anchors, summaries) and filled at replay from the
current call, and the key digests the params' shape rather than their
values. Numbers, booleans, short strings, and secret names stay literal.
* chore: changesets, type assertions, and level scope note for the runner fixes
* fix(runner): review follow-ups for skip, focus, and templated params
- A runtime skip keeps the suite realm: only failed, timed-out, and
interrupted attempts discard it, so beforeAll runs once per suite.
- A serial member that skipped itself stays skipped when the group passes.
- Focus (`.only`) is decided among the tests positionals selected; a `.only`
in an unnamed file neither runs nor silences the named files, and CI's
ONLY_IN_CI applies to the same selection.
- A templatable param the recorded text never contained is pinned in the
entry (`literalParams`) and a replay with another value misses, so a
param that steered the flow without appearing in it cannot replay the
wrong actions under a shape-only key.
* refactor(runner): one status vocabulary, one trace text walk, explicit cache key shape
Structural follow-ups from review, behavior unchanged except where noted:
- `records.ts` owns `AttemptStatus` and `isFailedStatus`; the realm,
serial, retry, and video-retain decisions all ask it instead of spelling
the status list by hand. `execute.ts` emits unit results through one
helper and stays under 1000 lines.
- The whole runtime-skip API lives in `internal/skip.ts`: `parseSkipCall`
tells the three call shapes apart, `skipRunningTest` refuses the cases
that may not skip (outside a body, inside a setup), and the executor's
catch is one uniform line. The published attempt carries `testKind`.
- `trace.ts` owns `mapTraceText`, the single list of text-bearing trace
fields; `template.ts` is now the placeholder grammar and the params walk
only. Location paths are templated too, so a value in the URL follows
the run instead of handing off forever.
- Placeholders name params by JSON Pointer, so `{ 'a.b' }` and
`{ a: { b } }` can no longer collide; the key digests an explicit
`{ literal, templated }` model instead of an in-band marker; templating
is one pass that records what it bound; a recording that already spells
a placeholder is not written rather than mislabelled `truncated`.
- Collection stamps `selected` on files and tests; `Collection` drops the
redundant `discovered` and `selectedFiles`.
- `getByRole({ level })` fails closed on the device engine, which reports
no heading level, and the contract says so.
- The four ordering matchers share one body; `isProjectedSecret` is owned
by the params projection that writes it.
* chore: rebase on #324, keep main's heading-level support
#324 landed level on the contract, the locator expression, the Playwright mapping, and the device engine, so this branch drops its own copies and their changesets; the matcher, skip, and expect changes to types.ts are re-applied on main's file.
* chore: split trace-cache param templating out of this PR
The templated params work moves to its own branch so the selection, skip,
and matcher changes can land on their own. This restores cache/identity.ts,
cache/trace.ts, agent/step-cache.ts, and agent/act-validation.ts to main.