From cd5ff1125981f0fc37a56df25eee7310286a3f89 Mon Sep 17 00:00:00 2001 From: Mario Zechner Date: Mon, 28 Sep 2026 16:47:23 +0200 Subject: [PATCH] docs: reorder durable harness milestones --- packages/durable/docs/pico-v5-handoff.md | 389 +++++++++++++---------- 1 file changed, 222 insertions(+), 167 deletions(-) diff --git a/packages/durable/docs/pico-v5-handoff.md b/packages/durable/docs/pico-v5-handoff.md index 8f11089e0..ac79e1cb1 100644 --- a/packages/durable/docs/pico-v5-handoff.md +++ b/packages/durable/docs/pico-v5-handoff.md @@ -265,201 +265,256 @@ commits; delivery Context value preservation; retained earlier revision stability; trusted mutation footguns; retirement before start and while active; recreation; idempotent stop; second-start rejection; cancellation during acquisition; cancellation/close during a callback without aborting or joining it; -listener-error settlement; and invocation-owned cleanup in package 15. +listener-error settlement; and invocation-owned cleanup in package 14. -## 13. Conversations and entries +Packages 13–20 are vertical milestones. Each must leave one real public path +working end to end; do not defer all integration to the last package. -Implement conversation history/ownership records, entry creation, -conversation-bound cursor-based fork-aware scans, head lookup, and entry edits. -Expose public history pagination through `Conversation.entries()`, never through -`Harness`. +Cross-cutting constraints for these milestones: -Test conversation creation and actual forks, deep ancestor caps, same-commit -entry prefixes, newest-edit wins, self-head resolution, raw head-to-tail -transcript, and ownership traversal. +- Introduce each persisted built-in document kind once, with its final schema, + history/fork policy, migration, checkpoint predicate, and public mount path. + Record that concrete protocol in the normative specification when the kind + lands. Do not add temporary built-in kinds or later replace a fake schema. +- Keep all visible progress durable. Tests may use faux models and fake effects, + but production code must use pi-ai's exported `Models` interface and the real + task chain; do not add a Pico model adapter or production fake successor. +- Prepare every loaded `ConversationView` revision from the complete candidate + Session commit before Storage admission. Never rebuild a view by subscribing + to already-committed table/document publications. +- A milestone may leave later operations unimplemented, but it must not expose a + temporary public facade. Extend the final §2.2 objects as later behavior lands. -## 14. Context derivation and system messages +## 13. Openable Harness -Implement model-context reduction, PR #9548 positional `SystemMessage` replay, -tool-result ordering, and missing post-fork tool results. Replay `content`, -ordered named `sections` with `null` removal, then tool removals/additions. +Implement the first usable slice of the final §2.2 surface: +`Harness.open/close`, stable root creation and lookup, conversation lookup and +creation, concrete-entry forks, `onConversation`, conversation-bound `commit()` +and `entries()`, and generic inherited Session document APIs. Implement the +public `Entry` definition token and `Conversation` handles rather than adding an +intermediate capability facade. -Test model-less and excluded-stop-reason entries, replacements/omissions, -multiple heads, section replacement/removal/re-addition order, order-only -configuration changes between separate request preparations, rejection of -integer-like section keys, tool addition/removal/replacement order, and raw-view -versus model context. +Bring conversation and entry semantics up to the normative contract: explicit +ownership, fork-aware cursor pagination, head lookup, entry edits, and model +context derivation. Context reduction includes newest-edit wins, positional PR +#9548 `SystemMessage` replay, tool-result ordering, missing post-fork tool +results, excluded stop reasons, and separate raw-history/model-context results. -## 15. Task definitions and invocations +Define the final rewindable/as-of conversation configuration document now. It +contains model selection, default `"off"` thinking, ordered section values, and +active tool names. Implement every configuration getter/setter and atomic +configuration seeding/override needed by root, independent conversation, and +fork creation. Definitions supplied through final Harness options may be used +for section/tool identity validation; dynamic execution and hooks arrive later. +Do not create temporary fixed document accessors. + +Use an internal final-form bootstrap transaction for reserved root ID `1`; do not +expose a temporary root-creation API or split empty-storage root/config creation +across commits. `options.root` applies only to empty storage. For this milestone, +Harness open/reopen acceptance is explicitly limited to storage with no live +tasks. Package 14 removes that limitation by adding complete open-time task +reconciliation; Package 13 must not invent partial reconciliation behavior. + +Acceptance: open persistent storage, obtain the root, mutate configuration and +history through public handles, create and fork conversations, close, reopen, +and verify stable root/conversation identity and state. Test atomic root and +conversation/config creation; actual forks; deep ancestor caps; same-commit +entry prefixes; cursor boundaries; self-head resolution; newest-edit wins; raw +head-to-tail transcript versus model context; model-less and excluded assistant +entries; replacements/omissions; multiple heads; section replacement/removal/ +re-addition order; integer-like section-key rejection; positional tool changes; +missing post-fork results; every configuration getter/setter; explicit active- +tool duplicate/unregistered rejection; default active registry snapshot; as-of +configuration inheritance including unavailable historical names; seed +overrides; listener initial/future delivery and isolation; and reopen. + +## 14. Durable task runtime Implement `defineTask`, exhaustive phase maps, full checkpoint replacement, -kind migration, runtime commits, memos, and invocation close gates. +kind migration, runtime commits and memos, invocation lifetime gates, scheduler +reservation, dependencies, terminal outcomes, typed waits, holds, quiescence, +joins, and orphaning. Complete the task-facing §2.2 methods: `resume`, `suspend`, +`hold`, task-kind registration, `getTask`, `waitForTask`, `markTask`, +`abortTask`, and the task-aware portion of idle waits. -Use a fake two-phase effect. Test intent/effect/outcome recovery, -unchanged-checkpoint faulting, same-phase checkpoint progress, cancellation -precedence, thrown-handler faulting, close/reopen without abort marks, outcomes, -or task-document retirement, no fresh phase/abort dispatch while closing, -first-writer-wins memos, and automatic watch cleanup. +Include the execution-critical abort core: durable direct-task marks, +signal-and-join of an active run, fresh abort invocation, run-commit rejection +after a mark, and close precedence. Deep owned-subtree cascading and background +boundaries remain Package 18. Open now reconciles every surviving `running` task +to `pending`, migrates registered kinds, and atomically orphans unknown or +unmigratable live kinds as required by the normative specification. No handler +dispatches during open, and dynamic registration does not resurrect a task +settled by that pass. -## 16. Scheduler and terminal tasks +Use a fake two-phase external effect to test the real runtime. Acceptance is an +intent/effect/outcome task interrupted after intent, closed, reopened on the same +storage, and safely resumed to a durable terminal receipt. Also test unchanged- +checkpoint faulting, same-phase progress, cancellation precedence, thrown +handlers, dependencies, result values and entry IDs, first-writer-wins memos, +terminal checkpoint/memo removal, task-document retirement, close/reopen without +abort marks or fabricated outcomes, no fresh phase/abort dispatch while closing, +watch cleanup, holds and quiescence with eligible work, unknown kinds, mark-only +versus signalling abort, and crashes at every direct-task abort stage. -Implement reservation, running-task reopen reconciliation, dependencies, -terminal outcomes, waits, holds, joins, and orphaning. +## 15. First runnable no-tool chat turn -Test result values and entry IDs, terminal records after reopen, dependency -eligibility, unknown kinds, and terminal removal of checkpoints/memos. +Implement the smallest real input-to-answer vertical path. Define the final +inbox, turn-control, and generation presentation documents needed by this path; +do not use provisional kinds or schemas. Add input `Submission` admission, +request-ID deduplication, reacquisition and waiting, idle placement, active-turn +ownership, successful answer settlement, and terminal failure cleanup. Expose +the final `SubmissionDraft` union rather than an interim input-only API. Complete +the optional initial input path on conversation creation so conversation, +configuration, sections, and input admission commit atomically. Busy steer/ +follow-up behavior, passive writes, and reset remain Package 17. -## 17. Abort and owned conversations +Implement section registration and no-tool request preparation, including exact +persisted rendered strings, positional system baselines/deltas, head-cut +rebaselining with `ContextEdit` omissions, and preparation revision checks. +Implement the ordinary generation phases needed for one response: preparation, +request intent, durable throttled partials, attempts/retry classification, +deferred handle polling/cancellation, assistant entry settlement, and input- +submission completion. -Implement durable abort marks, signal/join/fresh-abort invocation, owned -conversation creation, subtree traversal, background behavior, and idle waits. +Call pi-ai only through its exported `Models` methods: `getModel()`, +`streamSimple()`, `fetchDeferred()`, and `cancelDeferred()`. The test double must +implement that same interface; production code gets no adapter. A missing configured model produces +the durable `no_model` failure. All progress exposed to observers is committed +state, never raw provider frames. -Test commit rejection after a run task is marked, crashes at every abort stage, -close precedence over a previously marked task, deep ownership trees, atomic -retirement of task-scoped documents, default non-inheritance, inheritance from -the current committed tail, an empty source conversation, document fork -policies, and explicit model/section seed overrides. +Acceptance: `Harness.open → root → resume → submit(input) → Submission.wait → +durable assistant answer → close/reopen`, using faux Models in tests. Exercise +interruption and reopen before/after every implemented generation phase, +aborted-partial conversion, deferred polling/cancellation, retryable and +terminal model errors, no-visible-undurable updates, exact section order/value +patches, complete post-head baselines, retained system deltas on both sides of a +head marker, atomic input/ +configuration/section creation, and durable submission settlement. This is also +the first print-mode smoke path: print awaits its own input submission rather +than global idle. -## 18. Submissions and positional inbox +## 16. First coding-agent tool turn -Define the initial inbox and turn-control documents, then implement strict input/ -write `SubmissionRecord` variants, `Conversation.submit()`, request-ID -deduplication, awaitable/reacquirable submissions, busy admission, withdrawal, queue modes, -and `postTools`/`final` boundaries. Successful input settlement requires an -answer; write settlement means entry placement and never starts a turn. Use a fake successor -task. +Implement task/tool registries and Session/owned-subtree hooks, then wire the +real generation → tool tasks → post-tools → generation chain. Implement offered- +set checks, declaration and argument validation, hook composition, durable +execution intent, stored replay policy, bounded stream/progress documents, +result entries, post-tools joining, controls, and `postTools`/`final` boundaries. +The generation task now classifies tool calls and continues through the real +built-in task chain; neither side uses a production fake successor. + +Implement runtime registration lifetimes and preparation behavior for tool +loadout additions/removals, same-name replacement ordering, complete baseline +tool declarations, hook memos, and configuration/registry revision retries. +Do not implement in-process replacement of executing Session-side extension +code; the normative v1 close/reopen boundary remains Package 20. + +Acceptance: input → model tool call → registered local read/bash/edit operation → +tool result → model answer → durable submission settlement. Run that path once +normally and once interrupted/reopened. Test recovery from every tool and post- +tools phase; offered-history enforcement; before/after hook rules; both stored/ +current replay-policy directions; default and overridden output bounds; streamed +content fallback; progress replacement and coalesced commit settlement; drain- +before-terminal ordering; abort/close with buffered output; invocation-bound +handles and watches; `missing_active_tool` settlement; atomic assistant/tool/ +post-tools commits; and all registry lifetime and positional tool-history cases. + +## 17. Live UI and product state + +Complete submissions and inbox behavior: busy `steer`/`followUp`/`reject`, +passive writes, withdrawal, ordered `postTools`/`final` selection, stale targets, +self-head cuts, successor turns, queued reset/handoff, and every terminal cleanup. +Successful inputs still require an answer; writes settle on placement and never +start generation. + +Define any remaining built-in preference/presentation documents once with final +schemas. Implement the structural `{ conversation, entries, docs }` +`ConversationView`, `viewState()`, and `watch()`. Build the first revision lazily +on the Session line. For each affected commit, derive and prepare one mounted +operation batch from the complete candidate transaction before Storage +admission; after success, only install prepared pointers/cursors and enqueue the +exact immutable frame. Do not derive the mount through `subscribeCommits()`. + +Add the §9.4 notification adapter directly from uncoalesced committed +publication, without another tracker or persistence authority. Wire TUI +hydration to structural state/watch, print to its own `Submission`, and JSON/RPC +to correlated commands plus ordered committed notifications. Transport +backpressure and disconnect policy stay in the mode adapter. Table-test every submission transition, cross-type request-ID conflicts, -interleaved steer/follow-up/write selection, self-head cuts, stale targets, -successor triggers, reopen waits, writes pending without a later boundary, -compact large-payload removals, abort results for queued/placed/terminal -submissions, and orphan/fault cleanup of active turn control. +interleaved queue selection, compact positional removal of large payloads, +abort results, reopen waits, writes pending without a later boundary, busy reset +placement, and orphan/fault cleanup. Test one view publication per Session +commit; atomic entry/preview settlement; parent-linked active entries and heads; +mounted create/recreate/retire; preparation rollback before Storage; empty-batch +suppression and redundant nonempty revisions; contiguous delivery; stable public +paths; O(1) immutable acquisition; structural sharing; serialized consumers; +bounded reset behind an in-flight callback; durable retry/tool/collapse status; +output truncation metadata; the documented placement of diagnostics in entries, +terminal details, or bounded state; asynchronous consumer initialization; and +absence of semantic projection. Verify watch overflow cannot erase a separately +subscribed notification lifecycle, late clients hydrate structurally, and +notifications expose committed throttled +progress rather than raw provider frames. -## 19. Remaining built-in documents and view +## 18. Ownership and subagents -Define the concrete configuration, preference, and live presentation documents; -reuse the approved inbox/turn definitions. Record all IDs, fields, history, -fork settings, migration, and checkpoint predicates in the normative -specification. +Complete the remaining owned-conversation and abort semantics: durable foreground +subtree cascades, signal/join/fresh-abort across ownership edges, background +boundaries, ordinary and full traversal, conversation abort, and exact idle +waits. Finish invocation-bound owned APIs used by tools and the foreground and +background subagent provisioning patterns, including atomic task/conversation/ +registry creation and request-ID-safe submission recovery. -Implement `{ conversation, entries, docs }` as immutable structurally shared -revisions produced by the optimized immutable applier. Build the first revision -lazily on the Session line. For every later affected Session commit, derive the -mounted operation batch and prepare its next revision before Storage admission; -a failure rolls back normally. After Storage succeeds, finalization only installs -prepared pointers/cursors and enqueues publication. Conversation views expose a read-only Chord state directly; their watches use -Package 12's O(1) acquisition and bounded exact-frame buffering. +Test deep ownership trees, owner edges after terminal settlement, nested +background boundaries, conversation abort/join with surviving passive writes and +background tasks, cancellation of waiters without cancellation of work, and +atomic cancellation intent. Test default non-inheritance, inheritance from the +current committed tail, empty source conversations, document fork policies, +explicit model/section/tool seed overrides, foreground subagent cascade, and +background supervisor recovery before and after submission admission. -Test direct task writes, one publication per Session commit, atomic -entry/preview settlement, parent-linked active-entry reconstruction, head -changes, mounted create/recreate/retire transitions, preparation failure before -Storage, empty mounted-batch suppression, redundant nonempty mounted revisions, -contiguous revisions, stable public paths, immutable O(1) acquisition, -asynchronous consumer -initialization, serialized updates, revision/payload structural sharing, -bounded buffering and reset behind an in-flight callback, retry/collapse late-join status, -bounded-output truncation metadata, and absence of semantic projection. Specify -which diagnostics become entries, terminal details, or bounded document state. +## 19. Collapse and overflow -## 20. Registries, hooks, and sections +Implement manual, threshold, and generation-overflow collapse; exchange-boundary +range selection; summarization; retry policy; staleness checks; and headed +summary entries. Wire generation's real overflow path directly to the collapse +task, and complete `Conversation.collapse()` so it returns the admitted task ID. -Implement task/tool/section registries, Session and owned-subtree hooks, -positional PR #9548 section/tool updates, complete baselines after a head cut, -and preparation revision checks. Do not add a Session-kernel semantic event -journal or extension-state router; package 24 adds the thin product notification -adapter from specification §9.4. +Test model context before and after collapse, raw history preservation, provider +failure, declined and stale work, manual/threshold/overflow admission, late-join +presentation state, and reopen from every phase. Rerun generation overflow +integration without a fake collapse kind. -Test registration lifetimes, hook replay with memos, exact persisted rendered -section strings, minimal section patches and `null` removals, complete baseline -tool declarations, and a head cut that retains earlier system messages. Verify -the new baseline entry omits those messages through `ContextEdit` before replay, -including retained `content`, section order, and tool changes. Include a retained -delta whose ID precedes the head-carrying entry: select omissions by retained -context membership, not an ID comparison with the head entry. Test tool loadout -additions/removals and preparation retry after registry movement. Do -not implement in-process replacement of Session-side extension code; a host extension -change uses the Harness close/reopen boundary. +## 20. Reload and final conformance -## 21. Tool and post-tools tasks +Complete any remaining §2.2 surface and lifecycle gates, then implement the +normative v1 host-extension reload path: stop admission/reservation, close and +join, dispose registrations/facets, rebuild over the same storage with new +definition tokens and task/tool/section definitions, reopen/migrate live tasks, +and resume. Ordinary documents migrate on later typed access. The separate live- +registries proposal remains non-normative unless it is first merged into +`pico-v5.md`; do not silently substitute it for §7.4. -Implement offered-set checks, argument validation, tool hooks, durable bounded -progress, owned APIs, interrupted/replay-safe recovery, result entries, -post-tools joining, controls, and boundaries using fake tools. +Test that close seals commit and mutation admission, lets already-admitted +storage settlement finish despite caller cancellation, stops future state/watch +delivery, joins task/tool/hook invocations outside the Session line, writes no +abort or terminal outcome, starts no fresh abort invocation, and never runs old +and new Harness generations concurrently. Include cancellation during watch +acquisition and an already-running callback that remains caller-owned across +shutdown. Verify service withdrawal and client detach. -Test recovery from every phase, both stored/current replay-policy directions, -default and overridden bounds, streamed-content fallback, progress replacement -and coalesced commit settlement, drain-before-terminal ordering, -abort/close with buffered output, invocation-bound owned handles, and atomic -assistant/tool/post-tools settlement. Use a fake -generation successor; package 22 replaces it and reruns integration. +Run the exhaustive public conformance matrix: stable persisted root identity; +all root/create/lookup/fork/reset/collapse/abort/idle paths; every configuration +getter/setter and fork override; typed input/write submissions; task wait/abort; +generic document access; task/tool/section registration between open and resume; +conversation listener isolation; structural watches; and no resurrection of a +task settled during open. Compile-test every §2.2 and §3 owner/key/seed overload, +the normative usage sequences, and the Chord guide. The erased registry test must +use a concrete narrowed-input task with multiple checkpoint phases and custom +hooks. Verify that a Chord root-replacement delta remains distinct from a +Session-selected storage checkpoint. -## 22. Generation task - -Implement preparation, request intent, durable throttled partials, attempts, -retry policy, response classification, continuation, and deferred polling/ -cancellation through pi-ai's exported `Models` interface. Do not add a Pico -model adapter. The faux test double implements that same interface. - -Test every phase before and after reopen, aborted partial conversion, overflow -through a fake collapse kind, input-submission settlement, and no visible-undurable update. -Replace the fake tool successor and rerun the package 21 integration tests. - -## 23. Collapse task - -Implement manual, threshold, and overflow collapse; exchange-boundary range -selection; summarization; retries; staleness; and headed summary entries. - -Test context before/after collapse, provider failure, declined/stale work, and -reopen from every phase. Replace generation's fake overflow target and rerun its -overflow integration test. - -## 24. Harness integration - -Implement the exact public surface in specification §2.2: -`Harness.open/resume/suspend/close`, lifecycle gates, root/create/lookup -`Conversation` objects, typed input/write `submit()` and `Submission` objects, -conversation-bound commits and history pagination, fork/collapse/reset/abort/idle, -typed task wait/abort, generic document access, task/tool/section registries, and -structural conversation watches. Do not restore Pico3's -namespace router, fixed document accessors, semantic view events, or manual Chord -view bridge. - -Expose service withdrawal/client detach and product wiring. Implement the §9.4 -agent-mode notification adapter directly from uncoalesced committed publication, -without another tracker or persistence authority. Migrate TUI hydration to the -structural conversation watch, make print await its own input `Submission`, and expose -JSON/RPC correlated commands plus ordered committed notifications. Test that -watch overflow resets cannot erase a separately subscribed notification -lifecycle, late clients use structural hydration rather than event -replay, progress notifications -reflect durable throttled state rather than every provider frame, and stdout -backpressure/disconnect policy stays in the mode adapter. - -Implement the v1 host-extension reload path as stop admission, close/join, dispose, -rebuild with new document tokens and registered task/tool/section definitions, reopen/migrate live tasks, and resume. Ordinary documents migrate -on later typed access. Test that closing seals commit and -mutation admission, lets storage settlement for already-prepared admitted -commits finish despite caller cancellation, stops future watch deliveries, joins -task/tool/hook invocations outside the Session line, writes no abort or terminal -outcome, starts no fresh abort invocation, and does not run old and new -generations concurrently. Include cancellation during watch acquisition; an -already-running watch callback remains caller-owned across shutdown. - -Test stable persisted root identity; atomic conversation/config/section/input- -submission creation; default `"off"` thinking; every configuration getter/setter; explicit -active-tool seed duplicate/unregistered rejection; default active registry -snapshot; as-of fork inheritance including unavailable historical names; durable -`missing_active_tool` settlement; fork seed overrides; concrete-entry forks; -collapse task-ID return; -busy reset admission and later placement; mark-only versus signalling abort; -conversation abort/join with surviving passive writes and background tasks; -quiescence with eligible work; listener initial/future delivery and isolation; -and runtime registration between open and resume without resurrection of a task -settled during open. - -Compile-test every §2.2 and §3 owner/key/seed overload plus the usage sequences -in the normative specification and Chord guide. Verify that a Chord root -replacement delta remains distinct from a Session-selected storage checkpoint. The erased registry test must include a concrete task with narrowed -input, multiple checkpoint phases, and custom hooks. Run all package-specific -tests and the repository check. Verify a local -coding-agent turn and a reopened interrupted turn, then stop for final review. +Run all package-specific tests and the repository check. Finish with a local +coding-agent turn and a reopened interrupted turn through the public Harness, +then stop for final review.