mirror of
https://github.com/THU-MAIC/OpenMAIC.git
synced 2026-10-24 06:33:30 +08:00
@openmaic/generation@0.3.13
21
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 wyuc
|
3b4567076f |
refactor(generation): consume the package everywhere and delete lib/generation (#1090)
Part D of #1057 — the switch. All consumers import @openmaic/generation; lib/generation deleted (net -11,900 lines). PBL wiring preserved (single-call -> classified fallback -> injected loop), streaming outline route deduplicated onto buildOutlinePrompt, serverExternalPackages wired, and the change verified by a real-model four-kind end-to-end run against the production build with a rendered-classroom screenshot check. Closes #1057. |
||
|
wyuc
|
0ae21aa559 |
refactor(pbl): retire v1 write path and dead UI, converge on v2 (#1060)
* refactor(pbl): retire v1 write path and dead UI (#1058) Remove the legacy generator, renderer branches, chat API, and compatibility mirror writes. Keep stored v1 scenes readable through documented legacy adapters, and surface ordinary PBL planner double failures instead of degrading. * fix(pbl): contain planner failure to the scene, support legacy scenes in action regeneration, tighten legacy guards - Contain typed planner failures per scene while preserving retry and skip behavior. - Upgrade legacy PBL content before regenerating scene actions. - Apply video-export import boundaries to legacy readers. - Harden legacy rendering and empty-state handling. - Restore legacy upgrade compatibility coverage. - Remove retired translations and refresh fallback narration and legacy cover documentation. * fix(pbl): keep tolerant legacy configs renderable, guard invalid chat timestamps * chore(dsl): bump to 0.6.3 for README accuracy fix The README no longer references the deleted PBLProjectConfig symbol; the version gate requires a bump for any publishable package input change. * fix(pbl): harden legacy upgrade and persistence contract * fix(pbl): align action prompt with v2 workflow and validate project containers * fix(i18n): remove retired PBL v1 keys from the new fr-FR locale The fr-FR locale landed on main after this branch removed the v1 UI key families; prune the same 38 keys so all 10 locales align with en-US. * fix(generation): sync packaged pbl-actions template with the v2 wording The generation package that landed on main carries a byte-identical copy of the app's pbl-actions template; sync the v2 rewording and bump the package version. * fix(pbl): normalize legacy content and preserve planner status * fix(pbl): propagate provider status and ground action prompts * fix(pbl): classify planner failures for retry, default the project summary prompt input * fix(pbl): fall back to legacy data when v2 payloads are damaged, harden summary building * refactor(pbl): resolve hybrid content through a validity-aware resolver * fix(pbl): keep one v2 acceptance criterion across validator, resolver and persistence Persistence gates on the same container predicate as the validator and the renderer's resolver: payloads the renderer will not treat as v2 are inert bytes everywhere, avoiding runtime side effects for documents the write barrier would reject. An explicit null proficiencyAssessment stays processable. * fix(pbl): require runnable projects and harden legacy fallback * test(pbl): exercise the real save reload and hydration chain * fix(pbl): require an instructor and usable microtasks before v2 takes precedence * fix(pbl): fall back to the loop planner unless the provider failed or the user cancelled * fix(pbl): require role and task identities before v2 takes precedence Runtime normalization can repair status and create the instructor thread, but it cannot synthesize the role id that binds the thread, the microtask ids used by lookup and update paths, or the labels the workspace renders. The runnable predicate and its video-export mirror now require them, so identity-less payloads fall back to usable legacy data. |
||
|
wyucand杨慎
|
d12f8f07eb |
fix(diagram): honor outline node constraints (#911)
Co-authored-by: 杨慎 <117187635+cosarah@users.noreply.github.com> |
||
|
Yanpeng Wangandwyuc
|
af4c9ac69e |
fix(prompts): add postMessage listener contract to diagram/game/code widgets (#872)
* fix(prompts): add postMessage listener contract to diagram/game/code widgets Widget actions are delivered to the iframe via postMessage (lib/action/engine.ts:691-717), so a widget must register a 'message' listener to be driven by scene actions. The simulation, visualization3d, and procedural-skill content prompts document this contract; diagram, game, and code did not, so widgets of those types never emit the handler and every action targeting them silently no-ops. Add the postMessage listener section (SET_WIDGET_STATE / HIGHLIGHT_ELEMENT / ANNOTATE_ELEMENT / REVEAL_ELEMENT) to the three prompts, adapting the handler to each widget type's DOM, plus an Element Naming Convention note so highlight/annotate/reveal have stable targets. Refs #871 * fix(prompts): guard undeclared editor in code-widget SET_WIDGET_STATE sample editor?.setValue still throws ReferenceError when no `editor` variable is declared (e.g. a textarea-only code widget), so the #code-input fallback never runs. Use `typeof editor !== 'undefined'` to guard the identifier itself before the fallback. --------- Co-authored-by: wyuc <wang-yc24@mails.tsinghua.edu.cn> |
||
|
Yanpeng Wang
|
4b821272ff |
feat(interactive-actions): feed real HTML element inventory into the prompt (#829)
* feat(interactive-actions): feed real HTML element inventory into the prompt
Teacher actions (highlight/setState/annotation/reveal) for interactive scenes
frequently miss because the model has to guess selectors from the prompt's
naming conventions (e.g. `#{var}-slider`, `#start-btn`) that don't match the
ids the content model actually emits (`#score-val`, `#launch-btn`, …). The
game widget category is the worst hit because its conventions cover almost
nothing.
Extract an inventory of interactable elements (ids + notable classes +
aria/role/data-step-id) from the generated widget HTML and thread it into
`INTERACTIVE_ACTIONS` under an `{{elementInventory}}` variable. The system
prompt now instructs the model to select `target` from the inventory first
and only fall back to conventions when nothing matches.
The inventory is:
- Persisted on the runtime `InteractiveContent` so pro-mode
`regenerate_scene_actions` gets the same real selectors, not a stale guess.
- Refreshed by `edit_interactive_html` — the tool re-runs the extractor on
the post-edit HTML and the client writes it back into `scene.content`, so
a subsequent action regen sees the new dom.
- Computed on the fly in `generateSceneActions` when a legacy scene lacks it,
so pre-existing courses benefit without a data migration.
Tailwind / utility class tokens are filtered so the 30-class inventory cap
keeps semantic hooks like `.pairing-rules`, `.dna-card`, `.btn-launch` from
being crowded out by layout tokens.
* style: prettier format
* fix(interactive-actions): compute element inventory at prompt-build time
Address PR #829 review feedback:
1. Drop the persisted `elementInventory` field on
`InteractiveContent`/`GeneratedInteractiveContent`. `generateSceneActions`
already computes the inventory on the fly from `content.html`; make that
the only path. Removes born-stale mismatches (post-processing rewrite vs
raw-html extraction), same-turn staleness of `sceneContextMap`, and the
silent divergence between `createSceneWithActions` (drops the field) and
`scene-builder.ts` (keeps it) — server-generated classrooms already ran
in recompute-only mode today.
2. Extractor hardening:
- Strip HTML comments alongside `<script>` / `<style>` so commented-out
markup can't forge phantom inventory entries the prompt would prefer.
- Drop content after an unmatched `<script` open so a truncated
generation doesn't inventory ids inside `innerHTML` template strings.
- Parse attributes into a name→value map by walking the tag grammar
instead of per-attribute regex over the flat string. Prevents
`aria-label="try name=alpha or id=fake"` from lifting phantom
`name=alpha` / `id=fake` attributes.
- Collapse whitespace and truncate each attribute value (~120 chars)
when building inventory lines so a multi-line or hostile aria-label
can't fake prompt sections.
- Keep semantic hooks whose names collide with Tailwind category
prefixes (`.grid-cell`, `.fill-blank`, `.text-input`, `.select-btn`,
`.ring-carbon`) when they are declared in the page's own `<style>`
block.
* style: prettier format
* fix(interactive-actions): surface id-less data-attribute targets in inventory
The interactive-actions system prompt tells the model to prefer
`[data-step-id="step-1"]` selectors for procedural-skill widgets whose
step rows typically carry only `data-step-id` (no id, and often no
surviving semantic class). Under the previous extractor those rows were
invisible: `data-step-id` and `data-action` only surfaced as decoration
on id rows, so a widget with all-data-only steps could hit the
"(no interactive elements detected)" sentinel while the system prompt
kept telling the model to prefer the inventory — degrading exactly the
one widget family the conventions already served well.
Add a third `Stable data attributes:` section that emits an
`[data-step-id="step-1"] <li>` line for each id-less element carrying
data-step-id or data-action, deduped on attribute value and capped at 30
entries. Elements that already have an id keep those attributes as row
decoration only (no duplicate row).
|
||
|
Yanpeng Wangandwyuc
|
b6fe2814d2 |
fix(quiz): stop leaking questions on entry and pass results to chat agent (#823)
* fix(quiz): stop leaking questions on entry and pass results to chat agent Fixes #822. Two related defects on quiz scenes: 1. The `quiz-actions` opening monologue could enumerate question stems, compare options, and trigger a `discussion` action that walked agents through the quiz BEFORE the learner had answered anything. 2. After the learner submitted, the chat/discussion agent never saw their answers, correctness, the canonical `analysis`, or the `aiComment` from `/api/quiz-grade` — so post-quiz feedback was generic and often re-explained the whole quiz from scratch. Changes: - `lib/prompts/templates/quiz-actions/{system,user}.md`: collapse the generator down to a brief 1-2 segment opening, forbid `discussion` actions in scene.actions, and add explicit "no preview, no answer, no detailed concept re-teach" safety rules. - `lib/types/chat.ts` + `lib/chat/agent-loop.ts`: add optional `quizResults` to `StatelessChatRequest.storeState` / `AgentLoopStoreState` (sceneId + answers + per-question status/earned/aiComment). - `components/chat/use-chat-sessions.ts`: hydrate `quizResults` from `readSubmittedState(sceneId)` once per agent-loop iteration (and on initial dispatch) so retries are picked up. - `lib/orchestration/summarizers/state-context.ts`: when results are present, surface per-question student answer / correct answer / verdict / analysis / aiComment with a "address THIS student's mistakes" instruction; when unsubmitted, keep question visibility for clarifying questions but pin strict no-leak rules so a learner saying "I'm done" cannot bait the agent into reciting the quiz. Discussion / multi-agent round-robin and the canonical session flow are untouched. * style: apply prettier formatting to state-context --------- Co-authored-by: wyuc <wang-yc24@mails.tsinghua.edu.cn> |
||
|
|
cec3c6aaea |
Refactor widget actions into the scene action pipeline (#796)
* refactor: align widget actions with scene pipeline * feat(edit): migrate legacy teacherActions off interactive content on load The widget-actions pipeline no longer persists a teacherActions authoring layer; playback reads only scene.actions. Add migrateInteractiveContent to strip the dead field from existing documents as they load through migrateScene (wired into all store/classroom load paths). Satisfies the #794 forward-migration acceptance item. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(generation): default widget_setState.state to {} when the LLM omits it The removed TeacherAction->Action converter always defaulted state to {}. Restore that guard in the structured-output parser so an action missing state can't forward state: undefined into the widget iframe, where SET_WIDGET_STATE handlers dereference it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(edit): also strip nested widgetConfig.teacherActions in migration Legacy WidgetConfig variants each declared their own teacherActions, so existing documents can carry the dead field nested inside content.widgetConfig as well as at the top level. Extend migrateInteractiveContent to drop both, preserving all other widgetConfig fields. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(prompts): document stable widget-action selectors for diagram/game/viz3d The interactive-actions prompt documented stable target selectors only for procedural-skill and simulation widgets. Add the canonical selectors the diagram / game / visualization3d content prompts actually emit (node IDs via revealOrder; #zoom-in-btn / #canvas-container / #speed-slider / ...; #start-btn), plus a rule to use widget_setState or a speech-only beat instead of guessing a target when no stable selector is known (e.g. free-form code widgets). Hardens against widget actions silently no-oping on non-existent selectors. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: 杨慎 <117187635+cosarah@users.noreply.github.com> |
||
|
杨慎
|
09863af421 |
Add PBL v2 core schema and generation path (#795)
* feat: add PBL v2 core planner * chore: restore unrelated lint comments * fix: document PBL v2 scenario outline flags * fix: expose PBL scenario subtype in outlines * chore: clear PBL v2 completion stats lint warning * chore: address PBL instructor code quality comments * fix: address PBL v2 review findings * chore: restore unrelated lint comments |
||
|
wyucandwyuc
|
267965d6ed |
feat(generation): infer concise courseTitle from outlines for readable course names (#756)
* feat(generation): infer concise courseTitle from outlines for readable course names
Generated courses were named after the raw user prompt (truncated to 500 chars
client-side, or 50 chars / first outline title server-side), which is long and
unreadable. The outline-generation step already understands the full course
semantics, so have it emit a concise `courseTitle` alongside `languageDirective`
and `outlines`, and use it as the stage name.
- Prompts (requirements-to-outlines, interactive-outlines): top-level JSON gains
a required `courseTitle` (≤30 chars, in the teaching language, noun phrase).
- Streaming route: extract `courseTitle` head-bound like `languageDirective`,
emit a `courseTitle` SSE event, and include it in the `done` event.
- Non-streaming outline-generator: parse and return optional `courseTitle`
(defensive trim + 120-char cap).
- Naming: stage.name = courseTitle || outlines[0]?.title || requirement fallback
(client preview path backfills stage.name after outlines resolve; server
classroom-generation uses it directly).
- Session: persist `courseTitle` on GenerationSessionState for reload safety.
- Tests: add courseTitle parsing coverage; e2e mock done event carries it.
Fully optional in the pipeline — every consumer falls back to the previous
behavior when the field is absent, so legacy/LLM-skipped cases are unaffected.
* fix(generation): make courseTitle propagation robust across all outline paths
Address review feedback on the courseTitle threading.
Prompts — align every outline-prompt schema statement to the 3-key shape.
The user-prompt "final reminder" / top-level shape blocks for
requirements-to-outlines, interactive-outlines, and task-engine-outlines still
demanded the old two-key {languageDirective, outlines} object; as the last
instruction the model reads, that caused it to omit courseTitle and the stage
name to fall back to the raw requirement. Task Engine prompts now emit
courseTitle as well.
Streaming route — normalize the streamed courseTitle (trim, ignore
whitespace-only, cap length) to match the non-streaming parser, and add a
full-buffer fallback so a title emitted after the outlines array or beyond the
head-scan window is still recovered before the done event.
Client — reset latched languageDirective/title on outline retry so a
succeeding attempt that omits them falls back instead of inheriting a failed
attempt's stale value; also carry courseTitle in the stream-end resolve path
that fires when the stream closes without an explicit done event.
courseTitle remains optional on every consumer path.
---------
Co-authored-by: wyuc <zdq1204@gmail.com>
|
||
|
|
612a1471e7 |
fix(agent): respond to the user's turn before lecturing (+ answer-content eval) (#699)
* test(orchestration): add answer-content eval for agent first-reply answering The existing director question-answering eval only checks that the director ROUTES to the teacher; it never generates the dispatched agent's reply, so it cannot catch the case where routing is correct but the agent's first sentence drifts (greets / opens a lecture / pivots) and only a later turn answers. This adds an answer-content eval that runs the real agent-generate inputs (buildStructuredPrompt + convertMessagesToOpenAI), parses the structured output with the app's runtime parser, and uses an LLM judge for two signals: leads_with_answer (first sentence addresses the literal ask) and answered_anywhere (addressed at all). Their gap quantifies "drift-then-answer". A/B mirrors the answering-runner rule-13 strip: baseline removes the "Answering the User's Question" section from agent-system, with_rule is as-shipped. Scenarios are synthetic/anonymized (opening-lecture override, ignored format/capability/navigation request, ignored correction, frustration re-ask, adjacent pivot, vague-clarify) plus clean controls; no real user data. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test(orchestration): give answer-content scenarios live-shaped classroom context The minimal store state (empty scenes, null stage, no profile) made the assembled agent prompt far thinner than the live agent-generate path and suppressed the slide-narration "lecture reflex" that drives opener drift. Each scenario now carries a topical slide scene (title + body + latex elements with ids), a stage (name/description/languageDirective), and a student profile, so buildStructuredPrompt emits a "# Current State", "# Student Profile" and "# Language" of the same shape and bulk as production. The runner reads this context from the scenario instead of a hardcoded empty state. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(agent): respond to the user's turn before continuing the lecture Generalize the agent-system "Answering the User's Question" section into "Responding to the User's Turn": the user's latest message always takes priority over continuing the planned lecture. Beyond factual questions it now explicitly covers the interaction requests agents were ignoring — navigation/pacing ("skip to next page"), format/language ("explain in Chinese"), capability requests it cannot fulfill ("make a video" -> say so + offer an alternative), corrections, frustration re-asks, and vague asks (ask one clarifying question). Strengthen the answer-content eval that measures this: - the navigation case now uses a 3-slide deck so "skip to next page" is well-posed; runner supports multi-scene state and strips the renamed section for the A/B; - judge is fairer to legitimate behavior: explaining primarily in the requested language while keeping technical terms standard counts; a verbal transition to the next slide counts as honoring navigation (no page-turn action exists). Result (gemini-3-flash-preview, n=10): mean leads-with-answer rises from 42% (section stripped) to 93% (as-shipped); all 12 scenarios pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(agent): don't over-respond to bare acknowledgements; guard empty eval set Code-review follow-ups on the "Responding to the User's Turn" change: - Prompt: the absolute "every user message needs a direct response / treat continuing the lecture as a failure" wording had no carve-out for bare acknowledgements ("ok" / "嗯" / "good" / "thanks"), risking the agent halting the lesson to manufacture a reply, and conflicted with agent-initiated discussions where no user turn exists. Add a bare-acknowledgement carve-out, allow a brief "好的"/"Sure" before the answer, and scope the override to actual requests (and to "if there is a user turn to respond to"). - Eval: answer-content-runner exited 0 with a PASS verdict and NaN% aggregates when the scenario set was empty (e.g. an EVAL_SCENARIO typo). Guard it: error and exit 1 when no scenario matches. Re-run unchanged otherwise (gemini-3-flash-preview, n=10): mean leads-with-answer 42% (stripped) -> 94% (as-shipped), 11/12 scenarios pass; the chem opener is variance-prone around the 70% bar and not over-tuned. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * refactor(eval): cr-loop cleanups on answer-content-runner - ScenarioAgentSpec extends the shared ScenarioAgent (./types) instead of re-declaring id/name/role/priority. - Drop the dead expectedPreFix plumbing from ScenarioResult (it was copied in but never read; it stays as an informational annotation in the scenarios). - extractTexts: replace the hand-rolled JSON fallback with the app's finalizeParser, which recovers plain-prose / unclosed-array output the same way the runtime does. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: tighten acknowledgement carve-out; don't surface action-JSON as text (cr round 2) Round-2 review follow-ups. Prompt: the bare-acknowledgement carve-out was too loose — it could swallow ack+request hybrids ("嗯，那为什么X?") as acks (re-introducing drift), double-listed "好的" as both an allowed lead-in and a no-response ack, and listed pacing words ("继续"/"go on") that collide with the navigation bullet. Restrict it to a STANDALONE acknowledgement with nothing else attached; route an embedded question/request to that request; route pacing words to the navigation bullet. Eval: extractTexts gated finalizeParser recovery on !state.jsonStarted, so an actions-only / unclosed array yields "no text" (the judge's intended empty case) instead of finalizeParser's raw-buffer fallback surfacing action JSON as fake speech. Re-run (gemini-3-flash-preview, n=5): no regression — 11/12 pass, meta-requests 100/100/80/100; the chem opener remains variance-prone around the 70% bar. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: address review — honest slide-nav, strict judge booleans, errors fail (cr) Review follow-ups from @cosarah: - Navigation (P2): agents have no slide-navigation action, so "do it" + a verbal transition let "skip to next page" pass the eval while the UI stayed on the same slide. Reframe prompt, judge, and scenario so a correct reply ACKNOWLEDGES the request and is HONEST that it cannot change the slide itself (continue verbally / tell the user how to navigate); pretending it flipped the slide no longer counts. - Judge booleans (P2): answer-content-judge coerced fields with Boolean(...), so a string "false" became true. Parse strictly (real bool or "true"/"false"); malformed output is flagged as an error. - Error accounting (P3): aggregate excluded errored samples from the denominator, letting a scenario pass on one good sample. Errors now count as failures (denominator = all requested samples). Re-run (gemini-3-flash-preview, n=10): navigation 0% -> 100% under the honest framing; aggregate leads-with-answer ~93%, no regression. Opener scenarios (math/chem) remain variance-prone near the 70% bar. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(eval): judge first-sentence lead; use canonical teacher actions (cr round 2) Second review round from @cosarah: - leads_with_answer was judged against texts[0] (the whole first JSON text block), so "Today we'll discuss parabolas. The formula is x=-b/(2a)." counted as leading even though the first sentence drifted. Now extract the first ~2 sentences as the "opening" and have the judge make the call: a brief acknowledgement of the user before the answer counts as leading, but a greeting / self-intro / topic preamble before the answer does not. (Two sentences + semantic judge handles "Sorry, let me clarify. m is inertial mass." vs "Welcome! …" without a brittle keyword list.) - The runner hardcoded TEACHER_ACTIONS, omitting canonical teacher actions (play_video, wb_draw_chart). Use getActionsForRole('teacher') so the available- actions section matches production. Re-run (gemini-3-flash-preview agent, n=10): 11/12 pass, aggregate leads ~90%. The chem opener now correctly fails (~10%): its "Welcome! Today we…" persona greets first, so at sentence granularity it is genuine first-sentence drift (answered-anywhere ~90%) — exactly what the stricter lead measurement should catch. Note: the judge model is deepseek-v4-pro in this run because the anthropic gateway is currently returning Bad Request in our environment; the eval is judge-model-agnostic via EVAL_JUDGE_MODEL. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(eval): judge the opening across all text blocks, not just texts[0] (cr round 3) Review follow-up from @cosarah: leadFromTexts only looked inside the first text block, so a valid multi-block reply like [{"type":"text","content":"Sure."},{"type":"text","content":"The derivative is 2x."}] was judged on "Sure." alone and failed leads_with_answer despite following the allowed acknowledgement-then-answer pattern. Build the opening from the first ~2 speech sentences across ALL ordered text chunks. Re-run (gemini-3-flash-preview, n=10, deepseek judge): 11/12 pass, aggregate leads ~93%. The chem opener stays sub-threshold (~50%) — its "Welcome! Today we…" persona greets before answering about half the time, a genuine first-sentence drift. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: 杨慎 <117187635+cosarah@users.noreply.github.com> |
||
|
RongHuiQiang
|
e6debb093b | fix(vocational): address review feedback | ||
|
jackefn
|
c58893e76f | feat(vocational): add task engine and procedural skill widgets | ||
|
|
893f080f51 |
fix(prompts): constrain action narration to a single teacher voice (#671) (#673)
Action generators sometimes scripted role-play dialogue into the speech `text` — speaker-label parentheticals like （AI助教）：/（显眼包）：/（学生）： and stage-direction cues like （好奇发出）/（笔记动作）/（插话） — which then got read aloud by TTS and shown as narration, even on single-presenter slides. The `Classroom Agents` roster (incl. student personas) is injected into these prompts for `discussion.agentId` selection, and the model dramatized those personas into the teacher's monologue. Add an explicit "single voice, teacher only" constraint to all four *-actions system prompts (slide / quiz / interactive / pbl): no dialogue or lines for other speakers, no parenthetical speaker labels, no stage directions; the agent roster is only for picking a `discussion` agentId; route student reactions through the `discussion` action instead of voicing them inline. Also drop the dialogue-inviting word "conversational" from the slide prompt. Verified by replaying real production scene-actions prompts against gemini-3-flash-preview: cue rate dropped from ~10% to 0/80 on the cue-prone set, with narration quality intact (student reactions now route to a discussion action). Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: 杨慎 <117187635+cosarah@users.noreply.github.com> |
||
|
wyucandClaude Opus 4.8
|
ea049417cd |
fix(prompts): gate media snippets in interactive-outlines template (#626) (#628)
The interactive-outlines template (Ultra Mode) never included the media-generation instructions, so the `mediaGenerations` field was only ever produced by strong cloud models inferring it from context. Weak local models followed the prompt literally and omitted it, so images silently never generated. Mirror requirements-to-outlines by including the same conditional snippet blocks (image / video / media-safety), gated on the imageEnabled / videoEnabled / mediaEnabled variables that both outline paths already pass to buildPrompt. Reuses the existing snippets as the single source of truth rather than duplicating the instructions. Extends media-conditional.test.ts to hold INTERACTIVE_OUTLINES to the same media-gating contract as the standard template. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
wyuc
|
f064590a5e |
test+fix(orchestration): rip maxTurns; restore answer-the-user's-question behavior (#598 follow-up) (#599)
* test(orchestration): add premature-END regression eval (#598 follow-up) Adds eval/orchestration/ following the outline-language pattern: a runner that A/B-tests the director against the same scenario with two prompt variants — current main (post-#554) and a synthesised pre-#554 baseline that strips the role-aware summary labels AND the new system.md rules 10/11/12 together. Pass criterion is now framed as a regression guard rather than a fixture discrimination test: every scenario's post-fix END rate must stay below EVAL_END_THRESHOLD (default 20%). The pre-vs-post Δ is reported as informational data, since #554's reviewer feedback was that earlier fixtures didn't discriminate. Empirical finding across 7 shipped model configs (gpt-4.1-mini, gpt-4o-mini, gpt-5.4-nano, qwen-plus, qwen3.5-flash, deepseek-chat, deepseek-v4-flash, gemini-2.5-flash, claude-haiku-4-5) with 5 scenarios modelled on #511 (incl. the exact Tiananmen-3D objection trace, soft pushback after a long resolved-looking discussion, topic pivot, brief acknowledgement, and explicit teacher-signals-end-then-user-objects): no scenario produced a non-zero END rate in either variant. Useful data for the #554/#598 discussion — the prompt-layer rules don't measurably change behavior on these shipped models with these prompts, but the eval is now wired so future regressions show up. * test(orchestration): add director question-answering eval (#598 follow-up) Adds an A/B eval probing whether the director routes correctly when the conversation contains an unanswered user question — both in the first-turn drift case (agents started answering but drifted onto adjacent topics, no frustration yet) and in the escalated frustration case (user has already complained multiple times). The prod symptom this targets ("答非所问"): agents reply on adjacent topics rather than answering the literal question, and the director keeps picking peer agents for "variety" instead of routing back to the teacher to actually answer. Scenarios: 5 synthesized cases across math, biology, English grammar, physics, and calculus. 3 are first-turn drift (no frustration yet), 2 include frustration signals. Each captures: user asks specific question, agents reply on adjacent topics, director is asked to pick the next move. Decision rule-judged into USER | TEACHER | OTHER_AGENT | END: - USER → ✓ valid (cue user to clarify) - TEACHER → ✓ valid (re-route to teacher to answer) - OTHER_AGENT → ✗ wrong (peer-agent "variety" routing — the bug) - END → ✗ wrong A/B: - baseline : current director template with rule 13 stripped - with_rule : current director template as-shipped Pass = with_rule.correctRate ≥ EVAL_PASS_THRESHOLD (default 0.7). Pre-vs-post Δ is reported as informational only. Note this commit only adds the eval; the rule it strips/restores in the A/B is injected in the next commit. * fix(orchestration): inject director rule for answering user questions Adds rule 13 to lib/prompts/templates/director/system.md so the director recognizes when the conversation contains an unanswered user question and routes appropriately — to the teacher (literal role match) by default, or to USER cue when the question is ambiguous or the user has already complained multiple times. This addresses the "答非所问" symptom: when agents drift onto adjacent topics, the director keeps picking peer agents for "variety" rather than routing back to the teacher to actually answer. The rule is preventive (applies whenever the question is unanswered, including the first turn after drift) rather than reactive-only (waiting for the user to express frustration). Rule wording uses constrained-output framing: literal `role: teacher` match (not "best fit" / "highest priority" / "most knowledgeable"), and an explicit list of forbidden choices (peer agents, END). Earlier softer wordings let the model pick the assistant when it seemed more topically relevant; the literal-match wording prevents that. Validated by the question-answering eval added in the previous commit. On google:gemini-3-flash-preview, 5 samples per variant across 5 synthesized scenarios: | Scenario | baseline | with_rule | |----------------------------------------------------|----------|-----------| | math_quadratic_axis_drift_no_frustration | 100% | 100% | | bio_dark_reaction_drift_no_frustration | 80% | 100% | | english_team_isare_drift_no_frustration | 20% | 100% | | physics_inertial_mass_escalated_frustration | 0% | 100% | | calculus_product_rule_drift_no_frustration | 60% | 100% | | **mean** | **52%** | **100%** | 3 of 5 scenarios reproduce the bug at baseline (correct rate ≤ 60%); the rule lifts every scenario to 100%. The remaining 2 baselines are already high — the rule doesn't regress them. * fix(orchestration): teach agents to answer the user's literal question Adds an "Answering the User's Question" section to the agent system prompt. Before this, the agent SP had no rule for direct-answering — the persona, role guideline, current-state context, and length guidelines all pushed the agent to advance the lesson and inspire thought, with no explicit directive to first answer the user's literal question. The result observed in prod ("答非所问"): when the user asks a specific question (a formula, a yes/no, a term), agents reply with adjacent concepts, examples, or follow-up questions — but never the literal answer. The new section directs the agent, on every response where the user message contains a question, to: - lead with the concrete answer to the literal question, - not pivot to an adjacent topic even when it seems pedagogically richer, - treat "inspire thought" and peer-differentiation guidance as applying only AFTER the literal question is answered, - say "I don't know" directly when uncertain rather than answering a different question, - on frustration signals from prior turns, look back at the original question (before the frustration) and answer THAT specifically. This is the agent-side counterpart to the director rule landed in the previous commit. Director routes to the teacher when the question is unanswered; the teacher now has explicit guidance for what to do when dispatched. * refactor+fix(orchestration): rip maxTurns end-to-end; distinguish errors from completion Two coupled cleanups in one commit since the outcome-handler rewrite naturally falls out of removing the client-side max-turns loop. The director graph used `maxTurns: turnCount + 1` to force "one director→agent cycle per request" — a hack expressing the single-round contract as a turn limit. This commit removes the hack and the surrounding user-facing setting, replacing implicit intent with explicit graph topology. Server: - `director-graph.ts`: drop `maxTurns` from state, drop the hard-cap check at the top of `directorNode`, change `agent_generate → director` edge to `agent_generate → END`. Single-round contract is now expressed by topology. - `lib/types/chat.ts`: drop `maxTurns` and `currentTurn` from `SessionConfig`; drop `maxTurns?` from `CreateSessionRequest.trigger`. Client: - `lib/store/settings.ts`: drop `maxTurns` state, `setMaxTurns` action, the `'10'` default, and the localStorage migration path. - `components/agent/agent-bar.tsx`: drop the entire compact-stepper UI block plus the unused MessageSquare/Minus/Plus lucide imports. - `components/settings/agent-settings.tsx`: drop the AgentSettings prop and the multi-agent-only Input block. - `lib/chat/agent-loop.ts`: drop `maxTurns` parameter, change `while (turnCount < maxTurns)` to `while (true)` controlled by inner exits, drop `'max_turns'` reason from `AgentLoopOutcome`. - `components/chat/use-chat-sessions.ts`: drop the maxTurns lookup, drop the trailing arg to runAgentLoop, drop vestigial `maxTurns: 0` / `currentTurn: 0` from session creation. - `eval/whiteboard-layout/runner.ts`: drop unused MAX_AGENT_TURNS plus the trailing arg. i18n: drop `settings.maxTurns` and `settings.maxTurnsDesc` from six locales (en-US, zh-CN, zh-TW, ja-JP, ar-SA, ru-RU). Motivation for removing the user-facing setting: the LLM director controls round length via cue_user / END decisions, so a numeric "max discussion turns" knob is redundant with rule-driven routing. When the director prompt was buggy, the cap masked the bug by force- ending; once director routing is fixed, the cap just removes user agency. Before, every non-`cue_user` outcome of the agent loop mapped to `status: 'completed'` in `use-chat-sessions.ts`. That conflated four distinct paths: - `end`: director chose END (legitimate completion) - `empty_turns`: two consecutive agents returned no text or actions (parser failure, model timeout, etc.) - `no_done`: SSE stream ended without a `done` event (infrastructure failure) - `aborted`: user pressed stop (handled elsewhere) The new switch routes `end` to completed, `empty_turns` and `no_done` through `clearLiveSessionAfterError` so the user sees an inline System message describing what went wrong instead of a quiet "completed". - `lib/types/chat.ts`: add `'error'` to `SessionStatus`. - `components/chat/use-chat-sessions.ts`: rewrite the outcome handler as a switch; have `clearLiveSessionAfterError` set `status: 'error'` so all entry points (the runAgentLoop outcome AND existing throw paths from fetchChat / SSE error events) converge on the same visible state. - `components/chat/session-list.tsx`: add `AlertCircle` red icon for the new error status. - i18n: add `chat.error.emptyAgentResponses` and `chat.error.streamInterrupted` strings to all six locales. This is the in-product fix that the prod investigation surfaced as an independent root cause of the #511 "discussion ended unexpectedly" symptom — separate from the director and agent prompt fixes in the earlier commits of this PR. * fix(chat): drop dead settingsState binding; wire eval:orchestration:answering script Two minor CR follow-ups: - components/chat/use-chat-sessions.ts: the `settingsState` lookup at the top of runAgentLoopFn was the entry point for the maxTurns lookup ripped in the previous commit; with maxTurns gone, the binding is unused and ESLint flags it. Drop the line. (Other uses of useSettingsStore.getState() elsewhere in this file are unaffected.) - package.json: the answering eval runner had no convenience script, while the premature-end runner did. Add eval:orchestration:answering for symmetry and discoverability. * fix(orchestration): route ambiguous/unaddressed user turns to teacher, never USER cue Addresses reviewer issues 2 & 3 (ambiguous "帮我看一下这个" / "Can you take a look at this?"): the director was cueing USER for vague input, which makes no agent speak — the user faces dead air with nothing to react to ("nobody responds"); and when an agent did speak it guessed a topic instead of asking what the user meant. Root cause was the previous rule 13, which offered `{"next_agent":"USER"}` as a valid response to an ambiguous user turn. A USER cue is the wrong tool here: it returns control to the user without any agent speaking. The right behavior is to dispatch the teacher, who either answers or — if the message is too vague — asks a clarifying question out loud. Director (`director/system.md` rule 13): - When the most recent user turn is unaddressed (clear question, vague request, OR frustration signal), the only acceptable output is the literal `role: teacher` agent id. - USER cue is now explicitly forbidden as a response to a user turn (reserved for rule 7, where an agent has just asked the user something). - Tightened the END prohibition: a high turn count or a thoroughly-discussed topic does NOT mean the specific question was answered. Added rule 5 (don't-drag-on) to the override list, since that pressure was driving the director to END on well-discussed-but-unanswered turns. Agent (`agent-system/system.md`): - Added a clause to "Answering the User's Question": if the user's message is too vague to answer (no clear referent), ask ONE short clarifying question with a concrete option or two — do not guess a topic and lecture, do not stay silent. Eval (`answering-runner.ts` + `scenarios/answering.json`): - Reclassified USER as wrong (dead air); only TEACHER counts as correct now (teacher answers, or asks a clarifying question for vague input). - Added two ambiguity scenarios: a bare "帮我看一下这个" opener, and a vague "再帮我看看这个吧" follow-up after a real Q&A. - Made the rule-13 strip regex match by rule number rather than heading text, so it survives heading rewrites. On google:gemini-3-flash-preview, 10 samples per variant, all 7 scenarios PASS (with_rule mean ~97%, baseline ~57%). The two ambiguity scenarios: ambiguous_look_at_this 100%/100%, ambiguous_vague_followup 20%→100%. Manually verified the agent side: for "帮我看一下这个" the teacher now replies "…你想讨论的是具体的代码逻辑，还是 Agent 的决策流程图呢?" (clarifying question + concrete options) instead of guessing. Note: reviewer issue 1 (follow-up not answered) could not be reproduced on this branch across 7 configurations; it is consistent with the deployed environment running older code and/or provider API flakiness, which the C7 error-vs-completion distinction in this PR already surfaces instead of a silent 已结束. * docs(eval): fix stale physics scenario description after USER→wrong reclassification The physics_inertial_mass scenario description still said "USER-cue signal — but TEACHER re-answer is also valid", which contradicts the new classifier (USER is dead air = wrong; only TEACHER is correct). Cosmetic only — the description field is printed into the report, never used in classification. |
||
|
Ashutosh Ranaand杨慎
|
e5148bece5 |
fix(orchestration): restore agent attribution in director summary (#554)
* fix(orchestration): use metadata-based human/agent discrimination in director Closes #511 The original approach used AGENT_PREFIX_RE on converted message content to distinguish human turns from agent turns. This was unreliable: message-converter.ts applies a [senderName]: prefix to ALL role:'user' messages unconditionally, so extractLastHumanMessage() matched every human message as an agent turn and always returned null — the Open Student Question section never fired. Fix per reviewer feedback (use real structured metadata, not display text): - extractLastHumanMessage() now takes StatelessChatRequest['messages'] (pre-conversion) and uses msg.metadata.originalRole === 'user' as the source-of-truth discriminator. Text is extracted from msg.parts directly — no content parsing. - summarizeConversation() uses msg.role for human/agent discrimination. In the director path, convertMessagesToOpenAI() is called without currentAgentId, so peer agent messages remain role:'assistant' and role:'user' reliably identifies human turns. The [senderName]: prefix is stripped from summary output cosmetically only. - buildDirectorPrompt() 10th param changed from OpenAIMessage[] to StatelessChatRequest['messages']; director-graph.ts passes state.messages (already in scope at line 164, no extra computation). - Tests updated to use director-path message shapes: humanMsg() carries [You]: prefix, agentMsg() uses role:'assistant'. extractLastHumanMessage tests use originalRole metadata. * refactor(orchestration): move attribution fix to prompt layer; add eval fixtures Replaces procedural extractLastHumanMessage + buildOpenStudentQuestionSection wiring with three declarative rules in director/system.md — labels are authoritative, END is blocked by an unresolved student question, and a brief acknowledgment does not constitute resolution. Removes: extractLastHumanMessage, buildOpenStudentQuestionSection, {{openStudentQuestionSection}} placeholder, originalMessages 10th param. Keeps: summarizeConversation role-based discrimination (still the core fix). Adds eval/orchestration/ (runner.ts, judge.ts, reporter.ts, types.ts, scenarios/premature-end.json) following the outline-language eval pattern, with a deterministic premature-END scenario as the must-have regression guard. * chore(orchestration): remove eval fixtures and stale tests per review Per wyuc's review on PR #554: the eval/orchestration fixtures pass on main before the fix, so they don't discriminate — pull them out so the PR lands as a clean prompt-layer change. Track eval as a follow-up. - Remove eval/orchestration/ (judge, runner, reporter, types, scenario) - Remove eval:orchestration npm script from package.json - Delete 3 stale tests from tests/prompts/templates.test.ts that called buildDirectorPrompt with a 10th param removed in the refactor - Fix prettier formatting in conversation-summary.test.ts and templates.test.ts (was masking the TS break in CI) All checks pass: prettier clean, 333 unit tests green, 0 lint errors, tsc --noEmit clean. --------- Co-authored-by: 杨慎 <117187635+cosarah@users.noreply.github.com> |
||
|
wyuc
|
2dff6d10be |
feat: add generated video manifest refs (#540)
* feat: add generated video manifest refs * style: format video manifest changes * fix: address video manifest review feedback |
||
|
Yizuki_Ame
|
3fafca5f99 |
feat(prompts): conditional media snippets in generation prompts (#490)
* feat(prompts): conditional media snippets in generation prompts (#483) Replace always-included media instructions + negative-override pattern with conditional snippet includes gated on imageEnabled/videoEnabled flags. - Extend prompt loader with {{snippet:name?if=condition}} syntax - Extract 15 media instruction snippets from outline and slide templates - Wire imageEnabled/videoEnabled through outline-generator, scene-generator, and route handler - Remove mediaGenerationPolicy negative-override from all call sites - Close gap: slide content generation now receives media policy - Unify pre-output checklist to labeled bullet format * refactor(prompts): replace ?if= snippet syntax with {{#if}} inline blocks Address review feedback on PR #490 — the file-per-fragment pattern was too fine-grained (6 of 15 new snippets were ≤2 lines). Changes: - Add {{#if name}}...{{/if}} block syntax to the loader (~10 lines) - Inline 9 micro-snippets (≤4 lines each) back into host templates - Keep 3 large instruction snippets as {{snippet:name}} wrapped in {{#if}} at the call site - Drop ?if= syntax and PromptLoadOptions.conditions — conditional blocks read directly from buildPrompt's variables - Update tests and README Processing order: snippets → conditional blocks → variable interpolation * style: format media-conditional test with prettier |
||
|
|
fc6b186929 |
refactor(whiteboard): file-based prompts + geometry conflict detection (#485)
* feat(prompts): declare whiteboard role prompt IDs + whiteboard-reference snippet * feat(prompts): add whiteboard role template skeletons + reference snippet stub * refactor(orchestration): move buildWhiteboardGuidelines to role-specific markdown templates * docs(prompts): whiteboard guidelines now in markdown templates * feat(prompts): whiteboard-reference canvas + JSON output sections * feat(prompts): whiteboard-reference action schemas for text/shape/line * feat(prompts): whiteboard-reference action schemas for latex/chart/table * feat(prompts): whiteboard-reference action schemas for code/delete/clear/close * feat(prompts): whiteboard-reference LaTeX JSON escape section * feat(prompts): whiteboard-reference bounds + overlap section * feat(prompts): whiteboard-reference font + latex height tables * feat(prompts): whiteboard-reference pre-output checklist * feat(prompts): expand teacher whiteboard template * feat(prompts): expand assistant whiteboard template * feat(prompts): expand student whiteboard template * test(prompts): assert whiteboard-reference sections reach every role prompt * tune(prompts): cap teacher element budget at 4/turn, encourage wb_clear on crowded board * tune(prompts): simplify teacher template — single conservative rule + state-awareness emphasis * tune(prompts): add LaTeX-height × text-fontSize pairing table for visual weight consistency * feat(orchestration): preserve image parts from UIMessage to AI SDK ModelMessage * feat(eval): attach prior-turn whiteboard screenshot as user message image part Gated behind EVAL_ATTACH_PRIOR_SCREENSHOT=1 env. When on, captures a screenshot after every turn (not just checkpoints) and attaches it as a file part with mediaType:'image/png' on the next user message. Teacher prompt gets a new 'Prior-state image' section telling the agent to treat it as direct visual feedback. * fix(orchestration): strip data: prefix before passing image to AI SDK Vercel AI SDK's streamText/generateText treats ImagePart.image as a URL to fetch when it's a string. data: URLs fail the http/https scheme check and throw AI_DownloadError. Strip the data URL prefix and pass raw base64 with mediaType separately — AI SDK treats base64 strings as data content. * refactor(whiteboard): add conflict summarizer, drop prior-screenshot feature Eval across flash (image-on vs off × repeat-3) and pro (same) showed the prior-screenshot image-input feature gives ~0 net benefit: +0.4 overall on flash and −0.6 on pro. The image feedback loop's theoretical upside (agent self-correcting from visual) didn't materialize — weak models couldn't act on the image and strong models "found things to fix" and over-corrected. Meanwhile programmatic geometry detection on the raw JSON cleanly lifted overlap from 6.3 → 8.1 on flash. So: remove the image-input plumbing, keep a programmatic equivalent. ## Removed (reverts |
||
|
wyucandClaude Opus 4.7
|
550c43a3f3 |
fix(generation): unify languageDirective across outline + scene pipeline (#472) (#474)
* fix(generation): unify languageDirective across outline + scene pipeline (#472) Root issue: languageDirective was dropped, renamed, or hardcoded at several points in the scene-generation pipeline, silently degrading prompt-level language control. Slide paths survived because their fallback happened to hint requirement-language; widget paths degraded to zh-CN default. Fix plan (all code-only threading + prompt hygiene, no schema changes): 1. Thread languageDirective through every buildPrompt in scene-generator.ts: - Add param to generateSlideContent / generateQuizContent / generatePBLSceneContent - Forward from generateSceneContent's slide / quiz / pbl branches - Include key in the four generateSceneActions buildPrompt calls - Demote generatePBLSceneContent's hardcoded English to a fallback - Drop the dishonest `as 'zh-CN' | 'en-US'` cast on the widget path 2. Unify variable naming across widget templates + callers: - Rename {{language}} → {{languageDirective}} in 6 widget templates (simulation / diagram / code / game / visualization3d / widget-teacher-actions) - generateWidgetContent and generateWidgetTeacherActions now pass `languageDirective: languageDirective || ''` instead of a locale-code shim 3. Make the outline LLM reliably emit the wrapper object: - requirements-to-outlines/system.md + user.md restructured so the "top-level JSON object with languageDirective + outlines" contract is surfaced up-front, short, and unambiguous — drowning-by-scene-examples removed. Important Reminders reordered: top-level shape first. - interactive-outlines/system.md + user.md aligned with main outliner: adds full Language Inference section, requires wrapper output, drops the orphan {{language}} placeholder that never had a value source. 4. Remove lib/prompts/loader.ts prompt/snippet caches: - Disk reads for prompt templates are not on any hot path (LLM call dominates by 5 orders of magnitude). The cache's only observable effect was blocking prompt edits from taking effect without restart — which masked the wrapper bug during this investigation. Verification: - 7 new regression tests in tests/generation cover directive threading at each buildPrompt site plus the PBL caller-forwarding guarantee. - outline-language eval (50 scenarios, gemini-3-flash-preview inference, qwen-3.5-flash judge): 48/50 (96%). The 2 persistent failures are in the proxy-request category (teacher / tutor preparing lessons for students) where the system prompt's rule 5 has genuine ambiguity — orthogonal to this fix. - Curl burst tests: 8/8 runs each for regular-mode and interactive-mode outline endpoints emit the wrapper object and extract a detailed inferred directive. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * style: apply prettier formatting to loader.ts and new test file Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: thread languageDirective in scene-builder and classroom-generation Code review caught two additional sites missing the directive: - lib/server/classroom-generation.ts:371 — generateSceneActions call was missing languageDirective, reproducing the issue #472 symptom in the shareable-classroom server generation path. - lib/generation/scene-builder.ts:114 — buildSceneFromOutline's generateSceneActions call had the same miss; latent (no current live caller) but wrong for consistency. Also renumber duplicate `8.` in requirements-to-outlines/system.md scene-level rules. Extend regression tests with a buildSceneFromOutline end-to-end case that captures both the content and actions prompts via a stub aiCall and asserts the directive reaches both phases — this would have caught the scene-builder miss. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * review: widget-path test + docs + fallback constant (PR #474) Address @cosarah's review feedback: - Add widget-path regression test (tests/generation/...): interactive scene with widgetType=simulation captures both widget content and widget-teacher-actions prompts, asserts the directive reaches each and no `{{language}}` / `{{languageDirective}}` placeholder leaks. This is exactly the path issue #472 was filed for. - Rewrite lib/prompts/README.md "Caching" section: the cache was removed in this PR; README still instructed callers to use a clearPromptCache() function that no longer exists. - Sync interactive-outlines/user.md with the main outliner's "## Language Context" reminder block (line 13-19) so both outline paths give the LLM the same language-inference reminders. - Hoist 'Teach in the language that matches the user requirement.' into DEFAULT_LANGUAGE_DIRECTIVE, exported from outline-generator. Replaces 4 copies across outline-generator, scene-generator, and the scene-outlines-stream route. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
wyucandClaude Opus 4.7
|
f40c92f1aa |
refactor(orchestration): migrate prompt builders to file-based templates (#459)
* refactor(prompts): move loader to lib/prompts/ Move file-based prompt loader from lib/generation/prompts/ to project-level lib/prompts/ so it can be shared by orchestration and PBL prompts in subsequent commits. - Templates and snippets moved alongside the loader - getPromptsDir() now points at lib/prompts/ - Generation pipeline imports updated to @/lib/prompts (outline-generator, scene-generator, search-query-builder, scene-outlines-stream route) - Add tests/prompts/loader.test.ts as sanity gate Pure file move — no behavior change. * test(orchestration): add baseline snapshots for prompt builders Snapshot tests cover variant matrix: - buildStructuredPrompt: role × scene-type × peer/ledger/discussion/profile - convertMessagesToOpenAI: mixed message kinds - summarizeConversation: truncation - buildDirectorPrompt: Q&A vs discussion, ledger, profile - buildPBLSystemPrompt: default config These lock current output so the body refactor in subsequent commits can be verified byte-equal. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test(orchestration): plug snapshot coverage gaps from review - Split convertMessagesToOpenAI test: same-agent (existing) + cross-agent (new). The original test passed currentAgentId matching the message's agentId, so the cross-agent role-conversion branch was never exercised. - Add 'teacher / quiz scene' snapshot to lock the spotlight/laser strip path in getEffectiveActions. The previous variant-7 test (whiteboard-open) only triggered the mutual-exclusion warning, not the strip — renamed it to reflect what it actually tests. - Add assistant-role variant (was uncovered: ROLE_GUIDELINES, buildLengthGuidelines, buildWhiteboardGuidelines all branch on role). * refactor(orchestration): extract summarizers from prompt-builder.ts Move state-context, virtual whiteboard ledger replay, peer context, message converter, and conversation summary out of the 890-line prompt-builder.ts into focused modules under lib/orchestration/summarizers/. prompt-builder.ts now only owns prompt assembly. director-graph imports updated to the new locations. Snapshot tests pass byte-equal — no behavior change. * refactor(orchestration): move agent-system prompt to template file The buildStructuredPrompt template literal is now lib/prompts/templates/agent-system/system.md, assembled by a thin variable-pass through the shared prompt loader. Per-variant ternaries (format example, ordering, spotlight examples, mutual-exclusion note) are kept as module-level constants in prompt-builder.ts. ROLE_GUIDELINES, buildLengthGuidelines, and buildWhiteboardGuidelines stay too (they may further migrate to template partials in Phase 2). Also: PROMPT_IDS gains a 'satisfies Record<string, PromptId>' clause to prevent constant/union drift as more IDs are added. Snapshot tests (12 prompt-builder + 4 director + 1 pbl) pass byte-equal. * refactor(prompts): align agent-system template with project conventions - Rename agent-system placeholders snake_case → camelCase to match the convention already used by slide-actions/user.md and other generation templates. Settles convention before Task 5 lands two more templates that would otherwise inherit drift. - Extract '## Speech Guidelines (CRITICAL)' into lib/prompts/snippets/speech-guidelines.md for reuse by director and PBL prompts in Task 5. Snapshot tests pass byte-equal — placeholder name changes are invisible in rendered output, and snippet inclusion happens at load time. * refactor(orchestration,pbl): move director + PBL prompts to templates director-prompt.ts and pbl-system-prompt.ts now use the shared template loader. Bodies collapse to thin variable assembly. Three orchestration-tier prompts now share one infrastructure: agent-system, director, pbl-design. Snapshot tests pass byte-equal. * refactor(orchestration): extract WhiteboardActionRecord + AgentTurnSummary to types module These two interfaces are imported by 6+ modules including summarizers/ — having them live in director-prompt.ts created an awkward upward import direction (summarizers reaching back to a sibling prompt builder for types). Move them to lib/orchestration/types.ts and update all callers. director-prompt.ts now imports them too. Pure type-location refactor — snapshots pass byte-equal. * test: remove snapshot tests in favor of eval-as-integration-test Per review feedback: byte-equal snapshots were the wrong invariant for this refactor. The goal was a stable substrate for editing prompts, not a frozen contract. Snapshots would have generated maintenance churn (~100-200 line diffs per intentional Phase 2 prompt tweak) without providing value the eval doesn't already. Test posture going forward: - tests/prompts/loader.test.ts (3 tests) — bounds the loader infrastructure (template load, snippet inclusion, variable interpolation, missing-id behavior) - pnpm eval:whiteboard — bounds end-to-end agent loop, template composition, and integration with chat/director/state-manager Removed: 1879 lines of .snap + 320 lines of test scaffolding + fixtures. Net PR diff drops by ~2200 lines. * docs(prompts): add README + structural assertion tests (review feedback) Addresses two items flagged by PR #459 final review: 1. lib/prompts/README.md — conventions, syntax, gotchas, and local-testing recipe for template authors. Previously there was no doc for the stated goal of "non-engineers can edit prompts." 2. tests/prompts/templates.test.ts — 8 structural assertions covering: - no surviving {{...}} placeholders in any rendered template - role dispatch (teacher → LEAD TEACHER, student → not) - scene-type action stripping (quiz scene has no spotlight/laser) - director prompt output spec mentions next_agent These replace the removed byte-equal snapshot suite at much lower maintenance cost — they assert behaviors the refactor must preserve, not exact bytes. * chore(prompts): relocate templates added on main during rebase PR #461 (interactive mode clean) added 7 new templates under lib/generation/prompts/templates/ (the pre-refactor location). Rebase onto origin/main landed them at the old path; git's rename detection didn't forward them through the directory move in db56c40. Manually move them to lib/prompts/templates/ to match the new convention, and fix scene-generator.ts's relative import ('./prompts/types' → '@/lib/prompts/types') since the dir no longer exists as a sibling. No behavior change — templates read from PROMPT_IDS values via getPromptsDir() which already points at lib/prompts/. * refactor(prompts): address review feedback from @cosarah - loadSnippet now throws on missing file instead of silently returning a literal {{snippet:id}} string. A typo like {{snippet:speach-guidelines}} now fails at load time instead of reaching the LLM. - README gains a "Still in TypeScript" section listing the role-conditional content that still lives in prompt-builder.ts (ROLE_GUIDELINES, buildLengthGuidelines, buildWhiteboardGuidelines) so contributors expecting a pure-markdown workflow know where to look. - Expand tests/prompts/templates.test.ts to cover the conditional branches Phase 2 is most likely to touch (9 new assertions): - assistant role dispatch - peer-context section toggles on agentResponses presence - language constraint toggles on stage.languageDirective presence - director Q&A vs discussion mode branching - pbl-design {{issueCount}} substituted at all 3 occurrences - placeholder-naming-convention lint scans every template for non-camelCase placeholders (slide-content grandfathered) - Comment above interpolateVariables regex documents why kebab-case placeholders pass through silently (the lint test now catches them). - New test in loader.test.ts locks the throw-on-missing-snippet behavior. --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |