mirror of
https://github.com/THU-MAIC/OpenMAIC.git
synced 2026-10-17 05:29:46 +08:00
@openmaic/generation@0.3.0
31
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
ceb05adb41 |
feat(media): allocate generated assets through the registry (#1007 part 2, step b) (#1039)
* feat(media): establish shared asset ownership primitives Introduce global browser asset-pool ownership, asset-reference collection, stage reclamation planning, and lease-based URL access. Carry allocated media identity through storage and generation boundaries with regression coverage. * fix(media): enforce safe resolution across every consumer Route image, video, thumbnail, presentation, and video-export consumers through one resolution state machine. Prevent opaque allocated or generated references from reaching render and export sinks, with fallback and ownership tests. * fix(media): protect document-owned assets across mutations Allocate pool bytes before compatibility writes and document commits, then roll back uncommitted generations safely. Preserve document ownership across edits, retries, imports, speech generation, scene changes, and stage deletion. * fix(media): scope retries and tighten ownership guard Scope retries to the target scene and slide and refuse ambiguous shared-reference mutations. Expand retry rendering coverage and keep direct pool URL resolution behind the shared lease owner. * fix(storage): avoid nested lock during stage cleanup Execute prepared reclamation plans against an explicitly deleted document so the compatibility cascade cannot re-enter the per-document lock. Cover deletion of stage-owned media rows even when the document has no references. * refactor(media): confine reclamation to stage deletion Remove inline pool and compatibility-row cleanup from element, speech, scene, and audio replacement flows. Keep whole-stage reclamation behind explicit stage ownership, preserve stage-less legacy audio rows, and document deferred document-truth sweeping. * fix(media): close retry and resolution gaps Restore shared source tasks after successful forks, scope retries across both whiteboard locations, and wait for parallel TTS workers before rollback. Resolve background media through import, rendering, and PPTX export paths while keeping retry controls visible over last-good bytes. * test(media): execute the consumer safety matrix Replace source-substring checks with resolver seam execution across all six UI consumers, stage hydration, and both exporters. Pin each rollback layer independently and harden the ownership guard against aliased pool imports. * chore(packages): publish the additive DSL field Bump the DSL patch version for the optional speech-action field. Keep the transitional reclamation policy app-owned and document the legitimate transaction rollback removals there. * fix(media): make retry rollback task-safe Delay allocation task re-keying until final document reconciliation succeeds, restore shared source tasks on failed forks, and require exact stage-whiteboard targets before falling through from a missed scene. * fix(media): clear private assets and refresh leases Delete the asset-pool database during the confirmed local-data wipe. Notify the app lease layer after same-id replacement so mounted consumers re-resolve current bytes without reaching into storage internals. * test(media): pin closure safety guards Exercise the CSS allocation boundary, unfiltered legacy-row ownership, delete-and-undo byte survival, and real distinct consumer seams. Restore the video-only manifest overwrite condition and share the direct video resolution hook across both element variants. * test(media): narrow rollback element assertion Narrow the reconciled slide element to an image before checking its source so the rollback regression remains type-safe under the full root compiler configuration. * fix(media): close lease refresh races Gate the first batch publication by unique resolved refs, then publish every replacement snapshot without mutating the prior React state object. Serialize invalidation behind pending releases, evict rejected refreshes, and register replacement observation at the pool boundary. * fix(media): reopen pool after clear failures Always evict the singleton once its store has been closed, including blocked and failed database deletion paths. Report blocked deletion as deferred and prove a later write uses a fresh live store. * fix(media): isolate shared retry progress Track forked regeneration under the selected element until it receives a fresh asset identity, leaving the shared source task and bytes untouched. Surface targeted failures through renderer task lookup and skip the redundant fork reconciliation lock. * fix(media): close final asset retry gaps Keep blocked asset clears fail-loud until a successful retry, with actionable settings guidance. Clear failed shared-fork state across durable and live key spaces, and pin renderer lookups, lease publication identity, and committed-ref rollback protection. * fix: preserve actionable retry failures Localize the blocked cache-clear recovery hint across every supported locale and pin the deferred-error mapping. Retain durable fork failure rows while retries run, deleting them only after successful generation so unstructured failures survive reload. * fix(media): hydrate legacy stored video thumbnails Home-page recent-video thumbnails regressed for legacy Dexie mediaFiles rows keyed by gen_vid placeholders: the reworked hydration resolved the row's bytes through the sealed resolver but never surfaced the stored blob (and its poster) as object URLs for the preview card, so the CI recent-video-thumbnail e2e specs found no visible element. Hydration now materializes legacy stored video rows into blob URLs for both the element src and poster while keeping the resolver invariants: opaque refs still never reach a DOM src, and concrete addresses are never blanked. Unit pins cover the seam so the vitest suite catches this class without a browser. * fix(media): resolve sole restored legacy video Classroom playback restored tasks by exact document media references. Legacy gen_vid references can outlive the key used by the one persisted video row, leaving the player on a placeholder even though bytes were restored. Select the sole completed stage video only for legacy sequential refs after exact and reconciled matches. Keep exact failures authoritative, refuse ambiguous candidates, and cover success, ambiguity, and failure precedence in unit tests. * fix(media): scope legacy video recovery Decide restored legacy video recovery once from the complete document and record it through the shared task lookup consumed by playback, editing, and resolved slides. Keep ambiguous documents as placeholders, apply the same decision to thumbnail hydration, and preserve exact failure precedence. * fix(media): exclude claimed video recovery tasks Model restored legacy recovery as a two-pass match across document video elements and task rows. Remove tasks claimed by exact, targeted, or placeholder lookup before applying the sole-candidate fallback, and cover the ownership/cardinality matrix. * fix(media): unify video element resolution Centralize source, task, poster, and legacy recovery decisions for every video consumer. Ensure direct URLs win over opaque refs and play_video waits on element-targeted retry tasks. * refactor(media): route legacy recovery through resolver Let document-aware consumers request legacy video recovery through the unified element binding API, keeping thumbnail hydration on a single decision path. * fix(media): preserve import refs and prefer pool bytes Recognize unambiguous extensionless relative media addresses during classroom import. Resolve allocated export and thumbnail assets from the shared pool before falling back to lagging compatibility rows. * fix(export): preserve concrete video sources Route PPTX video elements through the shared media binding resolver so an unresolved opaque reference cannot replace a playable source. Complete browser and persisted-store cleanup when asset-pool deletion is deferred, while retaining distinct hard-failure behavior. * fix(media): fork retries without exclusive ownership Enumerate logical asset owners across every persisted document before allowing global pool replacement. Thread explicit targets through fresh-id rewrites and cover cross-document aliases plus unreadable ownership. * fix(media): guard global asset reclamation Share a fail-closed persisted-document liveness check between stage deletion and retry replacement. Preserve cross-document pool aliases while deleting stage-owned compatibility rows and cover enumeration failures. * fix(media): cover complete slide asset references Route slide media traversal through a shared mutable slot contract so backgrounds participate in export, thumbnail hydration, collection, and rewrite lifecycles. Snapshot complete surviving-document refs once per reclamation and preserve manifest-only owners while retaining fail-closed behavior. * fix(media): preserve exclusive retry asset ids Allow targeted retries to replace exclusively owned pool assets in place. Keep shared and unprovable ownership paths on fresh allocations, and pin production-shaped retries plus compatibility-row cleanup. * fix(media): revalidate asset bindings at completion Recheck repository-wide ownership before replacing generated media and fork scoped retries when exclusivity changed. Route video export selection through the unified resolver and keep concrete posters independent of task state. * fix(media): count unflushed owners and broadcast replacements The completion-time exclusivity proof read only the persisted document, but slide duplication updates the Zustand aggregate synchronously and schedules persistence behind a debounce. A retry finishing inside that window saw a single persisted owner and replaced the bytes behind a reference the duplicate also held. The proof now also counts owners in the live stage snapshot when that snapshot represents the stage being retried, so an unflushed duplicate forks instead. Same-id replacement notifications were realm-local, so a second tab showing the same classroom kept its lease pinned to the superseded blob URL. The notification now travels over a BroadcastChannel; each receiving realm runs its own observers against its own pool, so a spoofed message can at most force a re-resolve. A missing or failing channel never fails the replacement. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(media): bind replacement listeners and spare shared audio rows A realm that only renders never sends a replacement, so binding the channel from the sender path left passive tabs deaf to peers. Binding now happens where the observer is registered, when the asset-pool module loads, and the receiving realm resolves its own pool lazily so a cleared or unavailable pool degrades to the next resolve instead of throwing. Observer notifications ran under Promise.all, so a rejection surfaced after BrowserAssetStore.replace had already committed and turned a durable success into a reported failure. They are settled individually now; the callback is wrapped because a synchronous throw would otherwise escape before allSettled sees the array. audioFiles rows are keyed globally by audioId, so deleting a stage removed the sole row for an id a surviving document still referenced — playback and both export paths read that table directly and cannot fall back to the preserved pool blob. Rows are now filtered against surviving references, while a failed enumeration still withholds only the irreversible pool removal. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(media): serve replaced bytes from the pool everywhere Unknown survivor liveness deleted every planned audioFiles row. Those rows are keyed globally by audioId, and playback plus both export paths read the table directly, so losing one is as irreversible for them as removing the pool entry. Unknown liveness now preserves the rows too, leaving bounded garbage for a later pass that can prove exclusivity. The earlier pool-first change covered PPTX, video collection and thumbnail hydration but missed classroom ZIP export, which still serialized the stale compatibility row after a lagged same-id replacement, shipping media the classroom no longer renders. Auditing every direct reader of the media and audio tables surfaced the same gap in playback: speech regeneration also replaces bytes under a stable id and does not roll the pool back when the compatibility write fails, so the player kept serving superseded narration. It now resolves the pool first and falls back to stored rows for legacy and imported audio. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(audio): replace exclusively owned speech clips in place regenerateSpeechAudio accepted the action's audioId but always passed undefined as replaceAssetId, so regenerating an exclusively owned pool-backed clip allocated a new asset every time, rewrote the action and orphaned the previous pool entry and compatibility row until stage reclamation — contradicting the stable-id path generateAndStoreTTS already implements for media. Ownership is now established before synthesis, and the rule itself moved to the shared reference module so media retries, poster replacement and speech regeneration consume one implementation instead of restating it. An exclusively owned clip keeps its id and has its bytes replaced; a shared clip, a legacy id with no pool entry, or unprovable ownership still gets a fresh allocation so other holders keep their audio. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * chore(media): drop the import left behind by the ownership move Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(media): resolve audio pool-first and fence peer realms Stable-id TTS regeneration commits replaced narration to the pool before the audioFiles mirror write, so a failed mirror leaves the row stale. AudioPlayer already resolved pool-first, but audioObjectUrl, collectAudioFiles and the video timeline dependencies still read Dexie directly and would serve the superseded clip. All allocated-audio readers now share one resolver, with Dexie kept as the fallback for legacy and imported rows. The exclusivity proof modelled unflushed owners in the active realm only, so another tab duplicating the same asset during its save debounce could still be observed as a single owner and have its bytes replaced globally. A peer's pending state cannot be read across realms, so presence is probed instead: any realm holding the stage forces the fork path. A deferred asset-pool deletion no longer reloads the page. The database is still on disk and the guidance asks the user to close the other tab and retry, which the reload discarded. The decision moved into a shared helper so the rule is pinned rather than living inline in the component. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(media): fail closed when presence cannot be probed The presence helper documented that an unanswerable probe must count as a peer, but every unavailable path — no BroadcastChannel, a constructor that threw, a send that failed, and the window before the pool's asynchronous binding completes — returned false, so the ownership proof cleared a single local owner and replaced globally shared bytes in place. Probing now returns present, absent or unknown, and only a probe that was actually sent and went unanswered is absent; the ownership decision treats unknown exactly like present. The pool declares its binding intent synchronously so a probe issued during the load-time window waits for the bind instead of concluding that presence is unavailable, and releases that gate if the import fails. Coverage reaches the write boundary: with presence unknown, a production-shaped targeted retry forks to a fresh id instead of calling replace, and the original bytes stay intact for a peer's unflushed owner. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * chore(dsl): bump to 0.6.3 after the release dedupe took 0.6.2 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(i18n): add the deferred-clear guidance to fr-FR The locale landed on main after this branch added the key, so the alignment check flagged it as the one missing translation. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: 杨慎 <117187635+cosarah@users.noreply.github.com> |
||
|
zhifu gaoandLauraGPT
|
2b3f2eaa37 |
feat(asr): add local FunASR provider (#1044)
Signed-off-by: LauraGPT <199134975+LauraGPT@users.noreply.github.com> Co-authored-by: LauraGPT <199134975+LauraGPT@users.noreply.github.com> |
||
|
Nguyễn Lê TrườngandClaude Opus 5
|
4f20c5151a |
fix(i18n): translate ASR provider names in the generation toolbar (#1028)
The provider dropdown in the media popover rendered `p.name` straight from
`ASR_PROVIDERS`. That field is a registry label, not a UI string, and
several entries are written in Chinese:
browser-native → 浏览器原生 ASR (Web Speech API)
qwen-asr → Qwen ASR (阿里云百炼)
So a user on any non-Chinese locale saw Chinese in the toolbar, while the
settings dialog showed the same providers correctly — it resolves the name
through `t('settings.providerBrowserNative')` and friends.
The id→i18n-key tables move to `lib/audio/provider-display.ts` and both
surfaces use them, so they cannot drift apart again. The settings dialog
keeps its own handling of custom providers, which carry a user-supplied
name rather than an i18n key.
TTS is included in the shared module for symmetry (`doubao-tts` and
`qwen-tts` have the same problem), though the popover's TTS tab is
currently a toggle with no provider list.
Image and video groups in the same popover also render `p.name`, but those
registry labels are already English, so no user-visible string changes;
left alone to keep this focused.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
|
b4e3145c90 |
feat(roster): single-source the generated agent roster on the stage document (#994)
* feat(roster): single-source the generated agent roster on the stage document The generated agent roster used to live in two places at once: the stage document carried generatedAgentConfigs, while a per-stage IndexedDB mirror (db.generatedAgents) was the actual read authority — and the only home of each agent's voice binding. The split caused real losses: voices vanished on device switches, exports, and shares (the mirror never travels), the load path trusted a device-local table over the portable document, and roster edits persisted through a bespoke debounce with no beforeunload flush. Make the stage document the single source of truth: - Contract: add optional voiceConfig/voiceDesign to GeneratedAgentConfig in @openmaic/dsl (VoiceDesign moves into the contract; the app re-exports it). Additive optional fields — not a breaking serialized-shape change, so DSL_VERSION stays put per the version policy. - Writes: remove every mirror writer (registry saveGeneratedAgents, the stage store's debouncedSaveAgents bypass, classroom-load hydration, import direct writes). setStageAgents now only marks the stage dirty for the shared persistence scheduler plus synchronous in-memory registry/selection mirrors; generation and import embed the roster on the stage. - Reads: classroom load hydrates the registry from the loaded stage's generatedAgentConfigs; export reads the stage roster directly. - Lazy migration: the mirror is retained read-only. On load, a roster (or voice fields) missing from the document is backfilled from the mirror, committed onto the in-memory stage, and persisted by the next flush — idempotent, and gated on the current stage id after every await so a classroom switch cannot leak a stale roster into the registry. - Deletion: deleteStageData discards the deleted stage's pending persistence work so a queued flush cannot resurrect the document. - Round trip: classroom ZIP manifests now carry the voice fields, so export/import preserves agent voices. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(roster): harden deletion, migration, selection, and ingress paths Review hardening on top of the roster single-source change: - Deletion vs in-flight persistence: deleting a stage now registers a session-scoped tombstone (lib/utils/deleted-stages.ts) before the cascade starts. Every persistence landing point checks it — the scheduled flush and the departing-stage retry drop their snapshots, and the aggregate/incremental saves re-check under the document lock, including the incremental path's full-save fallback that would otherwise rebuild a deleted document from the in-memory snapshot. A deletion that fails before the document is removed lifts the tombstone so the still-existing stage keeps persisting. - Lazy migration termination: a fruitless legacy-mirror probe (nothing to merge — e.g. server-generated rosters, which are voiceless by design) is remembered per stage for the session instead of re-querying the mirror on every load forever. Successful merges are not memoized so a failed flush is retried on the next load. Docstrings now state the real lifecycle. - Selection provenance: setStageAgents no longer overwrites a user-set agent selection with the full roster. Stage-derived selections still track the roster; a user-set auto selection is only narrowed to surviving agents (newcomers are not auto-selected, the user-set flag is untouched), and a user-set preset selection is left alone — matching restoreAgentSelection. - Mirror hygiene: deleteStageData clears the deleted stage's rows from the legacy roster mirror (best-effort), and the mirror table's docs now describe its actual access pattern (migration reads + deletion hygiene). - Registry persistence: the agent registry's persist snapshot partializes generated agents out of localStorage, making the stage document the roster's only durable home; the rehydration merge filter remains as defense in depth. - Ingress validation: imported manifest voice fields are structurally validated (malformed bindings dropped per field, the agent survives), and applyGeneratedAgentsToRegistry validates providerId against the known TTS provider registry instead of casting, treating unknown providers as "no bound voice". - Contract: drop the speculative AgentVoiceConfig.modelId — an audit of every roster producer, current and historical, found none that ever emitted it. Contract fields are added once a producer exists, not before. New tests cover the deletion tombstone at both the store and storage layers (including the missing-destination full-save pin and the lock-race re-check), the fruitless-probe memo, the selection-provenance matrix, persistence exclusion of generated agents, and voice-field sanitization on import. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(roster): close tombstone lifecycle, probe-memo, and selection-intent gaps Second hardening pass over the deletion/migration/selection fixes: - Tombstone lifecycle: a deleted classroom id can legitimately come back (deletion is client-side only). Explicit (re)creation points now lift the tombstone — server-copy restore in applyClassroomStageAndScenes and backup restore in importDatabase — so edits after a same-session restore persist again, while in-flight flushes stay fenced. - Failed deletion now restores the dirt it discarded: deleteStageData snapshots the pending map plus the in-flight round before tombstoning, and re-marks it when the delete fails with the document still present. - Legacy-mirror probe memo now distinguishes a FAILED read (null, retried on the next load) from a confirmed-empty mirror (memoized), so a transient IndexedDB error cannot suppress migration for the whole session. - Incremental save tail (currentScene KV row + chat sessions) is tombstone- fenced like the aggregate path, and both paths re-check the tombstone immediately before every write, with comments scoped honestly for the lock-free (no Web Locks) best-effort LWW fallback. - Roster-edit selection mirror: an empty intersection now falls back to the full roster and clears the user-set flag (matching restoreAgentSelection's length > 0 gate), and a user-set selection that equals the pre-edit full roster keeps tracking the roster wholesale so newcomers are not excluded forever by the AgentBar auto-toggle snapshot. - TTS/ASR provider lookups use Object.hasOwn instead of `in`/bare indexing, so prototype-chain keys ('toString', 'constructor', ...) no longer pass the provider whitelist or resolve to Object.prototype members. - Documented the per-tab limit of the tombstone set. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(roster): replace the boolean deletion tombstone with a deletion-generation model A boolean tombstone cannot express the deletion lifecycle once a deleted id legitimately comes back (server-copy restore on revisit, backup import): after the restore lifts the flag, a pre-delete flush still in flight is indistinguishable from a post-restore edit and can overwrite the restored document with pre-delete content. Two rounds of point fixes kept leaking variants of this race, so this restructures the fence instead. deleted-stages.ts now keeps a per-stage monotonic deletion epoch plus a deleted flag. markStageDeleted bumps the epoch; unmarkStageDeleted clears only the flag and never rewinds the epoch. Every persistence path captures the epoch at the moment it captures the data it will write (flush-round snapshot, departing-stage snapshot, aggregate save entry) and re-checks "captured epoch is current AND not deleted" immediately before each individual write. Invariants: - A write captured before a deletion can never land after it, even when a same-id restore has lifted the deleted flag (the captured epoch is permanently stale). - A write capturing data after a restore observes the current epoch and persists normally. - A failed deletion lifts only the flag; the restored pending changes are re-queued as descriptors, so their flush re-captures current store state under the current epoch and is neither dropped nor a stale replay. Also fixed in the same restructuring: - Incremental tail writes (currentScene, chats) each re-check independently; a delete landing while the first tail write is awaiting now fences the chat write instead of riding the earlier check. - Deleting a classroom evicts it from the warm in-memory store, and loadFromStorage treats a warm-but-deleted stage as not loaded (discarding the ghost), so navigating Back to a deleted classroom reaches the server-restore path instead of rendering an editable ghost whose every edit is silently dropped. - importDatabase records each document's pre-import deletion state and reinstates it when a failed import rolls the document back, so an outstanding flush cannot recreate the rolled-back document. - Docstrings on the failed-deletion restore path now state that only dirt captured before the deletion is restored; edits refused during the deletion window are not part of the snapshot. New tests cover the epoch invariants at the storage layer (pre-delete rounds dropped across restores on the incremental, aggregate, and tail paths; post-restore captures landing; strictly increasing epochs), the scheduler layer (departing-retry delete+restore straddle; restored dirt flushing under the current epoch), the warm-ghost restore driven through the real loadFromStorage, and the import lift/rollback pair. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(deletion): report fenced drops, gate ghost eviction on cascade settlement Edge hardening around the deletion-epoch fence; the epoch model itself is unchanged. - An epoch-stale drop is now reported, not disguised: saveStageData / saveStageDataIncremental return a distinct 'stale-dropped' status (entry check, every in-mutation re-check, and both tail-write re-checks) instead of a shape identical to success. saveToStorage reacts by skipping all success bookkeeping - no chatSnapshot rebind to a snapshot that never landed, no pending-dirt clearing - and returns false, keeping its "true means verified write" contract honest. persistDirtySnapshot keeps its existing (correct) semantics explicitly: a drop has nothing to retry. - Warm-ghost eviction is settlement-gated: the deletion cascade records an in-flight bit (begun by deleteStageData, settled in its finally), and the deleted-warm branch of loadFromStorage keeps the warm state while the cascade is unsettled - a delete can still fail before removing the document, in which case the warm state plus the restored pending dirt is the only copy of the pre-delete edits. Once settled, the ghost discard and server-restore path behave as before. - Read-side landing re-check: loadFromStorage re-checks isStageDeleted after hydration, immediately before set(), mirroring the write-side "re-check immediately before landing" discipline - a lock-free mid-cascade read can no longer re-materialize a ghost classroom. - clearStoreForDeletedStage re-checks isStageDeleted at eviction time, so a same-id restore completing inside the cascade-tail window is never wiped. - capturedEpoch is now a required parameter on both storage entry points: the default (call-time capture) silently reopened the capture-point/ validation-point split for future callers; the type system now enforces the pairing. All production callers already passed it explicitly. New tests cover the dropped-status contract, the cascade in-flight lifecycle (including both failure shapes), keep-warm during an unsettled delete with end-to-end pending restore after a failed delete, settlement- then-eviction on success, the eviction guard against a tail-window restore, and the read-side hydration re-check. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(deletion): settle the deleted-warm load by awaiting the cascade outcome Hardening around the deletion settlement gate introduced previously; the epoch model is unchanged. - The deleted-warm branch of loadFromStorage no longer completes the load while the deletion cascade is undecided (which exposed a classroom whose edits were refused without being restorable, and left a successful deletion's eviction with no follow-up reload). It now parks on a per-stage settlement promise (stageDeletionSettled) and branches on the outcome: a failed delete keeps the warm, restored classroom; a successful delete falls through to a full cold load so the server-restore path recovers the route; a navigation that moved on during the park is stopped by the load token guard. The eviction (clearStoreForDeletedStage) deliberately stops claiming a new load token so the parked load stays current - ghost re-materialization is fenced by the read-side deleted re-checks instead. No deadlock: a parked load holds no document lock, and the cascade never waits on a load. - 'stale-dropped' now propagates through the debounced flush path: persistDirtySnapshot returns the sentinel instead of an empty failure set, and startFlushRound skips the chatSnapshot rebind on it (mirroring saveToStorage) - a falsely rebound baseline would let later chat saves no-op-skip chats that never landed and corrupt cross-tab conflict detection. Pending-map clearing keeps its empty-failure semantics (restored descriptors mint fresh revisions). - The cascade in-flight state is a counter (begin++/settle--) instead of a boolean, and deleteStageData is single-flight per stage (a concurrent second call joins the first cascade), so overlapping deletions can never expose an undecided cascade as settled. - The StaleDroppedSave docstring no longer overpromises: a tail saveCurrentScene can land after the cascade's clearCurrentScene; the orphaned cursor row is ignored by the load path rather than removed by the cascade. Net +8 tests: overlapping-cascade counting, settlement-promise resolution (success and failure), concurrent double-delete single-flight join, the parked mid-cascade Back in both outcomes plus the navigation-away guard, and the fenced flush round keeping an honest chat baseline end-to-end. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(deletion): re-verify store identity on keep-warm resume, re-run joined deletes after a mid-cascade restore Three hardening refinements on the deletion/persistence fencing: - loadFromStorage's parked keep-warm branch now mirrors the success path's store-identity guard: if a tokenless writer (e.g. a database import) replaced the stage during the park, a failed deletion falls through to the cold load instead of reporting another classroom as the kept warm state. The ghost-discard block is correspondingly gated on the deleted flag so the fall-through does not mislabel a surviving document as a ghost. - deleteStageData: a joined call carries its own deletion intent. When the first cascade fulfills but a same-id restore lifted the deleted flag in the meantime, the joined caller now runs one fresh cascade against the restored document instead of reporting the pre-restore outcome. Exactly one re-check per call (no recursion); a rejected first cascade still propagates unchanged to every joined caller. - Narrowed the failed-delete restore-ownership comments: the restore covers only the pending map plus an in-flight flush round's dirt; a departing-stage snapshot is outside that capture and is fenced and dropped by design. Comment-only, no behavior change. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(roster): gate deleted-warm handling on stage identity, keep an explicit empty roster authoritative Final-audit fixes, three items: - loadFromStorage keyed its deleted-warm settlement handling (park / keep-warm / ghost discard) behind the scenes.length > 0 shortcut, so a zero-scene warm ghost of a deleted classroom skipped it entirely: the cold load found no document, the ghost stayed in the store, and the classroom loader's server-fallback gate (!getCurrentStage()) never ran — the silently-uneditable trap this branch exists to close, in its scene-less form. Deletion handling now keys on stage identity alone, before any scene-count shortcut; the non-deleted warm skip still requires scenes. - rosterNeedsLegacyFallback collapsed "roster field absent" (document predates roster persistence) with "explicitly persisted empty roster". A future writer persisting [] on a device whose read-only mirror still held stale rows would have its emptied roster resurrected on every load. The full-lift branch now triggers only when the field is absent (undefined); an explicit [] is authoritative and never probes the mirror. No current writer produces [], but the read side no longer depends on that invariant. - The GeneratedAgentConfig docstring claimed the voice fields are not a breaking change; that only holds for this codebase's tolerant structural validators. The generated stage.schema.json sets additionalProperties: false, so consumers pinning an older published schema artifact reject documents carrying the new fields. The docstring now states both sides honestly. Tests: a zero-scene deleted warm ghost is restored through the real loadFromStorage (fails pre-fix), an explicit empty roster does not resurrect stale mirror rows (fails pre-fix), and the rosterNeedsLegacyFallback unit matrix covers undefined vs []. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(deletion): re-queue aggregate persistence when a failed deletion is recovered Direct aggregate saves (saveToStorage: generation completion, server-restore hydration) run outside the pending map and the flush round, so the deletion path's pre-delete dirt snapshot cannot describe them. When a deletion bumped the epoch while such a save was in flight, the save was correctly fenced ('stale-dropped') — but if the deletion then failed before removing the document, the recovery only re-queued the snapshotted scheduler descriptors. The fenced aggregate-only content stayed memory-only with no retry anywhere, and a later reload lost it. The failed-deletion restore now merges a full re-mark of the aggregate (structure / stage / outline / currentScene / chats plus a descriptor for every current scene) into the recovery set, still guarded on the store holding the stage. Because the flush recaptures the CURRENT store state under the CURRENT epoch, this necessarily carries whatever the fenced aggregate save held — and, for the same reason, edits refused by the scheduler during the deletion window, which previously stayed memory-only as well; the scope docstrings now say so instead of declaring them excluded. New integration coverage drives the real store scheduler and the real storage layer end to end: a genuinely fenced in-flight saveToStorage (generation completion) plus a pre-removal deletion failure ends with the completion flag durable on the next flush; a deletion-window scene edit survives the same way; and a successful deletion re-marks nothing (control). Both recovery tests fail against the previous restore logic. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: 杨慎 <117187635+cosarah@users.noreply.github.com> |
||
|
|
356d454ea4 |
feat(audio): persist audio duration when storing TTS (#861) (#862)
* feat(audio): persist audio duration when storing TTS (#861) Video export (#854) maps each narration clip onto a timeline segment, which needs the clip's duration. The IndexedDB AudioFileRecord already declares a `duration` field but the TTS store path never populated it, so export/import (which round-trip duration via the media manifest) had nothing to carry. Measure duration once when the client persists TTS audio instead of decoding at render time — that keeps the export manifest inspectable in tests without FFmpeg/Chrome (#854 criteria). Scope: client persistence path only. Duration is an asset property, so it lives on AudioFileRecord (matching mediaIndex.duration in export / import), not on the SpeechAction DSL contract. The server generation path has no AudioFileRecord and is left to a later slice. - Add dependency-free `measureAudioDuration` (WAV RIFF walk + MP3 Xing/CBR estimate); pure byte parsing, no DOM, returns null for unparseable input so callers degrade gracefully. - Client path (`generateAndStoreTTS`): populate AudioFileRecord.duration. - Unit tests for WAV/MP3 parsing, format dispatch, and degradation. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(audio): sniff content before trusting format hint; add VBRI + ID3v1 handling Address PR #862 review: - [P2] measureAudioDuration now sniffs magic bytes first and only falls back to the reported `format` hint when the content is unrecognizable. getAudioResponseFormat defaults to `mp3` on a missing Content-Type, so a WAV clip could reach the MP3 parser and false-sync into a wrong duration; content now always wins over a mismatched hint. - [P3] Implement the VBRI frame-count path the comment already claimed. - [P3] Exclude a trailing ID3v1 tag from the CBR byte-count estimate. - Add regression tests: mismatched hint (both directions), Xing/Info frame count, VBRI frame count, trailing ID3v1. 17 passing (was 11). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: wyuc <wang-yc24@mails.tsinghua.edu.cn> |
||
|
Yanpeng Wang
|
d54e62f5c4 |
feat(token-plan): one-click token-plan setup + deployment usage dashboard (#784)
* feat(usage): add usage normalization, pricing, model-fetch, balance, storage Foundational layer for token-plan usage tracking (cc-switch-modeled): - lib/usage/normalize.ts: AI SDK v6 usage → four-class token shape - lib/usage/pricing.ts + defaults: per-class USD pricing table - lib/server/model-fetch.ts: /models candidate-URL multi-fallback (ported) - lib/usage/balance-providers.ts: built-in balance queries + detection - lib/server/usage-storage.ts: fire-and-forget jsonl logging to data/usage All pure/storage logic covered by vitest (32 tests). * feat(usage): capture token usage at the callLLM/streamLLM chokepoint - callLLM records result.usage before returning - streamLLM wraps onFinish to record totalUsage on stream completion, preserving any caller-supplied onFinish - provider/model derived from the model instance (no route changes) - fs-backed storage imported dynamically; fire-and-forget, never throws - include_usage is already sent by @ai-sdk/openai, so streaming is covered * feat(usage): add probe-models, balance, and usage API routes - POST /api/provider/probe-models: discover chat models via /models with candidate fallback; SSRF-guarded; filters non-chat ids; 401/404 typed - POST /api/provider/balance: built-in balance detection + billing fallback - GET /api/usage: aggregate jsonl by model/day/source with costIncomplete flag Verified e2e against the live MAIC gateway: 16 chat models (6 filtered), balance detected, and a real callLLM writes a costed usage row. * feat(settings): add token-plan preset picker to provider dialog - lib/config/token-plan-presets.ts: data-driven vendor presets (Huawei/MiniMax/ Xiaomi token plans, OpenRouter/SiliconFlow gateways, DeepSeek/GLM/Qwen/Hunyuan/ Doubao direct) with baseURL, protocol, optional modelsUrl, category - add-provider-dialog: '选择厂商/自定义' tabs; picking a preset auto-fills baseURL+protocol+modelsUrl, custom tab unchanged - ProviderSettings.modelsUrl carries the optional /models override - i18n keys added across all 8 locales Verified in-browser: preset picker renders grouped by category. * feat(settings): add Fetch Models button and balance bar to provider panel - 拉取模型: probes /models, merges discovered ids into the model list (dedupe, keeps manual additions), with success/no-endpoint/auth messages - 查询余额: queries /api/provider/balance, renders a balance bar or a 'check console' hint when unsupported - index.tsx: handleModelsFetched merges probe results into provider config - i18n keys across all 8 locales; removed a now-unused eslint-disable Verified in-browser against MAIC gateway: 16 models fetched, balance shown. * feat(settings): add usage dashboard to System Settings - usage-dashboard.tsx: echarts dual-axis daily trend (tokens + cost), totals cards, by-model table, refresh; reads GET /api/usage - mounted at the top of GeneralSettings (系统设置) - honest disclaimer + costIncomplete marker when a model lacks pricing - i18n keys across all 8 locales Verified in-browser: shows 1 request / 31 tokens / $0.0003 from a prior call. * feat(token-plan): multi-modal one-click setup in System Settings - token-plan-presets.ts: presets now declare per-modality targets (llm/image/video/tts/webSearch); MiniMax is the full-set template, others LLM-only — extend by adding entries (at-our-best adaptation) - apply-token-plan.ts: fills one key into every declared modality via injected store setters, isolating per-modality failures (TDD, 4 tests) - token-plan-settings.tsx: new sidebar page — pick plan, enter key, one-click apply lights up adapted modalities (+LLM model probe), shows 'not adapted yet' for the rest, balance bar reused - reverted add-provider dialog to plain custom form (preset picker moved here) - i18n across all 8 locales Verified in-browser: Token Plan page renders, MiniMax shows all 5 modalities, apply/balance UI wired. 859 tests pass, build clean. * feat(token-plan): add custom token plan entry to Token Plan page - Custom card at the bottom of the provider list: pick it to manually enter name + protocol + baseURL (LLM-only), then key + one-click apply via the same applyTokenPlan flow (mirrors cc-switch's custom provider) - effectivePreset unifies preset vs custom for apply/balance/probe - i18n keys (customGroup/customName/customHint) across all 8 locales Verified in-browser: custom card expands the manual form; preset flow intact. * feat(token-plan): reflect persisted config on the Token Plan page Other settings panels read the store directly, so they survive section switches; the Token Plan page only wrote the store, so it looked blank on return. Now it also reads providersConfig: - selecting a preset prefills its saved API key - configured presets show a '已配置/Configured' badge State persistence was never broken (apply writes zustand→localStorage); this fixes the missing read-back so the page reflects it. * fix(settings): isLLMProviderConfigured crashed on providers without models Root cause of 'token plan config disappears': applyTokenPlan writes a new LLM provider with no models yet (probe fills them later). isLLMProviderConfigured did config.models.length unguarded → threw inside setProviderConfig's resolver → the whole set() aborted → key/baseUrl/type never persisted. - guard models in isLLMProviderConfigured (shared validator; affects any provider written without a models array) - seed models:[] in applyTokenPlan's LLM write for a valid initial shape - regression tests: validator no longer throws; apply+probe write keeps apiKey Verified in-browser: DeepSeek persists key and shows 已配置 after section switch. * feat(token-plan): add remove/teardown for a configured token plan Apply had no inverse. Add removeTokenPlan: clears the API key and disables (enabled:false) every modality the plan declared; a custom LLM provider is deleted entirely (removeProvider), built-ins keep the cleared shape. - removeTokenPlan in apply-token-plan.ts (mirrors applyTokenPlan, isolated per-modality, injected setters; 3 tests) - trash button on configured preset cards in the Token Plan page; resets page state if the removed plan was selected - i18n 'remove' across all 8 locales Verified in-browser: removing DeepSeek empties its key and drops the 已配置 badge. * feat(usage): track multimodal usage, drop cost/pricing Reframe usage stats as pure usage (no cost), per user decision: - usage-storage: drop all cost fields; add kind (llm/image/video/tts/asr) + quantity + unit; LLM keeps token counts, others store quantity - delete lib/usage/pricing.ts + pricing-defaults.json + its test - instrument image/video/tts at the server-only API routes (not in the provider dispatch — those files are in the client graph and importing fs-backed usage-storage broke the client bundle) - /api/usage: aggregate by model/day/modality, no cost - dashboard: per-modality usage (tokens/images/seconds/chars), token-only daily trend, no '$'; i18n cost keys replaced with modality/unit keys - backward compatible: legacy rows (no kind, stray cost fields) read as llm 912 tests pass, build clean. * feat(usage): per-modality dashboard layout + softer dark-mode chart - group usage into per-modality sections (LLM/image/video/tts/asr), each table's usage column uses one consistent unit (token/image/sec/char) — no more mixed units in a single column - summary chips per modality with their own unit - trend chart now plots daily REQUESTS (unit-agnostic, works for any modality) instead of LLM-only tokens - theme-aware chart: faint thin line + soft gradient area, muted axis/grid colors via useTheme — fixes the harsh solid stroke in dark mode Verified in-browser (dark): TTS shows '字符', LLM shows 'Token', separate sections; chart no longer has a hard line. * refactor(usage): dedupe model/fetch/usage helpers, parallel balance probe Review cleanups on the token-plan/usage branch: - extract modelInfoFromId() (shared vision heuristic + ModelInfo shape) - extract fetchWithTimeout() shared by model-fetch and balance-providers - extract recordGenerationUsage() to dedupe the image/tts/video routes - queryBalance: fetch billing subscription + usage in parallel - parseOneApiBilling: report quota without remaining when usage endpoint is unavailable, instead of implying zero spend (full balance) - split the two jammed imports in settings/index.tsx * feat(token-plan): drop custom token-plan support Custom token plans (manual baseURL/protocol entry) added complexity for little gain — a one-off provider is better configured directly on the Providers page. Token Plan is now preset-only: - remove custom mode, manual fields, and the custom card from the UI - collapse effectivePreset back to the selected preset - drop the now-dead removeProvider action + custom-id branch in removeTokenPlan - remove orphaned customGroup/customName/customHint i18n keys (8 locales) * feat(token-plan): add Volcengine/Tencent/Bailian plans, drop balance feature Add three vendor token-plan presets (all map to existing built-in LLM providers, so it's data-only — no new adapters): - 火山方舟 Volcengine Ark → doubao, OpenAI /api/v3 - 腾讯 TokenHub Token Plan → tencent-hunyuan, OpenAI /plan/v3 (the plan-specific base; /v1 is the pay-as-you-go gateway) - 阿里百炼 Token Plan → qwen, cross-model plan (Qwen + DeepSeek/Kimi/ GLM/MiniMax) on one key; model list is probed/entered Remove the balance/quota feature entirely — we now track usage, not cost, and every vendor's balance query needs its own cloud AK/SK + signature (Volcengine SigV4 / Tencent TC3 / Aliyun BSS), which the Bearer-key billing-endpoint probe never supported anyway: - delete lib/usage/balance-providers.ts, /api/provider/balance, its test - strip the Check Balance button + balance bar from the token-plan page and the provider config panel - remove the 4 balance i18n keys across all 8 locales - restore an eslint-disable the branch had dropped in provider-config-panel * feat(token-plan): use cloud-brand logos for vendor token plans The three vendor plans are cloud offerings, not single-model products, so icon them with the cloud brand rather than a model logo: - 火山方舟 → volcengine.svg (was doubao.svg) - 腾讯 TokenHub → tencentcloud.svg (was hunyuan.svg) - 阿里百炼 → alibabacloud.svg (was bailian.svg) Logos are the colored brand variants from lobehub/lobe-icons, matching the existing colored-logo style (plain <img>, no dark:invert needed). * feat(token-plan): keep only MiniMax and Volcengine presets Trim the token-plan list to the two we want to ship: MiniMax (full-set template) and 火山方舟 Volcengine Ark. Drop the Tencent/Bailian plans and the OpenRouter/SiliconFlow/DeepSeek/GLM/Qwen entries. - remove the now-unused tencentcloud.svg / alibabacloud.svg logos (volcengine.svg stays; the other logos are still used by the provider registry) - retarget the LLM-only apply test from the deleted deepseek preset to volcengine-ark * feat(token-plan): restore aggregator/third-party presets Previous commit over-trimmed: the intent was to drop only the Tencent and Bailian token plans, not the OpenRouter/SiliconFlow/DeepSeek/GLM/Qwen entries. Bring those back; keep only Tencent/Bailian removed. - token_plan: MiniMax, 火山方舟 Volcengine Ark - aggregator: OpenRouter, SiliconFlow - third_party: DeepSeek, GLM, Qwen Revert the apply test back to the deepseek fixture (restored). tencentcloud.svg / alibabacloud.svg stay deleted (their plans are gone). * fix(token-plan): point Volcengine plan at the Coding Plan endpoint The plan's ark--prefixed API keys authenticate only against /api/coding/v3, not the general /api/v3 endpoint — the latter rejects them with "The API key format is incorrect", so model probing returned nothing. Switch the base URL to https://ark.cn-beijing.volces.com/api/coding/v3. * fix(token-plan): Volcengine is an Agent Plan (Anthropic /api/plan) Per the Ark Agent Plan docs, the ark--prefixed keys authenticate ONLY against the dedicated Anthropic-compatible base https://ark.cn-beijing. volces.com/api/plan ("其他 Base URL 无法在 Agent Plan 中使用"). The general /api/v3 and the Coding Plan /api/coding endpoints both reject the key as "API key format is incorrect", which is why model probing kept returning 0. - baseUrl → https://ark.cn-beijing.volces.com/api/plan/v1 (the /v1 lets the Anthropic SDK land on /api/plan/v1/messages) - apiFormat → anthropic - rename to 火山方舟 Agent Plan Probe still targets /api/plan/v1/models (the path exists); if the Anthropic gateway doesn't return an OpenAI-shaped list, users fall back to typing a model id like ark-code-latest. * fix(token-plan): Volcengine Agent Plan = OpenAI /api/plan/v3 + ark-code-latest Settled after probing the real key and reading cc-switch's approach: - The ark- plan key works on the OpenAI-compatible /api/plan/v3 endpoint (chat/completions returns 200); switch apiFormat back to openai. - The plan exposes NO /models list (every /api/plan/*/models is 404), which is why probing kept returning 0. cc-switch handles this by hardcoding a single ark-code-latest (an auto-routing alias valid on any tier) and does NOT use AK/SK for model listing — so we do the same. - Seed defaultModels: ['ark-code-latest'] only; users add specific ids by hand. Supporting machinery (kept, general-purpose): - applyTokenPlan seeds models from defaultModels instead of wiping to [] - handleApply uses defaultModels and skips the doomed probe when present - drop stray .playwright-mcp/ debug artifacts and gitignore them * style: fix prettier formatting in usage files CI runs prettier on the whole repo (prettier . --check); these four files predate this branch's formatting pass and tripped the check. * feat(token-plan): verify Volcengine Agent Plan's published model set The Agent Plan publishes a fixed model set but exposes no /models endpoint, so carry the documented models as CANDIDATES and verify each on apply: - add verifyModels flag to TokenPlanModalityTarget - new /api/provider/probe-chat-models route: sends a minimal chat request per candidate (OpenAI /chat/completions or Anthropic /messages) in parallel, returns the subset that succeeds; SSRF-guarded, auth-failure short-circuits - handleApply gains a verify branch (before the fixed-defaultModels fast path), falling back to the seeded list if verification fails - Volcengine preset now carries the 12 published Agent Plan text models (doubao-seed-2.0-*/deepseek-v4-*/minimax-m*/glm-5.2/kimi-k2.*) as candidates This auto-prunes retired (docs flag deepseek-v3.2/glm-5.1 as 即将下线) and tier-gated models without code changes. Verified all 12 resolve against a real plan key. * feat(token-plan): wire Volcengine Agent Plan image + video modalities Make the Ark seedream/seedance adapters path-configurable and light up the image/video modalities on the Volcengine plan: - seedream/seedance adapters: resolveArkRoot() uses baseUrl verbatim when it already carries an /api/... path (token plan's /api/plan/v3), else appends the standard /api/v3 — no regression for the pay-as-you-go default host. - applyTokenPlan: image/video branches inject a modality's defaultModels as customModels and set them as the active provider+model, so generation works out of the box. New optional setImageProvider/ModelId + setVideoProvider/ ModelId actions (UI passes the store setters; tests omit them). - Volcengine preset declares image (doubao-seedream-5.0-lite, verified 200 on /api/plan/v3/images/generations) and video (doubao-seedance-2.0/1.5-pro — Medium+ tiers only; lower tiers reject at call time, no code change needed to upgrade). Applying the plan overwrites the shared seedream/seedance slot with the plan config (same overwrite model as LLM); switching back to pay-as-you-go is a manual edit or plan removal. Verified image end-to-end with a real plan key. * feat(token-plan): verify image/video models on apply, disable unsupported tiers The Volcengine plan lit up video optimistically, but lower tiers (Small) don't include video — so using it 404'd with UnsupportedModel. Probe media models on apply and only keep what the tier actually supports: - generalize /api/provider/probe-chat-models with a `kind` (chat|image|video): image hits /images/generations, video hits /contents/generations/tasks with empty content. The model-support check (404 UnsupportedModel) runs before any billable work, so probing never starts a real image/video job; for media, "supported" = any non-404 response. - handleApply: after lighting up image/video, probe each verifyModels modality; prune to the verified model set + re-select a working model, or disable the modality entirely if none pass (no false "available"). - Volcengine preset: image/video targets gain verifyModels: true. - add settings.tokenPlan.tierUnsupported across 8 locales. Verified with a real Small-tier key: image (seedream-5.0-lite) passes and is kept; video (seedance-2.0/1.5-pro) 404s and is disabled. * feat(web-search): add Doubao (豆包搜索) provider Doubao Search (Custom 版) over its REST endpoint POST open.feedcoopapi.com/search_api/web_search with Bearer auth — the same endpoint the askecho-search-infinity MCP server wraps, so the Volcengine Agent Plan key authenticates directly. Mirrors the MiniMax adapter: maps Result.WebResults to WebSearchSource (prefers Summary, the query-relevant excerpt, over Snippet for LLM use) and surfaces errors from ResponseMetadata.Error. - register 'doubao' in WebSearchProviderId + WEB_SEARCH_PROVIDERS - searchWithDoubao adapter, searchWeb dispatch, store default config - SSRF allowlist entry for the search host * feat(audio): support Agent Plan single-key auth for Doubao TTS generateDoubaoTTS now picks auth + endpoint from the key shape, since Volcengine exposes Seed-TTS as two products with separate credentials (verified: a plan key 401s on the normal endpoint, and the plan endpoint rejects appId-style auth): - single key (no colon) -> X-Api-Key, for the Agent Plan /plan endpoint - appId:accessKey -> X-Api-App-Id + X-Api-Access-Key (unchanged) A malformed pair (empty half) fails clearly instead of sending an empty header. Reuses the existing NDJSON/base64-mp3 parsing and voice list. * fix(media): map MiniMax video 720p to its real 768P tier normalizeVideoOptions defaults minimax-video to '720p' (the first supported resolution), but Hailuo 2.3 only accepts 768P/1080P and rejects 720P with '2013 ... does not support resolution 720P'. MiniMax's mid tier is 768P, not 720P (the adapter already falls back to 768P, as does the connectivity test), so map the shared enum's '720p' to 768P. Regression tests lock the mapping. * feat(token-plan): add web search + TTS to Volcengine Agent Plan, widen image tiers Extend the volcengine-ark preset now that the adapters exist: - webSearch -> doubao (own host open.feedcoopapi.com, not the ark endpoint) - tts -> doubao-tts on the /api/plan/tts endpoint (single-key auth) - image defaultModels widened to a best-first Seedream 5.0/4.5/4.0 list so a higher tier keeps the strongest model while verifyModels prunes the rest; video keeps the 2.0 + 1.5-pro candidates Comments record the verified host/auth quirks of each modality. * feat(token-plan): show result panel only after probing, with two clear states Addresses review feedback that a green check implied generation works when it only meant 'configured'. The panel now renders after probing finishes (gated on results && !applying) so it reflects the final set, and uses two states: green when the modality is configured/usable, muted when a live probe proved it unavailable (e.g. video on a tier without it). * feat(token-plan): scope presets to true multi-modal token plans Drop the single-modality LLM presets (OpenRouter, SiliconFlow, DeepSeek, GLM, Qwen) from Token Plan. A token plan's defining trait is one key spanning many modalities; those entries are ordinary LLM API providers already covered by the add-provider flow, and listing them here muddied the 'one key, every modality' promise. Only MiniMax and the Volcengine Ark Agent Plan remain. The UI already hides categories with no entries. apply-token-plan's LLM-only test now uses a local fixture instead of the removed deepseek preset. * feat(token-plan): progressive reveal of probe results on apply The result panel previously rendered all at once after probing finished, reading as dead air during the model probe. Now rows appear immediately on Apply: modalities with a live probe in flight show a spinner ('pending') and resolve to lit/failed independently as each probe returns, while non-probe modalities show lit right away. Probes run in parallel (Promise.all) instead of sequentially. A row only turns green once its own probe confirms, so this reveals structure + live progress without a premature green — complementing the earlier 'render only after probing' intent rather than reverting it. Per review feedback from @wyuc on #784. * fix(token-plan): enrich seeded models with built-in thinking capability Token Plan built ModelInfo objects from probed ids via modelInfoFromId(), filling only streaming/tools/vision — so a model that supports configurable thinking lost capabilities.thinking and InlineThinkingControl was hidden. modelInfoFromId now takes an optional providerId and overlays the catalog thinking capability for that (provider, model) pair; applyTokenPlan does the same for its synchronously-seeded list. Added the Ark Agent Plan's dotted aliases to the metadata table: - native Doubao Seed 2.0 family (doubao-seed-2.0-pro/code/lite/mini) - cross-vendor models the plan serves through its OpenAI-compatible endpoint (deepseek-v4-pro/flash, glm-5.2, kimi-k2.7-code/k2.6, minimax-m3/m2.7, ark-code-latest) All verified against a live plan key: each accepts the gateway's unified reasoning_effort field (low/medium/high) and actually reasons. They share the doubao effort adapter, which disables via 'minimal' (not 'none') — matching what the plan endpoint accepts (it rejects reasoning_effort:'none'). Addresses review point #1 from @wyuc on #784. * Improve token plan capability setup UI * fix token plan setup flow * chore: prettier format tts-providers.ts |
||
|
|
6fb87e7d0b |
fix(tts): respect string context when splitting the Doubao stream (#677)
Doubao streams a run of concatenated JSON objects with no delimiter. The
splitter counted `{`/`}` braces without tracking whether it was inside a
string literal, so a brace in a string value — e.g. an error
`{"message":"bad {input}"}` — mis-aligned the object boundaries, dropping
the chunk (lost audio, or a swallowed error).
Extract a string-aware `splitConcatenatedJsonObjects` helper (mirrors the
scanner in json-repair.ts: tracks inString + escapes) and use it. Behaviour
is otherwise unchanged: parse failures are skipped, code 20000000 ends the
stream, rate-limit/error codes still throw.
Closes #676
Co-authored-by: ly-wang19 <ly-wang19@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: wyuc <wang-yc24@mails.tsinghua.edu.cn>
|
||
|
wyuc
|
1d1ce80e04 |
feat(maic-agent): editor-agent line (v0) — Pro-mode "Edit with AI" (#777)
Promote the editor-agent line to main: read-then-act editor agent (read_scene_content, regenerate_scene, regenerate_scene_actions, edit_interactive_html), a routable maic-agent model stage, the agent sidebar UI, interactive-scene runtime-error capture, and the timeline/script editor. Cross-reviewed and smoke-verified; see #777. |
||
|
wyuc
|
e83db6a531 |
fix(agents): persist AgentBar voice picks and mode/selection across reloads (#723)
* feat(audio): let resolveAgentVoice honor per-agent voice overrides * feat(settings): persist per-agent voice overrides * fix(agents): persist AgentBar voice picks via settings overrides * fix(classroom): keep user's agent mode and selection across loads * fix(agents): only carry user-set agent selection across classrooms Review follow-ups: stage-derived defaults written by a classroom load were re-read as a user choice on the next load, so visiting a preset classroom permanently downgraded auto classrooms to preset agents. Track the origin with a persisted agentSelectionIsUserSet flag (set by AgentBar interactions, cleared by fallback writes). Also: voice overrides are now clearable (undefined deletes), the override type is single-sourced in voice-resolver, and persist rehydration of the new fields is pinned by tests. * fix(agents): don't mark stage-derived selections as user choices Round-2 review: clicking the already-active mode tab no longer flags the selection as user-set (a visual no-op must not promote stage defaults to a choice), generation-preview clears the flag when it writes the generated roster, and tests now pin that setter writes actually reach storage. * fix(agents): restore generated agent selection when switching back to Auto Review follow-up (#723): handleModeChange('auto') left the preset ids selected, so the UI said Auto while discussion playback kept using preset agents, and the persisted user-set auto pair could never validate on restore. Switching to Auto now selects the loaded classroom's generated agents (empty when none are loaded, falling back to stage defaults on the next classroom load). |
||
|
|
d7c0d80afe |
feat(tts): per-agent auto-voice quality + register-once timbre stability (#670) (#672)
* feat(tts): voxcpm voice-design types + deterministic voice id helpers (#670) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(tts): emit + persist per-agent voiceDesign descriptor (#670) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(tts): voxcpm voice registration backend client (#670) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(tts): voxcpm-voice ensure/register endpoint (#670) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(tts): client auto-voice registration + reference-clip cache (#670) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(tts): reference registered voice id in vLLM-Omni speech (#670) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(tts): thread voiceDesign + backend through tts call sites (#670) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(tts): match vLLM-Omni voice registration contract (#670) Live e2e against the real backend revealed the assumed multipart contract was wrong: POST /v1/audio/voices needs name + consent + audio_sample (not voice_id/file), and there is no per-name GET (405) — existence must list /v1/audio/voices and check membership. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * refactor(tts): make auto-voice register-once pattern provider-neutral (#670) The descriptor + register-once/reference-by-id pattern is not VoxCPM-specific. De-couple it into a provider-neutral seam so other registration-capable providers (ElevenLabs/MiniMax/Doubao voice cloning, …) can plug in: - lib/audio/voice-design.ts: VoiceDesign + buildVoiceDesignPrompt / normalizeVoiceDesign / getDeterministicVoiceId (neutral 'auto-<hash>' id, namespaced by providerId). - lib/audio/voice-registration.ts: VoiceRegistrationAdapter interface + providerId->adapter registry + supportsVoiceRegistration. - lib/audio/voice-registration-client.ts: neutral ensureRegisteredVoice() + IndexedDB clip cache. - app/api/generate/voice (replaces .../voxcpm-voice): dispatches by providerId. - voxcpm-registration.ts becomes the VoxCPM adapter (sole registered provider). - AgentConfig.voiceDesign + DB cache table renamed neutral. Behavior-preserving; VoxCPM-specific bits (inline (prompt)text, backend kinds, capability gate) stay in the voxcpm modules. Full suite green (674). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(tts): build the voice clip Blob from an inline Uint8Array (#670) A factored-out helper returning Uint8Array widened to Uint8Array<ArrayBufferLike>, which next build (stricter than bare tsc) rejects as a BlobPart. Inline the buffer like the rest of the repo does. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(tts): managed TTS providers resolve model server-side (#670) A server-managed TTS provider's model was still client-driven, but the managed-provider settings UI hides the model field — so a backend whose model id isn't the client default (e.g. VoxCPM/vLLM-Omni serving the full model path) always 500'd. resolveTTSModel() makes the model authoritative from server config (${PREFIX}_MODELS, first entry) for managed providers, like key/baseUrl; unmanaged/unconfigured providers keep the client model unchanged. Used by the tts and voice routes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(tts): single agent-voice resolver + stable teacher narration voice (#670) Route all TTS voice resolution (narration, discussion, preview) through one resolveAgentVoiceOptions(agent, ...) that reads the agent profile and registers + references the voice by id. Eagerly warm up generated agents' voices on save. Critically fixes the teacher narration drifting (male/female jumps): the registry is always seeded with DEFAULT_AGENTS, so the old narration lookup find(role==='teacher') returned the default teacher (no voiceDesign) instead of the generated one — so narration never registered a voice and fell back to the inline prompt. pickNarratorAgent() now prefers the teacher carrying a voiceDesign. Also: drop language from the deterministic voice id (descriptor already encodes it) so narration (directive) and discussion (locale) resolve the same id; log the effective registeredVoiceId. Regression tests for pickNarratorAgent. Verified e2e: agent-profiles -> 1 voice registration -> 16 narration TTS all referencing the same registeredVoiceId; voice present on the backend. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(tts): fall back to persona as the voice seed for agents without a voiceDesign (#670) Preset/default agents carry no LLM voiceDesign. Rather than add a static field, derive the bootstrap descriptor from the agent's persona when voiceDesign is absent, so they still register one stable reference voice (stable-but-generic; persona is not a vocal spec). Generated agents keep their LLM voiceDesign. Also hardens replay: stage snapshots drop voiceDesign but keep persona. Unit-tested: resolveAgentVoiceOptions uses real voiceDesign when present, persona otherwise. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(tts): match Auto Voice by its localized label in the agent voice picker (#670) The picker filtered on voice.name ('Auto Voice'), but Auto Voice is shown via its localized label (自动音色). Searching '自动' returned '没有匹配音色'. Match the localized label for the Auto Voice option so it's findable in any language. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(tts): address code-review round 1 (#670) - strip parentheses in the voice-design prompt so a paren in the descriptor/ persona can't corrupt the (prompt)text bootstrap delimiter - gate POST /api/generate/voice on isServerTTSProviderDisabled (#665): a force-disabled provider was off for the tts route but not this sibling - check voiceExists before re-registering a client-cached clip, so a cached voice that's still live isn't needlessly re-uploaded every session - dedup concurrent ensureRegisteredVoice calls via an in-flight promise map (eager warm-up + first utterance no longer double bootstrap/register) - warm up only the narrator (teacher), not every generated agent, to avoid synthesizing voices at save time for agents that may never speak - collapse the duplicated vLLM-Omni speech payload into one object Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(tts): prettier formatting (#670) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(tts): key auto-voice memo by (voiceId, backend) so a base-URL change re-registers (#670) Addresses review (cosarah): registeredThisSession/inFlight were keyed by voiceId alone, so switching the VoxCPM base URL mid-session made ensureRegisteredVoice short-circuit and return an id registered only on the old backend — TTS then sent that stale registeredVoiceId and skipped the inline fallback, failing with voice-not-found on the new backend. Memo key now includes the base URL; the IndexedDB clip cache stays keyed by voiceId (the reference clip is backend-independent and reused to re-register elsewhere). Regression test added. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(tts): include API key in the auto-voice memo key (#670) Addresses review (cosarah, round 2): memoKeyFor was keyed by (voiceId, baseUrl) but not the API key. Registration/existence checks and speech calls are auth-scoped, so switching account/key on the same base URL could reuse a registeredVoiceId from the old credentials and skip re-validation. Memo key now includes the API key (in-memory only, never persisted/logged). Regression test covers the key-switch case. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: 杨慎 <117187635+cosarah@users.noreply.github.com> |
||
|
|
3e1e9f083b |
feat(tts): unify provider-enablement model, default browser-native off (#665) (#666)
A single TTS provider-enablement model honored everywhere (picker,
auto-assignment, fallback, lecture/playback, server).
Model
- lib/audio/provider-enablement.ts: isTTSProviderConfigured /
isTTSProviderEnabled / listEnabledTTSProviderIds. Availability drops the
registry defaultBaseUrl fallback (fixes Lemonade's always-shown voices),
honors the per-provider `enabled` flag, server-disable overrides (server
precedence).
- voice-resolver: getEnabledProvidersWithVoices (enabled server/custom
providers) and getSelectableProvidersWithVoices (the ONE client source of
truth = enabled providers + opt-in browser-native when the browser exposes
voices). resolveAgentVoice picks deterministically among the given providers,
honors a voiceConfig only while its provider is present.
- Picker + discussion build their list from getSelectableProvidersWithVoices, so
the AgentBar and the live discussion agree on selectable voices (fixes
student agents going silent in auto mode). Students resolve via
resolveAgentVoice; the teacher uses the global lecture voice verbatim (same
voice the pre-generated lecture sends), so lecture and discussion never
diverge.
- Voice assignment vs fallback: an agent's voice is bound at generation (the
agent-profile LLM picks from availableVoices; preset uses the user's pick).
The fallback for a missing/invalid/unavailable voice is applied only at the
live TTS calls, not baked into the saved agent — so it reflects current
provider availability.
Defaults / opt-in
- Browser-native: first-class, default OFF, opt-in. Built-in providers default
enabled:true (only surface once configured).
- TTS master toggle defaults OFF; first server-sync auto-enables it when a
provider is configured (mirrors image/video).
- Persist v3->v4 migration normalizes the never-user-editable flags.
Server/admin disable (server precedence): server-providers.yml `enabled:false`
and env `TTS_<P>_ENABLED=false` (incl. browser-native). Surfaced as a per-provider
`disabled` flag on /api/server-providers; merged as serverDisabled; enforced at
the /api/generate/tts trust boundary, classroom media generation, and /api/health.
UI
- Settings TTS panel: per-provider enable toggle, disabled (with a hint) when the
provider is unavailable, locked off (with a note) when server-disabled.
- AgentBar voice pills render disabled with a "No voice" placeholder when no
provider is enabled. The toolbar media-popover TTS tab keeps its existing
on/off toggle (no picker/CTA).
Tests: predicates, resolver determinism + unified selectable list, server
force-disable, store merge + migration + TTS auto-enable, i18n parity. Verified
in a real browser (Playwright): settings toggle states + a full Gemini classroom
generation whose TTS requests used the picked voice.
Found on main @
|
||
|
|
2586cb5e05 |
fix(tts): surface provider rate-limit errors as HTTP 429 (#644)
TTS providers throw a plain `Error` on any non-OK response, and the `/api/generate/tts` route catches everything as `GENERATION_FAILED` (500), so an upstream 429 (quota/concurrency) was indistinguishable from a real failure — clients had no signal to retry/back off. `TTSRateLimitError` already existed for exactly this (with a TODO noting the route collapses everything uniformly), but only Doubao threw it and the route never mapped it. - Add a `RATE_LIMITED` API error code. - Map upstream HTTP 429 to `TTSRateLimitError` at every provider's `!response.ok` site via a shared `throwIfTtsRateLimited` helper. - Return `429 RATE_LIMITED` from the TTS route when a `TTSRateLimitError` is caught (all other errors unchanged). Purely additive: non-429 responses keep their existing error messages. Co-authored-by: ly-wang19 <ly-wang19@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: wyuc <wang-yc24@mails.tsinghua.edu.cn> |
||
|
|
35d36905d1 |
Make server-configured providers admin-managed, not client-overridable (#620) (#624)
* refactor(providers): make server-configured providers admin-managed, not client-overridable A server-configured provider is now authoritative: the operator owns its API key and base URL, and the client cannot override either. This collapses the tri-state (server key / client base URL / client key) that was the root of a recurring "missing API key" bug class — when the settings UI echoed a base URL with an empty key, routes that treated "any base URL ⇒ fully client-configured" dropped the server key. #533 patched that per-route via canUseServerApiKeyForBaseUrl; this removes the guard entirely by fixing the design. Server: - resolve*ApiKey/BaseUrl ignore client values when the provider is server- configured; add isServerConfiguredProvider(section, id); drop canUseServerApiKeyForBaseUrl + normalizeProviderBaseUrl. - getServer*Providers() no longer return base URLs — only allowed models (LLM/image) and the managed flag (presence) — so an internal gateway/proxy endpoint is never exposed to the client. - routes (chat/image/video/tts/asr/pdf + verify-*) resolve managed providers purely server-side; SSRF validation now applies only to unmanaged (client) base URLs. Client: - drop serverBaseUrl from ProviderSettings, the settings store, request-building (TTS base URL no longer echoes the server URL) and voice resolution. - settings UI renders managed providers read-only: the key/base-URL override inputs and region toggles are hidden, leaving the "managed by server" notice. Tradeoff: overriding a server provider's key with your own for the same provider is removed; add a separate custom provider instead. Closes #620. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(providers): refresh managed-provider copy, drop dead audio-settings, add #620 e2e - serverConfiguredNotice now says the provider is managed and not editable here, pointing users to add a separate provider for their own credentials (7 locales). - delete components/settings/audio-settings.tsx — unreferenced dead code. - add e2e/tests/managed-provider-620.spec.ts: managed provider hides the key/base-URL inputs; unmanaged provider keeps them. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(providers): lock the model catalog for managed providers with a pinned list When the operator pins an allowed model list (MODELS env/yaml ⇒ serverModels), the model catalog is admin-managed too: the LLM provider panel shows the pinned models read-only with a "Server" badge and no add / reset / edit / delete affordances. Without a pinned list the server manages only credentials and the user keeps curating models. This removes the prior inconsistency where the settings panel let you add models to a pinned managed provider that the generation toolbar then filtered out. e2e: the managed-openai case now also asserts the catalog is locked. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(providers): ignore (not reject) client overrides for managed web-search; clear legacy serverBaseUrl Addresses review feedback on #624: 1. Managed web-search providers now ignore a client-sent key/baseUrl instead of validating and rejecting it. A server-configured provider is admin-owned, so the server config is authoritative and a stale client base URL is dropped rather than 400-ing the request. The allowlist validation still runs for unmanaged providers, where the client base URL is actually used. 2. Drop the removed `serverBaseUrl` field from persisted localStorage on rehydrate (settings version 2 -> 3, stripped in both migrate and merge), so old clients don't keep a stale server URL in client state. Tests: web-search route test split into unmanaged-rejects-allowlist and managed-ignores-client-baseUrl; settings rehydrate test asserts a legacy serverBaseUrl is stripped. Full suite 381 pass; e2e 2 pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: 杨慎 <117187635+cosarah@users.noreply.github.com> |
||
|
Ismail Ariyan
|
07115dff17 |
feat(asr): add Azure STT (Fast Transcription) provider (#175)
* add Azure ASR config and localization * add transcribeAzureASR func * fix(settings): prevent Azure STT crash in ASR modal * feat(i18n): add Azure STT provider support in language locales |
||
|
|
be759a8252 |
feat: add lemonade to provider (#508)
* feat: add lemonade to provider * fix(lemonade): correct TTS/ASR model IDs and expand Kokoro voices - TTS model: kokoro → kokoro-v1 - ASR model: single 'whisper' → 6 Whisper variants (Base/Large-v3/Large-v3-Turbo/Medium/Small/Tiny), default Whisper-Base - Expand lemonade-tts voices to full Kokoro v1 set (54 voices across en-US/en-GB/zh-CN/ja-JP/es/fr-FR/hi/it-IT/pt-BR) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(lemonade): zh-TW i18n parity, real Kokoro default voice, language code consistency - zh-TW.json: add missing providerLemonadeTTS/ASR/Image keys - TTS default voice: 'default' → 'af_heart' (Kokoro expects real voice ID, not placeholder) - Voice language codes: 'es'/'hi' → 'es-ES'/'hi-IN' for consistency Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test(lemonade): add unit tests for image/TTS/ASR adapters and wav-utils Covers: - lib/media/adapters/lemonade-image-adapter: request shape, base URL fallback, custom model, optional Bearer auth, error paths, connectivity probe - lib/audio/tts-providers (lemonade-tts case): request body (model/voice/format), af_heart fallback, optional auth, error path - lib/audio/asr-providers (lemonade-asr case): WAV-only validation, model/language form fields, "audio is empty" graceful empty result, error propagation, default model fallback - lib/audio/wav-utils: isWavBlob detection by MIME and filename, normalizeASRUploadAudio pass-through and wav handling Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(lemonade): remove text path fork and harden fallbacks * style: format lemonade tts files * i18n: add lemonade voice language labels * fix(lemonade): align image default model and harden wav detection * test(lemonade): fix File BlobPart typing * fix(lemonade): simplify tts voice selection and clear asr test state * docs: highlight Lemonade support --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-authored-by: wyuc <wang-yc24@mails.tsinghua.edu.cn> |
||
|
wyuc
|
5e12ad892c |
[codex] add VoxCPM2 TTS provider (#496)
* add voxcpm2 tts provider * address voxcpm review feedback * translate voxcpm locale strings * narrow toolbar model pill * refine toolbar media controls * address voxcpm review followups |
||
|
杨慎andClaude Opus 4.6
|
470aa5e42e |
feat: inline language inference for outline and PBL generation (#412)
* feat: inline language inference in outline generation Replace manual language selection with automatic inference from user requirement text. The outline generator now produces a languageDirective that propagates through the entire generation pipeline. Key changes: - Rewrite Language Inference section in outline prompt with decision rules for foreign language learning, cross-language PDF, proxy requests, and terminology handling - Reorder pipeline: outlines before agents, so agent profiles can use the inferred languageDirective - Pass languageDirective through scene content/actions generation - Add SSE streaming of languageDirective from outline generation - Remove manual language selection UI Eval test suite: - 50 test cases covering 9 scenario types: single-language, language learning, immersive, explicit instruction, code-switching, minimal input, user profiles, cross-language PDF, locale mismatch - LLM-as-judge evaluation with configurable inference/judge models - 50/50 pass rate with gemini-3-flash-preview + gpt-4o judge - Excluded from CI (requires LLM API keys) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * chore: remove useless attributes in test case * feat: pass languageDirective to PBL, remove hardcoded zh-CN/en-US - Remove pblConfig.language field, use languageDirective from outline inference instead - Delete all Chinese prompt duplicates in pbl-system-prompt.ts, agent-templates.ts; keep English templates with languageDirective injection - Remove zh-CN/en-US branches in generate-pbl.ts (initial prompt, post-process context, welcome message) - Pipe languageDirective through scene-generator → generatePBLContent → IssueboardMCP → agent templates - Remove i18n template wrapping from PBL welcome messages in pbl-renderer.tsx and use-pbl-chat.ts; use LLM-generated content directly to avoid language mismatch when UI locale ≠ course language - Update pblConfig schema in outline prompt (remove language field) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * refactor: clean up legacy language fields, pass languageDirective through buildSceneFromOutline - Add languageDirective param to buildSceneFromOutline, combining with outline.languageNote via buildLanguageText - Remove dead types: AudienceProfile, StylePreferences, LegacyUserRequirements, SceneOutline.language - Remove Stage.language and all its propagation (storage, stageInfo, prompt-builder fallback) - Remove GenerateClassroomInput.language and job store inputSummary.language - Remove unused pdfLanguageSample from outline SSE route Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: persist languageDirective to IndexedDB, fix TTS preview and dead refs - Add languageDirective to StageRecord and stage-storage write path - Add DB v9 migration: convert legacy language locale codes to directives - Use voice.language for TTS preview text instead of dead localStorage key - Omit empty ## Language section in PBL agent prompts - Use JSON.parse for extractLanguageDirective unescape (\uXXXX support) - Remove dead i18n keys: toolbar.languageHint, pbl.chat.welcomeMessage Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
||
|
wyucandClaude Opus 4.6
|
9a0060e2f0 |
feat(audio): add custom OpenAI-compatible TTS/ASR provider support (#357) (#409)
* feat(audio): extend TTSProviderId/ASRProviderId to support custom providers * feat(audio): update provider registry helpers for custom provider support * feat(audio): route custom providers to OpenAI-compatible implementations Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(audio): add custom TTS/ASR provider CRUD to settings store * feat(audio): add dialog for creating custom audio providers * feat(audio): add custom provider UI with voice management and delete Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(i18n): add custom audio provider translation strings * fix(audio): resolve remaining type errors for custom provider support Add `as keyof typeof` casts where TTSProviderId/ASRProviderId (which now include template literal types) index into the built-in-only registries TTS_PROVIDERS, ASR_PROVIDERS, DEFAULT_TTS_VOICES, and DEFAULT_TTS_MODELS. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(audio): allow custom provider IDs to reach switch default branch The early guard `if (!provider) throw` blocked custom provider IDs from reaching the switch-case default branch where they get routed to OpenAI-compatible implementations. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(audio): add default model field to dialog and polish voice list UI - Add "Default Model" input to custom audio provider dialog (fixes Kokoro needing model:"kokoro" instead of fallback "gpt-4o-mini-tts") - Redesign voice list as aligned table with column headers, hover reveal delete buttons, and empty state message - Add Enter key support to voice add inputs - Use consistent grid layout for add voice row Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(audio): fallback to customDefaultBaseUrl when testing custom TTS provider The baseUrl field in provider config is empty by default (it's an override field). Custom providers store their default URL in customDefaultBaseUrl, which needs to be used as fallback for test. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(audio): auto-select first voice when adding to custom TTS provider When a custom provider's voice is 'default' (initial state before any voices are added), adding the first voice now automatically updates the active voice selection so test TTS works immediately. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(audio): fallback to customDefaultBaseUrl in all TTS call sites Scene generator, discussion TTS, and agent bar voice preview all read providerConfig.baseUrl which is empty for custom providers (the default URL is stored in customDefaultBaseUrl). Add fallback chain to all 4 call sites so custom providers work during actual playback, not just in the settings test. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(audio): include custom provider voices in agent voice picker getAvailableProvidersWithVoices only iterated built-in TTS_PROVIDERS, so custom providers never appeared in the agent bar voice dropdown. Now also iterates ttsProvidersConfig for custom-tts-* entries and includes their customVoices in the picker. Also updated getServerVoiceList, findVoiceDisplayName, and resolveAgentVoice to handle custom provider voice lookups. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(audio): fallback to customDefaultBaseUrl in all ASR call sites Same issue as TTS: audio-recorder, asr-settings, and audio-settings all read providerConfig.baseUrl which is empty for custom providers. Add customDefaultBaseUrl fallback to all 3 ASR call sites. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(audio): add model list management for custom ASR providers Custom ASR providers now support adding/removing models with a table UI (same design as TTS voice list). The model selector dropdown shows custom models and auto-selects the first one. Also adds custom ASR providers to the Toolbar's ASR provider dropdown and fixes custom provider handling in audio-settings.tsx (baseUrl fallback, request URL preview, API key section visibility). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(audio): seed customModels from defaultModel and fix ASR model label - When creating custom ASR provider with a defaultModel, auto-add it to customModels so the model list isn't empty on first visit - Fix ASR settings label from "TTS Model" to "Default Model" Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(audio): separate ASR model management from creation dialog ASR models should be managed in provider settings (add/remove from list, select default), not pre-filled at creation time. Unlike TTS where a single default model suffices, ASR users need to manage multiple models (e.g. whisper-small, whisper-large-v3). - Remove Default Model field from ASR creation dialog (keep for TTS) - Remove defaultModel parameter from addCustomASRProvider - ASR models are added via the model list UI in provider settings Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(audio): simplify ASR model management — first model is default Remove the separate model selector dropdown for custom ASR providers. The first model in the list is always used as default. Users manage the model list only (add/remove), no separate "default" selection. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(audio): add loading state and detailed errors to ASR test - Show spinner with "Processing..." while waiting for transcription - Disable record button during processing - Show specific error messages from API (e.g. "No transcription generated") instead of generic failure text - Handle empty transcription result as an error with helpful message Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(i18n): use correct key for processing label in ASR test button * fix(audio): polish custom provider UX and type safety - Add proper type declarations for custom provider fields (customName, customDefaultBaseUrl, customVoices, etc.) eliminating 30+ unsafe casts - Show custom ASR providers in toolbar with default language list - Hide custom ASR providers without models from toolbar selection - Add no-models warning in ASR settings panel - Add delete confirmation dialog for custom TTS/ASR providers - Add duplicate voice/model ID validation with toast error - Add dedicated i18n keys for ASR model management (modelNamePlaceholder, addModel) - Fix i18n key for ASR processing label (settings.asrProcessing) - Add radio-button model selection in custom ASR provider settings - Add visible OpenAI-compatible description in provider creation dialog - Add CUSTOM_ASR_DEFAULT_LANGUAGES constant for toolbar language options Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(test): add isCustomTTSProvider/isCustomASRProvider to audio types mock The settings-server-sync test mocked @/lib/audio/types as an empty object, but settings.ts now imports isCustomASRProvider for provider validation. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(audio): add customDefaultBaseUrl fallback in all TTS/ASR call sites When a custom provider is created, the actual URL is stored in customDefaultBaseUrl while baseUrl is initialized empty. Several call sites only checked baseUrl, causing fetch failures during course generation. Add fallback to customDefaultBaseUrl in: - generation-preview/page.tsx (course generation TTS) - tts-providers.ts (getCurrentTTSConfig) - asr-providers.ts (getCurrentASRConfig) - media-popover.tsx (TTS preview) - tts-config-popover.tsx (TTS preview) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * style: format generation-preview/page.tsx Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: remove unused ttsSpeedRange variable in media-popover Flagged by CodeQL code quality check. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> |
||
|
wyucand杨慎
|
3c687dd523 |
fix(tts): unify model selection to per-provider and fix ElevenLabs model_id (#326)
* fix(tts): unify model selection to per-provider and fix ElevenLabs model_id (#324) - Move model selection from global ttsModelId to per-provider ttsProvidersConfig[providerId].modelId - Remove supportsModelSelection flag, use models.length > 0 instead - Add defaultModelId to TTSProviderConfig and ASRProviderConfig - Add providerOptions field to TTSModelConfig for future extensibility - Fix ElevenLabs hardcoded model_id: 'eleven_multilingual_v2' - Fix MiniMax TTS using config.model instead of config.modelId - Update model lists from provider docs: - Qwen: add qwen3-tts-instruct-flash, qwen-tts - ElevenLabs: add eleven_flash_v2_5, eleven_flash_v2 - MiniMax: remove non-existent speech-2.8 series - Replace custom model CRUD UI with simple Select dropdown - Remove MiniMax-specific hardcoded model selector from audio-settings - Add migration for old ttsModelId/asrModelId/model fields * feat(tts): voice picker groups by provider:model, sets both model and voice Voice selection in agent bar now groups voices by model. Picking a voice from a model group sets both modelId and voiceId on the agent. TTS settings shows available models as display-only chips. - Add compatibleModels field to TTSVoiceInfo (OpenAI marin/cedar restricted to gpt-4o-mini-tts) - Add modelId to agent voiceConfig and ResolvedVoice - Add ModelVoiceGroup to ProviderWithVoices, build groups in getAvailableProvidersWithVoices - Update AgentVoicePill and TeacherVoicePill to iterate modelGroups - Thread modelId from voice selection through discussion TTS pipeline - Replace model Select with display-only chips in tts-settings * fix(tts): show provider name in voice picker group headers * fix(tts): restore MiniMax speech-01 models per official API docs * fix(tts): add MiniMax speech-2.8 models per official docs, default to speech-2.8-hd * feat(tts): expand MiniMax voice list with recommended voices * fix(tts): remove non-official MiniMax recommended voices, keep 10 from official list * fix: address CR findings - align defaults, fix test mocks, ASR modelId fallback * fix(tts): pass modelId in voice preview and classroom generation * fix(tts): voice picker highlights only the exact model:voice match --------- Co-authored-by: 杨慎 <117187635+cosarah@users.noreply.github.com> |
||
|
wyucandHi-Jiajun
|
9badd59856 |
feat: add and expand MiniMax provider support (#182)
Add MiniMax LLM models (M2–M2.7), TTS provider (speech-2.8/2.6/02/01), image generation (image-01), and video generation (Hailuo 2.3/02, T2V-01). Unify base URLs to api.minimaxi.com and add MiniMax-specific base URL normalization for the Anthropic-compatible endpoint. Co-authored-by: Hi-Jiajun <3138947285@qq.com> |
||
|
wyucand刘少杰
|
5adf91c6f3 |
feat(settings): add configurable model selection for TTS and ASR (#108)
Add ttsModelId/asrModelId to persisted settings, model management UI for providers that support model selection, and propagate selected model through preview, generation, and transcription flows. Co-authored-by: 刘少杰 <liushaojie@cvte.com> |
||
|
wyucandClaude Opus 4.6
|
694570783a |
feat(tts): add Doubao TTS 2.0 (Volcengine Seed-TTS 2.0) provider (#283)
* feat(tts): add Doubao TTS 2.0 (Volcengine Seed-TTS 2.0) provider Add Doubao TTS as a new TTS provider with 17 voices (14 Chinese, 3 English), streaming response parsing for Volcengine's chunked JSON format, rate-limit error handling, and full integration across types, constants, provider implementation, server config, settings store, i18n, and UI components. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * ui(tts): split Doubao API key into separate App ID and Access Key fields Show two input fields (App ID + Access Key) for Doubao TTS in settings, instead of requiring the compound "appId:accessKey" format. The values are combined internally so no changes are needed to the API layer. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(tts): address code review feedback for Doubao TTS provider - Add TODO note explaining TTSRateLimitError's future use for retry/backoff - Replace atob() + manual byte loop with Buffer.from() for server-side code - Trim supportedFormats to ['mp3'] since only mp3 is used in requests - Add i18n keys for Doubao App ID / Access Key labels (zh-CN + en-US) - Add show/hide toggle to Access Key input field Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
||
|
|
ddb52241fb |
feat: add Discussion TTS with per-agent voice assignment (#211)
* feat(tts): add resolveVoice() and getServerVoiceList() utilities
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* feat(tts): add AudioIndicator equalizer bars component
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* feat(tts): add onSegmentSealed callback to StreamBuffer
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* feat(tts): add voiceOverrides field to AgentConfig and AgentTemplate
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* feat(tts): add useDiscussionTTS hook with audio queue and cleanup
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(tts): add audio state indicator to Roundtable bubble
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* feat(tts): wire onSegmentSealed callback through chat sessions
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(tts): add per-agent voice dropdown to AgentBar
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(tts): integrate useDiscussionTTS in Stage and pass state to Roundtable
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* style(tts): refine voice dropdown to pill-style selector
Replace native select styling with a compact rounded-full pill
that blends into the agent row. Remove border, use muted bg,
smaller text, and a custom chevron icon.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* style(tts): use shadcn Select for voice dropdown, link with TTS toggle
- Replace native <select> with shadcn Select component for consistent UI
- Hide voice dropdown when TTS is muted (ttsMuted)
- Compact pill-style trigger with rounded-full, no border, muted bg
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* style(tts): add voice label prefix and always show dropdown
Show "音色: Alloy" instead of plain "Alloy" in the voice pill.
Always show dropdown regardless of mute state (voice config is
independent of playback).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* style(tts): add volume icon hint in collapsed AgentBar
Show a small Volume2 icon in the collapsed pill when voice config
is available, hinting that voice settings are inside.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(tts): fix voice dropdown layout and click handling
- Move voice pill below agent name (second line) to prevent
horizontal overflow in English
- Wrap Select in div with onPointerDown stopPropagation to fix
Radix click-through to parent row
- Add line-clamp-1 to descriptions for consistent row height
- Use items-start instead of items-center for better multi-line alignment
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* refactor(tts): redesign AgentBar voice layout for compactness
- Single-line layout: checkbox · avatar · name · role · voice pill
- Remove descriptions from agent rows (saves vertical space)
- Extract AgentVoicePill component to isolate Select event handling
- Smaller avatars (size-7), tighter row padding (py-1.5)
- Voice pill uses Volume2 icon + voice name (no prefix text)
- Works in both Chinese and English without overflow
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(tts): cross-provider voice selection per agent
- Change voiceConfig from per-provider lookup to explicit
{ providerId, voiceId } per agent
- Each agent can use a different TTS provider's voice
- Voice picker dropdown groups voices by provider
- useDiscussionTTS routes TTS requests per agent's provider
- resolveAgentVoice falls back to global provider if no config
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(tts): align role badge and voice pill across agent rows
Give role badge fixed width (w-14 text-right) so role text
and voice pills align vertically across all rows regardless
of agent name length.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(tts): fix role badge and voice pill alignment
Wrap role badge + voice pill in a fixed-width container
(w-[9.5rem] justify-end) so both align vertically across
all agent rows regardless of name or role text length.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* style(tts): align role badge and voice pill across agent rows
Add min-w-[52px] text-right to role badge so it starts at a
consistent position regardless of agent name length.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(tts): use fixed w-[60px] for role badge alignment
* fix(tts): use fixed w-[88px] for voice pill alignment
* fix(tts): prevent click-outside from closing AgentBar when Select portal is open
* fix(tts): comprehensive voice picker rewrite
- Replace Radix Select with Popover + button list (fixes click issue)
- Fix getAvailableProvidersWithVoices to always include global provider
- Widen panel from w-80 to w-96 (prevents name truncation)
- Voice pill uses primary color instead of gray (more visible)
- Extract renderAgentRow helper to reduce duplication
- Popover shows voices grouped by provider with active state
- Add findVoiceDisplayName utility
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(tts): align voice provider availability with toolbar logic
* fix(tts): fallback to first available provider when global provider has no voices
* refactor(tts): remove global provider fallback from voice resolution
Voice resolution now only depends on available providers (those with
API keys or server-configured). No more globalProviderId parameter.
Fallback is first available provider, then browser-native-tts.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(tts): add browser native TTS voices to agent voice picker
Load speechSynthesis.getVoices() in AgentBar and include as a
"Browser Native" provider group in the voice popover. No API key
needed - always available if browser supports it.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(tts): simplify toolbar TTS to on/off toggle, add disabled state
Toolbar:
- Replace volume slider with simple TTS on/off toggle button
- Remove ttsMuted/ttsVolume/onVolumeChange props from CanvasToolbar
- Toggle now controls ttsEnabled (not ttsMuted)
AgentBar:
- Collapsed: show VolumeX icon when TTS disabled
- Voice pills show disabled state (gray, cursor-not-allowed, no popover)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* refactor(tts): simplify Settings TTS tab to toggle + provider config
Remove voice selection, speed slider, preview/test, Azure locale
filter from Settings TTS tab. Voice is now per-agent in AgentBar.
Keep: on/off toggle, provider selector, API key + base URL config.
Add hint text pointing to AgentBar for voice configuration.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* refactor(tts): simplify media popover TTS tab to toggle only
* fix(tts): add voice config hint to media popover TTS tab
* feat(tts): add per-voice preview button in voice picker
Each voice row in the popover has a small speaker icon button.
Click to preview the voice with "欢迎来到AI课堂" / "Welcome to
AI Classroom" (follows i18n). Browser native uses Web Speech API,
server TTS calls /api/generate/tts. Click again or close popover
to stop. Shows spinner while generating.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(tts): preview text follows course language instead of UI language
* refactor(tts): redesign AgentBar expanded panel layout
- Teacher always at top with voice pill (works in both modes)
- Mode tabs moved below teacher
- Auto mode: single compact row with shuffle icon + description
- Max turns: compact inline row with smaller input
- Preset mode: only student agents listed (teacher already above)
- Remove large shuffle animation from auto mode
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* refactor(tts): merge max turns into teacher row
* refactor(tts): separate teacher row and max turns, use stepper UI
- Teacher row: avatar + name + voice pill only
- Max turns: bottom row with MessageSquare icon + compact stepper
(minus/number/plus in a rounded pill)
- Remove Input component dependency
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(tts): increase voice pill contrast in dark mode
* fix(tts): make max turns input editable, tighten panel padding
* fix(tts): restore shuffle animation in auto mode (compact version)
* fix(tts): adjust auto mode text spacing and add voice auto-assign hint
* fix(tts): auto-close voice popover after selecting a voice
* fix(tts): increase auto mode vertical padding for better balance
* fix(tts): push auto mode text toward bottom with flex spacer
* fix(tts): reduce auto mode bottom padding
* feat(tts): wait for TTS audio to finish before next agent turn
Add waitForDrain() to useDiscussionTTS that returns a promise
resolving when the audio queue is empty. The agent loop in
useChatSessions now awaits this after buffer drain, so the next
agent's turn doesn't start until the current agent's TTS audio
finishes playing.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(tts): keep bubble visible while TTS audio is still playing
When buffer drains (text=null) but audio indicator is still active,
don't clear liveSpeech. Clear it only when audio state goes idle.
This keeps the speech bubble visible until TTS finishes.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(tts): hold discussion bubble until TTS audio finishes
When StreamBuffer fires the done signal (onLiveSpeech null), Stage
now checks if TTS is still playing. If so, it defers clearing the
bubble state. The bubble stays visible until onAllAudioEnd fires
from the TTS hook (queue empty + nothing playing), then clears.
This prevents the jarring UX where the bubble disappears while
the agent's voice is still audible.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(tts): fix bubble hold - guard onStopSession instead of onLiveSpeech
Root cause: bubble disappears because doSessionCleanup fires via
onStopSession when the agent loop ends naturally, NOT because of
onLiveSpeech(null, null).
Fix: when onStopSession fires and TTS is still playing, defer
doSessionCleanup to onAllAudioEnd callback. Manual stop (user
presses button) still cleans up immediately via handleStopDiscussion.
Use doSessionCleanupRef to avoid circular dependency between
discussionTTS hook and doSessionCleanup useCallback.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(tts): guard BOTH onLiveSpeech and onStopSession for bubble hold
Two paths clear the bubble:
1. onLiveSpeech(null, null) from StreamBuffer done → clears liveSpeech
2. onStopSession → doSessionCleanup → clears all state
Both fire when agent loop ends. Path 1 fires first (tick loop),
path 2 fires after (waitUntilDrained resolves). Both must be
guarded when TTS is still playing.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(tts): hold bubble during TTS playback and respect playback speed
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(tts): LLM picks voice matching agent persona during generation
- Client sends available voices (providerId + voiceId + name) to
/api/generate/agent-profiles
- LLM prompt asks to pick a voice matching each agent's personality
- Parse "providerId::voiceId" from response, save as voiceConfig
- Fallback to index-based assignment if LLM doesn't pick
- Browser native voices hidden when server providers are available
- saveGeneratedAgents accepts and persists voiceConfig
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(tts): restore volume slider in classroom toolbar
Revert the toolbar simplification from
|
||
|
|
3a3e27bac0 |
feat: add ElevenLabs TTS provider (#134)
* feat: add ElevenLabs TTS provider * fix(tts): wire ElevenLabs provider config Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent) Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai> --------- Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai> Co-authored-by: wyuc <wang-yc24@mails.tsinghua.edu.cn> |
||
|
|
fbe8399678 |
feat(generate): server-side media & TTS generation (#75)
* feat(generate): server-side media & TTS generation in classroom pipeline When `enableImageGeneration`, `enableVideoGeneration`, or `enableTTS` flags are passed to /api/generate-classroom, the server now actually generates media files and TTS audio, persists them to disk, and replaces placeholders in the classroom JSON with serving URLs. - Add media serving API route (GET /api/classroom-media/[id]/[...path]) - Add server-side media generation utility (image, video, TTS) - Wire generation phases into classroom pipeline with progress steps - Update AudioPlayer to support URL-based playback (server-generated TTS) - Add audioUrl field to SpeechAction type - Extend /api/health with capabilities detection - Pass through feature flags in generate-classroom route - Update skill docs for optional feature configuration Closes #49 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * style: fix prettier formatting Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(generate): add agentMode support for server-side classroom generation Allow callers to choose between built-in default agents ('default') or LLM-generated custom agent profiles ('generate') via a new agentMode field on GenerateClassroomInput. The resolved agents are injected into outline, scene content, and scene action prompts via teacherContext. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(media): harden download, parallelize media gen, fix TTS concatenation - Add timeout (2min) and max size (100MB) guard to downloadToBuffer - Run image and video generation in parallel via Promise.all - Only split TTS text for MP3 (frame-based); skip for WAV/OGG/AAC whose container headers break on naive byte concatenation - Remove unnecessary type assertion for speechAction.audioUrl Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * style: fix prettier formatting Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(media): harden media serving and TTS generation - Add realpath check to prevent symlink-based path traversal in media route - Stream files via createReadStream instead of readFile to avoid OOM on large videos - Truncate TTS text for non-concatenable formats (WAV/OGG/AAC) when exceeding provider limit - Add 'researching' step to distinguish web search from outline generation progress Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(tts): split long speech actions instead of concatenating audio bytes Adopt the same strategy as client-side: split a long speech action into multiple shorter actions before TTS generation, so each sub-action gets its own audio file. This avoids the broken byte-concatenation approach that corrupted non-MP3 formats (WAV/OGG/AAC container headers). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * refactor(tts): extract shared TTS splitting logic into lib/audio/tts-utils Both client-side (use-scene-generator) and server-side (classroom-media-generation) had duplicated TTS_MAX_TEXT_LENGTH, text splitting, and action splitting logic. Extract into a shared module to keep them in sync. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * chore: remove unused TTS_MAX_TEXT_LENGTH import Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: wyuc <wang-yc24@mails.tsinghua.edu.cn> |
||
|
|
db5ac82167 |
fix: use browser speechSynthesis for playback when browser-native-tts is selected (#28)
* fix: use browser speechSynthesis for playback when browser-native-tts is selected Previously, selecting browser-native-tts as the TTS provider would produce sound in the settings test but remain silent during classroom playback. This happened because: 1. The scene generator correctly skipped pre-generation for browser TTS (it runs client-side, not via API) 2. The playback engine fell back to a silent reading timer when no pre-generated audio was found, instead of calling speechSynthesis This commit adds Web Speech API integration directly in the PlaybackEngine: - New playBrowserTTS() method speaks text via speechSynthesis - Properly wires onend/onerror to advance to the next action - pause()/resume() now handle speechSynthesis.pause()/resume() - stop() and handleUserInterrupt() cancel browser TTS Fixes #25, fixes #12, fixes #5 * fix: handle "default" ttsVoice, extract CJK_LANG_THRESHOLD constant - When ttsVoice is "default" (set by Browser Native TTS which has no voice picker), the voiceURI lookup silently fails and no lang is set, causing Chinese text to be spoken with an English voice. - Extract the 0.3 CJK detection threshold as a named constant CJK_LANG_THRESHOLD with JSDoc explaining the rationale. - Fall through to language auto-detection when voice lookup fails, regardless of the reason (missing voice, "default" sentinel, etc.). * fix: support browser-native tts previews * refactor: extract shared TTS preview logic into useTTSPreview hook The browser-native and API-based TTS preview code was duplicated across tts-config-popover, media-popover, and tts-settings. Extract it into a reusable useTTSPreview hook that handles refs, cancellation, audio lifecycle, and staleness checks in one place. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: yangshen <1322568757@qq.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
||
|
yangshenandClaude Opus 4.6
|
1f2ad6fbff |
fix(tts): use provider's default voice when testing non-active TTS provider
When testing a TTS provider different from the currently active one, the voice from the active provider was sent (e.g. OpenAI's 'alloy' to GLM API), causing error 1214. Now falls back to the selected provider's default voice. Also extracts duplicated default voice mappings into a shared constant. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
wyucandClaude Opus 4.6
|
843d33e5cc |
chore: enforce Prettier formatting and fix lint issues
- Add .prettierignore to exclude vendor packages, lock files, markdown, and YAML - Update .prettierrc: printWidth 100, singleQuote, trailingComma "all" - Run Prettier across all source files for consistent formatting - Fix unused imports (UserRequirements, setTTSProvider) - Fix eslint-disable comment placement after Prettier reformat Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
wyucandClaude Opus 4.6
|
71559690e4 |
feat: redesign MediaPopover with tab layout, grouped selects, and TTS preview
Replace the card-based media popover with a modern segmented-control tab UI. Add GroupedSelect for provider+model selection (filtered to available providers), TTS voice preview with audio playback, voice name localization, and remove unsupported speedRange from qwen-tts provider. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
wyucandClaude Opus 4.6
|
0610053d58 |
Refactor: remove Canvas toolbar, extract model resolution, delete dead routes
- Delete CanvasToolbar, CanvasTool, and toolbar types entirely - Remove toolbar-related props/state from Stage, CanvasArea, Roundtable - Delete 3 obsolete generation routes (generate/, scene/, scene-outlines/) - Extract resolveModel/resolveModelFromHeaders into lib/server/resolve-model.ts reducing ~15 lines of boilerplate per route to 1 line (9 routes updated) - Delete unused lib/audio/qwen.html (156KB unreferenced doc snapshot) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
wyucandClaude Opus 4.6
|
0d20abfed9 |
Initial commit: OpenMAIC AI classroom platform
Next.js 16 App Router application with: - AI-powered interactive classroom generation from PDF/text requirements - Multi-agent discussion system (teacher, students, assistant roles) - Provider abstraction for LLM, TTS, ASR, image, video, web search - Local-first data architecture (IndexedDB/Dexie) - i18n support (zh-CN/en-US) - Structured logger, SSRF guard, lint/build passing (0 errors) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> |