mirror of
https://github.com/THU-MAIC/OpenMAIC.git
synced 2026-10-18 05:40:25 +08:00
@openmaic/renderer@0.0.4
14
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
b4e3145c90 |
feat(roster): single-source the generated agent roster on the stage document (#994)
* feat(roster): single-source the generated agent roster on the stage document The generated agent roster used to live in two places at once: the stage document carried generatedAgentConfigs, while a per-stage IndexedDB mirror (db.generatedAgents) was the actual read authority — and the only home of each agent's voice binding. The split caused real losses: voices vanished on device switches, exports, and shares (the mirror never travels), the load path trusted a device-local table over the portable document, and roster edits persisted through a bespoke debounce with no beforeunload flush. Make the stage document the single source of truth: - Contract: add optional voiceConfig/voiceDesign to GeneratedAgentConfig in @openmaic/dsl (VoiceDesign moves into the contract; the app re-exports it). Additive optional fields — not a breaking serialized-shape change, so DSL_VERSION stays put per the version policy. - Writes: remove every mirror writer (registry saveGeneratedAgents, the stage store's debouncedSaveAgents bypass, classroom-load hydration, import direct writes). setStageAgents now only marks the stage dirty for the shared persistence scheduler plus synchronous in-memory registry/selection mirrors; generation and import embed the roster on the stage. - Reads: classroom load hydrates the registry from the loaded stage's generatedAgentConfigs; export reads the stage roster directly. - Lazy migration: the mirror is retained read-only. On load, a roster (or voice fields) missing from the document is backfilled from the mirror, committed onto the in-memory stage, and persisted by the next flush — idempotent, and gated on the current stage id after every await so a classroom switch cannot leak a stale roster into the registry. - Deletion: deleteStageData discards the deleted stage's pending persistence work so a queued flush cannot resurrect the document. - Round trip: classroom ZIP manifests now carry the voice fields, so export/import preserves agent voices. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(roster): harden deletion, migration, selection, and ingress paths Review hardening on top of the roster single-source change: - Deletion vs in-flight persistence: deleting a stage now registers a session-scoped tombstone (lib/utils/deleted-stages.ts) before the cascade starts. Every persistence landing point checks it — the scheduled flush and the departing-stage retry drop their snapshots, and the aggregate/incremental saves re-check under the document lock, including the incremental path's full-save fallback that would otherwise rebuild a deleted document from the in-memory snapshot. A deletion that fails before the document is removed lifts the tombstone so the still-existing stage keeps persisting. - Lazy migration termination: a fruitless legacy-mirror probe (nothing to merge — e.g. server-generated rosters, which are voiceless by design) is remembered per stage for the session instead of re-querying the mirror on every load forever. Successful merges are not memoized so a failed flush is retried on the next load. Docstrings now state the real lifecycle. - Selection provenance: setStageAgents no longer overwrites a user-set agent selection with the full roster. Stage-derived selections still track the roster; a user-set auto selection is only narrowed to surviving agents (newcomers are not auto-selected, the user-set flag is untouched), and a user-set preset selection is left alone — matching restoreAgentSelection. - Mirror hygiene: deleteStageData clears the deleted stage's rows from the legacy roster mirror (best-effort), and the mirror table's docs now describe its actual access pattern (migration reads + deletion hygiene). - Registry persistence: the agent registry's persist snapshot partializes generated agents out of localStorage, making the stage document the roster's only durable home; the rehydration merge filter remains as defense in depth. - Ingress validation: imported manifest voice fields are structurally validated (malformed bindings dropped per field, the agent survives), and applyGeneratedAgentsToRegistry validates providerId against the known TTS provider registry instead of casting, treating unknown providers as "no bound voice". - Contract: drop the speculative AgentVoiceConfig.modelId — an audit of every roster producer, current and historical, found none that ever emitted it. Contract fields are added once a producer exists, not before. New tests cover the deletion tombstone at both the store and storage layers (including the missing-destination full-save pin and the lock-race re-check), the fruitless-probe memo, the selection-provenance matrix, persistence exclusion of generated agents, and voice-field sanitization on import. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(roster): close tombstone lifecycle, probe-memo, and selection-intent gaps Second hardening pass over the deletion/migration/selection fixes: - Tombstone lifecycle: a deleted classroom id can legitimately come back (deletion is client-side only). Explicit (re)creation points now lift the tombstone — server-copy restore in applyClassroomStageAndScenes and backup restore in importDatabase — so edits after a same-session restore persist again, while in-flight flushes stay fenced. - Failed deletion now restores the dirt it discarded: deleteStageData snapshots the pending map plus the in-flight round before tombstoning, and re-marks it when the delete fails with the document still present. - Legacy-mirror probe memo now distinguishes a FAILED read (null, retried on the next load) from a confirmed-empty mirror (memoized), so a transient IndexedDB error cannot suppress migration for the whole session. - Incremental save tail (currentScene KV row + chat sessions) is tombstone- fenced like the aggregate path, and both paths re-check the tombstone immediately before every write, with comments scoped honestly for the lock-free (no Web Locks) best-effort LWW fallback. - Roster-edit selection mirror: an empty intersection now falls back to the full roster and clears the user-set flag (matching restoreAgentSelection's length > 0 gate), and a user-set selection that equals the pre-edit full roster keeps tracking the roster wholesale so newcomers are not excluded forever by the AgentBar auto-toggle snapshot. - TTS/ASR provider lookups use Object.hasOwn instead of `in`/bare indexing, so prototype-chain keys ('toString', 'constructor', ...) no longer pass the provider whitelist or resolve to Object.prototype members. - Documented the per-tab limit of the tombstone set. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(roster): replace the boolean deletion tombstone with a deletion-generation model A boolean tombstone cannot express the deletion lifecycle once a deleted id legitimately comes back (server-copy restore on revisit, backup import): after the restore lifts the flag, a pre-delete flush still in flight is indistinguishable from a post-restore edit and can overwrite the restored document with pre-delete content. Two rounds of point fixes kept leaking variants of this race, so this restructures the fence instead. deleted-stages.ts now keeps a per-stage monotonic deletion epoch plus a deleted flag. markStageDeleted bumps the epoch; unmarkStageDeleted clears only the flag and never rewinds the epoch. Every persistence path captures the epoch at the moment it captures the data it will write (flush-round snapshot, departing-stage snapshot, aggregate save entry) and re-checks "captured epoch is current AND not deleted" immediately before each individual write. Invariants: - A write captured before a deletion can never land after it, even when a same-id restore has lifted the deleted flag (the captured epoch is permanently stale). - A write capturing data after a restore observes the current epoch and persists normally. - A failed deletion lifts only the flag; the restored pending changes are re-queued as descriptors, so their flush re-captures current store state under the current epoch and is neither dropped nor a stale replay. Also fixed in the same restructuring: - Incremental tail writes (currentScene, chats) each re-check independently; a delete landing while the first tail write is awaiting now fences the chat write instead of riding the earlier check. - Deleting a classroom evicts it from the warm in-memory store, and loadFromStorage treats a warm-but-deleted stage as not loaded (discarding the ghost), so navigating Back to a deleted classroom reaches the server-restore path instead of rendering an editable ghost whose every edit is silently dropped. - importDatabase records each document's pre-import deletion state and reinstates it when a failed import rolls the document back, so an outstanding flush cannot recreate the rolled-back document. - Docstrings on the failed-deletion restore path now state that only dirt captured before the deletion is restored; edits refused during the deletion window are not part of the snapshot. New tests cover the epoch invariants at the storage layer (pre-delete rounds dropped across restores on the incremental, aggregate, and tail paths; post-restore captures landing; strictly increasing epochs), the scheduler layer (departing-retry delete+restore straddle; restored dirt flushing under the current epoch), the warm-ghost restore driven through the real loadFromStorage, and the import lift/rollback pair. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(deletion): report fenced drops, gate ghost eviction on cascade settlement Edge hardening around the deletion-epoch fence; the epoch model itself is unchanged. - An epoch-stale drop is now reported, not disguised: saveStageData / saveStageDataIncremental return a distinct 'stale-dropped' status (entry check, every in-mutation re-check, and both tail-write re-checks) instead of a shape identical to success. saveToStorage reacts by skipping all success bookkeeping - no chatSnapshot rebind to a snapshot that never landed, no pending-dirt clearing - and returns false, keeping its "true means verified write" contract honest. persistDirtySnapshot keeps its existing (correct) semantics explicitly: a drop has nothing to retry. - Warm-ghost eviction is settlement-gated: the deletion cascade records an in-flight bit (begun by deleteStageData, settled in its finally), and the deleted-warm branch of loadFromStorage keeps the warm state while the cascade is unsettled - a delete can still fail before removing the document, in which case the warm state plus the restored pending dirt is the only copy of the pre-delete edits. Once settled, the ghost discard and server-restore path behave as before. - Read-side landing re-check: loadFromStorage re-checks isStageDeleted after hydration, immediately before set(), mirroring the write-side "re-check immediately before landing" discipline - a lock-free mid-cascade read can no longer re-materialize a ghost classroom. - clearStoreForDeletedStage re-checks isStageDeleted at eviction time, so a same-id restore completing inside the cascade-tail window is never wiped. - capturedEpoch is now a required parameter on both storage entry points: the default (call-time capture) silently reopened the capture-point/ validation-point split for future callers; the type system now enforces the pairing. All production callers already passed it explicitly. New tests cover the dropped-status contract, the cascade in-flight lifecycle (including both failure shapes), keep-warm during an unsettled delete with end-to-end pending restore after a failed delete, settlement- then-eviction on success, the eviction guard against a tail-window restore, and the read-side hydration re-check. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(deletion): settle the deleted-warm load by awaiting the cascade outcome Hardening around the deletion settlement gate introduced previously; the epoch model is unchanged. - The deleted-warm branch of loadFromStorage no longer completes the load while the deletion cascade is undecided (which exposed a classroom whose edits were refused without being restorable, and left a successful deletion's eviction with no follow-up reload). It now parks on a per-stage settlement promise (stageDeletionSettled) and branches on the outcome: a failed delete keeps the warm, restored classroom; a successful delete falls through to a full cold load so the server-restore path recovers the route; a navigation that moved on during the park is stopped by the load token guard. The eviction (clearStoreForDeletedStage) deliberately stops claiming a new load token so the parked load stays current - ghost re-materialization is fenced by the read-side deleted re-checks instead. No deadlock: a parked load holds no document lock, and the cascade never waits on a load. - 'stale-dropped' now propagates through the debounced flush path: persistDirtySnapshot returns the sentinel instead of an empty failure set, and startFlushRound skips the chatSnapshot rebind on it (mirroring saveToStorage) - a falsely rebound baseline would let later chat saves no-op-skip chats that never landed and corrupt cross-tab conflict detection. Pending-map clearing keeps its empty-failure semantics (restored descriptors mint fresh revisions). - The cascade in-flight state is a counter (begin++/settle--) instead of a boolean, and deleteStageData is single-flight per stage (a concurrent second call joins the first cascade), so overlapping deletions can never expose an undecided cascade as settled. - The StaleDroppedSave docstring no longer overpromises: a tail saveCurrentScene can land after the cascade's clearCurrentScene; the orphaned cursor row is ignored by the load path rather than removed by the cascade. Net +8 tests: overlapping-cascade counting, settlement-promise resolution (success and failure), concurrent double-delete single-flight join, the parked mid-cascade Back in both outcomes plus the navigation-away guard, and the fenced flush round keeping an honest chat baseline end-to-end. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(deletion): re-verify store identity on keep-warm resume, re-run joined deletes after a mid-cascade restore Three hardening refinements on the deletion/persistence fencing: - loadFromStorage's parked keep-warm branch now mirrors the success path's store-identity guard: if a tokenless writer (e.g. a database import) replaced the stage during the park, a failed deletion falls through to the cold load instead of reporting another classroom as the kept warm state. The ghost-discard block is correspondingly gated on the deleted flag so the fall-through does not mislabel a surviving document as a ghost. - deleteStageData: a joined call carries its own deletion intent. When the first cascade fulfills but a same-id restore lifted the deleted flag in the meantime, the joined caller now runs one fresh cascade against the restored document instead of reporting the pre-restore outcome. Exactly one re-check per call (no recursion); a rejected first cascade still propagates unchanged to every joined caller. - Narrowed the failed-delete restore-ownership comments: the restore covers only the pending map plus an in-flight flush round's dirt; a departing-stage snapshot is outside that capture and is fenced and dropped by design. Comment-only, no behavior change. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(roster): gate deleted-warm handling on stage identity, keep an explicit empty roster authoritative Final-audit fixes, three items: - loadFromStorage keyed its deleted-warm settlement handling (park / keep-warm / ghost discard) behind the scenes.length > 0 shortcut, so a zero-scene warm ghost of a deleted classroom skipped it entirely: the cold load found no document, the ghost stayed in the store, and the classroom loader's server-fallback gate (!getCurrentStage()) never ran — the silently-uneditable trap this branch exists to close, in its scene-less form. Deletion handling now keys on stage identity alone, before any scene-count shortcut; the non-deleted warm skip still requires scenes. - rosterNeedsLegacyFallback collapsed "roster field absent" (document predates roster persistence) with "explicitly persisted empty roster". A future writer persisting [] on a device whose read-only mirror still held stale rows would have its emptied roster resurrected on every load. The full-lift branch now triggers only when the field is absent (undefined); an explicit [] is authoritative and never probes the mirror. No current writer produces [], but the read side no longer depends on that invariant. - The GeneratedAgentConfig docstring claimed the voice fields are not a breaking change; that only holds for this codebase's tolerant structural validators. The generated stage.schema.json sets additionalProperties: false, so consumers pinning an older published schema artifact reject documents carrying the new fields. The docstring now states both sides honestly. Tests: a zero-scene deleted warm ghost is restored through the real loadFromStorage (fails pre-fix), an explicit empty roster does not resurrect stale mirror rows (fails pre-fix), and the rosterNeedsLegacyFallback unit matrix covers undefined vs []. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(deletion): re-queue aggregate persistence when a failed deletion is recovered Direct aggregate saves (saveToStorage: generation completion, server-restore hydration) run outside the pending map and the flush round, so the deletion path's pre-delete dirt snapshot cannot describe them. When a deletion bumped the epoch while such a save was in flight, the save was correctly fenced ('stale-dropped') — but if the deletion then failed before removing the document, the recovery only re-queued the snapshotted scheduler descriptors. The fenced aggregate-only content stayed memory-only with no retry anywhere, and a later reload lost it. The failed-deletion restore now merges a full re-mark of the aggregate (structure / stage / outline / currentScene / chats plus a descriptor for every current scene) into the recovery set, still guarded on the store holding the stage. Because the flush recaptures the CURRENT store state under the CURRENT epoch, this necessarily carries whatever the fenced aggregate save held — and, for the same reason, edits refused by the scheduler during the deletion window, which previously stayed memory-only as well; the scope docstrings now say so instead of declaring them excluded. New integration coverage drives the real store scheduler and the real storage layer end to end: a genuinely fenced in-flight saveToStorage (generation completion) plus a pre-removal deletion failure ends with the completion flag durable on the next flush; a deletion-window scene edit survives the same way; and a successful deletion re-marks nothing (control). Both recovery tests fail against the previous restore logic. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: 杨慎 <117187635+cosarah@users.noreply.github.com> |
||
|
|
153195ca73 |
test(tts): cover splitLongSpeechText / splitLongSpeechActions (#694)
lib/audio/tts-utils.ts had no tests. Add coverage for the sentence -> clause -> hard-split chunking, the invariant that no chunk exceeds maxLength (nor is empty), and the splitLongSpeechActions contract (no-limit provider untouched; short/non-speech untouched; over-limit split into <id>_tts_<n> sub-actions with the parent audioId dropped and text preserved). Behavior verified against adversarial inputs first — no source change, coverage only. Closes #693 Co-authored-by: ly-wang19 <ly-wang19@users.noreply.github.com> Co-authored-by: wyuc <wang-yc24@mails.tsinghua.edu.cn> |
||
|
|
356d454ea4 |
feat(audio): persist audio duration when storing TTS (#861) (#862)
* feat(audio): persist audio duration when storing TTS (#861) Video export (#854) maps each narration clip onto a timeline segment, which needs the clip's duration. The IndexedDB AudioFileRecord already declares a `duration` field but the TTS store path never populated it, so export/import (which round-trip duration via the media manifest) had nothing to carry. Measure duration once when the client persists TTS audio instead of decoding at render time — that keeps the export manifest inspectable in tests without FFmpeg/Chrome (#854 criteria). Scope: client persistence path only. Duration is an asset property, so it lives on AudioFileRecord (matching mediaIndex.duration in export / import), not on the SpeechAction DSL contract. The server generation path has no AudioFileRecord and is left to a later slice. - Add dependency-free `measureAudioDuration` (WAV RIFF walk + MP3 Xing/CBR estimate); pure byte parsing, no DOM, returns null for unparseable input so callers degrade gracefully. - Client path (`generateAndStoreTTS`): populate AudioFileRecord.duration. - Unit tests for WAV/MP3 parsing, format dispatch, and degradation. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(audio): sniff content before trusting format hint; add VBRI + ID3v1 handling Address PR #862 review: - [P2] measureAudioDuration now sniffs magic bytes first and only falls back to the reported `format` hint when the content is unrecognizable. getAudioResponseFormat defaults to `mp3` on a missing Content-Type, so a WAV clip could reach the MP3 parser and false-sync into a wrong duration; content now always wins over a mismatched hint. - [P3] Implement the VBRI frame-count path the comment already claimed. - [P3] Exclude a trailing ID3v1 tag from the CBR byte-count estimate. - Add regression tests: mismatched hint (both directions), Xing/Info frame count, VBRI frame count, trailing ID3v1. 17 passing (was 11). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: wyuc <wang-yc24@mails.tsinghua.edu.cn> |
||
|
Yanpeng Wang
|
d54e62f5c4 |
feat(token-plan): one-click token-plan setup + deployment usage dashboard (#784)
* feat(usage): add usage normalization, pricing, model-fetch, balance, storage Foundational layer for token-plan usage tracking (cc-switch-modeled): - lib/usage/normalize.ts: AI SDK v6 usage → four-class token shape - lib/usage/pricing.ts + defaults: per-class USD pricing table - lib/server/model-fetch.ts: /models candidate-URL multi-fallback (ported) - lib/usage/balance-providers.ts: built-in balance queries + detection - lib/server/usage-storage.ts: fire-and-forget jsonl logging to data/usage All pure/storage logic covered by vitest (32 tests). * feat(usage): capture token usage at the callLLM/streamLLM chokepoint - callLLM records result.usage before returning - streamLLM wraps onFinish to record totalUsage on stream completion, preserving any caller-supplied onFinish - provider/model derived from the model instance (no route changes) - fs-backed storage imported dynamically; fire-and-forget, never throws - include_usage is already sent by @ai-sdk/openai, so streaming is covered * feat(usage): add probe-models, balance, and usage API routes - POST /api/provider/probe-models: discover chat models via /models with candidate fallback; SSRF-guarded; filters non-chat ids; 401/404 typed - POST /api/provider/balance: built-in balance detection + billing fallback - GET /api/usage: aggregate jsonl by model/day/source with costIncomplete flag Verified e2e against the live MAIC gateway: 16 chat models (6 filtered), balance detected, and a real callLLM writes a costed usage row. * feat(settings): add token-plan preset picker to provider dialog - lib/config/token-plan-presets.ts: data-driven vendor presets (Huawei/MiniMax/ Xiaomi token plans, OpenRouter/SiliconFlow gateways, DeepSeek/GLM/Qwen/Hunyuan/ Doubao direct) with baseURL, protocol, optional modelsUrl, category - add-provider-dialog: '选择厂商/自定义' tabs; picking a preset auto-fills baseURL+protocol+modelsUrl, custom tab unchanged - ProviderSettings.modelsUrl carries the optional /models override - i18n keys added across all 8 locales Verified in-browser: preset picker renders grouped by category. * feat(settings): add Fetch Models button and balance bar to provider panel - 拉取模型: probes /models, merges discovered ids into the model list (dedupe, keeps manual additions), with success/no-endpoint/auth messages - 查询余额: queries /api/provider/balance, renders a balance bar or a 'check console' hint when unsupported - index.tsx: handleModelsFetched merges probe results into provider config - i18n keys across all 8 locales; removed a now-unused eslint-disable Verified in-browser against MAIC gateway: 16 models fetched, balance shown. * feat(settings): add usage dashboard to System Settings - usage-dashboard.tsx: echarts dual-axis daily trend (tokens + cost), totals cards, by-model table, refresh; reads GET /api/usage - mounted at the top of GeneralSettings (系统设置) - honest disclaimer + costIncomplete marker when a model lacks pricing - i18n keys across all 8 locales Verified in-browser: shows 1 request / 31 tokens / $0.0003 from a prior call. * feat(token-plan): multi-modal one-click setup in System Settings - token-plan-presets.ts: presets now declare per-modality targets (llm/image/video/tts/webSearch); MiniMax is the full-set template, others LLM-only — extend by adding entries (at-our-best adaptation) - apply-token-plan.ts: fills one key into every declared modality via injected store setters, isolating per-modality failures (TDD, 4 tests) - token-plan-settings.tsx: new sidebar page — pick plan, enter key, one-click apply lights up adapted modalities (+LLM model probe), shows 'not adapted yet' for the rest, balance bar reused - reverted add-provider dialog to plain custom form (preset picker moved here) - i18n across all 8 locales Verified in-browser: Token Plan page renders, MiniMax shows all 5 modalities, apply/balance UI wired. 859 tests pass, build clean. * feat(token-plan): add custom token plan entry to Token Plan page - Custom card at the bottom of the provider list: pick it to manually enter name + protocol + baseURL (LLM-only), then key + one-click apply via the same applyTokenPlan flow (mirrors cc-switch's custom provider) - effectivePreset unifies preset vs custom for apply/balance/probe - i18n keys (customGroup/customName/customHint) across all 8 locales Verified in-browser: custom card expands the manual form; preset flow intact. * feat(token-plan): reflect persisted config on the Token Plan page Other settings panels read the store directly, so they survive section switches; the Token Plan page only wrote the store, so it looked blank on return. Now it also reads providersConfig: - selecting a preset prefills its saved API key - configured presets show a '已配置/Configured' badge State persistence was never broken (apply writes zustand→localStorage); this fixes the missing read-back so the page reflects it. * fix(settings): isLLMProviderConfigured crashed on providers without models Root cause of 'token plan config disappears': applyTokenPlan writes a new LLM provider with no models yet (probe fills them later). isLLMProviderConfigured did config.models.length unguarded → threw inside setProviderConfig's resolver → the whole set() aborted → key/baseUrl/type never persisted. - guard models in isLLMProviderConfigured (shared validator; affects any provider written without a models array) - seed models:[] in applyTokenPlan's LLM write for a valid initial shape - regression tests: validator no longer throws; apply+probe write keeps apiKey Verified in-browser: DeepSeek persists key and shows 已配置 after section switch. * feat(token-plan): add remove/teardown for a configured token plan Apply had no inverse. Add removeTokenPlan: clears the API key and disables (enabled:false) every modality the plan declared; a custom LLM provider is deleted entirely (removeProvider), built-ins keep the cleared shape. - removeTokenPlan in apply-token-plan.ts (mirrors applyTokenPlan, isolated per-modality, injected setters; 3 tests) - trash button on configured preset cards in the Token Plan page; resets page state if the removed plan was selected - i18n 'remove' across all 8 locales Verified in-browser: removing DeepSeek empties its key and drops the 已配置 badge. * feat(usage): track multimodal usage, drop cost/pricing Reframe usage stats as pure usage (no cost), per user decision: - usage-storage: drop all cost fields; add kind (llm/image/video/tts/asr) + quantity + unit; LLM keeps token counts, others store quantity - delete lib/usage/pricing.ts + pricing-defaults.json + its test - instrument image/video/tts at the server-only API routes (not in the provider dispatch — those files are in the client graph and importing fs-backed usage-storage broke the client bundle) - /api/usage: aggregate by model/day/modality, no cost - dashboard: per-modality usage (tokens/images/seconds/chars), token-only daily trend, no '$'; i18n cost keys replaced with modality/unit keys - backward compatible: legacy rows (no kind, stray cost fields) read as llm 912 tests pass, build clean. * feat(usage): per-modality dashboard layout + softer dark-mode chart - group usage into per-modality sections (LLM/image/video/tts/asr), each table's usage column uses one consistent unit (token/image/sec/char) — no more mixed units in a single column - summary chips per modality with their own unit - trend chart now plots daily REQUESTS (unit-agnostic, works for any modality) instead of LLM-only tokens - theme-aware chart: faint thin line + soft gradient area, muted axis/grid colors via useTheme — fixes the harsh solid stroke in dark mode Verified in-browser (dark): TTS shows '字符', LLM shows 'Token', separate sections; chart no longer has a hard line. * refactor(usage): dedupe model/fetch/usage helpers, parallel balance probe Review cleanups on the token-plan/usage branch: - extract modelInfoFromId() (shared vision heuristic + ModelInfo shape) - extract fetchWithTimeout() shared by model-fetch and balance-providers - extract recordGenerationUsage() to dedupe the image/tts/video routes - queryBalance: fetch billing subscription + usage in parallel - parseOneApiBilling: report quota without remaining when usage endpoint is unavailable, instead of implying zero spend (full balance) - split the two jammed imports in settings/index.tsx * feat(token-plan): drop custom token-plan support Custom token plans (manual baseURL/protocol entry) added complexity for little gain — a one-off provider is better configured directly on the Providers page. Token Plan is now preset-only: - remove custom mode, manual fields, and the custom card from the UI - collapse effectivePreset back to the selected preset - drop the now-dead removeProvider action + custom-id branch in removeTokenPlan - remove orphaned customGroup/customName/customHint i18n keys (8 locales) * feat(token-plan): add Volcengine/Tencent/Bailian plans, drop balance feature Add three vendor token-plan presets (all map to existing built-in LLM providers, so it's data-only — no new adapters): - 火山方舟 Volcengine Ark → doubao, OpenAI /api/v3 - 腾讯 TokenHub Token Plan → tencent-hunyuan, OpenAI /plan/v3 (the plan-specific base; /v1 is the pay-as-you-go gateway) - 阿里百炼 Token Plan → qwen, cross-model plan (Qwen + DeepSeek/Kimi/ GLM/MiniMax) on one key; model list is probed/entered Remove the balance/quota feature entirely — we now track usage, not cost, and every vendor's balance query needs its own cloud AK/SK + signature (Volcengine SigV4 / Tencent TC3 / Aliyun BSS), which the Bearer-key billing-endpoint probe never supported anyway: - delete lib/usage/balance-providers.ts, /api/provider/balance, its test - strip the Check Balance button + balance bar from the token-plan page and the provider config panel - remove the 4 balance i18n keys across all 8 locales - restore an eslint-disable the branch had dropped in provider-config-panel * feat(token-plan): use cloud-brand logos for vendor token plans The three vendor plans are cloud offerings, not single-model products, so icon them with the cloud brand rather than a model logo: - 火山方舟 → volcengine.svg (was doubao.svg) - 腾讯 TokenHub → tencentcloud.svg (was hunyuan.svg) - 阿里百炼 → alibabacloud.svg (was bailian.svg) Logos are the colored brand variants from lobehub/lobe-icons, matching the existing colored-logo style (plain <img>, no dark:invert needed). * feat(token-plan): keep only MiniMax and Volcengine presets Trim the token-plan list to the two we want to ship: MiniMax (full-set template) and 火山方舟 Volcengine Ark. Drop the Tencent/Bailian plans and the OpenRouter/SiliconFlow/DeepSeek/GLM/Qwen entries. - remove the now-unused tencentcloud.svg / alibabacloud.svg logos (volcengine.svg stays; the other logos are still used by the provider registry) - retarget the LLM-only apply test from the deleted deepseek preset to volcengine-ark * feat(token-plan): restore aggregator/third-party presets Previous commit over-trimmed: the intent was to drop only the Tencent and Bailian token plans, not the OpenRouter/SiliconFlow/DeepSeek/GLM/Qwen entries. Bring those back; keep only Tencent/Bailian removed. - token_plan: MiniMax, 火山方舟 Volcengine Ark - aggregator: OpenRouter, SiliconFlow - third_party: DeepSeek, GLM, Qwen Revert the apply test back to the deepseek fixture (restored). tencentcloud.svg / alibabacloud.svg stay deleted (their plans are gone). * fix(token-plan): point Volcengine plan at the Coding Plan endpoint The plan's ark--prefixed API keys authenticate only against /api/coding/v3, not the general /api/v3 endpoint — the latter rejects them with "The API key format is incorrect", so model probing returned nothing. Switch the base URL to https://ark.cn-beijing.volces.com/api/coding/v3. * fix(token-plan): Volcengine is an Agent Plan (Anthropic /api/plan) Per the Ark Agent Plan docs, the ark--prefixed keys authenticate ONLY against the dedicated Anthropic-compatible base https://ark.cn-beijing. volces.com/api/plan ("其他 Base URL 无法在 Agent Plan 中使用"). The general /api/v3 and the Coding Plan /api/coding endpoints both reject the key as "API key format is incorrect", which is why model probing kept returning 0. - baseUrl → https://ark.cn-beijing.volces.com/api/plan/v1 (the /v1 lets the Anthropic SDK land on /api/plan/v1/messages) - apiFormat → anthropic - rename to 火山方舟 Agent Plan Probe still targets /api/plan/v1/models (the path exists); if the Anthropic gateway doesn't return an OpenAI-shaped list, users fall back to typing a model id like ark-code-latest. * fix(token-plan): Volcengine Agent Plan = OpenAI /api/plan/v3 + ark-code-latest Settled after probing the real key and reading cc-switch's approach: - The ark- plan key works on the OpenAI-compatible /api/plan/v3 endpoint (chat/completions returns 200); switch apiFormat back to openai. - The plan exposes NO /models list (every /api/plan/*/models is 404), which is why probing kept returning 0. cc-switch handles this by hardcoding a single ark-code-latest (an auto-routing alias valid on any tier) and does NOT use AK/SK for model listing — so we do the same. - Seed defaultModels: ['ark-code-latest'] only; users add specific ids by hand. Supporting machinery (kept, general-purpose): - applyTokenPlan seeds models from defaultModels instead of wiping to [] - handleApply uses defaultModels and skips the doomed probe when present - drop stray .playwright-mcp/ debug artifacts and gitignore them * style: fix prettier formatting in usage files CI runs prettier on the whole repo (prettier . --check); these four files predate this branch's formatting pass and tripped the check. * feat(token-plan): verify Volcengine Agent Plan's published model set The Agent Plan publishes a fixed model set but exposes no /models endpoint, so carry the documented models as CANDIDATES and verify each on apply: - add verifyModels flag to TokenPlanModalityTarget - new /api/provider/probe-chat-models route: sends a minimal chat request per candidate (OpenAI /chat/completions or Anthropic /messages) in parallel, returns the subset that succeeds; SSRF-guarded, auth-failure short-circuits - handleApply gains a verify branch (before the fixed-defaultModels fast path), falling back to the seeded list if verification fails - Volcengine preset now carries the 12 published Agent Plan text models (doubao-seed-2.0-*/deepseek-v4-*/minimax-m*/glm-5.2/kimi-k2.*) as candidates This auto-prunes retired (docs flag deepseek-v3.2/glm-5.1 as 即将下线) and tier-gated models without code changes. Verified all 12 resolve against a real plan key. * feat(token-plan): wire Volcengine Agent Plan image + video modalities Make the Ark seedream/seedance adapters path-configurable and light up the image/video modalities on the Volcengine plan: - seedream/seedance adapters: resolveArkRoot() uses baseUrl verbatim when it already carries an /api/... path (token plan's /api/plan/v3), else appends the standard /api/v3 — no regression for the pay-as-you-go default host. - applyTokenPlan: image/video branches inject a modality's defaultModels as customModels and set them as the active provider+model, so generation works out of the box. New optional setImageProvider/ModelId + setVideoProvider/ ModelId actions (UI passes the store setters; tests omit them). - Volcengine preset declares image (doubao-seedream-5.0-lite, verified 200 on /api/plan/v3/images/generations) and video (doubao-seedance-2.0/1.5-pro — Medium+ tiers only; lower tiers reject at call time, no code change needed to upgrade). Applying the plan overwrites the shared seedream/seedance slot with the plan config (same overwrite model as LLM); switching back to pay-as-you-go is a manual edit or plan removal. Verified image end-to-end with a real plan key. * feat(token-plan): verify image/video models on apply, disable unsupported tiers The Volcengine plan lit up video optimistically, but lower tiers (Small) don't include video — so using it 404'd with UnsupportedModel. Probe media models on apply and only keep what the tier actually supports: - generalize /api/provider/probe-chat-models with a `kind` (chat|image|video): image hits /images/generations, video hits /contents/generations/tasks with empty content. The model-support check (404 UnsupportedModel) runs before any billable work, so probing never starts a real image/video job; for media, "supported" = any non-404 response. - handleApply: after lighting up image/video, probe each verifyModels modality; prune to the verified model set + re-select a working model, or disable the modality entirely if none pass (no false "available"). - Volcengine preset: image/video targets gain verifyModels: true. - add settings.tokenPlan.tierUnsupported across 8 locales. Verified with a real Small-tier key: image (seedream-5.0-lite) passes and is kept; video (seedance-2.0/1.5-pro) 404s and is disabled. * feat(web-search): add Doubao (豆包搜索) provider Doubao Search (Custom 版) over its REST endpoint POST open.feedcoopapi.com/search_api/web_search with Bearer auth — the same endpoint the askecho-search-infinity MCP server wraps, so the Volcengine Agent Plan key authenticates directly. Mirrors the MiniMax adapter: maps Result.WebResults to WebSearchSource (prefers Summary, the query-relevant excerpt, over Snippet for LLM use) and surfaces errors from ResponseMetadata.Error. - register 'doubao' in WebSearchProviderId + WEB_SEARCH_PROVIDERS - searchWithDoubao adapter, searchWeb dispatch, store default config - SSRF allowlist entry for the search host * feat(audio): support Agent Plan single-key auth for Doubao TTS generateDoubaoTTS now picks auth + endpoint from the key shape, since Volcengine exposes Seed-TTS as two products with separate credentials (verified: a plan key 401s on the normal endpoint, and the plan endpoint rejects appId-style auth): - single key (no colon) -> X-Api-Key, for the Agent Plan /plan endpoint - appId:accessKey -> X-Api-App-Id + X-Api-Access-Key (unchanged) A malformed pair (empty half) fails clearly instead of sending an empty header. Reuses the existing NDJSON/base64-mp3 parsing and voice list. * fix(media): map MiniMax video 720p to its real 768P tier normalizeVideoOptions defaults minimax-video to '720p' (the first supported resolution), but Hailuo 2.3 only accepts 768P/1080P and rejects 720P with '2013 ... does not support resolution 720P'. MiniMax's mid tier is 768P, not 720P (the adapter already falls back to 768P, as does the connectivity test), so map the shared enum's '720p' to 768P. Regression tests lock the mapping. * feat(token-plan): add web search + TTS to Volcengine Agent Plan, widen image tiers Extend the volcengine-ark preset now that the adapters exist: - webSearch -> doubao (own host open.feedcoopapi.com, not the ark endpoint) - tts -> doubao-tts on the /api/plan/tts endpoint (single-key auth) - image defaultModels widened to a best-first Seedream 5.0/4.5/4.0 list so a higher tier keeps the strongest model while verifyModels prunes the rest; video keeps the 2.0 + 1.5-pro candidates Comments record the verified host/auth quirks of each modality. * feat(token-plan): show result panel only after probing, with two clear states Addresses review feedback that a green check implied generation works when it only meant 'configured'. The panel now renders after probing finishes (gated on results && !applying) so it reflects the final set, and uses two states: green when the modality is configured/usable, muted when a live probe proved it unavailable (e.g. video on a tier without it). * feat(token-plan): scope presets to true multi-modal token plans Drop the single-modality LLM presets (OpenRouter, SiliconFlow, DeepSeek, GLM, Qwen) from Token Plan. A token plan's defining trait is one key spanning many modalities; those entries are ordinary LLM API providers already covered by the add-provider flow, and listing them here muddied the 'one key, every modality' promise. Only MiniMax and the Volcengine Ark Agent Plan remain. The UI already hides categories with no entries. apply-token-plan's LLM-only test now uses a local fixture instead of the removed deepseek preset. * feat(token-plan): progressive reveal of probe results on apply The result panel previously rendered all at once after probing finished, reading as dead air during the model probe. Now rows appear immediately on Apply: modalities with a live probe in flight show a spinner ('pending') and resolve to lit/failed independently as each probe returns, while non-probe modalities show lit right away. Probes run in parallel (Promise.all) instead of sequentially. A row only turns green once its own probe confirms, so this reveals structure + live progress without a premature green — complementing the earlier 'render only after probing' intent rather than reverting it. Per review feedback from @wyuc on #784. * fix(token-plan): enrich seeded models with built-in thinking capability Token Plan built ModelInfo objects from probed ids via modelInfoFromId(), filling only streaming/tools/vision — so a model that supports configurable thinking lost capabilities.thinking and InlineThinkingControl was hidden. modelInfoFromId now takes an optional providerId and overlays the catalog thinking capability for that (provider, model) pair; applyTokenPlan does the same for its synchronously-seeded list. Added the Ark Agent Plan's dotted aliases to the metadata table: - native Doubao Seed 2.0 family (doubao-seed-2.0-pro/code/lite/mini) - cross-vendor models the plan serves through its OpenAI-compatible endpoint (deepseek-v4-pro/flash, glm-5.2, kimi-k2.7-code/k2.6, minimax-m3/m2.7, ark-code-latest) All verified against a live plan key: each accepts the gateway's unified reasoning_effort field (low/medium/high) and actually reasons. They share the doubao effort adapter, which disables via 'minimal' (not 'none') — matching what the plan endpoint accepts (it rejects reasoning_effort:'none'). Addresses review point #1 from @wyuc on #784. * Improve token plan capability setup UI * fix token plan setup flow * chore: prettier format tts-providers.ts |
||
|
|
6fb87e7d0b |
fix(tts): respect string context when splitting the Doubao stream (#677)
Doubao streams a run of concatenated JSON objects with no delimiter. The
splitter counted `{`/`}` braces without tracking whether it was inside a
string literal, so a brace in a string value — e.g. an error
`{"message":"bad {input}"}` — mis-aligned the object boundaries, dropping
the chunk (lost audio, or a swallowed error).
Extract a string-aware `splitConcatenatedJsonObjects` helper (mirrors the
scanner in json-repair.ts: tracks inString + escapes) and use it. Behaviour
is otherwise unchanged: parse failures are skipped, code 20000000 ends the
stream, rate-limit/error codes still throw.
Closes #676
Co-authored-by: ly-wang19 <ly-wang19@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: wyuc <wang-yc24@mails.tsinghua.edu.cn>
|
||
|
杨慎andwyuc
|
0f44e9a46f |
Add PBL v2 runtime APIs and classroom UI (#799)
* feat: add PBL v2 runtime UI * fix: ship PBL v2 instructor avatar * fix: ship PBL v2 workspace logo * fix: prevent scenario PBL legacy fallback * chore: restore package lint comments * test: cover ordinary PBL legacy fallback * fix: front-load PBL single-call validation * fix: preserve PBL v2 content when assembling scenes --------- Co-authored-by: wyuc <wang-yc24@mails.tsinghua.edu.cn> |
||
|
wyuc
|
e83db6a531 |
fix(agents): persist AgentBar voice picks and mode/selection across reloads (#723)
* feat(audio): let resolveAgentVoice honor per-agent voice overrides * feat(settings): persist per-agent voice overrides * fix(agents): persist AgentBar voice picks via settings overrides * fix(classroom): keep user's agent mode and selection across loads * fix(agents): only carry user-set agent selection across classrooms Review follow-ups: stage-derived defaults written by a classroom load were re-read as a user choice on the next load, so visiting a preset classroom permanently downgraded auto classrooms to preset agents. Track the origin with a persisted agentSelectionIsUserSet flag (set by AgentBar interactions, cleared by fallback writes). Also: voice overrides are now clearable (undefined deletes), the override type is single-sourced in voice-resolver, and persist rehydration of the new fields is pinned by tests. * fix(agents): don't mark stage-derived selections as user choices Round-2 review: clicking the already-active mode tab no longer flags the selection as user-set (a visual no-op must not promote stage defaults to a choice), generation-preview clears the flag when it writes the generated roster, and tests now pin that setter writes actually reach storage. * fix(agents): restore generated agent selection when switching back to Auto Review follow-up (#723): handleModeChange('auto') left the preset ids selected, so the UI said Auto while discussion playback kept using preset agents, and the persisted user-set auto pair could never validate on restore. Switching to Auto now selects the loaded classroom's generated agents (empty when none are loaded, falling back to stage defaults on the next classroom load). |
||
|
|
d7c0d80afe |
feat(tts): per-agent auto-voice quality + register-once timbre stability (#670) (#672)
* feat(tts): voxcpm voice-design types + deterministic voice id helpers (#670) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(tts): emit + persist per-agent voiceDesign descriptor (#670) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(tts): voxcpm voice registration backend client (#670) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(tts): voxcpm-voice ensure/register endpoint (#670) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(tts): client auto-voice registration + reference-clip cache (#670) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(tts): reference registered voice id in vLLM-Omni speech (#670) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(tts): thread voiceDesign + backend through tts call sites (#670) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(tts): match vLLM-Omni voice registration contract (#670) Live e2e against the real backend revealed the assumed multipart contract was wrong: POST /v1/audio/voices needs name + consent + audio_sample (not voice_id/file), and there is no per-name GET (405) — existence must list /v1/audio/voices and check membership. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * refactor(tts): make auto-voice register-once pattern provider-neutral (#670) The descriptor + register-once/reference-by-id pattern is not VoxCPM-specific. De-couple it into a provider-neutral seam so other registration-capable providers (ElevenLabs/MiniMax/Doubao voice cloning, …) can plug in: - lib/audio/voice-design.ts: VoiceDesign + buildVoiceDesignPrompt / normalizeVoiceDesign / getDeterministicVoiceId (neutral 'auto-<hash>' id, namespaced by providerId). - lib/audio/voice-registration.ts: VoiceRegistrationAdapter interface + providerId->adapter registry + supportsVoiceRegistration. - lib/audio/voice-registration-client.ts: neutral ensureRegisteredVoice() + IndexedDB clip cache. - app/api/generate/voice (replaces .../voxcpm-voice): dispatches by providerId. - voxcpm-registration.ts becomes the VoxCPM adapter (sole registered provider). - AgentConfig.voiceDesign + DB cache table renamed neutral. Behavior-preserving; VoxCPM-specific bits (inline (prompt)text, backend kinds, capability gate) stay in the voxcpm modules. Full suite green (674). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(tts): build the voice clip Blob from an inline Uint8Array (#670) A factored-out helper returning Uint8Array widened to Uint8Array<ArrayBufferLike>, which next build (stricter than bare tsc) rejects as a BlobPart. Inline the buffer like the rest of the repo does. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(tts): managed TTS providers resolve model server-side (#670) A server-managed TTS provider's model was still client-driven, but the managed-provider settings UI hides the model field — so a backend whose model id isn't the client default (e.g. VoxCPM/vLLM-Omni serving the full model path) always 500'd. resolveTTSModel() makes the model authoritative from server config (${PREFIX}_MODELS, first entry) for managed providers, like key/baseUrl; unmanaged/unconfigured providers keep the client model unchanged. Used by the tts and voice routes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(tts): single agent-voice resolver + stable teacher narration voice (#670) Route all TTS voice resolution (narration, discussion, preview) through one resolveAgentVoiceOptions(agent, ...) that reads the agent profile and registers + references the voice by id. Eagerly warm up generated agents' voices on save. Critically fixes the teacher narration drifting (male/female jumps): the registry is always seeded with DEFAULT_AGENTS, so the old narration lookup find(role==='teacher') returned the default teacher (no voiceDesign) instead of the generated one — so narration never registered a voice and fell back to the inline prompt. pickNarratorAgent() now prefers the teacher carrying a voiceDesign. Also: drop language from the deterministic voice id (descriptor already encodes it) so narration (directive) and discussion (locale) resolve the same id; log the effective registeredVoiceId. Regression tests for pickNarratorAgent. Verified e2e: agent-profiles -> 1 voice registration -> 16 narration TTS all referencing the same registeredVoiceId; voice present on the backend. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * feat(tts): fall back to persona as the voice seed for agents without a voiceDesign (#670) Preset/default agents carry no LLM voiceDesign. Rather than add a static field, derive the bootstrap descriptor from the agent's persona when voiceDesign is absent, so they still register one stable reference voice (stable-but-generic; persona is not a vocal spec). Generated agents keep their LLM voiceDesign. Also hardens replay: stage snapshots drop voiceDesign but keep persona. Unit-tested: resolveAgentVoiceOptions uses real voiceDesign when present, persona otherwise. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(tts): match Auto Voice by its localized label in the agent voice picker (#670) The picker filtered on voice.name ('Auto Voice'), but Auto Voice is shown via its localized label (自动音色). Searching '自动' returned '没有匹配音色'. Match the localized label for the Auto Voice option so it's findable in any language. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(tts): address code-review round 1 (#670) - strip parentheses in the voice-design prompt so a paren in the descriptor/ persona can't corrupt the (prompt)text bootstrap delimiter - gate POST /api/generate/voice on isServerTTSProviderDisabled (#665): a force-disabled provider was off for the tts route but not this sibling - check voiceExists before re-registering a client-cached clip, so a cached voice that's still live isn't needlessly re-uploaded every session - dedup concurrent ensureRegisteredVoice calls via an in-flight promise map (eager warm-up + first utterance no longer double bootstrap/register) - warm up only the narrator (teacher), not every generated agent, to avoid synthesizing voices at save time for agents that may never speak - collapse the duplicated vLLM-Omni speech payload into one object Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(tts): prettier formatting (#670) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(tts): key auto-voice memo by (voiceId, backend) so a base-URL change re-registers (#670) Addresses review (cosarah): registeredThisSession/inFlight were keyed by voiceId alone, so switching the VoxCPM base URL mid-session made ensureRegisteredVoice short-circuit and return an id registered only on the old backend — TTS then sent that stale registeredVoiceId and skipped the inline fallback, failing with voice-not-found on the new backend. Memo key now includes the base URL; the IndexedDB clip cache stays keyed by voiceId (the reference clip is backend-independent and reused to re-register elsewhere). Regression test added. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(tts): include API key in the auto-voice memo key (#670) Addresses review (cosarah, round 2): memoKeyFor was keyed by (voiceId, baseUrl) but not the API key. Registration/existence checks and speech calls are auth-scoped, so switching account/key on the same base URL could reuse a registeredVoiceId from the old credentials and skip re-validation. Memo key now includes the API key (in-memory only, never persisted/logged). Regression test covers the key-switch case. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: 杨慎 <117187635+cosarah@users.noreply.github.com> |
||
|
|
9248c385f4 |
fix(audio): revoke blob URL when audio.play() rejects (#652)
In the IndexedDB playback branch, the object URL created via URL.createObjectURL is revoked only inside the 'ended' listener. When audio.play() rejects (autoplay policy, decode error, interrupted load) that listener never fires, so the blob URL — and its backing audio blob — leaked for the lifetime of the document. During lecture playback play() is called once per speech action, so repeated failures (e.g. autoplay blocked before the first user gesture, or rapid stop/skip) accumulate. Revoke the URL when play() rejects, then re-throw. Adds a regression test. Co-authored-by: ly-wang19 <ly-wang19@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: wyuc <wang-yc24@mails.tsinghua.edu.cn> |
||
|
|
3e1e9f083b |
feat(tts): unify provider-enablement model, default browser-native off (#665) (#666)
A single TTS provider-enablement model honored everywhere (picker,
auto-assignment, fallback, lecture/playback, server).
Model
- lib/audio/provider-enablement.ts: isTTSProviderConfigured /
isTTSProviderEnabled / listEnabledTTSProviderIds. Availability drops the
registry defaultBaseUrl fallback (fixes Lemonade's always-shown voices),
honors the per-provider `enabled` flag, server-disable overrides (server
precedence).
- voice-resolver: getEnabledProvidersWithVoices (enabled server/custom
providers) and getSelectableProvidersWithVoices (the ONE client source of
truth = enabled providers + opt-in browser-native when the browser exposes
voices). resolveAgentVoice picks deterministically among the given providers,
honors a voiceConfig only while its provider is present.
- Picker + discussion build their list from getSelectableProvidersWithVoices, so
the AgentBar and the live discussion agree on selectable voices (fixes
student agents going silent in auto mode). Students resolve via
resolveAgentVoice; the teacher uses the global lecture voice verbatim (same
voice the pre-generated lecture sends), so lecture and discussion never
diverge.
- Voice assignment vs fallback: an agent's voice is bound at generation (the
agent-profile LLM picks from availableVoices; preset uses the user's pick).
The fallback for a missing/invalid/unavailable voice is applied only at the
live TTS calls, not baked into the saved agent — so it reflects current
provider availability.
Defaults / opt-in
- Browser-native: first-class, default OFF, opt-in. Built-in providers default
enabled:true (only surface once configured).
- TTS master toggle defaults OFF; first server-sync auto-enables it when a
provider is configured (mirrors image/video).
- Persist v3->v4 migration normalizes the never-user-editable flags.
Server/admin disable (server precedence): server-providers.yml `enabled:false`
and env `TTS_<P>_ENABLED=false` (incl. browser-native). Surfaced as a per-provider
`disabled` flag on /api/server-providers; merged as serverDisabled; enforced at
the /api/generate/tts trust boundary, classroom media generation, and /api/health.
UI
- Settings TTS panel: per-provider enable toggle, disabled (with a hint) when the
provider is unavailable, locked off (with a note) when server-disabled.
- AgentBar voice pills render disabled with a "No voice" placeholder when no
provider is enabled. The toolbar media-popover TTS tab keeps its existing
on/off toggle (no picker/CTA).
Tests: predicates, resolver determinism + unified selectable list, server
force-disable, store merge + migration + TTS auto-enable, i18n parity. Verified
in a real browser (Playwright): settings toggle states + a full Gemini classroom
generation whose TTS requests used the picked voice.
Found on main @
|
||
|
|
2586cb5e05 |
fix(tts): surface provider rate-limit errors as HTTP 429 (#644)
TTS providers throw a plain `Error` on any non-OK response, and the `/api/generate/tts` route catches everything as `GENERATION_FAILED` (500), so an upstream 429 (quota/concurrency) was indistinguishable from a real failure — clients had no signal to retry/back off. `TTSRateLimitError` already existed for exactly this (with a TODO noting the route collapses everything uniformly), but only Doubao threw it and the route never mapped it. - Add a `RATE_LIMITED` API error code. - Map upstream HTTP 429 to `TTSRateLimitError` at every provider's `!response.ok` site via a shared `throwIfTtsRateLimited` helper. - Return `429 RATE_LIMITED` from the TTS route when a `TTSRateLimitError` is caught (all other errors unchanged). Purely additive: non-429 responses keep their existing error messages. Co-authored-by: ly-wang19 <ly-wang19@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: wyuc <wang-yc24@mails.tsinghua.edu.cn> |
||
|
|
be759a8252 |
feat: add lemonade to provider (#508)
* feat: add lemonade to provider * fix(lemonade): correct TTS/ASR model IDs and expand Kokoro voices - TTS model: kokoro → kokoro-v1 - ASR model: single 'whisper' → 6 Whisper variants (Base/Large-v3/Large-v3-Turbo/Medium/Small/Tiny), default Whisper-Base - Expand lemonade-tts voices to full Kokoro v1 set (54 voices across en-US/en-GB/zh-CN/ja-JP/es/fr-FR/hi/it-IT/pt-BR) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(lemonade): zh-TW i18n parity, real Kokoro default voice, language code consistency - zh-TW.json: add missing providerLemonadeTTS/ASR/Image keys - TTS default voice: 'default' → 'af_heart' (Kokoro expects real voice ID, not placeholder) - Voice language codes: 'es'/'hi' → 'es-ES'/'hi-IN' for consistency Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test(lemonade): add unit tests for image/TTS/ASR adapters and wav-utils Covers: - lib/media/adapters/lemonade-image-adapter: request shape, base URL fallback, custom model, optional Bearer auth, error paths, connectivity probe - lib/audio/tts-providers (lemonade-tts case): request body (model/voice/format), af_heart fallback, optional auth, error path - lib/audio/asr-providers (lemonade-asr case): WAV-only validation, model/language form fields, "audio is empty" graceful empty result, error propagation, default model fallback - lib/audio/wav-utils: isWavBlob detection by MIME and filename, normalizeASRUploadAudio pass-through and wav handling Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(lemonade): remove text path fork and harden fallbacks * style: format lemonade tts files * i18n: add lemonade voice language labels * fix(lemonade): align image default model and harden wav detection * test(lemonade): fix File BlobPart typing * fix(lemonade): simplify tts voice selection and clear asr test state * docs: highlight Lemonade support --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-authored-by: wyuc <wang-yc24@mails.tsinghua.edu.cn> |
||
|
wyucand杨慎
|
3c687dd523 |
fix(tts): unify model selection to per-provider and fix ElevenLabs model_id (#326)
* fix(tts): unify model selection to per-provider and fix ElevenLabs model_id (#324) - Move model selection from global ttsModelId to per-provider ttsProvidersConfig[providerId].modelId - Remove supportsModelSelection flag, use models.length > 0 instead - Add defaultModelId to TTSProviderConfig and ASRProviderConfig - Add providerOptions field to TTSModelConfig for future extensibility - Fix ElevenLabs hardcoded model_id: 'eleven_multilingual_v2' - Fix MiniMax TTS using config.model instead of config.modelId - Update model lists from provider docs: - Qwen: add qwen3-tts-instruct-flash, qwen-tts - ElevenLabs: add eleven_flash_v2_5, eleven_flash_v2 - MiniMax: remove non-existent speech-2.8 series - Replace custom model CRUD UI with simple Select dropdown - Remove MiniMax-specific hardcoded model selector from audio-settings - Add migration for old ttsModelId/asrModelId/model fields * feat(tts): voice picker groups by provider:model, sets both model and voice Voice selection in agent bar now groups voices by model. Picking a voice from a model group sets both modelId and voiceId on the agent. TTS settings shows available models as display-only chips. - Add compatibleModels field to TTSVoiceInfo (OpenAI marin/cedar restricted to gpt-4o-mini-tts) - Add modelId to agent voiceConfig and ResolvedVoice - Add ModelVoiceGroup to ProviderWithVoices, build groups in getAvailableProvidersWithVoices - Update AgentVoicePill and TeacherVoicePill to iterate modelGroups - Thread modelId from voice selection through discussion TTS pipeline - Replace model Select with display-only chips in tts-settings * fix(tts): show provider name in voice picker group headers * fix(tts): restore MiniMax speech-01 models per official API docs * fix(tts): add MiniMax speech-2.8 models per official docs, default to speech-2.8-hd * feat(tts): expand MiniMax voice list with recommended voices * fix(tts): remove non-official MiniMax recommended voices, keep 10 from official list * fix: address CR findings - align defaults, fix test mocks, ASR modelId fallback * fix(tts): pass modelId in voice preview and classroom generation * fix(tts): voice picker highlights only the exact model:voice match --------- Co-authored-by: 杨慎 <117187635+cosarah@users.noreply.github.com> |
||
|
wyucandHi-Jiajun
|
9badd59856 |
feat: add and expand MiniMax provider support (#182)
Add MiniMax LLM models (M2–M2.7), TTS provider (speech-2.8/2.6/02/01), image generation (image-01), and video generation (Hailuo 2.3/02, T2V-01). Unify base URLs to api.minimaxi.com and add MiniMax-specific base URL normalization for the Anthropic-compatible endpoint. Co-authored-by: Hi-Jiajun <3138947285@qq.com> |