mirror of
https://github.com/THU-MAIC/OpenMAIC.git
synced 2026-11-07 08:54:09 +08:00
* ci: run PR checks against the runtime-server-backend integration branch (#939) * feat(storage): RuntimeStore HTTP contract + HttpRuntimeStore client (#939 Part B) (#940) * feat(storage): RuntimeStore HTTP contract + HttpRuntimeStore client (#939 Part B) - documented JSON HTTP contract for every RuntimeStore operation (server-assigned seq, learnerKey derived from auth — never trusted from the request, machine-readable error codes, idempotency notes) - HttpRuntimeStore client with injected fetch + auth headers hook; session reads (including createSession responses) migrate forward on the runtime line so an older server cannot leak stale envelopes - conformance test server bridging the contract onto the browser backend; full runRuntimeStoreContract green over HTTP plus error mapping cases * fix(storage): harden HTTP runtime backend after cross-review (#939 Part B) Cross-review fixes: - appendRecord gates payloads through assertJsonValue: fail loud on values JSON cannot represent (Map, Date, NaN, nested undefined, NUL strings) instead of silently mangling them in transit; payload domain documented in the contract - appendRecord/listRecords responses validate via validateRuntimeRecord (server-assigned seq is no longer trusted verbatim) and listRecords sorts by seq like listSessions already sorted defensively - request headers no longer require an ambient Headers constructor when fetch is injected - conformance server classifies errors structurally (existence checks, DSL validators, version stamps) instead of regex-sniffing messages that interpolate caller-controlled ids - router preserves empty path segments so empty-key calls keep browser no-op semantics instead of shifting onto other routes - stray runtimeDslVersion/seq in request bodies are ignored (store- assigned wins), matching the browser reference; docs updated - loopback integration test exercises the real listening server - docs state the conformance server is test-only; auth-derived learnerKey is Part D's reference server * fix(storage): round-2 cross-review hardening for the HTTP backend (#939 Part B) - json-value guard rewritten: additionally rejects -0, symbol-keyed and non-enumerable own properties, and non-index own properties on arrays; U+2028/U+2029 are accepted again (they round-trip through JSON per RFC 8259 — rejecting them lost legitimate pasted text) - conformance server classifies racing duplicate creates as 409 via a structural post-check, gates payloads and merge learner keys as 400 - segment() rejects '.'/'..' ids the URL layer would fold away - list responses must be arrays and mergeLearner's moved must be a finite number, else a typed MALFORMED_RESPONSE error * fix(storage): round-3 cross-review hardening for the HTTP backend (#939 Part B) - json-value guard: NUL is rejected in object keys too, and unpaired UTF-16 surrogates are rejected in string values and keys (jsonb refuses both; other JSON stacks corrupt lone surrogates to U+FFFD) - typed storage errors survive non-object 200 responses instead of degrading into TypeErrors on .id access - mergeLearner's moved must be a non-negative integer - conformance server classifies missing/non-object request bodies as 400 VALIDATION_FAILED - createSession/appendRecord gate the full init envelope, not only the payload (stray Date/undefined properties, NUL in ids) * fix(storage): round-4 convergence fixes for the HTTP backend (#939 Part B) - envelope JSON gate tolerates explicitly-undefined optional anchors (sceneId: undefined behaves like omission, matching the browser) - body-carried identifiers reject '.'/'..' so nothing persists that the URL layer cannot later address; conformance server mirrors it - mergeLearner keys pass the JSON-domain gate on client and server - json-value guard v4: rejects enumerable accessors (validation/serialize TOCTOU), Array subclasses and null-proto arrays, prototype-supplied array indices; sparse-array test now constructs a genuine hole * fix(storage): round-5 convergence fixes for the HTTP backend (#939 Part B) - undefined-stripping is limited to the DSL-declared optional anchors (sceneId/actionIndex/subAnchor); any other undefined member fails the JSON gate loud instead of being dropped silently - json-value guard: array and object prototype checks are realm-agnostic (chain-shape instead of identity), so ordinary values from another realm are accepted while subclasses stay rejected; shared isLosslessJsonString predicate exported for SQL key guards - conformance server merge route validates target-key addressability * fix(storage): close the toJSON prototype channel in the JSON guard (#939) toJSON is the single channel through which a prototype can alter JSON output — prototype properties never serialize and own accessors are already rejected — so refusing any value with a callable toJSON closes prototype influence on serialization entirely, including prototypes crafted to pass the realm-agnostic chain-shape check. * fix(storage): probe toJSON via descriptor walk, not property read (#939) Reading value.toJSON would execute an inherited accessor, letting a stateful getter hide from validation and reappear at stringify time. The probe now walks own descriptors up the prototype chain without invoking any user code. Post-validation mutation of the caller's object graph is documented as out of scope — it is equally unpreventable for every other validated property. * fix(storage): toJSON probe mirrors JSON.stringify semantics exactly (#939) The own-most descriptor decides: absent is safe, a non-callable data value is an ordinary shadowing member and is safe, a callable data value is rejected, and an accessor is rejected because it cannot be inspected without invoking it. No daylight remains between the guard and the serializer on this property. * feat(storage): PgRuntimeStore — Postgres runtime backend (#939 Part C) (#941) * feat(storage): PgRuntimeStore — Postgres runtime backend over an injected queryable (#939 Part C) - PgRuntimeStore implements RuntimeStore over a minimal injected Queryable (node-postgres and PGlite both satisfy it) — the package keeps zero runtime dependencies beyond @openmaic/dsl - runtime_sessions / runtime_records schema exported as RUNTIME_PG_SCHEMA with idempotent ensureSchema() - appends serialize per session via a session-row lock; MAX(seq)+1 and the insert share one transaction, UNIQUE(session_id, seq) plus bounded retry backstop non-cooperating writers - envelope semantics mirror the browser backend: version stamping, validation gates, migrate-on-read, fail-loud on future-stamped rows - full runRuntimeStoreContract green on PGlite plus PG-specific cases (concurrent-append seq atomicity, ensureSchema and mergeLearner idempotence) * fix(storage): require pinned transactions + JSON payload gate in PG backend (#939 Part C) Cross-review fixes: - withTransaction is now required; the BEGIN/COMMIT fallback is removed (on a pg Pool it spread BEGIN/body/COMMIT across different connections — no real transaction, leaked idle-in-transaction clients, broken mergeLearner atomicity; on a shared pinned client concurrent calls interleaved transactions) - single-statement deletes no longer wrap in a transaction hook call - appendRecord gates payloads through assertJsonValue: fail loud on values JSON cannot represent (Map, Date, NaN, nested undefined, NUL strings) instead of silently persisting something different - append retry also covers 40001/40P01; READ COMMITTED assumption documented at the retry loop - ensureSchema documented as create-only; redundant stage index dropped - deterministic interleaving test proves the 23505 retry path; real PostgreSQL contract lane added (postgres:16 service workflow, pg driver suite skipped locally without PG_CONTRACT_URL) * fix(storage): round-2 cross-review hardening for the PG backend (#939 Part C) - json-value guard rewritten (shared with the HTTP backend): additionally rejects -0, symbol-keyed and non-enumerable own properties, and non-index own properties on arrays; accepts U+2028/U+2029 - storage-pg-contract workflow now triggers for the runtime-server-backend integration branch and fails loud (STORAGE_PG_CONTRACT_REQUIRED=1) when PG_CONTRACT_URL is missing instead of silently skipping - loadSession distinguishes corrupt non-object rows from absent rows so getSession fails loud instead of reporting the session missing * fix(storage): round-3 cross-review hardening for the PG backend (#939 Part C) - json-value guard (shared): NUL rejected in object keys, unpaired UTF-16 surrogates rejected in values and keys - createSession/appendRecord gate the full persisted envelope through assertJsonValue, not only the payload — stray Date/undefined properties and NUL in ids fail loud instead of silently diverging from the returned record or leaking raw 22P05 - real-PG lane covers a genuine 23505 conflict from a second connection and recovery after an aborted transaction - retry-set asymmetry and mergeLearner's unbounded lock set documented * fix(storage): round-4 convergence fixes for the PG backend (#939 Part C) - NUL/lone-surrogate lookup and delete keys resolve to absent/no-op instead of leaking 22021 driver errors; setSessionStatus reports the session missing; mergeLearner from-key moves 0 - mergeLearner destination key passes the JSON-domain gate fail-loud - envelope JSON gate tolerates explicitly-undefined optional anchors, matching the browser backend - json-value guard v4 (shared) + tests for accessors, Array subclasses, prototype-supplied indices * fix(storage): round-5 convergence fixes for the PG backend (#939 Part C) - appendRecord pre-checks the session key like every other lookup path, so a NUL/lone-surrogate sessionId reports 'no session' instead of leaking a 22021 driver error - the queryable-key predicate now structurally reuses the shared isLosslessJsonString export instead of restating the string rule - undefined-stripping limited to the DSL-declared optional anchors - json-value guard: realm-agnostic prototype checks (shared) * fix(storage): close the toJSON prototype channel in the JSON guard (#939) Shared guard change with the HTTP backend branch; adds the crafted null-proto-prototype regression test. Both final-gate reviewers independently converged on this same channel. * fix(storage): probe toJSON via descriptor walk, not property read (#939) Reading value.toJSON would execute an inherited accessor, letting a stateful getter hide from validation and reappear at stringify time. The probe now walks own descriptors up the prototype chain without invoking any user code. Post-validation mutation of the caller's object graph is documented as out of scope — it is equally unpreventable for every other validated property. * fix(storage): toJSON probe mirrors JSON.stringify semantics exactly (#939) The own-most descriptor decides: absent is safe, a non-callable data value is an ordinary shadowing member and is safe, a callable data value is rejected, and an accessor is rejected because it cannot be inspected without invoking it. No daylight remains between the guard and the serializer on this property. * fix(storage): conform HTTP/PG backends to the post-#926 RuntimeStore interface (#943) The chat cutover (#926) added deleteAllRuntime() to the RuntimeStore contract while the HTTP and Postgres backends were developed in parallel against the pre-#926 interface. Implements the method on both backends (single-statement wipe on PG via the FK cascade; DELETE /runtime on the HTTP contract, documented as an operator-gated administrative endpoint), restoring a green typecheck and contract suite on the integration branch. * feat(runtime): injectable RuntimeStore backend + learner-key provider (#939 Part A) (#944) * wip(runtime): backend injection seam — pending gate verification * fix(runtime): cross-review hardening for the storage injection seam - stage-deletion IndexedDB probe applies only to the default browser backend; an injected store always receives deleteStageRuntime - explicit kv argument takes precedence over the configured learner-key provider, matching the store seam's explicit-beats-global rule - configured learner-key resolution is latched with in-flight dedup, mirroring the store singleton; identity changes require app-level handling - factory retry semantics documented; seal errors explain the module-level bootstrap requirement; isRuntimeStorageConfigured() probe and a test-only reset added - client-bootstrap-only contract documented (SSR/HMR caveats) * fix(runtime): snapshot configuration and make the test reset complete - configureRuntimeStorage copies the option fields so mutating the caller's object after configuring cannot swap the sealed backend or identity provider - resetRuntimeStorageForTests now clears every latched consumer cache (store singleton, learner-key in-flight promise) via a reset-hook registry, so a reset-then-reconfigure test actually gets the new backend * fix(runtime): reset also clears the default learner-key caches resetRuntimeStorageForTests left defaultInFlight/defaultKv latched, so a default-path test could leak its anonymous key or KV store into the next test despite the documented full-reset promise. * feat(storage): runtime reference server — auth-derived learnerKey (#939 Part D) (#945) * wip(storage): reference server — pending deleteAllRuntime route + gates * feat(storage): runtime reference server — auth-derived learnerKey enforcement (#939 Part D) - createRuntimeHttpHandler(store, options) wires the documented HTTP contract onto any injected RuntimeStore over node http - authenticate is required; every learner-scoped operation verifies the path/body learnerKey against the authenticated principal (403 FORBIDDEN_LEARNER) — the client-supplied value is never trusted - mergeLearner requires an explicit authorizeMerge grant and admin surfaces (stage cascade, DELETE /runtime) require authorizeAdmin; both default-deny - runnable reference entry demonstrates a pg Pool withTransaction and bearer-token authentication, marked as demo-only - contract suite green through HttpRuntimeStore -> listening reference handler -> PgRuntimeStore(pglite); security matrix tested (401/403 paths, admin default-deny); threat model documented * fix(storage): cross-review hardening for the reference server (#939 Part D) - principal learnerKey is optional: admin/merge-only credentials no longer fabricate learner identity; learner-scoped routes 403 without ownership - full session/record envelopes pass the JSON-domain gate at the handler, so NUL/lone-surrogate identifiers map to 400 instead of 500 - payload validation follows the injected store's validator map (options.payloadValidators) instead of imposing DSL defaults - reference factory accepts authenticate/authorizeMerge/authorizeAdmin/ payloadValidators overrides; docs no longer claim the factory binds to localhost; demo-impersonation warning hardened - 500 responses carry a generic message; details go to the server log - ownership checks precede version checks and unowned sessions read as 404, closing existence/version oracles; cross-learner denial matrix tested per route - merge/delete concurrency documented as linearizable-equivalent with the ownership re-check narrowed to the delete call * fix(storage): align reference-server semantics with the store contract - future-stamped sessions read and delete through unchanged; 409 FUTURE_VERSION applies only to guarded writes (status, append, merge) - check-then-write races reclassify structurally via a post-failure re-fetch: missing session 404, non-active session 400, never a message-sniffed or generic 500 * fix(storage): close the reference CLI's pg pool on startup failure ensureSchema opens connections inside createReferenceRuntimeServer, so a failed schema init or occupied port left the pool holding database resources until its idle timeout. * fix(storage): close the records-route existence oracle cosarah's review point on #946: the records list answered 200 [] for an absent session but 404 for another learner's, so the 404 leaked that an id exists. Absent and foreign sessions now answer identically (404 SESSION_NOT_FOUND) and the HTTP client maps that code back to an empty list, preserving the store contract's absent-lists-as-empty semantics. Contract doc records the server MAY/SHOULD and the client MUST.
102 lines
4.1 KiB
TypeScript
102 lines
4.1 KiB
TypeScript
/**
|
|
* Device-anonymous learner identity for the runtime layer (#869).
|
|
*
|
|
* `learnerKey` partitions all learner-runtime data (RuntimeStore sessions).
|
|
* Until sign-in exists it is a per-device anonymous key: minted once and kept
|
|
* in the KV `device` scope, which never syncs across devices — a synced key
|
|
* would merge two people's runtime into one partition. When sign-in lands,
|
|
* `RuntimeStore.mergeLearner(anonKey, accountKey)` is the migration path.
|
|
*
|
|
* Client-only: the default KV store lazily touches `localStorage`. Server
|
|
* code must not import this without injecting its own `KVStore`.
|
|
*/
|
|
import { BrowserKVStore, type KVStore } from '@openmaic/storage';
|
|
|
|
import { registerRuntimeStorageResetHook, resolveConfiguredLearnerKey } from './config';
|
|
|
|
export const LEARNER_KEY_KV_KEY = 'runtime.learnerKey';
|
|
|
|
const LEARNER_KEY_LOCK = 'maic:learner-key';
|
|
|
|
let defaultKv: KVStore | undefined;
|
|
let defaultInFlight: Promise<string> | undefined;
|
|
let configuredInFlight: Promise<string> | undefined;
|
|
|
|
registerRuntimeStorageResetHook(() => {
|
|
configuredInFlight = undefined;
|
|
defaultInFlight = undefined;
|
|
defaultKv = undefined;
|
|
});
|
|
|
|
function mintLearnerKey(): string {
|
|
const uuid =
|
|
typeof crypto !== 'undefined' && 'randomUUID' in crypto
|
|
? crypto.randomUUID()
|
|
: `${Date.now()}-${Math.random().toString(36).slice(2)}`;
|
|
return `anon:${uuid}`;
|
|
}
|
|
|
|
async function mintPersisted(store: KVStore): Promise<string> {
|
|
const minted = mintLearnerKey();
|
|
await store.set(LEARNER_KEY_KV_KEY, minted, 'device');
|
|
// Return the PERSISTED value, not the local mint: if another writer raced
|
|
// us, everyone converges on the stored winner instead of keeping an
|
|
// orphaned key of their own.
|
|
return (await store.get<string>(LEARNER_KEY_KV_KEY, 'device')) ?? minted;
|
|
}
|
|
|
|
async function readOrMint(store: KVStore): Promise<string> {
|
|
const existing = await store.get<string>(LEARNER_KEY_KV_KEY, 'device');
|
|
if (existing) return existing;
|
|
|
|
if (typeof navigator !== 'undefined' && navigator.locks) {
|
|
// Cross-tab mutual exclusion: only one tab mints under the lock; a tab
|
|
// that lost the race re-reads the winner's key inside its own grant. An
|
|
// existing key is never overwritten, so the per-tab memo stays safe.
|
|
return await navigator.locks.request(LEARNER_KEY_LOCK, async () => {
|
|
const won = await store.get<string>(LEARNER_KEY_KV_KEY, 'device');
|
|
return won ?? mintPersisted(store);
|
|
});
|
|
}
|
|
// No Web Locks (older browsers, non-window contexts): read-after-write
|
|
// only. Residual race: a tab whose re-read lands before another tab's
|
|
// write keeps an orphaned key. Accepted here — it merely splits one
|
|
// anonymous learner's local history, and these contexts are rare.
|
|
return mintPersisted(store);
|
|
}
|
|
|
|
/**
|
|
* Resolve the client session's learner partition key.
|
|
*
|
|
* An explicit KV store takes priority over app-wide configuration. Without an
|
|
* explicit store, a configured provider is invoked only on first resolution:
|
|
* concurrent calls share its in-flight promise and the first resolved value is
|
|
* retained for the session. Identity changes mid-session belong in the
|
|
* application layer (reload or a `mergeLearner` flow).
|
|
*
|
|
* Client-only: this singleton is not SSR-safe or intended for request-scoped
|
|
* identity. Partial dev-mode HMR can recreate this cache independently of the
|
|
* configuration module; reload after changing bootstrap configuration.
|
|
*/
|
|
export function getLearnerKey(kv?: KVStore): Promise<string> {
|
|
// Injected stores (tests, server-side callers) bypass the memo but stay
|
|
// race-safe through the lock / read-after-write above.
|
|
if (kv) return readOrMint(kv);
|
|
|
|
const configured = configuredInFlight ?? resolveConfiguredLearnerKey();
|
|
if (configured) {
|
|
configuredInFlight ??= configured.catch((error) => {
|
|
configuredInFlight = undefined;
|
|
throw error;
|
|
});
|
|
return configuredInFlight;
|
|
}
|
|
// Concurrent same-bundle callers share one in-flight read/mint. A failure
|
|
// is not cached — a transient storage error must not pin every later call.
|
|
defaultInFlight ??= readOrMint((defaultKv ??= new BrowserKVStore())).catch((error) => {
|
|
defaultInFlight = undefined;
|
|
throw error;
|
|
});
|
|
return defaultInFlight;
|
|
}
|