mirror of
https://github.com/THU-MAIC/OpenMAIC.git
synced 2026-10-02 09:24:43 +08:00
* docs(storage): specify the asset registry HTTP contract (#1007) Adds the contract document and the server-side store, byte-layer, and handler types for the asset layer, the only layer with no server backend. Runtime behaviour is unchanged: the new modules are deliberately unreferenced until the backends land. Reclamation moves offline, and that decision carries the design. `remove` deletes a registry entry and nothing else -- it never reads, counts, or deletes bytes. A separate collector takes byte rows that have had no references for a grace period. Three things follow. The hardest race in the design disappears, because no request can delete bytes another request is adopting, so no request path needs a lock on the byte row and the write path never has to reconcile taking one with never branching on whether the bytes already existed. A side channel closes, because a `remove` that only deletes a row costs the same whether or not another principal holds the same bytes. And nothing is given up: under global deduplication `remove` could never promise the bytes were destroyed, so deleting them synchronously bought far less than it appeared to. The cost is that storage grows until the collector runs, which a deployment must configure rather than inherit. Offline reclamation is also what makes the byte layer pluggable, so this keeps #1007's pluggable-backend model rather than narrowing it. With collection out of every request path, the requirement on a byte layer is an ordering rule rather than an atomicity one: bytes are written before the row that references them, deleted after the last row referencing them, and the reference count is serialized on the blob row that every implementation keeps regardless of where bytes sit. Two implementations are planned -- bytes in a column of the transactional store, and an object store keyed by content hash -- which is why the interface exists; an interface with one implementation is not a seam, and this package has removed such a seam before. The browser backend keeps no such seam, because it reclaims inline and its bytes therefore cannot leave its transaction; four comments that stated this as a storage difference now state it as a reclamation one. Metadata crosses the wire as a multipart part. It is open-ended and callers already fill it with generated-narration text and image prompts, so a query parameter would put unbounded user content into the request target and into every intermediary's logs. That also motivates a rule the other layers do not need: metadata must never appear in a URL, a response header, a log line, or an error message. Metadata gets a value domain, and the document is explicit that this is a capability reduction -- the browser backend's structured clone carries Date, Map, Set, and cycles faithfully, so a deployment moving to a server backend must audit its metadata rather than expect a quietly lossy path to tighten. Resolution yields a client-minted object URL, superseding the resolution that a server deployment resolves to a self-hosted proxy path. A media element sends only ambient credentials, so a proxy path works for cookie-session deployments and silently fails for header-authenticated ones; closing that gap means minting URL tokens, a second credential form inside a storage library. Fetching bytes over the client's own authenticated request needs none and removes a class of hash-disclosure surface. The client obligations that make it preserve the browser backend's semantics are specified, and split into the two the shared suite already pins and the four an HTTP backend must pin itself. The cost, no range requests and no progressive playback, is stated. Channels that could carry a principal or a content hash are closed structurally rather than by enumeration: the route table admits no segment that could carry either, and no route takes a query string at all. Headers are closed by not reading them -- the contract assigns no meaning to any header beyond content type and length, and a handler must not derive identity from one outside its authenticate hook. Rejecting header names by pattern was tried and removed: no pattern broad enough to catch X-Remote-User spares User-Agent, and the forwarded-identity headers such a rule would reject are exactly what an authenticating reverse proxy injects for that hook to read. Ids that cannot be a path segment are resolved by the client as misses rather than raised as errors, a deliberate divergence from the KV contract. An unknown id is a miss by this layer's id-domain rule and the shared suite pins the empty string as one, so a client that threw would fail that suite and reintroduce the shape oracle the rule prevents. Also specified: response values may not derive from the shared byte row (Last-Modified off that row is an existence oracle), a collision- resistant digest is required because writes are unconditional and an object layer keys on that digest, byte responses carry the revision and refuse Range, HEAD errors carry the code in a header since they have no body, quota exhaustion has a status and a code, and relabelling a non-renderable type is not refusal -- resolve still returns a URL. * feat(storage): add pluggable server asset store (#1007) * fix(storage): write asset bytes only after claiming the blob row (#1007) The write path wrote bytes twice: once before the registry transaction and once inside it, after the upsert that claims the blob row. Only the second one carries any safety, and the first is what a large-media deployment pays for on every upload. The ordering that matters is claim, write, reference. Claiming the blob row takes its lock, so a collector already holding that lock finishes before the write proceeds; writing bytes before the claim instead lets the collector delete them while the claim waits, leaving a fresh entry that points at nothing. Writing them before the entry keeps every surviving entry backed by bytes that were actually stored. Dropping the first write preserves both ends of that order and removes a crash window that produced orphans for no reason. For a byte layer inside the transactional store this also means a failed write now leaves nothing at all, rather than an orphan the collector had to sweep later. The four tests this changed were each pinned to the removed write: - the statement-sequence test hardcoded the old four-statement shape; it now pins claim, write, entry, and still asserts the two existence paths are identical, which is the property that matters; - the quota test asserted two writes per accepted put; - the crash-window test asserted an orphan that a transactional byte layer no longer produces. It is now two tests: a transactional layer leaves nothing, and a non-transactional one strands an object with no blob row -- which the collector deliberately cannot see, since recovering it is deployment housekeeping rather than reference counting. It also no longer depends on state left behind by earlier tests in the file; - the collector-race test drove its interleaving off the removed write. It was choreography rather than concurrency in any case: PGlite is single-connection, and the test serialized transactions, so no row lock was ever contended. It is replaced by the two invariants that actually make the race safe -- the byte write is unconditional, and bytes the collector has already removed are re-stored by an adopting put. The contended-lock case belongs to the real-PostgreSQL suite, where it can be tested rather than mimed. Reverting the unconditional write fails two of them, so they pin it. The contract said a failed write leaves no partially written bytes while its byte-layer section permits exactly that orphan; the two are reconciled, and the ordering rule now states the claim step and the unconditional write it depends on, along with the cost that follows -- the byte write happens with the registry transaction open, so an object store holds a row lock across a network upload. * test(storage): fix the real-PostgreSQL asset tests against the new write order Both were written against the removed pre-transaction byte write, and both failed once it went away -- one of them by hanging, which also timed out the next test's TRUNCATE. The lock-contention test waited for the adopting put to start writing bytes before releasing the collector. Bytes are now written after the upsert that claims the blob row, and the collector holds that lock, so the write could never start: a circular wait. It now waits for a backend to actually appear in pg_stat_activity blocked on a lock, which is the condition it meant to wait for, observed rather than signalled. Its final assertion carries real weight now -- had the bytes been written before the claim, the collector would have deleted them and the adopting entry would resolve to nothing. The orphan test asserted that a failed registry transaction strands bytes. With this byte layer inside the registry's transaction it strands nothing, so it now asserts that instead. Object storage does strand an object, and that case is covered where it belongs. All of this ran against PostgreSQL 16: 781 tests pass with 3 skipped, and those 3 pass against a real S3-compatible server. Nothing in the suite is now unverified for lack of infrastructure. * feat(storage): add asset HTTP backend (#1007) * docs(storage): separate trusting a header from reading one (#1007) The header rule said a server reads exactly Content-Type and Content-Length, while the size rules require reading Content-Encoding in order to reject it -- the two cannot both be literal, and the implementation had to pick one. What the rule means is narrower: no request header may be trusted to say who is asking, outside the authenticate hook. Reading a header to decide how to frame or refuse a request is a different thing, and the transport headers are read and acted on. * chore(storage): bump to 0.2.3 for the asset server backend (#1007) Additive: new asset registry, byte layers, collector, HTTP handler and client, and their entry points. Nothing existing is removed or changed incompatibly, so this is a patch under the pre-1.0 rule. * fix(storage): close four defects found reviewing the asset backend (#1007) **Concurrent writes could exceed a principal's logical quota.** The check ran on the pool before the write transaction, so two concurrent puts both read the old total and both passed; enough concurrency amplified it arbitrarily. It now runs inside the write transaction behind a transaction-scoped advisory lock on the principal, taken only when a quota is configured -- a branch on deployment configuration, never on data. Pinned by a real-PostgreSQL test: four concurrent six-byte writes against a ten-byte quota, of which exactly one may be accepted. Removing the lock accepts three. **A part name smuggled inside a quoted filename was read as a real part name.** `Content-Disposition` was scanned with a regex that does not understand quoted strings, so `filename="x; name=meta; y"` parsed as a `name` parameter here while an RFC-aware intermediary sees an unnamed part. That is exactly the parser differential the contract requires be rejected. Replaced with a tokenizer that honours quoted strings and escapes and requires exactly one real `name`. **Any S3 404 was reported as an absent object.** `NoSuchBucket`, a misdirected endpoint, and a revoked access point all answer 404 while the bytes still exist, so a storage outage surfaced as `404 ASSET_NOT_FOUND` and a caller clearing its reference on that would turn it into real data loss -- the same failure the contract spells out for `401`. Only key-absent codes map to a miss now; everything else propagates to `500 INTERNAL_ERROR`. **Exceeding `maxParts` answered `400` rather than `413`.** It is one of the declared multipart resource limits and now answers like the other three. Verified against PostgreSQL 16 and a real S3-compatible server: 840 tests, none skipped. * fix(storage): repair two defects the previous fix round introduced (#1007) Both were created by the fixes themselves rather than surviving them, which is the failure mode a second review round exists to catch. **An over-quota replace answered 500.** Moving the quota check inside the write transaction put it under a catch that re-threw only AssetNotFoundError, so AssetQuotaExceededError was collapsed into a generic registry failure and the handler mapped it to INTERNAL_ERROR -- losing the status and code the contract gives that condition. Both typed errors now survive the catch. **The hand-written disposition tokenizer accepted malformed input and trusted it.** The quoted-value loop never checked that it found a closing quote, so `name="meta` ran to the end of the header and returned `meta`; `name="meta"junk` returned `meta` as well; and an extended `name*` form competing with a plain one was silently ignored. Each recreates the parser differential the tokenizer was written to remove. A quoted value now requires its closing quote, rejects a dangling escape, and permits only whitespace and a separator after it, and an extended name form is refused rather than resolved. Also widened the quota lock key from `hashtext` to `hashtextextended`. The 32-bit form collides -- two unrelated principals sharing a key would block each other for the whole transaction, which spans a byte write that may be a network upload. Regression tests cover all three: an over-quota replace raises the quota error and leaves the original bytes, and each of the three malformed dispositions is refused. Verified against PostgreSQL 16 and a real S3-compatible server: 842 tests, none skipped. * fix(storage): narrow the multipart disposition surface and the error boundary (#1007) Three review rounds have each found a defect in the part-disposition parser: a regex that misread quoted strings, then a tokenizer that accepted an unterminated quote, then RFC 2231 continuation forms (`name*0*=UTF-8''bytes`) and malformed empty parameter slots. Patching it a fourth time would have been the wrong move. This contract needs exactly one parameter, so it now accepts exactly one: a part disposition is `form-data` plus a single `name` whose value is `meta` or `bytes`, and everything else is a validation failure. That removes the continuation forms, the encoded forms, the empty slots, and `filename` -- the parameter every one of these attacks travelled in -- without having to model RFC 2231 at all. That rule turned out to bind our own client too: it emitted a fixed `filename="asset"`. Nothing reads it, and the contract already forbids deriving a response disposition filename from caller data, so the client no longer sends one. Separately, `instanceof` was being used as provenance. The byte layer is pluggable, so a byte store raising a same-named `AssetQuotaExceededError` or `AssetNotFoundError` inside the transaction was re-thrown as a logical registry outcome, carrying the byte layer's own message to a direct caller. The registry's checks now raise module-private sentinels, and only those are converted -- at the boundary, into fresh public errors with this package's fixed messages. Anything else, whatever its name, collapses to the generic registry failure. Also pinned the advisory-lock key width. Reverting `hashtextextended` to the 32-bit `hashtext` left every test green, so the emitted lock statement is now asserted. A SQL-text assertion is deliberate here: the behavioural difference is a collision between two particular keys under a database-internal hash, which is not a stable thing to assert against. Verified against PostgreSQL 16 and a real S3-compatible server: 845 tests, none skipped. * fix(storage): define multipart disposition grammar (#1007) * fix(storage): delegate multipart parsing to Fetch (#1007) * fix(storage): require both multipart parts to be files (#1007) Delegating the framing to the platform parser cost two rules for a metadata part sent without a filename: such a part comes back as a string with its headers discarded, so its `application/json` type could no longer be checked, and its bytes had already been replacement-decoded -- `{"x":"<0xFF>"}" was stored as `{"x":"\uFFFD"}` instead of refused. The bytes part already had to be a file for binary safety. Metadata is now symmetric: the client sends a fixed filename on both, the server requires both to be files, and with the part preserved the media type is checked and the bytes are decoded fatally before parsing. Neither filename is read anywhere. The limits are also described honestly now. `maxParts`, `maxMetaBytes` and `maxAssetBytes` are validated after the parser has materialized every part, so they bound what is accepted rather than what is parsed, and `maxRequestBytes` is what caps memory. The contract said `maxParts` bounded parser work; it does not. Restoring that would mean adopting a streaming parser with its own grammar, which is the disagreement with intermediaries that delegating to the platform exists to remove -- so the trade is stated rather than reversed. Also corrected the media-type wording: the two branches are metadata present versus absent, and retention of an untyped replacement's prior type is a browser-backend behaviour the HTTP path cannot reproduce, because a conforming parser supplies a default type for a file part whose header omits one. Verified against PostgreSQL 16 and a real S3-compatible server: 853 tests, none skipped. * docs(storage): the asset server backend ships in this branch (#1007) The contract still described the server backend as not yet shipped, which this branch is what changes. The conformance server remains test-only. * fix(storage): make asset HEAD bytes-free and pin reads (#1007) * fix(storage): harden asset write boundaries (#1007) * chore(storage): bump to 0.2.4 for the asset server backend (#1007) main merged in a 0.2.3 of its own, so this branch's bump no longer increased the version. Additive relative to the merged base -- new asset registry, byte layers, collector, HTTP handler and client, with nothing removed or narrowed -- so a patch bump under the pre-1.0 rule. --------- Co-authored-by: 杨慎 <117187635+cosarah@users.noreply.github.com>
This commit is contained in:
@@ -54,11 +54,14 @@ a browser.
|
||||
branches do not reveal whether the bytes were already present. The browser
|
||||
registry embeds its `blobs` table in the same database because reference
|
||||
counting, byte writes, and reclamation must share one transaction. This is
|
||||
not a replaceable browser-side blob backend; a replaceable blob interface is
|
||||
a server-backend concern (delivery plan part 4), where consistency is enforced
|
||||
server-side. Resource-accounting channels remain:
|
||||
not a replaceable blob backend *in the browser*, because that store reclaims
|
||||
inline. The server backend collects offline instead, so no request deletes
|
||||
bytes and its byte layer is pluggable — a column of the transactional store,
|
||||
or an object store keyed by content hash. Resource-accounting channels remain:
|
||||
quota errors, storage estimates, and server billing or metering can disclose
|
||||
existence, so server deployments must budget them per principal. Object URLs
|
||||
existence, so server deployments must budget them per principal — the
|
||||
[asset registry HTTP contract](./docs/asset-http-contract.md) requires quota
|
||||
to be accounted on a principal's logical bytes for exactly that reason. Object URLs
|
||||
are minted per id, not shared per `contentHash`: sharing would let a holder of
|
||||
two ids learn that their bytes match by comparing URL strings. Each
|
||||
`replace(id, ...)` followed by `resolve(id)` adds one retired snapshot that
|
||||
@@ -174,11 +177,12 @@ adding a caller-configurable allocation path.
|
||||
- [x] DocumentStore PostgreSQL backend
|
||||
- [x] `KVStore` (`account`) HTTP backend + HTTP contract
|
||||
- [ ] `KVStore` server-side reference backend and reference-server route
|
||||
- [ ] asset server backend — registry (principal column, server-derived) +
|
||||
replaceable blob storage, over the global resource pool model (#1007),
|
||||
with consistency enforced server-side. It must validate or allowlist
|
||||
content types before serving bytes rather than reflecting cross-principal
|
||||
metadata into response content types
|
||||
- [x] asset registry HTTP contract (#1007)
|
||||
- [ ] asset server backend — registry (principal column, server-derived) over a
|
||||
pluggable byte layer, with transactional-store and object-store
|
||||
implementations and an offline byte collector (#1007). It must allowlist
|
||||
content types before serving bytes, and it must account quota on a
|
||||
principal's logical bytes rather than on bytes physically written
|
||||
- [ ] asset manifest: the one enumeration of "which `AssetId`s does this course
|
||||
reference?" the export paths converge on (#1007)
|
||||
|
||||
|
||||
@@ -0,0 +1,320 @@
|
||||
# Asset registry HTTP contract
|
||||
|
||||
This contract exposes the asset operations — `put` / `identify` / `resolve` / `remove` / `replace` — over HTTP. All paths below are relative to a deployment-defined base URL. Path segments are percent-encoded UTF-8 strings. Metadata travels as JSON; asset bytes travel as bytes, as a multipart part on writes and as the whole body on reads. Successful operations with no return value respond with `204 No Content`.
|
||||
|
||||
The server backend for this contract ships as `createAssetHttpHandler`, with asset requests dispatched by `createStorageHttpHandler`. The conformance server this package provides remains test-only: it implements no authentication or authorization model beyond what the contract's response codes require, because deriving the principal from an authenticated session is the reference server's job.
|
||||
|
||||
The design this contract serves has three layers — an allocated asset id names a registry entry, and a registry entry names a content hash, and a content hash names bytes. **Only the first layer is on the wire.** The content hash is an internal deduplication key: it MUST NOT appear in any URL, response header, response body, error message, error details, or log line.
|
||||
|
||||
That prohibition is not satisfied by omitting the hash itself. Bytes are shared across principals and their storage row has no owner, so **no response header, body value, or status may be derived from the blob row except the representation's byte length.** Every other value on a byte response comes from the registry entry — its media type, its revision, its own creation time. The byte length is allowed because it is an inherent property of the representation and is the same value a `GET` obtains from the returned bytes; retaining it in the registry lets `HEAD` reproduce `Content-Length` without fetching those bytes. `Last-Modified` is the trap worth naming: sourced from the shared row's creation time, as a static-file idiom naturally would, it reports a timestamp earlier than the caller's own write and thereby proves another principal stored those bytes first. Byte responses MUST NOT carry `Last-Modified`, `Age`, or any other value read from the shared row; note that the `no-store` requirement below already makes `Last-Modified` pointless. Frameworks that generate `ETag` and `Last-Modified` by default MUST have both disabled on these routes.
|
||||
|
||||
The hash MUST be a collision-resistant digest of at least SHA-256 strength, untruncated. This is a security requirement rather than an implementation preference, and it is the unstated precondition of two rules below: a server writes bytes unconditionally, so under a colliding digest one principal's write silently substitutes the bytes every other principal's entries resolve to — and an object-storage byte layer names each object by that digest, so a collision there is a cross-principal overwrite at the storage layer as well.
|
||||
|
||||
## An asset id is opaque and unconstrained
|
||||
|
||||
An asset id is a string this package allocated, but nothing in this contract may depend on its shape. Ids are compared, never parsed. There is no id validator that can *reject* an id on either side, and a server MUST NOT refuse one for its content. (The client does inspect an id for the three transport classes below, but only ever to answer "miss" locally — never to reject.)
|
||||
|
||||
The consequence is the one that matters: **an id this registry never allocated is a miss, not an error.** An id from another id space, an empty string, a string containing NUL, four kilobytes of padding, `../../etc/passwd` — each is an ordinary lookup that finds nothing. This is not leniency. An id whose *shape* could be rejected would answer a question the caller is not entitled to ask, and a caller who can distinguish "malformed" from "not yours" has learned something about the id space. The browser backend documents this rule for the same reason, and the shared conformance suite pins each case.
|
||||
|
||||
A server therefore stores and looks up the decoded id as an opaque value — a bound query parameter, or a derived constrained storage id — **never as a path component, a filename, or an unescaped fragment of a query**. An id like `a/../../b`, arriving percent-encoded as `a%2F..%2F..%2Fb`, is one opaque lookup that traverses nothing.
|
||||
|
||||
### Ids that cannot be a path segment
|
||||
|
||||
Validity is a property of the id; reachability is a property of the transport. An id sits **mid-path** here, before the trailing `content` segment, and three classes of id cannot be carried there:
|
||||
|
||||
1. **An unpaired UTF-16 surrogate** has no percent-encoding; `encodeURIComponent` throws.
|
||||
2. **A whole-id `.` or `..`** is normalized away by URL path parsing before the request is sent, so `/assets/../content` would address `/content`.
|
||||
3. **The empty id** collapses `/assets//content`, which intermediaries that merge slashes rewrite to `/assets/content`.
|
||||
|
||||
The third has no counterpart in the [KVStore HTTP contract](./kv-http-contract.md), whose key is a trailing segment and round-trips when empty. The danger in all three is identical and it is not that the request fails — it is that the request **succeeds against something else**.
|
||||
|
||||
The client MUST therefore resolve these three locally, before building any request. **It does not raise an error: it answers as a miss** — `resolve` returns `null`, `remove` succeeds as a no-op, and `replace` fails as it does for any id the registry does not hold, with a synthesized `ASSET_NOT_FOUND` since there is no response to reconstitute one from. This is where this contract deliberately diverges from KV, which refuses an unencodable key with a client-side `KEY_NOT_ENCODABLE`. Throwing would be wrong here: an unknown id is a miss by the rule above, the shared conformance suite pins the empty string as a miss on both `resolve` and `remove`, and a client that threw would both fail that suite and reintroduce the shape oracle the rule exists to prevent. The distinction to preserve is not loud-versus-quiet; it is that the id must never address something it does not name.
|
||||
|
||||
Note that ids which merely *look* structural need none of this. `../../etc/passwd` percent-encodes to `..%2F..%2Fetc%2Fpasswd`, one ordinary segment that reaches the server and misses there; it illustrates the id-domain rule, not the local-resolution one.
|
||||
|
||||
### Transport hazards a deployment owns
|
||||
|
||||
Beyond those three, the path carries an id through intermediaries that may refuse or rewrite it, and the shared suite pins ids in each category. A conforming deployment MUST pass the id segment through unmodified, and MUST verify that it does:
|
||||
|
||||
- **A percent-encoded NUL** (`%00`) is rejected outright by some servers and filters.
|
||||
- **A percent-encoded slash** (`%2F`) is rejected or normalized by servers that do not allow encoded slashes by default — which the `bucket/path/to/object.png` shape produces.
|
||||
- **A very long id** may exceed a request-target ceiling and be refused with `431` before routing. No length is invalid; a long id is simply not reachable past that bound.
|
||||
|
||||
Each of these turns a pinned miss into a thrown error, and none is predictable client-side, so none may be papered over by mapping an unclassifiable `4xx` to a miss — that would break the rule that only a specific code becomes a miss. They are deployment configuration, and the shared suite's cases stand as the acceptance test for it. Real ids, being allocated by this package, encounter none of them.
|
||||
|
||||
## Endpoints
|
||||
|
||||
| Method | Path | Purpose | Success |
|
||||
| --- | --- | --- | --- |
|
||||
| `POST` | `/assets` | Allocate a new id and store the submitted bytes under it. | `201` with `{ "id": "ast_…" }` and `X-Asset-Revision` |
|
||||
| `GET` | `/assets/{id}/content` | Read the bytes stored under an id. | `200` with the bytes |
|
||||
| `HEAD` | `/assets/{id}/content` | Read identity headers without reading the byte layer. | `200`, no body |
|
||||
| `PUT` | `/assets/{id}/content` | Replace the bytes stored under an existing id. | `204` with `X-Asset-Revision` |
|
||||
| `DELETE` | `/assets/{id}` | Remove the registry entry. | `204` for any id the policy admits |
|
||||
|
||||
The route table admits exactly one segment after `assets`, and it is the id. There is no principal segment and no digest segment to supply, so any other path shape is `404 ROUTE_NOT_FOUND` — a routing outcome that requires inspecting nothing. A path that *does* match a table entry but with a method that entry does not list is `405 METHOD_NOT_ALLOWED`, carrying an `Allow` header naming that route's methods; since route matching never consults the registry, neither outcome discloses anything about an id.
|
||||
|
||||
**No route takes a query parameter.** A server MUST reject any request whose target contains `?` at all with `400 VALIDATION_FAILED` — stated on the raw target rather than on a parsed query, so that a bare trailing `?` cannot be read as absent by one implementation and present by another. That is total where an enumeration of forbidden parameter names would not be, and it costs nothing, because there is no parameter here to preserve.
|
||||
|
||||
There is deliberately **no metadata read route**. No backend offers one: the browser store's public surface is `put` / `resolve` / `release` / `replace` / `remove` / `close`, and no metadata member is ever returned to a caller. (`contentType` is read back, but only by the store itself, to label the bytes it hands out.) A route no backend can satisfy is not a contract but a promise. Warm-cache revalidation is what `HEAD` is for.
|
||||
|
||||
`HEAD` performs an ownership-checked registry identity read and MUST NOT read or materialize the asset bytes. Its status and headers are identical to `GET` for the same registry state; only the response body is absent. The registry's recorded byte length supplies `Content-Length`, so reproducing the `GET` headers does not require a byte-layer read.
|
||||
|
||||
Read routes MUST be served with `Cache-Control: private, no-store` and `Vary: Cookie, Authorization`, and MUST NOT be cached by any intermediary; the client sends its reads with `cache: 'no-store'` for the same reason, since a UA cache serving a stale `200` would defeat revision revalidation invisibly. Note that this differs from a content-addressed design, where bytes named by their digest never change and are safe to cache forever: here `replace` mutates the bytes behind a **stable** id, so no HTTP cache may serve a response for an id without revalidating. The client's own snapshot cache is a different thing and is governed below.
|
||||
|
||||
Byte responses MUST NOT advertise `Accept-Ranges` and MUST ignore a `Range` header, always answering `200` with the complete representation. A `206` would carry a response model this contract does not define, on a route whose headers are load-bearing for revalidation.
|
||||
|
||||
## Transport: bytes and metadata
|
||||
|
||||
`POST /assets` and `PUT /assets/{id}/content` take `multipart/form-data`, with the parts in this order:
|
||||
|
||||
- **`meta`** — `application/json`, the metadata object. Required on `POST`. **Optional on `PUT`**, where its absence is meaningful; see below.
|
||||
- **`bytes`** — the asset bytes, carrying the asset's own `Content-Type`. The package client uses `application/octet-stream` when the source blob has no type, because multipart parsers otherwise supply their own default media type.
|
||||
|
||||
The multipart framing MUST be parsed by a standards-conforming multipart parser; a deployment MUST NOT hand-roll one. A bespoke grammar disagrees with the parsers intermediaries use, and every such disagreement is a way for a scanner and the application to see different parts.
|
||||
|
||||
Both parts MUST carry a `filename`. Without one, a conforming parser returns the part as a string, discards its headers, and text-decodes its payload, silently replacing invalid UTF-8. Each filename is fixed by the package client, never read by this package, and never derived from caller data. Preserving `meta` as a file allows the server to require its `Content-Type` to be `application/json` and to decode its bytes as UTF-8 with errors reported rather than replacement characters inserted.
|
||||
|
||||
Multipart rather than metadata alongside a raw body, because there is nowhere safe to put the metadata. It is an open-ended object, and real callers fill it with generated-narration text and image-generation prompts — **unbounded caller-supplied content**. In a query parameter that content would sit in the request target, where it hits the request-target ceiling and is written verbatim into every intermediary's access log. A custom header has the same size problem for the same reason. Multipart is a standards-defined framing rather than a bespoke one, which is the property that matters; `maxMetaBytes` must be enforced against the parsed metadata part before the JSON is parsed rather than after.
|
||||
|
||||
From this, one rule the other layers have no need for:
|
||||
|
||||
> **Metadata is unbounded caller-supplied content. It MUST NOT appear in any URL, response header, response body, log line, error message, or error details.** The single exception is `contentType`, which determines a byte response's `Content-Type` as specified below. In particular a `Content-Disposition` filename MUST NOT be derived from metadata: it would put caller content in a header, and an unescaped one is a header-injection primitive. Use a fixed name or the id.
|
||||
|
||||
The client's request-header hook MUST NOT set `Content-Type`, and a client whose hook does MUST fail loud rather than choose a winner. Under multipart this is more serious than mislabelling: overwriting `Content-Type` destroys the boundary parameter, and every write from that deployment fails to parse with an error naming nothing an operator can act on.
|
||||
|
||||
### Sizes
|
||||
|
||||
Three byte limits and one part-count limit, because they bound different things:
|
||||
|
||||
| Limit | Default | Bounds | Measured on |
|
||||
| --- | --- | --- | --- |
|
||||
| `maxRequestBytes` | 33 MiB | The whole request | Raw octets off the wire, before parsing |
|
||||
| `maxAssetBytes` | 32 MiB | The `bytes` part | Decoded part content |
|
||||
| `maxMetaBytes` | 64 KiB | The `meta` part | Decoded part content |
|
||||
| `maxParts` | 8 | Number of multipart parts | Parsed frames |
|
||||
|
||||
Only `maxRequestBytes` is enforced before multipart parsing. It bounds the raw body read from the wire and is therefore the only ceiling on parser work and request-derived memory. The platform's `Response.formData()` then materializes every part before `maxParts`, `maxMetaBytes`, and `maxAssetBytes` are checked on the parsed result; those three limits bound what the handler accepts, not what the multipart parser processes.
|
||||
|
||||
This trade is deliberate. A streaming multipart parser with its own limits would introduce a second grammar that could disagree with the standards-conforming platform parser, recreating the scanner/application parser differential this delegation exists to remove. Bounded memory is preserved by `maxRequestBytes`; `maxParts`, `maxMetaBytes`, and `maxAssetBytes` remain finer-grained admission rules for the parsed request. Separate asset and metadata limits are still necessary because one shared admission limit either caps assets at metadata scale or admits metadata at asset scale. Per-part header bounding belongs to the standards-conforming parser rather than to this contract.
|
||||
|
||||
The outer bound is measured differently from the inner two, and deliberately so: it exists to stop reading, so it cannot wait for a decode. A handler MUST assert at construction that `maxRequestBytes` exceeds `maxAssetBytes + maxMetaBytes` with room for multipart framing, or the outer bound silently masks the inner one — with the defaults above equal, an asset at exactly `maxAssetBytes` would always be rejected by the request bound, reporting the wrong limit.
|
||||
|
||||
None may be inferred from `Content-Length`, which is a claim by the sender. Counting bytes as they are read is necessary but not sufficient for the decoded limits — under a `Content-Encoding` the bytes read are a fraction of the bytes stored, and the expansion lands inside the transaction that holds the write. A server therefore MUST reject `Content-Encoding` on these requests with `400 VALIDATION_FAILED`. Exceeding any size limit is `413 PAYLOAD_TOO_LARGE`, raised **before any bytes are stored**.
|
||||
|
||||
### Malformed and hostile bodies
|
||||
|
||||
A request whose `Content-Type` is not `multipart/form-data` is `415 UNSUPPORTED_MEDIA_TYPE`. A failure by the multipart parser, including invalid boundary framing, is `400 VALIDATION_FAILED` with a fixed package message. The expected entries are exactly `meta` and `bytes` on `POST`, and `bytes` with optional `meta` on `PUT`; any other entry name or any duplicate is `400 VALIDATION_FAILED`. Duplicate entries in particular MUST be rejected rather than resolved by a first-wins or last-wins rule: a scanning intermediary and the application choosing differently is a parser differential, and the two would disagree about which metadata a request carried.
|
||||
|
||||
When both entries are present, `meta` MUST precede `bytes`.
|
||||
|
||||
### Recording the media type
|
||||
|
||||
On `POST`, the recorded media type is `meta.contentType` when that member is **present** — including when it is the empty string — and the `bytes` part's own `Content-Type` only when it is absent. The distinction is `??` rather than `||`, and it is observable: an explicitly empty `contentType` records the empty string and is therefore served as an attachment, where a fallback would have served the blob's type inline.
|
||||
|
||||
On `PUT`, when `meta` is present it replaces the entry's metadata wholesale and the media type is derived the same way. When `meta` is **absent**, the entry's existing metadata is retained and its media type is retained unless the `bytes` part carries one.
|
||||
|
||||
The package client always carries a media type on `bytes`: an untyped source blob is sent as `application/octet-stream`. This is necessary because a standards-conforming parser supplies a default media type for a file part whose header omits one, so omission cannot survive parsing as an empty type and cannot signal retention. Metadata omission still retains the existing metadata object; only the media type follows the replacement bytes.
|
||||
|
||||
The two branches are metadata present versus metadata absent, matching the browser backend's metadata replacement and retention behavior. The absent branch is the one that matters: regenerating bytes in place while keeping the recorded provenance is the use case `replace` exists for. A wire on which `meta` were mandatory would force a client to send `{}`, erasing the accumulated prompt, model, narration, and voice on every regeneration — through a call that returns `204`.
|
||||
|
||||
Media-type retention is a deliberate divergence. The browser backend can retain the existing media type when metadata is absent and the replacement blob is untyped. The HTTP path cannot reproduce that behavior because a conforming multipart parser supplies a default type for a file part whose header omits one. The package client therefore sends `application/octet-stream` for an untyped replacement, and that type replaces the recorded media type even while the existing metadata object is retained.
|
||||
|
||||
### Serving bytes
|
||||
|
||||
Every response carrying bytes MUST send `X-Content-Type-Options: nosniff` and MUST serve a `Content-Type` from a renderable allowlist. The default allowlist is:
|
||||
|
||||
```
|
||||
image/png image/jpeg image/gif image/webp image/avif
|
||||
audio/mpeg audio/mp4 audio/ogg audio/wav audio/webm
|
||||
video/mp4 video/webm video/ogg
|
||||
```
|
||||
|
||||
Matching is exact, case-insensitive, and whole-string, with no parameters accepted. A deployment may narrow this list; widening it is a decision about executable content. **Excluded, and MUST remain excluded:** `image/svg+xml` and `image/svg`, `text/html`, `application/xhtml+xml`, `text/xml` and `application/xml`, and `application/pdf`. These are document formats that execute script, not media, and serving one same-origin turns stored bytes into stored script.
|
||||
|
||||
A recorded media type that is not a string, is empty, or is outside the allowlist is served as `application/octet-stream` with `Content-Disposition: attachment`. The empty case is real when metadata explicitly carries an empty `contentType`. The client MUST label its minted object URL from the **served** `Content-Type` and from nothing else; labelling it from metadata would reintroduce inside a `blob:` URL exactly what the allowlist excludes.
|
||||
|
||||
**Relabelling is not refusal.** `resolve` returns a minted URL for every stored asset regardless of media type; a non-renderable asset yields a URL labelled `application/octet-stream` that a media element will not render, and that is the intended outcome rather than a miss. Returning `null` instead would fail the shared conformance suite, every blob in which is `text/plain` — outside any sensible allowlist — and would conflate "these bytes are not safe to render inline" with "there are no bytes." The disposition header governs direct navigation and does no work on this path, where bytes reach the caller through `fetch`; the protection that carries the weight is the relabelling together with `nosniff`.
|
||||
|
||||
The three-layer design improves on a content-addressed one here, and it is worth stating why. Where bytes are shared across principals and the recorded media type travels with the bytes, one principal's metadata can determine how another principal's response is labelled — a stored cross-principal typing attack. Here the media type lives on the **registry entry**, and every entry belongs to exactly one principal, so a principal can only mislabel their own bytes. The attack is unrepresentable rather than mitigated, given the collision-resistant digest required above and the rule that no response value derives from the shared row.
|
||||
|
||||
### `X-Asset-Revision`
|
||||
|
||||
`GET` and `HEAD` on `/assets/{id}/content`, and the success responses of `POST /assets` and `PUT /assets/{id}/content`, MUST carry an `X-Asset-Revision` header: a monotonically increasing integer on the registry entry, starting at 1 and incremented by each `replace`. It is opaque to the client except for equality comparison.
|
||||
|
||||
It MUST NOT be derived from the content in any way. A content-derived validator — the reflex choice, and what an `ETag` from a static file server or object store would be — is the content hash under a different header name, and would disclose byte equality across ids to anyone holding two of them. Byte responses MUST NOT carry a content-derived `ETag` for the same reason.
|
||||
|
||||
Carrying it on the write responses too means a client learns the revision it produced without a second request, which would otherwise race a concurrent `replace`.
|
||||
|
||||
The headers and the body of a `GET` byte response MUST be produced from **one transactionally coordinated read**. Reading the revision and media type, releasing the transaction, and then reading the bytes lets a concurrent `replace` pair revision *n* headers with revision *n+1* bytes, which the client then caches as authoritative. The browser backend closes the same window deliberately, reading the registry entry in the same transaction as the bytes. `HEAD` reads only the registry identity and recorded byte length in one ownership-checked query.
|
||||
|
||||
## Resolution yields a client-minted object URL
|
||||
|
||||
`resolve` returns a URL the caller can use as an `<img>` / `<audio>` / `<video>` source. Over this contract, the client fetches the bytes through its own authenticated request and mints a local object URL from them. **The server exposes bytes; it does not hand out a URL for a media element to load.**
|
||||
|
||||
This supersedes the earlier resolution that a server-backed deployment would resolve to a self-hosted proxy path by default. That shape does not survive contact with how a resolved URL is consumed. A media element sends only ambient credentials — it cannot carry a client's request-header hook — so a proxy path works for deployments whose sessions are cookie-based and **silently fails** for header-authenticated ones. Closing that gap requires the deployment to mint and verify short-lived URL tokens, which is a second credential form invented inside a storage library and offered as the default.
|
||||
|
||||
Fetching the bytes through the client's ordinary authenticated request path needs no second credential form. It also preserves the browser backend's semantics, which is what lets consumers written against that backend keep working — but only if the client observes the rules below, which are the substance of that claim rather than a consequence of it. And it removes a class of disclosure surface: the client never receives a URL into which a hash could be embedded, so content-derived validators, cache keys, and URL shapes stop being questions this contract has to get right.
|
||||
|
||||
The cost, stated plainly: there are no range requests and no progressive playback. The bytes are downloaded in full before a media element sees them, and a resolved asset is held in memory until released — on the server side too, since a byte read materializes the asset rather than streaming it. This is what the browser backend already does, so it is not a regression for any existing consumer, but it is a real ceiling on large media. Self-hosted proxy paths and signed object-storage URLs remain possible later shapes; they are additive, and adding one when a deployment needs it beats defaulting to a shape that works for only half of them.
|
||||
|
||||
### Client resolution semantics
|
||||
|
||||
These are normative. Two of them the shared conformance suite already pins; the rest it cannot, and saying which is which matters, because an implementer will otherwise trust a green suite to have checked them.
|
||||
|
||||
Pinned by the shared suite:
|
||||
|
||||
- **Concurrent `resolve` calls for one id coalesce** onto a single request and a single minted URL. Minting twice orphans one of them — nothing can revoke it — and pins the asset in memory twice.
|
||||
- **A `remove` invalidates the warm snapshot**, and a `resolve` that misses retires it.
|
||||
|
||||
Normative but **not** pinned by the shared suite, and therefore the HTTP backend's own tests must pin them when it lands:
|
||||
|
||||
- **A returned URL is an immutable snapshot.** A later mutation retires it rather than revoking it; already-issued URLs stay valid. Only `release(id)` and `close()` revoke. (The suite has no `release` coverage; the browser backend's own tests carry it.)
|
||||
- **The object-URL cache is keyed per asset id, never per content.** A client that keys it on a digest of the response bytes discloses byte equality to a holder of two ids — the disclosure the whole design exists to prevent, reintroduced client-side. The suite asserts identical bytes yield distinct *ids*, never that they yield distinct *URLs*.
|
||||
- **A `replace` invalidates the warm snapshot and any in-flight resolution for that id.** (A `put` allocates a fresh id, which by construction has no cache entry to invalidate.)
|
||||
- **The warm-hit identity is the revision together with the served media type** — the HTTP analogue of the browser backend's content-hash-and-media-type pair. The suite is transport-agnostic and has no vocabulary for revisions.
|
||||
- **A client records the revision carried by the response that delivered the bytes, never one learned from a `HEAD`.** A `HEAD` decides only whether to re-fetch. Recording the `HEAD`'s revision against bytes fetched afterwards labels revision *n+1* bytes as revision *n*, and every later `HEAD` then confirms them as fresh.
|
||||
|
||||
One limit of this design is worth stating rather than leaving to be discovered: a `replace` performed elsewhere — another tab, another device — leaves a warm snapshot stale until the next revalidation, where the browser backend re-reads the registry inside every `resolve`. That is inherent to a network backend, and it is the one place these rules do not reproduce the browser's semantics.
|
||||
|
||||
### Cross-origin deployments
|
||||
|
||||
The client reads non-safelisted response headers on both reads and writes. A cross-origin server MUST send `Access-Control-Expose-Headers: X-Asset-Revision, X-Error-Code` on `GET` and `HEAD`, error responses included, and MUST expose `X-Asset-Revision` on the `201` success response from `POST` and the `204` success response from `PUT`. Without the read revision, the client's cache either never validates or validates as always-fresh. Without the error code, a client cannot classify any `HEAD` error and falls back to a full byte `GET` on every miss — so the mechanism `HEAD` exists to provide stops working, silently, in exactly the deployment shape this section addresses. Without the write revision, the client reports a committed allocation or replacement as a malformed response; retrying the non-idempotent `POST` then allocates a second entry for the same bytes.
|
||||
|
||||
A client accepts a `credentials` option and passes it through untouched. A credentialed cross-origin deployment answers with a concrete origin and `Access-Control-Allow-Credentials: true`, never `*`.
|
||||
|
||||
## Allocation discloses nothing
|
||||
|
||||
Every successful `POST /assets` allocates a **new** id, including for bytes the registry already holds. Returning an existing id would let any caller test whether arbitrary bytes are already stored — an existence oracle over data the caller never stored, and the reason the three-layer design exists at all.
|
||||
|
||||
The requirement is stronger than the return value. Every branch taken and every value returned on the success path MUST be independent of whether the bytes were already present. A server MUST NOT read-then-write: it writes the blob unconditionally and inserts the registry entry, so both paths execute the same statements in the same order.
|
||||
|
||||
Three channels stay outside the guarantee, and a deployment MUST budget for them:
|
||||
|
||||
- **Quota and metering.** These MUST be accounted on the principal's **logical** bytes — the sum of the bytes referenced by that principal's registry entries — and never on bytes physically newly written. Under physical accounting a deduplicated write costs nothing while a fresh write costs quota, so two equally sized writes answer the oracle the rest of this design closes. The same applies to any storage estimate or billing readout. A deployment MUST also raise `ASSET_QUOTA_EXCEEDED` from the **logical** check before attempting a physical write, and MUST keep physical headroom above the sum of logical quotas — otherwise a write near capacity succeeds when the bytes deduplicate and fails when they are novel, and the filesystem answers the question the accounting rule closed.
|
||||
- **Cost, on the write path only.** A deduplicated write does less physical work than a novel one, so `POST` and `PUT` latency and write volume vary with whether the bytes were already present. This contract does not promise constant-cost writes, and a deployment requiring that must provide it above this layer. Note that `remove` is *not* in this category: because reclamation is offline, it deletes one row and costs the same whether or not another principal holds the same bytes — a channel that a synchronous reclaimer would have left open. The reference-count value itself is unobservable in any response.
|
||||
- **Anything a deployment adds.** A diagnostic, a storage estimate, or a rate-limit counter need not be *read from* the blob row to leak: it is enough that its value correlates with whether a write deduplicated. A deployment adding any such value to a response owes it the same analysis as the two channels above, over and beyond the blob-row rule, which closes only the values read directly from that row.
|
||||
|
||||
## Where the bytes live
|
||||
|
||||
The registry — ids, owners, media types, metadata, and the reference count — lives in a transactional store. The **bytes are a pluggable layer** behind it: a column in that same store, or a filesystem, or an object store addressed by content hash.
|
||||
|
||||
That is possible because of a decision made in the next section: **reclamation is offline**, never part of a request. Were a request allowed to delete bytes, a concurrent write deduplicating onto those same bytes could commit against them, and a byte layer outside the transaction would have no way to serialize the two — which is what would force bytes into the transactional store. Offline reclamation removes that race from every request path, and what remains is coordination on a row rather than on the bytes.
|
||||
|
||||
The invariant a byte layer must satisfy is therefore an **ordering** rule, not an atomicity one:
|
||||
|
||||
- The **blob row is claimed first**, in the transactional store, before the bytes are written. This is the step that serializes a write against the collector: claiming the row takes its lock, so a collector already holding it finishes first and the write then re-claims the row. Writing bytes before claiming would not be safe — the collector could delete them while the claim waited, leaving a fresh entry that points at nothing.
|
||||
- Bytes are then written, **before** the entry that references them. A crash between the two leaves bytes nobody references, which costs storage and loses nothing; the reverse order would leave a registry entry pointing at bytes that were never stored.
|
||||
- The write is **unconditional**. It must not be skipped because the bytes appear to be present already — partly because the interleaving above produces exactly that appearance, and partly because branching on prior presence is what the allocation rule forbids.
|
||||
- Bytes are deleted **after** the last row referencing them, and only by the offline collector.
|
||||
- The reference count and the decision to collect are serialized on that same **blob row**, which every byte layer keeps regardless of where the bytes themselves sit.
|
||||
|
||||
One consequence is worth stating rather than leaving to be discovered: byte-layer I/O happens while the registry transaction is open. The write path holds the blob row's write lock across the upload, and the read path holds a shared blob-row lock across the download so the collector cannot delete the object in flight. For a byte layer inside the registry store this costs nothing; for an object store it means a row lock and a database connection are held across network I/O, which is a consideration for how a deployment sizes `maxAssetBytes` and its connection pool. The read lock is shared, not exclusive, so concurrent reads of the same bytes do not serialize against one another.
|
||||
|
||||
Two implementations ship, and the interface exists because both are real rather than because one might be one day. A byte layer in the transactional store keeps bytes in a column: nothing to reconcile, since a rolled-back transaction leaves nothing behind, but its bytes flow through the write-ahead log and the backups, so backup and replication volume scales with stored assets — which is the reason to size `maxAssetBytes` deliberately, and eventually the reason to move off it. An object-storage byte layer keeps that volume out of the database at the cost of one housekeeping duty: the crash window above leaves unreferenced objects, so a deployment reconciles or expires them. Naming each object by its content hash keeps those orphans harmless — the next write of the same bytes overwrites one idempotently, so nothing has to hunt for them.
|
||||
|
||||
### Reclamation is offline
|
||||
|
||||
Bytes are shared across principals; a registry entry is not. `remove` deletes the registry entry and **nothing else**: it never reads, counts, or deletes bytes. Bytes outlive their last reference until a separate, offline collector takes them — one that finds byte rows with no references older than a grace period, locks each, re-checks, and deletes.
|
||||
|
||||
Three things follow, and each is a reason the design is this way rather than a consequence to tolerate:
|
||||
|
||||
1. **Request paths never delete bytes.** The write path claims the blob row before storing, and the read path holds a shared lock while reading bytes. The collector takes an exclusive lock before deletion, so it waits for in-flight reads and writes without serializing readers against one another.
|
||||
2. **A side channel closes.** Were `remove` to reclaim, its cost would vary with whether another principal still held the same bytes, so a caller could learn that by deleting their own asset and timing it. A `remove` that only deletes one row costs the same either way.
|
||||
3. **Nothing is given up, because the guarantee never existed.** With global deduplication, `remove` could never promise the bytes were destroyed — another principal referencing them keeps them. Synchronous deletion therefore bought far less than it appeared to.
|
||||
|
||||
The cost is real and belongs to the deployment: storage grows until the collector runs, and bytes a user deleted remain on disk until then. A deployment MUST configure the collector and its grace period; leaving both unset is a design that grows without bound. The retention posture that follows is the one deduplication already implied, now with a stated delay rather than an implied one.
|
||||
|
||||
The reference count spans all principals — that is what makes global deduplication reclaimable — and its value MUST NOT be observable in any response.
|
||||
|
||||
Collection is a host decision, not a contract event. A registry entry whose bytes are gone resolves to a miss rather than an error, because inventing an error would make collection observable.
|
||||
|
||||
Reference counting has a second level in the wider design, from documents to asset ids. That level depends on manifest work that has not shipped, and a server implementing this contract implements the id-to-bytes level only.
|
||||
|
||||
## Metadata domain
|
||||
|
||||
Metadata is a JSON object whose one contract-relevant member is `contentType`; every other member is caller-defined and stored verbatim.
|
||||
|
||||
Because it travels through JSON, this backend accepts only plain JSON values that survive serialization without changing meaning. It MUST fail loud, before sending, on values such as `Map`, `Set`, `Date`, non-finite numbers, negative zero, nested `undefined`, `bigint`, sparse arrays, symbol-keyed properties, non-enumerable properties, strings containing U+0000, class instances, and circular references. U+2028 and U+2029 are valid JSON string contents and MUST be accepted.
|
||||
|
||||
**This is a real capability reduction, not a tightening of a lossy path**, and a deployment moving from the browser backend to a server one must audit its metadata against it. The browser backend persists a structured clone, which carries `Date`, `Map`, `Set`, `ArrayBuffer`, `bigint`, `-0`, and cycles **faithfully**, and throws only for what it genuinely cannot represent. So these values do not degrade quietly today; over this transport a `put` that works locally starts failing. (Structured clone does drop symbol-keyed and non-enumerable properties, so those two are lossy on both sides.) U+0000 deserves its own mention in the other direction: it is legal in the browser and rejected by common server text and JSON column types, so without this rule it becomes a `500`.
|
||||
|
||||
The list above binds the **client**, before sending, and every JSON-backed store, before writing. In particular, callers may use `PgAssetStore` directly without crossing the HTTP boundary, so its `put` and metadata-replacing `replace` branches MUST enforce the same lossless domain before any registry or byte write. The decision MUST be made by **inspecting the value, never by trial-serializing it**: a `JSON.stringify` pre-flight runs caller code — `toJSON`, getters — before validation has looked at anything, and a stateful accessor can show the probe one value and the serializer another. A native `TypeError` from serialization MUST NOT escape; it is replaced by a typed error, as a native `SyntaxError` is on the read path. That is not only about error typing: V8's circular-structure message enumerates property names from the object it failed on, which would put caller-supplied metadata into an error message in direct violation of the egress rule above. A validation error names the offending path and MUST NOT include the offending value; that error-egress rule binds direct store calls exactly as it binds HTTP calls.
|
||||
|
||||
A server receives the output of a JSON parse, so most of that list cannot reach it — non-finite numbers are a syntax error and the rest are unrepresentable in JSON text. Its obligation is correspondingly short, and is what the `400` row of the error table refers to: reject a `meta` part that is not a JSON object, and reject `-0` and U+0000 inside any string. Those are the only members of the domain that survive parsing.
|
||||
|
||||
Nothing reads metadata back to a caller. It is persisted so that provenance recorded at generation time — what produced these bytes, from what prompt or narration, with what voice — survives alongside the asset for later manifest and export work. A deployment should know what it is holding: metadata accumulates caller-supplied text that no code path currently returns, and it is subject to whatever retention and privacy posture that text requires.
|
||||
|
||||
## Principal derivation and authorization
|
||||
|
||||
The principal is derived server-side from the authenticated session and **never appears in a path, query parameter, or body**. This is the same non-negotiable that governs the other layers: a client-submitted principal is not proof of identity, and trusting one turns every route here into a lateral-authorization vulnerability. Every route requires authentication; there is no anonymous surface. A deployment's `authenticate` hook returning a principal in a malformed shape is `500 INTERNAL_ERROR` rather than `400`, because that is the deployment's bug and not the caller's.
|
||||
|
||||
A principal carries a **required, opaque identity key**, and that key is what every registry entry is partitioned on. This differs from the document layer, whose principal is all-optional because documents are author assets with no per-row ownership model; assets are partitioned, so the field the partition is keyed on cannot be optional. A principal that does not carry one has no asset capability at all and receives `403 FORBIDDEN_ASSETS` on every route — never a shared partition. The distinction matters because an absent key is not a value: entries stored against one would collide into a single partition that every such principal could read, replace, and delete, which is a lateral-authorization hole produced by the type rather than by a bug.
|
||||
|
||||
Two of the three channels a principal or a hash could arrive through are closed structurally rather than by inspection: the route table admits no segment that could carry either, and no route takes a query string.
|
||||
|
||||
The third channel, headers, is closed by **deriving no identity from it**, and this contract deliberately does not attempt to police header names. A server MUST NOT take a principal, an owner, or a content hash from any request header except through the deployment's `authenticate` hook; no other header carries identity here, and one the implementation never consults for identity smuggles nothing. The rule is about what the implementation trusts, not a filter on traffic.
|
||||
|
||||
This is narrower than "reads no other header", and deliberately so: the transport headers below are read and acted on — `Content-Type` for the multipart boundary, `Content-Length` as an untrusted hint, and `Content-Encoding` rejected outright because it would defeat the decoded-size limits. Reading a header to decide how to frame or refuse a request is not the same as trusting one to say who is asking.
|
||||
|
||||
Rejecting suspicious header names instead would be worse than useless, and it is worth recording why so it is not reintroduced. A name-shaped denylist cannot distinguish a header that asserts identity from one whose name merely contains an identity word: `User-Agent` is caught by any pattern broad enough to catch `X-Remote-User`, and a server enforcing such a pattern answers `400` to every browser, SDK, and command-line client. More decisively, `X-Forwarded-User`, `X-Remote-User`, `On-Behalf-Of`, and their kin are exactly what an authenticating reverse proxy — `auth_request`, an OAuth2 proxy, an identity-aware gateway — injects for `authenticate` to read. A contract that rejects them forbids the deployment shape it most needs to support, and one that exempts them has rebuilt the enumeration it replaced.
|
||||
|
||||
A deployment that authenticates from front-door headers owns one obligation this contract states but cannot enforce: that front door MUST strip and reset those headers on every inbound request, so a client cannot supply its own. That is a property of the proxy, not of this handler.
|
||||
|
||||
Bodyless methods (`GET`, `HEAD`, `DELETE`) MUST reject **any** request body outright: a server that routes them without reading the body would let a prohibited field arrive unexamined. In the `meta` part of a write, a `principal` or `contentHash` member is rejected.
|
||||
|
||||
`authorizeAssets` is evaluated on the principal, the method, and the route **before the entry is read**, and MUST NOT receive the entry — otherwise a policy decision becomes entry-dependent and defeats the ordering rule below. It defaults to **allowing** any authenticated principal, matching document authorization rather than the deny-by-default of the administrative hooks. That default is not an absence of protection: the enforcing boundary is the principal recorded on every registry entry, checked on every route including every byte read. The hook is an additional policy layer for deployments that want one; the administrative hooks default to denial because they are privilege-escalation surfaces with no such per-row boundary behind them. A denial is `403` uniformly, on `DELETE` as on every other route — answering `204` to a caller whose policy forbids deletion would have them clear their reference and orphan the entry permanently.
|
||||
|
||||
### Indistinguishability
|
||||
|
||||
An id belonging to another principal and an id that was never allocated MUST produce **byte-identical** responses — same status, same code, same message, same details, and the same headers — on every route in the table. Headers are named explicitly because this contract puts an outcome-varying value in one: a later diagnostic or `Retry-After` could break indistinguishability exactly as an added `details` member would. `DELETE` answers `204` for both, and for an already-deleted id. Ownership MUST be checked **before** any classification that could vary the response, including anything derived from the entry's revision or media type; otherwise the classification itself becomes the oracle. `ASSET_NOT_FOUND` responses carry a fixed message and **no** `details`, so that a diagnostic added later cannot break this.
|
||||
|
||||
An asset id is an identifier, not a capability. Authorizing once and treating the id as a bearer token afterwards is forbidden: **every** byte read re-checks ownership. A URL valid for one principal must not serve bytes to another that obtained it.
|
||||
|
||||
## Errors
|
||||
|
||||
Every non-2xx response has this machine-readable JSON shape:
|
||||
|
||||
```json
|
||||
{
|
||||
"error": {
|
||||
"code": "VALIDATION_FAILED",
|
||||
"message": "@openmaic/storage: asset write body must carry a \"bytes\" part",
|
||||
"details": []
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
`details` is optional. One class of response cannot carry this envelope: a `431`, and anything else the framework emits before the handler runs, has neither a body of this shape nor an `X-Error-Code`. That is why the `431` row below names no code, and it is the reason the client rule is phrased as it is — **a response the client cannot classify is never a miss.**
|
||||
|
||||
Because `HEAD` responses have no body, every error response — on `HEAD` and, so that the two stay identical, on `GET` as well — MUST also carry the code in an `X-Error-Code` response header. Without it a client cannot tell `ASSET_NOT_FOUND` from `ROUTE_NOT_FOUND` from a gateway's own `404` on precisely the route it uses to decide whether a cached snapshot is still real. **A `HEAD` response a client cannot classify is never a miss:** it falls back to `GET`.
|
||||
|
||||
The client raises `HttpAssetStoreError` in every throwing case below.
|
||||
|
||||
| Condition | HTTP status | Error code | Client behavior |
|
||||
| --- | --- | --- | --- |
|
||||
| Malformed multipart, a missing or duplicate part, a query string, a prohibited header, a body on a bodyless method, or metadata outside the value domain (an *id* is opaque and never a validation failure) | `400` | `VALIDATION_FAILED` | Throw with the server message |
|
||||
| Request `Content-Type` is not `multipart/form-data` | `415` | `UNSUPPORTED_MEDIA_TYPE` | Throw |
|
||||
| Method not allowed on this route | `405` | `METHOD_NOT_ALLOWED` | Throw |
|
||||
| Metadata, bytes, or the whole request exceed the deployment's bound | `413` | `PAYLOAD_TOO_LARGE` | Throw |
|
||||
| The principal's logical bytes would exceed its quota | `507` | `ASSET_QUOTA_EXCEEDED` | Throw |
|
||||
| Request target exceeds the deployment's ceiling | `431` | (transport) | A transport limit for a pathologically long id, not an id-domain rejection |
|
||||
| No entry is stored under the id, or the entry is not this principal's | `404` | `ASSET_NOT_FOUND` | `resolve` returns `null`; `remove` succeeds; `replace` throws |
|
||||
| Route does not exist | `404` | `ROUTE_NOT_FOUND` | Throw |
|
||||
| Missing or invalid credential | `401` | `UNAUTHENTICATED` | Throw |
|
||||
| Principal may not perform the operation | `403` | `FORBIDDEN_ASSETS` | Throw |
|
||||
| Unexpected server failure | `500` | `INTERNAL_ERROR` | Throw; the handler does not expose internal details |
|
||||
|
||||
Only `ASSET_NOT_FOUND` becomes a miss, and the client MUST match the **code together with the status**. Status alone is not sufficient: `ROUTE_NOT_FOUND` shares its status and means a broken deployment, and a `401` or `403` MUST NEVER be reported as a miss. A gateway answering `401` while echoing an error code must not be read as "the asset was deleted" — that reading turns an auth outage into apparent data loss, and a caller that reacts by clearing references turns it into real data loss.
|
||||
|
||||
A response the client cannot interpret — a non-JSON body under a 2xx status, an allocation response without an `id` member — raises `MALFORMED_RESPONSE`, a client-side code with no server counterpart, carried by the same error type. The client never lets a native `SyntaxError`, `TypeError`, or `URIError` escape in its place.
|
||||
|
||||
The content hash MUST NOT appear in any `message` or `details`, on any path, including internal failures. `5xx` responses collapse to `INTERNAL_ERROR` with a fixed message for that reason as well as the usual one.
|
||||
|
||||
## Retry and atomicity guarantees
|
||||
|
||||
`GET` and `HEAD` are reads. `DELETE` is idempotent: removing an absent, already-removed, or foreign id succeeds. `PUT /assets/{id}/content` is idempotent in its effect on the bytes for the same body, though each application advances the revision — so a retry after an ambiguous failure leaves the bytes correct and the revision advanced twice, which is why the revision is opaque and compared only for equality.
|
||||
|
||||
**`POST /assets` is not idempotent and is not implicitly retry-safe.** Every call allocates, so a retried request that in fact succeeded leaves a second entry holding the same bytes. This is a direct consequence of server-assigned ids, and it is the property `POST` carries in the [RuntimeStore HTTP contract](./runtime-http-contract.md). A caller that retries blindly should expect an orphan; a caller that cannot tolerate one must track the allocation it received.
|
||||
|
||||
A failed write leaves **no registry entry**, and no entry ever survives pointing at bytes that were not stored. It may leave bytes that nothing references — the crash window described under the byte layer, which costs storage rather than correctness. Whether it does depends on the layer: one inside the transactional store rolls back with the transaction and leaves nothing, while an object store cannot join that transaction and strands an object for reconciliation to take. There is no compare-and-swap. Concurrent `replace` calls on one id are last-writer-wins, and the revision reflects the order the server applied them. A `resolve` racing a `replace` observes either the pre- or the post-replace committed entry and never a mixture of the two.
|
||||
@@ -1,6 +1,6 @@
|
||||
# RuntimeStore and DocumentStore reference server
|
||||
# RuntimeStore, DocumentStore, and AssetStore reference server
|
||||
|
||||
The `@openmaic/storage/server` subpath exports a Node-only HTTP request handler implementing the [RuntimeStore HTTP contract](./runtime-http-contract.md) and, when a document store is supplied, the [DocumentStore HTTP contract](./document-http-contract.md). It accepts injected `RuntimeStore` and `DocumentStore` implementations; the runnable `@openmaic/storage/server/reference` composition creates and initializes `PgRuntimeStore`, accepts an optional host-created document store, and demonstrates the required node-postgres checkout/transaction/release pattern.
|
||||
The `@openmaic/storage/server` subpath exports a Node-only HTTP request handler implementing the [RuntimeStore HTTP contract](./runtime-http-contract.md), plus the [DocumentStore HTTP contract](./document-http-contract.md) and [AssetStore HTTP contract](./asset-http-contract.md) when their stores are supplied. It accepts injected store implementations; the runnable `@openmaic/storage/server/reference` composition creates and initializes `PgRuntimeStore`, accepts optional host-created document and asset stores, and demonstrates the required node-postgres checkout/transaction/release pattern.
|
||||
|
||||
The OpenMAIC application also mounts these same composed handlers as an
|
||||
app-integrated Next.js route at `/api/persistence`. That embedded route is the
|
||||
@@ -20,8 +20,9 @@ DATABASE_URL=postgres://user:password@host/database PORT=3000 \
|
||||
|
||||
The executable `main()` binds to `127.0.0.1`; `createReferenceRuntimeServer()` only creates and returns an unbound Node `Server`. The default bearer token payload is used directly as the demo `learnerKey`, self-merge is the only allowed merge, and admin operations are denied. Supplying `documentStore` adds the DocumentStore routes to that same server; any authenticated principal is allowed by default, or `authorizeDocuments` can enforce deployment policy. The factory also accepts `authenticate`, `authorizeMerge`, `authorizeAdmin`, and validator overrides. Replace the policy hooks before exposing a deployment:
|
||||
|
||||
- `authenticate(req)` must validate a real credential and derive the canonical learner partition from server-controlled identity state.
|
||||
- `authenticate(req)` must validate a real credential and derive canonical learner and asset partition keys from server-controlled identity state.
|
||||
- `authorizeDocuments(principal, req)` must establish that the principal may access the requested author document operation. The default permits every authenticated principal.
|
||||
- `authorizeAssets(principal, req)` must establish that the principal may access the requested asset operation. The default permits every authenticated principal carrying an asset key; registry ownership remains mandatory on every operation.
|
||||
- `authorizeMerge(principal, fromKey, toKey)` must explicitly establish that the principal may migrate the complete source partition into the destination identity. Default denial is intentional.
|
||||
- `authorizeAdmin(principal)` must require a separately protected administrative role. Default denial is intentional.
|
||||
|
||||
@@ -53,6 +54,20 @@ the lower-level handler yourself. Full response, validation, version, and retry
|
||||
semantics are specified in the
|
||||
[DocumentStore HTTP contract](./document-http-contract.md).
|
||||
|
||||
## Asset endpoints
|
||||
|
||||
Every asset route requires an authenticated principal with an asset partition key. Supplying `assetStore` adds these routes to the composed server; `authorizeAssets` can apply an additional deployment policy before any registry entry is read.
|
||||
|
||||
| Method | Path | Purpose |
|
||||
| --- | --- | --- |
|
||||
| `POST` | `/assets` | Allocate a new id and store bytes |
|
||||
| `GET` | `/assets/{id}/content` | Read bytes and identity headers |
|
||||
| `HEAD` | `/assets/{id}/content` | Read identity headers without bytes |
|
||||
| `PUT` | `/assets/{id}/content` | Replace bytes behind an existing id |
|
||||
| `DELETE` | `/assets/{id}` | Remove an entry; absent and foreign ids are no-ops |
|
||||
|
||||
Writes use bounded `multipart/form-data`; reads are private and uncached. The full media-type allowlist, size limits, response headers, and client snapshot rules are specified in the [AssetStore HTTP contract](./asset-http-contract.md).
|
||||
|
||||
## Endpoint authorization matrix
|
||||
|
||||
The matrix treats learner, merge, and admin credentials as separate capabilities. An admin-only or merge-only credential does not implicitly own a learner partition; a deployment may combine capabilities, but every applicable check still has to pass.
|
||||
@@ -70,6 +85,11 @@ The matrix treats learner, merge, and admin credentials as separate capabilities
|
||||
| `DELETE /runtime/stages/{stageId}/learners/{learnerKey}` | Deny (`401`) | Allow | Deny (`403`) | Deny | Deny |
|
||||
| `DELETE /runtime/stages/{stageId}` | Deny (`401`) | Deny (`403`) | Deny (`403`) | Deny (`403`) | Allow |
|
||||
| `DELETE /runtime` | Deny (`401`) | Deny (`403`) | Deny (`403`) | Deny (`403`) | Allow |
|
||||
| `POST /assets` | Deny (`401`) | Allow in own partition | Allow in own partition | Deny without asset key | Deny without asset key |
|
||||
| `GET /assets/{id}/content` | Deny (`401`) | Allow for own id | Not found (`404`) | Deny without asset key | Deny without asset key |
|
||||
| `HEAD /assets/{id}/content` | Deny (`401`) | Allow for own id | Not found (`404`) | Deny without asset key | Deny without asset key |
|
||||
| `PUT /assets/{id}/content` | Deny (`401`) | Allow for own id | Not found (`404`) | Deny without asset key | Deny without asset key |
|
||||
| `DELETE /assets/{id}` | Deny (`401`) | Allow | Allow as no-op | Deny without asset key | Deny without asset key |
|
||||
|
||||
## Threat model
|
||||
|
||||
@@ -77,6 +97,8 @@ The matrix treats learner, merge, and admin credentials as separate capabilities
|
||||
|
||||
Session-scoped routes deliberately conceal whether another learner's session ID exists. A credential with a different `learnerKey` receives the same `404 SESSION_NOT_FOUND` as an absent session, and ownership is checked before future-version classification so version metadata cannot disclose existence. A principal with no `learnerKey` is different: it lacks the learner capability entirely and receives `403 FORBIDDEN_LEARNER` on every learner-scoped route.
|
||||
|
||||
An asset id is an identifier, not a bearer token. Every asset operation rechecks the authenticated principal's ownership, including every byte read. An id belonging to another principal and an id that was never allocated are indistinguishable on every route: reads and replacements return the same fixed `404 ASSET_NOT_FOUND`, while deletion succeeds as the same no-op.
|
||||
|
||||
Merge is a privilege-escalation boundary because it rewrites every source session across every stage. Merely owning either key is insufficient in a real identity system: the authorization hook must verify the account-linking or identity-upgrade proof for both the source and destination. The default is deny.
|
||||
|
||||
Stage cascade deletion and whole-runtime deletion are admin-plane capabilities. If exposed to ordinary learners they can erase every partition on one stage or across the entire runtime store, so both routes are controlled by the separate admin authorization hook and denied by default. Production systems should isolate admin credentials, audit decisions, protect against confused-deputy use, and avoid deriving admin authority from a learner-controlled claim.
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "@openmaic/storage",
|
||||
"version": "0.2.3",
|
||||
"version": "0.2.4",
|
||||
"description": "The MAIC pluggable persistence layer: document / runtime / KV / asset primitives with browser and HTTP backends, depending only on @openmaic/dsl.",
|
||||
"type": "module",
|
||||
"main": "./dist/index.js",
|
||||
@@ -23,6 +23,10 @@
|
||||
"types": "./dist/kv/http.d.ts",
|
||||
"import": "./dist/kv/http.js"
|
||||
},
|
||||
"./asset/http": {
|
||||
"types": "./dist/asset/http.d.ts",
|
||||
"import": "./dist/asset/http.js"
|
||||
},
|
||||
"./document/pg": {
|
||||
"types": "./dist/document/pg.d.ts",
|
||||
"import": "./dist/document/pg.js"
|
||||
@@ -31,6 +35,22 @@
|
||||
"types": "./dist/runtime/pg.d.ts",
|
||||
"import": "./dist/runtime/pg.js"
|
||||
},
|
||||
"./asset/pg": {
|
||||
"types": "./dist/asset/pg.d.ts",
|
||||
"import": "./dist/asset/pg.js"
|
||||
},
|
||||
"./asset/pg-bytes": {
|
||||
"types": "./dist/asset/pg-bytes.d.ts",
|
||||
"import": "./dist/asset/pg-bytes.js"
|
||||
},
|
||||
"./asset/s3-bytes": {
|
||||
"types": "./dist/asset/s3-bytes.d.ts",
|
||||
"import": "./dist/asset/s3-bytes.js"
|
||||
},
|
||||
"./asset/collector": {
|
||||
"types": "./dist/asset/collector.d.ts",
|
||||
"import": "./dist/asset/collector.js"
|
||||
},
|
||||
"./server": {
|
||||
"types": "./dist/server/index.d.ts",
|
||||
"import": "./dist/server/index.js"
|
||||
@@ -75,7 +95,16 @@
|
||||
"dependencies": {
|
||||
"@openmaic/dsl": "workspace:^"
|
||||
},
|
||||
"peerDependencies": {
|
||||
"@aws-sdk/client-s3": "^3.0.0"
|
||||
},
|
||||
"peerDependenciesMeta": {
|
||||
"@aws-sdk/client-s3": {
|
||||
"optional": true
|
||||
}
|
||||
},
|
||||
"devDependencies": {
|
||||
"@aws-sdk/client-s3": "^3.1048.0",
|
||||
"@electric-sql/pglite": "^0.3.14",
|
||||
"@types/pg": "^8.15.5",
|
||||
"fake-indexeddb": "^6.0.0",
|
||||
|
||||
@@ -3,9 +3,13 @@
|
||||
* byte table, the hashing operation, and the object-URL cache used by resolve.
|
||||
*
|
||||
* These are implementation building blocks, not a replaceable blob-backend
|
||||
* seam. The browser registry keeps byte writes, reference counting, and
|
||||
* reclamation in the same IndexedDB transaction. Replaceable blob storage is a
|
||||
* server-backend concern, where the server enforces consistency.
|
||||
* seam *for the browser*. The browser registry keeps byte writes, reference
|
||||
* counting, and reclamation in one IndexedDB transaction, and its reclamation
|
||||
* happens inline, so its bytes cannot move out of that transaction.
|
||||
*
|
||||
* A server backend does have a byte seam — see `./byte-store.js` — because its
|
||||
* reclamation is offline rather than inline, which is the property that lets
|
||||
* bytes sit outside the registry's transaction.
|
||||
*/
|
||||
import type { AssetRef, BinaryBlob } from '@openmaic/dsl';
|
||||
|
||||
|
||||
@@ -22,10 +22,16 @@
|
||||
* databases: reference counting happens *inside* the same transaction that
|
||||
* deletes the entry, so the last entry's removal can never reclaim bytes a
|
||||
* concurrent `put` has just adopted. The byte table is deliberately not a
|
||||
* replaceable browser-side backend: byte writes, reference counting, and
|
||||
* reclamation must share this transaction. A genuinely replaceable blob
|
||||
* service belongs to the server backend in delivery plan part 4, where the
|
||||
* server enforces consistency.
|
||||
* replaceable backend *here*: byte writes, reference counting, and reclamation
|
||||
* must share this transaction, because this store reclaims **inline** — a
|
||||
* `remove` that drops the last reference deletes the bytes in the same
|
||||
* transaction. Bytes outside that transaction could be deleted while a
|
||||
* concurrent `put` adopted them, with nothing able to serialize the two.
|
||||
*
|
||||
* The server backend is not bound this way, and the difference is reclamation
|
||||
* rather than storage. It collects offline, so no request path ever deletes
|
||||
* bytes, and its byte layer is genuinely pluggable — a column of the
|
||||
* transactional store or an object store keyed by content hash.
|
||||
*
|
||||
* On successful `put` calls, every returned value and every branch taken
|
||||
* reveals nothing about whether the bytes already existed — see
|
||||
|
||||
@@ -0,0 +1,62 @@
|
||||
/**
|
||||
* The pluggable byte layer beneath a server asset registry.
|
||||
*
|
||||
* The registry — ids, owners, media types, metadata, and the reference count —
|
||||
* always lives in a transactional store. The bytes do not have to: they may sit
|
||||
* in a column of that store, or in an object store addressed by content hash.
|
||||
* Two implementations ship, which is why this interface exists at all. An
|
||||
* interface with one implementation is not a seam, and this package has removed
|
||||
* one such seam before rather than carry it.
|
||||
*
|
||||
* What makes the layer separable is that **reclamation is offline**. If a
|
||||
* request could delete bytes, a concurrent write deduplicating onto those same
|
||||
* bytes could commit against them, and a byte layer outside the registry's
|
||||
* transaction would have no way to serialize the two — which is exactly what
|
||||
* would force bytes into that transaction. With collection moved out of every
|
||||
* request path, what remains is an ordering rule rather than an atomicity one:
|
||||
*
|
||||
* - bytes are written **before** the row that references them, so a crash
|
||||
* between the two costs storage and loses nothing (the reverse order would
|
||||
* leave a registry entry pointing at bytes that were never stored);
|
||||
* - bytes are deleted **after** the last row referencing them, and only by the
|
||||
* offline collector;
|
||||
* - the reference count and the decision to collect are serialized on the blob
|
||||
* row, which every implementation keeps regardless of where bytes live.
|
||||
*
|
||||
* Note what this interface deliberately does not have: no reference counting,
|
||||
* no ownership, no deduplication decision. Those belong to the registry. A byte
|
||||
* layer stores and returns bytes under a hash and knows nothing about who
|
||||
* references them, which is what keeps the global byte pool free of principals.
|
||||
*/
|
||||
import type { ContentHash } from './blob.js';
|
||||
|
||||
export interface AssetByteStore {
|
||||
/**
|
||||
* Store bytes under their content hash.
|
||||
*
|
||||
* Unconditional and idempotent: writing bytes that are already stored MUST
|
||||
* succeed and MUST NOT report that they were present. The registry above
|
||||
* relies on this to keep its two write paths indistinguishable, so an
|
||||
* implementation must not read first, and must not return anything that
|
||||
* varies with prior presence.
|
||||
*/
|
||||
write(hash: ContentHash, bytes: Uint8Array): Promise<void>;
|
||||
|
||||
/**
|
||||
* Read the bytes stored under a hash, or `null` if there are none.
|
||||
*
|
||||
* A miss is not an error. A registry entry whose bytes have been collected
|
||||
* out from under it resolves to a miss, and inventing an error here would
|
||||
* make collection observable.
|
||||
*/
|
||||
read(hash: ContentHash): Promise<Uint8Array | null>;
|
||||
|
||||
/**
|
||||
* Delete the bytes stored under a hash.
|
||||
*
|
||||
* **Only the offline collector calls this.** No request path may, which is
|
||||
* the property that lets bytes live outside the registry's transaction at
|
||||
* all. Idempotent: deleting bytes that are absent succeeds.
|
||||
*/
|
||||
delete(hash: ContentHash): Promise<void>;
|
||||
}
|
||||
@@ -0,0 +1,114 @@
|
||||
/** Offline reclamation for unreferenced server asset bytes. */
|
||||
import type { ContentHash } from './blob.js';
|
||||
import type { AssetByteStore } from './byte-store.js';
|
||||
import type { Queryable, WithTransaction } from '../runtime/pg.js';
|
||||
|
||||
/** One hour. A deployment may choose a longer retention window. */
|
||||
export const DEFAULT_ASSET_COLLECTION_GRACE_MS = 60 * 60 * 1000;
|
||||
|
||||
export interface AssetCollectorOptions {
|
||||
/** Pin each per-blob callback to a fresh PostgreSQL transaction. */
|
||||
withTransaction: WithTransaction;
|
||||
/** Minimum age of an unreferenced row before collection. Defaults to one hour. */
|
||||
graceMs?: number;
|
||||
/** Clock override for deterministic hosts and tests. */
|
||||
now?: () => Date;
|
||||
}
|
||||
|
||||
interface CandidateRow extends Record<string, unknown> {
|
||||
content_hash: ContentHash;
|
||||
}
|
||||
|
||||
interface TransactionalByteDeleter extends AssetByteStore {
|
||||
deleteWith(queryable: Queryable, hash: ContentHash): Promise<void>;
|
||||
}
|
||||
|
||||
function hasTransactionalDeleter(store: AssetByteStore): store is TransactionalByteDeleter {
|
||||
return 'deleteWith' in store && typeof store.deleteWith === 'function';
|
||||
}
|
||||
|
||||
function collectorFailure(): Error {
|
||||
return new Error('@openmaic/storage: asset collection failed');
|
||||
}
|
||||
|
||||
/**
|
||||
* Re-runnable collector for the byte rows left behind by request operations.
|
||||
*
|
||||
* This is the only component that calls `AssetByteStore.delete`. Hosts must
|
||||
* schedule it: leaving it unscheduled lets unreferenced storage grow without
|
||||
* bound.
|
||||
*/
|
||||
export class AssetCollector {
|
||||
private readonly transactionHook: WithTransaction;
|
||||
private readonly graceMs: number;
|
||||
private readonly now: () => Date;
|
||||
|
||||
constructor(
|
||||
private readonly queryable: Queryable,
|
||||
private readonly byteStore: AssetByteStore,
|
||||
options: AssetCollectorOptions,
|
||||
) {
|
||||
if (typeof options?.withTransaction !== 'function') {
|
||||
throw new Error('@openmaic/storage: withTransaction is required for AssetCollector');
|
||||
}
|
||||
const graceMs = options.graceMs ?? DEFAULT_ASSET_COLLECTION_GRACE_MS;
|
||||
if (!Number.isSafeInteger(graceMs) || graceMs < 0) {
|
||||
throw new Error('@openmaic/storage: graceMs must be a non-negative safe integer');
|
||||
}
|
||||
this.transactionHook = options.withTransaction;
|
||||
this.graceMs = graceMs;
|
||||
this.now = options.now ?? (() => new Date());
|
||||
}
|
||||
|
||||
async collect(): Promise<number> {
|
||||
const cutoff = new Date(this.now().getTime() - this.graceMs).toISOString();
|
||||
let candidates;
|
||||
try {
|
||||
candidates = await this.queryable.query<CandidateRow>(
|
||||
`SELECT content_hash
|
||||
FROM asset_blobs
|
||||
WHERE unreferenced_at < $1::timestamptz
|
||||
ORDER BY unreferenced_at ASC, content_hash ASC`,
|
||||
[cutoff],
|
||||
);
|
||||
} catch {
|
||||
throw collectorFailure();
|
||||
}
|
||||
|
||||
let collected = 0;
|
||||
for (const candidate of candidates.rows) {
|
||||
try {
|
||||
const didCollect = await this.transactionHook(async (queryable) => {
|
||||
const locked = await queryable.query<CandidateRow>(
|
||||
`SELECT content_hash
|
||||
FROM asset_blobs
|
||||
WHERE content_hash = $1
|
||||
AND unreferenced_at < $2::timestamptz
|
||||
AND NOT EXISTS (
|
||||
SELECT 1 FROM asset_entries WHERE content_hash = $1
|
||||
)
|
||||
FOR UPDATE`,
|
||||
[candidate.content_hash, cutoff],
|
||||
);
|
||||
if (!locked.rows[0]) return false;
|
||||
if (hasTransactionalDeleter(this.byteStore)) {
|
||||
await this.byteStore.deleteWith(queryable, candidate.content_hash);
|
||||
} else {
|
||||
await this.byteStore.delete(candidate.content_hash);
|
||||
}
|
||||
await queryable.query('DELETE FROM asset_blobs WHERE content_hash = $1', [
|
||||
candidate.content_hash,
|
||||
]);
|
||||
return true;
|
||||
});
|
||||
if (didCollect) collected += 1;
|
||||
} catch {
|
||||
throw collectorFailure();
|
||||
}
|
||||
}
|
||||
return collected;
|
||||
}
|
||||
}
|
||||
|
||||
export type { AssetByteStore } from './byte-store.js';
|
||||
export type { Queryable, WithTransaction } from '../runtime/pg.js';
|
||||
@@ -0,0 +1,498 @@
|
||||
import type { AssetMeta, AssetRef, BinaryBlob, StorageProvider } from '@openmaic/dsl';
|
||||
import { assertHttpBaseUrl } from '../http/base-url.js';
|
||||
import { assertJsonValue } from '../runtime/json-value.js';
|
||||
import { ObjectUrlCache } from './blob.js';
|
||||
import type { AssetId } from './id.js';
|
||||
|
||||
export interface HttpAssetHeadersContext {
|
||||
method: string;
|
||||
path: string;
|
||||
}
|
||||
|
||||
export type HttpAssetHeadersHook = (
|
||||
context: HttpAssetHeadersContext,
|
||||
) => HeadersInit | Promise<HeadersInit>;
|
||||
|
||||
export interface HttpAssetStoreOptions {
|
||||
/** Root URL before the contract's `/assets/...` paths. */
|
||||
baseUrl: string;
|
||||
/** Fetch implementation. Defaults to `globalThis.fetch`. */
|
||||
fetch?: typeof globalThis.fetch;
|
||||
/** Called for every request so deployments can attach authentication headers. */
|
||||
headers?: HttpAssetHeadersHook;
|
||||
/** Passed through to fetch unchanged. */
|
||||
credentials?: RequestCredentials;
|
||||
}
|
||||
|
||||
interface ErrorResponseBody {
|
||||
error?: { code?: unknown; message?: unknown; details?: unknown };
|
||||
}
|
||||
|
||||
interface ObjectUrlIdentity {
|
||||
revision: string;
|
||||
mediaType: string;
|
||||
}
|
||||
|
||||
// These fixed filenames preserve part bytes under standards-conforming parsers. Neither filename
|
||||
// is read by this package or derived from caller data.
|
||||
const ASSET_META_FILENAME = 'metadata.json';
|
||||
const ASSET_BYTES_FILENAME = 'asset';
|
||||
const DEFAULT_ASSET_CONTENT_TYPE = 'application/octet-stream';
|
||||
|
||||
/** An asset HTTP failure, retaining its machine-readable identity. */
|
||||
export class HttpAssetStoreError extends Error {
|
||||
constructor(
|
||||
readonly status: number,
|
||||
readonly code: string,
|
||||
message: string,
|
||||
readonly details?: unknown,
|
||||
) {
|
||||
super(message);
|
||||
this.name = 'HttpAssetStoreError';
|
||||
}
|
||||
}
|
||||
|
||||
function normalizeHeaders(init: HeadersInit | undefined): Record<string, string> {
|
||||
const normalized: Record<string, string> = {};
|
||||
const set = (name: string, value: string): void => {
|
||||
normalized[name.toLowerCase()] = value;
|
||||
};
|
||||
if (init === undefined) return normalized;
|
||||
if (Array.isArray(init)) {
|
||||
for (const [name, value] of init) set(name, value);
|
||||
} else if (typeof (init as Headers).forEach === 'function') {
|
||||
(init as Headers).forEach((value, name) => set(name, value));
|
||||
} else {
|
||||
for (const [name, value] of Object.entries(init)) set(name, value);
|
||||
}
|
||||
return normalized;
|
||||
}
|
||||
|
||||
function malformed(status: number, message: string): HttpAssetStoreError {
|
||||
return new HttpAssetStoreError(status, 'MALFORMED_RESPONSE', message);
|
||||
}
|
||||
|
||||
function localNotFound(): HttpAssetStoreError {
|
||||
return new HttpAssetStoreError(
|
||||
404,
|
||||
'ASSET_NOT_FOUND',
|
||||
'@openmaic/storage: no asset is stored under that id',
|
||||
);
|
||||
}
|
||||
|
||||
function addressableSegment(id: string): string | null {
|
||||
if (id === '' || id === '.' || id === '..') return null;
|
||||
try {
|
||||
return encodeURIComponent(id);
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
function assertMetadata(meta: AssetMeta): void {
|
||||
if (typeof meta !== 'object' || meta === null || Array.isArray(meta)) {
|
||||
throw new HttpAssetStoreError(
|
||||
0,
|
||||
'VALIDATION_FAILED',
|
||||
'@openmaic/storage: asset metadata must be a plain JSON object',
|
||||
);
|
||||
}
|
||||
if ('principal' in meta || 'contentHash' in meta) {
|
||||
throw new HttpAssetStoreError(
|
||||
0,
|
||||
'VALIDATION_FAILED',
|
||||
'@openmaic/storage: asset metadata contains a prohibited member',
|
||||
);
|
||||
}
|
||||
try {
|
||||
assertJsonValue(meta, 'asset metadata');
|
||||
} catch (error) {
|
||||
throw new HttpAssetStoreError(
|
||||
0,
|
||||
'VALIDATION_FAILED',
|
||||
error instanceof Error ? error.message : '@openmaic/storage: invalid asset metadata',
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
function responseIdentity(response: Response): ObjectUrlIdentity | null {
|
||||
const revision = response.headers.get('x-asset-revision');
|
||||
const mediaType = response.headers.get('content-type');
|
||||
return revision === null || revision === '' || mediaType === null || mediaType === ''
|
||||
? null
|
||||
: { revision, mediaType };
|
||||
}
|
||||
|
||||
function sameIdentity(left: ObjectUrlIdentity, right: ObjectUrlIdentity): boolean {
|
||||
return left.revision === right.revision && left.mediaType === right.mediaType;
|
||||
}
|
||||
|
||||
function containsBytes(haystack: Uint8Array, needle: Uint8Array): boolean {
|
||||
if (needle.byteLength === 0) return true;
|
||||
for (let offset = 0; offset <= haystack.byteLength - needle.byteLength; offset += 1) {
|
||||
let equal = true;
|
||||
for (let index = 0; index < needle.byteLength; index += 1) {
|
||||
if (haystack[offset + index] !== needle[index]) {
|
||||
equal = false;
|
||||
break;
|
||||
}
|
||||
}
|
||||
if (equal) return true;
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
/** AssetStore client that downloads bytes and mints authenticated object URLs locally. */
|
||||
export class HttpAssetStore implements StorageProvider {
|
||||
private readonly baseUrl: string;
|
||||
private readonly fetchImpl: typeof globalThis.fetch;
|
||||
private readonly headersHook: HttpAssetHeadersHook | undefined;
|
||||
private readonly credentials: RequestCredentials | undefined;
|
||||
private readonly urls = new ObjectUrlCache<ObjectUrlIdentity>(sameIdentity);
|
||||
private readonly identities = new Map<AssetRef, ObjectUrlIdentity>();
|
||||
private readonly inFlight = new Map<AssetRef, Promise<string | null>>();
|
||||
private readonly generations = new Map<AssetRef, number>();
|
||||
private closed = false;
|
||||
|
||||
constructor(options: HttpAssetStoreOptions) {
|
||||
const selectedFetch = options.fetch ?? globalThis.fetch;
|
||||
if (typeof selectedFetch !== 'function') {
|
||||
throw new Error('@openmaic/storage: HttpAssetStore requires a fetch implementation');
|
||||
}
|
||||
this.baseUrl = assertHttpBaseUrl(options.baseUrl, 'HttpAssetStore');
|
||||
this.fetchImpl = selectedFetch.bind(globalThis);
|
||||
this.headersHook = options.headers;
|
||||
this.credentials = options.credentials;
|
||||
}
|
||||
|
||||
private generation(id: AssetRef): number {
|
||||
return this.generations.get(id) ?? 0;
|
||||
}
|
||||
|
||||
private async headers(
|
||||
method: string,
|
||||
path: string,
|
||||
multipart: boolean,
|
||||
): Promise<Record<string, string>> {
|
||||
let headers: Record<string, string>;
|
||||
try {
|
||||
headers = normalizeHeaders(await this.headersHook?.({ method, path }));
|
||||
} catch {
|
||||
throw new HttpAssetStoreError(
|
||||
0,
|
||||
'HTTP_REQUEST_FAILED',
|
||||
'@openmaic/storage: asset request headers could not be constructed',
|
||||
);
|
||||
}
|
||||
if (multipart && headers['content-type'] !== undefined) {
|
||||
throw new HttpAssetStoreError(
|
||||
0,
|
||||
'CONTENT_TYPE_CONFLICT',
|
||||
'@openmaic/storage: the headers hook must not set Content-Type for multipart asset writes',
|
||||
);
|
||||
}
|
||||
return headers;
|
||||
}
|
||||
|
||||
private async fetchResponse(method: string, path: string, body?: Blob): Promise<Response> {
|
||||
if (this.closed) {
|
||||
throw new HttpAssetStoreError(
|
||||
0,
|
||||
'STORE_CLOSED',
|
||||
'@openmaic/storage: HttpAssetStore is closed',
|
||||
);
|
||||
}
|
||||
const headers = await this.headers(method, path, body !== undefined);
|
||||
if (body !== undefined) headers['content-type'] = body.type;
|
||||
try {
|
||||
return await this.fetchImpl(`${this.baseUrl}${path}`, {
|
||||
method,
|
||||
headers,
|
||||
...(this.credentials === undefined ? {} : { credentials: this.credentials }),
|
||||
...(method === 'GET' || method === 'HEAD' ? { cache: 'no-store' as RequestCache } : {}),
|
||||
...(body === undefined ? {} : { body }),
|
||||
});
|
||||
} catch {
|
||||
throw new HttpAssetStoreError(
|
||||
0,
|
||||
'HTTP_REQUEST_FAILED',
|
||||
'@openmaic/storage: asset HTTP request failed',
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
private async httpError(response: Response): Promise<HttpAssetStoreError> {
|
||||
let body: ErrorResponseBody | undefined;
|
||||
if (response.status !== 204) {
|
||||
try {
|
||||
body = (await response.json()) as ErrorResponseBody;
|
||||
} catch {
|
||||
// Preserve a typed HTTP error for a non-conforming response.
|
||||
}
|
||||
}
|
||||
const headerCode = response.headers.get('x-error-code');
|
||||
const bodyCode = typeof body?.error?.code === 'string' ? body.error.code : undefined;
|
||||
const code = bodyCode ?? headerCode ?? 'HTTP_ERROR';
|
||||
const message =
|
||||
typeof body?.error?.message === 'string'
|
||||
? body.error.message
|
||||
: `@openmaic/storage: asset HTTP request failed with status ${response.status}`;
|
||||
return new HttpAssetStoreError(response.status, code, message, body?.error?.details);
|
||||
}
|
||||
|
||||
private async writeForm(
|
||||
data: BinaryBlob,
|
||||
meta: AssetMeta | undefined,
|
||||
includeMeta: boolean,
|
||||
): Promise<Blob> {
|
||||
if (includeMeta) assertMetadata(meta ?? {});
|
||||
let bytes: ArrayBuffer;
|
||||
try {
|
||||
bytes = await data.arrayBuffer();
|
||||
} catch {
|
||||
throw new HttpAssetStoreError(
|
||||
0,
|
||||
'VALIDATION_FAILED',
|
||||
'@openmaic/storage: asset bytes could not be read',
|
||||
);
|
||||
}
|
||||
if (!/^[\x20-\x7e]*$/.test(data.type)) {
|
||||
throw new HttpAssetStoreError(
|
||||
0,
|
||||
'VALIDATION_FAILED',
|
||||
'@openmaic/storage: asset media type contains characters that cannot be carried in a header',
|
||||
);
|
||||
}
|
||||
let encoded: string | undefined;
|
||||
if (includeMeta) {
|
||||
try {
|
||||
encoded = JSON.stringify(meta ?? {});
|
||||
} catch {
|
||||
throw new HttpAssetStoreError(
|
||||
0,
|
||||
'VALIDATION_FAILED',
|
||||
'@openmaic/storage: asset metadata could not be serialized',
|
||||
);
|
||||
}
|
||||
}
|
||||
let boundary: string;
|
||||
try {
|
||||
const content = new Uint8Array(bytes);
|
||||
do {
|
||||
const random = new Uint8Array(16);
|
||||
globalThis.crypto.getRandomValues(random);
|
||||
const suffix = Array.from(random, (value) => value.toString(16).padStart(2, '0')).join('');
|
||||
boundary = `openmaic-${suffix}`;
|
||||
} while (
|
||||
encoded?.includes(boundary) === true ||
|
||||
containsBytes(content, new TextEncoder().encode(boundary))
|
||||
);
|
||||
} catch {
|
||||
throw new HttpAssetStoreError(
|
||||
0,
|
||||
'HTTP_REQUEST_FAILED',
|
||||
'@openmaic/storage: a safe multipart boundary could not be generated',
|
||||
);
|
||||
}
|
||||
const parts: BlobPart[] = [];
|
||||
if (encoded !== undefined) {
|
||||
parts.push(
|
||||
`--${boundary}\r\n` +
|
||||
`Content-Disposition: form-data; name="meta"; filename="${ASSET_META_FILENAME}"\r\n` +
|
||||
'Content-Type: application/json\r\n\r\n' +
|
||||
`${encoded}\r\n`,
|
||||
);
|
||||
}
|
||||
parts.push(
|
||||
`--${boundary}\r\n` +
|
||||
`Content-Disposition: form-data; name="bytes"; filename="${ASSET_BYTES_FILENAME}"\r\n` +
|
||||
`Content-Type: ${data.type === '' ? DEFAULT_ASSET_CONTENT_TYPE : data.type}\r\n` +
|
||||
'\r\n',
|
||||
bytes,
|
||||
`\r\n--${boundary}--\r\n`,
|
||||
);
|
||||
return new Blob(parts, { type: `multipart/form-data; boundary=${boundary}` });
|
||||
}
|
||||
|
||||
async put(data: BinaryBlob, meta?: AssetMeta): Promise<AssetRef> {
|
||||
const path = '/assets';
|
||||
const response = await this.fetchResponse('POST', path, await this.writeForm(data, meta, true));
|
||||
if (!response.ok) throw await this.httpError(response);
|
||||
let body: unknown;
|
||||
try {
|
||||
body = await response.json();
|
||||
} catch {
|
||||
throw malformed(
|
||||
response.status,
|
||||
'@openmaic/storage: asset allocation response was not valid JSON',
|
||||
);
|
||||
}
|
||||
const id =
|
||||
typeof body === 'object' && body !== null && 'id' in body
|
||||
? (body as { id?: unknown }).id
|
||||
: undefined;
|
||||
if (
|
||||
response.status !== 201 ||
|
||||
typeof id !== 'string' ||
|
||||
id === '' ||
|
||||
response.headers.get('x-asset-revision') === null ||
|
||||
response.headers.get('x-asset-revision') === ''
|
||||
) {
|
||||
throw malformed(
|
||||
response.status,
|
||||
'@openmaic/storage: asset allocation response must carry an id and revision',
|
||||
);
|
||||
}
|
||||
return id;
|
||||
}
|
||||
|
||||
private async get(
|
||||
id: AssetRef,
|
||||
encoded: string,
|
||||
): Promise<{ url: string | null; retry: boolean }> {
|
||||
const generation = this.generation(id);
|
||||
const response = await this.fetchResponse('GET', `/assets/${encoded}/content`);
|
||||
if (!response.ok) {
|
||||
const error = await this.httpError(response);
|
||||
if (error.status === 404 && error.code === 'ASSET_NOT_FOUND') {
|
||||
this.identities.delete(id);
|
||||
await this.urls.invalidate(id);
|
||||
return { url: null, retry: false };
|
||||
}
|
||||
throw error;
|
||||
}
|
||||
const identity = responseIdentity(response);
|
||||
if (response.status !== 200 || identity === null) {
|
||||
throw malformed(
|
||||
response.status,
|
||||
'@openmaic/storage: asset byte response must carry content type and revision',
|
||||
);
|
||||
}
|
||||
let bytes: ArrayBuffer;
|
||||
try {
|
||||
bytes = await response.arrayBuffer();
|
||||
} catch {
|
||||
throw malformed(response.status, '@openmaic/storage: asset byte response could not be read');
|
||||
}
|
||||
if (this.generation(id) !== generation) return { url: null, retry: true };
|
||||
const url = await this.urls.resolve(id, identity, async () => {
|
||||
let minted: string;
|
||||
try {
|
||||
minted = URL.createObjectURL(new Blob([bytes], { type: identity.mediaType }));
|
||||
} catch {
|
||||
throw malformed(
|
||||
response.status,
|
||||
'@openmaic/storage: asset object URL could not be created',
|
||||
);
|
||||
}
|
||||
return { identity, url: minted };
|
||||
});
|
||||
if (this.generation(id) !== generation) {
|
||||
await this.urls.invalidate(id);
|
||||
return { url: null, retry: true };
|
||||
}
|
||||
this.identities.set(id, identity);
|
||||
return { url, retry: false };
|
||||
}
|
||||
|
||||
private async resolveFresh(id: AssetRef, encoded: string): Promise<string | null> {
|
||||
while (true) {
|
||||
const known = this.identities.get(id);
|
||||
if (known !== undefined) {
|
||||
const response = await this.fetchResponse('HEAD', `/assets/${encoded}/content`);
|
||||
if (response.ok) {
|
||||
const identity = response.status === 200 ? responseIdentity(response) : null;
|
||||
if (identity !== null && sameIdentity(known, identity)) {
|
||||
return this.urls.resolve(id, known, async () => null);
|
||||
}
|
||||
// A successful but unclassifiable HEAD is not a miss; GET decides.
|
||||
} else {
|
||||
const code = response.headers.get('x-error-code');
|
||||
if (response.status === 404 && code === 'ASSET_NOT_FOUND') {
|
||||
this.identities.delete(id);
|
||||
await this.urls.invalidate(id);
|
||||
return null;
|
||||
}
|
||||
if (code !== null) throw await this.httpError(response);
|
||||
// A HEAD error without a classifiable code falls back to GET.
|
||||
}
|
||||
}
|
||||
const result = await this.get(id, encoded);
|
||||
if (!result.retry) return result.url;
|
||||
}
|
||||
}
|
||||
|
||||
async resolve(id: AssetRef): Promise<string | null> {
|
||||
const encoded = addressableSegment(id);
|
||||
if (encoded === null) {
|
||||
this.identities.delete(id);
|
||||
await this.urls.invalidate(id);
|
||||
return null;
|
||||
}
|
||||
const current = this.inFlight.get(id);
|
||||
if (current !== undefined) return current;
|
||||
const resolution = this.resolveFresh(id, encoded);
|
||||
this.inFlight.set(id, resolution);
|
||||
try {
|
||||
return await resolution;
|
||||
} finally {
|
||||
if (this.inFlight.get(id) === resolution) this.inFlight.delete(id);
|
||||
}
|
||||
}
|
||||
|
||||
private async invalidate(id: AssetRef): Promise<void> {
|
||||
this.generations.set(id, this.generation(id) + 1);
|
||||
this.inFlight.delete(id);
|
||||
this.identities.delete(id);
|
||||
await this.urls.invalidate(id);
|
||||
}
|
||||
|
||||
async remove(id: AssetRef): Promise<void> {
|
||||
const encoded = addressableSegment(id);
|
||||
if (encoded === null) {
|
||||
await this.invalidate(id);
|
||||
return;
|
||||
}
|
||||
const response = await this.fetchResponse('DELETE', `/assets/${encoded}`);
|
||||
if (!response.ok) throw await this.httpError(response);
|
||||
if (response.status !== 204) {
|
||||
throw malformed(response.status, '@openmaic/storage: asset removal response must be 204');
|
||||
}
|
||||
await this.invalidate(id);
|
||||
}
|
||||
|
||||
async replace(id: AssetId, data: BinaryBlob, meta?: AssetMeta): Promise<void> {
|
||||
const encoded = addressableSegment(id);
|
||||
if (encoded === null) throw localNotFound();
|
||||
const includeMeta = meta !== undefined;
|
||||
const response = await this.fetchResponse(
|
||||
'PUT',
|
||||
`/assets/${encoded}/content`,
|
||||
await this.writeForm(data, meta, includeMeta),
|
||||
);
|
||||
if (!response.ok) throw await this.httpError(response);
|
||||
const revision = response.headers.get('x-asset-revision');
|
||||
if (response.status !== 204 || revision === null || revision === '') {
|
||||
throw malformed(
|
||||
response.status,
|
||||
'@openmaic/storage: asset replacement response must be 204 with a revision',
|
||||
);
|
||||
}
|
||||
await this.invalidate(id);
|
||||
}
|
||||
|
||||
async release(id: AssetRef): Promise<void> {
|
||||
this.generations.set(id, this.generation(id) + 1);
|
||||
this.inFlight.delete(id);
|
||||
this.identities.delete(id);
|
||||
await this.urls.release(id);
|
||||
}
|
||||
|
||||
async close(): Promise<void> {
|
||||
if (this.closed) return;
|
||||
this.closed = true;
|
||||
this.inFlight.clear();
|
||||
this.identities.clear();
|
||||
await this.urls.close();
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,91 @@
|
||||
/**
|
||||
* PostgreSQL byte storage for the server asset registry.
|
||||
*
|
||||
* Asset bytes stored here flow through PostgreSQL's write-ahead log, replicas,
|
||||
* and backups. Deployments should size their asset limit with that replication
|
||||
* and backup volume in mind.
|
||||
*/
|
||||
import type { ContentHash } from './blob.js';
|
||||
import type { AssetByteStore } from './byte-store.js';
|
||||
import type { Queryable } from '../runtime/pg.js';
|
||||
|
||||
function byteStoreFailure(operation: string): Error {
|
||||
return new Error(`@openmaic/storage: PostgreSQL asset byte ${operation} failed`);
|
||||
}
|
||||
|
||||
interface ByteRow extends Record<string, unknown> {
|
||||
bytes: Uint8Array | ArrayBuffer | null;
|
||||
}
|
||||
|
||||
function asUint8Array(value: Uint8Array | ArrayBuffer): Uint8Array {
|
||||
if (value instanceof Uint8Array) return new Uint8Array(value);
|
||||
return new Uint8Array(value.slice(0));
|
||||
}
|
||||
|
||||
export class PgAssetByteStore implements AssetByteStore {
|
||||
constructor(private readonly queryable: Queryable) {}
|
||||
|
||||
async write(hash: ContentHash, bytes: Uint8Array): Promise<void> {
|
||||
try {
|
||||
await this.queryable.query(
|
||||
`INSERT INTO asset_blobs (content_hash, byte_size, bytes, unreferenced_at)
|
||||
VALUES ($1, $2, $3, now())
|
||||
ON CONFLICT (content_hash) DO UPDATE
|
||||
SET byte_size = EXCLUDED.byte_size,
|
||||
bytes = EXCLUDED.bytes,
|
||||
unreferenced_at = now()`,
|
||||
[hash, bytes.byteLength, bytes],
|
||||
);
|
||||
} catch {
|
||||
throw byteStoreFailure('write');
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Use a transaction-pinned queryable for the registry's coordinated rewrite.
|
||||
* This is an implementation hook, not an additional byte-store capability.
|
||||
*/
|
||||
async writeWith(queryable: Queryable, hash: ContentHash, bytes: Uint8Array): Promise<void> {
|
||||
await queryable.query(
|
||||
`UPDATE asset_blobs
|
||||
SET byte_size = $2,
|
||||
bytes = $3
|
||||
WHERE content_hash = $1`,
|
||||
[hash, bytes.byteLength, bytes],
|
||||
);
|
||||
}
|
||||
|
||||
async read(hash: ContentHash): Promise<Uint8Array | null> {
|
||||
try {
|
||||
return await this.readWith(this.queryable, hash);
|
||||
} catch {
|
||||
throw byteStoreFailure('read');
|
||||
}
|
||||
}
|
||||
|
||||
/** Use a transaction-pinned queryable when the registry resolves an entry. */
|
||||
async readWith(queryable: Queryable, hash: ContentHash): Promise<Uint8Array | null> {
|
||||
const result = await queryable.query<ByteRow>(
|
||||
'SELECT bytes FROM asset_blobs WHERE content_hash = $1',
|
||||
[hash],
|
||||
);
|
||||
const bytes = result.rows[0]?.bytes;
|
||||
return bytes === undefined || bytes === null ? null : asUint8Array(bytes);
|
||||
}
|
||||
|
||||
async delete(hash: ContentHash): Promise<void> {
|
||||
try {
|
||||
await this.deleteWith(this.queryable, hash);
|
||||
} catch {
|
||||
throw byteStoreFailure('delete');
|
||||
}
|
||||
}
|
||||
|
||||
/** Use the collector's transaction-pinned queryable for byte clearing. */
|
||||
async deleteWith(queryable: Queryable, hash: ContentHash): Promise<void> {
|
||||
await queryable.query('UPDATE asset_blobs SET bytes = NULL WHERE content_hash = $1', [hash]);
|
||||
}
|
||||
}
|
||||
|
||||
export type { AssetByteStore } from './byte-store.js';
|
||||
export type { Queryable } from '../runtime/pg.js';
|
||||
@@ -0,0 +1,470 @@
|
||||
/**
|
||||
* PostgreSQL registry for server assets over a pluggable byte layer.
|
||||
*
|
||||
* Within a write, the blob row is claimed first, then the bytes are written,
|
||||
* then the entry that references them. The order is load-bearing at both ends:
|
||||
* claiming the row first serializes the write against the collector, which
|
||||
* could otherwise delete those bytes while this upsert waited for the lock, and
|
||||
* writing bytes before the entry keeps every surviving entry backed by bytes
|
||||
* that were actually stored. The byte write is unconditional, so both existence
|
||||
* paths emit the same statements.
|
||||
*
|
||||
* Request paths never delete bytes; the offline collector is the only reclaimer.
|
||||
* `withTransaction` must pin all queries in its body to one freshly checked-out
|
||||
* transaction.
|
||||
*/
|
||||
import type { AssetMeta, AssetRef, BinaryBlob } from '@openmaic/dsl';
|
||||
import { contentHashOf, type ContentHash } from './blob.js';
|
||||
import type { AssetByteStore } from './byte-store.js';
|
||||
import { newAssetId, type AssetId } from './id.js';
|
||||
import {
|
||||
AssetNotFoundError,
|
||||
AssetQuotaExceededError,
|
||||
type AssetBytes,
|
||||
type AssetIdentity,
|
||||
type AssetPrincipal,
|
||||
type AssetStore,
|
||||
} from './types.js';
|
||||
import { assertJsonValue, isLosslessJsonString } from '../runtime/json-value.js';
|
||||
import type { Queryable, WithTransaction } from '../runtime/pg.js';
|
||||
|
||||
export type { QueryResult, Queryable, WithTransaction } from '../runtime/pg.js';
|
||||
export type { AssetByteStore } from './byte-store.js';
|
||||
export type { AssetBytes, AssetIdentity, AssetPrincipal, AssetStore } from './types.js';
|
||||
export { AssetNotFoundError, AssetQuotaExceededError } from './types.js';
|
||||
|
||||
export interface PgAssetStoreOptions {
|
||||
/** Pin each callback to a fresh PostgreSQL transaction and connection. */
|
||||
withTransaction: WithTransaction;
|
||||
/** Physical byte storage used beneath the registry. */
|
||||
byteStore: AssetByteStore;
|
||||
/** Optional logical-byte ceiling for each principal. */
|
||||
quotaBytes?: number;
|
||||
}
|
||||
|
||||
/** One PGlite-compatible statement per entry, in dependency order. */
|
||||
export const ASSET_PG_SCHEMA: readonly string[] = [
|
||||
`CREATE TABLE IF NOT EXISTS asset_blobs (
|
||||
content_hash TEXT PRIMARY KEY,
|
||||
byte_size BIGINT NOT NULL,
|
||||
bytes BYTEA,
|
||||
unreferenced_at TIMESTAMPTZ
|
||||
)`,
|
||||
`CREATE TABLE IF NOT EXISTS asset_entries (
|
||||
id TEXT PRIMARY KEY,
|
||||
principal TEXT NOT NULL,
|
||||
content_hash TEXT NOT NULL REFERENCES asset_blobs(content_hash),
|
||||
mime TEXT NOT NULL,
|
||||
meta JSONB NOT NULL,
|
||||
revision INTEGER NOT NULL DEFAULT 1,
|
||||
created_at DOUBLE PRECISION NOT NULL
|
||||
)`,
|
||||
`CREATE INDEX IF NOT EXISTS asset_entries_principal_idx
|
||||
ON asset_entries (principal, id)`,
|
||||
`CREATE INDEX IF NOT EXISTS asset_entries_content_hash_idx
|
||||
ON asset_entries (content_hash)`,
|
||||
`CREATE INDEX IF NOT EXISTS asset_blobs_unreferenced_idx
|
||||
ON asset_blobs (unreferenced_at) WHERE unreferenced_at IS NOT NULL`,
|
||||
];
|
||||
|
||||
export async function ensureAssetSchema(queryable: Queryable): Promise<void> {
|
||||
for (const statement of ASSET_PG_SCHEMA) await queryable.query(statement);
|
||||
}
|
||||
|
||||
interface UsageRow extends Record<string, unknown> {
|
||||
logical_bytes: number | string;
|
||||
}
|
||||
|
||||
interface ReplaceUsageRow extends UsageRow {
|
||||
current_bytes: number | string;
|
||||
}
|
||||
|
||||
interface EntryRow extends Record<string, unknown> {
|
||||
content_hash: ContentHash;
|
||||
mime: string;
|
||||
meta: unknown;
|
||||
revision: number | string;
|
||||
}
|
||||
|
||||
interface IdentityRow extends Record<string, unknown> {
|
||||
mime: string;
|
||||
revision: number | string;
|
||||
byte_size: number | string;
|
||||
}
|
||||
|
||||
interface HashRow extends Record<string, unknown> {
|
||||
content_hash: ContentHash;
|
||||
}
|
||||
|
||||
interface TransactionalByteWriter extends AssetByteStore {
|
||||
writeWith(queryable: Queryable, hash: ContentHash, bytes: Uint8Array): Promise<void>;
|
||||
}
|
||||
|
||||
interface TransactionalByteReader extends AssetByteStore {
|
||||
readWith(queryable: Queryable, hash: ContentHash): Promise<Uint8Array | null>;
|
||||
}
|
||||
|
||||
function hasTransactionalWriter(store: AssetByteStore): store is TransactionalByteWriter {
|
||||
return 'writeWith' in store && typeof store.writeWith === 'function';
|
||||
}
|
||||
|
||||
function hasTransactionalReader(store: AssetByteStore): store is TransactionalByteReader {
|
||||
return 'readWith' in store && typeof store.readWith === 'function';
|
||||
}
|
||||
|
||||
function registryFailure(operation: string): Error {
|
||||
return new Error(`@openmaic/storage: asset registry ${operation} failed`);
|
||||
}
|
||||
|
||||
class RegistryAssetNotFound extends Error {}
|
||||
|
||||
class RegistryAssetQuotaExceeded extends Error {}
|
||||
|
||||
function encodeMeta(meta: AssetMeta): string {
|
||||
assertJsonValue(meta, 'asset metadata');
|
||||
try {
|
||||
const encoded = JSON.stringify(meta);
|
||||
if (encoded === undefined) throw new TypeError('not serializable');
|
||||
return encoded;
|
||||
} catch {
|
||||
throw new Error('@openmaic/storage: asset metadata is not JSON-serializable');
|
||||
}
|
||||
}
|
||||
|
||||
function byteView(buffer: ArrayBuffer): Uint8Array {
|
||||
return new Uint8Array(buffer);
|
||||
}
|
||||
|
||||
export class PgAssetStore implements AssetStore {
|
||||
private readonly transactionHook: WithTransaction;
|
||||
private readonly byteStore: AssetByteStore;
|
||||
private readonly quotaBytes?: number;
|
||||
|
||||
constructor(
|
||||
private readonly queryable: Queryable,
|
||||
options: PgAssetStoreOptions,
|
||||
) {
|
||||
if (typeof options?.withTransaction !== 'function') {
|
||||
throw new Error(
|
||||
'@openmaic/storage: withTransaction is required and must pin a fresh connection and transaction for every call',
|
||||
);
|
||||
}
|
||||
if (!options.byteStore) {
|
||||
throw new Error('@openmaic/storage: byteStore is required for PgAssetStore');
|
||||
}
|
||||
if (
|
||||
options.quotaBytes !== undefined &&
|
||||
(!Number.isSafeInteger(options.quotaBytes) || options.quotaBytes < 0)
|
||||
) {
|
||||
throw new Error('@openmaic/storage: quotaBytes must be a non-negative safe integer');
|
||||
}
|
||||
this.transactionHook = options.withTransaction;
|
||||
this.byteStore = options.byteStore;
|
||||
this.quotaBytes = options.quotaBytes;
|
||||
}
|
||||
|
||||
private transaction<T>(body: (queryable: Queryable) => Promise<T>): Promise<T> {
|
||||
return this.transactionHook(body);
|
||||
}
|
||||
|
||||
private async coordinatedWrite(
|
||||
queryable: Queryable,
|
||||
hash: ContentHash,
|
||||
bytes: Uint8Array,
|
||||
): Promise<void> {
|
||||
if (hasTransactionalWriter(this.byteStore)) {
|
||||
await this.byteStore.writeWith(queryable, hash, bytes);
|
||||
} else {
|
||||
await this.byteStore.write(hash, bytes);
|
||||
}
|
||||
}
|
||||
|
||||
private readBytes(queryable: Queryable, hash: ContentHash): Promise<Uint8Array | null> {
|
||||
return hasTransactionalReader(this.byteStore)
|
||||
? this.byteStore.readWith(queryable, hash)
|
||||
: this.byteStore.read(hash);
|
||||
}
|
||||
|
||||
/**
|
||||
* Serialize this principal's writes before reading their usage.
|
||||
*
|
||||
* A quota read outside the write transaction is stale by the time it is
|
||||
* used: two concurrent writes both observe the old total, both pass, and the
|
||||
* principal ends up over quota by as much as the concurrency allows. The
|
||||
* lock is transaction-scoped, so it releases on commit or rollback, and it
|
||||
* is taken only when a quota is configured -- a branch on deployment
|
||||
* configuration, never on data, so it discloses nothing.
|
||||
*/
|
||||
private async lockPrincipal(queryable: Queryable, principal: AssetPrincipal): Promise<void> {
|
||||
// hashtextextended is 64-bit: hashtext is 32-bit, and colliding principals
|
||||
// would block each other for the whole transaction -- which spans a byte
|
||||
// write that may be a network upload.
|
||||
await queryable.query('SELECT pg_advisory_xact_lock(hashtextextended($1, 0))', [principal.key]);
|
||||
}
|
||||
|
||||
private async assertPutQuota(
|
||||
queryable: Queryable,
|
||||
principal: AssetPrincipal,
|
||||
addedBytes: number,
|
||||
): Promise<void> {
|
||||
if (this.quotaBytes === undefined) return;
|
||||
await this.lockPrincipal(queryable, principal);
|
||||
let result;
|
||||
try {
|
||||
result = await queryable.query<UsageRow>(
|
||||
`SELECT COALESCE(SUM(blobs.byte_size), 0)::text AS logical_bytes
|
||||
FROM asset_entries AS entries
|
||||
JOIN asset_blobs AS blobs ON blobs.content_hash = entries.content_hash
|
||||
WHERE entries.principal = $1`,
|
||||
[principal.key],
|
||||
);
|
||||
} catch {
|
||||
throw registryFailure('quota check');
|
||||
}
|
||||
const used = Number(result.rows[0]?.logical_bytes ?? 0);
|
||||
if (used + addedBytes > this.quotaBytes) throw new RegistryAssetQuotaExceeded();
|
||||
}
|
||||
|
||||
private async assertReplaceQuota(
|
||||
queryable: Queryable,
|
||||
principal: AssetPrincipal,
|
||||
ref: AssetId,
|
||||
replacementBytes: number,
|
||||
): Promise<void> {
|
||||
if (this.quotaBytes === undefined) return;
|
||||
await this.lockPrincipal(queryable, principal);
|
||||
let result;
|
||||
try {
|
||||
result = await queryable.query<ReplaceUsageRow>(
|
||||
`SELECT current_blob.byte_size::text AS current_bytes,
|
||||
usage.logical_bytes
|
||||
FROM asset_entries AS current_entry
|
||||
JOIN asset_blobs AS current_blob
|
||||
ON current_blob.content_hash = current_entry.content_hash
|
||||
CROSS JOIN (
|
||||
SELECT COALESCE(SUM(blobs.byte_size), 0)::text AS logical_bytes
|
||||
FROM asset_entries AS entries
|
||||
JOIN asset_blobs AS blobs ON blobs.content_hash = entries.content_hash
|
||||
WHERE entries.principal = $2
|
||||
) AS usage
|
||||
WHERE current_entry.id = $1 AND current_entry.principal = $2`,
|
||||
[ref, principal.key],
|
||||
);
|
||||
} catch {
|
||||
throw registryFailure('quota check');
|
||||
}
|
||||
const row = result.rows[0];
|
||||
if (!row) throw new RegistryAssetNotFound();
|
||||
if (
|
||||
Number(row.logical_bytes) - Number(row.current_bytes) + replacementBytes >
|
||||
this.quotaBytes
|
||||
) {
|
||||
throw new RegistryAssetQuotaExceeded();
|
||||
}
|
||||
}
|
||||
|
||||
async put(principal: AssetPrincipal, data: BinaryBlob, meta?: AssetMeta): Promise<AssetId> {
|
||||
const storedMeta = meta ?? {};
|
||||
const encodedMeta = encodeMeta(storedMeta);
|
||||
const mime = storedMeta.contentType ?? data.type;
|
||||
const { contentHash, bytes: buffer } = await contentHashOf(data);
|
||||
const bytes = byteView(buffer);
|
||||
let id: AssetId;
|
||||
try {
|
||||
id = newAssetId();
|
||||
} catch {
|
||||
throw registryFailure('put');
|
||||
}
|
||||
|
||||
try {
|
||||
await this.transaction(async (queryable) => {
|
||||
// Inside the transaction, and before anything is written: a check on
|
||||
// the pool is already stale when it is acted on.
|
||||
await this.assertPutQuota(queryable, principal, bytes.byteLength);
|
||||
await queryable.query(
|
||||
`INSERT INTO asset_blobs (content_hash, byte_size, unreferenced_at)
|
||||
VALUES ($1, $2, NULL)
|
||||
ON CONFLICT (content_hash) DO UPDATE
|
||||
SET unreferenced_at = NULL,
|
||||
byte_size = EXCLUDED.byte_size`,
|
||||
[contentHash, bytes.byteLength],
|
||||
);
|
||||
// Bytes are written only after the upsert above has taken the blob
|
||||
// row's lock. Writing before it instead would not be safe: the
|
||||
// collector can hold that lock, delete those bytes, and commit while
|
||||
// this upsert waits, leaving a fresh entry pointing at nothing. The
|
||||
// write is unconditional, so both existence paths still emit the same
|
||||
// sequence. The entry is inserted after it, so no row ever references
|
||||
// bytes that were not stored first.
|
||||
await this.coordinatedWrite(queryable, contentHash, bytes);
|
||||
await queryable.query(
|
||||
`INSERT INTO asset_entries
|
||||
(id, principal, content_hash, mime, meta, revision, created_at)
|
||||
VALUES ($1, $2, $3, $4, $5::jsonb, 1, $6)`,
|
||||
[id, principal.key, contentHash, mime, encodedMeta, Date.now()],
|
||||
);
|
||||
});
|
||||
} catch (error) {
|
||||
if (error instanceof RegistryAssetQuotaExceeded) throw new AssetQuotaExceededError();
|
||||
throw registryFailure('put');
|
||||
}
|
||||
return id;
|
||||
}
|
||||
|
||||
async resolve(principal: AssetPrincipal, ref: AssetRef): Promise<AssetBytes | null> {
|
||||
if (!isLosslessJsonString(ref) || !isLosslessJsonString(principal.key)) return null;
|
||||
try {
|
||||
return await this.transaction(async (queryable) => {
|
||||
const result = await queryable.query<EntryRow>(
|
||||
`SELECT content_hash, mime, revision
|
||||
FROM asset_entries
|
||||
WHERE id = $1 AND principal = $2`,
|
||||
[ref, principal.key],
|
||||
);
|
||||
const entry = result.rows[0];
|
||||
if (!entry) return null;
|
||||
const locked = await queryable.query(
|
||||
`SELECT 1
|
||||
FROM asset_blobs
|
||||
WHERE content_hash = $1
|
||||
FOR SHARE`,
|
||||
[entry.content_hash],
|
||||
);
|
||||
if (!locked.rows[0]) return null;
|
||||
const bytes = await this.readBytes(queryable, entry.content_hash);
|
||||
if (bytes === null) return null;
|
||||
return { bytes, mime: entry.mime, revision: Number(entry.revision) };
|
||||
});
|
||||
} catch {
|
||||
throw registryFailure('resolve');
|
||||
}
|
||||
}
|
||||
|
||||
async identify(principal: AssetPrincipal, ref: AssetRef): Promise<AssetIdentity | null> {
|
||||
if (!isLosslessJsonString(ref) || !isLosslessJsonString(principal.key)) return null;
|
||||
try {
|
||||
const result = await this.queryable.query<IdentityRow>(
|
||||
`SELECT entries.mime, entries.revision, blobs.byte_size
|
||||
FROM asset_entries AS entries
|
||||
JOIN asset_blobs AS blobs ON blobs.content_hash = entries.content_hash
|
||||
WHERE entries.id = $1 AND entries.principal = $2`,
|
||||
[ref, principal.key],
|
||||
);
|
||||
const identity = result.rows[0];
|
||||
if (!identity) return null;
|
||||
return {
|
||||
mime: identity.mime,
|
||||
revision: Number(identity.revision),
|
||||
byteLength: Number(identity.byte_size),
|
||||
};
|
||||
} catch {
|
||||
throw registryFailure('identify');
|
||||
}
|
||||
}
|
||||
|
||||
async remove(principal: AssetPrincipal, ref: AssetRef): Promise<void> {
|
||||
if (!isLosslessJsonString(ref) || !isLosslessJsonString(principal.key)) return;
|
||||
try {
|
||||
await this.transaction(async (queryable) => {
|
||||
const deleted = await queryable.query<HashRow>(
|
||||
`DELETE FROM asset_entries
|
||||
WHERE id = $1 AND principal = $2
|
||||
RETURNING content_hash`,
|
||||
[ref, principal.key],
|
||||
);
|
||||
const hash = deleted.rows[0]?.content_hash;
|
||||
if (!hash) return;
|
||||
await queryable.query(
|
||||
`UPDATE asset_blobs
|
||||
SET unreferenced_at = now()
|
||||
WHERE content_hash = $1
|
||||
AND NOT EXISTS (
|
||||
SELECT 1 FROM asset_entries WHERE content_hash = $1
|
||||
)`,
|
||||
[hash],
|
||||
);
|
||||
});
|
||||
} catch {
|
||||
throw registryFailure('remove');
|
||||
}
|
||||
}
|
||||
|
||||
async replace(
|
||||
principal: AssetPrincipal,
|
||||
ref: AssetId,
|
||||
data: BinaryBlob,
|
||||
meta?: AssetMeta,
|
||||
): Promise<number> {
|
||||
if (!isLosslessJsonString(ref) || !isLosslessJsonString(principal.key)) {
|
||||
throw new AssetNotFoundError();
|
||||
}
|
||||
const storedMeta = meta === undefined ? undefined : meta;
|
||||
const encodedMeta = storedMeta === undefined ? undefined : encodeMeta(storedMeta);
|
||||
const replacementMime = storedMeta?.contentType ?? data.type;
|
||||
const { contentHash, bytes: buffer } = await contentHashOf(data);
|
||||
const bytes = byteView(buffer);
|
||||
try {
|
||||
return await this.transaction(async (queryable) => {
|
||||
await this.assertReplaceQuota(queryable, principal, ref, bytes.byteLength);
|
||||
const existing = await queryable.query<EntryRow>(
|
||||
`SELECT content_hash, mime, meta, revision
|
||||
FROM asset_entries
|
||||
WHERE id = $1 AND principal = $2
|
||||
FOR UPDATE`,
|
||||
[ref, principal.key],
|
||||
);
|
||||
const oldEntry = existing.rows[0];
|
||||
if (!oldEntry) throw new RegistryAssetNotFound();
|
||||
|
||||
await queryable.query(
|
||||
`INSERT INTO asset_blobs (content_hash, byte_size, unreferenced_at)
|
||||
VALUES ($1, $2, NULL)
|
||||
ON CONFLICT (content_hash) DO UPDATE
|
||||
SET unreferenced_at = NULL,
|
||||
byte_size = EXCLUDED.byte_size`,
|
||||
[contentHash, bytes.byteLength],
|
||||
);
|
||||
await this.coordinatedWrite(queryable, contentHash, bytes);
|
||||
|
||||
let updated;
|
||||
if (storedMeta === undefined) {
|
||||
updated = await queryable.query<{ revision: number | string }>(
|
||||
`UPDATE asset_entries
|
||||
SET content_hash = $3,
|
||||
mime = CASE WHEN $4 = '' THEN mime ELSE $4 END,
|
||||
revision = revision + 1
|
||||
WHERE id = $1 AND principal = $2
|
||||
RETURNING revision`,
|
||||
[ref, principal.key, contentHash, data.type],
|
||||
);
|
||||
} else {
|
||||
updated = await queryable.query<{ revision: number | string }>(
|
||||
`UPDATE asset_entries
|
||||
SET content_hash = $3,
|
||||
mime = $4,
|
||||
meta = $5::jsonb,
|
||||
revision = revision + 1
|
||||
WHERE id = $1 AND principal = $2
|
||||
RETURNING revision`,
|
||||
[ref, principal.key, contentHash, replacementMime, encodedMeta],
|
||||
);
|
||||
}
|
||||
|
||||
await queryable.query(
|
||||
`UPDATE asset_blobs
|
||||
SET unreferenced_at = now()
|
||||
WHERE content_hash = $1
|
||||
AND NOT EXISTS (
|
||||
SELECT 1 FROM asset_entries WHERE content_hash = $1
|
||||
)`,
|
||||
[oldEntry.content_hash],
|
||||
);
|
||||
return Number(updated.rows[0]!.revision);
|
||||
});
|
||||
} catch (error) {
|
||||
if (error instanceof RegistryAssetNotFound) throw new AssetNotFoundError();
|
||||
if (error instanceof RegistryAssetQuotaExceeded) throw new AssetQuotaExceededError();
|
||||
throw registryFailure('replace');
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,99 @@
|
||||
/**
|
||||
* S3 byte storage for the server asset registry.
|
||||
*
|
||||
* Each object key is exactly its content hash. A crash after PUT and before the
|
||||
* registry row is committed leaves an object that reference counting cannot
|
||||
* see. Configure a bucket lifecycle policy for such old objects, or
|
||||
* periodically reconcile bucket keys against `asset_blobs`. Hash-named orphans
|
||||
* are harmless and a later write of the same bytes overwrites them
|
||||
* idempotently; no pending-object state is required.
|
||||
*/
|
||||
import {
|
||||
DeleteObjectCommand,
|
||||
GetObjectCommand,
|
||||
PutObjectCommand,
|
||||
type S3Client,
|
||||
} from '@aws-sdk/client-s3';
|
||||
import type { ContentHash } from './blob.js';
|
||||
import type { AssetByteStore } from './byte-store.js';
|
||||
|
||||
export interface S3AssetByteStoreOptions {
|
||||
/** An AWS SDK v3 S3 client, or a compatible client used by a test double. */
|
||||
client: Pick<S3Client, 'send'>;
|
||||
/** Bucket dedicated to content-hash-named asset objects. */
|
||||
bucket: string;
|
||||
}
|
||||
|
||||
function s3Failure(operation: string): Error {
|
||||
return new Error(`@openmaic/storage: S3 asset byte ${operation} failed`);
|
||||
}
|
||||
|
||||
/**
|
||||
* Whether an error means *this key* is absent, and nothing else.
|
||||
*
|
||||
* Deliberately narrow. A bare `404` also covers `NoSuchBucket`, a misdirected
|
||||
* endpoint, and a revoked access point — conditions under which the bytes very
|
||||
* much still exist. Reporting one of those as an absent object walks into the
|
||||
* failure the contract warns about for `401`: the caller reads "deleted",
|
||||
* clears a live registry reference, and a storage outage becomes real data
|
||||
* loss. Only key-absent codes map to a miss; everything else propagates and
|
||||
* surfaces as `500 INTERNAL_ERROR`.
|
||||
*/
|
||||
function isNotFound(error: unknown): boolean {
|
||||
if (typeof error !== 'object' || error === null) return false;
|
||||
const candidate = error as { name?: unknown; Code?: unknown; code?: unknown };
|
||||
return (
|
||||
candidate.name === 'NoSuchKey' ||
|
||||
candidate.Code === 'NoSuchKey' ||
|
||||
candidate.code === 'NoSuchKey'
|
||||
);
|
||||
}
|
||||
|
||||
export class S3AssetByteStore implements AssetByteStore {
|
||||
private readonly client: Pick<S3Client, 'send'>;
|
||||
private readonly bucket: string;
|
||||
|
||||
constructor(options: S3AssetByteStoreOptions) {
|
||||
this.client = options.client;
|
||||
this.bucket = options.bucket;
|
||||
}
|
||||
|
||||
async write(hash: ContentHash, bytes: Uint8Array): Promise<void> {
|
||||
try {
|
||||
await this.client.send(
|
||||
new PutObjectCommand({
|
||||
Bucket: this.bucket,
|
||||
Key: hash,
|
||||
Body: bytes,
|
||||
ContentLength: bytes.byteLength,
|
||||
}),
|
||||
);
|
||||
} catch {
|
||||
throw s3Failure('write');
|
||||
}
|
||||
}
|
||||
|
||||
async read(hash: ContentHash): Promise<Uint8Array | null> {
|
||||
try {
|
||||
const output = await this.client.send(
|
||||
new GetObjectCommand({ Bucket: this.bucket, Key: hash }),
|
||||
);
|
||||
if (!output.Body) throw s3Failure('read');
|
||||
return Uint8Array.from(await output.Body.transformToByteArray());
|
||||
} catch (error) {
|
||||
if (isNotFound(error)) return null;
|
||||
throw s3Failure('read');
|
||||
}
|
||||
}
|
||||
|
||||
async delete(hash: ContentHash): Promise<void> {
|
||||
try {
|
||||
await this.client.send(new DeleteObjectCommand({ Bucket: this.bucket, Key: hash }));
|
||||
} catch (error) {
|
||||
if (isNotFound(error)) return;
|
||||
throw s3Failure('delete');
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
export type { AssetByteStore } from './byte-store.js';
|
||||
@@ -0,0 +1,273 @@
|
||||
/**
|
||||
* The server-side asset store interface.
|
||||
*
|
||||
* The backend shape only: the handler's options live in `../server/asset.js`,
|
||||
* which can name a Node request type that this browser-reachable module cannot
|
||||
* import. The backends themselves, the HTTP handler, and the HTTP client are
|
||||
* separate units.
|
||||
*
|
||||
* The difference from the browser backend is the principal. In a browser the
|
||||
* store *is* the boundary — one origin, one database, one user — so ownership
|
||||
* needs no representation. On a server one registry holds every principal's
|
||||
* entries, so every operation takes the principal it acts on behalf of, and
|
||||
* that principal is the enforcing boundary. See `docs/asset-http-contract.md`,
|
||||
* which is normative for everything below.
|
||||
*/
|
||||
import type { AssetMeta, AssetRef, BinaryBlob } from '@openmaic/dsl';
|
||||
|
||||
import type { AssetId } from './id.js';
|
||||
|
||||
/**
|
||||
* The identity an operation acts on behalf of.
|
||||
*
|
||||
* Derived server-side from an authenticated session. It is never accepted from
|
||||
* a request path, query parameter, or body: a client-submitted principal is not
|
||||
* proof of identity, and trusting one makes every operation a
|
||||
* lateral-authorization vulnerability.
|
||||
*
|
||||
* `key` is required, and deliberately so. Assets are partitioned per principal,
|
||||
* and every registry entry is partitioned on this value; an optional field
|
||||
* would make `{}` a conforming principal and collapse every principal lacking
|
||||
* one into a single shared partition they could all read, replace, and delete.
|
||||
* That is why this does not copy the document layer's all-optional principal —
|
||||
* documents are author assets with no per-row ownership model, assets are not.
|
||||
* A deployment whose authenticator cannot produce a key has no asset capability
|
||||
* and must be refused, not given a partition keyed on nothing.
|
||||
*/
|
||||
export interface AssetPrincipal {
|
||||
/** Opaque, deployment-defined partition key. Compared, never parsed. */
|
||||
readonly key: string;
|
||||
/** Carried so one principal object can serve several layers. Not the partition key. */
|
||||
readonly learnerKey?: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* Raised when an id names no entry this principal holds.
|
||||
*
|
||||
* A class rather than an interface because a handler has to tell "no such
|
||||
* entry" (a `404`) from an internal failure (a `500`), and the alternatives are
|
||||
* both antipatterns: classifying from an error string, or duck-typing a `code`
|
||||
* member with no shared constructor two backends could agree on.
|
||||
*
|
||||
* An unknown id and another principal's id raise this identically — a
|
||||
* difference between them is an existence oracle.
|
||||
*/
|
||||
export class AssetNotFoundError extends Error {
|
||||
readonly code = 'ASSET_NOT_FOUND' as const;
|
||||
|
||||
constructor(message = '@openmaic/storage: no asset is stored under that id') {
|
||||
super(message);
|
||||
this.name = 'AssetNotFoundError';
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Raised when a write would take the principal past its logical byte quota.
|
||||
*
|
||||
* Declared for the same reason as {@link AssetNotFoundError}: a handler must
|
||||
* map this to its own status rather than collapsing it into an internal error,
|
||||
* and a caller must be able to tell "you are over quota" from "the server
|
||||
* broke" and from "this one request was too large".
|
||||
*/
|
||||
export class AssetQuotaExceededError extends Error {
|
||||
readonly code = 'ASSET_QUOTA_EXCEEDED' as const;
|
||||
|
||||
constructor(message = '@openmaic/storage: asset quota exceeded for this principal') {
|
||||
super(message);
|
||||
this.name = 'AssetQuotaExceededError';
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Media types served inline by default.
|
||||
*
|
||||
* A deployment may narrow this. Widening it is a decision about executable
|
||||
* content, which is what {@link EXCLUDED_RENDERABLE_TYPES} exists to bound.
|
||||
*/
|
||||
export const DEFAULT_RENDERABLE_TYPES: readonly string[] = [
|
||||
'image/png',
|
||||
'image/jpeg',
|
||||
'image/gif',
|
||||
'image/webp',
|
||||
'image/avif',
|
||||
'audio/mpeg',
|
||||
'audio/mp4',
|
||||
'audio/ogg',
|
||||
'audio/wav',
|
||||
'audio/webm',
|
||||
'video/mp4',
|
||||
'video/webm',
|
||||
'video/ogg',
|
||||
];
|
||||
|
||||
/**
|
||||
* Media types that MUST never be served inline, whatever a deployment
|
||||
* configures.
|
||||
*
|
||||
* Each executes script in a browsing context, so serving one same-origin turns
|
||||
* stored bytes into stored script. A handler rejects a configured allowlist
|
||||
* containing any of these at construction rather than trusting the prose: this
|
||||
* is the one setting that converts a storage bug into cross-site scripting, and
|
||||
* a rule enforced only by documentation is not enforced.
|
||||
*/
|
||||
export const EXCLUDED_RENDERABLE_TYPES: readonly string[] = [
|
||||
'image/svg+xml',
|
||||
'image/svg',
|
||||
'text/html',
|
||||
'application/xhtml+xml',
|
||||
'text/xml',
|
||||
'application/xml',
|
||||
'application/pdf',
|
||||
];
|
||||
|
||||
/**
|
||||
* A store of allocated assets, partitioned by principal.
|
||||
*
|
||||
* Server implementations write bytes before the registry transaction and
|
||||
* reclaim them only through an offline collector. The registry transaction
|
||||
* owns ids, principals, metadata, revisions, and the reference-count decision;
|
||||
* the byte layer may live in PostgreSQL or an object store. A crash between the
|
||||
* byte write and the registry transaction may therefore leave harmless orphan
|
||||
* bytes, but it must never leave a registry entry that points at bytes which
|
||||
* were not stored.
|
||||
*
|
||||
* Request paths never delete bytes. Reads hold a shared blob-row lock while
|
||||
* accessing the byte layer, and collection takes an exclusive lock before
|
||||
* deletion. This separation is what makes the byte layer replaceable without
|
||||
* changing the observable store semantics.
|
||||
*/
|
||||
export interface AssetStore {
|
||||
/**
|
||||
* Store bytes and return a newly allocated id.
|
||||
*
|
||||
* Every successful call allocates a **new** id, including for bytes the
|
||||
* registry already holds. Returning an existing id would let any caller test
|
||||
* whether arbitrary bytes are already stored, which is an existence oracle
|
||||
* over data the caller never stored.
|
||||
*
|
||||
* The requirement covers more than the return value: every branch taken and
|
||||
* every value returned on the success path must be independent of whether
|
||||
* the bytes were already present. An implementation writes the bytes
|
||||
* unconditionally rather than reading first, so both cases execute the same
|
||||
* statements in the same order.
|
||||
*
|
||||
* Quota, metering, and cost stay outside that guarantee. Quota MUST be
|
||||
* accounted on the principal's logical bytes — the sum of bytes referenced by
|
||||
* that principal's entries — never on bytes physically newly written, and the
|
||||
* check MUST precede any physical write, raising
|
||||
* {@link AssetQuotaExceededError}, so that a capacity failure is never what a
|
||||
* caller observes first.
|
||||
*/
|
||||
put(principal: AssetPrincipal, data: BinaryBlob, meta?: AssetMeta): Promise<AssetId>;
|
||||
|
||||
/**
|
||||
* Read the response identity stored under an id without reading its bytes.
|
||||
*
|
||||
* Ownership and miss behavior are identical to {@link resolve}. The byte
|
||||
* length is registry data used only to reproduce the `GET` response headers
|
||||
* on a bytes-free HTTP `HEAD` request.
|
||||
*/
|
||||
identify(principal: AssetPrincipal, ref: AssetRef): Promise<AssetIdentity | null>;
|
||||
|
||||
/**
|
||||
* Read the bytes stored under an id, or `null` if there are none.
|
||||
*
|
||||
* An id this registry never allocated, an id belonging to another principal,
|
||||
* an id from another id space, an empty string, a string containing NUL — all
|
||||
* are misses, and none is an error. There is no id validator to disagree
|
||||
* with, and an implementation that rejected an id for its shape would answer
|
||||
* a question the caller is not entitled to ask.
|
||||
*
|
||||
* An entry whose bytes are gone is also a miss; inventing an error there
|
||||
* would make reclamation observable. That state must be reachable only
|
||||
* through host-level collection or tampering, never through concurrent
|
||||
* operations on this interface.
|
||||
*
|
||||
* The returned media type and revision MUST come from the same transactional
|
||||
* snapshot as the bytes, or a concurrent `replace` pairs one revision's
|
||||
* headers with another revision's bytes.
|
||||
*/
|
||||
resolve(principal: AssetPrincipal, ref: AssetRef): Promise<AssetBytes | null>;
|
||||
|
||||
/**
|
||||
* Remove the entry stored under an id.
|
||||
*
|
||||
* A no-op for an unknown id, another principal's id, and an already-removed
|
||||
* id — all three succeed, indistinguishably. Any difference between them is
|
||||
* an existence oracle.
|
||||
*
|
||||
* When the removed entry was the last one naming its bytes, those bytes
|
||||
* become reclaimable. The count that decides this spans all principals, which
|
||||
* is what makes global deduplication reclaimable, and its value must not
|
||||
* appear in any response.
|
||||
*/
|
||||
remove(principal: AssetPrincipal, ref: AssetRef): Promise<void>;
|
||||
|
||||
/**
|
||||
* Replace the bytes behind an allocated id without changing the id.
|
||||
*
|
||||
* This is what makes an allocated id a stable reference: bytes can be
|
||||
* regenerated in place without rewriting every document that points at them.
|
||||
* It has no counterpart on the DSL storage interface, which is why a server
|
||||
* backend must implement it explicitly — a backend without it silently loses
|
||||
* in-place regeneration.
|
||||
*
|
||||
* `meta` is optional, and the two branches differ in a way callers depend on.
|
||||
* Supplied, it replaces the entry's metadata and the media type comes from its
|
||||
* `contentType` when that member is present — including when it is the empty
|
||||
* string — falling back to the blob's own type only when it is absent.
|
||||
* **Omitted, the entry's metadata is retained** and its media type is
|
||||
* retained unless the blob carries one. Regenerating bytes while keeping
|
||||
* recorded provenance is the reason this method exists, so an implementation
|
||||
* that folded omission into an empty object would erase that provenance on
|
||||
* every regeneration.
|
||||
*
|
||||
* Advances and returns the entry's revision. The exact value lets an HTTP
|
||||
* handler report the revision produced by this write without a racy follow-up
|
||||
* read. Rejects an unknown id and another principal's id identically, with
|
||||
* {@link AssetNotFoundError}.
|
||||
*/
|
||||
replace(
|
||||
principal: AssetPrincipal,
|
||||
ref: AssetId,
|
||||
data: BinaryBlob,
|
||||
meta?: AssetMeta,
|
||||
): Promise<number>;
|
||||
}
|
||||
|
||||
/**
|
||||
* The bytes stored under an id, with the identity needed to revalidate them.
|
||||
*/
|
||||
export interface AssetBytes {
|
||||
/** The stored bytes. */
|
||||
readonly bytes: Uint8Array;
|
||||
/**
|
||||
* The recorded media type, used to label the response.
|
||||
*
|
||||
* Labelling is subject to the renderable allowlist: a type outside it — and
|
||||
* the empty string, which is what an untyped blob with an absent
|
||||
* `contentType` records — is relabelled and served as an attachment. It is
|
||||
* still served: relabelling is not refusal.
|
||||
*/
|
||||
readonly mime: string;
|
||||
/**
|
||||
* A monotonically increasing counter on the registry entry, starting at 1 and
|
||||
* advanced by each `replace`.
|
||||
*
|
||||
* Opaque except for equality comparison, and **never derived from the
|
||||
* content**. A content-derived validator is the content hash under another
|
||||
* name: it would disclose byte equality across ids to anyone holding two of
|
||||
* them, which is precisely what the registry layer exists to prevent.
|
||||
*/
|
||||
readonly revision: number;
|
||||
}
|
||||
|
||||
/** Registry identity and representation length for a bytes-free read. */
|
||||
export interface AssetIdentity {
|
||||
/** The recorded media type, subject to the HTTP renderable allowlist. */
|
||||
readonly mime: string;
|
||||
/** The registry entry revision. */
|
||||
readonly revision: number;
|
||||
/** The stored representation length, in bytes. */
|
||||
readonly byteLength: number;
|
||||
}
|
||||
@@ -15,9 +15,11 @@
|
||||
* allocated `AssetId` names a registry entry and the registry names
|
||||
* content-addressed bytes (#1007). Its byte table is embedded in the registry's
|
||||
* IndexedDB database so writes, reference counting, and reclamation share one
|
||||
* transaction. A replaceable blob interface belongs to the future asset server
|
||||
* backend, where consistency is enforced server-side; no HTTP asset backend is
|
||||
* exposed here yet.
|
||||
* transaction, and its inline reclamation is what keeps them there. A server
|
||||
* backend collects bytes offline instead, which lets its byte layer be
|
||||
* pluggable — a column of the transactional store, or an object store keyed by
|
||||
* content hash. The HTTP backend downloads bytes through authenticated fetches
|
||||
* and mints object URLs locally.
|
||||
*/
|
||||
export type { DeviceSafeKVStore, KVScope, KVStore, LocalKVStore } from './kv/types.js';
|
||||
export { assertKVScope, DEFAULT_KV_SCOPE, KVScopeViolationError } from './kv/types.js';
|
||||
@@ -33,7 +35,36 @@ export {
|
||||
type HttpKVStoreOptions,
|
||||
} from './kv/http.js';
|
||||
export { BrowserAssetStore, type BrowserAssetStoreOptions } from './asset/browser-store.js';
|
||||
export {
|
||||
HttpAssetStore,
|
||||
HttpAssetStoreError,
|
||||
type HttpAssetHeadersContext,
|
||||
type HttpAssetHeadersHook,
|
||||
type HttpAssetStoreOptions,
|
||||
} from './asset/http.js';
|
||||
export { newAssetId, toAssetId, type AssetId } from './asset/id.js';
|
||||
export {
|
||||
AssetNotFoundError,
|
||||
AssetQuotaExceededError,
|
||||
DEFAULT_RENDERABLE_TYPES,
|
||||
EXCLUDED_RENDERABLE_TYPES,
|
||||
type AssetBytes,
|
||||
type AssetIdentity,
|
||||
type AssetPrincipal,
|
||||
type AssetStore,
|
||||
} from './asset/types.js';
|
||||
export {
|
||||
ASSET_PG_SCHEMA,
|
||||
PgAssetStore,
|
||||
ensureAssetSchema,
|
||||
type PgAssetStoreOptions,
|
||||
} from './asset/pg.js';
|
||||
export { PgAssetByteStore } from './asset/pg-bytes.js';
|
||||
export {
|
||||
AssetCollector,
|
||||
DEFAULT_ASSET_COLLECTION_GRACE_MS,
|
||||
type AssetCollectorOptions,
|
||||
} from './asset/collector.js';
|
||||
|
||||
export {
|
||||
kvPersistStorage,
|
||||
|
||||
@@ -0,0 +1,562 @@
|
||||
import type { IncomingMessage, RequestListener, ServerResponse } from 'node:http';
|
||||
import type { AssetMeta } from '@openmaic/dsl';
|
||||
import type { AssetId } from '../asset/id.js';
|
||||
import {
|
||||
AssetNotFoundError,
|
||||
AssetQuotaExceededError,
|
||||
DEFAULT_RENDERABLE_TYPES,
|
||||
EXCLUDED_RENDERABLE_TYPES,
|
||||
type AssetPrincipal,
|
||||
type AssetStore,
|
||||
} from '../asset/types.js';
|
||||
|
||||
/** Derive the asset principal from the authenticated request session. */
|
||||
export type AssetHttpAuthenticate = (req: IncomingMessage) => Promise<AssetPrincipal | undefined>;
|
||||
|
||||
/** Additional deployment policy, evaluated before the handler reads an entry. */
|
||||
export type AssetHttpAuthorize = (
|
||||
principal: AssetPrincipal,
|
||||
req: IncomingMessage,
|
||||
) => boolean | Promise<boolean>;
|
||||
|
||||
/** Options for the asset registry HTTP contract handler. */
|
||||
export interface AssetHttpHandlerOptions {
|
||||
authenticate: AssetHttpAuthenticate;
|
||||
/** Defaults to allowing every authenticated principal carrying an asset key. */
|
||||
authorizeAssets?: AssetHttpAuthorize;
|
||||
/** Exact media types served inline; executable document types are always refused. */
|
||||
renderableTypes?: readonly string[];
|
||||
/** Raw whole-request limit. Defaults to 33 MiB. */
|
||||
maxRequestBytes?: number;
|
||||
/** Decoded bytes-part limit. Defaults to 32 MiB. */
|
||||
maxAssetBytes?: number;
|
||||
/** Decoded metadata-part limit. Defaults to 64 KiB. */
|
||||
maxMetaBytes?: number;
|
||||
/** Multipart frame-count limit. Defaults to 8. */
|
||||
maxParts?: number;
|
||||
}
|
||||
|
||||
export const DEFAULT_MAX_ASSET_REQUEST_BYTES = 33 * 1024 * 1024;
|
||||
export const DEFAULT_MAX_ASSET_BYTES = 32 * 1024 * 1024;
|
||||
export const DEFAULT_MAX_ASSET_META_BYTES = 64 * 1024;
|
||||
export const DEFAULT_MAX_ASSET_PARTS = 8;
|
||||
|
||||
interface ErrorBody {
|
||||
error: { code: string; message: string; details?: unknown };
|
||||
}
|
||||
|
||||
interface ParsedWrite {
|
||||
data: Blob;
|
||||
meta?: AssetMeta;
|
||||
}
|
||||
|
||||
class AssetHttpError extends Error {
|
||||
constructor(
|
||||
readonly status: number,
|
||||
readonly code: string,
|
||||
message: string,
|
||||
readonly details?: unknown,
|
||||
readonly headers: Record<string, string> = {},
|
||||
) {
|
||||
super(message);
|
||||
}
|
||||
}
|
||||
|
||||
function validationFailure(message: string): AssetHttpError {
|
||||
return new AssetHttpError(400, 'VALIDATION_FAILED', message);
|
||||
}
|
||||
|
||||
function payloadTooLarge(message: string): AssetHttpError {
|
||||
return new AssetHttpError(413, 'PAYLOAD_TOO_LARGE', message);
|
||||
}
|
||||
|
||||
function sendJson(
|
||||
req: IncomingMessage,
|
||||
res: ServerResponse,
|
||||
status: number,
|
||||
body: unknown,
|
||||
headers: Record<string, string> = {},
|
||||
): void {
|
||||
res.sendDate = false;
|
||||
const encoded = JSON.stringify(body);
|
||||
const errorCode =
|
||||
status >= 300 && typeof body === 'object' && body !== null
|
||||
? (body as ErrorBody).error.code
|
||||
: undefined;
|
||||
res.writeHead(status, {
|
||||
'content-type': 'application/json',
|
||||
...(req.method === 'GET' || req.method === 'HEAD'
|
||||
? { 'content-length': String(Buffer.byteLength(encoded)) }
|
||||
: {}),
|
||||
...(errorCode !== undefined && (req.method === 'GET' || req.method === 'HEAD')
|
||||
? {
|
||||
'x-error-code': errorCode,
|
||||
'access-control-expose-headers': 'X-Asset-Revision, X-Error-Code',
|
||||
}
|
||||
: {}),
|
||||
...headers,
|
||||
});
|
||||
res.end(req.method === 'HEAD' ? undefined : encoded);
|
||||
}
|
||||
|
||||
function sendNoContent(res: ServerResponse, headers: Record<string, string> = {}): void {
|
||||
res.sendDate = false;
|
||||
res.writeHead(204, headers);
|
||||
res.end();
|
||||
}
|
||||
|
||||
function assertPositiveSafeInteger(value: number, label: string): void {
|
||||
if (!Number.isSafeInteger(value) || value <= 0) {
|
||||
throw new Error(`@openmaic/storage: ${label} must be a positive safe integer`);
|
||||
}
|
||||
}
|
||||
|
||||
function assertMultipartContentType(contentType: string | undefined): string {
|
||||
if (
|
||||
contentType === undefined ||
|
||||
contentType.split(';', 1)[0]?.trim().toLowerCase() !== 'multipart/form-data'
|
||||
) {
|
||||
throw new AssetHttpError(
|
||||
415,
|
||||
'UNSUPPORTED_MEDIA_TYPE',
|
||||
'@openmaic/storage: asset writes require multipart/form-data',
|
||||
);
|
||||
}
|
||||
return contentType;
|
||||
}
|
||||
|
||||
async function readBoundedBody(req: IncomingMessage, maxRequestBytes: number): Promise<Buffer> {
|
||||
const chunks: Buffer[] = [];
|
||||
let total = 0;
|
||||
for await (const chunk of req) {
|
||||
const buffer = typeof chunk === 'string' ? Buffer.from(chunk) : (chunk as Buffer);
|
||||
total += buffer.byteLength;
|
||||
if (total > maxRequestBytes) {
|
||||
throw payloadTooLarge(
|
||||
`@openmaic/storage: request body exceeds maxRequestBytes (${maxRequestBytes})`,
|
||||
);
|
||||
}
|
||||
chunks.push(buffer);
|
||||
}
|
||||
return Buffer.concat(chunks, total);
|
||||
}
|
||||
|
||||
async function assertBodyless(req: IncomingMessage): Promise<void> {
|
||||
for await (const chunk of req) {
|
||||
const size = typeof chunk === 'string' ? Buffer.byteLength(chunk) : chunk.byteLength;
|
||||
if (size > 0) {
|
||||
throw validationFailure('@openmaic/storage: this asset route does not accept a body');
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
function assertServerMetadataValue(value: unknown): void {
|
||||
const visit = (member: unknown): void => {
|
||||
if (typeof member === 'string' && member.includes('\u0000')) {
|
||||
throw validationFailure('@openmaic/storage: asset metadata contains U+0000');
|
||||
}
|
||||
if (typeof member === 'number' && Object.is(member, -0)) {
|
||||
throw validationFailure('@openmaic/storage: asset metadata contains negative zero');
|
||||
}
|
||||
if (Array.isArray(member)) {
|
||||
for (const nested of member) visit(nested);
|
||||
} else if (typeof member === 'object' && member !== null) {
|
||||
for (const [key, nested] of Object.entries(member)) {
|
||||
visit(key);
|
||||
visit(nested);
|
||||
}
|
||||
}
|
||||
};
|
||||
visit(value);
|
||||
}
|
||||
|
||||
async function parseMeta(part: Blob): Promise<AssetMeta> {
|
||||
if (part.type.split(';', 1)[0]?.trim().toLowerCase() !== 'application/json') {
|
||||
throw validationFailure('@openmaic/storage: the meta part must be application/json');
|
||||
}
|
||||
let text: string;
|
||||
try {
|
||||
text = new TextDecoder('utf-8', { fatal: true }).decode(await part.arrayBuffer());
|
||||
} catch {
|
||||
throw validationFailure('@openmaic/storage: the meta part must contain valid UTF-8');
|
||||
}
|
||||
let value: unknown;
|
||||
try {
|
||||
value = JSON.parse(text) as unknown;
|
||||
} catch {
|
||||
throw validationFailure('@openmaic/storage: the meta part must contain valid JSON');
|
||||
}
|
||||
if (typeof value !== 'object' || value === null || Array.isArray(value)) {
|
||||
throw validationFailure('@openmaic/storage: the meta part must contain a JSON object');
|
||||
}
|
||||
if ('principal' in value || 'contentHash' in value) {
|
||||
throw validationFailure('@openmaic/storage: asset metadata contains a prohibited member');
|
||||
}
|
||||
assertServerMetadataValue(value);
|
||||
return value as AssetMeta;
|
||||
}
|
||||
|
||||
async function readWrite(
|
||||
req: IncomingMessage,
|
||||
requiredMeta: boolean,
|
||||
limits: {
|
||||
maxRequestBytes: number;
|
||||
maxParts: number;
|
||||
maxMetaBytes: number;
|
||||
maxAssetBytes: number;
|
||||
},
|
||||
): Promise<ParsedWrite> {
|
||||
if (req.headers['content-encoding'] !== undefined) {
|
||||
throw validationFailure('@openmaic/storage: Content-Encoding is not accepted on asset writes');
|
||||
}
|
||||
const contentType = assertMultipartContentType(req.headers['content-type']);
|
||||
const body = await readBoundedBody(req, limits.maxRequestBytes);
|
||||
let form: FormData;
|
||||
try {
|
||||
const bytes = body.buffer.slice(
|
||||
body.byteOffset,
|
||||
body.byteOffset + body.byteLength,
|
||||
) as ArrayBuffer;
|
||||
form = await new Response(bytes, { headers: { 'content-type': contentType } }).formData();
|
||||
} catch {
|
||||
throw validationFailure('@openmaic/storage: malformed multipart body');
|
||||
}
|
||||
const parts = [...form.entries()];
|
||||
if (parts.length > limits.maxParts) {
|
||||
throw payloadTooLarge(
|
||||
`@openmaic/storage: asset write body exceeds maxParts (${limits.maxParts})`,
|
||||
);
|
||||
}
|
||||
const named = new Map<'meta' | 'bytes', string | Blob>();
|
||||
for (const [name, part] of parts) {
|
||||
if (name !== 'meta' && name !== 'bytes') {
|
||||
throw validationFailure('@openmaic/storage: asset write body contains an unrecognized part');
|
||||
}
|
||||
if (named.has(name)) {
|
||||
throw validationFailure('@openmaic/storage: asset write body contains a duplicate part');
|
||||
}
|
||||
named.set(name, part);
|
||||
}
|
||||
const expectedLengths = requiredMeta ? [2] : [1, 2];
|
||||
if (!expectedLengths.includes(parts.length)) {
|
||||
throw validationFailure('@openmaic/storage: asset write body has the wrong number of parts');
|
||||
}
|
||||
const metaPart = named.get('meta');
|
||||
const bytesPart = named.get('bytes');
|
||||
if (bytesPart === undefined) {
|
||||
throw validationFailure('@openmaic/storage: asset write body must carry a "bytes" part');
|
||||
}
|
||||
if (requiredMeta && metaPart === undefined) {
|
||||
throw validationFailure('@openmaic/storage: asset write body must carry a "meta" part');
|
||||
}
|
||||
if (metaPart !== undefined && parts[0]?.[0] !== 'meta') {
|
||||
throw validationFailure('@openmaic/storage: the meta part must precede the bytes part');
|
||||
}
|
||||
if (typeof bytesPart === 'string') {
|
||||
throw validationFailure('@openmaic/storage: the bytes part must be sent as a file');
|
||||
}
|
||||
if (typeof metaPart === 'string') {
|
||||
throw validationFailure('@openmaic/storage: the meta part must be sent as a file');
|
||||
}
|
||||
if (bytesPart.size > limits.maxAssetBytes) {
|
||||
throw payloadTooLarge(
|
||||
`@openmaic/storage: bytes part exceeds maxAssetBytes (${limits.maxAssetBytes})`,
|
||||
);
|
||||
}
|
||||
if (metaPart !== undefined) {
|
||||
if (metaPart.size > limits.maxMetaBytes) {
|
||||
throw payloadTooLarge(
|
||||
`@openmaic/storage: meta part exceeds maxMetaBytes (${limits.maxMetaBytes})`,
|
||||
);
|
||||
}
|
||||
}
|
||||
const meta = metaPart === undefined ? undefined : await parseMeta(metaPart);
|
||||
return {
|
||||
data: bytesPart,
|
||||
...(meta === undefined ? {} : { meta }),
|
||||
};
|
||||
}
|
||||
|
||||
function parsePath(req: IncomingMessage): string[] {
|
||||
const target = req.url ?? '/';
|
||||
if (target.includes('?')) {
|
||||
throw validationFailure('@openmaic/storage: asset routes do not accept a query string');
|
||||
}
|
||||
const raw = target.split('#', 1)[0] ?? '/';
|
||||
const rawParts = raw.split('/');
|
||||
if (rawParts[0] === '') rawParts.shift();
|
||||
try {
|
||||
return rawParts.map((part) => decodeURIComponent(part));
|
||||
} catch {
|
||||
throw validationFailure('@openmaic/storage: request path is not valid percent-encoded UTF-8');
|
||||
}
|
||||
}
|
||||
|
||||
function routeShape(parts: string[]): { kind: 'collection' | 'content' | 'item'; allow: string } {
|
||||
if (parts.length === 1 && parts[0] === 'assets') return { kind: 'collection', allow: 'POST' };
|
||||
if (parts.length === 3 && parts[0] === 'assets' && parts[2] === 'content') {
|
||||
return { kind: 'content', allow: 'GET, HEAD, PUT' };
|
||||
}
|
||||
if (parts.length === 2 && parts[0] === 'assets') return { kind: 'item', allow: 'DELETE' };
|
||||
throw new AssetHttpError(404, 'ROUTE_NOT_FOUND', 'route not found');
|
||||
}
|
||||
|
||||
function assertMethod(
|
||||
method: string,
|
||||
kind: 'collection' | 'content' | 'item',
|
||||
allow: string,
|
||||
): void {
|
||||
const accepted =
|
||||
(kind === 'collection' && method === 'POST') ||
|
||||
(kind === 'content' && (method === 'GET' || method === 'HEAD' || method === 'PUT')) ||
|
||||
(kind === 'item' && method === 'DELETE');
|
||||
if (!accepted) {
|
||||
throw new AssetHttpError(
|
||||
405,
|
||||
'METHOD_NOT_ALLOWED',
|
||||
'@openmaic/storage: method not allowed for this asset route',
|
||||
undefined,
|
||||
{ allow },
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
function missingAsset(): AssetHttpError {
|
||||
return new AssetHttpError(
|
||||
404,
|
||||
'ASSET_NOT_FOUND',
|
||||
'@openmaic/storage: no asset is stored under that id',
|
||||
);
|
||||
}
|
||||
|
||||
function classifyStoreError(error: unknown): never {
|
||||
if (error instanceof AssetNotFoundError) throw missingAsset();
|
||||
if (error instanceof AssetQuotaExceededError) {
|
||||
throw new AssetHttpError(
|
||||
507,
|
||||
'ASSET_QUOTA_EXCEEDED',
|
||||
'@openmaic/storage: asset quota exceeded for this principal',
|
||||
);
|
||||
}
|
||||
throw error;
|
||||
}
|
||||
|
||||
function mappedError(error: unknown): {
|
||||
status: number;
|
||||
body: ErrorBody;
|
||||
headers: Record<string, string>;
|
||||
} {
|
||||
if (error instanceof AssetHttpError) {
|
||||
return {
|
||||
status: error.status,
|
||||
body: {
|
||||
error: {
|
||||
code: error.code,
|
||||
message: error.message,
|
||||
...(error.details === undefined ? {} : { details: error.details }),
|
||||
},
|
||||
},
|
||||
headers: error.headers,
|
||||
};
|
||||
}
|
||||
return {
|
||||
status: 500,
|
||||
body: {
|
||||
error: { code: 'INTERNAL_ERROR', message: '@openmaic/storage: internal server error' },
|
||||
},
|
||||
headers: {},
|
||||
};
|
||||
}
|
||||
|
||||
async function route(
|
||||
req: IncomingMessage,
|
||||
res: ServerResponse,
|
||||
store: AssetStore,
|
||||
options: AssetHttpHandlerOptions,
|
||||
config: {
|
||||
renderableTypes: ReadonlySet<string>;
|
||||
maxRequestBytes: number;
|
||||
maxAssetBytes: number;
|
||||
maxMetaBytes: number;
|
||||
maxParts: number;
|
||||
},
|
||||
): Promise<void> {
|
||||
const parts = parsePath(req);
|
||||
const shape = routeShape(parts);
|
||||
const method = req.method ?? 'GET';
|
||||
assertMethod(method, shape.kind, shape.allow);
|
||||
|
||||
const candidate = (await options.authenticate(req)) as unknown;
|
||||
if (candidate === undefined) {
|
||||
throw new AssetHttpError(401, 'UNAUTHENTICATED', '@openmaic/storage: authentication required');
|
||||
}
|
||||
if (typeof candidate !== 'object' || candidate === null) {
|
||||
throw new Error('@openmaic/storage: asset authenticator returned a malformed principal');
|
||||
}
|
||||
if (!('key' in candidate)) {
|
||||
throw new AssetHttpError(
|
||||
403,
|
||||
'FORBIDDEN_ASSETS',
|
||||
'@openmaic/storage: asset authorization required',
|
||||
);
|
||||
}
|
||||
if (typeof (candidate as { key?: unknown }).key !== 'string') {
|
||||
throw new Error('@openmaic/storage: asset authenticator returned a malformed principal');
|
||||
}
|
||||
const principal = candidate as AssetPrincipal;
|
||||
if (!(await (options.authorizeAssets?.(principal, req) ?? true))) {
|
||||
throw new AssetHttpError(
|
||||
403,
|
||||
'FORBIDDEN_ASSETS',
|
||||
'@openmaic/storage: asset authorization required',
|
||||
);
|
||||
}
|
||||
|
||||
if (method === 'GET' || method === 'HEAD' || method === 'DELETE') await assertBodyless(req);
|
||||
|
||||
if (shape.kind === 'collection') {
|
||||
const write = await readWrite(req, true, config);
|
||||
let id: AssetId;
|
||||
try {
|
||||
id = await store.put(principal, write.data, write.meta);
|
||||
} catch (error) {
|
||||
classifyStoreError(error);
|
||||
}
|
||||
if (typeof id !== 'string') {
|
||||
throw new Error('@openmaic/storage: asset store returned a malformed id');
|
||||
}
|
||||
sendJson(
|
||||
req,
|
||||
res,
|
||||
201,
|
||||
{ id },
|
||||
{
|
||||
'x-asset-revision': '1',
|
||||
'access-control-expose-headers': 'X-Asset-Revision',
|
||||
},
|
||||
);
|
||||
return;
|
||||
}
|
||||
|
||||
const id = parts[1]!;
|
||||
if (shape.kind === 'item') {
|
||||
try {
|
||||
await store.remove(principal, id);
|
||||
} catch (error) {
|
||||
classifyStoreError(error);
|
||||
}
|
||||
sendNoContent(res);
|
||||
return;
|
||||
}
|
||||
|
||||
if (method === 'PUT') {
|
||||
const write = await readWrite(req, false, config);
|
||||
let revision: number;
|
||||
try {
|
||||
revision = await store.replace(principal, id as AssetId, write.data, write.meta);
|
||||
} catch (error) {
|
||||
classifyStoreError(error);
|
||||
}
|
||||
if (!Number.isSafeInteger(revision) || revision < 1) {
|
||||
throw new Error('@openmaic/storage: asset store returned a malformed revision');
|
||||
}
|
||||
sendNoContent(res, {
|
||||
'x-asset-revision': String(revision),
|
||||
'access-control-expose-headers': 'X-Asset-Revision',
|
||||
});
|
||||
return;
|
||||
}
|
||||
|
||||
let asset;
|
||||
try {
|
||||
asset =
|
||||
method === 'HEAD' ? await store.identify(principal, id) : await store.resolve(principal, id);
|
||||
} catch (error) {
|
||||
classifyStoreError(error);
|
||||
}
|
||||
if (asset === null) throw missingAsset();
|
||||
if (!Number.isSafeInteger(asset.revision) || asset.revision < 1) {
|
||||
throw new Error('@openmaic/storage: asset store returned a malformed revision');
|
||||
}
|
||||
if ('byteLength' in asset && (!Number.isSafeInteger(asset.byteLength) || asset.byteLength < 0)) {
|
||||
throw new Error('@openmaic/storage: asset store returned a malformed byte length');
|
||||
}
|
||||
const recordedType = typeof asset.mime === 'string' ? asset.mime.toLowerCase() : '';
|
||||
const inline = config.renderableTypes.has(recordedType);
|
||||
const servedType = inline ? recordedType : 'application/octet-stream';
|
||||
const headers: Record<string, string> = {
|
||||
'content-type': servedType,
|
||||
'content-length': String('byteLength' in asset ? asset.byteLength : asset.bytes.byteLength),
|
||||
'x-asset-revision': String(asset.revision),
|
||||
'x-content-type-options': 'nosniff',
|
||||
'cache-control': 'private, no-store',
|
||||
vary: 'Cookie, Authorization',
|
||||
'access-control-expose-headers': 'X-Asset-Revision, X-Error-Code',
|
||||
...(inline ? {} : { 'content-disposition': 'attachment' }),
|
||||
};
|
||||
res.sendDate = false;
|
||||
res.writeHead(200, headers);
|
||||
res.end('bytes' in asset ? asset.bytes : undefined);
|
||||
}
|
||||
|
||||
/** Create a Node HTTP request handler for the complete AssetStore HTTP contract. */
|
||||
export function createAssetHttpHandler(
|
||||
store: AssetStore,
|
||||
options: AssetHttpHandlerOptions,
|
||||
): RequestListener {
|
||||
if (!store) throw new Error('@openmaic/storage: createAssetHttpHandler requires an asset store');
|
||||
if (typeof options?.authenticate !== 'function') {
|
||||
throw new Error('@openmaic/storage: createAssetHttpHandler requires authenticate');
|
||||
}
|
||||
const maxRequestBytes = options.maxRequestBytes ?? DEFAULT_MAX_ASSET_REQUEST_BYTES;
|
||||
const maxAssetBytes = options.maxAssetBytes ?? DEFAULT_MAX_ASSET_BYTES;
|
||||
const maxMetaBytes = options.maxMetaBytes ?? DEFAULT_MAX_ASSET_META_BYTES;
|
||||
const maxParts = options.maxParts ?? DEFAULT_MAX_ASSET_PARTS;
|
||||
for (const [label, value] of [
|
||||
['maxRequestBytes', maxRequestBytes],
|
||||
['maxAssetBytes', maxAssetBytes],
|
||||
['maxMetaBytes', maxMetaBytes],
|
||||
['maxParts', maxParts],
|
||||
] as const) {
|
||||
assertPositiveSafeInteger(value, label);
|
||||
}
|
||||
if (maxParts < 2) {
|
||||
throw new Error('@openmaic/storage: maxParts must allow the two required POST parts');
|
||||
}
|
||||
if (maxRequestBytes <= maxAssetBytes + maxMetaBytes) {
|
||||
throw new Error(
|
||||
'@openmaic/storage: maxRequestBytes must exceed maxAssetBytes + maxMetaBytes for multipart framing',
|
||||
);
|
||||
}
|
||||
|
||||
const configuredTypes = options.renderableTypes ?? DEFAULT_RENDERABLE_TYPES;
|
||||
const renderableTypes = new Set(configuredTypes.map((value) => value.toLowerCase()));
|
||||
const excluded = new Set(EXCLUDED_RENDERABLE_TYPES.map((value) => value.toLowerCase()));
|
||||
if (configuredTypes.some((value) => excluded.has(value.toLowerCase()))) {
|
||||
throw new Error('@openmaic/storage: renderableTypes contains an excluded executable type');
|
||||
}
|
||||
if (configuredTypes.some((value) => value !== value.trim() || value.includes(';'))) {
|
||||
throw new Error('@openmaic/storage: renderableTypes must contain exact media types');
|
||||
}
|
||||
|
||||
const config = {
|
||||
renderableTypes,
|
||||
maxRequestBytes,
|
||||
maxAssetBytes,
|
||||
maxMetaBytes,
|
||||
maxParts,
|
||||
};
|
||||
return (req, res) => {
|
||||
void route(req, res, store, options, config).catch((error: unknown) => {
|
||||
if (res.headersSent) {
|
||||
res.destroy(error instanceof Error ? error : undefined);
|
||||
return;
|
||||
}
|
||||
if (!(error instanceof AssetHttpError) || error.status >= 500) {
|
||||
console.error('@openmaic/storage: Asset HTTP handler internal error');
|
||||
}
|
||||
const mapped = mappedError(error);
|
||||
sendJson(req, res, mapped.status, mapped.body, mapped.headers);
|
||||
});
|
||||
};
|
||||
}
|
||||
@@ -27,9 +27,22 @@ import type {
|
||||
import { RuntimeAppendConflictError } from '../runtime/types.js';
|
||||
import type { Scene, Stage } from '@openmaic/dsl';
|
||||
import type { DocumentStore, SceneLike } from '../document/types.js';
|
||||
import type { AssetPrincipal, AssetStore } from '../asset/types.js';
|
||||
import { createAssetHttpHandler, type AssetHttpHandlerOptions } from './asset.js';
|
||||
import { createDocumentHttpHandler, type DocumentHttpHandlerOptions } from './document.js';
|
||||
import { assertMaxBodyBytes, DEFAULT_MAX_BODY_BYTES, readJsonObject } from './read-json.js';
|
||||
|
||||
export {
|
||||
createAssetHttpHandler,
|
||||
DEFAULT_MAX_ASSET_BYTES,
|
||||
DEFAULT_MAX_ASSET_META_BYTES,
|
||||
DEFAULT_MAX_ASSET_PARTS,
|
||||
DEFAULT_MAX_ASSET_REQUEST_BYTES,
|
||||
type AssetHttpAuthenticate,
|
||||
type AssetHttpAuthorize,
|
||||
type AssetHttpHandlerOptions,
|
||||
} from './asset.js';
|
||||
|
||||
export {
|
||||
createDocumentHttpHandler,
|
||||
type DocumentHttpAuthenticate,
|
||||
@@ -689,31 +702,65 @@ export function createRuntimeHttpHandler(
|
||||
export interface StorageHttpHandlerOptions
|
||||
extends
|
||||
RuntimeHttpHandlerOptions,
|
||||
Pick<DocumentHttpHandlerOptions, 'authorizeDocuments' | 'validateScene' | 'validateStage'> {}
|
||||
Pick<DocumentHttpHandlerOptions, 'authorizeDocuments' | 'validateScene' | 'validateStage'>,
|
||||
Pick<
|
||||
AssetHttpHandlerOptions,
|
||||
| 'authorizeAssets'
|
||||
| 'renderableTypes'
|
||||
| 'maxRequestBytes'
|
||||
| 'maxAssetBytes'
|
||||
| 'maxMetaBytes'
|
||||
| 'maxParts'
|
||||
> {
|
||||
/** When supplied, the composed handler exposes the `/assets` contract. */
|
||||
assetStore?: AssetStore;
|
||||
}
|
||||
|
||||
/**
|
||||
* Compose the runtime and author-document contracts into one request handler.
|
||||
* Existing runtime routing is delegated unchanged; `/documents` is dispatched
|
||||
* to the document handler and shares the same authentication hook.
|
||||
* Compose the runtime, optional author-document, and optional asset contracts
|
||||
* into one request handler. Existing runtime routing is delegated unchanged;
|
||||
* the other handlers share its authentication hook.
|
||||
*/
|
||||
export function createStorageHttpHandler<
|
||||
TScene extends SceneLike = Scene,
|
||||
TStage extends Stage = Stage,
|
||||
>(
|
||||
runtimeStore: RuntimeStore,
|
||||
documentStore: DocumentStore<TScene, TStage>,
|
||||
documentStore: DocumentStore<TScene, TStage> | undefined,
|
||||
options: StorageHttpHandlerOptions,
|
||||
): RequestListener {
|
||||
const runtime = createRuntimeHttpHandler(runtimeStore, options);
|
||||
const documents = createDocumentHttpHandler(documentStore, {
|
||||
authenticate: options.authenticate,
|
||||
...(options.authorizeDocuments === undefined
|
||||
? {}
|
||||
: { authorizeDocuments: options.authorizeDocuments }),
|
||||
...(options.validateScene === undefined ? {} : { validateScene: options.validateScene }),
|
||||
...(options.validateStage === undefined ? {} : { validateStage: options.validateStage }),
|
||||
...(options.maxBodyBytes === undefined ? {} : { maxBodyBytes: options.maxBodyBytes }),
|
||||
});
|
||||
const documents =
|
||||
documentStore === undefined
|
||||
? undefined
|
||||
: createDocumentHttpHandler(documentStore, {
|
||||
authenticate: options.authenticate,
|
||||
...(options.authorizeDocuments === undefined
|
||||
? {}
|
||||
: { authorizeDocuments: options.authorizeDocuments }),
|
||||
...(options.validateScene === undefined ? {} : { validateScene: options.validateScene }),
|
||||
...(options.validateStage === undefined ? {} : { validateStage: options.validateStage }),
|
||||
...(options.maxBodyBytes === undefined ? {} : { maxBodyBytes: options.maxBodyBytes }),
|
||||
});
|
||||
const assets =
|
||||
options.assetStore === undefined
|
||||
? undefined
|
||||
: createAssetHttpHandler(options.assetStore, {
|
||||
authenticate: async (req) =>
|
||||
(await options.authenticate(req)) as AssetPrincipal | undefined,
|
||||
...(options.authorizeAssets === undefined
|
||||
? {}
|
||||
: { authorizeAssets: options.authorizeAssets }),
|
||||
...(options.renderableTypes === undefined
|
||||
? {}
|
||||
: { renderableTypes: options.renderableTypes }),
|
||||
...(options.maxRequestBytes === undefined
|
||||
? {}
|
||||
: { maxRequestBytes: options.maxRequestBytes }),
|
||||
...(options.maxAssetBytes === undefined ? {} : { maxAssetBytes: options.maxAssetBytes }),
|
||||
...(options.maxMetaBytes === undefined ? {} : { maxMetaBytes: options.maxMetaBytes }),
|
||||
...(options.maxParts === undefined ? {} : { maxParts: options.maxParts }),
|
||||
});
|
||||
return (req, res) => {
|
||||
let pathname: string;
|
||||
try {
|
||||
@@ -722,7 +769,18 @@ export function createStorageHttpHandler<
|
||||
runtime(req, res);
|
||||
return;
|
||||
}
|
||||
if (pathname === '/documents' || pathname.startsWith('/documents/')) documents(req, res);
|
||||
else runtime(req, res);
|
||||
if (
|
||||
documents !== undefined &&
|
||||
(pathname === '/documents' || pathname.startsWith('/documents/'))
|
||||
) {
|
||||
documents(req, res);
|
||||
} else if (
|
||||
assets !== undefined &&
|
||||
(pathname === '/assets' || pathname.startsWith('/assets/'))
|
||||
) {
|
||||
assets(req, res);
|
||||
} else {
|
||||
runtime(req, res);
|
||||
}
|
||||
};
|
||||
}
|
||||
|
||||
@@ -14,6 +14,7 @@ import { pathToFileURL } from 'node:url';
|
||||
import { PgRuntimeStore, ensureSchema } from '../runtime/pg.js';
|
||||
import type { Queryable, WithTransaction } from '../runtime/pg.js';
|
||||
import type { RuntimePayloadValidator } from '../runtime/types.js';
|
||||
import type { AssetStore } from '../asset/types.js';
|
||||
import type {
|
||||
DocumentStore,
|
||||
SceneLike,
|
||||
@@ -24,6 +25,7 @@ import type { Scene, Stage } from '@openmaic/dsl';
|
||||
import { createRuntimeHttpHandler, createStorageHttpHandler } from './index.js';
|
||||
import type {
|
||||
DocumentHttpAuthorize,
|
||||
AssetHttpAuthorize,
|
||||
RuntimeHttpAuthenticate,
|
||||
RuntimeHttpAuthorizeAdmin,
|
||||
RuntimeHttpAuthorizeMerge,
|
||||
@@ -46,6 +48,9 @@ export interface ReferenceRuntimeServerOptions<
|
||||
/** When supplied, the same server also exposes the `/documents` contract. */
|
||||
documentStore?: DocumentStore<TScene, TStage>;
|
||||
authorizeDocuments?: DocumentHttpAuthorize;
|
||||
/** When supplied, the same server also exposes the `/assets` contract. */
|
||||
assetStore?: AssetStore;
|
||||
authorizeAssets?: AssetHttpAuthorize;
|
||||
/** Pass the same validators configured on documentStore. */
|
||||
validateScene?: SceneValidator;
|
||||
validateStage?: StageValidator;
|
||||
@@ -104,7 +109,7 @@ export async function createReferenceRuntimeServer<
|
||||
return undefined;
|
||||
}
|
||||
const learnerKey = authorization.slice('Bearer '.length);
|
||||
return learnerKey === '' ? undefined : { learnerKey };
|
||||
return learnerKey === '' ? undefined : { learnerKey, key: learnerKey };
|
||||
}),
|
||||
authorizeMerge:
|
||||
options.authorizeMerge ??
|
||||
@@ -116,12 +121,14 @@ export async function createReferenceRuntimeServer<
|
||||
...(options.authorizeDocuments === undefined
|
||||
? {}
|
||||
: { authorizeDocuments: options.authorizeDocuments }),
|
||||
...(options.assetStore === undefined ? {} : { assetStore: options.assetStore }),
|
||||
...(options.authorizeAssets === undefined ? {} : { authorizeAssets: options.authorizeAssets }),
|
||||
...(options.validateScene === undefined ? {} : { validateScene: options.validateScene }),
|
||||
...(options.validateStage === undefined ? {} : { validateStage: options.validateStage }),
|
||||
...(options.maxBodyBytes === undefined ? {} : { maxBodyBytes: options.maxBodyBytes }),
|
||||
};
|
||||
return createServer(
|
||||
options.documentStore === undefined
|
||||
options.documentStore === undefined && options.assetStore === undefined
|
||||
? createRuntimeHttpHandler(store, handlerOptions)
|
||||
: createStorageHttpHandler(store, options.documentStore, handlerOptions),
|
||||
);
|
||||
|
||||
@@ -0,0 +1,58 @@
|
||||
import { describe, expect, test } from 'vitest';
|
||||
import type { AssetByteStore } from '../src/asset/byte-store.js';
|
||||
import { contentHashOf, type ContentHash } from '../src/asset/blob.js';
|
||||
|
||||
const bytes = (value: string): Uint8Array => new TextEncoder().encode(value);
|
||||
|
||||
async function hashFor(value: string): Promise<ContentHash> {
|
||||
return (await contentHashOf(new Blob([value]))).contentHash;
|
||||
}
|
||||
|
||||
export function runAssetByteStoreContract(
|
||||
name: string,
|
||||
makeStore: () => AssetByteStore,
|
||||
namespace = name,
|
||||
): void {
|
||||
const value = (label: string): string => `${namespace}:${label}`;
|
||||
|
||||
describe(`asset byte store contract: ${name}`, () => {
|
||||
test('write and read round-trip bytes', async () => {
|
||||
const store = makeStore();
|
||||
const payload = value('round trip');
|
||||
const hash = await hashFor(payload);
|
||||
await store.write(hash, bytes(payload));
|
||||
expect(await store.read(hash)).toEqual(bytes(payload));
|
||||
});
|
||||
|
||||
test('writing the same bytes twice is idempotent', async () => {
|
||||
const store = makeStore();
|
||||
const payload = value('idempotent write');
|
||||
const hash = await hashFor(payload);
|
||||
await expect(store.write(hash, bytes(payload))).resolves.toBeUndefined();
|
||||
await expect(store.write(hash, bytes(payload))).resolves.toBeUndefined();
|
||||
expect(await store.read(hash)).toEqual(bytes(payload));
|
||||
});
|
||||
|
||||
test('a missing hash reads as null', async () => {
|
||||
const store = makeStore();
|
||||
expect(await store.read(await hashFor(value('missing')))).toBeNull();
|
||||
});
|
||||
|
||||
test('zero-byte values are legal', async () => {
|
||||
const store = makeStore();
|
||||
const hash = await hashFor('');
|
||||
await store.write(hash, new Uint8Array());
|
||||
expect(await store.read(hash)).toEqual(new Uint8Array());
|
||||
});
|
||||
|
||||
test('delete removes bytes and is idempotent', async () => {
|
||||
const store = makeStore();
|
||||
const payload = value('delete');
|
||||
const hash = await hashFor(payload);
|
||||
await store.write(hash, bytes(payload));
|
||||
await expect(store.delete(hash)).resolves.toBeUndefined();
|
||||
expect(await store.read(hash)).toBeNull();
|
||||
await expect(store.delete(hash)).resolves.toBeUndefined();
|
||||
});
|
||||
});
|
||||
}
|
||||
@@ -0,0 +1,131 @@
|
||||
// Test-only HTTP adapter for the asset contract. Each namespace owns a real
|
||||
// PgAssetStore on PGlite, so the shared provider suite crosses multipart,
|
||||
// handler, registry, transaction, and byte-storage boundaries end to end.
|
||||
import { createServer, type IncomingMessage, type Server } from 'node:http';
|
||||
import { PGlite } from '@electric-sql/pglite';
|
||||
import type { AssetByteStore } from '../src/asset/byte-store.js';
|
||||
import { PgAssetByteStore } from '../src/asset/pg-bytes.js';
|
||||
import {
|
||||
PgAssetStore,
|
||||
ensureAssetSchema,
|
||||
type Queryable,
|
||||
type WithTransaction,
|
||||
} from '../src/asset/pg.js';
|
||||
import type { AssetPrincipal, AssetStore } from '../src/asset/types.js';
|
||||
import {
|
||||
createAssetHttpHandler,
|
||||
type AssetHttpAuthorize,
|
||||
type AssetHttpHandlerOptions,
|
||||
} from '../src/server/asset.js';
|
||||
|
||||
export interface AssetConformanceServer {
|
||||
baseUrl: string;
|
||||
fetch: typeof globalThis.fetch;
|
||||
close(): Promise<void>;
|
||||
}
|
||||
|
||||
export interface AssetConformanceServerOptions extends Omit<
|
||||
AssetHttpHandlerOptions,
|
||||
'authenticate' | 'authorizeAssets'
|
||||
> {
|
||||
authenticate?: (req: IncomingMessage) => Promise<AssetPrincipal | undefined>;
|
||||
authorizeAssets?: AssetHttpAuthorize;
|
||||
byteStore?: (db: PGlite) => AssetByteStore;
|
||||
store?: (db: PGlite) => AssetStore;
|
||||
}
|
||||
|
||||
interface NamespaceState {
|
||||
db: PGlite;
|
||||
ready: Promise<void>;
|
||||
handler: ReturnType<typeof createAssetHttpHandler>;
|
||||
}
|
||||
|
||||
function transactions(db: PGlite): WithTransaction {
|
||||
return (body) => db.transaction((tx: Queryable) => body(tx));
|
||||
}
|
||||
|
||||
function header(req: IncomingMessage, name: string): string | undefined {
|
||||
const value = req.headers[name];
|
||||
return typeof value === 'string' ? value : undefined;
|
||||
}
|
||||
|
||||
/** Start the asset conformance server on an ephemeral loopback port. */
|
||||
export async function startAssetConformanceServer(
|
||||
options: AssetConformanceServerOptions = {},
|
||||
): Promise<AssetConformanceServer> {
|
||||
const states = new Map<string, NamespaceState>();
|
||||
const authenticate =
|
||||
options.authenticate ??
|
||||
(async (req: IncomingMessage) => ({ key: header(req, 'x-asset-principal') ?? 'default' }));
|
||||
|
||||
const stateFor = (req: IncomingMessage): NamespaceState => {
|
||||
const namespace = header(req, 'x-asset-store-id') ?? 'default';
|
||||
let state = states.get(namespace);
|
||||
if (state !== undefined) return state;
|
||||
const db = new PGlite();
|
||||
const store =
|
||||
options.store?.(db) ??
|
||||
new PgAssetStore(db, {
|
||||
withTransaction: transactions(db),
|
||||
byteStore: options.byteStore?.(db) ?? new PgAssetByteStore(db),
|
||||
});
|
||||
state = {
|
||||
db,
|
||||
ready: db.waitReady.then(() => ensureAssetSchema(db)),
|
||||
handler: createAssetHttpHandler(store, {
|
||||
authenticate,
|
||||
...(options.authorizeAssets === undefined
|
||||
? {}
|
||||
: { authorizeAssets: options.authorizeAssets }),
|
||||
...(options.renderableTypes === undefined
|
||||
? {}
|
||||
: { renderableTypes: options.renderableTypes }),
|
||||
...(options.maxRequestBytes === undefined
|
||||
? {}
|
||||
: { maxRequestBytes: options.maxRequestBytes }),
|
||||
...(options.maxAssetBytes === undefined ? {} : { maxAssetBytes: options.maxAssetBytes }),
|
||||
...(options.maxMetaBytes === undefined ? {} : { maxMetaBytes: options.maxMetaBytes }),
|
||||
...(options.maxParts === undefined ? {} : { maxParts: options.maxParts }),
|
||||
}),
|
||||
};
|
||||
states.set(namespace, state);
|
||||
return state;
|
||||
};
|
||||
|
||||
const server: Server = createServer((req, res) => {
|
||||
const state = stateFor(req);
|
||||
void state.ready.then(
|
||||
() => state.handler(req, res),
|
||||
() => {
|
||||
res.writeHead(500, { 'content-type': 'application/json' });
|
||||
res.end(
|
||||
JSON.stringify({
|
||||
error: {
|
||||
code: 'INTERNAL_ERROR',
|
||||
message: '@openmaic/storage: internal server error',
|
||||
},
|
||||
}),
|
||||
);
|
||||
},
|
||||
);
|
||||
});
|
||||
|
||||
await new Promise<void>((resolve, reject) => {
|
||||
server.once('error', reject);
|
||||
server.listen(0, '127.0.0.1', resolve);
|
||||
});
|
||||
const address = server.address();
|
||||
if (address === null || typeof address === 'string') {
|
||||
throw new Error('Asset conformance server did not bind a TCP port');
|
||||
}
|
||||
return {
|
||||
baseUrl: `http://127.0.0.1:${address.port}`,
|
||||
fetch: globalThis.fetch.bind(globalThis),
|
||||
close: async () => {
|
||||
await new Promise<void>((resolve, reject) => {
|
||||
server.close((error) => (error ? reject(error) : resolve()));
|
||||
});
|
||||
await Promise.all([...states.values()].map((state) => state.db.close()));
|
||||
},
|
||||
};
|
||||
}
|
||||
@@ -103,7 +103,7 @@ function base32(digest: Uint8Array, alphabet: string): string {
|
||||
return encoded;
|
||||
}
|
||||
|
||||
async function commonDigestEncodings(data: Blob): Promise<string[]> {
|
||||
export async function commonDigestEncodings(data: Blob): Promise<string[]> {
|
||||
const digest = new Uint8Array(await crypto.subtle.digest('SHA-256', await data.arrayBuffer()));
|
||||
const binary = String.fromCharCode(...digest);
|
||||
const base64 = btoa(binary);
|
||||
@@ -118,7 +118,7 @@ async function commonDigestEncodings(data: Blob): Promise<string[]> {
|
||||
];
|
||||
}
|
||||
|
||||
async function expectNoDigestSubstring(id: AssetRef, data: Blob): Promise<void> {
|
||||
export async function expectNoDigestSubstring(id: AssetRef, data: Blob): Promise<void> {
|
||||
const minimumLeakLength = 12;
|
||||
for (const encoding of await commonDigestEncodings(data)) {
|
||||
for (const idForm of new Set([id, id.toLowerCase()])) {
|
||||
@@ -136,7 +136,7 @@ async function expectNoDigestSubstring(id: AssetRef, data: Blob): Promise<void>
|
||||
* an ordinary miss — the id domain is opaque and unvalidated, so there is no
|
||||
* such thing as a malformed id, only an id nothing is stored under.
|
||||
*/
|
||||
const FOREIGN_IDS: ReadonlyArray<readonly [label: string, id: string]> = [
|
||||
export const FOREIGN_IDS: ReadonlyArray<readonly [label: string, id: string]> = [
|
||||
['empty string', ''],
|
||||
['whitespace', ' '],
|
||||
['bare prefix', 'ast_'],
|
||||
|
||||
@@ -993,6 +993,7 @@ test('the package entry does not expose internal asset-layer symbols', () => {
|
||||
'ContentHash',
|
||||
'BlobStore',
|
||||
'BrowserAssetProvider',
|
||||
'S3AssetByteStore',
|
||||
]) {
|
||||
expect(storageExports).not.toHaveProperty(name);
|
||||
}
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,49 @@
|
||||
import { afterEach, beforeEach, describe, expect, test } from 'vitest';
|
||||
import { PGlite } from '@electric-sql/pglite';
|
||||
import { contentHashOf } from '../src/asset/blob.js';
|
||||
import { PgAssetByteStore } from '../src/asset/pg-bytes.js';
|
||||
import { ensureAssetSchema } from '../src/asset/pg.js';
|
||||
import { expectNoDigestSubstring } from './asset-contract.js';
|
||||
import { runAssetByteStoreContract } from './asset-byte-store-contract.js';
|
||||
|
||||
describe('PgAssetByteStore with PGlite', () => {
|
||||
let db: PGlite;
|
||||
let store: PgAssetByteStore;
|
||||
|
||||
beforeEach(async () => {
|
||||
db = new PGlite();
|
||||
await db.waitReady;
|
||||
await ensureAssetSchema(db);
|
||||
store = new PgAssetByteStore(db);
|
||||
});
|
||||
|
||||
afterEach(async () => {
|
||||
await db.close();
|
||||
});
|
||||
|
||||
runAssetByteStoreContract('PostgreSQL bytes (PGlite)', () => store);
|
||||
|
||||
test('sanitizes query failures so the digest cannot escape', async () => {
|
||||
const data = new Blob(['postgres byte failure']);
|
||||
const { contentHash, bytes } = await contentHashOf(data);
|
||||
const failing = new PgAssetByteStore({
|
||||
async query() {
|
||||
throw new Error(contentHash);
|
||||
},
|
||||
});
|
||||
for (const operation of [
|
||||
() => failing.write(contentHash, new Uint8Array(bytes)),
|
||||
() => failing.read(contentHash),
|
||||
() => failing.delete(contentHash),
|
||||
]) {
|
||||
let thrown: unknown;
|
||||
try {
|
||||
await operation();
|
||||
} catch (error) {
|
||||
thrown = error;
|
||||
}
|
||||
expect(thrown).toBeInstanceOf(Error);
|
||||
await expectNoDigestSubstring(String(thrown), data);
|
||||
}
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,283 @@
|
||||
import { afterAll, beforeAll, beforeEach, describe, expect, test } from 'vitest';
|
||||
import { Pool } from 'pg';
|
||||
import { contentHashOf, type ContentHash } from '../src/asset/blob.js';
|
||||
import type { AssetByteStore } from '../src/asset/byte-store.js';
|
||||
import { AssetCollector } from '../src/asset/collector.js';
|
||||
import { PgAssetByteStore } from '../src/asset/pg-bytes.js';
|
||||
import {
|
||||
AssetQuotaExceededError,
|
||||
PgAssetStore,
|
||||
ensureAssetSchema,
|
||||
type QueryResult,
|
||||
type Queryable,
|
||||
type WithTransaction,
|
||||
} from '../src/asset/pg.js';
|
||||
|
||||
const contractUrl = process.env.PG_CONTRACT_URL;
|
||||
|
||||
if (process.env.STORAGE_PG_CONTRACT_REQUIRED === '1' && !contractUrl) {
|
||||
throw new Error(
|
||||
'@openmaic/storage: STORAGE_PG_CONTRACT_REQUIRED=1 requires PG_CONTRACT_URL; refusing to skip the PostgreSQL asset suite',
|
||||
);
|
||||
}
|
||||
|
||||
function transactionFor(pool: Pool): WithTransaction {
|
||||
return async (body) => {
|
||||
const client = await pool.connect();
|
||||
try {
|
||||
await client.query('BEGIN');
|
||||
const result = await body(client as Queryable);
|
||||
await client.query('COMMIT');
|
||||
return result;
|
||||
} catch (error) {
|
||||
try {
|
||||
await client.query('ROLLBACK');
|
||||
} catch {
|
||||
// Preserve the transaction body's original error.
|
||||
}
|
||||
throw error;
|
||||
} finally {
|
||||
client.release();
|
||||
}
|
||||
};
|
||||
}
|
||||
|
||||
async function waitForLockWaiter(pool: { query: Queryable['query'] }): Promise<void> {
|
||||
for (let attempt = 0; attempt < 400; attempt += 1) {
|
||||
const waiting = await pool.query(
|
||||
`SELECT 1 FROM pg_stat_activity
|
||||
WHERE wait_event_type = 'Lock' AND datname = current_database()`,
|
||||
);
|
||||
if (waiting.rows.length > 0) return;
|
||||
await new Promise((resolve) => setTimeout(resolve, 25));
|
||||
}
|
||||
throw new Error('no backend blocked on a lock: the operation never contended for the blob row');
|
||||
}
|
||||
|
||||
class BlockingReadByteStore implements AssetByteStore {
|
||||
private readonly values = new Map<ContentHash, Uint8Array>();
|
||||
private signalReadStarted!: () => void;
|
||||
private allowReadToFinish!: () => void;
|
||||
readonly readStarted = new Promise<void>((resolve) => {
|
||||
this.signalReadStarted = resolve;
|
||||
});
|
||||
private readonly mayFinishRead = new Promise<void>((resolve) => {
|
||||
this.allowReadToFinish = resolve;
|
||||
});
|
||||
|
||||
async write(hash: ContentHash, value: Uint8Array): Promise<void> {
|
||||
this.values.set(hash, new Uint8Array(value));
|
||||
}
|
||||
|
||||
async read(hash: ContentHash): Promise<Uint8Array | null> {
|
||||
this.signalReadStarted();
|
||||
await this.mayFinishRead;
|
||||
const value = this.values.get(hash);
|
||||
return value === undefined ? null : new Uint8Array(value);
|
||||
}
|
||||
|
||||
async delete(hash: ContentHash): Promise<void> {
|
||||
this.values.delete(hash);
|
||||
}
|
||||
|
||||
finishRead(): void {
|
||||
this.allowReadToFinish();
|
||||
}
|
||||
}
|
||||
|
||||
describe.skipIf(!contractUrl)('PgAssetStore with PostgreSQL 16', () => {
|
||||
let pool: Pool;
|
||||
let bytes: PgAssetByteStore;
|
||||
let store: PgAssetStore;
|
||||
const principal = { key: 'postgres-principal' };
|
||||
|
||||
beforeAll(async () => {
|
||||
pool = new Pool({ connectionString: contractUrl, max: 12 });
|
||||
await ensureAssetSchema(pool as Queryable);
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
await pool.query('TRUNCATE asset_entries, asset_blobs');
|
||||
bytes = new PgAssetByteStore(pool as Queryable);
|
||||
store = new PgAssetStore(pool as Queryable, {
|
||||
byteStore: bytes,
|
||||
withTransaction: transactionFor(pool),
|
||||
});
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await pool.end();
|
||||
});
|
||||
|
||||
test('provisions the non-cascading foreign key and stores BYTEA bytes', async () => {
|
||||
const id = await store.put(principal, new Blob(['postgres bytes']));
|
||||
const foreignKey = await pool.query<{ delete_rule: string }>(
|
||||
`SELECT delete_rule
|
||||
FROM information_schema.referential_constraints
|
||||
WHERE constraint_schema = current_schema()
|
||||
AND constraint_name = 'asset_entries_content_hash_fkey'`,
|
||||
);
|
||||
expect(foreignKey.rows).toEqual([{ delete_rule: 'NO ACTION' }]);
|
||||
expect((await store.resolve(principal, id))?.bytes).toEqual(
|
||||
new TextEncoder().encode('postgres bytes'),
|
||||
);
|
||||
});
|
||||
|
||||
test('an adopting put survives a collector that already holds the blob row lock', async () => {
|
||||
const data = new Blob(['locked adoption']);
|
||||
const original = await store.put(principal, data);
|
||||
await store.remove(principal, original);
|
||||
await pool.query(`UPDATE asset_blobs SET unreferenced_at = '2000-01-01T00:00:00.000Z'`);
|
||||
|
||||
let locked!: () => void;
|
||||
const rowLocked = new Promise<void>((resolve) => {
|
||||
locked = resolve;
|
||||
});
|
||||
let release!: () => void;
|
||||
const mayDelete = new Promise<void>((resolve) => {
|
||||
release = resolve;
|
||||
});
|
||||
const collector = new AssetCollector(pool as Queryable, bytes, {
|
||||
graceMs: 0,
|
||||
now: () => new Date('2026-01-01T00:00:00.000Z'),
|
||||
withTransaction: async (body) => {
|
||||
const client = await pool.connect();
|
||||
try {
|
||||
await client.query('BEGIN');
|
||||
const result = await body({
|
||||
async query<TRow extends Record<string, unknown> = Record<string, unknown>>(
|
||||
text: string,
|
||||
params?: unknown[],
|
||||
): Promise<QueryResult<TRow>> {
|
||||
const result = await (client as Queryable).query<TRow>(text, params);
|
||||
if (text.includes('FOR UPDATE')) {
|
||||
locked();
|
||||
await mayDelete;
|
||||
}
|
||||
return result;
|
||||
},
|
||||
});
|
||||
await client.query('COMMIT');
|
||||
return result;
|
||||
} catch (error) {
|
||||
await client.query('ROLLBACK');
|
||||
throw error;
|
||||
} finally {
|
||||
client.release();
|
||||
}
|
||||
},
|
||||
});
|
||||
|
||||
const collection = collector.collect();
|
||||
await rowLocked;
|
||||
|
||||
// The adopting put blocks on the collector's row lock before it can write
|
||||
// any bytes -- claim first, then write, is the ordering under test. Observe
|
||||
// a backend actually waiting on a lock rather than sleeping or signalling
|
||||
// off an implementation detail.
|
||||
const adopter = new PgAssetStore(pool as Queryable, {
|
||||
byteStore: bytes,
|
||||
withTransaction: transactionFor(pool),
|
||||
});
|
||||
const adoption = adopter.put(principal, data);
|
||||
await waitForLockWaiter(pool);
|
||||
release();
|
||||
|
||||
expect(await collection).toBe(1);
|
||||
const adoptedId = await adoption;
|
||||
expect((await adopter.resolve(principal, adoptedId))?.bytes).toEqual(
|
||||
new TextEncoder().encode('locked adoption'),
|
||||
);
|
||||
});
|
||||
|
||||
test('a resolving read pins the blob row until its byte read completes', async () => {
|
||||
const layer = new BlockingReadByteStore();
|
||||
const registry = new PgAssetStore(pool as Queryable, {
|
||||
byteStore: layer,
|
||||
withTransaction: transactionFor(pool),
|
||||
});
|
||||
const data = new Blob(['pinned read']);
|
||||
const { contentHash } = await contentHashOf(data);
|
||||
const id = await registry.put(principal, data);
|
||||
// Make the row a collector candidate while it is still referenced. The
|
||||
// collector's transaction re-checks references, so deleting the entry
|
||||
// after the read starts isolates the lock interleaving under test.
|
||||
await pool.query(`UPDATE asset_blobs SET unreferenced_at = '2000-01-01T00:00:00.000Z'`);
|
||||
|
||||
const resolving = registry.resolve(principal, id);
|
||||
await layer.readStarted;
|
||||
await pool.query('DELETE FROM asset_entries WHERE id = $1', [id]);
|
||||
|
||||
const collector = new AssetCollector(pool as Queryable, layer, {
|
||||
graceMs: 0,
|
||||
now: () => new Date('2026-01-01T00:00:00.000Z'),
|
||||
withTransaction: transactionFor(pool),
|
||||
});
|
||||
const collection = collector.collect();
|
||||
await waitForLockWaiter(pool);
|
||||
|
||||
layer.finishRead();
|
||||
expect((await resolving)?.bytes).toEqual(new TextEncoder().encode('pinned read'));
|
||||
expect(await collection).toBe(1);
|
||||
expect(await layer.read(contentHash)).toBeNull();
|
||||
});
|
||||
|
||||
test('concurrent writes cannot exceed a principal logical quota', async () => {
|
||||
// A quota read on the pool is already stale when it is acted on: two
|
||||
// concurrent writes both observe the old total and both pass. Enforcement
|
||||
// has to happen inside the write transaction, behind a per-principal lock,
|
||||
// which only a real connection pool can exercise -- PGlite is
|
||||
// single-connection and cannot contend.
|
||||
const quoted = new PgAssetStore(pool as Queryable, {
|
||||
byteStore: bytes,
|
||||
withTransaction: transactionFor(pool),
|
||||
quotaBytes: 10,
|
||||
});
|
||||
|
||||
const results = await Promise.allSettled(
|
||||
Array.from({ length: 4 }, (_, index) =>
|
||||
quoted.put(principal, new Blob([`${index}`.repeat(6)])),
|
||||
),
|
||||
);
|
||||
|
||||
const accepted = results.filter((result) => result.status === 'fulfilled');
|
||||
const usage = await pool.query<{ total: string }>(
|
||||
`SELECT COALESCE(SUM(blobs.byte_size), 0)::text AS total
|
||||
FROM asset_entries AS entries
|
||||
JOIN asset_blobs AS blobs ON blobs.content_hash = entries.content_hash
|
||||
WHERE entries.principal = $1`,
|
||||
[principal.key],
|
||||
);
|
||||
|
||||
expect(accepted).toHaveLength(1);
|
||||
expect(Number(usage.rows[0]!.total)).toBeLessThanOrEqual(10);
|
||||
for (const result of results) {
|
||||
if (result.status === 'rejected') {
|
||||
expect(result.reason).toBeInstanceOf(AssetQuotaExceededError);
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
test('a failed registry transaction leaves no PostgreSQL bytes behind', async () => {
|
||||
// This byte layer writes through the registry's own transaction, so a
|
||||
// rollback takes the bytes with it and there is no orphan to collect. An
|
||||
// object store cannot join that transaction and does strand one; that case
|
||||
// is deployment housekeeping, not reference counting.
|
||||
const data = new Blob(['postgres orphan']);
|
||||
const { contentHash } = await contentHashOf(data);
|
||||
const failing = new PgAssetStore(pool as Queryable, {
|
||||
byteStore: bytes,
|
||||
withTransaction: (body) =>
|
||||
transactionFor(pool)(async (queryable) => {
|
||||
await body(queryable);
|
||||
throw new Error('injected failure after the body');
|
||||
}) as Promise<never>,
|
||||
});
|
||||
|
||||
await expect(failing.put(principal, data)).rejects.toThrow(/registry put failed/);
|
||||
|
||||
expect((await pool.query('SELECT * FROM asset_entries')).rows).toEqual([]);
|
||||
expect((await pool.query('SELECT * FROM asset_blobs')).rows).toEqual([]);
|
||||
expect(await bytes.read(contentHash)).toBeNull();
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,724 @@
|
||||
import { afterEach, beforeEach, describe, expect, test } from 'vitest';
|
||||
import { PGlite } from '@electric-sql/pglite';
|
||||
import type { AssetMeta, AssetRef, BinaryBlob, StorageProvider } from '@openmaic/dsl';
|
||||
import { contentHashOf, ObjectUrlCache, type ContentHash } from '../src/asset/blob.js';
|
||||
import type { AssetByteStore } from '../src/asset/byte-store.js';
|
||||
import { AssetCollector } from '../src/asset/collector.js';
|
||||
import { __setAssetIdFactoryForTesting, type AssetId } from '../src/asset/id.js';
|
||||
import { PgAssetByteStore } from '../src/asset/pg-bytes.js';
|
||||
import {
|
||||
ASSET_PG_SCHEMA,
|
||||
PgAssetStore,
|
||||
ensureAssetSchema,
|
||||
type PgAssetStoreOptions,
|
||||
type QueryResult,
|
||||
type Queryable,
|
||||
type WithTransaction,
|
||||
} from '../src/asset/pg.js';
|
||||
import { AssetNotFoundError, AssetQuotaExceededError } from '../src/asset/types.js';
|
||||
import {
|
||||
commonDigestEncodings,
|
||||
expectNoDigestSubstring,
|
||||
runAssetStoreContract,
|
||||
} from './asset-contract.js';
|
||||
import { blobForObjectUrl } from './setup.js';
|
||||
|
||||
const PRINCIPAL = { key: 'principal-a' } as const;
|
||||
const OTHER_PRINCIPAL = { key: 'principal-b' } as const;
|
||||
const bytes = (value: string): Uint8Array => new TextEncoder().encode(value);
|
||||
const blob = (value: string, type = 'text/plain'): Blob => new Blob([value], { type });
|
||||
|
||||
function transactions(db: PGlite): WithTransaction {
|
||||
return (body) => db.transaction((tx: Queryable) => body(tx));
|
||||
}
|
||||
|
||||
function options(
|
||||
db: PGlite,
|
||||
byteStore: AssetByteStore,
|
||||
extra: Partial<PgAssetStoreOptions> = {},
|
||||
): PgAssetStoreOptions {
|
||||
return { withTransaction: transactions(db), byteStore, ...extra };
|
||||
}
|
||||
|
||||
interface UrlIdentity {
|
||||
revision: number;
|
||||
mime: string;
|
||||
}
|
||||
|
||||
class LazyPgProvider implements StorageProvider {
|
||||
readonly db = new PGlite();
|
||||
readonly byteStore = new PgAssetByteStore(this.db);
|
||||
readonly registry = new PgAssetStore(this.db, options(this.db, this.byteStore));
|
||||
readonly ready = this.db.waitReady.then(() => ensureAssetSchema(this.db));
|
||||
private readonly urls = new ObjectUrlCache<UrlIdentity>(
|
||||
(left, right) => left.revision === right.revision && left.mime === right.mime,
|
||||
);
|
||||
|
||||
async put(data: BinaryBlob, meta?: AssetMeta): Promise<AssetId> {
|
||||
await this.ready;
|
||||
return this.registry.put(PRINCIPAL, data, meta);
|
||||
}
|
||||
|
||||
async resolve(ref: AssetRef): Promise<string | null> {
|
||||
await this.ready;
|
||||
const asset = await this.registry.resolve(PRINCIPAL, ref);
|
||||
if (!asset) {
|
||||
await this.urls.invalidate(ref);
|
||||
return null;
|
||||
}
|
||||
const identity = { revision: asset.revision, mime: asset.mime };
|
||||
return this.urls.resolve(ref, identity, async () => ({
|
||||
identity,
|
||||
url: URL.createObjectURL(
|
||||
new Blob(
|
||||
[
|
||||
asset.bytes.buffer.slice(
|
||||
asset.bytes.byteOffset,
|
||||
asset.bytes.byteOffset + asset.bytes.byteLength,
|
||||
) as ArrayBuffer,
|
||||
],
|
||||
{ type: asset.mime },
|
||||
),
|
||||
),
|
||||
}));
|
||||
}
|
||||
|
||||
async remove(ref: AssetRef): Promise<void> {
|
||||
await this.ready;
|
||||
await this.registry.remove(PRINCIPAL, ref);
|
||||
await this.urls.invalidate(ref);
|
||||
}
|
||||
|
||||
async replace(ref: AssetId, data: Blob, meta?: AssetMeta): Promise<void> {
|
||||
await this.ready;
|
||||
await this.registry.replace(PRINCIPAL, ref, data, meta);
|
||||
await this.urls.invalidate(ref);
|
||||
}
|
||||
|
||||
async close(): Promise<void> {
|
||||
await this.urls.close();
|
||||
await this.db.close();
|
||||
}
|
||||
}
|
||||
|
||||
describe('PgAssetStore shared contract with PGlite', () => {
|
||||
const providers: LazyPgProvider[] = [];
|
||||
|
||||
afterEach(async () => {
|
||||
__setAssetIdFactoryForTesting(null);
|
||||
await Promise.all(providers.splice(0).map((provider) => provider.close()));
|
||||
});
|
||||
|
||||
runAssetStoreContract(
|
||||
'PgAssetStore (PGlite)',
|
||||
{
|
||||
makeStore: () => {
|
||||
const provider = new LazyPgProvider();
|
||||
providers.push(provider);
|
||||
return provider;
|
||||
},
|
||||
withAllocator: async (allocator, run) => {
|
||||
__setAssetIdFactoryForTesting(allocator);
|
||||
try {
|
||||
return await run();
|
||||
} finally {
|
||||
__setAssetIdFactoryForTesting(null);
|
||||
}
|
||||
},
|
||||
},
|
||||
async (url) => {
|
||||
const stored = blobForObjectUrl(url);
|
||||
if (!stored) throw new Error('object URL is not registered');
|
||||
return new Uint8Array(await stored.arrayBuffer());
|
||||
},
|
||||
);
|
||||
});
|
||||
|
||||
function normalizeSql(sql: string): string {
|
||||
return sql.replace(/\s+/g, ' ').trim();
|
||||
}
|
||||
|
||||
function recordingQueryable(queryable: Queryable, statements: string[]): Queryable {
|
||||
return {
|
||||
async query<TRow extends Record<string, unknown> = Record<string, unknown>>(
|
||||
text: string,
|
||||
params?: unknown[],
|
||||
): Promise<QueryResult<TRow>> {
|
||||
statements.push(normalizeSql(text));
|
||||
return queryable.query<TRow>(text, params);
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
function recordingTransactions(db: PGlite, statements: string[]): WithTransaction {
|
||||
return (body) => db.transaction((tx: Queryable) => body(recordingQueryable(tx, statements)));
|
||||
}
|
||||
|
||||
class MemoryByteStore implements AssetByteStore {
|
||||
readonly values = new Map<ContentHash, Uint8Array>();
|
||||
onWrite?: () => void;
|
||||
|
||||
async write(hash: ContentHash, value: Uint8Array): Promise<void> {
|
||||
this.values.set(hash, new Uint8Array(value));
|
||||
this.onWrite?.();
|
||||
}
|
||||
|
||||
async read(hash: ContentHash): Promise<Uint8Array | null> {
|
||||
const value = this.values.get(hash);
|
||||
return value ? new Uint8Array(value) : null;
|
||||
}
|
||||
|
||||
async delete(hash: ContentHash): Promise<void> {
|
||||
this.values.delete(hash);
|
||||
}
|
||||
}
|
||||
|
||||
describe('PgAssetStore registry behavior with PGlite', () => {
|
||||
let db: PGlite;
|
||||
let byteStore: PgAssetByteStore;
|
||||
let store: PgAssetStore;
|
||||
|
||||
beforeEach(async () => {
|
||||
db = new PGlite();
|
||||
await db.waitReady;
|
||||
await ensureAssetSchema(db);
|
||||
byteStore = new PgAssetByteStore(db);
|
||||
store = new PgAssetStore(db, options(db, byteStore));
|
||||
});
|
||||
|
||||
afterEach(async () => {
|
||||
__setAssetIdFactoryForTesting(null);
|
||||
await db.close();
|
||||
});
|
||||
|
||||
test('schema is idempotent and has one PGlite-compatible statement per entry', async () => {
|
||||
const statements: string[] = [];
|
||||
await ensureAssetSchema(recordingQueryable(db, statements));
|
||||
await ensureAssetSchema(recordingQueryable(db, statements));
|
||||
expect(statements).toEqual([...ASSET_PG_SCHEMA, ...ASSET_PG_SCHEMA].map(normalizeSql));
|
||||
expect(ASSET_PG_SCHEMA).toHaveLength(5);
|
||||
expect(ASSET_PG_SCHEMA.every((statement) => !statement.includes(';'))).toBe(true);
|
||||
});
|
||||
|
||||
test('zero-byte assets get distinct ids backed by one blob row', async () => {
|
||||
const first = await store.put(PRINCIPAL, blob(''));
|
||||
const second = await store.put(PRINCIPAL, blob(''));
|
||||
expect(first).not.toBe(second);
|
||||
expect((await db.query('SELECT * FROM asset_entries')).rows).toHaveLength(2);
|
||||
expect((await db.query('SELECT * FROM asset_blobs')).rows).toHaveLength(1);
|
||||
expect((await store.resolve(PRINCIPAL, first))?.bytes).toEqual(new Uint8Array());
|
||||
});
|
||||
|
||||
test('ownership is checked on resolve, replace, and remove', async () => {
|
||||
const id = await store.put(PRINCIPAL, blob('private'));
|
||||
expect(await store.identify(PRINCIPAL, id)).toEqual({
|
||||
mime: 'text/plain',
|
||||
revision: 1,
|
||||
byteLength: 7,
|
||||
});
|
||||
expect(await store.identify(OTHER_PRINCIPAL, id)).toBeNull();
|
||||
expect(await store.resolve(OTHER_PRINCIPAL, id)).toBeNull();
|
||||
await expect(store.replace(OTHER_PRINCIPAL, id, blob('foreign'))).rejects.toBeInstanceOf(
|
||||
AssetNotFoundError,
|
||||
);
|
||||
await store.remove(OTHER_PRINCIPAL, id);
|
||||
expect((await store.resolve(PRINCIPAL, id))?.bytes).toEqual(bytes('private'));
|
||||
});
|
||||
|
||||
test('replace preserves or replaces metadata and MIME according to omission', async () => {
|
||||
const id = await store.put(PRINCIPAL, blob('original', 'image/png'), {
|
||||
contentType: '',
|
||||
provenance: 'first',
|
||||
});
|
||||
expect((await store.resolve(PRINCIPAL, id))?.mime).toBe('');
|
||||
|
||||
await store.replace(PRINCIPAL, id, blob('untyped', ''));
|
||||
expect(await store.resolve(PRINCIPAL, id)).toMatchObject({ mime: '', revision: 2 });
|
||||
let row = await db.query<{ meta: unknown }>('SELECT meta FROM asset_entries WHERE id = $1', [
|
||||
id,
|
||||
]);
|
||||
expect(row.rows[0]?.meta).toEqual({ contentType: '', provenance: 'first' });
|
||||
|
||||
await store.replace(PRINCIPAL, id, blob('typed', 'audio/mpeg'));
|
||||
expect(await store.resolve(PRINCIPAL, id)).toMatchObject({ mime: 'audio/mpeg', revision: 3 });
|
||||
|
||||
await store.replace(PRINCIPAL, id, blob('supplied', 'video/mp4'), {
|
||||
contentType: '',
|
||||
provenance: 'replacement',
|
||||
});
|
||||
expect(await store.resolve(PRINCIPAL, id)).toMatchObject({ mime: '', revision: 4 });
|
||||
row = await db.query<{ meta: unknown }>('SELECT meta FROM asset_entries WHERE id = $1', [id]);
|
||||
expect(row.rows[0]?.meta).toEqual({ contentType: '', provenance: 'replacement' });
|
||||
});
|
||||
|
||||
test('rejects JSON-lossy metadata on put and replace without changing entries', async () => {
|
||||
const id = await store.put(PRINCIPAL, blob('original', 'image/png'), {
|
||||
contentType: 'image/png',
|
||||
provenance: 'original',
|
||||
});
|
||||
const original = (await db.query('SELECT * FROM asset_entries WHERE id = $1', [id])).rows[0];
|
||||
const originalBlobs = (await db.query('SELECT * FROM asset_blobs ORDER BY content_hash')).rows;
|
||||
const cases: Array<[string, AssetMeta, string, string?]> = [
|
||||
[
|
||||
'Date',
|
||||
{ invalid: new Date('2026-01-01T00:00:00.000Z') } as unknown as AssetMeta,
|
||||
'/invalid',
|
||||
'2026-01-01',
|
||||
],
|
||||
[
|
||||
'Map',
|
||||
{ invalid: new Map([['map-secret', 'value']]) } as unknown as AssetMeta,
|
||||
'/invalid',
|
||||
'map-secret',
|
||||
],
|
||||
['negative zero', { invalid: -0 } as AssetMeta, '/invalid'],
|
||||
[
|
||||
'nested undefined',
|
||||
{ nested: { invalid: undefined } } as unknown as AssetMeta,
|
||||
'/nested/invalid',
|
||||
],
|
||||
['U+0000', { invalid: 'nul-secret\u0000tail' } as AssetMeta, '/invalid', 'nul-secret'],
|
||||
];
|
||||
|
||||
for (const [name, invalidMeta, path, hiddenValue] of cases) {
|
||||
for (const [operation, write] of [
|
||||
['put', () => store.put(PRINCIPAL, blob('new bytes'), invalidMeta)],
|
||||
['replace', () => store.replace(PRINCIPAL, id, blob('replacement'), invalidMeta)],
|
||||
] as const) {
|
||||
let thrown: unknown;
|
||||
try {
|
||||
await write();
|
||||
} catch (error) {
|
||||
thrown = error;
|
||||
}
|
||||
expect(thrown, `${operation} ${name}`).toBeInstanceOf(Error);
|
||||
expect((thrown as Error).message, `${operation} ${name}`).toContain(`'${path}'`);
|
||||
if (hiddenValue !== undefined) {
|
||||
expect((thrown as Error).message, `${operation} ${name}`).not.toContain(hiddenValue);
|
||||
}
|
||||
expect(
|
||||
(await db.query('SELECT * FROM asset_entries ORDER BY id')).rows,
|
||||
`${operation} ${name}`,
|
||||
).toEqual([original]);
|
||||
expect(
|
||||
(await db.query('SELECT * FROM asset_blobs ORDER BY content_hash')).rows,
|
||||
`${operation} ${name}`,
|
||||
).toEqual(originalBlobs);
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
test('accepts plain metadata objects on put and replace', async () => {
|
||||
const id = await store.put(PRINCIPAL, blob('plain put'), {
|
||||
contentType: 'image/png',
|
||||
nested: { accepted: true },
|
||||
});
|
||||
await expect(
|
||||
store.replace(PRINCIPAL, id, blob('plain replace'), {
|
||||
contentType: 'audio/mpeg',
|
||||
nested: { accepted: ['yes'] },
|
||||
}),
|
||||
).resolves.toBe(2);
|
||||
expect(
|
||||
(await db.query<{ meta: unknown }>('SELECT meta FROM asset_entries WHERE id = $1', [id]))
|
||||
.rows[0]?.meta,
|
||||
).toEqual({ contentType: 'audio/mpeg', nested: { accepted: ['yes'] } });
|
||||
});
|
||||
|
||||
test('an entry whose bytes are gone resolves as a miss', async () => {
|
||||
const id = await store.put(PRINCIPAL, blob('missing bytes'));
|
||||
await db.query('UPDATE asset_blobs SET bytes = NULL');
|
||||
expect(await store.resolve(PRINCIPAL, id)).toBeNull();
|
||||
});
|
||||
|
||||
test('an over-quota replace raises the quota error rather than a generic failure', async () => {
|
||||
// The quota check runs inside the write transaction, so the transaction's
|
||||
// catch has to let this error through. Collapsing it would answer 500 for
|
||||
// a condition the contract gives a status and a code of its own.
|
||||
const quotaStore = new PgAssetStore(db, options(db, byteStore, { quotaBytes: 5 }));
|
||||
const id = await quotaStore.put(PRINCIPAL, blob('1'));
|
||||
|
||||
await expect(quotaStore.replace(PRINCIPAL, id, blob('123456'))).rejects.toBeInstanceOf(
|
||||
AssetQuotaExceededError,
|
||||
);
|
||||
expect((await quotaStore.resolve(PRINCIPAL, id))?.bytes).toEqual(bytes('1'));
|
||||
});
|
||||
|
||||
test('logical quota counts every principal entry and runs before byte writes', async () => {
|
||||
const writes: string[] = [];
|
||||
const observingBytes: AssetByteStore = {
|
||||
write: async () => {
|
||||
writes.push('write');
|
||||
},
|
||||
read: async () => null,
|
||||
delete: async () => undefined,
|
||||
};
|
||||
const quotaStore = new PgAssetStore(db, options(db, observingBytes, { quotaBytes: 5 }));
|
||||
await quotaStore.put(PRINCIPAL, blob('12345'));
|
||||
await expect(quotaStore.put(PRINCIPAL, blob('1'))).rejects.toBeInstanceOf(
|
||||
AssetQuotaExceededError,
|
||||
);
|
||||
// One write for the accepted put, none for the rejected one: the quota check
|
||||
// runs before any byte reaches the byte store.
|
||||
expect(writes).toEqual(['write']);
|
||||
});
|
||||
|
||||
test('principal quota locks use the 64-bit hash function', async () => {
|
||||
const statements: string[] = [];
|
||||
const instrumented = new PgAssetStore(recordingQueryable(db, statements), {
|
||||
byteStore,
|
||||
withTransaction: recordingTransactions(db, statements),
|
||||
quotaBytes: 10,
|
||||
});
|
||||
|
||||
await instrumented.put(PRINCIPAL, blob('lock'));
|
||||
|
||||
// This SQL-text assertion deliberately pins the 64-bit lock key: a
|
||||
// behavioral test would depend on unstable collisions in PostgreSQL's
|
||||
// internal hash function.
|
||||
expect(statements).toContain('SELECT pg_advisory_xact_lock(hashtextextended($1, 0))');
|
||||
});
|
||||
|
||||
test('byte-layer quota errors collapse to generic registry failures', async () => {
|
||||
const internalDetail = 'internal byte-layer detail';
|
||||
const failingBytes: AssetByteStore = {
|
||||
write: async () => {
|
||||
throw new AssetQuotaExceededError(internalDetail);
|
||||
},
|
||||
read: async () => null,
|
||||
delete: async () => undefined,
|
||||
};
|
||||
const failing = new PgAssetStore(db, options(db, failingBytes));
|
||||
|
||||
let putError: unknown;
|
||||
try {
|
||||
await failing.put(PRINCIPAL, blob('put'));
|
||||
} catch (error) {
|
||||
putError = error;
|
||||
}
|
||||
expect(putError).toBeInstanceOf(Error);
|
||||
expect(putError).not.toBeInstanceOf(AssetQuotaExceededError);
|
||||
expect((putError as Error).message).toBe('@openmaic/storage: asset registry put failed');
|
||||
expect((putError as Error).message).not.toContain(internalDetail);
|
||||
|
||||
const id = await store.put(PRINCIPAL, blob('original'));
|
||||
let replaceError: unknown;
|
||||
try {
|
||||
await failing.replace(PRINCIPAL, id, blob('replacement'));
|
||||
} catch (error) {
|
||||
replaceError = error;
|
||||
}
|
||||
expect(replaceError).toBeInstanceOf(Error);
|
||||
expect(replaceError).not.toBeInstanceOf(AssetQuotaExceededError);
|
||||
expect((replaceError as Error).message).toBe(
|
||||
'@openmaic/storage: asset registry replace failed',
|
||||
);
|
||||
expect((replaceError as Error).message).not.toContain(internalDetail);
|
||||
});
|
||||
|
||||
test('put emits an identical statement sequence for existing and new bytes', async () => {
|
||||
async function observe(seed: boolean): Promise<string[]> {
|
||||
const local = new PGlite();
|
||||
await local.waitReady;
|
||||
await ensureAssetSchema(local);
|
||||
const directBytes = new PgAssetByteStore(local);
|
||||
const base = new PgAssetStore(local, options(local, directBytes));
|
||||
if (seed) await base.put(PRINCIPAL, blob('statement equality'));
|
||||
|
||||
const statements: string[] = [];
|
||||
const recorded = recordingQueryable(local, statements);
|
||||
const recordedBytes = new PgAssetByteStore(recorded);
|
||||
const instrumented = new PgAssetStore(recorded, {
|
||||
byteStore: recordedBytes,
|
||||
withTransaction: recordingTransactions(local, statements),
|
||||
});
|
||||
await instrumented.put(PRINCIPAL, blob(seed ? 'statement equality' : 'brand new'));
|
||||
await local.close();
|
||||
return statements;
|
||||
}
|
||||
|
||||
const existing = await observe(true);
|
||||
const fresh = await observe(false);
|
||||
expect(existing).toEqual(fresh);
|
||||
// Blob row claimed, then bytes, then the entry. The byte write sits between
|
||||
// the two registry writes deliberately: it must follow the upsert that takes
|
||||
// the blob row's lock, and precede the entry that references it.
|
||||
expect(existing.map((sql) => sql.split(' ')[0])).toEqual(['INSERT', 'UPDATE', 'INSERT']);
|
||||
});
|
||||
|
||||
test('remove emits the same statements with and without another principal reference', async () => {
|
||||
async function observe(shared: boolean): Promise<string[]> {
|
||||
const local = new PGlite();
|
||||
await local.waitReady;
|
||||
await ensureAssetSchema(local);
|
||||
const baseBytes = new PgAssetByteStore(local);
|
||||
const base = new PgAssetStore(local, options(local, baseBytes));
|
||||
const id = await base.put(PRINCIPAL, blob('remove cost'));
|
||||
if (shared) await base.put(OTHER_PRINCIPAL, blob('remove cost'));
|
||||
|
||||
const statements: string[] = [];
|
||||
const instrumented = new PgAssetStore(recordingQueryable(local, statements), {
|
||||
byteStore: baseBytes,
|
||||
withTransaction: recordingTransactions(local, statements),
|
||||
});
|
||||
await instrumented.remove(PRINCIPAL, id);
|
||||
await local.close();
|
||||
return statements;
|
||||
}
|
||||
|
||||
expect(await observe(false)).toEqual(await observe(true));
|
||||
});
|
||||
|
||||
test('a transactional byte layer leaves nothing behind when the registry write fails', async () => {
|
||||
// Its bytes are written through the registry's own transaction, so a
|
||||
// rollback takes them with it. This layer has nothing to reconcile.
|
||||
const local = new PGlite();
|
||||
await local.waitReady;
|
||||
await ensureAssetSchema(local);
|
||||
const layer = new PgAssetByteStore(local);
|
||||
const failing = new PgAssetStore(local, {
|
||||
byteStore: layer,
|
||||
withTransaction: (body) =>
|
||||
local.transaction(async (tx: Queryable) => {
|
||||
await body(tx);
|
||||
throw new Error('injected failure after the body committed nothing');
|
||||
}) as Promise<never>,
|
||||
});
|
||||
const data = blob('crash window');
|
||||
const { contentHash } = await contentHashOf(data);
|
||||
|
||||
await expect(failing.put(PRINCIPAL, data)).rejects.toThrow(/registry put failed/);
|
||||
|
||||
expect((await local.query('SELECT * FROM asset_entries')).rows).toEqual([]);
|
||||
expect((await local.query('SELECT * FROM asset_blobs')).rows).toEqual([]);
|
||||
expect(await layer.read(contentHash)).toBeNull();
|
||||
await local.close();
|
||||
});
|
||||
|
||||
test('a non-transactional byte layer leaves an orphan the collector cannot see', async () => {
|
||||
// An object store cannot join the registry transaction, so a rollback
|
||||
// strands its bytes -- and strands them with no blob row, which is what
|
||||
// puts them beyond reference counting. Recovering them is deployment
|
||||
// housekeeping (a lifecycle rule or a bucket-versus-table reconciliation),
|
||||
// not something the collector can do.
|
||||
const local = new PGlite();
|
||||
await local.waitReady;
|
||||
await ensureAssetSchema(local);
|
||||
const layer = new MemoryByteStore();
|
||||
const failing = new PgAssetStore(local, {
|
||||
byteStore: layer,
|
||||
withTransaction: (body) =>
|
||||
local.transaction(async (tx: Queryable) => {
|
||||
await body(tx);
|
||||
throw new Error('injected failure after the body committed nothing');
|
||||
}) as Promise<never>,
|
||||
});
|
||||
const data = blob('crash window');
|
||||
const { contentHash } = await contentHashOf(data);
|
||||
|
||||
await expect(failing.put(PRINCIPAL, data)).rejects.toThrow(/registry put failed/);
|
||||
|
||||
expect((await local.query('SELECT * FROM asset_entries')).rows).toEqual([]);
|
||||
expect((await local.query('SELECT * FROM asset_blobs')).rows).toEqual([]);
|
||||
expect(await layer.read(contentHash)).toEqual(bytes('crash window'));
|
||||
|
||||
const collector = new AssetCollector(local, layer, {
|
||||
withTransaction: transactions(local),
|
||||
graceMs: 0,
|
||||
now: () => new Date('2026-01-01T00:00:00.000Z'),
|
||||
});
|
||||
expect(await collector.collect()).toBe(0);
|
||||
expect(await layer.read(contentHash)).toEqual(bytes('crash window'));
|
||||
await local.close();
|
||||
});
|
||||
|
||||
test('collector observes grace, re-checks references, and is re-runnable', async () => {
|
||||
const oldId = await store.put(PRINCIPAL, blob('old unreferenced'));
|
||||
const referencedId = await store.put(PRINCIPAL, blob('still referenced'));
|
||||
await store.remove(PRINCIPAL, oldId);
|
||||
await db.query(
|
||||
`UPDATE asset_blobs
|
||||
SET unreferenced_at = '2026-01-01T00:00:00.000Z'
|
||||
WHERE unreferenced_at IS NOT NULL`,
|
||||
);
|
||||
await db.query(
|
||||
`UPDATE asset_blobs
|
||||
SET unreferenced_at = '2000-01-01T00:00:00.000Z'
|
||||
WHERE content_hash = (
|
||||
SELECT content_hash FROM asset_entries WHERE id = $1
|
||||
)`,
|
||||
[referencedId],
|
||||
);
|
||||
const collector = new AssetCollector(db, byteStore, {
|
||||
withTransaction: transactions(db),
|
||||
graceMs: 60 * 60 * 1000,
|
||||
now: () => new Date('2026-01-01T00:30:00.000Z'),
|
||||
});
|
||||
expect(await collector.collect()).toBe(0);
|
||||
|
||||
const later = new AssetCollector(db, byteStore, {
|
||||
withTransaction: transactions(db),
|
||||
graceMs: 60 * 60 * 1000,
|
||||
now: () => new Date('2026-01-01T02:00:00.000Z'),
|
||||
});
|
||||
expect(await later.collect()).toBe(1);
|
||||
expect(await later.collect()).toBe(0);
|
||||
expect((await store.resolve(PRINCIPAL, referencedId))?.bytes).toEqual(
|
||||
bytes('still referenced'),
|
||||
);
|
||||
});
|
||||
|
||||
test('put writes bytes unconditionally, even when they are already stored', async () => {
|
||||
// This is what makes an adopting write safe against the collector. The
|
||||
// collector can hold the blob row's lock, delete those bytes and commit
|
||||
// while this put's upsert waits; the put then re-claims the row and writes
|
||||
// again. A put that skipped the write when the bytes looked present would
|
||||
// resolve to nothing -- and would also be branching on prior existence,
|
||||
// which the allocation rule forbids.
|
||||
const local = new PGlite();
|
||||
await local.waitReady;
|
||||
await ensureAssetSchema(local);
|
||||
const layer = new MemoryByteStore();
|
||||
let writes = 0;
|
||||
layer.onWrite = () => {
|
||||
writes += 1;
|
||||
};
|
||||
const registry = new PgAssetStore(local, options(local, layer));
|
||||
|
||||
await registry.put(PRINCIPAL, blob('unconditional'));
|
||||
await registry.put(PRINCIPAL, blob('unconditional'));
|
||||
|
||||
expect(writes).toBe(2);
|
||||
await local.close();
|
||||
});
|
||||
|
||||
test('a put re-stores bytes the collector has already removed', async () => {
|
||||
const local = new PGlite();
|
||||
await local.waitReady;
|
||||
await ensureAssetSchema(local);
|
||||
const layer = new MemoryByteStore();
|
||||
const registry = new PgAssetStore(local, options(local, layer));
|
||||
const data = blob('adopting write');
|
||||
const { contentHash } = await contentHashOf(data);
|
||||
|
||||
const first = await registry.put(PRINCIPAL, data);
|
||||
await registry.remove(PRINCIPAL, first);
|
||||
await local.query(`UPDATE asset_blobs SET unreferenced_at = '2000-01-01T00:00:00.000Z'`);
|
||||
const collector = new AssetCollector(local, layer, {
|
||||
withTransaction: transactions(local),
|
||||
graceMs: 0,
|
||||
now: () => new Date('2026-01-01T00:00:00.000Z'),
|
||||
});
|
||||
expect(await collector.collect()).toBe(1);
|
||||
expect(await layer.read(contentHash)).toBeNull();
|
||||
|
||||
const adopted = await registry.put(PRINCIPAL, data);
|
||||
expect(await registry.resolve(PRINCIPAL, adopted)).toMatchObject({
|
||||
bytes: bytes('adopting write'),
|
||||
revision: 1,
|
||||
});
|
||||
await local.close();
|
||||
});
|
||||
|
||||
test('every registry error path keeps common digest encodings out of thrown values', async () => {
|
||||
const data = blob('digest-sensitive failure');
|
||||
const { contentHash } = await contentHashOf(data);
|
||||
expect(await commonDigestEncodings(data)).toHaveLength(5);
|
||||
|
||||
const digestFailureBytes: AssetByteStore = {
|
||||
write: async () => {
|
||||
throw new Error(contentHash);
|
||||
},
|
||||
read: async () => {
|
||||
throw new Error(contentHash);
|
||||
},
|
||||
delete: async () => {
|
||||
throw new Error(contentHash);
|
||||
},
|
||||
};
|
||||
const failures: Array<() => Promise<unknown>> = [
|
||||
() => new PgAssetStore(db, options(db, digestFailureBytes)).put(PRINCIPAL, data),
|
||||
async () => {
|
||||
const id = await store.put(PRINCIPAL, data);
|
||||
return new PgAssetStore(db, options(db, digestFailureBytes)).resolve(PRINCIPAL, id);
|
||||
},
|
||||
() => store.replace(PRINCIPAL, 'unknown' as AssetId, data),
|
||||
async () => {
|
||||
const id = await store.put(PRINCIPAL, data);
|
||||
return store.replace(OTHER_PRINCIPAL, id, data);
|
||||
},
|
||||
() => new PgAssetStore(db, options(db, byteStore, { quotaBytes: 0 })).put(PRINCIPAL, data),
|
||||
() =>
|
||||
new PgAssetStore(db, {
|
||||
byteStore: new MemoryByteStore(),
|
||||
withTransaction: async () => {
|
||||
throw new Error(contentHash);
|
||||
},
|
||||
}).put(PRINCIPAL, data),
|
||||
async () => {
|
||||
const id = await store.put(PRINCIPAL, data);
|
||||
return new PgAssetStore(db, {
|
||||
byteStore,
|
||||
withTransaction: async () => {
|
||||
throw new Error(contentHash);
|
||||
},
|
||||
}).resolve(PRINCIPAL, id);
|
||||
},
|
||||
async () => {
|
||||
const id = await store.put(PRINCIPAL, data);
|
||||
return new PgAssetStore(db, {
|
||||
byteStore,
|
||||
withTransaction: async () => {
|
||||
throw new Error(contentHash);
|
||||
},
|
||||
}).remove(PRINCIPAL, id);
|
||||
},
|
||||
async () => {
|
||||
const id = await store.put(PRINCIPAL, data);
|
||||
return new PgAssetStore(db, {
|
||||
byteStore: new MemoryByteStore(),
|
||||
withTransaction: async () => {
|
||||
throw new Error(contentHash);
|
||||
},
|
||||
}).replace(PRINCIPAL, id, data);
|
||||
},
|
||||
];
|
||||
|
||||
for (const fail of failures) {
|
||||
let thrown: unknown;
|
||||
try {
|
||||
await fail();
|
||||
} catch (error) {
|
||||
thrown = error;
|
||||
}
|
||||
expect(thrown).toBeInstanceOf(Error);
|
||||
await expectNoDigestSubstring(String(thrown), data);
|
||||
}
|
||||
|
||||
await db.query('TRUNCATE asset_entries, asset_blobs');
|
||||
const collectorBytes: AssetByteStore = {
|
||||
write: async () => undefined,
|
||||
read: async () => null,
|
||||
delete: async () => {
|
||||
throw new Error(contentHash);
|
||||
},
|
||||
};
|
||||
const collectorStore = new PgAssetStore(db, options(db, collectorBytes));
|
||||
const collectorId = await collectorStore.put(PRINCIPAL, data);
|
||||
await collectorStore.remove(PRINCIPAL, collectorId);
|
||||
await db.query(`UPDATE asset_blobs SET unreferenced_at = '2000-01-01T00:00:00.000Z'`);
|
||||
const collector = new AssetCollector(db, collectorBytes, {
|
||||
withTransaction: transactions(db),
|
||||
graceMs: 0,
|
||||
now: () => new Date('2026-01-01T00:00:00.000Z'),
|
||||
});
|
||||
let collectorError: unknown;
|
||||
try {
|
||||
await collector.collect();
|
||||
} catch (error) {
|
||||
collectorError = error;
|
||||
}
|
||||
expect(collectorError).toBeInstanceOf(Error);
|
||||
await expectNoDigestSubstring(String(collectorError), data);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,69 @@
|
||||
import { afterAll, beforeAll, describe, expect, test } from 'vitest';
|
||||
import { HeadBucketCommand, S3Client } from '@aws-sdk/client-s3';
|
||||
import { contentHashOf, type ContentHash } from '../src/asset/blob.js';
|
||||
import { S3AssetByteStore } from '../src/asset/s3-bytes.js';
|
||||
|
||||
const endpoint = process.env.S3_CONTRACT_ENDPOINT;
|
||||
const bucket = process.env.S3_CONTRACT_BUCKET;
|
||||
const configured = Boolean(endpoint && bucket);
|
||||
|
||||
if (process.env.STORAGE_S3_CONTRACT_REQUIRED === '1' && !configured) {
|
||||
throw new Error(
|
||||
'@openmaic/storage: STORAGE_S3_CONTRACT_REQUIRED=1 requires S3_CONTRACT_ENDPOINT and S3_CONTRACT_BUCKET; refusing to skip the S3 asset suite',
|
||||
);
|
||||
}
|
||||
|
||||
describe.skipIf(!configured)('S3AssetByteStore with an S3-compatible server', () => {
|
||||
let client: S3Client;
|
||||
let store: S3AssetByteStore;
|
||||
const written = new Set<ContentHash>();
|
||||
|
||||
beforeAll(async () => {
|
||||
client = new S3Client({
|
||||
endpoint,
|
||||
forcePathStyle: true,
|
||||
region: process.env.S3_CONTRACT_REGION ?? 'us-east-1',
|
||||
credentials:
|
||||
process.env.S3_CONTRACT_ACCESS_KEY && process.env.S3_CONTRACT_SECRET_KEY
|
||||
? {
|
||||
accessKeyId: process.env.S3_CONTRACT_ACCESS_KEY,
|
||||
secretAccessKey: process.env.S3_CONTRACT_SECRET_KEY,
|
||||
}
|
||||
: undefined,
|
||||
});
|
||||
await client.send(new HeadBucketCommand({ Bucket: bucket! }));
|
||||
store = new S3AssetByteStore({ client, bucket: bucket! });
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await Promise.all([...written].map((hash) => store.delete(hash)));
|
||||
client.destroy();
|
||||
});
|
||||
|
||||
test('round-trips and deletes an object against real S3 semantics', async () => {
|
||||
const data = new Blob([`s3-real-${crypto.randomUUID()}`]);
|
||||
const { contentHash, bytes } = await contentHashOf(data);
|
||||
written.add(contentHash);
|
||||
await store.write(contentHash, new Uint8Array(bytes));
|
||||
expect(await store.read(contentHash)).toEqual(new Uint8Array(bytes));
|
||||
await store.delete(contentHash);
|
||||
written.delete(contentHash);
|
||||
expect(await store.read(contentHash)).toBeNull();
|
||||
});
|
||||
|
||||
test('plain PUT is idempotent for an existing hash-named key', async () => {
|
||||
const data = new Blob([`s3-idempotent-${crypto.randomUUID()}`]);
|
||||
const { contentHash, bytes } = await contentHashOf(data);
|
||||
written.add(contentHash);
|
||||
await store.write(contentHash, new Uint8Array(bytes));
|
||||
await expect(store.write(contentHash, new Uint8Array(bytes))).resolves.toBeUndefined();
|
||||
expect(await store.read(contentHash)).toEqual(new Uint8Array(bytes));
|
||||
});
|
||||
|
||||
test('missing reads and repeated deletes are idempotent', async () => {
|
||||
const { contentHash } = await contentHashOf(new Blob([`s3-missing-${crypto.randomUUID()}`]));
|
||||
expect(await store.read(contentHash)).toBeNull();
|
||||
await expect(store.delete(contentHash)).resolves.toBeUndefined();
|
||||
await expect(store.delete(contentHash)).resolves.toBeUndefined();
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,104 @@
|
||||
import { DeleteObjectCommand, GetObjectCommand, PutObjectCommand } from '@aws-sdk/client-s3';
|
||||
import { describe, expect, test } from 'vitest';
|
||||
import { contentHashOf } from '../src/asset/blob.js';
|
||||
import { S3AssetByteStore } from '../src/asset/s3-bytes.js';
|
||||
import { expectNoDigestSubstring } from './asset-contract.js';
|
||||
import { runAssetByteStoreContract } from './asset-byte-store-contract.js';
|
||||
|
||||
class MemoryS3Client {
|
||||
readonly objects = new Map<string, Uint8Array>();
|
||||
|
||||
async send(command: unknown): Promise<unknown> {
|
||||
if (command instanceof PutObjectCommand) {
|
||||
const body = command.input.Body;
|
||||
if (!(body instanceof Uint8Array)) throw new Error('expected byte body');
|
||||
this.objects.set(command.input.Key!, new Uint8Array(body));
|
||||
return {};
|
||||
}
|
||||
if (command instanceof GetObjectCommand) {
|
||||
const bytes = this.objects.get(command.input.Key!);
|
||||
if (!bytes) throw Object.assign(new Error('missing'), { name: 'NoSuchKey' });
|
||||
return {
|
||||
Body: {
|
||||
transformToByteArray: async () => new Uint8Array(bytes),
|
||||
},
|
||||
};
|
||||
}
|
||||
if (command instanceof DeleteObjectCommand) {
|
||||
this.objects.delete(command.input.Key!);
|
||||
return {};
|
||||
}
|
||||
throw new Error('unsupported command');
|
||||
}
|
||||
}
|
||||
|
||||
function makeStore(client = new MemoryS3Client()): S3AssetByteStore {
|
||||
return new S3AssetByteStore({
|
||||
client: client as never,
|
||||
bucket: 'asset-contract',
|
||||
});
|
||||
}
|
||||
|
||||
runAssetByteStoreContract('S3 bytes (in-memory SDK double)', () => makeStore());
|
||||
|
||||
describe('S3AssetByteStore commands and failures', () => {
|
||||
test('uses the content hash as the complete object key', async () => {
|
||||
const client = new MemoryS3Client();
|
||||
const store = makeStore(client);
|
||||
const data = new Blob(['keyed bytes']);
|
||||
const { contentHash, bytes } = await contentHashOf(data);
|
||||
await store.write(contentHash, new Uint8Array(bytes));
|
||||
expect([...client.objects.keys()]).toEqual([contentHash]);
|
||||
});
|
||||
|
||||
test('maps not-found reads to null', async () => {
|
||||
const store = makeStore();
|
||||
const { contentHash } = await contentHashOf(new Blob(['absent']));
|
||||
await expect(store.read(contentHash)).resolves.toBeNull();
|
||||
});
|
||||
|
||||
test('a bucket-level 404 is a failure, not an absent object', async () => {
|
||||
// NoSuchBucket, a misdirected endpoint, and a revoked access point all
|
||||
// answer 404 while the bytes still exist. Reporting one as a miss would
|
||||
// have the registry answer ASSET_NOT_FOUND for a live entry, and a caller
|
||||
// that clears its reference on that turns an outage into data loss.
|
||||
const failing = {
|
||||
async send(): Promise<never> {
|
||||
throw Object.assign(new Error('bucket is gone'), {
|
||||
name: 'NoSuchBucket',
|
||||
$metadata: { httpStatusCode: 404 },
|
||||
});
|
||||
},
|
||||
};
|
||||
const store = makeStore(failing as never);
|
||||
const { contentHash } = await contentHashOf(new Blob(['still here']));
|
||||
|
||||
await expect(store.read(contentHash)).rejects.toThrow(/S3 asset byte read failed/);
|
||||
});
|
||||
|
||||
test('sanitizes SDK failures so the digest cannot escape', async () => {
|
||||
const data = new Blob(['secret failure bytes']);
|
||||
const { contentHash, bytes } = await contentHashOf(data);
|
||||
const failing = {
|
||||
send: async () => {
|
||||
throw new Error(contentHash);
|
||||
},
|
||||
};
|
||||
const store = new S3AssetByteStore({ client: failing as never, bucket: 'failure' });
|
||||
|
||||
for (const operation of [
|
||||
() => store.write(contentHash, new Uint8Array(bytes)),
|
||||
() => store.read(contentHash),
|
||||
() => store.delete(contentHash),
|
||||
]) {
|
||||
let thrown: unknown;
|
||||
try {
|
||||
await operation();
|
||||
} catch (error) {
|
||||
thrown = error;
|
||||
}
|
||||
expect(thrown).toBeInstanceOf(Error);
|
||||
await expectNoDigestSubstring(String(thrown), data);
|
||||
}
|
||||
});
|
||||
});
|
||||
Generated
+376
-17
@@ -55,7 +55,7 @@ importers:
|
||||
version: 1.2.0(@types/react@19.2.14)(react-dom@19.2.3(react@19.2.3))(react@19.2.3)
|
||||
'@copilotkit/backend':
|
||||
specifier: ^0.37.0
|
||||
version: 0.37.0(@aws-crypto/sha256-js@5.2.0)(@aws-sdk/client-bedrock-runtime@3.1048.0)(@aws-sdk/credential-provider-node@3.972.57)(@smithy/util-utf8@2.3.0)(axios@1.13.6)(fast-xml-parser@5.7.3)(ignore@5.3.2)(jsdom@29.1.1(@noble/hashes@1.8.0)(canvas@3.2.3))(lodash@4.17.23)(pg@8.22.0)(playwright@1.58.2)(ws@8.19.0)
|
||||
version: 0.37.0(@aws-crypto/sha256-js@5.2.0)(@aws-sdk/client-bedrock-runtime@3.1048.0)(@aws-sdk/client-s3@3.1106.0)(@aws-sdk/credential-provider-node@3.972.78)(@smithy/util-utf8@2.3.0)(axios@1.13.6)(fast-xml-parser@5.7.3)(ignore@5.3.2)(jsdom@29.1.1(@noble/hashes@1.8.0)(canvas@3.2.3))(lodash@4.17.23)(pg@8.22.0)(playwright@1.58.2)(ws@8.19.0)
|
||||
'@copilotkit/runtime':
|
||||
specifier: ^1.51.2
|
||||
version: 1.53.0(@ag-ui/encoder@0.0.47)(@cfworker/json-schema@4.1.1)(@copilotkitnext/shared@1.53.0)(@langchain/core@1.1.31(@opentelemetry/api@1.9.0)(openai@4.104.0(ws@8.19.0)(zod@4.3.6)))(@langchain/langgraph-sdk@1.7.1(@langchain/core@1.1.31(@opentelemetry/api@1.9.0)(openai@4.104.0(ws@8.19.0)(zod@4.3.6))))(@opentelemetry/api@1.9.0)(react-dom@19.2.3(react@19.2.3))(react@19.2.3)(ws@8.19.0)
|
||||
@@ -661,6 +661,9 @@ importers:
|
||||
specifier: workspace:^
|
||||
version: link:../dsl
|
||||
devDependencies:
|
||||
'@aws-sdk/client-s3':
|
||||
specifier: ^3.1048.0
|
||||
version: 3.1048.0
|
||||
'@electric-sql/pglite':
|
||||
specifier: ^0.3.14
|
||||
version: 0.3.16
|
||||
@@ -1090,6 +1093,9 @@ packages:
|
||||
resolution: {integrity: sha512-nLbCWqQNgUiwwtFsen1AdzAtvuLRsQS8rYgMuxCrdKf9kOssamGLuPwyTY9wyYblNr9+1XM8v6zoDTPPSIeANg==}
|
||||
engines: {node: '>=16.0.0'}
|
||||
|
||||
'@aws-crypto/sha1-browser@5.2.0':
|
||||
resolution: {integrity: sha512-OH6lveCFfcDjX4dbAvCFSYUjJZjDr/3XJ3xHtjn3Oj5b9RjojQo8npoLeA/bNwkOkrSQ0wgrHzXk4tDRxGKJeg==}
|
||||
|
||||
'@aws-crypto/sha256-browser@5.2.0':
|
||||
resolution: {integrity: sha512-AXfN/lGotSQwu6HNcEsIASo7kWXZ5HYWvfOmSNKDsEqC4OashTp8alTmaz+F7TC2L083SFv5RdB+qU3Vs1kZqw==}
|
||||
|
||||
@@ -1103,6 +1109,10 @@ packages:
|
||||
'@aws-crypto/util@5.2.0':
|
||||
resolution: {integrity: sha512-4RkU9EsI6ZpBve5fseQlGNUWKMa1RLPQ1dnjnQoe07ldfIzcsGb5hC5W0Dm7u423KWzawlrpbjXBrXCEv9zazQ==}
|
||||
|
||||
'@aws-sdk/checksums@3.1000.26':
|
||||
resolution: {integrity: sha512-CGznePoL+1oWCSzqmkvlYMpWQEZohPR3LntjKXXfVH+oh6Kd8d+yHLkjpiAGwPIHAxFe4OAq3aKfRnv/wqWIyA==}
|
||||
engines: {node: '>=20.0.0'}
|
||||
|
||||
'@aws-sdk/client-bedrock-runtime@3.1048.0':
|
||||
resolution: {integrity: sha512-u+NT61JZEkRFtpL0CAw1N1dwxnaLgwVXQl/zjJxTGgLyS/jTIdg2SdoEoCTHxgDyCnqa1HEi9QOoE9/pYRNpOQ==}
|
||||
engines: {node: '>=20.0.0'}
|
||||
@@ -1111,6 +1121,14 @@ packages:
|
||||
resolution: {integrity: sha512-3OEn8zvtfJoN0jFfjVJ9jF2GVRDL3IjDfk6CAgVTAqjfCVjajiUD0iFAGQ4cOzdcv1LGsZ0b/snJDWalY3OePQ==}
|
||||
engines: {node: '>=20.0.0'}
|
||||
|
||||
'@aws-sdk/client-s3@3.1048.0':
|
||||
resolution: {integrity: sha512-SrJn5FteqqtcDBgQIvqLKk3Qn/2vSsi5XR03I53EDDR4CbCdLysVSNgUnjVncEECMua9Pz+nxO0/lEx3TP+6mA==}
|
||||
engines: {node: '>=20.0.0'}
|
||||
|
||||
'@aws-sdk/client-s3@3.1106.0':
|
||||
resolution: {integrity: sha512-hUTlnyRRGlVdfvJLL3hCEnMm7CmunSzc/lxFVRX8g1fjJTMUVbyQCfhfMhp7dZ7JBftLU6OYresu3Hje4nvkJw==}
|
||||
engines: {node: '>=20.0.0'}
|
||||
|
||||
'@aws-sdk/core@3.974.22':
|
||||
resolution: {integrity: sha512-YofH63shc6YRdXjz80BJkpJW+Bkn0Cuu2dn4Rv7s9G2Idt58tgtzQEWxrR2xVljlVfIBeUjPuULnSVYLke3sUQ==}
|
||||
engines: {node: '>=20.0.0'}
|
||||
@@ -1119,6 +1137,10 @@ packages:
|
||||
resolution: {integrity: sha512-7ur3kCKuvPLqlsZ2XlvnNBVQ7KkpSu6Y6dOTwSPHLrFpTEfZM8isLBJc4cgv96WB7GifeVM436mpycwxBd2vEA==}
|
||||
engines: {node: '>=20.0.0'}
|
||||
|
||||
'@aws-sdk/core@3.977.6':
|
||||
resolution: {integrity: sha512-QiaJV4/zDrB4ZY2mfeSXSzSTc36W16sZXcGz+SPFk0CJ26gziO0cS+4LjJUMAbdeeBOvS0k0Aq1cZpfGdUXxSw==}
|
||||
engines: {node: '>=20.0.0'}
|
||||
|
||||
'@aws-sdk/credential-provider-cognito-identity@3.972.58':
|
||||
resolution: {integrity: sha512-s5uoABv5eOzuH/S+XngHjHSrY8mK0UTBUFs8pm1ynBNuxXmYp176zarDyxN9lUS3Rry0wjzNvJUV09QROaO98g==}
|
||||
engines: {node: '>=20.0.0'}
|
||||
@@ -1127,34 +1149,66 @@ packages:
|
||||
resolution: {integrity: sha512-h6FEC95fbexUd6zxm4PdgS82bTcI2PRtUb2ZwMipb/Xr8bPwtf0G8rBo2jp7NA24Mbx2JA8/WingiYpA9RCCyw==}
|
||||
engines: {node: '>=20.0.0'}
|
||||
|
||||
'@aws-sdk/credential-provider-env@3.972.67':
|
||||
resolution: {integrity: sha512-rcIpk5kxUqDaaNa6Xk23pQ6ViY7jlqzmfFWCahQcBT97ddXaXYYwzCen9Tz1Jvo6aJft6wDl5bN44/Jw5B4oLA==}
|
||||
engines: {node: '>=20.0.0'}
|
||||
|
||||
'@aws-sdk/credential-provider-http@3.972.50':
|
||||
resolution: {integrity: sha512-lJO3OLpjvz5m/RSBQmsG/CEUGsvCy5ruxKwPQaOCqxqCMuyYT2BZwQUTDZVVwqQ9LrZKuK24JSa6r31hL/tvkg==}
|
||||
engines: {node: '>=20.0.0'}
|
||||
|
||||
'@aws-sdk/credential-provider-http@3.972.69':
|
||||
resolution: {integrity: sha512-nggwJtZ4eeNsUw5IeWBMXsi1ryct5idi0K+/SCRF3kybLubOMaNTb3XCihXpWMiVpyzyPeIrl0zTkzhBH9porA==}
|
||||
engines: {node: '>=20.0.0'}
|
||||
|
||||
'@aws-sdk/credential-provider-ini@3.972.55':
|
||||
resolution: {integrity: sha512-TBoF4buBGYhXjdZAryayY2TrkQj2B2KfE/msG4V53XCt+w0EhEwM2JRjx8p2grJ2C6gtH5++SAwEvGMRdi0yyw==}
|
||||
engines: {node: '>=20.0.0'}
|
||||
|
||||
'@aws-sdk/credential-provider-ini@3.973.12':
|
||||
resolution: {integrity: sha512-pNEf/OeyN5X3VmLKlgSO6TqaWmW10CvI3TfwL1XhsuhYjSLT2VDaxFnCPHnOeQXSaFisMX4jNhpETriqN8DOmg==}
|
||||
engines: {node: '>=20.0.0'}
|
||||
|
||||
'@aws-sdk/credential-provider-login@3.972.54':
|
||||
resolution: {integrity: sha512-hBWI3wZTdTGiuMfmPts6AWbAjFfRniOQnqx68tc2cQvRKWawFbN9wkLOVPWM1FAOyowZU73mC6Fi+rHSHNyLFw==}
|
||||
engines: {node: '>=20.0.0'}
|
||||
|
||||
'@aws-sdk/credential-provider-login@3.972.74':
|
||||
resolution: {integrity: sha512-0AQfDcf99TNmqVKv0owHrw/TQs6i4ZE5t9qmz6NvO53bE/sA/tpXhXL9AAcEP1qHc6Zzjd1UMb69+/9zdhvY3g==}
|
||||
engines: {node: '>=20.0.0'}
|
||||
|
||||
'@aws-sdk/credential-provider-node@3.972.57':
|
||||
resolution: {integrity: sha512-u6dClpzNdWf1HGWz4wwhdXi1wiOofCLniM9S4BQQGlLAN9TW7VB+ld5V533GdKrYMaFeBGFqKnj0JCYvynLqwQ==}
|
||||
engines: {node: '>=20.0.0'}
|
||||
|
||||
'@aws-sdk/credential-provider-node@3.972.78':
|
||||
resolution: {integrity: sha512-OgPAnfvbGAMWac6yvxJ1ihslrvDpPVwR68D2csospdNCCyPvHk9JLzYKwz48SNiS1T2znDwHauywRKRFfpyYng==}
|
||||
engines: {node: '>=20.0.0'}
|
||||
|
||||
'@aws-sdk/credential-provider-process@3.972.48':
|
||||
resolution: {integrity: sha512-w6VZwojPt12WnEkAUy6Nu4K6sWCbBmR7QX390b0nE6vRvkXbrYr9Lq9VySGkfjiMjpUA87op+J4EgvRmtWIDoQ==}
|
||||
engines: {node: '>=20.0.0'}
|
||||
|
||||
'@aws-sdk/credential-provider-process@3.972.67':
|
||||
resolution: {integrity: sha512-IlUEejorGTWKb4/Dm7K5Yw4QxUmXLThLhrvBmzVBqZFTbW72cv9LTcITmo1dsnYriALE4h68mOq4LB99x6sQ7Q==}
|
||||
engines: {node: '>=20.0.0'}
|
||||
|
||||
'@aws-sdk/credential-provider-sso@3.972.54':
|
||||
resolution: {integrity: sha512-23uZpIpF2SIFDCa1fcWa202tK4gGeyvX6GIIAjiB8WBsvsVRBMnJ/7dCxHzxf7eZT7GToJg837LDIBnZsl/VUg==}
|
||||
engines: {node: '>=20.0.0'}
|
||||
|
||||
'@aws-sdk/credential-provider-sso@3.973.11':
|
||||
resolution: {integrity: sha512-gAQBkBZxUB84d71+pPcI9L+jh2ujhuAVxc/4FgGiWFDjkPBlMKxzd5XDtkSXTFX8Ro7ansnT88+XadasxMeCRw==}
|
||||
engines: {node: '>=20.0.0'}
|
||||
|
||||
'@aws-sdk/credential-provider-web-identity@3.972.54':
|
||||
resolution: {integrity: sha512-0Iv5QttS6wcATlodYKgvQj6B9Db51rx7NU9fqu0PoLeS4BIgdYMc/QK4smwLwpm5RFrs02V/eLyEFp3FklvlNQ==}
|
||||
engines: {node: '>=20.0.0'}
|
||||
|
||||
'@aws-sdk/credential-provider-web-identity@3.972.73':
|
||||
resolution: {integrity: sha512-SnlEmQa6SjOgs6iOPLUQl1Eyq4AKiAdPQlkOhFhqNfDtDCwibMGvL6QlkSmf3o6vAUSImzdPCxowT5dfQUZP1A==}
|
||||
engines: {node: '>=20.0.0'}
|
||||
|
||||
'@aws-sdk/credential-providers@3.1045.0':
|
||||
resolution: {integrity: sha512-J+it58HUGyMIAquB6pWtvmO4m0E/gQ/Tz9Xcoogk3Rety13likU5U8HioeIgE+aN1DDOAB//MARoIdLZS1Mpfw==}
|
||||
engines: {node: '>=20.0.0'}
|
||||
@@ -1163,14 +1217,30 @@ packages:
|
||||
resolution: {integrity: sha512-tqPJv0dz4+O0hWGm1a6YekcMZyPhDFs/zH73Von7icaVT5n0Jqvm86typ3jRrG+qoUdPhALOnboRLTmnWQTlYQ==}
|
||||
engines: {node: '>=20.0.0'}
|
||||
|
||||
'@aws-sdk/middleware-bucket-endpoint@3.972.45':
|
||||
resolution: {integrity: sha512-DOnfqPMWenYvKddq5xwdzykreVmmhWzSWwrybXPyxV7eHfZ4SrmZoh+l3kBgXmlm2vo9oVybPt/LhkkQrotUQA==}
|
||||
engines: {node: '>=20.0.0'}
|
||||
|
||||
'@aws-sdk/middleware-eventstream@3.972.18':
|
||||
resolution: {integrity: sha512-OHpk8YoZi3yexPq8aFt1vN1IxA2zLKvsIR5GpWYylX/ve6kQmY7wxHNSFy/D3t2apMZ16rs76Co4dJWcDyIk3A==}
|
||||
engines: {node: '>=20.0.0'}
|
||||
|
||||
'@aws-sdk/middleware-expect-continue@3.972.41':
|
||||
resolution: {integrity: sha512-XCEYwXNiFQLweHrMvNcqAQ/ye6oiHTxKIZKX/tCX38lrPW83xkMGDV7DKpN4R8SiBqd5Y38cnwHSJdFGnigNyA==}
|
||||
engines: {node: '>=20.0.0'}
|
||||
|
||||
'@aws-sdk/middleware-flexible-checksums@3.974.51':
|
||||
resolution: {integrity: sha512-BLWVpJfoHU6OGy1fHgznPCq/eP8MBDfYQdug6Ilyvp6PKe/20sU1LH+Emd/dNs7FyCmzDdtjlk4S7z734Ibydw==}
|
||||
engines: {node: '>=20.0.0'}
|
||||
|
||||
'@aws-sdk/middleware-host-header@3.972.34':
|
||||
resolution: {integrity: sha512-NmaFH7Wvrh4SICPO7hRW7M8i+9OvjHxC155HHi/C3JwMUM9CSSXdOjtYhw1Q4cZAoKFnrcu9gIfWPE9seZY5Nw==}
|
||||
engines: {node: '>=20.0.0'}
|
||||
|
||||
'@aws-sdk/middleware-location-constraint@3.972.38':
|
||||
resolution: {integrity: sha512-SQO9mbk4ancAKumhBGVrSHH9bj9p1d+Y7Jr5M0NQAYnhoS1mz2zFQhzDACcfENAw8BnRT7WVDYPI1rcFMsdTIQ==}
|
||||
engines: {node: '>=20.0.0'}
|
||||
|
||||
'@aws-sdk/middleware-logger@3.972.33':
|
||||
resolution: {integrity: sha512-wnyJLhO66WsWMCo4TPDFQj1Pa3w4IbA0KDsjQIf1ComTyVPs/HxOmqlXMim5ZPXJQsrWvqRgfCe+vYaa/k39SA==}
|
||||
engines: {node: '>=20.0.0'}
|
||||
@@ -1179,6 +1249,14 @@ packages:
|
||||
resolution: {integrity: sha512-FEdTIfdyyVfMOYfcSEawLZt8ejRBlPoJ51YZOLBdVTBteH5SwLaiDmvAyVwSAhqiI1ooMesdGK0KOwlV+4FDgQ==}
|
||||
engines: {node: '>=20.0.0'}
|
||||
|
||||
'@aws-sdk/middleware-sdk-s3@3.972.72':
|
||||
resolution: {integrity: sha512-lSAoVPvQxX1d8TOM6waKDBQrvvZcm4w6pCldFAsRUffEaXq6lYY0pPyew3KlLu6Xqb74DXI42hGvSsbGBLljlw==}
|
||||
engines: {node: '>=20.0.0'}
|
||||
|
||||
'@aws-sdk/middleware-ssec@3.972.38':
|
||||
resolution: {integrity: sha512-SwLYSlQXpfnCUcazZepBr1DoBy7UUTJWhazX2KWfYAgGR8tsMUiOet4wb6L8CpHEEUaRB8nBHnJ7nlrtAHhb5A==}
|
||||
engines: {node: '>=20.0.0'}
|
||||
|
||||
'@aws-sdk/middleware-user-agent@3.972.63':
|
||||
resolution: {integrity: sha512-ciBdN5V+RjFX2oLXmxqdVbKfADSYIZb4UTFv0eZ2mD9DwARaeSKW9UI9qfX9hJrRjbmYIYtdbMPqFla20x8ZOA==}
|
||||
engines: {node: '>=20.0.0'}
|
||||
@@ -1195,6 +1273,10 @@ packages:
|
||||
resolution: {integrity: sha512-dVZOroI/r3/ENvqNGgjMPul+jjlz9GddfVusgTXlVjfZj5isibOxecLkGQbRPp8XOuX+RAfjXLFgPkD1JS5xrw==}
|
||||
engines: {node: '>=20.0.0'}
|
||||
|
||||
'@aws-sdk/nested-clients@3.997.41':
|
||||
resolution: {integrity: sha512-RDHqPGQWlF6tatA/Tp3rg6oIwtgN9IVderxE+9av2Y93Dfyu+mO1hZ5Bu2jpfZg2rwdNbsssnwM+sLafIczMlQ==}
|
||||
engines: {node: '>=20.0.0'}
|
||||
|
||||
'@aws-sdk/region-config-resolver@3.972.37':
|
||||
resolution: {integrity: sha512-O0mWXYkM1ASgX72KKW3zNG+pEvpmUG5yl6uykgA/p2xkVFi/SjUxwAZBv2ToqKba/TBwDU8tv2wL5Rq5ZTcdMg==}
|
||||
engines: {node: '>=20.0.0'}
|
||||
@@ -1207,6 +1289,10 @@ packages:
|
||||
resolution: {integrity: sha512-QMUytg+FQMGouc8gHS00KoYih3+N6cqmVI/pQGOIo7Nr7OpQaiXjSYOuL+vsPZ1tymY4LAQ8MYcHJmws5LRxng==}
|
||||
engines: {node: '>=20.0.0'}
|
||||
|
||||
'@aws-sdk/signature-v4-multi-region@3.996.43':
|
||||
resolution: {integrity: sha512-lKekx8bLBXSv4O+cslk9Zfnw2XKSkWBs3uWL5QGhH2ZAQfNS7FE0vcSSN2vD/AhxX54ZTywWxR4STThoeOXlBA==}
|
||||
engines: {node: '>=20.0.0'}
|
||||
|
||||
'@aws-sdk/token-providers@3.1048.0':
|
||||
resolution: {integrity: sha512-k0y/GcuesuSfWyUM0WamrGyeZmltRYaPbHO82UDA6mZ/doB+FOHKutikPAtSXMn/hDz970cF+iRuuiYO9VEbAA==}
|
||||
engines: {node: '>=20.0.0'}
|
||||
@@ -1215,6 +1301,10 @@ packages:
|
||||
resolution: {integrity: sha512-4LDW2Qob6LoLFuqYSYZq2AyTE9koSE9+i+n5UZcm10GpmQOK0zRD9L4uYlzItiTKksIWgC/qMFChAi3RvKYtMg==}
|
||||
engines: {node: '>=20.0.0'}
|
||||
|
||||
'@aws-sdk/token-providers@3.1103.0':
|
||||
resolution: {integrity: sha512-N4wy26MNn31ItGVHYHPrEuCIFY4MBBjC+C5v1lJKqIUSA7OZBdhleCY53zCCrXn27hsk7YNOaTuhQu807S4AfQ==}
|
||||
engines: {node: '>=20.0.0'}
|
||||
|
||||
'@aws-sdk/types@3.973.13':
|
||||
resolution: {integrity: sha512-pEHZqRkAlHfnfAU9tK+WpKv/gBNjGJrHMgA3A0iYRGyswBS2t0pfez+lWlwktb3Bqa0ovh7w/QJTFwp3fDxLNg==}
|
||||
engines: {node: '>=20.0.0'}
|
||||
@@ -1246,6 +1336,10 @@ packages:
|
||||
resolution: {integrity: sha512-RdGmS1GLrtaTOLE1ElSluMldNrpk9Emq6uYs8SS8iHlu5xTAmM9rRkM91o48+rIRryBtyO9t+uLYCoMG6jVMVA==}
|
||||
engines: {node: '>=20.0.0'}
|
||||
|
||||
'@aws-sdk/xml-builder@3.972.37':
|
||||
resolution: {integrity: sha512-zKq4HQum8JwDyEuyfuI4bbiAcU0KxP6qy+9PR/IsR92IyE/DaBAikzAS50tjxip4bqIIANpCcG+Yyj6CVhXupg==}
|
||||
engines: {node: '>=20.0.0'}
|
||||
|
||||
'@aws/lambda-invoke-store@0.2.4':
|
||||
resolution: {integrity: sha512-iY8yvjE0y651BixKNPgmv1WrQc+GZ142sb0z4gYnChDDY2YqI4P/jsSopBWrKfAt7LOJAkOXt7rC/hms+WclQQ==}
|
||||
engines: {node: '>=18.0.0'}
|
||||
@@ -2115,6 +2209,7 @@ packages:
|
||||
'@copilotkitnext/shared@1.53.0':
|
||||
resolution: {integrity: sha512-RdhudJUq79RM0zLLzIJDAg//aLc0vW2WocaJmsQzz/awZEbcbo13Ag6jL+oZOOlq++Lu9xWd7SQ83jhoZwCJXg==}
|
||||
engines: {node: '>=18'}
|
||||
deprecated: Use @copilotkit/shared instead.
|
||||
|
||||
'@csstools/color-helpers@6.1.0':
|
||||
resolution: {integrity: sha512-064IFJdjTfUqnjpCVpMOdbr8FLQBhinbZj6yRv2An2E41O/pLEXqfFRWqGq/SxlE5PEUYTlvWsG2r8MswAVvkg==}
|
||||
@@ -5399,10 +5494,18 @@ packages:
|
||||
resolution: {integrity: sha512-i0dk2t5B+CwV/dcJdUHILYkOQF5lof8f44dFCfDWToGCxjT9YQ+CgHqTAvJxzc3+zqQwm2QtVoJ5IqiNar/CnQ==}
|
||||
engines: {node: '>=18.0.0'}
|
||||
|
||||
'@smithy/core@3.31.1':
|
||||
resolution: {integrity: sha512-CyogUINxvi7C7LDsh8Syo6hVJOT9ckz4rG8dRZfTJ8r91HkMY59PnNooaj7WcHyxEkxPfBAmbgztZU+xTo76lg==}
|
||||
engines: {node: '>=18.0.0'}
|
||||
|
||||
'@smithy/credential-provider-imds@4.4.1':
|
||||
resolution: {integrity: sha512-TSAF5NHgxEsllbErYWbK8aLnl5L601NGc5VYJlSPsKnf3YlkhdoBN+geGcaU00oiw2OK3QO5LA3QNXiiWhCidQ==}
|
||||
engines: {node: '>=18.0.0'}
|
||||
|
||||
'@smithy/credential-provider-imds@4.4.16':
|
||||
resolution: {integrity: sha512-QfuLWAkLzptffFW980AFeHZFdqds2B64rpEd3uJ6lgs3xVn9QegGMUgUcj+4d7dRrAsya3r58ZKpku97WcFb4w==}
|
||||
engines: {node: '>=18.0.0'}
|
||||
|
||||
'@smithy/eventstream-codec@4.4.10':
|
||||
resolution: {integrity: sha512-yG1n59zLQMa979xvXlTQ9+FpNmm4RRPR+2rYZo57wk28E27zmKZN4ST9wc2pKkyspLpDal2/84ST72KLJhbP1w==}
|
||||
engines: {node: '>=18.0.0'}
|
||||
@@ -5411,6 +5514,10 @@ packages:
|
||||
resolution: {integrity: sha512-96JrD1q71anokymx9Iblb+zKmNQYNstlV/25A9ZYIJ2A0rp1r7/GZAIm0bDWSmVvz3DpNOCZuabzsiL+w0UHhw==}
|
||||
engines: {node: '>=18.0.0'}
|
||||
|
||||
'@smithy/fetch-http-handler@5.6.13':
|
||||
resolution: {integrity: sha512-4fW86pEUOMbrD5nkbyl/tTvPHHWJFbuB2odl6ps9lWfHoXf9HWh3Q/Smh59qH1g7+c/BSZghX6bbUk4gsiMs8A==}
|
||||
engines: {node: '>=18.0.0'}
|
||||
|
||||
'@smithy/fetch-http-handler@5.6.7':
|
||||
resolution: {integrity: sha512-3zpg8yqqyXzoK2TsRDdkqVOj2RDBFfLXwCczOZ5c7TWB4eiaebfSCsbMjDPYB3PJ9ihV62QaeadZ+wLadZtNGA==}
|
||||
engines: {node: '>=18.0.0'}
|
||||
@@ -5459,6 +5566,10 @@ packages:
|
||||
resolution: {integrity: sha512-emtXvoky671puri18ETf64AFIQUGIEA093F2drXpBgB0OGnBLjcwNR3CA2mYu62IAqNsS56xa5lnTxAgPq7cjw==}
|
||||
engines: {node: '>=18.0.0'}
|
||||
|
||||
'@smithy/node-http-handler@4.9.13':
|
||||
resolution: {integrity: sha512-Nmd/Nl35zfYrd+a6OO2cDJb3GPh9bgTjIUhcM+JFfjpp8/osCgboDV5nCT1I01Pv6R13eSKDKLSoVa5ZB6Zsfw==}
|
||||
engines: {node: '>=18.0.0'}
|
||||
|
||||
'@smithy/node-http-handler@4.9.7':
|
||||
resolution: {integrity: sha512-wCU8HCLjAtAVqxxe0j2xff9LcEPw3yjBbg5IdQDIYFnxnPxbxcSLc7rgex7kqm9L/WYOnJEgaWQlfDkZleozMA==}
|
||||
engines: {node: '>=18.0.0'}
|
||||
@@ -5475,6 +5586,10 @@ packages:
|
||||
resolution: {integrity: sha512-X9rVls3En0z3NtrmguTmpRM0/NqtWUxBjal6fcAkwtsub+gOdLZ6kD+V7xhUgFMGdG14bHbZ7M5QjaRI1+DatQ==}
|
||||
engines: {node: '>=18.0.0'}
|
||||
|
||||
'@smithy/signature-v4@5.6.12':
|
||||
resolution: {integrity: sha512-I6KLtq3H0qqSuV9vLglfi8puHqzygzWHOnI4z/Rdoo+q50vvo18vBRdPAvvEtcaKROz7Zn6qnPa14kRfPH6PcQ==}
|
||||
engines: {node: '>=18.0.0'}
|
||||
|
||||
'@smithy/signature-v4@5.6.6':
|
||||
resolution: {integrity: sha512-efP6DN3UTFrzIsGO42/xcabv8jU7+9nwEdphFUH7yL0k010ERyAWaO41KFQIDLcFZLZ8xzIQr4wplFxNzslSGQ==}
|
||||
engines: {node: '>=18.0.0'}
|
||||
@@ -13807,6 +13922,15 @@ snapshots:
|
||||
'@aws-sdk/types': 3.973.13
|
||||
tslib: 2.8.1
|
||||
|
||||
'@aws-crypto/sha1-browser@5.2.0':
|
||||
dependencies:
|
||||
'@aws-crypto/supports-web-crypto': 5.2.0
|
||||
'@aws-crypto/util': 5.2.0
|
||||
'@aws-sdk/types': 3.974.2
|
||||
'@aws-sdk/util-locate-window': 3.965.8
|
||||
'@smithy/util-utf8': 2.3.0
|
||||
tslib: 2.8.1
|
||||
|
||||
'@aws-crypto/sha256-browser@5.2.0':
|
||||
dependencies:
|
||||
'@aws-crypto/sha256-js': 5.2.0
|
||||
@@ -13820,7 +13944,7 @@ snapshots:
|
||||
'@aws-crypto/sha256-js@5.2.0':
|
||||
dependencies:
|
||||
'@aws-crypto/util': 5.2.0
|
||||
'@aws-sdk/types': 3.973.13
|
||||
'@aws-sdk/types': 3.974.2
|
||||
tslib: 2.8.1
|
||||
|
||||
'@aws-crypto/supports-web-crypto@5.2.0':
|
||||
@@ -13833,6 +13957,14 @@ snapshots:
|
||||
'@smithy/util-utf8': 2.3.0
|
||||
tslib: 2.8.1
|
||||
|
||||
'@aws-sdk/checksums@3.1000.26':
|
||||
dependencies:
|
||||
'@aws-sdk/core': 3.977.6
|
||||
'@aws-sdk/types': 3.974.2
|
||||
'@smithy/core': 3.31.1
|
||||
'@smithy/types': 4.16.1
|
||||
tslib: 2.8.1
|
||||
|
||||
'@aws-sdk/client-bedrock-runtime@3.1048.0':
|
||||
dependencies:
|
||||
'@aws-crypto/sha256-browser': 5.2.0
|
||||
@@ -13892,6 +14024,42 @@ snapshots:
|
||||
'@smithy/util-utf8': 4.4.10
|
||||
tslib: 2.8.1
|
||||
|
||||
'@aws-sdk/client-s3@3.1048.0':
|
||||
dependencies:
|
||||
'@aws-crypto/sha1-browser': 5.2.0
|
||||
'@aws-crypto/sha256-browser': 5.2.0
|
||||
'@aws-crypto/sha256-js': 5.2.0
|
||||
'@aws-sdk/core': 3.977.6
|
||||
'@aws-sdk/credential-provider-node': 3.972.78
|
||||
'@aws-sdk/middleware-bucket-endpoint': 3.972.45
|
||||
'@aws-sdk/middleware-expect-continue': 3.972.41
|
||||
'@aws-sdk/middleware-flexible-checksums': 3.974.51
|
||||
'@aws-sdk/middleware-location-constraint': 3.972.38
|
||||
'@aws-sdk/middleware-sdk-s3': 3.972.72
|
||||
'@aws-sdk/middleware-ssec': 3.972.38
|
||||
'@aws-sdk/signature-v4-multi-region': 3.996.43
|
||||
'@aws-sdk/types': 3.974.2
|
||||
'@smithy/core': 3.31.1
|
||||
'@smithy/fetch-http-handler': 5.6.13
|
||||
'@smithy/node-http-handler': 4.9.13
|
||||
'@smithy/types': 4.16.1
|
||||
tslib: 2.8.1
|
||||
|
||||
'@aws-sdk/client-s3@3.1106.0':
|
||||
dependencies:
|
||||
'@aws-sdk/checksums': 3.1000.26
|
||||
'@aws-sdk/core': 3.977.6
|
||||
'@aws-sdk/credential-provider-node': 3.972.78
|
||||
'@aws-sdk/middleware-sdk-s3': 3.972.72
|
||||
'@aws-sdk/signature-v4-multi-region': 3.996.43
|
||||
'@aws-sdk/types': 3.974.2
|
||||
'@smithy/core': 3.31.1
|
||||
'@smithy/fetch-http-handler': 5.6.13
|
||||
'@smithy/node-http-handler': 4.9.13
|
||||
'@smithy/types': 4.16.1
|
||||
tslib: 2.8.1
|
||||
optional: true
|
||||
|
||||
'@aws-sdk/core@3.974.22':
|
||||
dependencies:
|
||||
'@aws-sdk/types': 3.973.13
|
||||
@@ -13914,6 +14082,17 @@ snapshots:
|
||||
bowser: 2.14.1
|
||||
tslib: 2.8.1
|
||||
|
||||
'@aws-sdk/core@3.977.6':
|
||||
dependencies:
|
||||
'@aws-sdk/types': 3.974.2
|
||||
'@aws-sdk/xml-builder': 3.972.37
|
||||
'@aws/lambda-invoke-store': 0.3.0
|
||||
'@smithy/core': 3.31.1
|
||||
'@smithy/signature-v4': 5.6.12
|
||||
'@smithy/types': 4.16.1
|
||||
bowser: 2.14.1
|
||||
tslib: 2.8.1
|
||||
|
||||
'@aws-sdk/credential-provider-cognito-identity@3.972.58':
|
||||
dependencies:
|
||||
'@aws-sdk/nested-clients': 3.997.33
|
||||
@@ -13930,6 +14109,14 @@ snapshots:
|
||||
'@smithy/types': 4.15.0
|
||||
tslib: 2.8.1
|
||||
|
||||
'@aws-sdk/credential-provider-env@3.972.67':
|
||||
dependencies:
|
||||
'@aws-sdk/core': 3.977.6
|
||||
'@aws-sdk/types': 3.974.2
|
||||
'@smithy/core': 3.31.1
|
||||
'@smithy/types': 4.16.1
|
||||
tslib: 2.8.1
|
||||
|
||||
'@aws-sdk/credential-provider-http@3.972.50':
|
||||
dependencies:
|
||||
'@aws-sdk/core': 3.974.22
|
||||
@@ -13940,6 +14127,16 @@ snapshots:
|
||||
'@smithy/types': 4.15.0
|
||||
tslib: 2.8.1
|
||||
|
||||
'@aws-sdk/credential-provider-http@3.972.69':
|
||||
dependencies:
|
||||
'@aws-sdk/core': 3.977.6
|
||||
'@aws-sdk/types': 3.974.2
|
||||
'@smithy/core': 3.31.1
|
||||
'@smithy/fetch-http-handler': 5.6.13
|
||||
'@smithy/node-http-handler': 4.9.13
|
||||
'@smithy/types': 4.16.1
|
||||
tslib: 2.8.1
|
||||
|
||||
'@aws-sdk/credential-provider-ini@3.972.55':
|
||||
dependencies:
|
||||
'@aws-sdk/core': 3.974.22
|
||||
@@ -13956,6 +14153,22 @@ snapshots:
|
||||
'@smithy/types': 4.15.0
|
||||
tslib: 2.8.1
|
||||
|
||||
'@aws-sdk/credential-provider-ini@3.973.12':
|
||||
dependencies:
|
||||
'@aws-sdk/core': 3.977.6
|
||||
'@aws-sdk/credential-provider-env': 3.972.67
|
||||
'@aws-sdk/credential-provider-http': 3.972.69
|
||||
'@aws-sdk/credential-provider-login': 3.972.74
|
||||
'@aws-sdk/credential-provider-process': 3.972.67
|
||||
'@aws-sdk/credential-provider-sso': 3.973.11
|
||||
'@aws-sdk/credential-provider-web-identity': 3.972.73
|
||||
'@aws-sdk/nested-clients': 3.997.41
|
||||
'@aws-sdk/types': 3.974.2
|
||||
'@smithy/core': 3.31.1
|
||||
'@smithy/credential-provider-imds': 4.4.16
|
||||
'@smithy/types': 4.16.1
|
||||
tslib: 2.8.1
|
||||
|
||||
'@aws-sdk/credential-provider-login@3.972.54':
|
||||
dependencies:
|
||||
'@aws-sdk/core': 3.974.22
|
||||
@@ -13965,6 +14178,15 @@ snapshots:
|
||||
'@smithy/types': 4.15.0
|
||||
tslib: 2.8.1
|
||||
|
||||
'@aws-sdk/credential-provider-login@3.972.74':
|
||||
dependencies:
|
||||
'@aws-sdk/core': 3.977.6
|
||||
'@aws-sdk/nested-clients': 3.997.41
|
||||
'@aws-sdk/types': 3.974.2
|
||||
'@smithy/core': 3.31.1
|
||||
'@smithy/types': 4.16.1
|
||||
tslib: 2.8.1
|
||||
|
||||
'@aws-sdk/credential-provider-node@3.972.57':
|
||||
dependencies:
|
||||
'@aws-sdk/credential-provider-env': 3.972.48
|
||||
@@ -13979,6 +14201,20 @@ snapshots:
|
||||
'@smithy/types': 4.15.0
|
||||
tslib: 2.8.1
|
||||
|
||||
'@aws-sdk/credential-provider-node@3.972.78':
|
||||
dependencies:
|
||||
'@aws-sdk/credential-provider-env': 3.972.67
|
||||
'@aws-sdk/credential-provider-http': 3.972.69
|
||||
'@aws-sdk/credential-provider-ini': 3.973.12
|
||||
'@aws-sdk/credential-provider-process': 3.972.67
|
||||
'@aws-sdk/credential-provider-sso': 3.973.11
|
||||
'@aws-sdk/credential-provider-web-identity': 3.972.73
|
||||
'@aws-sdk/types': 3.974.2
|
||||
'@smithy/core': 3.31.1
|
||||
'@smithy/credential-provider-imds': 4.4.16
|
||||
'@smithy/types': 4.16.1
|
||||
tslib: 2.8.1
|
||||
|
||||
'@aws-sdk/credential-provider-process@3.972.48':
|
||||
dependencies:
|
||||
'@aws-sdk/core': 3.974.22
|
||||
@@ -13987,6 +14223,14 @@ snapshots:
|
||||
'@smithy/types': 4.15.0
|
||||
tslib: 2.8.1
|
||||
|
||||
'@aws-sdk/credential-provider-process@3.972.67':
|
||||
dependencies:
|
||||
'@aws-sdk/core': 3.977.6
|
||||
'@aws-sdk/types': 3.974.2
|
||||
'@smithy/core': 3.31.1
|
||||
'@smithy/types': 4.16.1
|
||||
tslib: 2.8.1
|
||||
|
||||
'@aws-sdk/credential-provider-sso@3.972.54':
|
||||
dependencies:
|
||||
'@aws-sdk/core': 3.974.22
|
||||
@@ -13997,6 +14241,16 @@ snapshots:
|
||||
'@smithy/types': 4.15.0
|
||||
tslib: 2.8.1
|
||||
|
||||
'@aws-sdk/credential-provider-sso@3.973.11':
|
||||
dependencies:
|
||||
'@aws-sdk/core': 3.977.6
|
||||
'@aws-sdk/nested-clients': 3.997.41
|
||||
'@aws-sdk/token-providers': 3.1103.0
|
||||
'@aws-sdk/types': 3.974.2
|
||||
'@smithy/core': 3.31.1
|
||||
'@smithy/types': 4.16.1
|
||||
tslib: 2.8.1
|
||||
|
||||
'@aws-sdk/credential-provider-web-identity@3.972.54':
|
||||
dependencies:
|
||||
'@aws-sdk/core': 3.974.22
|
||||
@@ -14006,6 +14260,15 @@ snapshots:
|
||||
'@smithy/types': 4.15.0
|
||||
tslib: 2.8.1
|
||||
|
||||
'@aws-sdk/credential-provider-web-identity@3.972.73':
|
||||
dependencies:
|
||||
'@aws-sdk/core': 3.977.6
|
||||
'@aws-sdk/nested-clients': 3.997.41
|
||||
'@aws-sdk/types': 3.974.2
|
||||
'@smithy/core': 3.31.1
|
||||
'@smithy/types': 4.16.1
|
||||
tslib: 2.8.1
|
||||
|
||||
'@aws-sdk/credential-providers@3.1045.0':
|
||||
dependencies:
|
||||
'@aws-sdk/client-cognito-identity': 3.1045.0
|
||||
@@ -14036,6 +14299,11 @@ snapshots:
|
||||
'@smithy/types': 4.15.0
|
||||
tslib: 2.8.1
|
||||
|
||||
'@aws-sdk/middleware-bucket-endpoint@3.972.45':
|
||||
dependencies:
|
||||
'@aws-sdk/middleware-sdk-s3': 3.972.72
|
||||
tslib: 2.8.1
|
||||
|
||||
'@aws-sdk/middleware-eventstream@3.972.18':
|
||||
dependencies:
|
||||
'@aws-sdk/types': 3.973.13
|
||||
@@ -14043,11 +14311,26 @@ snapshots:
|
||||
'@smithy/types': 4.15.0
|
||||
tslib: 2.8.1
|
||||
|
||||
'@aws-sdk/middleware-expect-continue@3.972.41':
|
||||
dependencies:
|
||||
'@aws-sdk/middleware-sdk-s3': 3.972.72
|
||||
tslib: 2.8.1
|
||||
|
||||
'@aws-sdk/middleware-flexible-checksums@3.974.51':
|
||||
dependencies:
|
||||
'@aws-sdk/checksums': 3.1000.26
|
||||
tslib: 2.8.1
|
||||
|
||||
'@aws-sdk/middleware-host-header@3.972.34':
|
||||
dependencies:
|
||||
'@aws-sdk/core': 3.975.3
|
||||
tslib: 2.8.1
|
||||
|
||||
'@aws-sdk/middleware-location-constraint@3.972.38':
|
||||
dependencies:
|
||||
'@aws-sdk/middleware-sdk-s3': 3.972.72
|
||||
tslib: 2.8.1
|
||||
|
||||
'@aws-sdk/middleware-logger@3.972.33':
|
||||
dependencies:
|
||||
'@aws-sdk/core': 3.975.3
|
||||
@@ -14058,6 +14341,20 @@ snapshots:
|
||||
'@aws-sdk/core': 3.975.3
|
||||
tslib: 2.8.1
|
||||
|
||||
'@aws-sdk/middleware-sdk-s3@3.972.72':
|
||||
dependencies:
|
||||
'@aws-sdk/core': 3.977.6
|
||||
'@aws-sdk/signature-v4-multi-region': 3.996.43
|
||||
'@aws-sdk/types': 3.974.2
|
||||
'@smithy/core': 3.31.1
|
||||
'@smithy/types': 4.16.1
|
||||
tslib: 2.8.1
|
||||
|
||||
'@aws-sdk/middleware-ssec@3.972.38':
|
||||
dependencies:
|
||||
'@aws-sdk/middleware-sdk-s3': 3.972.72
|
||||
tslib: 2.8.1
|
||||
|
||||
'@aws-sdk/middleware-user-agent@3.972.63':
|
||||
dependencies:
|
||||
'@aws-sdk/core': 3.975.3
|
||||
@@ -14097,6 +14394,17 @@ snapshots:
|
||||
'@smithy/types': 4.16.1
|
||||
tslib: 2.8.1
|
||||
|
||||
'@aws-sdk/nested-clients@3.997.41':
|
||||
dependencies:
|
||||
'@aws-sdk/core': 3.977.6
|
||||
'@aws-sdk/signature-v4-multi-region': 3.996.43
|
||||
'@aws-sdk/types': 3.974.2
|
||||
'@smithy/core': 3.31.1
|
||||
'@smithy/fetch-http-handler': 5.6.13
|
||||
'@smithy/node-http-handler': 4.9.13
|
||||
'@smithy/types': 4.16.1
|
||||
tslib: 2.8.1
|
||||
|
||||
'@aws-sdk/region-config-resolver@3.972.37':
|
||||
dependencies:
|
||||
'@aws-sdk/core': 3.975.3
|
||||
@@ -14116,6 +14424,13 @@ snapshots:
|
||||
'@smithy/types': 4.16.1
|
||||
tslib: 2.8.1
|
||||
|
||||
'@aws-sdk/signature-v4-multi-region@3.996.43':
|
||||
dependencies:
|
||||
'@aws-sdk/types': 3.974.2
|
||||
'@smithy/signature-v4': 5.6.12
|
||||
'@smithy/types': 4.16.1
|
||||
tslib: 2.8.1
|
||||
|
||||
'@aws-sdk/token-providers@3.1048.0':
|
||||
dependencies:
|
||||
'@aws-sdk/core': 3.974.22
|
||||
@@ -14134,6 +14449,15 @@ snapshots:
|
||||
'@smithy/types': 4.15.0
|
||||
tslib: 2.8.1
|
||||
|
||||
'@aws-sdk/token-providers@3.1103.0':
|
||||
dependencies:
|
||||
'@aws-sdk/core': 3.977.6
|
||||
'@aws-sdk/nested-clients': 3.997.41
|
||||
'@aws-sdk/types': 3.974.2
|
||||
'@smithy/core': 3.31.1
|
||||
'@smithy/types': 4.16.1
|
||||
tslib: 2.8.1
|
||||
|
||||
'@aws-sdk/types@3.973.13':
|
||||
dependencies:
|
||||
'@smithy/types': 4.15.0
|
||||
@@ -14175,6 +14499,11 @@ snapshots:
|
||||
'@smithy/types': 4.16.1
|
||||
tslib: 2.8.1
|
||||
|
||||
'@aws-sdk/xml-builder@3.972.37':
|
||||
dependencies:
|
||||
'@smithy/types': 4.16.1
|
||||
tslib: 2.8.1
|
||||
|
||||
'@aws/lambda-invoke-store@0.2.4': {}
|
||||
|
||||
'@aws/lambda-invoke-store@0.3.0': {}
|
||||
@@ -15165,14 +15494,14 @@ snapshots:
|
||||
|
||||
'@chevrotain/types@11.1.2': {}
|
||||
|
||||
'@copilotkit/backend@0.37.0(@aws-crypto/sha256-js@5.2.0)(@aws-sdk/client-bedrock-runtime@3.1048.0)(@aws-sdk/credential-provider-node@3.972.57)(@smithy/util-utf8@2.3.0)(axios@1.13.6)(fast-xml-parser@5.7.3)(ignore@5.3.2)(jsdom@29.1.1(@noble/hashes@1.8.0)(canvas@3.2.3))(lodash@4.17.23)(pg@8.22.0)(playwright@1.58.2)(ws@8.19.0)':
|
||||
'@copilotkit/backend@0.37.0(@aws-crypto/sha256-js@5.2.0)(@aws-sdk/client-bedrock-runtime@3.1048.0)(@aws-sdk/client-s3@3.1106.0)(@aws-sdk/credential-provider-node@3.972.78)(@smithy/util-utf8@2.3.0)(axios@1.13.6)(fast-xml-parser@5.7.3)(ignore@5.3.2)(jsdom@29.1.1(@noble/hashes@1.8.0)(canvas@3.2.3))(lodash@4.17.23)(pg@8.22.0)(playwright@1.58.2)(ws@8.19.0)':
|
||||
dependencies:
|
||||
'@copilotkit/shared': 0.37.0
|
||||
'@google/generative-ai': 0.11.5
|
||||
'@langchain/core': 0.1.63(openai@4.104.0(ws@8.19.0)(zod@4.3.6))
|
||||
'@langchain/openai': 0.0.28(ws@8.19.0)
|
||||
js-tiktoken: 1.0.21
|
||||
langchain: 0.1.37(@aws-crypto/sha256-js@5.2.0)(@aws-sdk/client-bedrock-runtime@3.1048.0)(@aws-sdk/credential-provider-node@3.972.57)(@smithy/util-utf8@2.3.0)(axios@1.13.6)(fast-xml-parser@5.7.3)(ignore@5.3.2)(jsdom@29.1.1(@noble/hashes@1.8.0)(canvas@3.2.3))(lodash@4.17.23)(openai@4.104.0(ws@8.19.0)(zod@4.3.6))(pg@8.22.0)(playwright@1.58.2)(ws@8.19.0)
|
||||
langchain: 0.1.37(@aws-crypto/sha256-js@5.2.0)(@aws-sdk/client-bedrock-runtime@3.1048.0)(@aws-sdk/client-s3@3.1106.0)(@aws-sdk/credential-provider-node@3.972.78)(@smithy/util-utf8@2.3.0)(axios@1.13.6)(fast-xml-parser@5.7.3)(ignore@5.3.2)(jsdom@29.1.1(@noble/hashes@1.8.0)(canvas@3.2.3))(lodash@4.17.23)(openai@4.104.0(ws@8.19.0)(zod@4.3.6))(pg@8.22.0)(playwright@1.58.2)(ws@8.19.0)
|
||||
openai: 4.104.0(ws@8.19.0)(zod@3.25.76)
|
||||
zod: 3.25.76
|
||||
transitivePeerDependencies:
|
||||
@@ -16292,7 +16621,7 @@ snapshots:
|
||||
'@jridgewell/resolve-uri': 3.1.2
|
||||
'@jridgewell/sourcemap-codec': 1.5.5
|
||||
|
||||
'@langchain/community@0.0.57(@aws-crypto/sha256-js@5.2.0)(@aws-sdk/client-bedrock-runtime@3.1048.0)(@aws-sdk/credential-provider-node@3.972.57)(@smithy/util-utf8@2.3.0)(jsdom@29.1.1(@noble/hashes@1.8.0)(canvas@3.2.3))(lodash@4.17.23)(openai@4.104.0(ws@8.19.0)(zod@4.3.6))(pg@8.22.0)(ws@8.19.0)':
|
||||
'@langchain/community@0.0.57(@aws-crypto/sha256-js@5.2.0)(@aws-sdk/client-bedrock-runtime@3.1048.0)(@aws-sdk/credential-provider-node@3.972.78)(@smithy/util-utf8@2.3.0)(jsdom@29.1.1(@noble/hashes@1.8.0)(canvas@3.2.3))(lodash@4.17.23)(openai@4.104.0(ws@8.19.0)(zod@4.3.6))(pg@8.22.0)(ws@8.19.0)':
|
||||
dependencies:
|
||||
'@langchain/core': 0.1.63(openai@4.104.0(ws@8.19.0)(zod@4.3.6))
|
||||
'@langchain/openai': 0.0.28(ws@8.19.0)
|
||||
@@ -16305,7 +16634,7 @@ snapshots:
|
||||
optionalDependencies:
|
||||
'@aws-crypto/sha256-js': 5.2.0
|
||||
'@aws-sdk/client-bedrock-runtime': 3.1048.0
|
||||
'@aws-sdk/credential-provider-node': 3.972.57
|
||||
'@aws-sdk/credential-provider-node': 3.972.78
|
||||
'@smithy/util-utf8': 2.3.0
|
||||
jsdom: 29.1.1(@noble/hashes@1.8.0)(canvas@3.2.3)
|
||||
lodash: 4.17.23
|
||||
@@ -18575,12 +18904,23 @@ snapshots:
|
||||
'@smithy/types': 4.16.1
|
||||
tslib: 2.8.1
|
||||
|
||||
'@smithy/core@3.31.1':
|
||||
dependencies:
|
||||
'@smithy/types': 4.16.1
|
||||
tslib: 2.8.1
|
||||
|
||||
'@smithy/credential-provider-imds@4.4.1':
|
||||
dependencies:
|
||||
'@smithy/core': 3.25.1
|
||||
'@smithy/types': 4.15.0
|
||||
tslib: 2.8.1
|
||||
|
||||
'@smithy/credential-provider-imds@4.4.16':
|
||||
dependencies:
|
||||
'@smithy/core': 3.31.1
|
||||
'@smithy/types': 4.16.1
|
||||
tslib: 2.8.1
|
||||
|
||||
'@smithy/eventstream-codec@4.4.10':
|
||||
dependencies:
|
||||
'@smithy/core': 3.29.5
|
||||
@@ -18592,6 +18932,12 @@ snapshots:
|
||||
'@smithy/types': 4.15.0
|
||||
tslib: 2.8.1
|
||||
|
||||
'@smithy/fetch-http-handler@5.6.13':
|
||||
dependencies:
|
||||
'@smithy/core': 3.31.1
|
||||
'@smithy/types': 4.16.1
|
||||
tslib: 2.8.1
|
||||
|
||||
'@smithy/fetch-http-handler@5.6.7':
|
||||
dependencies:
|
||||
'@smithy/core': 3.29.5
|
||||
@@ -18654,6 +19000,12 @@ snapshots:
|
||||
'@smithy/types': 4.15.0
|
||||
tslib: 2.8.1
|
||||
|
||||
'@smithy/node-http-handler@4.9.13':
|
||||
dependencies:
|
||||
'@smithy/core': 3.31.1
|
||||
'@smithy/types': 4.16.1
|
||||
tslib: 2.8.1
|
||||
|
||||
'@smithy/node-http-handler@4.9.7':
|
||||
dependencies:
|
||||
'@smithy/core': 3.29.5
|
||||
@@ -18676,6 +19028,12 @@ snapshots:
|
||||
'@smithy/types': 4.15.0
|
||||
tslib: 2.8.1
|
||||
|
||||
'@smithy/signature-v4@5.6.12':
|
||||
dependencies:
|
||||
'@smithy/core': 3.31.1
|
||||
'@smithy/types': 4.16.1
|
||||
tslib: 2.8.1
|
||||
|
||||
'@smithy/signature-v4@5.6.6':
|
||||
dependencies:
|
||||
'@smithy/core': 3.29.5
|
||||
@@ -21268,8 +21626,8 @@ snapshots:
|
||||
'@next/eslint-plugin-next': 16.1.2
|
||||
eslint: 9.39.4(jiti@2.6.1)
|
||||
eslint-import-resolver-node: 0.3.9
|
||||
eslint-import-resolver-typescript: 3.10.1(eslint-plugin-import@2.32.0)(eslint@9.39.4(jiti@2.6.1))
|
||||
eslint-plugin-import: 2.32.0(eslint-import-resolver-typescript@3.10.1)(eslint@9.39.4(jiti@2.6.1))
|
||||
eslint-import-resolver-typescript: 3.10.1(eslint-plugin-import@2.32.0(eslint@9.39.4(jiti@2.6.1)))(eslint@9.39.4(jiti@2.6.1))
|
||||
eslint-plugin-import: 2.32.0(eslint-import-resolver-typescript@3.10.1(eslint-plugin-import@2.32.0(eslint@9.39.4(jiti@2.6.1)))(eslint@9.39.4(jiti@2.6.1)))(eslint@9.39.4(jiti@2.6.1))
|
||||
eslint-plugin-jsx-a11y: 6.10.2(eslint@9.39.4(jiti@2.6.1))
|
||||
eslint-plugin-react: 7.37.5(eslint@9.39.4(jiti@2.6.1))
|
||||
eslint-plugin-react-hooks: 7.0.1(eslint@9.39.4(jiti@2.6.1))
|
||||
@@ -21291,7 +21649,7 @@ snapshots:
|
||||
transitivePeerDependencies:
|
||||
- supports-color
|
||||
|
||||
eslint-import-resolver-typescript@3.10.1(eslint-plugin-import@2.32.0)(eslint@9.39.4(jiti@2.6.1)):
|
||||
eslint-import-resolver-typescript@3.10.1(eslint-plugin-import@2.32.0(eslint@9.39.4(jiti@2.6.1)))(eslint@9.39.4(jiti@2.6.1)):
|
||||
dependencies:
|
||||
'@nolyfill/is-core-module': 1.0.39
|
||||
debug: 4.4.3(supports-color@8.1.1)
|
||||
@@ -21302,21 +21660,21 @@ snapshots:
|
||||
tinyglobby: 0.2.15
|
||||
unrs-resolver: 1.11.1
|
||||
optionalDependencies:
|
||||
eslint-plugin-import: 2.32.0(eslint-import-resolver-typescript@3.10.1)(eslint@9.39.4(jiti@2.6.1))
|
||||
eslint-plugin-import: 2.32.0(eslint-import-resolver-typescript@3.10.1(eslint-plugin-import@2.32.0(eslint@9.39.4(jiti@2.6.1)))(eslint@9.39.4(jiti@2.6.1)))(eslint@9.39.4(jiti@2.6.1))
|
||||
transitivePeerDependencies:
|
||||
- supports-color
|
||||
|
||||
eslint-module-utils@2.12.1(eslint-import-resolver-node@0.3.9)(eslint-import-resolver-typescript@3.10.1)(eslint@9.39.4(jiti@2.6.1)):
|
||||
eslint-module-utils@2.12.1(eslint-import-resolver-node@0.3.9)(eslint-import-resolver-typescript@3.10.1(eslint-plugin-import@2.32.0(eslint@9.39.4(jiti@2.6.1)))(eslint@9.39.4(jiti@2.6.1)))(eslint@9.39.4(jiti@2.6.1)):
|
||||
dependencies:
|
||||
debug: 3.2.7
|
||||
optionalDependencies:
|
||||
eslint: 9.39.4(jiti@2.6.1)
|
||||
eslint-import-resolver-node: 0.3.9
|
||||
eslint-import-resolver-typescript: 3.10.1(eslint-plugin-import@2.32.0)(eslint@9.39.4(jiti@2.6.1))
|
||||
eslint-import-resolver-typescript: 3.10.1(eslint-plugin-import@2.32.0(eslint@9.39.4(jiti@2.6.1)))(eslint@9.39.4(jiti@2.6.1))
|
||||
transitivePeerDependencies:
|
||||
- supports-color
|
||||
|
||||
eslint-plugin-import@2.32.0(eslint-import-resolver-typescript@3.10.1)(eslint@9.39.4(jiti@2.6.1)):
|
||||
eslint-plugin-import@2.32.0(eslint-import-resolver-typescript@3.10.1(eslint-plugin-import@2.32.0(eslint@9.39.4(jiti@2.6.1)))(eslint@9.39.4(jiti@2.6.1)))(eslint@9.39.4(jiti@2.6.1)):
|
||||
dependencies:
|
||||
'@rtsao/scc': 1.1.0
|
||||
array-includes: 3.1.9
|
||||
@@ -21327,7 +21685,7 @@ snapshots:
|
||||
doctrine: 2.1.0
|
||||
eslint: 9.39.4(jiti@2.6.1)
|
||||
eslint-import-resolver-node: 0.3.9
|
||||
eslint-module-utils: 2.12.1(eslint-import-resolver-node@0.3.9)(eslint-import-resolver-typescript@3.10.1)(eslint@9.39.4(jiti@2.6.1))
|
||||
eslint-module-utils: 2.12.1(eslint-import-resolver-node@0.3.9)(eslint-import-resolver-typescript@3.10.1(eslint-plugin-import@2.32.0(eslint@9.39.4(jiti@2.6.1)))(eslint@9.39.4(jiti@2.6.1)))(eslint@9.39.4(jiti@2.6.1))
|
||||
hasown: 2.0.2
|
||||
is-core-module: 2.16.1
|
||||
is-glob: 4.0.3
|
||||
@@ -23642,10 +24000,10 @@ snapshots:
|
||||
|
||||
kleur@4.1.5: {}
|
||||
|
||||
langchain@0.1.37(@aws-crypto/sha256-js@5.2.0)(@aws-sdk/client-bedrock-runtime@3.1048.0)(@aws-sdk/credential-provider-node@3.972.57)(@smithy/util-utf8@2.3.0)(axios@1.13.6)(fast-xml-parser@5.7.3)(ignore@5.3.2)(jsdom@29.1.1(@noble/hashes@1.8.0)(canvas@3.2.3))(lodash@4.17.23)(openai@4.104.0(ws@8.19.0)(zod@4.3.6))(pg@8.22.0)(playwright@1.58.2)(ws@8.19.0):
|
||||
langchain@0.1.37(@aws-crypto/sha256-js@5.2.0)(@aws-sdk/client-bedrock-runtime@3.1048.0)(@aws-sdk/client-s3@3.1106.0)(@aws-sdk/credential-provider-node@3.972.78)(@smithy/util-utf8@2.3.0)(axios@1.13.6)(fast-xml-parser@5.7.3)(ignore@5.3.2)(jsdom@29.1.1(@noble/hashes@1.8.0)(canvas@3.2.3))(lodash@4.17.23)(openai@4.104.0(ws@8.19.0)(zod@4.3.6))(pg@8.22.0)(playwright@1.58.2)(ws@8.19.0):
|
||||
dependencies:
|
||||
'@anthropic-ai/sdk': 0.9.1
|
||||
'@langchain/community': 0.0.57(@aws-crypto/sha256-js@5.2.0)(@aws-sdk/client-bedrock-runtime@3.1048.0)(@aws-sdk/credential-provider-node@3.972.57)(@smithy/util-utf8@2.3.0)(jsdom@29.1.1(@noble/hashes@1.8.0)(canvas@3.2.3))(lodash@4.17.23)(openai@4.104.0(ws@8.19.0)(zod@4.3.6))(pg@8.22.0)(ws@8.19.0)
|
||||
'@langchain/community': 0.0.57(@aws-crypto/sha256-js@5.2.0)(@aws-sdk/client-bedrock-runtime@3.1048.0)(@aws-sdk/credential-provider-node@3.972.78)(@smithy/util-utf8@2.3.0)(jsdom@29.1.1(@noble/hashes@1.8.0)(canvas@3.2.3))(lodash@4.17.23)(openai@4.104.0(ws@8.19.0)(zod@4.3.6))(pg@8.22.0)(ws@8.19.0)
|
||||
'@langchain/core': 0.1.63(openai@4.104.0(ws@8.19.0)(zod@4.3.6))
|
||||
'@langchain/openai': 0.0.28(ws@8.19.0)
|
||||
'@langchain/textsplitters': 0.0.3(openai@4.104.0(ws@8.19.0)(zod@4.3.6))
|
||||
@@ -23663,7 +24021,8 @@ snapshots:
|
||||
zod: 3.25.76
|
||||
zod-to-json-schema: 3.25.1(zod@3.25.76)
|
||||
optionalDependencies:
|
||||
'@aws-sdk/credential-provider-node': 3.972.57
|
||||
'@aws-sdk/client-s3': 3.1106.0
|
||||
'@aws-sdk/credential-provider-node': 3.972.78
|
||||
axios: 1.13.6
|
||||
fast-xml-parser: 5.7.3
|
||||
ignore: 5.3.2
|
||||
|
||||
Reference in New Issue
Block a user