mirror of
https://github.com/THU-MAIC/OpenMAIC.git
synced 2026-10-17 05:29:46 +08:00
wyucand杨慎
6825363de1
* docs(storage): specify the asset registry HTTP contract (#1007) Adds the contract document and the server-side store, byte-layer, and handler types for the asset layer, the only layer with no server backend. Runtime behaviour is unchanged: the new modules are deliberately unreferenced until the backends land. Reclamation moves offline, and that decision carries the design. `remove` deletes a registry entry and nothing else -- it never reads, counts, or deletes bytes. A separate collector takes byte rows that have had no references for a grace period. Three things follow. The hardest race in the design disappears, because no request can delete bytes another request is adopting, so no request path needs a lock on the byte row and the write path never has to reconcile taking one with never branching on whether the bytes already existed. A side channel closes, because a `remove` that only deletes a row costs the same whether or not another principal holds the same bytes. And nothing is given up: under global deduplication `remove` could never promise the bytes were destroyed, so deleting them synchronously bought far less than it appeared to. The cost is that storage grows until the collector runs, which a deployment must configure rather than inherit. Offline reclamation is also what makes the byte layer pluggable, so this keeps #1007's pluggable-backend model rather than narrowing it. With collection out of every request path, the requirement on a byte layer is an ordering rule rather than an atomicity one: bytes are written before the row that references them, deleted after the last row referencing them, and the reference count is serialized on the blob row that every implementation keeps regardless of where bytes sit. Two implementations are planned -- bytes in a column of the transactional store, and an object store keyed by content hash -- which is why the interface exists; an interface with one implementation is not a seam, and this package has removed such a seam before. The browser backend keeps no such seam, because it reclaims inline and its bytes therefore cannot leave its transaction; four comments that stated this as a storage difference now state it as a reclamation one. Metadata crosses the wire as a multipart part. It is open-ended and callers already fill it with generated-narration text and image prompts, so a query parameter would put unbounded user content into the request target and into every intermediary's logs. That also motivates a rule the other layers do not need: metadata must never appear in a URL, a response header, a log line, or an error message. Metadata gets a value domain, and the document is explicit that this is a capability reduction -- the browser backend's structured clone carries Date, Map, Set, and cycles faithfully, so a deployment moving to a server backend must audit its metadata rather than expect a quietly lossy path to tighten. Resolution yields a client-minted object URL, superseding the resolution that a server deployment resolves to a self-hosted proxy path. A media element sends only ambient credentials, so a proxy path works for cookie-session deployments and silently fails for header-authenticated ones; closing that gap means minting URL tokens, a second credential form inside a storage library. Fetching bytes over the client's own authenticated request needs none and removes a class of hash-disclosure surface. The client obligations that make it preserve the browser backend's semantics are specified, and split into the two the shared suite already pins and the four an HTTP backend must pin itself. The cost, no range requests and no progressive playback, is stated. Channels that could carry a principal or a content hash are closed structurally rather than by enumeration: the route table admits no segment that could carry either, and no route takes a query string at all. Headers are closed by not reading them -- the contract assigns no meaning to any header beyond content type and length, and a handler must not derive identity from one outside its authenticate hook. Rejecting header names by pattern was tried and removed: no pattern broad enough to catch X-Remote-User spares User-Agent, and the forwarded-identity headers such a rule would reject are exactly what an authenticating reverse proxy injects for that hook to read. Ids that cannot be a path segment are resolved by the client as misses rather than raised as errors, a deliberate divergence from the KV contract. An unknown id is a miss by this layer's id-domain rule and the shared suite pins the empty string as one, so a client that threw would fail that suite and reintroduce the shape oracle the rule prevents. Also specified: response values may not derive from the shared byte row (Last-Modified off that row is an existence oracle), a collision- resistant digest is required because writes are unconditional and an object layer keys on that digest, byte responses carry the revision and refuse Range, HEAD errors carry the code in a header since they have no body, quota exhaustion has a status and a code, and relabelling a non-renderable type is not refusal -- resolve still returns a URL. * feat(storage): add pluggable server asset store (#1007) * fix(storage): write asset bytes only after claiming the blob row (#1007) The write path wrote bytes twice: once before the registry transaction and once inside it, after the upsert that claims the blob row. Only the second one carries any safety, and the first is what a large-media deployment pays for on every upload. The ordering that matters is claim, write, reference. Claiming the blob row takes its lock, so a collector already holding that lock finishes before the write proceeds; writing bytes before the claim instead lets the collector delete them while the claim waits, leaving a fresh entry that points at nothing. Writing them before the entry keeps every surviving entry backed by bytes that were actually stored. Dropping the first write preserves both ends of that order and removes a crash window that produced orphans for no reason. For a byte layer inside the transactional store this also means a failed write now leaves nothing at all, rather than an orphan the collector had to sweep later. The four tests this changed were each pinned to the removed write: - the statement-sequence test hardcoded the old four-statement shape; it now pins claim, write, entry, and still asserts the two existence paths are identical, which is the property that matters; - the quota test asserted two writes per accepted put; - the crash-window test asserted an orphan that a transactional byte layer no longer produces. It is now two tests: a transactional layer leaves nothing, and a non-transactional one strands an object with no blob row -- which the collector deliberately cannot see, since recovering it is deployment housekeeping rather than reference counting. It also no longer depends on state left behind by earlier tests in the file; - the collector-race test drove its interleaving off the removed write. It was choreography rather than concurrency in any case: PGlite is single-connection, and the test serialized transactions, so no row lock was ever contended. It is replaced by the two invariants that actually make the race safe -- the byte write is unconditional, and bytes the collector has already removed are re-stored by an adopting put. The contended-lock case belongs to the real-PostgreSQL suite, where it can be tested rather than mimed. Reverting the unconditional write fails two of them, so they pin it. The contract said a failed write leaves no partially written bytes while its byte-layer section permits exactly that orphan; the two are reconciled, and the ordering rule now states the claim step and the unconditional write it depends on, along with the cost that follows -- the byte write happens with the registry transaction open, so an object store holds a row lock across a network upload. * test(storage): fix the real-PostgreSQL asset tests against the new write order Both were written against the removed pre-transaction byte write, and both failed once it went away -- one of them by hanging, which also timed out the next test's TRUNCATE. The lock-contention test waited for the adopting put to start writing bytes before releasing the collector. Bytes are now written after the upsert that claims the blob row, and the collector holds that lock, so the write could never start: a circular wait. It now waits for a backend to actually appear in pg_stat_activity blocked on a lock, which is the condition it meant to wait for, observed rather than signalled. Its final assertion carries real weight now -- had the bytes been written before the claim, the collector would have deleted them and the adopting entry would resolve to nothing. The orphan test asserted that a failed registry transaction strands bytes. With this byte layer inside the registry's transaction it strands nothing, so it now asserts that instead. Object storage does strand an object, and that case is covered where it belongs. All of this ran against PostgreSQL 16: 781 tests pass with 3 skipped, and those 3 pass against a real S3-compatible server. Nothing in the suite is now unverified for lack of infrastructure. * feat(storage): add asset HTTP backend (#1007) * docs(storage): separate trusting a header from reading one (#1007) The header rule said a server reads exactly Content-Type and Content-Length, while the size rules require reading Content-Encoding in order to reject it -- the two cannot both be literal, and the implementation had to pick one. What the rule means is narrower: no request header may be trusted to say who is asking, outside the authenticate hook. Reading a header to decide how to frame or refuse a request is a different thing, and the transport headers are read and acted on. * chore(storage): bump to 0.2.3 for the asset server backend (#1007) Additive: new asset registry, byte layers, collector, HTTP handler and client, and their entry points. Nothing existing is removed or changed incompatibly, so this is a patch under the pre-1.0 rule. * fix(storage): close four defects found reviewing the asset backend (#1007) **Concurrent writes could exceed a principal's logical quota.** The check ran on the pool before the write transaction, so two concurrent puts both read the old total and both passed; enough concurrency amplified it arbitrarily. It now runs inside the write transaction behind a transaction-scoped advisory lock on the principal, taken only when a quota is configured -- a branch on deployment configuration, never on data. Pinned by a real-PostgreSQL test: four concurrent six-byte writes against a ten-byte quota, of which exactly one may be accepted. Removing the lock accepts three. **A part name smuggled inside a quoted filename was read as a real part name.** `Content-Disposition` was scanned with a regex that does not understand quoted strings, so `filename="x; name=meta; y"` parsed as a `name` parameter here while an RFC-aware intermediary sees an unnamed part. That is exactly the parser differential the contract requires be rejected. Replaced with a tokenizer that honours quoted strings and escapes and requires exactly one real `name`. **Any S3 404 was reported as an absent object.** `NoSuchBucket`, a misdirected endpoint, and a revoked access point all answer 404 while the bytes still exist, so a storage outage surfaced as `404 ASSET_NOT_FOUND` and a caller clearing its reference on that would turn it into real data loss -- the same failure the contract spells out for `401`. Only key-absent codes map to a miss now; everything else propagates to `500 INTERNAL_ERROR`. **Exceeding `maxParts` answered `400` rather than `413`.** It is one of the declared multipart resource limits and now answers like the other three. Verified against PostgreSQL 16 and a real S3-compatible server: 840 tests, none skipped. * fix(storage): repair two defects the previous fix round introduced (#1007) Both were created by the fixes themselves rather than surviving them, which is the failure mode a second review round exists to catch. **An over-quota replace answered 500.** Moving the quota check inside the write transaction put it under a catch that re-threw only AssetNotFoundError, so AssetQuotaExceededError was collapsed into a generic registry failure and the handler mapped it to INTERNAL_ERROR -- losing the status and code the contract gives that condition. Both typed errors now survive the catch. **The hand-written disposition tokenizer accepted malformed input and trusted it.** The quoted-value loop never checked that it found a closing quote, so `name="meta` ran to the end of the header and returned `meta`; `name="meta"junk` returned `meta` as well; and an extended `name*` form competing with a plain one was silently ignored. Each recreates the parser differential the tokenizer was written to remove. A quoted value now requires its closing quote, rejects a dangling escape, and permits only whitespace and a separator after it, and an extended name form is refused rather than resolved. Also widened the quota lock key from `hashtext` to `hashtextextended`. The 32-bit form collides -- two unrelated principals sharing a key would block each other for the whole transaction, which spans a byte write that may be a network upload. Regression tests cover all three: an over-quota replace raises the quota error and leaves the original bytes, and each of the three malformed dispositions is refused. Verified against PostgreSQL 16 and a real S3-compatible server: 842 tests, none skipped. * fix(storage): narrow the multipart disposition surface and the error boundary (#1007) Three review rounds have each found a defect in the part-disposition parser: a regex that misread quoted strings, then a tokenizer that accepted an unterminated quote, then RFC 2231 continuation forms (`name*0*=UTF-8''bytes`) and malformed empty parameter slots. Patching it a fourth time would have been the wrong move. This contract needs exactly one parameter, so it now accepts exactly one: a part disposition is `form-data` plus a single `name` whose value is `meta` or `bytes`, and everything else is a validation failure. That removes the continuation forms, the encoded forms, the empty slots, and `filename` -- the parameter every one of these attacks travelled in -- without having to model RFC 2231 at all. That rule turned out to bind our own client too: it emitted a fixed `filename="asset"`. Nothing reads it, and the contract already forbids deriving a response disposition filename from caller data, so the client no longer sends one. Separately, `instanceof` was being used as provenance. The byte layer is pluggable, so a byte store raising a same-named `AssetQuotaExceededError` or `AssetNotFoundError` inside the transaction was re-thrown as a logical registry outcome, carrying the byte layer's own message to a direct caller. The registry's checks now raise module-private sentinels, and only those are converted -- at the boundary, into fresh public errors with this package's fixed messages. Anything else, whatever its name, collapses to the generic registry failure. Also pinned the advisory-lock key width. Reverting `hashtextextended` to the 32-bit `hashtext` left every test green, so the emitted lock statement is now asserted. A SQL-text assertion is deliberate here: the behavioural difference is a collision between two particular keys under a database-internal hash, which is not a stable thing to assert against. Verified against PostgreSQL 16 and a real S3-compatible server: 845 tests, none skipped. * fix(storage): define multipart disposition grammar (#1007) * fix(storage): delegate multipart parsing to Fetch (#1007) * fix(storage): require both multipart parts to be files (#1007) Delegating the framing to the platform parser cost two rules for a metadata part sent without a filename: such a part comes back as a string with its headers discarded, so its `application/json` type could no longer be checked, and its bytes had already been replacement-decoded -- `{"x":"<0xFF>"}" was stored as `{"x":"\uFFFD"}` instead of refused. The bytes part already had to be a file for binary safety. Metadata is now symmetric: the client sends a fixed filename on both, the server requires both to be files, and with the part preserved the media type is checked and the bytes are decoded fatally before parsing. Neither filename is read anywhere. The limits are also described honestly now. `maxParts`, `maxMetaBytes` and `maxAssetBytes` are validated after the parser has materialized every part, so they bound what is accepted rather than what is parsed, and `maxRequestBytes` is what caps memory. The contract said `maxParts` bounded parser work; it does not. Restoring that would mean adopting a streaming parser with its own grammar, which is the disagreement with intermediaries that delegating to the platform exists to remove -- so the trade is stated rather than reversed. Also corrected the media-type wording: the two branches are metadata present versus absent, and retention of an untyped replacement's prior type is a browser-backend behaviour the HTTP path cannot reproduce, because a conforming parser supplies a default type for a file part whose header omits one. Verified against PostgreSQL 16 and a real S3-compatible server: 853 tests, none skipped. * docs(storage): the asset server backend ships in this branch (#1007) The contract still described the server backend as not yet shipped, which this branch is what changes. The conformance server remains test-only. * fix(storage): make asset HEAD bytes-free and pin reads (#1007) * fix(storage): harden asset write boundaries (#1007) * chore(storage): bump to 0.2.4 for the asset server backend (#1007) main merged in a 0.2.3 of its own, so this branch's bump no longer increased the version. Additive relative to the merged base -- new asset registry, byte layers, collector, HTTP handler and client, with nothing removed or narrowed -- so a patch bump under the pre-1.0 rule. --------- Co-authored-by: 杨慎 <117187635+cosarah@users.noreply.github.com>