3 Commits

Include renames

Author	SHA1	Message	Date
wyuc	335180ec9c	feat(generation): scaffold @openmaic/generation with pipeline types and packaged prompt assets (#1063) ... Part A of #1057. Package skeleton (ESM, tsc, dsl/storage pattern), pipeline type contracts, prompt loader with package-relative asset resolution, 13 templates + 7 snippets byte-identical with golden tests, allowlist boundary lint, full publish/CI wiring, installed-tarball smoke coverage.	2026-08-05 22:22:54 -04:00
wyuc	90f5f3942b	fix(release): dedupe the published dsl, close two release-path blind spots (#1019) ... * fix(release): dedupe the published dsl, close two release-path blind spots Three independent defects on the release path. 1. `@openmaic/storage`, `@openmaic/renderer` and `@openmaic/importer` declared `"@openmaic/dsl": "workspace:*"`, which pnpm publishes as an EXACT pin. The tarballs on the registry today say storage@0.1.1 needs dsl 0.5.0, renderer@0.0.3 needs dsl 0.4.0 and importer@0.1.1 needs dsl 0.4.0, so installing storage together with renderer yields two copies of the dsl. The dsl carries the schema, the validators and the version constants, so a document produced against one copy can be validated by the other's schema revision. `workspace:^` publishes as `^0.5.0`. Under 0.x that admits 0.5.x, so a dsl patch reaches dependents without republishing them while a dsl minor still requires a deliberate dependent release. All three packages take a patch version increase, both because check-package-version-bumps.mjs requires one and because the corrected range only reaches consumers through a new release. The tarball smoke test now asserts the packed range, so an exact pin cannot come back unnoticed. 2. The publish workflow's build-integrity guard ran `git diff --quiet -- packages/@openmaic`, which compares the worktree to the INDEX. The steps before it run third-party package code, which needs no credential to rewrite a tracked file and `git add` it, after which the guard reported clean and the rewritten bytes were packed and published. Both copies of the guard now compare against HEAD, the commit actually being released. 3. `@openmaic/storage`'s PostgreSQL contract suites refuse to skip when STORAGE_PG_CONTRACT_REQUIRED=1, but that refusal is a `throw` inside the test modules, so it never fires if vitest stops collecting them. Collection is decided by `packages/@openmaic/storage/vitest.config.ts`, which is on the ignore list of publishable inputs, so excluding `*.pg.test.ts` there turns both PostgreSQL backends off with a green run and no version bump. scripts/assert-pg-contract-suites.mjs reads vitest's json results and requires both suite files to be present, passed and non-empty, keyed on the file names rather than on a count. It runs in the publish workflow's validate job and in the storage PostgreSQL contract workflow. Merging this publishes @openmaic/storage@0.1.2, @openmaic/renderer@0.0.4 and @openmaic/importer@0.1.2. That republish is the fix for (1), not a side effect of it: the corrected dependency range exists only in a tarball that has not been cut yet. * fix(release): audit the contract database, pin the format version to a minor Follow-up on cross-vendor review of the three release-path fixes. 1. The PostgreSQL assertion proved far less than it claimed. Vitest's `assertionResults` are test CASES, not `expect()` calls, so one `test('x', () => {})` per required filename satisfied it — and the whole surface is reachable without a version bump, because `test/` is on the same ignore list as `vitest.config.ts`. A `vi.mock('pg', ...)` in the already-wired `test/setup.ts`, or a stubbed suite body, made it print "ran against a real database" having checked nothing of the sort. The script now runs in two phases. Phase 1 keeps the file/status checks. Phase 2 connects to PG_CONTRACT_URL from outside the vitest process and requires all five tables the two backends own to exist and to show inserts in pg_stat_user_tables. Nothing inside `test/` can forge that. Cumulative insert counters rather than surviving rows, because the suites clean up after themselves. The success message now states only what was checked. 2. Validation packed with lifecycle scripts while publishing runs `--ignore-scripts`, so committing `workspace:*` plus a `prepack` that rewrites it to `workspace:^` passed the tarball assertion and then published the exact pin. Both the smoke test and the workflow's dry-run pack now pass `--config.ignore-scripts=true`, so validation and publication have identical lifecycle semantics. 3. `workspace:^` unpins the serialized-format version that the exact pin was silently holding. DSL_VERSION and RUNTIME_DSL_VERSION are decoupled from the npm version by design, and storage compares them by value across the package boundary. Under `^0.5.0` a dsl PATCH could therefore hand a new format to an already-published storage: one install resolves it and stamps the new version into document_stages, another install pinned to the older patch reads that row and hard- fails. Two installs of one published storage version, data- incompatible. check-package-version-bumps.mjs now requires a change to either constant to carry at least a MINOR increase of the dsl package version, since a caret does not cross a 0.x minor. The rule is documented next to both constants. That doc edit is itself a publishable change, so dsl takes a patch bump to 0.5.1. 4. The range and build-integrity guards ran only in the publish workflow, so a pull request restoring `workspace:*` or leaving a stale generated file failed at release time, after a version number had been spent. ci.yml gains a cheap source-level range check (scripts/check-internal-dependency-ranges.mjs) and the `git diff --quiet HEAD` integrity check after its install step. The tarball assertion stays where it is as the release-time proof of actual published behaviour. Merging this now publishes @openmaic/dsl@0.5.1 as well as @openmaic/storage@0.1.2, @openmaic/renderer@0.0.4 and @openmaic/importer@0.1.2. * fix(release): fail the gates closed, and state what they actually prove Confirmation round on the release-path fixes. 1. The format-version rule failed open if version.ts moved. It hard-coded one source path and returned successfully when either revision lacked it, so renaming the file, changing DSL_VERSION and bumping dsl by a patch passed both checks. It now reports an error whenever the file is absent at either revision while dsl has any publishable change. The constant regex is anchored to the start of a line, so a commented-out declaration cannot be read as the value, and a duplicate declaration is an error rather than a coin flip. 2. "At least a MINOR" was wrong past 1.0.0: `^1.0.0` admits minors, so a format change shipped as a minor would float into published dependents exactly as the rule exists to prevent. The rule is now stated as what it needs to be — an increase the dependents' caret range will NOT admit — computed from the pre-change version: a minor while dsl is 0.x, a major once it reaches 1.0.0. The error message and the docs in version.ts say the same thing. 3. Internal declarations outside `dependencies` bypassed both range checks. Keeping `dependencies` at `workspace:^` while adding an exact `peerDependencies` or `optionalDependencies` entry published a second, tighter constraint and passed. An owned package must now appear exactly once, in `dependencies`; entries in the peer and optional fields are rejected outright, in the source check and in the packed-manifest assertion. The global count test is replaced by naming the three dependents that must be present. 4. The format rule compared against the base tip, so an un-rebased branch produced an inverted message. It now uses the merge base, matching the `base...HEAD` form the rest of diff mode uses. Deliberately not applied to the version comparison, where the tip is the correct reference. 5. Phase 2 had no baseline, so cumulative counters from an earlier run satisfied it forever against a non-ephemeral database. It now captures the counters before the vitest step and requires a positive delta. Both workflows gained the capture step. 6. Phase 2 overclaimed. It proves rows were inserted into those five tables during the run, not that the built PgDocumentStore and PgRuntimeStore inserted them — `test/` is on the publishable-input ignore list, so test code could write directly. The success message now says exactly that, and the script documents the threat model: this catches accidental silencing, which is the defect it was written for; it is not a defence against someone who can merge changes to `test/`, who can alter production code just as easily. Also: scripts/openmaic-packages.mjs is now the single source for the owned package list, consumed by all three checking scripts, and it cross-checks itself against both the packages directory and publish-packages.yml so a package added in one place and forgotten in another fails rather than going quietly exempt. The workflow's own YAML lists cannot import it, which is why they are cross-checked instead. * fix(release): let the format-version source move without opening the gate Failing closed on a missing version.ts is right, but on its own it made a rename impossible to merge: the new path does not exist at the base revision either, so the check refused both before and after the move. DSL_VERSION_SOURCES is now an ordered candidate list, resolved per revision against the first path that exists there. A rename lands by prepending the new path in the same change, and the comparison still happens across the move. Absent everywhere at either revision, while dsl has any publishable change, remains an error. * fix(release): report package-list drift instead of crashing on it Every check in check-internal-dependency-ranges.mjs reads the shared package list, so a list naming a package that does not exist threw an ENOENT out of readManifest before any finding was printed. It failed closed, but unreadably. The list is now validated first and fatally, with its own headline. * fix(release): anchor the caret to the base branch, and check the gates' own inputs Final confirmation round. 1. The caret escape was computed from the merge base, which is the wrong reference for it. With merge base dsl 0.5.1, an ordinary minor landing on main taking the tip to 0.6.0, and HEAD at 0.6.1 with a changed DSL_VERSION, the rule computed the escape from 0.5.1, got 0.6.0, and accepted 0.6.1 — but dependents released against 0.6.0 publish `^0.6.0`, which admits 0.6.1, so the new format reached them anyway. 0.6.1 is also the minimum the ordinary version check allows, making that the default outcome rather than an unlucky one. The escape is now computed from the HIGHEST dsl version reachable on the base branch. Whether the format moved still comes from the merge base, because that is a question about the branch. The comment on mergeBaseWithHead now records that it is a no-op in both current CI invocations (checkout uses the merge ref on pull_request, and the before-SHA is an ancestor on push) rather than claiming a case that is not exercised there. 2. The workflow cross-check matched raw text, so a commented-out trigger path still satisfied it while silently disabling that package's release. Full-line comments are now stripped first; the `on.push.paths` list and every `for pkg in ...` loop are compared as exact sets, so an unexpected entry fails as well as a missing one; and `--filter` usage is checked more loosely, because individual steps legitimately filter subsets. The limits of a textual check are stated where it lives. 3. `locate()` returned the first candidate format-version source that existed, so during a half-finished rename the gate could compare a file that is no longer the exported one. It now resolves every candidate per revision and fails unless exactly one declares the constants. 4. INTERNAL_DEPENDENTS was checked in one direction only, so deleting an entry exempted that package from the source check and from the packed assertion that iterates the same map. It is now cross-checked both ways, an owned package in another owned package's devDependencies is rejected outright, and multiple owned dependencies no longer collapse to whichever was declared last. 5. assertPackageListIsComplete() now also runs in release mode, which is the gate that decides what gets published; and the PostgreSQL script consumes flag values properly instead of treating the first non-flag argument as the results path. Also documents, in openmaic-packages.mjs, the threat model these gates share: they catch mistakes, and anyone who can merge edits to scripts/ can edit production code directly, so deliberate subversion is out of scope and cannot be closed by adding further checks.	2026-08-02 22:44:34 -04:00
wyucandClaude Opus 5	cc95851dc5	chore(packages): publish storage and enforce version bumps (#998) ... * chore(packages): publish storage and enforce version bumps * fix(packages): cover all publish inputs and tarballs * fix(packages): scope version guard to release inputs * fix(packages): fail closed on new package inputs * docs(packages): clarify version guard boundaries * fix(packages): harden release guard edge cases * fix(packages): align smoke peers and git inputs * docs(packages): make release smoke policy explicit * fix(packages): anchor version guard at repository root * fix(packages): report version guard setup errors * fix(packages): harden release validation * fix(ci): validate complete main pushes * test(storage): pin the exported PostgreSQL schemas DOCUMENT_PG_SCHEMA and RUNTIME_PG_SCHEMA are public API. A deployment that provisions these tables with its own migration tooling has to reproduce the DDL exactly for ensureDocumentSchema() / ensureSchema() to stay the intended no-op against an already-provisioned database. Nothing guarded that today: every statement is CREATE ... IF NOT EXISTS, so PostgreSQL silently accepts whatever table already exists under the name. A column type, a nullability, an index or a FK action can drift apart from a downstream migration without an error, and the first symptom is a store query failing in production or succeeding against the wrong types. Pin both constants verbatim, and assert every statement stays IF NOT EXISTS guarded so the ensure functions remain idempotent. The pin does not judge the DDL; it makes changing it impossible to do by accident. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(packages): gate publishing on the version check The version check ran only in ci.yml. publish-packages.yml is a separate workflow, so it could reach `pnpm publish` on a main push, a matching tag, or a manual run with dry_run=false even when CI was red, skipped, or never ran. Run the validation inside the publish job, before publishing, in a new release mode that works for every trigger. A push range only exists for branch pushes, so release mode judges each package against the registry instead: a version that is not published yet is a release, and a version that is already published is accepted only when the package source has not moved since that release. `pnpm publish` skips an already-published version, so without that second half a drifted package releases as a silent no-op. The anchor for "since that release" is the @openmaic/<name>@<version> tag the job now writes after every successful publish, falling back to the push range for packages released before those tags existed. With neither anchor the release stops instead of guessing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(packages): make the release gate prove only what it can Three independent reviews of the previous revision converged on one root cause: it used git tags and a push range as the record of which tree produced a published version. Neither is trustworthy. Tags are mutable and can be created or moved by hand, so "the tag resolves" was being read as "the tag is authentic"; pnpm records no gitHead; and a push range describes one push, not the origin of a release. Packages released before the scheme existed had no anchor at all, which made the tag and manual triggers fail closed from the first run with no way to self-heal, dry runs included. Drop the anchor machinery. Drift is prevented where it is provable, in diff mode at merge time, which sees every commit before it can be released. The pre-publish gate is now limited to claims a release can establish: every version it publishes is new and moves forward, an already-published version is reported and left alone, and any registry answer that is not a definitive 404 stops the release. Around it, close what the reviews found in the workflow: - real publishes only from a commit contained in main, so a tag or manual run cannot ship an unreviewed ref while holding NPM_TOKEN - one repository-wide concurrency group, so a main push and a tag push cannot each decide the same version is unpublished - publish and mark each package individually, so a partial failure stays retryable instead of leaving published packages unrecorded - fail if the build rewrites tracked package files, which would otherwise publish content that is not in the released commit - compare against every published version when ordering, and refuse to guess when the registry holds versions this check cannot order The schema pin also asserted the constants without asserting what the ensure functions execute, so those could diverge. Both are now run against a recording queryable and their exact statement sequence is asserted. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(packages): refuse to release from a tree behind the registry Simplifying the gate introduced a hole that a mutation check caught: a package whose local version is already published took the "already published, skip" path before any ordering check, so a tree rolled back to an older published version passed silently. That is not a harmless no-op, because pnpm publish rewrites each workspace dependency to the version in the tree, and a sibling released alongside it would be published declaring a dependency on the older package. Order every package against the registry first, then decide: at the highest published version it is a legitimate skip, below it the release stops. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(packages): couple publishing to a green CI run for the same commit Second review round, on the reworked design. The headline defect was a hard blocker: `git fetch --depth=0` is not valid git (that is an actions/checkout convention), so under `set -e` every real publish aborted before the gate ever ran. Remove the depth argument; the checkout already has full history. The rest closes the gap both reviewers reached independently, that being on main is not the same as having passed the gate: - wait for this exact commit's CI conclusion and require success. ci.yml runs concurrently with this workflow on a push to main and blocks nothing, so an unpublished version whose source drifted could be released while its own version check was still running or already red. - require the commit on main's FIRST-PARENT history. Plain reachability also accepts every intermediate commit of every branch merged with a merge commit. A pull request that adds a bad tree and reverts it in the next commit leaves that tree an ancestor of main forever, and a tag pointing at it published it. - do not cancel CI runs on main. Each run validates only its own push range, so cancelling one drops the range that contained the change, and its replacement compares against the cancelled tip and sees nothing. - run the version check on every main push. A branch creation or force push reports an unusable `before`, which used to skip the gate entirely; fall back to the first parent instead. - publish with --ignore-scripts. Every package's prepublishOnly reruns the build, deleting and regenerating the dist that was just verified and smoke-tested, so npm could receive bytes nothing had checked. - run storage's PostgreSQL contract suites before publishing it. They skip themselves without a database, so the one backend that needs a real PostgreSQL was shipping unexercised. - order registry versions with real semver, prereleases included. Rejecting every non-x.y.z version meant one historical prerelease anywhere in one package's history would refuse every future release of every package. - warn when a release marker exists but points at another commit. Note what is NOT enforceable here: a workflow_dispatch or tag run executes the workflow definition from the selected ref, so a branch that edits this file can delete these checks. The boundary has to be a GitHub Environment holding NPM_TOKEN behind a main-only deployment branch rule. The job now declares that environment and the header states the required setup; the in-file checks are defence in depth. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(packages): stop the first-parent check failing on SIGPIPE Third review round. The blocker this time: `git rev-list --first-parent | grep -qx` under `set -o pipefail`. grep exits at the first match, rev-list then dies of SIGPIPE, and the pipeline returns 141, so a commit that IS on main is read as "not on main" and every real publish is refused. It does not reproduce in a shallow clone, where main's history fits in the pipe buffer; with the full history this job checks out it always will. Count matches instead, which reads the whole stream. Also from this round: - identify the CI run by workflow file and triggering event rather than by a check-run display name. Names are not unique, so any other workflow or app publishing a check called "Lint, Typecheck & Unit Tests" could stand in for a red version gate. - fail the main-push version check when the push range is unusable instead of substituting HEAD^. A force push can replace many commits at once, so the previous commit is not the range that needs checking, and a green run here is what publishing depends on. - correct the stated limitation. It claimed the package-directory input model was exact for dsl and storage; it is not. Their dist is whatever the lockfile's TypeScript emits, and dsl's shipped JSON schema comes from the lockfile's ts-json-schema-generator, so a toolchain bump can change any of the four tarballs with no diff under the package directory. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * refactor(packages): make a merged version bump the only release input The tag trigger was incoherent. Its own documentation admitted that any @openmaic/* tag republishes the whole family and that which packages actually go out is decided by the manifests, so the tag name never participated in the decision. It was a second entry point to one decision function, it bypassed review entirely because a tag is not a diff and can be created by anyone with write access on any commit, and it cannot be guarded: environment deployment rules match GITHUB_REF, so refs/tags/* and a main-only rule are mutually exclusive. Drop it. A version bump that landed on main is now the only release input, and `@openmaic/<name>@<version>` tags are an output written after a package reaches the registry. Nothing is lost: a manual dispatch from main still republishes without a new commit, and that path is inside the protected environment. Split the workflow so the token has a boundary. `validate` holds everything that does not need NPM_TOKEN and runs from any ref, which keeps dry runs useful to contributors. `publish` is the only job declaring the `release` environment, so the token is never attached to a run that is merely validating. It rebuilds rather than sharing state, which is the right trade for a release. Also add the `actions: read` permission the CI-conclusion query needs; it was switched from the checks API to the Actions API without updating the scope, which would have failed with 403 on every real publish. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs(contributing): explain when a package version must be bumped CONTRIBUTING said nothing about versioning or releases, while this branch adds a CI check that fails a contributor's PR with "publishable package inputs changed but version did not increase". A gate that rejects work without telling anyone the rule is the kind that ends up disabled. State the rule, the exact failure message, which files are exempt, and how to choose the number, with a specific warning for @openmaic/dsl: it is the contract the other packages validate against, so narrowing what an existing document may contain is breaking even when the diff is small. Also state what contributors do NOT do: publishing is automatic once a bump lands on main, and the release tag is a marker written afterwards rather than something to push. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(packages): run renderer's tests before publishing renderer The pre-publish gate ran renderer's typecheck but never its test suite, so the one package whose behaviour is hardest to typecheck was the one shipping unexercised. It has 25 test files and 242 tests; they just were not in the filter list. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(packages): keep the git credential away from package code Review finding: the publish job held `contents: write` while actions/checkout had persisted that credential, and it then ran `pnpm install` and the package builds. Any dependency or build script executed with a credentialed git remote in reach and a write scope to use it, which is a wider blast radius than publishing needs. Separate the two capabilities so no job holds both: - `validate` and `publish` run package code. Both now check out with `persist-credentials: false`, `validate` is pinned to `contents: read`, and `publish` drops write scope entirely, keeping only the `actions: read` it needs for the CI conclusion and `id-token: write` for provenance. - a new `mark` job is the only holder of `contents: write` and a git credential, and it installs nothing and builds nothing. `mark` also reconciles instead of only recording this run. A marker whose push failed previously could never be repaired before, because the release plan excludes versions already on the registry and the marker logic sat behind that plan. It now marks any registry version whose tag is missing, and leaves an existing tag alone rather than moving it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-07-29 23:54:23 -04:00