Files
paperclip/scripts/cloud-readiness.mjs
T
Devin FoleyandPaperclip 4cc387f907 fix(ci): remove npm propagation from cloud readiness (#13456)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Paperclip Cloud needs a verified image and matching database
migrator before it can deploy a merge.
> - New npm package versions can take minutes to become downloadable
after the package build finishes.
> - The direct producer now publishes signed archives and a complete
dependency lockfile for each master commit.
> - This pull request makes readiness verify those artifacts and removes
the duplicate automatic npm migrator run.
> - Deployment still requires all source checks, exact image identity,
migration compatibility, and pinned dependencies.

## Linked Issues or Issue Description

Refs: #13455, #13454, #13192

**What existing behavior does this improve?**
The time from a master merge to the `Cloud deployable v1` signal.

**Current behavior**
Readiness polls npm metadata for the new DB and shared versions. An
automatic dispatcher also starts a separate npm-only migrator workflow.
A measured source built its packages at 06:22:41 UTC on 2026-09-15, but
both npm archives were not downloadable until 06:31:56 UTC.

**Proposed behavior**
Wait for the successful exact-source direct producer, verify its signed
manifest and all pinned downloads, and publish readiness only after the
existing source and image jobs pass. Keep manual npm migrators and
branch previews available.

**Reason and benefit**
Remove new-version npm propagation from merge-to-deployable time. The
gain depends on whether image building or source verification finishes
later; it is not a fixed subtraction from every run.

## What Changed

- Require a successful producer from the canonical repository, exact
commit, master ref, expected workflow, and approved event.
- Verify the manifest's GitHub attestation with the hosted GitHub CLI.
Enforce the exact source SHA, master workflow identity, and hosted
runner.
- Download and validate both archives and the complete dependency
lockfile after publication succeeds. Reject invalid signatures,
inaccessible objects, corrupt bytes, and source mismatches.
- Remove automatic npm-only migrator dispatch. Retain manual release and
branch-preview publication.
- Document the cloud feature-switch prerequisite and coordinated
rollback.

## Verification

- `node --test .github/scripts/tests/*.test.mjs`: 405 pass.
- Focused readiness, routing, preview, and artifact tests: 249 pass.
- Workflow lint and `git diff --check`: pass.
- `pnpm test:release-registry`: 139 pass after installing this
worktree's dependencies.
- All latest-head GitHub CI checks passed. Greptile is 5/5 with no
unresolved comments.
- Application source is unchanged. Common-source local typecheck and
build passed. The full local application suite has the documented macOS
read-only-directory rename limitation from #13454 (13 failures in two
unchanged suites); Linux CI is the final application gate.
- Live readiness verification of master
da77a0c28c passed in 6.52 seconds,
including the real GitHub CLI signature policy and all artifact
downloads.
- Cloud consumer resolution with the certificate encoding fix passed in
5.45 seconds with zero npm metadata requests or npm processes. The
consumer is deployed and enabled in staging and production; their live
resolution APIs passed in 2.34 and 2.38 seconds. Both report the
expected fixed harness commit. A fresh tenant deployment follows this
cutover merge.

## Risks

- `Cloud deployable v1` no longer promises npm preview availability.
Enable the cloud direct-artifact consumer in staging and production
before merging this change.
- Artifact storage and GitHub attestations become required services for
new direct releases. Missing or invalid evidence fails explicitly.
- Restore the old npm dispatcher and readiness gate together before
disabling the consumer switch. Retain artifacts referenced by existing
releases.
- This change does not expand AWS runner access. The producer and
readiness bookkeeping use GitHub-hosted runners. Existing PR allowlists
and source verification gates remain enforced.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, repository tools, and code
execution. The exact serving model ID and context window are not exposed
by this environment.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (focused checks; full
application host limitation documented above)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-09-15 01:14:43 -07:00

103 lines
5.1 KiB
JavaScript

#!/usr/bin/env node
import { pathToFileURL } from "node:url";
import { execFileSync } from "node:child_process";
import { mkdtempSync, rmSync, writeFileSync } from "node:fs";
import os from "node:os";
import path from "node:path";
import { imageExists, versionFor } from "./preview-artifacts.mjs";
import { verifyPublished } from "./cloud-migrator-artifacts.mjs";
const repository = "paperclipai/paperclip";
const workflow = ".github/workflows/cloud-migrator-artifacts.yml";
export async function migratorPublished(sha, fetchImpl, token) {
let pending = false;
const failures = [];
for (let page = 1; page <= 10; page++) {
const response = await fetchImpl(`https://api.github.com/repos/${repository}/actions/workflows/cloud-migrator-artifacts.yml/runs?branch=master&head_sha=${sha}&per_page=100&page=${page}`, {
headers: { Accept: "application/vnd.github+json", ...(token ? { Authorization: `Bearer ${token}` } : {}) },
redirect: "error", signal: AbortSignal.timeout(30_000),
});
if (!response.ok) throw new Error(`Migrator producer lookup failed: HTTP ${response.status}`);
const body = await response.json();
if (!Array.isArray(body.workflow_runs) || !Number.isSafeInteger(body.total_count) || body.total_count < 0 ||
(page === 1 && (body.total_count === 0) !== (body.workflow_runs.length === 0))) throw new Error("Invalid migrator producer response.");
if (body.total_count === 0) return false;
for (const run of body.workflow_runs) {
if (run.head_sha !== sha || run.head_branch !== "master" || run.path !== workflow ||
run.head_repository?.id !== 1170821064 || run.head_repository.full_name !== repository ||
!["push", "workflow_dispatch"].includes(run.event)) throw new Error("Migrator producer identity mismatch.");
// Publication is immutable. A later failed manual run must not hide a
// successful exact-source publisher; the signed bundle is checked next.
if (run.status === "completed" && run.conclusion === "success") return true;
if (run.status !== "completed") pending = true;
else failures.push(`${run.id}: ${run.conclusion}`);
}
if (page * 100 >= body.total_count) {
if (pending) return false;
throw new Error(`Migrator producers failed: ${failures.join(", ")}.`);
}
}
throw new Error("Too many migrator producer runs to establish publication.");
}
export function verifyManifestProvenance(bytes, sha, { exec = execFileSync } = {}) {
versionFor(sha);
const scratch = mkdtempSync(path.join(os.tmpdir(), "cloud-readiness-attestation-"));
try {
const file = path.join(scratch, "manifest.json");
writeFileSync(file, bytes);
exec("gh", ["attestation", "verify", file, "--repo", repository,
"--source-digest", sha, "--source-ref", "refs/heads/master",
"--cert-identity", `https://github.com/${repository}/${workflow}@refs/heads/master`,
"--deny-self-hosted-runners"], { stdio: "inherit", timeout: 60_000 });
} finally { rmSync(scratch, { recursive: true, force: true }); }
}
/** Read-only availability gate. Deployment still resolves and pins artifacts. */
export async function waitForCloudArtifacts(sha, {
fetchImpl = fetch,
token = process.env.GH_TOKEN,
verifyProvenance = verifyManifestProvenance,
now = () => performance.now(),
sleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms)),
timeoutMs = 30 * 60_000,
intervalMs = 20_000,
log = console.log,
} = {}) {
const version = versionFor(sha);
if (!Number.isFinite(timeoutMs) || timeoutMs <= 0 || !Number.isFinite(intervalMs) || intervalMs <= 0) {
throw new Error("Cloud readiness requires positive finite timeout and poll interval.");
}
const deadline = now() + timeoutMs;
let previous;
let missing = ["image", "migrator"];
while (now() < deadline) {
// Recheck the image and exact-source publisher on the successful poll.
// Only a missing/pending producer waits; failed publication fails closed.
const results = await Promise.all([
imageExists(sha, fetchImpl),
migratorPublished(sha, fetchImpl, token),
]);
missing = ["image", "migrator"].filter((_, index) => !results[index]);
if (missing.length === 0) {
// Verify the exact signed bytes and all pinned downloads after the
// publisher succeeds. An inaccessible or corrupt artifact cannot pass.
await verifyPublished(sha, fetchImpl, { verifyProvenance });
log(`Cloud artifacts available for ${sha}: verified image and exact-source migrator ${version}.`);
return { version: 1, sha, packageVersion: version };
}
const state = missing.join(", ");
if (state !== previous) log(`Waiting for cloud artifacts for ${sha}: ${state}.`);
previous = state;
const remaining = deadline - now();
if (remaining > 0) await sleep(Math.min(intervalMs, remaining));
}
throw new Error(`Cloud artifacts timed out for ${sha}; missing: ${missing.join(", ")}.`);
}
if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) {
try { await waitForCloudArtifacts(process.argv[2]); }
catch (error) { console.error(error.message); process.exitCode = 1; }
}