trending/paperclip
1
0
Fork 0
mirror of https://github.com/paperclipai/paperclip.git synced 2026-10-02 02:07:25 +08:00
Code Issues Packages Projects Releases Wiki Activity

ci: retire recurring public cloud image builds (#13827)

Browse Source 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Core publishes standard images and source verification for
downstream services.
> - Managed services can now compose private images from the signed
standard image.
> - Core still builds a second public cloud image on every master push
and release.
> - That duplicate producer consumes build capacity and retains an
obsolete readiness contract.
> - This pull request retires recurring cloud publication while
preserving the standard producer and rollback artifacts.

## Linked Issues or Issue Description

Refs #13797 and #13789. Related: #12856 changes image dependency
packaging; it does not retire this producer.

**What existing behavior does this improve?**

Core's recurring Docker publication and Cloud readiness workflow.

**Current behavior**

Master pushes call the legacy cloud publisher from Cloud readiness.
Release tags and manual Docker runs call it too. Canary promotion also
requires the legacy image.

**Proposed behavior**

Publish standard Core images and retain `Cloud source verified v1`. Let
downstream services build their managed image. Keep explicit commit
previews and existing images available.

## What Changed

- Remove `docker-cloud.yml`, its master and release callers, and its
unused cache selector.
- Remove the legacy image/migrator wait and `Cloud deployable v1` job.
Keep the full source verification workflow and exact source-proof name.
- Make canary promotion inspect and promote the standard image only.
- Preserve signed standard-image publication, direct migrator
publication, and explicit `release.yml` previews. The preview path still
uses the Dockerfile `cloud` target.
- Update workflow, preview, build-stamp, and packaging tests. Exercise
the promotion shell with mocked registry commands, including
missing-image and missing-tag cases.
- Document frozen legacy aliases, consumer requirements, preview
compatibility, and rollback retention.

## Verification

- All 377 workflow tests pass: `node --test
.github/scripts/tests/*.test.mjs`.
- All 129 release-registry tests pass: `pnpm test:release-registry`.
- Focused source-proof, standard-image, preview, and workflow tests
pass: 256 tests.
- Focused image packaging/build-stamp tests pass: 16 tests.
- Actionlint passes on all three changed workflow files. `git diff
--check` passes.
- Full local `pnpm build` and `pnpm -r typecheck` pass.
- The policy follow-up updates an old assertion that required the
removed readiness job. All 37 source-proof/release-workflow tests pass
locally.
- Full local `pnpm test:run` did not complete successfully while the Mac
ran out of disk space. No full-suite pass is claimed. Removed 1.2 GiB of
generated Cargo output from this isolated worktree with `cargo clean`.
GitHub CI passed on the final head: 52 successful checks and 2 optional
skips.
- Fresh Greptile review for `4f5fe1951f0bd7f7739cf6655d395ff78f1ed944`:
**5/5**, successful current-head check, zero review threads.
- September 23 refresh: the unchanged PR head merges cleanly with
current master `db8f8fe5b73a2697684a30261b0d306a9c631aba`. In an
isolated temporary worktree, all 377 workflow tests and 29
release/preview tests pass on the combined tree. `git diff --cached
--check` passes.
- Refreshed Actionlint workflow validation passes with ShellCheck
disabled. Full Actionlint reports the same 10 existing ShellCheck
diagnostics as master, with no added diagnostics. No source changes or
new PR commits were needed.
- The full local build/typecheck and current-head Linux CI results above
remain the verification for the unchanged PR head. They were not rerun
for this metadata-only refresh. No image publication or tenant
deployment was initiated for this refresh.

## Risks

**Deployment prerequisite satisfied (September 23):** The combined
cleanup release is deployed to staging and production, and production
Support is verified. Active managed-fleet automation uses standard-image
composition. Explicit immutable previews remain supported by the
retained preview publisher. This PR is ready for maintainer review; keep
auto-merge disabled and wait for explicit merge authorization.

- A consumer still selecting `Cloud deployable v1` will stop advancing
at the last legacy-ready commit. Confirm active automatic consumers use
the standard-image composition contract before merge.
- Legacy cloud release-channel aliases stop advancing. Standard
self-hosted aliases continue.
- This PR deletes no registry images, cache tags, migrators,
credentials, or runner infrastructure. Existing immutable releases
remain usable for rollback.
- Explicit legacy previews remain for commit-specific operator
deployments. Retiring that compatibility path requires a separate
consumer migration.
- These changes affect CI publication, not database schema or
application behavior.

## Model Used

OpenAI Codex, GPT-6. The runtime does not expose a more specific model
identifier or context-window size. Used repository inspection,
reasoning, code editing, shell tools, and test execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
Devin Foley
2026-09-23 19:35:19 -07:00
committed by GitHub
co-authored by Paperclip
parent db8f8fe5b7
commit 8ee8f1fd6e
23 changed files with 197 additions and 1187 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.github/scripts/tests/cloud-readiness.test.mjs
+21 -134
View File
@@ -1,144 +1,31 @@
import test from "node:test";
import assert from "node:assert/strict";
import { existsSync, readFileSync } from "node:fs";
import { gzipSync } from "node:zlib";
import { waitForCloudArtifacts, verifyManifestProvenance, migratorPublished } from "../../../scripts/cloud-readiness.mjs";
import { artifactBase, descriptor } from "../../../scripts/cloud-migrator-artifacts.mjs";
import { previewManifest } from "../../../scripts/preview-artifacts.mjs";
const sha = "a".repeat(40);
const version = `0.0.0-preview.g${sha}`;
const digest = `sha256:${"b".repeat(64)}`;
const json = (body, status = 200) => new Response(JSON.stringify(body), { status });
const producer = { id: 123, head_sha: sha, head_branch: "master", path: ".github/workflows/cloud-migrator-artifacts.yml",
head_repository: { id: 1170821064, full_name: "paperclipai/paperclip" }, event: "push", status: "completed", conclusion: "success" };
function bundle() {
const packages = {}; const files = new Map();
const entries = { "": { dependencies: { "@paperclipai/db": version } } };
for (const name of ["db", "shared"]) {
const metadata = previewManifest({ name: `@paperclipai/${name}`, dependencies: {} }, sha);
const bytes = Buffer.from(JSON.stringify(metadata));
const header = Buffer.alloc(512); header.write("package/package.json"); header.write(bytes.length.toString(8).padStart(11, "0"), 124, 11); header[156] = 48;
const padded = Buffer.alloc(Math.ceil(bytes.length / 512) * 512); bytes.copy(padded);
const archive = gzipSync(Buffer.concat([header, padded, Buffer.alloc(1024)]));
const pin = descriptor(archive, "tgz"); packages[name] = pin; files.set(pin.url, archive);
entries[`node_modules/@paperclipai/${name}`] = { version, resolved: pin.url, integrity: pin.integrity, dependencies: metadata.dependencies };
}
const lock = Buffer.from(JSON.stringify({ lockfileVersion: 3, packages: entries }));
const manifest = { version: 1, sourceSha: sha, packageVersion: version, packages, lockfile: descriptor(lock, "json") };
files.set(manifest.lockfile.url, lock);
const bytes = Buffer.from(JSON.stringify(manifest) + "\n");
files.set(`${artifactBase}/${sha}/manifest.json`, bytes);
return { manifest, bytes, files };
}
function registry({ missing = new Set(), failure, wrongImage = false, run = producer, objects = bundle() } = {}) {
return async (url, options) => {
assert.ok(!url.startsWith("https://registry.npmjs.org/"), "readiness must never wait for npm");
if (failure) return json({}, failure);
if (url.startsWith("https://api.github.com/")) {
assert.match(url, new RegExp(`head_sha=${sha}&per_page=100&page=1$`));
return json({ total_count: missing.has("migrator") ? 0 : 1, workflow_runs: missing.has("migrator") ? [] : [run] });
}
if (url.startsWith(artifactBase)) {
assert.equal(options.headers?.Authorization, undefined, "GitHub credentials stay off the artifact origin");
return objects.files.has(url) ? new Response(objects.files.get(url)) : json({}, 403);
}
if (url.includes("/token?")) return json({ token: "fixture" });
if (url.includes("/manifests/")) return missing.has("image") ? json({}, 404) : json({ config: { digest } });
if (url.includes("/blobs/")) return json({ config: { Labels: { "org.opencontainers.image.revision": wrongImage ? "c".repeat(40) : sha } } });
throw new Error(`Unexpected request: ${url}`);
};
}
const noSignature = async () => {}; // Signature enforcement is exercised separately below.
const workflow = readFileSync(new URL("../../workflows/cloud-readiness.yml", import.meta.url), "utf8");
test("readiness rechecks image and publisher, then verifies the exact signed bundle with no npm requests", async () => {
const missing = new Set(["image", "migrator"]); const objects = bundle(); let clock = 0; let signatures = 0;
const result = await waitForCloudArtifacts(sha, {
fetchImpl: registry({ missing, objects }), token: "fixture", now: () => clock, intervalMs: 10, timeoutMs: 100, log: () => {},
verifyProvenance: async (bytes, source) => { assert.deepEqual(bytes, objects.bytes); assert.equal(source, sha); signatures++; },
sleep: async (ms) => {
clock += ms;
if (clock === 10) missing.delete("image");
if (clock === 20) { missing.delete("migrator"); missing.add("image"); }
if (clock === 30) missing.delete("image");
},
});
assert.equal(clock, 30); assert.equal(signatures, 1);
assert.deepEqual(result, { version: 1, sha, packageVersion: version });
});
test("missing or in-progress publishers time out with a precise inventory and bounded sleep", async () => {
for (const fixture of [{ missing: new Set(["migrator"]) }, { run: { ...producer, status: "in_progress", conclusion: null } }]) {
let clock = 0; const sleeps = [];
await assert.rejects(waitForCloudArtifacts(sha, {
fetchImpl: registry(fixture), now: () => clock, timeoutMs: 25, intervalMs: 20, log: () => {}, verifyProvenance: noSignature,
sleep: async (ms) => { sleeps.push(ms); clock += ms; },
}), /timed out.*missing: migrator/);
assert.deepEqual(sleeps, [20, 5]);
}
});
for (const fixture of [{ failure: 403 }, { failure: 503 }, { wrongImage: true },
...["failure", "cancelled", "skipped"].map((conclusion) => ({ run: { ...producer, conclusion } })),
...[{ head_sha: "b".repeat(40) }, { head_branch: "feature" }, { path: ".github/workflows/evil.yml" },
{ head_repository: { id: 123, full_name: "someone/paperclip" } }, { event: "pull_request" }].map((wrong) => ({ run: { ...producer, ...wrong } }))]) {
test(`upstream errors, failed publication and identity mismatches fail immediately: ${JSON.stringify(fixture)}`, async () => {
await assert.rejects(waitForCloudArtifacts(sha, { fetchImpl: registry(fixture), verifyProvenance: noSignature,
sleep: async () => assert.fail("must not retry an invalid artifact or upstream error"), log: () => {} }));
});
}
test("successful publication cannot hide inaccessible or corrupt archives or an invalid signature", async () => {
for (const corrupt of [false, true]) {
const objects = bundle();
if (corrupt) objects.files.set(objects.manifest.packages.db.url, Buffer.from("corrupt"));
else objects.files.delete(objects.manifest.packages.db.url);
await assert.rejects(waitForCloudArtifacts(sha, { fetchImpl: registry({ objects }), verifyProvenance: noSignature, log: () => {} }), /download failed|immutable pin/);
}
await assert.rejects(waitForCloudArtifacts(sha, { fetchImpl: registry(), verifyProvenance: async () => { throw new Error("invalid signature"); }, log: () => {} }), /invalid signature/);
});
test("CLI verifies the exact bytes, source, master workflow and hosted runner and cleans up on failure", () => {
let temporary;
assert.throws(() => verifyManifestProvenance(Buffer.from("exact manifest\n"), sha, { exec: (cmd, args) => {
assert.equal(cmd, "gh"); assert.deepEqual(args.slice(0, 2), ["attestation", "verify"]); temporary = args[2];
assert.equal(readFileSync(temporary, "utf8"), "exact manifest\n");
for (const [flag, value] of [["--repo", "paperclipai/paperclip"], ["--source-digest", sha], ["--source-ref", "refs/heads/master"],
["--cert-identity", "https://github.com/paperclipai/paperclip/.github/workflows/cloud-migrator-artifacts.yml@refs/heads/master"]]) assert.equal(args[args.indexOf(flag) + 1], value);
assert.ok(args.includes("--deny-self-hosted-runners")); throw new Error("verification rejected");
} }), /verification rejected/);
assert.equal(existsSync(temporary), false);
});
test("invalid source and timing configuration are rejected before registry access", async () => {
const fetchImpl = async () => assert.fail("invalid inputs must not reach a registry");
await assert.rejects(waitForCloudArtifacts("master", { fetchImpl }), /full immutable commit SHA/);
for (const options of [{ timeoutMs: 0 }, { intervalMs: -1 }, { timeoutMs: Infinity }]) await assert.rejects(waitForCloudArtifacts(sha, { ...options, fetchImpl }), /positive finite/);
});
test("versioned readiness retains every source gate and removes duplicate automatic npm publication", () => {
const workflow = readFileSync(new URL("../../workflows/cloud-readiness.yml", import.meta.url), "utf8");
test("retirement preserves exact-source verification without issuing legacy deployment readiness", () => {
assert.match(workflow, /push:\s*\n\s*branches: \[master\]/);
assert.match(workflow, /uses: \.\/\.github\/workflows\/release-verify.yml\s+with:\s+ref: \$\{\{ github.sha \}\}/);
assert.match(workflow, /uses: \.\/\.github\/workflows\/docker-cloud.yml/);
assert.match(workflow, /attestations: read/); assert.match(workflow, /GH_TOKEN: \$\{\{ github.token \}\}/);
const ready = workflow.split(" ready:")[1];
assert.match(ready, /name: Cloud deployable v1/); assert.match(ready, /needs: \[verify, image, artifacts\]/);
assert.match(ready, /if: github.repository == 'paperclipai\/paperclip' && github.ref == 'refs\/heads\/master'/);
assert.doesNotMatch(ready, /^\s*(?:if:.*always\(|continue-on-error:)/m);
assert.doesNotMatch(workflow, /secrets: inherit|id-token: write|actions: write|checks: write|uses: .*@v\d\b/);
assert.equal(existsSync(new URL("../../workflows/cloud-artifacts.yml", import.meta.url)), false);
const proof = workflow.split(" source_verified:")[1];
assert.ok(proof);
assert.match(proof, /name: Cloud source verified v1/);
assert.match(proof, /needs: \[verify\]/);
assert.match(proof, /if: github.repository == 'paperclipai\/paperclip' && github.ref == 'refs\/heads\/master'/);
assert.doesNotMatch(proof, /^\s*(?:if:.*always\(|continue-on-error:)/m);
assert.doesNotMatch(workflow, /Cloud deployable v1|docker-cloud.yml|cloud-readiness.mjs|^ (image|artifacts|ready):/m);
assert.doesNotMatch(workflow, /packages: write|secrets: inherit|id-token: write|actions: write|checks: write|uses: .*@v\d\b/);
assert.equal(existsSync(new URL("../../workflows/docker-cloud.yml", import.meta.url)), false);
});
test("later manual failures or pending retries cannot hide an earlier successful immutable publication", async () => {
for (const latest of [{ status: "completed", conclusion: "failure" }, { status: "in_progress", conclusion: null }]) {
let calls = 0;
assert.equal(await migratorPublished(sha, async (url) => {
calls++;
if (url.endsWith("page=1")) return json({ total_count: 101, workflow_runs: Array.from({ length: 100 }, (_, i) => ({ ...producer, ...latest, id: 200 + i, event: "workflow_dispatch" })) });
assert.ok(url.endsWith("page=2")); return json({ total_count: 101, workflow_runs: [producer] });
}), true);
assert.equal(calls, 2);
}
test("standard image provenance and independent exact-source migrators remain available", () => {
const docker = readFileSync(new URL("../../workflows/docker.yml", import.meta.url), "utf8");
assert.match(docker, /target: production/);
assert.match(docker, /type=raw,value=sha-\$\{\{ github.sha \}\}/);
assert.match(docker, /run: node scripts\/standard-image-contract.mjs --resolve "\$GITHUB_SHA"/);
assert.match(docker, /subject-digest: \$\{\{ steps.standard.outputs.digest \}\}/);
const migrator = readFileSync(new URL("../../workflows/cloud-migrator-artifacts.yml", import.meta.url), "utf8");
assert.match(migrator, /push:\s*\n\s*branches: \[master\]/);
assert.match(migrator, /uses: actions\/attest@/);
assert.doesNotMatch(docker, /build-and-push-cloud|docker-cloud.yml|canary-cloud/);
});
.github/scripts/tests/cloud-runner-routing.test.mjs
-35
View File
@@ -1,35 +0,0 @@
import test from "node:test";
import assert from "node:assert/strict";
import { readFileSync } from "node:fs";
import { runInNewContext } from "node:vm";
const workflow = readFileSync(new URL("../../workflows/docker-cloud.yml", import.meta.url), "utf8");
// Exercise the workflow's actual boolean expression. Its string comparisons and
// boolean operators have the same results in JS for these canonical contexts.
const expression = workflow.match(/^ runs-on: \$\{\{ (.+) \}\}$/m)?.[1];
assert.ok(expression, "cloud routing must remain an explicit job expression");
const timeoutExpression = workflow.match(/^ timeout-minutes: \$\{\{ (.+) \}\}$/m)?.[1];
assert.ok(timeoutExpression, "AWS jobs must finish before the Fleet instance lifetime");
const fleet = "runs-on/fleet=paperclip-cloud-build-x64/env=public-ci";
const base = { repository: "paperclipai/paperclip", repository_id: "1170821064", ref: "refs/heads/master", event_name: "push" };
for (const { name, github = {}, enabled = "true", expected = "ubuntu-latest" } of [
{ name: "canonical master push", expected: fleet },
{ name: "manual master build", github: { event_name: "workflow_dispatch" }, expected: fleet },
{ name: "disabled switch", enabled: "false" },
{ name: "missing switch", enabled: "" },
{ name: "invalid switch", enabled: "yes" },
{ name: "fork", github: { repository: "someone/paperclip", repository_id: "123" } },
{ name: "wrong repository identity", github: { repository_id: "123" } },
{ name: "pull request", github: { event_name: "pull_request", ref: "refs/pull/123/merge" } },
{ name: "privileged PR event", github: { event_name: "pull_request_target" } },
{ name: "release tag", github: { ref: "refs/tags/v2026.911.0" } },
{ name: "branch push", github: { ref: "refs/heads/feature" } },
{ name: "manual branch build", github: { event_name: "workflow_dispatch", ref: "refs/heads/feature" } },
{ name: "workflow completion event", github: { event_name: "workflow_run" } },
]) {
test(`cloud runner routing: ${name}`, () => {
const context = { github: { ...base, ...github }, vars: { AWS_CLOUD_BUILDS_ENABLED: enabled } };
assert.equal(runInNewContext(expression, context), expected);
assert.equal(runInNewContext(timeoutExpression, context), expected === fleet ? 40 : 60);
});
}
.github/scripts/tests/docker-canary-promotion.test.mjs
+56
View File
@@ -0,0 +1,56 @@
import test from "node:test";
import assert from "node:assert/strict";
import { mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import path from "node:path";
import { spawnSync } from "node:child_process";
const workflow = readFileSync(new URL("../../workflows/docker.yml", import.meta.url), "utf8");
const job = workflow.split(" promote_canary_channel:\n")[1];
const script = job.split(" run: |\n")[1].split("\n").map(line => line.replace(/^ {10}/, "")).join("\n");
const sha = "a".repeat(40);
test("canary promotion waits for the standard manifest and keeps its serialized channel", () => {
assert.match(job, /needs: \[merge-and-push\]/);
assert.match(job, /group: docker-canary-channel-promotion/);
assert.match(job, /cancel-in-progress: false/);
});
for (const [name, commitPresent, imagePresent] of [
["promotes the current npm canary without a cloud image", true, true],
["waits when the standard image is missing", true, false],
["waits when the npm canary tag has not resolved", false, false],
]) {
test(name, () => {
const dir = mkdtempSync(path.join(tmpdir(), "standard-canary-promotion-"));
const log = path.join(dir, "calls.jsonl");
const fixture = `#!${process.execPath}
const fs = require("node:fs");
const command = require("node:path").basename(process.argv[1]);
const args = process.argv.slice(2);
fs.appendFileSync(process.env.TEST_CALLS, JSON.stringify({ command, args }) + "\\n");
if (command === "curl") process.stdout.write(JSON.stringify({ canary: "2026.922.0-canary.1" }));
else if (command === "gh") { if (process.env.COMMIT_PRESENT !== "true") process.exit(1); process.stdout.write(process.env.TEST_SHA); }
else if (args.slice(0, 3).join(" ") === "buildx imagetools inspect") process.exit(process.env.IMAGE_PRESENT === "true" ? 0 : 1);
else if (args.slice(0, 3).join(" ") !== "buildx imagetools create") process.exit(99);
`;
try {
for (const command of ["curl", "gh", "docker"]) writeFileSync(path.join(dir, command), fixture, { mode: 0o755 });
const result = spawnSync("bash", ["-e", "-o", "pipefail", "-c", script], {
encoding: "utf8", env: {
...process.env, PATH: `${dir}${path.delimiter}${process.env.PATH}`,
IMAGE: "ghcr.io/paperclipai/paperclip", GITHUB_REPOSITORY: "paperclipai/paperclip",
TEST_CALLS: log, TEST_SHA: sha, COMMIT_PRESENT: String(commitPresent), IMAGE_PRESENT: String(imagePresent),
},
});
assert.equal(result.status, 0, result.stderr);
const calls = readFileSync(log, "utf8").trim().split("\n").map(line => JSON.parse(line));
assert.ok(calls.find(call => call.command === "gh").args.some(arg => arg.includes("canary%2Fv2026.922.0-canary.1")));
const docker = calls.filter(call => call.command === "docker").map(call => call.args);
assert.deepEqual(docker, [
...(commitPresent ? [["buildx", "imagetools", "inspect", "ghcr.io/paperclipai/paperclip:sha-aaaaaaa"]] : []),
...(commitPresent && imagePresent ? [["buildx", "imagetools", "create", "-t", "ghcr.io/paperclipai/paperclip:canary", "ghcr.io/paperclipai/paperclip:sha-aaaaaaa"]] : []),
]);
} finally { rmSync(dir, { recursive: true, force: true }); }
});
}
.github/scripts/tests/docker-disk-workflow.test.mjs
-65
View File
@@ -1,65 +0,0 @@
import test from "node:test";
import assert from "node:assert/strict";
import { mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import path from "node:path";
import { spawnSync } from "node:child_process";
const workflow = readFileSync(new URL("../../workflows/docker-cloud.yml", import.meta.url), "utf8");
const step = workflow.split(" - name: Free runner disk")[1].split(" - name: Login to GitHub Container Registry")[0];
const script = step.split(" run: |\n")[1].split("\n").map((line) => line.replace(/^ {10}/, "")).join("\n");
const threshold = 64 * 1024 * 1024;
for (const { name, dockerFree, workspaceFree, dfStatus = "0", infoStatus = "0", cleanup } of [
{ name: "ample free space", dockerFree: threshold + 1, workspaceFree: threshold + 1, cleanup: false },
{ name: "exactly the headroom threshold", dockerFree: threshold, workspaceFree: threshold, cleanup: false },
{ name: "Docker filesystem below threshold", dockerFree: threshold - 1, workspaceFree: threshold + 1, cleanup: true },
{ name: "workspace filesystem below threshold", dockerFree: threshold + 1, workspaceFree: threshold - 1, cleanup: true },
{ name: "invalid Docker measurement", dockerFree: "unknown", workspaceFree: threshold + 1, cleanup: true },
{ name: "invalid workspace measurement", dockerFree: threshold + 1, workspaceFree: "unknown", cleanup: true },
{ name: "failed df command", dockerFree: threshold + 1, workspaceFree: threshold + 1, dfStatus: "1", cleanup: true },
{ name: "failed Docker inspection", dockerFree: threshold + 1, workspaceFree: threshold + 1, infoStatus: "1", cleanup: true },
]) {
test(`cloud disk cleanup: ${name}`, () => {
const dir = mkdtempSync(path.join(tmpdir(), "cloud-disk-test-"));
const log = path.join(dir, "commands.log");
// Every mutating command is a recording fixture; no real SDKs, caches,
// images, or directories are deleted when the workflow shell executes.
const fixture = `#!/bin/bash
printf '%s %s\\n' "\${0##*/}" "$*" >> "$COMMAND_LOG"
case "\${0##*/}" in
df)
printf 'Filesystem 1024-blocks Used Available Capacity Mounted on\\n'
if [ "$1" = '-Pk' ]; then
printf '/dev/docker 200000000 1 %s 1%% /docker\\n' "$DOCKER_FREE"
printf '/dev/workspace 200000000 1 %s 1%% /workspace\\n' "$WORKSPACE_FREE"
exit "$DF_STATUS"
fi
;;
docker)
if [ "$1" = 'info' ]; then
printf '/docker-data\\n'
exit "$INFO_STATUS"
fi
;;
esac
`;
try {
for (const command of ["df", "docker", "pnpm", "sudo"]) {
writeFileSync(path.join(dir, command), fixture, { mode: 0o755 });
}
const result = spawnSync("bash", ["-c", script], {
encoding: "utf8",
env: { ...process.env, PATH: `${dir}:${process.env.PATH}`, GITHUB_WORKSPACE: "/workspace", COMMAND_LOG: log,
DOCKER_FREE: String(dockerFree), WORKSPACE_FREE: String(workspaceFree), DF_STATUS: dfStatus, INFO_STATUS: infoStatus },
});
assert.equal(result.status, 0, result.stderr);
const commands = readFileSync(log, "utf8");
if (infoStatus === "0") assert.match(commands, /df -Pk \/docker-data \/workspace/);
assert.equal(commands.includes("pnpm store prune"), cleanup);
assert.equal(commands.includes("sudo rm -rf /usr/share/dotnet"), cleanup);
assert.equal(commands.includes("docker system prune -af"), cleanup);
assert.equal(result.stdout.includes("skipping cleanup"), !cleanup);
} finally { rmSync(dir, { recursive: true, force: true }); }
});
}
.github/scripts/tests/lockfile-refresh-workflows.test.mjs
-1
View File
@@ -6,7 +6,6 @@ const workflows = [
'.github/workflows/refresh-lockfile.yml',
'.github/workflows/pr-trusted.yml',
'.github/workflows/docker.yml',
'.github/workflows/docker-cloud.yml',
];
test('lockfile repair workflows resolve dependencies instead of updating metadata only', async () => {
.github/scripts/tests/post-merge-runner-routing.test.mjs
-4
View File
@@ -87,9 +87,7 @@ test("Cloud readiness bookkeeping never waits for the AWS verification fleet", (
const workflow = readFileSync(new URL("../../workflows/cloud-readiness.yml", import.meta.url), "utf8");
const bodies = new Map();
for (const [name, needs] of [
["artifacts", null],
["source_verified", "[verify]"],
["ready", "[verify, image, artifacts]"],
]) {
const body = workflow.match(new RegExp(`^ ${name}:\\n([\\s\\S]*?)(?=^ [a-z_]+:|(?![\\s\\S]))`, "m"))?.[1];
assert.ok(body, `missing ${name} job`);
@@ -100,8 +98,6 @@ test("Cloud readiness bookkeeping never waits for the AWS verification fleet", (
assert.match(body, /^ +SOURCE_SHA: \$\{\{ github.sha \}\}$/m);
assert.equal(body.match(/^ needs: (.+)$/m)?.[1] ?? null, needs, `${name} prerequisites`);
}
assert.match(bodies.get("artifacts"), /^ run: node scripts\/cloud-readiness.mjs "\$SOURCE_SHA"$/m);
assert.match(bodies.get("source_verified"), /^ run: node --test scripts\/cloud-source-verification.test.mjs$/m);
assert.match(bodies.get("source_verified"), /echo "Cloud source verified v1: \$SOURCE_SHA"/);
assert.match(bodies.get("ready"), /echo "Cloud deployable v1: \$SOURCE_SHA"/);
});
.github/workflows/cloud-readiness.yml
-50
View File
@@ -14,13 +14,6 @@ concurrency:
cancel-in-progress: false
jobs:
image:
if: github.repository == 'paperclipai/paperclip' && github.ref == 'refs/heads/master'
permissions:
contents: read
packages: write
uses: ./.github/workflows/docker-cloud.yml
verify:
if: github.repository == 'paperclipai/paperclip' && github.ref == 'refs/heads/master'
permissions:
@@ -29,29 +22,6 @@ jobs:
with:
ref: ${{ github.sha }}
artifacts:
if: github.repository == 'paperclipai/paperclip' && github.ref == 'refs/heads/master'
name: Wait for exact-source cloud artifacts
# Bookkeeping must not wait for the AWS builders it observes.
runs-on: ubuntu-latest
timeout-minutes: 35
permissions:
contents: read
actions: read
attestations: read
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
with:
persist-credentials: false
- uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7
with:
node-version: 24
- name: Wait for verified image and exact-source migrator
env:
GH_TOKEN: ${{ github.token }}
SOURCE_SHA: ${{ github.sha }}
run: node scripts/cloud-readiness.mjs "$SOURCE_SHA"
source_verified:
# npm canary publication reuses this exact-source verification proof.
# Keep it independent of image/migrator availability, and fail closed when
@@ -79,23 +49,3 @@ jobs:
SOURCE_SHA: ${{ github.sha }}
run: |
echo "Cloud source verified v1: $SOURCE_SHA" >> "$GITHUB_STEP_SUMMARY"
ready:
# Versioned consumer contract. Never add always() or continue-on-error:
# failed, cancelled, or skipped prerequisites must not report readiness.
name: Cloud deployable v1
needs: [verify, image, artifacts]
if: github.repository == 'paperclipai/paperclip' && github.ref == 'refs/heads/master'
# Bookkeeping must not wait for the AWS builders it observes.
runs-on: ubuntu-latest
timeout-minutes: 5
steps:
- name: Record cloud readiness
env:
SOURCE_SHA: ${{ github.sha }}
run: |
{
echo "Cloud deployable v1: $SOURCE_SHA"
echo "Source verification passed; the full-SHA image and exact-source migrator are available."
echo "Deployment tooling must still resolve and pin the image and migrator and validate migration compatibility."
} >> "$GITHUB_STEP_SUMMARY"
.github/workflows/docker-cloud.yml
-290
View File
@@ -1,290 +0,0 @@
name: Docker cloud
on:
workflow_dispatch:
workflow_call:
permissions: {}
# Independent SHAs can build immediately on separate runners.
# Repeated requests for the same source serialize without cancelling a build.
# No mutable canary channel is promoted here; docker.yml owns that operation.
concurrency:
group: docker-cloud-${{ github.sha }}
cancel-in-progress: false
jobs:
build-and-push-cloud:
# Only canonical master builds can consume the release Fleet. The runner
# group must also allow this workflow only at refs/heads/master.
# Keep an operator switch for a full-run retry on GitHub-hosted runners.
runs-on: ${{ vars.AWS_CLOUD_BUILDS_ENABLED == 'true' && github.repository == 'paperclipai/paperclip' && github.repository_id == '1170821064' && github.ref == 'refs/heads/master' && (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && 'runs-on/fleet=paperclip-cloud-build-x64/env=public-ci' || 'ubuntu-latest' }}
# Fleet instances expire after 45 minutes, including bootstrap and cleanup.
timeout-minutes: ${{ vars.AWS_CLOUD_BUILDS_ENABLED == 'true' && github.repository == 'paperclipai/paperclip' && github.repository_id == '1170821064' && github.ref == 'refs/heads/master' && (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && 40 || 60 }}
permissions:
contents: read
packages: write
steps:
- name: Checkout
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
with:
# Full history and tags so `git describe` below can compute the
# release version to stamp into the image.
fetch-depth: 0
# `.git` is dockerignored, so a running image cannot derive its own
# version and otherwise reports the source package.json placeholder in
# analytics and the debug panel. Compute it here from the pristine
# checkout (real CalVer drift from the nearest release tag) and pass it
# into the build. Empty when no release tag is reachable — the server
# then keeps its existing fallbacks.
- name: Compute build version
id: build-version
run: |
set -euo pipefail
case "${GITHUB_REF}" in
refs/tags/nightly/v*)
# Lane tags carry the exact published version; stamp it verbatim
# instead of describing drift from the nearest stable tag.
version="${GITHUB_REF#refs/tags/nightly/v}"
;;
refs/tags/beta/v*)
version="${GITHUB_REF#refs/tags/beta/v}"
;;
*)
version="$(git describe --tags --match 'v*' --long --dirty 2>/dev/null || true)"
;;
esac
echo "version=${version}" >> "$GITHUB_OUTPUT"
echo "Stamping build version: ${version:-<none>}"
# ISO week stamp for the Dockerfile's tool layer: the layer caches
# across commits and re-pulls the @latest CLI tools when the week rolls
# over, instead of on every build.
- name: Compute tool cache epoch
id: tools-epoch
run: echo "epoch=$(date -u +%G-W%V)" >> "$GITHUB_OUTPUT"
- name: Setup pnpm
uses: pnpm/action-setup@0977fd99725f1db4007ccb2928dbb4e90d06cc86 # v6
with:
version: 9.15.4
run_install: false
# No dependency cache here: this workflow publishes release images, and
# restoring a shared Actions cache into the build inputs would let a
# poisoned cache entry reach the published artifact.
- name: Setup Node.js
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7
with:
node-version: 24
- name: Refresh lockfile for Docker build context
run: |
set -euo pipefail
pnpm install --resolution-only --ignore-scripts --no-frozen-lockfile
changed="$(git status --porcelain)"
if [ -z "$changed" ]; then
echo "Lockfile already matches package metadata."
exit 0
fi
if printf '%s\n' "$changed" | grep -Fvq ' pnpm-lock.yaml'; then
echo "Unexpected files changed during lockfile refresh:"
echo "$changed"
exit 1
fi
echo "Using refreshed pnpm-lock.yaml in the Docker build context."
- name: Free runner disk
run: |
set -euo pipefail
echo "Disk before cleanup:"
df -h
# A measured hosted cloud build started with 86 GB available.
# Keep ample headroom for BuildKit and image verification, but
# avoid minutes deleting SDKs when neither filesystem needs space.
minimum_free_kib=$((64 * 1024 * 1024))
if docker_root="$(docker info --format '{{.DockerRootDir}}')" \
&& available_kib="$(df -Pk "$docker_root" "$GITHUB_WORKSPACE" | awk 'NR > 1 { rows++; if ($4 !~ /^[0-9]+$/) invalid = 1; if (min == "" || $4 < min) min = $4 } END { if (invalid || rows != 2) exit 1; print min }')" \
&& [[ "$available_kib" =~ ^[0-9]+$ ]] \
&& (( available_kib >= minimum_free_kib )); then
echo "At least 64 GiB is available for Docker and the workspace; skipping cleanup."
exit 0
fi
pnpm store prune || true
sudo apt-get clean || true
sudo rm -rf \
/usr/share/dotnet \
/usr/share/swift \
/usr/local/lib/android \
/usr/local/share/boost \
/usr/local/share/powershell \
/opt/ghc \
/opt/hostedtoolcache/CodeQL \
/opt/hostedtoolcache/PyPy \
/opt/hostedtoolcache/Ruby || true
docker system prune -af || true
echo "Disk after cleanup:"
df -h
- name: Login to GitHub Container Registry
uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4
with:
registry: ghcr.io
username: ${{ github.repository_owner }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@37fe631027851001ddb9b187196cc803df7f5f0e # v4
# Mixing several historical manifests missed otherwise reusable native
# layers on fresh builders. Import the nearest available complete cache.
- name: Select cloud cache ancestry
id: cloud-cache
env:
CACHE_IMAGE: ghcr.io/${{ github.repository }}
run: node scripts/select-cloud-cache.mjs
# Deployment tooling reads these labels from the registry to verify an
# image's schema expectations against a migrator before deploying it,
# without pulling the image. The server refuses to start when the
# database is missing bundled migrations, so orchestrators need a cheap
# way to check image/migrator compatibility up front.
- name: Compute schema migration labels
id: schema
run: |
set -euo pipefail
last=$(ls packages/db/src/migrations/*.sql | sed 's|.*/||' | LC_ALL=C sort | tail -1)
count=$(ls packages/db/src/migrations/*.sql | wc -l | tr -d ' ')
echo "last=${last}" >> "$GITHUB_OUTPUT"
echo "count=${count}" >> "$GITHUB_OUTPUT"
# Published under the same lane tag set as the self-hosted image, with a
# `-cloud` suffix (nightly-cloud, latest-cloud, <version>-cloud,
# sha-<short>-cloud). `:canary-cloud` follows the same retag-step
# ownership rule as `:canary` above.
- name: Docker meta (cloud)
id: meta-cloud
uses: docker/metadata-action@dc802804100637a589fabce1cb79ff13a1411302 # v6
with:
images: ghcr.io/${{ github.repository }}
flavor: |
suffix=-cloud,onlatest=true
tags: |
type=raw,value=nightly,enable=${{ startsWith(github.ref, 'refs/tags/nightly/v') }}
type=raw,value=beta,enable=${{ startsWith(github.ref, 'refs/tags/beta/v') }}
type=raw,value=latest,enable=${{ startsWith(github.ref, 'refs/tags/v') }}
type=semver,pattern={{version}},enable=${{ startsWith(github.ref, 'refs/tags/v') }}
type=semver,pattern={{major}}.{{minor}},enable=${{ startsWith(github.ref, 'refs/tags/v') }}
type=sha
labels: |
io.github.paperclipai.schema.last-migration=${{ steps.schema.outputs.last }}
io.github.paperclipai.schema.migration-count=${{ steps.schema.outputs.count }}
- name: Build and push (cloud)
id: build-cloud
uses: docker/build-push-action@53b7df96c91f9c12dcc8a07bcb9ccacbed38856a # v7
with:
context: .
target: cloud
# Space-separated sandbox-provider directory names to build into
# the variant; add here when managed deployments need another.
# CLOUD_BUNDLED_SERVER_DEPS names the optional peer packages the
# variant installs from server/package.json's declared version;
# add another name there when a managed tenant needs it.
build-args: |
USER_UID=1001
USER_GID=1001
CLOUD_BUNDLED_PLUGINS=daytona
CLOUD_BUNDLED_SERVER_DEPS=@sentry/node
PAPERCLIP_BUILD_VERSION=${{ steps.build-version.outputs.version }}
PAPERCLIP_BUILD_COMMIT=${{ github.sha }}
CLI_TOOLS_CACHE_EPOCH=${{ steps.tools-epoch.outputs.epoch }}
# amd64 only, unlike the self-hosted image above: the cloud variant
# is consumed exclusively by managed-deployment hosts, which run
# amd64. The QEMU-emulated arm64 half dominated this job's wall
# clock, and dropping it roughly halves time-to-deployable-image.
platforms: linux/amd64
push: true
# Same-SHA builds serialize above; different SHAs never share a
# writable cache ref. Registry layers are content-addressed and
# shared even when cache manifests have separate tags.
cache-from: ${{ steps.cloud-cache.outputs.source }}
cache-to: type=registry,ref=ghcr.io/${{ github.repository }}:buildcache-cloud-${{ github.sha }},mode=max
tags: ${{ steps.meta-cloud.outputs.tags }}
labels: ${{ steps.meta-cloud.outputs.labels }}
# The cloud target installs @sentry/node at the version
# server/package.json declares, into a directory the server's own
# module resolution walks. Verify the image this job just pushed, not
# a local build, so a build-cache or layer-ordering regression is
# caught before any tenant runs the image.
- name: Verify the pushed image resolves the declared Sentry version
env:
IMAGE: ghcr.io/${{ github.repository }}@${{ steps.build-cloud.outputs.digest }}
run: |
set -euo pipefail
expected="$(node -e "process.stdout.write(require('./server/package.json').peerDependencies['@sentry/node'])")"
test -n "$expected"
installed="$(docker run --rm --pull always \
-v "$PWD/scripts/assert-cloud-image-sentry.mjs:/app/server/.ci-sentry-probe.mjs:ro" \
--entrypoint node "$IMAGE" /app/server/.ci-sentry-probe.mjs)"
echo "Declared optional peer version: $expected"
echo "Installed in the pushed image: $installed"
if [ "$installed" != "$expected" ]; then
echo "ERROR: the pushed image resolves @sentry/node@$installed, expected @sentry/node@$expected" >&2
exit 1
fi
echo "The pushed image resolves the declared @sentry/node version."
# Managed hosts run node as 1001:1001. Bake that identity into the image
# so usermod does not walk the mounted home on every container start.
# Check before the entrypoint can repair a wrongly built identity.
- name: Verify cloud runtime user
env:
IMAGE: ghcr.io/${{ github.repository }}@${{ steps.build-cloud.outputs.digest }}
run: |
set -euo pipefail
docker run --rm --entrypoint sh "$IMAGE" -ec '
test "$(id -u node)" = 1001
test "$(id -g node)" = 1001
test "$USER_UID" = 1001
test "$USER_GID" = 1001
'
docker run --rm -e USER_UID=1001 -e USER_GID=1001 "$IMAGE" sh -ec '
test "$(id -u)" = 1001
test "$(id -g)" = 1001
test -w "$PAPERCLIP_HOME"
'
# Verify the independently published cloud image without waiting for
# the self-hosted manifest job. The Sentry check already pulled it.
- name: Verify cloud PID 1 reaps orphaned processes
env:
IMAGE: ghcr.io/${{ github.repository }}@${{ steps.build-cloud.outputs.digest }}
run: docker run --rm -i "$IMAGE" sh -s < scripts/assert-orphan-reaping.sh
# Cloud's commit resolver and preview-artifact planner use the full SHA.
# Publish that address only after checking this build's exact digest.
# Retagging reuses the registry manifest and does not rebuild the image.
- name: Publish verified full-SHA cloud tag
env:
IMAGE: ghcr.io/${{ github.repository }}@${{ steps.build-cloud.outputs.digest }}
FULL_SHA_TAG: ghcr.io/${{ github.repository }}:sha-${{ github.sha }}-cloud
run: |
set -euo pipefail
revision="$(docker image inspect "$IMAGE" --format '{{ index .Config.Labels "org.opencontainers.image.revision" }}')"
platform="$(docker image inspect "$IMAGE" --format '{{ .Os }}/{{ .Architecture }}')"
test "$revision" = "$GITHUB_SHA"
test "$platform" = linux/amd64
docker buildx imagetools create --prefer-index=false --tag "$FULL_SHA_TAG" "$IMAGE"
.github/workflows/docker.yml
+6 -18
View File
@@ -53,9 +53,8 @@ jobs:
# ten master commits sampled that day never produced an image at all.
#
# Each platform now builds on a runner of its own architecture and pushes by
# digest; `merge` assembles the manifest list. arm64 is kept rather than
# dropped (the cloud variant below dropped it and is amd64-only) because
# this is the self-hosted image, and ARM hosts consume it.
# digest; `merge` assembles the manifest list. Both architectures remain
# available to self-hosted installations and downstream image composers.
build-and-push:
strategy:
# Independent legs: one architecture failing should still publish
@@ -353,7 +352,7 @@ jobs:
# until the cgroup pid limit is exhausted and every fork() in the
# container fails. Run against the pushed manifest rather than a local
# build: the legs push by digest, so nothing is loaded into this
# runner's daemon. The independent cloud workflow checks its own image.
# runner's daemon.
- name: Verify PID 1 reaps orphaned processes
env:
# Through the environment, not interpolated into the script body, so
@@ -387,17 +386,7 @@ jobs:
subject-digest: ${{ steps.standard.outputs.digest }}
push-to-registry: true
# Master cloud builds start independently in docker-cloud.yml. Tag builds
# and manual Docker dispatches call the same implementation, preserving the
# release tags and the canary promotion dependency below.
build-and-push-cloud:
if: github.event_name != 'push' || github.ref != 'refs/heads/master'
uses: ./.github/workflows/docker-cloud.yml
permissions:
contents: read
packages: write
# Moves the mutable `:canary` / `:canary-cloud` channel tags. Kept OUT
# Moves the mutable `:canary` channel tag. Kept OUT
# of the build jobs and serialized in its own lane, and — the load-
# bearing property — CONVERGENT rather than self-interested: a
# promotion does not promote "its own" canary, it retags the channel
@@ -418,7 +407,7 @@ jobs:
# merge-and-push, not build-and-push: the per-arch legs push untagged
# digests, and the production `sha-*` tags this promotion retags only
# exist once the manifest merge has named them.
needs: [merge-and-push, build-and-push-cloud]
needs: [merge-and-push]
runs-on: ubuntu-latest
timeout-minutes: 10
permissions:
@@ -447,10 +436,9 @@ jobs:
exit 0
fi
short="$(printf '%s' "$sha" | cut -c1-7)"
if ! docker buildx imagetools inspect "$IMAGE:sha-${short}-cloud" >/dev/null 2>&1; then
if ! docker buildx imagetools inspect "$IMAGE:sha-${short}" >/dev/null 2>&1; then
echo "images for ${current} (sha-${short}) not published yet; its own promotion converges the channel"
exit 0
fi
docker buildx imagetools create -t "$IMAGE:canary" "$IMAGE:sha-${short}"
docker buildx imagetools create -t "$IMAGE:canary-cloud" "$IMAGE:sha-${short}-cloud"
echo "channel tags moved to canary ${current} (sha-${short})"
.github/workflows/release.yml
+1 -1
View File
@@ -417,7 +417,7 @@ jobs:
# explicitly, exactly like the nightly and beta lanes: the run keys
# its concurrency off the tag ref, so no master push can supersede
# it, and docker.yml's `type=sha` mapping publishes the
# sha-<short> and sha-<short>-cloud images either way.
# standard sha-<short> images either way.
- name: Build Docker images for the canary tag
env:
GH_TOKEN: ${{ github.token }}
doc/DOCKER.md
+19 -30
View File
@@ -32,40 +32,29 @@ docker build -t paperclip-local \
--build-arg USER_UID=$(id -u) --build-arg USER_GID=$(id -g) .
```
## Cloud image addresses
## Standard images and downstream composition
The Docker workflow publishes the managed deployment image for Linux AMD64.
`Cloud readiness` starts `Docker cloud` on each master push independently of the
multi-platform self-hosted build. Different commits use separate concurrency groups and existing
GitHub-hosted runners, so an older production or cloud build does not hold the
new commit in a workflow queue. Available GitHub runner capacity still applies.
Release tags and manual `Docker` dispatches call the same cloud build workflow.
The Docker workflow publishes the standard `production` target for Linux AMD64
and ARM64. Canonical master pushes also publish
`ghcr.io/paperclipai/paperclip:sha-<FULL_SHA>` and a GitHub/Sigstore attestation
for its immutable multi-platform digest. Downstream services can compose their
own images from this public base without rebuilding Core.
Each commit exports to its own `buildcache-cloud-<FULL_SHA>` registry tag.
Builds import the current commit and nine first-parent ancestors, plus the
legacy `buildcache-cloud` fallback. This preserves reusable layers without
letting concurrent builds overwrite one shared cache manifest. Retain recent
cache tags if registry cleanup is configured; deleting them makes builds colder.
The legacy recurring public `-cloud` publisher is retired. Master pushes,
release tags, and manual `Docker` dispatches no longer build that variant.
Existing `-cloud` tags and digests remain in the registry for rollback; their
release-channel aliases no longer advance. This change deletes no images,
cache tags, or migrator artifacts.
Cloud CI skips SDK and cache cleanup when both the Docker data filesystem and
the checkout filesystem have at least 64 GiB available. Below that conservative
headroom threshold, or when the measurement fails, it retains the existing
cleanup. The threshold selects the fast path; it is not a new minimum disk
requirement for local builds or smaller runners.
The `cloud` Dockerfile target remains available for explicit
[preview builds](preview-release-artifacts.md). Those requests still publish a
full-SHA `-cloud` tag when needed. They do not advance a release channel or
replace downstream private composition.
After the pushed image passes its Sentry and orphan-reaping checks, the workflow verifies its
commit label and platform and adds `ghcr.io/paperclipai/paperclip:sha-<full-commit-sha>-cloud`.
This address lets commit-based deployment tooling reuse the normal build.
Existing short-SHA and release tags remain available.
The full-SHA tag identifies the source commit. It does not certify that source
tests passed or that a compatible database migrator is available. Deployment
tooling must still check those prerequisites and pin the resolved image digest;
a rebuild of the same source can update the tag's digest.
The separate [cloud readiness check](cloud-build-readiness.md) combines source
verification, successful cloud image checks, and exact-source migrator
availability. It runs outside the full npm release's concurrency queue.
A published image alone does not prove source tests or migration compatibility.
Downstream deployment tooling must verify [source proof](cloud-build-readiness.md),
the standard image attestation, the exact-source migrator, and its own composed
image before rollout. Resolve immutable digests instead of deploying mutable tags.
## One-liner (build + run)
doc/RELEASE-CHECKLIST.md
+2 -2
View File
@@ -19,7 +19,7 @@ The release captain's checklist for every lane. The mechanics live in
- [ ] the release smoke suite passed against the exact candidate canary
before anything published
- [ ] `npm view paperclipai@nightly version` shows the new `-nightly.N`
- [ ] `nightly/v*` tag pushed; `:nightly` and `:nightly-cloud` images built
- [ ] `nightly/v*` tag pushed; `:nightly` image built
- [ ] on a tag-push rejection (workflows-permission error), follow the
recovery commands in the job summary
@@ -34,7 +34,7 @@ Happy path:
- [ ] dispatch `release.yml` with `channel: beta`
- [ ] approve the `npm-beta` environment gate
- [ ] `npm view paperclipai@beta version` shows the new `-beta.N`
- [ ] `beta/v*` tag pushed; `:beta` and `:beta-cloud` images built
- [ ] `beta/v*` tag pushed; `:beta` image built
- [ ] post-publish smoke (`smoke_beta`) is green
- [ ] `draft_stable_notes` pushed `release-notes/v<beta-version>`; open the
notes PR from the job-summary link
doc/cloud-build-readiness.md
+53 -96
View File
@@ -1,38 +1,25 @@
# Cloud build readiness
The `Cloud readiness` workflow starts for every master push. Its versioned
`Cloud deployable v1` job succeeds only after all three prerequisites succeed:
The `Cloud readiness` workflow starts for every master push and retains the
versioned `Cloud source verified v1` job. It calls the full `Release Verify`
workflow for that exact commit, including typecheck, builds, general and
serialized tests, and Runner verification. The source proof depends on every
source check and fails closed if verification fails, is cancelled, or is skipped.
- The existing `Release Verify` workflow checks that exact commit, including
typecheck, builds, general and serialized tests, and Runner verification.
- The reusable `Docker cloud` workflow builds and verifies its Linux AMD64
image, including Sentry resolution and orphan reaping, then publishes the
full-SHA cloud tag. Cloud readiness owns the master trigger so there is one
cloud build per push. Release tags and manual Docker runs retain their callers.
- The full-SHA image is visible and the exact-source `Cloud migrator artifacts`
workflow has succeeded. Readiness verifies the manifest's GitHub attestation
against the full SHA, canonical master workflow, and GitHub-hosted runner,
then downloads and validates both package archives and the prepared dependency
lockfile. The database package pins the matching shared package. New-version
npm metadata and tarball propagation are outside this path.
The recurring public `-cloud` publisher and its `Cloud deployable v1` gate are
retired. The workflow no longer builds a legacy image or waits for one.
Keep its filename and source-proof job name stable: npm canary publication and
downstream image composers consume that exact contract.
The Cloud workflow builds the image with `USER_UID=1001` and `USER_GID=1001`,
matching the managed runtime. This avoids a startup user remap, which can walk
the mounted home and delay health checks. Before publishing the full-SHA tag,
the workflow checks the baked identity without running the entrypoint, then
checks the normal entrypoint's effective user and writable home. Volume ownership
repair still runs when needed. The Dockerfile defaults remain `1000:1000` for
self-hosted builds, and runtime identity overrides remain supported. The first
build with the new identity must rebuild layers that depend on the base image;
later builds can reuse those layers.
Standard images still publish independently through `docker.yml`. The
`cloud-migrator-artifacts.yml` workflow still publishes signed exact-source
migrators independently. A downstream composer must verify those artifacts,
build and test its own image, and record separate deployment readiness.
Verification and image building run concurrently, outside the full npm release's
concurrency group. Different commits have independent groups. The npm canary
release reuses `Cloud source verified v1` for the exact master push instead of
starting a second copy of `Release Verify`. This source-only job depends on every
source check but does not wait for Docker or migrator publication. npm canary
publication remains possible when source verification passes and an image build
fails. Stable releases and candidate-branch betas still run full verification.
Verification runs outside the full npm release's concurrency group. Different
commits have independent groups. The npm canary release reuses the source proof
for its exact master push instead of starting another `Release Verify` run.
Stable releases and candidate-branch betas still run full verification.
The canary consumer requires the expected workflow ID and path, upstream source
repository, master push event, full SHA, and a successful job in the latest run
@@ -57,41 +44,36 @@ before that bot's PR merges. Verification must install and test that commit
without waiting for another merge. The generated lockfile stays in the job's
workspace; these checks do not commit it back to the repository.
The artifact wait runs for up to 30 minutes and reports what is missing. A
missing image or an exact-source publisher with no successful run yet means publication
is pending. An earlier successful push or manual run remains valid after a failed
retry because publication is immutable. If all matching runs failed, readiness
fails. An invalid signature, inaccessible or corrupt
bundle, authorization error, or identity mismatch fails the job. A failed, cancelled, or skipped prerequisite
cannot produce a successful readiness job. Retry the failed publication or build,
then rerun the failed readiness workflow jobs to check the same commit again.
## Consumer contract and retirement boundary
## Consumer contract
Accept `Cloud source verified v1` only from the latest attempt of the canonical
`cloud-readiness.yml` master push for the expected repository identity and full
source SHA. Check the job itself and reject failed, skipped, cancelled, or
ambiguous proof. This signal verifies source only. It creates no release record,
certifies no composed image, and deploys no instance.
`Cloud deployable v1` is a source-and-artifact readiness signal. A deployment
consumer must still resolve and pin the image digest and migrator integrity/lockfile,
validate migration contents and compatibility, and apply its target health gates.
The check creates no release record and deploys no instance. A full-SHA tag by
itself, or a successful migrator dispatch, is not this readiness signal.
Downstream deployment consumers must separately verify the standard image's
immutable digest and attestation, the exact-source migrator's signature and
integrity, migration compatibility, their own image composition, and target
health. Order automatic candidates by master ancestry, not completion time.
For automatic selection, accept only a successful job named exactly
`Cloud deployable v1` in the latest attempt of a successful
`.github/workflows/cloud-readiness.yml` run in `paperclipai/paperclip`, with
event `push`, head branch `master`, and the expected full head SHA and repository.
Do not trust a similarly named check from another workflow or a manual branch run.
Order candidates by master ancestry, not job completion time: an older commit
finishing late must not roll a fleet backward. Fail closed on API errors.
Merge this retirement only after all active automatic deployment consumers use
the standard-image composition contract. A consumer still selecting
`Cloud deployable v1` will stop advancing at the last legacy-ready commit.
Do not rename the source proof to the old readiness name or weaken a consumer
check to hide that dependency.
Cloud consumers must enable `CLOUD_HARNESS_DIRECT_MIGRATOR_ARTIFACTS` before
this gate is adopted: readiness no longer promises preview npm availability.
The automatic npm-only migrator dispatcher has been removed. Manual
`release.yml` runs with `channel=cloud-migrator`, branch previews, and stable
releases retain their npm publisher for legacy consumers and rollback.
Existing release records, immutable image digests, migrator artifacts, and
registry tags are retained for rollback. No registry deletion or live deployment
is part of this change. Explicit `release.yml` preview requests still use the
legacy `cloud` Dockerfile target for a specified source commit. They do not
restart recurring legacy publication. Keep that compatibility path until its
operator consumers migrate separately.
For rollback, restore the npm dispatcher and gate together before disabling the
cloud direct-artifact switch. Already-created releases retain their immutable
archive URLs and lockfiles; keep those objects available. The master producer
can be retried independently without republishing or overwriting a valid bundle.
The old `nightly-cloud`, `beta-cloud`, `latest-cloud`, and `canary-cloud` aliases
stop advancing. Self-hosted standard release aliases continue unchanged. A
rollback to an already published image needs no rebuild; restoring recurring
legacy publication would require reverting the publisher retirement.
## Timing and rollout
@@ -107,7 +89,8 @@ exact commit. Keep readiness and deployment as separate milestones:
| --- | --- | --- |
| Merge | Merged PR timestamp and full merge commit SHA | Merge |
| Image available | Successful full-SHA image publication and verification | Merge |
| Cloud deployable | Successful `Cloud deployable v1` job in the accepted push run and attempt | Merge |
| Source verified | Successful `Cloud source verified v1` job in the accepted push run and attempt | Merge |
| Composed image ready | Downstream composition verification and publication succeed | Merge |
| Canary healthy | Deployment consumer's canary health gate confirms the target commit | Merge |
| Fleet complete | Campaign succeeds for all eligible targets at that commit | Merge |
@@ -118,7 +101,7 @@ does not measure automatic merge-to-deploy latency. A preparation-only run
resolves artifacts without deploying a target and must not be counted as a
successful deployment.
Record queue time and the image, source-verification, and artifact-wait durations
Record queue time and the image, source-verification, migrator, and composition durations
separately. The slowest prerequisite determines readiness; shortening an already
faster prerequisite may have no effect on the total. After readiness, measure
consumer discovery delay, artifact resolution, canary health, and fleet rollout.
@@ -132,23 +115,14 @@ excluded or sleeping targets, retries, and failures with the fleet result. Recor
runner queue conditions and cache state; one warm or cold run is a sample, not a
latency guarantee.
Land full-SHA image publication, independent cloud builds, and migrator-only
publication before enabling this workflow. Until those producers are present,
the artifact wait cannot succeed. A manual dispatch on master can verify the
wiring, but automatic consumers should use push runs. Source verification and
registry checks can be rerun without deploying or changing mutable npm channels.
When reverting this workflow, restore the master push trigger in
`docker-cloud.yml` in the same change so master images continue to build.
## Reserved AWS verification capacity
`AWS_POST_MERGE_CI_ENABLED=true` routes cloud source verification, artifact
waiting, readiness signals, and exact-master migrator preparation to the
`AWS_POST_MERGE_CI_ENABLED=true` routes cloud source verification and
exact-master migrator preparation to the
`paperclip-post-merge` runner group. The separate Fleet label is
`runs-on/fleet=paperclip-post-merge-x64/env=public-ci`. Its 36 reserved slots use
the same four-vCPU, 16-GiB machines as approved PR jobs. PR capacity is reduced
to 64; image capacity stays at eight. The total ceiling remains 108 runners.
to 64; the separately provisioned image capacity is unchanged by this retirement.
This keeps PR bursts from consuming every post-merge verification slot.
Every selector checks the canonical repository name and ID, master ref, and a
@@ -172,29 +146,12 @@ Disable the switch and rerun the whole workflow to restore GitHub-hosted
placement. Assigned jobs keep their original runners. Readiness requirements,
source checks, and npm integrity checks are unchanged.
## AWS cloud build routing
`AWS_CLOUD_BUILDS_ENABLED=true` routes the Docker cloud job to the
`paperclip-cloud-build-x64` RunsOn Fleet for canonical `paperclipai/paperclip`
master pushes and manual master runs. Forks, pull requests, and release tags
retain GitHub-hosted runners. The separate `AWS_CI_ENABLED` and
`AWS_CI_TRUSTED_USER_IDS` variables control PR routing.
The cloud Fleet uses a separate runner group, `paperclip-cloud-build`, restricted
to this repository and `.github/workflows/docker-cloud.yml@refs/heads/master`.
Provision that group and Fleet before enabling the variable. The cloud runners
need at least 64 GiB free for Docker and the workspace; the initial configuration
uses 120 GiB disks with the existing 4-vCPU, 16-GiB machine size. AWS jobs have
a 40-minute workflow timeout so they finish before the 45-minute instance
lifetime; GitHub-hosted jobs retain their 60-minute timeout. Keep the registry
cache and all pushed-image verification steps enabled.
To roll back routing, set `AWS_CLOUD_BUILDS_ENABLED=false`, then rerun the cloud
workflow. Changing the variable does not migrate an already assigned job.
Check the Actions job's runner name and runner group to verify placement. Record
queue time, image verification completion, and `Cloud deployable v1` separately;
source verification and the migrator still run on GitHub-hosted runners.
## Retired AWS cloud build routing
`AWS_CLOUD_BUILDS_ENABLED` and the `paperclip-cloud-build` runner group no longer
route a public image job after this retirement. This source change does not
delete runner groups, Fleets, credentials, registry images, or cache tags. Review
shared infrastructure ownership separately before removing those resources.
### Typecheck Rust dependency cache
doc/preview-release-artifacts.md
+20 -25
View File
@@ -21,6 +21,9 @@ definitions that do not run from `master`.
## Outputs and reuse
The image uses `ghcr.io/paperclipai/paperclip:sha-<FULL_SHA>-cloud`.
This explicit operator path is retained after retirement of the recurring public
`-cloud` publisher. Existing images remain reusable; missing images still build
the `cloud` Dockerfile target. It is separate from private image composition.
Full-SHA tags keep separate commits with the same short prefix isolated. Normal
release images retain their existing short-tag convention. Build arguments carry the full commit SHA.
Preview builds do not import or overwrite the shared release cache or release
@@ -51,26 +54,18 @@ version 1, request ID, SHA, stage `build`, and status `ready`. It expires after
### Migrator publication on merge
The `Cloud artifacts` workflow starts a `cloud-migrator` dispatch of `release.yml`
for every push to `master`. This dispatch builds and publishes only the exact-source
`@paperclipai/shared` and `@paperclipai/db` preview packages. It starts independently
of the full npm release and does not wait for the Docker image. The normal Docker
workflow supplies the image separately.
`cloud-migrator-artifacts.yml` publishes a signed exact-source DB/shared bundle
on each canonical master push, independently of image publication and npm.
See [Direct cloud migrator artifacts](#direct-cloud-migrator-artifacts) below.
Downstream deployment tooling must verify source, image, and migrator separately.
The run title is `Cloud migrator <FULL_SHA>`. A successful `Cloud artifacts`
dispatch job only confirms that GitHub accepted the request. Inspect the matching
`release.yml` run to confirm publication completed. This path does not produce a
`stack-deploy-result` or certify source-test success or deployment readiness.
Cloud must still verify all deployment prerequisites.
To retry one commit, dispatch `release.yml` on `master` with `channel=cloud-migrator`,
the full SHA as `source_ref`, a new UUID v4 as `request_id`, and `dry_run=false`.
`preview_migrator` is not required for this channel. Existing packages are verified
and reused. Preview and migrator-only runs use separate workflow concurrency
groups. Only their package publication jobs share a group for the same SHA, so
they cannot publish the same version concurrently and the migrator does not wait
for a preview's image build. Different SHAs publish in separate groups; the full
release keeps its existing group.
The manual npm compatibility path remains available: dispatch `release.yml` on
`master` with `channel=cloud-migrator`, the full SHA as `source_ref`, a new UUID v4
as `request_id`, and `dry_run=false`. `preview_migrator` is not required for this
channel. Existing packages are verified and reused. Preview and migrator-only
runs use separate workflow concurrency groups. Only their npm publication jobs
share a group for the same SHA. This prevents duplicate publication without
making the migrator wait for a preview image. Different SHAs remain independent.
### Publisher identity
@@ -170,12 +165,12 @@ The deploy policies are checked in under `.github/cloud-migrator-deploy/`:
There is no lifecycle expiry on this prefix. Keep referenced artifacts for
rollback; deleting them can prevent a fresh migrator install for an old release.
Cloud readiness consumes the signed direct bundle after its exact-source
publisher succeeds. It retains source verification, image identity, and the
cloud runner's integrity and migration compatibility checks. The cloud direct
artifact switch must be enabled before adopting this gate. Automatic npm-only
migrator dispatch is removed; explicit npm previews and manual migrator runs
remain available. See `doc/cloud-build-readiness.md` for coordinated rollback.
Downstream deployment consumers verify the signed direct bundle after its
exact-source publisher succeeds. Source verification, image identity, archive
integrity and migration compatibility remain required. The retired public
`Cloud deployable v1` gate no longer waits for this bundle. Explicit npm previews
and manual migrator runs remain available for compatible consumers and rollback.
See `doc/cloud-build-readiness.md` for the source proof and retirement boundary.
Local verification:
package.json
+1 -1
View File
@@ -59,7 +59,7 @@
"smoke:posthog-live": "node scripts/smoke/posthog-live.mjs",
"smoke:pipelines-tutorial": "./scripts/smoke/pipelines-tutorial-smoke.sh",
"smoke:terminal-bench-loop-skill": "node scripts/smoke/terminal-bench-loop-skill-smoke.mjs",
"test:release-registry": "node --test scripts/verify-release-registry-state.test.mjs scripts/release-package-map.test.mjs scripts/check-release-package-bootstrap.test.mjs scripts/check-no-git-push.test.mjs scripts/release-lib.test.mjs scripts/release-registry-versions.test.mjs scripts/link-plugin-dev-sdk.test.js scripts/acpx-patch-packaging.test.mjs scripts/service-onboard-smoke.test.mjs scripts/docker-onboard-smoke.test.mjs scripts/preview-artifacts.test.mjs scripts/cloud-migrator-artifacts.test.mjs scripts/select-cloud-cache.test.mjs",
"test:release-registry": "node --test scripts/verify-release-registry-state.test.mjs scripts/release-package-map.test.mjs scripts/check-release-package-bootstrap.test.mjs scripts/check-no-git-push.test.mjs scripts/release-lib.test.mjs scripts/release-registry-versions.test.mjs scripts/link-plugin-dev-sdk.test.js scripts/acpx-patch-packaging.test.mjs scripts/service-onboard-smoke.test.mjs scripts/docker-onboard-smoke.test.mjs scripts/preview-artifacts.test.mjs scripts/cloud-migrator-artifacts.test.mjs",
"storybook-visual:baseline": "node scripts/storybook-visual-baseline.mjs",
"test:storybook-visual": "node scripts/storybook-visual-baseline.mjs download && node scripts/storybook-visual-baseline.mjs verify && pnpm build-storybook && npx playwright test --config tests/storybook-visual/playwright.config.ts",
"test:storybook-visual:update": "node scripts/storybook-visual-baseline.mjs download && pnpm build-storybook && npx playwright test --config tests/storybook-visual/playwright.config.ts --update-snapshots && node scripts/storybook-visual-baseline.mjs pack",
scripts/__tests__/release-verify-workflow.test.mjs
+2 -2
View File
@@ -63,13 +63,13 @@ test("canary reuses exact-source proof while stable keeps full verification", ()
test("source proof requires every source check and does not wait on image publication", () => {
const readiness = readWorkflow("cloud-readiness.yml");
const proof = readiness.split(" source_verified:\n")[1].split("\n ready:")[0];
const proof = readiness.split(" source_verified:\n")[1];
assert.match(proof, /name: Cloud source verified v1/);
assert.match(proof, /needs: \[verify\]/);
assert.match(proof, /node --test scripts\/cloud-source-verification.test.mjs/);
assert.match(proof, /SOURCE_SHA: \$\{\{ github\.sha \}\}/);
assert.doesNotMatch(proof, /always\(\)|continue-on-error|needs:.*(?:image|artifacts)/);
assert.match(readiness.split(" ready:\n")[1], /needs: \[verify, image, artifacts\]/);
assert.doesNotMatch(readiness, /^ (?:image|artifacts|ready):/m);
});
test("onboard smoke container binds beyond loopback so the mapped port is reachable", () => {
scripts/cloud-readiness.mjs
-102
View File
@@ -1,102 +0,0 @@
#!/usr/bin/env node
import { pathToFileURL } from "node:url";
import { execFileSync } from "node:child_process";
import { mkdtempSync, rmSync, writeFileSync } from "node:fs";
import os from "node:os";
import path from "node:path";
import { imageExists, versionFor } from "./preview-artifacts.mjs";
import { verifyPublished } from "./cloud-migrator-artifacts.mjs";
const repository = "paperclipai/paperclip";
const workflow = ".github/workflows/cloud-migrator-artifacts.yml";
export async function migratorPublished(sha, fetchImpl, token) {
let pending = false;
const failures = [];
for (let page = 1; page <= 10; page++) {
const response = await fetchImpl(`https://api.github.com/repos/${repository}/actions/workflows/cloud-migrator-artifacts.yml/runs?branch=master&head_sha=${sha}&per_page=100&page=${page}`, {
headers: { Accept: "application/vnd.github+json", ...(token ? { Authorization: `Bearer ${token}` } : {}) },
redirect: "error", signal: AbortSignal.timeout(30_000),
});
if (!response.ok) throw new Error(`Migrator producer lookup failed: HTTP ${response.status}`);
const body = await response.json();
if (!Array.isArray(body.workflow_runs) || !Number.isSafeInteger(body.total_count) || body.total_count < 0 ||
(page === 1 && (body.total_count === 0) !== (body.workflow_runs.length === 0))) throw new Error("Invalid migrator producer response.");
if (body.total_count === 0) return false;
for (const run of body.workflow_runs) {
if (run.head_sha !== sha || run.head_branch !== "master" || run.path !== workflow ||
run.head_repository?.id !== 1170821064 || run.head_repository.full_name !== repository ||
!["push", "workflow_dispatch"].includes(run.event)) throw new Error("Migrator producer identity mismatch.");
// Publication is immutable. A later failed manual run must not hide a
// successful exact-source publisher; the signed bundle is checked next.
if (run.status === "completed" && run.conclusion === "success") return true;
if (run.status !== "completed") pending = true;
else failures.push(`${run.id}: ${run.conclusion}`);
}
if (page * 100 >= body.total_count) {
if (pending) return false;
throw new Error(`Migrator producers failed: ${failures.join(", ")}.`);
}
}
throw new Error("Too many migrator producer runs to establish publication.");
}
export function verifyManifestProvenance(bytes, sha, { exec = execFileSync } = {}) {
versionFor(sha);
const scratch = mkdtempSync(path.join(os.tmpdir(), "cloud-readiness-attestation-"));
try {
const file = path.join(scratch, "manifest.json");
writeFileSync(file, bytes);
exec("gh", ["attestation", "verify", file, "--repo", repository,
"--source-digest", sha, "--source-ref", "refs/heads/master",
"--cert-identity", `https://github.com/${repository}/${workflow}@refs/heads/master`,
"--deny-self-hosted-runners"], { stdio: "inherit", timeout: 60_000 });
} finally { rmSync(scratch, { recursive: true, force: true }); }
}
/** Read-only availability gate. Deployment still resolves and pins artifacts. */
export async function waitForCloudArtifacts(sha, {
fetchImpl = fetch,
token = process.env.GH_TOKEN,
verifyProvenance = verifyManifestProvenance,
now = () => performance.now(),
sleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms)),
timeoutMs = 30 * 60_000,
intervalMs = 20_000,
log = console.log,
} = {}) {
const version = versionFor(sha);
if (!Number.isFinite(timeoutMs) || timeoutMs <= 0 || !Number.isFinite(intervalMs) || intervalMs <= 0) {
throw new Error("Cloud readiness requires positive finite timeout and poll interval.");
}
const deadline = now() + timeoutMs;
let previous;
let missing = ["image", "migrator"];
while (now() < deadline) {
// Recheck the image and exact-source publisher on the successful poll.
// Only a missing/pending producer waits; failed publication fails closed.
const results = await Promise.all([
imageExists(sha, fetchImpl),
migratorPublished(sha, fetchImpl, token),
]);
missing = ["image", "migrator"].filter((_, index) => !results[index]);
if (missing.length === 0) {
// Verify the exact signed bytes and all pinned downloads after the
// publisher succeeds. An inaccessible or corrupt artifact cannot pass.
await verifyPublished(sha, fetchImpl, { verifyProvenance });
log(`Cloud artifacts available for ${sha}: verified image and exact-source migrator ${version}.`);
return { version: 1, sha, packageVersion: version };
}
const state = missing.join(", ");
if (state !== previous) log(`Waiting for cloud artifacts for ${sha}: ${state}.`);
previous = state;
const remaining = deadline - now();
if (remaining > 0) await sleep(Math.min(intervalMs, remaining));
}
throw new Error(`Cloud artifacts timed out for ${sha}; missing: ${missing.join(", ")}.`);
}
if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) {
try { await waitForCloudArtifacts(process.argv[2]); }
catch (error) { console.error(error.message); process.exitCode = 1; }
}
scripts/preview-artifacts.test.mjs
-134
View File
@@ -4,9 +4,7 @@ import { planArtifacts } from "./preview-artifacts.mjs";
import { readFileSync, mkdtempSync, writeFileSync, rmSync } from "node:fs";
import { tmpdir } from "node:os";
import path from "node:path";
import { fileURLToPath } from "node:url";
import { gzipSync } from "node:zlib";
import { execFileSync, spawnSync } from "node:child_process";
import { previewManifest, assertMetadata, validateRequest, versionFor, tarManifest, packageExists, imageExists, publishPreview, publishImage } from "./preview-artifacts.mjs";
const sha = "a".repeat(40);
@@ -222,135 +220,3 @@ test("commits sharing a short prefix use separate full-SHA image addresses", asy
await imageExists(other, fetchImpl);
assert.deepEqual(urls.filter((url) => url.includes("/manifests/")), [sha, other].map((commit) => `https://ghcr.io/v2/paperclipai/paperclip/manifests/sha-${commit}-cloud`));
});
test("cloud builds start per commit and preserve tag promotion dependencies", () => {
const docker = readFileSync(new URL("../.github/workflows/docker.yml", import.meta.url), "utf8");
const cloud = readFileSync(new URL("../.github/workflows/docker-cloud.yml", import.meta.url), "utf8");
const readiness = readFileSync(new URL("../.github/workflows/cloud-readiness.yml", import.meta.url), "utf8");
assert.match(readiness, /branches: \[master\]/);
assert.match(readiness, /uses: \.\/\.github\/workflows\/docker-cloud.yml/);
assert.doesNotMatch(cloud, /^ push:/m);
assert.match(cloud, /workflow_call:/);
assert.match(cloud, /group: docker-cloud-\$\{\{ github.sha \}\}/);
assert.match(cloud, /cancel-in-progress: false/);
assert.doesNotMatch(cloud, /uses: .*@v\d\b/);
assert.match(cloud, /cache-to: type=registry,ref=ghcr.io\/\$\{\{ github.repository \}\}:buildcache-cloud-\$\{\{ github.sha \}\},mode=max/);
const caller = docker.split(" build-and-push-cloud:")[1].split(" promote_canary_channel:")[0];
assert.match(caller, /if: github.event_name != 'push' \|\| github.ref != 'refs\/heads\/master'/);
assert.match(caller, /uses: .\/.github\/workflows\/docker-cloud.yml/);
assert.match(docker.split(" promote_canary_channel:")[1], /needs: \[merge-and-push, build-and-push-cloud\]/);
const reaping = cloud.indexOf(" - name: Verify cloud PID 1 reaps orphaned processes");
assert.ok(reaping > cloud.indexOf(" - name: Verify the pushed image resolves the declared Sentry version"));
assert.ok(reaping < cloud.indexOf(" - name: Publish verified full-SHA cloud tag"));
});
test("cloud builds bake the managed runtime identity and verify it before publication", () => {
const workflow = readFileSync(new URL("../.github/workflows/docker-cloud.yml", import.meta.url), "utf8");
const build = workflow.split(" - name: Build and push (cloud)")[1].split(" - name:")[0];
assert.match(build, /build-args: \|\n\s+USER_UID=1001\n\s+USER_GID=1001\n/);
const verify = workflow.indexOf(" - name: Verify cloud runtime user");
assert.ok(verify > workflow.indexOf(" - name: Verify the pushed image resolves the declared Sentry version"));
assert.ok(verify < workflow.indexOf(" - name: Publish verified full-SHA cloud tag"));
const step = workflow.slice(verify).split("\n - name:")[0];
assert.match(step, /IMAGE: ghcr.io\/\$\{\{ github.repository \}\}@\$\{\{ steps.build-cloud.outputs.digest \}\}/);
assert.doesNotMatch(step, /continue-on-error:|if:/);
assert.ok(step.indexOf('--entrypoint sh "$IMAGE"') < step.indexOf('-e USER_UID=1001 -e USER_GID=1001'));
for (const flag of ["u", "g"]) {
assert.ok(step.includes(`test "$(id -${flag} node)" = 1001`));
assert.ok(step.includes(`test "$(id -${flag})" = 1001`));
}
assert.ok(step.includes('test -w "$PAPERCLIP_HOME"'));
});
test("cloud cache imports are bounded, follow master ancestry, and retain the legacy fallback", () => {
const workflow = readFileSync(new URL("../.github/workflows/docker-cloud.yml", import.meta.url), "utf8");
const selector = workflow.indexOf(" - name: Select cloud cache ancestry");
assert.ok(selector > workflow.indexOf(" - name: Login to GitHub Container Registry"));
assert.ok(selector > workflow.indexOf(" - name: Set up Docker Buildx"));
assert.ok(selector < workflow.indexOf(" - name: Build and push (cloud)"));
assert.match(workflow, /run: node scripts\/select-cloud-cache.mjs/);
assert.match(workflow, /cache-from: \$\{\{ steps.cloud-cache.outputs.source \}\}/);
const script = fileURLToPath(new URL("./select-cloud-cache.mjs", import.meta.url));
const dir = mkdtempSync(path.join(tmpdir(), "cloud-cache-test-"));
const output = path.join(dir, "output");
const env = { ...process.env, GIT_AUTHOR_NAME: "Test", GIT_AUTHOR_EMAIL: "test@example.test", GIT_COMMITTER_NAME: "Test", GIT_COMMITTER_EMAIL: "test@example.test" };
const git = (...args) => execFileSync("git", ["-c", "core.hooksPath=/dev/null", "-c", "commit.gpgsign=false", ...args], { cwd: dir, env, encoding: "utf8", stdio: ["ignore", "pipe", "pipe"] }).trim();
try {
git("init", "--initial-branch=master");
const commits = [];
for (let i = 0; i < 12; i++) {
git("commit", "--allow-empty", "-m", `main ${i}`);
commits.unshift(git("rev-parse", "HEAD"));
}
git("checkout", "-b", "topic", "HEAD~1");
git("commit", "--allow-empty", "-m", "topic");
git("checkout", "master");
git("merge", "--no-ff", "topic", "-m", "merge topic");
commits.unshift(git("rev-parse", "HEAD"));
const available = `ghcr.io/paperclipai/paperclip:buildcache-cloud-${commits[2]}`;
const inspections = path.join(dir, "inspections");
writeFileSync(path.join(dir, "docker"), `#!/usr/bin/env node
const fs = require("node:fs");
fs.appendFileSync(process.env.CACHE_INSPECTIONS, process.argv.at(-1) + "\\n");
if (process.argv.at(-1) !== process.env.AVAILABLE_CACHE) {
process.stderr.write("manifest unknown");
process.exit(1);
}
`, { mode: 0o755 });
const result = spawnSync(process.execPath, [script], {
cwd: dir, encoding: "utf8", env: {
...env, PATH: `${dir}${path.delimiter}${env.PATH}`, CACHE_IMAGE: "ghcr.io/paperclipai/paperclip",
GITHUB_OUTPUT: output, AVAILABLE_CACHE: available, CACHE_INSPECTIONS: inspections,
},
});
assert.equal(result.status, 0, result.stderr);
assert.equal(readFileSync(output, "utf8"), `source=type=registry,ref=${available}\n`);
assert.deepEqual(readFileSync(inspections, "utf8").trim().split("\n"), commits.slice(0, 3).map((commit) => `ghcr.io/paperclipai/paperclip:buildcache-cloud-${commit}`));
} finally { rmSync(dir, { recursive: true, force: true }); }
});
test("normal cloud builds publish the checked digest only when source and platform match", () => {
const workflow = readFileSync(new URL("../.github/workflows/docker-cloud.yml", import.meta.url), "utf8");
const cloud = workflow.split(" build-and-push-cloud:")[1];
const verify = cloud.indexOf(" - name: Verify the pushed image resolves the declared Sentry version");
const publish = cloud.indexOf(" - name: Publish verified full-SHA cloud tag");
assert.ok(verify >= 0 && publish > verify);
const verification = cloud.slice(verify, publish);
assert.match(verification, /IMAGE: ghcr.io\/\$\{\{ github.repository \}\}@\$\{\{ steps.build-cloud.outputs.digest \}\}/);
assert.doesNotMatch(verification, /continue-on-error:|if: always\(/);
const step = cloud.slice(publish).split(/\n(?: #| - name:)/)[0];
assert.doesNotMatch(step, /continue-on-error:|if:/);
assert.match(step, /FULL_SHA_TAG: ghcr.io\/\$\{\{ github.repository \}\}:sha-\$\{\{ github.sha \}\}-cloud/);
const script = step.split(" run: |\n")[1].split("\n").map((line) => line.replace(/^ {10}/, "")).join("\n");
const dir = mkdtempSync(path.join(tmpdir(), "cloud-tag-test-"));
const image = `ghcr.io/paperclipai/paperclip@sha256:${"b".repeat(64)}`;
const tag = `ghcr.io/paperclipai/paperclip:sha-${sha}-cloud`;
try {
writeFileSync(path.join(dir, "docker"), `#!/bin/sh
case "$1 $2" in
'image inspect')
case "$5" in
*revision*) printf '%s\\n' "$TEST_REVISION" ;;
*) printf '%s\\n' "$TEST_PLATFORM" ;;
esac ;;
'buildx imagetools') printf '%s\\n' "$@" > "$TEST_CALLS" ;;
*) exit 99 ;;
esac
`, { mode: 0o755 });
for (const [revision, platform, succeeds] of [[sha, "linux/amd64", true], ["c".repeat(40), "linux/amd64", false], [sha, "linux/arm64", false]]) {
const calls = path.join(dir, "calls");
rmSync(calls, { force: true });
const result = spawnSync("bash", ["-c", script], { encoding: "utf8", env: {
...process.env, PATH: `${dir}${path.delimiter}${process.env.PATH}`, GITHUB_SHA: sha,
IMAGE: image, FULL_SHA_TAG: tag, TEST_REVISION: revision, TEST_PLATFORM: platform, TEST_CALLS: calls,
} });
if (succeeds) {
assert.equal(result.status, 0, result.stderr);
assert.deepEqual(readFileSync(calls, "utf8").trim().split("\n"), ["buildx", "imagetools", "create", "--prefer-index=false", "--tag", tag, image]);
} else {
assert.notEqual(result.status, 0);
assert.throws(() => readFileSync(calls), { code: "ENOENT" });
}
}
} finally { rmSync(dir, { recursive: true, force: true }); }
});
scripts/select-cloud-cache.mjs
-66
View File
@@ -1,66 +0,0 @@
#!/usr/bin/env node
import { execFileSync } from "node:child_process";
import { appendFileSync } from "node:fs";
import { pathToFileURL } from "node:url";
export function cloudCacheCandidates(image, commits) {
if (!/^ghcr\.io\/[a-z0-9._-]+\/[a-z0-9._-]+$/.test(image ?? "")) {
throw new Error("Expected a GHCR owner/repository cache image.");
}
if (!Array.isArray(commits) || commits.length === 0 || commits.some((sha) => !/^[a-f0-9]{40}$/.test(sha))) {
throw new Error("Cloud cache ancestry requires full commit SHAs.");
}
return [
...[...new Set(commits)].slice(0, 10).map((sha) => `${image}:buildcache-cloud-${sha}`),
`${image}:buildcache-cloud`,
];
}
export async function selectCloudCache(image, commits, {
exists = registryCacheExists,
log = console.log,
} = {}) {
for (const ref of cloudCacheCandidates(image, commits)) {
try {
if (!await exists(ref)) continue;
log(`Using cloud cache: ${ref}`);
return `type=registry,ref=${ref}`;
} catch {
// Cache availability must not turn an otherwise valid build into a
// failure. A later ancestor may still be available during a rollout.
log(`Could not inspect cloud cache ${ref}; trying the next ancestor.`);
}
}
log("No cloud cache is available; this build will populate one.");
return "";
}
function registryCacheExists(ref) {
try {
// Use the preceding Docker login, including for private registry caches.
// Inspect metadata only: no layer download and no image execution.
execFileSync("docker", ["buildx", "imagetools", "inspect", "--raw", ref], {
timeout: 10_000,
maxBuffer: 1024 * 1024,
stdio: ["ignore", "pipe", "pipe"],
});
return true;
} catch (error) {
if (/manifest unknown|not found|NAME_UNKNOWN/i.test(String(error.stderr ?? ""))) return false;
throw error;
}
}
if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) {
try {
if (!process.env.GITHUB_OUTPUT) throw new Error("GITHUB_OUTPUT is required.");
const commits = execFileSync("git", ["rev-list", "--first-parent", "--max-count=10", "HEAD"], {
encoding: "utf8",
}).trim().split("\n");
const source = await selectCloudCache(process.env.CACHE_IMAGE, commits, { exists: registryCacheExists });
appendFileSync(process.env.GITHUB_OUTPUT, `source=${source}\n`);
} catch (error) {
console.error(error.message);
process.exitCode = 1;
}
}
scripts/select-cloud-cache.test.mjs
-65
View File
@@ -1,65 +0,0 @@
import assert from "node:assert/strict";
import test from "node:test";
import { cloudCacheCandidates, selectCloudCache } from "./select-cloud-cache.mjs";
const image = "ghcr.io/paperclipai/paperclip";
const commits = ["a".repeat(40), "b".repeat(40), "c".repeat(40)];
const candidates = cloudCacheCandidates(image, commits);
test("a same-SHA rerun imports only its existing cache", async () => {
const inspected = [];
const source = await selectCloudCache(image, commits, {
exists: async (ref) => { inspected.push(ref); return true; }, log() {},
});
assert.equal(source, `type=registry,ref=${candidates[0]}`);
assert.deepEqual(inspected, candidates.slice(0, 1));
});
test("a new merge imports only its nearest available ancestor", async () => {
const inspected = [];
const source = await selectCloudCache(image, commits, {
exists: async (ref) => { inspected.push(ref); return ref === candidates[1]; }, log() {},
});
assert.equal(source, `type=registry,ref=${candidates[1]}`);
assert.deepEqual(inspected, candidates.slice(0, 2));
assert.equal(source.includes("\n"), false);
});
test("a still-building parent falls back to an older completed cache", async () => {
assert.equal(await selectCloudCache(image, commits, {
exists: async (ref) => ref === candidates[2], log() {},
}), `type=registry,ref=${candidates[2]}`);
});
test("the legacy cache is used only if no SHA cache exists", async () => {
const inspected = [];
assert.equal(await selectCloudCache(image, commits, {
exists: async (ref) => { inspected.push(ref); return ref === candidates.at(-1); }, log() {},
}), `type=registry,ref=${candidates.at(-1)}`);
assert.deepEqual(inspected, candidates);
});
test("missing caches permit a cold build", async () => {
assert.equal(await selectCloudCache(image, commits, { exists: async () => false, log() {} }), "");
});
test("a failed lookup can fall back without failing image publication", async () => {
const messages = [];
assert.equal(await selectCloudCache(image, commits, {
exists: async (ref) => {
if (ref === candidates[0]) throw new Error("registry temporarily unavailable");
return true;
},
log: (message) => messages.push(message),
}), `type=registry,ref=${candidates[1]}`);
assert.match(messages[0], /Could not inspect cloud cache/);
});
test("ancestry is bounded, deduplicated, and rejects output injection", () => {
const many = Array.from({ length: 20 }, (_, i) => i.toString(16).padStart(40, "0"));
assert.equal(cloudCacheCandidates(image, many).length, 11);
assert.deepEqual(cloudCacheCandidates(image, [commits[0], commits[0]]), [candidates[0], candidates.at(-1)]);
assert.throws(() => cloudCacheCandidates(`${image}\nsource=untrusted`, commits));
assert.throws(() => cloudCacheCandidates(image, ["master"]));
assert.throws(() => cloudCacheCandidates(image, []));
});
server/src/__tests__/cloud-image-bundled-plugins.test.ts
+6 -50
View File
@@ -5,22 +5,18 @@ import { describe, expect, it } from "vitest";
import { BUNDLED_PLUGIN_CATALOG } from "../services/bundled-plugins.js";
/**
* Drift guard for the cloud image variant (Dockerfile `cloud` target).
* Drift guard for the explicit preview image (Dockerfile `cloud` target).
*
* The cloud image builds the sandbox-provider plugins named in the
* The preview image builds the sandbox-provider plugins named in the
* CLOUD_BUNDLED_PLUGINS build arg so managed instances can auto-install
* them from the bundled catalog at boot. That contract spans three places
* that nothing else ties together: the Dockerfile ARG default, the docker
* workflow's build-arg, and BUNDLED_PLUGIN_CATALOG. A rename or removal in
* any one of them would otherwise surface only when the image build fails
* on master — or worse, as a silent "bundle not present" skip at instance
* boot.
* them from the bundled catalog at boot. The Dockerfile default and
* BUNDLED_PLUGIN_CATALOG must agree even after the recurring public cloud
* publisher is retired. Explicit previews still use this build target.
*/
const repoRoot = path.resolve(path.dirname(fileURLToPath(import.meta.url)), "..", "..", "..");
const dockerfile = readFileSync(path.join(repoRoot, "Dockerfile"), "utf8");
const workflow = readFileSync(path.join(repoRoot, ".github", "workflows", "docker.yml"), "utf8");
const cloudWorkflow = readFileSync(path.join(repoRoot, ".github", "workflows", "docker-cloud.yml"), "utf8");
function parseList(source: string, pattern: RegExp, label: string): string[] {
const match = source.match(pattern);
@@ -35,18 +31,9 @@ const dockerfileDefault = parseList(
/^ARG CLOUD_BUNDLED_PLUGINS="([^"]*)"/m,
"Dockerfile",
);
const workflowArg = parseList(
cloudWorkflow,
/^\s*CLOUD_BUNDLED_PLUGINS=(.*)$/m,
"docker workflow",
);
describe("cloud image bundled plugins", () => {
it("keeps the Dockerfile default and the workflow build-arg in sync", () => {
expect(workflowArg).toEqual(dockerfileDefault);
});
it.each([...new Set([...dockerfileDefault, ...workflowArg])])(
it.each(dockerfileDefault)(
"plugin %s is buildable and resolvable by the auto-installer",
(name) => {
const dir = path.join(repoRoot, "packages", "plugins", "sandbox-providers", name);
@@ -77,37 +64,6 @@ describe("cloud image bundled plugins", () => {
expect(workflow).toMatch(/^\s*target: production$/m);
});
it("publishes the cloud image in its own job with no needs coupling", () => {
const caller = workflow.split(" build-and-push-cloud:")[1]?.split(" promote_canary_channel:")[0];
expect(caller, "tag and manual builds must call the cloud workflow").toContain("uses: ./.github/workflows/docker-cloud.yml");
expect(caller, "the reusable caller must also remain independent of production").not.toMatch(/^\s*needs:/m);
// The reusable cloud workflow owns its job and SHA concurrency group.
// Production publication must not gate, delay, or skip the cloud build.
const jobsSection = cloudWorkflow.slice(cloudWorkflow.indexOf("\njobs:\n"));
const headers = [...jobsSection.matchAll(/^ {2}([\w-]+):[^\n]*$/gm)];
expect(
headers.length,
"docker-cloud.yml must declare a cloud build job under jobs:",
).toBeGreaterThanOrEqual(1);
// Locate the job block that carries the cloud build (target: cloud) and
// assert it declares no `needs:` — coupling it to another job would
// reintroduce the shared failure the split job exists to remove.
const cloudHeaderIdx = headers.findIndex((header, i) => {
const start = header.index ?? 0;
const end = headers[i + 1]?.index ?? jobsSection.length;
return jobsSection.slice(start, end).includes("target: cloud");
});
expect(cloudHeaderIdx, "one job must build the cloud target").toBeGreaterThanOrEqual(0);
const start = headers[cloudHeaderIdx].index ?? 0;
const end = headers[cloudHeaderIdx + 1]?.index ?? jobsSection.length;
const cloudJobBlock = jobsSection.slice(start, end);
expect(
cloudJobBlock,
"the cloud job must not couple to another job via needs:",
).not.toMatch(/^\s*needs:/m);
});
it("throttles the docker workflow with cancel-in-progress: false", () => {
// Concurrency is declared at the workflow (top) level so a single group
// spans the whole run, and cancel-in-progress is false so an in-flight
server/src/__tests__/cloud-image-sentry.test.ts
+4 -9
View File
@@ -14,12 +14,12 @@ import { fileURLToPath } from "node:url";
import { describe, expect, it } from "vitest";
/**
* Drift guard for the cloud image variant's bundled Sentry server package
* Drift guard for the explicit preview image's bundled Sentry server package
* (Dockerfile `cloud` target).
*
* The self-hosted image, built from the `production` target, keeps
* `@sentry/node` as a true optional peer dependency: the operator installs
* it themselves. The hosted (cloud) image installs the packages the
* it themselves. The explicit preview image installs the packages the
* `CLOUD_BUNDLED_SERVER_DEPS` build argument names, so a managed tenant
* gets server error reports with no separate install step. The stage
* reads each package's version from the `peerDependencies` block of
@@ -30,14 +30,13 @@ import { describe, expect, it } from "vitest";
* workflow carry no literal version pin (they read the version from
* `server/package.json` at build time instead); the `cloud-server-deps`
* stage declares the `CLOUD_BUNDLED_SERVER_DEPS` build argument with a
* default that names `@sentry/node`; the docker workflow passes that same
* argument to the cloud build; and no committed manifest re-declares the
* default that names `@sentry/node`; and no committed manifest re-declares the
* version.
*/
const repoRoot = path.resolve(path.dirname(fileURLToPath(import.meta.url)), "..", "..", "..");
const dockerfile = readFileSync(path.join(repoRoot, "Dockerfile"), "utf8");
const workflow = readFileSync(path.join(repoRoot, ".github", "workflows", "docker-cloud.yml"), "utf8");
const workflow = readFileSync(path.join(repoRoot, ".github", "workflows", "release.yml"), "utf8");
const serverPackageJson = JSON.parse(
readFileSync(path.join(repoRoot, "server", "package.json"), "utf8"),
) as { peerDependencies?: Record<string, string> };
@@ -164,10 +163,6 @@ describe("cloud image Sentry install", () => {
).toContain("@sentry/node");
});
it("passes CLOUD_BUNDLED_SERVER_DEPS to the cloud build in the docker workflow", () => {
expect(workflow).toMatch(/^\s*CLOUD_BUNDLED_SERVER_DEPS=@sentry\/node\s*$/m);
});
it("declares no committed manifest that re-states the version", () => {
expect(
existsSync(path.join(repoRoot, "docker", "cloud-server-deps")),
server/src/__tests__/docker-build-stamp.test.ts
+6 -7
View File
@@ -21,7 +21,7 @@ import { describe, expect, it } from "vitest";
const repoRoot = path.resolve(path.dirname(fileURLToPath(import.meta.url)), "..", "..", "..");
const dockerfile = readFileSync(path.join(repoRoot, "Dockerfile"), "utf8");
const workflow = readFileSync(path.join(repoRoot, ".github", "workflows", "docker.yml"), "utf8");
const cloudWorkflow = readFileSync(path.join(repoRoot, ".github", "workflows", "docker-cloud.yml"), "utf8");
const previewWorkflow = readFileSync(path.join(repoRoot, ".github", "workflows", "release.yml"), "utf8");
/**
* Return the text of the Dockerfile stage that starts at the named target.
@@ -68,12 +68,11 @@ describe("docker build-stamp wiring", () => {
).toBeLessThan(serverBuildIdx);
});
it("passes PAPERCLIP_BUILD_COMMIT as a build-arg for both image targets", () => {
const argLines = [...`${workflow}\n${cloudWorkflow}`.matchAll(/^\s*PAPERCLIP_BUILD_COMMIT=.*$/gm)];
expect(
argLines.length,
"the docker workflow must pass PAPERCLIP_BUILD_COMMIT for the production and cloud builds",
).toBeGreaterThanOrEqual(2);
it("passes PAPERCLIP_BUILD_COMMIT as a build-arg for standard and explicit preview builds", () => {
for (const [name, source] of [["standard", workflow], ["preview", previewWorkflow]]) {
expect(source, `${name} must pass the source commit into the image build`)
.toMatch(/^\s*PAPERCLIP_BUILD_COMMIT=\$\{\{ (?:github.sha|inputs.source_ref) \}\}$/m);
}
});
});