mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-02 02:07:25 +08:00
ci: retire recurring public cloud image builds (#13827)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Core publishes standard images and source verification for downstream services. > - Managed services can now compose private images from the signed standard image. > - Core still builds a second public cloud image on every master push and release. > - That duplicate producer consumes build capacity and retains an obsolete readiness contract. > - This pull request retires recurring cloud publication while preserving the standard producer and rollback artifacts. ## Linked Issues or Issue Description Refs #13797 and #13789. Related: #12856 changes image dependency packaging; it does not retire this producer. **What existing behavior does this improve?** Core's recurring Docker publication and Cloud readiness workflow. **Current behavior** Master pushes call the legacy cloud publisher from Cloud readiness. Release tags and manual Docker runs call it too. Canary promotion also requires the legacy image. **Proposed behavior** Publish standard Core images and retain `Cloud source verified v1`. Let downstream services build their managed image. Keep explicit commit previews and existing images available. ## What Changed - Remove `docker-cloud.yml`, its master and release callers, and its unused cache selector. - Remove the legacy image/migrator wait and `Cloud deployable v1` job. Keep the full source verification workflow and exact source-proof name. - Make canary promotion inspect and promote the standard image only. - Preserve signed standard-image publication, direct migrator publication, and explicit `release.yml` previews. The preview path still uses the Dockerfile `cloud` target. - Update workflow, preview, build-stamp, and packaging tests. Exercise the promotion shell with mocked registry commands, including missing-image and missing-tag cases. - Document frozen legacy aliases, consumer requirements, preview compatibility, and rollback retention. ## Verification - All 377 workflow tests pass: `node --test .github/scripts/tests/*.test.mjs`. - All 129 release-registry tests pass: `pnpm test:release-registry`. - Focused source-proof, standard-image, preview, and workflow tests pass: 256 tests. - Focused image packaging/build-stamp tests pass: 16 tests. - Actionlint passes on all three changed workflow files. `git diff --check` passes. - Full local `pnpm build` and `pnpm -r typecheck` pass. - The policy follow-up updates an old assertion that required the removed readiness job. All 37 source-proof/release-workflow tests pass locally. - Full local `pnpm test:run` did not complete successfully while the Mac ran out of disk space. No full-suite pass is claimed. Removed 1.2 GiB of generated Cargo output from this isolated worktree with `cargo clean`. GitHub CI passed on the final head: 52 successful checks and 2 optional skips. - Fresh Greptile review for `4f5fe1951f0bd7f7739cf6655d395ff78f1ed944`: **5/5**, successful current-head check, zero review threads. - September 23 refresh: the unchanged PR head merges cleanly with current master `db8f8fe5b73a2697684a30261b0d306a9c631aba`. In an isolated temporary worktree, all 377 workflow tests and 29 release/preview tests pass on the combined tree. `git diff --cached --check` passes. - Refreshed Actionlint workflow validation passes with ShellCheck disabled. Full Actionlint reports the same 10 existing ShellCheck diagnostics as master, with no added diagnostics. No source changes or new PR commits were needed. - The full local build/typecheck and current-head Linux CI results above remain the verification for the unchanged PR head. They were not rerun for this metadata-only refresh. No image publication or tenant deployment was initiated for this refresh. ## Risks **Deployment prerequisite satisfied (September 23):** The combined cleanup release is deployed to staging and production, and production Support is verified. Active managed-fleet automation uses standard-image composition. Explicit immutable previews remain supported by the retained preview publisher. This PR is ready for maintainer review; keep auto-merge disabled and wait for explicit merge authorization. - A consumer still selecting `Cloud deployable v1` will stop advancing at the last legacy-ready commit. Confirm active automatic consumers use the standard-image composition contract before merge. - Legacy cloud release-channel aliases stop advancing. Standard self-hosted aliases continue. - This PR deletes no registry images, cache tags, migrators, credentials, or runner infrastructure. Existing immutable releases remain usable for rollback. - Explicit legacy previews remain for commit-specific operator deployments. Retiring that compatibility path requires a separate consumer migration. - These changes affect CI publication, not database schema or application behavior. ## Model Used OpenAI Codex, GPT-6. The runtime does not expose a more specific model identifier or context-window size. Used repository inspection, reasoning, code editing, shell tools, and test execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
Devin Foley
@@ -63,13 +63,13 @@ test("canary reuses exact-source proof while stable keeps full verification", ()
|
||||
|
||||
test("source proof requires every source check and does not wait on image publication", () => {
|
||||
const readiness = readWorkflow("cloud-readiness.yml");
|
||||
const proof = readiness.split(" source_verified:\n")[1].split("\n ready:")[0];
|
||||
const proof = readiness.split(" source_verified:\n")[1];
|
||||
assert.match(proof, /name: Cloud source verified v1/);
|
||||
assert.match(proof, /needs: \[verify\]/);
|
||||
assert.match(proof, /node --test scripts\/cloud-source-verification.test.mjs/);
|
||||
assert.match(proof, /SOURCE_SHA: \$\{\{ github\.sha \}\}/);
|
||||
assert.doesNotMatch(proof, /always\(\)|continue-on-error|needs:.*(?:image|artifacts)/);
|
||||
assert.match(readiness.split(" ready:\n")[1], /needs: \[verify, image, artifacts\]/);
|
||||
assert.doesNotMatch(readiness, /^ (?:image|artifacts|ready):/m);
|
||||
});
|
||||
|
||||
test("onboard smoke container binds beyond loopback so the mapped port is reachable", () => {
|
||||
|
||||
@@ -1,102 +0,0 @@
|
||||
#!/usr/bin/env node
|
||||
import { pathToFileURL } from "node:url";
|
||||
import { execFileSync } from "node:child_process";
|
||||
import { mkdtempSync, rmSync, writeFileSync } from "node:fs";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import { imageExists, versionFor } from "./preview-artifacts.mjs";
|
||||
import { verifyPublished } from "./cloud-migrator-artifacts.mjs";
|
||||
|
||||
const repository = "paperclipai/paperclip";
|
||||
const workflow = ".github/workflows/cloud-migrator-artifacts.yml";
|
||||
|
||||
export async function migratorPublished(sha, fetchImpl, token) {
|
||||
let pending = false;
|
||||
const failures = [];
|
||||
for (let page = 1; page <= 10; page++) {
|
||||
const response = await fetchImpl(`https://api.github.com/repos/${repository}/actions/workflows/cloud-migrator-artifacts.yml/runs?branch=master&head_sha=${sha}&per_page=100&page=${page}`, {
|
||||
headers: { Accept: "application/vnd.github+json", ...(token ? { Authorization: `Bearer ${token}` } : {}) },
|
||||
redirect: "error", signal: AbortSignal.timeout(30_000),
|
||||
});
|
||||
if (!response.ok) throw new Error(`Migrator producer lookup failed: HTTP ${response.status}`);
|
||||
const body = await response.json();
|
||||
if (!Array.isArray(body.workflow_runs) || !Number.isSafeInteger(body.total_count) || body.total_count < 0 ||
|
||||
(page === 1 && (body.total_count === 0) !== (body.workflow_runs.length === 0))) throw new Error("Invalid migrator producer response.");
|
||||
if (body.total_count === 0) return false;
|
||||
for (const run of body.workflow_runs) {
|
||||
if (run.head_sha !== sha || run.head_branch !== "master" || run.path !== workflow ||
|
||||
run.head_repository?.id !== 1170821064 || run.head_repository.full_name !== repository ||
|
||||
!["push", "workflow_dispatch"].includes(run.event)) throw new Error("Migrator producer identity mismatch.");
|
||||
// Publication is immutable. A later failed manual run must not hide a
|
||||
// successful exact-source publisher; the signed bundle is checked next.
|
||||
if (run.status === "completed" && run.conclusion === "success") return true;
|
||||
if (run.status !== "completed") pending = true;
|
||||
else failures.push(`${run.id}: ${run.conclusion}`);
|
||||
}
|
||||
if (page * 100 >= body.total_count) {
|
||||
if (pending) return false;
|
||||
throw new Error(`Migrator producers failed: ${failures.join(", ")}.`);
|
||||
}
|
||||
}
|
||||
throw new Error("Too many migrator producer runs to establish publication.");
|
||||
}
|
||||
|
||||
export function verifyManifestProvenance(bytes, sha, { exec = execFileSync } = {}) {
|
||||
versionFor(sha);
|
||||
const scratch = mkdtempSync(path.join(os.tmpdir(), "cloud-readiness-attestation-"));
|
||||
try {
|
||||
const file = path.join(scratch, "manifest.json");
|
||||
writeFileSync(file, bytes);
|
||||
exec("gh", ["attestation", "verify", file, "--repo", repository,
|
||||
"--source-digest", sha, "--source-ref", "refs/heads/master",
|
||||
"--cert-identity", `https://github.com/${repository}/${workflow}@refs/heads/master`,
|
||||
"--deny-self-hosted-runners"], { stdio: "inherit", timeout: 60_000 });
|
||||
} finally { rmSync(scratch, { recursive: true, force: true }); }
|
||||
}
|
||||
|
||||
/** Read-only availability gate. Deployment still resolves and pins artifacts. */
|
||||
export async function waitForCloudArtifacts(sha, {
|
||||
fetchImpl = fetch,
|
||||
token = process.env.GH_TOKEN,
|
||||
verifyProvenance = verifyManifestProvenance,
|
||||
now = () => performance.now(),
|
||||
sleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms)),
|
||||
timeoutMs = 30 * 60_000,
|
||||
intervalMs = 20_000,
|
||||
log = console.log,
|
||||
} = {}) {
|
||||
const version = versionFor(sha);
|
||||
if (!Number.isFinite(timeoutMs) || timeoutMs <= 0 || !Number.isFinite(intervalMs) || intervalMs <= 0) {
|
||||
throw new Error("Cloud readiness requires positive finite timeout and poll interval.");
|
||||
}
|
||||
const deadline = now() + timeoutMs;
|
||||
let previous;
|
||||
let missing = ["image", "migrator"];
|
||||
while (now() < deadline) {
|
||||
// Recheck the image and exact-source publisher on the successful poll.
|
||||
// Only a missing/pending producer waits; failed publication fails closed.
|
||||
const results = await Promise.all([
|
||||
imageExists(sha, fetchImpl),
|
||||
migratorPublished(sha, fetchImpl, token),
|
||||
]);
|
||||
missing = ["image", "migrator"].filter((_, index) => !results[index]);
|
||||
if (missing.length === 0) {
|
||||
// Verify the exact signed bytes and all pinned downloads after the
|
||||
// publisher succeeds. An inaccessible or corrupt artifact cannot pass.
|
||||
await verifyPublished(sha, fetchImpl, { verifyProvenance });
|
||||
log(`Cloud artifacts available for ${sha}: verified image and exact-source migrator ${version}.`);
|
||||
return { version: 1, sha, packageVersion: version };
|
||||
}
|
||||
const state = missing.join(", ");
|
||||
if (state !== previous) log(`Waiting for cloud artifacts for ${sha}: ${state}.`);
|
||||
previous = state;
|
||||
const remaining = deadline - now();
|
||||
if (remaining > 0) await sleep(Math.min(intervalMs, remaining));
|
||||
}
|
||||
throw new Error(`Cloud artifacts timed out for ${sha}; missing: ${missing.join(", ")}.`);
|
||||
}
|
||||
|
||||
if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) {
|
||||
try { await waitForCloudArtifacts(process.argv[2]); }
|
||||
catch (error) { console.error(error.message); process.exitCode = 1; }
|
||||
}
|
||||
@@ -4,9 +4,7 @@ import { planArtifacts } from "./preview-artifacts.mjs";
|
||||
import { readFileSync, mkdtempSync, writeFileSync, rmSync } from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import path from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import { gzipSync } from "node:zlib";
|
||||
import { execFileSync, spawnSync } from "node:child_process";
|
||||
import { previewManifest, assertMetadata, validateRequest, versionFor, tarManifest, packageExists, imageExists, publishPreview, publishImage } from "./preview-artifacts.mjs";
|
||||
|
||||
const sha = "a".repeat(40);
|
||||
@@ -222,135 +220,3 @@ test("commits sharing a short prefix use separate full-SHA image addresses", asy
|
||||
await imageExists(other, fetchImpl);
|
||||
assert.deepEqual(urls.filter((url) => url.includes("/manifests/")), [sha, other].map((commit) => `https://ghcr.io/v2/paperclipai/paperclip/manifests/sha-${commit}-cloud`));
|
||||
});
|
||||
|
||||
test("cloud builds start per commit and preserve tag promotion dependencies", () => {
|
||||
const docker = readFileSync(new URL("../.github/workflows/docker.yml", import.meta.url), "utf8");
|
||||
const cloud = readFileSync(new URL("../.github/workflows/docker-cloud.yml", import.meta.url), "utf8");
|
||||
const readiness = readFileSync(new URL("../.github/workflows/cloud-readiness.yml", import.meta.url), "utf8");
|
||||
assert.match(readiness, /branches: \[master\]/);
|
||||
assert.match(readiness, /uses: \.\/\.github\/workflows\/docker-cloud.yml/);
|
||||
assert.doesNotMatch(cloud, /^ push:/m);
|
||||
assert.match(cloud, /workflow_call:/);
|
||||
assert.match(cloud, /group: docker-cloud-\$\{\{ github.sha \}\}/);
|
||||
assert.match(cloud, /cancel-in-progress: false/);
|
||||
assert.doesNotMatch(cloud, /uses: .*@v\d\b/);
|
||||
assert.match(cloud, /cache-to: type=registry,ref=ghcr.io\/\$\{\{ github.repository \}\}:buildcache-cloud-\$\{\{ github.sha \}\},mode=max/);
|
||||
const caller = docker.split(" build-and-push-cloud:")[1].split(" promote_canary_channel:")[0];
|
||||
assert.match(caller, /if: github.event_name != 'push' \|\| github.ref != 'refs\/heads\/master'/);
|
||||
assert.match(caller, /uses: .\/.github\/workflows\/docker-cloud.yml/);
|
||||
assert.match(docker.split(" promote_canary_channel:")[1], /needs: \[merge-and-push, build-and-push-cloud\]/);
|
||||
const reaping = cloud.indexOf(" - name: Verify cloud PID 1 reaps orphaned processes");
|
||||
assert.ok(reaping > cloud.indexOf(" - name: Verify the pushed image resolves the declared Sentry version"));
|
||||
assert.ok(reaping < cloud.indexOf(" - name: Publish verified full-SHA cloud tag"));
|
||||
});
|
||||
|
||||
test("cloud builds bake the managed runtime identity and verify it before publication", () => {
|
||||
const workflow = readFileSync(new URL("../.github/workflows/docker-cloud.yml", import.meta.url), "utf8");
|
||||
const build = workflow.split(" - name: Build and push (cloud)")[1].split(" - name:")[0];
|
||||
assert.match(build, /build-args: \|\n\s+USER_UID=1001\n\s+USER_GID=1001\n/);
|
||||
const verify = workflow.indexOf(" - name: Verify cloud runtime user");
|
||||
assert.ok(verify > workflow.indexOf(" - name: Verify the pushed image resolves the declared Sentry version"));
|
||||
assert.ok(verify < workflow.indexOf(" - name: Publish verified full-SHA cloud tag"));
|
||||
const step = workflow.slice(verify).split("\n - name:")[0];
|
||||
assert.match(step, /IMAGE: ghcr.io\/\$\{\{ github.repository \}\}@\$\{\{ steps.build-cloud.outputs.digest \}\}/);
|
||||
assert.doesNotMatch(step, /continue-on-error:|if:/);
|
||||
assert.ok(step.indexOf('--entrypoint sh "$IMAGE"') < step.indexOf('-e USER_UID=1001 -e USER_GID=1001'));
|
||||
for (const flag of ["u", "g"]) {
|
||||
assert.ok(step.includes(`test "$(id -${flag} node)" = 1001`));
|
||||
assert.ok(step.includes(`test "$(id -${flag})" = 1001`));
|
||||
}
|
||||
assert.ok(step.includes('test -w "$PAPERCLIP_HOME"'));
|
||||
});
|
||||
|
||||
test("cloud cache imports are bounded, follow master ancestry, and retain the legacy fallback", () => {
|
||||
const workflow = readFileSync(new URL("../.github/workflows/docker-cloud.yml", import.meta.url), "utf8");
|
||||
const selector = workflow.indexOf(" - name: Select cloud cache ancestry");
|
||||
assert.ok(selector > workflow.indexOf(" - name: Login to GitHub Container Registry"));
|
||||
assert.ok(selector > workflow.indexOf(" - name: Set up Docker Buildx"));
|
||||
assert.ok(selector < workflow.indexOf(" - name: Build and push (cloud)"));
|
||||
assert.match(workflow, /run: node scripts\/select-cloud-cache.mjs/);
|
||||
assert.match(workflow, /cache-from: \$\{\{ steps.cloud-cache.outputs.source \}\}/);
|
||||
const script = fileURLToPath(new URL("./select-cloud-cache.mjs", import.meta.url));
|
||||
const dir = mkdtempSync(path.join(tmpdir(), "cloud-cache-test-"));
|
||||
const output = path.join(dir, "output");
|
||||
const env = { ...process.env, GIT_AUTHOR_NAME: "Test", GIT_AUTHOR_EMAIL: "test@example.test", GIT_COMMITTER_NAME: "Test", GIT_COMMITTER_EMAIL: "test@example.test" };
|
||||
const git = (...args) => execFileSync("git", ["-c", "core.hooksPath=/dev/null", "-c", "commit.gpgsign=false", ...args], { cwd: dir, env, encoding: "utf8", stdio: ["ignore", "pipe", "pipe"] }).trim();
|
||||
try {
|
||||
git("init", "--initial-branch=master");
|
||||
const commits = [];
|
||||
for (let i = 0; i < 12; i++) {
|
||||
git("commit", "--allow-empty", "-m", `main ${i}`);
|
||||
commits.unshift(git("rev-parse", "HEAD"));
|
||||
}
|
||||
git("checkout", "-b", "topic", "HEAD~1");
|
||||
git("commit", "--allow-empty", "-m", "topic");
|
||||
git("checkout", "master");
|
||||
git("merge", "--no-ff", "topic", "-m", "merge topic");
|
||||
commits.unshift(git("rev-parse", "HEAD"));
|
||||
const available = `ghcr.io/paperclipai/paperclip:buildcache-cloud-${commits[2]}`;
|
||||
const inspections = path.join(dir, "inspections");
|
||||
writeFileSync(path.join(dir, "docker"), `#!/usr/bin/env node
|
||||
const fs = require("node:fs");
|
||||
fs.appendFileSync(process.env.CACHE_INSPECTIONS, process.argv.at(-1) + "\\n");
|
||||
if (process.argv.at(-1) !== process.env.AVAILABLE_CACHE) {
|
||||
process.stderr.write("manifest unknown");
|
||||
process.exit(1);
|
||||
}
|
||||
`, { mode: 0o755 });
|
||||
const result = spawnSync(process.execPath, [script], {
|
||||
cwd: dir, encoding: "utf8", env: {
|
||||
...env, PATH: `${dir}${path.delimiter}${env.PATH}`, CACHE_IMAGE: "ghcr.io/paperclipai/paperclip",
|
||||
GITHUB_OUTPUT: output, AVAILABLE_CACHE: available, CACHE_INSPECTIONS: inspections,
|
||||
},
|
||||
});
|
||||
assert.equal(result.status, 0, result.stderr);
|
||||
assert.equal(readFileSync(output, "utf8"), `source=type=registry,ref=${available}\n`);
|
||||
assert.deepEqual(readFileSync(inspections, "utf8").trim().split("\n"), commits.slice(0, 3).map((commit) => `ghcr.io/paperclipai/paperclip:buildcache-cloud-${commit}`));
|
||||
} finally { rmSync(dir, { recursive: true, force: true }); }
|
||||
});
|
||||
|
||||
test("normal cloud builds publish the checked digest only when source and platform match", () => {
|
||||
const workflow = readFileSync(new URL("../.github/workflows/docker-cloud.yml", import.meta.url), "utf8");
|
||||
const cloud = workflow.split(" build-and-push-cloud:")[1];
|
||||
const verify = cloud.indexOf(" - name: Verify the pushed image resolves the declared Sentry version");
|
||||
const publish = cloud.indexOf(" - name: Publish verified full-SHA cloud tag");
|
||||
assert.ok(verify >= 0 && publish > verify);
|
||||
const verification = cloud.slice(verify, publish);
|
||||
assert.match(verification, /IMAGE: ghcr.io\/\$\{\{ github.repository \}\}@\$\{\{ steps.build-cloud.outputs.digest \}\}/);
|
||||
assert.doesNotMatch(verification, /continue-on-error:|if: always\(/);
|
||||
const step = cloud.slice(publish).split(/\n(?: #| - name:)/)[0];
|
||||
assert.doesNotMatch(step, /continue-on-error:|if:/);
|
||||
assert.match(step, /FULL_SHA_TAG: ghcr.io\/\$\{\{ github.repository \}\}:sha-\$\{\{ github.sha \}\}-cloud/);
|
||||
const script = step.split(" run: |\n")[1].split("\n").map((line) => line.replace(/^ {10}/, "")).join("\n");
|
||||
const dir = mkdtempSync(path.join(tmpdir(), "cloud-tag-test-"));
|
||||
const image = `ghcr.io/paperclipai/paperclip@sha256:${"b".repeat(64)}`;
|
||||
const tag = `ghcr.io/paperclipai/paperclip:sha-${sha}-cloud`;
|
||||
try {
|
||||
writeFileSync(path.join(dir, "docker"), `#!/bin/sh
|
||||
case "$1 $2" in
|
||||
'image inspect')
|
||||
case "$5" in
|
||||
*revision*) printf '%s\\n' "$TEST_REVISION" ;;
|
||||
*) printf '%s\\n' "$TEST_PLATFORM" ;;
|
||||
esac ;;
|
||||
'buildx imagetools') printf '%s\\n' "$@" > "$TEST_CALLS" ;;
|
||||
*) exit 99 ;;
|
||||
esac
|
||||
`, { mode: 0o755 });
|
||||
for (const [revision, platform, succeeds] of [[sha, "linux/amd64", true], ["c".repeat(40), "linux/amd64", false], [sha, "linux/arm64", false]]) {
|
||||
const calls = path.join(dir, "calls");
|
||||
rmSync(calls, { force: true });
|
||||
const result = spawnSync("bash", ["-c", script], { encoding: "utf8", env: {
|
||||
...process.env, PATH: `${dir}${path.delimiter}${process.env.PATH}`, GITHUB_SHA: sha,
|
||||
IMAGE: image, FULL_SHA_TAG: tag, TEST_REVISION: revision, TEST_PLATFORM: platform, TEST_CALLS: calls,
|
||||
} });
|
||||
if (succeeds) {
|
||||
assert.equal(result.status, 0, result.stderr);
|
||||
assert.deepEqual(readFileSync(calls, "utf8").trim().split("\n"), ["buildx", "imagetools", "create", "--prefer-index=false", "--tag", tag, image]);
|
||||
} else {
|
||||
assert.notEqual(result.status, 0);
|
||||
assert.throws(() => readFileSync(calls), { code: "ENOENT" });
|
||||
}
|
||||
}
|
||||
} finally { rmSync(dir, { recursive: true, force: true }); }
|
||||
});
|
||||
|
||||
@@ -1,66 +0,0 @@
|
||||
#!/usr/bin/env node
|
||||
import { execFileSync } from "node:child_process";
|
||||
import { appendFileSync } from "node:fs";
|
||||
import { pathToFileURL } from "node:url";
|
||||
|
||||
export function cloudCacheCandidates(image, commits) {
|
||||
if (!/^ghcr\.io\/[a-z0-9._-]+\/[a-z0-9._-]+$/.test(image ?? "")) {
|
||||
throw new Error("Expected a GHCR owner/repository cache image.");
|
||||
}
|
||||
if (!Array.isArray(commits) || commits.length === 0 || commits.some((sha) => !/^[a-f0-9]{40}$/.test(sha))) {
|
||||
throw new Error("Cloud cache ancestry requires full commit SHAs.");
|
||||
}
|
||||
return [
|
||||
...[...new Set(commits)].slice(0, 10).map((sha) => `${image}:buildcache-cloud-${sha}`),
|
||||
`${image}:buildcache-cloud`,
|
||||
];
|
||||
}
|
||||
|
||||
export async function selectCloudCache(image, commits, {
|
||||
exists = registryCacheExists,
|
||||
log = console.log,
|
||||
} = {}) {
|
||||
for (const ref of cloudCacheCandidates(image, commits)) {
|
||||
try {
|
||||
if (!await exists(ref)) continue;
|
||||
log(`Using cloud cache: ${ref}`);
|
||||
return `type=registry,ref=${ref}`;
|
||||
} catch {
|
||||
// Cache availability must not turn an otherwise valid build into a
|
||||
// failure. A later ancestor may still be available during a rollout.
|
||||
log(`Could not inspect cloud cache ${ref}; trying the next ancestor.`);
|
||||
}
|
||||
}
|
||||
log("No cloud cache is available; this build will populate one.");
|
||||
return "";
|
||||
}
|
||||
|
||||
function registryCacheExists(ref) {
|
||||
try {
|
||||
// Use the preceding Docker login, including for private registry caches.
|
||||
// Inspect metadata only: no layer download and no image execution.
|
||||
execFileSync("docker", ["buildx", "imagetools", "inspect", "--raw", ref], {
|
||||
timeout: 10_000,
|
||||
maxBuffer: 1024 * 1024,
|
||||
stdio: ["ignore", "pipe", "pipe"],
|
||||
});
|
||||
return true;
|
||||
} catch (error) {
|
||||
if (/manifest unknown|not found|NAME_UNKNOWN/i.test(String(error.stderr ?? ""))) return false;
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
|
||||
if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) {
|
||||
try {
|
||||
if (!process.env.GITHUB_OUTPUT) throw new Error("GITHUB_OUTPUT is required.");
|
||||
const commits = execFileSync("git", ["rev-list", "--first-parent", "--max-count=10", "HEAD"], {
|
||||
encoding: "utf8",
|
||||
}).trim().split("\n");
|
||||
const source = await selectCloudCache(process.env.CACHE_IMAGE, commits, { exists: registryCacheExists });
|
||||
appendFileSync(process.env.GITHUB_OUTPUT, `source=${source}\n`);
|
||||
} catch (error) {
|
||||
console.error(error.message);
|
||||
process.exitCode = 1;
|
||||
}
|
||||
}
|
||||
@@ -1,65 +0,0 @@
|
||||
import assert from "node:assert/strict";
|
||||
import test from "node:test";
|
||||
import { cloudCacheCandidates, selectCloudCache } from "./select-cloud-cache.mjs";
|
||||
|
||||
const image = "ghcr.io/paperclipai/paperclip";
|
||||
const commits = ["a".repeat(40), "b".repeat(40), "c".repeat(40)];
|
||||
const candidates = cloudCacheCandidates(image, commits);
|
||||
|
||||
test("a same-SHA rerun imports only its existing cache", async () => {
|
||||
const inspected = [];
|
||||
const source = await selectCloudCache(image, commits, {
|
||||
exists: async (ref) => { inspected.push(ref); return true; }, log() {},
|
||||
});
|
||||
assert.equal(source, `type=registry,ref=${candidates[0]}`);
|
||||
assert.deepEqual(inspected, candidates.slice(0, 1));
|
||||
});
|
||||
|
||||
test("a new merge imports only its nearest available ancestor", async () => {
|
||||
const inspected = [];
|
||||
const source = await selectCloudCache(image, commits, {
|
||||
exists: async (ref) => { inspected.push(ref); return ref === candidates[1]; }, log() {},
|
||||
});
|
||||
assert.equal(source, `type=registry,ref=${candidates[1]}`);
|
||||
assert.deepEqual(inspected, candidates.slice(0, 2));
|
||||
assert.equal(source.includes("\n"), false);
|
||||
});
|
||||
|
||||
test("a still-building parent falls back to an older completed cache", async () => {
|
||||
assert.equal(await selectCloudCache(image, commits, {
|
||||
exists: async (ref) => ref === candidates[2], log() {},
|
||||
}), `type=registry,ref=${candidates[2]}`);
|
||||
});
|
||||
|
||||
test("the legacy cache is used only if no SHA cache exists", async () => {
|
||||
const inspected = [];
|
||||
assert.equal(await selectCloudCache(image, commits, {
|
||||
exists: async (ref) => { inspected.push(ref); return ref === candidates.at(-1); }, log() {},
|
||||
}), `type=registry,ref=${candidates.at(-1)}`);
|
||||
assert.deepEqual(inspected, candidates);
|
||||
});
|
||||
|
||||
test("missing caches permit a cold build", async () => {
|
||||
assert.equal(await selectCloudCache(image, commits, { exists: async () => false, log() {} }), "");
|
||||
});
|
||||
|
||||
test("a failed lookup can fall back without failing image publication", async () => {
|
||||
const messages = [];
|
||||
assert.equal(await selectCloudCache(image, commits, {
|
||||
exists: async (ref) => {
|
||||
if (ref === candidates[0]) throw new Error("registry temporarily unavailable");
|
||||
return true;
|
||||
},
|
||||
log: (message) => messages.push(message),
|
||||
}), `type=registry,ref=${candidates[1]}`);
|
||||
assert.match(messages[0], /Could not inspect cloud cache/);
|
||||
});
|
||||
|
||||
test("ancestry is bounded, deduplicated, and rejects output injection", () => {
|
||||
const many = Array.from({ length: 20 }, (_, i) => i.toString(16).padStart(40, "0"));
|
||||
assert.equal(cloudCacheCandidates(image, many).length, 11);
|
||||
assert.deepEqual(cloudCacheCandidates(image, [commits[0], commits[0]]), [candidates[0], candidates.at(-1)]);
|
||||
assert.throws(() => cloudCacheCandidates(`${image}\nsource=untrusted`, commits));
|
||||
assert.throws(() => cloudCacheCandidates(image, ["master"]));
|
||||
assert.throws(() => cloudCacheCandidates(image, []));
|
||||
});
|
||||
Reference in New Issue
Block a user