ci: halve the isolated native Runner check by seeding it from the public master build cache (#13736)

## What

Recurring CI health check (PAP-31): on the most recent fully-green PR
run (`cb703ac`, run
[35519542997](https://github.com/paperclipai/paperclip/actions/runs/35519542997)),
the slowest check was **Compile isolated native Runner** at **452s** —
ahead of the largest test shards (379s). Every run recompiled the full
Rust dependency tree from zero, even though `docker.yml` already
refreshes a **public** `mode=max` BuildKit cache
(`ghcr.io/paperclipai/paperclip:buildcache-{amd64,arm64}`) on every
master push, containing exactly these layers. This PR seeds **only the
baseline build** with that registry cache, via anonymous pull.

**Measured on this PR's own CI (which exercises the seeded path): the
check completed in 123s, down from 452s — a 73% reduction, ~5.5 minutes
saved per run.**

## Thinking Path

Cost breakdown of the 452s from the job log: `cargo chef cook`
dependency compile 214.7s, local cache export 43.1s, runner-core build
37.0s, metadata proof layer 36.8s, `cargo install cargo-chef` 36.4s,
rebuild-verification build ~65s, setup/teardown ~20s. The dependency
compile and toolchain layers are identical to what the production
`docker.yml` build already caches publicly on every master push, so
recompiling them here bought no signal — the check's real assertions
live in the *verification* build, not the baseline. A first attempt used
`actions/cache` plus a master `push` trigger, but the CI bot's GitHub
App lacks `workflows` permission; the registry-cache approach is
strictly better anyway (shared across PRs immediately, no 10GB
Actions-cache quota pressure, no workflow change).

## What Changed

- `scripts/check-docker-runner-cache.sh`: the baseline build now adds
`--cache-from
type=registry,ref=ghcr.io/paperclipai/paperclip:buildcache-{amd64|arm64}`
(selected by host arch). `RUNNER_CHECK_SEED_CACHE` overrides the ref, or
set it empty to force the old cold path. The script header documents the
anonymous external read.
- `.github/workflows/docker-runner-check.yml` (comment-only): the stale
"no external cache" note now describes the anonymous GHCR seed and the
verification build's local-cache-only isolation. This was pushed in a
follow-up commit with workflow-edit permissions; the original CI-bot
token could not touch workflow files.

No Dockerfile stages or verification assertions changed.

## Verification

- This PR's own `Compile isolated native Runner` check runs the seeded
path (the script is in the workflow's trigger paths): **passed in 123s**
vs the 452s baseline.
- The rebuild-verification semantics are untouched: it still runs on a
**fresh builder** importing **only the local cache exported by this
run's baseline**, so it proves exactly what it proved before — that the
runner image rebuilds reproducibly from this run's own exported layers.
- Verified `ghcr.io/paperclipai/paperclip:buildcache-amd64` is
anonymously readable (unauthenticated manifest pull succeeds), so the
check gains no credential or secret dependency.

## Risks

- **Stale or missing seed cache:** if the GHCR ref is unreachable,
private, or garbage-collected, BuildKit logs a warning and falls back to
the pre-PR cold compile — the check gets slower, never wrong.
`RUNNER_CHECK_SEED_CACHE=""` restores the cold path explicitly.
- **Cache trust:** the seed only accelerates the *baseline* build; the
verification build still runs on a fresh builder against only this run's
locally exported cache, so a stale or poisoned registry cache cannot
make verification pass spuriously. The ref lives under
`ghcr.io/paperclipai/*`, written only by repo CI on master pushes.

## Model Used

Claude Fable 5 (`claude-fable-5`) via Paperclip agent **Bender
(Fable)**, issue PAP-31.

---------

Co-authored-by: Bender (Fable) <bender-fable@paperclip.local>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Devin Foley
2026-09-21 20:22:48 -07:00
committed by GitHub
co-authored by Bender Claude Fable 5
parent ded156a904
commit 3c2bc4f546
2 changed files with 29 additions and 2 deletions
+4 -1
View File
@@ -33,6 +33,9 @@ jobs:
uses: docker/setup-buildx-action@37fe631027851001ddb9b187196cc803df7f5f0e # v4
# Compile the real target, then change source in a disposable context.
# A fresh builder must import dependencies and produce changed binary metadata.
# No registry credentials, external cache, or image publication.
# The baseline build anonymously seeds from the public BuildKit cache at
# ghcr.io/paperclipai/paperclip:buildcache-{amd64,arm64}; the verification
# build imports only this run's locally exported cache on a fresh builder.
# No registry credentials or image publication.
- name: Verify native build and dependency cache reuse
run: bash scripts/check-docker-runner-cache.sh
+25 -1
View File
@@ -1,6 +1,10 @@
#!/usr/bin/env bash
# Build the real Docker target on two fresh builders using an exported cache.
# Export only metadata, avoiding a multi-gigabyte test image in the daemon.
# External access: the baseline build anonymously reads the public BuildKit
# cache at ghcr.io/paperclipai/paperclip:buildcache-{amd64,arm64} (see
# RUNNER_CHECK_SEED_CACHE below). No credentials are used or required, and
# nothing is pushed.
set -euo pipefail
repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
probe_dir="$(mktemp -d "${TMPDIR:-/tmp}/paperclip-runner-cache.XXXXXX")"
@@ -34,7 +38,27 @@ build_proof() {
docker buildx build --builder "$builder" --file "$probe_dir/cache-probe.Dockerfile" --target cache-proof-export --output "type=local,dest=$probe_dir/$result" --progress plain "$@" . 2>&1 | tee "$probe_dir/$result.log"
}
docker buildx create --name "$baseline_builder" --driver docker-container
build_proof baseline "$baseline_builder" --cache-to "type=local,dest=$probe_dir/cache,mode=max"
# Seed only the baseline with the public BuildKit cache that docker.yml
# refreshes on every master push. The rust stages consume none of that
# build's args, so their layer keys match, and mode=max re-exports the
# imported layers into $probe_dir/cache — the verification build below
# still proves what it always proved from this run's exported cache
# alone, on a fresh builder. Anonymous pull only, nothing is pushed; a
# missing or unreachable ref is a BuildKit warning and the baseline
# degrades to the previous cold compile. Set RUNNER_CHECK_SEED_CACHE to
# another ref, or to the empty string to force the cold path.
if [[ -z "${RUNNER_CHECK_SEED_CACHE+x}" ]]; then
case "$(uname -m)" in
x86_64) RUNNER_CHECK_SEED_CACHE="ghcr.io/paperclipai/paperclip:buildcache-amd64" ;;
aarch64 | arm64) RUNNER_CHECK_SEED_CACHE="ghcr.io/paperclipai/paperclip:buildcache-arm64" ;;
*) RUNNER_CHECK_SEED_CACHE="" ;;
esac
fi
seed_args=()
if [[ -n "$RUNNER_CHECK_SEED_CACHE" ]]; then
seed_args=(--cache-from "type=registry,ref=${RUNNER_CHECK_SEED_CACHE}")
fi
build_proof baseline "$baseline_builder" ${seed_args[@]+"${seed_args[@]}"} --cache-to "type=local,dest=$probe_dir/cache,mode=max"
# Removing the first builder proves the second build cannot use daemon-local
# state, and releases its disk space before importing the exported cache.
docker buildx rm "$baseline_builder"