fix(ci): tolerate missing Docker cache blobs (#3945)

## Description

The Docker workflow on `main` failed when BuildKit restored a valid GHA
cache index whose backing blob had already been evicted. The ARM64
`slim-nonroot` build received `404 BlobNotFound`, so it never uploaded
its digest artifact and the manifest job failed secondarily.

This makes remote cache strictly best-effort: cache export errors are
ignored, and a failed cached build is retried once with remote cache
disabled. A real build failure remains fatal because the uncached retry
does not use `continue-on-error`.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature
- [ ] Breaking change
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring

## Changes Made

- Mark the cached bake attempt as recoverable and make GHA cache export
non-fatal.
- Retry the identical per-platform digest build without cache
import/export when the cached attempt fails.
- Select digest metadata from the attempt that actually succeeded.
- Add structural regression coverage for fallback failure semantics and
metadata selection.

## Testing

- [x] Unit tests pass (`tests/test_release_workflows.py`)
- [x] Linting passes for changed Python test
- [ ] Type checking passes (`mypy headroom`) — no typed Python
production code changed
- [x] New tests added for the fix
- [x] Manual workflow validation performed

### Test Output

```text
uv run --extra dev python -m pytest tests/test_release_workflows.py -q
54 passed in 2.79s

uv run --extra dev ruff check tests/test_release_workflows.py
All checks passed!

uv run --extra dev ruff format --check tests/test_release_workflows.py
1 file already formatted

actionlint v1.7.12 .github/workflows/docker.yml
(exit 0)

git diff --check upstream/main...HEAD
(exit 0)
```

A repository-wide `pytest -q` run completed with 14,208 passed and 617
skipped, plus 53 unrelated failures and 20 setup errors in existing
Windows/environment-sensitive suites. The changed release-workflow suite
passed within that run.

## Real Behavior Proof

- Environment: GitHub Actions Docker workflow on `main`, ARM64 native
runner.
- Exact failure:
https://github.com/headroomlabs-ai/headroom/actions/runs/36963024403/job/110700641054
- Observed result: BuildKit failed lazy cache copy with `404
BlobNotFound`; `digests-slim-nonroot-arm64` was absent and the matching
manifest job failed.
- Not tested: hosted fallback execution cannot be forced safely because
GitHub controls cache eviction; PR CI validates workflow syntax and
regression invariants.

## Runtime Rollout Safety

- Rollout-managed feature(s): none; CI-only change.
- Stable/default behavior changed: Docker publishing retries uncached
after a failed cached bake.
- Kill switch / disable path: revert this commit or remove the fallback
step.
- Qualification impact: none.
- Rollback path: revert the PR.

## Review Readiness

- [x] I performed self-review
- [x] PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I performed a self-review
- [x] I commented the cache failure behavior
- [x] Changes generate no new warnings
- [x] I added tests proving the fix
- [x] Relevant existing unit tests pass locally
- [x] I did not edit `CHANGELOG.md`

## Additional Notes

The fallback intentionally retries any failed cached bake, not only a
parsed cache error. That avoids brittle log matching; deterministic
build defects still fail on the required uncached attempt.
<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/3945"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->
This commit is contained in:
JD Davis
2026-10-02 10:59:08 -07:00
committed by GitHub
parent 038c923b4b
commit 540f4da199
2 changed files with 62 additions and 3 deletions
+29 -2
View File
@@ -124,6 +124,7 @@ jobs:
- name: Build and push by digest (single platform)
id: bake
uses: docker/bake-action@v7
continue-on-error: true
with:
files: |
./docker-bake.hcl
@@ -147,7 +148,33 @@ jobs:
*.platform=${{ matrix.arch.platform }}
*.output=type=image,name=${{ env.REGISTRY }}/${{ steps.image-name.outputs.image_name }},push-by-digest=true,name-canonical=true,push=true
*.cache-from=type=gha,scope=${{ matrix.variant.name || 'root' }}-${{ matrix.arch.name }}
*.cache-to=type=gha,mode=max,scope=${{ matrix.variant.name || 'root' }}-${{ matrix.arch.name }}
*.cache-to=type=gha,mode=max,ignore-error=true,scope=${{ matrix.variant.name || 'root' }}-${{ matrix.arch.name }}
# BuildKit can load a valid GHA cache index whose backing blob has
# since been evicted. The resulting lazy-copy 404 is fatal to cache
# import. Use a fresh daemon for the retry so the failed builder's
# imported records cannot be selected from its local state.
- name: Set up clean Docker Buildx fallback
id: fallback-buildx
if: steps.bake.outcome == 'failure'
uses: docker/setup-buildx-action@v4
# A real build failure remains fatal on this uncached attempt.
- name: Retry build without remote cache
id: bake-fallback
if: steps.bake.outcome == 'failure'
uses: docker/bake-action@v7
with:
builder: ${{ steps.fallback-buildx.outputs.name }}
no-cache: true
files: |
./docker-bake.hcl
cwd://${{ steps.meta.outputs.bake-file-labels }}
targets: ${{ matrix.variant.bake_target }}
push: true
set: |
*.platform=${{ matrix.arch.platform }}
*.output=type=image,name=${{ env.REGISTRY }}/${{ steps.image-name.outputs.image_name }},push-by-digest=true,name-canonical=true,push=true
- name: Export digest
id: digest
@@ -159,7 +186,7 @@ jobs:
# runner exports every env entry at bash spawn, so a large
# metadata blob fails the step with E2BIG before it runs.
cat > "${RUNNER_TEMP}/bake_meta.json" <<'__HEADROOM_BAKE_META_EOF__'
${{ steps.bake.outputs.metadata }}
${{ steps.bake.outcome == 'success' && steps.bake.outputs.metadata || steps.bake-fallback.outputs.metadata }}
__HEADROOM_BAKE_META_EOF__
digest="$(jq -r 'to_entries[0].value."containerimage.digest" // empty' \
"${RUNNER_TEMP}/bake_meta.json")"
+33 -1
View File
@@ -210,7 +210,39 @@ def test_docker_bake_metadata_never_travels_through_env() -> None:
if step.get("name") == "Export digest"
)
assert "<<'__HEADROOM_BAKE_META_EOF__'" in export["run"]
assert "${{ steps.bake.outputs.metadata }}" in export["run"]
assert (
"${{ steps.bake.outcome == 'success' && steps.bake.outputs.metadata || steps.bake-fallback.outputs.metadata }}"
in export["run"]
)
def test_docker_build_cache_failures_are_best_effort() -> None:
"""A missing remote cache blob must not block publishing an image."""
workflow = yaml.safe_load((ROOT / ".github" / "workflows" / "docker.yml").read_text())
build_steps = workflow["jobs"]["docker-build"]["steps"]
cached = next(step for step in build_steps if step.get("id") == "bake")
fallback_builder = next(step for step in build_steps if step.get("id") == "fallback-buildx")
fallback = next(step for step in build_steps if step.get("id") == "bake-fallback")
digest = next(step for step in build_steps if step.get("id") == "digest")
cached_overrides = cached["with"]["set"].splitlines()
fallback_overrides = fallback["with"]["set"].splitlines()
cache_to = next(line for line in cached_overrides if ".cache-to=" in line)
assert cached["continue-on-error"] is True
assert "ignore-error=true" in cache_to
assert fallback_builder["if"] == "steps.bake.outcome == 'failure'"
assert fallback_builder["uses"] == "docker/setup-buildx-action@v4"
assert fallback["if"] == "steps.bake.outcome == 'failure'"
assert fallback["uses"] == cached["uses"]
assert fallback["with"]["builder"] == "${{ steps.fallback-buildx.outputs.name }}"
assert fallback["with"]["no-cache"] is True
assert not any(".cache-from=" in line for line in fallback_overrides)
assert not any(".cache-to=" in line for line in fallback_overrides)
assert (
"steps.bake.outcome == 'success' && steps.bake.outputs.metadata || steps.bake-fallback.outputs.metadata"
in digest["run"]
)
def test_docker_latest_promotion_is_owned_by_root_manifest_cell() -> None: