mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(ci): tolerate missing Docker cache blobs (#3945)
## Description The Docker workflow on `main` failed when BuildKit restored a valid GHA cache index whose backing blob had already been evicted. The ARM64 `slim-nonroot` build received `404 BlobNotFound`, so it never uploaded its digest artifact and the manifest job failed secondarily. This makes remote cache strictly best-effort: cache export errors are ignored, and a failed cached build is retried once with remote cache disabled. A real build failure remains fatal because the uncached retry does not use `continue-on-error`. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature - [ ] Breaking change - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring ## Changes Made - Mark the cached bake attempt as recoverable and make GHA cache export non-fatal. - Retry the identical per-platform digest build without cache import/export when the cached attempt fails. - Select digest metadata from the attempt that actually succeeded. - Add structural regression coverage for fallback failure semantics and metadata selection. ## Testing - [x] Unit tests pass (`tests/test_release_workflows.py`) - [x] Linting passes for changed Python test - [ ] Type checking passes (`mypy headroom`) — no typed Python production code changed - [x] New tests added for the fix - [x] Manual workflow validation performed ### Test Output ```text uv run --extra dev python -m pytest tests/test_release_workflows.py -q 54 passed in 2.79s uv run --extra dev ruff check tests/test_release_workflows.py All checks passed! uv run --extra dev ruff format --check tests/test_release_workflows.py 1 file already formatted actionlint v1.7.12 .github/workflows/docker.yml (exit 0) git diff --check upstream/main...HEAD (exit 0) ``` A repository-wide `pytest -q` run completed with 14,208 passed and 617 skipped, plus 53 unrelated failures and 20 setup errors in existing Windows/environment-sensitive suites. The changed release-workflow suite passed within that run. ## Real Behavior Proof - Environment: GitHub Actions Docker workflow on `main`, ARM64 native runner. - Exact failure: https://github.com/headroomlabs-ai/headroom/actions/runs/36963024403/job/110700641054 - Observed result: BuildKit failed lazy cache copy with `404 BlobNotFound`; `digests-slim-nonroot-arm64` was absent and the matching manifest job failed. - Not tested: hosted fallback execution cannot be forced safely because GitHub controls cache eviction; PR CI validates workflow syntax and regression invariants. ## Runtime Rollout Safety - Rollout-managed feature(s): none; CI-only change. - Stable/default behavior changed: Docker publishing retries uncached after a failed cached bake. - Kill switch / disable path: revert this commit or remove the fallback step. - Qualification impact: none. - Rollback path: revert the PR. ## Review Readiness - [x] I performed self-review - [x] PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I performed a self-review - [x] I commented the cache failure behavior - [x] Changes generate no new warnings - [x] I added tests proving the fix - [x] Relevant existing unit tests pass locally - [x] I did not edit `CHANGELOG.md` ## Additional Notes The fallback intentionally retries any failed cached bake, not only a parsed cache error. That avoids brittle log matching; deterministic build defects still fail on the required uncached attempt. <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/3945" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end -->
This commit is contained in:
@@ -124,6 +124,7 @@ jobs:
|
||||
- name: Build and push by digest (single platform)
|
||||
id: bake
|
||||
uses: docker/bake-action@v7
|
||||
continue-on-error: true
|
||||
with:
|
||||
files: |
|
||||
./docker-bake.hcl
|
||||
@@ -147,7 +148,33 @@ jobs:
|
||||
*.platform=${{ matrix.arch.platform }}
|
||||
*.output=type=image,name=${{ env.REGISTRY }}/${{ steps.image-name.outputs.image_name }},push-by-digest=true,name-canonical=true,push=true
|
||||
*.cache-from=type=gha,scope=${{ matrix.variant.name || 'root' }}-${{ matrix.arch.name }}
|
||||
*.cache-to=type=gha,mode=max,scope=${{ matrix.variant.name || 'root' }}-${{ matrix.arch.name }}
|
||||
*.cache-to=type=gha,mode=max,ignore-error=true,scope=${{ matrix.variant.name || 'root' }}-${{ matrix.arch.name }}
|
||||
|
||||
# BuildKit can load a valid GHA cache index whose backing blob has
|
||||
# since been evicted. The resulting lazy-copy 404 is fatal to cache
|
||||
# import. Use a fresh daemon for the retry so the failed builder's
|
||||
# imported records cannot be selected from its local state.
|
||||
- name: Set up clean Docker Buildx fallback
|
||||
id: fallback-buildx
|
||||
if: steps.bake.outcome == 'failure'
|
||||
uses: docker/setup-buildx-action@v4
|
||||
|
||||
# A real build failure remains fatal on this uncached attempt.
|
||||
- name: Retry build without remote cache
|
||||
id: bake-fallback
|
||||
if: steps.bake.outcome == 'failure'
|
||||
uses: docker/bake-action@v7
|
||||
with:
|
||||
builder: ${{ steps.fallback-buildx.outputs.name }}
|
||||
no-cache: true
|
||||
files: |
|
||||
./docker-bake.hcl
|
||||
cwd://${{ steps.meta.outputs.bake-file-labels }}
|
||||
targets: ${{ matrix.variant.bake_target }}
|
||||
push: true
|
||||
set: |
|
||||
*.platform=${{ matrix.arch.platform }}
|
||||
*.output=type=image,name=${{ env.REGISTRY }}/${{ steps.image-name.outputs.image_name }},push-by-digest=true,name-canonical=true,push=true
|
||||
|
||||
- name: Export digest
|
||||
id: digest
|
||||
@@ -159,7 +186,7 @@ jobs:
|
||||
# runner exports every env entry at bash spawn, so a large
|
||||
# metadata blob fails the step with E2BIG before it runs.
|
||||
cat > "${RUNNER_TEMP}/bake_meta.json" <<'__HEADROOM_BAKE_META_EOF__'
|
||||
${{ steps.bake.outputs.metadata }}
|
||||
${{ steps.bake.outcome == 'success' && steps.bake.outputs.metadata || steps.bake-fallback.outputs.metadata }}
|
||||
__HEADROOM_BAKE_META_EOF__
|
||||
digest="$(jq -r 'to_entries[0].value."containerimage.digest" // empty' \
|
||||
"${RUNNER_TEMP}/bake_meta.json")"
|
||||
|
||||
@@ -210,7 +210,39 @@ def test_docker_bake_metadata_never_travels_through_env() -> None:
|
||||
if step.get("name") == "Export digest"
|
||||
)
|
||||
assert "<<'__HEADROOM_BAKE_META_EOF__'" in export["run"]
|
||||
assert "${{ steps.bake.outputs.metadata }}" in export["run"]
|
||||
assert (
|
||||
"${{ steps.bake.outcome == 'success' && steps.bake.outputs.metadata || steps.bake-fallback.outputs.metadata }}"
|
||||
in export["run"]
|
||||
)
|
||||
|
||||
|
||||
def test_docker_build_cache_failures_are_best_effort() -> None:
|
||||
"""A missing remote cache blob must not block publishing an image."""
|
||||
workflow = yaml.safe_load((ROOT / ".github" / "workflows" / "docker.yml").read_text())
|
||||
build_steps = workflow["jobs"]["docker-build"]["steps"]
|
||||
cached = next(step for step in build_steps if step.get("id") == "bake")
|
||||
fallback_builder = next(step for step in build_steps if step.get("id") == "fallback-buildx")
|
||||
fallback = next(step for step in build_steps if step.get("id") == "bake-fallback")
|
||||
digest = next(step for step in build_steps if step.get("id") == "digest")
|
||||
|
||||
cached_overrides = cached["with"]["set"].splitlines()
|
||||
fallback_overrides = fallback["with"]["set"].splitlines()
|
||||
cache_to = next(line for line in cached_overrides if ".cache-to=" in line)
|
||||
|
||||
assert cached["continue-on-error"] is True
|
||||
assert "ignore-error=true" in cache_to
|
||||
assert fallback_builder["if"] == "steps.bake.outcome == 'failure'"
|
||||
assert fallback_builder["uses"] == "docker/setup-buildx-action@v4"
|
||||
assert fallback["if"] == "steps.bake.outcome == 'failure'"
|
||||
assert fallback["uses"] == cached["uses"]
|
||||
assert fallback["with"]["builder"] == "${{ steps.fallback-buildx.outputs.name }}"
|
||||
assert fallback["with"]["no-cache"] is True
|
||||
assert not any(".cache-from=" in line for line in fallback_overrides)
|
||||
assert not any(".cache-to=" in line for line in fallback_overrides)
|
||||
assert (
|
||||
"steps.bake.outcome == 'success' && steps.bake.outputs.metadata || steps.bake-fallback.outputs.metadata"
|
||||
in digest["run"]
|
||||
)
|
||||
|
||||
|
||||
def test_docker_latest_promotion_is_owned_by_root_manifest_cell() -> None:
|
||||
|
||||
Reference in New Issue
Block a user