mirror of
https://github.com/ever-co/ever-gauzy.git
synced 2026-10-02 01:54:50 +08:00
perf(ci): build.yml installs the monorepo ONCE instead of five times in parallel
Five jobs each ran their own full `yarn install` of the same tree, concurrently, on the same self-hosted pool — so they competed with each other for I/O. Same commit, same run, measured on PR #10010 (run 32358690732): build-api install 1h49m -> passed build-monorepo-root install 1h51m -> passed build-libs install 2h40m -> KILLED at the 180-minute ceiling build-web install 2h40m -> KILLED at the 180-minute ceiling The two that died never compiled a line: `Build packages` and `Build web` are recorded as SKIPPED. Which two survive is a coin flip, and this gate has failed that way four times during this work on code that builds fine. `build-monorepo-root` already did install + postinstall and nothing else, so it becomes the producer: it publishes the tree as one compressed archive, and the other four `needs:` it and unpack in minutes instead of installing. A CACHE, not an artifact — the opposite choice to test_playwright.yml, for two reasons that only apply here. build.yml runs on every pull_request, so a ~2.9 GB artifact per run would meter against the org's Actions storage (the quota that just stopped Playwright reports uploading); caches are unbilled and live in a separate 10 GB per-repo budget. And a cache persists ACROSS runs keyed on the lockfile, so once develop populates it every PR restores rather than installing — which an artifact, scoped to its own run, cannot do. Net cache usage goes DOWN: `cache: yarn` is removed from all five jobs, which frees the 4.66 GB setup-node yarn cache it kept per ref, and the ~2.9 GB tree archive replaces it. The archive is only re-saved when yarn.lock, patches/ or .scripts/postinstall.js change, so most PRs restore and never write. The archive/restore shell is now shared with test_playwright.yml as .github/scripts/archive-node-modules.sh and restore-node-modules.sh — one copy behind eight call sites, rather than the drift that already bit the e2e workflow once. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Ruslan Konviser
co-authored by
Claude Opus 5
Claude Opus 5
parent
e78ac94a0d
commit
d7c56524d2
+69
-21
@@ -45,6 +45,9 @@ env:
|
||||
# Nx Cloud is disabled for this org; without this `nx run-many` hard-fails with
|
||||
# "Nx Cloud: Workspace is unable to be authorized. Exiting run."
|
||||
NX_NO_CLOUD: true
|
||||
# The dependency tree travels between jobs as this one compressed file. Workspace-relative so the
|
||||
# same string works for `tar` in a run step and for actions/cache.
|
||||
NODE_MODULES_ARCHIVE: node-modules.tar.zst
|
||||
|
||||
# Least-privilege scope for the automatic GITHUB_TOKEN.
|
||||
# This workflow only builds/tests/deploys from a checkout — read access is sufficient.
|
||||
@@ -67,20 +70,42 @@ jobs:
|
||||
- uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: 24
|
||||
cache: yarn
|
||||
|
||||
- name: Restore node_modules archive
|
||||
id: cache
|
||||
uses: actions/cache/restore@v4
|
||||
with:
|
||||
path: ${{ env.NODE_MODULES_ARCHIVE }}
|
||||
key: ${{ runner.os }}-${{ runner.arch }}-node-modules-${{ hashFiles('yarn.lock', 'patches/**', '.scripts/postinstall.js') }}
|
||||
|
||||
- name: Install dependencies
|
||||
if: steps.cache.outputs.cache-hit != 'true'
|
||||
run: yarn install --network-timeout 1000000 --frozen-lockfile --ignore-scripts
|
||||
|
||||
- name: Run postinstall manually
|
||||
if: steps.cache.outputs.cache-hit != 'true'
|
||||
run: yarn postinstall.manual
|
||||
|
||||
- name: Archive node_modules
|
||||
if: steps.cache.outputs.cache-hit != 'true'
|
||||
shell: bash
|
||||
run: .github/scripts/archive-node-modules.sh
|
||||
|
||||
- name: Save node_modules archive
|
||||
if: steps.cache.outputs.cache-hit != 'true'
|
||||
uses: actions/cache/save@v4
|
||||
# Explicit save so a FAILED install can never publish a half-built tree.
|
||||
with:
|
||||
path: ${{ env.NODE_MODULES_ARCHIVE }}
|
||||
key: ${{ runner.os }}-${{ runner.arch }}-node-modules-${{ hashFiles('yarn.lock', 'patches/**', '.scripts/postinstall.js') }}
|
||||
|
||||
# The strict library verdict, as its own early-reporting check: this is the task set whose
|
||||
# per-library `strictTemplates` tsconfigs catch template type errors (the class that broke the
|
||||
# demo webapp image). It has no `needs` edge and no app build behind it, so it reports as soon
|
||||
# as install + the 71 library builds finish.
|
||||
build-libs:
|
||||
name: build-libs
|
||||
needs: build-monorepo-root
|
||||
runs-on: ${{ vars.RUNNER_LINUX_X64_8 || 'ubuntu-latest' }}
|
||||
timeout-minutes: 180
|
||||
steps:
|
||||
@@ -89,19 +114,25 @@ jobs:
|
||||
- uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: 24
|
||||
cache: yarn
|
||||
|
||||
- name: Install dependencies
|
||||
run: yarn install --network-timeout 1000000 --frozen-lockfile --ignore-scripts
|
||||
- name: Restore node_modules archive
|
||||
uses: actions/cache/restore@v4
|
||||
with:
|
||||
path: ${{ env.NODE_MODULES_ARCHIVE }}
|
||||
key: ${{ runner.os }}-${{ runner.arch }}-node-modules-${{ hashFiles('yarn.lock', 'patches/**', '.scripts/postinstall.js') }}
|
||||
|
||||
- name: Run postinstall manually
|
||||
run: yarn postinstall.manual
|
||||
- name: Restore node_modules
|
||||
shell: bash
|
||||
# Unpacks the tree build-monorepo-root published, or installs from scratch if the cache is
|
||||
# unavailable. One step, so there is no `if:` on a previous step's outcome to get wrong.
|
||||
run: .github/scripts/restore-node-modules.sh
|
||||
|
||||
- name: Build packages
|
||||
run: yarn build:package:all
|
||||
|
||||
build-api:
|
||||
name: build-api
|
||||
needs: build-monorepo-root
|
||||
runs-on: ${{ vars.RUNNER_LINUX_X64_8 || 'ubuntu-latest' }}
|
||||
timeout-minutes: 180
|
||||
steps:
|
||||
@@ -110,19 +141,25 @@ jobs:
|
||||
- uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: 24
|
||||
cache: yarn
|
||||
|
||||
- name: Install dependencies
|
||||
run: yarn install --network-timeout 1000000 --frozen-lockfile --ignore-scripts
|
||||
- name: Restore node_modules archive
|
||||
uses: actions/cache/restore@v4
|
||||
with:
|
||||
path: ${{ env.NODE_MODULES_ARCHIVE }}
|
||||
key: ${{ runner.os }}-${{ runner.arch }}-node-modules-${{ hashFiles('yarn.lock', 'patches/**', '.scripts/postinstall.js') }}
|
||||
|
||||
- name: Run postinstall manually
|
||||
run: yarn postinstall.manual
|
||||
- name: Restore node_modules
|
||||
shell: bash
|
||||
# Unpacks the tree build-monorepo-root published, or installs from scratch if the cache is
|
||||
# unavailable. One step, so there is no `if:` on a previous step's outcome to get wrong.
|
||||
run: .github/scripts/restore-node-modules.sh
|
||||
|
||||
- name: Build API
|
||||
run: yarn build:api:prod:ci
|
||||
|
||||
build-web:
|
||||
name: build-web
|
||||
needs: build-monorepo-root
|
||||
runs-on: ${{ vars.RUNNER_LINUX_X64_8 || 'ubuntu-latest' }}
|
||||
timeout-minutes: 180
|
||||
steps:
|
||||
@@ -131,19 +168,25 @@ jobs:
|
||||
- uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: 24
|
||||
cache: yarn
|
||||
|
||||
- name: Install dependencies
|
||||
run: yarn install --network-timeout 1000000 --frozen-lockfile --ignore-scripts
|
||||
- name: Restore node_modules archive
|
||||
uses: actions/cache/restore@v4
|
||||
with:
|
||||
path: ${{ env.NODE_MODULES_ARCHIVE }}
|
||||
key: ${{ runner.os }}-${{ runner.arch }}-node-modules-${{ hashFiles('yarn.lock', 'patches/**', '.scripts/postinstall.js') }}
|
||||
|
||||
- name: Run postinstall manually
|
||||
run: yarn postinstall.manual
|
||||
- name: Restore node_modules
|
||||
shell: bash
|
||||
# Unpacks the tree build-monorepo-root published, or installs from scratch if the cache is
|
||||
# unavailable. One step, so there is no `if:` on a previous step's outcome to get wrong.
|
||||
run: .github/scripts/restore-node-modules.sh
|
||||
|
||||
- name: Build web
|
||||
run: yarn build:gauzy:prod:ci
|
||||
|
||||
build-desktop:
|
||||
name: build-desktop
|
||||
needs: build-monorepo-root
|
||||
# Matches the CircleCI branch filter: desktop was never built on ordinary PR branches.
|
||||
if: github.ref == 'refs/heads/develop' || github.ref == 'refs/heads/stage' || github.ref == 'refs/heads/master'
|
||||
runs-on: ${{ vars.RUNNER_LINUX_X64_8 || 'ubuntu-latest' }}
|
||||
@@ -154,7 +197,6 @@ jobs:
|
||||
- uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: 24
|
||||
cache: yarn
|
||||
|
||||
- name: Install system dependencies for Electron
|
||||
run: |
|
||||
@@ -162,11 +204,17 @@ jobs:
|
||||
sudo apt-get install -y --no-install-recommends \
|
||||
build-essential icnsutils graphicsmagick binutils libappindicator3-1 || true
|
||||
|
||||
- name: Install dependencies
|
||||
run: yarn install --network-timeout 1000000 --frozen-lockfile --ignore-scripts
|
||||
- name: Restore node_modules archive
|
||||
uses: actions/cache/restore@v4
|
||||
with:
|
||||
path: ${{ env.NODE_MODULES_ARCHIVE }}
|
||||
key: ${{ runner.os }}-${{ runner.arch }}-node-modules-${{ hashFiles('yarn.lock', 'patches/**', '.scripts/postinstall.js') }}
|
||||
|
||||
- name: Run postinstall manually
|
||||
run: yarn postinstall.manual
|
||||
- name: Restore node_modules
|
||||
shell: bash
|
||||
# Unpacks the tree build-monorepo-root published, or installs from scratch if the cache is
|
||||
# unavailable. One step, so there is no `if:` on a previous step's outcome to get wrong.
|
||||
run: .github/scripts/restore-node-modules.sh
|
||||
|
||||
- name: Build desktop
|
||||
run: yarn build:desktop
|
||||
|
||||
@@ -96,46 +96,8 @@ jobs:
|
||||
# was killed at the 120-minute job timeout 1 h 58 m in. All four shards died that way and
|
||||
# the 12 h 34 m run produced ZERO test results.
|
||||
#
|
||||
# And this job no longer writes a cache AT ALL. After pruning duplicates the budget holds
|
||||
# 9.33 GB of caches that other workflows genuinely need — `build.yml`'s 4.66 GB yarn cache
|
||||
# (the PR gate, five jobs, every pull request) and the 4.64 GB macOS desktop-build cache.
|
||||
# Storing even a ~3 GB archive would evict one of them, making the PR gate slower to speed up
|
||||
# a suite that runs only on stage pushes. That is a bad trade, so deps pays the bootstrap and
|
||||
# the artifact makes sure nothing downstream pays it again.
|
||||
#
|
||||
# zstd on one file also beats the cache action's per-file handling of a huge tree in both
|
||||
# directions. `--warning=no-file-changed` because yarn's own daemon can touch files while we
|
||||
# read them; `|| [ $? -eq 1 ]` accepts tar's "file changed as we read it" (exit 1) but still
|
||||
# fails on a real error (exit 2).
|
||||
run: |
|
||||
set -o pipefail
|
||||
PATHS=()
|
||||
for p in node_modules apps/*/node_modules packages/*/node_modules \
|
||||
packages/plugins/*/node_modules tools/node_modules; do
|
||||
[ -d "$p" ] && PATHS+=("$p")
|
||||
done
|
||||
if [ ${#PATHS[@]} -eq 0 ]; then
|
||||
echo "::error::bootstrap produced no node_modules directories"; exit 1
|
||||
fi
|
||||
echo "archiving ${#PATHS[@]} directories"
|
||||
if command -v zstd >/dev/null 2>&1; then
|
||||
COMPRESS="zstd -3 -T0"
|
||||
else
|
||||
echo "::warning::zstd unavailable — falling back to gzip (larger archive, slower)"
|
||||
COMPRESS="gzip -3"
|
||||
fi
|
||||
tar --warning=no-file-changed --use-compress-program="$COMPRESS" \
|
||||
-cf "$NODE_MODULES_ARCHIVE" "${PATHS[@]}" || [ $? -eq 1 ]
|
||||
ls -lh "$NODE_MODULES_ARCHIVE"
|
||||
# Tolerating tar's exit 1 means a TRUNCATED archive could be published, and every
|
||||
# downstream job would then fall back to its own multi-hour bootstrap. A cheap size floor
|
||||
# catches the gross case; a full `tar -t` would re-read every byte for little extra.
|
||||
size=$(stat -c%s "$NODE_MODULES_ARCHIVE")
|
||||
echo "archive bytes: $size"
|
||||
if [ "$size" -lt 104857600 ]; then
|
||||
echo "::error::archive is only $size bytes — the dependency tree cannot be that small"
|
||||
exit 1
|
||||
fi
|
||||
# Shares its implementation with build.yml — see the script.
|
||||
run: .github/scripts/archive-node-modules.sh
|
||||
|
||||
- name: Upload node_modules archive
|
||||
uses: actions/upload-artifact@v7
|
||||
|
||||
Reference in New Issue
Block a user