## Related Issues
Follow-up to #8229.
## Summary of Changes
Locate the private auto-testing checkout from the lane root or the workspace root so the nested security checkout can record functional-chain evidence.
## Verification
The exact PR head b66129ab9f passed one mechanical correctness and simplicity review, nine real-Git layout and provenance checks, and sixteen existing evidence/envelope tests. The baseline sibling layout failed with git exit 128; the corrected layout succeeded while revision mismatches and missing checkouts stayed rejected. Current PR checks are completed with successful or skipped conclusions, including the aggregate.
## Impact
Both lane and private-script revision checks remain intact. No time limits, assertions, production behavior, or evidence validation requirements change. The synthetic layout checks do not execute the actual scheduled security suite; integrated main CI and release acceptance remain separate gates.
## Additional Notes
Approved review 5374559393 is bound to the exact head above. Reverting the single-file change restores the previous checkout lookup.
The security workflow checks the repository out into rustfs-repo/ (#7212)
but its chain evidence steps still invoke
scripts/functional_chain_evidence.py relative to the workspace root, which
on the persistent shared runner resolves to a stale checkout left by another
job. The evidence gate then compares that checkout's HEAD against the chain
workflow_sha and rejects the lane before any case runs
("lane checkout differs from chain workflow source").
Run the evidence script from the lane's own checkout so ROOT resolves to
rustfs-repo/, whose HEAD is exactly the chain-pinned workflow_sha.
* ci: give the pool run a real wall-clock budget
The pool suite budgets up to 24h each for rebalance and decommission
(REBALANCE_TIMEOUT / DECOMMISSION_TIMEOUT in rustfs_pool_expand.sh),
but the workflow step capped it at 45 minutes. Four consecutive runs
died identically: steps 1-7 all PASS, then the rebalance wait was
killed at exactly 45:12 - chain 35680308150, standalone 35702803577,
chain 35757773373, chain 36283120750 - with rebalance at completed=1/4
(~21 minutes in), so a full pass has never been observed.
Make the budget an input (default 240 minutes: covers the observed
rebalance pace plus one decommission pass) and document why. The suite
keeps failing the job through its [POOL-STEP] marker adjudication.
* fix(ci): align pool job budget and timeout contract tests
* fix(connect): preserve legacy heartbeats and isolate I/O tests
---------
Signed-off-by: Hauser <housemecn@gmail.com>
Co-authored-by: overtrue <anzhengchao@gmail.com>
Co-authored-by: Hauser <housemecn@gmail.com>
Co-authored-by: RustFS <hello@rustfs.com>
The contract check asserts exact lines in build.yml, and five recent
build-workflow PRs drifted it out of sync, so Security Audit's Workflow
Pin Report job fails for every PR and every push to main since Sep 27
17:34Z (first failure on push run 36337450094). Refresh the assertions
to the current, intentional shapes - contract intent is preserved and
in fact tightened:
- #8160 added PROFILE_ARGS to the cross/native cargo build lines.
- #8171 added 'should_publish == true' as a stricter prefix on the R2
publication guard and the create-release / upload-release-assets /
publish-release / update-latest-version conditions.
Verified locally: the full check (including the docker and helm
workflow contract tests) passes against main's files with these
assertions.
Route the two linux-aarch64-* release legs to the dedicated arm64 runner
and build them natively (cross: false) instead of cross-compiling via
cargo-zigbuild on x86_64 hosts.
With native legs now executable, extend the packaged-artifact checks that
were previously gated to x86_64-unknown-linux-gnu only:
- Every Linux leg runs rustfs --version and rustfs-cli --help from its own
package, proving the artifact matches the runner architecture.
- The packaged-console runtime smoke (server boot + console HTTP 200) also
covers aarch64-unknown-linux-gnu, so both architectures get full runtime
verification. Musl legs run the binary checks but skip server boot for
now.
* ci: run the chain orchestration jobs where the gh CLI exists
#8112 moved these jobs to the sm-standard-2 pool, whose images ship
without the gh CLI (a known property of the build fleet, see #7572).
Last night's chain died in prepare before any suite ran:
resolve_functional_candidate.py:34 does subprocess.run(["gh", "api"])
and got FileNotFoundError; complete-chain and the hourly
functional-chain-health canary fail the same way. On Sep 22/23 the same
jobs ran green on GitHub-hosted runners.
Move prepare, complete-chain and the health canary to ubuntu-latest.
The suite lanes stay on smoke-testing.
* ci: run the chain orchestration jobs where the gh CLI exists
#8112 moved these jobs to the sm-standard-2 pool, whose images ship
without the gh CLI (a known property of the build fleet, see #7572).
Last night's chain died in prepare before any suite ran:
resolve_functional_candidate.py:34 does subprocess.run(["gh", "api"])
and got FileNotFoundError; complete-chain and the hourly
functional-chain-health canary fail the same way. On Sep 22/23 the same
jobs ran green on GitHub-hosted runners.
Move prepare, complete-chain and the health canary to ubuntu-latest.
The suite lanes stay on smoke-testing.
* ci: point the chain orchestration jobs at the smoke-testing runner
Per maintainer decision, consolidate them onto the same runner as the
test lanes. Verified on rustfs-smoke-testing as the runner user:
gh 2.45.0, jq 1.7.
* ci: point the chain orchestration jobs at the smoke-testing runner
Per maintainer decision, consolidate them onto the same runner as the
test lanes. Verified on rustfs-smoke-testing as the runner user:
gh 2.45.0, jq 1.7.
Companion to rustfs/auto-testing#109: the 4x1 topology is removed from
MATRIX_TOPOS because the product rejects a 1-drive-per-node distributed
pool at startup (FATAL BelowMinimum). The unfiltered sweep is now 36
combinations / 300 cases; update the contract totals and comments so an
unfiltered sweep is still held to an exact expected case count.
#8112 replaced ubuntu-latest with sm-standard-2 across workflows. Both nightly
KMS Vault lanes need a working Docker daemon; the sm-standard-2 pool does not
provide one, so both lanes die in seconds on 'Docker is not available' while
the package compiles and publishes fine - and the functional chain never fires
because its gate requires a successful build run. Restore the lanes to
ubuntu-latest, as documented in the lane comment before #8112.
* ci: replace ubuntu-latest runner with sm-standard-2 across workflows
* ci: keep scheduled-validation monitors on hosted runners
The freshness and watchdog jobs report stalled scheduled validations.
Running them on the same sm-standard-2 pool means a pool outage stalls
the monitors too, so nothing reports it.
---------
Co-authored-by: overtrue <anzhengchao@gmail.com>
* ci(package): build gnu and musl DEB/RPM variants with distinct file names
The Build and Release workflow produces four Linux binaries
(x86_64-gnu, aarch64-gnu, x86_64-musl, aarch64-musl), but packaging
only consumed the two gnu artifacts. Add matrix entries for the two
musl artifacts so every release ships all four DEB/RPM variants.
The libc variant is now part of the package file names, which would
otherwise collide between gnu and musl builds of the same version:
- deb: rustfs_<version>_<libc>_<arch>.deb
- rpm: rustfs-<libc>-<version>-<release>.<arch>.rpm
The dpkg Package and rpm Name stay plain "rustfs", so gnu and musl
remain mutually exclusive upgrades of one package rather than
co-installable packages fighting over /usr/bin/rustfs.
Dependency declarations now follow the linkage: gnu binaries
dynamically link glibc and keep Depends: libc6 (>= 2.31) /
glibc >= 2.31; musl binaries are statically linked and declare no
libc dependency. The libc variant is also visible in the package
description.
scripts/release/package_versions.sh gains a LIBC argument and its
contract tests cover both variants plus the invalid-libc cases.
* ci(package): align deb/rpm file names with the zip artifact naming
Rename the package file names so every release asset of one build
shares the same stem as its binary artifact, differing only by
extension:
- before: rustfs_<deb_version>_<libc>_<deb_arch>.deb
rustfs-<libc>-<rpm_version>-<rpm_release>.<rpm_arch>.rpm
- after: rustfs-linux-<arch>-<libc>-v<version>.deb / .rpm
e.g. rustfs-linux-x86_64-gnu-v1.0.0.zip,
rustfs-linux-x86_64-gnu-v1.0.0.deb,
rustfs-linux-x86_64-gnu-v1.0.0.rpm.
Non-development builds embed the raw release tag (with 'v'), like the
zips; development builds embed dev-<full sha>. The dpkg/rpm versions
(including the '~' prerelease ordering) are unchanged - they live in
the package metadata, and a side effect is that release asset names no
longer contain '~' (which GitHub normalizes to '.').
package_versions.sh now takes the target arch (x86_64|aarch64) instead
of the deb/rpm arch pair; the deb Architecture (amd64/arm64) in the
control metadata still comes from the workflow matrix. The two test
workflows that assemble deb download URLs from a release tag
(rustfs-table-test, rustfs-upgrade-test) are updated to the new name,
which also removes their '~'-to-'.' asset name workaround.
Adds rustfs-fault-tolerance-matrix.yml, a manually dispatched workflow that
runs the --matrix topology x EC outage sweep from rustfs/auto-testing (38
combinations, 320 cases). The scenario suite hardcodes --all and a 60-minute
budget, so the exhaustive sweep had no CI entry point.
The 09-22 nightly chain failed all 12 lanes in seconds at the 'Bind
functional candidate' step:
ValueError: private script pin differs from chain
resolve_functional_candidate.py's >24h staleness fallback (added by
#8026, made functional by #8041's token fix) legitimately sets
manifest.testing_sha to auto-testing main HEAD, but current_chain()
still required it to equal .config/functional-script-revision.txt -
a check written for the pre-fallback world where the two could never
diverge. Once the fallback finally fired, prepare produced testing_sha
21edcf4 while the pin file still holds 27e9584 and every lane aborted
before checking out the test scripts.
Drop the pin-file comparison and keep what the lane actually needs to
guarantee: testing_sha is a valid commit sha (current_chain), the lane
checked out exactly that sha (record: private_head == testing_sha, kept
as-is), and the health checker validates the same format instead of
re-reading the pin file. Tests updated: a fallback testing_sha that
differs from the pin is accepted; a non-sha testing_sha is rejected.
Verified: python3 -m unittest test_functional_chain
test_functional_chain_health -> 39 tests OK.
#8052 renamed the fault-tolerance step to 'Run fault-tolerance
scenarios (A, B, C, C2, D, E)' and raised the evidence gate to 44
cases, but scripts/test_security_workflow.py still expected the old
step name and a 38-case fixture, breaking Quick Checks with three
KeyErrors and one assertion failure. Update DIRECT_TESTS and the
write_result fixture to the new contract.
The fault-tolerance suite now registers and runs scenario E
(rustfs/auto-testing#100): --all covers A, B, C, C2, D, E with 38 + 6
= 44 expected cases. The chain-evidence assertion still hardcoded
[A, B, C, C2, D] and 38 everywhere, which would fail every green FT
run's evidence validation once E is registered.
Bump the scenario list to include E and all count assertions from 38
to 44; step name updated to match. No other lanes reference 38.
resolve_functional_candidate.py probes rustfs/auto-testing (private) to
age the pinned functional-script revision and fall back to main HEAD
after 24h. The prepare step passed github.token, which cannot see the
private repo, so every nightly chain logged
staleness probe failed (...exit status 1.); keeping pinned revision
and replayed the 09-14 harness. On the 09-20 nightly that harness died
on the dpkg conffile prompt in all 12 lanes (see rustfs/auto-testing#97)
because the --force-confold and other fixes never reached the chain.
Use PF_TESTING_GH_TOKEN - already required by the other steps in this
workflow - so the probe can actually run and the >24h fallback works.
The functional chain pins the test harness to the revision recorded in
.config/functional-script-revision.txt, which is only refreshed by
release syncs - it went stale on Sep 16 and the nightly chain has been
replaying a Sep-17-era harness ever since. Last night that meant:
the performance lane died on a dpkg conffile prompt (fix merged as
#88 but never reached the chain), the FT lane died on 'unknown option:
--results-file' (harness/workflow version mismatch), and the
STS-105 signing fix (#94) sat untested.
When the pinned revision's commit is older than 24 hours, fall back to
auto-testing main HEAD so the chain always runs the current harness.
Any staleness-probe failure keeps the pinned revision (fail-safe).
The revision file stays in place as the audit trail and the release
sync continues to manage it.
* fix(package): preserve service state across upgrades
* fix(package): match legacy DEB versions in tilde form
Published prerelease packages carry ~ in the dpkg control version
(package_versions.sh maps the SemVer prerelease - to ~), so the
legacy fallback list written with dots never matched 1.0.0~rc.x and
upgrades away from those DEBs still left the service stopped (#8011).
Fix the legacy glob to the tilde form and update the contract test,
which had enshrined the dot form. Verified on Ubuntu 24.04 systemd
containers: DEB upgrade 1.0.0~rc.5 -> 1.0.1 now keeps the service
running; 1.0.0 -> 1.0.1 and 1.0.1 -> 1.0.2 marker path still pass.
* docs(package): expand /etc/default/rustfs example template
Document the commonly used RUSTFS_* settings as commented examples in
the packaged conffile and point to docs.rustfs.com.
The performance suite runs on the pf-testing fleet - a completely
separate environment (own runner, own target nodes) from the
smoke-testing fleet the functional suites share. Holding
rustfs-shared-functional-tests-v2 made an in-flight performance run
block every functional dispatch (observed: kms queued pending with
zero jobs for 4+ hours behind a 4-hour performance run), and
performance benchmarks queued behind functional suites in return.
Give it its own group; nothing else changes.
The fault-tolerance suite was the only one without a dashboard upload
step - its reports never reached functional-reports/fault-tolerance/
without manual uploads. Add the standard upload step (continue-on-error,
same shape as the other ten suites), placed before the artifact upload.
Co-authored-by: Hauser <housemecn@gmail.com>
The fault-tolerance suite (disk-outage and peer-failure scenarios,
FT-CASE verdicts) has never been dispatchable: the workflow file only
exists on release, and GitHub registers workflow_dispatch targets
exclusively from the default branch. The suite is otherwise ready - it
runs on the smoke-testing fleet and already uses the
rustfs-shared-functional-tests-v2 lock.
Adds the workflow adapted to this branch's current (pre-sync) gate
semantics - the run step's continue-on-error is dropped so the existing
contract holds - plus the evidence init aligned with the shared
four-variable pattern (FUNCTIONAL_ARTIFACTS_DIR under
rustfs-fault-tolerance-*, LOG_FILE/REPORT_FILE/EVIDENCE_DIR/TMPDIR,
scratch dir) and the standard artifact allowlist.
Contract registration: JOBS / DIRECT_TESTS entries, upload allowlist,
and fixture teaching for the FT-specific shapes (composite
package-url fallback expression, script-driven --cleanup instead of
ssh-driven cleanup). The generic legacy matrices exclude FT the same
way they exclude table/performance until the release->main sync lands.
ci(chain): gate the performance lane on its fleet probe (rebase + contract)
Rebased onto latest main; adds the contract-test update that the first
push of this branch missed:
- test_functional_chain: the performance lane now asserts the new
probe-gated 'if' (prepare success AND performance_ready == online);
the other nine lanes keep the plain prepare gate; prepare must carry
the two preflight steps (hard smoke-testing check, pf-testing probe)
and publish the performance_ready output
Carried over unchanged from the previous push: per-fleet preflight
(prepare hard-fails only on smoke-testing), the pf-testing probe step,
the probe-gated performance lane, and aggregate's --allow-skipped
(permitted skips shrink the evidence set; unpermitted skips still fail
aggregation). Chain contract tests: 27/27 locally.
* ci: register the table suite workflow on the default branch
The table suite workflow (#7873) only exists on release so far, and
GitHub registers workflow_dispatch targets exclusively from the default
branch — the suite cannot be triggered until this file lands on main
(same mechanism that keeps the fault-tolerance suite undispatchable).
This adds the identical rustfs-table-test.yml (byte-for-byte the
release version) plus the contract-test registration (JOBS /
DIRECT_TESTS / upload allowlist), and nothing else from release.
* ci: pin actions in the table workflow to full commit SHAs
The default branch enforces check_workflow_pins.sh (unpinned
third-party action refs fail CI). Use the same pinned SHAs as the other
suite workflows on main: actions/checkout@9c091bb... (#v7) and
actions/upload-artifact@b7c566a... (#v6).
Note: the release copy of this workflow still uses @v4 tags; release
does not run the pin gate. Release and main will converge on the pinned
form during the full sync.
#7847 routed schedule-failure alerting to rustfs/backlog by switching the
alert jobs' token to secrets.BACKLOG_ISSUE_TOKEN, but check_test_wiring.py
still asserts the literal 'github-token: ${{ secrets.GITHUB_TOKEN }}', so
Quick Checks fails on every open PR. Update the wiring assertions and the
self-test fixtures to the new literal.
Both `--self-test` (52 tests) and the full `validate()` pass locally.
Scheduled pipeline failures currently open [scheduled-failure] tracking
issues in rustfs/rustfs itself, adding bot noise to the code repository's
issue tracker. Route that alerting to rustfs/backlog instead.
- schedule-failure-issue gains target-repository (default rustfs/backlog);
dedupe lookup, comment appends and issue creation target that repository.
- New source-token input (default ${{ github.token }}) reads the failed-job
list from the reported run; the run still lives in and links to
rustfs/rustfs. The issue body now carries a '- Repository:' line.
- All consumer workflows pass secrets.BACKLOG_ISSUE_TOKEN; the
workflow-scoped GITHUB_TOKEN cannot open issues cross-repo. Alert job
permissions blocks are unchanged.
Adds a fault-tolerance suite that verifies read/write behavior under
drive and node loss against the erasure-coding contract and snapshots
health-endpoint responses at every degradation tier. Scenarios derived
from product source (default_parity_count, erasure set sizing):
- A: single-node 4 drives (EC:2, read quorum 2): hide 1/2/3 drives
- B: multi-node 4x1 (one set of 4, EC:2): stop 1/2/3 nodes
- C: multi-node 4x4 (one set of 16, EC:4, read quorum 12): 1 node down
lands exactly on the read-quorum boundary; 2 nodes down breaks it
- C2: multi-node 4x4 with RUSTFS_STORAGE_CLASS_STANDARD=EC:8 (read
quorum 8, write quorum 9, lock majority 9): 2 nodes down puts reads
inside the reported divergence window (read quorum met while the
lock majority is broken)
By default a reads-refused-despite-met-read-quorum observation is
reported as known-divergence without failing the suite; the strict
input escalates it. Chain order becomes:
upgrade -> s3 -> kms -> tier -> storage -> heal -> pool -> security ->
replication -> fault-tolerance -> performance.
feat(nightly): publish packages as assets of the rolling 'nightly' release (#7592)
Replace the assets-branch scheme with a proper GitHub Release on
rustfs/auto-testing: a single 'nightly' release whose deb/rpm assets
are replaced in place on every build. This is the standard channel —
visible on the repo's Releases page, stable download URLs, no git
history growth (release assets live outside the repository).
- New scripts/release/publish_nightly_assets.sh: resolves-or-creates
the 'nightly' release via the REST API, deletes same-name assets,
uploads rustfs-nightly-latest.{deb,rpm}, then PATCHes the release
body with the build provenance (ref@sha, run link, sizes, SHA256).
Plain curl + python3, no gh CLI (the build fleet has none — #7586).
- The workflow step shrinks to invoking the script; full flow
exercised end-to-end against the real release with probe files
(create / upload / overwrite / download round-trip / body update).
The sm-standard-4 runners used by the nightly build have no gh binary
(functional-chain workflows run elsewhere, on the jumpbox). The assets
publish step died on 'gh: command not found' at the credential-helper
setup, and the preceding clone failure had been masked by 2>/dev/null,
misleading the step into the orphan path. Swap clone and remote setup
to plain git with the token embedded in the URL; push semantics are
unchanged.
Two fixes for the functional test chain:
1. Report upload fails with 'jq: Argument list too long' when the base64
report is passed through '--arg content' (pool reports exceed the OS
argv limit; last night's pool run lost its Step Results report this
way). Write the base64 payload to a temp file and load it in jq via
--rawfile instead. Applied uniformly to all nine suite workflows
that share this upload step.
2. The security workflow cloned rustfs/auto-testing into the workspace
and then ran actions/checkout at the workspace root for the OIDC
live gate script, which wiped the auto-testing clone and killed the
suite with 'chmod: cannot access auto-testing/rustfs-security-test.sh'.
Check out the repository into the rustfs-repo/ subdirectory instead
and point RUSTFS_SECURITY_OIDC_LIVE_SCRIPT there.
Co-authored-by: rustfs-ci <ci@rustfs.com>
The workflow_dispatch inputs already accept arbitrary release tags, but
the run failed late and unclearly when a tag had no .deb asset, and the
from_version default pointed at 1.0.0-rc.4-preview.1, whose release
ships no .deb at all - so scheduled runs died on a 404 while installing
the old package.
- Add a fail-fast preflight that resolves each requested tag via the
GitHub release API and verifies the rustfs_<tag>_amd64.deb asset
exists before the suite starts, with an actionable error message
otherwise (e.g. 1.0.0-rc.4 ships only zip/sbom assets).
- Change the from_version default to 1.0.0-rc.3, the newest release
that actually ships a .deb asset.
- Reword the from_version/to_version descriptions so manual triggers
state the .deb-asset requirement and the nightly fallback.
- Pass PF_TESTING_GH_TOKEN as GH_TOKEN to the suite step for the gh api
release lookups, matching the other functional workflows.
Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
* ci(upgrade): render an upgrade matrix in the report; fix from default
The upgrade report only ever showed the requested deb URLs and a case
table. The nightly chain runs died installing the OLD package (default
from_version 1.0.0-rc.4-preview.1 has no .deb asset on its release, and
release 1.0.0-rc.4 ships none either), leaving an empty Total: 0 report
with no indication of what was upgraded.
- Default from_version is now 1.0.0-rc.3 (ships rustfs_1.0.0.rc.3_amd64.deb).
Matches the auto-testing default from PR #32.
- The report generator also parses the [UPG-TOPO] lines the suite now
emits and renders an 'Upgrade Matrix' section: per topology and KMS
backend, the versions actually in place before/after (captured via
'rustfs --version' on the node) and the aggregated result. When the
suite dies before any topology completes, the matrix says so instead
of silently showing nothing.
* fix(ci): use English headers in the upgrade matrix table
* ci(heal,pool): render step results and version in the reports
The heal and pool-expansion reports were only a raw log tail: no
structured indication of which steps passed, no overall verdict, and no
version information for the cluster under test.
The suites now emit machine-readable lines (auto-testing PR):
[HEAL-STEP] <n> <desc> PASS|FAIL [POOL-STEP] <n> <desc> PASS|FAIL
[HEAL-VERSION] <ver> (node <n>) [POOL-VERSION] <ver> (node <n>)
[HEAL-RESULT] PASS|FAIL <detail> [POOL-RESULT] PASS|FAIL <detail>
Both report generators parse them and emit a '## Step Results' section
before the log tail: the version captured in place via 'rustfs --version'
on a node, the overall verdict, and a per-step table. When a run dies
before any step reports (old script or early crash), the table shows a
NOT RUN placeholder row instead of silently showing nothing.
PR #7023 rewrote the handoff retry scripts with shell parameter
expansions collapsed into Actions expression syntax: ${GH_TOKEN:-},
${{attempt}}, ${{DISPATCHED:-0}}, ${{TITLE}} etc. GitHub parses
${{...}} as workflow expressions, and bare identifiers are invalid
there, so all seven shared-VM suite workflows (upgrade, s3-compat, kms,
tier, storage, heal, pool-expand) were rejected as invalid workflow
files on main.
Symptoms since 2026-09-01 23:11 +0800 (bba9347):
- every push to any branch produced 'failure' runs with no jobs
('This run likely failed because of a workflow file issue')
- the nightly functional chain dispatched rustfs-chain-upgrade at
17:08Z but the event was silently dropped: zero repository_dispatch
runs for all eight shared-VM suites overnight (only performance,
whose file was untouched, ran)
- the workflows API listed them by path instead of name
Fix: restore the shell expansions (${VAR}, ${VAR:-default}); quote the
expected-event name without legacy backticks; render the markdown fence
via printf so shellcheck can parse the block. actionlint and YAML
validation now pass clean on all eleven rustfs-*.yml workflows.
- New RustFS Replication Test workflow (rustfs-replication-test.yml):
standalone workflow_dispatch (suite selector bucket/site/all) and
repository_dispatch rustfs-chain-replication; runs on the shared
smoke-testing runner under the shared functional concurrency group.
- Suite never fails the workflow (continue-on-error): failures are filed
as redacted issues in rustfs/backlog (deduped per run) and the report is
uploaded to rustfs/dashboard functional-reports/replication/<date>.md.
- Security now hands off to Replication, making it the tenth and final
link: upgrade -> s3 -> kms -> tier -> storage -> heal -> pool ->
security -> replication (performance stays parallel on pf-testing).
- Depends on rustfs/auto-testing#27 (rustfs-replication-test.sh).
Co-authored-by: houseme <housemecn@gmail.com>
The repository_dispatch handoff step was continue-on-error with a single
attempt: if the call failed (token lacking contents:write, transient API
error), the chain stalled silently while every job stayed green.
Each handoff now retries 3x and, if all attempts fail, files an alert
issue in rustfs/backlog with the exact recovery command before exiting 1
(still continue-on-error, so suite workflows themselves never fail).
Problem: the nightly functional chain has not completed end-to-end.
Evidence from recent runs:
- workflow_run events are fire-and-forget: after KMS finished at 17:09Z
on 8/31 no tier run was created; rustfs-storage-test.yml has never run.
- 'if: conclusion == success' gates skip downstream suites on any
failure (security was skipped after pool failed on 9/1 01:48Z).
- rustfs-pool-expand-test.yml embedded a heal pass without
continue-on-error, so a heal failure failed the whole workflow.
Fixes:
- Add rustfs-functional-chain.yml: entry point that dispatches the first
suite via repository_dispatch; each suite hands off to the next with an
explicit, re-drivable API call instead of workflow_run triggers.
- Split heal out of the pool workflow (renamed to RustFS Pool Expansion
Test): heal now runs exactly once per chain, in rustfs-heal-test.yml
(storage -> heal -> pool).
- Every suite job gets continue-on-error so a failing test never fails
the workflow; failures are filed as issues in rustfs/backlog (report
+ redacted log tail) and the chain moves on.
- Clone rustfs/auto-testing with the PF token via 'gh repo clone' plus a
5-attempt retry loop (transient clone failures aborted whole suites).
- Stop rewriting functional/index.html from every suite (divergent
copies raced each other with stale SHAs); the canonical index now
lives in the dashboard repo.
- Standalone workflow_dispatch runs are unchanged and never forward the
chain; performance runs on its own runner, dispatched in parallel.
New RustFS Upgrade Test workflow (SUITE: upgrade) runs first in the
nightly functional chain:
- Nightly GNU Build -> Upgrade -> S3 -> KMS -> Tier -> Pool/Heal -> Security
- S3 compatibility now triggers on "RustFS Upgrade Test" completion, so an
upgrade regression gates the rest of the chain.
- Security suite moves to the end, after pool/heal, on the shared VMs.
- The upgrade suite drives auto-testing's rustfs-upgrade-test.sh
(UPG-101..402): seed golden data/identity/config on the OLD deb, upgrade
in place to the NEW deb, verify byte-identical preservation, and publish
functional-reports/upgrade/<date>.md.
- Add the Upgrade tab to every dashboard index writer so the shared
functional/index.html stays consistent.
The check_preview_release_workflow.sh script uses exact line matching
(grep -Fxq) to verify the cleanup-preview-releases job contains:
gh release delete "$preview_tag" --yes
The extra --repo flag is unnecessary in GitHub Actions context since
gh auto-detects the repository from GITHUB_REPOSITORY, and it causes
the Workflow Pin Report check to fail on all PRs.
The test workflows checkout the private rustfs/auto-testing repository, but
the default GITHUB_TOKEN only has access to rustfs/rustfs, so every checkout
failed with 'repository ... not found' (nightly runs on 2026-08-28).
Pass secrets.PF_TESTING_GH_TOKEN (the existing cross-repo PAT already used
by the performance workflow) to the auto-testing checkout steps in all three
workflows.
warp is killed at the write threshold and can leave in-flight multipart
uploads behind. rc.4-preview.1's decommission post-check refuses to
finalize a pool that still contains one (data is already moved, then the
pool is marked failed with 'resolve it before retrying'). Abort any
multipart uploads in the test bucket before starting decommission
(ListMultipartUploads + AbortMultipartUpload via the admin API).
* test(heal): relative disk target and fail fast on terminal-but-short
The absolute 40 GiB heal target was calibrated to the background scanner
(auto-heal), which is now disabled for determinism; with only the explicit
heal the recovered node lands at ~36 GiB for 40 GiB survivors. Make the
success criterion relative: the outage node must reach at least 90% of the
least-used surviving node (absolute HEAL_TARGET_GB floor optional, default
0 = relative only).
Also fail fast when the heal task reaches a terminal success but the disk
target is not met (previously the monitor kept polling until timeout), and
drop the misleading 'progress absent' warning on the final (cleaned) task
response — mid-run progress is reported correctly.
Validated live: heal summary=finished, 0 failed, vm000/vm001=40GB,
vm002=40GB (target 36GB), test PASSED.
* test(heal): gate success on server verdict + data read-back, drop disk GB gate
The per-node disk-usage target (40 GiB / 90% of survivors) is not a
code-level invariant: EC distributes different shards per node, so the
final GB per node depends on the layout, not on heal correctness. Gate the
test on what the server actually verifies:
- Heal task terminal success (finished/completed) with objectsFailed == 0
(the server's per-object scan/repair verdict).
- S3 read-back verification: list the test bucket and GET a sample of
objects, requiring HTTP 200 for every read (end-to-end proof the data is
still reconstructable after repair). The GET uses a discard mode so
binary bodies are not captured (no null-byte warnings / SIGPIPE).
Per-node disk usage stays in the output as observability (with a warning if
the outage node gained no usage), not as the pass/fail gate. Removes the
heal_target_gb input and the relative-target logic.
Validated live: heal summary=finished, 0 failed, 20/20 objects read back,
vm002_used=40GB, PASS.
A fixed /tmp path (log file, final heal status, warp log) can be owned by
another user on the shared runner (e.g. a previous root run), which made the
github-runner user fail: tee could not append the test log, the final heal
status write killed step 6 with EACCES, and upload-artifact could not read
stale root-owned warp logs. The heal scenario itself had passed
(summary=finished, vm002 reached the target) before the status-save died.
- Log files fall back to a unique mktemp path when the configured path is not
writable (heal + pool scripts).
- The final heal status is written to a mktemp file (best effort).
- Workflow artifact uploads use globs for the fallback names.
* test(heal): add node-outage heal E2E script and workflow
RustFS heal test on the 3x4 cluster (3 nodes x 4 disks, same
RUSTFS_VOLUMES expression on every node): write data with warp, stop the
outage node mid-write, restart it, start cluster heal via the admin API,
and pass only when the heal task finishes with 0 failures AND the outage
node's disk usage reaches the target.
Includes the GitHub Actions workflow (smoke-testing runner, nightly deb by
default) and a README. Validated end-to-end on the test environment:
40/40/16 GiB before heal -> 40/40/40 GiB after heal, summary=finished.
The script also writes RUSTFS_HEAL_TASK_TIMEOUT_SECS (default 6h) into the
node config because the server default (5 min) is far too short for
healing tens of GiB.
* ci(pool-test): chain heal regression after the pool test
The pool-expansion workflow is now triggered by the Nightly GNU Build
(workflow_run, replacing the schedule) and runs two sequential jobs on the
shared test environment:
1. pool-expansion-test (existing) — skipped if the nightly build failed.
2. heal-test — runs after the pool test regardless of its outcome
(if: always()): a pool failure makes the run red but does not block the
heal regression. Runs the heal script (reset -> install/start 3x4 ->
write/outage -> heal -> verify -> reset).
* test(heal): address review — camelCase progress, fail-closed, workflow hygiene
- Heal progress fields are camelCase in the API (objectsScanned/objectsHealed/
objectsFailed/progressPercentage); read them with a snake_case fallback and
distinguish null (absent) progress from zero, logging null as evidence
(rustfs/backlog#2035) instead of silently coercing.
- Fail closed in step 3: the outage node must actually be inactive after stop,
the write target must be reached, and an unobserved outage or incomplete
write fails the test instead of warning.
- Step 4 waits (bounded) for the cluster to report an active pool after the
outage-node restart instead of swallowing the verification error.
- Heal start fails fast on 400/403 (deterministic request/auth problems) and
only retries transient server errors.
- Disable the background scanner (RUSTFS_HEAL_AUTO_HEAL_ENABLE=false) so the
explicit heal is the only repair mechanism and the outage is observable.
- Workflows: heal and pool share one concurrency group; workflow_run requires
an exact successful nightly conclusion; checkout is pinned to the triggering
SHA; comma-separated step args are quoted (actionlint SC2054).
* test(pool): fix warp log path and retry rebalance start
- warp writes now use a unique mktemp log file instead of a fixed
/tmp/rustfs-warp.log: the runner user could not write the stale
root-owned file, which made the background warp process die instantly
(warp never ran). The workflow uploads /tmp/rustfs-warp.*.log.
- rebalance start is retried (6x, 20s apart): nightly builds gate
rebalance activation on a live cross-pool fence fleet capability proof
that takes ~10-20s to re-establish after a pool joins. Verified live:
attempt 1 fails with 500 'pool activation requires a live fleet
capability proof', attempt 2 succeeds.
* test(pool): annotate known server-side issues in failure output
When a node fails to start, grab the rustfs journal tail and match known
server-side error signatures (e.g. the fleet capability proof cold-start
regression, rustfs/backlog#2031), printing a hint with the tracking issue.
Also annotate the rebalance-start retry exhaustion and the rc.3 decommission
metacache-listing failure with actionable guidance.
* fix(ecstore): defer rebalance activation without fleet proof
---------
Co-authored-by: 马登山 <cxymds@qq.com>
Co-authored-by: cxymds <cxymds@gmail.com>
ci(pool-test): fix scheduled runs and read env from secrets or vars
workflow_dispatch inputs are empty for schedule events, so the scheduled
pool test built a broken package URL (--version "") and failed preflight.
Fall back to the latest nightly deb (R2) when no version/package_url input
is given, default the thresholds/duration/pools, and default cleanup to
enabled. Also read RUSTFS_API_ENDPOINT / RUSTFS_NODES / RUSTFS_SSH_USER
from secrets first (variables as fallback) so either configuration works.
* ci(nightly): persist the nightly deb on Cloudflare R2
Upload the deb to artifacts/rustfs/packages/nightly/ (dated name plus a
rustfs-nightly-latest.deb alias) through the same R2 channel package.yml
uses, so the nightly package can be downloaded later with a stable URL.
The step is skipped when the R2 secrets are not configured, keeping the
artifact-only mode intact.
* test: add pool expansion / decommission E2E script and workflow
Add the admin-API based pool expansion, rebalance and decommission test
script (scripts/test/rustfs_pool_expand.sh) plus a workflow_dispatch /
nightly workflow that runs it on a self-hosted runner against real nodes.
The workflow accepts a release tag or a direct .deb URL (e.g. nightly/R2
package) via the package_url input.
* ci(pool-test): run the pool expansion test on the smoke-testing runner
The nightly GNU build now also packages the release binary as
rustfs-nightly-<YYYY-MM-DD>.deb (Asia/Shanghai date, matching the schedule
timezone) and uploads it as a workflow artifact. Packaging mirrors
package.yml: DEBIAN control/conffiles and the systemd service from
deploy/build/, built with fakeroot dpkg-deb.
Add an optional TLS passthrough listener to the Gateway API support. When gatewayApi.listeners.tls.enabled is true, the Gateway gets a TLS listener with tls.mode: Passthrough and a TLSRoute is rendered to the RustFS service so TLS terminates at the backend (end-to-end encryption).
Refs rustfs/rustfs#3862.
The RPM build step fails because fpm's --config-files flag requires
/etc/default/rustfs to exist in the staging area, but unlike the DEB
build (which creates it in its package directory structure), the fpm
command has no prior step creating this file.
Create the config file in a temporary directory and pass it to fpm
via a source=dest mapping, matching the DEB build's behavior.
The DEB version substitution used ${VERSION/-/~} which caused bash
to expand ~ to $HOME (e.g. /home/runner), producing an invalid
version string like '1.0.0/home/runnerrc.1'.
Store ~ in a variable first to prevent tilde expansion.
hector