mirror of
https://github.com/agent-substrate/substrate.git
synced 2026-10-02 03:24:42 +08:00
Aditya ShantanuandAditya Shantanu
04bdb19b12
## Summary Adds `hack/metrics/ci-failure-analysis.py`, a script that samples recent failed `pr-workflow` CI runs, fetches their job logs, and classifies each failure by root cause rather than treating all CI failures as equivalent. **Failure categories:** - `named_test_fail` — a specific Go test emitted `--- FAIL: TestName` - `no_free_workers` — envtest/integration tests hit worker resource contention - `e2e_timeout` — e2e tests timed out waiting for actor/service responses (503s, connection resets) - `gcs_access` — GCS/S3 bucket access denied when fetching sandbox assets - `license_check` — `hack/verify/licenses.sh` detected uncommitted LICENSES changes - `verify_fail` — other `hack/verify-all.sh` step failed - `unknown` — could not classify from log output **Usage:** ```bash python hack/metrics/ci-failure-analysis.py \ --repo agent-substrate/substrate \ --fetch-limit 500 --sample 40 # Save for later comparison python hack/metrics/ci-failure-analysis.py \ --repo agent-substrate/substrate \ --fetch-limit 500 --sample 40 \ --save ci-breakdown.json ``` **Sample output (40 sampled runs, Aug 2026):** ``` Named Go test failure 50% ████████████████ No free workers 30% ██████████ E2e timeout / 503 5% █ Unclassified 15% █████ Named test failures: 6x TestDurableDirLifecycle 4x TestActorLifecycle 2x TestMultipleDurableDirLifecycle 2x TestSyncer_UpdateWorker_RetryOnVersionConflict 2x TestLoaderConcurrentHandshakes ``` This informed the timeout fixes in #800 and is intended to feed the planned scheduled CI remediation agent. ## Checklist - [x] Issue is linked above (see #799) - [x] Tests pass locally (`go test ./...`) - [x] Documentation updated if behavior changed (script is new, no existing docs affected) --------- Co-authored-by: Aditya Shantanu <aditya-shantanu@users.noreply.github.com>