fix(ci): keep hard watchdog out of pytest shards (#3845)

## Summary

- disable the production hard watchdog only in the sharded pytest job
- keep the existing 30-minute job timeout responsible for reporting CI
stalls
- add a workflow-contract regression test so the opt-out cannot
disappear silently

## Root cause

After the hard watchdog merged, `main` CI shard 1 began terminating with
exit code 1 during `tests/test_compress_session_mode.py`, without a
pytest failure or summary. A matching Ubuntu 24.04 / Python 3.12 shard
reproduced the abrupt termination locally. The same shard can complete
when `HEADROOM_HARD_WATCHDOG_SECS=0`.

The watchdog deliberately uses `faulthandler.dump_traceback_later(...,
exit=True)` to terminate a production process when native code holds the
GIL. That behavior is appropriate under a service supervisor, but inside
pytest it destroys the diagnostic process before pytest or the job
timeout can identify the stalled test. Production defaults and the
dedicated subprocess watchdog tests remain unchanged.

The contemporaneous Docker failures were separate: Docker Hub reset the
OAuth connection, then the manifest job could not download the missing
amd64 digest artifact.

## Verification

- `uv run --frozen --extra dev pytest scripts/tests/test_ci_workflow.py
tests/test_hard_watchdog.py -q` — 6 passed
- `uv run --frozen --extra dev ruff check
scripts/tests/test_ci_workflow.py` — passed
- `uv run --frozen --extra dev ruff format --check
scripts/tests/test_ci_workflow.py` — passed
- `git diff --check` — passed
- Ubuntu 24.04 / Python 3.12 exact shard command with the CI opt-out —
completed successfully

No production runtime behavior is changed.
This commit is contained in:
JD Davis
2026-09-29 10:57:20 -05:00
committed by GitHub
parent f519fa8b97
commit 25274310ca
2 changed files with 12 additions and 0 deletions
+3
View File
@@ -291,6 +291,9 @@ jobs:
shard: [1, 2, 3, 4]
env:
TRANSFORMERS_OFFLINE: "1"
# The production watchdog intentionally hard-exits the whole process.
# CI's job timeout must report stalls without erasing pytest diagnostics.
HEADROOM_HARD_WATCHDOG_SECS: "0"
steps:
- uses: actions/checkout@v7
- uses: actions/setup-python@v7
+9
View File
@@ -4,6 +4,8 @@ from __future__ import annotations
from pathlib import Path
import yaml
def test_sharded_ci_verifies_offline_huggingface_cache_before_pytest() -> None:
workflow = Path(".github/workflows/ci.yml").read_text(encoding="utf-8")
@@ -21,3 +23,10 @@ def test_sharded_ci_uploads_only_explicit_coverage_reports() -> None:
assert "files: coverage-${{ matrix.shard }}.xml" in upload_step
assert "disable_search: true" in upload_step
def test_sharded_ci_disables_process_killing_hard_watchdog() -> None:
"""Pytest must report a stalled test instead of being hard-exited."""
workflow = yaml.safe_load(Path(".github/workflows/ci.yml").read_text(encoding="utf-8"))
assert workflow["jobs"]["test"]["env"]["HEADROOM_HARD_WATCHDOG_SECS"] == "0"