mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
fix(ci): keep hard watchdog out of pytest shards (#3845)
## Summary - disable the production hard watchdog only in the sharded pytest job - keep the existing 30-minute job timeout responsible for reporting CI stalls - add a workflow-contract regression test so the opt-out cannot disappear silently ## Root cause After the hard watchdog merged, `main` CI shard 1 began terminating with exit code 1 during `tests/test_compress_session_mode.py`, without a pytest failure or summary. A matching Ubuntu 24.04 / Python 3.12 shard reproduced the abrupt termination locally. The same shard can complete when `HEADROOM_HARD_WATCHDOG_SECS=0`. The watchdog deliberately uses `faulthandler.dump_traceback_later(..., exit=True)` to terminate a production process when native code holds the GIL. That behavior is appropriate under a service supervisor, but inside pytest it destroys the diagnostic process before pytest or the job timeout can identify the stalled test. Production defaults and the dedicated subprocess watchdog tests remain unchanged. The contemporaneous Docker failures were separate: Docker Hub reset the OAuth connection, then the manifest job could not download the missing amd64 digest artifact. ## Verification - `uv run --frozen --extra dev pytest scripts/tests/test_ci_workflow.py tests/test_hard_watchdog.py -q` — 6 passed - `uv run --frozen --extra dev ruff check scripts/tests/test_ci_workflow.py` — passed - `uv run --frozen --extra dev ruff format --check scripts/tests/test_ci_workflow.py` — passed - `git diff --check` — passed - Ubuntu 24.04 / Python 3.12 exact shard command with the CI opt-out — completed successfully No production runtime behavior is changed.
This commit is contained in:
@@ -291,6 +291,9 @@ jobs:
|
||||
shard: [1, 2, 3, 4]
|
||||
env:
|
||||
TRANSFORMERS_OFFLINE: "1"
|
||||
# The production watchdog intentionally hard-exits the whole process.
|
||||
# CI's job timeout must report stalls without erasing pytest diagnostics.
|
||||
HEADROOM_HARD_WATCHDOG_SECS: "0"
|
||||
steps:
|
||||
- uses: actions/checkout@v7
|
||||
- uses: actions/setup-python@v7
|
||||
|
||||
@@ -4,6 +4,8 @@ from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
import yaml
|
||||
|
||||
|
||||
def test_sharded_ci_verifies_offline_huggingface_cache_before_pytest() -> None:
|
||||
workflow = Path(".github/workflows/ci.yml").read_text(encoding="utf-8")
|
||||
@@ -21,3 +23,10 @@ def test_sharded_ci_uploads_only_explicit_coverage_reports() -> None:
|
||||
|
||||
assert "files: coverage-${{ matrix.shard }}.xml" in upload_step
|
||||
assert "disable_search: true" in upload_step
|
||||
|
||||
|
||||
def test_sharded_ci_disables_process_killing_hard_watchdog() -> None:
|
||||
"""Pytest must report a stalled test instead of being hard-exited."""
|
||||
workflow = yaml.safe_load(Path(".github/workflows/ci.yml").read_text(encoding="utf-8"))
|
||||
|
||||
assert workflow["jobs"]["test"]["env"]["HEADROOM_HARD_WATCHDOG_SECS"] == "0"
|
||||
|
||||
Reference in New Issue
Block a user