Commit Graph
2 Commits
Author SHA1 Message Date
c22cd1f1b6 ci(pr-health): flag stale PR branches by ref comparison (#3229)
## Description

The `status: needs rebase` label in `.github/workflows/pr-health.yml` is
driven by `mergeStateStatus == BEHIND`. GitHub only reports `BEHIND`
when the base branch requires branches to be up to date, and `main` does
not (`required_status_checks.strict = false`, per #3066). The label has
therefore never been applied since it was added in #933, so stale-base
risk is invisible on the PR.

This computes the drift directly with a ref comparison instead, so the
label works regardless of the branch-protection setting. It does not
replace the settings decision in #3066 (only a repo admin can enable
strict checks or a merge queue) — it makes the risk visible while that
is pending.

Refs #3066

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [x] Code refactoring (no functional changes)

## Changes Made

- `.github/workflows/pr-health.yml`: the `label` job now asks `gh pr
view` for `baseRefName`, `headRefOid` and `files`, calls
`repos/{repo}/compare/{base}...{head}` for `behind_by` plus the merge
base, and — only when the branch is behind — calls
`compare/{merge_base}...{base}` for the files the base branch changed in
the meantime. That second call has no `[]` fallback: `base_files` is set
to an empty-string sentinel first and reset to it when the call fails,
so partial output from a call that died midway is discarded and a
missing merge base takes the same path.
- `.github/scripts/pr-health-labels.py`: added `drift_state()`,
`parse_behind_by()` and `parse_base_files()`, plus a `--field
checks|drift` switch with `--behind-by` / `--base-files`. A PR is
`stale` when it is behind **and** the base moved on files the PR also
changes, `current` otherwise, and `unknown` when either comparison call
failed. `parse_base_files()` maps the sentinel, invalid JSON and any
non-list payload to `None`, which is what makes a failed base-file
comparison `unknown` rather than an empty set of moved files; a
successful comparison that returns zero files still classifies as
`current`. `unknown` leaves the label untouched, mirroring the existing
`mergeStateStatus == UNKNOWN` guard, so a transient API error never
clears a real label. An explicit `mergeStateStatus == BEHIND` is still
honoured, so nothing regresses if strict checks are enabled later.
- The same `stale` verdict now holds back `status: ready for review`,
replacing the old `BEHIND` condition.
- File overlap is the deliberate narrowing: plain "behind by at least
one commit" matches 18 of 20 open PRs on a base branch this busy, which
would make the label noise. Overlap targets the case that hides a
semantic merge conflict from per-PR checks, which is what #3066
describes.
- Cost is 1-2 extra API calls per PR, so at most ~200 per scheduled run
at the existing 100-PR cap.
- Added unit tests for the new logic and workflow-contract assertions,
including a regression that runs the extracted `base_files` shell block
under `bash` with `gh` stubbed to print partial output and exit 1,
asserts the sentinel survives, feeds it to the script for an `unknown`
verdict, and asserts the drift dispatch has no `else` so `unknown`
neither adds nor removes the label.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ python -m pytest scripts/tests/test_pr_health_labels.py scripts/tests/test_pr_health_workflow.py -q
scripts\tests\test_pr_health_labels.py .............                     [ 65%]
scripts\tests\test_pr_health_workflow.py .......                         [100%]
============================= 20 passed in 0.10s ==============================

$ ruff check .github/scripts/pr-health-labels.py scripts/tests/
All checks passed!

$ ruff format --check .github/scripts/pr-health-labels.py scripts/tests/test_pr_health_labels.py scripts/tests/test_pr_health_workflow.py
3 files already formatted
```

## Real Behavior Proof

- Environment: local checkout of the branch at 738f984d, Windows 11,
Python 3.13, `gh` 2.x authenticated against `headroomlabs-ai/headroom`
- Exact command / steps: replayed the new workflow steps against the 25
most recent open PRs — for each one, `gh pr view --json
baseRefName,files,headRefOid,isDraft,labels,mergeStateStatus,reviewDecision,statusCheckRollup`,
then `gh api repos/headroomlabs-ai/headroom/compare/{base}...{head}` and
`compare/{merge_base}...{base}`, feeding the results to `python
.github/scripts/pr-health-labels.py --state-json ... --field drift
--behind-by ... --base-files ...`
- Observed result: review follow-up on 2026-08-27, replayed #3192,
#3180, #3177 and this PR with a successful base-file comparison and
again with a failed one. The successful verdicts are unchanged (`stale`,
`stale`, `stale`, `current`); the failed comparison now returns
`unknown` on all four, where the pre-fix script returned `current` and
the label step would have removed `status: needs rebase`. Reverting the
workflow block to the old `|| echo '[]'` fallback fails both new
workflow tests (`assert 'partia[]' == ''`). Original sweep: 3 of 25 PRs
classified `stale` — #3192, #3180 and #3177, each 13 commits behind
`main` and each touching `headroom/proxy/server.py`, which `main` also
changed in that window. The other 22 came back `current`, including PRs
that are behind by 1 to 13 commits on unrelated files. Before this
change none of the 25 reported `mergeStateStatus == BEHIND` and none
carried `status: needs rebase`. Workflow YAML still parses via
`yaml.safe_load`.
- Not tested: no run of the workflow on GitHub Actions
(`pull_request_target` and the cron schedule can only fire from the
default branch after merge), and no `actionlint`/`act` run locally as
neither tool is installed on this machine — CI covers both.

## Runtime Rollout Safety

- Rollout-managed feature(s): none, this is CI-only automation and
touches no runtime code path
- Minimum rollout channel: not applicable
- Stable/default behavior changed: no product behavior changes; the only
observable change is which PRs receive the `status: needs rebase` label
- Kill switch / disable path: revert this commit, or delete the label —
the `label` job keeps working, and every `gh pr edit` failure is already
tolerated
- Unsafe override required: no
- Qualification impact: none
- Rollback path: `git revert` of this commit restores the previous
`mergeStateStatus == BEHIND` condition

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

- Documentation checkbox is unchecked because the label semantics are
described in the label description itself (updated by this PR) and no
docs page covers the PR-health labels.
- `mypy headroom` is unchecked because nothing under `headroom/`
changed; the touched script lives in `.github/scripts/`.
- The overlap comparison uses the files `gh pr view --json files`
returns, so a PR with an unusually large file list could under-report
overlap. That fails safe (no label) rather than mislabelling.
- This PR intentionally does not close #3066: the acceptance criteria
there are strict status checks or a merge queue on `main`, which need
admin rights on the repo.

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-09-25 10:34:18 -05:00
RAPHAEL LUGO 99c874d423 fix(codex): PR health label check state (#986)
## Description

Fix the PR health label job so `status: ci failing` reflects the latest
check attempt for each check, not historical failed or cancelled
attempts that still appear in `statusCheckRollup`.

This showed up on #984: the current checks were green, but the label job
kept `status: ci failing` because older failed template runs were still
present in the rollup payload.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- Added a small `.github/scripts/pr-health-labels.py` helper that groups
check-rollup entries by logical check name and evaluates only the newest
entry for each check.
- Updated the PR health workflow label job to call the helper instead of
treating any historical failing rollup entry as current failure.
- Added regression tests for historical failures followed by latest
passing attempts, plus current latest failure behavior.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
PYTEST_ADDOPTS='-p no:cacheprovider' pytest scripts/tests -q
47 passed, 1 warning in 0.39s

python .github/scripts/pr-health-labels.py --state-json '<payload with old FAILURE and latest SUCCESS>'
passing

data=$(gh pr view 984 --repo chopratejas/headroom --json statusCheckRollup)
python .github/scripts/pr-health-labels.py --state-json "$data"
passing
```

## Real Behavior Proof

- Environment: macOS local checkout, Python 3.11.7, live GitHub PR #984
check-rollup payload fetched with `gh pr view`.
- Exact command / steps: Added regression coverage for historical
failed/cancelled check runs followed by latest successful runs, ran the
scripts test suite, and evaluated live PR #984's `statusCheckRollup`
with the new helper.
- Observed result: The helper returns `passing` for #984's live payload
even though older failed/cancelled check runs are still present, while
still returning `failing` when the latest attempt for a check failed.
- Not tested: A full GitHub Actions run of the updated workflow on
upstream before merge; this PR should exercise the workflow on itself.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review
2026-06-15 16:52:26 -05:00