Commit Graph
5 Commits
Author SHA1 Message Date
2973ca593d ci: pin every GitHub Action to a commit SHA (#4671)
## Description

Every third-party action in `.github/` ran from a version tag like
`actions/checkout@v7`. A tag can be moved to other code at any time.
That is how `tj-actions/changed-files` leaked CI secrets from ~23,000
repos in March 2025.

Here a moved tag would run with the release job's PyPI OIDC token
(`id-token: write`) and the Docker job's `packages: write` token.
Pinning to a commit SHA closes that, and a new test keeps it closed.

Closes #

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [x] Code refactoring (no functional changes)

## Changes Made

- All 195 third-party `uses:` refs in 25 files now name the commit their
tag points to today, with the version as a comment:
`actions/checkout@3d3c42e5... # v7.0.1`. No action version changes.
- Dependabot's existing `github-actions` config updates SHA pins and
their version comments, so the pins stay current.
- Pin dtolnay/rust-toolchain stable and 1.95.0 branches to their
existing commit heads. The stable action retains its runtime toolchain
input/default; the 1.95.0 action embeds toolchain: 1.95.0. This
preserves each call site's existing toolchain selection.
- New `test_every_action_is_pinned_to_a_commit_sha` in
`scripts/tests/test_ci_workflow.py` fails on any tag-pinned action and
lists each one.
- Three tests in `tests/test_release_workflows.py` matched the old tag
text (`setup-buildx-action@v4`, `gh-action-pypi-publish@vX.Y.Z`,
`release-please-action@v4/v5`). They now match the SHA pin and its
version comment.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ uv run --frozen --extra dev pytest -q scripts/tests/test_ci_workflow.py scripts/tests/test_pr_health_workflow.py scripts/tests/test_normalize_release_pr_bodies.py tests/test_release_workflows.py
87 passed in 0.87s

$ uv run --frozen --extra dev pytest -q scripts/tests/test_pr_health_labels.py scripts/tests/test_version_sync.py tests/test_cli/test_wrap_copilot.py tests/test_learn/test_subagent_scanning.py tests/test_plugin_manifests.py tests/test_pr208_changes.py
114 passed in 1.32s

$ uvx ruff check scripts/tests/test_ci_workflow.py tests/test_release_workflows.py && uvx ruff format --check <same>
All checks passed!
```

## Real Behavior Proof

- Environment: macOS, Python 3.12 via `uv run --frozen --extra dev`,
branch cut from `main` at c01cf07d.
- Exact command / steps: resolved each tag with `git ls-remote --tags
--heads https://github.com/<owner>/<repo>` (peeled for annotated tags),
rewrote only the `uses:` lines, then ran the test files above and a
`yaml.safe_load` over every file in `.github/`.
- Observed result: 195 lines changed, 195 removed, no other line touched
(mixed LF/CRLF in `pr-health.yml` kept). All 201 tests pass. On `main`
the new test fails and lists the unpinned refs.
- Not tested: `scripts/validate-workflows.sh` (actionlint plus act dry
runs) was not run locally. The `workflow-validation` job runs it on this
PR.

## Runtime Rollout Safety

- Rollout-managed feature(s): None. CI workflow files and their tests
only.
- Minimum rollout channel: N/A, no runtime code changes.
- Stable/default behavior changed: No. Each SHA is the commit its tag
points to today.
- Kill switch / disable path: N/A, revert the commit.
- Unsafe override required: No.
- Qualification impact: None.
- Rollback path: Revert this PR.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Screenshots (if applicable)

N/A, no UI change.

## Additional Notes

- mypy and docs are N/A: no `headroom/` code or documented behavior
changes.
- Dependabot does not bump branch pins, so the two
`dtolnay/rust-toolchain` pins stay on today's branch head until someone
moves them. That action rarely changes.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4671"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

### Maintainer current-main requalification

Exact head `57b7edd8d05b6dc5e3bc4eac1dff55f21519afe8`, upstream main
`c01cf07d943d816cfd9d97e6838ed4e38f1ff9a4`.

Reviewed the27-file delta and existing discussion/Devin no-issues result
against current upstream main c01cf07d94,
already included in the contributor head. Independent full-workflow
comparison proves only uses refs/comments change across the25
workflow/action files. All195 replaced refs resolve exactly to their
original upstream tag/branch commits (29 unique pins across26
repositories), with annotated tags peeled; no mismatches. Pinned dtolnay
action source independently confirms stable runtime default versus
embedded1.95.0 branch behavior, preserving toolchain selection.87
workflow/release tests and113 adjacent tests pass,1skip; changed Python
lint/format passes. Maintainer fix only removes CRLF endings on two
edited pr-health lines to clear whitespace; provenance reverified
afterward. No contributor workflow logic altered. Description is
corrected to explain embedded1.95.0 semantics. No publish/release
workflow execution claimed; full hosted qualification must run on this
new mechanical-fix head. Maintainers handle CI. Nothing merged.


Exact-head hosted checks are being handled by maintainers. No new
finding in the reviewed scope; formal approval awaits exact-head hosted
qualification; nothing merged.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-11-06 09:16:15 -06:00
be2a298ed0 ci: upload shard coverage even when a test in the shard fails (#3876)
## Description

The Codecov upload step in the sharded `test` job has no `if:`, so a
single failing test in a shard skips it. Every line that only that shard
covers then drops out of `codecov/patch`. Because of this, a PR can show
0% patch coverage when its own tests passed but an unrelated test in the
same shard failed. pytest-cov still writes `coverage-N.xml` when tests
fail, so the report exists and only the upload is skipped.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)

## Changes Made

- `.github/workflows/ci.yml`: add `if: ${{ !cancelled() }}` to `Upload
coverage shard N to Codecov`.
- `scripts/tests/test_ci_workflow.py`: extend the existing upload-step
contract test to require that condition.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] New tests added for new functionality

### Test Output

```text
$ uv run --frozen --extra dev pytest scripts/tests/test_ci_workflow.py -q
3 passed
# with the ci.yml change reverted: 1 failed, 2 passed
```

## Real Behavior Proof

- Environment: GitHub Actions `ci.yml` sharded test job, as observed on
PR #3874 (run 36671680550).
- Exact command / steps: Compared the failing shards' step lists against
the Codecov status on the PR.
- Observed result: Shard 3 failed on an unrelated test, and its `Upload
coverage` step was skipped although pytest had written `coverage-3.xml`.
With this change the step runs after a test failure and is still skipped
on cancellation.
- Not tested: A live Codecov upload from this branch; this PR's own CI
run exercises it.

## Runtime Rollout Safety

- Rollout-managed feature(s): None.
- Minimum rollout channel: Stable.
- Stable/default behavior changed: No.
- Kill switch / disable path: Not applicable.
- Unsafe override required: No.
- Qualification impact: None.
- Rollback path: Revert this commit.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

This does not recover coverage from a shard that is killed outright. A
hard-exited process never writes its XML, so there is nothing to upload;
that case is fixed separately in fix/pytest-hard-watchdog.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
2026-09-30 11:04:01 -05:00
JD Davis 25274310ca fix(ci): keep hard watchdog out of pytest shards (#3845)
## Summary

- disable the production hard watchdog only in the sharded pytest job
- keep the existing 30-minute job timeout responsible for reporting CI
stalls
- add a workflow-contract regression test so the opt-out cannot
disappear silently

## Root cause

After the hard watchdog merged, `main` CI shard 1 began terminating with
exit code 1 during `tests/test_compress_session_mode.py`, without a
pytest failure or summary. A matching Ubuntu 24.04 / Python 3.12 shard
reproduced the abrupt termination locally. The same shard can complete
when `HEADROOM_HARD_WATCHDOG_SECS=0`.

The watchdog deliberately uses `faulthandler.dump_traceback_later(...,
exit=True)` to terminate a production process when native code holds the
GIL. That behavior is appropriate under a service supervisor, but inside
pytest it destroys the diagnostic process before pytest or the job
timeout can identify the stalled test. Production defaults and the
dedicated subprocess watchdog tests remain unchanged.

The contemporaneous Docker failures were separate: Docker Hub reset the
OAuth connection, then the manifest job could not download the missing
amd64 digest artifact.

## Verification

- `uv run --frozen --extra dev pytest scripts/tests/test_ci_workflow.py
tests/test_hard_watchdog.py -q` — 6 passed
- `uv run --frozen --extra dev ruff check
scripts/tests/test_ci_workflow.py` — passed
- `uv run --frozen --extra dev ruff format --check
scripts/tests/test_ci_workflow.py` — passed
- `git diff --check` — passed
- Ubuntu 24.04 / Python 3.12 exact shard command with the CI opt-out —
completed successfully

No production runtime behavior is changed.
2026-09-29 10:57:20 -05:00
Raúl 3f2ca99fe1 fix(ci): restrict Codecov shard uploads (#2745)
## Description

Closes #2744

Restrict each Codecov Action v5 matrix upload to its declared
`coverage-${{ matrix.shard }}.xml` report. This prevents automatic
discovery
from uploading the unsharded `coverage.xml` alongside every shard.

## Type of Change

- [x] Bug fix (non-breaking change fixes an issue)

## Changes Made

- Set Codecov Action `disable_search: true` for Python shard uploads.
- Add a CI workflow contract test that protects the explicit-report-only
setup.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check`)
- [x] Type checking passes (`mypy headroom`) (not applicable:
workflow/test-only change)
- [x] New tests added new functionality
- [x] Manual testing performed (not applicable: GitHub Actions will
execute the workflow)

### Test Output

```text
$ uv run --with ruff ruff format --check scripts/tests/test_ci_workflow.py
1 file already formatted

$ uv run --with ruff ruff check scripts/tests/test_ci_workflow.py
All checks passed!

$ uv run --with pytest pytest scripts/tests/test_ci_workflow.py -q
2 passed
```

## Real Behavior Proof

- Environment: GitHub Actions Ubuntu runner using Python 3.12.13;
Codecov Action v5.
- Exact command / steps: Run the CI Python test matrix, which writes
`coverage-${{ matrix.shard }}.xml`, then runs the Codecov Action upload
step. Inspect the uploader's discovered/uploaded report list.
- Observed result: Before this change, raw CI logs showed the Action
explicitly uploading `coverage-2.xml` and additionally
discovering/uploading `coverage.xml`. This PR configures
`disable_search: true`; the workflow contract test confirms the explicit
report setting and search disablement. Runtime upload evidence will be
added from this draft PR's CI run.
- Not tested: Codecov's final cross-shard patch calculation; that
depends on Codecov processing the reports after CI completes.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR ready for human review

## Checklist

- [x] My code follows project's style guidelines
- [x] I have performed a self-review
- [x] I commented my code, particularly in hard-to-understand areas
- [x] I made corresponding changes documentation (not applicable)
- [x] My changes generate no new warnings
- [x] I added tests prove my fix is effective or feature works
- [x] New and existing unit tests pass locally changes
- [x] I did **not** edit `CHANGELOG.md` — generated by release-please
from Conventional Commit PR title (a CI guard enforces this)

## Additional Notes

This is intentionally limited to the Codecov upload configuration and
its
workflow contract test. It does not include the unrelated Copilot
Keychain fix.
2026-08-03 14:20:17 -07:00
JD Davis adb793bee1 ci: harden PR governance and model cache checks (#1401)
## Description

Hardens two routine PR-review pain points from the recent open-PR sweep:

- PR Governance reruns could keep validating the stale
`pull_request_target` event body even after the live PR description had
been fixed.
- Main CI model-cache misses could surface as dozens of unrelated
memory-test failures instead of one clear cache-preflight failure.

This intentionally avoids PyPI/package-bloat and release/nightly
workflow changes so the PR stays scoped to review and CI stabilization.

## Type of Change

- [x] Bug fix
- [ ] New feature
- [ ] Documentation
- [x] Refactor
- [x] Tests only

## Changes Made

- Added `--body-file` support to `scripts/pr-governance.py` so workflows
can validate the current PR body rather than stale rerun payloads.
- Updated PR Governance to fetch the live PR body via the GitHub API
before validating template fields.
- Added a CI preflight script that loads the default
sentence-transformer model in offline mode and verifies the expected
embedding dimension.
- Wired that preflight into the sharded CI job before pytest starts,
turning missing/corrupt Hugging Face caches into one early, actionable
failure.
- Added workflow/script regression tests for the live-body override and
model-cache preflight placement.

## Testing

- [x] Unit tests
- [x] Lint/static checks
- [ ] Integration tests
- [ ] Manual testing

### Test Output

```text
uv run --with pytest --with pytest-asyncio python -m pytest scripts/tests/test_pr_governance.py scripts/tests/test_pr_health_workflow.py scripts/tests/test_ci_workflow.py -q
9 passed in 0.04s

uv run ruff check scripts/pr-governance.py scripts/ci/verify_hf_model_cache.py scripts/tests/test_pr_governance.py scripts/tests/test_pr_health_workflow.py scripts/tests/test_ci_workflow.py
All checks passed!

python -m py_compile scripts\ci\verify_hf_model_cache.py scripts\pr-governance.py
# passed
```

## Real Behavior Proof

- Environment: Windows 11, Python 3.13.3, isolated worktree
`C:\git\headroom\.worktrees\stabilization-hardening`.
- Exact command / steps: Ran the focused governance/workflow tests, ruff
on touched Python files, and `py_compile` for the executable scripts.
- Observed result: Governance tests prove a stale event body can be
overridden by the live PR body; workflow tests prove CI validates live
PR body and runs the Hugging Face offline-cache preflight before pytest
shards.
- Not tested: Full GitHub CI before PR creation; that will run on this
PR. The new Hugging Face preflight itself is intentionally not run
locally because it depends on the CI-warmed offline model cache.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review
2026-06-26 21:34:34 -07:00