mirror of
https://github.com/NVIDIA/OpenShell.git
synced 2026-10-26 06:52:57 +08:00
feat(skills): add release-captain skill for the weekly release cadence (#4435)
* feat(skills): add release-captain skill for the weekly release cadence Add an internal skill with Monday pre-flight, Tuesday stable cut, and recovery steps for the RFC 0014 release train. All tag, workflow, merge, and Slack actions require explicit maintainer approval. Closes #4433 Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * docs(skills): document starting a release train after tag-push triggers Pre-release tag pushes now start Release Tag, so the skill no longer dispatches after pushing. Add seed-commit selection from Release Dev results, a commit-type summary for the patch/minor decision, and a recovery row for a tag push that starts no run. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * docs(skills): create lightweight release tags regardless of signing config With tag.gpgSign enabled, git tag creates an annotated tag and blocks in an editor. Disable signing for the release tag commands so they create the same lightweight tags as Auto-Tag. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * docs(skills): capture release captain lessons from starting 0.1.5 Note that Auto-Tag skips report success and show how to tell from the log, the UTC time of the schedule across DST, where pre-releases appear instead of the Releases page, why tags need a forced fetch, reading workflow triggers from origin/main, the rolling dev tag, and the common compatibility-review false positive for unpublished internal crates. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * docs(skills): add recovery for pre-releases failing on new RustSec advisories Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * docs(skills): recommend fixing forward to a green pre-release A failed pre-release doesn't end the train. Document merging the fix and tagging the next pre-release by hand once its Release Dev run passes, letting the failed run finish for its remaining signal. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * docs(skills): describe the triage channel and thread replies for pre-release failures Name #openshell-triage, where TriageBot posts each failed Release Tag attempt, and ask the captain to reply in the failure's thread so others know a fix is underway. List common transient failure types and rerun once when a failure looks unrelated to the commit. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * docs(skills): drop Codex Security from release captain run matching Codex Security no longer runs in release qualification (#4469), so the stable release can't rely on a pre-release's Codex result and runs can't be matched by its job name. Match dispatched runs by the qualification artifact name instead. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * docs(skills): describe where and how to post the release coordination thread Name the CNCF Slack channel, note that the release ships from main with no release branch or cherry-picks, and add an example post. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * docs(skills): make seeding the next train the outgoing captain's handoff Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * docs(skills): update release captain for the Release workflow and add usage guide Stable releases are now dispatched with release.yml, which selects the qualified pre-release and creates the tag, and pre-releases run in release-prerelease.yml. Replace the stable tag push steps and stale recovery rows, add a preview of the selected candidate, add a section on when to use each part of the skill, and trim redundant explanations. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * docs(skills): explain what the release cutoff guarantees The release ships the final pre-release's commit only if it passes, so the coordination post must not promise that every PR merged by the cutoff ships. Describe the Auto-Tag and hand-tagged cutoffs, and tag the final pre-release by hand on Tuesday for a later cutoff. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * docs(skills): add release notes review and v0.1.5 release day lessons Review the release notes preview from the Pre-release run before approval, and describe how Release regenerates them. Correct the thread when the shipped pre-release was cut before the posted cutoff, require maintainer agreement for accepted-risk breaks in a patch release, and add recovery rows for startup failures and release notes failures. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * docs(skills): make starting the next train the captain's final duty Move starting the next train after closing the loop, so it is the last Tuesday step. The captain's week ends when the next -pre.1 Pre-release run passes or its failure is handed off. Also move open milestone items when closing the loop. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * docs(skills): trim the release captain skill to its durable steps Remove repeated guidance, week-specific examples, fragile commands, and debugging detail an agent can derive from the workflows. Keep the approval guardrails, the weekly steps, and a short recovery table. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * docs(ci): document Release Auto-Tag and pre-release notes previews Add Release Auto-Tag to the workflow table, including that each train's -pre.1 is pushed by hand, and note that every Pre-release run writes a release notes preview. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * docs(skills): add a template for the release announcement reply Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * docs(skills): watch pre-releases in the week before your turn Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * docs(skills): remove internal channel references from release captain Check pre-release runs directly instead of relying on an internal notification channel, and drop internal Slack channel IDs. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> * docs(skills): fix pre-release failures in a new number via Auto-Tag A rerun repeats the same commit, so a failure that needs a fix ships in the next -pre.N. Cut it by dispatching Release Auto-Tag from the tip of main, and tag a specific commit by hand only when needed. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> --------- Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
This commit is contained in:
Matthew Grossman
@@ -0,0 +1,228 @@
|
||||
---
|
||||
name: release-captain
|
||||
description: Run the weekly OpenShell release captain checklist. Monday checks the pre-release train, triages failures and the compatibility review, and drafts the coordination post. Tuesday promotes the qualified pre-release after approval, verifies publication, and starts the next train. Trigger keywords - release captain, weekly release, Tuesday release, cut the release, release checklist, pre-release status, release train, start next train, stable release.
|
||||
metadata:
|
||||
internal: true
|
||||
---
|
||||
|
||||
# Release Captain
|
||||
|
||||
OpenShell publishes a stable release every Tuesday from a qualified
|
||||
pre-release ([RFC 0014](../../../rfc/0014-release-stability/README.md)). The
|
||||
release captain keeps the week's pre-release train green, coordinates the last
|
||||
merges, cuts the release, and starts the next train.
|
||||
|
||||
## How to use this skill
|
||||
|
||||
| When | Ask | What happens |
|
||||
|---|---|---|
|
||||
| The week before your turn | "I'm release captain next week. Is the train ready?" | Checks that the next version's `-pre.1` exists. If not, ask this week's captain to start the train (Tuesday step 6). Then check each weekday that Auto-Tag's pre-release ran and passed (`gh run list --workflow release-prerelease.yml --limit 3`), and fix failures as they happen (Monday step 3) instead of on Monday. |
|
||||
| Monday | "Run the release captain Monday checklist." | Monday steps 1–4. |
|
||||
| A pre-release failed | "`vX.Y.Z-pre.N` failed, help me fix it." | Monday step 3 and Recovery. |
|
||||
| Tuesday | "Cut the release." | Tuesday steps 1–6, ending with starting the next train. |
|
||||
|
||||
## Guardrails
|
||||
|
||||
Get explicit approval, with the exact command and target SHA, before pushing a
|
||||
tag, dispatching, rerunning, or cancelling a workflow, merging or approving a
|
||||
PR, or posting in Slack or GitHub. Never repoint a stable tag, delete a `-pre.N`
|
||||
tag, or start a second `Pre-release` run for a tag; it has no concurrency guard.
|
||||
|
||||
## How the train works
|
||||
|
||||
- A train has one base version, such as `v0.1.4`, fixed by its `-pre.1` tag. A
|
||||
patch train can't take breaking changes to Stable interfaces.
|
||||
- `Release Auto-Tag` runs on weekdays at 14:00 Europe/Zurich. If `main` has
|
||||
commits since the latest `-pre.N`, it tags the next one and dispatches
|
||||
`Pre-release` (`release-prerelease.yml`, about 45 minutes). It skips when the
|
||||
train has no `-pre.1` or the base version is already stable.
|
||||
- `Pre-release` builds, then publishes and qualifies in parallel, and writes a
|
||||
`release-notes` preview artifact. Pushing a `-pre.N` tag by hand starts it.
|
||||
- `Release` (`release.yml`) is dispatched by hand with the stable version. It
|
||||
picks the highest `-pre.N` on `main` whose whole `Pre-release` run passed,
|
||||
creates the stable tag on that commit, and publishes without requalifying.
|
||||
|
||||
## Setup
|
||||
|
||||
```shell
|
||||
export GH_REPO=NVIDIA/OpenShell
|
||||
git fetch origin main --tags --force
|
||||
PREV=$(git tag -l 'v*' --sort=-v:refname | grep -E '^v[0-9]+\.[0-9]+\.[0-9]+$' | head -1)
|
||||
git tag -l 'v*' --sort=-v:refname | grep -E -- '-pre\.[0-9]+$' | head -5
|
||||
```
|
||||
|
||||
Find the `Pre-release` run for a tag. A tag pushed by hand runs on the tag; an
|
||||
Auto-Tag run is dispatched on `main`:
|
||||
|
||||
```shell
|
||||
TAG=vX.Y.Z-pre.N
|
||||
gh run list --workflow release-prerelease.yml --branch "$TAG"
|
||||
gh run list --workflow release-prerelease.yml --event workflow_dispatch --commit "$(git rev-parse "$TAG^{commit}")"
|
||||
```
|
||||
|
||||
Preview the candidate `Release` would promote, read-only:
|
||||
|
||||
```shell
|
||||
VERSION=vX.Y.Z
|
||||
wt=$(mktemp -d)
|
||||
git worktree add -q --detach "$wt" origin/main
|
||||
(cd "$wt" && GITHUB_REPOSITORY=NVIDIA/OpenShell uv run -q --no-project python tasks/scripts/select-release-candidate.py "$VERSION" --branch main | jq .)
|
||||
git worktree remove --force "$wt"
|
||||
```
|
||||
|
||||
## Monday: pre-flight
|
||||
|
||||
1. **Check the candidate.** Run the preview. If the latest `-pre.N` isn't the
|
||||
candidate, its run failed; read why:
|
||||
|
||||
```shell
|
||||
RUN=<run-id>
|
||||
gh run view "$RUN" --json jobs --jq '.jobs[] | select(.conclusion == "failure" or .conclusion == "timed_out") | "\(.name)\t\(.url)"'
|
||||
gh run view "$RUN" --log-failed
|
||||
```
|
||||
|
||||
If the next version has no `-pre.1`, start the train (Tuesday step 6).
|
||||
|
||||
2. **Triage the compatibility review** for the candidate's run. It's advisory,
|
||||
and a green job doesn't mean it found no breaks:
|
||||
|
||||
```shell
|
||||
ATTEMPT=$(gh run view "$RUN" --json attempt --jq .attempt)
|
||||
dir=$(mktemp -d)
|
||||
gh run download "$RUN" -n "compatibility-review-run-$RUN-attempt-$ATTEMPT" -D "$dir"
|
||||
jq '{status, verdict: .review.verdict}' "$dir/report.json"
|
||||
cat "$dir/report.md"
|
||||
```
|
||||
|
||||
Only `complete` with `no_unaddressed_breaks` is clean. A real break needs a
|
||||
fix, a revert, a minor train, or a maintainer-approved exception with upgrade
|
||||
steps in the release notes.
|
||||
|
||||
3. **Fix forward.** A failed pre-release doesn't end the train. Find the cause,
|
||||
and let the failed run finish rather than cancelling it.
|
||||
|
||||
A rerun repeats the same commit and workflow, so it only helps with a
|
||||
transient failure (see Recovery). Anything that needs a code or workflow
|
||||
fix ships in a new number: that `-pre.N` stays failed, and `Release` skips
|
||||
it. Merge the fix, then cut the next `-pre.N` from the tip of `main` after
|
||||
approval:
|
||||
|
||||
```shell
|
||||
gh workflow run release-auto-tag.yml --ref main
|
||||
```
|
||||
|
||||
Tag a specific commit by hand (as in Tuesday step 6) only when the tip of
|
||||
`main` isn't the commit you want.
|
||||
|
||||
4. **Draft the coordination post.** Start a thread in the OpenShell channel of
|
||||
the CNCF Slack. The release ships from `main` with no release branch, so the
|
||||
thread is for getting ready PRs reviewed and merged before the cutoff. List
|
||||
approved PRs:
|
||||
|
||||
```shell
|
||||
gh pr list --base main --state open --search "review:approved" --json number,title,author --jq '.[] | "#\(.number) \(.title) (@\(.author.login))"'
|
||||
```
|
||||
|
||||
The release ships the final pre-release's commit, if it passes. Pick a
|
||||
cutoff before Tuesday's Auto-Tag run, or a later one after which you tag the
|
||||
final pre-release by hand. Don't promise that everything merged by the
|
||||
cutoff ships, or name the next version.
|
||||
|
||||
```text
|
||||
:thread: OpenShell vX.Y.Z release, Tuesday <date>. I'm release captain this week.
|
||||
|
||||
Status: vX.Y.Z-pre.N (<commit>) passed qualification. <blockers and owners, or "No blockers.">
|
||||
|
||||
Reply here with any ready PRs you'd like in vX.Y.Z. Merge cutoff is <HH:MM UTC> (<local time>) Tuesday. <"Tuesday's automatic pre-release picks up everything merged by then" | "I'll cut the final pre-release after the cutoff">, and vX.Y.Z ships from it if it passes qualification. Anything that misses it goes in the next release.
|
||||
```
|
||||
|
||||
If the pre-release that ships was cut before the posted cutoff, say so in
|
||||
the thread, with its commit and time.
|
||||
|
||||
## Tuesday: cut the release
|
||||
|
||||
1. **Select the final candidate.** For a late cutoff, cut the final `-pre.N`
|
||||
after the cutoff as in Monday step 3. When it finishes, run the preview and
|
||||
repeat Monday step 2 for its run.
|
||||
|
||||
2. **Request approval.** Present the candidate tag, commit, qualification run,
|
||||
compatibility assessment, commits since `$PREV`, and release notes preview:
|
||||
|
||||
```shell
|
||||
git log --oneline --no-merges "$PREV..<candidate_tag>"
|
||||
gh run download "$RUN" -n release-notes -D "$dir/notes" && cat "$dir/notes/release-notes.md"
|
||||
```
|
||||
|
||||
The notes come from each PR's "Release Notes" section. To fix them, edit the
|
||||
PR, then preview with
|
||||
`gh workflow run release.yml --ref main -f tag="$VERSION" -f notes-only=true`.
|
||||
|
||||
3. **Dispatch the release.** After approval:
|
||||
|
||||
```shell
|
||||
gh workflow run release.yml --ref main -f tag="$VERSION"
|
||||
```
|
||||
|
||||
4. **Watch and verify.** Check that `Select Qualified Prerelease` names the
|
||||
approved candidate, the GitHub Release has assets, no job failed, and the
|
||||
wheel reached PyPI:
|
||||
|
||||
```shell
|
||||
gh run watch <run-id> --exit-status
|
||||
gh release view "$VERSION" --json isPrerelease,publishedAt,assets --jq '{isPrerelease, publishedAt, assets: (.assets | length)}'
|
||||
```
|
||||
|
||||
5. **Close the loop.** Draft a short reply to the coordination thread. Point
|
||||
to the release notes rather than repeating them:
|
||||
|
||||
```text
|
||||
:white_check_mark: :ship: OpenShell vX.Y.Z is out: https://github.com/NVIDIA/OpenShell/releases/tag/vX.Y.Z
|
||||
|
||||
It shipped from vX.Y.Z-pre.N (<commit>). I'll be starting the <next version>-pre.* jobs shortly for next week.
|
||||
|
||||
Please check the release notes for upgrade notes that operators might care about. I'm also moving everything still open in the X.Y.Z milestone to <next milestone>.
|
||||
|
||||
Thanks everyone!
|
||||
```
|
||||
|
||||
After approval, move the milestone's open items to the next milestone.
|
||||
|
||||
6. **Start the next train.** This is this week's captain's last duty. Until the
|
||||
next `-pre.1` exists, Auto-Tag skips. Agree with the maintainers on patch
|
||||
(the default) or minor
|
||||
([version selection](../../../rfc/0014-release-stability/release-version-selection.md)).
|
||||
Seed it on a commit after `$VERSION` with a passing `Release Dev` run:
|
||||
|
||||
```shell
|
||||
for sha in $(git rev-list --max-count=10 "$VERSION..origin/main"); do
|
||||
state=$(gh run list --workflow release-dev.yml --commit "$sha" --json status,conclusion --jq '.[0] | if .status == "completed" then .conclusion else .status end')
|
||||
echo "${sha:0:9} ${state:-none} $(git log -1 --format=%s "$sha")"
|
||||
done
|
||||
```
|
||||
|
||||
After approval, push the tag. The `-c` and `/dev/null` keep `git` from
|
||||
signing the tag and opening an editor:
|
||||
|
||||
```shell
|
||||
NEXT=vX.Y.Z-pre.1 NEXT_SHA=<commit>
|
||||
git -c tag.gpgSign=false tag "$NEXT" "$NEXT_SHA" </dev/null
|
||||
git push origin "refs/tags/$NEXT"
|
||||
```
|
||||
|
||||
You're done when that `Pre-release` run passes, or its failure is handed to
|
||||
next week's captain.
|
||||
|
||||
## Recovery
|
||||
|
||||
| Symptom | Action |
|
||||
|---|---|
|
||||
| Transient infrastructure failure in a pre-release, including a registry push | After approval, `gh run rerun <run-id> --failed`. `Release` won't select the run until all of it passes. |
|
||||
| Pre-release run ends in `startup_failure` | The workflow file is invalid at that commit. Fix it in a new `-pre.N` (Monday step 3). |
|
||||
| Code defect or `Cargo Deny` advisory | Rerunning won't help. Fix it in a new `-pre.N` (Monday step 3). |
|
||||
| `Release` fails after selection | After approval, rerun only the failed jobs that haven't published. |
|
||||
|
||||
## Related
|
||||
|
||||
- `release-openshell`: the short procedure behind Tuesday steps 1–4.
|
||||
- `watch-github-actions`: general `gh run` monitoring and log retrieval.
|
||||
- `CI.md`: release workflows, qualification profile, and scanner policies.
|
||||
@@ -27,6 +27,7 @@ Run after adding, removing, or renaming a skill or crate; changing issue or PR c
|
||||
| Inference providers or native model endpoints | `debug-inference`, `openshell-cli` |
|
||||
| TUI behavior | `tui-development` |
|
||||
| Release artifacts or smoke coverage | `test-release-canary` |
|
||||
| Release cadence, tagging, or qualification workflows | `release-captain` |
|
||||
| CI diagnostics | `watch-github-actions` |
|
||||
| Gator supervision | `launch-openshell-gator` |
|
||||
| SBOM workflow | `sbom` |
|
||||
|
||||
@@ -637,6 +637,7 @@ These workflows run after merge to publish dev/tagged artifacts and verify them.
|
||||
|---|---|
|
||||
| `.github/workflows/release-dev.yml` | Publishes the rolling `dev` build on every push to `main`. Builds gateway, sandbox, and supervisor images and binaries, packages, wheels, and pushes the Helm chart as `oci://ghcr.io/nvidia/openshell/helm-chart:0.0.0-dev` (plus an immutable `0.0.0-dev.<sha>` pin). Also dispatchable manually. |
|
||||
| `.github/workflows/release.yml` | Manually promote the latest qualified prerelease for a version and branch, create its stable tag, then rebuild and publish. Support releases preserve shared latest channels. Publishes the prerelease's notes preview, retagged, or generates notes when it is unavailable. The notes-only mode generates a release notes preview artifact. |
|
||||
| `.github/workflows/release-auto-tag.yml` | Weekdays at 14:00 Europe/Zurich, tags the next `-pre.N` on `main` when commits landed since the latest one, then dispatches Pre-release. It skips when the train has no `-pre.1` or its version is already stable, so push each train's `-pre.1` tag by hand. |
|
||||
| `.github/workflows/release-prerelease.yml` | Pre-release orchestrator, triggered by prerelease tag pushes or dispatched by Release Auto-Tag or manually: build, then publish and qualify in parallel. Qualification failure fails the run and sends the failure notification; published artifacts remain available. |
|
||||
| `.github/workflows/release-build.yml` | Shared version computation, binary and image builds, and package builds. Validates that the tag matches the caller's release kind. |
|
||||
| `.github/workflows/release-publish.yml` | Shared artifact assembly, OCI image tags and Helm chart publication, and package publication. Stable releases also publish to GitHub Releases, snap, npm, PyPI, and Fern. |
|
||||
|
||||
@@ -117,6 +117,7 @@ Contributor and maintainer skills live in `.agents/skills/`. They are marked int
|
||||
| Platform | `build-openshell-mxc-windows` | Maintain and validate the build-only x64 and ARM64 Windows MSVC lane |
|
||||
| Documentation | `update-docs-from-commits` | Scan recent commits and draft doc updates for user-facing changes |
|
||||
| Maintenance | `sync-agent-infra` | Detect and fix drift across agent-first infrastructure files |
|
||||
| Maintenance | `release-captain` | Run the weekly Monday pre-flight and Tuesday stable release checklist |
|
||||
| Reference | `sbom` | Generate SBOMs and resolve dependency licenses |
|
||||
|
||||
### Issue Workflow
|
||||
|
||||
Reference in New Issue
Block a user