mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
docs(release): define qualification decisions (#3636)
Current maintenance status (2026-10-09): head
`0467f6e5b7ec0fbdfd4a20cf668c792c5f14b99e` includes current main
`01d9194` through a normal, conflict-free merge. The PR-specific diff
remains documentation-only (seven files). All 54 release-workflow tests
pass locally; workflow files match main. Hosted CI on this exact head is
terminal: 19 successful checks and four skips, with no failures or
running checks. The branch includes current main and is mergeable.
Independent human review and merge approval remain required. Earlier
evidence below applies to the prior head. This update does not merge the
PR or advance dependent PRs' upstream stack bases; independent human
merge approval is required.
## Description
Part of HEAOSS-6; split from #3382. Stack PR 1 defines five
release-qualification decisions: main/release semantics, artifact
identity, A1/B comparison, black-box benchmarks, and fail-closed
overrides. The split specification and implementation plan document the
remaining layers.
Current head: f4790a5359. Next PR: #3637.
## Type of Change
- [x] Documentation update
- [x] Bug fix (the restore command stages ADRs; the plan must inspect
the index)
## Changes Made
- Add the five ADRs and the specification/plan for the focused release
stack.
- Require both unstaged and cached whitespace checks after restoring
ADRs, stopping on either failure.
- Scope the placeholder scan to this slice's seven documents and
distinguish no matches from a failed scan.
## Testing
- [x] Actual disposable Git replay reproduces the staged-whitespace gap
and verifies the cached check catches it.
- [x] Both checks pass after correcting and restaging the document.
- [x] Seven actual changed documents are readable UTF-8, nonempty, and
contain no unresolved TBD/TODO markers.
- [x] Unstaged/staged whitespace checks pass on the published change.
```text
PASS reproduction: unstaged-only diff check exits 0; staged check exits 2 and reports trailing whitespace
PASS corrected staged document: both whitespace checks exit 0
PASS seven actual changed documents: readable UTF-8, nonempty, no unresolved TBD/TODO markers
PASS original source ADR untouched
```
## Real Behavior Proof
- Environment: Windows/Python 3.12.13/Git, disposable repository on G;
real ADR bytes copied from this checkout.
- Exact command / steps: `python
G:/git/headroom/.remember/tmp/pr3636-resume40-doc-replay.py`. Commit the
actual ADR as a baseline, append trailing whitespace, stage it, and
execute both Git checks. Correct/restage and rerun both checks; inspect
the seven real changed documents and verify the original ADR's digest is
unchanged.
- Observed result: The old unstaged-only check falsely succeeds while
the cached check exits 2 and identifies the staged violation. Both exit
0 after correction; the source ADR remains untouched and the changed
documentation has no unresolved placeholders.
- Not tested: Full six-PR reconstruction, release controller behavior,
candidate dispatch, production publishing, or documentation
publication/rendering. This is documentation-integrity proof.
- Evidence: pr3636-resume40-doc-replay.py/.log under
G:/git/headroom/.remember/tmp; owned replay repository
G:/tmp/headroom-review/resume40-3636-doc-replay.
## Runtime Rollout Safety
- Rollout-managed feature(s): Release qualification design;
documentation only.
- Minimum rollout channel: No runtime channel change.
- Stable/default behavior changed: The implementation plan checks staged
documents correctly; no executable release behavior is introduced here.
- Kill switch / disable path: Do not execute the plan; revert the
documentation.
- Unsafe override required: No.
- Qualification impact: Documents the intended fail-closed evidence
model; implementation/hosted qualification belongs to later stack PRs.
- Rollback path: Revert the ADR/plan slice; existing publisher remains
operational.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
Independent review remains required. No PR merge, self-approval or
release publication performed.
This commit is contained in:
@@ -0,0 +1,40 @@
|
||||
# ADR 0001: Main and release branch semantics
|
||||
|
||||
- Status: Accepted
|
||||
- Date: 2026-08-13
|
||||
|
||||
## Context
|
||||
|
||||
Headroom currently develops and releases from `main`. The target lifecycle needs
|
||||
`main` to continue moving while an exact source revision is qualified. A branch
|
||||
name alone is not evidence, and creating a `release` branch before qualification
|
||||
is authoritative would make unqualified code appear publishable.
|
||||
|
||||
## Decision
|
||||
|
||||
`main` remains the default integration branch and the target for ordinary pull
|
||||
requests. The future protected `release` branch represents qualified source. It
|
||||
may receive only an ancestry-preserving promotion of an exact Gate-A-qualified
|
||||
`main` SHA, a verified metadata-only Release Please change, an authorized hotfix,
|
||||
or release-control maintenance.
|
||||
|
||||
Promotion must leave the qualified candidate SHA reachable from `release`.
|
||||
Squash and rebase merges are therefore forbidden for promotion pull requests.
|
||||
Normal pull requests into `main` may retain the repository's usual merge policy.
|
||||
|
||||
The first `release` branch must be created from an exact SHA with a passing,
|
||||
unrevoked qualification manifest. Branch creation, protection, lifecycle
|
||||
activation, and publication credentials remain explicit administrator actions.
|
||||
They are not performed by this contract-only change.
|
||||
|
||||
Gate A admits source to `release`. Gate B independently authorizes publication
|
||||
of final distribution bytes. Neither branch membership nor a tag or GitHub
|
||||
Release substitutes for either gate.
|
||||
|
||||
## Consequences
|
||||
|
||||
Release Please will eventually watch `release`, but only after the qualification
|
||||
controller and migration guard are operational. Promotion automation must use an
|
||||
ancestry-preserving merge. A hotfix made on `release` requires qualification and
|
||||
forward synchronization to `main`; `release` must not become a second development
|
||||
branch.
|
||||
@@ -0,0 +1,35 @@
|
||||
# ADR 0002: Release artifact identity and provenance
|
||||
|
||||
- Status: Accepted
|
||||
- Date: 2026-08-13
|
||||
|
||||
## Context
|
||||
|
||||
Rebuilding a moving source reference can produce different bytes. Source identity
|
||||
alone cannot prove what qualification tested or what publication shipped.
|
||||
|
||||
## Decision
|
||||
|
||||
Every candidate is bound to an exact full Git commit SHA and an immutable artifact
|
||||
identified by SHA-256, filename, package name, version, size, and build-run
|
||||
identity. When runtime behavior comes from a payload that can differ from package
|
||||
bytes, its SHA-256 is recorded separately.
|
||||
|
||||
Evidence documents are themselves immutable bytes with a SHA-256 and an opaque
|
||||
URI. A producer identity records repository, workflow, run ID, attempt, and the
|
||||
producer commit. Cross-repository consumers verify the digest before processing
|
||||
the artifact or evidence. Secrets and credentials are never evidence fields.
|
||||
|
||||
The candidate artifact is built once. Downstream deterministic, integration, and
|
||||
benchmark lanes consume those exact bytes; they do not rebuild `main` and reuse
|
||||
the candidate identity. Final Gate B compares published bytes with the qualified
|
||||
artifact or applies an explicitly approved, policy-versioned runtime-payload
|
||||
equivalence rule.
|
||||
|
||||
## Consequences
|
||||
|
||||
Artifact stores and workflow runs require immutable retention sufficient for
|
||||
audit. A digest, source SHA, schema version, or producer mismatch makes evidence
|
||||
`inconclusive` or `fail` according to policy; it can never pass silently. Future
|
||||
controllers must enforce equality across documents in addition to validating
|
||||
each document's JSON Schema.
|
||||
@@ -0,0 +1,37 @@
|
||||
# ADR 0003: A1 versus B is the release benchmark claim
|
||||
|
||||
- Status: Accepted
|
||||
- Date: 2026-08-13
|
||||
|
||||
## Context
|
||||
|
||||
Comparing Headroom against a direct provider path changes more than compression
|
||||
and cannot isolate Headroom's effect. Release evidence needs a paired comparison
|
||||
whose intended difference is only optimization state.
|
||||
|
||||
## Decision
|
||||
|
||||
The headline release comparison is:
|
||||
|
||||
- A1 (`a1_passthrough`): the Headroom proxy and candidate artifact with
|
||||
optimization disabled.
|
||||
- B (`b_headroom`): the same Headroom proxy and candidate artifact with
|
||||
optimization enabled.
|
||||
|
||||
A1 and B must have equal candidate source SHA, artifact digest, rollout registry
|
||||
and snapshot digests, model, agent, taskset, environment image, and resource
|
||||
limits. Tasks are paired, arm order is randomized, and the versioned policy sets
|
||||
repeat count and statistical decision rules. A direct-provider A0 arm may be
|
||||
recorded for diagnosis but is not the headline comparator.
|
||||
|
||||
The benchmark producer reports an explicit identity-verification verdict. Any
|
||||
identity mismatch makes the experiment and release evidence inconclusive. JSON
|
||||
Schema constrains the arm shapes and intended optimization states; the benchmark
|
||||
worker and qualification controller must enforce equality between arms.
|
||||
|
||||
## Consequences
|
||||
|
||||
A single noisy run is not hard release evidence. Missing telemetry, provider
|
||||
outages, unequal environments, or identity drift remain visible and cannot be
|
||||
coerced into `pass`. Accuracy, savings, latency, reliability, and missing-data
|
||||
rules live in versioned policy rather than workflow shell code.
|
||||
@@ -0,0 +1,38 @@
|
||||
# ADR 0004: Black-box release benchmark boundary
|
||||
|
||||
- Status: Accepted
|
||||
- Date: 2026-08-13
|
||||
|
||||
## Context
|
||||
|
||||
Internal unit and compression tests are necessary but do not prove that the
|
||||
packaged proxy preserves real agent, streaming, and tool-use behavior. Conversely,
|
||||
benchmark infrastructure must not gain release credentials or execute arbitrary
|
||||
release commands.
|
||||
|
||||
## Decision
|
||||
|
||||
Release benchmarks exercise Headroom through its supported public protocol and
|
||||
observable provenance surfaces, using the immutable candidate artifact. They
|
||||
record taskset revision and digest, model and agent identities, environment,
|
||||
resource limits, rollout snapshot, artifact digest, attempts, telemetry
|
||||
completeness, and raw-result identity.
|
||||
|
||||
The benchmark repository receives only the minimum candidate and experiment
|
||||
identities needed to run. It verifies artifact digests before installation and
|
||||
returns an immutable result reference with authenticated producer identity. It
|
||||
does not receive provider secrets through release evidence, package-publish
|
||||
credentials, or permission to mutate the Headroom repository.
|
||||
|
||||
Failures and cancellations preserve terminal evidence. A provider or
|
||||
infrastructure failure is `inconclusive`, not a candidate pass and not necessarily
|
||||
a candidate-caused failure. The qualification controller applies the registered
|
||||
policy to the returned reference; callback identity alone does not establish a
|
||||
passing result.
|
||||
|
||||
## Consequences
|
||||
|
||||
Hermetic PR simulators, scheduled live smoke tests, and release-candidate lanes
|
||||
can share one result shape while retaining their execution depth. Cross-repository
|
||||
authentication, artifact transport, callback authorization, and retention need
|
||||
separate implementation and security review in later PRs.
|
||||
@@ -0,0 +1,40 @@
|
||||
# ADR 0005: Release overrides and fail-closed policy
|
||||
|
||||
- Status: Accepted
|
||||
- Date: 2026-08-13
|
||||
|
||||
## Context
|
||||
|
||||
Headroom product transforms may fail open to preserve requests, but release
|
||||
control makes a different safety decision: publication must not proceed without
|
||||
complete, matching, current evidence. Operational exceptions still need an
|
||||
auditable representation.
|
||||
|
||||
## Decision
|
||||
|
||||
Release evidence uses exactly four policy statuses: `pass`, `fail`,
|
||||
`inconclusive`, and `skipped_by_policy`. Missing, stale, invalid, mismatched,
|
||||
revoked, unauthorized, or schema-incompatible required evidence cannot produce
|
||||
`pass`. `skipped_by_policy` is allowed only when the selected policy version does
|
||||
not require that evidence.
|
||||
|
||||
Policy is versioned data. It maps risk classes to required Gate A and Gate B
|
||||
evidence and deterministic rules. Missing-data behavior is `inconclusive`.
|
||||
Thresholds and exceptions are not embedded as ad hoc workflow conditions.
|
||||
|
||||
An override records a unique ID, narrow scope, reason, human approver, approval
|
||||
time, expiry, and optional ticket URI. It never contains credentials. Controllers
|
||||
authenticate the approver, reject expired or unauthorized overrides, and include
|
||||
accepted overrides in the gate result. An unsafe runtime rollout override makes
|
||||
the rollout qualification-ineligible and cannot be overridden into a release
|
||||
pass by this mechanism.
|
||||
|
||||
Qualification revocation is explicit, immutable evidence linked to the original
|
||||
qualification. A revoked qualification cannot authorize promotion or publication.
|
||||
|
||||
## Consequences
|
||||
|
||||
The stable GitHub check will be named `release-policy` even when risk-specific
|
||||
evidence varies. Workflow code remains a thin adapter around a tested policy
|
||||
evaluator. Operational urgency stays visible to reviewers instead of silently
|
||||
weakening the evidence chain.
|
||||
@@ -0,0 +1,283 @@
|
||||
# HEAOSS-6 Pull Request Split Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Replace PR #3382 with six focused draft pull requests whose cumulative tip preserves its reviewed HEAOSS-6 behavior.
|
||||
|
||||
**Architecture:** Build a linear Git branch stack from current upstream `main`. Each branch introduces one independently useful layer: decisions, policy contracts, evidence contracts, manifest tooling, reusable release builds, then candidate orchestration.
|
||||
|
||||
**Tech Stack:** Git, GitHub CLI, Python 3.13, pytest, JSON Schema 2020-12, GitHub Actions, Ruff, mypy, actionlint
|
||||
|
||||
**Spec:** `docs/superpowers/specs/2026-09-17-heaoss-6-pr-split-design.md`
|
||||
|
||||
## Global Constraints
|
||||
|
||||
- The first branch is based on current upstream `main`; each later branch is based on its immediate predecessor.
|
||||
- Every pull request is opened as a draft and targets the preceding branch.
|
||||
- The original PR #3382 stays unchanged until all six drafts and the cumulative-equivalence audit exist.
|
||||
- No unrelated merge-commit changes from PR #3382 may enter the new stack.
|
||||
- Release publishing behavior remains intact after every executable slice.
|
||||
- Every PR body states `Part of HEAOSS-6; split from #3382.` and identifies its stack position.
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Release Qualification Decisions
|
||||
|
||||
**Files:**
|
||||
- Create: `docs/adr/0001-main-release-branch-semantics.md`
|
||||
- Create: `docs/adr/0002-release-artifact-identity.md`
|
||||
- Create: `docs/adr/0003-a1-b-release-benchmark.md`
|
||||
- Create: `docs/adr/0004-black-box-release-benchmarks.md`
|
||||
- Create: `docs/adr/0005-release-overrides-and-fail-closed-policy.md`
|
||||
- Existing: `docs/superpowers/specs/2026-09-17-heaoss-6-pr-split-design.md`
|
||||
- Existing: `docs/superpowers/plans/2026-09-17-heaoss-6-pr-split.md`
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: HEAOSS-6 decisions preserved on `upstream/pr-3382`.
|
||||
- Produces: Stable terminology for source, artifact, rollout, policy, benchmark, and override identity used by later PRs.
|
||||
|
||||
- [ ] **Step 1: Restore the five reviewed ADRs**
|
||||
|
||||
```powershell
|
||||
git checkout upstream/pr-3382 -- docs/adr/0001-main-release-branch-semantics.md docs/adr/0002-release-artifact-identity.md docs/adr/0003-a1-b-release-benchmark.md docs/adr/0004-black-box-release-benchmarks.md docs/adr/0005-release-overrides-and-fail-closed-policy.md
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Verify documentation integrity**
|
||||
|
||||
Run both `git diff --check` and `git diff --cached --check`; stop if either exits nonzero. The restore command stages the ADRs, so the cached check is required to inspect those files.
|
||||
Run `rg -n 'TB[D]|TO[D]O'` against the five restored ADRs and this split's specification and plan. Exit 1 means no placeholders; exit 0 reports unresolved markers, and exit 2 is a scan error.
|
||||
Expected: both whitespace checks exit 0 and no unresolved placeholder is present in these seven documents. Unrelated plans and ADRs are outside this slice's placeholder check.
|
||||
|
||||
- [ ] **Step 3: Commit the documentation slice**
|
||||
|
||||
```powershell
|
||||
git add docs/adr docs/superpowers
|
||||
git commit -m "docs(release): define qualification decisions"
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Push and open draft PR 1**
|
||||
|
||||
```powershell
|
||||
git push -u origin jd/heaoss-6-01-release-qualification-decisions
|
||||
gh pr create --repo headroomlabs-ai/headroom --draft --base main --head JerrettDavis:jd/heaoss-6-01-release-qualification-decisions --title "docs(release): define qualification decisions" --body-file <body-file>
|
||||
```
|
||||
|
||||
### Task 2: Shared Policy Contracts
|
||||
|
||||
**Files:**
|
||||
- Create: `release/contracts/common.schema.json`
|
||||
- Create: `release/contracts/policy.schema.json`
|
||||
- Create: `release/contracts/examples/policy.valid.json`
|
||||
- Create: `release/contracts/README.md`
|
||||
- Modify: `pyproject.toml`
|
||||
- Modify: `uv.lock`
|
||||
- Create: `tests/test_release_contracts.py`
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: ADR terminology from Task 1.
|
||||
- Produces: JSON Schema definitions referenced as `common.schema.json#/$defs/...` and a complete `policy.schema.json` consumer.
|
||||
|
||||
- [ ] **Step 1: Create the stacked branch and restore contract sources**
|
||||
|
||||
```powershell
|
||||
git switch -c jd/heaoss-6-02-release-policy-contracts
|
||||
git checkout upstream/pr-3382 -- release/contracts/common.schema.json release/contracts/policy.schema.json release/contracts/examples/policy.valid.json release/contracts/README.md pyproject.toml uv.lock
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Add focused policy contract tests**
|
||||
|
||||
Extract the schema-registry helpers and policy-specific cases from PR #3382's
|
||||
`tests/test_release_contracts.py`. The resulting test file must load local
|
||||
schemas through `referencing.Registry`, validate `policy.valid.json`, reject an
|
||||
unknown schema version, and enforce policy pass/fail invariants.
|
||||
|
||||
- [ ] **Step 3: Validate and commit**
|
||||
|
||||
Run: `uv lock --check`
|
||||
|
||||
Run: `uv run --extra dev pytest tests/test_release_contracts.py -q`
|
||||
|
||||
Run: `uv run --extra dev ruff check tests/test_release_contracts.py`
|
||||
|
||||
Expected: all commands exit 0.
|
||||
|
||||
```powershell
|
||||
git add release/contracts tests/test_release_contracts.py pyproject.toml uv.lock
|
||||
git commit -m "feat(release): define versioned policy contracts"
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Push and open draft PR 2 targeting PR 1's branch**
|
||||
|
||||
Use title `feat(release): define versioned policy contracts` and base
|
||||
`jd/heaoss-6-01-release-qualification-decisions`.
|
||||
|
||||
### Task 3: Qualification Evidence Contracts
|
||||
|
||||
**Files:**
|
||||
- Create: `release/contracts/benchmark-result-ref.schema.json`
|
||||
- Create: `release/contracts/candidate-manifest.schema.json`
|
||||
- Create: `release/contracts/gate-result.schema.json`
|
||||
- Create: `release/contracts/integration-result.schema.json`
|
||||
- Create: `release/contracts/qualification-manifest.schema.json`
|
||||
- Create: matching files under `release/contracts/examples/`
|
||||
- Modify: `tests/test_release_contracts.py`
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: `common.schema.json` definitions and the release policy contract.
|
||||
- Produces: Complete evidence formats used by candidate tooling and future qualification lanes.
|
||||
|
||||
- [ ] **Step 1: Create the branch and restore all remaining schemas/examples**
|
||||
|
||||
Create `jd/heaoss-6-03-qualification-evidence-contracts` from Task 2, then
|
||||
restore the declared files from `upstream/pr-3382`.
|
||||
|
||||
- [ ] **Step 2: Restore the complete reviewed contract test suite**
|
||||
|
||||
```powershell
|
||||
git checkout upstream/pr-3382 -- tests/test_release_contracts.py
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Validate and commit**
|
||||
|
||||
Run: `uv run --extra dev pytest tests/test_release_contracts.py -q`
|
||||
|
||||
Run: `uv run --extra dev ruff check tests/test_release_contracts.py`
|
||||
|
||||
Expected: all contract examples and negative invariants pass.
|
||||
|
||||
Commit as `feat(release): add qualification evidence contracts`.
|
||||
|
||||
- [ ] **Step 4: Push and open draft PR 3 targeting PR 2's branch**
|
||||
|
||||
### Task 4: Candidate Manifest CLI
|
||||
|
||||
**Files:**
|
||||
- Create: `scripts/candidate_manifest.py`
|
||||
- Create: `tests/test_candidate_manifest.py`
|
||||
- Modify: `.gitignore`
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: `release/contracts/candidate-manifest.schema.json` and common definitions.
|
||||
- Produces: `inventory`, `create`, and `verify` CLI commands used by Task 6.
|
||||
|
||||
- [ ] **Step 1: Create the branch and restore reviewed CLI files**
|
||||
|
||||
Create `jd/heaoss-6-04-candidate-manifest-cli`, then restore the three declared
|
||||
paths from `upstream/pr-3382`.
|
||||
|
||||
- [ ] **Step 2: Validate behavior and types**
|
||||
|
||||
Run: `uv run --extra dev pytest tests/test_candidate_manifest.py tests/test_release_contracts.py -q`
|
||||
|
||||
Run: `uv run --extra dev ruff check scripts/candidate_manifest.py tests/test_candidate_manifest.py`
|
||||
|
||||
Run: `uv run --extra dev mypy scripts/candidate_manifest.py`
|
||||
|
||||
Expected: all commands exit 0.
|
||||
|
||||
- [ ] **Step 3: Commit, push, and open draft PR 4 targeting PR 3's branch**
|
||||
|
||||
Commit as `feat(release): add immutable candidate manifest verifier`.
|
||||
|
||||
### Task 5: Read-Only Release Build Workflow
|
||||
|
||||
**Files:**
|
||||
- Create: `.github/workflows/release-build.yml`
|
||||
- Modify: `.github/workflows/release.yml`
|
||||
- Modify: `tests/test_release_workflows.py`
|
||||
- Modify: `docs/content/docs/releases.mdx`
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: Existing release build and smoke jobs.
|
||||
- Produces: Reusable `workflow_call` accepting `source_sha`, `manual_version`, and `release_tag`, with version/npm/canonical outputs and retained artifacts.
|
||||
|
||||
- [ ] **Step 1: Create the branch and restore the reviewed extraction**
|
||||
|
||||
Create `jd/heaoss-6-05-read-only-release-build`, then restore the four declared
|
||||
paths from `upstream/pr-3382`.
|
||||
|
||||
- [ ] **Step 2: Validate workflow structure and release invariants**
|
||||
|
||||
Run: `uv run --extra dev pytest tests/test_release_workflows.py -q`
|
||||
|
||||
Run: `actionlint .github/workflows/release.yml .github/workflows/release-build.yml`
|
||||
|
||||
Expected: 50 or more release workflow tests pass and actionlint emits no findings.
|
||||
|
||||
- [ ] **Step 3: Commit, push, and open draft PR 5 targeting PR 4's branch**
|
||||
|
||||
Commit as `refactor(release): extract read-only build workflow`.
|
||||
|
||||
### Task 6: Immutable Candidate Workflow
|
||||
|
||||
**Files:**
|
||||
- Create: `.github/workflows/candidate-artifact.yml`
|
||||
- Create: `tests/test_candidate_workflow.py`
|
||||
- Modify: `docs/content/docs/releases.mdx`
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: Task 4 CLI and Task 5 reusable workflow.
|
||||
- Produces: A manual exact-main-SHA candidate artifact and independently verified retained bytes.
|
||||
|
||||
- [ ] **Step 1: Create the branch and restore candidate orchestration**
|
||||
|
||||
Create `jd/heaoss-6-06-immutable-candidate-workflow`, then restore the workflow
|
||||
and test from `upstream/pr-3382`. Reconcile the release documentation so only
|
||||
candidate-specific additions remain in this branch.
|
||||
|
||||
- [ ] **Step 2: Run cumulative focused validation**
|
||||
|
||||
Run: `uv run --extra dev pytest tests/test_candidate_workflow.py tests/test_candidate_manifest.py tests/test_release_contracts.py tests/test_release_workflows.py -q`
|
||||
|
||||
Run: `uv run --extra dev ruff check scripts/candidate_manifest.py tests/test_candidate_manifest.py tests/test_candidate_workflow.py tests/test_release_contracts.py tests/test_release_workflows.py`
|
||||
|
||||
Run: `uv run --extra dev mypy scripts/candidate_manifest.py`
|
||||
|
||||
Run: `actionlint .github/workflows/candidate-artifact.yml .github/workflows/release-build.yml .github/workflows/release.yml`
|
||||
|
||||
Expected: 96 focused tests or the current-main-adjusted equivalent pass; lint,
|
||||
types, and workflow validation exit 0.
|
||||
|
||||
- [ ] **Step 3: Audit cumulative equivalence**
|
||||
|
||||
```powershell
|
||||
git diff --stat upstream/pr-3382...HEAD
|
||||
git diff upstream/pr-3382 HEAD -- .github docs/adr docs/content/docs/releases.mdx release scripts tests/test_candidate_manifest.py tests/test_candidate_workflow.py tests/test_release_contracts.py tests/test_release_workflows.py pyproject.toml uv.lock .gitignore
|
||||
```
|
||||
|
||||
Expected: only the approved split documents and intentional current-main
|
||||
reconciliation differ. Investigate and document every other difference.
|
||||
|
||||
- [ ] **Step 4: Commit, push, and open draft PR 6 targeting PR 5's branch**
|
||||
|
||||
Commit as `feat(release): build immutable exact-SHA candidates`.
|
||||
|
||||
### Task 7: Link the Stack and Supersede PR #3382
|
||||
|
||||
**Files:**
|
||||
- Modify: GitHub draft PR bodies only.
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: URLs and exact head/base SHAs for all six draft PRs.
|
||||
- Produces: Navigable stack metadata and a non-destructive supersession notice on #3382.
|
||||
|
||||
- [ ] **Step 1: Update each draft body with complete previous/next links**
|
||||
|
||||
Use `gh pr edit` and verify the rendered bodies with `gh pr view --json body`.
|
||||
|
||||
- [ ] **Step 2: Convert #3382 to draft and prepend the supersession notice**
|
||||
|
||||
The notice must link all six drafts and retain the prior validation history
|
||||
below it. Do not close #3382.
|
||||
|
||||
- [ ] **Step 3: Verify authoritative GitHub state**
|
||||
|
||||
Run `gh pr view` for all seven PRs and verify draft status, base branch, head
|
||||
branch, mergeability, and URLs. Confirm every new PR head SHA matches the pushed
|
||||
local branch.
|
||||
|
||||
- [ ] **Step 4: Report hosted-check status without claiming unobserved success**
|
||||
|
||||
List terminal, pending, or failing checks for each exact head. Hosted checks may
|
||||
continue after handoff; local validation and PR creation are required before
|
||||
the task is reported complete.
|
||||
@@ -0,0 +1,243 @@
|
||||
# HEAOSS-6 Pull Request Split Design
|
||||
|
||||
## Purpose
|
||||
|
||||
Replace the 3,279-line change in PR #3382 with a reviewable stack of six draft
|
||||
pull requests. Every pull request must have one coherent deliverable, pass its
|
||||
own focused validation, and leave the repository in a useful state if the stack
|
||||
stops there.
|
||||
|
||||
The final stack must preserve the behavior and safety properties already
|
||||
reviewed on PR #3382: versioned qualification evidence contracts, fail-closed
|
||||
candidate manifest verification, exact-main-SHA builds, a read-only build call
|
||||
tree, deterministic artifact inventory, and independent verification of the
|
||||
retained candidate bytes.
|
||||
|
||||
## Constraints
|
||||
|
||||
- Base the first branch on the current upstream `main`.
|
||||
- Stack every later branch on its immediate predecessor.
|
||||
- Keep each pull request draft until the whole stack is reconstructed and its
|
||||
cumulative tip is equivalent to PR #3382, except for deliberate documentation
|
||||
and history changes described here.
|
||||
- Do not modify or delete the original PR #3382 branch while constructing the
|
||||
stack.
|
||||
- Do not include unrelated changes from merge commits or the dirty primary
|
||||
checkout.
|
||||
- Preserve the release workflow's existing publish behavior at every executable
|
||||
step of the stack.
|
||||
- Use conventional commit messages and include the HEAOSS-6 context and stack
|
||||
position in every pull request body.
|
||||
|
||||
## Stack Topology
|
||||
|
||||
| Position | Branch | Base | Deliverable |
|
||||
| --- | --- | --- | --- |
|
||||
| 1 | `jd/heaoss-6-01-release-qualification-decisions` | `main` | Release qualification ADRs and this split specification |
|
||||
| 2 | `jd/heaoss-6-02-release-policy-contracts` | PR 1 branch | Shared evidence vocabulary and versioned release policy contract |
|
||||
| 3 | `jd/heaoss-6-03-qualification-evidence-contracts` | PR 2 branch | Candidate, benchmark, integration, gate, and qualification evidence schemas |
|
||||
| 4 | `jd/heaoss-6-04-candidate-manifest-cli` | PR 3 branch | Local candidate inventory, manifest creation, and fail-closed verification CLI |
|
||||
| 5 | `jd/heaoss-6-05-read-only-release-build` | PR 4 branch | Reusable read-only build/smoke workflow used by the existing release pipeline |
|
||||
| 6 | `jd/heaoss-6-06-immutable-candidate-workflow` | PR 5 branch | Manual exact-SHA candidate build, bundling, retention, and independent verification |
|
||||
|
||||
GitHub draft PRs use the preceding branch as their base. This makes each diff
|
||||
show only its own deliverable while retaining an executable cumulative tip.
|
||||
|
||||
## PR 1: Release Qualification Decisions
|
||||
|
||||
### Scope
|
||||
|
||||
- Add ADR 0001 defining authoritative-main release semantics.
|
||||
- Add ADR 0002 defining artifact and runtime-payload identity.
|
||||
- Add ADR 0003 defining the A1/B benchmark comparison.
|
||||
- Add ADR 0004 defining black-box release benchmarks.
|
||||
- Add ADR 0005 defining overrides and fail-closed policy.
|
||||
- Add this stack design document.
|
||||
|
||||
### Value at Merge
|
||||
|
||||
The repository gains an explicit, reviewable release-qualification model before
|
||||
automation encodes it. No runtime or workflow behavior changes.
|
||||
|
||||
### Validation
|
||||
|
||||
- Markdown and formatting checks applicable to documentation.
|
||||
- Confirm the ADRs contain no unresolved placeholders and agree on identity,
|
||||
policy, and override terminology.
|
||||
|
||||
## PR 2: Shared Policy Contracts
|
||||
|
||||
### Scope
|
||||
|
||||
- Add `release/contracts/common.schema.json` with shared identity, status,
|
||||
rollout, producer, environment, override, and equivalence definitions.
|
||||
- Add `release/contracts/policy.schema.json` and its valid example.
|
||||
- Add `release/contracts/README.md` covering schema versioning and validation.
|
||||
- Add `jsonschema>=4.23.0,<5.0` to the development dependencies and refresh
|
||||
`uv.lock`.
|
||||
- Introduce the reusable schema loading and validation test helpers in
|
||||
`tests/test_release_contracts.py`, with positive and negative policy tests.
|
||||
|
||||
### Value at Merge
|
||||
|
||||
Policy authors can publish and validate a complete versioned release policy.
|
||||
The shared vocabulary is exercised by a real consumer in the same pull request;
|
||||
it is not unused scaffolding.
|
||||
|
||||
### Validation
|
||||
|
||||
- `uv lock --check`
|
||||
- `uv run --extra dev pytest tests/test_release_contracts.py -q`
|
||||
- `uv run --extra dev ruff check tests/test_release_contracts.py`
|
||||
|
||||
## PR 3: Qualification Evidence Contracts
|
||||
|
||||
### Scope
|
||||
|
||||
- Add candidate-manifest, benchmark-result-reference, integration-result,
|
||||
gate-result, and qualification-manifest schemas.
|
||||
- Add one valid example for every schema.
|
||||
- Extend contract tests to validate every example, reject malformed identities,
|
||||
enforce pass/fail invariants, enforce Gate B runtime-payload equivalence, and
|
||||
reject unsupported schema versions.
|
||||
|
||||
### Value at Merge
|
||||
|
||||
Qualification producers and consumers have a complete, machine-validated
|
||||
evidence interchange format. The contracts are independently useful without
|
||||
the candidate workflow and can be adopted by later qualification lanes.
|
||||
|
||||
### Validation
|
||||
|
||||
- `uv run --extra dev pytest tests/test_release_contracts.py -q`
|
||||
- `uv run --extra dev ruff check tests/test_release_contracts.py`
|
||||
- Validate that all relative `$ref` targets resolve from the checked-in schema
|
||||
registry.
|
||||
|
||||
## PR 4: Candidate Manifest CLI
|
||||
|
||||
### Scope
|
||||
|
||||
- Add `scripts/candidate_manifest.py` with `inventory`, `create`, and `verify`
|
||||
commands.
|
||||
- Add `.candidate/` to `.gitignore` for local command output.
|
||||
- Add focused tests for deterministic ordering, hashes, sizes, package/version
|
||||
identity, source SHA, producer repository/workflow/run identity, rollout
|
||||
identity, safe qualification eligibility, schema versions, missing files,
|
||||
changed bytes, and unsupported embedded schemas.
|
||||
|
||||
### Value at Merge
|
||||
|
||||
Maintainers can create and independently verify immutable candidate manifests
|
||||
locally or from any later CI workflow. The CLI is fully functional before it is
|
||||
wired into GitHub Actions.
|
||||
|
||||
### Validation
|
||||
|
||||
- `uv run --extra dev pytest tests/test_candidate_manifest.py tests/test_release_contracts.py -q`
|
||||
- `uv run --extra dev ruff check scripts/candidate_manifest.py tests/test_candidate_manifest.py`
|
||||
- `uv run --extra dev mypy scripts/candidate_manifest.py`
|
||||
|
||||
## PR 5: Read-Only Release Build Workflow
|
||||
|
||||
### Scope
|
||||
|
||||
- Extract version resolution, package construction, the cross-platform wheel
|
||||
matrix, distribution collection, and smoke imports from `release.yml` into
|
||||
the reusable `release-build.yml` workflow.
|
||||
- Make `release.yml` call the reusable workflow, then retain all publishing and
|
||||
release creation in `release.yml`.
|
||||
- Ensure every publisher waits for the complete build/smoke group.
|
||||
- Preserve the release workflow entry points and published outputs.
|
||||
- Update release workflow tests and release documentation to describe the new
|
||||
boundary.
|
||||
|
||||
### Value at Merge
|
||||
|
||||
The production release uses a reusable, contents-read-only build boundary with
|
||||
no publishing job in its call tree. This is immediately useful even without the
|
||||
candidate workflow and removes duplicated future build logic.
|
||||
|
||||
### Validation
|
||||
|
||||
- `uv run --extra dev pytest tests/test_release_workflows.py -q`
|
||||
- `actionlint .github/workflows/release.yml .github/workflows/release-build.yml`
|
||||
- Hosted PR dry-run must complete all platform builds and smoke-import jobs;
|
||||
publishing jobs must remain skipped.
|
||||
|
||||
## PR 6: Immutable Candidate Workflow
|
||||
|
||||
### Scope
|
||||
|
||||
- Add `candidate-artifact.yml` as a manual workflow accepting only a full SHA
|
||||
reachable from authoritative `main`.
|
||||
- Call `release-build.yml` directly under `contents: read` permissions.
|
||||
- Produce a deterministic artifact inventory and X-001A candidate manifest.
|
||||
- Verify the manifest before upload, download the retained artifact in a
|
||||
separate job, and verify the exact bytes again.
|
||||
- Reject wrong source, repository, producer, workflow, run, attempt,
|
||||
package/version, filename, size, digest, rollout identity, unsafe rollout,
|
||||
qualification-ineligible rollout, or unsupported schema.
|
||||
- Add focused structural workflow tests and document the candidate workflow.
|
||||
|
||||
### Value at Merge
|
||||
|
||||
Maintainers can build and retain an immutable release candidate from an exact
|
||||
commit on authoritative `main` without exposing publishing permissions. The
|
||||
same bytes are verified independently after artifact retention.
|
||||
|
||||
### Validation
|
||||
|
||||
- `uv run --extra dev pytest tests/test_candidate_workflow.py tests/test_candidate_manifest.py tests/test_release_contracts.py tests/test_release_workflows.py -q`
|
||||
- `uv run --extra dev ruff check scripts/candidate_manifest.py tests/test_candidate_manifest.py tests/test_candidate_workflow.py tests/test_release_contracts.py tests/test_release_workflows.py`
|
||||
- `uv run --extra dev mypy scripts/candidate_manifest.py`
|
||||
- `actionlint .github/workflows/candidate-artifact.yml .github/workflows/release-build.yml .github/workflows/release.yml`
|
||||
- Hosted fork validation must build all five platform targets, run all smoke
|
||||
jobs, emit the candidate, download it independently, and verify its bytes.
|
||||
No publishing job may run.
|
||||
|
||||
## Reconstruction Method
|
||||
|
||||
For each branch, restore only the files or coherent hunks assigned to that pull
|
||||
request from PR #3382's reviewed tip. Where one original test file spans two
|
||||
stack positions, split its tests by the behavior introduced at that position.
|
||||
Commit and validate each branch before creating the next branch from it.
|
||||
|
||||
After PR 6, compare its cumulative tree to PR #3382 with `git diff`, allowing
|
||||
only this specification and any necessary stack metadata. Review every
|
||||
remaining difference rather than assuming equivalence from file counts.
|
||||
|
||||
## Draft Pull Request Format
|
||||
|
||||
Each draft body must include:
|
||||
|
||||
- `Stack: N of 6` and links to adjacent draft PRs when available.
|
||||
- The branch/base relationship.
|
||||
- A concise statement of the independently deliverable value.
|
||||
- Exact local validation results for that branch.
|
||||
- A note that hosted validation is pending or a link to its exact run.
|
||||
- `Part of HEAOSS-6; split from #3382.`
|
||||
- Runtime rollout safety and rollback notes appropriate to the slice.
|
||||
|
||||
## Original PR Transition
|
||||
|
||||
Keep #3382 unchanged until all six draft heads exist and the cumulative tip has
|
||||
passed local equivalence and validation checks. Then convert #3382 to draft and
|
||||
replace its leading status section with a supersession notice linking the six
|
||||
drafts. Do not close it until the stack is confirmed visible and complete; its
|
||||
existing discussion and hosted validation remain useful review evidence.
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
The split is complete only when:
|
||||
|
||||
1. Six draft PRs exist with the topology above.
|
||||
2. Every PR diff contains only its declared slice and has passing focused local
|
||||
validation.
|
||||
3. PR 5 demonstrates unchanged release build/smoke behavior with publication
|
||||
skipped on PR validation.
|
||||
4. PR 6 demonstrates candidate creation and independent byte verification with
|
||||
no publishing permissions or publishing jobs in its call tree.
|
||||
5. The cumulative PR 6 tree is reconciled against PR #3382 and every difference
|
||||
is documented and intentional.
|
||||
6. PR #3382 is a draft with a supersession notice linking the complete stack.
|
||||
Reference in New Issue
Block a user