Files
paperclip/scripts/verify-release-registry-state.test.mjs
Devin Foley 8f7b8b3fda feat(release): add human-gated beta channel with stable soak enforcement (#11008)
> Follow-up to #11006 (merged): rebased onto master and ready for
review.

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The release subsystem now publishes canary (every master push),
nightly (scheduled, smoke-gated, added in #11006), and stable (manual)
> - There is still no human-approved release-candidate lane between
nightly and stable, and nothing enforces that a stable actually soaked
anywhere before shipping
> - Betas need a real approval gate, and stables need a soak policy that
is data, not prose
> - This pull request adds the beta channel: a manual promotion of a
chosen nightly behind the `npm-beta` environment gate, re-smoked after
publish, plus a stable preflight that enforces a 3-day beta soak with a
written-justification bypass
> - The benefit is a complete canary → nightly → beta → stable train
where every stable shipped as a beta first, and emergencies leave a
written trace

## Linked Issues or Issue Description

**Subsystem affected**

Release automation: `scripts/release.sh`, `scripts/release-lib.sh`,
`.github/workflows/release.yml`, `.github/workflows/docker.yml`,
`.github/workflows/release-smoke.yml`.

**Problem or motivation**

After #11006 the project has canary and nightly prerelease lanes, but no
release-candidate lane. Stable promotion has no enforced soak: any ref
can ship as stable directly. There is no approval boundary for a
broader-audience prerelease, and no structured way to record why an
emergency release skipped validation.

**Proposed solution**

Add a `beta` channel: a manual dispatch that promotes a chosen nightly's
source commit, publishes behind the `npm-beta` GitHub environment
(required reviewers are the gate), re-smokes the published beta, and
tags `beta/vX`. Enforce in the stable path that the source commit
shipped as a beta at least 3 days earlier (measured from the beta's npm
publish time), with a `skip_soak_justification` input as the recorded
emergency bypass.

**Alternatives considered**

Codifying the soak policy in docs only. Rejected: an unenforced policy
decays; the preflight makes the policy executable while the
justification input keeps the emergency path usable and auditable.

## What Changed

- `scripts/release.sh` + `scripts/release-lib.sh`: `beta` channel —
requires HEAD to carry a `nightly/v*` tag, publishes the package set as
`YYYY.MDD.P-beta.N` under dist-tag `beta`, tags
`beta/vYYYY.MDD.P-beta.N`
- `.github/workflows/release.yml`:
- `channel: beta` dispatch path: `select_beta` resolves the newest (or
an explicit `source_version`) nightly and fails loudly on selection
problems; `publish_beta` runs behind the `npm-beta` environment, pushes
the tag, and dispatches `docker.yml`; `smoke_beta` re-runs the release
smoke suite against the exact published beta version
- stable path: new `preflight_stable` job enforces the 3-day beta soak
from the beta's npm publish time; `skip_soak_justification` bypasses
with the reason echoed into the job summary; dry runs report without
blocking
- `.github/workflows/docker.yml`: `beta/v*` tags publish `:beta` on both
images, with exact version stamping
- `.github/workflows/release-smoke.yml`: `beta` added to the dispatch
choice list
- Docs: `CHANNELS.md` beta entries; `RELEASING.md` beta lane, soak gate,
and failure playbook; `RELEASE-AUTOMATION-SETUP.md` `npm-beta`
environment setup, including the warning to create the environment
before the first beta dispatch (GitHub auto-creates unprotected
environments on first reference)
- Tests: beta version-counting coverage in
`scripts/release-registry-versions.test.mjs`; beta identity and
nightly-tag guard coverage in
`scripts/__tests__/release-dry-run-notes.test.mjs`

## Verification

- `node --test` on the two touched suites: 17 pass, including the 3 new
beta tests
- `bash -n` on both shell scripts and YAML parse of all three workflows
- After merge, in order: create the `npm-beta` environment, dispatch
`channel: beta` with `dry_run: true` to preview, then a real promotion
of a published nightly through the approval gate, then a stable dry-run
against a young beta to see the soak gate report

## Risks

- If the `npm-beta` environment does not exist when the first beta
dispatch runs, GitHub creates it with no protection rules and the beta
publishes without approval. Mitigated by documentation and by creating
the environment before merge (operator step)
- Until the first beta exists, every stable dispatch requires
`skip_soak_justification`. This is deliberate — the first beta ships
immediately after this merges — but it is a behavior change to the
stable dispatch
- The soak clock reads the beta's npm publish time from the registry; a
registry outage makes the preflight fall back to requiring justification
(fail-closed)

## Model Used

Claude Fable 5 (`claude-fable-5`, Anthropic) in Claude Code, with
extended thinking and full tool use (repository exploration, local test
execution, live registry and git verification). All code, tests, and
docs in this PR were model-authored under human direction.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green (pending — will confirm before
merge)
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
(pending — will confirm before merge)
- [x] I will address all Greptile and reviewer comments before
requesting merge
2026-08-10 16:52:59 -07:00

402 lines
10 KiB
JavaScript

import assert from "node:assert/strict";
import test from "node:test";
import {
collectInternalDependencyProblems,
createManifestLookupKey,
fetchRegistryJson,
isCanaryVersion,
isPrereleaseVersion,
verifyPackageRegistryProblems,
verifyPackageRegistryState,
} from "./verify-release-registry-state.mjs";
test("isCanaryVersion matches release canaries", () => {
assert.equal(isCanaryVersion("2026.427.0-canary.3"), true);
assert.equal(isCanaryVersion("2026.427.0"), false);
});
test("isPrereleaseVersion matches canary and nightly versions", () => {
assert.equal(isPrereleaseVersion("2026.427.0-canary.3"), true);
assert.equal(isPrereleaseVersion("2026.427.0-nightly.0"), true);
assert.equal(isPrereleaseVersion("2026.427.0-beta.2"), true);
assert.equal(isPrereleaseVersion("2026.427.0"), false);
});
test("collectInternalDependencyProblems flags missing internal versions", () => {
const manifest = {
dependencies: {
"@paperclipai/plugin-sdk": "2026.425.0-canary.5",
e2b: "^2.19.0",
},
};
const packageDocsByName = new Map([
[
"@paperclipai/plugin-sdk",
{
versions: {
"2026.427.0-canary.3": {},
},
},
],
]);
assert.deepEqual(
collectInternalDependencyProblems(manifest, packageDocsByName),
["dependencies requires @paperclipai/plugin-sdk@2026.425.0-canary.5, but npm does not expose that version"],
);
});
test("collectInternalDependencyProblems accepts version-specific manifests when the root document is stale", () => {
const manifest = {
dependencies: {
"@paperclipai/plugin-sdk": "2026.425.0-canary.5",
},
};
const packageDocsByName = new Map([
[
"@paperclipai/plugin-sdk",
{
versions: {},
},
],
]);
const packageManifestsByKey = new Map([
[
createManifestLookupKey("@paperclipai/plugin-sdk", "2026.425.0-canary.5"),
{ name: "@paperclipai/plugin-sdk", version: "2026.425.0-canary.5" },
],
]);
assert.deepEqual(
collectInternalDependencyProblems(manifest, packageDocsByName, packageManifestsByKey),
[],
);
});
test("collectInternalDependencyProblems ignores peer dependency range specifiers", () => {
const manifest = {
peerDependencies: {
"@paperclipai/server": "^2026.430.0-canary.0",
},
};
assert.deepEqual(
collectInternalDependencyProblems(manifest, new Map()),
[],
);
});
test("collectInternalDependencyProblems reports unfetched transitive dependency metadata neutrally", () => {
const manifest = {
optionalDependencies: {
"@paperclipai/browser": "2026.430.0-canary.0",
},
};
assert.deepEqual(
collectInternalDependencyProblems(manifest, new Map()),
[
"optionalDependencies requires @paperclipai/browser@2026.430.0-canary.0, but npm publication metadata was not fetched for that dependency",
],
);
});
test("verifyPackageRegistryState tolerates a stale root versions map when dist-tags and direct manifests are correct", () => {
const packageDocsByName = new Map([
[
"@paperclipai/ui",
{
"dist-tags": {
canary: "2026.430.0-canary.0",
latest: "2026.430.0",
},
versions: {},
},
],
[
"@paperclipai/shared",
{
versions: {},
},
],
]);
const packageManifestsByKey = new Map([
[
createManifestLookupKey("@paperclipai/ui", "2026.430.0-canary.0"),
{
name: "@paperclipai/ui",
version: "2026.430.0-canary.0",
dependencies: {
"@paperclipai/shared": "2026.430.0-canary.0",
},
},
],
[
createManifestLookupKey("@paperclipai/shared", "2026.430.0-canary.0"),
{
name: "@paperclipai/shared",
version: "2026.430.0-canary.0",
},
],
]);
assert.deepEqual(
verifyPackageRegistryState({
packageName: "@paperclipai/ui",
packageDoc: packageDocsByName.get("@paperclipai/ui"),
packageDocsByName,
packageManifestsByKey,
channel: "canary",
distTag: "canary",
targetVersion: "2026.430.0-canary.0",
allowCanaryLatest: false,
}),
[],
);
});
test("verifyPackageRegistryState fails when canary latest is left in place by default", () => {
const packageDocsByName = new Map([
[
"@paperclipai/plugin-e2b",
{
"dist-tags": {
latest: "2026.425.0-canary.5",
canary: "2026.427.0-canary.3",
},
versions: {
"2026.425.0-canary.5": {
dependencies: {
"@paperclipai/plugin-sdk": "2026.425.0-canary.5",
},
},
"2026.427.0-canary.3": {
dependencies: {
"@paperclipai/plugin-sdk": "2026.427.0-canary.3",
},
},
},
},
],
[
"@paperclipai/plugin-sdk",
{
versions: {
"2026.427.0-canary.3": {},
},
},
],
]);
assert.deepEqual(
verifyPackageRegistryState({
packageName: "@paperclipai/plugin-e2b",
packageDoc: packageDocsByName.get("@paperclipai/plugin-e2b"),
packageDocsByName,
channel: "canary",
distTag: "canary",
targetVersion: "2026.427.0-canary.3",
allowCanaryLatest: false,
}),
[
"@paperclipai/plugin-e2b: latest dist-tag still resolves to prerelease 2026.425.0-canary.5; if that state is intentional, rerun the verification script directly with --allow-canary-latest",
"@paperclipai/plugin-e2b@2026.425.0-canary.5 via latest: dependencies requires @paperclipai/plugin-sdk@2026.425.0-canary.5, but npm does not expose that version",
],
);
});
test("verifyPackageRegistryProblems marks canary latest drift as non-retriable", () => {
const packageDocsByName = new Map([
[
"@paperclipai/plugin-e2b",
{
"dist-tags": {
latest: "2026.425.0-canary.5",
canary: "2026.427.0-canary.3",
},
versions: {
"2026.427.0-canary.3": {},
},
},
],
]);
const problems = verifyPackageRegistryProblems({
packageName: "@paperclipai/plugin-e2b",
packageDoc: packageDocsByName.get("@paperclipai/plugin-e2b"),
packageDocsByName,
channel: "canary",
distTag: "canary",
targetVersion: "2026.427.0-canary.3",
allowCanaryLatest: false,
});
assert.equal(problems[0]?.retriable, false);
assert.match(problems[0]?.message ?? "", /latest dist-tag still resolves to prerelease/);
});
test("verifyPackageRegistryProblems accepts the nightly channel and flags nightly latest drift", () => {
const packageDocsByName = new Map([
[
"@paperclipai/plugin-e2b",
{
"dist-tags": {
latest: "2026.425.0-nightly.1",
nightly: "2026.427.0-nightly.0",
},
versions: {
"2026.427.0-nightly.0": {},
},
},
],
]);
const problems = verifyPackageRegistryProblems({
packageName: "@paperclipai/plugin-e2b",
packageDoc: packageDocsByName.get("@paperclipai/plugin-e2b"),
packageDocsByName,
channel: "nightly",
distTag: "nightly",
targetVersion: "2026.427.0-nightly.0",
allowCanaryLatest: false,
});
assert.equal(problems[0]?.retriable, false);
assert.match(problems[0]?.message ?? "", /latest dist-tag still resolves to prerelease 2026\.425\.0-nightly\.1/);
});
test("verifyPackageRegistryState allows intentional canary latest but still checks dependencies", () => {
const packageDocsByName = new Map([
[
"paperclipai",
{
"dist-tags": {
latest: "2026.427.0-canary.3",
canary: "2026.427.0-canary.3",
},
versions: {
"2026.427.0-canary.3": {
dependencies: {
"@paperclipai/server": "2026.427.0-canary.3",
},
},
},
},
],
[
"@paperclipai/server",
{
versions: {
"2026.427.0-canary.3": {},
},
},
],
]);
assert.deepEqual(
verifyPackageRegistryState({
packageName: "paperclipai",
packageDoc: packageDocsByName.get("paperclipai"),
packageDocsByName,
channel: "canary",
distTag: "canary",
targetVersion: "2026.427.0-canary.3",
allowCanaryLatest: true,
}),
[],
);
});
test("verifyPackageRegistryState still fails when the dist-tag is stale", () => {
const packageDocsByName = new Map([
[
"@paperclipai/ui",
{
"dist-tags": {
canary: "2026.429.0-canary.2",
},
versions: {},
},
],
]);
const packageManifestsByKey = new Map([
[
createManifestLookupKey("@paperclipai/ui", "2026.430.0-canary.0"),
{
name: "@paperclipai/ui",
version: "2026.430.0-canary.0",
},
],
]);
assert.deepEqual(
verifyPackageRegistryState({
packageName: "@paperclipai/ui",
packageDoc: packageDocsByName.get("@paperclipai/ui"),
packageDocsByName,
packageManifestsByKey,
channel: "canary",
distTag: "canary",
targetVersion: "2026.430.0-canary.0",
allowCanaryLatest: false,
}),
["@paperclipai/ui: dist-tag canary resolves to 2026.429.0-canary.2, expected 2026.430.0-canary.0"],
);
});
test("verifyPackageRegistryState ignores internal peer dependency ranges", () => {
const packageDocsByName = new Map([
[
"@paperclipai/plugin-sdk",
{
"dist-tags": {
canary: "2026.430.0-canary.0",
},
versions: {
"2026.430.0-canary.0": {
peerDependencies: {
"@paperclipai/server": "^2026.430.0-canary.0",
},
},
},
},
],
]);
assert.deepEqual(
verifyPackageRegistryState({
packageName: "@paperclipai/plugin-sdk",
packageDoc: packageDocsByName.get("@paperclipai/plugin-sdk"),
packageDocsByName,
channel: "canary",
distTag: "canary",
targetVersion: "2026.430.0-canary.0",
allowCanaryLatest: false,
}),
[],
);
});
test("fetchRegistryJson times out hung requests", async () => {
const originalFetch = globalThis.fetch;
globalThis.fetch = (_url, { signal }) =>
new Promise((_resolve, reject) => {
signal.addEventListener(
"abort",
() => reject(new DOMException("The operation was aborted.", "AbortError")),
{ once: true },
);
});
try {
await assert.rejects(
fetchRegistryJson(new URL("https://registry.npmjs.org/@paperclipai%2Fui"), { timeoutMs: 1 }),
/timed out/,
);
} finally {
globalThis.fetch = originalFetch;
}
});