Files
JD DavisandJerrett Davis b235bc2d27 fix(ci): verify installer provenance in build workflows (#4677)
## Current maintainer status (2026-11-10, current-main refresh)

Published head: `db4bdd71e7ddc1a72316445ef0d16a7d90b62951`, containing
upstream main `a385ea8e60a14c1511d82b21d518421af35deab2` through a clean
mechanical merge. All 13 remediation files are byte-identical to
reviewed head `81ff3ed384afbd404c48fd79d6bed41e8d647a99`; no installer,
action metadata, or checksum selection changed.

Fresh focused Windows checks: 61 passed and 23 platform skips. Fresh WSL
offline bootstrap/hook fixtures: 23 passed, with one pytest
configuration warning. These local checks use fixtures and do not claim
live vendor, provider, or native GUI execution.

Exact-head CI, including Docker-native, and the Init, Wrap, Rust,
governance, changelog, and merge-conflict workflows succeeded. Security
remains FAILURE: the OpenCode npm audit reports two moderate findings
through Effect advisory GHSA-chp3-j5qr-2vfr. The security workflow and
affected OpenCode package manifest/lock are byte-identical to current
main, establishing an inherited dependency finding. This is not an
all-green result or a fresh audit of main. No workflow rerun, dependency
downgrade, or advisory override was performed on this branch.

Independent human review and integration remain required. No
self-approval or merge was performed. Other branch publication and
execution holds remain active. All older qualification below belongs to
its recorded historical heads.

---



## Description

Build validation jobs fetched mutable installer scripts, and the pinned
commitlint action still declared a mutable Docker image tag. This pins
that image digest, verifies workflow-tool archives and cached binaries,
and replaces the Docker bootstrap scripts with pinned, verified sources.


## Type of Change

- [x] Bug fix



## Changes Made

- Preserve Rust 1.95.0, minimal/rustfmt/clippy options, Node 24, npm
12.2.1, and OpenCode 1.17.8.
- Preserve Debian userland architecture detection, Rustup ARM fallback,
and OpenCode x64 AVX2/baseline selection.
- Use an official Node image pinned by manifest digest; retain
npm/corepack trees and explicit runtime libraries.
- Add offline bootstrap tests for failed downloads, corrupt bytes,
unsupported targets, symlinks, and option forwarding.


## Testing

- [x] Offline bootstrap tests pass (stdlib unittest, 16 cases).

- [x] Ruff lint/format pass for the two new test files; shell syntax and
whitespace checks pass.
- [x] Exact-head hosted CI passed, including platform tests and
Docker-native, Init, and Wrap build/container checks.


### Test Output

```text

python3 -m unittest discover -s tests/test_install -p 'test_*bootstrap.py'

................

Ran 16 tests in 0.770s

OK

```



## Real Behavior Proof

- Environment: Ubuntu 24.04 under WSL, exact candidate
4ee38e8619.
- Exact command / steps: Run the unittest discovery command above.
Separately stream the recorded release archives and official Node image
layer through SHA-256 and inspect their tar headers.
- Observed result: Invalid bytes stop before extraction/execution;
fixture platform and CPU cases select the intended archives. Actual
workflow-tool/OpenCode archives and the Node layer match published
digests. The Rustup x64 binary matches its vendor checksum; all nine
embedded target checksums match vendor records.
- Not tested: No live model/provider behavior or unexercised
architecture is claimed. Local bootstrap fixtures use mocks; the hosted
Docker/build checks below exercised the actual pinned installation
paths.


## Runtime Rollout Safety

- Rollout-managed feature(s): None; CI/build bootstrap only.

- Minimum rollout channel: Not applicable.

- Stable/default behavior changed: Build inputs become pinned; proxy
defaults, Rust 1.95.0/components, npm 12.2.1 and OpenCode 1.17.8 stay
unchanged. Node remains on the 24 LTS line with a fixed image digest.
- Kill switch / disable path: Revert the maintenance change through
review; do not bypass checksum failures.
- Unsafe override required: None for this exact reviewed candidate.

- Qualification impact: The exact-head hosted checks below passed.
Existing PR heads retain their separate security/qualification holds
until main is remediated and their own gates are refreshed.
- Rollback path: Revert this maintenance PR if build compatibility
regresses.


## Review Readiness

- [x] I performed a self-review.

- [x] This PR is ready for human review.



## Checklist

- [x] New security regression tests are included.

- [x] CHANGELOG.md is unchanged.



## Additional Notes

Static independent code review found no defects. No merge or
self-approval is requested. Completed historical CI evidence is retained
within its original scope; this change qualifies future installer
execution separately.


Initial hosted validation of b2592de8d stopped at rustup filename
dispatch. Follow-up 4ee38e861 retains the required rustup-init basename
in a private directory and adds the reproducing regression test. All
nine latest applicable hosted workflows passed on the corrected head.


### Hosted validation on 4ee38e861

- [CI: platform tests and Docker-native
build/install/compose/wrap/init](https://github.com/headroomlabs-ai/headroom/actions/runs/42352878789):
success.
- [Wrap
E2E](https://github.com/headroomlabs-ai/headroom/actions/runs/42352878762),
[Init
E2E](https://github.com/headroomlabs-ai/headroom/actions/runs/42352878760),
[Init Native
E2E](https://github.com/headroomlabs-ai/headroom/actions/runs/42352878767),
and
[Security](https://github.com/headroomlabs-ai/headroom/actions/runs/42352878768):
success.
- Governance, changelog, Rust and merge-conflict checks also passed. The
older cancelled governance duplicate was superseded by a successful run.


Independent human review is required. No self-approval or merge has been
performed.




### Local-action metadata coverage follow-up



`38337d8be` adds `.github/actions/**` to both code and workflow filters.
The normal pytest suite now checks the vendored commitlint metadata
structure and immutable Docker digest; this is separate from actionlint
workflow parsing. Six focused CI regression tests pass, including a
filter regression that failed before the fix. Independent delta review
found no issues. `38337d8be` now has successful latest applicable CI,
Wrap E2E, Init E2E, Init Native E2E, Security, Rust, Changelog Guard,
Merge Conflicts, and PR Governance runs. The successful runs above
remain historical evidence for `4ee38e861`.




Current-head
[CI](https://github.com/headroomlabs-ai/headroom/actions/runs/42360858297),
[Wrap](https://github.com/headroomlabs-ai/headroom/actions/runs/42360858271),
[Init](https://github.com/headroomlabs-ai/headroom/actions/runs/42360858296),
[Init
Native](https://github.com/headroomlabs-ai/headroom/actions/runs/42360858178),
and
[Security](https://github.com/headroomlabs-ai/headroom/actions/runs/42360858352)
passed. Independent human review remains required; no self-approval or
merge.


<!-- maintainer-current-main-qualification -->

## Current-main qualification



Published head `9381378ffb07e466acef6a57eb39322aa84d4582` includes
upstream main `bb8d18a56abb07ca7e811235c86ea6b626eb1f8b` and is
mergeable. All nine exact-head workflows succeeded; all 53 jobs are
terminal success or explicit skips, including CI Docker-native,
wrap/init and native-init validation. Local metadata/release checks: 61
passed; 16 Linux-only bootstrap tests skipped on Windows. Remediation
installer/action sources are unchanged by the clean main integration.


Independent human review and merge are required. No self-approval or
merge performed. This PR does not lift the inherited
execution-provenance hold on main or other branches before human
integration.
<!-- /maintainer-current-main-qualification -->



### Maintainer current-main refresh (a2bcbd101)

Published `a20f072a238a1f8de422adf6a25d3f9d664151e4` incorporates
upstream main `a2bcbd101b1391debfd218ec8c63cb72d2bc240a` through a clean
merge. All 13 previously screened CI provenance remediation files are
byte-identical to prior published head
`9381378ffb07e466acef6a57eb39322aa84d4582`; no
installer/action/remediation logic was rewritten. Fresh focused checks:
61 passed, 23 platform-specific shell cases skipped on Windows. No
vendor installer, Docker image or downloaded binary was executed
locally. Earlier Linux bootstrap and hosted evidence retain their
original scope.

Fresh exact-head hosted validation completed successfully: all 10
observed workflow runs succeeded; all 56 checks are terminal successful
or explicitly skipped. Ready for independent human review on this
head/current main. This remediation branch alone is permitted to run its
screened checks; inherited execution/publication holds on main and other
PRs remain until independent human integration. Independent review is
required; no self-approval, merge or release publication.

---------

Co-authored-by: Jerrett Davis <JerrettDavis@users.noreply.github.com>
2026-11-10 21:02:42 -06:00
..

scripts/

Utility scripts bundled with the Headroom repo. Most are one-off operator tools; a few are runnable as part of development workflows.

Reproducing the reconnect storm

repro_codex_replay.py reproduces the multi-agent Codex reconnect/retry storm against a local Headroom proxy (default http://127.0.0.1:8787). Use it to:

  • Regression-check that /livez stays responsive under a cold-start storm.
  • Empirically tune the Unit 4 pre-upstream semaphore default (HEADROOM_ANTHROPIC_PRE_UPSTREAM_CONCURRENCY).
  • Exercise the Codex WS lifecycle + Anthropic HTTP path simultaneously without needing to replay captured production traffic.

Run

# Default: 8 WS + 4 HTTP clients, 30s storm, p99 /livez must stay <= 500ms.
python scripts/repro_codex_replay.py

# Tighter budget, shorter run:
python scripts/repro_codex_replay.py \
    --url http://127.0.0.1:8787 \
    --ws-clients 16 \
    --anthropic-clients 8 \
    --duration 60 \
    --livez-threshold-ms 100

# Dump the full summary as JSON for downstream tooling:
python scripts/repro_codex_replay.py --json

Exit code:

  • 0 — warmup succeeded (or was skipped), storm ran for the requested duration, and /livez p99 stayed under --livez-threshold-ms.
  • 1 — soft assertion failed, proxy unreachable, or unhandled exception. Proxy-unreachable is detected and reported within ~5 seconds.

Fixtures

The script loads two hand-crafted, fully synthetic JSON fixtures:

  • scripts/fixtures/anthropic_replay_body.json — shape of a large agent reconnect replay /v1/messages?beta=true POST body.
  • scripts/fixtures/codex_response_create_frame.json — first Codex WS frame with the {"type": "response.create", "response": {...}} envelope.

Override via --ws-frame-fixture / --anthropic-body-fixture if you have captured traffic to replay instead.

Interpretation

  • /livez p99 under threshold means the event loop is not starved during the storm. If it rises with the semaphore unbounded (HEADROOM_ANTHROPIC_PRE_UPSTREAM_CONCURRENCY=10000) and drops back under the default, Unit 4's backpressure is working.
  • Codex WS: opened should equal --ws-clients. response.completed typically stays low when upstream auth isn't configured locally — the goal is handshake + relay wiring, not real upstream traffic.
  • Anthropic HTTP: ok_2xx + non_2xx + timed_out + errors should roughly equal attempted. Sustained non-zero timed_out during the storm is the failure signal the plan targets.

A smoke test at tests/test_scripts/test_repro_codex_replay_smoke.py exercises the script against a mock FastAPI server on every PR.

Install scripts

  • install.sh — POSIX installer.
  • install.ps1 — Windows PowerShell installer.

These are generated by the release pipeline; edit with care.

Windows development bootstrap

bootstrap-windows-dev.ps1 prepares a Windows development checkout. It resolves or creates a repo-local Python virtual environment, checks for Rust, installs Python build/test tooling, installs npm dependencies for the TypeScript SDK and OpenClaw plugin, and runs a small smoke set.

powershell -ExecutionPolicy Bypass -File scripts/bootstrap-windows-dev.ps1

Use -CheckOnly to print detected tool versions without installing packages. Use -SkipSmoke, -SkipDocs, -SkipNode, or -SkipRust when intentionally debugging one part of the environment.

npm release asset smoke

build_npm_release_assets.mjs locally reproduces the release workflow's npm asset build. It builds the TypeScript SDK tarball, installs that tarball into OpenClaw, rewrites OpenClaw's release dependency to the same version, regenerates dist/package.json, packs OpenClaw, and then runs verify_npm_release_assets.mjs.

node scripts/build_npm_release_assets.mjs <version>

By default, output goes into a timestamped release-assets-local/<version>-* directory. Pass an explicit empty directory when you want a predictable path:

node scripts/build_npm_release_assets.mjs <version> release-assets-local/smoke

Expected tarballs:

  • headroom-ai-<version>.tgz
  • headroom-openclaw-<version>.tgz

The script restores package metadata after it finishes so the source tree keeps the registry-installable development dependency range.

Python release artifact smoke

build_python_release_smoke.py locally reproduces the Python artifact smoke: it builds a wheel with maturin, builds an sdist, verifies the sdist License-File metadata against tarball contents, installs the wheel into a fresh python -m venv environment, and imports the native headroom._core extension from that installed wheel.

python scripts/build_python_release_smoke.py

By default, the wheel uses the faster Cargo ci profile and output goes into a timestamped release-assets-local/python-<version>-* directory. Use --release when you want the slower shipped-wheel profile:

python scripts/build_python_release_smoke.py --release --out release-assets-local/python-release-smoke

Expected artifacts:

  • headroom_ai-<version>-*.whl
  • headroom_ai-<version>.tar.gz

Full local release smoke

release_smoke_all.py is the one-command local release gate. It first runs scripts/verify-versions.py, then runs the npm release asset smoke and the Python wheel/sdist smoke into sibling output directories.

python scripts/release_smoke_all.py

By default, output goes into release-assets-local/all-<version>-*/npm and release-assets-local/all-<version>-*/python. Pass an explicit empty output directory for a predictable evidence path:

python scripts/release_smoke_all.py --out release-assets-local/full-release-smoke

Use --python-release when the Python smoke should build with maturin's slower release profile. Use --skip-npm or --skip-python only when intentionally debugging one side of the artifact pipeline.