mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
feat/https-compose-caddy
55
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
8f6366cf66 |
fix(ci): keep the npm publish token out of install and build steps (#4690)
## Description
In `.github/workflows/release.yml`, job `publish-npm`, each package's
publish step set `NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}` for the
whole step, and the same step ran `npm ci` and `npm run build` before
`npm publish`. `npm ci` runs the install lifecycle scripts of every
dependency, so those scripts could read the publish token. This PR
splits each of the three packages (SDK, openclaw, opencode) into a
Prepare step with no token and a Publish step that carries the token and
runs only `npm publish`.
Fixes #4688
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `release.yml`: new `Prepare ... for npmjs.org` step per package (no
`env`): the "already published" `npm view` check, `cd`, `npm ci`, `npm
run build`, `npm version`, and for openclaw/opencode the dependency
rewrite and (openclaw) `node prepare-dist.mjs`. Same commands, same
order as before.
- `release.yml`: the existing `Publish ... to npmjs.org` step (same name
and id) now has `working-directory`, keeps `NODE_AUTH_TOKEN`, and runs
only `npm publish --access public`.
- The skip for an already-published version is passed between the two
steps with a step output (`skip=true` in `$GITHUB_OUTPUT`), and each
Publish step has `if: steps.<id>-prepare.outputs.skip != 'true'`.
Re-running after a partial release behaves as before.
- `tests/test_release_workflows.py`: new
`test_publish_npm_token_is_only_in_whoami_and_publish_steps`; the two
existing order tests now start at the Prepare step, where `npm version`
and the dependency rewrite now live.
- No `package.json` of the three packages has a `prepublishOnly`,
`prepack` or `prepare` script, so nothing needed at publish time was
moved out of the token step. I did not add `--ignore-scripts` to `npm
publish`.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
Fail before (new test only, on the unmodified workflow):
```text
FAILED tests/test_release_workflows.py::test_publish_npm_token_is_only_in_whoami_and_publish_steps
E assert ('npm ci' not in 'version="${...ess public\n'
E 'npm ci' is contained here:
E npm ci
E npm run build
E npm version "$version" --no-git-tag-version --allow-same-version
E npm publish --access public)
1 failed, 4 passed, 54 deselected
```
Pass after (whole file):
```text
tests/test_release_workflows.py .................ss..................... [ 67%]
................... [100%]
57 passed, 2 skipped in 277.72s (0:04:37)
```
Lint (touched test file): `ruff check` "All checks passed!", `ruff
format --check` "1 file already formatted". `mypy` is not relevant (no
`headroom` code changed).
Step-by-step env keys of the `publish-npm` job, parsed with PyYAML,
before:
```text
2 Verify npm publishing authentication env=['NODE_AUTH_TOKEN'] cmds=['npm whoami']
3 Publish ... (TypeScript SDK) to npmjs.org env=['NODE_AUTH_TOKEN'] cmds=['npm ci', 'npm run build', 'npm publish']
4 Publish ... openclaw to npmjs.org env=['NODE_AUTH_TOKEN'] cmds=['npm ci', 'npm run build', 'npm publish']
5 Publish ... opencode to npmjs.org env=['NODE_AUTH_TOKEN'] cmds=['npm ci', 'npm run build', 'npm publish']
```
after:
```text
2 Verify npm publishing authentication env=['NODE_AUTH_TOKEN'] cmds=['npm whoami']
3 Prepare ... (TypeScript SDK) env=[] cmds=['npm ci', 'npm run build']
4 Publish ... (TypeScript SDK) env=['NODE_AUTH_TOKEN'] wd=sdk/typescript cmds=['npm publish']
5 Prepare ... openclaw env=[] cmds=['npm ci', 'npm run build']
6 Publish ... openclaw env=['NODE_AUTH_TOKEN'] wd=plugins/openclaw cmds=['npm publish']
7 Prepare ... opencode env=[] cmds=['npm ci', 'npm run build']
8 Publish ... opencode env=['NODE_AUTH_TOKEN'] wd=plugins/opencode cmds=['npm publish']
```
## Real Behavior Proof
- Environment: Windows 11 host, workflow parsed in WSL Ubuntu with
PyYAML and pytest from the repo's test venv, against a clone at
`12864fc8` plus this change.
- Exact command / steps: parsed `.github/workflows/release.yml` with
`yaml.safe_load` and printed every `publish-npm` step's env keys,
working directory, `if` and npm commands before and after (output
above); ran `pytest tests/test_release_workflows.py`.
- Observed result: before, all three install+build+publish steps carry
`NODE_AUTH_TOKEN`; after, the token is only on `npm whoami` and the
three `npm publish` steps, and `npm ci`/`npm run build` steps have no
env. The whole workflow test file passes.
- Not tested: an actual GitHub Actions run (the workflow only publishes
on a release and needs `NPM_TOKEN`; I did not run it or publish
anything). `actionlint` is not installed here, so
`scripts/validate-workflows.sh` was not run. I did not check that
setup-node's `.npmrc` placeholder `NODE_AUTH_TOKEN` lets the token-less
`npm view` and `npm ci` run as before; the `npm view` of a public
package should not need auth, but a maintainer should confirm on the
first release.
## Runtime Rollout Safety
- Rollout-managed feature(s): none (release workflow only).
- Minimum rollout channel: n/a.
- Stable/default behavior changed: No product behavior. The release
workflow publishes the same packages with the same commands; only the
step boundaries and which steps see the token changed.
- Kill switch / disable path: n/a; the existing `publish-npm` gating of
the workflow is unchanged.
- Unsafe override required: No.
- Qualification impact: None.
- Rollback path: revert this commit.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Additional Notes
- Not changed, left for a maintainer: npm trusted publishing (OIDC,
`--provenance`) as suggested in the issue, which would remove the stored
`NPM_TOKEN`; and the same split in the `publish-github-packages` job,
which has the same shape with `GITHUB_TOKEN`.
- Documentation checkbox is N/A (no user-facing docs describe the
workflow steps).
Produced with AI assistance (Claude Code) operated by me (HetCreep); the
tests and proof above were run on my machine against a clone of the
repository.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- devin-review-badge-begin -->
---
<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4690"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
2973ca593d |
ci: pin every GitHub Action to a commit SHA (#4671)
## Description Every third-party action in `.github/` ran from a version tag like `actions/checkout@v7`. A tag can be moved to other code at any time. That is how `tj-actions/changed-files` leaked CI secrets from ~23,000 repos in March 2025. Here a moved tag would run with the release job's PyPI OIDC token (`id-token: write`) and the Docker job's `packages: write` token. Pinning to a commit SHA closes that, and a new test keeps it closed. Closes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [x] Code refactoring (no functional changes) ## Changes Made - All 195 third-party `uses:` refs in 25 files now name the commit their tag points to today, with the version as a comment: `actions/checkout@3d3c42e5... # v7.0.1`. No action version changes. - Dependabot's existing `github-actions` config updates SHA pins and their version comments, so the pins stay current. - Pin dtolnay/rust-toolchain stable and 1.95.0 branches to their existing commit heads. The stable action retains its runtime toolchain input/default; the 1.95.0 action embeds toolchain: 1.95.0. This preserves each call site's existing toolchain selection. - New `test_every_action_is_pinned_to_a_commit_sha` in `scripts/tests/test_ci_workflow.py` fails on any tag-pinned action and lists each one. - Three tests in `tests/test_release_workflows.py` matched the old tag text (`setup-buildx-action@v4`, `gh-action-pypi-publish@vX.Y.Z`, `release-please-action@v4/v5`). They now match the SHA pin and its version comment. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ uv run --frozen --extra dev pytest -q scripts/tests/test_ci_workflow.py scripts/tests/test_pr_health_workflow.py scripts/tests/test_normalize_release_pr_bodies.py tests/test_release_workflows.py 87 passed in 0.87s $ uv run --frozen --extra dev pytest -q scripts/tests/test_pr_health_labels.py scripts/tests/test_version_sync.py tests/test_cli/test_wrap_copilot.py tests/test_learn/test_subagent_scanning.py tests/test_plugin_manifests.py tests/test_pr208_changes.py 114 passed in 1.32s $ uvx ruff check scripts/tests/test_ci_workflow.py tests/test_release_workflows.py && uvx ruff format --check <same> All checks passed! ``` ## Real Behavior Proof - Environment: macOS, Python 3.12 via `uv run --frozen --extra dev`, branch cut from `main` at |
||
|
|
66e1467682 |
feat(plugins): headroom skill for Claude Code, Codex, Copilot and skills.sh (#4533)
## Description In the unbranded sandbox run the agent never searched and wrote a 300-line trimmer by hand; nothing in its environment said Headroom exists. An Agent Skill is the one artifact that fires on the job words (cut input tokens, trim tool outputs, fit a context window) with zero fetches. Four unofficial "headroom" skills already exist on skills.sh; there is no official one. Depends on #4529 for the `/docs/fastapi` link in the skill body. ## Type of Change - [x] New feature (non-breaking change that adds functionality) ## Changes Made - `skills/headroom/SKILL.md`: job-language description; body is the no-key `compress()` path, the zero-network check, `model_limit` / `protect_recent`, the TypeScript proxy caveat, the coding-agent path, and a short do-not list (no savings claims, no `CompressionMiddleware` for apps that call a provider, no `[all]` unless asked). - `plugins/headroom-skill/`: the same SKILL.md under `skills/`, `.claude-plugin/plugin.json` and `.github/plugin/plugin.json` (identical), README with install lines for Claude Code, Copilot CLI and `npx skills add`. - Both marketplace manifests: `headroom-skill` entry, inserted after the hooks plugin so `plugins[0]` stays as the existing tests expect. - `scripts/version-sync.py` and `scripts/verify-versions.py`: the two new manifests, so the next release PR passes the version gate. `scripts/tests/test_version_sync.py` fixture and assertion extended; `tests/test_release_workflows.py` gate list extended. - `tests/test_plugin_manifests.py`: the two SKILL.md copies must be byte-identical (the repo has no symlinks, for Windows checkouts), frontmatter has `name: headroom`; the skill plugin's two manifests match each other and the marketplace entry. ## Testing - [x] Unit tests pass: `tests/test_plugin_manifests.py scripts/tests/test_version_sync.py tests/test_release_workflows.py -k "manifest or version or skill or sync"`: 30 passed - [x] `ruff check` / `ruff format --check` on the touched Python - [x] `python scripts/verify-versions.py`: "All versions aligned at 0.42.0" - [x] Manual testing performed ### Test Output ```text pytest tests/test_plugin_manifests.py scripts/tests/test_version_sync.py tests/test_release_workflows.py -k "manifest or version or skill or sync": 30 passed python scripts/verify-versions.py: All versions aligned at 0.42.0 claude plugin validate .: ✔ Validation passed ``` ## Real Behavior Proof - Environment: macOS arm64, Claude Code CLI. - Exact command / steps: `claude plugin validate .` at the repo root of this branch. - Observed result: `✔ Validation passed`. The only warning is pre-existing and about `plugins/headroom-agent-hooks/README.md` lacking a marketplace install line. - Not tested: installing the plugin from the marketplace into a fresh Claude Code or Copilot CLI session and triggering the skill on a real task; skills.sh indexing (happens on their side once this is on `main`). ## Runtime Rollout Safety - Rollout-managed feature(s): none; documentation and metadata only, no runtime code path changes - Minimum rollout channel: n/a - Stable/default behavior changed: no - Kill switch / disable path: n/a (revert the PR) - Unsafe override required: no - Qualification impact: none - Rollback path: revert the commit ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> Co-authored-by: JD Davis <mxjerrett@gmail.com> |
||
|
|
670767eeb4 |
fix(docker): pass RUNTIME_HOME=/root for root bake targets (#4445)
## Description Fixes #4443. The four root bake targets (`runtime`, `runtime-code`, `runtime-slim`, `runtime-code-slim`) set `RUNTIME_USER=root` but not `RUNTIME_HOME`, so the Dockerfile default `/home/nonroot` was used for `VOLUME` and `WORKDIR` while the process runs with `HOME=/root`. `~/.headroom` therefore landed in the container layer instead of the declared volume. Passing `RUNTIME_HOME=/root` fixes the two python-slim targets, but the two slim targets build the distroless `runtime-slim` stage, which declared no `ARG RUNTIME_HOME` and no `VOLUME` at all, so the arg was a silent no-op there. That stage now declares `ARG RUNTIME_HOME` and `VOLUME ${RUNTIME_HOME}/.headroom`, and since distroless has no shell for `RUN mkdir`, the state directory is built in a small `runtime-slim-state` stage and copied in owned by `RUNTIME_USER`. Closes #4443 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `docker-bake.hcl`: pass `RUNTIME_HOME = "/root"` in the four targets that set `RUNTIME_USER = "root"`. - `Dockerfile`: new `runtime-slim-state` stage (`python:${PYTHON_VERSION}-slim`) that creates `/state/.headroom`; the distroless `runtime-slim` stage now declares `ARG RUNTIME_HOME=/home/nonroot`, copies that directory to `${RUNTIME_HOME}/` with `--chown=${RUNTIME_USER}:${RUNTIME_USER}`, and declares `VOLUME ${RUNTIME_HOME}/.headroom`, matching the python-slim stage. - `tests/test_release_workflows.py`: `test_docker_bake_runtime_args_are_declared_by_the_selected_stage` walks every bake target, resolves the Dockerfile stage it builds (including inherited parent stages), and checks that each `RUNTIME_*` arg it passes is declared by that stage, that root targets use `/root`, and that the stage puts `VOLUME ${RUNTIME_HOME}/.headroom` under the home it was given. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ pytest tests/test_release_workflows.py -q # head |
||
|
|
33f204406c |
fix(openclaw/release): ship responsive npm integrations (#1458)
## Description Fixes #977 and #2353. The v0.35.0 release built all three JavaScript packages successfully, but npm rejected every publish with ENEEDAUTH. Because each publish step used continue-on-error, the publish-npm job still concluded success. npm therefore remains on headroom-ai 0.22.4 and headroom-openclaw 0.1.0, while headroom-opencode is absent. The checked-in OpenClaw source had already corrected the old strict-schema mismatch and added a circuit breaker, but its assemble hook still called compression for every tiny conversation and allowed a 30-second default wait. This PR completes both the runtime correction and the delivery path instead of merely exposing one failure. ## Changes ### OpenClaw runtime - Skip proxy compression below an 800-character context floor (roughly 200 tokens), avoiding synchronous HTTP work on short conversations that cannot produce meaningful savings. - Measure actual content, including tool-call arguments, so a single large tool result remains eligible. - Allow operators to disable the floor with minContextChars=0. - Lower the interactive request ceiling from 30 seconds to 2 seconds. - Pass that timeout into the SDK, whose AbortSignal cancels the HTTP request, instead of racing a promise while the fetch continues in the background. - Preserve circuit-breaker fail-open behavior and document/schema the new defaults. ### npm release integrity - Authenticate with npm before publishing any package, so a missing or invalid NPM_TOKEN fails before a partial release begins. - Make SDK, OpenClaw, and OpenCode npm publish failures fatal when npm publishing is enabled. - Preserve NPM_SKIP=true as the explicit opt-out. - Make retries safe after partial publication by skipping only the exact package version already present, then continuing to missing packages. - Remove the false-success notice path. ## Existing #977 corrections verified - The current OpenClaw schema declares startupTimeoutMs, routeCodexViaProxy, gatewayProviderIds, request timeout, and circuit-breaker fields written/used by the installer and runtime. - wrap defaults to the real published package name, headroom-openclaw. - unwrap removes managed values while preserving unrelated user configuration. - The coding savings profile uses protect_recent=0 paired with type-based recent-read protection, avoiding the old blanket short-session protection without weakening tool-result fidelity. ## Testing - 80 OpenClaw plugin tests pass. - TypeScript typecheck and production build pass. - 123 OpenClaw wrap/provider, savings-profile, compress endpoint, and MCP contract tests pass. - 47 release-workflow and MCP contract tests pass. - The npm release asset builder installed/imported and verified the actual 0.35.1 SDK and OpenClaw tarballs. - Workflow YAML parses; Ruff, format, and git diff checks pass. ## Required repository action A maintainer must replace or restore the npm publishing credential in NPM_TOKEN, then rerun the v0.35.0 release workflow (or publish the next version). This PR makes that failure visible and safely rerunnable; repository code cannot provision the external npm credential. Co-authored with @liuyongcheng0000, whose original PR identified the masked npm failures. <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/1458" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> --------- Co-authored-by: Jerrett Davis <mxjerrett@gmail.com> |
||
|
|
8d5dde7765 |
fix(docker): install the redis extra in every image (#4514)
## Description The 0.42 images ship the replica-shared compress session store, but no published variant installs the `redis` client. Setting `HEADROOM_COMPRESS_SESSION_STORE=redis://...` stopped startup with `ConfigurationError`, and `HEADROOM_CCR_BACKEND=redis` degraded to the per-process store. This PR adds the `redis` extra to the Dockerfile default and to all nine `docker-bake.hcl` targets. It locks the invariant with a release test. It also documents the session fields of `/v1/compress` and the shared session store. Closes #4513 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (a fix or feature that changes existing functionality) - [x] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `Dockerfile`, `docker-bake.hcl`: `HEADROOM_EXTRAS` gains `redis` in the default and in all nine bake targets. - `tests/test_release_workflows.py`: new `test_every_docker_variant_includes_the_redis_extra` guards the Dockerfile default, the bake targets, and the `pyproject.toml` extra. - `docs/content/docs/installation.mdx`: new `redis` extras row. The Docker tag table shows the new extras. - `docs/content/docs/proxy.mdx`, `wiki/proxy.md`: documented the `session` response fields, `HEADROOM_COMPRESS_SESSION_STORE`, its failure behavior, and the `session_affinity` interaction. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text $ python -m pytest tests/test_release_workflows.py -q Pytest: 55 passed $ uvx ruff@0.17.0 check . All checks passed! ``` ## Real Behavior Proof - Environment: macOS worktree of this repository, Python 3.13. - Exact command / steps: `python -m pytest tests/test_release_workflows.py -q` and `uvx ruff@0.17.0 check .`. - Observed result: 55 tests passed, ruff clean. The new test fails against the pre-change `Dockerfile`, because its `ARG HEADROOM_EXTRAS` line lacked `redis`. - Not tested: the Docker images were not built. The release test asserts the build args statically, as the sibling Bedrock test does. ## Runtime Rollout Safety - Rollout-managed feature(s): none. The change touches build args, docs, and a test. - Minimum rollout channel: not applicable. - Stable/default behavior changed: every published image now installs the `redis` client. With the extra installed, `HEADROOM_COMPRESS_SESSION_STORE=redis://...` starts instead of failing, and `HEADROOM_CCR_BACKEND=redis` uses Redis instead of the per-process fallback. - Kill switch / disable path: drop `redis` from `HEADROOM_EXTRAS` at build time, or unset the two environment variables at runtime. - Unsafe override required: no. - Qualification impact: none. No runtime code path changed. The `redis` import stays lazy, so deployments that do not set the variables are unaffected. - Rollback path: revert this PR and rebuild. ## Review Readiness - [x] I performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I performed a self-review of my code - [x] I commented my code, particularly in hard-to-understand areas - [x] I made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md`. It is generated by release-please from my Conventional Commit PR title, and a CI guard enforces this. ## Screenshots (if applicable) Not applicable. ## Additional Notes The full test suite and `mypy headroom` were not run. This PR changes no file under `headroom/`, so I ran the affected suite (`tests/test_release_workflows.py`) and lint only. The docs half also answers part of #3184 (documentation of the storage backends). The CI half of #3184 stays open, and PR #3282 covers it. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4514" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: JD Davis <mxjerrett@gmail.com> Co-authored-by: Tejas Chopra <tejas@headroomlabs.ai> |
||
|
|
67ce7d0f31 |
fix(ci): clear dependency audit and gate Docker publishing (#3984)
This maintainer-requested corrective PR fixes red-main CI failures. The
task explicitly asks for the main-CI fix, providing the exception
context for CONTRIBUTING.md's CI-only restriction. Maintainers retain
the merge decision.
Docker manifest publishing previously used `always()` and ran after
cancelled builds, then failed downloading missing digest artifacts (main
Docker run 37372683144). Require `success()` for the complete build
matrix before publishing. Update the existing release regression test
while retaining signing order and root-only latest-promotion assertions.
The production dependency audit identified CVE-2026-104851 in fsspec
2025.10.0 and CVE-2026-104874 in multidict 6.7.1. Update only those lock
entries to the reported fixed versions, 2026.6.0 and 6.9.1.
## Real behavior proof
- Setup: Windows, Python 3.13.3, uv; workflow regression tests do not
use an LLM provider or credentials. Hosted CI uses its existing Linux
matrix.
- Before fix: hosted test shard 4 failed with `KeyError: 'if'`. The
updated success-gate regression failed locally before adding the
explicit gate: 1 failed, 53 passed. Hosted production audit reproduced
the fsspec vulnerability; a subsequent local audit reproduced the
multidict vulnerability.
- After fix: `uv run --no-project --with pytest --with pyyaml pytest
tests/test_release_workflows.py -q --noconftest` reports 54 passed.
`--noconftest` isolates these YAML regression tests from unrelated
application fixtures.
- `uv lock --check`, `actionlint .github/workflows/docker.yml`, and `git
diff --check` pass.
- Exported CI's production scope with `uv export --frozen --no-dev
--no-emit-project --no-hashes --extra all --format requirements-txt
--output-file <requirements>`; `uvx pip-audit --no-deps --disable-pip -r
<requirements>` reports no known vulnerabilities after both updates.
- Not tested locally: actual Docker image publication/signing, the full
application test suite, or remote-provider calls. Hosted validation
completed successfully on head
|
||
|
|
0d841feb89 |
ci: move Apple Silicon jobs off the retiring macos-14 runner (#3953)
## Description
GitHub retires the `macos-14` runner image on **2026-11-02**. Before
that, jobs on it fail during scheduled brownouts, the first being
**2026-10-05 14:00 UTC - 2026-10-06 00:00 UTC**, then Oct 12, 16, 19,
23, 26, 29 and 30 (same hours).
Three jobs still use it, all on the Apple Silicon path:
- `release.yml` `build-wheels` (the `aarch64-apple-darwin` wheel)
- `release.yml` `smoke-import-wheels` (the macOS native smoke gate,
which `publish-pypi` waits on)
- `rust.yml` wheel build
A release cut during a brownout fails, and after Nov 2 no
`aarch64-apple-darwin` wheel can be built.
This moves all three to `macos-26`, the newest arm64 GA image. I pinned
it explicitly instead of using `macos-latest`, so the next image
rotation is a reviewed change and not a silent toolchain jump.
`macos-15-intel` is unaffected and left alone.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
## Changes Made
- `release.yml`: `macos-14` -> `macos-26` in the `build-wheels` matrix
and the `smoke-import-wheels` matrix row. The row's `wheel_artifact`
changes with it (`wheels-macos-26-aarch64-apple-darwin`), because the
build job uploads as `wheels-${{ matrix.os }}-${{ matrix.target }}`.
- `release.yml`: the smoke-step comment no longer names an image
version.
- `rust.yml`: `macos-14` -> `macos-26`.
- `tests/test_release_workflows.py`: the required smoke-matrix entry is
now `runner: macos-26`.
## Testing
- [x] Unit tests pass (`pytest`)
```text
$ uv run --frozen --extra dev pytest tests/test_release_workflows.py -q
54 passed in 2.16s
```
## Real Behavior Proof
- Environment: GitHub-hosted `macos-26` arm64 runners, via this PR's own
CI. The PR touches `release.yml`, so its path filter triggers the
release dry-run (full wheel matrix plus `smoke-import-wheels`, publish
jobs skipped).
- Exact command / steps: push this branch; CI ran `Release` run
[37109226385](https://github.com/headroomlabs-ai/headroom/actions/runs/37109226385)
and `rust` run
[37109226215](https://github.com/headroomlabs-ai/headroom/actions/runs/37109226215)
on the new runner.
- Observed result: the macOS arm64 wheel builds on `macos-26`, keeps the
`macosx_11_0_arm64` tag, and passes the smoke-import gate. Per job:
- [`build-wheels (macos-26,
aarch64-apple-darwin)`](https://github.com/headroomlabs-ai/headroom/actions/runs/37109226385/job/111163877183):
success. It built `headroom_ai-1.0.0-cp310-abi3-macosx_11_0_arm64.whl`,
the same `macosx_11_0_arm64` platform tag the `macos-14` builds
published, so the minimum supported macOS is unchanged.
- [`smoke-import-wheels (macos-26, 3.13, aarch64-apple-darwin,
...)`](https://github.com/headroomlabs-ai/headroom/actions/runs/37109226385/job/111165038700):
success. The wheel installs into a fresh `python3.13` venv on the runner
and passes the staged smoke-import script.
- [`rust / wheels
(aarch64-apple-darwin)`](https://github.com/headroomlabs-ai/headroom/actions/runs/37109226215/job/111163758381):
success.
- Every other platform row in both runs also passed. The new toolchain
(Xcode 26.6, clang 21, up from Xcode 15.4, clang 15) builds and links
the Rust extension and ORT.
- Compatibility, checked against the `actions/runner-images` readmes
(macos-14-arm64 20260831, macos-26-arm64 20260907): both images ship the
same Python toolcache (3.11.9, 3.12.10, 3.13.15, 3.14.7), so the smoke
step's `python3.13` and the build's `--interpreter` list resolve the
same way.
- Not tested: an actual `release: published` run. The dry-run covers the
same build and smoke jobs; only the publish steps differ.
## Runtime Rollout Safety
- Rollout-managed feature(s): none. CI runner labels only; no runtime
code or `FEATURES` entry changes.
- Minimum rollout channel: N/A (no runtime behavior; nothing ships to
users except a wheel with the same platform tag).
- Stable/default behavior changed: no.
- Kill switch / disable path: N/A at runtime. In CI, revert this PR (or
set the three runner labels to `macos-15`, also a supported arm64
image).
- Unsafe override required: no.
- Qualification impact: none. The published macOS arm64 wheel keeps its
`macosx_11_0_arm64` tag and passed the same smoke-import gate.
- Rollback path: revert this PR. After 2026-11-02 a revert to `macos-14`
no longer runs, so roll back to `macos-15` instead.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] I have performed a self-review of my code
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`
## Additional Notes
The two red checks, `test (2)`
(`test_section_lifecycle.py::test_category_losing_its_last_pattern_drops_its_heading`)
and `test-dashboard-ui`
(`test_dashboard_co2_playwright.py::test_co2_card_renders_estimated_savings`,
`_open_dashboard() missing 1 required positional argument: 'lifetime'`),
fail identically on `main` ([run
37100184603](https://github.com/headroomlabs-ai/headroom/actions/runs/37100184603))
and are unrelated to this change.
`macos-15` would also work. I picked `macos-26` because it has the
longest runway and the same Python toolcache, so the larger Xcode jump
is the only extra risk, and this PR's dry-run exercises it. Happy to
switch to `macos-15` if you'd rather take the smaller toolchain step.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- devin-review-badge-begin -->
---
<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/3953"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: JD Davis <mxjerrett@gmail.com>
|
||
|
|
540f4da199 |
fix(ci): tolerate missing Docker cache blobs (#3945)
## Description The Docker workflow on `main` failed when BuildKit restored a valid GHA cache index whose backing blob had already been evicted. The ARM64 `slim-nonroot` build received `404 BlobNotFound`, so it never uploaded its digest artifact and the manifest job failed secondarily. This makes remote cache strictly best-effort: cache export errors are ignored, and a failed cached build is retried once with remote cache disabled. A real build failure remains fatal because the uncached retry does not use `continue-on-error`. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature - [ ] Breaking change - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring ## Changes Made - Mark the cached bake attempt as recoverable and make GHA cache export non-fatal. - Retry the identical per-platform digest build without cache import/export when the cached attempt fails. - Select digest metadata from the attempt that actually succeeded. - Add structural regression coverage for fallback failure semantics and metadata selection. ## Testing - [x] Unit tests pass (`tests/test_release_workflows.py`) - [x] Linting passes for changed Python test - [ ] Type checking passes (`mypy headroom`) — no typed Python production code changed - [x] New tests added for the fix - [x] Manual workflow validation performed ### Test Output ```text uv run --extra dev python -m pytest tests/test_release_workflows.py -q 54 passed in 2.79s uv run --extra dev ruff check tests/test_release_workflows.py All checks passed! uv run --extra dev ruff format --check tests/test_release_workflows.py 1 file already formatted actionlint v1.7.12 .github/workflows/docker.yml (exit 0) git diff --check upstream/main...HEAD (exit 0) ``` A repository-wide `pytest -q` run completed with 14,208 passed and 617 skipped, plus 53 unrelated failures and 20 setup errors in existing Windows/environment-sensitive suites. The changed release-workflow suite passed within that run. ## Real Behavior Proof - Environment: GitHub Actions Docker workflow on `main`, ARM64 native runner. - Exact failure: https://github.com/headroomlabs-ai/headroom/actions/runs/36963024403/job/110700641054 - Observed result: BuildKit failed lazy cache copy with `404 BlobNotFound`; `digests-slim-nonroot-arm64` was absent and the matching manifest job failed. - Not tested: hosted fallback execution cannot be forced safely because GitHub controls cache eviction; PR CI validates workflow syntax and regression invariants. ## Runtime Rollout Safety - Rollout-managed feature(s): none; CI-only change. - Stable/default behavior changed: Docker publishing retries uncached after a failed cached bake. - Kill switch / disable path: revert this commit or remove the fallback step. - Qualification impact: none. - Rollback path: revert the PR. ## Review Readiness - [x] I performed self-review - [x] PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I performed a self-review - [x] I commented the cache failure behavior - [x] Changes generate no new warnings - [x] I added tests proving the fix - [x] Relevant existing unit tests pass locally - [x] I did not edit `CHANGELOG.md` ## Additional Notes The fallback intentionally retries any failed cached bake, not only a parsed cache error. That avoids brittle log matching; deterministic build defects still fail on the required uncached attempt. <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/3945" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> |
||
|
|
23c2054552 |
ci(docker): stop passing bake metadata through env in Export digest (#3785)
## Description The `Export digest` step in `docker.yml` can fail before its script runs: ```text ##[error]An error occurred trying to start process '/usr/bin/bash' with working directory '/home/runner/work/headroom/headroom'. Argument list too long ``` The step already writes the bake metadata to a file through a heredoc. That was the fix in |
||
|
|
7f44a64490 |
ci(release): announce completed GitHub Releases in Slack (#3607)
## Description - announce a GitHub Release in Slack only after `create-release` has successfully uploaded its assets - build an accessible Block Kit payload with the exact release name, tag, repository, actor, and GitHub Release link - isolate the incoming webhook to the delivery step and fail closed on missing configuration or an unexpected Slack response - add a reproducible `act` smoke workflow whose `ACT=true` guard builds and validates the production payload without sending network traffic ## Changes Made - Adds a gated release-notification job, a local composite action, a payload builder, and smoke/regression coverage. ## Boundary and rollout This notification means **the GitHub Release and its assets are ready**. It intentionally does not claim that every package registry publish succeeded, matching the existing `create-release` contract. Configure `SLACK_RELEASES_WEBHOOK_URL` as a repository Actions secret containing an incoming webhook already bound to `#headroom`. Slack incoming webhooks cannot override their configured channel. No Slack request was made while developing or validating this PR. This is additive to the current publication flow and does not implement or bypass the future Gate B work tracked by Linear HEAOSS-19. ## Testing - Rebased onto current `main` at `94206e265`; refreshed focused qualification on `477f68047`. - [x] Focused tests pass ### Test Output ```text 68 passed ``` - `uv run --extra dev pytest tests/test_release_workflows.py tests/test_release_version.py -q` (68 passed) - `uv run --extra dev ruff check .github/actions/release-slack-notification/build_payload.py tests/test_release_workflows.py` - `uv run --extra dev ruff format --check .github/actions/release-slack-notification/build_payload.py tests/test_release_workflows.py` - `actionlint .github/act/release-slack-notification.yml .github/workflows/release.yml` - `act workflow_dispatch -W .github/act/release-slack-notification.yml -j smoke --pull=false` ## Type of Change - [x] CI / release automation ## Real Behavior Proof - Environment: maintainer macOS/Python 3.13 checkout at aedf2c2e. - Exact command / steps: `python -m pytest tests/test_release_workflows.py tests/test_release_version.py -q`. - Observed result: 68 passed; current hosted build, test and security checks pass. Payload and workflow guards are covered by the focused tests. - Not tested: delivery to the production Slack webhook. No notification or release was sent during this review. ## Runtime Rollout Safety - Rollout-managed feature(s): no runtime feature; release notification only. - Minimum rollout channel: human-reviewed repository workflow deployment. - Stable/default behavior changed: successful GitHub Release asset publication triggers the notification. - Kill switch / disable path: disable or revert the notification job; missing webhook configuration fails that job. - Unsafe override required: no. - Qualification impact: notification signifies GitHub Release assets ready, not registry-wide publication success. - Rollback path: revert the additive notification workflow/action. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review |
||
|
|
04cdf79ab0 |
fix: stabilize release checks and consolidate dependency updates (#3531)
## Description Consolidates the open dependency updates into one draft and fixes the remaining release 0.38.0 test failures. Release packaging already includes the merged Node 24 fix from #3516. The concurrency test now proves request overlap with a barrier, and the release workflow tests verify registry-range consistency and publication failure gating without hard-coding obsolete dependency versions. Updates npm, Cargo, Python, and GitHub Actions dependencies. Adds recurring audits of all five npm lockfiles at every severity. Upgrades CrewAI to remove its vulnerable json-repair 0.25.2 pin, and replaces yanked chacha20 and pypdfium2 releases. This remains a draft. All 67 hosted checks pass on |
||
|
|
96dfc7005f |
chore: finish the org and docs-domain migration (#3389)
Two stale identities left over from moves that were only half applied. ## GitHub org `install/state.py` already migrated Docker images to `ghcr.io/headroomlabs-ai/headroom` and carries a `_DEPRECATED_IMAGE_REPO` constant that rewrites the old path on load — so this move was decided long ago. Everything else still said `chopratejas/headroom`. Rewritten across packaging (`pyproject.toml`, `Cargo.toml`), both plugin marketplaces, the agent-hooks manifests, install scripts, CI templates, crate docs and tests. **Two deliberate exclusions:** - **`ghcr.io/chopratejas/headroom` is untouched everywhere.** It is the *input* to `_migrate_deprecated_image()` and to the tests covering it. Rewriting it would silently disable the migration. - **`CHANGELOG.md` keeps its 238 historical references.** GitHub's rename redirect resolves them, and rewriting would bury this diff in noise. **HuggingFace model ids are NOT affected and were never in scope.** `chopratejas/kompress-base`, `kompress-v2-base`, `kompress-finance`, `technique-router`, `technique-router-onnx` and `siglip-image-encoder-onnx` are account-scoped identifiers on a different service. All six were validated against the HF API before this change: | Model | HTTP | |---|---| | `kompress-base` | 200 | | `kompress-v2-base` | 200 | | `technique-router` | 200 | | `technique-router-onnx` | 200 | | `siglip-image-encoder-onnx` | 200 | | `kompress-finance` | **401 (private)** | The 401 is unrelated to this migration, but worth knowing: `kompress_compressor.py:1255` references it, so that path is not reachable by an OSS user without access. **One functional fix, not cosmetic:** `_marketplace_source()` (`headroom/cli/init.py`) returned the old org as its fallback, so plugin installs were pointed at the retired repo. ## Docs domain The site moved to `docs.headroomlabs.ai`, but eleven references still named the vercel.app host — including **`llms.txt` (27 links)**, which is served at `docs.headroomlabs.ai/llms.txt` and read by LLM crawlers, the **dashboard's own Documentation link**, and a `headroom wrap` help message. The docs app's canonical-URL fallbacks were fixed separately in #3388. ## Also folded in `RUST_DEV.md` corrections found while verifying the above: - `make build-wheel` points at `Cargo.toml`, not `pyproject.toml` - the Rust toolchain is pinned, not tracking `stable` - `headroom-simulators` was missing from the workspace listing - the Python `CompressionStore` defaults to **SQLite**, not in-memory, when `HEADROOM_CCR_BACKEND` is unset (verified by execution) — this one matters for multi-worker reasoning And the install scripts pointed at `docs/docker-install.md`, a path that no longer exists; they now point at the live docs page. ## Verification 297 tests pass, including the plugin-manifest, install-state and release-workflow suites that assert on these exact strings. `ruff check` clean. Unrelated lockfile churn (`uv.lock`, both `package-lock.json`) is deliberately excluded. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01HRaEPjQcCSAwD2xbQMNENu --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com> |
||
|
|
5d25abd356 |
test: repair three suite failures that are red on main (#3196)
## Summary
Three tests fail on a clean `main` full-suite run. None is a product
defect — all three are tests that stopped describing reality, and they
will noise up or block the 0.36.4 release.
| Test | Why it fails | Fix |
|---|---|---|
| `test_release_workflows::test_no_native_tls_in_wheel_build_tree` |
Shells out to `cargo`; raises `FileNotFoundError` wherever the Rust
toolchain is absent | Copied the skip guards its own dual already had |
|
`test_learn/test_integration::TestCodexIntegration::test_full_pipeline`
| Asserts `"Bash" in all_tools` against **real local Codex data**; Codex
renamed its shell tool | Assert what the test is for, across Codex
versions |
|
`test_graceful_shutdown::test_run_server_installs_cancelled_error_filter`
| Counts installs on the **process-global** `uvicorn.error` logger;
order-dependent | Isolate the global state; assert the real contract |
## 1. native-tls / cargo
The `openssl-sys` gate 30 lines above is described in-code as this
test's dual. It already skips when `cargo` is missing, **and** when
cargo fails for a reason other than `"package did not match"` (the Linux
wheel target not being installed locally). The native-tls test never
copied either guard.
Not disabled: CI installs the toolchain via `dtolnay/rust-toolchain`, so
the check still executes there. The skip only applies where cargo is
genuinely absent.
## 2. Codex tool vocabulary
This test runs against whatever Codex sessions the machine actually has
(gated by `HAS_CODEX_DATA`), and asserted:
```python
# Codex has only Bash tool (shell)
assert "Bash" in all_tools
```
Codex has since renamed its shell tool (`Bash` → `shell` → `exec`), and
0.149.0 added agent tools (`spawn_agent`, `send_message`, `wait`) beside
it. The assertion pinned one release's vocabulary, so it fails on any
current install.
It now asserts what the pipeline is actually being tested for — that
tool calls were extracted, including a shell-execution tool under any of
its known names — and names the remedy in the failure message for the
next rename.
**Still discriminating** (verified, not assumed):
| Scenario | Result |
|---|---|
| pipeline parsed nothing | fails ✓ |
| tool names garbled | fails ✓ |
| agent tools only, no shell tool | fails ✓ |
| real current Codex data | passes ✓ |
## 3. Global logger state
```python
if not any(isinstance(item, _SuppressCancelledErrorFilter) for item in uvicorn_error_logger.filters):
uvicorn_error_logger.addFilter(_SuppressCancelledErrorFilter())
```
`run_server` is deliberately idempotent and `uvicorn.error` is a
process-global logger, so any earlier test in the session that reached
`run_server` leaves the filter attached — and this test then observes
**zero** installs against its `== 1` assertion. It passes alone and
fails in a full run, which is exactly the symptom.
The test now clears and restores that global state around itself, and
additionally asserts the idempotence guard that is the real contract:
calling `run_server` twice must not stack a duplicate filter. The test
got stronger, not just quieter.
## Scope
Tests only — no product code is touched.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Tejas Chopra <tejas@Tejass-MacBook-Pro.local>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
|
||
|
|
bf651c3dc1 |
fix(docker): give :latest exactly one writer (#3154)
## Description Closes #3150. `ghcr.io/headroomlabs-ai/headroom:latest` resolved to the distroless `code-slim` build, whose `import onnxruntime` segfaults on arm64. The proxy imports onnxruntime at startup in cache mode, so the container never bound its port and `headroom deploy` crash-looped (exit 139) on Apple Silicon. @ricwo's report is exceptionally good — it isolates the base image with a copy-`site-packages`-onto-`debian:trixie-slim` experiment, and explicitly retracts an earlier wrong theory about the `cpuid_info` line. I verified the tagging half independently against the live registry: ``` latest sha256:6b34905489e3... <- identical 0.36.0-code-slim sha256:6b34905489e3... <- identical 0.36.0 sha256:bb8e77d01b54... ``` **Root cause, proven from the job log rather than inferred.** `docker/metadata-action` defaults to `latest=auto`, which appends a bare `latest` for any semver release — and its own log line reads `suffixLatest=false`, meaning the per-tag `suffix=` that keeps every other tag variant-scoped never reaches it. All eight variant cells therefore emitted `:latest`, and the last to finish won. From the 0.36.0 `code-slim` cell: ``` latest=auto suffixLatest=false tags: [..."ghcr.io/headroomlabs-ai/headroom:code-slim", "ghcr.io/headroomlabs-ai/headroom:latest"] pushing sha256:fbcbb68... to ghcr.io/headroomlabs-ai/headroom:latest ``` It landed on `code-slim` by scheduling luck. Any of the eight could have won on any release. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) ## Changes Made - **`flavor: latest=false`** on the `docker-manifest` metadata-action. Stops the tag being generated at all, leaving the root-cell promotion step as the single writer of `:latest`. - **A runtime guard** in `Create multi-arch manifest`: if a suffixed variant reaches the push carrying a bare `latest`, the job fails instead of publishing. `VARIANT_NAME` is passed via `env:` rather than spliced inline. - **A test that encodes the missing half of the contract.** `test_docker_latest_promotion_is_owned_by_root_manifest_cell` already existed and passed throughout — it asserted the *intended* writer was the root cell but never the *absence of unintended ones*. The new test asserts exclusivity: `latest=false` is set, no tag rule reintroduces `value=latest`, and the guard runs before anything is pushed. ## Testing - [x] Unit tests pass (`pytest`) ### Test Output ```text tests/test_release_workflows.py 48 passed, 1 skipped, 1 failed The failure is test_no_native_tls_in_wheel_build_tree: FileNotFoundError: [Errno 2] No such file or directory: 'cargo' Pre-existing and environmental — cargo is not installed on this machine; it fails identically on a clean main checkout. ruff check: All checks passed ruff format --check: 1 file already formatted YAML parses; flavor='latest=false', env keys ['IMAGE','DIGEST_DIR','VARIANT_NAME']. ``` ## Real Behavior Proof - Environment: macOS (darwin 25.4.0), worktree off `main`. Live registry queried anonymously via the GHCR token endpoint. - Exact command / steps: (1) resolved `latest`, `0.36.0` and all four variant tags to manifest digests directly from `ghcr.io/v2/.../manifests/*` to confirm the aliasing; (2) pulled the `docker-manifest (code-slim)` job log from the 0.36.0 release run to see which tags that cell actually pushed; (3) applied the fix and ran the workflow test suite; (4) **removed `latest=false` again and re-ran the new test** to confirm it reproduces the bug. - Observed result: `:latest` and `:0.36.0-code-slim` share digest `sha256:6b34905489e3...` while `:0.36.0` is `sha256:bb8e77d01b54...`, exactly as reported. The code-slim job log shows `latest=auto` / `suffixLatest=false` and `pushing ... to ghcr.io/headroomlabs-ai/headroom:latest`. With the fix removed the new test fails on `assert 'latest=false' in ''`; with it restored, it passes. - Not tested: I could not exercise the arm64 segfault or a real multi-arch push from here — no ghcr write credential and no arm64 runner. The tagging fix is verified at the config layer plus the registry evidence above; the end-to-end proof is the re-run described below. ## Runtime Rollout Safety - Rollout-managed feature(s): None. - Minimum rollout channel: n/a - Stable/default behavior changed: Yes, and that is the fix — `:latest` will track the plain Debian-based build instead of whichever variant cell happened to finish last. - Kill switch / disable path: n/a (CI tagging policy). - Unsafe override required: No. - Qualification impact: A variant cell that would publish a bare `latest` now fails the Docker job loudly rather than silently repointing the default tag. - Rollback path: Revert the commit. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Additional Notes **The live `:latest` is still wrong until the images are re-tagged.** Merging this fixes future releases but does not touch the registry. Once merged, run `docker.yml` via `workflow_dispatch` with `version=0.36.0` to rebuild and repoint `:latest` at the plain build. I don't hold a `write:packages` credential, so that step needs a maintainer. **Not fixed here, and it outlives this PR:** the distroless arm64 segfault itself. After this change `:latest` points at the Debian build that works, but `0.36.0-slim` and `0.36.0-code-slim` remain broken on arm64 for anyone selecting them explicitly. @ricwo's evidence points squarely at the distroless base — same wheel, same numpy 2.5.2, same Python 3.13.5, works on `debian:trixie-slim` and segfaults on distroless. That deserves its own issue; the two failures are independent and this one is a release-tagging bug, exactly as the report says. Related but separate, from an earlier audit of this same file: the four bare variants set `RUNTIME_USER = "root"` in `docker-bake.hcl` while `Dockerfile:162` defaults to `nonroot`, and the `runtime-default` (nonroot) bake target is referenced by the docs but by no workflow. Worth its own change. Co-authored-by: Tejas Chopra <tejas@Tejass-MacBook-Pro.local> |
||
|
|
ac8646aa3c |
fix(ci): scope the release credential and stop persisting it to disk (#3062)
## Description `RELEASE_PLEASE_TOKEN` is currently a maintainer's personal PAT. It bypasses branch and tag protection on `main` (`release-please.yml` says so in its own comment), and forging a tag with it fires `release.yml` and `docker.yml` on `release: published`, which publish to PyPI, npm and GHCR. If it is a classic token with `repo` scope it is also valid against every other repository that account can reach. `release-metadata-sync.yml` made that credential readable on the runner. `actions/checkout` defaults to `persist-credentials: true`, writing the token into `.git/config`, and the very next step runs `scripts/version-sync.py` **from the checked-out branch**. The trigger is a push to the glob `release-please--branches--**`, which is not a protected namespace, so a principal with push access could land a modified `version-sync.py` and read it. Closes #2955. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update ## Changes Made - Both release workflows now prefer a GitHub App installation token — scoped to this repository, expiring in an hour — over the PAT, via `actions/create-github-app-token@v3`. - The minting step is gated on `vars.RELEASE_APP_ID` and marked `continue-on-error`, so an unconfigured app falls through to the existing `PAT -> GITHUB_TOKEN` chain and nothing breaks today. - `release-metadata-sync.yml`'s checkout no longer persists credentials, and no longer receives a token at all. - The final push supplies the credential through the step's own `env` and an explicit remote URL, so it is never on disk while branch-supplied code runs. ## Testing - [x] Unit tests pass - [x] Linting passes (ruff check + format on the test file) - [ ] Type checking passes — N/A (YAML + test only) - [x] New tests added for new functionality ### Test Output ```text $ .venv/bin/python -m pytest tests/test_release_workflows.py -q 1 failed, 44 passed, 1 skipped in 0.23s ``` The single failure is `test_no_native_tls_in_wheel_build_tree`, which shells out to `cargo`. It reproduces identically on unmodified `main` on this machine (no Rust toolchain installed) and is unrelated to this change. New tests only: ```text $ .venv/bin/python -m pytest tests/test_release_workflows.py -q -k "persist_credentials or scoped_app_token" 3 passed, 46 deselected in 0.18s ``` Against the parent commit: ```text FAILED test_metadata_sync_does_not_persist_credentials_for_branch_supplied_code FAILED test_release_workflows_prefer_scoped_app_token[release-please.yml-release-please] FAILED test_release_workflows_prefer_scoped_app_token[release-metadata-sync.yml-sync] 3 failed, 46 deselected ``` ## Real Behavior Proof - Environment: macOS 15 (darwin 25.4.0), Python 3.12.13; workflows parsed with PyYAML, not executed on a runner. - Exact command / steps: parse both workflow files and assert (a) every `actions/checkout` step sets `persist-credentials: false` and receives no `token`, (b) exactly one gated `create-github-app-token` step exists per workflow, and (c) every credential consumer places `steps.app-token.outputs.token` ahead of `secrets.RELEASE_PLEASE_TOKEN` in its fallback chain. - Observed result: all three assertions pass on this branch and fail on the parent commit. Both files remain valid YAML. - **Not tested — important:** none of this has executed on a GitHub runner. I have not minted a real installation token, not confirmed the app-token step's `continue-on-error` fallback behaves as expected when `vars.RELEASE_APP_ID` is unset, and not performed a real push with the explicit-remote-URL form. The first live release run is the real test. ## Runtime Rollout Safety - Rollout-managed feature(s): none. - Minimum rollout channel: N/A. - Stable/default behavior changed: no, unless `vars.RELEASE_APP_ID` is set — without it both workflows resolve to exactly today's credential chain. - Kill switch / disable path: unset `vars.RELEASE_APP_ID` to fall back to the PAT. - Unsafe override required: none. - Qualification impact: none. - Rollback path: revert this commit. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective - [x] New and existing unit tests pass locally with my changes ## Additional Notes **This narrows blast radius; it does not make the trigger safe on its own.** For an `on: push` workflow GitHub reads the workflow file from the pushed ref, so a principal with push access can still edit this file on their branch. The durable fix is the scoped app token *plus revoking the personal PAT* — the revocation is a console action and is deliberately not in this commit. **Two repo settings are required to actually complete #2955**, and neither can land in git: ``` vars.RELEASE_APP_ID (repository variable) secrets.RELEASE_APP_PRIVATE_KEY (repository secret) ``` Until those exist this PR is a no-op on behavior and a defense-in-depth improvement on the `persist-credentials` path only. Co-authored-by: Tejas Chopra <tejas@Tejass-MacBook-Pro.local> |
||
|
|
eafdf11a2c |
fix(docker): ship Bedrock auth and current registry (#2982)
## Description Fixes #1551 and #1692. Every published Headroom Docker image now installs the existing `bedrock` extra, so `--backend bedrock` can authenticate with temporary STS, SSO, and credential-process credentials instead of failing because `botocore` is absent. Public Docker instructions now consistently use `ghcr.io/headroomlabs-ai/headroom`. Several still pointed at the old personal package, which is frozen at 0.27.0 and caused users to report that no latest image existed. ## Type of Change - [x] Bug fix - [ ] New feature - [ ] Breaking change - [x] Documentation update - [x] Build / CI ## Changes Made - Add `bedrock` to the standalone Dockerfile default extras. - Add `bedrock` to all nine root/code/slim/nonroot bake targets. - Replace obsolete personal GHCR references in README, llms.txt, Compose guidance, testing guidance, and wiki docs. - Add release contract tests for Bedrock dependencies and the current organization registry. ## Testing - [x] Focused Docker release and Bedrock preflight tests pass. - [x] Full updater suites pass: 69 tests. - [x] `uv run ruff check tests/test_release_workflows.py` - [x] `docker buildx bake --print` - [x] `git diff --check` ## Real Behavior Proof Before this change, every published bake target installed only `proxy` or `proxy,code`, so `AWS_SESSION_TOKEN` selected an unavailable botocore path. Public copy-paste commands also referenced `ghcr.io/chopratejas/headroom`, which the existing migration code and changelog identify as frozen at 0.27.0. After this change, all nine parsed bake targets install `bedrock`; the regression resolves that package extra and confirms `boto3` plus `botocore`. Every public Docker instruction covered by the contract names `ghcr.io/headroomlabs-ai/headroom`. ## Runtime Rollout Safety This changes image contents and documentation only; proxy routing and non-Docker installs are unchanged. Static AWS credentials remain unaffected. Existing manifests using the deprecated image continue to be migrated by the established install-state logic. Rollback is a Docker/bake extras and documentation revert. ## Review Readiness - [x] Two related Docker blockers batched in one PR - [x] Regression coverage included - [x] No unrelated lockfile changes - [x] Ready for review |
||
|
|
e269afb935 |
fix(ci): unjam release and Docker publishing (#2958)
## Description
Fixes two release-automation defects exposed by the 0.35.0 release:
1. Release Please grouped the single package into a PR titled `chore:
release main`, which could not be matched back to the `headroom-ai`
component/version and therefore never emitted the release event that
starts PyPI publishing.
2. Docker manifest jobs downloaded digest artifacts with overlapping
variant globs. For example, `digests-code-*` also selected code-nonroot,
code-slim, and code-slim-nonroot artifacts, yielding eight markers where
exactly two were required.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- Route the single root package through Release Please's normal
versioned-PR path.
- Preserve both `${component}` and `${version}` in generated release PR
titles.
- Download Docker amd64 and arm64 digest artifacts by exact name instead
of an overlapping variant glob.
- Add regression assertions for both Release Please title matching and
Docker artifact isolation.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
python -m pytest tests/test_release_workflows.py -q
44 passed in 2.29s
pre-commit hooks:
Sync plugin versions........................Passed
Verify Ruff version alignment...............Passed
check for merge conflicts...................Passed
ruff (legacy alias).........................Passed
ruff format.................................Passed
mypy........................................Passed
```
## Real Behavior Proof
- Environment: GitHub Actions release runs 31653073583 and 31664975515,
`main` at the 0.35.0 release merge (`93f2d7a2`).
- Exact command / steps: Inspected the Release Please rerun and each
failed Docker manifest job; enumerated the digest artifacts downloaded
by their configured patterns.
- Observed result: Release Please logged `There are untagged, merged
release PRs outstanding - aborting`. Docker's `slim`, `code`, and
`code-slim` manifest cells found 4, 8, and 4 digest markers respectively
instead of 2 because their prefix globs included related variants. The
new Docker workflow requests `digests-<variant>-amd64` and
`digests-<variant>-arm64` by exact name.
- Not tested: A synthetic release was not published because registry
versions/tags are irreversible. Both configuration invariants are
covered by regression tests, and GitHub's PR workflow validation runs
against this branch.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)
## Screenshots (if applicable)
N/A — workflow configuration and regression-test changes only.
## Additional Notes
- Documentation is N/A: these are internal release workflow corrections
with no user-facing command or API changes.
- The 0.35.0 Python release recovery is proceeding separately through
the existing wheel smoke-test and PyPI publication gates.
- The already-started 0.35.0 Docker run used the old workflow from tag
commit `93f2d7a2`; this PR prevents the artifact collision in subsequent
Docker runs.
|
||
|
|
78591545ce |
fix: publish headroom-opencode in release workflow (#2372)
## Description `headroom-opencode` is documented as an npm package, but the release workflow never published it, so installs failed with a registry 404 even though the plugin source already lived under `plugins/opencode`. This wires the existing package into the npm release path, keeps its version synced with root releases, and adds release guards for the new package. Closes #76. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - added `headroom-opencode` to the npm release workflow, including release-version stamping and `headroom-ai` dependency rewrite before publish - added `plugins/opencode/package.json` to release-please and local version-sync guards - synced the source opencode package version to the current release line and documented the new npm package in the release docs - added focused release workflow and version-sync tests for the opencode package - aligned the two failing dashboard Playwright tests with the current Session/Lifetime split and `/stats-lifetime` fixture contract ## Testing - [x] Unit tests pass (`uv run pytest scripts/tests/test_version_sync.py -q`, `uv run pytest tests/test_release_workflows.py -q -k 'publish_npm_rewrites_opencode_dependency_after_version_and_before_publish or opencode_source_dependency_matches_lockfile_registry_range or release_please_manifest_config_consistency'`) - [x] Unit tests pass (`uv run pytest tests/test_dashboard_cache_lifetime_playwright.py tests/test_dashboard_cache_ttl_playwright.py -q`) - [x] Linting passes (`uv run ruff check scripts/verify-versions.py scripts/version-sync.py scripts/tests/test_version_sync.py tests/test_release_workflows.py`) - [ ] Type checking passes (`uv run mypy headroom`) - [x] New tests added for new functionality when applicable - [x] Manual testing performed ### Test Output ```text $ uv run pytest scripts/tests/test_version_sync.py -q 8 passed, 1 warning in 0.51s $ uv run pytest tests/test_release_workflows.py -q -k 'publish_npm_rewrites_opencode_dependency_after_version_and_before_publish or opencode_source_dependency_matches_lockfile_registry_range or release_please_manifest_config_consistency' 2 passed, 38 deselected, 1 warning in 0.07s $ uv run pytest tests/test_dashboard_cache_lifetime_playwright.py tests/test_dashboard_cache_ttl_playwright.py -q 4 passed, 1 warning in 4.04s $ uv run ruff check scripts/verify-versions.py scripts/version-sync.py scripts/tests/test_version_sync.py tests/test_release_workflows.py All checks passed! $ npm ci && npm run build (plugins/opencode) Build success; dist/index.js, dist/entry.opencode.js, and DTS outputs emitted ``` ## Real Behavior Proof - Environment: Windows, Python 3.11.15, Node v24.15.0, npm 11.16.0 - Exact command / steps: inspected `.github/workflows/release.yml`, updated the npm publish path for `plugins/opencode`, aligned the two failing dashboard Playwright tests with the current Session/Lifetime split, then ran the focused pytest commands above plus `npm ci && npm run build` in `plugins/opencode` - Observed result: the release workflow now versions and publishes `headroom-opencode`, release-please and version-sync track `plugins/opencode/package.json`, the dashboard tests now fetch durable cache and setup-url data from the Lifetime view, and the opencode package still builds locally from source - Not tested: GitHub Package Registry publish ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I have updated the CHANGELOG.md if applicable ## Additional Notes `CHANGELOG.md` is unchanged because release-please owns changelog generation here. --------- Co-authored-by: JD Davis <mxjerrett@gmail.com> |
||
|
|
5568d738af |
fix(ci): publish latest from the root Docker manifest (#2252)
## Description A successful root Docker image can miss `:latest` when any optional variant manifest fails. The release workflow currently gates the standalone `promote-latest` job on the aggregate `docker-manifest` matrix, so one sibling failure skips promotion even when the signed root amd64+arm64 manifest exists. This moves `:latest` promotion into the successful root manifest cell. Optional variant failures remain visible and continue to fail their jobs, but they no longer suppress the image used by `headroom install`, which defaults to `ghcr.io/headroomlabs-ai/headroom:latest`. Refs #1583 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - Publish `:latest` from the root `docker-manifest` matrix cell after its versioned multi-architecture manifest is created and signed. - Remove the aggregate `promote-latest` dependency that allowed unrelated variant failures to suppress publication. - Keep all existing root, slim, code, and nonroot variants. - Preserve native linux/amd64 and linux/arm64 manifest assembly. - Add a focused workflow-contract regression test. ## Testing - [x] Unit tests pass (`uv run pytest tests/test_release_workflows.py -q -k "docker or latest"`) - [x] Linting passes (`uv run ruff check tests/test_release_workflows.py`) - [x] Formatting passes (`uv run ruff format --check tests/test_release_workflows.py`) - [x] New tests added for new functionality - [ ] Manual testing performed ### Test Output ```text Focused checks pass: `5 passed, 34 deselected` for `uv run pytest tests/test_release_workflows.py -q -k "docker or latest"`; the full test file has one unrelated Windows `FileNotFoundError` in `test_no_native_tls_in_wheel_build_tree` because its external command is unavailable. `uv run ruff check tests/test_release_workflows.py` and `uv run ruff format tests/test_release_workflows.py --check` pass. The repository-wide format check reports eight pre-existing files outside this target. Proof report: `D:\Repos\.claude\pr-sweep\headroom-PR-TARGET-1583-PROOF.md`. ``` ## Real Behavior Proof - Environment: Windows, Python managed by `uv`, repository workflow-contract tests; production publication owned by GitHub Actions and GHCR. - Exact command / steps: run the focused release-workflow tests; after merge, inspect the next Docker release run and execute `docker buildx imagetools inspect ghcr.io/headroomlabs-ai/headroom:latest` without registry login. - Observed result: local workflow-contract proof passes for root-owned promotion and both native architecture inputs; live GHCR publication remains unverified until the next release. - Not tested: production GHCR publication before merge. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [ ] My code follows the project's style guidelines - [ ] I have performed a self-review of my code - [x] Workflow comments explain the non-obvious root-only promotion boundary - [x] Documentation outside the changelog is unchanged because the CLI image reference is already correct - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective - [x] New and existing focused tests pass locally with my changes ## Screenshots (if applicable) Not applicable. ## Additional Notes Release run https://github.com/headroomlabs-ai/headroom/actions/runs/28404020512 demonstrated the cascade: the root manifest succeeded, a nonroot manifest failed during Buildx setup, and `promote-latest` was skipped. PR CI can prove the workflow dependency and architecture-preservation contracts. GHCR availability and anonymous package visibility require the next production release plus an unauthenticated registry inspection. |
||
|
|
5383c6bf2f |
fix(release): sync generated version metadata on the release branch (#2659)
## Description The 0.33.0 release PR (#2339) has sat in `changes-requested` since 2026-07-17. Root cause: **release-please only rewrites `pyproject.toml` and its configured `extra-files`**, but other tracked files also carry the version — and `server.json` is asserted byte-for-byte against `render_server_json()`, which derives its version from `pyproject.toml`. So the bump alone fails `tests/test_mcp_registry/test_server_json.py::test_root_server_json_matches_builder` (the `test (2)` shard) on every regenerated release PR. Nothing in the repo regenerated `server.json` at all, so it fell behind every release. Unblocks #2339. ### Why the release *build* passes but the release PR does not `release.yml` already runs `scripts/version-sync.py` immediately before its own `verify-versions.py` gate (lines 145 and 278). That is why `build` and `build-wheels` are green on #2339 despite the drift — it syncs in the workspace, uncommitted. The regular CI test job does **not** sync, so the fix has to be committed to the branch. This also explains why reviewers kept seeing `verify-versions.py` fail locally while CI's build jobs passed: the verifier is never run un-synced inside `release.yml`. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - **`scripts/version-sync.py`**: also write `server.json`. It was the one version-carrying file with no writer anywhere. Values are rewritten in place so key order and formatting keep matching the builder's byte-for-byte output (verified: the file is pure ASCII and round-trips exactly through `json.dumps(..., indent=2) + "\n"`). - **`.github/workflows/release-metadata-sync.yml`** (new): on a push to `release-please--branches--**`, run version-sync → gate on verify-versions → commit if changed. - **Keyed off the branch push** because release-please force-regenerates that branch on every merge to main. That is precisely what wiped the hand-pushed metadata fixes on #2339 (`2a86c8ff`, `d5ea4dc5`) — a push trigger re-heals after every regeneration instead of being lost. - **Uses the same PAT as `release-please.yml`**: a `GITHUB_TOKEN` push does not trigger workflows, so the release PR's checks would never re-run against the synced commit and would stay red. - **Idempotent**: the self-triggered rerun finds no diff and exits before pushing, so the loop terminates after one no-op run. - **Corrected pre-existing drift on `main`**: the agent-hooks plugin manifests, both marketplace manifests, and `.releasemetadata` were stranded at **0.31.0** — never bumped for 0.32.0 either. `verify-versions.py` now passes on `main`. ### Why not more `extra-files` entries That would need ~13 jsonpath entries restating what `version-sync.py` already knows, and a jsonpath that fails to match **fails silently** — the same class of failure this PR removes, discoverable only after a real release PR regenerates. There is also no precedent for nested jsonpath (`$.packages[0].version`, `$.metadata.version`) in the config today; both existing entries are plain `$.version`. Running the script keeps one source of truth, and files added to it later are covered with no change here. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) — no `headroom/` sources touched - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ python -m pytest scripts/tests/ tests/test_release_workflows.py tests/test_mcp_registry/ -q 207 passed in 2.69s $ ruff check scripts/version-sync.py scripts/tests/test_version_sync.py tests/test_release_workflows.py All checks passed! $ ruff format --check <same> 3 files already formatted $ actionlint .github/workflows/release-metadata-sync.yml (clean) ``` New tests: - `test_server_json_version_is_synchronized` — version-sync moves both `server.json` version fields and preserves the other keys. - `test_release_metadata_sync_runs_on_release_please_branch` — asserts the trigger, the sync→verify→commit ordering, the no-op guard, and the PAT. - `test_version_sync_covers_every_file_the_verifier_gates` — guards `version-sync.py` and `verify-versions.py` against drifting apart again, which is the root cause here. ## Real Behavior Proof - **Environment:** macOS (Darwin arm64), Python 3.12, repo venv. - **Exact command / steps:** reproduced the CI failure locally by simulating release-please's partial bump, then applying the fix. **Reproducing the exact `test (2)` failure** — set `pyproject` to 0.33.0 while `server.json` stays at 0.32.0, as release-please leaves it: ```text $ python -m pytest tests/test_mcp_registry/test_server_json.py -q FAILED tests/test_mcp_registry/test_server_json.py::test_root_server_json_matches_builder 1 failed, 3 passed ``` **After `version-sync.py`:** ```text $ python scripts/version-sync.py && python -m pytest tests/test_mcp_registry/test_server_json.py -q 4 passed ``` **Both gates green on a simulated 0.33.0 bump:** ```text $ python scripts/version-sync.py --version 0.33.0 Version synchronized to 0.33.0 $ python scripts/verify-versions.py All versions aligned at 0.33.0 $ python -m pytest tests/test_mcp_registry/test_server_json.py -q 4 passed ``` **Idempotency** (the property the workflow's loop-termination relies on): re-running against an already-synced tree leaves `pyproject.toml`, `server.json`, `openclaw`, and `sdk/typescript` untouched. - **Not tested:** the workflow has not executed on a real release-please branch regeneration — that can only be exercised once this is on `main` and release-please next updates #2339. The PAT push path and the self-trigger no-op are reasoned from `release-please.yml`'s existing token comment and from local idempotency, not observed in CI. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` ## Additional Notes **Context on the v0.32.0 release failure, since it is easy to misread as "images never build".** Every artifact built for v0.32.0 — all 5 wheel platforms including Windows, all 16 Docker builds + 8 manifests + `promote-latest`, npm, and GitHub Packages. Only `publish-pypi` failed (PyPI attestations, already fixed by `f9cbdd6e` / #2405), and `create-release` was skipped because it depends on it. That is why the release looked like it produced nothing. **Separate, approaching blocker — not addressed here.** PyPI is at **9.69 GB of its 10 GB project cap (96.9%)**, leaving ~305 MB against ~68 MB per release, so roughly 4 more releases fit. The `0.21.x` series alone holds **6.58 GB across 31 releases**, from the old every-push-is-a-release era; pruning it would reclaim two thirds of the quota. Worth a separate issue. **`.releasemetadata` is written but never read** by anything outside `version-sync.py` and its test. It is kept in sync here for internal consistency, but it may be a deletion candidate. |
||
|
|
5709291914 |
chore(release): harden local artifact smokes (#1824)
## Description
Harden the release artifact workflow so maintainers can reproduce npm
and Python release checks locally and so OpenClaw packaging does not
depend on a not-yet-published SDK version. This follow-up also keeps the
OpenClaw loader export contract explicit and updates the PR after the
branch was merged with current `headroomlabs/main`.
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [x] New feature (non-breaking change that adds functionality)
- [x] Documentation update
## Changes Made
- Added reusable local release smoke scripts for npm assets, Python
wheel/sdist artifacts, and the combined release gate.
- Switched the release workflow npm asset build to the reusable npm
builder.
- Made OpenClaw release asset building install against the just-built
local SDK tarball before rewriting packed metadata to the release
dependency range.
- Kept the OpenClaw source dependency registry-installable at
`headroom-ai@^0.22.3` while the packed artifact still ships
`^<release-version>`.
- Added `registerHeadroomPlugin` as a named export while preserving the
default `{ register }` OpenClaw loader contract.
- Added Windows development bootstrap docs/script and regression tests
for version sync, npm asset ordering, OpenClaw source installability,
and Python wheel smoke import isolation.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
uv run --with pytest python -m pytest tests/test_release_workflows.py scripts/tests/test_version_sync.py -q
44 passed, 1 warning
node --check scripts/build_npm_release_assets.mjs
node --check scripts/verify_npm_release_assets.mjs
python -m py_compile scripts/build_python_release_smoke.py scripts/release_smoke_all.py scripts/version-sync.py
python scripts/verify-versions.py
All versions aligned at 0.31.0
npm ci (plugins/openclaw)
added 88 packages, audited 89 packages, found 0 vulnerabilities
python scripts/release_smoke_all.py --out release-assets-local/all-pr1824-postmerge-fixed-20260710-104739
Verified npm release assets for 0.31.0
wheel metadata OK: headroom_ai-0.31.0-cp310-abi3-win_amd64.whl contains headroom/_core.pyd
sdist License-File metadata OK: ['LICENSE', 'NOTICE']
smoke-import OK: version=0.31.0 hello=headroom-core
```
## Real Behavior Proof
- Environment: Windows 11, Python 3.14.3, Node v24.14.0, npm 11.9.0, uv
0.11.15, Rust/Cargo already installed in the local development
environment.
- Exact command / steps: Ran `python scripts/release_smoke_all.py --out
release-assets-local/all-pr1824-postmerge-fixed-20260710-104739` after
syncing this branch with current `headroomlabs/main`.
- Observed result: The command built `headroom-ai-0.31.0.tgz`,
`headroom-openclaw-0.31.0.tgz`,
`headroom_ai-0.31.0-cp310-abi3-win_amd64.whl`, and
`headroom_ai-0.31.0.tar.gz`; npm verifier passed; the wheel installed
into a fresh venv and imported `headroom._core`.
- Not tested: Full GitHub Actions release matrix and publish jobs with
real PyPI/npm/GitHub Packages credentials.
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable
## Screenshots (if applicable)
N/A.
## Additional Notes
The post-merge full smoke initially failed because the OpenClaw source
package depended on `headroom-ai@^0.31.0`, which is not available on
public npm yet. The builder now installs OpenClaw against the just-built
local SDK tarball before build/pack, and then rewrites packed metadata
to `^0.31.0`.
---------
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
Co-authored-by: Tejas Chopra <chopratejas@gmail.com>
|
||
|
|
c46cd8f950 |
fix(core): load ONNX Runtime dynamically so headroom._core imports on non-AVX2 x86-64 (#1715)
## Description
`import headroom._core` dies with SIGILL (`Illegal instruction`) on
x86-64 CPUs without AVX2 (Pentium N4200, Celeron N4500, AMD FX 8350 —
all reported on the issue). The repo sets no `RUSTFLAGS`/`target-cpu`
anywhere, so first-party Rust code is baseline x86-64; the AVX2 code
comes from Microsoft's prebuilt ONNX Runtime, statically linked into the
extension by fastembed's `ort-download-binaries-rustls-tls` feature on
non-Windows targets. Because it is statically linked, its code is mapped
and initialized when the extension module loads — **before** the runtime
AVX2 guard from #1162 can run, which is why that fix helped Magika init
but not the import-time crash.
Fix, mirroring what Windows already does for its own reasons (DirectML
link libs): build with `ort-load-dynamic` on every platform, so ONNX
Runtime is only `dlopen`'d at first use, where the #1162 AVX2 guard
falls back to the non-ONNX detection tiers on unsupported CPUs. Since
both target blocks became identical, they are collapsed into one
platform-independent `fastembed` dependency.
To keep Magika/fastembed working out of the box on Linux/macOS, the
existing `ORT_DYLIB_PATH` auto-pin (`headroom/_ort.py`, previously
Windows-only) now resolves the pip `onnxruntime` package's shared
library on all platforms (`onnxruntime.dll` / `libonnxruntime.so*` /
`libonnxruntime*.dylib`). The pip `onnxruntime` CPU wheels use runtime
CPU dispatch, so they also work on pre-AVX2 machines — non-AVX2 users
get working ML detection instead of a crash. Without the `onnxruntime`
package, ML detection degrades gracefully to the non-ONNX tiers exactly
as it already does on Windows.
Fixes #1278
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `crates/headroom-core/Cargo.toml`: replaced the per-target `fastembed`
blocks (`ort-download-binaries-rustls-tls` on non-Windows,
`ort-load-dynamic` on Windows) with a single platform-independent
dependency on `ort-load-dynamic`, with a comment documenting both the
DirectML and the AVX2/#1278 rationale.
- `Cargo.lock`: regenerated — `ort-sys` drops its static-download
dependencies (`hmac-sha256`, `lzma-rust2`, `ureq`); no version bumps.
- `headroom/_ort.py`: `ORT_DYLIB_PATH` auto-pin extended from
Windows-only to all platforms via a small `_find_dylib` helper that
resolves the platform's shared-library name inside the pip `onnxruntime`
package.
- `tests/test_transforms/test_ort_dylib.py`: replaced the obsolete
`test_noop_on_non_windows` with Linux (versioned `.so`) and macOS
(`.dylib`) pin tests; module docstring updated.
- `docs/content/docs/configuration.mdx`: `ORT_DYLIB_PATH` row updated
from Windows-only wording to the cross-platform behavior.
## Testing
- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ cargo fmt --all -- --check && cargo clippy --workspace -- -D warnings && cargo test -p headroom-core --lib
clean
test result: 844 passed; 0 failed; 1 ignored
$ python -m pytest tests/test_transforms/test_ort_dylib.py -q
8 passed
$ ruff check headroom/_ort.py tests/test_transforms/test_ort_dylib.py
All checks passed!
```
## Real Behavior Proof
- Environment: Windows 11 (AVX2-capable — the SIGILL itself is not
reproducible on this machine), Python 3.13, Rust 1.95.0, local checkout
branched from `upstream/main` (
|
||
|
|
fce93bf39a |
fix(ci/deps): clear audit and release smoke failures (#2190)
## Summary - add a uv constraint floor for `setuptools>=83.0.0` to address `PYSEC-2026-3447` - refresh `uv.lock` so the production audit export resolves with `setuptools 83.0.0` - harden the Release wheel smoke-import gate by retrying Ubuntu-container `apt-get` operations and using `--fix-missing` - keep the generated `requirements-prod.txt` uncommitted; it is produced by the security workflow ## Why This clears the new dependency audit alert that made PRs red: - `setuptools 80.10.2` - `PYSEC-2026-3447` - fixed in `83.0.0` While validating the queue, the same PR class also hit a Release smoke-import failure in the Ubuntu 22.04 ARM container due apt mirror skew: `E: Failed to fetch ... python3-httplib2_0.20.2-2ubuntu0.1_all.deb 404 Not Found` The smoke gate should still fail for broken wheels, but transient apt mirror skew should not make unrelated PRs red. ## Lockfile impact - `setuptools 80.10.2 -> 83.0.0` - `torch 2.12.1 -> 2.13.0`, required for pip resolver compatibility with `setuptools 83.0.0` in the exported audit set - `cuda-toolkit 13.0.2 -> 13.0.3.0`, pulled by the torch lock refresh - uv also refreshed the existing project metadata for the sandbox extra so `uv lock --check` passes ## Validation - `uv lock --check` - `uv export --frozen --no-dev --no-emit-project --no-hashes --extra all --format requirements-txt > requirements-prod.txt` - confirmed generated `requirements-prod.txt` contains `setuptools==83.0.0`, `torch==2.13.0`, `cuda-toolkit==13.0.3.0` - `uvx pip-audit -r requirements-prod.txt` -> No known vulnerabilities found - `python -m pytest tests/test_release_workflows.py -q` -> 32 passed - `uvx ruff@0.15.17 check tests/test_release_workflows.py` -> All checks passed - `git diff --check` |
||
|
|
32ce99e4b4 |
fix(build): enable Intel macOS pip installs via ort-load-dynamic (#1538)
## Description
Fix `pip install headroom-ai` on Intel Mac (`x86_64-apple-darwin`).
Source installs failed because `ort-sys 2.0.0-rc.12` (transitive via
`fastembed`) does not ship prebuilt ONNX Runtime binaries for that
target, causing maturin/cargo to exit during the wheel build.
This PR mirrors the existing Windows fix: build the Rust core with
`ort-load-dynamic`, pin `ORT_DYLIB_PATH` to the pip `onnxruntime` native
library at import time, and publish Intel macOS wheels from CI.
Closes #
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
- `crates/headroom-core/Cargo.toml`: use `ort-load-dynamic` for
`x86_64-apple-darwin` instead of `ort-download-binaries-rustls-tls`.
- `headroom/_ort.py`: extend the ORT dylib pin hook to Intel macOS
(`darwin` + `x86_64`), resolving `libonnxruntime*.dylib` from the pip
`onnxruntime` package.
- `.github/workflows/release.yml` and `.github/workflows/rust.yml`: add
`macos-15-intel` / `x86_64-apple-darwin` wheel matrix entries.
- `tests/test_release_workflows.py` and
`tests/test_transforms/test_ort_dylib.py`: update/add coverage for the
new target and dylib pin behavior.
- `README.md`: note that prebuilt wheels are published for Intel macOS.
## Testing
- [x] Unit tests pass (`pytest`)
- [ ] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed
### Test Output
```text
$ pytest tests/test_transforms/test_ort_dylib.py \
tests/test_release_workflows.py::test_fastembed_uses_dynamic_ort_on_windows \
tests/test_release_workflows.py::test_build_wheels_matrix_includes_intel_macos_with_dynamic_ort -q
.......................... [100%]
10 passed in 0.18s
$ maturin build --release -o /tmp/headroom-dist
📦 Built wheel for abi3 Python ≥ 3.10 to /tmp/headroom-dist/headroom_ai-0.27.0-cp310-abi3-macosx_10_12_x86_64.whl
$ python3.11 -m venv /tmp/hr-venv && /tmp/hr-venv/bin/pip install .
Successfully built headroom-ai
Successfully installed headroom-ai-0.27.0
$ cd /tmp && /tmp/hr-venv/bin/python -c "import headroom; import headroom._core; print('ok')"
version 0.27.0
_core ok
```
## Real Behavior Proof
- Environment: macOS `x86_64-apple-darwin`, Python 3.11.5, Rust 1.95.0
- Exact command / steps: Reproduced the reported failure with `pip
install headroom-ai` (sdist build dies in `ort-sys` for
`x86_64-apple-darwin`); after this patch ran `maturin build --release`,
then `pip install .` in a clean venv, then `python -c "import
headroom._core"`.
- Observed result: Before fix, cargo/maturin exit 101 on missing ORT
prebuilts; after fix, wheel build succeeds and `headroom._core` imports
cleanly (`version 0.27.0`, `_core ok`).
- Not tested: `macos-15-intel` GitHub Actions wheel matrix row (will be
validated by CI after merge).
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md if applicable
## Additional Notes
- ML features (magika detection, fastembed embeddings) still require
`onnxruntime` at runtime on Intel Mac. Users should install
`headroom-ai[proxy]` or `pip install onnxruntime`; `_ort.py` auto-pins
`ORT_DYLIB_PATH` when that package is present.
- Apple Silicon (`aarch64-apple-darwin`) behavior is unchanged: it
continues to bundle ORT via `ort-download-binaries-rustls-tls`.
- Lint/mypy not re-run locally in this pass; targeted pytest +
maturin/pip install proof covers the changed surface.
---------
Co-authored-by: Bor <you@example.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
|
||
|
|
5771a8020e |
fix(deps): remediate dependency CVEs and publish SBOM (#1509)
## Description
Supply-chain hardening: takes the **shipped** dependency surface from
**26 known CVEs to 0**. `pip install headroom-ai[all]` now resolves with
no known vulnerabilities (verified with Anchore syft + grype). Also
publishes a checked-in SBOM package (`sbom/`) so any user — especially
pilots running their own security review — can verify what's inside and
that we track it.
This addresses the Dependabot alerts on `main` (9 high / 4 moderate / 7
low at time of writing).
Closes #
## Type of Change
- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [x] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)
## Changes Made
**Rust**
- `pyo3` 0.24 → 0.29 (GHSA-36hh-v3qg-5jq4 High, GHSA-chgr-c6px-7xpp
Med). Migrated `Python::allow_threads` → `Python::detach` (10 sites) and
added `from_py_object` to the `Clone`-deriving `#[pyclass]` types (both
required by the 0.25+ API).
- `pyo3-log` 0.12 → 0.13; `lru` 0.12 → 0.18 (GHSA-rhfx-m35p-ff5j).
**Python**
- `torch` → 2.12.1, `mem0ai` → 2.x.
- Floor-pinned transitive CVE deps via `[tool.uv]
constraint-dependencies`: `pygments>=2.20.0`,
`pydantic-settings>=2.14.2`, `gitpython>=3.1.50`, `langsmith>=0.9.0`.
- **Removed `benchmark` from the `[all]` aggregate** so the default
install is CVE-free. `lm-eval` is invoked as an external subprocess
(`python -m lm_eval`) and never imported, so it is not a true runtime
dep — it remains available via the opt-in `[benchmark]` extra. See
[Accepted Risks](#additional-notes).
**npm (build/test tooling — never shipped in the
wheel/container/published SDK)**
- `esbuild` override `>=0.28.1` in `sdk/typescript` + `plugins/openclaw`
(GHSA-g7r4-m6w7-qqqr).
- `docs/`: `@anthropic-ai/sdk` → `^0.106.0` (GHSA-p7fg-763f-g4gf),
`postcss` override to force Next.js's bundled copy ≥8.5.10
(GHSA-qx2v-qp2m-jg93); regenerated a stale `bun.lock` that carried a
**Critical** vitest/vite.
**CI**
- Pinned `pypa/gh-action-pypi-publish` `@release/v1` → `@v1.13.0`
(GHSA-vxmw-7h4f-hqxh) in `release.yml` + `publish.yml`.
**SBOM**
- New `sbom/` directory: CycloneDX 1.7 + SPDX 2.3 SBOMs, grype scan
evidence, 330-package license inventory, and a regeneration guide.
## Testing
- [ ] Unit tests pass (`pytest`) — N/A, no Python source changed
(deps/config only)
- [x] Linting passes — `cargo fmt --check` + `cargo clippy` clean on the
changed crate; 0 `.py` files changed so `ruff`/`mypy` scope is
unaffected
- [x] Type checking passes — `cargo check --workspace` (0 errors)
- [ ] New tests added — N/A (dependency bumps; covered by existing
suites)
- [x] Manual testing performed — see Real Behavior Proof
### Test Output
```text
# headroom-ai[all] product surface — the number that matters
$ grype sbom:sbom/headroom-sbom-all-extra.cdx.json
No vulnerabilities found
# full repo scan (universal lock incl. opt-in [benchmark] + dev)
$ grype sbom:sbom/headroom-sbom.cdx.json
NAME INSTALLED TYPE VULNERABILITY SEVERITY
sqlitedict 2.1.0 python GHSA-g4r7-86gm-pgqc High # [benchmark]-only, unpatchable, accepted
nltk 3.9.4 python GHSA-p4gq-832x-fm9v High # [benchmark]-only, unpatchable, accepted
# pyo3 0.29 migration — extension builds + imports + runs
$ cargo check --workspace
Finished `dev` profile [unoptimized + debuginfo] target(s)
$ maturin develop && python -c "from headroom._core import DiffCompressor, SmartCrusher; ..."
extension OK — detach + from_py_object paths exercised
# lru 0.18 — eviction path
$ cargo test -p headroom-proxy --lib drift
14 passed, 213 filtered out
# per-ecosystem npm audits
$ (cd sdk/typescript && npm audit) -> found 0 vulnerabilities
$ (cd plugins/openclaw && npm audit) -> found 0 vulnerabilities
$ (cd docs && npm audit && bun audit) -> found 0 vulnerabilities / No vulnerabilities found
```
## Real Behavior Proof
- Environment: macOS (darwin 25.4.0, arm64), Python 3.12 `.venv`, Rust
1.95 toolchain, syft 1.46.0, grype 0.115.0, bun 1.3.14, maturin 1.13.3.
- Exact command / steps: (1) `uv export --extra all --no-dev
--no-emit-project | syft → grype` for the product surface; (2) `cargo
check --workspace` + `maturin develop` + extension import/compress smoke
test; (3) `cargo test -p headroom-proxy --lib drift`; (4) `cargo fmt
--check` + `cargo clippy -p headroom-py`; (5) `npm audit` in
sdk/openclaw/docs + `bun audit` in docs.
- Observed result: `headroom-ai[all]` resolution scans clean — "No
vulnerabilities found" (179 pkgs); full/prod SBOM shows only the 2
documented accepted CVEs; pyo3 0.29 extension imports and runs (detach +
from_py_object paths exercised); drift tests 14/14 pass; cargo fmt +
clippy clean; all npm/bun audits report 0.
- Not tested: full `pytest` suite (no Python source changed);
release-profile wheel build (used dev-profile `maturin develop` for the
import proof — the extension is semantically identical).
## Review Readiness
- [x] I have performed a self-review
- [x] This PR is ready for human review
## Checklist
- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have made corresponding changes to the documentation
(`sbom/README.md`)
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective — N/A
(dependency bumps; existing suites + scans cover it)
- [x] New and existing unit tests pass locally with my changes
- [ ] I have updated the CHANGELOG.md — N/A (Release Please
auto-generates from the conventional commit)
## Additional Notes
**Accepted risks (the 2 residual CVEs).** Both originate solely from the
EleutherAI `lm-evaluation-harness` under the **opt-in `[benchmark]`
extra**, which Headroom invokes as a subprocess (never imports):
- `sqlitedict` CVE-2024-35515 (High) — pickle deserialization; package
abandoned (last release 2021), **no upstream fix exists**.
- `nltk` CVE-2026-54293 (High) — path traversal in `nltk.data.load()`;
affects ≤3.9.4 (current latest), **no patched release**.
Neither is in `[all]`, the published wheel, or the container. They are
documented in `sbom/README.md` and will be picked up automatically once
upstream ships fixes.
**Release/CHANGELOG:** N/A items above are because this is a
dependency/security PR with no Python source changes; CHANGELOG is
Release-Please-managed via the conventional commit message.
|
||
|
|
6d3f39f213 |
feat: add dashboard agent usage stats (#814)
## Description Add a clear dashboard view for per-agent token usage so end users can see Cursor, Claude, Codex, and other detected clients with before/after token counts, tokens saved, and savings percentages. The stats API now exposes a stable `agent_usage` object that the dashboard renders near the top of the session view. Fixes # ## Type of Change - [ ] Bug fix (non-breaking change that fixes an issue) - [x] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made ### New Files **Tests:** - `tests/test_dashboard_agent_usage.py` — Covers agent classification, exact per-request aggregation, and aggregate fallback behavior. ### Modified Files - `headroom/proxy/server.py` — Adds per-agent usage aggregation to `/stats` with before tokens, after tokens, output tokens, saved tokens, savings percentage, source, providers, and models. - `headroom/dashboard/templates/dashboard.html` — Adds a prominent Agent Usage panel with totals, coverage status, per-agent token-flow bars, request counts, before/after tokens, saved tokens, and share of savings. ## Testing - [x] Unit tests pass: `.venv312/bin/pytest tests/test_dashboard_agent_usage.py` - [x] Linting passes: `.venv312/bin/ruff check headroom/proxy/server.py tests/test_dashboard_agent_usage.py` - [x] Diff whitespace check passes: `git diff --check origin/main...HEAD` - [x] Dashboard smoke render: local proxy on `127.0.0.1:8790`, captured Chrome headless screenshot of `/dashboard` - [x] New tests added for new functionality ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing relevant unit tests pass locally with my changes - [ ] I have made corresponding changes to the documentation - [ ] I have updated the CHANGELOG.md if applicable ## Additional Notes The agent usage panel uses exact request-log data when available. If detailed request logs are empty, it falls back to aggregate provider/model request counts and labels the coverage as aggregate fallback so users are not misled. |
||
|
|
b723874d12 |
ci: limit commitlint to pull requests (#843)
## Summary - limit the CI commitlint job to pull request events - prevent squash-merge commit subjects on `main` from failing post-merge CI - keep commitlint as a pre-merge PR gate ## Context - fixes the main-branch CI failure from https://github.com/chopratejas/headroom/actions/runs/27320913096/job/80711562040 ## Validation - `diff --check` - `actionlint .github/workflows/ci.yml .github/workflows/release.yml .github/workflows/release-please.yml .github/workflows/docker.yml` - `act workflow_dispatch -W .github/workflows/release.yml -e .github/act/dry-run.json -n` - `act release -W .github/workflows/release.yml -e .github/act/release-published.json -n` - `act push -W .github/workflows/release-please.yml -e .github/act/push-feat.json -n` - `act workflow_dispatch -W .github/workflows/docker.yml -e .github/act/docker-version.json -n` |
||
|
|
53d2342291 |
ci: speed up GitHub Actions — path filters, caching, timeouts, version upgrades (#620)
* ci: speed up GitHub Actions - path filters, caching, timeouts, version upgrades
Performance improvements:
- init-e2e.yml, wrap-e2e.yml: add path filters so e2e Docker builds only run when
e2e-related files change (saves ~10 min per irrelevant PR push)
- init-e2e.yml, wrap-e2e.yml: add concurrency groups to cancel superseded PR runs
- ci.yml: add pip caching to lint and build jobs
- ci.yml: cache actionlint + act binaries in workflow-validation (skip curl on hits)
- eval.yml: add pip caching to smoke-test and weekly-suite jobs
- docs.yml: add pip caching for mkdocs-material install
- rust.yml: replace cargo install --locked cargo-audit/deny with taiki-e/install-action
(prebuilt binaries; saves 2-5 min per audit run)
Bug fixes:
- docker.yml: fix actions/checkout@v6 -> @v4 (v6 does not exist; would break all
Docker builds on every release/PR touching docker paths)
Version upgrades:
- wagoid/commitlint-github-action: @v5 -> @v6
- devcontainers.yml: docker/setup-buildx-action@v3 -> @v4 (align with docker.yml)
Safety improvements:
- ci.yml: add timeout-minutes to all 13 jobs (changes, lint, build-wheel,
prefetch-model, test x4, test-extras, test-agno, commitlint, build,
workflow-validation, docker-native-e2e, windows-native-wrapper, macos-native-wrapper)
- docker.yml: add timeout-minutes to docker-build (75m), docker-manifest (20m),
promote-latest (10m)
- eval.yml: add timeout-minutes to smoke-test (30m); bump weekly-suite 60->90m
- rust.yml: add timeout-minutes to test (30m), wheels (45m), audit (20m)
Observed wall-clock impact on recent PRs:
- Init E2E and Wrap E2E were running on every single PR push regardless of content
- CI workflow was taking 12-17 min; path filters reduce unnecessary e2e runs to 0
* fix(ci): bust actionlint+act cache when workflow file changes
Static cache key 'ci-tools-actionlint-act-v1' never invalidated on
tool version updates. Switched to hashFiles('.github/workflows/ci.yml')
so the cache busts automatically whenever the download scripts are
updated to point at a newer release.
Flagged by adversarial review (Architecture + Testing/Reliability personas).
* fix(ci): add missing Dockerfile COPY paths to e2e path filters
e2e/init/Dockerfile and e2e/wrap/Dockerfile COPY files not covered
by the initial path filter set:
init-e2e: Cargo.toml, Cargo.lock, rust-toolchain.toml, uv.lock,
.claude-plugin, .github/plugin/**, plugins/headroom-agent-hooks/**
wrap-e2e: Cargo.toml, Cargo.lock, rust-toolchain.toml, uv.lock,
sdk/typescript/**, plugins/openclaw/**
Without these, a Rust toolchain bump or SDK change on a PR would
skip the e2e gate entirely, only catching it on the merge to main.
Flagged by adversarial review (Domain/Correctness persona).
* fix(devcontainer): upgrade uv to >=0.7.0 to parse uv.lock revision=3
* fix(devcontainer): set UV_SKIP_WHEEL_FILENAME_CHECK=1 in post-create.sh for gitpython wheel
* ci: bump actions/checkout and actions/setup-node to v5 (Node.js 20 EOL Jun 16)
* fix(devcontainer): export UV_SKIP_WHEEL_FILENAME_CHECK so uv run also skips wheel check
* ci: bump all GitHub Actions to latest versions (Node.js 24)
* fix(test): accept release-please-action v4 or v5 in workflow assertion
* fix(format): ruff format test_release_workflows.py
|
||
|
|
0f3e3af6b2 |
fix(release): tag format vX.Y.Z (drop release-please component prefix)
release-please's default for manifest configs is `include-component-in-tag: true`, which prepends the package name to every tag: `headroom-ai-v0.22.4`. The repo's existing tags are plain `vX.Y.Z` (v0.22.0, v0.22.1, v0.22.2, v0.22.3 — all created by the prior release_version.py + create-release flow). Without this override, the first release-please run on main opened PR #496 trying to ship `headroom-ai-v0.23.0` and computed its changelog from genesis because it couldn't find any tag matching its expected prefix. Set `include-component-in-tag: false` so the bot uses `vX.Y.Z` and finds the existing baseline (v0.22.3) cleanly. Add a regression test pinning this setting. |
||
|
|
74843d1d62 |
fix(tests): tomllib fallback to tomli on python 3.10
`tomllib` is a Python 3.11+ stdlib module. The project supports 3.10+ (per pyproject.toml `requires-python = ">=3.10"`) and the test (3.10) matrix job correctly caught this: ModuleNotFoundError: No module named 'tomllib' at tests/test_release_workflows.py:1036 Apply the same try/except fallback pattern used in headroom/release_version.py: tomllib on 3.11+, tomli as the backport on 3.10. |
||
|
|
c8e347f1ac |
ci(release): adopt release-please for gated publishes
Replace "every push to main = release" with release-please's release-PR pattern: the bot watches main and maintains a single "chore: release vX.Y.Z" PR aggregating conventional commits; merging that PR creates the tag + GitHub Release, which fires the release:published event that release.yml now triggers on. Why --- Per-merge releases burned PyPI's 10 GiB per-project storage quota (one fresh wheel matrix ~= 200 MB per merged fix/feat PR). publish-pypi has failed on every main merge since PR #482 with "400 Project size too large". Consolidating many fixes into one release cuts upload frequency ~5x. What changed ------------ - .github/workflows/release-please.yml: bot watching main - .release-please-config.json: python release-type + extra-files for sdk/typescript and plugins/openclaw package.json - .release-please-manifest.json: tracks current 0.9.1 - .github/workflows/release.yml: * trigger: push to main -> release: published * detect-version: reads tag from github.event.release.tag_name (strips leading "v") so release_version.py does not re-bump past the bot's tag * create-release: when release already exists (typical release-please path), do not pass --notes-file -- that would clobber the bot's auto-generated changelog body Tests ----- Five new regression tests in test_release_workflows.py prevent silent reversion to per-push triggering and assert the bot workflow + config invariants. Note ---- This commit does NOT fix the existing quota breach. Request a PyPI quota increase, yank old releases, or shrink the wheel matrix to free immediate space. This PR ensures the future release cadence stops growing the problem. |
||
|
|
c83687798b | Fix Windows ORT builds and Docker signing retries | ||
|
|
37c0df23b6 |
fix(ci): update sdist license-packaging invariant test to match new shape
The structural-invariant test in test_release_workflows.py pinned
literal strings from the previous single-LICENSE shape:
- 'include = [{ path = "LICENSE", format = "sdist" }]' (single line)
- 'name: Verify sdist includes top-level LICENSE'
- 'license_path = f"{root}/LICENSE"' (verifier line)
The previous commit broke all three intentionally — sdist include is
now multi-line and lists NOTICE alongside LICENSE; the workflow step
was renamed to "Verify sdist license-file metadata matches tarball
contents"; the verifier no longer hardcodes a single license_path,
it parses every License-File entry from PKG-INFO.
Update the test to:
- Pin substrings (not full lines) for both LICENSE and NOTICE
entries so future formatting changes won't break the test
- Pin the new step name
- Pin two distinctive verifier signatures (the License-File parse
line and the missing-files error message) so a refactor that
silently drops the cross-check fails loudly
- Add a docstring explaining the regression history so a future
"let me clean up this test" pass can't accidentally re-loosen
the invariant
|
||
|
|
9492386398 | fix: harden release gating and clarify pipx compatibility | ||
|
|
b9f84fa815 |
feat(ci): X2 — PR-time release dry-run via path-filtered pull_request trigger
The X1 smoke-import gate (PR #387) catches runtime symbol mismatches on the wheel before publish, but only at release time. Recent break patterns were upstream of that: - #379 (docker bake `name=` regression in PR #376) - #382 (sdist `os: ubuntu-latest` → `ubuntu-24.04` rename) - #384 / #385 / #386 (glibc shim alias / link-order iterations) - #387's own heredoc-indent regression that broke main on the FIRST release run after merge — the heredoc was inside `bash -ec '...'`, no PR-time check exercised it X2 adds a `pull_request:` trigger to release.yml with a NARROW path filter so the dry-run runs at PR time for changes that affect wheel layout / release pipeline, but skips for source-only PRs to `crates/headroom-core` / `crates/headroom-proxy`. Path filter: release.yml, docker.yml, crates/headroom-py/**, pyproject.toml, root Cargo.toml, Cargo.lock. publish-pypi / publish-npm / publish-github-packages / publish-docker / create-release all gate on `github.event_name != 'pull_request'`, so a PR run never publishes — the dry-run is build + collect-dist + smoke-import only. concurrency rules: - PR runs: namespaced by PR number (`pr-N`), cancel-in-progress=true. - main runs: namespaced by ref_name, cancel-in-progress=false (a tag-push release that's mid-flight must not be cancelled). Test pin covers all four invariants (trigger, path filter, publish gates, concurrency split). 21 tests in test_release_workflows.py, all green locally. |
||
|
|
596212b428 |
fix(ci): smoke-import wheels on customer-representative envs before publish (X1)
Issue #355 plus the three follow-on hotfixes (#384/#385/#386) all share a pattern: the wheel is technically valid (clippy passes, tests pass, auditwheel is happy, the static-symbol audit added in #384 is happy) but FAILS at runtime on a customer's box because of a dynamic-link symbol mismatch. None of our pre-publish gates actually `import headroom._core` on a representative customer environment. They only build it. What X1 adds ------------ A `smoke-import-wheels` job that runs after `build-wheels` and before `publish-pypi` / `publish-docker` / `create-release`. Matrix (6 jobs in parallel, ~3 min wall-clock): - `manylinux_2_28_x86_64` + Python 3.11 (the floor we promise) - `ubuntu:22.04` (glibc 2.35) + Python 3.12 (issue #355's env) - `ubuntu:20.04` (glibc 2.31) + Python 3.10 (older LTS) - `manylinux_2_28_aarch64` + Python 3.11 (aarch64 floor) - `ubuntu:22.04` arm64 + Python 3.12 (aarch64 customer env) - `macos-14` host + Python 3.13 (Apple Silicon) Each job downloads its arch's wheel artifact, installs the wheel matching its Python version inside the container, and runs the exact command the proxy's `_check_rust_core` runs at startup: from headroom._core import hello as _rust_hello If any matrix entry fails, `publish-pypi` / `publish-docker` / `create-release` are blocked. The matrix tells us exactly which customer environment combination breaks. Regression test in tests/test_release_workflows.py: `test_release_workflow_has_smoke_import_wheel_gate` pins the job's existence, the required matrix entries, and — critically — the gating wires (publish-pypi / publish-docker / create-release all need-and-require-success on the smoke job). A future "this slow CI step always passes anyway, drop it" refactor fails at PR time. Companion tests `test_glibc_compat_shim_present_in_headroom_py` and `test_release_workflow_audits_wheel_glibc_symbols` (added in #384) cover the static-symbol gate; this PR is the dynamic-link gate. Both are needed. |
||
|
|
e2146724af |
fix: ship glibc 2.38 compat shim + wheel symbol audit (closes #355)
Issue #355: published headroom_ai-*-manylinux_2_28_*.whl fails to import on Ubuntu 22.04, Debian 11/12, Conda envs with libc < 2.38: 'ImportError: undefined symbol: __isoc23_strtoll'. Root cause: ORT prebuilt artifacts (downloaded via fastembed's ort-download-binaries-rustls-tls feature) are compiled with gcc-14.2.1 on a glibc-2.38+ host and reference __isoc23_strtoll. Our manylinux build host has glibc 2.38 so the link succeeds; end users with older glibc don't. Two-part fix: (1) crates/headroom-py/glibc_compat.c provides weak-alias definitions for __isoc23_strtol/strtoll/strtoul/strtoull delegating to the older strtol* family, compiled by build.rs on Linux/glibc only. The dynamic linker prefers glibc's strong symbol when present (>= 2.38) and falls back to ours when not (< 2.38). (2) scripts/audit_wheel_glibc_symbols.py is a release.yml gate that runs objdump -T on every Linux wheel and rejects any UND symbol whose required glibc version exceeds the wheel's manylinux floor. Validated: the audit correctly rejects the actually-broken v0.20.26 wheel with a precise diagnostic. The shim itself is a tiny static link with zero runtime cost. Regression tests in tests/test_release_workflows.py pin the shim's load-bearing pieces (.c file, build.rs trigger, [build-dependencies] cc dep) and the audit invocation in release.yml. Future drift fails at PR time, not in the next release. This is the same bug class as PR #371 (rustls-everywhere). Each instance gets fixed; the audit gate now catches the *class* — any future static linkage that introduces a post-floor symbol gets blocked before publish. |
||
|
|
75576dae2b |
fix(ci): rebuild sdist on the renamed wheel-matrix host
PR #376 pinned the wheel matrix's `os:` from `ubuntu-latest` to `ubuntu-24.04` (explicit pinning, no semantic change). It silently disabled the sdist build, whose conditional was if: matrix.os == 'ubuntu-latest' && matrix.target == 'x86_64-unknown-linux-gnu' The literal `'ubuntu-latest'` no longer matched, sdist never built, `release-assets/*.tar.gz` was empty, and `create-release` failed at gh release upload release-assets/*.tar.gz --clobber with "no matches found". v0.20.22's release didn't get its sdist on the GitHub Release. Fix: key the conditional on `matrix.target` only. sdist is platform-independent so any single matrix row is fine; using `target` decouples the sdist build from any future host-runner rename. Regression test in tests/test_release_workflows.py: `test_sdist_build_conditional_keyed_on_target_not_os` pins the target-only form. A future "let's add `os` back for clarity" refactor will fail at PR time, not 8 minutes into a release. This is a follow-up to PR #379 (docker bake `name=`) — same root cause class: PR #376's matrix-shape changes silently broke a downstream conditional whose literal value was tied to the OLD matrix shape. Both fixes are now in place + pinned by tests. |
||
|
|
8f6bc5865c |
fix(ci): docker per-arch bake needs explicit image name in output
PR #376's per-arch fan-out correctly removed `bake-file-tags` from the docker-build step (tags belong on the multi-arch manifest, not on per-arch images). But that left bake without ANY reference for the push target — no tags AND no explicit `name=` in the output spec. Every release-time docker-build job failed with the misleading message: ERROR: tag is needed when pushing to registry Buildx's actual constraint is "no tags AND no `name=` in the output = no push target." Since push-by-digest discards tags anyway, the fix is to specify `name=<registry>/<image>` directly in the `*.output` spec. Tags are still applied later, only on the multi-arch manifest by `docker-manifest`. Regression test in tests/test_release_workflows.py: `test_docker_per_arch_build_specifies_image_name_in_output` pins the `name=` substring so a future "the labels block already has the registry, surely buildx can figure it out" refactor will fail at PR time rather than 2 minutes into release. |
||
|
|
ed36676c9c |
ci: native arm64 runners — drop QEMU, cut wheel + docker build time
GitHub-hosted Linux arm64 runners (`ubuntu-24.04-arm`) went GA in Aug
2025 and are free for public repositories. Switching the aarch64
wheel + the multi-arch docker matrix off `ubuntu-latest`+QEMU onto
the native runner cuts wall-clock on both surfaces.
release.yml — build-wheels matrix
* `aarch64-unknown-linux-gnu`: `ubuntu-latest` → `ubuntu-24.04-arm`.
maturin-action still runs inside `quay.io/pypa/manylinux_2_28_aarch64`,
but the container now executes natively on an aarch64 kernel
instead of through QEMU emulation. Aarch64 wheel build drops
from ~50–60 min to ~10 min.
* `x86_64-unknown-linux-gnu`: `ubuntu-latest` → `ubuntu-24.04`
(pin the moving alias for reproducibility; no semantic change).
docker.yml — fan-out + manifest merge
* Pre-#377: one `docker-variant-tags` matrix job per variant on
`ubuntu-latest`, using bake's `platforms = [amd64, arm64]` with
QEMU for the arm64 leg. ~1h per variant, 8 variants.
* Post-#377: split into `docker-build` (variant × arch = 16
parallel jobs, each on its native runner, single-platform
push-by-digest) and `docker-manifest` (per variant, merges the
two arch digests into a multi-arch tagged manifest with
`docker buildx imagetools create`, signs the index manifest
with cosign). Wall-clock drops from ~1h per variant to ~10 min.
* `docker/setup-qemu-action` removed — there's no QEMU left.
* Per-(variant, arch) GHA cache scopes so the two arches don't
collide on cache keys.
* `promote-latest` rewired to depend on `docker-manifest`.
Behavior change: cosign now signs only the multi-arch index digest
per variant, not each per-platform image. `cosign verify <repo>:tag`
(the typical flow) is unchanged because cosign resolves the tag to
the index digest. Verifiers pinning a specific per-arch digest will
need to verify the index digest instead.
Regression tests in tests/test_release_workflows.py:
* `test_aarch64_wheel_uses_native_arm64_runner` — pins the
aarch64 row to `ubuntu-24.04-arm` (and the amd64 row to
`ubuntu-24.04`, not `-latest`), so a future "let me unify on
ubuntu-latest" refactor surfaces the QEMU regression at PR time.
* `test_docker_workflow_builds_on_native_arch_runners` — pins the
fan-out matrix's arch entries, asserts push-by-digest, asserts
`setup-qemu-action` is absent from non-comment lines, asserts
the manifest-merge job exists.
Verified:
* Both workflow files parse as valid YAML with the expected job
graph (`docker-build` → `docker-manifest` → `promote-latest`,
16 fan-out jobs, 8 manifest jobs).
* `docker buildx imagetools inspect <tag> --format '{{ json . }}'`
exposes the index digest at `.manifest.digest` (confirmed via
Docker's official reference).
* `ubuntu-24.04-arm` is the correct GitHub-hosted runner label
(GA 2025-08-07, free for public repos).
* `make ci-precheck-rust` and `make ci-precheck-python` both pass
locally; `tests/test_release_workflows.py` is 15/15 green
(13 existing + 2 new).
|
||
|
|
6f2c0a8400 |
fix(ci): rustls-everywhere — eliminate openssl-sys from build tree
# Root cause of the wheel-build cascade We have shipped 5 release-pipeline hot-fixes in 12 hours, each addressing a different symptom of the same architectural problem: 1. PR #363 — npm artifact downloads + tried `yum openssl-devel` 2. PR #367 — vendored OpenSSL in `headroom-proxy` + dropped Intel mac 3. PR #369 — Debian-cross perl install (`perl` not `libipc-cmd-perl`) 4. PR #370 — moved `openssl/vendored` from headroom-proxy to headroom-py 5. (this PR) — ELIMINATE OpenSSL entirely Each fix exposed a different missing system package or feature flag in a different build surface (manylinux x86_64 vs aarch64-cross-Debian vs macOS Intel vs e2e/wrap Dockerfile vs e2e/init Dockerfile vs main Dockerfile vs devcontainer). We were playing whack-a-mole because every Cargo dep change to the OpenSSL surface required matching system-package updates in 6+ different Dockerfiles and workflows, and the PR-level CI didn't exercise all of them. # Why this PR is the structural fix `fastembed` exposes clean rustls feature flags: - `hf-hub-rustls-tls` (replaces default `hf-hub-native-tls`) - `ort-download-binaries-rustls-tls` (replaces default `…native-tls`) By disabling fastembed's default features and enabling the rustls variants explicitly, we remove `native-tls` (and therefore `openssl-sys`, `openssl`, `openssl-src`, perl modules, OpenSSL build-time deps, vendored OpenSSL ~30s build cost) from the entire workspace dep tree. Verified locally: $ cargo tree -p headroom-py -i openssl-sys error: package ID specification `openssl-sys` did not match any packages $ cargo tree -p headroom-py -i native-tls error: package ID specification `native-tls` did not match any packages $ cargo build --release -p headroom-py Finished `release` profile [optimized] target(s) in 25.57s (Down from 1m+ with vendored OpenSSL.) # Cleanups enabled by this change - crates/headroom-py/Cargo.toml — dropped the `openssl/vendored` workaround from PR #370. - crates/headroom-proxy/Cargo.toml — same dep removed. - e2e/wrap/Dockerfile — dropped `yum install openssl-devel pkgconfig perl-IPC-Cmd`. Comment retained explaining why. - e2e/init/Dockerfile — same. - Dockerfile (main) — dropped `pkg-config libssl-dev` from apt-get. - .devcontainer/Dockerfile — dropped `pkg-config libssl-dev`. - .github/workflows/release.yml — removed the entire before-script-linux block (perl install probe + multi-package-manager dispatch + fail-loud assertion). No longer needed. # Regression gate Three new structural tests in tests/test_release_workflows.py: - test_no_openssl_sys_in_wheel_build_tree — runs `cargo tree -p <crate> -i openssl-sys` for headroom-py / headroom-proxy / headroom-core. If openssl-sys reappears (a future native-tls enabler creeping in via a new dep), this fails AT PR TIME with an actionable message. - test_no_native_tls_in_wheel_build_tree — same shape, native-tls is the proximate cause. - test_fastembed_uses_rustls_features — checks the Cargo.toml so a future "let me bump fastembed and forget the features" doesn't silently re-introduce OpenSSL. Plus two cleanup gates: - test_dockerfiles_no_longer_install_openssl_devel - test_release_yml_does_not_install_openssl_or_perl_for_wheels All 13 release-workflow tests pass. `make ci-precheck` PASSED. # What this teaches us about rollouts (per user's ultrathink ask) The 5-fix cascade exposed three meta-problems: 1. PR checks don't block merges. PR #370 had docker-init-e2e, docker-wrap-e2e, docker-native-e2e all FAILED yet got merged. Branch protection should require these checks. Operator action needed (cannot fix in code). 2. Local validation is misleading. `cargo build -p headroom-py` from the workspace root used the workspace lockfile and looked green; CI did fresh resolution against headroom-py's manifest alone where the feature wasn't enabled. Lesson: verify structural invariants with `cargo tree -e features` before trusting that a build "works." 3. 6+ build surfaces with independent system-dep state. Every Cargo change required matching updates in 6 places. The structural answer (this PR) is to NOT depend on system OpenSSL at all. Where structural fixes are not possible, the answer is a single shared scripts/install-rust-build-deps.sh — but with this PR there's nothing left to install. |
||
|
|
a5c7f6fed9 |
fix(ci): vendored OpenSSL must live in headroom-py, not headroom-proxy
PR #367 added `openssl = { features = ["vendored"] }` to `crates/headroom-proxy/Cargo.toml`, expecting Cargo's feature unification to propagate the vendored feature throughout the workspace. PR #369 unblocked aarch64 by fixing the perl install. The next release run on `1323830f` (the #369 merge) showed aarch64 + macOS wheels build successfully but the **x86_64 Linux wheel still fails** with: The system library `openssl` required by crate `openssl-sys` was not found. # Why my previous reasoning was wrong Cargo's feature unification only applies to the resolution graph that's actually built. `maturin build --manifest-path crates/headroom-py/Cargo.toml` resolves headroom-py's deps only — headroom-py does NOT depend on headroom-proxy, so the `openssl/vendored` enable in headroom-proxy never propagates to the wheel build. `cargo tree -p headroom-py -e features` confirms: openssl-sys is built with `default` only, no `vendored`. (I missed this when I tested locally because `cargo build -p headroom-py` from the workspace root reads the workspace's Cargo.lock, which was already populated with `openssl-src` from when I added the dep to headroom-proxy. The CI job runs `cargo rustc` with a fresh resolution against headroom-py's manifest, where the vendored feature isn't enabled.) # Fix Move the `openssl = { version = "0.10", features = ["vendored"] }` dep from `headroom-proxy/Cargo.toml` to `headroom-py/Cargo.toml`. The wheel-producing crate now declares the feature directly. `cargo tree -p headroom-py -e features` after this change: openssl-sys feature "vendored" └── openssl-sys feature "openssl-src" └── openssl-src feature "default" └── openssl-src v300.6.0+3.6.2 Local `cargo build --release -p headroom-py` confirms openssl-src is now compiled (35s wheel build). # Tests `test_headroom_py_vendors_openssl` (renamed from `test_headroom_proxy_vendors_openssl`) gates the dep on the right crate. Docstring explains WHY it must live in headroom-py — so a future refactor doesn't move it back to headroom-proxy and silently break the wheel build. All 11 release-workflow tests pass. `make ci-precheck` PASSED. |
||
|
|
cf4ea02432 |
fix(ci): wheel build before-script-linux must work on Debian aarch64-cross
Previous wheel hot-fix (#367) introduced a NEW failure mode that the F1-merge release run surfaced: E: Unable to locate package libipc-cmd-perl The aarch64-unknown-linux-gnu maturin-action target does NOT use the AlmaLinux 8 manylinux_2_28 image — it uses a Debian/Ubuntu-based cross-compile container. The previous hot-fix's apt branch installed `libipc-cmd-perl` which is a deprecated alias and is no longer in the default Debian/Ubuntu sources. The build failed before openssl-src could even start. # What the script actually needs to do `IPC::Cmd` is a Perl core module since 5.10, so any working `perl` install provides it. The fix: 1. **Probe first.** `perl -MIPC::Cmd -e 1` exits cleanly when the module is already importable — skip the install entirely. Some manylinux images already ship it; others don't. 2. **Cover every package manager.** dnf (modern RHEL family) → yum (older RHEL) → apt-get (Debian/Ubuntu) → apk (Alpine/musllinux). The maturin-action uses different containers per (target, manylinux) combo and we don't get to pick. 3. **Use `perl` not `libipc-cmd-perl` on Debian.** The plain `perl` meta-package pulls `perl-modules-*` which contains IPC::Cmd. Works on every Debian/Ubuntu version we'll see; `libipc-cmd-perl` is gone from default sources. 4. **Fail loud after install.** `perl -MIPC::Cmd -e 'print "loaded OK"'` runs unconditionally at the end. If somehow the module is STILL missing, we fail here — not 5 minutes later in the openssl-src compile step where the error message is harder to debug. Matches the project's "no silent fallback" rule. # Tests `test_build_wheels_installs_perl_ipc_cmd_for_vendored_openssl` updated to gate the new shape: - Asserts `perl -MIPC::Cmd -e 1` pre-probe is present - Asserts dnf/yum/apt-get/apk branches all exist - Asserts apt branch installs `perl` not `libipc-cmd-perl` - Asserts `libipc-cmd-perl` does not appear on any non-comment line - Asserts the final `perl -MIPC::Cmd` fail-loud assertion is present All 11 release-workflow tests pass. `make ci-precheck` PASSED. |
||
|
|
1314842b19 |
fix(ci): vendor OpenSSL via cargo + drop x86_64 macOS from wheel matrix
The previous hot-fix (#363) addressed npm artifact downloads and added openssl-devel installs in the manylinux container, but the wheel build still fails on three of four matrix entries with three distinct errors: 1. ubuntu-x86_64 with `manylinux: auto` resolved to manylinux2014 (CentOS 7 / OpenSSL 1.0.2k). `openssl-sys 0.9` requires OpenSSL 1.1.0+ — "different version of OpenSSL was found". 2. ubuntu-aarch64 cross-compiles via `aarch64-unknown-linux-gnu-gcc` from an x86_64 manylinux container. The `yum install openssl-devel` we added installs x86_64 headers; `/usr/aarch64-unknown-linux-gnu/ include/` has no OpenSSL — "openssl/opensslv.h: No such file or directory". 3. macos-15-intel fails on `ort-sys` (transitive via the ML compression backend), which has no prebuilt ONNX Runtime binaries for `x86_64-apple-darwin`. Unrelated to OpenSSL; an upstream limitation. Why the workspace pulls openssl-sys at all: `hf-hub` (transitive via `fastembed`) hard-codes `native-tls` as a default feature. Cargo's feature unification then enables openssl-sys for the whole workspace despite our `reqwest`/`tokio-tungstenite`/`tokio-rustls` preferences. # Fix 1: vendored OpenSSL Add `openssl = { version = "0.10", features = ["vendored"] }` to `crates/headroom-proxy/Cargo.toml`. The `vendored` feature compiles OpenSSL from source as part of the cargo build — works on every target uniformly. Local build verified: cargo now pulls `openssl-src v300.6.0+3.6.2` and compiles it. ~30s extra one-time build cost. The `openssl/vendored` feature DEFEATS `OPENSSL_DIR`. We therefore remove the previous hot-fix's "Install OpenSSL (macOS)" step that exported `OPENSSL_DIR` — leaving it would silently regress to the system-OpenSSL path that broke originally. # Fix 2: pin manylinux floor to 2_28 Change x86_64-unknown-linux-gnu from `manylinux: auto` to `manylinux: 2_28` (matching aarch64 + the e2e Dockerfiles). This isn't strictly required with vendored OpenSSL — the floor is now glibc 2.28 / AlmaLinux 8 which has modern toolchain — but it removes the CentOS-7 surface entirely and matches our runtime container target. # Fix 3: drop x86_64-apple-darwin from the matrix `ort-sys 2.0.0-rc.12` has no prebuilt ONNX Runtime binaries for that target. Building ORT from source would add CMake + ~5 minutes per build. Apple Silicon macOS (`aarch64-apple-darwin`) is fully covered; Intel-mac users install from the platform-independent sdist this matrix also produces. Tracked as a follow-up: switch the ML backend to `ort-tract` or upstream a request for x86_64 macOS prebuilts. # before-script-linux: keep perl-IPC-Cmd, drop openssl-devel OpenSSL's vendored `Configure` script needs `IPC::Cmd` (without it the build fails with "Can't locate IPC/Cmd.pm"). System openssl-devel is no longer needed. # Tests 4 new regression tests gate this: - `test_headroom_proxy_vendors_openssl` - `test_build_wheels_installs_perl_ipc_cmd_for_vendored_openssl` - `test_build_wheels_does_not_set_openssl_dir` - `test_build_wheels_matrix_excludes_intel_macos` Plus the previous 7. All 11 release-workflow tests pass. `make ci-precheck` PASSED. Local `cargo build --release -p headroom-py` green. |
||
|
|
7ba47e257b |
fix(ci): unbreak release pipeline — wheel openssl + dead npm artifact downloads
Three independent failures on the post-merge release run for PR #360, all introduced by the single-wheel maturin refactor: 1. Linux wheel matrix (`ubuntu-x86_64`, `aarch64-unknown-linux-gnu`) failed inside the manylinux container with: Could not find openssl via pkg-config The system library `openssl` required by crate `openssl-sys` was not found. `openssl-sys` is transitive via `fastembed` → `hf-hub` → `ureq` → `native-tls`. The e2e Dockerfiles (`e2e/wrap`, `e2e/init`) we wrote for #360 install `openssl-devel` upfront, but the `build-wheels` matrix uses `PyO3/maturin-action@v1` which spins up its OWN manylinux container that does not inherit those installs. Fix: add a `before-script-linux:` to the action with a yum/apt-get conditional so it works on RHEL-family (manylinux2014, manylinux_2_28) and Debian-family musllinux variants. 2. macOS x86_64 wheel build failed with `maturin` exit 1 from the same `openssl-sys` lookup. The aarch64 macos-14 runner happens to have `/opt/homebrew/opt/openssl@3` on `openssl-sys`'s default discovery path; the Intel macos-15-intel runner uses `/usr/local/Cellar` which is NOT on that path. Fix: add a pre-maturin step that runs `brew install openssl@3` and exports `OPENSSL_DIR` / `OPENSSL_LIB_DIR` / `OPENSSL_INCLUDE_DIR` / `PKG_CONFIG_PATH`. The aarch64 runner happily picks up the explicit env vars too — no regression. 3. `publish-npm` and `publish-github-packages` both fail with "Artifact not found for name: dist". Both jobs `npm pack` + `npm publish` directly from the checked-out source tree — they never consume the Python `dist` artifact. The `Download dist artifact` step was vestigial dead code carried over from a prior workflow shape; the only reason it didn't fail before #360 is that the pre-refactor `build` job DID upload a `dist` artifact. Post-#360, `dist` is produced by `collect-dist` and neither publish job is gated on it (by design — npm vs PyPI ecosystems publish independently). Fix: remove the dead download step from both jobs. Loose coupling is preserved; `create-release` still gates the GitHub Release tag on all of build / build-wheels / collect-dist / publish-* succeeding. Why the PR-level CI didn't catch any of this: `release.yml` only runs on `push: branches: [main]`. PR #360's CI exercised `ci.yml` which has a separate `ci-build-wheels-on-pr` matrix that uses a different setup. The release surface only fires post-merge. Tests added (regression gates): - `test_build_wheels_installs_openssl_devel_on_linux_via_before_script` - `test_build_wheels_resolves_openssl_dir_explicitly_on_macos` - `test_npm_publish_jobs_do_not_download_dist_artifact` All 9 release-workflow tests pass. `make ci-precheck` PASSED. |
||
|
|
b31a34b4ac |
fix(ci): multi-stage manylinux build for e2e dockerfiles + release workflow test
## Two distinct failures on PR #360 ### docker-init-e2e + docker-wrap-e2e + docker-native-e2e Building headroom-ai from source inside `node:22-bookworm` produced a `_core.so` that referenced `__isoc23_strtoll` (a glibc 2.38+ symbol). The same image's runtime libc.so.6 (whatever it actually ships) can't resolve it at import time: ImportError: /workspace/headroom/_core.cpython-311-x86_64-linux-gnu.so: undefined symbol: __isoc23_strtoll Most likely cause: cc-rs invoking the bookworm gcc against headers that have C23 wrappers exposed (libc6-dev backport, gcc 13 default mode, or something similar), generating object code that references a symbol the runtime libc.so doesn't actually have. Fix: multi-stage docker build. Stage 1 builds the wheel inside `quay.io/pypa/manylinux_2_28_x86_64` (AlmaLinux 8, glibc 2.28 baseline). Stage 2 (node:22-bookworm) just installs the prebuilt wheel — no rust toolchain needed at runtime, no build inside the runtime image. Same pattern release.yml already uses for cross-platform wheel matrix. Removed `COPY headroom/` and `COPY pyproject.toml` from the runtime stage to prevent the source-only `headroom/` from shadowing the installed wheel via cwd (Python would import the .py-only package and miss `_core.so`). ### test (3.10/3.11/3.12/3.13) The release-workflows test asserts the literal `needs:` list of the create-release job. The single-wheel maturin refactor added `build-wheels` and `collect-dist` jobs between `build` and the publish jobs; create-release now waits for those too. Updated the assertion + added explicit checks for the new `needs.<job>.result == 'success'` guards. |
||
|
|
efd2ac1ca4 |
chore: renormalize line endings to LF
`.gitattributes` declares `*.py text eol=lf` and `*.sh text eol=lf`, but 74 files (73 .py, 1 .sh) are stored in the index with CRLF line endings, violating that contract. Every macOS/Linux clone reports these files as "modified" on fresh checkout because git's diff engine sees the stored bytes don't match the attribute contract, even though the working tree and index match byte-for-byte. Running `git add --renormalize .` rewrites each affected blob so the stored form matches the attribute declaration. No semantic changes — every affected file's diff is "N insertions, N deletions" with inserts and deletes being the same lines modulo line endings. Follow-up commit adds `.git-blame-ignore-revs` so `git blame` / GitHub blame skip this mechanical commit. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
|
796afd085b |
chore: apply ruff format to tests/test_release_workflows.py
Drive-by: main is currently failing `ruff format --check .` because of
two missing blank lines between two top-level functions in this file
(introduced in
|
||
|
|
e854ba1771 | Merge branch 'main' into fix/python-github-packages-publish |