mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-23 02:49:17 +08:00
## Description `e2e/wrap/tools/npm-audit-exceptions.json` excepted 8 advisories in npm 12.2.1's bundled copies and gave "no newer npm release is available" as part of the reason. npm 12.2.2 is published and bundles cleanness for 6 of those 8: undici 6.29.0 → 6.29.1 and ip-address 10.7.2 → 10.7.5. Moving the harness pin drops those exceptions; glob 13.0.6 and minimatch 10.2.5 are unchanged, so their two exceptions stay and the file's expiry moves off the 2026-11-15 cliff, where the security job would otherwise fail every PR. Closes #4774 ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `e2e/wrap/tools/package.json` and `e2e/wrap/Dockerfile`: pin npm 12.2.2 instead of 12.2.1, in both the dependency override and the image's global install. - `e2e/wrap/tools/package-lock.json`: regenerated by installing the lock with npm 12.2.2 itself. `npm install --package-lock-only` does **not** refresh the `inBundle` child entries, so it left undici at 6.29.0; a real `npm install --ignore-scripts` updated them. Only the bundled copies npm 12.2.2 changed moved — `@npmcli/promise-spawn` 10.0.1, `bin-links` 7.0.1, `ip-address` 10.7.5, `libnpmexec` 11.0.5, `pacote` 22.0.1, `semver` 7.8.6, `undici` 6.29.1 — with no unrelated transitive drift and the npm entry itself at 12.2.2 with the registry's `dist.integrity`. - `e2e/wrap/tools/npm-audit-exceptions.json`: removed the 5 undici and 1 ip-address entries, corrected the reason (they are fixed in what 12.2.2 bundles), and moved `expires` to 2026-12-15 so the two glob/minimatch exceptions left are not a four-day fuse. - `e2e/wrap/tools/README.md`: the pin references, the paragraph describing which bundled copies the audit still reports, and a separate procedure for bumping the `npm` override, since `npm install --package-lock-only` does not refresh npm's `inBundle` children. - `e2e/wrap/tools/dependency-regression.test.cjs` and `e2e/wrap/Dockerfile`: a regression test that reads npm's bundled undici and ip-address versions, fails if either is below its fixed floor (6.29.1, 10.7.5) or still has an entry in `npm-audit-exceptions.json`, and runs in the image too. ## Testing - [ ] Unit tests pass (`pytest`) - [ ] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text # npm 12.2.2 tarball, from registry.npmjs.org (sha512 matches dist.integrity) $ tar -xzf npm-12.2.2.tgz undici 6.29.1 ip-address 10.7.5 glob 13.0.6 minimatch 10.2.5 12.2.2 # every bundled copy the registry tarball ships, vs npm 12.2.1 changed: 7 @npmcli/promise-spawn: 10.0.0 -> 10.0.1 bin-links: 7.0.0 -> 7.0.1 ip-address: 10.7.2 -> 10.7.5 libnpmexec: 11.0.4 -> 11.0.5 pacote: 22.0.0 -> 22.0.1 semver: 7.8.5 -> 7.8.6 undici: 6.29.0 -> 6.29.1 # CI's audit step from .github/workflows/security.yml, run locally on the new lock $ cd e2e/wrap/tools && npm audit --package-lock-only --json > audit.json && python3 ci_audit_check.py audit.json Only excepted bundled advisories remain (2), until 2026-12-15. # same script, same lock, with the runner's npm 11 instead of 12.2.2 Only excepted bundled advisories remain (2), until 2026-12-15. # raw audit totals, before -> after {"moderate":33,"high":1,"total":34} # undici high + ip-address moderate {"moderate":33,"high":0,"total":33} # glob + minimatch only; the rest inherit from them # clean install of the new lock with npm 12.2.2, in a scratch prefix like /opt/wrap-tools $ npm ci --prefix <dir> --no-fund --no-audit --strict-allow-scripts added 363 packages in 28s $ npm test # this branch 1..4 # tests 4 # pass 4 # fail 0 # the same test file against base (npm 12.2.1 + its 8 exceptions), installed from the old lock $ node node_modules/npm/bin/npm-cli.js test not ok 1 - npm bundles the fixed undici and ip-address instead of excepting them error: 'npm bundles undici 6.29.0, below the fixed 6.29.1' # pass 3 # fail 1 # and against npm 12.2.2 while the old exceptions file is still in place not ok 1 - npm bundles the fixed undici and ip-address instead of excepting them # (fails on the leftover exception entries, as intended) ``` ## Real Behavior Proof - Environment: Windows 11, Node 26.10.0, npm 12.2.2 taken from the downloaded registry tarball; no Docker engine on this machine. - Exact command / steps: (1) unpack `npm-12.2.2.tgz` and read the bundled `package.json` versions, comparing all 132 entries against 12.2.1. (2) `npm install --ignore-scripts` into a copy of `e2e/wrap/tools` so the lock's `inBundle` entries regenerate. (3) `npm audit --package-lock-only` on that lock, then the Python exception check copied verbatim from `.github/workflows/security.yml`. (4) `npm ci --prefix <dir> --no-fund --no-audit --strict-allow-scripts`, then `npm test`, `openclaw --version`, `codex --version` in that prefix. (5) Cross-check every one of the 147 `node_modules/npm/node_modules/*` lock entries against the tree npm 12.2.2 actually installed. (6) Run the new test file against base's lock installed with npm 12.2.1, and against the same tree with base's exceptions file restored. - Observed result: the lock resolves npm 12.2.2 and its 7 new bundled copies. The CI check prints `Only excepted bundled advisories remain (2), until 2026-12-15.` with exit 0, for both the bundled npm 12.2.2 and the runner's npm 11. The high-severity undici finding is gone, and no `STALE EXCEPTION` line appears because the removed entries and their advisories are gone together. The prefix install is byte-consistent with the lock: 147/147 bundled entries match, 0 mismatches. `npm test` is 4/4 on this branch; the new check fails on base with `npm bundles undici 6.29.0, below the fixed 6.29.1`, and also fails on 12.2.2 if the old exception entries are put back. `openclaw --version` → `OpenClaw 2026.10.10 (7dc3350)`, `codex --version` → `codex-cli 0.166.0`. - Not tested: the Docker image build and the wrap E2E run itself — neither can run here without Docker. `docker-native-e2e` and Wrap E2E cover that in this PR, since their path filters include `e2e/**`. The installed CLIs were exercised only via `--version`, not a full Codex/OpenClaw session. ## Runtime Rollout Safety - Rollout-managed feature(s): none - Minimum rollout channel: n/a, dependency pins for the E2E harness only - Stable/default behavior changed: no. The wheel, SDK and plugins are untouched; this only changes the npm the E2E image installs with and the lock the harness installs from. - Kill switch / disable path: none needed - Unsafe override required: no - Qualification impact: none; no shipped artifact embeds npm or these bundled copies - Rollback path: revert this commit ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` - it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Screenshots (if applicable) N/A ## Additional Notes **Dependency justification** (CONTRIBUTING, "Dependencies & supply chain"): | Package | Why this version | Maintainer, activity | Install surface | |---|---|---|---| | `npm` 12.2.1 → 12.2.2 | Security patch in the bundled copies: 12.2.2 ships undici 6.29.1 and ip-address 10.7.5, which clear 5 undici advisories (GHSA-4qv8-2p9j-2685, GHSA-xjj6-f33q-929c, GHSA-2m5v-pch6-74hx, GHSA-qrxm-fqhp-jwvm, GHSA-9fm3-97px-rjf6) and GHSA-rxp7-6hxx-x8q5 that 12.2.1's undici 6.29.0 and ip-address 10.7.2 carried. It is also `latest` and the only release that closes them. | npm, Inc., maintained at [npm/cli](https://github.com/npm/cli) under the same release process as every other npm; monthly minors, patch releases within days of a bundled-dependency fix. | Same as the 12.2.1 pin it replaces: a global CLI install in the E2E image only. No new package, no new install script, no native code. Registry tarball carries `dist.integrity`, which the lock now pins. | The six bundled copies also move (`@npmcli/promise-spawn`, `bin-links`, `libnpmexec`, `pacote`, `semver` patch bumps — they are npm's own release, not a resolution I chose). None is reachable from the wheel or SDK; the whole tree lives in `/opt/wrap-tools` inside the E2E image. Why the expiry moves rather than disappearing: glob GHSA-3g3m-2fjr-6pw4 (fixed only in glob 14.0.2) and minimatch GHSA-g3hv-75wg-m249 (fixed only in 11.0.1) still apply, because npm 12.2.2 bundles glob 13.0.6 and minimatch 10.2.5 — both majors npm does not bundle. Deleting the file would fail the security job on those two; keeping 2026-11-15 would fail it in four days regardless of this change. 2026-12-15 is one month, matching the cadence the file had. N/A items: `pytest`, `ruff`, `mypy` — no Python changed. The new test is the one CONTRIBUTING asks for on a bug fix: it fails on base (`npm bundles undici 6.29.0, below the fixed 6.29.1`) and passes here, and it keeps holding the invariant for the next npm bump rather than only pinning this one. The remaining glob/minimatch exceptions are still verified by the security job's own check, which rejects stale entries. Answering the two review comments: both are addressed — the regression test is the new `dependency-regression.test.cjs` case, and the README now documents the npm override bump as its own procedure with the reason `--package-lock-only` is not enough.
148 lines
7.6 KiB
Docker
148 lines
7.6 KiB
Docker
# ─── Stage 1: build the headroom-ai wheel in manylinux ─────────────────────
|
|
# Building from source inside `node:22-bookworm` produced a `_core.so` that
|
|
# referenced `__isoc23_strtoll` (a glibc 2.38+ symbol). The runtime in the
|
|
# same image couldn't resolve it at import time because something in the
|
|
# build chain — gcc version, libc6-dev backport, or cc-rs compiling
|
|
# transitive C deps against newer headers — emitted C23-era symbols that
|
|
# the runtime libc.so.6 doesn't have. Building inside the manylinux_2_28
|
|
# container (AlmaLinux 8, glibc 2.28 baseline) guarantees the wheel works
|
|
# on any glibc 2.28+ runtime, including bookworm.
|
|
FROM quay.io/pypa/manylinux_2_28_x86_64 AS builder
|
|
|
|
# No OpenSSL system deps required. As of the rustls-everywhere refactor,
|
|
# all HTTP+TLS in our Rust crates goes through `rustls`. fastembed's
|
|
# `hf-hub-rustls-tls` + `ort-download-binaries-rustls-tls` features
|
|
# replace the default native-tls path; `cargo tree -p headroom-py -i
|
|
# openssl-sys` returns "not found", confirming this Dockerfile no
|
|
# longer needs `openssl-devel`, `pkgconfig`, or any perl modules
|
|
# (Time::Piece, IPC::Cmd) for the openssl-src vendored Configure
|
|
# script. The historical `yum install openssl-devel pkgconfig
|
|
# perl-IPC-Cmd` line is intentionally absent.
|
|
|
|
# Install rust toolchain into the manylinux container. Match the
|
|
# rust-toolchain.toml at repo root (1.95.0 + rustfmt + clippy) so cargo
|
|
# doesn't try to mutate the toolchain on first invocation.
|
|
ENV CARGO_HOME=/usr/local/cargo \
|
|
RUSTUP_HOME=/usr/local/rustup \
|
|
PATH=/usr/local/cargo/bin:/opt/python/cp311-cp311/bin:${PATH}
|
|
COPY scripts/install-rustup.sh /tmp/install-rustup.sh
|
|
RUN sh /tmp/install-rustup.sh -y --no-modify-path --profile minimal -c rustfmt -c clippy --default-toolchain 1.95.0
|
|
|
|
WORKDIR /build
|
|
|
|
COPY pyproject.toml uv.lock README.md README-pypi.md ./
|
|
COPY Cargo.toml Cargo.lock rust-toolchain.toml ./
|
|
COPY crates/ crates/
|
|
COPY headroom/ headroom/
|
|
|
|
# Build the wheel with maturin. PYO3_USE_ABI3_FORWARD_COMPATIBILITY allows
|
|
# building against Python 3.14+ until we bump pyo3 past 0.22.
|
|
ENV PYO3_USE_ABI3_FORWARD_COMPATIBILITY=1
|
|
# Build for Python 3.11 — aider-chat==0.86.2 requires Python <3.12, and
|
|
# trixie's default python3.13 is too new for aider. The manylinux image
|
|
# has python interpreters for every version at /opt/python/cpXY-cpXY/.
|
|
# Stage 2 uses `python:3.11-slim` (now trixie-based, glibc 2.41).
|
|
RUN /opt/python/cp311-cp311/bin/pip install 'maturin>=1.5,<2.0' && \
|
|
/opt/python/cp311-cp311/bin/maturin build --release --out /dist --interpreter python3.11
|
|
|
|
# ─── Stage 2: python+node runtime ───────────────────────────────────────────
|
|
# Base on `python:3.11-slim` (now trixie, glibc 2.41) instead of
|
|
# `node:22-bookworm` (glibc 2.36): the wheel's `_core.so` references
|
|
# three glibc 2.38+ C23 wrappers (`__isoc23_strtol{,l,ul}`) that one of
|
|
# our transitive C/C++ deps emits during cc-rs compilation even when
|
|
# built inside manylinux_2_28 (probably libstdc++'s `<cstdlib>` resolving
|
|
# `std::strtoll` to the C23 variant when the toolchain has newer
|
|
# headers). Bookworm's libc.so.6 doesn't export these wrappers, so
|
|
# `import headroom._core` fails at runtime. Trixie's glibc 2.41 does.
|
|
#
|
|
# Node 24.22.2 from the official multi-platform image; no NodeSource setup script.
|
|
FROM node:24-bookworm-slim@sha256:01d566e8d94d8c619591d78652babfccd3c6992a3dffa9ed5ab39e11cf6c70e9 AS wrap-node
|
|
|
|
# We need Python 3.11 here (aider-chat==0.86.2 requires Python <3.12).
|
|
# Trixie's default `python3` is 3.13 — too new for aider. python:3.11-slim
|
|
# gives us Python 3.11 + trixie glibc + apt access for installing Node.
|
|
FROM python:3.11-slim
|
|
|
|
ENV DEBIAN_FRONTEND=noninteractive \
|
|
AIDER_CHAT_VERSION=0.86.2 \
|
|
OPENCODE_VERSION=1.17.8 \
|
|
PATH="/opt/wrap-tools/node_modules/.bin:/opt/headroom-venv/bin:/opt/aider-venv/bin:${PATH}" \
|
|
PIP_DISABLE_PIP_VERSION_CHECK=1 \
|
|
PIP_NO_CACHE_DIR=1 \
|
|
PYTHONUNBUFFERED=1 \
|
|
PYTHONDONTWRITEBYTECODE=1
|
|
|
|
# OpenClaw 2026.10.10 requires Node >=24.16.0 <25 (or >=26.1.0). Install the
|
|
# Node 24 LTS line alongside Python 3.11. Also install git for packages
|
|
# that clone at install time.
|
|
RUN apt-get update && \
|
|
apt-get install -y --no-install-recommends \
|
|
ca-certificates \
|
|
curl \
|
|
git \
|
|
gnupg \
|
|
libstdc++6 \
|
|
libatomic1 && \
|
|
rm -rf /var/lib/apt/lists/*
|
|
|
|
COPY --from=wrap-node /usr/local/bin/node /usr/local/bin/node
|
|
COPY --from=wrap-node /usr/local/lib/node_modules/ /usr/local/lib/node_modules/
|
|
RUN ln -s ../lib/node_modules/npm/bin/npm-cli.js /usr/local/bin/npm && \
|
|
ln -s ../lib/node_modules/npm/bin/npx-cli.js /usr/local/bin/npx && \
|
|
ln -s ../lib/node_modules/corepack/dist/corepack.js /usr/local/bin/corepack
|
|
|
|
# npm 10.9.8 crashes in Arborist's peer-set resolver when installing the local
|
|
# OpenClaw plugin (Cannot read properties of null, reading 'edgesOut'). Pin the
|
|
# tested npm version so both wrap-e2e callers use the same working installer.
|
|
RUN npm install -g --no-fund --no-audit npm@12.2.2
|
|
|
|
# Freeze transitive CLI dependencies as well as the top-level versions.
|
|
# Global npm install re-resolves ranges and can select partially published
|
|
# dependency releases; npm ci consumes the reviewed lockfile instead.
|
|
COPY e2e/wrap/tools/package.json e2e/wrap/tools/package-lock.json e2e/wrap/tools/.npmrc /opt/wrap-tools/
|
|
RUN npm ci --prefix /opt/wrap-tools --no-fund --no-audit --strict-allow-scripts
|
|
|
|
# Exercise the installed dependency fixes in both existing wrap-E2E CI callers.
|
|
# The exceptions file is copied too: one check asserts it holds no advisory for a
|
|
# bundled copy that now carries the fix.
|
|
COPY e2e/wrap/tools/dependency-regression.test.cjs e2e/wrap/tools/npm-audit-exceptions.json /opt/wrap-tools/
|
|
RUN npm test --prefix /opt/wrap-tools
|
|
|
|
WORKDIR /workspace
|
|
|
|
# Bring in the prebuilt manylinux wheel from stage 1.
|
|
COPY --from=builder /dist/*.whl /tmp/wheels/
|
|
|
|
# `sdk/typescript` and `plugins/openclaw` are wired into the wrap-e2e
|
|
# harness (e2e/wrap/run.py invokes `npx openclaw`). DO NOT copy
|
|
# `headroom/` or `pyproject.toml` from the workspace — they would shadow
|
|
# the installed wheel's `headroom/` package via cwd and Python would
|
|
# import the source-only `headroom/` (which has no `_core.so`),
|
|
# triggering the same `ModuleNotFoundError` the wheel install fixes.
|
|
COPY sdk/typescript ./sdk/typescript
|
|
COPY plugins/openclaw ./plugins/openclaw
|
|
COPY scripts/install-opencode.sh /tmp/install-opencode.sh
|
|
|
|
# Install the prebuilt manylinux wheel with [proxy] extras. Pip resolves
|
|
# extras from the wheel's metadata and pulls deps from public PyPI. The
|
|
# wheel's `_core.so` was compiled against glibc 2.28, so it loads cleanly
|
|
# against bookworm's glibc 2.36 at runtime.
|
|
RUN python -m venv /opt/headroom-venv && \
|
|
/opt/headroom-venv/bin/python -m pip install --upgrade pip && \
|
|
/opt/headroom-venv/bin/python -m pip install "$(ls /tmp/wheels/headroom_ai-*.whl)[proxy]" && \
|
|
/opt/headroom-venv/bin/python -c "from headroom._core import DiffCompressor; print('headroom._core OK')" && \
|
|
python -m venv /opt/aider-venv && \
|
|
/opt/aider-venv/bin/python -m pip install --upgrade pip && \
|
|
/opt/aider-venv/bin/python -m pip install "aider-chat==${AIDER_CHAT_VERSION}" && \
|
|
sh /tmp/install-opencode.sh "${OPENCODE_VERSION}"
|
|
|
|
# Expose the installed binary to non-login shells (including CMD, docker run,
|
|
# and e2e shims spawned by subprocess). The e2e shim shadows
|
|
# opencode anyway, but this guarantees `which opencode` works for
|
|
# diagnostics.
|
|
ENV PATH="/root/.opencode/bin:${PATH}"
|
|
|
|
COPY e2e/wrap ./e2e/wrap
|
|
|
|
CMD ["python", "e2e/wrap/run.py"]
|