Files
headroom/scripts/verify_npm_release_assets.mjs
Tejas ChopraandClaude Opus 5.5 a88e2d5145 fix(security): resolve 23 open CodeQL alerts (#4045)
## Description

CodeQL had 43 open alerts on `headroom` (19 high, 24 medium). We need 0
for SOC 2 (risk R-10: developers introduce vulnerabilities). This PR
fixes the 23 that are real problems in code we ship or run. The 16 in
`tests/` are handled by #4047. 11 were dismissed in GitHub with written
reasons: 7 stale alerts from the retired default-setup scan, and 4 false
positives (#21, #105, #109, #135).

**In plain words:**
1. **Error replies no longer include internal error text.** When a
request body couldn't be read, the proxy sent back Python's raw error
message. It now sends a fixed message that still says what kind of
problem it was.
2. **A benchmark no longer logs a possible API key.** `--model_args` is
shown as `<redacted>` in the log, but `lm_eval` still receives the real
value.
3. **A test no longer prints part of the Gemini API key.**
4. **Five workflows declare read-only permissions**, and CodeQL scans
GitHub Actions workflows again.
5. **The npm release scripts no longer pass arguments through Windows
`cmd.exe`**, which reinterprets `&` and `|` inside arguments, file paths
included.

Closes CodeQL alerts #20, #23, #24, #25, #26, #27, #28, #29, #31, #33,
#193 (stack-trace exposure), #177, #51, #145, #146, #147. The
workflow-permission fixes correspond to the dismissed #2, #7, #67, #85,
#86, #87, #125.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/proxy/helpers.py`:
- New `invalid_request_body_message(exc)` returns a fixed string per
error type and logs the type:
    - `…: malformed JSON`
    - `…: not valid UTF-8 (possibly compressed?)`
    - `…: must be a JSON object`
    - `…: too large`
    - plain `Invalid request body` for anything else
- New `RequestBodyNotObject(ValueError)` replaces a bare `ValueError`,
so every existing `except ValueError` still catches it.
- `headroom/proxy/handlers/{anthropic,openai,gemini,batch}.py`:
  - All 10 `f"Invalid request body: {e!s}"` replies now use the helper.
- The Google batch forwarder's 500 reply drops the exception text; it
was already logged on the line above.
- `headroom/evals/comprehensive_benchmark.py`: `--model_args` is
appended after the command is logged, and the log line shows it as
`<redacted>`.
- `tests/test_google_multimodal_e2e.py`: stops printing the first 10
characters of the API key.
-
`.github/workflows/{devcontainers,eval,init-e2e,init-native-e2e,wrap-e2e}.yml`:
top-level `permissions: contents: read`. None of these jobs write to the
repo; caching and artifact upload don't use the repo token.
- `.github/workflows/security.yml`: CodeQL matrix adds `actions`.
- `scripts/{build,verify}_npm_release_assets.mjs`:
- On Windows, npm runs as `node <npm-cli.js> …` instead of `cmd.exe /c
npm.cmd …`.
- `npm-cli.js` is found via `npm_execpath`, then next to node, then
beside each `npm.cmd` on PATH. This handles node and npm in different
directories, per Devin's review.
  - If none is found, it fails listing every path searched.
- New tests:
  - `tests/test_invalid_request_body_message.py`
- `tests/test_evals/test_comprehensive_benchmark_log_redaction.py`, per
Devanshi's review.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`): one error remains, the
same one `main` has, in an untouched file. See notes.
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ pytest -q tests/test_release_workflows.py tests/test_invalid_request_body_message.py \
    tests/test_evals/test_comprehensive_benchmark_log_redaction.py
============================== 62 passed in 3.10s ==============================

# The redaction test against the pre-fix comprehensive_benchmark.py (origin/main):
========================= 1 failed, 1 passed in 0.50s ==========================
# With the fix:
============================== 2 passed in 0.57s ===============================

$ pytest -q (proxy/handler/batch/eval/request-body files, run one file at a time)
187 files: 182 OK; the 5 below fail or hang identically on unchanged origin/main:
  tests/test_proxy_uds.py                          11 failed  (macOS 104-byte sun_path limit)
  tests/test_anthropic_buffered_ccr_usage.py        6 failed
  tests/test_proxy_copilot_auth_hooks.py            2 failed
  tests/gateway/test_parity_proxy_vs_gateway.py     2 failed
  tests/test_cli_proxy_improvements.py             hangs >300s

$ ruff check . -q                      -> clean
$ ruff format --check headroom tests   -> clean
$ mypy headroom
headroom/integrations/langchain/chat_model.py:394: error: Function "RunnableBinding" could always be true in boolean context  [truthy-function]
Found 1 error in 1 file (checked 566 source files)   # identical on origin/main

$ node --check scripts/build_npm_release_assets.mjs && node --check scripts/verify_npm_release_assets.mjs  -> ok
```

CI: the `test (3)` failures (`tests/test_onnx_dependency_contract.py`, 5
tests) and `test (4)` failures (`tests/test_xai_cache_economics.py`, 5
tests) also fail on `main`'s latest CI run (run 37806990929), so they're
unrelated to this PR.

## Real Behavior Proof

- Environment: macOS 26 (arm64), Python 3.12.13, Node 24, npm 11.19.0.
Worktree of `origin/main` at 87e05b68 plus this branch.
- Exact command / steps: the three checks below.
1. **Error replies:** `invalid_request_body_message` was exercised with
each exception type, including a `JSONDecodeError` whose document is
`"secret-doc"`. Each case asserts the reply names the category and
contains neither the exception text nor the document.
2. **Benchmark:**
`run_lm_eval(model_args="model=gpt-4o-mini,api_key=sk-test-FAKE-…")` was
called with the subprocess call mocked. The log shows `--model_args
<redacted>` and not the fake key, while the mocked call received
`model=gpt-4o-mini,api_key=sk-test-FAKE-…,base_url=http://localhost:8787/v1`.
3. **Windows npm lookup:** `findNpmCli()` was extracted from each script
and run against a temporary layout reproducing Devin's case: node shim
in `dev/bin`, `npm.cmd` plus `node_modules/npm/bin/npm-cli.js` in
`Program Files/nodejs`, both on PATH.
- Observed result: all three Windows npm lookup cases pass in both
scripts (output below); checks 1 and 2 pass as shown in Test Output.
  ```text
  == scripts/build_npm_release_assets.mjs
  separate node/npm dirs -> found npm-cli.js beside npm.cmd on PATH
  npm_execpath set       -> uses npm_execpath
  missing npm            -> clear error: true
  == scripts/verify_npm_release_assets.mjs
  separate node/npm dirs -> found npm-cli.js beside npm.cmd on PATH
  npm_execpath set       -> uses npm_execpath
  missing npm            -> clear error: true
  ```
- Not tested: the two items below.
- A real Windows run of the release scripts. No Windows machine was
available, and CI runs these scripts on Ubuntu, where the code path is
unchanged (`npm` spawned directly, no shell).
- The new CodeQL `actions` scan. It runs on this PR's CI and on `main`
after merge.

### Maintainer verification at `128f271f` (2026-10-08)

- Environment: actual Windows 11, Python 3.12.13, Node 22.22.0, npm
10.9.4. Contributor logic unchanged; this head contains current `main`.
- Commands: the focused `test_invalid_request_body_message.py`,
`test_comprehensive_benchmark_log_redaction.py`, and
`test_release_workflows.py` suites; the existing Windows npm lookup
probe against both checked-in scripts; and a two-process-boundary
loopback uvicorn/httpx malformed-body probe against this head.
- Observed: **62 focused tests passed**. Both npm scripts invoked the
real npm CLI under Node with `shell:false`, including Node copied into a
separate `G:/tmp/headroom-review/resume49-node shim & literal` directory
while npm remained under `C:/Program Files/nodejs`; `npm_execpath` was
unset, so PATH lookup found the separate npm installation. npm reported
10.9.4 and a cache argument containing spaces and `&` was preserved
literally. **36 actual TCP requests** (four malformed-body categories
across nine endpoints) returned the expected fixed 400 messages, with no
private marker disclosed and no upstream requests. Only outbound
provider requests were replaced with a fail-if-called boundary; HTTP
parsing and proxy handlers were real.
- Hosted qualification at this exact head: **67 passed, 11 skipped**,
including all three CodeQL languages. Evidence is retained in the
maintainer review artifacts. The earlier simulated Windows-layout proof
remains below as historical author evidence.

## Runtime Rollout Safety

- Rollout-managed feature(s): none.
- Minimum rollout channel: n/a.
- Stable/default behavior changed: yes, the text of 400 replies for
unreadable request bodies. It was `Invalid request body: <python
exception text>` and is now a fixed category message. Status codes,
error `type`/`code` fields and response shape are unchanged. Clients
that parse the free-text message, rather than the status and type, would
see different text.
- Kill switch / disable path: none needed; this is message text only.
- Unsafe override required: no.
- Qualification impact: none.
- Rollback path: revert this PR.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation (none
needed)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`

## Additional Notes

- Review feedback addressed in 921de19b: 
- Devin: Windows npm lookup when node and npm are in different
directories.
  - Devanshi: benchmark redaction test.
- **#21 (`security_block` reply in `anthropic.py`) is left as is.** It
returns the security plugin's deliberate, user-facing reason for
blocking a request. It was dismissed as a false positive with that
reason.
- The `mypy` error is pre-existing: the same `truthy-function` error
appears on `origin/main`, in a file this PR doesn't touch.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 22:40:04 -05:00

204 lines
6.6 KiB
JavaScript

#!/usr/bin/env node
import { copyFileSync, existsSync, mkdtempSync, readFileSync, rmSync } from "node:fs";
import { tmpdir } from "node:os";
import path from "node:path";
import { spawnSync } from "node:child_process";
const [assetsDirArg, version] = process.argv.slice(2);
if (!assetsDirArg || !version) {
console.error("Usage: node scripts/verify_npm_release_assets.mjs <assets-dir> <version>");
process.exit(2);
}
const assetsDir = path.resolve(assetsDirArg);
const packages = [
{
name: "headroom-ai",
tarball: `headroom-ai-${version}.tgz`,
},
{
name: "headroom-openclaw",
tarball: `headroom-openclaw-${version}.tgz`,
dependencies: {
"headroom-ai": `^${version}`,
},
},
];
const tarballPaths = new Map();
function extractPackageJson(tarballPath) {
return extractJsonFromTarball(tarballPath, "package/package.json");
}
function extractDistPackageJson(tarballPath) {
return extractJsonFromTarball(tarballPath, "package/dist/package.json");
}
function extractJsonFromTarball(tarballPath, packageJsonPath) {
const workdir = mkdtempSync(path.join(tmpdir(), "headroom-npm-asset-"));
try {
const result = spawnSync("tar", ["-xzf", tarballPath, "-C", workdir], {
encoding: "utf8",
});
if (result.status !== 0) {
throw new Error(
`tar failed for ${tarballPath}: ${result.stderr || result.stdout || "unknown error"}`,
);
}
return JSON.parse(readFileSync(path.join(workdir, packageJsonPath), "utf8"));
} finally {
rmSync(workdir, { recursive: true, force: true });
}
}
function assertNoFileDependencies(pkg) {
for (const field of ["dependencies", "peerDependencies", "optionalDependencies"]) {
for (const [name, spec] of Object.entries(pkg[field] || {})) {
if (typeof spec === "string" && (spec.startsWith("file:") || spec.includes("release-assets"))) {
throw new Error(`${pkg.name} has non-portable ${field}.${name} spec: ${spec}`);
}
}
}
}
// npm on Windows is a .cmd shim, which only runs through cmd.exe, and cmd.exe
// reinterprets characters such as & and | inside arguments (paths included).
// Run npm's own CLI script under this Node binary instead, so no shell is
// involved on any platform.
function npmCommand(args) {
if (process.platform !== "win32") {
return ["npm", args];
}
return [process.execPath, [findNpmCli(), ...args]];
}
// Node and npm can live in different directories (a node shim, or npm
// upgraded into the global prefix), so look where npm itself says it is, then
// next to node, then next to every npm.cmd on PATH. Every npm.cmd install
// keeps its CLI at node_modules/npm/bin/npm-cli.js beside the shim.
function findNpmCli() {
const cliPath = path.join("node_modules", "npm", "bin", "npm-cli.js");
const candidates = [];
if (process.env.npm_execpath?.endsWith("npm-cli.js")) {
candidates.push(process.env.npm_execpath);
}
candidates.push(path.join(path.dirname(process.execPath), cliPath));
for (const dir of (process.env.PATH || "").split(path.delimiter)) {
if (dir && existsSync(path.join(dir, "npm.cmd"))) {
candidates.push(path.join(dir, cliPath));
}
}
const found = candidates.find((candidate) => existsSync(candidate));
if (!found) {
throw new Error(`npm CLI (npm-cli.js) not found. Looked in: ${candidates.join(", ")}`);
}
return found;
}
function runNpm(args, cwd) {
const [command, commandArgs] = npmCommand(args);
return spawnSync(command, commandArgs, {
cwd,
encoding: "utf8",
});
}
function assertOpenClawExtensionContract(cwd) {
const smoke = `
const mod = await import("headroom-openclaw");
if (typeof mod.default?.register !== "function") {
throw new Error("headroom-openclaw default export must expose register(api)");
}
if (typeof mod.registerHeadroomPlugin !== "function") {
throw new Error("headroom-openclaw must export registerHeadroomPlugin(api)");
}
if (mod.default.register !== mod.registerHeadroomPlugin) {
throw new Error("headroom-openclaw default.register must match registerHeadroomPlugin");
}
`;
const result = spawnSync(process.execPath, ["--input-type=module", "-e", smoke], {
cwd,
encoding: "utf8",
});
if (result.status !== 0) {
throw new Error(
`headroom-openclaw import smoke failed: ${
result.error?.message || result.stderr || result.stdout || "unknown error"
}`,
);
}
}
for (const expected of packages) {
const tarballPath = path.join(assetsDir, expected.tarball);
tarballPaths.set(expected.name, tarballPath);
const pkg = extractPackageJson(tarballPath);
if (pkg.name !== expected.name) {
throw new Error(`${expected.tarball} package name mismatch: expected ${expected.name}, got ${pkg.name}`);
}
if (pkg.version !== version) {
throw new Error(`${expected.tarball} version mismatch: expected ${version}, got ${pkg.version}`);
}
assertNoFileDependencies(pkg);
for (const [name, spec] of Object.entries(expected.dependencies || {})) {
const actual = pkg.dependencies?.[name];
if (actual !== spec) {
throw new Error(`${pkg.name} dependency ${name} mismatch: expected ${spec}, got ${actual}`);
}
}
if (expected.name === "headroom-openclaw") {
const distPkg = extractDistPackageJson(tarballPath);
if (distPkg.name !== expected.name) {
throw new Error(`${expected.tarball} dist package name mismatch: expected ${expected.name}, got ${distPkg.name}`);
}
if (distPkg.version !== version) {
throw new Error(`${expected.tarball} dist package version mismatch: expected ${version}, got ${distPkg.version}`);
}
assertNoFileDependencies(distPkg);
for (const [name, spec] of Object.entries(expected.dependencies || {})) {
const actual = distPkg.dependencies?.[name];
if (actual !== spec) {
throw new Error(`${distPkg.name} dist dependency ${name} mismatch: expected ${spec}, got ${actual}`);
}
}
}
}
const installDir = mkdtempSync(path.join(tmpdir(), "headroom-npm-install-"));
try {
for (const expected of packages) {
copyFileSync(tarballPaths.get(expected.name), path.join(installDir, expected.tarball));
}
const result = runNpm(
[
"install",
"--ignore-scripts",
"--no-audit",
"--no-fund",
`./${packages[0].tarball}`,
`./${packages[1].tarball}`,
],
installDir,
);
if (result.status !== 0) {
throw new Error(
`clean npm install failed: ${
result.error?.message || result.stderr || result.stdout || "unknown error"
}`,
);
}
assertOpenClawExtensionContract(installDir);
} finally {
rmSync(installDir, { recursive: true, force: true });
}
console.log(`Verified npm release assets for ${version}`);