mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 15:10:00 +08:00
## Description CodeQL had 43 open alerts on `headroom` (19 high, 24 medium). We need 0 for SOC 2 (risk R-10: developers introduce vulnerabilities). This PR fixes the 23 that are real problems in code we ship or run. The 16 in `tests/` are handled by #4047. 11 were dismissed in GitHub with written reasons: 7 stale alerts from the retired default-setup scan, and 4 false positives (#21, #105, #109, #135). **In plain words:** 1. **Error replies no longer include internal error text.** When a request body couldn't be read, the proxy sent back Python's raw error message. It now sends a fixed message that still says what kind of problem it was. 2. **A benchmark no longer logs a possible API key.** `--model_args` is shown as `<redacted>` in the log, but `lm_eval` still receives the real value. 3. **A test no longer prints part of the Gemini API key.** 4. **Five workflows declare read-only permissions**, and CodeQL scans GitHub Actions workflows again. 5. **The npm release scripts no longer pass arguments through Windows `cmd.exe`**, which reinterprets `&` and `|` inside arguments, file paths included. Closes CodeQL alerts #20, #23, #24, #25, #26, #27, #28, #29, #31, #33, #193 (stack-trace exposure), #177, #51, #145, #146, #147. The workflow-permission fixes correspond to the dismissed #2, #7, #67, #85, #86, #87, #125. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/proxy/helpers.py`: - New `invalid_request_body_message(exc)` returns a fixed string per error type and logs the type: - `…: malformed JSON` - `…: not valid UTF-8 (possibly compressed?)` - `…: must be a JSON object` - `…: too large` - plain `Invalid request body` for anything else - New `RequestBodyNotObject(ValueError)` replaces a bare `ValueError`, so every existing `except ValueError` still catches it. - `headroom/proxy/handlers/{anthropic,openai,gemini,batch}.py`: - All 10 `f"Invalid request body: {e!s}"` replies now use the helper. - The Google batch forwarder's 500 reply drops the exception text; it was already logged on the line above. - `headroom/evals/comprehensive_benchmark.py`: `--model_args` is appended after the command is logged, and the log line shows it as `<redacted>`. - `tests/test_google_multimodal_e2e.py`: stops printing the first 10 characters of the API key. - `.github/workflows/{devcontainers,eval,init-e2e,init-native-e2e,wrap-e2e}.yml`: top-level `permissions: contents: read`. None of these jobs write to the repo; caching and artifact upload don't use the repo token. - `.github/workflows/security.yml`: CodeQL matrix adds `actions`. - `scripts/{build,verify}_npm_release_assets.mjs`: - On Windows, npm runs as `node <npm-cli.js> …` instead of `cmd.exe /c npm.cmd …`. - `npm-cli.js` is found via `npm_execpath`, then next to node, then beside each `npm.cmd` on PATH. This handles node and npm in different directories, per Devin's review. - If none is found, it fails listing every path searched. - New tests: - `tests/test_invalid_request_body_message.py` - `tests/test_evals/test_comprehensive_benchmark_log_redaction.py`, per Devanshi's review. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`): one error remains, the same one `main` has, in an untouched file. See notes. - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ pytest -q tests/test_release_workflows.py tests/test_invalid_request_body_message.py \ tests/test_evals/test_comprehensive_benchmark_log_redaction.py ============================== 62 passed in 3.10s ============================== # The redaction test against the pre-fix comprehensive_benchmark.py (origin/main): ========================= 1 failed, 1 passed in 0.50s ========================== # With the fix: ============================== 2 passed in 0.57s =============================== $ pytest -q (proxy/handler/batch/eval/request-body files, run one file at a time) 187 files: 182 OK; the 5 below fail or hang identically on unchanged origin/main: tests/test_proxy_uds.py 11 failed (macOS 104-byte sun_path limit) tests/test_anthropic_buffered_ccr_usage.py 6 failed tests/test_proxy_copilot_auth_hooks.py 2 failed tests/gateway/test_parity_proxy_vs_gateway.py 2 failed tests/test_cli_proxy_improvements.py hangs >300s $ ruff check . -q -> clean $ ruff format --check headroom tests -> clean $ mypy headroom headroom/integrations/langchain/chat_model.py:394: error: Function "RunnableBinding" could always be true in boolean context [truthy-function] Found 1 error in 1 file (checked 566 source files) # identical on origin/main $ node --check scripts/build_npm_release_assets.mjs && node --check scripts/verify_npm_release_assets.mjs -> ok ``` CI: the `test (3)` failures (`tests/test_onnx_dependency_contract.py`, 5 tests) and `test (4)` failures (`tests/test_xai_cache_economics.py`, 5 tests) also fail on `main`'s latest CI run (run 37806990929), so they're unrelated to this PR. ## Real Behavior Proof - Environment: macOS 26 (arm64), Python 3.12.13, Node 24, npm 11.19.0. Worktree of `origin/main` at87e05b68plus this branch. - Exact command / steps: the three checks below. 1. **Error replies:** `invalid_request_body_message` was exercised with each exception type, including a `JSONDecodeError` whose document is `"secret-doc"`. Each case asserts the reply names the category and contains neither the exception text nor the document. 2. **Benchmark:** `run_lm_eval(model_args="model=gpt-4o-mini,api_key=sk-test-FAKE-…")` was called with the subprocess call mocked. The log shows `--model_args <redacted>` and not the fake key, while the mocked call received `model=gpt-4o-mini,api_key=sk-test-FAKE-…,base_url=http://localhost:8787/v1`. 3. **Windows npm lookup:** `findNpmCli()` was extracted from each script and run against a temporary layout reproducing Devin's case: node shim in `dev/bin`, `npm.cmd` plus `node_modules/npm/bin/npm-cli.js` in `Program Files/nodejs`, both on PATH. - Observed result: all three Windows npm lookup cases pass in both scripts (output below); checks 1 and 2 pass as shown in Test Output. ```text == scripts/build_npm_release_assets.mjs separate node/npm dirs -> found npm-cli.js beside npm.cmd on PATH npm_execpath set -> uses npm_execpath missing npm -> clear error: true == scripts/verify_npm_release_assets.mjs separate node/npm dirs -> found npm-cli.js beside npm.cmd on PATH npm_execpath set -> uses npm_execpath missing npm -> clear error: true ``` - Not tested: the two items below. - A real Windows run of the release scripts. No Windows machine was available, and CI runs these scripts on Ubuntu, where the code path is unchanged (`npm` spawned directly, no shell). - The new CodeQL `actions` scan. It runs on this PR's CI and on `main` after merge. ### Maintainer verification at `128f271f` (2026-10-08) - Environment: actual Windows 11, Python 3.12.13, Node 22.22.0, npm 10.9.4. Contributor logic unchanged; this head contains current `main`. - Commands: the focused `test_invalid_request_body_message.py`, `test_comprehensive_benchmark_log_redaction.py`, and `test_release_workflows.py` suites; the existing Windows npm lookup probe against both checked-in scripts; and a two-process-boundary loopback uvicorn/httpx malformed-body probe against this head. - Observed: **62 focused tests passed**. Both npm scripts invoked the real npm CLI under Node with `shell:false`, including Node copied into a separate `G:/tmp/headroom-review/resume49-node shim & literal` directory while npm remained under `C:/Program Files/nodejs`; `npm_execpath` was unset, so PATH lookup found the separate npm installation. npm reported 10.9.4 and a cache argument containing spaces and `&` was preserved literally. **36 actual TCP requests** (four malformed-body categories across nine endpoints) returned the expected fixed 400 messages, with no private marker disclosed and no upstream requests. Only outbound provider requests were replaced with a fail-if-called boundary; HTTP parsing and proxy handlers were real. - Hosted qualification at this exact head: **67 passed, 11 skipped**, including all three CodeQL languages. Evidence is retained in the maintainer review artifacts. The earlier simulated Windows-layout proof remains below as historical author evidence. ## Runtime Rollout Safety - Rollout-managed feature(s): none. - Minimum rollout channel: n/a. - Stable/default behavior changed: yes, the text of 400 replies for unreadable request bodies. It was `Invalid request body: <python exception text>` and is now a fixed category message. Status codes, error `type`/`code` fields and response shape are unchanged. Clients that parse the free-text message, rather than the status and type, would see different text. - Kill switch / disable path: none needed; this is message text only. - Unsafe override required: no. - Qualification impact: none. - Rollback path: revert this PR. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation (none needed) - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` ## Additional Notes - Review feedback addressed in921de19b: - Devin: Windows npm lookup when node and npm are in different directories. - Devanshi: benchmark redaction test. - **#21 (`security_block` reply in `anthropic.py`) is left as is.** It returns the security plugin's deliberate, user-facing reason for blocking a request. It was dismissed as a false positive with that reason. - The `mypy` error is pre-existing: the same `truthy-function` error appears on `origin/main`, in a file this PR doesn't touch. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
204 lines
6.6 KiB
JavaScript
204 lines
6.6 KiB
JavaScript
#!/usr/bin/env node
|
|
|
|
import { copyFileSync, existsSync, mkdtempSync, readFileSync, rmSync } from "node:fs";
|
|
import { tmpdir } from "node:os";
|
|
import path from "node:path";
|
|
import { spawnSync } from "node:child_process";
|
|
|
|
const [assetsDirArg, version] = process.argv.slice(2);
|
|
|
|
if (!assetsDirArg || !version) {
|
|
console.error("Usage: node scripts/verify_npm_release_assets.mjs <assets-dir> <version>");
|
|
process.exit(2);
|
|
}
|
|
|
|
const assetsDir = path.resolve(assetsDirArg);
|
|
|
|
const packages = [
|
|
{
|
|
name: "headroom-ai",
|
|
tarball: `headroom-ai-${version}.tgz`,
|
|
},
|
|
{
|
|
name: "headroom-openclaw",
|
|
tarball: `headroom-openclaw-${version}.tgz`,
|
|
dependencies: {
|
|
"headroom-ai": `^${version}`,
|
|
},
|
|
},
|
|
];
|
|
|
|
const tarballPaths = new Map();
|
|
|
|
function extractPackageJson(tarballPath) {
|
|
return extractJsonFromTarball(tarballPath, "package/package.json");
|
|
}
|
|
|
|
function extractDistPackageJson(tarballPath) {
|
|
return extractJsonFromTarball(tarballPath, "package/dist/package.json");
|
|
}
|
|
|
|
function extractJsonFromTarball(tarballPath, packageJsonPath) {
|
|
const workdir = mkdtempSync(path.join(tmpdir(), "headroom-npm-asset-"));
|
|
try {
|
|
const result = spawnSync("tar", ["-xzf", tarballPath, "-C", workdir], {
|
|
encoding: "utf8",
|
|
});
|
|
if (result.status !== 0) {
|
|
throw new Error(
|
|
`tar failed for ${tarballPath}: ${result.stderr || result.stdout || "unknown error"}`,
|
|
);
|
|
}
|
|
return JSON.parse(readFileSync(path.join(workdir, packageJsonPath), "utf8"));
|
|
} finally {
|
|
rmSync(workdir, { recursive: true, force: true });
|
|
}
|
|
}
|
|
|
|
function assertNoFileDependencies(pkg) {
|
|
for (const field of ["dependencies", "peerDependencies", "optionalDependencies"]) {
|
|
for (const [name, spec] of Object.entries(pkg[field] || {})) {
|
|
if (typeof spec === "string" && (spec.startsWith("file:") || spec.includes("release-assets"))) {
|
|
throw new Error(`${pkg.name} has non-portable ${field}.${name} spec: ${spec}`);
|
|
}
|
|
}
|
|
}
|
|
}
|
|
|
|
// npm on Windows is a .cmd shim, which only runs through cmd.exe, and cmd.exe
|
|
// reinterprets characters such as & and | inside arguments (paths included).
|
|
// Run npm's own CLI script under this Node binary instead, so no shell is
|
|
// involved on any platform.
|
|
function npmCommand(args) {
|
|
if (process.platform !== "win32") {
|
|
return ["npm", args];
|
|
}
|
|
return [process.execPath, [findNpmCli(), ...args]];
|
|
}
|
|
|
|
// Node and npm can live in different directories (a node shim, or npm
|
|
// upgraded into the global prefix), so look where npm itself says it is, then
|
|
// next to node, then next to every npm.cmd on PATH. Every npm.cmd install
|
|
// keeps its CLI at node_modules/npm/bin/npm-cli.js beside the shim.
|
|
function findNpmCli() {
|
|
const cliPath = path.join("node_modules", "npm", "bin", "npm-cli.js");
|
|
const candidates = [];
|
|
if (process.env.npm_execpath?.endsWith("npm-cli.js")) {
|
|
candidates.push(process.env.npm_execpath);
|
|
}
|
|
candidates.push(path.join(path.dirname(process.execPath), cliPath));
|
|
for (const dir of (process.env.PATH || "").split(path.delimiter)) {
|
|
if (dir && existsSync(path.join(dir, "npm.cmd"))) {
|
|
candidates.push(path.join(dir, cliPath));
|
|
}
|
|
}
|
|
const found = candidates.find((candidate) => existsSync(candidate));
|
|
if (!found) {
|
|
throw new Error(`npm CLI (npm-cli.js) not found. Looked in: ${candidates.join(", ")}`);
|
|
}
|
|
return found;
|
|
}
|
|
|
|
function runNpm(args, cwd) {
|
|
const [command, commandArgs] = npmCommand(args);
|
|
return spawnSync(command, commandArgs, {
|
|
cwd,
|
|
encoding: "utf8",
|
|
});
|
|
}
|
|
|
|
function assertOpenClawExtensionContract(cwd) {
|
|
const smoke = `
|
|
const mod = await import("headroom-openclaw");
|
|
if (typeof mod.default?.register !== "function") {
|
|
throw new Error("headroom-openclaw default export must expose register(api)");
|
|
}
|
|
if (typeof mod.registerHeadroomPlugin !== "function") {
|
|
throw new Error("headroom-openclaw must export registerHeadroomPlugin(api)");
|
|
}
|
|
if (mod.default.register !== mod.registerHeadroomPlugin) {
|
|
throw new Error("headroom-openclaw default.register must match registerHeadroomPlugin");
|
|
}
|
|
`;
|
|
const result = spawnSync(process.execPath, ["--input-type=module", "-e", smoke], {
|
|
cwd,
|
|
encoding: "utf8",
|
|
});
|
|
if (result.status !== 0) {
|
|
throw new Error(
|
|
`headroom-openclaw import smoke failed: ${
|
|
result.error?.message || result.stderr || result.stdout || "unknown error"
|
|
}`,
|
|
);
|
|
}
|
|
}
|
|
|
|
for (const expected of packages) {
|
|
const tarballPath = path.join(assetsDir, expected.tarball);
|
|
tarballPaths.set(expected.name, tarballPath);
|
|
const pkg = extractPackageJson(tarballPath);
|
|
|
|
if (pkg.name !== expected.name) {
|
|
throw new Error(`${expected.tarball} package name mismatch: expected ${expected.name}, got ${pkg.name}`);
|
|
}
|
|
if (pkg.version !== version) {
|
|
throw new Error(`${expected.tarball} version mismatch: expected ${version}, got ${pkg.version}`);
|
|
}
|
|
|
|
assertNoFileDependencies(pkg);
|
|
|
|
for (const [name, spec] of Object.entries(expected.dependencies || {})) {
|
|
const actual = pkg.dependencies?.[name];
|
|
if (actual !== spec) {
|
|
throw new Error(`${pkg.name} dependency ${name} mismatch: expected ${spec}, got ${actual}`);
|
|
}
|
|
}
|
|
|
|
if (expected.name === "headroom-openclaw") {
|
|
const distPkg = extractDistPackageJson(tarballPath);
|
|
if (distPkg.name !== expected.name) {
|
|
throw new Error(`${expected.tarball} dist package name mismatch: expected ${expected.name}, got ${distPkg.name}`);
|
|
}
|
|
if (distPkg.version !== version) {
|
|
throw new Error(`${expected.tarball} dist package version mismatch: expected ${version}, got ${distPkg.version}`);
|
|
}
|
|
assertNoFileDependencies(distPkg);
|
|
for (const [name, spec] of Object.entries(expected.dependencies || {})) {
|
|
const actual = distPkg.dependencies?.[name];
|
|
if (actual !== spec) {
|
|
throw new Error(`${distPkg.name} dist dependency ${name} mismatch: expected ${spec}, got ${actual}`);
|
|
}
|
|
}
|
|
}
|
|
}
|
|
|
|
const installDir = mkdtempSync(path.join(tmpdir(), "headroom-npm-install-"));
|
|
try {
|
|
for (const expected of packages) {
|
|
copyFileSync(tarballPaths.get(expected.name), path.join(installDir, expected.tarball));
|
|
}
|
|
const result = runNpm(
|
|
[
|
|
"install",
|
|
"--ignore-scripts",
|
|
"--no-audit",
|
|
"--no-fund",
|
|
`./${packages[0].tarball}`,
|
|
`./${packages[1].tarball}`,
|
|
],
|
|
installDir,
|
|
);
|
|
if (result.status !== 0) {
|
|
throw new Error(
|
|
`clean npm install failed: ${
|
|
result.error?.message || result.stderr || result.stdout || "unknown error"
|
|
}`,
|
|
);
|
|
}
|
|
assertOpenClawExtensionContract(installDir);
|
|
} finally {
|
|
rmSync(installDir, { recursive: true, force: true });
|
|
}
|
|
|
|
console.log(`Verified npm release assets for ${version}`);
|