mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
## Description CodeQL had 43 open alerts on `headroom` (19 high, 24 medium). We need 0 for SOC 2 (risk R-10: developers introduce vulnerabilities). This PR fixes the 23 that are real problems in code we ship or run. The 16 in `tests/` are handled by #4047. 11 were dismissed in GitHub with written reasons: 7 stale alerts from the retired default-setup scan, and 4 false positives (#21, #105, #109, #135). **In plain words:** 1. **Error replies no longer include internal error text.** When a request body couldn't be read, the proxy sent back Python's raw error message. It now sends a fixed message that still says what kind of problem it was. 2. **A benchmark no longer logs a possible API key.** `--model_args` is shown as `<redacted>` in the log, but `lm_eval` still receives the real value. 3. **A test no longer prints part of the Gemini API key.** 4. **Five workflows declare read-only permissions**, and CodeQL scans GitHub Actions workflows again. 5. **The npm release scripts no longer pass arguments through Windows `cmd.exe`**, which reinterprets `&` and `|` inside arguments, file paths included. Closes CodeQL alerts #20, #23, #24, #25, #26, #27, #28, #29, #31, #33, #193 (stack-trace exposure), #177, #51, #145, #146, #147. The workflow-permission fixes correspond to the dismissed #2, #7, #67, #85, #86, #87, #125. ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - `headroom/proxy/helpers.py`: - New `invalid_request_body_message(exc)` returns a fixed string per error type and logs the type: - `…: malformed JSON` - `…: not valid UTF-8 (possibly compressed?)` - `…: must be a JSON object` - `…: too large` - plain `Invalid request body` for anything else - New `RequestBodyNotObject(ValueError)` replaces a bare `ValueError`, so every existing `except ValueError` still catches it. - `headroom/proxy/handlers/{anthropic,openai,gemini,batch}.py`: - All 10 `f"Invalid request body: {e!s}"` replies now use the helper. - The Google batch forwarder's 500 reply drops the exception text; it was already logged on the line above. - `headroom/evals/comprehensive_benchmark.py`: `--model_args` is appended after the command is logged, and the log line shows it as `<redacted>`. - `tests/test_google_multimodal_e2e.py`: stops printing the first 10 characters of the API key. - `.github/workflows/{devcontainers,eval,init-e2e,init-native-e2e,wrap-e2e}.yml`: top-level `permissions: contents: read`. None of these jobs write to the repo; caching and artifact upload don't use the repo token. - `.github/workflows/security.yml`: CodeQL matrix adds `actions`. - `scripts/{build,verify}_npm_release_assets.mjs`: - On Windows, npm runs as `node <npm-cli.js> …` instead of `cmd.exe /c npm.cmd …`. - `npm-cli.js` is found via `npm_execpath`, then next to node, then beside each `npm.cmd` on PATH. This handles node and npm in different directories, per Devin's review. - If none is found, it fails listing every path searched. - New tests: - `tests/test_invalid_request_body_message.py` - `tests/test_evals/test_comprehensive_benchmark_log_redaction.py`, per Devanshi's review. ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [x] Type checking passes (`mypy headroom`): one error remains, the same one `main` has, in an untouched file. See notes. - [x] New tests added for new functionality - [x] Manual testing performed ### Test Output ```text $ pytest -q tests/test_release_workflows.py tests/test_invalid_request_body_message.py \ tests/test_evals/test_comprehensive_benchmark_log_redaction.py ============================== 62 passed in 3.10s ============================== # The redaction test against the pre-fix comprehensive_benchmark.py (origin/main): ========================= 1 failed, 1 passed in 0.50s ========================== # With the fix: ============================== 2 passed in 0.57s =============================== $ pytest -q (proxy/handler/batch/eval/request-body files, run one file at a time) 187 files: 182 OK; the 5 below fail or hang identically on unchanged origin/main: tests/test_proxy_uds.py 11 failed (macOS 104-byte sun_path limit) tests/test_anthropic_buffered_ccr_usage.py 6 failed tests/test_proxy_copilot_auth_hooks.py 2 failed tests/gateway/test_parity_proxy_vs_gateway.py 2 failed tests/test_cli_proxy_improvements.py hangs >300s $ ruff check . -q -> clean $ ruff format --check headroom tests -> clean $ mypy headroom headroom/integrations/langchain/chat_model.py:394: error: Function "RunnableBinding" could always be true in boolean context [truthy-function] Found 1 error in 1 file (checked 566 source files) # identical on origin/main $ node --check scripts/build_npm_release_assets.mjs && node --check scripts/verify_npm_release_assets.mjs -> ok ``` CI: the `test (3)` failures (`tests/test_onnx_dependency_contract.py`, 5 tests) and `test (4)` failures (`tests/test_xai_cache_economics.py`, 5 tests) also fail on `main`'s latest CI run (run 37806990929), so they're unrelated to this PR. ## Real Behavior Proof - Environment: macOS 26 (arm64), Python 3.12.13, Node 24, npm 11.19.0. Worktree of `origin/main` at87e05b68plus this branch. - Exact command / steps: the three checks below. 1. **Error replies:** `invalid_request_body_message` was exercised with each exception type, including a `JSONDecodeError` whose document is `"secret-doc"`. Each case asserts the reply names the category and contains neither the exception text nor the document. 2. **Benchmark:** `run_lm_eval(model_args="model=gpt-4o-mini,api_key=sk-test-FAKE-…")` was called with the subprocess call mocked. The log shows `--model_args <redacted>` and not the fake key, while the mocked call received `model=gpt-4o-mini,api_key=sk-test-FAKE-…,base_url=http://localhost:8787/v1`. 3. **Windows npm lookup:** `findNpmCli()` was extracted from each script and run against a temporary layout reproducing Devin's case: node shim in `dev/bin`, `npm.cmd` plus `node_modules/npm/bin/npm-cli.js` in `Program Files/nodejs`, both on PATH. - Observed result: all three Windows npm lookup cases pass in both scripts (output below); checks 1 and 2 pass as shown in Test Output. ```text == scripts/build_npm_release_assets.mjs separate node/npm dirs -> found npm-cli.js beside npm.cmd on PATH npm_execpath set -> uses npm_execpath missing npm -> clear error: true == scripts/verify_npm_release_assets.mjs separate node/npm dirs -> found npm-cli.js beside npm.cmd on PATH npm_execpath set -> uses npm_execpath missing npm -> clear error: true ``` - Not tested: the two items below. - A real Windows run of the release scripts. No Windows machine was available, and CI runs these scripts on Ubuntu, where the code path is unchanged (`npm` spawned directly, no shell). - The new CodeQL `actions` scan. It runs on this PR's CI and on `main` after merge. ### Maintainer verification at `128f271f` (2026-10-08) - Environment: actual Windows 11, Python 3.12.13, Node 22.22.0, npm 10.9.4. Contributor logic unchanged; this head contains current `main`. - Commands: the focused `test_invalid_request_body_message.py`, `test_comprehensive_benchmark_log_redaction.py`, and `test_release_workflows.py` suites; the existing Windows npm lookup probe against both checked-in scripts; and a two-process-boundary loopback uvicorn/httpx malformed-body probe against this head. - Observed: **62 focused tests passed**. Both npm scripts invoked the real npm CLI under Node with `shell:false`, including Node copied into a separate `G:/tmp/headroom-review/resume49-node shim & literal` directory while npm remained under `C:/Program Files/nodejs`; `npm_execpath` was unset, so PATH lookup found the separate npm installation. npm reported 10.9.4 and a cache argument containing spaces and `&` was preserved literally. **36 actual TCP requests** (four malformed-body categories across nine endpoints) returned the expected fixed 400 messages, with no private marker disclosed and no upstream requests. Only outbound provider requests were replaced with a fail-if-called boundary; HTTP parsing and proxy handlers were real. - Hosted qualification at this exact head: **67 passed, 11 skipped**, including all three CodeQL languages. Evidence is retained in the maintainer review artifacts. The earlier simulated Windows-layout proof remains below as historical author evidence. ## Runtime Rollout Safety - Rollout-managed feature(s): none. - Minimum rollout channel: n/a. - Stable/default behavior changed: yes, the text of 400 replies for unreadable request bodies. It was `Invalid request body: <python exception text>` and is now a fixed category message. Status codes, error `type`/`code` fields and response shape are unchanged. Clients that parse the free-text message, rather than the status and type, would see different text. - Kill switch / disable path: none needed; this is message text only. - Unsafe override required: no. - Qualification impact: none. - Rollback path: revert this PR. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation (none needed) - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` ## Additional Notes - Review feedback addressed in921de19b: - Devin: Windows npm lookup when node and npm are in different directories. - Devanshi: benchmark redaction test. - **#21 (`security_block` reply in `anthropic.py`) is left as is.** It returns the security plugin's deliberate, user-facing reason for blocking a request. It was dismissed as a false positive with that reason. - The `mypy` error is pre-existing: the same `truthy-function` error appears on `origin/main`, in a file this PR doesn't touch. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
56 lines
1.7 KiB
Python
56 lines
1.7 KiB
Python
"""Client-facing 400 text for unreadable request bodies (CodeQL py/stack-trace-exposure).
|
|
|
|
The handlers used to echo ``str(exc)`` back to the client. These pin that the
|
|
replacement still tells the client *what kind* of problem it was, and never
|
|
carries the exception's own text.
|
|
"""
|
|
|
|
import json
|
|
|
|
import pytest
|
|
|
|
from headroom.proxy.helpers import (
|
|
RequestBodyNotObject,
|
|
RequestBodyTooLarge,
|
|
invalid_request_body_message,
|
|
)
|
|
|
|
|
|
def _unicode_error() -> ValueError:
|
|
try:
|
|
b"\xff".decode("utf-8")
|
|
except UnicodeDecodeError as cause:
|
|
try:
|
|
raise ValueError(f"Request body is not valid UTF-8: {cause}") from cause
|
|
except ValueError as exc:
|
|
return exc
|
|
raise AssertionError("unreachable")
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
("exc", "expected"),
|
|
[
|
|
(RequestBodyTooLarge("Request body exceeds 100MB (Content-Length: 1)"), "too large"),
|
|
(RequestBodyNotObject("Request body must be a JSON object, not list"), "JSON object"),
|
|
(json.JSONDecodeError("Expecting value", "secret-doc", 0), "malformed JSON"),
|
|
(_unicode_error(), "not valid UTF-8"),
|
|
(ValueError("zstd frame header at offset 3"), None),
|
|
],
|
|
)
|
|
def test_message_names_the_category_without_exception_text(exc, expected):
|
|
message = invalid_request_body_message(exc)
|
|
|
|
assert message.startswith("Invalid request body")
|
|
if expected is None:
|
|
assert message == "Invalid request body"
|
|
else:
|
|
assert expected in message
|
|
assert str(exc) not in message
|
|
assert "secret-doc" not in message
|
|
|
|
|
|
def test_not_object_is_still_a_value_error():
|
|
# Existing ``except (json.JSONDecodeError, ValueError)`` call sites must keep
|
|
# catching it.
|
|
assert issubclass(RequestBodyNotObject, ValueError)
|