fix(security): resolve 23 open CodeQL alerts (#4045)

## Description

CodeQL had 43 open alerts on `headroom` (19 high, 24 medium). We need 0
for SOC 2 (risk R-10: developers introduce vulnerabilities). This PR
fixes the 23 that are real problems in code we ship or run. The 16 in
`tests/` are handled by #4047. 11 were dismissed in GitHub with written
reasons: 7 stale alerts from the retired default-setup scan, and 4 false
positives (#21, #105, #109, #135).

**In plain words:**
1. **Error replies no longer include internal error text.** When a
request body couldn't be read, the proxy sent back Python's raw error
message. It now sends a fixed message that still says what kind of
problem it was.
2. **A benchmark no longer logs a possible API key.** `--model_args` is
shown as `<redacted>` in the log, but `lm_eval` still receives the real
value.
3. **A test no longer prints part of the Gemini API key.**
4. **Five workflows declare read-only permissions**, and CodeQL scans
GitHub Actions workflows again.
5. **The npm release scripts no longer pass arguments through Windows
`cmd.exe`**, which reinterprets `&` and `|` inside arguments, file paths
included.

Closes CodeQL alerts #20, #23, #24, #25, #26, #27, #28, #29, #31, #33,
#193 (stack-trace exposure), #177, #51, #145, #146, #147. The
workflow-permission fixes correspond to the dismissed #2, #7, #67, #85,
#86, #87, #125.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/proxy/helpers.py`:
- New `invalid_request_body_message(exc)` returns a fixed string per
error type and logs the type:
    - `…: malformed JSON`
    - `…: not valid UTF-8 (possibly compressed?)`
    - `…: must be a JSON object`
    - `…: too large`
    - plain `Invalid request body` for anything else
- New `RequestBodyNotObject(ValueError)` replaces a bare `ValueError`,
so every existing `except ValueError` still catches it.
- `headroom/proxy/handlers/{anthropic,openai,gemini,batch}.py`:
  - All 10 `f"Invalid request body: {e!s}"` replies now use the helper.
- The Google batch forwarder's 500 reply drops the exception text; it
was already logged on the line above.
- `headroom/evals/comprehensive_benchmark.py`: `--model_args` is
appended after the command is logged, and the log line shows it as
`<redacted>`.
- `tests/test_google_multimodal_e2e.py`: stops printing the first 10
characters of the API key.
-
`.github/workflows/{devcontainers,eval,init-e2e,init-native-e2e,wrap-e2e}.yml`:
top-level `permissions: contents: read`. None of these jobs write to the
repo; caching and artifact upload don't use the repo token.
- `.github/workflows/security.yml`: CodeQL matrix adds `actions`.
- `scripts/{build,verify}_npm_release_assets.mjs`:
- On Windows, npm runs as `node <npm-cli.js> …` instead of `cmd.exe /c
npm.cmd …`.
- `npm-cli.js` is found via `npm_execpath`, then next to node, then
beside each `npm.cmd` on PATH. This handles node and npm in different
directories, per Devin's review.
  - If none is found, it fails listing every path searched.
- New tests:
  - `tests/test_invalid_request_body_message.py`
- `tests/test_evals/test_comprehensive_benchmark_log_redaction.py`, per
Devanshi's review.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`): one error remains, the
same one `main` has, in an untouched file. See notes.
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ pytest -q tests/test_release_workflows.py tests/test_invalid_request_body_message.py \
    tests/test_evals/test_comprehensive_benchmark_log_redaction.py
============================== 62 passed in 3.10s ==============================

# The redaction test against the pre-fix comprehensive_benchmark.py (origin/main):
========================= 1 failed, 1 passed in 0.50s ==========================
# With the fix:
============================== 2 passed in 0.57s ===============================

$ pytest -q (proxy/handler/batch/eval/request-body files, run one file at a time)
187 files: 182 OK; the 5 below fail or hang identically on unchanged origin/main:
  tests/test_proxy_uds.py                          11 failed  (macOS 104-byte sun_path limit)
  tests/test_anthropic_buffered_ccr_usage.py        6 failed
  tests/test_proxy_copilot_auth_hooks.py            2 failed
  tests/gateway/test_parity_proxy_vs_gateway.py     2 failed
  tests/test_cli_proxy_improvements.py             hangs >300s

$ ruff check . -q                      -> clean
$ ruff format --check headroom tests   -> clean
$ mypy headroom
headroom/integrations/langchain/chat_model.py:394: error: Function "RunnableBinding" could always be true in boolean context  [truthy-function]
Found 1 error in 1 file (checked 566 source files)   # identical on origin/main

$ node --check scripts/build_npm_release_assets.mjs && node --check scripts/verify_npm_release_assets.mjs  -> ok
```

CI: the `test (3)` failures (`tests/test_onnx_dependency_contract.py`, 5
tests) and `test (4)` failures (`tests/test_xai_cache_economics.py`, 5
tests) also fail on `main`'s latest CI run (run 37806990929), so they're
unrelated to this PR.

## Real Behavior Proof

- Environment: macOS 26 (arm64), Python 3.12.13, Node 24, npm 11.19.0.
Worktree of `origin/main` at 87e05b68 plus this branch.
- Exact command / steps: the three checks below.
1. **Error replies:** `invalid_request_body_message` was exercised with
each exception type, including a `JSONDecodeError` whose document is
`"secret-doc"`. Each case asserts the reply names the category and
contains neither the exception text nor the document.
2. **Benchmark:**
`run_lm_eval(model_args="model=gpt-4o-mini,api_key=sk-test-FAKE-…")` was
called with the subprocess call mocked. The log shows `--model_args
<redacted>` and not the fake key, while the mocked call received
`model=gpt-4o-mini,api_key=sk-test-FAKE-…,base_url=http://localhost:8787/v1`.
3. **Windows npm lookup:** `findNpmCli()` was extracted from each script
and run against a temporary layout reproducing Devin's case: node shim
in `dev/bin`, `npm.cmd` plus `node_modules/npm/bin/npm-cli.js` in
`Program Files/nodejs`, both on PATH.
- Observed result: all three Windows npm lookup cases pass in both
scripts (output below); checks 1 and 2 pass as shown in Test Output.
  ```text
  == scripts/build_npm_release_assets.mjs
  separate node/npm dirs -> found npm-cli.js beside npm.cmd on PATH
  npm_execpath set       -> uses npm_execpath
  missing npm            -> clear error: true
  == scripts/verify_npm_release_assets.mjs
  separate node/npm dirs -> found npm-cli.js beside npm.cmd on PATH
  npm_execpath set       -> uses npm_execpath
  missing npm            -> clear error: true
  ```
- Not tested: the two items below.
- A real Windows run of the release scripts. No Windows machine was
available, and CI runs these scripts on Ubuntu, where the code path is
unchanged (`npm` spawned directly, no shell).
- The new CodeQL `actions` scan. It runs on this PR's CI and on `main`
after merge.

### Maintainer verification at `128f271f` (2026-10-08)

- Environment: actual Windows 11, Python 3.12.13, Node 22.22.0, npm
10.9.4. Contributor logic unchanged; this head contains current `main`.
- Commands: the focused `test_invalid_request_body_message.py`,
`test_comprehensive_benchmark_log_redaction.py`, and
`test_release_workflows.py` suites; the existing Windows npm lookup
probe against both checked-in scripts; and a two-process-boundary
loopback uvicorn/httpx malformed-body probe against this head.
- Observed: **62 focused tests passed**. Both npm scripts invoked the
real npm CLI under Node with `shell:false`, including Node copied into a
separate `G:/tmp/headroom-review/resume49-node shim & literal` directory
while npm remained under `C:/Program Files/nodejs`; `npm_execpath` was
unset, so PATH lookup found the separate npm installation. npm reported
10.9.4 and a cache argument containing spaces and `&` was preserved
literally. **36 actual TCP requests** (four malformed-body categories
across nine endpoints) returned the expected fixed 400 messages, with no
private marker disclosed and no upstream requests. Only outbound
provider requests were replaced with a fail-if-called boundary; HTTP
parsing and proxy handlers were real.
- Hosted qualification at this exact head: **67 passed, 11 skipped**,
including all three CodeQL languages. Evidence is retained in the
maintainer review artifacts. The earlier simulated Windows-layout proof
remains below as historical author evidence.

## Runtime Rollout Safety

- Rollout-managed feature(s): none.
- Minimum rollout channel: n/a.
- Stable/default behavior changed: yes, the text of 400 replies for
unreadable request bodies. It was `Invalid request body: <python
exception text>` and is now a fixed category message. Status codes,
error `type`/`code` fields and response shape are unchanged. Clients
that parse the free-text message, rather than the status and type, would
see different text.
- Kill switch / disable path: none needed; this is message text only.
- Unsafe override required: no.
- Qualification impact: none.
- Rollback path: revert this PR.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation (none
needed)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`

## Additional Notes

- Review feedback addressed in 921de19b: 
- Devin: Windows npm lookup when node and npm are in different
directories.
  - Devanshi: benchmark redaction test.
- **#21 (`security_block` reply in `anthropic.py`) is left as is.** It
returns the security plugin's deliberate, user-facing reason for
blocking a request. It was dismissed as a false positive with that
reason.
- The `mypy` error is pre-existing: the same `truthy-function` error
appears on `origin/main`, in a file this PR doesn't touch.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Tejas Chopra
2026-10-08 22:40:04 -05:00
committed by GitHub
co-authored by Claude Opus 5.5
parent c7a223dac0
commit a88e2d5145
16 changed files with 263 additions and 31 deletions
+3
View File
@@ -17,6 +17,9 @@ on:
- "uv.lock"
workflow_dispatch:
permissions:
contents: read
jobs:
validate:
runs-on: ubuntu-latest
+3
View File
@@ -10,6 +10,9 @@ on:
- 'headroom/evals/**'
- 'headroom/compress.py'
permissions:
contents: read
jobs:
# Fast smoke test on PRs touching compression code (~$0.05, ~2 min)
smoke-test:
+3
View File
@@ -31,6 +31,9 @@ concurrency:
group: init-e2e-${{ github.ref }}
cancel-in-progress: ${{ github.ref != 'refs/heads/main' }}
permissions:
contents: read
jobs:
docker-init-e2e:
runs-on: ubuntu-latest
+3
View File
@@ -32,6 +32,9 @@ on:
branches: [main]
workflow_dispatch:
permissions:
contents: read
jobs:
init-native:
runs-on: ${{ matrix.os }}
+1 -1
View File
@@ -120,7 +120,7 @@ jobs:
strategy:
fail-fast: false
matrix:
language: [python, javascript-typescript]
language: [python, javascript-typescript, actions]
steps:
- uses: actions/checkout@v7
+3
View File
@@ -30,6 +30,9 @@ concurrency:
group: wrap-e2e-${{ github.ref }}
cancel-in-progress: ${{ github.ref != 'refs/heads/main' }}
permissions:
contents: read
jobs:
docker-wrap-e2e:
runs-on: ubuntu-latest
+8 -3
View File
@@ -248,8 +248,7 @@ def run_lm_eval(
if base_url:
args_parts.append(f"base_url={base_url}")
if args_parts:
cmd.extend(["--model_args", ",".join(args_parts)])
model_args_value = ",".join(args_parts) if args_parts else None
if num_fewshot is not None:
cmd.extend(["--num_fewshot", str(num_fewshot)])
@@ -264,7 +263,13 @@ def run_lm_eval(
cmd.extend(["--output_path", output_path])
logger.info(f"Running: {' '.join(cmd)}")
# --model_args can carry an API key, so it is appended after the command is
# logged and the log shows it redacted.
logger.info(
f"Running: {' '.join(cmd)}" + (" --model_args <redacted>" if model_args_value else "")
)
if model_args_value:
cmd.extend(["--model_args", model_args_value])
# Run lm-eval
start_time = time.time()
+3 -2
View File
@@ -56,6 +56,7 @@ from headroom.proxy.compression_decision import CompressionDecision
from headroom.proxy.handlers._debug_dump import _debug_dump_mode, _redact_debug_value
from headroom.proxy.helpers import (
extract_tags,
invalid_request_body_message,
relocate_system_messages_to_top_level,
sanitize_forwarded_response_headers,
)
@@ -1166,7 +1167,7 @@ class AnthropicHandlerMixin:
"type": "error",
"error": {
"type": "invalid_request_error",
"message": f"Invalid request body: {e!s}",
"message": invalid_request_body_message(e),
},
},
)
@@ -5653,7 +5654,7 @@ class AnthropicHandlerMixin:
"type": "error",
"error": {
"type": "invalid_request_error",
"message": f"Invalid request body: {e!s}",
"message": invalid_request_body_message(e),
},
},
)
+4 -3
View File
@@ -19,6 +19,7 @@ from headroom.proxy.helpers import (
COMPRESSION_TIMEOUT_SECONDS,
_headroom_bypass_enabled,
extract_tags,
invalid_request_body_message,
)
from headroom.proxy.outcome import RequestOutcome
@@ -86,7 +87,7 @@ class BatchHandlerMixin:
content={
"error": {
"code": 400,
"message": f"Invalid request body: {e!s}",
"message": invalid_request_body_message(e),
"status": "INVALID_ARGUMENT",
}
},
@@ -397,7 +398,7 @@ class BatchHandlerMixin:
content={
"error": {
"code": 500,
"message": f"Failed to forward batch request: {e!s}",
"message": "Failed to forward batch request",
"status": "INTERNAL",
}
},
@@ -853,7 +854,7 @@ class BatchHandlerMixin:
status_code=400,
content={
"error": {
"message": f"Invalid request body: {e!s}",
"message": invalid_request_body_message(e),
"type": "invalid_request_error",
"code": "invalid_json",
}
+8 -4
View File
@@ -22,7 +22,11 @@ from headroom.agent_savings import proxy_pipeline_kwargs
from headroom.copilot_auth import build_copilot_upstream_url
from headroom.proxy.auth_mode import classify_client
from headroom.proxy.compression_decision import CompressionDecision
from headroom.proxy.helpers import COMPRESSION_TIMEOUT_SECONDS, extract_tags
from headroom.proxy.helpers import (
COMPRESSION_TIMEOUT_SECONDS,
extract_tags,
invalid_request_body_message,
)
from headroom.proxy.identity import resolve_memory_identity
from headroom.proxy.outcome import RequestOutcome
from headroom.proxy.rate_limit_identity import rate_limit_identity
@@ -367,7 +371,7 @@ class GeminiHandlerMixin:
status_code=400,
content={
"error": {
"message": f"Invalid request body: {e!s}",
"message": invalid_request_body_message(e),
"code": 400,
}
},
@@ -1133,7 +1137,7 @@ class GeminiHandlerMixin:
status_code=400,
content={
"error": {
"message": f"Invalid request body: {e!s}",
"message": invalid_request_body_message(e),
"code": 400,
}
},
@@ -1361,7 +1365,7 @@ class GeminiHandlerMixin:
status_code=400,
content={
"error": {
"message": f"Invalid request body: {e!s}",
"message": invalid_request_body_message(e),
"code": 400,
}
},
+4 -3
View File
@@ -30,6 +30,7 @@ from headroom.proxy.helpers import (
COMPRESSION_TIMEOUT_SECONDS,
_headroom_bypass_enabled,
extract_tags,
invalid_request_body_message,
jitter_delay_ms,
sanitize_forwarded_response_headers,
)
@@ -3621,7 +3622,7 @@ class OpenAIHandlerMixin:
status_code=400,
content={
"error": {
"message": f"Invalid request body: {e!s}",
"message": invalid_request_body_message(e),
"type": "invalid_request_error",
"code": "invalid_json",
}
@@ -6031,7 +6032,7 @@ class OpenAIHandlerMixin:
status_code=400,
content={
"error": {
"message": f"Invalid request body: {e!s}",
"message": invalid_request_body_message(e),
"type": "invalid_request_error",
"code": "invalid_json",
}
@@ -10652,7 +10653,7 @@ class OpenAIHandlerMixin:
except (json.JSONDecodeError, ValueError) as e:
return JSONResponse(
status_code=400,
content={"error": f"Invalid request body: {e!s}"},
content={"error": invalid_request_body_message(e)},
)
messages = body.get("messages", [])
_bypass_payload = {
+30 -2
View File
@@ -3015,6 +3015,30 @@ class RequestBodyTooLarge(ValueError):
"""
class RequestBodyNotObject(ValueError):
"""The request body parsed as JSON but is not an object (e.g. a list)."""
def invalid_request_body_message(exc: Exception) -> str:
"""Client-facing text for a request body that failed to read or parse.
Fixed strings only, chosen by exception type. The exception's own text can
carry decoder internals and request metadata, and echoing it back is what
CodeQL's ``py/stack-trace-exposure`` flags. The type is logged so a 400 can
still be diagnosed from the proxy side.
"""
logger.info("rejected request body (%s)", type(exc).__name__)
if isinstance(exc, RequestBodyTooLarge):
return "Invalid request body: too large"
if isinstance(exc, RequestBodyNotObject):
return "Invalid request body: must be a JSON object"
if isinstance(exc, json.JSONDecodeError):
return "Invalid request body: malformed JSON"
if isinstance(exc.__cause__, UnicodeDecodeError):
return "Invalid request body: not valid UTF-8 (possibly compressed?)"
return "Invalid request body"
def _inflate_bounded(raw: bytes, *, wbits: int, label: str, multi_member: bool = False) -> bytes:
"""Incrementally inflate ``raw``, stopping the instant output passes the cap.
@@ -3347,7 +3371,9 @@ async def _read_request_json(request: Request) -> dict[str, Any]:
result = json.loads(text)
if not isinstance(result, dict):
raise ValueError("Request body must be a JSON object, not " + type(result).__name__)
raise RequestBodyNotObject(
"Request body must be a JSON object, not " + type(result).__name__
)
# Drop output-only blocks the request schema rejects (see
# ``strip_output_only_request_blocks``). Callers of this bytes-less reader
@@ -3387,7 +3413,9 @@ async def read_request_json_with_bytes(request: Request) -> tuple[dict[str, Any]
result = json.loads(text)
if not isinstance(result, dict):
raise ValueError("Request body must be a JSON object, not " + type(result).__name__)
raise RequestBodyNotObject(
"Request body must be a JSON object, not " + type(result).__name__
)
# Drop output-only blocks (see ``strip_output_only_request_blocks``) before
# any downstream deepcopy / compression / 400-retry path. This is the shared
+36 -5
View File
@@ -86,12 +86,43 @@ function run(command, args, cwd) {
}
}
function runNpm(args, cwd) {
if (process.platform === "win32") {
run("cmd.exe", ["/d", "/s", "/c", "npm.cmd", ...args], cwd);
return;
// npm on Windows is a .cmd shim, which only runs through cmd.exe, and cmd.exe
// reinterprets characters such as & and | inside arguments (paths included).
// Run npm's own CLI script under this Node binary instead, so no shell is
// involved on any platform.
function npmCommand(args) {
if (process.platform !== "win32") {
return ["npm", args];
}
run("npm", args, cwd);
return [process.execPath, [findNpmCli(), ...args]];
}
// Node and npm can live in different directories (a node shim, or npm
// upgraded into the global prefix), so look where npm itself says it is, then
// next to node, then next to every npm.cmd on PATH. Every npm.cmd install
// keeps its CLI at node_modules/npm/bin/npm-cli.js beside the shim.
function findNpmCli() {
const cliPath = path.join("node_modules", "npm", "bin", "npm-cli.js");
const candidates = [];
if (process.env.npm_execpath?.endsWith("npm-cli.js")) {
candidates.push(process.env.npm_execpath);
}
candidates.push(path.join(path.dirname(process.execPath), cliPath));
for (const dir of (process.env.PATH || "").split(path.delimiter)) {
if (dir && existsSync(path.join(dir, "npm.cmd"))) {
candidates.push(path.join(dir, cliPath));
}
}
const found = candidates.find((candidate) => existsSync(candidate));
if (!found) {
throw new Error(`npm CLI (npm-cli.js) not found. Looked in: ${candidates.join(", ")}`);
}
return found;
}
function runNpm(args, cwd) {
const [command, commandArgs] = npmCommand(args);
run(command, commandArgs, cwd);
}
function runNode(args, cwd) {
+36 -8
View File
@@ -1,6 +1,6 @@
#!/usr/bin/env node
import { copyFileSync, mkdtempSync, readFileSync, rmSync } from "node:fs";
import { copyFileSync, existsSync, mkdtempSync, readFileSync, rmSync } from "node:fs";
import { tmpdir } from "node:os";
import path from "node:path";
import { spawnSync } from "node:child_process";
@@ -65,15 +65,43 @@ function assertNoFileDependencies(pkg) {
}
}
function runNpm(args, cwd) {
if (process.platform === "win32") {
return spawnSync("cmd.exe", ["/d", "/s", "/c", "npm.cmd", ...args], {
cwd,
encoding: "utf8",
});
// npm on Windows is a .cmd shim, which only runs through cmd.exe, and cmd.exe
// reinterprets characters such as & and | inside arguments (paths included).
// Run npm's own CLI script under this Node binary instead, so no shell is
// involved on any platform.
function npmCommand(args) {
if (process.platform !== "win32") {
return ["npm", args];
}
return [process.execPath, [findNpmCli(), ...args]];
}
return spawnSync("npm", args, {
// Node and npm can live in different directories (a node shim, or npm
// upgraded into the global prefix), so look where npm itself says it is, then
// next to node, then next to every npm.cmd on PATH. Every npm.cmd install
// keeps its CLI at node_modules/npm/bin/npm-cli.js beside the shim.
function findNpmCli() {
const cliPath = path.join("node_modules", "npm", "bin", "npm-cli.js");
const candidates = [];
if (process.env.npm_execpath?.endsWith("npm-cli.js")) {
candidates.push(process.env.npm_execpath);
}
candidates.push(path.join(path.dirname(process.execPath), cliPath));
for (const dir of (process.env.PATH || "").split(path.delimiter)) {
if (dir && existsSync(path.join(dir, "npm.cmd"))) {
candidates.push(path.join(dir, cliPath));
}
}
const found = candidates.find((candidate) => existsSync(candidate));
if (!found) {
throw new Error(`npm CLI (npm-cli.js) not found. Looked in: ${candidates.join(", ")}`);
}
return found;
}
function runNpm(args, cwd) {
const [command, commandArgs] = npmCommand(args);
return spawnSync(command, commandArgs, {
cwd,
encoding: "utf8",
});
@@ -0,0 +1,63 @@
"""run_lm_eval must not log --model_args, which can carry an API key (CodeQL #177).
The redaction is log-only: lm_eval itself still has to receive the real value,
or the benchmark silently runs against the wrong endpoint/credentials.
"""
import logging
from types import SimpleNamespace
import pytest
from headroom.evals import comprehensive_benchmark
FAKE_SECRET = "sk-test-FAKE-0123456789abcdef"
def test_model_args_redacted_in_log_but_passed_to_lm_eval(monkeypatch, caplog, tmp_path):
calls: list[list[str]] = []
def fake_run(cmd, **_kwargs):
calls.append(list(cmd))
# Non-zero exit stops run_lm_eval right after the call, before it
# looks for result files we did not create.
return SimpleNamespace(returncode=1, stdout="", stderr="stopped by test")
monkeypatch.setattr(comprehensive_benchmark, "run", fake_run)
with caplog.at_level(logging.INFO, logger=comprehensive_benchmark.__name__):
with pytest.raises(RuntimeError, match="stopped by test"):
comprehensive_benchmark.run_lm_eval(
model_args=f"model=gpt-4o-mini,api_key={FAKE_SECRET}",
base_url="http://localhost:8787/v1",
tasks=["arc_easy"],
output_path=str(tmp_path),
)
running = [r.getMessage() for r in caplog.records if r.getMessage().startswith("Running:")]
assert len(running) == 1
assert "--model_args <redacted>" in running[0]
assert FAKE_SECRET not in caplog.text
assert len(calls) == 1
cmd = calls[0]
assert cmd[cmd.index("--model_args") + 1] == (
f"model=gpt-4o-mini,api_key={FAKE_SECRET},base_url=http://localhost:8787/v1"
)
def test_no_model_args_logs_no_placeholder(monkeypatch, caplog, tmp_path):
calls: list[list[str]] = []
def fake_run(cmd, **_kwargs):
calls.append(list(cmd))
return SimpleNamespace(returncode=1, stdout="", stderr="stopped by test")
monkeypatch.setattr(comprehensive_benchmark, "run", fake_run)
with caplog.at_level(logging.INFO, logger=comprehensive_benchmark.__name__):
with pytest.raises(RuntimeError):
comprehensive_benchmark.run_lm_eval(tasks=["arc_easy"], output_path=str(tmp_path))
assert "--model_args" not in caplog.text
assert "--model_args" not in calls[0]
@@ -0,0 +1,55 @@
"""Client-facing 400 text for unreadable request bodies (CodeQL py/stack-trace-exposure).
The handlers used to echo ``str(exc)`` back to the client. These pin that the
replacement still tells the client *what kind* of problem it was, and never
carries the exception's own text.
"""
import json
import pytest
from headroom.proxy.helpers import (
RequestBodyNotObject,
RequestBodyTooLarge,
invalid_request_body_message,
)
def _unicode_error() -> ValueError:
try:
b"\xff".decode("utf-8")
except UnicodeDecodeError as cause:
try:
raise ValueError(f"Request body is not valid UTF-8: {cause}") from cause
except ValueError as exc:
return exc
raise AssertionError("unreachable")
@pytest.mark.parametrize(
("exc", "expected"),
[
(RequestBodyTooLarge("Request body exceeds 100MB (Content-Length: 1)"), "too large"),
(RequestBodyNotObject("Request body must be a JSON object, not list"), "JSON object"),
(json.JSONDecodeError("Expecting value", "secret-doc", 0), "malformed JSON"),
(_unicode_error(), "not valid UTF-8"),
(ValueError("zstd frame header at offset 3"), None),
],
)
def test_message_names_the_category_without_exception_text(exc, expected):
message = invalid_request_body_message(exc)
assert message.startswith("Invalid request body")
if expected is None:
assert message == "Invalid request body"
else:
assert expected in message
assert str(exc) not in message
assert "secret-doc" not in message
def test_not_object_is_still_a_value_error():
# Existing ``except (json.JSONDecodeError, ValueError)`` call sites must keep
# catching it.
assert issubclass(RequestBodyNotObject, ValueError)