Commit Graph
4 Commits
Author SHA1 Message Date
Tejas ChopraandClaude Opus 5.5 a88e2d5145 fix(security): resolve 23 open CodeQL alerts (#4045)
## Description

CodeQL had 43 open alerts on `headroom` (19 high, 24 medium). We need 0
for SOC 2 (risk R-10: developers introduce vulnerabilities). This PR
fixes the 23 that are real problems in code we ship or run. The 16 in
`tests/` are handled by #4047. 11 were dismissed in GitHub with written
reasons: 7 stale alerts from the retired default-setup scan, and 4 false
positives (#21, #105, #109, #135).

**In plain words:**
1. **Error replies no longer include internal error text.** When a
request body couldn't be read, the proxy sent back Python's raw error
message. It now sends a fixed message that still says what kind of
problem it was.
2. **A benchmark no longer logs a possible API key.** `--model_args` is
shown as `<redacted>` in the log, but `lm_eval` still receives the real
value.
3. **A test no longer prints part of the Gemini API key.**
4. **Five workflows declare read-only permissions**, and CodeQL scans
GitHub Actions workflows again.
5. **The npm release scripts no longer pass arguments through Windows
`cmd.exe`**, which reinterprets `&` and `|` inside arguments, file paths
included.

Closes CodeQL alerts #20, #23, #24, #25, #26, #27, #28, #29, #31, #33,
#193 (stack-trace exposure), #177, #51, #145, #146, #147. The
workflow-permission fixes correspond to the dismissed #2, #7, #67, #85,
#86, #87, #125.

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/proxy/helpers.py`:
- New `invalid_request_body_message(exc)` returns a fixed string per
error type and logs the type:
    - `…: malformed JSON`
    - `…: not valid UTF-8 (possibly compressed?)`
    - `…: must be a JSON object`
    - `…: too large`
    - plain `Invalid request body` for anything else
- New `RequestBodyNotObject(ValueError)` replaces a bare `ValueError`,
so every existing `except ValueError` still catches it.
- `headroom/proxy/handlers/{anthropic,openai,gemini,batch}.py`:
  - All 10 `f"Invalid request body: {e!s}"` replies now use the helper.
- The Google batch forwarder's 500 reply drops the exception text; it
was already logged on the line above.
- `headroom/evals/comprehensive_benchmark.py`: `--model_args` is
appended after the command is logged, and the log line shows it as
`<redacted>`.
- `tests/test_google_multimodal_e2e.py`: stops printing the first 10
characters of the API key.
-
`.github/workflows/{devcontainers,eval,init-e2e,init-native-e2e,wrap-e2e}.yml`:
top-level `permissions: contents: read`. None of these jobs write to the
repo; caching and artifact upload don't use the repo token.
- `.github/workflows/security.yml`: CodeQL matrix adds `actions`.
- `scripts/{build,verify}_npm_release_assets.mjs`:
- On Windows, npm runs as `node <npm-cli.js> …` instead of `cmd.exe /c
npm.cmd …`.
- `npm-cli.js` is found via `npm_execpath`, then next to node, then
beside each `npm.cmd` on PATH. This handles node and npm in different
directories, per Devin's review.
  - If none is found, it fails listing every path searched.
- New tests:
  - `tests/test_invalid_request_body_message.py`
- `tests/test_evals/test_comprehensive_benchmark_log_redaction.py`, per
Devanshi's review.

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [x] Type checking passes (`mypy headroom`): one error remains, the
same one `main` has, in an untouched file. See notes.
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ pytest -q tests/test_release_workflows.py tests/test_invalid_request_body_message.py \
    tests/test_evals/test_comprehensive_benchmark_log_redaction.py
============================== 62 passed in 3.10s ==============================

# The redaction test against the pre-fix comprehensive_benchmark.py (origin/main):
========================= 1 failed, 1 passed in 0.50s ==========================
# With the fix:
============================== 2 passed in 0.57s ===============================

$ pytest -q (proxy/handler/batch/eval/request-body files, run one file at a time)
187 files: 182 OK; the 5 below fail or hang identically on unchanged origin/main:
  tests/test_proxy_uds.py                          11 failed  (macOS 104-byte sun_path limit)
  tests/test_anthropic_buffered_ccr_usage.py        6 failed
  tests/test_proxy_copilot_auth_hooks.py            2 failed
  tests/gateway/test_parity_proxy_vs_gateway.py     2 failed
  tests/test_cli_proxy_improvements.py             hangs >300s

$ ruff check . -q                      -> clean
$ ruff format --check headroom tests   -> clean
$ mypy headroom
headroom/integrations/langchain/chat_model.py:394: error: Function "RunnableBinding" could always be true in boolean context  [truthy-function]
Found 1 error in 1 file (checked 566 source files)   # identical on origin/main

$ node --check scripts/build_npm_release_assets.mjs && node --check scripts/verify_npm_release_assets.mjs  -> ok
```

CI: the `test (3)` failures (`tests/test_onnx_dependency_contract.py`, 5
tests) and `test (4)` failures (`tests/test_xai_cache_economics.py`, 5
tests) also fail on `main`'s latest CI run (run 37806990929), so they're
unrelated to this PR.

## Real Behavior Proof

- Environment: macOS 26 (arm64), Python 3.12.13, Node 24, npm 11.19.0.
Worktree of `origin/main` at 87e05b68 plus this branch.
- Exact command / steps: the three checks below.
1. **Error replies:** `invalid_request_body_message` was exercised with
each exception type, including a `JSONDecodeError` whose document is
`"secret-doc"`. Each case asserts the reply names the category and
contains neither the exception text nor the document.
2. **Benchmark:**
`run_lm_eval(model_args="model=gpt-4o-mini,api_key=sk-test-FAKE-…")` was
called with the subprocess call mocked. The log shows `--model_args
<redacted>` and not the fake key, while the mocked call received
`model=gpt-4o-mini,api_key=sk-test-FAKE-…,base_url=http://localhost:8787/v1`.
3. **Windows npm lookup:** `findNpmCli()` was extracted from each script
and run against a temporary layout reproducing Devin's case: node shim
in `dev/bin`, `npm.cmd` plus `node_modules/npm/bin/npm-cli.js` in
`Program Files/nodejs`, both on PATH.
- Observed result: all three Windows npm lookup cases pass in both
scripts (output below); checks 1 and 2 pass as shown in Test Output.
  ```text
  == scripts/build_npm_release_assets.mjs
  separate node/npm dirs -> found npm-cli.js beside npm.cmd on PATH
  npm_execpath set       -> uses npm_execpath
  missing npm            -> clear error: true
  == scripts/verify_npm_release_assets.mjs
  separate node/npm dirs -> found npm-cli.js beside npm.cmd on PATH
  npm_execpath set       -> uses npm_execpath
  missing npm            -> clear error: true
  ```
- Not tested: the two items below.
- A real Windows run of the release scripts. No Windows machine was
available, and CI runs these scripts on Ubuntu, where the code path is
unchanged (`npm` spawned directly, no shell).
- The new CodeQL `actions` scan. It runs on this PR's CI and on `main`
after merge.

### Maintainer verification at `128f271f` (2026-10-08)

- Environment: actual Windows 11, Python 3.12.13, Node 22.22.0, npm
10.9.4. Contributor logic unchanged; this head contains current `main`.
- Commands: the focused `test_invalid_request_body_message.py`,
`test_comprehensive_benchmark_log_redaction.py`, and
`test_release_workflows.py` suites; the existing Windows npm lookup
probe against both checked-in scripts; and a two-process-boundary
loopback uvicorn/httpx malformed-body probe against this head.
- Observed: **62 focused tests passed**. Both npm scripts invoked the
real npm CLI under Node with `shell:false`, including Node copied into a
separate `G:/tmp/headroom-review/resume49-node shim & literal` directory
while npm remained under `C:/Program Files/nodejs`; `npm_execpath` was
unset, so PATH lookup found the separate npm installation. npm reported
10.9.4 and a cache argument containing spaces and `&` was preserved
literally. **36 actual TCP requests** (four malformed-body categories
across nine endpoints) returned the expected fixed 400 messages, with no
private marker disclosed and no upstream requests. Only outbound
provider requests were replaced with a fail-if-called boundary; HTTP
parsing and proxy handlers were real.
- Hosted qualification at this exact head: **67 passed, 11 skipped**,
including all three CodeQL languages. Evidence is retained in the
maintainer review artifacts. The earlier simulated Windows-layout proof
remains below as historical author evidence.

## Runtime Rollout Safety

- Rollout-managed feature(s): none.
- Minimum rollout channel: n/a.
- Stable/default behavior changed: yes, the text of 400 replies for
unreadable request bodies. It was `Invalid request body: <python
exception text>` and is now a fixed category message. Status codes,
error `type`/`code` fields and response shape are unchanged. Clients
that parse the free-text message, rather than the status and type, would
see different text.
- Kill switch / disable path: none needed; this is message text only.
- Unsafe override required: no.
- Qualification impact: none.
- Rollback path: revert this PR.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation (none
needed)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`

## Additional Notes

- Review feedback addressed in 921de19b: 
- Devin: Windows npm lookup when node and npm are in different
directories.
  - Devanshi: benchmark redaction test.
- **#21 (`security_block` reply in `anthropic.py`) is left as is.** It
returns the security plugin's deliberate, user-facing reason for
blocking a request. It was dismissed as a false positive with that
reason.
- The `mypy` error is pre-existing: the same `truthy-function` error
appears on `origin/main`, in a file this PR doesn't touch.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 22:40:04 -05:00
chopratejas 3290a3d582 Remove LLMLingua: Kompress is the sole text compressor
LLMLingua was the original ML text compressor (BERT-based). Kompress
(ModernBERT, trained on 330K structured tool outputs) replaced it with
better compression quality and simpler architecture.

Removed across 35 files:
- Deleted headroom/transforms/llmlingua_compressor.py
- Deleted tests/test_transforms/test_llmlingua_compressor.py
- Deleted tests/test_proxy_llmlingua.py
- Removed all enable_llmlingua config, _get_llmlingua methods,
  LLMLingua fallback paths, LLMLINGUA strategy enum values
- Removed CLI flags, model configs, compression handler references
- Simplified ContentRouter: Kompress is primary and only text compressor
2026-03-26 11:11:00 -07:00
Claude Code BotandClaude Opus 4.5 9ff67fbd1b fix(tests): skip HTML extractor tests when trafilatura not installed
Add pytest.importorskip("trafilatura") to HTML extractor test modules
to skip tests gracefully when the optional trafilatura dependency is
not installed. This fixes CI failures in the base test matrix that
doesn't include the html extras.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-31 15:39:55 -08:00
chopratejasandClaude Opus 4.5 d1a28322cc Add HTMLExtractor for web content extraction with OSS benchmarks
HTMLExtractor uses trafilatura to extract main content from HTML pages,
removing scripts, styles, navigation, and ads. This achieves 94.9%
compression while preserving 98.2% recall on the Scrapinghub benchmark.

Key features:
- Automatic HTML detection in content router
- Configurable output format (markdown or text)
- Metadata extraction (title, author, date, description)
- Batch extraction support

Evaluation framework:
- OSS benchmark integration (Scrapinghub Article Extraction Benchmark)
- LLM-as-judge evaluation for QA accuracy preservation
- F1 score: 0.919 on 181-sample benchmark (baseline: 0.958)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-31 10:08:29 -08:00