Files
headroom/tests/test_google_multimodal_e2e.py
Tejas ChopraandClaude Opus 5.5 c7a223dac0 ci(codeql): exclude tests/ from CodeQL scanning (#4047)
## Description

**In plain words:** 17 of our open CodeQL alerts are in `tests/`. 16 are
normal test code that CodeQL misreads, and **1 was a real problem, fixed
in this PR before the exclusion takes effect**:
- **15** are tests checking that a URL appears in some output, e.g.
`assert "https://…" in output`. CodeQL reads that as a weak URL security
check.
- **1** is a fake test server listening on all network interfaces
(`tests/gateway/fake_provider.py`).
- **1 is real:** `tests/test_google_multimodal_e2e.py` printed the first
10 characters of the runner's real `GOOGLE_API_KEY` (alert 51). This PR
removes that print (`dc01e95c`). It is the same one-line change as in
#4045, so the two merge cleanly in either order.

Once that print is gone, what remains under `tests/` is noise that
buries real findings: test code never ships to users. Excluding `tests/`
is standard practice, and production code (`headroom/`, `scripts/`,
plugins, workflows) is still scanned.

Part of getting CodeQL to 0 open alerts for SOC 2 (risk R-10). Companion
to #4045.

## Type of Change

- [ ] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [x] Code refactoring (no functional changes): CI configuration only

## Changes Made

- `.github/workflows/security.yml`: the CodeQL `init` step gets
  ```yaml
  config: |
    paths-ignore:
      - tests
  ```
plus a comment explaining why. Nothing else changes: same languages,
same `security-extended` query suite, same triggers.
- `tests/test_google_multimodal_e2e.py`: `print(f"Using API key:
{api_key[:10]}...")` becomes `print("Using API key from environment")`.

## Testing

- [ ] Unit tests pass (`pytest`): not applicable; the only Python change
is one `print` in a manual e2e script, checked with `py_compile`
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`): not applicable
- [ ] New tests added for new functionality
- [x] Manual testing performed

### Test Output

```text
$ python -c "import yaml; d=yaml.safe_load(open('.github/workflows/security.yml')); \
    print([s['with'] for s in d['jobs']['codeql']['steps'] if s.get('name')=='Initialize CodeQL'])"
[{'languages': '${{ matrix.language }}', 'queries': 'security-extended', 'config': 'paths-ignore:\n  - tests\n'}]

# Open CodeQL alerts on main before this PR, by location:
$ gh api 'repos/headroomlabs-ai/headroom/code-scanning/alerts?ref=refs/heads/main&state=open' ...
     15 code
     17 tests/
```

CI on this PR: `test (3)` (`tests/test_onnx_dependency_contract.py`) and
`test (4)` (`tests/test_xai_cache_economics.py`) fail identically on
`main`'s latest CI run (run 37806990929). This PR changes no Python.

## Real Behavior Proof

- Environment: GitHub Actions CodeQL (`github/codeql-action@v4`),
`security.yml` on `main`.
- Exact command / steps: the YAML parses, and the `config` block reaches
the CodeQL `init` step as shown above. The CodeQL scans on this PR
completed successfully for `python` and `javascript-typescript`.
- Observed result: before merge, `main` has 17 open alerts under
`tests/`.
- Not tested: the closure of those 17 alerts can only be observed after
merge. On pull requests, CodeQL reports only alerts on changed lines, so
this PR's own scan can't show it. After merge, the push-triggered scan
of `main` should close all 17. I'll confirm the count on `main` and post
it here.

## Runtime Rollout Safety

- Rollout-managed feature(s): none.
- Minimum rollout channel: n/a.
- Stable/default behavior changed: no product behavior changes. CodeQL
stops analysing files under `tests/`.
- Kill switch / disable path: remove the `config` block.
- Unsafe override required: no.
- Qualification impact: none.
- Rollback path: revert this PR. The next scan of `main` will re-report
test-code alerts.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation (none
needed)
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works (CI config only; verified after merge as described above)
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md`

## Additional Notes

- Trade-off: a real vulnerability inside test code would no longer be
flagged. Test code doesn't ship, and the alternative was dismissing 17
alerts one by one as "used in tests" and repeating that for every new
one.
- **Devin's review asked for runtime proof.** See "Not tested" above:
the proof is the post-merge scan of `main`, which I'll post as a
comment.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 22:38:47 -05:00

376 lines
13 KiB
Python

"""End-to-end test for Google Gemini multimodal content preservation.
This test uses the real Google Gemini API to verify that non-text content
(images, function calls) is preserved through the proxy's compression pipeline.
These tests require a GOOGLE_API_KEY environment variable and are skipped in CI.
Run manually with: GOOGLE_API_KEY=your_key python tests/test_google_multimodal_e2e.py
"""
import asyncio
import os
import httpx
import pytest
# 10x10 red pixel PNG for testing (valid image generated by PIL)
TINY_RED_PNG = "iVBORw0KGgoAAAANSUhEUgAAAAoAAAAKCAIAAAACUFjqAAAAEklEQVR4nGP8z4APMOGVHbHSAEEsAROxCnMTAAAAAElFTkSuQmCC"
@pytest.fixture
def api_key():
"""Get API key from environment, skip if not available."""
key = os.environ.get("GOOGLE_API_KEY")
if not key:
pytest.skip("GOOGLE_API_KEY not set - skipping E2E tests")
return key
@pytest.mark.skipif(
not os.environ.get("GOOGLE_API_KEY"),
reason="GOOGLE_API_KEY not set - E2E tests require real API access",
)
@pytest.mark.asyncio
async def test_text_only_request(api_key):
"""Test that pure text requests work normally."""
print("\n=== Test 1: Pure Text Request ===")
url = f"https://generativelanguage.googleapis.com/v1beta/models/gemini-2.0-flash:generateContent?key={api_key}"
payload = {
"contents": [
{"role": "user", "parts": [{"text": "What is 2 + 2? Reply with just the number."}]}
]
}
async with httpx.AsyncClient() as client:
response = await client.post(url, json=payload, timeout=30)
print(f"Status: {response.status_code}")
if response.status_code == 200:
data = response.json()
text = (
data.get("candidates", [{}])[0].get("content", {}).get("parts", [{}])[0].get("text", "")
)
print(f"Response: {text[:100]}")
print("✅ Text-only request works")
return True
else:
print(f"Error: {response.text[:200]}")
return False
@pytest.mark.skipif(
not os.environ.get("GOOGLE_API_KEY"),
reason="GOOGLE_API_KEY not set - E2E tests require real API access",
)
@pytest.mark.asyncio
async def test_image_request(api_key):
"""Test that image content is preserved and processed."""
print("\n=== Test 2: Image Request (inlineData) ===")
url = f"https://generativelanguage.googleapis.com/v1beta/models/gemini-2.0-flash:generateContent?key={api_key}"
# Request with inline image
payload = {
"contents": [
{
"role": "user",
"parts": [
{"text": "What color is this tiny image? Reply with just the color name."},
{"inlineData": {"mimeType": "image/png", "data": TINY_RED_PNG}},
],
}
]
}
async with httpx.AsyncClient() as client:
response = await client.post(url, json=payload, timeout=30)
print(f"Status: {response.status_code}")
if response.status_code == 200:
data = response.json()
text = (
data.get("candidates", [{}])[0].get("content", {}).get("parts", [{}])[0].get("text", "")
)
print(f"Response: {text[:100]}")
print("✅ Image request works - model processed the image")
return True
else:
print(f"Error: {response.text[:200]}")
return False
@pytest.mark.skipif(
not os.environ.get("GOOGLE_API_KEY"),
reason="GOOGLE_API_KEY not set - E2E tests require real API access",
)
@pytest.mark.asyncio
async def test_function_calling(api_key):
"""Test that function calling works (functionCall in response)."""
print("\n=== Test 3: Function Calling ===")
url = f"https://generativelanguage.googleapis.com/v1beta/models/gemini-2.0-flash:generateContent?key={api_key}"
# Request with function declaration
payload = {
"contents": [{"role": "user", "parts": [{"text": "What's the weather in New York?"}]}],
"tools": [
{
"functionDeclarations": [
{
"name": "get_weather",
"description": "Get the current weather for a location",
"parameters": {
"type": "object",
"properties": {
"location": {"type": "string", "description": "The city name"}
},
"required": ["location"],
},
}
]
}
],
}
async with httpx.AsyncClient() as client:
response = await client.post(url, json=payload, timeout=30)
print(f"Status: {response.status_code}")
if response.status_code == 200:
data = response.json()
parts = data.get("candidates", [{}])[0].get("content", {}).get("parts", [])
# Check if model made a function call
has_function_call = any("functionCall" in part for part in parts)
if has_function_call:
func_call = next(p["functionCall"] for p in parts if "functionCall" in p)
print(f"Function called: {func_call.get('name')} with args: {func_call.get('args')}")
print("✅ Function calling works")
return True
else:
# Model might have answered directly
text = parts[0].get("text", "") if parts else ""
print(f"Model responded with text instead: {text[:100]}")
print("⚠️ Model didn't use function call (acceptable)")
return True
else:
print(f"Error: {response.text[:200]}")
return False
@pytest.mark.skipif(
not os.environ.get("GOOGLE_API_KEY"),
reason="GOOGLE_API_KEY not set - E2E tests require real API access",
)
@pytest.mark.asyncio
async def test_function_response_flow(api_key):
"""Test complete function call + response flow."""
print("\n=== Test 4: Function Call + Response Flow ===")
url = f"https://generativelanguage.googleapis.com/v1beta/models/gemini-2.0-flash:generateContent?key={api_key}"
# Multi-turn with function response
payload = {
"contents": [
{"role": "user", "parts": [{"text": "What's the weather in Tokyo?"}]},
{
"role": "model",
"parts": [{"functionCall": {"name": "get_weather", "args": {"location": "Tokyo"}}}],
},
{
"role": "user",
"parts": [
{
"functionResponse": {
"name": "get_weather",
"response": {"temperature": 22, "condition": "sunny", "humidity": 45},
}
}
],
},
],
"tools": [
{
"functionDeclarations": [
{
"name": "get_weather",
"description": "Get the current weather for a location",
"parameters": {
"type": "object",
"properties": {"location": {"type": "string"}},
},
}
]
}
],
}
async with httpx.AsyncClient() as client:
response = await client.post(url, json=payload, timeout=30)
print(f"Status: {response.status_code}")
if response.status_code == 200:
data = response.json()
text = (
data.get("candidates", [{}])[0].get("content", {}).get("parts", [{}])[0].get("text", "")
)
print(f"Response: {text[:150]}")
print("✅ Function response flow works - model used the function result")
return True
else:
print(f"Error: {response.text[:300]}")
return False
@pytest.mark.skipif(
not os.environ.get("GOOGLE_API_KEY"),
reason="GOOGLE_API_KEY not set - E2E tests require real API access",
)
@pytest.mark.asyncio
async def test_mixed_conversation(api_key):
"""Test a conversation mixing text and images."""
print("\n=== Test 5: Mixed Conversation (Text + Image) ===")
url = f"https://generativelanguage.googleapis.com/v1beta/models/gemini-2.0-flash:generateContent?key={api_key}"
payload = {
"contents": [
{"role": "user", "parts": [{"text": "I'll show you an image and ask about it."}]},
{
"role": "model",
"parts": [{"text": "Sure, please share the image and I'll help you with it."}],
},
{
"role": "user",
"parts": [
{"text": "Here it is. What color do you see?"},
{"inlineData": {"mimeType": "image/png", "data": TINY_RED_PNG}},
],
},
]
}
async with httpx.AsyncClient() as client:
response = await client.post(url, json=payload, timeout=30)
print(f"Status: {response.status_code}")
if response.status_code == 200:
data = response.json()
text = (
data.get("candidates", [{}])[0].get("content", {}).get("parts", [{}])[0].get("text", "")
)
print(f"Response: {text[:150]}")
print("✅ Mixed conversation works - model saw and processed the image")
return True
else:
print(f"Error: {response.text[:200]}")
return False
@pytest.mark.skipif(
not os.environ.get("GOOGLE_API_KEY"),
reason="GOOGLE_API_KEY not set - E2E tests require real API access",
)
@pytest.mark.asyncio
async def test_through_proxy(api_key, proxy_url: str = "http://localhost:8080"):
"""Test multimodal requests through the Headroom proxy."""
print(f"\n=== Test 6: Through Headroom Proxy ({proxy_url}) ===")
# The proxy expects requests at /v1beta/models/{model}:generateContent
url = f"{proxy_url}/v1beta/models/gemini-2.0-flash:generateContent"
payload = {
"contents": [
{
"role": "user",
"parts": [
{"text": "Describe this image in one word."},
{"inlineData": {"mimeType": "image/png", "data": TINY_RED_PNG}},
],
}
]
}
headers = {"x-goog-api-key": api_key, "Content-Type": "application/json"}
try:
async with httpx.AsyncClient() as client:
response = await client.post(url, json=payload, headers=headers, timeout=30)
print(f"Status: {response.status_code}")
if response.status_code == 200:
data = response.json()
text = (
data.get("candidates", [{}])[0]
.get("content", {})
.get("parts", [{}])[0]
.get("text", "")
)
print(f"Response: {text[:150]}")
print("✅ Proxy preserved the image and forwarded correctly!")
return True
else:
print(f"Error: {response.text[:300]}")
return False
except httpx.ConnectError:
print("⚠️ Proxy not running - skipping proxy test")
print(" To test through proxy, start it with: uv run headroom-proxy")
return None
async def main():
api_key = os.environ.get("GOOGLE_API_KEY")
if not api_key:
print("ERROR: GOOGLE_API_KEY environment variable not set")
print("Usage: GOOGLE_API_KEY=your_key python tests/test_google_multimodal_e2e.py")
return False
print("=" * 60)
print("Google Gemini Multimodal E2E Tests")
print("=" * 60)
print("Using API key from environment")
results = []
# Test 1: Pure text
results.append(("Text Only", await test_text_only_request(api_key)))
# Test 2: Image
results.append(("Image (inlineData)", await test_image_request(api_key)))
# Test 3: Function calling
results.append(("Function Calling", await test_function_calling(api_key)))
# Test 4: Function response
results.append(("Function Response Flow", await test_function_response_flow(api_key)))
# Test 5: Mixed conversation
results.append(("Mixed Conversation", await test_mixed_conversation(api_key)))
# Test 6: Through proxy (if running)
proxy_result = await test_through_proxy(api_key)
if proxy_result is not None:
results.append(("Through Proxy", proxy_result))
print("\n" + "=" * 60)
print("SUMMARY")
print("=" * 60)
passed = sum(1 for _, r in results if r)
total = len(results)
for name, result in results:
status = "✅ PASS" if result else "❌ FAIL"
print(f" {name}: {status}")
print(f"\nTotal: {passed}/{total} passed")
return passed == total
if __name__ == "__main__":
success = asyncio.run(main())
exit(0 if success else 1)