Files
headroom/tests/test_reporting.py
Tejas ChopraandClaude Opus 5.5 dea9092e19 fix(security): resolve high-severity CodeQL alerts (#4005)
## Description

Resolves the high-severity CodeQL alerts in product code. Three are real
leaks (an API key in the proxy log, a credential in an oauth2 warning,
XSS in the HTML report); the rest are hardening.

Closes #

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- **#63, Gemini API key written to the proxy log.** Google forwarders
build `...?key=<api key>` URLs (`batch.py:442` and others) and
`log_outbound_request` logged the full URL at INFO. The log line now
drops the query string, fragment and userinfo, keeping scheme, host
(IPv6 literals stay bracketed) and path. A URL with an invalid port logs
`<unparseable>` rather than raising.
- **#104, credential in an oauth2 warning.** A static header written as
`Authorization: Bearer x=y` splits on the first `=`, so `Authorization:
Bearer x` lands in the key, is rejected as malformed, and was echoed by
the warning. The warning no longer prints the key.
- **#131, XSS in the HTML report.** `model` and other request fields are
client supplied. Jinja2 autoescape is now on; no report field relies on
raw HTML.
- **#127:** `SNOWFLAKE_HOST` must end with `.snowflakecomputing.com`
instead of merely containing it.
- **#19, #128, #129, #130:** the `/\/+$/` trailing-slash regex in the TS
SDK and opencode plugin backtracks quadratically on long runs of `/`.
Replaced with a loop. The committed wheel bundle
(`headroom/providers/opencode/_dist/entry.opencode.js`) is rebuilt to
match.

Not changed (false positives, to dismiss): **#105** (URL already reduced
to `scheme://host` by `_redact_url`) and **#109** (`token_fingerprint`
is a short SHA-256 fingerprint of a high-entropy token for comparison,
not password storage).

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

New tests: outbound log keeps no `key=` value or userinfo, keeps IPv6
brackets, and handles relative paths and invalid ports; the report
escapes `<script>`; the Snowflake host must end with the domain; both TS
trailing-slash helpers are tested on normal inputs and on a 100k-slash
input (the old regex took ~5.2 s on it; the test bound is 1 s).

### Test Output

```text
$ HEADROOM_REQUIRE_RUST_CORE=false pytest -q tests/test_proxy_byte_faithful_forwarding.py tests/test_reporting.py tests/test_provider_cortex_code.py
105 passed in 4.65s

$ HEADROOM_REQUIRE_RUST_CORE=false pytest -q plugins/headroom-oauth2/tests
70 passed

$ ruff check headroom/proxy/helpers.py tests/test_proxy_byte_faithful_forwarding.py && ruff format --check ...
All checks passed!
2 files already formatted

$ (cd plugins/opencode && npm run typecheck && npx vitest run)
Test Files  3 passed (3)
     Tests  38 passed (38)

$ (cd sdk/typescript && npx vitest run)
Test Files  19 passed | 5 skipped (24)
     Tests  351 passed | 33 skipped (384)

$ (cd plugins/opencode && npm run build:standalone) && cmp dist-standalone/entry.opencode.js ../../headroom/providers/opencode/_dist/entry.opencode.js
(no output: bundle matches source)
```

The two oauth2 tests that start the full proxy need the Rust core, which
was not built in the local worktree, so they ran with
`HEADROOM_REQUIRE_RUST_CORE=false`.

## Real Behavior Proof

- Environment: macOS 26 (Darwin 25.6), Python 3.12 venv, this branch at
`bb634a3b`, `HEADROOM_REQUIRE_RUST_CORE=false`. A local fake Gemini
upstream (Python `http.server`) listening on IPv6 `[::1]:9911`.
- Exact command / steps:
1. `python fake_gemini.py` (accepts POST, logs the request path, returns
`{"batch":{"name":"batches/fake"}}`)
  2. `headroom proxy --port 8799 --gemini-api-url 'http://[::1]:9911'`
3. `curl -X POST
'http://127.0.0.1:8799/v1beta/models/gemini-2.5-pro:batchGenerateContent'
-H 'x-goog-api-key: AIzaFAKE-RUNTIME-PROOF-123' -H 'content-type:
application/json' -d
'{"batch":{"displayName":"proof","inputConfig":{"requests":{"requests":[]}}}}'`
  4. `grep outbound_request proxy.log; grep -c AIzaFAKE proxy.log`
- Observed result: the client got `HTTP 200`. The upstream still
received the key
(`/v1beta/models/gemini-2.5-pro:batchGenerateContent?key=AIzaFAKE-RUNTIME-PROOF-123`),
so forwarding is unchanged. The proxy log line was
`event=outbound_request forwarder=google_batch_passthrough method=POST
path=http://[::1]:9911/v1beta/models/gemini-2.5-pro:batchGenerateContent
body_bytes=76 ...`, and the key appeared in the proxy log **0** times.
- Not tested: real Google, Snowflake or oauth2 provider endpoints; the
HTML report was exercised only through unit tests; the opencode plugin
was not loaded into a live opencode session; Windows.

## Runtime Rollout Safety

- Rollout-managed feature(s): None. No rollout-managed feature or flag
is touched.
- Minimum rollout channel: Stable. These are security fixes with no
opt-in behaviour.
- Stable/default behavior changed: Only diagnostics and validation: the
`outbound_request` log line no longer includes the query string,
fragment or userinfo; the oauth2 malformed-header warning no longer
prints the key; HTML report fields are escaped; a `SNOWFLAKE_HOST` that
merely contains `.snowflakecomputing.com` is now rejected. Request
forwarding is byte-for-byte unchanged.
- Kill switch / disable path: None needed; nothing new is enabled.
Reverting the PR restores the previous behaviour.
- Unsafe override required: No.
- Qualification impact: None. No compression, routing or
token-accounting path changes.
- Rollback path: Revert this PR.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

Possible overlap: #3999 also replaces the opencode trailing-slash regex.
Whichever lands second will need a small rebase.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 23:11:40 -05:00

373 lines
13 KiB
Python

from __future__ import annotations
import builtins
import sys
from dataclasses import dataclass
from datetime import datetime, timedelta
from types import SimpleNamespace
import pytest
import headroom.reporting as reporting
from headroom.models.registry import ModelRegistry
from headroom.reporting import generator
from headroom.utils import format_cost
@dataclass
class FakeMetrics:
request_id: str
model: str
mode: str
timestamp: datetime
tokens_input_before: int
tokens_input_after: int
cache_alignment_score: float
waste_signals: dict[str, int]
class FakeStorage:
def __init__(self, stats: dict, items: list[FakeMetrics]) -> None:
self._stats = stats
self._items = items
self.closed = False
def get_summary_stats(self, start_time, end_time):
return dict(self._stats)
def iter_all(self):
return iter(self._items)
def close(self) -> None:
self.closed = True
def test_reporting_public_export() -> None:
assert reporting.generate_report is generator.generate_report
assert reporting.__all__ == ["generate_report"]
def test_get_jinja2_template_success_with_stub(monkeypatch) -> None:
class FakeTemplate:
def __init__(self, template_str: str, **options) -> None:
self.template_str = template_str
self.options = options
def render(self, **kwargs) -> str:
return f"{self.template_str}:{kwargs['name']}"
monkeypatch.setitem(sys.modules, "jinja2", SimpleNamespace(Template=FakeTemplate))
template = generator._get_jinja2_template("hello")
assert template.render(name="world") == "hello:world"
assert template.options == {"autoescape": True}
def test_report_template_escapes_client_supplied_fields() -> None:
pytest.importorskip("jinja2")
template = generator._get_jinja2_template("<td>{{ model }}</td>")
html = template.render(model="<script>alert(1)</script>")
assert "<script>" not in html
assert "&lt;script&gt;" in html
def test_get_jinja2_template_raises_helpful_error(monkeypatch) -> None:
real_import = builtins.__import__
def fake_import(name, globals=None, locals=None, fromlist=(), level=0):
if name == "jinja2":
raise ImportError("missing")
return real_import(name, globals, locals, fromlist, level)
monkeypatch.setattr(builtins, "__import__", fake_import)
with pytest.raises(ImportError, match="jinja2 is required for report generation"):
generator._get_jinja2_template("ignored")
def test_build_waste_histogram_empty_and_filtered_data() -> None:
now = datetime(2026, 4, 23, 12, 0, 0)
metrics = [
FakeMetrics(
request_id="before",
model="gpt-4o",
mode="audit",
timestamp=now - timedelta(days=2),
tokens_input_before=100,
tokens_input_after=90,
cache_alignment_score=10,
waste_signals={"json_bloat": 5},
),
FakeMetrics(
request_id="inside",
model="gpt-4o",
mode="optimize",
timestamp=now,
tokens_input_before=200,
tokens_input_after=100,
cache_alignment_score=70,
waste_signals={"json_bloat": 30, "html_noise": 10, "dynamic_date": 5, "reread": 20},
),
FakeMetrics(
request_id="flat",
model="gpt-4o",
mode="audit",
timestamp=now,
tokens_input_before=50,
tokens_input_after=50,
cache_alignment_score=50,
waste_signals={"whitespace": 4},
),
FakeMetrics(
request_id="after",
model="gpt-4o",
mode="audit",
timestamp=now + timedelta(days=2),
tokens_input_before=100,
tokens_input_after=20,
cache_alignment_score=20,
waste_signals={"base64": 50},
),
]
histogram = generator._build_waste_histogram(
FakeStorage({}, metrics),
start_time=now - timedelta(hours=1),
end_time=now + timedelta(hours=1),
)
assert histogram[0] == {"label": "History Bloat", "tokens": 55, "percentage": 100.0}
assert histogram[1] == pytest.approx(
{"label": "Tool JSON Bloat", "tokens": 30, "percentage": 54.54545454545454}
)
# "reread" surfaces in the histogram but is excluded from known_waste,
# so History Bloat above stays 100 - 45 = 55.
assert any(
item["label"] == "Re-served Tool Results" and item["tokens"] == 20 for item in histogram
)
assert any(item["label"] == "HTML Noise" and item["tokens"] == 10 for item in histogram)
assert any(item["label"] == "Dynamic Dates" and item["tokens"] == 5 for item in histogram)
assert any(item["label"] == "Base64 Blobs" and item["tokens"] == 0 for item in histogram)
empty = generator._build_waste_histogram(FakeStorage({}, []), None, None)
assert all(item["tokens"] == 0 and item["percentage"] == 0 for item in empty)
def test_get_top_waste_requests_sorts_filters_and_limits() -> None:
now = datetime(2026, 4, 23, 12, 0, 0)
metrics = [
FakeMetrics("one", "gpt-4o", "audit", now, 400, 100, 80, {}),
FakeMetrics("two", "gpt-4o-mini", "optimize", now, 350, 330, 70, {}),
FakeMetrics("three", "claude", "audit", now - timedelta(days=3), 1000, 10, 50, {}),
FakeMetrics("four", "claude", "audit", now + timedelta(days=3), 1000, 200, 40, {}),
]
top_requests = generator._get_top_waste_requests(
FakeStorage({}, metrics),
start_time=now - timedelta(hours=1),
end_time=now + timedelta(hours=1),
limit=1,
)
assert top_requests == [
{
"request_id": "one",
"model": "gpt-4o",
"mode": "audit",
"tokens_before": 400,
"tokens_saved": 300,
"cache_alignment": 80,
}
]
def test_generate_recommendations_for_heavy_waste_and_for_getting_started() -> None:
stats = {
"avg_cache_alignment": 40,
"audit_count": 7,
"optimize_count": 3,
"total_tokens_saved": 120000,
"estimated_savings": "$1.23",
}
histogram = [
{"label": "Tool JSON Bloat", "tokens": 15000, "percentage": 100},
{"label": "History Bloat", "tokens": 60000, "percentage": 50},
]
recommendations = generator._generate_recommendations(stats, histogram, top_requests=[{}])
titles = [item["title"] for item in recommendations]
assert titles == [
"Improve Cache Alignment",
"Enable Tool Output Compression",
"Review Rolling Window Settings",
"Switch to Optimize Mode",
"Continue Monitoring",
]
assert "15,000" in recommendations[1]["description"]
assert "60,000" in recommendations[2]["description"]
starter = generator._generate_recommendations(
{
"avg_cache_alignment": 90,
"audit_count": 1,
"optimize_count": 1,
"total_tokens_saved": 0,
"estimated_savings": "$0.00",
},
[{"label": "Tool JSON Bloat", "tokens": 1, "percentage": 100}],
top_requests=[],
)
assert starter == [
{
"title": "Get Started",
"description": "No optimizations applied yet. Try setting headroom_mode='optimize' "
"on your next request to start seeing token savings.",
}
]
@pytest.mark.parametrize(
("start_time", "end_time", "expected_period"),
[
(
datetime(2026, 4, 20, 8, 0, 0),
datetime(2026, 4, 23, 18, 0, 0),
"2026-04-20 to 2026-04-23",
),
(datetime(2026, 4, 20, 8, 0, 0), None, "Since 2026-04-20"),
(None, datetime(2026, 4, 23, 18, 0, 0), "Until 2026-04-23"),
(None, None, "All time"),
],
)
def test_generate_report_writes_output_and_closes_storage(
monkeypatch, tmp_path, start_time, end_time, expected_period
) -> None:
storage = FakeStorage(
{
"total_requests": 3,
"total_tokens_saved": 50,
"avg_tokens_saved": 16.6,
"total_tokens_before": 100,
"total_tokens_after": 0,
"avg_cache_alignment": 82,
"audit_count": 1,
"optimize_count": 2,
},
[],
)
render_calls: list[dict] = []
class FakeTemplate:
def render(self, **kwargs) -> str:
render_calls.append(kwargs)
return "<html>report</html>"
monkeypatch.setattr(generator, "create_storage", lambda store_url: storage)
monkeypatch.setattr(
generator, "_build_waste_histogram", lambda *args: [{"label": "x", "tokens": 1}]
)
monkeypatch.setattr(
generator, "_get_top_waste_requests", lambda *args, **kwargs: [{"request_id": "abc"}]
)
monkeypatch.setattr(
generator, "_generate_recommendations", lambda *args: [{"title": "Keep going"}]
)
monkeypatch.setattr(generator, "_get_jinja2_template", lambda template_str: FakeTemplate())
output_path = tmp_path / "report.html"
result = generator.generate_report(
"sqlite:///demo.db",
output_path=str(output_path),
start_time=start_time,
end_time=end_time,
)
assert result == str(output_path)
assert output_path.read_text() == "<html>report</html>"
assert render_calls[0]["period"] == expected_period
assert render_calls[0]["stats"]["tpm_multiplier"] == 100.0
assert storage.closed is True
def _capture_stats(monkeypatch, tmp_path, stats: dict) -> dict:
"""Run generate_report() over *stats* and hand back the rendered stats dict.
Only the storage and template seams are patched: pricing stays the product's
own, so a report that cannot price itself shows up here as a wrong number.
"""
render_calls: list[dict] = []
class FakeTemplate:
def render(self, **kwargs) -> str:
render_calls.append(kwargs)
return "<html>report</html>"
monkeypatch.setattr(generator, "create_storage", lambda store_url: FakeStorage(stats, []))
monkeypatch.setattr(generator, "_build_waste_histogram", lambda *args: [])
monkeypatch.setattr(generator, "_get_top_waste_requests", lambda *args, **kwargs: [])
monkeypatch.setattr(generator, "_generate_recommendations", lambda *args: [])
monkeypatch.setattr(generator, "_get_jinja2_template", lambda template_str: FakeTemplate())
generator.generate_report("sqlite:///demo.db", output_path=str(tmp_path / "report.html"))
return render_calls[0]["stats"]
SAVINGS_STATS = {
"total_requests": 2,
"total_tokens_saved": 245_000,
"avg_tokens_saved": 122_500,
"total_tokens_before": 350_000,
"total_tokens_after": 105_000,
"avg_cache_alignment": 80,
"audit_count": 0,
"optimize_count": 2,
}
def test_generate_report_prices_the_savings_tile(monkeypatch, tmp_path) -> None:
"""245k tokens saved must cost something, not read as a $0.0000 claim."""
stats = _capture_stats(monkeypatch, tmp_path, dict(SAVINGS_STATS))
before = ModelRegistry.estimate_cost("gpt-4o", 350_000, 0)
after = ModelRegistry.estimate_cost("gpt-4o", 105_000, 0)
assert before is not None and after is not None, "gpt-4o left the bundled rate card"
assert stats["estimated_savings"] == format_cost(before - after)
assert stats["estimated_savings"] != "$0.00"
def test_generate_report_says_unavailable_when_it_cannot_price(monkeypatch, tmp_path) -> None:
"""An unknown price is a missing number, not zero saved."""
monkeypatch.setattr(
ModelRegistry,
"estimate_cost",
classmethod(lambda cls, model, *args, **kwargs: None),
)
stats = _capture_stats(monkeypatch, tmp_path, dict(SAVINGS_STATS))
assert stats["estimated_savings"] == "unavailable"
def test_generate_report_closes_storage_when_render_fails(monkeypatch, tmp_path) -> None:
storage = FakeStorage(
{
"total_requests": 0,
"total_tokens_saved": 0,
"avg_tokens_saved": 0,
"total_tokens_before": 0,
"total_tokens_after": 0,
"avg_cache_alignment": 0,
"audit_count": 0,
"optimize_count": 0,
},
[],
)
class FakeTemplate:
def render(self, **kwargs) -> str:
raise RuntimeError("boom")
monkeypatch.setattr(generator, "create_storage", lambda store_url: storage)
monkeypatch.setattr(generator, "_build_waste_histogram", lambda *args: [])
monkeypatch.setattr(generator, "_get_top_waste_requests", lambda *args, **kwargs: [])
monkeypatch.setattr(generator, "_generate_recommendations", lambda *args: [])
monkeypatch.setattr(generator, "_get_jinja2_template", lambda template_str: FakeTemplate())
with pytest.raises(RuntimeError, match="boom"):
generator.generate_report("sqlite:///demo.db", output_path=str(tmp_path / "report.html"))
assert storage.closed is True