mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 18:41:07 +08:00
## Description Resolves the high-severity CodeQL alerts in product code. Three are real leaks (an API key in the proxy log, a credential in an oauth2 warning, XSS in the HTML report); the rest are hardening. Closes # ## Type of Change - [x] Bug fix (non-breaking change that fixes an issue) - [ ] New feature (non-breaking change that adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] Documentation update - [ ] Performance improvement - [ ] Code refactoring (no functional changes) ## Changes Made - **#63, Gemini API key written to the proxy log.** Google forwarders build `...?key=<api key>` URLs (`batch.py:442` and others) and `log_outbound_request` logged the full URL at INFO. The log line now drops the query string, fragment and userinfo, keeping scheme, host (IPv6 literals stay bracketed) and path. A URL with an invalid port logs `<unparseable>` rather than raising. - **#104, credential in an oauth2 warning.** A static header written as `Authorization: Bearer x=y` splits on the first `=`, so `Authorization: Bearer x` lands in the key, is rejected as malformed, and was echoed by the warning. The warning no longer prints the key. - **#131, XSS in the HTML report.** `model` and other request fields are client supplied. Jinja2 autoescape is now on; no report field relies on raw HTML. - **#127:** `SNOWFLAKE_HOST` must end with `.snowflakecomputing.com` instead of merely containing it. - **#19, #128, #129, #130:** the `/\/+$/` trailing-slash regex in the TS SDK and opencode plugin backtracks quadratically on long runs of `/`. Replaced with a loop. The committed wheel bundle (`headroom/providers/opencode/_dist/entry.opencode.js`) is rebuilt to match. Not changed (false positives, to dismiss): **#105** (URL already reduced to `scheme://host` by `_redact_url`) and **#109** (`token_fingerprint` is a short SHA-256 fingerprint of a high-entropy token for comparison, not password storage). ## Testing - [x] Unit tests pass (`pytest`) - [x] Linting passes (`ruff check .`) - [ ] Type checking passes (`mypy headroom`) - [x] New tests added for new functionality - [x] Manual testing performed New tests: outbound log keeps no `key=` value or userinfo, keeps IPv6 brackets, and handles relative paths and invalid ports; the report escapes `<script>`; the Snowflake host must end with the domain; both TS trailing-slash helpers are tested on normal inputs and on a 100k-slash input (the old regex took ~5.2 s on it; the test bound is 1 s). ### Test Output ```text $ HEADROOM_REQUIRE_RUST_CORE=false pytest -q tests/test_proxy_byte_faithful_forwarding.py tests/test_reporting.py tests/test_provider_cortex_code.py 105 passed in 4.65s $ HEADROOM_REQUIRE_RUST_CORE=false pytest -q plugins/headroom-oauth2/tests 70 passed $ ruff check headroom/proxy/helpers.py tests/test_proxy_byte_faithful_forwarding.py && ruff format --check ... All checks passed! 2 files already formatted $ (cd plugins/opencode && npm run typecheck && npx vitest run) Test Files 3 passed (3) Tests 38 passed (38) $ (cd sdk/typescript && npx vitest run) Test Files 19 passed | 5 skipped (24) Tests 351 passed | 33 skipped (384) $ (cd plugins/opencode && npm run build:standalone) && cmp dist-standalone/entry.opencode.js ../../headroom/providers/opencode/_dist/entry.opencode.js (no output: bundle matches source) ``` The two oauth2 tests that start the full proxy need the Rust core, which was not built in the local worktree, so they ran with `HEADROOM_REQUIRE_RUST_CORE=false`. ## Real Behavior Proof - Environment: macOS 26 (Darwin 25.6), Python 3.12 venv, this branch at `bb634a3b`, `HEADROOM_REQUIRE_RUST_CORE=false`. A local fake Gemini upstream (Python `http.server`) listening on IPv6 `[::1]:9911`. - Exact command / steps: 1. `python fake_gemini.py` (accepts POST, logs the request path, returns `{"batch":{"name":"batches/fake"}}`) 2. `headroom proxy --port 8799 --gemini-api-url 'http://[::1]:9911'` 3. `curl -X POST 'http://127.0.0.1:8799/v1beta/models/gemini-2.5-pro:batchGenerateContent' -H 'x-goog-api-key: AIzaFAKE-RUNTIME-PROOF-123' -H 'content-type: application/json' -d '{"batch":{"displayName":"proof","inputConfig":{"requests":{"requests":[]}}}}'` 4. `grep outbound_request proxy.log; grep -c AIzaFAKE proxy.log` - Observed result: the client got `HTTP 200`. The upstream still received the key (`/v1beta/models/gemini-2.5-pro:batchGenerateContent?key=AIzaFAKE-RUNTIME-PROOF-123`), so forwarding is unchanged. The proxy log line was `event=outbound_request forwarder=google_batch_passthrough method=POST path=http://[::1]:9911/v1beta/models/gemini-2.5-pro:batchGenerateContent body_bytes=76 ...`, and the key appeared in the proxy log **0** times. - Not tested: real Google, Snowflake or oauth2 provider endpoints; the HTML report was exercised only through unit tests; the opencode plugin was not loaded into a live opencode session; Windows. ## Runtime Rollout Safety - Rollout-managed feature(s): None. No rollout-managed feature or flag is touched. - Minimum rollout channel: Stable. These are security fixes with no opt-in behaviour. - Stable/default behavior changed: Only diagnostics and validation: the `outbound_request` log line no longer includes the query string, fragment or userinfo; the oauth2 malformed-header warning no longer prints the key; HTML report fields are escaped; a `SNOWFLAKE_HOST` that merely contains `.snowflakecomputing.com` is now rejected. Request forwarding is byte-for-byte unchanged. - Kill switch / disable path: None needed; nothing new is enabled. Reverting the PR restores the previous behaviour. - Unsafe override required: No. - Qualification impact: None. No compression, routing or token-accounting path changes. - Rollback path: Revert this PR. ## Review Readiness - [x] I have performed a self-review - [x] This PR is ready for human review ## Checklist - [x] My code follows the project's style guidelines - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] New and existing unit tests pass locally with my changes - [x] I did **not** edit `CHANGELOG.md` — it is generated by release-please from my Conventional Commit PR title (a CI guard enforces this) ## Additional Notes Possible overlap: #3999 also replaces the opencode trailing-slash regex. Whichever lands second will need a small rebase. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
373 lines
13 KiB
Python
373 lines
13 KiB
Python
from __future__ import annotations
|
|
|
|
import builtins
|
|
import sys
|
|
from dataclasses import dataclass
|
|
from datetime import datetime, timedelta
|
|
from types import SimpleNamespace
|
|
|
|
import pytest
|
|
|
|
import headroom.reporting as reporting
|
|
from headroom.models.registry import ModelRegistry
|
|
from headroom.reporting import generator
|
|
from headroom.utils import format_cost
|
|
|
|
|
|
@dataclass
|
|
class FakeMetrics:
|
|
request_id: str
|
|
model: str
|
|
mode: str
|
|
timestamp: datetime
|
|
tokens_input_before: int
|
|
tokens_input_after: int
|
|
cache_alignment_score: float
|
|
waste_signals: dict[str, int]
|
|
|
|
|
|
class FakeStorage:
|
|
def __init__(self, stats: dict, items: list[FakeMetrics]) -> None:
|
|
self._stats = stats
|
|
self._items = items
|
|
self.closed = False
|
|
|
|
def get_summary_stats(self, start_time, end_time):
|
|
return dict(self._stats)
|
|
|
|
def iter_all(self):
|
|
return iter(self._items)
|
|
|
|
def close(self) -> None:
|
|
self.closed = True
|
|
|
|
|
|
def test_reporting_public_export() -> None:
|
|
assert reporting.generate_report is generator.generate_report
|
|
assert reporting.__all__ == ["generate_report"]
|
|
|
|
|
|
def test_get_jinja2_template_success_with_stub(monkeypatch) -> None:
|
|
class FakeTemplate:
|
|
def __init__(self, template_str: str, **options) -> None:
|
|
self.template_str = template_str
|
|
self.options = options
|
|
|
|
def render(self, **kwargs) -> str:
|
|
return f"{self.template_str}:{kwargs['name']}"
|
|
|
|
monkeypatch.setitem(sys.modules, "jinja2", SimpleNamespace(Template=FakeTemplate))
|
|
template = generator._get_jinja2_template("hello")
|
|
assert template.render(name="world") == "hello:world"
|
|
assert template.options == {"autoescape": True}
|
|
|
|
|
|
def test_report_template_escapes_client_supplied_fields() -> None:
|
|
pytest.importorskip("jinja2")
|
|
template = generator._get_jinja2_template("<td>{{ model }}</td>")
|
|
html = template.render(model="<script>alert(1)</script>")
|
|
assert "<script>" not in html
|
|
assert "<script>" in html
|
|
|
|
|
|
def test_get_jinja2_template_raises_helpful_error(monkeypatch) -> None:
|
|
real_import = builtins.__import__
|
|
|
|
def fake_import(name, globals=None, locals=None, fromlist=(), level=0):
|
|
if name == "jinja2":
|
|
raise ImportError("missing")
|
|
return real_import(name, globals, locals, fromlist, level)
|
|
|
|
monkeypatch.setattr(builtins, "__import__", fake_import)
|
|
with pytest.raises(ImportError, match="jinja2 is required for report generation"):
|
|
generator._get_jinja2_template("ignored")
|
|
|
|
|
|
def test_build_waste_histogram_empty_and_filtered_data() -> None:
|
|
now = datetime(2026, 4, 23, 12, 0, 0)
|
|
metrics = [
|
|
FakeMetrics(
|
|
request_id="before",
|
|
model="gpt-4o",
|
|
mode="audit",
|
|
timestamp=now - timedelta(days=2),
|
|
tokens_input_before=100,
|
|
tokens_input_after=90,
|
|
cache_alignment_score=10,
|
|
waste_signals={"json_bloat": 5},
|
|
),
|
|
FakeMetrics(
|
|
request_id="inside",
|
|
model="gpt-4o",
|
|
mode="optimize",
|
|
timestamp=now,
|
|
tokens_input_before=200,
|
|
tokens_input_after=100,
|
|
cache_alignment_score=70,
|
|
waste_signals={"json_bloat": 30, "html_noise": 10, "dynamic_date": 5, "reread": 20},
|
|
),
|
|
FakeMetrics(
|
|
request_id="flat",
|
|
model="gpt-4o",
|
|
mode="audit",
|
|
timestamp=now,
|
|
tokens_input_before=50,
|
|
tokens_input_after=50,
|
|
cache_alignment_score=50,
|
|
waste_signals={"whitespace": 4},
|
|
),
|
|
FakeMetrics(
|
|
request_id="after",
|
|
model="gpt-4o",
|
|
mode="audit",
|
|
timestamp=now + timedelta(days=2),
|
|
tokens_input_before=100,
|
|
tokens_input_after=20,
|
|
cache_alignment_score=20,
|
|
waste_signals={"base64": 50},
|
|
),
|
|
]
|
|
histogram = generator._build_waste_histogram(
|
|
FakeStorage({}, metrics),
|
|
start_time=now - timedelta(hours=1),
|
|
end_time=now + timedelta(hours=1),
|
|
)
|
|
|
|
assert histogram[0] == {"label": "History Bloat", "tokens": 55, "percentage": 100.0}
|
|
assert histogram[1] == pytest.approx(
|
|
{"label": "Tool JSON Bloat", "tokens": 30, "percentage": 54.54545454545454}
|
|
)
|
|
# "reread" surfaces in the histogram but is excluded from known_waste,
|
|
# so History Bloat above stays 100 - 45 = 55.
|
|
assert any(
|
|
item["label"] == "Re-served Tool Results" and item["tokens"] == 20 for item in histogram
|
|
)
|
|
assert any(item["label"] == "HTML Noise" and item["tokens"] == 10 for item in histogram)
|
|
assert any(item["label"] == "Dynamic Dates" and item["tokens"] == 5 for item in histogram)
|
|
assert any(item["label"] == "Base64 Blobs" and item["tokens"] == 0 for item in histogram)
|
|
|
|
empty = generator._build_waste_histogram(FakeStorage({}, []), None, None)
|
|
assert all(item["tokens"] == 0 and item["percentage"] == 0 for item in empty)
|
|
|
|
|
|
def test_get_top_waste_requests_sorts_filters_and_limits() -> None:
|
|
now = datetime(2026, 4, 23, 12, 0, 0)
|
|
metrics = [
|
|
FakeMetrics("one", "gpt-4o", "audit", now, 400, 100, 80, {}),
|
|
FakeMetrics("two", "gpt-4o-mini", "optimize", now, 350, 330, 70, {}),
|
|
FakeMetrics("three", "claude", "audit", now - timedelta(days=3), 1000, 10, 50, {}),
|
|
FakeMetrics("four", "claude", "audit", now + timedelta(days=3), 1000, 200, 40, {}),
|
|
]
|
|
top_requests = generator._get_top_waste_requests(
|
|
FakeStorage({}, metrics),
|
|
start_time=now - timedelta(hours=1),
|
|
end_time=now + timedelta(hours=1),
|
|
limit=1,
|
|
)
|
|
assert top_requests == [
|
|
{
|
|
"request_id": "one",
|
|
"model": "gpt-4o",
|
|
"mode": "audit",
|
|
"tokens_before": 400,
|
|
"tokens_saved": 300,
|
|
"cache_alignment": 80,
|
|
}
|
|
]
|
|
|
|
|
|
def test_generate_recommendations_for_heavy_waste_and_for_getting_started() -> None:
|
|
stats = {
|
|
"avg_cache_alignment": 40,
|
|
"audit_count": 7,
|
|
"optimize_count": 3,
|
|
"total_tokens_saved": 120000,
|
|
"estimated_savings": "$1.23",
|
|
}
|
|
histogram = [
|
|
{"label": "Tool JSON Bloat", "tokens": 15000, "percentage": 100},
|
|
{"label": "History Bloat", "tokens": 60000, "percentage": 50},
|
|
]
|
|
recommendations = generator._generate_recommendations(stats, histogram, top_requests=[{}])
|
|
titles = [item["title"] for item in recommendations]
|
|
assert titles == [
|
|
"Improve Cache Alignment",
|
|
"Enable Tool Output Compression",
|
|
"Review Rolling Window Settings",
|
|
"Switch to Optimize Mode",
|
|
"Continue Monitoring",
|
|
]
|
|
assert "15,000" in recommendations[1]["description"]
|
|
assert "60,000" in recommendations[2]["description"]
|
|
|
|
starter = generator._generate_recommendations(
|
|
{
|
|
"avg_cache_alignment": 90,
|
|
"audit_count": 1,
|
|
"optimize_count": 1,
|
|
"total_tokens_saved": 0,
|
|
"estimated_savings": "$0.00",
|
|
},
|
|
[{"label": "Tool JSON Bloat", "tokens": 1, "percentage": 100}],
|
|
top_requests=[],
|
|
)
|
|
assert starter == [
|
|
{
|
|
"title": "Get Started",
|
|
"description": "No optimizations applied yet. Try setting headroom_mode='optimize' "
|
|
"on your next request to start seeing token savings.",
|
|
}
|
|
]
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
("start_time", "end_time", "expected_period"),
|
|
[
|
|
(
|
|
datetime(2026, 4, 20, 8, 0, 0),
|
|
datetime(2026, 4, 23, 18, 0, 0),
|
|
"2026-04-20 to 2026-04-23",
|
|
),
|
|
(datetime(2026, 4, 20, 8, 0, 0), None, "Since 2026-04-20"),
|
|
(None, datetime(2026, 4, 23, 18, 0, 0), "Until 2026-04-23"),
|
|
(None, None, "All time"),
|
|
],
|
|
)
|
|
def test_generate_report_writes_output_and_closes_storage(
|
|
monkeypatch, tmp_path, start_time, end_time, expected_period
|
|
) -> None:
|
|
storage = FakeStorage(
|
|
{
|
|
"total_requests": 3,
|
|
"total_tokens_saved": 50,
|
|
"avg_tokens_saved": 16.6,
|
|
"total_tokens_before": 100,
|
|
"total_tokens_after": 0,
|
|
"avg_cache_alignment": 82,
|
|
"audit_count": 1,
|
|
"optimize_count": 2,
|
|
},
|
|
[],
|
|
)
|
|
render_calls: list[dict] = []
|
|
|
|
class FakeTemplate:
|
|
def render(self, **kwargs) -> str:
|
|
render_calls.append(kwargs)
|
|
return "<html>report</html>"
|
|
|
|
monkeypatch.setattr(generator, "create_storage", lambda store_url: storage)
|
|
monkeypatch.setattr(
|
|
generator, "_build_waste_histogram", lambda *args: [{"label": "x", "tokens": 1}]
|
|
)
|
|
monkeypatch.setattr(
|
|
generator, "_get_top_waste_requests", lambda *args, **kwargs: [{"request_id": "abc"}]
|
|
)
|
|
monkeypatch.setattr(
|
|
generator, "_generate_recommendations", lambda *args: [{"title": "Keep going"}]
|
|
)
|
|
monkeypatch.setattr(generator, "_get_jinja2_template", lambda template_str: FakeTemplate())
|
|
|
|
output_path = tmp_path / "report.html"
|
|
result = generator.generate_report(
|
|
"sqlite:///demo.db",
|
|
output_path=str(output_path),
|
|
start_time=start_time,
|
|
end_time=end_time,
|
|
)
|
|
|
|
assert result == str(output_path)
|
|
assert output_path.read_text() == "<html>report</html>"
|
|
assert render_calls[0]["period"] == expected_period
|
|
assert render_calls[0]["stats"]["tpm_multiplier"] == 100.0
|
|
assert storage.closed is True
|
|
|
|
|
|
def _capture_stats(monkeypatch, tmp_path, stats: dict) -> dict:
|
|
"""Run generate_report() over *stats* and hand back the rendered stats dict.
|
|
|
|
Only the storage and template seams are patched: pricing stays the product's
|
|
own, so a report that cannot price itself shows up here as a wrong number.
|
|
"""
|
|
render_calls: list[dict] = []
|
|
|
|
class FakeTemplate:
|
|
def render(self, **kwargs) -> str:
|
|
render_calls.append(kwargs)
|
|
return "<html>report</html>"
|
|
|
|
monkeypatch.setattr(generator, "create_storage", lambda store_url: FakeStorage(stats, []))
|
|
monkeypatch.setattr(generator, "_build_waste_histogram", lambda *args: [])
|
|
monkeypatch.setattr(generator, "_get_top_waste_requests", lambda *args, **kwargs: [])
|
|
monkeypatch.setattr(generator, "_generate_recommendations", lambda *args: [])
|
|
monkeypatch.setattr(generator, "_get_jinja2_template", lambda template_str: FakeTemplate())
|
|
|
|
generator.generate_report("sqlite:///demo.db", output_path=str(tmp_path / "report.html"))
|
|
return render_calls[0]["stats"]
|
|
|
|
|
|
SAVINGS_STATS = {
|
|
"total_requests": 2,
|
|
"total_tokens_saved": 245_000,
|
|
"avg_tokens_saved": 122_500,
|
|
"total_tokens_before": 350_000,
|
|
"total_tokens_after": 105_000,
|
|
"avg_cache_alignment": 80,
|
|
"audit_count": 0,
|
|
"optimize_count": 2,
|
|
}
|
|
|
|
|
|
def test_generate_report_prices_the_savings_tile(monkeypatch, tmp_path) -> None:
|
|
"""245k tokens saved must cost something, not read as a $0.0000 claim."""
|
|
stats = _capture_stats(monkeypatch, tmp_path, dict(SAVINGS_STATS))
|
|
|
|
before = ModelRegistry.estimate_cost("gpt-4o", 350_000, 0)
|
|
after = ModelRegistry.estimate_cost("gpt-4o", 105_000, 0)
|
|
assert before is not None and after is not None, "gpt-4o left the bundled rate card"
|
|
assert stats["estimated_savings"] == format_cost(before - after)
|
|
assert stats["estimated_savings"] != "$0.00"
|
|
|
|
|
|
def test_generate_report_says_unavailable_when_it_cannot_price(monkeypatch, tmp_path) -> None:
|
|
"""An unknown price is a missing number, not zero saved."""
|
|
monkeypatch.setattr(
|
|
ModelRegistry,
|
|
"estimate_cost",
|
|
classmethod(lambda cls, model, *args, **kwargs: None),
|
|
)
|
|
|
|
stats = _capture_stats(monkeypatch, tmp_path, dict(SAVINGS_STATS))
|
|
|
|
assert stats["estimated_savings"] == "unavailable"
|
|
|
|
|
|
def test_generate_report_closes_storage_when_render_fails(monkeypatch, tmp_path) -> None:
|
|
storage = FakeStorage(
|
|
{
|
|
"total_requests": 0,
|
|
"total_tokens_saved": 0,
|
|
"avg_tokens_saved": 0,
|
|
"total_tokens_before": 0,
|
|
"total_tokens_after": 0,
|
|
"avg_cache_alignment": 0,
|
|
"audit_count": 0,
|
|
"optimize_count": 0,
|
|
},
|
|
[],
|
|
)
|
|
|
|
class FakeTemplate:
|
|
def render(self, **kwargs) -> str:
|
|
raise RuntimeError("boom")
|
|
|
|
monkeypatch.setattr(generator, "create_storage", lambda store_url: storage)
|
|
monkeypatch.setattr(generator, "_build_waste_histogram", lambda *args: [])
|
|
monkeypatch.setattr(generator, "_get_top_waste_requests", lambda *args, **kwargs: [])
|
|
monkeypatch.setattr(generator, "_generate_recommendations", lambda *args: [])
|
|
monkeypatch.setattr(generator, "_get_jinja2_template", lambda template_str: FakeTemplate())
|
|
|
|
with pytest.raises(RuntimeError, match="boom"):
|
|
generator.generate_report("sqlite:///demo.db", output_path=str(tmp_path / "report.html"))
|
|
assert storage.closed is True
|