Commit Graph
5 Commits
Author SHA1 Message Date
Tejas ChopraandClaude Opus 5.5 dea9092e19 fix(security): resolve high-severity CodeQL alerts (#4005)
## Description

Resolves the high-severity CodeQL alerts in product code. Three are real
leaks (an API key in the proxy log, a credential in an oauth2 warning,
XSS in the HTML report); the rest are hardening.

Closes #

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- **#63, Gemini API key written to the proxy log.** Google forwarders
build `...?key=<api key>` URLs (`batch.py:442` and others) and
`log_outbound_request` logged the full URL at INFO. The log line now
drops the query string, fragment and userinfo, keeping scheme, host
(IPv6 literals stay bracketed) and path. A URL with an invalid port logs
`<unparseable>` rather than raising.
- **#104, credential in an oauth2 warning.** A static header written as
`Authorization: Bearer x=y` splits on the first `=`, so `Authorization:
Bearer x` lands in the key, is rejected as malformed, and was echoed by
the warning. The warning no longer prints the key.
- **#131, XSS in the HTML report.** `model` and other request fields are
client supplied. Jinja2 autoescape is now on; no report field relies on
raw HTML.
- **#127:** `SNOWFLAKE_HOST` must end with `.snowflakecomputing.com`
instead of merely containing it.
- **#19, #128, #129, #130:** the `/\/+$/` trailing-slash regex in the TS
SDK and opencode plugin backtracks quadratically on long runs of `/`.
Replaced with a loop. The committed wheel bundle
(`headroom/providers/opencode/_dist/entry.opencode.js`) is rebuilt to
match.

Not changed (false positives, to dismiss): **#105** (URL already reduced
to `scheme://host` by `_redact_url`) and **#109** (`token_fingerprint`
is a short SHA-256 fingerprint of a high-entropy token for comparison,
not password storage).

## Testing

- [x] Unit tests pass (`pytest`)
- [x] Linting passes (`ruff check .`)
- [ ] Type checking passes (`mypy headroom`)
- [x] New tests added for new functionality
- [x] Manual testing performed

New tests: outbound log keeps no `key=` value or userinfo, keeps IPv6
brackets, and handles relative paths and invalid ports; the report
escapes `<script>`; the Snowflake host must end with the domain; both TS
trailing-slash helpers are tested on normal inputs and on a 100k-slash
input (the old regex took ~5.2 s on it; the test bound is 1 s).

### Test Output

```text
$ HEADROOM_REQUIRE_RUST_CORE=false pytest -q tests/test_proxy_byte_faithful_forwarding.py tests/test_reporting.py tests/test_provider_cortex_code.py
105 passed in 4.65s

$ HEADROOM_REQUIRE_RUST_CORE=false pytest -q plugins/headroom-oauth2/tests
70 passed

$ ruff check headroom/proxy/helpers.py tests/test_proxy_byte_faithful_forwarding.py && ruff format --check ...
All checks passed!
2 files already formatted

$ (cd plugins/opencode && npm run typecheck && npx vitest run)
Test Files  3 passed (3)
     Tests  38 passed (38)

$ (cd sdk/typescript && npx vitest run)
Test Files  19 passed | 5 skipped (24)
     Tests  351 passed | 33 skipped (384)

$ (cd plugins/opencode && npm run build:standalone) && cmp dist-standalone/entry.opencode.js ../../headroom/providers/opencode/_dist/entry.opencode.js
(no output: bundle matches source)
```

The two oauth2 tests that start the full proxy need the Rust core, which
was not built in the local worktree, so they ran with
`HEADROOM_REQUIRE_RUST_CORE=false`.

## Real Behavior Proof

- Environment: macOS 26 (Darwin 25.6), Python 3.12 venv, this branch at
`bb634a3b`, `HEADROOM_REQUIRE_RUST_CORE=false`. A local fake Gemini
upstream (Python `http.server`) listening on IPv6 `[::1]:9911`.
- Exact command / steps:
1. `python fake_gemini.py` (accepts POST, logs the request path, returns
`{"batch":{"name":"batches/fake"}}`)
  2. `headroom proxy --port 8799 --gemini-api-url 'http://[::1]:9911'`
3. `curl -X POST
'http://127.0.0.1:8799/v1beta/models/gemini-2.5-pro:batchGenerateContent'
-H 'x-goog-api-key: AIzaFAKE-RUNTIME-PROOF-123' -H 'content-type:
application/json' -d
'{"batch":{"displayName":"proof","inputConfig":{"requests":{"requests":[]}}}}'`
  4. `grep outbound_request proxy.log; grep -c AIzaFAKE proxy.log`
- Observed result: the client got `HTTP 200`. The upstream still
received the key
(`/v1beta/models/gemini-2.5-pro:batchGenerateContent?key=AIzaFAKE-RUNTIME-PROOF-123`),
so forwarding is unchanged. The proxy log line was
`event=outbound_request forwarder=google_batch_passthrough method=POST
path=http://[::1]:9911/v1beta/models/gemini-2.5-pro:batchGenerateContent
body_bytes=76 ...`, and the key appeared in the proxy log **0** times.
- Not tested: real Google, Snowflake or oauth2 provider endpoints; the
HTML report was exercised only through unit tests; the opencode plugin
was not loaded into a live opencode session; Windows.

## Runtime Rollout Safety

- Rollout-managed feature(s): None. No rollout-managed feature or flag
is touched.
- Minimum rollout channel: Stable. These are security fixes with no
opt-in behaviour.
- Stable/default behavior changed: Only diagnostics and validation: the
`outbound_request` log line no longer includes the query string,
fragment or userinfo; the oauth2 malformed-header warning no longer
prints the key; HTML report fields are escaped; a `SNOWFLAKE_HOST` that
merely contains `.snowflakecomputing.com` is now rejected. Request
forwarding is byte-for-byte unchanged.
- Kill switch / disable path: None needed; nothing new is enabled.
Reverting the PR restores the previous behaviour.
- Unsafe override required: No.
- Qualification impact: None. No compression, routing or
token-accounting path changes.
- Rollback path: Revert this PR.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

Possible overlap: #3999 also replaces the opencode trailing-slash regex.
Whichever lands second will need a small rebase.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 23:11:40 -05:00
Aashish MahatoandJerrettDavis f519fa8b97 fix(reporting): price the savings tile instead of fabricating $0.00 (#3821)
## Description

`generate_report()` priced nothing and reported the absence as a number.
It called `headroom.utils.estimate_cost()` without a `provider`; that
function returns `None` before doing any work
(`headroom/utils.py:238-239`), and the call site swallowed the `None`
with `or 0.0`:

```python
cost_before = estimate_cost(stats["total_tokens_before"], 0, "gpt-4o") or 0.0
cost_after = estimate_cost(stats["total_tokens_after"], 0, "gpt-4o") or 0.0
estimated_savings = format_cost(cost_before - cost_after)
```

Both sides are therefore always `0.0`, so the **Est. Cost Savings** tile
read `$0.0000` on every report the function ever produced — verified on
the commit that shipped as 0.39.1. This is a fabricated fact, not a
missing value: a report saying "you saved $0" while its neighbouring
tile says "245,000 tokens saved" is the same failure #3744 just fixed
for negative savings.

The fix prices through `ModelRegistry.estimate_cost()` — the repo's own
LiteLLM-database pricer, which needs no provider instance and no
credentials — and when the rate card genuinely has no entry it renders
`unavailable` instead of `$0.0000`.

Closes #3820

## Type of Change

- [x] Bug fix (non-breaking change that fixes an issue)
- [ ] New feature (non-breaking change that adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Documentation update
- [ ] Performance improvement
- [ ] Code refactoring (no functional changes)

## Changes Made

- `headroom/reporting/generator.py`: price the two totals through
`ModelRegistry.estimate_cost("gpt-4o", tokens, 0)`; report `unavailable`
when a price is unobtainable instead of coercing it to `0.0`. The pricer
is imported inside `generate_report()` — imported at module level it
costs **1.19 s** and pulls `litellm` into a module that currently
imports in **0.071 s** with `litellm` unloaded (both numbers measured
below), which would tax every importer of `headroom.reporting` for a
value only the HTML report needs.
- `tests/test_reporting.py`: two new tests that leave pricing unpatched
(the report is priced against the real bundled rate card), one for the
unpriceable path, plus removal of the two stubs that hid the bug.
`test_generate_report_writes_output_and_closes_storage` patched
`generator.estimate_cost` with `lambda tokens, output_tokens, model:
{100: 2.0, 0: None}[tokens]` — a stub that answers the exact
3-positional call shape the real function rejects, and whose `$2.00` the
test then asserted. That is why 11 green tests coexisted with a
permanently wrong tile.

## Testing

- [x] Unit tests pass (`pytest`) — for the suites below; see *Not
tested* for the local limit
- [x] Linting passes (`ruff check .`)
- [ ] Type checking: `mypy headroom` is already red on `main` (see
*Gates*) — my file is clean and I add no errors
- [x] New tests added for new functionality
- [x] Manual testing performed

### Test Output

Red before the fix — the two new tests, everything else untouched, run
against `main` at `d13e196` with only the test file changed:

```text
tests/test_reporting.py ..........FF.                                    [100%]

_________________ test_generate_report_prices_the_savings_tile _________________
    assert stats["estimated_savings"] == format_cost(before - after)
E   AssertionError: assert '$0.0000' == '$0.61'
__________ test_generate_report_says_unavailable_when_it_cannot_price __________
    assert stats["estimated_savings"] == "unavailable"
E   AssertionError: assert '$0.0000' == 'unavailable'
========================= 2 failed, 11 passed in 1.51s =========================
```

Green after:

```text
tests/test_reporting.py .............                                    [100%]
============================== 13 passed in 1.40s ==============================
```

Control suites around the changed seams (`test_reporting`, `test_utils`,
`test_pricing`, `test_pricing_litellm`, `test_models`), same command on
the base worktree for the pre-existing subset:

```text
============================== 69 passed in 6.21s ==============================
```

Mutation checks — each regression is caught by the test that owns it:

```text
# `or 0.0` swallow put back:
FAILED tests/test_reporting.py::test_generate_report_says_unavailable_when_it_cannot_price
E   AssertionError: assert '$0.0000' == 'unavailable'
========================= 1 failed, 12 passed in 6.84s =========================

# None branch dropped:
FAILED tests/test_reporting.py::test_generate_report_says_unavailable_when_it_cannot_price
E   TypeError: unsupported operand type(s) for -: 'NoneType' and 'NoneType'
========================= 1 failed, 12 passed in 6.50s =========================
```

Gates:

```text
$ .venv/bin/python -m ruff check .
All checks passed!
$ .venv/bin/python -m ruff format --check .
1658 files already formatted
$ .venv/bin/python -m mypy headroom
headroom/integrations/langchain/chat_model.py:394: error: Function "RunnableBinding" could always be true in boolean context  [truthy-function]
Found 1 error in 1 file (checked 543 source files)
$ .venv/bin/python -m mypy headroom/reporting/generator.py
Success: no issues found in 1 source file
```

The repo-wide `mypy headroom` line is **not** caused by this change: the
same command on an untouched `main` worktree at `d13e196` produces the
identical single error in
`headroom/integrations/langchain/chat_model.py`, a file this PR does not
touch. I did not fix it here — it belongs to whichever change owns that
adapter, and folding it in would widen a two-file diff.

Import-cost numbers behind keeping the pricer inside the function:

```text
import headroom.reporting: 0.071s litellm_loaded=False   # before, and after this patch
from headroom.models.registry import ModelRegistry: 1.192s litellm_loaded=True
```

**Not tested locally:** the full `pytest tests`. 211 test modules import
the compiled `headroom._core`, which I did not build (`uv sync --extra
dev --no-install-project`), and the collectible remainder is slow enough
to need CI's sharding — `tests/test_bundled_tools_savings.py` alone
takes 73.46 s on the base commit `d13e196`. I checked that file
separately and it passes; nothing in it touches `headroom.reporting`.

## Real Behavior Proof

- **Environment:** macOS 27 arm64, Python 3.14.4, source tree at `main`
= `d13e196` (`chore: release 0.39.1 (#3807)`, `_version` reports
`0.39.2-dev`), deps from `uv sync --extra dev --no-install-project`, no
provider credentials, no network calls in the path under test.
- **Exact command / steps:** write two real requests into a real SQLite
metrics DB through `create_storage()`, call the documented
`headroom.generate_report()` (it is exported from `headroom/__init__.py`
and in `docs/content/docs/api-reference.mdx`), then read the tile text
out of the generated HTML with a regex rather than looking at the page:

```python
storage = create_storage("sqlite:///tmp/r.db")
for i, (before, after) in enumerate([(200_000, 60_000), (150_000, 45_000)]):
    storage.save(RequestMetrics(
        request_id=f"req-{i}", timestamp=datetime(2026, 9, 27, 10, i),
        model="gpt-4o", stream=False, mode="optimize",
        tokens_input_before=before, tokens_input_after=after, tokens_output=1000))

html = open(generate_report("sqlite:///tmp/r.db", "/tmp/report.html")).read()
```

- **Observed result — before the patch** (this is the issue's
reproduction, run on `d13e196`):

```text
stored summary: {'total_requests': 2, 'total_tokens_before': 350000,
                 'total_tokens_after': 105000, 'total_tokens_saved': 245000}
report written: /tmp/hr-repro2/report.html
  'Tokens Saved'    tile -> 245,000
  'Est. Cost Savings' tile -> $0.0000
```

- **Observed result — after the patch**, same script, same DB contents,
only `headroom/reporting/generator.py` changed:

```text
stored summary: {'total_requests': 2, 'total_tokens_before': 350000,
                 'total_tokens_after': 105000, 'total_tokens_saved': 245000}
report written: /tmp/hr-repro2b/report.html
  'Tokens Saved'    tile -> 245,000
  'Est. Cost Savings' tile -> $0.61

  registry gpt-4o pricing: before=$0.8750 after=$0.2625 savings=$0.6125
```

`$0.61` is `format_cost(0.8750 - 0.2625)`, i.e. the tile now agrees with
the rate card the report already names, and the number moves when the
stored tokens move.
- **Not tested:** Windows; the `headroom-ai[proxy]` wheel's Rust path
(`headroom/reporting/` is pure Python with no Rust mirror — I ran it
from the source tree, not the wheel); opening the report in a browser
(the tile text was read from the generated HTML, not screenshotted);
what happens to this tile when a deployment's traffic is mostly a model
LiteLLM has no price for — that case is now labelled `unavailable` and I
have not seen it in real traffic.

## Runtime Rollout Safety

- Rollout-managed feature(s): none — `generate_report()` is a plain SDK
call, not behind a rollout flag.
- Minimum rollout channel: n/a.
- Stable/default behavior changed: yes, deliberately. Every report's
**Est. Cost Savings** tile changes from the constant `$0.0000` to a
computed price, or to `unavailable` when the rate card has no entry. No
other tile, no stored field, no on-disk format changes.
- Kill switch / disable path: none needed; reverting the single commit
restores the previous (incorrect) behaviour.
- Unsafe override required: none.
- Qualification impact: none known — no qualification asserts on this
string, and the two tests that did assert on it asserted on their own
stub's return value.
- Rollback path: `git revert` the commit; it touches two files and adds
no state.

## Review Readiness

- [x] I have performed a self-review
- [x] This PR is ready for human review

## Checklist

- [x] My code follows the project's style guidelines
- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] New and existing unit tests pass locally with my changes
- [x] I did **not** edit `CHANGELOG.md` — it is generated by
release-please from my Conventional Commit PR title (a CI guard enforces
this)

## Additional Notes

- **Docs left alone deliberately.**
`docs/content/docs/api-reference.mdx` documents `generate_report()`
without promising a dollar figure, so nothing becomes wrong here. I did
not add a line about the `unavailable` string — one word in a tile that
needs no legend felt like noise, but I'll document it if you'd rather it
be explicit.
- **The hardcoded `"gpt-4o"` rate card is untouched on purpose.** A
model-agnostic report pricing all savings at one reference model is a
real question, and it is a different change from "the tile always said
zero". Widening this PR would make the before/after harder to read.
- **`utils.estimate_cost()`'s `None` contract is left alone.** The
silent-swallow hazard exists for any future caller too, but
`headroom/utils.py` is being edited in the open by #3116, so I stayed
out of that file rather than create a conflict for them.
- Diff is 2 files, +72/−13. No dependency, lockfile, or generated file
is touched.

Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-09-28 23:12:53 -05:00
Focused InstabilityandAsh Rhodes 5f1d88ad27 feat: detect re-served tool results as over-compression waste signal (#854)
Closes #853

## What

Adds a `reread` waste signal: identical `tool_result` content appearing
at more than one message position means the agent re-fetched something
already in context — the dominant failure signature of over-compression
(Manus context-engineering; JetBrains "Complexity Trap",
arXiv:2508.21433). Per-request savings can't see this cost; this signal
makes it visible.

- `WasteSignals.reread_tokens` — new field, in `total()`, exported as
`"reread"` in `to_dict()`.
- `parse_messages()` groups `tool_result` blocks by their **existing**
`content_hash` and counts every repeat beyond the first serve. No new
hashing or tokenization; one O(blocks) dict pass.
- `REREAD_MIN_TOKENS = 50` guard: short outputs ("ok", empty diffs)
legitimately repeat and are skipped. Duplicates within a single message
(same `source_index`) are not counted.
- Works across all formats the parser already normalizes to
`tool_result` blocks: OpenAI `role=tool`, Anthropic `tool_result`,
Strands/Bedrock `toolResult` (#813/#815).
- Flows through existing generic plumbing with zero handler changes:
pipeline → `RequestOutcome.waste_signals` → Prometheus
`headroom_waste_signal_tokens_total{signal="reread"}` → dashboard "Waste
Detected" panel. Dashboard gains label/color entries for the new key.

## Tests

7 new tests in `tests/test_parser.py::TestRereadDetection` (red before,
green after): OpenAI + Anthropic format detection, repeat-counting
semantics (first serve free), single-occurrence, short-duplicate guard,
same-message guard, `total()`/`to_dict()` participation. Updated 2
exact-shape assertions in `tests/test_config.py`.

Local runs: `tests/test_parser.py` (72 passed), `tests/test_config.py` +
outcome/reporting/observability/storage/proxy-hooks suites (190 passed),
`tests/test_canonical_pipeline.py` +
`tests/test_proxy_pipeline_lifecycle.py` (11 passed). `ruff check` +
`ruff format --check` clean.

## Real behavior proof

**Setup:** macOS (Darwin 25.5), Python 3.11.9, this branch, real proxy
server (`python -m headroom.proxy.server --port 18970
--anthropic-api-url http://127.0.0.1:18971`) with a local mock Anthropic
upstream returning a canned `/v1/messages` response (no real key
needed).

**Steps:** POSTed an Anthropic-format conversation to the live proxy:
agent fetches a 14 KB JSON log array via `get_logs` tool, then fetches
the identical content again under a different `tool_use_id` (the
re-read).

**Observed result** — `curl http://127.0.0.1:18970/metrics` after the
request:

```
# HELP headroom_waste_signal_tokens_total Tokens attributed to detected waste signals
# TYPE headroom_waste_signal_tokens_total counter
headroom_waste_signal_tokens_total{signal="json_bloat"} 9858
headroom_waste_signal_tokens_total{signal="reread"} 4935
```

`reread` = 4935 tokens, exactly the second serve of the ~4.9k-token tool
result (json_bloat counts both occurrences ≈ 2×). `/stats` shows the
same: `"waste_signals": {"json_bloat": 9858, "reread": 4935, ...}` —
which is what the dashboard panel renders.

Also verified the negative path live: a conversation whose tool results
contain non-compressible plain code text produced no waste-signal
entries (the pipeline only attributes waste when compression actually
engaged, unchanged behavior).

**Not tested:** Gemini `functionResponse` path (parser doesn't produce
`tool_result` blocks for it — pre-existing gap tracked in #819);
dashboard rendering only verified via the `/stats` payload the panel
binds to, not a browser screenshot.

## Out of scope (per #853)

Tool-call argument matching, compression-marker attribution,
tokens-per-task metric, cache hit-rate panel.

---------

Co-authored-by: Ash Rhodes <ashley.rhodes@king.com>
2026-06-11 13:07:04 -05:00
GarmandClaude Opus 4.7 efd2ac1ca4 chore: renormalize line endings to LF
`.gitattributes` declares `*.py text eol=lf` and `*.sh text eol=lf`, but
74 files (73 .py, 1 .sh) are stored in the index with CRLF line endings,
violating that contract. Every macOS/Linux clone reports these files as
"modified" on fresh checkout because git's diff engine sees the stored
bytes don't match the attribute contract, even though the working tree
and index match byte-for-byte.

Running `git add --renormalize .` rewrites each affected blob so the
stored form matches the attribute declaration. No semantic changes —
every affected file's diff is "N insertions, N deletions" with inserts
and deletes being the same lines modulo line endings.

Follow-up commit adds `.git-blame-ignore-revs` so `git blame` / GitHub
blame skip this mechanical commit.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-24 15:33:30 +02:00
JerrettDavisandCopilot 38bf3e639c test: expand coverage across helper slices
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-04-23 07:39:52 -05:00