Files
headroom/tests/test_codeql_triage_script.py
5698e91b2a chore(security): beacon rate limit, dev environment, weekly CodeQL triage (#4008)
SOC 2 follow-ups from the 2026-10-06 firewall review (R3, R4) and
incident INC-2026-001.

## Beacon rate limit (R3)
`deploy/beacon/wrangler.toml` said a WAF rule limited `/v1/logs` to 60
req/min/IP. That rule could not exist: headroomlabs.ai is not a
Cloudflare zone, so the Worker is only on `workers.dev`. This adds a
[Workers rate-limit
binding](https://developers.cloudflare.com/workers/runtime-apis/bindings/rate-limit/)
checked in `fetch()` **before** the body is read.

- Keyed on one constant, not the IP: the Worker still never reads
`cf-connecting-ip` (privacy promise in the file header). Trade-off: a
flood also sheds real events for that minute.
- 3000 req/min **per Cloudflare location** (~10x today's ~300/min
total).
- Returns 429, which the client does **not** treat as a gzip refusal
(`_GZIP_REFUSED_STATUSES` is 400/415), so rate-limited clients keep
compressing.
- Fails open if the binding is absent **or the limiter itself errors**
(review fix): a rate-limit service outage lets beacons through rather
than dropping all of them. `node deploy/beacon/test-rollup.mjs` covers
429 over the limit, fail-open on outage and when unbound, and the single
shared key.

## Dev environment (R4)
`[env.dev]` deploys `headroom-beacon-dev` bound to the new
`headroom-telemetry-dev` bucket (600/min limit). Already deployed and
tested: a sample event returns 204, and a 2,500-request burst got 384 ×
429.

## Weekly CodeQL triage
`.github/workflows/codeql-triage.yml` opens an issue every Monday,
assigned to @chopratejas (tejas@headroomlabs.ai), listing open
critical/high alerts with checkboxes. Alert #63 sat unowned for five
months; this gives alerts an owner and leaves an audit trail.

**One issue per week, however often it runs (review fix).** The logic is
in `scripts/codeql_triage.sh`:
- It keys the issue on the ISO week, using a hidden marker in the body.
- It finds the existing issue by listing security-labelled issues,
closed ones included. It uses a listing, not search, so an issue created
seconds earlier is not missed.
- It reuses that issue and never edits, reopens or retitles it, so
ticked boxes and closure stand. Alerts opened since then are added once,
as a comment.
- The jq programs live in quoted heredocs, which clears ShellCheck
SC2016 without any shell expansion.
- `tests/test_codeql_triage_script.py` runs the real script against a
fake `gh`. It covers same-day re-runs, a Friday dispatch on a closed
issue with ticked boxes, the next week, and the ISO year boundary
(2027-01-01 is in 2026-W53).

### Test output

```text
$ pytest -q tests/test_codeql_triage_script.py
6 passed
$ node deploy/beacon/test-rollup.mjs
rate limit: 429 over, fail-open on outage and when unbound
ok
$ shellcheck scripts/codeql_triage.sh && actionlint .github/workflows/codeql-triage.yml
(clean)
```

## ⚠️ Before deploying production
The live `headroom-beacon` Worker predates #3253 (gzip inflate, schema
v2 allowlist). `wrangler deploy` from main will ship that too, not just
this change.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4008"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-10-12 11:18:47 -07:00

223 lines
7.3 KiB
Python

"""scripts/codeql_triage.sh: one triage issue per ISO week, however often it runs.
The script runs for real against a fake ``gh`` on PATH that keeps alerts and
issues in a JSON file, so re-runs, later dispatches in the same week, a closed
issue with ticked boxes, and the ISO year boundary are all exercised end to end.
"""
from __future__ import annotations
import json
import os
import shutil
import subprocess
import sys
from pathlib import Path
import pytest
ROOT = Path(__file__).resolve().parents[1]
SCRIPT = ROOT / "scripts" / "codeql_triage.sh"
pytestmark = pytest.mark.skipif(
sys.platform == "win32" or not (shutil.which("bash") and shutil.which("jq")),
reason="needs bash and jq (present on the Linux runner that executes the workflow)",
)
FAKE_GH = r'''#!/usr/bin/env python3
"""Just enough of `gh` for scripts/codeql_triage.sh, backed by $FAKE_GH_STATE."""
import json, os, sys
path = os.environ["FAKE_GH_STATE"]
state = json.load(open(path))
args = sys.argv[1:]
def save():
json.dump(state, open(path, "w"))
def opt(name):
return args[args.index(name) + 1]
if args[0] == "api":
target = next(a for a in args[1:] if a.startswith("repos/"))
if "/code-scanning/alerts" in target:
print(json.dumps(state["alerts"]))
elif "/issues?" in target:
print(json.dumps(state["issues"]))
elif target.endswith("/comments"):
number = int(target.split("/")[-2])
issue = next(i for i in state["issues"] if i["number"] == number)
print(json.dumps([{"body": c} for c in issue["comments"]]))
else:
number = int(target.rsplit("/", 1)[1])
print(json.dumps(next(i for i in state["issues"] if i["number"] == number)))
elif args[:2] == ["issue", "create"]:
number = len(state["issues"]) + 1
state["issues"].append({
"number": number,
"title": opt("--title"),
"body": open(opt("--body-file")).read(),
"state": "open",
"labels": [opt("--label")],
"comments": [],
})
save()
print(f"https://github.com/o/r/issues/{number}")
elif args[:2] == ["issue", "comment"]:
number = int(args[2])
issue = next(i for i in state["issues"] if i["number"] == number)
issue["comments"].append(open(opt("--body-file")).read())
save()
else:
sys.exit(f"fake gh: unsupported {args}")
'''
def _alert(number: int, severity: str = "high") -> dict:
return {
"number": number,
"html_url": f"https://github.com/o/r/security/code-scanning/{number}",
"rule": {"id": f"py/rule-{number}", "security_severity_level": severity},
"most_recent_instance": {"location": {"path": "headroom/x.py", "start_line": number}},
"created_at": "2026-10-01T00:00:00Z",
}
@pytest.fixture
def harness(tmp_path):
bindir = tmp_path / "bin"
bindir.mkdir()
gh = bindir / "gh"
gh.write_text(FAKE_GH)
gh.chmod(0o755)
state_path = tmp_path / "state.json"
workdir = tmp_path / "work"
workdir.mkdir()
def set_alerts(*alerts):
state = json.loads(state_path.read_text()) if state_path.exists() else {"issues": []}
state["alerts"] = list(alerts)
state_path.write_text(json.dumps(state))
def state():
return json.loads(state_path.read_text())
def edit_issue(number, **changes):
s = state()
next(i for i in s["issues"] if i["number"] == number).update(changes)
state_path.write_text(json.dumps(s))
def run(today: str) -> subprocess.CompletedProcess:
env = {
**os.environ,
"PATH": f"{bindir}{os.pathsep}{os.environ['PATH']}",
"FAKE_GH_STATE": str(state_path),
"REPO": "o/r",
"TRIAGE_ASSIGNEE": "owner",
"TRIAGE_TODAY": today,
}
result = subprocess.run(
["bash", str(SCRIPT)], cwd=workdir, env=env, capture_output=True, text=True
)
assert result.returncode == 0, result.stderr
return result
set_alerts()
return set_alerts, state, edit_issue, run
def test_first_run_opens_one_issue_keyed_to_the_iso_week(harness):
set_alerts, state, _, run = harness
set_alerts(_alert(1), _alert(2, "critical"), _alert(3, "medium"))
run("2026-10-07") # a Wednesday
(issue,) = state()["issues"]
assert issue["title"] == "CodeQL triage: week of 2026-10-05 (2 critical/high open)"
assert "<!-- codeql-triage:2026-W41 -->" in issue["body"]
assert "[#1]" in issue["body"] and "[#2]" in issue["body"]
assert "[#3]" not in issue["body"] # medium is out of scope
def test_rerun_same_day_creates_nothing(harness):
set_alerts, state, _, run = harness
set_alerts(_alert(1))
run("2026-10-05")
result = run("2026-10-05")
(issue,) = state()["issues"]
assert issue["comments"] == []
assert "Nothing to do" in result.stdout
def test_later_dispatch_reuses_a_closed_issue_and_keeps_its_ticks(harness):
set_alerts, state, edit_issue, run = harness
set_alerts(_alert(1))
run("2026-10-05") # Monday's scheduled run
ticked = state()["issues"][0]["body"].replace("- [ ]", "- [x]")
edit_issue(1, body=ticked, state="closed") # #1 fixed, box ticked
set_alerts(_alert(7)) # a new high alert midweek
run("2026-10-09") # Friday's manual dispatch
(issue,) = state()["issues"]
assert issue["state"] == "closed"
assert issue["body"] == ticked
(comment,) = issue["comments"]
assert "[#7]" in comment and "[#1]" not in comment
run("2026-10-11") # Sunday: same ISO week, nothing new since the comment
(issue,) = state()["issues"]
assert len(issue["comments"]) == 1 # #7 is not reported twice
def test_next_week_opens_a_new_issue(harness):
set_alerts, state, _, run = harness
set_alerts(_alert(1))
run("2026-10-05")
run("2026-10-12")
titles = [i["title"] for i in state()["issues"]]
assert titles == [
"CodeQL triage: week of 2026-10-05 (1 critical/high open)",
"CodeQL triage: week of 2026-10-12 (1 critical/high open)",
]
def test_iso_year_boundary_shares_one_week(harness):
set_alerts, state, _, run = harness
set_alerts(_alert(1))
run("2026-12-28") # Monday of 2026-W53
run("2027-01-01") # Friday, same ISO week in a new calendar year
(issue,) = state()["issues"]
assert "<!-- codeql-triage:2026-W53 -->" in issue["body"]
assert issue["title"].startswith("CodeQL triage: week of 2026-12-28")
def test_no_alerts_still_records_the_review(harness):
_, state, _, run = harness
run("2026-10-05")
(issue,) = state()["issues"]
assert "(0 critical/high open)" in issue["title"]
assert "None. Close this issue to record the review." in issue["body"]
def test_an_alert_reopened_after_its_box_was_ticked_is_reported_again(harness):
set_alerts, state, edit_issue, run = harness
set_alerts(_alert(1))
run("2026-10-05")
ticked = state()["issues"][0]["body"].replace("- [ ]", "- [x]")
edit_issue(1, body=ticked, state="closed") # dismissed and ticked
# Reopened later the same week: open again, and its only mention is ticked.
run("2026-10-08")
(issue,) = state()["issues"]
(comment,) = issue["comments"]
assert "[#1]" in comment
assert issue["body"] == ticked and issue["state"] == "closed"
run("2026-10-09") # listed unticked in that comment now: not repeated
assert len(state()["issues"][0]["comments"]) == 1