Files
headroom/scripts/codeql_triage.sh
5698e91b2a chore(security): beacon rate limit, dev environment, weekly CodeQL triage (#4008)
SOC 2 follow-ups from the 2026-10-06 firewall review (R3, R4) and
incident INC-2026-001.

## Beacon rate limit (R3)
`deploy/beacon/wrangler.toml` said a WAF rule limited `/v1/logs` to 60
req/min/IP. That rule could not exist: headroomlabs.ai is not a
Cloudflare zone, so the Worker is only on `workers.dev`. This adds a
[Workers rate-limit
binding](https://developers.cloudflare.com/workers/runtime-apis/bindings/rate-limit/)
checked in `fetch()` **before** the body is read.

- Keyed on one constant, not the IP: the Worker still never reads
`cf-connecting-ip` (privacy promise in the file header). Trade-off: a
flood also sheds real events for that minute.
- 3000 req/min **per Cloudflare location** (~10x today's ~300/min
total).
- Returns 429, which the client does **not** treat as a gzip refusal
(`_GZIP_REFUSED_STATUSES` is 400/415), so rate-limited clients keep
compressing.
- Fails open if the binding is absent **or the limiter itself errors**
(review fix): a rate-limit service outage lets beacons through rather
than dropping all of them. `node deploy/beacon/test-rollup.mjs` covers
429 over the limit, fail-open on outage and when unbound, and the single
shared key.

## Dev environment (R4)
`[env.dev]` deploys `headroom-beacon-dev` bound to the new
`headroom-telemetry-dev` bucket (600/min limit). Already deployed and
tested: a sample event returns 204, and a 2,500-request burst got 384 ×
429.

## Weekly CodeQL triage
`.github/workflows/codeql-triage.yml` opens an issue every Monday,
assigned to @chopratejas (tejas@headroomlabs.ai), listing open
critical/high alerts with checkboxes. Alert #63 sat unowned for five
months; this gives alerts an owner and leaves an audit trail.

**One issue per week, however often it runs (review fix).** The logic is
in `scripts/codeql_triage.sh`:
- It keys the issue on the ISO week, using a hidden marker in the body.
- It finds the existing issue by listing security-labelled issues,
closed ones included. It uses a listing, not search, so an issue created
seconds earlier is not missed.
- It reuses that issue and never edits, reopens or retitles it, so
ticked boxes and closure stand. Alerts opened since then are added once,
as a comment.
- The jq programs live in quoted heredocs, which clears ShellCheck
SC2016 without any shell expansion.
- `tests/test_codeql_triage_script.py` runs the real script against a
fake `gh`. It covers same-day re-runs, a Friday dispatch on a closed
issue with ticked boxes, the next week, and the ISO year boundary
(2027-01-01 is in 2026-W53).

### Test output

```text
$ pytest -q tests/test_codeql_triage_script.py
6 passed
$ node deploy/beacon/test-rollup.mjs
rate limit: 429 over, fail-open on outage and when unbound
ok
$ shellcheck scripts/codeql_triage.sh && actionlint .github/workflows/codeql-triage.yml
(clean)
```

## ⚠️ Before deploying production
The live `headroom-beacon` Worker predates #3253 (gzip inflate, schema
v2 allowlist). `wrangler deploy` from main will ship that too, not just
this change.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4008"
target="_blank"><picture><source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img
src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4"
alt="Devin Review"></picture></a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
2026-10-12 11:18:47 -07:00

97 lines
4.4 KiB
Bash
Executable File

#!/usr/bin/env bash
# Weekly CodeQL triage issue, run by .github/workflows/codeql-triage.yml.
#
# Idempotent per ISO week. The week key is the ISO week of "today" (UTC), so a
# re-run of Monday's job and a manual dispatch later the same week both find
# the issue already opened for that week, including a closed one, and reuse it
# instead of opening another. A reused issue is never edited, reopened or
# retitled, so ticked checkboxes and its closure stand. Open critical/high
# alerts it does not list unticked (new ones, or ones reopened after being
# ticked) are added as a comment; otherwise the run does nothing.
#
# Env: REPO (owner/name), TRIAGE_ASSIGNEE (login). TRIAGE_TODAY (YYYY-MM-DD)
# overrides the date for tests. Needs gh, jq and python3.
set -euo pipefail
: "${REPO:?REPO is required}"
: "${TRIAGE_ASSIGNEE:?TRIAGE_ASSIGNEE is required}"
today="${TRIAGE_TODAY:-$(date -u +%Y-%m-%d)}"
# ISO week key (e.g. 2026-W41) and that week's Monday. Python's isocalendar
# handles the year boundary (2027-01-01 is in 2026-W53) on any platform's date.
read -r week_key week_start < <(python3 -c '
import datetime, sys
d = datetime.date.fromisoformat(sys.argv[1])
y, w, wd = d.isocalendar()
print(f"{y}-W{w:02d}", d - datetime.timedelta(days=wd - 1))
' "$today")
marker="<!-- codeql-triage:${week_key} -->"
# jq programs live in quoted heredocs: they are literal jq, not shell, so
# nothing in them is expanded and no single-quoted '$' or backtick is left for
# ShellCheck (SC2016) to misread.
alert_lines_jq=$(cat <<'JQ'
.[] | select(.rule.security_severity_level == "critical" or .rule.security_severity_level == "high")
| "- [ ] [#\(.number)](\(.html_url)) **\(.rule.security_severity_level)** `\(.rule.id)` in `\(.most_recent_instance.location.path):\(.most_recent_instance.location.start_line)` (open since \(.created_at[:10]))"
JQ
)
week_issue_jq=$(cat <<'JQ'
.[] | select(.pull_request | not) | select((.body // "") | contains($marker)) | .number
JQ
)
alerts="$(gh api --paginate -X GET "repos/$REPO/code-scanning/alerts" -f state=open -f per_page=100 \
| jq -r "$alert_lines_jq")"
# Listing, not search: the search index lags, so an issue created seconds ago
# by the previous run could be missed. state=all so a closed issue counts.
existing="$(gh api --paginate "repos/$REPO/issues?state=all&labels=security&per_page=100" \
| jq -r --arg marker "$marker" "$week_issue_jq" | head -n 1)"
if [ -n "$existing" ]; then
# Alerts already listed, unticked, in the issue or an earlier re-run
# comment. A ticked alert that is open again (dismissed, then reopened) is
# reported again rather than hidden behind its old tick.
known="$(gh api "repos/$REPO/issues/$existing" | jq -r '.body // ""')"
known="$known"$'\n'"$(gh api --paginate "repos/$REPO/issues/$existing/comments" | jq -r '.[].body // ""')"
new=""
while IFS= read -r line; do
[ -n "$line" ] || continue
ref="${line#*\[#}" # after "- [ ] [#"
ref="- [ ] [#${ref%%\]*}](" # "- [ ] [#123]("
case "$known" in *"$ref"*) ;; *) new="${new}${line}"$'\n' ;; esac
done <<< "$alerts"
if [ -z "$new" ]; then
echo "Week $week_key already has issue #$existing and no new critical/high alerts. Nothing to do."
exit 0
fi
{
echo "Re-run on $today: critical/high alerts that are open and not yet listed here (new, or reopened after being ticked):"
echo
printf '%s' "$new"
} > comment.md
gh issue comment "$existing" --repo "$REPO" --body-file comment.md
echo "Week $week_key: added $(printf '%s' "$new" | grep -c '^- ') new alert(s) to #$existing."
exit 0
fi
count="$(printf '%s' "$alerts" | grep -c '^- ' || true)"
{
echo "$marker"
echo "Weekly code scanning triage for the week of **$week_start** ($week_key). Owner: @$TRIAGE_ASSIGNEE (tejas@headroomlabs.ai)."
echo
echo "For each alert: fix it (link the PR) or dismiss it in code scanning with a reason. Close this issue when every box is ticked."
echo "Target: critical and high fixed or dismissed within the vulnerability management SLA."
echo
echo "### Open critical/high alerts: $count"
echo
if [ "$count" -eq 0 ]; then echo "None. Close this issue to record the review."; else printf '%s\n' "$alerts"; fi
echo
echo "All open alerts: https://github.com/$REPO/security/code-scanning"
} > body.md
gh issue create --repo "$REPO" \
--title "CodeQL triage: week of $week_start ($count critical/high open)" \
--body-file body.md \
--assignee "$TRIAGE_ASSIGNEE" \
--label security