mirror of
https://github.com/headroomlabs-ai/headroom.git
synced 2026-11-22 13:29:19 +08:00
SOC 2 follow-ups from the 2026-10-06 firewall review (R3, R4) and incident INC-2026-001. ## Beacon rate limit (R3) `deploy/beacon/wrangler.toml` said a WAF rule limited `/v1/logs` to 60 req/min/IP. That rule could not exist: headroomlabs.ai is not a Cloudflare zone, so the Worker is only on `workers.dev`. This adds a [Workers rate-limit binding](https://developers.cloudflare.com/workers/runtime-apis/bindings/rate-limit/) checked in `fetch()` **before** the body is read. - Keyed on one constant, not the IP: the Worker still never reads `cf-connecting-ip` (privacy promise in the file header). Trade-off: a flood also sheds real events for that minute. - 3000 req/min **per Cloudflare location** (~10x today's ~300/min total). - Returns 429, which the client does **not** treat as a gzip refusal (`_GZIP_REFUSED_STATUSES` is 400/415), so rate-limited clients keep compressing. - Fails open if the binding is absent **or the limiter itself errors** (review fix): a rate-limit service outage lets beacons through rather than dropping all of them. `node deploy/beacon/test-rollup.mjs` covers 429 over the limit, fail-open on outage and when unbound, and the single shared key. ## Dev environment (R4) `[env.dev]` deploys `headroom-beacon-dev` bound to the new `headroom-telemetry-dev` bucket (600/min limit). Already deployed and tested: a sample event returns 204, and a 2,500-request burst got 384 × 429. ## Weekly CodeQL triage `.github/workflows/codeql-triage.yml` opens an issue every Monday, assigned to @chopratejas (tejas@headroomlabs.ai), listing open critical/high alerts with checkboxes. Alert #63 sat unowned for five months; this gives alerts an owner and leaves an audit trail. **One issue per week, however often it runs (review fix).** The logic is in `scripts/codeql_triage.sh`: - It keys the issue on the ISO week, using a hidden marker in the body. - It finds the existing issue by listing security-labelled issues, closed ones included. It uses a listing, not search, so an issue created seconds earlier is not missed. - It reuses that issue and never edits, reopens or retitles it, so ticked boxes and closure stand. Alerts opened since then are added once, as a comment. - The jq programs live in quoted heredocs, which clears ShellCheck SC2016 without any shell expansion. - `tests/test_codeql_triage_script.py` runs the real script against a fake `gh`. It covers same-day re-runs, a Friday dispatch on a closed issue with ticked boxes, the next week, and the ISO year boundary (2027-01-01 is in 2026-W53). ### Test output ```text $ pytest -q tests/test_codeql_triage_script.py 6 passed $ node deploy/beacon/test-rollup.mjs rate limit: 429 over, fail-open on outage and when unbound ok $ shellcheck scripts/codeql_triage.sh && actionlint .github/workflows/codeql-triage.yml (clean) ``` ## ⚠️ Before deploying production The live `headroom-beacon` Worker predates #3253 (gzip inflate, schema v2 allowlist). `wrangler deploy` from main will ship that too, not just this change. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/headroomlabs-ai/headroom/pull/4008" target="_blank"><picture><source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-devin-review-dark.svg?v=4"><img src="https://static.devin.ai/assets/gh-devin-review-light.svg?v=4" alt="Devin Review"></picture></a> <!-- devin-review-badge-end --> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: JerrettDavis <mxjerrett@gmail.com>
97 lines
4.4 KiB
Bash
Executable File
97 lines
4.4 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# Weekly CodeQL triage issue, run by .github/workflows/codeql-triage.yml.
|
|
#
|
|
# Idempotent per ISO week. The week key is the ISO week of "today" (UTC), so a
|
|
# re-run of Monday's job and a manual dispatch later the same week both find
|
|
# the issue already opened for that week, including a closed one, and reuse it
|
|
# instead of opening another. A reused issue is never edited, reopened or
|
|
# retitled, so ticked checkboxes and its closure stand. Open critical/high
|
|
# alerts it does not list unticked (new ones, or ones reopened after being
|
|
# ticked) are added as a comment; otherwise the run does nothing.
|
|
#
|
|
# Env: REPO (owner/name), TRIAGE_ASSIGNEE (login). TRIAGE_TODAY (YYYY-MM-DD)
|
|
# overrides the date for tests. Needs gh, jq and python3.
|
|
set -euo pipefail
|
|
|
|
: "${REPO:?REPO is required}"
|
|
: "${TRIAGE_ASSIGNEE:?TRIAGE_ASSIGNEE is required}"
|
|
today="${TRIAGE_TODAY:-$(date -u +%Y-%m-%d)}"
|
|
|
|
# ISO week key (e.g. 2026-W41) and that week's Monday. Python's isocalendar
|
|
# handles the year boundary (2027-01-01 is in 2026-W53) on any platform's date.
|
|
read -r week_key week_start < <(python3 -c '
|
|
import datetime, sys
|
|
d = datetime.date.fromisoformat(sys.argv[1])
|
|
y, w, wd = d.isocalendar()
|
|
print(f"{y}-W{w:02d}", d - datetime.timedelta(days=wd - 1))
|
|
' "$today")
|
|
marker="<!-- codeql-triage:${week_key} -->"
|
|
|
|
# jq programs live in quoted heredocs: they are literal jq, not shell, so
|
|
# nothing in them is expanded and no single-quoted '$' or backtick is left for
|
|
# ShellCheck (SC2016) to misread.
|
|
alert_lines_jq=$(cat <<'JQ'
|
|
.[] | select(.rule.security_severity_level == "critical" or .rule.security_severity_level == "high")
|
|
| "- [ ] [#\(.number)](\(.html_url)) **\(.rule.security_severity_level)** `\(.rule.id)` in `\(.most_recent_instance.location.path):\(.most_recent_instance.location.start_line)` (open since \(.created_at[:10]))"
|
|
JQ
|
|
)
|
|
week_issue_jq=$(cat <<'JQ'
|
|
.[] | select(.pull_request | not) | select((.body // "") | contains($marker)) | .number
|
|
JQ
|
|
)
|
|
|
|
alerts="$(gh api --paginate -X GET "repos/$REPO/code-scanning/alerts" -f state=open -f per_page=100 \
|
|
| jq -r "$alert_lines_jq")"
|
|
|
|
# Listing, not search: the search index lags, so an issue created seconds ago
|
|
# by the previous run could be missed. state=all so a closed issue counts.
|
|
existing="$(gh api --paginate "repos/$REPO/issues?state=all&labels=security&per_page=100" \
|
|
| jq -r --arg marker "$marker" "$week_issue_jq" | head -n 1)"
|
|
|
|
if [ -n "$existing" ]; then
|
|
# Alerts already listed, unticked, in the issue or an earlier re-run
|
|
# comment. A ticked alert that is open again (dismissed, then reopened) is
|
|
# reported again rather than hidden behind its old tick.
|
|
known="$(gh api "repos/$REPO/issues/$existing" | jq -r '.body // ""')"
|
|
known="$known"$'\n'"$(gh api --paginate "repos/$REPO/issues/$existing/comments" | jq -r '.[].body // ""')"
|
|
new=""
|
|
while IFS= read -r line; do
|
|
[ -n "$line" ] || continue
|
|
ref="${line#*\[#}" # after "- [ ] [#"
|
|
ref="- [ ] [#${ref%%\]*}](" # "- [ ] [#123]("
|
|
case "$known" in *"$ref"*) ;; *) new="${new}${line}"$'\n' ;; esac
|
|
done <<< "$alerts"
|
|
if [ -z "$new" ]; then
|
|
echo "Week $week_key already has issue #$existing and no new critical/high alerts. Nothing to do."
|
|
exit 0
|
|
fi
|
|
{
|
|
echo "Re-run on $today: critical/high alerts that are open and not yet listed here (new, or reopened after being ticked):"
|
|
echo
|
|
printf '%s' "$new"
|
|
} > comment.md
|
|
gh issue comment "$existing" --repo "$REPO" --body-file comment.md
|
|
echo "Week $week_key: added $(printf '%s' "$new" | grep -c '^- ') new alert(s) to #$existing."
|
|
exit 0
|
|
fi
|
|
|
|
count="$(printf '%s' "$alerts" | grep -c '^- ' || true)"
|
|
{
|
|
echo "$marker"
|
|
echo "Weekly code scanning triage for the week of **$week_start** ($week_key). Owner: @$TRIAGE_ASSIGNEE (tejas@headroomlabs.ai)."
|
|
echo
|
|
echo "For each alert: fix it (link the PR) or dismiss it in code scanning with a reason. Close this issue when every box is ticked."
|
|
echo "Target: critical and high fixed or dismissed within the vulnerability management SLA."
|
|
echo
|
|
echo "### Open critical/high alerts: $count"
|
|
echo
|
|
if [ "$count" -eq 0 ]; then echo "None. Close this issue to record the review."; else printf '%s\n' "$alerts"; fi
|
|
echo
|
|
echo "All open alerts: https://github.com/$REPO/security/code-scanning"
|
|
} > body.md
|
|
gh issue create --repo "$REPO" \
|
|
--title "CodeQL triage: week of $week_start ($count critical/high open)" \
|
|
--body-file body.md \
|
|
--assignee "$TRIAGE_ASSIGNEE" \
|
|
--label security
|