ci(release): double-verify ML antivirus false positives instead of re-rolling builds

Release run 30464288732 was blocked by the VirusTotal gate: three linux-amd64
binaries flagged 1/62 by Microsoft's Wacatac.B!ml -- fully stripped binaries
(0 symbols, verified on the exact artifacts), the state that scanned clean in
the two previous cycles. Meanwhile a real Defender endpoint (engine
1.1.26060.3008, signatures 1.455.410.0 updated the same day, RTP on) scans
the identical bytes clean. Four cycles of evidence now say the same thing:
this verdict is an unstable ML decision boundary, not a property of the code,
and no build-side lever moves it durably -- stripping, downloader removal and
metadata changes each "worked" only until a later build flipped it back.

So stop treating the flag as buildable-away and verify it honestly instead:

  check-virustotal.sh may downgrade BLOCKED to TOLERATED only when ALL hold:
    - pre-release version (-rc./-pre/-alpha/-beta); stable releases never
    - every failing file flagged by exactly ONE engine
    - that engine is Microsoft and the verdict ends in "!ml" (never a
      signature name)
    - hash-pinned Defender ENDPOINT evidence is attached to the draft release
      (defender-endpoint-verification.txt) proving Microsoft's shipping
      product, signature-updated at scan time, reports the exact bytes clean

  av-endpoint-verify.sh (new) produces that evidence: downloads the draft
  assets, scans them on the local Windows VM endpoint, refuses to attest if
  RTP is off or Defender itself detects, uploads the hash-pinned result.
  The gate prints the exact command when evidence is missing; re-running the
  failed verify job does not rebuild, so the bytes stay fixed.

  append-vt-notes.sh (new, extracted from inline YAML per venue-parity) then
  renders the release-notes table honestly: a tolerated file reads "1/62 ML
  false positive, endpoint-verified clean", never "0 detections".

tests/test_vt_gate_tolerance_contract.sh pins all nine decision directions
against a stubbed VT API and release store -- clean pass, stable-never,
missing/stale/DETECTED evidence, signature-named verdict, non-Microsoft
engine, multi-engine -- so the tolerance provably fails closed.

Also: release.yml gains skip_tests for re-releases of an already test-green
tree (build/smoke/soak/verify always run; lint failures still gate via
!cancelled() && !failure()).

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
This commit is contained in:
Martin Vogel
2026-07-29 20:45:13 +02:00
parent e9ce367a05
commit 48fc942418
5 changed files with 474 additions and 31 deletions
+24 -28
View File
@@ -31,6 +31,11 @@ on:
required: false
type: boolean
default: false
skip_tests:
description: "Skip the test phase entirely (re-release of an already test-green tree; build/smoke/soak/verify still run)"
required: false
type: boolean
default: false
permissions:
contents: read
@@ -46,7 +51,12 @@ jobs:
uses: ./.github/workflows/_lint.yml
# ── 2. Tests (all platforms, full suite for release) ────────────
# skip_tests exists for RE-releases of a tree whose tests are already green:
# when only packaging/gating changed and the previous run proved the suites,
# re-running ~2h of tests adds no information. It skips ONLY this phase —
# build, smoke, soak and verify always run against the fresh artifacts.
test:
if: ${{ !inputs.skip_tests }}
needs: [lint]
uses: ./.github/workflows/_test.yml
with:
@@ -55,7 +65,10 @@ jobs:
shard_suites: true
# ── 3. Build all platforms ──────────────────────────────────────
# !cancelled() && !failure(): run when `test` is deliberately skipped, but
# never when lint or test actually failed.
build:
if: ${{ !cancelled() && !failure() }}
needs: [test]
permissions:
contents: read
@@ -220,44 +233,27 @@ jobs:
vt_api_key: ${{ secrets.VIRUS_TOTAL_SCANNER_API_KEY }}
files: binaries/*
# VERSION + GH_TOKEN enable the pre-release ML false-positive tolerance:
# the gate may downgrade a single-engine Microsoft "!ml" verdict to
# TOLERATED, but only for -rc./-pre versions and only when hash-pinned
# Defender ENDPOINT evidence is attached to the draft release
# (defender-endpoint-verification.txt, from scripts/ci/av-endpoint-verify.sh).
- name: Wait for VirusTotal results
env:
VT_API_KEY: ${{ secrets.VIRUS_TOTAL_SCANNER_API_KEY }}
VT_ANALYSIS: ${{ steps.virustotal.outputs.analysis }}
VERSION: ${{ inputs.version }}
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: scripts/ci/check-virustotal.sh
# Renders per-binary sha256 + VT links into the notes; when the gate
# tolerated an ML false positive it states that honestly instead of
# claiming "0 detections".
- name: Append VirusTotal scan links to release notes
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
VERSION: ${{ inputs.version }}
run: |
# Hash the extracted binaries (not the archives — VT indexes binary hashes)
TABLE="\n\n## Security Verification\n\n"
TABLE+="All release binaries scanned with 70+ antivirus engines — **0 detections**.\n\n"
TABLE+="| Binary | SHA-256 | VirusTotal |\n"
TABLE+="|--------|---------|------------|\n"
for bin in binaries/codebase-memory-mcp-*; do
[ -f "$bin" ] || continue
name=$(basename "$bin")
# Skip UI variants and non-binary files
echo "$name" | grep -qE \
'^codebase-memory-mcp-(linux|darwin|windows)-(amd64|arm64)(\.exe)?$' || continue
sha256=$(sha256sum "$bin" 2>/dev/null | awk '{print $1}' \
|| shasum -a 256 "$bin" | awk '{print $1}')
label=$(echo "$name" \
| sed 's/^codebase-memory-mcp-//' \
| sed 's/\.exe$//')
short="${sha256:0:20}..."
vt_url="https://www.virustotal.com/gui/file/${sha256}/detection"
TABLE+="| \`${label}\` | \`${short}\` | [0/72 ✅](${vt_url}) |\n"
done
CURRENT=$(gh release view "$VERSION" \
--json body --jq '.body // ""' --repo "$GITHUB_REPOSITORY")
printf '%s%b' "$CURRENT" "$TABLE" > /tmp/release_notes.md
gh release edit "$VERSION" \
--notes-file /tmp/release_notes.md --repo "$GITHUB_REPOSITORY"
run: scripts/ci/append-vt-notes.sh
# ── 8. Publish package wrappers (npm + PyPI) ──────────────────
# Wrappers in pkg/npm and pkg/pypi download the released binary at