mirror of
https://github.com/DeusData/codebase-memory-mcp.git
synced 2026-10-31 07:33:32 +08:00
Microsoft's ML flagged the rc.1 release binaries. The decisive evidence is that the SAME sha256 went from 0/62 clean to Microsoft-detected in about an hour with no byte change, so the verdict lives partly in scanner-side state and no code change can promise a clean result. What code CAN do is stop shipping things that have no business in a release artifact, which is worth doing on its own merits and incidentally widens the classifier margin. Every claim below is verified against a built binary by the new gate, not by reading source. Executable stack (the worst of the findings). vendored/nomic/code_vectors_blob.S is the only assembly in the build and carried no .note.GNU-stack. An unannotated object makes ld assume the worst for the whole link, so EVERY Linux release we have ever shipped had GNU_STACK RWE. Adds the note (cause) plus ELF-only -Wl,-z,noexecstack (outcome); the gate fails the release if it returns. Test seams are now opt-in, never opt-out. TEST_SEAMS=1 defines CBM_ENABLE_TEST_SEAMS; without it the crash-orphan probe -- which forks a child that ignores SIGTERM and loops forever, then writes its pid to a caller-supplied path -- and the lease-ownership marker compile to trivial stubs, so call sites are untouched and the binary holds no fork, no signal handler and no env-var string. Opt-IN is the point: forgetting the flag yields a clean binary rather than a leaky one. scripts/test.sh requests it in the leg that consumes it, and tests/test_worker_watchdog.sh now asserts the capability up front instead of dying later with an opaque "Killed: 9". The daemon's background version check is gone. It spawned curl against api.github.com/repos/.../releases/latest on the first eligible session of every run to say "a newer version exists" -- a release URL and an outbound request in every shipped binary, for something the install scripts already report. The INJECTABLE SEAM survives: update_ops is still honoured, the fakes in tests/test_daemon_application.c still cover notice/ownership/cancellation/replay, and with no provider application_update_subscribe_locked returns early so no generation ever starts. "No network request by default" is now structural. Dead capability out of release builds. The tar.gz/zip extraction block (gzip_decompress through cbm_extract_binary_from_zip, plus its cli.h declarations) moves under CBM_CLI_ENABLE_TEST_API -- verified self-contained, zero uses of any helper outside it, only callers the excluded updater and tests/test_cli.c. Downloading an archive, decompressing it, picking an executable out of it and marking it executable is the canonical dropper composite; it is now absent rather than merely unreachable. SQLite is built with -DSQLITE_OMIT_LOAD_EXTENSION (no caller of load_extension anywhere in src/ or internal/), removing that API surface and part of the dlopen/dlsym surface. Temp files and environment scanning (S2/S3). Predictable paths in mcp.c, artifact.c and diagnostics.c are created privately and exclusively and written through the returned descriptor; pass_envscan.c no longer descends symlinked directories out of the project root, and its fixed 512-byte path buffers no longer truncate into pointer arithmetic that could land outside the buffer. Build-time entropy. mimalloc's version banner baked __DATE__/__TIME__ into every binary, so two builds of identical source seconds apart could never share a hash and no release could inherit a false-positive determination made about its predecessor. Local patch removes it (marked to survive refreshes), -Wdate-time makes any future use a build error, and -Wl,--no-insert-timestamp stops the PE header carrying the link clock. scripts/ci/check-binary-composition.sh is the proof that each removal stays removed, wired into package-release.sh after strip so the local artifact-flow smoke enforces exactly what CI does. It asserts absences plus a CANARY string, so handing it a compressed, stubbed or empty file fails instead of passing vacuously, and a missing tool is a hard error -- a skipped assertion must never look like a satisfied one. Two build-system traps found by that gate, both of which had silently defeated a fix: the product binary is compiled in one shot from sources, so a flag flip did not rebuild it (now tracked by a .build-config stamp that also removes the binary, making it independent of mtime granularity); and prod_sqlite3.o / prod_mimalloc.o depended on a single named source, so SQLITE_OMIT_LOAD_EXTENSION and the mimalloc patch BOTH compiled to nothing on the first incremental build. Source review would have called them done. Deliberately NOT changed. Three seams stay in release artifacts because scripts/smoke-test.sh runs against the real artifact and needs them: CBM_TEST_CRASH_ON and CBM_TEST_HANG_ON inject the faults that prove supervisor recovery, and CBM_TEST_WINDOWS_USER_PATH_RUN_ID is what keeps the PATH smoke from writing the tester's actual PATH. The gate treats those as an allowlist, so a NOVEL seam still fails. The true no-UI standard build is deferred rather than rushed: src/ui/* is in PROD_SRCS and four files outside src/ui reference UI symbols, including the daemon that serves the UI, so that assertion reports instead of failing until the split lands -- a gate everyone knows is red teaches people to ignore gates. No grammar is removed. ObjectScript accounts for essentially all binary growth since the last provably-clean release (+21.4MB rodata, +1.1MB text from two four-line shims), which made it the obvious ablation candidate, but a dry run performed twelve real Defender endpoint scans across standard/UI and amd64/arm64 with ObjectScript, the daemon and the expanded hooks all present and every scan was clean. Nothing there is a deterministic trigger, so cutting a community-contributed language would spend a real feature on unproven margin. Lean is not a candidate either: at 99.6MB of source it is by far the largest grammar, but it shipped in v0.9.0 which scanned 20/20 clean, so removing it would produce a novel unscanned profile instead of restoring a known-good one. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
198 lines
8.5 KiB
Bash
Executable File
198 lines
8.5 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# package-release.sh — THE canonical release-archive step. Every venue that
|
|
# turns built binaries into a release archive (release/_build.yml, the local
|
|
# artifact-flow smoke lane) runs this file; workflows provide only
|
|
# checkout/toolchain/upload around it. Archive names and contents are defined
|
|
# HERE, nowhere else, so a local artifact smoke provably exercises the same
|
|
# bytes-layout the release publishes.
|
|
#
|
|
# This script ARCHIVES what scripts/build.sh already produced — it never
|
|
# builds the product itself (the Windows launcher image is the one deliberate
|
|
# exception: it is part of the archive, not of the product build).
|
|
|
|
set -euo pipefail
|
|
|
|
ROOT="$(cd "$(dirname "$0")/.." && pwd)"
|
|
cd "$ROOT"
|
|
|
|
usage() {
|
|
cat <<'EOF'
|
|
Usage: scripts/package-release.sh <goos> <goarch> [--variant standard|ui]
|
|
[--out-dir DIR] [VAR=VAL ...]
|
|
|
|
The canonical release-archive step: identical in the release build and the
|
|
local artifact-flow smoke lane.
|
|
|
|
goos linux | darwin | windows
|
|
goarch arch label used verbatim in the archive name (amd64, arm64,
|
|
arm64-portable, ...)
|
|
--variant standard (default) | ui — selects the archive NAME prefix; the
|
|
matching binary must already have been built (--with-ui for ui).
|
|
--out-dir where to place the archive (default: repository root).
|
|
|
|
Make passthrough (VAR=VAL, forwarded to the build):
|
|
CC= CXX= compiler override, e.g. CC=clang CXX=clang++.
|
|
|
|
Environment:
|
|
BUILD_DIR build tree to archive from (default build/c).
|
|
|
|
Archive contents (defined here, canonical) — ONE binary per platform:
|
|
unix: codebase-memory-mcp LICENSE install.sh THIRD_PARTY_NOTICES.md (.tar.gz)
|
|
windows: codebase-memory-mcp.exe LICENSE install.ps1
|
|
THIRD_PARTY_NOTICES.md (.zip)
|
|
EOF
|
|
}
|
|
|
|
GOOS=""
|
|
GOARCH=""
|
|
VARIANT="standard"
|
|
OUT_DIR="$ROOT"
|
|
MAKE_ARGS=()
|
|
expect_value=""
|
|
for arg in "$@"; do
|
|
case "$expect_value" in
|
|
variant) VARIANT="$arg"; expect_value=""; continue ;;
|
|
out-dir) OUT_DIR="$arg"; expect_value=""; continue ;;
|
|
esac
|
|
case "$arg" in
|
|
-h | --help) usage; exit 0 ;;
|
|
--variant) expect_value="variant" ;;
|
|
--variant=*) VARIANT="${arg#--variant=}" ;;
|
|
--out-dir) expect_value="out-dir" ;;
|
|
--out-dir=*) OUT_DIR="${arg#--out-dir=}" ;;
|
|
-*)
|
|
echo "package-release: unknown option '$arg'. Please consult --help." >&2
|
|
exit 2
|
|
;;
|
|
*=*) MAKE_ARGS+=("$arg") ;;
|
|
*)
|
|
if [ -z "$GOOS" ]; then GOOS="$arg"
|
|
elif [ -z "$GOARCH" ]; then GOARCH="$arg"
|
|
else
|
|
echo "package-release: unexpected argument '$arg'. Please consult --help." >&2
|
|
exit 2
|
|
fi
|
|
;;
|
|
esac
|
|
done
|
|
[ -n "$GOOS" ] && [ -n "$GOARCH" ] || { usage >&2; exit 2; }
|
|
case "$GOOS" in
|
|
linux | darwin | windows) ;;
|
|
*) echo "package-release: goos must be linux, darwin or windows." >&2; exit 2 ;;
|
|
esac
|
|
case "$VARIANT" in
|
|
standard) SUFFIX="" ;;
|
|
ui) SUFFIX="-ui" ;;
|
|
*) echo "package-release: variant must be 'standard' or 'ui'." >&2; exit 2 ;;
|
|
esac
|
|
[ -n "$expect_value" ] && { echo "package-release: --$expect_value needs a value." >&2; exit 2; }
|
|
|
|
BUILD_DIR="${BUILD_DIR:-build/c}"
|
|
OUT_DIR="$(mkdir -p "$OUT_DIR" && cd "$OUT_DIR" && pwd)"
|
|
NAME="codebase-memory-mcp${SUFFIX}-${GOOS}-${GOARCH}"
|
|
|
|
# Ship every release binary stripped. Production already builds without -g, but
|
|
# the linker still keeps a ~536 KB .symtab, so releases carried their full
|
|
# symbol table to users: bigger downloads and a free map of the internals, with
|
|
# nothing gained. Nothing symbolizes at runtime (mem_profile.c is not in the
|
|
# production build and never calls backtrace_symbols), so this costs no
|
|
# diagnostics.
|
|
#
|
|
# It also had a concrete cost. Microsoft's ML scored the unstripped linux-amd64
|
|
# binary Trojan:Script/Wacatac.B!ml (1 engine of 62) and blocked release run
|
|
# 30398064336 at the VirusTotal gate. That verdict is a decision-boundary
|
|
# artifact rather than a property of the code -- the dry-run build two days
|
|
# earlier is the same program plus 10 KB and scans clean, and the ui build of
|
|
# the same commit was never flagged. Stripping removes the symbol surface those
|
|
# models score and cleared BOTH flagged builds (Wacatac.B and Wacatac.C)
|
|
# without changing what the program does.
|
|
#
|
|
# macOS is ad-hoc signed by the build workflow BEFORE this script runs, and
|
|
# stripping invalidates that signature, so Mach-O is re-signed here. Skipping
|
|
# the re-sign ships a binary the kernel refuses to exec.
|
|
strip_release_binary() {
|
|
local binary="$1"
|
|
[ -f "$binary" ] || return 0
|
|
# The right flags differ per format, and the WRONG ones fail silently in
|
|
# the dangerous direction. Measured on the flagged darwin-arm64 artifact:
|
|
#
|
|
# llvm-strip --strip-all 373 symbols scanned CLEAN
|
|
# strip (no flags) 378 symbols equivalent
|
|
# strip -x -S 4058 symbols the state VirusTotal FLAGGED
|
|
# strip -X / -u -r 4058 symbols likewise
|
|
#
|
|
# Apple's strip returns success for `-x -S`, so a helper that just tries
|
|
# candidates until one exits 0 would quietly reship the flagged binary.
|
|
# GNU/LLVM `--strip-all` is not even accepted by Apple's strip, which is why
|
|
# generalising it to every platform broke the macOS build -- loudly, which
|
|
# was the lucky outcome.
|
|
#
|
|
# So: --strip-all where it is understood, plain `strip` for Mach-O, and a
|
|
# hard error when no candidate can do the job. Never a weaker fallback.
|
|
local stripped=""
|
|
for tool in "${STRIP:-}" llvm-strip strip; do
|
|
[ -n "$tool" ] || continue
|
|
command -v "$tool" >/dev/null 2>&1 || continue
|
|
if "$tool" --strip-all "$binary" 2>/dev/null; then
|
|
stripped="$tool --strip-all"
|
|
elif [ "$GOOS" = "darwin" ] && "$tool" "$binary" 2>/dev/null; then
|
|
stripped="$tool"
|
|
fi
|
|
[ -n "$stripped" ] && break
|
|
done
|
|
if [ -z "$stripped" ]; then
|
|
echo "package-release: no working strip for $binary" >&2
|
|
return 1
|
|
fi
|
|
if [ "$GOOS" = "darwin" ]; then
|
|
command -v codesign >/dev/null 2>&1 &&
|
|
codesign --sign - --force "$binary" 2>/dev/null
|
|
fi
|
|
echo "=== package-release: stripped $(basename "$binary") ==="
|
|
return 0
|
|
}
|
|
|
|
if [ "$GOOS" = "windows" ]; then
|
|
# Windows ships ONE binary, exactly like every other platform. There is no
|
|
# launcher stub: a small unsigned PE whose entire job is to verify and
|
|
# execute another binary is statically indistinguishable from a dropper,
|
|
# and Defender's ML scored it Trojan:Win32/Wacatac.B!ml on x64 regardless
|
|
# of what we changed (bcrypt-free, stripped, versioned, and even
|
|
# resource-free builds were all flagged, while the product binary itself
|
|
# scans clean on every platform). Self-update — the launcher's whole reason
|
|
# to exist — moves OUT of the running process into install.ps1: Windows'
|
|
# executable lock only blocks a process from replacing ITSELF.
|
|
PAYLOAD="$BUILD_DIR/codebase-memory-mcp"
|
|
[ -f "${PAYLOAD}.exe" ] && PAYLOAD="${PAYLOAD}.exe"
|
|
[ -f "$PAYLOAD" ] || { echo "package-release: build first; missing $PAYLOAD" >&2; exit 2; }
|
|
PACK_DIR="$(mktemp -d "${TMPDIR:-/tmp}/cbm-package.XXXXXX")"
|
|
trap 'rm -rf "$PACK_DIR"' EXIT
|
|
cp "$PAYLOAD" "$PACK_DIR/codebase-memory-mcp.exe"
|
|
strip_release_binary "$PACK_DIR/codebase-memory-mcp.exe" || exit 2
|
|
# Gate the artifact AFTER strip: strip is the last byte-changing step, so
|
|
# this inspects exactly what goes into the archive. Runs here rather than in
|
|
# a workflow step so the local artifact-flow smoke enforces the same thing.
|
|
scripts/ci/check-binary-composition.sh --variant="$VARIANT" \
|
|
"$PACK_DIR/codebase-memory-mcp.exe" || exit 2
|
|
cp LICENSE install.ps1 "$PACK_DIR/"
|
|
scripts/gen-third-party-notices.sh "$PACK_DIR/THIRD_PARTY_NOTICES.md"
|
|
(
|
|
cd "$PACK_DIR"
|
|
rm -f "$OUT_DIR/$NAME.zip"
|
|
zip -q "$OUT_DIR/$NAME.zip" \
|
|
codebase-memory-mcp.exe LICENSE install.ps1 THIRD_PARTY_NOTICES.md
|
|
)
|
|
echo "=== package-release: $OUT_DIR/$NAME.zip ==="
|
|
else
|
|
[ -f "$BUILD_DIR/codebase-memory-mcp" ] ||
|
|
{ echo "package-release: build first; missing $BUILD_DIR/codebase-memory-mcp" >&2; exit 2; }
|
|
strip_release_binary "$BUILD_DIR/codebase-memory-mcp" || exit 2
|
|
scripts/ci/check-binary-composition.sh --variant="$VARIANT" \
|
|
"$BUILD_DIR/codebase-memory-mcp" || exit 2
|
|
cp LICENSE install.sh "$BUILD_DIR/"
|
|
scripts/gen-third-party-notices.sh "$BUILD_DIR/THIRD_PARTY_NOTICES.md"
|
|
tar -czf "$OUT_DIR/$NAME.tar.gz" -C "$BUILD_DIR" \
|
|
codebase-memory-mcp LICENSE install.sh THIRD_PARTY_NOTICES.md
|
|
echo "=== package-release: $OUT_DIR/$NAME.tar.gz ==="
|
|
fi
|