mirror of
https://github.com/DeusData/codebase-memory-mcp.git
synced 2026-10-04 05:48:43 +08:00
Martin Vogel
92e7e6a913
Indexing broke on x86-64 Linux: the index worker refused to start and
SIGKILLed its own process group, so every venue reported nothing but
"index worker ended with killed (exit=-1, signal=9)".
glibc places a thread's static TLS block inside the stack allocation it
makes for that thread. This branch's new thread-local caches -- a
512-slot field cache (~24 KB), per-depth tree cursors (~4 KB) and two
parked CBMArenas (~8 KB; an arena carries blocks[256] + block_sizes[256])
-- took static TLS from 47,905 to 93,513 bytes. The 64 KB parent-death
watchdog thread then could not be created at all, and a worker without
process-tree containment correctly refuses to index.
Four layers, because there were four things wrong:
- The big thread-locals move to the heap behind a thread-local POINTER,
allocated once per thread and reused, each with a correct fallback if
allocation fails (the field cache and cursor pool are caches; a miss
is slower, never wrong). Static TLS is back to 52,233 bytes.
- PARENT_WATCHDOG_STACK_SIZE 64 KB -> 256 KB. "The watchdog only polls,
a tiny stack suffices" was true when written and false once TLS
reached ~50 KB, because the TLS block comes out of that same
allocation.
- cbm_thread_create retries with the default stack on EINVAL. A stack
size is a hint about what a thread needs; the platform refusing the
hint is not a reason to fail to create the thread.
- tests/test_thread_stack_tls_contract.sh (Step 5a of scripts/test.sh)
keeps static TLS a quarter clear of the smallest requested stack,
reading that smallest size from the sources so a new small stack
tightens the gate automatically.
Two diagnostics that made this findable at all, and that stay:
smoke-test.sh prints the worker log its failure message names (CI deletes
that sandbox with the job, so the one artifact naming the cause was
unreadable), and worker_containment_unavailable() now reports WHICH of
its conditions failed, with pid/pgrp/ppid.
Verified in the amd64 container, the venue nothing on the local ladder
covers: baseline main indexes 9039 nodes/15067 edges, this branch before
the fix refuses to start the worker, and after it indexes 9039/15067.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
85 lines
2.9 KiB
Bash
Executable File
85 lines
2.9 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# fuzz.sh — run the libFuzzer targets of build/fuzz/cbm-fuzz.
|
|
#
|
|
# Usage:
|
|
# scripts/fuzz.sh <extract|cypher|config|all> [--runs N] [--max-total-time S]
|
|
# [--jobs N] [--seed N] [--no-build]
|
|
#
|
|
# Two shapes, one driver:
|
|
# --runs N --seed S a fixed number of executions from a fixed seed: the same
|
|
# inputs every time, so a smoke in CI is a function of the
|
|
# code (O9). This is the default (--runs 20000 --seed 1).
|
|
# --max-total-time S exploration for dry-run and nightly: new inputs are the
|
|
# point, and a crash is saved with its reproducer.
|
|
#
|
|
# The checked-in seeds (tests/fuzz/corpus/<target>/) are copied into a working
|
|
# corpus under build/fuzz/, so a run never rewrites the repository. A crash,
|
|
# leak or timeout leaves its input in build/fuzz/artifacts-<target>/ and the
|
|
# script exits non-zero; reproduce with
|
|
# CBM_FUZZ_TARGET=<target> build/fuzz/cbm-fuzz <artifact>
|
|
set -euo pipefail
|
|
|
|
ROOT="$(cd "$(dirname "$0")/.." && pwd)"
|
|
# --help before the positional target: the venue-parity interface probe calls
|
|
# every canonical entry as `<entry> --help` with no other argument.
|
|
case "${1:-}" in
|
|
-h | --help) sed -n '2,21p' "$0"; exit 0 ;;
|
|
esac
|
|
TARGET="${1:-}"
|
|
[ $# -gt 0 ] && shift
|
|
RUNS=20000
|
|
SEED=1
|
|
MAX_TIME=""
|
|
JOBS=1
|
|
BUILD=1
|
|
while [ $# -gt 0 ]; do
|
|
case "$1" in
|
|
--runs) RUNS="$2"; shift 2 ;;
|
|
--seed) SEED="$2"; shift 2 ;;
|
|
--max-total-time) MAX_TIME="$2"; RUNS=""; shift 2 ;;
|
|
--jobs) JOBS="$2"; shift 2 ;;
|
|
--no-build) BUILD=0; shift ;;
|
|
-h | --help) sed -n '2,21p' "$0"; exit 0 ;;
|
|
*) echo "fuzz.sh: unknown argument $1. Please consult --help." >&2; exit 2 ;;
|
|
esac
|
|
done
|
|
|
|
case "$TARGET" in
|
|
extract | cypher | config) TARGETS=("$TARGET") ;;
|
|
all) TARGETS=(extract cypher config) ;;
|
|
*) echo "fuzz.sh: target must be extract, cypher, config or all. Please consult --help." >&2
|
|
exit 2 ;;
|
|
esac
|
|
|
|
BIN="$ROOT/build/fuzz/cbm-fuzz"
|
|
if [ "$BUILD" -eq 1 ]; then
|
|
make -C "$ROOT" -f Makefile.cbm -j"$(getconf _NPROCESSORS_ONLN)" fuzz >/dev/null
|
|
fi
|
|
if [ ! -x "$BIN" ]; then
|
|
echo "fuzz.sh: $BIN is missing (build failed, or --no-build without a build)" >&2
|
|
exit 2
|
|
fi
|
|
|
|
status=0
|
|
for t in "${TARGETS[@]}"; do
|
|
corpus="$ROOT/build/fuzz/corpus-$t"
|
|
artifacts="$ROOT/build/fuzz/artifacts-$t/"
|
|
mkdir -p "$corpus" "$artifacts"
|
|
cp -R "$ROOT/tests/fuzz/corpus/$t/." "$corpus/"
|
|
args=(-artifact_prefix="$artifacts" -seed="$SEED" -rss_limit_mb=4096 -timeout=30 -print_final_stats=1)
|
|
if [ -n "$RUNS" ]; then
|
|
args+=(-runs="$RUNS")
|
|
else
|
|
args+=(-max_total_time="$MAX_TIME")
|
|
fi
|
|
if [ "$JOBS" -gt 1 ]; then
|
|
args+=(-jobs="$JOBS" -workers="$JOBS")
|
|
fi
|
|
echo "=== fuzz $t: ${RUNS:+$RUNS runs}${MAX_TIME:+$MAX_TIME s} (seed $SEED) ==="
|
|
if ! CBM_FUZZ_TARGET="$t" "$BIN" "${args[@]}" "$corpus"; then
|
|
echo "fuzz.sh: target $t FAILED -- reproducer in $artifacts" >&2
|
|
status=1
|
|
fi
|
|
done
|
|
exit "$status"
|