mirror of
https://github.com/colbymchenry/codegraph.git
synced 2026-11-14 22:43:17 +08:00
Colby MchenryandClaude Opus 5.5
1941caf7bc
scripts/add-lang/verify-extraction.mjs ran execFileSync('codegraph', ...).
On Windows, npm installs codegraph as a .cmd shim (so do npm link and
local-install.sh), and Node neither finds a .cmd by bare name nor runs
one without a shell. The check always died with "spawnSync codegraph
ENOENT" (exit 2), and /add-lang's bench.sh read that as broken
extraction and skipped its A/B.
The script now asks, in order:
- CODEGRAPH_BIN, when set. A .js, .mjs or .cjs path runs with the
current node; anything else runs directly (a .cmd or .bat through
cmd.exe).
- On Windows, the codegraph Windows itself would pick: the first PATH
folder holding codegraph under one of PATHEXT's runnable extensions.
A .cmd or .bat runs through cmd.exe (shell: true) as one command
string. Each token is quoted for the C runtime and caret-escaped for
cmd, so spaces, &, parentheses, a %NAME% and a trailing backslash in
the repo path arrive intact. An .exe or .com needs no shell.
- Otherwise codegraph by bare name, exactly as before. On macOS and
Linux without CODEGRAPH_BIN, the call and the output are unchanged.
bench.sh takes CODEGRAPH_BIN too. It makes a path absolute (indexing
runs from inside the repo), exports it so verify-extraction.mjs reads
the index with the same build, and runs its own --version and init calls
with it. Without it, bench.sh behaves exactly as before. The A/B's MCP
server stays run-all.sh's own CG_BIN.
Validation (Windows 11, Node 22.22.0):
- New __tests__/add-lang-verify-extraction.test.ts covers CODEGRAPH_BIN
with a .js stand-in (all platforms), a .cmd shim on PATH with an
awkward repo path (Windows only), and the bare-name call (macOS and
Linux only). With main's script both tests that run here fail ("could
not read codegraph status"); with the fix, 2 pass and 1 is skipped.
- The fixture from PR 1246's review (a codegraph.cmd on PATH): main
exits 2 with ENOENT; the fix reads the status.
- 13 scenarios with npm's own generated shims, and a real index in a
folder named "lua repo & co (x86) 100%": main fails every one that
needs the shim or CODEGRAPH_BIN, and the fix passes them with the
exact argv reaching codegraph. With no codegraph anywhere, both exit 2
the same way.
- bench.sh with the A/B stubbed: with npm link's shims on PATH, main's
copy skips the A/B (verify exit 2) and the fix reaches it (PASS).
CODEGRAPH_BIN as an absolute or a relative path reaches it too, and a
missing one stops with a clear message.
- With process.platform reported as linux or darwin, main's script and
the new one make the same execFileSync call and print the same output.
Raised by PR 1246 (@bslatner).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
80 lines
3.3 KiB
Bash
Executable File
80 lines
3.3 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# Add-lang benchmark for ONE repo:
|
|
# clone -> wipe+index (with the codegraph on PATH) -> verify extraction ->
|
|
# with/without retrieval A/B (reuses scripts/agent-eval/run-all.sh).
|
|
#
|
|
# Assumes the codegraph dev build is already built + linked on PATH — the skill
|
|
# runs `npm run build && ./scripts/local-install.sh` ONCE before looping repos.
|
|
# The A/B is skipped if extraction fails its critical checks (don't burn $ on a
|
|
# broken extractor); set FORCE_AB=1 to run it anyway.
|
|
#
|
|
# Usage: bench.sh <lang> <repo-name> <repo-url> "<question>" [headless|tmux|all]
|
|
# Env: CORPUS corpus dir (default /tmp/codegraph-corpus, shared with agent-eval)
|
|
# CODEGRAPH_BIN index and verify with this codegraph instead of the one on
|
|
# PATH: a binary, or a build's dist/bin/codegraph.js (run with
|
|
# node). Exported, so verify-extraction.mjs uses it too. The A/B's
|
|
# MCP server is run-all.sh's own CG_BIN (default: codegraph on PATH).
|
|
set -uo pipefail
|
|
|
|
LANG_TOKEN="${1:?usage: bench.sh <lang> <repo-name> <repo-url> \"<question>\" [mode]}"
|
|
NAME="${2:?repo-name required}"
|
|
URL="${3:?repo-url required}"
|
|
Q="${4:?question required}"
|
|
MODE="${5:-headless}"
|
|
|
|
HARNESS="$(cd "$(dirname "$0")" && pwd)"
|
|
AGENT_EVAL="$(cd "$HARNESS/../agent-eval" && pwd)"
|
|
CORPUS="${CORPUS:-/tmp/codegraph-corpus}"
|
|
REPO="$CORPUS/$NAME"
|
|
|
|
if [ -n "${CODEGRAPH_BIN:-}" ]; then
|
|
# A path is made absolute, since indexing runs from inside the repo.
|
|
if [ -e "$CODEGRAPH_BIN" ]; then
|
|
CODEGRAPH_BIN="$(cd "$(dirname "$CODEGRAPH_BIN")" && pwd)/$(basename "$CODEGRAPH_BIN")"
|
|
else
|
|
command -v "$CODEGRAPH_BIN" >/dev/null || { echo "CODEGRAPH_BIN not found: $CODEGRAPH_BIN"; exit 1; }
|
|
fi
|
|
export CODEGRAPH_BIN
|
|
# Stands in for the PATH codegraph in this script's own calls.
|
|
case "$CODEGRAPH_BIN" in
|
|
*.js|*.mjs|*.cjs) codegraph() { node "$CODEGRAPH_BIN" "$@"; } ;;
|
|
*) codegraph() { "$CODEGRAPH_BIN" "$@"; } ;;
|
|
esac
|
|
else
|
|
command -v codegraph >/dev/null || { echo "no codegraph on PATH (build + ./scripts/local-install.sh first)"; exit 1; }
|
|
fi
|
|
|
|
echo "==================== add-lang bench: $NAME ($LANG_TOKEN) ===================="
|
|
echo "codegraph: ${CODEGRAPH_BIN:-$(command -v codegraph)} -> $(codegraph --version 2>/dev/null || echo '?')"
|
|
|
|
# 1. Ensure the repo (shallow clone, reuse if present).
|
|
mkdir -p "$CORPUS"
|
|
if [ -d "$REPO/.git" ]; then
|
|
echo "→ reusing checkout: $REPO"
|
|
else
|
|
echo "→ cloning $URL"
|
|
git clone --depth 1 "$URL" "$REPO" || { echo "git clone failed"; exit 1; }
|
|
fi
|
|
|
|
# 2. Wipe + index with the binary under test.
|
|
echo "→ wiping .codegraph and indexing"
|
|
rm -rf "$REPO/.codegraph"
|
|
( cd "$REPO" && codegraph init -i ) || { echo "indexing failed"; exit 1; }
|
|
|
|
# 3. Verify extraction (cheap guard before the paid A/B).
|
|
echo "→ verifying extraction"
|
|
node "$HARNESS/verify-extraction.mjs" "$REPO" "$LANG_TOKEN"
|
|
VERIFY=$?
|
|
|
|
# 4. Retrieval A/B (skipped if extraction is broken, unless FORCE_AB=1).
|
|
if [ "$VERIFY" -ne 0 ] && [ "${FORCE_AB:-0}" != "1" ]; then
|
|
echo "→ SKIPPING A/B — extraction failed critical checks (set FORCE_AB=1 to override)"
|
|
else
|
|
echo "→ retrieval A/B (mode=$MODE)"
|
|
bash "$AGENT_EVAL/run-all.sh" "$REPO" "$Q" "$MODE"
|
|
fi
|
|
|
|
echo "==================== bench complete: $NAME (verify exit=$VERIFY) ===================="
|
|
# Exit reflects extraction: 0 = pass/warn, 1 = critical fail, 2 = couldn't read status.
|
|
exit "$VERIFY"
|