Merge upstream/main at 2469ecc3 into integrate-upstream-split-search-tools while retaining branch-specific incremental indexing, MCP API consolidation, and graph extraction behavior.
The staged tree adds upstream's Mojo and Perl grammar support, issue reproduction suites, Graph UI changes, Windows smoke coverage, TOON graph output, subprocess supervision, and Laravel facade route handling. Conflict resolutions in src/pipeline/httplink.c and pass_httplinks.c retain dynamic growth and IMPORTS edge identity; scripts/smoke-test.sh and scripts/soak-test.sh retain hermetic cache isolation.
Validation against the merge plus the follow-up working-tree reconciliation: ASan/UBSan MCP suite 216/216 passed. Follow-up semantic and concurrency fixes remain intentionally unstaged for the next focused commit.
Measure idle CPU from process CPU-time deltas instead of the lifetime average reported by ps. Expose user and system CPU counters in diagnostics, validate configurable idle and response limits, and fail closed when the server or its protocol pipe exits.
Disable session-CWD auto-indexing inside the explicit-index soak harness so unrelated repository work cannot contend for the pipeline lock, inflate cache/RSS results, or masquerade as idle CPU. Preserve crash/restart coverage and exact cleanup of harness-owned resources.
Validated with the ASan/UBSan MCP suite (171 tests), a bounded 30-second idle soak with crash recovery (0.0% idle CPU, 197 MiB RSS, stable cache/FDs), shell syntax and ShellCheck, source-safety lint, clang-format diff, configuration rejection cases, and diff hygiene.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Restore the upstream query-only leak workflow and timeout budgets so multi-hour jobs are neither dormant nor truncated. Preserve mixed-workload behavior while adding a dedicated read-only mode across Linux, macOS, and Windows.
Run every soak in an isolated project/cache/FIFO root, clean owned servers and diagnostics on normal or abnormal exit, and convert MSYS paths before placing them in JSON or environment variables. Validate initialize and tool responses, retain failure state across crash recovery, inject the crash before waiting for an index response, and require a checked post-restart reindex.
Record cache, database, and WAL sizes alongside RSS and FD metrics. Use post-index peak/final ratios and long-run slope as the portable default memory gates; keep an explicit CBM_SOAK_RSS_MAX_MB override for measured platform budgets instead of the obsolete universal 200 MB ceiling.
Validation:
- bash -n scripts/soak-test.sh
- shellcheck -x scripts/soak-test.sh
- actionlint on soak workflows
- source-safety and diff hygiene
- zero-minute product lifecycle smoke: passed
- asynchronous crash/restart/reindex smoke: passed
- one-minute query-leak run: 106 RPCs, zero failures
- invalid config and failed-server cleanup paths: rejected and cleaned
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Make install.ps1 preserve an existing executable before replacement, treat nonzero native --version status as verification failure, and restore the prior binary when copy or verification fails. Abort safely if the existing executable cannot be preserved instead of risking an unrollbackable overwrite.
Remove the rename-aside backup only after the new binary verifies and place the complete download/extract/install flow under a finally cleanup so temporary cbm-install directories are removed on success, explicit exits, and unexpected errors.
Strengthen the existing Windows smoke path to seed an installed binary, fail on installer errors, and reject an orphaned .old backup after successful replacement.
Validation: bash syntax; source-safety and protocol stdout checks; isolated install security audit; repository PowerShell grammar parse with zero extraction errors; diff hygiene. Native PowerShell replacement execution remains covered by the Windows CI smoke job because PowerShell is unavailable on this host.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
The 8e gate compared uname -s to the bare literal MINGW64_NT, which
never matches the real value (MINGW64_NT-10.0-...), so the POSIX
branch silently ran on Windows and looked for the extensionless shim
that #929 no longer installs. Branch by platform prefix: Windows
expects the .cmd shim and the absence of the legacy extensionless
twin; POSIX unchanged. Content checks (never blocks, delegates to
hook-augment) now run on both.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
cbm_writer_open used a raw ANSI-CP fopen for the hand-rolled SQLite
writer - the one file-creating call on the dump chain without a
UTF-8-to-wide conversion. On a non-ASCII Windows profile
(C:\Users\Kovacs Janos) extract and resolve succeeded but the dump
failed with pipeline.err phase=dump: fopen could not create graph.db
under the profile-derived cache dir while cbm_mkdir_p (already wide)
had happily created the directory around it. Routed through cbm_fopen,
matching #700 (pass readers) and #973 (canonicalization); POSIX is
byte-transparent.
New Windows guard test_non_ascii_cache_dump.py indexes an ASCII repo
into a non-ASCII CBM_CACHE_DIR and requires a query_graph readback
with Function count > 0 - proving the DB was written to and reopened
from the non-ASCII cache, not merely that no error surfaced. The
existing non-ASCII guard only varied the repo path against an ASCII
cache, which is why this class went unseen.
The USERPROFILE-sourced env encoding question (Layer 2 - _environ is
ANSI-CP, not UTF-8) is deliberately not bundled; it needs verification
on a real Windows box and will be filed separately if the guard alone
does not settle it.
Closes#996
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
SIGABRT (free of a non-libmalloc pointer in ts_stack_delete) on the
second index_repository in one process, whenever a sequential run is
followed by a parallel one. The sequential path parses on the CALLING
thread and left its TLS parser alive; the next parallel run switches
the global tree-sitter allocator to the slab (cbm_slab_install), the
calling thread participates in extraction, reuses the stale parser,
and its per-file teardown frees mimalloc-epoch memory through
slab_free's foreign-pointer fallback -> plain free() -> libmalloc
abort. The v0.9.0 supervisor masks this on the default MCP path (fresh
worker process per index); the in-process path and every embedded
consumer died. Root-caused with allocator-epoch instrumentation: the
victim parser is created at epoch 0 in cbm_pipeline_pass_definitions
and deleted at epoch 1 in extract_worker.
Two boundary fixes:
- run_sequential_pipeline destroys the calling thread's parser at run
end (also bounds its memory like the parallel workers already do);
- cbm_parallel_extract destroys the calling thread's parser BEFORE
cbm_slab_install, so no parser ever crosses the allocator switch.
Guards: prod-tier smoke invariant inv_second_index_inprocess
(sequential-then-parallel double index over stdio, RED with signal 6
on the unfixed binary — ASan builds mask the allocator seam, so the
unit tier cannot catch this class) + suite-tier fork-isolated
index_second_inprocess_run_survives_issue773 pinning the in-process
double-index contract. Repro 5/5 clean after the fix (was 5/5 abort).
Closes#773
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Completes the compact-output pass across the remaining query tools
(consumers are LLM agents; every response byte is context tokens):
- trace_path: callees[N]{qn,hop}/callers[N]{qn,hop} tables (risk/test/
args columns per flags); the per-hop JSON key envelope was 84% of the
payload. 3,589B -> 1,634B on a representative trace.
- query_graph: rows[N]{cols}: table (columns double as the header);
format:"json" restores columns/rows arrays.
- search_code compact mode: results/raw/dirs tables; drops the
field (duplicate of the qn tail). 2,488B -> 1,135B for 17 hits.
- search_code full mode: per-hit source capped at a 60-line window
anchored on the first match, with source_start/source_truncated
markers — uncapped whole-symbol dumps ran to 142KB per response; the
complete symbol stays one get_code_snippet call away.
- get_architecture: default (no aspects) is now a summary (languages,
packages, entry_points + always-on totals/label/type counts) with an
aspects_hint — the old default rendered EVERYTHING including the full
file_tree (94KB -> 4.5KB -> 2.8KB with TOON tables); all sections
emit as TOON tables, aspects:[...] and ["all"] keep full access
(52KB vs 94KB even for the explicit full dump).
- get_code_snippet: drops the property-blob enrichment (41% of the
response; signature/docstring are literally in the source; fp/sp/bt
never belonged there). Metrics remain reachable via search_graph
fields=[...].
- list_projects: per-project 12-field git block (mostly null) replaced
by the branch name; name/root/nodes/edges/size kept.
- search_graph description: in/out documented as TOTAL all-edge degree,
NOT caller/callee counts (agent evals mis-read it as callers).
- format:"json" escape hatch on every converted tool.
Guards: tool_output_byte_budgets pins absolute byte ceilings on default
search/trace outputs (a property blob sneaking back into row emission
blows them immediately); snippet_enriched_properties inverted to pin
the no-spill contract. Suite 5,984 green; smoke updated to the TOON
contract end-to-end.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Tool responses land in an LLM agent's context window, where the legacy
per-node JSON objects were dominated by data no agent can use: the
similarity/semantic pipeline intermediates fp (~450B minhash hex), bt
(body-token bag) and sp (structural profile) plus 14 complexity metrics
were grafted onto every result by enrich_node_properties — ~64% of every
node emission, ~370 tokens per search hit.
search_graph now defaults to TOON (Token-Oriented Object Notation,
toonformat.dev): scalars as 'key: value' lines and results as a
'results[N]{qn,label,file,lines,in,out}:' header + one row per hit.
Published benchmarks show equal-or-better LLM retrieval accuracy at
40-60% fewer tokens than JSON; rows now also carry line ranges, which
the legacy output lacked.
- fields:[...] opts into per-node property columns (complexity,
signature, docstring, ...); fp/sp/bt are blocklisted even there
- format:"json" restores the legacy verbose objects byte-identically;
include_connected forces it (nested neighbor lists)
- BM25 and semantic modes emit the same table shape (rank/score column)
- semantic-only calls no longer run the unfiltered regex search that
silently prepended up to 200 enriched nodes (102.8KB -> 4.2KB)
- default limit 200 -> 50 (BM25 100 -> 50): cheap first page, page via
has_more/offset as before
- new emitter module src/mcp/compact_out.{c,h} (string builder + TOON
scalar/table emission with spec quoting rules)
Measured on a real index (14-hit regex search): 20,793B -> 1,327B
(-94%); BM25 19,667B -> 5,081B; semantic 102,837B -> 4,181B. A typical
discovery session drops ~50.9K -> ~4.7K tokens combined with the
follow-up slices.
Guards: tool_search_graph_toon_never_leaks_internal_fields (sentinel
fp/sp/bt never emitted, even when requested), rewritten
tool_search_graph_includes_node_properties pins default-lean/fields-
opt-in/json-escape-hatch; suite 5983 passed, smoke green.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Perl LSP landed in #461; scan its implementation against PerlNavigator
(MIT, the leading OSS Perl language server) for defensive
copy-detection, same rationale as the phpactor entry. Verified clean:
no verbatim string/comment overlap.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Check 8d still locked the matcher to the pre-#963 'Grep|Glob' and failed
the pr-smoke leg on all platforms. The matcher now includes Read (the
augmenter injects the coverage note when a not-fully-indexed file is
read; structurally non-blocking, so the issue-#362 gate hazard cannot
recur). The guard now pins the exact new matcher and still rejects
Search/catch-all creep.
Refs #963
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
install.ps1 kept detecting arch=amd64 on windows-11-arm because it runs under x64
emulation, where neither $env:PROCESSOR_ARCHITECTURE nor .NET OSArchitecture (on
Windows PowerShell 5.1) reports the real Arm64. Add a CBM_ARCH env override to
install.ps1 (wins over auto-detect; also a genuine escape hatch for emulated
invocations) and have smoke Phase 13 pass the authoritative DL_ARCH. Deterministic
-- no reliance on in-process detection under emulation. Same emulated-arch class
as #907 (uname) and #908.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The persistent windows-11-arm smoke Phase 12a failure was a 404, not a network
error: it requested codebase-memory-mcp-windows-amd64.zip on the arm64 leg. Cause:
DL_ARCH came from `uname -m`, which on windows-11-arm is an emulated x86_64 MSYS2
uname reporting "x86_64" -> wrong (amd64) archive -> 404 (server has arm64). Phase
14 worked because the binary's own detect_arch() is native. Prefer SMOKE_ARCH
(passed from the smoke workflow's matrix.arch) over uname; fall back to uname for
local runs. This is the real cause the earlier curl/proxy/ipv4 attempts masked --
the 404 was swallowed by 2>/dev/null until #905 surfaced it.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Phase 12a swallowed curl's stderr (2>/dev/null), so the persistent windows-arm64
"curl download failed" was undiagnosable. Two changes: (1) add --noproxy '*' so
curl never routes the local 127.0.0.1 test server through a proxy env var -- a
strong candidate since the app's own WinHTTP downloader reaches the server in
Phase 14 while only msys2 curl fails, instantly; (2) print curl's stderr on
failure so the actual reason is visible if it persists. Harmless on all
platforms (the smoke server is always local).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Closes#423, #20. A non-ASCII repo path passed to
`codebase-memory-mcp cli index_repository "<json>"` was mangled on Windows, so the
command failed with "repo_path is required" instead of indexing the real directory.
POSIX is unaffected (argv is already UTF-8). Two boundaries had to be fixed:
1. argv read: main() took only the narrow `int main(int argc, char **argv)`, so the
CRT handed it argv in the active ANSI code page. On Windows main() now rebuilds
argv from the wide command line (GetCommandLineW + CommandLineToArgvW) and
converts each element to UTF-8 (cbm_wide_to_utf8).
2. worker spawn: the index supervisor (and the UI index thread) spawned the worker
via CreateProcessA, which re-interpreted the UTF-8 command line through the ANSI
code page -- re-mangling the path at the parent->worker boundary even after (1).
Both spawns now build a wide command line (cbm_utf8_to_wide) and use CreateProcessW.
Without (2) the default (supervisor-enabled) path stayed broken while only the
in-process path worked, so this is the complete end-to-end fix, not just argv.
Promotes tests/windows/test_cli_non_ascii_arg.py from an opt-in known-red to a green
Windows guard (scripts/test-windows.ps1 + the test-windows-guards CI job). The guard
now exercises the DEFAULT supervised path (it drops the suite's CBM_INDEX_SUPERVISOR=0
so it actually crosses the worker-spawn boundary). Verified on native Windows: RED
before the fix, GREEN after (a non-ASCII repo indexes, nodes>0). Also adds -lshell32
explicitly (main.c now uses CommandLineToArgvW) and refreshes RED_TEST_ANALYSIS.md.
Signed-off-by: Flipper <jacobphilipp@ymail.com>
Insert the FastAPI benchmark probe as an APIRouter method instead of a top-level function before an indented method declaration. The old mutation produced invalid Python and contaminated scoped-LSP overlay diagnostics with parser fallout.
Compile the mutated source before writing it so future FastAPI probe changes fail fast with a precise error instead of producing misleading benchmark data.
Validation: uv run python -m py_compile scripts/benchmark-incremental-speed.py; uv run python -m py_compile /private/tmp/cbm-fastapi-valid-probe-current-20260704T2250/fastapi_insert_probe/repo/fastapi/routing.py; bash scripts/check-source-safety.sh; git diff --check; FastAPI matrix /private/tmp/cbm-fastapi-valid-probe-current-20260704T2250.json passed with canonical_equal=true and publish_kind=full.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Add a real FastAPI matrix scenario that copies git-tracked HEAD files from an existing checkout into an isolated case repo, applies a routing.py probe insertion, and records source git metadata. Cloning is explicit opt-in via --clone-missing-real-repos; normal runs reuse --fastapi-repo, CBM_FASTAPI_REPO, or common local paths.
Current evidence: /private/tmp/cbm-fastapi-matrix-current-20260705T-postpatch.json passed with canonical_equal=true, publish_kind=full, exact_reason=frontier_too_large, and speedup_full_rebuild_over_incremental=0.43653250773993807. This proves the safe fallback path and preserves the FastAPI scoped-LSP performance blocker as a rerunnable benchmark target.
Validation: uv run python -m py_compile scripts/benchmark-incremental-speed.py; uv run python scripts/benchmark-incremental-speed.py --binary build/c/codebase-memory-mcp --matrix --matrix-scenarios fastapi_insert_probe --fastapi-repo /private/tmp/cbm-fastapi-cli-inbound-debug-20260705T-repo --work-root /private/tmp/cbm-fastapi-matrix-current-20260705T-postpatch --out /private/tmp/cbm-fastapi-matrix-current-20260705T-postpatch.json --timeout 240 --include-logs; bash scripts/check-source-safety.sh; git diff --check.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
libgit2 is licensed GPLv2-with-linking-exception, and this project is
deliberately GPL-free (see scripts/license-policy.json). libgit2 was only
an OPTIONAL, faster git-history code path guarded by HAVE_LIBGIT2 and
auto-detected via pkg-config; it always shipped with a popen("git log ...")
fallback that release binaries already used (they were built without
libgit2). Making that popen fallback the SOLE git-history implementation
means there is NO change to shipped behavior.
Changes:
- pass_githistory.c: collapse the HAVE_LIBGIT2 #ifdef so only the popen
parse_git_log remains; drop the now-unused <git2.h>/<time.h> includes.
- cbm.c / cbm.h / main.c / subprocess.c / index_supervisor.h: remove the
libgit2 mimalloc allocator bind and its >=1.7.0 version guards; keep the
tree-sitter + sqlite3 binds; correct the comments.
- Makefile.cbm: remove the optional-libgit2 pkg-config detection block,
the REQUIRE_LIBGIT2 error gate, and all LIBGIT2 CFLAGS/LIBS/FLAGS.
- .github/workflows/_test.yml: drop the REQUIRE_LIBGIT2 test-matrix leg
plus its libgit2-dev/pkg-config apt install (lower CI cost, no new
gating, no trigger change).
- flake.nix: drop libgit2/pkg-config from the dev shell (now-dead dep).
HAVE_LIBGIT2 no longer exists anywhere in the tree.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Distilled from #789 (safe features only). Adds backend dead-code classification
(status + in_calls in the layout JSON), a GET /api/repo-info endpoint for GitHub
deep-links, and the frontend dead-code filters, node code preview, and deep-links.
The render-cap revert was dropped (kept 2000 for DEFAULT_MAX_NODES/HARD_MAX_NODES
and GRAPH_RENDER_NODE_LIMIT); the sidebar regex-search refactor was omitted.
Security fixes over the original: repo-info strips credentials from any returned
remote_url; the legitimate https blob-URL construction is allow-listed so the
static gate passes; libgit2 is not re-initialized/shutdown per request (reuses the
process-wide init from cbm_alloc_init); deep-link path segments are URL-encoded.
Co-authored-by: Andy Zehady <azehady@ciroos.ai>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Split active overlay readability from file replacement tombstones. Replacement overlay publishes still hide canonical file rows, while the new additive publish API layers safe overlay facts without tombstoning the file.
Update the active-overlay CTE, node-view summary, and benchmark mirror so additive overlay rows are visible while canonical rows for the same file remain visible.
Add a store canary proving an additive overlay symbol is readable without creating a file tombstone or hiding existing canonical rows.
Validation: uv py_compile benchmark harness passed; ASan/UBSan test-runner build passed; store_nodes suite passed 108; source-safety passed; diff check clean; production build passed.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Project active overlay edges as logical graph edges instead of per-file overlay ownership rows. Cross-file edges can be owned by multiple changed files in one overlay generation for cleanup and compaction bookkeeping, but AI-facing active views must not inflate degree or schema counts.
Mirror the production active-edge semantics in the self-dogfood benchmark and add a store regression for duplicate cross-file overlay edge ownership.
Validation: CBM_ONLY_SUITE=store_nodes build/c/test-runner; make -f Makefile.cbm cbm; self-dogfood full matrix at /private/tmp/cbm-overlay-active-edge-dedupe-full-self-dogfood-20260704T.json; source-safety at /private/tmp/cbm-overlay-active-edge-dedupe-source-safety-20260704T.log.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Guard C-family header changes from active overlay publication until active edge parity is proven against fresh rebuilds. Bounded header edits can still use exact base-delta publication, preserving correctness without forcing every safe case to full rebuild.
Add a focused C-header incremental canary that verifies no overlay publish, exact frontier expansion, exact publish stats, and fresh-rebuild equality. Clean up stale direct-frontier helper code from the failed cap64 overlay experiment.
Validation: CBM_ONLY_SUITE=pipeline build/c/test-runner -> 323 passed; make -f Makefile.cbm lint-source-safety -> OK; self-dogfood cap16/cap64 store_pipeline_batch passed canonical equality but still fell back with frontier_too_large at 2.81x/2.74x.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
The index supervisor gate (cbm_index_supervisor_should_wrap) wrapped ANY
embedder of cbm_mcp_handle_tool: it resolved the CURRENT executable and
re-invoked it as '<self> cli --index-worker index_repository ...'. In a
test binary 'self' is the test runner, which ignores those args and
re-runs its suites instead, producing recursive spawn chains (observed
11-minute hangs, and multiplied kernel VM-map/memory load during the
2026-07-04 host panics). The gate is now opt-in: only a process that
called cbm_index_supervisor_mark_host() — the real binary's main(),
before any subcommand dispatch — may spawn workers. Embedders index
in-process by default; the test mains additionally pin
CBM_INDEX_SUPERVISOR=0 as belt-and-suspenders. Guarded by
index_supervisor_gate_requires_marked_host_issue845 (fork+alarm harness,
new spawn-count test hook; red on the old gate, green without the env
kill switch on the new one).
A supervised worker whose supervisor died also kept indexing as an
orphan: the parent-death watchdog only ran in MCP-server mode. run_cli
now starts the same watchdog (detached; on ppid change it only writes
to stderr and _exit(0)s, so it is safe outside server mode) when it
detects --index-worker. Guarded by tests/test_worker_watchdog.sh
(scripts/test.sh Step 5b): a CBM_TEST_HANG_ON busy-spin pins the worker
mid-index, the wrapper parent is killed, and the worker must self-exit
within 15s — red without the wiring, green with it.
Fixes#845.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Rebased onto current main and reworked in response to review. Three of the four
Windows reds were fixed upstream since the branch was cut at b075f05, so they are
now green regression guards; the fourth stays a genuine known-red.
- test_non_ascii_path.py (#636/#357): green guard - fixed by #700 (per-pass
readers now route through cbm_fopen -> _wfopen). Re-verified green on main.
- test_hook_augment.py (#618): green guard - fixed by #619 (cbm_is_walkable_abs_path
accepts drive-letter X:/ cwd). Re-verified green on main.
- test_ui_drive_listing.py (#548): rewritten. The fix exposes drives via a new
roots[] field, not the dirs[] array the old test asserted (which would stay red
against fixed code). Now asserts every fixed drive is in roots and browsable.
Re-verified green on main (drives C:/D:/E:).
- test_cli_non_ascii_arg.py (#423/#20): unchanged - main() is still narrow-argv
with no wide command line, so this remains genuinely red (the keeper).
- scripts/test-windows.ps1: split green guards (gate CI) from opt-in known-reds;
add -GuardsOnly; run indexing in-process (CBM_INDEX_SUPERVISOR=0) so a guard
reflects the path/hook/drive fix under test, not the index-worker spawn path.
- .github/workflows/_test.yml: new test-windows-guards job builds the product+UI
binary (scripts/build.sh --with-ui) and runs the guards with -GuardsOnly so
#700/#619/#548 stay enforced on Windows CI.
- RED_TEST_ANALYSIS.md: refreshed to record the landed fixes and current status.
Signed-off-by: Flipper <jacobphilipp@ymail.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Extends the Windows red-test suite with two more reproduced, Windows-specific
failures. No production fixes.
- windows_hook_augment_emits_context (integration, #618): the PreToolUse
Grep/Glob augmenter `hook-augment` emits empty stdout for every payload on
Windows. src/cli/hook_augment.c gates on POSIX-style absolute paths
(cwd[0] == '/' and the walk-up loop's dir[0] == '/'), which a Windows
drive-letter cwd never satisfies, so the graph augmentation never fires. A
control search_graph confirms the symbol is indexed.
- windows_ui_picker_reaches_all_drives (integration, #548): the UI directory
picker's GET /api/browse?path=/ returns no entries and never enumerates
logical drives, so drives other than the system drive (D:\, E:\) cannot be
selected. handle_browse in src/ui/http_server.c uses opendir without a
GetLogicalDriveStrings root case. Needs a UI build (cbm-with-ui) and >1 drive;
otherwise it reports a precondition (exit 2).
Also records additional ruled-out seed areas (get_code_snippet sanitizes
non-UTF-8 to U+FFFD #530.3; stdio handshake/flush works #513/#530.1/#635; mapped
subst-drive indexing keeps the DB #227/#367) and cross-platform items left out
of this Windows-only PR (#530.2 nested gitignore, #530.5 .git/info/exclude,
#530.4 libgit2 build, #581 memory soak).
Signed-off-by: Flipper <jacobphilipp@ymail.com>
Adds Windows-only red tests and analysis for native Windows failures found
during a Windows red-test campaign. This change contains no production fixes.
- windows_non_ascii_repo_path_preserves_definitions (integration): byte-identical
TypeScript fixtures indexed under non-ASCII parent paths (Latin-1 accents,
Cyrillic, CJK, Greek) extract zero definitions and only File/Folder nodes
(5 nodes / 4 edges) versus the ASCII baseline (12 nodes / 20 edges / 5
definitions). The pipeline source readers open files with fopen() on a UTF-8
path, which the Windows CRT interprets in the active ANSI code page; directory
discovery already uses the wide API, so files are listed but never parsed.
- windows_cli_non_ascii_repo_path_is_honored (integration): the documented
`cli index_repository` entrypoint rejects a non-ASCII repo_path because main()
does not read the wide command line, so argv arrives in the ANSI code page.
Both reproduce at the product surface (real MCP process, real stdio, real SQLite
DB), are deterministic, and pass on Linux/macOS. A PowerShell runner builds the
binary and runs the suite; standard-library Python only. See
tests/windows/RED_TEST_ANALYSIS.md for environment, commands, ruled-out seed
areas, and suspected fix locations.
Signed-off-by: Flipper <jacobphilipp@ymail.com>
Add an active-overlay graph comparator to the incremental benchmark harness and use it as the structural gate for incremental_overlay rows. Canonical/noop/exact/containment rows continue to use canonical graph equality.
This caught the same-stem File-node collision that answer-level MCP oracles missed, and the follow-up route-handler benchmark now passes with active_overlay_graph.equal=true.
Validation: uv run python -m py_compile scripts/benchmark-incremental-speed.py; /private/tmp/cbm-pan87-active-graph-route-handler-mcp-after-file-qn-20260704T.json passed with 1842 ms incremental vs 22607 ms fresh full, 12.27x speedup.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Keep canonical graph equality as the default graph gate, but let self-dogfood incremental_overlay rows pass on active answer oracles and freshness metadata because overlay publishes intentionally do not mutate canonical rows.
Record the selected graph_gate in benchmark output so maintainers can see whether a row used canonical_graph or overlay_active_oracles policy.
Validation: uv run python -m py_compile scripts/benchmark-incremental-speed.py; route_handler MCP self-dogfood rerun passed with publish_kind=incremental_overlay, graph_gate=overlay_active_oracles, active oracles passed, cleanup removed, and 13.09x speedup.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Add a default-off overlay_publish policy for bounded exact-delta incremental runs. When enabled, eligible non-noop exact deltas publish ready overlay rows and dirty freshness without mutating canonical graph rows, while default canonical behavior remains unchanged.
Use the latest complete canonical generation as the overlay base, avoid reserving a canonical generation for successful overlay publishes, and report incremental_overlay in pipeline telemetry and benchmark parsing.
Add focused store and pipeline canaries for latest complete generation lookup, opt-in overlay visibility, canonical invisibility, dirty overlay_ready state, and absence of stray reserved canonical generations.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
- format the include trailing comments and wrap the over-long reap log line
(clang-format-20 violations flagged by lint / lint)
- allowlist subprocess.c's fork with justification (security-static): the
supervisor primitive forks and execs immediately; no code runs in the
forked image
- move bin into the fork-child block alongside argv (cppcheck variableScope
consistency)
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
On NixOS (and other non-FHS systems) /bin/bash does not exist, so
scripts with an absolute shebang fail to run. Switch the remaining
holdouts to /usr/bin/env bash: eleven scripts/*.sh,
test-infrastructure/run.sh, and the three Claude Code hook scripts
emitted by src/cli/cli.c (gate, session reminder, subagent reminder).
Distilled from PR #674, with parser-test coverage preserved: the
infra_parse_shell* fixtures in tests/test_pipeline.c intentionally keep
#!/bin/bash so absolute-path shebang extraction stays covered, and
tests/repro fixtures are untouched.
Also replace the GitHub-PAT-shaped fixture string flagged in the #674
thread with an obviously fake placeholder (ghp_FAKE...) that still
matches the ghp_ + 36-alnum secret detector.
Co-authored-by: Sandro Jäckel <sandro.jaeckel@gmail.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Completes the crash/hang supervisor. A file that makes the indexer make NO
progress (an external-scanner infinite loop, not a crash) is now contained the
same way a crash is: the supervisor's no-progress quiet-timeout kills the worker,
classifies it as a hang, and the recovery loop quarantines the marker-pinned file
and re-runs until a clean run indexes the good files.
- src/mcp/index_supervisor.c: worker_quiet_timeout_ms() defaults to a generous
15 min NO-PROGRESS timeout (reset by the worker's periodic log heartbeat), with
the CBM_INDEX_WORKER_TIMEOUT_S override. (--progress deliberately NOT added: it
installs a replace-mode sink + carriage-return updates that would remove the
newline heartbeat the quiet-timeout relies on.)
- Quarantine entries now carry the phase ("rel\tphase"): a fault signal -> "crash",
a no-progress kill -> "hang". internal/cbm/cbm.c parses it (bare path tolerated,
defaults crash) and exposes cbm_index_quarantine_phase(); the extract loops
(pass_parallel.c, pass_definitions.c) report the real phase + reason in skipped[].
- src/mcp/mcp.c: recovery loop logs the outcome, and a best-effort-PARTIAL terminal
spawn — when the loop cannot converge but files were quarantined, one final
quarantine-only run yields a partial index (good files indexed, known-bad ones
skipped) + a high-severity index.supervisor.partial log, instead of a hard error.
- scripts/smoke-invariants.sh: inv_hanger_skipped_cli (hang twin of the crasher
guard — baseline rc=124 vacuity check; supervised -> status indexed + hanger
phase=hang + good file indexed). Hardened run_bounded to force-kill (SIGKILL) a
SIGTERM-catching busy-spin and to not block on the fallback fifo open — the hang
guard is the first invariant to actually hang and exposed this latent harness bug.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Stage 3b contained a crash to the worker but returned a failure response. Now the
supervisor pins the exact crashing file, quarantines it, and re-indexes the
healthy files: index_repository returns status="indexed" with the crasher listed
in skipped[] (phase="crash") and every other file indexed normally.
On a crash/hang the supervisor re-runs the worker single-threaded with a per-file
marker (CBM_INDEX_MARKER_FILE) so the marker names the EXACT crasher; that file is
appended to a quarantine list (CBM_INDEX_QUARANTINE_FILE) and the worker is
re-spawned until a clean run indexes the good files. Bounded by
CBM_INDEX_MAX_RESTARTS (default 100); each crash quarantines one new file, so the
loop makes monotonic progress.
- internal/cbm/cbm.c/.h: per-file marker + a lazily-loaded crash-quarantine set,
with a HARD guard at the top of cbm_extract_file (the one choke point every
extract pass funnels through) that short-circuits a quarantined file to an empty
result BEFORE the marker/parser — so no re-extracting pass (sequential
pass_calls/usages/semantic on a cache miss) can crash on it again.
- src/pipeline/pipeline.c: effective_worker_count() honours CBM_INDEX_SINGLE_THREAD
(=1 worker, sequential path) for exact single-threaded attribution.
- src/pipeline/pass_parallel.c, pass_definitions.c: report a quarantined file as a
phase="crash" skip via the Stage-2 skip plumbing so it surfaces in skipped[].
- src/mcp/index_supervisor.{h,c}: cbm_index_spawn_worker gains single_thread /
marker / quarantine knobs, passed to the child as inherited env around the
sequential spawn.
- src/mcp/mcp.c: index_run_supervised recovery loop (attribute -> quarantine ->
re-run until clean; else a contained-failure response).
- scripts/smoke-invariants.sh: inv_crasher_skipped_cli upgrades the 3b guard to the
skip-continue contract (baseline crash escapes; supervised -> status indexed +
crasher phase=crash + good file indexed + good file NOT skipped).
In-run quarantine only (not persisted across separate index runs); content-hash
persistence + re-admit is a follow-up.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
tools/list is cursor-paginated (MCP_TOOLS_PAGE_SIZE=8): the server returns a
page of tools plus result.nextCursor, and a compliant client follows the cursor
to retrieve the rest. inv_tools_list only read the first page and expected all
14 tools in one response, so it failed with "got 8, expected 14" even though
the server is correct. Follow nextCursor across pages and assert the union.
No server change — the pagination behavior was already correct.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A single pathological file that hard-crashes the native indexer (SIGSEGV / stack
overflow / abort / tree-sitter assertion) previously took down the whole MCP
server or CLI (issue #668). index_repository now runs in a worker subprocess: the
parent reaps it, and a crash is contained — the parent survives and reports it
instead of dying.
- src/mcp/index_supervisor.{h,c}: worker-role state + cbm_index_spawn_worker(),
which spawns `<self> cli --index-worker index_repository <json> --response-out
<tmp>` via the subprocess primitive, reaps, and classifies. fork+exec only;
recursion is prevented by the --index-worker argv flag, never an ambient env
var (which could misfire and silently run unprotected in-process).
- src/mcp/mcp.c: a supervisor gate at the top of handle_index_repository wraps the
run unless this process is the worker or CBM_INDEX_SUPERVISOR=0 (kill switch).
Clean exit passes the worker's response through; a crash/hang returns a
contained-failure response; a spawn failure degrades to the in-process path.
- src/main.c: parse --index-worker / --response-out; the worker writes its result
to the response file for the parent to read back.
- src/ui/http_server.c: its existing index spawn now passes --index-worker so it
does not re-supervise (one isolation layer, no redundant process nesting).
- src/foundation/subprocess.c: the forked child redirects output with open()+dup2()
instead of freopen() — async-signal-safe, no malloc between fork and exec, since
the parent may be multithreaded (server threads + mimalloc/sqlite/libgit2 state).
- internal/cbm/cbm.c: CBM_TEST_CRASH_ON / CBM_TEST_HANG_ON deterministic fault
injectors (test-only, env-gated) so the guard is honest — green iff a real fault
is genuinely contained, not a fixture that may stop faulting.
- scripts/smoke-invariants.sh: inv_crasher_contained_cli asserts the crash escapes
as a signal without the supervisor (rc>=128) and is contained + reported with it
(rc<128, outcome=crash).
Precise skip-and-continue (quarantine the culprit file, index the rest) is layered
on in a follow-up; this change contains the crash and keeps the process alive.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Use sqlite3's online backup API when preserving benchmark snapshots instead of copying .db files directly. This keeps WAL-mode databases consistent for canonical graph comparisons and removes stale destination sidecars before writing the snapshot.
Validation:
- git diff --check
- uv run python -m py_compile scripts/benchmark-incremental-speed.py
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Add optional exact_delta metadata to index_repository responses so benchmark artifacts can distinguish changed, affected, and published exact-delta path counts without scraping logs. Keep the benchmark parser backward-compatible by reading response metadata first and stderr markers for older binaries.
Validation: make -j16 -f Makefile.cbm cbm; make -j16 -f Makefile.cbm build/c/test-runner; CBM_ONLY_SUITE=mcp CBM_ONLY_TEST=tool_index_repository_reports_incremental_containment_reason build/c/test-runner; CBM_ONLY_SUITE=mcp build/c/test-runner; bash scripts/check-source-safety.sh; uv run python -m py_compile scripts/benchmark-incremental-speed.py; MCP telemetry smoke /private/tmp/cbm-pan2-20260703T0353Z-exact-delta-telemetry-smoke.json.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Add an opt-in rank_refresh=stale_on_incremental policy while preserving eager as the default and stale_on_exact as exact-only. Replacement publishes now mark rank-derived views stale before refresh, and MCP index, autoindex, and watcher paths share a publish-kind mapper so refresh policy is applied consistently.
Validation: make -f Makefile.cbm -j8 cbm rc=0 (/private/tmp/cbm-rank-refresh-dry-build1.log); make -f Makefile.cbm -j8 build/c/test-runner rc=0 (/private/tmp/cbm-rank-refresh-dry-test-runner-build1.log); CBM_ONLY_SUITE=pagerank build/c/test-runner rc=0, 57/57 (/private/tmp/cbm-rank-refresh-dry-pagerank2.log); CBM_ONLY_SUITE=pipeline build/c/test-runner rc=0, 307/307 (/private/tmp/cbm-rank-refresh-dry-pipeline2.log); make -f Makefile.cbm lint-source-safety rc=0 (/private/tmp/cbm-rank-refresh-dry-source-safety1.log); git diff --check clean.
Benchmark note: stale_on_exact vs stale_on_incremental self-dogfood comparison is still pending because the escalated uv benchmark run was rejected by the environment usage limit. Historical baseline remains /private/tmp/cbm-pan6-no-late-fallback-selfdogfood-cap256.json.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Add config-backed exact delta changed/affected frontier limits while preserving the conservative defaults. Exact delete and upsert paths now read the effective limits through shared pipeline getters instead of hardcoded live caps.
Add TDD coverage for default clamping, registry entries, and a configured 5-file inbound frontier that publishes exact and matches a fresh rebuild. Extend the incremental benchmark harness with repeatable --config key=value overrides so cap experiments are reproducible.
Validation: ASan/UBSan test-runner build rc=0; CBM_ONLY_SUITE=pipeline passed 304/304; source-safety rc=0 after harness edit; production cbm build rc=0; benchmark pycompile rc=0. Self-dogfood one_source_file with incremental_exact_max_affected_paths=64 passed canonical/oracle/cleanup checks but still fell back to containment with exact_reason=inbound_edges_require_full and only 3.30x speedup, so incremental remains off by default.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Record subprocess return code, stdout/stderr byte counts and tails, plus raw stdout/stderr/meta files when benchmark CLI subprocesses fail. Surface the structured details in benchmark JSON reports so failed runs can be diagnosed without rerunning expensive cases.
Keep artifacts under each benchmark work root's failures directory, with a temp fallback only when no CBM_CACHE_DIR is present.
Validation: py_compile with cfile in /private/tmp passed, expected-failure wrapper produced error_detail.returncode=37 and raw artifacts, tiny success benchmark returned rc=0 and passed under --min-speedup 0. Temporary work roots and wrapper were cleaned after standalone evidence was saved.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Andrew Hundt