PR CI ran tests but never built the binary or smoke-tested it, so built-binary
regressions only surfaced at the release dry run — e.g. the Windows
CreateProcess argv-quoting bug (worker got mangled JSON) slipped all the way to
the release gate.
Add a path-gated light smoke: on PRs that touch product code (src/, internal/,
Makefile.cbm, build.sh, smoke-test.sh, env.sh), build the PRODUCTION binary and
run scripts/smoke-test.sh on ubuntu-latest + macos-14 + windows-latest (native,
reliable — not the flaky broad/emulated legs). Every leg gates. Docs/CI/test PRs
skip it and stay fast (gh pr diff --name-only filter). The full broad/emulated
smoke stays in the release dry run.
SMOKE_DOWNLOAD_URL is intentionally unset so smoke-test.sh's download/update
phases self-skip; the core index/search/trace phases (which catch the
CreateProcess class) still run. ci-ok now treats skipped as OK so PRs that skip
smoke still pass.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The release dry run must be all-green on every platform — drop the
optional/continue-on-error flags from the broad matrix legs (ubuntu-22.04,
ubuntu-22.04-arm, macos-15, windows-2025) so a failure on any of them gates
the run. Only affects broad_platforms=true (the dry run); the core PR-CI
matrix is unchanged.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
mem_rss_reflects_external_resident_memory asserted cbm_mem_rss() >= 128MB
after touching a 256MB region, but on Windows cbm_mem_rss() reads
WorkingSetSize (GetProcessMemoryInfo), which the OS trims under memory
pressure — a stressed windows-11-arm runner kept only ~97MB resident and
the leg (a required gate) flaked. Re-touch the region immediately before
measuring and assert a 32MB Windows threshold that survives aggressive
trimming while staying far above the ~1MB mimalloc warm buffer, so it still
guards the real regression (a broken small-counter RSS). The Linux #else
branch keeps its 128MB assertion (the actual undercount it was written for).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
cbm_mem_peak_rss() returned mimalloc's peak_rss (from getrusage ru_maxrss,
KB-granular) while on Linux cbm_mem_rss() reads the live /proc/self/statm
value (page-granular, the RSS-undercount fallback). The two sources can
disagree by a few pages, so a live current read could momentarily exceed the
reported peak and break the definitional peak >= current invariant — flaking
the mem_peak_rss_gte_rss test on the Linux/ARM CI leg (observed as peak 172KB
under current). Reconcile the sources so the reported peak is at least the
current RSS. Not observable on macOS, where both come from mimalloc.
The mem_peak_rss_gte_rss guard now faults in a 32MB buffer before asserting so
the invariant is checked against a non-trivial live current read.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
libgit2 is licensed GPLv2-with-linking-exception, and this project is
deliberately GPL-free (see scripts/license-policy.json). libgit2 was only
an OPTIONAL, faster git-history code path guarded by HAVE_LIBGIT2 and
auto-detected via pkg-config; it always shipped with a popen("git log ...")
fallback that release binaries already used (they were built without
libgit2). Making that popen fallback the SOLE git-history implementation
means there is NO change to shipped behavior.
Changes:
- pass_githistory.c: collapse the HAVE_LIBGIT2 #ifdef so only the popen
parse_git_log remains; drop the now-unused <git2.h>/<time.h> includes.
- cbm.c / cbm.h / main.c / subprocess.c / index_supervisor.h: remove the
libgit2 mimalloc allocator bind and its >=1.7.0 version guards; keep the
tree-sitter + sqlite3 binds; correct the comments.
- Makefile.cbm: remove the optional-libgit2 pkg-config detection block,
the REQUIRE_LIBGIT2 error gate, and all LIBGIT2 CFLAGS/LIBS/FLAGS.
- .github/workflows/_test.yml: drop the REQUIRE_LIBGIT2 test-matrix leg
plus its libgit2-dev/pkg-config apt install (lower CI cost, no new
gating, no trigger change).
- flake.nix: drop libgit2/pkg-config from the dev shell (now-dead dep).
HAVE_LIBGIT2 no longer exists anywhere in the tree.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The SBOM hardcoded '158-grammars' for the tree-sitter-grammars aggregate;
the true count is now 159 (Mojo + InterSystems ObjectScript vendored, nim
dropped since the literal was last set), so the shipped SBOM under-counted.
Compute the count from the vendored grammar directories so it cannot drift
again. Also correct the non-MIT summary, which named only clojure CC0-1.0 +
jinja2/just Apache-2.0 while omitting fennel (CC0), the other eight Apache
grammars, fish (Unlicense) and pine/templ (ISC); it now lists the full
non-MIT family set and is explicitly non-exhaustive, deferring to
MANIFEST.md as authoritative. Data-only change to generated sbom.json; no
build or gating behavior changes.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The incr_full_index guard measured 2050-2072MB peak for the 1100-file
Python index across CI runs after the retention/source-text-cap and
RAM-tiering work shifted the x86 baseline up. The old 2048MB limit sat
right on the line and flaked marginally (2050 vs 2048). Raise the x86
base budget to 2304MB to restore headroom; a genuine leak (GBs over)
still trips it, and the ARM/large-page bumps to 2816 are unchanged.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Distilled from #789 (safe features only). Adds backend dead-code classification
(status + in_calls in the layout JSON), a GET /api/repo-info endpoint for GitHub
deep-links, and the frontend dead-code filters, node code preview, and deep-links.
The render-cap revert was dropped (kept 2000 for DEFAULT_MAX_NODES/HARD_MAX_NODES
and GRAPH_RENDER_NODE_LIMIT); the sidebar regex-search refactor was omitted.
Security fixes over the original: repo-info strips credentials from any returned
remote_url; the legitimate https blob-URL construction is allow-listed so the
static gate passes; libgit2 is not re-initialized/shutdown per request (reuses the
process-wide init from cbm_alloc_init); deep-link path segments are URL-encoded.
Co-authored-by: Andy Zehady <azehady@ciroos.ai>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Distilled from #776's 132460f5. This project sets arena_eager_commit=0 +
purge_decommits=1 + purge_delay=0 in cbm_mem_init to reduce upfront memory, so
mimalloc's committed-page counter reads low. On Linux mi_process_info() never
sets current_rss (vendored/mimalloc/src/prim/unix/prim.c only fills peak_rss),
so current_rss defaults to that low committed counter — cbm_mem_rss() returned a
few MB while true RSS was multiple GB, leaving cbm_mem_over_budget() backpressure,
the memory ceiling, and the host RAM tier blind on Linux.
Prefer os_rss() (/proc/self/statm) as the primary source on Linux; macOS and
Windows are unchanged (their mi_process_info current_rss is accurate via
task_info / GetProcessMemoryInfo). cbm_mem_peak_rss() is untouched. Complements
#752 by making the RAM tiers actually bite on Linux.
Reproduce-first: mem_rss_reflects_external_resident_memory pins mimalloc's
committed counter low with a live mi_malloc, then grows true RSS via a raw 256MB
mmap (invisible to mimalloc); unfixed Linux returns the ~few-MB committed counter
(RED), fixed Linux returns /proc RSS (GREEN). macOS/Windows pass either way, so
the RED manifests on the Linux CI leg.
Co-authored-by: petercoxphoto <info@petercox.ie>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Distilled from #782 keeping its retire-on-live_count correctness, replacing the
global-lock+linear-scan with aligned-page O(1) lookup + per-page lock-free
remote-free queue (no hot-path contention). Each 64KB slab page is allocated
aligned to its own size, so the owning page of any chunk is recovered by masking
the pointer; a lock-free 3-level radix page map (cold-path writes only) tells
slab_free whether a masked base is a real slab page or a plain heap pointer
without dereferencing an unrelated address. Owner frees hit the thread-local
free list; foreign frees CAS onto that page's lock-free MPSC remote-free stack.
Pages carry refcount = handed-out chunks + owner guard; reclaim/destroy retire a
page that still holds foreign-live chunks and free it when the final chunk
returns, so a page holding a live tree-sitter lexer chunk is never freed under
it. Keeps #782's resolve-worker terminal slab teardown.
Closes the #852 use-after-free and the cross-thread invalid free. Reproduce-first
guards in test_mem.c are RED on main (heap-use-after-free / bad-free under ASan)
and GREEN with this fix.
Closes#852
Supersedes #782
Co-authored-by: SS-42 <47749027+SS-42@users.noreply.github.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The idx832/idx853 fork-harness helpers are used only in the POSIX (#else)
branch of their tests; on Windows (SKIP_PLATFORM) they are unused →
-Werror,-Wunused-function on the CLANG64 leg. Guard the definitions to
match their call sites.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Inline Rust test fns (#[test], #[tokio::test], #[async_std::test],
#[actix_rt::test], #[test_case::case]) inside a regular .rs file were indexed
as ordinary Functions (is_test=false) and leaked past the store.c is_test!=1
filter into graph/agent context — test detection was file-path-based only
(cbm_is_test_file: *_test.rs / test_*). Add rust_def_is_test(decorators)
(mirroring rust_cfg_qualified_name's scan) and set def.is_test in
extract_func_def's Rust branch.
Closes#855.
Signed-off-by: Greg Tiller <tiller@dal.ca>
The #832 supervisor keystone added a second watcher-registration site in
autoindex_thread's supervised-success branch, but wired it as a direct
cbm_watcher_watch() guarded only by `if (srv->watcher)`. srv->watcher is set
unconditionally (cbm_mcp_server_set_watcher), so that guard does NOT honour the
auto_watch config. PR #849 had routed ALL watcher registration through
register_watcher_if_enabled() (which checks auto_watch_enabled() and logs
watcher.register.skipped reason=auto_watch_off). On a supervisor-marked prod
host with a fresh (unindexed) project and `config set auto_watch false`, the
supervised path therefore registered a background watcher against the user's
explicit opt-out — defeating #849's multi-project containment. The in-process
twin ~20 lines below already used register_watcher_if_enabled() correctly.
Fix: replace the ungated block in the supervised-success branch with
register_watcher_if_enabled(srv), matching the in-process branch. Surrounding
supervised-success logging is unchanged.
Reproduce-first: mcp_auto_watch_false_skips_supervised_autoindex_issue853 drives
the real public entry initialize -> maybe_auto_index -> autoindex_thread on a
supervisor-marked host (kill switch off) with a FRESH project and
auto_watch=false, in a fork+alarm-bounded child (like the #832/#845 gate tests).
cbm_mcp_server_free() joins the autoindex thread so the registration decision has
run before the watch count is read. It asserts spawn_count advanced (so the probe
cannot pass vacuously by skipping the supervised branch) and watch_count == 0.
RED on the unfixed code (watch_count == 1, exit 61); GREEN after the fix
(register_watcher_if_enabled skips → watcher.register.skipped reason=auto_watch_off).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Distilled from #685 (nguyentamdat) rebased onto current main, plus two
research-driven refinements and a genuine reproduce-first guard.
The parallel extract retains each file's source text so the fused cross-file
LSP resolve can re-parse it. That retention is transient but a peak-RSS driver.
On main the caps are flat (100 MiB/file, 2 GiB total) and a file over the cap is
silently unretained AND its cross-file resolution is skipped -- a graph-quality
gap. This change bounds retention AND keeps every cross-file edge.
- Source-text cap as a FLOOR, not just a ceiling: retention total defaults to
min(cbm_mem_budget()/8, 1 GiB), per-file min(32 MiB, total). Following the
rust-analyzer memory model, the RAM-derived default is clamped to a small
absolute ceiling so a huge-RAM host does not hold tens of GB it would re-read
cheaply. Both caps env-overridable via CBM_RETAIN_TOTAL_MB /
CBM_RETAIN_PER_FILE_MB (limits.c convention); ceilings bound only the
auto-derived default, never an explicit operator/caller choice. A dropped file
emits one index.retain_capped WARN per run.
- Bounded re-read fallback (the correctness guarantee): resolve_worker re-reads
an unretained file's source on demand (bounded, freed immediately) instead of
skipping resolution, wired at every cross-LSP site that consumes source. The
cap now only trades retained RAM for a bounded re-read, never a lost edge.
- cbm_parallel_extract_ex + opts struct (cbm_parallel_extract is now a wrapper
passing NULL -> env-derived defaults); malloc/calloc NULL-check hardening.
Reproduce-first: parallel_cross_file_reread_preserves_unretained_edges uses a
Java<->Kotlin pair whose cross-file lsp edges are genuinely source-dependent;
the edges are lost when the caller is unretained and the fallback is absent
(RED), present with it (GREEN), with a retained CONTROL scenario proving
non-vacuity. #685's original Python red test was a false guard (per-file py_lsp
already resolves those calls) and is replaced. Peak-bound guards
(retained_bytes <= total_cap; retain_sources=false retains nothing) in test_mem.c.
Verify: make -f Makefile.cbm cbm && make -f Makefile.cbm lint-ci; test-runner
parallel pipeline incremental py_lsp ts_lsp java_lsp kotlin_lsp c_lsp cs_lsp
go_lsp rust_lsp mem -> 2323 passed.
Co-authored-by: nguyentamdat <nguyentamdat@gmail.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The long-lived MCP server ran the full index pipeline in-process on two
background paths -- the session auto-index (autoindex_thread, mcp.c) and the
watcher re-index (watcher_index_fn, main.c). The parallel pipeline's worker
threads abandon their mimalloc pages at thread exit, and mimalloc v3 defaults
page_reclaim_on_free=0, so those pages are never reclaimed when the main thread
later frees their blocks (mi_collect cannot touch abandoned pages either). RSS
then ratchets across re-index cycles and never comes back down (#832).
The index_repository tool path already avoided this by running the index in a
supervised worker subprocess: on child exit the kernel reclaims 100% of the
child's RSS. Route the two background paths through that same supervised runner:
- supervisor_invalidate_store() is now NULL-safe (the watcher path has no MCP
server / cached store).
- New static index_run_supervised_path() builds {"repo_path": root} and reuses
the existing spawn + skip-and-continue recovery loop (no duplication). New
public cbm_mcp_index_run_supervised_path() is the srv-less entry main.c calls.
- autoindex_thread and watcher_index_fn gate on cbm_index_supervisor_should_wrap():
supervised on a marked host with the kill switch off, else fall through to the
unchanged in-process pipeline (kill switch / spawn-failure degrade). The
watcher keeps its non-blocking try_lock gate + g_shutdown check exactly; the
lock still serialises re-indexes. cbm_mem_collect() stays on the in-process
branch only.
Also fold in the in-process allocator fallback: cbm_mem_init sets
mi_option_page_reclaim_on_free=1, restoring v2 behaviour so any path that stays
in-process (kill switch, degrade, embedders) also reclaims abandoned worker
pages. The subprocess is the primary cure (child exit returns RSS every cycle);
this is the fallback.
Tests: a deterministic gating guard (test_mcp.c) proves the background paths now
spawn a worker child and index the fixture (RED when routed in-process:
spawn_count unchanged; GREEN after). The test-runner acts as a real in-process
worker when spawned as `cli --index-worker` (public APIs only, no production
seam). A non-gating prod-binary RSS reproduction (tests/repro/issue832_rss.py)
shows the supervised server flat at ~10 MB vs ~131 MB in-process.
Scope: JobObject, the #841 spurious-trigger fix, retention caps, budget tiering
and the backpressure busy-spin are follow-ups.
Refs #832, #841.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The fly-to animation moved only the camera and called lookAt, leaving the OrbitControls pivot at the origin, so OrbitControls re-centred the view on the next frame and snapped the camera back once the animation ended.
Lerp the controls target to the focus point as well so the view stays on the clicked node or cluster and orbits around it afterwards.
Signed-off-by: Zadak <rarepops@protonmail.com>
Rename the ambiguous Clear button to Clear selection so it reads distinctly from Refresh, and point the empty-state hint at the Projects tab instead of the renamed Stats tab.
Signed-off-by: Zadak <rarepops@protonmail.com>
Store the active tab and selected project in the URL query string so the view survives refreshes and can be bookmarked or shared, syncing back on browser back and forward via popstate.
Query params are used rather than path segments because the embedded server only serves index.html at the root path. Opening the Projects tab now clears the active project, and the Graph tab stays disabled until a project is selected.
Signed-off-by: Zadak <rarepops@protonmail.com>
Martin Vogel