Use sqlite3's online backup API when preserving benchmark snapshots instead of copying .db files directly. This keeps WAL-mode databases consistent for canonical graph comparisons and removes stale destination sidecars before writing the snapshot.
Validation:
- git diff --check
- uv run python -m py_compile scripts/benchmark-incremental-speed.py
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Add optional exact_delta metadata to index_repository responses so benchmark artifacts can distinguish changed, affected, and published exact-delta path counts without scraping logs. Keep the benchmark parser backward-compatible by reading response metadata first and stderr markers for older binaries.
Validation: make -j16 -f Makefile.cbm cbm; make -j16 -f Makefile.cbm build/c/test-runner; CBM_ONLY_SUITE=mcp CBM_ONLY_TEST=tool_index_repository_reports_incremental_containment_reason build/c/test-runner; CBM_ONLY_SUITE=mcp build/c/test-runner; bash scripts/check-source-safety.sh; uv run python -m py_compile scripts/benchmark-incremental-speed.py; MCP telemetry smoke /private/tmp/cbm-pan2-20260703T0353Z-exact-delta-telemetry-smoke.json.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Add an opt-in rank_refresh=stale_on_incremental policy while preserving eager as the default and stale_on_exact as exact-only. Replacement publishes now mark rank-derived views stale before refresh, and MCP index, autoindex, and watcher paths share a publish-kind mapper so refresh policy is applied consistently.
Validation: make -f Makefile.cbm -j8 cbm rc=0 (/private/tmp/cbm-rank-refresh-dry-build1.log); make -f Makefile.cbm -j8 build/c/test-runner rc=0 (/private/tmp/cbm-rank-refresh-dry-test-runner-build1.log); CBM_ONLY_SUITE=pagerank build/c/test-runner rc=0, 57/57 (/private/tmp/cbm-rank-refresh-dry-pagerank2.log); CBM_ONLY_SUITE=pipeline build/c/test-runner rc=0, 307/307 (/private/tmp/cbm-rank-refresh-dry-pipeline2.log); make -f Makefile.cbm lint-source-safety rc=0 (/private/tmp/cbm-rank-refresh-dry-source-safety1.log); git diff --check clean.
Benchmark note: stale_on_exact vs stale_on_incremental self-dogfood comparison is still pending because the escalated uv benchmark run was rejected by the environment usage limit. Historical baseline remains /private/tmp/cbm-pan6-no-late-fallback-selfdogfood-cap256.json.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Add config-backed exact delta changed/affected frontier limits while preserving the conservative defaults. Exact delete and upsert paths now read the effective limits through shared pipeline getters instead of hardcoded live caps.
Add TDD coverage for default clamping, registry entries, and a configured 5-file inbound frontier that publishes exact and matches a fresh rebuild. Extend the incremental benchmark harness with repeatable --config key=value overrides so cap experiments are reproducible.
Validation: ASan/UBSan test-runner build rc=0; CBM_ONLY_SUITE=pipeline passed 304/304; source-safety rc=0 after harness edit; production cbm build rc=0; benchmark pycompile rc=0. Self-dogfood one_source_file with incremental_exact_max_affected_paths=64 passed canonical/oracle/cleanup checks but still fell back to containment with exact_reason=inbound_edges_require_full and only 3.30x speedup, so incremental remains off by default.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Record subprocess return code, stdout/stderr byte counts and tails, plus raw stdout/stderr/meta files when benchmark CLI subprocesses fail. Surface the structured details in benchmark JSON reports so failed runs can be diagnosed without rerunning expensive cases.
Keep artifacts under each benchmark work root's failures directory, with a temp fallback only when no CBM_CACHE_DIR is present.
Validation: py_compile with cfile in /private/tmp passed, expected-failure wrapper produced error_detail.returncode=37 and raw artifacts, tiny success benchmark returned rc=0 and passed under --min-speedup 0. Temporary work roots and wrapper were cleaned after standalone evidence was saved.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Extend scripts/benchmark-incremental-speed.py with a real-repo self-dogfood mode that reuses the existing CLI/MCP transport, isolated cache setup, canonical graph comparison, publish-kind parsing, freshness capture, and cleanup reporting.
The new mode creates detached temporary git worktrees, applies deterministic CBM-prefixed marker edits, probes search_graph/search_code/query_graph/get_architecture/route freshness, records source git metadata, and resolves the default binary relative to either the caller cwd or the script repo root.
Also make canonical SQLite comparison surrogate-safe for real-repo rows and scope marker search_code probes with file_pattern plus path_filter so oracle latency stays practical.
Validation: no-bytecode compile log /private/tmp/cbm-pan4-20260702T-self-dogfood-compile-filepattern.log; git diff --check; create-path matrix smoke /private/tmp/cbm-pan4-20260702T-self-dogfood-harness-matrix-create-smoke.json passed with publish_kind=incremental_exact.
Current known blocker captured by the harness: /private/tmp/cbm-pan4-20260702T-self-dogfood-one-source-mcp-filepattern.json fails canonical equality for internal.cbm.ac.CBMAutomaton (containment ac.c vs fresh full ac.h), so broad self-dogfood expansion remains blocked pending root-cause work.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Add compatibility-preserving freshness metadata to stale derived-view MCP responses while preserving existing warnings text. Responses now include freshness.state=stale_with_warning and freshness.stale_views for the stale derived views already detected by search_graph, query_graph, trace_path, and get_architecture.
Update the existing incremental benchmark parser to retain freshness and freshness_state from tool responses instead of introducing a parallel harness.
Validation: make -f Makefile.cbm cbm; bash scripts/check-source-safety.sh; CBM_ONLY_SUITE=mcp 131/131; CBM_ONLY_SUITE=tool_consolidation 98/98; CBM_ONLY_SUITE=pipeline 298/298; CBM_ONLY_SUITE=pagerank 55/55; uv run python -m py_compile scripts/benchmark-incremental-speed.py. Dogfood: isolated CBM self-index in /private/tmp/cbm-pan2-dogfood-cache.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Route MCP existence-only probes through existing CBM file and directory helpers instead of raw access/stat checks. Reuse project_db_path for QN cold-start DB lookup, length-check ADR probes, and keep metadata-dependent stat calls unchanged.
Extend source-safety to block newly added raw access() and strndup() in production diffs, with self-tests for both guard paths.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Add cbm_str_copy() as the shared allocation-free fixed-buffer copy helper and use it in httplink, envscan, and store file-tree code.
Remove guarded production strncpy() sites, add a source-safety rule and self-test for future strncpy regressions, and tighten the logging thread-safety comment to match the implementation.
Validation: make -f Makefile.cbm lint-source-safety; CBM_ONLY_SUITE=httplink make -f Makefile.cbm test; CBM_ONLY_SUITE=str_util build/c/test-runner; CBM_ONLY_SUITE=pipeline CBM_ONLY_TEST=envscan build/c/test-runner; CBM_ONLY_SUITE=store_arch build/c/test-runner.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Detect exact file deltas whose canonical graph rows already match the persisted store, then refresh only file freshness and incremental metadata under a completed generation. This keeps graph-changing exact deltas on the existing planner/apply path while body-only edits publish as incremental_noop with graph_changed=false.
Also update the incremental benchmark harness so incremental_noop is treated as an explicit incremental route while preserving the existing explicit_exact_or_fallback field for compatibility.
Validation: git diff --check; bash scripts/check-source-safety.sh; bash scripts/test-source-safety.sh; python3 -m py_compile scripts/benchmark-incremental-speed.py; make -j8 -f Makefile.cbm build/c/test-runner; CBM_ONLY_SUITE=store_nodes 81 passed; CBM_ONLY_SUITE=pipeline 295 passed; CBM_ONLY_SUITE=incremental 161 passed; make -j8 -f Makefile.cbm cbm; persistent-MCP affected-frontier matrix 8/8 canonical-equal.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Add pipeline-owned publish_reason metadata and surface it in index_repository responses when incremental indexing falls back to containment. The reason is set on existing exact-route skip/fallback paths and cleared on exact publish/full run reset, so MCP and CLI JSON callers can distinguish a safe containment rebuild from an opaque incremental result.
Update the affected-frontier benchmark harness to prefer structured publish_reason while keeping stderr parsing for older binaries. Add focused pipeline assertions and an MCP tool test for incremental_containment with missing_existing_ownership.
Validated with git diff --check, bash scripts/check-source-safety.sh, bash scripts/test-source-safety.sh, python3 -m py_compile scripts/benchmark-incremental-speed.py, escalated make -j8 -f Makefile.cbm build/c/test-runner, escalated CBM_ONLY_SUITE=pipeline build/c/test-runner, escalated CBM_ONLY_SUITE=mcp build/c/test-runner, escalated make -j8 -f Makefile.cbm cbm, and escalated persistent MCP affected-frontier matrix 8/8 pass.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Extend the existing incremental benchmark with an opt-in --matrix mode for affected-frontier scenarios. The matrix isolates repos and caches per case, records publish_kind and exact/fallback reasons, compares incremental output to a fresh FAST rebuild with canonical SQLite graph rows, and cleans up generated work roots by default.
This is a measurement and regression harness, not a default-policy change. incremental_reindex remains default-off until the broader matrix, freshness semantics, cleanup, and speedup gates are satisfied.
Validated with python3 -m py_compile scripts/benchmark-incremental-speed.py, git diff --check, bash scripts/check-source-safety.sh, and an escalated two-case matrix smoke for go_modify_1/go_modify_2.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Add --transport=cli|mcp to the incremental speed benchmark. The MCP path keeps one JSON-RPC server process alive for each measurement phase and detects full versus exact incremental routes from index_repository publish_kind metadata instead of stderr logs.
The harness still preserves the cold CLI path and cleanup behavior, and reports publish_kind alongside existing wall/logged timing fields for current and future performance comparisons.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Add indexed_work_elapsed_ms and unlogged_overhead_ms to the incremental speed benchmark output while preserving the existing elapsed_ms and logged_elapsed_ms fields. This separates cold CLI wall time from the logged indexing path so future speedup work can target the correct latency term.
Validation: python3 -m py_compile scripts/benchmark-incremental-speed.py; git diff --check; scripts/benchmark-incremental-speed.py --files 20 --functions-per-file 4 --changed-files 1 --min-speedup 0.5 --timeout 120 --out /private/tmp/cbm-incr-speed-overhead-fields-smoke-pass.json
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Add an opt-in benchmark that compares fast-mode exact incremental indexing against a fresh fast full rebuild in an isolated temp repo and CBM_CACHE_DIR. The gate requires the incremental.exact.done marker, records rank-refresh markers and bounded log tails, and cleans only paths it creates.
This does not change defaults. The current implementation is expected to fail the default 10x threshold until exact-delta post-passes are scoped or marked stale safely.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Extend the source-safety self-test to prove fprintf(stdout, ...) is rejected in MCP/pipeline code, matching the existing protocol-output guard.
Reuse store.c's heap_strdup helper for neighbor-name and LIKE-hint allocations instead of direct strdup calls. This keeps ownership behavior unchanged while reducing allocator-wrapper drift in touched store paths.
Validation: make -j8 -f Makefile.cbm build/c/test-runner; CBM_ONLY_SUITE=store_search ./build/c/test-runner; CBM_ONLY_SUITE=mcp ./build/c/test-runner; bash scripts/check-source-safety.sh; bash scripts/test-source-safety.sh; git diff --check.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Replace raw strdup calls in the incremental inbound-edge snapshot with cbm_strdup so the touched path follows the repository portability wrapper convention. Add a source-safety diff guard and self-test that reject newly introduced raw strdup calls in production code without failing historical existing uses.
Validation: git diff --check; make -j8 -f Makefile.cbm build/c/test-runner; make -f Makefile.cbm lint-source-safety; CBM_ONLY_SUITE=pipeline ./build/c/test-runner; CBM_ONLY_SUITE=incremental ./build/c/test-runner.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Add a root-overridable self-test harness for scripts/check-source-safety.sh so the guard can be tested against synthetic trees without scanning the real repository.
Cover clean input, MCP/server stdout writes, unsafe string APIs, and new raw filesystem APIs in production diffs. Add a dedicated lint-source-safety target because the broader lint-no-suppress path is currently blocked by existing NOLINT audit failures before this guard can run.
Validation: bash scripts/test-source-safety.sh; bash scripts/check-source-safety.sh; git diff --check; make -f Makefile.cbm lint-source-safety.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Stabilize several graph-producing paths and add stronger regression coverage before the incremental redesign work continues.
Key changes:
- resolve call import maps from resolved graph-buffer IMPORTS edges so re-exported targets are reused consistently
- make semantic, similarity, complexity, route/section upserts, and resolver scoring more deterministic across full/incremental and worker-order differences
- propagate incremental phase/persist failures instead of continuing toward publication after failed phases
- add canonical graph diff helpers and stricter incremental artifacts for full-vs-incremental comparison
- add source-safety lint coverage for MCP stdout, unsafe string APIs, and new raw env/fs API usage
- replace touched unsafe string copies with bounded or project-wrapper-based forms
Validation:
- make -f Makefile.cbm build/c/test-runner
- CBM_ONLY_SUITE=pipeline ./build/c/test-runner: 217 passed
- CBM_ONLY_SUITE=graph_buffer ./build/c/test-runner: 55 passed
- CBM_ONLY_SUITE=mcp ./build/c/test-runner: 117 passed
- CBM_ONLY_SUITE=simhash ./build/c/test-runner: 24 passed
- CBM_ONLY_SUITE=parallel ./build/c/test-runner: 21 passed
- CBM_ONLY_SUITE=watcher ./build/c/test-runner: 57 passed
- bash scripts/check-source-safety.sh: OK
Known remaining:
- CBM_ONLY_SUITE=incremental still fails incr_accuracy_vs_full: 159 passed, 1 failed. Current artifacts show stale IMPORTS/CALLS/USAGE targets in unchanged importers and downstream SEMANTICALLY_RELATED drift; this is tracked in the forensic notes as a fallback-first incremental redesign task.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Deep --no-ff merge of upstream DeusData/main (597 commits, merge-base 988d975)
into fork api-consolidation (HEAD 3018b22), producing a superset that combines
upstream breadth with the fork's surviving depth.
Upstream additions integrated: semantic/embedding search (vendored/nomic),
minhash/SimHash, cross-repo + cross-file LSP + route + pkgmap passes, artifact
import/export, path_alias, git_context, hook_augment, Windows support
(compat_fs/compat_thread), the UI layer, and ~140 expanded tree-sitter grammars.
Fork depth retained: PageRank/LinkRank ranking (src/pagerank/; sort_by=
relevance/linkrank/degree/calls), dependency indexing (src/depindex/;
CBM_INDEX_MODE=DEP), token reduction, and HTTP-endpoint discovery
(src/pipeline/httplink.c, pass_httplinks.c).
Conflict resolution (27 regions across the 39-file intersection):
- src/mcp/mcp.c: kept the fork superset handler pattern (resolve_project_store +
expand_project_param + inject_context_once) as the single path; UNIONed the
tool dispatch table; emit status:"indexed", excluded[], cross_service trace.
- src/store/store.c: upstream base + fork pagerank/linkrank/node_degree tables
and sort_by plumbing; cbm_store_node_degree now excludes INHERITS
(WHERE type != 'INHERITS'); removed the duplicate cbm_store_get_db.
- src/pipeline/pipeline.c: took upstream run_extraction_phase; grafted the fork
passes (httplinks, normalize, flush_store dump branch); call
cbm_pipeline_set_committed_counts before cbm_gbuf_dump_to_sqlite.
- internal/cbm/lang_specs.c: 158 entries combining the fork section_node_types
with upstream ts_factory/embedded_imports; wired 13 orphaned type-arrays.
- src/depindex/depindex.h: renamed CBM_PATH_MAX -> CBM_DEP_PATH_MAX (it collided
with constants.h enum CBM_PATH_MAX=1024) + 11 call sites in depindex.c.
- Makefile.cbm: upstream grammar/nomic units + fork CGO targets; added
httplink.c + pass_httplinks.c to PIPELINE_SRCS; removed mongoose NOSAN
machinery (mongoose was deleted upstream).
- httplinks restored (the merge silently deleted them because the fork had left
them unchanged since the base): httplink.c/.h, pass_httplinks.c,
tests/test_httplink.c; registered RUN_SUITE(httplink) in tests/test_main.c.
- CBM_INDEX_MODE enum unified to FULL=0, MODERATE=1, FAST=2, DEP=3.
Completeness (dual-parent check): 0 upstream files dropped from the merge tree;
6 fork files dropped, all intentional (upstream-superseded): CHANGELOG.md,
vendored/mimalloc/src/{arena-abandon,segment-map,segment}.c,
vendored/mongoose/{mongoose.c,mongoose.h}. Fork features (pagerank, depindex,
token_reduction, httplink) verified present and registered in tests/test_main.c.
src/pipeline/pass_parallel.c has an identical function set to upstream (49=49,
none dropped); the large line-deltas vs upstream are within-function divergence
of the fork extractor, not dropped functions.
Gate (run manually; hooks skip on merge commits): build green
(make -f Makefile.cbm cbm, exit 0); full C test suite 5980 passed / 2 failed /
0 crashes. The fork's pre-existing ASan crash in build_project_list_error_srv is
fixed by upstream's mcp.c rewrite, so the merge is strictly better than the fork
baseline on the crash axis.
Known issue, tracked separately (NOT a merge defect): 2 nondeterministic
failures in tests/test_incremental.c — incr_db_deleted_recovery:772 (diff_pct)
and incr_accuracy_vs_full:803 (full_nodes vs incr_nodes). Cause: the fork's
extractor (internal/cbm/extract_*.c) and parallel-resolve Route-node creation
are not thread-safe under the concurrent use of upstream's parallel pipeline
(run_parallel_pipeline), producing nondeterministic route/infra HANDLES edges
(~234k edge swing between runs). Sequential extraction (CBM_WORKERS=1) is
deterministic and correct. Root cause, bisect, and repro are documented in
notes/2026-06-24-fork-vs-upstream-divergence-assessment.md (section 8f); the fix
is deferred to a dedicated thread-safety audit (the make test-tsan target is
currently a stub and needs wiring).
The MSYS2 CLANG64 toolchain bakes its package-tracker URL
(https://github.com/msys2/MINGW-packages) into the static Windows .exe via
libc++/compiler-rt. The Layer 2 binary string audit flagged it as an
unauthorized URL, failing smoke-windows (both the standard and ui variants)
while Linux smoke stayed green. Add it to ALLOWED_URLS alongside the other
compiler/linker toolchain URLs (gcc.gnu.org, sourceware.org, launchpad).
Add tests/test_security_strings_allowlist.sh as a reproduce-first regression
guard: it reproduces the BLOCKED failure with a binary fixture and includes a
negative control proving a genuinely unauthorized URL is still rejected. Wired
into scripts/test.sh as Step 6.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The UI license enumeration invoked npm by bare name via subprocess,
which fails on Windows (npm is npm.cmd there — WinError 2 in the
Windows UI-archive packaging step). Resolve it through the shell, which
handles platform shims everywhere; the command is a constant string.
Signed-off-by: Martin Vogel <martin.vogel@datadice.io>
It names license identifiers in its verdict maps by design — the same
documented false-positive category as the other license tooling.
Signed-off-by: Martin Vogel <martin.vogel@datadice.io>
Every commit must now carry a Signed-off-by trailer matching its
author, certifying the right to submit the change under the project's
MIT license (DCO 1.1, the Linux kernel mechanism). Enforcement is
strict at three layers: a commit-msg hook rejects unsigned commits
locally (scripts/install-git-hooks.sh), and the new DCO workflow
rejects every push and pull request containing one. Merge commits and
bot authors are exempt, matching standard DCO checks.
Signed-off-by: Martin Vogel <martin.vogel@datadice.io>
The license gate now proves itself before running: a --selftest mode
plants an unlicensed vendored file and asserts the structural layer
detects it, so a silently broken gate can no longer pass. The
byte-identity provenance audit gains exit-code semantics (any verdict
outside the accepted set fails) and runs as a blocking step in the
security workflow for both dry runs and releases.
A byte-identity audit of all vendored license files against their
upstream repositories found 113 copies differing only by a missing
trailing newline (vendoring artifact) — replaced with the exact
upstream bytes. Five needed real corrections: fennel carried an
unfilled MIT template although its upstream is CC0-1.0 at the pinned
commit; ron now carries the repo's LICENSE-MIT (dual MIT/Apache
upstream); python, wit, and verilog were refreshed to current upstream
bytes. The manifest's first-party table was also corrected: six of the
twelve grammars are self-maintained forks whose retained upstream
licenses are now byte-verified (assembly's upstream has been deleted
from GitHub; the retained MIT copy is the surviving grant). The audit
tool is kept as scripts/audit-license-provenance.py for future sweeps.
Structural layer: drop the hardcoded component list — any directory
containing vendored source or data files must now be covered by a
license file in itself or an ancestor within the vendored tree, so
newly vendored code without a license fails immediately. New third
layer: the graph-UI npm production tree is resolved (postinstall
scripts disabled) and every bundled package must be on the policy
allow-list; unknown licenses fail. Reclassifies @tailwindcss/vite as a
devDependency — it is a build plugin and was inflating the production
tree with native tooling.
The guard now runs a token-level clone detector (jscpd, with an npx
fallback) comparing internal/cbm/lsp/ C sources against clangd C++ — the
one reference close enough to C to tokenize alike — staging both trees as
a common format and reporting only clone pairs that span them. Catches
structural ports that rewrote strings/identifiers, which the verbatim
string/comment scan misses. clangd verified: no structural clones.
Scans internal/cbm/lsp/ for verbatim string/comment overlap against the
reference language servers (pyright, gopls, tsserver, Roslyn, clangd,
Eclipse JDT LS, kotlin-language-server, rust-analyzer, phpactor). Run
before committing new LSP work; add a new language's upstream reference to
the REFS manifest when implementing it. All 9 current languages verified
with no verbatim overlap.
The with-ui archives now append the per-package license texts of the
frontend bundle's production dependency tree to THIRD_PARTY_NOTICES.md
(platform-specific native build tooling is listed but excluded — its
code never reaches the browser bundle). The Glama check image carries
LICENSE and the notices file alongside the binary. Also renames a
TypeScript LSP test to describe its generic ODM-interface fixture.
The graph-UI HTTP server now owns its listening socket directly, so the
Layer 1 static audit's network-call scan would flag its socket()/bind()
calls. Exempt src/ui/httpd.c — it binds 127.0.0.1 only and is audited
separately by security-ui.sh — while still forbidding raw network calls
in every other source file.
Release archives now carry THIRD_PARTY_NOTICES.md, generated by
scripts/gen-third-party-notices.sh from THIRD_PARTY.md, the grammar
manifest, and the per-component license texts; the Homebrew formula
and AUR PKGBUILD install it alongside the binary. The SBOM gains
per-component license metadata, corrected versions, and the previously
missing vendored libraries. The security workflow gains a
vendored-license scan with an explicit allow-list policy, and the
release workflow exposes a skip_perf input for releases that do not
touch pipeline logic.
Removes the heaviest vendored grammar (66 MB) along with its language
wiring and test fixtures; the supported-language count moves to 158
across README, site, and manifest. The grammar manifest additionally
records the canonical-source decisions for the five
registry-disagreement grammars and the per-directory license files
restored earlier this week.
The web server behind the graph UI is now a purpose-built in-house
module (src/ui/httpd.c): localhost-only listener, strict HTTP/1.1
parsing with fixed request caps, a per-connection receive deadline,
and Connection: close semantics. http_server.c keeps the routing and
handlers, rewritten against the new transport API; the public server
API and main.c are unchanged. The previously vendored third-party
server is removed entirely.
Comes with a new 28-test transport + routing suite (tests/test_httpd.c)
covering parsing edge cases (strict CRLF, Content-Length limits, raw
path matching, percent-decode rules) and live-socket behavior (CORS
policy, RPC dispatch, receive deadline, clean shutdown). The security
audit scripts now check the new file layout and treat any network call
in vendored code as a failure.
mimalloc bumped to v3.3.2 (MI_MALLOC_VERSION 30302), which adds static-CRT malloc override on Windows. Makefile: define the real MI_MALLOC_OVERRIDE=1 ONLY on MinGW (enables the static-CRT override entry points — the generic Windows fix for #424); on Unix keep static-link-order override (defining MI_MALLOC_OVERRIDE there compiles alloc-override.c's forwarding defs and crashes macOS with 'mi_free: invalid pointer' since system libs keep the system allocator). The old -DMI_OVERRIDE=1 was never read by mimalloc source — kept as a prod marker. smoke-test.sh: hard-fail on any 'mimalloc: error' in stderr. Regenerated scripts/vendored-checksums.txt for the new tree (security-vendored.sh passes; no dangerous calls in vendored mimalloc).
The standard linux release binary dynamically links glibc 2.38+ and
GLIBCXX_3.4.32, so it fails to start on Debian 11, Ubuntu 20.04/22.04,
RHEL/Rocky 8/9, Amazon Linux 2, etc. — yet install.sh, the npm and PyPI
wrappers, and the binary's own self-update all fetched it by default,
contradicting the "single static binary" promise.
Point every linux install + self-update path at the fully-static
"-portable" asset (gcc -static), which has no glibc floor. macOS/Windows
are unaffected and unchanged.
- install.sh, pkg/npm/install.js, pkg/pypi _cli.py: select -portable on linux
- src/cli/cli.c: self-update download URL AND checksum archive name both use
-portable on linux (they must match or the update fails checksum verify)
- scripts/smoke-test.sh: assert linux self-update targets the -portable asset
- scripts/ci/check-glibc-compat.sh: new guard — runs the binary inside
debian:bullseye (glibc 2.31) and asserts it starts
- _smoke.yml: run the guard on the portable binary in smoke-linux-portable
Reproduced: standard binary -> "GLIBC_2.38 not found" on glibc 2.31;
portable binary runs cleanly.
The C++ heap-corruption crash only manifests in the production binary (MI_OVERRIDE=1), so the smoke job (which runs the prod binary) is where it reproduces — most reliably on Windows. A ~10k-line templated header exercises the parse-churn path; indexing it must succeed.
Phase 16's FIFO writer-close surfaced as POLLHUP rather than a clean POLLIN+EOF, which the server's poll()-based read loop does not treat as shutdown, so the probe left the process alive and the smoke job failed on every platform. Switch to a plain '< /dev/null' immediate EOF — the server exits reliably (~1s) via its normal getline-EOF path, and it is fully portable (POSIX + MSYS2 bash), so no OS gate is needed. Verified end-to-end: bash scripts/smoke-test.sh reaches 'ALL PASSED'.
Phase 16 of smoke-test.sh launches the stdio MCP server with its stdin held open (a running instance), closes stdin to trigger shutdown, and asserts the process terminates within ~6s — guarding the orphaned-background-process failure mode behind #406 from the prod binary on every platform. POSIX-gated (Windows uses job objects; mkfifo is unavailable on MSYS2).
Two reviewed external-fork contributions, re-implemented as clean maintainer code.
#406 / PR #407 (thanks @nvt-pankajsharma): a POSIX parent-death watchdog so the stdio MCP server exits when its launching parent dies instead of lingering on stdin. Refactors signal_handler into an idempotent request_shutdown(); a watchdog thread polls getppid() (500ms) and shuts down + exit(0) once the initial ppid (>1) changes; Windows unaffected (#ifndef _WIN32). The fork's getppid()<=1 startup early-exit was dropped (it could wrongly kill a legitimately-launched server during reparent races / in container launchers; the initial_ppid>1 guard already no-ops safely), and thread-create failure is now non-fatal (matches the watcher/HTTP background-thread policy). Adds tests/test_parent_watchdog.sh + scripts/test.sh wiring.
#413 / PR #414 (thanks @santanusinha): runtime log-level control via CBM_LOG_LEVEL (case-insensitive debug|info|warn|error|none, or numeric 0..4; unset/unknown leaves the level unchanged). getenv() before threads start (no race). Adds tests/test_log.c cases + README env-var row.
Four overload-scoring lookups (cbm_registry_lookup_method_by_types/_by_args, _symbol_by_types/_by_args) linear-scanned the entire project registry on every call site, ignoring the pre-built method_buckets/func_qn_buckets hash index that the by-name lookups already use. On a large C++ header (e.g. 10989 defs) that is O(calls x project_funcs) per file (~34s/file), so big codebases never finish indexing. Add a hashed fast path (O(overloads)) when the registry is finalized; preserve first-match semantics via lowest registration index (equivalence verified over 1606 lookups, 0 divergence); keep the linear fallback for un-finalized single-file registries so normal files are unaffected. Adds tests/test_cpp_index_hang.sh (scale-tier subprocess+timeout repro, opt-in via CBM_RUN_HANG_TEST).
The cypher unsupported-function check captured stdout, but the CLI prints the parse error to stderr (which cli() redirects into $CLI_STDERR) and exits non-zero, leaving stdout empty — so the assertion always failed once the run reached the smoke stage. Verify the exit status and the stderr channel where the binary actually reports the loud failure. Confirmed locally: full smoke-test.sh passes.
cyp_first_cell built the query_graph JSON by interpolation, so a query with a double-quoted string literal (e.g. replace(f.name, "a", "A")) produced invalid JSON and an empty result (FAIL: replace empty). Escape embedded double-quotes before building the JSON.
The update/star notice in src/mcp/mcp.c references the project's own GitHub URL, which scripts/security-audit.sh blocked as not allow-listed. Add it to scripts/security-allowlist.txt.
Add scripts/check-no-test-skips.sh (run from lint) which fails the lint phase on any plain SKIP() or direct tf_skip_count manipulation; only SKIP_PLATFORM() (for genuinely platform-specific tests) is tolerated. Add FAIL() and SKIP_PLATFORM() helpers to the test framework and convert the remaining SKIP()/perf-gated skips across the suite into pass-or-fail assertions, so a suite that cannot meet its preconditions reports a red failure instead of a silent skip.
#272: '&' is neutralised by the single-/double-quoting of the grep/Select-String/
git commands, so it no longer fails validation. A relaxed validate_search_path_arg
(cbm_validate_shell_arg minus '&') is used for search_code + detect_changes
root_path/file_pattern; all other shell metacharacters stay rejected and
base_branch keeps the strict check.
#282: search_code defaulted regex=false, so 'foo|bar' matched the literal pipe
and silently returned 0 results. The result now carries a warnings[] array that
flags a literal '|' under regex=false (with the regex=true hint), so the trap is
visible instead of looking like a legitimate no-match. (The related invalid-regex
silent-empty case was already fixed in #283.)
Also: every search_code result now reports elapsed_ms, and a perf warning fires
when a search exceeds 5s (also logged as search.slow) so slow calls are visible.
Tests for both issues + the multi-word/invalid-regex no-regression; generous
search_code smoke coverage (basic+elapsed_ms, literal-| warning, '&' acceptance).
Closes#272Closes#282
Wire the Leiden detector into get_architecture via a new 'clusters' aspect.
arch_clusters loads Function/Method/Class nodes + CALLS edges (capped for very
large graphs), runs cbm_leiden, and reports the top-N communities compactly —
each with a dominant-package label, member count, cohesion (internal vs boundary
edge ratio), representative top nodes (by degree), and the packages spanned.
Singletons are skipped to avoid noise. The mcp.c serializer already emitted the
clusters array; it is now populated.
Tests: arch_clusters_basic (two cliques + a bridge -> >=2 connected clusters).
Generous smoke coverage added for the full Cypher surface (labels/type/id/keys/
properties/size/reverse/replace/left/coalesce/substring, NOT EXISTS dead-code,
CASE, unsupported-function error) plus the get_architecture clusters aspect.
Andrew Hundt