Share the route path/handler scanner between sequential and parallel call passes, add FastAPI/Starlette websocket route-registration suffixes, and let sequential unresolved route-registration suffix calls emit Route registration facts when the existing route-literal guard accepts the path.
Add a sequential/parallel parity canary for FastAPI websocket decorators. Validation: make -f Makefile.cbm -j16 build/c/test-runner; CBM_ONLY_SUITE=parallel build/c/test-runner; CBM_ONLY_SUITE=pipeline CBM_ONLY_TEST=fast_route build/c/test-runner; make -f Makefile.cbm -j16 cbm; bash scripts/check-source-safety.sh; git diff --check.
FastAPI matrix evidence: guarded production remains safe by full fallback in /private/tmp/cbm-fastapi-matrix-websocket-parity-current-20260704T2215.json. Forced exact cap64 now preserves the previously missing tests/test_ws_router websocket Route nodes and 9 route-registration edges, but still fails canonical edge parity; keep scoped-LSP full fallback until the remaining edge-attribution mismatch is fixed.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Guard C-family header changes from active overlay publication until active edge parity is proven against fresh rebuilds. Bounded header edits can still use exact base-delta publication, preserving correctness without forcing every safe case to full rebuild.
Add a focused C-header incremental canary that verifies no overlay publish, exact frontier expansion, exact publish stats, and fresh-rebuild equality. Clean up stale direct-frontier helper code from the failed cap64 overlay experiment.
Validation: CBM_ONLY_SUITE=pipeline build/c/test-runner -> 323 passed; make -f Makefile.cbm lint-source-safety -> OK; self-dogfood cap16/cap64 store_pipeline_batch passed canonical equality but still fell back with frontier_too_large at 2.81x/2.74x.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Serialize call args consistently for call-derived edges across sequential and parallel indexing, including CONFIGURES edges used by MCP graph responses and data-flow derivation.
Add a generic argument-list fallback for call extraction so grammars exposing arguments as argument_list still populate CBMCall.args.
Guard exact delta and overlay publication against cross-file node-QN collisions, falling back instead of letting a changed header override a canonical source definition in the active overlay view.
Validation: build/c test runner and binary rebuild passed; CBM_ONLY_SUITE=pipeline passed; lint-source-safety passed; full self-dogfood MCP matrix passed 5/5 in /private/tmp/cbm-cross-file-qn-guard-full-self-dogfood-20260704T.json.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Preserve macro-expanded call recovery without leaking metadata from the preprocessor pass by adding a calls-only unified extraction path.
When exact delta cannot bound a C-family header frontier, fall back to a full rebuild instead of containment. For existing DBs, publish that fallback through the store project-replacement path so dependency sibling projects are preserved and MCP dependency refreshes cannot retain stale skipped-dir rows.
Validation:
- make -j8 -f Makefile.cbm cbm
- CBM_ONLY_SUITE=pipeline build/c/test-runner
- CBM_ONLY_SUITE=mcp build/c/test-runner
- CBM_ONLY_SUITE=extraction build/c/test-runner
- CBM_ONLY_SUITE=watcher build/c/test-runner
- uv run python scripts/benchmark-incremental-speed.py --self-dogfood --transport mcp --self-dogfood-scenarios one_source_file
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Make duplicate graph-buffer upserts and worker merges choose richer source spans for definition-like nodes, so full and containment rebuilds no longer depend on worker order for declaration-vs-definition locations.
Pass C/C++ macro extraction mode through extraction contexts instead of relying on process-global pipeline state, preventing MCP runs with mixed modes from leaking macro-node policy across pipelines.
Recompute configlink-owned CONFIGURES edges deterministically by clearing only key_symbol/dependency_import strategy edges, preserving direct CONFIGURES links, and sizing candidate arrays from discovered graph counts instead of fixed caps. Reuse cbm_path_base for manifest basename handling and make it recognize both POSIX and Windows separators.
Validation: git diff --check; ASan/UBSan test-runner rebuild /private/tmp/cbm-pan4-20260702T112332Z-build-test-runner-after-basename.log; str_util 65/65 and configlink 9/9 /private/tmp/cbm-pan4-20260702T112531Z-affected-strutil-configlink.log; product build /private/tmp/cbm-pan4-20260702T112545Z-cbm-build-after-configlink.log; MCP self-dogfood one-source canonical equality /private/tmp/cbm-pan4-20260702T112614Z-one-source-mcp-after-configlink.json; graph_buffer 63/63 and extraction 256/256 /private/tmp/cbm-pan4-20260702T112822Z-affected-graphbuffer-extraction.log; incremental 161/161 /private/tmp/cbm-pan4-20260702T112851Z-incremental-suite.log.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Add cbm_env_flag_enabled() beside cbm_safe_getenv() so debug/probe feature flags share one parser for unset, empty, 0, false, off, and no.
Remove the LSP-only env wrapper and route LSP debug/probe checks directly through the foundation helper. Reuse the same helper for CBM_PROFILE and CBM_SEMANTIC_ENABLED, and use cbm_safe_getenv() for CBM_SEMANTIC_THRESHOLD.
This keeps debug/profile nonempty opt-in compatibility while making explicit false values disable them. It also lets semantic enabled flags use the same true-style config convention rather than only literal 1.
Validation logs: /private/tmp/cbm-p26-env-flag-source-safety.log; /private/tmp/cbm-p26-env-flag-source-safety-selftest.log; /private/tmp/cbm-p26-env-flag-test-runner-build.log; /private/tmp/cbm-p26-env-flag-platform.log; /private/tmp/cbm-p26-env-flag-ts-lsp.log; /private/tmp/cbm-p26-env-flag-all-lsp.log; /private/tmp/cbm-p26-env-flag-pipeline.log; /private/tmp/cbm-p26-env-flag-product-build.log.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Add a shared LSP env flag helper that uses cbm_safe_getenv and consistent false values for debug/probe flags.
Replace duplicated raw getenv checks for CBM_LSP_DEBUG, CBM_LSP_DISABLED, and CBM_LSP_KOTLIN_AST across language resolvers. Add -Isrc to grammar/LSP compile flags so lsp_all.c can include foundation headers in product, test, nosan, and tsan builds.
Add a TS LSP regression test proving CBM_LSP_DISABLED=0/false/off/no keeps the resolver enabled.
Validation logs: /private/tmp/cbm-p264-lsp-env-source-safety-final2.log; /private/tmp/cbm-p264-lsp-env-source-safety-selftest-final2.log; /private/tmp/cbm-p264-lsp-env-test-runner-build-5.log; /private/tmp/cbm-p264-lsp-env-ts-lsp-final.log; /private/tmp/cbm-p264-lsp-env-all-lsp-final.log; /private/tmp/cbm-p264-lsp-env-product-build-final.log.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Restore Puppet include_statement call candidates and allow the keyword-like include operation through the existing keyword exception path while preserving include import extraction.
Evidence: AST probe confirmed include_statement, failed-first extraction canary produced 254 passed / 1 failed before the fix, and the focused extraction suite now passes 255/255.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Add a shared Verilog/SystemVerilog leaf-identifier resolver for wrapped subroutine and tf_call nodes, and include SystemVerilog subroutine_call nodes exposed by the vendored grammar.
Evidence: AST probes confirmed the wrapper shapes, failed-first canaries produced 252 passed / 2 failed before the fix, and the focused extraction suite now passes 254/254.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Restore VHDL parenthesis_group call candidates and add a narrow VHDL resolver for the grammar shape where add(1, 2) parses as name + parenthesis_group instead of function_call.
Evidence: a temporary AST probe confirmed the shape, a failed-first extraction canary produced 251 passed / 1 failed before the fix, and the focused extraction suite now passes 252/252.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Restore the FunC function_application node to the call-node spec so the existing shared function-field resolver can emit exact CALLS records for ordinary helper() applications.
Evidence: a failed-first extraction canary produced 250 passed / 1 failed before the spec restoration, a temporary AST probe showed function_application function:(identifier), and the focused extraction suite now passes 251/251.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Use the vendored Nickel grammar's fun_expr and applicative nodes instead of the previous fun/infix_expr spec. The new Nickel resolver emits a call only for applicatives with a t2 argument and walks the bounded t1 chain to the leftmost ident so curried calls produce one edge.
Failed-first evidence: CBM_ONLY_SUITE=extraction ./build/c/test-runner failed 249 passed / 1 failed at nickel_function_application_edge before the fix. After the fix, the focused extraction suite passes 250/250. Also validated git diff --check and scripts/check-source-safety.sh.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Resolve COBOL call_statement nodes through the grammar's x field and reuse the existing quote stripping path so CALL 'SUBPROG' emits SUBPROG instead of no call edge.
Adds a fixed-format COBOL canary that failed before the resolver change with 248 passed / 1 failed and now passes in the focused extraction suite: CBM_ONLY_SUITE=extraction ./build/c/test-runner => 249 passed. Also validated git diff --check and scripts/check-source-safety.sh.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Add a failed-first NASM canary for ordinary call puts syntax and restore extraction by matching the grammar's actual_instruction node.
The NASM resolver is language-gated, requires the opcode to be call using portable ASCII case folding, and returns the first operand symbol so non-call instructions such as ret are ignored.
Validation: failed-first extraction produced 247 passed / 1 failed at nasm_call_edge; after the fix, git diff --check, scripts/check-source-safety.sh, make -j8 -f Makefile.cbm build/c/test-runner, and CBM_ONLY_SUITE=extraction ./build/c/test-runner passed with 248/248.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Use the LLVM grammar's instruction_call, instruction_invoke, and instruction_callbr node names instead of dead call/invoke token names, so call extraction reaches real instruction nodes.
Teach the shared field-based callee resolver to read a callee field and unwrap simple value/var/expression wrappers, which keeps the fix reusable for grammars that expose a semantic callee field.
Add an exact LLVM IR canary for call void @callee(). Failed-first validation produced 246 passed / 1 failed before the fix; after the fix, extraction passes 247/247. Also passed git diff --check, scripts/check-source-safety.sh, and make -j8 -f Makefile.cbm build/c/test-runner.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Use the vendored Meson grammar's normal_command node and command field to extract Meson command callees such as project and executable.
Add exact call-edge canaries for Meson and Just. The rebuilt failed-first extraction run showed Meson missing project/executable calls while the Just canary already passed through the existing generic function_call path.
Validation: git diff --check; bash scripts/check-source-safety.sh; make -j8 -f Makefile.cbm build/c/test-runner; CBM_ONLY_SUITE=extraction ./build/c/test-runner passed 246/246.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Restore verified callee extraction for Jsonnet functioncall, Typst call, Make function_call/shell_function, and Puppet function_call nodes.
Align the Make call-node specs with the verified shell_function path. Leave Puppet include syntax and other mismatched upstream helper candidates deferred to parse-probe work instead of broadening call-node behavior without evidence.
Validation: git diff --check; bash scripts/check-source-safety.sh; make -j8 -f Makefile.cbm build/c/test-runner; CBM_ONLY_SUITE=extraction ./build/c/test-runner (245 passed).
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Restore verified tree-sitter callee extraction for CSS function calls, SQL invocations, Elm function_call_expr targets, and Nix apply_expression heads.
Add extraction canaries for each restored path and keep unverified mismatched grammars deferred to parse-probe work instead of guessing tree shapes.
Validation: git diff --check; bash scripts/check-source-safety.sh; CBM_ONLY_SUITE=extraction ./build/c/test-runner (241 passed).
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Restore upstream-compatible textual CBMCall synthesis for Java method references, Kotlin operator/convention syntax, and C++/CUDA operator and implicit call forms. These call records are the join input for existing LSP resolutions, so the graph can form CALLS edges for method references, operator overloads, copy-init, conversion operators, and delete/destructor operands instead of silently dropping resolved calls.
Keep the implementation language-gated and reuse existing arena allocation, cbm_memmem, TS_FIELD, and cbm_calls_push helpers. Also normalize C++ template_function callees to the bare name so template-call LSP resolutions have the same textual join key.
Validation:
- git diff --check
- bash scripts/check-source-safety.sh
- make -j8 -f Makefile.cbm build/c/test-runner
- CBM_ONLY_SUITE=extraction ./build/c/test-runner
- CBM_ONLY_SUITE=java_lsp ./build/c/test-runner
- CBM_ONLY_SUITE=kotlin_lsp ./build/c/test-runner
- CBM_ONLY_SUITE=c_lsp ./build/c/test-runner
- CBM_ONLY_SUITE=matrix_known_classes ./build/c/test-runner
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Restore upstream-compatible extraction behavior for Objective-C, Dart, and Rust call attribution without reintroducing duplicate scope paths.
Objective-C now resolves top-level C function definitions and attributes method-body calls to the method under the implementation class. Dart function bodies inherit the preceding function or method signature scope. Rust impl items use the class-boundary path for impl type scopes.
Add deterministic canaries for ObjC helper calls, Dart body calls, Rust impl method calls, and update the Dart language contract now that the gap is closed.
Validation: git diff --check; bash scripts/check-source-safety.sh; make -j8 -f Makefile.cbm build/c/test-runner; CBM_ONLY_SUITE=extraction ./build/c/test-runner; CBM_ONLY_SUITE=lang_contract ./build/c/test-runner; CBM_ONLY_SUITE=grammar_labels ./build/c/test-runner; CBM_ONLY_SUITE=grammar_regression ./build/c/test-runner.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Restore focused upstream extraction fixes for PureScript exp_apply callees, SCSS @function call edges, Julia assignment-form short functions, and Agda body-call attribution.
Keep the changes branch-compatible: use existing arena/text helpers, preserve plain Julia assignments as variables, skip Agda type-signature expressions so types do not become CALLS edges, and add deterministic extraction regressions for each restored behavior.
Validation: git diff --check; bash scripts/check-source-safety.sh; make -j8 -f Makefile.cbm build/c/test-runner; CBM_ONLY_SUITE=extraction ./build/c/test-runner; CBM_ONLY_SUITE=grammar_regression ./build/c/test-runner; CBM_ONLY_SUITE=grammar_labels ./build/c/test-runner; CBM_ONLY_SUITE=lang_contract ./build/c/test-runner.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Port the next upstream correctness slice in fork-compatible form. TSX per-file JSX import resolution no longer emits raw relative module specifiers as final targets; Java static imports retry package-qualified classes by registered short name before falling back to text; OCaml unified call scoping now mirrors value_definition naming and skips nested local let scopes; C/C++ out-of-line method callers are rebuilt with the resolved class QN so LSP caller joins match the extractor.
Focused validation: git diff --check; bash scripts/check-source-safety.sh; make -j8 -f Makefile.cbm build/c/test-runner; CBM_ONLY_SUITE=ts_lsp ./build/c/test-runner; CBM_ONLY_SUITE=java_lsp ./build/c/test-runner; CBM_ONLY_SUITE=extraction ./build/c/test-runner; CBM_ONLY_SUITE=c_lsp ./build/c/test-runner.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Port the upstream Python/Kotlin builtin target fixes and the Windows ARM64 stdlib include onto the fork branch. Python call extraction now keeps keyword-filtered builtins only when the generated stdlib data provides real LSP target nodes, so LSP-resolved builtin calls can form CALLS edges. Kotlin now injects kotlin.Any plus toString/equals/hashCode target definitions before graph minting so lsp_kt_any edges point at real nodes. compat.h includes stdlib.h directly for getenv/_putenv_s declarations on stricter Windows toolchains.
Focused validation: git diff --check; bash scripts/check-source-safety.sh; make -j8 -f Makefile.cbm build/c/test-runner; CBM_ONLY_SUITE=extraction ./build/c/test-runner; CBM_ONLY_SUITE=kotlin_lsp ./build/c/test-runner; CBM_ONLY_SUITE=grammar_labels ./build/c/test-runner; CBM_ONLY_SUITE=platform ./build/c/test-runner.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Replace the plain cbm_init() static guard with a C11 atomic one-shot state machine so parallel extraction callers wait for allocator/library initialization instead of racing the guard.
Make the macOS mach timebase cache use the same release/acquire publication pattern while leaving the Linux, BSD, and Windows timing paths unchanged.
Add a parallel cbm_init() idempotence regression registered first in the parallel suite.
Validation: make -f Makefile.cbm build/c/test-runner; CBM_ONLY_SUITE=parallel ./build/c/test-runner; CBM_ONLY_SUITE=platform ./build/c/test-runner; bash scripts/check-source-safety.sh; git diff --check.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Remove process-global SQLite writer sort comparator state while preserving the existing parallel index-sort phase. Each sort job now carries immutable node/edge context, using qsort_r where available and a no-global fallback elsewhere.
Also make the shared source-code import edge helper build local_name JSON with yyjson so long aliases are not truncated after the import emission consolidation.
Validation: make -f Makefile.cbm cbm; make -f Makefile.cbm build/c/test-runner; CBM_ONLY_SUITE=pipeline ./build/c/test-runner; CBM_ONLY_SUITE=sqlite_writer ./build/c/test-runner; CBM_ONLY_SUITE=graph_buffer ./build/c/test-runner; bash scripts/check-source-safety.sh; git diff --check.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Guard C/C++ AST-discovered registry enrichment when the Tier-2 cross registry is shared across resolve workers. This covers default-arg min_params updates, bare function declarations, template type metadata, and class method pre-pass registration so worker-local arena pointers are not stored in shared registry state.
Add a focused c_lsp regression test and wire CBM_ONLY_SUITE=c_lsp so this path can be rerun cheaply. Also make every lsp_all.o variant depend on the included LSP sources/headers; otherwise resolver edits can silently link stale unity objects.
Validation: make -f Makefile.cbm build/c/test-runner; CBM_ONLY_SUITE=c_lsp ./build/c/test-runner > /private/tmp/cbm-logs/c-lsp-shared-registry.log 2>&1 (743 passed); CBM_ONLY_SUITE=parallel ./build/c/test-runner > /private/tmp/cbm-logs/parallel-c-lsp-shared-registry.log 2>&1 (22 passed); bash scripts/check-source-safety.sh > /private/tmp/cbm-logs/source-safety-c-lsp-shared-registry.log 2>&1 ([source-safety] OK).
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Stabilize several graph-producing paths and add stronger regression coverage before the incremental redesign work continues.
Key changes:
- resolve call import maps from resolved graph-buffer IMPORTS edges so re-exported targets are reused consistently
- make semantic, similarity, complexity, route/section upserts, and resolver scoring more deterministic across full/incremental and worker-order differences
- propagate incremental phase/persist failures instead of continuing toward publication after failed phases
- add canonical graph diff helpers and stricter incremental artifacts for full-vs-incremental comparison
- add source-safety lint coverage for MCP stdout, unsafe string APIs, and new raw env/fs API usage
- replace touched unsafe string copies with bounded or project-wrapper-based forms
Validation:
- make -f Makefile.cbm build/c/test-runner
- CBM_ONLY_SUITE=pipeline ./build/c/test-runner: 217 passed
- CBM_ONLY_SUITE=graph_buffer ./build/c/test-runner: 55 passed
- CBM_ONLY_SUITE=mcp ./build/c/test-runner: 117 passed
- CBM_ONLY_SUITE=simhash ./build/c/test-runner: 24 passed
- CBM_ONLY_SUITE=parallel ./build/c/test-runner: 21 passed
- CBM_ONLY_SUITE=watcher ./build/c/test-runner: 57 passed
- bash scripts/check-source-safety.sh: OK
Known remaining:
- CBM_ONLY_SUITE=incremental still fails incr_accuracy_vs_full: 159 passed, 1 failed. Current artifacts show stale IMPORTS/CALLS/USAGE targets in unchanged importers and downstream SEMANTICALLY_RELATED drift; this is tracked in the forensic notes as a fallback-first incremental redesign task.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Tighten service-pattern matching so short framework ids such as gin. do not match unrelated QNs like plugin.*, while keeping separator-based wrappers such as requests_get working.
Reject bare HTTP-verb decorators like unittest.mock.patch as route decorators unless they have a receiver, and skip package manifests when promoting infra URL string refs to Route nodes.
Validation: build/c/test-runner builds; CBM_ONLY_SUITE=infrascan 4/4; extraction 195/195; depindex 32/32; lang_contract 31/31; edge_types_probe 52/52; isolated autorun index now stores 0 Route nodes.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Make index writes and artifact refreshes use verified temp files plus atomic replacement, preserving Windows replace semantics and avoiding fixed temp-path collisions.
Thread the remaining pipeline confidence knobs through config while keeping zero-valued defaults backward compatible with existing upstream behavior.
Tighten route-node gates and architecture route output so infra URL nodes stay queryable in the graph but do not pollute get_architecture route summaries; preserve code-created absolute URL routes for all language extractors.
Harden MCP and CLI path handling around cache paths, grep scratch files, update downloads, checksum validation, stale checks, and project DB discovery without writing protocol noise to stdout.
Add regression coverage for corrupt-DB recovery, concurrent atomic writes, tmp-prefixed project DBs, duplicate search result fields, route output filtering, config registry entries, and focused suite selectors.
Validated with: git diff --check; make -f Makefile.cbm build/c/test-runner; focused suites security, artifact, graph_buffer, sqlite_writer, store_search, store_bulk, store_pragmas, store_checkpoint, watcher, infrascan, pipeline, store_nodes, mcp, cli, parallel, tool_consolidation, incremental, and store_arch; make -f Makefile.cbm cbm; isolated dogfood index/query run.
Signed-off-by: Andrew Hundt <ATHundt@gmail.com>
Deep --no-ff merge of upstream DeusData/main (597 commits, merge-base 988d975)
into fork api-consolidation (HEAD 3018b22), producing a superset that combines
upstream breadth with the fork's surviving depth.
Upstream additions integrated: semantic/embedding search (vendored/nomic),
minhash/SimHash, cross-repo + cross-file LSP + route + pkgmap passes, artifact
import/export, path_alias, git_context, hook_augment, Windows support
(compat_fs/compat_thread), the UI layer, and ~140 expanded tree-sitter grammars.
Fork depth retained: PageRank/LinkRank ranking (src/pagerank/; sort_by=
relevance/linkrank/degree/calls), dependency indexing (src/depindex/;
CBM_INDEX_MODE=DEP), token reduction, and HTTP-endpoint discovery
(src/pipeline/httplink.c, pass_httplinks.c).
Conflict resolution (27 regions across the 39-file intersection):
- src/mcp/mcp.c: kept the fork superset handler pattern (resolve_project_store +
expand_project_param + inject_context_once) as the single path; UNIONed the
tool dispatch table; emit status:"indexed", excluded[], cross_service trace.
- src/store/store.c: upstream base + fork pagerank/linkrank/node_degree tables
and sort_by plumbing; cbm_store_node_degree now excludes INHERITS
(WHERE type != 'INHERITS'); removed the duplicate cbm_store_get_db.
- src/pipeline/pipeline.c: took upstream run_extraction_phase; grafted the fork
passes (httplinks, normalize, flush_store dump branch); call
cbm_pipeline_set_committed_counts before cbm_gbuf_dump_to_sqlite.
- internal/cbm/lang_specs.c: 158 entries combining the fork section_node_types
with upstream ts_factory/embedded_imports; wired 13 orphaned type-arrays.
- src/depindex/depindex.h: renamed CBM_PATH_MAX -> CBM_DEP_PATH_MAX (it collided
with constants.h enum CBM_PATH_MAX=1024) + 11 call sites in depindex.c.
- Makefile.cbm: upstream grammar/nomic units + fork CGO targets; added
httplink.c + pass_httplinks.c to PIPELINE_SRCS; removed mongoose NOSAN
machinery (mongoose was deleted upstream).
- httplinks restored (the merge silently deleted them because the fork had left
them unchanged since the base): httplink.c/.h, pass_httplinks.c,
tests/test_httplink.c; registered RUN_SUITE(httplink) in tests/test_main.c.
- CBM_INDEX_MODE enum unified to FULL=0, MODERATE=1, FAST=2, DEP=3.
Completeness (dual-parent check): 0 upstream files dropped from the merge tree;
6 fork files dropped, all intentional (upstream-superseded): CHANGELOG.md,
vendored/mimalloc/src/{arena-abandon,segment-map,segment}.c,
vendored/mongoose/{mongoose.c,mongoose.h}. Fork features (pagerank, depindex,
token_reduction, httplink) verified present and registered in tests/test_main.c.
src/pipeline/pass_parallel.c has an identical function set to upstream (49=49,
none dropped); the large line-deltas vs upstream are within-function divergence
of the fork extractor, not dropped functions.
Gate (run manually; hooks skip on merge commits): build green
(make -f Makefile.cbm cbm, exit 0); full C test suite 5980 passed / 2 failed /
0 crashes. The fork's pre-existing ASan crash in build_project_list_error_srv is
fixed by upstream's mcp.c rewrite, so the merge is strictly better than the fork
baseline on the crash axis.
Known issue, tracked separately (NOT a merge defect): 2 nondeterministic
failures in tests/test_incremental.c — incr_db_deleted_recovery:772 (diff_pct)
and incr_accuracy_vs_full:803 (full_nodes vs incr_nodes). Cause: the fork's
extractor (internal/cbm/extract_*.c) and parallel-resolve Route-node creation
are not thread-safe under the concurrent use of upstream's parallel pipeline
(run_parallel_pipeline), producing nondeterministic route/infra HANDLES edges
(~234k edge swing between runs). Sequential extraction (CBM_WORKERS=1) is
deterministic and correct. Root cause, bisect, and repro are documented in
notes/2026-06-24-fork-vs-upstream-divergence-assessment.md (section 8f); the fix
is deferred to a dedicated thread-safety audit (the make test-tsan target is
currently a stub and needs wiring).
Vendors the two MIT-licensed grammars from intersystems/tree-sitter-objectscript
@a7ffcdf (the language vendor's official grammars, ABI 15) that PR #467 needs:
internal/cbm/vendored/grammars/objectscript_udl/ (.cls)
internal/cbm/vendored/grammars/objectscript_routine/ (.mac/.inc/.rtn/.int)
Each directory carries the verbatim generated parser.c + scanner.c + tree_sitter/
headers + LICENSE. The one adjustment: each scanner.c's upstream
`#include "../../common/scanner.h"` is repointed to a per-directory
`objectscript_common.h` (a verbatim copy of the upstream common/scanner.h), because
this repo's shared vendored/common/scanner.h belongs to the cfml/fsharp grammars and
differs. Updates MANIFEST.md + THIRD_PARTY.md.
These files are dormant until #467 (the grammar shims + extraction) is rebased on
top; main's build is unaffected.
Refs #462, #467.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
try_route_from_annotation read the annotation name via the "name" field
(ts_node_child_by_field_name). tree-sitter-kotlin annotation nodes have no
"name" field: the name lives in a nested type_identifier under (user_type)
or (constructor_invocation (user_type ...)). The lookup returned null, so
every Kotlin @RestController route was dropped (0 decorator-source routes)
while Java worked. Args are likewise under constructor_invocation as
value_arguments (each wrapped in value_argument), not Java argument_list.
Add annotation_name_node()/annotation_args_node() that fall back to the
Kotlin AST shape (Java path unchanged) and unwrap value_argument when
scanning for the route path. Kotlin @GetMapping/@PostMapping/etc. now yield
Route nodes and HANDLES edges. Adds handles_spring_kotlin regression test.
Signed-off-by: jsw6701 <86887824+jsw6701@users.noreply.github.com>
Python calls of the form iris_obj.classMethodValue('Pkg.Class', 'Method')
emitted no CALLS edge. The detection existed in walk_calls, which is dead
code — the pipeline uses cbm_extract_unified → handle_calls, not walk_calls.
Move the IRIS Native API string-dispatch detection to handle_calls. When a
Python call's callee ends in .classMethodValue, .classMethodVoid,
.classMethodBoolean, or .classMethodObject, and both string-literal arguments
can be extracted, emit a synthetic CALLS edge to Arg1.Arg2.
The pattern generalizes beyond this specific API: any library that takes
class/method names as string literals follows the same shape.
Regression test: python_iris_classMethodValue in test_extraction.c.
Signed-off-by: Thomas Dyar <tdyar@intersystems.com>
* fix(perl): stop extracting config strings as call targets
The Perl branch of extract_scripting_callee blindly returned the text of
child(0) of every call node. In config-heavy Perl (.cgi/.pl with embedded
log4perl-style config), tree-sitter-perl misparses dotted config tokens
(e.g. "log4perl.appender.File.utf8") into call-shaped nodes, and that
dotted string was emitted as a callee_name, later matched by the generic
short-name resolver to unrelated project subs.
Now the Perl branch pulls the real name token (method/function field, else
child(0)) and validates it as a bare Perl sub/method identifier via
perl_is_identifier_callee: must start with a letter or '_' and contain only
[A-Za-z0-9_:] (allowing the '::' package separator). Any '.', whitespace,
quote, or '/' disqualifies it and NULL is returned so no CALLS edge forms.
Gated to CBM_LANG_PERL; other languages are untouched.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Shane McCarron <shane.mccarron@corvexconnect.com>
* fix(resolver): don't match Perl builtins to project subs
Perl builtins (push/shift/keys/sprintf/...) carry no language or call-kind
awareness through the generic name-matcher in cbm_registry_resolve. When a
project defines a sub whose name collides with a builtin, an invocation of
the builtin was wired to that sub by same-module / suffix matching - a
false-positive CALLS edge.
Adds cbm_perl_is_builtin (curated, sorted bsearch set of 94 perlfunc core
builtins) and applies it in both call-resolution passes (sequential
resolve_single_call and parallel resolve_file_calls), gated on the file
language == CBM_LANG_PERL and only AFTER LSP resolution has declined, so a
genuine LSP-resolved call is never suppressed. The file language is threaded
into both resolvers via a new trailing CBMLanguage parameter; every other
language reaches cbm_registry_resolve unchanged (byte-identical behavior).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Shane McCarron <shane.mccarron@corvexconnect.com>
* fix(resolver): suppress generic short-name matching for Perl method calls
A Perl method call ($obj->m / Class->m) carries no receiver type at the
structural tier, so the generic short-name matcher in cbm_registry_resolve
would wire $dbh->commit / $cgi->param / $logger->log to any project sub
sharing the bare method name - the dominant source of false-positive CALLS
edges in CPAN/framework-heavy Perl. Resolving such a call correctly is the
LSP's job, not the bare-name matcher's.
Adds a CBMCall.is_method flag (zero-init false, so all other languages and
existing call sites are unaffected). method_call_expression is added to the
Perl call node set and handle_calls sets is_method=true only for that node
type when the file language is Perl. Both call-resolution passes then skip
generic resolution for Perl method calls (combined with the builtin guard
from the prior commit). Genuine intra-project function calls (non-method,
non-builtin) still resolve as before. LSP-resolved method calls are
unaffected because the guard runs only after LSP resolution declines.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Shane McCarron <shane.mccarron@corvexconnect.com>
* test(perl): cover call-graph noise fixes (builtins, methods, config strings)
Hermetic tests for the three Perl call-graph noise fixes:
test_extraction.c (extraction tier):
- config string is never emitted as a callee; genuine call still extracted
- builtin calls (push/keys) extracted but never flagged is_method
- arrow/method calls ($self->commit / $dbh->commit) set is_method=true,
while the genuine function call (helper) does not
- a JS method call never sets is_method (flag is Perl-only — other
languages unaffected)
test_registry.c (resolver tier):
- cbm_perl_is_builtin recognizes core builtins (incl. first/last of the
sorted set) and rejects project subs, case variants, empty, and NULL
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Shane McCarron <shane.mccarron@corvexconnect.com>
* fix(perl): address QA round 1
Round 1 (Claude panel + DO DeepSeek) findings:
- Rework the Perl noise guard so it suppresses only WEAK generic matches
(suffix_match/unique_name) and KEEPS high-confidence same_module/import_map.
The prior guard ran before cbm_registry_resolve and dropped genuine same-file
calls to builtin-named subs (e.g. a project sub log/index/open called as a
bare function) that pre-PR resolved via same_module. Extracted the decision
into pure, unit-tested cbm_perl_suppress_generic_match() shared by the
sequential (pass_calls.c) and parallel (pass_parallel.c) resolvers; corrected
the inaccurate comments (Perl has no LSP/textual stage before the guard).
- Tighten perl_is_identifier_callee to require '::' pairs (reject a lone ':',
':::', or trailing '::').
- Add resolver-contract tests covering weak-match suppression, same_module/
import_map retention, genuine-call survival, non-Perl no-op, and NULL strategy.
Verified on a real Perl monorepo: .cgi builtin/CPAN/config-string noise stays
eliminated while same_module edges to builtin-named subs are recovered.
Signed-off-by: Shane McCarron <shane.mccarron@corvexconnect.com>
* fix(perl): address QA round 2
Round 2 (Claude panel) caught a regression introduced by the round-1 refactor:
cbm_perl_suppress_generic_match whitelisted only the exact strategies
"same_module" and "import_map", but resolve_import_map can also return
"import_map_suffix" (confidence 0.85 — a genuine import-based resolution, not a
weak short-name guess). A '::'-qualified Perl builtin/method call resolved via
the import-suffix fallback was therefore dropped, contradicting the helper's
documented contract and partially missing acceptance criterion (d).
Add import_map_suffix to the kept (high-confidence) set so only the weak
short-name strategies (suffix_match / unique_name) are suppressed; update the
doc comment and add a unit-test case asserting import_map_suffix is retained.
Deferred as advisory (non-blocking, noted on the PR): a hypothetical leading-'::'
(main:: shorthand) under-extraction in perl_is_identifier_callee, and a
colon-edge-case coverage gap (logic correct by inspection).
Signed-off-by: Shane McCarron <shane.mccarron@corvexconnect.com>
---------
Signed-off-by: Shane McCarron <shane.mccarron@corvexconnect.com>
Co-authored-by: Shane McCarron <shane.mccarron@corvexconnect.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
A byte-identity audit of all vendored license files against their
upstream repositories found 113 copies differing only by a missing
trailing newline (vendoring artifact) — replaced with the exact
upstream bytes. Five needed real corrections: fennel carried an
unfilled MIT template although its upstream is CC0-1.0 at the pinned
commit; ron now carries the repo's LICENSE-MIT (dual MIT/Apache
upstream); python, wit, and verilog were refreshed to current upstream
bytes. The manifest's first-party table was also corrected: six of the
twelve grammars are self-maintained forks whose retained upstream
licenses are now byte-verified (assembly's upstream has been deleted
from GitHub; the retained MIT copy is the surviving grant). The audit
tool is kept as scripts/audit-license-provenance.py for future sweeps.
Removes the heaviest vendored grammar (66 MB) along with its language
wiring and test fixtures; the supported-language count moves to 158
across README, site, and manifest. The grammar manifest additionally
records the canonical-source decisions for the five
registry-disagreement grammars and the per-directory license files
restored earlier this week.
The alloc.h/array.h/parser.h headers in
internal/cbm/vendored/common/tree_sitter/ are part of the tree-sitter C
runtime (MIT, (c) 2018 Max Brunsfeld) but had no LICENSE file alongside
them. Copy the upstream MIT license into that directory and clarify in
THIRD_PARTY.md the split between the tree-sitter-html scanner helpers
(common/, (c) 2014) and the core runtime headers (common/tree_sitter/,
(c) 2018).
Restore upstream LICENSE files for every vendored grammar and C library
across both vendored trees (19 grammar dirs, lz4, zstd, simplecpp,
verstable, wyhash, ts_runtime, common, mongoose, xxhash, yyjson; SQLite
public-domain notice). Add the nomic-embed-code Apache-2.0 license plus a
NOTICE describing the embedding derivation. First-party grammars carry
the project MIT license.
THIRD_PARTY.md now defers grammar provenance to the verified
MANIFEST.md instead of a stale hand-written table (which misattributed
clojure as EPL-1.0 — it is CC0-1.0), covers all vendored libraries, and
documents the Hybrid LSP reference servers and stdlib type-data
provenance (typeshed, Go stdlib introspection, hand-curated specs).
Profiling symfony (416 s wall, one core pinned) convicted cbm_registry_lookup_type -> strcmp at 95% of samples: the PHP and Python def-registration loops probe the registry per Method (receiver-stub check) BEFORE the hash indexes exist, i.e. a linear scan over the growing type table - O(methods x types) per file, and the cross resolvers run per file. Java and Kotlin never called cbm_registry_finalize at all, leaving every lookup during their file walks linear over the whole cross registry. Fixes: (1) two-phase registration in php/py (types, finalize, then funcs/methods); (2) finalize added to the java/kotlin cross entry points; (3) lookups now tail-scan entries added after finalize - previously such entries were silently INVISIBLE to the hashed path (reproduce-first: tslsp_hash_registry_post_finalize_adds was red); (4) cbm_registry_finalize_into allocates the index from a per-call scratch arena - putting buckets into the pipeline-lifetime result arena per file accumulated +1.1 GB RSS on the FastAPI incremental test (incr_full_index caught it), and the py tier-2 builder (one def per call) skips mid-build finalize entirely. symfony: 416 s -> 12 s, identical node count; suite 5577/0.
The hand-rolled dump writer stored index cells fully inline regardless of payload size. SQLite's format spills index payloads above the max-local threshold (16,422 bytes at the 64 KB page size) to overflow pages, so a long key written inline makes the reader interpret key bytes as an overflow page number: PRAGMA integrity_check reported 'invalid page number 0x43654C6C' (ASCII from the key text) in idx_nodes_name on elasticsearch (very long Section names), and name lookups on that index silently returned nothing. write_index_btree now rewrites oversized cells (varint + local prefix + overflow chain via write_overflow_pages) before page building; promoted separators carry their chains with them. Reproduce-first: sw_long_index_keys_overflow (20 KB name; integrity_check failed pre-fix, ok post-fix). Also escapes the remaining raw text interpolations in DATA_FLOWS route props and HANDLES handler QNs (the ~51-edge malformed tail after 6c04ab7).
c_process_function's template-param attach wrote rf->type_param_names on the registry — in the Tier-2 cross-LSP phase that registry is shared READ-ONLY across all resolve workers, so the write raced concurrent readers AND stored a pointer into the writing worker's per-file arena; once that arena was recycled the shared registry held dangling memory, crashing other workers intermittently (bitcoin: SIGSEGV in c_process_function strcmp via cbm_run_c_lsp_cross_with_registry, ~1-in-2 runs). A registry_shared flag now marks the cross context and the attach is skipped there (cross-phase template deduction falls back to positional names — graceful degradation); the single-file extract phase keeps full fidelity. Real-repo-tier reproduction: 10x bitcoin index loop, 0 failures post-fix (pre-fix ~50% crash rate); suite 5575/0.
bind_lambda_args and the method-reference arity path indexed the resolved method signature's NULL-terminated param_types array by the CALL-SITE argument index. A call with more arguments than the resolved overload declares (overload mismatch, varargs) read past the terminator and dereferenced whatever followed in the arena as a CBMType* - SIGSEGV while indexing elasticsearch (same out-of-bounds family as #427 and the bitcoin targs fix). Both sites now bound the index by the walked param count. Reproduce-first: 6-line fixture - run(Runnable) invoked with six lambdas - crashed rc=134 pre-fix; fork+exit-signal regression test lsp_java_lambda_args_exceed_params_no_crash added; suite 5574/0.
The C/C++ LSP's explicit-template-argument call path (template_function, e.g. Using<Fmt>(v)) filled a stack array const CBMType *targs[16] without NULL-terminating it before cbm_type_substitute. With more declared type params than explicit args (bitcoin src/serialize.h: Using's Wrapper<Formatter, T&> with 2 params, 1 arg), the bounded args walk read uninitialized stack; any non-NULL garbage got bound to the type param and woven into the REGISTERED return-type graph, then dereferenced at a later call site - SIGSEGV (crash shape: string bytes misread as a CBMType*). All c_lsp targs arrays are now zero-filled, and cbm_type_substitute's args walk treats implausible values (misaligned / below the first page) as the terminator so contract violations can never leak garbage into a type graph. Reproduce-first: deterministic poison-args unit test (typerep_substitute_rejects_garbage_args_entries) + real-repo verification (bitcoin indexes clean under ASan; was a deterministic SIGSEGV).
Indexing large OSS during the perf sweep crashed three LSP resolvers, all by stack overflow in their recursive resolve walks (macOS crash reports): elasticsearch — SIGSEGV in bind_lambda_args under deep recursive java_resolve_calls_in_node frames; bitcoin — SIGSEGV in cbm_type_substitute via c_adl_resolve under deep c_resolve_calls_in_node frames; microsoft/TypeScript — SIGBUS under an unbounded lookup_member_type cycle (cyclic type graph in reallyLargeFile.ts). The expression-eval guards (eval_depth) did not cover these walks. Each resolver now has a depth-guarded entry (walk_depth cap 512 for the Java/C AST walks, member_depth cap 64 for TS member lookup); past the cap the subtree resolves as unknown — graceful degradation instead of a crash. Reproduce-first: prod CLI probes crashed pre-fix (nested-call Java rc=139, nested-call C++ rc=139); fork+exit-signal regression tests added (TS cyclic shape needs a real cross-file registry, so it is verified at the real-repo tier — microsoft/TypeScript now indexes clean). Side-finding documented in the sweep report: Java BLOCK nesting >=3000 is minutes-slow (separate pathology, not this fix).
C++/CUDA out-of-line method definitions (Foo::bar in a .cpp; the class body lives declaration-only in a header) were recorded as free Functions with no link to their class. A new helper cpp_out_of_line_parent_class() detects a qualified_identifier declarator on a function_definition and resolves the immediate enclosing class (descending nested scopes ns::Foo::bar to the direct parent); extract_func_def then promotes the def to label Method with parent_class set to the class QN so DEFINES_METHOD edges resolve. Distilled (reimplemented) from PR #428. Reproduce-first: cpp_out_of_line_method_issue428 (single-level out-of-line defs; the nested-namespace path isn't isolation-testable because tree-sitter-cpp parses a synthetic doubly-qualified def as an ERROR node, so it is exercised by real codebases).
The C++ LSP passes a class's declared template-param names (e.g. template<class T, class Alloc = ...>) alongside an instance's fewer template args (vector<int>). The substitution loop is bounded by type_params, so a param matching beyond the supplied args indexed type_args[i] past the array — reading adjacent arena memory (a type-name string byte) as a CBMType* that c_eval_expr_type later dereferenced -> SEGV (ASan: stack-buffer-overflow in cbm_type_substitute via c_eval_expr_type). Deterministic, all platforms; distinct from the #424 allocator crash. Fix at the single chokepoint: bound the args length by the (NULL-terminated) param count and gate every type_args[i] access by it, so no caller can over-read regardless of language. Reproduce-first: typerep_substitute_short_args_no_oob_issue427.
Defense-in-depth follow-up to the tree-sitter bind: route sqlite3 and libgit2
allocations through mimalloc explicitly so a correct binary never depends on the
fragile MI_OVERRIDE symbol override (where the Windows static-MinGW link can
resolve a library's malloc to mimalloc but its free to the CRT, corrupting the
heap). Prod-only, guarded by the existing CBM_BIND_TS_ALLOCATOR; the CRT+ASan
test build compiles these as no-ops and stays unchanged.
New cbm_alloc_init() consolidates all three binds behind a static idempotency
guard. main() calls it as its very first statement — before cbm_mcp_server_new
opens the in-memory store — because SQLITE_CONFIG_MALLOC must run before the
first sqlite3_open*/sqlite3_initialize, else sqlite3_config returns SQLITE_MISUSE
silently and the bind is ignored (the return code is asserted). cbm_init() also
calls cbm_alloc_init() so pipeline-pass entry points get the binds too; the
ts_set_allocator bind moved out of cbm_init into cbm_alloc_init.
sqlite3: a static sqlite3_mem_methods backed by mi_malloc/mi_free/mi_realloc,
xSize via mi_usable_size, xRoundup to an 8-byte boundary, xInit returning
SQLITE_OK, xShutdown a no-op (int-sized sqlite signatures wrapped with size_t
casts). libgit2: a static git_allocator (gmalloc/grealloc/gfree → mi_*) installed
via git_libgit2_opts(GIT_OPT_SET_ALLOCATOR). The allocator is set BEFORE any
git_libgit2_init: git_allocator_global_init keeps a custom allocator if one is
already set (git__allocator.gmalloc != git_failalloc_malloc), the pre-init global
is the fail-allocator, and there is no allocator reset on git_libgit2_shutdown —
so binding once at startup survives pass_githistory's per-call init/shutdown
pairs. libgit2 code is additionally guarded by HAVE_LIBGIT2.
Test build 5564/0 ASan-clean; prod build clean with CBM_BIND_TS_ALLOCATOR=1; the
HAVE_LIBGIT2 branch type-checked against the libgit2 sys/alloc.h struct shape;
lint-ci clean.
Andrew Hundt