Commit Graph
620 Commits
Author SHA1 Message Date
Martin Vogel 98910f9eda Merge origin/main into fix/scala-package-aware-imports
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-11-17 22:10:21 +01:00
Martin Vogel e6980cd60b Merge pull request #2547 from Zhuoxi2000/fix-haskell-escaped-char-swallow
fix(haskell): stop escaped char literals at the closing quote (#2439)
2026-11-17 22:06:27 +01:00
Martin Vogel f854641dcb Merge pull request #2988 from DeusData/fix/ci-slowest-suites-and-cpu-line
ci(test): fix the slowest-suites table on uutils sort; print CPU and SHA support
2026-11-14 14:52:16 +01:00
Martin Vogel a4364704a8 Merge origin/main into fix-haskell-escaped-char-swallow
Brings the branch up to main 1ec2f2623. Conflicts were the grammar
bookkeeping only:

- THIRD_PARTY.md: main's thirteen self-maintained forks (sql, vba added)
  plus haskell make fourteen; the haskell paragraph sits between pascal
  and main's new sql paragraph; main's tree-sitter-vba section is kept.
- MANIFEST.md summary: main's 164 grammars with haskell moved from
  vendored-from-upstream to self-maintained: 138 + 21 + 5 = 164,
  recounted from the tree (164 grammar dirs, 164 parser.c, ABI
  9/78/77). haskell stays only in the self-maintained forks table.
- scripts/vendored-checksums.txt: regenerated with
  scripts/security-vendored.sh --update.

scripts/audit-license-provenance.py merged cleanly: the haskell FORKS
row sits alphabetically between dotenv and javascript, next to main's
sql and vba rows.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-11-14 12:54:21 +01:00
Martin Vogel d6d9440fb0 ci(test): fix the slowest-suites table on uutils sort; print CPU and SHA support
The "8 slowest suites" summary in scripts/run-tests-parallel.sh sorted with
`sort -t= -k6 -rn`. uutils sort (Rust coreutils, shipped as sort on newer
runner images) reads the attached form -t= as -t with an inline "=value",
an empty separator, and exits with "separator must be exactly one character
long: ''", leaving the table empty. Pass the separator as its own argument,
which GNU, BSD and uutils sort all parse alike. Informational only: the
script has no set -e and its exit status is unchanged.

scripts/test.sh's environment line now also names the CPU and whether it
has SHA-256 instructions (x86 sha_ni / ARMv8 sha2), the signal
src/foundation/sha256.c uses to choose its hardware or portable backend.
A sanitized runner fingerprints its own large image on that path, so which
CPU a hosted job drew decides how long those fingerprints take. The helper
lives in scripts/env.sh (cpu_summary, opted into via print_env --cpu, so
build.sh, lint.sh and repro.sh print exactly what they did); it reads
/proc/cpuinfo or lscpu on Linux/MSYS2 and sysctl on macOS, reports
"unknown" for anything unreadable, and never fails the caller.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-11-14 10:35:55 +01:00
Martin Vogel 08ab4a09fe Merge pull request #2981 from DeusData/perf/linear-growth-sites
fix, perf: C# operator nodes, C increment writes and linear growth in extraction and eight language passes
2026-11-14 10:24:34 +01:00
Martin Vogel 334bad2e41 refactor(path-alias): allocate through the memory core
path_alias.c had 40 raw allocator sites outside the memory core. Every
allocation and its free now go through cbm_alloc/cbm_free in one class,
and the two cbm_strndup copies the file freed itself come from a
core-backed helper. cbm_path_alias_resolve keeps strdup, because
pass_pkgmap.c and the tests free its result with free(); the baseline
drops from 40 to 1. No behaviour changes.

The nearest-ancestor test built its collection with calloc/strdup and
freed it through cbm_path_alias_collection_free; it now frees what it
built with the allocator it used.

Checked under the production allocator, where a block freed by the wrong
side crashes: nest (34 tsconfig files with paths) and immich index
identically to before, and a synthetic project with an extending nested
tsconfig stays identical through full, fallback and incremental reindex
steps.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-11-14 00:47:41 +01:00
Martin Vogel eb44c04d4e Merge origin/main into feat/vendor-vba-grammar
Brings in #2974 (incl. the jsonnet scanner patch) and #2970. One conflict:
scripts/vendored-checksums.txt, where #2974 and the vba vendoring both
rewrite the MANIFEST.md line. Resolved by taking main's file and
regenerating it with scripts/security-vendored.sh --update; against main it
differs only in the MANIFEST.md line and the five vba lines, and the
integrity verify passes.

THIRD_PARTY.md, MANIFEST.md and tests/test_parse_coverage.c merged cleanly;
each equals main plus this branch's own change (identical patch-ids), and
test_parse_coverage.c has 76 TEST and 76 RUN_TEST entries (71 at the merge
base, +2 from main, +3 from this branch). The MANIFEST summary (164
grammars) and ABI line (9/78/77) match a recount from the tree.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-11-14 00:31:52 +01:00
Martin Vogel 4c52fb4e1a Merge origin/main into fix/scala-package-aware-imports
Brings in main up to 09bd6bb1 (#2967 FreeBSD peer pid, #2974 graph
variation and content loss fixes).

Resolutions:
- scripts/memory-core-baseline.txt (textual conflict): both sides
  lowered counts in the same hunk. pass_parallel.c keeps this branch's
  29 (arena QNs); pass_pkgmap.c takes main's 80 (#2974 dropped one raw
  free in cbm_pkgmap_build; this branch's pass_pkgmap.c additions add
  no raw site). Both values re-derived by scanning the merged tree with
  scripts/lint-memory-core.py; the lint passes.
- src/pipeline/pass_pkgmap.c (auto-merged): main's deterministic
  manifest order in cbm_pkgmap_build and this branch's Scala package
  index sit in separate functions; the merged file is main plus this
  branch's change unchanged (same per-file patch-id).
- tests/test_edge_imports.c, tests/test_pipeline.c (auto-merged): both
  sides' new tests are kept and registered; main's new tests call no
  function this branch re-signed. Same per-file patch-id as this
  branch's change.

Every other file equals main.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-11-14 00:17:43 +01:00
Martin Vogel e5b3972429 fix(grammar): give the jsonnet scanner per-parser state
Indexing go/grafana__loki produced a different graph from one run to the
next: each index reported one or two jsonnet files as parse_partial, a
different set every time, and indexing only its 68 jsonnet files reported
21-37 of them, with real definitions and their CALLS/DEFINES/IMPORTS/USAGE
edges missing. Parsed one at a time, every one of those files is clean,
and CBM_WORKERS=1 gave the same graph on every run.

The vendored jsonnet external scanner kept its state (inside a string,
and which quote) in three file-scope globals, so every indexing worker's
parser shared one copy. tree-sitter restores the scanner state right
before each scan and saves it right after; another thread's write in that
window changed the scan in flight, and a quoted string ran past its
closing quote. Its deserialize also ignored the zero-length call that
starts a parse, so a file ending inside an unterminated quote left that
state to every later parse in the process, and no later ||| text block
could be lexed.

The state now lives in the payload from ..._external_scanner_create(),
and deserialize resets it before reading. Lexing logic and the 3-byte
serialized state are unchanged. MANIFEST.md records the local patch next
to the properties scanner's, THIRD_PARTY.md notes it must be re-applied
on re-vendor, and scripts/vendored-checksums.txt is repinned.

Two single-threaded tests in parse_coverage pin it: a second parser run
from inside parser A's input callback, at the byte after A's opening
quote, must not break A's string; and a parse that ends inside a string
must not stop the next one from lexing a ||| block. Both fail on the
unpatched scanner. With the patch, 30 indexes of the 68 files and 5 of
the whole repo each give one result, with no jsonnet file partial.

lint-vendored-scanners (now part of lint-ci) rejects mutable file-scope
state in vendored scanners and the headers next to them. properties and
jsonnet both had it; haskell's and purescript's read-only cases are
whitelisted with their reasons.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-11-13 18:07:03 +01:00
Martin Vogel cd318253e4 fix(pkgmap): pick the manifest nearest the root when two declare one package
Two manifests can declare the same package name: cli/cli has its own go.mod and
a test fixture's copy with the same module path. cbm_pkgmap_build kept the
first entry in worker order, and which worker parses which manifest is thread
scheduling, so about one index run in thirty mapped the module to the fixture
directory. IMPORTS edges fell from 3,869 to 690 and CALLS from 19,898 to
12,392, with status "indexed" and nothing logged.

The candidates are now ordered by package name, depth below the repository
root and path before the first-wins insertion, so the winner is a property of
the repository. The sequential builders go through the same function.

Two tests hand the same manifests to the merge in both worker orders.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-11-13 18:06:53 +01:00
Martin Vogel ea98999e08 Merge origin/main into fix/scala-package-aware-imports
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-11-13 07:37:35 +01:00
Martin Vogel 56f4b91ff6 Merge origin/main into feat/vendor-vba-grammar
Brings in #2957 (sql became a self-maintained fork), #2959 and #2962.
Both sides added a self-maintained grammar fork, so every conflict keeps
both additions:

- MANIFEST.md summary: 164 grammars = 139 vendored-from-upstream + 20
  self-maintained (sql moved over on main, vba added here) + 5
  registry-disagreement; the ABI tally stays 9/78/77 (sql stayed ABI 15).
  The notes and the forks table carry both the sql and the vba rows.
- THIRD_PARTY.md: thirteen further self-maintained forks including sql and
  vba; main's sql paragraph, then this branch's tree-sitter-vba section.
- tests/test_parse_coverage.c: both test blocks and both sets of
  RUN_TEST registrations.
- scripts/vendored-checksums.txt: regenerated with
  security-vendored.sh --update; against main only the MANIFEST.md line
  and the five vba files differ.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-11-12 23:18:44 +01:00
Martin Vogel fcbfed48d2 Merge pull request #2957 from DeusData/distill/2246-sqlite-grammar
fix(grammar): stop flagging valid SQLite migrations as partially parsed
2026-11-12 22:57:52 +01:00
Martin Vogel c3c56c6ddb chore(vendor): vendor the VB6/VBA tree-sitter grammar
Vendors harumiWeb/tree-sitter-vba (MIT, (c) 2026 harumiWeb) at
83c5c3c3c721 (tag v0.14.5, upstream HEAD), its vba dialect, as a
self-maintained fork in tools/tree-sitter-vba/. This is the precondition
for VB6/VBA extraction (#2564, ported after #1179); it registers no
language yet.

Why generated here: upstream does not track its generated parser (its
ADR 0002), so the fork carries upstream's grammar source in upstream's
layout with no local patch, and parser.c is generated in this repository
with tree-sitter-cli 0.25.10 (ABI 15), the generator every fork here
uses. Upstream's own copy is generated by 0.26.9.

Why this pin: v0.13.0 (the pin #2564 carried) exposes the same procedures
but misreads statements without an ERROR node. On the three corpora
below it yields 1,276 expression_statement nodes where v0.14.5 yields 56
(`Err.Raise a, b, c` after a label loses its callee), and it leaves an
ERROR at `Case obj Is Nothing`.

Why the vba dialect rather than upstream's vb6 one: measured with the
CLI on kellyethridge/VBCorLib @ 100acbbae0c0 (655 VB6 files),
Kr00l/VBCCR @ 4b86c5c2c075 (161 VB6 files, 70 .ctl) and upstream's 481
VBA examples, procedures outside ERROR nodes are 14,084/14,084,
12,385/1,552 and 8,104/6,573 (vba/vb6). vb6 rejects the PtrSafe/LongPtr
forms dual-target VB6 keeps inside #If VBA7 blocks and then loses whole
files; vba never sees fewer procedures in any file. vb6's parser.c is
also 75.8 MB against 48.3 MB.

Provenance by hash: common/define-grammar.js is upstream's blob
5e1f7d758ce1; vba/grammar.js minus its 40-line provenance header is
upstream's blob 9247dbbfe166. Two independent generations are
byte-identical, and identical to a generation from upstream's own layout
and tree-sitter.json. The generated node-types.json is upstream's
checked-in file byte-identical (500e635610), and the parse-table counts
equal upstream's own generated parser. Review: 0 non-ASCII bytes, 0
unmatched characters on full tokenization, one comment (the generator
stamp), no string literal with a space, no external scanner, no call
outside the tree-sitter macros. No upstream script, package hook or
binary was run or vendored.

- internal/cbm/vendored/grammars/vba/: the generated parser.c, the
  tree_sitter/ headers (byte-identical to copies already vendored) and
  upstream's MIT LICENSE byte-identical.
- internal/cbm/grammar_vba.c: compiles and links tree_sitter_vba. No
  CBMLanguage, extension or alias routes to it yet; extraction adds them
  under the vb6/vba names (vb and visualbasic mean VB.NET).
- MANIFEST.md (fork note with the measurements and known gaps, forks
  table, 164 grammars, ABI recount), THIRD_PARTY.md (fork list and a vba
  section), scripts/vendored-checksums.txt, and the FORKS entry in
  scripts/audit-license-provenance.py.
- README.md and the Chocolatey nuspec say 164 languages: the
  language-count contract defines the count as the vendored grammar
  directories.
- tests/test_parse_coverage.c: a dual-target VB6 class module (IDE class
  header, Declare in both #If VBA7 branches, Property Get/Let/Set,
  Implements, events, a label, bare calls) parses with no ERROR node and
  every bare call keeps its callee; a CRLF VB6 form parses clean; broken
  source is still flagged. The class case fails on v0.13.0 and on the vb6
  dialect.

Refs #721, #2564

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-11-12 21:35:36 +01:00
Martin Vogel 583e71b6ac Merge origin/main into fix/scala-package-aware-imports (#2360)
Bring #2360 up to date with main b3ec08da (#2427 registry test verdict,
#2962 MSan CI, #2952 Windows stdio guard). The merge is clean and needs
no follow-up edit: #2360's code makes no cbm_registry_add* call and never
used is_test_qn, and Scala definitions register through the shared
seeders in pass_definitions.c, pass_parallel.c and pipeline_incremental.c,
which #2427 already passes the extractor verdict through for every
language. The branch's own change against main is byte-identical to
28948f29's against f5d57465.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-11-12 19:23:57 +01:00
Martin Vogel 74affb1990 Merge branch 'main' into fix/test-impact-gate 2026-11-12 19:07:58 +01:00
Martin Vogel ad257095a5 Merge branch 'main' into distill/2246-sqlite-grammar 2026-11-12 19:07:55 +01:00
Martin Vogel f9dc6503ae ci(test-impact): narrow every test leg, on fork PRs, with base tooling
Three changes to how PR CI consumes the test-impact selection (user
decisions R150 and R151, 2026-11-11).

Fork PRs. select-tests ran only when vars.TEST_IMPACT_MODE was `gate`, and
GitHub passes no repository variables to a pull_request run from a fork:
all 24 fork runs in the last 40 PR runs skipped the job, maintainer-started
ones included. The condition is now (vars.TEST_IMPACT_MODE || 'gate') ==
'gate'. Setting the variable to anything else (e.g. `shadow`) still turns
the gate off for same-repository and Dependabot PRs; for fork PRs the off
switch is the default in the expression on main. No pull_request_target,
no new permissions, no secrets.

Base tooling. select-tests built and ran the PR checkout's own engine,
test-impact-select.sh and selection.py, so a change to them could narrow
its own tests. The job now checks out the merge base (HEAD^1 of the merge
ref) into a worktree, builds that engine and runs that copy of the script
with --repo pointing at the PR checkout. test-impact-select.sh separates
the tooling (its own checkout) from the repository under test (--repo).
Until this lands, a PR's merge base has no --repo: the job then fails and,
as before, every test runs.

All legs. Only the Linux x86_64 test leg consumed a selection. A narrowed
answer now also produces portable/ (selection.py --whole-suites with the
platform-specific suites of the base and HEAD), and the arm64 and macOS
test-unix legs and the Windows leg run that view: platform-neutral suites
as selected, every platform-specific suite whole. No portable view, no
narrowing there. Sanitizer, TSan, MSan, diag and LSan lanes still run
every test. The shard harness already slices the selected list, so the
sharded arm64 leg and the shard-completeness proof need no change.

docs/SMART_CI.md and README.md follow; tests/test_lane_wiring_contract.sh
pins the new wiring.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-11-11 19:50:31 +01:00
Martin Vogel dd3c50f4d5 feat(test-impact): platform-suite classification and a portable view
selection.py learns two things the CI needs to narrow test legs other than
Linux x86_64, where the coverage map is recorded:

- `platform-suites --repo DIR --rev REV...` lists the platform-specific
  runner suites: those whose defining test file holds a SKIP_PLATFORM( call
  or an #if/#ifdef on an OS or architecture macro (the two markers
  scripts/check-no-test-skips.sh names for a genuinely platform-specific
  test), and those tests/test_main.c registers under such a conditional.
  Every revision given is read and the results are united, so a change
  cannot declassify a suite by dropping its conditionals. Any git failure
  exits 1.
- `filter --whole-suites FILE` writes the view for such a leg: every listed
  suite runs whole there, its per-test tokens collapsed to the suite; a
  listed suite the answer did not select is added and marked optional (a
  build that does not register it drops it with a notice, as a full run
  never runs it); the answer's own required suites stay required. An
  unusable list runs everything.

Nothing consumes the new view yet. Contract cases in
tests/test_test_impact_selection.sh.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-11-11 19:50:30 +01:00
Martin VogelandKerf d9b22e0b7f fix(grammar): stop flagging valid SQLite migrations as partially parsed
Valid SQLite migration files were reported parse_partial because the
vendored SQL grammar (DerekStride/tree-sitter-sql @ 851e9cb257ba) rejects
SQLite syntax they use: PRAGMA, AUTOINCREMENT, a trigger's BEGIN ... END
body, a column named `key` after the first column, WITHOUT ROWID, an
unparenthesized trigger WHEN, INSERT OR IGNORE, and CREATE VIRTUAL TABLE
... USING fts5(...). Each became an ERROR node; the trigger's error region
in atuin's ai_session schema also swallowed the CREATE TABLE after it.

The `key` case is lexical: after a comma inside a column list the grammar
also accepts MySQL's `KEY name (cols)` index, so `key` lexes as that
keyword there and `key TEXT NOT NULL` became a malformed index.

Fix: sql becomes a self-maintained fork of the pin we already vendored,
following the pascal model. tools/tree-sitter-sql/ holds upstream's
grammar.js + grammar/ (blob-identical to the pin and to upstream main @
3c99273f9b32, where its corpus lives) and one patch,
patches/sqlite-dialect.patch, touching grammar/ only:
- PRAGMA [schema.]name [= value | (value)] (new pragma_statement)
- AUTOINCREMENT as keyword_auto_increment, upstream's own change
  (c042cde1e2a6, 2026-11-08, after our pin)
- `key` accepted as a column name; the type keywords that take no
  arguments are accepted as a MySQL index name so `KEY name (name)` stays
  an index
- WITHOUT ROWID
- a trigger body as the existing block node, after an optional
  unparenthesized WHEN (precedence keeps Postgres `WHEN (...) EXECUTE
  FUNCTION` trees unchanged)
- INSERT OR {ABORT | FAIL | IGNORE | REPLACE | ROLLBACK}
- CREATE VIRTUAL TABLE [IF NOT EXISTS] name USING module(args), aliased
  to create_table (new module_argument node), so the table is extracted
  as a Table like any other
Upstream node types keep their shape: upstream's 511 corpus cases pass
unchanged; test/corpus/sqlite.txt adds 11 (9 fail on the unpatched
grammar, 2 controls pass on both). The vendored scanner.c is kept as is
(it carries our start_tag leak fix 0e61867d9, now listed under the
MANIFEST's "Local source patches"; upstream fixed the same leak after our
pin in ee2f84f9b857).

Provenance: without the patch, tree-sitter-cli 0.25.10 regenerates
upstream's shipped parser.c byte-for-byte except line 1 (upstream's
generator stamp has no version), and grammar.json, node-types.json and
tree_sitter/*.h byte-identical. Patched: three generations are
byte-identical (parser.c SHA-256 3c64e5c4...7887); ABI 15; states 30,622
-> 31,081, symbols 770 -> 782; 0 non-ASCII bytes, no string literal with
a space, one comment, the same call-position identifiers as before. Node
types only gain entries (pragma_statement, module_argument, four
keywords; create_table gains a `module` field).

Proof, indexed from scratch with isolated caches (main 6d34ca15 vs this
change): dani-garcia/vaultwarden @ 58a980f8 migrations/sqlite (116
files) parse_partial 3 -> 0, graph identical (303 nodes, 446 edges);
atuinsh/atuin @ 2535161e *.sql (69 files) 19 -> 7: all 12 valid SQLite
files clean; the 7 left are the invalid `ALTER TABLE ... DROP INDEX`
down migration and the 2 MySQL + 4 Postgres files, same ranges as
before; graph +2 Table nodes (the fts5 table and the table the trigger
error swallowed), +2 DEFINES, +3 USAGE lineage edges. No newly flagged
file. Control: helm @ 9b88efd6 indexes to an identical graph (12,251
nodes, 39,561 edges).

Registered like pascal: MANIFEST.md fork rows + notes (ABI tally
recounted, unchanged), the audit FORKS map, THIRD_PARTY.md, vendored
checksums. LICENSE stays upstream's MIT byte-identical.

Tests: parse_coverage carries the two SQLite tests from #2246 as written
(sqlite_valid_migration_not_flagged, red before this change, and the
sqlite_malformed_still_flagged control) and adds `key` columns, WITHOUT
ROWID, a WHEN trigger with INSERT OR IGNORE plus an fts5 table (asserted
as a Table), PRAGMA forms, and two controls: MySQL's KEY index still
clean, and ALTER TABLE ... DROP INDEX still flagged.

Distilled from #2246.

Co-authored-by: Kerf <kerfern@hotmail.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-11-11 19:44:47 +01:00
Martin Vogel 32a73eb7fd Merge origin/main into fix/scala-package-aware-imports (#2360)
Bring #2360 (package-aware Scala imports, closes #2153) up to date with
main f5d57465. The contributor's commit 14d43883 is untouched; every
conflict is resolved so that the change stays as written and main's
changes are kept.

Conflict resolutions:
- src/pipeline/pass_pkgmap.c: rebuilt as main's file plus the PR's own
  blocks. Main moved Strategy 2 (namespace map) out of
  cbm_pipeline_resolve_import_node into resolve_import_strategies, so the
  Scala dispatch (language == CBM_LANG_SCALA ->
  resolve_scala_namespace_import) and the declaring_file_is_scala guard on
  the generic namespace walk now sit there, directly before and inside
  Strategy 2 as in the PR; resolve_import_strategies takes the PR's
  language and scala_index. Main's namespace-map builder (fail-closed:
  NULL on a construction failure, an empty map when no file declares a
  namespace) stays the single implementation: it becomes the ctx-aware
  static namespace_map_build that records each clause as the File node's
  `package` property (the PR's file_node_set_package), and the public
  cbm_pipeline_namespace_map_build{,_names} and
  cbm_pipeline_namespace_map_free that main's tests and pipeline cleanup
  call are kept. The PR's key conversion to cbm_mem_strdup was already on
  main. cbm_pipeline_import_maps_build builds both maps on top of it.
  Main's PHP IMPORTS writer (cbm_pipeline_php_create_import_edges) calls
  the re-signed resolver with CBM_LANG_PHP and no Scala index.
- src/pipeline/pass_definitions.c, src/pipeline/pass_parallel.c: main's PHP
  branch in the IMPORTS writers kept, the PR's call with language and
  scala_index in the generic branch. Main publishes the namespace map to
  the pipeline (#1355) instead of freeing it, so only the Scala index is
  freed at the end of the pass; main's two new ABORT_PRESERVE_DB exits in
  the registry loop free the Scala index as well.
- src/pipeline/pass_calls.c: the PR's build_import_map(..., language, ...)
  call next to main's PHP vendor-name lines.
- src/pipeline/pipeline_internal.h: the PR's cbm_file_namespace_t, the
  cbm_scala_index_t declaration, the stored_namespaces context fields and
  the import-maps API next to main's env-join and Terraform types, stem
  marks, env-join base and PHP vendor declarations; main's public
  namespace-map declarations kept.
- src/pipeline/pipeline_incremental.c: incr_load_stored_namespaces and
  incr_free_stored_namespaces next to main's coverage helpers. Closure
  route: main's doc-link scopes are loaded first, then the stored package
  clauses, so a failed doc-link read cannot leak them; the context carries
  the PR's fields and main's. Legacy route: main removed the ADR capture
  (ADRs live in the sidecar database), so the PR's shared ADR/namespace
  abort becomes a read of the stored clauses after a successful graph load
  that shares the failed load's cleanup (a failed load logs and returns
  exactly as on main; stored_namespaces_failed returns
  ABORT_PRESERVE_DB as in the PR).
- tests/test_pipeline.c: main's file plus the PR's own hunks; the four
  RUN_TEST lines follow the same anchor line.
- tests/test_edge_imports.c: main's two PHP resolver tests (#1186) adapted
  to the re-signed cbm_pipeline_resolve_import_node (CBM_LANG_PHP, NULL
  Scala index), the convention the PR used for its own call sites.
- tests/test_extraction.c, tests/test_edge_imports.c: the PR's reformat of
  extract_ts_await_generic_call_issue2210 and its pipeline_internal.h
  include were already on main (e8a46c16e), so they drop out of the diff.
- scripts/memory-core-baseline.txt: recomputed against main's baseline:
  pass_definitions.c 11 -> 9 and pass_parallel.c 31 -> 29 (the PR drops
  their raw rels arrays); no line raised. pass_pkgmap.c is one raw site
  above main's baseline after this merge; the follow-up commit removes it.

The index format bump for the new File-node property follows in the next
commit.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-11-11 19:41:48 +01:00
Edson abbaba65a0 Merge origin/main into fix-haskell-escaped-char-swallow
Resolve the vendored-registry conflicts with the vbnet grammar (#1179):
keep both the haskell fork and vbnet entries in THIRD_PARTY.md and
MANIFEST.md, recount the manifest totals (163 = 139 upstream + 19
self-maintained + 5 registry-disagreement; ABI line unchanged), and
refresh the MANIFEST.md checksum.

Signed-off-by: Edson <zhuoxi2000@gmail.com>
2026-11-10 21:21:48 -05:00
c05de88247 test(windows): guard clean MCP stdio startup
Adds a native Windows guard for #654: a successful MCP initialize +
tools/list exchange, run against an isolated cache and runtime directory,
must leave nothing on stderr but the server's own structured log lines.

- tests/windows/test_mcp_stdio.py: the guard. The English and Spanish
  path-not-found messages reported in #654 are named cases; any other
  non-empty stderr line that is not a structured "level=<name> msg=<event>"
  record fails as well, so the same leak on a machine in another language
  is caught without listing its translation. The log format is pinned to
  text so an inherited CBM_LOG_FORMAT=json cannot change the verdict. Cache
  and runtime live under one temp root, so both are removed with it, also
  on failure, after the daemon is stopped.
- tests/windows/mcp_stdio.py: close() joins the stderr reader, and the new
  stderr_complete() reports whether the reader reached EOF. The guard reads
  stderr only after it has; a reader still running after the server exited
  is a SETUP failure (exit 2), never a verdict on a partial capture.
- scripts/test-windows.ps1: registers the guard in the green guard list
  and documents it in the header.

#2213 was an earlier duplicate of the same check.

Refs #654.

Distilled from #2217.

Co-authored-by: lorenzozanee <wyz0707@proton.me>
Co-authored-by: Shinken009 <182390694+Shinken009@users.noreply.github.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-11-11 00:57:46 +01:00
Martin Vogel a8222606b2 fix(grammar): stop the VB.NET XML-literal scan at attribute lines
The vendored vbnet scanner lexes a whole VB XML literal as one external
token, and the parser offers it a '<' wherever a declaration or member
starts. There, in valid VB, '<' opens an attribute block
(`<Serializable()> _`), never an XML literal. The scanner nevertheless
took `<Serializable` for an element start and read the rest of the file
looking for a closing tag before giving up: O(rest of file) per
attribute, O(n^2) per file. Generated proxies and designer code put an
attribute on every member: 2,500 attributes in a 77 KB file advanced
38.5 M characters, and a 1.35 MB synthetic file took 21 s to parse.

The scan now fails at the first character well-formed XML cannot
contain: inside a tag, outside a quoted value, anything other than name
characters, blanks, '=', quotes, '/>', '>' and '<%='; in element text, a
'<' that starts no tag, closing tag, comment, PI or embedded expression,
and an '&' that starts no reference. No valid XML literal lexes
differently: parse trees are byte-identical before and after on all 25
sample and synthetic files tried, XML literals of every shape included.
The 1.35 MB file now parses in 0.08 s.

Residual, recorded in MANIFEST.md: an attribute without an argument list
(`<Fact>`) is a well-formed start tag, so its scan still runs until a
character that cannot be XML, or the end of the file.

The change is kept as tools/tree-sitter-vbnet/patches/
xml-literal-tag-syntax.patch (CodeGraph's scanner 8334fdb -> b3473d1)
and in MANIFEST.md's local-patch table. The new
complexity_vbnet_attribute_lexing_is_linear counts the bytes the lexer
pulls through a chunked TSInput for k and 2k attribute-decorated
classes: ratio 3.99 before the fix, 2.0 after.

Refs #1178, #1179

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-11-10 21:37:56 +01:00
Martin Vogel 970dd5656a chore(vendor): vendor the VB.NET tree-sitter grammar
Vendors govindbanura/tree-sitter-vbnet (MIT, (c) 2025 Govind Banura) at
538b7087bf80 (tag v0.1-beta, upstream HEAD) with CodeGraph's real-world
VB.NET patch (colbymchenry/codegraph docs/grammars/tree-sitter-vbnet.patch
at fe7542b6f6a4, MIT, (c) 2026 Colby Mchenry), as a self-maintained fork
in tools/tree-sitter-vbnet/. This is the precondition for VB.NET
extraction (#1179, folding in #2885); it registers no language yet.

Why this grammar: it is the licensed one. CodeAnt-AI/tree-sitter-vb-dotnet,
which #1179 was built on, carries no license and cannot be vendored.

Why the patch: upstream checks in grammar.js only and does not parse the
code VB.NET projects are made of. Unpatched, the Single literals in every
WinForms .Designer.vb (New SizeF(6.0!, 13.0!)) and body-less MustOverride
members are ERROR nodes. The 2026-10-06 revision of the patch is pinned
because it also keeps names that begin with a keyword whole: the earlier
revision (the one #2885 carried) turns `Sub NewItem()` into an ERROR and
`Public SharedCache As Integer` into a Shared field named Cache.

Provenance by hash: the patch file is byte-identical to CodeGraph's (blob
37918d81); upstream grammar.js at the pin is the patch's pre-image
(21c8523), and tools/tree-sitter-vbnet/grammar.js minus its 41-line
provenance header is the post-image (c9c1d85), as is scanner.c (8334fdb).
parser.c is generated here with tree-sitter-cli 0.25.10 (ABI 15); two
independent generations are byte-identical. No upstream script, package
hook or binary was run or vendored.

- internal/cbm/vendored/grammars/vbnet/: the generated parser.c,
  CodeGraph's external scanner.c (two stateless tokens), the tree_sitter/
  headers, upstream's MIT LICENSE byte-identical, and CodeGraph's MIT text
  as LICENSE-codegraph: the scanner and the patched rules are CodeGraph's
  work, so its notice has to ship with the binary too.
- internal/cbm/grammar_vbnet.c: compiles and links tree_sitter_vbnet. No
  CBMLanguage, extension or alias routes to it yet; extraction adds them,
  with `vb` and `visualbasic` meaning VB.NET (VB6 keeps its own name).
- MANIFEST.md (fork note, forks table, 163 grammars, ABI recount),
  THIRD_PARTY.md (fork list and a vbnet section; the stale "160
  pre-generated parsers" now reads 163), scripts/vendored-checksums.txt,
  and the FORKS entry in scripts/audit-license-provenance.py.
- README.md and the Chocolatey nuspec say 163 languages: the
  language-count contract defines the count as the vendored grammar
  directories.
- tests/test_parse_coverage.c: real VB.NET (a WinForms Designer partial
  class, Handles, Inherits/Implements, MustOverride, #Region, Property,
  keyword-prefixed names) parses with no ERROR node. The case fails on
  unpatched upstream (8 ERROR nodes) and on the earlier patch revision (1).

Refs #1178, #1179, #2885

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-11-10 21:20:25 +01:00
Martin Vogel 612eee2518 Merge pull request #2911 from DeusData/ci/freebsd-dispatch-workflow
ci(freebsd): add a dispatch-only, informational FreeBSD build workflow
2026-11-05 00:21:43 +01:00
Martin Vogel fe55b86c61 Merge pull request #2749 from tahodev/argv-grep-find-2634
refactor(search): run POSIX unscoped scan as argv stages, no shell (#2634)
2026-11-05 00:21:20 +01:00
Martin VogelandPedro Ramos e750c0531a ci(freebsd): add a dispatch-only, informational FreeBSD build workflow
Add .github/workflows/freebsd.yml, a standalone workflow triggered only
by workflow_dispatch. It builds the UI-embedded binary inside FreeBSD
15.1 and 14.4 x86_64 VMs (vmactions/freebsd-vm, full-SHA pin to v1.5.9)
with the canonical scripts/build.sh and runs the binary with --version.
It declares read-only permissions, uploads and publishes nothing, is not
part of ci-ok and is not called by release.yml or dry-run.yml, so a
FreeBSD red never gates a pull request or a release.

scripts/env.sh now detects FreeBSD and selects gmake there (BSD make
cannot read the GNU make syntax of Makefile.cbm) and make everywhere
else; the environment summary line reports the choice. scripts/build.sh
runs $MAKE at all three build call sites. macOS, Linux and Windows still
run make, so their builds are unchanged.

Distilled from #1422 by PR9000: the release matrix, the x86_64-host
choice, the aarch64 exclusion, the vmactions VM approach with a SHA pin,
fail-fast: false and the gmake detection all come from that pull
request, which built on the FreeBSD groundwork of #1093, #1423 and #1467.

Co-authored-by: Pedro Ramos <131530838+pr9000@users.noreply.github.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-11-04 18:10:27 +01:00
Martin Vogel 980c8ed932 Merge origin/main into persistent-install-hook-opt-out
Bring the branch up to main e0c7a1da6c
(#2395), which includes #1724 (index resource worker and publication
fence, 008b6c3585).

Conflict resolved (1 file):
- docs/CONFIGURATION.md: both sides appended rows to the "Current keys"
  table at the same spot. Resolved as main rows index_max_rss_mb and
  index_max_duration_seconds first, then the branch row install_hooks
  and its paragraph that closes the table. No conflict side was
  concatenated as-is; the branch paragraph must follow the last table
  row.

Auto-merged without conflict: README.md, src/cli/cli.c (CONFIG_KEYS
now holds the new index_max_rss_mb / index_max_duration_seconds keys
and install_hooks), src/main.c (worker publish-fence argv next to the
--hooks/--no-hooks help line).

Check: for all seven branch files, the zero-context patch-id of the
merged tree against main equals the branch own change. Semantic check:
the branch re-signs no function; main re-signed the daemon worker start
callback and the supervisor result struct, which the branch does not
use. No call site adapted.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-11-03 22:40:15 +01:00
Martin Vogel b9ba6c0360 Merge origin/main into argv-grep-find-2634
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-11-03 22:03:56 +01:00
Martin Vogel 008b6c3585 Merge pull request #1724 from liuchong/feat/index-resource-worker
feat(index): add worker resource watchdogs
2026-11-03 21:20:43 +01:00
Martin Vogel 6ab3653b85 Merge origin/main into feat/index-resource-worker
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-11-03 18:09:48 +01:00
Martin Vogel 714a0cf04e Merge origin/main into fix/lint-mem-gate-warning-suffix
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-11-03 07:44:32 +01:00
Martin Vogel 7e67581448 Merge origin/main into persistent-install-hook-opt-out
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-11-03 03:04:38 +01:00
Martin Vogel 334a408b63 Merge pull request #2198 from astandrik/fix/1696-bench-runtime-isolation
fix(test): isolate benchmark harness daemon runtimes
2026-11-03 01:51:49 +01:00
Martin Vogel 1d67510e1e Merge origin/main into fix/lint-mem-gate-warning-suffix
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-11-02 18:24:55 +01:00
Martin Vogel 140e2998a1 Merge origin/main into feat/index-resource-worker
Brings the branch up to main 9c19b5dd.

Two same-spot insertion conflicts, each resolved as the main version
of the file plus the own diff of this branch:
- src/daemon/application.c: the job struct gained watch_owners on main
  and request_started_ms plus the request facts here; both kept.
- tests/test_daemon_application.c: main added the #1818
  autoindex-skipped helpers where this branch added the worker resource
  failure test; both kept, the test placed after the doc_candidates
  test.

Everything else merged cleanly. src/main.c changed on both sides: main
(#2838) lowers the worker CPU priority inside the worker role block,
and this branch strips the publication fence flag before that block
and installs the fence after the worker role options; the hunks are
disjoint and both are kept. tests/test_pipeline.c gained the Next.js
route tests (#2747) on main and the publication gate test here, also
disjoint. No call site needed adapting: no new main code calls a
function this branch re-signed, and main re-signed none that this
branch calls.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-11-02 18:15:55 +01:00
Martin Vogel a7a9f27fae Merge origin/main into fix/1696-bench-runtime-isolation
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-11-02 18:10:57 +01:00
Martin Vogel a508219299 Merge origin/main into feat/ruby-lsp-per-file
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-11-02 18:09:59 +01:00
Martin Vogel 1adabd8c57 Merge origin/main into carry/1862-install-hook-optout
Maintainer refresh of #1862 (persistent install-hook opt-out, direction
decided in #753) onto main a5b248f4. The first refresh is 2577a86a and
carries the semantic fixes for shell-free Claude hooks.

Textual conflicts, all resolved by keeping both sides:

- src/cli/cli.h: main added CBM_CONFIG_DOC_CANDIDATES on the line where
  the PR added CBM_CONFIG_INSTALL_HOOKS. Both defines are kept, in
  alphabetical order.

- src/cli/cli.c CONFIG_KEYS[]: main added the watch_non_git row where the
  PR added install_hooks. Main row stays first, followed by the
  install_hooks row.

- src/cli/cli.c Codex hook install: main (#2719) declared pretool_written
  in front of the hooks.json branch the PR prefixed with its --no-hooks
  removal branch. The declaration stays first, then the PR removal
  branch, then main hooks.json and config.toml upsert branches unchanged.

Check: for every file the PR touches, the added and removed lines of the
merged tree against main equal those of the PR against its old base.

No compile-level semantic conflict: no function the PR calls changed its
signature, and the PR re-signs no function main calls. The new Codex
PreToolUse hook from #2719 is gated by the opt-out in the next commit.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-11-02 01:37:59 +01:00
Martin Vogel 020671b58b Merge origin/main into fix/lint-mem-gate-warning-suffix
Brings the branch up to date with main (a5b248f4). No conflicts.
scripts/test.sh keeps Step 0e4 next to Step 0e3; the gate script and
its test are unchanged by main.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-11-02 01:12:11 +01:00
Martin Vogel 91070cb2f0 test(lint-mem): run the warning-suffix gate test on every leg
scripts/test.sh lists every shell contract test explicitly and does
not glob tests/, so tests/test_lint_mem_gate.sh never ran on any leg.
Register it as Step 0e4, next to the other gate contract (Step 0e3),
in the same two-line shape as its neighbours. Step 0 runs in every
default scripts/test.sh leg (macOS, Linux, Windows) and in the PR
contracts-only job. The wiring, Windows included, is a maintainer
decision (R1).

Make the test portable to the Windows leg. There python3 is a native
MSYS2 build: MSYS converts path arguments for it, but never the
diagnostic text on stdin, so the absolute fixture path inside the
finding does not resolve there, and a C:/ path would not parse (the
file field of FINDING_RE ends at the first colon). The gate now runs
from inside the fixture directory, in a subshell, with relative paths.
CRLF from a native python3 is stripped from the --hash output, the
whitelist is written once the hash is known (no placeholder rewrite),
the gate exit status is asserted, and every failed assertion is
reported together with the gate output.

Refs #2442

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-11-02 01:11:32 +01:00
Martin Vogel 760d73c91b Merge origin/main into fix/1696-bench-runtime-isolation
Brings #2198 (benchmark harness runtime isolation, #1696) up to date
with main a5b248f4.

No conflicts. Of the files this branch touches, main changed only
scripts/test.sh since the merge base 8530c8cd: it adds Step 5-alloc
(production allocator contract), far from this branch's Step 0i3,
which is unchanged. scripts/test-runtime.sh, which the harnesses and
their contract test source, is unchanged on main. No caller of either
harness outside docs/EVALUATION_PLAN.md exists on main.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-11-02 00:08:59 +01:00
Martin Vogel 78bf6a2745 fix(test): hand over a kept benchmark runtime only after its daemon stopped
With CBM_BENCH_KEEP_RUNTIME=1, bench_finish ran `daemon stop` with its
output and exit status discarded and wrote runtime-root.txt regardless.
`daemon stop` refuses while a committed client still uses the daemon and
fails when the daemon does not answer; the handoff then promised a
stopped daemon that was still running.

A refused stop now prints the explanation of the product, writes no
handoff, keeps the root without cleaning it up (the daemon may still be
using it) and exits non-zero. A stop with nothing to stop still exits 0,
so a clean run is unaffected.

Section 7 step 8 of the evaluation plan follows the same rule: the kept
root is removed only if its daemon stopped, and is kept with a message
otherwise.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-11-02 00:06:52 +01:00
Martin Vogel b9851700a4 fix(test): surface why a benchmark daemon did not start
Both harnesses discarded the output of `daemon start`, so a refused
start was reported only as "private daemon did not start", the symptom
without its cause. The output now goes to a run-private file next to the
index stderr, the shape both scripts already use for the index call, and
is printed under "--- daemon start output ---" when the start fails.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-11-02 00:06:52 +01:00
Martin Vogel 14b8da00c3 fix(test): drop a stale benchmark handoff before setup can fail
benchmark-index.sh removed the runtime-root.txt of an earlier run only
after the repository path, the private runtime and the temp directory had
been set up. A run that failed in one of those steps exited non-zero but left
the previous handoff in the reused results directory, and section 7 of
the evaluation plan sources that file unconditionally, so the graph
session would be pointed at a root that step 8 has already removed.

The results directory is now created and the stale handoff removed right
after argument parsing, before any setup step that can fail.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-11-02 00:06:52 +01:00
Martin Vogel d3cf9660c2 Merge origin/main into fix-haskell-escaped-char-swallow
Brings the branch up to origin/main 9f278b7e. One conflict, in
scripts/vendored-checksums.txt: both sides re-hashed the MANIFEST.md
line (main for its objectscript scanner patches, this branch for the
haskell fork). Resolved by taking main's file and re-running
scripts/security-vendored.sh --update on the merged tree; against
origin/main the only lines that differ are MANIFEST.md (the merged
file) and haskell/parser.c (this branch's regenerated parser). Every
other file merged cleanly and carries exactly this branch's change.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-11-01 23:09:45 +01:00
Martin Vogel ed2056eb53 fix(grammar): drop the duplicate haskell MANIFEST row and re-hash it
The fork registration put the haskell fork row in the "Self-maintained
forks" table and also replaced the original row in the "Vendored from
verified upstream" table with the same text, so haskell was listed
twice and the upstream table carried 140 rows against the 139 the
summary line declares. pascal, the model for this fork, is listed only
in the forks table; drop the copy from the upstream table so both
tables agree with the summary counts (139 upstream, 18 self-maintained).

The license audit is unaffected: haskell is routed through the FORKS
map, and its manifest parser never matched the fork-format row.

MANIFEST.md is itself covered by scripts/vendored-checksums.txt, so
re-hash it with scripts/security-vendored.sh --update; that MANIFEST.md
line is the only one that changes. The vendored haskell parser.c is not
touched.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-11-01 23:05:39 +01:00
Martin Vogel 79f674efe0 Merge origin/main into fix/termux-portable-guard
Brings the branch up to current main (it was 2061 commits behind) so
the new Step 0x2 runs inside the current scripts/test.sh. No conflicts.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
2026-11-01 22:44:24 +01:00