scripts/memwaste.sh set no memory budget, so the daemon derived one from
the machine's RAM: 1792 MB on the 7 GB macos-14 runner, about 4 GB on the
16 GB Linux and Windows runners. The event layer itself holds 0.5-1 GB, and
on macOS the index stopped with over_memory_budget before it measured
anything, in both full-mode runs since the lane was added (dry runs
37212102899 and 38186491924, peaks 3449 and 3111 MB at the stop).
The script now passes CBM_MEM_BUDGET_MB=4096, what the larger runners
already derived, unless the caller sets one.
Checked on macOS arm64 with CBM_WORKERS=3, the macos-14 runner's core
count:
- budget 1792: over_memory_budget in parallel_extract (rss 3904 MB), the
CI failure;
- budget 4096: the index completes and the report is written;
- the edited script with no budget in the environment: 4096 is applied,
rc 0.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Five suites (grammar_regression grammar_labels pipeline lang_contract
grammar_probe_e) were excluded from the MSan lane for "stack-overflow
under instrumentation". That was never deep recursion. MSan aborted while
REPORTING real uninitialized reads:
- every abort was on the main thread. MSan's "T<n>" is the OS thread id
and equalled the pid in all five x86-64 aborts (CI job 91868259558);
the old block read it as a worker-thread ordinal, so the worker-stack
knob, MSAN_ORIGINS and CBM_WORKERS=1 could never change anything;
- with the default slow unwinder (the image's MSan-instrumented
libunwind) a report exhausted the main thread's 256 MiB stack, and the
overflow handler then failed with "nested bug in the same thread";
- with fast_unwind_on_fatal=1 the same test prints the
use-of-uninitialized-value report instead (measured on one tree and
test: slow unwind aborts at the stack limit, fast unwind reports
wit/scanner.c:13).
The two reads are fixed in the previous commits of this series, so:
- MSAN_OPTIONS defaults to fast_unwind_on_fatal=1 (the build already
keeps frame pointers);
- MSAN_EXCLUDE is down to cli, whose install-path failure is a separate,
still-undiagnosed cause;
- the known-red block, the header (it still claimed CI runs without
exclusions) and the MSAN_ORIGINS and stack comments say what is true;
- the partial-lane warning and the exclude-name typo check lived in a
branch behind `elif true` that never ran. They now run in the
per-suite path, the check as a case match instead of a grep -q pipe
that pipefail can misreport;
- the comments in scripts/ci/msan-lane.sh and above the test-msan job
in _test.yml no longer say the five suites overflow.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
tree_sitter_wit_external_scanner_create() is declared to return void *
but had an empty body. tree-sitter's runtime stores its result as the
scanner payload (parser.c ts_parser__external_scanner_create), so every
WIT parse used an indeterminate value, which C11 6.9.1p12 makes
undefined. The scanner never reads its payload, so ordinary builds were
unaffected, and the vendored build's -w hid -Wreturn-type.
MemorySanitizer reports it at wit/scanner.c:13 from grammar_labels,
grammar_probe_e, grammar_regression and lang_contract. Those four suites
were excluded from the MSan lane because the report itself crashed and
posed as a stack overflow (fixed in the msan.sh commit of this series).
Recorded in MANIFEST.md's local-patch table. Upstream
bytecodealliance/tree-sitter-wit still has the empty body at
059bfdd1af72, the last change to src/scanner.c.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Reconcile the complete Atlas and Lens stream with current main while retaining both histories. Preserve architecture functionality, current process and watcher safeguards, test infrastructure, and dependency security updates. Add coverage for Git-environment isolation combined with child-local overrides.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A bare import specifier was always external. The TS resolver now asks the
pipeline what it already knows for import resolution, without indexing
package.json or tsconfig.json: the nearest tsconfig/jsconfig `paths` and
`baseUrl` (the alias collection; the first target, as TypeScript tries it
first), then the workspace package whose `name` the specifier is (the
package map), bound to the package's entry module (`exports`, `main`,
`module`) when the repository indexes it. Every other bare specifier, a
package's subpath included, stays external. Incremental runs stay exact:
a tsconfig change re-resolves the files under it, a package.json change is
a manifest change.
path_alias gets cbm_path_alias_resolve_into, which writes into the caller's
buffer; cbm_path_alias_resolve now wraps it (two raw allocations fewer, the
memory-core baseline lowered to match).
Also fixed: a declaration in an index file has no `index` in its QN (fqn.c
strips it from a symbol's QN, not from the module's), so a reference to one
missed its node; the resolver now names a file's declarations with that
prefix.
Proof (ship build, Apple M5 Pro / 48 GB): the TypeScript compiler
b465fdbfe175 unchanged (69 edges, 57 of the prototype's 61 pairs);
discord.js 4dc1acc6 unchanged (no doc reference names a workspace import;
its packages' entries are build output). Tests: a tsconfig alias, a
workspace package's entry and an index file's declaration each bind, each
red with its rule reverted; path_alias, pipeline, edge_imports, incremental
and all doc suites pass (1298).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
81264814 built the production allocator as mimalloc's release build
(-DMI_DEBUG=0). That bought a large speed-up, but it raised what macOS
charges an index worker, its phys_footprint:
| peak phys_footprint | v0.11.0 | main |
|---|---:|---:|
| TypeScript | 1.25 GB | 1.94 GB |
| dotnet/runtime | 4.5 GB | 12.7 GB |
The 2026-10-09 bench missed most of this, because it sampled RSS.
The difference is one compile-time branch in mimalloc's unix decommit.
With MI_DEBUG or MI_SECURE>2, a purged range is mprotect(PROT_NONE)'d
and flagged needs_recommit. A release build leaves it "committed".
When the arena hands such a range out again, it is re-accounted as a
whole with MADV_FREE_REUSE (arena.c, _mi_os_reuse). macOS then charges
every page of it to the footprint, touched or not.
On macOS, take the recommit path in every build. Linux and Windows keep
upstream's behaviour; there MADV_DONTNEED already drops RSS at once.
Measured on Apple arm64 (macOS 26.6), main ec4fbd3a vs this change, cold
full indexes interleaved with main (C#: 2 runs each, in the same session
as main):
| corpus | footprint before | footprint after | change | wall time |
|---|---:|---:|---:|---|
| TypeScript | 1.98 GB | 1.09 GB | -45 % | unchanged |
| kubernetes | 2.92 GB | 1.88 GB | -35 % | unchanged |
| dotnet/runtime | 12.6 GB | 4.9 GB | -61 % | unchanged |
| linux kernel | 16.0 GB | 13.5 GB | -16 % | 574 -> 559 s |
The graph is byte-identical: TypeScript, kubernetes and django give
the same nodes, edges and CALLS-set hash.
dotnet/runtime is now close to v0.11.0's 4.5 GB.
Measured but not taken:
- the whole debug build, which costs ~5 % wall time on C#;
- MI_STAT=2, MI_PADDING=1 or MI_ENCODE_FREELIST=1 alone, none of which
moves the footprint;
- MIMALLOC_PURGE_DELAY=0, which is already set.
Vendored file changed in place (precedent 1f674c80); its hash in
scripts/vendored-checksums.txt is updated and security-vendored.sh passes.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Resolves the definition walks next to each other: main's decl_namespace (#2120) and this branch's extra_props are both compacted and both counted in the result census.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The PDF text layer carries data tables (D3). Their sources and notices are
now in THIRD_PARTY.md (shipped in every archive through
THIRD_PARTY_NOTICES.md) and in the SBOM:
- the Adobe Glyph List subset (BSD-3-Clause, Copyright 2002-2019 Adobe),
- the printable-ASCII advance widths of Helvetica and Times-Roman from
Adobe's core-14 font metrics (the AFM files are not distributed),
- the Unicode Character Database data of pdf_unicode.c (Unicode
License v3, full text included).
scripts/gen-pdf-unicode.py (stdlib CPython only) regenerates
pdf_unicode.c byte for byte with CPython 3.14.6 (Unicode 16.0.0);
pdf_tables.c is maintained data from here on. License texts were taken
from their sources (adobe-type-tools/agl-aglfn, unicode.org).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The test-impact engine certifies the history it diffs and refuses a shallow
clone ("Graft or shallow state prevents history certification"). The shadow
job, the gate-mode select-tests job and the main-branch producer all checked
out with fetch-depth 2. So every shadow answer since #2458 was run_all:
GIT_UNAVAILABLE, then POLICY_UNAVAILABLE, because the merge base's
.codebase-memory.json is read through the same git facts. That was 24 of 24
measured PR runs. All three jobs now check out full history (about 400 MiB,
these jobs only).
The producer never published a bundle either. coverage-map.py exits 1 when a
suite fails under coverage: the map is still written and that suite's tests
are incomplete. test-impact-publish.sh ran it under set -e, so the publish
aborted. The failing suite was test_impact_git. Its hostile-diff probe
re-enters the runner as a child, which inherited CBM_TEST_COVERAGE_DIR,
started a coverage run of its own, and refused the per-test LLVM_PROFILE_FILE
it inherits.
Two changes:
- The probe now clears CBM_TEST_COVERAGE_DIR, as it already clears
CBM_TEST_ONLY_FILE. The child's profile then goes to the spawning test, like
any other child process.
- publish treats exit 1 as a written, conservative map (incomplete tests run
whole), with a warning that names the suite. It fails only on exit 2.
Proof:
- Depth-2 clone at f9ce3865: test-impact select answers run_all
[GIT_UNAVAILABLE, POLICY_UNAVAILABLE] with "Graft or shallow state ...", as
in CI. Same commit with full history: graph certified, run_all
[RUNNER_REACHED] (the change reaches the runner's main).
- Coverage runner (Homebrew LLVM, macOS), test_impact_git: before, 43 passed
and 2 failed ("a coverage run starts with LLVM_PROFILE_FILE=/dev/null");
after, 45 passed. coverage-map.py --suite test_impact_git: 45 tests,
0 incomplete, exit 0.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The full Windows leg on the VM stopped at the coverage-map contract and was
reported green. Two defects, one hiding the other:
- The VM verdict counted the completion marker anywhere in the log. The
isolated-venue contract's stubbed entry prints that marker, and a capacity
case prints "expected=0 passed=True", so a leg that stopped at a later
contract, with no suite run, read as complete. The marker now counts only as
the log's last step header, which is where scripts/test.sh prints it; the
verdict contract pins the observed shape (14 cases).
- Under the native python3 the Windows legs install, the test-impact
contracts failed: the coverage-map builder cannot start the leg's bash
fakes, the scripts wrote CRLF, and cmp (diffutils) is not installed there.
selection.py, coverage-merge.py and coverage-map.py now write LF on every
host, so their outputs are byte-identical across platforms; the two cmp
checks compare bytes in the shell; the builder's functional leg is skipped
under a native Windows python with the reason and what was tried (the map
is built only by the ubuntu publish job), while both pure-python parser
tests still run there.
Production was not affected by the CRLF: run-tests-parallel.sh strips CR from
the selection and the runner trims its tokens.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
verify-shard-union.sh piped the shard slices into sha256sum, which macOS
runners do not ship. The new lane-gate contract cases run the script on
every venue, so test-unix (macos-14) failed with "sha256sum: command not
found" (GATE-QUALITY LOSS on a complete leg). Fall back to shasum -a 256,
as run-tests-parallel.sh already does.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
pr.yml and test-impact-artifact.yml run scripts/ci/test-impact-select.sh,
-bundle.sh and -publish.sh as commands, but they were committed 100644
(core.fileMode is false in this clone, so the disk mode never reached the
index), and the PR shadow job failed with "Permission denied" (exit 126).
tests/test_script_exec_bit_contract.sh reports exactly this; it skipped the
scripts locally while they were still untracked.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Ten files changed on both sides; every conflict was a union of independent
additions:
- workflows (_lint.yml, _test.yml): keep the branch's lane `if:` and main's
pinned ubuntu-24.04 runner with its reason.
- internal/cbm/cbm.h: main's callable-identity and module-doc fields plus the
branch's configured-test fields.
- Makefile.cbm: both new test files; prod_lsp_all.o keeps main's
.build-config dependency and the branch's coverage flags.
- scripts/test.sh: main's setup-scripts contract becomes Step 0y3, ahead of
the branch's 0z series and the contracts-only exit.
- src/foundation/subprocess.c: main's cbm_posix_close_fds_from (it closes the
error descriptor too).
- src/mcp/mcp.c: main's repository default base for detect_changes, with the
early returns released through the branch's detect_release.
- src/pipeline/lsp_resolve.h, tests/test_main.c: both includes, suites and
start-up steps (main's git environment reset first).
- scripts/memory-core-baseline.txt: recounted on the merged tree and lowered
(mcp.c 773, config_toml_edit.c 73).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Producer. test-impact-artifact.yml publishes, on every push to main, the
commit's team artifact bundle: its frozen graph, built incrementally from the
previous bundle, and its per-test coverage map, re-run only for the suites
the push selects (scripts/ci/test-impact-publish.sh). A weekly run rebuilds
both from scratch and fails when the incremental graph's topology differs
from the full one. Bundles are Actions artifacts kept 30 days, never
committed.
Shadow. The report-only test-impact-shadow job also builds the checkout and
runs `test-impact select` against the base side of the PR merge commit
(scripts/ci/test-impact-select.sh). A bundle is used only when it comes from
a successful main-branch producer run for exactly that commit
(scripts/ci/test-impact-bundle.sh). The job stays continue-on-error and
outside ci-ok.
Narrowing switch, PR CI only. While the repository variable
TEST_IMPACT_MODE is `gate`, a select-tests job turns the engine's answer into
the tests the test-unix and test-windows legs run. Unset, nothing changes.
It fails open by construction:
- scripts/test-impact/selection.py is the only place an answer becomes a
runner filter. Only a well-formed selected lane whose filter matches its
own suite list narrows; run-all answers, a lane that selects nothing,
unknown schemas or tokens, and any crash run everything.
- `test` passes a selection only from a successful, narrowed select-tests
run; ci-ok does not need select-tests. A broken selection costs minutes,
never coverage.
- scripts/run-tests-parallel.sh launches only the selected suites and keeps
every guard on the selected list. A selected suite the build does not
register fails the run, unless the engine saw its registration as
conditional; then it is dropped with a notice, exactly as a full run of
that build never runs it. Each runner judges its own tokens, so a selected
test that does not exist fails its suite.
- The shard manifest records the selection's digest, and
verify-shard-union.sh requires every shard of a leg to agree on it.
- Dry runs and releases never pass a selection (lane-wiring contract).
Also fixed on the way:
- publish crashed on any run-all push: a run-all lane carries
"suites": null.
- the git-facts hostile-diff probe re-ran the runner with the parent's
CBM_TEST_ONLY_FILE still set; under a narrowed run naming its suite it
re-ran the spawning test, which spawned again. The child now clears it
(a new test fails without that).
- the select summary crashed on the same null suites.
Contracts: tests/test_test_impact_selection.sh (Step 0z8),
tests/test_harness_selection.sh (Step 2c, the real runner through the real
harness), and new cases in the lane-gate and lane-wiring contracts.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A main commit can now publish its test-impact bundle — the commit's frozen
graph, its per-test coverage map and receipt.json binding both — and a PR
selection given the bundle of its merge base, verified by the caller,
builds its graph incrementally from the bundle's graph and narrows suites
by the admitted map. Without a verified bundle nothing changes: the graph
is built from scratch and every reached suite runs whole.
Producer and consumer:
- `codebase-memory-mcp test-impact publish|select` (in process, no
daemon): publish builds the bundle of the checked-out commit, from the
previous bundle's graph when that bundle verifies; select is the
selection with optional --artifact/--artifact-commit/--artifact-verified.
The MCP surface is unchanged.
- receipt.json binds the exact imported graph (graph digest), the graph's
content and topology, the map's three files and a compatibility digest
(platform label + the bytes of test_impact.coverage.compatibility_paths).
- A bundle is used only when the caller verified its source, it is the
merge base's own, and its imported graph is the one the receipt names;
the map is admitted by the existing receipt policy (source, ancestry,
content, age, compatibility, metadata). Refusals become rejection
reasons in the answer's receipt.
- C_eff: a changed function the map knows contributes its own id; one it
does not know (new, platform-only, a type or macro) contributes its
nearest mapped callers on the static inbound closure; deleted names
contribute every mapped function of that name.
Graph:
- the frozen build accepts a base candidate and repairs it through the
incremental closure route; any doubt declines to the full build. Pinned
to one worker like the full frozen build.
- incremental closure repair skipped the interprocedural complexity pass:
re-parsed functions lost transitive_loop_depth/recursive and callers kept
stale values. One traversal now serves the in-memory pass and a new
store-level recompute that runs before the importance rescore.
- test-impact's frozen builds use a fixed display Git context: the per-run
pin path and the commit ids made two builds of one tree differ and made
every commit a semantic-input change, so repair always declined.
- graph content and topology digests (store_content_digest.c): the exact
graph digest binds one database (ids, db_uid, generation), never equal
across builds; the content digest keys rows by what they say, the
topology digest by what selection reads.
Measured on this repository (frozen publish, macOS arm64): a two-file C
change repairs in 18-20 s instead of 75-80 s with the topology of the full
build; a docs-only change repairs with identical content; selections with
and without the bundle are identical on all seven replayed pairs
(two repaired incrementally, five declined to the full build). Recorded, not
fixed: closure repair still differs from a full build in edge PROPERTIES
(argument quoting and key order, CONFIGURES args) and in the corpus-wide
SIMILAR_TO/SEMANTICALLY_RELATED edges; selection reads neither, so the
topology digest is the incremental-equals-full check and the receipt
carries both.
Tooling: scripts/test-impact/coverage-merge.py builds a push's incremental
coverage map (re-run suites verbatim, the rest carried with ids remapped
onto the new image's function table), with its contract as test.sh Step
0z7.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
One request pins the change (merge base .. HEAD), builds the graph and the
runnable test registry of exactly the HEAD snapshot, seeds the graph from
the change, walks what depends on it and answers which tests to run
(cbm.test_impact). Missing or doubtful evidence never narrows: it becomes a
run-all answer that names why.
Snapshot (classified ledger -> frozen graph -> registry):
- one shared language classifier for discovery and pinned blobs (raw bytes
on every platform); the classified ledger feeds the frozen build, which
indexes exactly the listed files and records partial parses as per-file
PARSE_GAP instead of refusing the graph;
- the pinned tree skips symlinks and submodule links (counted), as
discovery never indexes them;
- the test model scopes uncertainty to the suites of its file (they run
whole), reads line splices where they occur, treats a mid-file
`#ifndef X` + `#define X` as an ordinary condition, and no longer reads a
plain identifier named like a macro as an invocation. On this repository
the registry is now narrowable and matches a text census exactly
(8,365 cases, 141 runner suites).
Seeding and reach (port of the measured replay, rule B):
- hunk -> overlapping definitions, comment-only hunks dropped, pure
deletions -> enclosing definition, no-definition hunks -> whole file,
deleted names found by text, header changes propagated through the
include closure and the headers' macros (pastes included), file-level
escalation for unclassifiable header lines;
- inbound fixpoint walk over CALLS, ASYNC_CALLS, USAGE, CALL_REFERENCE,
READS (+ routes); cases, suites and the runner's entry points are sinks;
a reached runner main runs everything (RUNNER_REACHED);
- a parse gap whose text names a reached definition reaches the
definition around it (or its file);
- macro-registered tests run their suites whole; cases and suites map to
graph nodes by their start line.
The line patterns reproduce the reference regexes exactly on 1,383,850
lines of this repository and 37,167,926 kernel lines; on 24 replayed
changes the engine selects at least what the reference selected (13
identical, the rest only conservative supersets on the same graph).
Also:
- configured test declarations that cannot be mapped degrade their file
(no configured roles, a test_declarations diagnostic) instead of
aborting the whole index;
- the selector keeps other runners' suites out of the selection, and the
result codec judges the model by narrowability like the selector;
- a change to .codebase-memory.json runs everything (built-in rule);
- .codebase-memory.json with the c-cbm preset and this repository's rules.
Gate fixes (suites, lint-ci, Windows cross-compile):
- the request runs on its own thread with a default-size stack (daemon
tool calls run on 256 KiB threads, where the in-process build died);
- a request-supplied config_path is no longer followed by an unlink of an
uninitialized path buffer;
- the inventory's frame-local reader no longer stays reachable from the
caller's source after the filter call;
- FILE_ID_INFO needs _WIN32_WINNT >= 0x0602; the pinned-tree header pins
it before any system header (toolchains defaulting lower failed);
- memory-core ratchet: the win_utf8 converters gain core-backed
`_in(cls, ...)` forms over one implementation (the legacy names keep
the C library for unmigrated callers), the Git facts module uses them,
and the Windows spawn path releases everything through one failure
exit; pipeline, extraction and config allocations that this work
touched moved to the core in matched pairs; baselines tightened;
- the branch's C files formatted with the project's clang-format.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
gcc's libasan turns detect_stack_use_after_return ON by default; clang's
runtime leaves it off. On aarch64, gcc 13/14 free small fake frames
without clearing their in-use flag, so once 8,192 frames of a size class
have been used, every instrumented call scans the flag array (~8 us
instead of ~6 ns) and the check silently stops detecting anything. CI's
ubuntu-24.04-arm gcc leg runs the extraction suite in 688 s against a
900 s cap (16 s on amd64), and the deep extraction tests of #2478 push
it over the cap.
The lanes already document the intent: test-diag in _test.yml and
`make -f Makefile.cbm diag` set detect_stack_use_after_return=1
explicitly, as one of the ASan checks that are off by default in every
other lane.
scripts/test.sh now appends detect_stack_use_after_return=0 to
ASAN_OPTIONS (keeping any options already set, ':'-separated) unless
ASAN_OPTIONS already mentions the flag, and exports it, so test-diag's
explicit =1 wins. Lanes built with clang already ran with it off and
are unchanged. The venue-parity contract (Step 0j) gains a layer that
drives the real scripts/test.sh with a no-op make and a probe runner
and asserts three cases: unset, other options kept with the flag
appended, explicit =1 kept.
Measured in the local Linux arm64 container (Colima, gcc 13.3,
ASan + UBSan), wall time of `scripts/test.sh --suites extraction`:
origin/main 268a9d88 without this change 628.6 s
origin/main 268a9d88 with this change 5.1 s
#2478 head 818ef0ec with this change 9.7 s
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Preserve the current selector, frozen-input inventory, result codec,
coverage collection, runner filtering, and isolated Windows venue work.
Component-level regression proofs are recorded in the PR handover.
The live classified provider, public execution integration, artifact
admission, fresh oracle/shadow evidence, and full platform gates remain
incomplete. This checkpoint does not enable or claim verified narrowing.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
scripts/setup.sh and scripts/setup-windows.ps1 carried their own copy of
the release download: resolve the latest tag, fetch the archive, unpack it
into the install directory. Two implementations of the same step drifted
apart: the setup scripts did not follow the installers as those gained
their checks, their architecture selection and their staging directory.
Both setup scripts now fetch the installer from the origin and branch they
are themselves served from and hand the download and install over to it,
so there is exactly one implementation and the setup scripts behave like
the installers: the same conditions stop them with a non-zero exit before
anything is installed, setup-windows.ps1 selects the archive through the
installer's architecture detection instead of always amd64 (CBM_ARCH pins
it) and stages in a freshly created directory, and the file is pure ASCII
again. CBM_DOWNLOAD_URL, the
installers' download-base override, also moves the installer fetch so both
scripts can be driven against a local fixture.
Everything the setup scripts do beyond installing the binary is unchanged:
the --from-source / -FromSource builds, the interactive Claude Code
configuration, the PATH hint, the git-hook activation and the uninstall
notes.
tests/test_setup_scripts_contract.sh runs setup.sh against a loopback
fixture release built from a stub binary, in file mode and in curl | bash
mode, covers the cases in which the installer refuses, and pins the static
contract for setup-windows.ps1. It is Step 0z of scripts/test.sh.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The client-detection fix moved the Windows Hermes home to
%LOCALAPPDATA%\hermes and stopped counting a bare ~/.claude directory as a
Claude Code install. The smoke script followed only part of the way.
Hermes: 8x and 8x-i read the new HERMES_HOME_DIR, but 8x-ii still looked
for the pre_llm_call hook in ~/.hermes/config.yaml, and the uninstall
absence checks 9m and 9n still named ~/.hermes. On Linux and macOS
HERMES_HOME_DIR is ~/.hermes, so the stale paths pointed at the right
directory and nothing showed. Only on Windows do the two differ: 8x-ii
failed on a hook that install had written to the new home, and 9m and 9n
passed without looking at any file install had produced. All three now use
HERMES_HOME_DIR. The assertions are unchanged.
Claude Code: the fixtures of 9b-2, 9b-3, 9b-8 and both Phase 13 legs
created only an empty ~/.claude, which is no longer a Claude Code home.
Wherever no claude executable is installed, 9b-2 then finds no .claude.json
entry and 13e finds no agent config, and 9b-3, 9b-8 and the install.ps1 leg
stop exercising the Claude paths without saying so. The Linux test
container reproduces both failures. The Windows job never got that far,
because 8x-ii stops it in Phase 8. The fixtures now seed the .claude.json
user config through one helper, the same seed the unit tests use.
9b-1b pins the new rule end to end: where the empty-home control shows no
Claude Code, a home with only a bare ~/.claude gets no Claude config. A
venue that has a claude executable skips it, with the reason at the case.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The test-impact selector has to hand the C runner individual tests, and the
map behind it has to know what each test executes. The runner could do
neither: argv selects whole suites.
Selection. CBM_TEST_ONLY (comma-separated) and CBM_TEST_ONLY_FILE (one per
line; blank lines and # comments allowed) take suite:test tokens, and are
unioned when both are set. A suite no token names never executes its body.
Inside a named suite only the named tests run; the rest are counted as
deselected and the summary reports the count. Argv still selects suites and
the tokens narrow inside that selection, so the parallel harness hands every
per-suite process the same list with no change of its own.
Tokens are strict: one that names a suite this build does not register, or a
test its suite never ran, is printed and fails the run. An unreadable file, a
malformed token, and a selection that names no test stop the run before any
suite. Without a selection the output is byte-identical to before, and
--list-suites ignores the selection.
Coverage. TEST_COVERAGE=1 compiles clang source coverage into our own code
(the one-shot sources, the LSP unity object, the preprocessor; vendored
objects stay uninstrumented) and the runner's hooks into the test runner.
`make test-runner-cov` and `make cbm-cov` build the instrumented runner and
product binary into build/cov. With CBM_TEST_COVERAGE_DIR set the runner
writes <suite>/<test>.parent.profraw for itself around each test, points
LLVM_PROFILE_FILE at <suite>/<test>.%p.profraw for the children the test
spawns, and sends what runs between tests to _setup.<n>.profraw. No other
build compiles any of it.
tests/test_harness_test_only.sh pins the selection contract against the built
runner and runs as Step 2b of scripts/test.sh, before the suites.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The vendored tree-sitter-c-sharp (88366631d598) predated upstream's C# 12
collection-expression support (tree-sitter-c-sharp #402, first released in
v0.23.4). A `[...]` literal could only be parsed as an
element_binding_expression, so `c ? [] : x`, wrapped `? []` ternaries and
both-branch collection conditionals had no valid parse. Error recovery then
ate the surrounding statement, the file was reported as parse_partial, and
nearby declarations could fall out of the graph.
Refresh the generated grammar bundle byte-for-byte from upstream tag v0.23.5
(cac6d5fb595f5811a076336682d5d595ac1c9e85). The ABI stays at 15, the MIT
LICENSE is byte-identical, and scanner.c is content-identical. The only new
node types are collection_expression, collection_element,
expression_element and spread_element. No node type is removed or renamed,
and the C# extractor and LSP depend on none of the old or new names.
MANIFEST.md and the vendored checksums are updated.
Add parse_coverage regressions for the reported variants plus a `? null`
control. A second test checks that calls inside a collection expression
(element and spread) are still extracted. A malformed-source guard checks
that broken C# is still flagged.
On dotnet/runtime (60108ba6), 4 files are no longer parse_partial and no
file becomes newly partial. Utf8JsonWriterTests.cs gets its class back, and
216 orphaned Functions are re-attached as Methods. Two files that were
already partial for unrelated unsupported syntax (the preview `closed`
modifier, `async` as an identifier) recover differently and lose 46 and
10 definitions.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Overloads that share a base QN (Java/C#/C++/Kotlin/Swift/Scala/ObjC) merge
into one graph node today, and that node collects the CALLS of every
overload. The agreed cure is one signature-qualified callable identity for
all overloading languages:
callable_qn := base_qn [<tparams>] (params) [cvref]
This change lands the plumbing with every language still at
CBM_CALLABLE_ID_NONE, so the graph is byte-identical; each language is
switched on later in its own change (with its index-format bump).
- internal/cbm/callable_sig.{c,h}: the per-language mode side table
(NONE|TYPED|LABELED_TYPED|LABELED|ARITY; kept out of CBMLangSpec like
cbm_string_dispatch_suffixes, since ~160 positional rows would otherwise
all need the member under -Wmissing-field-initializers), the shared
builder cbm_callable_sig[_mode] for the def and the call-scope QN, and
the inverse cbm_qn_callable_base_len[_named]. Suffix contract: no '.',
'::' or '->'; whitespace-normalized; comments/annotations/attributes
dropped; qualified types keep their last segment; '->' spelled '=>';
variadic '[]' (Java) / '~'; C++ top-level cv dropped, arrays decay,
cvref kept; <= 200 bytes, longer ones keep whole leading entries and end
in '#<fnv1a64 of the full suffix>)'. The builder is iterative (cursor
walk, bounded declarator loop) and self-checks against the inverse.
- The inverse is anchored to the node's bare name wherever the name is
known: real QNs already end in ')' without being callables (a Java field
whose name is its declarator `X = f()`, make `$(VAR)` targets), and only
the anchor makes the inverse exact on them.
- Leaf splitters strip a suffix first (no-op today): registry by-name key
(indexed by the bare name for a suffixed callable), cbm_lsp_bare_segment,
cbm_pipeline_qn_class_method_tail, pxc_last_component, pxc_qn_leaf,
go_index_visit_node, collect_prod_defs, sg_qn_prefix_len.
- New unused fields: CBMDefinition.qn_sig_off, CBMResolvedCall.callee_sig
(relocated by result compaction), CBMRegisteredFunc.sig, CBMLSPDef.sig.
The lsp_surface codec is unchanged (no format bump in this change).
- get_code_snippet and trace_path gain base-QN tiers
(cbm_store_find_nodes_by_qn_base) that find nothing until suffixes exist.
- store: cbm_store_find_nodes_by_qn_suffix built its LIKE pattern in a
512-byte buffer (a long suffix was truncated into a pattern that matched
nothing) and did not escape '_'/'%' (`my_func` also matched `myXfunc`).
The pattern is now heap-built, escaped, and matched with ESCAPE '\'.
Proof: full sorted node+edge+lsp_surface dumps are byte-identical before
and after on 7 corpora (this repo, elasticsearch libs, Exposed, dotnet
System.Linq, System.Collections, coreclr/gc, a C corpus).
Refs #2061
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Git exports repository-local variables (GIT_DIR, GIT_INDEX_FILE,
GIT_WORK_TREE, ...) into hooks, and they take precedence over
`git -C <dir>`. Every fixture git command in the test runner therefore
acted on the caller's real repository whenever the suite ran with those
variables set: with GIT_DIR alone, fixture assertions failed and commits,
branches and linked worktrees were written into the caller's clone; with
the full hook set, fixture commits failed and git-dependent tests degraded
to "git not available" SKIPs. The pre-commit hook already clears the
variables (7883507d), but any other caller (a shell with GIT_DIR set, an
editor hook, `GIT_DIR=... scripts/test.sh`) still hit it.
The test runner and the repro runner now clear git's local-env list
(`git rev-parse --local-env-vars`) once at startup, before any fixture or
re-exec'd child spawns git, so every fixture scopes to its own repository
no matter who invokes the suite.
tests/test_git_env_isolation_contract.sh (Step 3a of scripts/test.sh)
drives the real runner's git_context suite with GIT_DIR, and with the full
hook set, aimed at a throwaway decoy repository. It requires the same
pass/skip summary as a clean run and a decoy left untouched (no refs, no
linked worktrees).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Under R1, a T0b push (test-infrastructure/, lint config, graph-ui, memwaste
tooling) selected lint and no test leg, so the static contract steps that
police exactly those files -- the VM scripts, the compose and run.sh wiring,
the venue-parity rules -- ran nowhere. Replayed over September, 13 of the 18
T0b pushes ran no contract step at all.
The R1 table now gives T0b lint + contracts, so those PRs get the
contracts job (scripts/test.sh --contracts-only) as T0a PRs do. The
fixture table's six T0b cases expect it, and the history replay asserts
that every push runs the contract steps (the contracts lane or any test
leg). Comments that described the job as docs-only are updated.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The pulls/files API lists at most 3000 files and says nothing when it stops,
and the lane decision is only as complete as that list: a cut list can drop
exactly the file that needed a lane.
select-lanes.sh takes --expect-files N (the PR's changed_files; batch rows
take "expect_files") and decides full with a file_list_truncated reason when
the list has fewer distinct paths than N (a complete list never does: renames
only add their old path), when N is at or over the 3000-file cap, or when the
list itself reaches the cap. The tier still classifies what was listed. An
empty or non-numeric count is a usage error, so a workflow that loses the
count fails loudly instead of skipping the check.
pr.yml passes github.event.pull_request.changed_files. The selector contract
covers short, complete, rename-inflated, at-cap and just-under-cap lists and
the github-output and batch channels; the wiring contract requires the count
to be passed.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A new pr.yml job, test-impact-shadow, predicts from the code graph which C
test suites a change can affect and uploads the prediction, so graph-based
selection can be judged against what actually failed before anything gates
on it. It is continue-on-error and not in ci-ok's needs: it can never block
or pass a merge.
scripts/ci/test-impact-shadow.sh --fetch-release downloads the latest
published release archive for the host plus checksums.txt with the job's
read-only token and verifies the SHA-256 (seconds instead of a build, and a
PR that breaks the indexer cannot corrupt its own prediction); pr.yml builds
the checkout only if that fails. --binary/--out then indexes the checkout
into a throwaway HOME/cache with a private CBM_RUNTIME_DIR (short, under
/tmp, so the daemon socket fits sun_path), runs detect_changes against the
merge commit's base at depth 15 and hands it to predict.py. The graph is
only consulted when C sources changed.
scripts/test-impact/predict.py ports the replay's mapping: impacted test
functions -> registered suites (perf excluded; nested RUN_SUITE children
resolve to the registered suite that runs them), changed test files ->
whole suite, fixtures -> suites that name them, grammars -> grammar suites,
and run-all on the selector's full, a test header, test infrastructure, or
a saturated/truncated impact set. It writes prediction.json and a short
step-summary table.
tests/test_test_impact_predict.sh (Step 0z5) pins the mapping on a
synthetic tree; the wiring contract requires the job to stay
continue-on-error; the script joins the venue-parity interface probes.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
pr.yml's `changes` job now lists the changed files (renames with their old
path too), runs scripts/ci/select-lanes.sh in a credential-less step and
outputs tier, full, lanes and the pr-smoke matrix. security, lint and test
pass the lanes to their reusable workflows, which take a `lanes` input
defaulting to "all": dry-run.yml and release.yml pass none and run exactly
as before (setup-matrix output for every broad/shard combination is
byte-identical; the TSan matrix moved into setup-matrix with the same legs).
Under a selection, setup-matrix keeps only the tagged legs of the selected
lanes (an untagged leg is an error, never a silent drop) and each lane job
has an `if:` on its lane. pr-smoke and memwaste run on their lanes instead of
the old product regex. Docs-only PRs (T0a) get a `contracts` job running the
new `scripts/test.sh --contracts-only` (Step 0 contracts, no build). ci-ok
needs contracts, lists the run's jobs (actions: read) and passes LANES/JOBS
to require-all-green.sh; shard-completeness passes --lanes.
tests/test_lane_wiring_contract.sh (Step 0z4) pins it: dry run and release
pass no lanes and the inputs default to all, pr.yml wires the selection
through, and every lane is served by a tagged leg or an `if:` naming it.
The smoke-fixture contract now checks the selector's decisions for the
smoke-harness paths instead of the removed regex; select-lanes.sh joins the
venue-parity interface probes.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Under lane selection a PR runs a subset of its jobs, and "skipped" stops
meaning "not needed": a selected lane that was skipped, cancelled or never
created is a silent gate loss that the needs-level ci-ok waves through.
require-all-green.sh keeps the needs-level check and, when given the
selector's LANES and the run's JOBS (Actions API job list), requires every
selected lane to have at least one job and all of them to have succeeded.
Unselected lanes may be skipped or absent but may not fail; memwaste stays
report-only. A selection it cannot trust (not a JSON list, unknown lanes, or
neither lint nor contracts) is red. Matrix jobs are mapped by the lane value
in their name, single jobs by their job id.
verify-shard-union.sh takes an optional --lanes: no manifests at all is
green only when the selection holds no test leg; without --lanes (dry run,
release) or with "all" it fails exactly as before.
tests/test_lane_gate_contract.sh drives both with synthetic inputs
(selected-skipped/cancelled/missing -> red, unselected-skipped -> green,
no legs -> green only for an empty leg selection); Step 0z3.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
scripts/ci/select-lanes.sh maps a PR's changed-file list to a tier, a lane
set, a full flag and the reasons: each path gets a category, the categories
give a tier (T0a docs .. T5 core), the tier gives the R1 lanes, and a few
categories add lanes of their own (pkg -> wrappers + 3-OS smoke, graph-ui ->
Windows guards, memory core -> memwaste, ...). It fails safe: tier T5, an
unmatched path, an empty list, the selector itself, any workflow, the test
harness, Makefile.cbm and vendored code select every lane. The output is a
pure function of the file set.
tests/test_select_lanes.sh pins the decision table (43 cases), determinism
(order, duplicates, CRLF), the input channels and the lane universe.
scripts/test-impact/replay-selector.sh replays all 767 September 2026 PR
pushes and the 22 real failures from a compact fixture: every push gets its
expected tier (729 equal to the analysis, 38 documented deviations where the
analysis split Makefile.cbm diffs by content and the selector, seeing names
only, is conservatively full), and every real failure's lane is selected.
Both run as Step 0z/0z2 of scripts/test.sh.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The fork notes for javascript + tsx changed MANIFEST.md, which is covered by
scripts/vendored-checksums.txt; its entry still held the pre-change hash, so
Layer 8 (security-vendored.sh) failed. Regenerated with --update; the only
line that changed is MANIFEST.md.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A plain '&' inside JSX -- a URL query in an attribute
(href="https://fonts.googleapis.com/css2?family=Inter&display=swap"),
or text such as <p>Tom & Jerry</p> -- produced a tree-sitter ERROR node
in .js, .jsx and .tsx files. The file was flagged parse_partial in the
coverage view, and the error recovery turned words from the URL
(create-next-app -> create/next/app) into identifiers that resolved to
unrelated symbols as false USAGE edges.
Root cause is upstream tree-sitter-javascript, which the tsx dialect
builds on: unescaped_{double,single}_jsx_string_fragment is
/([^"&]|&[^#A-Za-z])+/, so '&' is only accepted before a space/digit
or as a complete html_character_reference (& / &). jsx_text is
scanned by the external scanner, which stops at every '&', and nothing
in _jsx_child accepted a '&' that is not a reference. So '&' + letter in
a string, and any lone '&' in JSX text, had no valid token.
Fix: javascript and tsx become self-maintained forks of the pins we
already vendored (javascript @ 58404d8cf191; typescript tsx @
75b3874edb2d on its javascript base @ 3a837b6f3658), following the
arkts model. One patch (tools/tree-sitter-javascript/patches/
jsx-lone-ampersand.patch) is applied to both javascript bases:
- a one-character '&' token at lexical precedence -1, aliased to the
existing string_fragment (in strings) and jsx_text (in text); the
lower precedence keeps & / & lexing as html_character_reference
- the string-fragment regexes exclude a '&' directly before the closing
quote, so "a&" no longer runs past the end of the string
Node types are unchanged (same type/named set). The external scanners
stay upstream bytes. Parsers regenerated with tree-sitter-cli 0.25.10
(ABI 15; tsx moves from ABI 14, and its tree_sitter/ headers now match
arkts). Without the patch both fork dirs regenerate upstream exactly.
The upstream corpus passes unchanged: javascript 116/116; tsx gives the
same results as unpatched upstream.
Proof on vercel/next.js examples/ (canary 725705bd, 1,722 js/jsx/tsx
files): parse_partial 92 -> 47 (tsx 33 -> 0, jsx 2 -> 0, js 12 -> 2; the
two left are Babel do-expressions and Flow types), no newly flagged
file, code-graph nodes 14,739 -> 14,739, edges 24,773 -> 24,657: the
115 USAGE edges removed were all false cross-example edges from URL
fragments, plus a 4 SIMILAR_TO / +3 SEMANTICALLY_RELATED reshuffle
from the corrected ASTs.
Registered like arkts: MANIFEST.md fork rows + notes (ABI tally
recounted), audit FORKS map, THIRD_PARTY.md, vendored checksums.
LICENSEs stay upstream's MIT byte-identical.
Tests: parse_coverage gains a .js/.jsx/.tsx matrix (double/single-quoted
attribute, '&' before the quote, '&' + letter/space/digit in text, plus
&/& controls) and a guard that broken JSX is still flagged.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The parallel-scheduler contract failed twice on test-windows CLANG64 while
running fake suites only:
- PR #2394 (run 36364479687): "Windows timeout race refused without a
surviving descendant to refuse over". The refusal came 38s after the
forced leader exit (a cold powershell/CIM descendant probe, once in the
wave loop and again in the cleanup pass), while the fixture's descendant
was a `time.sleep(30)` that had already exited on its own.
- PR #2345 (run 36199749518): "suite 'hang_after_summary' taskkill could
not prove process-tree cleanup". The scheduler bounded taskkill.exe with
--kill-grace, which the contract passes as 1s, so a slow cold start of
the tool was reported as a failed cleanup.
Both verdicts were a race between a clock and runner latency. An audit of
the fixture found more of the same shape:
- Every hanging fixture ended on its own after 30s. A starved scheduler
could therefore see a "hung" leader exit 0, and a descendant the
scheduler failed to kill could vanish before the leak check looked.
- The 1s --timeout could kill a slow leader before it had printed its
summary (asserted pass=1), spawned its descendant or written the
descendant pid. That pid was then read back with int() from a file
written create-then-write, so a reader could see no file or an empty one.
The empty-file read recorded earlier was the scheduler's ready file,
which 3b9052c9 already publishes atomically; the fixture's own writer
had the same shape.
- The descendant installed its SIGTERM-ignore during its own start-up. A
SIGTERM that won that race skipped the SIGKILL escalation the POSIX leg
exists to cover, silently and with the verdict unchanged.
- On Windows, the stubborn-tree leak check was a single look right after
taskkill, but TerminateProcess returns before its target has exited.
Fix it by construction, not by budget:
- A gate. Every process that must outlive the scheduler's decision blocks
reading stdin. That is a pipe whose only write end the contract holds,
handed down through the scheduler, and the read returns only at EOF. The
contract closes the pipe once the verdict is recorded. If the contract
dies, EOF releases everything, so nothing is orphaned and no fixture
ends on a clock.
- The three hanging scenarios run under the scheduler's existing
pre-terminate barrier. They are released only once the fixture has
published `<suite>.established`, which happens after the summary is
flushed, the descendant has confirmed it is armed, and its pid is known.
The marker is written to a temp file and moved into place with
os.replace, so a reader sees either no file or the whole file.
- The Windows leak check waits on the descendant's process handle. That
wait is exact, not a race: the descendant is held by the gate, so the
scheduler's kill is the only way it can end.
- run-test-wave.py runs taskkill through windows_taskkill_tree() with its
own stable-state budget (WINDOWS_TASKKILL_SECONDS = 15), as 132e8fc3 did
for the descendant probe. --kill-grace keeps bounding what its name
says. Real runs already pass --kill-grace 15, so they are unchanged.
The contract pins this structurally, the same way it pins the probe:
exactly one taskkill site, no kill-grace there, and a helper bounded by
the constant that fails closed on a timeout, a start failure or a
non-zero exit.
Every assertion keeps its contract:
- A leader that hangs after a green summary is recorded as rc=124 pass=1.
- A zero-test child is recorded as rc=97.
- The wave continues after a bounded failure.
- A SIGTERM-resistant tree is killed in full.
- A leader that exits between the timeout decision and termination is
still cleaned up on POSIX. On Windows the scheduler refuses (rc=2,
naming the cleanup failure) over a descendant that is really alive.
A new self-check fails loudly if a held leader or descendant is not live
while the barrier holds, that is, if the fixture no longer exercises what
those assertions claim.
Proof, on macOS arm64 and a Linux arm64 container. The Windows branch was
reasoned through; the VM is down.
- Torture hook (proof only) that delays each hanging leader 2s, past the
1s timeout. The old fixture is RED 3/3 on every scenario (pass=0,
FileNotFoundError on descendant.pid, "lost its bounded result"). The new
one is green, including at 8s, and 10/10 under the hook plus CPU load.
- The #2394 mechanism: after an injected 31s refusal latency the old
descendant is gone and the new one is live. It exits 0.02s after the
gate closes.
- The taskkill pin is RED on the old scheduler and green on the new one.
- Stress, 20 runs each under CPU load. The old full contract passed 20/20
on macOS and 20/20 on Linux, so these hosts do not reproduce the CI
timing. The rebuilt held scenarios passed 20/20 on macOS and 20/20 on
Linux. The full contract after the change passes a single run on macOS
and on Linux, and 20/20 on each leg under CPU load (4 load workers;
the Linux container is capped at 4 CPUs).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Martin Vogel