README.md, the site files and pkg/npm/README.md are read by test_cli, so
a change to one now runs the test legs (T1-tests). The September replay
held 15 README pushes to the docs tier the analysis assigned and failed
every test leg's contract step. Those rows are now a documented
deviation, read-by-tests, which like makefile-by-name may only raise a
tier: 767 pushes as expected, 53 documented fail-safe deviations, all 22
real failures still caught.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The first commit of this branch is the Smart CI change as of the rebase
onto 9f85f104. A review of it found places where the engine could run
fewer tests than a change needs; each now runs everything or selects the
right tests, with a test that fails when its fix is reverted.
Engine:
- A `#define NAME "literal"` line counted as naming a role but selected
nobody, so the tests that start the blocking-git probe through their
marker macros would not have run. The scan now follows NAME, as a whole
word, to the code that uses it, through chains of up to 4 macros; a
define it cannot follow runs everything.
- main dispatches through the role table, so no edge leads from main to a
handler. A table entry the engine cannot read, or whose handler is not
exactly one function, was skipped; it now runs everything.
- In a runner file, only a test case names a role's literal. The runner's
own helper that reads the marker no longer counts as a spawner, so a
role only the runner names runs everything again.
- A handler reached first by a parse gap or by the spawn lift was never
lifted. Role lifts and parse gaps now run to a fixpoint, before the
runner check and again after the spawn lift.
- A reached sink that main calls (the blocking-git probe) counted as a way
into main and hid the real one: #2644 now names the start-up path main
-> cbm_index_supervisor_capture_build_fingerprint.
- No pointer past the end of a role line; the RUNNER_REACHED note clamps
its prefix.
CI:
- pkg/npm/README.md, which test_cli reads, runs the test legs instead of
the docs tier.
- The build rule lists the harness scripts the test legs run:
run-test-wave.py, path-safety.sh, clean.sh.
- The revert guard rejects the ancestor's coverage when a diff fails.
- The vetoed selection is uploaded (test-selection-vetoed).
Docs: SMART_CI.md states the new fail-safe cases, the version that lacks
the new config keys, the selection base, and that the narrowed leg builds
with gcc while the coverage comes from clang.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The v0.12.1 release smoke failed on every leg in Phase 13w, the
package-wrapper install gate added in #2664:
- macOS: the runner images have no Go on PATH, and
SMOKE_REQUIRE_WRAPPERS=1 rightly fails without one. pr-smoke and the
three release smoke jobs now set up Go from pkg/go/go.mod, with the
same pinned step the wrapper test job uses.
- Windows: `mkdir -m 700` changes the mode in a second step, which MSYS2
cannot do under the smoke profile's owner-only DACL. mktemp -d sets
the mode at creation.
- Linux: the runner's older Go downloaded the 1.26.1 toolchain into the
smoke profile's HOME. Go writes its module cache read-only, so the
harness could not delete its profile afterwards. All Go state now stays
in the wrapper's own work tree, and its cleanup restores write
permission before removing it.
npm and PyPI passed on every leg that reached them; the gate itself is
unchanged.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
v0.12.0 could not be installed through npm, pnpm, pip or the Go wrapper on
any platform (#2657): each wrapper refused an archive member above 256 MiB,
and every v0.12.0 binary is 331-335 MB. The v0.11.0 binaries were already
about 300 MB, so the Go wrapper, which has had the cap since v0.10.0, was
broken before; npm and PyPI gained it in v0.12.0. The archive is verified
against the release's checksums.txt before it is unpacked, so a byte cap
only ever refused what we ship.
- npm, PyPI, Go: delete every byte-size cap (compressed archive, per member,
aggregate, checksum manifest, download size). The checksum, the exact
member set, path and link refusal, the member count, HTTPS and the
--version check stay.
- All three honour CBM_DOWNLOAD_URL like install.sh, for testing: https, or
plain http to an exact loopback authority with no proxy and no redirect.
- Smoke: scripts/smoke-wrappers.sh installs the three wrappers from the
checkout against the release fixture and runs the binary each one
installed. smoke-test.sh Phase 13w runs it; SMOKE_REQUIRE_WRAPPERS=1 makes
it required on the PR smoke lanes and in the release smoke, which gates
publishing.
Other install paths found broken while checking every channel:
- Homebrew: the formula was still at 0.10.3 and installed the dynamic Linux
build, which needs glibc 2.38 (Ubuntu 22.04, Debian 12 and RHEL 9 lack it).
It now installs 0.12.0 with the static -portable Linux assets and tracks
the newest release in the version contract like the other surfaces.
- Go: `go install .../pkg/go/cmd/codebase-memory-mcp@vX.Y.Z` never worked
(no pkg/go/v* tags, so Go fetched the whole repository, over its 500 MB
module limit), and @latest built main's wrapper, whose version constant
lags each release. The release now also tags pkg/go/vX.Y.Z, and the
wrapper installs its own module version (the constant only for local and
untagged builds).
- npm: a stable release also moves the `next` dist-tag to itself; `next`
still pointed to 0.9.1-rc.1.
- Docs: winget and Chocolatey were listed as install channels but are not
published; they are removed until they are. Windows arm64 ships and is
now listed.
Fixes#2657.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The license gate installed `scancode-toolkit` unpinned. pluggy 1.7.0
(uploaded to PyPI 2026-10-15 09:50 UTC) no longer finds the hookspecs that
plugincode 32.0.0 declares on a class (`@pre_scan_spec class
PreScanPlugin`), so scancode crashes at import with "ValueError: did not
find any 'pre_scan' hooks" and `security / license-gate` fails on every
run, whatever the tree. The last green gate started at 09:50:14 UTC;
every run since is red. `ci-ok` and the release `verify` job both need the
security workflow, so this blocks every PR and the release.
Install exactly scancode-toolkit 32.5.0 (the version every green run
used) with pluggy constrained to 1.6.0 through PIP_CONSTRAINT, which the
pip that pipx runs reads. The local hint in license-gate.sh names the
same command.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The v0.12.0 release smoke blocked every binary in its string audit:
"BLOCKED: Dangerous commands found in binary: wget". The line is the bare
string "wget" that src/pipeline/pass_shell.c compares a script's program
name with, and records as the `via` of a curl / wget HTTP_CALLS edge
(3b93db06). The binary never runs it.
Allow that whole line only, like the rst grammar's bare "telnet". A line
that carries a wget command (flags, a target, a URL) stays blocked; the
contract test now pins both cases.
Proof on the release run's own artifacts (run 38644765106, main 8df0639c):
linux-amd64-portable and darwin-arm64 binaries BLOCKED before, all three
audits OK after. test_security_strings_allowlist.sh: 49 passed; with the
allowlist line reverted, the new bare-token case fails.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
On the kernel index (8.5 M nodes, 1.9 M CALLS edges) `MATCH (a)-[:CALLS]->(b)
RETURN count(*)` peaked at 18-25 GB RSS (killed in a 12 GB container): the
store materialised every start node with all its strings (4.9 GB), every one
was deep-copied into a 1.6 KB binding, and every expanded row was materialised
again before projection.
cypher: execute_single now runs the start nodes through the pipeline in chunks
of 1024 (pattern-0 hops, late WHERE, WITH, RETURN). Each hop emits a binding's
expansions in input order and every consumer (RETURN rows / star / default
projection / aggregation, WITH aggregation / projection) keeps its state
between chunks, so the concatenated output is the single-pass output: same
rows, row order, first-seen group order, ORDER BY ties, DISTINCT, LIMIT and
max_rows truncation, OPTIONAL fallback rows, warnings. Queries with additional
MATCH patterns keep one pass (their join strategy depends on the first binding
of the whole set). Start nodes and fetched neighbours move their strings into
the binding instead of being deep-copied and freed.
store: cbm_store_node_scan_open/_next/_close hand out the rows of
cbm_store_find_nodes / _by_label a page at a time. Pages are keyset queries
along the index the eager query walks (idx_nodes_file: NULL file_path rows by
id, then the rest of the last file by id, then the files after it;
idx_nodes_label: by id), each reset before it returns, so no read lock is held
between pages. open verifies the eager plan with EXPLAIN QUERY PLAN and falls
back to the eager rows for any other plan.
Kernel, same index, medians of 3 (macOS arm64):
count(*) peak RSS 24.9 GB -> 256 MB, 23.0 s -> 13.0 s
top-callee peak RSS 12.4 GB -> 1.26 GB, 11.7 s -> 10.9 s
Responses byte-identical to the baseline on django, redis and the kernel
(19 Cypher + 24 tool cases each); the streamed scan equals the eager scan
field by field, in order, for every project and label of all three indexes.
A scan that fails part-way fails the whole query ("query aborted: reading the graph failed") instead of answering from the pages read so far:
The streamed start-node scan answered from the rows of the pages it had read
when a later page failed (a SQLite step error or a page re-prepare failure):
a partial result that looked complete. A failed page, a scan that cannot
open, or a failed eager fallback now marks the start source failed;
execute_chunk returns an error and cbm_cypher_execute reports
"query aborted: reading the graph failed (<store error>)" with no rows. The
store error is logged as cypher.scan_failed.
The test seam cbm_store_node_scan_test_fail_page_after (CBM_ENABLE_TEST_SEAMS)
fails one page after a given number of successful pages; the new test fails
the second page of a label scan (rows and count(*)) and a later page of the
all-nodes scan, and asserts an error result with no rows.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
lint-memory-core failed on #2642: the new search_code scanner, result index, root containment cache and Cypher sort/group/distinct scratch used malloc/calloc/realloc/free directly (mcp.c +22, cypher.c +26, git_context.c +1). They now go through cbm_alloc/cbm_calloc/cbm_realloc/cbm_free/cbm_mem_strdup: file buffers as CBM_MEM_CLASS_EXTRACT, the rest as CBM_MEM_CLASS_OTHER, each block freed with the class it was allocated with. The result index reserves its slot before a result's strings exist, so its insert cannot fail and needs no cleanup; cbm_git_branch_resolve writes into caller buffers instead of returning heap strings. cypher.c ends below its baseline (128 < 132), which is tightened. Output unchanged: 42/42 tool and Cypher responses identical on django; mcp, cypher and git_context suites green under ASan.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Ansible's composition produced no structure: kubespray's 40 playbooks,
roles and task files, tied together by 217 import_playbook /
include_tasks / import_tasks / include_role / import_role / roles:
directives, were unconnected YAML files.
A playbook or task file -- a YAML file whose top level is a sequence;
manifests, values, CI and compose files are mappings -- now IMPORTS
{local_name: <directive>, via: ansible, line} what each directive names:
import_playbook a playbook from its directory; include_tasks /
import_tasks / include a task file from its directory, else from its
role's tasks/; roles: items and include_role / import_role a role's
Folder, roles/<name> at or above its directory (a role by "./" / "../"
path from the directory). The ansible.builtin. prefix is understood; a
templated target links nothing.
Measured on 17 infrastructure repositories: 211 of 211 composition pairs
of kubespray against an independent reading, nothing else in any
corpus; deterministic over three runs; the incremental re-index equals
a full index in 4 scenarios (an included file edited or deleted, an
include pointed elsewhere, a playbook edited).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The soak's query-leak leg failed on GitHub's windows-11-arm runner in the
v0.12.0 release run (38460145605) and in the 10-12 dry run (38204190338),
both times on one search_code call. On that runner image the
PowerShell-backed search_code answers in ~23 s per call (mean 23.3 s over
23 calls). One call crossed the harness's fixed 30 s read, counted as a
failed operation, and failed the leg, even though the harness's own latency
check (Check 4) allows 60 s.
It is the runner environment, not the product: the same leg on a local
Windows ARM64 VM gives search_code 1.10 s mean for v0.11.0 and 1.06 s for the
v0.12.0 candidate artifact, both PASSED. Linux runs it in ~50 ms, macOS in
~150-230 ms, Windows x64 in ~0.9 s.
One constant, SOAK_MAX_QUERY_LATENCY_S (60), now drives the latency check.
The per-call read waits 30 s past it, so the measured latency, not a
timeout, gives the verdict; a reply that never comes still ends the read
and fails the call.
Checked: bash -n; tests/test_soak_daemon_recovery_contract.sh and
tests/test_smoke_fixture_contract.sh pass; the patched harness on the
Windows ARM64 VM (query-leak, 2 min, candidate binary) PASSED.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The variables a container sets (K8s env, compose environment, Dockerfile ENV)
were container-scoped EnvVars of their own, joined to nothing: the readers of
a name point at the global EnvVar, which every program reading that name
shares, so a join through it was in the right program for 23 % of the pairs.
Each scoped EnvVar now CONFIGURES {strategy:env_read, dockerfile} the readers
of its name inside the files the container's image is built from:
- the image: a K8s container's literal image, built from exactly one
Dockerfile (Image -DEPENDS_ON {via:build}->); a compose service's own build,
else its image's; a Dockerfile's own ENV. A K8s image reference now names
its container (USAGE {ref:image, container}), so a two-container pod joins
each container to its own image.
- its files: the context sources COPY / ADD put into the final stage, the
stages it is built FROM and the stages it copies from, resolved in the
build context (the Dockerfile's directory or the repository root, the one
the sources exist under), less the context's .dockerignore, never a test
file. They are kept on the Dockerfile's Module (image_context, image_files,
image_ignore).
pass_docker now reads COPY / ADD from the instruction's own text, as Docker
does: the grammar lost a JSON form opening with `/` and, after a word with a
trailing blank, took the following instructions for more sources.
Incremental: every run recomputes every join edge. A closure repair reads the
inputs of the files it leaves alone from the store and replaces the stored
edges wholesale; a .dockerignore (now a semantic-manifest control) changing,
or a deletion of a file an image copies, rebuilds in full.
CBM_SEMANTIC_INDEX_VERSION 8 -> 9.
Measured on 17 infrastructure repositories: 80 pairs, 80 correct by hand;
the graph matches an independent reading of the Dockerfiles 80/80.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The v0.12.0 release run's license gate blocked @img/sharp-wasm32@0.35.5
(Apache-2.0 AND LGPL-3.0-or-later AND MIT). sharp sat in graph-ui's tree
only because @huggingface/transformers lists it for its Node image path.
CBM Lens runs Transformers in the browser (a tokenizer and a causal LM on
WebGPU), never processes images, and the Transformers web build already
replaces sharp with an empty object. Yet sharp installed 14 LGPL packages
(10 @img/sharp-libvips-*, 3 @img/sharp-win32-*, @img/sharp-wasm32), and
the gate let them through two ways:
- a reviewed exception for exactly @img/sharp-wasm32@0.35.0 ("installed
but not shipped"), which the 0.35.5 bump in #2632 tripped;
- collect() skipped every os/cpu-restricted package as "build tooling",
which hid all ten LGPL libvips binaries.
A non-permissive license has no place in the tree, shipped or not:
- sharp now resolves to graph-ui/stubs/sharp, our own MIT stand-in that
throws if anything ever calls it (direct dependency file:./stubs/sharp
plus override "sharp": "$sharp"). The whole sharp family leaves the
lockfile: 441 -> 411 packages, zero LGPL/GPL left (audited).
- ui_browser_licenses.py loses the exception and keeps only the browser
build evidence check; license-gate-check-npm.py validates that evidence
unconditionally, checks platform-restricted packages like any other,
and has no exclusion path; gen-ui-licenses.py drops the exception
section.
- tests: the exception tests become guards: an unshipped LGPL package
blocks, a platform-restricted LGPL package blocks, a platform-restricted
MIT package passes. Against the old scripts both LGPL guards fail.
The shipped browser bundle is unchanged: rebuilt, it differs from main's
only by the build's own -dirty version marker. License gate stage 3 on
this tree: 211 production packages, all allow-listed.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A file and the directory beside it with the same stem (genproto.sh next
to genproto/) got one qualified name: the file's Module QN equals the
directory's Folder QN, and the graph buffer keeps the structural node, so
the file's Module was dropped, and with it the edges that pointed at it.
On 17 infrastructure repositories this lost 27 Modules (shell scripts,
YAML and JSON manifests, a Makefile, Rust, Elixir, Python, Terraform).
One side now yields, decided once per run from the discovered files:
- the Folder is spelled proj.pkg.foo/ (no module QN ends in a slash);
- where the directory is a Java/Go package it IS that package's module
and Go imports resolve to it, so the Folder keeps its QN and the file's
Module is spelled proj.pkg.foo#sh instead.
Symbol QNs do not change.
The closure-repair planner declines when a deletion touches a collision
on either side, since the spelling depends on the file set. The kustomize
resolver binds a directory entry to its Folder even when a same-stem
manifest sits beside it. CBM_SEMANTIC_INDEX_VERSION 7 -> 8, so existing
indexes rebuild once.
Measured on main 56700bdb over the 17 corpora: colliding files without a
Module 27 -> 0 (19 marked Folders, 8 #ext Modules). k8s.io resolves 3
more kustomize entries, and deckhouse's `source shell_lib.sh` imports
land on the script instead of the directory.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Reuse a stable arena-owned function QN order for ambiguity marking and
submodule prefix lookup, preserving exact lookup policy and appended tails.
Add prefix/index failure regressions and a pipeline operation-count case.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The Python namespace checks added two more raw free(dep_targets) calls
on decline paths, growing pipeline_incremental.c past its memory-core
baseline. The first file that declines now stops both the changed and
the deleted loop; one exit after the loops skips the dependent query and
releases dep_targets and the row index for every reason, including the
existing missing-row and added-name declines.
closure_python_namespace_decline maps the namespace comparison to its
decline reason. Decline order and reasons are unchanged. The file drops
to 140 raw sites; the baseline is tightened to match.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Delphi code that calls 1C's OLE API through late-bound Variants uses
Cyrillic member names (v8_Base.<Cyrillic>.<Cyrillic>('1', Now),
NM.<Cyrillic>()). Every such name produced a tree-sitter ERROR node, so the
unit was reported as parse_partial and a procedure whose own name is
Cyrillic was dropped from the graph.
Root cause is upstream tree-sitter-pascal (Isopod, the pin we vendor):
its identifier token is ASCII-only, /[&]?[a-zA-Z_]+[0-9_a-zA-Z]*/, and
there is no other token a non-ASCII letter can lex as.
Fix: pascal becomes a self-maintained fork of the pin we already vendored
(Isopod/tree-sitter-pascal @ 042119eca2e1, v0.10.2 = upstream master),
following the arkts model. One patch
(tools/tree-sitter-pascal/patches/unicode-identifiers.patch) changes only
that token to /[&]?[\p{L}_][\p{L}\p{M}\p{Nd}_]*/. Keywords are still
extracted from this `word` token, so begin/end/procedure lex as before
(`begin` glued to Cyrillic letters is one identifier; `&begin` stays an
escaped identifier). Node types are unchanged. The \p{} classes compile
to numeric code-point ranges: the vendored parser.c has 0 non-ASCII bytes.
Regenerated with tree-sitter-cli 0.25.10 --abi 14 (ABI unchanged, no
external scanner; tree_sitter/parser.h is byte-identical to chialisp's).
Fidelity: the UNPATCHED grammar regenerated with 0.25.10 gives the same
symbol/token/field/production counts as upstream's shipped parser and
byte-identical parse trees on upstream's 88 corpus cases and its three
examples/*.pas (37,773 tree lines for codetoolmanager.pas); the patched
parser gives the same identical trees on all of those ASCII inputs and
passes all 88 corpus cases.
Proof (7 Pascal files: the report's snippet, a Cyrillic procedure with
keyword-boundary cases, an ASCII control, a broken unit, upstream's three
examples; fresh caches): parse_partial 3 -> 1 (only the genuinely broken
unit remains), nodes 581 -> 582 (the Cyrillic-named procedure), edges
1480 -> 1482 (+1 DEFINES, +1 SEMANTICALLY_RELATED); per-file function
counts for the ASCII files are unchanged.
Registered like arkts: MANIFEST.md fork row + note, audit FORKS map,
THIRD_PARTY.md, vendored checksums. LICENSE stays upstream's MIT,
byte-identical.
Tests: parse_coverage gains the report's snippet and a Cyrillic-named
procedure (not partial, definitions extracted), an ASCII control, and a
guard that broken Pascal is still flagged; the fork carries
test/corpus/unicode_identifiers.txt.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A Windows sanitizer job went red on one suite that aborted under
AddressSanitizer, and the job log showed an empty "every failure site"
section followed by the shadow-byte legend. The summary in
scripts/run-tests-parallel.sh searched the suite log for "FAIL" only. A
sanitizer report never contains that word, and its useful part (error
kind, access size, top frames) is at the start of the report, far above
the last 15 lines.
The summary now lives in scripts/suite-failure-report.sh, which the
harness sources. Between the unchanged failure-site and last-15-lines
sections it prints, for a failing suite, the test that was running and
the first sanitizer report. The framework flushes each test name before
the test body runs, so the last test line ahead of a report is the test
that was running; a report that comes after the last test finished is
labelled as such instead of blaming a test. The report is shown from its
header (AddressSanitizer, LeakSanitizer, ThreadSanitizer,
MemorySanitizer, UndefinedBehaviorSanitizer) to its own SUMMARY line, at
most 40 lines, without the shadow dump and the legend; a recoverable
"runtime error:" line is shown with its stack only. Further sanitizer
headers, "runtime error:" lines and SUMMARY lines follow (at most 10),
then signal and timeout lines (at most 5).
A suite that dies with no report and no completion summary gets the
running test named. A trap-mode UBSan build stops on an illegal
instruction and prints nothing, so that name is the only evidence its
log holds. A log with an assertion failure and no report prints exactly
what it printed before.
The same file keeps the logs a red run needs. An EXIT trap in the
harness copies the log of every suite that did not record rc=0, plus
results.txt, into test-logs/failed/. The trap covers the scheduler
giving up and the union guard as well as the normal end, and it never
changes the exit status.
scripts/run-test-wave.py reported pass=0 for a suite that died without
its summary line. In that case, and only then, it now counts the PASS
markers in the log. A suite with a summary line is never recounted, so
the totals of a green run are unchanged.
tests/test_failure_report_contract.sh drives the real functions with
fabricated suite logs and runs as Step 0i2 of scripts/test.sh. It fails
on the previous behaviour in every sanitizer, running-test and
result-line check and passes on it in the unchanged-output checks. The
awk code uses no interval expressions and no character classes; the test
passes with BSD awk and grep on macOS and with mawk and GNU grep on
Ubuntu. scripts/README.md names the new internal file.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Smoke's binary string audit blocked on every platform in the dry run on
70543ba8 (run 38204190338, all 12 smoke jobs): "Credential patterns
found in binary". #2068 rebuilt the graph-ui bundle. The two reviewed
Monaco 0.56.0 quick-input lines are still there but minified to different
bytes, so their SHA-256 no longer matched the pair recorded on 09-09.
Re-reviewed: the CI-built bundle has exactly two flagged lines (15,644 and
72,472 bytes). Every match is the input box's password-mode flag
(_password=!1, password=!!e.password, inputBox.password=this.password,
inputBox.password=!1), not a credential. All 12 smoke logs show the same
two lines. The new pair is computed as filter_reviewed_lines does
(printf '%s' line | sha256) on CI's binaries: GNU strings on the Linux
amd64 artifact and Apple strings on the darwin arm64 artifact give the same
two hashes, and both are exempted. The comment now says how to redo this
when the bundle changes again.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Brings in #2596 (MSan: hidden uninitialized reads) and #2068 (CodeAtlas
web).
README: main's CBM Atlas line and this branch's infrastructure-as-code
line are both kept.
tools/list metadata: main and this branch each stayed under the 20 KiB
discovery budget, but together they reached 20,809 bytes. The branch's
wording is shorter now (search_graph kind/namespace, one hint per trace
mode: 136 bytes less), and the reviewed budget is 21 KiB so the
topology/deploy/run modes keep their usage hints.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
scripts/memwaste.sh set no memory budget, so the daemon derived one from
the machine's RAM: 1792 MB on the 7 GB macos-14 runner, about 4 GB on the
16 GB Linux and Windows runners. The event layer itself holds 0.5-1 GB, and
on macOS the index stopped with over_memory_budget before it measured
anything, in both full-mode runs since the lane was added (dry runs
37212102899 and 38186491924, peaks 3449 and 3111 MB at the stop).
The script now passes CBM_MEM_BUDGET_MB=4096, what the larger runners
already derived, unless the caller sets one.
Checked on macOS arm64 with CBM_WORKERS=3, the macos-14 runner's core
count:
- budget 1792: over_memory_budget in parallel_extract (rss 3904 MB), the
CI failure;
- budget 4096: the index completes and the report is written;
- the edited script with no budget in the environment: 4096 is applied,
rc 0.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Five suites (grammar_regression grammar_labels pipeline lang_contract
grammar_probe_e) were excluded from the MSan lane for "stack-overflow
under instrumentation". That was never deep recursion. MSan aborted while
REPORTING real uninitialized reads:
- every abort was on the main thread. MSan's "T<n>" is the OS thread id
and equalled the pid in all five x86-64 aborts (CI job 91868259558);
the old block read it as a worker-thread ordinal, so the worker-stack
knob, MSAN_ORIGINS and CBM_WORKERS=1 could never change anything;
- with the default slow unwinder (the image's MSan-instrumented
libunwind) a report exhausted the main thread's 256 MiB stack, and the
overflow handler then failed with "nested bug in the same thread";
- with fast_unwind_on_fatal=1 the same test prints the
use-of-uninitialized-value report instead (measured on one tree and
test: slow unwind aborts at the stack limit, fast unwind reports
wit/scanner.c:13).
The two reads are fixed in the previous commits of this series, so:
- MSAN_OPTIONS defaults to fast_unwind_on_fatal=1 (the build already
keeps frame pointers);
- MSAN_EXCLUDE is down to cli, whose install-path failure is a separate,
still-undiagnosed cause;
- the known-red block, the header (it still claimed CI runs without
exclusions) and the MSAN_ORIGINS and stack comments say what is true;
- the partial-lane warning and the exclude-name typo check lived in a
branch behind `elif true` that never ran. They now run in the
per-suite path, the check as a case match instead of a grep -q pipe
that pipefail can misreport;
- the comments in scripts/ci/msan-lane.sh and above the test-msan job
in _test.yml no longer say the five suites overflow.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
tree_sitter_wit_external_scanner_create() is declared to return void *
but had an empty body. tree-sitter's runtime stores its result as the
scanner payload (parser.c ts_parser__external_scanner_create), so every
WIT parse used an indeterminate value, which C11 6.9.1p12 makes
undefined. The scanner never reads its payload, so ordinary builds were
unaffected, and the vendored build's -w hid -Wreturn-type.
MemorySanitizer reports it at wit/scanner.c:13 from grammar_labels,
grammar_probe_e, grammar_regression and lang_contract. Those four suites
were excluded from the MSan lane because the report itself crashed and
posed as a stack overflow (fixed in the msan.sh commit of this series).
Recorded in MANIFEST.md's local-patch table. Upstream
bytecodealliance/tree-sitter-wit still has the empty body at
059bfdd1af72, the last change to src/scanner.c.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Reconcile the complete Atlas and Lens stream with current main while retaining both histories. Preserve architecture functionality, current process and watcher safeguards, test infrastructure, and dependency security updates. Add coverage for Git-environment isolation combined with child-local overrides.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Dockerfiles were Module + ENV/ARG Variables only: the 261 FROM and
114 COPY --from lines of the infra corpora joined nothing, and build
files named for a variant were not indexed at all.
- pass_docker (new, end of the K8s pass): FROM <image> and
COPY --from=<image> -> DEPENDS_ON {via: from | copy-from, image,
stage, line} from the Dockerfile's Module to an Image node. ARG
defaults are expanded (global ARGs in FROM, a stage's ARG / ENV in
COPY); a stage (by name or index), scratch and an ARG without a value
are no image.
- Image (new label, as approved): one node per repository, normalised
(docker.io for no registry host, library/ for an official image, a
registry host and port kept); tag and digest stay on the edge.
- Detection: Dockerfile.<x>, Dockerfile-<x>, Dockerfile_<x>,
Containerfile[.<x>] and <x>.Dockerfile are Dockerfiles, unless the
suffix is a language extension of its own (Dockerfile.j2 stays a
Jinja template) or a BuildKit ignore file.
- QNs: a Dockerfile.<variant> keeps its suffix (fqn.c and helpers.c,
which must agree); dropping it gave the variant the plain
Dockerfile's QN, merging both Modules. Covered by this branch's
index-format bump.
- Incremental: an Image no DEPENDS_ON / USAGE edge touches any more is
dropped on both routes, like a Process without SPAWNS; the
unreferenced-node GC takes a list of edge types.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
"Which CI jobs run hack/ci/run-e2e-tests.sh, and what does that script
start?" had no answer: a command running a script of the repository drew
no edge (SPAWNS reached Process nodes only, and repository scripts were
left out), and CI configurations were not read at all.
- A shell script, Make recipe or CI command that runs a script of the
repository -- by its path (./hack/x.sh, "$ROOT/hack/x.sh") or through a
shell (bash -e hack/x.sh) -- SPAWNS that script: its Module (the source
of the script's own edges, so CI -> script -> deploy chains connect),
else its File. The path must resolve to exactly one file; a program that
is no repository file keeps its Process edge.
- A new pass_ci.c reads GitHub Actions workflows and composite actions
(run: steps with their working directory, shell, env literals,
${{ github.workspace }} and ${{ github.action_path }}; no pwsh step, no
step of a Windows job) and Prow job configs (container command + args,
exec'd; bash -c TEXT as shell; a decorated job starts in the checkout).
The shell layer evaluates them like a script; their DEPLOYS and SPAWNS
edges come from the workflow / job file and carry "ci" and "job".
- env A="$X/y" PROGRAM: an assignment with an unknown value is no longer
taken for the program (it made a Process named after the value's last
segment and hid the program).
Measured on 17 repositories against an independent reading (libyaml for
the CI files, a separate shell reader for the commands): 292 of 292
script runs linked, 292 of 292 correct -- from Prow jobs (115 containers),
GitHub Actions steps, shell scripts and Make recipes, across 10
repositories (kubermatic, gardener, kube-hetzner, cluster-api, deckhouse,
k8s.io, ...); CI Process and deploy edges 4 of 4. Only SPAWNS (+295),
DEPLOYS (+1), Process (+3) and Chart (+1) changed; three runs build
identical graphs; index time unchanged (kubermatic 2.95 -> 2.93 s,
gardener 5.25 -> 5.25 s, kube-hetzner 1.13 -> 1.14 s, medians); editing or deleting scripts, workflows,
Prow jobs and Makefiles leaves the graph a fresh index builds (two
scenarios differ in SEMANTICALLY_RELATED edges and Make pattern-rule
nodes exactly as the previous build does).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A Terraform module installing a chart with `resource "helm_release"` had no
edge to the chart: "which modules install this chart" had no answer.
A new Terraform / OpenTofu pass (run at the end of the K8s pass, like the
shell pass) reads the helm_release blocks of .tf / .tofu files and links,
to existing nodes only, from the block's node (resource.helm_release.<name>):
- chart = "./charts/x" or "${path.module}/charts/x": the Chart of that
directory of the module;
- a repository and a chart name, or chart = "oci://.../<name>": the shared
external Chart of that name (the node chart dependencies use).
Values are read from string literals and ${path.module} only; a variable
chart, or a directory without a Chart.yaml, is no edge. Properties: via
terraform, the chart, external, line.
Measured on 17 repositories against an independent reading of the
Terraform files: 21 of 21 releases linked, 21 of 21 correct (EKS
blueprints 17, k8s.io 3, kube-hetzner 1). Only these DEPLOYS edges and 8
external Charts were added; three runs build identical graphs; index time
unchanged (cloud-foundation-fabric 1.91 -> 1.89 s, EKS blueprints 0.87 ->
0.88 s, k8s.io 3.44 -> 3.45 s, medians); editing or deleting the files
leaves the graph a fresh index builds.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A script or Make recipe running kubectl apply, helm upgrade or a kustomize
build piped into kubectl puts manifests, a kustomization or a chart on a
cluster, and none of it was an edge: "which script deploys this manifest"
had no answer in the graph.
A new pass (the Bash + Make layer, run at the end of the K8s pass) reads
scripts and Makefiles that name a deploy tool and links, to existing nodes
only:
- kubectl apply|create|replace -f: the File of a manifest, a directory's
manifests (-R: below it too), a glob's; -k: the Kustomization;
- kustomize build D (or kubectl kustomize D) | ... | kubectl apply -f -:
the Kustomization of D;
- helm install|upgrade [NAME] CHART, helm template piped into kubectl:
the Chart of a chart dir, else the shared external Chart of repo/name or
oci://.../name;
- find D -name G -exec kubectl apply -f {} \;: D's matching manifests.
The edge starts at the bash function or Make rule running the command,
top-level code at its file, with via (kubectl / helm / kustomize), the
deployed path or chart, and the line.
Words are evaluated by constant propagation, nothing is executed:
assignments (local / export / readonly / declare, arrays), ${X:-default},
$0 / BASH_SOURCE, $(dirname ..), $(cd .. && pwd), $(realpath ..),
$(git rev-parse --show-toplevel); cd / pushd / popd track the directory.
Anything else is unknown, and an unknown target is no edge: a value set in
a branch or loop is unknown after it, a subshell's changes do not leak, an
unquoted unknown word that may shift helm's positionals blocks the chart.
With no directory tracked, a relative path resolves against the root and
the script's directory, linked only when exactly one of them has a target.
Make: the Makefile's variables expand into each recipe line ($(shell pwd) /
$(CURDIR) = the Makefile's directory; a conditional's value is unknown),
which is then read as a shell line.
Measured on 17 repositories against an independent reading of the scripts
(its own tokenizer, variable and directory tracking, the file system): 86
of 86 deploys linked, 86 of 86 correct (k8s.io 66, kubermatic 12, gardener
8). Only DEPLOYS edges and 3 external Charts were added; three runs build
identical graphs; index time unchanged within 1 s (deckhouse 10 s, gardener
5-6 s, k8s.io 3-4 s, median of 3). Nine incremental edits (deploy lines
edited, removed and added, a Makefile, a deployed manifest, a new manifest
and a new non-manifest in an applied directory) leave the graph a fresh
index builds.
tree-sitter-bash folds `a | b` followed by a line `cat <<EOF | c` into
`a | (b cat | c)`; that shape continues the outer pipe.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The complexity guard (O(n^2) detection) had no infrastructure corpus, and
the infra resolvers looked candidates up by name across the whole
repository and then filtered them: an object reference by "Kind/name", a
selector by its rarest label pair, a custom resource by its CRD's
group/kind, a controller's kind by the Go type's name. Independent copies
share every name, so with k copies each lookup visited all k: the work
grew ~4x per doubling while the edges stayed ~2x.
Candidates are now looked up where they can match:
- K8s references, selectors and CRDs through indexes keyed by the bundle
(kustomize build root / chart / dir, and the deploy scopes for
references) and the name, type or label pair; a source visits only the
bundles it is in. The repository-wide fallback for a custom resource
without a CRD beside it stays (it stands for every copy by design);
- the operator pass through a (package dir, type) index: a controller's
kind is one lookup in the imported or own package; only an import no
module of the repository has still checks the same-named types. Group
declarations are deduplicated through a set, not a scan.
Child access in the K8s extractor goes through a tree cursor:
ts_node_named_child(n, i) walks from the first child on every call, so the
index loops over mapping pairs, sequence items and a file's documents were
quadratic in a wide node.
The guard gains fixtures (tests/fixtures/complexity/Kubernetes: a
kustomize bundle with workloads, Service, Ingress, PDB, NetworkPolicy,
HPA, RBAC, CRD + CR, cert-manager; tests/fixtures/complexity/Go: a
controller-runtime operator with +groupName types and an embedded CRD),
work counters g_k8s_candidates / g_op_candidates, and a gate on them and
on the fixtures' own node and edge growth; fixture dirs now also apply to
templated languages. An API group is one identity across a repository, so
the fixtures' CRDs use groups no copy's Go code declares -- the guard
measures accidental growth, not that by-design join. The originality
check covers the infra sources and scans them against yaml-language-server,
helm-ls, kustomize, Flux's kustomize package and controller-tools.
Before: k8s candidates 126 -> 504 (4.00x), operator 45 -> 288 (6.40x)
per doubling; after: 42 -> 84 and 6 -> 12 (2.00x), fixture edges 2.01x.
On 17 infrastructure repositories the graphs are identical to before;
three runs build identical graphs. Index wall time against origin/main,
median of three: deckhouse 9.58 -> 9.59 s, kubermatic 2.93 -> 2.93 s,
cloud-foundation-fabric 1.90 -> 1.90 s. A manifest of thousands of
documents indexes ~10 % faster (2.40 -> 2.13 s).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A bare import specifier was always external. The TS resolver now asks the
pipeline what it already knows for import resolution, without indexing
package.json or tsconfig.json: the nearest tsconfig/jsconfig `paths` and
`baseUrl` (the alias collection; the first target, as TypeScript tries it
first), then the workspace package whose `name` the specifier is (the
package map), bound to the package's entry module (`exports`, `main`,
`module`) when the repository indexes it. Every other bare specifier, a
package's subpath included, stays external. Incremental runs stay exact:
a tsconfig change re-resolves the files under it, a package.json change is
a manifest change.
path_alias gets cbm_path_alias_resolve_into, which writes into the caller's
buffer; cbm_path_alias_resolve now wraps it (two raw allocations fewer, the
memory-core baseline lowered to match).
Also fixed: a declaration in an index file has no `index` in its QN (fqn.c
strips it from a symbol's QN, not from the module's), so a reference to one
missed its node; the resolver now names a file's declarations with that
prefix.
Proof (ship build, Apple M5 Pro / 48 GB): the TypeScript compiler
b465fdbfe175 unchanged (69 edges, 57 of the prototype's 61 pairs);
discord.js 4dc1acc6 unchanged (no doc reference names a workspace import;
its packages' entries are build output). Tests: a tsconfig alias, a
workspace package's entry and an index file's declaration each bind, each
red with its rule reverted; path_alias, pipeline, edge_imports, incremental
and all doc suites pass (1298).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
81264814 built the production allocator as mimalloc's release build
(-DMI_DEBUG=0). That bought a large speed-up, but it raised what macOS
charges an index worker, its phys_footprint:
| peak phys_footprint | v0.11.0 | main |
|---|---:|---:|
| TypeScript | 1.25 GB | 1.94 GB |
| dotnet/runtime | 4.5 GB | 12.7 GB |
The 2026-10-09 bench missed most of this, because it sampled RSS.
The difference is one compile-time branch in mimalloc's unix decommit.
With MI_DEBUG or MI_SECURE>2, a purged range is mprotect(PROT_NONE)'d
and flagged needs_recommit. A release build leaves it "committed".
When the arena hands such a range out again, it is re-accounted as a
whole with MADV_FREE_REUSE (arena.c, _mi_os_reuse). macOS then charges
every page of it to the footprint, touched or not.
On macOS, take the recommit path in every build. Linux and Windows keep
upstream's behaviour; there MADV_DONTNEED already drops RSS at once.
Measured on Apple arm64 (macOS 26.6), main ec4fbd3a vs this change, cold
full indexes interleaved with main (C#: 2 runs each, in the same session
as main):
| corpus | footprint before | footprint after | change | wall time |
|---|---:|---:|---:|---|
| TypeScript | 1.98 GB | 1.09 GB | -45 % | unchanged |
| kubernetes | 2.92 GB | 1.88 GB | -35 % | unchanged |
| dotnet/runtime | 12.6 GB | 4.9 GB | -61 % | unchanged |
| linux kernel | 16.0 GB | 13.5 GB | -16 % | 574 -> 559 s |
The graph is byte-identical: TypeScript, kubernetes and django give
the same nodes, edges and CALLS-set hash.
dotnet/runtime is now close to v0.11.0's 4.5 GB.
Measured but not taken:
- the whole debug build, which costs ~5 % wall time on C#;
- MI_STAT=2, MI_PADDING=1 or MI_ENCODE_FREELIST=1 alone, none of which
moves the footprint;
- MIMALLOC_PURGE_DELAY=0, which is already set.
Vendored file changed in place (precedent 1f674c80); its hash in
scripts/vendored-checksums.txt is updated and security-vendored.sh passes.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The Helm file:// dependency records and the exact kustomize entry
resolver added raw strdup/free sites, so pass_k8s.c went from 6 to 9
raw allocator sites and pass_pkgmap.c from 82 to 85 -- the
memory-core ratchet (make lint-memory-core, part of lint-ci) fails on
both files.
pass_k8s.c: a dependency record now holds core-owned copies (class
other) of its chart path and target QN. The QN helper copies the
libc string fqn.c hands out into the core and releases it in one
place, and the record array grows before anything is allocated, so
the failure path frees nothing. 9 -> 6, the baseline.
pass_pkgmap.c: the resolver's three frees release fqn.c's
libc-allocated QNs and paths and have to stay raw. The manifest read
buffer, private to this file, now comes from the core (class extract:
per-file source text) and its three callers release it there. 85 ->
81; the baseline drops from 82 to 81 in this change.
No graph change: kubermatic and kubernetes/k8s.io indexes are
identical to the previous build node for node and edge for edge,
deckhouse in every edge and node identity. Its remaining difference,
the order of the variants list on QN-colliding Go methods, also
appears between two runs of the same binary and is not caused here.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
pass_infrascan.c carried line-by-line parsers for Dockerfiles, .env
files, shell scripts and Terraform, five file classifiers and a JSON
bracket cleaner; pass_envscan.c walked a repository for URL-valued
environment assignments. No production code calls any of them: the
graph gets these files from the tree-sitter extractors, and the
envscan walk said so in its own header comment. Their only callers
were 40 tests in test_pipeline.c that asserted the parsers' output,
so the suite measured code the index never runs.
Deleted: cbm_is_dockerfile, cbm_is_compose_file,
cbm_is_cloudbuild_file, cbm_is_env_file, cbm_is_shell_script,
cbm_clean_json_brackets, cbm_parse_{dockerfile,dotenv,shell,terraform}_source,
cbm_scan_project_env_urls(_excluded), their result structs, and the
tests and test helpers that existed only for them.
Kept, all with production callers or a planned one:
cbm_is_kustomize_file and cbm_is_k8s_manifest (pass_k8s), cbm_infra_qn,
the Helm Chart.yaml parser, and the secret detectors
cbm_is_secret_binding / cbm_is_secret_value (still covered by
infra_secret_detection), which the secret-hygiene work reuses. The
secret detector's prefix offsets now have their own names instead of
borrowing the Dockerfile parser's keyword lengths.
No graph change: the deleted functions were unreachable from the
index. Suites green: pipeline 299 (339 minus the 40 deleted), platform,
grammar_probe_f/g, infrascan, discover, language, lang_contract,
incremental, integration, edge_structural, configlink.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Resolves the definition walks next to each other: main's decl_namespace (#2120) and this branch's extra_props are both compacted and both counted in the result census.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The PDF text layer carries data tables (D3). Their sources and notices are
now in THIRD_PARTY.md (shipped in every archive through
THIRD_PARTY_NOTICES.md) and in the SBOM:
- the Adobe Glyph List subset (BSD-3-Clause, Copyright 2002-2019 Adobe),
- the printable-ASCII advance widths of Helvetica and Times-Roman from
Adobe's core-14 font metrics (the AFM files are not distributed),
- the Unicode Character Database data of pdf_unicode.c (Unicode
License v3, full text included).
scripts/gen-pdf-unicode.py (stdlib CPython only) regenerates
pdf_unicode.c byte for byte with CPython 3.14.6 (Unicode 16.0.0);
pdf_tables.c is maintained data from here on. License texts were taken
from their sources (adobe-type-tools/agl-aglfn, unicode.org).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The test-impact engine certifies the history it diffs and refuses a shallow
clone ("Graft or shallow state prevents history certification"). The shadow
job, the gate-mode select-tests job and the main-branch producer all checked
out with fetch-depth 2. So every shadow answer since #2458 was run_all:
GIT_UNAVAILABLE, then POLICY_UNAVAILABLE, because the merge base's
.codebase-memory.json is read through the same git facts. That was 24 of 24
measured PR runs. All three jobs now check out full history (about 400 MiB,
these jobs only).
The producer never published a bundle either. coverage-map.py exits 1 when a
suite fails under coverage: the map is still written and that suite's tests
are incomplete. test-impact-publish.sh ran it under set -e, so the publish
aborted. The failing suite was test_impact_git. Its hostile-diff probe
re-enters the runner as a child, which inherited CBM_TEST_COVERAGE_DIR,
started a coverage run of its own, and refused the per-test LLVM_PROFILE_FILE
it inherits.
Two changes:
- The probe now clears CBM_TEST_COVERAGE_DIR, as it already clears
CBM_TEST_ONLY_FILE. The child's profile then goes to the spawning test, like
any other child process.
- publish treats exit 1 as a written, conservative map (incomplete tests run
whole), with a warning that names the suite. It fails only on exit 2.
Proof:
- Depth-2 clone at f9ce3865: test-impact select answers run_all
[GIT_UNAVAILABLE, POLICY_UNAVAILABLE] with "Graft or shallow state ...", as
in CI. Same commit with full history: graph certified, run_all
[RUNNER_REACHED] (the change reaches the runner's main).
- Coverage runner (Homebrew LLVM, macOS), test_impact_git: before, 43 passed
and 2 failed ("a coverage run starts with LLVM_PROFILE_FILE=/dev/null");
after, 45 passed. coverage-map.py --suite test_impact_git: 45 tests,
0 incomplete, exit 0.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Martin Vogel