The first commit of this branch is the Smart CI change as of the rebase
onto 9f85f104. A review of it found places where the engine could run
fewer tests than a change needs; each now runs everything or selects the
right tests, with a test that fails when its fix is reverted.
Engine:
- A `#define NAME "literal"` line counted as naming a role but selected
nobody, so the tests that start the blocking-git probe through their
marker macros would not have run. The scan now follows NAME, as a whole
word, to the code that uses it, through chains of up to 4 macros; a
define it cannot follow runs everything.
- main dispatches through the role table, so no edge leads from main to a
handler. A table entry the engine cannot read, or whose handler is not
exactly one function, was skipped; it now runs everything.
- In a runner file, only a test case names a role's literal. The runner's
own helper that reads the marker no longer counts as a spawner, so a
role only the runner names runs everything again.
- A handler reached first by a parse gap or by the spawn lift was never
lifted. Role lifts and parse gaps now run to a fixpoint, before the
runner check and again after the spawn lift.
- A reached sink that main calls (the blocking-git probe) counted as a way
into main and hid the real one: #2644 now names the start-up path main
-> cbm_index_supervisor_capture_build_fingerprint.
- No pointer past the end of a role line; the RUNNER_REACHED note clamps
its prefix.
CI:
- pkg/npm/README.md, which test_cli reads, runs the test legs instead of
the docs tier.
- The build rule lists the harness scripts the test legs run:
run-test-wave.py, path-safety.sh, clean.sh.
- The revert guard rejects the ancestor's coverage when a diff fails.
- The vetoed selection is uploaded (test-selection-vetoed).
Docs: SMART_CI.md states the new fail-safe cases, the version that lacks
the new config keys, the selection base, and that the narrowed leg builds
with gcc while the coverage comes from clang.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
v0.12.0 could not be installed through npm, pnpm, pip or the Go wrapper on
any platform (#2657): each wrapper refused an archive member above 256 MiB,
and every v0.12.0 binary is 331-335 MB. The v0.11.0 binaries were already
about 300 MB, so the Go wrapper, which has had the cap since v0.10.0, was
broken before; npm and PyPI gained it in v0.12.0. The archive is verified
against the release's checksums.txt before it is unpacked, so a byte cap
only ever refused what we ship.
- npm, PyPI, Go: delete every byte-size cap (compressed archive, per member,
aggregate, checksum manifest, download size). The checksum, the exact
member set, path and link refusal, the member count, HTTPS and the
--version check stay.
- All three honour CBM_DOWNLOAD_URL like install.sh, for testing: https, or
plain http to an exact loopback authority with no proxy and no redirect.
- Smoke: scripts/smoke-wrappers.sh installs the three wrappers from the
checkout against the release fixture and runs the binary each one
installed. smoke-test.sh Phase 13w runs it; SMOKE_REQUIRE_WRAPPERS=1 makes
it required on the PR smoke lanes and in the release smoke, which gates
publishing.
Other install paths found broken while checking every channel:
- Homebrew: the formula was still at 0.10.3 and installed the dynamic Linux
build, which needs glibc 2.38 (Ubuntu 22.04, Debian 12 and RHEL 9 lack it).
It now installs 0.12.0 with the static -portable Linux assets and tracks
the newest release in the version contract like the other surfaces.
- Go: `go install .../pkg/go/cmd/codebase-memory-mcp@vX.Y.Z` never worked
(no pkg/go/v* tags, so Go fetched the whole repository, over its 500 MB
module limit), and @latest built main's wrapper, whose version constant
lags each release. The release now also tags pkg/go/vX.Y.Z, and the
wrapper installs its own module version (the constant only for local and
untagged builds).
- npm: a stable release also moves the `next` dist-tag to itself; `next`
still pointed to 0.9.1-rc.1.
- Docs: winget and Chocolatey were listed as install channels but are not
published; they are removed until they are. Windows arm64 ships and is
now listed.
Fixes#2657.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A Chart node had one kind of outgoing edge, DEPENDS_ON for a dependency
its Chart.yaml declares. A chart without dependencies, most of them, was
an isolated leaf: nothing linked it to the objects its templates render,
to its values files, or to the images it runs. On 20 pinned infra
corpora, 238 of 260 Charts had no outgoing edge, while 2,626 objects sat
in chart templates/ directories.
- Chart -DEPLOYS {via: chart}-> each Resource of <chart>/templates/, for
its nearest enclosing chart: a subchart's objects are the subchart's.
The same edge a Flux Kustomization draws to what it applies.
- values.yaml / values-<x>.yaml beside Chart.yaml -CONFIGURES {strategy:
helm_values}-> the Chart.
- Chart -DEPLOYS {via: chart, ref: image}-> each Image its objects name
literally, read after pass_docker.c moved references onto component
Images. A templated image names no Image, and no values default is read.
An incremental closure repair re-runs the K8s pass over its plan only, so
a values file or a template edited alone would lose the chart's edge; such
an edit now declines to the full rebuild, like a Chart.yaml edit.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
An image reference that names no registry host was stored under an
invented one: `FROM nginx` became docker.io/library/nginx, `image:
acme/web` docker.io/acme/web, and a local `docker build -t myapp` or a
skaffold artifact `image: api` docker.io/library/myapp. docker.io is only
Docker's default. Podman and CRI-O resolve a short name through a search
list, mirrors rewrite it, and a local build tag may exist on no registry
at all. The graph stated a host the code never names.
An Image is now the repository as written, tag and digest dropped, with
a `registry` property: the host the reference names, else "unknown". A
stated host is kept as written; Docker Hub's spellings (docker.io,
index.docker.io, registry-1.docker.io) are docker.io, with library/ for
an official one-segment name. This holds for every source: FROM and
COPY --from, compose services, K8s containers, shell and skaffold build
tags, and the per-component Images of a repository built from several
Dockerfiles.
`FROM nginx` and `image: docker.io/library/nginx` are now two Image
nodes. Relating them would state the host again, so the graph does not.
The Image QN changes (__image__nginx instead of
__image__docker.io/library/nginx), so CBM_INDEX_FORMAT_VERSION moves to
3. Refreshed in place, an older index would give only touched files the
new identity and hold both forms of one image; every existing index is
rebuilt once on its first index after upgrading.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Nondeterministic Windows reds came from races on _config.db:
test-windows-guards section_start_status_port / daemon_lifecycle ("the
daemon did not accept the UI configuration"), section_churn_stability
("could not load saved runtime settings"), the windows-2025 release
smoke whose `--ui=true --port` server exited about 5 s after start
(Phase 15, read as "no embedded UI assets"), and section_crash_recovery
("the stale daemon state never cleared").
- Every _config.db connection now waits out a committing writer
(cbm_runtime_settings_wait_for_writers: one busy handler, logs
config_db.busy_wait once per busy wait, a 3 s bound for a wedged
holder). Startup's 50 ms window let a daemon's routine commit end
unrelated CLI starts; cbm_config_open and the UI load had no wait at
all, so a concurrent commit failed the daemon's UI save and made the
load keep defaults.
- A process killed mid-write (crash, taskkill /F, power loss) left a hot
rollback journal that the read-only settings reads cannot roll back
(SQLITE_READONLY_ROLLBACK). Every later command then failed with
"could not load saved runtime settings", permanently. Settings startup
and the UI load now recover once through a read-write open that rolls
the interrupted write back (config_db.hot_journal_recovered); a failed
recovery is reported with the journal path and stays fatal.
- write_value no longer rewrites an unchanged value. The upsert fired the
revision trigger, so every `daemon start` on a UI build committed and
locked readers out although nothing changed.
- `daemon start` and the stdio server wait for the set_context and
set_ui_config replies within their startup budget instead of the 1 s
probe budget (MAIN_CONNECT_TIMEOUT_MS) that decided the command
whenever the daemon's save ran long.
Tests, red on 8df0639c and green here:
- runtime_settings suite: three single-threaded lock tests (a log sink
commits the held lock when the waiter logs config_db.busy_wait, so no
test depends on a duration) and two hot-journal tests (a portable
fixture snapshots the database and its journal mid-transaction and
checks the journal is hot before the code under test runs).
- tests/windows/test_daemon_stability.py: new section
start_port_waits_for_settings_writer holds the lock against a real
`daemon start --port` and releases it on events; the crash section
now prints the status output it saw.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The v0.12.0 release smoke blocked every binary in its string audit:
"BLOCKED: Dangerous commands found in binary: wget". The line is the bare
string "wget" that src/pipeline/pass_shell.c compares a script's program
name with, and records as the `via` of a curl / wget HTTP_CALLS edge
(3b93db06). The binary never runs it.
Allow that whole line only, like the rst grammar's bare "telnet". A line
that carries a wget command (flags, a target, a URL) stays blocked; the
contract test now pins both cases.
Proof on the release run's own artifacts (run 38644765106, main 8df0639c):
linux-amd64-portable and darwin-arm64 binaries BLOCKED before, all three
audits OK after. test_security_strings_allowlist.sh: 49 passed; with the
allowlist line reverted, the new bare-token case fails.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The daemon keeps adjacency layers (store/graph_mem.h) resident across
requests -- 237 MB for the kernel -- with no end of life but LRU pressure.
- Idle eviction: a layer nobody has held for CBM_GRAPH_MEM_IDLE_EVICT_MS
(10 minutes, measured from its last release) is evicted on the next
request: cbm_mcp_handle_tool sweeps after releasing the request store, and
every acquire sweeps too. No timer thread. Ten minutes outlasts the gaps
between an agent's graph calls within one task, while a session that moved
on stops pinning the memory; a wrong guess costs one rebuild (1.2-1.5 s on
the 17 GB kernel index, tens of ms on ordinary repos). The clock is a test
seam (cbm_graph_mem_set_clock_for_testing), so tests move time instead of
waiting.
- Worker budget: application_worker_memory_slice_locked hands index workers
the aggregate minus the resident layer bytes (cbm_graph_mem_cached_bytes),
at least one byte per active job so a slice never reads as "no cap".
- Memory class: layers are charged to the new CBM_MEM_CLASS_GRAPH_CACHE
("graph_cache" in census and class logs) instead of STORE.
Responses stay byte-identical to the baseline on django and redis (tools
and Cypher) and on the kernel trace/hub and extra BFS/degree cases.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
On a kernel-scale index (8.5M nodes, 16.9M edges) trace_path's recursive
CTE and search_graph's per-row degree subqueries spend seconds in B-tree
seeks. graph_mem reads the edges once, in one sequential scan of the
covering UNIQUE index, into a typed CSR (node id -> out/in neighbours with
interned edge types) and answers both from RAM:
- store_bfs (shortest-path mode): BFS over the layer with the CTE's exact
semantics (minimal hop per node, the start only via a cycle back to it,
(hop, id) order, probe row, JOIN-skips for ids without a node row); node
rows and edge collection still come from SQL. Trail mode is unchanged.
- cbm_store_search with min/max_degree: the two correlated COUNT(*)
subqueries become cbm_adj_degree(); filter, order, page and total are the
same SQL.
Layers live in a process-wide registry (the daemon closes its stores after
every request), refcounted across request threads, allocated through the
memory core (CBM_MEM_CLASS_STORE) and bounded: all layers stay under a
quarter of the budget, idle ones are evicted oldest-first, and a layer that
does not fit is not built (SQL path, never a partial layer). A layer is
keyed by db path + store generation + the rollback-journal file state (the
header change counter every rollback commit bumps), read in one read
transaction with the scan; only read-only query stores of sealed DBs
qualify. The file state matters: the cross-repo pass writes CROSS_* edges
into other projects' published DBs in place without a new generation; such
a DB is in WAL mode and keeps the SQL path until the next publish.
Kernel, macOS, same index, fresh daemon: trace_path schedule 7.5 s -> 21 ms
warm (1.23 s first call incl. a 1.2 s build), search_graph Function
min_degree 50 4.2 s -> 170 ms warm (1.35 s first). Layer: 16,890,273 edges,
237 MB. Responses byte-identical to the baseline on django, redis and the
kernel (tools, Cypher and extra BFS/degree cases).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A config key linked to every code symbol whose name contained it. In
YAML, JSON and HCL files that guess was retired in WP0.9; TOML and INI
files kept it, and in 17 infrastructure repositories every one of their
72 key_symbol edges was a coincidence: `.gitleaks.toml useDefault` linked
to TestConfig_UseDefault and three other symbols, a toml-merge tool's test
data `config_path` to 46 `*ConfigPath` constants all over deckhouse,
`grafana.ini admin_user` to user-cluster constants, a Rust Cargo.toml
dependency to other services' go.mod requirements.
A .toml, .ini, .cfg or .conf key now links like an infrastructure key:
by exact name (camelCase and snake_case still normalize to one key) and
only to code inside the config file's component, its directory or the
one around a config/, conf/, settings/ or etc/ directory. .env and
.properties keys keep substring matching; the tests that covered it on a
TOML key now use a .properties key.
Measured on the 17 repositories: key_symbol 95 -> 23, the 23 being the
infrastructure edges WP0.9 reviewed, the 72 removed all TOML/INI and all
coincidences, no other edge or node changed; deterministic over three
runs; the incremental re-index equals a full index for INI, TOML, code
and tfvars edits.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Three query rewrites over the existing indexes; every MCP response stays
byte-identical to c9cacdf0 (tools + Cypher suites on django, redis and the
linux kernel, same index each).
- arch_packages: GROUP BY +n.name. With a bare column the planner walked
idx_nodes_name over every node to get groups in name order, with a table
lookup per node to test the label (6.3 s of the 7.3 s get_architecture on
the 8.5M-node kernel graph, for 5 Package nodes). The unary plus seeks
idx_nodes_label instead; the name tie-break is explicit because a sorted
GROUP BY follows the ORDER BY direction and reversed tied counts.
- get_schema_impl: label / edge-type counts step through the distinct values
(MIN(col) > previous) and count each value's index range instead of a
GROUP BY scan: kernel 0.26 -> 0.16 s and 0.54 -> 0.32 s. Values reach
ORDER BY in the same ascending order, so ties keep their order.
- cbm_store_search: an anchored name_pattern ("^schedule$", "^kmalloc")
adds idx_nodes_name ranges for its literal prefix (every ASCII letter case
of the first three letters under iregexp, which runs in the C locale),
one UNION ALL arm per range merged by ORDER BY name, id without a sort;
the count sums the ranges. LIKE hints and the regex stay in the WHERE, so
the rows are those of the plain scan. Patterns that do not compile keep
the plain query (and its error).
Kernel, interleaved medians n=5: get_architecture 9.2-10.3 s -> 1.03 s,
get_graph_schema 0.76 s -> 0.48 s, cycles 2.24 s -> 2.04 s, search_graph
^schedule$ 1.15 s -> 14 ms, ^kmalloc + label 0.64 s -> 14 ms. Peak daemon
RSS for get_architecture +67 MB -> +22 MB.
The ranged SQL is built with a bounds-checked append (no snprintf-derived sizes) into CBM_MEM_CLASS_STORE buffers; a build that does not fit rolls its binds back and the plain query runs.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
An external Chart was <project>.__helm_dep__.<name>: keyed by its name,
inside one project. Two repositories naming the same published chart held
two unrelated nodes, and a project naming two charts of one name from two
publishers (kubermatic's cert-manager from charts.jetstack.io and from its
own mirror) would have merged them into one.
An external Chart whose repository URL is stated is now
__chart__<repository>/<name>, the same QN in every project, the URL in a
`repository` property: scheme and host lower-cased, a trailing slash and
`.git` dropped. A Chart.yaml dependency, a kustomize helmCharts entry, a
Flux HelmRelease chartRef and a Terraform helm_release of an oci:// chart
that name one published chart meet in one node. A chart reference (a Flux
OCIRepository url, `chart = "oci://..."`) is cut to its repository; a
repository URL never is, so repository oci://public.ecr.aws/karpenter
with chart karpenter stays oci://public.ecr.aws/karpenter/karpenter. A
repository that is no URL (an @alias, no scheme, file://, credentials in
the authority, an unexpanded ${...}) or that the pass does not read (a
Flux HelmRepository source, a Terraform `repository =`, a shell
`repo/name`) keeps the per-project node: no identity is claimed for what
was not stated. A query or fragment (`?token=...`) takes the per-project
node too, so a token never becomes part of a QN or a property.
One predicate now tells an external chart from a chart of the repository
where Terraform, shell and the Go installer index the chart directories:
an external chart a Chart.yaml depends on carries that file's path, and
must never stand in for the chart the directory declares.
Measured on 17 infrastructure repositories against an independent
reading: 141 of 141 edges into external charts reach the identity their
stated repository gives (114 by URL, 27 per project). The only
differences to the previous index are the 79 external Chart nodes that
moved to their URL identity, each with its repository property, and the
114 edges into them, re-pointed with type, source and properties
unchanged. Every chart gate (Helm includes, helmCharts, Terraform, the Go
installer, shell, CI, GitOps deploys) reads the same as before; three
runs are identical; the incremental re-index equals a full index for a
shared chart's owning file edited and deleted, a URL respelled, a
dependency moved to another publisher and to an alias, a chartRef URL
edit and a new Terraform oci:// release. The index rebuilds once on
upgrade with the other version 9 changes.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
On the kernel index (8.5 M nodes, 1.9 M CALLS edges) `MATCH (a)-[:CALLS]->(b)
RETURN count(*)` peaked at 18-25 GB RSS (killed in a 12 GB container): the
store materialised every start node with all its strings (4.9 GB), every one
was deep-copied into a 1.6 KB binding, and every expanded row was materialised
again before projection.
cypher: execute_single now runs the start nodes through the pipeline in chunks
of 1024 (pattern-0 hops, late WHERE, WITH, RETURN). Each hop emits a binding's
expansions in input order and every consumer (RETURN rows / star / default
projection / aggregation, WITH aggregation / projection) keeps its state
between chunks, so the concatenated output is the single-pass output: same
rows, row order, first-seen group order, ORDER BY ties, DISTINCT, LIMIT and
max_rows truncation, OPTIONAL fallback rows, warnings. Queries with additional
MATCH patterns keep one pass (their join strategy depends on the first binding
of the whole set). Start nodes and fetched neighbours move their strings into
the binding instead of being deep-copied and freed.
store: cbm_store_node_scan_open/_next/_close hand out the rows of
cbm_store_find_nodes / _by_label a page at a time. Pages are keyset queries
along the index the eager query walks (idx_nodes_file: NULL file_path rows by
id, then the rest of the last file by id, then the files after it;
idx_nodes_label: by id), each reset before it returns, so no read lock is held
between pages. open verifies the eager plan with EXPLAIN QUERY PLAN and falls
back to the eager rows for any other plan.
Kernel, same index, medians of 3 (macOS arm64):
count(*) peak RSS 24.9 GB -> 256 MB, 23.0 s -> 13.0 s
top-callee peak RSS 12.4 GB -> 1.26 GB, 11.7 s -> 10.9 s
Responses byte-identical to the baseline on django, redis and the kernel
(19 Cypher + 24 tool cases each); the streamed scan equals the eager scan
field by field, in order, for every project and label of all three indexes.
A scan that fails part-way fails the whole query ("query aborted: reading the graph failed") instead of answering from the pages read so far:
The streamed start-node scan answered from the rows of the pages it had read
when a later page failed (a SQLite step error or a page re-prepare failure):
a partial result that looked complete. A failed page, a scan that cannot
open, or a failed eager fallback now marks the start source failed;
execute_chunk returns an error and cbm_cypher_execute reports
"query aborted: reading the graph failed (<store error>)" with no rows. The
store error is logged as cypher.scan_failed.
The test seam cbm_store_node_scan_test_fail_page_after (CBM_ENABLE_TEST_SEAMS)
fails one page after a given number of successful pages; the new test fails
the second page of a label scan (rows and count(*)) and a later page of the
all-nodes scan, and asserts an error result with no rows.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
kubespray applies its cluster add-ons with its own `kube` module: a
task runs kubectl apply on a manifest the play rendered from a template
a few tasks earlier. Both tasks were Functions and the template a File,
with nothing between them: no query reached the manifest a task applies.
A kube (kubespray's library module) or k8s (kubernetes.core /
community.kubernetes) task now DEPLOYS {via: ansible, module, line} the
repository file its play rendered the applied manifest from: a template
task's src (the role's templates/, else the task file's directory) or a
copy task's (its files/; never a remote_src copy, which reads the host).
The applied filename is traced through the play: a renderer whose dest
is the very string; {{ item.item.K }} over "{{ R.results }}" -> each
item of the renderer registering R; {{ item.K }} -> each of the task's
own items through the renderer of that dest; a literal filename a looped
renderer renders to; a k8s template: argument. A loop list is a literal,
a set_fact of the file, or a list of the role's defaults / vars main
file. Templates are found before copies to the same dest. remote_src is
read the way Ansible's boolean() reads it (y / yes / on / 1 / true / t,
any case): `remote_src: true` is a YAML boolean, not a string, and a
string-only read had kept every remote copy as a renderer.
The lists in a role's defaults / vars reach past the task file, so a
change to one takes the full rebuild (closure_ansible_decline); without
it, a list that names another template kept the edge to the old one. A
file holding Ansible tasks that the repair would re-walk as a dependent
-- a template it applies was deleted -- also takes the full rebuild
(closure_ansible_dependent_decline): the repair walks it without its
role's defaults, handlers and other task files, and drew none of its
list-fed edges.
Measured on 17 infrastructure repositories: 164 of 164 applied files
(52 of kubespray's 62 kube / k8s tasks, 158 templates) against an
independent reading, all in kubespray; nothing in the other 16; the rest
of the graph unchanged (+164 DEPLOYS only); deterministic over three
runs; the incremental re-index equals a full index for a template body
edit, a deleted template (with and without a same-named fallback beside
the task file), a task list edit, a copied manifest edit, a defaults
list edit and a deleted vars file. Of the 10 tasks left out, 4 act on
an object by name, one builds its definition with a template lookup,
one applies a remote_src copy, and 4 name their file through a filter, a
variable or two joined results lists.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Every index_repository call copied the whole existing database into a
stage (sqlite3 backup) before routing, and the exact no-op then sealed
that copy and renamed it back over the live file. On the Linux kernel
index the copy alone was a 19 GB write per unchanged re-index.
The copy is now deferred until routing needs it. The exact no-op is
decided read-only on the live generation with the same checks the
staged route applies to its copy (shallow integrity, format version,
current coverage metadata and doc-link layer, byte-identical semantic
manifest); the run then publishes nothing and only refreshes the
artifact from the live database. Any other outcome copies and routes
exactly as before, so every change -- including a same-size edit that
keeps the original mtime -- still rebuilds from content hashes.
cbm_pipeline_new no longer resolves the Git context the run start
always re-resolved (seven git subprocesses per call).
No-op index_repository, macOS, median of three runs of n=3:
django 840.8 -> 539.0 ms, redis 450.7 -> 310.5 ms, Linux kernel
18449.5 -> 2405.0 ms. Responses and graphs are byte-identical to the
baseline for the no-op and for modify/add/delete/rename edits.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
lint-memory-core failed on #2642: the new search_code scanner, result index, root containment cache and Cypher sort/group/distinct scratch used malloc/calloc/realloc/free directly (mcp.c +22, cypher.c +26, git_context.c +1). They now go through cbm_alloc/cbm_calloc/cbm_realloc/cbm_free/cbm_mem_strdup: file buffers as CBM_MEM_CLASS_EXTRACT, the rest as CBM_MEM_CLASS_OTHER, each block freed with the class it was allocated with. The result index reserves its slot before a result's strings exist, so its insert cannot fail and needs no cleanup; cbm_git_branch_resolve writes into caller buffers instead of returning heap strings. cypher.c ends below its baseline (128 < 132), which is tightened. Output unchanged: 42/42 tool and Cypher responses identical on django; mcp, cypher and git_context suites green under ASan.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
kubespray's 1,425 tasks and 65 handlers were text inside YAML files: a
search for "Containerd | restart containerd" found no node, and nothing
said which task triggers which handler.
Each task and handler of an Ansible file is now a Function, named by its
`name:` (else its module), {source: ansible, kind, module}, which its file
DEFINES -- a playbook's pre_tasks / tasks / post_tasks / handlers, a role's
tasks/ and handlers/ files, and the files those include; a block's block /
rescue / always tasks, never the block itself. A notify CALLS {via:
notify, line} every handler of that name or listen topic in the task's own
role (the directory before its tasks/, so nested roles stay apart), else
in its file.
A top-level YAML list is not Ansible by itself: JSON patches, image maps
and service lists are lists too, and would have made some 3,000 false
tasks in five other repositories. A file counts when it is a playbook, a
role's tasks or handlers file, or included by one (to a fixpoint).
Quoted names have their YAML escapes decoded. A notify joins two files,
so a change to Ansible content takes the full rebuild
(closure_ansible_decline); without it, renaming a listen topic or adding
a notifying task left the incremental graph different from a full index.
Measured on 17 infrastructure repositories: 1,490 of 1,490 tasks and
handlers (file, line, kind, name, module) and 276 of 276 notify edges
against an independent reading, all in kubespray; nothing in the other
16; the P7.1 include edges unchanged (211 of 211); deterministic over
three runs; the incremental re-index equals a full index for handler and
task file edits, a deleted handler file and a playbook edit. The only
other change: as every function joins the similarity corpus, four
SEMANTICALLY_RELATED scores in kubespray's Python module moved and one
pair was added.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
search_code
- Scan the indexed files in process instead of through grep (POSIX) or a
PowerShell Select-String pipeline (Windows), with the same grep -f
semantics: one pattern per line, ERE regex, binary-file and NUL handling,
-Hn records. The subprocess runner stays for the unindexed recursive
fallback and a Windows file_pattern it cannot prefilter.
- Windows now matches like macOS and Linux: Select-String matched
case-insensitively, used .NET regex syntax and read NUL-containing
files as text. A trailing CR is still ignored there, so `foo$` keeps
matching CRLF checkouts.
- Result merge, root containment and source reads no longer scale with
hits x results: a node-id index, one canonical root per request with a
per-file verdict cache, and fread spans instead of per-byte fgetc.
query_graph
- ORDER BY (RETURN and WITH) was a bubble sort. A stable merge sort keeps
the same comparisons and so the same order; WITH keys are resolved once
per row instead of twice per comparison.
- GROUP BY, DISTINCT (RETURN and WITH) and COUNT(DISTINCT) looked every
row up in a linear list. They hash the same keys now.
Store reads
- Read tools run their statements in one deferred read transaction,
renewed every 100 ms, instead of taking the database file lock once per
statement.
list_projects
- Resolve only the branch the listing prints: 2 git processes per project
instead of 7.
On macOS and Linux every response is byte-identical to before:
search_code 30/30 (django), 28/28 (this repo) and 28/28 (Linux, GNU
grep); all tools and the Cypher cases 86/86 on shared django and redis
indexes. On django: search_code 908 -> 355 ms, WITH ... ORDER BY
41 s -> 62 ms, count(*) 508 -> 184 ms, list_projects 94 -> 40 ms. On the
Windows ARM VM: search_code 1.6-3.4 s -> 0.2-0.5 s.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Cluster API deploys its manager as `capi-controller-manager` in
`capi-system`: config/default/kustomization.yaml puts `namePrefix: capi-`
and `namespace: capi-system` on the `controller-manager` of
config/manager/manager.yaml. The graph only had the declared name, so the
name the cluster runs -- the one docs, scripts and tests quote -- matched
no node.
A kustomization with a namePrefix or nameSuffix now gives every object of
its build a variant: a Resource named with the deployed name, in the
namespace the build gives it, `ecosystem: kustomize`, which the
kustomization's File DEFINES and which OVERRIDEs {via: namePrefix |
nameSuffix, kustomization, line} the object. Prefixes of nested
kustomizations compose outermost first, suffixes innermost first, and the
outermost namespace wins. CustomResourceDefinitions, Namespaces and
apiregistration APIServices keep their names, as kustomize's own
transformers skip them. Variants are minted after the Kubernetes joins:
inside a build, references use the declared names.
Measured on 17 infrastructure repositories: 56 of 56 renamed objects in
the 6 renaming kustomizations against an independent reading, with
deployed name, namespace and base object; nothing else changed (56 nodes,
112 edges added, no property of an existing node); deterministic over
three runs; the incremental re-index equals a full index when a prefix
changes, goes, nests, or a renamed object is renamed or deleted.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A `kubernetes_*` resource in a .tf file -- eks-blueprints' network
policies and storage classes, k8s.io's metrics-server stack -- declares
a cluster object, but the graph only had the Terraform block, named by
its Terraform address: no node carried the object's kind and name.
A kubernetes_<type> resource (also _vN) whose metadata name is a literal
is now a Resource of the same identity scheme as a manifest's object:
kind from the resource type, or a kubernetes_manifest's own apiVersion
and kind; name and namespace from the object's own metadata, never from
spec; `ecosystem: terraform` on its properties. The .tf file DEFINES it
and the block DEPLOYS it {via: terraform, line}. The objects are minted
before the Kubernetes joins, so a manifest of the same bundle references
them as it does a manifest's object (an Ingress maps to the Service). A
computed name, and a type that acts on an object declared elsewhere
(kubernetes_annotations, kubernetes_labels, ...), make none. Labels are
not read, so a Service selector does not reach a Terraform workload.
Measured on 17 infrastructure repositories: 21 of 21 literal-named
objects against an independent reading, with group, kind, name and
namespace; nothing else changed (21 nodes, 42 edges added, no property
of an existing node); deterministic over three runs; the incremental
re-index equals a full index when an object is renamed, gains a
namespace, or its file is added or deleted.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A .tfvars file in a directory without Terraform -- kubespray's
sample-inventory/cluster.tfvars, eks-blueprints' spokes/workspaces/*.tfvars --
is applied to a module further up with a -var-file the repository does
not state, and linked nothing.
When the file's own directory declares none of its names, it now sets
the variables of the nearest ancestor module that does: each assignment
to a declared variable OVERRIDEs {via: tfvars, inferred: true, line} the
`variable` block, from the file's Variable. A file whose own directory
declares one of its names keeps the exact edge, without `inferred`.
Measured on 17 infrastructure repositories: 238 of 238 assignments --
124 to their own directory's module, 114 inferred to an ancestor's --
against an independent reading, with the inferred flag; nothing else
changed; deterministic over three runs; the incremental re-index equals
a full index when an ancestor-linked file changes.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Ansible's composition produced no structure: kubespray's 40 playbooks,
roles and task files, tied together by 217 import_playbook /
include_tasks / import_tasks / include_role / import_role / roles:
directives, were unconnected YAML files.
A playbook or task file -- a YAML file whose top level is a sequence;
manifests, values, CI and compose files are mappings -- now IMPORTS
{local_name: <directive>, via: ansible, line} what each directive names:
import_playbook a playbook from its directory; include_tasks /
import_tasks / include a task file from its directory, else from its
role's tasks/; roles: items and include_role / import_role a role's
Folder, roles/<name> at or above its directory (a role by "./" / "../"
path from the directory). The ansible.builtin. prefix is understood; a
templated target links nothing.
Measured on 17 infrastructure repositories: 211 of 211 composition pairs
of kubespray against an independent reading, nothing else in any
corpus; deterministic over three runs; the incremental re-index equals
a full index in 4 scenarios (an included file edited or deleted, an
include pointed elsewhere, a playbook edited).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A Terragrunt unit (terragrunt.hcl) composes itself from other files --
`include` blocks, read_terragrunt_config calls, dependencies on other
units -- and the graph held none of it: its blocks only met same-named
blocks elsewhere through the generic name matcher.
Every unit is now evaluated in its own context, the way Terragrunt
evaluates it (also the files it includes): an include IMPORTS
{local_name, via: terragrunt, line} the file its path names; a
read_terragrunt_config IMPORTS {via: terragrunt, read, line} the file it
names -- from the block holding the call or, for a call in an included
file, from the unit's include block that brings that file in, `read`
naming the file holding the call; a dependency DEPENDS_ON the other
unit's terragrunt.hcl; a local terraform source IMPORTS its Folder.
Paths may be string templates of find_in_parent_folders (searched from
the unit's parent directory upward), dirname and get_terragrunt_dir;
anything else is unknown and links nothing. A run holding any .hcl file
re-links every unit (a .hcl change takes the full rebuild since the
previous commit).
Measured on 17 infrastructure repositories: 30 of 30 composition pairs
of the 6 units of the Terragrunt repository against an independent
evaluator, nothing else; deterministic over three runs; the incremental
re-index equals a full index in 5 scenarios (a read file edited or
deleted, the included root file edited or reading another file, a unit
edited).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A Terraform module's output read can live in one file (outputs.tf:
module.net.vpc_id) and the module's source in another (main.tf). When
only main.tf changed -- its source pointed elsewhere -- the incremental
repair recomputed that file's edges alone, so the read, held by two
unchanged files, stayed in the graph although a full reindex no longer
has it. Measured on kube-hetzner: 18 stale output reads after moving
one module's source.
Terraform, OpenTofu and Terragrunt inputs reach past their own file the
way Kubernetes bundles do: the files of a module directory resolve each
other's references, a module call decides what another file reads, a
.tfvars file sets the variables beside it. A change to a .tf, .tofu,
.tfvars or .hcl file now declines the repair to the full rebuild, as a
kustomization or manifest change already does.
The incremental re-index now equals a full index in all 18 Terraform
scenarios, the stale-read one included; a Terraform edit costs a full
rebuild (fabric: 1.64 -> 1.90 s). A pipeline test moves a module's
source and requires the read to go.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A .tfvars file's keys were Variable nodes connected to nothing: nothing
said which variable `region = "eu"` sets.
An assignment to a variable the module of the file's own directory
declares now OVERRIDEs {via: tfvars, line} that `variable` block, from
the file's Variable (child to base, as a values or patch overlay). A
.tfvars file elsewhere is applied by a -var-file the repository does not
state, and links nothing. A run touching a .tfvars file re-resolves like
one touching a .tf file.
Measured on 17 infrastructure repositories: 124 of 124 assignments of
the 19 files that set variables of their own directory, nothing else;
deterministic over three runs; the incremental re-index equals a full
index in 4 scenarios (a value edited, the variables file edited, a
.tfvars file deleted).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A module call with a local source -- `module "net" { source =
"./modules/net" }` -- linked nothing: 379 such calls on the 7 Terraform
repositories of 17 infrastructure corpora, and every `module.net.id`
reached at most the calling block, never the output it reads.
Such a call now IMPORTS {local_name, via: terraform, line} the Folder of
its source directory (the repository root: the Project), when that
directory lies inside the repository and holds .tf / .tofu files; a
registry, git or http source, or a path not starting with ./ or ../,
makes none. `module.net.id` -- also `module.net[0].id`, `module.net.0.id`
and `module.net[*].id` -- USAGEs {ref: output} the `output "id"` of that
directory, beside the USAGE of `module.net`.
Output reads cross directories, so a run touching any .tf / .tofu file
now resolves every module directory of the repository: an unchanged
caller keeps its reads when the callee file is re-parsed. A full index
takes as long as before; an incremental re-index touching a .tf file
pays for parsing them all (+0.24 s on a 2,000-file repository).
Measured: 349 of the 379 calls name a directory that exists in the
repository and IMPORT it, nothing else (the 30 others name paths a test
harness creates or that are absent); 644 of 644 output reads (holding
block to output) against an independent reading; the reference pairs
of the module resolver stay exact; only edges are added; deterministic
over three runs; the incremental re-index equals a full index in 6
scenarios (an output's value edited in the callee, its outputs file
deleted, a source moved, a widely called module edited, the root
module's caller edited).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Terraform references were resolved by the generic name matcher: by their
last name segment, across the whole repository. `format()` CALLS a
variable named format of another module, `local.x` meets a resource named
local, `.id` an output id anywhere. Measured on the 7 Terraform
repositories of 17 infrastructure corpora: 12,437 edges, recall 0.63 and
precision 0.39 against an independent reading of the references, and not
one of them left the block holding its reference.
A reference -- var.X, local.X, module.X, data.T.N, T.N -- now USAGEs
{ref, via: terraform, line}, from the top-level block holding it, the
block declaring it in the same module: the .tf / .tofu files of its
directory (a locals entry: its locals block); an entry of the block's
depends_on list DEPENDS_ON the object instead. The generic matcher makes
no edge for a .tf / .tofu file any more; a name only mentioned in string
text is no reference, nor is a block's reference to itself. A changed
file re-resolves its whole directory, so an unchanged file's reference
meets the declaration a change adds and loses the one it removes.
A bare block label (`variable region {}`, valid HCL) now names its block
like a quoted one; such blocks used to share the keyword's name and QN.
Measured: 7,771 of 7,771 file-level and 13,397 of 13,397 block-level
reference pairs (edge type included) and nothing else; the 10 other
corpora are unchanged; deterministic over three runs; the incremental
re-index equals a full index in 7 scenarios (rename a variable, a module,
a locals entry; delete the variables file; add a file; add a declaration
an unchanged file references; quote a bare label); index time unchanged
(within 1 % on the four largest Terraform corpora).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A Kubernetes object's metadata.labels were not in the graph: neither a
search for an object by label nor a kustomize patch whose target selects
by label could find anything.
A Resource now keeps its labels as `labels: "k=v,k2=v2"` (sorted; a
templated label, or one a selector cannot name, is left out), and a
patch target's labelSelector meets the objects of its build carrying its
terms -- `k=v` / `k==v`, `k!=v`, `k`, `!k`; a set-based term names no key
and meets nothing. A target with only a selector selects among every
kind; an annotationSelector is not evaluated and names nothing.
Measured on 17 infrastructure repositories: 1,956 Resources gain labels
and nothing else; an independent reading of metadata.labels agrees on
every one of the 1,804 objects both read (152 more are Go-templated,
which only the graph reads, through its skeleton). The 4 label-selector
patch pairs now resolve: kustomize patches 130 of 130.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A kustomization's `helmCharts` inflate Helm charts into its build -- the
fourth way a repository puts a chart on a cluster, beside a Flux
HelmRelease, a Terraform helm_release and a helm command; the graph read
none of them.
Each entry now DEPLOYS from the kustomization -- its Kustomization
Resource, else its File: the unit a Flux or Argo deploy of it reaches -- to
the shared external Chart of that name, with {via: helmcharts, external,
chart, url, version, release, namespace, values, line}. A chart name with
a path names nothing; the values file inside the repository is an IMPORTS
(local_name: valuesFile) of the kustomization.
Measured on 17 infrastructure repositories: 20 edges, every one in an
independent reading of the kustomizations (20 entries, cluster-api 10,
k8s.io 10); no other node or edge changes.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A kustomize overlay changes its base through patches; the graph read none of
the v5 `patches:` entries (path / patch / target mappings), so a patch file
showed up as a second copy of the object it patches, with no edge to it.
Each patch of a kustomization now OVERRIDEs, child -> base, the objects of
its build it changes, with {via: patch, kustomization, patch, line}:
- a patch file's documents: the object of the same group, kind and name
(and namespace, when the document declares one);
- an inline patch (from the kustomization's File) or a JSON6902 operation
file (from its own File): what its `target` selects -- kind, group,
version, name, namespace; a label or annotation selector is not
evaluated and names nothing;
- in the build of the kustomization stating it; a component's in the build
of each kustomization including it.
A patch file is an IMPORTS (local_name: patches) of its kustomization and
never a member of its build: its documents select nothing, are selected by
nothing.
Resources now keep the version of their apiVersion (`version`), which the
identity leaves out and a target may name.
Each build indexes its own objects by name once: a lookup in the
repository-wide name index would visit every copy of a name for every
patch. The complexity fixture gains patch files (one of four documents), an
inline and a kind-only patch.
Measured on 17 infrastructure repositories: 126 edges, every one in an
independent reading of the 173 patches (122 resolvable in the
repositories); the 4 pairs without an edge use a label selector.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Shell scripts, Make recipes and CI steps download tools and call services
with curl and wget; the graph had no edge for any of them, so neither what
a repository fetches nor the scripts that call its own services were
visible.
Such a command now HTTP_CALLS the Route of each URL it states, with
{via: curl | wget, method, url_path, host, line}:
- the method from the flags: -X / --request / --method, a body (-d, -F,
--data*, --json, --form*, wget --post-data / --post-file) POST, -T PUT,
-I HEAD, -G back to GET;
- the URL from constant words (VER=1.2.3 ... v${VER}), unknown parts `{}`
as in a Route path -- also in an unquoted word tree-sitter-bash splits
around its expansions; a URL whose host is not literal, or the value of a
proxy / referer option, is no request;
- never the user info, the query or the fragment: a token in a URL does not
reach the graph;
- the Route: path-only (__route__POST__/v1/items) when the host is a
Kubernetes Service of the repository in a service DNS form, where that
service's handler meets it as a code client's request does; any other
host -- a download, localhost, a compose service -- keeps it
(__route__GET__https://dl.k8s.io/release/{}/bin/linux/amd64/kubectl,
props {method, host, external}), and route_nodes makes no path-only twin
of it.
Incremental indexing now drops a Route without a file that no edge touches
any more -- its last caller's file changed or went. A full index never
holds one; before, the Route of a deleted code client stayed behind too.
Measured on 17 infrastructure repositories: 92 edges, every one correct by
hand; an independent reading of the scripts and Makefiles finds 82
requests, 81 of them in the graph (the last is lost in a file
tree-sitter-bash cannot parse past `[ "$v" \< "1.33" ]`). No property of
any other node or edge changes.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A workload's environment names the services it talks to
(PRODUCT_CATALOG_SERVICE_ADDR: productcatalogservice:3550, a URL, a
<svc>.<ns>.svc name); the graph had no edge for it, so the service topology
of an application was invisible.
A Kubernetes workload or a compose service now CONNECTS_TO the Service its
containers' environment addresses, with {via: env, key, container} -- never
the value:
- the address: the host of a URL or of host:port, a <svc>.<ns>.svc name in
the value, or the whole value when the key names an address (*_ADDR,
*_ADDRESS, *_HOST, *_URL, *_ENDPOINT, *_URI) and it is a bare DNS name; a
templated value, localhost or an IP addresses nothing, and only service DNS
forms count (svc, svc.ns, svc.ns.svc...), never another domain's host
(cbm_env_address, shared by both sides);
- Kubernetes: a reference resolved like every other one -- a Service of the
same bundle, in the named namespace or the object's own;
- compose: a service of the same file; a variable without a value takes the
project's .env, whose values compose expands against the lines before them
(CART_ADDR=cart:${CART_PORT}) -- the dotenv reader does that now too.
Consumers: trace_path's topology mode, the CLI's edge list and the README
(which also gains DEPLOYS and the env join of the previous commit).
Measured on 17 infrastructure repositories: 129 edges, every one correct by
hand; an independent reading of the values finds 133 pairs repo-wide, the 5
without an edge belong to another bundle or namespace.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The variables a container sets (K8s env, compose environment, Dockerfile ENV)
were container-scoped EnvVars of their own, joined to nothing: the readers of
a name point at the global EnvVar, which every program reading that name
shares, so a join through it was in the right program for 23 % of the pairs.
Each scoped EnvVar now CONFIGURES {strategy:env_read, dockerfile} the readers
of its name inside the files the container's image is built from:
- the image: a K8s container's literal image, built from exactly one
Dockerfile (Image -DEPENDS_ON {via:build}->); a compose service's own build,
else its image's; a Dockerfile's own ENV. A K8s image reference now names
its container (USAGE {ref:image, container}), so a two-container pod joins
each container to its own image.
- its files: the context sources COPY / ADD put into the final stage, the
stages it is built FROM and the stages it copies from, resolved in the
build context (the Dockerfile's directory or the repository root, the one
the sources exist under), less the context's .dockerignore, never a test
file. They are kept on the Dockerfile's Module (image_context, image_files,
image_ignore).
pass_docker now reads COPY / ADD from the instruction's own text, as Docker
does: the grammar lost a JSON form opening with `/` and, after a word with a
trailing blank, took the following instructions for more sources.
Incremental: every run recomputes every join edge. A closure repair reads the
inputs of the files it leaves alone from the store and replaces the stored
edges wholesale; a .dockerignore (now a semantic-manifest control) changing,
or a deletion of a file an image copies, rebuilds in full.
CBM_SEMANTIC_INDEX_VERSION 8 -> 9.
Measured on 17 infrastructure repositories: 80 pairs, 80 correct by hand;
the graph matches an independent reading of the Dockerfiles 80/80.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The v0.12.0 release run's license gate blocked @img/sharp-wasm32@0.35.5
(Apache-2.0 AND LGPL-3.0-or-later AND MIT). sharp sat in graph-ui's tree
only because @huggingface/transformers lists it for its Node image path.
CBM Lens runs Transformers in the browser (a tokenizer and a causal LM on
WebGPU), never processes images, and the Transformers web build already
replaces sharp with an empty object. Yet sharp installed 14 LGPL packages
(10 @img/sharp-libvips-*, 3 @img/sharp-win32-*, @img/sharp-wasm32), and
the gate let them through two ways:
- a reviewed exception for exactly @img/sharp-wasm32@0.35.0 ("installed
but not shipped"), which the 0.35.5 bump in #2632 tripped;
- collect() skipped every os/cpu-restricted package as "build tooling",
which hid all ten LGPL libvips binaries.
A non-permissive license has no place in the tree, shipped or not:
- sharp now resolves to graph-ui/stubs/sharp, our own MIT stand-in that
throws if anything ever calls it (direct dependency file:./stubs/sharp
plus override "sharp": "$sharp"). The whole sharp family leaves the
lockfile: 441 -> 411 packages, zero LGPL/GPL left (audited).
- ui_browser_licenses.py loses the exception and keeps only the browser
build evidence check; license-gate-check-npm.py validates that evidence
unconditionally, checks platform-restricted packages like any other,
and has no exclusion path; gen-ui-licenses.py drops the exception
section.
- tests: the exception tests become guards: an unshipped LGPL package
blocks, a platform-restricted LGPL package blocks, a platform-restricted
MIT package passes. Against the old scripts both LGPL guards fail.
The shipped browser bundle is unchanged: rebuilt, it differs from main's
only by the build's own -dirty version marker. License gate stage 3 on
this tree: 211 production packages, all allow-listed.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
`cli list_projects` with no arguments blocked forever whenever fd 0 was a
pipe nobody closes: an ssh session's stdin, or a Node child_process.spawn
parent that never calls stdin.end(). That is the #1359 deadlock again.
The #1359 gate only skipped the stdin read for a tool whose schema
declares NO properties; #1181 then gave list_projects optional paging
properties, and the read came back for exactly the callers #1359 fixed.
It showed up as a 25-minute stall in the Windows smoke's Phase 1b over
ssh, and it is the same in v0.11.0.
The gate now asks what stdin is (terminal, regular file, or stream) and
what the tool needs:
- terminal: never read (unchanged);
- no properties: never read (unchanged);
- optional properties only: a regular file (`cli list_projects < args.json`)
is read, since a file always reaches EOF; a stream is not, and the call
runs with `{}` exactly as an interactive one does;
- any required property: file or stream, the documented
`echo '<json>' | cli <tool>` channel, unchanged; the bare call cannot
succeed without it.
Only list_projects is optional-only today; the other 16 tools all require
an argument. Piping JSON into list_projects is no longer read; flags,
--args-file and `< file` still are.
Proof (macOS, the release-tree binary vs this one, isolated runtime dirs):
- held-open pipe: `list_projects` hung until killed at 20 s -> returns in 0.4 s;
- `list_projects < args.json` ({"format":"json"}): JSON on both;
- `printf '{"project":"nope"}' | cli index_status`: same "project not
found" on both.
Tests: the #1359 schema seam now takes the stdin kind; new
cli_optional_argument_tool_never_reads_stream_stdin (fails before: the
old gate admitted a stream for list_projects) and
cli_stdin_kind_classifies_file_and_pipe; the whole-table sweep checks
file and stream separately and asserts it really covers an optional-only
tool.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A file and the directory beside it with the same stem (genproto.sh next
to genproto/) got one qualified name: the file's Module QN equals the
directory's Folder QN, and the graph buffer keeps the structural node, so
the file's Module was dropped, and with it the edges that pointed at it.
On 17 infrastructure repositories this lost 27 Modules (shell scripts,
YAML and JSON manifests, a Makefile, Rust, Elixir, Python, Terraform).
One side now yields, decided once per run from the discovered files:
- the Folder is spelled proj.pkg.foo/ (no module QN ends in a slash);
- where the directory is a Java/Go package it IS that package's module
and Go imports resolve to it, so the Folder keeps its QN and the file's
Module is spelled proj.pkg.foo#sh instead.
Symbol QNs do not change.
The closure-repair planner declines when a deletion touches a collision
on either side, since the spelling depends on the file set. The kustomize
resolver binds a directory entry to its Folder even when a same-stem
manifest sits beside it. CBM_SEMANTIC_INDEX_VERSION 7 -> 8, so existing
indexes rebuild once.
Measured on main 56700bdb over the 17 corpora: colliding files without a
Module 27 -> 0 (19 marked Folders, 8 #ext Modules). k8s.io resolves 3
more kustomize entries, and deckhouse's `source shell_lib.sh` imports
land on the script instead of the directory.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The #961/#1989 rescue re-adopts definitions from the preprocessed second
pass when they sit in a raw parse error, and drops a copy only when its QN
equals a raw one. When the two parses scope the same definition
differently, both copies survived:
- Linux lib/tests/randstruct_kunit.c: a macro-generated struct body makes
the raw parse nest every later struct inside one ERROR region; the flat
expanded copies were adopted next to the mis-nested raw ones (5
duplicate groups).
- coreclr/rapidjson: the expanded parse nests utilcode.h classes wrongly
or loses rapidjson's scopes, and both copies survived.
One rule at one decision point: an expanded and a raw definition with the
same label, name and start line but different QNs are one definition. The
raw copy wins and whatever the expanded pass attributed to the expanded QN
moves to it -- unless
- the raw copy's nesting exists only because of a raw ERROR region that
the expanded parse resolved (the randstruct shape, six conditions in
cbm_misnested_raw_twin): then the flat expanded copy replaces the raw
one and the raw children (Fields, nested defs, call/usage scopes) are
re-parented to it; or
- the raw copy's QN is already held by a raw definition of another label
(coreclr inc/holder.h: the raw parse reads a FORCEINLINE constructor as
a Function of the class's scope-less QN): then the expanded copy keeps
its own, correct QN instead of vanishing into the Function's node.
Proof (macOS arm64):
- randstruct_kunit.c: 5 -> 0 duplicate groups, 12 Fields re-parented to
the flat structs, CALLS/USAGE/WRITES unchanged.
- dotnet/runtime native subset (src/coreclr + src/native): 700 -> 183
groups; the remaining groups are no raw/expanded pairs at one site.
- redis: node and edge sets byte-identical.
- New tests (RED without the change): extraction
extract_c_misnested_raw_struct_replaced_by_flat_copy,
extract_cpp_preprocessed_rescue_skips_raw_def_on_same_line,
extract_cpp_rescue_keeps_expanded_class_when_raw_qn_is_taken (fixture
delta-minimized from holder.h).
- conditional_variants, extraction, c_lsp, pipeline, edge_types_probe,
parse_coverage, lsp_resolution_probe: 1854 passed.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
42a93202 gave a non-static C-family function under a platform directory
(arch/x86/, src/win/, ...) or file suffix a platform-neutral QN, so the
platform variants of one function shared a node. On the Linux kernel that
merged only 644 of 17,413 such definitions; the other 16,769 lost their
path for nothing (`arch/arm64/mm/init.c` mem_init became
`arch.mm.init.mem_init`, and a lookup by its path QN found nothing),
arches missing from the token list kept their paths, and calls bound to
merged nodes produced impossible arch -> other-arch CALLS.
Every definition keeps its path-derived QN again (the v0.11.0 identity).
The platform-neutral QN is now a key: definitions carrying one get a
`variant_key` node property, and nodes of a project sharing a key are one
variant group. The test-impact walk (store_impact `follow_variant_keys`,
enabled by the test-impact engine) treats a reached variant as reaching its
whole group, at seeding and at every step, so a change confined to the
Windows variant still selects the tests that reach the POSIX variant's
callers. The key is a pure function of a definition's own path, name and
linkage: incremental re-parsing of the changed file writes what a full
index writes. Same-QN variants (#if/#else, Rust cfg, Go build tags,
.c/.h pairs) and their `variants` property are unchanged. No new edge type.
Proof (macOS arm64):
- Linux kernel subset (arch, include, kernel, mm, lib; 27,005 files):
arch functions whose QN is not under their own path 12,873 -> 0;
arch/A -> arch/B CALLS 10,353 -> 7,101; 581 variant groups (1,815
members), each one of the former cross-file merges; the path QN of
arch/arm64's mem_init resolves.
- This repository: 13 groups (test_impact_tree_{posix,windows},
mem_override_{posix,win}); before, 22 of 49 platform-path functions had a
stripped QN.
- New/rewritten tests (RED on the old tree or without the walk
expansion): conditional_variants cv_c_platform_files,
cv_c_platform_directories, cv_incremental_platform_files;
test_impact_engine
test_impact_engine_platform_variants_reach_each_others_callers;
store_impact impact_walk_crosses_platform_variants_only_when_asked;
cv_incremental_variant_partners re-fixtured to Go build tags.
- 18 affected suites: 2710 passed, 4 Windows-only skips.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Doc-section -> code candidates (doc_link_candidates, #2574/#2610) are the
documentation-in-the-graph feature of this release. They now run by
default for full indexes and not for moderate or fast ones; an explicit
`doc_candidates` argument, or a `doc_candidates` setting the user set,
overrides the default either way in every mode.
- pipeline: the switch is unset/on/off; unset means on only when the
effective mode is full.
- MCP: forwards only an explicit choice (argument, then a user-set config
value) to daemon jobs and workers; daemon auto-index and watcher/UI jobs
now carry a user-set value too (their workers have no config, so
`doc_candidates false` was ignored there).
- fast + doc_candidates on runs the semantic pass's corpus phases and the
doc phase only: candidates, and still no SIMILAR_TO or
SEMANTICALLY_RELATED edges.
- config: `doc_candidates` joins the config key table (default `auto`);
`config set doc_candidates ...` was refused as an unknown key before.
README, docs/CONFIGURATION.md and the MCP schema describe the default.
Proof (macOS arm64, django fb11376):
full 11,304 rows · moderate 0 · fast 0 · full+off 0 · moderate+on 1 ·
fast+on 1 (moderate/fast discovery skips docs/) -- nodes and edges are
identical between on and off in every mode; fast+on adds 0 semantic
edges.
- New tests (RED on the old code): pipeline
pipeline_doc_candidates_default_follows_mode,
pipeline_doc_candidates_explicit_wins_over_mode,
pipeline_doc_candidates_fast_on_legacy_incremental; mcp
tool_index_repository_doc_candidates_follow_mode,
tool_index_repository_doc_candidates_forwarded_only_when_explicit;
daemon_application
daemon_application_programmatic_index_forwards_set_doc_candidates.
- mcp 374 passed (4 Windows-only skips), daemon_application 73 passed,
pipeline/store_arch/cli/runtime_settings/index_policy/pipeline_frozen
green on the release tree.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The preprocessed second pass rebuilt the expanded source with simplecpp's
TokenList::stringify(), which puts one space between every pair of tokens:
`template<> struct bitonic<int64_t, AVX2>` became
`bitonic < int64_t , AVX2 >`. Every QN builder takes the class segment from
node text, so a definition the #961 rescue re-adopted from that buffer
(vxsort's INLINE/NOINLINE macros break the raw parse on every method line)
got a second, token-spaced QN, and the QN-equality dedup kept both. CALLS
split across the two spellings (dotnet/runtime: 902 compact + 996 spaced
into the bitonic sorters).
The expanded buffer now keeps the source spacing: tokens that touched in
the source (same file and line, no macro expansion between them) stay
touching, every other gap is still one space, and the line structure and
#line directives are unchanged, so every line remap holds. One fix point
instead of a normaliser at each QN builder; no vendored code changes.
Proof (macOS arm64, dotnet/runtime 60108ba6, two identical runs each):
- duplicate definition groups 706 -> 217 (v0.11.0: 545); nodes with a
token-spaced QN 495 -> 0; CALLS into bitonic 902 + 996 -> 998, one
spelling. The remaining groups are coreclr gc.cpp's SVR/WKS builds (184)
and 33 scope disagreements between the two parses.
- Side effect: rapidjson document.h's GenericValue members now exist once,
with the raw parse's scope (the token-spaced copy that carried the
nested scope is gone); `operator[]` / `operator()` in the expanded pass
now parse as single tokens.
- New test (RED without the change): extraction
extract_cpp_template_specialization_preprocessed_qn_single_spelling.
Three c_lsp collision fixtures recalibrated to the new byte offsets
(comment padding and offsets only, assertions unchanged).
- extraction, c_lsp, pipeline, edge_types_probe, parse_coverage,
conditional_variants, lsp_resolution_probe: 1852 passed.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Martin Vogel