Terraform references were resolved by the generic name matcher: by their
last name segment, across the whole repository. `format()` CALLS a
variable named format of another module, `local.x` meets a resource named
local, `.id` an output id anywhere. Measured on the 7 Terraform
repositories of 17 infrastructure corpora: 12,437 edges, recall 0.63 and
precision 0.39 against an independent reading of the references, and not
one of them left the block holding its reference.
A reference -- var.X, local.X, module.X, data.T.N, T.N -- now USAGEs
{ref, via: terraform, line}, from the top-level block holding it, the
block declaring it in the same module: the .tf / .tofu files of its
directory (a locals entry: its locals block); an entry of the block's
depends_on list DEPENDS_ON the object instead. The generic matcher makes
no edge for a .tf / .tofu file any more; a name only mentioned in string
text is no reference, nor is a block's reference to itself. A changed
file re-resolves its whole directory, so an unchanged file's reference
meets the declaration a change adds and loses the one it removes.
A bare block label (`variable region {}`, valid HCL) now names its block
like a quoted one; such blocks used to share the keyword's name and QN.
Measured: 7,771 of 7,771 file-level and 13,397 of 13,397 block-level
reference pairs (edge type included) and nothing else; the 10 other
corpora are unchanged; deterministic over three runs; the incremental
re-index equals a full index in 7 scenarios (rename a variable, a module,
a locals entry; delete the variables file; add a file; add a declaration
an unchanged file references; quote a bare label); index time unchanged
(within 1 % on the four largest Terraform corpora).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A Kubernetes object's metadata.labels were not in the graph: neither a
search for an object by label nor a kustomize patch whose target selects
by label could find anything.
A Resource now keeps its labels as `labels: "k=v,k2=v2"` (sorted; a
templated label, or one a selector cannot name, is left out), and a
patch target's labelSelector meets the objects of its build carrying its
terms -- `k=v` / `k==v`, `k!=v`, `k`, `!k`; a set-based term names no key
and meets nothing. A target with only a selector selects among every
kind; an annotationSelector is not evaluated and names nothing.
Measured on 17 infrastructure repositories: 1,956 Resources gain labels
and nothing else; an independent reading of metadata.labels agrees on
every one of the 1,804 objects both read (152 more are Go-templated,
which only the graph reads, through its skeleton). The 4 label-selector
patch pairs now resolve: kustomize patches 130 of 130.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A kustomization's `helmCharts` inflate Helm charts into its build -- the
fourth way a repository puts a chart on a cluster, beside a Flux
HelmRelease, a Terraform helm_release and a helm command; the graph read
none of them.
Each entry now DEPLOYS from the kustomization -- its Kustomization
Resource, else its File: the unit a Flux or Argo deploy of it reaches -- to
the shared external Chart of that name, with {via: helmcharts, external,
chart, url, version, release, namespace, values, line}. A chart name with
a path names nothing; the values file inside the repository is an IMPORTS
(local_name: valuesFile) of the kustomization.
Measured on 17 infrastructure repositories: 20 edges, every one in an
independent reading of the kustomizations (20 entries, cluster-api 10,
k8s.io 10); no other node or edge changes.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A kustomize overlay changes its base through patches; the graph read none of
the v5 `patches:` entries (path / patch / target mappings), so a patch file
showed up as a second copy of the object it patches, with no edge to it.
Each patch of a kustomization now OVERRIDEs, child -> base, the objects of
its build it changes, with {via: patch, kustomization, patch, line}:
- a patch file's documents: the object of the same group, kind and name
(and namespace, when the document declares one);
- an inline patch (from the kustomization's File) or a JSON6902 operation
file (from its own File): what its `target` selects -- kind, group,
version, name, namespace; a label or annotation selector is not
evaluated and names nothing;
- in the build of the kustomization stating it; a component's in the build
of each kustomization including it.
A patch file is an IMPORTS (local_name: patches) of its kustomization and
never a member of its build: its documents select nothing, are selected by
nothing.
Resources now keep the version of their apiVersion (`version`), which the
identity leaves out and a target may name.
Each build indexes its own objects by name once: a lookup in the
repository-wide name index would visit every copy of a name for every
patch. The complexity fixture gains patch files (one of four documents), an
inline and a kind-only patch.
Measured on 17 infrastructure repositories: 126 edges, every one in an
independent reading of the 173 patches (122 resolvable in the
repositories); the 4 pairs without an edge use a label selector.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A workload's environment names the services it talks to
(PRODUCT_CATALOG_SERVICE_ADDR: productcatalogservice:3550, a URL, a
<svc>.<ns>.svc name); the graph had no edge for it, so the service topology
of an application was invisible.
A Kubernetes workload or a compose service now CONNECTS_TO the Service its
containers' environment addresses, with {via: env, key, container} -- never
the value:
- the address: the host of a URL or of host:port, a <svc>.<ns>.svc name in
the value, or the whole value when the key names an address (*_ADDR,
*_ADDRESS, *_HOST, *_URL, *_ENDPOINT, *_URI) and it is a bare DNS name; a
templated value, localhost or an IP addresses nothing, and only service DNS
forms count (svc, svc.ns, svc.ns.svc...), never another domain's host
(cbm_env_address, shared by both sides);
- Kubernetes: a reference resolved like every other one -- a Service of the
same bundle, in the named namespace or the object's own;
- compose: a service of the same file; a variable without a value takes the
project's .env, whose values compose expands against the lines before them
(CART_ADDR=cart:${CART_PORT}) -- the dotenv reader does that now too.
Consumers: trace_path's topology mode, the CLI's edge list and the README
(which also gains DEPLOYS and the env join of the previous commit).
Measured on 17 infrastructure repositories: 129 edges, every one correct by
hand; an independent reading of the values finds 133 pairs repo-wide, the 5
without an edge belong to another bundle or namespace.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The variables a container sets (K8s env, compose environment, Dockerfile ENV)
were container-scoped EnvVars of their own, joined to nothing: the readers of
a name point at the global EnvVar, which every program reading that name
shares, so a join through it was in the right program for 23 % of the pairs.
Each scoped EnvVar now CONFIGURES {strategy:env_read, dockerfile} the readers
of its name inside the files the container's image is built from:
- the image: a K8s container's literal image, built from exactly one
Dockerfile (Image -DEPENDS_ON {via:build}->); a compose service's own build,
else its image's; a Dockerfile's own ENV. A K8s image reference now names
its container (USAGE {ref:image, container}), so a two-container pod joins
each container to its own image.
- its files: the context sources COPY / ADD put into the final stage, the
stages it is built FROM and the stages it copies from, resolved in the
build context (the Dockerfile's directory or the repository root, the one
the sources exist under), less the context's .dockerignore, never a test
file. They are kept on the Dockerfile's Module (image_context, image_files,
image_ignore).
pass_docker now reads COPY / ADD from the instruction's own text, as Docker
does: the grammar lost a JSON form opening with `/` and, after a word with a
trailing blank, took the following instructions for more sources.
Incremental: every run recomputes every join edge. A closure repair reads the
inputs of the files it leaves alone from the store and replaces the stored
edges wholesale; a .dockerignore (now a semantic-manifest control) changing,
or a deletion of a file an image copies, rebuilds in full.
CBM_SEMANTIC_INDEX_VERSION 8 -> 9.
Measured on 17 infrastructure repositories: 80 pairs, 80 correct by hand;
the graph matches an independent reading of the Dockerfiles 80/80.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The #961/#1989 rescue re-adopts definitions from the preprocessed second
pass when they sit in a raw parse error, and drops a copy only when its QN
equals a raw one. When the two parses scope the same definition
differently, both copies survived:
- Linux lib/tests/randstruct_kunit.c: a macro-generated struct body makes
the raw parse nest every later struct inside one ERROR region; the flat
expanded copies were adopted next to the mis-nested raw ones (5
duplicate groups).
- coreclr/rapidjson: the expanded parse nests utilcode.h classes wrongly
or loses rapidjson's scopes, and both copies survived.
One rule at one decision point: an expanded and a raw definition with the
same label, name and start line but different QNs are one definition. The
raw copy wins and whatever the expanded pass attributed to the expanded QN
moves to it -- unless
- the raw copy's nesting exists only because of a raw ERROR region that
the expanded parse resolved (the randstruct shape, six conditions in
cbm_misnested_raw_twin): then the flat expanded copy replaces the raw
one and the raw children (Fields, nested defs, call/usage scopes) are
re-parented to it; or
- the raw copy's QN is already held by a raw definition of another label
(coreclr inc/holder.h: the raw parse reads a FORCEINLINE constructor as
a Function of the class's scope-less QN): then the expanded copy keeps
its own, correct QN instead of vanishing into the Function's node.
Proof (macOS arm64):
- randstruct_kunit.c: 5 -> 0 duplicate groups, 12 Fields re-parented to
the flat structs, CALLS/USAGE/WRITES unchanged.
- dotnet/runtime native subset (src/coreclr + src/native): 700 -> 183
groups; the remaining groups are no raw/expanded pairs at one site.
- redis: node and edge sets byte-identical.
- New tests (RED without the change): extraction
extract_c_misnested_raw_struct_replaced_by_flat_copy,
extract_cpp_preprocessed_rescue_skips_raw_def_on_same_line,
extract_cpp_rescue_keeps_expanded_class_when_raw_qn_is_taken (fixture
delta-minimized from holder.h).
- conditional_variants, extraction, c_lsp, pipeline, edge_types_probe,
parse_coverage, lsp_resolution_probe: 1854 passed.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
42a93202 gave a non-static C-family function under a platform directory
(arch/x86/, src/win/, ...) or file suffix a platform-neutral QN, so the
platform variants of one function shared a node. On the Linux kernel that
merged only 644 of 17,413 such definitions; the other 16,769 lost their
path for nothing (`arch/arm64/mm/init.c` mem_init became
`arch.mm.init.mem_init`, and a lookup by its path QN found nothing),
arches missing from the token list kept their paths, and calls bound to
merged nodes produced impossible arch -> other-arch CALLS.
Every definition keeps its path-derived QN again (the v0.11.0 identity).
The platform-neutral QN is now a key: definitions carrying one get a
`variant_key` node property, and nodes of a project sharing a key are one
variant group. The test-impact walk (store_impact `follow_variant_keys`,
enabled by the test-impact engine) treats a reached variant as reaching its
whole group, at seeding and at every step, so a change confined to the
Windows variant still selects the tests that reach the POSIX variant's
callers. The key is a pure function of a definition's own path, name and
linkage: incremental re-parsing of the changed file writes what a full
index writes. Same-QN variants (#if/#else, Rust cfg, Go build tags,
.c/.h pairs) and their `variants` property are unchanged. No new edge type.
Proof (macOS arm64):
- Linux kernel subset (arch, include, kernel, mm, lib; 27,005 files):
arch functions whose QN is not under their own path 12,873 -> 0;
arch/A -> arch/B CALLS 10,353 -> 7,101; 581 variant groups (1,815
members), each one of the former cross-file merges; the path QN of
arch/arm64's mem_init resolves.
- This repository: 13 groups (test_impact_tree_{posix,windows},
mem_override_{posix,win}); before, 22 of 49 platform-path functions had a
stripped QN.
- New/rewritten tests (RED on the old tree or without the walk
expansion): conditional_variants cv_c_platform_files,
cv_c_platform_directories, cv_incremental_platform_files;
test_impact_engine
test_impact_engine_platform_variants_reach_each_others_callers;
store_impact impact_walk_crosses_platform_variants_only_when_asked;
cv_incremental_variant_partners re-fixtured to Go build tags.
- 18 affected suites: 2710 passed, 4 Windows-only skips.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
doclink_rst.c sorted d.toks unconditionally; the vector stays NULL until
the first token, and qsort's base is declared nonnull even for zero
elements. GCC UBSan on Linux amd64 reported it in rst_section_text_cut and
diagrams_rst ("null pointer passed as argument 1, which is declared to
never be null", doclink_rst.c:1767) -- the same class as the k8s pass fix.
Sort only when there are at least two tokens, as the file already does for
its section titles.
Proof: doc_links_rst on Linux amd64 (GCC, ASan+UBSan) reports the
violation without the change and none with it; doc_links_rst + diagrams on
macOS: 27 passed.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The preprocessed second pass rebuilt the expanded source with simplecpp's
TokenList::stringify(), which puts one space between every pair of tokens:
`template<> struct bitonic<int64_t, AVX2>` became
`bitonic < int64_t , AVX2 >`. Every QN builder takes the class segment from
node text, so a definition the #961 rescue re-adopted from that buffer
(vxsort's INLINE/NOINLINE macros break the raw parse on every method line)
got a second, token-spaced QN, and the QN-equality dedup kept both. CALLS
split across the two spellings (dotnet/runtime: 902 compact + 996 spaced
into the bitonic sorters).
The expanded buffer now keeps the source spacing: tokens that touched in
the source (same file and line, no macro expansion between them) stay
touching, every other gap is still one space, and the line structure and
#line directives are unchanged, so every line remap holds. One fix point
instead of a normaliser at each QN builder; no vendored code changes.
Proof (macOS arm64, dotnet/runtime 60108ba6, two identical runs each):
- duplicate definition groups 706 -> 217 (v0.11.0: 545); nodes with a
token-spaced QN 495 -> 0; CALLS into bitonic 902 + 996 -> 998, one
spelling. The remaining groups are coreclr gc.cpp's SVR/WKS builds (184)
and 33 scope disagreements between the two parses.
- Side effect: rapidjson document.h's GenericValue members now exist once,
with the raw parse's scope (the token-spaced copy that carried the
nested scope is gone); `operator[]` / `operator()` in the expanded pass
now parse as single tokens.
- New test (RED without the change): extraction
extract_cpp_template_specialization_preprocessed_qn_single_spelling.
Three c_lsp collision fixtures recalibrated to the new byte offsets
(comment padding and offsets only, assertions unchanged).
- extraction, c_lsp, pipeline, edge_types_probe, parse_coverage,
conditional_variants, lsp_resolution_probe: 1852 passed.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
#2513 (77bc4927) computes `url_context` -- is the callee an HTTP client or a
global fetch? -- for every positional argument of every call, in every
language, through cbm_service_pattern_match: a linear strstr over every
service pattern. Only the URL-folding functions read it, and they return
NULL outside Python and JS/TS; the one other reader is the template-string
branch. On the C# corpus (dotnet/runtime, 451k functions) extraction went
from 24.7 s to 37.4 s (+51 %); a macOS sample put the difference in
handle_calls -> cbm_service_pattern_match -> match_qn -> strstr.
A per-call url_context_t now computes the answer at most once and only when
an argument needs it: the folding paths ask only in folding languages, the
template-string branch asks for itself.
Proof (macOS arm64, Apple M5 Pro):
- C# parallel_extract 37.4 s -> 24.2 s, wall 127-132 s -> 119.7 s (the
pre-#2513 level, 24.7 s / 121.9 s).
- Byte-identical graphs: the full node and edge sets (properties included)
hash the same before and after on django (Python), typescript (TS) and
csharp, HTTP_CALLS included (457 / 283 / 39).
- extraction, extraction_imports, pipeline, parallel, infrascan and
edge_types_probe: 1009 passed.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The C# and Java Tier-2 registry builders register types, finalize, then
register the rest from `defs + type_count`. With no definitions of that
language the array is NULL, and NULL + 0 is undefined: clang's UBSan
reports "applying zero offset to null pointer" at cs_lsp.c:3886 on the
macOS legs of dry run 38321896354 (reached by
incremental_k8s_edits_reach_other_files, a project without C# files).
The second registration now runs only when there is something to register;
both finalize calls are unchanged.
cs_lsp, cs_lsp_bench, java_lsp, java_lsp_coverage, lsp_resolution_probe:
529 passed (macOS). The report reproduces on CI's macOS clang, not on the
local toolchain, so CI's macOS legs are the check.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A call no project symbol resolves, whose callee an import binds to an
external library, is now spelled through that import and classified with
the existing service patterns: `send` under
`from paho.mqtt import publish as send` is paho.mqtt.publish (ASYNC_CALLS
to its topic), `navigate` under `import { get as navigate } from 'axios'`
is axios.get (a typed GET HTTP_CALLS edge). Before, such calls had no
service edge at all.
- CBMImport gains `member`: the exported name an ES named import binds
(`get` in `{ get as navigate }`), NULL for default, namespace and other
imports and for every other language. Filled by process_named_imports,
carried through result compaction (and so through spill), counted in the
extraction census with the other import strings.
- One helper, cbm_pipeline_external_import_service, serves both call
resolvers: a per-file index (local name -> binding import), built on the
first unresolved call that asks, then one hash lookup per call.
- Precision: only unresolved calls, after the callee-name, route-suffix and
global-fetch fallbacks; never a relative import, an import an IMPORTS
edge materialized (a project module), a name bound by two different
imports, or a Python callee rebound as a parameter. An HTTP edge needs a
request verb in the spelled name, so urllib.parse.urljoin is not a
request. Python, JavaScript, TypeScript and TSX.
Tests: extraction_imports covers the member and its survival through
compaction; a pipeline test runs both resolvers over a typed GET, a typed
POST and a publish through aliases, and over controls that must stay
silent (urljoin, lodash and json aliases, a local definition and a
parameter shadowing the import, a materialized project package named
httpx). Each guard was reverted once and its control went red.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Refs #2291, #706, #1147. Preserve raw argument values and carry URL projections separately through compaction and spill.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Preserve registry array order and full-scan fallback when optional query
storage is unavailable. Keep temporary range storage scoped to each module
operation and shared type indexes in the registry owner arena.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
lookup_func_self guards a NULL qualified_name in its indexed bucket scan
and its post-finalize tail scan, but not in the linear scan an
unfinalized registry takes. Iterating a fallback chain checks every base
entry for shadowing against the head, so a head holding a func without a
qualified name crashed in strcmp (ASan SEGV in
pylsp_prefix_order_matches_iterator_and_chain_shadowing). The linear
scan now skips such entries like the other two.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Two gaps let a call on a vendor class escape the #1186 evidence:
- A vendor class the PHP LSP knows from its seeded library data (for
example GuzzleHttp.Client) resolved as php_static_resolved or
php_method_typed. Its target has no graph node either, but without the
php_external_receiver strategy the pipeline fell through to a
name-only registry binding. Methods found on a class reached through a
vendor `use` now carry php_external_receiver as well.
- A vendor call skips the registry, and the extractor names a scoped
call by its method alone (only a literal `Route::` scope is kept), so
`GuzzleClient::get('/x')` reached the route-verb fallback and became a
route registration instead of an HTTP call. When the callee text names
no service, the service patterns now also see the vendor method the
LSP proved (`GuzzleHttp.Client.get`, `Kafka.Producer.publish`), in
both the sequential and the parallel pass.
Refs #1186
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
collect_use_declaration dropped the group prefix of `use A\B\{C as D}`
and read the function/const keyword only from the declaration, so a
grouped alias mapped to the bare `C` with the declaration's kind. The
cross-file LSP then could not recognise a grouped vendor class import as
external, and a grouped `function`/`const` alias was registered as a
class. Prefix each grouped clause with the group namespace and take the
keyword from the clause when the declaration has none, mirroring
emit_php_use_decl in extract_imports.c.
Refs #1186
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Reuse a stable arena-owned function QN order for ambiguity marking and
submodule prefix lookup, preserving exact lookup policy and appended tails.
Add prefix/index failure regressions and a pipeline operation-count case.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The __all__ literal guard compared a uint32_t child count against
SIZE_MAX / sizeof(char *), which is always false on LP64 and failed the
GCC build (-Werror=type-limits). One bound now covers both limits on
every platform: the names are counted in an int, and
count * sizeof(*names) <= INT_MAX also fits a 32-bit size_t.
Building one import fact moves into py_ns_import_fact, and the
module-name child is skipped with ts_node_eq: for a plain import the
module is the null node, which equals no child. The loop no longer tests
`from` after the from-only prelude, which cppcheck 2.20 misread as
always false (knownConditionTrueFalse). Emitted facts are unchanged.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Capture ordered Python namespace facts and retain them through compact, spill,
and versioned LSP surfaces. Reject unavailable metadata before incremental
reuse and rebuild on Python namespace changes or deletion.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Resolve symbols declared in __init__.py under their package while retaining
the raw module identity for file and top-level caller nodes. Carry explicit
language through registry callers, select Python package imports, preserve
long exact names, and report scope allocation failures without partial output.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Preserve the arena-owned raw Module caller name so top-level resolutions
join the extractor in both pipeline drivers. Add occurrence and ownership
controls, including package modules and named lambdas.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The depth guard of the C/C++ and Python expression evaluators
(C_EVAL_DEPTH_LIMIT, PY_LSP_MAX_EVAL_DEPTH = 256) now reports a sticky
per-file failure that stops the walk and drops every LSP resolution of the
file. Both evaluators recursed once per chain link, so a 300-term sum or a
C++ fluent chain of ~130 calls counted as 256 levels of nesting: one long
expression cost its whole file every LSP call edge. A chain's length is not
nesting, and the guard must not decide graph content for it.
c_eval_expr_type and py_eval_expr_type now first walk down the chain
operands below the node (the operand the evaluator evaluates first and
derives the link's type from), then evaluate them innermost first at the
caller's depth, keeping pending links on a small arena-owned stack in the
context that nested evaluations share. When a link is evaluated, its chain
operand is already memoized, so the evaluator's own recursive call into it
is a memo hit one frame deep. Every node is still evaluated by the
unchanged inner evaluator, evaluation does not mutate scope within one
outermost call, and only operands the recursion itself evaluates are
pre-evaluated, so the types are exactly the recursive ones.
No longer counted as depth (any length):
- C/C++: binary operators other than == != < > <= >= && || (those yield
bool without evaluating an operand), member access (. and ->), calls on
a non-name callee, subscripts, conditional consequences, parentheses,
comma lists.
- Python: binary operators, attribute access, calls on obj.m(...), f()()
and (expr)(), subscripts, the else branch of a conditional, parentheses.
Still counted, unchanged guard and reporting (genuine nesting):
- C/C++: call arguments used for template argument deduction,
std::move/std::forward operands, unary, pointer and update operands,
assignment left sides, lambda return expressions, co_await, fold and
_Generic operands.
- Python: tuple/list/set elements, dictionary keys and values, the true
branch of a conditional, await operands, next()/iter()/assert_type()
arguments, called lambdas.
Proof:
- New c_lsp and py_lsp tests build chains of 300, 2,000 and 20,000 terms
(a binary + chain and a .a() receiver chain) next to an unrelated s.m().
On the previous evaluator, and with this change reverted, all four fail
at 300 terms: the depth error is recorded and the file resolves 0 calls
(C++ per-file; Python per-file and through the cross-file dispatch).
With it, every size records no error, resolves run -> S.m, and the
chain's result type resolves x.done() / the final .done().
- Real input, isolated cache, before and after:
redis 4f20cb48 (C): 36,876 CALLS, 10,699 LSP-resolved;
django fb113765 (Python): 60,123 CALLS, 25,153 LSP-resolved.
Identical CALLS edge sets on both (0 lost, 0 gained, 0 strategy
changes), 0 files with an LSP error before and after, wall time
unchanged (redis 5.5 s, django 9.1-10.2 s both ways).
Refs #1527
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Memoize C expression evaluation, index repeated scope and registry queries,
and report incomplete walks explicitly on memo or depth failure.
Refs #1527
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Four syntax-tree walkers recursed once per nesting level of the indexed
file, so their stack use followed the depth of the parse tree rather than
anything bounded. They walk iteratively now, with identical results.
- terminal_callee_leaf (extract_calls.c) and its PHP LSP mirror
php_reference_callee_leaf descended one wrapper node per frame on a
callee such as ((((f))))(...). Both are loops now: each step either
returns or moves to the child the search continues in, so the nodes
visited and the leaf chosen are unchanged.
- cbm_collect_error_regions and cbm_mark_pp_error_rows (cbm.c) recursed
into every child that contains an error. One cursor walk now drives
both. It stops on exactly the visible children ts_node_child
enumerated, in the same order, and never enters an ERROR subtree.
- is_direct_argument_value (extract_usages.c) moved from a labeled
argument's value to the argument itself by recursing. The grammar bounds
that step, but it is a loop now, taking the slow-parent test note once
per level as before.
calls_append_args (pass_calls.c) formatted each argument into a 512-byte
local and assumed the result fit. An argument that does not fit now ends
the array like one the props buffer cannot take. The parallel twin formats in
place and already stops on truncation.
Indexing this repository's src/ and internal/cbm/ with and without the
change gives byte-identical sorted dumps of nodes, edges per type, CALLS
edges and parse-coverage rows.
CBM_WORKER_STACK_SIZE moves to worker_pool.h so the new extraction tests
run on a thread of exactly the pipeline's size, in a forked child on POSIX
so an overflow is a signal instead of a dead runner. They nest 200,000
parentheses around a first-class callable and around an invalid token,
pin the labeled-argument classification across the loop conversion, and
hand the formatter an argument longer than its local buffer.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Delphi code that calls 1C's OLE API through late-bound Variants uses
Cyrillic member names (v8_Base.<Cyrillic>.<Cyrillic>('1', Now),
NM.<Cyrillic>()). Every such name produced a tree-sitter ERROR node, so the
unit was reported as parse_partial and a procedure whose own name is
Cyrillic was dropped from the graph.
Root cause is upstream tree-sitter-pascal (Isopod, the pin we vendor):
its identifier token is ASCII-only, /[&]?[a-zA-Z_]+[0-9_a-zA-Z]*/, and
there is no other token a non-ASCII letter can lex as.
Fix: pascal becomes a self-maintained fork of the pin we already vendored
(Isopod/tree-sitter-pascal @ 042119eca2e1, v0.10.2 = upstream master),
following the arkts model. One patch
(tools/tree-sitter-pascal/patches/unicode-identifiers.patch) changes only
that token to /[&]?[\p{L}_][\p{L}\p{M}\p{Nd}_]*/. Keywords are still
extracted from this `word` token, so begin/end/procedure lex as before
(`begin` glued to Cyrillic letters is one identifier; `&begin` stays an
escaped identifier). Node types are unchanged. The \p{} classes compile
to numeric code-point ranges: the vendored parser.c has 0 non-ASCII bytes.
Regenerated with tree-sitter-cli 0.25.10 --abi 14 (ABI unchanged, no
external scanner; tree_sitter/parser.h is byte-identical to chialisp's).
Fidelity: the UNPATCHED grammar regenerated with 0.25.10 gives the same
symbol/token/field/production counts as upstream's shipped parser and
byte-identical parse trees on upstream's 88 corpus cases and its three
examples/*.pas (37,773 tree lines for codetoolmanager.pas); the patched
parser gives the same identical trees on all of those ASCII inputs and
passes all 88 corpus cases.
Proof (7 Pascal files: the report's snippet, a Cyrillic procedure with
keyword-boundary cases, an ASCII control, a broken unit, upstream's three
examples; fresh caches): parse_partial 3 -> 1 (only the genuinely broken
unit remains), nodes 581 -> 582 (the Cyrillic-named procedure), edges
1480 -> 1482 (+1 DEFINES, +1 SEMANTICALLY_RELATED); per-file function
counts for the ASCII files are unchanged.
Registered like arkts: MANIFEST.md fork row + note, audit FORKS map,
THIRD_PARTY.md, vendored checksums. LICENSE stays upstream's MIT,
byte-identical.
Tests: parse_coverage gains the report's snippet and a Cyrillic-named
procedure (not partial, definitions extracted), an ASCII control, and a
guard that broken Pascal is still flagged; the fork carries
test/corpus/unicode_identifiers.txt.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Why: a function-like macro in a definition's declarator,
`void CBasePlayer::__API_HOOK(ReloadWeapons)(int slot)` or
`static void FNAME(first_caller)(void)`, parses as a function_declarator
nested directly inside the definition's own function_declarator. The
declarator walk descended to the terminal identifier, which is the MACRO,
so every definition wrapped by the same macro got one qualified name.
ReGameDLL_CS's hook definitions landed as 20 nodes literally named
__API_HOOK, and the Linux kernel's arch/x86/kvm/mmu/paging_tmpl.h (19
FNAME(...) definitions) as a single Function FNAME. The last definition
won, calls in every other body were sourced to it with a call line
outside its range, and callers never reached the real functions. The C
LSP had its own copy of the walk: it named the caller after the macro and
bound the macro's argument list as the function's parameters, so member
calls through a real parameter went unresolved.
Fix: cbm_c_macro_wrapped_name_node() recognises the shape. Valid C/C++
never nests two function_declarators directly (a function cannot return a
function; a function-pointer return has a parenthesized_declarator in
between), so the nesting always means a macro call. When that macro is a
plain or class-qualified name taking exactly one bare-name argument, the
definition is named after the argument; any other shape (multi-argument
wrappers, non-name arguments) keeps today's walk. The shared C-family
declarator resolver uses it, so the def name, the call-scope QN and the
unified extractor agree; the C LSP uses it too and keeps the outer
(real) parameter list. The class scope of `Cls::MACRO(x)` was already
derived from the qualified name and is unchanged.
Proof: dev builds of origin/main 1f9b4db1 (before) and this change
(after), same machine, fresh indexes.
rehlds/ReGameDLL_CS @ 4a50c42 (#946):
- nodes named __API_HOOK: 20 -> 0
- Function 986 -> 1003, Method 3768 -> 3859: 129 hook definitions now
have their own node, and every added definition starts on a
`__API_HOOK(<name>)(` line (0 false definitions). Removed: the 20
__API_HOOK nodes, plus CGrenade::Explode(Vector, Vector), which now
shares CGrenade.Explode with its hooked overload, as any C++ overload
pair does today (one node per QN).
- CALLS 9436 -> 9916; CALLS sourced from __API_HOOK 410 -> 0; callable-
sourced CALLS whose call line lies outside the caller's range 565 -> 198.
- parse_partial files 131 -> 131.
- 5 samples checked against source (CBasePlayer::Spawn 5771-6142 with 32
in-range out-calls incl. ReloadWeapons via lsp_implicit_this,
CBasePlayer::TakeDamage, PM_Move, CGrenade::ShootSmokeGrenade,
CBotManager::OnEvent).
- 18 of the 147 hook definitions are still not extracted, before and
after: the parser loses them to raw ERROR regions (a `T *EXT_FUNC`
pointer return misparsed as a multiplication, a preceding
semicolon-less LINK_HOOK_*(...) invocation, an #ifdef-split
signature). That is parser recovery, not naming, and is left for a
follow-up.
Linux arch/x86/kvm/mmu @ 2d2338c93 (#1957):
- paging_tmpl.h: 1 Function FNAME -> 19 distinct functions (all 19
FNAME(...) definitions; 0 false definitions in the directory).
- CALLS from paging_tmpl.h functions with the call line outside the
caller's range: 32 -> 0 (directory: 35 -> 3, the 3 pre-existing in
mmu.c); e.g. fetch 651-786 now owns its 13 calls, sync_spte 934-1005
its 8.
- 5 samples checked against source (prefetch_gpte, walk_addr_generic,
fetch, sync_spte, page_fault).
- #1957's own reproduction: first_caller and later_function are distinct
Functions, sink() at line 9 is sourced to first_caller (7-10), and
main -> first_caller resolves.
Tests: extraction::extract_cpp_macro_wrapped_declarator_names_def_issue946
(qualified + unqualified + EXT_FUNC hooks, macro in a header: def names,
QNs, range, real parameter count, per-body call scope),
extraction::extract_c_fname_macro_definitions_stay_distinct_issue1957
(the reported C reproduction incl. the LSP caller QN, and the kernel
template shape with a token-pasting FNAME defined only under #if),
extraction::extract_c_macro_wrapped_declarator_controls_issue946
(precision controls: file-scope and in-body macro call statements, a call
through a returned function pointer, a real function returning a function
pointer, a macro-wrapped prototype and a two-argument wrapper mint no
definition named after a macro argument), and
c_lsp::clsp_macro_wrapped_declarator_caller_qn_issue946 (LSP caller QN,
lsp_implicit_this, typed parameter). RED 3x before, GREEN after, RED on
revert; the LSP half alone binds the c_lsp test and the #1957 test.
Fixes#946Fixes#1957
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Brings in #2596 (MSan: hidden uninitialized reads) and #2068 (CodeAtlas
web).
README: main's CBM Atlas line and this branch's infrastructure-as-code
line are both kept.
tools/list metadata: main and this branch each stayed under the 20 KiB
discovery budget, but together they reached 20,809 bytes. The branch's
wording is shorter now (search_graph kind/namespace, one hint per trace
mode: 136 bytes less), and the reviewed budget is 21 KiB so the
topology/deploy/run modes keep their usage hints.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
tree_sitter_wit_external_scanner_create() is declared to return void *
but had an empty body. tree-sitter's runtime stores its result as the
scanner payload (parser.c ts_parser__external_scanner_create), so every
WIT parse used an indeterminate value, which C11 6.9.1p12 makes
undefined. The scanner never reads its payload, so ordinary builds were
unaffected, and the vendored build's -w hid -Wreturn-type.
MemorySanitizer reports it at wit/scanner.c:13 from grammar_labels,
grammar_probe_e, grammar_regression and lang_contract. Those four suites
were excluded from the MSan lane because the report itself crashed and
posed as a stack overflow (fixed in the msan.sh commit of this series).
Recorded in MANIFEST.md's local-patch table. Upstream
bytecodealliance/tree-sitter-wit still has the empty body at
059bfdd1af72, the last change to src/scanner.c.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Each variable a K8s container's env (containers, initContainers,
ephemeralContainers at any depth), a compose service's environment (map
or list) or a Dockerfile's last stage ENV (with the stages it is built
FROM) sets is now an EnvVar of that container -- of the image, for ENV:
`<owner QN>#env/<container | ->/<name>` {env_key, scope, container,
source}, owned by the declaring file, and the owner CONFIGURES it
{strategy: env_set, source, line}. `source` says where the value comes
from (value, configMapKeyRef, secretKeyRef, fieldRef, resourceFieldRef,
fileKeyRef, compose, dockerfile); the value itself is never read into
the graph.
Two containers setting PORT are two nodes, and neither is joined to the
global EnvVar PORT a program's readers share: the same name is read by
other programs (measured on 17 infra repositories, a join through the
global node is in the setter's own program for 23 % of the pairs).
Docker drops comment lines before it joins continued lines, but the
grammar ends an instruction at one; the Dockerfile pass now reads such a
comment line as a bare continuation (rows and offsets unchanged), so
`ENV A=1 \` / `# url` / `B=2` sets B too.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Reconcile the complete Atlas and Lens stream with current main while retaining both histories. Preserve architecture functionality, current process and watcher safeguards, test infrastructure, and dependency security updates. Add coverage for Git-environment isolation combined with child-local overrides.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A Kubernetes object running a container now links to the Image of its
literal image reference: containers, initContainers and
ephemeralContainers at any depth, so pod templates of workloads,
CronJobs and custom resources all count. It is the node a Dockerfile
building on, or building, that image joins -- the run half of
Service -> ... -> main().
- extract_k8s: a new CBM_K8S_FACT_IMAGE for a literal image leaf;
a shell substitution ($(X), ${X}) or a template expression is none.
- pass_k8s: the facts become USAGE {ref: image, image} edges from the
object's Resource to the shared Image node.
- pass_docker: cbm_pipeline_image_node() is the one place an image
reference becomes an Image (normalised repository); a reference with
a character no image reference can hold -- a Helm template's
placeholder such as <.Values.image.repository> -- is no image.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The held-out audit of the eight new languages is scored. Each tier is a
language and a reference form, read on a sample of 150 links (census below
150) from repositories never used to build or tune the resolvers. Three
blind reading rounds; bar: Wilson 95% lower bound >= 0.90 and point >= 0.95.
tier population judged correct Wilson low
go doc_link 5980 150 150 0.975
java link 23934 150 150 0.975
java linkplain 1372 150 150 0.975
java see 7711 150 150 0.975
java throws 650 150 150 0.975
java value 79 79 79 0.954
kotlin kdoc_link 3888 150 150 0.975
kotlin sample 279 150 150 0.975
kotlin see 369 150 150 0.975
kotlin throws 73 73 73 0.950
rust intra_doc 14119 150 149 0.963
rust link_def 1491 150 150 0.975
perl pod_L 2255 150 150 0.975
ts/js link 1222 150 150 0.975
php see 5171 150 150 0.975
c doxygen_ref 463 150 150 0.975
c see 509 150 145 0.924
These 17 families now ship: a resolved reference is a MENTIONS edge.
TS/JS {@linkcode} (10 held-out links) and @see (18) were all judged
correct, but are too few to reach the bar; they stay held, as do the
diagram and OpenAPI link families (3 links on 21 held-out repositories).
The Go ship-gate test turned its family off and restored only the held
families afterwards; with Go shipping it left Go off for the tests that
follow it. It restores the production gates first now.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A Markdown Section's docstring is its body collapsed and cut at 500 bytes.
A diagram fence in that body put the format's syntax into it (`-->`,
`participant`, `classDiagram`, `subgraph`), which names no code, and pushed
the section's prose out of the cap. The doc -> code candidates are scored
on that text.
Each top-level diagram fence (mermaid, plantuml, puml, uml, dot, graphviz,
d2) is now replaced in the section's text by the labels its diagram parser
reads: the names of its elements, then its arrow labels, in source order.
The parse runs into a scratch result, so the file's own Diagram and
DiagramElement nodes are untouched. A section without such a fence keeps
its text byte for byte; other fences stay as written.
Measured before it was chosen, on held-out repositories (147 sections that
embed diagrams, 503 section/candidate pairs, three blind reading rounds):
fence as written 277 stored precision 0.45 42 sections with a correct candidate
labels (this) 310 stored precision 0.54 56 sections with a correct candidate
fence dropped 127 stored precision 0.54 29 sections with a correct candidate
Labels gave 20 sections a correct candidate that the fence as written did
not, and lost 6 (exact McNemar p = 0.009). Dropping the fence lost more
than it gained.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Dockerfiles were Module + ENV/ARG Variables only: the 261 FROM and
114 COPY --from lines of the infra corpora joined nothing, and build
files named for a variant were not indexed at all.
- pass_docker (new, end of the K8s pass): FROM <image> and
COPY --from=<image> -> DEPENDS_ON {via: from | copy-from, image,
stage, line} from the Dockerfile's Module to an Image node. ARG
defaults are expanded (global ARGs in FROM, a stage's ARG / ENV in
COPY); a stage (by name or index), scratch and an ARG without a value
are no image.
- Image (new label, as approved): one node per repository, normalised
(docker.io for no registry host, library/ for an official image, a
registry host and port kept); tag and digest stay on the edge.
- Detection: Dockerfile.<x>, Dockerfile-<x>, Dockerfile_<x>,
Containerfile[.<x>] and <x>.Dockerfile are Dockerfiles, unless the
suffix is a language extension of its own (Dockerfile.j2 stays a
Jinja template) or a BuildKit ignore file.
- QNs: a Dockerfile.<variant> keeps its suffix (fqn.c and helpers.c,
which must agree); dropping it gave the variant the plain
Dockerfile's QN, merging both Modules. Covered by this branch's
index-format bump.
- Incremental: an Image no DEPENDS_ON / USAGE edge touches any more is
dropped on both routes, like a Process without SPAWNS; the
unreferenced-node GC takes a list of edge types.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Martin Vogel