cppcheck 2.20 reports the post-cut `if (!name[0])` as always false
(knownConditionTrueFalse): it cannot see that writing through `lt`
may empty `name`. The name is non-empty here, so the cut empties it
only when `<` is the first character. Test exactly that before the
cut. The behaviour is unchanged.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The cross-file Kotlin walk resolved a call on a typed method parameter by
type, but a call on a class property - a primary-constructor val, a body
property or `this.prop` - fell to the name guess. The cross registry held
each own-file class from its extraction defs, whose properties carry no type
(constructor vals none at all), and the walk's typed AST registration of the
same class was added behind that entry under the same QN, so every lookup
returned the field-less one. Constructor injection is the common Kotlin
dependency shape, so `repository.inserir(...)` on an injected port bound to
whichever same-named method the guess found, including an implementation of
another port (#1555).
The walk now registers the file's own declarations into an overlay chained
to the cross registry and checked first, indexed once the top-level
declarations are collected. The sole-definer fallback walks the whole chain
so its candidate set is unchanged.
Typing more receivers exposed resolution gaps that the untyped receivers had
hidden; each would otherwise turn a typed call into a wrong name guess:
- a member of the receiver class wins over a same-named extension of the
same file (Kotlin resolution order; the overlay holds both);
- `x.hashCode()` / `toString()` / `equals()` on a receiver typed by another
file: the per-file pass emits the kotlin.Any placeholder, the cross pass
the declared override, and two targets at one occurrence are ambiguous to
the call join. The per-file placeholder is dropped when the cross pass
resolves the same occurrence;
- the same members on a stdlib or JDK type (`count.toString()` on an Int)
resolve to the kotlin.Any builtin instead of a seeded stdlib member that
has no graph node;
- a property READ on a typed receiver whose QN the graph spells differently
(a nested class's property) never becomes a CALL_REFERENCE to a
same-named function: the reference join falls back by name to
non-callable targets only, which binds the property's own node;
- a typed value without an `invoke` operator (`private val handshake:
Handshake?`) no longer shadows the same-named member function;
- inside a lambda whose typed receiver becomes `this` (`repo.let { .. }`),
and inside a member extension, the enclosing class's members stay
callable bare (the implicit-receiver chain); a call naming a class stays
a constructor call.
Fixes#1555
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The extractor fix above names a generic impl's methods `Gen.in_generic`
on both the def and the call-scope side. The Rust LSP still built its
self type from the impl's raw text, `Gen<T>`. That broke three things:
- its caller QN (`Gen<T>.in_generic`) no longer joined the pipeline's
call scope, so every LSP row inside a generic or lifetime impl was
dropped for a registry guess. On 9 Rust repos that turned 1,927
lsp_* edges into suffix_match/unique_name;
- `self.` and `Self::` dispatch never found the impl's own methods;
`Self::assoc_g()` had no edge at all, on main as well;
- the return-type harvest never matched those methods, so chained
calls on them stayed untyped.
Strip the generic arguments from the impl's self type in both places.
Text that starts with `<` (a qualified path) is left alone.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
`Opts::write(true)` is an associated function. When the Rust LSP cannot
place it, the registry can (import_map or qualified_suffix onto
fsx.Opts.write). The two free-function short-name fallbacks (the
cross-crate member lookup and the project-wide lsp_short_name_unique)
ran first, though. They bound the call to the project's only free
`write`, and that LSP row pre-empted the right answer.
While the Rust registry also held other languages' definitions, a
same-named JS or Perl function often made the free name ambiguous and
hid this. With Rust-only registries the hijack replaced 516
qualified_suffix and 84 import_map edges on 7 Rust repos.
A call whose qualifying segment names a type now skips both fallbacks.
That covers an ASCII-uppercase initial, `Self`, and a qualified
`<T as Tr>::f` path.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The def side names a Rust impl method after the implementing type with
its generic arguments stripped (`impl<T> Gen<T>` -> `Gen`, extract_rust_impl).
The call-scope QN in extract_unified.c used the impl's type text verbatim,
`Gen<T>` / `Life<'a>`. So every method of a generic or lifetime-parameterised
impl had a scope QN that no node carries, and every call in its body was
sourced from the File node instead of the method. Base census, 9f85f104 par
(lane L7's #763 measurement): 87,855 Rust CALLS start at a File/Module node,
84,331 of them (96%) on a line inside a function span. Go is 0%.
Strip the generic arguments on the call side exactly as the def side does.
Plain impls, trait impls, free functions and inline modules were already
consistent (checked on a fixture with the base binary).
Test: extract_rust_generic_impl_call_scope_matches_def (plain, type-generic
and lifetime-generic impls; def QN == call scope).
Refs #763
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Both Rust registries were built from the pipeline's mixed-language def
set. The shared Tier-2 build (cbm_rust_build_cross_registry) and the
per-file conversion (pxc_lspdefs_to_rust) copied every def, so a Perl
`sub new` or any C function became a Rust free function. The
lsp_short_name_unique fallback then bound `Store::new()` to the Perl
constructor at confidence 0.90 on both venues: its "crate scope" check
compares only the first module-QN segment, which is the project name.
(Found by the generic-engine lane on a 12-language fixture.)
Register Rust defs only, at both conversion points, exactly as the Go
builder filters to Go. Rust impl-relation records already carry
CBM_LANG_RUST. As a side effect, the shared Rust build no longer
registers the C defs of a mixed C+Rust tree.
Tests: pipeline_rust_call_never_binds_another_language (sequential and
parallel): no CALLS from the Rust caller into the Perl file, and the
Rust associated function still binds.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Re-port onto current main (agreed with lane L7):
- The Python LSP binds every def/class a function body defines as a local
before walking the body (LEGB): an undecorated def as an exact callable
alias to its nested QN, anything else as a hard shadow, so no call falls
through to a same-named module symbol. Defs inside the body's compound
statements count too, and their bodies are now walked.
- Classes declared in a function are walked under the function's QN
(py_process_class_in), so super()/self calls in their methods resolve.
- cbm_registry_resolve_lexical moves AFTER the LSP override: both resolvers
call one entry point, cbm_registry_resolve_call_scoped, at the same place
(before the language-aware chain). It never uses the per-file resolve cache.
- Main's cbm_registry_add_lang carries the def-local exclusion, so every
registration site (full, parallel, incremental seed) applies it; the
separate cbm_registry_add_for_language and the second incremental language
derivation are dropped.
- The pass_pkgmap.c def-local import skip is its own continue.
Tests: local-vs-module shadowing in both orders, a decorated local, and a
function-local class's super()/self calls, sequential and parallel.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Why
The Python def walk stopped at every function_definition: it extracted
the def and never looked at its body. Every def or class declared inside
a function -- closures, decorator wrappers, the route handlers of a Flask
or FastAPI app factory (`@app.route` inside create_app), local classes and
their methods -- was absent from the graph. Blocks at module level (if /
try / for / with) were transparent, so only function bodies were blind.
The calls inside those defs were not re-attributed either: the call-scope
walk named a nested def by its class or module only (`pkg.mod.nested`, a
QN no node had), so the edges fell back to the file node, and inbound
traces of the called function silently lost the caller. A def nested in a
method got the method's own QN (`Tree.visit` for the `def visit` inside
method Tree.visit), so its calls, recursion included, landed on the
method. On an app-factory codebase this removes the whole HTTP layer.
Fix
- Extraction (extract_defs.c): after a Python def is extracted, its body
is walked with the def's QN as the enclosing scope; methods' bodies are
walked under the method QN. A nested def or class is qualified by the
def it sits in -- pkg.mod.create_app.index, pkg.mod.Tree.visit.visit,
pkg.mod.create_app.Settings.value -- the QN the Python LSP already gives
the calls inside it (py_process_function). Decorated nested defs keep
their decorators, so their route_path mints the Route and HANDLES.
Lambdas and comprehensions hold no statements and stay as they are.
- Call scope (extract_unified.c): for Python the nearest named scope, a
def or a class, prefixes a nested def or class, so every call, usage and
env read inside a nested def carries the nested def's QN.
- A def nested in a function is a local of it (Python LEGB), so it must
not become a global name: a Python symbol whose QN extends a
Function/Method node is registered for exact lookups only, never in the
short-name index (registry.c cbm_registry_add_for_language, used by the
full, parallel and incremental registration sites alike). Without this,
django's per-test `class MyModelAdmin` and every decorator's `def
wrapper` became unique_name / suffix_match targets for unrelated calls.
The incremental seed derives each stored file's language the way
discovery does (name, then shebang), so it keeps the same symbols out.
The import symbol-name fallback skips such locals too (pass_pkgmap.c),
and the Python LSP no longer binds a def-local class module-wide by its
short name (py_lsp.c).
- A bare call now resolves lexically first (registry.c
cbm_registry_resolve_lexical, strategy lexical_scope): the def or class
bound in the nearest enclosing function, skipping class bodies, stopping
at the module. The sequential and parallel call passes use it
identically, and so does the handler lookup of a call-registered route:
`app.add_url_rule("/ping", "ping", ping)` in create_app reaches the
local ping.
Proof (real code; every node and edge diffed, main 1f9b4db1 vs this)
- pallets/flask examples/tutorial (flaskr, create_app factory): the
nested `hello` handler is now create_app.hello with HANDLES to
/hello; 4 nested defs/classes added, 0 nodes and 0 edges removed.
- pallets/flask d73fa1c (tests declare handlers inside test functions):
Function 532 -> 1068, Method 318 -> 381, Class 130 -> 209. The +678
equals the 678 unique nested def/class QNs Python's own ast finds.
Handler HANDLES (decorators and add_url_rule) 32 -> 296. 0 nodes
removed, 0 duplicate QNs.
The 94 removed CALLS: 90 moved from the file node onto the nested def
that makes the call, 4 replaced by the lexically bound local.
- django fb11376 (local, 2,927 .py files): nodes 55,483 -> 60,052,
Function 1,992 -> 3,027, Method 28,850 -> 29,491, Class 8,187 ->
11,073. 0 nodes removed and 0 duplicate QNs. Every added
Function/Method/Class sits under a Function/Method of the same file,
and they match ast's 4,567 nested QNs except for 5 methods declared in
an `if` inside a local class body (class-body `if` is not walked for
any class). CALLS 62,430 -> 63,815. Every added CALLS edge has a nested
def at one end: 1,015 from nested defs, 1,219 lexical_scope into a
local, and none from outside the local's function. Of the 849 removed:
507 moved from the file node onto the nested def, 334 replaced by the
local the name binds, and 8 were false on main, all checked (e.g. the
recursive `get_actions` nested in call_command had been suffix-matched
to ModelAdmin.get_actions). HANDLES unchanged (135).
- Samples checked against the source:
AbstractBaseUser.check_password.setter -> set_password / save;
View.as_view.view -> setup / dispatch;
Apps.lazy_model_operation -> lazy_model_operation.apply_next_model
(line 426);
InstrumentedASGIRequest.META -> the local InstrumentedDict (line 279);
flask test_provide_automatic_options_arg_disable.index -> GET /.
Tests
- extraction: extract_python_nested_def_qualified_by_enclosing_def,
_calls_attributed_to_it, _decorated_nested_route_handler,
_nested_class, _nested_def_named_like_its_method,
_lambda_and_comprehension_stay_in_enclosing_def (control),
_nested_def_env_access_attributed_to_it (all _issue1548).
- pipeline: pipeline_python_app_factory_routes, _nested_def_is_function_
local, _local_classes_stay_in_their_def, _import_never_binds_a_local_def,
_nested_defs_parallel (>= 50 files, CBM_WORKERS=4),
_nested_def_local_after_incremental (all _issue1548).
- registry: registry_python_function_local_is_exact_only,
registry_resolve_lexical_walks_enclosing_defs (both _issue1548).
The extraction and pipeline tests were RED on main 3/3 for the asserted
reason and are green with the fix; the registry tests cover the new
registry API. Each production piece was reverted on its own, and each
revert turns at least one of these tests RED.
Fixes#1548
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
(cherry picked from commit 62fda43fd66200879db62a58903a32dab0dd1a20)
Terraform references were resolved by the generic name matcher: by their
last name segment, across the whole repository. `format()` CALLS a
variable named format of another module, `local.x` meets a resource named
local, `.id` an output id anywhere. Measured on the 7 Terraform
repositories of 17 infrastructure corpora: 12,437 edges, recall 0.63 and
precision 0.39 against an independent reading of the references, and not
one of them left the block holding its reference.
A reference -- var.X, local.X, module.X, data.T.N, T.N -- now USAGEs
{ref, via: terraform, line}, from the top-level block holding it, the
block declaring it in the same module: the .tf / .tofu files of its
directory (a locals entry: its locals block); an entry of the block's
depends_on list DEPENDS_ON the object instead. The generic matcher makes
no edge for a .tf / .tofu file any more; a name only mentioned in string
text is no reference, nor is a block's reference to itself. A changed
file re-resolves its whole directory, so an unchanged file's reference
meets the declaration a change adds and loses the one it removes.
A bare block label (`variable region {}`, valid HCL) now names its block
like a quoted one; such blocks used to share the keyword's name and QN.
Measured: 7,771 of 7,771 file-level and 13,397 of 13,397 block-level
reference pairs (edge type included) and nothing else; the 10 other
corpora are unchanged; deterministic over three runs; the incremental
re-index equals a full index in 7 scenarios (rename a variable, a module,
a locals entry; delete the variables file; add a file; add a declaration
an unchanged file references; quote a bare label); index time unchanged
(within 1 % on the four largest Terraform corpora).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A Kubernetes object's metadata.labels were not in the graph: neither a
search for an object by label nor a kustomize patch whose target selects
by label could find anything.
A Resource now keeps its labels as `labels: "k=v,k2=v2"` (sorted; a
templated label, or one a selector cannot name, is left out), and a
patch target's labelSelector meets the objects of its build carrying its
terms -- `k=v` / `k==v`, `k!=v`, `k`, `!k`; a set-based term names no key
and meets nothing. A target with only a selector selects among every
kind; an annotationSelector is not evaluated and names nothing.
Measured on 17 infrastructure repositories: 1,956 Resources gain labels
and nothing else; an independent reading of metadata.labels agrees on
every one of the 1,804 objects both read (152 more are Go-templated,
which only the graph reads, through its skeleton). The 4 label-selector
patch pairs now resolve: kustomize patches 130 of 130.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A kustomization's `helmCharts` inflate Helm charts into its build -- the
fourth way a repository puts a chart on a cluster, beside a Flux
HelmRelease, a Terraform helm_release and a helm command; the graph read
none of them.
Each entry now DEPLOYS from the kustomization -- its Kustomization
Resource, else its File: the unit a Flux or Argo deploy of it reaches -- to
the shared external Chart of that name, with {via: helmcharts, external,
chart, url, version, release, namespace, values, line}. A chart name with
a path names nothing; the values file inside the repository is an IMPORTS
(local_name: valuesFile) of the kustomization.
Measured on 17 infrastructure repositories: 20 edges, every one in an
independent reading of the kustomizations (20 entries, cluster-api 10,
k8s.io 10); no other node or edge changes.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A kustomize overlay changes its base through patches; the graph read none of
the v5 `patches:` entries (path / patch / target mappings), so a patch file
showed up as a second copy of the object it patches, with no edge to it.
Each patch of a kustomization now OVERRIDEs, child -> base, the objects of
its build it changes, with {via: patch, kustomization, patch, line}:
- a patch file's documents: the object of the same group, kind and name
(and namespace, when the document declares one);
- an inline patch (from the kustomization's File) or a JSON6902 operation
file (from its own File): what its `target` selects -- kind, group,
version, name, namespace; a label or annotation selector is not
evaluated and names nothing;
- in the build of the kustomization stating it; a component's in the build
of each kustomization including it.
A patch file is an IMPORTS (local_name: patches) of its kustomization and
never a member of its build: its documents select nothing, are selected by
nothing.
Resources now keep the version of their apiVersion (`version`), which the
identity leaves out and a target may name.
Each build indexes its own objects by name once: a lookup in the
repository-wide name index would visit every copy of a name for every
patch. The complexity fixture gains patch files (one of four documents), an
inline and a kind-only patch.
Measured on 17 infrastructure repositories: 126 edges, every one in an
independent reading of the 173 patches (122 resolvable in the
repositories); the 4 pairs without an edge use a label selector.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A workload's environment names the services it talks to
(PRODUCT_CATALOG_SERVICE_ADDR: productcatalogservice:3550, a URL, a
<svc>.<ns>.svc name); the graph had no edge for it, so the service topology
of an application was invisible.
A Kubernetes workload or a compose service now CONNECTS_TO the Service its
containers' environment addresses, with {via: env, key, container} -- never
the value:
- the address: the host of a URL or of host:port, a <svc>.<ns>.svc name in
the value, or the whole value when the key names an address (*_ADDR,
*_ADDRESS, *_HOST, *_URL, *_ENDPOINT, *_URI) and it is a bare DNS name; a
templated value, localhost or an IP addresses nothing, and only service DNS
forms count (svc, svc.ns, svc.ns.svc...), never another domain's host
(cbm_env_address, shared by both sides);
- Kubernetes: a reference resolved like every other one -- a Service of the
same bundle, in the named namespace or the object's own;
- compose: a service of the same file; a variable without a value takes the
project's .env, whose values compose expands against the lines before them
(CART_ADDR=cart:${CART_PORT}) -- the dotenv reader does that now too.
Consumers: trace_path's topology mode, the CLI's edge list and the README
(which also gains DEPLOYS and the env join of the previous commit).
Measured on 17 infrastructure repositories: 129 edges, every one correct by
hand; an independent reading of the values finds 133 pairs repo-wide, the 5
without an edge belong to another bundle or namespace.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The variables a container sets (K8s env, compose environment, Dockerfile ENV)
were container-scoped EnvVars of their own, joined to nothing: the readers of
a name point at the global EnvVar, which every program reading that name
shares, so a join through it was in the right program for 23 % of the pairs.
Each scoped EnvVar now CONFIGURES {strategy:env_read, dockerfile} the readers
of its name inside the files the container's image is built from:
- the image: a K8s container's literal image, built from exactly one
Dockerfile (Image -DEPENDS_ON {via:build}->); a compose service's own build,
else its image's; a Dockerfile's own ENV. A K8s image reference now names
its container (USAGE {ref:image, container}), so a two-container pod joins
each container to its own image.
- its files: the context sources COPY / ADD put into the final stage, the
stages it is built FROM and the stages it copies from, resolved in the
build context (the Dockerfile's directory or the repository root, the one
the sources exist under), less the context's .dockerignore, never a test
file. They are kept on the Dockerfile's Module (image_context, image_files,
image_ignore).
pass_docker now reads COPY / ADD from the instruction's own text, as Docker
does: the grammar lost a JSON form opening with `/` and, after a word with a
trailing blank, took the following instructions for more sources.
Incremental: every run recomputes every join edge. A closure repair reads the
inputs of the files it leaves alone from the store and replaces the stored
edges wholesale; a .dockerignore (now a semantic-manifest control) changing,
or a deletion of a file an image copies, rebuilds in full.
CBM_SEMANTIC_INDEX_VERSION 8 -> 9.
Measured on 17 infrastructure repositories: 80 pairs, 80 correct by hand;
the graph matches an independent reading of the Dockerfiles 80/80.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The #961/#1989 rescue re-adopts definitions from the preprocessed second
pass when they sit in a raw parse error, and drops a copy only when its QN
equals a raw one. When the two parses scope the same definition
differently, both copies survived:
- Linux lib/tests/randstruct_kunit.c: a macro-generated struct body makes
the raw parse nest every later struct inside one ERROR region; the flat
expanded copies were adopted next to the mis-nested raw ones (5
duplicate groups).
- coreclr/rapidjson: the expanded parse nests utilcode.h classes wrongly
or loses rapidjson's scopes, and both copies survived.
One rule at one decision point: an expanded and a raw definition with the
same label, name and start line but different QNs are one definition. The
raw copy wins and whatever the expanded pass attributed to the expanded QN
moves to it -- unless
- the raw copy's nesting exists only because of a raw ERROR region that
the expanded parse resolved (the randstruct shape, six conditions in
cbm_misnested_raw_twin): then the flat expanded copy replaces the raw
one and the raw children (Fields, nested defs, call/usage scopes) are
re-parented to it; or
- the raw copy's QN is already held by a raw definition of another label
(coreclr inc/holder.h: the raw parse reads a FORCEINLINE constructor as
a Function of the class's scope-less QN): then the expanded copy keeps
its own, correct QN instead of vanishing into the Function's node.
Proof (macOS arm64):
- randstruct_kunit.c: 5 -> 0 duplicate groups, 12 Fields re-parented to
the flat structs, CALLS/USAGE/WRITES unchanged.
- dotnet/runtime native subset (src/coreclr + src/native): 700 -> 183
groups; the remaining groups are no raw/expanded pairs at one site.
- redis: node and edge sets byte-identical.
- New tests (RED without the change): extraction
extract_c_misnested_raw_struct_replaced_by_flat_copy,
extract_cpp_preprocessed_rescue_skips_raw_def_on_same_line,
extract_cpp_rescue_keeps_expanded_class_when_raw_qn_is_taken (fixture
delta-minimized from holder.h).
- conditional_variants, extraction, c_lsp, pipeline, edge_types_probe,
parse_coverage, lsp_resolution_probe: 1854 passed.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
42a93202 gave a non-static C-family function under a platform directory
(arch/x86/, src/win/, ...) or file suffix a platform-neutral QN, so the
platform variants of one function shared a node. On the Linux kernel that
merged only 644 of 17,413 such definitions; the other 16,769 lost their
path for nothing (`arch/arm64/mm/init.c` mem_init became
`arch.mm.init.mem_init`, and a lookup by its path QN found nothing),
arches missing from the token list kept their paths, and calls bound to
merged nodes produced impossible arch -> other-arch CALLS.
Every definition keeps its path-derived QN again (the v0.11.0 identity).
The platform-neutral QN is now a key: definitions carrying one get a
`variant_key` node property, and nodes of a project sharing a key are one
variant group. The test-impact walk (store_impact `follow_variant_keys`,
enabled by the test-impact engine) treats a reached variant as reaching its
whole group, at seeding and at every step, so a change confined to the
Windows variant still selects the tests that reach the POSIX variant's
callers. The key is a pure function of a definition's own path, name and
linkage: incremental re-parsing of the changed file writes what a full
index writes. Same-QN variants (#if/#else, Rust cfg, Go build tags,
.c/.h pairs) and their `variants` property are unchanged. No new edge type.
Proof (macOS arm64):
- Linux kernel subset (arch, include, kernel, mm, lib; 27,005 files):
arch functions whose QN is not under their own path 12,873 -> 0;
arch/A -> arch/B CALLS 10,353 -> 7,101; 581 variant groups (1,815
members), each one of the former cross-file merges; the path QN of
arch/arm64's mem_init resolves.
- This repository: 13 groups (test_impact_tree_{posix,windows},
mem_override_{posix,win}); before, 22 of 49 platform-path functions had a
stripped QN.
- New/rewritten tests (RED on the old tree or without the walk
expansion): conditional_variants cv_c_platform_files,
cv_c_platform_directories, cv_incremental_platform_files;
test_impact_engine
test_impact_engine_platform_variants_reach_each_others_callers;
store_impact impact_walk_crosses_platform_variants_only_when_asked;
cv_incremental_variant_partners re-fixtured to Go build tags.
- 18 affected suites: 2710 passed, 4 Windows-only skips.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
doclink_rst.c sorted d.toks unconditionally; the vector stays NULL until
the first token, and qsort's base is declared nonnull even for zero
elements. GCC UBSan on Linux amd64 reported it in rst_section_text_cut and
diagrams_rst ("null pointer passed as argument 1, which is declared to
never be null", doclink_rst.c:1767) -- the same class as the k8s pass fix.
Sort only when there are at least two tokens, as the file already does for
its section titles.
Proof: doc_links_rst on Linux amd64 (GCC, ASan+UBSan) reports the
violation without the change and none with it; doc_links_rst + diagrams on
macOS: 27 passed.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The preprocessed second pass rebuilt the expanded source with simplecpp's
TokenList::stringify(), which puts one space between every pair of tokens:
`template<> struct bitonic<int64_t, AVX2>` became
`bitonic < int64_t , AVX2 >`. Every QN builder takes the class segment from
node text, so a definition the #961 rescue re-adopted from that buffer
(vxsort's INLINE/NOINLINE macros break the raw parse on every method line)
got a second, token-spaced QN, and the QN-equality dedup kept both. CALLS
split across the two spellings (dotnet/runtime: 902 compact + 996 spaced
into the bitonic sorters).
The expanded buffer now keeps the source spacing: tokens that touched in
the source (same file and line, no macro expansion between them) stay
touching, every other gap is still one space, and the line structure and
#line directives are unchanged, so every line remap holds. One fix point
instead of a normaliser at each QN builder; no vendored code changes.
Proof (macOS arm64, dotnet/runtime 60108ba6, two identical runs each):
- duplicate definition groups 706 -> 217 (v0.11.0: 545); nodes with a
token-spaced QN 495 -> 0; CALLS into bitonic 902 + 996 -> 998, one
spelling. The remaining groups are coreclr gc.cpp's SVR/WKS builds (184)
and 33 scope disagreements between the two parses.
- Side effect: rapidjson document.h's GenericValue members now exist once,
with the raw parse's scope (the token-spaced copy that carried the
nested scope is gone); `operator[]` / `operator()` in the expanded pass
now parse as single tokens.
- New test (RED without the change): extraction
extract_cpp_template_specialization_preprocessed_qn_single_spelling.
Three c_lsp collision fixtures recalibrated to the new byte offsets
(comment padding and offsets only, assertions unchanged).
- extraction, c_lsp, pipeline, edge_types_probe, parse_coverage,
conditional_variants, lsp_resolution_probe: 1852 passed.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
#2513 (77bc4927) computes `url_context` -- is the callee an HTTP client or a
global fetch? -- for every positional argument of every call, in every
language, through cbm_service_pattern_match: a linear strstr over every
service pattern. Only the URL-folding functions read it, and they return
NULL outside Python and JS/TS; the one other reader is the template-string
branch. On the C# corpus (dotnet/runtime, 451k functions) extraction went
from 24.7 s to 37.4 s (+51 %); a macOS sample put the difference in
handle_calls -> cbm_service_pattern_match -> match_qn -> strstr.
A per-call url_context_t now computes the answer at most once and only when
an argument needs it: the folding paths ask only in folding languages, the
template-string branch asks for itself.
Proof (macOS arm64, Apple M5 Pro):
- C# parallel_extract 37.4 s -> 24.2 s, wall 127-132 s -> 119.7 s (the
pre-#2513 level, 24.7 s / 121.9 s).
- Byte-identical graphs: the full node and edge sets (properties included)
hash the same before and after on django (Python), typescript (TS) and
csharp, HTTP_CALLS included (457 / 283 / 39).
- extraction, extraction_imports, pipeline, parallel, infrascan and
edge_types_probe: 1009 passed.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The C# and Java Tier-2 registry builders register types, finalize, then
register the rest from `defs + type_count`. With no definitions of that
language the array is NULL, and NULL + 0 is undefined: clang's UBSan
reports "applying zero offset to null pointer" at cs_lsp.c:3886 on the
macOS legs of dry run 38321896354 (reached by
incremental_k8s_edits_reach_other_files, a project without C# files).
The second registration now runs only when there is something to register;
both finalize calls are unchanged.
cs_lsp, cs_lsp_bench, java_lsp, java_lsp_coverage, lsp_resolution_probe:
529 passed (macOS). The report reproduces on CI's macOS clang, not on the
local toolchain, so CI's macOS legs are the check.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A call no project symbol resolves, whose callee an import binds to an
external library, is now spelled through that import and classified with
the existing service patterns: `send` under
`from paho.mqtt import publish as send` is paho.mqtt.publish (ASYNC_CALLS
to its topic), `navigate` under `import { get as navigate } from 'axios'`
is axios.get (a typed GET HTTP_CALLS edge). Before, such calls had no
service edge at all.
- CBMImport gains `member`: the exported name an ES named import binds
(`get` in `{ get as navigate }`), NULL for default, namespace and other
imports and for every other language. Filled by process_named_imports,
carried through result compaction (and so through spill), counted in the
extraction census with the other import strings.
- One helper, cbm_pipeline_external_import_service, serves both call
resolvers: a per-file index (local name -> binding import), built on the
first unresolved call that asks, then one hash lookup per call.
- Precision: only unresolved calls, after the callee-name, route-suffix and
global-fetch fallbacks; never a relative import, an import an IMPORTS
edge materialized (a project module), a name bound by two different
imports, or a Python callee rebound as a parameter. An HTTP edge needs a
request verb in the spelled name, so urllib.parse.urljoin is not a
request. Python, JavaScript, TypeScript and TSX.
Tests: extraction_imports covers the member and its survival through
compaction; a pipeline test runs both resolvers over a typed GET, a typed
POST and a publish through aliases, and over controls that must stay
silent (urljoin, lodash and json aliases, a local definition and a
parameter shadowing the import, a materialized project package named
httpx). Each guard was reverted once and its control went red.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Refs #2291, #706, #1147. Preserve raw argument values and carry URL projections separately through compaction and spill.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Preserve registry array order and full-scan fallback when optional query
storage is unavailable. Keep temporary range storage scoped to each module
operation and shared type indexes in the registry owner arena.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
lookup_func_self guards a NULL qualified_name in its indexed bucket scan
and its post-finalize tail scan, but not in the linear scan an
unfinalized registry takes. Iterating a fallback chain checks every base
entry for shadowing against the head, so a head holding a func without a
qualified name crashed in strcmp (ASan SEGV in
pylsp_prefix_order_matches_iterator_and_chain_shadowing). The linear
scan now skips such entries like the other two.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Two gaps let a call on a vendor class escape the #1186 evidence:
- A vendor class the PHP LSP knows from its seeded library data (for
example GuzzleHttp.Client) resolved as php_static_resolved or
php_method_typed. Its target has no graph node either, but without the
php_external_receiver strategy the pipeline fell through to a
name-only registry binding. Methods found on a class reached through a
vendor `use` now carry php_external_receiver as well.
- A vendor call skips the registry, and the extractor names a scoped
call by its method alone (only a literal `Route::` scope is kept), so
`GuzzleClient::get('/x')` reached the route-verb fallback and became a
route registration instead of an HTTP call. When the callee text names
no service, the service patterns now also see the vendor method the
LSP proved (`GuzzleHttp.Client.get`, `Kafka.Producer.publish`), in
both the sequential and the parallel pass.
Refs #1186
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
collect_use_declaration dropped the group prefix of `use A\B\{C as D}`
and read the function/const keyword only from the declaration, so a
grouped alias mapped to the bare `C` with the declaration's kind. The
cross-file LSP then could not recognise a grouped vendor class import as
external, and a grouped `function`/`const` alias was registered as a
class. Prefix each grouped clause with the group namespace and take the
keyword from the clause when the declaration has none, mirroring
emit_php_use_decl in extract_imports.c.
Refs #1186
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Reuse a stable arena-owned function QN order for ambiguity marking and
submodule prefix lookup, preserving exact lookup policy and appended tails.
Add prefix/index failure regressions and a pipeline operation-count case.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The __all__ literal guard compared a uint32_t child count against
SIZE_MAX / sizeof(char *), which is always false on LP64 and failed the
GCC build (-Werror=type-limits). One bound now covers both limits on
every platform: the names are counted in an int, and
count * sizeof(*names) <= INT_MAX also fits a 32-bit size_t.
Building one import fact moves into py_ns_import_fact, and the
module-name child is skipped with ts_node_eq: for a plain import the
module is the null node, which equals no child. The loop no longer tests
`from` after the from-only prelude, which cppcheck 2.20 misread as
always false (knownConditionTrueFalse). Emitted facts are unchanged.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Capture ordered Python namespace facts and retain them through compact, spill,
and versioned LSP surfaces. Reject unavailable metadata before incremental
reuse and rebuild on Python namespace changes or deletion.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Resolve symbols declared in __init__.py under their package while retaining
the raw module identity for file and top-level caller nodes. Carry explicit
language through registry callers, select Python package imports, preserve
long exact names, and report scope allocation failures without partial output.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Preserve the arena-owned raw Module caller name so top-level resolutions
join the extractor in both pipeline drivers. Add occurrence and ownership
controls, including package modules and named lambdas.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The depth guard of the C/C++ and Python expression evaluators
(C_EVAL_DEPTH_LIMIT, PY_LSP_MAX_EVAL_DEPTH = 256) now reports a sticky
per-file failure that stops the walk and drops every LSP resolution of the
file. Both evaluators recursed once per chain link, so a 300-term sum or a
C++ fluent chain of ~130 calls counted as 256 levels of nesting: one long
expression cost its whole file every LSP call edge. A chain's length is not
nesting, and the guard must not decide graph content for it.
c_eval_expr_type and py_eval_expr_type now first walk down the chain
operands below the node (the operand the evaluator evaluates first and
derives the link's type from), then evaluate them innermost first at the
caller's depth, keeping pending links on a small arena-owned stack in the
context that nested evaluations share. When a link is evaluated, its chain
operand is already memoized, so the evaluator's own recursive call into it
is a memo hit one frame deep. Every node is still evaluated by the
unchanged inner evaluator, evaluation does not mutate scope within one
outermost call, and only operands the recursion itself evaluates are
pre-evaluated, so the types are exactly the recursive ones.
No longer counted as depth (any length):
- C/C++: binary operators other than == != < > <= >= && || (those yield
bool without evaluating an operand), member access (. and ->), calls on
a non-name callee, subscripts, conditional consequences, parentheses,
comma lists.
- Python: binary operators, attribute access, calls on obj.m(...), f()()
and (expr)(), subscripts, the else branch of a conditional, parentheses.
Still counted, unchanged guard and reporting (genuine nesting):
- C/C++: call arguments used for template argument deduction,
std::move/std::forward operands, unary, pointer and update operands,
assignment left sides, lambda return expressions, co_await, fold and
_Generic operands.
- Python: tuple/list/set elements, dictionary keys and values, the true
branch of a conditional, await operands, next()/iter()/assert_type()
arguments, called lambdas.
Proof:
- New c_lsp and py_lsp tests build chains of 300, 2,000 and 20,000 terms
(a binary + chain and a .a() receiver chain) next to an unrelated s.m().
On the previous evaluator, and with this change reverted, all four fail
at 300 terms: the depth error is recorded and the file resolves 0 calls
(C++ per-file; Python per-file and through the cross-file dispatch).
With it, every size records no error, resolves run -> S.m, and the
chain's result type resolves x.done() / the final .done().
- Real input, isolated cache, before and after:
redis 4f20cb48 (C): 36,876 CALLS, 10,699 LSP-resolved;
django fb113765 (Python): 60,123 CALLS, 25,153 LSP-resolved.
Identical CALLS edge sets on both (0 lost, 0 gained, 0 strategy
changes), 0 files with an LSP error before and after, wall time
unchanged (redis 5.5 s, django 9.1-10.2 s both ways).
Refs #1527
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Memoize C expression evaluation, index repeated scope and registry queries,
and report incomplete walks explicitly on memo or depth failure.
Refs #1527
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Martin Vogel