py_lsp.c: #2697 moved py_lookup_attribute below the C3 code (own members, C3 order, then the depth-first walk) and declares it up front; the trivial definition this branch still carried next to its module-member lookup is dropped, the module-member block is kept, and the C3 section takes main's header.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
py_lsp.c: the module-member lookup of this branch and the C3 super() lookup of #2696 were added at the same place; both are kept.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The memory-core linter counts raw allocator calls per file and fails when a
file grows. The TYPE_CHECKING change freed the copied local name of an
annotation-only import with a new raw free in two import-map builders.
build_import_map now asks the predicate on the name's span in the edge
properties before copying it (cbm_imports_local_span_binds_runtime_name,
which cbm_imports_local_binds_runtime_name now wraps), and the cross-LSP
builder routes an annotation-only name through its existing no-value path.
Same behaviour, no new raw allocator site.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A namespace fact stream is checked by cbm_py_namespace_facts_valid, which
accepts a shadow only for the value, decorated, unproven-class and
unsupported reasons. The withdrawn TYPE_CHECKING re-export carried the
compound reason, so the whole stream was rejected and the package published
no names at all (a lazy-module package lost its TYPE_CHECKING exports). It
now carries the unsupported reason: the builder cannot tell which source the
name has at runtime. python_namespace_facts_type_checking_alternatives pins
both shapes as complete, valid streams.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A package namespace read in the type-checker view takes the consequence of
`if TYPE_CHECKING:` as executed (lazy-module packages publish their names
there). The compat shape imports a name there and again in `else: try:
from runtime.place import name`: the module's runtime export comes from the
else branch, so publishing the TYPE_CHECKING source as the re-export target
was a guess. Every name an elif/else import binds now becomes a shadow fact
(the re-export is withdrawn, as for a later assignment); a star import or an
import form the builder does not read there makes the whole namespace
unknown. The branch is walked iteratively.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
cbm_run_php_lsp never finalized its type registry, so every lookup during
single-file resolution scanned the stdlib plus every definition of the
file linearly: O(lookups x definitions), quadratic in the size of one
file. The cross-file path already indexes its registry.
The registry is now finalized with first-registered semantics (the first
registration of a duplicate QN wins), which keeps exactly the answers of
the linear scan, and re-indexed once after trait flattening adds methods.
The index lives in a scratch arena that dies with the call.
A test-only work ceiling (CBM_LSP_WORK_CEILING_HIT, seam builds only) lets
the work-counter test stop a reverted, unbounded loop deterministically
instead of hanging the runner; product builds have no ceiling.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A package `__init__.py` re-exports its submodules' names (`from .fields
import *`, `from .x import Orig as Renamed`), and callers reach them as
module attributes (`models.CharField()`) or through `from pkg import Name`.
The Python namespace facts already capture each module's top-level bindings
in order; nothing consumed them, so those calls fell to name guesses that
pick the first same-named definition under the package.
- Records: cbm_pxc_collect_all_defs replays each module's COMPLETE fact
stream into re-export, star-import and `__all__` records next to its
defs, including for an import-only `__init__.py` (zero defs). Records
carry the raw module spec; the Python registry resolves it (relative
exactly, absolute by unique scope tail, else unknown).
- Fold: the shared Python registry folds the records once, after finalize
and before the seal, into a sorted per-module export table (binary
search; no lookup in the registration loop). Order-deterministic; a
cycle contributes what it has so far.
- States (conservative): an incomplete fact stream publishes nothing; a
shadow, delete or whole-namespace unknown (a top-level try/if/with)
withdraws the names it covers; a star whose source cannot be enumerated
publishes nothing; a literal `__all__` restricts a star, else public
names only.
- Consult: the Python LSP looks a module attribute up in the export table
on a miss (calls, expression types, callable values) and when binding
`from pkg import Y` where Y is no def of pkg.
- Namespace facts read a module-level `if TYPE_CHECKING:` consequence as
executed: lazy-module packages publish their names there.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The compat shape imports a name under `if TYPE_CHECKING:` and again in the
else branch, usually inside try/except ImportError. The name is bound at
runtime there, so its TYPE_CHECKING import is no annotation-only import:
recording it flagged made every CALLS import map skip the name and lost the
calls and bases that bind through the runtime import (airflow mask_secret,
OpenLineageRedactor(SecretsMasker)). The imports of the elif/else branches
are collected (any depth, iteratively), every TYPE_CHECKING import whose
local name they bind is dropped with them, and a star import there drops the
whole block: those names keep the behaviour they had before #1277.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The import an annotated field's type comes from usually sits under a
module-level `if TYPE_CHECKING:` (to avoid an import cycle).
parse_python_imports walked only the module's direct statements, so that
import was never recorded and the reporter's chain
trainer.strategies.process_batch() lost its type exactly where real code
writes it.
- Extraction records the imports of a module-level `if TYPE_CHECKING:` /
`if <module>.TYPE_CHECKING:` consequence block, flagged
CBMImport.type_checking_only. elif/else branches stay unrecorded, as
every other compound statement's imports do.
- One rule for every CALLS import-map builder:
cbm_import_binds_runtime_name() (false for a flagged import). The
sequential build_import_map applies it in its fast and slow paths; the
shared cbm_pxc_build_import_map (= _ex(..., false)) applies it in both
loops, so no CALLS import map ever binds an annotation-only name.
- The annotation consumers take cbm_pxc_build_import_map_ex(..., true):
the cross-file base/field-type resolution and the Python LSP dispatch.
The sequential dispatch always takes it; the parallel one builds that
map only for a file that records a flagged import and otherwise passes
its CALLS map, which is identical there by construction.
- The Python LSP binds those names into an annotation-only frame consulted
by annotation resolution after the ordinary scope chain: a runtime call
through such a name never resolves.
Tests: python_type_checking_imports_flagged_issue1277 (extraction),
python_type_checking_typed_field_issue1277(_parallel) (pipeline, both
venues; controls: no import_map and no LSP CALLS edge through the name).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The second half of #1277: the reporter's typed-field chain with both
annotation imports under `if TYPE_CHECKING:` (the usual shape when the
import would cycle), through `typing.TYPE_CHECKING` too. The call resolves
by the LSP as with runtime imports, and a runtime call through a
TYPE_CHECKING-only name gets no CALLS edge at all. The extraction test reads
the new CBMImport.type_checking_only flag, declared here and not yet set.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Ordinary attribute lookup walked the bases one at a time, each through its
whole ancestry, so on a diamond self.m() and e.m() reached the shared base
before a later base that overrides it. It now follows the class's C3 order,
the order super() uses, after the class's own members; a hierarchy without
a C3 order keeps the depth-first walk.
Each class is linearized once per file: the orders live in a per-file,
arena-allocated open-addressing cache keyed by class QN, which also replaces
the per-lookup cache of the super() resolver. The registry is complete
before the walk starts, so an entry never goes stale; keys and the class
entry of an order are arena copies. Linearizing a class costs one merge of
its bases' cached orders, so no lookup re-linearizes a deep hierarchy.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
super() binds to the enclosing class's method resolution order, Python's C3
linearization, in which a common base follows every class deriving from it.
The super() resolver walked the bases one by one, each through its whole
ancestry, so with RadarrCalendarEntity(RadarrEntity, CalendarEntity) the
first base reached Entity through CoordinatorEntity before CalendarEntity,
which overrides it, was ever looked at.
py_c3_linearize computes the order (memoized per lookup, capped in depth and
length; -1 when the hierarchy has none), and super() takes the first class
after the enclosing one that declares the member. A hierarchy without a C3
order keeps the base-by-base walk, as does the synthesized __init__ of an
undeclared constructor.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
py_process_class qualified every class under the module, so a class nested
in a class (proj.m.Outer.Inner, the QN extraction gives it) was walked as
proj.m.Inner: its methods' LSP rows carried callers no node has and never
joined. A typed-parameter call in such a method got no edge at all, a self
call fell to unique_name and a constructor to same_module. A nested class,
decorated or not, is now walked under its enclosing class through the same
py_process_class_in the function-local classes use.
Tests: pipeline_python_nested_class_methods_resolve(_parallel).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
push_nested_class_nodes descended into field/template/declaration and conditional wrappers but not into Python's decorated_definition, so a decorated nested class (`@dataclass class Meta:` inside a class) was never extracted: no Class node, no Method nodes, and the calls in it fell to the FILE node.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
cppcheck 2.20 reports the post-cut `if (!name[0])` as always false
(knownConditionTrueFalse): it cannot see that writing through `lt`
may empty `name`. The name is non-empty here, so the cut empties it
only when `<` is the first character. Test exactly that before the
cut. The behaviour is unchanged.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The cross-file Kotlin walk resolved a call on a typed method parameter by
type, but a call on a class property - a primary-constructor val, a body
property or `this.prop` - fell to the name guess. The cross registry held
each own-file class from its extraction defs, whose properties carry no type
(constructor vals none at all), and the walk's typed AST registration of the
same class was added behind that entry under the same QN, so every lookup
returned the field-less one. Constructor injection is the common Kotlin
dependency shape, so `repository.inserir(...)` on an injected port bound to
whichever same-named method the guess found, including an implementation of
another port (#1555).
The walk now registers the file's own declarations into an overlay chained
to the cross registry and checked first, indexed once the top-level
declarations are collected. The sole-definer fallback walks the whole chain
so its candidate set is unchanged.
Typing more receivers exposed resolution gaps that the untyped receivers had
hidden; each would otherwise turn a typed call into a wrong name guess:
- a member of the receiver class wins over a same-named extension of the
same file (Kotlin resolution order; the overlay holds both);
- `x.hashCode()` / `toString()` / `equals()` on a receiver typed by another
file: the per-file pass emits the kotlin.Any placeholder, the cross pass
the declared override, and two targets at one occurrence are ambiguous to
the call join. The per-file placeholder is dropped when the cross pass
resolves the same occurrence;
- the same members on a stdlib or JDK type (`count.toString()` on an Int)
resolve to the kotlin.Any builtin instead of a seeded stdlib member that
has no graph node;
- a property READ on a typed receiver whose QN the graph spells differently
(a nested class's property) never becomes a CALL_REFERENCE to a
same-named function: the reference join falls back by name to
non-callable targets only, which binds the property's own node;
- a typed value without an `invoke` operator (`private val handshake:
Handshake?`) no longer shadows the same-named member function;
- inside a lambda whose typed receiver becomes `this` (`repo.let { .. }`),
and inside a member extension, the enclosing class's members stay
callable bare (the implicit-receiver chain); a call naming a class stays
a constructor call.
Fixes#1555
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A package-relative from-import (`from . import x`, `from .. import x`,
`from . import x as y`) has a module made only of dots. The extractor joined
it to the imported name with another '.', recording "..x" for `from . import x`:
the parent package. The import resolver then missed the real binding and fell
to the symbol-name guess (the lexicographically first same-named definition in
the project), so the import map, and the Python LSP binding that consumes it,
sent every call to an unrelated function at import_map / lsp_callable_alias
confidence. On home-assistant, 7,175 caller/callee pairs bound to another
integration's function (webhook's `async_register` to accuweather's).
Join without the extra separator when the module already ends in '.'.
`from .m import f` (".m.f") and absolute imports are unchanged.
Tests: python_package_relative_import_paths (extraction: the recorded paths);
pipeline_python_package_relative_imports{,_parallel} (all three forms resolve
to the own / parent package, never to the namesake that sorts first, without a
name guess, on both resolvers); pipeline_python_from_import_bare_call_issue1237
{,_parallel} (#1237 guard: a plain from-import and bare calls bind the imported
def and keep it across a re-index that adds a file).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The extractor fix above names a generic impl's methods `Gen.in_generic`
on both the def and the call-scope side. The Rust LSP still built its
self type from the impl's raw text, `Gen<T>`. That broke three things:
- its caller QN (`Gen<T>.in_generic`) no longer joined the pipeline's
call scope, so every LSP row inside a generic or lifetime impl was
dropped for a registry guess. On 9 Rust repos that turned 1,927
lsp_* edges into suffix_match/unique_name;
- `self.` and `Self::` dispatch never found the impl's own methods;
`Self::assoc_g()` had no edge at all, on main as well;
- the return-type harvest never matched those methods, so chained
calls on them stayed untyped.
Strip the generic arguments from the impl's self type in both places.
Text that starts with `<` (a qualified path) is left alone.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
`Opts::write(true)` is an associated function. When the Rust LSP cannot
place it, the registry can (import_map or qualified_suffix onto
fsx.Opts.write). The two free-function short-name fallbacks (the
cross-crate member lookup and the project-wide lsp_short_name_unique)
ran first, though. They bound the call to the project's only free
`write`, and that LSP row pre-empted the right answer.
While the Rust registry also held other languages' definitions, a
same-named JS or Perl function often made the free name ambiguous and
hid this. With Rust-only registries the hijack replaced 516
qualified_suffix and 84 import_map edges on 7 Rust repos.
A call whose qualifying segment names a type now skips both fallbacks.
That covers an ASCII-uppercase initial, `Self`, and a qualified
`<T as Tr>::f` path.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The def side names a Rust impl method after the implementing type with
its generic arguments stripped (`impl<T> Gen<T>` -> `Gen`, extract_rust_impl).
The call-scope QN in extract_unified.c used the impl's type text verbatim,
`Gen<T>` / `Life<'a>`. So every method of a generic or lifetime-parameterised
impl had a scope QN that no node carries, and every call in its body was
sourced from the File node instead of the method. Base census, 9f85f104 par
(lane L7's #763 measurement): 87,855 Rust CALLS start at a File/Module node,
84,331 of them (96%) on a line inside a function span. Go is 0%.
Strip the generic arguments on the call side exactly as the def side does.
Plain impls, trait impls, free functions and inline modules were already
consistent (checked on a fixture with the base binary).
Test: extract_rust_generic_impl_call_scope_matches_def (plain, type-generic
and lifetime-generic impls; def QN == call scope).
Refs #763
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Both Rust registries were built from the pipeline's mixed-language def
set. The shared Tier-2 build (cbm_rust_build_cross_registry) and the
per-file conversion (pxc_lspdefs_to_rust) copied every def, so a Perl
`sub new` or any C function became a Rust free function. The
lsp_short_name_unique fallback then bound `Store::new()` to the Perl
constructor at confidence 0.90 on both venues: its "crate scope" check
compares only the first module-QN segment, which is the project name.
(Found by the generic-engine lane on a 12-language fixture.)
Register Rust defs only, at both conversion points, exactly as the Go
builder filters to Go. Rust impl-relation records already carry
CBM_LANG_RUST. As a side effect, the shared Rust build no longer
registers the C defs of a mixed C+Rust tree.
Tests: pipeline_rust_call_never_binds_another_language (sequential and
parallel): no CALLS from the Rust caller into the Perl file, and the
Rust associated function still binds.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Re-port onto current main (agreed with lane L7):
- The Python LSP binds every def/class a function body defines as a local
before walking the body (LEGB): an undecorated def as an exact callable
alias to its nested QN, anything else as a hard shadow, so no call falls
through to a same-named module symbol. Defs inside the body's compound
statements count too, and their bodies are now walked.
- Classes declared in a function are walked under the function's QN
(py_process_class_in), so super()/self calls in their methods resolve.
- cbm_registry_resolve_lexical moves AFTER the LSP override: both resolvers
call one entry point, cbm_registry_resolve_call_scoped, at the same place
(before the language-aware chain). It never uses the per-file resolve cache.
- Main's cbm_registry_add_lang carries the def-local exclusion, so every
registration site (full, parallel, incremental seed) applies it; the
separate cbm_registry_add_for_language and the second incremental language
derivation are dropped.
- The pass_pkgmap.c def-local import skip is its own continue.
Tests: local-vs-module shadowing in both orders, a decorated local, and a
function-local class's super()/self calls, sequential and parallel.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A controller routed with ASP.NET Core attributes produced no Route node
and no HANDLES edge:
[Route("api/[controller]")] class QuotesController
[HttpGet("{symbol}")] -> nothing
[HttpPost] + [Route("orders/{id}/confirm")] -> nothing
Root cause: the route extractor only knew decorator calls (Flask, FastAPI,
Express-style) and Spring/JAX-RS annotation names. HttpGet/HttpPost/.../
Route matched none of them, the class-prefix join ran only for Java and
Kotlin, and ASP.NET templates are relative ("{symbol}", "api/[controller]"),
while the path filter and ensure_one_decorator_route accept only
'/'-prefixed paths. #1759 made every stacked C# attribute visible as
DECORATES but did not turn any of them into a route.
Fix: a C# path in push_method_def, next to the JVM prefix join, that
reads the attributes of the action and of its class:
- HttpGet/Post/Put/Delete/Patch/Head/Options with or without a template,
[Route] on the class and on the action, [AcceptVerbs("GET", Route = "x")];
qualified names and the "Attribute" suffix are accepted.
- [controller] is the class name minus "Controller", [action] the method
name minus "Async" (the ASP.NET Core default) or its [ActionName], and
[area] comes from [Area].
- A template is relative to the class prefix; "/" or "~/" overrides it.
- Constraints, defaults and optional markers reduce to the bare parameter
("{id:int}", "{code:regex(^\d{{3}}$)}", "{page=1}" -> "{id}", "{code}",
"{page}"), so the path is '/'-led and canonicalizes like every other
decorator route. This works with both the current cbm_route_canon_path
and the depth-matched variant.
An action without a verb or [Route] attribute stays route-less, and other
C# attributes never become routes.
Known limits, unchanged in kind from the JVM path: one route per action
(a stacked [HttpGet] + [HttpHead] records the first), and a [Route]
inherited from a base controller in another file is not applied.
Proof on jellyfin/jellyfin (shallow clone, depth 1, 208c278): Route
nodes 0 -> 360, HANDLES 0 -> 388, e.g.
GET /Items/{itemId}/Images/{imageType} <- ImageController.GetItemImage.
Tests (edge_types_probe, exact Route-name sets): the #1692 controller,
the #1041 AuthController, [controller]/[action]/[ActionName]/[AcceptVerbs]
tokens, "/" and "~/" overrides with constraints plus a controller without
a class [Route], and a non-controller C# control.
Refs #1692
Refs #1041
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Why
The Python def walk stopped at every function_definition: it extracted
the def and never looked at its body. Every def or class declared inside
a function -- closures, decorator wrappers, the route handlers of a Flask
or FastAPI app factory (`@app.route` inside create_app), local classes and
their methods -- was absent from the graph. Blocks at module level (if /
try / for / with) were transparent, so only function bodies were blind.
The calls inside those defs were not re-attributed either: the call-scope
walk named a nested def by its class or module only (`pkg.mod.nested`, a
QN no node had), so the edges fell back to the file node, and inbound
traces of the called function silently lost the caller. A def nested in a
method got the method's own QN (`Tree.visit` for the `def visit` inside
method Tree.visit), so its calls, recursion included, landed on the
method. On an app-factory codebase this removes the whole HTTP layer.
Fix
- Extraction (extract_defs.c): after a Python def is extracted, its body
is walked with the def's QN as the enclosing scope; methods' bodies are
walked under the method QN. A nested def or class is qualified by the
def it sits in -- pkg.mod.create_app.index, pkg.mod.Tree.visit.visit,
pkg.mod.create_app.Settings.value -- the QN the Python LSP already gives
the calls inside it (py_process_function). Decorated nested defs keep
their decorators, so their route_path mints the Route and HANDLES.
Lambdas and comprehensions hold no statements and stay as they are.
- Call scope (extract_unified.c): for Python the nearest named scope, a
def or a class, prefixes a nested def or class, so every call, usage and
env read inside a nested def carries the nested def's QN.
- A def nested in a function is a local of it (Python LEGB), so it must
not become a global name: a Python symbol whose QN extends a
Function/Method node is registered for exact lookups only, never in the
short-name index (registry.c cbm_registry_add_for_language, used by the
full, parallel and incremental registration sites alike). Without this,
django's per-test `class MyModelAdmin` and every decorator's `def
wrapper` became unique_name / suffix_match targets for unrelated calls.
The incremental seed derives each stored file's language the way
discovery does (name, then shebang), so it keeps the same symbols out.
The import symbol-name fallback skips such locals too (pass_pkgmap.c),
and the Python LSP no longer binds a def-local class module-wide by its
short name (py_lsp.c).
- A bare call now resolves lexically first (registry.c
cbm_registry_resolve_lexical, strategy lexical_scope): the def or class
bound in the nearest enclosing function, skipping class bodies, stopping
at the module. The sequential and parallel call passes use it
identically, and so does the handler lookup of a call-registered route:
`app.add_url_rule("/ping", "ping", ping)` in create_app reaches the
local ping.
Proof (real code; every node and edge diffed, main 1f9b4db1 vs this)
- pallets/flask examples/tutorial (flaskr, create_app factory): the
nested `hello` handler is now create_app.hello with HANDLES to
/hello; 4 nested defs/classes added, 0 nodes and 0 edges removed.
- pallets/flask d73fa1c (tests declare handlers inside test functions):
Function 532 -> 1068, Method 318 -> 381, Class 130 -> 209. The +678
equals the 678 unique nested def/class QNs Python's own ast finds.
Handler HANDLES (decorators and add_url_rule) 32 -> 296. 0 nodes
removed, 0 duplicate QNs.
The 94 removed CALLS: 90 moved from the file node onto the nested def
that makes the call, 4 replaced by the lexically bound local.
- django fb11376 (local, 2,927 .py files): nodes 55,483 -> 60,052,
Function 1,992 -> 3,027, Method 28,850 -> 29,491, Class 8,187 ->
11,073. 0 nodes removed and 0 duplicate QNs. Every added
Function/Method/Class sits under a Function/Method of the same file,
and they match ast's 4,567 nested QNs except for 5 methods declared in
an `if` inside a local class body (class-body `if` is not walked for
any class). CALLS 62,430 -> 63,815. Every added CALLS edge has a nested
def at one end: 1,015 from nested defs, 1,219 lexical_scope into a
local, and none from outside the local's function. Of the 849 removed:
507 moved from the file node onto the nested def, 334 replaced by the
local the name binds, and 8 were false on main, all checked (e.g. the
recursive `get_actions` nested in call_command had been suffix-matched
to ModelAdmin.get_actions). HANDLES unchanged (135).
- Samples checked against the source:
AbstractBaseUser.check_password.setter -> set_password / save;
View.as_view.view -> setup / dispatch;
Apps.lazy_model_operation -> lazy_model_operation.apply_next_model
(line 426);
InstrumentedASGIRequest.META -> the local InstrumentedDict (line 279);
flask test_provide_automatic_options_arg_disable.index -> GET /.
Tests
- extraction: extract_python_nested_def_qualified_by_enclosing_def,
_calls_attributed_to_it, _decorated_nested_route_handler,
_nested_class, _nested_def_named_like_its_method,
_lambda_and_comprehension_stay_in_enclosing_def (control),
_nested_def_env_access_attributed_to_it (all _issue1548).
- pipeline: pipeline_python_app_factory_routes, _nested_def_is_function_
local, _local_classes_stay_in_their_def, _import_never_binds_a_local_def,
_nested_defs_parallel (>= 50 files, CBM_WORKERS=4),
_nested_def_local_after_incremental (all _issue1548).
- registry: registry_python_function_local_is_exact_only,
registry_resolve_lexical_walks_enclosing_defs (both _issue1548).
The extraction and pipeline tests were RED on main 3/3 for the asserted
reason and are green with the fix; the registry tests cover the new
registry API. Each production piece was reverted on its own, and each
revert turns at least one of these tests RED.
Fixes#1548
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
(cherry picked from commit 62fda43fd66200879db62a58903a32dab0dd1a20)
Terraform references were resolved by the generic name matcher: by their
last name segment, across the whole repository. `format()` CALLS a
variable named format of another module, `local.x` meets a resource named
local, `.id` an output id anywhere. Measured on the 7 Terraform
repositories of 17 infrastructure corpora: 12,437 edges, recall 0.63 and
precision 0.39 against an independent reading of the references, and not
one of them left the block holding its reference.
A reference -- var.X, local.X, module.X, data.T.N, T.N -- now USAGEs
{ref, via: terraform, line}, from the top-level block holding it, the
block declaring it in the same module: the .tf / .tofu files of its
directory (a locals entry: its locals block); an entry of the block's
depends_on list DEPENDS_ON the object instead. The generic matcher makes
no edge for a .tf / .tofu file any more; a name only mentioned in string
text is no reference, nor is a block's reference to itself. A changed
file re-resolves its whole directory, so an unchanged file's reference
meets the declaration a change adds and loses the one it removes.
A bare block label (`variable region {}`, valid HCL) now names its block
like a quoted one; such blocks used to share the keyword's name and QN.
Measured: 7,771 of 7,771 file-level and 13,397 of 13,397 block-level
reference pairs (edge type included) and nothing else; the 10 other
corpora are unchanged; deterministic over three runs; the incremental
re-index equals a full index in 7 scenarios (rename a variable, a module,
a locals entry; delete the variables file; add a file; add a declaration
an unchanged file references; quote a bare label); index time unchanged
(within 1 % on the four largest Terraform corpora).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A Kubernetes object's metadata.labels were not in the graph: neither a
search for an object by label nor a kustomize patch whose target selects
by label could find anything.
A Resource now keeps its labels as `labels: "k=v,k2=v2"` (sorted; a
templated label, or one a selector cannot name, is left out), and a
patch target's labelSelector meets the objects of its build carrying its
terms -- `k=v` / `k==v`, `k!=v`, `k`, `!k`; a set-based term names no key
and meets nothing. A target with only a selector selects among every
kind; an annotationSelector is not evaluated and names nothing.
Measured on 17 infrastructure repositories: 1,956 Resources gain labels
and nothing else; an independent reading of metadata.labels agrees on
every one of the 1,804 objects both read (152 more are Go-templated,
which only the graph reads, through its skeleton). The 4 label-selector
patch pairs now resolve: kustomize patches 130 of 130.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A kustomization's `helmCharts` inflate Helm charts into its build -- the
fourth way a repository puts a chart on a cluster, beside a Flux
HelmRelease, a Terraform helm_release and a helm command; the graph read
none of them.
Each entry now DEPLOYS from the kustomization -- its Kustomization
Resource, else its File: the unit a Flux or Argo deploy of it reaches -- to
the shared external Chart of that name, with {via: helmcharts, external,
chart, url, version, release, namespace, values, line}. A chart name with
a path names nothing; the values file inside the repository is an IMPORTS
(local_name: valuesFile) of the kustomization.
Measured on 17 infrastructure repositories: 20 edges, every one in an
independent reading of the kustomizations (20 entries, cluster-api 10,
k8s.io 10); no other node or edge changes.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
A kustomize overlay changes its base through patches; the graph read none of
the v5 `patches:` entries (path / patch / target mappings), so a patch file
showed up as a second copy of the object it patches, with no edge to it.
Each patch of a kustomization now OVERRIDEs, child -> base, the objects of
its build it changes, with {via: patch, kustomization, patch, line}:
- a patch file's documents: the object of the same group, kind and name
(and namespace, when the document declares one);
- an inline patch (from the kustomization's File) or a JSON6902 operation
file (from its own File): what its `target` selects -- kind, group,
version, name, namespace; a label or annotation selector is not
evaluated and names nothing;
- in the build of the kustomization stating it; a component's in the build
of each kustomization including it.
A patch file is an IMPORTS (local_name: patches) of its kustomization and
never a member of its build: its documents select nothing, are selected by
nothing.
Resources now keep the version of their apiVersion (`version`), which the
identity leaves out and a target may name.
Each build indexes its own objects by name once: a lookup in the
repository-wide name index would visit every copy of a name for every
patch. The complexity fixture gains patch files (one of four documents), an
inline and a kind-only patch.
Measured on 17 infrastructure repositories: 126 edges, every one in an
independent reading of the 173 patches (122 resolvable in the
repositories); the 4 pairs without an edge use a label selector.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Martin Vogel