diff --git a/CHANGELOG.md b/CHANGELOG.md index 7152433d2..7f9ba642b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -204,6 +204,10 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - In React with TypeScript, a component that renders a generic component and passes it type arguments, like outline's ` items={…} />` or excalidraw's ` …>`, now links to the component it renders. Before, a tag followed by type arguments was skipped, so the generic component looked unused by the screens that render it, and `codegraph_explore`'s call paths, callers and impact stopped short of it. Re-index React projects after upgrading. - In React, a component that a file declares itself as a value is now what that file's JSX renders, whatever else in the repository shares its name. A lazily loaded page, like `const RegisterPage = Loadable(lazy(() => import('./pages/Register')))`, a `dynamic(() => import(…))` component or one loaded only in development (`IS_DEV ? lazy(…) : null`), links to the component its module exports, including the one its loader picks and one an `export default observer(Login)` wraps; a wrapped component like `const Settings = observer(function Settings() {…})`, `React.memo(Rows, areEqual)` or `forwardRef(ComboboxInner) as {…}` links to the function it wraps; and an alias like `const DialogPortal = DialogPrimitive.Portal` links to its own declaration. Before, `` was linked to any lone component of that name elsewhere, often in another app of the same repository, so `codegraph_explore`, callers and impact followed it into the wrong app while the page it really renders had no caller. React Router layouts declared this way follow the same rule, and a TSX type named like a value the file declares now stays in that file. Re-index React projects after upgrading. - In React, a JSX tag imported from one of your own modules now links to the component that module exports under that name, read the way a lazily loaded page's module already is. That includes going through an `index` file or barrel that passes the component on, like outline's `Sidebar/index.ts` with `import Sidebar from './App'; export default Sidebar` or its `Avatar/index.ts` with `const Avatar = AvatarWithHoverCard`, and past a wrapper such as `export default observer(AppSidebar)`, `React.memo(StaticCanvas, areEqual)` or mastodon's `const MemoizedBlurhash = memo(Blurhash)`. Before, a tag was matched by its name and the import only broke ties, so outline's `` linked to an unrelated styled `Sidebar` in a sibling file, a default import under another name linked to whatever other component its module exported, and an import through a barrel could land on another app's component of the same name. A tag whose module exports an unnamed component, like `export default () => …`, now links to nothing instead of to a namesake elsewhere. A lazily loaded React Router page exported as such a value or wrapper links to the component it holds too, the one with the markup. `codegraph sync` and the file watcher redraw these links when only the file that passes the component on changes. Re-index React projects after upgrading. +- Python calls through an attribute now reach the method your code gives that attribute. `self.sink.stop()` follows `self.sink = Sink()`, a `sink: Sink` annotation, or a typed `__init__` parameter to `Sink`'s `stop`, including a `stop` that `Sink` inherits, and an attribute of a typed local or parameter resolves the same way. An attribute holding a list, a dict or a class from another package, like `requests.Session()`, no longer links its calls to a same-named method of your own, and one whose type the code never names is matched by the attribute's own name, like `self.client` to a `Client` class, but no longer by the method's name alone. Re-index Python projects after upgrading. Thanks @inth3shadows. (#1704) +- Python attributes set by a factory or a base class now lead to the right method. `self.client.send()` resolves when the class gets `self.client` from `Client.from_env()` or `make_client()` — read from the factory's return type, or from what every `return` in it builds — and when a base class's `__init__` in another file sets it. Calls straight on a factory's result, like `Client.from_env().send()` or `Truncator(value).chars(n)`, resolve the same way. A factory whose returns build different classes, a generator, and a name rebound inside the function give no type, and an attribute only a base class from another package sets, like `self.client` in a Django app's tests, no longer links to a method of your own. `codegraph sync` re-checks these calls when you edit the factory or base class they depend on; on large projects, an edit to a module much of the project imports can leave a call that only now becomes resolvable for the next full index. Re-index to pick up the change. Thanks @inth3shadows. (#1704, #750) +- A call on a variable or attribute named after a class, like `client` or `nodelist`, now prefers the class it spells whole, `Client` or `NodeList`, over another that only shares a word with it, like `RedisCacheClient`. +- A call through an attribute of an imported value, like flask's `app.jinja_env.filters.keys()`, no longer links to the value's own `jinja_env`: the method called is `keys`. Re-index to update an existing project. - A call through a class name, like `Logger.log()`, or through a variable named after one, like `logger.log()`, now links to that class's own method or the one it inherits, such as Django's `form.is_valid()` reaching `BaseForm`, instead of the same-named method of a class whose name merely contains it, such as `FileLogger`. These calls also no longer link to a class a test declares inside one of its functions, or, from your own code, to a class in a test suite. Re-index to update an existing project. Thanks @inth3shadows. - Python imports now resolve the way Python finds modules: from your project's package roots — the repository root, `src/` layouts, each service with its own `pyproject.toml`, `setup.cfg` or `setup.py`, and the package directories those declare — so `from shop.cart import total` reaches `src/shop/cart.py`, `from pkg import name` reaches what the package's `__init__.py` defines or re-exports (respecting `__all__`), and `import json` or `import requests` no longer links to a same-named file nested in a test fixture, an example, a vendored copy or another package. After a module, package or `__init__.py` is added or removed, a package's `__init__.py` is edited, or a build config is edited, the next sync brings Python imports back in line with a fresh index; the file watcher doesn't react to a build-config edit by itself, so run `codegraph sync` after one. Re-index Python projects after upgrading. (#1704) - An Express route keeps its named handler when a middleware before it contains an arrow function. `rateLimit({ keyGenerator: (req) => req.ip })` or `(req, res, next) => next()` no longer hides the handler at the end of the route (re-index to update an existing project). diff --git a/__tests__/fixtures/kernel-parity/torture.py b/__tests__/fixtures/kernel-parity/torture.py index 001d64f49..af83300d3 100644 --- a/__tests__/fixtures/kernel-parity/torture.py +++ b/__tests__/fixtures/kernel-parity/torture.py @@ -61,3 +61,16 @@ def bucket_chains(d, k, v): make().run() (lambda: make)()().run() obj.make().run().again() + + +# --- non-call, non-identifier receivers (#66) --------------------------------- +def fabrication_shapes(rows_by_file, key): + # Attribute chain and subscript. Each must keep its receiver text as a + # qualifier — a bare `append`/`get` exact-matches an unrelated project + # function of that name. (The call-chain shape lives in bucket_chains + # above, which #1748 encodes as `().` instead.) + self_like = rows_by_file + self_like.rows.append({"x": 1}) + rows_by_file[key].append(2) + rows_by_file[key].get(key, None) + return rows_by_file diff --git a/__tests__/kernel-python-call-fabrication.test.ts b/__tests__/kernel-python-call-fabrication.test.ts new file mode 100644 index 000000000..342310ed6 --- /dev/null +++ b/__tests__/kernel-python-call-fabrication.test.ts @@ -0,0 +1,98 @@ +/** + * Python non-identifier call receivers must not fabricate edges — ON THE ARM + * THAT SHIPS. + * + * `resolution.test.ts` already covers this end-to-end, but python is in the + * kernel's DEFAULT_ROUTED set, so in a from-source checkout (no staged .node) + * that test exercises the *wasm* TreeSitterExtractor while every published + * bundle extracts python with codegraph-kernel/src/python.rs. A fix applied to + * only one arm is a green test over code that never runs. + * + * This suite forces the kernel arm and asserts the same two properties: + * - `ledger.append(row)`, where `ledger` is an imported project module that + * exports `append`, RESOLVES (the false-negative half, #66). + * - `self.data.append(...)` / `rows[k].append(...)` — attribute-chain and + * subscript receivers — do NOT fabricate an edge to that same `append` + * (the false-positive half; `python.rs` used to emit a BARE `append`, + * which exact-matched the only project-wide symbol of that name). + * + * Skips when no kernel binary is staged, like the parity suites. + */ + +import { describe, it, expect, beforeEach, afterEach } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src/index'; + +const KERNEL_PATH = path.join( + __dirname, + '..', + 'codegraph-kernel', + 'prebuilds', + `${process.platform}-${process.arch}`, + 'codegraph-kernel.node' +); +const kernelBuilt = fs.existsSync(KERNEL_PATH); + +describe.skipIf(!kernelBuilt)('python call-receiver fabrication (kernel arm)', () => { + let tempDir: string; + let cg: CodeGraph | undefined; + let savedLangs: string | undefined; + + beforeEach(() => { + savedLangs = process.env.CODEGRAPH_KERNEL_LANGS; + process.env.CODEGRAPH_KERNEL_LANGS = 'python'; + tempDir = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-py-fab-')); + }); + + afterEach(() => { + cg?.close(); + cg = undefined; + if (savedLangs === undefined) delete process.env.CODEGRAPH_KERNEL_LANGS; + else process.env.CODEGRAPH_KERNEL_LANGS = savedLangs; + fs.rmSync(tempDir, { recursive: true, force: true }); + }); + + it('resolves a module-qualified call and fabricates none from chained receivers (#66)', async () => { + fs.writeFileSync( + path.join(tempDir, 'ledger.py'), + 'def append(row):\n return True\n\n\ndef path():\n return "ledger.jsonl"\n' + ); + fs.writeFileSync( + path.join(tempDir, 'record.py'), + 'from . import ledger\n\n\ndef add_outcome(row):\n if not ledger.append(row):\n return None\n return ledger.path()\n' + ); + fs.writeFileSync( + path.join(tempDir, 'unrelated.py'), + 'class Box:\n def __init__(self):\n self.data = []\n\n def build_map(self):\n self.data.append({"x": 1})\n rows = {}\n rows["k"] = []\n rows["k"].append(2)\n return rows\n' + ); + + cg = await CodeGraph.init(tempDir, { index: true }); + + const ledgerAppend = cg + .getNodesByKind('function') + .find((n) => n.name === 'append' && n.filePath.replace(/\\/g, '/').endsWith('ledger.py')); + expect(ledgerAppend).toBeDefined(); + + const addOutcome = cg.getNodesByKind('function').find((n) => n.name === 'add_outcome'); + expect(addOutcome).toBeDefined(); + expect( + cg + .getOutgoingEdges(addOutcome!.id) + .filter((e) => e.kind === 'calls') + .map((e) => e.target) + ).toContain(ledgerAppend!.id); + + // Both non-identifier receivers live in build_map; neither may reach + // ledger.py's append. + const buildMap = cg.getNodesByKind('method').find((n) => n.name === 'build_map'); + expect(buildMap).toBeDefined(); + expect( + cg + .getOutgoingEdges(buildMap!.id) + .filter((e) => e.kind === 'calls') + .map((e) => e.target) + ).not.toContain(ledgerAppend!.id); + }); +}); diff --git a/__tests__/python-attr-chain.test.ts b/__tests__/python-attr-chain.test.ts new file mode 100644 index 000000000..b050970e3 --- /dev/null +++ b/__tests__/python-attr-chain.test.ts @@ -0,0 +1,494 @@ +/** + * A python call through an ATTRIBUTE CHAIN is a silent miss, never a guess + * (#66 follow-up). + * + * Since #66 the extractor keeps the receiver's text, so `self.data.append(1)` + * arrives as `self.data.append` rather than a bare `append`. The comment on that + * change claimed the qualifier confined it to import/module-member resolution. + * It did not: `matchMethodCall`'s dotMatch splits it into receiver `self.data` + + * method `append`, and the bare-name strategies below then bound it to any + * project method of that name. + * + * Go (#1276) and Rust (#1585) already treat a chained receiver as + * validated-inference-or-nothing, and PHP's `this->prop.method` likewise. + * + * Every negative test here uses a DISTRACTOR — a second project symbol with the + * same method name — because without one the old fallback found the right target + * by single-candidate luck and the test would pass on both arms, proving + * nothing. (That is exactly how an earlier version of this suite was vacuous.) + */ +import { describe, it, expect, beforeEach, afterEach } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src/index'; + +describe('python attribute-chain receivers', () => { + let tempDir: string; + let cg: CodeGraph | undefined; + + beforeEach(() => { + tempDir = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-py-chain-')); + }); + afterEach(() => { + cg?.close(); + cg = undefined; + fs.rmSync(tempDir, { recursive: true, force: true }); + }); + + const callsFrom = (kind: 'function' | 'method', name: string) => { + const src = cg!.getNodesByKind(kind).find((n) => n.name === name); + expect(src).toBeDefined(); + return cg! + .getOutgoingEdges(src!.id) + .filter((e) => e.kind === 'calls') + .map((e) => cg!.getNode(e.target)!) + .map((n) => `${n.kind}:${n.name}@${n.filePath.replace(/\\/g, '/')}`); + }; + + it('does not bind a self-attribute chain to an unrelated same-named method', async () => { + fs.writeFileSync( + path.join(tempDir, 'sink.py'), + 'class Sink:\n def append(self, x):\n return x\n\n def get(self, k):\n return k\n' + ); + fs.writeFileSync( + path.join(tempDir, 'box.py'), + 'class Box:\n def __init__(self):\n self.data = []\n self.inner = {}\n\n' + + ' def build(self, k):\n self.data.append(1)\n rows = {}\n rows[k] = []\n' + + ' rows[k].append(2)\n return self.inner.get(k)\n' + ); + + cg = await CodeGraph.init(tempDir, { index: true }); + + // `self.data` is a list and `self.inner` a dict; nothing in the graph says + // so, which is exactly why guessing is wrong. Silence, not Sink. + expect(callsFrom('method', 'build')).toEqual([]); + }); + + it('does not bind a deep chain either', async () => { + fs.writeFileSync( + path.join(tempDir, 'leaf.py'), + 'class Leaf:\n def go(self, x):\n return x\n' + ); + fs.writeFileSync( + path.join(tempDir, 'deep.py'), + 'class Deep:\n def __init__(self, a):\n self.a = a\n\n' + + ' def run(self, x):\n return self.a.b.go(x)\n' + ); + + cg = await CodeGraph.init(tempDir, { index: true }); + expect(callsFrom('method', 'run')).toEqual([]); + }); + + it('guesses the class the last link spells whole before one that only shares its word', async () => { + // Django's test `Client` and the cache's `RedisCacheClient` both have a + // `get`; `self.client` is the one it is named after. `self.nodelist` is a + // `NodeList` though no single word of it matches. + writeTree({ + 'cache.py': 'class RedisCacheClient:\n def get(self, k):\n return k\n', + 'client.py': 'class Client:\n def get(self, k):\n return k\n', + 'nodes.py': 'class NodeList:\n def render(self, c):\n return c\n\n\nclass Other:\n def render(self, c):\n return c\n', + 'tests.py': 'class Tests:\n def setUp(self, client, nodelist):\n self.client = client\n self.nodelist = nodelist\n\n' + + ' def run(self):\n self.client.get(1)\n self.nodelist.render(2)\n', + }); + cg = await CodeGraph.init(tempDir, { index: true }); + expect(callsFrom('method', 'run').sort()).toEqual(['method:get@client.py', 'method:render@nodes.py']); + }); + + it('guesses an untyped chain only through its head noun', async () => { + // mealie's `database.group_shopping_lists` shares only `group` with the AI + // providers' repository; its head noun is `lists`. `group_ai_providers` + // names a provider, and `client` the mixin Django's test client gets + // `force_login` from. Django's `request.META` is a dict, not a `request`. + writeTree({ + 'repos.py': 'class GroupRepositoryAIProvider:\n def create(self):\n pass\n\n\nclass Other:\n def create(self):\n pass\n', + 'client.py': 'class ClientMixin:\n def force_login(self, u):\n return u\n\n\nclass OtherMixin:\n def force_login(self, u):\n return u\n', + 'factory.py': 'class RequestFactory:\n def get(self, path):\n return path\n', + 'use.py': 'def lists(database):\n database.group_shopping_lists.create()\n\n\ndef providers(database):\n database.group_ai_providers.create()\n\n\n' + + 'def login(case):\n case.client.force_login(1)\n\n\ndef meta(request):\n return request.META.get("HTTP_HOST")\n', + }); + cg = await CodeGraph.init(tempDir, { index: true }); + expect(callsFrom('function', 'lists')).toEqual([]); + expect(callsFrom('function', 'providers')).toEqual(['method:create@repos.py']); + expect(callsFrom('function', 'login')).toEqual(['method:force_login@client.py']); + expect(callsFrom('function', 'meta')).toEqual([]); + }); + + it('leaves a chain through a module to the module', async () => { + // `os.path.exists()` is the stdlib's, not a project storage's `exists` + // whose class shares the word `Path`. + writeTree({ + 'storage.py': 'class PathNotImplementedStorage:\n def exists(self, p):\n return p\n\n\nclass Storage:\n def exists(self, p):\n return p\n', + 'use.py': 'import os\n\n\ndef has(p):\n return os.path.exists(p)\n', + }); + cg = await CodeGraph.init(tempDir, { index: true }); + expect(callsFrom('function', 'has')).toEqual([]); + }); + + it('guesses an untyped chain only by the name of its last link, as any receiver', async () => { + // Nothing types `cfg`, so the type reader has no answer. `cfg.client` is + // then guessed as a `Client` by its name — the guess every untyped + // receiver gets (`client.send()`), and how Django's `self.client.get()` + // reaches its test client — while `cfg.conn` names neither class: no + // edge, rather than one of the two `send`s. + fs.writeFileSync(path.join(tempDir, 'client.py'), 'class Client:\n def send(self, x):\n return x\n'); + fs.writeFileSync(path.join(tempDir, 'other.py'), 'class Other:\n def send(self, x):\n return x\n'); + fs.writeFileSync( + path.join(tempDir, 'use.py'), + 'def run(cfg, x):\n return cfg.client.send(x)\n\n\ndef run_conn(cfg, x):\n return cfg.conn.send(x)\n' + ); + + cg = await CodeGraph.init(tempDir, { index: true }); + expect(callsFrom('function', 'run')).toEqual(['method:send@client.py']); + expect(callsFrom('function', 'run_conn')).toEqual([]); + }); + + // --- boundaries the gate must NOT cross ----------------------------------- + + it('still resolves a single-segment receiver whose type is inferable', async () => { + fs.writeFileSync( + path.join(tempDir, 'client.py'), + 'class Client:\n def send(self, payload):\n return payload\n' + ); + fs.writeFileSync( + path.join(tempDir, 'other.py'), + 'class Other:\n def send(self, payload):\n return payload\n' + ); + fs.writeFileSync( + path.join(tempDir, 'svc.py'), + 'from client import Client\n\n\ndef run(payload):\n c = Client()\n return c.send(payload)\n' + ); + + cg = await CodeGraph.init(tempDir, { index: true }); + // With `Other.send` present, only the local-variable type inference can + // pick the right one — and it still runs, because `c` has no dot in it. + expect(callsFrom('function', 'run')).toEqual(['method:send@client.py']); + }); + + it('still resolves a module-qualified call, which is a different strategy', async () => { + // A guard, not a gate test: `ledger.record` has a single-segment receiver, + // so the gate never sees it. It is here because the gate sits in the same + // function as the module path and a careless widening would take it out. + fs.writeFileSync(path.join(tempDir, 'ledger.py'), 'def record(row):\n return row\n'); + fs.writeFileSync( + path.join(tempDir, 'use.py'), + 'from . import ledger\n\n\ndef save(row):\n return ledger.record(row)\n' + ); + + cg = await CodeGraph.init(tempDir, { index: true }); + expect(callsFrom('function', 'save')).toContain('function:record@ledger.py'); + }); + + // --- recall the qualifier must not cost ------------------------------------ + // + // Before the receiver was kept, each of these arrived as a bare method name + // and resolved by name. The qualifier routes them to evidence instead: the + // module the import names, or the type the class body declares. Every case + // carries a same-named distractor so a name-only guess cannot pass. + + const writeTree = (files: Record) => { + for (const [rel, body] of Object.entries(files)) { + const abs = path.join(tempDir, rel); + fs.mkdirSync(path.dirname(abs), { recursive: true }); + fs.writeFileSync(abs, body); + } + }; + + it('resolves a call through a dotted module path', async () => { + writeTree({ + 'app/__init__.py': '', + 'app/utils/__init__.py': '', + 'app/utils/helpers.py': 'def do_x():\n return 1\n', + 'app/other.py': 'def do_x():\n return 2\n', + 'app/main.py': + 'import app.utils.helpers\nfrom app import utils\n\n\n' + + 'def full():\n return app.utils.helpers.do_x()\n\n\n' + + 'def via_package():\n return utils.helpers.do_x()\n', + }); + cg = await CodeGraph.init(tempDir, { index: true }); + expect(callsFrom('function', 'full')).toEqual(['function:do_x@app/utils/helpers.py']); + expect(callsFrom('function', 'via_package')).toEqual(['function:do_x@app/utils/helpers.py']); + }); + + it('resolves a class-attribute receiver through the class or cls', async () => { + writeTree({ + 'sink.py': 'class Sink:\n def lookup(self, k):\n return k\n', + 'main.py': + 'class Registry:\n def lookup(self, k):\n return k\n\n\n' + + 'class Svc:\n registry = Registry()\n\n' + + ' def run(self):\n return Svc.registry.lookup("k")\n\n' + + ' @classmethod\n def c(cls):\n return cls.registry.lookup("k")\n', + }); + cg = await CodeGraph.init(tempDir, { index: true }); + expect(callsFrom('method', 'run')).toEqual(['method:lookup@main.py']); + expect(callsFrom('method', 'c')).toEqual(['method:lookup@main.py']); + }); + + it('resolves a self attribute to the class the class body gives it', async () => { + writeTree({ + 'pkg/__init__.py': '', + 'pkg/a.py': 'class Capture:\n def stop(self):\n pass\n', + 'pkg/b.py': 'class Sink:\n def stop(self):\n pass\n', + 'pkg/models.py': 'class User:\n def save(self):\n pass\n\n\nclass Other:\n def save(self):\n pass\n', + 'pkg/main.py': + 'from pkg.a import Capture\nfrom pkg.b import Sink\nfrom pkg import models\n\n\n' + + 'class Svc:\n def __init__(self, cap: Capture):\n self.cap = cap\n' + + ' self.sink = Sink()\n self.user = models.User()\n\n' + + ' def by_param(self):\n self.cap.stop()\n\n' + + ' def by_ctor(self):\n self.sink.stop()\n\n' + + ' def by_module_type(self):\n self.user.save()\n', + }); + cg = await CodeGraph.init(tempDir, { index: true }); + expect(callsFrom('method', 'by_param')).toEqual(['method:stop@pkg/a.py']); + expect(callsFrom('method', 'by_ctor')).toEqual(['method:stop@pkg/b.py']); + expect(callsFrom('method', 'by_module_type')).toEqual(['method:save@pkg/models.py']); + }); + + it('follows the attribute type to a method its base class declares', async () => { + writeTree({ + 'app/__init__.py': '', + 'app/capture.py': 'class Base:\n def stop(self):\n pass\n\n\nclass Capture(Base):\n def start(self):\n pass\n', + 'app/sink.py': 'class Sink:\n def stop(self):\n pass\n\n def start(self):\n pass\n', + 'app/main.py': + 'from app.capture import Capture\n\n\n' + + 'class Svc:\n def __init__(self):\n self.cap = Capture()\n\n' + + ' def run(self):\n self.cap.start()\n self.cap.stop()\n', + }); + cg = await CodeGraph.init(tempDir, { index: true }); + expect(callsFrom('method', 'run').sort()).toEqual(['method:start@app/capture.py', 'method:stop@app/capture.py']); + }); + + it('resolves an attribute of a typed local or parameter', async () => { + writeTree({ + 'models.py': + 'class Llm:\n def rebuild(self):\n pass\n\n\nclass Other:\n def rebuild(self):\n pass\n\n\n' + + 'class Config:\n llm: Llm\n\n\nclass App:\n def __init__(self):\n self.config = Config()\n', + 'use.py': + 'from models import App, Config\n\n\n' + + 'def by_param(config: Config):\n config.llm.rebuild()\n\n\n' + + 'def by_local():\n app = App()\n app.config.llm.rebuild()\n cfg = app.config\n', + }); + cg = await CodeGraph.init(tempDir, { index: true }); + expect(callsFrom('function', 'by_param')).toEqual(['method:rebuild@models.py']); + // Two attribute hops is a deeper chain than the class body answers: it is + // guessed by the name of its last link, `llm`, as an untyped one is. + expect(callsFrom('function', 'by_local')).toEqual(['method:rebuild@models.py']); + }); + + it('leaves an attribute typed as an external class unresolved', async () => { + writeTree({ + 'session.py': 'class Session:\n def get(self, url):\n return url\n', + 'svc.py': + 'import requests\n\n\nclass Svc:\n def __init__(self):\n self.http = requests.Session()\n\n' + + ' def fetch(self, url):\n return self.http.get(url)\n', + }); + cg = await CodeGraph.init(tempDir, { index: true }); + expect(callsFrom('method', 'fetch')).toEqual([]); + }); + + it('types an attribute only by a call that is the whole assigned value', async () => { + const lib = + 'class Client:\n def send(self):\n pass\n\n def close(self):\n pass\n\n\n' + + 'class Fake:\n def send(self):\n pass\n\n\n' + + 'class Pool:\n def acquire(self):\n return Client()\n\n def close(self):\n pass\n'; + writeTree({ + 'lib.py': lib, + 'svc.py': + 'from lib import Client, Fake, Pool\n\n\nclass Svc:\n def __init__(self, t):\n' + + ' self.pooled = Pool().acquire()\n' + + ' self.either = Client() if t else Fake()\n' + + ' self.wrapped = Client() \\\n if t else Fake()\n' + + ' self.closed = Client().close()\n' + + ' self.same = Client(1) if t else Client(2)\n' + + ' self.multi = Client(\n t,\n )\n\n' + + ' def pooled_(self):\n self.pooled.close()\n\n' + + ' def either_(self):\n self.either.send()\n\n' + + ' def wrapped_(self):\n self.wrapped.send()\n\n' + + ' def closed_(self):\n self.closed.send()\n\n' + + ' def multi_(self):\n self.multi.send()\n\n' + + ' def same_(self):\n self.same.send()\n', + }); + cg = await CodeGraph.init(tempDir, { index: true }); + // `Pool().acquire()` holds what `acquire` returns, not a Pool; a + // conditional whose arms differ holds either; `Client().close()` holds what `close` + // returns. A call spread over lines is still one whole call. + expect(callsFrom('method', 'pooled_')).toEqual([]); + expect(callsFrom('method', 'either_')).toEqual([]); + expect(callsFrom('method', 'wrapped_')).toEqual([]); + expect(callsFrom('method', 'closed_')).toEqual([]); + expect(callsFrom('method', 'multi_')).toEqual(['method:send@lib.py']); + // Both arms constructing the same class is that class. + expect(callsFrom('method', 'same_')).toEqual(['method:send@lib.py']); + }); + + it('never skips a statement after a value that does not close', async () => { + const lib = 'class Client:\n def send(self):\n pass\n\n\nclass Fake:\n def send(self):\n pass\n'; + const run = ' def run(self):\n self.c.send()\n'; + writeTree({ + 'lib.py': lib, + // A string that merely mentions `self.c =` and has an unmatched `(`. + 'a.py': 'from lib import Client, Fake\n\n\nclass A:\n def __init__(self):\n self.c = Client()\n\n' + + ' def check(self):\n raise ValueError("self.c = must be set (")\n\n' + + ' def reset(self):\n self.c = Fake()\n\n' + run, + // A continuation line `c=1)` at class-body indent reads like `c = 1)`. + 'b.py': 'from lib import Client, Fake\n\n\nclass B:\n def __init__(self):\n self.c = Client()\n\n' + + ' opts = dict(\n c=1)\n\n def reset(self):\n self.c = Fake()\n\n' + run, + // A Python 3.12 f-string reusing its own quote inside the braces. + 'c.py': 'from lib import Client, Fake\n\n\nclass C:\n def __init__(self):\n self.c = Client()\n\n' + + ' def check(self):\n self.c = f"{\'"\'}"\n self.c = Fake()\n\n' + run, + }); + cg = await CodeGraph.init(tempDir, { index: true }); + // Each class also assigns `Fake()`, so the answer is contested: no edge. + const runs = cg.getNodesByKind('method').filter((n) => n.name === 'run'); + expect(runs).toHaveLength(3); + for (const r of runs) { + expect(cg.getOutgoingEdges(r.id).filter((e) => e.kind === 'calls')).toEqual([]); + } + }); + describe('review round 2026-09-30', () => { + const write = (rel: string, src: string) => { + fs.mkdirSync(path.dirname(path.join(tempDir, rel)), { recursive: true }); + fs.writeFileSync(path.join(tempDir, rel), src); + }; + + it('an attribute assigned in the calling method still needs the whole value to be a call', async () => { + write('pool.py', 'class Pool:\n def acquire(self):\n return object()\n\n def send(self, x):\n return x\n'); + write('svc.py', 'from pool import Pool\n\nclass Svc:\n def run(self):\n self.conn = Pool().acquire()\n return self.conn.send(1)\n'); + cg = await CodeGraph.init(tempDir, { index: true }); + expect(callsFrom('method', 'run')).not.toContain('method:send@pool.py'); + }); + + it('an attribute assigned in the calling method from a rebound class name is not that class', async () => { + write('backend.py', 'class Backend:\n def run(self):\n return 1\n'); + write('svc.py', 'from backend import Backend\n\ndef load():\n return object\n\nclass Svc:\n def go(self):\n Backend = load()\n self.b = Backend()\n return self.b.run()\n'); + cg = await CodeGraph.init(tempDir, { index: true }); + expect(callsFrom('method', 'go')).not.toContain('method:run@backend.py'); + }); + + it('an attribute assigned in the calling method from an out-of-repo class is not a same-named project class', async () => { + write('app/local.py', 'class Session:\n def get(self, url):\n return url\n'); + write('app/api.py', 'from requests import Session\n\nclass Api:\n def fetch(self):\n self.session = Session()\n return self.session.get("x")\n'); + cg = await CodeGraph.init(tempDir, { index: true }); + expect(callsFrom('method', 'fetch')).not.toContain('method:get@app/local.py'); + }); + + it('a parameter rebound before the assignment no longer carries its annotation', async () => { + write('box.py', 'class Box:\n def run(self):\n return 1\n'); + // `held`, named after no class, so no guess by name stands in for the + // annotation this must not carry. + write('svc.py', 'from box import Box\n\ndef wrap(b):\n return b\n\nclass Svc:\n def __init__(self, box: Box):\n box = wrap(box)\n self.held = box\n\n def go(self):\n return self.held.run()\n'); + cg = await CodeGraph.init(tempDir, { index: true }); + expect(callsFrom('method', 'go')).not.toContain('method:run@box.py'); + }); + + it('descends a dotted module path from a relative package import', async () => { + write('app/__init__.py', ''); + write('app/rel/__init__.py', ''); + write('app/rel/utils/__init__.py', ''); + write('app/rel/utils/helpers.py', 'def do_x():\n return 1\n'); + write('app/rel/use.py', 'from . import utils\n\ndef rel_caller():\n return utils.helpers.do_x()\n'); + cg = await CodeGraph.init(tempDir, { index: true }); + expect(callsFrom('function', 'rel_caller')).toContain('function:do_x@app/rel/utils/helpers.py'); + }); + }); + describe('review round 2026-10-01', () => { + const write = (rel: string, src: string) => { + fs.mkdirSync(path.dirname(path.join(tempDir, rel)), { recursive: true }); + fs.writeFileSync(path.join(tempDir, rel), src); + }; + const FOO = 'class Foo:\n def m(self):\n return 1\n'; + const OTHER = 'class Other:\n def m(self):\n return 2\n'; + const resolves = async (svc: string, fn = 'go') => { + write('foo.py', FOO); + write('other.py', OTHER); + write('svc.py', svc); + cg = await CodeGraph.init(tempDir, { index: true }); + return callsFrom('method', fn).filter((c) => c.startsWith('method:m@')); + }; + + it('a keyword argument passing a parameter through is not a rebinding', async () => { + expect(await resolves('from foo import Foo\n\nclass Base:\n def __init__(self, client=None):\n pass\n\nclass Svc(Base):\n def __init__(self, client: Foo):\n super().__init__(\n client=client,\n )\n validate(client=client)\n self.client = client\n\n def go(self):\n return self.client.m()\n\ndef validate(client=None):\n return client\n')) + .toEqual(['method:m@foo.py']); + }); + + it('a keyword argument named like the class is not a rebinding', async () => { + expect(await resolves('from foo import Foo\n\ndef reg(**kw):\n return kw\n\nclass Svc:\n def __init__(self):\n reg(Foo=1)\n reg(\n Foo=2,\n )\n self.b = Foo()\n\n def go(self):\n return self.b.m()\n')) + .toEqual(['method:m@foo.py']); + }); + + it('a function-local import of the class is not a rebinding', async () => { + expect(await resolves('class Svc:\n def __init__(self):\n from foo import Foo\n self.b = Foo()\n\n def go(self):\n return self.b.m()\n')) + .toEqual(['method:m@foo.py']); + }); + + it('a nested def above the assignment does not rebind the outer name', async () => { + expect(await resolves('from foo import Foo\n\nclass Svc:\n def __init__(self):\n def cb(Foo=None):\n Foo = 1\n return Foo\n self.b = Foo()\n\n def go(self):\n return self.b.m()\n')) + .toEqual(['method:m@foo.py']); + }); + + it('text in a string is not a rebinding', async () => { + expect(await resolves('from foo import Foo\n\nclass Svc:\n def __init__(self):\n print("Foo = %s" % 1)\n self.b = Foo()\n\n def go(self):\n return self.b.m()\n')) + .toEqual(['method:m@foo.py']); + }); + + it('a walrus still rebinds', async () => { + expect(await resolves('from foo import Foo\n\ndef load():\n return object\n\nclass Svc:\n def __init__(self, k):\n if (Foo := load()):\n pass\n self.b = Foo()\n\n def go(self):\n return self.b.m()\n')).toEqual([]); + }); + + it('a one-line compound statement still rebinds', async () => { + expect(await resolves('from foo import Foo\n\ndef load():\n return object\n\nclass Svc:\n def __init__(self, k):\n if k: Foo = load()\n self.b = Foo()\n\n def go(self):\n return self.b.m()\n')).toEqual([]); + }); + + it('leaves a deeper attribute chain unresolved, even one the calling method assigned', async () => { + expect(await resolves('from foo import Foo\n\nclass Svc:\n def go(self):\n self.a.b = Foo()\n return self.a.b.m()\n')).toEqual([]); + }); + + it('leaves an attribute on an untyped parameter unresolved', async () => { + write('foo.py', FOO); + write('other.py', OTHER); + write('svc.py', 'from foo import Foo\n\ndef go(c):\n c.client = Foo()\n return c.client.m()\n'); + cg = await CodeGraph.init(tempDir, { index: true }); + expect(callsFrom('function', 'go').filter((c) => c.startsWith('method:m@'))).toEqual([]); + }); + + it('assignments that disagree anywhere in the class are no evidence, even in the calling method', async () => { + expect(await resolves('from foo import Foo\nfrom other import Other\n\nclass Svc:\n def reset(self):\n self.x = Other()\n\n def go(self):\n self.x = Foo()\n return self.x.m()\n')).toEqual([]); + }); + }); + describe('review round 2026-10-02', () => { + const write = (rel: string, src: string) => { + fs.mkdirSync(path.dirname(path.join(tempDir, rel)), { recursive: true }); + fs.writeFileSync(path.join(tempDir, rel), src); + }; + const resolves = async (svc: string) => { + write('foo.py', 'class Foo:\n def m(self):\n return 1\n'); + write('other.py', 'class Other:\n def m(self):\n return 2\n'); + write('svc.py', 'from foo import Foo\nfrom other import Other\n\n' + svc); + cg = await CodeGraph.init(tempDir, { index: true }); + return callsFrom('method', 'go').filter((c) => c.startsWith('method:m@')); + }; + + it('storing into a container keyed by the class, or setting its attribute, is not a rebinding', async () => { + expect(await resolves('class Svc:\n def __init__(self, d, c):\n d[Foo] = 1\n self._h[Foo] = c\n Foo.attr = 1\n self.b = Foo()\n\n def go(self):\n return self.b.m()\n')).toEqual(['method:m@foo.py']); + }); + + it('branches of the calling method that disagree are no evidence', async () => { + expect(await resolves('class Svc:\n def go(self, a):\n if a:\n self.x = Foo()\n else:\n self.x = Other()\n return self.x.m()\n')).toEqual([]); + }); + + it('a branch in the calling method does not override the class body', async () => { + expect(await resolves('class Svc:\n def __init__(self):\n self.x = Other()\n\n def go(self, a):\n if a:\n self.x = Foo()\n return self.x.m()\n')).toEqual([]); + }); + + it('an assignment inside a nested def is not typed by the enclosing method', async () => { + expect(await resolves('class Svc:\n def __init__(self, c: Foo):\n def on_ready(c):\n self.x = c\n self.cb = on_ready\n\n def go(self):\n return self.x.m()\n')).toEqual([]); + cg!.close(); cg = undefined; + fs.rmSync(path.join(tempDir, '.codegraph'), { recursive: true, force: true }); + expect(await resolves('def load():\n return object\n\nclass Svc:\n def __init__(self):\n def setup():\n Foo = load()\n self.x = Foo()\n setup()\n\n def go(self):\n return self.x.m()\n')).toEqual([]); + }); + + it('an augmented assignment is not typed by its right-hand side', async () => { + expect(await resolves('class Svc:\n def go(self):\n self.x += Foo()\n return self.x.m()\n')).toEqual([]); + }); + }); +}); diff --git a/__tests__/python-attr-types.test.ts b/__tests__/python-attr-types.test.ts new file mode 100644 index 000000000..920fb0a25 --- /dev/null +++ b/__tests__/python-attr-types.test.ts @@ -0,0 +1,628 @@ +/** + * Python attribute types a class body does not spell out itself (#1704, #750). + * + * `self.client.send()` resolves when the class body says what `self.client` + * holds (python-attr-chain.test.ts). These are the two common ways it says so + * only INDIRECTLY: + * + * - through a factory: `self.client = Client.from_env()` / `make_client()`, + * typed by the factory's `-> T` annotation or by what every `return` in its + * body constructs (the Python instance of the #750 chained-factory family); + * - through a base class: `self.repo` assigned in a base's `__init__`, in + * another file, and used in a subclass. + * + * Each positive test plants a DISTRACTOR — a second project class with the same + * method name — so a bare-name guess cannot pass it. Each negative test is a + * shape where the type is unknowable or contested: silence, never a guess. + */ +import { describe, it, expect, beforeEach, afterEach, vi } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src/index'; + +describe('python attribute types from factories and base classes', () => { + let tempDir: string; + let cg: CodeGraph | undefined; + + beforeEach(() => { + tempDir = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-py-attr-types-')); + }); + afterEach(() => { + cg?.close(); + cg = undefined; + fs.rmSync(tempDir, { recursive: true, force: true }); + }); + + const write = (files: Record) => { + for (const [rel, text] of Object.entries(files)) { + const abs = path.join(tempDir, rel); + fs.mkdirSync(path.dirname(abs), { recursive: true }); + fs.writeFileSync(abs, text); + } + }; + + const callsFrom = (name: string) => { + const src = cg!.getNodesByKind('method').find((n) => n.name === name); + expect(src).toBeDefined(); + return cg! + .getOutgoingEdges(src!.id) + .filter((e) => e.kind === 'calls') + .map((e) => cg!.getNode(e.target)!) + .map((n) => `${n.qualifiedName}@${n.filePath.replace(/\\/g, '/')}`); + }; + + const DISTRACTOR = 'class Other:\n def send(self):\n pass\n\n def save(self):\n pass\n\n def go(self):\n pass\n'; + + describe('factories', () => { + it('types an attribute by a classmethod return annotation', async () => { + write({ + 'client.py': 'class Client:\n @classmethod\n def from_env(cls) -> "Client":\n return cls()\n\n def send(self):\n pass\n', + 'other.py': DISTRACTOR, + 'svc.py': 'from client import Client\n\nclass Svc:\n def __init__(self):\n self.client = Client.from_env()\n\n def run(self):\n self.client.send()\n', + }); + cg = await CodeGraph.init(tempDir, { index: true }); + expect(callsFrom('run')).toEqual(['Client::send@client.py']); + }); + + it('types an attribute by a classmethod that returns cls(...)', async () => { + write({ + 'client.py': 'class Client:\n @classmethod\n def from_env(cls):\n return cls()\n\n def send(self):\n pass\n', + 'other.py': DISTRACTOR, + 'svc.py': 'from client import Client\n\nclass Svc:\n def __init__(self):\n self.client = Client.from_env()\n\n def run(self):\n self.client.send()\n', + }); + cg = await CodeGraph.init(tempDir, { index: true }); + expect(callsFrom('run')).toEqual(['Client::send@client.py']); + }); + + it('types cls(...) as the class the factory is called on, not where it is defined', async () => { + write({ + 'base.py': 'class Base:\n @classmethod\n def create(cls):\n return cls()\n\n def send(self):\n pass\n', + 'client.py': 'from base import Base\n\nclass Client(Base):\n def send(self):\n pass\n', + 'svc.py': 'from client import Client\n\nclass Svc:\n def __init__(self):\n self.client = Client.create()\n\n def run(self):\n self.client.send()\n', + }); + cg = await CodeGraph.init(tempDir, { index: true }); + expect(callsFrom('run')).toEqual(['Client::send@client.py']); + }); + + it('types an attribute by an imported factory function annotation', async () => { + write({ + 'client.py': 'class Client:\n def send(self):\n pass\n\ndef make_client() -> Client:\n return Client()\n', + 'other.py': DISTRACTOR, + 'svc.py': 'from client import make_client\n\nclass Svc:\n def __init__(self):\n self.client = make_client()\n\n def run(self):\n self.client.send()\n', + }); + cg = await CodeGraph.init(tempDir, { index: true }); + expect(callsFrom('run')).toEqual(['Client::send@client.py']); + }); + + it('types an attribute by what an unannotated factory returns', async () => { + write({ + 'client.py': 'class Client:\n def send(self):\n pass\n\ndef make_client(url=None):\n if url is None:\n return None\n return Client()\n', + 'other.py': DISTRACTOR, + 'svc.py': 'from client import make_client\n\nclass Svc:\n def __init__(self):\n self.client = make_client("x")\n\n def run(self):\n self.client.send()\n', + }); + cg = await CodeGraph.init(tempDir, { index: true }); + expect(callsFrom('run')).toEqual(['Client::send@client.py']); + }); + + it('types a factory called through its module', async () => { + write({ + 'app/__init__.py': '', + 'app/clients.py': 'class Client:\n def send(self):\n pass\n\ndef build() -> "Client":\n return Client()\n', + 'other.py': DISTRACTOR, + 'svc.py': 'from app import clients\n\nclass Svc:\n def __init__(self):\n self.client = clients.build()\n\n def run(self):\n self.client.send()\n', + }); + cg = await CodeGraph.init(tempDir, { index: true }); + expect(callsFrom('run')).toEqual(['Client::send@app/clients.py']); + }); + + it('stays silent when the factory returns something it cannot name', async () => { + write({ + 'client.py': 'REGISTRY = {}\n\nclass Client:\n def send(self):\n pass\n\ndef make_client(kind):\n return REGISTRY[kind]()\n', + 'other.py': DISTRACTOR, + // `conn`, named after no class: what is tested is the type reader, + // not the guess by name an untyped `self.client` would get. + 'svc.py': 'from client import make_client\n\nclass Svc:\n def __init__(self):\n self.conn = make_client("x")\n\n def run(self):\n self.conn.send()\n', + }); + cg = await CodeGraph.init(tempDir, { index: true }); + expect(callsFrom('run')).toEqual([]); + }); + + it('types a factory only by returns that are whole calls, and by agreeing arms', async () => { + const lib = 'class Client:\n def send(self):\n pass\n\n def close(self):\n pass\n\n' + + 'class Fake:\n def send(self):\n pass\n\n' + + 'class Pool:\n def acquire(self):\n return Client()\n\n def close(self):\n pass\n'; + write({ + 'lib.py': lib, + 'f.py': 'from lib import Client, Fake, Pool\n\n' + + 'def either(t):\n return Client() if t else Fake()\n\n' + + 'def wrapped(t):\n return Client() \\\n if t else Fake()\n\n' + + 'def chained():\n return Pool().acquire()\n\n' + + 'def closed():\n return Client().close()\n\n' + + 'def same(t):\n return Client(1) if t else Client(\n 2,\n )\n\n' + + 'def make_pool():\n return Pool()\n', + 'svc.py': 'from f import either, wrapped, chained, closed, same, make_pool\n\n' + + 'class Svc:\n def __init__(self, t):\n' + + ' self.a = either(t)\n self.b = wrapped(t)\n self.c = chained()\n' + + ' self.d = closed()\n self.e = same(t)\n self.g = make_pool().acquire()\n\n' + + ' def a_(self):\n self.a.send()\n\n def b_(self):\n self.b.send()\n\n' + + ' def c_(self):\n self.c.close()\n\n def d_(self):\n self.d.send()\n\n' + + ' def e_(self):\n self.e.send()\n\n def g_(self):\n self.g.close()\n', + }); + cg = await CodeGraph.init(tempDir, { index: true }); + expect(callsFrom('a_')).toEqual([]); + expect(callsFrom('b_')).toEqual([]); + expect(callsFrom('c_')).toEqual([]); + expect(callsFrom('d_')).toEqual([]); + expect(callsFrom('e_')).toEqual(['Client::send@lib.py']); + expect(callsFrom('g_')).toEqual([]); + }); + + it('stays silent when the factory returns two different classes', async () => { + write({ + 'client.py': 'class Client:\n def send(self):\n pass\n\nclass Fake:\n def send(self):\n pass\n\ndef make_client(test):\n if test:\n return Fake()\n return Client()\n', + 'svc.py': 'from client import make_client\n\nclass Svc:\n def __init__(self):\n self.conn = make_client(False)\n\n def run(self):\n self.conn.send()\n', + }); + cg = await CodeGraph.init(tempDir, { index: true }); + expect(callsFrom('run')).toEqual([]); + }); + + it('stays silent for a generator or an un-awaited coroutine factory', async () => { + write({ + 'client.py': 'class Client:\n def send(self):\n pass\n\ndef clients():\n yield Client()\n\nasync def connect():\n return Client()\n', + 'other.py': DISTRACTOR, + 'svc.py': 'from client import clients, connect\n\nclass Svc:\n def __init__(self):\n self.a = clients()\n self.b = connect()\n\n def run(self):\n self.a.send()\n self.b.send()\n', + }); + cg = await CodeGraph.init(tempDir, { index: true }); + expect(callsFrom('run')).toEqual([]); + }); + + it('types an awaited coroutine factory', async () => { + write({ + 'client.py': 'class Client:\n def send(self):\n pass\n\nasync def connect() -> Client:\n return Client()\n', + 'other.py': DISTRACTOR, + 'svc.py': 'from client import connect\n\nclass Svc:\n async def start(self):\n self.client = await connect()\n\n def run(self):\n self.client.send()\n', + }); + cg = await CodeGraph.init(tempDir, { index: true }); + expect(callsFrom('run')).toEqual(['Client::send@client.py']); + }); + + it('stays silent when the returned class name is rebound locally', async () => { + write({ + 'signing.py': 'from loader import import_string\n\nclass Signer:\n def unsign(self, v):\n return v\n\ndef get_signer(backend):\n Signer = import_string(backend)\n return Signer()\n', + 'loader.py': 'def import_string(p):\n return None\n', + 'other.py': 'class Other:\n def unsign(self, v):\n return v\n', + 'svc.py': 'from signing import Signer, get_signer\nfrom loader import import_string\n\nclass Svc:\n def __init__(self, path):\n self.a = get_signer(path)\n Backend = import_string(path)\n self.b = Backend()\n\n def run(self):\n self.a.unsign(1)\n self.b.unsign(2)\n', + }); + cg = await CodeGraph.init(tempDir, { index: true }); + expect(callsFrom('run')).toEqual([]); + }); + + it('keeps a parameter-typed attribute whose parameter shadows an imported module', async () => { + write({ + 'shapes/__init__.py': '', + 'shapes/box.py': 'class Box:\n def substitute(self):\n pass\n', + 'other.py': 'class Other:\n def substitute(self):\n pass\n', + 'shapes/table.py': 'from typing import Optional\nfrom . import box\n\nclass Table:\n def __init__(self, box: Optional[box.Box] = None):\n self.box = box\n\n def run(self):\n self.box.substitute()\n', + }); + cg = await CodeGraph.init(tempDir, { index: true }); + expect(callsFrom('run')).toEqual(['Box::substitute@shapes/box.py']); + }); + + it('does not follow a factory imported from outside the project', async () => { + write({ + 'client.py': 'class Session:\n def send(self):\n pass\n\ndef session() -> Session:\n return Session()\n', + 'svc.py': 'from requests import session\n\nclass Svc:\n def __init__(self):\n self.http = session()\n\n def run(self):\n self.http.send()\n', + }); + cg = await CodeGraph.init(tempDir, { index: true }); + expect(callsFrom('run')).toEqual([]); + }); + + it('does not let a same-named factory in another module answer', async () => { + write({ + 'p/__init__.py': '', + 'p/a.py': 'class Client:\n def send(self):\n pass\n\ndef make() -> Client:\n return Client()\n', + 'q/__init__.py': '', + 'q/a.py': 'class Fake:\n def send(self):\n pass\n\ndef make() -> Fake:\n return Fake()\n', + 'svc.py': 'from q.a import make\n\nclass Svc:\n def __init__(self):\n self.client = make()\n\n def run(self):\n self.client.send()\n', + }); + cg = await CodeGraph.init(tempDir, { index: true }); + expect(callsFrom('run')).toEqual(['Fake::send@q/a.py']); + }); + }); + + describe('chained factory calls (#750)', () => { + it('resolves a method called straight on a factory result', async () => { + write({ + 'client.py': 'class Client:\n @classmethod\n def from_env(cls):\n return cls()\n\n def send(self):\n pass\n\ndef make_client() -> Client:\n return Client()\n', + 'other.py': DISTRACTOR, + 'svc.py': 'from client import Client, make_client\n\nclass Svc:\n def run(self):\n Client.from_env().send()\n make_client().send()\n Client().send()\n', + }); + cg = await CodeGraph.init(tempDir, { index: true }); + expect(callsFrom('run').filter((c) => c.includes('send'))).toEqual([ + 'Client::send@client.py', 'Client::send@client.py', 'Client::send@client.py', + ]); + }); + + it('stays silent on a chained call through a factory it cannot type', async () => { + write({ + 'client.py': 'class Client:\n def send(self):\n pass\n\ndef make_client(kind):\n return kind()\n', + 'other.py': DISTRACTOR, + 'svc.py': 'from client import make_client\n\nclass Svc:\n def run(self):\n make_client(1).send()\n', + }); + cg = await CodeGraph.init(tempDir, { index: true }); + expect(callsFrom('run').filter((c) => c.includes('send'))).toEqual([]); + }); + }); + + describe('base classes', () => { + it('types an attribute assigned in a base class __init__ in another file', async () => { + write({ + 'repo.py': 'class Repo:\n def save(self):\n pass\n', + 'base.py': 'from repo import Repo\n\nclass Base:\n def __init__(self):\n self.repo = Repo()\n', + 'other.py': DISTRACTOR, + 'child.py': 'from base import Base\n\nclass Child(Base):\n def run(self):\n self.repo.save()\n', + }); + cg = await CodeGraph.init(tempDir, { index: true }); + expect(callsFrom('run')).toEqual(['Repo::save@repo.py']); + }); + + it('follows a chain of bases', async () => { + write({ + 'repo.py': 'class Repo:\n def save(self):\n pass\n', + 'base.py': 'from repo import Repo\n\nclass Base:\n def __init__(self, repo: Repo):\n self.repo = repo\n', + 'mid.py': 'from base import Base\n\nclass Mid(Base):\n pass\n', + 'other.py': DISTRACTOR, + 'child.py': 'from mid import Mid\n\nclass Child(Mid):\n def run(self):\n self.repo.save()\n', + }); + cg = await CodeGraph.init(tempDir, { index: true }); + expect(callsFrom('run')).toEqual(['Repo::save@repo.py']); + }); + + it('lets the subclass own assignment shadow the base', async () => { + write({ + 'repo.py': 'class Repo:\n def save(self):\n pass\n\nclass Cache:\n def save(self):\n pass\n', + 'base.py': 'from repo import Repo\n\nclass Base:\n def __init__(self):\n self.repo = Repo()\n', + 'child.py': 'from base import Base\nfrom repo import Cache\n\nclass Child(Base):\n def __init__(self):\n super().__init__()\n self.repo = Cache()\n\n def run(self):\n self.repo.save()\n', + }); + cg = await CodeGraph.init(tempDir, { index: true }); + expect(callsFrom('run')).toEqual(['Cache::save@repo.py']); + }); + + it('stays silent when an untyped subclass assignment shadows a typed base', async () => { + write({ + 'repo.py': 'class Repo:\n def save(self):\n pass\n', + 'base.py': 'from repo import Repo\n\nclass Base:\n def __init__(self):\n self.store = Repo()\n', + 'other.py': DISTRACTOR, + 'child.py': 'from base import Base\n\nclass Child(Base):\n def __init__(self, repo):\n super().__init__()\n self.store = repo\n\n def run(self):\n self.store.save()\n', + }); + cg = await CodeGraph.init(tempDir, { index: true }); + expect(callsFrom('run')).toEqual([]); + }); + + it('stays silent when two bases disagree', async () => { + write({ + 'repo.py': 'class Repo:\n def save(self):\n pass\n\nclass Cache:\n def save(self):\n pass\n', + 'bases.py': 'from repo import Repo, Cache\n\nclass A:\n def __init__(self):\n self.store = Repo()\n\nclass B:\n def __init__(self):\n self.store = Cache()\n', + 'child.py': 'from bases import A, B\n\nclass Child(A, B):\n def run(self):\n self.store.save()\n', + }); + cg = await CodeGraph.init(tempDir, { index: true }); + expect(callsFrom('run')).toEqual([]); + }); + + it('stays silent for a base class from outside the project', async () => { + write({ + 'repo.py': 'class Repo:\n def save(self):\n pass\n', + 'other.py': DISTRACTOR, + 'child.py': 'from django.views import View\n\nclass Child(View):\n def run(self):\n self.repo.save()\n', + }); + cg = await CodeGraph.init(tempDir, { index: true }); + // Nothing of the project's assigns `self.repo`, so it is the outside + // base's — not even guessed as the project's `Repo` by its name. + expect(callsFrom('run')).toEqual([]); + }); + + it('does not rule a method out of a class whose base the reader cannot find', async () => { + // `MultiValueField` reaches `forms.py` through a package that star-imports + // it, which the reader does not follow: `ComplexField` may inherit + // `clean`, so the call is guessed by name rather than left unresolved. + write({ + 'fields/__init__.py': 'from .impl import *\n', + 'fields/impl.py': 'class MultiValueField:\n def clean(self, v):\n return v\n', + 'forms.py': 'from fields import MultiValueField\n\nclass ComplexField(MultiValueField):\n pass\n', + 'other.py': 'class Other:\n def clean(self, v):\n return v\n', + 'tests.py': 'from forms import ComplexField\n\nclass T:\n def setUp(self):\n self.field = ComplexField()\n\n def run(self):\n self.field.clean(1)\n', + }); + cg = await CodeGraph.init(tempDir, { index: true }); + expect(callsFrom('run')).toEqual(['MultiValueField::clean@fields/impl.py']); + }); + + it("guesses by name an attribute whose only outside base is one that gives none, like ABC", async () => { + write({ + 'repo.py': 'class Repo:\n def save(self):\n pass\n', + 'other.py': DISTRACTOR, + 'child.py': 'from abc import ABC\n\nclass Child(ABC):\n def run(self):\n self.repo.save()\n', + }); + cg = await CodeGraph.init(tempDir, { index: true }); + expect(callsFrom('run')).toEqual(['Repo::save@repo.py']); + }); + + it('guesses by name a class-body field an outside class builds, which a descriptor may hand something else', async () => { + // Django's `author = models.ForeignKey(Author)`: instances hold an Author. + write({ + 'models.py': 'from django.db import models\n\nclass Author(models.Model):\n def save(self):\n pass\n\n' + + 'class Book(models.Model):\n author = models.ForeignKey(Author, on_delete=models.CASCADE)\n\n def run(self):\n self.author.save()\n', + 'other.py': DISTRACTOR, + }); + cg = await CodeGraph.init(tempDir, { index: true }); + expect(callsFrom('run')).toEqual(['Author::save@models.py']); + }); + + it('leaves an attribute an outside class builds in a method unresolved', async () => { + write({ + 'session.py': 'class Session:\n def get(self):\n pass\n', + 'api.py': 'import requests\n\nclass Api:\n def __init__(self):\n self.session = requests.Session()\n\n def run(self):\n self.session.get()\n', + }); + cg = await CodeGraph.init(tempDir, { index: true }); + expect(callsFrom('run')).toEqual([]); + }); + }); + + describe('already typed by PR-A (pinned)', () => { + it('types a dotted constructor and a dotted annotation', async () => { + write({ + 'app/__init__.py': '', + 'app/models.py': 'class User:\n def save(self):\n pass\n', + 'other.py': DISTRACTOR, + 'svc.py': 'import app.models\nfrom app import models\n\nclass Svc:\n def __init__(self, u):\n self.user = models.User()\n self.other: app.models.User = u\n\n def run(self):\n self.user.save()\n self.other.save()\n', + }); + cg = await CodeGraph.init(tempDir, { index: true }); + expect(callsFrom('run')).toEqual(['User::save@app/models.py', 'User::save@app/models.py']); + }); + + it('needs every branch of a conditional assignment to agree', async () => { + write({ + 'a.py': 'class A:\n def go(self):\n pass\n\nclass B:\n def go(self):\n pass\n', + 'other.py': DISTRACTOR, + 'svc.py': 'from a import A, B\n\nclass Svc:\n def __init__(self, f):\n if f:\n self.x = A(1)\n else:\n self.x = A(2)\n try:\n self.y = A()\n except Exception:\n self.y = B()\n\n def run(self):\n self.x.go()\n self.y.go()\n', + }); + cg = await CodeGraph.init(tempDir, { index: true }); + expect(callsFrom('run')).toEqual(['A::go@a.py']); + }); + }); + + describe('sync converges to a fresh index', () => { + const edgesOf = (graph: CodeGraph) => + graph + .getNodesByKind('method') + .filter((n) => n.name === 'run') + .flatMap((n) => graph.getOutgoingEdges(n.id).filter((e) => e.kind === 'calls')) + .map((e) => `${graph.getNode(e.target)!.qualifiedName}@${graph.getNode(e.target)!.filePath}`) + .sort(); + + const convergeAfter = async (edit: Record) => { + await cg!.sync(); + const synced = edgesOf(cg!); + cg!.close(); + fs.rmSync(path.join(tempDir, '.codegraph'), { recursive: true, force: true }); + cg = await CodeGraph.init(tempDir, { index: true }); + expect(synced).toEqual(edgesOf(cg)); + void edit; + return synced; + }; + + it('re-types an attribute when a factory in another file changes its return', async () => { + write({ + 'client.py': 'class Client:\n def send(self):\n pass\n\nclass Fake:\n def send(self):\n pass\n', + 'factory.py': 'from client import Client, Fake\n\ndef make():\n return Client()\n', + 'svc.py': 'from factory import make\n\nclass Svc:\n def __init__(self):\n self.client = make()\n\n def run(self):\n self.client.send()\n', + }); + cg = await CodeGraph.init(tempDir, { index: true }); + expect(edgesOf(cg)).toEqual(['Client::send@client.py']); + const edit = { 'factory.py': 'from client import Client, Fake\n\ndef make():\n return Fake()\n' }; + write(edit); + expect(await convergeAfter(edit)).toEqual(['Fake::send@client.py']); + }); + + it('types an attribute once an untyped factory gains an annotation', async () => { + write({ + 'client.py': 'class Client:\n def send(self):\n pass\n', + 'other.py': DISTRACTOR, + 'factory.py': 'from client import Client\n\ndef make(kind):\n return kind()\n', + 'svc.py': 'from factory import make\n\nclass Svc:\n def __init__(self):\n self.conn = make(1)\n\n def run(self):\n self.conn.send()\n', + }); + cg = await CodeGraph.init(tempDir, { index: true }); + expect(edgesOf(cg)).toEqual([]); + const edit = { 'factory.py': 'from client import Client\n\ndef make(kind) -> Client:\n return kind()\n' }; + write(edit); + expect(await convergeAfter(edit)).toEqual(['Client::send@client.py']); + }); + + it('re-types a guess by name once the factory gains an annotation', async () => { + // Untyped, `self.client` is guessed by its name as a `Client`. The guess + // records the factory's file, so annotating it `-> Fake` re-opens it. + write({ + 'client.py': 'class Client:\n def send(self):\n pass\n\nclass Fake:\n def send(self):\n pass\n', + 'factory.py': 'from client import Client, Fake\n\ndef make(kind):\n return kind()\n', + 'svc.py': 'from factory import make\n\nclass Svc:\n def __init__(self):\n self.client = make(1)\n\n def run(self):\n self.client.send()\n', + }); + cg = await CodeGraph.init(tempDir, { index: true }); + expect(edgesOf(cg)).toEqual(['Client::send@client.py']); + const edit = { 'factory.py': 'from client import Client, Fake\n\ndef make(kind) -> Fake:\n return kind()\n' }; + write(edit); + expect(await convergeAfter(edit)).toEqual(['Fake::send@client.py']); + }); + + it('re-types an inherited attribute when the base __init__ changes', async () => { + write({ + 'repo.py': 'class Repo:\n def save(self):\n pass\n\nclass Cache:\n def save(self):\n pass\n', + 'base.py': 'from repo import Repo, Cache\n\nclass Base:\n def __init__(self):\n self.repo = Repo()\n', + 'child.py': 'from base import Base\n\nclass Child(Base):\n def run(self):\n self.repo.save()\n', + }); + cg = await CodeGraph.init(tempDir, { index: true }); + expect(edgesOf(cg)).toEqual(['Repo::save@repo.py']); + const edit = { 'base.py': 'from repo import Repo, Cache\n\nclass Base:\n def __init__(self):\n self.repo = Cache()\n' }; + write(edit); + expect(await convergeAfter(edit)).toEqual(['Cache::save@repo.py']); + }); + + it('drops an inherited edge when the base class file is deleted', async () => { + write({ + 'repo.py': 'class Repo:\n def save(self):\n pass\n', + 'other.py': DISTRACTOR, + 'base.py': 'from repo import Repo\n\nclass Base:\n def __init__(self):\n self.repo = Repo()\n', + 'child.py': 'from base import Base\n\nclass Child(Base):\n def run(self):\n self.repo.save()\n', + }); + cg = await CodeGraph.init(tempDir, { index: true }); + expect(edgesOf(cg)).toEqual(['Repo::save@repo.py']); + fs.rmSync(path.join(tempDir, 'base.py')); + expect(await convergeAfter({})).toEqual([]); + }); + + it('re-opens nothing when an edit touches no file an answer was read from', async () => { + write({ + 'repo.py': 'class Repo:\n def save(self):\n pass\n', + 'base.py': 'from repo import Repo\n\nclass Base:\n def __init__(self):\n self.repo = Repo()\n', + 'child.py': 'from base import Base\n\nclass Child(Base):\n def run(self):\n self.repo.save()\n', + 'unrelated.py': 'def f():\n return 1\n', + }); + cg = await CodeGraph.init(tempDir, { index: true }); + const spy = vi.spyOn((cg as unknown as { orchestrator: { resurrectTypeDependentEdges: (...a: unknown[]) => string[] } }).orchestrator, 'resurrectTypeDependentEdges'); + write({ 'unrelated.py': 'def f():\n return 2\n' }); + await cg.sync(); + expect(spy).toHaveBeenCalledTimes(1); + expect(spy.mock.results[0]!.value).toEqual([]); + expect(edgesOf(cg)).toEqual(['Repo::save@repo.py']); + }); + + it('retries a failed call in a file whose other attribute edge was re-opened', async () => { + write({ + 'client.py': 'class Client:\n def send(self):\n pass\n\nclass Fake:\n def send(self):\n pass\n', + 'f.py': 'from client import Client, Fake\n\ndef make():\n return Client()\n\ndef make2(k):\n return k()\n', + // Two classes in one file, both `run`: A's edge is re-opened by the + // edit, B's call only now resolves. + 'svc.py': 'from f import make, make2\n\nclass A:\n def __init__(self):\n self.a = make()\n\n def run(self):\n self.a.send()\n\n' + + 'class B:\n def __init__(self):\n self.b = make2(1)\n\n def run(self):\n self.b.send()\n', + }); + cg = await CodeGraph.init(tempDir, { index: true }); + expect(edgesOf(cg)).toEqual(['Client::send@client.py']); + write({ 'f.py': 'from client import Client, Fake\n\ndef make():\n return Fake()\n\ndef make2(k) -> Client:\n return k()\n' }); + expect(await convergeAfter({})).toEqual(['Client::send@client.py', 'Fake::send@client.py']); + }); + + it('types an attribute once the factory it imports comes into existence', async () => { + write({ + 'client.py': 'class Client:\n def send(self):\n pass\n', + 'other.py': DISTRACTOR, + 'f.py': 'from client import Client\n\ndef make_v2():\n return Client()\n', + 'svc.py': 'from f import make\n\nclass Svc:\n def __init__(self):\n self.c = make()\n\n def run(self):\n self.c.send()\n', + }); + cg = await CodeGraph.init(tempDir, { index: true }); + expect(edgesOf(cg)).toEqual([]); + write({ 'f.py': 'from client import Client\n\ndef make():\n return Client()\n' }); + expect(await convergeAfter({})).toEqual(['Client::send@client.py']); + }); + + it('narrows the retry to what the edited file can reach when there are too many candidates', async () => { + const svc = 'from client import make, helper\n\nclass Svc:\n def __init__(self, x, y):\n' + + ' self.c = make(1)\n self.d = x\n self.e = y\n helper()\n\n' + + ' def run(self):\n self.c.send()\n self.d.send()\n self.e.send()\n'; + write({ + 'client.py': 'class Client:\n def send(self):\n pass\n\ndef make(k):\n return k()\n\ndef helper():\n pass\n', + 'other.py': DISTRACTOR, + 'svc.py': svc, + }); + process.env.CODEGRAPH_PY_ATTR_RETRY_CEILING = '1'; + try { + cg = await CodeGraph.init(tempDir, { index: true }); + expect(edgesOf(cg)).toEqual([]); + // Three failed candidates, over the ceiling of 1: only `self.c` — + // assigned from `make`, which the edited file defines — is kept. + write({ 'client.py': 'class Client:\n def send(self):\n pass\n\ndef make(k) -> Client:\n return k()\n\ndef helper():\n pass\n' }); + expect(await convergeAfter({})).toEqual(['Client::send@client.py']); + } finally { + delete process.env.CODEGRAPH_PY_ATTR_RETRY_CEILING; + } + }); + + it('drops an inherited edge when the base stops typing the attribute', async () => { + // `store`, named after no class, so no guess by name replaces the edge. + write({ + 'repo.py': 'class Repo:\n def save(self):\n pass\n', + 'other.py': DISTRACTOR, + 'base.py': 'from repo import Repo\n\nclass Base:\n def __init__(self):\n self.store = Repo()\n', + 'child.py': 'from base import Base\n\nclass Child(Base):\n def run(self):\n self.store.save()\n', + }); + cg = await CodeGraph.init(tempDir, { index: true }); + expect(edgesOf(cg)).toEqual(['Repo::save@repo.py']); + const edit = { 'base.py': 'from repo import Repo\n\nclass Base:\n def __init__(self, repo):\n self.store = repo\n' }; + write(edit); + expect(await convergeAfter(edit)).toEqual([]); + }); + }); + describe('review round 2026-09-30', () => { + it('sees a return written on the same line as its condition', async () => { + write({ + 'a.py': 'class A:\n def send(self):\n pass\n', + 'b.py': 'class B:\n def send(self):\n pass\n', + 'make.py': 'from a import A\nfrom b import B\n\ndef make(k):\n if k: return A()\n return B()\n', + 'svc.py': 'from make import make\n\nclass Svc:\n def __init__(self):\n self.c = make(1)\n\n def run(self):\n self.c.send()\n', + }); + cg = await CodeGraph.init(tempDir, { index: true }); + expect(callsFrom('run')).toEqual([]); + }); + + it('still agrees when every one-line return builds the same class', async () => { + write({ + 'a.py': 'class A:\n def send(self):\n pass\n', + 'other.py': DISTRACTOR, + 'make.py': 'from a import A\n\ndef make(k):\n if k: return A()\n else: return A()\n', + 'svc.py': 'from make import make\n\nclass Svc:\n def __init__(self):\n self.c = make(1)\n\n def run(self):\n self.c.send()\n', + }); + cg = await CodeGraph.init(tempDir, { index: true }); + expect(callsFrom('run')).toEqual(['A::send@a.py']); + }); + + it('does not type a chained call through a class name the method rebinds', async () => { + write({ + 'backend.py': 'class Backend:\n def run(self):\n pass\n', + 'svc.py': 'from backend import Backend\n\ndef load():\n return object\n\nclass Svc:\n def go(self):\n Backend = load()\n Backend().run()\n', + }); + cg = await CodeGraph.init(tempDir, { index: true }); + expect(callsFrom('go').filter((c) => c.startsWith('Backend::'))).toEqual([]); + }); + + it('does not type a chained call through a parameter that shadows a factory', async () => { + write({ + 'client.py': 'class Client:\n def send(self):\n pass\n', + 'make.py': 'from client import Client\n\ndef make():\n return Client()\n', + 'svc.py': 'from make import make\n\nclass Svc:\n def go(self, make):\n make().send()\n', + }); + cg = await CodeGraph.init(tempDir, { index: true }); + expect(callsFrom('go')).not.toContain('Client::send@client.py'); + }); + }); + describe('review round 2026-10-01', () => { + it('a return or yield inside a string is not a statement', async () => { + write({ + 'a.py': 'class A:\n def send(self):\n pass\n', + 'other.py': DISTRACTOR, + 'fac.py': 'from a import A\n\nK = 1\n\ndef make():\n if not K:\n raise ValueError("bad config: return value missing")\n print("will yield")\n return A()\n', + 'svc.py': 'from fac import make\n\nclass Svc:\n def __init__(self):\n self.c = make()\n\n def run(self):\n self.c.send()\n', + }); + cg = await CodeGraph.init(tempDir, { index: true }); + expect(callsFrom('run')).toEqual(['A::send@a.py']); + }); + + it('a factory the calling method assigns is typed like the class body\'s', async () => { + write({ + 'a.py': 'class A:\n def send(self):\n pass\n', + 'other.py': DISTRACTOR, + 'fac.py': 'from a import A\n\ndef make() -> A:\n return A()\n', + 'svc.py': 'from fac import make\n\nclass Svc:\n def run(self):\n self.c = make()\n self.c.send()\n', + }); + cg = await CodeGraph.init(tempDir, { index: true }); + expect(callsFrom('run').filter((c) => c.includes('::send@'))).toEqual(['A::send@a.py']); + }); + }); +}); diff --git a/codegraph-kernel/src/python.rs b/codegraph-kernel/src/python.rs index 1fe325f05..bdec80f99 100644 --- a/codegraph-kernel/src/python.rs +++ b/codegraph-kernel/src/python.rs @@ -731,6 +731,30 @@ impl<'t> Walker<'t> { // TreeSitterExtractor.extractCall. let Some(inner) = self.plain_inner_callee(r) else { return }; callee_name = format!("{inner}().{method_name}"); + } else if let Some(r) = receiver { + // Any receiver shape the arms above do not claim — attribute + // chain (`self.data.append`) and subscript (`d[k].append`). + // The call-chain shape (`rows.setdefault(k, []).append(x)`) + // is taken by the `().` arm above (#1748), + // which is matched first. + // These used to fall through to a BARE method_name. A bare name + // matching a common list/dict/str method (`append`, `get`, + // `update`, ...) exact-matches an unrelated top-level project + // function sharing that name, fabricating a call edge (#66, + // same class as #1230/#1276). Keep the receiver's source text + // as a qualifier so the ref can only resolve through + // import/module-member resolution, never the bare-name + // fallback; an unresolvable qualifier is a silent miss, never + // a wrong edge. Mirrors TreeSitterExtractor.extractCall's + // python branch — the whitespace collapse is JS `\s` + // semantics, not `char::is_whitespace`, so both arms emit the + // same bytes for the parity sweep. + let receiver_text = collapse_js_whitespace(self.text(r)); + callee_name = if receiver_text.is_empty() { + method_name.to_string() + } else { + format!("{receiver_text}.{method_name}") + }; } else { callee_name = method_name.to_string(); } @@ -1105,6 +1129,35 @@ fn is_stoplisted(name: &str) -> bool { ) } +/// `s.replace(/\s+/g, ' ').trim()` with JavaScript's `\s` class, so a receiver +/// qualifier is byte-identical to the one TreeSitterExtractor emits. Rust's +/// `char::is_whitespace` is NOT the same set: it counts U+0085 (NEL), which JS +/// `\s` does not, and omits U+FEFF (ZWNBSP), which JS `\s` includes. +fn collapse_js_whitespace(s: &str) -> String { + fn is_js_space(c: char) -> bool { + matches!( + c, + '\t' | '\n' | '\u{b}' | '\u{c}' | '\r' | ' ' | '\u{a0}' | '\u{feff}' + | '\u{2028}' | '\u{2029}' + | '\u{1680}' | '\u{2000}'..='\u{200a}' | '\u{202f}' | '\u{205f}' | '\u{3000}' + ) + } + let mut out = String::with_capacity(s.len()); + let mut in_space = false; + for c in s.chars() { + if is_js_space(c) { + in_space = true; + } else { + if in_space && !out.is_empty() { + out.push(' '); + } + in_space = false; + out.push(c); + } + } + out +} + /// LITERAL_RECEIVER_TYPES membership (shared table; python names among them). fn is_literal_receiver(kind: &str) -> bool { matches!( diff --git a/src/db/queries.ts b/src/db/queries.ts index 2dc7b24c2..fe2f73d9e 100644 --- a/src/db/queries.ts +++ b/src/db/queries.ts @@ -213,6 +213,11 @@ function rowToNode(row: NodeRow): Node { /** * Convert database row to Edge object */ +/** A failed python ref through one attribute (`self.c.send`) or a call receiver (`make().send`). */ +const FAILED_PY_ATTR_REF = `status = 'failed' AND language = 'python' + AND reference_name NOT GLOB '*.*.*.*' + AND (reference_name GLOB '*.*.*' OR reference_name GLOB '*().*')`; + function rowToEdge(row: EdgeRow): Edge { return { source: row.source, @@ -4713,6 +4718,129 @@ export class QueryBuilder { * Replace resolution edges with their original unresolved references as one * transaction. If ref insertion fails, the edge deletion is rolled back. */ + /** + * Resolution edges whose answer was read from other files + * (`metadata.typeFrom` — python attribute types, see + * `resurrectTypeDependentEdges`), with the source file context a + * resurrection needs. One scan; only python attribute calls carry the key. + */ + getEdgesWithTypeFrom(): Array { + const rows = this.db + .prepare( + `SELECT e.*, src.file_path AS source_file_path, src.language AS source_language + FROM edges e + JOIN nodes src ON src.id = e.source + WHERE e.metadata LIKE '%"typeFrom"%'` + ) + .all() as Array; + return rows.map((row) => ({ + ...rowToEdge(row), + edgeId: row.id, + sourceFilePath: row.source_file_path, + sourceLanguage: row.source_language, + })); + } + + /** + * The python files whose failed attribute-chain refs (`self.client.send`, + * `make().send`) an edit to `filePaths` can affect. Such an answer is read + * only from files reached through imports — the owner class, its bases, a + * factory and the class it returns are each looked up through the imports + * of the file before — so these are the files that import a touched file, + * directly or within `depth` hops, and the touched files themselves. + */ + getPythonImporterClosure(filePaths: string[], depth: number = 6): string[] { + const closure = new Set(filePaths); + let frontier = [...filePaths]; + for (let hop = 0; hop < depth && frontier.length > 0; hop++) { + const next: string[] = []; + for (let i = 0; i < frontier.length; i += SQLITE_PARAM_CHUNK_SIZE) { + const chunk = frontier.slice(i, i + SQLITE_PARAM_CHUNK_SIZE); + const placeholders = chunk.map(() => '?').join(','); + const rows = this.db + .prepare( + `SELECT DISTINCT src.file_path AS file_path + FROM edges e + JOIN nodes tgt ON tgt.id = e.target + JOIN nodes src ON src.id = e.source + WHERE e.kind = 'imports' AND tgt.file_path IN (${placeholders}) AND src.language = 'python'` + ) + .all(...chunk) as Array<{ file_path: string }>; + for (const row of rows) { + if (!closure.has(row.file_path)) { + closure.add(row.file_path); + next.push(row.file_path); + } + } + } + frontier = next; + } + return [...closure]; + } + + /** + * The failed python attribute-chain refs — one attribute hop + * (`self.client.send`) or a call receiver (`make().send`) — in `filePaths`, + * as just the columns a sync needs to choose among them. + */ + getFailedPythonAttrRefKeys(filePaths: string[]): Array<{ id: number; fromNodeId: string; referenceName: string; filePath: string }> { + // One scan, filtered here: a closure is often most of the project, and a + // `file_path IN (...)` per chunk re-runs the GLOB filters over every row. + const wanted = new Set(filePaths); + const rows = this.db + .prepare(`SELECT id, from_node_id, reference_name, file_path FROM unresolved_refs WHERE ${FAILED_PY_ATTR_REF}`) + .all() as Array<{ id: number; from_node_id: string; reference_name: string; file_path: string }>; + const out: Array<{ id: number; fromNodeId: string; referenceName: string; filePath: string }> = []; + for (const r of rows) { + if (wanted.has(r.file_path)) out.push({ id: r.id, fromNodeId: r.from_node_id, referenceName: r.reference_name, filePath: r.file_path }); + } + return out; + } + + /** Unresolved refs by row id. */ + getUnresolvedReferencesByIds(ids: number[]): UnresolvedReference[] { + const rows: UnresolvedRefRow[] = []; + for (let i = 0; i < ids.length; i += SQLITE_PARAM_CHUNK_SIZE) { + const chunk = ids.slice(i, i + SQLITE_PARAM_CHUNK_SIZE); + const placeholders = chunk.map(() => '?').join(','); + const chunkRows = this.db + .prepare(`SELECT * FROM unresolved_refs WHERE id IN (${placeholders})`) + .all(...chunk) as UnresolvedRefRow[]; + for (const row of chunkRows) rows.push(row); + } + return rows.map((row) => ({ + fromNodeId: row.from_node_id, + referenceName: row.reference_name, + referenceKind: row.reference_kind as EdgeKind, + line: row.line, + column: row.col, + candidates: row.candidates ? safeJsonParse(row.candidates, undefined) : undefined, + filePath: row.file_path, + language: row.language as Language, + rowId: row.id, + })); + } + + /** + * The python files with an edge of one of `kinds` into a node of `filePath` + * whose kind is one of `targetKinds` — who subclasses, calls or constructs + * what a file defines. + */ + getPythonFilesWithEdgesInto(filePath: string, kinds: string[], targetKinds: string[]): string[] { + const kp = kinds.map(() => '?').join(','); + const tp = targetKinds.map(() => '?').join(','); + return (this.db + .prepare( + `SELECT DISTINCT src.file_path AS file_path + FROM edges e + JOIN nodes tgt ON tgt.id = e.target + JOIN nodes src ON src.id = e.source + WHERE tgt.file_path = ? AND e.kind IN (${kp}) AND tgt.kind IN (${tp}) + AND src.language = 'python' AND src.file_path != ?` + ) + .all(filePath, ...kinds, ...targetKinds, filePath) as Array<{ file_path: string }>).map((r) => r.file_path); + } + replaceResolutionEdgesWithUnresolvedRefs( edgeIds: number[], refs: UnresolvedReference[] diff --git a/src/extraction/index.ts b/src/extraction/index.ts index 76a41ff0a..001648bd8 100644 --- a/src/extraction/index.ts +++ b/src/extraction/index.ts @@ -164,6 +164,8 @@ export interface SyncResult { * nothing downstream. */ definitionDelta?: string[]; + /** Tracked files this sync found gone and deleted from the index. */ + removedFilePaths?: string[]; } /** @@ -3337,6 +3339,39 @@ export class ExtractionOrchestrator { return refs.length; } + /** + * Re-open resolution edges whose answer was READ from a file this sync + * touched, although neither end of the edge changed: a python attribute typed + * through a factory or a base class in another file records those files as + * `metadata.typeFrom`. The definition delta cannot see these — editing what a + * factory returns, or what a base `__init__` assigns, adds or removes no + * definition — so without this a synced index keeps the old type forever. + * The edges go back to pending refs; returns the files they come from, whose + * pending rows the caller resolves. + */ + resurrectTypeDependentEdges(touchedPaths: string[], changedFilePaths: string[]): string[] { + const touched = new Set(touchedPaths.filter((p) => /\.pyi?$/.test(p))); + if (touched.size === 0) return []; + const alreadyFresh = new Set(changedFilePaths); + + const edgeIds: number[] = []; + const refs: UnresolvedReference[] = []; + const files = new Set(); + for (const e of this.queries.getEdgesWithTypeFrom()) { + if (alreadyFresh.has(e.sourceFilePath)) continue; + const from = e.metadata?.typeFrom; + if (!Array.isArray(from) || !from.some((f) => typeof f === 'string' && touched.has(f))) continue; + const ref = resurrectRefFromDroppedEdge(e); + if (!ref) continue; + edgeIds.push(e.edgeId); + refs.push(ref); + files.add(e.sourceFilePath); + } + if (refs.length === 0) return []; + this.queries.replaceResolutionEdgesWithUnresolvedRefs(edgeIds, refs); + return [...files]; + } + /** * Re-open every resolution a language made, for the sync's sweep to redo * against the current graph — the answer when an input to that language's @@ -3466,6 +3501,7 @@ export class ExtractionOrchestrator { let nodesUpdated = 0; const changedFilePaths: string[] = []; const addedFilePaths: string[] = []; + const removedFilePaths: string[] = []; // `file\0name` definition pairs for the files this sync touches, sampled // BEFORE their nodes are replaced/deleted. Compared against the post-store // pairs below to derive `definitionDelta` (CG-33). @@ -3575,6 +3611,7 @@ export class ExtractionOrchestrator { } onFileChange?.(tracked.path); this.queries.deleteFile(tracked.path); + removedFilePaths.push(tracked.path); filesRemoved++; }; const removed: FileRecord[] = []; @@ -3735,6 +3772,7 @@ export class ExtractionOrchestrator { ...(addedFilePaths.length > 0 ? { addedFilePaths } : {}), ...(failedFilePaths.length > 0 ? { failedFilePaths } : {}), definitionDelta: definitionDelta.length > 0 ? definitionDelta : undefined, + ...(removedFilePaths.length > 0 ? { removedFilePaths } : {}), }; } diff --git a/src/extraction/tree-sitter.ts b/src/extraction/tree-sitter.ts index 091aa9ee7..e3c7b0683 100644 --- a/src/extraction/tree-sitter.ts +++ b/src/extraction/tree-sitter.ts @@ -5779,6 +5779,21 @@ export class TreeSitterExtractor { // Emit nothing: a silent miss, never a wrong edge. Mirrored in // the kernel's extract_call (tsjs/extractors.rs). return; + } else if (this.language === 'python' && receiver) { + // Any receiver shape the branches above do not claim — attribute + // chain (`self.data.append`) and subscript (`d[k].append`). The + // call-chain shape (`rows.setdefault(k, []).append(x)`) is taken + // by the `().` branch above (#1748), which runs + // first. These previously fell through to a bare methodName. A + // bare name matching a common + // list/dict/str method (`append`, `get`, `update`, ...) can exact-match + // an unrelated top-level project function sharing that name, fabricating + // a call edge (#66, same class as #1230/#1276). Keep the receiver's + // source text as a qualifier so the ref can only resolve through + // import/module-member resolution, never the bare-name fallback; an + // unresolvable qualifier is a silent miss, never a wrong edge. + const receiverText = getNodeText(receiver, this.source).replace(/\s+/g, ' ').trim(); + calleeName = receiverText ? `${receiverText}.${methodName}` : methodName; } else { calleeName = methodName; } diff --git a/src/index.ts b/src/index.ts index bb907095e..e17a36571 100644 --- a/src/index.ts +++ b/src/index.ts @@ -1347,6 +1347,32 @@ export class CodeGraph { result.pendingRefsUnresolved = recovery.stats.unresolved; } + // Python attribute types read from OTHER files — a factory's return, a + // base class's `__init__` — change with a body-only edit the definition + // delta above cannot see. Re-open the edges whose answer was read from a + // touched file, and retry the failed attribute calls in files that + // import one; both resolve right here. After the orphan sweep, so the + // import edges this sync created (a caller whose `from f import make` + // only now resolves) are there to find the importers by. + if (process.env.CODEGRAPH_NO_REBIND !== '1') { + const touched = [...(result.changedFilePaths ?? []), ...(result.removedFilePaths ?? [])]; + const touchedPy = touched.filter((p) => /\.pyi?$/.test(p)); + if (touchedPy.length > 0) { + const tTypes = Date.now(); + const reopenedFiles = this.orchestrator.resurrectTypeDependentEdges(touchedPy, result.changedFilePaths ?? []); + // CODEGRAPH_PY_ATTR_RETRY_CEILING overrides the candidate ceiling (tests). + const ceiling = Number(process.env.CODEGRAPH_PY_ATTR_RETRY_CEILING ?? '') || 2000; + const failed = this.resolver.pythonAttrRetryCandidates(touchedPy, ceiling); + const retry = [...this.queries.getUnresolvedReferencesByFiles(reopenedFiles), ...(failed ?? [])]; + if (retry.length > 0) await this.resolver.resolveAndPersistListYielding(retry); + if (process.env.CODEGRAPH_SYNTH_TIMINGS) { + console.error( + `[phase-timing] sync-type-rebind: ${Date.now() - tTypes}ms (${retry.length} refs${failed ? '' : ', failed-ref fan-in over ceiling'})` + ); + } + } + } + if (filesChanged || orphanCount > 0) { // Second pass: chained calls whose method lives on a supertype the // receiver conforms to (protocol-extension / inherited). Needs the diff --git a/src/resolution/import-resolver.ts b/src/resolution/import-resolver.ts index a401af1dd..96df63e24 100644 --- a/src/resolution/import-resolver.ts +++ b/src/resolution/import-resolver.ts @@ -3002,7 +3002,12 @@ function resolvePythonModuleMember( } for (const imp of imports) { - if (imp.localName !== receiver) continue; + // `import pkg.mod` binds `pkg`, so a call spells the whole path out — + // `pkg.mod.func()`. The mapping names the module by its last segment, which + // never matches that receiver; match the full dotted source instead. + const spelledOut = imp.isNamespace && imp.localName === imp.source.split('.').pop() + && imp.source.includes('.') && ref.referenceName.startsWith(`${imp.source}.`); + if (imp.localName !== receiver && !spelledOut) continue; // `import mod` / `import numpy as np` bind the module at `source` itself; // `from . import certs` / `from pkg import mod` bind a SUBMODULE whose @@ -3015,27 +3020,62 @@ function resolvePythonModuleMember( // form (where the two names coincide) worked (#1626). For an unaliased // import the two are identical, so this changes nothing there. const moduleName = imp.exportedName === '*' ? imp.localName : imp.exportedName; - const modulePath = imp.isNamespace + let modulePath = imp.isNamespace ? imp.source : imp.source.endsWith('.') ? imp.source + moduleName : imp.source + '.' + moduleName; + const rest = ref.referenceName + .substring(spelledOut ? imp.source.length + 1 : dotIdx + 1) + .split('.'); if (!imp.isNamespace && pythonPackageBinds(imp, ref.filePath, context)) continue; - const resolvedPath = resolveImportPath(modulePath, ref.filePath, ref.language, context); + let resolvedPath = resolveImportPath(modulePath, ref.filePath, ref.language, context); if (!resolvedPath || resolvedPath === ref.filePath) continue; - // The member as the module binds it: a module-level def (never a method — - // `mod.foo` must not land on a same-named class method, and never a nested - // def), or a name the module re-exports from elsewhere (`pkg/__init__.py` - // doing `from .impl import foo`). - const target = findExportedSymbol( - resolvedPath, - { isDefault: false, isNamespace: false, exportedName: member, memberName: null }, + // The symbol `name` as the module at `filePath` binds it: a module-level + // def (never a method — `mod.foo` must not land on a same-named class + // method, and never a nested def), or a name the module re-exports from + // elsewhere (`pkg/__init__.py` doing `from .impl import foo`). + const bound = (filePath: string, name: string): Node | undefined => findExportedSymbol( + filePath, + { isDefault: false, isNamespace: false, exportedName: name, memberName: null }, 'python', context, new Set() ); + + // A CALL through a longer path — `utils.helpers.do_x()`, `mod.Klass.run()` + // — names submodules and then one symbol. Descend while the next segment + // is a module file; what is left must be a top-level callable, or a class + // and one of its methods. Anything else is not something this path can + // prove, so it is no edge — never the first segment's symbol, which for + // `mod.Klass.run()` would record a call to the class. Other reference + // kinds keep the single-member lookup below. + if (ref.referenceKind === 'calls' && rest.length > 1) { + while (rest.length > 1) { + // `modulePath` is spelled from the importer (`.utils.helpers`), so it + // resolves from there, not from the module file just reached. + const sub = resolveImportPath(`${modulePath}.${rest[0]}`, ref.filePath, ref.language, context); + if (!sub) break; + modulePath = `${modulePath}.${rest.shift()}`; + resolvedPath = sub; + } + const head = bound(resolvedPath, rest[0]!); + let target: Node | undefined; + if (rest.length === 1) { + target = head && (head.kind === 'function' || head.kind === 'class') ? head : undefined; + } else if (rest.length === 2 && head?.kind === 'class') { + target = context.getNodesInFile(head.filePath).find((n) => + n.kind === 'method' && n.name === rest[1] && n.qualifiedName === `${head.qualifiedName}::${rest[1]}`); + } + if (target) { + return { original: ref, targetNodeId: target.id, confidence: 0.85, resolvedBy: 'import' }; + } + continue; + } + + const target = bound(resolvedPath, rest[0]!); if (target) { return { original: ref, targetNodeId: target.id, confidence: 0.85, resolvedBy: 'import' }; } @@ -3705,8 +3745,10 @@ function resolveImportedInstanceMember( ): ResolvedRef | null { if (ref.referenceKind !== 'calls') return null; if (value.kind !== 'constant' && value.kind !== 'variable') return null; - const member = ref.referenceName.slice(localName.length + 1).split('.')[0]; - if (!member) return null; + // The method called on the value itself: `app.jinja_env.filters.keys()` + // calls `keys`, no method of the app's type — never `jinja_env`. + const member = ref.referenceName.slice(localName.length + 1); + if (!member || member.includes('.')) return null; const source = context.readFile(value.filePath); if (!source) return null; diff --git a/src/resolution/index.ts b/src/resolution/index.ts index 41dc786ae..80d3bae76 100644 --- a/src/resolution/index.ts +++ b/src/resolution/index.ts @@ -21,7 +21,7 @@ import { isImportableKind, CPP_DEFINE_SIGNATURE, } from './types'; -import { isPythonSelfCall, matchJsStoreBindingCall, isUnresolvedJsMemberCall, matchObjectPathCall, thisScopeCaller, isVisibleAcrossFiles, matchReference, matchByFilePath, matchFunctionRef, matchDottedCallChain, matchScopedCallChain, matchMethodCall, sameLanguageFamily, crossesCodeBoundary, gateLanguageMatch, dumpNameMatcherProfile, clearNameMatcherMemos, isRustNameInScope, isRustForeignImplCallable, CASE_INSENSITIVE_LANGUAGES, isDartMemberRead, matchDartMemberRead, isDartChainLink, matchDartChainLink, isDartAnnotation, matchDartAnnotation, isStdMethodName, isGoUnknownQualified, isGoProjectLocalCall, GO_STDLIB_PACKAGES, isGoBareName, goTypePositionTarget, GO_TYPE_KINDS, matchGoAssertedCall, isGoAssertedLocal, isGoLocallyBound, isPythonFromImportModule } from './name-matcher'; +import { isPythonSelfCall, matchJsStoreBindingCall, isUnresolvedJsMemberCall, matchObjectPathCall, thisScopeCaller, isVisibleAcrossFiles, matchReference, matchByFilePath, matchFunctionRef, matchDottedCallChain, matchScopedCallChain, matchMethodCall, sameLanguageFamily, crossesCodeBoundary, gateLanguageMatch, dumpNameMatcherProfile, clearNameMatcherMemos, isRustNameInScope, isRustForeignImplCallable, CASE_INSENSITIVE_LANGUAGES, isDartMemberRead, matchDartMemberRead, isDartChainLink, matchDartChainLink, isDartAnnotation, matchDartAnnotation, isStdMethodName, isGoUnknownQualified, isGoProjectLocalCall, GO_STDLIB_PACKAGES, isGoBareName, goTypePositionTarget, GO_TYPE_KINDS, matchGoAssertedCall, isGoAssertedLocal, isGoLocallyBound, isPythonFromImportModule, pythonAttrRefResolves } from './name-matcher'; import { isVisibleCppMacro, clearCppMacroVisibility } from './cpp-macro-visibility'; import { isCppConstructorRef, matchCppConstructor } from './cpp-constructor'; import { isCppSupertypeRef, matchCppSupertype, clearCppSupertypeMemos } from './cpp-supertypes'; @@ -45,6 +45,7 @@ import { loadProjectAliases, type AliasMap } from './path-aliases'; import { findGoModuleForImport, goModulePackageDir, loadGoModule, type GoModule } from './go-module'; import { loadWorkspacePackages, type WorkspacePackages } from './workspace-packages'; import { logDebug } from '../errors'; +import { stripCommentsForRegex } from './strip-comments'; import { lexicalPathWithinRoot } from '../utils'; import type { CrossFileNodes, ReExport } from './types'; import { resurrectRefFromDroppedEdge } from './resurrect-ref'; @@ -1230,6 +1231,97 @@ export class ReferenceResolver { * Calls that land on an alias binding then forward once to the callable * the alias names (see ./alias-binding), regardless of the strategy. */ + /** + * The failed python attribute-chain refs that an edit to `touched` makes + * resolvable now — what a sync retries after a factory or a base class + * changed in another file (`pythonAttrRefResolves` answers each one). + * + * The candidates are the refs in the files that import a touched file + * within a few hops, counted once per (calling symbol, name). When there + * are more than `ceiling` of them — an edit to a module most of the project + * imports — only those the edit can plausibly reach are kept: an attribute + * a touched file assigns, a call receiver it defines, a class that + * subclasses one of its classes, or an attribute assigned from a call to + * something it defines. Null when even that is over the ceiling; those refs + * wait for a full index. + * + * Past the ceiling, then, these become resolvable only at the next full + * index: a factory two hops from the edit, a subclass two hops away, and an + * attribute typed by an annotation — a parameter `c: Client`, or a + * class-level `c: Client` in a class that only gains the base or import that + * resolves `Client` in the touched file. + */ + pythonAttrRetryCandidates(touched: string[], ceiling: number = 2000): UnresolvedReference[] | null { + const touchedSet = new Set(touched); + const closure = this.queries.getPythonImporterClosure(touched).filter((f) => !touchedSet.has(f)); + // One answer per (calling symbol, name): the same call written twice in + // one function resolves the same way, so it is asked — and counted — once. + const keyOf = (ref: { fromNodeId: string; referenceName: string }) => `${ref.fromNodeId}\0${ref.referenceName}`; + const resolvable = (refs: UnresolvedReference[]) => { + const answers = new Map(); + return refs.filter((ref) => { + if (!ref.filePath || !ref.language) return false; + const key = keyOf(ref); + let ok = answers.get(key); + if (ok === undefined) answers.set(key, (ok = pythonAttrRefResolves(ref as UnresolvedRef, this.context))); + return ok; + }); + }; + const candidates = this.queries.getFailedPythonAttrRefKeys(closure); + const load = (keys: typeof candidates) => this.queries.getUnresolvedReferencesByIds(keys.map((k) => k.id)); + if (new Set(candidates.map(keyOf)).size <= ceiling) return resolvable(load(candidates)); + + const assigned = new Set(); + const defined = new Set(); + const subclassing = new Set(); + const calling = new Set(); + for (const file of touched) { + const source = this.context.readFile(file); + if (source) { + const text = stripCommentsForRegex(source, 'python'); + for (const m of text.matchAll(/\b(?:self|cls)\.(\w+)\s*(?::[^=\n]*)?=(?!=)|^[ \t]+(\w+)\s*(?::[^=\n]*)?=(?!=)/gm)) { + assigned.add((m[1] ?? m[2])!); + } + } + for (const n of this.context.getNodesInFile(file)) { + if (n.kind === 'class' || n.kind === 'function' || n.kind === 'method') defined.add(n.name); + } + for (const f of this.queries.getPythonFilesWithEdgesInto(file, ['extends'], ['class'])) subclassing.add(f); + for (const f of this.queries.getPythonFilesWithEdgesInto(file, ['calls', 'instantiates'], ['function', 'method', 'class'])) { + calling.add(f); + } + } + + const callees = new Map>>(); // file → attr → callee names + const assignedCallees = (file: string, attr: string): Set => { + let byAttr = callees.get(file); + if (!byAttr) { + byAttr = new Map(); + const source = this.context.readFile(file); + const text = source ? stripCommentsForRegex(source, 'python') : ''; + for (const m of text.matchAll(/\b(\w+)\s*(?::[^=\n]*)?=(?!=)\s*(?:await\s+)?(?:[\w.]*\.)?(\w+)\s*\(/g)) { + let set = byAttr.get(m[1]!); + if (!set) byAttr.set(m[1]!, (set = new Set())); + set.add(m[2]!); + } + callees.set(file, byAttr); + } + return byAttr.get(attr) ?? new Set(); + }; + + const kept = candidates.filter((ref) => { + const name = ref.referenceName; + if (name.includes('().')) return defined.has(name.split('().')[0]!.split('.').pop()!); + const segs = name.split('.'); + if (segs.length !== 3 || !segs.every((seg) => /^[A-Za-z_]\w*$/.test(seg))) return false; + const attr = segs[1]!; + if (assigned.has(attr)) return true; + if (subclassing.has(ref.filePath) && (segs[0] === 'self' || segs[0] === 'cls')) return true; + return calling.has(ref.filePath) && [...assignedCallees(ref.filePath, attr)].some((c) => defined.has(c)); + }); + return new Set(kept.map(keyOf)).size <= ceiling ? resolvable(load(kept)) : null; + } + resolveOne(ref: UnresolvedRef): ResolvedRef | null { // A C/C++ "call" whose name is a function-like macro visible in this // translation unit is a macro expansion, not a call — it must never bind diff --git a/src/resolution/name-matcher.ts b/src/resolution/name-matcher.ts index e8780e1d4..c00616299 100644 --- a/src/resolution/name-matcher.ts +++ b/src/resolution/name-matcher.ts @@ -3932,20 +3932,29 @@ function isPythonProjectModule(module: string, fromFile: string, context: Resolu * method of the same name (#2599). */ export function isPythonOutsideModuleReceiver(name: string, ref: UnresolvedRef, context: ResolutionContext): boolean { - if (ref.language !== 'python' || !/^\w+$/.test(name)) return false; - const outside = pythonNameScan(context, `outside\0${ref.filePath}\0${name}`, () => { - const keys = pythonImportKeys(name, ref.filePath, context); + if (ref.language !== 'python') return false; + return isPythonOutsideModuleName(name, ref.filePath, context) && !pythonBindsLocally(name, ref, context, false); +} + +/** + * The module-scope half of isPythonOutsideModuleReceiver: `filePath` binds + * `name` only by importing it from outside the project, and never rebinds it + * at module scope. + */ +function isPythonOutsideModuleName(name: string, filePath: string, context: ResolutionContext): boolean { + if (!/^\w+$/.test(name)) return false; + return pythonNameScan(context, `outside\0${filePath}\0${name}`, () => { + const keys = pythonImportKeys(name, filePath, context); if (keys.size === 0) return false; - if (pythonGlobalBindings(name, ref.filePath, context).some((b) => b.kind !== 'import')) return false; - if (context.getNodesInFile(ref.filePath).some((n) => n.name === name && (n.kind === 'class' || n.kind === 'function'))) return false; + if (pythonGlobalBindings(name, filePath, context).some((b) => b.kind !== 'import')) return false; + if (context.getNodesInFile(filePath).some((n) => n.name === name && (n.kind === 'class' || n.kind === 'function'))) return false; // `import a.b as x` keys as `a.b:*`; `from a import x` as `a:x`, where only // the module `a` tells that `x` comes from outside. return [...keys].every((key) => { const module = key.slice(0, key.indexOf(':')); - return !module.startsWith('.') && !isPythonProjectModuleAnyRoot(module, ref.filePath, context); + return !module.startsWith('.') && !isPythonProjectModuleAnyRoot(module, filePath, context); }); }); - return outside && !pythonBindsLocally(name, ref, context, false); } /** @@ -16980,6 +16989,13 @@ export function clearNameMatcherMemos(context: ResolutionContext): void { TS_CREATE_INSTANCE_DECLARED.delete(context); TS_NAMED_TYPES.delete(context); TARGET_LANGUAGE.delete(context); + PY_LINES.delete(context); + PY_ATTR_MEMO.delete(context); + PY_FACTORY_MEMO.delete(context); + PY_STATEMENTS_MEMO.delete(context); + PY_BASES_MEMO.delete(context); + PY_FILE_CLASSES.delete(context); + PY_ATTR_CALL_MEMO.delete(context); } function memoPatterns(key: string, build: () => RegExp[]): RegExp[] { @@ -18041,6 +18057,9 @@ function inferPhpAssignedPropertyType( return null; } +/** Python attribute-chain refs being guessed by name after the type reader found nothing. */ +const PY_CHAIN_GUESSING = new WeakSet(); + /** * Try to resolve by method name on a class/object */ @@ -18132,7 +18151,13 @@ export function matchMethodCall( const [, objectOrClass, methodName] = match; // A simple `receiver.method` / `receiver:method` / `receiver$method` shape whose // receiver type we can try to infer from its local declaration. - const inferableReceiver = dotMatch || luaColonMatch || rDollarMatch; + // A python attribute receiver (`self.conn`, `cfg.client`) is not one: the + // python branch below owns it, and the shared inferrer would answer first by + // the assignment's leading name — `self.conn = Pool().acquire()` read as a + // `Pool`, `Session` read as the project's class under `from requests import + // Session` — skipping that branch's whole-value, rebinding and import checks. + const pythonAttrReceiver = ref.language === 'python' && !!dotMatch && objectOrClass!.includes('.'); + const inferableReceiver = (dotMatch || luaColonMatch || rDollarMatch) && !pythonAttrReceiver; // A Go call of a standard method name — `String`, `Error`, `Len`, `Lock` — // on a plain variable: a name nearly every type carries, so the call says // nothing of the receiver's type. @@ -18513,6 +18538,33 @@ export function matchMethodCall( return matchTsThisFieldCall(objectOrClass!.slice('this.'.length), methodName!, ref, context); } + // Python call through an attribute — `self.data.append(x)`, + // `cls.registry.lookup(k)`, `cfg.client.send(x)` — which the extractor emits + // with its receiver (`self.data.append`) instead of a bare method name. The + // type the class body gives the attribute decides when there is one: its + // method, or no edge for a builtin or outside type (a bare-name match is how + // `self.data.append(1)` bound to an unrelated class's `append`). With no + // such evidence the call is guessed by name below, as any receiver is. A + // dotted MODULE path (`pkg.mod.func()`) never gets here: import resolution + // claims it first, through the module the import names. + if (pythonAttrReceiver && !PY_CHAIN_GUESSING.has(ref)) { + const read = new Set(); + const typed = matchPythonAttrCall(objectOrClass!, methodName!, ref, context, read); + if (typed !== undefined) return typed; + // The guess records the files the type reader looked at, as a typed answer + // does (`typeFrom`): a sync that makes one of them type the attribute — + // a factory gaining `-> Client` — re-opens it. + PY_CHAIN_GUESSING.add(ref); + let guess: ResolvedRef | null; + try { + guess = matchMethodCall(ref, context); + } finally { + PY_CHAIN_GUESSING.delete(ref); + } + read.delete(ref.filePath); + return guess && read.size > 0 ? { ...guess, metadata: { ...guess.metadata, typeFrom: [...read].sort() } } : guess; + } + // Java/Kotlin: receiver may be a field whose name doesn't match the type by // Java naming convention (`userbo` → class `UserBO`, abbreviated). Look up // the field in the enclosing class to get its declared type, then resolve @@ -18780,13 +18832,19 @@ export function matchMethodCall( targetMethods = kept; } + // A python attribute chain nothing typed (matchPythonAttrCall) is guessed + // only by the name of its last link, in the scoring below — Django's + // `self.client.get()` is its test `Client`'s — never as the one method of + // the name: `self.a.b.go()` is no `Leaf.go` the receiver says nothing of. + const pythonChain = ref.language === 'python' && objectOrClass!.includes('.'); + // If only one same-language method with this name exists, use it — // except in Ruby, where nothing types a receiver: there the one method // must also belong to something the receiver is named after // (`web_push_request.legacy_encrypt` → WebPushRequest). rubocop's // `node.loc` on a rubocop-ast node went to the one `loc` in the project // 1,201 times; lobsters' `value.to_s` to a short-id class's. - if (targetMethods.length === 1 && !narrowed && !scopedOut && targetMethods[0]!.language === ref.language && + if (targetMethods.length === 1 && !narrowed && !scopedOut && !pythonChain && targetMethods[0]!.language === ref.language && // A test double is only what a test names — as in the scoring below: // allauth's `resp.json()` on a Django test response is not the one // `json` of its `MockedResponse`. @@ -18810,11 +18868,16 @@ export function matchMethodCall( } // Multiple methods: score by receiver name word overlap with class name - if (targetMethods.length > 1 || (targetMethods.length === 1 && scopedOut && !narrowed)) { + if (targetMethods.length > 1 || (targetMethods.length === 1 && (scopedOut || pythonChain) && !narrowed)) { // What the receiver is named after is its last link: `builder.tokeniser` - // is a Tokeniser, `table.Columns` no kind of table. - const receiverWords = splitCamelCase(receiverLink(objectOrClass!)); + // is a Tokeniser, `table.Columns` no kind of table. In a python chain the + // last link as written: Django's `request.META` and `request.GET` are + // dicts, not constants of `request`'s type. + const link = pythonChain ? objectOrClass!.slice(objectOrClass!.lastIndexOf('.') + 1) : receiverLink(objectOrClass!); + const receiverWords = splitCamelCase(link); const head = receiverWords[receiverWords.length - 1]?.toLowerCase(); + const wholeReceiver = link.replace(/_/g, '').toLowerCase(); + const singular = (w: string): string => w.toLowerCase().replace(/s$/, ''); let bestMatch: typeof targetMethods[0] | undefined; let bestScore = 0; let tied: typeof targetMethods = []; @@ -18829,6 +18892,12 @@ export function matchMethodCall( // shares `Client` with every `httpClient`) nor the method's. const cut = method.qualifiedName.lastIndexOf('::'); const classWords = cut > 0 ? splitCamelCase(method.qualifiedName.slice(0, cut).split(/::|\./).pop()!) : []; + // An untyped python attribute chain is a guess only through its head + // noun: `self.session` names a `SessionBase` and `self.client` a + // `ClientMixin`, but mealie's `database.group_shopping_lists` no + // `GroupRepositoryAIProvider`, which only shares its `group`. + if (pythonChain && head !== undefined && !classWords.some((cw) => singular(cw) === singular(head)) && + wholeReceiver !== classWords.join('').toLowerCase()) continue; let score = receiverWords.filter(w => classWords.some(cw => cw.toLowerCase() === w.toLowerCase()) ).length; @@ -18840,6 +18909,11 @@ export function matchMethodCall( // The receiver's head noun naming the owner's: `bookPage` is a Page // before it is anything of a Book's. if (head !== undefined && head === classWords[classWords.length - 1]?.toLowerCase()) score += 1; + // The receiver spelling the owner's whole name: Django's + // `self.client` is a `Client` before a `RedisCacheClient`, which only + // ends in the word, and `self.nodelist` a `NodeList`, though no one + // word of it matches. + if (classWords.length > 0 && wholeReceiver === classWords.join('').toLowerCase()) score += 2; // Bonus for same language if (method.language === ref.language) score += 1; if (score > bestScore) { @@ -19635,6 +19709,1094 @@ function goDotImportDirs(filePath: string, context: ResolutionContext): string[] return dirs; } +/** Python types whose methods are the runtime's, never a project symbol's. */ +const PYTHON_RUNTIME_TYPES: ReadonlySet = new Set([ + 'list', 'dict', 'set', 'frozenset', 'tuple', 'str', 'bytes', 'bytearray', + 'int', 'float', 'complex', 'bool', 'object', 'type', 'range', 'slice', + 'List', 'Dict', 'Set', 'FrozenSet', 'Tuple', 'Sequence', 'Mapping', + 'MutableMapping', 'MutableSequence', 'Iterable', 'Iterator', 'Callable', + 'deque', 'defaultdict', 'OrderedDict', 'Counter', + 'collections.deque', 'collections.defaultdict', 'collections.OrderedDict', 'collections.Counter', +]); + +/** + * Per-context memos for the python attribute-type reader below: a file's + * comment-blanked lines, and the answers for `(class, attribute)` and + * `(file, factory)` together with the files each answer was read from. All are + * derived from file text, so they drop with the context's file caches + * (clearNameMatcherMemos). Answers are memoized only for top-level questions, + * never for a step inside a base-class walk, whose answer depends on the walk + * — a memo must not make a fresh index and a sync ask in a different order and + * get different edges. + */ +const PY_LINES = new WeakMap>(); +const PY_ATTR_MEMO = new WeakMap>(); +const PY_FACTORY_MEMO = new WeakMap>(); +/** Per class: what its own body says of an attribute, and its declared bases — + * each read from that class's file alone, so safe to memoize at any depth. */ +const PY_STATEMENTS_MEMO = new WeakMap>(); +const PY_BASES_MEMO = new WeakMap>>(); +const PY_LINES_CAP = 1024; +const PY_MEMO_CAP = 65536; + +function pyMemo(store: WeakMap>, context: ResolutionContext): Map { + let m = store.get(context); + if (!m) { + m = new Map(); + store.set(context, m); + } + return m; +} + +/** `filePath`'s lines with comments and docstrings blanked, memoized. */ +function pythonLines(context: ResolutionContext, filePath: string): string[] | null { + const memo = pyMemo(PY_LINES, context); + if (memo.has(filePath)) return memo.get(filePath)!; + const source = context.readFile(filePath); + const lines = source === null ? null : stripCommentsForRegex(source, 'python').split('\n'); + if (memo.size >= PY_LINES_CAP) memo.delete(memo.keys().next().value!); + memo.set(filePath, lines); + return lines; +} + +/** + * What one python statement says an attribute holds: a type it names, a + * factory whose result it holds (`Client.from_env()`, `make_client()`, + * `await connect()`), either of two answers (`A if c else B`), a builtin, or + * nothing usable (null). + */ +type PythonAttrEvidence = + // `classLevel`: assigned in the class body (`author = ForeignKey(Author)`), + // where a framework's descriptor may hand instances something else. + | { type: string; classLevel?: boolean } + | { factory: string; awaited: boolean } + | { arms: [PythonEvidence, PythonEvidence] } + | 'runtime' + | null; +type PythonEvidence = Exclude; + +/** + * The type an annotation names — `Client`, `"Client"`, `Optional[Client]`, + * `Client | None`, `models.Client` — or 'runtime' for a builtin container, or + * null when it says nothing usable (`Any`, a generic of a project type). + */ +function pythonAnnotationEvidence(raw: string): PythonAttrEvidence { + let t = raw.trim().replace(/^['"]|['"]$/g, '').trim(); + for (let i = 0; i < 4 && t; i++) { + const opt = t.match(/^(?:typing\.)?Optional\s*\[\s*(.+)\s*\]$/) + ?? t.match(/^(?:typing\.)?Union\s*\[\s*([^,\]]+?)\s*,\s*None\s*\]$/) + ?? t.match(/^(.+?)\s*\|\s*None$/) + ?? t.match(/^None\s*\|\s*(.+)$/); + if (!opt) break; + t = opt[1]!.trim().replace(/^['"]|['"]$/g, '').trim(); + } + const head = t.split('[')[0]!.trim().replace(/^typing\./, ''); + if (PYTHON_RUNTIME_TYPES.has(head)) return 'runtime'; + if (t.includes('[') || !/^[A-Za-z_][\w.]*$/.test(t) || head === 'Any') return null; + return { type: t }; +} + +/** + * The top-level shape of a python expression: string literals collapse to + * `S` and everything inside brackets is dropped, so `Client(a, f(b))` reads + * `Client()`, `Pool().acquire()` stays `Pool().acquire()` and + * `Client() if t else Fake()` keeps its `if`. `open` says why there is no + * shape yet: a bracket or triple-quoted string still open, or a trailing `\` + * — the expression continues on the next line — versus a single-line string + * left open or a bracket closed that was never opened, which no further line + * can repair. + */ +function pythonExprScan(raw: string): { shape: string | null; open: 'none' | 'continues' | 'broken' } { + let out = ''; + let depth = 0; + for (let i = 0; i < raw.length; i++) { + const c = raw[i]!; + if (c === '"' || c === "'") { + const triple = raw.startsWith(c.repeat(3), i); + const close = triple ? c.repeat(3) : c; + let j = i + close.length; + while (j < raw.length && !raw.startsWith(close, j)) j += raw[j] === '\\' ? 2 : 1; + if (j >= raw.length) return { shape: null, open: triple ? 'continues' : 'broken' }; + if (depth === 0) out = out.replace(/[rbfuRBFU]{1,2}$/, '') + 'S'; + i = j + close.length - 1; + } else if (c === '(' || c === '[' || c === '{') { + if (depth === 0) out += c; + depth++; + } else if (c === ')' || c === ']' || c === '}') { + depth--; + if (depth < 0) return { shape: null, open: 'broken' }; + if (depth === 0) out += c; + } else if (depth === 0) { + out += c; + } + } + const shape = out.replace(/\s+/g, ' ').trim(); + if (depth > 0 || shape.endsWith('\\')) return { shape: null, open: 'continues' }; + return { shape, open: 'none' }; +} + +function pythonExprShape(raw: string): string | null { + return pythonExprScan(raw).shape; +} + +/** + * The two arms of a top-level conditional expression — `A if c else B` → + * `[A, B]` — or null when `raw` is not one. Keywords inside brackets or + * strings do not count. + */ +function pythonConditionalArms(raw: string): [string, string] | null { + let depth = 0; + let ifAt = -1; + for (let i = 0; i < raw.length; i++) { + const c = raw[i]!; + if (c === '"' || c === "'") { + const close = raw.startsWith(c.repeat(3), i) ? c.repeat(3) : c; + let j = i + close.length; + while (j < raw.length && !raw.startsWith(close, j)) j += raw[j] === '\\' ? 2 : 1; + i = j + close.length - 1; + } else if ('([{'.includes(c)) depth++; + else if (')]}'.includes(c)) depth--; + else if (depth === 0 && /\s/.test(raw[i - 1] ?? '')) { + if (ifAt < 0 && /^if\s/.test(raw.slice(i))) ifAt = i; + else if (ifAt >= 0 && /^else\s/.test(raw.slice(i))) { + return [raw.slice(0, ifAt).trim(), raw.slice(i + 4).trim()]; + } + } + } + return null; +} + +/** + * Same question for an assigned value: `Client(...)`, `[]`, `cap` (a + * parameter), or a call to something that is not a class — a factory, typed + * later by what it returns. A call counts only when it IS the whole value: + * `Pool().acquire()` holds what `acquire` returns and `Client().close()` what + * `close` returns, so neither is evidence of `Pool` or `Client`. A conditional + * `A if c else B` holds either arm; it is evidence only if both arms agree, + * which is decided once they are resolved. + */ +function pythonValueEvidence(raw: string, signature: string | null): PythonAttrEvidence { + let v = raw.trim(); + const arms = pythonConditionalArms(v); + if (arms) { + const [a, b] = arms.map((arm) => pythonValueEvidence(arm, signature)); + return a && b ? { arms: [a, b] } : null; + } + const awaited = /^await\s/.test(v); + if (awaited) v = v.replace(/^await\s+/, ''); + const shape = pythonExprShape(v); + if (shape === null) return null; + if (/^(?:S(?: S)*|-?\d[\w.]*|True|False|\[\]|\{\}|\(\))$/.test(shape)) return awaited ? null : 'runtime'; + const ctor = /^([A-Za-z_][\w.]*) ?\(\)$/.exec(shape); + if (ctor) { + if (PYTHON_RUNTIME_TYPES.has(ctor[1]!)) return awaited ? null : 'runtime'; + // A capitalised callee constructs that class; anything else is a factory + // (`make_client()`, `Client.from_env()`) whose result its body decides. + if (/^[A-Z]/.test(ctor[1]!.split('.').pop()!)) return awaited ? null : { type: ctor[1]! }; + return { factory: ctor[1]!, awaited }; + } + // `self.cap = cap` — the parameter's annotation in the enclosing `def`. + if (!awaited && signature && /^[A-Za-z_]\w*$/.test(v)) { + const param = new RegExp(`[(,]\\s*\\*{0,2}${v}\\s*:\\s*([^,)=]+)`).exec(signature); + if (param) return pythonAnnotationEvidence(param[1]!); + } + return null; +} + +/** + * The value starting at `first` on line `ln`, joined with the lines that + * continue it — an open bracket or triple-quoted string, or a trailing `\` — + * up to a small cap. Returns the text and the last line it used. A value that + * does not close within the cap, or cannot (a single-line string left open, + * a stray closing bracket), is returned as its first line alone with no line + * consumed: the lines after it are statements of their own and must still be + * read. + */ +function pythonJoinedValue(lines: string[], ln: number, first: string, lastLine: number): { text: string; end: number } { + if (pythonExprScan(first).open !== 'continues') return { text: first, end: ln }; + let text = first; + for (let end = ln; end < lastLine && end - ln < 30; ) { + text = text.replace(/\\\s*$/, '') + ' ' + (lines[end] ?? '').trim(); + end++; + const { open } = pythonExprScan(text); + if (open === 'none') return { text, end }; + if (open === 'broken') break; + } + return { text: first, end: ln }; +} + +/** + * The names `value` calls as a whole expression or conditional arm — + * `Client(...)`, `make()` — whose local rebinding would make the evidence + * wrong. + */ +function pythonValueCallees(value: string): string[] { + const arms = pythonConditionalArms(value.trim()); + if (arms) return arms.flatMap(pythonValueCallees); + const shape = pythonExprShape(value.trim().replace(/^await\s+/, '')); + const call = shape && /^([A-Za-z_][\w.]*) ?\(\)$/.exec(shape); + return call ? [call[1]!] : []; +} + +/** + * `line` with the contents of its string literals blanked (quotes kept), so a + * string never reads as code: `log("a: return b")` holds no `return`, + * `print("Foo = %s")` no assignment. A quote left open blanks the rest of the + * line (a triple-quoted string continuing below). + */ +function pythonBlankStrings(line: string): string { + const out = line.split(''); + for (let i = 0; i < line.length; i++) { + const c = line[i]!; + if (c !== '"' && c !== "'") continue; + const q = line.startsWith(c.repeat(3), i) ? c.repeat(3) : c; + let j = i + q.length; + while (j < line.length && !line.startsWith(q, j)) j += line[j] === '\\' ? 2 : 1; + for (let k = i + q.length; k < Math.min(j, line.length); k++) out[k] = ' '; + i = Math.min(j + q.length, line.length) - 1; + } + return out.join(''); +} + +/** + * The assignment targets of one python statement (strings blanked): `a, (b, + * c) = …` → [`a, (b, c)`], `x: T = …` → [`x`], `x += …` → [`x`], `a = b = …` → + * [`a`, `b`], and the statement a one-line compound header guards (`if c: x = + * …`). A `=` inside brackets is a keyword argument or a default, `==` / `!=` / + * `<=` / `>=` compare, and a `lambda`'s defaults bind nothing here — none is a + * target. + */ +function pythonAssignTargets(statement: string): string[] { + const stmt = statement.replace(/^\s*(?:(?:el)?if|while|else|try|finally|except|with|for)\b[^:]*:(?!=)/, ''); + const targets: string[] = []; + let depth = 0; + let from = 0; + for (let i = 0; i < stmt.length; i++) { + const c = stmt[i]!; + if (c === '(' || c === '[' || c === '{') depth++; + else if (c === ')' || c === ']' || c === '}') depth = Math.max(0, depth - 1); + else if (c === '=' && depth === 0) { + if (stmt[i + 1] === '=') { i++; continue; } + const prev = stmt[i - 1] ?? ''; + if (prev === '!' || prev === ':') continue; // `!=`, walrus + if ((prev === '<' || prev === '>') && stmt[i - 2] !== prev) continue; // `<=`, `>=` + let seg = stmt.slice(from, i); + if (/\blambda\b/.test(seg)) break; + seg = seg.replace(/(?:\*\*|\/\/|>>|<<|[-+*/%&|^@])$/, ''); + targets.push(seg.split(':')[0]!.trim()); + from = i + 1; + } + } + return targets; +} + +/** + * The statement-level lines of the `def` on line `defLine`, after its header + * and before line `upto`: each with its line number and its statements + * (strings blanked, split on `;`). A line that continues a bracketed or + * `\`-continued statement is not one — `client=client,` inside a call is a + * keyword argument — and neither is a line inside a nested `def` or `class`, + * whose header is reported as `nested` (its name is bound here). + */ +function pythonDefStatements( + lines: string[], + defLine: number, + upto: number, +): { header: string; body: Array<{ line: number; statements: string[]; nested?: string }> } { + let header = ''; + let ln = defLine; + for (; ln <= upto && ln <= lines.length; ln++) { + header += ' ' + (lines[ln - 1] ?? '').trim(); + if (/\)\s*(?:->[^:]*)?:\s*$/.test(header)) break; + } + const body: Array<{ line: number; statements: string[]; nested?: string }> = []; + const indent = (l: string) => l.length - l.trimStart().length; + let depth = 0; + let continued = false; + let nestedIndent = -1; + for (let k = ln + 1; k < upto; k++) { + const raw = lines[k - 1] ?? ''; + if (!raw.trim()) continue; + const blanked = pythonBlankStrings(raw); + const inside = depth > 0 || continued; + for (const c of blanked) { + if (c === '(' || c === '[' || c === '{') depth++; + else if (c === ')' || c === ']' || c === '}') depth = Math.max(0, depth - 1); + } + continued = /\\\s*$/.test(blanked); + if (inside) continue; + if (nestedIndent >= 0) { + if (indent(raw) > nestedIndent) continue; + nestedIndent = -1; + } + const text = blanked.trim(); + const nested = /^(?:async\s+)?(?:def|class)\s+(\w+)/.exec(text); + if (nested) { + nestedIndent = indent(raw); + body.push({ line: k, statements: [], nested: nested[1]! }); + continue; + } + body.push({ line: k, statements: text.split(';').map((t) => t.trim()).filter(Boolean) }); + } + return { header, body }; +} + +/** + * Whether an assignment target binds `name` itself — `name`, `*name`, or an + * element of a tuple / list target (`a, (name, b)`, `[name, b]`). A target that + * only USES the name does not: `d[name] = 1` and `self._h[name] = c` store + * into a container, `name.attr = 1` sets an attribute. + */ +function pythonTargetBinds(target: string, name: string): boolean { + const n = name.replace(/[.*+?^${}()|[\]\\]/g, '\\$&'); + for (const m of target.matchAll(new RegExp(`(? p.trim().replace(/^\*{1,2}/, '').split(/[:=]/)[0]!.trim()); + if (paramNames.includes(name)) return true; + } + const loop = new RegExp(`^(?:async\\s+)?for\\s+[\\w\\s,()\\[\\]]*(? pythonTargetBinds(t, name))) return true; + if (loop.test(st) || walrus.test(st)) return true; + if (!/^(?:from|import)\b/.test(st) && as.test(st)) return true; + } + } + return false; +} + +/** + * Every statement in `owner`'s body that assigns `attr` — a class-level + * declaration (`registry = Registry()`, `conn: Client`) or an assignment + * through `self` / `cls` (`self.cap = Capture()`, `self.cap: Capture`, + * `self.cap = cap` with `cap: Capture` in the signature) — and whether the body + * mentions it at all. An untyped statement (`self.cap = load()` of an unknown) + * contributes no evidence but still counts as a mention: it shadows any base. + * + * Comments and docstrings are blanked before reading, and nested classes are + * skipped, so neither can supply the type: a regex over raw class lines took a + * type from a docstring once and turned a correct edge into a wrong one. + */ +function pythonAttrStatements( + owner: Node, + attr: string, + context: ResolutionContext, +): { mentioned: boolean; evidence: PythonEvidence[] } { + const memo = pyMemo(PY_STATEMENTS_MEMO, context); + const key = `${owner.id}\0${attr}`; + let hit = memo.get(key); + if (!hit) { + hit = pythonAttrStatementsUncached(owner, attr, context); + if (memo.size >= PY_MEMO_CAP) memo.clear(); + memo.set(key, hit); + } + return hit; +} + +function pythonAttrStatementsUncached( + owner: Node, + attr: string, + context: ResolutionContext, +): { mentioned: boolean; evidence: PythonEvidence[] } { + const lines = pythonLines(context, owner.filePath); + if (!lines) return { mentioned: false, evidence: [] }; + const nested = context + .getNodesInFile(owner.filePath) + .filter((n) => n.kind === 'class' && n.id !== owner.id && n.startLine > owner.startLine && n.endLine <= owner.endLine); + const indent = (l: string) => l.length - l.trimStart().length; + let bodyIndent = -1; + const a = attr.replace(/[.*+?^${}()|[\]\\]/g, '\\$&'); + const classLevel = new RegExp(`^${a}\\s*(?::\\s*([^=]+?))?\\s*(?:=\\s*(.+?))?\\s*$`); + const viaSelf = new RegExp(`\\b(?:self|cls)\\.${a}\\s*(?::\\s*([^=]+?))?\\s*(?:=(?!=)\\s*(.+?))?\\s*$`); + let signature: string | null = null; + let defStart = -1; + let nestedDefIndent = -1; + let mentioned = false; + const evidence: PythonEvidence[] = []; + + for (let ln = owner.startLine + 1; ln <= owner.endLine; ln++) { + const line = lines[ln - 1]; + if (!line || !line.trim()) continue; + if (nested.some((n) => ln >= n.startLine && ln <= n.endLine)) continue; + if (bodyIndent < 0) bodyIndent = indent(line); + const text = line.trim(); + if (/^(?:async\s+)?def\s/.test(text) && indent(line) === bodyIndent) { + // The enclosing method's signature, possibly over several lines. A def + // nested in a method is not one: its parameters and locals are its own. + defStart = ln; + nestedDefIndent = -1; + signature = text; + for (let k = ln; k < owner.endLine && !/\)\s*(?:->[^:]*)?:\s*$/.test(signature); k++) { + signature += ' ' + (lines[k] ?? '').trim(); + } + continue; + } + if (nestedDefIndent >= 0 && indent(line) <= nestedDefIndent) nestedDefIndent = -1; + if (nestedDefIndent < 0 && /^(?:async\s+)?def\s/.test(text) && indent(line) > bodyIndent) { + nestedDefIndent = indent(line); + continue; + } + const m = indent(line) === bodyIndent ? classLevel.exec(text) : viaSelf.exec(text); + if (!m || (!m[1] && !m[2])) continue; + mentioned = true; + // `self.x = c` inside a def nested in a method (a callback) reads that + // def's own parameters and locals, which the method's scope does not see: + // it shadows a base but says nothing usable here. + if (nestedDefIndent >= 0 && indent(line) > nestedDefIndent) continue; + const inDef = indent(line) !== bodyIndent && defStart > 0; + const at = ln; + let value = m[2]; + if (!m[1] && value) { + const joined = pythonJoinedValue(lines, ln, value, owner.endLine); + value = joined.text; + ln = joined.end; + } + // A value read through a name the method rebinds first — `Backend = + // load(); self.b = Backend()`, or `box = wrap(box); self.box = box` for a + // `box: Box` parameter — is not what the module's class or the annotation + // says. That statement is no evidence. + if (!m[1] && inDef) { + const v = value!.trim(); + const reads = pythonValueCallees(v).map((c) => c.split('.')[0]!); + if (reads.some((c) => pythonBoundInDef(lines, defStart, at, c))) continue; + if (/^[A-Za-z_]\w*$/.test(v) && pythonBoundInDef(lines, defStart, at, v, false)) continue; + } + const ev = m[1] + ? pythonAnnotationEvidence(m[1]) + : pythonValueEvidence(value!, inDef ? signature : null); + if (ev) evidence.push(!m[1] && !inDef && typeof ev === 'object' && 'type' in ev ? { ...ev, classLevel: true } : ev); + } + return { mentioned, evidence }; +} + +/** + * The class `className` declared in the python module `modulePath` as seen + * from `fromFile` — `pkg.a` / `.a` / `..core` — or null. Relative paths are + * anchored at `fromFile`'s package; an absolute one may sit under a source root + * (`src/pkg/a.py`), so it matches by path suffix and must be unique. With + * `kind: 'function'` the same lookup finds a module-level function. + */ +function pythonModuleClass( + modulePath: string, + className: string, + fromFile: string, + context: ResolutionContext, + kind: 'class' | 'function' = 'class', +): Node | null { + const dots = /^\.*/.exec(modulePath)![0].length; + let rel = modulePath.slice(dots).replace(/\./g, '/'); + if (dots > 0) { + const dir = fromFile.replace(/\\/g, '/').split('/').slice(0, -1); + for (let i = 1; i < dots; i++) { + if (dir.length === 0) return null; + dir.pop(); + } + rel = [...dir, ...(rel ? [rel] : [])].join('/'); + } + if (!rel) return null; + const wanted = [`${rel}.py`, `${rel}/__init__.py`, `${rel}.pyi`]; + const hits = context.getNodesByName(className).filter((n) => { + if (n.kind !== kind || n.language !== 'python') return false; + if (kind === 'function' && n.qualifiedName !== n.name) return false; // module level only + const fp = n.filePath.replace(/\\/g, '/'); + return wanted.some((w) => fp === w || (dots === 0 && fp.endsWith(`/${w}`))); + }); + return hits.length === 1 ? hits[0]! : null; +} + +/** + * The project class a python type name means in `fromFile`: the class in the + * module its import names, or the file's own declaration. An import that names + * a module outside the project (`requests.Session`) is null, never a same-named + * project class — that is the guess this whole path exists to stop. With + * `kind: 'function'` it answers the same question for a function name + * (`make_client`, `clients.build`). + */ +function pythonClassNamed( + typeName: string, + fromFile: string, + context: ResolutionContext, + kind: 'class' | 'function' = 'class', +): Node | null { + const segs = typeName.split('.'); + const className = segs[segs.length - 1]!; + const imports = context.getImportMappings(fromFile, 'python'); + const candidates: Array = []; + for (const imp of imports) { + if (imp.isNamespace && segs.length > 1 && typeName.startsWith(`${imp.source}.`)) { + // `import pkg.models` then `pkg.models.User`. + const middle = typeName.slice(imp.source.length + 1).split('.').slice(0, -1); + candidates.push(pythonModuleClass([imp.source, ...middle].join('.'), className, fromFile, context, kind)); + } else if (imp.localName === segs[0]) { + if (segs.length === 1) { + // `from pkg.a import Capture` / `... import Capture as C`. + candidates.push(imp.isNamespace ? null : pythonModuleClass(imp.source, imp.exportedName, fromFile, context, kind)); + } else { + // `from pkg import models` then `models.User`; `import models as m` then `m.User`. + const base = imp.isNamespace + ? imp.source + : imp.source.endsWith('.') ? imp.source + imp.exportedName : `${imp.source}.${imp.exportedName}`; + candidates.push(pythonModuleClass([base, ...segs.slice(1, -1)].join('.'), className, fromFile, context, kind)); + } + } + } + if (candidates.length > 0) { + // Every binding of the name must agree on one class. + const ids = new Set(candidates.map((c) => c?.id ?? '')); + return ids.size === 1 && candidates[0] ? candidates[0] : null; + } + if (segs.length > 1) return null; + const local = context + .getNodesByName(typeName) + .filter((n) => + n.kind === kind && n.language === 'python' && n.filePath === fromFile && + (kind === 'class' || n.qualifiedName === n.name)); + return local.length === 1 ? local[0]! : null; +} + +/** + * The bases `cls`'s own `class` line names, each resolved from `cls`'s file + * (null for one outside the project). Reading them off the header rather than + * the `extends` edges keeps this answerable in the first pass. + */ +function pythonClassBases(cls: Node, context: ResolutionContext): Array { + const memo = pyMemo(PY_BASES_MEMO, context); + let hit = memo.get(cls.id); + if (!hit) { + hit = pythonClassBasesUncached(cls, context); + if (memo.size >= PY_MEMO_CAP) memo.clear(); + memo.set(cls.id, hit); + } + return hit; +} + +function pythonClassBasesUncached(cls: Node, context: ResolutionContext): Array { + return pythonClassBaseNames(cls, context).map((b) => pythonClassNamed(b, cls.filePath, context)); +} + +/** The bases `cls`'s own `class` line names, as written (`Base`, `models.Model`). */ +function pythonClassBaseNames(cls: Node, context: ResolutionContext): string[] { + const lines = pythonLines(context, cls.filePath); + if (!lines) return []; + const header = lines.slice(cls.startLine - 1, cls.startLine + 4).join(' '); + const bases = new RegExp(`\\bclass\\s+${cls.name}\\s*\\(([^)]*)\\)`).exec(header)?.[1]; + return (bases ?? '') + .split(',') + .map((b) => b.trim()) + .filter((b) => /^[A-Za-z_][\w.]*$/.test(b) && b !== 'object'); // `metaclass=...`, generics +} + +/** + * Whether every base in `cls`'s hierarchy is known — a project class, whose + * bases are known in turn, or a class from outside the project. A base the + * reader cannot find (a name a package re-exports from elsewhere) may hold a + * method, so a class with one is no evidence that a method is missing. + */ +function pythonHierarchyKnown(cls: Node, context: ResolutionContext, seen: Set = new Set()): boolean { + if (seen.has(cls.id) || seen.size >= 5) return true; + seen.add(cls.id); + const bases = pythonClassBases(cls, context); + return pythonClassBaseNames(cls, context).every((name, i) => { + const base = bases[i]; + return base ? pythonHierarchyKnown(base, context, seen) : isPythonOutsideModuleName(name.split('.')[0]!, cls.filePath, context); + }); +} + +/** + * `method` on class `cls`, or on a base its own `class` line names, with + * whether it was inherited. Files whose text was read are added to `read`. + */ +function pythonFindMethod( + cls: Node, + methodName: string, + context: ResolutionContext, + read: Set, + seen: Set = new Set(), +): { method: Node; inherited: boolean } | null { + if (seen.has(cls.id) || seen.size >= 5) return null; + seen.add(cls.id); + const own = context + .getNodesByName(methodName) + .find((m) => + m.kind === 'method' && m.language === 'python' && m.filePath === cls.filePath && + (m.qualifiedName === `${cls.name}::${methodName}` || m.qualifiedName.endsWith(`::${cls.name}::${methodName}`))); + if (own) return { method: own, inherited: false }; + + read.add(cls.filePath); + for (const base of pythonClassBases(cls, context)) { + const hit = base && pythonFindMethod(base, methodName, context, read, seen); + if (hit) return { method: hit.method, inherited: true }; + } + return null; +} + +/** `method` on class `cls` (or a declared base) as a resolution. */ +function pythonMethodOnClass( + cls: Node, + methodName: string, + ref: UnresolvedRef, + context: ResolutionContext, + read: Set, +): ResolvedRef | null { + const hit = pythonFindMethod(cls, methodName, context, read); + if (!hit) return null; + return { original: ref, targetNodeId: hit.method.id, confidence: hit.inherited ? 0.8 : 0.85, resolvedBy: 'instance-method' }; +} + +/** + * A base-typed attribute can hold a subclass-only method (#1820): the one + * descendant of `cls` that defines `methodName`, else nothing. Same rule as the + * method-value path in `matchMemberFunctionRef`; unrelated same-name methods + * never win. + */ +function pythonSubclassMethod( + cls: Node, + methodName: string, + ref: UnresolvedRef, + context: ResolutionContext, +): ResolvedRef | null { + const descendants = context.getNodesByName(methodName).filter((m) => { + if (m.kind !== 'method' || m.language !== 'python') return false; + const parent = context.getNodesInFile(m.filePath).find((c) => + c.kind === 'class' && m.qualifiedName === `${c.qualifiedName}::${methodName}`); + return !!parent && pythonDerivesFrom(parent, cls, ref, context); + }); + return descendants.length === 1 + ? { original: ref, targetNodeId: descendants[0]!.id, confidence: 0.8, resolvedBy: 'instance-method' } + : null; +} + +/** A class, a builtin, or unknown/contested — what one piece of evidence comes to. */ +type PythonTypeAnswer = Node | 'runtime' | null; + +/** One answer from several: they must all agree, and a builtin is not a project class. */ +function pythonAgree(answers: PythonTypeAnswer[]): PythonTypeAnswer { + if (answers.length === 0 || answers.some((a) => a === null)) return null; + const keys = new Set(answers.map((a) => (a === 'runtime' ? '' : (a as Node).id))); + return keys.size === 1 ? answers[0]! : null; +} + +/** + * The class one piece of evidence comes to, read from `fromFile`: a named type + * through the file's imports, a factory by what it produces, a conditional + * only when both arms agree. `cls(...)` is `clsIs` — the class a classmethod + * factory is called on — and unknown anywhere else. + */ +function pythonEvidenceClass( + ev: PythonEvidence, + fromFile: string, + context: ResolutionContext, + read: Set, + depth: number, + clsIs: Node | null, +): PythonTypeAnswer { + if (ev === 'runtime') return 'runtime'; + if ('arms' in ev) { + return pythonAgree(ev.arms.map((arm) => pythonEvidenceClass(arm, fromFile, context, read, depth, clsIs))); + } + // A class from outside the project (`requests.Session()`, a `Session` from + // `from requests import Session`) is no project class either: like a + // builtin, none of its methods is the project's. Not in a class body, where + // Django's `author = models.ForeignKey(Author)` hands instances an Author. + if ('type' in ev) { + return pythonClassNamed(ev.type, fromFile, context) ?? + (!ev.classLevel && isPythonOutsideModuleName(ev.type.split('.')[0]!, fromFile, context) ? 'runtime' : null); + } + if (ev.factory === 'cls') return ev.awaited ? null : clsIs; + return pythonFactoryType(ev.factory, ev.awaited, fromFile, context, read, depth); +} + +/** + * The type a python factory call produces — `make_client()`, + * `clients.build()`, `Client.from_env()` — read from `fromFile`: the callee's + * `-> T` annotation (`Self` meaning the class it is called on), else what every + * `return` in its body produces, which must agree. `cls(...)` in a classmethod + * is the class the factory is CALLED on, not where it is defined. A generator, + * a coroutine used without `await`, or any return this cannot name is unknown + * (null) — the callee decides the type, so an unreadable callee is no evidence. + * Files whose text was read are added to `read`. + */ +function pythonFactoryType( + callee: string, + awaited: boolean, + fromFile: string, + context: ResolutionContext, + read: Set, + depth = 0, +): PythonTypeAnswer { + if (depth > 0) return pythonFactoryTypeUncached(callee, awaited, fromFile, context, read, depth); + const memo = pyMemo(PY_FACTORY_MEMO, context); + const key = `${fromFile}\0${callee}\0${awaited ? 1 : 0}`; + let hit = memo.get(key); + if (!hit) { + const own = new Set(); + hit = { answer: pythonFactoryTypeUncached(callee, awaited, fromFile, context, own, 0), read: [...own] }; + if (memo.size >= PY_MEMO_CAP) memo.clear(); + memo.set(key, hit); + } + for (const f of hit.read) read.add(f); + return hit.answer; +} + +function pythonFactoryTypeUncached( + callee: string, + awaited: boolean, + fromFile: string, + context: ResolutionContext, + read: Set, + depth: number, +): PythonTypeAnswer { + if (depth > 3) return null; + const segs = callee.split('.'); + const fname = segs[segs.length - 1]!; + const prefix = segs.slice(0, -1).join('.'); + + let fn: Node | null = null; + let receiver: Node | null = null; + if (prefix && /^[A-Z]/.test(segs[segs.length - 2]!)) { + receiver = pythonClassNamed(prefix, fromFile, context); + fn = receiver ? pythonFindMethod(receiver, fname, context, read)?.method ?? null : null; + } else { + fn = pythonClassNamed(callee, fromFile, context, 'function'); + } + if (!fn) return null; + read.add(fn.filePath); + + const lines = pythonLines(context, fn.filePath); + if (!lines) return null; + const defLine = (lines[fn.startLine - 1] ?? '').trim(); + const isAsync = /^async\s+def\b/.test(defLine); + if (isAsync !== awaited) return null; + + const returns = /->\s*(.+)$/.exec(fn.signature ?? ''); + if (returns) { + const ann = pythonAnnotationEvidence(returns[1]!); + if (ann === 'runtime') return 'runtime'; + if (!ann || !('type' in ann)) return null; + if (/^(?:typing(?:_extensions)?\.)?Self$/.test(ann.type)) return receiver; + return pythonClassNamed(ann.type, fn.filePath, context); + } + + // No annotation: every `return` in the body, outside nested defs/classes. + const nested = context + .getNodesInFile(fn.filePath) + .filter((n) => n.id !== fn!.id && n.startLine > fn!.startLine && n.endLine <= fn!.endLine && + (n.kind === 'function' || n.kind === 'method' || n.kind === 'class')); + let decorated = ''; + for (let ln = fn.startLine - 2; ln >= 0 && /^\s*@/.test(lines[ln] ?? ''); ln--) decorated += lines[ln]; + const isClassmethod = /@classmethod\b/.test(decorated); + const answers: PythonTypeAnswer[] = []; + for (let ln = fn.startLine + 1; ln <= fn.endLine; ln++) { + if (nested.some((n) => ln >= n.startLine && ln <= n.endLine)) continue; + const text = (lines[ln - 1] ?? '').trim(); + // Keywords are read with string contents blanked: `log("note: return x")` + // and `print("will yield")` are not statements. + const code = pythonBlankStrings(text); + if (/^yield\b|[=(\s]yield\b/.test(code)) return null; // a generator yields, it does not return + // A return may follow a compound header or another statement on the same + // line — `if k: return A()`, `else: return B()`, `x = 1; return C()` — and + // every one of them must agree, so none may be skipped. + const ret = /(?:^|[:;]\s*)return\b\s*/.exec(code); + if (!ret) continue; + const at = ln; + const joined = pythonJoinedValue(lines, ln, text.slice(ret.index + ret[0].length), fn.endLine); + ln = joined.end; + const expr = joined.text.trim(); + if (expr === '' || expr === 'None') continue; + // The same whole-expression rule as an assignment: `return Pool().acquire()` + // returns what `acquire` returns, `return A() if t else B()` either arm. + const ev = pythonValueEvidence(expr, null); + if (!ev) return null; + if (pythonValueCallees(expr).some((c) => c !== 'cls' && pythonBoundInDef(lines, fn.startLine, at, c.split('.')[0]!))) return null; + answers.push(pythonEvidenceClass(ev, fn.filePath, context, read, depth + 1, isClassmethod ? receiver : null)); + } + return pythonAgree(answers); +} + +/** + * The class `owner.attr` holds, or null. The owner's own body decides when it + * mentions the attribute at all (a typed statement, or an untyped one that + * shadows); otherwise every declared base that types it must agree — the + * attribute a subclass uses without assigning is the one a base's `__init__` + * set. A base outside the project contributes nothing. Files whose text was + * read are added to `read`. + */ +function pythonAttrClass( + owner: Node, + attr: string, + context: ResolutionContext, + read: Set, +): PythonTypeAnswer | 'absent' { + const memo = pyMemo(PY_ATTR_MEMO, context); + const key = `${owner.id}\0${attr}`; + let hit = memo.get(key); + if (!hit) { + const own = new Set(); + hit = { answer: pythonAttrClassWalk(owner, attr, context, own, new Set()), read: [...own] }; + if (memo.size >= PY_MEMO_CAP) memo.clear(); + memo.set(key, hit); + } + for (const f of hit.read) read.add(f); + return hit.answer; +} + +function pythonAttrClassWalk( + owner: Node, + attr: string, + context: ResolutionContext, + read: Set, + seen: Set, +): PythonTypeAnswer | 'absent' { + if (seen.has(owner.id) || seen.size >= 5) return 'absent'; + seen.add(owner.id); + read.add(owner.filePath); + + const { mentioned, evidence } = pythonAttrStatements(owner, attr, context); + if (mentioned) { + // Untyped statements are no evidence either way — and a factory whose + // result cannot be read is one of them; the typed ones must agree. Two + // spellings of one class (`models.User()`, `User()`) agree. + const answers: PythonTypeAnswer[] = []; + for (const ev of evidence) { + const made = pythonEvidenceClass(ev, owner.filePath, context, read, 0, null); + if (made || ev === 'runtime' || !('factory' in ev)) answers.push(made); + } + const keys = new Set(answers.map((a) => (a === null ? '?' : a === 'runtime' ? '' : a.id))); + if (keys.size !== 1) return null; + return answers[0]!; + } + + const fromBases: PythonTypeAnswer[] = []; + for (const base of pythonClassBases(owner, context)) { + if (!base) continue; + const got = pythonAttrClassWalk(base, attr, context, read, seen); + if (got !== 'absent') fromBases.push(got); + } + if (fromBases.length > 0) return pythonAgree(fromBases); + // No class of the project's in the hierarchy assigns it: a base from + // outside — Django's `TestCase` or `View` in an app — is where it comes + // from (`self.client`, `self.request`), and none of its methods is the + // project's. + return pythonHasOutsideBase(owner, context) ? 'runtime' : 'absent'; +} + +/** Bases that give a subclass no attributes: `ABC`, `Protocol`, an enum, an exception. */ +const PYTHON_ATTRLESS_BASE = /^(?:ABC|Protocol|NamedTuple|TypedDict|Enum|IntEnum|StrEnum|Flag|IntFlag|\w*(?:Error|Exception|Warning))$/; + +/** Whether `cls`'s own `class` line names a base from outside the project that could give it attributes. */ +function pythonHasOutsideBase(cls: Node, context: ResolutionContext): boolean { + return pythonClassBaseNames(cls, context) + .filter((b) => !PYTHON_ATTRLESS_BASE.test(b.split('.').pop()!)) + .some((b) => isPythonOutsideModuleName(b.split('.')[0]!, cls.filePath, context)); +} + +/** + * Python call through one attribute of a class — `self.cap.stop()`, + * `cls.registry.lookup()`, `Svc.registry.lookup()`, `config.llm.rebuild()`. + * The class is the one the call sits in (`self` / `cls`), the one the receiver + * names, or the declared type of a local; the attribute's type comes from that + * class's body — directly, through a factory it calls, or from a base class + * that assigns it; the method is looked up on that type. + * + * Three answers. The method, when the attribute's type is a project class + * that has it. Null — no edge — when the evidence rules every project method + * out: the attribute holds a builtin (`self.data = []`) or a class from + * outside the project (`requests.Session()`), or the class it holds has no + * such method. Undefined when nothing here types the attribute — an untyped + * local, a factory whose result is unknowable, bases that disagree, an + * attribute no class body mentions, a deeper chain: the caller then guesses + * by name as for any other receiver, which is how Django's + * `self.client.get()` reaches its test `Client`. The edge records the other + * files the answer was read from (`typeFrom`), so a sync that edits one of + * them re-opens it. + */ +function matchPythonAttrCall( + receiver: string, + methodName: string, + ref: UnresolvedRef, + context: ResolutionContext, + read: Set = new Set(), +): ResolvedRef | null | undefined { + const segs = receiver.split('.'); + // A chain through a module — `os.path.exists()`, `np.linalg.norm()`, an + // `import utils` of the project's — is the module's: import resolution + // answers the project's own members, and no class method of the project's + // is a guess for one, not even `PathNotImplementedStorage.exists` by its + // shared word. + const root = segs[0]!; + if (isPythonOutsideModuleReceiver(root, ref, context) || + (context.getImportMappings(ref.filePath, 'python').some((m) => m.isNamespace && m.localName === root) && + !pythonBindsLocally(root, ref, context, false))) return null; + if (segs.length !== 2) return undefined; + const [head, attr] = segs as [string, string]; + + let owner: Node | null = null; + if (head === 'self' || head === 'cls') { + // The innermost class whose extent holds the call. + for (const n of pythonFileClasses(ref.filePath, context)) { + if (n.startLine > ref.line || n.endLine < ref.line) continue; + if (!owner || n.startLine > owner.startLine) owner = n; + } + } else if (/^[A-Z]/.test(head)) { + owner = pythonClassNamed(head, ref.filePath, context); + } else { + // A local or parameter whose own type the scope states — `config: Config`, + // `controller = Controller(...)` — then the attribute on that class. + const headType = inferLocalReceiverType(head, ref, context); + if (headType) owner = pythonClassNamed(headType, ref.filePath, context); + } + if (!owner) return undefined; + + const hit = pythonAttrCallOn(owner, attr, methodName, ref, context, read); + if (!hit) return hit; + read.delete(ref.filePath); + return read.size > 0 ? { ...hit, metadata: { typeFrom: [...read].sort() } } : hit; +} + +/** Per context: a python file's classes, for the class around a `self.x.m()` call. */ +const PY_FILE_CLASSES = new WeakMap>(); + +function pythonFileClasses(filePath: string, context: ResolutionContext): Node[] { + const memo = pyMemo(PY_FILE_CLASSES, context); + let hit = memo.get(filePath); + if (!hit) { + hit = context.getNodesInFile(filePath).filter((n) => n.kind === 'class'); + if (memo.size >= PY_MEMO_CAP) memo.clear(); + memo.set(filePath, hit); + } + return hit; +} + +/** Per context: owner + attribute + method → matchPythonAttrCall's answer for that class, and the files it read. */ +const PY_ATTR_CALL_MEMO = new WeakMap>(); + +/** + * matchPythonAttrCall's answer once the owner class is known: the same for + * every call of `methodName` through `attr` of that class — Django's + * thousand `self.client.get()`s in one test class read it once. + */ +function pythonAttrCallOn( + owner: Node, + attr: string, + methodName: string, + ref: UnresolvedRef, + context: ResolutionContext, + read: Set, +): ResolvedRef | null | undefined { + const memo = pyMemo(PY_ATTR_CALL_MEMO, context); + const key = `${owner.id}\0${attr}\0${methodName}`; + let hit = memo.get(key); + if (!hit) { + const own = new Set(); + const cls = pythonAttrClass(owner, attr, context, own); + let answer: ResolvedRef | null | undefined; + if (cls === 'runtime') answer = null; + else if (!cls || cls === 'absent') answer = undefined; + else { + // The class has no such method of the project's — unless a base the + // reader cannot find holds it: Django's `ComplexField(MultiValueField)`, + // its base imported from a package that re-exports it. + answer = pythonMethodOnClass(cls, methodName, ref, context, own) ?? pythonSubclassMethod(cls, methodName, ref, context) ?? + (pythonHierarchyKnown(cls, context) ? null : undefined); + } + hit = { answer, read: [...own] }; + if (memo.size >= PY_MEMO_CAP) memo.clear(); + memo.set(key, hit); + } + for (const f of hit.read) read.add(f); + return hit.answer ? { ...hit.answer, original: ref } : hit.answer; +} + +/** + * Python chained factory call — `Client.from_env().send()`, + * `make_client().send()`, `Client().send()` — the python instance of the #750 + * chained-factory family. The receiver's class is what the factory produces + * (see `pythonFactoryType`), or the class itself for a constructor; a factory + * this cannot type is no edge. Records the files it read, like + * `matchPythonAttrCall`. + */ +function matchPythonFactoryChain(ref: UnresolvedRef, context: ResolutionContext): ResolvedRef | null { + const m = /^([A-Za-z_][\w.]*)\(\)\.(\w+)$/.exec(ref.referenceName); + if (!m) return null; + const [, callee, methodName] = m as unknown as [string, string, string]; + // `Backend = load(); Backend().run()`, or a `make` parameter: the name the + // call spells is a local there, not the module's class or factory. + const lines = pythonLines(context, ref.filePath); + let fn: Node | null = null; + for (const n of context.getNodesInFile(ref.filePath)) { + if ((n.kind !== 'function' && n.kind !== 'method') || n.startLine > ref.line || n.endLine < ref.line) continue; + if (!fn || n.startLine > fn.startLine) fn = n; + } + if (lines && fn && pythonBoundInDef(lines, fn.startLine, ref.line, callee.split('.')[0]!)) return null; + const read = new Set(); + const made: PythonTypeAnswer = /^[A-Z]/.test(callee.split('.').pop()!) + ? pythonClassNamed(callee, ref.filePath, context) + : pythonFactoryType(callee, false, ref.filePath, context, read); + if (!made || made === 'runtime') return null; + const hit = pythonMethodOnClass(made, methodName, ref, context, read); + if (!hit) return null; + read.delete(ref.filePath); + return read.size > 0 ? { ...hit, metadata: { typeFrom: [...read].sort() } } : hit; +} + +/** + * Whether a python attribute-chain ref — `self.client.send`, + * `Svc.registry.lookup`, `make().send` — resolves through the attribute-type + * reader now. That reader is the only answer a body-only edit in ANOTHER file + * can change (a factory's return, a base `__init__`), so a sync asks this of + * the failed refs such an edit could affect and retries only those that it + * now answers, instead of every one in the import fan-in of the edited file. + */ +export function pythonAttrRefResolves(ref: UnresolvedRef, context: ResolutionContext): boolean { + if (ref.language !== 'python') return false; + if (ref.referenceName.includes('().')) return matchPythonFactoryChain(ref, context) !== null; + const dot = ref.referenceName.lastIndexOf('.'); + const receiver = dot > 0 ? ref.referenceName.slice(0, dot) : ''; + if (!receiver.includes('.')) return false; + return !!matchPythonAttrCall(receiver, ref.referenceName.slice(dot + 1), ref, context); +} + /** Go builtin/primitive field types that can never carry a project method. */ const GO_BUILTIN_FIELD_TYPES = new Set([ 'string', 'bool', 'byte', 'rune', 'error', 'any', @@ -22506,6 +23668,12 @@ function matchReferenceInner( ref.referenceName.includes('().') && (ref.language === 'typescript' || ref.language === 'javascript' || ref.language === 'tsx' || ref.language === 'jsx' || ref.language === 'python') ) { + if (ref.language === 'python') { + // `Client.from_env().send()`, `make_client().send()` — the factory's own + // `-> T` or returns name the receiver (#750); otherwise still nothing. + const chained = nmTimed('pythonFactoryChain', ref, () => matchPythonFactoryChain(ref, context)); + if (chained) return chained; + } return nmTimed('storeAccessorChain', ref, () => matchStoreAccessorChain(ref, context)); }