mirror of
https://github.com/colbymchenry/codegraph.git
synced 2026-10-28 07:34:23 +08:00
fix(sync): a Python import of a name a package re-exports links when the package appears later (#2507)
`from fastapi import FastAPI` reaches `fastapi/applications.py` through `fastapi/__init__.py`, which re-exports it and declares no node named `FastAPI`. While that `__init__.py` was missing, or before it gained the re-export, the calls through the name failed and were parked under the name, which the package file that then appears or changes never carries. A fresh index linked them; a sync never did. An unaliased from-import of a module in the project's own folders is now parked under the module's key, as an aliased one is (#2501): a relative import, or an absolute one whose dotted path, or the package holding it, is a project folder at any depth. An outside package's names (`from django.db.models import Q`) keep their tails, since on netbox they would crowd 'module:models', 'module:urls' and others past the retry ceiling. #2450's by-name lookup still finds every parked ref by its own name. Two limits made the restored package's importers wait anyway: - A module key gets a retry ceiling of its own, 5,000 rather than the 500 a name tail has. A large population under a module key is every importer of a package that was missing, not noise: fastapi's 1,069 `from fastapi import ...` refs, Django's 2,257 under 'module:urls'. - The CG-33 rebind's per-name ceiling counts only the edges that existed before the retry. Relinking every `from django.urls import path` pushed `path` over it, so the `path` edges the sync's new definitions moved were never rebound; the retry's own edges are still rebound under a name that stays below it, as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5.5
parent
5171a6aa9c
commit
a3fdb920a8
@@ -79,6 +79,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
||||
- A file in a `unittests/` folder, where llvm/llvm-project and google/breakpad keep their unit tests, or in a folder named like a unittest, such as `dcheck_unittest/` in google/glog, now counts as a test whatever its own name says, like LLVM's `CountCopyAndMove.h` helper. Before, such a file was taken for production code unless its name said test, and none of them were kept apart from the program the way a `tests/` folder is: search and `codegraph_explore` ranked them alongside production code, `codegraph affected` missed them, and production code linked to a unit test's own declarations, as LLVM's `Ctx` and `OpInfo` variables did to ones its unit tests declare. A folder named just `unittest/`, like CPython's `Lib/unittest/`, is still production code. Re-index projects that have such folders after upgrading.
|
||||
- `codegraph sync` and the file watcher now link a file to a module it imports under a name of its own when the module is added after the file was indexed, comes back after being deleted, or gets the export in a later edit: a default import like `import tagsController from './tag/tag.controller'`, a namespace import like `import * as api from './api'`, or an aliased one like `import { Component as GroupsPageWrapper } from './groups_page'`, in JavaScript, TypeScript, Vue, Svelte and Astro. Before, the import and every use of the name, such as a call, `new`, a base class, a JSX or template tag or an Express `app.use` mount, stayed unlinked until the importing file changed or the project was re-indexed, so the module looked unused and `codegraph affected` missed the files that use it. Re-index after upgrading to pick up links an earlier version missed.
|
||||
- `codegraph sync` and the file watcher now do the same for a Python import under a name of its own, like `from .models import User as UserModel`, `from . import helpers as h` or `from app.pkg import sub as subpkg`: when the module or package is added after the importing file was indexed, comes back after being deleted, or gets the imported name in a later edit, the import and every use of the alias, such as a call, a base class or the class passed as a value, now link to it. Before, they stayed unlinked until the importing file changed or the project was re-indexed. Re-index Python projects after upgrading to pick up links an earlier version missed.
|
||||
- `codegraph sync` and the file watcher now also link a Python import of a name a package re-exports, like `from fastapi import FastAPI` through `fastapi/__init__.py`, when that `__init__.py` is added after the importing files were indexed or gains the re-export in a later edit. Before, calls through the name stayed unlinked until each importing file changed or the project was re-indexed. Large projects gain the most: sync also used to skip, in any language, a module or package that more than 500 references were waiting for, and now restoring Django's `django/urls/__init__.py` relinks every file that imports `path` from it. Re-index Python projects after upgrading to pick up links an earlier version missed.
|
||||
- In C, C++ and Objective-C, a struct or class member declared as a pointer, reference, array or function, like `SharedState* shared;`, `jv elements[];` or `virtual Status Put(…) = 0;`, no longer makes the struct look like it inherits from the member's type. Before, `codegraph_explore` and impact listed those types as base classes, every struct holding a pointer to a type as one of its subclasses, and a method of that struct as an override of the type's method with the same name. Real base classes like `class Derived : public Base`, Objective-C superclasses and Go's embedded fields are unchanged. Re-index C, C++ and Objective-C projects after upgrading.
|
||||
- In C++, a call on a variable, parameter or member declared through a `typedef` or `using` alias now links to the method of the type the alias names, with the alias looked up the way C++ does it: in the calling function, its class and the classes that class inherits from, then the namespaces around it. In google/leveldb, `table_.Insert(buf)` on a `Table table_;` declared next to `typedef SkipList<const char*, KeyComparator> Table;` now reaches `SkipList::Insert` instead of the unrelated `HandleTable::Insert`, and `Table::Iterator iter(&table_);` reaches `SkipList::Iterator`'s constructor. A class that only shares the alias's name is no longer taken for it, and when the aliased type has no such method, as with an alias of `std::vector` or of a template parameter, no method is guessed from the receiver's name. Re-index C++ projects after upgrading.
|
||||
- In C++, a call on a variable, parameter or member declared with template arguments nested inside its template arguments, like `autovector<std::pair<int, FileMetaData*>>` or `Striped<CacheAlignedWrapper<port::Mutex>>`, now links to the method of the class it is declared as. Before, the nested arguments hid that class: in RocksDB, `files_marked_for_compaction_.clear()` on such an `autovector` linked to an unrelated `CompactionInputFiles::clear` that only shares words with the variable's name, and `mutex_.Get(key)` on a `Striped` linked to nothing. Re-index C++ projects after upgrading.
|
||||
|
||||
@@ -12,6 +12,12 @@
|
||||
* importer changed or the project was indexed again. #2450 parked the ES-module
|
||||
* bindings under their module; a Python import names its module by a dotted
|
||||
* path, relative to the importing file's package when it starts with dots.
|
||||
*
|
||||
* An unaliased name has the same gap when a package re-exports it:
|
||||
* `from fastapi import FastAPI` reaches `fastapi/applications.py` through
|
||||
* `fastapi/__init__.py`, which declares no node named `FastAPI`. So a package
|
||||
* `__init__.py` that appears after its importers, or gains the re-export,
|
||||
* left them unlinked too.
|
||||
*/
|
||||
import { describe, it, expect, afterEach } from 'vitest';
|
||||
import * as fs from 'fs';
|
||||
@@ -19,6 +25,7 @@ import * as os from 'os';
|
||||
import * as path from 'path';
|
||||
import CodeGraph from '../src/index';
|
||||
import { DatabaseConnection, getDatabasePath } from '../src/db';
|
||||
import { QueryBuilder } from '../src/db/queries';
|
||||
import { moduleReferenceKeys } from '../src/db/reference-tail';
|
||||
|
||||
type Files = Record<string, string>;
|
||||
@@ -104,6 +111,46 @@ const BY_NAME: Files = {
|
||||
};
|
||||
const BY_NAME_ADDED: Files = { 'app/schemas.py': 'class FormSchema:\n pass\n' };
|
||||
|
||||
/**
|
||||
* Unaliased imports of what a package re-exports from the modules that declare
|
||||
* it: absolutely (`from shop import Order`) and from the package above
|
||||
* (`from .. import Order`). The package renames what it re-exports, so nothing
|
||||
* named `Order` or `price` exists for a name lookup to find.
|
||||
*/
|
||||
const REEXPORT_IMPORTERS: Files = {
|
||||
'shop/models.py': 'class _Order:\n pass\n',
|
||||
'shop/services/__init__.py': '',
|
||||
'shop/services/pricing.py': 'def base_price():\n return 1\n',
|
||||
'shop/api/__init__.py': '',
|
||||
'shop/api/views.py': 'from .. import Order\n\n\ndef view():\n return Order()\n',
|
||||
'checkout.py': 'from shop import Order, price\n\n\ndef run():\n return Order(), price()\n',
|
||||
};
|
||||
const REEXPORT_PACKAGE: Files = {
|
||||
'shop/__init__.py': 'from .models import _Order as Order\nfrom .services.pricing import base_price as price\n',
|
||||
};
|
||||
const REEXPORT_LINKS = [
|
||||
'Order: imports file checkout.py -> class shop/models.py::_Order',
|
||||
'Order: imports file shop/__init__.py -> class shop/models.py::_Order',
|
||||
'Order: imports file shop/api/views.py -> class shop/models.py::_Order',
|
||||
'Order: instantiates function checkout.py -> class shop/models.py::_Order',
|
||||
'Order: instantiates function shop/api/views.py -> class shop/models.py::_Order',
|
||||
'price: calls function checkout.py -> function shop/services/pricing.py::base_price',
|
||||
'price: imports file checkout.py -> function shop/services/pricing.py::base_price',
|
||||
'price: imports file shop/__init__.py -> function shop/services/pricing.py::base_price',
|
||||
];
|
||||
|
||||
/** fastapi's shape: `from fastapi import FastAPI` through `from .applications import FastAPI as FastAPI`. */
|
||||
const SAME_NAME: Files = {
|
||||
'kit/applications.py': 'class Kit:\n pass\n',
|
||||
'app.py': 'from kit import Kit\n\n\ndef make():\n return Kit()\n',
|
||||
};
|
||||
const SAME_NAME_PACKAGE: Files = { 'kit/__init__.py': 'from .applications import Kit as Kit\n' };
|
||||
const SAME_NAME_LINKS = [
|
||||
'Kit: imports file app.py -> class kit/applications.py::Kit',
|
||||
'Kit: imports file kit/__init__.py -> class kit/applications.py::Kit',
|
||||
'Kit: instantiates function app.py -> class kit/applications.py::Kit',
|
||||
];
|
||||
|
||||
let roots: string[] = [];
|
||||
let graphs: CodeGraph[] = [];
|
||||
|
||||
@@ -145,9 +192,9 @@ function edgesOf(graph: CodeGraph): string[] {
|
||||
.sort();
|
||||
}
|
||||
|
||||
/** The {@link LINKS} the graph has. */
|
||||
function linksOf(graph: CodeGraph): string[] {
|
||||
const names = new Set(LINKS.map((link) => link.slice(0, link.indexOf(':'))));
|
||||
/** The links of `expected`'s reference names the graph has ({@link LINKS} by default). */
|
||||
function linksOf(graph: CodeGraph, expected: string[] = LINKS): string[] {
|
||||
const names = new Set(expected.map((link) => link.slice(0, link.indexOf(':'))));
|
||||
const ids = graph.getFiles().flatMap((file) => graph.getNodesInFile(file.path)).map((n) => n.id);
|
||||
const links: string[] = [];
|
||||
for (const edge of graph.getOutgoingEdgesFrom(ids)) {
|
||||
@@ -162,9 +209,13 @@ function linksOf(graph: CodeGraph): string[] {
|
||||
|
||||
/** The final tree, indexed from scratch in a folder of its own: what the sync has to match. */
|
||||
async function fullIndexOf(files: Files): Promise<string[]> {
|
||||
return edgesOf(await freshGraphOf(files));
|
||||
}
|
||||
|
||||
async function freshGraphOf(files: Files): Promise<CodeGraph> {
|
||||
const root = tempRoot();
|
||||
write(root, files);
|
||||
return edgesOf(await index(root));
|
||||
return index(root);
|
||||
}
|
||||
|
||||
describe('sync links references through a Python import alias to their module', () => {
|
||||
@@ -229,21 +280,64 @@ describe('sync links references through a Python import alias to their module',
|
||||
}, 120_000);
|
||||
});
|
||||
|
||||
describe('sync links an unaliased Python import through the package that re-exports it', () => {
|
||||
it('when the package is added after its importers', async () => {
|
||||
const root = tempRoot();
|
||||
write(root, REEXPORT_IMPORTERS);
|
||||
const graph = await index(root);
|
||||
expect(linksOf(graph, REEXPORT_LINKS)).toEqual([]);
|
||||
|
||||
write(root, REEXPORT_PACKAGE);
|
||||
const result = await graph.sync();
|
||||
expect(result.filesAdded).toBe(1);
|
||||
expect(result.filesModified).toBe(0);
|
||||
expect(linksOf(graph, REEXPORT_LINKS)).toEqual(REEXPORT_LINKS);
|
||||
expect(edgesOf(graph)).toEqual(await fullIndexOf({ ...REEXPORT_IMPORTERS, ...REEXPORT_PACKAGE }));
|
||||
}, 120_000);
|
||||
|
||||
it('when an edit gives the package the re-export', async () => {
|
||||
const root = tempRoot();
|
||||
write(root, { ...REEXPORT_IMPORTERS, 'shop/__init__.py': '' });
|
||||
const graph = await index(root);
|
||||
|
||||
write(root, REEXPORT_PACKAGE);
|
||||
expect((await graph.sync()).filesModified).toBe(1);
|
||||
expect(linksOf(graph, REEXPORT_LINKS)).toEqual(REEXPORT_LINKS);
|
||||
expect(edgesOf(graph)).toEqual(await fullIndexOf({ ...REEXPORT_IMPORTERS, ...REEXPORT_PACKAGE }));
|
||||
}, 120_000);
|
||||
|
||||
it('when it re-exports the name its module declares', async () => {
|
||||
// While the package is missing, the import itself is linked to `Kit` by
|
||||
// name, which a full index links through the package: the same link.
|
||||
const root = tempRoot();
|
||||
write(root, SAME_NAME);
|
||||
const graph = await index(root);
|
||||
|
||||
write(root, SAME_NAME_PACKAGE);
|
||||
expect((await graph.sync()).filesAdded).toBe(1);
|
||||
expect(linksOf(graph, SAME_NAME_LINKS)).toEqual(SAME_NAME_LINKS);
|
||||
expect(linksOf(await freshGraphOf({ ...SAME_NAME, ...SAME_NAME_PACKAGE }), SAME_NAME_LINKS)).toEqual(SAME_NAME_LINKS);
|
||||
}, 120_000);
|
||||
});
|
||||
|
||||
describe('a failed reference through a Python import alias', () => {
|
||||
it('waits for its module, and every other reference keeps its own tail', async () => {
|
||||
const root = tempRoot();
|
||||
write(root, {
|
||||
...IMPORTERS,
|
||||
...BY_NAME,
|
||||
// An unaliased import, a standard-library alias and a member read
|
||||
// through an alias wait for the name they use.
|
||||
...REEXPORT_IMPORTERS,
|
||||
// An unaliased import of an outside package, a standard-library alias
|
||||
// and a member read through an alias wait for the name they use; an
|
||||
// unaliased import of the project's own module waits for its module.
|
||||
'app/other.py': [
|
||||
'from .models import Group',
|
||||
'from django.db.models import Q',
|
||||
'from collections import OrderedDict as OD',
|
||||
'from .models import User as UserModel',
|
||||
'',
|
||||
'def build():',
|
||||
' return Group(), OD(), UserModel.create()',
|
||||
' return Group(), Q(), OD(), UserModel.create()',
|
||||
'',
|
||||
].join('\n'),
|
||||
});
|
||||
@@ -273,7 +367,13 @@ describe('a failed reference through a Python import alias', () => {
|
||||
// Still found by its own name too: sync's lookup by whole name.
|
||||
expect(tailOf('app/form.py', 'FormSchema', 'extends')).toBe('module:lib');
|
||||
|
||||
expect(tailOf('app/other.py', 'Group', 'calls')).toBe('Group');
|
||||
// An unaliased import of a module in the project's own folders.
|
||||
expect(tailOf('app/other.py', 'Group', 'calls')).toBe('module:models');
|
||||
expect(tailOf('checkout.py', 'Order', 'calls')).toBe('module:shop');
|
||||
expect(tailOf('checkout.py', 'price', 'imports')).toBe('module:shop');
|
||||
expect(tailOf('shop/api/views.py', 'Order', 'calls')).toBe('module:shop');
|
||||
|
||||
expect(tailOf('app/other.py', 'Q', 'calls')).toBe('Q');
|
||||
expect(tailOf('app/other.py', 'OD', 'calls')).toBe('OD');
|
||||
expect(tailOf('app/other.py', 'UserModel.create', 'calls')).toBe('create');
|
||||
|
||||
@@ -287,3 +387,54 @@ describe('a failed reference through a Python import alias', () => {
|
||||
}
|
||||
}, 60_000);
|
||||
});
|
||||
|
||||
describe('the retry ceiling', () => {
|
||||
let db: DatabaseConnection | undefined;
|
||||
|
||||
afterEach(() => {
|
||||
db?.close();
|
||||
db = undefined;
|
||||
});
|
||||
|
||||
/** 600 failed calls of `FastAPI` parked under 'module:fastapi', and 600 of `get` under their name. */
|
||||
function fixture(): QueryBuilder {
|
||||
db = DatabaseConnection.initialize(path.join(tempRoot(), 'test.db'));
|
||||
const queries = new QueryBuilder(db.getDb());
|
||||
queries.insertNode({ id: 'f', kind: 'file', name: 'app.py', qualifiedName: 'app.py', filePath: 'app.py',
|
||||
language: 'python', startLine: 1, endLine: 1, startColumn: 0, endColumn: 0, updatedAt: 0 });
|
||||
const refs = Array.from({ length: 1200 }, (_, i) => ({
|
||||
fromNodeId: 'f', referenceName: i < 600 ? 'FastAPI' : 'get', referenceKind: 'calls' as const,
|
||||
line: i + 1, column: 0, filePath: 'app.py', language: 'python' as const,
|
||||
}));
|
||||
queries.insertUnresolvedRefsBatch(refs);
|
||||
db.getDb().exec("UPDATE unresolved_refs SET status = 'failed', name_tail = CASE reference_name WHEN 'FastAPI' THEN 'module:fastapi' ELSE 'get' END");
|
||||
return queries;
|
||||
}
|
||||
|
||||
it('lets every importer of a package that appeared retry, and still skips a common name', () => {
|
||||
const queries = fixture();
|
||||
// Every importer of the package waits for it: a large population is not noise.
|
||||
expect(queries.getRetryableFailedReferences(['module:fastapi'])).toHaveLength(600);
|
||||
// A name shared by 600 failed calls is, as before; so is the by-name lookup of parked rows.
|
||||
expect(queries.getRetryableFailedReferences(['get'])).toHaveLength(0);
|
||||
expect(queries.getRetryableFailedReferences(['FastAPI'])).toHaveLength(0);
|
||||
// The module ceiling still bounds the work.
|
||||
expect(queries.getRetryableFailedReferences(['module:fastapi'], 500, 599)).toHaveLength(0);
|
||||
});
|
||||
|
||||
it("leaves the edges a sync's retry just made out of the rebind's count, not out of the rebind", () => {
|
||||
// Django: relinking every `from django.urls import path` must not stop a
|
||||
// new definition of `path` from rebinding the calls it may now answer.
|
||||
const queries = fixture();
|
||||
queries.insertNode({ id: 'p', kind: 'variable', name: 'path', qualifiedName: 'path', filePath: 'conf.py',
|
||||
language: 'python', startLine: 1, endLine: 1, startColumn: 0, endColumn: 0, updatedAt: 0 });
|
||||
const edge = (line: number) => ({ source: 'f', target: 'p', kind: 'calls' as const, line, column: 0,
|
||||
metadata: { refName: 'path', resolvedBy: 'exact-match', confidence: 0.7 } });
|
||||
queries.insertEdges([edge(1)]);
|
||||
const beforeRetry = queries.getMaxEdgeId();
|
||||
queries.insertEdges(Array.from({ length: 600 }, (_, i) => edge(i + 2)));
|
||||
|
||||
expect(queries.getResolutionEdgesByTargetName(['path'])).toHaveLength(0);
|
||||
expect(queries.getResolutionEdgesByTargetName(['path'], 500, beforeRetry)).toHaveLength(601);
|
||||
});
|
||||
});
|
||||
|
||||
+27
-5
@@ -22,7 +22,7 @@ import { kindBonus, nameMatchBonus, scorePathRelevance } from '../search/query-u
|
||||
import { parseQuery, boundedEditDistance } from '../search/query-parser';
|
||||
import { isGeneratedFile } from '../extraction/generated-detection';
|
||||
import { splitIdentifierSegments } from '../search/identifier-segments';
|
||||
import { referenceNameTail } from './reference-tail';
|
||||
import { isModuleKey, referenceNameTail } from './reference-tail';
|
||||
|
||||
/**
|
||||
* Files that should not be candidates for "dominant file" detection: test/spec
|
||||
@@ -3841,8 +3841,14 @@ export class QueryBuilder {
|
||||
* reference still waits for its own name too, looked up by its whole name:
|
||||
* a fresh index links a binding whose module never resolves it by that
|
||||
* name alone, to whichever declaration carries it.
|
||||
*
|
||||
* A module key has a ceiling of its own, `perModuleCeiling`. What waits
|
||||
* under it imports a module of that name, so a large population is not
|
||||
* noise: it is every importer of a package whose `__init__.py` was missing
|
||||
* (`from fastapi import FastAPI` across a whole test suite), which the file
|
||||
* that just appeared resolves at once. The ceiling only bounds the work.
|
||||
*/
|
||||
getRetryableFailedReferences(names: string[], perNameCeiling: number = 500): UnresolvedReference[] {
|
||||
getRetryableFailedReferences(names: string[], perNameCeiling: number = 500, perModuleCeiling: number = 5000): UnresolvedReference[] {
|
||||
if (names.length === 0) return [];
|
||||
|
||||
const lookups = [
|
||||
@@ -3863,7 +3869,8 @@ export class QueryBuilder {
|
||||
.prepare(`SELECT ${column} AS key, COUNT(*) AS count FROM unresolved_refs WHERE ${where} AND ${column} IN (${placeholders}) GROUP BY ${column}`)
|
||||
.all(...chunk) as Array<{ key: string; count: number }>;
|
||||
for (const row of counts) {
|
||||
if (row.count <= perNameCeiling) retryKeys.push(row.key);
|
||||
const ceiling = column === 'name_tail' && isModuleKey(row.key) ? perModuleCeiling : perNameCeiling;
|
||||
if (row.count <= ceiling) retryKeys.push(row.key);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -4018,6 +4025,11 @@ export class QueryBuilder {
|
||||
return changed;
|
||||
}
|
||||
|
||||
/** The highest edge id so far: edges inserted later have higher ids (AUTOINCREMENT). */
|
||||
getMaxEdgeId(): number {
|
||||
return (this.db.prepare('SELECT COALESCE(MAX(id), 0) AS id FROM edges').get() as { id: number }).id;
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolution edges whose TARGET symbol is named one of `names` — the edges a
|
||||
* sync must re-resolve after `names` gained or lost a definition (CG-33).
|
||||
@@ -4041,15 +4053,24 @@ export class QueryBuilder {
|
||||
* population the name is generic (`get`, `clear`, …), one definition changing
|
||||
* won't flip most of them, and rebinding an arbitrary subset is both wasted
|
||||
* work and incoherent coverage.
|
||||
*
|
||||
* With `countedUpToEdgeId`, only edges up to that id count toward the
|
||||
* ceiling: the ones a sync's failed-ref retry made just now were resolved
|
||||
* against the graph this sync built, and relinking every `from django.urls
|
||||
* import path` must not push `path` over the ceiling and leave the edges the
|
||||
* new definitions moved unrebound. They are still returned under a name
|
||||
* that stays below it.
|
||||
*/
|
||||
getResolutionEdgesByTargetName(
|
||||
names: string[],
|
||||
perNameCeiling: number = 500
|
||||
perNameCeiling: number = 500,
|
||||
countedUpToEdgeId?: number
|
||||
): Array<Edge & { edgeId: number; sourceFilePath: string; sourceLanguage: Language }> {
|
||||
if (names.length === 0) return [];
|
||||
|
||||
// Pass 1: per-name edge counts, chunked under the SQLite parameter limit.
|
||||
const keep: string[] = [];
|
||||
const counted = countedUpToEdgeId === undefined ? '' : 'AND e.id <= ?';
|
||||
for (let i = 0; i < names.length; i += SQLITE_PARAM_CHUNK_SIZE) {
|
||||
const chunk = names.slice(i, i + SQLITE_PARAM_CHUNK_SIZE);
|
||||
const placeholders = chunk.map(() => '?').join(',');
|
||||
@@ -4060,9 +4081,10 @@ export class QueryBuilder {
|
||||
JOIN nodes tgt ON tgt.id = e.target
|
||||
WHERE tgt.name IN (${placeholders})
|
||||
AND (e.provenance IS NULL OR e.provenance != 'heuristic')
|
||||
${counted}
|
||||
GROUP BY tgt.name`
|
||||
)
|
||||
.all(...chunk) as Array<{ name: string; count: number }>;
|
||||
.all(...chunk, ...(countedUpToEdgeId === undefined ? [] : [countedUpToEdgeId])) as Array<{ name: string; count: number }>;
|
||||
for (const row of counts) {
|
||||
if (row.count <= perNameCeiling) keep.push(row.name);
|
||||
}
|
||||
|
||||
@@ -73,6 +73,11 @@ const MODULE_REFERENCE = /^(?:layout:)?(?:lazy-import:(.+)|import:(.+)#[^#]+)$/;
|
||||
/** What a module reference's tail starts with. No symbol's name does, so a lookup by symbol names never finds one. */
|
||||
const MODULE_KEY = 'module:';
|
||||
|
||||
/** Whether a tail or lookup key waits for a module (`module:Team`) rather than for a name. */
|
||||
export function isModuleKey(key: string): boolean {
|
||||
return key.startsWith(MODULE_KEY);
|
||||
}
|
||||
|
||||
/**
|
||||
* The tail of a reference that waits for a module rather than for a name: the
|
||||
* module path's stem behind 'module:' — `./pages/Team` → 'module:Team',
|
||||
|
||||
@@ -3208,14 +3208,15 @@ export class ExtractionOrchestrator {
|
||||
* - edges whose source is in a file this sync already re-extracted are
|
||||
* skipped — their references were re-resolved from scratch moments ago;
|
||||
* - very common names are skipped by the per-name ceiling in
|
||||
* `getResolutionEdgesByTargetName`.
|
||||
* `getResolutionEdgesByTargetName`, counted over the edges up to
|
||||
* `countedUpToEdgeId` when given (those before the sync's failed-ref retry).
|
||||
*
|
||||
* Returns the number of references resurrected.
|
||||
*/
|
||||
resurrectStaleResolutionEdges(definitionDelta: string[], changedFilePaths: string[]): number {
|
||||
resurrectStaleResolutionEdges(definitionDelta: string[], changedFilePaths: string[], countedUpToEdgeId?: number): number {
|
||||
if (definitionDelta.length === 0) return 0;
|
||||
const alreadyFresh = new Set(changedFilePaths);
|
||||
const candidates = this.queries.getResolutionEdgesByTargetName(definitionDelta);
|
||||
const candidates = this.queries.getResolutionEdgesByTargetName(definitionDelta, undefined, countedUpToEdgeId);
|
||||
|
||||
const edgeIds: number[] = [];
|
||||
const refs: UnresolvedReference[] = [];
|
||||
|
||||
+7
-1
@@ -998,6 +998,10 @@ export class CodeGraph {
|
||||
|
||||
// Resolve references if files were updated
|
||||
const filesChanged = result.filesAdded > 0 || result.filesModified > 0;
|
||||
// The last edge before the failed-ref retry below: the rebind's
|
||||
// per-name ceiling counts only edges up to it (see
|
||||
// resurrectStaleResolutionEdges).
|
||||
let edgesBeforeRetry: number | undefined;
|
||||
if (filesChanged) {
|
||||
if (result.changedFilePaths) {
|
||||
// Scope resolution to changed files (git fast path — bounded set)
|
||||
@@ -1037,6 +1041,7 @@ export class CodeGraph {
|
||||
// an edit can satisfy it as well as an added file. On a sync
|
||||
// where no failed ref matches, this is one indexed lookup.
|
||||
const tRetry = Date.now();
|
||||
edgesBeforeRetry = this.queries.getMaxEdgeId();
|
||||
const retryable = this.queries.getRetryableFailedReferences([...new Set([
|
||||
...this.queries.getNodeNamesByFiles(result.changedFilePaths),
|
||||
...result.changedFilePaths.flatMap(moduleReferenceKeys),
|
||||
@@ -1122,7 +1127,8 @@ export class CodeGraph {
|
||||
const tRebind = Date.now();
|
||||
const rebound = this.orchestrator.resurrectStaleResolutionEdges(
|
||||
result.definitionDelta,
|
||||
result.changedFilePaths ?? []
|
||||
result.changedFilePaths ?? [],
|
||||
edgesBeforeRetry
|
||||
);
|
||||
if (process.env.CODEGRAPH_SYNTH_TIMINGS) {
|
||||
console.error(
|
||||
|
||||
@@ -323,6 +323,7 @@ export function clearImportResolverMemos(context: ResolutionContext): void {
|
||||
cobolCopybookIndexes.delete(context);
|
||||
pythonModuleFileMemos.delete(context);
|
||||
PY_MODULE_SYMBOLS.delete(context);
|
||||
PROJECT_FOLDER_SUFFIXES.delete(context);
|
||||
}
|
||||
|
||||
export function resolveImportPath(
|
||||
@@ -3151,22 +3152,76 @@ export function importBindingTail(ref: UnresolvedRef, context: ResolutionContext
|
||||
* import a submodule of it (`helpers` is `app/helpers.py`), whose keys hold
|
||||
* the package's name too (moduleReferenceKeys).
|
||||
*
|
||||
* An unaliased import keeps its tail: the module, or the one it re-exports the
|
||||
* name from, declares that name. So does an import of the standard library. An
|
||||
* installed package's import can't be told from a module the project doesn't
|
||||
* An installed package's alias can't be told from a module the project doesn't
|
||||
* have yet (`from settings import Config as Cfg`), so it waits for its module
|
||||
* too, and a sync still finds it by its own name.
|
||||
* too, and a sync still finds it by its own name. An import of the standard
|
||||
* library keeps its tail.
|
||||
*
|
||||
* So does an unaliased `import x`, and an unaliased `from fastapi import
|
||||
* FastAPI` unless its module is in one of the project's own folders
|
||||
* ({@link isProjectPythonModule}). The name is the one the module has, but a
|
||||
* package often re-exports it from the module that declares it
|
||||
* (`fastapi/__init__.py`: `from .applications import FastAPI as FastAPI`), and
|
||||
* a package `__init__.py` that appears or gains the re-export declares no node
|
||||
* of that name. An outside package's names stay off the module keys: every
|
||||
* `from django.db.models import Q` would crowd 'module:models' past the retry
|
||||
* ceiling, and the project's own modules with it.
|
||||
*/
|
||||
function pythonImportBindingTail(ref: UnresolvedRef, context: ResolutionContext): string | undefined {
|
||||
if (!IDENTIFIER.test(ref.referenceName)) return undefined;
|
||||
const binding = context.getImportMappings(ref.filePath, 'python').find((m) => m.localName === ref.referenceName);
|
||||
if (!binding) return undefined;
|
||||
if (!binding || isExternalImport(binding.source, 'python', context)) return undefined;
|
||||
const declared = binding.isNamespace ? binding.source.slice(binding.source.lastIndexOf('.') + 1) : binding.exportedName;
|
||||
if (declared === binding.localName || isExternalImport(binding.source, 'python', context)) return undefined;
|
||||
if (declared === binding.localName && (binding.isNamespace || !isProjectPythonModule(binding.source, context))) return undefined;
|
||||
const modulePath = pythonModulePath(binding.source, ref.filePath);
|
||||
return modulePath ? moduleTail(modulePath) || undefined : undefined;
|
||||
}
|
||||
|
||||
/**
|
||||
* Whether a Python import's module is in one of the project's own folders: a
|
||||
* relative one is, and an absolute one when a folder of the project ends in its
|
||||
* dotted path (`app/api` for `app.api`, the package) or in the package holding
|
||||
* it (`app` for `app.api`, a module file), at any depth, as `src/` layouts put
|
||||
* it. A folder, not a module: `fastapi/__init__.py` is what goes missing, while
|
||||
* `fastapi/` keeps its other files.
|
||||
*/
|
||||
function isProjectPythonModule(source: string, context: ResolutionContext): boolean {
|
||||
if (source.startsWith('.')) return true;
|
||||
const folders = projectFolderSuffixes(context);
|
||||
const modulePath = source.replace(/\./g, '/');
|
||||
const slash = modulePath.lastIndexOf('/');
|
||||
return folders.has(modulePath) || (slash > 0 && folders.has(modulePath.slice(0, slash)));
|
||||
}
|
||||
|
||||
/**
|
||||
* Per-context memo: every trailing run of folder names the project's files sit
|
||||
* in (`a/b/c.py` gives 'a/b', 'b' and 'a'). The file set is that of the
|
||||
* resolution pass; dropped by clearImportResolverMemos.
|
||||
*/
|
||||
const PROJECT_FOLDER_SUFFIXES = new WeakMap<ResolutionContext, Set<string>>();
|
||||
|
||||
function projectFolderSuffixes(context: ResolutionContext): Set<string> {
|
||||
let suffixes = PROJECT_FOLDER_SUFFIXES.get(context);
|
||||
if (suffixes) return suffixes;
|
||||
const folders = new Set<string>();
|
||||
for (const file of context.getAllFiles()) {
|
||||
for (let i = file.lastIndexOf('/'); i > 0; i = file.lastIndexOf('/', i - 1)) {
|
||||
const folder = file.slice(0, i);
|
||||
if (folders.has(folder)) break;
|
||||
folders.add(folder);
|
||||
}
|
||||
}
|
||||
suffixes = new Set<string>();
|
||||
for (const folder of folders) {
|
||||
for (let i = -1; ; i = folder.indexOf('/', i + 1)) {
|
||||
suffixes.add(folder.slice(i + 1));
|
||||
if (folder.indexOf('/', i + 1) === -1) break;
|
||||
}
|
||||
}
|
||||
PROJECT_FOLDER_SUFFIXES.set(context, suffixes);
|
||||
return suffixes;
|
||||
}
|
||||
|
||||
/**
|
||||
* A Python import's module as a project path: `app.models` → 'app/models';
|
||||
* from `app/sub/views.py`, `.models` → 'app/sub/models', `..` → 'app' and
|
||||
|
||||
Reference in New Issue
Block a user