diff --git a/CHANGELOG.md b/CHANGELOG.md index d8d9c4e2a..be4d0d65d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -223,6 +223,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - In JavaScript and TypeScript, a function the file imports from a package and passes along as a value, like react-router's `Link` in `styled(Link)`, lodash's `noop` in `createContext({ onClose: noop })` or chart.js's `Legend` in `Chart.register(Legend)`, no longer links to a project function that only shares its name. Before, whenever the project had exactly one function of that name, every such value linked to it: outline's breadcrumbs, menus and document lists showed up among the callers of an email template's `Link`, and a test's own `noop` stub in cord-field among the code the app's contexts use, so `codegraph callers`, impact and `codegraph_explore` followed references that never happen. A function the file declares itself, including one that hides the import inside a function, and one imported from a file of your project link as before. Re-index JavaScript and TypeScript projects after upgrading. - In JavaScript and TypeScript, a function a file loads with a dynamic import from Node or a package, like `const { createServer } = await import('node:http')`, `const { default: chalk } = await import('chalk')` or `import('node:fs/promises').then(({ readFile }) => …)`, no longer links to a project function that only shares its name. Before, vite's call to Node's `createServer` showed up among the callers of vite's own `createServer`. The import counts only where it is in scope, so the rest of the file links as before, and a dynamic import of your own project's module still leads to that module. Thanks @Ancienttwo for the report. (#2491) - In JavaScript and TypeScript, code outside your tests no longer links to a Jest or Vitest manual mock under `__mocks__/`. Before, when an import of a workspace package couldn't be followed, a call like `formatLabel(name)` linked by name to the mock beside it instead of the package's own function, so the real function lost its callers, and the same happened to rendered components and event handlers. Production code is now matched as if the mocks weren't there, so it reaches the real function, and adding or deleting a mock leaves it alone. Tests, other mocks, and a file that imports a mock by its path, like a `jest.setup.ts`, still reach the mock. Thanks @Y-Osawa2022 for the report and @danusha2345. (#2455) +- In JavaScript and TypeScript, an import of a package from your own repository now leads to the file its `package.json` names: the `exports` entry for the import, conditions and subpaths like `"./utils": "./src/utils.ts"` included, or else its `main`, `module`, `types` or `source`. A package another one depends on by path, like `"acme-shared": "file:../common"` or `link:`, now counts as part of your project too, for the files of the package that declares it, even with no root `workspaces`. Before, an import of a pnpm workspace member declared with `"main": "./src/index.ts"` and no `index` file at its root, or of a `file:` dependency, couldn't be followed, so its calls were matched by name alone and a same-named private function nearer the caller could take them. An entry that names build output, like `./dist/esm/index.mjs`, or an import of a path inside it, like `@jsonjoy.com/fs-node-utils/lib/types/misc`, leads to the matching source file, like `src/index.ts` or `src/types/misc.ts`, when your project has one, and to nothing otherwise. `codegraph sync` follows edits to these `package.json` fields. Re-index JavaScript and TypeScript monorepos after upgrading. Thanks @elderbernardi and @Y-Osawa2022 for the reports, and @maxmilian and @danusha2345. (#2502, #2456) - In JavaScript and TypeScript, a value the file imports from your own project and passes along now links only to what it imports: a constant like `isCloudHosted`, a class like a `Star` model, a whole module (`import * as Slack from '../slack'`), or a stylesheet, image or GraphQL document (`import TCO21 from './themes/TCO21.scss'`) no longer links to a function elsewhere that only shares its name. Before, when the project had exactly one function of that name, such a value linked to it: outline's settings screens showed up among the callers of the server's `isCloudHosted()` check, and its stores among those of the `` icon, so `codegraph callers`, impact and `codegraph_explore` followed references that never happen. A component that wraps the function declared beside it, like `export default styled(Disclosure)`, and a page loaded lazily, like `lazy(() => import('../pages/Home'))`, still lead to that function. Re-index JavaScript and TypeScript projects after upgrading. - In JavaScript, an import of a TypeScript module written without an extension, like `import AudioModal from './audio_modal'` in a `.jsx` file next to `audio_modal.tsx`, now links to that module, as it already did from a TypeScript file. Before, a `.js` or `.jsx` file's imports of TypeScript modules were never followed, so in a codebase moving to TypeScript they were matched by name instead: an import under another name linked nothing, and a name that repeats could link to the wrong file, so mastodon's `` and `` tags rendered other components of those names than the ones each file imports. A JavaScript module beside a TypeScript one of the same name still comes first. Re-index JavaScript projects after upgrading. - `codegraph_explore`'s Flow now goes through the implementation your query names when an interface method has several that lead to the same place. Before, it took whichever one came first in the index: asking about prometheus's `Engine.execEvalStmt Queryable.Querier fanout.Querier NewMergeQuerier` traced the call through the TSDB's `DB.Querier` and left the `fanout.Querier` you named off the Flow. Any two routes of the same length are now settled this way, in favor of the one that passes through more of the symbols you named. diff --git a/__tests__/workspace-package-entries.test.ts b/__tests__/workspace-package-entries.test.ts new file mode 100644 index 000000000..855f36e0b --- /dev/null +++ b/__tests__/workspace-package-entries.test.ts @@ -0,0 +1,370 @@ +/** + * Issues #2456 and #2502: a package of the repository is imported by its name, + * and that import has to land on the file the package's manifest names. + * + * - #2502: a pnpm (or npm/yarn/bun) workspace member that declares its entry + * with `main` / `module` / `types` or an `exports` map, like + * `"main": "./src/index.ts"`, had no `index.*` at its root, so + * `import { caseStatus } from '@acme/core'` failed and its calls fell back to + * name matching: a same-named private function nearer the caller took them. + * - #2456: a sibling package an npm `file:` dependency points at, with no root + * `workspaces`, was not known as a package at all. + * + * A declared entry that names build output the index does not hold (`dist/`) + * is read as the source it is built from when the mapping is clear + * (`dist/index.js` → `src/index.ts`), and otherwise links nothing new. + */ +import { describe, it, expect, afterEach } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; + +let roots: string[] = []; +let graphs: CodeGraph[] = []; + +afterEach(() => { + for (const graph of graphs) graph.close(); + graphs = []; + for (const root of roots) fs.rmSync(root, { recursive: true, force: true }); + roots = []; +}); + +function write(root: string, files: Record): void { + for (const [rel, content] of Object.entries(files)) { + const file = path.join(root, rel); + if (content === null) { + fs.rmSync(file, { force: true }); + continue; + } + fs.mkdirSync(path.dirname(file), { recursive: true }); + fs.writeFileSync(file, content); + } +} + +async function index(files: Record): Promise { + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-ws-entries-')); + roots.push(root); + write(root, files); + const graph = await CodeGraph.init(root, { index: true }); + graphs.push(graph); + return graph; +} + +/** The files `file`'s import statements link it to. */ +function moduleImports(graph: CodeGraph, file: string): string[] { + const out = new Set(); + for (const node of graph.getNodesInFile(file)) { + for (const e of graph.getOutgoingEdges(node.id)) { + if (e.kind !== 'imports') continue; + const target = graph.getNode(e.target)!; + if (target.kind === 'file') out.add(target.filePath); + } + } + return [...out].sort(); +} + +/** `name@file` of every call `file` makes. */ +function callsFrom(graph: CodeGraph, file: string): string[] { + const out: string[] = []; + for (const node of graph.getNodesInFile(file)) { + for (const e of graph.getOutgoingEdges(node.id)) { + if (e.kind !== 'calls') continue; + const target = graph.getNode(e.target)!; + out.push(`${target.name}@${target.filePath}`); + } + } + return out.sort(); +} + +/** A pnpm workspace whose `@acme/core` member has the given manifest. */ +function pnpm(core: Record, extra: Record = {}): Record { + return { + 'package.json': JSON.stringify({ name: 'repo', private: true }), + 'pnpm-workspace.yaml': 'packages:\n - "packages/*"\n - "apps/*"\n', + 'packages/core/package.json': JSON.stringify({ name: '@acme/core', ...core }), + 'packages/core/src/index.ts': 'export function coreFn(): number { return 1; }\n', + 'apps/web/package.json': JSON.stringify({ name: '@acme/web', dependencies: { '@acme/core': 'workspace:*' } }), + 'apps/web/main.ts': "import { coreFn } from '@acme/core';\nexport const run = () => coreFn();\n", + ...extra, + }; +} + +describe('a workspace member imported by name lands on the entry its manifest declares (#2502)', () => { + it.each([ + ['main', { main: './src/index.ts' }], + ['module', { module: './src/index.ts' }], + ['types', { types: './src/index.ts' }], + ['a string `exports`', { exports: './src/index.ts' }], + ['`exports["."]`', { exports: { '.': './src/index.ts' } }], + ['conditions straight under `exports`', { exports: { types: './dist/index.d.ts', import: './src/index.ts' } }], + ])('through %s', async (_label, manifest) => { + const graph = await index(pnpm(manifest)); + expect(moduleImports(graph, 'apps/web/main.ts')).toEqual(['packages/core/src/index.ts']); + expect(callsFrom(graph, 'apps/web/main.ts')).toContain('coreFn@packages/core/src/index.ts'); + }, 120_000); + + it('through conditions, nested ones included, whichever the index holds', async () => { + // `types` and `require` name build output; the source sits under `import` → `default`. + const exports = { + '.': { + types: './dist/index.d.ts', + import: { types: './dist/index.d.mts', default: './src/index.ts' }, + require: './dist/index.cjs', + }, + }; + const graph = await index(pnpm({ exports })); + expect(moduleImports(graph, 'apps/web/main.ts')).toEqual(['packages/core/src/index.ts']); + }, 120_000); + + it("through the condition its importers resolve by, read as its source, before a custom condition's source", async () => { + // @wdio/logger's layout: `import` names build output, `browserSource` a browser-only source file. + const exports = { + '.': { + browserSource: './src/browser.ts', + browser: './build/browser.js', + types: './build/index.d.ts', + import: './build/index.js', + require: './build/index.cjs', + }, + }; + const graph = await index(pnpm({ exports }, { + 'packages/core/src/browser.ts': 'export function coreFn(): number { return 5; }\n', + })); + expect(moduleImports(graph, 'apps/web/main.ts')).toEqual(['packages/core/src/index.ts']); + }, 120_000); + + it('through the implementation a condition names before a declaration file', async () => { + const graph = await index(pnpm({ exports: { '.': { types: './src/index.d.ts', import: './src/index.ts' } } }, { + 'packages/core/src/index.d.ts': 'export declare function coreFn(): number;\n', + })); + expect(moduleImports(graph, 'apps/web/main.ts')).toEqual(['packages/core/src/index.ts']); + }, 120_000); + + it('never through build output that is on disk but not indexed', async () => { + // A local build left `dist/` behind; `.gitignore` keeps it out of the index. + const graph = await index(pnpm({ main: './dist/index.js' }, { + '.gitignore': 'dist/\n', + 'packages/core/dist/index.js': 'exports.coreFn = function () { return 1; };\n', + })); + expect(moduleImports(graph, 'apps/web/main.ts')).toEqual(['packages/core/src/index.ts']); + }, 120_000); + + it('through a subpath export and a `./*` pattern', async () => { + const graph = await index(pnpm( + { exports: { '.': './src/index.ts', './utils': './src/utils.ts', './schema/*': { import: './src/schema/*.ts' } } }, + { + 'packages/core/src/utils.ts': 'export function trim(s: string): string { return s.trim(); }\n', + 'packages/core/src/schema/case.ts': "export const caseTable = 'case';\n", + 'apps/web/text.ts': "import { trim } from '@acme/core/utils';\nexport const clean = (s: string) => trim(s);\n", + 'apps/web/table.ts': "import { caseTable } from '@acme/core/schema/case';\nexport const tableName = () => caseTable;\n", + }, + )); + expect(moduleImports(graph, 'apps/web/text.ts')).toEqual(['packages/core/src/utils.ts']); + expect(callsFrom(graph, 'apps/web/text.ts')).toContain('trim@packages/core/src/utils.ts'); + expect(moduleImports(graph, 'apps/web/table.ts')).toEqual(['packages/core/src/schema/case.ts']); + }, 120_000); + + it('through build output the index lacks, read as the source it is built from', async () => { + const graph = await index(pnpm({ + main: './dist/cjs/index.cjs', + types: './dist/types/index.d.ts', + exports: { '.': { import: './dist/esm/index.mjs', require: './dist/cjs/index.cjs' }, './utils': './dist/utils.js' }, + }, { + 'packages/core/src/utils.ts': 'export function trim(s: string): string { return s.trim(); }\n', + 'apps/web/text.ts': "import { trim } from '@acme/core/utils';\nexport const clean = (s: string) => trim(s);\n", + })); + expect(moduleImports(graph, 'apps/web/main.ts')).toEqual(['packages/core/src/index.ts']); + expect(moduleImports(graph, 'apps/web/text.ts')).toEqual(['packages/core/src/utils.ts']); + }, 120_000); + + it('through a path into its build output, read as the source it is built from', async () => { + // memfs's layout: `import { … } from '@jsonjoy.com/fs-node-utils/lib/types/misc'` is + // `lib/types/misc.js`, built from `src/types/misc.ts`. A local build on disk changes nothing. + const graph = await index(pnpm({ main: './lib/index.js' }, { + '.gitignore': 'lib/\n', + 'packages/core/lib/strings.js': 'exports.trim = function (s) { return s; };\n', + 'packages/core/src/strings.ts': 'export function trim(s: string): string { return s.trim(); }\n', + 'packages/core/src/types/misc.ts': 'export type Encoding = string;\n', + 'apps/web/text.ts': + "import { trim } from '@acme/core/lib/strings';\nimport type { Encoding } from '@acme/core/lib/types/misc';\nexport const clean = (s: string, e: Encoding) => trim(s) + e;\n", + })); + expect(moduleImports(graph, 'apps/web/text.ts')).toEqual(['packages/core/src/strings.ts', 'packages/core/src/types/misc.ts']); + expect(callsFrom(graph, 'apps/web/text.ts')).toEqual(['trim@packages/core/src/strings.ts']); + }, 120_000); + + it('through the source tree a build folder mirrors before the folder read as a build format', async () => { + // Vite's layout: `dist/node/index.js` is built from `src/node/index.ts`, not `src/index.ts`. + const graph = await index(pnpm({ main: './dist/node/index.js', types: './dist/node/index.d.ts' }, { + 'packages/core/src/node/index.ts': 'export function coreFn(): number { return 3; }\n', + })); + expect(moduleImports(graph, 'apps/web/main.ts')).toEqual(['packages/core/src/node/index.ts']); + expect(callsFrom(graph, 'apps/web/main.ts')).toContain('coreFn@packages/core/src/node/index.ts'); + }, 120_000); + + it("through `exports` first, then the root's `main`-style fields when `exports` leads to no indexed file", async () => { + // `exports` read as its source beats a root `index.js` that `main` names… + const first = await index(pnpm({ exports: { '.': './dist/index.mjs' }, main: './index.js' }, { + 'packages/core/index.js': "module.exports = require('./dist/index.cjs');\n", + })); + expect(moduleImports(first, 'apps/web/main.ts')).toEqual(['packages/core/src/index.ts']); + // …and `source` is read when `exports` names build output with no clear source. + const then = await index(pnpm({ exports: { '.': './dist/acme-core.mjs' }, source: './src/main.ts' }, { + 'packages/core/src/main.ts': 'export function coreFn(): number { return 4; }\n', + })); + expect(moduleImports(then, 'apps/web/main.ts')).toEqual(['packages/core/src/main.ts']); + }, 120_000); + + it('and links nothing new when build output has no clear source', async () => { + // `dist/bundle.js` has no `src/bundle.*`: no guess at another file. + const graph = await index(pnpm({ main: './dist/bundle.js' })); + expect(moduleImports(graph, 'apps/web/main.ts')).toEqual([]); + }, 120_000); + + it("so a same-named private function no longer takes the package's callers (the issue's repro)", async () => { + const graph = await index({ + 'package.json': JSON.stringify({ name: 'repro', private: true }), + 'pnpm-workspace.yaml': 'packages:\n - "packages/*"\n - "apps/*"\n', + 'packages/core/package.json': JSON.stringify({ + name: '@acme/core', type: 'module', main: './src/index.ts', exports: { '.': './src/index.ts' }, + }), + 'packages/core/src/index.ts': "export * from './status.js';\n", + 'packages/core/src/status.ts': + "export function caseStatus(input: { open: boolean }): { value: string } {\n return { value: input.open ? 'open' : 'closed' };\n}\n", + 'packages/db/package.json': JSON.stringify({ name: '@acme/db', type: 'module' }), + 'packages/db/src/load.ts': + "import { caseStatus } from '@acme/core';\nexport const loadStatus = (open: boolean) => caseStatus({ open }).value;\n", + 'apps/web/package.json': JSON.stringify({ name: '@acme/web', type: 'module' }), + 'apps/web/lib/list.ts': + "import { caseStatus } from '@acme/core';\nexport const listCases = (cases: { open: boolean }[]) => cases.map((c) => caseStatus(c).value);\n", + 'apps/worker/package.json': JSON.stringify({ name: '@acme/worker', type: 'module' }), + 'apps/worker/src/run.ts': + "import { caseStatus } from '@acme/core';\nexport const run = () => caseStatus({ open: true }).value;\n", + 'apps/worker/src/interpret.ts': + 'function caseStatus(raw: string): string { return raw.trim(); }\nexport const interpret = (raw: string) => caseStatus(raw);\n', + }); + for (const file of ['packages/db/src/load.ts', 'apps/web/lib/list.ts', 'apps/worker/src/run.ts']) { + expect(callsFrom(graph, file)).toContain('caseStatus@packages/core/src/status.ts'); + expect(callsFrom(graph, file)).not.toContain('caseStatus@apps/worker/src/interpret.ts'); + } + }, 120_000); +}); + +describe('a package an npm `file:` dependency points at is imported by its name (#2456)', () => { + /** The issue's layout: no root manifest, `acme-api` depends on `file:../common`. */ + const fileDep = (spec: string, extra: Record = {}): Record => ({ + 'packages/common/package.json': JSON.stringify({ name: 'acme-shared', version: '1.0.0' }), + 'packages/common/utils.ts': + "export function formatLabel(name: string): string { return name.trim(); }\nexport function isAdmin(role: string): boolean { return role === 'admin'; }\n", + 'packages/api/package.json': JSON.stringify({ name: 'acme-api', dependencies: { 'acme-shared': spec } }), + 'packages/api/src/handler.ts': + "import { formatLabel, isAdmin } from 'acme-shared/utils';\nexport function handleRequest(name: string, role: string): string {\n return isAdmin(role) ? formatLabel(name) : 'forbidden';\n}\n", + ...extra, + }); + + it.each(['file:../common', 'link:../common'])('with %s and no root `workspaces`', async (spec) => { + const graph = await index(fileDep(spec, { + // A same-named function nearer the caller, which name matching would pick. + 'packages/api/src/format.ts': "export function formatLabel(name: string): string { return '!' + name; }\n", + })); + expect(moduleImports(graph, 'packages/api/src/handler.ts')).toEqual(['packages/common/utils.ts']); + expect(callsFrom(graph, 'packages/api/src/handler.ts')).toEqual([ + 'formatLabel@packages/common/utils.ts', 'isAdmin@packages/common/utils.ts', + ]); + }, 120_000); + + it('for every file when the root manifest declares it', async () => { + // An Expo app's local native module, as `link:` from the root package.json. + const graph = await index({ + 'package.json': JSON.stringify({ name: 'app', dependencies: { '@acme/native': 'link:./modules/native' } }), + 'modules/native/package.json': JSON.stringify({ name: '@acme/native', main: './src/index.ts' }), + 'modules/native/src/index.ts': 'export function share(text: string): string { return text; }\n', + 'src/screens/Share.ts': "import { share } from '@acme/native';\nexport const onShare = () => share('hi');\n", + }); + expect(moduleImports(graph, 'src/screens/Share.ts')).toEqual(['modules/native/src/index.ts']); + expect(callsFrom(graph, 'src/screens/Share.ts')).toEqual(['share@modules/native/src/index.ts']); + }, 120_000); + + it("only for the files under the manifest that declares it", async () => { + // A test fixture's `left-pad` stub is the fixture's; the app's `left-pad` stays the npm package. + const graph = await index({ + 'package.json': JSON.stringify({ name: 'app', dependencies: { 'left-pad': '^1.3.0' } }), + 'src/pad.ts': "import leftPad from 'left-pad';\nexport const pad = (s: string) => leftPad(s, 4);\n", + 'test/fixture/package.json': JSON.stringify({ name: 'fixture', dependencies: { 'left-pad': 'file:../stub-left-pad' } }), + 'test/fixture/use.ts': "import leftPad from 'left-pad';\nexport const use = () => leftPad('x', 2);\n", + 'test/stub-left-pad/package.json': JSON.stringify({ name: 'left-pad', main: 'index.ts' }), + 'test/stub-left-pad/index.ts': 'export default function leftPad(s: string, n: number): string { return s.padStart(n); }\n', + }); + expect(moduleImports(graph, 'test/fixture/use.ts')).toEqual(['test/stub-left-pad/index.ts']); + expect(moduleImports(graph, 'src/pad.ts')).toEqual([]); + expect(callsFrom(graph, 'src/pad.ts')).toEqual([]); + }, 120_000); + + it("through the target's own `exports`", async () => { + const graph = await index({ + 'packages/common/package.json': JSON.stringify({ name: 'acme-shared', exports: { './utils': './src/utils.ts' } }), + 'packages/common/src/utils.ts': 'export function formatLabel(name: string): string { return name; }\n', + 'packages/api/package.json': JSON.stringify({ name: 'acme-api', dependencies: { 'acme-shared': 'file:../common' } }), + 'packages/api/handler.ts': "import { formatLabel } from 'acme-shared/utils';\nexport const h = (n: string) => formatLabel(n);\n", + }); + expect(moduleImports(graph, 'packages/api/handler.ts')).toEqual(['packages/common/src/utils.ts']); + }, 120_000); +}); + +describe('sync follows a manifest edit (#2456, #2502)', () => { + it("re-links importers when a member's `main` or `exports` changes, as a fresh index would", async () => { + const files = pnpm({ main: './dist/bundle.js' }, { + 'packages/core/src/next.ts': 'export function coreFn(): number { return 2; }\n', + }); + const graph = await index(files); + const root = roots[roots.length - 1]!; + expect(moduleImports(graph, 'apps/web/main.ts')).toEqual([]); + + write(root, { 'packages/core/package.json': JSON.stringify({ name: '@acme/core', main: './src/index.ts' }) }); + await graph.sync(); + expect(moduleImports(graph, 'apps/web/main.ts')).toEqual(['packages/core/src/index.ts']); + + write(root, { 'packages/core/package.json': JSON.stringify({ name: '@acme/core', exports: { '.': { import: './src/next.ts' } } }) }); + await graph.sync(); + expect(moduleImports(graph, 'apps/web/main.ts')).toEqual(['packages/core/src/next.ts']); + expect(callsFrom(graph, 'apps/web/main.ts')).toContain('coreFn@packages/core/src/next.ts'); + }, 120_000); + + it('re-links importers when a `file:` dependency appears', async () => { + const graph = await index({ + 'packages/common/package.json': JSON.stringify({ name: 'acme-shared' }), + 'packages/common/utils.ts': 'export function formatLabel(name: string): string { return name; }\n', + 'packages/api/package.json': JSON.stringify({ name: 'acme-api' }), + 'packages/api/handler.ts': "import { formatLabel } from 'acme-shared/utils';\nexport const h = (n: string) => formatLabel(n);\n", + }); + const root = roots[roots.length - 1]!; + expect(moduleImports(graph, 'packages/api/handler.ts')).toEqual([]); + write(root, { 'packages/api/package.json': JSON.stringify({ name: 'acme-api', dependencies: { 'acme-shared': 'file:../common' } }) }); + await graph.sync(); + expect(moduleImports(graph, 'packages/api/handler.ts')).toEqual(['packages/common/utils.ts']); + }, 120_000); +}); + +describe('sync follows the entry a manifest declares when its file appears (#2502)', () => { + it.each([ + ['an `exports` entry', { exports: { '.': './src/index.ts' } }], + ['the source of a `main` in `dist/`', { main: './dist/index.js' }], + ])("re-links importers that fell back to the package's root index, through %s", async (_label, manifest) => { + const files = pnpm(manifest, { + 'packages/core/index.ts': "export function caseStatus(open: boolean): string { return 'root'; }\n", + 'packages/core/src/status.ts': "export function caseStatus(open: boolean): string { return open ? 'open' : 'closed'; }\n", + 'apps/web/main.ts': "import { caseStatus } from '@acme/core';\nexport const run = () => caseStatus(true);\n", + }); + delete files['packages/core/src/index.ts']; + const graph = await index(files); + const root = roots[roots.length - 1]!; + expect(moduleImports(graph, 'apps/web/main.ts')).toEqual(['packages/core/index.ts']); + expect(callsFrom(graph, 'apps/web/main.ts')).toEqual(['caseStatus@packages/core/index.ts']); + + // The barrel names no `caseStatus` of its own: only the import leads the call there. + write(root, { 'packages/core/src/index.ts': "export * from './status';\n" }); + await graph.sync(); + expect(moduleImports(graph, 'apps/web/main.ts')).toEqual(['packages/core/src/index.ts']); + expect(callsFrom(graph, 'apps/web/main.ts')).toEqual(['caseStatus@packages/core/src/status.ts']); + }, 120_000); +}); diff --git a/src/db/queries.ts b/src/db/queries.ts index 58904634b..88004853d 100644 --- a/src/db/queries.ts +++ b/src/db/queries.ts @@ -3403,6 +3403,37 @@ export class QueryBuilder { })); } + /** + * The module edges (`imports` edges into a file node) into any file under + * the folder `dir` (projectRoot-relative, no trailing slash), with the + * source file and language a resurrection needs: where an import of a + * package by its name can land, whichever file of the package its manifest + * or the folder's index makes the entry. + */ + getModuleImportsUnder( + dir: string + ): Array { + // `dir/` up to `dir0` ('0' is the byte after '/'). + const rows = this.db + .prepare( + `SELECT e.*, tgt.file_path AS target_file_path, src.file_path AS source_file_path, src.language AS source_language + FROM nodes tgt + JOIN edges e ON e.target = tgt.id AND e.kind = 'imports' + JOIN nodes src ON src.id = e.source + WHERE tgt.kind = 'file' AND tgt.file_path >= ? AND tgt.file_path < ?` + ) + .all(`${dir}/`, `${dir}0`) as Array< + EdgeRow & { target_file_path: string; source_file_path: string; source_language: Language } + >; + return rows.map((row) => ({ + ...rowToEdge(row), + edgeId: row.id, + targetFilePath: row.target_file_path, + sourceFilePath: row.source_file_path, + sourceLanguage: row.source_language, + })); + } + /** * The module edges (`imports` edges into a file node) into the files an * import of `dir/name` can find instead of a file added there: `dir/name.*` diff --git a/src/index.ts b/src/index.ts index b702890af..0a62d0241 100644 --- a/src/index.ts +++ b/src/index.ts @@ -68,6 +68,7 @@ import { loadDeprioritizePatterns } from './project-config'; import { CodeGraphPackageVersion } from './mcp/version'; import { extractSegmentSearchWords, segmentLookupVariants, splitIdentifierSegments } from './search/identifier-segments'; import { createYielder } from './resolution/cooperative-yield'; +import { ESM_IMPORT_LANGUAGES } from './resolution/import-resolver'; import { minRefsForPool } from './resolution/resolver-pool'; // Re-export types for consumers @@ -150,6 +151,8 @@ export interface IndexOptions { */ /** project_metadata key for the Python package-root digest (see sync). */ const PYTHON_ROOTS_METADATA_KEY = 'python_root_fingerprint'; +/** Metadata key: the digest of the workspace packages and path dependencies the index was resolved by. */ +const WORKSPACE_METADATA_KEY = 'js_workspace_fingerprint'; export class CodeGraph { private db: DatabaseConnection; @@ -772,6 +775,7 @@ export class CodeGraph { // real full index — a sync touches a subset, so it must NOT advance the // extraction stamp (the bulk would still be stale). See extraction-version.ts. if (result.success) this.stampPythonRootFingerprint(); + if (result.success) this.stampWorkspaceFingerprint(); if (result.success && result.filesIndexed > 0) { try { @@ -882,6 +886,13 @@ export class CodeGraph { } catch { /* metadata is advisory */ } } + /** Record the workspace-package digest a later sync compares against (see sync). */ + private stampWorkspaceFingerprint(fingerprint = this.resolver.getWorkspaceFingerprint()): void { + try { + this.queries.setMetadata(WORKSPACE_METADATA_KEY, fingerprint); + } catch { /* metadata is advisory */ } + } + async sync(options: IndexOptions = {}): Promise { return this.indexMutex.withLock(async () => { const startedAt = Date.now(); @@ -1047,6 +1058,35 @@ export class CodeGraph { } } + // Workspace packages and path dependencies (#2456, #2502). A + // package.json edit — a member's name, its `main` or `exports`, a + // `file:` dependency — moves what an import of the package resolves to + // in files it never touched, and no indexed file changes with it. A + // full reconcile (`codegraph sync`, a git hook, the watcher's catch-up) + // reads the manifests again; when what they declare moved, every + // JavaScript-family reference is re-opened, as a moved Python root set + // re-opens Python's below, and the orphan sweep resolves them by the + // new map exactly as a fresh index would. Package.json edits that leave + // the map alone (a version bump, a new registry dependency) cost one + // read. The watcher's scoped syncs follow source files only and skip + // the read: a monorepo's manifests take a walk of the tree to find. + // An index stamped by an older engine has no digest: record one. + if (fullReconcile) { + const tWorkspace = Date.now(); + const before = this.queries.getMetadata(WORKSPACE_METADATA_KEY); + const now = this.resolver.refreshWorkspacePackages(); + if (before !== null && before !== now) { + let reopened = 0; + for (const language of ESM_IMPORT_LANGUAGES) reopened += this.orchestrator.resurrectResolutionEdgesForLanguage(language); + refreshSynthesis = true; + this.queries.setMetadata('synthesis_pending', '1'); + if (process.env.CODEGRAPH_SYNTH_TIMINGS) { + console.error(`[phase-timing] sync-workspace-packages: ${Date.now() - tWorkspace}ms (${reopened} refs re-opened)`); + } + } + if (before !== now) this.stampWorkspaceFingerprint(now); + } + // Resolve references if files were updated const filesChanged = result.filesAdded > 0 || result.filesModified > 0; // The last edge before the failed-ref retry below: the rebind's @@ -1170,7 +1210,9 @@ export class CodeGraph { // `index.ts` re-exporting what it imports: what the files importing // it bound by name in its absence, or failed, goes back to pending // for the sweep below, which binds it through the module as a fresh - // index does. (The removal half runs before removed files are deleted.) + // index does. So does what an import of a workspace package bound + // through another of its files while the entry its manifest names + // was missing. (The removal half runs before removed files are deleted.) const importReader = createImportReader(this.projectRoot); const added = (result.addedFilePaths ?? []).flatMap((filePath) => { const file = this.queries.getFileByPath(filePath); @@ -1180,7 +1222,8 @@ export class CodeGraph { const tAdded = Date.now(); const reopened = reopenReferencesThroughAddedFiles( this.queries, importReader, added, edgesBeforeResolution, - (specifier, fromFile, language) => this.resolver.resolveModuleFile(specifier, fromFile, language) + (specifier, fromFile, language) => this.resolver.resolveModuleFile(specifier, fromFile, language), + (file) => this.resolver.workspacePackageDirOf(file) ); if (process.env.CODEGRAPH_SYNTH_TIMINGS) console.error(`[phase-timing] sync-added-module-importers: ${Date.now() - tAdded}ms (${reopened} refs re-opened)`); } diff --git a/src/resolution/import-resolver.ts b/src/resolution/import-resolver.ts index 6d11a974e..01712149b 100644 --- a/src/resolution/import-resolver.ts +++ b/src/resolution/import-resolver.ts @@ -11,7 +11,7 @@ import { Language, Node } from '../types'; import { UnresolvedRef, ResolvedRef, ResolutionContext, ImportMapping, ReExport } from './types'; import { applyAliases, aliasAndBaseUrlCandidates } from './path-aliases'; import { extractLocalExportAliases } from './alias-binding'; -import { resolveWorkspaceImport } from './workspace-packages'; +import { resolveWorkspaceImport, workspaceImportTargets, type WorkspaceImportTargets } from './workspace-packages'; import { stripCommentsForRegex } from './strip-comments'; import { makeLineAt } from './synth-utils'; import { dartDirectiveFile } from './dart-libraries'; @@ -389,7 +389,7 @@ function resolveImportPathUncached( // alias map first (custom prefixes like `@components/*` would // otherwise be misclassified as npm). What still looks like a package // may name a file through the importing app's own config. - if (isExternalImport(importPath, language, context)) { + if (isExternalImport(importPath, language, context, { fromFile })) { return resolveFromOwnConfig(importPath, fromFile, language, context); } @@ -1105,12 +1105,16 @@ const RUST_STDLIB_ROOTS = new Set(['std', 'core', 'alloc', 'proc_macro']); * asked `resolveImportPath` whether an alias maps the specifier to a file. * Matching a prefix proves nothing by itself: a catch-all `"*"` pattern * (`"*": ["./typings/*"]`) has an empty prefix and matches every package. + * + * `fromFile`, the importing file, lets a package another package depends on + * by path (`"acme-shared": "file:../common"`) count as the project's for the + * files under the manifest that declares it (#2456). */ export function isExternalImport( importPath: string, language: Language, context?: ResolutionContext, - options: { aliasPrefixes?: boolean } = {} + options: { aliasPrefixes?: boolean; fromFile?: string } = {} ): boolean { // Relative imports are not external if (importPath.startsWith('.')) { @@ -1122,7 +1126,7 @@ export function isExternalImport( // workspace map first so they aren't misclassified as external (#629). The // map is null for single-package repos, so this is a no-op there. const workspaces = context?.getWorkspacePackages?.(); - if (workspaces && resolveWorkspaceImport(importPath, workspaces)) { + if (workspaces && resolveWorkspaceImport(importPath, workspaces, options.fromFile)) { return false; } @@ -1244,7 +1248,8 @@ const EMITTED_TO_SOURCE_EXTENSIONS: ReadonlyArray boolean = (file) => context.fileExists(file) ): string | null { if (!EMITTED_SPECIFIER_LANGUAGES.has(language)) return null; for (const [emitted, sources] of EMITTED_TO_SOURCE_EXTENSIONS) { @@ -1252,7 +1257,7 @@ function findSourceForEmittedSpecifier( const stem = relativePath.replace(emitted, ''); for (const ext of sources) { const candidate = stem + ext; - if (context.fileExists(candidate)) return candidate; + if (exists(candidate)) return candidate; } return null; } @@ -1297,16 +1302,23 @@ function resolveAliasedImport( } } - // 1.5 Workspace packages (`@scope/ui/widgets` → `packages/ui/widgets`). - // Resolves a monorepo member import to the member's directory; the - // extension/index permutations below then find its barrel (#629). + // 1.5 Workspace packages and path dependencies: the entry the package's + // manifest declares for the import (#2502), else the member's + // directory (`@scope/ui/widgets` → `packages/ui/widgets`), where the + // extension/index permutations find its barrel (#629). A path into the + // package's build output the index lacks (`@acme/utils/lib/path`) is + // read as the source it is built from (`src/path.ts`), as a declared + // entry is, before a file only a local build left on disk. const workspaces = context.getWorkspacePackages?.(); - if (workspaces) { - const base = resolveWorkspaceImport(importPath, workspaces); - if (base) { - const hit = tryWithExt(base); - if (hit) return hit; - } + const targets = workspaces ? workspaceImportTargets(importPath, workspaces, fromFile) : null; + if (targets) { + const declared = declaredWorkspaceEntry(targets, language, context); + if (declared) return declared; + const direct = tryWithExt(targets.directory); + if (direct && (context.isIndexedFile?.(direct) ?? true)) return direct; + const built = builtSourceOf(targets.directory, targets.packageDir, language, context); + if (built) return built; + if (direct) return direct; } // 2. Hard-coded fallback list. Kept for projects that use these @@ -1364,6 +1376,80 @@ function indexedSpelling(filePath: string, language: Language): string { : filePath; } +/** + * The file a package's declared entry for an import lands on (#2502): the + * first declared target, in order (`exports` by condition, then the root's + * `main`-style fields), that the index holds, or that is build output the + * index lacks (`dist/index.js`), read as the source it is built from + * (`src/index.ts`). So `import`'s `./build/index.js` leads to `src/index.ts` + * before a custom condition's `./src/browser.ts` does. Null when none holds + * — the caller then tries the package's directory, as it did before entries + * were read. + */ +function declaredWorkspaceEntry(targets: WorkspaceImportTargets, language: Language, context: ResolutionContext): string | null { + for (const target of targets.declared) { + const hit = indexedTarget(target, language, context) ?? builtSourceOf(target, targets.packageDir, language, context); + if (hit) return hit; + } + return null; +} + +/** `target`, build output the index lacks, as the indexed source it is built from (see builtSourcesOf); null when there is none. */ +function builtSourceOf(target: string, packageDir: string, language: Language, context: ResolutionContext): string | null { + for (const source of builtSourcesOf(target, packageDir)) { + const hit = indexedTarget(source, language, context); + if (hit) return hit; + } + return null; +} + +/** + * `target` as a file the index holds: as written, with the language's + * extensions or index files, or as the source of an emitted `.js`. Only the + * index counts: build output a developer has built locally is on disk, but + * nothing links to it. + */ +function indexedTarget(target: string, language: Language, context: ResolutionContext): string | null { + const indexed = (file: string): boolean => context.isIndexedFile?.(file) ?? context.fileExists(file); + const asIs = indexedSpelling(target, language); + if (indexed(asIs)) return asIs; + for (const ext of EXTENSION_RESOLUTION[language] || []) { + const candidate = indexedSpelling(target + ext, language); + if (indexed(candidate)) return candidate; + } + return findSourceForEmittedSpecifier(asIs, language, context, indexed); +} + +/** A package's build folder: `dist/`, `build/`, `lib/`, `out/`, `es/`, `esm/`, `cjs/`. */ +const BUILD_ROOT = /^(?:dist|build|lib|out|output|es|esm|cjs|umd)\//; +/** The format subfolder a bundler adds below it (`dist/esm/`, `dist/types/`), or the source folder tsc mirrors (`dist/src/`). */ +const BUILD_FORMAT = /^(?:esm|cjs|es|mjs|module|modules|commonjs|types|typings|umd|node|browser|src)\//; +/** An emitted file's extension, declaration files included. */ +const EMITTED_EXTENSION = /(?:\.d)?\.(?:[cm]?[jt]s|[jt]sx)$/; + +/** + * The source files a package's build output may be built from, + * extension-less, most literal first: the build folder mirrored under + * `src/` (vite's `dist/node/index.js` → `src/node/index`), then without a + * format subfolder (`dist/esm/index.mjs`, `dist/types/index.d.ts` → + * `src/index`). The caller links one only if the index holds it, so a + * mapping that is not clear links nothing. Empty for a target outside a + * build folder. + */ +function builtSourcesOf(target: string, packageDir: string): string[] { + const prefix = packageDir === '' ? '' : packageDir + '/'; + if (!target.startsWith(prefix)) return []; + const inPackage = target.slice(prefix.length); + const root = BUILD_ROOT.exec(inPackage); + if (!root) return []; + const mirrored = inPackage.slice(root[0].length); + const format = BUILD_FORMAT.exec(mirrored); + const stems = [mirrored, ...(format ? [mirrored.slice(format[0].length)] : [])] + .map((rest) => rest.replace(EMITTED_EXTENSION, '')) + .filter((stem) => stem !== '' && !stem.startsWith('src/')); + return [...new Set(stems)].map((stem) => `${prefix}src/${stem}`); +} + /** * A bare specifier `isExternalImport` takes for a package, resolved through * the importing app's own tsconfig / jsconfig — the one nearest the file, @@ -4242,7 +4328,7 @@ export function isBoundToOutOfRepoImport( if (imp.localName !== name) continue; // A package the repository declares itself (`declare module 'x'`) has // its declarations in the repository. - return isExternalImport(imp.source, ref.language, context) && + return isExternalImport(imp.source, ref.language, context, { fromFile: ref.filePath }) && resolveImportPath(imp.source, ref.filePath, ref.language, context) === null && !ambientModuleDeclares(imp.source, imp.exportedName, imp.isDefault, ref, context); } @@ -4315,7 +4401,7 @@ export function importBindingTail(ref: UnresolvedRef, context: ResolutionContext const binding = mappings.find((m) => m.localName === name) ?? (receiver && ref.referenceKind === 'calls' ? localTypeBinding(receiver, ref, context, mappings) : undefined); if (!binding) return undefined; - if (isExternalImport(binding.source, ref.language, context) && + if (isExternalImport(binding.source, ref.language, context, { fromFile: ref.filePath }) && // A monorepo app's own tsconfig alias (`#/views/…`): isExternalImport reads the root's only. !context.getNearestAliases?.(ref.filePath)?.patterns.some((p) => p.prefix !== '' && binding.source.startsWith(p.prefix)) && // A module the app's `baseUrl` or catch-all `paths` reads (`layout/MainLayout`). diff --git a/src/resolution/index.ts b/src/resolution/index.ts index 06d435e49..4093a92d9 100644 --- a/src/resolution/index.ts +++ b/src/resolution/index.ts @@ -45,7 +45,7 @@ import { createYielder, type MaybeYield } from './cooperative-yield'; import { MAX_SOURCE_FILE_SIZE_BYTES } from '../file-limits'; import { loadProjectAliases, type AliasMap } from './path-aliases'; import { findGoModuleForImport, goModulePackageDir, loadGoModule, type GoModule } from './go-module'; -import { loadWorkspacePackages, type WorkspacePackages } from './workspace-packages'; +import { loadWorkspacePackages, workspaceFingerprint, workspacePackageDirOf, type WorkspacePackages } from './workspace-packages'; import { logDebug } from '../errors'; import { stripCommentsForRegex } from './strip-comments'; import { lexicalPathWithinRoot } from '../utils'; @@ -365,6 +365,9 @@ export class ReferenceResolver { * Initialize the resolver (detect frameworks, etc.) */ initialize(): void { + // A full index resolves by the package.json files on disk now, as the + // fingerprint it stamps says, even in a process that read them earlier. + this.workspacePackages = undefined; this.frameworks = detectFrameworks(this.context); this.frameworksByLanguage.clear(); this.clearCaches(); @@ -706,7 +709,7 @@ export class ReferenceResolver { // `"lit/decorators": ["./node_modules/lit/decorators.js"]` lands in // `node_modules`, topcoder's `config` package on its `config/` folder. isOutOfRepoImport: (source, fromFile, language) => { - if (!isExternalImport(source, language, this.context, { aliasPrefixes: false })) return false; + if (!isExternalImport(source, language, this.context, { aliasPrefixes: false, fromFile })) return false; const resolved = resolveImportPath(source, fromFile, language, this.context); if (resolved !== null && this.isIndexedFile(resolved)) return false; return this.isDeclaredOutsidePackage(source, fromFile); @@ -2850,6 +2853,35 @@ export class ReferenceResolver { return pythonRootFingerprint(this.diskContext()); } + /** + * The workspace packages and path dependencies as package.json files + * declare them now, read again from disk, and their fingerprint (see + * `workspaceFingerprint`). The resolver keeps the map for its lifetime, so + * a sync calls this to see a manifest edit; when the map moved, the + * resolver takes the new one and drops every memo resolved through the old + * one, and the caller re-opens what was linked by it. + */ + refreshWorkspacePackages(): string { + const fresh = loadWorkspacePackages(this.projectRoot); + const fingerprint = workspaceFingerprint(fresh); + if (this.workspacePackages === undefined || workspaceFingerprint(this.workspacePackages) !== fingerprint) { + this.workspacePackages = fresh; + this.clearCaches(); + } + return fingerprint; + } + + /** The fingerprint of the workspace map this resolver resolves by (see `workspaceFingerprint`). */ + getWorkspaceFingerprint(): string { + return workspaceFingerprint(this.context.getWorkspacePackages?.() ?? null); + } + + /** The folder of the workspace package or path dependency `file` is in (see `workspacePackageDirOf`), or null. */ + workspacePackageDirOf(file: string): string | null { + const ws = this.context.getWorkspacePackages?.(); + return ws ? workspacePackageDirOf(ws, file) : null; + } + /** What a sync must re-open after these Python files were added or removed (see `pythonReopenScope`). */ getPythonReopenScope(changedFiles: string[]): ReturnType { return pythonReopenScope(this.diskContext(), changedFiles); diff --git a/src/resolution/module-importers.ts b/src/resolution/module-importers.ts index e1ff0a6e3..f6a82da32 100644 --- a/src/resolution/module-importers.ts +++ b/src/resolution/module-importers.ts @@ -161,6 +161,9 @@ export function reopenReferencesThroughRemovedFiles( /** Resolves a module specifier written in `fromFile` as its import does: the project file it names, or null. */ export type ModuleResolver = (specifier: string, fromFile: string, language: Language) => string | null; +/** The folder of the project's own package (a workspace member, a path dependency) `file` is in, or null. */ +export type PackageDirOf = (file: string) => string | null; + /** * The other half: after a sync's retries, re-open the references the files * importing any of `added` make through those imports — the edges a name @@ -170,8 +173,10 @@ export type ModuleResolver = (specifier: string, fromFile: string, language: Lan * them. (The failed-import retry looks those up by the module's name, and * skips a name as common as `models` or `test`.) With `resolveModule`, also an * import that resolved to a file beside an added one that now outranks it - * (`./button` to `button.vue` while `button.ts` was missing), and what that - * statement binds. Returns the number re-opened. + * (`./button` to `button.vue` while `button.ts` was missing), or, with + * `packageDirOf`, to another file of the package an added file is in (see + * outrankedImports), and what that statement binds. Returns the number + * re-opened. */ export function reopenReferencesThroughAddedFiles( queries: QueryBuilder, @@ -179,8 +184,9 @@ export function reopenReferencesThroughAddedFiles( added: ReadonlyArray<{ path: string; language: Language }>, resolvedUpToEdgeId: number, resolveModule?: ModuleResolver, + packageDirOf?: PackageDirOf, ): number { - const outranked = resolveModule ? outrankedImports(queries, added, resolveModule) : []; + const outranked = resolveModule ? outrankedImports(queries, added, resolveModule, packageDirOf) : []; const bindings = bindingsOf(importersOf(queries, added, reader, outranked), reader); const failed = [...bindings].flatMap(([file, { modules, names }]) => queries.getFailedReferencesThrough(file, modules, names)); return reopenModuleEdges(queries, outranked) + @@ -430,17 +436,38 @@ interface OutrankedImport { * another extension of its name, or its folder's own entry — when the * import's specifier now names the added file: `./button` resolved to * `button.vue` while `button.ts` was missing, and `button.ts` comes first. + * With `packageDirOf`, also the imports of a package by its name into any + * file of the package an added file is in: the entry its manifest declares + * (`./src/index.ts`, or the source of `./dist/index.js`), or the source of a + * path into its build output, outranks the file an import fell back to while + * it was missing, such as the folder's own index (#2502). */ function outrankedImports( queries: QueryBuilder, added: ReadonlyArray<{ path: string; language: Language }>, resolveModule: ModuleResolver, + packageDirOf?: PackageDirOf, ): OutrankedImport[] { const out: OutrankedImport[] = []; + const packageImports = new Map>(); for (const file of added) { if (!BINDING_LANGUAGES.has(file.language)) continue; const { dir, name } = importedName(file.path); - for (const edge of queries.getModuleImportsNamed(dir, name)) { + const edges = queries.getModuleImportsNamed(dir, name); + const packageDir = packageDirOf?.(file.path); + if (packageDir) { + let imports = packageImports.get(packageDir); + if (!imports) { + imports = queries.getModuleImportsUnder(packageDir).filter((edge) => { + const specifier = edge.metadata?.refName; + return typeof specifier === 'string' && !specifier.startsWith('.') && !specifier.startsWith('/'); + }); + packageImports.set(packageDir, imports); + } + const seen = new Set(edges.map((edge) => edge.edgeId)); + for (const edge of imports) if (!seen.has(edge.edgeId)) edges.push(edge); + } + for (const edge of edges) { const specifier = edge.metadata?.refName; if (edge.targetFilePath === file.path || typeof specifier !== 'string') continue; if (resolveModule(specifier, edge.sourceFilePath, edge.sourceLanguage) === file.path) { diff --git a/src/resolution/workspace-packages.ts b/src/resolution/workspace-packages.ts index 0d1e2de8e..5594de535 100644 --- a/src/resolution/workspace-packages.ts +++ b/src/resolution/workspace-packages.ts @@ -11,27 +11,37 @@ * `0 callers` on a live component (issue #629). * * This module maps each member package's declared `name` to its - * directory so the resolver can rewrite `@scope/ui/widgets` → - * `packages/ui/widgets` and then run normal extension/index resolution. + * directory, and its package.json's declared entries (`exports`, `main`, + * `module`, `types`) to files, so the resolver can rewrite `@scope/ui` → + * `packages/ui/src/index.ts` and `@scope/ui/widgets` → the file its + * `exports` names, or `packages/ui/widgets` with normal extension/index + * resolution when the manifest names none (#2502). * - * Scope deliberately small for v1 (mirrors path-aliases.ts): * - reads `workspaces` (array OR `{ packages: [...] }`) from package.json, * plus a minimal `pnpm-workspace.yaml` `packages:` list * - expands `*`, `?` and `{a,b}` within a folder name and `**` across * folders (`packages/*`, pnpm's nested `packages/**\/*`), and leaves out * what a negated glob (`!**\/example`) matches - * - subpath resolution is directory-based (`@scope/ui/sub` → `/sub`); - * it does NOT yet honour a member's `exports` map or `main` field - * - returns null when the project declares no workspaces, so single- + * - reads the sibling packages a package.json depends on by path + * (`"acme-shared": "file:../common"`, `link:`, `portal:`), which need no + * workspace declaration at all: the root manifest's for every file, any + * other's for the files under it, as npm installs a package's + * dependencies for that package (#2456) + * - returns null when the project declares none of these, so single- * package repos pay nothing and see no behaviour change. */ +import { createHash } from 'crypto'; import * as fs from 'fs'; import * as path from 'path'; import { logDebug } from '../errors'; export interface WorkspacePackages { - /** Member package `name` → directory relative to projectRoot (posix). */ + /** + * Package `name` → directory relative to projectRoot (posix): the workspace + * members, ohpm's `file:` dependencies, and the root package.json's path + * dependencies. + */ byName: Map; /** * Member package `name` → its declared ENTRY FILE relative to projectRoot @@ -40,15 +50,64 @@ export interface WorkspacePackages { * the member's real barrel even when it doesn't follow an index-file * convention — and independent of the CONSUMER's language (a `.ts` file * importing an `.ets` barrel resolves without `.ets` in the TS candidate - * list). Absent for npm/pnpm members (their index conventions cover it). + * list). npm/pnpm packages use `entriesByDir`. */ entryByName?: Map; + /** + * The entries an npm/pnpm/yarn/bun package's package.json declares, by the + * package's directory (see PackageEntries). + */ + entriesByDir?: Map; + /** + * The path dependencies a package.json below the root declares: its + * directory → dependency name → target directory (all projectRoot-relative + * posix). Seen only from the files under that manifest — a test fixture's + * `"left-pad": "file:../stub"` is the fixture's own, and `left-pad` stays the + * npm package for everyone else. The root manifest's are in `byName`. + */ + scopedByName?: Map>; +} + +/** + * What a package.json declares a package's importers reach, as targets + * relative to projectRoot (posix), best candidate first. A target may name + * build output the index does not hold; the resolver then reads the source it + * is built from, where that is clear. + */ +export interface PackageEntries { + /** + * The `exports` map: `.`, `./utils`, `./schema/*` → targets. A string, an + * array or a conditions-only `exports` is the `.` entry. A key mapped to + * `null` (not exported) holds no target. + */ + exports?: Map; + /** + * The package root's `source`, `module`, `main`, `types` and `typings` + * fields, read after `exports` (as TypeScript's older `node` resolution + * reads them in its place). + */ + main?: string[]; +} + +/** Where an import of a workspace or path-dependency package lands, before any file is probed. */ +export interface WorkspaceImportTargets { + /** + * The entries the package declares for the import, best first: what + * `exports` names for it in condition order, then, for the package root, + * its `main`-style fields. + */ + declared: string[]; + /** The package's directory plus the import's subpath (`packages/ui/widgets`), extension-less. */ + directory: string; + /** The package's directory. */ + packageDir: string; } /** * Load workspace member packages for `projectRoot`. Returns `null` when - * the project declares no workspaces (the common single-package case) — - * callers then skip all workspace logic. + * the project declares no workspaces and no package depends on another by + * path (the common single-package case) — callers then skip all workspace + * logic. * * Cheap to call repeatedly only via the resolver's per-instance cache; * this function itself touches the filesystem, so the resolver memoises it @@ -75,19 +134,105 @@ export function loadWorkspacePackages(projectRoot: string): WorkspacePackages | // oh-package.json5 declares its local siblings as `"data": "file:../../ // core/data"` dependencies, and code then imports the bare name // (`import { CartRepository } from "data"`). Same monorepo problem as npm - // workspaces, different manifest. + // workspaces, different manifest. An npm package does the same with + // `"acme-shared": "file:../common"` and no root `workspaces` (#2456). + const { ohpm, npm } = collectPathDeps(projectRoot); const entryByName = new Map(); - for (const [name, dir] of collectOhpmFileDeps(projectRoot)) { + for (const [name, dir] of ohpm) { if (byName.has(name)) continue; byName.set(name, dir); const entry = readOhpmMain(projectRoot, dir); if (entry) entryByName.set(name, entry); } + for (const [name, dir] of npm.get('') ?? []) { + if (!byName.has(name)) byName.set(name, dir); + } + npm.delete(''); - if (byName.size === 0) return null; + if (byName.size === 0 && npm.size === 0) return null; - logDebug('workspace packages loaded', { count: byName.size }); - return { byName, entryByName: entryByName.size > 0 ? entryByName : undefined }; + // What each npm-style package's manifest declares for its importers. + const entriesByDir = new Map(); + const packageDirs = new Set(byName.values()); + for (const names of npm.values()) for (const dir of names.values()) packageDirs.add(dir); + for (const dir of packageDirs) { + const entries = readPackageEntries(projectRoot, dir); + if (entries) entriesByDir.set(dir, entries); + } + + logDebug('workspace packages loaded', { count: byName.size, scoped: npm.size, entries: entriesByDir.size }); + return { + byName, + entryByName: entryByName.size > 0 ? entryByName : undefined, + entriesByDir: entriesByDir.size > 0 ? entriesByDir : undefined, + scopedByName: npm.size > 0 ? npm : undefined, + }; +} + +/** + * Condition keys an `exports` target is read through first: the ones + * TypeScript and the bundlers resolve by. Any other condition (`browser`, + * `development`, a custom `@acme/source`) comes next, and `types`/`typings` + * last — a declaration file is the package as tsc reads it, the + * implementation is what its importers run. + */ +const RESOLVING_CONDITIONS = new Set(['import', 'require', 'node', 'module', 'module-sync', 'default']); +const TYPE_CONDITIONS = new Set(['types', 'typings']); + +/** An `exports` value's targets, best first: a string, a fallback array, or (nested) conditions, by key order within each tier. */ +function exportTargets(value: unknown): string[] { + if (typeof value === 'string') return [value]; + if (Array.isArray(value)) return value.flatMap(exportTargets); + if (!value || typeof value !== 'object') return []; + const conditions = Object.entries(value as Record); + const tier = (key: string): number => (RESOLVING_CONDITIONS.has(key) ? 0 : TYPE_CONDITIONS.has(key) ? 2 : 1); + return [0, 1, 2].flatMap((t) => conditions.filter(([key]) => tier(key) === t).flatMap(([, v]) => exportTargets(v))); +} + +/** The package.json fields an importer of the package root reaches without `exports`, best first. */ +const MAIN_FIELDS = ['source', 'module', 'main', 'types', 'typings']; + +/** Read a package's declared entries (see PackageEntries); null when its manifest declares none. */ +function readPackageEntries(projectRoot: string, dirRel: string): PackageEntries | null { + let pkg: Record | null; + try { + pkg = JSON.parse(fs.readFileSync(path.join(projectRoot, dirRel, 'package.json'), 'utf-8')) as Record | null; + } catch { + return null; + } + if (!pkg || typeof pkg !== 'object') return null; + const inPackage = (target: string): string | null => packageTarget(projectRoot, dirRel, target); + const out: PackageEntries = {}; + const exp = pkg.exports; + if (exp !== undefined && exp !== null) { + const exports = new Map(); + const subpaths = typeof exp === 'object' && !Array.isArray(exp) && Object.keys(exp).some((key) => key.startsWith('.')); + const entries: Array<[string, unknown]> = subpaths + ? Object.entries(exp as Record).filter(([key]) => key.startsWith('.')) + : [['.', exp]]; + for (const [key, value] of entries) { + exports.set(key, [...new Set(exportTargets(value).map(inPackage).filter((t): t is string => t !== null))]); + } + out.exports = exports; + } + const main = MAIN_FIELDS + .map((field) => pkg![field]) + .filter((value): value is string => typeof value === 'string') + .map(inPackage) + .filter((t): t is string => t !== null); + if (main.length > 0) out.main = [...new Set(main)]; + return out.exports || out.main ? out : null; +} + +/** A package-relative target (`./src/index.ts`) → projectRoot-relative posix, or null if it leaves the package. */ +function packageTarget(projectRoot: string, dirRel: string, target: string): string | null { + const trimmed = target.trim(); + if (!trimmed) return null; + const packageDir = path.resolve(projectRoot, dirRel); + const abs = path.resolve(packageDir, trimmed); + const inside = path.relative(packageDir, abs); + if (inside.startsWith('..') || path.isAbsolute(inside)) return null; + return path.relative(projectRoot, abs).replace(/\\/g, '/'); } /** @@ -124,11 +269,18 @@ function readOhpmMain(projectRoot: string, dirRel: string): string | null { * cross-module link. Registry dependencies (`@ohos/axios: "^2.0.0"`) don't * use `file:` and are ignored, staying external. * + * The same walk reads every package.json's `file:`, `link:` and `portal:` + * dependencies (#2456), by the directory of the manifest that declares them + * (`''` for the root): npm installs a package's dependencies for that package, + * so the files under it are the ones that see them. Only a directory of the + * project counts, other than the root and nothing under `node_modules` — a + * tarball or a path into installed packages maps nothing. + * * The walk is bounded (depth + directory budget) and prunes build/dependency - * dirs, so non-ArkTS projects pay one readdir at the root and nothing else - * (they have no oh-package.json5 anywhere shallow). + * dirs; a manifest is parsed only where the walk finds one. */ const OHPM_MANIFEST = 'oh-package.json5'; +const NPM_MANIFEST = 'package.json'; const OHPM_WALK_MAX_DEPTH = 6; const OHPM_WALK_DIR_BUDGET = 8000; const OHPM_SKIP_DIRS = new Set([ @@ -136,9 +288,10 @@ const OHPM_SKIP_DIRS = new Set([ 'build', 'dist', 'out', 'oh-package-lock.json5', ]); -function collectOhpmFileDeps(projectRoot: string): Map { - const byName = new Map(); +function collectPathDeps(projectRoot: string): { ohpm: Map; npm: Map> } { + const ohpm = new Map(); const ambiguous = new Set(); + const npm = new Map>(); const queue: Array<{ rel: string; depth: number }> = [{ rel: '', depth: 0 }]; let visited = 0; @@ -161,6 +314,18 @@ function collectOhpmFileDeps(projectRoot: string): Map { queue.push({ rel: rel ? `${rel}/${e.name}` : e.name, depth: depth + 1 }); continue; } + if (e.name === NPM_MANIFEST) { + const deps = new Map(); + for (const [name, target] of readNpmPathDeps(path.join(abs, e.name))) { + const targetAbs = path.resolve(abs, target); + const targetRel = path.relative(projectRoot, targetAbs).replace(/\\/g, '/'); + if (targetRel === '' || targetRel.startsWith('..') || path.isAbsolute(targetRel)) continue; + if (/(?:^|\/)node_modules(?:\/|$)/.test(targetRel) || !isDirectory(targetAbs)) continue; + deps.set(name, targetRel); + } + if (deps.size > 0) npm.set(rel, deps); + continue; + } if (e.name !== OHPM_MANIFEST) continue; const deps = readOhpmFileDeps(path.join(abs, e.name)); @@ -168,18 +333,47 @@ function collectOhpmFileDeps(projectRoot: string): Map { const targetAbs = path.resolve(abs, target); const targetRel = path.relative(projectRoot, targetAbs).replace(/\\/g, '/'); if (targetRel.startsWith('..')) continue; // escapes the project - const existing = byName.get(name); + const existing = ohpm.get(name); if (existing === undefined) { - if (!ambiguous.has(name)) byName.set(name, targetRel); + if (!ambiguous.has(name)) ohpm.set(name, targetRel); } else if (existing !== targetRel) { - byName.delete(name); + ohpm.delete(name); ambiguous.add(name); } } } } - return byName; + return { ohpm, npm }; +} + +function isDirectory(abs: string): boolean { + try { + return fs.statSync(abs).isDirectory(); + } catch { + return false; + } +} + +/** A package.json's dependencies on a path (`file:`, `link:`, `portal:`), from every dependency field → [name, target]. */ +function readNpmPathDeps(manifestAbs: string): Array<[string, string]> { + let pkg: Record | null; + try { + pkg = JSON.parse(fs.readFileSync(manifestAbs, 'utf-8')) as Record | null; + } catch { + return []; + } + if (!pkg || typeof pkg !== 'object') return []; + const out: Array<[string, string]> = []; + for (const field of ['dependencies', 'devDependencies', 'optionalDependencies', 'peerDependencies']) { + const deps = pkg[field]; + if (!deps || typeof deps !== 'object') continue; + for (const [name, value] of Object.entries(deps as Record)) { + const m = typeof value === 'string' ? /^(?:file|link|portal):(.+)$/.exec(value.trim()) : null; + if (m && m[1]!.trim()) out.push([name, m[1]!.trim()]); + } + } + return out; } /** Parse one oh-package.json5's dependencies → [name, file-target] pairs. */ @@ -208,32 +402,145 @@ function readOhpmFileDeps(manifestAbs: string): Array<[string, string]> { * Rewrite a bare workspace import to a path relative to projectRoot, * WITHOUT an extension — the caller applies the language's extension/index * resolution. `@scope/ui/widgets` → `packages/ui/widgets`; the bare package - * name `@scope/ui` → its directory. Returns `null` when no member package - * name matches. + * name `@scope/ui` → its directory, or the ohpm entry its manifest names. + * `fromFile`, the importing file, sees the path dependencies the manifests + * above it declare. Returns `null` when no package name matches. */ export function resolveWorkspaceImport( importPath: string, - ws: WorkspacePackages + ws: WorkspacePackages, + fromFile?: string ): string | null { - // Longest matching package name wins, so `@scope/ui/core` prefers a - // `@scope/ui/core` package over a `@scope/ui` one when both exist. - let bestName: string | null = null; - for (const name of ws.byName.keys()) { - if (importPath === name || importPath.startsWith(name + '/')) { - if (!bestName || name.length > bestName.length) bestName = name; - } - } - if (!bestName) return null; - const dir = ws.byName.get(bestName)!; - const subpath = importPath.slice(bestName.length); // '' or '/widgets' + const hit = matchPackage(importPath, ws, fromFile); + if (!hit) return null; + const subpath = importPath.slice(hit.name.length); // '' or '/widgets' // A bare member import resolves straight to the member's declared entry - // file when the manifest names one (ohpm `main`) — the caller's exact-path - // check hits it without extension/index guessing. - if (!subpath) { - const entry = ws.entryByName?.get(bestName); - if (entry) return entry; + // file when the ohpm manifest names one — the caller's exact-path check + // hits it without extension/index guessing. + const ohpm = subpath ? undefined : ws.entryByName?.get(hit.name); + return ohpm ?? (hit.dir + subpath).replace(/\/{2,}/g, '/'); +} + +/** + * Where an import of a package of the project lands (see + * WorkspaceImportTargets): what its manifest declares for the import, and + * the directory form. Null when no package name matches. + */ +export function workspaceImportTargets( + importPath: string, + ws: WorkspacePackages, + fromFile?: string +): WorkspaceImportTargets | null { + const hit = matchPackage(importPath, ws, fromFile); + if (!hit) return null; + const subpath = importPath.slice(hit.name.length); // '' or '/widgets' + const directory = (hit.dir + subpath).replace(/\/{2,}/g, '/'); + const ohpm = subpath ? undefined : ws.entryByName?.get(hit.name); + const declared = ohpm ? [ohpm] : declaredEntries(ws.entriesByDir?.get(hit.dir), subpath); + return { declared, directory, packageDir: hit.dir }; +} + +/** + * The package an import names: a path dependency of the nearest manifest + * above `fromFile` that declares a matching name, else a workspace member or + * a root path dependency. The longest name wins, so `@scope/ui/core` prefers + * a `@scope/ui/core` package over a `@scope/ui` one when both exist. + */ +function matchPackage(importPath: string, ws: WorkspacePackages, fromFile?: string): { name: string; dir: string } | null { + const scoped = scopedPackage(importPath, ws, fromFile); + if (scoped) return scoped; + const name = longestName(importPath, ws.byName.keys()); + return name === null ? null : { name, dir: ws.byName.get(name)! }; +} + +/** The path dependency the nearest declaring manifest above `fromFile` maps `importPath` to. */ +function scopedPackage(importPath: string, ws: WorkspacePackages, fromFile?: string): { name: string; dir: string } | null { + if (!fromFile || !ws.scopedByName) return null; + let dir = fromFile.replace(/\\/g, '/'); + while (dir !== '') { + const slash = dir.lastIndexOf('/'); + dir = slash < 0 ? '' : dir.slice(0, slash); + const names = ws.scopedByName.get(dir); + if (!names) continue; + const name = longestName(importPath, names.keys()); + if (name !== null) return { name, dir: names.get(name)! }; } - return (dir + subpath).replace(/\/{2,}/g, '/'); + return null; +} + +function longestName(importPath: string, names: Iterable): string | null { + let best: string | null = null; + for (const name of names) { + if ((importPath === name || importPath.startsWith(name + '/')) && (best === null || name.length > best.length)) best = name; + } + return best; +} + +/** + * The targets a package's manifest declares for `subpath` (`''` or + * `/widgets`), best first: its `exports` entry, then, for the package root, + * its `main`-style fields. + */ +function declaredEntries(entries: PackageEntries | undefined, subpath: string): string[] { + if (!entries) return []; + const exported = entries.exports ? exportedTargets(entries.exports, '.' + subpath) : []; + return subpath ? exported : [...new Set([...exported, ...(entries.main ?? [])])]; +} + +/** `exports`' targets for `key`: the exact key's, else the `*` pattern's with the longest prefix, its `*`s filled in. */ +function exportedTargets(exports: Map, key: string): string[] { + const exact = exports.get(key); + if (exact) return exact; + let best: { prefix: string; match: string; targets: string[] } | null = null; + for (const [pattern, targets] of exports) { + const star = pattern.indexOf('*'); + if (star < 0 || pattern.indexOf('*', star + 1) >= 0) continue; + const prefix = pattern.slice(0, star); + const suffix = pattern.slice(star + 1); + if (key.length < prefix.length + suffix.length || !key.startsWith(prefix) || !key.endsWith(suffix)) continue; + if (best && prefix.length <= best.prefix.length) continue; + best = { prefix, match: key.slice(prefix.length, key.length - suffix.length), targets }; + } + return best ? best.targets.map((t) => t.split('*').join(best!.match)) : []; +} + +/** + * The workspace map as a stable string: every name, scope, directory and + * declared entry an import can be resolved by. Two loads that agree on it + * resolve every package import the same way; sync compares it with the one + * the index was built with to see a package.json edit (#2456, #2502). + */ +export function workspaceFingerprint(ws: WorkspacePackages | null): string { + if (!ws) return 'none'; + const sorted = (map: Map | undefined): Array<[string, V]> => + map ? [...map.entries()].sort(([a], [b]) => (a < b ? -1 : a > b ? 1 : 0)) : []; + const canonical = JSON.stringify({ + byName: sorted(ws.byName), + entryByName: sorted(ws.entryByName), + scoped: sorted(ws.scopedByName).map(([dir, names]) => [dir, sorted(names)]), + entries: sorted(ws.entriesByDir).map(([dir, e]) => [dir, sorted(e.exports), e.main ?? []]), + }); + return createHash('sha256').update(canonical).digest('hex'); +} + +/** Each map's package folders, outermost first. */ +const PACKAGE_DIRS = new WeakMap(); + +/** + * The folder of the package `file` is in — a workspace member, or a path + * dependency's target — the outermost when packages nest, or null. Every + * import of a package by its name lands in its folder, inner packages' + * files included. + */ +export function workspacePackageDirOf(ws: WorkspacePackages, file: string): string | null { + let dirs = PACKAGE_DIRS.get(ws); + if (!dirs) { + const all = new Set(ws.byName.values()); + for (const names of ws.scopedByName?.values() ?? []) for (const dir of names.values()) all.add(dir); + dirs = [...all].filter((dir) => dir !== '').sort((a, b) => a.length - b.length); + PACKAGE_DIRS.set(ws, dirs); + } + return dirs.find((dir) => file.startsWith(dir + '/')) ?? null; } /** Read workspace glob patterns from package.json + pnpm-workspace.yaml. */