GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — Load JavaScript modules from Node packages by billti · Pull Request #7075 · microsoft/TypeScript

Conversation

@billti

@billti Bill Ticehurst (billti) commented Feb 14, 2016

Copy link
Copy Markdown
Member

I'm not looking to push this now, but opening for discussion on both a) the approach in the implementation, and b) some design questions it raises.

This change would allow for the loading of JavaScript modules from a search of top level Node packages. This could be useful in TypeScript, but is especially useful in editing JavaScript.

To avoid transitively loading 100s of JavaScript files from an NPM installed dependency graph, this sets a limit on how deep in the dependency chain to crawl (which only applies to JavaScript, not TypeScript files). For example, if chalk depends on has-ansi which depends on ansi-regex, then setting maxNodeModuleJsDepth to 0 (the default currently) wouldn't load JavaScript modules for any of them. Setting to 1 would load only the JavaScript modules for chalk, to 2 would load has-ansi also, etc.. If a module exposes imports from a dependency, then a greater search depth flows up richer type information (e.g. packages such as gulp commonly reexport members of their imported dependencies, such as vinyl-fs).

In order to avoid recompiling/overwriting JavaScript files installed under node_modules, this change also suppresses emitting any code for files found with a search of installed Node packages. This is a change in behavior (as can be seen in some baseline changes below), but what is desirable here is up for debate. It seems bad if you end up renaming a property in a dependency you installed from NPM (which will be reverted on next npm install, likely resulting in a mismatch of property names), yet some folks also npm link across multiple projects under development in order to treat the whole as a "solution".

Vladimir Matveev (@vladima) The approach below no longer modifies the ResolvedModule type (it uses the existing isExternalLibraryImport flag), but does add a member to SourceFile. Let me know if this requires managed side changes also.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

nissoh reacted with thumbs up emoji ivanz reacted with hooray emoji clark-stevenson reacted with heart emoji
All reactions
  • 👍 1 reaction
  • 🎉 1 reaction
  • ❤️ 1 reaction
Bill Ticehurst (billti) added 3 commits February 13, 2016 07:48
@billti
Load JS from node_modules
dfb0dcd
@billti
Added tests
6126f7b
@billti
Updated tests
e2df645
@billti Bill Ticehurst (billti) added the Salsa label Feb 14, 2016
@billti Bill Ticehurst (billti) self-assigned this Feb 14, 2016
@billti Bill Ticehurst (billti) added this to the TypeScript 2.0 milestone Feb 14, 2016
@msftclas Microsoft Contribution License Agreements (msftclas) added the cla-not-required label Feb 14, 2016
@Bigous

Richard Natal (Bigous) commented Feb 17, 2016

Copy link
Copy Markdown

This is awesome, I think it could be added to 1.9 or 1.8 ... very, very useful.
I think depth 2 would be enough for the majority. So it could be the default.
Cache the findings of Salsa as a pseudo .d.ts (or a real .d.ts in typings/generated) could be great too.
Another approach could be a simple line command option to generate all .d.ts from node_modules that does not have .d.ts or .ts itself. It could help people to write the .d.ts from packages without having to start from 0.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

This was referenced Feb 18, 2016
Expecting index.js to be resolved when require a directory #7129
Closed
[Salsa] load .js files from node packages #6670
Closed
@massimocode

massimocode commented Feb 18, 2016

Copy link
Copy Markdown

I've not looked at the code but this looks like a very sensible/useful idea

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@Bigous Richard Natal (Bigous) mentioned this pull request Feb 22, 2016
Typescript definition file googleapis/google-api-nodejs-client#503
Closed
mhegazy
mhegazy reviewed Mar 9, 2016
View reviewed changes
Comment thread src/compiler/program.ts Outdated
const i = moduleName.lastIndexOf("./", 1);
const startsWithDotSlashOrDotDotSlash = i === 0 || (i === 1 && moduleName.charCodeAt(0) === CharacterCodes.dot);
return !startsWithDotSlashOrDotDotSlash;
return !startsWithDotSlashOrDotDotSlash.test(moduleName);

@mhegazy Mohamed Hegazy (mhegazy) Mar 9, 2016

Copy link
Copy Markdown
Contributor

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The idea here is that looking up the fist 2 chars is cheeper than running a regexp. is there a reason why we need to change it?

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@mhegazy

Mohamed Hegazy (mhegazy) commented Mar 9, 2016

Copy link
Copy Markdown
Contributor

a few minor comments. other than that looks good to me.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@billti Bill Ticehurst (billti) mentioned this pull request Mar 16, 2016
Unclear on how to use allowJS with untyped modules installed with npm #7542
Closed
@raybooysen

Ray Booysen (raybooysen) commented Mar 17, 2016

Copy link
Copy Markdown

Mohamed Hegazy (@mhegazy) Is this still on for a 2.0 release? Is it too late for a 1.9?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@goldenc

Cathal Golden (goldenc) commented Mar 17, 2016

Copy link
Copy Markdown

I could really use this getting into the 1.9 release.. pretty please!

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@dunkymole

Duncan Mole (dunkymole) commented Mar 17, 2016

Copy link
Copy Markdown

a 1.9 release would help us tremendously also

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@jonclare

Jon Clare (jonclare) commented Mar 17, 2016

Copy link
Copy Markdown

+1 for getting this in the 1.9 release - would be a great help for us.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@bnwan

Benjamin Nwaneampeh (bnwan) commented Mar 17, 2016

Copy link
Copy Markdown

It would be great if this made it to 1.9

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@sarbbottam

Sarbbottam Bandyopadhyay (sarbbottam) commented Apr 25, 2016

Copy link
Copy Markdown

Bill Ticehurst (@billti) could you resolve the conflicts?

When could this PR be merged? I look forward to see #6670 resolved

cc/ egamma

Thanks!

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@mhegazy

Mohamed Hegazy (mhegazy) commented May 11, 2016

Copy link
Copy Markdown
Contributor

Bill Ticehurst (@billti) is this ready to go? can you update?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@billti

Bill Ticehurst (billti) commented May 12, 2016

Copy link
Copy Markdown
Member Author

I'll try and get to it later tonight or tomorrow... just got a couple of other tasks ahead of it.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@mhegazy Mohamed Hegazy (mhegazy) mentioned this pull request May 17, 2016
{ "moduleResolution": "node" } support incomplete; doesn't understand package.json files #8527
Closed
Bill Ticehurst (billti) added 4 commits May 23, 2016 17:00
@billti
Merge remote-tracking branch 'upstream/master' into loadJsFromModules
c76dc72
@billti
Fixed diagnostic message wording
ac54194
@billti
Merge remote-tracking branch 'upstream/master' into loadJsFromModules
75d84dc
@billti
Updated test
dd31bf9
@billti

Bill Ticehurst (billti) commented Jun 21, 2016

Copy link
Copy Markdown
Member Author

They say necessity is the mother of invention... by rethinking how to handle this with the new requirement that a SourceFile might be reused across programs, I think this is actually much simpler now.

Mohamed Hegazy (@mhegazy), can you take a look and let me know if you spot any flaws with this approach?

I don't handle the suppression of emit if the file is located under node_modules, but that is a separate issue that is present already. I can add that and more tests if desired before checking in.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@billti

Bill Ticehurst (billti) commented Jun 21, 2016

Copy link
Copy Markdown
Member Author

Actually, trying to solve the emit problem I realized where this falls down in the general case.

If module A imports B imports C, and is doesn't add C when processing B because it's too many hops, if I then later directly import B, is will already have processed the file and won't reprocess imports and see that C is now not too far and should be added. 😢 Back to the drawing board...

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@mhegazy Mohamed Hegazy (mhegazy) mentioned this pull request Jun 22, 2016
importing JavaScript containing JsDoc comments #9317
Closed
@mhegazy

Mohamed Hegazy (mhegazy) commented Jun 22, 2016

Copy link
Copy Markdown
Contributor

👍

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@mhegazy

Mohamed Hegazy (mhegazy) commented Jun 22, 2016

Copy link
Copy Markdown
Contributor

Can you also add a test for --maxNodeModuleJsDepth

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

Bill Ticehurst (billti) added 5 commits June 26, 2016 08:45
@billti
Remove extra baselines
6fba804
@billti
Fixed bugs and linting
e9a0c56
@billti
Added project tests for node_modules JavaScript searches
885b0e9
@billti
Merge branch 'master' into loadJsFromModules
0174bda
@billti
Removed old TODO comment
1b43bd8
@billti

Bill Ticehurst (billti) commented Jun 27, 2016 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown
Member Author

Mohamed Hegazy (@mhegazy) This should be pretty comprehensive now. It tracks if a JavaScript file was loaded from searching node_modules so it knows not to compile it (a minor tweak to this logic could also solve #6964 if desired). It also tracks if a module has some imports skipped, so that if the same module is later processed higher up the import graph it will re-processes the imports.

I added a few project tests to test a) the eliding once the max depth is reached,b) increasing the depth via the config option, and c) the reprocessing of imports if a module is later imported higher up the chain. I used Project tests as with baselines its hard to add files that exist but aren't root files (as the files under node_modules need to be imported but not root files). The project tests also contain useful info about what files were input, resolved, and output.

Let me know if you have any other suggestions for improvements.

p.s. I had to fix a regular expression to get the Project tests working with the .js files (else it was putting a full path from my machine in the error baselines), but that did change a lot of other baselines (for the better).

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@billti
Fixed the regexp for removing full paths
de559fb
billti
billti reviewed Jun 27, 2016
View reviewed changes
Comment thread tests/baselines/reference/library-reference-5.errors.txt
/// <reference types="alpha" />
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
!!! message TS4090: Conflicting library definitions for 'alpha' found at 'index.d.ts' and 'index.d.ts'. Copy the correct file to the 'typings' folder to resolve this conflict.
!!! message TS4090: Conflicting library definitions for 'alpha' found at '/node_modules/bar/node_modules/alpha/index.d.ts' and '/node_modules/foo/node_modules/alpha/index.d.ts'. Copy the correct file to the 'typings' folder to resolve this conflict.

@billti Bill Ticehurst (billti) Jun 27, 2016

Copy link
Copy Markdown
Member Author

There was a problem hiding this comment.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixing the regular expression makes messages such as this useful.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@billti
Added emitHost method to return source from node modules
d8e3bd9
@mhegazy

Mohamed Hegazy (mhegazy) commented Jun 27, 2016

Copy link
Copy Markdown
Contributor

👍

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@billti

Bill Ticehurst (billti) commented Jun 27, 2016

Copy link
Copy Markdown
Member Author

How about this change?

Note that I'm using a Map for the collection of file names as this seems quicker on large projects with lots of modules from node_modules, than using a array of file paths and scanning this every time to check for existence (and splicing when I need to remove an entry). I could use a simple string[] if that seems cleaner, but this works well currently.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@mhegazy

Mohamed Hegazy (mhegazy) commented Jun 27, 2016

Copy link
Copy Markdown
Contributor

I would have preferred a function that takes a file path and returns a boolean. but this is fine too.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@mhegazy

Mohamed Hegazy (mhegazy) commented Jun 27, 2016

Copy link
Copy Markdown
Contributor

can you mark this as /* @internal */ since we are the only users for it.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@billti
Marked new method internal
ddad35a
@billti

Bill Ticehurst (billti) commented Jun 27, 2016

Copy link
Copy Markdown
Member Author

Like so?

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@mhegazy

Mohamed Hegazy (mhegazy) commented Jun 28, 2016

Copy link
Copy Markdown
Contributor

thanks!

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@billti

Bill Ticehurst (billti) commented Jun 28, 2016

Copy link
Copy Markdown
Member Author

The build is taking forever :-/ I'm sure it'll pass... but waiting as a precaution.

I'll port and merge this to release-2.0 also unless you have any objections.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@mhegazy

Mohamed Hegazy (mhegazy) commented Jun 28, 2016

Copy link
Copy Markdown
Contributor

no objections. thanks!

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@billti Bill Ticehurst (billti) mentioned this pull request Jun 28, 2016
Load js from modules 2.0 #9384
Merged
@billti
Bill Ticehurst (billti) merged commit 8bf3d54 into master Jun 28, 2016
@billti
Bill Ticehurst (billti) deleted the loadJsFromModules branch June 28, 2016 01:42
@sarbbottam

Sarbbottam Bandyopadhyay (sarbbottam) commented Jun 28, 2016

Copy link
Copy Markdown

👏 Bill Ticehurst (@billti) Mohamed Hegazy (@mhegazy)

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

@mhegazy Mohamed Hegazy (mhegazy) mentioned this pull request Jun 28, 2016
Discovering .ts files under node_modules resulting in them being compiled #6964
Closed
@microsoft Microsoft (microsoft) locked and limited conversation to collaborators Jun 19, 2018
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

No reviews

Uh oh!

There was an error while loading. Please reload this page.

Assignees

@billti Bill Ticehurst (billti)

Labels

None yet

Projects

None yet

Milestone

TypeScript 2.0

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

11 participants

@billti @Bigous @massimocode @mhegazy @raybooysen @goldenc @dunkymole @jonclare @bnwan @sarbbottam @msftclas
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.